981 lines
53 KiB
Markdown
981 lines
53 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 📊 MONITORS
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Observe, measure, and report — never act.** Scripts in this folder watch the system
|
|
and notify when something needs attention. They do not restart containers, do not
|
|
reboot servers, do not delete files. The other folders handle intervention. This folder
|
|
handles awareness.
|
|
|
|
```
|
|
Monitors/ ← observes and reports (this folder)
|
|
Docker_Essentials/ ← acts on containers
|
|
unRAID_Essentials/ ← acts on the server
|
|
Media/ ← maintains the library
|
|
```
|
|
|
|
> **All monitor scripts are safe to run at any time.** They are read-only except for
|
|
> `bandwidth_monitor.sh` (one bounded append per rsync run) and
|
|
> `system_tuning_monitor.sh` (one bounded append every 6 hours). Every other script
|
|
> in this folder writes nothing. None of them can break anything.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### 🔴 Certs Expiring Silently Despite Renewal
|
|
|
|
Certbot runs on schedule. NPM shows the cert as renewed. But the web server was not
|
|
reloaded after renewal — the old expired cert is still being served to external clients.
|
|
NPM's dashboard shows green. `openssl s_client` from the outside shows a cert that
|
|
expired three days ago. Users see certificate errors. You see a healthy NPM status.
|
|
|
|
The fix: `cert_monitor.sh` connects directly via `openssl` — not to NPM's API, not
|
|
to any internal check, but to the actual TLS handshake the outside world sees. It
|
|
catches renewed-but-not-reloaded certs, wrong certs being served, chain issues that
|
|
are invisible from the inside. If users would see an error, this script catches it.
|
|
|
|
---
|
|
|
|
### 🔴 No Early Warning on Drive Degradation
|
|
|
|
A drive accumulates reallocated sectors silently over weeks. The count climbs from 0
|
|
to 12 to 47. Nothing fails. No error in the Docker logs. No alert from unRAID.
|
|
The drive is in progressive failure — heading toward a point of no return — and the
|
|
only way to know is to actively query SMART data. Which nobody does until something
|
|
goes wrong.
|
|
|
|
The fix: `smart_health.sh` queries SMART attributes for every drive every week.
|
|
Reallocated sectors, pending sectors, uncorrectable sectors, temperature — anything
|
|
concerning triggers a notification. Drive failure has a warning curve. Weekly SMART
|
|
checks give you time to act on it.
|
|
|
|
---
|
|
|
|
### 🔴 rsync Succeeds But the Copy Is Wrong
|
|
|
|
`rsync` reports success. Transfer completed. Exit code 0. But a storage hardware
|
|
issue on either server silently corrupted a file during the write. The file exists
|
|
on both ends. rsync is satisfied — sizes match, timestamps match. Nobody knows the
|
|
content is wrong until someone tries to play the file and it fails, or worse, until
|
|
a drive failure reveals the backup was corrupt from the start.
|
|
|
|
The fix: `backup_verify.sh` independently computes MD5 checksums on both sides of
|
|
the rsync mirror and compares them. It does not trust rsync's exit code. It does not
|
|
trust that the file exists. It verifies the actual content matches. Silent data
|
|
corruption is the failure mode backup systems exist to catch — and the only way to
|
|
catch it is to verify independently.
|
|
|
|
---
|
|
|
|
### 🔴 inotify Exhaustion Producing Unexplained Symptoms
|
|
|
|
inotify watches are silently exhausted. The kernel limit is hit. New file events
|
|
stop being delivered — silently. Downloads finish but arr containers don't see them.
|
|
Live TV stutter appears because media files aren't being detected as they arrive.
|
|
Library updates stop happening. None of these produce clear error messages — the
|
|
containers are running, the files are there, nothing is obviously wrong. The symptom
|
|
looks like a network problem or a container bug. It is an inotify problem.
|
|
|
|
The same issue appears with PHP-FPM worker exhaustion — WebGUI requests start
|
|
queueing, timeouts appear, the dashboard feels sluggish. The worker count is at
|
|
ceiling and you have no visibility into how often it happens or how close it gets.
|
|
|
|
The fix: `system_tuning_monitor.sh` snapshots inotify and PHP-FPM utilisation every
|
|
6 hours. The weekly coffee report shows peak, average, and warning count. You see
|
|
the trend before it becomes a problem. `inotify_tuning.sh` and `php_fpm_max_children.sh`
|
|
fix the limits — this script tells you whether the fix is holding.
|
|
|
|
---
|
|
|
|
### 🔴 No Single View of What the Continuous Scripts Are Doing
|
|
|
|
Three scripts run continuously as background processes: `system_watchdog.sh`,
|
|
`docker_watchdog.sh`, and `failover.sh`. Are they running? How long have they been
|
|
up? Are there active strikes against any container? Is failover in a non-NORMAL state?
|
|
Is the skip list empty? You have to check each state file individually, parse the
|
|
format yourself, and mentally correlate the information.
|
|
|
|
The fix: `continuous_scripts_status.sh` — a live dashboard that reads all state files
|
|
and presents a single coherent view. Run it any time to know exactly what the watchdogs
|
|
are doing, what strikes are active, what the failover state is, and how the system
|
|
health looks right now. One command, complete picture.
|
|
|
|
---
|
|
|
|
## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Four distinct observation roles:
|
|
|
|
---
|
|
|
|
### 🔍 Live Connectivity Checks — Active probes against real endpoints
|
|
|
|
`cert_monitor.sh` and `backup_verify.sh` go out and test things. They connect,
|
|
compute, compare. The result is a direct measurement of the actual state of the
|
|
world — not a reading of a cached status.
|
|
|
|
---
|
|
|
|
### 📋 State File Aggregation — Synthesising the ecosystem's own data
|
|
|
|
`weekly_health_digest.sh` and `continuous_scripts_status.sh` read the state files
|
|
the watchdogs and orchestrators have already written. They don't gather new data —
|
|
they organise existing data into human-readable summaries.
|
|
|
|
---
|
|
|
|
### 📈 Time-Series Snapshots — Tracking trends over time
|
|
|
|
`bandwidth_monitor.sh` and `system_tuning_monitor.sh` write one bounded log entry
|
|
per interval. Over time this builds a picture of trends — whether inotify pressure
|
|
is growing, whether transfer sizes are increasing, whether PHP-FPM is consistently
|
|
near ceiling.
|
|
|
|
---
|
|
|
|
### 🖥️ Hardware Health — Checking what the hardware itself reports
|
|
|
|
`smart_health.sh` and `zfs_memory_snapshot.sh` query the hardware and kernel
|
|
directly — drive SMART attributes, ZFS pool state, ARC statistics, kernel memory
|
|
pressure. Things that are invisible in application-level monitoring.
|
|
|
|
---
|
|
|
|
## ━━━ FLASH DRIVE WRITE POLICY ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
unRAID boots from a USB flash drive. Flash drives have limited write cycles.
|
|
Every script in this folder is designed with this constraint in mind.
|
|
|
|
| Script | Writes to flash | Detail |
|
|
|--------|----------------|--------|
|
|
| `cert_monitor.sh` | Never | openssl connections only |
|
|
| `smart_health.sh` | Never | smartctl reads only |
|
|
| `zfs_memory_snapshot.sh` | Never | writes to `/var/log/` (tmpfs) |
|
|
| `backup_verify.sh` | Never | SSH + MD5 comparison only |
|
|
| `emby_session_report.sh` | Never | API queries only |
|
|
| `weekly_health_digest.sh` | Never | reads existing state files only |
|
|
| `continuous_scripts_status.sh` | Never | reads existing state files only |
|
|
| `bandwidth_monitor.sh` | One append + one trim per rsync run | Bounded — never exceeds `BANDWIDTH_LOG_RETENTION` lines |
|
|
| `system_tuning_monitor.sh` | One append + one trim every 6 hours | Bounded — trimmed to `TUNING_LOG_RETENTION` days |
|
|
|
|
The two scripts that write to flash are designed to be minimal and bounded. They
|
|
write one line and trim old entries on every write — the files never grow.
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
| Script | What It Watches | When |
|
|
|--------|----------------|------|
|
|
| `cert_monitor.sh` | SSL cert expiry via live TLS connection | Weekly — Sunday 9am |
|
|
| `smart_health.sh` | Drive SMART health attributes | Weekly — Sunday 7am |
|
|
| `zfs_memory_snapshot.sh` | ZFS pool health, ARC stats, memory pressure | Weekly — Sunday 6am |
|
|
| `backup_verify.sh` | rsync mirror integrity via MD5 checksums | Weekly — Sunday 10am |
|
|
| `bandwidth_monitor.sh` | rsync transfer history and trends | Auto (by rsync.sh) + weekly report |
|
|
| `weekly_health_digest.sh` | Full ecosystem health aggregation | Daily (profile controls notify) |
|
|
| `emby_session_report.sh` | Emby streaming usage and library stats | Weekly — Sunday 11am |
|
|
| `system_tuning_monitor.sh` | inotify + PHP-FPM utilisation trends | Every 6 hours |
|
|
| `continuous_scripts_status.sh` | Live dashboard for all background processes | On demand |
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🔐 cert_monitor.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
SSL certificate expiry monitoring via live TLS connection. Connects directly to each
|
|
domain via `openssl s_client` and reads the certificate actually being served —
|
|
not the certificate file, not NPM's API, not any cached status.
|
|
|
|
```bash
|
|
# Scheduled: 0 9 * * 0 (Sunday 9am weekly)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Direct openssl, Not an API ──────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# API-based cert checks ask the certificate manager if the cert is valid.
|
|
# openssl checks ask the server what cert it is actually serving.
|
|
# These are not the same question and the answers can differ.
|
|
#
|
|
# Direct openssl catches:
|
|
# - Cert renewed in NPM but web server not reloaded (old cert still serving)
|
|
# - Wrong cert being served (subdomain pointing to wrong vhost)
|
|
# - Chain issues visible externally but not internally
|
|
# - NPM reporting healthy while clients see an expired cert
|
|
#
|
|
# If a user would see a certificate error in their browser, this script
|
|
# catches it. If the cert manager says it's fine but the server is serving
|
|
# an old cert, this script catches that too.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Results Per Domain ───────────────────────────────────────────────────────
|
|
|
|
```
|
|
HEALTHY > CERT_WARN_DAYS remaining → silent ✅
|
|
WARNING ≤ CERT_WARN_DAYS remaining → notification
|
|
CRITICAL ≤ CERT_CRIT_DAYS remaining → urgent notification
|
|
FAILED could not connect or parse cert → notification
|
|
```
|
|
|
|
Notifications are batched by severity — one message lists all domains at WARNING,
|
|
a separate message lists all domains at CRITICAL. Not one notification per domain.
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Each server monitors its own domains — detect_hosts() aliases the correct list.
|
|
# Each domain and subdomain is a separate entry — they have independent certs.
|
|
#
|
|
HOST1_CERT_MONITOR_DOMAINS=(
|
|
"Gmer4Lfe.com" # primary domain — checks the actual TLS handshake
|
|
"Gmer4Lfe.us" # secondary domain
|
|
# "auth.Gmer4Lfe.com" # add subdomains separately — they have separate certs
|
|
# "cloud.Gmer4Lfe.com"
|
|
)
|
|
|
|
# master.conf
|
|
CERT_WARN_DAYS=30 # warn this many days before expiry — time to investigate
|
|
CERT_CRIT_DAYS=7 # critical alert — action needed now
|
|
CERT_TIMEOUT=10 # seconds per domain before declaring FAILED
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
cert_monitor.sh # normal run — silent if all healthy
|
|
cert_monitor.sh --dry-run # check all certs and show results, no notification sent
|
|
cert_monitor.sh --log # verbose output per domain
|
|
cert_monitor.sh --status # show domain list and thresholds
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 💾 smart_health.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Drive SMART health monitoring. Queries SMART attributes for every drive on the system
|
|
weekly. Catches drive degradation before it becomes drive failure — most spinning
|
|
drives go through a warning period before dying outright.
|
|
|
|
```bash
|
|
# Scheduled: 0 7 * * 0 (Sunday 7am weekly)
|
|
```
|
|
|
|
---
|
|
|
|
### ── What It Monitors Per Drive ──────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# The attributes that actually indicate drive health problems:
|
|
#
|
|
Reallocated_Sector_Ct any > 0 = warning
|
|
The drive found a bad sector and swapped in a spare.
|
|
0 spare = healthy. 1 spare = warning. More = escalating concern.
|
|
This is the first sign a drive is wearing out.
|
|
|
|
Current_Pending_Sector any > 0 = warning
|
|
Sectors the drive suspects are bad but hasn't confirmed yet.
|
|
May recover on next read. May escalate to Reallocated. Watch it.
|
|
|
|
Offline_Uncorrectable any > 0 = critical
|
|
Sectors the drive tried to correct during offline tests and couldn't.
|
|
No spares. No recovery. Data loss risk.
|
|
|
|
Temperature_Celsius vs thresholds from dynamix.cfg
|
|
High sustained temperature shortens drive life significantly.
|
|
Script reads unRAID's own configured thresholds — no duplication needed.
|
|
|
|
Overall SMART status PASSED / FAILED
|
|
The drive's own self-assessment. FAILED = get data off this drive now.
|
|
|
|
Power_On_Hours informational
|
|
Drive age estimate — useful context for evaluating other attributes.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Drive Discovery + Temperature Source ────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Drives discovered automatically — /dev/sd* and /dev/nvme* on every run.
|
|
# NVMe drives use different attribute names — detected and handled automatically.
|
|
# No drive list to maintain in configuration.
|
|
#
|
|
# Temperature thresholds read from /boot/config/plugins/dynamix/dynamix.cfg:
|
|
# hot, max, hotssd, maxssd — unRAID's own configured values.
|
|
# Falls back to SMART_TEMP_WARN / SMART_TEMP_CRIT from master.conf if not found.
|
|
# This means smart_health.sh and unRAID's dashboard use the same thresholds.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Drives to skip — typically the boot USB flash drive.
|
|
# unRAID boots from USB which appears as sda. Flash drives either don't
|
|
# support SMART or report meaningless values. Skip it.
|
|
#
|
|
HOST1_SMART_IGNORE_DRIVES=(
|
|
"sda" # boot USB flash drive — no meaningful SMART data
|
|
)
|
|
|
|
# master.conf
|
|
SMART_TEMP_WARN=45 # °C — fallback if dynamix.cfg not found
|
|
SMART_TEMP_CRIT=55 # °C — fallback
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
smart_health.sh # normal run — silent if all drives healthy
|
|
smart_health.sh --dry-run # show which drives would be checked, no smartctl
|
|
smart_health.sh --log # verbose — full attribute dump per drive
|
|
smart_health.sh --status # show config and ignore list
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🗄️ zfs_memory_snapshot.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Weekly ZFS pool health and memory diagnostic report. Combines ZFS pool status, ARC
|
|
statistics, Docker memory usage, and kernel memory pressure into a single snapshot.
|
|
Informational only — `system_watchdog.sh` handles threshold-based intervention.
|
|
|
|
```bash
|
|
# Scheduled: 0 6 * * 0 (Sunday 6am weekly — before other Sunday scripts)
|
|
```
|
|
|
|
---
|
|
|
|
### ── What It Reports ─────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# ZFS pool health:
|
|
# Status (ONLINE/DEGRADED/FAULTED) per pool
|
|
# Size, allocated, free, capacity, health per pool
|
|
# Pools in ZFS_REPORT_IGNORE_POOLS are excluded from reporting
|
|
# (they remain fully monitored by unRAID — this only affects the report)
|
|
#
|
|
# ARC statistics:
|
|
# Current ARC size vs max — what percentage of max is being used
|
|
# Metadata ARC — how much ARC is occupied by metadata vs data
|
|
# ARC hit rate — cache effectiveness
|
|
# Warns if ARC utilisation exceeds ZFS_REPORT_ARC_WARN_PCT
|
|
#
|
|
# Memory status:
|
|
# Total, free, available RAM
|
|
# Warns if free < ZFS_REPORT_FREE_WARN_GB
|
|
# Warns if available < ZFS_REPORT_AVAIL_WARN_GB
|
|
#
|
|
# Docker memory:
|
|
# Top ZFS_REPORT_DOCKER_TOP containers by memory usage
|
|
# Useful for spotting containers approaching their watchdog limits
|
|
#
|
|
# Kernel pressure:
|
|
# vmstat snapshot — 3 samples showing CPU and memory activity
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Output Goes to Console and Log ─────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Output is written to both console (for User Scripts output log) and ZFS_REPORT_LOG.
|
|
# The log file lets you compare pool health week over week without having to
|
|
# remember what last week's numbers were. Open it to see the trend.
|
|
# In dry-run mode — console only, nothing written to log.
|
|
#
|
|
ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log" # tmpfs — resets on reboot
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Single-disk ZFS array members can be excluded from the report.
|
|
# They remain fully monitored by unRAID — this only affects the weekly snapshot.
|
|
# Each disk in a JBOD ZFS configuration appears as its own pool — there can be many.
|
|
#
|
|
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
|
"disk10" # JBOD member — high usage expected, exclude from report noise
|
|
"disk9"
|
|
"disk8"
|
|
"disk6"
|
|
"disk5"
|
|
)
|
|
|
|
# master.conf
|
|
ZFS_REPORT_ARC_WARN_PCT=90 # warn if ARC using more than 90% of its max
|
|
ZFS_REPORT_FREE_WARN_GB=10 # warn if free RAM below 10GB
|
|
ZFS_REPORT_AVAIL_WARN_GB=20 # warn if available RAM below 20GB
|
|
ZFS_REPORT_DOCKER_TOP=10 # top 10 Docker containers by memory
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
zfs_memory_snapshot.sh # normal run — writes to log + console
|
|
zfs_memory_snapshot.sh --dry-run # console only, no log write
|
|
zfs_memory_snapshot.sh --log # verbose output
|
|
zfs_memory_snapshot.sh --status # show config and pool ignore list
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## ✅ backup_verify.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
rsync mirror integrity verification via independent MD5 checksums. Randomly samples
|
|
files from each share, computes checksums on both sides, and compares. Does not trust
|
|
rsync's exit code. Verifies the actual content matches.
|
|
|
|
```bash
|
|
# Scheduled: 0 10 * * 0 (Sunday 10am weekly)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Verify Independently ─────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# rsync reports success when:
|
|
# - The transfer completed without network errors
|
|
# - The file sizes and modification times match
|
|
#
|
|
# rsync does NOT detect:
|
|
# - Silent corruption during the transfer (bitflip in transit)
|
|
# - Corruption written to storage at rest (faulty drive sector)
|
|
# - Files that matched size/mtime but had wrong content (rare, happens)
|
|
#
|
|
# These failure modes produce exit code 0. rsync says "done."
|
|
# backup_verify.sh checks whether "done" means "correct."
|
|
#
|
|
# Sample size is intentionally small (10 files per share, default) —
|
|
# this is a spot check, not an exhaustive verify. It catches systematic
|
|
# problems and hardware issues while running in minutes, not hours.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Results Per File ─────────────────────────────────────────────────────────
|
|
|
|
```
|
|
MATCH checksums identical on both sides → correctly mirrored ✅
|
|
MISMATCH file exists on both, checksums differ → sync failure or corruption
|
|
MISSING file exists locally but not on remote → not yet synced or deleted
|
|
```
|
|
|
|
All MISMATCHes and any significant number of MISSINGs trigger a notification.
|
|
|
|
---
|
|
|
|
### ── Pre-flight Checks Before Any SSH ───────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Two checks before computing a single checksum:
|
|
#
|
|
# check_connectivity() — is the remote Tailscale IP reachable?
|
|
# Without this: all files would show as MISSING (SSH timeout per file).
|
|
# False alarm from a network hiccup.
|
|
#
|
|
# check_remote_array() — is the remote /mnt/user mounted?
|
|
# Without this: remote files don't exist yet (array not started).
|
|
# All files would show as MISSING. False catastrophic alarm.
|
|
#
|
|
# Both must pass before any checksums are computed.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
BACKUP_VERIFY_SAMPLE=10 # random files sampled per share per run
|
|
BACKUP_VERIFY_MIN_SIZE="1M" # skip files smaller than this (checksums waste
|
|
# time on tiny files with low corruption risk)
|
|
|
|
# master_host*.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Leave empty to use HOST*_DAILY_SYNC_SHARES automatically.
|
|
# No additional configuration needed for the standard setup.
|
|
#
|
|
HOST1_BACKUP_VERIFY_SHARES=(
|
|
# empty — uses HOST1_DAILY_SYNC_SHARES automatically
|
|
# "/mnt/user/Movies" # override here to check specific shares only
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
backup_verify.sh # normal run — silent if all samples match
|
|
backup_verify.sh --dry-run # show which files would be sampled, no checksums
|
|
backup_verify.sh --log # verbose — show checksum comparison per file
|
|
backup_verify.sh --status # show share list and sample configuration
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 📡 bandwidth_monitor.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
rsync transfer history logging and weekly summary reporting. Called automatically by
|
|
`rsync.sh` after each sync — no manual scheduling needed for the logging part. The
|
|
report can be run manually or on a weekly schedule.
|
|
|
|
```bash
|
|
# Log mode: called automatically by rsync.sh — no manual scheduling needed
|
|
# Report mode — scheduled: 0 11 * * 0 (Sunday 11am weekly)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Two Modes ────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# --log-transfer profile duration_seconds status bytes_transferred
|
|
# Called by rsync.sh after each sync completes.
|
|
# Appends one line to BANDWIDTH_LOG.
|
|
# Trims entries older than BANDWIDTH_LOG_RETENTION days.
|
|
# Flags transfers exceeding BANDWIDTH_WARN_GB for report highlighting.
|
|
# You never call this manually.
|
|
#
|
|
# --report (or no arguments)
|
|
# Reads the accumulated log and generates a weekly summary.
|
|
# Per-profile breakdown: run count, total transferred, average duration, failures.
|
|
# Last 7 days activity timeline.
|
|
# Flags any days or transfers that exceeded BANDWIDTH_WARN_GB.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Log Format — Version-Proof Design ──────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# One line per transfer:
|
|
# YYYY-MM-DD|HH:MM|profile|duration_seconds|status|bytes_transferred
|
|
#
|
|
# Example entries:
|
|
# 2026-04-14|01:23|critical-data|143|success|2847362048
|
|
# 2026-04-14|01:31|arrs_stack|287|success|891234567
|
|
# 2026-04-14|02:15|movies|1847|failed|0
|
|
#
|
|
# Why this format:
|
|
# Earlier designs parsed rsync's human-readable output for bytes transferred.
|
|
# rsync changes its output format between versions — those parsers break silently.
|
|
# Status (success/failed) and duration are always available regardless of rsync version.
|
|
# Bytes captured from rsync --stats output via awk — version-stable field names.
|
|
# This format survives any rsync update with no changes.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
BANDWIDTH_LOG="/boot/config/bandwidth_history.db" # survives reboots
|
|
BANDWIDTH_LOG_RETENTION=90 # days — file stays bounded, never grows unbounded
|
|
BANDWIDTH_WARN_GB=50 # flag transfers or daily totals exceeding this
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
bandwidth_monitor.sh # generate report (default mode)
|
|
bandwidth_monitor.sh --report # generate report (explicit)
|
|
bandwidth_monitor.sh --status # show config and log statistics
|
|
bandwidth_monitor.sh --log # verbose output
|
|
|
|
# This is called automatically by rsync.sh — never call manually:
|
|
bandwidth_monitor.sh --log-transfer "profile" 287 "success" 891234567
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 📋 weekly_health_digest.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Full ecosystem health aggregation. Reads state files from across the entire system —
|
|
watchdog strikes, failover state, skip list, bandwidth history, transcode stats, cert
|
|
status — and compiles them into a single digest report. Reads only, writes nothing.
|
|
|
|
```bash
|
|
# Scheduled: 0 8 * * * (8am daily — DIGEST_PROFILE controls when it notifies)
|
|
```
|
|
|
|
---
|
|
|
|
### ── The Key Design — Schedule Daily, Profile Controls Behaviour ─────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# The cron schedule never changes — always daily at 8am.
|
|
# DIGEST_PROFILE controls what actually happens on each run.
|
|
# Switch profiles by changing one line in master.conf — no cron edit needed.
|
|
#
|
|
DIGEST_PROFILE="weekly" # always | smart | weekly
|
|
```
|
|
|
|
---
|
|
|
|
### ── Three Profiles ───────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# always — sends every run regardless of findings
|
|
# Use: daily digest of everything, even when healthy
|
|
# Result: daily notification with full system snapshot
|
|
#
|
|
# smart — sends only when something worth reporting is found
|
|
# Use: quiet operation, alerts on issues
|
|
# Result: silent on clean days, notification when something needs attention
|
|
# What triggers "worth reporting" is configurable via DIGEST_SMART_ON_* toggles
|
|
#
|
|
# weekly — sends once per week on DIGEST_DAY, silent all other days
|
|
# Use: weekly summary without daily noise
|
|
# Result: one notification on Sunday, silent Monday-Saturday
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
DIGEST_DAY="Sunday" # for weekly profile — must match `date +%A` output
|
|
|
|
# Smart profile triggers — each independently toggleable
|
|
DIGEST_SMART_ON_WATCHDOG=true # send if any active watchdog strikes
|
|
DIGEST_SMART_ON_FAILOVER=true # send if failover state is not NORMAL
|
|
DIGEST_SMART_ON_CERT_WARN=true # send if any cert within CERT_WARN_DAYS
|
|
DIGEST_SMART_ON_BANDWIDTH=true # send if any transfer exceeded BANDWIDTH_WARN_GB
|
|
```
|
|
|
|
---
|
|
|
|
### ── Data Sources — Reads Only ────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Every source is a read — this script writes nothing, changes nothing.
|
|
#
|
|
FAILOVER_STATE_FILE # current failover state (NORMAL/FAILOVER/etc.)
|
|
SYS_WATCHDOG_FAILED_FILE # container skip list — anything here needs attention
|
|
WATCHDOG_STATE_FILE # active container watchdog strikes
|
|
SYS_WATCHDOG_STATE_FILE # active system watchdog strikes
|
|
BANDWIDTH_LOG # yesterday's transfer history
|
|
TRANSCODE_DAILY_LOG # weekly transcode statistics
|
|
CERT_MONITOR_DOMAINS # live openssl cert check per domain
|
|
RAMDISK_PATH / TRANSCODE_LINK # current transcode location and usage
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
weekly_health_digest.sh # normal run — profile determines if notification sent
|
|
weekly_health_digest.sh --dry-run # show digest output, no notification regardless of profile
|
|
weekly_health_digest.sh --log # verbose per-section output
|
|
weekly_health_digest.sh --status # show profile, day, and trigger configuration
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🎬 emby_session_report.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Emby usage report via the Emby API. Queries activity logs and session history to
|
|
produce a weekly summary of streaming activity. No persistent state — queries fresh
|
|
on every run.
|
|
|
|
```bash
|
|
# Scheduled: 0 11 * * 0 (Sunday 11am weekly)
|
|
```
|
|
|
|
---
|
|
|
|
### ── What It Reports ─────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
Server info: name, version, uptime
|
|
Active sessions: current streams, direct play vs transcode breakdown
|
|
Library stats: movie count, episode count, song count
|
|
Activity history: play events from the last EMBY_REPORT_DAYS days
|
|
Top content: most played items in the period (top EMBY_REPORT_TOP_N)
|
|
Most active users: who watched the most in the period
|
|
Transcode ratio: how often transcoding was needed vs direct play
|
|
notifies if > 80% of streams are transcoding (config issue)
|
|
Ramdisk status: current transcode location and usage
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# detect_hosts() aliases these from HOST*_ prefixed vars.
|
|
# Each server reports on its own Emby instance automatically.
|
|
#
|
|
HOST1_EMBY_URL="http://192.168.50.2:8096"
|
|
HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829"
|
|
#
|
|
# To get an API key:
|
|
# Emby UI → Settings → API Keys → New API Key
|
|
# Give it a name (e.g. "unraid_scripts") — copy the key
|
|
|
|
# master.conf
|
|
EMBY_REPORT_DAYS=7 # report period in days
|
|
EMBY_REPORT_TOP_N=10 # top N content items to show
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
emby_session_report.sh # generate report
|
|
emby_session_report.sh --dry-run # test API connectivity only, no notification
|
|
emby_session_report.sh --log # verbose per-section output
|
|
emby_session_report.sh --status # show config and Emby connection status
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## ⚙️ system_tuning_monitor.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
inotify and PHP-FPM utilisation tracking. Snapshots current usage every 6 hours and
|
|
writes to a bounded log. The weekly coffee report reads this log for peak/average/
|
|
warning counts over the week. Silent when healthy — warns when thresholds are hit.
|
|
|
|
```bash
|
|
# Scheduled: 0 */6 * * * (every 6 hours)
|
|
```
|
|
|
|
---
|
|
|
|
### ── What It Tracks ───────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# inotify instances:
|
|
# Current in use vs kernel limit (INOTIFY_MAX_INSTANCES)
|
|
# Utilisation % — warns above INOTIFY_WARN_PCT (default 80%)
|
|
# Top 5 consumers by instance count
|
|
#
|
|
# What inotify exhaustion looks like at the application level:
|
|
# Downloads complete but arrs don't detect them
|
|
# Live TV stutter as media files aren't seen arriving
|
|
# Library updates stop happening
|
|
# No clear error anywhere — things just stop working
|
|
#
|
|
# php-fpm workers:
|
|
# Active workers vs PHP_MAX_CHILDREN limit
|
|
# Utilisation % — warns above PHP_FPM_WARN_PCT (default 80%)
|
|
#
|
|
# What PHP-FPM exhaustion looks like:
|
|
# unRAID WebGUI becomes slow or times out
|
|
# Dashboard requests queue behind each other
|
|
# Settings saves hang
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Log Format ──────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# One line per snapshot — trimmed to TUNING_LOG_RETENTION days on each write:
|
|
# DATE|TIME|INOTIFY_USED|INOTIFY_LIMIT|INOTIFY_PCT|INOTIFY_WARN|
|
|
# PHPFPM_ACTIVE|PHPFPM_MAX|PHPFPM_PCT|PHPFPM_WARN
|
|
#
|
|
# INOTIFY_WARN and PHPFPM_WARN are 1/0 flags — 1 means threshold was exceeded
|
|
# during this snapshot. The weekly coffee report counts warnings over the week.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
INOTIFY_WARN_PCT=80 # warn if inotify instances above 80% of limit
|
|
PHP_FPM_WARN_PCT=80 # warn if php-fpm workers above 80% of max
|
|
TUNING_MONITOR_LOG="$DATA_DIR/tuning_monitor.db"
|
|
TUNING_LOG_RETENTION=30 # days — file stays bounded
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🖥️ continuous_scripts_status.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Live dashboard for all continuously running background scripts. Run any time to get
|
|
an immediate picture of the ecosystem's operational state. No schedule — on demand only.
|
|
|
|
---
|
|
|
|
### ── What It Shows ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# For each continuous script (system_watchdog, docker_watchdog, failover):
|
|
# Running state, PID, uptime, approximate cycle count
|
|
#
|
|
# system_watchdog:
|
|
# Rootfs usage, free RAM, ZFS ARC size, load average, zombie count, CPU temp
|
|
# Active strikes (any check with non-zero strike count)
|
|
# Recent watchdog-triggered reboots
|
|
#
|
|
# docker_watchdog:
|
|
# Running vs stopped vs unhealthy container counts
|
|
# Required containers — which are running, which are stopped
|
|
# Memory-monitored containers — current usage vs hard limits
|
|
# Active container strikes
|
|
# Recent restart history (per container)
|
|
# Skip list — anything here needs human attention
|
|
#
|
|
# failover:
|
|
# Current state (NORMAL/FAILOVER/NO_INTERNET/DARK)
|
|
# Which tier containers are active
|
|
# Outage duration if in FAILOVER
|
|
# Remote server Tailscale visibility (ping result)
|
|
# DDNS containers running on this server
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Run any time — no cron, no schedule. This is an on-demand status tool.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
continuous_scripts_status.sh # live dashboard
|
|
continuous_scripts_status.sh --log # verbose output with full state file dumps
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ RECOMMENDED SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Daily
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
0 8 * * * weekly_health_digest.sh # DIGEST_PROFILE controls notify frequency
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Every 6 hours — background snapshot
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
0 */6 * * * system_tuning_monitor.sh # inotify + php-fpm utilisation tracking
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Sunday morning block — runs after nightly maintenance completes (~3am)
|
|
# By 6am the weekly restarts, log clears, and media maintenance have finished.
|
|
# Monitors see a freshly maintained system.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
0 6 * * 0 zfs_memory_snapshot.sh # ZFS + memory — first, before everything
|
|
0 7 * * 0 smart_health.sh # drive SMART health
|
|
0 9 * * 0 cert_monitor.sh # SSL cert expiry
|
|
0 10 * * 0 backup_verify.sh # rsync mirror integrity
|
|
0 11 * * 0 emby_session_report.sh # Emby streaming usage
|
|
0 11 * * 0 bandwidth_monitor.sh --report # rsync transfer summary
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Automatic — no scheduling needed
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# bandwidth_monitor.sh --log-transfer is called by rsync.sh after each sync.
|
|
# continuous_scripts_status.sh is run on demand — no cron entry.
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ ADDING A NEW MONITOR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
The monitor pattern is consistent across all scripts in this folder:
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# All monitor scripts share these properties:
|
|
#
|
|
# SILENT_MODE=false — output is the point (monitors are meant to produce output)
|
|
# No action taken — observe only, intervention is handled elsewhere
|
|
# acquire_lock — prevent concurrent runs producing duplicate output
|
|
# detect_hosts() — correct configuration per server via MY_ID
|
|
# validate_unraid_cmd — notify script validated before use
|
|
# --dry-run support — run checks without sending notifications
|
|
# --status support — show configuration and exit
|
|
# --log support — verbose per-item output
|
|
# Silent on healthy — no notification when everything passes
|
|
#
|
|
# To add a new monitor:
|
|
# 1. Source load_config.sh — gives you all common functions
|
|
# 2. Set SILENT_MODE=false — this is a monitor, output is expected
|
|
# 3. Call detect_hosts() — correct per-host configuration
|
|
# 4. Call acquire_lock — prevent duplicate runs
|
|
# 5. Do your checks — read, connect, compare
|
|
# 6. notify() only on problems — silent on healthy results
|
|
# 7. Support --dry-run (no notify) and --status (show config, exit)
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
``` |