Varaverk/Monitors/README-Monitors.md

━━━━━ MONITORS ━━━━━

Observe, measure, and report — never act. Scripts in this folder watch the system and notify when something needs attention. They do not restart containers, do not reboot servers, do not delete files. The other folders handle intervention. This folder handles awareness.

All monitor scripts are safe to run at any time. They are read-only except for bandwidth_monitor.sh (one bounded append per rsync run) and system_tuning_monitor.sh (one bounded append every 6 hours). Every other script in this folder writes nothing. None of them can break anything.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━

Certs Expiring Silently Despite Renewal
Certbot runs on schedule. NPM shows the cert as renewed. But the web server was not reloaded after renewal — the old expired cert is still being served to external clients. NPM's dashboard shows green. Users see certificate errors.
Fix: cert_monitor.sh connects directly via openssl — not to NPM's API, but to the actual TLS handshake the outside world sees. If users would see a certificate error, this script catches it.

No Early Warning on Drive Degradation
A drive accumulates reallocated sectors silently over weeks. Nothing fails. No error in the Docker logs. No alert from unRAID. The drive is heading toward failure and the only way to know is to actively query SMART data.
Fix: smart_health.sh queries SMART attributes for every drive every week. Drive failure has a warning curve — weekly checks give you time to act on it.

rsync Succeeds But the Copy Is Wrong
rsync reports success. Exit code 0. But a storage hardware issue on either server silently corrupted a file during the write. rsync is satisfied — sizes match, timestamps match — but the content is wrong.
Fix: backup_verify.sh independently computes MD5 checksums on both sides and compares. It does not trust rsync's exit code. It verifies the actual content matches.

inotify Exhaustion Producing Unexplained Symptoms
inotify watches are silently exhausted. Downloads finish but arr containers don't see them. Live TV stutters. Library updates stop. None of these produce clear error messages.
Fix: system_tuning_monitor.sh snapshots inotify and PHP-FPM utilisation every 6 hours. The weekly digest shows peak, average, and warning count so you see the trend before it becomes a problem.

No Single View of the Ecosystem's Health
System state is spread across a dozen state files — watchdog strikes, fallback state, bandwidth history, cert status, transcode stats. No single place to see how everything is doing.
Fix: weekly_health_digest.sh reads all state files and compiles them into one digest. Three profiles (always / smart / weekly) control how often it notifies.

---

━━━ WHAT THIS FOLDER DOES ━━━

Four distinct observation roles:

Live Connectivity Checks — Active probes against real endpoints
cert_monitor.sh and backup_verify.sh go out and test things directly. They connect, compute, compare. The result is a measurement of the actual state of the world — not a reading of a cached status.

State File Aggregation — Synthesising the ecosystem's own data
weekly_health_digest.sh reads the state files the watchdogs and orchestrators have already written. It doesn't gather new data — it organises existing data into a human-readable summary.

Time-Series Snapshots — Tracking trends over time
bandwidth_monitor.sh and system_tuning_monitor.sh write one bounded log entry per interval. Over time this builds a picture of trends — whether inotify pressure is growing, whether transfer sizes are increasing, whether PHP-FPM is consistently near ceiling.

Hardware Health — Checking what the hardware itself reports
smart_health.sh and zfs_memory_snapshot.sh query the hardware and kernel directly — drive SMART attributes, ZFS pool state, ARC statistics, kernel memory pressure.

---

━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

Monitors/            ← observes and reports          (this folder)
Docker_Essentials/   ← acts on containers            (docker_watchdog starts/stops)
System_Essentials/   ← acts on the server            (stability_watchdog, inotify_tuning)
Fallback/            ← acts on the full stack        (fallback, handback)
Rsync/               ← calls bandwidth_monitor.sh    (auto-logs each sync)

weekly_health_digest.sh reads state files written by scripts in Docker_Essentials, System_Essentials, Fallback, and Rsync. It is the only script in this folder with runtime dependencies on other folders' output — everything else is fully independent.

---

━━━ FLASH DRIVE WRITE POLICY ━━━

unRAID boots from a USB flash drive with limited write cycles. Every script in this folder is designed with this constraint in mind.

Script	Writes to flash	Detail
cert_monitor.sh	Never	openssl connections only
smart_health.sh	Never	smartctl reads only
zfs_memory_snapshot.sh	Never	writes to /var/log/ (tmpfs)
backup_verify.sh	Never	SSH + MD5 comparison only
emby_session_report.sh	Never	API queries only
weekly_health_digest.sh	Never	reads existing state files only
bandwidth_monitor.sh	One append + one trim per rsync run	Bounded — never exceeds BANDWIDTH_LOG_RETENTION lines
system_tuning_monitor.sh	One append + one trim every 6 hours	Bounded — trimmed to TUNING_LOG_RETENTION days
mesh_monitor.sh	Never	read-only conf parse — no API calls, no SSH

---

━━━ SCRIPTS IN THIS FOLDER ━━━

Script	What It Watches	When
zfs_memory_snapshot.sh	ZFS pool health, ARC stats, memory pressure	Weekly — Sunday 6am
smart_health.sh	Drive SMART health attributes	Weekly — Sunday 7am
cert_monitor.sh	SSL cert expiry via live TLS connection	Weekly — Sunday 9am
backup_verify.sh	rsync mirror integrity via MD5 checksums	Weekly — Sunday 10am
emby_session_report.sh	Emby streaming usage and library stats	Weekly — Sunday 11am
bandwidth_monitor.sh	rsync transfer history and trends	Auto (called by rsync.sh) + weekly report Sunday 11am
weekly_health_digest.sh	Full ecosystem health aggregation	Daily 8am (DIGEST_PROFILE controls notify)
system_tuning_monitor.sh	inotify + PHP-FPM utilisation trends	Every 6 hours
mesh_monitor.sh	Mesh membership — who covers whom, partnership state	On demand

---

━━━ HOW THE SCRIPTS RELATE ━━━

rsync.sh ──────────────────────────────────► bandwidth_monitor.sh --log-transfer
  (after each sync completes)                  (one log entry per sync, bounded)

Sunday morning block (after nightly maintenance):
  6am  zfs_memory_snapshot.sh ─────────────► ZFS_REPORT_LOG (tmpfs)
  7am  smart_health.sh ───────────────────── (notification only)
  9am  cert_monitor.sh ───────────────────── (notification only)
 10am  backup_verify.sh ────────────────────  (notification only)
 11am  emby_session_report.sh ──────────────  (notification only)
 11am  bandwidth_monitor.sh --report ───────► reads BANDWIDTH_LOG

Every 6h:
  system_tuning_monitor.sh ──────────────────► TUNING_MONITOR_LOG

On demand:
  mesh_monitor.sh ──────────────────────────── reads host*.conf only (no API, no SSH)
                                                MEMBERS / COVERAGE / PARTNERSHIP sections

Daily 8am:
  weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE
                          ── reads ──────────► WATCHDOG_STATE_FILE
                          ── reads ──────────► SYS_WATCHDOG_STATE_FILE
                          ── reads ──────────► DOCKER_WATCHDOG_FAILED_FILE
                          ── reads ──────────► BANDWIDTH_LOG
                          ── reads ──────────► TRANSCODE_DAILY_LOG
                          ── reads ──────────► TUNING_MONITOR_LOG
                          ── live openssl ────► CERT_MONITOR_DOMAINS