# ━━━━━ MONITORS ━━━━━

**Observe, measure, and report — never act.** Scripts in this folder watch the system
and notify when something needs attention. They do not restart containers, do not
reboot servers, do not delete files. The other folders handle intervention. This folder
handles awareness.

> **All monitor scripts are safe to run at any time.** They are read-only except for
> `bandwidth_monitor.sh` (one bounded append per rsync run) and
> `system_tuning_monitor.sh` (one bounded append every 6 hours). Every other script
> in this folder writes nothing. None of them can break anything.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━

**Certs Expiring Silently Despite Renewal**  
Certbot runs on schedule. NPM shows the cert as renewed. But the web server was not
reloaded after renewal — the old expired cert is still being served to external clients.
NPM's dashboard shows green. Users see certificate errors.  
Fix: `cert_monitor.sh` connects directly via `openssl` — not to NPM's API, but to the
actual TLS handshake the outside world sees. If users would see a certificate error,
this script catches it.

**No Early Warning on Drive Degradation**  
A drive accumulates reallocated sectors silently over weeks. Nothing fails. No error
in the Docker logs. No alert from unRAID. The drive is heading toward failure and the
only way to know is to actively query SMART data.  
Fix: `smart_health.sh` queries SMART attributes for every drive every week. Drive failure
has a warning curve — weekly checks give you time to act on it.

**rsync Succeeds But the Copy Is Wrong**  
rsync reports success. Exit code 0. But a storage hardware issue on either server
silently corrupted a file during the write. rsync is satisfied — sizes match, timestamps
match — but the content is wrong.  
Fix: `backup_verify.sh` independently computes MD5 checksums on both sides and compares.
It does not trust rsync's exit code. It verifies the actual content matches.

**inotify Exhaustion Producing Unexplained Symptoms**  
inotify watches are silently exhausted. Downloads finish but arr containers don't see
them. Live TV stutters. Library updates stop. None of these produce clear error messages.  
Fix: `system_tuning_monitor.sh` snapshots inotify and PHP-FPM utilisation every 6 hours.
The weekly digest shows peak, average, and warning count so you see the trend before it
becomes a problem.

**No Single View of the Ecosystem's Health**  
System state is spread across a dozen state files — watchdog strikes, fallback state,
bandwidth history, cert status, transcode stats. No single place to see how everything
is doing.  
Fix: `weekly_health_digest.sh` reads all state files and compiles them into one digest.
Three profiles (always / smart / weekly) control how often it notifies.

---

## ━━━ WHAT THIS FOLDER DOES ━━━

Four distinct observation roles:

**Live Connectivity Checks** — Active probes against real endpoints  
`cert_monitor.sh` and `backup_verify.sh` go out and test things directly. They connect,
compute, compare. The result is a measurement of the actual state of the world — not a
reading of a cached status.

**State File Aggregation** — Synthesising the ecosystem's own data  
`weekly_health_digest.sh` reads the state files the watchdogs and orchestrators have
already written. It doesn't gather new data — it organises existing data into a
human-readable summary.

**Time-Series Snapshots** — Tracking trends over time  
`bandwidth_monitor.sh` and `system_tuning_monitor.sh` write one bounded log entry per
interval. Over time this builds a picture of trends — whether inotify pressure is growing,
whether transfer sizes are increasing, whether PHP-FPM is consistently near ceiling.

**Hardware Health** — Checking what the hardware itself reports  
`smart_health.sh` and `zfs_memory_snapshot.sh` query the hardware and kernel directly —
drive SMART attributes, ZFS pool state, ARC statistics, kernel memory pressure.

---

## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

```
Monitors/            ← observes and reports          (this folder)
Docker_Essentials/   ← acts on containers            (docker_watchdog starts/stops)
System_Essentials/   ← acts on the server            (stability_watchdog, inotify_tuning)
Fallback/            ← acts on the full stack        (fallback, handback)
Rsync/               ← calls bandwidth_monitor.sh    (auto-logs each sync)
```

`weekly_health_digest.sh` reads state files written by scripts in Docker_Essentials,
System_Essentials, Fallback, and Rsync. It is the only script in this folder with
runtime dependencies on other folders' output — everything else is fully independent.

---

## ━━━ FLASH DRIVE WRITE POLICY ━━━

unRAID boots from a USB flash drive with limited write cycles. Every script in this
folder is designed with this constraint in mind.

| Script | Writes to flash | Detail |
|--------|----------------|--------|
| `cert_monitor.sh` | Never | openssl connections only |
| `smart_health.sh` | Never | smartctl reads only |
| `zfs_memory_snapshot.sh` | Never | writes to `/var/log/` (tmpfs) |
| `backup_verify.sh` | Never | SSH + MD5 comparison only |
| `emby_session_report.sh` | Never | API queries only |
| `weekly_health_digest.sh` | Never | reads existing state files only |
| `bandwidth_monitor.sh` | One append + one trim per rsync run | Bounded — never exceeds `BANDWIDTH_LOG_RETENTION` lines |
| `system_tuning_monitor.sh` | One append + one trim every 6 hours | Bounded — trimmed to `TUNING_LOG_RETENTION` days |
| `mesh_monitor.sh` | Never | read-only conf parse — no API calls, no SSH |

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━

| Script | What It Watches | When |
|--------|----------------|------|
| `zfs_memory_snapshot.sh` | ZFS pool health, ARC stats, memory pressure | Weekly — Sunday 6am |
| `smart_health.sh` | Drive SMART health attributes | Weekly — Sunday 7am |
| `cert_monitor.sh` | SSL cert expiry via live TLS connection | Weekly — Sunday 9am |
| `backup_verify.sh` | rsync mirror integrity via MD5 checksums | Weekly — Sunday 10am |
| `emby_session_report.sh` | Emby streaming usage and library stats | Weekly — Sunday 11am |
| `bandwidth_monitor.sh` | rsync transfer history and trends | Auto (called by rsync.sh) + weekly report Sunday 11am |
| `weekly_health_digest.sh` | Full ecosystem health aggregation | Daily 8am (DIGEST_PROFILE controls notify) |
| `system_tuning_monitor.sh` | inotify + PHP-FPM utilisation trends | Every 6 hours |
| `mesh_monitor.sh` | Mesh membership — who covers whom, partnership state | On demand |

---

## ━━━ HOW THE SCRIPTS RELATE ━━━

```
rsync.sh ──────────────────────────────────► bandwidth_monitor.sh --log-transfer
  (after each sync completes)                  (one log entry per sync, bounded)

Sunday morning block (after nightly maintenance):
  6am  zfs_memory_snapshot.sh ─────────────► ZFS_REPORT_LOG (tmpfs)
  7am  smart_health.sh ───────────────────── (notification only)
  9am  cert_monitor.sh ───────────────────── (notification only)
 10am  backup_verify.sh ────────────────────  (notification only)
 11am  emby_session_report.sh ──────────────  (notification only)
 11am  bandwidth_monitor.sh --report ───────► reads BANDWIDTH_LOG

Every 6h:
  system_tuning_monitor.sh ──────────────────► TUNING_MONITOR_LOG

On demand:
  mesh_monitor.sh ──────────────────────────── reads host*.conf only (no API, no SSH)
                                                MEMBERS / COVERAGE / PARTNERSHIP sections

Daily 8am:
  weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE
                          ── reads ──────────► WATCHDOG_STATE_FILE
                          ── reads ──────────► SYS_WATCHDOG_STATE_FILE
                          ── reads ──────────► DOCKER_WATCHDOG_FAILED_FILE
                          ── reads ──────────► BANDWIDTH_LOG
                          ── reads ──────────► TRANSCODE_DAILY_LOG
                          ── reads ──────────► TUNING_MONITOR_LOG
                          ── live openssl ────► CERT_MONITOR_DOMAINS
```
