Gmer4Lfe
2fa78a21b6
- Tools/Manual-Tools.md: fallback_state_reset.sh marked not yet built; manual workaround added; TOC entry updated; failover→fallback rename throughout - Watchdogs/Manual-Watchdogs.md: System Watchdog config block updated — remove pre-split vars (MEM_WARN_GB, MEM_SHUTDOWN_GB, MEM_GB, MEM_RECOVER_GB, SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED, SYS_WATCHDOG_REQUIRED_CONTAINERS); add SYS_WATCHDOG_MEM_GB (reboot trigger only); fix state file path names (watchdog_state.db → container_watchdog_state.db, sys_watchdog_state.db → system_watchdog_state.db) - Orchestrators/README-Orchestrators.md: add monthly_maintenance.sh section with two-gate model, configuration reference, and usage examples - Monitors/README-Monitors.md: add mesh_monitor.sh to both tables and diagram - Monitors/Manual-Monitors.md: add mesh_monitor.sh section - README-User_Script_Plug-in.md, Rsync/README-Rsync.md, Rsync/Manual-Rsync.md, README.md: emby-failover → emby-fallback (missed from prior batch)
162 lines
8.5 KiB
Markdown
162 lines
8.5 KiB
Markdown
# ━━━━━ MONITORS ━━━━━
|
|
|
|
**Observe, measure, and report — never act.** Scripts in this folder watch the system
|
|
and notify when something needs attention. They do not restart containers, do not
|
|
reboot servers, do not delete files. The other folders handle intervention. This folder
|
|
handles awareness.
|
|
|
|
> **All monitor scripts are safe to run at any time.** They are read-only except for
|
|
> `bandwidth_monitor.sh` (one bounded append per rsync run) and
|
|
> `system_tuning_monitor.sh` (one bounded append every 6 hours). Every other script
|
|
> in this folder writes nothing. None of them can break anything.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
|
|
|
**Certs Expiring Silently Despite Renewal**
|
|
Certbot runs on schedule. NPM shows the cert as renewed. But the web server was not
|
|
reloaded after renewal — the old expired cert is still being served to external clients.
|
|
NPM's dashboard shows green. Users see certificate errors.
|
|
Fix: `cert_monitor.sh` connects directly via `openssl` — not to NPM's API, but to the
|
|
actual TLS handshake the outside world sees. If users would see a certificate error,
|
|
this script catches it.
|
|
|
|
**No Early Warning on Drive Degradation**
|
|
A drive accumulates reallocated sectors silently over weeks. Nothing fails. No error
|
|
in the Docker logs. No alert from unRAID. The drive is heading toward failure and the
|
|
only way to know is to actively query SMART data.
|
|
Fix: `smart_health.sh` queries SMART attributes for every drive every week. Drive failure
|
|
has a warning curve — weekly checks give you time to act on it.
|
|
|
|
**rsync Succeeds But the Copy Is Wrong**
|
|
rsync reports success. Exit code 0. But a storage hardware issue on either server
|
|
silently corrupted a file during the write. rsync is satisfied — sizes match, timestamps
|
|
match — but the content is wrong.
|
|
Fix: `backup_verify.sh` independently computes MD5 checksums on both sides and compares.
|
|
It does not trust rsync's exit code. It verifies the actual content matches.
|
|
|
|
**inotify Exhaustion Producing Unexplained Symptoms**
|
|
inotify watches are silently exhausted. Downloads finish but arr containers don't see
|
|
them. Live TV stutters. Library updates stop. None of these produce clear error messages.
|
|
Fix: `system_tuning_monitor.sh` snapshots inotify and PHP-FPM utilisation every 6 hours.
|
|
The weekly digest shows peak, average, and warning count so you see the trend before it
|
|
becomes a problem.
|
|
|
|
**No Single View of the Ecosystem's Health**
|
|
System state is spread across a dozen state files — watchdog strikes, fallback state,
|
|
bandwidth history, cert status, transcode stats. No single place to see how everything
|
|
is doing.
|
|
Fix: `weekly_health_digest.sh` reads all state files and compiles them into one digest.
|
|
Three profiles (always / smart / weekly) control how often it notifies.
|
|
|
|
---
|
|
|
|
## ━━━ WHAT THIS FOLDER DOES ━━━
|
|
|
|
Four distinct observation roles:
|
|
|
|
**Live Connectivity Checks** — Active probes against real endpoints
|
|
`cert_monitor.sh` and `backup_verify.sh` go out and test things directly. They connect,
|
|
compute, compare. The result is a measurement of the actual state of the world — not a
|
|
reading of a cached status.
|
|
|
|
**State File Aggregation** — Synthesising the ecosystem's own data
|
|
`weekly_health_digest.sh` reads the state files the watchdogs and orchestrators have
|
|
already written. It doesn't gather new data — it organises existing data into a
|
|
human-readable summary.
|
|
|
|
**Time-Series Snapshots** — Tracking trends over time
|
|
`bandwidth_monitor.sh` and `system_tuning_monitor.sh` write one bounded log entry per
|
|
interval. Over time this builds a picture of trends — whether inotify pressure is growing,
|
|
whether transfer sizes are increasing, whether PHP-FPM is consistently near ceiling.
|
|
|
|
**Hardware Health** — Checking what the hardware itself reports
|
|
`smart_health.sh` and `zfs_memory_snapshot.sh` query the hardware and kernel directly —
|
|
drive SMART attributes, ZFS pool state, ARC statistics, kernel memory pressure.
|
|
|
|
---
|
|
|
|
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
|
|
|
```
|
|
Monitors/ ← observes and reports (this folder)
|
|
Docker_Essentials/ ← acts on containers (docker_watchdog starts/stops)
|
|
unRAID_Essentials/ ← acts on the server (stability_watchdog, inotify_tuning)
|
|
Fallback/ ← acts on the full stack (failover, handback)
|
|
Rsync/ ← calls bandwidth_monitor.sh (auto-logs each sync)
|
|
```
|
|
|
|
`weekly_health_digest.sh` reads state files written by scripts in Docker_Essentials,
|
|
unRAID_Essentials, Fallback, and Rsync. It is the only script in this folder with
|
|
runtime dependencies on other folders' output — everything else is fully independent.
|
|
|
|
---
|
|
|
|
## ━━━ FLASH DRIVE WRITE POLICY ━━━
|
|
|
|
unRAID boots from a USB flash drive with limited write cycles. Every script in this
|
|
folder is designed with this constraint in mind.
|
|
|
|
| Script | Writes to flash | Detail |
|
|
|--------|----------------|--------|
|
|
| `cert_monitor.sh` | Never | openssl connections only |
|
|
| `smart_health.sh` | Never | smartctl reads only |
|
|
| `zfs_memory_snapshot.sh` | Never | writes to `/var/log/` (tmpfs) |
|
|
| `backup_verify.sh` | Never | SSH + MD5 comparison only |
|
|
| `emby_session_report.sh` | Never | API queries only |
|
|
| `weekly_health_digest.sh` | Never | reads existing state files only |
|
|
| `bandwidth_monitor.sh` | One append + one trim per rsync run | Bounded — never exceeds `BANDWIDTH_LOG_RETENTION` lines |
|
|
| `system_tuning_monitor.sh` | One append + one trim every 6 hours | Bounded — trimmed to `TUNING_LOG_RETENTION` days |
|
|
| `mesh_monitor.sh` | Never | read-only conf parse — no API calls, no SSH |
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
|
|
|
| Script | What It Watches | When |
|
|
|--------|----------------|------|
|
|
| `zfs_memory_snapshot.sh` | ZFS pool health, ARC stats, memory pressure | Weekly — Sunday 6am |
|
|
| `smart_health.sh` | Drive SMART health attributes | Weekly — Sunday 7am |
|
|
| `cert_monitor.sh` | SSL cert expiry via live TLS connection | Weekly — Sunday 9am |
|
|
| `backup_verify.sh` | rsync mirror integrity via MD5 checksums | Weekly — Sunday 10am |
|
|
| `emby_session_report.sh` | Emby streaming usage and library stats | Weekly — Sunday 11am |
|
|
| `bandwidth_monitor.sh` | rsync transfer history and trends | Auto (called by rsync.sh) + weekly report Sunday 11am |
|
|
| `weekly_health_digest.sh` | Full ecosystem health aggregation | Daily 8am (DIGEST_PROFILE controls notify) |
|
|
| `system_tuning_monitor.sh` | inotify + PHP-FPM utilisation trends | Every 6 hours |
|
|
| `mesh_monitor.sh` | Mesh membership — who covers whom, partnership state | On demand |
|
|
|
|
---
|
|
|
|
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
|
|
|
```
|
|
rsync.sh ──────────────────────────────────► bandwidth_monitor.sh --log-transfer
|
|
(after each sync completes) (one log entry per sync, bounded)
|
|
|
|
Sunday morning block (after nightly maintenance):
|
|
6am zfs_memory_snapshot.sh ─────────────► ZFS_REPORT_LOG (tmpfs)
|
|
7am smart_health.sh ───────────────────── (notification only)
|
|
9am cert_monitor.sh ───────────────────── (notification only)
|
|
10am backup_verify.sh ──────────────────── (notification only)
|
|
11am emby_session_report.sh ────────────── (notification only)
|
|
11am bandwidth_monitor.sh --report ───────► reads BANDWIDTH_LOG
|
|
|
|
Every 6h:
|
|
system_tuning_monitor.sh ──────────────────► TUNING_MONITOR_LOG
|
|
|
|
On demand:
|
|
mesh_monitor.sh ──────────────────────────── reads host*.conf only (no API, no SSH)
|
|
MEMBERS / COVERAGE / PARTNERSHIP sections
|
|
|
|
Daily 8am:
|
|
weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE
|
|
── reads ──────────► WATCHDOG_STATE_FILE
|
|
── reads ──────────► SYS_WATCHDOG_STATE_FILE
|
|
── reads ──────────► SYS_WATCHDOG_FAILED_FILE
|
|
── reads ──────────► BANDWIDTH_LOG
|
|
── reads ──────────► TRANSCODE_DAILY_LOG
|
|
── reads ──────────► TUNING_MONITOR_LOG
|
|
── live openssl ────► CERT_MONITOR_DOMAINS
|
|
```
|