Correct and extend folder docs for Docker Essentials, Watchdogs, Media and Arrs Stack

The docs had drifted from the scripts — a script that no longer exists, three wrong variable
names, a reversed run order, and seven scheduled scripts that were never documented at all.
This commit is contained in:
Gmer4Lfe
2026-08-01 22:59:07 -04:00
parent c377ddfcca
commit 8a2707ee37
6 changed files with 331 additions and 30 deletions
+50
View File
@@ -234,3 +234,53 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
---
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━
These run unattended every 15 minutes and hold the only irreversible remedy in the
ecosystem, so the guards matter as much as the checks. Each script's header documents its
full set; these are the folder-wide ones.
**Nothing acts on a single reading.** Every check that can trigger an action is strike-gated.
One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves
when the condition resolves, so a transient blip never accumulates toward an alarm across
unrelated weeks.
**Escalation is asymmetric on purpose.** Protective actions happen fast; undoing them happens
slowly. `resource_watchdog.sh` throttles before it pauses and pauses before it stops, then
requires `RW_RECOVER_CYCLES` clear cycles and de-escalates one level at a time. Recovering on
a single good reading would flap under sustained load.
**Ownership boundaries are enforced, not conventional.** Container health belongs to
`docker_watchdog.sh`; `stability_watchdog.sh` deliberately does not check it, and its Docker
daemon check writes a flag for `docker_watchdog.sh` rather than acting. Two watchdogs
remediating the same subsystem would race.
**Watchdogs coordinate through state files.** `resource_watchdog.sh` publishes
`mem_shutdown_active` so `docker_watchdog.sh` defers restarts — otherwise one would stop
containers to free RAM while the other restarted them to restore health. The reader applies a
staleness guard so a crashed writer cannot suppress restarts forever.
**Data safety outranks uptime.** `stability_watchdog.sh` aborts a reboot while a ZFS pool is
unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system
cannot recover from and which worsen every cycle — bypasses that.
**Reboot loops end in shutdown, not more reboots.** No more than
`SYS_WATCHDOG_REBOOT_LIMIT` reboots within `SYS_WATCHDOG_REBOOT_WINDOW_HRS`. On hitting the
limit the host powers off: a fault that survives repeated reboots will not be fixed by more
of them, and a cleanly-down box is better than one cycling endlessly.
**An aborted reboot restores what it stopped.** An EXIT trap is armed the moment containers
start being stopped for a reboot and disarmed only once the reboot commits. If the script
dies in between, everything it stopped comes back — the failure mode is a running system,
never a host left stripped with no reboot.
**An unconfigured job list fails loudly.** An empty `WATCHDOG_ORCHESTRATOR_SCRIPTS` or
`SYSTEM_WATCHDOG_SCRIPTS` would otherwise report "0/0 passed" and exit 0 every cycle —
indistinguishable from healthy while nothing is monitored at all.
**Verify the path, not the process.** `network_watchdog.sh` checks NPM by fetching an external
URL rather than asking whether the container is running. A running container behind broken DNS
or a broken upstream still serves nothing.