Correct and extend folder docs for Docker Essentials, Watchdogs, Media and Arrs Stack
The docs had drifted from the scripts — a script that no longer exists, three wrong variable names, a reversed run order, and seven scheduled scripts that were never documented at all.
This commit is contained in:
@@ -234,3 +234,53 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
|
||||
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
||||
|
||||
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━
|
||||
|
||||
These run unattended every 15 minutes and hold the only irreversible remedy in the
|
||||
ecosystem, so the guards matter as much as the checks. Each script's header documents its
|
||||
full set; these are the folder-wide ones.
|
||||
|
||||
**Nothing acts on a single reading.** Every check that can trigger an action is strike-gated.
|
||||
One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves
|
||||
when the condition resolves, so a transient blip never accumulates toward an alarm across
|
||||
unrelated weeks.
|
||||
|
||||
**Escalation is asymmetric on purpose.** Protective actions happen fast; undoing them happens
|
||||
slowly. `resource_watchdog.sh` throttles before it pauses and pauses before it stops, then
|
||||
requires `RW_RECOVER_CYCLES` clear cycles and de-escalates one level at a time. Recovering on
|
||||
a single good reading would flap under sustained load.
|
||||
|
||||
**Ownership boundaries are enforced, not conventional.** Container health belongs to
|
||||
`docker_watchdog.sh`; `stability_watchdog.sh` deliberately does not check it, and its Docker
|
||||
daemon check writes a flag for `docker_watchdog.sh` rather than acting. Two watchdogs
|
||||
remediating the same subsystem would race.
|
||||
|
||||
**Watchdogs coordinate through state files.** `resource_watchdog.sh` publishes
|
||||
`mem_shutdown_active` so `docker_watchdog.sh` defers restarts — otherwise one would stop
|
||||
containers to free RAM while the other restarted them to restore health. The reader applies a
|
||||
staleness guard so a crashed writer cannot suppress restarts forever.
|
||||
|
||||
**Data safety outranks uptime.** `stability_watchdog.sh` aborts a reboot while a ZFS pool is
|
||||
unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system
|
||||
cannot recover from and which worsen every cycle — bypasses that.
|
||||
|
||||
**Reboot loops end in shutdown, not more reboots.** No more than
|
||||
`SYS_WATCHDOG_REBOOT_LIMIT` reboots within `SYS_WATCHDOG_REBOOT_WINDOW_HRS`. On hitting the
|
||||
limit the host powers off: a fault that survives repeated reboots will not be fixed by more
|
||||
of them, and a cleanly-down box is better than one cycling endlessly.
|
||||
|
||||
**An aborted reboot restores what it stopped.** An EXIT trap is armed the moment containers
|
||||
start being stopped for a reboot and disarmed only once the reboot commits. If the script
|
||||
dies in between, everything it stopped comes back — the failure mode is a running system,
|
||||
never a host left stripped with no reboot.
|
||||
|
||||
**An unconfigured job list fails loudly.** An empty `WATCHDOG_ORCHESTRATOR_SCRIPTS` or
|
||||
`SYSTEM_WATCHDOG_SCRIPTS` would otherwise report "0/0 passed" and exit 0 every cycle —
|
||||
indistinguishable from healthy while nothing is monitored at all.
|
||||
|
||||
**Verify the path, not the process.** `network_watchdog.sh` checks NPM by fetching an external
|
||||
URL rather than asking whether the container is running. A running container behind broken DNS
|
||||
or a broken upstream still serves nothing.
|
||||
|
||||
Reference in New Issue
Block a user