Watchdogs/ docs: new README + Manual, update affected folders

New docs:
  Watchdogs/README-Watchdogs.md  — design, relationships, script table, state file map
  Watchdogs/Manual-Watchdogs.md  — full config reference for all 4 watchdogs

Docker_Essentials/:
  README — remove docker_watchdog, update folder description and diagrams
  Manual  — strip watchdog config sections, add pointer to Watchdogs/Manual

unRAID_Essentials/:
  README — remove system/resource watchdog, update diagrams and script table
  Manual  — strip system/resource watchdog sections, update TOC + config reference

README.md:
  Add Watchdogs/ to folder structure
  Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start
  Fix daily cycle and monitoring diagrams
This commit is contained in:
Gmer4Lfe
2026-05-22 17:26:21 -04:00
parent ec79a926e8
commit 8a3e22c9b8
7 changed files with 912 additions and 707 deletions
+22 -49
View File
@@ -154,24 +154,16 @@ its history after you've fixed the problem. No manual file editing required.
```
Docker_Essentials/ ← acts on containers (this folder)
unRAID_Essentials/ ← acts on the server itself
Watchdogs/ ← reactive monitoring + last-resort stability
Monitors/ ← observes, measures, reports
Rsync/ ← moves data between servers
```
Five distinct responsibilities, each handled by dedicated scripts:
> `docker_watchdog.sh` has moved to `Watchdogs/`. Container healing, memory limits,
> HTTP health checks, and skip list management are documented in
> `Watchdogs/README-Watchdogs.md` and `Watchdogs/Manual-Watchdogs.md`.
---
### 🔁 Reactive Healing — `docker_watchdog.sh`
Continuous two-tier monitoring that catches problems as they happen and acts on them
immediately. Runs as a background process started at array start. Every 15 minutes it
checks the full stack and fixes what it can — silently when everything is fine, visibly
when something needs attention.
Two tiers because different containers need different monitoring strategies:
- **Tier 1** — explicit per-container configuration with specific thresholds
- **Tier 2** — global scan of everything that's running with catch-all health checks
Four distinct responsibilities in this folder, each handled by dedicated scripts:
---
@@ -212,34 +204,22 @@ or in-progress downloads.
---
## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## ━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Two watchdogs run simultaneously. They are designed to work together, not compete:
`docker_watchdog.sh` has moved to `Watchdogs/` and is now one of four coordinated
single-pass scripts called every minute by `Orchestrators/watchdog_orchestrator.sh`.
```
system_watchdog.sh watches the server: RAM, CPU, disk, kernel, daemon health
docker_watchdog.sh ← watches the containers: memory, CPU, HTTP response, crashes
Watchdogs/resource_watchdog.sh ← reduces pressure before healing attempts
Watchdogs/docker_watchdog.sh heals containers (reads resource_watchdog state)
Watchdogs/storage_watchdog.sh ← pool growth + runaway log detection
Watchdogs/system_watchdog.sh ← last resort — reboots when healing has failed
```
**The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops
non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh`
would see stopped containers on its next cycle and restart them — directly undoing the
RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover.
The system would eventually hit the reboot threshold anyway, having accomplished nothing.
**The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When
`system_watchdog.sh` triggers a RAM emergency shutdown it writes
`mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every
cycle and defers all container restart logic until it clears. Health URL checks for
excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still
run. Everything else stands down.
**The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state
file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh`
checks how long ago the state file was modified. If it's more than 2 hours old while
`mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running.
`docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced
indefinitely by a stale flag from a process that's no longer running.
Scripts in this folder (daily restart, updates, network connect) are unaffected —
they run on their own schedules via the maintenance orchestrators and are not part
of the every-minute watchdog cycle. See `Watchdogs/README-Watchdogs.md` for the
full coordination model between all four watchdogs.
---
@@ -247,7 +227,6 @@ indefinitely by a stale flag from a process that's no longer running.
| Script | Role | When It Runs |
|--------|------|-------------|
| `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop — array start |
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window |
@@ -265,18 +244,12 @@ indefinitely by a stale flag from a process that's no longer running.
```
Array starts
── docker_network_connect.sh ────── run once at start
ensure networks + connections exist
silent if correct, notify if creating
└── docker_watchdog.sh ──────────── continuous background loop (every 15min)
Tier 1: memory, CPU, HTTP, required containers
Tier 2: global unhealthy / OOM / crash / dead scan
reads system_watchdog state (RAM emergency deferral)
│ (on skip list event → operator uses)
└── Tools/watchdog_skip_list_manager.sh
inspect state, clear after fixing root cause
── docker_network_connect.sh ────── run once at start
ensure networks + connections exist
silent if correct, notify if creating
Every minute (Orchestrators/watchdog_orchestrator.sh):
└── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md
Daily maintenance window (1am):
daily_sync_maintenance.sh