Watchdogs/ docs: new README + Manual, update affected folders
New docs: Watchdogs/README-Watchdogs.md — design, relationships, script table, state file map Watchdogs/Manual-Watchdogs.md — full config reference for all 4 watchdogs Docker_Essentials/: README — remove docker_watchdog, update folder description and diagrams Manual — strip watchdog config sections, add pointer to Watchdogs/Manual unRAID_Essentials/: README — remove system/resource watchdog, update diagrams and script table Manual — strip system/resource watchdog sections, update TOC + config reference README.md: Add Watchdogs/ to folder structure Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start Fix daily cycle and monitoring diagrams
This commit is contained in:
@@ -154,24 +154,16 @@ its history after you've fixed the problem. No manual file editing required.
|
||||
```
|
||||
Docker_Essentials/ ← acts on containers (this folder)
|
||||
unRAID_Essentials/ ← acts on the server itself
|
||||
Watchdogs/ ← reactive monitoring + last-resort stability
|
||||
Monitors/ ← observes, measures, reports
|
||||
Rsync/ ← moves data between servers
|
||||
```
|
||||
|
||||
Five distinct responsibilities, each handled by dedicated scripts:
|
||||
> `docker_watchdog.sh` has moved to `Watchdogs/`. Container healing, memory limits,
|
||||
> HTTP health checks, and skip list management are documented in
|
||||
> `Watchdogs/README-Watchdogs.md` and `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
---
|
||||
|
||||
### 🔁 Reactive Healing — `docker_watchdog.sh`
|
||||
|
||||
Continuous two-tier monitoring that catches problems as they happen and acts on them
|
||||
immediately. Runs as a background process started at array start. Every 15 minutes it
|
||||
checks the full stack and fixes what it can — silently when everything is fine, visibly
|
||||
when something needs attention.
|
||||
|
||||
Two tiers because different containers need different monitoring strategies:
|
||||
- **Tier 1** — explicit per-container configuration with specific thresholds
|
||||
- **Tier 2** — global scan of everything that's running with catch-all health checks
|
||||
Four distinct responsibilities in this folder, each handled by dedicated scripts:
|
||||
|
||||
---
|
||||
|
||||
@@ -212,34 +204,22 @@ or in-progress downloads.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Two watchdogs run simultaneously. They are designed to work together, not compete:
|
||||
`docker_watchdog.sh` has moved to `Watchdogs/` and is now one of four coordinated
|
||||
single-pass scripts called every minute by `Orchestrators/watchdog_orchestrator.sh`.
|
||||
|
||||
```
|
||||
system_watchdog.sh ← watches the server: RAM, CPU, disk, kernel, daemon health
|
||||
docker_watchdog.sh ← watches the containers: memory, CPU, HTTP response, crashes
|
||||
Watchdogs/resource_watchdog.sh ← reduces pressure before healing attempts
|
||||
Watchdogs/docker_watchdog.sh ← heals containers (reads resource_watchdog state)
|
||||
Watchdogs/storage_watchdog.sh ← pool growth + runaway log detection
|
||||
Watchdogs/system_watchdog.sh ← last resort — reboots when healing has failed
|
||||
```
|
||||
|
||||
**The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops
|
||||
non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh`
|
||||
would see stopped containers on its next cycle and restart them — directly undoing the
|
||||
RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover.
|
||||
The system would eventually hit the reboot threshold anyway, having accomplished nothing.
|
||||
|
||||
**The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When
|
||||
`system_watchdog.sh` triggers a RAM emergency shutdown it writes
|
||||
`mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every
|
||||
cycle and defers all container restart logic until it clears. Health URL checks for
|
||||
excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still
|
||||
run. Everything else stands down.
|
||||
|
||||
**The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state
|
||||
file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh`
|
||||
checks how long ago the state file was modified. If it's more than 2 hours old while
|
||||
`mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running.
|
||||
`docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced
|
||||
indefinitely by a stale flag from a process that's no longer running.
|
||||
Scripts in this folder (daily restart, updates, network connect) are unaffected —
|
||||
they run on their own schedules via the maintenance orchestrators and are not part
|
||||
of the every-minute watchdog cycle. See `Watchdogs/README-Watchdogs.md` for the
|
||||
full coordination model between all four watchdogs.
|
||||
|
||||
---
|
||||
|
||||
@@ -247,7 +227,6 @@ indefinitely by a stale flag from a process that's no longer running.
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop — array start |
|
||||
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
||||
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
||||
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window |
|
||||
@@ -265,18 +244,12 @@ indefinitely by a stale flag from a process that's no longer running.
|
||||
```
|
||||
Array starts
|
||||
│
|
||||
├── docker_network_connect.sh ────── run once at start
|
||||
│ ensure networks + connections exist
|
||||
│ silent if correct, notify if creating
|
||||
│
|
||||
└── docker_watchdog.sh ──────────── continuous background loop (every 15min)
|
||||
Tier 1: memory, CPU, HTTP, required containers
|
||||
Tier 2: global unhealthy / OOM / crash / dead scan
|
||||
reads system_watchdog state (RAM emergency deferral)
|
||||
│
|
||||
│ (on skip list event → operator uses)
|
||||
└── Tools/watchdog_skip_list_manager.sh
|
||||
inspect state, clear after fixing root cause
|
||||
└── docker_network_connect.sh ────── run once at start
|
||||
ensure networks + connections exist
|
||||
silent if correct, notify if creating
|
||||
|
||||
Every minute (Orchestrators/watchdog_orchestrator.sh):
|
||||
└── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md
|
||||
|
||||
Daily maintenance window (1am):
|
||||
daily_sync_maintenance.sh
|
||||
|
||||
Reference in New Issue
Block a user