Gmer4Lfe
8a3e22c9b8
New docs: Watchdogs/README-Watchdogs.md — design, relationships, script table, state file map Watchdogs/Manual-Watchdogs.md — full config reference for all 4 watchdogs Docker_Essentials/: README — remove docker_watchdog, update folder description and diagrams Manual — strip watchdog config sections, add pointer to Watchdogs/Manual unRAID_Essentials/: README — remove system/resource watchdog, update diagrams and script table Manual — strip system/resource watchdog sections, update TOC + config reference README.md: Add Watchdogs/ to folder structure Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start Fix daily cycle and monitoring diagrams
177 lines
9.2 KiB
Markdown
177 lines
9.2 KiB
Markdown
# ━━━━━ WATCHDOGS ━━━━━
|
|
|
|
**Four single-pass scripts that run every minute through `watchdog_orchestrator.sh`,
|
|
each with a clear lane:** reduce system pressure → heal containers → protect storage →
|
|
reboot if nothing else worked. They never run standalone loops. The orchestrator calls
|
|
them in order, once per cron cycle.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
|
|
|
**Container Memory Leaks Going Undetected for Days**
|
|
Emby's transcode session handling occasionally leaks memory. SABnzbd's Python process
|
|
expands slowly across downloads. Neither crashes dramatically — they just consume more
|
|
RAM until the system starts swapping. Docker reports both containers as `Up 14 days`.
|
|
Nothing alerts. By the time someone notices, the system has been degraded for hours.
|
|
|
|
Fix: `docker_watchdog.sh` — hard per-container memory ceilings. When a container
|
|
exceeds its limit the watchdog restarts it immediately. No strikes, no waiting.
|
|
A memory leak is not a transient spike.
|
|
|
|
**Containers That Look Running But Aren't Responding**
|
|
Docker reports a container as `Up` while its application layer has been frozen for
|
|
hours. The reverse proxy forwards traffic to a service that returns nothing. Users see
|
|
a broken page. Docker sees a healthy container.
|
|
|
|
Fix: `docker_watchdog.sh` — HTTP health checks on the actual service port every cycle.
|
|
Two consecutive non-responses trigger a restart. Process running and service responding
|
|
are not the same thing.
|
|
|
|
**System Pressure Causing Docker Watchdog to Undo Itself**
|
|
Resource pressure builds. RAM drops. docker_watchdog.sh tries to restart a container
|
|
into a system that's already swapping — the restarted container fails immediately
|
|
and goes on the skip list. The real problem (RAM pressure) is never addressed.
|
|
|
|
Fix: `resource_watchdog.sh` runs first in the orchestrator sequence. At Level 1 it
|
|
throttles downloaders. At Level 2 it pauses non-critical containers. At Level 3 it
|
|
stops heavy services and signals docker_watchdog to defer all restarts. By the time
|
|
docker_watchdog runs, the system has breathing room to actually heal.
|
|
|
|
**Runaway Log Files Filling a Pool Before Anyone Notices**
|
|
A game server container was offline for a year, restarted for a weekend, and wrote
|
|
130GB of logs to the appdata pool. The pool grew 13% in days. No alert fired —
|
|
nothing was watching for growth at the data level, only at the container level.
|
|
|
|
Fix: `storage_watchdog.sh` — growth-rate scan of every container's appdata directory
|
|
every cycle. No per-container configuration required. Runaway growth gets three
|
|
cycles to be confirmed, then alerts and (optionally) truncates logs automatically.
|
|
|
|
**Rootfs at 99% With SSH Failing Silently**
|
|
Rootfs fills. SSH stops accepting new connections. Docker can't write log files. State
|
|
files fail silently. The server is functionally dead but still technically running.
|
|
Nothing in the container layer can detect or recover from this — it requires a reboot.
|
|
|
|
Fix: `system_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
|
|
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
|
|
sequence so container healing and pressure reduction always get a chance first.
|
|
|
|
---
|
|
|
|
## ━━━ WHAT THIS FOLDER DOES ━━━
|
|
|
|
Four watchdogs. One purpose each. Fixed execution order via `watchdog_orchestrator.sh`.
|
|
|
|
```
|
|
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
|
|
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
|
|
Storage protection storage_watchdog.sh — pool growth rate + runaway log detection
|
|
Last resort system_watchdog.sh — reboot only when nothing else can recover
|
|
```
|
|
|
|
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
|
|
attempts restarts — containers restarted into a pressure-bound system just fail again.
|
|
Storage is checked after containers are healed — no false alerts from containers that
|
|
were already being restarted. System watchdog runs last — reboot is always the last
|
|
option, not the first.
|
|
|
|
**None of these scripts run standalone loops.** Each is a single-pass script called
|
|
once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles
|
|
startup grace, overlap protection, heartbeat, and sequencing.
|
|
|
|
---
|
|
|
|
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
|
|
|
```
|
|
Orchestrators/
|
|
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
|
|
──────────────────► docker_watchdog.sh (2nd)
|
|
──────────────────► storage_watchdog.sh (3rd)
|
|
──────────────────► system_watchdog.sh (4th — last resort)
|
|
|
|
Tools/
|
|
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
|
|
(operator utility — inspect + clear after fixing a crash-looping container)
|
|
|
|
Docker_Essentials/
|
|
All container lifecycle scripts (daily restart, updates, network) — unaffected.
|
|
docker_watchdog.sh coordinates with them via shared state, not direct calls.
|
|
|
|
unRAID_Essentials/
|
|
Server-level scripts (WebGUI restart, inotify tuning, log hygiene) — unaffected.
|
|
system_watchdog.sh runs in the same ecosystem but is independent of those scripts.
|
|
```
|
|
|
|
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
|
|
**watchdog_skip_list_manager.sh stays in Tools/** — it's an operator utility, not a watchdog.
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
|
|
|
| Script | Role | Called By |
|
|
|--------|------|-----------|
|
|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute |
|
|
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute |
|
|
| `storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `watchdog_orchestrator.sh` — 3rd every minute |
|
|
| `system_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
|
|
|
|
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
|
|
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
|
|
|
|
---
|
|
|
|
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
|
|
|
```
|
|
Every minute — watchdog_orchestrator.sh fires:
|
|
|
|
Step 1 — resource_watchdog.sh
|
|
│ RAM/load OK → pass through (no action)
|
|
│ Level 1 (soft): throttle SABnzbd + qBit download speeds
|
|
│ Level 2 (medium): further throttle + docker pause non-critical containers
|
|
│ Level 3 (hard): docker stop optional services
|
|
│ writes mem_shutdown_active=true → RW_STATE_FILE
|
|
│ ↓
|
|
Step 2 — docker_watchdog.sh
|
|
│ reads RW_STATE_FILE — if mem_shutdown_active=true: skip all restarts
|
|
│
|
|
│ Tier 1 — explicit per-container checks (configured in host*.conf):
|
|
│ memory hard limits → immediate restart (no strikes)
|
|
│ CPU high sustained → 2-strike restart
|
|
│ HTTP non-response → 2-strike restart
|
|
│ required stopped → restart (dependency ordering respected)
|
|
│
|
|
│ Tier 2 — global scan of all running containers:
|
|
│ unhealthy / OOM / crashloop / dead / non-zero exit → restart
|
|
│ N restarts in window → skip list + critical notify → human required
|
|
│
|
|
│ (skip list management) → Tools/watchdog_skip_list_manager.sh
|
|
│
|
|
Step 3 — storage_watchdog.sh
|
|
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
|
|
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
|
|
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
|
|
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
|
|
│
|
|
Step 4 — system_watchdog.sh
|
|
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
|
|
Tier 1 CRITICAL → immediate reboot (no strikes)
|
|
Tier 2 URGENT → reboot if OOM confirmed
|
|
Tier 3 STANDARD → N consecutive failures → reboot
|
|
Abort conditions → ZFS unhealthy / parity running / mover active
|
|
Rate limit → max N reboots per window before switching to notify
|
|
```
|
|
|
|
**State file coordination between scripts:**
|
|
|
|
| State File | Written By | Read By | Purpose |
|
|
|-----------|-----------|---------|---------|
|
|
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
|
|
| `SYS_WATCHDOG_STATE_FILE` | `system_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
|
|
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
|
|
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
|
|
| `STORAGE_WATCHDOG_STATE_FILE` | `storage_watchdog.sh` | itself | Growth + log strike counts |
|
|
| `WATCHDOG_APPDATA_GROWTH_FILE` | `storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
|