━━━━━ WATCHDOGS ━━━━━

Four tiers that run every 15 minutes through watchdog_orchestrator.sh, each with a clear lane: reduce system pressure → heal containers → check system components → reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle. System component checks (storage, WebGUI, network) run inside a thin system_watchdog.sh orchestrator that can also be run standalone.

━━━ THE PROBLEM THAT BUILT THIS ━━━

Container Memory Leaks Going Undetected for Days Emby's transcode session handling occasionally leaks memory. SABnzbd's Python process expands slowly across downloads. Neither crashes dramatically — they just consume more RAM until the system starts swapping. Docker reports both containers as Up 14 days. Nothing alerts. By the time someone notices, the system has been degraded for hours.

Fix: docker_watchdog.sh — hard per-container memory ceilings. When a container exceeds its limit the watchdog restarts it immediately. No strikes, no waiting. A memory leak is not a transient spike.

Containers That Look Running But Aren't Responding Docker reports a container as Up while its application layer has been frozen for hours. The reverse proxy forwards traffic to a service that returns nothing. Users see a broken page. Docker sees a healthy container.

Fix: docker_watchdog.sh — HTTP health checks on the actual service port every cycle. Two consecutive non-responses trigger a restart. Process running and service responding are not the same thing.

System Pressure Causing Docker Watchdog to Undo Itself Resource pressure builds. RAM drops. docker_watchdog.sh tries to restart a container into a system that's already swapping — the restarted container fails immediately and goes on the skip list. The real problem (RAM pressure) is never addressed.

Fix: resource_watchdog.sh runs first in the orchestrator sequence. At Level 1 it throttles downloaders. At Level 2 it pauses non-critical containers. At Level 3 it stops heavy services and signals docker_watchdog to defer all restarts. By the time docker_watchdog runs, the system has breathing room to actually heal.

Runaway Log Files Filling a Pool Before Anyone Notices A game server container was offline for a year, restarted for a weekend, and wrote 130GB of logs to the appdata pool. The pool grew 13% in days. No alert fired — nothing was watching for growth at the data level, only at the container level.

Fix: storage_watchdog.sh — growth-rate scan of every container's appdata directory every cycle. No per-container configuration required. Runaway growth gets three cycles to be confirmed, then alerts and (optionally) truncates logs automatically.

Rootfs at 99% With SSH Failing Silently Rootfs fills. SSH stops accepting new connections. Docker can't write log files. State files fail silently. The server is functionally dead but still technically running. Nothing in the container layer can detect or recover from this — it requires a reboot.

Fix: stability_watchdog.sh — watches the server itself: RAM, CPU, disk, kernel, daemon health. Only script in the stack authorized to reboot. Runs last in the orchestrator sequence so container healing and pressure reduction always get a chance first.

━━━ WHAT THIS FOLDER DOES ━━━

Four tiers. Fixed execution order via watchdog_orchestrator.sh.

Pressure reduction    resource_watchdog.sh   — throttle/pause/stop before healing fails
Container healing     docker_watchdog.sh     — memory, CPU, HTTP, required containers
System components     system_watchdog.sh     — thin orchestrator: storage + WebGUI health
  └─ System/          storage_watchdog.sh    — pool growth rate + runaway log detection
                      webgui_watchdog.sh     — WebGUI availability, nginx → php-fpm → emhttp
                      network_watchdog.sh    — internet, DDNS sync, Tailscale, NPM proxy
Last resort           stability_watchdog.sh  — reboot only when nothing else can recover

The execution order is the design. Resource pressure is reduced before docker_watchdog attempts restarts — containers restarted into a pressure-bound system just fail again. System component checks run after containers are healed. Stability watchdog runs last — reboot is always the last option, not the first.

None of these scripts run standalone loops. Each is a single-pass script called once every 15 minutes by Orchestrators/watchdog_orchestrator.sh. The orchestrator handles startup grace, overlap protection, heartbeat, and sequencing. system_watchdog.sh can also be run standalone to check all system component watchdogs at once.

━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

Orchestrators/
  watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh    (1st — every 15 min)
                            ──────────────────► docker_watchdog.sh     (2nd)
                            ──────────────────► system_watchdog.sh     (3rd — thin orchestrator)
                            ──────────────────► stability_watchdog.sh  (4th — last resort)

Watchdogs/System/              ← called by system_watchdog.sh each cycle
  storage_watchdog.sh          pool growth rate + runaway log detection
  webgui_watchdog.sh           WebGUI availability — nginx → php-fpm → emhttp escalation
  network_watchdog.sh          internet reachability, DDNS sync, Tailscale, NPM proxy

Tools/
  watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
    (operator utility — inspect + clear after fixing a crash-looping container)

Docker_Essentials/
  All container lifecycle scripts (daily restart, updates, network) — unaffected.
  docker_watchdog.sh coordinates with them via shared state, not direct calls.

watchdog_orchestrator.sh stays in Orchestrators/ — it's a job runner, not a watchdog. watchdog_skip_list_manager.sh stays in Tools/ — it's an operator utility, not a watchdog.

━━━ SCRIPTS IN THIS FOLDER ━━━

Script	Role	Called By
`resource_watchdog.sh`	Three-level pressure reduction — throttle, pause, stop	`watchdog_orchestrator.sh` — 1st every 15 min
`docker_watchdog.sh`	Two-tier container healing — memory, CPU, HTTP, required	`watchdog_orchestrator.sh` — 2nd every 15 min
`system_watchdog.sh`	Thin orchestrator — runs System/ component watchdogs in sequence	`watchdog_orchestrator.sh` — 3rd every 15 min
`stability_watchdog.sh`	Last-resort server watchdog — reboots when healing has failed	`watchdog_orchestrator.sh` — 4th every 15 min
`System/storage_watchdog.sh`	Pool growth rate + runaway log detection and remediation	`system_watchdog.sh` — every 15 min
`System/webgui_watchdog.sh`	WebGUI availability — nginx → php-fpm → emhttp escalation	`system_watchdog.sh` — every 15 min
`System/network_watchdog.sh`	Internet reachability, DDNS sync, Tailscale, NPM proxy	`system_watchdog.sh` — every 15 min

watchdog_orchestrator.sh is in Orchestrators/. watchdog_skip_list_manager.sh is in Tools/. Neither is a watchdog — they sit at the edges of this system.

━━━ HOW THE SCRIPTS RELATE ━━━

Every 15 minutes — watchdog_orchestrator.sh fires:

  Step 1 — resource_watchdog.sh
    │  RAM/load OK → pass through (no action)
    │  Level 1 (soft):   throttle SABnzbd + qBit download speeds
    │  Level 2 (medium): further throttle + docker pause non-critical containers
    │  Level 3 (hard):   docker stop optional services
    │                    writes mem_shutdown_active=true → RW_STATE_FILE
    │                                  ↓
  Step 2 — docker_watchdog.sh
    │  reads RW_STATE_FILE — if mem_shutdown_active=true: skip all restarts
    │
    │  Tier 1 — explicit per-container checks (configured in host*.conf):
    │    memory hard limits  → immediate restart (no strikes)
    │    CPU high sustained  → 2-strike restart
    │    HTTP non-response   → 2-strike restart
    │    required stopped    → restart (dependency ordering respected)
    │
    │  Tier 2 — global scan of all running containers:
    │    unhealthy / OOM / crashloop / dead / non-zero exit → restart
    │    N restarts in window → skip list + critical notify → human required
    │
    │  (skip list management) → Tools/watchdog_skip_list_manager.sh
    │
  Step 3 — system_watchdog.sh (thin orchestrator)
    │  runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially:
    │
    │  └─ storage_watchdog.sh
    │       growth rate scan: du -sm appdata/* → compare to previous cycle baseline
    │         growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
    │       log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
    │         oversize log found → 3-strike warn → truncate (if enabled) or alert
    │
    │  └─ webgui_watchdog.sh
    │       curl check → WebGUI responding → exit 0 (silent)
    │       not responding → nginx restart → wait → recheck
    │       still down    → php-fpm restart → wait → recheck
    │       still down    → emhttp restart → wait → recheck
    │       all three failed → critical notify, manual intervention needed
    │
    │  └─ network_watchdog.sh
    │       internet check → fail = alert + skip remaining (prevents false positives)
    │       DDNS: public IP vs dig @1.1.1.1 → mismatch = restart Cloudflare DDNS container
    │       Tailscale: status Running → pass; not running = notify (no auto-restart)
    │       NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
    │
  Step 4 — stability_watchdog.sh
       checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
       Tier 1 CRITICAL  → immediate reboot (no strikes)
       Tier 2 URGENT    → reboot if OOM confirmed
       Tier 3 STANDARD  → N consecutive failures → reboot
       Abort conditions → ZFS unhealthy / parity running / mover active
       Rate limit       → max N reboots per window before switching to notify

State file coordination between scripts:

State File	Written By	Read By	Purpose
`RW_STATE_FILE`	`resource_watchdog.sh`	`docker_watchdog.sh`	`mem_shutdown_active` flag — defer restarts during RAM emergency
`SYS_WATCHDOG_STATE_FILE`	`stability_watchdog.sh`	`docker_watchdog.sh`	`watchdog_cycle` heartbeat — stale guard (2hr timeout)
`WATCHDOG_STATE_FILE`	`docker_watchdog.sh`	itself	CPU/HTTP strike counts per container
`SYS_WATCHDOG_FAILED_FILE`	`docker_watchdog.sh`	`watchdog_skip_list_manager.sh`	Container skip list
`STORAGE_WATCHDOG_STATE_FILE`	`System/storage_watchdog.sh`	itself	Growth + log strike counts
`WATCHDOG_APPDATA_GROWTH_FILE`	`System/storage_watchdog.sh`	itself	Per-container size baseline for growth rate
`NETWORK_WATCHDOG_NPM_STATE_FILE`	`System/network_watchdog.sh`	itself	NPM proxy strike count

11 KiB Raw Blame History