Varaverk/Watchdogs/README-Watchdogs.md

━━━━━ WATCHDOGS ━━━━━

Four tiers that run every 15 minutes through watchdog_orchestrator.sh, each with a clear lane: reduce system pressure → heal containers → check system components → reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle. System component checks (storage, WebGUI, network, conf cache) run inside a thin system_watchdog.sh orchestrator that can also be run standalone.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━

Container Memory Leaks Going Undetected for Days Emby's transcode session handling occasionally leaks memory. SABnzbd's Python process expands slowly across downloads. Neither crashes dramatically — they just consume more RAM until the system starts swapping. Docker reports both containers as Up 14 days. Nothing alerts. By the time someone notices, the system has been degraded for hours.

Fix: docker_watchdog.sh — hard per-container memory ceilings. When a container exceeds its limit the watchdog restarts it immediately. No strikes, no waiting. A memory leak is not a transient spike.

Containers That Look Running But Aren't Responding Docker reports a container as Up while its application layer has been frozen for hours. The reverse proxy forwards traffic to a service that returns nothing. Users see a broken page. Docker sees a healthy container.

Fix: docker_watchdog.sh — HTTP health checks on the actual service port every cycle. Two consecutive non-responses trigger a restart. Process running and service responding are not the same thing.

System Pressure Causing Docker Watchdog to Undo Itself Resource pressure builds. RAM drops. docker_watchdog.sh tries to restart a container into a system that's already swapping — the restarted container fails immediately and goes on the skip list. The real problem (RAM pressure) is never addressed.

Fix: resource_watchdog.sh runs first in the orchestrator sequence. At Level 1 it throttles downloaders. At Level 2 it pauses non-critical containers. At Level 3 it stops heavy services and signals docker_watchdog to defer all restarts. By the time docker_watchdog runs, the system has breathing room to actually heal.

Runaway Log Files Filling a Pool Before Anyone Notices A game server container was offline for a year, restarted for a weekend, and wrote 130GB of logs to the appdata pool. The pool grew 13% in days. No alert fired — nothing was watching for growth at the data level, only at the container level.

Fix: storage_watchdog.sh — growth-rate scan of every container's appdata directory every cycle. No per-container configuration required. Runaway growth gets three cycles to be confirmed, then alerts and (optionally) truncates logs automatically.

Rootfs at 99% With SSH Failing Silently Rootfs fills. SSH stops accepting new connections. Docker can't write log files. State files fail silently. The server is functionally dead but still technically running. Nothing in the container layer can detect or recover from this — it requires a reboot.

Fix: stability_watchdog.sh — watches the server itself: RAM, CPU, disk, kernel, daemon health. Only script in the stack authorized to reboot. Runs last in the orchestrator sequence so container healing and pressure reduction always get a chance first.

Partner Conf Backup Going Stale During a Long Outage The partner goes offline overnight. On the next reboot (planned maintenance), the persistent conf backup saved at shutdown is stale — it reflects the state from before the outage, not the most recent live values. conf_cache_restore.sh loads it faithfully, but fallback.sh may be working with old tier delays or container names.

Fix: conf_cache_watchdog.sh — refreshes the persistent backup from the RAM cache every 15 minutes while the partner is offline, keeping it current throughout the outage.

---

━━━ WHAT THIS FOLDER DOES ━━━

Four tiers. Fixed execution order via watchdog_orchestrator.sh.

Pressure reduction    resource_watchdog.sh   — throttle/pause/stop before healing fails
Container healing     docker_watchdog.sh     — memory, CPU, HTTP, required containers
System components     system_watchdog.sh     — thin orchestrator: system component health
  └─ System/          storage_watchdog.sh    — pool growth rate + runaway log detection
                      network_watchdog.sh    — internet, DDNS sync, Tailscale, NPM proxy
                      conf_cache_watchdog.sh — maintain persistent partner conf backup
Last resort           stability_watchdog.sh  — reboot only when nothing else can recover

Plugin/unraid/Watchdogs/System/webgui_watchdog.sh is also called by system_watchdog.sh but lives in the plugin tree because it calls Unraid-specific service commands (nginx, php-fpm, emhttp via platform adapter). See Plugin/unraid/Watchdogs/System/.

The execution order is the design. Resource pressure is reduced before docker_watchdog attempts restarts — containers restarted into a pressure-bound system just fail again. System component checks run after containers are healed. Stability watchdog runs last — reboot is always the last option, not the first.

None of these scripts run standalone loops. Each is a single-pass script called once every 15 minutes by Orchestrators/watchdog_orchestrator.sh. The orchestrator handles startup grace, overlap protection, heartbeat, and sequencing. system_watchdog.sh can also be run standalone to check all system component watchdogs at once.

---

━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

Orchestrators/
  watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh    (1st — every 15 min)
                            ──────────────────► docker_watchdog.sh     (2nd)
                            ──────────────────► system_watchdog.sh     (3rd — thin orchestrator)
                            ──────────────────► stability_watchdog.sh  (4th — last resort)

Watchdogs/System/              ← called by system_watchdog.sh each cycle
  storage_watchdog.sh          pool growth rate + runaway log detection
  network_watchdog.sh          internet reachability, DDNS sync, Tailscale, NPM proxy
  conf_cache_watchdog.sh       maintain persistent partner conf backup during outages

Plugin/unraid/Watchdogs/System/
  webgui_watchdog.sh           WebGUI availability — nginx → php-fpm → emhttp escalation
                               (Unraid-specific platform calls — lives in plugin tree)

Plugin/unraid/System_Essentials/
  unraid_api_key_renew.sh      re-registers the Varaverk plugin API key each watchdog cycle
                               (inserted between docker_watchdog and stability_watchdog)

Tools/
  watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
    (operator utility — inspect + clear after fixing a crash-looping container)

Docker_Essentials/
  All container lifecycle scripts (daily restart, updates, network) — unaffected.
  docker_watchdog.sh coordinates with them via shared state, not direct calls.

System_Essentials/
  conf_cache_save.sh / conf_cache_restore.sh — at-stop/at-start bookends for the
  persistent backup that conf_cache_watchdog.sh keeps current in between.

watchdog_orchestrator.sh stays in Orchestrators/ — it's a job runner, not a watchdog. watchdog_skip_list_manager.sh stays in Tools/ — it's an operator utility, not a watchdog.

---

━━━ SCRIPTS IN THIS FOLDER ━━━

Script	Role	Called By
resource_watchdog.sh	Three-level pressure reduction — throttle, pause, stop	watchdog_orchestrator.sh — 1st every 15 min
docker_watchdog.sh	Two-tier container healing — memory, CPU, HTTP, required	watchdog_orchestrator.sh — 2nd every 15 min
system_watchdog.sh	Thin orchestrator — runs SYSTEM_WATCHDOG_SCRIPTS in sequence	watchdog_orchestrator.sh — 3rd every 15 min
stability_watchdog.sh	Last-resort server watchdog — reboots when healing has failed	watchdog_orchestrator.sh — 4th every 15 min
System/storage_watchdog.sh	Pool growth rate + runaway log detection and remediation	system_watchdog.sh — every 15 min
System/network_watchdog.sh	Internet reachability, DDNS sync, Tailscale, NPM proxy	system_watchdog.sh — every 15 min
System/conf_cache_watchdog.sh	Refresh persistent partner conf backup while partner is offline	system_watchdog.sh — every 15 min

Plugin/unraid/Watchdogs/System/webgui_watchdog.sh is also called by system_watchdog.sh but is not in this folder. watchdog_orchestrator.sh is in Orchestrators/. watchdog_skip_list_manager.sh is in Tools/. Neither is a watchdog.

---

━━━ HOW THE SCRIPTS RELATE ━━━

Every 15 minutes — watchdog_orchestrator.sh fires:

  Step 1 — resource_watchdog.sh
    │  RAM/load OK → pass through (no action)
    │  Level 1 (soft):   throttle SABnzbd + qBit download speeds
    │  Level 2 (medium): further throttle + docker pause non-critical containers
    │  Level 3 (hard):   docker stop optional services
    │                    writes mem_shutdown_active=true → RW_STATE_FILE
    │                                  ↓
  Step 2 — docker_watchdog.sh
    │  reads RW_STATE_FILE — if mem_shutdown_active=true: skip all restarts
    │
    │  Tier 1 — explicit per-container checks (configured in host*.conf):
    │    memory hard limits  → immediate restart (no strikes)
    │    CPU high sustained  → 2-strike restart
    │    HTTP non-response   → 2-strike restart
    │    required stopped    → restart (dependency ordering respected)
    │
    │  Tier 2 — global scan of all running containers:
    │    unhealthy / OOM / crashloop / dead / non-zero exit → restart
    │    N restarts in window → skip list + critical notify → human required
    │
    │  (skip list management) → Tools/watchdog_skip_list_manager.sh
    │
  Step 3 — system_watchdog.sh (thin orchestrator)
    │  runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially:
    │
    │  └─ storage_watchdog.sh
    │       growth rate scan: du -sm appdata/* → compare to previous cycle baseline
    │         growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
    │       log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
    │         oversize log found → 3-strike warn → truncate (if enabled) or alert
    │
    │  └─ Plugin/unraid/Watchdogs/System/webgui_watchdog.sh
    │       curl check → WebGUI responding → exit 0 (silent)
    │       not responding → nginx restart → wait → recheck
    │       still down    → php-fpm restart → wait → recheck
    │       still down    → emhttp restart → wait → recheck
    │       all three failed → critical notify, manual intervention needed
    │
    │  └─ network_watchdog.sh
    │       internet check → fail = alert + skip remaining (prevents false positives)
    │       DDNS: public IP vs dig @1.1.1.1 → mismatch = restart Cloudflare DDNS container
    │       Tailscale: status Running → pass; not running = notify (no auto-restart)
    │       NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
    │
    │  └─ conf_cache_watchdog.sh
    │       remote online  → remove persistent backup (conf_sync gets fresh on next boot)
    │       remote offline → refresh backup from RAM cache → backup stays current
    │       silent when remote is online and no backup exists (normal state)
    │
    │  (+ Plugin/unraid/System_Essentials/unraid_api_key_renew.sh — between steps 3 and 4)
    │
  Step 4 — stability_watchdog.sh
       checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
       Tier 1 CRITICAL  → immediate reboot (no strikes)
       Tier 2 URGENT    → reboot if OOM confirmed
       Tier 3 STANDARD  → N consecutive failures → reboot
       Abort conditions → ZFS unhealthy / parity running / mover active
       Rate limit       → max N reboots per window before switching to notify

State file coordination between scripts:

State File	Written By	Read By	Purpose
RW_STATE_FILE	resource_watchdog.sh	docker_watchdog.sh	mem_shutdown_active flag — defer restarts during RAM emergency
SYS_WATCHDOG_STATE_FILE	stability_watchdog.sh	docker_watchdog.sh	watchdog_cycle heartbeat — stale guard (2hr timeout)
WATCHDOG_STATE_FILE	docker_watchdog.sh	itself	CPU/HTTP strike counts per container
DOCKER_WATCHDOG_FAILED_FILE	docker_watchdog.sh	watchdog_skip_list_manager.sh	Container skip list
STORAGE_WATCHDOG_STATE_FILE	System/storage_watchdog.sh	itself	Growth + log strike counts
WATCHDOG_APPDATA_GROWTH_FILE	System/storage_watchdog.sh	itself	Per-container size baseline for growth rate
NETWORK_WATCHDOG_NPM_STATE_FILE	System/network_watchdog.sh	itself	NPM proxy strike count

All state files are in $STATE_DIR (adapts to storage mode). See master.conf for actual variable values.

---

━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━

These run unattended every 15 minutes and hold the only irreversible remedy in the ecosystem, so the guards matter as much as the checks. Each script's header documents its full set; these are the folder-wide ones.

Nothing acts on a single reading. Every check that can trigger an action is strike-gated. One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves when the condition resolves, so a transient blip never accumulates toward an alarm across unrelated weeks.

Escalation is asymmetric on purpose. Protective actions happen fast; undoing them happens slowly. resource_watchdog.sh throttles before it pauses and pauses before it stops, then requires RW_RECOVER_CYCLES clear cycles and de-escalates one level at a time. Recovering on a single good reading would flap under sustained load.

Ownership boundaries are enforced, not conventional. Container health belongs to docker_watchdog.sh; stability_watchdog.sh deliberately does not check it, and its Docker daemon check writes a flag for docker_watchdog.sh rather than acting. Two watchdogs remediating the same subsystem would race.

Watchdogs coordinate through state files. resource_watchdog.sh publishes mem_shutdown_active so docker_watchdog.sh defers restarts — otherwise one would stop containers to free RAM while the other restarted them to restore health. The reader applies a staleness guard so a crashed writer cannot suppress restarts forever.

Data safety outranks uptime. stability_watchdog.sh aborts a reboot while a ZFS pool is unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system cannot recover from and which worsen every cycle — bypasses that.

Reboot loops end in shutdown, not more reboots. No more than SYS_WATCHDOG_REBOOT_LIMIT reboots within SYS_WATCHDOG_REBOOT_WINDOW_HRS. On hitting the limit the host powers off: a fault that survives repeated reboots will not be fixed by more of them, and a cleanly-down box is better than one cycling endlessly.

An aborted reboot restores what it stopped. An EXIT trap is armed the moment containers start being stopped for a reboot and disarmed only once the reboot commits. If the script dies in between, everything it stopped comes back — the failure mode is a running system, never a host left stripped with no reboot.

An unconfigured job list fails loudly. An empty WATCHDOG_ORCHESTRATOR_SCRIPTS or SYSTEM_WATCHDOG_SCRIPTS would otherwise report "0/0 passed" and exit 0 every cycle — indistinguishable from healthy while nothing is monitored at all.

Verify the path, not the process. network_watchdog.sh checks NPM by fetching an external URL rather than asking whether the container is running. A running container behind broken DNS or a broken upstream still serves nothing.