# ━━━━━ WATCHDOGS ━━━━━

**Four tiers that run every minute through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → check system components →
reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
System component checks (storage, WebGUI) run inside a thin `system_watchdog.sh` orchestrator
that can also be run standalone.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━

**Container Memory Leaks Going Undetected for Days**
Emby's transcode session handling occasionally leaks memory. SABnzbd's Python process
expands slowly across downloads. Neither crashes dramatically — they just consume more
RAM until the system starts swapping. Docker reports both containers as `Up 14 days`.
Nothing alerts. By the time someone notices, the system has been degraded for hours.

Fix: `docker_watchdog.sh` — hard per-container memory ceilings. When a container
exceeds its limit the watchdog restarts it immediately. No strikes, no waiting.
A memory leak is not a transient spike.

**Containers That Look Running But Aren't Responding**
Docker reports a container as `Up` while its application layer has been frozen for
hours. The reverse proxy forwards traffic to a service that returns nothing. Users see
a broken page. Docker sees a healthy container.

Fix: `docker_watchdog.sh` — HTTP health checks on the actual service port every cycle.
Two consecutive non-responses trigger a restart. Process running and service responding
are not the same thing.

**System Pressure Causing Docker Watchdog to Undo Itself**
Resource pressure builds. RAM drops. docker_watchdog.sh tries to restart a container
into a system that's already swapping — the restarted container fails immediately
and goes on the skip list. The real problem (RAM pressure) is never addressed.

Fix: `resource_watchdog.sh` runs first in the orchestrator sequence. At Level 1 it
throttles downloaders. At Level 2 it pauses non-critical containers. At Level 3 it
stops heavy services and signals docker_watchdog to defer all restarts. By the time
docker_watchdog runs, the system has breathing room to actually heal.

**Runaway Log Files Filling a Pool Before Anyone Notices**
A game server container was offline for a year, restarted for a weekend, and wrote
130GB of logs to the appdata pool. The pool grew 13% in days. No alert fired —
nothing was watching for growth at the data level, only at the container level.

Fix: `storage_watchdog.sh` — growth-rate scan of every container's appdata directory
every cycle. No per-container configuration required. Runaway growth gets three
cycles to be confirmed, then alerts and (optionally) truncates logs automatically.

**Rootfs at 99% With SSH Failing Silently**
Rootfs fills. SSH stops accepting new connections. Docker can't write log files. State
files fail silently. The server is functionally dead but still technically running.
Nothing in the container layer can detect or recover from this — it requires a reboot.

Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
sequence so container healing and pressure reduction always get a chance first.

---

## ━━━ WHAT THIS FOLDER DOES ━━━

Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.

```
Pressure reduction    resource_watchdog.sh   — throttle/pause/stop before healing fails
Container healing     docker_watchdog.sh     — memory, CPU, HTTP, required containers
System components     system_watchdog.sh     — thin orchestrator: storage + WebGUI health
  └─ System/          storage_watchdog.sh    — pool growth rate + runaway log detection
                      webgui_watchdog.sh     — WebGUI availability, nginx → php-fpm → emhttp
                      network_watchdog.sh    — internet, DDNS sync, Tailscale, NPM proxy
Last resort           stability_watchdog.sh  — reboot only when nothing else can recover
```

**The execution order is the design.** Resource pressure is reduced before docker_watchdog
attempts restarts — containers restarted into a pressure-bound system just fail again.
System component checks run after containers are healed. Stability watchdog runs last —
reboot is always the last option, not the first.

**None of these scripts run standalone loops.** Each is a single-pass script called
once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles
startup grace, overlap protection, heartbeat, and sequencing. `system_watchdog.sh` can
also be run standalone to check all system component watchdogs at once.

---

## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

```
Orchestrators/
  watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh    (1st — every minute)
                            ──────────────────► docker_watchdog.sh     (2nd)
                            ──────────────────► system_watchdog.sh     (3rd — thin orchestrator)
                            ──────────────────► stability_watchdog.sh  (4th — last resort)

Watchdogs/System/              ← called by system_watchdog.sh each cycle
  storage_watchdog.sh          pool growth rate + runaway log detection
  webgui_watchdog.sh           WebGUI availability — nginx → php-fpm → emhttp escalation
  network_watchdog.sh          internet reachability, DDNS sync, Tailscale, NPM proxy

Tools/
  watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
    (operator utility — inspect + clear after fixing a crash-looping container)

Docker_Essentials/
  All container lifecycle scripts (daily restart, updates, network) — unaffected.
  docker_watchdog.sh coordinates with them via shared state, not direct calls.
```

**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
**watchdog_skip_list_manager.sh stays in Tools/** — it's an operator utility, not a watchdog.

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━

| Script | Role | Called By |
|--------|------|-----------|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute |
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute |
| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every minute |
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute |
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute |
| `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every minute |

> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.

---

## ━━━ HOW THE SCRIPTS RELATE ━━━

```
Every minute — watchdog_orchestrator.sh fires:

  Step 1 — resource_watchdog.sh
    │  RAM/load OK → pass through (no action)
    │  Level 1 (soft):   throttle SABnzbd + qBit download speeds
    │  Level 2 (medium): further throttle + docker pause non-critical containers
    │  Level 3 (hard):   docker stop optional services
    │                    writes mem_shutdown_active=true → RW_STATE_FILE
    │                                  ↓
  Step 2 — docker_watchdog.sh
    │  reads RW_STATE_FILE — if mem_shutdown_active=true: skip all restarts
    │
    │  Tier 1 — explicit per-container checks (configured in host*.conf):
    │    memory hard limits  → immediate restart (no strikes)
    │    CPU high sustained  → 2-strike restart
    │    HTTP non-response   → 2-strike restart
    │    required stopped    → restart (dependency ordering respected)
    │
    │  Tier 2 — global scan of all running containers:
    │    unhealthy / OOM / crashloop / dead / non-zero exit → restart
    │    N restarts in window → skip list + critical notify → human required
    │
    │  (skip list management) → Tools/watchdog_skip_list_manager.sh
    │
  Step 3 — system_watchdog.sh (thin orchestrator)
    │  runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially:
    │
    │  └─ storage_watchdog.sh
    │       growth rate scan: du -sm appdata/* → compare to previous cycle baseline
    │         growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
    │       log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
    │         oversize log found → 3-strike warn → truncate (if enabled) or alert
    │
    │  └─ webgui_watchdog.sh
    │       curl check → WebGUI responding → exit 0 (silent)
    │       not responding → nginx restart → wait → recheck
    │       still down    → php-fpm restart → wait → recheck
    │       still down    → emhttp restart → wait → recheck
    │       all three failed → critical notify, manual intervention needed
    │
    │  └─ network_watchdog.sh
    │       internet check → fail = alert + skip remaining (prevents false positives)
    │       DDNS: public IP vs dig @1.1.1.1 → mismatch = restart Cloudflare DDNS container
    │       Tailscale: status Running → pass; not running = notify (no auto-restart)
    │       NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
    │
  Step 4 — stability_watchdog.sh
       checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
       Tier 1 CRITICAL  → immediate reboot (no strikes)
       Tier 2 URGENT    → reboot if OOM confirmed
       Tier 3 STANDARD  → N consecutive failures → reboot
       Abort conditions → ZFS unhealthy / parity running / mover active
       Rate limit       → max N reboots per window before switching to notify
```

**State file coordination between scripts:**

| State File | Written By | Read By | Purpose |
|-----------|-----------|---------|---------|
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
