FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update all docs to reflect watchdog restructure

Browse Source 
README-Watchdogs.md: new tier diagram, updated script table, execution flow,
  state file table, and folder relationship diagram
Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog,
  System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting
README.md: updated schedule listing, folder description, daily cycle examples
README-User_Script_Plug-in.md: stability_watchdog references throughout
unRAID_Essentials/README + Manual: stability_watchdog, updated storage path
Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
This commit is contained in:
Gmer4Lfe
2026-05-22 20:26:36 -04:00
parent cf21efaeea
commit 13ce8497f3
9 changed files with 150 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Watchdogs/README-Watchdogs.md
+46 -31
View File
@@ -1,9 +1,10 @@
# ━━━━━ WATCHDOGS ━━━━━
**Four single-pass scripts that run every minute through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → protect storage →
reboot if nothing else worked. They never run standalone loops. The orchestrator calls
them in order, once per cron cycle.
**Four tiers that run every minute through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → check system components →
reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
System component checks (storage, WebGUI) run inside a thin `system_watchdog.sh` orchestrator
that can also be run standalone.
---
@@ -52,7 +53,7 @@ Rootfs fills. SSH stops accepting new connections. Docker can't write log files.
files fail silently. The server is functionally dead but still technically running.
Nothing in the container layer can detect or recover from this — it requires a reboot.
Fix: `system_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
sequence so container healing and pressure reduction always get a chance first.
@@ -60,24 +61,26 @@ sequence so container healing and pressure reduction always get a chance first.
## ━━━ WHAT THIS FOLDER DOES ━━━
Four watchdogs. One purpose each. Fixed execution order via `watchdog_orchestrator.sh`.
Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
```
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
Storage protection storage_watchdog.sh — pool growth rate + runaway log detection
Last resort system_watchdog.sh — reboot only when nothing else can recover
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
Last resort stability_watchdog.sh — reboot only when nothing else can recover
```
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
attempts restarts — containers restarted into a pressure-bound system just fail again.
Storage is checked after containers are healed — no false alerts from containers that
were already being restarted. System watchdog runs last — reboot is always the last
option, not the first.
System component checks run after containers are healed. Stability watchdog runs last —
reboot is always the last option, not the first.
**None of these scripts run standalone loops.** Each is a single-pass script called
once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles
startup grace, overlap protection, heartbeat, and sequencing.
startup grace, overlap protection, heartbeat, and sequencing. `system_watchdog.sh` can
also be run standalone to check all system component watchdogs at once.
---
@@ -85,10 +88,14 @@ startup grace, overlap protection, heartbeat, and sequencing.
```
Orchestrators/
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
──────────────────► docker_watchdog.sh (2nd)
──────────────────► storage_watchdog.sh (3rd)
──────────────────► system_watchdog.sh (4th — last resort)
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
──────────────────► docker_watchdog.sh (2nd)
──────────────────► system_watchdog.sh (3rd — thin orchestrator)
──────────────────► stability_watchdog.sh (4th — last resort)
Watchdogs/System/ ← called by system_watchdog.sh each cycle
storage_watchdog.sh pool growth rate + runaway log detection
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
Tools/
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
@@ -97,10 +104,6 @@ Tools/
Docker_Essentials/
All container lifecycle scripts (daily restart, updates, network) — unaffected.
docker_watchdog.sh coordinates with them via shared state, not direct calls.
unRAID_Essentials/
Server-level scripts (WebGUI restart, inotify tuning, log hygiene) — unaffected.
system_watchdog.sh runs in the same ecosystem but is independent of those scripts.
```
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
@@ -114,8 +117,10 @@ unRAID_Essentials/
|--------|------|-----------|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute |
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute |
| `storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `watchdog_orchestrator.sh` — 3rd every minute |
| `system_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every minute |
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute |
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute |
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
@@ -149,13 +154,23 @@ Every minute — watchdog_orchestrator.sh fires:
│
│ (skip list management) → Tools/watchdog_skip_list_manager.sh
│
Step 3 — storage_watchdog.sh
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
Step 3 — system_watchdog.sh (thin orchestrator)
│ runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially:
│
Step 4 — system_watchdog.sh
│ └─ storage_watchdog.sh
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
│
│ └─ webgui_watchdog.sh
│ curl check → WebGUI responding → exit 0 (silent)
│ not responding → nginx restart → wait → recheck
│ still down → php-fpm restart → wait → recheck
│ still down → emhttp restart → wait → recheck
│ all three failed → critical notify, manual intervention needed
│
Step 4 — stability_watchdog.sh
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
Tier 1 CRITICAL → immediate reboot (no strikes)
Tier 2 URGENT → reboot if OOM confirmed
@@ -169,8 +184,8 @@ Every minute — watchdog_orchestrator.sh fires:
| State File | Written By | Read By | Purpose |
|-----------|-----------|---------|---------|
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
| `SYS_WATCHDOG_STATE_FILE` | `system_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `STORAGE_WATCHDOG_STATE_FILE` | `storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |