FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update all docs to reflect watchdog restructure

Browse Source 
README-Watchdogs.md: new tier diagram, updated script table, execution flow,
  state file table, and folder relationship diagram
Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog,
  System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting
README.md: updated schedule listing, folder description, daily cycle examples
README-User_Script_Plug-in.md: stability_watchdog references throughout
unRAID_Essentials/README + Manual: stability_watchdog, updated storage path
Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
This commit is contained in:
Gmer4Lfe
2026-05-22 20:26:36 -04:00
parent cf21efaeea
commit 13ce8497f3
9 changed files with 150 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Watchdogs/Manual-Watchdogs.md
+74 -16
View File
@@ -1,7 +1,7 @@
# ━━━━━ WATCHDOGS — Manual ━━━━━
Configuration reference, operational procedures, and troubleshooting for all four
watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
Configuration reference, operational procedures, and troubleshooting for all watchdog
scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
---
@@ -11,8 +11,10 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
- [Output Tiers](#output-tiers)
- [resource_watchdog.sh](#resource_watchdogsh)
- [docker_watchdog.sh](#docker_watchdogsh)
- [storage_watchdog.sh](#storage_watchdogsh)
- [system_watchdog.sh](#system_watchdogsh)
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
- [stability_watchdog.sh](#stability_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -23,7 +25,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
All watchdog scripts use a two-tier output model: `echo` lines are always visible;
`log` lines only appear when `--log` is passed.
All four watchdogs are **single-pass scripts** called once per minute by the orchestrator.
All watchdog scripts are **single-pass scripts** called once per minute by the orchestrator.
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
visible. Per-check detail is suppressed on clean cycles.
@@ -285,9 +287,26 @@ docker start ContainerName
---
## storage_watchdog.sh
## system_watchdog.sh
Runs third in the orchestrator sequence. Two independent checks per cycle:
Thin orchestrator — runs `SYSTEM_WATCHDOG_SCRIPTS` from master.conf sequentially each cycle.
Called third by `watchdog_orchestrator.sh`. Covers all system component watchdogs.
Can also be run standalone to check all system components at once.
### Usage
```bash
system_watchdog.sh # run all system component watchdogs
system_watchdog.sh --status # show configured scripts and their paths
system_watchdog.sh --dry-run # preview without executing anything
system_watchdog.sh --log # verbose output
```
---
## System/storage_watchdog.sh
Called by `system_watchdog.sh` each cycle. Two independent checks per cycle:
growth rate detection (automatic, zero config) and oversize log detection.
Uses its own strike state file — independent from docker_watchdog.
@@ -343,7 +362,7 @@ Log strikes auto-clear when the file drops below threshold.
### Usage
```bash
storage_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
storage_watchdog.sh # single pass (called by system_watchdog.sh)
storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings
storage_watchdog.sh --dry-run # show what would be alerted/truncated
storage_watchdog.sh --log # verbose per-container output
@@ -351,7 +370,46 @@ storage_watchdog.sh --log # verbose per-container output
---
## system_watchdog.sh
## System/webgui_watchdog.sh
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
through three restart steps if unresponsive. Silent when healthy.
### Escalation Path
```
WebGUI responding → exit 0 (silent)
Not responding:
Step 1 — nginx restart → wait WEBGUI_NGINX_WAIT → recheck
Step 2 — php-fpm restart → wait WEBGUI_PHP_WAIT → recheck
Step 3 — emhttp restart → wait WEBGUI_EMHTTP_WAIT → recheck
All three failed → critical notify, manual intervention needed
```
### Configuration
```bash
# master.conf
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30
```
### Usage
```bash
webgui_watchdog.sh # single pass (called by system_watchdog.sh)
webgui_watchdog.sh --status # current WebGUI state + nginx/php-fpm/emhttp status
webgui_watchdog.sh --dry-run # show which services would be restarted
webgui_watchdog.sh --log # verbose per-step output
```
---
## stability_watchdog.sh
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
to reboot. Watches the server itself — not containers, not storage. Reboots only when
@@ -425,10 +483,10 @@ reboots → something crashes again immediately → reboot again.
### Usage
```bash
system_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
system_watchdog.sh --dry-run # run detection logic without rebooting
system_watchdog.sh --status # thresholds, current state, strike counts
system_watchdog.sh --log # verbose per-check output
stability_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
stability_watchdog.sh --dry-run # run detection logic without rebooting
stability_watchdog.sh --status # thresholds, current state, strike counts
stability_watchdog.sh --log # verbose per-check output
```
---
@@ -616,7 +674,7 @@ storage_watchdog.sh --status
# )
```
### system_watchdog Rebooted Unexpectedly
### stability_watchdog Rebooted Unexpectedly
```bash
# Check the reboot log (survives reboots):
@@ -624,13 +682,13 @@ cat /boot/config/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot
# Check syslog near the reboot time:
grep "system_watchdog" /var/log/syslog | tail -20
grep "stability_watchdog" /var/log/syslog | tail -20
```
### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
### stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
```bash
# All four watchdogs run as single-pass scripts — there is no background process to check.
# All watchdogs run as single-pass scripts — there is no background process to check.
# If the orchestrator reports a timeout, one pass took longer than expected.
# Check the orchestrator itself:
Watchdogs/README-Watchdogs.md
+46 -31
View File
@@ -1,9 +1,10 @@
# ━━━━━ WATCHDOGS ━━━━━
**Four single-pass scripts that run every minute through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → protect storage →
reboot if nothing else worked. They never run standalone loops. The orchestrator calls
them in order, once per cron cycle.
**Four tiers that run every minute through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → check system components →
reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
System component checks (storage, WebGUI) run inside a thin `system_watchdog.sh` orchestrator
that can also be run standalone.
---
@@ -52,7 +53,7 @@ Rootfs fills. SSH stops accepting new connections. Docker can't write log files.
files fail silently. The server is functionally dead but still technically running.
Nothing in the container layer can detect or recover from this — it requires a reboot.
Fix: `system_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
sequence so container healing and pressure reduction always get a chance first.
@@ -60,24 +61,26 @@ sequence so container healing and pressure reduction always get a chance first.
## ━━━ WHAT THIS FOLDER DOES ━━━
Four watchdogs. One purpose each. Fixed execution order via `watchdog_orchestrator.sh`.
Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
```
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
Storage protection storage_watchdog.sh — pool growth rate + runaway log detection
Last resort system_watchdog.sh — reboot only when nothing else can recover
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
Last resort stability_watchdog.sh — reboot only when nothing else can recover
```
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
attempts restarts — containers restarted into a pressure-bound system just fail again.
Storage is checked after containers are healed — no false alerts from containers that
were already being restarted. System watchdog runs last — reboot is always the last
option, not the first.
System component checks run after containers are healed. Stability watchdog runs last —
reboot is always the last option, not the first.
**None of these scripts run standalone loops.** Each is a single-pass script called
once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles
startup grace, overlap protection, heartbeat, and sequencing.
startup grace, overlap protection, heartbeat, and sequencing. `system_watchdog.sh` can
also be run standalone to check all system component watchdogs at once.
---
@@ -85,10 +88,14 @@ startup grace, overlap protection, heartbeat, and sequencing.
```
Orchestrators/
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
──────────────────► docker_watchdog.sh (2nd)
──────────────────► storage_watchdog.sh (3rd)
──────────────────► system_watchdog.sh (4th — last resort)
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
──────────────────► docker_watchdog.sh (2nd)
──────────────────► system_watchdog.sh (3rd — thin orchestrator)
──────────────────► stability_watchdog.sh (4th — last resort)
Watchdogs/System/ ← called by system_watchdog.sh each cycle
storage_watchdog.sh pool growth rate + runaway log detection
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
Tools/
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
@@ -97,10 +104,6 @@ Tools/
Docker_Essentials/
All container lifecycle scripts (daily restart, updates, network) — unaffected.
docker_watchdog.sh coordinates with them via shared state, not direct calls.
unRAID_Essentials/
Server-level scripts (WebGUI restart, inotify tuning, log hygiene) — unaffected.
system_watchdog.sh runs in the same ecosystem but is independent of those scripts.
```
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
@@ -114,8 +117,10 @@ unRAID_Essentials/
|--------|------|-----------|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute |
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute |
| `storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `watchdog_orchestrator.sh` — 3rd every minute |
| `system_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every minute |
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute |
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute |
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
@@ -149,13 +154,23 @@ Every minute — watchdog_orchestrator.sh fires:
│
│ (skip list management) → Tools/watchdog_skip_list_manager.sh
│
Step 3 — storage_watchdog.sh
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
Step 3 — system_watchdog.sh (thin orchestrator)
│ runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially:
│
Step 4 — system_watchdog.sh
│ └─ storage_watchdog.sh
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
│
│ └─ webgui_watchdog.sh
│ curl check → WebGUI responding → exit 0 (silent)
│ not responding → nginx restart → wait → recheck
│ still down → php-fpm restart → wait → recheck
│ still down → emhttp restart → wait → recheck
│ all three failed → critical notify, manual intervention needed
│
Step 4 — stability_watchdog.sh
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
Tier 1 CRITICAL → immediate reboot (no strikes)
Tier 2 URGENT → reboot if OOM confirmed
@@ -169,8 +184,8 @@ Every minute — watchdog_orchestrator.sh fires:
| State File | Written By | Read By | Purpose |
|-----------|-----------|---------|---------|
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
| `SYS_WATCHDOG_STATE_FILE` | `system_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `STORAGE_WATCHDOG_STATE_FILE` | `storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |