Update all docs to reflect watchdog restructure

README-Watchdogs.md: new tier diagram, updated script table, execution flow,
  state file table, and folder relationship diagram
Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog,
  System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting
README.md: updated schedule listing, folder description, daily cycle examples
README-User_Script_Plug-in.md: stability_watchdog references throughout
unRAID_Essentials/README + Manual: stability_watchdog, updated storage path
Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
This commit is contained in:
Gmer4Lfe
2026-05-22 20:26:36 -04:00
parent cf21efaeea
commit 13ce8497f3
9 changed files with 150 additions and 77 deletions
+74 -16
View File
@@ -1,7 +1,7 @@
# ━━━━━ WATCHDOGS — Manual ━━━━━
Configuration reference, operational procedures, and troubleshooting for all four
watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
Configuration reference, operational procedures, and troubleshooting for all watchdog
scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
---
@@ -11,8 +11,10 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
- [Output Tiers](#output-tiers)
- [resource_watchdog.sh](#resource_watchdogsh)
- [docker_watchdog.sh](#docker_watchdogsh)
- [storage_watchdog.sh](#storage_watchdogsh)
- [system_watchdog.sh](#system_watchdogsh)
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
- [stability_watchdog.sh](#stability_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -23,7 +25,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
All watchdog scripts use a two-tier output model: `echo` lines are always visible;
`log` lines only appear when `--log` is passed.
All four watchdogs are **single-pass scripts** called once per minute by the orchestrator.
All watchdog scripts are **single-pass scripts** called once per minute by the orchestrator.
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
visible. Per-check detail is suppressed on clean cycles.
@@ -285,9 +287,26 @@ docker start ContainerName
---
## storage_watchdog.sh
## system_watchdog.sh
Runs third in the orchestrator sequence. Two independent checks per cycle:
Thin orchestrator — runs `SYSTEM_WATCHDOG_SCRIPTS` from master.conf sequentially each cycle.
Called third by `watchdog_orchestrator.sh`. Covers all system component watchdogs.
Can also be run standalone to check all system components at once.
### Usage
```bash
system_watchdog.sh # run all system component watchdogs
system_watchdog.sh --status # show configured scripts and their paths
system_watchdog.sh --dry-run # preview without executing anything
system_watchdog.sh --log # verbose output
```
---
## System/storage_watchdog.sh
Called by `system_watchdog.sh` each cycle. Two independent checks per cycle:
growth rate detection (automatic, zero config) and oversize log detection.
Uses its own strike state file — independent from docker_watchdog.
@@ -343,7 +362,7 @@ Log strikes auto-clear when the file drops below threshold.
### Usage
```bash
storage_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
storage_watchdog.sh # single pass (called by system_watchdog.sh)
storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings
storage_watchdog.sh --dry-run # show what would be alerted/truncated
storage_watchdog.sh --log # verbose per-container output
@@ -351,7 +370,46 @@ storage_watchdog.sh --log # verbose per-container output
---
## system_watchdog.sh
## System/webgui_watchdog.sh
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
through three restart steps if unresponsive. Silent when healthy.
### Escalation Path
```
WebGUI responding → exit 0 (silent)
Not responding:
Step 1 — nginx restart → wait WEBGUI_NGINX_WAIT → recheck
Step 2 — php-fpm restart → wait WEBGUI_PHP_WAIT → recheck
Step 3 — emhttp restart → wait WEBGUI_EMHTTP_WAIT → recheck
All three failed → critical notify, manual intervention needed
```
### Configuration
```bash
# master.conf
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30
```
### Usage
```bash
webgui_watchdog.sh # single pass (called by system_watchdog.sh)
webgui_watchdog.sh --status # current WebGUI state + nginx/php-fpm/emhttp status
webgui_watchdog.sh --dry-run # show which services would be restarted
webgui_watchdog.sh --log # verbose per-step output
```
---
## stability_watchdog.sh
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
to reboot. Watches the server itself — not containers, not storage. Reboots only when
@@ -425,10 +483,10 @@ reboots → something crashes again immediately → reboot again.
### Usage
```bash
system_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
system_watchdog.sh --dry-run # run detection logic without rebooting
system_watchdog.sh --status # thresholds, current state, strike counts
system_watchdog.sh --log # verbose per-check output
stability_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
stability_watchdog.sh --dry-run # run detection logic without rebooting
stability_watchdog.sh --status # thresholds, current state, strike counts
stability_watchdog.sh --log # verbose per-check output
```
---
@@ -616,7 +674,7 @@ storage_watchdog.sh --status
# )
```
### system_watchdog Rebooted Unexpectedly
### stability_watchdog Rebooted Unexpectedly
```bash
# Check the reboot log (survives reboots):
@@ -624,13 +682,13 @@ cat /boot/config/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot
# Check syslog near the reboot time:
grep "system_watchdog" /var/log/syslog | tail -20
grep "stability_watchdog" /var/log/syslog | tail -20
```
### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
### stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
```bash
# All four watchdogs run as single-pass scripts — there is no background process to check.
# All watchdogs run as single-pass scripts — there is no background process to check.
# If the orchestrator reports a timeout, one pass took longer than expected.
# Check the orchestrator itself: