Update all docs to reflect watchdog restructure
README-Watchdogs.md: new tier diagram, updated script table, execution flow, state file table, and folder relationship diagram Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog, System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting README.md: updated schedule listing, folder description, daily cycle examples README-User_Script_Plug-in.md: stability_watchdog references throughout unRAID_Essentials/README + Manual: stability_watchdog, updated storage path Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
# ━━━━━ WATCHDOGS — Manual ━━━━━
|
||||
|
||||
Configuration reference, operational procedures, and troubleshooting for all four
|
||||
watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
|
||||
Configuration reference, operational procedures, and troubleshooting for all watchdog
|
||||
scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
|
||||
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
|
||||
|
||||
---
|
||||
@@ -11,8 +11,10 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
||||
- [Output Tiers](#output-tiers)
|
||||
- [resource_watchdog.sh](#resource_watchdogsh)
|
||||
- [docker_watchdog.sh](#docker_watchdogsh)
|
||||
- [storage_watchdog.sh](#storage_watchdogsh)
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
|
||||
- [stability_watchdog.sh](#stability_watchdogsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
@@ -23,7 +25,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
||||
All watchdog scripts use a two-tier output model: `echo` lines are always visible;
|
||||
`log` lines only appear when `--log` is passed.
|
||||
|
||||
All four watchdogs are **single-pass scripts** called once per minute by the orchestrator.
|
||||
All watchdog scripts are **single-pass scripts** called once per minute by the orchestrator.
|
||||
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
|
||||
visible. Per-check detail is suppressed on clean cycles.
|
||||
|
||||
@@ -285,9 +287,26 @@ docker start ContainerName
|
||||
|
||||
---
|
||||
|
||||
## storage_watchdog.sh
|
||||
## system_watchdog.sh
|
||||
|
||||
Runs third in the orchestrator sequence. Two independent checks per cycle:
|
||||
Thin orchestrator — runs `SYSTEM_WATCHDOG_SCRIPTS` from master.conf sequentially each cycle.
|
||||
Called third by `watchdog_orchestrator.sh`. Covers all system component watchdogs.
|
||||
Can also be run standalone to check all system components at once.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # run all system component watchdogs
|
||||
system_watchdog.sh --status # show configured scripts and their paths
|
||||
system_watchdog.sh --dry-run # preview without executing anything
|
||||
system_watchdog.sh --log # verbose output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## System/storage_watchdog.sh
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Two independent checks per cycle:
|
||||
growth rate detection (automatic, zero config) and oversize log detection.
|
||||
Uses its own strike state file — independent from docker_watchdog.
|
||||
|
||||
@@ -343,7 +362,7 @@ Log strikes auto-clear when the file drops below threshold.
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
storage_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
storage_watchdog.sh # single pass (called by system_watchdog.sh)
|
||||
storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings
|
||||
storage_watchdog.sh --dry-run # show what would be alerted/truncated
|
||||
storage_watchdog.sh --log # verbose per-container output
|
||||
@@ -351,7 +370,46 @@ storage_watchdog.sh --log # verbose per-container output
|
||||
|
||||
---
|
||||
|
||||
## system_watchdog.sh
|
||||
## System/webgui_watchdog.sh
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
|
||||
through three restart steps if unresponsive. Silent when healthy.
|
||||
|
||||
### Escalation Path
|
||||
|
||||
```
|
||||
WebGUI responding → exit 0 (silent)
|
||||
|
||||
Not responding:
|
||||
Step 1 — nginx restart → wait WEBGUI_NGINX_WAIT → recheck
|
||||
Step 2 — php-fpm restart → wait WEBGUI_PHP_WAIT → recheck
|
||||
Step 3 — emhttp restart → wait WEBGUI_EMHTTP_WAIT → recheck
|
||||
All three failed → critical notify, manual intervention needed
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WEBGUI_URL="http://localhost"
|
||||
WEBGUI_TIMEOUT=5
|
||||
WEBGUI_NGINX_WAIT=15
|
||||
WEBGUI_PHP_WAIT=10
|
||||
WEBGUI_EMHTTP_WAIT=30
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
webgui_watchdog.sh # single pass (called by system_watchdog.sh)
|
||||
webgui_watchdog.sh --status # current WebGUI state + nginx/php-fpm/emhttp status
|
||||
webgui_watchdog.sh --dry-run # show which services would be restarted
|
||||
webgui_watchdog.sh --log # verbose per-step output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## stability_watchdog.sh
|
||||
|
||||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||||
to reboot. Watches the server itself — not containers, not storage. Reboots only when
|
||||
@@ -425,10 +483,10 @@ reboots → something crashes again immediately → reboot again.
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
system_watchdog.sh --dry-run # run detection logic without rebooting
|
||||
system_watchdog.sh --status # thresholds, current state, strike counts
|
||||
system_watchdog.sh --log # verbose per-check output
|
||||
stability_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
stability_watchdog.sh --dry-run # run detection logic without rebooting
|
||||
stability_watchdog.sh --status # thresholds, current state, strike counts
|
||||
stability_watchdog.sh --log # verbose per-check output
|
||||
```
|
||||
|
||||
---
|
||||
@@ -616,7 +674,7 @@ storage_watchdog.sh --status
|
||||
# )
|
||||
```
|
||||
|
||||
### system_watchdog Rebooted Unexpectedly
|
||||
### stability_watchdog Rebooted Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check the reboot log (survives reboots):
|
||||
@@ -624,13 +682,13 @@ cat /boot/config/system_watchdog_reboots.db
|
||||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||||
|
||||
# Check syslog near the reboot time:
|
||||
grep "system_watchdog" /var/log/syslog | tail -20
|
||||
grep "stability_watchdog" /var/log/syslog | tail -20
|
||||
```
|
||||
|
||||
### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
|
||||
### stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
|
||||
|
||||
```bash
|
||||
# All four watchdogs run as single-pass scripts — there is no background process to check.
|
||||
# All watchdogs run as single-pass scripts — there is no background process to check.
|
||||
# If the orchestrator reports a timeout, one pass took longer than expected.
|
||||
|
||||
# Check the orchestrator itself:
|
||||
|
||||
Reference in New Issue
Block a user