Update all docs to reflect watchdog restructure

README-Watchdogs.md: new tier diagram, updated script table, execution flow,
  state file table, and folder relationship diagram
Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog,
  System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting
README.md: updated schedule listing, folder description, daily cycle examples
README-User_Script_Plug-in.md: stability_watchdog references throughout
unRAID_Essentials/README + Manual: stability_watchdog, updated storage path
Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
This commit is contained in:
Gmer4Lfe
2026-05-22 20:26:36 -04:00
parent cf21efaeea
commit 13ce8497f3
9 changed files with 150 additions and 77 deletions
+1 -1
View File
@@ -82,7 +82,7 @@ drive SMART attributes, ZFS pool state, ARC statistics, kernel memory pressure.
``` ```
Monitors/ ← observes and reports (this folder) Monitors/ ← observes and reports (this folder)
Docker_Essentials/ ← acts on containers (docker_watchdog starts/stops) Docker_Essentials/ ← acts on containers (docker_watchdog starts/stops)
unRAID_Essentials/ ← acts on the server (system_watchdog, inotify_tuning) unRAID_Essentials/ ← acts on the server (stability_watchdog, inotify_tuning)
Fallback/ ← acts on the full stack (failover, handback) Fallback/ ← acts on the full stack (failover, handback)
Rsync/ ← calls bandwidth_monitor.sh (auto-logs each sync) Rsync/ ← calls bandwidth_monitor.sh (auto-logs each sync)
``` ```
+8 -8
View File
@@ -111,14 +111,14 @@ docker_syslog_filter.sh — suppresses Docker veth noise before containers crea
php_fpm_max_children.sh — sets PHP-FPM workers to 250 before WebGUI serves requests php_fpm_max_children.sh — sets PHP-FPM workers to 250 before WebGUI serves requests
ramdisk_setup.sh — creates /mnt/ramdisk_transcodes and symlink before Emby starts ramdisk_setup.sh — creates /mnt/ramdisk_transcodes and symlink before Emby starts
docker_network_connect.sh— ensures custom networks exist + containers connected docker_network_connect.sh— ensures custom networks exist + containers connected
system_watchdog.sh ← continuous — three-tier server stability watchdog stability_watchdog.sh ← last-resort — three-tier server stability watchdog
docker_watchdog.sh ← continuous — two-tier container healing watchdog docker_watchdog.sh ← continuous — two-tier container healing watchdog
failover.sh ← continuous — mutual failover state machine failover.sh ← continuous — mutual failover state machine
``` ```
**After running:** **After running:**
One-shot scripts exit cleanly. Three continuous processes run in the background until One-shot scripts exit cleanly. Three continuous processes run in the background until
the array stops. Check with: `pgrep -f "system_watchdog\|docker_watchdog\|failover"` the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|failover"`
--- ---
@@ -899,7 +899,7 @@ Background: NO
**What it shows:** **What it shows:**
Strike counts for all monitored containers. Current skip list with running/stopped state. Strike counts for all monitored containers. Current skip list with running/stopped state.
Whether startup grace period is active and how long remains. Whether RAM emergency Whether startup grace period is active and how long remains. Whether RAM emergency
deferral is active (mem_shutdown_active from system_watchdog). Last cycle timing. deferral is active (mem_shutdown_active from stability_watchdog). Last cycle timing.
--- ---
@@ -1200,7 +1200,7 @@ flip count, session split ramdisk vs SSD).
## 🖥️ UNRAID_ESSENTIALS ## 🖥️ UNRAID_ESSENTIALS
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> **system_watchdog.sh is started by array_start.sh — not scheduled separately.** > **stability_watchdog.sh is called by watchdog_orchestrator.sh — not scheduled separately.**
> **webgui_watchdog.sh, system_tuning_monitor.sh, clear_logs.sh are scheduled below.** > **webgui_watchdog.sh, system_tuning_monitor.sh, clear_logs.sh are scheduled below.**
> **inotify_tuning.sh, php_fpm_max_children.sh, docker_syslog_filter.sh run via array_start.sh.** > **inotify_tuning.sh, php_fpm_max_children.sh, docker_syslog_filter.sh run via array_start.sh.**
@@ -1264,11 +1264,11 @@ Active containers like Emby and SABnzbd grow fastest. Truncation not rotation
--- ---
### ── system_watchdog_status ────────────────────────────────────────────────── ### ── stability_watchdog_status ──────────────────────────────────────────────────
``` ```
Category: unRAID_Essentials Category: unRAID_Essentials
Name: system_watchdog_status Name: stability_watchdog_status
Schedule: None — run manually Schedule: None — run manually
Background: NO Background: NO
``` ```
@@ -1632,7 +1632,7 @@ Correct startup setup:
Wrong startup setup (old approach): Wrong startup setup (old approach):
ramdisk_setup — At Startup of Array ramdisk_setup — At Startup of Array
docker_syslog_filter — At Startup of Array docker_syslog_filter — At Startup of Array
system_watchdog — At Startup of Array stability_watchdog — Every Minute via watchdog_orchestrator
docker_watchdog — At Startup of Array docker_watchdog — At Startup of Array
failover — At Startup of Array failover — At Startup of Array
(unpredictable order, no coordination) (unpredictable order, no coordination)
@@ -1707,5 +1707,5 @@ critical_sync_maintenance verify Critical-Data sync completes to HOST2
daily_sync_maintenance --dry-run verify remote connectivity and share list daily_sync_maintenance --dry-run verify remote connectivity and share list
failover_status confirm NORMAL state on both servers failover_status confirm NORMAL state on both servers
docker_watchdog_status confirm required containers all running docker_watchdog_status confirm required containers all running
system_watchdog_status confirm no threshold alerts on first check stability_watchdog_status confirm no threshold alerts on first check
``` ```
+10 -10
View File
@@ -448,8 +448,8 @@ Failover coverage:
serves Gmer4Lfe.us via DDNS serves Gmer4Lfe.us via DDNS
Monitoring: Monitoring:
Both servers resource/docker/storage/ resource/docker/storage/ Both servers resource/docker/system/ resource/docker/system/
system_watchdog.sh system_watchdog.sh stability_watchdog.sh stability_watchdog.sh
failover.sh failover.sh failover.sh failover.sh
Sunday morning coffee report Sunday morning coffee report Sunday morning coffee report Sunday morning coffee report
``` ```
@@ -476,7 +476,7 @@ Unraid_Scripts/
├── Failover/ ← Mutual automatic failover — continuous background process ├── Failover/ ← Mutual automatic failover — continuous background process
│ README: README-Failover.md │ README: README-Failover.md
├── Watchdogs/ ← All watchdog scripts: resource, docker, storage, system ├── Watchdogs/ ← All watchdog scripts: resource, docker, system, stability
│ README: README-Watchdogs.md │ README: README-Watchdogs.md
├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks ├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks
@@ -548,8 +548,8 @@ Every minute:
watchdog_orchestrator.sh fires each watchdog in sequence watchdog_orchestrator.sh fires each watchdog in sequence
→ resource_watchdog.sh reduce pressure before healing attempts → resource_watchdog.sh reduce pressure before healing attempts
→ docker_watchdog.sh two-tier container healing → docker_watchdog.sh two-tier container healing
→ storage_watchdog.sh pool growth + runaway log detection → system_watchdog.sh system components: storage + WebGUI health
→ system_watchdog.sh last resort — reboots when all else fails → stability_watchdog.sh last resort — reboots when all else fails
Every 3 minutes: Every 3 minutes:
transcode_management.sh cleanup → manager (order non-negotiable) transcode_management.sh cleanup → manager (order non-negotiable)
@@ -593,8 +593,8 @@ What actually happens on a typical day, from the ecosystem's perspective:
Throughout the day (every minute via watchdog_orchestrator.sh): Throughout the day (every minute via watchdog_orchestrator.sh):
resource_watchdog.sh managing: system pressure (throttle/pause/stop) resource_watchdog.sh managing: system pressure (throttle/pause/stop)
docker_watchdog.sh healing: memory leaks, HTTP failures, required containers docker_watchdog.sh healing: memory leaks, HTTP failures, required containers
storage_watchdog.sh watching: appdata growth rate, runaway log files system_watchdog.sh watching: system components (storage growth, WebGUI)
system_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon stability_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
Throughout the day: Throughout the day:
failover.sh watching: remote server, internet connectivity failover.sh watching: remote server, internet connectivity
@@ -639,13 +639,13 @@ Problem: Authelia crashed and won't restart
→ Tools: watchdog_skip_list_manager.sh to clear after fixing → Tools: watchdog_skip_list_manager.sh to clear after fixing
Problem: RAM drops to 3GB Problem: RAM drops to 3GB
→ system_watchdog.sh: stop non-essential containers → stability_watchdog.sh: stop non-essential containers
→ Wait for RAM to recover to 30GB → Wait for RAM to recover to 30GB
→ docker_watchdog.sh defers (reads mem_shutdown_active=true) → docker_watchdog.sh defers (reads mem_shutdown_active=true)
→ If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot → If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot
Problem: Docker daemon hung Problem: Docker daemon hung
→ system_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart → stability_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
→ If still hung: immediate reboot (no strikes needed — daemon can't be managed) → If still hung: immediate reboot (no strikes needed — daemon can't be managed)
Problem: HOST1 loses power Problem: HOST1 loses power
@@ -713,7 +713,7 @@ Things that are different from what you might expect:
# scheduled directly. array_start.sh is the only "At Startup of Array" entry. # scheduled directly. array_start.sh is the only "At Startup of Array" entry.
# 2. Scripts are silent when healthy. # 2. Scripts are silent when healthy.
# system_watchdog.sh runs 288 times per day. It produces zero output on clean # stability_watchdog.sh runs 288 times per day. It produces zero output on clean
# cycles. If you see output, something needs attention. If you see nothing, # cycles. If you see output, something needs attention. If you see nothing,
# the system is healthy. This is by design. # the system is healthy. This is by design.
+1 -1
View File
@@ -439,7 +439,7 @@ bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh
This single entry launches everything defined in ARRAY_START_SCRIPTS from master.conf: This single entry launches everything defined in ARRAY_START_SCRIPTS from master.conf:
inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup, inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup,
docker_network_connect, system_watchdog, docker_watchdog, failover. docker_network_connect, stability_watchdog, docker_watchdog, failover.
### Cron Schedule ### Cron Schedule
+1 -1
View File
@@ -562,7 +562,7 @@ changes to any running process, container, or state file.
### What It Shows ### What It Shows
``` ```
system_watchdog stability_watchdog
Running state, PID, uptime, approximate cycle count Running state, PID, uptime, approximate cycle count
Active strikes, recent restart history Active strikes, recent restart history
Live snapshot: rootfs, RAM, ZFS ARC, load, zombie count, CPU temp Live snapshot: rootfs, RAM, ZFS ARC, load, zombie count, CPU temp
+74 -16
View File
@@ -1,7 +1,7 @@
# ━━━━━ WATCHDOGS — Manual ━━━━━ # ━━━━━ WATCHDOGS — Manual ━━━━━
Configuration reference, operational procedures, and troubleshooting for all four Configuration reference, operational procedures, and troubleshooting for all watchdog
watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`. scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`. For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
--- ---
@@ -11,8 +11,10 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
- [Output Tiers](#output-tiers) - [Output Tiers](#output-tiers)
- [resource_watchdog.sh](#resource_watchdogsh) - [resource_watchdog.sh](#resource_watchdogsh)
- [docker_watchdog.sh](#docker_watchdogsh) - [docker_watchdog.sh](#docker_watchdogsh)
- [storage_watchdog.sh](#storage_watchdogsh)
- [system_watchdog.sh](#system_watchdogsh) - [system_watchdog.sh](#system_watchdogsh)
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
- [stability_watchdog.sh](#stability_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference) - [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting) - [Troubleshooting](#troubleshooting)
@@ -23,7 +25,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
All watchdog scripts use a two-tier output model: `echo` lines are always visible; All watchdog scripts use a two-tier output model: `echo` lines are always visible;
`log` lines only appear when `--log` is passed. `log` lines only appear when `--log` is passed.
All four watchdogs are **single-pass scripts** called once per minute by the orchestrator. All watchdog scripts are **single-pass scripts** called once per minute by the orchestrator.
Without `--log`, only state transitions, warnings, errors, and the conclusion line are Without `--log`, only state transitions, warnings, errors, and the conclusion line are
visible. Per-check detail is suppressed on clean cycles. visible. Per-check detail is suppressed on clean cycles.
@@ -285,9 +287,26 @@ docker start ContainerName
--- ---
## storage_watchdog.sh ## system_watchdog.sh
Runs third in the orchestrator sequence. Two independent checks per cycle: Thin orchestrator — runs `SYSTEM_WATCHDOG_SCRIPTS` from master.conf sequentially each cycle.
Called third by `watchdog_orchestrator.sh`. Covers all system component watchdogs.
Can also be run standalone to check all system components at once.
### Usage
```bash
system_watchdog.sh # run all system component watchdogs
system_watchdog.sh --status # show configured scripts and their paths
system_watchdog.sh --dry-run # preview without executing anything
system_watchdog.sh --log # verbose output
```
---
## System/storage_watchdog.sh
Called by `system_watchdog.sh` each cycle. Two independent checks per cycle:
growth rate detection (automatic, zero config) and oversize log detection. growth rate detection (automatic, zero config) and oversize log detection.
Uses its own strike state file — independent from docker_watchdog. Uses its own strike state file — independent from docker_watchdog.
@@ -343,7 +362,7 @@ Log strikes auto-clear when the file drops below threshold.
### Usage ### Usage
```bash ```bash
storage_watchdog.sh # single pass (called by watchdog_orchestrator.sh) storage_watchdog.sh # single pass (called by system_watchdog.sh)
storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings
storage_watchdog.sh --dry-run # show what would be alerted/truncated storage_watchdog.sh --dry-run # show what would be alerted/truncated
storage_watchdog.sh --log # verbose per-container output storage_watchdog.sh --log # verbose per-container output
@@ -351,7 +370,46 @@ storage_watchdog.sh --log # verbose per-container output
--- ---
## system_watchdog.sh ## System/webgui_watchdog.sh
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
through three restart steps if unresponsive. Silent when healthy.
### Escalation Path
```
WebGUI responding → exit 0 (silent)
Not responding:
Step 1 — nginx restart → wait WEBGUI_NGINX_WAIT → recheck
Step 2 — php-fpm restart → wait WEBGUI_PHP_WAIT → recheck
Step 3 — emhttp restart → wait WEBGUI_EMHTTP_WAIT → recheck
All three failed → critical notify, manual intervention needed
```
### Configuration
```bash
# master.conf
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30
```
### Usage
```bash
webgui_watchdog.sh # single pass (called by system_watchdog.sh)
webgui_watchdog.sh --status # current WebGUI state + nginx/php-fpm/emhttp status
webgui_watchdog.sh --dry-run # show which services would be restarted
webgui_watchdog.sh --log # verbose per-step output
```
---
## stability_watchdog.sh
Runs last in the orchestrator sequence. The only script in the ecosystem authorized Runs last in the orchestrator sequence. The only script in the ecosystem authorized
to reboot. Watches the server itself — not containers, not storage. Reboots only when to reboot. Watches the server itself — not containers, not storage. Reboots only when
@@ -425,10 +483,10 @@ reboots → something crashes again immediately → reboot again.
### Usage ### Usage
```bash ```bash
system_watchdog.sh # single pass (called by watchdog_orchestrator.sh) stability_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
system_watchdog.sh --dry-run # run detection logic without rebooting stability_watchdog.sh --dry-run # run detection logic without rebooting
system_watchdog.sh --status # thresholds, current state, strike counts stability_watchdog.sh --status # thresholds, current state, strike counts
system_watchdog.sh --log # verbose per-check output stability_watchdog.sh --log # verbose per-check output
``` ```
--- ---
@@ -616,7 +674,7 @@ storage_watchdog.sh --status
# ) # )
``` ```
### system_watchdog Rebooted Unexpectedly ### stability_watchdog Rebooted Unexpectedly
```bash ```bash
# Check the reboot log (survives reboots): # Check the reboot log (survives reboots):
@@ -624,13 +682,13 @@ cat /boot/config/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot # Shows timestamp and reason for each watchdog-triggered reboot
# Check syslog near the reboot time: # Check syslog near the reboot time:
grep "system_watchdog" /var/log/syslog | tail -20 grep "stability_watchdog" /var/log/syslog | tail -20
``` ```
### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout ### stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
```bash ```bash
# All four watchdogs run as single-pass scripts — there is no background process to check. # All watchdogs run as single-pass scripts — there is no background process to check.
# If the orchestrator reports a timeout, one pass took longer than expected. # If the orchestrator reports a timeout, one pass took longer than expected.
# Check the orchestrator itself: # Check the orchestrator itself:
+46 -31
View File
@@ -1,9 +1,10 @@
# ━━━━━ WATCHDOGS ━━━━━ # ━━━━━ WATCHDOGS ━━━━━
**Four single-pass scripts that run every minute through `watchdog_orchestrator.sh`, **Four tiers that run every minute through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → protect storage each with a clear lane:** reduce system pressure → heal containers → check system components
reboot if nothing else worked. They never run standalone loops. The orchestrator calls reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
them in order, once per cron cycle. System component checks (storage, WebGUI) run inside a thin `system_watchdog.sh` orchestrator
that can also be run standalone.
--- ---
@@ -52,7 +53,7 @@ Rootfs fills. SSH stops accepting new connections. Docker can't write log files.
files fail silently. The server is functionally dead but still technically running. files fail silently. The server is functionally dead but still technically running.
Nothing in the container layer can detect or recover from this — it requires a reboot. Nothing in the container layer can detect or recover from this — it requires a reboot.
Fix: `system_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
health. Only script in the stack authorized to reboot. Runs last in the orchestrator health. Only script in the stack authorized to reboot. Runs last in the orchestrator
sequence so container healing and pressure reduction always get a chance first. sequence so container healing and pressure reduction always get a chance first.
@@ -60,24 +61,26 @@ sequence so container healing and pressure reduction always get a chance first.
## ━━━ WHAT THIS FOLDER DOES ━━━ ## ━━━ WHAT THIS FOLDER DOES ━━━
Four watchdogs. One purpose each. Fixed execution order via `watchdog_orchestrator.sh`. Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
``` ```
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
Storage protection storage_watchdog.sh — pool growth rate + runaway log detection System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
Last resort system_watchdog.sh reboot only when nothing else can recover └─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
Last resort stability_watchdog.sh — reboot only when nothing else can recover
``` ```
**The execution order is the design.** Resource pressure is reduced before docker_watchdog **The execution order is the design.** Resource pressure is reduced before docker_watchdog
attempts restarts — containers restarted into a pressure-bound system just fail again. attempts restarts — containers restarted into a pressure-bound system just fail again.
Storage is checked after containers are healed — no false alerts from containers that System component checks run after containers are healed. Stability watchdog runs last —
were already being restarted. System watchdog runs last — reboot is always the last reboot is always the last option, not the first.
option, not the first.
**None of these scripts run standalone loops.** Each is a single-pass script called **None of these scripts run standalone loops.** Each is a single-pass script called
once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles
startup grace, overlap protection, heartbeat, and sequencing. startup grace, overlap protection, heartbeat, and sequencing. `system_watchdog.sh` can
also be run standalone to check all system component watchdogs at once.
--- ---
@@ -85,10 +88,14 @@ startup grace, overlap protection, heartbeat, and sequencing.
``` ```
Orchestrators/ Orchestrators/
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute) watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
──────────────────► docker_watchdog.sh (2nd) ──────────────────► docker_watchdog.sh (2nd)
──────────────────► storage_watchdog.sh (3rd) ──────────────────► system_watchdog.sh (3rd — thin orchestrator)
──────────────────► system_watchdog.sh (4th — last resort) ──────────────────► stability_watchdog.sh (4th — last resort)
Watchdogs/System/ ← called by system_watchdog.sh each cycle
storage_watchdog.sh pool growth rate + runaway log detection
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
Tools/ Tools/
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
@@ -97,10 +104,6 @@ Tools/
Docker_Essentials/ Docker_Essentials/
All container lifecycle scripts (daily restart, updates, network) — unaffected. All container lifecycle scripts (daily restart, updates, network) — unaffected.
docker_watchdog.sh coordinates with them via shared state, not direct calls. docker_watchdog.sh coordinates with them via shared state, not direct calls.
unRAID_Essentials/
Server-level scripts (WebGUI restart, inotify tuning, log hygiene) — unaffected.
system_watchdog.sh runs in the same ecosystem but is independent of those scripts.
``` ```
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog. **watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
@@ -114,8 +117,10 @@ unRAID_Essentials/
|--------|------|-----------| |--------|------|-----------|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute | | `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute |
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute | | `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute |
| `storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `watchdog_orchestrator.sh` — 3rd every minute | | `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every minute |
| `system_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute | | `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute |
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute |
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh` > `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system. > is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
@@ -149,13 +154,23 @@ Every minute — watchdog_orchestrator.sh fires:
│ (skip list management) → Tools/watchdog_skip_list_manager.sh │ (skip list management) → Tools/watchdog_skip_list_manager.sh
Step 3 — storage_watchdog.sh Step 3 — system_watchdog.sh (thin orchestrator)
growth rate scan: du -sm appdata/* → compare to previous cycle baseline runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially:
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
Step 4 — system_watchdog.sh │ └─ storage_watchdog.sh
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
│ └─ webgui_watchdog.sh
│ curl check → WebGUI responding → exit 0 (silent)
│ not responding → nginx restart → wait → recheck
│ still down → php-fpm restart → wait → recheck
│ still down → emhttp restart → wait → recheck
│ all three failed → critical notify, manual intervention needed
Step 4 — stability_watchdog.sh
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
Tier 1 CRITICAL → immediate reboot (no strikes) Tier 1 CRITICAL → immediate reboot (no strikes)
Tier 2 URGENT → reboot if OOM confirmed Tier 2 URGENT → reboot if OOM confirmed
@@ -169,8 +184,8 @@ Every minute — watchdog_orchestrator.sh fires:
| State File | Written By | Read By | Purpose | | State File | Written By | Read By | Purpose |
|-----------|-----------|---------|---------| |-----------|-----------|---------|---------|
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency | | `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
| `SYS_WATCHDOG_STATE_FILE` | `system_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) | | `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container | | `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list | | `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `STORAGE_WATCHDOG_STATE_FILE` | `storage_watchdog.sh` | itself | Growth + log strike counts | | `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `storage_watchdog.sh` | itself | Per-container size baseline for growth rate | | `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
@@ -4,7 +4,7 @@ Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start. adding or reordering scripts at array start.
> **Watchdog scripts have moved.** `system_watchdog.sh` and `resource_watchdog.sh` > **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
> now live in `Watchdogs/`. Their configuration reference and troubleshooting > now live in `Watchdogs/`. Their configuration reference and troubleshooting
> procedures are in `Watchdogs/Manual-Watchdogs.md`. > procedures are in `Watchdogs/Manual-Watchdogs.md`.
@@ -452,8 +452,8 @@ server_reboot.sh --reason="disk work" # include reason in notification
## Full Configuration Reference ## Full Configuration Reference
> Watchdog configuration (`system_watchdog.sh`, `resource_watchdog.sh`, > Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
> `docker_watchdog.sh`, `storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`. > `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
```bash ```bash
# master.conf # master.conf
@@ -494,8 +494,8 @@ REBOOT_VM_WAIT=30
## Troubleshooting ## Troubleshooting
> Watchdog troubleshooting (system_watchdog, resource_watchdog, docker_watchdog, > Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
> storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`. > System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
### rsync_stop Killed the Wrong Thing ### rsync_stop Killed the Wrong Thing
@@ -16,7 +16,7 @@ The restart hangs. The server needs a reboot, but nothing in the ecosystem is
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
write log files. The server is functionally dead but still technically running. write log files. The server is functionally dead but still technically running.
Fix: `system_watchdog.sh` — three-tier response: immediate reboot on critical Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
failures, OOM-confirmed bypass for RAM crises, strike system for sustained failures, OOM-confirmed bypass for RAM crises, strike system for sustained
threshold breaches. Last line of defense before a hard crash. threshold breaches. Last line of defense before a hard crash.
@@ -82,7 +82,7 @@ Graceful operations mover_stop.sh — clean mover stop
server_reboot.sh — clean reboot with pre-flight warnings server_reboot.sh — clean reboot with pre-flight warnings
``` ```
> `system_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`. > `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite. > See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
--- ---
@@ -99,9 +99,9 @@ Orchestrators/
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally) server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
Watchdogs/ Watchdogs/
system_watchdog.sh and resource_watchdog.sh now live here. stability_watchdog.sh and resource_watchdog.sh now live here.
See Watchdogs/README-Watchdogs.md for how they relate to each other See Watchdogs/README-Watchdogs.md for how they relate to each other
and to docker_watchdog.sh and storage_watchdog.sh. and to docker_watchdog.sh and System/storage_watchdog.sh.
``` ```
--- ---