Update all docs to reflect watchdog restructure

README-Watchdogs.md: new tier diagram, updated script table, execution flow,
  state file table, and folder relationship diagram
Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog,
  System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting
README.md: updated schedule listing, folder description, daily cycle examples
README-User_Script_Plug-in.md: stability_watchdog references throughout
unRAID_Essentials/README + Manual: stability_watchdog, updated storage path
Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
This commit is contained in:
Gmer4Lfe
2026-05-22 20:26:36 -04:00
parent cf21efaeea
commit 13ce8497f3
9 changed files with 150 additions and 77 deletions
+10 -10
View File
@@ -448,8 +448,8 @@ Failover coverage:
serves Gmer4Lfe.us via DDNS
Monitoring:
Both servers resource/docker/storage/ resource/docker/storage/
system_watchdog.sh system_watchdog.sh
Both servers resource/docker/system/ resource/docker/system/
stability_watchdog.sh stability_watchdog.sh
failover.sh failover.sh
Sunday morning coffee report Sunday morning coffee report
```
@@ -476,7 +476,7 @@ Unraid_Scripts/
├── Failover/ ← Mutual automatic failover — continuous background process
│ README: README-Failover.md
├── Watchdogs/ ← All watchdog scripts: resource, docker, storage, system
├── Watchdogs/ ← All watchdog scripts: resource, docker, system, stability
│ README: README-Watchdogs.md
├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks
@@ -548,8 +548,8 @@ Every minute:
watchdog_orchestrator.sh fires each watchdog in sequence
→ resource_watchdog.sh reduce pressure before healing attempts
→ docker_watchdog.sh two-tier container healing
→ storage_watchdog.sh pool growth + runaway log detection
→ system_watchdog.sh last resort — reboots when all else fails
→ system_watchdog.sh system components: storage + WebGUI health
→ stability_watchdog.sh last resort — reboots when all else fails
Every 3 minutes:
transcode_management.sh cleanup → manager (order non-negotiable)
@@ -593,8 +593,8 @@ What actually happens on a typical day, from the ecosystem's perspective:
Throughout the day (every minute via watchdog_orchestrator.sh):
resource_watchdog.sh managing: system pressure (throttle/pause/stop)
docker_watchdog.sh healing: memory leaks, HTTP failures, required containers
storage_watchdog.sh watching: appdata growth rate, runaway log files
system_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
system_watchdog.sh watching: system components (storage growth, WebGUI)
stability_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
Throughout the day:
failover.sh watching: remote server, internet connectivity
@@ -639,13 +639,13 @@ Problem: Authelia crashed and won't restart
→ Tools: watchdog_skip_list_manager.sh to clear after fixing
Problem: RAM drops to 3GB
→ system_watchdog.sh: stop non-essential containers
→ stability_watchdog.sh: stop non-essential containers
→ Wait for RAM to recover to 30GB
→ docker_watchdog.sh defers (reads mem_shutdown_active=true)
→ If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot
Problem: Docker daemon hung
→ system_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
→ stability_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
→ If still hung: immediate reboot (no strikes needed — daemon can't be managed)
Problem: HOST1 loses power
@@ -713,7 +713,7 @@ Things that are different from what you might expect:
# scheduled directly. array_start.sh is the only "At Startup of Array" entry.
# 2. Scripts are silent when healthy.
# system_watchdog.sh runs 288 times per day. It produces zero output on clean
# stability_watchdog.sh runs 288 times per day. It produces zero output on clean
# cycles. If you see output, something needs attention. If you see nothing,
# the system is healthy. This is by design.