Update all docs to reflect watchdog restructure
README-Watchdogs.md: new tier diagram, updated script table, execution flow, state file table, and folder relationship diagram Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog, System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting README.md: updated schedule listing, folder description, daily cycle examples README-User_Script_Plug-in.md: stability_watchdog references throughout unRAID_Essentials/README + Manual: stability_watchdog, updated storage path Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
This commit is contained in:
@@ -448,8 +448,8 @@ Failover coverage:
|
||||
serves Gmer4Lfe.us via DDNS
|
||||
|
||||
Monitoring:
|
||||
Both servers resource/docker/storage/ resource/docker/storage/
|
||||
system_watchdog.sh system_watchdog.sh
|
||||
Both servers resource/docker/system/ resource/docker/system/
|
||||
stability_watchdog.sh stability_watchdog.sh
|
||||
failover.sh failover.sh
|
||||
Sunday morning coffee report Sunday morning coffee report
|
||||
```
|
||||
@@ -476,7 +476,7 @@ Unraid_Scripts/
|
||||
├── Failover/ ← Mutual automatic failover — continuous background process
|
||||
│ README: README-Failover.md
|
||||
│
|
||||
├── Watchdogs/ ← All watchdog scripts: resource, docker, storage, system
|
||||
├── Watchdogs/ ← All watchdog scripts: resource, docker, system, stability
|
||||
│ README: README-Watchdogs.md
|
||||
│
|
||||
├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks
|
||||
@@ -548,8 +548,8 @@ Every minute:
|
||||
watchdog_orchestrator.sh fires each watchdog in sequence
|
||||
→ resource_watchdog.sh reduce pressure before healing attempts
|
||||
→ docker_watchdog.sh two-tier container healing
|
||||
→ storage_watchdog.sh pool growth + runaway log detection
|
||||
→ system_watchdog.sh last resort — reboots when all else fails
|
||||
→ system_watchdog.sh system components: storage + WebGUI health
|
||||
→ stability_watchdog.sh last resort — reboots when all else fails
|
||||
|
||||
Every 3 minutes:
|
||||
transcode_management.sh cleanup → manager (order non-negotiable)
|
||||
@@ -593,8 +593,8 @@ What actually happens on a typical day, from the ecosystem's perspective:
|
||||
Throughout the day (every minute via watchdog_orchestrator.sh):
|
||||
resource_watchdog.sh managing: system pressure (throttle/pause/stop)
|
||||
docker_watchdog.sh healing: memory leaks, HTTP failures, required containers
|
||||
storage_watchdog.sh watching: appdata growth rate, runaway log files
|
||||
system_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
|
||||
system_watchdog.sh watching: system components (storage growth, WebGUI)
|
||||
stability_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
|
||||
|
||||
Throughout the day:
|
||||
failover.sh watching: remote server, internet connectivity
|
||||
@@ -639,13 +639,13 @@ Problem: Authelia crashed and won't restart
|
||||
→ Tools: watchdog_skip_list_manager.sh to clear after fixing
|
||||
|
||||
Problem: RAM drops to 3GB
|
||||
→ system_watchdog.sh: stop non-essential containers
|
||||
→ stability_watchdog.sh: stop non-essential containers
|
||||
→ Wait for RAM to recover to 30GB
|
||||
→ docker_watchdog.sh defers (reads mem_shutdown_active=true)
|
||||
→ If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot
|
||||
|
||||
Problem: Docker daemon hung
|
||||
→ system_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
|
||||
→ stability_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
|
||||
→ If still hung: immediate reboot (no strikes needed — daemon can't be managed)
|
||||
|
||||
Problem: HOST1 loses power
|
||||
@@ -713,7 +713,7 @@ Things that are different from what you might expect:
|
||||
# scheduled directly. array_start.sh is the only "At Startup of Array" entry.
|
||||
|
||||
# 2. Scripts are silent when healthy.
|
||||
# system_watchdog.sh runs 288 times per day. It produces zero output on clean
|
||||
# stability_watchdog.sh runs 288 times per day. It produces zero output on clean
|
||||
# cycles. If you see output, something needs attention. If you see nothing,
|
||||
# the system is healthy. This is by design.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user