FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Watchdogs/ docs: new README + Manual, update affected folders

Browse Source 
New docs:
  Watchdogs/README-Watchdogs.md  — design, relationships, script table, state file map
  Watchdogs/Manual-Watchdogs.md  — full config reference for all 4 watchdogs

Docker_Essentials/:
  README — remove docker_watchdog, update folder description and diagrams
  Manual  — strip watchdog config sections, add pointer to Watchdogs/Manual

unRAID_Essentials/:
  README — remove system/resource watchdog, update diagrams and script table
  Manual  — strip system/resource watchdog sections, update TOC + config reference

README.md:
  Add Watchdogs/ to folder structure
  Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start
  Fix daily cycle and monitoring diagrams
This commit is contained in:
Gmer4Lfe
2026-05-22 17:26:21 -04:00
parent ec79a926e8
commit 8a3e22c9b8
7 changed files with 912 additions and 707 deletions
Download Patch File Download Diff File Expand all files Collapse all files
unRAID_Essentials/README-Unraid_Essentials.md
+14 -32
View File
@@ -71,8 +71,6 @@ SIGTERM (graceful — finishes current file), SIGKILL only if needed.
## ━━━ WHAT THIS FOLDER DOES ━━━
```
Last-resort stability system_watchdog.sh — reboots before crash
Pressure reduction resource_watchdog.sh — throttle/pause/stop under load
WebGUI availability webgui_restart.sh — nginx → php-fpm → emhttp escalation
Kernel tuning inotify_tuning.sh — file watch limits
php_fpm_max_children.sh — PHP worker count
@@ -84,6 +82,9 @@ Graceful operations mover_stop.sh — clean mover stop
server_reboot.sh — clean reboot with pre-flight warnings
```
> `system_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
---
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
@@ -93,33 +94,22 @@ Orchestrators/
array_start.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
─────────────────────────► docker_syslog_filter.sh (second)
─────────────────────────► php_fpm_max_children.sh
─────────────────────────► system_watchdog.sh (background loop)
watchdog_orchestrator.sh ───────────────► resource_watchdog.sh (every minute)
weekly_maintenance.sh ──────────────────► clear_logs.sh
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
Docker_Essentials/
docker_watchdog.sh ◄─── reads ──────────── resource_watchdog.sh state
(mem_shutdown_active flag)
Watchdogs/
system_watchdog.sh and resource_watchdog.sh now live here.
See Watchdogs/README-Watchdogs.md for how they relate to each other
and to docker_watchdog.sh and storage_watchdog.sh.
```
`system_watchdog.sh` and `docker_watchdog.sh` (in Docker_Essentials/) are
designed to work together — docker_watchdog heals containers first,
system_watchdog reboots only when healing has failed. `resource_watchdog.sh`
coordinates with docker_watchdog via the `mem_shutdown_active` state flag to
prevent docker_watchdog from restarting containers that resource_watchdog just
stopped to free RAM.
---
## ━━━ SCRIPTS IN THIS FOLDER ━━━
| Script | Role | When It Runs |
|--------|------|-------------|
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop via array_start.sh |
| `resource_watchdog.sh` | Pressure reduction — throttle/pause/stop under load | Every minute via watchdog_orchestrator.sh |
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
@@ -137,20 +127,14 @@ stopped to free RAM.
```
Array starts
│
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
├─ php_fpm_max_children.sh ← before WebGUI is under load
└─ system_watchdog.sh ← starts background loop
Every minute (watchdog_orchestrator.sh):
└─ resource_watchdog.sh
Level 1 (soft): throttle SABnzbd + qBit download speeds
Level 2 (medium): further throttle + docker pause non-essential containers
Level 3 (hard): docker stop optional services + set mem_shutdown_active=true
↓
docker_watchdog.sh reads mem_shutdown_active — defers restarts
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
└─ php_fpm_max_children.sh ← before WebGUI is under load
Every minute (watchdog_orchestrator.sh in Orchestrators/):
→ Watchdogs/resource_watchdog.sh → Watchdogs/docker_watchdog.sh
→ Watchdogs/storage_watchdog.sh → Watchdogs/system_watchdog.sh
(see Watchdogs/README-Watchdogs.md for full flow)
Every 10 minutes (User Scripts):
└─ webgui_restart.sh
@@ -161,13 +145,11 @@ Every 10 minutes (User Scripts):
Step 3: restart emhttp → recheck
All failed → notify, exit 1
Weekly (weekly_maintenance.sh):
└─ clear_logs.sh
System logs: clear if > LOG_MIN_SIZE_MB
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
Manual operations:
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)