FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Watchdogs/ docs: new README + Manual, update affected folders

Browse Source 
New docs:
  Watchdogs/README-Watchdogs.md  — design, relationships, script table, state file map
  Watchdogs/Manual-Watchdogs.md  — full config reference for all 4 watchdogs

Docker_Essentials/:
  README — remove docker_watchdog, update folder description and diagrams
  Manual  — strip watchdog config sections, add pointer to Watchdogs/Manual

unRAID_Essentials/:
  README — remove system/resource watchdog, update diagrams and script table
  Manual  — strip system/resource watchdog sections, update TOC + config reference

README.md:
  Add Watchdogs/ to folder structure
  Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start
  Fix daily cycle and monitoring diagrams
This commit is contained in:
Gmer4Lfe
2026-05-22 17:26:21 -04:00
parent ec79a926e8
commit 8a3e22c9b8
7 changed files with 912 additions and 707 deletions
Download Patch File Download Diff File Expand all files Collapse all files
unRAID_Essentials/Manual-Unraid_Essentials.md
+14 -254
View File
@@ -4,13 +4,15 @@ Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start.
> **Watchdog scripts have moved.** `system_watchdog.sh` and `resource_watchdog.sh`
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
---
## ━━━ CONTENTS ━━━
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
- [system_watchdog.sh](#system_watchdogsh)
- [resource_watchdog.sh](#resource_watchdogsh)
- [webgui_restart.sh](#webgui_restartsh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
@@ -30,9 +32,9 @@ adding or reordering scripts at array start.
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
**Daemon scripts** (`system_watchdog.sh`, `resource_watchdog.sh`, `webgui_restart.sh`):
run on every cycle. Without `--log`, only state transitions, warnings, errors, and
the clean-cycle conclusion line are visible. Per-check detail suppressed.
**Daemon scripts** (`webgui_restart.sh`): run on every cycle. Without `--log`, only
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
Per-check detail suppressed.
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
@@ -59,8 +61,9 @@ ARRAY_START_SCRIPTS=(
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
...
"system_watchdog.sh" # LAST or near-last — starts background loop
)
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every minute via
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
```
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
@@ -73,151 +76,6 @@ generates veth messages — these will appear in syslog if the filter isn't acti
---
## system_watchdog.sh
### Three-Tier Response
The watchdog categorizes every failure into one of three tiers:
**Tier 1 — CRITICAL (immediate reboot, no strikes)**
| Condition | Threshold | Why immediate |
|-----------|-----------|---------------|
| Docker daemon unresponsive | N/A | Nothing can be healed; every docker command hangs |
| rootfs usage | SYS_WATCHDOG_ROOTFS_CRITICAL_PCT (99%) | SSH stops; state files fail silently |
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
| File descriptor exhaustion | SYS_WATCHDOG_FD_CRITICAL_PCT (95%) | New connections silently failing |
| /boot read-only unexpectedly | write test fails | Config writes silently failing |
**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
RAM below MEM_GB AND OOM kills this cycle >= SYS_WATCHDOG_OOM_LIMIT.
Both conditions required — RAM alone without OOM uses the standard strike system.
OOM confirms the system is dying faster than watchdogs can heal.
**Tier 3 — STANDARD (SYS_WATCHDOG_STRIKES consecutive failures → reboot)**
| Check | Threshold |
|-------|-----------|
| Free RAM | MEM_WARN_GB → MEM_SHUTDOWN_GB → MEM_GB |
| Load average | SYS_WATCHDOG_LOAD_MULTIPLIER × cpu_count |
| CPU temperature | SYS_WATCHDOG_CPU_TEMP |
| Zombie processes | SYS_WATCHDOG_ZOMBIES |
| /var/log usage | SYS_WATCHDOG_VAR_LOG_PCT |
| /tmp usage | SYS_WATCHDOG_TMP_PCT |
| Array disk errors | mdstat error delta > 0 |
| NIC state | interface operstate != "up" |
| Required containers | containers in SYS_WATCHDOG_REQUIRED_CONTAINERS |
### RAM Tiers
```
MEM_WARN_GB (10GB) → warn + notify, no action
MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery
MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM)
MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart
```
At MEM_SHUTDOWN_GB, all containers NOT listed in
`SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED` are stopped. Excluded containers by default:
NginxProxyManager, Authelia, Mariadb, Redis, Emby, Dispatcharr. Adjust in
master.conf for your critical services.
### Abort Conditions
These conditions prevent a reboot — running them would cause data loss:
```bash
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
SYS_WATCHDOG_ABORT_ON_PARITY=true # Parity check/rebuild running
SYS_WATCHDOG_ABORT_ON_MOVER=true # Mover running
```
CRITICAL tier bypasses all abort conditions — an imminent crash outweighs
data safety concerns.
### Strike System
The strike count tracks consecutive failures. A single recovery resets strikes
to 0. The reboot only fires after SYS_WATCHDOG_STRIKES consecutive failures on
the same check — transient spikes (a brief load burst, a momentary RAM dip) don't
trigger reboots.
### Reboot Rate Limit
```bash
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # window in hours
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots within the window
```
If the server has rebooted SYS_WATCHDOG_MAX_REBOOTS times within the window,
system_watchdog stops rebooting and notifies instead. This prevents a boot loop
where the watchdog reboots → something crashes again immediately → reboot again.
### Usage
```bash
system_watchdog.sh # start continuous monitoring loop
system_watchdog.sh --dry-run # run detection logic without rebooting
system_watchdog.sh --status # show thresholds, current state, strike counts
system_watchdog.sh --log # verbose per-cycle output
```
### Verify Running
```bash
pgrep -a -f system_watchdog.sh
# Expected: shows PID and path
```
---
## resource_watchdog.sh
### Pressure Levels
```bash
# master.conf
RW_RAM_SOFT_GB=20 # Level 1 trigger — throttle downloaders
RW_RAM_MEDIUM_GB=15 # Level 2 trigger — throttle + pause containers
RW_RAM_HARD_GB=10 # Level 3 trigger — stop containers
RW_RAM_RECOVER_GB=25 # recover to this before un-stopping at level 3
RW_LOAD_SOFT_MULTIPLIER=2.0 # load > 2× cpu count = level 1
RW_LOAD_MEDIUM_MULTIPLIER=3.0 # load > 3× cpu count = level 2
RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalating
```
### Per-Host Container Lists
```bash
# host1.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at medium pressure
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at hard pressure
```
Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
pressure level. Default includes: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
### docker_watchdog Coordination
At level 3, resource_watchdog writes `mem_shutdown_active=true` to `RW_STATE_FILE`.
docker_watchdog.sh reads this flag each cycle and skips all container restart logic
while it is set. Without this coordination, docker_watchdog would immediately
restart containers that resource_watchdog just stopped to free RAM.
The flag is cleared when level 3 pressure resolves and containers are restarted.
### Usage
```bash
resource_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
resource_watchdog.sh --dry-run # show what would be throttled/paused/stopped
resource_watchdog.sh --status # current level, active actions, recovery cycle count
resource_watchdog.sh --log # verbose per-check output
```
---
## webgui_restart.sh
### Escalation Logic
@@ -594,75 +452,12 @@ server_reboot.sh --reason="disk work" # include reason in notification
## Full Configuration Reference
> Watchdog configuration (`system_watchdog.sh`, `resource_watchdog.sh`,
> `docker_watchdog.sh`, `storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
```bash
# master.conf
# ── System Watchdog ────────────────────────────────────────────────────────────
SYS_WATCHDOG_INTERVAL=300 # seconds between check cycles
SYS_WATCHDOG_STRIKES=3 # consecutive failures before reboot
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # rate limit window
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots in window
SYS_WATCHDOG_OOM_LIMIT=3 # OOM kills/cycle for URGENT bypass
MEM_WARN_GB=10 # warn + notify
MEM_SHUTDOWN_GB=6 # stop non-essential containers
MEM_GB=4 # strike → reboot
MEM_RECOVER_GB=30 # recovery threshold
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99 # Tier 1 trigger
SYS_WATCHDOG_FD_CRITICAL_PCT=95 # Tier 1 trigger
SYS_WATCHDOG_LOAD_MULTIPLIER=4 # Tier 3 — ×cpu_count
SYS_WATCHDOG_CPU_TEMP=85 # Tier 3 — Celsius
SYS_WATCHDOG_ZOMBIES=20 # Tier 3 — process count
SYS_WATCHDOG_VAR_LOG_PCT=80 # Tier 3 — percent full
SYS_WATCHDOG_TMP_PCT=85 # Tier 3 — percent full
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=true
SYS_WATCHDOG_ABORT_ON_MOVER=true
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
"NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
)
SYS_WATCHDOG_REQUIRED_CONTAINERS=() # containers that must be running
# State files:
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
SYS_WATCHDOG_REBOOT_LOG="/tmp/sys_watchdog_reboots.db"
SYS_WATCHDOG_FAILED_FILE="/tmp/sys_watchdog_failed.db"
SYS_WATCHDOG_OOM_FILE="/tmp/sys_watchdog_oom.db"
# ── Resource Watchdog ──────────────────────────────────────────────────────────
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
RW_RAM_SOFT_GB=20
RW_RAM_MEDIUM_GB=15
RW_RAM_HARD_GB=10
RW_RAM_RECOVER_GB=25
RW_LOAD_SOFT_MULTIPLIER=2.0
RW_LOAD_MEDIUM_MULTIPLIER=3.0
RW_RECOVER_CYCLES=3
RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"
RW_SABNZBD_SPEED_MEDIUM="10M"
RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200 # KB/s
RW_QBIT_DL_MEDIUM=10240
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
# ── Per-Host (host*.conf) ───────────────────────────────────────────────
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
@@ -699,43 +494,8 @@ REBOOT_VM_WAIT=30
## Troubleshooting
### system_watchdog.sh Not Starting
```bash
# Check if already running (acquire_lock prevents second instance):
pgrep -a -f system_watchdog.sh
# Check state file permissions:
ls -la /tmp/sys_watchdog_*.db
# Run with --log to see startup:
system_watchdog.sh --log
```
### system_watchdog Rebooted Unexpectedly
```bash
# Check reboot log:
cat /tmp/sys_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot
# Check what condition triggered it:
# Look in /var/log/syslog for "system_watchdog" near the reboot time
grep "system_watchdog" /var/log/syslog | tail -20
```
### resource_watchdog Paused Containers It Shouldn't Have
```bash
# Check current state:
resource_watchdog.sh --status
# Add the container to RW_CRITICAL_CONTAINERS in master.conf:
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
# Un-pause manually if needed:
docker unpause MyContainer
```
> Watchdog troubleshooting (system_watchdog, resource_watchdog, docker_watchdog,
> storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
### rsync_stop Killed the Wrong Thing
unRAID_Essentials/README-Unraid_Essentials.md
+14 -32
View File
@@ -71,8 +71,6 @@ SIGTERM (graceful — finishes current file), SIGKILL only if needed.
## ━━━ WHAT THIS FOLDER DOES ━━━
```
Last-resort stability system_watchdog.sh — reboots before crash
Pressure reduction resource_watchdog.sh — throttle/pause/stop under load
WebGUI availability webgui_restart.sh — nginx → php-fpm → emhttp escalation
Kernel tuning inotify_tuning.sh — file watch limits
php_fpm_max_children.sh — PHP worker count
@@ -84,6 +82,9 @@ Graceful operations mover_stop.sh — clean mover stop
server_reboot.sh — clean reboot with pre-flight warnings
```
> `system_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
---
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
@@ -93,33 +94,22 @@ Orchestrators/
array_start.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
─────────────────────────► docker_syslog_filter.sh (second)
─────────────────────────► php_fpm_max_children.sh
─────────────────────────► system_watchdog.sh (background loop)
watchdog_orchestrator.sh ───────────────► resource_watchdog.sh (every minute)
weekly_maintenance.sh ──────────────────► clear_logs.sh
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
Docker_Essentials/
docker_watchdog.sh ◄─── reads ──────────── resource_watchdog.sh state
(mem_shutdown_active flag)
Watchdogs/
system_watchdog.sh and resource_watchdog.sh now live here.
See Watchdogs/README-Watchdogs.md for how they relate to each other
and to docker_watchdog.sh and storage_watchdog.sh.
```
`system_watchdog.sh` and `docker_watchdog.sh` (in Docker_Essentials/) are
designed to work together — docker_watchdog heals containers first,
system_watchdog reboots only when healing has failed. `resource_watchdog.sh`
coordinates with docker_watchdog via the `mem_shutdown_active` state flag to
prevent docker_watchdog from restarting containers that resource_watchdog just
stopped to free RAM.
---
## ━━━ SCRIPTS IN THIS FOLDER ━━━
| Script | Role | When It Runs |
|--------|------|-------------|
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop via array_start.sh |
| `resource_watchdog.sh` | Pressure reduction — throttle/pause/stop under load | Every minute via watchdog_orchestrator.sh |
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
@@ -137,20 +127,14 @@ stopped to free RAM.
```
Array starts
│
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
├─ php_fpm_max_children.sh ← before WebGUI is under load
└─ system_watchdog.sh ← starts background loop
Every minute (watchdog_orchestrator.sh):
└─ resource_watchdog.sh
Level 1 (soft): throttle SABnzbd + qBit download speeds
Level 2 (medium): further throttle + docker pause non-essential containers
Level 3 (hard): docker stop optional services + set mem_shutdown_active=true
↓
docker_watchdog.sh reads mem_shutdown_active — defers restarts
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
└─ php_fpm_max_children.sh ← before WebGUI is under load
Every minute (watchdog_orchestrator.sh in Orchestrators/):
→ Watchdogs/resource_watchdog.sh → Watchdogs/docker_watchdog.sh
→ Watchdogs/storage_watchdog.sh → Watchdogs/system_watchdog.sh
(see Watchdogs/README-Watchdogs.md for full flow)
Every 10 minutes (User Scripts):
└─ webgui_restart.sh
@@ -161,13 +145,11 @@ Every 10 minutes (User Scripts):
Step 3: restart emhttp → recheck
All failed → notify, exit 1
Weekly (weekly_maintenance.sh):
└─ clear_logs.sh
System logs: clear if > LOG_MIN_SIZE_MB
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
Manual operations:
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)