FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: fix watchdog README and Manual — 15min schedule, dead vars, renamed vars, abort defaults, removed Required containers from stability tier

Browse Source
This commit is contained in:
Gmer4Lfe
2026-06-01 19:33:49 -04:00
parent d3454a3dab
commit 4b28529736
2 changed files with 35 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Watchdogs/Manual-Watchdogs.md
+22 -25
View File
@@ -26,7 +26,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
All watchdog scripts use a two-tier output model: `echo` lines are always visible;
`log` lines only appear when `--log` is passed.
All watchdog scripts are **single-pass scripts** called once per minute by the orchestrator.
All watchdog scripts are **single-pass scripts** called once every 15 minutes by the orchestrator.
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
visible. Per-check detail is suppressed on clean cycles.
@@ -528,18 +528,17 @@ RAM below `SYS_WATCHDOG_MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_L
Both conditions required — RAM alone uses the standard strike system.
OOM confirms the system is dying faster than watchdogs can heal.
**Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKES` consecutive failures → reboot)**
**Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKE_LIMIT` consecutive failures → reboot)**
| Check | Threshold |
|-------|-----------|
| Free RAM | `SYS_WATCHDOG_MEM_GB` (reboot trigger — earlier tiers handled by resource_watchdog) |
| Load average | `SYS_WATCHDOG_LOAD_MULTIPLIER` × cpu_count |
| CPU temperature | `SYS_WATCHDOG_CPU_TEMP` |
| Zombie processes | `SYS_WATCHDOG_ZOMBIES` |
| /var/log usage | `SYS_WATCHDOG_VAR_LOG_PCT` |
| CPU temperature | `SYS_WATCHDOG_CPU_TEMP_MAX` |
| Zombie processes | `SYS_WATCHDOG_ZOMBIE_LIMIT` |
| /var/log usage | `SYS_WATCHDOG_LOG_PCT` |
| /tmp usage | `SYS_WATCHDOG_TMP_PCT` |
| Array disk errors | mdstat error delta > 0 |
| NIC state | interface operstate != "up" |
| Required containers | containers in `HOST*_WATCHDOG_REQUIRED_CONTAINERS` (host*.conf) |
### RAM Tiers
@@ -563,8 +562,8 @@ Prevent reboot — running them would risk data loss:
```bash
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
SYS_WATCHDOG_ABORT_ON_PARITY=true # Parity check/rebuild running
SYS_WATCHDOG_ABORT_ON_MOVER=true # Mover running
SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity mid-check is worse than crashing
SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting move mid-run is worse than crashing
```
Tier 1 CRITICAL bypasses all abort conditions — an imminent crash outweighs data
@@ -574,10 +573,10 @@ safety concerns.
```bash
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # window in hours
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots within the window
SYS_WATCHDOG_REBOOT_LIMIT=3 # max reboots within the window
```
If the server reboots `SYS_WATCHDOG_MAX_REBOOTS` times within the window, the watchdog
If the server reboots `SYS_WATCHDOG_REBOOT_LIMIT` times within the window, the watchdog
switches from rebooting to notifying only. Prevents a boot loop where the watchdog
reboots → something crashes again immediately → reboot again.
@@ -629,7 +628,6 @@ HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"
# ── Docker Watchdog ────────────────────────────────────────────────────────────
DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes)
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
@@ -684,10 +682,9 @@ STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
# Reboot trigger only — warn/shutdown/recover RAM tiers handled by resource_watchdog.sh
# RW_RAM_HARD_GB > SYS_WATCHDOG_MEM_GB always (resource_watchdog acts before watchdog reboots)
SYS_WATCHDOG_STRIKE_LIMIT=2
SYSTEM_WATCHDOG_INTERVAL=300
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2
SYS_WATCHDOG_MAX_REBOOTS=3
SYS_WATCHDOG_REBOOT_WINDOW_HRS=12
SYS_WATCHDOG_REBOOT_LIMIT=3
SYS_WATCHDOG_OOM_LIMIT=3
SYS_WATCHDOG_MEM_GB=4 # strike system → reboot (last resort — below resource_watchdog hard stop)
@@ -695,20 +692,20 @@ SYS_WATCHDOG_MEM_GB=4 # strike system → reboot (last resort — b
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99
SYS_WATCHDOG_FD_CRITICAL_PCT=95
SYS_WATCHDOG_LOAD_MULTIPLIER=4
SYS_WATCHDOG_CPU_TEMP=85
SYS_WATCHDOG_ZOMBIES=20
SYS_WATCHDOG_VAR_LOG_PCT=80
SYS_WATCHDOG_TMP_PCT=85
SYS_WATCHDOG_CPU_TEMP_MAX=95
SYS_WATCHDOG_ZOMBIE_LIMIT=50
SYS_WATCHDOG_LOG_PCT=95
SYS_WATCHDOG_TMP_PCT=90
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=true
SYS_WATCHDOG_ABORT_ON_MOVER=true
SYS_WATCHDOG_ABORT_ON_PARITY=false
SYS_WATCHDOG_ABORT_ON_MOVER=false
# State files:
SYS_WATCHDOG_STATE_FILE="/tmp/system_watchdog_state.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
# State files (all in State_Files/ — survive reboots):
SYS_WATCHDOG_STATE_FILE="State_Files/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="State_Files/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="State_Files/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="State_Files/system_watchdog_oom.db"
# ── Network Watchdog ───────────────────────────────────────────────────────────
NETWORK_WATCHDOG_ENABLED=true