feat: watchdog architecture v2 — resource manager + single-pass orchestrator
Introduce a four-layer self-healing stack replacing the continuous-loop watchdogs: - resource_manager.sh (new): single-pass pressure reduction layer; throttles SABnzbd/qBit at level 1, docker-pauses background containers at level 2, docker-stops optional containers and signals docker_watchdog to defer at level 3; graduated recovery with hysteresis - watchdog_orchestrator.sh (new, Orchestrators/): runs resource_manager → docker_watchdog → system_watchdog in sequence; intended for per-minute cron via User Scripts; startup grace, acquire_lock to prevent pile-up, heartbeat - docker_watchdog.sh: de-looped to single-pass; daemon strikes persisted to state file across runs; cross-script coordination reads RM_STATE_FILE instead of SYS_WATCHDOG_STATE_FILE - system_watchdog.sh: de-looped to single-pass; stripped of all container management (shutdown_non_essential_containers removed); reboot-only last resort - master.conf: removed system_watchdog and docker_watchdog from ARRAY_START_SCRIPTS; added WATCHDOG ORCHESTRATOR and RESOURCE MANAGER sections - master_host1.conf: added RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS arrays - common.sh: aliased RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS via detect_hosts() - continuous_scripts_status.sh: moved to Tools/ (preserved for future use) - sunday_morning_coffee_report.sh: watchdog section updated to use state file mtime checks instead of is_running; added Resource Manager subsection; fixed mem_shutdown grep filter pointing to wrong state file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Gmer4Lfe
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
f16c962ac0
commit
309546e615
+86
-36
@@ -278,7 +278,9 @@
|
||||
# Scripts launched by array_started.sh when the array comes online.
|
||||
# Launched in order — each as a background process.
|
||||
# One-shot scripts (ramdisk, syslog, fpm, inotify, network) run and exit naturally.
|
||||
# Continuous scripts (watchdogs, failover) run until array stops.
|
||||
# Continuous scripts (failover) run until array stops.
|
||||
# Watchdogs (resource_manager, docker_watchdog, system_watchdog) are cronned via
|
||||
# watchdog_orchestrator.sh — NOT launched here.
|
||||
ARRAY_START_SCRIPTS=(
|
||||
"git_pull_execute.sh" # pull latest scripts before anything starts
|
||||
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
|
||||
@@ -286,9 +288,7 @@
|
||||
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
|
||||
"unRAID_Essentials/inotify_tuning.sh" # bump inotify limits — containers miss events if exhausted
|
||||
"Docker_Essentials/docker_network_connect.sh" # ensure networks exist + connect containers
|
||||
"unRAID_Essentials/system_watchdog.sh" # system health monitor — continuous loop
|
||||
"Docker_Essentials/docker_watchdog.sh" # container health monitor — continuous loop
|
||||
"Fallback/fallback.sh" # mutual failover — HOST2 back online
|
||||
"Fallback/fallback.sh" # mutual failover — continuous
|
||||
)
|
||||
|
||||
# ━━━ Intermediate Sync Maintenance ━━━
|
||||
@@ -1114,12 +1114,85 @@
|
||||
EMBY_REPORT_DAYS=7 # days to include in the report period
|
||||
EMBY_REPORT_TOP_N=10 # number of top content items to show
|
||||
|
||||
# ==============================================================================================
|
||||
# ── WATCHDOG ORCHESTRATOR ─────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Runs resource_manager → docker_watchdog → system_watchdog in sequence each cron cycle.
|
||||
# Schedule: * * * * * (every minute via User Scripts plugin)
|
||||
# NOT in ARRAY_START_SCRIPTS — has its own cron entry.
|
||||
|
||||
WATCHDOG_ORCHESTRATOR_HEARTBEAT=true
|
||||
WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS=1
|
||||
|
||||
# ==============================================================================================
|
||||
# ── RESOURCE MANAGER ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Pressure reduction layer — keeps the system comfortable before things break.
|
||||
# Called by watchdog_orchestrator.sh. Single-pass, not a continuous loop.
|
||||
#
|
||||
# ── PRESSURE LEVELS ───────────────────────────────────────────────────────────────────────────
|
||||
# Level 1 (soft) — throttle SABnzbd + qBit download speeds
|
||||
# Level 2 (medium) — further throttle + docker pause background containers
|
||||
# Level 3 (hard) — docker stop optional containers, signal docker_watchdog to defer
|
||||
#
|
||||
# ── PER-HOST CONTAINER LISTS ──────────────────────────────────────────────────────────────────
|
||||
# HOST*_RM_PAUSE_CONTAINERS — docker pause at medium pressure (in master_host*.conf)
|
||||
# HOST*_RM_STOP_CONTAINERS — docker stop at hard pressure (in master_host*.conf)
|
||||
|
||||
RM_ENABLED=true
|
||||
RM_STATE_FILE="/tmp/resource_manager_state.db"
|
||||
|
||||
# ━━━ Pressure Thresholds ━━━
|
||||
# Graduated RAM response — resource_manager acts before system_watchdog reboots.
|
||||
# RM_RAM_SOFT_GB > RM_RAM_MEDIUM_GB > RM_RAM_HARD_GB > SYS_WATCHDOG_MEM_GB always
|
||||
RM_RAM_SOFT_GB=12 # throttle start — reduce background load
|
||||
RM_RAM_MEDIUM_GB=8 # pause background containers
|
||||
RM_RAM_HARD_GB=6 # stop optional containers (was SYS_WATCHDOG_MEM_SHUTDOWN_GB)
|
||||
RM_RAM_RECOVER_GB=20 # RAM must reach this before restoring hard-stopped containers
|
||||
|
||||
# Load average thresholds — multiplier × core count
|
||||
RM_LOAD_SOFT_MULTIPLIER=2.0 # soft pressure: 2× cores sustained
|
||||
RM_LOAD_MEDIUM_MULTIPLIER=3.0 # medium pressure: 3× cores sustained
|
||||
|
||||
# Consecutive runs at lower pressure before de-escalating
|
||||
RM_RECOVER_CYCLES=3
|
||||
|
||||
# ━━━ SABnzbd Throttle ━━━
|
||||
# Speed values: "50M" = 50 MB/s, "0" = unlimited
|
||||
RM_SABNZBD_ENABLED=true
|
||||
RM_SABNZBD_SPEED_SOFT="50M"
|
||||
RM_SABNZBD_SPEED_MEDIUM="10M"
|
||||
|
||||
# ━━━ qBittorrent Throttle ━━━
|
||||
# KB/s — 0 = unlimited
|
||||
RM_QBIT_ENABLED=true
|
||||
RM_QBIT_DL_SOFT=51200 # 50 MB/s
|
||||
RM_QBIT_DL_MEDIUM=10240 # 10 MB/s
|
||||
|
||||
# ━━━ Critical Containers ━━━
|
||||
# Never paused or stopped regardless of pressure level.
|
||||
# Keep DNS, auth, media serving, and live TV always running.
|
||||
RM_CRITICAL_CONTAINERS=(
|
||||
"NginxProxyManager" # reverse proxy — internet access
|
||||
"Authelia" # auth — nothing accessible without it
|
||||
"Authelia-Secondary"
|
||||
"Mariadb-Authelia" # Authelia dependency
|
||||
"Mariadb-Authelia-Secondary"
|
||||
"Redis-Authelia" # Authelia dependency
|
||||
"Redis-Authelia-Secondary"
|
||||
"AdGuard-Home" # DNS — all LAN resolution
|
||||
"Emby" # media server — Live TV buffering
|
||||
"Dispatcharr" # Live TV scheduler — loses state if stopped
|
||||
"Dispatcharr-Basic"
|
||||
"Dispatcharr-Iptv-Users"
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Continuous system health monitoring — last line of defense before a crash.
|
||||
# Started by array_started.sh — runs until array stops.
|
||||
# Re-sources all three conf files each cycle — config changes take effect on next cycle.
|
||||
# Single-pass system health check — last line of defense before a crash.
|
||||
# Called by watchdog_orchestrator.sh every minute. NOT started by array_started.sh.
|
||||
# Re-sources config at each orchestrator run — config changes take effect immediately.
|
||||
#
|
||||
# ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
|
||||
# CRITICAL — bypass ALL strikes, reboot immediately
|
||||
@@ -1132,16 +1205,9 @@
|
||||
# STANDARD — strike system (N consecutive failures → reboot)
|
||||
# RAM tiers, high load, CPU temp, zombies, /var/log, /tmp, containers
|
||||
#
|
||||
# ── RAM TIERS ─────────────────────────────────────────────────────────────────────────────────
|
||||
# MEM_WARN_GB — warn + notify only (informational)
|
||||
# MEM_SHUTDOWN_GB — stop non-essential containers, recover above MEM_RECOVER_GB
|
||||
# MEM_GB — strike system → reboot (or bypass with OOM)
|
||||
#
|
||||
# ── CONTAINER SHUTDOWN ────────────────────────────────────────────────────────────────────────
|
||||
# At MEM_SHUTDOWN_GB: stop all containers NOT in MEM_SHUTDOWN_EXCLUDED list
|
||||
# Excluded containers stay running — DNS, auth, Emby, Dispatcharr
|
||||
# Stopped containers stay stopped until RAM recovers above MEM_RECOVER_GB
|
||||
# Strike list applied — doesn't flip-flop every cycle
|
||||
# ── RAM ───────────────────────────────────────────────────────────────────────────────────────
|
||||
# SYS_WATCHDOG_MEM_GB — strike system → reboot (or OOM bypass)
|
||||
# Warn/shutdown/recover RAM tiers are handled by resource_manager.sh
|
||||
|
||||
# ━━━ State Files ━━━
|
||||
SYS_WATCHDOG_STATE_FILE="/tmp/system_watchdog_state.db" # /tmp — resets on reboot ✅
|
||||
@@ -1167,26 +1233,10 @@
|
||||
SYSTEM_WATCHDOG_HEARTBEAT=true
|
||||
SYSTEM_WATCHDOG_HEARTBEAT_HOURS=1
|
||||
|
||||
# ━━━ RAM Tiers ━━━
|
||||
# Three-level RAM response — graduated action instead of single threshold.
|
||||
# HOST1 has 128GB, HOST2 has 64GB — adjust accordingly.
|
||||
# MEM_WARN_GB > MEM_SHUTDOWN_GB > MEM_GB always
|
||||
SYS_WATCHDOG_MEM_WARN_GB=10 # warn + notify — informational only
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_GB=6 # stop non-essential containers
|
||||
# ━━━ RAM Reboot Threshold ━━━
|
||||
# Reboot trigger only — warn/shutdown/recover handled by resource_manager.sh
|
||||
# RM_RAM_HARD_GB > SYS_WATCHDOG_MEM_GB always (RM acts before watchdog reboots)
|
||||
SYS_WATCHDOG_MEM_GB=4 # strike system → reboot
|
||||
SYS_WATCHDOG_MEM_RECOVER_GB=30 # RAM must recover above this before restarting containers
|
||||
|
||||
# Containers excluded from RAM emergency shutdown.
|
||||
# These stay running regardless of RAM pressure.
|
||||
# DNS and auth must stay up, Emby and Dispatcharr for Live TV continuity.
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
|
||||
"NginxProxyManager" # DNS / reverse proxy — internet access
|
||||
"Authelia" # auth — without this nothing is accessible
|
||||
"Mariadb" # Authelia dependency
|
||||
"Redis" # Authelia dependency
|
||||
"Emby" # media server — Live TV buffering
|
||||
"Dispatcharr" # Live TV scheduler — loses state if stopped
|
||||
)
|
||||
|
||||
# ━━━ OOM Bypass Settings ━━━
|
||||
# OOM bypass: if RAM is critically low AND kernel OOM kills exceed this threshold
|
||||
|
||||
Reference in New Issue
Block a user