feat: watchdog architecture v2 — resource manager + single-pass orchestrator

Introduce a four-layer self-healing stack replacing the continuous-loop watchdogs:

- resource_manager.sh (new): single-pass pressure reduction layer; throttles
  SABnzbd/qBit at level 1, docker-pauses background containers at level 2,
  docker-stops optional containers and signals docker_watchdog to defer at
  level 3; graduated recovery with hysteresis

- watchdog_orchestrator.sh (new, Orchestrators/): runs resource_manager →
  docker_watchdog → system_watchdog in sequence; intended for per-minute cron
  via User Scripts; startup grace, acquire_lock to prevent pile-up, heartbeat

- docker_watchdog.sh: de-looped to single-pass; daemon strikes persisted to
  state file across runs; cross-script coordination reads RM_STATE_FILE instead
  of SYS_WATCHDOG_STATE_FILE

- system_watchdog.sh: de-looped to single-pass; stripped of all container
  management (shutdown_non_essential_containers removed); reboot-only last resort

- master.conf: removed system_watchdog and docker_watchdog from
  ARRAY_START_SCRIPTS; added WATCHDOG ORCHESTRATOR and RESOURCE MANAGER sections

- master_host1.conf: added RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS arrays

- common.sh: aliased RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS via detect_hosts()

- continuous_scripts_status.sh: moved to Tools/ (preserved for future use)

- sunday_morning_coffee_report.sh: watchdog section updated to use state file
  mtime checks instead of is_running; added Resource Manager subsection;
  fixed mem_shutdown grep filter pointing to wrong state file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Gmer4Lfe
2026-05-12 17:41:59 -04:00
co-authored by Claude Sonnet 4.6
parent f16c962ac0
commit 309546e615
9 changed files with 925 additions and 277 deletions
+26
View File
@@ -670,6 +670,32 @@
# Enable only if HOST1 has no CPU-intensive workloads.
HOST1_SYS_WATCHDOG_CHECK_RUNAWAY=false
# ==============================================================================================
# ── RESOURCE MANAGER ──────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Containers to manage under pressure — see master.conf RM_CRITICAL_CONTAINERS for exclusions.
# docker pause at medium pressure (RAM < RM_RAM_MEDIUM_GB or load > medium threshold)
# Suspended in-place — instant to pause/unpause, no state lost, no restart delay.
HOST1_RM_PAUSE_CONTAINERS=(
"Huntarr" # arr search automation — safe to suspend
"Cleanuparr" # download cleanup — safe to suspend
"Healarr" # arr health checks — safe to suspend
"Soularr" # Slskd automation — background only
"ChannelTube" # YouTube archiver — background only
"Pinchflat" # YouTube archiver — background only
)
# docker stop at hard pressure (RAM < RM_RAM_HARD_GB)
# Full stop — these are optional/heavy services that free significant RAM when stopped.
# resource_manager.sh restarts them when pressure fully clears (RAM >= RM_RAM_RECOVER_GB).
HOST1_RM_STOP_CONTAINERS=(
"LocalAI" # GPU/CPU heavy — largest RAM consumer when idle
"7DaysToDie" # game server — optional
"V-Rising" # game server — optional
"Code-Server" # IDE — not needed during pressure events
)
# ==============================================================================================
# ──────────────────────── End Of HOST1 Variables ──────────────────────────────────────────────
# ==============================================================================================