FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Structural reorganization: watchdog taxonomy + server_reboot integration

Browse Source 
Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
This commit is contained in:
Gmer4Lfe
2026-05-22 20:17:22 -04:00
parent 670f25fbbd
commit cf21efaeea
18 changed files with 962 additions and 824 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Watchdogs/docker_watchdog.sh
+7 -7
View File
@@ -52,9 +52,9 @@
# Parity awareness — skips restart actions during parity check
# Timeout protection — all docker commands wrapped in timeout
# Docker daemon check — each cycle begins with daemon health check; hung daemon →
# restart via rc.docker → system_watchdog.sh escalates if needed
# restart via rc.docker → stability_watchdog.sh escalates if needed
# RAM emergency defer — reads SYS_WATCHDOG_STATE_FILE; stands down while
# system_watchdog.sh is managing a RAM emergency
# stability_watchdog.sh is managing a RAM emergency
#
# ==============================================================================================
# DESIGN PRINCIPLES
@@ -113,13 +113,13 @@
# Docker Daemon Health Check
# First operation every cycle. Daemon not responding within DOCKER_TIMEOUT →
# restart via /etc/rc.d/rc.docker → verify recovery. If still hung: log
# critical, skip cycle. system_watchdog.sh handles further escalation.
# critical, skip cycle. stability_watchdog.sh handles further escalation.
#
# RAM Emergency Deferral
# Reads SYS_WATCHDOG_STATE_FILE each cycle. If system_watchdog.sh has set
# Reads SYS_WATCHDOG_STATE_FILE each cycle. If stability_watchdog.sh has set
# mem_shutdown_active=true, all restart logic defers until the flag clears.
# Stale state guard: if file is >2 hours old with flag still set,
# system_watchdog.sh has likely stopped — watchdog resumes normal operation.
# stability_watchdog.sh has likely stopped — watchdog resumes normal operation.
#
# Timeout Protection
# All docker commands wrapped in timeout. Daemon hangs cannot stall the
@@ -465,7 +465,7 @@ is_parity_running() {
# At WATCHDOG_DAEMON_STRIKE_LIMIT → attempt daemon restart via rc.docker
# After restart → wait WATCHDOG_DAEMON_RESTART_WAIT seconds → verify
# If verified → clear strikes, continue cycle ✅
# If still hung → notify critical, skip cycle → system_watchdog.sh escalates from here
# If still hung → notify critical, skip cycle → stability_watchdog.sh escalates from here
#
# Returns: 0 = daemon healthy | 1 = daemon down, skip this cycle
@@ -543,7 +543,7 @@ check_docker_daemon() {
if [[ "$WATCHDOG_DAEMON_RESTARTED" == true ]]; then
error "Docker daemon still unresponsive after restart attempt"
error "system_watchdog.sh will handle further escalation"
error "stability_watchdog.sh will handle further escalation"
queue_notify "Docker daemon hung on $(hostname) — restart failed — manual intervention needed" "critical"
flush_notify
return 1