FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Structural reorganization: watchdog taxonomy + server_reboot integration

Browse Source 
Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
This commit is contained in:
Gmer4Lfe
2026-05-22 20:17:22 -04:00
parent 670f25fbbd
commit cf21efaeea
18 changed files with 962 additions and 824 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Configurations/master.conf
+11 -3
View File
@@ -291,17 +291,25 @@
# Schedule: * * * * * (every minute)
# NOT in ARRAY_START_SCRIPTS — has its own cron entry.
# Order matters — resource first (frees pressure), docker second (heals with freed resources),
# storage third (pool/data health — never reboots), system last (last line of defense).
# system third (storage + webgui component health), stability last (last line of defense).
WATCHDOG_ORCHESTRATOR_SCRIPTS=(
"Watchdogs/resource_watchdog.sh" # reduce system pressure before healing attempts
"Watchdogs/docker_watchdog.sh" # heal containers with freed resources
"Watchdogs/storage_watchdog.sh" # pool and appdata health — alert and remediate
"Watchdogs/system_watchdog.sh" # reboot if all else fails — last line of defense
"Watchdogs/system_watchdog.sh" # system component health — storage + webgui
"Watchdogs/stability_watchdog.sh" # reboot if all else fails — last line of defense
)
WATCHDOG_ORCHESTRATOR_HEARTBEAT=true
WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS=1
# ━━━ System Watchdog ━━━
# system_watchdog.sh runs SYSTEM_WATCHDOG_SCRIPTS sequentially each cycle.
# Called by watchdog_orchestrator.sh — not scheduled directly.
SYSTEM_WATCHDOG_SCRIPTS=(
"Watchdogs/System/storage_watchdog.sh" # pool growth + runaway log detection
"Watchdogs/System/webgui_watchdog.sh" # WebGUI availability — nginx → php-fpm → emhttp
)
# ━━━ Critical Sync Maintenance ━━━
# critical_sync_maintenance.sh runs every 15 minutes.
# Order: CRITICAL_MAINTENANCE_SCRIPTS (jobs) → CRITICAL_SYNC_SHARES (rsync) → partnership --check