fix: correct state file ownership and function naming across watchdogs

stability_watchdog: remove SYS_WATCHDOG_FAILED_FILE from state file
touch list — skip list is owned by docker_watchdog, not stability.

docker_watchdog: rename check_system_watchdog_state →
check_resource_watchdog_state (reads RW_STATE_FILE from resource_watchdog,
not SYS_WATCHDOG_STATE_FILE from stability_watchdog). Update all comments
and state file docs to reflect correct ownership.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Gmer4Lfe
2026-05-31 14:46:18 -04:00
co-authored by Claude Sonnet 4.6
parent cce6fc5b65
commit 9d6772fbd6
2 changed files with 11 additions and 14 deletions
+10 -13
View File
@@ -53,8 +53,8 @@
# Timeout protection — all docker commands wrapped in timeout
# Docker daemon check — each cycle begins with daemon health check; hung daemon →
# restart via rc.docker → stability_watchdog.sh escalates if needed
# RAM emergency defer — reads SYS_WATCHDOG_STATE_FILE; stands down while
# stability_watchdog.sh is managing a RAM emergency
# RAM emergency defer — reads RW_STATE_FILE; stands down while
# resource_watchdog.sh is managing a RAM emergency
#
# ==============================================================================================
# DESIGN PRINCIPLES
@@ -116,10 +116,10 @@
# critical, skip cycle. stability_watchdog.sh handles further escalation.
#
# RAM Emergency Deferral
# Reads SYS_WATCHDOG_STATE_FILE each cycle. If stability_watchdog.sh has set
# Reads RW_STATE_FILE each cycle. If resource_watchdog.sh has set
# mem_shutdown_active=true, all restart logic defers until the flag clears.
# Stale state guard: if file is >2 hours old with flag still set,
# stability_watchdog.sh has likely stopped — watchdog resumes normal operation.
# resource_watchdog.sh has likely stopped — watchdog resumes normal operation.
#
# Timeout Protection
# All docker commands wrapped in timeout. Daemon hangs cannot stall the
@@ -133,13 +133,10 @@
# STATE FILES
# ==============================================================================================
#
# WATCHDOG_STATE_FILE — strike counts (default: /tmp — resets on reboot)
# SYS_WATCHDOG_FAILED_FILE — skip list (default: /boot/config — survives reboots)
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection
# SYS_WATCHDOG_STATE_FILE — shared state with system_watchdog.sh (RAM emergency flag)
#
# /tmp files reset on reboot — correct, pre-reboot strike counts are meaningless after it.
# /boot/config files survive reboots — correct, a skip-listed container is still broken after one.
# WATCHDOG_STATE_FILE — strike counts, daemon flags (STATE_DIR — survives reboots)
# SYS_WATCHDOG_FAILED_FILE — container skip list (STATE_DIR — survives reboots)
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection (DATA_DIR)
# RW_STATE_FILE — read-only: resource_watchdog RAM emergency flag
#
# ==============================================================================================
# CONFIGURATION
@@ -489,7 +486,7 @@ _dr_raw=$(get_strikes "daemon_restarted_flag" "$WATCHDOG_STATE_FILE")
# 0 = normal — run all checks
# 1 = RAM emergency active — defer container management this cycle
check_system_watchdog_state() {
check_resource_watchdog_state() {
# Returns 0 = normal operation | 1 = defer, resource_watchdog RAM emergency active
local state_file="$RW_STATE_FILE"
@@ -635,7 +632,7 @@ CYCLE_START=$(date +%s)
# ── RAM emergency check — resource_watchdog.sh managing containers ────────────────────────
# If resource_watchdog.sh has triggered a hard RAM shutdown, defer all container
# management this run to prevent undoing the emergency stop and re-pressuring RAM.
if ! check_system_watchdog_state; then
if ! check_resource_watchdog_state; then
MEM_KB=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
MEM_GB=$(( MEM_KB / 1024 / 1024 ))
warn "RAM emergency active (${MEM_GB}GB free) — resource_watchdog.sh managing containers"
+1 -1
View File
@@ -134,7 +134,7 @@ SYS_WATCHDOG_REBOOT_WINDOW=$(( SYS_WATCHDOG_REBOOT_WINDOW_HRS * 3600 ))
# Ensure state files exist
for state_file in "$SYS_WATCHDOG_STATE_FILE" "$SYS_WATCHDOG_REBOOT_LOG" \
"$SYS_WATCHDOG_FAILED_FILE" "$SYS_WATCHDOG_OOM_FILE"; do
"$SYS_WATCHDOG_OOM_FILE"; do
touch "$state_file" 2>/dev/null || {
error "Cannot create state file: $state_file"
exit 1