Bring script headers onto the template and close safeguard gaps

Headers claimed protections the code never had, and several destructive paths had no
guard against a collapsed config value.
This commit is contained in:
Gmer4Lfe
2026-08-01 20:37:59 -04:00
parent cdce877601
commit e8b114094a
78 changed files with 3301 additions and 277 deletions
+86 -2
View File
@@ -54,6 +54,39 @@
# (Container health is owned by docker_watchdog — not checked here.)
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Last Line of Defense
# Every other watchdog tries to heal a specific subsystem. This one assumes
# those attempts have already failed and holds the only irreversible remedy in
# the ecosystem — a reboot. That authority is why nearly every check here is
# gated behind strikes, tiers and abort conditions.
#
# Evidence Before Reboot
# Strikes are the default; bypassing them requires corroboration, not just a
# worse number. Tier 2 needs low RAM AND active OOM kills before it acts —
# low RAM alone is a reading, low RAM plus processes being killed is a crisis.
#
# Unrecoverable Conditions Skip the Queue
# Tier 1 conditions share one property: the system cannot heal from them and
# waiting makes recovery less likely. A full rootfs or a kernel oops degrades
# further every cycle, and strike-counting through it only guarantees the
# reboot happens from a worse state.
#
# Data Safety Outranks Uptime
# Reboots abort while a ZFS pool is unhealthy, parity is running, or the mover
# is active. Interrupting those risks the data itself, which no amount of
# uptime justifies. Tier 1 is the sole exception — an imminent crash will
# interrupt them anyway, less gracefully.
#
# Clear Ownership Boundaries
# Container health belongs to docker_watchdog.sh and is deliberately not
# checked here. The Docker daemon check writes a flag for docker_watchdog
# rather than acting on it. Two watchdogs remediating the same subsystem
# would race.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
@@ -61,10 +94,61 @@
# Reboot and container stop require root.
#
# Single Instance Lock
# acquire_lock prevents a second watchdog instance from starting.
# acquire_lock prevents a second watchdog instance from starting. Two instances
# could each count strikes against the same condition and reach the reboot
# threshold in half the intended time.
#
# State File Verification
# All state files verified writable at startup — errors if any cannot be created.
# Strike counts and the reboot log live in these files; if they silently failed
# to persist, every cycle would look like strike 1 and the reboot rate limit
# would never accumulate.
#
# Reboot Rate Limiting
# No more than SYS_WATCHDOG_REBOOT_LIMIT reboots within
# SYS_WATCHDOG_REBOOT_WINDOW_HRS. On hitting the limit the host powers off
# instead of rebooting — a fault that survives repeated reboots will not be
# fixed by more of them, and a box cycling endlessly is worse than one that
# is cleanly down and obviously needs attention.
#
# Abort Conditions
# Reboots are aborted while a ZFS pool is unhealthy, parity is running, or the
# mover is active — each individually toggleable. Interrupting any of these
# risks the data itself.
#
# Critical Tier Override
# Tier 1 conditions bypass both strikes and abort conditions. These are states
# the system cannot recover from and which degrade every cycle; waiting only
# guarantees the eventual reboot happens from a worse position.
#
# Strike Threshold
# Tier 3 requires SYS_WATCHDOG_STRIKE_LIMIT consecutive failing cycles. A
# single bad sample — a momentary load spike, a transient RAM dip — never
# reboots the system.
#
# OOM Corroboration
# Tier 2 escalation requires low RAM AND active OOM kills in the same cycle.
# Low RAM alone stays in the strike system.
#
# Ownership Boundary
# Container health is not checked here — docker_watchdog.sh owns it. The Docker
# daemon check writes daemon_confirmed_down for docker_watchdog rather than
# remediating, so the two never act on the same subsystem.
#
# Aborted-Reboot Recovery
# An EXIT trap is armed the moment containers start being stopped for a reboot
# and disarmed only once the reboot is committed. If the script dies anywhere
# in between, the trap restarts everything it stopped — the failure mode is a
# running system, never a host left with all containers down and no reboot.
#
# Sync Before Reboot
# sync is issued before both /sbin/poweroff and /sbin/reboot so pending writes
# are flushed. Container stop is additionally bounded by a 60 second timeout so
# one unresponsive container cannot hold the shutdown sequence open forever.
#
# Dry Run Support
# --dry-run runs the full detection path and reports the reboot or shutdown
# that would occur without issuing either.
#
# ==============================================================================================
# CONFIGURATION
@@ -74,7 +158,7 @@
# Full variable listing in master.conf. Key variables:
#
# SYS_WATCHDOG_REBOOT_WINDOW_HRS — reboot rate limit window (default: 12)
# SYS_WATCHDOG_MAX_REBOOTS — max reboots in window before giving up (default: 3)
# SYS_WATCHDOG_REBOOT_LIMIT — max reboots in window before giving up (default: 3)
# SYS_WATCHDOG_STRIKE_LIMIT — consecutive failures before reboot (default: 2)
# SYS_WATCHDOG_OOM_LIMIT — OOM kills/cycle to trigger URGENT bypass (default: 3)
# SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED — containers exempt from memory shutdown