Bring script headers onto the template and close safeguard gaps
Headers claimed protections the code never had, and several destructive paths had no guard against a collapsed config value.
This commit is contained in:
@@ -54,6 +54,39 @@
|
||||
# (Container health is owned by docker_watchdog — not checked here.)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Last Line of Defense
|
||||
# Every other watchdog tries to heal a specific subsystem. This one assumes
|
||||
# those attempts have already failed and holds the only irreversible remedy in
|
||||
# the ecosystem — a reboot. That authority is why nearly every check here is
|
||||
# gated behind strikes, tiers and abort conditions.
|
||||
#
|
||||
# Evidence Before Reboot
|
||||
# Strikes are the default; bypassing them requires corroboration, not just a
|
||||
# worse number. Tier 2 needs low RAM AND active OOM kills before it acts —
|
||||
# low RAM alone is a reading, low RAM plus processes being killed is a crisis.
|
||||
#
|
||||
# Unrecoverable Conditions Skip the Queue
|
||||
# Tier 1 conditions share one property: the system cannot heal from them and
|
||||
# waiting makes recovery less likely. A full rootfs or a kernel oops degrades
|
||||
# further every cycle, and strike-counting through it only guarantees the
|
||||
# reboot happens from a worse state.
|
||||
#
|
||||
# Data Safety Outranks Uptime
|
||||
# Reboots abort while a ZFS pool is unhealthy, parity is running, or the mover
|
||||
# is active. Interrupting those risks the data itself, which no amount of
|
||||
# uptime justifies. Tier 1 is the sole exception — an imminent crash will
|
||||
# interrupt them anyway, less gracefully.
|
||||
#
|
||||
# Clear Ownership Boundaries
|
||||
# Container health belongs to docker_watchdog.sh and is deliberately not
|
||||
# checked here. The Docker daemon check writes a flag for docker_watchdog
|
||||
# rather than acting on it. Two watchdogs remediating the same subsystem
|
||||
# would race.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
@@ -61,10 +94,61 @@
|
||||
# Reboot and container stop require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents a second watchdog instance from starting.
|
||||
# acquire_lock prevents a second watchdog instance from starting. Two instances
|
||||
# could each count strikes against the same condition and reach the reboot
|
||||
# threshold in half the intended time.
|
||||
#
|
||||
# State File Verification
|
||||
# All state files verified writable at startup — errors if any cannot be created.
|
||||
# Strike counts and the reboot log live in these files; if they silently failed
|
||||
# to persist, every cycle would look like strike 1 and the reboot rate limit
|
||||
# would never accumulate.
|
||||
#
|
||||
# Reboot Rate Limiting
|
||||
# No more than SYS_WATCHDOG_REBOOT_LIMIT reboots within
|
||||
# SYS_WATCHDOG_REBOOT_WINDOW_HRS. On hitting the limit the host powers off
|
||||
# instead of rebooting — a fault that survives repeated reboots will not be
|
||||
# fixed by more of them, and a box cycling endlessly is worse than one that
|
||||
# is cleanly down and obviously needs attention.
|
||||
#
|
||||
# Abort Conditions
|
||||
# Reboots are aborted while a ZFS pool is unhealthy, parity is running, or the
|
||||
# mover is active — each individually toggleable. Interrupting any of these
|
||||
# risks the data itself.
|
||||
#
|
||||
# Critical Tier Override
|
||||
# Tier 1 conditions bypass both strikes and abort conditions. These are states
|
||||
# the system cannot recover from and which degrade every cycle; waiting only
|
||||
# guarantees the eventual reboot happens from a worse position.
|
||||
#
|
||||
# Strike Threshold
|
||||
# Tier 3 requires SYS_WATCHDOG_STRIKE_LIMIT consecutive failing cycles. A
|
||||
# single bad sample — a momentary load spike, a transient RAM dip — never
|
||||
# reboots the system.
|
||||
#
|
||||
# OOM Corroboration
|
||||
# Tier 2 escalation requires low RAM AND active OOM kills in the same cycle.
|
||||
# Low RAM alone stays in the strike system.
|
||||
#
|
||||
# Ownership Boundary
|
||||
# Container health is not checked here — docker_watchdog.sh owns it. The Docker
|
||||
# daemon check writes daemon_confirmed_down for docker_watchdog rather than
|
||||
# remediating, so the two never act on the same subsystem.
|
||||
#
|
||||
# Aborted-Reboot Recovery
|
||||
# An EXIT trap is armed the moment containers start being stopped for a reboot
|
||||
# and disarmed only once the reboot is committed. If the script dies anywhere
|
||||
# in between, the trap restarts everything it stopped — the failure mode is a
|
||||
# running system, never a host left with all containers down and no reboot.
|
||||
#
|
||||
# Sync Before Reboot
|
||||
# sync is issued before both /sbin/poweroff and /sbin/reboot so pending writes
|
||||
# are flushed. Container stop is additionally bounded by a 60 second timeout so
|
||||
# one unresponsive container cannot hold the shutdown sequence open forever.
|
||||
#
|
||||
# Dry Run Support
|
||||
# --dry-run runs the full detection path and reports the reboot or shutdown
|
||||
# that would occur without issuing either.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -74,7 +158,7 @@
|
||||
# Full variable listing in master.conf. Key variables:
|
||||
#
|
||||
# SYS_WATCHDOG_REBOOT_WINDOW_HRS — reboot rate limit window (default: 12)
|
||||
# SYS_WATCHDOG_MAX_REBOOTS — max reboots in window before giving up (default: 3)
|
||||
# SYS_WATCHDOG_REBOOT_LIMIT — max reboots in window before giving up (default: 3)
|
||||
# SYS_WATCHDOG_STRIKE_LIMIT — consecutive failures before reboot (default: 2)
|
||||
# SYS_WATCHDOG_OOM_LIMIT — OOM kills/cycle to trigger URGENT bypass (default: 3)
|
||||
# SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED — containers exempt from memory shutdown
|
||||
|
||||
Reference in New Issue
Block a user