Prune stale container strike counts, not just skip-list entries

The 2026-07-19 prune covered the skip list and intentional stops but not the
per-container strike counts, which have the same failure: set_strikes 0 only
fires when a container is seen running again, so an uninstalled one keeps its
count forever and every consumer reads the host as unhealthy. Reserved daemon_*
keys are excluded.
This commit is contained in:
Gmer4Lfe
2026-08-02 00:45:29 -04:00
parent 43b5443b30
commit 0b4ff27be4
+33
View File
@@ -137,6 +137,17 @@
# All docker commands wrapped in timeout. Daemon hangs cannot stall the # All docker commands wrapped in timeout. Daemon hangs cannot stall the
# watchdog and leave containers unmonitored between cycles. # watchdog and leave containers unmonitored between cycles.
# #
# Stale Entry Pruning
# Every cycle, entries naming a container that no longer exists are removed from
# DOCKER_WATCHDOG_FAILED_FILE, DOCKER_WATCHDOG_INTENTIONAL_FILE, and the per-container
# strike counts in WATCHDOG_STATE_FILE. None of these can clear themselves: every
# clearing path requires seeing the container running again, which never happens once
# it is uninstalled. Without pruning, one removed container makes this host report
# unhealthy permanently. Reserved daemon_* keys in WATCHDOG_STATE_FILE are never pruned.
# Pruning runs under --dry-run as well: the documented contract for that flag is that
# nothing gets restarted, and dropping a record of a container that no longer exists is
# housekeeping, not an action. It is idempotent — repeated runs converge.
#
# Notification Batching # Notification Batching
# Events collected across a full cycle and sent as a single summary. # Events collected across a full cycle and sent as a single summary.
# Prevents notification floods when a shared dependency failure cascades. # Prevents notification floods when a shared dependency failure cascades.
@@ -710,6 +721,28 @@ CYCLE_START=$(date +%s)
done done
unset _prune_file _prune_snapshot _prune_container unset _prune_file _prune_snapshot _prune_container
# Same failure, different file. WATCHDOG_STATE_FILE cannot join the loop above: it stores
# key:value rather than one name per line, and it holds reserved daemon_* keys alongside the
# per-container strike counts. A strike entry for an uninstalled container never clears on its
# own — set_strikes "$container" 0 only fires when the container is seen running again, which
# by definition never happens — so the count sits there and every consumer that treats a
# non-zero strike as unhealthy reports this host unhealthy forever. Confirmed live 2026-08-02:
# claudeclaw_docker at 2119 strikes long after removal, which is what kept the monitor page's
# watchdog summary from ever going green.
if [[ -s "$WATCHDOG_STATE_FILE" ]]; then
mapfile -t _prune_snapshot < "$WATCHDOG_STATE_FILE"
for _prune_line in "${_prune_snapshot[@]}"; do
_prune_container="${_prune_line%%:*}"
[[ -z "$_prune_container" ]] && continue
[[ "$_prune_container" == daemon_* ]] && continue # reserved keys, not containers
if ! timeout "$DOCKER_TIMEOUT" docker inspect "$_prune_container" &>/dev/null; then
sed -i "/^${_prune_container}:/d" "$WATCHDOG_STATE_FILE" 2>/dev/null
warn "$_prune_container no longer exists — removed from $(basename "$WATCHDOG_STATE_FILE")"
fi
done
unset _prune_snapshot _prune_line _prune_container
fi
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs) _skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
[[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed" [[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed"
_intentional_contents=$(cat "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null | tr '\n' ' ' | xargs) _intentional_contents=$(cat "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null | tr '\n' ' ' | xargs)