Prune stale container strike counts, not just skip-list entries
The 2026-07-19 prune covered the skip list and intentional stops but not the per-container strike counts, which have the same failure: set_strikes 0 only fires when a container is seen running again, so an uninstalled one keeps its count forever and every consumer reads the host as unhealthy. Reserved daemon_* keys are excluded.
This commit is contained in:
@@ -137,6 +137,17 @@
|
|||||||
# All docker commands wrapped in timeout. Daemon hangs cannot stall the
|
# All docker commands wrapped in timeout. Daemon hangs cannot stall the
|
||||||
# watchdog and leave containers unmonitored between cycles.
|
# watchdog and leave containers unmonitored between cycles.
|
||||||
#
|
#
|
||||||
|
# Stale Entry Pruning
|
||||||
|
# Every cycle, entries naming a container that no longer exists are removed from
|
||||||
|
# DOCKER_WATCHDOG_FAILED_FILE, DOCKER_WATCHDOG_INTENTIONAL_FILE, and the per-container
|
||||||
|
# strike counts in WATCHDOG_STATE_FILE. None of these can clear themselves: every
|
||||||
|
# clearing path requires seeing the container running again, which never happens once
|
||||||
|
# it is uninstalled. Without pruning, one removed container makes this host report
|
||||||
|
# unhealthy permanently. Reserved daemon_* keys in WATCHDOG_STATE_FILE are never pruned.
|
||||||
|
# Pruning runs under --dry-run as well: the documented contract for that flag is that
|
||||||
|
# nothing gets restarted, and dropping a record of a container that no longer exists is
|
||||||
|
# housekeeping, not an action. It is idempotent — repeated runs converge.
|
||||||
|
#
|
||||||
# Notification Batching
|
# Notification Batching
|
||||||
# Events collected across a full cycle and sent as a single summary.
|
# Events collected across a full cycle and sent as a single summary.
|
||||||
# Prevents notification floods when a shared dependency failure cascades.
|
# Prevents notification floods when a shared dependency failure cascades.
|
||||||
@@ -710,6 +721,28 @@ CYCLE_START=$(date +%s)
|
|||||||
done
|
done
|
||||||
unset _prune_file _prune_snapshot _prune_container
|
unset _prune_file _prune_snapshot _prune_container
|
||||||
|
|
||||||
|
# Same failure, different file. WATCHDOG_STATE_FILE cannot join the loop above: it stores
|
||||||
|
# key:value rather than one name per line, and it holds reserved daemon_* keys alongside the
|
||||||
|
# per-container strike counts. A strike entry for an uninstalled container never clears on its
|
||||||
|
# own — set_strikes "$container" 0 only fires when the container is seen running again, which
|
||||||
|
# by definition never happens — so the count sits there and every consumer that treats a
|
||||||
|
# non-zero strike as unhealthy reports this host unhealthy forever. Confirmed live 2026-08-02:
|
||||||
|
# claudeclaw_docker at 2119 strikes long after removal, which is what kept the monitor page's
|
||||||
|
# watchdog summary from ever going green.
|
||||||
|
if [[ -s "$WATCHDOG_STATE_FILE" ]]; then
|
||||||
|
mapfile -t _prune_snapshot < "$WATCHDOG_STATE_FILE"
|
||||||
|
for _prune_line in "${_prune_snapshot[@]}"; do
|
||||||
|
_prune_container="${_prune_line%%:*}"
|
||||||
|
[[ -z "$_prune_container" ]] && continue
|
||||||
|
[[ "$_prune_container" == daemon_* ]] && continue # reserved keys, not containers
|
||||||
|
if ! timeout "$DOCKER_TIMEOUT" docker inspect "$_prune_container" &>/dev/null; then
|
||||||
|
sed -i "/^${_prune_container}:/d" "$WATCHDOG_STATE_FILE" 2>/dev/null
|
||||||
|
warn "$_prune_container no longer exists — removed from $(basename "$WATCHDOG_STATE_FILE")"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
unset _prune_snapshot _prune_line _prune_container
|
||||||
|
fi
|
||||||
|
|
||||||
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
||||||
[[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed"
|
[[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed"
|
||||||
_intentional_contents=$(cat "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
_intentional_contents=$(cat "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
||||||
|
|||||||
Reference in New Issue
Block a user