docker_watchdog: config dump at startup (thresholds/limits), skip list shown when active, per-container healthy log for Tier1 required + mem/CPU monitored containers
stability_watchdog: config dump with all tier thresholds, log() on pass for rootfs/log/tmp/load/zombies/NIC checks (previously silent on clean)
resource_watchdog: config dump with all pressure thresholds and container lists, log normal pressure state with live RAM/load values
system_watchdog: per-script timing on each child script run
network_watchdog: config dump (internet URL, DDNS domain/container, NPM URL, strike limit)
storage_watchdog: config dump (growth threshold, log max, paths, suppress ceilings)
webgui_watchdog: log nginx worker and php-fpm worker counts on healthy check
Two root causes of the May 31 19:30 hang (orchestrator stuck for 45min):
1. /etc/rc.d/rc.docker restart had no timeout — on a host with many
containers, stopping them all before daemon restart can take 30+
minutes. Added timeout 180 so it's bounded.
2. WATCHDOG_DAEMON_RESTART_WAIT=900 (15min sleep after restart) —
unnecessary. The daemon itself comes up in seconds; containers are
handled by subsequent docker_watchdog cycles. Reduced to 90s in
master.conf, cutting worst-case orchestrator block from 45min to ~5min.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove dead DOCKER_TIMEOUT=10 from stability_watchdog (daemon check
is delegated to docker_watchdog; variable was never used post-refactor)
- Fix parity check detection: parity-date.txt is gone in Unraid 7.3;
now reads mdResync from var.ini (non-zero = check/sync in progress),
with parity-date.txt fallback for older versions
- Same is_parity_running() fix in docker_watchdog
- DOCKER_TIMEOUT in docker_watchdog now reads WATCHDOG_DAEMON_TIMEOUT
from master.conf (default 20s) instead of hardcoded 15s
- WATCHDOG_DAEMON_RESTART_WAIT now reads from master.conf
(default 900s/15min) instead of hardcoded 30s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
common.sh: add is_docker_enabled() and is_vm_manager_enabled() helpers
reading /boot/config/docker.cfg and /boot/config/domain.cfg.
docker_watchdog: exit cleanly if Docker not enabled in Unraid settings.
stability_watchdog: skip Docker daemon check and Docker container stop
if Docker not enabled; skip virsh VM shutdown if VM Manager not enabled.
server_reboot: skip VM shutdown and libvirt stop if VM Manager not enabled.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Skip list belongs to docker_watchdog, not stability_watchdog (legacy
naming from when docker+system were the only two watchdogs). Rename
variable and physical file system_watchdog_failed.db →
docker_watchdog_failed.db across all files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
stability_watchdog: remove SYS_WATCHDOG_FAILED_FILE from state file
touch list — skip list is owned by docker_watchdog, not stability.
docker_watchdog: rename check_system_watchdog_state →
check_resource_watchdog_state (reads RW_STATE_FILE from resource_watchdog,
not SYS_WATCHDOG_STATE_FILE from stability_watchdog). Update all comments
and state file docs to reflect correct ownership.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously only cleared inside the strikes>0 block. After a reboot,
strikes reset to 0 but the flag persists on flash — daemon could be
healthy but flag stays true, causing stability_watchdog to keep striking.
Now cleared on every healthy docker info check regardless of strike count.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docker_watchdog now writes daemon_confirmed_down=true to its state file
when restart is attempted and daemon is still unresponsive. Clears the
flag on recovery.
stability_watchdog removes the duplicate CRITICAL daemon check (which
bypassed all strikes, all abort conditions, and had its own rc.docker
restart). Replaced with a standard strike check reading the
daemon_confirmed_down flag from docker_watchdog's state file.
Result: docker_watchdog owns all daemon restart logic. stability_watchdog
escalates to reboot only after N consecutive strike cycles confirming the
daemon is truly unrecoverable — same pattern as all other standard checks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
- network_watchdog: remove ISSUES++ for Tailscale — it's notify-only by
design, no restart possible; cascading to system watchdog exit 1 was noise
- docker_watchdog: move banner to top of cycle so body errors appear under
the correct section header, not visually inside Resource Manager output
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
Gmer4Lfe