Commit Graph
9 Commits
Author SHA1 Message Date
Gmer4Lfe d3454a3dab docs: update watchdog headers — single-pass cron model, correct caller refs, stability vs system watchdog 2026-06-01 19:31:16 -04:00
Gmer4LfeandClaude Sonnet 4.6 a943a5c294 fix: rename System Watchdog → Stability Watchdog in labels and output
The stability_watchdog.sh script was being called "System Watchdog"
everywhere, conflicting with system_watchdog.sh (which orchestrates
storage/webgui/network sub-watchdogs). Fixes:

- --status header: "SYSTEM WATCHDOG STATUS" → "STABILITY WATCHDOG STATUS"
- reboot banner: "SYSTEM WATCHDOG — REBOOT TRIGGERED" → "STABILITY WATCHDOG"
- Interval line: replace undefined SYSTEM_WATCHDOG_INTERVAL variable
  with hardcoded "60s (cron — every minute)"
- --status check toggles: remove containers= (dead config — container
  check was removed from stability_watchdog in a prior refactor)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 19:52:12 -04:00
Gmer4LfeandClaude Sonnet 4.6 4187d52bac fix: watchdog hardening for Unraid 7.3 + Docker 29.x
- Remove dead DOCKER_TIMEOUT=10 from stability_watchdog (daemon check
  is delegated to docker_watchdog; variable was never used post-refactor)
- Fix parity check detection: parity-date.txt is gone in Unraid 7.3;
  now reads mdResync from var.ini (non-zero = check/sync in progress),
  with parity-date.txt fallback for older versions
- Same is_parity_running() fix in docker_watchdog
- DOCKER_TIMEOUT in docker_watchdog now reads WATCHDOG_DAEMON_TIMEOUT
  from master.conf (default 20s) instead of hardcoded 15s
- WATCHDOG_DAEMON_RESTART_WAIT now reads from master.conf
  (default 900s/15min) instead of hardcoded 30s

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 19:13:54 -04:00
Gmer4LfeandClaude Sonnet 4.6 46b27f609a safeguard: check Docker/VM Manager enabled before acting
common.sh: add is_docker_enabled() and is_vm_manager_enabled() helpers
reading /boot/config/docker.cfg and /boot/config/domain.cfg.

docker_watchdog: exit cleanly if Docker not enabled in Unraid settings.
stability_watchdog: skip Docker daemon check and Docker container stop
if Docker not enabled; skip virsh VM shutdown if VM Manager not enabled.
server_reboot: skip VM shutdown and libvirt stop if VM Manager not enabled.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 15:18:58 -04:00
Gmer4LfeandClaude Sonnet 4.6 9d6772fbd6 fix: correct state file ownership and function naming across watchdogs
stability_watchdog: remove SYS_WATCHDOG_FAILED_FILE from state file
touch list — skip list is owned by docker_watchdog, not stability.

docker_watchdog: rename check_system_watchdog_state →
check_resource_watchdog_state (reads RW_STATE_FILE from resource_watchdog,
not SYS_WATCHDOG_STATE_FILE from stability_watchdog). Update all comments
and state file docs to reflect correct ownership.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 14:46:18 -04:00
Gmer4LfeandClaude Sonnet 4.6 f6fb3add77 refactor: delegate Docker daemon escalation from stability to docker_watchdog
docker_watchdog now writes daemon_confirmed_down=true to its state file
when restart is attempted and daemon is still unresponsive. Clears the
flag on recovery.

stability_watchdog removes the duplicate CRITICAL daemon check (which
bypassed all strikes, all abort conditions, and had its own rc.docker
restart). Replaced with a standard strike check reading the
daemon_confirmed_down flag from docker_watchdog's state file.

Result: docker_watchdog owns all daemon restart logic. stability_watchdog
escalates to reboot only after N consecutive strike cycles confirming the
daemon is truly unrecoverable — same pattern as all other standard checks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 14:38:03 -04:00
Gmer4LfeandClaude Sonnet 4.6 9c68429e25 fix: remove container check from stability_watchdog, add USB boot warning
stability_watchdog: container health is owned by docker_watchdog — rebooting
when docker_watchdog skip-lists a container creates a reboot loop. Removed
SYS_WATCHDOG_CHECK_CONTAINERS check entirely. docker_watchdog handles
strikes, restart attempts, skip-listing, and notifications.

varaverk.plg: warn on USB/removable boot device at install time. Plugin is
designed for internal NVMe/SSD boot. Warning is non-blocking but clear.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 14:30:48 -04:00
Gmer4Lfe afd9129869 watchdogs: add echo banners to network/storage/stability
All watchdog scripts now open with a consistent echo banner:
  ━━━ <icon> <Name> Watchdog — <timestamp> ━━━

Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.

Also: network_watchdog now logs "Network healthy " on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
2026-05-24 17:20:55 -04:00
Gmer4Lfe cf21efaeea Structural reorganization: watchdog taxonomy + server_reboot integration
Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
2026-05-22 20:17:22 -04:00