Commit Graph
6 Commits
Author SHA1 Message Date
Gmer4LfeandClaude Sonnet 4.6 46b27f609a safeguard: check Docker/VM Manager enabled before acting
common.sh: add is_docker_enabled() and is_vm_manager_enabled() helpers
reading /boot/config/docker.cfg and /boot/config/domain.cfg.

docker_watchdog: exit cleanly if Docker not enabled in Unraid settings.
stability_watchdog: skip Docker daemon check and Docker container stop
if Docker not enabled; skip virsh VM shutdown if VM Manager not enabled.
server_reboot: skip VM shutdown and libvirt stop if VM Manager not enabled.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 15:18:58 -04:00
Gmer4LfeandClaude Sonnet 4.6 9d6772fbd6 fix: correct state file ownership and function naming across watchdogs
stability_watchdog: remove SYS_WATCHDOG_FAILED_FILE from state file
touch list — skip list is owned by docker_watchdog, not stability.

docker_watchdog: rename check_system_watchdog_state →
check_resource_watchdog_state (reads RW_STATE_FILE from resource_watchdog,
not SYS_WATCHDOG_STATE_FILE from stability_watchdog). Update all comments
and state file docs to reflect correct ownership.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 14:46:18 -04:00
Gmer4LfeandClaude Sonnet 4.6 f6fb3add77 refactor: delegate Docker daemon escalation from stability to docker_watchdog
docker_watchdog now writes daemon_confirmed_down=true to its state file
when restart is attempted and daemon is still unresponsive. Clears the
flag on recovery.

stability_watchdog removes the duplicate CRITICAL daemon check (which
bypassed all strikes, all abort conditions, and had its own rc.docker
restart). Replaced with a standard strike check reading the
daemon_confirmed_down flag from docker_watchdog's state file.

Result: docker_watchdog owns all daemon restart logic. stability_watchdog
escalates to reboot only after N consecutive strike cycles confirming the
daemon is truly unrecoverable — same pattern as all other standard checks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 14:38:03 -04:00
Gmer4LfeandClaude Sonnet 4.6 9c68429e25 fix: remove container check from stability_watchdog, add USB boot warning
stability_watchdog: container health is owned by docker_watchdog — rebooting
when docker_watchdog skip-lists a container creates a reboot loop. Removed
SYS_WATCHDOG_CHECK_CONTAINERS check entirely. docker_watchdog handles
strikes, restart attempts, skip-listing, and notifications.

varaverk.plg: warn on USB/removable boot device at install time. Plugin is
designed for internal NVMe/SSD boot. Warning is non-blocking but clear.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 14:30:48 -04:00
Gmer4Lfe afd9129869 watchdogs: add echo banners to network/storage/stability
All watchdog scripts now open with a consistent echo banner:
  ━━━ <icon> <Name> Watchdog — <timestamp> ━━━

Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.

Also: network_watchdog now logs "Network healthy " on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
2026-05-24 17:20:55 -04:00
Gmer4Lfe cf21efaeea Structural reorganization: watchdog taxonomy + server_reboot integration
Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
2026-05-22 20:17:22 -04:00