Commit Graph
4 Commits
Author SHA1 Message Date
Gmer4Lfe ea9f39a803 watchdogs: make healthy-state outputs always visible
log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
2026-05-24 17:25:33 -04:00
Gmer4Lfe afd9129869 watchdogs: add echo banners to network/storage/stability
All watchdog scripts now open with a consistent echo banner:
  ━━━ <icon> <Name> Watchdog — <timestamp> ━━━

Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.

Also: network_watchdog now logs "Network healthy " on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
2026-05-24 17:20:55 -04:00
Gmer4Lfe 4535804c5d Add System/network_watchdog.sh — services-layer connectivity watchdog
Checks:
  1. Internet reachability (gates all remaining checks)
  2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
  3. Tailscale status — notify only, no auto-restart
  4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart

Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle

Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
2026-05-22 20:51:22 -04:00
Gmer4Lfe cf21efaeea Structural reorganization: watchdog taxonomy + server_reboot integration
Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
2026-05-22 20:17:22 -04:00