log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
All watchdog scripts now open with a consistent echo banner:
━━━ <icon> <Name> Watchdog — <timestamp> ━━━
Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.
Also: network_watchdog now logs "Network healthy ✅" on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
network_watchdog: Tailscale JSON uses "BackendState": "Running" (with space)
but grep pattern matched "BackendState":"Running" (no space) — always failed.
Switched to grep -qE with \s* to handle both forms.
system_watchdog: add echo banner at run start so its output is visually
separated from docker_watchdog in orchestrator logs. Previously appeared
as a continuation of the docker section since docker_watchdog has a banner
but system_watchdog did not.
- network_watchdog: remove ISSUES++ for Tailscale — it's notify-only by
design, no restart possible; cascading to system watchdog exit 1 was noise
- docker_watchdog: move banner to top of cycle so body errors appear under
the correct section header, not visually inside Resource Manager output
Checks:
1. Internet reachability (gates all remaining checks)
2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
3. Tailscale status — notify only, no auto-restart
4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart
Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle
Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
Gmer4Lfe