Scripts calling acquire_lock followed by acquire_rsync_lock (rsync.sh)
or a custom EXIT trap (resource_watchdog.sh) would overwrite the lock
release trap, leaving the first lock file behind on exit.
Replace per-acquire trap with a _LOCK_FILES registry. _register_lock()
appends each lock file to the array and sets a single _release_all_locks
trap. All acquired locks release together on exit regardless of how many
traps are subsequently set or overwritten.
resource_watchdog.sh: chain _release_all_locks into both its custom trap
and the state-persisted disarm path so the lock releases in all exit paths.
log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
All watchdog scripts now open with a consistent echo banner:
━━━ <icon> <Name> Watchdog — <timestamp> ━━━
Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.
Also: network_watchdog now logs "Network healthy ✅" on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
network_watchdog: Tailscale JSON uses "BackendState": "Running" (with space)
but grep pattern matched "BackendState":"Running" (no space) — always failed.
Switched to grep -qE with \s* to handle both forms.
system_watchdog: add echo banner at run start so its output is visually
separated from docker_watchdog in orchestrator logs. Previously appeared
as a continuation of the docker section since docker_watchdog has a banner
but system_watchdog did not.
- network_watchdog: remove ISSUES++ for Tailscale — it's notify-only by
design, no restart possible; cascading to system watchdog exit 1 was noise
- docker_watchdog: move banner to top of cycle so body errors appear under
the correct section header, not visually inside Resource Manager output
Update the Three-Tier Response and RAM Tiers sections to reflect the
resource_watchdog/stability_watchdog split: MEM_WARN_GB/MEM_SHUTDOWN_GB/
MEM_RECOVER_GB moved to RW_RAM_SOFT/MEDIUM/HARD/RECOVER_GB in resource_watchdog;
stability_watchdog now only holds SYS_WATCHDOG_MEM_GB (reboot trigger). Update
required containers reference from SYS_WATCHDOG_REQUIRED_CONTAINERS (removed)
to HOST*_WATCHDOG_REQUIRED_CONTAINERS in host*.conf.
Checks:
1. Internet reachability (gates all remaining checks)
2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
3. Tailscale status — notify only, no auto-restart
4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart
Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle
Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
watchdog_orchestrator.sh stays in Orchestrators/ — it's an orchestrator, not a watchdog.
watchdog_skip_list_manager.sh stays in Tools/ — it's a management utility.
Only the 4 watchdog scripts belong in Watchdogs/:
docker_watchdog.sh, resource_watchdog.sh, storage_watchdog.sh, system_watchdog.sh
Gmer4Lfe