Storage-mode awareness pass + doc update for System_Essentials through Partnership

All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.

PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.

conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.

fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.

README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
This commit is contained in:
Gmer4Lfe
2026-06-19 19:32:39 -04:00
parent 0564580605
commit bf3e7cc2c4
35 changed files with 835 additions and 489 deletions
+46 -11
View File
@@ -3,8 +3,8 @@
**Four tiers that run every 15 minutes through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → check system components →
reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
System component checks (storage, WebGUI, network) run inside a thin `system_watchdog.sh`
orchestrator that can also be run standalone.
System component checks (storage, WebGUI, network, conf cache) run inside a thin
`system_watchdog.sh` orchestrator that can also be run standalone.
---
@@ -57,6 +57,15 @@ Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kern
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
sequence so container healing and pressure reduction always get a chance first.
**Partner Conf Backup Going Stale During a Long Outage**
The partner goes offline overnight. On the next reboot (planned maintenance), the
persistent conf backup saved at shutdown is stale — it reflects the state from before
the outage, not the most recent live values. `conf_cache_restore.sh` loads it
faithfully, but fallback.sh may be working with old tier delays or container names.
Fix: `conf_cache_watchdog.sh` — refreshes the persistent backup from the RAM cache
every 15 minutes while the partner is offline, keeping it current throughout the outage.
---
## ━━━ WHAT THIS FOLDER DOES ━━━
@@ -66,13 +75,17 @@ Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
```
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
System components system_watchdog.sh — thin orchestrator: system component health
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
network_watchdog.sh — internet, DDNS sync, Tailscale, NPM proxy
conf_cache_watchdog.sh — maintain persistent partner conf backup
Last resort stability_watchdog.sh — reboot only when nothing else can recover
```
> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
> but lives in the plugin tree because it calls Unraid-specific service commands
> (nginx, php-fpm, emhttp via platform adapter). See `Plugin/unraid/Watchdogs/System/`.
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
attempts restarts — containers restarted into a pressure-bound system just fail again.
System component checks run after containers are healed. Stability watchdog runs last —
@@ -96,8 +109,16 @@ Orchestrators/
Watchdogs/System/ ← called by system_watchdog.sh each cycle
storage_watchdog.sh pool growth rate + runaway log detection
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
network_watchdog.sh internet reachability, DDNS sync, Tailscale, NPM proxy
conf_cache_watchdog.sh maintain persistent partner conf backup during outages
Plugin/unraid/Watchdogs/System/
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
(Unraid-specific platform calls — lives in plugin tree)
Plugin/unraid/System_Essentials/
unraid_api_key_renew.sh re-registers the Varaverk plugin API key each watchdog cycle
(inserted between docker_watchdog and stability_watchdog)
Tools/
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
@@ -106,6 +127,10 @@ Tools/
Docker_Essentials/
All container lifecycle scripts (daily restart, updates, network) — unaffected.
docker_watchdog.sh coordinates with them via shared state, not direct calls.
System_Essentials/
conf_cache_save.sh / conf_cache_restore.sh — at-stop/at-start bookends for the
persistent backup that conf_cache_watchdog.sh keeps current in between.
```
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
@@ -119,14 +144,15 @@ Docker_Essentials/
|--------|------|-----------|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every 15 min |
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every 15 min |
| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
| `system_watchdog.sh` | Thin orchestrator — runs SYSTEM_WATCHDOG_SCRIPTS in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every 15 min |
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every 15 min |
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every 15 min |
| `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every 15 min |
| `System/conf_cache_watchdog.sh` | Refresh persistent partner conf backup while partner is offline | `system_watchdog.sh` — every 15 min |
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
> but is not in this folder. `watchdog_orchestrator.sh` is in `Orchestrators/`.
> `watchdog_skip_list_manager.sh` is in `Tools/`. Neither is a watchdog.
---
@@ -166,7 +192,7 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
│ └─ webgui_watchdog.sh
│ └─ Plugin/unraid/Watchdogs/System/webgui_watchdog.sh
│ curl check → WebGUI responding → exit 0 (silent)
│ not responding → nginx restart → wait → recheck
│ still down → php-fpm restart → wait → recheck
@@ -179,6 +205,13 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
│ Tailscale: status Running → pass; not running = notify (no auto-restart)
│ NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
│ └─ conf_cache_watchdog.sh
│ remote online → remove persistent backup (conf_sync gets fresh on next boot)
│ remote offline → refresh backup from RAM cache → backup stays current
│ silent when remote is online and no backup exists (normal state)
│ (+ Plugin/unraid/System_Essentials/unraid_api_key_renew.sh — between steps 3 and 4)
Step 4 — stability_watchdog.sh
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
Tier 1 CRITICAL → immediate reboot (no strikes)
@@ -195,7 +228,9 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `DOCKER_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.