Storage-mode awareness pass + doc update for System_Essentials through Partnership

All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.

PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.

conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.

fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.

README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
This commit is contained in:
Gmer4Lfe
2026-06-19 19:32:39 -04:00
parent 0564580605
commit bf3e7cc2c4
35 changed files with 835 additions and 489 deletions
+55 -15
View File
@@ -13,8 +13,9 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
- [docker_watchdog.sh](#docker_watchdogsh)
- [system_watchdog.sh](#system_watchdogsh)
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) ← Plugin/unraid/Watchdogs/System/
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
- [System/conf_cache_watchdog.sh](#systemconf_cache_watchdogsh)
- [stability_watchdog.sh](#stability_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -373,6 +374,10 @@ storage_watchdog.sh --log # verbose per-container output
## System/webgui_watchdog.sh
> **Lives in `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh`** — calls Unraid-specific
> service commands (`rc.nginx`, `rc.php-fpm`, `emhttp`) via the platform adapter. Called by
> `system_watchdog.sh` via `SYSTEM_WATCHDOG_SCRIPTS` in master.conf.
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
through three restart steps if unresponsive. Silent when healthy.
@@ -446,7 +451,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
@@ -506,6 +511,40 @@ tailscale up
---
## System/conf_cache_watchdog.sh
Called by `system_watchdog.sh` each cycle. Maintains the persistent partner conf backup
at `$PERSISTENT_CONF_CACHE` while the partner is offline.
### Behaviour
**Remote online:** removes the persistent backup if one exists. It is not needed —
`conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to
`$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
an extended outage. If this host reboots while the partner is still down,
`conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
valid partner vars.
Silent when remote is online and no backup exists (the normal steady state).
### Gates
- `FALLBACK_ENABLED=false` → no-op (no fallback means no need for partner vars)
- `CONF_SYNC_ENABLED=false` → no-op
- `PARTNERSHIP_ENABLED=false` → exits silently (require_partnership gate)
### Usage
```bash
conf_cache_watchdog.sh # single pass (called by system_watchdog.sh)
conf_cache_watchdog.sh --dry-run # show what would be written or removed
conf_cache_watchdog.sh --log # verbose output
```
---
## stability_watchdog.sh
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
@@ -653,9 +692,9 @@ WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
WATCHDOG_BATCH_NOTIFY=true
# State files:
WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
# State files (all paths adapt to storage mode via STATE_DIR / DATA_DIR):
WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
# host*.conf
HOST1_WATCHDOG_CONTAINERS=() # "ContainerName:LimitMB"
@@ -670,8 +709,8 @@ WATCHDOG_APPDATA_GROWTH_GB=2
WATCHDOG_APPDATA_LOG_MAX_GB=2
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
WATCHDOG_APPDATA_STRIKE_LIMIT=3
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"
# host*.conf (optional — only for suppress ceilings)
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
@@ -701,11 +740,12 @@ SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=false
SYS_WATCHDOG_ABORT_ON_MOVER=false
# State files (all in State_Files/ — survive reboots):
SYS_WATCHDOG_STATE_FILE="State_Files/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="State_Files/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="State_Files/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="State_Files/system_watchdog_oom.db"
# State files all in $STATE_DIR (survive reboots; adapt to storage mode):
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"
# ── Network Watchdog ───────────────────────────────────────────────────────────
NETWORK_WATCHDOG_ENABLED=true
@@ -714,7 +754,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
@@ -781,8 +821,8 @@ storage_watchdog.sh --status
### stability_watchdog Rebooted Unexpectedly
```bash
# Check the reboot log (survives reboots):
cat /boot/config/system_watchdog_reboots.db
# Check the reboot log (survives reboots — in State_Files/):
cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot
# Check syslog near the reboot time: