Storage-mode awareness pass + doc update for System_Essentials through Partnership

All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.

PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.

conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.

fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.

README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).

Watchdogs/Manual-Watchdogs.md

@@ -13,8 +13,9 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
 - [docker_watchdog.sh](#docker_watchdogsh)
 - [system_watchdog.sh](#system_watchdogsh)
 - [System/storage_watchdog.sh](#systemstorage_watchdogsh)
-- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
+- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) ← Plugin/unraid/Watchdogs/System/
 - [System/network_watchdog.sh](#systemnetwork_watchdogsh)
+- [System/conf_cache_watchdog.sh](#systemconf_cache_watchdogsh)
 - [stability_watchdog.sh](#stability_watchdogsh)
 - [Full Configuration Reference](#full-configuration-reference)
 - [Troubleshooting](#troubleshooting)
@@ -373,6 +374,10 @@ storage_watchdog.sh --log      # verbose per-container output
 
 ## System/webgui_watchdog.sh
 
+> **Lives in `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh`** — calls Unraid-specific
+> service commands (`rc.nginx`, `rc.php-fpm`, `emhttp`) via the platform adapter. Called by
+> `system_watchdog.sh` via `SYSTEM_WATCHDOG_SCRIPTS` in master.conf.
+
 Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
 through three restart steps if unresponsive. Silent when healthy.
 
@@ -446,7 +451,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
 NETWORK_WATCHDOG_CHECK_TAILSCALE=true
 NETWORK_WATCHDOG_NPM_TIMEOUT=10
 NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
-NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
+NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
 
 # host*.conf (host-specific)
 HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
@@ -506,6 +511,40 @@ tailscale up
 
 ---
 
+## System/conf_cache_watchdog.sh
+
+Called by `system_watchdog.sh` each cycle. Maintains the persistent partner conf backup
+at `$PERSISTENT_CONF_CACHE` while the partner is offline.
+
+### Behaviour
+
+**Remote online:** removes the persistent backup if one exists. It is not needed —
+`conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
+
+**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to
+`$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
+an extended outage. If this host reboots while the partner is still down,
+`conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
+valid partner vars.
+
+Silent when remote is online and no backup exists (the normal steady state).
+
+### Gates
+
+- `FALLBACK_ENABLED=false` → no-op (no fallback means no need for partner vars)
+- `CONF_SYNC_ENABLED=false` → no-op
+- `PARTNERSHIP_ENABLED=false` → exits silently (require_partnership gate)
+
+### Usage
+
+```bash
+conf_cache_watchdog.sh              # single pass (called by system_watchdog.sh)
+conf_cache_watchdog.sh --dry-run    # show what would be written or removed
+conf_cache_watchdog.sh --log        # verbose output
+```
+
+---
+
 ## stability_watchdog.sh
 
 Runs last in the orchestrator sequence. The only script in the ecosystem authorized
@@ -653,9 +692,9 @@ WATCHDOG_RESTART_DEAD=true
 WATCHDOG_RESTART_CRASHED=true
 WATCHDOG_BATCH_NOTIFY=true
 
-# State files:
-WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
-WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
+# State files (all paths adapt to storage mode via STATE_DIR / DATA_DIR):
+WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
+WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
 
 # host*.conf
 HOST1_WATCHDOG_CONTAINERS=()          # "ContainerName:LimitMB"
@@ -670,8 +709,8 @@ WATCHDOG_APPDATA_GROWTH_GB=2
 WATCHDOG_APPDATA_LOG_MAX_GB=2
 WATCHDOG_APPDATA_TRUNCATE_LOGS=false
 WATCHDOG_APPDATA_STRIKE_LIMIT=3
-WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
-STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
+WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
+STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"
 
 # host*.conf (optional — only for suppress ceilings)
 # declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
@@ -701,11 +740,12 @@ SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
 SYS_WATCHDOG_ABORT_ON_PARITY=false
 SYS_WATCHDOG_ABORT_ON_MOVER=false
 
-# State files (all in State_Files/ — survive reboots):
-SYS_WATCHDOG_STATE_FILE="State_Files/system_watchdog_state.db"
-DOCKER_WATCHDOG_FAILED_FILE="State_Files/docker_watchdog_failed.db"
-SYS_WATCHDOG_REBOOT_LOG="State_Files/system_watchdog_reboots.db"
-SYS_WATCHDOG_OOM_FILE="State_Files/system_watchdog_oom.db"
+# State files — all in $STATE_DIR (survive reboots; adapt to storage mode):
+SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
+DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
+SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
+SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
+RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"
 
 # ── Network Watchdog ───────────────────────────────────────────────────────────
 NETWORK_WATCHDOG_ENABLED=true
@@ -714,7 +754,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
 NETWORK_WATCHDOG_CHECK_TAILSCALE=true
 NETWORK_WATCHDOG_NPM_TIMEOUT=10
 NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
-NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
+NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
 
 # host*.conf (host-specific)
 HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
@@ -781,8 +821,8 @@ storage_watchdog.sh --status
 ### stability_watchdog Rebooted Unexpectedly
 
 ```bash
-# Check the reboot log (survives reboots):
-cat /boot/config/system_watchdog_reboots.db
+# Check the reboot log (survives reboots — in State_Files/):
+cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db
 # Shows timestamp and reason for each watchdog-triggered reboot
 
 # Check syslog near the reboot time:

Watchdogs/README-Watchdogs.md

@@ -3,8 +3,8 @@
 **Four tiers that run every 15 minutes through `watchdog_orchestrator.sh`,
 each with a clear lane:** reduce system pressure → heal containers → check system components →
 reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
-System component checks (storage, WebGUI, network) run inside a thin `system_watchdog.sh`
-orchestrator that can also be run standalone.
+System component checks (storage, WebGUI, network, conf cache) run inside a thin
+`system_watchdog.sh` orchestrator that can also be run standalone.
 
 ---
 
@@ -57,6 +57,15 @@ Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kern
 health. Only script in the stack authorized to reboot. Runs last in the orchestrator
 sequence so container healing and pressure reduction always get a chance first.
 
+**Partner Conf Backup Going Stale During a Long Outage**
+The partner goes offline overnight. On the next reboot (planned maintenance), the
+persistent conf backup saved at shutdown is stale — it reflects the state from before
+the outage, not the most recent live values. `conf_cache_restore.sh` loads it
+faithfully, but fallback.sh may be working with old tier delays or container names.
+
+Fix: `conf_cache_watchdog.sh` — refreshes the persistent backup from the RAM cache
+every 15 minutes while the partner is offline, keeping it current throughout the outage.
+
 ---
 
 ## ━━━ WHAT THIS FOLDER DOES ━━━
@@ -66,13 +75,17 @@ Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
 ```
 Pressure reduction    resource_watchdog.sh   — throttle/pause/stop before healing fails
 Container healing     docker_watchdog.sh     — memory, CPU, HTTP, required containers
-System components     system_watchdog.sh     — thin orchestrator: storage + WebGUI health
+System components     system_watchdog.sh     — thin orchestrator: system component health
   └─ System/          storage_watchdog.sh    — pool growth rate + runaway log detection
-                      webgui_watchdog.sh     — WebGUI availability, nginx → php-fpm → emhttp
                       network_watchdog.sh    — internet, DDNS sync, Tailscale, NPM proxy
+                      conf_cache_watchdog.sh — maintain persistent partner conf backup
 Last resort           stability_watchdog.sh  — reboot only when nothing else can recover
 ```
 
+> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
+> but lives in the plugin tree because it calls Unraid-specific service commands
+> (nginx, php-fpm, emhttp via platform adapter). See `Plugin/unraid/Watchdogs/System/`.
+
 **The execution order is the design.** Resource pressure is reduced before docker_watchdog
 attempts restarts — containers restarted into a pressure-bound system just fail again.
 System component checks run after containers are healed. Stability watchdog runs last —
@@ -96,8 +109,16 @@ Orchestrators/
 
 Watchdogs/System/              ← called by system_watchdog.sh each cycle
   storage_watchdog.sh          pool growth rate + runaway log detection
-  webgui_watchdog.sh           WebGUI availability — nginx → php-fpm → emhttp escalation
   network_watchdog.sh          internet reachability, DDNS sync, Tailscale, NPM proxy
+  conf_cache_watchdog.sh       maintain persistent partner conf backup during outages
+
+Plugin/unraid/Watchdogs/System/
+  webgui_watchdog.sh           WebGUI availability — nginx → php-fpm → emhttp escalation
+                               (Unraid-specific platform calls — lives in plugin tree)
+
+Plugin/unraid/System_Essentials/
+  unraid_api_key_renew.sh      re-registers the Varaverk plugin API key each watchdog cycle
+                               (inserted between docker_watchdog and stability_watchdog)
 
 Tools/
   watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
@@ -106,6 +127,10 @@ Tools/
 Docker_Essentials/
   All container lifecycle scripts (daily restart, updates, network) — unaffected.
   docker_watchdog.sh coordinates with them via shared state, not direct calls.
+
+System_Essentials/
+  conf_cache_save.sh / conf_cache_restore.sh — at-stop/at-start bookends for the
+  persistent backup that conf_cache_watchdog.sh keeps current in between.
 ```
 
 **watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
@@ -119,14 +144,15 @@ Docker_Essentials/
 |--------|------|-----------|
 | `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every 15 min |
 | `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every 15 min |
-| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
+| `system_watchdog.sh` | Thin orchestrator — runs SYSTEM_WATCHDOG_SCRIPTS in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
 | `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every 15 min |
 | `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every 15 min |
-| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every 15 min |
 | `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every 15 min |
+| `System/conf_cache_watchdog.sh` | Refresh persistent partner conf backup while partner is offline | `system_watchdog.sh` — every 15 min |
 
-> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
-> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
+> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
+> but is not in this folder. `watchdog_orchestrator.sh` is in `Orchestrators/`.
+> `watchdog_skip_list_manager.sh` is in `Tools/`. Neither is a watchdog.
 
 ---
 
@@ -166,7 +192,7 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
     │       log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
     │         oversize log found → 3-strike warn → truncate (if enabled) or alert
     │
-    │  └─ webgui_watchdog.sh
+    │  └─ Plugin/unraid/Watchdogs/System/webgui_watchdog.sh
     │       curl check → WebGUI responding → exit 0 (silent)
     │       not responding → nginx restart → wait → recheck
     │       still down    → php-fpm restart → wait → recheck
@@ -179,6 +205,13 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
     │       Tailscale: status Running → pass; not running = notify (no auto-restart)
     │       NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
     │
+    │  └─ conf_cache_watchdog.sh
+    │       remote online  → remove persistent backup (conf_sync gets fresh on next boot)
+    │       remote offline → refresh backup from RAM cache → backup stays current
+    │       silent when remote is online and no backup exists (normal state)
+    │
+    │  (+ Plugin/unraid/System_Essentials/unraid_api_key_renew.sh — between steps 3 and 4)
+    │
   Step 4 — stability_watchdog.sh
        checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
        Tier 1 CRITICAL  → immediate reboot (no strikes)
@@ -195,7 +228,9 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
 | `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
 | `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
 | `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
-| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
+| `DOCKER_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
 | `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
 | `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
 | `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
+
+All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.

Watchdogs/System/conf_cache_watchdog.sh

@@ -5,7 +5,7 @@
 #
 # PURPOSE
 # ─────────────────────────────────────────────────────────────────────────────
-# Maintains the persistent partner conf backup at /boot/config/.cache/vv/d/.
+# Maintains the persistent partner conf backup at $PERSISTENT_CONF_CACHE.
 # Runs every 15 minutes via SYSTEM_WATCHDOG_SCRIPTS.
 #
 # When remote is OFFLINE:
@@ -33,7 +33,7 @@ require_partnership
 [[ -z "${REMOTE_ID:-}"         ]]            && exit 0
 
 RAM_CACHE="/tmp/.cache/vv/d"
-SAVE_DIR="/boot/config/.cache/vv/d"
+SAVE_DIR="$PERSISTENT_CONF_CACHE"
 
 if ping_remote; then
     if [[ -d "$SAVE_DIR" ]]; then