Storage-mode awareness pass + doc update for System_Essentials through Partnership
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR / PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the ecosystem works in both internal and appdata storage modes. PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php, config.php): all state reads switched to STATE_DIR constant; remote state reads use the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via their varaverk.cfg before building the path. conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added _remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull. fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED, FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card. README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync, Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync, conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener, upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to $STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js, not PHP -S).
This commit is contained in:
@@ -13,8 +13,9 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
||||
- [docker_watchdog.sh](#docker_watchdogsh)
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
|
||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) ← Plugin/unraid/Watchdogs/System/
|
||||
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
|
||||
- [System/conf_cache_watchdog.sh](#systemconf_cache_watchdogsh)
|
||||
- [stability_watchdog.sh](#stability_watchdogsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
@@ -373,6 +374,10 @@ storage_watchdog.sh --log # verbose per-container output
|
||||
|
||||
## System/webgui_watchdog.sh
|
||||
|
||||
> **Lives in `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh`** — calls Unraid-specific
|
||||
> service commands (`rc.nginx`, `rc.php-fpm`, `emhttp`) via the platform adapter. Called by
|
||||
> `system_watchdog.sh` via `SYSTEM_WATCHDOG_SCRIPTS` in master.conf.
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
|
||||
through three restart steps if unresponsive. Silent when healthy.
|
||||
|
||||
@@ -446,7 +451,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
|
||||
|
||||
# host*.conf (host-specific)
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
@@ -506,6 +511,40 @@ tailscale up
|
||||
|
||||
---
|
||||
|
||||
## System/conf_cache_watchdog.sh
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Maintains the persistent partner conf backup
|
||||
at `$PERSISTENT_CONF_CACHE` while the partner is offline.
|
||||
|
||||
### Behaviour
|
||||
|
||||
**Remote online:** removes the persistent backup if one exists. It is not needed —
|
||||
`conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
|
||||
|
||||
**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to
|
||||
`$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
|
||||
an extended outage. If this host reboots while the partner is still down,
|
||||
`conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
|
||||
valid partner vars.
|
||||
|
||||
Silent when remote is online and no backup exists (the normal steady state).
|
||||
|
||||
### Gates
|
||||
|
||||
- `FALLBACK_ENABLED=false` → no-op (no fallback means no need for partner vars)
|
||||
- `CONF_SYNC_ENABLED=false` → no-op
|
||||
- `PARTNERSHIP_ENABLED=false` → exits silently (require_partnership gate)
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
conf_cache_watchdog.sh # single pass (called by system_watchdog.sh)
|
||||
conf_cache_watchdog.sh --dry-run # show what would be written or removed
|
||||
conf_cache_watchdog.sh --log # verbose output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## stability_watchdog.sh
|
||||
|
||||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||||
@@ -653,9 +692,9 @@ WATCHDOG_RESTART_DEAD=true
|
||||
WATCHDOG_RESTART_CRASHED=true
|
||||
WATCHDOG_BATCH_NOTIFY=true
|
||||
|
||||
# State files:
|
||||
WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
|
||||
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
|
||||
# State files (all paths adapt to storage mode via STATE_DIR / DATA_DIR):
|
||||
WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
|
||||
WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
|
||||
|
||||
# host*.conf
|
||||
HOST1_WATCHDOG_CONTAINERS=() # "ContainerName:LimitMB"
|
||||
@@ -670,8 +709,8 @@ WATCHDOG_APPDATA_GROWTH_GB=2
|
||||
WATCHDOG_APPDATA_LOG_MAX_GB=2
|
||||
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
|
||||
WATCHDOG_APPDATA_STRIKE_LIMIT=3
|
||||
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
|
||||
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
|
||||
WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
|
||||
STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"
|
||||
|
||||
# host*.conf (optional — only for suppress ceilings)
|
||||
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||||
@@ -701,11 +740,12 @@ SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=false
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=false
|
||||
|
||||
# State files (all in State_Files/ — survive reboots):
|
||||
SYS_WATCHDOG_STATE_FILE="State_Files/system_watchdog_state.db"
|
||||
DOCKER_WATCHDOG_FAILED_FILE="State_Files/docker_watchdog_failed.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="State_Files/system_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_OOM_FILE="State_Files/system_watchdog_oom.db"
|
||||
# State files — all in $STATE_DIR (survive reboots; adapt to storage mode):
|
||||
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
|
||||
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
|
||||
RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"
|
||||
|
||||
# ── Network Watchdog ───────────────────────────────────────────────────────────
|
||||
NETWORK_WATCHDOG_ENABLED=true
|
||||
@@ -714,7 +754,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
|
||||
|
||||
# host*.conf (host-specific)
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
@@ -781,8 +821,8 @@ storage_watchdog.sh --status
|
||||
### stability_watchdog Rebooted Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check the reboot log (survives reboots):
|
||||
cat /boot/config/system_watchdog_reboots.db
|
||||
# Check the reboot log (survives reboots — in State_Files/):
|
||||
cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db
|
||||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||||
|
||||
# Check syslog near the reboot time:
|
||||
|
||||
Reference in New Issue
Block a user