Compare commits

..
8 Commits
Author SHA1 Message Date
Gmer4Lfe bf3e7cc2c4 Storage-mode awareness pass + doc update for System_Essentials through Partnership
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.

PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.

conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.

fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.

README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
2026-06-19 19:32:39 -04:00
Gmer4Lfe 0564580605 Make PARTNERSHIP_ENABLED the authoritative gate for all cross-server operations
Adds require_partnership() to common.sh — exits cleanly when PARTNERSHIP_ENABLED=false.
Removes FALLBACK_PARTNERSHIP_REQUIRED toggle — partnership is now always required,
not optional. Cross-server scripts (rsync, conf sync, fallback, arr sync, play state,
backup verify) all call require_partnership after detect_hosts.
2026-06-19 18:27:41 -04:00
Gmer4Lfe 7a12c3eee6 Fix watchdog orchestrator schedule comment — every 15 min not every minute 2026-06-19 18:08:29 -04:00
Gmer4Lfe 82810ab168 Add conf_cache_watchdog.sh — watchdog-driven persistent conf backup
Writes partner confs from RAM cache to /boot/config/.cache/vv/d/ while
remote is offline, and removes the backup when remote comes back. Called
each minute via SYSTEM_WATCHDOG_SCRIPTS so crashes and power loss are
covered — not just graceful shutdowns.
2026-06-19 18:06:31 -04:00
Gmer4Lfe 4749e5857c Relocate conf cache to more discrete paths 2026-06-19 17:56:17 -04:00
Gmer4Lfe 28d434b76c Propagate new fallback model to all remaining script references
All FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER* references updated to
FALLBACK_${REMOTE_ID}_TIER* across fallback_test.sh, partnership_manager.sh,
docker_update.sh, mesh_monitor.sh, and monitor.php. mesh_monitor.sh drops
the inner covering-host loop — tier data now lives in the covered host's own
conf so no cross-host scan is needed. monitor.php reads from the covered
host's conf file rather than the local host's.
2026-06-19 17:49:26 -04:00
Gmer4Lfe b27b2d62f5 Flip fallback coverage model: each host defines its own recovery profile
Previously HOST1 defined what it would run for HOST2 (FALLBACK_HOST1_COVERS_HOST2_TIER*).
Now each host defines what it wants run when IT goes down (FALLBACK_HOST1_TIER*), and the
covering server reads the down host's conf via the RAM/persistent cache.

get_tier_containers() reads FALLBACK_${REMOTE_ID}_TIER* instead of
FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*. Tier data migrated to the correct host confs.
Writeback paths and delays were already REMOTE_ID-based — no change needed there.
2026-06-19 17:44:53 -04:00
Gmer4Lfe 7332eb81e0 Add persistent partner conf backup across reboots
conf_cache_save.sh runs first on array stop — snapshots partner confs from RAM
cache to /boot/config/varaverk/conf_bak/ before anything else shuts down.

conf_cache_restore.sh runs after conf_sync.sh on array start — if partner was
unreachable and RAM cache is incomplete, loads the backup into RAM then removes
it. Normal reboots: backup written, fresh pull succeeds, backup deleted unused.
Edge case (partner down at boot): backup fills the gap so fallback.sh has the
partner vars it needs to operate correctly.
2026-06-19 17:40:46 -04:00
45 changed files with 1181 additions and 627 deletions
+1
View File
@@ -10,6 +10,7 @@ Configurations/*.bak
# ── Runtime state, data, logs ─────────────────────────────────────────────────
data/
State_Files/
.cache/
*.log
*.lock
+8 -6
View File
@@ -123,6 +123,7 @@
# Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no /boot/config root.
DATA_DIR="${SCRIPTS_DIR}/data"
STATE_DIR="${SCRIPTS_DIR}/State_Files"
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
# ── Version Parity ──
# Controls behaviour when local and remote unRAID versions differ.
@@ -295,7 +296,8 @@
# watchdog_orchestrator.sh — NOT launched here.
ARRAY_START_SCRIPTS=(
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # re-register Varaverk API key — registry is ephemeral
"System_Essentials/conf_sync.sh" # pull partner confs + push own conf into /tmp/.vv/ RAM cache
"System_Essentials/conf_sync.sh" # pull partner confs + push own conf into /tmp/.cache/vv/d/ RAM cache
"System_Essentials/conf_cache_restore.sh" # load partner confs from persistent backup if conf_sync couldn't reach partner
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
"System_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
"Plugin/unraid/System_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
@@ -311,6 +313,7 @@
# Run sequentially (foreground) — each must complete before the next starts.
# Order matters: user scripts first (prevents new ops), then data movement, then containers.
ARRAY_STOP_SCRIPTS=(
"System_Essentials/conf_cache_save.sh" # snapshot partner conf RAM cache → /boot before anything stops
"Plugin/unraid/System_Essentials/user_scripts_stop.sh" # stop background scripts before they start new ops
"Fallback/fallback.sh --stop" # gracefully stop fallback (not caught by user_scripts_stop)
"System_Essentials/rsync_stop.sh --rsync-only" # kill rsync; skip container recovery (handled below)
@@ -320,7 +323,7 @@
# ━━━ Watchdog Orchestrator ━━━
# watchdog_orchestrator.sh runs WATCHDOG_ORCHESTRATOR_SCRIPTS in order each cron cycle.
# Schedule: * * * * * (every minute)
# Schedule: */15 * * * * (every 15 minutes)
# NOT in ARRAY_START_SCRIPTS — has its own cron entry.
# Order matters — resource first (frees pressure), docker second (heals with freed resources),
# system third (storage + webgui component health), stability last (last line of defense).
@@ -336,9 +339,10 @@
# system_watchdog.sh runs SYSTEM_WATCHDOG_SCRIPTS sequentially each cycle.
# Called by watchdog_orchestrator.sh — not scheduled directly.
SYSTEM_WATCHDOG_SCRIPTS=(
"Watchdogs/System/storage_watchdog.sh" # pool growth + runaway log detection
"Watchdogs/System/storage_watchdog.sh" # pool growth + runaway log detection
"Plugin/unraid/Watchdogs/System/webgui_watchdog.sh" # WebGUI availability — nginx → php-fpm → emhttp
"Watchdogs/System/network_watchdog.sh" # internet, DDNS, Tailscale, NPM proxy
"Watchdogs/System/network_watchdog.sh" # internet, DDNS, Tailscale, NPM proxy
"Watchdogs/System/conf_cache_watchdog.sh" # maintain persistent conf backup while partner is offline
)
# ━━━ Critical Sync Maintenance ━━━
@@ -684,8 +688,6 @@
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
FALLBACK_ENABLED=true # HOST2 back online
# false = suppresses "not running" warnings in status scripts
FALLBACK_PARTNERSHIP_REQUIRED=true # gate fallback on an active partnership
# false = standalone mode, no partnership dependency
FALLBACK_PARTNERSHIP_SUSPEND_AFTER=120 # minutes without active partnership before suspending
# 0 = suspend immediately when partnership goes inactive
+1 -1
View File
@@ -10,7 +10,7 @@
# EMPTY fields, never overwrites existing values unless --overwrite is passed.
#
# After populating, pushes the updated conf to all partners via conf_sync.sh
# so they have the fresh keys in their /tmp/.vv/ cache immediately.
# so they have the fresh keys in their /tmp/.cache/vv/d/ cache immediately.
#
# ==============================================================================================
# AUTO-DETECTED FIELDS
+12 -12
View File
@@ -40,8 +40,8 @@
# ── FALLBACK ───────────────────────────────────────────────────────────────────────────────
# DDNS DDNS containers managed by this host
# INTERNET LOSS containers stopped when internet is lost
# FALLBACK TIERS what this host runs for the remote per tier
# TIER DELAYS delays before each tier activates
# FALLBACK TIERS what this host wants the partner to run when this host is down
# TIER DELAYS how long this host must be down before each tier activates
# RSYNC WRITEBACK appdata synced back on handback
#
# ── DOCKER ─────────────────────────────────────────────────────────────────────────────────
@@ -255,27 +255,27 @@
# "MyServer.com"
)
# ━━━ Fallback Tiers — HOSTN Runs for Partner ━━━
# Containers this host starts when the partner goes down.
# Replace REMOTE_ID below with the actual remote host ID (HOST1, HOST2, etc.)
FALLBACK_HOSTN_COVERS_REMOTE_ID_TIER1=(
# "Partner-DDNS-Container"
# ━━━ Fallback Tiers — What HOSTN Wants Covered When Down ━━━
# Containers the partner starts for HOSTN when HOSTN goes down.
# Tier 1 is always immediate. Higher tiers activate after HOSTN_TIER*_DELAY minutes.
FALLBACK_HOSTN_TIER1=(
# "MyServer-DDNS-Container"
)
FALLBACK_HOSTN_COVERS_REMOTE_ID_TIER2=(
FALLBACK_HOSTN_TIER2=(
# "container-placeholder"
)
FALLBACK_HOSTN_COVERS_REMOTE_ID_TIER3=(
FALLBACK_HOSTN_TIER3=(
# "container-placeholder"
)
FALLBACK_HOSTN_COVERS_REMOTE_ID_TIER4=(
FALLBACK_HOSTN_TIER4=(
# "container-placeholder"
)
# ━━━ Tier Delays — This Host's Outage Timers ━━━
# How long THIS host must be down before each tier activates on the partner.
# ━━━ Tier Delays — HOSTN Outage Timers ━━━
# How long HOSTN must be down before each tier activates on the partner.
HOSTN_TIER2_DELAY=240 # 4 hours
HOSTN_TIER3_DELAY=720 # 12 hours
HOSTN_TIER4_DELAY=1440 # 24 hours
+9 -6
View File
@@ -120,6 +120,7 @@
# Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no /boot/config root.
DATA_DIR="/boot/config/plugins/varaverk/data"
STATE_DIR="/boot/config/plugins/varaverk/State_Files"
PERSISTENT_CONF_CACHE="/boot/config/plugins/varaverk/.cache/vv/d"
# ── Version Parity ──
# Controls behaviour when local and remote unRAID versions differ.
@@ -292,6 +293,8 @@
# Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via
# watchdog_orchestrator.sh — NOT launched here.
ARRAY_START_SCRIPTS=(
"System_Essentials/conf_sync.sh" # pull partner confs + push own conf into /tmp/.vv/ RAM cache
"System_Essentials/conf_cache_restore.sh" # load partner confs from persistent backup if conf_sync couldn't reach partner
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
@@ -306,6 +309,7 @@
# Run sequentially (foreground) — each must complete before the next starts.
# Order matters: user scripts first (prevents new ops), then data movement, then containers.
ARRAY_STOP_SCRIPTS=(
"System_Essentials/conf_cache_save.sh" # snapshot partner conf RAM cache → /boot before anything stops
"unRAID_Essentials/user_scripts_stop.sh" # stop background scripts before they start new ops
"Fallback/fallback.sh --stop" # gracefully stop fallback (not caught by user_scripts_stop)
"unRAID_Essentials/rsync_stop.sh --rsync-only" # kill rsync; skip container recovery (handled below)
@@ -315,7 +319,7 @@
# ━━━ Watchdog Orchestrator ━━━
# watchdog_orchestrator.sh runs WATCHDOG_ORCHESTRATOR_SCRIPTS in order each cron cycle.
# Schedule: * * * * * (every minute)
# Schedule: */15 * * * * (every 15 minutes)
# NOT in ARRAY_START_SCRIPTS — has its own cron entry.
# Order matters — resource first (frees pressure), docker second (heals with freed resources),
# system third (storage + webgui component health), stability last (last line of defense).
@@ -330,9 +334,10 @@
# system_watchdog.sh runs SYSTEM_WATCHDOG_SCRIPTS sequentially each cycle.
# Called by watchdog_orchestrator.sh — not scheduled directly.
SYSTEM_WATCHDOG_SCRIPTS=(
"Watchdogs/System/storage_watchdog.sh" # pool growth + runaway log detection
"Watchdogs/System/webgui_watchdog.sh" # WebGUI availability — nginx → php-fpm → emhttp
"Watchdogs/System/network_watchdog.sh" # internet, DDNS, Tailscale, NPM proxy
"Watchdogs/System/storage_watchdog.sh" # pool growth + runaway log detection
"Watchdogs/System/webgui_watchdog.sh" # WebGUI availability — nginx → php-fpm → emhttp
"Watchdogs/System/network_watchdog.sh" # internet, DDNS, Tailscale, NPM proxy
"Watchdogs/System/conf_cache_watchdog.sh" # maintain persistent conf backup while partner is offline
)
# ━━━ Critical Sync Maintenance ━━━
@@ -678,8 +683,6 @@
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
FALLBACK_ENABLED=false # set true once both servers are configured and paired
# false = suppresses "not running" warnings in status scripts
FALLBACK_PARTNERSHIP_REQUIRED=true # gate fallback on an active partnership
# false = standalone mode, no partnership dependency
FALLBACK_PARTNERSHIP_SUSPEND_AFTER=120 # minutes without active partnership before suspending
# 0 = suspend immediately when partnership goes inactive
@@ -139,7 +139,7 @@ happens after an update, and you want to know when it does.
A container keeps appearing in a broken state. You SSH in and see it's stopped. You
don't know if the watchdog tried to restart it and failed, gave up and skip-listed it,
is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip
list file on `/boot/config/`, check the restart history file, check the state file —
list file in `$STATE_DIR`, check the restart history file, check the state file —
none of which have obvious formats.
The fix: `watchdog_skip_list_manager.sh` (in `Tools/`). One command to see exactly what's
+1 -1
View File
@@ -219,7 +219,7 @@ if [[ "$REMAINDER_MODE" == true ]]; then
# Updating them here risks version divergence: if remote's writeback after handback
# encounters data written by a newer version, it may not handle it correctly.
for _tier in 1 2 3 4; do
_tier_var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${_tier}"
_tier_var="FALLBACK_${REMOTE_ID}_TIER${_tier}"
eval "_tier_arr=(\"\${${_tier_var}[@]:-}\")" 2>/dev/null
for _c in "${_tier_arr[@]}"; do
[[ -n "$_c" ]] && _exclude["$_c"]=1
+22 -16
View File
@@ -37,11 +37,11 @@ up required before handback sequence starts. **(default: 3)**
---
```
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
```
Path to the persistent state file. Lives on `/boot/` intentionally — survives reboots.
If the server was in FALLBACK state when it rebooted, it resumes FALLBACK on restart
rather than assuming everything is normal.
Path to the persistent state file. In `$STATE_DIR` — survives reboots whether storage
mode is internal (boot device) or appdata (array). If the server was in FALLBACK state
when it rebooted, it resumes FALLBACK on restart rather than assuming everything is normal.
---
@@ -172,7 +172,7 @@ daily_sync_maintenance.sh uses, in the opposite direction. No separate TIER4 lis
FALLBACK_ENABLED=true
FALLBACK_CHECK_INTERVAL=30
FALLBACK_HANDBACK_STRIKES=3
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
FALLBACK_RSYNC_ENABLED=true
EXTERNAL_IP=8.8.8.8
FALLBACK_TEST_BLOCK_WAIT=60
@@ -293,7 +293,10 @@ FALLBACK_HOST2_WRITEBACK_TIER1=(
## ━━━ STATE FILE REFERENCE ━━━
Location: `/boot/config/fallback_state.db` (survives reboots)
Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata)
> In a shell where load_config.sh is not sourced, use the full path:
> `/boot/config/plugins/varaverk/State_Files/fallback_state.db` (internal storage mode)
```
state=NORMAL # NORMAL | FALLBACK | NO_INTERNET | DARK
@@ -304,8 +307,8 @@ tier3_started=false # whether Tier 3 containers started
tier4_started=false # whether Tier 4 containers started
```
View state: `cat /boot/config/fallback_state.db`
Check state: `fallback.sh --status`
View state: `fallback.sh --status` (preferred — parsed output)
Raw file: `cat "$STATE_DIR/fallback_state.db"` (requires STATE_DIR set, or use full path)
The file is managed exclusively by fallback.sh. Do not edit it while fallback.sh is
running — the next cycle will overwrite your changes. Use the Manual State Reset
@@ -384,7 +387,7 @@ servers must be running it continuously for mutual coverage.
pgrep -f "fallback.sh"
# Check the state file
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
```
Start via User Scripts plugin on both servers.
@@ -405,7 +408,7 @@ in smart mode — a non-NORMAL state at digest time needs attention.
```bash
fallback.sh --status # full state snapshot
cat /boot/config/fallback_state.db # raw state file
cat "$STATE_DIR/fallback_state.db" # raw state file
```
---
@@ -426,7 +429,7 @@ fallback_test.sh
# Step 3 — check state after test completes
fallback.sh --status
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
```
If the test doesn't complete cleanly, the state file may be left in FALLBACK. The
@@ -459,11 +462,14 @@ ping -c 5 [remote-tailscale-ip]
**Stop fallback.sh first (via User Scripts Abort), then reset:**
```bash
# Set STATE_DIR (or source load_config.sh to get it from the environment)
source /boot/config/plugins/varaverk/load_config.sh
# View current state
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
# Write a clean NORMAL state
cat > /boot/config/fallback_state.db << 'EOF'
cat > "$STATE_DIR/fallback_state.db" << 'EOF'
state=NORMAL
fallback_start=0
handback_strikes=0
@@ -473,7 +479,7 @@ tier4_started=false
EOF
# Verify the write
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
```
Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its next cycle.
@@ -510,7 +516,7 @@ Can this server reach the remote Tailscale IP?
What does fallback.sh report?
→ fallback.sh --status
→ cat /boot/config/fallback_state.db
→ cat "$STATE_DIR/fallback_state.db"
```
### Handback Not Completing
@@ -559,7 +565,7 @@ If it has happened:
3. Understand the state before resetting
fallback.sh --status
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
4. Perform Manual State Reset above on the server in a bad state
+78 -87
View File
@@ -75,10 +75,10 @@
# failed handback: containers stop on the covering server, remote goes down
# again mid-rsync. Strikes prevent this.
#
# MY_ID-Based Routing
# All tier arrays, DDNS lists, and writeback paths are selected via MY_ID
# set by detect_hosts() — not hostname string comparison. The same script
# and same config file handle both directions symmetrically.
# REMOTE_ID-Based Coverage Config
# Tier arrays, writeback paths, and delays are all defined in the covered
# host's own conf and read via REMOTE_ID — not MY_ID. Each host owns its
# complete recovery profile. The same script handles both directions.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
@@ -116,12 +116,11 @@
# Exits if the host cannot be identified — prevents running on an unknown machine.
#
# Partnership Gate
# When FALLBACK_PARTNERSHIP_REQUIRED=true, fallback is only permitted when the
# local partnership DB reports an active partnership. If partnership goes inactive,
# a grace timer starts. After FALLBACK_PARTNERSHIP_SUSPEND_AFTER minutes the loop
# suspends: no new FALLBACK is entered, and any live fallback containers are
# stopped cleanly. Resumes automatically when partnership becomes active again.
# Set FALLBACK_PARTNERSHIP_REQUIRED=false to run fallback without a partnership.
# Fallback is only permitted when the local partnership DB reports an active
# partnership. If partnership goes inactive, a grace timer starts. After
# FALLBACK_PARTNERSHIP_SUSPEND_AFTER minutes the loop suspends: no new FALLBACK
# is entered, and any live fallback containers are stopped cleanly. Resumes
# automatically when partnership becomes active again.
#
# Silent by Default
# State transitions: warn() — always visible. Healthy routine cycles: log()
@@ -131,11 +130,13 @@
# STATE FILES
# ==============================================================================================
#
# FALLBACK_STATE_FILE — /boot/config/fallback_state.db (survives reboots)
# FALLBACK_STATE_FILE — $STATE_DIR/fallback_state.db (survives reboots)
# Keys: state, fallback_start, handback_strikes, tier2_started,
# tier3_started, tier4_started, partnership_suspended, partner_lost_at.
# Lives on /boot/ intentionally — if the server was in FALLBACK when it
# rebooted, it resumes FALLBACK on restart.
# Survives reboots — $STATE_DIR is on the boot device (internal) or appdata
# (flash). Either way the array is up before this script runs, so the file
# is always accessible. If the server was in FALLBACK when it rebooted,
# it resumes FALLBACK on restart.
#
# ==============================================================================================
# CONFIGURATION
@@ -150,9 +151,10 @@
# FALLBACK_HOST*_STOP_ON_NO_NET
# Containers stopped when this host loses internet
#
# FALLBACK_HOST*_COVERS_HOST*_TIER14
# Containers this host starts for the remote when remote is down, by tier.
# Variable pattern: FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}
# FALLBACK_HOST*_TIER14
# Containers started on this host when the partner is down — defined in the
# partner's own conf (what the partner wants covered). Read via REMOTE_ID.
# Variable pattern: FALLBACK_${REMOTE_ID}_TIER${N}
#
# HOST*_TIER2_DELAY / TIER3_DELAY / TIER4_DELAY
# Minutes after FALLBACK entry before activating each tier.
@@ -173,11 +175,6 @@
# FALLBACK_ENABLED
# false = exit cleanly (e.g. remote server being rebuilt). (default: false)
#
# FALLBACK_PARTNERSHIP_REQUIRED
# true = fallback is gated on an active partnership — suspends when no active
# partner is found after FALLBACK_PARTNERSHIP_SUSPEND_AFTER minutes.
# false = fallback runs standalone regardless of partnership state. (default: true)
#
# FALLBACK_PARTNERSHIP_SUSPEND_AFTER
# Minutes without an active partnership before fallback suspends itself.
# Grace period: fallback continues normally during this window.
@@ -190,7 +187,7 @@
# Consecutive remote-up checks required before handback begins. (default: 3)
#
# FALLBACK_STATE_FILE
# State file path — /boot/config/fallback_state.db — survives reboots.
# State file path — $STATE_DIR/fallback_state.db — survives reboots.
#
# FALLBACK_RSYNC_ENABLED
# Gate for writeback rsync jobs during handback. (default: true)
@@ -289,6 +286,7 @@ if ! command -v docker &>/dev/null; then
fi
detect_hosts
require_partnership
resolve_remote_ip
# Validate unRAID notify script — used throughout for state change notifications
@@ -300,13 +298,13 @@ DOCKER_TIMEOUT=15
SSH_TIMEOUT=10
CONTAINER_VERIFY_WAIT=5 # seconds after start before verifying container is up
log "$ICON_GEAR Config: check-interval=${FALLBACK_CHECK_INTERVAL}s handback-strikes=${FALLBACK_HANDBACK_STRIKES} partnership-required=${FALLBACK_PARTNERSHIP_REQUIRED:-true} rsync-enabled=${FALLBACK_RSYNC_ENABLED:-true}"
log "$ICON_GEAR Config: check-interval=${FALLBACK_CHECK_INTERVAL}s handback-strikes=${FALLBACK_HANDBACK_STRIKES} rsync-enabled=${FALLBACK_RSYNC_ENABLED:-true}"
log "$ICON_GEAR Timeouts: docker=${DOCKER_TIMEOUT}s ssh=${SSH_TIMEOUT}s verify-wait=${CONTAINER_VERIFY_WAIT}s"
# ==============================================================================================
# ── STATE FILE HELPERS ────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# State file on /boot/config — survives reboots.
# State file at $FALLBACK_STATE_FILE ($STATE_DIR/fallback_state.db) — survives reboots.
# Format: key=value one per line.
# Keys: state, fallback_start, handback_strikes, tier2_started, tier3_started, tier4_started
@@ -537,8 +535,7 @@ remote_ddns_stop() {
get_tier_containers() {
local tier="$1"
local remote_id="${REMOTE_ID}"
local var_name="FALLBACK_${MY_ID}_COVERS_${remote_id}_TIER${tier}"
local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}"
eval "echo \"\${${var_name}[@]:-}\""
}
@@ -627,19 +624,15 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo "$ICON_GEAR Enabled: $FALLBACK_ENABLED"
echo "$ICON_GEAR Dry Run: $DRY_RUN"
if [[ "${FALLBACK_PARTNERSHIP_REQUIRED:-true}" == "true" ]]; then
SUSPENDED=$(state_get partnership_suspended)
PLOST=$(state_get partner_lost_at)
if [[ "$SUSPENDED" == "true" ]]; then
echo "$ICON_GEAR Partnership: SUSPENDED (no active partnership)"
elif [[ -n "$PLOST" && "$PLOST" != "0" ]]; then
PLOST_MIN=$(( ($(date +%s) - PLOST) / 60 ))
echo "$ICON_GEAR Partnership: GRACE (${PLOST_MIN}min / ${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}min until suspend)"
else
echo "$ICON_GEAR Partnership: ACTIVE"
fi
SUSPENDED=$(state_get partnership_suspended)
PLOST=$(state_get partner_lost_at)
if [[ "$SUSPENDED" == "true" ]]; then
echo "$ICON_GEAR Partnership: SUSPENDED (no active partnership)"
elif [[ -n "$PLOST" && "$PLOST" != "0" ]]; then
PLOST_MIN=$(( ($(date +%s) - PLOST) / 60 ))
echo "$ICON_GEAR Partnership: GRACE (${PLOST_MIN}min / ${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}min until suspend)"
else
echo "$ICON_GEAR Partnership: REQUIRED=false (standalone mode)"
echo "$ICON_GEAR Partnership: ACTIVE"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━"
@@ -862,57 +855,55 @@ while [[ "$FALLBACK_RUNNING" == true ]]; do
log "Remote: $REMOTE_UP | Internet: $INTERNET_UP | State: $CURRENT_STATE"
# ── Partnership gate ──────────────────────────────────────────────────────────────────────
# When FALLBACK_PARTNERSHIP_REQUIRED=true, a fallback without an active partner is
# meaningless — who are we covering for? The gate suspends the loop after a grace
# period and aborts any live fallback containers. Clears automatically when partnership
# is restored.
if [[ "${FALLBACK_PARTNERSHIP_REQUIRED:-true}" == "true" ]]; then
if check_partnership_active; then
# Partnership healthy — clear any accumulated grace state
if [[ "$(state_get partnership_suspended)" == "true" ]]; then
warn "Active partnership restored — re-enabling fallback monitoring"
state_set partnership_suspended "false"
state_set partner_lost_at "0"
notify "Fallback re-enabled on $(hostname) — active partnership restored" \
"Fallback" "normal"
elif [[ "$(state_get partner_lost_at)" != "0" ]]; then
log "Partnership recovered within grace window — resetting timer"
state_set partner_lost_at "0"
# ── Partnership Gate ──────────────────────────────────────────────────────────────────────
# A fallback without an active partner is meaningless — who are we covering for?
# The gate suspends the loop after a grace period and aborts any live fallback
# containers. Clears automatically when partnership becomes active again.
if check_partnership_active; then
# Partnership healthy — clear any accumulated grace state
if [[ "$(state_get partnership_suspended)" == "true" ]]; then
warn "Active partnership restored — re-enabling fallback monitoring"
state_set partnership_suspended "false"
state_set partner_lost_at "0"
notify "Fallback re-enabled on $(hostname) — active partnership restored" \
"Fallback" "normal"
elif [[ "$(state_get partner_lost_at)" != "0" ]]; then
log "Partnership recovered within grace window — resetting timer"
state_set partner_lost_at "0"
fi
else
PARTNER_LOST_AT=$(state_get partner_lost_at)
if [[ -z "$PARTNER_LOST_AT" || "$PARTNER_LOST_AT" == "0" ]]; then
warn "No active partnership detected — grace timer started (${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}min before suspend)"
state_set partner_lost_at "$NOW"
PARTNER_LOST_AT=$NOW
fi
PARTNER_LOST_MIN=$(( (NOW - PARTNER_LOST_AT) / 60 ))
SUSPEND_AFTER=${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}
if [[ "$(state_get partnership_suspended)" == "true" || "$PARTNER_LOST_MIN" -ge "$SUSPEND_AFTER" ]]; then
# Suspend — abort any live fallback containers, then skip this cycle entirely
if [[ "$(state_get partnership_suspended)" != "true" ]]; then
warn "No active partnership for ${PARTNER_LOST_MIN}min — suspending fallback"
state_set partnership_suspended "true"
notify "Fallback suspended on $(hostname) — no active partnership for ${PARTNER_LOST_MIN}min" \
"Fallback" "warning"
fi
if [[ "$CURRENT_STATE" == "FALLBACK" ]]; then
warn "Aborting in-progress fallback — partnership suspended"
_abort_fallback_containers
state_set state "NORMAL"
state_set fallback_start "0"
state_set handback_strikes "0"
notify "Fallback aborted on $(hostname) — no active partnership, containers stopped" \
"Fallback" "warning"
fi
log "Partnership suspended — skipping fallback cycle"
sleep "$FALLBACK_CHECK_INTERVAL" & wait $!
continue
else
PARTNER_LOST_AT=$(state_get partner_lost_at)
if [[ -z "$PARTNER_LOST_AT" || "$PARTNER_LOST_AT" == "0" ]]; then
warn "No active partnership detected — grace timer started (${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}min before suspend)"
state_set partner_lost_at "$NOW"
PARTNER_LOST_AT=$NOW
fi
PARTNER_LOST_MIN=$(( (NOW - PARTNER_LOST_AT) / 60 ))
SUSPEND_AFTER=${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}
if [[ "$(state_get partnership_suspended)" == "true" || "$PARTNER_LOST_MIN" -ge "$SUSPEND_AFTER" ]]; then
# Suspend — abort any live fallback containers, then skip this cycle entirely
if [[ "$(state_get partnership_suspended)" != "true" ]]; then
warn "No active partnership for ${PARTNER_LOST_MIN}min — suspending fallback"
state_set partnership_suspended "true"
notify "Fallback suspended on $(hostname) — no active partnership for ${PARTNER_LOST_MIN}min" \
"Fallback" "warning"
fi
if [[ "$CURRENT_STATE" == "FALLBACK" ]]; then
warn "Aborting in-progress fallback — partnership suspended"
_abort_fallback_containers
state_set state "NORMAL"
state_set fallback_start "0"
state_set handback_strikes "0"
notify "Fallback aborted on $(hostname) — no active partnership, containers stopped" \
"Fallback" "warning"
fi
log "Partnership suspended — skipping fallback cycle"
sleep "$FALLBACK_CHECK_INTERVAL" & wait $!
continue
else
log "No active partnership — ${PARTNER_LOST_MIN}min elapsed / ${SUSPEND_AFTER}min until suspend"
fi
log "No active partnership — ${PARTNER_LOST_MIN}min elapsed / ${SUSPEND_AFTER}min until suspend"
fi
fi
+4 -4
View File
@@ -206,7 +206,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
fi
# Show Tier 1 containers for this host
TIER1_VAR="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1"
TIER1_VAR="FALLBACK_${REMOTE_ID}_TIER1"
eval "TIER1_CONTAINERS=(\"\${${TIER1_VAR}[@]:-}\")"
echo "$ICON_CONTAINERS Tier 1 to test: ${TIER1_CONTAINERS[*]:-none configured}"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
@@ -223,8 +223,8 @@ TOTAL_START=$(date +%s)
phase_pass() { PHASES_PASS+=("$1"); warn "$ICON_DONE Phase: $1 — PASSED ✅"; }
phase_fail() { PHASES_FAIL+=("$1"); error "Phase: $1 — FAILED ❌"; }
# Get Tier 1 containers for this server's fallback responsibility
TIER1_VAR="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1"
# Get Tier 1 containers defined in remote's own conf
TIER1_VAR="FALLBACK_${REMOTE_ID}_TIER1"
eval "TIER1_CONTAINERS=(\"\${${TIER1_VAR}[@]:-}\")"
# ==============================================================================================
@@ -286,7 +286,7 @@ fi
# Tier 1 containers configured
if [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then
error "No Tier 1 containers configured for $MY_ID$REMOTE_ID"
error "Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in host*.conf"
error "Check FALLBACK_${REMOTE_ID}_TIER1 in ${REMOTE_ID,,}.conf"
phase_fail "Pre-flight"
exit 1
fi
+3 -3
View File
@@ -409,7 +409,7 @@ pgrep -a -f fallback.sh
Check current state:
```bash
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
# state=NORMAL — both servers up
```
@@ -510,7 +510,7 @@ bash Orchestrators/daily_sync_maintenance.sh --dry-run --log
### ── Check fallback state ────────────────────────────────────────────────────
```bash
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
# Expected:
# state=NORMAL
# fallback_start=0
@@ -600,7 +600,7 @@ The remote server is unreachable on the first check. Common causes:
```bash
# Confirm which state fallback is in
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
# If stuck in FALLBACK after remote comes back: reset state
bash Tools/fallback_state_reset.sh
```
+45 -4
View File
@@ -132,8 +132,8 @@ LIDARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan
LIDARR_VERSION_MAJOR=3 # expected Lidarr major version (API safety check)
LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc")
LIDARR_TRACKED_COUNT_FILE=/boot/config/lidarr_tracked_count # persistent baseline
ARR_CLEANUP_STATS=/boot/config/arr_cleanup_stats.db # read by coffee report
LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count" # persistent baseline
ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db" # read by coffee report
```
---
@@ -171,7 +171,7 @@ RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.a
```bash
ARR_SYNC_ENABLED=true
ARR_SYNC_BLOCKLIST=/boot/config/arr_sync_blocklist.tsv # tombstone file
ARR_SYNC_BLOCKLIST="$DATA_DIR/arr_sync_blocklist.tsv" # tombstone file
ARR_SYNC_CONNECT_TIMEOUT=10 # SSH connect timeout in seconds
ARR_SYNC_API_TIMEOUT=60 # curl API call timeout in seconds
DOCKER_APPDATA_BASE=/mnt/user/appdata
@@ -189,7 +189,7 @@ ARR_IMPORT_RECOVERY_AGE=6 # hours — items newer than this are skipped
SONARR_VERSION_MAJOR=4
RADARR_VERSION_MAJOR=6
LIDARR_VERSION_MAJOR=3
ARR_RECOVERY_STATS=/boot/config/arr_recovery_stats.db # read by coffee report
ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db" # read by coffee report
```
---
@@ -274,6 +274,47 @@ Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
---
### Play State Sync
```bash
PLAY_STATE_SYNC_ENABLED=true # toggle entire sync
PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items
```
Emby and Jellyfin servers configured per-host:
```bash
# host*.conf
HOST1_EMBY_URL="http://192.168.50.2:8096"
HOST1_EMBY_API_KEY="..."
HOST1_JELLYFIN_URL="" # empty = skip Jellyfin on this host
HOST1_JELLYFIN_API_KEY=""
```
---
### Upgrade Webhook
```bash
# master.conf
WEBHOOK_PORT=9456 # 0 = disable listener
WEBHOOK_SECRET="" # auto-generated on first start if empty
WEBHOOK_UPGRADE_RSYNC_ENABLED=true # push upgraded file to remote nodes
```
```bash
# host*.conf
HOST1_WEBHOOK_LAN_IP="192.168.50.2" # IP arrs call for webhook delivery
```
When `WEBHOOK_PORT=0`, `start_webhook_listener.sh` exits cleanly and no listener starts.
When `WEBHOOK_SECRET` is empty, a 32-byte hex secret is generated on first start and
written back to `master.conf`. Run `Tools/webhook_setup.sh` to register the URL in arrs.
Webhook log: `/var/log/varaverk/upgrade_webhook.log`
---
### Orchestrator Job Order
```bash
+48 -2
View File
@@ -4,8 +4,9 @@ Library health, consistency, sync, and behavior-driven discovery for a multi-ser
stack. Correct permissions so arrs can manage files. Junk removal so orphan detection
isn't confused by scene debris. Library sync so every node tracks the same content.
Orphan cleanup against live arr APIs so deleted content actually leaves disk. Emby
notified automatically after every deletion. Weekly discovery adds new music, movies,
and TV shows based on what you actually play — no manual browsing required.
notified automatically after every deletion. Watch state synced across Emby/Jellyfin
every 30 minutes. Quality upgrades propagate to all nodes immediately via webhook.
Weekly discovery adds new music, movies, and TV shows based on what you actually play.
> **These scripts permanently delete files.** The arr cleanup scripts are protected by
> multiple safety layers that must all pass before anything is touched — but dry runs and
@@ -91,6 +92,19 @@ free overnight recovery.
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
fallback sources. Never overwrites existing files.
**Play State Sync**
`play_state_sync.sh` — syncs watched/played state and resume positions across all
configured Emby and Jellyfin servers. Newest timestamp wins. Runs every 30 minutes
via critical_sync_maintenance.sh.
**Upgrade Propagation**
`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr
OnUpgrade webhooks. Continuous; started at array start. Writes to
`/var/log/varaverk/upgrade_webhook.log`.
`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded
item folder to every remote node immediately, then triggers an arr library rescan on
each remote so the upgraded file is accepted without triggering a redundant quality search.
**Discovery**
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
@@ -138,6 +152,24 @@ playback_aware_radarr_discovery.sh — score watch history → TMDB recommenda
playback_aware_sonarr_discovery.sh — score episode history → TMDB TV recommendations → add to Sonarr
```
**Every 30 min via `critical_sync_maintenance.sh` (CRITICAL_MAINTENANCE_SCRIPTS):**
```
play_state_sync.sh — sync watched/resume state across Emby + Jellyfin
```
**Continuous (started by `array_started.sh`):**
```
start_webhook_listener.sh — Node.js webhook server; dispatch upgrade_webhook_handler.sh
```
**On every arr upgrade (triggered by webhook):**
```
upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan
```
**Ad-hoc or separate schedule:**
```
@@ -189,6 +221,9 @@ in `HOST*_MEDIA_PERMISSION_SHARES` and `HOST*_MEDIA_CLEAN_FOLDERS` in host*.conf
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Ad-hoc or separate schedule |
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Ad-hoc or weekly |
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Ad-hoc or weekly |
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min via `critical_sync_maintenance.sh` |
| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous (started by `array_started.sh`) |
| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade (via webhook) |
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly via `weekly_sync_maintenance.sh` |
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly via `weekly_sync_maintenance.sh` |
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly via `weekly_sync_maintenance.sh` |
@@ -228,6 +263,17 @@ Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
└── each discovery script fires arr search immediately after successful add
Every 30 min (critical_sync_maintenance.sh CRITICAL_MAINTENANCE_SCRIPTS):
play_state_sync.sh ─── newest timestamp wins → watched/resume state synced
across all configured Emby + Jellyfin servers
Continuous (started by array_started.sh):
start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
│ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
└── upgrade_webhook_handler.sh
├── rsync upgraded folder → all remote nodes immediately
└── trigger arr library rescan on each remote (accept new file, no quality search)
Ad-hoc enrichment:
lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
+2 -1
View File
@@ -38,7 +38,7 @@
# ==============================================================================================
#
# Remote API Access — Cache-First, SSH Fallback
# If conf_sync.sh has populated /tmp/.vv/config/cached/.confs/ and
# If conf_sync.sh has populated /tmp/.cache/vv/d/ and
# load_config.sh has sourced it, HOST*_<ARR>_API_KEY vars are available
# in the environment. Remote functions use them to call the arr API
# directly over Tailscale (no SSH, no remote shell). If the cached key
@@ -174,6 +174,7 @@ done
unset _tool
detect_hosts
require_partnership
# ── Runtime config with defaults ──────────────────────────────────────────────────────────────
ARR_SYNC_ENABLED="${ARR_SYNC_ENABLED:-true}"
+1
View File
@@ -80,6 +80,7 @@ for _a in "$@"; do
[[ "$_a" == "--full" ]] && FULL_SYNC=true || _FILTERED+=("$_a")
done
parse_args "${_FILTERED[@]}"
require_partnership
# ==============================================================================================
# ━━━ Setup ━━━
+5 -2
View File
@@ -5,7 +5,7 @@
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Starts a standalone PHP HTTP server that receives Sonarr/Radarr/Lidarr
# Starts a standalone Node.js HTTP server that receives Sonarr/Radarr/Lidarr
# OnUpgrade webhooks and dispatches upgrade_webhook_handler.sh.
#
# Runs outside Unraid nginx — no session auth required. The shared secret in
@@ -13,9 +13,12 @@
#
# http://<HOST_LAN_IP>:<WEBHOOK_PORT>/webhook?key=<WEBHOOK_SECRET>
#
# Runs as a continuous script started by array_started.sh. Execs php -S which
# Runs as a continuous script started by array_started.sh. Execs node which
# replaces this process — the PID stays the same for array_started.sh's check.
#
# Uses Node.js instead of php -S: php -S on Unraid PHP 8.4 silently drops
# POST request bodies, making webhook payloads arrive empty.
#
# If WEBHOOK_SECRET is empty in master.conf: generates and saves one, then starts.
# If WEBHOOK_PORT is 0: exits cleanly (disables the listener).
#
+5 -5
View File
@@ -151,14 +151,14 @@ BACKUP_VERIFY_MIN_SIZE="1M" # skip files smaller than this
```bash
# master.conf
BANDWIDTH_LOG="/boot/config/bandwidth_history.db" # survives reboots
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db" # survives reboots
BANDWIDTH_LOG_RETENTION=90 # days — file stays bounded, never grows unbounded
BANDWIDTH_WARN_GB=50 # flag transfers or daily totals exceeding this
```
**Why `/boot/config/`**: The log needs to survive reboots to build a useful history.
`/boot/config/` is on the USB flash drive, which survives reboots and is backed up
by unRAID's flash backup. The log is bounded by `BANDWIDTH_LOG_RETENTION` so it never
**Why `$DATA_DIR`**: The log needs to survive reboots to build a useful history.
`$DATA_DIR` (`$SCRIPTS_DIR/data/`) is on the boot device (internal) or appdata (flash),
both of which survive reboots. The log is bounded by `BANDWIDTH_LOG_RETENTION` so it never
grows unbounded.
**`BANDWIDTH_WARN_GB`**: Set to a value that represents "unexpectedly large" for your
@@ -401,7 +401,7 @@ BACKUP_VERIFY_SAMPLE=10
BACKUP_VERIFY_MIN_SIZE="1M"
# bandwidth_monitor.sh
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90
BANDWIDTH_WARN_GB=50
+1
View File
@@ -124,6 +124,7 @@ acquire_lock
# detect_hosts() sets MY_ID and aliases BACKUP_VERIFY_SHARES + DAILY_SYNC_SHARES
detect_hosts
require_partnership
# Share selection — configured list or fallback to daily sync shares
if [[ ${#BACKUP_VERIFY_SHARES[@]} -gt 0 ]]; then
+15 -16
View File
@@ -142,30 +142,29 @@ for covered in "${ALL_HOST_IDS[@]}"; do
covered_owner_var="${covered}_OWNER"; covered_owner="${!covered_owner_var:-$covered}"
covered_email_var="${covered}_OWNER_EMAIL"; covered_email="${!covered_email_var:-}"
# Find who covers this host and collect their tiers
# Each host defines its own recovery profile — read directly from covered host's vars
any_tiers=false
declare -A tier_containers=()
declare -A tier_delays=()
covered_by=""
for covering in "${ALL_HOST_IDS[@]}"; do
[[ "$covering" == "$covered" ]] && continue
for tier in 1 2 3 4; do
containers=$(get_array "FALLBACK_${covered}_TIER${tier}")
[[ -z "$containers" ]] && continue
any_tiers=true
tier_containers[$tier]="$containers"
delay_var="${covered}_TIER${tier}_DELAY"
tier_delays[$tier]="${!delay_var:-0}"
done
for tier in 1 2 3 4; do
containers=$(get_array "FALLBACK_${covering}_COVERS_${covered}_TIER${tier}")
[[ -z "$containers" ]] && continue
any_tiers=true
tier_containers[$tier]="$containers"
delay_var="${covered}_TIER${tier}_DELAY"
tier_delays[$tier]="${!delay_var:-0}"
done
[[ "$any_tiers" == true ]] && {
if [[ "$any_tiers" == true ]]; then
for covering in "${ALL_HOST_IDS[@]}"; do
[[ "$covering" == "$covered" ]] && continue
covering_owner_var="${covering}_OWNER"
covering_owner="${!covering_owner_var:-$covering}"
covered_by="$covering ($covering_owner)"
}
done
covered_by="${covered_by:+$covered_by, }$covering ($covering_owner)"
done
fi
[[ "$any_tiers" == false ]] && continue
+2 -2
View File
@@ -849,7 +849,7 @@ long tests — that should only run on stable systems that have been up for at l
Two gates must both pass before any job runs:
1. Server uptime ≥ `MONTHLY_UPTIME_THRESHOLD_DAYS`
2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file on `/boot/config/` — survives reboots)
2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file in `$STATE_DIR` — survives reboots)
If either gate fails, the script exits 0 with no output. This is expected — it runs
daily and most days are no-ops.
@@ -866,7 +866,7 @@ MONTHLY_MAINTENANCE_SCRIPTS=(
)
MONTHLY_UPTIME_THRESHOLD_DAYS=30
MONTHLY_RUN_INTERVAL_DAYS=30
MONTHLY_LAST_RUN_FILE="/boot/config/monthly_maintenance_last_run.db"
MONTHLY_LAST_RUN_FILE="$STATE_DIR/monthly_maintenance_last_run.db"
```
Scripts are commented out by default — uncomment what applies to your hardware.
@@ -6,7 +6,7 @@
# Schedule: 0 */4 * * * (every 4 hours)
#
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
# 1. conf_sync.sh --pull-only — refresh partner conf cache in RAM (/tmp/.vv/)
# 1. conf_sync.sh --pull-only — refresh partner conf cache in RAM (/tmp/.cache/vv/d/)
# 2. arr_sync.sh — sync Lidarr/Sonarr/Radarr libraries across all nodes
# 3. Rsync window (optional) — INTERMEDIATE_SYNC_SHARES, if any configured
# 4. INTERMEDIATE_MAINTENANCE_SCRIPTS — artwork fetch and any future 4-hour jobs
+4 -4
View File
@@ -574,7 +574,7 @@ Partnership/partnership_manager.sh --status # shows both sides via SSH
```bash
# Check the offline counter:
cat /boot/config/partnership_offline_days.db
cat "$STATE_DIR/partnership_offline_days.db"
# Extended Tailscale outage may have incremented the counter.
# Check Tailscale peer visibility:
@@ -591,9 +591,9 @@ Partnership/partnership_manager.sh --onboard
## ━━━ STATE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```bash
/boot/config/partnership_HOST1.db # HOST1 writes only
/boot/config/partnership_HOST2.db # HOST2 writes only
/boot/config/partnership_blocklist.db # hostname|timestamp|reason
$STATE_DIR/partnership_HOST1.db # HOST1 writes only
$STATE_DIR/partnership_HOST2.db # HOST2 writes only
$STATE_DIR/partnership_blocklist.db # hostname|timestamp|reason
# Example state file:
state=ACTIVE
+5 -5
View File
@@ -93,11 +93,11 @@
# STATE FILES
# ==============================================================================================
#
# /boot/config/partnership_HOST1.db — HOST1 writes, HOST2 reads via SSH
# /boot/config/partnership_HOST2.db — HOST2 writes, HOST1 reads via SSH
# /boot/config/partnership_blocklist.db hostname|timestamp|reason, persists until cleared
# $STATE_DIR/partnership_<hostname>.db — each host writes its own, partner reads via SSH
# $STATE_DIR/partnership_blocklist.db hostname|timestamp|reason, persists until cleared
# $STATE_DIR/partnership_offline_days.db — cumulative offline day counter
#
# On /boot/config — survives reboots, available before array starts, minimal flash wear.
# All in STATE_DIR — survives reboots (on /boot in internal mode, appdata in flash mode).
#
# ==============================================================================================
# CONFIGURATION
@@ -512,7 +512,7 @@ gather_partner_fallback_containers() {
eval "${out_var}=()"
local tier var
for tier in TIER1 TIER2 TIER3 TIER4; do
var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_${tier}"
var="FALLBACK_${REMOTE_ID}_${tier}"
# Check if var is set and is an array
if declare -p "$var" >/dev/null 2>&1; then
local -a _tmp_arr
+1 -1
View File
@@ -274,7 +274,7 @@ platform_get_templates_dir() {
# Writes the path to the persistent Varaverk setup/wizard state database.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_setup_db_path() {
echo "/boot/config/varaverk_setup.db"
echo "${STATE_DIR}/varaverk_setup.db"
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
+1 -1
View File
@@ -13,7 +13,7 @@ $ramUsedMb = $ramTotalMb - $res['ram_free_mb'];
$ramPct = $ramTotalMb > 0 ? (int)round($ramUsedMb / $ramTotalMb * 100) : 0;
// Fallback state (fast file read, no exec)
$fbRaw = @file_get_contents('/tmp/fallback_state.db') ?: '';
$fbRaw = @file_get_contents(STATE_DIR . '/fallback_state.db') ?: '';
$fbData = vv_parse_kv_db($fbRaw);
$fallbackState = $fbData['state'] ?? 'UNKNOWN';
+24 -5
View File
@@ -34,8 +34,7 @@ function vv_setup_state_write(array $data): void {
}
// Push the setup state file to all remote hosts via scp.
// Unlike master.conf push, this does NOT require the plugin to be installed on the remote —
// it only needs SSH to be reachable, and pushes to /boot/config/ (always available).
// Reads the remote's varaverk.cfg to find their actual SCRIPTS_DIR (handles appdata mode).
function vv_push_setup_state(): void {
if (!file_exists(VV_SETUP_STATE_FILE)) return;
$myHostId = vv_detect_host();
@@ -56,9 +55,20 @@ function vv_push_setup_state(): void {
if (!$ip) continue;
$sshBase = 'ssh -i ' . escapeshellarg($sshKey)
. ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@' . $ip;
// Ensure the target dir exists (it always should on Unraid, but be safe)
shell_exec($sshBase . ' "mkdir -p /boot/config" 2>/dev/null');
$dest = escapeshellarg('root@' . $ip . ':/boot/config/varaverk_setup.db');
// Get remote SCRIPTS_DIR from varaverk.cfg — handles appdata mode on remote.
// Falls back to the default install path if varaverk.cfg is absent (pre-install).
$cfgRaw = trim(shell_exec($sshBase . ' "cat /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null"') ?: '');
$remoteSD = '/boot/config/plugins/varaverk';
foreach (explode("\n", $cfgRaw) as $line) {
if (str_starts_with(trim($line), 'SCRIPTS_DIR=')) {
$remoteSD = trim(substr(trim($line), strlen('SCRIPTS_DIR=')), '"\'');
break;
}
}
$remoteStatePath = $remoteSD . '/State_Files/varaverk_setup.db';
shell_exec($sshBase . ' "mkdir -p ' . escapeshellarg(dirname($remoteStatePath)) . '" 2>/dev/null');
$dest = escapeshellarg('root@' . $ip . ':' . $remoteStatePath);
exec('scp -i ' . escapeshellarg($sshKey)
. ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no'
. ' ' . escapeshellarg(VV_SETUP_STATE_FILE) . ' ' . $dest . ' 2>&1');
@@ -391,6 +401,15 @@ function vv_auto_create_api_key(string $hostId, string $confFile): array {
return ['ok' => true, 'key_preview' => $key ? substr($key, 0, 8) . '...' . substr($key, -4) : 'registered'];
}
// Build a bash command that reads a state file from the REMOTE host's State_Files/.
// Reads the remote's varaverk.cfg to resolve their SCRIPTS_DIR (may differ from ours
// when the remote is in appdata mode). Falls back to the internal plugin path.
function vv_remote_state_cmd(string $filename): string {
$fn = basename($filename);
return 'sd=$(grep -m1 SCRIPTS_DIR= /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null'
. ' | cut -d\'"\' -f2); cat "${sd:-/boot/config/plugins/varaverk}/State_Files/' . $fn . '" 2>/dev/null';
}
// Local LAN IP via routing table — static-cached per request.
// Previously duplicated in include/docker_folders.php and inline in include/docker.php.
function vv_local_ip(): string {
+35 -23
View File
@@ -18,12 +18,14 @@ function vv_fb_scalar(string $raw, string $varname): string {
function vv_fb_parse_state(string $text): array {
$out = [
'state' => 'UNKNOWN',
'fallback_start' => 0,
'handback_strikes' => 0,
'tier2_started' => false,
'tier3_started' => false,
'tier4_started' => false,
'state' => 'UNKNOWN',
'fallback_start' => 0,
'handback_strikes' => 0,
'tier2_started' => false,
'tier3_started' => false,
'tier4_started' => false,
'partnership_suspended' => false,
'partner_lost_at' => 0,
];
foreach (explode("\n", $text) as $line) {
$line = trim($line);
@@ -31,24 +33,26 @@ function vv_fb_parse_state(string $text): array {
[$k, $v] = array_pad(explode('=', $line, 2), 2, '');
$k = trim($k); $v = trim($v, '"\'');
switch ($k) {
case 'state': $out['state'] = $v; break;
case 'fallback_start': $out['fallback_start'] = (int)$v; break;
case 'handback_strikes': $out['handback_strikes'] = (int)$v; break;
case 'tier2_started': $out['tier2_started'] = $v === 'true'; break;
case 'tier3_started': $out['tier3_started'] = $v === 'true'; break;
case 'tier4_started': $out['tier4_started'] = $v === 'true'; break;
case 'state': $out['state'] = $v; break;
case 'fallback_start': $out['fallback_start'] = (int)$v; break;
case 'handback_strikes': $out['handback_strikes'] = (int)$v; break;
case 'tier2_started': $out['tier2_started'] = $v === 'true'; break;
case 'tier3_started': $out['tier3_started'] = $v === 'true'; break;
case 'tier4_started': $out['tier4_started'] = $v === 'true'; break;
case 'partnership_suspended': $out['partnership_suspended'] = $v === 'true'; break;
case 'partner_lost_at': $out['partner_lost_at'] = (int)$v; break;
}
}
return $out;
}
function vv_fb_local_state(): array {
$path = '/boot/config/fallback_state.db';
$path = STATE_DIR . '/fallback_state.db';
return vv_fb_parse_state(file_exists($path) ? file_get_contents($path) : '');
}
function vv_fb_remote_state(string $ip, string $sshKey): array {
$out = vv_pt_ssh($ip, $sshKey, 'cat /boot/config/fallback_state.db 2>/dev/null');
$out = vv_pt_ssh($ip, $sshKey, vv_remote_state_cmd('fallback_state.db'));
return vv_fb_parse_state($out);
}
@@ -67,11 +71,10 @@ function vv_fb_remote_running(string $ip, string $sshKey): array {
// ── Covers — what a node runs for the other when it's down ───────────────────
function vv_fb_covers(string $covering, string $remote, string $coveringRaw, string $remoteRaw): array {
$cu = strtoupper($covering); // HOST1
$ru = strtoupper($remote); // HOST2
$ru = strtoupper($remote); // HOST2 — covered host owns its own tier lists
$tiers = [];
for ($t = 1; $t <= 4; $t++) {
$tiers["tier$t"] = vv_fb_bash_array($coveringRaw, "FALLBACK_{$cu}_COVERS_{$ru}_TIER{$t}");
$tiers["tier$t"] = vv_fb_bash_array($remoteRaw, "FALLBACK_{$ru}_TIER{$t}");
}
// Delays: how long the remote (covered) host must be down before each tier fires.
// Stored in the *remote* host's conf as REMOTE_TIER*_DELAY.
@@ -95,8 +98,13 @@ function vv_fb_all(): array {
$currentHost = vv_detect_host();
$hosts = vv_fb_known_hosts();
$tsPeers = vv_pt_ts_peers();
$handbackReq = (int)(vv_fb_scalar(vv_read_conf_raw('master.conf'), 'FALLBACK_HANDBACK_STRIKES') ?: 3);
$fbEnabled = vv_fb_scalar(vv_read_conf_raw('master.conf'), 'FALLBACK_ENABLED') === 'true';
$masterRaw = vv_read_conf_raw('master.conf');
$handbackReq = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_HANDBACK_STRIKES') ?: 3);
$fbEnabled = vv_fb_scalar($masterRaw, 'FALLBACK_ENABLED') === 'true';
$ptEnabled = vv_fb_scalar($masterRaw, 'PARTNERSHIP_ENABLED') === 'true';
$rsyncEnabled = vv_fb_scalar($masterRaw, 'FALLBACK_RSYNC_ENABLED') !== 'false';
$checkInterval = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_CHECK_INTERVAL') ?: 30);
$suspendAfter = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_PARTNERSHIP_SUSPEND_AFTER') ?: 120);
// Read all host conf raws upfront
$raws = [];
@@ -163,9 +171,13 @@ function vv_fb_all(): array {
}
return [
'ts' => time(),
'fb_enabled' => $fbEnabled,
'handback_req' => $handbackReq,
'nodes' => $nodes,
'ts' => time(),
'fb_enabled' => $fbEnabled,
'partnership_enabled' => $ptEnabled,
'fb_rsync_enabled' => $rsyncEnabled,
'handback_req' => $handbackReq,
'check_interval' => $checkInterval,
'suspend_after' => $suspendAfter,
'nodes' => $nodes,
];
}
+6 -10
View File
@@ -56,14 +56,13 @@ function vv_fallback_state(): array {
$enabled = ($vars['FALLBACK_ENABLED'] ?? 'false') === 'true';
$interval = (int)($vars['FALLBACK_CHECK_INTERVAL'] ?? 30);
$reqStrikes = (int)($vars['FALLBACK_HANDBACK_STRIKES'] ?? 3);
$ptRequired = ($vars['FALLBACK_PARTNERSHIP_REQUIRED'] ?? 'true') === 'true';
$suspendAfter = (int)($vars['FALLBACK_PARTNERSHIP_SUSPEND_AFTER'] ?? 120);
$stateFile = '/tmp/fallback_state.db';
$stateFile = STATE_DIR . '/fallback_state.db';
if (!file_exists($stateFile)) {
return ['state' => 'UNKNOWN', 'enabled' => $enabled, 'check_interval' => $interval,
'handback_strikes' => 0, 'handback_strikes_required' => $reqStrikes,
'partnership_required' => $ptRequired, 'partnership_suspended' => false,
'partnership_suspended' => false,
'partner_lost_at' => 0, 'partnership_suspend_after' => $suspendAfter];
}
$raw = [];
@@ -83,7 +82,6 @@ function vv_fallback_state(): array {
'enabled' => $enabled,
'check_interval' => $interval,
'handback_strikes_required' => $reqStrikes,
'partnership_required' => $ptRequired,
'partnership_suspend_after' => $suspendAfter,
];
}
@@ -109,19 +107,17 @@ function vv_fallback_active(): array {
if ($c) $running[strtolower($c['n'])] = $c['i'];
}
// Parse FALLBACK arrays from this host's conf
$confFile = strtolower($myId) . '.conf';
$rawConf = vv_read_conf_raw($confFile);
$result = [];
foreach ($allHostIds as $covered) {
if ($covered === $myId) continue;
$coveredHostname = $vars[$covered] ?? '';
if (!$coveredHostname) continue;
// Covered host defines its own recovery profile — read from their conf
$coveredRaw = vv_read_conf_raw(strtolower($covered) . '.conf');
$names = [];
for ($tier = 1; $tier <= 4; $tier++)
$names = array_merge($names, vv_parse_bash_array($rawConf, "FALLBACK_{$myId}_COVERS_{$covered}_TIER{$tier}"));
$names = array_merge($names, vv_parse_bash_array($coveredRaw, "FALLBACK_{$covered}_TIER{$tier}"));
$active = [];
foreach ($names as $name) {
@@ -168,7 +164,7 @@ function vv_watchdog_summary(): array {
usort($restarts, fn($a, $b) => $b['ts'] - $a['ts']);
// Reboots (12 h)
$rebootRaw = @file_get_contents('/boot/config/system_watchdog_reboots.db') ?: '';
$rebootRaw = @file_get_contents(STATE_DIR . '/system_watchdog_reboots.db') ?: '';
$rbootCutoff = time() - 43200;
$reboots = 0;
foreach (explode("\n", trim($rebootRaw)) as $line) {
+4 -5
View File
@@ -10,7 +10,7 @@ require_once __DIR__ . '/common.php'; // vv_system_info(), vv_docker_containers(
function vv_pt_config(): array {
$v = vv_conf_vars();
$offlineDays = null;
$odFile = '/boot/config/partnership_offline_days.db';
$odFile = STATE_DIR . '/partnership_offline_days.db';
if (file_exists($odFile)) {
$raw = trim(@file_get_contents($odFile) ?: '');
if (is_numeric($raw)) $offlineDays = (int)$raw;
@@ -209,12 +209,11 @@ function vv_pt_nodes(): array {
// Fallback state
$fbState = 'UNKNOWN';
$fbPath = '/boot/config/fallback_state.db';
if ($isMe) {
$fb = vv_pt_read_db($fbPath);
$fb = vv_pt_read_db(STATE_DIR . '/fallback_state.db');
$fbState = $fb['state'] ?? 'UNKNOWN';
} elseif ($ts['online'] && $ts['ip'] && $mySshKey) {
$out = vv_pt_ssh($ts['ip'], $mySshKey, 'cat /boot/config/fallback_state.db 2>/dev/null');
$out = vv_pt_ssh($ts['ip'], $mySshKey, vv_remote_state_cmd('fallback_state.db'));
if ($out) {
$fb = [];
foreach (explode("\n", $out) as $line) {
@@ -226,7 +225,7 @@ function vv_pt_nodes(): array {
}
// Partnership DB — local only (each server writes its own)
$dbPath = "/boot/config/partnership_{$hostname}.db";
$dbPath = STATE_DIR . "/partnership_{$hostname}.db";
$ptDb = vv_pt_read_db($dbPath);
// System info
+8 -8
View File
@@ -127,14 +127,14 @@ function vv_wd_parse_network_state(string $raw): array {
// ── Local state files ─────────────────────────────────────────────────────────
function vv_wd_local_states(string $restartLogPath): array {
$rwRaw = @file_get_contents('/tmp/resource_watchdog_state.db') ?: '';
$dockRaw = @file_get_contents('/tmp/container_watchdog_state.db') ?: '';
$skipRaw = @file_get_contents('/boot/config/system_watchdog_failed.db') ?: '';
$sysRaw = @file_get_contents('/tmp/system_watchdog_state.db') ?: '';
$rebootRaw = @file_get_contents('/boot/config/system_watchdog_reboots.db')?: '';
$restartRaw= @file_get_contents($restartLogPath) ?: '';
$storRaw = @file_get_contents('/tmp/storage_watchdog_state.db') ?: '';
$netWdRaw = @file_get_contents('/tmp/network_watchdog_state.db') ?: '';
$rwRaw = @file_get_contents(STATE_DIR . '/resource_watchdog_state.db') ?: '';
$dockRaw = @file_get_contents(STATE_DIR . '/container_watchdog_state.db') ?: '';
$skipRaw = @file_get_contents(STATE_DIR . '/docker_watchdog_failed.db') ?: '';
$sysRaw = @file_get_contents(STATE_DIR . '/system_watchdog_state.db') ?: '';
$rebootRaw = @file_get_contents(STATE_DIR . '/system_watchdog_reboots.db') ?: '';
$restartRaw= @file_get_contents($restartLogPath) ?: '';
$storRaw = @file_get_contents(STATE_DIR . '/storage_watchdog_state.db') ?: '';
$netWdRaw = @file_get_contents(STATE_DIR . '/network_watchdog_state.db') ?: '';
$rw = vv_wd_parse_kv($rwRaw);
$dock = vv_wd_parse_kv($dockRaw);
+200 -27
View File
@@ -1,4 +1,5 @@
<style>
/* ── Existing status styles ── */
.vv-fb-active { background:#1a1200;border:1px solid #5a3800;border-radius:6px;padding:12px 14px; }
.vv-fb-active-h { display:flex;align-items:baseline;gap:10px;margin-bottom:8px; }
.vv-fb-badge { font-size:11px;font-weight:bold;letter-spacing:.06em;padding:2px 7px;border-radius:3px;flex-shrink:0; }
@@ -6,6 +7,7 @@
.vv-fb-badge.norm { background:#1a2a1a;color:#4caf50; }
.vv-fb-badge.dark { background:#2a1a2a;color:#9c27b0; }
.vv-fb-badge.nonet{ background:#1a1a2a;color:#5c7cfa; }
.vv-fb-badge.susp { background:#2a1a00;color:#888; }
.vv-fb-meta { display:flex;gap:18px;flex-wrap:wrap;margin-bottom:10px; }
.vv-fb-meta-item{ display:flex;flex-direction:column;gap:1px; }
.vv-fb-meta-val { font-size:17px;font-weight:bold;color:#ffb74d; }
@@ -31,17 +33,109 @@
.vv-fb-state-dot { width:6px;height:6px;border-radius:50%;flex-shrink:0;margin-top:3px; }
.vv-fb-sep { border:none;border-top:1px solid #222;margin:8px 0; }
.vv-fb-disabled { grid-column:1/-1;color:#3a3a3a;font-size:12px;padding:20px 0;text-align:center; }
/* ── Controls + settings card ── */
.vv-fb-card { background:#161616;border:1px solid #2a2a2a;border-radius:6px;padding:14px 16px;margin-bottom:14px; }
.vv-fb-card-hdr { font-size:11px;font-weight:700;color:#666;text-transform:uppercase;letter-spacing:.07em;margin-bottom:12px; }
.vv-fb-ctrl-row { display:flex;justify-content:space-between;align-items:center;gap:12px;padding:5px 0; }
.vv-fb-ctrl-lbl { font-size:12px;color:#888; }
.vv-fb-ctrl-sub { font-size:10px;color:#3a3a3a;margin-top:2px; }
.vv-fb-tog { width:32px;height:18px;border-radius:9px;background:#222;border:1px solid #333;
position:relative;transition:background .15s,border-color .15s;flex-shrink:0;cursor:pointer; }
.vv-fb-tog.on { background:#1a3a1a;border-color:#2d5a2d; }
.vv-fb-tog::after { content:'';position:absolute;top:2px;left:2px;width:12px;height:12px;
border-radius:50%;background:#555;transition:left .15s,background .15s; }
.vv-fb-tog.on::after { left:16px;background:#4caf50; }
.vv-fb-set-row { display:flex;justify-content:space-between;align-items:center;padding:5px 0; }
.vv-fb-set-lbl { font-size:11px;color:#555; }
.vv-fb-set-inp { background:#0d0d0d;border:1px solid #2a2a2a;border-radius:3px;color:#888;
font-size:12px;padding:4px 8px;outline:none;width:72px;font-family:monospace;
text-align:right;box-sizing:border-box; }
.vv-fb-set-inp:focus { border-color:#444; }
.vv-fb-set-unit { font-size:10px;color:#3a3a3a;min-width:56px; }
.vv-fb-save-btn { background:#1a1a1a;border:1px solid #333;color:#888;font-size:11px;
padding:5px 14px;border-radius:3px;cursor:pointer; }
.vv-fb-save-btn:hover { border-color:#555;color:#ccc; }
.vv-fb-save-btn:disabled { opacity:.4;cursor:default; }
</style>
<!-- Controls card -->
<div class="vv-fb-card">
<div class="vv-fb-card-hdr">Controls</div>
<div class="vv-fb-ctrl-row">
<div>
<div class="vv-fb-ctrl-lbl">Partnership</div>
<div class="vv-fb-ctrl-sub">Master gate disabling stops all cross-server operations</div>
</div>
<div class="vv-fb-tog" id="vv-fb-pt-tog" onclick="vvFbToggle(this,'PARTNERSHIP_ENABLED')"></div>
</div>
<hr class="vv-fb-sep">
<div class="vv-fb-ctrl-row">
<div>
<div class="vv-fb-ctrl-lbl">Fallback</div>
<div class="vv-fb-ctrl-sub">Mutual container failover between nodes</div>
</div>
<div class="vv-fb-tog" id="vv-fb-en-tog" onclick="vvFbToggle(this,'FALLBACK_ENABLED')"></div>
</div>
<hr class="vv-fb-sep">
<div class="vv-fb-ctrl-row">
<div>
<div class="vv-fb-ctrl-lbl">Rsync on handback</div>
<div class="vv-fb-ctrl-sub">Writeback rsync when the covered host recovers and containers return</div>
</div>
<div class="vv-fb-tog" id="vv-fb-rsync-tog" onclick="vvFbToggle(this,'FALLBACK_RSYNC_ENABLED')"></div>
</div>
</div>
<!-- Status section -->
<div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px;padding:0 2px;">
<span style="font-size:13px;font-weight:bold;color:#888;text-transform:uppercase;letter-spacing:.06em;">FallBack</span>
<span style="font-size:13px;font-weight:bold;color:#888;text-transform:uppercase;letter-spacing:.06em;">Status</span>
<span style="font-size:11px;color:#3a3a3a;" id="vv-fb-ts"></span>
</div>
<div id="vv-fb-grid" style="display:grid;grid-template-columns:repeat(8,1fr);gap:12px;">
<div id="vv-fb-grid" style="display:grid;grid-template-columns:repeat(8,1fr);gap:12px;margin-bottom:14px;">
<div style="grid-column:1/-1;color:#444;font-size:12px;padding:16px 0;text-align:center;">Loading…</div>
</div>
<!-- Settings card -->
<div class="vv-fb-card">
<div class="vv-fb-card-hdr">Settings</div>
<div class="vv-fb-set-row">
<span class="vv-fb-set-lbl">Check interval</span>
<div style="display:flex;align-items:center;gap:8px;">
<input class="vv-fb-set-inp" id="vv-fb-interval" type="number" min="5" max="300">
<span class="vv-fb-set-unit">seconds</span>
</div>
</div>
<div class="vv-fb-set-row">
<span class="vv-fb-set-lbl">Handback strikes</span>
<div style="display:flex;align-items:center;gap:8px;">
<input class="vv-fb-set-inp" id="vv-fb-strikes" type="number" min="1" max="20">
<span class="vv-fb-set-unit">consecutive</span>
</div>
</div>
<div class="vv-fb-set-row">
<span class="vv-fb-set-lbl">Partnership suspend after</span>
<div style="display:flex;align-items:center;gap:8px;">
<input class="vv-fb-set-inp" id="vv-fb-suspend" type="number" min="0" max="1440">
<span class="vv-fb-set-unit">minutes</span>
</div>
</div>
<div style="display:flex;justify-content:flex-end;align-items:center;gap:10px;margin-top:10px;padding-top:8px;border-top:1px solid #1e1e1e;">
<span id="vv-fb-set-fb" style="font-size:11px;"></span>
<button class="vv-fb-save-btn" id="vv-fb-save-btn" onclick="vvFbSaveSettings()">Save</button>
</div>
</div>
<script>
(function() {
@@ -78,6 +172,7 @@ function _stateBadge(st) {
OFFLINE: ['dark', 'OFFLINE'],
UNREACHABLE: ['dark', 'UNREACHABLE'],
UNKNOWN: ['dark', 'UNKNOWN'],
SUSPENDED: ['susp', 'SUSPENDED'],
};
const [cls, label] = map[st] || ['dark', st];
return `<span class="vv-fb-badge ${cls}">${label}</span>`;
@@ -87,7 +182,7 @@ function _stateDot(st) {
const col = {
NORMAL:'#4caf50', FALLBACK:'#ffb74d',
NO_INTERNET:'#5c7cfa', DARK:'#9c27b0',
OFFLINE:'#555', UNREACHABLE:'#555', UNKNOWN:'#333',
OFFLINE:'#555', UNREACHABLE:'#555', UNKNOWN:'#333', SUSPENDED:'#444',
}[st] || '#333';
return `<span class="vv-fb-state-dot" style="background:${col}"></span>`;
}
@@ -102,7 +197,6 @@ function _activeCard(nodes, handbackReq) {
const cov = covering.covers;
const covered = cov ? cov.hostname : '?';
// All containers that should be running at current tier
let expected = [...(cov?.tier1 || [])];
if (tier >= 2) expected = expected.concat(cov?.tier2 || []);
if (tier >= 3) expected = expected.concat(cov?.tier3 || []);
@@ -169,9 +263,19 @@ function _tierSection(tiers, activeTier, delays) {
}).join('');
}
function _nodeCard(node) {
function _ptStatus(st) {
if (!st) return '';
if (st.partnership_suspended) return _stateBadge('SUSPENDED');
if (st.partner_lost_at && st.partner_lost_at > 0) {
const minGone = Math.floor((Date.now() / 1000 - st.partner_lost_at) / 60);
return `<span class="vv-fb-badge susp">GRACE ${minGone}m</span>`;
}
return '';
}
function _nodeCard(node, suspendAfter) {
const st = node.state || {};
const state = st.state || 'UNKNOWN';
const state = st.partnership_suspended ? 'SUSPENDED' : (st.state || 'UNKNOWN');
const cov = node.covers;
const active = _activeTier(state === 'FALLBACK' ? st : null);
@@ -183,6 +287,8 @@ function _nodeCard(node) {
? _tierSection(cov, active, cov.delays)
: '<div style="color:#3a3a3a;font-size:11px;">No coverage configured</div>';
const ptBadge = node.is_me ? _ptStatus(st) : '';
return `<div class="vv-card vv-fb-node">
<div class="vv-fb-node-h">
${_stateDot(state)}
@@ -190,6 +296,7 @@ function _nodeCard(node) {
<span class="vv-fb-node-nm">${node.hostname}</span>
${covTarget}
<span style="flex:1"></span>
${ptBadge}
${_stateBadge(state)}
</div>
<hr class="vv-fb-sep">
@@ -197,30 +304,47 @@ function _nodeCard(node) {
</div>`;
}
function _setToggles(data) {
const pairs = [
['vv-fb-pt-tog', !!data.partnership_enabled],
['vv-fb-en-tog', !!data.fb_enabled],
['vv-fb-rsync-tog', !!data.fb_rsync_enabled],
];
pairs.forEach(([id, on]) => {
const el = document.getElementById(id);
if (el) el.classList.toggle('on', on);
});
}
function _setInputs(data) {
const fields = [
['vv-fb-interval', data.check_interval ?? 30],
['vv-fb-strikes', data.handback_req ?? 3],
['vv-fb-suspend', data.suspend_after ?? 120],
];
fields.forEach(([id, val]) => {
const el = document.getElementById(id);
if (el && el !== document.activeElement) el.value = val;
});
}
function _render(data) {
if (!data.fb_enabled) {
document.getElementById('vv-fb-grid').innerHTML =
'<div class="vv-fb-disabled">FALLBACK_ENABLED=false — fallback monitoring is disabled</div>';
return;
_setToggles(data);
_setInputs(data);
const grid = document.getElementById('vv-fb-grid');
if (!data.partnership_enabled) {
grid.innerHTML = '<div class="vv-fb-disabled">PARTNERSHIP_ENABLED=false — all cross-server operations disabled</div>';
} else if (!data.fb_enabled) {
grid.innerHTML = '<div class="vv-fb-disabled">FALLBACK_ENABLED=false — fallback monitoring is disabled</div>';
} else {
const nodes = data.nodes || [];
let html = _activeCard(nodes, data.handback_req || 3);
for (const node of nodes) html += _nodeCard(node, data.suspend_after || 120);
grid.innerHTML = html || '<div class="vv-fb-disabled">No nodes configured.</div>';
}
const nodes = data.nodes || [];
let html = '';
// Top: active fallback card (if any)
html += _activeCard(nodes, data.handback_req || 3);
// Per-node cards
for (const node of nodes) {
html += _nodeCard(node);
}
if (!html) {
html = '<div class="vv-fb-disabled">No nodes configured.</div>';
}
document.getElementById('vv-fb-grid').innerHTML = html;
const ts = data.ts
? new Date(data.ts * 1000).toLocaleString([], {
month:'numeric', day:'numeric', year:'numeric',
@@ -239,6 +363,55 @@ function vvFbLoad() {
});
}
window.vvFbToggle = function(track, key) {
const on = !track.classList.contains('on');
track.classList.toggle('on', on);
const fd = new FormData();
fd.append('id', 'fallback');
fd.append('changes', JSON.stringify([{ file: 'master.conf', key, value: on ? 'true' : 'false', type: 'scalar' }]));
fetch('/plugins/varaverk/api/confform.php', { method: 'POST', body: fd })
.then(r => r.json())
.then(d => { if (!d.ok) track.classList.toggle('on', !on); })
.catch(() => track.classList.toggle('on', !on));
};
window.vvFbSaveSettings = function() {
const ivEl = document.getElementById('vv-fb-interval');
const stEl = document.getElementById('vv-fb-strikes');
const suEl = document.getElementById('vv-fb-suspend');
const fb = document.getElementById('vv-fb-set-fb');
const btn = document.getElementById('vv-fb-save-btn');
const interval = parseInt(ivEl.value, 10);
const strikes = parseInt(stEl.value, 10);
const suspend = parseInt(suEl.value, 10);
if ([interval, strikes, suspend].some(n => isNaN(n) || n < 0)) {
fb.style.color = '#ef5350'; fb.textContent = 'Invalid values'; return;
}
btn.disabled = true; btn.textContent = 'Saving…'; fb.textContent = '';
const fd = new FormData();
fd.append('id', 'fallback');
fd.append('changes', JSON.stringify([
{ file: 'master.conf', key: 'FALLBACK_CHECK_INTERVAL', value: String(interval), type: 'scalar' },
{ file: 'master.conf', key: 'FALLBACK_HANDBACK_STRIKES', value: String(strikes), type: 'scalar' },
{ file: 'master.conf', key: 'FALLBACK_PARTNERSHIP_SUSPEND_AFTER', value: String(suspend), type: 'scalar' },
]));
fetch('/plugins/varaverk/api/confform.php', { method: 'POST', body: fd })
.then(r => r.json())
.then(d => {
btn.disabled = false; btn.textContent = 'Save';
fb.style.color = d.ok ? '#4caf50' : '#ef5350';
fb.textContent = d.ok ? 'Saved ✓' : (d.error || 'Failed');
if (d.ok) setTimeout(() => { fb.textContent = ''; }, 3000);
})
.catch(() => {
btn.disabled = false; btn.textContent = 'Save';
fb.style.color = '#ef5350'; fb.textContent = 'Request failed';
});
};
vvFbLoad();
setInterval(vvFbLoad, 30000);
+2 -2
View File
@@ -715,8 +715,8 @@ Change `--dry-run` to nothing and run during a maintenance window.
> forced kill of fallback.sh:
> ```bash
> # Verify first — right containers on right server, DDNS correct, fallback.sh not running
> cat /boot/config/fallback_state.db
> echo "state=NORMAL" > /boot/config/fallback_state.db
> cat "$STATE_DIR/fallback_state.db"
> echo "state=NORMAL" > "$STATE_DIR/fallback_state.db"
> ```
> Resets state only — does NOT start or stop any containers.
+1 -1
View File
@@ -621,7 +621,7 @@ SLEEP=60 # seconds between retries
ROOTFS_WARN_PCT=75 # abort if remote rootfs above this %
# Shared with Monitors/
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90
BANDWIDTH_WARN_GB=50
+1
View File
@@ -186,6 +186,7 @@ fi
detect_hosts
require_partnership
# Tier 1 global gate — Tier 2 (per-orchestrator) checked by caller
if ! check_rsync_enabled; then
+192 -238
View File
@@ -4,23 +4,24 @@ Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start.
> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) and their configuration live in
> `Plugin/unraid/System_Essentials/`. Watchdog configuration (`stability_watchdog.sh`,
> `resource_watchdog.sh`, `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives
> in `Watchdogs/Manual-Watchdogs.md`.
---
## ━━━ CONTENTS ━━━
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
- [webgui_watchdog.sh](#webgui_watchdogsh)
- [ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order](#array_start_scripts--array_stop_scripts-order)
- [conf_sync.sh](#conf_syncsh)
- [conf_cache_save.sh](#conf_cache_savesh)
- [conf_cache_restore.sh](#conf_cache_restoresh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
- [clear_logs.sh](#clear_logssh)
- [mover_stop.sh](#mover_stopsh)
- [rsync_stop.sh](#rsync_stopsh)
- [user_scripts_stop.sh](#user_scripts_stopsh)
- [server_reboot.sh](#server_rebootsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -32,109 +33,155 @@ adding or reordering scripts at array start.
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
Per-check detail suppressed.
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
**One-shot scripts** (`conf_sync.sh`, `conf_cache_save.sh`, `conf_cache_restore.sh`,
`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`, `rsync_stop.sh`,
`server_reboot.sh`): without `--log`, section headers, per-step results, and the
final summary are visible. Per-item detail suppressed.
---
## ARRAY_START_SCRIPTS Order
## ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
> scripts.** Getting it wrong causes subtle failures that don't show up
> immediately.
> **Order matters.** Scripts that set up conditions other scripts depend on must run
> first. Getting the order wrong causes subtle failures.
```bash
# master.conf
# master.conf (current order — from ARRAY_START_SCRIPTS)
ARRAY_START_SCRIPTS=(
"inotify_tuning.sh" # 1 — FIRST: kernel limits must be set before
# any container starts. Containers inherit
# inotify limits at launch, not dynamically.
"docker_syslog_filter.sh" # 2 — SECOND: before any veth interfaces are
# created. If a container starts first, its
# veth creation is already in syslog.
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
...
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # plugin API key — ephemeral registry
"System_Essentials/conf_sync.sh" # 1st cross-server op — pull partner confs into RAM cache
"System_Essentials/conf_cache_restore.sh" # fill gaps if partner was down at boot
"Transcodes/ramdisk_setup.sh" # ramdisk before Emby starts
"System_Essentials/docker_syslog_filter.sh" # before veth interfaces are created
"Plugin/unraid/System_Essentials/php_fpm_max_children.sh" # WebGUI tuning
"System_Essentials/inotify_tuning.sh" # before docker_network_connect and continuous scripts
"Docker_Essentials/docker_network_connect.sh" # ensure networks + container connections
"Tools/claude_startup.sh" # persist Claude data to appdata; re-symlink on boot
"Media/start_webhook_listener.sh" # arr upgrade webhook — continuous
"Fallback/fallback.sh" # mutual failover — continuous
)
ARRAY_STOP_SCRIPTS=(
"System_Essentials/conf_cache_save.sh" # FIRST: snapshot RAM cache while fresh
"Plugin/unraid/System_Essentials/user_scripts_stop.sh"
"Fallback/fallback.sh --stop"
"System_Essentials/rsync_stop.sh --rsync-only"
"Plugin/unraid/System_Essentials/mover_stop.sh"
"Docker_Essentials/docker_container_stop.sh"
)
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
```
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
Why conf_sync before conf_cache_restore: the sync runs first to get the freshest
possible partner conf. The restore only fills in for confs that the sync couldn't
fetch — it is a fallback, not the primary path.
Why docker_syslog_filter SECOND: The filter must be in place before any container
starts creating veth interfaces. The first container start after array start
generates veth messages — these will appear in syslog if the filter isn't active.
Why docker_syslog_filter before inotify: the filter must be in place before any
container creates veth interfaces. inotify must be set before docker_network_connect
and the continuous scripts (start_webhook_listener, fallback), which are the first
steps that actually touch containers.
Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at
`/tmp/.cache/vv/d/` is wiped on reboot. Saving it must happen before anything
else shuts down or changes state.
---
## webgui_watchdog.sh
## conf_sync.sh
### Escalation Logic
### Modes
```
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
Not responding:
1. platform_restart_service nginx
wait WEBGUI_NGINX_WAIT (15s) → recheck
→ recovered: notify, exit 0
2. platform_restart_service php-fpm
wait WEBGUI_PHP_WAIT (10s) → recheck
→ recovered: notify, exit 0
3. platform_restart_service emhttp
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
→ recovered: notify, exit 0
All three failed → notify warning, exit 1
conf_sync.sh Full sync: pull from all partners + push to all partners
conf_sync.sh --push-only Push own conf to all partners (fast — for conf-save hook)
conf_sync.sh --pull-only Pull partner confs only (intermediate orchestrator)
conf_sync.sh --dry-run Show what would happen, no changes
conf_sync.sh --log Verbose output
```
### Configuration
### What It Syncs
- **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk
via SCP → writes to local `/tmp/.cache/vv/d/${partner_id}.conf`
- **Push**: sends own `Configurations/${my_id}.conf` to partner's
`/tmp/.cache/vv/d/${my_id}.conf` via SCP
- **Own conf in local cache**: copies own conf to `/tmp/.cache/vv/d/${my_id}.conf`
on full sync (so the cache has a complete snapshot of all confs)
Only partner confs are sourced from cache — `load_config.sh` always reads own conf
from disk to avoid sourcing a stale cached copy.
### Remote SCRIPTS_DIR Resolution
The pull path reads the partner's `/boot/config/plugins/varaverk/varaverk.cfg` to
find their actual `SCRIPTS_DIR` before building the SCP path. This handles the case
where the partner is in appdata storage mode and their conf is at
`/mnt/user/appdata/Varaverk/Configurations/` rather than the internal path.
### PARTNERSHIP_ENABLED Gate
conf_sync.sh calls `require_partnership` — if `PARTNERSHIP_ENABLED=false`, it exits
silently with 0. The conf cache will be empty for partner confs while partnership
is disabled.
### If Conf Pull Fails at Boot
If the partner is unreachable, `conf_sync.sh` logs a warning and exits 1.
`conf_cache_restore.sh` then runs (next in ARRAY_START_SCRIPTS) and loads the
persistent backup from `$PERSISTENT_CONF_CACHE` if available.
A notification fires if any partner fails — check partner reachability via Tailscale.
---
## conf_cache_save.sh
### What It Does
At array stop, copies all partner confs from `/tmp/.cache/vv/d/` to
`$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives
the reboot and is used by `conf_cache_restore.sh` at next array start if the sync
can't reach the partner.
`conf_cache_watchdog.sh` (in `Watchdogs/System/`) refreshes this backup every
15 minutes while the partner is offline — keeping it current even during extended
outages.
### PARTNERSHIP_ENABLED Gate
Exits silently when `PARTNERSHIP_ENABLED=false`. No backup is written.
### Usage
```bash
WEBGUI_URL="http://localhost" # URL to check
WEBGUI_TIMEOUT=5 # curl timeout in seconds
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
conf_cache_save.sh # save partner confs from RAM cache (called at array stop)
conf_cache_save.sh --dry-run # show what would be saved
conf_cache_save.sh --log # verbose output
```
### WebGUI Frozen — Manual Recovery
---
## conf_cache_restore.sh
### What It Does
At array start (after `conf_sync.sh`), checks which partner confs are missing from
`/tmp/.cache/vv/d/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
if a backup exists there.
Always removes the persistent backup when done — whether used or not. On a normal
reboot where the partner was reachable, the sync already populated the cache and the
backup is cleaned up without being used.
### PARTNERSHIP_ENABLED Gate
Exits silently when `PARTNERSHIP_ENABLED=false`.
### Usage
```bash
# Check which services are running:
webgui_watchdog.sh --status
# Try manual restart sequence (mirrors what the script calls):
# Source the ecosystem first to get platform functions:
source /boot/config/plugins/varaverk/load_config.sh
platform_restart_service nginx
# wait 15s, then:
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
# If nginx did not fix it, php-fpm:
platform_restart_service php-fpm
# If still down, emhttp:
platform_restart_service emhttp
# Raw equivalents (no source needed — paste directly into terminal):
# /etc/rc.d/rc.nginx restart
# /etc/rc.d/rc.php-fpm restart
# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
# If all three failed:
server_reboot.sh --status # check for active sessions first
conf_cache_restore.sh # restore missing confs from backup (called at array start)
conf_cache_restore.sh --dry-run # show what would be restored
conf_cache_restore.sh --log # verbose output
```
---
@@ -176,48 +223,6 @@ inotify_tuning.sh --log
---
## php_fpm_max_children.sh
### What It Sets
```bash
PHP_MAX_CHILDREN=250 # target pm.max_children (default: 4-8 on unRAID)
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
```
250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
The default of 48 saturates immediately under load on a busy server.
### Verify
```bash
php_fpm_max_children.sh --status
# Shows current value vs target, PHP-FPM worker count
# Manual verify:
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
# Expected: pm.max_children = 250
```
### If WebGUI Is Slow Despite the Setting
```bash
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
# pm.max_children value may still be too low, or there's a PHP worker leak.
# Check running worker count:
pgrep -fc php-fpm
# Compare to pm.max_children — if equal, workers are saturated
# Increase if needed:
# master.conf: PHP_MAX_CHILDREN=350
# Then: php_fpm_max_children.sh --log (will update and restart php-fpm)
```
---
## docker_syslog_filter.sh
### What It Creates
@@ -230,8 +235,7 @@ pgrep -fc php-fpm
```
This drops any syslog message containing "veth" or "docker0" before it reaches
any output target, including the log file. The drop rule is applied at rsyslog
level — not at the log viewer level.
any output target, including the log file.
### Verify
@@ -298,39 +302,6 @@ du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
---
## mover_stop.sh
### Stop Sequence
```
1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
2. Wall message to all logged-in terminal users
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
4. SIGTERM — mover finishes its current file operation, then stops
5. Wait 5 seconds → verify stopped
6. SIGKILL if still running — forced stop, partial files possible
7. Final verify — error if still running after SIGKILL
```
SIGTERM first because the mover can finish the file it is currently moving,
leaving no partial copies split across cache and array. SIGKILL is a last resort.
### Configuration
```bash
MOVER_STOP_TIMEOUT=30 # seconds between wall warning and SIGTERM
```
### Usage
```bash
mover_stop.sh # check and stop if running
mover_stop.sh --status # show current mover state and PID
mover_stop.sh --dry-run # show what would happen without stopping
```
---
## rsync_stop.sh
### Auto-Detection Logic
@@ -355,8 +326,8 @@ After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTA
for any that were stopped by the interrupted rsync session and restarts them.
Remote containers are left for docker_watchdog.sh to recover.
Skip container recovery with `--rsync-only` — used when called by other scripts
that handle recovery themselves.
Skip container recovery with `--rsync-only` — used when called by array_stopping.sh
(which handles container stop separately via docker_container_stop.sh).
### Usage
@@ -371,34 +342,6 @@ rsync_stop.sh --full-stop --dry-run # preview full stop
---
## user_scripts_stop.sh
### Process Identification
Scans `/proc/*/cmdline` for any process whose command line contains
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
what the script is named or how it was launched.
Script names are extracted from the path for display: you see which scripts are
being stopped, not just PIDs.
### Self-Exclusion
If this script is run via the User Scripts plugin, it would find its own PID in
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
from the kill list.
### Usage
```bash
user_scripts_stop.sh # stop all User Script processes
user_scripts_stop.sh --status # show running scripts with names and elapsed time
user_scripts_stop.sh --dry-run # show what would be stopped
```
---
## server_reboot.sh
### Full Shutdown Sequence
@@ -422,11 +365,12 @@ user_scripts_stop.sh --dry-run # show what would be stopped
6. /etc/rc.d/rc.libvirt stop (VM Manager)
7. Orchestrators/array_stopping.sh — safe ordered array stop:
- user_scripts_stop.sh stop background User Scripts
- fallback.sh --stop graceful fallback teardown
- rsync_stop.sh --rsync-only kill active rsync transfers
- mover_stop.sh stop mover
- docker_container_stop.sh stop all containers gracefully
- conf_cache_save.sh snapshot partner conf RAM cache
- user_scripts_stop.sh stop background User Scripts
- fallback.sh --stop graceful fallback teardown
- rsync_stop.sh --rsync-only kill active rsync transfers
- mover_stop.sh stop mover
- docker_container_stop.sh stop all containers gracefully
8. sync (flush filesystem buffers to disk)
@@ -446,7 +390,6 @@ For a clean reboot when services are active:
```bash
rsync_stop.sh # stop any active rsync (smart mode)
mover_stop.sh # stop mover gracefully
server_reboot.sh --status # check what's still running
server_reboot.sh --reason="planned maintenance"
```
@@ -464,27 +407,17 @@ server_reboot.sh --reason="disk work" # include reason in notification
## Full Configuration Reference
> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
```bash
# master.conf
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30
# ── Conf Sync ──────────────────────────────────────────────────────────────────
CONF_SYNC_ENABLED=true # toggle: false disables conf_sync.sh entirely
# ── inotify Tuning ─────────────────────────────────────────────────────────────
INOTIFY_MAX_INSTANCES=1024
INOTIFY_MAX_WATCHES=1048576
INOTIFY_MAX_QUEUED_EVENTS=32768
# ── PHP-FPM ────────────────────────────────────────────────────────────────────
PHP_MAX_CHILDREN=250
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
INOTIFY_WARN_PCT=80 # warn (in reports) if instances exceed this % of limit
# ── Syslog Filter ──────────────────────────────────────────────────────────────
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
@@ -494,20 +427,48 @@ LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
LOG_MIN_SIZE_MB=10
LOG_DOCKER_MAX_MB=100
# ── Mover Stop ─────────────────────────────────────────────────────────────────
MOVER_STOP_TIMEOUT=30
# ── Server Reboot ──────────────────────────────────────────────────────────────
REBOOT_SLEEP=30
REBOOT_VM_WAIT=30
# ── State Paths (auto-adapt to storage mode) ───────────────────────────────────
# STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
# They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
# (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
STATE_DIR="${SCRIPTS_DIR}/State_Files"
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
```
---
## Troubleshooting
> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
### Partner Conf Not In RAM Cache After Boot
```bash
# Check what's in the RAM cache:
ls -la /tmp/.cache/vv/d/
# Check what's in the persistent backup:
ls -la "$PERSISTENT_CONF_CACHE/" # set SCRIPTS_DIR first or use full path
# Re-run conf sync manually:
/boot/config/plugins/varaverk/System_Essentials/conf_sync.sh --log
# If partner is reachable but pull failed, check SSH key:
ssh -i /path/to/ssh_key root@<partner-tailscale-ip> "echo ok"
```
### Persistent Backup Is Stale or Empty
```bash
# conf_cache_watchdog.sh refreshes the backup while partner is offline.
# Check watchdog state:
cat "$STATE_DIR/conf_cache_watchdog_state.db"
# Force a manual backup from current RAM cache:
/boot/config/plugins/varaverk/System_Essentials/conf_cache_save.sh --log
```
### rsync_stop Killed the Wrong Thing
@@ -521,36 +482,29 @@ rsync_stop.sh --dry-run # shows smart mode decision
rsync_stop.sh --full-stop --dry-run # shows full-stop decision
```
### WebGUI Recovery After All Three Steps Failed
### inotify Exhaustion After Boot
```bash
# Check if processes are running:
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
pgrep emhttpd && echo "emhttp: yes" || echo "emhttp: no"
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
# Verify limits are applied:
sysctl fs.inotify.max_user_watches # expect 1048576
sysctl fs.inotify.max_user_instances # expect 1024
# Check recent nginx errors:
cat /var/log/nginx/error.log | tail -20
# If not set — run manually:
inotify_tuning.sh --log
# Check emhttp log:
tail -20 /var/log/syslog | grep emhttp
# Last resort — reboot:
server_reboot.sh --reason="WebGUI unrecoverable"
# Check current usage:
inotify_tuning.sh --status
```
### PHP-FPM Config Not Found After unRAID Update
unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
errors with "config file not found":
### Syslog Still Noisy After Array Start
```bash
# Find the new config path:
find /etc -name "www.conf" 2>/dev/null
# Check filter is in place:
docker_syslog_filter.sh --status
# Update PHP_CONF in master.conf:
PHP_CONF="/etc/php84/php-fpm.d/www.conf" # example for php84
# Re-apply if needed:
docker_syslog_filter.sh --log
# Verify with:
php_fpm_max_children.sh --status
# Restart rsyslog to pick up the filter:
/etc/rc.d/rc.rsyslogd restart
```
+79 -87
View File
@@ -1,35 +1,18 @@
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
**System-level scripts that act on the server itself — not containers,
not media, not monitoring.** Keeping the server stable under load, recovering a
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
shutdowns with proper warning sequences.
not media, not monitoring.** Kernel limits, log hygiene, conf synchronisation
between servers, and graceful shutdowns with proper warning sequences.
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) live in `Plugin/unraid/System_Essentials/`
> because they call Unraid-specific service commands and paths. This folder
> contains scripts that would run unchanged on any Linux host.
---
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
**Server Getting Into Unstable States With No Recovery Path**
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
watchdog tries to restart the container — but Docker itself is barely responding.
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
write log files. The server is functionally dead but still technically running.
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
threshold breaches. Last line of defense before a hard crash.
**WebGUI Freezing and Nobody Noticing**
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
load, settings don't save, containers can't be started or stopped via the UI. No
container-level alert fires because this isn't a container problem — it's a web
server problem. By the time someone notices it may have been broken for hours.
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
php-fpm → emhttp. Lightest fix first. Silent when healthy.
**50+ Containers Starting and Filling Syslog With Veth Noise**
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
network interface for each one. Each interface generates multiple syslog entries.
@@ -40,68 +23,79 @@ In the first minute after array start, syslog is buried under 200400 lines of
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
starts. Applied at array start. Idempotent — silent when already correct.
**WebGUI Queuing Requests Under Load Without Explanation**
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 48 workers. With
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
those workers saturate immediately. New requests queue behind active ones.
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
**inotify Exhaustion Producing Unexplained Failures**
When inotify limits are exhausted, containers silently stop receiving filesystem
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
file changes." Code-Server with node_modules alone can consume 100K200K watches,
and all containers share the same pool.
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
run FIRST in ARRAY_START_SCRIPTS before any containers start.
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start, before
any container-connecting or continuous scripts start.
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
The mover is running — moving a large batch of files from cache to array. A reboot
is triggered. The mover stops mid-file. The file exists partially on both cache and
array simultaneously. unRAID's deduplication layer is confused.
**Partner Conf Lost Across Reboots When Partner is Down**
Scripts like `fallback.sh` need the partner's conf vars (credentials, container
names, tier delays) to operate. The partner conf lives in a RAM cache at
`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls
a fresh copy from the partner. But if the partner is offline at boot time, the
pull fails and fallback has no partner vars to work with.
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
Fix: `conf_cache_save.sh` + `conf_cache_restore.sh` — snapshot the RAM cache to
`$PERSISTENT_CONF_CACHE` on array stop; reload on next start for any confs the
sync couldn't fetch.
**Mover/Rsync Interruption During Reboot**
Rsync transfers or mover runs are in progress when a reboot is triggered. Stopping
them uncleanly leaves partial files.
Fix: `server_reboot.sh` — orchestrates `array_stopping.sh` which stops rsync, mover,
and containers in the correct order before calling `/sbin/reboot`.
---
## ━━━ WHAT THIS FOLDER DOES ━━━
```
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
Kernel tuning inotify_tuning.sh — file watch limits
php_fpm_max_children.sh — PHP worker count
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
clear_logs.sh — weekly log trimming
Graceful operations mover_stop.sh — clean mover stop
rsync_stop.sh — smart rsync stop (orchestrator-aware)
user_scripts_stop.sh stop running User Scripts
clear_logs.sh — size-threshold log trimming
Conf synchronisation conf_sync.sh pull/push partner confs → RAM cache
conf_cache_save.sh — snapshot RAM cache → persistent at stop
conf_cache_restore.sh — reload from snapshot at start (offline partner)
Graceful operations rsync_stop.sh — smart rsync stop (orchestrator-aware)
server_reboot.sh — clean reboot with pre-flight warnings
```
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
---
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
```
Orchestrators/
array_started.sh ────────────────────────inotify_tuning.sh (first in sequence)
────────────────────────► docker_syslog_filter.sh (second)
────────────────────────► php_fpm_max_children.sh
array_started.sh ────────────────────────► conf_sync.sh (pulls partner confs)
────────────────────────► conf_cache_restore.sh (fills gaps if partner down)
────────────────────────► docker_syslog_filter.sh (before containers start)
────────────────────────► inotify_tuning.sh (before docker_network_connect)
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
weekly_maintenance.sh ──────────────────► clear_logs.sh
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
Watchdogs/System/
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
(runs every 15 min via watchdog_orchestrator.sh)
Watchdogs/
stability_watchdog.sh and resource_watchdog.sh now live here.
See Watchdogs/README-Watchdogs.md for how they relate to each other
and to docker_watchdog.sh and System/storage_watchdog.sh.
System_Essentials/
server_reboot.sh ────────────────────────► array_stopping.sh (via Orchestrators/)
────────────────────────► mover_stop.sh, user_scripts_stop.sh
(via Plugin/unraid/System_Essentials/)
Plugin/unraid/System_Essentials/
php_fpm_max_children.sh — WebGUI tuning (Unraid-specific PHP paths)
mover_stop.sh — clean mover stop (Unraid mover daemon)
user_scripts_stop.sh — stop User Scripts plugin processes
unraid_api_key_renew.sh — Varaverk plugin API key renewal
Plugin/unraid/Watchdogs/System/
webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
```
---
@@ -110,14 +104,13 @@ Watchdogs/
| Script | Role | When It Runs |
|--------|------|-------------|
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — before container ops |
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
---
@@ -125,27 +118,28 @@ Watchdogs/
## ━━━ HOW THE SCRIPTS RELATE ━━━
```
Array starts
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
php_fpm_max_children.sh ← before WebGUI is under load
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
│ push own conf to partner's /tmp/.cache/vv/d/
conf_cache_restore.sh ← if partner was down: load last-known-good conf from
│ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live)
└─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts
Every minute (watchdog_orchestrator.sh in Orchestrators/):
→ Watchdogs/resource_watchdog.sh
→ Watchdogs/docker_watchdog.sh
→ Watchdogs/system_watchdog.sh (thin orchestrator)
└─ Watchdogs/System/storage_watchdog.sh
└─ Watchdogs/System/webgui_watchdog.sh
WebGUI OK → silent exit
Not responding:
Step 1: restart nginx → recheck
Step 2: restart php-fpm → recheck
Step 3: restart emhttp → recheck
All failed → notify, exit 1
└─ Watchdogs/System/network_watchdog.sh
→ Watchdogs/stability_watchdog.sh
(see Watchdogs/README-Watchdogs.md for full flow)
Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
│ while RAM cache is still fresh
├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
└─ ...other stop scripts...
Every 15 minutes (watchdog_orchestrator.sh):
→ Watchdogs/System/conf_cache_watchdog.sh
If partner is offline and persistent backup is stale → refresh from last RAM cache
Every 4 hours (intermediate_sync_maintenance.sh):
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
Weekly (weekly_maintenance.sh):
└─ clear_logs.sh
@@ -153,8 +147,6 @@ Weekly (weekly_maintenance.sh):
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
Manual operations:
mover_stop.sh wall → SIGTERM → SIGKILL → verify stopped
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
rsync_stop.sh → detect orchestratorkill rsync (or orchestrator+rsync)
server_reboot.sh → pre-flight → array_stopping.sh → reboot
```
+78
View File
@@ -0,0 +1,78 @@
#!/bin/bash
# ==============================================================================================
# ============================= Conf Cache Restore =============================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Runs at array start after conf_sync.sh. Checks whether partner confs made it
# into the RAM cache. If any are missing (partner was unreachable at boot) and
# a persistent backup exists from the previous shutdown, loads the missing confs
# from backup into the RAM cache so scripts like fallback.sh have partner vars.
#
# Always removes the persistent backup when done — used or not.
#
# Normal reboot (partner up):
# conf_sync.sh pulls fresh → RAM cache complete → backup not needed → removed
#
# Edge case (partner down at boot):
# conf_sync.sh fails to pull → RAM cache missing partner → backup loaded into
# RAM → backup removed → fallback.sh runs with last-known-good partner vars
#
# Own conf is never in the backup — it's always on disk.
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
detect_hosts
require_partnership
RAM_CACHE="/tmp/.cache/vv/d"
SAVE_DIR="$PERSISTENT_CONF_CACHE"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
if [[ ! -d "$SAVE_DIR" ]]; then
log "No persistent conf backup found — nothing to restore"
exit 0
fi
restored=0
for conf in "$SAVE_DIR"/host*.conf; do
[[ -f "$conf" ]] || continue
base="$(basename "$conf")"
[[ "${base,,}" == "${MY_ID,,}.conf" ]] && continue
if [[ -f "$RAM_CACHE/$base" ]]; then
log "$base already in RAM cache (conf_sync succeeded) — skipping"
continue
fi
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would restore $base → RAM cache"
(( restored++ ))
continue
fi
mkdir -p "$RAM_CACHE"
if cp "$conf" "$RAM_CACHE/$base"; then
echo "Restored $base from persistent backup → RAM cache ✅"
(( restored++ ))
else
warn "Failed to restore $base"
fi
done
[[ "$restored" -gt 0 ]] && \
warn "Loaded $restored partner conf(s) from persistent backup — partner was unreachable at array start"
if [[ "$DRY_RUN" == false ]]; then
rm -rf "$SAVE_DIR"
log "Persistent conf backup cleared"
else
warn "DRY RUN — would remove $SAVE_DIR"
fi
exit 0
+63
View File
@@ -0,0 +1,63 @@
#!/bin/bash
# ==============================================================================================
# ============================= Conf Cache Save ================================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Snapshots the partner conf RAM cache to $PERSISTENT_CONF_CACHE on
# array stop. Survives reboot. Used by conf_cache_restore.sh at next array start
# to reload partner vars into RAM when the partner is unreachable at boot time.
#
# Runs as the first step in ARRAY_STOP_SCRIPTS — while the RAM cache is fresh
# and before anything else changes. If partner is reachable at next start,
# conf_sync.sh gets a fresh copy and the backup is never used. If partner is
# down, the backup fills the gap so fallback.sh has the vars it needs.
#
# Only partner confs are saved — own conf is always on disk.
# Path adapts to storage mode: $SCRIPTS_DIR/.cache/vv/d (internal or appdata).
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
detect_hosts
require_partnership
RAM_CACHE="/tmp/.cache/vv/d"
SAVE_DIR="$PERSISTENT_CONF_CACHE"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be written"
if [[ ! -d "$RAM_CACHE" ]]; then
log "No RAM conf cache found — nothing to save"
exit 0
fi
saved=0
for conf in "$RAM_CACHE"/host*.conf; do
[[ -f "$conf" ]] || continue
base="$(basename "$conf")"
[[ "${base,,}" == "${MY_ID,,}.conf" ]] && continue
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would save $base$SAVE_DIR/"
(( saved++ ))
continue
fi
mkdir -p "$SAVE_DIR"
if cp "$conf" "$SAVE_DIR/$base"; then
echo "Saved $base$SAVE_DIR"
(( saved++ ))
else
warn "Failed to save $base"
fi
done
if [[ "$saved" -eq 0 ]]; then
log "No partner confs in RAM cache — nothing saved"
else
echo "Conf cache saved: $saved partner conf(s) → $SAVE_DIR"
fi
+32 -11
View File
@@ -5,21 +5,24 @@
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Maintains a RAM-resident conf cache at /tmp/.vv/config/cached/.confs/.
# Maintains a RAM-resident conf cache at /tmp/.cache/vv/d/.
# Credentials and partner keys live in RAM only — never on disk across hosts.
#
# On array start (default / --array-start):
# 1. Copy own conf to local cache
# 2. Pull each available partner's conf from their disk → local cache
# 3. Push own conf to each available partner's /tmp/.vv/ cache
# 3. Push own conf to each available partner's /tmp/.cache/vv/d/ cache
#
# On conf save (--push-only):
# Fast path — push updated own conf to all partners' /tmp/.vv/ cache only.
# Fast path — push updated own conf to all partners' /tmp/.cache/vv/d/ only.
# No pulls, no local cache rebuild.
#
# Cache is /tmp (tmpfs) — cleared every reboot, repopulated by this script
# on next array start. Scripts source from cache for partner vars; own vars
# always come from disk (load_config.sh skips cached copy of own conf).
# Cache is /tmp/.cache/vv/d (tmpfs) — cleared every reboot, repopulated by
# this script on next array start. Scripts source from cache for partner vars;
# own vars always come from disk (load_config.sh skips cached copy of own conf).
#
# Pull path resolves the remote's SCRIPTS_DIR from their varaverk.cfg so it
# works whether the remote is in internal or appdata storage mode.
#
# ==============================================================================================
# RUNTIME MODES
@@ -48,16 +51,33 @@ for arg in "$@"; do
done
parse_args "${FILTERED_ARGS[@]}"
detect_hosts
require_partnership
if [[ "${CONF_SYNC_ENABLED:-true}" == false ]]; then
log "CONF_SYNC_ENABLED=false — skipping"
exit 0
fi
CACHE_DIR="/tmp/.vv/config/cached/.confs"
MY_CONF="$SCRIPTS_ROOT/Configurations/${MY_ID,,}.conf"
CACHE_DIR="/tmp/.cache/vv/d"
MY_CONF="$SCRIPTS_DIR/Configurations/${MY_ID,,}.conf"
SSH_TIMEOUT=10
# Reads the remote's varaverk.cfg to find their actual SCRIPTS_DIR.
# Handles the case where the remote is in appdata storage mode.
_remote_scripts_dir() {
local ip="$1"
local cfg line sd
cfg=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
"root@${ip}" "cat /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null" 2>/dev/null) || true
while IFS= read -r line; do
[[ "$line" == SCRIPTS_DIR=* ]] || continue
sd="${line#SCRIPTS_DIR=}"; sd="${sd//\"/}"; sd="${sd//\'/}"
echo "$sd"; return
done <<< "$cfg"
echo "/boot/config/plugins/varaverk"
}
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
# ── Ensure cache dir exists ───────────────────────────────────────────────────
@@ -100,7 +120,8 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
# ── Pull: grab partner's conf from their disk → our local cache ──────────
if [[ "$PUSH_ONLY" == false ]]; then
remote_conf="${SCRIPTS_DIR}/Configurations/${partner_slot}.conf"
remote_sd=$(_remote_scripts_dir "$partner_ip")
remote_conf="${remote_sd}/Configurations/${partner_slot}.conf"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would pull $partner_host:$remote_conf$CACHE_DIR/${partner_slot}.conf"
elif timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
@@ -115,13 +136,13 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
fi
fi
# ── Push: send own conf to partner's /tmp/.vv/ cache ────────────────────
# ── Push: send own conf to partner's /tmp/.cache/vv/d/ cache ───────────
if [[ "$PULL_ONLY" == true ]]; then
continue
fi
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would push ${MY_ID,,}.conf → $partner_host:/tmp/.vv/config/cached/.confs/"
warn "DRY RUN — would push ${MY_ID,,}.conf → $partner_host:/tmp/.cache/vv/d/"
continue
fi
+1 -1
View File
@@ -214,7 +214,7 @@ View and manage the persistent container skip list used by `docker_watchdog.sh`.
```
docker_watchdog.sh restarts the same container N times within the rolling window
→ container added to skip list on /boot/config/
→ container added to skip list in $STATE_DIR
→ critical notification sent
→ watchdog stops touching it entirely
+55 -15
View File
@@ -13,8 +13,9 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
- [docker_watchdog.sh](#docker_watchdogsh)
- [system_watchdog.sh](#system_watchdogsh)
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) ← Plugin/unraid/Watchdogs/System/
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
- [System/conf_cache_watchdog.sh](#systemconf_cache_watchdogsh)
- [stability_watchdog.sh](#stability_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -373,6 +374,10 @@ storage_watchdog.sh --log # verbose per-container output
## System/webgui_watchdog.sh
> **Lives in `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh`** — calls Unraid-specific
> service commands (`rc.nginx`, `rc.php-fpm`, `emhttp`) via the platform adapter. Called by
> `system_watchdog.sh` via `SYSTEM_WATCHDOG_SCRIPTS` in master.conf.
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
through three restart steps if unresponsive. Silent when healthy.
@@ -446,7 +451,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
@@ -506,6 +511,40 @@ tailscale up
---
## System/conf_cache_watchdog.sh
Called by `system_watchdog.sh` each cycle. Maintains the persistent partner conf backup
at `$PERSISTENT_CONF_CACHE` while the partner is offline.
### Behaviour
**Remote online:** removes the persistent backup if one exists. It is not needed —
`conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to
`$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
an extended outage. If this host reboots while the partner is still down,
`conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
valid partner vars.
Silent when remote is online and no backup exists (the normal steady state).
### Gates
- `FALLBACK_ENABLED=false` → no-op (no fallback means no need for partner vars)
- `CONF_SYNC_ENABLED=false` → no-op
- `PARTNERSHIP_ENABLED=false` → exits silently (require_partnership gate)
### Usage
```bash
conf_cache_watchdog.sh # single pass (called by system_watchdog.sh)
conf_cache_watchdog.sh --dry-run # show what would be written or removed
conf_cache_watchdog.sh --log # verbose output
```
---
## stability_watchdog.sh
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
@@ -653,9 +692,9 @@ WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
WATCHDOG_BATCH_NOTIFY=true
# State files:
WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
# State files (all paths adapt to storage mode via STATE_DIR / DATA_DIR):
WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
# host*.conf
HOST1_WATCHDOG_CONTAINERS=() # "ContainerName:LimitMB"
@@ -670,8 +709,8 @@ WATCHDOG_APPDATA_GROWTH_GB=2
WATCHDOG_APPDATA_LOG_MAX_GB=2
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
WATCHDOG_APPDATA_STRIKE_LIMIT=3
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"
# host*.conf (optional — only for suppress ceilings)
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
@@ -701,11 +740,12 @@ SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=false
SYS_WATCHDOG_ABORT_ON_MOVER=false
# State files (all in State_Files/ — survive reboots):
SYS_WATCHDOG_STATE_FILE="State_Files/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="State_Files/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="State_Files/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="State_Files/system_watchdog_oom.db"
# State files all in $STATE_DIR (survive reboots; adapt to storage mode):
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"
# ── Network Watchdog ───────────────────────────────────────────────────────────
NETWORK_WATCHDOG_ENABLED=true
@@ -714,7 +754,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
@@ -781,8 +821,8 @@ storage_watchdog.sh --status
### stability_watchdog Rebooted Unexpectedly
```bash
# Check the reboot log (survives reboots):
cat /boot/config/system_watchdog_reboots.db
# Check the reboot log (survives reboots — in State_Files/):
cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot
# Check syslog near the reboot time:
+46 -11
View File
@@ -3,8 +3,8 @@
**Four tiers that run every 15 minutes through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → check system components →
reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
System component checks (storage, WebGUI, network) run inside a thin `system_watchdog.sh`
orchestrator that can also be run standalone.
System component checks (storage, WebGUI, network, conf cache) run inside a thin
`system_watchdog.sh` orchestrator that can also be run standalone.
---
@@ -57,6 +57,15 @@ Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kern
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
sequence so container healing and pressure reduction always get a chance first.
**Partner Conf Backup Going Stale During a Long Outage**
The partner goes offline overnight. On the next reboot (planned maintenance), the
persistent conf backup saved at shutdown is stale — it reflects the state from before
the outage, not the most recent live values. `conf_cache_restore.sh` loads it
faithfully, but fallback.sh may be working with old tier delays or container names.
Fix: `conf_cache_watchdog.sh` — refreshes the persistent backup from the RAM cache
every 15 minutes while the partner is offline, keeping it current throughout the outage.
---
## ━━━ WHAT THIS FOLDER DOES ━━━
@@ -66,13 +75,17 @@ Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
```
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
System components system_watchdog.sh — thin orchestrator: system component health
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
network_watchdog.sh — internet, DDNS sync, Tailscale, NPM proxy
conf_cache_watchdog.sh — maintain persistent partner conf backup
Last resort stability_watchdog.sh — reboot only when nothing else can recover
```
> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
> but lives in the plugin tree because it calls Unraid-specific service commands
> (nginx, php-fpm, emhttp via platform adapter). See `Plugin/unraid/Watchdogs/System/`.
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
attempts restarts — containers restarted into a pressure-bound system just fail again.
System component checks run after containers are healed. Stability watchdog runs last —
@@ -96,8 +109,16 @@ Orchestrators/
Watchdogs/System/ ← called by system_watchdog.sh each cycle
storage_watchdog.sh pool growth rate + runaway log detection
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
network_watchdog.sh internet reachability, DDNS sync, Tailscale, NPM proxy
conf_cache_watchdog.sh maintain persistent partner conf backup during outages
Plugin/unraid/Watchdogs/System/
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
(Unraid-specific platform calls — lives in plugin tree)
Plugin/unraid/System_Essentials/
unraid_api_key_renew.sh re-registers the Varaverk plugin API key each watchdog cycle
(inserted between docker_watchdog and stability_watchdog)
Tools/
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
@@ -106,6 +127,10 @@ Tools/
Docker_Essentials/
All container lifecycle scripts (daily restart, updates, network) — unaffected.
docker_watchdog.sh coordinates with them via shared state, not direct calls.
System_Essentials/
conf_cache_save.sh / conf_cache_restore.sh — at-stop/at-start bookends for the
persistent backup that conf_cache_watchdog.sh keeps current in between.
```
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
@@ -119,14 +144,15 @@ Docker_Essentials/
|--------|------|-----------|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every 15 min |
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every 15 min |
| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
| `system_watchdog.sh` | Thin orchestrator — runs SYSTEM_WATCHDOG_SCRIPTS in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every 15 min |
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every 15 min |
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every 15 min |
| `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every 15 min |
| `System/conf_cache_watchdog.sh` | Refresh persistent partner conf backup while partner is offline | `system_watchdog.sh` — every 15 min |
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
> but is not in this folder. `watchdog_orchestrator.sh` is in `Orchestrators/`.
> `watchdog_skip_list_manager.sh` is in `Tools/`. Neither is a watchdog.
---
@@ -166,7 +192,7 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
│ └─ webgui_watchdog.sh
│ └─ Plugin/unraid/Watchdogs/System/webgui_watchdog.sh
│ curl check → WebGUI responding → exit 0 (silent)
│ not responding → nginx restart → wait → recheck
│ still down → php-fpm restart → wait → recheck
@@ -179,6 +205,13 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
│ Tailscale: status Running → pass; not running = notify (no auto-restart)
│ NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
│ └─ conf_cache_watchdog.sh
│ remote online → remove persistent backup (conf_sync gets fresh on next boot)
│ remote offline → refresh backup from RAM cache → backup stays current
│ silent when remote is online and no backup exists (normal state)
│ (+ Plugin/unraid/System_Essentials/unraid_api_key_renew.sh — between steps 3 and 4)
Step 4 — stability_watchdog.sh
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
Tier 1 CRITICAL → immediate reboot (no strikes)
@@ -195,7 +228,9 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `DOCKER_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
+70
View File
@@ -0,0 +1,70 @@
#!/bin/bash
# ==============================================================================================
# ============================= Conf Cache Watchdog ============================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Maintains the persistent partner conf backup at $PERSISTENT_CONF_CACHE.
# Runs every 15 minutes via SYSTEM_WATCHDOG_SCRIPTS.
#
# When remote is OFFLINE:
# Writes/refreshes partner confs from RAM cache → persistent location.
# Means if this host reboots while remote is still down, conf_cache_restore.sh
# can load the partner vars into RAM so fallback.sh has what it needs.
#
# When remote is ONLINE:
# Removes the persistent backup if present — not needed, conf_sync.sh will
# pull fresh on next boot.
#
# Silent when remote is online and no backup exists (normal state).
# No-op when FALLBACK_ENABLED=false or CONF_SYNC_ENABLED=false.
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../../load_config.sh"
parse_args "$@"
detect_hosts
require_partnership
[[ "${FALLBACK_ENABLED:-false}" != "true" ]] && exit 0
[[ "${CONF_SYNC_ENABLED:-true}" != "true" ]] && exit 0
[[ -z "${REMOTE_ID:-}" ]] && exit 0
RAM_CACHE="/tmp/.cache/vv/d"
SAVE_DIR="$PERSISTENT_CONF_CACHE"
if ping_remote; then
if [[ -d "$SAVE_DIR" ]]; then
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — would remove $SAVE_DIR (remote back online)" && exit 0
rm -rf "$SAVE_DIR"
log "Remote online — persistent conf backup removed"
fi
exit 0
fi
# Remote offline — write/refresh backup from RAM cache
if [[ ! -d "$RAM_CACHE" ]]; then
log "Remote offline but RAM cache not populated — nothing to back up"
exit 0
fi
saved=0
for conf in "$RAM_CACHE"/host*.conf; do
[[ -f "$conf" ]] || continue
base="$(basename "$conf")"
[[ "${base,,}" == "${MY_ID,,}.conf" ]] && continue
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would write $base$SAVE_DIR/"
(( saved++ ))
continue
fi
mkdir -p "$SAVE_DIR"
cp "$conf" "$SAVE_DIR/$base" && (( saved++ ))
done
[[ "$saved" -gt 0 ]] && log "Remote offline — partner conf backup refreshed ($saved file(s))"
exit 0
+6
View File
@@ -1924,3 +1924,9 @@ show_status() {
echo "Excludes: ${EXCLUDE_DIRS[*]:-n/a}"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
}
require_partnership() {
[[ "${PARTNERSHIP_ENABLED:-false}" == "true" ]] && return 0
log "PARTNERSHIP_ENABLED=false — skipping cross-server operation"
exit 0
}
+2 -2
View File
@@ -99,12 +99,12 @@
fi
# ━━━ Source partner confs from /tmp cache ━━━
# conf_sync.sh pulls partner host*.conf files into /tmp/.vv/config/cached/.confs/
# conf_sync.sh pulls partner host*.conf files into /tmp/.cache/vv/d/
# on array start and after any conf save. Sourcing them here makes partner vars
# (HOST2_*, HOST3_*, …) available without committing credentials to the git repo
# or violating sparse checkout — partner confs live in RAM only, cleared on reboot.
# Confs already loaded from disk are skipped — disk copy is authoritative.
_VV_CONF_CACHE="/tmp/.vv/config/cached/.confs"
_VV_CONF_CACHE="/tmp/.cache/vv/d"
if [[ -d "$_VV_CONF_CACHE" ]]; then
while IFS= read -r _conf; do
[[ -f "$_conf" ]] || continue