FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Storage-mode awareness pass + doc update for System_Essentials through Partnership

Browse Source 
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.

PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.

conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.

fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.

README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
This commit is contained in:
Gmer4Lfe
2026-06-19 19:32:39 -04:00
parent 0564580605
commit bf3e7cc2c4
35 changed files with 835 additions and 489 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -10,6 +10,7 @@ Configurations/*.bak
# ── Runtime state, data, logs ─────────────────────────────────────────────────
data/
State_Files/
.cache/
*.log
*.lock
Configurations/master.conf
+2 -1
View File
@@ -123,6 +123,7 @@
# Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no /boot/config root.
DATA_DIR="${SCRIPTS_DIR}/data"
STATE_DIR="${SCRIPTS_DIR}/State_Files"
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
# ── Version Parity ──
# Controls behaviour when local and remote unRAID versions differ.
@@ -295,7 +296,7 @@
# watchdog_orchestrator.sh — NOT launched here.
ARRAY_START_SCRIPTS=(
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # re-register Varaverk API key — registry is ephemeral
"System_Essentials/conf_sync.sh" # pull partner confs + push own conf into /tmp/.vv/ RAM cache
"System_Essentials/conf_sync.sh" # pull partner confs + push own conf into /tmp/.cache/vv/d/ RAM cache
"System_Essentials/conf_cache_restore.sh" # load partner confs from persistent backup if conf_sync couldn't reach partner
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
"System_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
Deployment/conf_populate.sh
+1 -1
View File
@@ -10,7 +10,7 @@
# EMPTY fields, never overwrites existing values unless --overwrite is passed.
#
# After populating, pushes the updated conf to all partners via conf_sync.sh
# so they have the fresh keys in their /tmp/.vv/ cache immediately.
# so they have the fresh keys in their /tmp/.cache/vv/d/ cache immediately.
#
# ==============================================================================================
# AUTO-DETECTED FIELDS
Deployment/master.conf.template
+1
View File
@@ -120,6 +120,7 @@
# Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no /boot/config root.
DATA_DIR="/boot/config/plugins/varaverk/data"
STATE_DIR="/boot/config/plugins/varaverk/State_Files"
PERSISTENT_CONF_CACHE="/boot/config/plugins/varaverk/.cache/vv/d"
# ── Version Parity ──
# Controls behaviour when local and remote unRAID versions differ.
Docker_Essentials/README-Docker_Essentials.md
+1 -1
View File
@@ -139,7 +139,7 @@ happens after an update, and you want to know when it does.
A container keeps appearing in a broken state. You SSH in and see it's stopped. You
don't know if the watchdog tried to restart it and failed, gave up and skip-listed it,
is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip
list file on `/boot/config/`, check the restart history file, check the state file —
list file in `$STATE_DIR`, check the restart history file, check the state file —
none of which have obvious formats.
The fix: `watchdog_skip_list_manager.sh` (in `Tools/`). One command to see exactly what's
Fallback/Manual-Fallback.md
+22 -16
View File
@@ -37,11 +37,11 @@ up required before handback sequence starts. **(default: 3)**
---
```
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
```
Path to the persistent state file. Lives on `/boot/` intentionally — survives reboots.
If the server was in FALLBACK state when it rebooted, it resumes FALLBACK on restart
rather than assuming everything is normal.
Path to the persistent state file. In `$STATE_DIR` — survives reboots whether storage
mode is internal (boot device) or appdata (array). If the server was in FALLBACK state
when it rebooted, it resumes FALLBACK on restart rather than assuming everything is normal.
---
@@ -172,7 +172,7 @@ daily_sync_maintenance.sh uses, in the opposite direction. No separate TIER4 lis
FALLBACK_ENABLED=true
FALLBACK_CHECK_INTERVAL=30
FALLBACK_HANDBACK_STRIKES=3
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
FALLBACK_RSYNC_ENABLED=true
EXTERNAL_IP=8.8.8.8
FALLBACK_TEST_BLOCK_WAIT=60
@@ -293,7 +293,10 @@ FALLBACK_HOST2_WRITEBACK_TIER1=(
## ━━━ STATE FILE REFERENCE ━━━
Location: `/boot/config/fallback_state.db` (survives reboots)
Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata)
> In a shell where load_config.sh is not sourced, use the full path:
> `/boot/config/plugins/varaverk/State_Files/fallback_state.db` (internal storage mode)
```
state=NORMAL # NORMAL | FALLBACK | NO_INTERNET | DARK
@@ -304,8 +307,8 @@ tier3_started=false # whether Tier 3 containers started
tier4_started=false # whether Tier 4 containers started
```
View state: `cat /boot/config/fallback_state.db`
Check state: `fallback.sh --status`
View state: `fallback.sh --status` (preferred — parsed output)
Raw file: `cat "$STATE_DIR/fallback_state.db"` (requires STATE_DIR set, or use full path)
The file is managed exclusively by fallback.sh. Do not edit it while fallback.sh is
running — the next cycle will overwrite your changes. Use the Manual State Reset
@@ -384,7 +387,7 @@ servers must be running it continuously for mutual coverage.
pgrep -f "fallback.sh"
# Check the state file
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
```
Start via User Scripts plugin on both servers.
@@ -405,7 +408,7 @@ in smart mode — a non-NORMAL state at digest time needs attention.
```bash
fallback.sh --status # full state snapshot
cat /boot/config/fallback_state.db # raw state file
cat "$STATE_DIR/fallback_state.db" # raw state file
```
---
@@ -426,7 +429,7 @@ fallback_test.sh
# Step 3 — check state after test completes
fallback.sh --status
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
```
If the test doesn't complete cleanly, the state file may be left in FALLBACK. The
@@ -459,11 +462,14 @@ ping -c 5 [remote-tailscale-ip]
**Stop fallback.sh first (via User Scripts Abort), then reset:**
```bash
# Set STATE_DIR (or source load_config.sh to get it from the environment)
source /boot/config/plugins/varaverk/load_config.sh
# View current state
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
# Write a clean NORMAL state
cat > /boot/config/fallback_state.db << 'EOF'
cat > "$STATE_DIR/fallback_state.db" << 'EOF'
state=NORMAL
fallback_start=0
handback_strikes=0
@@ -473,7 +479,7 @@ tier4_started=false
EOF
# Verify the write
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
```
Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its next cycle.
@@ -510,7 +516,7 @@ Can this server reach the remote Tailscale IP?
What does fallback.sh report?
→ fallback.sh --status
→ cat /boot/config/fallback_state.db
→ cat "$STATE_DIR/fallback_state.db"
```
### Handback Not Completing
@@ -559,7 +565,7 @@ If it has happened:
3. Understand the state before resetting
fallback.sh --status
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
4. Perform Manual State Reset above on the server in a bad state
Fallback/fallback.sh
+7 -5
View File
@@ -130,11 +130,13 @@
# STATE FILES
# ==============================================================================================
#
# FALLBACK_STATE_FILE — /boot/config/fallback_state.db (survives reboots)
# FALLBACK_STATE_FILE — $STATE_DIR/fallback_state.db (survives reboots)
# Keys: state, fallback_start, handback_strikes, tier2_started,
# tier3_started, tier4_started, partnership_suspended, partner_lost_at.
# Lives on /boot/ intentionally — if the server was in FALLBACK when it
# rebooted, it resumes FALLBACK on restart.
# Survives reboots — $STATE_DIR is on the boot device (internal) or appdata
# (flash). Either way the array is up before this script runs, so the file
# is always accessible. If the server was in FALLBACK when it rebooted,
# it resumes FALLBACK on restart.
#
# ==============================================================================================
# CONFIGURATION
@@ -185,7 +187,7 @@
# Consecutive remote-up checks required before handback begins. (default: 3)
#
# FALLBACK_STATE_FILE
# State file path — /boot/config/fallback_state.db — survives reboots.
# State file path — $STATE_DIR/fallback_state.db — survives reboots.
#
# FALLBACK_RSYNC_ENABLED
# Gate for writeback rsync jobs during handback. (default: true)
@@ -302,7 +304,7 @@ log "$ICON_GEAR Timeouts: docker=${DOCKER_TIMEOUT}s ssh=${SSH_TIMEOUT}s verify-w
# ==============================================================================================
# ── STATE FILE HELPERS ────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# State file on /boot/config — survives reboots.
# State file at $FALLBACK_STATE_FILE ($STATE_DIR/fallback_state.db) — survives reboots.
# Format: key=value one per line.
# Keys: state, fallback_start, handback_strikes, tier2_started, tier3_started, tier4_started
Manual.md
+3 -3
View File
@@ -409,7 +409,7 @@ pgrep -a -f fallback.sh
Check current state:
```bash
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
# state=NORMAL — both servers up
```
@@ -510,7 +510,7 @@ bash Orchestrators/daily_sync_maintenance.sh --dry-run --log
### ── Check fallback state ────────────────────────────────────────────────────
```bash
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
# Expected:
# state=NORMAL
# fallback_start=0
@@ -600,7 +600,7 @@ The remote server is unreachable on the first check. Common causes:
```bash
# Confirm which state fallback is in
cat /boot/config/fallback_state.db
cat "$STATE_DIR/fallback_state.db"
# If stuck in FALLBACK after remote comes back: reset state
bash Tools/fallback_state_reset.sh
```
Media/Manual-Media.md
+45 -4
View File
@@ -132,8 +132,8 @@ LIDARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan
LIDARR_VERSION_MAJOR=3 # expected Lidarr major version (API safety check)
LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc")
LIDARR_TRACKED_COUNT_FILE=/boot/config/lidarr_tracked_count # persistent baseline
ARR_CLEANUP_STATS=/boot/config/arr_cleanup_stats.db # read by coffee report
LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count" # persistent baseline
ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db" # read by coffee report
```
---
@@ -171,7 +171,7 @@ RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.a
```bash
ARR_SYNC_ENABLED=true
ARR_SYNC_BLOCKLIST=/boot/config/arr_sync_blocklist.tsv # tombstone file
ARR_SYNC_BLOCKLIST="$DATA_DIR/arr_sync_blocklist.tsv" # tombstone file
ARR_SYNC_CONNECT_TIMEOUT=10 # SSH connect timeout in seconds
ARR_SYNC_API_TIMEOUT=60 # curl API call timeout in seconds
DOCKER_APPDATA_BASE=/mnt/user/appdata
@@ -189,7 +189,7 @@ ARR_IMPORT_RECOVERY_AGE=6 # hours — items newer than this are skipped
SONARR_VERSION_MAJOR=4
RADARR_VERSION_MAJOR=6
LIDARR_VERSION_MAJOR=3
ARR_RECOVERY_STATS=/boot/config/arr_recovery_stats.db # read by coffee report
ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db" # read by coffee report
```
---
@@ -274,6 +274,47 @@ Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
---
### Play State Sync
```bash
PLAY_STATE_SYNC_ENABLED=true # toggle entire sync
PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items
```
Emby and Jellyfin servers configured per-host:
```bash
# host*.conf
HOST1_EMBY_URL="http://192.168.50.2:8096"
HOST1_EMBY_API_KEY="..."
HOST1_JELLYFIN_URL="" # empty = skip Jellyfin on this host
HOST1_JELLYFIN_API_KEY=""
```
---
### Upgrade Webhook
```bash
# master.conf
WEBHOOK_PORT=9456 # 0 = disable listener
WEBHOOK_SECRET="" # auto-generated on first start if empty
WEBHOOK_UPGRADE_RSYNC_ENABLED=true # push upgraded file to remote nodes
```
```bash
# host*.conf
HOST1_WEBHOOK_LAN_IP="192.168.50.2" # IP arrs call for webhook delivery
```
When `WEBHOOK_PORT=0`, `start_webhook_listener.sh` exits cleanly and no listener starts.
When `WEBHOOK_SECRET` is empty, a 32-byte hex secret is generated on first start and
written back to `master.conf`. Run `Tools/webhook_setup.sh` to register the URL in arrs.
Webhook log: `/var/log/varaverk/upgrade_webhook.log`
---
### Orchestrator Job Order
```bash
Media/README-Media.md
+48 -2
View File
@@ -4,8 +4,9 @@ Library health, consistency, sync, and behavior-driven discovery for a multi-ser
stack. Correct permissions so arrs can manage files. Junk removal so orphan detection
isn't confused by scene debris. Library sync so every node tracks the same content.
Orphan cleanup against live arr APIs so deleted content actually leaves disk. Emby
notified automatically after every deletion. Weekly discovery adds new music, movies,
and TV shows based on what you actually play — no manual browsing required.
notified automatically after every deletion. Watch state synced across Emby/Jellyfin
every 30 minutes. Quality upgrades propagate to all nodes immediately via webhook.
Weekly discovery adds new music, movies, and TV shows based on what you actually play.
> **These scripts permanently delete files.** The arr cleanup scripts are protected by
> multiple safety layers that must all pass before anything is touched — but dry runs and
@@ -91,6 +92,19 @@ free overnight recovery.
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
fallback sources. Never overwrites existing files.
**Play State Sync**
`play_state_sync.sh` — syncs watched/played state and resume positions across all
configured Emby and Jellyfin servers. Newest timestamp wins. Runs every 30 minutes
via critical_sync_maintenance.sh.
**Upgrade Propagation**
`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr
OnUpgrade webhooks. Continuous; started at array start. Writes to
`/var/log/varaverk/upgrade_webhook.log`.
`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded
item folder to every remote node immediately, then triggers an arr library rescan on
each remote so the upgraded file is accepted without triggering a redundant quality search.
**Discovery**
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
@@ -138,6 +152,24 @@ playback_aware_radarr_discovery.sh — score watch history → TMDB recommenda
playback_aware_sonarr_discovery.sh — score episode history → TMDB TV recommendations → add to Sonarr
```
**Every 30 min via `critical_sync_maintenance.sh` (CRITICAL_MAINTENANCE_SCRIPTS):**
```
play_state_sync.sh — sync watched/resume state across Emby + Jellyfin
```
**Continuous (started by `array_started.sh`):**
```
start_webhook_listener.sh — Node.js webhook server; dispatch upgrade_webhook_handler.sh
```
**On every arr upgrade (triggered by webhook):**
```
upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan
```
**Ad-hoc or separate schedule:**
```
@@ -189,6 +221,9 @@ in `HOST*_MEDIA_PERMISSION_SHARES` and `HOST*_MEDIA_CLEAN_FOLDERS` in host*.conf
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Ad-hoc or separate schedule |
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Ad-hoc or weekly |
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Ad-hoc or weekly |
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min via `critical_sync_maintenance.sh` |
| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous (started by `array_started.sh`) |
| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade (via webhook) |
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly via `weekly_sync_maintenance.sh` |
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly via `weekly_sync_maintenance.sh` |
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly via `weekly_sync_maintenance.sh` |
@@ -228,6 +263,17 @@ Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
│
└── each discovery script fires arr search immediately after successful add
Every 30 min (critical_sync_maintenance.sh CRITICAL_MAINTENANCE_SCRIPTS):
play_state_sync.sh ─── newest timestamp wins → watched/resume state synced
across all configured Emby + Jellyfin servers
Continuous (started by array_started.sh):
start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
│ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
└── upgrade_webhook_handler.sh
├── rsync upgraded folder → all remote nodes immediately
└── trigger arr library rescan on each remote (accept new file, no quality search)
Ad-hoc enrichment:
lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
Media/start_webhook_listener.sh
+5 -2
View File
@@ -5,7 +5,7 @@
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Starts a standalone PHP HTTP server that receives Sonarr/Radarr/Lidarr
# Starts a standalone Node.js HTTP server that receives Sonarr/Radarr/Lidarr
# OnUpgrade webhooks and dispatches upgrade_webhook_handler.sh.
#
# Runs outside Unraid nginx — no session auth required. The shared secret in
@@ -13,9 +13,12 @@
#
# http://<HOST_LAN_IP>:<WEBHOOK_PORT>/webhook?key=<WEBHOOK_SECRET>
#
# Runs as a continuous script started by array_started.sh. Execs php -S which
# Runs as a continuous script started by array_started.sh. Execs node which
# replaces this process — the PID stays the same for array_started.sh's check.
#
# Uses Node.js instead of php -S: php -S on Unraid PHP 8.4 silently drops
# POST request bodies, making webhook payloads arrive empty.
#
# If WEBHOOK_SECRET is empty in master.conf: generates and saves one, then starts.
# If WEBHOOK_PORT is 0: exits cleanly (disables the listener).
#
Monitors/Manual-Monitors.md
+5 -5
View File
@@ -151,14 +151,14 @@ BACKUP_VERIFY_MIN_SIZE="1M" # skip files smaller than this
```bash
# master.conf
BANDWIDTH_LOG="/boot/config/bandwidth_history.db" # survives reboots
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db" # survives reboots
BANDWIDTH_LOG_RETENTION=90 # days — file stays bounded, never grows unbounded
BANDWIDTH_WARN_GB=50 # flag transfers or daily totals exceeding this
```
**Why `/boot/config/`**: The log needs to survive reboots to build a useful history.
`/boot/config/` is on the USB flash drive, which survives reboots and is backed up
by unRAID's flash backup. The log is bounded by `BANDWIDTH_LOG_RETENTION` so it never
**Why `$DATA_DIR`**: The log needs to survive reboots to build a useful history.
`$DATA_DIR` (`$SCRIPTS_DIR/data/`) is on the boot device (internal) or appdata (flash),
both of which survive reboots. The log is bounded by `BANDWIDTH_LOG_RETENTION` so it never
grows unbounded.
**`BANDWIDTH_WARN_GB`**: Set to a value that represents "unexpectedly large" for your
@@ -401,7 +401,7 @@ BACKUP_VERIFY_SAMPLE=10
BACKUP_VERIFY_MIN_SIZE="1M"
# bandwidth_monitor.sh
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90
BANDWIDTH_WARN_GB=50
Orchestrators/README-Orchestrators.md
+2 -2
View File
@@ -849,7 +849,7 @@ long tests — that should only run on stable systems that have been up for at l
Two gates must both pass before any job runs:
1. Server uptime ≥ `MONTHLY_UPTIME_THRESHOLD_DAYS`
2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file on `/boot/config/` — survives reboots)
2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file in `$STATE_DIR` — survives reboots)
If either gate fails, the script exits 0 with no output. This is expected — it runs
daily and most days are no-ops.
@@ -866,7 +866,7 @@ MONTHLY_MAINTENANCE_SCRIPTS=(
)
MONTHLY_UPTIME_THRESHOLD_DAYS=30
MONTHLY_RUN_INTERVAL_DAYS=30
MONTHLY_LAST_RUN_FILE="/boot/config/monthly_maintenance_last_run.db"
MONTHLY_LAST_RUN_FILE="$STATE_DIR/monthly_maintenance_last_run.db"
```
Scripts are commented out by default — uncomment what applies to your hardware.
Orchestrators/intermediate_sync_maintenance.sh
+1 -1
View File
@@ -6,7 +6,7 @@
# Schedule: 0 */4 * * * (every 4 hours)
#
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
# 1. conf_sync.sh --pull-only — refresh partner conf cache in RAM (/tmp/.vv/)
# 1. conf_sync.sh --pull-only — refresh partner conf cache in RAM (/tmp/.cache/vv/d/)
# 2. arr_sync.sh — sync Lidarr/Sonarr/Radarr libraries across all nodes
# 3. Rsync window (optional) — INTERMEDIATE_SYNC_SHARES, if any configured
# 4. INTERMEDIATE_MAINTENANCE_SCRIPTS — artwork fetch and any future 4-hour jobs
Partnership/Manual-Partnership.md
+4 -4
View File
@@ -574,7 +574,7 @@ Partnership/partnership_manager.sh --status # shows both sides via SSH
```bash
# Check the offline counter:
cat /boot/config/partnership_offline_days.db
cat "$STATE_DIR/partnership_offline_days.db"
# Extended Tailscale outage may have incremented the counter.
# Check Tailscale peer visibility:
@@ -591,9 +591,9 @@ Partnership/partnership_manager.sh --onboard
## ━━━ STATE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```bash
/boot/config/partnership_HOST1.db # HOST1 writes only
/boot/config/partnership_HOST2.db # HOST2 writes only
/boot/config/partnership_blocklist.db # hostname|timestamp|reason
$STATE_DIR/partnership_HOST1.db # HOST1 writes only
$STATE_DIR/partnership_HOST2.db # HOST2 writes only
$STATE_DIR/partnership_blocklist.db # hostname|timestamp|reason
# Example state file:
state=ACTIVE
Partnership/partnership_manager.sh
+4 -4
View File
@@ -93,11 +93,11 @@
# STATE FILES
# ==============================================================================================
#
# /boot/config/partnership_HOST1.db — HOST1 writes, HOST2 reads via SSH
# /boot/config/partnership_HOST2.db — HOST2 writes, HOST1 reads via SSH
# /boot/config/partnership_blocklist.db — hostname|timestamp|reason, persists until cleared
# $STATE_DIR/partnership_<hostname>.db — each host writes its own, partner reads via SSH
# $STATE_DIR/partnership_blocklist.db — hostname|timestamp|reason, persists until cleared
# $STATE_DIR/partnership_offline_days.db — cumulative offline day counter
#
# On /boot/config — survives reboots, available before array starts, minimal flash wear.
# All in STATE_DIR — survives reboots (on /boot in internal mode, appdata in flash mode).
#
# ==============================================================================================
# CONFIGURATION
Plugin/unraid/adapter.sh
+1 -1
View File
@@ -274,7 +274,7 @@ platform_get_templates_dir() {
# Writes the path to the persistent Varaverk setup/wizard state database.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_setup_db_path() {
echo "/boot/config/varaverk_setup.db"
echo "${STATE_DIR}/varaverk_setup.db"
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
Plugin/unraid/api/snapshot.php
+1 -1
View File
@@ -13,7 +13,7 @@ $ramUsedMb = $ramTotalMb - $res['ram_free_mb'];
$ramPct = $ramTotalMb > 0 ? (int)round($ramUsedMb / $ramTotalMb * 100) : 0;
// Fallback state (fast file read, no exec)
$fbRaw = @file_get_contents('/tmp/fallback_state.db') ?: '';
$fbRaw = @file_get_contents(STATE_DIR . '/fallback_state.db') ?: '';
$fbData = vv_parse_kv_db($fbRaw);
$fallbackState = $fbData['state'] ?? 'UNKNOWN';
Plugin/unraid/include/config.php
+24 -5
View File
@@ -34,8 +34,7 @@ function vv_setup_state_write(array $data): void {
}
// Push the setup state file to all remote hosts via scp.
// Unlike master.conf push, this does NOT require the plugin to be installed on the remote —
// it only needs SSH to be reachable, and pushes to /boot/config/ (always available).
// Reads the remote's varaverk.cfg to find their actual SCRIPTS_DIR (handles appdata mode).
function vv_push_setup_state(): void {
if (!file_exists(VV_SETUP_STATE_FILE)) return;
$myHostId = vv_detect_host();
@@ -56,9 +55,20 @@ function vv_push_setup_state(): void {
if (!$ip) continue;
$sshBase = 'ssh -i ' . escapeshellarg($sshKey)
. ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@' . $ip;
// Ensure the target dir exists (it always should on Unraid, but be safe)
shell_exec($sshBase . ' "mkdir -p /boot/config" 2>/dev/null');
$dest = escapeshellarg('root@' . $ip . ':/boot/config/varaverk_setup.db');
// Get remote SCRIPTS_DIR from varaverk.cfg — handles appdata mode on remote.
// Falls back to the default install path if varaverk.cfg is absent (pre-install).
$cfgRaw = trim(shell_exec($sshBase . ' "cat /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null"') ?: '');
$remoteSD = '/boot/config/plugins/varaverk';
foreach (explode("\n", $cfgRaw) as $line) {
if (str_starts_with(trim($line), 'SCRIPTS_DIR=')) {
$remoteSD = trim(substr(trim($line), strlen('SCRIPTS_DIR=')), '"\'');
break;
}
}
$remoteStatePath = $remoteSD . '/State_Files/varaverk_setup.db';
shell_exec($sshBase . ' "mkdir -p ' . escapeshellarg(dirname($remoteStatePath)) . '" 2>/dev/null');
$dest = escapeshellarg('root@' . $ip . ':' . $remoteStatePath);
exec('scp -i ' . escapeshellarg($sshKey)
. ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no'
. ' ' . escapeshellarg(VV_SETUP_STATE_FILE) . ' ' . $dest . ' 2>&1');
@@ -391,6 +401,15 @@ function vv_auto_create_api_key(string $hostId, string $confFile): array {
return ['ok' => true, 'key_preview' => $key ? substr($key, 0, 8) . '...' . substr($key, -4) : 'registered'];
}
// Build a bash command that reads a state file from the REMOTE host's State_Files/.
// Reads the remote's varaverk.cfg to resolve their SCRIPTS_DIR (may differ from ours
// when the remote is in appdata mode). Falls back to the internal plugin path.
function vv_remote_state_cmd(string $filename): string {
$fn = basename($filename);
return 'sd=$(grep -m1 SCRIPTS_DIR= /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null'
. ' | cut -d\'"\' -f2); cat "${sd:-/boot/config/plugins/varaverk}/State_Files/' . $fn . '" 2>/dev/null';
}
// Local LAN IP via routing table — static-cached per request.
// Previously duplicated in include/docker_folders.php and inline in include/docker.php.
function vv_local_ip(): string {
Plugin/unraid/include/fallback.php
+33 -20
View File
@@ -18,12 +18,14 @@ function vv_fb_scalar(string $raw, string $varname): string {
function vv_fb_parse_state(string $text): array {
$out = [
'state' => 'UNKNOWN',
'fallback_start' => 0,
'handback_strikes' => 0,
'tier2_started' => false,
'tier3_started' => false,
'tier4_started' => false,
'state' => 'UNKNOWN',
'fallback_start' => 0,
'handback_strikes' => 0,
'tier2_started' => false,
'tier3_started' => false,
'tier4_started' => false,
'partnership_suspended' => false,
'partner_lost_at' => 0,
];
foreach (explode("\n", $text) as $line) {
$line = trim($line);
@@ -31,24 +33,26 @@ function vv_fb_parse_state(string $text): array {
[$k, $v] = array_pad(explode('=', $line, 2), 2, '');
$k = trim($k); $v = trim($v, '"\'');
switch ($k) {
case 'state': $out['state'] = $v; break;
case 'fallback_start': $out['fallback_start'] = (int)$v; break;
case 'handback_strikes': $out['handback_strikes'] = (int)$v; break;
case 'tier2_started': $out['tier2_started'] = $v === 'true'; break;
case 'tier3_started': $out['tier3_started'] = $v === 'true'; break;
case 'tier4_started': $out['tier4_started'] = $v === 'true'; break;
case 'state': $out['state'] = $v; break;
case 'fallback_start': $out['fallback_start'] = (int)$v; break;
case 'handback_strikes': $out['handback_strikes'] = (int)$v; break;
case 'tier2_started': $out['tier2_started'] = $v === 'true'; break;
case 'tier3_started': $out['tier3_started'] = $v === 'true'; break;
case 'tier4_started': $out['tier4_started'] = $v === 'true'; break;
case 'partnership_suspended': $out['partnership_suspended'] = $v === 'true'; break;
case 'partner_lost_at': $out['partner_lost_at'] = (int)$v; break;
}
}
return $out;
}
function vv_fb_local_state(): array {
$path = '/boot/config/fallback_state.db';
$path = STATE_DIR . '/fallback_state.db';
return vv_fb_parse_state(file_exists($path) ? file_get_contents($path) : '');
}
function vv_fb_remote_state(string $ip, string $sshKey): array {
$out = vv_pt_ssh($ip, $sshKey, 'cat /boot/config/fallback_state.db 2>/dev/null');
$out = vv_pt_ssh($ip, $sshKey, vv_remote_state_cmd('fallback_state.db'));
return vv_fb_parse_state($out);
}
@@ -94,8 +98,13 @@ function vv_fb_all(): array {
$currentHost = vv_detect_host();
$hosts = vv_fb_known_hosts();
$tsPeers = vv_pt_ts_peers();
$handbackReq = (int)(vv_fb_scalar(vv_read_conf_raw('master.conf'), 'FALLBACK_HANDBACK_STRIKES') ?: 3);
$fbEnabled = vv_fb_scalar(vv_read_conf_raw('master.conf'), 'FALLBACK_ENABLED') === 'true';
$masterRaw = vv_read_conf_raw('master.conf');
$handbackReq = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_HANDBACK_STRIKES') ?: 3);
$fbEnabled = vv_fb_scalar($masterRaw, 'FALLBACK_ENABLED') === 'true';
$ptEnabled = vv_fb_scalar($masterRaw, 'PARTNERSHIP_ENABLED') === 'true';
$rsyncEnabled = vv_fb_scalar($masterRaw, 'FALLBACK_RSYNC_ENABLED') !== 'false';
$checkInterval = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_CHECK_INTERVAL') ?: 30);
$suspendAfter = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_PARTNERSHIP_SUSPEND_AFTER') ?: 120);
// Read all host conf raws upfront
$raws = [];
@@ -162,9 +171,13 @@ function vv_fb_all(): array {
}
return [
'ts' => time(),
'fb_enabled' => $fbEnabled,
'handback_req' => $handbackReq,
'nodes' => $nodes,
'ts' => time(),
'fb_enabled' => $fbEnabled,
'partnership_enabled' => $ptEnabled,
'fb_rsync_enabled' => $rsyncEnabled,
'handback_req' => $handbackReq,
'check_interval' => $checkInterval,
'suspend_after' => $suspendAfter,
'nodes' => $nodes,
];
}
Plugin/unraid/include/monitor.php
+2 -2
View File
@@ -58,7 +58,7 @@ function vv_fallback_state(): array {
$reqStrikes = (int)($vars['FALLBACK_HANDBACK_STRIKES'] ?? 3);
$suspendAfter = (int)($vars['FALLBACK_PARTNERSHIP_SUSPEND_AFTER'] ?? 120);
$stateFile = '/tmp/fallback_state.db';
$stateFile = STATE_DIR . '/fallback_state.db';
if (!file_exists($stateFile)) {
return ['state' => 'UNKNOWN', 'enabled' => $enabled, 'check_interval' => $interval,
'handback_strikes' => 0, 'handback_strikes_required' => $reqStrikes,
@@ -164,7 +164,7 @@ function vv_watchdog_summary(): array {
usort($restarts, fn($a, $b) => $b['ts'] - $a['ts']);
// Reboots (12 h)
$rebootRaw = @file_get_contents('/boot/config/system_watchdog_reboots.db') ?: '';
$rebootRaw = @file_get_contents(STATE_DIR . '/system_watchdog_reboots.db') ?: '';
$rbootCutoff = time() - 43200;
$reboots = 0;
foreach (explode("\n", trim($rebootRaw)) as $line) {
Plugin/unraid/include/partnership.php
+4 -5
View File
@@ -10,7 +10,7 @@ require_once __DIR__ . '/common.php'; // vv_system_info(), vv_docker_containers(
function vv_pt_config(): array {
$v = vv_conf_vars();
$offlineDays = null;
$odFile = '/boot/config/partnership_offline_days.db';
$odFile = STATE_DIR . '/partnership_offline_days.db';
if (file_exists($odFile)) {
$raw = trim(@file_get_contents($odFile) ?: '');
if (is_numeric($raw)) $offlineDays = (int)$raw;
@@ -209,12 +209,11 @@ function vv_pt_nodes(): array {
// Fallback state
$fbState = 'UNKNOWN';
$fbPath = '/boot/config/fallback_state.db';
if ($isMe) {
$fb = vv_pt_read_db($fbPath);
$fb = vv_pt_read_db(STATE_DIR . '/fallback_state.db');
$fbState = $fb['state'] ?? 'UNKNOWN';
} elseif ($ts['online'] && $ts['ip'] && $mySshKey) {
$out = vv_pt_ssh($ts['ip'], $mySshKey, 'cat /boot/config/fallback_state.db 2>/dev/null');
$out = vv_pt_ssh($ts['ip'], $mySshKey, vv_remote_state_cmd('fallback_state.db'));
if ($out) {
$fb = [];
foreach (explode("\n", $out) as $line) {
@@ -226,7 +225,7 @@ function vv_pt_nodes(): array {
}
// Partnership DB — local only (each server writes its own)
$dbPath = "/boot/config/partnership_{$hostname}.db";
$dbPath = STATE_DIR . "/partnership_{$hostname}.db";
$ptDb = vv_pt_read_db($dbPath);
// System info
Plugin/unraid/include/watchdog.php
+8 -8
View File
@@ -127,14 +127,14 @@ function vv_wd_parse_network_state(string $raw): array {
// ── Local state files ─────────────────────────────────────────────────────────
function vv_wd_local_states(string $restartLogPath): array {
$rwRaw = @file_get_contents('/tmp/resource_watchdog_state.db') ?: '';
$dockRaw = @file_get_contents('/tmp/container_watchdog_state.db') ?: '';
$skipRaw = @file_get_contents('/boot/config/system_watchdog_failed.db') ?: '';
$sysRaw = @file_get_contents('/tmp/system_watchdog_state.db') ?: '';
$rebootRaw = @file_get_contents('/boot/config/system_watchdog_reboots.db')?: '';
$restartRaw= @file_get_contents($restartLogPath) ?: '';
$storRaw = @file_get_contents('/tmp/storage_watchdog_state.db') ?: '';
$netWdRaw = @file_get_contents('/tmp/network_watchdog_state.db') ?: '';
$rwRaw = @file_get_contents(STATE_DIR . '/resource_watchdog_state.db') ?: '';
$dockRaw = @file_get_contents(STATE_DIR . '/container_watchdog_state.db') ?: '';
$skipRaw = @file_get_contents(STATE_DIR . '/docker_watchdog_failed.db') ?: '';
$sysRaw = @file_get_contents(STATE_DIR . '/system_watchdog_state.db') ?: '';
$rebootRaw = @file_get_contents(STATE_DIR . '/system_watchdog_reboots.db') ?: '';
$restartRaw= @file_get_contents($restartLogPath) ?: '';
$storRaw = @file_get_contents(STATE_DIR . '/storage_watchdog_state.db') ?: '';
$netWdRaw = @file_get_contents(STATE_DIR . '/network_watchdog_state.db') ?: '';
$rw = vv_wd_parse_kv($rwRaw);
$dock = vv_wd_parse_kv($dockRaw);
Plugin/unraid/pages/fallback.php
+200 -27
View File
@@ -1,4 +1,5 @@
<style>
/* ── Existing status styles ── */
.vv-fb-active { background:#1a1200;border:1px solid #5a3800;border-radius:6px;padding:12px 14px; }
.vv-fb-active-h { display:flex;align-items:baseline;gap:10px;margin-bottom:8px; }
.vv-fb-badge { font-size:11px;font-weight:bold;letter-spacing:.06em;padding:2px 7px;border-radius:3px;flex-shrink:0; }
@@ -6,6 +7,7 @@
.vv-fb-badge.norm { background:#1a2a1a;color:#4caf50; }
.vv-fb-badge.dark { background:#2a1a2a;color:#9c27b0; }
.vv-fb-badge.nonet{ background:#1a1a2a;color:#5c7cfa; }
.vv-fb-badge.susp { background:#2a1a00;color:#888; }
.vv-fb-meta { display:flex;gap:18px;flex-wrap:wrap;margin-bottom:10px; }
.vv-fb-meta-item{ display:flex;flex-direction:column;gap:1px; }
.vv-fb-meta-val { font-size:17px;font-weight:bold;color:#ffb74d; }
@@ -31,17 +33,109 @@
.vv-fb-state-dot { width:6px;height:6px;border-radius:50%;flex-shrink:0;margin-top:3px; }
.vv-fb-sep { border:none;border-top:1px solid #222;margin:8px 0; }
.vv-fb-disabled { grid-column:1/-1;color:#3a3a3a;font-size:12px;padding:20px 0;text-align:center; }
/* ── Controls + settings card ── */
.vv-fb-card { background:#161616;border:1px solid #2a2a2a;border-radius:6px;padding:14px 16px;margin-bottom:14px; }
.vv-fb-card-hdr { font-size:11px;font-weight:700;color:#666;text-transform:uppercase;letter-spacing:.07em;margin-bottom:12px; }
.vv-fb-ctrl-row { display:flex;justify-content:space-between;align-items:center;gap:12px;padding:5px 0; }
.vv-fb-ctrl-lbl { font-size:12px;color:#888; }
.vv-fb-ctrl-sub { font-size:10px;color:#3a3a3a;margin-top:2px; }
.vv-fb-tog { width:32px;height:18px;border-radius:9px;background:#222;border:1px solid #333;
position:relative;transition:background .15s,border-color .15s;flex-shrink:0;cursor:pointer; }
.vv-fb-tog.on { background:#1a3a1a;border-color:#2d5a2d; }
.vv-fb-tog::after { content:'';position:absolute;top:2px;left:2px;width:12px;height:12px;
border-radius:50%;background:#555;transition:left .15s,background .15s; }
.vv-fb-tog.on::after { left:16px;background:#4caf50; }
.vv-fb-set-row { display:flex;justify-content:space-between;align-items:center;padding:5px 0; }
.vv-fb-set-lbl { font-size:11px;color:#555; }
.vv-fb-set-inp { background:#0d0d0d;border:1px solid #2a2a2a;border-radius:3px;color:#888;
font-size:12px;padding:4px 8px;outline:none;width:72px;font-family:monospace;
text-align:right;box-sizing:border-box; }
.vv-fb-set-inp:focus { border-color:#444; }
.vv-fb-set-unit { font-size:10px;color:#3a3a3a;min-width:56px; }
.vv-fb-save-btn { background:#1a1a1a;border:1px solid #333;color:#888;font-size:11px;
padding:5px 14px;border-radius:3px;cursor:pointer; }
.vv-fb-save-btn:hover { border-color:#555;color:#ccc; }
.vv-fb-save-btn:disabled { opacity:.4;cursor:default; }
</style>
<!-- Controls card -->
<div class="vv-fb-card">
<div class="vv-fb-card-hdr">Controls</div>
<div class="vv-fb-ctrl-row">
<div>
<div class="vv-fb-ctrl-lbl">Partnership</div>
<div class="vv-fb-ctrl-sub">Master gate — disabling stops all cross-server operations</div>
</div>
<div class="vv-fb-tog" id="vv-fb-pt-tog" onclick="vvFbToggle(this,'PARTNERSHIP_ENABLED')"></div>
</div>
<hr class="vv-fb-sep">
<div class="vv-fb-ctrl-row">
<div>
<div class="vv-fb-ctrl-lbl">Fallback</div>
<div class="vv-fb-ctrl-sub">Mutual container failover between nodes</div>
</div>
<div class="vv-fb-tog" id="vv-fb-en-tog" onclick="vvFbToggle(this,'FALLBACK_ENABLED')"></div>
</div>
<hr class="vv-fb-sep">
<div class="vv-fb-ctrl-row">
<div>
<div class="vv-fb-ctrl-lbl">Rsync on handback</div>
<div class="vv-fb-ctrl-sub">Writeback rsync when the covered host recovers and containers return</div>
</div>
<div class="vv-fb-tog" id="vv-fb-rsync-tog" onclick="vvFbToggle(this,'FALLBACK_RSYNC_ENABLED')"></div>
</div>
</div>
<!-- Status section -->
<div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px;padding:0 2px;">
<span style="font-size:13px;font-weight:bold;color:#888;text-transform:uppercase;letter-spacing:.06em;">FallBack</span>
<span style="font-size:13px;font-weight:bold;color:#888;text-transform:uppercase;letter-spacing:.06em;">Status</span>
<span style="font-size:11px;color:#3a3a3a;" id="vv-fb-ts"></span>
</div>
<div id="vv-fb-grid" style="display:grid;grid-template-columns:repeat(8,1fr);gap:12px;">
<div id="vv-fb-grid" style="display:grid;grid-template-columns:repeat(8,1fr);gap:12px;margin-bottom:14px;">
<div style="grid-column:1/-1;color:#444;font-size:12px;padding:16px 0;text-align:center;">Loading…</div>
</div>
<!-- Settings card -->
<div class="vv-fb-card">
<div class="vv-fb-card-hdr">Settings</div>
<div class="vv-fb-set-row">
<span class="vv-fb-set-lbl">Check interval</span>
<div style="display:flex;align-items:center;gap:8px;">
<input class="vv-fb-set-inp" id="vv-fb-interval" type="number" min="5" max="300">
<span class="vv-fb-set-unit">seconds</span>
</div>
</div>
<div class="vv-fb-set-row">
<span class="vv-fb-set-lbl">Handback strikes</span>
<div style="display:flex;align-items:center;gap:8px;">
<input class="vv-fb-set-inp" id="vv-fb-strikes" type="number" min="1" max="20">
<span class="vv-fb-set-unit">consecutive</span>
</div>
</div>
<div class="vv-fb-set-row">
<span class="vv-fb-set-lbl">Partnership suspend after</span>
<div style="display:flex;align-items:center;gap:8px;">
<input class="vv-fb-set-inp" id="vv-fb-suspend" type="number" min="0" max="1440">
<span class="vv-fb-set-unit">minutes</span>
</div>
</div>
<div style="display:flex;justify-content:flex-end;align-items:center;gap:10px;margin-top:10px;padding-top:8px;border-top:1px solid #1e1e1e;">
<span id="vv-fb-set-fb" style="font-size:11px;"></span>
<button class="vv-fb-save-btn" id="vv-fb-save-btn" onclick="vvFbSaveSettings()">Save</button>
</div>
</div>
<script>
(function() {
@@ -78,6 +172,7 @@ function _stateBadge(st) {
OFFLINE: ['dark', 'OFFLINE'],
UNREACHABLE: ['dark', 'UNREACHABLE'],
UNKNOWN: ['dark', 'UNKNOWN'],
SUSPENDED: ['susp', 'SUSPENDED'],
};
const [cls, label] = map[st] || ['dark', st];
return `<span class="vv-fb-badge ${cls}">${label}</span>`;
@@ -87,7 +182,7 @@ function _stateDot(st) {
const col = {
NORMAL:'#4caf50', FALLBACK:'#ffb74d',
NO_INTERNET:'#5c7cfa', DARK:'#9c27b0',
OFFLINE:'#555', UNREACHABLE:'#555', UNKNOWN:'#333',
OFFLINE:'#555', UNREACHABLE:'#555', UNKNOWN:'#333', SUSPENDED:'#444',
}[st] || '#333';
return `<span class="vv-fb-state-dot" style="background:${col}"></span>`;
}
@@ -102,7 +197,6 @@ function _activeCard(nodes, handbackReq) {
const cov = covering.covers;
const covered = cov ? cov.hostname : '?';
// All containers that should be running at current tier
let expected = [...(cov?.tier1 || [])];
if (tier >= 2) expected = expected.concat(cov?.tier2 || []);
if (tier >= 3) expected = expected.concat(cov?.tier3 || []);
@@ -169,9 +263,19 @@ function _tierSection(tiers, activeTier, delays) {
}).join('');
}
function _nodeCard(node) {
function _ptStatus(st) {
if (!st) return '';
if (st.partnership_suspended) return _stateBadge('SUSPENDED');
if (st.partner_lost_at && st.partner_lost_at > 0) {
const minGone = Math.floor((Date.now() / 1000 - st.partner_lost_at) / 60);
return `<span class="vv-fb-badge susp">GRACE ${minGone}m</span>`;
}
return '';
}
function _nodeCard(node, suspendAfter) {
const st = node.state || {};
const state = st.state || 'UNKNOWN';
const state = st.partnership_suspended ? 'SUSPENDED' : (st.state || 'UNKNOWN');
const cov = node.covers;
const active = _activeTier(state === 'FALLBACK' ? st : null);
@@ -183,6 +287,8 @@ function _nodeCard(node) {
? _tierSection(cov, active, cov.delays)
: '<div style="color:#3a3a3a;font-size:11px;">No coverage configured</div>';
const ptBadge = node.is_me ? _ptStatus(st) : '';
return `<div class="vv-card vv-fb-node">
<div class="vv-fb-node-h">
${_stateDot(state)}
@@ -190,6 +296,7 @@ function _nodeCard(node) {
<span class="vv-fb-node-nm">${node.hostname}</span>
${covTarget}
<span style="flex:1"></span>
${ptBadge}
${_stateBadge(state)}
</div>
<hr class="vv-fb-sep">
@@ -197,30 +304,47 @@ function _nodeCard(node) {
</div>`;
}
function _setToggles(data) {
const pairs = [
['vv-fb-pt-tog', !!data.partnership_enabled],
['vv-fb-en-tog', !!data.fb_enabled],
['vv-fb-rsync-tog', !!data.fb_rsync_enabled],
];
pairs.forEach(([id, on]) => {
const el = document.getElementById(id);
if (el) el.classList.toggle('on', on);
});
}
function _setInputs(data) {
const fields = [
['vv-fb-interval', data.check_interval ?? 30],
['vv-fb-strikes', data.handback_req ?? 3],
['vv-fb-suspend', data.suspend_after ?? 120],
];
fields.forEach(([id, val]) => {
const el = document.getElementById(id);
if (el && el !== document.activeElement) el.value = val;
});
}
function _render(data) {
if (!data.fb_enabled) {
document.getElementById('vv-fb-grid').innerHTML =
'<div class="vv-fb-disabled">FALLBACK_ENABLED=false — fallback monitoring is disabled</div>';
return;
_setToggles(data);
_setInputs(data);
const grid = document.getElementById('vv-fb-grid');
if (!data.partnership_enabled) {
grid.innerHTML = '<div class="vv-fb-disabled">PARTNERSHIP_ENABLED=false — all cross-server operations disabled</div>';
} else if (!data.fb_enabled) {
grid.innerHTML = '<div class="vv-fb-disabled">FALLBACK_ENABLED=false — fallback monitoring is disabled</div>';
} else {
const nodes = data.nodes || [];
let html = _activeCard(nodes, data.handback_req || 3);
for (const node of nodes) html += _nodeCard(node, data.suspend_after || 120);
grid.innerHTML = html || '<div class="vv-fb-disabled">No nodes configured.</div>';
}
const nodes = data.nodes || [];
let html = '';
// Top: active fallback card (if any)
html += _activeCard(nodes, data.handback_req || 3);
// Per-node cards
for (const node of nodes) {
html += _nodeCard(node);
}
if (!html) {
html = '<div class="vv-fb-disabled">No nodes configured.</div>';
}
document.getElementById('vv-fb-grid').innerHTML = html;
const ts = data.ts
? new Date(data.ts * 1000).toLocaleString([], {
month:'numeric', day:'numeric', year:'numeric',
@@ -239,6 +363,55 @@ function vvFbLoad() {
});
}
window.vvFbToggle = function(track, key) {
const on = !track.classList.contains('on');
track.classList.toggle('on', on);
const fd = new FormData();
fd.append('id', 'fallback');
fd.append('changes', JSON.stringify([{ file: 'master.conf', key, value: on ? 'true' : 'false', type: 'scalar' }]));
fetch('/plugins/varaverk/api/confform.php', { method: 'POST', body: fd })
.then(r => r.json())
.then(d => { if (!d.ok) track.classList.toggle('on', !on); })
.catch(() => track.classList.toggle('on', !on));
};
window.vvFbSaveSettings = function() {
const ivEl = document.getElementById('vv-fb-interval');
const stEl = document.getElementById('vv-fb-strikes');
const suEl = document.getElementById('vv-fb-suspend');
const fb = document.getElementById('vv-fb-set-fb');
const btn = document.getElementById('vv-fb-save-btn');
const interval = parseInt(ivEl.value, 10);
const strikes = parseInt(stEl.value, 10);
const suspend = parseInt(suEl.value, 10);
if ([interval, strikes, suspend].some(n => isNaN(n) || n < 0)) {
fb.style.color = '#ef5350'; fb.textContent = 'Invalid values'; return;
}
btn.disabled = true; btn.textContent = 'Saving…'; fb.textContent = '';
const fd = new FormData();
fd.append('id', 'fallback');
fd.append('changes', JSON.stringify([
{ file: 'master.conf', key: 'FALLBACK_CHECK_INTERVAL', value: String(interval), type: 'scalar' },
{ file: 'master.conf', key: 'FALLBACK_HANDBACK_STRIKES', value: String(strikes), type: 'scalar' },
{ file: 'master.conf', key: 'FALLBACK_PARTNERSHIP_SUSPEND_AFTER', value: String(suspend), type: 'scalar' },
]));
fetch('/plugins/varaverk/api/confform.php', { method: 'POST', body: fd })
.then(r => r.json())
.then(d => {
btn.disabled = false; btn.textContent = 'Save';
fb.style.color = d.ok ? '#4caf50' : '#ef5350';
fb.textContent = d.ok ? 'Saved ✓' : (d.error || 'Failed');
if (d.ok) setTimeout(() => { fb.textContent = ''; }, 3000);
})
.catch(() => {
btn.disabled = false; btn.textContent = 'Save';
fb.style.color = '#ef5350'; fb.textContent = 'Request failed';
});
};
vvFbLoad();
setInterval(vvFbLoad, 30000);
README-User_Script_Plug-in.md
+2 -2
View File
@@ -715,8 +715,8 @@ Change `--dry-run` to nothing and run during a maintenance window.
> forced kill of fallback.sh:
> ```bash
> # Verify first — right containers on right server, DDNS correct, fallback.sh not running
> cat /boot/config/fallback_state.db
> echo "state=NORMAL" > /boot/config/fallback_state.db
> cat "$STATE_DIR/fallback_state.db"
> echo "state=NORMAL" > "$STATE_DIR/fallback_state.db"
> ```
> Resets state only — does NOT start or stop any containers.
Rsync/Manual-Rsync.md
+1 -1
View File
@@ -621,7 +621,7 @@ SLEEP=60 # seconds between retries
ROOTFS_WARN_PCT=75 # abort if remote rootfs above this %
# Shared with Monitors/
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90
BANDWIDTH_WARN_GB=50
System_Essentials/Manual-System_Essentials.md
+192 -238
View File
@@ -4,23 +4,24 @@ Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start.
> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) and their configuration live in
> `Plugin/unraid/System_Essentials/`. Watchdog configuration (`stability_watchdog.sh`,
> `resource_watchdog.sh`, `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives
> in `Watchdogs/Manual-Watchdogs.md`.
---
## ━━━ CONTENTS ━━━
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
- [webgui_watchdog.sh](#webgui_watchdogsh)
- [ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order](#array_start_scripts--array_stop_scripts-order)
- [conf_sync.sh](#conf_syncsh)
- [conf_cache_save.sh](#conf_cache_savesh)
- [conf_cache_restore.sh](#conf_cache_restoresh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
- [clear_logs.sh](#clear_logssh)
- [mover_stop.sh](#mover_stopsh)
- [rsync_stop.sh](#rsync_stopsh)
- [user_scripts_stop.sh](#user_scripts_stopsh)
- [server_reboot.sh](#server_rebootsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -32,109 +33,155 @@ adding or reordering scripts at array start.
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
Per-check detail suppressed.
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
**One-shot scripts** (`conf_sync.sh`, `conf_cache_save.sh`, `conf_cache_restore.sh`,
`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`, `rsync_stop.sh`,
`server_reboot.sh`): without `--log`, section headers, per-step results, and the
final summary are visible. Per-item detail suppressed.
---
## ARRAY_START_SCRIPTS Order
## ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
> scripts.** Getting it wrong causes subtle failures that don't show up
> immediately.
> **Order matters.** Scripts that set up conditions other scripts depend on must run
> first. Getting the order wrong causes subtle failures.
```bash
# master.conf
# master.conf (current order — from ARRAY_START_SCRIPTS)
ARRAY_START_SCRIPTS=(
"inotify_tuning.sh" # 1 — FIRST: kernel limits must be set before
# any container starts. Containers inherit
# inotify limits at launch, not dynamically.
"docker_syslog_filter.sh" # 2 — SECOND: before any veth interfaces are
# created. If a container starts first, its
# veth creation is already in syslog.
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
...
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # plugin API key — ephemeral registry
"System_Essentials/conf_sync.sh" # 1st cross-server op — pull partner confs into RAM cache
"System_Essentials/conf_cache_restore.sh" # fill gaps if partner was down at boot
"Transcodes/ramdisk_setup.sh" # ramdisk before Emby starts
"System_Essentials/docker_syslog_filter.sh" # before veth interfaces are created
"Plugin/unraid/System_Essentials/php_fpm_max_children.sh" # WebGUI tuning
"System_Essentials/inotify_tuning.sh" # before docker_network_connect and continuous scripts
"Docker_Essentials/docker_network_connect.sh" # ensure networks + container connections
"Tools/claude_startup.sh" # persist Claude data to appdata; re-symlink on boot
"Media/start_webhook_listener.sh" # arr upgrade webhook — continuous
"Fallback/fallback.sh" # mutual failover — continuous
)
ARRAY_STOP_SCRIPTS=(
"System_Essentials/conf_cache_save.sh" # FIRST: snapshot RAM cache while fresh
"Plugin/unraid/System_Essentials/user_scripts_stop.sh"
"Fallback/fallback.sh --stop"
"System_Essentials/rsync_stop.sh --rsync-only"
"Plugin/unraid/System_Essentials/mover_stop.sh"
"Docker_Essentials/docker_container_stop.sh"
)
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
```
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
Why conf_sync before conf_cache_restore: the sync runs first to get the freshest
possible partner conf. The restore only fills in for confs that the sync couldn't
fetch — it is a fallback, not the primary path.
Why docker_syslog_filter SECOND: The filter must be in place before any container
starts creating veth interfaces. The first container start after array start
generates veth messages — these will appear in syslog if the filter isn't active.
Why docker_syslog_filter before inotify: the filter must be in place before any
container creates veth interfaces. inotify must be set before docker_network_connect
and the continuous scripts (start_webhook_listener, fallback), which are the first
steps that actually touch containers.
Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at
`/tmp/.cache/vv/d/` is wiped on reboot. Saving it must happen before anything
else shuts down or changes state.
---
## webgui_watchdog.sh
## conf_sync.sh
### Escalation Logic
### Modes
```
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
Not responding:
1. platform_restart_service nginx
wait WEBGUI_NGINX_WAIT (15s) → recheck
→ recovered: notify, exit 0
2. platform_restart_service php-fpm
wait WEBGUI_PHP_WAIT (10s) → recheck
→ recovered: notify, exit 0
3. platform_restart_service emhttp
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
→ recovered: notify, exit 0
All three failed → notify warning, exit 1
conf_sync.sh Full sync: pull from all partners + push to all partners
conf_sync.sh --push-only Push own conf to all partners (fast — for conf-save hook)
conf_sync.sh --pull-only Pull partner confs only (intermediate orchestrator)
conf_sync.sh --dry-run Show what would happen, no changes
conf_sync.sh --log Verbose output
```
### Configuration
### What It Syncs
- **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk
via SCP → writes to local `/tmp/.cache/vv/d/${partner_id}.conf`
- **Push**: sends own `Configurations/${my_id}.conf` to partner's
`/tmp/.cache/vv/d/${my_id}.conf` via SCP
- **Own conf in local cache**: copies own conf to `/tmp/.cache/vv/d/${my_id}.conf`
on full sync (so the cache has a complete snapshot of all confs)
Only partner confs are sourced from cache — `load_config.sh` always reads own conf
from disk to avoid sourcing a stale cached copy.
### Remote SCRIPTS_DIR Resolution
The pull path reads the partner's `/boot/config/plugins/varaverk/varaverk.cfg` to
find their actual `SCRIPTS_DIR` before building the SCP path. This handles the case
where the partner is in appdata storage mode and their conf is at
`/mnt/user/appdata/Varaverk/Configurations/` rather than the internal path.
### PARTNERSHIP_ENABLED Gate
conf_sync.sh calls `require_partnership` — if `PARTNERSHIP_ENABLED=false`, it exits
silently with 0. The conf cache will be empty for partner confs while partnership
is disabled.
### If Conf Pull Fails at Boot
If the partner is unreachable, `conf_sync.sh` logs a warning and exits 1.
`conf_cache_restore.sh` then runs (next in ARRAY_START_SCRIPTS) and loads the
persistent backup from `$PERSISTENT_CONF_CACHE` if available.
A notification fires if any partner fails — check partner reachability via Tailscale.
---
## conf_cache_save.sh
### What It Does
At array stop, copies all partner confs from `/tmp/.cache/vv/d/` to
`$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives
the reboot and is used by `conf_cache_restore.sh` at next array start if the sync
can't reach the partner.
`conf_cache_watchdog.sh` (in `Watchdogs/System/`) refreshes this backup every
15 minutes while the partner is offline — keeping it current even during extended
outages.
### PARTNERSHIP_ENABLED Gate
Exits silently when `PARTNERSHIP_ENABLED=false`. No backup is written.
### Usage
```bash
WEBGUI_URL="http://localhost" # URL to check
WEBGUI_TIMEOUT=5 # curl timeout in seconds
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
conf_cache_save.sh # save partner confs from RAM cache (called at array stop)
conf_cache_save.sh --dry-run # show what would be saved
conf_cache_save.sh --log # verbose output
```
### WebGUI Frozen — Manual Recovery
---
## conf_cache_restore.sh
### What It Does
At array start (after `conf_sync.sh`), checks which partner confs are missing from
`/tmp/.cache/vv/d/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
if a backup exists there.
Always removes the persistent backup when done — whether used or not. On a normal
reboot where the partner was reachable, the sync already populated the cache and the
backup is cleaned up without being used.
### PARTNERSHIP_ENABLED Gate
Exits silently when `PARTNERSHIP_ENABLED=false`.
### Usage
```bash
# Check which services are running:
webgui_watchdog.sh --status
# Try manual restart sequence (mirrors what the script calls):
# Source the ecosystem first to get platform functions:
source /boot/config/plugins/varaverk/load_config.sh
platform_restart_service nginx
# wait 15s, then:
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
# If nginx did not fix it, php-fpm:
platform_restart_service php-fpm
# If still down, emhttp:
platform_restart_service emhttp
# Raw equivalents (no source needed — paste directly into terminal):
# /etc/rc.d/rc.nginx restart
# /etc/rc.d/rc.php-fpm restart
# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
# If all three failed:
server_reboot.sh --status # check for active sessions first
conf_cache_restore.sh # restore missing confs from backup (called at array start)
conf_cache_restore.sh --dry-run # show what would be restored
conf_cache_restore.sh --log # verbose output
```
---
@@ -176,48 +223,6 @@ inotify_tuning.sh --log
---
## php_fpm_max_children.sh
### What It Sets
```bash
PHP_MAX_CHILDREN=250 # target pm.max_children (default: 4-8 on unRAID)
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
```
250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
The default of 4–8 saturates immediately under load on a busy server.
### Verify
```bash
php_fpm_max_children.sh --status
# Shows current value vs target, PHP-FPM worker count
# Manual verify:
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
# Expected: pm.max_children = 250
```
### If WebGUI Is Slow Despite the Setting
```bash
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
# pm.max_children value may still be too low, or there's a PHP worker leak.
# Check running worker count:
pgrep -fc php-fpm
# Compare to pm.max_children — if equal, workers are saturated
# Increase if needed:
# master.conf: PHP_MAX_CHILDREN=350
# Then: php_fpm_max_children.sh --log (will update and restart php-fpm)
```
---
## docker_syslog_filter.sh
### What It Creates
@@ -230,8 +235,7 @@ pgrep -fc php-fpm
```
This drops any syslog message containing "veth" or "docker0" before it reaches
any output target, including the log file. The drop rule is applied at rsyslog
level — not at the log viewer level.
any output target, including the log file.
### Verify
@@ -298,39 +302,6 @@ du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
---
## mover_stop.sh
### Stop Sequence
```
1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
2. Wall message to all logged-in terminal users
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
4. SIGTERM — mover finishes its current file operation, then stops
5. Wait 5 seconds → verify stopped
6. SIGKILL if still running — forced stop, partial files possible
7. Final verify — error if still running after SIGKILL
```
SIGTERM first because the mover can finish the file it is currently moving,
leaving no partial copies split across cache and array. SIGKILL is a last resort.
### Configuration
```bash
MOVER_STOP_TIMEOUT=30 # seconds between wall warning and SIGTERM
```
### Usage
```bash
mover_stop.sh # check and stop if running
mover_stop.sh --status # show current mover state and PID
mover_stop.sh --dry-run # show what would happen without stopping
```
---
## rsync_stop.sh
### Auto-Detection Logic
@@ -355,8 +326,8 @@ After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTA
for any that were stopped by the interrupted rsync session and restarts them.
Remote containers are left for docker_watchdog.sh to recover.
Skip container recovery with `--rsync-only` — used when called by other scripts
that handle recovery themselves.
Skip container recovery with `--rsync-only` — used when called by array_stopping.sh
(which handles container stop separately via docker_container_stop.sh).
### Usage
@@ -371,34 +342,6 @@ rsync_stop.sh --full-stop --dry-run # preview full stop
---
## user_scripts_stop.sh
### Process Identification
Scans `/proc/*/cmdline` for any process whose command line contains
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
what the script is named or how it was launched.
Script names are extracted from the path for display: you see which scripts are
being stopped, not just PIDs.
### Self-Exclusion
If this script is run via the User Scripts plugin, it would find its own PID in
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
from the kill list.
### Usage
```bash
user_scripts_stop.sh # stop all User Script processes
user_scripts_stop.sh --status # show running scripts with names and elapsed time
user_scripts_stop.sh --dry-run # show what would be stopped
```
---
## server_reboot.sh
### Full Shutdown Sequence
@@ -422,11 +365,12 @@ user_scripts_stop.sh --dry-run # show what would be stopped
6. /etc/rc.d/rc.libvirt stop (VM Manager)
7. Orchestrators/array_stopping.sh — safe ordered array stop:
- user_scripts_stop.sh stop background User Scripts
- fallback.sh --stop graceful fallback teardown
- rsync_stop.sh --rsync-only kill active rsync transfers
- mover_stop.sh stop mover
- docker_container_stop.sh stop all containers gracefully
- conf_cache_save.sh snapshot partner conf RAM cache
- user_scripts_stop.sh stop background User Scripts
- fallback.sh --stop graceful fallback teardown
- rsync_stop.sh --rsync-only kill active rsync transfers
- mover_stop.sh stop mover
- docker_container_stop.sh stop all containers gracefully
8. sync (flush filesystem buffers to disk)
@@ -446,7 +390,6 @@ For a clean reboot when services are active:
```bash
rsync_stop.sh # stop any active rsync (smart mode)
mover_stop.sh # stop mover gracefully
server_reboot.sh --status # check what's still running
server_reboot.sh --reason="planned maintenance"
```
@@ -464,27 +407,17 @@ server_reboot.sh --reason="disk work" # include reason in notification
## Full Configuration Reference
> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
```bash
# master.conf
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30
# ── Conf Sync ──────────────────────────────────────────────────────────────────
CONF_SYNC_ENABLED=true # toggle: false disables conf_sync.sh entirely
# ── inotify Tuning ─────────────────────────────────────────────────────────────
INOTIFY_MAX_INSTANCES=1024
INOTIFY_MAX_WATCHES=1048576
INOTIFY_MAX_QUEUED_EVENTS=32768
# ── PHP-FPM ────────────────────────────────────────────────────────────────────
PHP_MAX_CHILDREN=250
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
INOTIFY_WARN_PCT=80 # warn (in reports) if instances exceed this % of limit
# ── Syslog Filter ──────────────────────────────────────────────────────────────
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
@@ -494,20 +427,48 @@ LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
LOG_MIN_SIZE_MB=10
LOG_DOCKER_MAX_MB=100
# ── Mover Stop ─────────────────────────────────────────────────────────────────
MOVER_STOP_TIMEOUT=30
# ── Server Reboot ──────────────────────────────────────────────────────────────
REBOOT_SLEEP=30
REBOOT_VM_WAIT=30
# ── State Paths (auto-adapt to storage mode) ───────────────────────────────────
# STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
# They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
# (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
STATE_DIR="${SCRIPTS_DIR}/State_Files"
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
```
---
## Troubleshooting
> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
### Partner Conf Not In RAM Cache After Boot
```bash
# Check what's in the RAM cache:
ls -la /tmp/.cache/vv/d/
# Check what's in the persistent backup:
ls -la "$PERSISTENT_CONF_CACHE/" # set SCRIPTS_DIR first or use full path
# Re-run conf sync manually:
/boot/config/plugins/varaverk/System_Essentials/conf_sync.sh --log
# If partner is reachable but pull failed, check SSH key:
ssh -i /path/to/ssh_key root@<partner-tailscale-ip> "echo ok"
```
### Persistent Backup Is Stale or Empty
```bash
# conf_cache_watchdog.sh refreshes the backup while partner is offline.
# Check watchdog state:
cat "$STATE_DIR/conf_cache_watchdog_state.db"
# Force a manual backup from current RAM cache:
/boot/config/plugins/varaverk/System_Essentials/conf_cache_save.sh --log
```
### rsync_stop Killed the Wrong Thing
@@ -521,36 +482,29 @@ rsync_stop.sh --dry-run # shows smart mode decision
rsync_stop.sh --full-stop --dry-run # shows full-stop decision
```
### WebGUI Recovery After All Three Steps Failed
### inotify Exhaustion After Boot
```bash
# Check if processes are running:
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
pgrep emhttpd && echo "emhttp: yes" || echo "emhttp: no"
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
# Verify limits are applied:
sysctl fs.inotify.max_user_watches # expect 1048576
sysctl fs.inotify.max_user_instances # expect 1024
# Check recent nginx errors:
cat /var/log/nginx/error.log | tail -20
# If not set — run manually:
inotify_tuning.sh --log
# Check emhttp log:
tail -20 /var/log/syslog | grep emhttp
# Last resort — reboot:
server_reboot.sh --reason="WebGUI unrecoverable"
# Check current usage:
inotify_tuning.sh --status
```
### PHP-FPM Config Not Found After unRAID Update
unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
errors with "config file not found":
### Syslog Still Noisy After Array Start
```bash
# Find the new config path:
find /etc -name "www.conf" 2>/dev/null
# Check filter is in place:
docker_syslog_filter.sh --status
# Update PHP_CONF in master.conf:
PHP_CONF="/etc/php84/php-fpm.d/www.conf" # example for php84
# Re-apply if needed:
docker_syslog_filter.sh --log
# Verify with:
php_fpm_max_children.sh --status
# Restart rsyslog to pick up the filter:
/etc/rc.d/rc.rsyslogd restart
```
System_Essentials/README-System_Essentials.md
+79 -87
View File
@@ -1,35 +1,18 @@
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
**System-level scripts that act on the server itself — not containers,
not media, not monitoring.** Keeping the server stable under load, recovering a
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
shutdowns with proper warning sequences.
not media, not monitoring.** Kernel limits, log hygiene, conf synchronisation
between servers, and graceful shutdowns with proper warning sequences.
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) live in `Plugin/unraid/System_Essentials/`
> because they call Unraid-specific service commands and paths. This folder
> contains scripts that would run unchanged on any Linux host.
---
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
**Server Getting Into Unstable States With No Recovery Path**
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
watchdog tries to restart the container — but Docker itself is barely responding.
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
write log files. The server is functionally dead but still technically running.
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
threshold breaches. Last line of defense before a hard crash.
**WebGUI Freezing and Nobody Noticing**
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
load, settings don't save, containers can't be started or stopped via the UI. No
container-level alert fires because this isn't a container problem — it's a web
server problem. By the time someone notices it may have been broken for hours.
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
php-fpm → emhttp. Lightest fix first. Silent when healthy.
**50+ Containers Starting and Filling Syslog With Veth Noise**
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
network interface for each one. Each interface generates multiple syslog entries.
@@ -40,68 +23,79 @@ In the first minute after array start, syslog is buried under 200–400 lines of
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
starts. Applied at array start. Idempotent — silent when already correct.
**WebGUI Queuing Requests Under Load Without Explanation**
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
those workers saturate immediately. New requests queue behind active ones.
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
**inotify Exhaustion Producing Unexplained Failures**
When inotify limits are exhausted, containers silently stop receiving filesystem
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
and all containers share the same pool.
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
run FIRST in ARRAY_START_SCRIPTS before any containers start.
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start, before
any container-connecting or continuous scripts start.
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
The mover is running — moving a large batch of files from cache to array. A reboot
is triggered. The mover stops mid-file. The file exists partially on both cache and
array simultaneously. unRAID's deduplication layer is confused.
**Partner Conf Lost Across Reboots When Partner is Down**
Scripts like `fallback.sh` need the partner's conf vars (credentials, container
names, tier delays) to operate. The partner conf lives in a RAM cache at
`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls
a fresh copy from the partner. But if the partner is offline at boot time, the
pull fails and fallback has no partner vars to work with.
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
Fix: `conf_cache_save.sh` + `conf_cache_restore.sh` — snapshot the RAM cache to
`$PERSISTENT_CONF_CACHE` on array stop; reload on next start for any confs the
sync couldn't fetch.
**Mover/Rsync Interruption During Reboot**
Rsync transfers or mover runs are in progress when a reboot is triggered. Stopping
them uncleanly leaves partial files.
Fix: `server_reboot.sh` — orchestrates `array_stopping.sh` which stops rsync, mover,
and containers in the correct order before calling `/sbin/reboot`.
---
## ━━━ WHAT THIS FOLDER DOES ━━━
```
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
Kernel tuning inotify_tuning.sh — file watch limits
php_fpm_max_children.sh — PHP worker count
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
clear_logs.sh — weekly log trimming
Graceful operations mover_stop.sh — clean mover stop
rsync_stop.sh — smart rsync stop (orchestrator-aware)
user_scripts_stop.sh — stop running User Scripts
clear_logs.sh — size-threshold log trimming
Conf synchronisation conf_sync.sh — pull/push partner confs → RAM cache
conf_cache_save.sh — snapshot RAM cache → persistent at stop
conf_cache_restore.sh — reload from snapshot at start (offline partner)
Graceful operations rsync_stop.sh — smart rsync stop (orchestrator-aware)
server_reboot.sh — clean reboot with pre-flight warnings
```
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
---
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
```
Orchestrators/
array_started.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
─────────────────────────► docker_syslog_filter.sh (second)
─────────────────────────► php_fpm_max_children.sh
array_started.sh ────────────────────────► conf_sync.sh (pulls partner confs)
────────────────────────► conf_cache_restore.sh (fills gaps if partner down)
────────────────────────► docker_syslog_filter.sh (before containers start)
────────────────────────► inotify_tuning.sh (before docker_network_connect)
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
weekly_maintenance.sh ──────────────────► clear_logs.sh
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
Watchdogs/System/
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
(runs every 15 min via watchdog_orchestrator.sh)
Watchdogs/
stability_watchdog.sh and resource_watchdog.sh now live here.
See Watchdogs/README-Watchdogs.md for how they relate to each other
and to docker_watchdog.sh and System/storage_watchdog.sh.
System_Essentials/
server_reboot.sh ────────────────────────► array_stopping.sh (via Orchestrators/)
────────────────────────► mover_stop.sh, user_scripts_stop.sh
(via Plugin/unraid/System_Essentials/)
Plugin/unraid/System_Essentials/
php_fpm_max_children.sh — WebGUI tuning (Unraid-specific PHP paths)
mover_stop.sh — clean mover stop (Unraid mover daemon)
user_scripts_stop.sh — stop User Scripts plugin processes
unraid_api_key_renew.sh — Varaverk plugin API key renewal
Plugin/unraid/Watchdogs/System/
webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
```
---
@@ -110,14 +104,13 @@ Watchdogs/
| Script | Role | When It Runs |
|--------|------|-------------|
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — before container ops |
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
---
@@ -125,27 +118,28 @@ Watchdogs/
## ━━━ HOW THE SCRIPTS RELATE ━━━
```
Array starts
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
│
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
└─ php_fpm_max_children.sh ← before WebGUI is under load
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
│ push own conf to partner's /tmp/.cache/vv/d/
├─ conf_cache_restore.sh ← if partner was down: load last-known-good conf from
│ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live)
└─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts
Every minute (watchdog_orchestrator.sh in Orchestrators/):
→ Watchdogs/resource_watchdog.sh
→ Watchdogs/docker_watchdog.sh
→ Watchdogs/system_watchdog.sh (thin orchestrator)
└─ Watchdogs/System/storage_watchdog.sh
└─ Watchdogs/System/webgui_watchdog.sh
WebGUI OK → silent exit
Not responding:
Step 1: restart nginx → recheck
Step 2: restart php-fpm → recheck
Step 3: restart emhttp → recheck
All failed → notify, exit 1
└─ Watchdogs/System/network_watchdog.sh
→ Watchdogs/stability_watchdog.sh
(see Watchdogs/README-Watchdogs.md for full flow)
Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
│
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
│ while RAM cache is still fresh
├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
└─ ...other stop scripts...
Every 15 minutes (watchdog_orchestrator.sh):
→ Watchdogs/System/conf_cache_watchdog.sh
If partner is offline and persistent backup is stale → refresh from last RAM cache
Every 4 hours (intermediate_sync_maintenance.sh):
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
Weekly (weekly_maintenance.sh):
└─ clear_logs.sh
@@ -153,8 +147,6 @@ Weekly (weekly_maintenance.sh):
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
Manual operations:
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
server_reboot.sh → pre-flight → array_stopping.sh → reboot
```
System_Essentials/conf_cache_restore.sh
+1 -1
View File
@@ -30,7 +30,7 @@ detect_hosts
require_partnership
RAM_CACHE="/tmp/.cache/vv/d"
SAVE_DIR="/boot/config/.cache/vv/d"
SAVE_DIR="$PERSISTENT_CONF_CACHE"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
System_Essentials/conf_cache_save.sh
+3 -3
View File
@@ -5,7 +5,7 @@
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Snapshots the partner conf RAM cache to /boot/config/.cache/vv/d/ on
# Snapshots the partner conf RAM cache to $PERSISTENT_CONF_CACHE on
# array stop. Survives reboot. Used by conf_cache_restore.sh at next array start
# to reload partner vars into RAM when the partner is unreachable at boot time.
#
@@ -15,7 +15,7 @@
# down, the backup fills the gap so fallback.sh has the vars it needs.
#
# Only partner confs are saved — own conf is always on disk.
# Location is outside the git repo and outside the main plugin folder.
# Path adapts to storage mode: $SCRIPTS_DIR/.cache/vv/d (internal or appdata).
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -26,7 +26,7 @@ detect_hosts
require_partnership
RAM_CACHE="/tmp/.cache/vv/d"
SAVE_DIR="/boot/config/.cache/vv/d"
SAVE_DIR="$PERSISTENT_CONF_CACHE"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be written"
System_Essentials/conf_sync.sh
+28 -8
View File
@@ -11,15 +11,18 @@
# On array start (default / --array-start):
# 1. Copy own conf to local cache
# 2. Pull each available partner's conf from their disk → local cache
# 3. Push own conf to each available partner's /tmp/.vv/ cache
# 3. Push own conf to each available partner's /tmp/.cache/vv/d/ cache
#
# On conf save (--push-only):
# Fast path — push updated own conf to all partners' /tmp/.vv/ cache only.
# Fast path — push updated own conf to all partners' /tmp/.cache/vv/d/ only.
# No pulls, no local cache rebuild.
#
# Cache is /tmp (tmpfs) — cleared every reboot, repopulated by this script
# on next array start. Scripts source from cache for partner vars; own vars
# always come from disk (load_config.sh skips cached copy of own conf).
# Cache is /tmp/.cache/vv/d (tmpfs) — cleared every reboot, repopulated by
# this script on next array start. Scripts source from cache for partner vars;
# own vars always come from disk (load_config.sh skips cached copy of own conf).
#
# Pull path resolves the remote's SCRIPTS_DIR from their varaverk.cfg so it
# works whether the remote is in internal or appdata storage mode.
#
# ==============================================================================================
# RUNTIME MODES
@@ -56,9 +59,25 @@ if [[ "${CONF_SYNC_ENABLED:-true}" == false ]]; then
fi
CACHE_DIR="/tmp/.cache/vv/d"
MY_CONF="$SCRIPTS_ROOT/Configurations/${MY_ID,,}.conf"
MY_CONF="$SCRIPTS_DIR/Configurations/${MY_ID,,}.conf"
SSH_TIMEOUT=10
# Reads the remote's varaverk.cfg to find their actual SCRIPTS_DIR.
# Handles the case where the remote is in appdata storage mode.
_remote_scripts_dir() {
local ip="$1"
local cfg line sd
cfg=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
"root@${ip}" "cat /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null" 2>/dev/null) || true
while IFS= read -r line; do
[[ "$line" == SCRIPTS_DIR=* ]] || continue
sd="${line#SCRIPTS_DIR=}"; sd="${sd//\"/}"; sd="${sd//\'/}"
echo "$sd"; return
done <<< "$cfg"
echo "/boot/config/plugins/varaverk"
}
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
# ── Ensure cache dir exists ───────────────────────────────────────────────────
@@ -101,7 +120,8 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
# ── Pull: grab partner's conf from their disk → our local cache ──────────
if [[ "$PUSH_ONLY" == false ]]; then
remote_conf="${SCRIPTS_DIR}/Configurations/${partner_slot}.conf"
remote_sd=$(_remote_scripts_dir "$partner_ip")
remote_conf="${remote_sd}/Configurations/${partner_slot}.conf"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would pull $partner_host:$remote_conf → $CACHE_DIR/${partner_slot}.conf"
elif timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
@@ -116,7 +136,7 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
fi
fi
# ── Push: send own conf to partner's /tmp/.vv/ cache ────────────────────
# ── Push: send own conf to partner's /tmp/.cache/vv/d/ cache ───────────
if [[ "$PULL_ONLY" == true ]]; then
continue
fi
Tools/Manual-Tools.md
+1 -1
View File
@@ -214,7 +214,7 @@ View and manage the persistent container skip list used by `docker_watchdog.sh`.
```
docker_watchdog.sh restarts the same container N times within the rolling window
→ container added to skip list on /boot/config/
→ container added to skip list in $STATE_DIR
→ critical notification sent
→ watchdog stops touching it entirely
Watchdogs/Manual-Watchdogs.md
+55 -15
View File
@@ -13,8 +13,9 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
- [docker_watchdog.sh](#docker_watchdogsh)
- [system_watchdog.sh](#system_watchdogsh)
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) ← Plugin/unraid/Watchdogs/System/
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
- [System/conf_cache_watchdog.sh](#systemconf_cache_watchdogsh)
- [stability_watchdog.sh](#stability_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -373,6 +374,10 @@ storage_watchdog.sh --log # verbose per-container output
## System/webgui_watchdog.sh
> **Lives in `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh`** — calls Unraid-specific
> service commands (`rc.nginx`, `rc.php-fpm`, `emhttp`) via the platform adapter. Called by
> `system_watchdog.sh` via `SYSTEM_WATCHDOG_SCRIPTS` in master.conf.
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
through three restart steps if unresponsive. Silent when healthy.
@@ -446,7 +451,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
@@ -506,6 +511,40 @@ tailscale up
---
## System/conf_cache_watchdog.sh
Called by `system_watchdog.sh` each cycle. Maintains the persistent partner conf backup
at `$PERSISTENT_CONF_CACHE` while the partner is offline.
### Behaviour
**Remote online:** removes the persistent backup if one exists. It is not needed —
`conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to
`$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
an extended outage. If this host reboots while the partner is still down,
`conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
valid partner vars.
Silent when remote is online and no backup exists (the normal steady state).
### Gates
- `FALLBACK_ENABLED=false` → no-op (no fallback means no need for partner vars)
- `CONF_SYNC_ENABLED=false` → no-op
- `PARTNERSHIP_ENABLED=false` → exits silently (require_partnership gate)
### Usage
```bash
conf_cache_watchdog.sh # single pass (called by system_watchdog.sh)
conf_cache_watchdog.sh --dry-run # show what would be written or removed
conf_cache_watchdog.sh --log # verbose output
```
---
## stability_watchdog.sh
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
@@ -653,9 +692,9 @@ WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
WATCHDOG_BATCH_NOTIFY=true
# State files:
WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
# State files (all paths adapt to storage mode via STATE_DIR / DATA_DIR):
WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
# host*.conf
HOST1_WATCHDOG_CONTAINERS=() # "ContainerName:LimitMB"
@@ -670,8 +709,8 @@ WATCHDOG_APPDATA_GROWTH_GB=2
WATCHDOG_APPDATA_LOG_MAX_GB=2
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
WATCHDOG_APPDATA_STRIKE_LIMIT=3
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"
# host*.conf (optional — only for suppress ceilings)
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
@@ -701,11 +740,12 @@ SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=false
SYS_WATCHDOG_ABORT_ON_MOVER=false
# State files (all in State_Files/ — survive reboots):
SYS_WATCHDOG_STATE_FILE="State_Files/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="State_Files/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="State_Files/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="State_Files/system_watchdog_oom.db"
# State files — all in $STATE_DIR (survive reboots; adapt to storage mode):
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"
# ── Network Watchdog ───────────────────────────────────────────────────────────
NETWORK_WATCHDOG_ENABLED=true
@@ -714,7 +754,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
@@ -781,8 +821,8 @@ storage_watchdog.sh --status
### stability_watchdog Rebooted Unexpectedly
```bash
# Check the reboot log (survives reboots):
cat /boot/config/system_watchdog_reboots.db
# Check the reboot log (survives reboots — in State_Files/):
cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot
# Check syslog near the reboot time:
Watchdogs/README-Watchdogs.md
+46 -11
View File
@@ -3,8 +3,8 @@
**Four tiers that run every 15 minutes through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → check system components →
reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
System component checks (storage, WebGUI, network) run inside a thin `system_watchdog.sh`
orchestrator that can also be run standalone.
System component checks (storage, WebGUI, network, conf cache) run inside a thin
`system_watchdog.sh` orchestrator that can also be run standalone.
---
@@ -57,6 +57,15 @@ Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kern
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
sequence so container healing and pressure reduction always get a chance first.
**Partner Conf Backup Going Stale During a Long Outage**
The partner goes offline overnight. On the next reboot (planned maintenance), the
persistent conf backup saved at shutdown is stale — it reflects the state from before
the outage, not the most recent live values. `conf_cache_restore.sh` loads it
faithfully, but fallback.sh may be working with old tier delays or container names.
Fix: `conf_cache_watchdog.sh` — refreshes the persistent backup from the RAM cache
every 15 minutes while the partner is offline, keeping it current throughout the outage.
---
## ━━━ WHAT THIS FOLDER DOES ━━━
@@ -66,13 +75,17 @@ Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
```
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
System components system_watchdog.sh — thin orchestrator: system component health
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
network_watchdog.sh — internet, DDNS sync, Tailscale, NPM proxy
conf_cache_watchdog.sh — maintain persistent partner conf backup
Last resort stability_watchdog.sh — reboot only when nothing else can recover
```
> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
> but lives in the plugin tree because it calls Unraid-specific service commands
> (nginx, php-fpm, emhttp via platform adapter). See `Plugin/unraid/Watchdogs/System/`.
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
attempts restarts — containers restarted into a pressure-bound system just fail again.
System component checks run after containers are healed. Stability watchdog runs last —
@@ -96,8 +109,16 @@ Orchestrators/
Watchdogs/System/ ← called by system_watchdog.sh each cycle
storage_watchdog.sh pool growth rate + runaway log detection
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
network_watchdog.sh internet reachability, DDNS sync, Tailscale, NPM proxy
conf_cache_watchdog.sh maintain persistent partner conf backup during outages
Plugin/unraid/Watchdogs/System/
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
(Unraid-specific platform calls — lives in plugin tree)
Plugin/unraid/System_Essentials/
unraid_api_key_renew.sh re-registers the Varaverk plugin API key each watchdog cycle
(inserted between docker_watchdog and stability_watchdog)
Tools/
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
@@ -106,6 +127,10 @@ Tools/
Docker_Essentials/
All container lifecycle scripts (daily restart, updates, network) — unaffected.
docker_watchdog.sh coordinates with them via shared state, not direct calls.
System_Essentials/
conf_cache_save.sh / conf_cache_restore.sh — at-stop/at-start bookends for the
persistent backup that conf_cache_watchdog.sh keeps current in between.
```
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
@@ -119,14 +144,15 @@ Docker_Essentials/
|--------|------|-----------|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every 15 min |
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every 15 min |
| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
| `system_watchdog.sh` | Thin orchestrator — runs SYSTEM_WATCHDOG_SCRIPTS in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every 15 min |
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every 15 min |
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every 15 min |
| `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every 15 min |
| `System/conf_cache_watchdog.sh` | Refresh persistent partner conf backup while partner is offline | `system_watchdog.sh` — every 15 min |
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
> but is not in this folder. `watchdog_orchestrator.sh` is in `Orchestrators/`.
> `watchdog_skip_list_manager.sh` is in `Tools/`. Neither is a watchdog.
---
@@ -166,7 +192,7 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
│
│ └─ webgui_watchdog.sh
│ └─ Plugin/unraid/Watchdogs/System/webgui_watchdog.sh
│ curl check → WebGUI responding → exit 0 (silent)
│ not responding → nginx restart → wait → recheck
│ still down → php-fpm restart → wait → recheck
@@ -179,6 +205,13 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
│ Tailscale: status Running → pass; not running = notify (no auto-restart)
│ NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
│
│ └─ conf_cache_watchdog.sh
│ remote online → remove persistent backup (conf_sync gets fresh on next boot)
│ remote offline → refresh backup from RAM cache → backup stays current
│ silent when remote is online and no backup exists (normal state)
│
│ (+ Plugin/unraid/System_Essentials/unraid_api_key_renew.sh — between steps 3 and 4)
│
Step 4 — stability_watchdog.sh
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
Tier 1 CRITICAL → immediate reboot (no strikes)
@@ -195,7 +228,9 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `DOCKER_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
Watchdogs/System/conf_cache_watchdog.sh
+2 -2
View File
@@ -5,7 +5,7 @@
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Maintains the persistent partner conf backup at /boot/config/.cache/vv/d/.
# Maintains the persistent partner conf backup at $PERSISTENT_CONF_CACHE.
# Runs every 15 minutes via SYSTEM_WATCHDOG_SCRIPTS.
#
# When remote is OFFLINE:
@@ -33,7 +33,7 @@ require_partnership
[[ -z "${REMOTE_ID:-}" ]] && exit 0
RAM_CACHE="/tmp/.cache/vv/d"
SAVE_DIR="/boot/config/.cache/vv/d"
SAVE_DIR="$PERSISTENT_CONF_CACHE"
if ping_remote; then
if [[ -d "$SAVE_DIR" ]]; then