Storage-mode awareness pass + doc update for System_Essentials through Partnership
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR / PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the ecosystem works in both internal and appdata storage modes. PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php, config.php): all state reads switched to STATE_DIR constant; remote state reads use the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via their varaverk.cfg before building the path. conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added _remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull. fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED, FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card. README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync, Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync, conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener, upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to $STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js, not PHP -S).
This commit is contained in:
Gmer4Lfe
@@ -10,6 +10,7 @@ Configurations/*.bak
|
||||
# ── Runtime state, data, logs ─────────────────────────────────────────────────
|
||||
data/
|
||||
State_Files/
|
||||
.cache/
|
||||
*.log
|
||||
*.lock
|
||||
|
||||
|
||||
@@ -123,6 +123,7 @@
|
||||
# Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no /boot/config root.
|
||||
DATA_DIR="${SCRIPTS_DIR}/data"
|
||||
STATE_DIR="${SCRIPTS_DIR}/State_Files"
|
||||
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
|
||||
|
||||
# ── Version Parity ──
|
||||
# Controls behaviour when local and remote unRAID versions differ.
|
||||
@@ -295,7 +296,7 @@
|
||||
# watchdog_orchestrator.sh — NOT launched here.
|
||||
ARRAY_START_SCRIPTS=(
|
||||
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # re-register Varaverk API key — registry is ephemeral
|
||||
"System_Essentials/conf_sync.sh" # pull partner confs + push own conf into /tmp/.vv/ RAM cache
|
||||
"System_Essentials/conf_sync.sh" # pull partner confs + push own conf into /tmp/.cache/vv/d/ RAM cache
|
||||
"System_Essentials/conf_cache_restore.sh" # load partner confs from persistent backup if conf_sync couldn't reach partner
|
||||
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
|
||||
"System_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
# EMPTY fields, never overwrites existing values unless --overwrite is passed.
|
||||
#
|
||||
# After populating, pushes the updated conf to all partners via conf_sync.sh
|
||||
# so they have the fresh keys in their /tmp/.vv/ cache immediately.
|
||||
# so they have the fresh keys in their /tmp/.cache/vv/d/ cache immediately.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# AUTO-DETECTED FIELDS
|
||||
|
||||
@@ -120,6 +120,7 @@
|
||||
# Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no /boot/config root.
|
||||
DATA_DIR="/boot/config/plugins/varaverk/data"
|
||||
STATE_DIR="/boot/config/plugins/varaverk/State_Files"
|
||||
PERSISTENT_CONF_CACHE="/boot/config/plugins/varaverk/.cache/vv/d"
|
||||
|
||||
# ── Version Parity ──
|
||||
# Controls behaviour when local and remote unRAID versions differ.
|
||||
|
||||
@@ -139,7 +139,7 @@ happens after an update, and you want to know when it does.
|
||||
A container keeps appearing in a broken state. You SSH in and see it's stopped. You
|
||||
don't know if the watchdog tried to restart it and failed, gave up and skip-listed it,
|
||||
is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip
|
||||
list file on `/boot/config/`, check the restart history file, check the state file —
|
||||
list file in `$STATE_DIR`, check the restart history file, check the state file —
|
||||
none of which have obvious formats.
|
||||
|
||||
The fix: `watchdog_skip_list_manager.sh` (in `Tools/`). One command to see exactly what's
|
||||
|
||||
+22
-16
@@ -37,11 +37,11 @@ up required before handback sequence starts. **(default: 3)**
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
|
||||
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
|
||||
```
|
||||
Path to the persistent state file. Lives on `/boot/` intentionally — survives reboots.
|
||||
If the server was in FALLBACK state when it rebooted, it resumes FALLBACK on restart
|
||||
rather than assuming everything is normal.
|
||||
Path to the persistent state file. In `$STATE_DIR` — survives reboots whether storage
|
||||
mode is internal (boot device) or appdata (array). If the server was in FALLBACK state
|
||||
when it rebooted, it resumes FALLBACK on restart rather than assuming everything is normal.
|
||||
|
||||
---
|
||||
|
||||
@@ -172,7 +172,7 @@ daily_sync_maintenance.sh uses, in the opposite direction. No separate TIER4 lis
|
||||
FALLBACK_ENABLED=true
|
||||
FALLBACK_CHECK_INTERVAL=30
|
||||
FALLBACK_HANDBACK_STRIKES=3
|
||||
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
|
||||
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
|
||||
FALLBACK_RSYNC_ENABLED=true
|
||||
EXTERNAL_IP=8.8.8.8
|
||||
FALLBACK_TEST_BLOCK_WAIT=60
|
||||
@@ -293,7 +293,10 @@ FALLBACK_HOST2_WRITEBACK_TIER1=(
|
||||
|
||||
## ━━━ STATE FILE REFERENCE ━━━
|
||||
|
||||
Location: `/boot/config/fallback_state.db` (survives reboots)
|
||||
Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata)
|
||||
|
||||
> In a shell where load_config.sh is not sourced, use the full path:
|
||||
> `/boot/config/plugins/varaverk/State_Files/fallback_state.db` (internal storage mode)
|
||||
|
||||
```
|
||||
state=NORMAL # NORMAL | FALLBACK | NO_INTERNET | DARK
|
||||
@@ -304,8 +307,8 @@ tier3_started=false # whether Tier 3 containers started
|
||||
tier4_started=false # whether Tier 4 containers started
|
||||
```
|
||||
|
||||
View state: `cat /boot/config/fallback_state.db`
|
||||
Check state: `fallback.sh --status`
|
||||
View state: `fallback.sh --status` (preferred — parsed output)
|
||||
Raw file: `cat "$STATE_DIR/fallback_state.db"` (requires STATE_DIR set, or use full path)
|
||||
|
||||
The file is managed exclusively by fallback.sh. Do not edit it while fallback.sh is
|
||||
running — the next cycle will overwrite your changes. Use the Manual State Reset
|
||||
@@ -384,7 +387,7 @@ servers must be running it continuously for mutual coverage.
|
||||
pgrep -f "fallback.sh"
|
||||
|
||||
# Check the state file
|
||||
cat /boot/config/fallback_state.db
|
||||
cat "$STATE_DIR/fallback_state.db"
|
||||
```
|
||||
|
||||
Start via User Scripts plugin on both servers.
|
||||
@@ -405,7 +408,7 @@ in smart mode — a non-NORMAL state at digest time needs attention.
|
||||
|
||||
```bash
|
||||
fallback.sh --status # full state snapshot
|
||||
cat /boot/config/fallback_state.db # raw state file
|
||||
cat "$STATE_DIR/fallback_state.db" # raw state file
|
||||
```
|
||||
|
||||
---
|
||||
@@ -426,7 +429,7 @@ fallback_test.sh
|
||||
|
||||
# Step 3 — check state after test completes
|
||||
fallback.sh --status
|
||||
cat /boot/config/fallback_state.db
|
||||
cat "$STATE_DIR/fallback_state.db"
|
||||
```
|
||||
|
||||
If the test doesn't complete cleanly, the state file may be left in FALLBACK. The
|
||||
@@ -459,11 +462,14 @@ ping -c 5 [remote-tailscale-ip]
|
||||
**Stop fallback.sh first (via User Scripts Abort), then reset:**
|
||||
|
||||
```bash
|
||||
# Set STATE_DIR (or source load_config.sh to get it from the environment)
|
||||
source /boot/config/plugins/varaverk/load_config.sh
|
||||
|
||||
# View current state
|
||||
cat /boot/config/fallback_state.db
|
||||
cat "$STATE_DIR/fallback_state.db"
|
||||
|
||||
# Write a clean NORMAL state
|
||||
cat > /boot/config/fallback_state.db << 'EOF'
|
||||
cat > "$STATE_DIR/fallback_state.db" << 'EOF'
|
||||
state=NORMAL
|
||||
fallback_start=0
|
||||
handback_strikes=0
|
||||
@@ -473,7 +479,7 @@ tier4_started=false
|
||||
EOF
|
||||
|
||||
# Verify the write
|
||||
cat /boot/config/fallback_state.db
|
||||
cat "$STATE_DIR/fallback_state.db"
|
||||
```
|
||||
|
||||
Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its next cycle.
|
||||
@@ -510,7 +516,7 @@ Can this server reach the remote Tailscale IP?
|
||||
|
||||
What does fallback.sh report?
|
||||
→ fallback.sh --status
|
||||
→ cat /boot/config/fallback_state.db
|
||||
→ cat "$STATE_DIR/fallback_state.db"
|
||||
```
|
||||
|
||||
### Handback Not Completing
|
||||
@@ -559,7 +565,7 @@ If it has happened:
|
||||
|
||||
3. Understand the state before resetting
|
||||
fallback.sh --status
|
||||
cat /boot/config/fallback_state.db
|
||||
cat "$STATE_DIR/fallback_state.db"
|
||||
|
||||
4. Perform Manual State Reset above on the server in a bad state
|
||||
|
||||
|
||||
@@ -130,11 +130,13 @@
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# FALLBACK_STATE_FILE — /boot/config/fallback_state.db (survives reboots)
|
||||
# FALLBACK_STATE_FILE — $STATE_DIR/fallback_state.db (survives reboots)
|
||||
# Keys: state, fallback_start, handback_strikes, tier2_started,
|
||||
# tier3_started, tier4_started, partnership_suspended, partner_lost_at.
|
||||
# Lives on /boot/ intentionally — if the server was in FALLBACK when it
|
||||
# rebooted, it resumes FALLBACK on restart.
|
||||
# Survives reboots — $STATE_DIR is on the boot device (internal) or appdata
|
||||
# (flash). Either way the array is up before this script runs, so the file
|
||||
# is always accessible. If the server was in FALLBACK when it rebooted,
|
||||
# it resumes FALLBACK on restart.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -185,7 +187,7 @@
|
||||
# Consecutive remote-up checks required before handback begins. (default: 3)
|
||||
#
|
||||
# FALLBACK_STATE_FILE
|
||||
# State file path — /boot/config/fallback_state.db — survives reboots.
|
||||
# State file path — $STATE_DIR/fallback_state.db — survives reboots.
|
||||
#
|
||||
# FALLBACK_RSYNC_ENABLED
|
||||
# Gate for writeback rsync jobs during handback. (default: true)
|
||||
@@ -302,7 +304,7 @@ log "$ICON_GEAR Timeouts: docker=${DOCKER_TIMEOUT}s ssh=${SSH_TIMEOUT}s verify-w
|
||||
# ==============================================================================================
|
||||
# ── STATE FILE HELPERS ────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# State file on /boot/config — survives reboots.
|
||||
# State file at $FALLBACK_STATE_FILE ($STATE_DIR/fallback_state.db) — survives reboots.
|
||||
# Format: key=value one per line.
|
||||
# Keys: state, fallback_start, handback_strikes, tier2_started, tier3_started, tier4_started
|
||||
|
||||
|
||||
@@ -409,7 +409,7 @@ pgrep -a -f fallback.sh
|
||||
|
||||
Check current state:
|
||||
```bash
|
||||
cat /boot/config/fallback_state.db
|
||||
cat "$STATE_DIR/fallback_state.db"
|
||||
# state=NORMAL — both servers up
|
||||
```
|
||||
|
||||
@@ -510,7 +510,7 @@ bash Orchestrators/daily_sync_maintenance.sh --dry-run --log
|
||||
### ── Check fallback state ────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
cat /boot/config/fallback_state.db
|
||||
cat "$STATE_DIR/fallback_state.db"
|
||||
# Expected:
|
||||
# state=NORMAL
|
||||
# fallback_start=0
|
||||
@@ -600,7 +600,7 @@ The remote server is unreachable on the first check. Common causes:
|
||||
|
||||
```bash
|
||||
# Confirm which state fallback is in
|
||||
cat /boot/config/fallback_state.db
|
||||
cat "$STATE_DIR/fallback_state.db"
|
||||
# If stuck in FALLBACK after remote comes back: reset state
|
||||
bash Tools/fallback_state_reset.sh
|
||||
```
|
||||
|
||||
+45
-4
@@ -132,8 +132,8 @@ LIDARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan
|
||||
LIDARR_VERSION_MAJOR=3 # expected Lidarr major version (API safety check)
|
||||
LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
|
||||
LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc")
|
||||
LIDARR_TRACKED_COUNT_FILE=/boot/config/lidarr_tracked_count # persistent baseline
|
||||
ARR_CLEANUP_STATS=/boot/config/arr_cleanup_stats.db # read by coffee report
|
||||
LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count" # persistent baseline
|
||||
ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db" # read by coffee report
|
||||
```
|
||||
|
||||
---
|
||||
@@ -171,7 +171,7 @@ RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.a
|
||||
|
||||
```bash
|
||||
ARR_SYNC_ENABLED=true
|
||||
ARR_SYNC_BLOCKLIST=/boot/config/arr_sync_blocklist.tsv # tombstone file
|
||||
ARR_SYNC_BLOCKLIST="$DATA_DIR/arr_sync_blocklist.tsv" # tombstone file
|
||||
ARR_SYNC_CONNECT_TIMEOUT=10 # SSH connect timeout in seconds
|
||||
ARR_SYNC_API_TIMEOUT=60 # curl API call timeout in seconds
|
||||
DOCKER_APPDATA_BASE=/mnt/user/appdata
|
||||
@@ -189,7 +189,7 @@ ARR_IMPORT_RECOVERY_AGE=6 # hours — items newer than this are skipped
|
||||
SONARR_VERSION_MAJOR=4
|
||||
RADARR_VERSION_MAJOR=6
|
||||
LIDARR_VERSION_MAJOR=3
|
||||
ARR_RECOVERY_STATS=/boot/config/arr_recovery_stats.db # read by coffee report
|
||||
ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db" # read by coffee report
|
||||
```
|
||||
|
||||
---
|
||||
@@ -274,6 +274,47 @@ Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
|
||||
|
||||
---
|
||||
|
||||
### Play State Sync
|
||||
|
||||
```bash
|
||||
PLAY_STATE_SYNC_ENABLED=true # toggle entire sync
|
||||
PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items
|
||||
```
|
||||
|
||||
Emby and Jellyfin servers configured per-host:
|
||||
|
||||
```bash
|
||||
# host*.conf
|
||||
HOST1_EMBY_URL="http://192.168.50.2:8096"
|
||||
HOST1_EMBY_API_KEY="..."
|
||||
HOST1_JELLYFIN_URL="" # empty = skip Jellyfin on this host
|
||||
HOST1_JELLYFIN_API_KEY=""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Upgrade Webhook
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WEBHOOK_PORT=9456 # 0 = disable listener
|
||||
WEBHOOK_SECRET="" # auto-generated on first start if empty
|
||||
WEBHOOK_UPGRADE_RSYNC_ENABLED=true # push upgraded file to remote nodes
|
||||
```
|
||||
|
||||
```bash
|
||||
# host*.conf
|
||||
HOST1_WEBHOOK_LAN_IP="192.168.50.2" # IP arrs call for webhook delivery
|
||||
```
|
||||
|
||||
When `WEBHOOK_PORT=0`, `start_webhook_listener.sh` exits cleanly and no listener starts.
|
||||
When `WEBHOOK_SECRET` is empty, a 32-byte hex secret is generated on first start and
|
||||
written back to `master.conf`. Run `Tools/webhook_setup.sh` to register the URL in arrs.
|
||||
|
||||
Webhook log: `/var/log/varaverk/upgrade_webhook.log`
|
||||
|
||||
---
|
||||
|
||||
### Orchestrator Job Order
|
||||
|
||||
```bash
|
||||
|
||||
+48
-2
@@ -4,8 +4,9 @@ Library health, consistency, sync, and behavior-driven discovery for a multi-ser
|
||||
stack. Correct permissions so arrs can manage files. Junk removal so orphan detection
|
||||
isn't confused by scene debris. Library sync so every node tracks the same content.
|
||||
Orphan cleanup against live arr APIs so deleted content actually leaves disk. Emby
|
||||
notified automatically after every deletion. Weekly discovery adds new music, movies,
|
||||
and TV shows based on what you actually play — no manual browsing required.
|
||||
notified automatically after every deletion. Watch state synced across Emby/Jellyfin
|
||||
every 30 minutes. Quality upgrades propagate to all nodes immediately via webhook.
|
||||
Weekly discovery adds new music, movies, and TV shows based on what you actually play.
|
||||
|
||||
> **These scripts permanently delete files.** The arr cleanup scripts are protected by
|
||||
> multiple safety layers that must all pass before anything is touched — but dry runs and
|
||||
@@ -91,6 +92,19 @@ free overnight recovery.
|
||||
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
|
||||
fallback sources. Never overwrites existing files.
|
||||
|
||||
**Play State Sync**
|
||||
`play_state_sync.sh` — syncs watched/played state and resume positions across all
|
||||
configured Emby and Jellyfin servers. Newest timestamp wins. Runs every 30 minutes
|
||||
via critical_sync_maintenance.sh.
|
||||
|
||||
**Upgrade Propagation**
|
||||
`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr
|
||||
OnUpgrade webhooks. Continuous; started at array start. Writes to
|
||||
`/var/log/varaverk/upgrade_webhook.log`.
|
||||
`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded
|
||||
item folder to every remote node immediately, then triggers an arr library rescan on
|
||||
each remote so the upgraded file is accepted without triggering a redundant quality search.
|
||||
|
||||
**Discovery**
|
||||
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
|
||||
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
|
||||
@@ -138,6 +152,24 @@ playback_aware_radarr_discovery.sh — score watch history → TMDB recommenda
|
||||
playback_aware_sonarr_discovery.sh — score episode history → TMDB TV recommendations → add to Sonarr
|
||||
```
|
||||
|
||||
**Every 30 min via `critical_sync_maintenance.sh` (CRITICAL_MAINTENANCE_SCRIPTS):**
|
||||
|
||||
```
|
||||
play_state_sync.sh — sync watched/resume state across Emby + Jellyfin
|
||||
```
|
||||
|
||||
**Continuous (started by `array_started.sh`):**
|
||||
|
||||
```
|
||||
start_webhook_listener.sh — Node.js webhook server; dispatch upgrade_webhook_handler.sh
|
||||
```
|
||||
|
||||
**On every arr upgrade (triggered by webhook):**
|
||||
|
||||
```
|
||||
upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan
|
||||
```
|
||||
|
||||
**Ad-hoc or separate schedule:**
|
||||
|
||||
```
|
||||
@@ -189,6 +221,9 @@ in `HOST*_MEDIA_PERMISSION_SHARES` and `HOST*_MEDIA_CLEAN_FOLDERS` in host*.conf
|
||||
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Ad-hoc or separate schedule |
|
||||
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Ad-hoc or weekly |
|
||||
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Ad-hoc or weekly |
|
||||
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min via `critical_sync_maintenance.sh` |
|
||||
| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous (started by `array_started.sh`) |
|
||||
| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade (via webhook) |
|
||||
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly via `weekly_sync_maintenance.sh` |
|
||||
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly via `weekly_sync_maintenance.sh` |
|
||||
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly via `weekly_sync_maintenance.sh` |
|
||||
@@ -228,6 +263,17 @@ Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
|
||||
│
|
||||
└── each discovery script fires arr search immediately after successful add
|
||||
|
||||
Every 30 min (critical_sync_maintenance.sh CRITICAL_MAINTENANCE_SCRIPTS):
|
||||
play_state_sync.sh ─── newest timestamp wins → watched/resume state synced
|
||||
across all configured Emby + Jellyfin servers
|
||||
|
||||
Continuous (started by array_started.sh):
|
||||
start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
|
||||
│ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
|
||||
└── upgrade_webhook_handler.sh
|
||||
├── rsync upgraded folder → all remote nodes immediately
|
||||
└── trigger arr library rescan on each remote (accept new file, no quality search)
|
||||
|
||||
Ad-hoc enrichment:
|
||||
lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
|
||||
radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Starts a standalone PHP HTTP server that receives Sonarr/Radarr/Lidarr
|
||||
# Starts a standalone Node.js HTTP server that receives Sonarr/Radarr/Lidarr
|
||||
# OnUpgrade webhooks and dispatches upgrade_webhook_handler.sh.
|
||||
#
|
||||
# Runs outside Unraid nginx — no session auth required. The shared secret in
|
||||
@@ -13,9 +13,12 @@
|
||||
#
|
||||
# http://<HOST_LAN_IP>:<WEBHOOK_PORT>/webhook?key=<WEBHOOK_SECRET>
|
||||
#
|
||||
# Runs as a continuous script started by array_started.sh. Execs php -S which
|
||||
# Runs as a continuous script started by array_started.sh. Execs node which
|
||||
# replaces this process — the PID stays the same for array_started.sh's check.
|
||||
#
|
||||
# Uses Node.js instead of php -S: php -S on Unraid PHP 8.4 silently drops
|
||||
# POST request bodies, making webhook payloads arrive empty.
|
||||
#
|
||||
# If WEBHOOK_SECRET is empty in master.conf: generates and saves one, then starts.
|
||||
# If WEBHOOK_PORT is 0: exits cleanly (disables the listener).
|
||||
#
|
||||
|
||||
@@ -151,14 +151,14 @@ BACKUP_VERIFY_MIN_SIZE="1M" # skip files smaller than this
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
BANDWIDTH_LOG="/boot/config/bandwidth_history.db" # survives reboots
|
||||
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db" # survives reboots
|
||||
BANDWIDTH_LOG_RETENTION=90 # days — file stays bounded, never grows unbounded
|
||||
BANDWIDTH_WARN_GB=50 # flag transfers or daily totals exceeding this
|
||||
```
|
||||
|
||||
**Why `/boot/config/`**: The log needs to survive reboots to build a useful history.
|
||||
`/boot/config/` is on the USB flash drive, which survives reboots and is backed up
|
||||
by unRAID's flash backup. The log is bounded by `BANDWIDTH_LOG_RETENTION` so it never
|
||||
**Why `$DATA_DIR`**: The log needs to survive reboots to build a useful history.
|
||||
`$DATA_DIR` (`$SCRIPTS_DIR/data/`) is on the boot device (internal) or appdata (flash),
|
||||
both of which survive reboots. The log is bounded by `BANDWIDTH_LOG_RETENTION` so it never
|
||||
grows unbounded.
|
||||
|
||||
**`BANDWIDTH_WARN_GB`**: Set to a value that represents "unexpectedly large" for your
|
||||
@@ -401,7 +401,7 @@ BACKUP_VERIFY_SAMPLE=10
|
||||
BACKUP_VERIFY_MIN_SIZE="1M"
|
||||
|
||||
# bandwidth_monitor.sh
|
||||
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
|
||||
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
|
||||
BANDWIDTH_LOG_RETENTION=90
|
||||
BANDWIDTH_WARN_GB=50
|
||||
|
||||
|
||||
@@ -849,7 +849,7 @@ long tests — that should only run on stable systems that have been up for at l
|
||||
|
||||
Two gates must both pass before any job runs:
|
||||
1. Server uptime ≥ `MONTHLY_UPTIME_THRESHOLD_DAYS`
|
||||
2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file on `/boot/config/` — survives reboots)
|
||||
2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file in `$STATE_DIR` — survives reboots)
|
||||
|
||||
If either gate fails, the script exits 0 with no output. This is expected — it runs
|
||||
daily and most days are no-ops.
|
||||
@@ -866,7 +866,7 @@ MONTHLY_MAINTENANCE_SCRIPTS=(
|
||||
)
|
||||
MONTHLY_UPTIME_THRESHOLD_DAYS=30
|
||||
MONTHLY_RUN_INTERVAL_DAYS=30
|
||||
MONTHLY_LAST_RUN_FILE="/boot/config/monthly_maintenance_last_run.db"
|
||||
MONTHLY_LAST_RUN_FILE="$STATE_DIR/monthly_maintenance_last_run.db"
|
||||
```
|
||||
|
||||
Scripts are commented out by default — uncomment what applies to your hardware.
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
# Schedule: 0 */4 * * * (every 4 hours)
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# 1. conf_sync.sh --pull-only — refresh partner conf cache in RAM (/tmp/.vv/)
|
||||
# 1. conf_sync.sh --pull-only — refresh partner conf cache in RAM (/tmp/.cache/vv/d/)
|
||||
# 2. arr_sync.sh — sync Lidarr/Sonarr/Radarr libraries across all nodes
|
||||
# 3. Rsync window (optional) — INTERMEDIATE_SYNC_SHARES, if any configured
|
||||
# 4. INTERMEDIATE_MAINTENANCE_SCRIPTS — artwork fetch and any future 4-hour jobs
|
||||
|
||||
@@ -574,7 +574,7 @@ Partnership/partnership_manager.sh --status # shows both sides via SSH
|
||||
|
||||
```bash
|
||||
# Check the offline counter:
|
||||
cat /boot/config/partnership_offline_days.db
|
||||
cat "$STATE_DIR/partnership_offline_days.db"
|
||||
|
||||
# Extended Tailscale outage may have incremented the counter.
|
||||
# Check Tailscale peer visibility:
|
||||
@@ -591,9 +591,9 @@ Partnership/partnership_manager.sh --onboard
|
||||
## ━━━ STATE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
/boot/config/partnership_HOST1.db # HOST1 writes only
|
||||
/boot/config/partnership_HOST2.db # HOST2 writes only
|
||||
/boot/config/partnership_blocklist.db # hostname|timestamp|reason
|
||||
$STATE_DIR/partnership_HOST1.db # HOST1 writes only
|
||||
$STATE_DIR/partnership_HOST2.db # HOST2 writes only
|
||||
$STATE_DIR/partnership_blocklist.db # hostname|timestamp|reason
|
||||
|
||||
# Example state file:
|
||||
state=ACTIVE
|
||||
|
||||
@@ -93,11 +93,11 @@
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# /boot/config/partnership_HOST1.db — HOST1 writes, HOST2 reads via SSH
|
||||
# /boot/config/partnership_HOST2.db — HOST2 writes, HOST1 reads via SSH
|
||||
# /boot/config/partnership_blocklist.db — hostname|timestamp|reason, persists until cleared
|
||||
# $STATE_DIR/partnership_<hostname>.db — each host writes its own, partner reads via SSH
|
||||
# $STATE_DIR/partnership_blocklist.db — hostname|timestamp|reason, persists until cleared
|
||||
# $STATE_DIR/partnership_offline_days.db — cumulative offline day counter
|
||||
#
|
||||
# On /boot/config — survives reboots, available before array starts, minimal flash wear.
|
||||
# All in STATE_DIR — survives reboots (on /boot in internal mode, appdata in flash mode).
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
|
||||
@@ -274,7 +274,7 @@ platform_get_templates_dir() {
|
||||
# Writes the path to the persistent Varaverk setup/wizard state database.
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
platform_setup_db_path() {
|
||||
echo "/boot/config/varaverk_setup.db"
|
||||
echo "${STATE_DIR}/varaverk_setup.db"
|
||||
}
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -13,7 +13,7 @@ $ramUsedMb = $ramTotalMb - $res['ram_free_mb'];
|
||||
$ramPct = $ramTotalMb > 0 ? (int)round($ramUsedMb / $ramTotalMb * 100) : 0;
|
||||
|
||||
// Fallback state (fast file read, no exec)
|
||||
$fbRaw = @file_get_contents('/tmp/fallback_state.db') ?: '';
|
||||
$fbRaw = @file_get_contents(STATE_DIR . '/fallback_state.db') ?: '';
|
||||
$fbData = vv_parse_kv_db($fbRaw);
|
||||
$fallbackState = $fbData['state'] ?? 'UNKNOWN';
|
||||
|
||||
|
||||
@@ -34,8 +34,7 @@ function vv_setup_state_write(array $data): void {
|
||||
}
|
||||
|
||||
// Push the setup state file to all remote hosts via scp.
|
||||
// Unlike master.conf push, this does NOT require the plugin to be installed on the remote —
|
||||
// it only needs SSH to be reachable, and pushes to /boot/config/ (always available).
|
||||
// Reads the remote's varaverk.cfg to find their actual SCRIPTS_DIR (handles appdata mode).
|
||||
function vv_push_setup_state(): void {
|
||||
if (!file_exists(VV_SETUP_STATE_FILE)) return;
|
||||
$myHostId = vv_detect_host();
|
||||
@@ -56,9 +55,20 @@ function vv_push_setup_state(): void {
|
||||
if (!$ip) continue;
|
||||
$sshBase = 'ssh -i ' . escapeshellarg($sshKey)
|
||||
. ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@' . $ip;
|
||||
// Ensure the target dir exists (it always should on Unraid, but be safe)
|
||||
shell_exec($sshBase . ' "mkdir -p /boot/config" 2>/dev/null');
|
||||
$dest = escapeshellarg('root@' . $ip . ':/boot/config/varaverk_setup.db');
|
||||
|
||||
// Get remote SCRIPTS_DIR from varaverk.cfg — handles appdata mode on remote.
|
||||
// Falls back to the default install path if varaverk.cfg is absent (pre-install).
|
||||
$cfgRaw = trim(shell_exec($sshBase . ' "cat /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null"') ?: '');
|
||||
$remoteSD = '/boot/config/plugins/varaverk';
|
||||
foreach (explode("\n", $cfgRaw) as $line) {
|
||||
if (str_starts_with(trim($line), 'SCRIPTS_DIR=')) {
|
||||
$remoteSD = trim(substr(trim($line), strlen('SCRIPTS_DIR=')), '"\'');
|
||||
break;
|
||||
}
|
||||
}
|
||||
$remoteStatePath = $remoteSD . '/State_Files/varaverk_setup.db';
|
||||
shell_exec($sshBase . ' "mkdir -p ' . escapeshellarg(dirname($remoteStatePath)) . '" 2>/dev/null');
|
||||
$dest = escapeshellarg('root@' . $ip . ':' . $remoteStatePath);
|
||||
exec('scp -i ' . escapeshellarg($sshKey)
|
||||
. ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no'
|
||||
. ' ' . escapeshellarg(VV_SETUP_STATE_FILE) . ' ' . $dest . ' 2>&1');
|
||||
@@ -391,6 +401,15 @@ function vv_auto_create_api_key(string $hostId, string $confFile): array {
|
||||
return ['ok' => true, 'key_preview' => $key ? substr($key, 0, 8) . '...' . substr($key, -4) : 'registered'];
|
||||
}
|
||||
|
||||
// Build a bash command that reads a state file from the REMOTE host's State_Files/.
|
||||
// Reads the remote's varaverk.cfg to resolve their SCRIPTS_DIR (may differ from ours
|
||||
// when the remote is in appdata mode). Falls back to the internal plugin path.
|
||||
function vv_remote_state_cmd(string $filename): string {
|
||||
$fn = basename($filename);
|
||||
return 'sd=$(grep -m1 SCRIPTS_DIR= /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null'
|
||||
. ' | cut -d\'"\' -f2); cat "${sd:-/boot/config/plugins/varaverk}/State_Files/' . $fn . '" 2>/dev/null';
|
||||
}
|
||||
|
||||
// Local LAN IP via routing table — static-cached per request.
|
||||
// Previously duplicated in include/docker_folders.php and inline in include/docker.php.
|
||||
function vv_local_ip(): string {
|
||||
|
||||
@@ -24,6 +24,8 @@ function vv_fb_parse_state(string $text): array {
|
||||
'tier2_started' => false,
|
||||
'tier3_started' => false,
|
||||
'tier4_started' => false,
|
||||
'partnership_suspended' => false,
|
||||
'partner_lost_at' => 0,
|
||||
];
|
||||
foreach (explode("\n", $text) as $line) {
|
||||
$line = trim($line);
|
||||
@@ -37,18 +39,20 @@ function vv_fb_parse_state(string $text): array {
|
||||
case 'tier2_started': $out['tier2_started'] = $v === 'true'; break;
|
||||
case 'tier3_started': $out['tier3_started'] = $v === 'true'; break;
|
||||
case 'tier4_started': $out['tier4_started'] = $v === 'true'; break;
|
||||
case 'partnership_suspended': $out['partnership_suspended'] = $v === 'true'; break;
|
||||
case 'partner_lost_at': $out['partner_lost_at'] = (int)$v; break;
|
||||
}
|
||||
}
|
||||
return $out;
|
||||
}
|
||||
|
||||
function vv_fb_local_state(): array {
|
||||
$path = '/boot/config/fallback_state.db';
|
||||
$path = STATE_DIR . '/fallback_state.db';
|
||||
return vv_fb_parse_state(file_exists($path) ? file_get_contents($path) : '');
|
||||
}
|
||||
|
||||
function vv_fb_remote_state(string $ip, string $sshKey): array {
|
||||
$out = vv_pt_ssh($ip, $sshKey, 'cat /boot/config/fallback_state.db 2>/dev/null');
|
||||
$out = vv_pt_ssh($ip, $sshKey, vv_remote_state_cmd('fallback_state.db'));
|
||||
return vv_fb_parse_state($out);
|
||||
}
|
||||
|
||||
@@ -94,8 +98,13 @@ function vv_fb_all(): array {
|
||||
$currentHost = vv_detect_host();
|
||||
$hosts = vv_fb_known_hosts();
|
||||
$tsPeers = vv_pt_ts_peers();
|
||||
$handbackReq = (int)(vv_fb_scalar(vv_read_conf_raw('master.conf'), 'FALLBACK_HANDBACK_STRIKES') ?: 3);
|
||||
$fbEnabled = vv_fb_scalar(vv_read_conf_raw('master.conf'), 'FALLBACK_ENABLED') === 'true';
|
||||
$masterRaw = vv_read_conf_raw('master.conf');
|
||||
$handbackReq = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_HANDBACK_STRIKES') ?: 3);
|
||||
$fbEnabled = vv_fb_scalar($masterRaw, 'FALLBACK_ENABLED') === 'true';
|
||||
$ptEnabled = vv_fb_scalar($masterRaw, 'PARTNERSHIP_ENABLED') === 'true';
|
||||
$rsyncEnabled = vv_fb_scalar($masterRaw, 'FALLBACK_RSYNC_ENABLED') !== 'false';
|
||||
$checkInterval = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_CHECK_INTERVAL') ?: 30);
|
||||
$suspendAfter = (int)(vv_fb_scalar($masterRaw, 'FALLBACK_PARTNERSHIP_SUSPEND_AFTER') ?: 120);
|
||||
|
||||
// Read all host conf raws upfront
|
||||
$raws = [];
|
||||
@@ -164,7 +173,11 @@ function vv_fb_all(): array {
|
||||
return [
|
||||
'ts' => time(),
|
||||
'fb_enabled' => $fbEnabled,
|
||||
'partnership_enabled' => $ptEnabled,
|
||||
'fb_rsync_enabled' => $rsyncEnabled,
|
||||
'handback_req' => $handbackReq,
|
||||
'check_interval' => $checkInterval,
|
||||
'suspend_after' => $suspendAfter,
|
||||
'nodes' => $nodes,
|
||||
];
|
||||
}
|
||||
|
||||
@@ -58,7 +58,7 @@ function vv_fallback_state(): array {
|
||||
$reqStrikes = (int)($vars['FALLBACK_HANDBACK_STRIKES'] ?? 3);
|
||||
$suspendAfter = (int)($vars['FALLBACK_PARTNERSHIP_SUSPEND_AFTER'] ?? 120);
|
||||
|
||||
$stateFile = '/tmp/fallback_state.db';
|
||||
$stateFile = STATE_DIR . '/fallback_state.db';
|
||||
if (!file_exists($stateFile)) {
|
||||
return ['state' => 'UNKNOWN', 'enabled' => $enabled, 'check_interval' => $interval,
|
||||
'handback_strikes' => 0, 'handback_strikes_required' => $reqStrikes,
|
||||
@@ -164,7 +164,7 @@ function vv_watchdog_summary(): array {
|
||||
usort($restarts, fn($a, $b) => $b['ts'] - $a['ts']);
|
||||
|
||||
// Reboots (12 h)
|
||||
$rebootRaw = @file_get_contents('/boot/config/system_watchdog_reboots.db') ?: '';
|
||||
$rebootRaw = @file_get_contents(STATE_DIR . '/system_watchdog_reboots.db') ?: '';
|
||||
$rbootCutoff = time() - 43200;
|
||||
$reboots = 0;
|
||||
foreach (explode("\n", trim($rebootRaw)) as $line) {
|
||||
|
||||
@@ -10,7 +10,7 @@ require_once __DIR__ . '/common.php'; // vv_system_info(), vv_docker_containers(
|
||||
function vv_pt_config(): array {
|
||||
$v = vv_conf_vars();
|
||||
$offlineDays = null;
|
||||
$odFile = '/boot/config/partnership_offline_days.db';
|
||||
$odFile = STATE_DIR . '/partnership_offline_days.db';
|
||||
if (file_exists($odFile)) {
|
||||
$raw = trim(@file_get_contents($odFile) ?: '');
|
||||
if (is_numeric($raw)) $offlineDays = (int)$raw;
|
||||
@@ -209,12 +209,11 @@ function vv_pt_nodes(): array {
|
||||
|
||||
// Fallback state
|
||||
$fbState = 'UNKNOWN';
|
||||
$fbPath = '/boot/config/fallback_state.db';
|
||||
if ($isMe) {
|
||||
$fb = vv_pt_read_db($fbPath);
|
||||
$fb = vv_pt_read_db(STATE_DIR . '/fallback_state.db');
|
||||
$fbState = $fb['state'] ?? 'UNKNOWN';
|
||||
} elseif ($ts['online'] && $ts['ip'] && $mySshKey) {
|
||||
$out = vv_pt_ssh($ts['ip'], $mySshKey, 'cat /boot/config/fallback_state.db 2>/dev/null');
|
||||
$out = vv_pt_ssh($ts['ip'], $mySshKey, vv_remote_state_cmd('fallback_state.db'));
|
||||
if ($out) {
|
||||
$fb = [];
|
||||
foreach (explode("\n", $out) as $line) {
|
||||
@@ -226,7 +225,7 @@ function vv_pt_nodes(): array {
|
||||
}
|
||||
|
||||
// Partnership DB — local only (each server writes its own)
|
||||
$dbPath = "/boot/config/partnership_{$hostname}.db";
|
||||
$dbPath = STATE_DIR . "/partnership_{$hostname}.db";
|
||||
$ptDb = vv_pt_read_db($dbPath);
|
||||
|
||||
// System info
|
||||
|
||||
@@ -127,14 +127,14 @@ function vv_wd_parse_network_state(string $raw): array {
|
||||
// ── Local state files ─────────────────────────────────────────────────────────
|
||||
|
||||
function vv_wd_local_states(string $restartLogPath): array {
|
||||
$rwRaw = @file_get_contents('/tmp/resource_watchdog_state.db') ?: '';
|
||||
$dockRaw = @file_get_contents('/tmp/container_watchdog_state.db') ?: '';
|
||||
$skipRaw = @file_get_contents('/boot/config/system_watchdog_failed.db') ?: '';
|
||||
$sysRaw = @file_get_contents('/tmp/system_watchdog_state.db') ?: '';
|
||||
$rebootRaw = @file_get_contents('/boot/config/system_watchdog_reboots.db')?: '';
|
||||
$rwRaw = @file_get_contents(STATE_DIR . '/resource_watchdog_state.db') ?: '';
|
||||
$dockRaw = @file_get_contents(STATE_DIR . '/container_watchdog_state.db') ?: '';
|
||||
$skipRaw = @file_get_contents(STATE_DIR . '/docker_watchdog_failed.db') ?: '';
|
||||
$sysRaw = @file_get_contents(STATE_DIR . '/system_watchdog_state.db') ?: '';
|
||||
$rebootRaw = @file_get_contents(STATE_DIR . '/system_watchdog_reboots.db') ?: '';
|
||||
$restartRaw= @file_get_contents($restartLogPath) ?: '';
|
||||
$storRaw = @file_get_contents('/tmp/storage_watchdog_state.db') ?: '';
|
||||
$netWdRaw = @file_get_contents('/tmp/network_watchdog_state.db') ?: '';
|
||||
$storRaw = @file_get_contents(STATE_DIR . '/storage_watchdog_state.db') ?: '';
|
||||
$netWdRaw = @file_get_contents(STATE_DIR . '/network_watchdog_state.db') ?: '';
|
||||
|
||||
$rw = vv_wd_parse_kv($rwRaw);
|
||||
$dock = vv_wd_parse_kv($dockRaw);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
<style>
|
||||
/* ── Existing status styles ── */
|
||||
.vv-fb-active { background:#1a1200;border:1px solid #5a3800;border-radius:6px;padding:12px 14px; }
|
||||
.vv-fb-active-h { display:flex;align-items:baseline;gap:10px;margin-bottom:8px; }
|
||||
.vv-fb-badge { font-size:11px;font-weight:bold;letter-spacing:.06em;padding:2px 7px;border-radius:3px;flex-shrink:0; }
|
||||
@@ -6,6 +7,7 @@
|
||||
.vv-fb-badge.norm { background:#1a2a1a;color:#4caf50; }
|
||||
.vv-fb-badge.dark { background:#2a1a2a;color:#9c27b0; }
|
||||
.vv-fb-badge.nonet{ background:#1a1a2a;color:#5c7cfa; }
|
||||
.vv-fb-badge.susp { background:#2a1a00;color:#888; }
|
||||
.vv-fb-meta { display:flex;gap:18px;flex-wrap:wrap;margin-bottom:10px; }
|
||||
.vv-fb-meta-item{ display:flex;flex-direction:column;gap:1px; }
|
||||
.vv-fb-meta-val { font-size:17px;font-weight:bold;color:#ffb74d; }
|
||||
@@ -31,17 +33,109 @@
|
||||
.vv-fb-state-dot { width:6px;height:6px;border-radius:50%;flex-shrink:0;margin-top:3px; }
|
||||
.vv-fb-sep { border:none;border-top:1px solid #222;margin:8px 0; }
|
||||
.vv-fb-disabled { grid-column:1/-1;color:#3a3a3a;font-size:12px;padding:20px 0;text-align:center; }
|
||||
|
||||
/* ── Controls + settings card ── */
|
||||
.vv-fb-card { background:#161616;border:1px solid #2a2a2a;border-radius:6px;padding:14px 16px;margin-bottom:14px; }
|
||||
.vv-fb-card-hdr { font-size:11px;font-weight:700;color:#666;text-transform:uppercase;letter-spacing:.07em;margin-bottom:12px; }
|
||||
.vv-fb-ctrl-row { display:flex;justify-content:space-between;align-items:center;gap:12px;padding:5px 0; }
|
||||
.vv-fb-ctrl-lbl { font-size:12px;color:#888; }
|
||||
.vv-fb-ctrl-sub { font-size:10px;color:#3a3a3a;margin-top:2px; }
|
||||
.vv-fb-tog { width:32px;height:18px;border-radius:9px;background:#222;border:1px solid #333;
|
||||
position:relative;transition:background .15s,border-color .15s;flex-shrink:0;cursor:pointer; }
|
||||
.vv-fb-tog.on { background:#1a3a1a;border-color:#2d5a2d; }
|
||||
.vv-fb-tog::after { content:'';position:absolute;top:2px;left:2px;width:12px;height:12px;
|
||||
border-radius:50%;background:#555;transition:left .15s,background .15s; }
|
||||
.vv-fb-tog.on::after { left:16px;background:#4caf50; }
|
||||
.vv-fb-set-row { display:flex;justify-content:space-between;align-items:center;padding:5px 0; }
|
||||
.vv-fb-set-lbl { font-size:11px;color:#555; }
|
||||
.vv-fb-set-inp { background:#0d0d0d;border:1px solid #2a2a2a;border-radius:3px;color:#888;
|
||||
font-size:12px;padding:4px 8px;outline:none;width:72px;font-family:monospace;
|
||||
text-align:right;box-sizing:border-box; }
|
||||
.vv-fb-set-inp:focus { border-color:#444; }
|
||||
.vv-fb-set-unit { font-size:10px;color:#3a3a3a;min-width:56px; }
|
||||
.vv-fb-save-btn { background:#1a1a1a;border:1px solid #333;color:#888;font-size:11px;
|
||||
padding:5px 14px;border-radius:3px;cursor:pointer; }
|
||||
.vv-fb-save-btn:hover { border-color:#555;color:#ccc; }
|
||||
.vv-fb-save-btn:disabled { opacity:.4;cursor:default; }
|
||||
</style>
|
||||
|
||||
<!-- Controls card -->
|
||||
<div class="vv-fb-card">
|
||||
<div class="vv-fb-card-hdr">Controls</div>
|
||||
|
||||
<div class="vv-fb-ctrl-row">
|
||||
<div>
|
||||
<div class="vv-fb-ctrl-lbl">Partnership</div>
|
||||
<div class="vv-fb-ctrl-sub">Master gate — disabling stops all cross-server operations</div>
|
||||
</div>
|
||||
<div class="vv-fb-tog" id="vv-fb-pt-tog" onclick="vvFbToggle(this,'PARTNERSHIP_ENABLED')"></div>
|
||||
</div>
|
||||
|
||||
<hr class="vv-fb-sep">
|
||||
|
||||
<div class="vv-fb-ctrl-row">
|
||||
<div>
|
||||
<div class="vv-fb-ctrl-lbl">Fallback</div>
|
||||
<div class="vv-fb-ctrl-sub">Mutual container failover between nodes</div>
|
||||
</div>
|
||||
<div class="vv-fb-tog" id="vv-fb-en-tog" onclick="vvFbToggle(this,'FALLBACK_ENABLED')"></div>
|
||||
</div>
|
||||
|
||||
<hr class="vv-fb-sep">
|
||||
|
||||
<div class="vv-fb-ctrl-row">
|
||||
<div>
|
||||
<div class="vv-fb-ctrl-lbl">Rsync on handback</div>
|
||||
<div class="vv-fb-ctrl-sub">Writeback rsync when the covered host recovers and containers return</div>
|
||||
</div>
|
||||
<div class="vv-fb-tog" id="vv-fb-rsync-tog" onclick="vvFbToggle(this,'FALLBACK_RSYNC_ENABLED')"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Status section -->
|
||||
<div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px;padding:0 2px;">
|
||||
<span style="font-size:13px;font-weight:bold;color:#888;text-transform:uppercase;letter-spacing:.06em;">FallBack</span>
|
||||
<span style="font-size:13px;font-weight:bold;color:#888;text-transform:uppercase;letter-spacing:.06em;">Status</span>
|
||||
<span style="font-size:11px;color:#3a3a3a;" id="vv-fb-ts"></span>
|
||||
</div>
|
||||
|
||||
<div id="vv-fb-grid" style="display:grid;grid-template-columns:repeat(8,1fr);gap:12px;">
|
||||
<div id="vv-fb-grid" style="display:grid;grid-template-columns:repeat(8,1fr);gap:12px;margin-bottom:14px;">
|
||||
<div style="grid-column:1/-1;color:#444;font-size:12px;padding:16px 0;text-align:center;">Loading…</div>
|
||||
</div>
|
||||
|
||||
<!-- Settings card -->
|
||||
<div class="vv-fb-card">
|
||||
<div class="vv-fb-card-hdr">Settings</div>
|
||||
|
||||
<div class="vv-fb-set-row">
|
||||
<span class="vv-fb-set-lbl">Check interval</span>
|
||||
<div style="display:flex;align-items:center;gap:8px;">
|
||||
<input class="vv-fb-set-inp" id="vv-fb-interval" type="number" min="5" max="300">
|
||||
<span class="vv-fb-set-unit">seconds</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="vv-fb-set-row">
|
||||
<span class="vv-fb-set-lbl">Handback strikes</span>
|
||||
<div style="display:flex;align-items:center;gap:8px;">
|
||||
<input class="vv-fb-set-inp" id="vv-fb-strikes" type="number" min="1" max="20">
|
||||
<span class="vv-fb-set-unit">consecutive</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="vv-fb-set-row">
|
||||
<span class="vv-fb-set-lbl">Partnership suspend after</span>
|
||||
<div style="display:flex;align-items:center;gap:8px;">
|
||||
<input class="vv-fb-set-inp" id="vv-fb-suspend" type="number" min="0" max="1440">
|
||||
<span class="vv-fb-set-unit">minutes</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style="display:flex;justify-content:flex-end;align-items:center;gap:10px;margin-top:10px;padding-top:8px;border-top:1px solid #1e1e1e;">
|
||||
<span id="vv-fb-set-fb" style="font-size:11px;"></span>
|
||||
<button class="vv-fb-save-btn" id="vv-fb-save-btn" onclick="vvFbSaveSettings()">Save</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
(function() {
|
||||
|
||||
@@ -78,6 +172,7 @@ function _stateBadge(st) {
|
||||
OFFLINE: ['dark', 'OFFLINE'],
|
||||
UNREACHABLE: ['dark', 'UNREACHABLE'],
|
||||
UNKNOWN: ['dark', 'UNKNOWN'],
|
||||
SUSPENDED: ['susp', 'SUSPENDED'],
|
||||
};
|
||||
const [cls, label] = map[st] || ['dark', st];
|
||||
return `<span class="vv-fb-badge ${cls}">${label}</span>`;
|
||||
@@ -87,7 +182,7 @@ function _stateDot(st) {
|
||||
const col = {
|
||||
NORMAL:'#4caf50', FALLBACK:'#ffb74d',
|
||||
NO_INTERNET:'#5c7cfa', DARK:'#9c27b0',
|
||||
OFFLINE:'#555', UNREACHABLE:'#555', UNKNOWN:'#333',
|
||||
OFFLINE:'#555', UNREACHABLE:'#555', UNKNOWN:'#333', SUSPENDED:'#444',
|
||||
}[st] || '#333';
|
||||
return `<span class="vv-fb-state-dot" style="background:${col}"></span>`;
|
||||
}
|
||||
@@ -102,7 +197,6 @@ function _activeCard(nodes, handbackReq) {
|
||||
const cov = covering.covers;
|
||||
const covered = cov ? cov.hostname : '?';
|
||||
|
||||
// All containers that should be running at current tier
|
||||
let expected = [...(cov?.tier1 || [])];
|
||||
if (tier >= 2) expected = expected.concat(cov?.tier2 || []);
|
||||
if (tier >= 3) expected = expected.concat(cov?.tier3 || []);
|
||||
@@ -169,9 +263,19 @@ function _tierSection(tiers, activeTier, delays) {
|
||||
}).join('');
|
||||
}
|
||||
|
||||
function _nodeCard(node) {
|
||||
function _ptStatus(st) {
|
||||
if (!st) return '';
|
||||
if (st.partnership_suspended) return _stateBadge('SUSPENDED');
|
||||
if (st.partner_lost_at && st.partner_lost_at > 0) {
|
||||
const minGone = Math.floor((Date.now() / 1000 - st.partner_lost_at) / 60);
|
||||
return `<span class="vv-fb-badge susp">GRACE ${minGone}m</span>`;
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function _nodeCard(node, suspendAfter) {
|
||||
const st = node.state || {};
|
||||
const state = st.state || 'UNKNOWN';
|
||||
const state = st.partnership_suspended ? 'SUSPENDED' : (st.state || 'UNKNOWN');
|
||||
const cov = node.covers;
|
||||
const active = _activeTier(state === 'FALLBACK' ? st : null);
|
||||
|
||||
@@ -183,6 +287,8 @@ function _nodeCard(node) {
|
||||
? _tierSection(cov, active, cov.delays)
|
||||
: '<div style="color:#3a3a3a;font-size:11px;">No coverage configured</div>';
|
||||
|
||||
const ptBadge = node.is_me ? _ptStatus(st) : '';
|
||||
|
||||
return `<div class="vv-card vv-fb-node">
|
||||
<div class="vv-fb-node-h">
|
||||
${_stateDot(state)}
|
||||
@@ -190,6 +296,7 @@ function _nodeCard(node) {
|
||||
<span class="vv-fb-node-nm">${node.hostname}</span>
|
||||
${covTarget}
|
||||
<span style="flex:1"></span>
|
||||
${ptBadge}
|
||||
${_stateBadge(state)}
|
||||
</div>
|
||||
<hr class="vv-fb-sep">
|
||||
@@ -197,30 +304,47 @@ function _nodeCard(node) {
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function _setToggles(data) {
|
||||
const pairs = [
|
||||
['vv-fb-pt-tog', !!data.partnership_enabled],
|
||||
['vv-fb-en-tog', !!data.fb_enabled],
|
||||
['vv-fb-rsync-tog', !!data.fb_rsync_enabled],
|
||||
];
|
||||
pairs.forEach(([id, on]) => {
|
||||
const el = document.getElementById(id);
|
||||
if (el) el.classList.toggle('on', on);
|
||||
});
|
||||
}
|
||||
|
||||
function _setInputs(data) {
|
||||
const fields = [
|
||||
['vv-fb-interval', data.check_interval ?? 30],
|
||||
['vv-fb-strikes', data.handback_req ?? 3],
|
||||
['vv-fb-suspend', data.suspend_after ?? 120],
|
||||
];
|
||||
fields.forEach(([id, val]) => {
|
||||
const el = document.getElementById(id);
|
||||
if (el && el !== document.activeElement) el.value = val;
|
||||
});
|
||||
}
|
||||
|
||||
function _render(data) {
|
||||
if (!data.fb_enabled) {
|
||||
document.getElementById('vv-fb-grid').innerHTML =
|
||||
'<div class="vv-fb-disabled">FALLBACK_ENABLED=false — fallback monitoring is disabled</div>';
|
||||
return;
|
||||
}
|
||||
_setToggles(data);
|
||||
_setInputs(data);
|
||||
|
||||
const grid = document.getElementById('vv-fb-grid');
|
||||
|
||||
if (!data.partnership_enabled) {
|
||||
grid.innerHTML = '<div class="vv-fb-disabled">PARTNERSHIP_ENABLED=false — all cross-server operations disabled</div>';
|
||||
} else if (!data.fb_enabled) {
|
||||
grid.innerHTML = '<div class="vv-fb-disabled">FALLBACK_ENABLED=false — fallback monitoring is disabled</div>';
|
||||
} else {
|
||||
const nodes = data.nodes || [];
|
||||
let html = '';
|
||||
|
||||
// Top: active fallback card (if any)
|
||||
html += _activeCard(nodes, data.handback_req || 3);
|
||||
|
||||
// Per-node cards
|
||||
for (const node of nodes) {
|
||||
html += _nodeCard(node);
|
||||
let html = _activeCard(nodes, data.handback_req || 3);
|
||||
for (const node of nodes) html += _nodeCard(node, data.suspend_after || 120);
|
||||
grid.innerHTML = html || '<div class="vv-fb-disabled">No nodes configured.</div>';
|
||||
}
|
||||
|
||||
if (!html) {
|
||||
html = '<div class="vv-fb-disabled">No nodes configured.</div>';
|
||||
}
|
||||
|
||||
document.getElementById('vv-fb-grid').innerHTML = html;
|
||||
|
||||
const ts = data.ts
|
||||
? new Date(data.ts * 1000).toLocaleString([], {
|
||||
month:'numeric', day:'numeric', year:'numeric',
|
||||
@@ -239,6 +363,55 @@ function vvFbLoad() {
|
||||
});
|
||||
}
|
||||
|
||||
window.vvFbToggle = function(track, key) {
|
||||
const on = !track.classList.contains('on');
|
||||
track.classList.toggle('on', on);
|
||||
const fd = new FormData();
|
||||
fd.append('id', 'fallback');
|
||||
fd.append('changes', JSON.stringify([{ file: 'master.conf', key, value: on ? 'true' : 'false', type: 'scalar' }]));
|
||||
fetch('/plugins/varaverk/api/confform.php', { method: 'POST', body: fd })
|
||||
.then(r => r.json())
|
||||
.then(d => { if (!d.ok) track.classList.toggle('on', !on); })
|
||||
.catch(() => track.classList.toggle('on', !on));
|
||||
};
|
||||
|
||||
window.vvFbSaveSettings = function() {
|
||||
const ivEl = document.getElementById('vv-fb-interval');
|
||||
const stEl = document.getElementById('vv-fb-strikes');
|
||||
const suEl = document.getElementById('vv-fb-suspend');
|
||||
const fb = document.getElementById('vv-fb-set-fb');
|
||||
const btn = document.getElementById('vv-fb-save-btn');
|
||||
|
||||
const interval = parseInt(ivEl.value, 10);
|
||||
const strikes = parseInt(stEl.value, 10);
|
||||
const suspend = parseInt(suEl.value, 10);
|
||||
|
||||
if ([interval, strikes, suspend].some(n => isNaN(n) || n < 0)) {
|
||||
fb.style.color = '#ef5350'; fb.textContent = 'Invalid values'; return;
|
||||
}
|
||||
|
||||
btn.disabled = true; btn.textContent = 'Saving…'; fb.textContent = '';
|
||||
const fd = new FormData();
|
||||
fd.append('id', 'fallback');
|
||||
fd.append('changes', JSON.stringify([
|
||||
{ file: 'master.conf', key: 'FALLBACK_CHECK_INTERVAL', value: String(interval), type: 'scalar' },
|
||||
{ file: 'master.conf', key: 'FALLBACK_HANDBACK_STRIKES', value: String(strikes), type: 'scalar' },
|
||||
{ file: 'master.conf', key: 'FALLBACK_PARTNERSHIP_SUSPEND_AFTER', value: String(suspend), type: 'scalar' },
|
||||
]));
|
||||
fetch('/plugins/varaverk/api/confform.php', { method: 'POST', body: fd })
|
||||
.then(r => r.json())
|
||||
.then(d => {
|
||||
btn.disabled = false; btn.textContent = 'Save';
|
||||
fb.style.color = d.ok ? '#4caf50' : '#ef5350';
|
||||
fb.textContent = d.ok ? 'Saved ✓' : (d.error || 'Failed');
|
||||
if (d.ok) setTimeout(() => { fb.textContent = ''; }, 3000);
|
||||
})
|
||||
.catch(() => {
|
||||
btn.disabled = false; btn.textContent = 'Save';
|
||||
fb.style.color = '#ef5350'; fb.textContent = 'Request failed';
|
||||
});
|
||||
};
|
||||
|
||||
vvFbLoad();
|
||||
setInterval(vvFbLoad, 30000);
|
||||
|
||||
|
||||
@@ -715,8 +715,8 @@ Change `--dry-run` to nothing and run during a maintenance window.
|
||||
> forced kill of fallback.sh:
|
||||
> ```bash
|
||||
> # Verify first — right containers on right server, DDNS correct, fallback.sh not running
|
||||
> cat /boot/config/fallback_state.db
|
||||
> echo "state=NORMAL" > /boot/config/fallback_state.db
|
||||
> cat "$STATE_DIR/fallback_state.db"
|
||||
> echo "state=NORMAL" > "$STATE_DIR/fallback_state.db"
|
||||
> ```
|
||||
> Resets state only — does NOT start or stop any containers.
|
||||
|
||||
|
||||
@@ -621,7 +621,7 @@ SLEEP=60 # seconds between retries
|
||||
ROOTFS_WARN_PCT=75 # abort if remote rootfs above this %
|
||||
|
||||
# Shared with Monitors/
|
||||
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
|
||||
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
|
||||
BANDWIDTH_LOG_RETENTION=90
|
||||
BANDWIDTH_WARN_GB=50
|
||||
|
||||
|
||||
@@ -4,23 +4,24 @@ Configuration reference, operational procedures, and troubleshooting for
|
||||
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
|
||||
adding or reordering scripts at array start.
|
||||
|
||||
> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
|
||||
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
|
||||
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
|
||||
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
|
||||
> `mover_stop.sh`, `user_scripts_stop.sh`) and their configuration live in
|
||||
> `Plugin/unraid/System_Essentials/`. Watchdog configuration (`stability_watchdog.sh`,
|
||||
> `resource_watchdog.sh`, `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives
|
||||
> in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
|
||||
- [webgui_watchdog.sh](#webgui_watchdogsh)
|
||||
- [ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order](#array_start_scripts--array_stop_scripts-order)
|
||||
- [conf_sync.sh](#conf_syncsh)
|
||||
- [conf_cache_save.sh](#conf_cache_savesh)
|
||||
- [conf_cache_restore.sh](#conf_cache_restoresh)
|
||||
- [inotify_tuning.sh](#inotify_tuningsh)
|
||||
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
|
||||
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
|
||||
- [clear_logs.sh](#clear_logssh)
|
||||
- [mover_stop.sh](#mover_stopsh)
|
||||
- [rsync_stop.sh](#rsync_stopsh)
|
||||
- [user_scripts_stop.sh](#user_scripts_stopsh)
|
||||
- [server_reboot.sh](#server_rebootsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
@@ -32,109 +33,155 @@ adding or reordering scripts at array start.
|
||||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||||
lines only appear when `--log` is passed.
|
||||
|
||||
**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
|
||||
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
|
||||
Per-check detail suppressed.
|
||||
|
||||
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
|
||||
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
|
||||
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
|
||||
**One-shot scripts** (`conf_sync.sh`, `conf_cache_save.sh`, `conf_cache_restore.sh`,
|
||||
`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`, `rsync_stop.sh`,
|
||||
`server_reboot.sh`): without `--log`, section headers, per-step results, and the
|
||||
final summary are visible. Per-item detail suppressed.
|
||||
|
||||
---
|
||||
|
||||
## ARRAY_START_SCRIPTS Order
|
||||
## ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order
|
||||
|
||||
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
|
||||
> scripts.** Getting it wrong causes subtle failures that don't show up
|
||||
> immediately.
|
||||
> **Order matters.** Scripts that set up conditions other scripts depend on must run
|
||||
> first. Getting the order wrong causes subtle failures.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# master.conf (current order — from ARRAY_START_SCRIPTS)
|
||||
ARRAY_START_SCRIPTS=(
|
||||
"inotify_tuning.sh" # 1 — FIRST: kernel limits must be set before
|
||||
# any container starts. Containers inherit
|
||||
# inotify limits at launch, not dynamically.
|
||||
"docker_syslog_filter.sh" # 2 — SECOND: before any veth interfaces are
|
||||
# created. If a container starts first, its
|
||||
# veth creation is already in syslog.
|
||||
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
|
||||
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
|
||||
...
|
||||
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # plugin API key — ephemeral registry
|
||||
"System_Essentials/conf_sync.sh" # 1st cross-server op — pull partner confs into RAM cache
|
||||
"System_Essentials/conf_cache_restore.sh" # fill gaps if partner was down at boot
|
||||
"Transcodes/ramdisk_setup.sh" # ramdisk before Emby starts
|
||||
"System_Essentials/docker_syslog_filter.sh" # before veth interfaces are created
|
||||
"Plugin/unraid/System_Essentials/php_fpm_max_children.sh" # WebGUI tuning
|
||||
"System_Essentials/inotify_tuning.sh" # before docker_network_connect and continuous scripts
|
||||
"Docker_Essentials/docker_network_connect.sh" # ensure networks + container connections
|
||||
"Tools/claude_startup.sh" # persist Claude data to appdata; re-symlink on boot
|
||||
"Media/start_webhook_listener.sh" # arr upgrade webhook — continuous
|
||||
"Fallback/fallback.sh" # mutual failover — continuous
|
||||
)
|
||||
|
||||
ARRAY_STOP_SCRIPTS=(
|
||||
"System_Essentials/conf_cache_save.sh" # FIRST: snapshot RAM cache while fresh
|
||||
"Plugin/unraid/System_Essentials/user_scripts_stop.sh"
|
||||
"Fallback/fallback.sh --stop"
|
||||
"System_Essentials/rsync_stop.sh --rsync-only"
|
||||
"Plugin/unraid/System_Essentials/mover_stop.sh"
|
||||
"Docker_Essentials/docker_container_stop.sh"
|
||||
)
|
||||
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
|
||||
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
|
||||
```
|
||||
|
||||
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
|
||||
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
|
||||
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
|
||||
Why conf_sync before conf_cache_restore: the sync runs first to get the freshest
|
||||
possible partner conf. The restore only fills in for confs that the sync couldn't
|
||||
fetch — it is a fallback, not the primary path.
|
||||
|
||||
Why docker_syslog_filter SECOND: The filter must be in place before any container
|
||||
starts creating veth interfaces. The first container start after array start
|
||||
generates veth messages — these will appear in syslog if the filter isn't active.
|
||||
Why docker_syslog_filter before inotify: the filter must be in place before any
|
||||
container creates veth interfaces. inotify must be set before docker_network_connect
|
||||
and the continuous scripts (start_webhook_listener, fallback), which are the first
|
||||
steps that actually touch containers.
|
||||
|
||||
Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at
|
||||
`/tmp/.cache/vv/d/` is wiped on reboot. Saving it must happen before anything
|
||||
else shuts down or changes state.
|
||||
|
||||
---
|
||||
|
||||
## webgui_watchdog.sh
|
||||
## conf_sync.sh
|
||||
|
||||
### Escalation Logic
|
||||
### Modes
|
||||
|
||||
```
|
||||
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
|
||||
|
||||
Not responding:
|
||||
1. platform_restart_service nginx
|
||||
wait WEBGUI_NGINX_WAIT (15s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
2. platform_restart_service php-fpm
|
||||
wait WEBGUI_PHP_WAIT (10s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
3. platform_restart_service emhttp
|
||||
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
All three failed → notify warning, exit 1
|
||||
conf_sync.sh Full sync: pull from all partners + push to all partners
|
||||
conf_sync.sh --push-only Push own conf to all partners (fast — for conf-save hook)
|
||||
conf_sync.sh --pull-only Pull partner confs only (intermediate orchestrator)
|
||||
conf_sync.sh --dry-run Show what would happen, no changes
|
||||
conf_sync.sh --log Verbose output
|
||||
```
|
||||
|
||||
### Configuration
|
||||
### What It Syncs
|
||||
|
||||
- **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk
|
||||
via SCP → writes to local `/tmp/.cache/vv/d/${partner_id}.conf`
|
||||
- **Push**: sends own `Configurations/${my_id}.conf` to partner's
|
||||
`/tmp/.cache/vv/d/${my_id}.conf` via SCP
|
||||
- **Own conf in local cache**: copies own conf to `/tmp/.cache/vv/d/${my_id}.conf`
|
||||
on full sync (so the cache has a complete snapshot of all confs)
|
||||
|
||||
Only partner confs are sourced from cache — `load_config.sh` always reads own conf
|
||||
from disk to avoid sourcing a stale cached copy.
|
||||
|
||||
### Remote SCRIPTS_DIR Resolution
|
||||
|
||||
The pull path reads the partner's `/boot/config/plugins/varaverk/varaverk.cfg` to
|
||||
find their actual `SCRIPTS_DIR` before building the SCP path. This handles the case
|
||||
where the partner is in appdata storage mode and their conf is at
|
||||
`/mnt/user/appdata/Varaverk/Configurations/` rather than the internal path.
|
||||
|
||||
### PARTNERSHIP_ENABLED Gate
|
||||
|
||||
conf_sync.sh calls `require_partnership` — if `PARTNERSHIP_ENABLED=false`, it exits
|
||||
silently with 0. The conf cache will be empty for partner confs while partnership
|
||||
is disabled.
|
||||
|
||||
### If Conf Pull Fails at Boot
|
||||
|
||||
If the partner is unreachable, `conf_sync.sh` logs a warning and exits 1.
|
||||
`conf_cache_restore.sh` then runs (next in ARRAY_START_SCRIPTS) and loads the
|
||||
persistent backup from `$PERSISTENT_CONF_CACHE` if available.
|
||||
|
||||
A notification fires if any partner fails — check partner reachability via Tailscale.
|
||||
|
||||
---
|
||||
|
||||
## conf_cache_save.sh
|
||||
|
||||
### What It Does
|
||||
|
||||
At array stop, copies all partner confs from `/tmp/.cache/vv/d/` to
|
||||
`$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives
|
||||
the reboot and is used by `conf_cache_restore.sh` at next array start if the sync
|
||||
can't reach the partner.
|
||||
|
||||
`conf_cache_watchdog.sh` (in `Watchdogs/System/`) refreshes this backup every
|
||||
15 minutes while the partner is offline — keeping it current even during extended
|
||||
outages.
|
||||
|
||||
### PARTNERSHIP_ENABLED Gate
|
||||
|
||||
Exits silently when `PARTNERSHIP_ENABLED=false`. No backup is written.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
WEBGUI_URL="http://localhost" # URL to check
|
||||
WEBGUI_TIMEOUT=5 # curl timeout in seconds
|
||||
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
|
||||
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
|
||||
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
|
||||
conf_cache_save.sh # save partner confs from RAM cache (called at array stop)
|
||||
conf_cache_save.sh --dry-run # show what would be saved
|
||||
conf_cache_save.sh --log # verbose output
|
||||
```
|
||||
|
||||
### WebGUI Frozen — Manual Recovery
|
||||
---
|
||||
|
||||
## conf_cache_restore.sh
|
||||
|
||||
### What It Does
|
||||
|
||||
At array start (after `conf_sync.sh`), checks which partner confs are missing from
|
||||
`/tmp/.cache/vv/d/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
|
||||
if a backup exists there.
|
||||
|
||||
Always removes the persistent backup when done — whether used or not. On a normal
|
||||
reboot where the partner was reachable, the sync already populated the cache and the
|
||||
backup is cleaned up without being used.
|
||||
|
||||
### PARTNERSHIP_ENABLED Gate
|
||||
|
||||
Exits silently when `PARTNERSHIP_ENABLED=false`.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Check which services are running:
|
||||
webgui_watchdog.sh --status
|
||||
|
||||
# Try manual restart sequence (mirrors what the script calls):
|
||||
# Source the ecosystem first to get platform functions:
|
||||
source /boot/config/plugins/varaverk/load_config.sh
|
||||
platform_restart_service nginx
|
||||
# wait 15s, then:
|
||||
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
|
||||
|
||||
# If nginx did not fix it, php-fpm:
|
||||
platform_restart_service php-fpm
|
||||
|
||||
# If still down, emhttp:
|
||||
platform_restart_service emhttp
|
||||
|
||||
# Raw equivalents (no source needed — paste directly into terminal):
|
||||
# /etc/rc.d/rc.nginx restart
|
||||
# /etc/rc.d/rc.php-fpm restart
|
||||
# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
|
||||
|
||||
# If all three failed:
|
||||
server_reboot.sh --status # check for active sessions first
|
||||
conf_cache_restore.sh # restore missing confs from backup (called at array start)
|
||||
conf_cache_restore.sh --dry-run # show what would be restored
|
||||
conf_cache_restore.sh --log # verbose output
|
||||
```
|
||||
|
||||
---
|
||||
@@ -176,48 +223,6 @@ inotify_tuning.sh --log
|
||||
|
||||
---
|
||||
|
||||
## php_fpm_max_children.sh
|
||||
|
||||
### What It Sets
|
||||
|
||||
```bash
|
||||
PHP_MAX_CHILDREN=250 # target pm.max_children (default: 4-8 on unRAID)
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||||
```
|
||||
|
||||
250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
|
||||
The default of 4–8 saturates immediately under load on a busy server.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
php_fpm_max_children.sh --status
|
||||
# Shows current value vs target, PHP-FPM worker count
|
||||
|
||||
# Manual verify:
|
||||
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
|
||||
# Expected: pm.max_children = 250
|
||||
```
|
||||
|
||||
### If WebGUI Is Slow Despite the Setting
|
||||
|
||||
```bash
|
||||
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
|
||||
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
|
||||
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
|
||||
# pm.max_children value may still be too low, or there's a PHP worker leak.
|
||||
|
||||
# Check running worker count:
|
||||
pgrep -fc php-fpm
|
||||
# Compare to pm.max_children — if equal, workers are saturated
|
||||
|
||||
# Increase if needed:
|
||||
# master.conf: PHP_MAX_CHILDREN=350
|
||||
# Then: php_fpm_max_children.sh --log (will update and restart php-fpm)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## docker_syslog_filter.sh
|
||||
|
||||
### What It Creates
|
||||
@@ -230,8 +235,7 @@ pgrep -fc php-fpm
|
||||
```
|
||||
|
||||
This drops any syslog message containing "veth" or "docker0" before it reaches
|
||||
any output target, including the log file. The drop rule is applied at rsyslog
|
||||
level — not at the log viewer level.
|
||||
any output target, including the log file.
|
||||
|
||||
### Verify
|
||||
|
||||
@@ -298,39 +302,6 @@ du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
|
||||
|
||||
---
|
||||
|
||||
## mover_stop.sh
|
||||
|
||||
### Stop Sequence
|
||||
|
||||
```
|
||||
1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
|
||||
2. Wall message to all logged-in terminal users
|
||||
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
|
||||
4. SIGTERM — mover finishes its current file operation, then stops
|
||||
5. Wait 5 seconds → verify stopped
|
||||
6. SIGKILL if still running — forced stop, partial files possible
|
||||
7. Final verify — error if still running after SIGKILL
|
||||
```
|
||||
|
||||
SIGTERM first because the mover can finish the file it is currently moving,
|
||||
leaving no partial copies split across cache and array. SIGKILL is a last resort.
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
MOVER_STOP_TIMEOUT=30 # seconds between wall warning and SIGTERM
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
mover_stop.sh # check and stop if running
|
||||
mover_stop.sh --status # show current mover state and PID
|
||||
mover_stop.sh --dry-run # show what would happen without stopping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## rsync_stop.sh
|
||||
|
||||
### Auto-Detection Logic
|
||||
@@ -355,8 +326,8 @@ After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTA
|
||||
for any that were stopped by the interrupted rsync session and restarts them.
|
||||
Remote containers are left for docker_watchdog.sh to recover.
|
||||
|
||||
Skip container recovery with `--rsync-only` — used when called by other scripts
|
||||
that handle recovery themselves.
|
||||
Skip container recovery with `--rsync-only` — used when called by array_stopping.sh
|
||||
(which handles container stop separately via docker_container_stop.sh).
|
||||
|
||||
### Usage
|
||||
|
||||
@@ -371,34 +342,6 @@ rsync_stop.sh --full-stop --dry-run # preview full stop
|
||||
|
||||
---
|
||||
|
||||
## user_scripts_stop.sh
|
||||
|
||||
### Process Identification
|
||||
|
||||
Scans `/proc/*/cmdline` for any process whose command line contains
|
||||
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
|
||||
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
|
||||
what the script is named or how it was launched.
|
||||
|
||||
Script names are extracted from the path for display: you see which scripts are
|
||||
being stopped, not just PIDs.
|
||||
|
||||
### Self-Exclusion
|
||||
|
||||
If this script is run via the User Scripts plugin, it would find its own PID in
|
||||
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
|
||||
from the kill list.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
user_scripts_stop.sh # stop all User Script processes
|
||||
user_scripts_stop.sh --status # show running scripts with names and elapsed time
|
||||
user_scripts_stop.sh --dry-run # show what would be stopped
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## server_reboot.sh
|
||||
|
||||
### Full Shutdown Sequence
|
||||
@@ -422,6 +365,7 @@ user_scripts_stop.sh --dry-run # show what would be stopped
|
||||
6. /etc/rc.d/rc.libvirt stop (VM Manager)
|
||||
|
||||
7. Orchestrators/array_stopping.sh — safe ordered array stop:
|
||||
- conf_cache_save.sh snapshot partner conf RAM cache
|
||||
- user_scripts_stop.sh stop background User Scripts
|
||||
- fallback.sh --stop graceful fallback teardown
|
||||
- rsync_stop.sh --rsync-only kill active rsync transfers
|
||||
@@ -446,7 +390,6 @@ For a clean reboot when services are active:
|
||||
|
||||
```bash
|
||||
rsync_stop.sh # stop any active rsync (smart mode)
|
||||
mover_stop.sh # stop mover gracefully
|
||||
server_reboot.sh --status # check what's still running
|
||||
server_reboot.sh --reason="planned maintenance"
|
||||
```
|
||||
@@ -464,27 +407,17 @@ server_reboot.sh --reason="disk work" # include reason in notification
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
|
||||
> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
|
||||
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
|
||||
WEBGUI_URL="http://localhost"
|
||||
WEBGUI_TIMEOUT=5
|
||||
WEBGUI_NGINX_WAIT=15
|
||||
WEBGUI_PHP_WAIT=10
|
||||
WEBGUI_EMHTTP_WAIT=30
|
||||
# ── Conf Sync ──────────────────────────────────────────────────────────────────
|
||||
CONF_SYNC_ENABLED=true # toggle: false disables conf_sync.sh entirely
|
||||
|
||||
# ── inotify Tuning ─────────────────────────────────────────────────────────────
|
||||
INOTIFY_MAX_INSTANCES=1024
|
||||
INOTIFY_MAX_WATCHES=1048576
|
||||
INOTIFY_MAX_QUEUED_EVENTS=32768
|
||||
|
||||
# ── PHP-FPM ────────────────────────────────────────────────────────────────────
|
||||
PHP_MAX_CHILDREN=250
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||||
INOTIFY_WARN_PCT=80 # warn (in reports) if instances exceed this % of limit
|
||||
|
||||
# ── Syslog Filter ──────────────────────────────────────────────────────────────
|
||||
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
|
||||
@@ -494,20 +427,48 @@ LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
|
||||
LOG_MIN_SIZE_MB=10
|
||||
LOG_DOCKER_MAX_MB=100
|
||||
|
||||
# ── Mover Stop ─────────────────────────────────────────────────────────────────
|
||||
MOVER_STOP_TIMEOUT=30
|
||||
|
||||
# ── Server Reboot ──────────────────────────────────────────────────────────────
|
||||
REBOOT_SLEEP=30
|
||||
REBOOT_VM_WAIT=30
|
||||
|
||||
# ── State Paths (auto-adapt to storage mode) ───────────────────────────────────
|
||||
# STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
|
||||
# They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
|
||||
# (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
|
||||
STATE_DIR="${SCRIPTS_DIR}/State_Files"
|
||||
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
|
||||
> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
|
||||
### Partner Conf Not In RAM Cache After Boot
|
||||
|
||||
```bash
|
||||
# Check what's in the RAM cache:
|
||||
ls -la /tmp/.cache/vv/d/
|
||||
|
||||
# Check what's in the persistent backup:
|
||||
ls -la "$PERSISTENT_CONF_CACHE/" # set SCRIPTS_DIR first or use full path
|
||||
|
||||
# Re-run conf sync manually:
|
||||
/boot/config/plugins/varaverk/System_Essentials/conf_sync.sh --log
|
||||
|
||||
# If partner is reachable but pull failed, check SSH key:
|
||||
ssh -i /path/to/ssh_key root@<partner-tailscale-ip> "echo ok"
|
||||
```
|
||||
|
||||
### Persistent Backup Is Stale or Empty
|
||||
|
||||
```bash
|
||||
# conf_cache_watchdog.sh refreshes the backup while partner is offline.
|
||||
# Check watchdog state:
|
||||
cat "$STATE_DIR/conf_cache_watchdog_state.db"
|
||||
|
||||
# Force a manual backup from current RAM cache:
|
||||
/boot/config/plugins/varaverk/System_Essentials/conf_cache_save.sh --log
|
||||
```
|
||||
|
||||
### rsync_stop Killed the Wrong Thing
|
||||
|
||||
@@ -521,36 +482,29 @@ rsync_stop.sh --dry-run # shows smart mode decision
|
||||
rsync_stop.sh --full-stop --dry-run # shows full-stop decision
|
||||
```
|
||||
|
||||
### WebGUI Recovery After All Three Steps Failed
|
||||
### inotify Exhaustion After Boot
|
||||
|
||||
```bash
|
||||
# Check if processes are running:
|
||||
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
|
||||
pgrep emhttpd && echo "emhttp: yes" || echo "emhttp: no"
|
||||
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
|
||||
# Verify limits are applied:
|
||||
sysctl fs.inotify.max_user_watches # expect 1048576
|
||||
sysctl fs.inotify.max_user_instances # expect 1024
|
||||
|
||||
# Check recent nginx errors:
|
||||
cat /var/log/nginx/error.log | tail -20
|
||||
# If not set — run manually:
|
||||
inotify_tuning.sh --log
|
||||
|
||||
# Check emhttp log:
|
||||
tail -20 /var/log/syslog | grep emhttp
|
||||
|
||||
# Last resort — reboot:
|
||||
server_reboot.sh --reason="WebGUI unrecoverable"
|
||||
# Check current usage:
|
||||
inotify_tuning.sh --status
|
||||
```
|
||||
|
||||
### PHP-FPM Config Not Found After unRAID Update
|
||||
|
||||
unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
|
||||
errors with "config file not found":
|
||||
### Syslog Still Noisy After Array Start
|
||||
|
||||
```bash
|
||||
# Find the new config path:
|
||||
find /etc -name "www.conf" 2>/dev/null
|
||||
# Check filter is in place:
|
||||
docker_syslog_filter.sh --status
|
||||
|
||||
# Update PHP_CONF in master.conf:
|
||||
PHP_CONF="/etc/php84/php-fpm.d/www.conf" # example for php84
|
||||
# Re-apply if needed:
|
||||
docker_syslog_filter.sh --log
|
||||
|
||||
# Verify with:
|
||||
php_fpm_max_children.sh --status
|
||||
# Restart rsyslog to pick up the filter:
|
||||
/etc/rc.d/rc.rsyslogd restart
|
||||
```
|
||||
|
||||
@@ -1,35 +1,18 @@
|
||||
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
|
||||
|
||||
**System-level scripts that act on the server itself — not containers,
|
||||
not media, not monitoring.** Keeping the server stable under load, recovering a
|
||||
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
|
||||
shutdowns with proper warning sequences.
|
||||
not media, not monitoring.** Kernel limits, log hygiene, conf synchronisation
|
||||
between servers, and graceful shutdowns with proper warning sequences.
|
||||
|
||||
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
|
||||
> `mover_stop.sh`, `user_scripts_stop.sh`) live in `Plugin/unraid/System_Essentials/`
|
||||
> because they call Unraid-specific service commands and paths. This folder
|
||||
> contains scripts that would run unchanged on any Linux host.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Server Getting Into Unstable States With No Recovery Path**
|
||||
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
|
||||
watchdog tries to restart the container — but Docker itself is barely responding.
|
||||
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
|
||||
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
|
||||
write log files. The server is functionally dead but still technically running.
|
||||
|
||||
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
|
||||
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
|
||||
threshold breaches. Last line of defense before a hard crash.
|
||||
|
||||
**WebGUI Freezing and Nobody Noticing**
|
||||
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
|
||||
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
|
||||
load, settings don't save, containers can't be started or stopped via the UI. No
|
||||
container-level alert fires because this isn't a container problem — it's a web
|
||||
server problem. By the time someone notices it may have been broken for hours.
|
||||
|
||||
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
|
||||
php-fpm → emhttp. Lightest fix first. Silent when healthy.
|
||||
|
||||
**50+ Containers Starting and Filling Syslog With Veth Noise**
|
||||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
|
||||
network interface for each one. Each interface generates multiple syslog entries.
|
||||
@@ -40,68 +23,79 @@ In the first minute after array start, syslog is buried under 200–400 lines of
|
||||
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
|
||||
starts. Applied at array start. Idempotent — silent when already correct.
|
||||
|
||||
**WebGUI Queuing Requests Under Load Without Explanation**
|
||||
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
|
||||
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
|
||||
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
|
||||
those workers saturate immediately. New requests queue behind active ones.
|
||||
|
||||
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||||
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
|
||||
|
||||
**inotify Exhaustion Producing Unexplained Failures**
|
||||
When inotify limits are exhausted, containers silently stop receiving filesystem
|
||||
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
|
||||
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
|
||||
and all containers share the same pool.
|
||||
|
||||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
|
||||
run FIRST in ARRAY_START_SCRIPTS before any containers start.
|
||||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start, before
|
||||
any container-connecting or continuous scripts start.
|
||||
|
||||
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
|
||||
The mover is running — moving a large batch of files from cache to array. A reboot
|
||||
is triggered. The mover stops mid-file. The file exists partially on both cache and
|
||||
array simultaneously. unRAID's deduplication layer is confused.
|
||||
**Partner Conf Lost Across Reboots When Partner is Down**
|
||||
Scripts like `fallback.sh` need the partner's conf vars (credentials, container
|
||||
names, tier delays) to operate. The partner conf lives in a RAM cache at
|
||||
`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls
|
||||
a fresh copy from the partner. But if the partner is offline at boot time, the
|
||||
pull fails and fallback has no partner vars to work with.
|
||||
|
||||
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
|
||||
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
|
||||
Fix: `conf_cache_save.sh` + `conf_cache_restore.sh` — snapshot the RAM cache to
|
||||
`$PERSISTENT_CONF_CACHE` on array stop; reload on next start for any confs the
|
||||
sync couldn't fetch.
|
||||
|
||||
**Mover/Rsync Interruption During Reboot**
|
||||
Rsync transfers or mover runs are in progress when a reboot is triggered. Stopping
|
||||
them uncleanly leaves partial files.
|
||||
|
||||
Fix: `server_reboot.sh` — orchestrates `array_stopping.sh` which stops rsync, mover,
|
||||
and containers in the correct order before calling `/sbin/reboot`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
```
|
||||
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
|
||||
Kernel tuning inotify_tuning.sh — file watch limits
|
||||
php_fpm_max_children.sh — PHP worker count
|
||||
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
|
||||
clear_logs.sh — weekly log trimming
|
||||
Graceful operations mover_stop.sh — clean mover stop
|
||||
rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||||
user_scripts_stop.sh — stop running User Scripts
|
||||
clear_logs.sh — size-threshold log trimming
|
||||
Conf synchronisation conf_sync.sh — pull/push partner confs → RAM cache
|
||||
conf_cache_save.sh — snapshot RAM cache → persistent at stop
|
||||
conf_cache_restore.sh — reload from snapshot at start (offline partner)
|
||||
Graceful operations rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||||
server_reboot.sh — clean reboot with pre-flight warnings
|
||||
```
|
||||
|
||||
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
|
||||
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
```
|
||||
Orchestrators/
|
||||
array_started.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
|
||||
─────────────────────────► docker_syslog_filter.sh (second)
|
||||
─────────────────────────► php_fpm_max_children.sh
|
||||
array_started.sh ────────────────────────► conf_sync.sh (pulls partner confs)
|
||||
────────────────────────► conf_cache_restore.sh (fills gaps if partner down)
|
||||
────────────────────────► docker_syslog_filter.sh (before containers start)
|
||||
────────────────────────► inotify_tuning.sh (before docker_network_connect)
|
||||
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
|
||||
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
|
||||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||||
|
||||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||||
Watchdogs/System/
|
||||
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
|
||||
(runs every 15 min via watchdog_orchestrator.sh)
|
||||
|
||||
Watchdogs/
|
||||
stability_watchdog.sh and resource_watchdog.sh now live here.
|
||||
See Watchdogs/README-Watchdogs.md for how they relate to each other
|
||||
and to docker_watchdog.sh and System/storage_watchdog.sh.
|
||||
System_Essentials/
|
||||
server_reboot.sh ────────────────────────► array_stopping.sh (via Orchestrators/)
|
||||
────────────────────────► mover_stop.sh, user_scripts_stop.sh
|
||||
(via Plugin/unraid/System_Essentials/)
|
||||
|
||||
Plugin/unraid/System_Essentials/
|
||||
php_fpm_max_children.sh — WebGUI tuning (Unraid-specific PHP paths)
|
||||
mover_stop.sh — clean mover stop (Unraid mover daemon)
|
||||
user_scripts_stop.sh — stop User Scripts plugin processes
|
||||
unraid_api_key_renew.sh — Varaverk plugin API key renewal
|
||||
|
||||
Plugin/unraid/Watchdogs/System/
|
||||
webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
|
||||
```
|
||||
|
||||
---
|
||||
@@ -110,14 +104,13 @@ Watchdogs/
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
|
||||
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — before container ops |
|
||||
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
|
||||
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
|
||||
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
|
||||
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
|
||||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
|
||||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
|
||||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
|
||||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||||
|
||||
---
|
||||
@@ -125,27 +118,28 @@ Watchdogs/
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
Array starts
|
||||
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
|
||||
│
|
||||
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
|
||||
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
|
||||
└─ php_fpm_max_children.sh ← before WebGUI is under load
|
||||
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
|
||||
│ push own conf to partner's /tmp/.cache/vv/d/
|
||||
├─ conf_cache_restore.sh ← if partner was down: load last-known-good conf from
|
||||
│ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
|
||||
├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live)
|
||||
└─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts
|
||||
|
||||
Every minute (watchdog_orchestrator.sh in Orchestrators/):
|
||||
→ Watchdogs/resource_watchdog.sh
|
||||
→ Watchdogs/docker_watchdog.sh
|
||||
→ Watchdogs/system_watchdog.sh (thin orchestrator)
|
||||
└─ Watchdogs/System/storage_watchdog.sh
|
||||
└─ Watchdogs/System/webgui_watchdog.sh
|
||||
WebGUI OK → silent exit
|
||||
Not responding:
|
||||
Step 1: restart nginx → recheck
|
||||
Step 2: restart php-fpm → recheck
|
||||
Step 3: restart emhttp → recheck
|
||||
All failed → notify, exit 1
|
||||
└─ Watchdogs/System/network_watchdog.sh
|
||||
→ Watchdogs/stability_watchdog.sh
|
||||
(see Watchdogs/README-Watchdogs.md for full flow)
|
||||
Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
|
||||
│
|
||||
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
|
||||
│ while RAM cache is still fresh
|
||||
├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
|
||||
└─ ...other stop scripts...
|
||||
|
||||
Every 15 minutes (watchdog_orchestrator.sh):
|
||||
→ Watchdogs/System/conf_cache_watchdog.sh
|
||||
If partner is offline and persistent backup is stale → refresh from last RAM cache
|
||||
|
||||
Every 4 hours (intermediate_sync_maintenance.sh):
|
||||
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
|
||||
|
||||
Weekly (weekly_maintenance.sh):
|
||||
└─ clear_logs.sh
|
||||
@@ -153,8 +147,6 @@ Weekly (weekly_maintenance.sh):
|
||||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||||
|
||||
Manual operations:
|
||||
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
|
||||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||||
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
|
||||
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
|
||||
server_reboot.sh → pre-flight → array_stopping.sh → reboot
|
||||
```
|
||||
|
||||
@@ -30,7 +30,7 @@ detect_hosts
|
||||
require_partnership
|
||||
|
||||
RAM_CACHE="/tmp/.cache/vv/d"
|
||||
SAVE_DIR="/boot/config/.cache/vv/d"
|
||||
SAVE_DIR="$PERSISTENT_CONF_CACHE"
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Snapshots the partner conf RAM cache to /boot/config/.cache/vv/d/ on
|
||||
# Snapshots the partner conf RAM cache to $PERSISTENT_CONF_CACHE on
|
||||
# array stop. Survives reboot. Used by conf_cache_restore.sh at next array start
|
||||
# to reload partner vars into RAM when the partner is unreachable at boot time.
|
||||
#
|
||||
@@ -15,7 +15,7 @@
|
||||
# down, the backup fills the gap so fallback.sh has the vars it needs.
|
||||
#
|
||||
# Only partner confs are saved — own conf is always on disk.
|
||||
# Location is outside the git repo and outside the main plugin folder.
|
||||
# Path adapts to storage mode: $SCRIPTS_DIR/.cache/vv/d (internal or appdata).
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -26,7 +26,7 @@ detect_hosts
|
||||
require_partnership
|
||||
|
||||
RAM_CACHE="/tmp/.cache/vv/d"
|
||||
SAVE_DIR="/boot/config/.cache/vv/d"
|
||||
SAVE_DIR="$PERSISTENT_CONF_CACHE"
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be written"
|
||||
|
||||
|
||||
@@ -11,15 +11,18 @@
|
||||
# On array start (default / --array-start):
|
||||
# 1. Copy own conf to local cache
|
||||
# 2. Pull each available partner's conf from their disk → local cache
|
||||
# 3. Push own conf to each available partner's /tmp/.vv/ cache
|
||||
# 3. Push own conf to each available partner's /tmp/.cache/vv/d/ cache
|
||||
#
|
||||
# On conf save (--push-only):
|
||||
# Fast path — push updated own conf to all partners' /tmp/.vv/ cache only.
|
||||
# Fast path — push updated own conf to all partners' /tmp/.cache/vv/d/ only.
|
||||
# No pulls, no local cache rebuild.
|
||||
#
|
||||
# Cache is /tmp (tmpfs) — cleared every reboot, repopulated by this script
|
||||
# on next array start. Scripts source from cache for partner vars; own vars
|
||||
# always come from disk (load_config.sh skips cached copy of own conf).
|
||||
# Cache is /tmp/.cache/vv/d (tmpfs) — cleared every reboot, repopulated by
|
||||
# this script on next array start. Scripts source from cache for partner vars;
|
||||
# own vars always come from disk (load_config.sh skips cached copy of own conf).
|
||||
#
|
||||
# Pull path resolves the remote's SCRIPTS_DIR from their varaverk.cfg so it
|
||||
# works whether the remote is in internal or appdata storage mode.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
@@ -56,9 +59,25 @@ if [[ "${CONF_SYNC_ENABLED:-true}" == false ]]; then
|
||||
fi
|
||||
|
||||
CACHE_DIR="/tmp/.cache/vv/d"
|
||||
MY_CONF="$SCRIPTS_ROOT/Configurations/${MY_ID,,}.conf"
|
||||
MY_CONF="$SCRIPTS_DIR/Configurations/${MY_ID,,}.conf"
|
||||
SSH_TIMEOUT=10
|
||||
|
||||
# Reads the remote's varaverk.cfg to find their actual SCRIPTS_DIR.
|
||||
# Handles the case where the remote is in appdata storage mode.
|
||||
_remote_scripts_dir() {
|
||||
local ip="$1"
|
||||
local cfg line sd
|
||||
cfg=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
|
||||
"root@${ip}" "cat /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null" 2>/dev/null) || true
|
||||
while IFS= read -r line; do
|
||||
[[ "$line" == SCRIPTS_DIR=* ]] || continue
|
||||
sd="${line#SCRIPTS_DIR=}"; sd="${sd//\"/}"; sd="${sd//\'/}"
|
||||
echo "$sd"; return
|
||||
done <<< "$cfg"
|
||||
echo "/boot/config/plugins/varaverk"
|
||||
}
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ── Ensure cache dir exists ───────────────────────────────────────────────────
|
||||
@@ -101,7 +120,8 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
|
||||
|
||||
# ── Pull: grab partner's conf from their disk → our local cache ──────────
|
||||
if [[ "$PUSH_ONLY" == false ]]; then
|
||||
remote_conf="${SCRIPTS_DIR}/Configurations/${partner_slot}.conf"
|
||||
remote_sd=$(_remote_scripts_dir "$partner_ip")
|
||||
remote_conf="${remote_sd}/Configurations/${partner_slot}.conf"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would pull $partner_host:$remote_conf → $CACHE_DIR/${partner_slot}.conf"
|
||||
elif timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
|
||||
@@ -116,7 +136,7 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Push: send own conf to partner's /tmp/.vv/ cache ────────────────────
|
||||
# ── Push: send own conf to partner's /tmp/.cache/vv/d/ cache ───────────
|
||||
if [[ "$PULL_ONLY" == true ]]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
@@ -214,7 +214,7 @@ View and manage the persistent container skip list used by `docker_watchdog.sh`.
|
||||
|
||||
```
|
||||
docker_watchdog.sh restarts the same container N times within the rolling window
|
||||
→ container added to skip list on /boot/config/
|
||||
→ container added to skip list in $STATE_DIR
|
||||
→ critical notification sent
|
||||
→ watchdog stops touching it entirely
|
||||
|
||||
|
||||
@@ -13,8 +13,9 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
||||
- [docker_watchdog.sh](#docker_watchdogsh)
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
|
||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) ← Plugin/unraid/Watchdogs/System/
|
||||
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
|
||||
- [System/conf_cache_watchdog.sh](#systemconf_cache_watchdogsh)
|
||||
- [stability_watchdog.sh](#stability_watchdogsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
@@ -373,6 +374,10 @@ storage_watchdog.sh --log # verbose per-container output
|
||||
|
||||
## System/webgui_watchdog.sh
|
||||
|
||||
> **Lives in `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh`** — calls Unraid-specific
|
||||
> service commands (`rc.nginx`, `rc.php-fpm`, `emhttp`) via the platform adapter. Called by
|
||||
> `system_watchdog.sh` via `SYSTEM_WATCHDOG_SCRIPTS` in master.conf.
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
|
||||
through three restart steps if unresponsive. Silent when healthy.
|
||||
|
||||
@@ -446,7 +451,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
|
||||
|
||||
# host*.conf (host-specific)
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
@@ -506,6 +511,40 @@ tailscale up
|
||||
|
||||
---
|
||||
|
||||
## System/conf_cache_watchdog.sh
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Maintains the persistent partner conf backup
|
||||
at `$PERSISTENT_CONF_CACHE` while the partner is offline.
|
||||
|
||||
### Behaviour
|
||||
|
||||
**Remote online:** removes the persistent backup if one exists. It is not needed —
|
||||
`conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
|
||||
|
||||
**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to
|
||||
`$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
|
||||
an extended outage. If this host reboots while the partner is still down,
|
||||
`conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
|
||||
valid partner vars.
|
||||
|
||||
Silent when remote is online and no backup exists (the normal steady state).
|
||||
|
||||
### Gates
|
||||
|
||||
- `FALLBACK_ENABLED=false` → no-op (no fallback means no need for partner vars)
|
||||
- `CONF_SYNC_ENABLED=false` → no-op
|
||||
- `PARTNERSHIP_ENABLED=false` → exits silently (require_partnership gate)
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
conf_cache_watchdog.sh # single pass (called by system_watchdog.sh)
|
||||
conf_cache_watchdog.sh --dry-run # show what would be written or removed
|
||||
conf_cache_watchdog.sh --log # verbose output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## stability_watchdog.sh
|
||||
|
||||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||||
@@ -653,9 +692,9 @@ WATCHDOG_RESTART_DEAD=true
|
||||
WATCHDOG_RESTART_CRASHED=true
|
||||
WATCHDOG_BATCH_NOTIFY=true
|
||||
|
||||
# State files:
|
||||
WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
|
||||
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
|
||||
# State files (all paths adapt to storage mode via STATE_DIR / DATA_DIR):
|
||||
WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
|
||||
WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
|
||||
|
||||
# host*.conf
|
||||
HOST1_WATCHDOG_CONTAINERS=() # "ContainerName:LimitMB"
|
||||
@@ -670,8 +709,8 @@ WATCHDOG_APPDATA_GROWTH_GB=2
|
||||
WATCHDOG_APPDATA_LOG_MAX_GB=2
|
||||
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
|
||||
WATCHDOG_APPDATA_STRIKE_LIMIT=3
|
||||
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
|
||||
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
|
||||
WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
|
||||
STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"
|
||||
|
||||
# host*.conf (optional — only for suppress ceilings)
|
||||
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||||
@@ -701,11 +740,12 @@ SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=false
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=false
|
||||
|
||||
# State files (all in State_Files/ — survive reboots):
|
||||
SYS_WATCHDOG_STATE_FILE="State_Files/system_watchdog_state.db"
|
||||
DOCKER_WATCHDOG_FAILED_FILE="State_Files/docker_watchdog_failed.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="State_Files/system_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_OOM_FILE="State_Files/system_watchdog_oom.db"
|
||||
# State files — all in $STATE_DIR (survive reboots; adapt to storage mode):
|
||||
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
|
||||
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
|
||||
RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"
|
||||
|
||||
# ── Network Watchdog ───────────────────────────────────────────────────────────
|
||||
NETWORK_WATCHDOG_ENABLED=true
|
||||
@@ -714,7 +754,7 @@ NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
|
||||
|
||||
# host*.conf (host-specific)
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
@@ -781,8 +821,8 @@ storage_watchdog.sh --status
|
||||
### stability_watchdog Rebooted Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check the reboot log (survives reboots):
|
||||
cat /boot/config/system_watchdog_reboots.db
|
||||
# Check the reboot log (survives reboots — in State_Files/):
|
||||
cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db
|
||||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||||
|
||||
# Check syslog near the reboot time:
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
**Four tiers that run every 15 minutes through `watchdog_orchestrator.sh`,
|
||||
each with a clear lane:** reduce system pressure → heal containers → check system components →
|
||||
reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
|
||||
System component checks (storage, WebGUI, network) run inside a thin `system_watchdog.sh`
|
||||
orchestrator that can also be run standalone.
|
||||
System component checks (storage, WebGUI, network, conf cache) run inside a thin
|
||||
`system_watchdog.sh` orchestrator that can also be run standalone.
|
||||
|
||||
---
|
||||
|
||||
@@ -57,6 +57,15 @@ Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kern
|
||||
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
|
||||
sequence so container healing and pressure reduction always get a chance first.
|
||||
|
||||
**Partner Conf Backup Going Stale During a Long Outage**
|
||||
The partner goes offline overnight. On the next reboot (planned maintenance), the
|
||||
persistent conf backup saved at shutdown is stale — it reflects the state from before
|
||||
the outage, not the most recent live values. `conf_cache_restore.sh` loads it
|
||||
faithfully, but fallback.sh may be working with old tier delays or container names.
|
||||
|
||||
Fix: `conf_cache_watchdog.sh` — refreshes the persistent backup from the RAM cache
|
||||
every 15 minutes while the partner is offline, keeping it current throughout the outage.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
@@ -66,13 +75,17 @@ Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
|
||||
```
|
||||
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
|
||||
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
|
||||
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
|
||||
System components system_watchdog.sh — thin orchestrator: system component health
|
||||
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
|
||||
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
|
||||
network_watchdog.sh — internet, DDNS sync, Tailscale, NPM proxy
|
||||
conf_cache_watchdog.sh — maintain persistent partner conf backup
|
||||
Last resort stability_watchdog.sh — reboot only when nothing else can recover
|
||||
```
|
||||
|
||||
> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
|
||||
> but lives in the plugin tree because it calls Unraid-specific service commands
|
||||
> (nginx, php-fpm, emhttp via platform adapter). See `Plugin/unraid/Watchdogs/System/`.
|
||||
|
||||
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
|
||||
attempts restarts — containers restarted into a pressure-bound system just fail again.
|
||||
System component checks run after containers are healed. Stability watchdog runs last —
|
||||
@@ -96,8 +109,16 @@ Orchestrators/
|
||||
|
||||
Watchdogs/System/ ← called by system_watchdog.sh each cycle
|
||||
storage_watchdog.sh pool growth rate + runaway log detection
|
||||
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
|
||||
network_watchdog.sh internet reachability, DDNS sync, Tailscale, NPM proxy
|
||||
conf_cache_watchdog.sh maintain persistent partner conf backup during outages
|
||||
|
||||
Plugin/unraid/Watchdogs/System/
|
||||
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
|
||||
(Unraid-specific platform calls — lives in plugin tree)
|
||||
|
||||
Plugin/unraid/System_Essentials/
|
||||
unraid_api_key_renew.sh re-registers the Varaverk plugin API key each watchdog cycle
|
||||
(inserted between docker_watchdog and stability_watchdog)
|
||||
|
||||
Tools/
|
||||
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
|
||||
@@ -106,6 +127,10 @@ Tools/
|
||||
Docker_Essentials/
|
||||
All container lifecycle scripts (daily restart, updates, network) — unaffected.
|
||||
docker_watchdog.sh coordinates with them via shared state, not direct calls.
|
||||
|
||||
System_Essentials/
|
||||
conf_cache_save.sh / conf_cache_restore.sh — at-stop/at-start bookends for the
|
||||
persistent backup that conf_cache_watchdog.sh keeps current in between.
|
||||
```
|
||||
|
||||
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
|
||||
@@ -119,14 +144,15 @@ Docker_Essentials/
|
||||
|--------|------|-----------|
|
||||
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every 15 min |
|
||||
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every 15 min |
|
||||
| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
|
||||
| `system_watchdog.sh` | Thin orchestrator — runs SYSTEM_WATCHDOG_SCRIPTS in sequence | `watchdog_orchestrator.sh` — 3rd every 15 min |
|
||||
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every 15 min |
|
||||
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every 15 min |
|
||||
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every 15 min |
|
||||
| `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every 15 min |
|
||||
| `System/conf_cache_watchdog.sh` | Refresh persistent partner conf backup while partner is offline | `system_watchdog.sh` — every 15 min |
|
||||
|
||||
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
|
||||
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
|
||||
> `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh` is also called by `system_watchdog.sh`
|
||||
> but is not in this folder. `watchdog_orchestrator.sh` is in `Orchestrators/`.
|
||||
> `watchdog_skip_list_manager.sh` is in `Tools/`. Neither is a watchdog.
|
||||
|
||||
---
|
||||
|
||||
@@ -166,7 +192,7 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
|
||||
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
|
||||
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
|
||||
│
|
||||
│ └─ webgui_watchdog.sh
|
||||
│ └─ Plugin/unraid/Watchdogs/System/webgui_watchdog.sh
|
||||
│ curl check → WebGUI responding → exit 0 (silent)
|
||||
│ not responding → nginx restart → wait → recheck
|
||||
│ still down → php-fpm restart → wait → recheck
|
||||
@@ -179,6 +205,13 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
|
||||
│ Tailscale: status Running → pass; not running = notify (no auto-restart)
|
||||
│ NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
|
||||
│
|
||||
│ └─ conf_cache_watchdog.sh
|
||||
│ remote online → remove persistent backup (conf_sync gets fresh on next boot)
|
||||
│ remote offline → refresh backup from RAM cache → backup stays current
|
||||
│ silent when remote is online and no backup exists (normal state)
|
||||
│
|
||||
│ (+ Plugin/unraid/System_Essentials/unraid_api_key_renew.sh — between steps 3 and 4)
|
||||
│
|
||||
Step 4 — stability_watchdog.sh
|
||||
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
|
||||
Tier 1 CRITICAL → immediate reboot (no strikes)
|
||||
@@ -195,7 +228,9 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
|
||||
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
|
||||
| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
|
||||
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
|
||||
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
|
||||
| `DOCKER_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
|
||||
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
|
||||
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
|
||||
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
||||
|
||||
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Maintains the persistent partner conf backup at /boot/config/.cache/vv/d/.
|
||||
# Maintains the persistent partner conf backup at $PERSISTENT_CONF_CACHE.
|
||||
# Runs every 15 minutes via SYSTEM_WATCHDOG_SCRIPTS.
|
||||
#
|
||||
# When remote is OFFLINE:
|
||||
@@ -33,7 +33,7 @@ require_partnership
|
||||
[[ -z "${REMOTE_ID:-}" ]] && exit 0
|
||||
|
||||
RAM_CACHE="/tmp/.cache/vv/d"
|
||||
SAVE_DIR="/boot/config/.cache/vv/d"
|
||||
SAVE_DIR="$PERSISTENT_CONF_CACHE"
|
||||
|
||||
if ping_remote; then
|
||||
if [[ -d "$SAVE_DIR" ]]; then
|
||||
|
||||
Reference in New Issue
Block a user