FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Storage-mode awareness pass + doc update for System_Essentials through Partnership

Browse Source 
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.

PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.

conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.

fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.

README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
This commit is contained in:
Gmer4Lfe
2026-06-19 19:32:39 -04:00
parent 0564580605
commit bf3e7cc2c4
35 changed files with 835 additions and 489 deletions
Download Patch File Download Diff File Expand all files Collapse all files
System_Essentials/Manual-System_Essentials.md
+192 -238
View File
@@ -4,23 +4,24 @@ Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start.
> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) and their configuration live in
> `Plugin/unraid/System_Essentials/`. Watchdog configuration (`stability_watchdog.sh`,
> `resource_watchdog.sh`, `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives
> in `Watchdogs/Manual-Watchdogs.md`.
---
## ━━━ CONTENTS ━━━
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
- [webgui_watchdog.sh](#webgui_watchdogsh)
- [ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order](#array_start_scripts--array_stop_scripts-order)
- [conf_sync.sh](#conf_syncsh)
- [conf_cache_save.sh](#conf_cache_savesh)
- [conf_cache_restore.sh](#conf_cache_restoresh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
- [clear_logs.sh](#clear_logssh)
- [mover_stop.sh](#mover_stopsh)
- [rsync_stop.sh](#rsync_stopsh)
- [user_scripts_stop.sh](#user_scripts_stopsh)
- [server_reboot.sh](#server_rebootsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -32,109 +33,155 @@ adding or reordering scripts at array start.
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
Per-check detail suppressed.
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
**One-shot scripts** (`conf_sync.sh`, `conf_cache_save.sh`, `conf_cache_restore.sh`,
`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`, `rsync_stop.sh`,
`server_reboot.sh`): without `--log`, section headers, per-step results, and the
final summary are visible. Per-item detail suppressed.
---
## ARRAY_START_SCRIPTS Order
## ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
> scripts.** Getting it wrong causes subtle failures that don't show up
> immediately.
> **Order matters.** Scripts that set up conditions other scripts depend on must run
> first. Getting the order wrong causes subtle failures.
```bash
# master.conf
# master.conf (current order — from ARRAY_START_SCRIPTS)
ARRAY_START_SCRIPTS=(
"inotify_tuning.sh" # 1 — FIRST: kernel limits must be set before
# any container starts. Containers inherit
# inotify limits at launch, not dynamically.
"docker_syslog_filter.sh" # 2 — SECOND: before any veth interfaces are
# created. If a container starts first, its
# veth creation is already in syslog.
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
...
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # plugin API key — ephemeral registry
"System_Essentials/conf_sync.sh" # 1st cross-server op — pull partner confs into RAM cache
"System_Essentials/conf_cache_restore.sh" # fill gaps if partner was down at boot
"Transcodes/ramdisk_setup.sh" # ramdisk before Emby starts
"System_Essentials/docker_syslog_filter.sh" # before veth interfaces are created
"Plugin/unraid/System_Essentials/php_fpm_max_children.sh" # WebGUI tuning
"System_Essentials/inotify_tuning.sh" # before docker_network_connect and continuous scripts
"Docker_Essentials/docker_network_connect.sh" # ensure networks + container connections
"Tools/claude_startup.sh" # persist Claude data to appdata; re-symlink on boot
"Media/start_webhook_listener.sh" # arr upgrade webhook — continuous
"Fallback/fallback.sh" # mutual failover — continuous
)
ARRAY_STOP_SCRIPTS=(
"System_Essentials/conf_cache_save.sh" # FIRST: snapshot RAM cache while fresh
"Plugin/unraid/System_Essentials/user_scripts_stop.sh"
"Fallback/fallback.sh --stop"
"System_Essentials/rsync_stop.sh --rsync-only"
"Plugin/unraid/System_Essentials/mover_stop.sh"
"Docker_Essentials/docker_container_stop.sh"
)
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
```
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
Why conf_sync before conf_cache_restore: the sync runs first to get the freshest
possible partner conf. The restore only fills in for confs that the sync couldn't
fetch — it is a fallback, not the primary path.
Why docker_syslog_filter SECOND: The filter must be in place before any container
starts creating veth interfaces. The first container start after array start
generates veth messages — these will appear in syslog if the filter isn't active.
Why docker_syslog_filter before inotify: the filter must be in place before any
container creates veth interfaces. inotify must be set before docker_network_connect
and the continuous scripts (start_webhook_listener, fallback), which are the first
steps that actually touch containers.
Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at
`/tmp/.cache/vv/d/` is wiped on reboot. Saving it must happen before anything
else shuts down or changes state.
---
## webgui_watchdog.sh
## conf_sync.sh
### Escalation Logic
### Modes
```
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
Not responding:
1. platform_restart_service nginx
wait WEBGUI_NGINX_WAIT (15s) → recheck
→ recovered: notify, exit 0
2. platform_restart_service php-fpm
wait WEBGUI_PHP_WAIT (10s) → recheck
→ recovered: notify, exit 0
3. platform_restart_service emhttp
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
→ recovered: notify, exit 0
All three failed → notify warning, exit 1
conf_sync.sh Full sync: pull from all partners + push to all partners
conf_sync.sh --push-only Push own conf to all partners (fast — for conf-save hook)
conf_sync.sh --pull-only Pull partner confs only (intermediate orchestrator)
conf_sync.sh --dry-run Show what would happen, no changes
conf_sync.sh --log Verbose output
```
### Configuration
### What It Syncs
- **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk
via SCP → writes to local `/tmp/.cache/vv/d/${partner_id}.conf`
- **Push**: sends own `Configurations/${my_id}.conf` to partner's
`/tmp/.cache/vv/d/${my_id}.conf` via SCP
- **Own conf in local cache**: copies own conf to `/tmp/.cache/vv/d/${my_id}.conf`
on full sync (so the cache has a complete snapshot of all confs)
Only partner confs are sourced from cache — `load_config.sh` always reads own conf
from disk to avoid sourcing a stale cached copy.
### Remote SCRIPTS_DIR Resolution
The pull path reads the partner's `/boot/config/plugins/varaverk/varaverk.cfg` to
find their actual `SCRIPTS_DIR` before building the SCP path. This handles the case
where the partner is in appdata storage mode and their conf is at
`/mnt/user/appdata/Varaverk/Configurations/` rather than the internal path.
### PARTNERSHIP_ENABLED Gate
conf_sync.sh calls `require_partnership` — if `PARTNERSHIP_ENABLED=false`, it exits
silently with 0. The conf cache will be empty for partner confs while partnership
is disabled.
### If Conf Pull Fails at Boot
If the partner is unreachable, `conf_sync.sh` logs a warning and exits 1.
`conf_cache_restore.sh` then runs (next in ARRAY_START_SCRIPTS) and loads the
persistent backup from `$PERSISTENT_CONF_CACHE` if available.
A notification fires if any partner fails — check partner reachability via Tailscale.
---
## conf_cache_save.sh
### What It Does
At array stop, copies all partner confs from `/tmp/.cache/vv/d/` to
`$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives
the reboot and is used by `conf_cache_restore.sh` at next array start if the sync
can't reach the partner.
`conf_cache_watchdog.sh` (in `Watchdogs/System/`) refreshes this backup every
15 minutes while the partner is offline — keeping it current even during extended
outages.
### PARTNERSHIP_ENABLED Gate
Exits silently when `PARTNERSHIP_ENABLED=false`. No backup is written.
### Usage
```bash
WEBGUI_URL="http://localhost" # URL to check
WEBGUI_TIMEOUT=5 # curl timeout in seconds
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
conf_cache_save.sh # save partner confs from RAM cache (called at array stop)
conf_cache_save.sh --dry-run # show what would be saved
conf_cache_save.sh --log # verbose output
```
### WebGUI Frozen — Manual Recovery
---
## conf_cache_restore.sh
### What It Does
At array start (after `conf_sync.sh`), checks which partner confs are missing from
`/tmp/.cache/vv/d/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
if a backup exists there.
Always removes the persistent backup when done — whether used or not. On a normal
reboot where the partner was reachable, the sync already populated the cache and the
backup is cleaned up without being used.
### PARTNERSHIP_ENABLED Gate
Exits silently when `PARTNERSHIP_ENABLED=false`.
### Usage
```bash
# Check which services are running:
webgui_watchdog.sh --status
# Try manual restart sequence (mirrors what the script calls):
# Source the ecosystem first to get platform functions:
source /boot/config/plugins/varaverk/load_config.sh
platform_restart_service nginx
# wait 15s, then:
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
# If nginx did not fix it, php-fpm:
platform_restart_service php-fpm
# If still down, emhttp:
platform_restart_service emhttp
# Raw equivalents (no source needed — paste directly into terminal):
# /etc/rc.d/rc.nginx restart
# /etc/rc.d/rc.php-fpm restart
# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
# If all three failed:
server_reboot.sh --status # check for active sessions first
conf_cache_restore.sh # restore missing confs from backup (called at array start)
conf_cache_restore.sh --dry-run # show what would be restored
conf_cache_restore.sh --log # verbose output
```
---
@@ -176,48 +223,6 @@ inotify_tuning.sh --log
---
## php_fpm_max_children.sh
### What It Sets
```bash
PHP_MAX_CHILDREN=250 # target pm.max_children (default: 4-8 on unRAID)
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
```
250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
The default of 4–8 saturates immediately under load on a busy server.
### Verify
```bash
php_fpm_max_children.sh --status
# Shows current value vs target, PHP-FPM worker count
# Manual verify:
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
# Expected: pm.max_children = 250
```
### If WebGUI Is Slow Despite the Setting
```bash
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
# pm.max_children value may still be too low, or there's a PHP worker leak.
# Check running worker count:
pgrep -fc php-fpm
# Compare to pm.max_children — if equal, workers are saturated
# Increase if needed:
# master.conf: PHP_MAX_CHILDREN=350
# Then: php_fpm_max_children.sh --log (will update and restart php-fpm)
```
---
## docker_syslog_filter.sh
### What It Creates
@@ -230,8 +235,7 @@ pgrep -fc php-fpm
```
This drops any syslog message containing "veth" or "docker0" before it reaches
any output target, including the log file. The drop rule is applied at rsyslog
level — not at the log viewer level.
any output target, including the log file.
### Verify
@@ -298,39 +302,6 @@ du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
---
## mover_stop.sh
### Stop Sequence
```
1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
2. Wall message to all logged-in terminal users
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
4. SIGTERM — mover finishes its current file operation, then stops
5. Wait 5 seconds → verify stopped
6. SIGKILL if still running — forced stop, partial files possible
7. Final verify — error if still running after SIGKILL
```
SIGTERM first because the mover can finish the file it is currently moving,
leaving no partial copies split across cache and array. SIGKILL is a last resort.
### Configuration
```bash
MOVER_STOP_TIMEOUT=30 # seconds between wall warning and SIGTERM
```
### Usage
```bash
mover_stop.sh # check and stop if running
mover_stop.sh --status # show current mover state and PID
mover_stop.sh --dry-run # show what would happen without stopping
```
---
## rsync_stop.sh
### Auto-Detection Logic
@@ -355,8 +326,8 @@ After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTA
for any that were stopped by the interrupted rsync session and restarts them.
Remote containers are left for docker_watchdog.sh to recover.
Skip container recovery with `--rsync-only` — used when called by other scripts
that handle recovery themselves.
Skip container recovery with `--rsync-only` — used when called by array_stopping.sh
(which handles container stop separately via docker_container_stop.sh).
### Usage
@@ -371,34 +342,6 @@ rsync_stop.sh --full-stop --dry-run # preview full stop
---
## user_scripts_stop.sh
### Process Identification
Scans `/proc/*/cmdline` for any process whose command line contains
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
what the script is named or how it was launched.
Script names are extracted from the path for display: you see which scripts are
being stopped, not just PIDs.
### Self-Exclusion
If this script is run via the User Scripts plugin, it would find its own PID in
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
from the kill list.
### Usage
```bash
user_scripts_stop.sh # stop all User Script processes
user_scripts_stop.sh --status # show running scripts with names and elapsed time
user_scripts_stop.sh --dry-run # show what would be stopped
```
---
## server_reboot.sh
### Full Shutdown Sequence
@@ -422,11 +365,12 @@ user_scripts_stop.sh --dry-run # show what would be stopped
6. /etc/rc.d/rc.libvirt stop (VM Manager)
7. Orchestrators/array_stopping.sh — safe ordered array stop:
- user_scripts_stop.sh stop background User Scripts
- fallback.sh --stop graceful fallback teardown
- rsync_stop.sh --rsync-only kill active rsync transfers
- mover_stop.sh stop mover
- docker_container_stop.sh stop all containers gracefully
- conf_cache_save.sh snapshot partner conf RAM cache
- user_scripts_stop.sh stop background User Scripts
- fallback.sh --stop graceful fallback teardown
- rsync_stop.sh --rsync-only kill active rsync transfers
- mover_stop.sh stop mover
- docker_container_stop.sh stop all containers gracefully
8. sync (flush filesystem buffers to disk)
@@ -446,7 +390,6 @@ For a clean reboot when services are active:
```bash
rsync_stop.sh # stop any active rsync (smart mode)
mover_stop.sh # stop mover gracefully
server_reboot.sh --status # check what's still running
server_reboot.sh --reason="planned maintenance"
```
@@ -464,27 +407,17 @@ server_reboot.sh --reason="disk work" # include reason in notification
## Full Configuration Reference
> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
```bash
# master.conf
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30
# ── Conf Sync ──────────────────────────────────────────────────────────────────
CONF_SYNC_ENABLED=true # toggle: false disables conf_sync.sh entirely
# ── inotify Tuning ─────────────────────────────────────────────────────────────
INOTIFY_MAX_INSTANCES=1024
INOTIFY_MAX_WATCHES=1048576
INOTIFY_MAX_QUEUED_EVENTS=32768
# ── PHP-FPM ────────────────────────────────────────────────────────────────────
PHP_MAX_CHILDREN=250
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
INOTIFY_WARN_PCT=80 # warn (in reports) if instances exceed this % of limit
# ── Syslog Filter ──────────────────────────────────────────────────────────────
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
@@ -494,20 +427,48 @@ LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
LOG_MIN_SIZE_MB=10
LOG_DOCKER_MAX_MB=100
# ── Mover Stop ─────────────────────────────────────────────────────────────────
MOVER_STOP_TIMEOUT=30
# ── Server Reboot ──────────────────────────────────────────────────────────────
REBOOT_SLEEP=30
REBOOT_VM_WAIT=30
# ── State Paths (auto-adapt to storage mode) ───────────────────────────────────
# STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
# They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
# (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
STATE_DIR="${SCRIPTS_DIR}/State_Files"
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
```
---
## Troubleshooting
> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
### Partner Conf Not In RAM Cache After Boot
```bash
# Check what's in the RAM cache:
ls -la /tmp/.cache/vv/d/
# Check what's in the persistent backup:
ls -la "$PERSISTENT_CONF_CACHE/" # set SCRIPTS_DIR first or use full path
# Re-run conf sync manually:
/boot/config/plugins/varaverk/System_Essentials/conf_sync.sh --log
# If partner is reachable but pull failed, check SSH key:
ssh -i /path/to/ssh_key root@<partner-tailscale-ip> "echo ok"
```
### Persistent Backup Is Stale or Empty
```bash
# conf_cache_watchdog.sh refreshes the backup while partner is offline.
# Check watchdog state:
cat "$STATE_DIR/conf_cache_watchdog_state.db"
# Force a manual backup from current RAM cache:
/boot/config/plugins/varaverk/System_Essentials/conf_cache_save.sh --log
```
### rsync_stop Killed the Wrong Thing
@@ -521,36 +482,29 @@ rsync_stop.sh --dry-run # shows smart mode decision
rsync_stop.sh --full-stop --dry-run # shows full-stop decision
```
### WebGUI Recovery After All Three Steps Failed
### inotify Exhaustion After Boot
```bash
# Check if processes are running:
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
pgrep emhttpd && echo "emhttp: yes" || echo "emhttp: no"
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
# Verify limits are applied:
sysctl fs.inotify.max_user_watches # expect 1048576
sysctl fs.inotify.max_user_instances # expect 1024
# Check recent nginx errors:
cat /var/log/nginx/error.log | tail -20
# If not set — run manually:
inotify_tuning.sh --log
# Check emhttp log:
tail -20 /var/log/syslog | grep emhttp
# Last resort — reboot:
server_reboot.sh --reason="WebGUI unrecoverable"
# Check current usage:
inotify_tuning.sh --status
```
### PHP-FPM Config Not Found After unRAID Update
unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
errors with "config file not found":
### Syslog Still Noisy After Array Start
```bash
# Find the new config path:
find /etc -name "www.conf" 2>/dev/null
# Check filter is in place:
docker_syslog_filter.sh --status
# Update PHP_CONF in master.conf:
PHP_CONF="/etc/php84/php-fpm.d/www.conf" # example for php84
# Re-apply if needed:
docker_syslog_filter.sh --log
# Verify with:
php_fpm_max_children.sh --status
# Restart rsyslog to pick up the filter:
/etc/rc.d/rc.rsyslogd restart
```
System_Essentials/README-System_Essentials.md
+79 -87
View File
@@ -1,35 +1,18 @@
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
**System-level scripts that act on the server itself — not containers,
not media, not monitoring.** Keeping the server stable under load, recovering a
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
shutdowns with proper warning sequences.
not media, not monitoring.** Kernel limits, log hygiene, conf synchronisation
between servers, and graceful shutdowns with proper warning sequences.
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) live in `Plugin/unraid/System_Essentials/`
> because they call Unraid-specific service commands and paths. This folder
> contains scripts that would run unchanged on any Linux host.
---
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
**Server Getting Into Unstable States With No Recovery Path**
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
watchdog tries to restart the container — but Docker itself is barely responding.
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
write log files. The server is functionally dead but still technically running.
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
threshold breaches. Last line of defense before a hard crash.
**WebGUI Freezing and Nobody Noticing**
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
load, settings don't save, containers can't be started or stopped via the UI. No
container-level alert fires because this isn't a container problem — it's a web
server problem. By the time someone notices it may have been broken for hours.
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
php-fpm → emhttp. Lightest fix first. Silent when healthy.
**50+ Containers Starting and Filling Syslog With Veth Noise**
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
network interface for each one. Each interface generates multiple syslog entries.
@@ -40,68 +23,79 @@ In the first minute after array start, syslog is buried under 200–400 lines of
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
starts. Applied at array start. Idempotent — silent when already correct.
**WebGUI Queuing Requests Under Load Without Explanation**
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
those workers saturate immediately. New requests queue behind active ones.
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
**inotify Exhaustion Producing Unexplained Failures**
When inotify limits are exhausted, containers silently stop receiving filesystem
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
and all containers share the same pool.
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
run FIRST in ARRAY_START_SCRIPTS before any containers start.
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start, before
any container-connecting or continuous scripts start.
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
The mover is running — moving a large batch of files from cache to array. A reboot
is triggered. The mover stops mid-file. The file exists partially on both cache and
array simultaneously. unRAID's deduplication layer is confused.
**Partner Conf Lost Across Reboots When Partner is Down**
Scripts like `fallback.sh` need the partner's conf vars (credentials, container
names, tier delays) to operate. The partner conf lives in a RAM cache at
`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls
a fresh copy from the partner. But if the partner is offline at boot time, the
pull fails and fallback has no partner vars to work with.
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
Fix: `conf_cache_save.sh` + `conf_cache_restore.sh` — snapshot the RAM cache to
`$PERSISTENT_CONF_CACHE` on array stop; reload on next start for any confs the
sync couldn't fetch.
**Mover/Rsync Interruption During Reboot**
Rsync transfers or mover runs are in progress when a reboot is triggered. Stopping
them uncleanly leaves partial files.
Fix: `server_reboot.sh` — orchestrates `array_stopping.sh` which stops rsync, mover,
and containers in the correct order before calling `/sbin/reboot`.
---
## ━━━ WHAT THIS FOLDER DOES ━━━
```
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
Kernel tuning inotify_tuning.sh — file watch limits
php_fpm_max_children.sh — PHP worker count
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
clear_logs.sh — weekly log trimming
Graceful operations mover_stop.sh — clean mover stop
rsync_stop.sh — smart rsync stop (orchestrator-aware)
user_scripts_stop.sh — stop running User Scripts
clear_logs.sh — size-threshold log trimming
Conf synchronisation conf_sync.sh — pull/push partner confs → RAM cache
conf_cache_save.sh — snapshot RAM cache → persistent at stop
conf_cache_restore.sh — reload from snapshot at start (offline partner)
Graceful operations rsync_stop.sh — smart rsync stop (orchestrator-aware)
server_reboot.sh — clean reboot with pre-flight warnings
```
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
---
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
```
Orchestrators/
array_started.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
─────────────────────────► docker_syslog_filter.sh (second)
─────────────────────────► php_fpm_max_children.sh
array_started.sh ────────────────────────► conf_sync.sh (pulls partner confs)
────────────────────────► conf_cache_restore.sh (fills gaps if partner down)
────────────────────────► docker_syslog_filter.sh (before containers start)
────────────────────────► inotify_tuning.sh (before docker_network_connect)
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
weekly_maintenance.sh ──────────────────► clear_logs.sh
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
Watchdogs/System/
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
(runs every 15 min via watchdog_orchestrator.sh)
Watchdogs/
stability_watchdog.sh and resource_watchdog.sh now live here.
See Watchdogs/README-Watchdogs.md for how they relate to each other
and to docker_watchdog.sh and System/storage_watchdog.sh.
System_Essentials/
server_reboot.sh ────────────────────────► array_stopping.sh (via Orchestrators/)
────────────────────────► mover_stop.sh, user_scripts_stop.sh
(via Plugin/unraid/System_Essentials/)
Plugin/unraid/System_Essentials/
php_fpm_max_children.sh — WebGUI tuning (Unraid-specific PHP paths)
mover_stop.sh — clean mover stop (Unraid mover daemon)
user_scripts_stop.sh — stop User Scripts plugin processes
unraid_api_key_renew.sh — Varaverk plugin API key renewal
Plugin/unraid/Watchdogs/System/
webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
```
---
@@ -110,14 +104,13 @@ Watchdogs/
| Script | Role | When It Runs |
|--------|------|-------------|
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — before container ops |
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
---
@@ -125,27 +118,28 @@ Watchdogs/
## ━━━ HOW THE SCRIPTS RELATE ━━━
```
Array starts
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
│
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
└─ php_fpm_max_children.sh ← before WebGUI is under load
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
│ push own conf to partner's /tmp/.cache/vv/d/
├─ conf_cache_restore.sh ← if partner was down: load last-known-good conf from
│ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live)
└─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts
Every minute (watchdog_orchestrator.sh in Orchestrators/):
→ Watchdogs/resource_watchdog.sh
→ Watchdogs/docker_watchdog.sh
→ Watchdogs/system_watchdog.sh (thin orchestrator)
└─ Watchdogs/System/storage_watchdog.sh
└─ Watchdogs/System/webgui_watchdog.sh
WebGUI OK → silent exit
Not responding:
Step 1: restart nginx → recheck
Step 2: restart php-fpm → recheck
Step 3: restart emhttp → recheck
All failed → notify, exit 1
└─ Watchdogs/System/network_watchdog.sh
→ Watchdogs/stability_watchdog.sh
(see Watchdogs/README-Watchdogs.md for full flow)
Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
│
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
│ while RAM cache is still fresh
├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
└─ ...other stop scripts...
Every 15 minutes (watchdog_orchestrator.sh):
→ Watchdogs/System/conf_cache_watchdog.sh
If partner is offline and persistent backup is stale → refresh from last RAM cache
Every 4 hours (intermediate_sync_maintenance.sh):
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
Weekly (weekly_maintenance.sh):
└─ clear_logs.sh
@@ -153,8 +147,6 @@ Weekly (weekly_maintenance.sh):
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
Manual operations:
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
server_reboot.sh → pre-flight → array_stopping.sh → reboot
```
System_Essentials/conf_cache_restore.sh
+1 -1
View File
@@ -30,7 +30,7 @@ detect_hosts
require_partnership
RAM_CACHE="/tmp/.cache/vv/d"
SAVE_DIR="/boot/config/.cache/vv/d"
SAVE_DIR="$PERSISTENT_CONF_CACHE"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
System_Essentials/conf_cache_save.sh
+3 -3
View File
@@ -5,7 +5,7 @@
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Snapshots the partner conf RAM cache to /boot/config/.cache/vv/d/ on
# Snapshots the partner conf RAM cache to $PERSISTENT_CONF_CACHE on
# array stop. Survives reboot. Used by conf_cache_restore.sh at next array start
# to reload partner vars into RAM when the partner is unreachable at boot time.
#
@@ -15,7 +15,7 @@
# down, the backup fills the gap so fallback.sh has the vars it needs.
#
# Only partner confs are saved — own conf is always on disk.
# Location is outside the git repo and outside the main plugin folder.
# Path adapts to storage mode: $SCRIPTS_DIR/.cache/vv/d (internal or appdata).
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -26,7 +26,7 @@ detect_hosts
require_partnership
RAM_CACHE="/tmp/.cache/vv/d"
SAVE_DIR="/boot/config/.cache/vv/d"
SAVE_DIR="$PERSISTENT_CONF_CACHE"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be written"
System_Essentials/conf_sync.sh
+28 -8
View File
@@ -11,15 +11,18 @@
# On array start (default / --array-start):
# 1. Copy own conf to local cache
# 2. Pull each available partner's conf from their disk → local cache
# 3. Push own conf to each available partner's /tmp/.vv/ cache
# 3. Push own conf to each available partner's /tmp/.cache/vv/d/ cache
#
# On conf save (--push-only):
# Fast path — push updated own conf to all partners' /tmp/.vv/ cache only.
# Fast path — push updated own conf to all partners' /tmp/.cache/vv/d/ only.
# No pulls, no local cache rebuild.
#
# Cache is /tmp (tmpfs) — cleared every reboot, repopulated by this script
# on next array start. Scripts source from cache for partner vars; own vars
# always come from disk (load_config.sh skips cached copy of own conf).
# Cache is /tmp/.cache/vv/d (tmpfs) — cleared every reboot, repopulated by
# this script on next array start. Scripts source from cache for partner vars;
# own vars always come from disk (load_config.sh skips cached copy of own conf).
#
# Pull path resolves the remote's SCRIPTS_DIR from their varaverk.cfg so it
# works whether the remote is in internal or appdata storage mode.
#
# ==============================================================================================
# RUNTIME MODES
@@ -56,9 +59,25 @@ if [[ "${CONF_SYNC_ENABLED:-true}" == false ]]; then
fi
CACHE_DIR="/tmp/.cache/vv/d"
MY_CONF="$SCRIPTS_ROOT/Configurations/${MY_ID,,}.conf"
MY_CONF="$SCRIPTS_DIR/Configurations/${MY_ID,,}.conf"
SSH_TIMEOUT=10
# Reads the remote's varaverk.cfg to find their actual SCRIPTS_DIR.
# Handles the case where the remote is in appdata storage mode.
_remote_scripts_dir() {
local ip="$1"
local cfg line sd
cfg=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
"root@${ip}" "cat /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null" 2>/dev/null) || true
while IFS= read -r line; do
[[ "$line" == SCRIPTS_DIR=* ]] || continue
sd="${line#SCRIPTS_DIR=}"; sd="${sd//\"/}"; sd="${sd//\'/}"
echo "$sd"; return
done <<< "$cfg"
echo "/boot/config/plugins/varaverk"
}
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
# ── Ensure cache dir exists ───────────────────────────────────────────────────
@@ -101,7 +120,8 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
# ── Pull: grab partner's conf from their disk → our local cache ──────────
if [[ "$PUSH_ONLY" == false ]]; then
remote_conf="${SCRIPTS_DIR}/Configurations/${partner_slot}.conf"
remote_sd=$(_remote_scripts_dir "$partner_ip")
remote_conf="${remote_sd}/Configurations/${partner_slot}.conf"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would pull $partner_host:$remote_conf → $CACHE_DIR/${partner_slot}.conf"
elif timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
@@ -116,7 +136,7 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
fi
fi
# ── Push: send own conf to partner's /tmp/.vv/ cache ────────────────────
# ── Push: send own conf to partner's /tmp/.cache/vv/d/ cache ───────────
if [[ "$PULL_ONLY" == true ]]; then
continue
fi