FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Storage-mode awareness pass + doc update for System_Essentials through Partnership

Browse Source 
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.

PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.

conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.

fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.

README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
This commit is contained in:
Gmer4Lfe
2026-06-19 19:32:39 -04:00
parent 0564580605
commit bf3e7cc2c4
35 changed files with 835 additions and 489 deletions
Download Patch File Download Diff File Expand all files Collapse all files
System_Essentials/README-System_Essentials.md
+79 -87
View File
@@ -1,35 +1,18 @@
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
**System-level scripts that act on the server itself — not containers,
not media, not monitoring.** Keeping the server stable under load, recovering a
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
shutdowns with proper warning sequences.
not media, not monitoring.** Kernel limits, log hygiene, conf synchronisation
between servers, and graceful shutdowns with proper warning sequences.
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) live in `Plugin/unraid/System_Essentials/`
> because they call Unraid-specific service commands and paths. This folder
> contains scripts that would run unchanged on any Linux host.
---
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
**Server Getting Into Unstable States With No Recovery Path**
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
watchdog tries to restart the container — but Docker itself is barely responding.
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
write log files. The server is functionally dead but still technically running.
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
threshold breaches. Last line of defense before a hard crash.
**WebGUI Freezing and Nobody Noticing**
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
load, settings don't save, containers can't be started or stopped via the UI. No
container-level alert fires because this isn't a container problem — it's a web
server problem. By the time someone notices it may have been broken for hours.
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
php-fpm → emhttp. Lightest fix first. Silent when healthy.
**50+ Containers Starting and Filling Syslog With Veth Noise**
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
network interface for each one. Each interface generates multiple syslog entries.
@@ -40,68 +23,79 @@ In the first minute after array start, syslog is buried under 200–400 lines of
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
starts. Applied at array start. Idempotent — silent when already correct.
**WebGUI Queuing Requests Under Load Without Explanation**
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
those workers saturate immediately. New requests queue behind active ones.
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
**inotify Exhaustion Producing Unexplained Failures**
When inotify limits are exhausted, containers silently stop receiving filesystem
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
and all containers share the same pool.
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
run FIRST in ARRAY_START_SCRIPTS before any containers start.
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start, before
any container-connecting or continuous scripts start.
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
The mover is running — moving a large batch of files from cache to array. A reboot
is triggered. The mover stops mid-file. The file exists partially on both cache and
array simultaneously. unRAID's deduplication layer is confused.
**Partner Conf Lost Across Reboots When Partner is Down**
Scripts like `fallback.sh` need the partner's conf vars (credentials, container
names, tier delays) to operate. The partner conf lives in a RAM cache at
`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls
a fresh copy from the partner. But if the partner is offline at boot time, the
pull fails and fallback has no partner vars to work with.
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
Fix: `conf_cache_save.sh` + `conf_cache_restore.sh` — snapshot the RAM cache to
`$PERSISTENT_CONF_CACHE` on array stop; reload on next start for any confs the
sync couldn't fetch.
**Mover/Rsync Interruption During Reboot**
Rsync transfers or mover runs are in progress when a reboot is triggered. Stopping
them uncleanly leaves partial files.
Fix: `server_reboot.sh` — orchestrates `array_stopping.sh` which stops rsync, mover,
and containers in the correct order before calling `/sbin/reboot`.
---
## ━━━ WHAT THIS FOLDER DOES ━━━
```
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
Kernel tuning inotify_tuning.sh — file watch limits
php_fpm_max_children.sh — PHP worker count
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
clear_logs.sh — weekly log trimming
Graceful operations mover_stop.sh — clean mover stop
rsync_stop.sh — smart rsync stop (orchestrator-aware)
user_scripts_stop.sh — stop running User Scripts
clear_logs.sh — size-threshold log trimming
Conf synchronisation conf_sync.sh — pull/push partner confs → RAM cache
conf_cache_save.sh — snapshot RAM cache → persistent at stop
conf_cache_restore.sh — reload from snapshot at start (offline partner)
Graceful operations rsync_stop.sh — smart rsync stop (orchestrator-aware)
server_reboot.sh — clean reboot with pre-flight warnings
```
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
---
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
```
Orchestrators/
array_started.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
─────────────────────────► docker_syslog_filter.sh (second)
─────────────────────────► php_fpm_max_children.sh
array_started.sh ────────────────────────► conf_sync.sh (pulls partner confs)
────────────────────────► conf_cache_restore.sh (fills gaps if partner down)
────────────────────────► docker_syslog_filter.sh (before containers start)
────────────────────────► inotify_tuning.sh (before docker_network_connect)
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
weekly_maintenance.sh ──────────────────► clear_logs.sh
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
Watchdogs/System/
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
(runs every 15 min via watchdog_orchestrator.sh)
Watchdogs/
stability_watchdog.sh and resource_watchdog.sh now live here.
See Watchdogs/README-Watchdogs.md for how they relate to each other
and to docker_watchdog.sh and System/storage_watchdog.sh.
System_Essentials/
server_reboot.sh ────────────────────────► array_stopping.sh (via Orchestrators/)
────────────────────────► mover_stop.sh, user_scripts_stop.sh
(via Plugin/unraid/System_Essentials/)
Plugin/unraid/System_Essentials/
php_fpm_max_children.sh — WebGUI tuning (Unraid-specific PHP paths)
mover_stop.sh — clean mover stop (Unraid mover daemon)
user_scripts_stop.sh — stop User Scripts plugin processes
unraid_api_key_renew.sh — Varaverk plugin API key renewal
Plugin/unraid/Watchdogs/System/
webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
```
---
@@ -110,14 +104,13 @@ Watchdogs/
| Script | Role | When It Runs |
|--------|------|-------------|
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — before container ops |
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
---
@@ -125,27 +118,28 @@ Watchdogs/
## ━━━ HOW THE SCRIPTS RELATE ━━━
```
Array starts
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
│
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
└─ php_fpm_max_children.sh ← before WebGUI is under load
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
│ push own conf to partner's /tmp/.cache/vv/d/
├─ conf_cache_restore.sh ← if partner was down: load last-known-good conf from
│ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live)
└─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts
Every minute (watchdog_orchestrator.sh in Orchestrators/):
→ Watchdogs/resource_watchdog.sh
→ Watchdogs/docker_watchdog.sh
→ Watchdogs/system_watchdog.sh (thin orchestrator)
└─ Watchdogs/System/storage_watchdog.sh
└─ Watchdogs/System/webgui_watchdog.sh
WebGUI OK → silent exit
Not responding:
Step 1: restart nginx → recheck
Step 2: restart php-fpm → recheck
Step 3: restart emhttp → recheck
All failed → notify, exit 1
└─ Watchdogs/System/network_watchdog.sh
→ Watchdogs/stability_watchdog.sh
(see Watchdogs/README-Watchdogs.md for full flow)
Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
│
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
│ while RAM cache is still fresh
├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
└─ ...other stop scripts...
Every 15 minutes (watchdog_orchestrator.sh):
→ Watchdogs/System/conf_cache_watchdog.sh
If partner is offline and persistent backup is stale → refresh from last RAM cache
Every 4 hours (intermediate_sync_maintenance.sh):
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
Weekly (weekly_maintenance.sh):
└─ clear_logs.sh
@@ -153,8 +147,6 @@ Weekly (weekly_maintenance.sh):
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
Manual operations:
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
server_reboot.sh → pre-flight → array_stopping.sh → reboot
```