Gmer4Lfe
bf3e7cc2c4
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR / PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the ecosystem works in both internal and appdata storage modes. PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php, config.php): all state reads switched to STATE_DIR constant; remote state reads use the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via their varaverk.cfg before building the path. conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added _remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull. fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED, FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card. README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync, Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync, conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener, upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to $STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js, not PHP -S).
7.9 KiB
━━━━━ SYSTEM ESSENTIALS ━━━━━
System-level scripts that act on the server itself — not containers, not media, not monitoring. Kernel limits, log hygiene, conf synchronisation between servers, and graceful shutdowns with proper warning sequences.
Platform-specific scripts (
webgui_watchdog.sh,php_fpm_max_children.sh,mover_stop.sh,user_scripts_stop.sh) live inPlugin/unraid/System_Essentials/because they call Unraid-specific service commands and paths. This folder contains scripts that would run unchanged on any Linux host.
━━━ THE PROBLEM THAT BUILT THIS ━━━
50+ Containers Starting and Filling Syslog With Veth Noise
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
network interface for each one. Each interface generates multiple syslog entries.
In the first minute after array start, syslog is buried under 200–400 lines of
veth renamed from eth0 and docker0: port entered forwarding state. Real events
— a failed mount, a permission error, a service that didn't start — are invisible.
Fix: docker_syslog_filter.sh — creates an rsyslog drop rule before any container
starts. Applied at array start. Idempotent — silent when already correct.
inotify Exhaustion Producing Unexplained Failures When inotify limits are exhausted, containers silently stop receiving filesystem events. Arrs don't detect completed downloads. VSCode shows "unable to watch for file changes." Code-Server with node_modules alone can consume 100K–200K watches, and all containers share the same pool.
Fix: inotify_tuning.sh — raises all three inotify limits at array start, before
any container-connecting or continuous scripts start.
Partner Conf Lost Across Reboots When Partner is Down
Scripts like fallback.sh need the partner's conf vars (credentials, container
names, tier delays) to operate. The partner conf lives in a RAM cache at
/tmp/.cache/vv/d/ — wiped every reboot. At array start, conf_sync.sh pulls
a fresh copy from the partner. But if the partner is offline at boot time, the
pull fails and fallback has no partner vars to work with.
Fix: conf_cache_save.sh + conf_cache_restore.sh — snapshot the RAM cache to
$PERSISTENT_CONF_CACHE on array stop; reload on next start for any confs the
sync couldn't fetch.
Mover/Rsync Interruption During Reboot Rsync transfers or mover runs are in progress when a reboot is triggered. Stopping them uncleanly leaves partial files.
Fix: server_reboot.sh — orchestrates array_stopping.sh which stops rsync, mover,
and containers in the correct order before calling /sbin/reboot.
━━━ WHAT THIS FOLDER DOES ━━━
Kernel tuning inotify_tuning.sh — file watch limits
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
clear_logs.sh — size-threshold log trimming
Conf synchronisation conf_sync.sh — pull/push partner confs → RAM cache
conf_cache_save.sh — snapshot RAM cache → persistent at stop
conf_cache_restore.sh — reload from snapshot at start (offline partner)
Graceful operations rsync_stop.sh — smart rsync stop (orchestrator-aware)
server_reboot.sh — clean reboot with pre-flight warnings
━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
Orchestrators/
array_started.sh ────────────────────────► conf_sync.sh (pulls partner confs)
────────────────────────► conf_cache_restore.sh (fills gaps if partner down)
────────────────────────► docker_syslog_filter.sh (before containers start)
────────────────────────► inotify_tuning.sh (before docker_network_connect)
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
weekly_maintenance.sh ──────────────────► clear_logs.sh
Watchdogs/System/
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
(runs every 15 min via watchdog_orchestrator.sh)
System_Essentials/
server_reboot.sh ────────────────────────► array_stopping.sh (via Orchestrators/)
────────────────────────► mover_stop.sh, user_scripts_stop.sh
(via Plugin/unraid/System_Essentials/)
Plugin/unraid/System_Essentials/
php_fpm_max_children.sh — WebGUI tuning (Unraid-specific PHP paths)
mover_stop.sh — clean mover stop (Unraid mover daemon)
user_scripts_stop.sh — stop User Scripts plugin processes
unraid_api_key_renew.sh — Varaverk plugin API key renewal
Plugin/unraid/Watchdogs/System/
webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
━━━ SCRIPTS IN THIS FOLDER ━━━
| Script | Role | When It Runs |
|---|---|---|
inotify_tuning.sh |
Raise inotify kernel limits | At array start — before container ops |
docker_syslog_filter.sh |
Suppress Docker veth syslog noise | At array start — before containers |
conf_sync.sh |
Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
conf_cache_save.sh |
Snapshot partner RAM cache → $PERSISTENT_CONF_CACHE |
At array stop — first step |
conf_cache_restore.sh |
Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
clear_logs.sh |
Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
rsync_stop.sh |
Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
server_reboot.sh |
Graceful reboot with pre-flight warnings | Manual |
━━━ HOW THE SCRIPTS RELATE ━━━
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
│
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
│ push own conf to partner's /tmp/.cache/vv/d/
├─ conf_cache_restore.sh ← if partner was down: load last-known-good conf from
│ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live)
└─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts
Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
│
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
│ while RAM cache is still fresh
├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
└─ ...other stop scripts...
Every 15 minutes (watchdog_orchestrator.sh):
→ Watchdogs/System/conf_cache_watchdog.sh
If partner is offline and persistent backup is stale → refresh from last RAM cache
Every 4 hours (intermediate_sync_maintenance.sh):
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
Weekly (weekly_maintenance.sh):
└─ clear_logs.sh
System logs: clear if > LOG_MIN_SIZE_MB
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
Manual operations:
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
server_reboot.sh → pre-flight → array_stopping.sh → reboot