Gmer4Lfe
bf3e7cc2c4
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR / PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the ecosystem works in both internal and appdata storage modes. PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php, config.php): all state reads switched to STATE_DIR constant; remote state reads use the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via their varaverk.cfg before building the path. conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added _remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull. fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED, FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card. README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync, Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync, conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener, upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to $STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js, not PHP -S).
153 lines
7.9 KiB
Markdown
153 lines
7.9 KiB
Markdown
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
|
||
|
||
**System-level scripts that act on the server itself — not containers,
|
||
not media, not monitoring.** Kernel limits, log hygiene, conf synchronisation
|
||
between servers, and graceful shutdowns with proper warning sequences.
|
||
|
||
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
|
||
> `mover_stop.sh`, `user_scripts_stop.sh`) live in `Plugin/unraid/System_Essentials/`
|
||
> because they call Unraid-specific service commands and paths. This folder
|
||
> contains scripts that would run unchanged on any Linux host.
|
||
|
||
---
|
||
|
||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||
|
||
**50+ Containers Starting and Filling Syslog With Veth Noise**
|
||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
|
||
network interface for each one. Each interface generates multiple syslog entries.
|
||
In the first minute after array start, syslog is buried under 200–400 lines of
|
||
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
|
||
— a failed mount, a permission error, a service that didn't start — are invisible.
|
||
|
||
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
|
||
starts. Applied at array start. Idempotent — silent when already correct.
|
||
|
||
**inotify Exhaustion Producing Unexplained Failures**
|
||
When inotify limits are exhausted, containers silently stop receiving filesystem
|
||
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
|
||
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
|
||
and all containers share the same pool.
|
||
|
||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start, before
|
||
any container-connecting or continuous scripts start.
|
||
|
||
**Partner Conf Lost Across Reboots When Partner is Down**
|
||
Scripts like `fallback.sh` need the partner's conf vars (credentials, container
|
||
names, tier delays) to operate. The partner conf lives in a RAM cache at
|
||
`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls
|
||
a fresh copy from the partner. But if the partner is offline at boot time, the
|
||
pull fails and fallback has no partner vars to work with.
|
||
|
||
Fix: `conf_cache_save.sh` + `conf_cache_restore.sh` — snapshot the RAM cache to
|
||
`$PERSISTENT_CONF_CACHE` on array stop; reload on next start for any confs the
|
||
sync couldn't fetch.
|
||
|
||
**Mover/Rsync Interruption During Reboot**
|
||
Rsync transfers or mover runs are in progress when a reboot is triggered. Stopping
|
||
them uncleanly leaves partial files.
|
||
|
||
Fix: `server_reboot.sh` — orchestrates `array_stopping.sh` which stops rsync, mover,
|
||
and containers in the correct order before calling `/sbin/reboot`.
|
||
|
||
---
|
||
|
||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||
|
||
```
|
||
Kernel tuning inotify_tuning.sh — file watch limits
|
||
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
|
||
clear_logs.sh — size-threshold log trimming
|
||
Conf synchronisation conf_sync.sh — pull/push partner confs → RAM cache
|
||
conf_cache_save.sh — snapshot RAM cache → persistent at stop
|
||
conf_cache_restore.sh — reload from snapshot at start (offline partner)
|
||
Graceful operations rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||
server_reboot.sh — clean reboot with pre-flight warnings
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||
|
||
```
|
||
Orchestrators/
|
||
array_started.sh ────────────────────────► conf_sync.sh (pulls partner confs)
|
||
────────────────────────► conf_cache_restore.sh (fills gaps if partner down)
|
||
────────────────────────► docker_syslog_filter.sh (before containers start)
|
||
────────────────────────► inotify_tuning.sh (before docker_network_connect)
|
||
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
|
||
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
|
||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||
|
||
Watchdogs/System/
|
||
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
|
||
(runs every 15 min via watchdog_orchestrator.sh)
|
||
|
||
System_Essentials/
|
||
server_reboot.sh ────────────────────────► array_stopping.sh (via Orchestrators/)
|
||
────────────────────────► mover_stop.sh, user_scripts_stop.sh
|
||
(via Plugin/unraid/System_Essentials/)
|
||
|
||
Plugin/unraid/System_Essentials/
|
||
php_fpm_max_children.sh — WebGUI tuning (Unraid-specific PHP paths)
|
||
mover_stop.sh — clean mover stop (Unraid mover daemon)
|
||
user_scripts_stop.sh — stop User Scripts plugin processes
|
||
unraid_api_key_renew.sh — Varaverk plugin API key renewal
|
||
|
||
Plugin/unraid/Watchdogs/System/
|
||
webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||
|
||
| Script | Role | When It Runs |
|
||
|--------|------|-------------|
|
||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — before container ops |
|
||
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
|
||
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
|
||
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
|
||
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
|
||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
|
||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||
|
||
---
|
||
|
||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||
|
||
```
|
||
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
|
||
│
|
||
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
|
||
│ push own conf to partner's /tmp/.cache/vv/d/
|
||
├─ conf_cache_restore.sh ← if partner was down: load last-known-good conf from
|
||
│ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
|
||
├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live)
|
||
└─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts
|
||
|
||
Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
|
||
│
|
||
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
|
||
│ while RAM cache is still fresh
|
||
├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
|
||
└─ ...other stop scripts...
|
||
|
||
Every 15 minutes (watchdog_orchestrator.sh):
|
||
→ Watchdogs/System/conf_cache_watchdog.sh
|
||
If partner is offline and persistent backup is stale → refresh from last RAM cache
|
||
|
||
Every 4 hours (intermediate_sync_maintenance.sh):
|
||
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
|
||
|
||
Weekly (weekly_maintenance.sh):
|
||
└─ clear_logs.sh
|
||
System logs: clear if > LOG_MIN_SIZE_MB
|
||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||
|
||
Manual operations:
|
||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||
server_reboot.sh → pre-flight → array_stopping.sh → reboot
|
||
```
|