Varaverk/Plugin/unraid/System_Essentials/README-System_Essentials.md

━━━━━ PLUGIN / UNRAID / SYSTEM ESSENTIALS ━━━━━

Four scripts that touch Unraid subsystems directly. They live here rather than in the top-level System_Essentials/ because each one manipulates something that only exists on Unraid — the mover, the User Scripts plugin, the PHP-FPM pool serving emhttp, and the unraid-api service registry.

There is no platform-agnostic version of any of them. On TrueNAS there is no mover to stop.

Script	Runs	Role
php_fpm_max_children.sh	Array start	Raise the PHP-FPM worker ceiling so the WebGUI stays responsive
unraid_api_key_renew.sh	Array start + every 15 min	Re-register Varaverk's key in the ephemeral unraid-api registry
user_scripts_stop.sh	Array stop	Stop User Scripts processes before the array goes down
mover_stop.sh	Array stop	Stop the mover gracefully before rsync or reboot

All four are wired through ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS in master.conf.

unraid_api_key_renew.sh is additionally listed in WATCHDOG_ORCHESTRATOR_SCRIPTS, so it also re-runs every 15 minutes. That is deliberate: the registry it repairs can be cleared by a service restart at any time, not only across a reboot, and the script is idempotent.

---

━━━ WHY EACH ONE EXISTS ━━━

🐘 php_fpm_max_children.sh — the WebGUI gets slow, not broken

Unraid's WebGUI runs through PHP-FPM, and the stock pm.max_children is very low (4–8). Under real load — several browser tabs polling, Docker operations running, a dashboard open — every worker saturates and new requests queue. The WebGUI becomes slow or stops answering, while nothing is actually wrong with the server.

Runs at array start because the setting does not survive an Unraid update — the OS replaces the pool config. Idempotent: already at the target value means no write and no restart, so a clean boot is silent.

🔑 unraid_api_key_renew.sh — the registry is ephemeral

Varaverk's enhanced monitoring authenticates against unraid-api. That registry is cleared by OS updates and service restarts, so a key that worked yesterday can simply be gone.

Re-registers unconditionally every array start rather than only when missing, because the failure it repairs is precisely a key present in the conf but absent from the registry — checking the conf would not detect it. Writes the resulting key into HOST*_UNRAID_API_KEY.

🛑 user_scripts_stop.sh — stop new work before shutting down

The User Scripts plugin spawns background processes that Varaverk does not own. During a shutdown sequence those can still be starting operations while everything else is being torn down.

Runs first in ARRAY_STOP_SCRIPTS for that reason — stop new work before stopping the things it would work on. Reports each process by script name, not just PID, because "stopping 4 processes" tells an operator nothing they can act on.

Matching is scoped strictly to processes the User Scripts plugin spawned. A broad pattern would catch Varaverk's own scripts — including, during a reboot, the very script doing the stopping.

💾 mover_stop.sh — mover and rsync must not overlap

The mover relocates files between cache and array. rsync reads those same paths. Both running at once can produce a corrupt or half-moved snapshot on the remote.

Ordering in the stop sequence is deliberate: rsync_stop.sh runs before mover_stop.sh, because both write to the same paths and stopping the reader first is the safer order.

Sends a wall warning first, then SIGTERM with a configurable window, then SIGKILL only if ignored. The mover is mid-file-move by definition — giving it the chance to finish the current file is the difference between a stopped transfer and a half-moved file.

An absent mover exits 0. Callers use this as a precondition ("ensure the mover is not running"), so treating "already stopped" as failure would abort every reboot on a quiet system.

---

━━━ WHERE THEY SIT IN THE LIFECYCLE ━━━

Array starting
    │
    ├── php_fpm_max_children.sh ──── WebGUI can handle the load that's about to arrive
    └── unraid_api_key_renew.sh ──── monitoring can authenticate again

        ... normal operation ...

Array stopping
    │
    ├── user_scripts_stop.sh ─────── stop new work first
    ├── fallback.sh --stop
    ├── rsync_stop.sh ────────────── stop the reader
    ├── mover_stop.sh ────────────── then the writer — same paths
    └── docker_container_stop.sh ─── containers last, verified one at a time

mover_stop.sh is also called directly by server_reboot.sh, and both stop scripts appear in the User Scripts master template for manual use.

---

━━━ SAFEGUARDS ━━━

All four enforce root, take a lock, and support --dry-run. Specific to this group:

Graceful before forced. mover_stop.sh and user_scripts_stop.sh both use SIGTERM with a window before SIGKILL. Neither kills first.

Absent target is success. No mover running, no User Scripts running — both exit 0. These are preconditions, not commands that must find something to do.

Idempotent where it matters. php_fpm_max_children.sh writes and restarts only when the value is actually wrong, so array start does not restart PHP-FPM every single boot.

Targeted process matching. user_scripts_stop.sh matches only what the User Scripts plugin spawned — see above for why a broad pattern is dangerous here specifically.

Stop only, never start. Neither stop script has a counterpart that restarts what it stopped. Unraid's own schedule owns when the mover runs; these only remove it from the picture for a window.

---

━━━ CONFIGURATION ━━━

Variable	Used by	Purpose
HOST*_UNRAID_API_KEY	unraid_api_key_renew.sh	Written at array start and every 15 min — output, not input
MOVER_STOP_TIMEOUT	mover_stop.sh	SIGTERM grace window before escalating
PHP_MAX_CHILDREN	php_fpm_max_children.sh	Target worker ceiling

See each script's CONFIGURATION header section for the authoritative list — and Deployment/master.conf.template, which is the versioned schema for all of them.