Varaverk/Plugin/unraid/System_Essentials/README-System_Essentials.md

# ━━━━━ PLUGIN / UNRAID / SYSTEM ESSENTIALS ━━━━━

Four scripts that touch Unraid subsystems directly. They live here rather than in the
top-level `System_Essentials/` because each one manipulates something that only exists on
Unraid — the mover, the User Scripts plugin, the PHP-FPM pool serving emhttp, and the
unraid-api service registry.

There is no platform-agnostic version of any of them. On TrueNAS there is no mover to stop.

| Script | Runs | Role |
|--------|------|------|
| `php_fpm_max_children.sh` | Array **start** | Raise the PHP-FPM worker ceiling so the WebGUI stays responsive |
| `unraid_api_key_renew.sh` | Array **start** | Re-register Varaverk's key in the ephemeral unraid-api registry |
| `user_scripts_stop.sh` | Array **stop** | Stop User Scripts processes before the array goes down |
| `mover_stop.sh` | Array **stop** | Stop the mover gracefully before rsync or reboot |

All four are wired through `ARRAY_START_SCRIPTS` / `ARRAY_STOP_SCRIPTS` in `master.conf` —
none are on a timer.

---

## ━━━ WHY EACH ONE EXISTS ━━━

### 🐘  `php_fpm_max_children.sh` — the WebGUI gets slow, not broken

Unraid's WebGUI runs through PHP-FPM, and the stock `pm.max_children` is very low (4–8). Under
real load — several browser tabs polling, Docker operations running, a dashboard open — every
worker saturates and new requests queue. The WebGUI becomes slow or stops answering, while
nothing is actually wrong with the server.

Runs at array start because the setting **does not survive an Unraid update** — the OS
replaces the pool config. Idempotent: already at the target value means no write and no
restart, so a clean boot is silent.

### 🔑  `unraid_api_key_renew.sh` — the registry is ephemeral

Varaverk's enhanced monitoring authenticates against unraid-api. That registry is cleared by
OS updates and service restarts, so a key that worked yesterday can simply be gone.

Re-registers unconditionally every array start rather than only when missing, because the
failure it repairs is precisely *a key present in the conf but absent from the registry* —
checking the conf would not detect it. Writes the resulting key into `HOST*_UNRAID_API_KEY`.

### 🛑  `user_scripts_stop.sh` — stop new work before shutting down

The User Scripts plugin spawns background processes that Varaverk does not own. During a
shutdown sequence those can still be starting operations while everything else is being torn
down.

Runs **first** in `ARRAY_STOP_SCRIPTS` for that reason — stop new work before stopping the
things it would work on. Reports each process by **script name**, not just PID, because
"stopping 4 processes" tells an operator nothing they can act on.

Matching is scoped strictly to processes the User Scripts plugin spawned. A broad pattern
would catch Varaverk's own scripts — including, during a reboot, the very script doing the
stopping.

### 💾  `mover_stop.sh` — mover and rsync must not overlap

The mover relocates files between cache and array. rsync reads those same paths. Both running
at once can produce a corrupt or half-moved snapshot on the remote.

Ordering in the stop sequence is deliberate: `rsync_stop.sh` runs **before** `mover_stop.sh`,
because both write to the same paths and stopping the reader first is the safer order.

Sends a wall warning first, then SIGTERM with a configurable window, then SIGKILL only if
ignored. The mover is mid-file-move by definition — giving it the chance to finish the current
file is the difference between a stopped transfer and a half-moved file.

An absent mover exits 0. Callers use this as a precondition ("ensure the mover is not
running"), so treating "already stopped" as failure would abort every reboot on a quiet
system.

---

## ━━━ WHERE THEY SIT IN THE LIFECYCLE ━━━

```
Array starting
    │
    ├── php_fpm_max_children.sh ──── WebGUI can handle the load that's about to arrive
    └── unraid_api_key_renew.sh ──── monitoring can authenticate again

        ... normal operation ...

Array stopping
    │
    ├── user_scripts_stop.sh ─────── stop new work first
    ├── fallback.sh --stop
    ├── rsync_stop.sh ────────────── stop the reader
    ├── mover_stop.sh ────────────── then the writer — same paths
    └── docker_container_stop.sh ─── containers last, verified one at a time
```

`mover_stop.sh` is also called directly by `server_reboot.sh`, and both stop scripts appear in
the User Scripts master template for manual use.

---

## ━━━ SAFEGUARDS ━━━

All four enforce root, take a lock, and support `--dry-run`. Specific to this group:

**Graceful before forced.** `mover_stop.sh` and `user_scripts_stop.sh` both use SIGTERM with a
window before SIGKILL. Neither kills first.

**Absent target is success.** No mover running, no User Scripts running — both exit 0. These
are preconditions, not commands that must find something to do.

**Idempotent where it matters.** `php_fpm_max_children.sh` writes and restarts only when the
value is actually wrong, so array start does not restart PHP-FPM every single boot.

**Targeted process matching.** `user_scripts_stop.sh` matches only what the User Scripts
plugin spawned — see above for why a broad pattern is dangerous here specifically.

**Stop only, never start.** Neither stop script has a counterpart that restarts what it
stopped. Unraid's own schedule owns when the mover runs; these only remove it from the picture
for a window.

---

## ━━━ CONFIGURATION ━━━

| Variable | Used by | Purpose |
|----------|---------|---------|
| `HOST*_UNRAID_API_KEY` | `unraid_api_key_renew.sh` | Written every array start — output, not input |
| `MOVER_STOP_TIMEOUT` | `mover_stop.sh` | SIGTERM grace window before escalating |
| `PHP_MAX_CHILDREN` | `php_fpm_max_children.sh` | Target worker ceiling |

See each script's `CONFIGURATION` header section for the authoritative list — and
`Deployment/master.conf.template`, which is the versioned schema for all of them.