Document the Unraid platform layer

The folder that translates the OS into Varaverk's vocabulary had no docs of its own, so the adapter contract and the three-layer web UI were only discoverable by reading the code.
2026-08-01 23:20:52 -04:00
parent ab169a6f48
commit 1ed6b92118
3 changed files with 436 additions and 0 deletions
@@ -0,0 +1,170 @@
 # ━━━━━ PLUGIN / UNRAID ━━━━━
 The Unraid platform layer. Everything in this folder exists to translate one specific
 operating system into the vocabulary Varaverk speaks — and to put a face on it in the Unraid
 WebGUI.
 Varaverk itself knows nothing about Unraid. No script outside this folder references
 `/etc/rc.d`, `/boot/config/plugins/dynamix`, `emhttp`, the mover, or a `.page` file. They call
 `platform_*()` and read `$STATE_DIR`; this folder decides what those mean on Unraid.
 ```
  adapter.sh              24 platform_*() functions — the entire OS contract
  Varaverk.page           WebGUI menu entry (Tasks:95)
  VaraverkSettings.page   WebGUI settings entry
  event/                  Unraid array lifecycle hooks
  pages/ api/ include/    the three-layer web UI
  System_Essentials/      Unraid-specific system scripts
  Tools/                  Unraid-specific operator tools
  Watchdogs/System/       Unraid-specific watchdog (WebGUI health)
  Partnership/            Unraid-specific container deploy/cleanup
 ```
 > Swapping platforms means writing `Plugin/truenas/adapter.sh` implementing the same 24
 > function names. `load_config.sh` sources `Plugin/$PLATFORM/adapter.sh` — nothing else in
 > the codebase changes.
 ---
 ## ━━━ THE PROBLEM THAT BUILT THIS ━━━
 **Every NAS OS Answers the Same Questions Differently**
 "Is the array mounted?" "Restart this service." "What are the disk temperature thresholds?"
 "Send a notification." Every platform has an answer, and every answer is different — a
 different path, a different init system, a different config format.
 Scattering those differences through 115 scripts would mean every script carrying an `if
 unraid ... elif truenas` branch, and every new platform touching all of them.
 **Varaverk Vars Are Not OS Vars**
 Varaverk works in terms of `STATE_DIR`, `SCRIPTS_DIR`, `DOCKER_APPDATA_BASE`,
 `WATCHDOG_CONTAINERS`. Unraid works in terms of `/boot/config/plugins/`, `emhttp`,
 `/mnt/user`, dockerMan XML templates, and `dynamix.cfg`. Something has to map one onto the
 other, in exactly one place, or the two vocabularies leak into each other.
 **A Headless Ecosystem Is Hard to Trust**
 115 scripts running unattended produce a lot of state nobody can see. Without a UI the only
 way to know what the watchdogs did last night is to read logs. The WebGUI exists so the
 system can be inspected without SSH.
 ---
 ## ━━━ THE ADAPTER — WHERE OS MEETS VARAVERK ━━━
 `adapter.sh` is the whole contract. 24 functions, sourced by `load_config.sh` after
 `common.sh`, so every script in the ecosystem has them.
 The most-called ones, by usage across the repo:
 | Function | Calls | What it hides |
 |----------|-------|---------------|
 | `platform_require_cmd` | 28 | Verifying a platform binary exists and is executable |
 | `platform_restart_service` | 13 | `/etc/rc.d/rc.<name>` init scripts |
 | `platform_is_service_running` | 13 | Unraid's service process conventions |
 | `platform_is_mover_running` | 10 | The Unraid mover — no other OS has one |
 | `platform_get_os_version` | 10 | Where the version string lives |
 | `platform_setup_db_path` | 9 | Where wizard/setup state persists |
 | `platform_storage_path` | 7 | `/mnt/user` as the user-share root |
 Plus: disk states and temperature thresholds from `dynamix.cfg`, parity/maintenance detection,
 native notifications through the dynamix notify script, container rebuild from dockerMan XML
 templates, and the WebGUI install path.
 **The contract is strict, and it is what makes the layer work:**
 - **Every function returns 0/1 and never calls `exit`.** A missing capability is information
  the caller needs, not a decision the adapter makes — a watchdog may want to skip where an
  installer wants to abort.
 - **Value-producing functions write to stdout**, captured with `$()`. Status travels in the
  exit code. Keeping those separate is what lets callers use them in conditionals.
 - **It reports; it does not remediate.** No retries, no escalation, no notification policy.
  Burying those here would make identical calls behave differently per platform.
 ---
 ## ━━━ THE WEB UI — THREE LAYERS ━━━
 ```
  pages/*.php      what you look at        ── requires ──►  include/
  api/*.php        what the page fetches   ── requires ──►  include/
  include/*.php    the actual logic — vv_*() functions, shared by both
 ```
 The split matters: a page and its API endpoint call **the same** `vv_*()` builders, so a
 cached response and a live one cannot disagree in shape. Adding a metric means adding it in
 `include/` once, and both the page and the endpoint get it.
 | Layer | Files | Role |
 |-------|-------|------|
 | `pages/` | 11 | One per WebGUI tab — monitor, docker, arrs, fallback, watchdog, rsync, partnership, scheduler, auth, settings, setup |
 | `api/` | 50 | JSON endpoints the pages poll, plus action endpoints (run a script, stop a job, toggle a flag) |
 | `include/` | 16 | Shared builders and helpers — `vv_monitor_*`, `vv_arrs_*`, `vv_docker_*`, config read/write, auth |
 **Caching.** Several endpoints serve from `/tmp/vv_cache` (tmpfs) rather than hitting live
 APIs on every page view, refreshed by `Tools/api_cache_writer.sh`. `?live=1` bypasses the
 cache. A missing or unparseable cache always falls back to a live call, so the cache can never
 be the reason a page fails to load.
 ---
 ## ━━━ ARRAY LIFECYCLE HOOKS ━━━
 `event/` plugs Varaverk into Unraid's own array lifecycle. These are how the ecosystem starts
 and stops with the array rather than with the OS.
 | Hook | Fires | Runs |
 |------|-------|------|
 | `disks_mounted/array_start_jobs` | Array started | `Orchestrators/array_started.sh` → `ARRAY_START_SCRIPTS` |
 | `disks_mounted/rebuild_cron` | Array started | Regenerates `varaverk.cron` from the scheduler config |
 | `unmounting_disks/array_stop_jobs` | Array stopping | `Orchestrators/array_stopping.sh` → `ARRAY_STOP_SCRIPTS` |
 Array-start rather than boot is the correct trigger: in flash storage mode `SCRIPTS_DIR` lives
 in appdata, which does not exist until the array mounts.
 ---
 ## ━━━ THE PLATFORM-SPECIFIC SCRIPT MIRRORS ━━━
 These folders mirror the top-level Varaverk structure, but hold work that **cannot** be
 platform-agnostic. A script lives here rather than in the root folder when it manipulates
 Unraid itself.
 | Folder | Contents | Why it is platform-specific |
 |--------|----------|----------------------------|
 | `System_Essentials/` | 4 scripts | Mover, User Scripts plugin, PHP-FPM pool, unraid-api registry — all Unraid subsystems |
 | `Tools/` | 5 files | Share `.cfg` recreation, storage-mode migration, API cache writers |
 | `Watchdogs/System/` | `webgui_watchdog.sh` | emhttp / nginx / php-fpm health — there is no generic "WebGUI" |
 | `Partnership/` | `containers.sh` | Container deploy from dockerMan CA XML templates |
 Each has its own README where the group is large enough to need one; single-script folders are
 documented in the script header.
 ---
 ## ━━━ HOW THIS FOLDER IS INSTALLED ━━━
 Two paths, deliberately different:
 **Development** — `Plugin/plugin_setup.sh` symlinks this directory into the WebGUI plugin
 location. Every edit is live immediately. This is the normal working mode.
 **Release** — `Plugin/build.sh` packages this directory into a Slackware `.txz` and rewrites
 the version and checksum in `varaverk.plg`. That is what Unraid reinstalls from flash on every
 boot.
 In flash storage mode `git_pull_execute.sh` syncs `Plugin/` back to `/boot/` after each pull,
 so the WebGUI always serves current PHP even though `SCRIPTS_DIR` points at appdata.
 ---
 ## ━━━ ADDING A NEW PLATFORM ━━━
 1. Create `Plugin/<platform>/adapter.sh` implementing all 24 `platform_*()` functions.
 2. Set `PLATFORM=<platform>` in `varaverk.cfg`.
 That is the whole contract for the script ecosystem. `load_config.sh` sources
 `Plugin/$PLATFORM/adapter.sh` and every script keeps working unchanged.
 The WebGUI is separate and optional — it is Unraid-specific by nature (`.page` files, emhttp
 conventions). A new platform can run the entire ecosystem headless with only the adapter, and
 add a UI later in whatever form that platform expects.
@@ -0,0 +1,131 @@
 # ━━━━━ PLUGIN / UNRAID / SYSTEM ESSENTIALS ━━━━━
 Four scripts that touch Unraid subsystems directly. They live here rather than in the
 top-level `System_Essentials/` because each one manipulates something that only exists on
 Unraid — the mover, the User Scripts plugin, the PHP-FPM pool serving emhttp, and the
 unraid-api service registry.
 There is no platform-agnostic version of any of them. On TrueNAS there is no mover to stop.
 | Script | Runs | Role |
 |--------|------|------|
 | `php_fpm_max_children.sh` | Array **start** | Raise the PHP-FPM worker ceiling so the WebGUI stays responsive |
 | `unraid_api_key_renew.sh` | Array **start** | Re-register Varaverk's key in the ephemeral unraid-api registry |
 | `user_scripts_stop.sh` | Array **stop** | Stop User Scripts processes before the array goes down |
 | `mover_stop.sh` | Array **stop** | Stop the mover gracefully before rsync or reboot |
 All four are wired through `ARRAY_START_SCRIPTS` / `ARRAY_STOP_SCRIPTS` in `master.conf` —
 none are on a timer.
 ---
 ## ━━━ WHY EACH ONE EXISTS ━━━
 ### 🐘  `php_fpm_max_children.sh` — the WebGUI gets slow, not broken
 Unraid's WebGUI runs through PHP-FPM, and the stock `pm.max_children` is very low (4–8). Under
 real load — several browser tabs polling, Docker operations running, a dashboard open — every
 worker saturates and new requests queue. The WebGUI becomes slow or stops answering, while
 nothing is actually wrong with the server.
 Runs at array start because the setting **does not survive an Unraid update** — the OS
 replaces the pool config. Idempotent: already at the target value means no write and no
 restart, so a clean boot is silent.
 ### 🔑  `unraid_api_key_renew.sh` — the registry is ephemeral
 Varaverk's enhanced monitoring authenticates against unraid-api. That registry is cleared by
 OS updates and service restarts, so a key that worked yesterday can simply be gone.
 Re-registers unconditionally every array start rather than only when missing, because the
 failure it repairs is precisely *a key present in the conf but absent from the registry* —
 checking the conf would not detect it. Writes the resulting key into `HOST*_UNRAID_API_KEY`.
 ### 🛑  `user_scripts_stop.sh` — stop new work before shutting down
 The User Scripts plugin spawns background processes that Varaverk does not own. During a
 shutdown sequence those can still be starting operations while everything else is being torn
 down.
 Runs **first** in `ARRAY_STOP_SCRIPTS` for that reason — stop new work before stopping the
 things it would work on. Reports each process by **script name**, not just PID, because
 "stopping 4 processes" tells an operator nothing they can act on.
 Matching is scoped strictly to processes the User Scripts plugin spawned. A broad pattern
 would catch Varaverk's own scripts — including, during a reboot, the very script doing the
 stopping.
 ### 💾  `mover_stop.sh` — mover and rsync must not overlap
 The mover relocates files between cache and array. rsync reads those same paths. Both running
 at once can produce a corrupt or half-moved snapshot on the remote.
 Ordering in the stop sequence is deliberate: `rsync_stop.sh` runs **before** `mover_stop.sh`,
 because both write to the same paths and stopping the reader first is the safer order.
 Sends a wall warning first, then SIGTERM with a configurable window, then SIGKILL only if
 ignored. The mover is mid-file-move by definition — giving it the chance to finish the current
 file is the difference between a stopped transfer and a half-moved file.
 An absent mover exits 0. Callers use this as a precondition ("ensure the mover is not
 running"), so treating "already stopped" as failure would abort every reboot on a quiet
 system.
 ---
 ## ━━━ WHERE THEY SIT IN THE LIFECYCLE ━━━
 ```
 Array starting
    │
    ├── php_fpm_max_children.sh ──── WebGUI can handle the load that's about to arrive
    └── unraid_api_key_renew.sh ──── monitoring can authenticate again
        ... normal operation ...
 Array stopping
    │
    ├── user_scripts_stop.sh ─────── stop new work first
    ├── fallback.sh --stop
    ├── rsync_stop.sh ────────────── stop the reader
    ├── mover_stop.sh ────────────── then the writer — same paths
    └── docker_container_stop.sh ─── containers last, verified one at a time
 ```
 `mover_stop.sh` is also called directly by `server_reboot.sh`, and both stop scripts appear in
 the User Scripts master template for manual use.
 ---
 ## ━━━ SAFEGUARDS ━━━
 All four enforce root, take a lock, and support `--dry-run`. Specific to this group:
 **Graceful before forced.** `mover_stop.sh` and `user_scripts_stop.sh` both use SIGTERM with a
 window before SIGKILL. Neither kills first.
 **Absent target is success.** No mover running, no User Scripts running — both exit 0. These
 are preconditions, not commands that must find something to do.
 **Idempotent where it matters.** `php_fpm_max_children.sh` writes and restarts only when the
 value is actually wrong, so array start does not restart PHP-FPM every single boot.
 **Targeted process matching.** `user_scripts_stop.sh` matches only what the User Scripts
 plugin spawned — see above for why a broad pattern is dangerous here specifically.
 **Stop only, never start.** Neither stop script has a counterpart that restarts what it
 stopped. Unraid's own schedule owns when the mover runs; these only remove it from the picture
 for a window.
 ---
 ## ━━━ CONFIGURATION ━━━
 | Variable | Used by | Purpose |
 |----------|---------|---------|
 | `HOST*_UNRAID_API_KEY` | `unraid_api_key_renew.sh` | Written every array start — output, not input |
 | `MOVER_STOP_TIMEOUT` | `mover_stop.sh` | SIGTERM grace window before escalating |
 | `PHP_MAX_CHILDREN` | `php_fpm_max_children.sh` | Target worker ceiling |
 See each script's `CONFIGURATION` header section for the authoritative list — and
 `Deployment/master.conf.template`, which is the versioned schema for all of them.
@@ -0,0 +1,135 @@
 # ━━━━━ PLUGIN / UNRAID / TOOLS ━━━━━
 Platform-specific tooling for the WebGUI and the Unraid installation itself. Two of these keep
 the UI fast, two are operator tools for specific recovery and migration situations.
 They live here rather than in the top-level `Tools/` because each one manipulates something
 Unraid-specific — share `.cfg` files, the plugin's own PHP payload builders, or the
 installation's storage location.
 | File | Type | Runs |
 |------|------|------|
 | `api_cache_writer.sh` + `.php` | UI cache | **Every minute** via cron |
 | `remote_arr_cache_writer.sh` | UI cache | **Every 2 hours** via cron |
 | `recreate_shares.sh` | Recovery | Manual — after a rebuild or fresh install |
 | `storage_migrate.sh` | Migration | Manual — switching storage mode |
 ---
 ## ━━━ THE CACHE WRITERS — WHY PAGES LOAD INSTANTLY ━━━
 The monitor and arrs pages present a lot of state: per-core CPU, memory, network, GPUs,
 containers, VMs, transcodes, arr library counts. Building that live on every page view means
 dozens of API calls and `docker inspect` runs per refresh, from a WebGUI that is already the
 first thing to slow down under load.
 Instead, both are written to `/tmp/vv_cache` (tmpfs — RAM-speed reads, cleared on reboot) and
 the pages serve from there.
 ### ⚡  `api_cache_writer.sh` — local payloads, every minute
 A deliberate one-line shim: `php api_cache_writer.php`. All logic is in the PHP because the
 payload builders (`vv_monitor_*`, `vv_arrs_*`) are the **same functions the live API endpoints
 call**. A bash reimplementation would be a second version of the same payload, free to drift.
 Nothing here is allowed to break the UI:
 - A failed run simply leaves the cache unrefreshed — pages fall back to live calls. Slower,
  still correct.
 - A missing or unparseable cache is treated as absent, never as empty data.
 - `?live=1` bypasses the cache entirely.
 - Runs unprivileged with no lock — a torn cache file is replaced within 60 seconds, and every
  reader already has the live fallback.
 ### 🌐  `remote_arr_cache_writer.sh` — partner payloads, every 2 hours
 SSHes to each partner and calls `vv_arrs_local_node()` **on their** PHP stack, caching the
 result locally as `arrs_remote_<hostid>.json`.
 The partner builds its own payload rather than this host querying the partner's arr APIs
 directly. That matters: the partner already has working local URLs and API keys for its own
 arrs, so no cross-host credentials and no path mapping are involved. This host never holds
 keys for a remote's arrs.
 A partner whose Tailscale IP will not resolve is skipped, not fatal — the arrs page shows what
 it has and falls back for the rest.
 ---
 ## ━━━ THE OPERATOR TOOLS ━━━
 ### 🗂️  `recreate_shares.sh` — after a rebuild
 Recreates share **directories** on the correct disks by reading `/boot/config/shares/*.cfg`
 and honouring each share's `shareInclude` disk list.
 The `.cfg` files are Unraid's own record of what a share is and which disks it spans — they
 are the source of truth, not a list Varaverk maintains. Typically needed on a host after a
 full disk replacement or fresh install, where the config survived but the folders did not.
 **Create only, never delete.** A directory that already exists is left alone. This runs at a
 moment when the operator's picture of what should exist may be out of date, and removing
 anything on that basis is how a recovery step becomes a data-loss step.
 **Refuses to run without `/mnt/user` mounted.** Creating share directories against an
 unmounted array writes them into the underlying root filesystem, where they then shadow the
 real shares once the array does mount.
 ### 📦  `storage_migrate.sh` — moving the whole installation
 Relocates Varaverk between the two storage modes:
 | Mode | `SCRIPTS_DIR` | Trade-off |
 |------|---------------|-----------|
 | Internal | `/boot/config/plugins/varaverk` | Direct git pull/push. Available before the array mounts. |
 | Flash | `/mnt/user/appdata/Varaverk` | Preserves USB flash lifetime. **Array must be started** for Varaverk to function at all. |
 This is the most destructive script in the folder — `rsync --delete`, `cp -a`, `rm -rf` — and
 it rewrites the pointers everything else derives from:
 ```
  varaverk.cfg   SCRIPTS_DIR                    ← the authoritative path
  master.conf    TARGET_DIR
  host*.conf     HOST*_STORAGE_MODE_INTERNAL
  varaverk.cron  regenerated so job paths follow the new SCRIPTS_DIR
 ```
 `STATE_DIR`, `DATA_DIR`, `PERSISTENT_CONF_CACHE` and every orchestrator job path are built
 from `SCRIPTS_DIR`. Changing storage mode moves all of them at once, which is why the cron is
 **regenerated** rather than edited.
 In flash mode, `git_pull_execute.sh` syncs `Plugin/` back to `/boot/` after each pull, so the
 WebGUI keeps serving current PHP even though the scripts live in appdata.
 ---
 ## ━━━ SAFEGUARDS ━━━
 **The cache writers are deliberately unprivileged and lockless.** Their output is regenerable
 within a minute and every consumer has a live fallback. There is no privileged operation to
 gate and no state worth locking. This is documented in their headers so it is not "corrected"
 later.
 **The operator tools are the opposite.** Both enforce root, take a lock, and support
 `--dry-run`. `storage_migrate.sh` additionally exits cleanly if already in the requested mode,
 and removes the old location only once the new one is confirmed in place.
 **`recreate_shares.sh` will not act on an unmounted array** — see above; this is the guard
 that prevents shadow directories.
 ---
 ## ━━━ RELATIONSHIP TO THE WEB UI ━━━
 ```
  pages/monitor.php ─┐
                     ├─► include/monitor.php  (vv_monitor_*)
  api/monitor.php ───┘            ▲
                                  │
  Tools/api_cache_writer.php ─────┘   same builders → cache can't disagree with live
        │
        └─► /tmp/vv_cache/monitor.json ──► served by api/monitor.php unless ?live=1
 ```
 Adding a metric means adding it in `include/` once. The page, the live endpoint, and the cache
 all pick it up together.