Sonarr and Lidarr had the same deadlock and the same missing confirmation as Radarr, so give them the same budget and strikes

The window editor is path-first, which is exact but slow for the case that is nearly all of them.
The chips only offer whole shares; a subpath already in the window marks its share partial rather
than offering to widen it, and the path box stays the only way to express a subpath or a profile.

.gitignore

@@ -3,14 +3,21 @@
 # Templates (*.template) are safe and remain tracked.
 Configurations/host*.conf
 Configurations/master.conf
-Configurations/*.bak
+# *.bak alone missed conf_upgrade's real output — it writes host1.conf.bak-20260802, which does
+# not end in .bak — so those sat untracked rather than ignored, one `git add -A` from being
+# pushed. The glob has to cover the suffix.
+Configurations/*.bak*
 .vscode
 
 # ── Personal scratch notes — dev-only, never pushed ───────────────────────────
 Notes_To-Do.md
 
 # ── Runtime state, data, logs ─────────────────────────────────────────────────
-data/
+# Contents, not the directory itself. Ignoring "data/" outright means git never descends into
+# it, and a negation for a file inside an excluded directory is silently ineffective — so the
+# README explaining what data/ is would be the one file missing from every installation of it.
+data/*
+!data/README.md
 State_Files/
 .cache/
 *.log
@@ -36,3 +43,15 @@ Plugin/dist/
 *.swp
 *~
 .vscode/
+
+# ── Local-only plugin surfaces (per-installation, never pushed) ───────────────
+# Varaverk.page discovers pages/local/*.php and registers each as a tab; api/local/ holds their
+# endpoints. Both are symlinks into a store outside this repo, so what they contain belongs to
+# one installation and is not part of the project. The tracked loader is deliberately generic —
+# it names no page — so the public mirror never learns what a given server runs here.
+#
+# No trailing slash on either pattern. These paths are symlinks, not directories, and git treats
+# a symlink as a blob — a "dir/" pattern does not match one, so the entries sat untracked rather
+# than ignored, which is the same near-miss the *.bak rule above documents.
+Plugin/unraid/pages/local
+Plugin/unraid/api/local

AI/.gitignore

@@ -0,0 +1,4 @@
+# The index is generated, host-specific, and regenerable in minutes. Never commit it.
+*.db
+*.db-wal
+*.db-shm

AI/README-AI.md

@@ -0,0 +1,370 @@
+# ━━━━━ AI ━━━━━
+
+Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an
+answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources.
+
+Everything here is **off by default and optional**. Varaverk works exactly as well with
+`AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder.
+
+| File | Role |
+|------|------|
+| `ai_index.sh` | Build / refresh the retrieval index |
+| `ai_query.sh` | Ask a question, or search the index directly |
+| `lib/chunk.js` | Split repo files into retrieval units |
+| `lib/index.js` | Embed chunks, store vectors in SQLite |
+| `lib/search.js` | Embed a query, score it, rank results |
+| `lib/cli.js` | Argument bridge the bash entry points call |
+
+---
+
+## ━━━ WHY THIS WORKS AT ALL ━━━
+
+**Because the header audit came first.**
+
+The single worst failure in naive retrieval is a chunk that contains half of one idea and half
+of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one
+vector. That problem does not exist here, because every script in the repo carries the same six
+sections at exact, greppable boundaries:
+
+```
+PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS
+        → CONFIGURATION → RUNTIME MODES
+```
+
+Split on those and every chunk is a coherent unit by construction. No token windows, no overlap
+heuristics, no tuning.
+
+**And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it
+does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that
+`removeCompletedDownloads` being true on both arrs is intentional. Those live in
+`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth
+more than an equivalent pile of source.
+
+---
+
+## ━━━ SECTION ROUTING ━━━
+
+Every chunk stores its section name as its own column, so a question's *shape* can steer
+retrieval before similarity is even considered:
+
+| Question shape | Steered toward |
+|----------------|----------------|
+| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
+| "which variable controls X" | `CONFIGURATION` |
+| "does this take --dry-run" | `RUNTIME MODES` |
+| "why is it built this way" | `DESIGN PRINCIPLES` |
+| "how does X work" | `OPERATIONAL MODEL` |
+
+Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic
+must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a
+hard filter when you actually want one.
+
+The boost still has a blind spot worth knowing about: **definitional questions.** "What is
+Varaverk?" matches the `PURPOSE` intent, so every script's one-line PURPOSE gets boosted above
+the top-level prose that actually answers it — and the model correctly replies that the context
+does not define the system. The corpus is fine; `README.md` is indexed. The routing simply
+buries it. `--kind=readme` is the hard filter for that case:
+
+```bash
+bash AI/ai_query.sh --kind=readme "what is Varaverk"
+```
+
+`--kind` filters on where a chunk came from — `header`, `readme`, `manual`, `template`, `doc`, `ui` —
+and composes with `--section`. Prefer it over `--section` for "what is" and "why does this
+exist" questions, where the answer is narrative rather than a header field.
+
+---
+
+## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━
+
+Sections alone were not granular enough, and the failure was instructive.
+
+`rsync.sh` documents fourteen distinct safeguards in one 2.8k-character
+`OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the
+correct chunk scored **0.558** — below unrelated chunks from other files — because the other
+thirteen safeguards dominated the vector.
+
+The header convention writes each safeguard as a named paragraph: an unindented title, an
+indented body. Splitting on those titles took the same query to **0.718** and first place.
+
+```
+Merge-Run Delete Interlock            ← its own chunk
+  --delete is applied only when pass 1 completed...
+```
+
+The parent section name is carried onto every sub-chunk, so routing still works. The title
+detection requires the *next* line to be indented — without that check, any wrapped prose line
+became a spurious boundary mid-sentence.
+
+---
+
+## ━━━ WHAT IS INDEXED ━━━
+
+Roughly 2,900 chunks across ~180 files:
+
+| Kind | Source |
+|------|--------|
+| `header` | Script headers — bash and PHP, sections and named paragraphs |
+| `readme` | Every `README-*.md` |
+| `manual` | Every `Manual-*.md` |
+| `template` | `Deployment/*.template` — the versioned conf schema |
+| `doc` | Top-level `README.md`, `Manual.md`, design notes |
+| `ui` | `Plugin/unraid/pages/readme/*.md` — the WebGUI's own help panels |
+
+**Script bodies are not indexed.** Headers state intent, code states mechanism; for the
+questions this answers, intent retrieves better and costs far less.
+
+---
+
+## ━━━ THE SAFETY BOUNDARY ━━━
+
+**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
+
+`Configurations/` and `data/` are gitignored, so every file holding a credential
+was never in the repo to begin with. The index therefore describes the full conf schema (via the
+tracked templates, which carry all the explanatory comments) while structurally **never
+containing a secret**.
+
+> Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret
+> averaged into a 768-dimension float cannot be found, let alone removed.
+
+A live conf value the model genuinely needs should arrive through a tool call at query time,
+subject to redaction — never baked into a vector at index time.
+
+---
+
+## ━━━ HOW IT IS BUILT ━━━
+
+**SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds
+per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and
+back up, in exchange for nothing at this scale.
+
+**Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised
+vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP
+reads the same blobs with `unpack('f*', $blob)` when the UI needs them.
+
+**Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A
+no-op refresh takes about 70 ms; a full rebuild takes a few minutes.
+
+**Node for the maths, bash for everything else.** The bash entry points own configuration,
+gating, locking and logging exactly as every other Varaverk job does. Node owns only float
+vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP.
+
+> `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it
+> needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is
+> regenerable in minutes — this is a disposable artefact, not a datastore.
+
+---
+
+## ━━━ USAGE ━━━
+
+```bash
+# Build (needs AI_ENABLED=true)
+bash AI/ai_index.sh                  # incremental
+bash AI/ai_index.sh --force          # full rebuild
+bash AI/ai_index.sh --dry-run        # what would be indexed; contacts nothing
+bash AI/ai_index.sh --status         # size, counts, last build
+
+# Ask
+bash AI/ai_query.sh "what stops rsync and the mover running at once"
+bash AI/ai_query.sh --search "why are the cache writers lockless"
+bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period"
+bash AI/ai_query.sh --kind=readme "what is Varaverk"          # definitional / narrative
+bash AI/ai_query.sh --json "..."     # for other scripts
+```
+
+**`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated.
+When the answer matters, use it — or read the sources the generated answer cites.
+
+---
+
+## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━
+
+The generation prompt instructs the model to answer only from retrieved context and to say what
+is missing rather than fill the gap from general knowledge. That instruction matters here more
+than usual: this repo's conventions are frequently *not* the conventional ones, and a confident
+generic answer about rsync, Docker or systemd is worse than no answer.
+
+It works — and its faithfulness cut both ways on the first real test. Asked which variable
+controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30
+seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the
+real value is 300 — and the model was quoting the header verbatim. **The header was stale.**
+
+That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The
+lesson is the operating principle for this whole folder:
+
+> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
+> wrong, check the cited source before blaming the model — it is usually reporting a real
+> problem in the repo.
+
+---
+
+## ━━━ WHAT THIS DOES NOT DO ━━━
+
+- **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with
+  an empty key whitelist. Nothing in this folder writes a setting.
+- **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The
+  per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately.
+- **It does not index code.** Headers and docs only.
+- **It does not sync.** The index is host-local and gitignored. Each node builds its own.
+- **It is not required.** Every script runs identically with `AI_ENABLED=false`.
+
+See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the
+Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be
+built before any of that is enabled.
+
+---
+
+## ━━━ SCHEDULING ━━━
+
+The index tracks the repo automatically, from the only moment the corpus actually changes — a
+successful pull. `git_pull_execute.sh` re-indexes behind three gates: the pull succeeded and
+changed tracked files, `AI_INDEX_ON_PULL=true`, and `AI_ENABLED=true`. It is never fatal — a
+failed index leaves the previous one in place and the pull still reports success.
+
+No cron entry and no `DAILY_MAINTENANCE_SCRIPTS` line are needed; the daily pull carries it.
+
+An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing
+and a pull that changed twelve files costs a few seconds.
+
+---
+
+## ━━━ PROFILES ━━━
+
+A profile is a contract plus a set of inputs. `Plugin/unraid/include/ai_profiles.php` is the one
+definition of both, read by the endpoint, the worker, the shared chat include and the Scheduler
+dock.
+
+| Profile | Turns | Retrieves | Notes |
+|---|---|---|---|
+| `varaverk` | 3 | yes | answers only from the index, with citations. The default. |
+| `chat` | 8 | **no** | ordinary conversation. Holds zero capabilities, deliberately. |
+| `code` | 4 | no | drafts shell for Custom Scripts; scans its own output for destructive ops |
+| `troubleshoot` | 2 | yes | reasons from evidence — an open log, or one you name. May file bug reports. |
+
+Capabilities are granted per profile — retrieval, live health, run evidence, scoped log,
+incidents, conf lookup, bug filing, code scanning. `chat` holding an empty list is a guarantee,
+not an oversight: anything added to it stops being general chat and becomes an assistant that
+sometimes invents claims about this installation.
+
+### Routing out of General Chat
+
+`chat` hands a question to whichever profile fits, decided by `vv_ai_route_from_chat()`. Ordered
+most specific first, because these overlap on purpose:
+
+| Question | Goes to | Why |
+|---|---|---|
+| "write me a script that prunes logs" | `code` | asked for something written |
+| "why did the daily orch fail" | `troubleshoot` | diagnostic phrasing **and** something here to diagnose |
+| "how did the daily orch go" | `varaverk` | about this install, but not a fault |
+| "what does arr_sync.sh do" | `varaverk` | names a script, wants documentation |
+| "why is the sky blue" | stays `chat` | diagnostic phrasing about nothing here |
+
+`code` is checked first because it is the only intent about a thing that does not exist yet, so
+nothing else can claim it — and it is anchored on the verb, which is what keeps "write me a
+script" apart from "what does this script do". Escalation adds capability, so a wrong escalation
+costs more than a missed one: anything unrecognised stays in `chat`, the profile that cannot
+invent claims about this system. The worker reverts to `chat` anyway if retrieval comes back
+empty.
+
+The answer opens with one line naming the profile that took it, because the button still shows
+the one you picked and an answer arriving under a different contract otherwise reads as the
+assistant ignoring you.
+
+Routing is asserted by `Plugin/unraid/Tools/ai_explain_check.sh` against
+`ai_explain_fixtures.txt` — every case runs through the worker's `--explain` mode, which stops
+where deterministic assembly ends and never calls the model.
+
+This used to live in five places — history depth in the endpoint, capabilities in `include/ai.php`,
+label and depth again in JavaScript, a prompt branch in the worker, and a label map on the
+Scheduler page. They had already drifted: the JavaScript knew three profiles where PHP knew four.
+The system prompts still live in `Tools/ai_chat_worker.php`, because they have one reader and
+moving them would relocate the most delicate text in the subsystem without removing a duplicate.
+
+## ━━━ CONVERSATIONS ━━━
+
+Chats are stored server-side under `AI_DATA_DIR/ai_chats/`, one JSON file each, saved
+automatically when a turn completes and pruned to `AI_CHAT_HISTORY_MAX` (default 10, oldest
+first by creation).
+
+There is no Save button. A conversation worth keeping is not reliably one you knew was worth
+keeping while you were having it.
+
+The same store backs the AI tab and the Monitor tab's AI row, so a thread started on the
+dashboard is the one you carry on in the tab. Messages are re-validated per message on the way
+in — a stored chat is replayed into a later prompt when reopened, so an unchecked role written
+there would be an injection that survives a reload rather than one turn.
+
+Reopened chats render as plain turns: sources, reasoning and timings describe one generation and
+are not stored, because redrawing them beside a transcript that may be continued under a
+different profile would be citing evidence for an answer no longer being made.
+
+### Secrets are redacted on the way to disk
+
+A conversation about settings is a conversation containing credentials — asking for an API key to
+be changed means typing one. Message bodies are redacted in `vv_ai_chat_save()`, and the question
+is redacted again before it reaches `ai.log`.
+
+**On the way out, never in flight.** The live turn keeps the real value, because the model needs
+it to carry out what was asked. What it does not need is that value still in the transcript a week
+later — and a stored chat is replayed into a later prompt when reopened, so an unredacted one
+would hand the credential back on every subsequent turn, indefinitely.
+
+Two passes, because they catch different things:
+
+| Pass | Catches | Method |
+|---|---|---|
+| Known values | a credential this host already holds | exact match against secret-shaped conf keys, longest first |
+| Assignment shapes | a credential arriving that is not in the conf yet | `NAME=value`, `"api_key": value`, "set the token to …" |
+
+The second pass is the one that matters for settings changes: *"change the Emby API key to X"* is
+a secret arriving, and X matches nothing on disk until after the write it is requesting.
+
+Ordinary prose is left alone — the patterns anchor on a secret-shaped *name*, so `CACHE_WARN_GB=100`
+and "turn off the zfs scrub" pass through untouched. `vv_conf_key_is_secret()` is shared with the
+conf audit log, so the two cannot disagree about what counts as a secret.
+
+## ━━━ TOKEN ACCOUNTING ━━━
+
+Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`):
+
+```
+date|time|host|profile|source|prompt_tokens|completion_tokens|tok_s
+2026-08-04|22:03:51|host1|varaverk|cli|2041|318|61.4
+```
+
+Both paths write it — this CLI (`source=cli`) and the WebGUI worker (`source=webgui`) — so the
+totals are not quietly the tab's alone. `ai_query.sh` passes `--token-db` and `--token-host`;
+called by hand without them, `cli.js` simply skips the row rather than guessing a path, because
+this file never reads conf itself.
+
+Read it on the plugin's AI tab, which aggregates today / last 7 days / all time, per host. Or
+straight from the shell, since it is just a delimited file:
+
+```bash
+# tokens used today
+awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai/ai_token_history.db
+```
+
+**The host column is where the turn ran, not where the file is read.** Each host writes only its
+own rows.
+
+`ai_token_sync.sh` pulls each partner's ledger into `$AI_TOKEN_CACHE_DIR/<slot>.tokens.db` (`/tmp/varaverk/ai/`) — the
+same trick `conf_sync.sh` uses for partner confs, and it runs from
+`INTERMEDIATE_MAINTENANCE_SCRIPTS` every four hours. The tab then reads every ledger it can see,
+so a fleet total is a fleet total.
+
+Pull only, no push: nothing here is needed by anyone else, and a reader that fetches its own data
+controls its own freshness instead of depending on the partner's cron. A partner file may only
+contribute rows whose `host` column matches its filename — a ledger copied into the wrong slot
+would otherwise be double-counted against a total that still looked plausible.
+
+The cache is tmpfs with **no save/restore pair**, unlike the conf cache. Stale counters are worse
+than absent ones: absent renders as "not collected here", stale renders as fact. An unreachable
+partner leaves its file alone and logs at info, because a partner being down for weeks is a
+normal state, not an incident.
+
+Pruning is by row count (`AI_TOKEN_RETAIN_ROWS`, default 20000) and happens on write, but only
+once the file passes a size threshold — an ordinary turn costs a `stat()` and an append. The CLI
+deliberately does not prune: duplicating a read-modify-write of the whole file in a second
+language is how the two drift apart.

AI/ai_index.sh

@@ -0,0 +1,265 @@
+#!/bin/bash
+# ==============================================================================================
+# ========================== AI Retrieval Index Builder ========================================
+# ==============================================================================================
+#
+# PURPOSE
+# ==============================================================================================
+# Builds and refreshes the retrieval index over Varaverk's own documentation — every script
+# header, folder README and Manual, and the conf templates. Chunks each file on the section
+# boundaries the header convention already defines, embeds each chunk through Ollama, and
+# stores the vectors in SQLite for AI/ai_query.sh to search.
+#
+# The index is derived data. It is gitignored, host-local, and rebuildable from the repo in
+# minutes — nothing depends on it surviving.
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+# Chunking mirrors the header convention rather than using a fixed token window:
+#
+#   bash headers      split on the six section names, then sub-split named-paragraph
+#                     safeguards and principles so one question finds one answer
+#   PHP headers       same six names plus the per-layer tails (EXPORTS, RENDERS, ...)
+#   markdown          split on headings
+#   conf templates    split on the ━━━ section rules
+#
+# Every chunk keeps its section name as a field, which is what lets a query about a safeguard
+# be steered toward OPERATIONAL SAFEGUARDS chunks before similarity is considered.
+#
+# Incremental by file mtime. A file whose mtime has not moved is skipped without being read,
+# so a routine refresh costs well under a second and a full rebuild costs a few minutes.
+#
+# Heavy lifting runs in Node — float vector maths and SQLite BLOBs are genuinely awkward in
+# bash. This follows the api_cache_writer.sh precedent: a bash shim owning config, gating,
+# locking and logging, in front of the language that fits the work.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Tracked Files Only
+#   Indexes exactly what `git ls-files` reports. Configurations/, State_Files/ and data/ are
+#   gitignored, so it is structurally impossible for a credential to reach the index — the
+#   files holding them were never in the repo. This must never become a filesystem walk: a
+#   secret written into a vector cannot be rotated back out of it.
+#
+# The Index Is Disposable
+#   Stored under DATA_DIR, gitignored, and never synced to a partner. Losing it costs one
+#   rebuild. Nothing reads it as a source of truth — it points at files, and the files are
+#   the truth.
+#
+# Documentation Is The Corpus, Not The Code
+#   Script bodies are not indexed. The headers state intent and the code states mechanism;
+#   for the questions this answers, intent retrieves far better and is far cheaper.
+#
+# Off By Default
+#   Does nothing unless AI_ENABLED is true. A node with AI off never pays for this.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Fail-Closed Gate
+#   Exits cleanly unless AI_ENABLED is exactly "true". Any other value, including unset,
+#   means off.
+#
+# Root Enforcement
+#   Writes into DATA_DIR alongside other Varaverk state.
+#
+# Concurrency Lock
+#   acquire_lock() prevents two indexers racing on the same database.
+#
+# Dependency Verification
+#   Verifies node and the Ollama endpoint before touching the database. A missing dependency
+#   is reported and exits non-zero rather than leaving a half-built index.
+#
+# Reachability Pre-flight
+#   Probes the resolved Ollama URL with AI_CONNECT_TIMEOUT before starting. An unreachable
+#   endpoint aborts immediately instead of failing once per batch across the whole corpus.
+#
+# Partial Failure Is Not Recorded As Success
+#   A file whose chunks all failed to embed keeps its previous rows and its old mtime, so the
+#   next run retries it. A run with any failed batch exits 3.
+#
+# Atomic Per-Run Write
+#   All database changes commit in one transaction. An interrupted run leaves the previous
+#   index intact rather than a partially rewritten one.
+#
+# Deleted Files Are Removed From The Index
+#   A file that has left the repo has its chunks deleted, so retrieval cannot cite something
+#   that no longer exists.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   AI_ENABLED
+#       Master switch. Fail-closed — must be exactly "true".
+#
+#   AI_INDEX_DB
+#       SQLite index path. (shipped default: $DATA_DIR/ai_index.db)
+#
+#   AI_INDEX_BATCH
+#       Chunks per embed request. (shipped default: 32)
+#
+#   AI_CONNECT_TIMEOUT
+#       Seconds for the reachability probe. (shipped default: 5)
+#
+#   AI_REQUEST_TIMEOUT
+#       Seconds for a single embed batch. (shipped default: 240)
+#
+# host*.conf
+#
+#   HOST*_OLLAMA_URL
+#       This host's Ollama endpoint. Empty means no local Ollama.
+#
+#   HOST*_OLLAMA_EMBED_MODEL
+#       Embedding model. The generation model cannot embed.
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# ai_index.sh
+#     Incremental refresh — only files whose mtime moved are re-embedded.
+#
+# ai_index.sh --force
+#     Full rebuild. Discards the existing index and re-embeds every chunk.
+#
+# ai_index.sh --dry-run
+#     Report what would be indexed. Contacts nothing and writes nothing.
+#
+# ai_index.sh --status
+#     Show index location, size, chunk counts by kind and section, and last build time.
+#
+# ai_index.sh --log
+#     Verbose — per-batch embedding progress.
+#
+# ==============================================================================================
+
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/../load_config.sh"
+
+detect_hosts
+
+DRY_RUN=false; FORCE=false; STATUS=false; LOG=false
+for arg in "$@"; do
+    case "$arg" in
+        --dry-run) DRY_RUN=true ;;
+        --force)   FORCE=true ;;
+        --status)  STATUS=true ;;
+        --log)     LOG=true ;;
+        *) echo "Unknown option: $arg" >&2; exit 1 ;;
+    esac
+done
+
+CLI="${SCRIPT_DIR}/lib/cli.js"
+DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
+
+_url_var="${MY_ID}_OLLAMA_URL"
+_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
+OLLAMA_URL="${!_url_var:-}"
+EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
+
+# ── Status ────────────────────────────────────────────────────────────────────────────────────
+if [[ "$STATUS" == true ]]; then
+    echo "$ICON_GEAR  AI Index Status"
+    echo "   Enabled:   ${AI_ENABLED:-false}"
+    echo "   Database:  $DB"
+    if [[ -f "$DB" ]]; then
+        echo "   Size:      $(du -h "$DB" 2>/dev/null | cut -f1)"
+        echo "   Chunks:    $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null || echo '?')"
+        echo "   Files:     $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_files;'  2>/dev/null || echo '?')"
+        _last=$(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='last_index';" 2>/dev/null)
+        [[ -n "$_last" ]] && echo "   Last built: $(date -d "@$_last" '+%Y-%m-%d %H:%M:%S' 2>/dev/null)"
+        echo "   Model:     $(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='embed_model';" 2>/dev/null || echo '?')"
+        echo "   By kind:"
+        sqlite3 "$DB" "SELECT '     '||kind||': '||COUNT(*) FROM vv_chunks GROUP BY kind ORDER BY COUNT(*) DESC;" 2>/dev/null
+    else
+        echo "   Database:  not built yet"
+    fi
+    echo "   Ollama:    ${OLLAMA_URL:-<none local>}"
+    echo "   Embed:     $EMBED_MODEL"
+    exit 0
+fi
+
+# ── Gate ──────────────────────────────────────────────────────────────────────────────────────
+if [[ "${AI_ENABLED:-false}" != "true" ]]; then
+    log "AI_ENABLED is not true — skipping index build"
+    exit 0
+fi
+
+# The mesh shares one AI, and the index belongs to the node that holds the model. A mirror has the
+# same checkout and could build one, but nothing there would read it: retrieval happens wherever
+# generation happens, which is the owner.
+#
+# A skip, not an error. This is reached from git_pull_execute.sh on every node after every pull;
+# before the AI became mesh-wide it ran here too and failed on the empty OLLAMA_URL, nightly and
+# silently, because the caller discards its output.
+_ai_owner="${AI_OWNER_HOST:-host1}"
+if [[ "${MY_ID,,}" != "${_ai_owner,,}" ]]; then
+    log "This node is not the AI owner ($_ai_owner) — the index lives there; skipping"
+    exit 0
+fi
+
+if [[ "$DRY_RUN" == false && "$EUID" -ne 0 ]]; then
+    error "Must be run as root"
+    exit 1
+fi
+
+acquire_lock
+
+command -v node >/dev/null 2>&1 || { error "node not found — required to build the index"; exit 1; }
+[[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; }
+
+# The AI owner has had data/ai since the subsystem was built, so nothing ever created it — cli.js
+# opens the DB by path and does not make the directory. On a first build the failure surfaces as a
+# sqlite open error rather than as the missing directory it is.
+if [[ "$DRY_RUN" == false ]] && ! mkdir -p "$(dirname "$DB")"; then
+    error "Cannot create $(dirname "$DB")"
+    exit 1
+fi
+
+if [[ -z "$OLLAMA_URL" ]]; then
+    error "${MY_ID}_OLLAMA_URL is empty — no local Ollama to index against"
+    exit 1
+fi
+
+# Pre-flight: fail once, up front, rather than once per batch across the whole corpus.
+if [[ "$DRY_RUN" == false ]]; then
+    if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
+        error "Ollama unreachable at $OLLAMA_URL"
+        exit 1
+    fi
+fi
+
+# ── Build ─────────────────────────────────────────────────────────────────────────────────────
+_args=(
+    index
+    "--root=${SCRIPTS_DIR}"
+    "--db=${DB}"
+    "--url=${OLLAMA_URL}"
+    "--model=${EMBED_MODEL}"
+    "--batch=${AI_INDEX_BATCH:-32}"
+    "--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))"
+)
+[[ "$FORCE"   == true ]] && _args+=(--force)
+[[ "$DRY_RUN" == true ]] && _args+=(--dry-run)
+[[ "$LOG"     == false ]] && _args+=(--quiet)
+
+log "$ICON_GEAR Building AI index → $DB"
+node --no-warnings "$CLI" "${_args[@]}"
+_rc=$?
+
+case "$_rc" in
+    0) log "$ICON_DONE AI index build complete" ;;
+    3) warn "AI index built with some batches failed — those files will retry next run" ;;
+    *) error "AI index build failed (exit $_rc)" ;;
+esac
+
+exit "$_rc"

AI/ai_query.sh

@@ -0,0 +1,209 @@
+#!/bin/bash
+# ==============================================================================================
+# ============================== AI Retrieval Query ============================================
+# ==============================================================================================
+#
+# PURPOSE
+# ==============================================================================================
+# Answers questions about Varaverk from Varaverk's own documentation. Embeds the question,
+# retrieves the closest chunks from the index AI/ai_index.sh built, and either prints them
+# directly or passes them to the generation model as grounding context.
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+# Two modes over the same retrieval:
+#
+#   --search    print the matching chunks and their sources. No generation model involved,
+#               so it is fast and its output is verbatim repo text.
+#   (default)   retrieve, then ask the generation model to answer strictly from what was
+#               retrieved, citing each claim.
+#
+# Retrieval is steered by question shape. A question about what prevents something is pushed
+# toward OPERATIONAL SAFEGUARDS chunks, one about a variable toward CONFIGURATION, one asking
+# why toward DESIGN PRINCIPLES. This is a score boost, not a filter — a heuristic must not be
+# able to exclude the chunk that actually holds the answer.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Grounded Or Silent
+#   The prompt instructs the model to answer only from retrieved context and to say what is
+#   missing rather than fill the gap. This repo's conventions are frequently not the
+#   conventional ones, and a confident generic answer about rsync or Docker is worse here
+#   than no answer.
+#
+# Sources Are Always Shown
+#   Every answer prints the chunks it drew on. An answer that cannot be traced back to a file
+#   is not usable for changing anything.
+#
+# Search Is The Trustworthy Mode
+#   --search returns repo text with nothing generated. When an answer matters, use it.
+#
+# Read-Only
+#   Retrieves and answers. Nothing here writes conf, touches state, or runs another script.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Fail-Closed Gate
+#   Exits cleanly unless AI_ENABLED is exactly "true".
+#
+# No Root Required
+#   Reads the index and calls Ollama. Nothing it does needs privilege, so it does not ask for
+#   any — this is the one AI script an ordinary user should be able to run.
+#
+# Missing Index Is Reported, Not Built
+#   An absent index exits with guidance to run ai_index.sh. Building a corpus-wide index as a
+#   side effect of a question would turn a two-second query into a several-minute one.
+#
+# Reachability Pre-flight
+#   Probes Ollama with AI_CONNECT_TIMEOUT before embedding, so an unreachable endpoint fails
+#   immediately with a clear message.
+#
+# Bounded Generation
+#   The request is capped at AI_REQUEST_TIMEOUT. A wedged model cannot hang the caller.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   AI_ENABLED
+#       Master switch. Fail-closed — must be exactly "true".
+#
+#   AI_INDEX_DB
+#       SQLite index to search. (shipped default: $DATA_DIR/ai_index.db)
+#
+#   AI_SEARCH_K
+#       Chunks retrieved per query. (shipped default: 8)
+#
+#   AI_SEARCH_PER_FILE
+#       Cap per file, so one document cannot fill the context. (shipped default: 3)
+#
+#   AI_REQUEST_TIMEOUT
+#       Seconds allowed for generation. (shipped default: 240)
+#
+# host*.conf
+#
+#   HOST*_OLLAMA_URL / HOST*_OLLAMA_MODEL / HOST*_OLLAMA_EMBED_MODEL
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# ai_query.sh "your question"
+#     Retrieve and answer, with sources.
+#
+# ai_query.sh --search "your question"
+#     Print matching chunks only. No generation.
+#
+# ai_query.sh --section=CONFIGURATION "your question"
+#     Restrict retrieval to one header section.
+#
+# ai_query.sh --kind=readme "what is Varaverk"
+#     Restrict retrieval to one chunk origin: header, readme, manual, template, doc.
+#     Use this for definitional and narrative questions. Intent routing boosts header
+#     sections such as PURPOSE, which answers "what does this script do" well but buries
+#     the top-level prose that explains what the system *is*. --kind=readme reaches it.
+#
+# ai_query.sh --json "your question"
+#     Machine-readable output for other scripts.
+#
+# ai_query.sh --status
+#     Show index and endpoint state.
+#
+# ==============================================================================================
+
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "${SCRIPT_DIR}/../load_config.sh"
+
+detect_hosts
+
+SEARCH_ONLY=false; JSON=false; STATUS=false; SECTION=""; KIND=""; QUERY=""
+for arg in "$@"; do
+    case "$arg" in
+        --search)     SEARCH_ONLY=true ;;
+        --json)       JSON=true ;;
+        --status)     STATUS=true ;;
+        --section=*)  SECTION="${arg#*=}" ;;
+        --kind=*)     KIND="${arg#*=}" ;;
+        --*)          echo "Unknown option: $arg" >&2; exit 1 ;;
+        *)            QUERY="$arg" ;;
+    esac
+done
+
+# --kind is a hard filter on chunk origin; --section filters the header section within a
+# chunk. They answer different questions and compose: --kind=readme --section=PURPOSE is
+# meaningful. Validated here rather than in the CLI so a typo costs nothing — an unknown kind
+# silently matches no rows, which reads as "the index has no answer" and is the most
+# misleading failure this tool can produce.
+if [[ -n "$KIND" ]]; then
+    case "$KIND" in
+        header|readme|manual|template|doc|ui) ;;
+        *) echo "Unknown --kind=$KIND (expected: header, readme, manual, template, doc, ui)" >&2
+           exit 1 ;;
+    esac
+fi
+
+CLI="${SCRIPT_DIR}/lib/cli.js"
+DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
+
+_url_var="${MY_ID}_OLLAMA_URL"
+_gen_var="${MY_ID}_OLLAMA_MODEL"
+_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
+OLLAMA_URL="${!_url_var:-}"
+GEN_MODEL="${!_gen_var:-}"
+EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
+
+if [[ "$STATUS" == true ]]; then
+    echo "$ICON_GEAR  AI Query Status"
+    echo "   Enabled:  ${AI_ENABLED:-false}"
+    echo "   Index:    $DB $([[ -f "$DB" ]] && echo "($(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null) chunks)" || echo '(not built)')"
+    echo "   Ollama:   ${OLLAMA_URL:-<none>}"
+    echo "   Generate: ${GEN_MODEL:-<unset>}"
+    echo "   Embed:    $EMBED_MODEL"
+    exit 0
+fi
+
+if [[ "${AI_ENABLED:-false}" != "true" ]]; then
+    echo "AI_ENABLED is not true — AI features are off" >&2
+    exit 0
+fi
+
+[[ -z "$QUERY" ]] && { echo "usage: ai_query.sh [--search] [--section=NAME] [--kind=KIND] \"your question\"" >&2; exit 1; }
+[[ -f "$DB" ]]   || { error "No index at $DB — run AI/ai_index.sh first"; exit 1; }
+[[ -f "$CLI" ]]  || { error "missing $CLI"; exit 1; }
+command -v node >/dev/null 2>&1 || { error "node not found"; exit 1; }
+[[ -z "$OLLAMA_URL" ]] && { error "${MY_ID}_OLLAMA_URL is empty"; exit 1; }
+
+if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
+    error "Ollama unreachable at $OLLAMA_URL"
+    exit 1
+fi
+
+_args=("--db=${DB}" "--url=${OLLAMA_URL}" "--query=${QUERY}")
+[[ -n "$SECTION" ]] && _args+=("--section=${SECTION}")
+[[ -n "$KIND"    ]] && _args+=("--kind=${KIND}")
+[[ "$JSON" == true ]] && _args+=(--json)
+
+if [[ "$SEARCH_ONLY" == true ]]; then
+    node --no-warnings "$CLI" search "${_args[@]}" \
+        "--model=${EMBED_MODEL}" \
+        "--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}"
+else
+    [[ -z "$GEN_MODEL" ]] && { error "${MY_ID}_OLLAMA_MODEL is empty — needed for generation"; exit 1; }
+    # Token accounting. Passed in rather than re-read in node, so the conf stays the shell's job
+    # and cli.js keeps taking everything it needs as arguments. Omitting either flag simply
+    # skips the row — the CLI must still work when called by hand outside this wrapper.
+    node --no-warnings "$CLI" ask "${_args[@]}" \
+        "--model=${GEN_MODEL}" "--embed-model=${EMBED_MODEL}" \
+        "--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}" \
+        "--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))" \
+        "--token-db=${AI_TOKEN_DB:-}" "--token-host=$(echo "$MY_ID" | tr '[:upper:]' '[:lower:]')"
+fi

AI/ai_token_sync.sh

@@ -0,0 +1,236 @@
+#!/bin/bash
+# ==============================================================================================
+# ============================= AI Token Ledger Sync ===========================================
+# ==============================================================================================
+#
+# PURPOSE
+# ─────────────────────────────────────────────────────────────────────────────
+# Pulls each partner's AI token ledger into a RAM cache at /tmp/.cache/vv/ai/, so the AI tab
+# can report usage for the whole fleet instead of only the host the browser happens to be on.
+#
+# Each host records its own turns to data/ai_token_history.db and nothing syncs that file, so
+# without this a host can only ever see its own totals. The tab is careful to say "not collected
+# here" rather than 0 for a partner it cannot see; this script is what turns that into a number.
+#
+# Same trick as conf_sync.sh, and deliberately so — resolve the partner over Tailscale, scp one
+# small file into a tmpfs cache, let the reader treat a missing file as "unknown".
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Gates — PARTNERSHIP_ENABLED, AI_ENABLED, AI_TOKEN_SYNC_ENABLED
+#   2. Per partner:
+#        a. Resolve their Tailscale IP
+#        b. Resolve their SCRIPTS_DIR from their varaverk.cfg (they may be in appdata mode)
+#        c. scp their data/ai_token_history.db → $AI_TOKEN_CACHE_DIR/<slot>.tokens.db
+#
+# Pull only, no push. conf_sync.sh pushes as well because a partner may be unable to reach us
+# and still needs our credentials; nothing here is needed by anyone else, and a reader that
+# fetches its own data controls its own freshness rather than depending on the partner's cron.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# A missing file means unknown, never zero.
+#   The whole point of the tab reporting "not collected here" is that it is a different claim
+#   from "this partner spent nothing". If a partner is dark, unreachable or has never run a turn,
+#   there is simply no cache file, and the reader is expected to say so rather than render a 0
+#   that looks like a measurement.
+#
+# The reader pulls; nobody pushes.
+#   conf_sync.sh pushes as well, because a partner that cannot reach us still needs our
+#   credentials. Nothing here is needed by anyone else, so a host that wants fleet totals fetches
+#   them and owns its own freshness instead of depending on someone else's cron having run.
+#
+# RAM, not flash.
+#   The cache lands in tmpfs. It is a copy of a file that already exists on the partner and is
+#   rebuilt on the next pass, so writing it to flash would cost wear for something that is never
+#   worth surviving a reboot.
+#
+# Same shape as conf_sync.sh, deliberately.
+#   Resolve over Tailscale, scp one small file into a tmpfs cache, let a missing file mean
+#   unknown. A second transport pattern for a second small file would be a second set of
+#   failure modes to learn.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+#   An unreachable partner is not a failure.
+#       HOST2 is expected to be down for long stretches during onboarding. A warn every four
+#       hours would train the operator to ignore this script's output, and the AI diagnostic
+#       path treats every log WARN as actionable. Unresolvable partners are counted and
+#       reported once at info level; only a partner that resolves and then fails to transfer
+#       is treated as an error.
+#
+#   The cache is never written directly.
+#       scp lands on a .part file that is renamed into place, so a transfer interrupted halfway
+#       cannot leave the reader parsing half a ledger. A truncated final row would be skipped by
+#       the field-count check on the PHP side, but a torn file should not reach it at all.
+#
+#   Nothing is ever written back to the partner.
+#       This script only reads. A bug here cannot corrupt a partner's accounting.
+#
+#   The cache is tmpfs and deliberately not preserved.
+#       Unlike the conf cache there is no save/restore pair. Stale counters are worse than
+#       absent ones: absent reads as "not collected here", stale reads as fact.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   AI_ENABLED               Whole AI subsystem gate
+#   AI_TOKEN_SYNC_ENABLED    This script's own toggle (default: true)
+#   PARTNERSHIP_ENABLED      Checked via require_partnership()
+#   SSH_KEY                  Key used for all partner ssh/scp operations
+#
+# load_config.sh
+#
+#   AI_TOKEN_CACHE_DIR       tmpfs directory the tab reads partner ledgers from
+#
+# host*.conf
+#
+#   HOST*  — hostnames used to build the partner list via detect_hosts()
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+#   ai_token_sync.sh             Pull every reachable partner's ledger
+#   ai_token_sync.sh --dry-run   Report what would be pulled, transfer nothing
+#   ai_token_sync.sh --log       Verbose output
+#
+# ==============================================================================================
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/../load_config.sh"
+
+parse_args "$@"
+
+if [[ "$EUID" -ne 0 ]]; then
+    error "Must be run as root"
+    exit 1
+fi
+
+acquire_lock
+
+detect_hosts
+require_partnership
+
+if [[ "${AI_ENABLED:-false}" != true ]]; then
+    log "AI_ENABLED=false — skipping"
+    exit 0
+fi
+
+if [[ "${AI_TOKEN_SYNC_ENABLED:-true}" == false ]]; then
+    log "AI_TOKEN_SYNC_ENABLED=false — skipping"
+    exit 0
+fi
+
+CACHE_DIR="$AI_TOKEN_CACHE_DIR"
+SSH_TIMEOUT=10
+
+# Mirrors conf_sync.sh — the remote may be in appdata storage mode, so its ledger is not
+# necessarily under /boot.
+_remote_scripts_dir() {
+    local ip="$1"
+    local cfg line sd
+    cfg=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+        "root@${ip}" "cat /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null" 2>/dev/null) || true
+    while IFS= read -r line; do
+        [[ "$line" == SCRIPTS_DIR=* ]] || continue
+        sd="${line#SCRIPTS_DIR=}"; sd="${sd//\"/}"; sd="${sd//\'/}"
+        echo "$sd"; return
+    done <<< "$cfg"
+    echo "/boot/config/plugins/varaverk"
+}
+
+[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
+
+if [[ "$DRY_RUN" == false ]]; then
+    mkdir -p "$CACHE_DIR" && chmod 755 "$CACHE_DIR"
+fi
+
+PULLED=0
+OFFLINE=0
+FAILED=0
+
+for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
+    partner_host="${!host_var}"
+    [[ -z "$partner_host" ]] && continue
+    [[ "${host_var,,}" == "${MY_ID,,}" ]] && continue
+
+    partner_slot="${host_var,,}"
+    partner_ip=$(resolve_tailscale_ip "$partner_host" 2>/dev/null || true)
+
+    if [[ -z "$partner_ip" ]]; then
+        log "$partner_host — unresolvable, leaving any cached ledger as-is"
+        (( OFFLINE++ ))
+        continue
+    fi
+
+    # Liveness and ledger presence are separate questions, probed separately on purpose.
+    # Tailscale hands back an IP for a peer that is registered but powered off, so resolving
+    # proves nothing. And a single `ssh test -f` answers both questions at once: it fails
+    # identically whether the host is down or the file is simply absent. Treating that one
+    # failure as "no ledger" would delete a perfectly good cached copy every time the partner
+    # blinked — turning "synced 3h ago" into "not collected here" on a transient.
+    if ! timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+        "root@${partner_ip}" true 2>/dev/null; then
+        log "$partner_host — not answering, keeping any cached ledger as-is"
+        (( OFFLINE++ ))
+        continue
+    fi
+
+    remote_sd=$(_remote_scripts_dir "$partner_ip")
+    remote_db="${remote_sd}/data/ai_token_history.db"
+
+    if [[ "$DRY_RUN" == true ]]; then
+        warn "DRY RUN — would pull $partner_host:$remote_db → $CACHE_DIR/${partner_slot}.tokens.db"
+        continue
+    fi
+
+    # Reachable, but nothing recorded there — a partner with AI off, or one that has simply
+    # never been asked anything. Now that liveness is established this is a real answer, so the
+    # stale copy goes: the tab should say "not collected here", not quote a number from before.
+    if ! timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+        "root@${partner_ip}" "[[ -f '$remote_db' ]]" 2>/dev/null; then
+        log "$partner_host — reachable, but no ledger there yet"
+        rm -f "$CACHE_DIR/${partner_slot}.tokens.db"
+        (( OFFLINE++ ))
+        continue
+    fi
+
+    if timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
+        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+        "root@${partner_ip}:${remote_db}" \
+        "$CACHE_DIR/${partner_slot}.tokens.db.part" 2>/dev/null \
+       && mv -f "$CACHE_DIR/${partner_slot}.tokens.db.part" "$CACHE_DIR/${partner_slot}.tokens.db"; then
+        chmod 644 "$CACHE_DIR/${partner_slot}.tokens.db" 2>/dev/null
+        echo "Pulled ${partner_slot} ledger from $partner_host ✅"
+        (( PULLED++ ))
+    else
+        rm -f "$CACHE_DIR/${partner_slot}.tokens.db.part"
+        warn "Could not pull ${partner_slot} ledger from $partner_host"
+        (( FAILED++ ))
+    fi
+done
+
+# Zero counts are omitted rather than printed. ${VAR:+...} keeps "0" because it is a non-empty
+# string, and a summary that always ends "0 failed" is what teaches you to stop reading it.
+_summary="AI token sync complete — pulled $PULLED"
+(( OFFLINE > 0 )) && _summary+=", $OFFLINE unavailable"
+(( FAILED  > 0 )) && _summary+=", $FAILED failed"
+info "$_summary"
+
+# Only a partner that answered and then failed the transfer is worth an exit code. An absent
+# partner is the normal state whenever a partner is not yet onboarded.
+[[ "$FAILED" -gt 0 ]] && exit 1
+exit 0

AI/lib/chunk.js

@@ -0,0 +1,276 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════════════════════════════════════
+// Chunker — turns repo files into retrieval units.
+//
+// The whole point of the header audit is that chunk boundaries are deterministic here. Bash
+// scripts split on their six section names, markdown on its headings, conf templates on their
+// ━━━ section rules. Nothing is split on a fixed token window, so no chunk ever contains half
+// of one idea and half of another.
+//
+// Every chunk carries its section name as its own field, because that is the metadata that
+// lets retrieval filter by question shape before it ever computes similarity.
+// ═══════════════════════════════════════════════════════════════════════════════════════════════
+
+const fs   = require('fs');
+const path = require('path');
+
+const BASH_SECTIONS = [
+    'PURPOSE', 'OPERATIONAL MODEL', 'DESIGN PRINCIPLES',
+    'OPERATIONAL SAFEGUARDS', 'CONFIGURATION', 'RUNTIME MODES',
+];
+
+// PHP headers reuse the first three names deliberately, then diverge per layer.
+const PHP_SECTIONS = [
+    'PURPOSE', 'OPERATIONAL MODEL', 'DESIGN PRINCIPLES', 'OPERATIONAL SAFEGUARDS',
+    'STATUS', 'EXPORTS', 'REQUEST CONTRACT', 'SIDE EFFECTS', 'RENDERS', 'DEPENDS ON',
+    'CONFIGURATION',
+];
+
+const MIN_CHARS = 40;      // below this a chunk carries no retrievable meaning
+const MAX_CHARS = 6000;    // above this, split on blank lines — protects the embed window
+
+// Banner rules and box-drawing art are everywhere in this repo's headers. They carry no
+// meaning to embed, and a chunk that is mostly rule characters is pure noise in the index.
+// Measure a chunk by what is left after the decoration is removed, not by raw length.
+function meaningful(s) {
+    return s.replace(/[═─━=_#\/*\s|+.-]/g, '').length;
+}
+const MIN_MEANINGFUL = 30;
+
+function isRealHeading(h) {
+    return !!h && /[A-Za-z0-9]/.test(h.replace(/[═─━=_]/g, ''));
+}
+
+function stripPrefix(line, prefix) {
+    // '#   text' -> 'text'   '//   text' -> 'text'
+    const re = new RegExp('^\\s*' + prefix + '\\s?');
+    return line.replace(re, '');
+}
+
+// ── Comment-header sectioning, shared by bash (#) and PHP (//) ────────────────────────────────
+function sectionsFromCommentHeader(text, prefix, names) {
+    const lines = text.split('\n');
+    const nameSet = new Set(names);
+    const found = [];
+
+    // headerEnd matters as much as the section starts. The last section (RUNTIME MODES in bash,
+    // DEPENDS ON in a page) would otherwise run to EOF and sweep up every unrelated comment in
+    // the file — scheduler.php alone contributed an 11k-char chunk of unrelated inline comments.
+    let headerEnd = lines.length;
+    for (let i = 0; i < lines.length; i++) {
+        const raw = lines[i];
+        if (!new RegExp('^\\s*' + prefix).test(raw)) {
+            // Header block ends at the first non-comment, non-blank line past the shebang.
+            // '<?php' and '?>' bracket a PHP header block and are not the end of it.
+            const t = raw.trim();
+            if (t !== '' && !/^#!/.test(t) && t !== '<?php' && t !== '?>' && found.length) {
+                headerEnd = i;
+                break;
+            }
+            continue;
+        }
+        const inner = stripPrefix(raw, prefix).trim();
+        if (nameSet.has(inner)) found.push({ name: inner, start: i });
+    }
+
+    const out = [];
+    for (let k = 0; k < found.length; k++) {
+        const start = found[k].start + 1;
+        const end   = k + 1 < found.length ? found[k + 1].start : headerEnd;
+        const body  = lines.slice(start, end)
+            .filter(l => new RegExp('^\\s*' + prefix).test(l))
+            .map(l => stripPrefix(l, prefix))
+            // drop pure separator rules (════, ────, ━━━) — they carry no meaning
+            .filter(l => !/^[\s═─━=_-]*$/.test(l) || l.trim() === '')
+            .join('\n')
+            .replace(/\n{3,}/g, '\n\n')
+            .trim();
+        if (meaningful(body) >= MIN_MEANINGFUL) {
+            for (const p of splitNamedParagraphs(body))
+                out.push({ section: found[k].name, title: p.title, content: p.content });
+        }
+    }
+    return out;
+}
+
+// ── Named-paragraph sub-chunking ──────────────────────────────────────────────────────────────
+// The header convention writes safeguards and principles as named paragraphs: an unindented
+// title line followed by an indented body. Embedding a whole section as one unit dilutes them —
+// rsync.sh's OPERATIONAL SAFEGUARDS holds eight distinct guarantees in 2.8k chars, and a query
+// about one of them scored below unrelated chunks because the other seven dominated the vector.
+// Splitting on the title lines is what makes a specific question find a specific answer.
+//
+// The section name is carried onto every sub-chunk, so section routing still works; the
+// paragraph title becomes the chunk's heading.
+function splitNamedParagraphs(body) {
+    const lines = body.split('\n');
+    const marks = [];
+    for (let i = 0; i < lines.length; i++) {
+        const l = lines[i];
+        if (!l.trim()) continue;
+        if (/^\s/.test(l)) continue;                 // indented => body, not a title
+        if (/^[-*•]/.test(l.trim())) continue;       // list item, not a title
+        if (l.trim().length > 80) continue;          // a long line is prose, not a heading
+        if (/[.:;,]$/.test(l.trim())) continue;      // ends like a sentence
+
+        // The decisive signal: a real title is followed by an indented body. Wrapped prose is
+        // followed by more unindented prose. Without this check, any short line in a paragraph
+        // that happened to wrap became a spurious chunk boundary mid-sentence.
+        let j = i + 1;
+        while (j < lines.length && !lines[j].trim()) j++;
+        if (j >= lines.length || !/^\s+\S/.test(lines[j])) continue;
+
+        marks.push(i);
+    }
+    // Fewer than two titles means this section is not written as named paragraphs — keep it whole.
+    if (marks.length < 2) return [{ title: null, content: body }];
+
+    const out = [];
+    if (marks[0] > 0) {
+        const pre = lines.slice(0, marks[0]).join('\n').trim();
+        if (meaningful(pre) >= MIN_MEANINGFUL) out.push({ title: null, content: pre });
+    }
+    for (let k = 0; k < marks.length; k++) {
+        const start = marks[k];
+        const end   = k + 1 < marks.length ? marks[k + 1] : lines.length;
+        const title = lines[start].trim();
+        const content = lines.slice(start, end).join('\n').trim();
+        if (meaningful(content) >= MIN_MEANINGFUL) out.push({ title, content });
+    }
+    return out;
+}
+
+// ── Markdown: split on ## headings, keep the heading with its body ─────────────────────────────
+function sectionsFromMarkdown(text) {
+    const lines = text.split('\n');
+    const marks = [];
+    let fence = false;
+
+    for (let i = 0; i < lines.length; i++) {
+        if (/^\s*```/.test(lines[i])) { fence = !fence; continue; }
+        if (fence) continue;
+        // A heading whose text is nothing but rule characters is a banner, not a section. The
+        // house style opens a document with three of them —
+        //     # ━━━━━━━━
+        //     # 🏠  VARAVERK
+        //     # ━━━━━━━━
+        // — and treating each as a boundary split the title into a section of its own, too small
+        // to survive, then gave the paragraph that actually defines the project a chunk headed by
+        // the rule beneath it: no heading, and content opening with 75 identical glyphs. That is
+        // why "what is Varaverk" returned five script PURPOSE headers and never README.md, which
+        // has been indexed the whole time. 35 of the 40 markdown files here open this way.
+        //
+        // Whole-string test, so an ordinary heading containing a dash is unaffected — "Set-up"
+        // does not reduce to empty, and a line of dashes does.
+        if (!/^#{1,3}\s+\S/.test(lines[i])) continue;
+        if (lines[i].replace(/^#+\s*/, '').replace(/[━─═=~_*\-\s]+/gu, '') === '') continue;
+        marks.push(i);
+    }
+    if (!marks.length) return [{ heading: null, content: text.trim() }];
+
+    const out = [];
+    // preamble before the first heading
+    if (marks[0] > 0) {
+        const pre = lines.slice(0, marks[0]).join('\n').trim();
+        if (meaningful(pre) >= MIN_MEANINGFUL) out.push({ heading: null, content: pre });
+    }
+    for (let k = 0; k < marks.length; k++) {
+        const start = marks[k];
+        const end   = k + 1 < marks.length ? marks[k + 1] : lines.length;
+        const heading = lines[start].replace(/^#+\s*/, '').replace(/[━─═]+/g, '').trim();
+        const content = lines.slice(start, end).join('\n').trim();
+        if (meaningful(content) >= MIN_MEANINGFUL) out.push({ heading: isRealHeading(heading) ? heading : null, content });
+    }
+    return out;
+}
+
+// ── Conf templates: split on the ━━━ / ── section rules ───────────────────────────────────────
+function sectionsFromConfTemplate(text) {
+    const lines = text.split('\n');
+    const marks = [];
+    for (let i = 0; i < lines.length; i++) {
+        const m = lines[i].match(/^#\s*[━─]{2,}\s*(.+?)\s*[━─]{2,}\s*$/);
+        if (m && isRealHeading(m[1])) marks.push({ i, name: m[1].trim() });
+    }
+    if (!marks.length) return [];
+
+    const out = [];
+    for (let k = 0; k < marks.length; k++) {
+        const start = marks[k].i;
+        const end   = k + 1 < marks.length ? marks[k + 1].i : lines.length;
+        const content = lines.slice(start, end).join('\n').replace(/\n{3,}/g, '\n\n').trim();
+        if (meaningful(content) >= MIN_MEANINGFUL) out.push({ heading: marks[k].name, content });
+    }
+    return out;
+}
+
+// Oversized chunks split on blank lines rather than mid-sentence.
+function capSize(chunks) {
+    const out = [];
+    for (const c of chunks) {
+        if (c.content.length <= MAX_CHARS) { out.push(c); continue; }
+        const paras = c.content.split(/\n\s*\n/);
+        let buf = [], len = 0, part = 1;
+        const flush = () => {
+            if (!buf.length) return;
+            out.push({ ...c, content: buf.join('\n\n'), part: part++ });
+            buf = []; len = 0;
+        };
+        for (const p of paras) {
+            if (len + p.length > MAX_CHARS && buf.length) flush();
+            buf.push(p); len += p.length + 2;
+        }
+        flush();
+    }
+    return out;
+}
+
+function classify(rel) {
+    const base = path.basename(rel);
+    if (rel.startsWith('Deployment/') && rel.endsWith('.template')) return 'template';
+    // WebGUI page docs, written for whoever is using the tab rather than maintaining it. Their
+    // own kind because every other kind here answers a maintainer's question: an operator asking
+    // "how do I stop this" needs the click path, and a corpus that is three-quarters script
+    // headers will otherwise always answer in conf edits. Matched on the folder, not the
+    // filename, so these can be named whatever reads best.
+    if (rel.startsWith('Plugin/unraid/pages/readme/') && base.endsWith('.md')) return 'ui';
+    if (base.endsWith('.md')) {
+        if (base.startsWith('Manual')) return 'manual';
+        if (base.startsWith('README') || base === 'README.md') return 'readme';
+        return 'doc';
+    }
+    if (base.endsWith('.sh'))  return 'header';
+    if (base.endsWith('.php')) return 'header';
+    return 'other';
+}
+
+function chunkFile(absPath, rel) {
+    const text = fs.readFileSync(absPath, 'utf8');
+    const kind = classify(rel);
+    let raw = [];
+
+    if (kind === 'header' && rel.endsWith('.sh')) {
+        raw = sectionsFromCommentHeader(text, '#', BASH_SECTIONS)
+            .map(s => ({ section: s.section, heading: s.title || null, content: s.content }));
+    } else if (kind === 'header' && rel.endsWith('.php')) {
+        raw = sectionsFromCommentHeader(text, '//', PHP_SECTIONS)
+            .map(s => ({ section: s.section, heading: s.title || null, content: s.content }));
+    } else if (kind === 'template') {
+        raw = sectionsFromConfTemplate(text)
+            .map(s => ({ section: null, heading: s.heading, content: s.content }));
+    } else if (kind === 'readme' || kind === 'manual' || kind === 'doc' || kind === 'ui') {
+        raw = sectionsFromMarkdown(text)
+            .map(s => ({ section: null, heading: s.heading, content: s.content }));
+    }
+
+    return capSize(raw).map(c => ({
+        path: rel,
+        kind,
+        section: c.section || null,
+        heading: c.heading || null,
+        part: c.part || null,
+        content: c.content,
+    }));
+}
+
+module.exports = { chunkFile, classify, BASH_SECTIONS, PHP_SECTIONS };

AI/lib/cli.js

@@ -0,0 +1,209 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════════════════════════════════════
+// CLI bridge — the thin layer the bash entry points call.
+//
+// The bash scripts own configuration, gating, locking and logging, exactly as they do for every
+// other Varaverk job. This file owns only the work that is genuinely awkward in bash: float
+// vector math and SQLite BLOBs. That split follows the existing api_cache_writer.sh precedent —
+// a bash shim in front of the language that fits the task.
+//
+// Every value arrives as an argument or an environment variable read by the caller. This file
+// never reads a conf file itself, so there is exactly one place that decides what the settings
+// are.
+// ═══════════════════════════════════════════════════════════════════════════════════════════════
+
+const fs             = require('fs');
+const { buildIndex } = require('./index.js');
+const { search }     = require('./search.js');
+
+function arg(name, dflt) {
+    const p = `--${name}=`;
+    const hit = process.argv.find(a => a.startsWith(p));
+    return hit ? hit.slice(p.length) : dflt;
+}
+function flag(name) {
+    return process.argv.includes(`--${name}`);
+}
+
+// Token accounting. Writes the same row shape as the WebGUI worker into the same file — one
+// ledger for both paths, or the totals quietly come to mean "whatever the tab happened to do".
+// Skipped silently when the caller passes neither flag, because cli.js has to stay runnable by
+// hand. Best-effort: a failed append must never cost a caller an answer it already has.
+//
+// Trimming is deliberately not done here. The PHP side prunes on write, and duplicating a
+// read-modify-write of the whole file in a second language is how the two drift apart.
+function recordTokens(profile, prompt, completion, tokS) {
+    const db = arg('token-db', ''), host = arg('token-host', '');
+    if (!db || !host || (prompt <= 0 && completion <= 0)) return;
+    const d  = new Date();
+    const p2 = n => String(n).padStart(2, '0');
+    const row = [
+        `${d.getFullYear()}-${p2(d.getMonth() + 1)}-${p2(d.getDate())}`,
+        `${p2(d.getHours())}:${p2(d.getMinutes())}:${p2(d.getSeconds())}`,
+        host, profile, 'cli', prompt, completion,
+        tokS === null ? '' : tokS.toFixed(1),
+    ].join('|') + '\n';
+    try { fs.appendFileSync(db, row); } catch { /* accounting is not the answer */ }
+}
+
+function fail(msg, code = 1) {
+    console.error(msg);
+    process.exit(code);
+}
+
+async function cmdIndex() {
+    const root  = arg('root');
+    const db    = arg('db');
+    const url   = arg('url');
+    const model = arg('model', 'nomic-embed-text');
+    if (!root || !db || !url) fail('index: --root, --db and --url are required');
+
+    const quiet = flag('quiet');
+    let stats;
+    try {
+        stats = await buildIndex({
+            root, dbPath: db, url, model,
+            batch:   parseInt(arg('batch', '32'), 10),
+            timeout: parseInt(arg('timeout', '120000'), 10),
+            force:   flag('force'),
+            dryRun:  flag('dry-run'),
+            onProgress: p => {
+                if (p.error) console.error(`embed batch failed: ${p.error}`);
+                else if (!quiet && p.done % 320 === 0) console.log(`  embedded ${p.done}/${p.total}`);
+            },
+        });
+    } catch (e) {
+        fail(`index failed: ${e.message}`, 2);
+    }
+
+    if (flag('json')) { console.log(JSON.stringify(stats)); return; }
+    if (stats.dryRun) {
+        console.log(`DRY RUN — ${stats.files} file(s) would be indexed, ${stats.chunks} chunk(s) embedded`);
+        console.log(`          ${stats.skipped} unchanged, ${stats.removed} stale entr(ies) would be dropped`);
+        return;
+    }
+    console.log(`indexed ${stats.files} file(s), ${stats.chunks} chunk(s) embedded`);
+    console.log(`  ${stats.skipped} unchanged, ${stats.removed} removed, ${stats.failed} failed`);
+    console.log(`  index now holds ${stats.total} chunk(s)`);
+    // A partial index is usable but not complete — say so in the exit code so a caller can act.
+    if (stats.failed) process.exit(3);
+}
+
+async function cmdSearch() {
+    const db    = arg('db');
+    const url   = arg('url');
+    const model = arg('model', 'nomic-embed-text');
+    const q     = arg('query');
+    if (!db || !url || !q) fail('search: --db, --url and --query are required');
+
+    let r;
+    try {
+        r = await search({
+            dbPath: db, url, embedModel: model, query: q,
+            k:       parseInt(arg('k', '8'), 10),
+            perFile: parseInt(arg('per-file', '3'), 10),
+            section: arg('section', null),
+            kind:    arg('kind', null),
+        });
+    } catch (e) {
+        fail(`search failed: ${e.message}`, 2);
+    }
+
+    if (flag('json')) { console.log(JSON.stringify(r)); return; }
+    if (!r.results.length) { console.log('no matches'); return; }
+    if (r.intents.length) console.log(`intent: ${r.intents.join(', ')}\n`);
+    for (const x of r.results) {
+        const label = x.heading || x.section || '-';
+        console.log(`── ${x.score.toFixed(3)}  ${x.path}  [${x.section || x.kind}] ${label}`);
+        console.log(x.content.split('\n').map(l => '   ' + l).join('\n'));
+        console.log('');
+    }
+}
+
+// Retrieval + generation. The prompt is built here so the context block and the instructions
+// stay in one reviewable place.
+async function cmdAsk() {
+    const db      = arg('db');
+    const url     = arg('url');
+    const embed   = arg('embed-model', 'nomic-embed-text');
+    const gen     = arg('model');
+    const q       = arg('query');
+    const timeout = parseInt(arg('timeout', '240000'), 10);
+    if (!db || !url || !gen || !q) fail('ask: --db, --url, --model and --query are required');
+
+    let r;
+    try {
+        // section and kind must be forwarded here too. They were not, so both filters worked
+        // under `search` and were silently ignored under `ask` — the documented
+        // --section=CONFIGURATION usage retrieved from the whole index and the answer looked
+        // plausible, which is the worst way for a filter to fail.
+        r = await search({
+            dbPath: db, url, embedModel: embed, query: q,
+            k: parseInt(arg('k', '6'), 10), perFile: parseInt(arg('per-file', '2'), 10),
+            section: arg('section', null),
+            kind:    arg('kind', null),
+        });
+    } catch (e) {
+        fail(`retrieval failed: ${e.message}`, 2);
+    }
+    if (!r.results.length) fail('no relevant context found in the index', 4);
+
+    const context = r.results.map((x, i) => {
+        const label = [x.path, x.section, x.heading].filter(Boolean).join(' › ');
+        return `[${i + 1}] ${label}\n${x.content}`;
+    }).join('\n\n');
+
+    const prompt =
+`You are answering questions about Varaverk, a two-server self-healing home media ecosystem.
+
+Answer ONLY from the context below. If the context does not contain the answer, say so plainly
+and name what is missing — do not fill the gap from general knowledge about Linux, Docker or
+rsync, because this system's conventions are frequently not the conventional ones.
+
+Cite the source of each claim as [n]. Be concise and concrete.
+
+CONTEXT
+${context}
+
+QUESTION
+${q}
+
+ANSWER`;
+
+    let res;
+    try {
+        res = await fetch(`${url.replace(/\/$/, '')}/api/generate`, {
+            method: 'POST',
+            headers: { 'Content-Type': 'application/json' },
+            body: JSON.stringify({
+                model: gen, prompt, stream: false,
+                options: { temperature: 0.2, num_ctx: 8192 },
+            }),
+            signal: AbortSignal.timeout(timeout),
+        });
+    } catch (e) {
+        fail(`generation failed: ${e.message}`, 2);
+    }
+    if (!res.ok) fail(`generation HTTP ${res.status}`, 2);
+    const j = await res.json();
+
+    // 'varaverk' rather than a CLI-specific name: this path retrieves and cites, so it is the
+    // same kind of turn the tab's default profile runs, and the two should aggregate together.
+    recordTokens('varaverk', j.prompt_eval_count || 0, j.eval_count || 0,
+                 j.eval_duration > 0 ? (j.eval_count / (j.eval_duration / 1e9)) : null);
+
+    if (flag('json')) {
+        console.log(JSON.stringify({ answer: j.response, sources: r.results.map(x => ({ path: x.path, section: x.section, heading: x.heading, score: x.score })) }));
+        return;
+    }
+    console.log((j.response || '').trim());
+    console.log('\nSources:');
+    r.results.forEach((x, i) => {
+        console.log(`  [${i + 1}] ${[x.path, x.section, x.heading].filter(Boolean).join(' › ')}`);
+    });
+}
+
+const cmd = process.argv[2];
+const table = { index: cmdIndex, search: cmdSearch, ask: cmdAsk };
+if (!table[cmd]) fail(`usage: cli.js <index|search|ask> [--flags]`);
+table[cmd]().catch(e => fail(e.message, 2));

AI/lib/index.js

@@ -0,0 +1,201 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════════════════════════════════════
+// Indexer — chunk the repo, embed each chunk, store vectors in SQLite.
+//
+// Incremental by file mtime: a file whose mtime has not moved since its last index is skipped
+// entirely, so a routine re-index costs seconds rather than re-embedding the whole corpus.
+//
+// Vectors are stored as raw little-endian float32 BLOBs. nomic-embed-text returns L2-normalised
+// vectors, so cosine similarity is a plain dot product at query time — no normalising, no
+// magnitude cache. PHP can read the same blobs with unpack('f*', $blob) when the UI needs them.
+// ═══════════════════════════════════════════════════════════════════════════════════════════════
+
+const fs   = require('fs');
+const path = require('path');
+const { execSync } = require('child_process');
+const { DatabaseSync } = require('node:sqlite');
+const { chunkFile, classify } = require('./chunk.js');
+
+const SCHEMA = `
+CREATE TABLE IF NOT EXISTS vv_files (
+    path     TEXT PRIMARY KEY,
+    mtime    INTEGER NOT NULL,
+    chunks   INTEGER NOT NULL,
+    indexed  INTEGER NOT NULL
+);
+CREATE TABLE IF NOT EXISTS vv_chunks (
+    id       INTEGER PRIMARY KEY,
+    path     TEXT NOT NULL,
+    kind     TEXT NOT NULL,
+    section  TEXT,
+    heading  TEXT,
+    part     INTEGER,
+    content  TEXT NOT NULL,
+    vector   BLOB NOT NULL,
+    indexed  INTEGER NOT NULL
+);
+CREATE INDEX IF NOT EXISTS idx_chunks_path    ON vv_chunks(path);
+CREATE INDEX IF NOT EXISTS idx_chunks_section ON vv_chunks(section);
+CREATE INDEX IF NOT EXISTS idx_chunks_kind    ON vv_chunks(kind);
+CREATE TABLE IF NOT EXISTS vv_meta (k TEXT PRIMARY KEY, v TEXT);
+`;
+
+function openDb(dbPath) {
+    fs.mkdirSync(path.dirname(dbPath), { recursive: true });
+    const db = new DatabaseSync(dbPath);
+    db.exec('PRAGMA journal_mode = WAL;');
+    db.exec('PRAGMA synchronous = NORMAL;');
+    db.exec(SCHEMA);
+    return db;
+}
+
+// Only ever index what git tracks. Configurations/, State_Files/ and data/ are gitignored, which
+// is what makes it structurally impossible for a credential to reach the index — the files that
+// hold them were never in the repo. Do not replace this with a filesystem walk.
+function trackedFiles(root) {
+    return execSync('git ls-files', { cwd: root, maxBuffer: 1 << 26 })
+        .toString().trim().split('\n')
+        .filter(Boolean)
+        .filter(f => classify(f) !== 'other');
+}
+
+async function embedBatch(url, model, inputs, timeoutMs) {
+    const ctl = AbortSignal.timeout(timeoutMs);
+    const res = await fetch(`${url.replace(/\/$/, '')}/api/embed`, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        body: JSON.stringify({ model, input: inputs }),
+        signal: ctl,
+    });
+    if (!res.ok) throw new Error(`embed HTTP ${res.status}: ${(await res.text()).slice(0, 200)}`);
+    const j = await res.json();
+    if (!j.embeddings || j.embeddings.length !== inputs.length)
+        throw new Error(`embed returned ${j.embeddings ? j.embeddings.length : 0} of ${inputs.length}`);
+    return j.embeddings;
+}
+
+function toBlob(vec) {
+    return Buffer.from(Float32Array.from(vec).buffer);
+}
+
+async function buildIndex(opts) {
+    const {
+        root, dbPath, url, model,
+        batch = 32, timeout = 120000, force = false, dryRun = false,
+        onProgress = () => {},
+    } = opts;
+
+    const db  = dryRun ? null : openDb(dbPath);
+    const now = Math.floor(Date.now() / 1000);
+
+    const known = new Map();
+    if (db) for (const r of db.prepare('SELECT path, mtime FROM vv_files').all()) known.set(r.path, r.mtime);
+
+    const files   = trackedFiles(root);
+    const present = new Set(files);
+
+    // Files that left the repo must leave the index with them.
+    let removed = 0;
+    if (db && !force) {
+        for (const p of known.keys()) {
+            if (!present.has(p)) {
+                db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(p);
+                db.prepare('DELETE FROM vv_files  WHERE path = ?').run(p);
+                removed++;
+            }
+        }
+    }
+    if (db && force) { db.exec('DELETE FROM vv_chunks; DELETE FROM vv_files;'); }
+
+    // ── Collect the chunks that actually need embedding ───────────────────────────────────────
+    const pending = [];
+    let skipped = 0, scanned = 0;
+
+    for (const rel of files) {
+        const abs = path.join(root, rel);
+        let st;
+        try { st = fs.statSync(abs); } catch { continue; }
+        const mtime = Math.floor(st.mtimeMs / 1000);
+        scanned++;
+
+        if (!force && known.has(rel) && known.get(rel) === mtime) { skipped++; continue; }
+
+        let chunks = [];
+        try { chunks = chunkFile(abs, rel); } catch (e) { continue; }
+        // A file that yields no chunks still gets a vv_files row so it is not re-chunked every
+        // run. Queued rather than written here, so every database change lands in the single
+        // commit below — a run interrupted mid-embed must leave the index exactly as it was.
+        pending.push({ rel, mtime, chunks });
+    }
+
+    const totalChunks = pending.reduce((n, f) => n + f.chunks.length, 0);
+    if (dryRun) {
+        return { dryRun: true, scanned, skipped, removed, files: pending.length, chunks: totalChunks };
+    }
+
+    // ── Embed in batches, write per file so an interrupted run leaves a consistent index ───────
+    const flat = [];
+    for (const f of pending) for (const c of f.chunks) flat.push({ f, c });
+
+    let done = 0, failed = 0;
+    for (let i = 0; i < flat.length; i += batch) {
+        const slice  = flat.slice(i, i + batch);
+        const inputs = slice.map(x => x.c.content);
+        let vecs;
+        try {
+            vecs = await embedBatch(url, model, inputs, timeout);
+        } catch (e) {
+            failed += slice.length;
+            onProgress({ done, total: flat.length, error: e.message });
+            continue;
+        }
+        slice.forEach((x, k) => { x.c.__vec = vecs[k]; });
+        done += slice.length;
+        onProgress({ done, total: flat.length });
+    }
+
+    const ins = db.prepare(
+        'INSERT INTO vv_chunks (path,kind,section,heading,part,content,vector,indexed) VALUES (?,?,?,?,?,?,?,?)'
+    );
+    const insF = db.prepare('INSERT OR REPLACE INTO vv_files VALUES (?,?,?,?)');
+
+    db.exec('BEGIN');
+    try {
+        for (const f of pending) {
+            // Nothing to index in this file at all — record it so it is not re-chunked next run.
+            if (!f.chunks.length) {
+                db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(f.rel);
+                insF.run(f.rel, f.mtime, 0, now);
+                continue;
+            }
+            const embedded = f.chunks.filter(c => c.__vec);
+            // A file whose chunks all failed to embed keeps its previous rows and its old mtime,
+            // so the next run retries it rather than recording a half-indexed file as current.
+            if (!embedded.length) continue;
+            db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(f.rel);
+            for (const c of embedded) {
+                ins.run(c.path, c.kind, c.section, c.heading, c.part, c.content, toBlob(c.__vec), now);
+            }
+            insF.run(f.rel, f.mtime, embedded.length, now);
+        }
+        db.prepare('INSERT OR REPLACE INTO vv_meta VALUES (?,?)').run('embed_model', model);
+        db.prepare('INSERT OR REPLACE INTO vv_meta VALUES (?,?)').run('last_index', String(now));
+        db.prepare('INSERT OR REPLACE INTO vv_meta VALUES (?,?)').run('dims', '768');
+        db.exec('COMMIT');
+    } catch (e) {
+        db.exec('ROLLBACK');
+        throw e;
+    }
+
+    const stats = {
+        scanned, skipped, removed,
+        files:  pending.length,
+        chunks: done,
+        failed,
+        total:  db.prepare('SELECT COUNT(*) n FROM vv_chunks').get().n,
+    };
+    db.close();
+    return stats;
+}
+
+module.exports = { buildIndex, openDb, toBlob, trackedFiles };

AI/lib/search.js

@@ -0,0 +1,139 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════════════════════════════════════
+// Search — embed a question, score it against the index, return the best chunks.
+//
+// nomic-embed-text returns L2-normalised vectors, so cosine similarity is a plain dot product.
+// At this corpus size (~1.7k chunks x 768 dims) that is a couple of million multiply-adds —
+// under a millisecond, with no vector database and no index structure to maintain.
+//
+// Section routing is the payoff from the header audit. Every chunk knows whether it is a
+// PURPOSE, a DESIGN PRINCIPLES, an OPERATIONAL SAFEGUARDS and so on, so a question's shape can
+// steer retrieval before similarity is even considered. It is applied as a score boost rather
+// than a hard filter — intent detection is a heuristic, and a heuristic should not be able to
+// exclude the one chunk that actually holds the answer.
+// ═══════════════════════════════════════════════════════════════════════════════════════════════
+
+const { DatabaseSync } = require('node:sqlite');
+
+// Question shape → the section most likely to answer it.
+const INTENTS = [
+    { section: 'OPERATIONAL SAFEGUARDS',
+      re: /\b(safe|safety|guard|protect|prevent|fail|failure|abort|refuse|lock|root|timeout|dry.?run|what stops|what happens if|race|corrupt|data.?loss)\b/i },
+    { section: 'CONFIGURATION',
+      re: /\b(variable|var|setting|conf|config|threshold|toggle|which key|what controls|where is .* set|default value|env)\b/i },
+    { section: 'RUNTIME MODES',
+      re: /\b(flag|argument|option|--\w+|how do i run|invoke|cli|command line|status mode|usage)\b/i },
+    { section: 'DESIGN PRINCIPLES',
+      re: /\b(why|rationale|reason|design|decision|deliberate|intentional|on purpose|trade.?off|chose|approach)\b/i },
+    { section: 'OPERATIONAL MODEL',
+      re: /\b(how does .* work|flow|sequence|order|tier|lifecycle|state machine|when does)\b/i },
+    { section: 'EXPORTS',
+      re: /\b(function|export|api surface|what does .* provide|helper|vv_\w+)\b/i },
+    { section: 'PURPOSE',
+      re: /\b(what is|what does .* do|purpose|responsible for|job of)\b/i },
+];
+
+const SECTION_BOOST = 0.06;   // enough to reorder near-ties, not enough to beat a real match
+const KIND_BOOST    = 0.02;   // docs answer "how do I" better than a script header does
+
+function detectIntent(q) {
+    const hits = [];
+    for (const i of INTENTS) if (i.re.test(q)) hits.push(i.section);
+    return hits;
+}
+
+function blobToVec(buf) {
+    const b = Buffer.from(buf);
+    return new Float32Array(b.buffer, b.byteOffset, b.length / 4);
+}
+
+function dot(a, b) {
+    let s = 0;
+    for (let i = 0; i < a.length; i++) s += a[i] * b[i];
+    return s;
+}
+
+async function embedQuery(url, model, text, timeoutMs = 60000) {
+    const res = await fetch(`${url.replace(/\/$/, '')}/api/embed`, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        body: JSON.stringify({ model, input: text }),
+        signal: AbortSignal.timeout(timeoutMs),
+    });
+    if (!res.ok) throw new Error(`embed HTTP ${res.status}`);
+    const j = await res.json();
+    if (!j.embeddings || !j.embeddings[0]) throw new Error('embed returned no vector');
+    return Float32Array.from(j.embeddings[0]);
+}
+
+// Keep at most `perFile` chunks from any one file, so a single large document cannot fill the
+// entire context window and crowd out a better answer living somewhere else.
+function diversify(rows, k, perFile) {
+    const seen = new Map();
+    const out  = [];
+    for (const r of rows) {
+        const n = seen.get(r.path) || 0;
+        if (n >= perFile) continue;
+        seen.set(r.path, n + 1);
+        out.push(r);
+        if (out.length >= k) break;
+    }
+    return out;
+}
+
+async function search(opts) {
+    const {
+        dbPath, url, embedModel, query,
+        k = 8, perFile = 3, section = null, kind = null, minScore = 0.0,
+    } = opts;
+
+    const db = new DatabaseSync(dbPath, { readOnly: true });
+
+    let sql = 'SELECT id,path,kind,section,heading,part,content,vector FROM vv_chunks';
+    const where = [], args = [];
+    if (section) { where.push('section = ?'); args.push(section); }
+    if (kind)    { where.push('kind = ?');    args.push(kind); }
+    if (where.length) sql += ' WHERE ' + where.join(' AND ');
+
+    const rows = db.prepare(sql).all(...args);
+    if (!rows.length) { db.close(); return { results: [], intents: [], scanned: 0 }; }
+
+    const qv  = await embedQuery(url, embedModel, query);
+    let intents = section ? [] : detectIntent(query);
+
+    // "What is Varaverk" and "what is arr_sync.sh" are not the same question, and the PURPOSE
+    // intent cannot tell them apart — it fires on both and boosts every PURPOSE block in the
+    // repository at once. There are a couple of hundred, each genuinely describing the purpose of
+    // something, and each a short sentence containing the word Varaverk. The project's own README
+    // then loses to a script that migrates storage modes, because a paragraph is more diluted
+    // than a one-line summary.
+    //
+    // A question that names the project and no component inside it is asking about the whole, so
+    // PURPOSE is precisely the wrong section to promote. Dropping only that intent, rather than
+    // all of them, leaves "why was Varaverk built this way" still routed to DESIGN PRINCIPLES.
+    const namesProject   = /\bvaraverk\b/i.test(query);
+    const namesComponent = /\b[\w.-]+\.(sh|php|js)\b|\b[A-Z][A-Z0-9]*(_[A-Z0-9]+)+\b/.test(query);
+    const projectLevel   = namesProject && !namesComponent;
+    if (projectLevel) intents = intents.filter(s => s !== 'PURPOSE');
+
+    // The same question wants the top-level prose, which is what the doc kinds are.
+    const wantDoc = projectLevel
+                 || /\b(how do i|steps|procedure|setup|install|troubleshoot|guide)\b/i.test(query);
+
+    const scored = rows.map(r => {
+        let s = dot(qv, blobToVec(r.vector));
+        if (intents.includes(r.section)) s += SECTION_BOOST;
+        if (wantDoc && (r.kind === 'manual' || r.kind === 'readme')) s += KIND_BOOST;
+        return {
+            id: r.id, path: r.path, kind: r.kind, section: r.section,
+            heading: r.heading, part: r.part, content: r.content, score: s,
+        };
+    });
+
+    scored.sort((a, b) => b.score - a.score);
+    const kept = diversify(scored.filter(r => r.score >= minScore), k, perFile);
+    db.close();
+    return { results: kept, intents, scanned: rows.length };
+}
+
+module.exports = { search, detectIntent, blobToVec, dot, embedQuery, INTENTS };

Arrs_Stack/README-Arrs_Stack.md

@@ -200,6 +200,75 @@ Lidarr configured — no HOST1_LIDARR_URL means nothing runs.
 | `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
 | `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
 | `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
+| `arr_download_orphan_cleaner.sh` | Clear orphaned completed downloads out of the SABnzbd Completed folders | Daily |
+| `sonarr_classification_scan.sh` | Detect series sitting in the wrong root (anime / kids / general); `--move` acts | Daily |
+| `radarr_classification_scan.sh` | Same for movies, plus junk-metadata detection via `--remove-junk` | Daily |
+| `lidarr_duplicate_artist_cleanup.sh` | Remove phantom zero-file duplicate artists; flag real ones for review | Daily |
+| `arr_cache_prefill.sh` | Warm the shared tracked-data cache so consumers never read cold | Array start + every 4hr |
+| `arr_corruption_scan.sh` | ffprobe every tracked video for corrupt headers; `--remediate` deletes + re-searches | Weekly |
+| `arr_full_rescan.sh` | Force a real disk↔database reconciliation on all three arrs | Weekly |
+
+---
+
+## ━━━ THE NEWER LAYERS ━━━
+
+The original folder was "delete what the arrs no longer track". These were added as distinct
+failure modes surfaced — each exists because something went wrong that the cleanups could not
+have caught.
+
+### 🗑️  Download-Side Orphans — `arr_download_orphan_cleaner.sh`
+
+Every cleanup script here walks the **library** side. Nothing walked the **download** side —
+so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed
+folders indefinitely. Discovered as **755 GB of orphaned TV downloads, oldest from 2022**,
+filling the cache pool to 89%.
+
+Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and
+acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr
+is skipped entirely, because without it there is no way to tell an active import from an
+orphan. A run total over `DOWNLOAD_ORPHAN_MAX_DELETE_GB` aborts — an abnormally large delete
+is the visible symptom of a partial queue fetch.
+
+### 🎭  Wrong-Root Detection — `sonarr_classification_scan.sh` + `radarr_classification_scan.sh`
+
+Overseerr lets any user request content into the wrong root folder — kids shows into general
+TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre,
+certification, network/studio, original language) and report where the computed classification
+disagrees with the folder the item actually sits in.
+
+Report-only by default. `--move` acts on forward misplacements and adult-content-in-kids-root
+leaks. It deliberately does **not** move non-anime content out of the anime root — deliberate
+style placements (Western animation grouped with anime by choice) are genuine judgment calls.
+
+Both poll the arr's async move command to completion before verifying, because `moveFiles=true`
+flips the database instantly while the physical move is still queued behind others.
+
+### 🎨  Phantom Artists — `lidarr_duplicate_artist_cleanup.sh`
+
+MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one
+holding nothing. Removes only the zero-file side, with `deleteFiles=false` so nothing on disk
+is touched. Pairs where both sides hold files are flagged for review, never auto-resolved.
+
+Gated on the tracked-count floor shared with `lidarr_cleanup.sh` — during a library-wide desync
+both sides of a real duplicate can read as zero-file phantoms.
+
+### 🩺  Corruption + Reconciliation — `arr_corruption_scan.sh` + `arr_full_rescan.sh`
+
+`arr_corruption_scan.sh` ffprobes tracked video files for corrupt headers. Report-only unless
+`--remediate`, which deletes the file record and triggers an explicit re-search. Requires
+repeat detections across separate runs before acting, so a transient probe failure cannot
+delete a healthy file.
+
+`arr_full_rescan.sh` forces a genuine disk↔database reconciliation. Organic scans only touch
+files involved in an import, so an untouched library silently drifts — confirmed when Lidarr
+reported **~23% of its true track count** for 1,004 of 1,357 artists with no scan running and
+every file present on disk.
+
+### ⚡  Cache Warmth — `arr_cache_prefill.sh`
+
+Populates the shared tracked-data cache at array start and every 4 hours, so consumers never
+pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still
+writes through on a cold cache.
 
 ---
 

Arrs_Stack/arr_cache_prefill.sh

@@ -30,6 +30,29 @@
 # An arr not configured on this host (e.g. Lidarr is HOST1-only) is skipped cleanly.
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Warm the Cache, Never Own It
+#   This script only pre-populates what arr_get_tracked_data() would fetch on demand
+#   anyway. Nothing depends on it having run — every consumer still writes through on a
+#   cold cache. It removes latency and staleness, it is not a dependency.
+#
+# Failure Is a No-Op, Not an Error
+#   An arr that never comes up in time simply leaves its cache cold, exactly as if this
+#   script did not exist. That is why a missed prefill is logged rather than notified —
+#   the fallback path is the normal path.
+#
+# Wait Ceiling Matched to the Trigger
+#   The array-start run tolerates a long wait because containers are genuinely still
+#   starting. The 30-minute recurring run does not, because a live fetch takes seconds
+#   and a long wait there would only serve to overlap the next tick.
+#
+# Per-Arr Independence
+#   Each arr is prefilled on its own. One unconfigured or slow-starting arr never
+#   prevents the other two from being warmed.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #

Arrs_Stack/arr_corruption_scan.sh

@@ -62,6 +62,94 @@
 #           logged in Sonarr's history as "UserInvokedSearch").
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Detect Always, Act Only on Request
+#   A bare run probes and reports. Deleting a file the arr believes it has is a
+#   destructive act, so it requires --remediate explicitly. The scan can be scheduled
+#   weekly and read without any risk of it removing media on its own.
+#
+# One Probe Result Is Not Evidence
+#   ffprobe can fail for reasons that have nothing to do with the file — a mid-write
+#   import, an NFS blip, a container restart. Corruption must be observed
+#   CORRUPTION_SCAN_STRIKE_LIMIT times consecutively before remediation acts, and a
+#   single clean re-probe resets the counter.
+#
+# Skip-Cache Over Re-Probing
+#   At 90k+ tracked files a full re-probe every run is not viable. Files unchanged by
+#   mtime and size since they last verified clean are skipped, so each run spends its
+#   time on what actually changed rather than re-proving the library from scratch.
+#
+# Delete the Record, Let the Arr Re-Acquire
+#   Remediation removes the file record and explicitly triggers a search. The arr is
+#   left to obtain a good copy through its normal path — this script never tries to
+#   repair a file in place.
+#
+# Explicit Search, Not the Background Cycle
+#   The re-search is triggered directly rather than left to the arr's own missing-search
+#   cycle, because that cycle skips unmonitored items entirely and would silently leave
+#   an unmonitored corrupt file deleted and never replaced.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   docker exec into the ffprobe container requires root.
+#
+# Lock Acquisition
+#   acquire_lock prevents overlapping runs. Two instances would both probe and could
+#   both count a strike against the same file, reaching the limit in half the intended
+#   number of observations.
+#
+# Host Detection
+#   detect_hosts() aliases the arr URLs, API keys and HOST*_FFPROBE_CONTAINER.
+#
+# jq Dependency Check
+#   Fails fast if jq is missing — the tracked-file lists and every hasFile verification
+#   are parsed with it.
+#
+# Report-Only Default
+#   Nothing is deleted without --remediate.
+#
+# ffprobe Configuration Check
+#   Exits cleanly if FFPROBE_CONTAINER / FFPROBE_BIN are unconfigured for this host.
+#
+# ffprobe Container Health Check
+#   check_container_health() verifies the container is running and healthy before any
+#   probing. Every probe is a docker exec into it — if it is stopped or unhealthy every
+#   exec fails, every file reads as corrupt, and two consecutive runs would clear the
+#   strike limit and hand --remediate the whole library to delete.
+#
+# API Reachability + Version Gate
+#   check_api then check_arr_version per arr. A version mismatch skips that arr rather
+#   than issuing deletes against an API whose file-record endpoints may have moved.
+#
+# Per-Arr Isolation
+#   Sonarr and Radarr run sequentially, and one failing, unconfigured or version-
+#   mismatched arr never blocks the other.
+#
+# Unmapped Path Skip
+#   Files whose arr-side path cannot be mapped into the ffprobe container's mount
+#   namespace are skipped and counted, never probed through a wrong path and never
+#   treated as corrupt because the probe could not see them.
+#
+# Strike Threshold
+#   CORRUPTION_SCAN_STRIKE_LIMIT consecutive corrupt detections are required before
+#   --remediate deletes anything. A transient ffprobe failure cannot trigger a delete,
+#   and a clean re-probe clears the counter.
+#
+# Post-Delete Verification
+#   The DELETE response is never trusted. hasFile is re-checked and must have flipped
+#   false before the re-search is issued, so a failed delete never leaves the arr
+#   searching for something it still believes it has.
+#
+# Targeted Deletion
+#   Only the specific episodefile/moviefile record for the corrupt file is removed —
+#   never the series, movie, or any sibling file.
+#
+# ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
 #
@@ -174,12 +262,20 @@ if [[ -z "$FFPROBE_CONTAINER" || -z "$FFPROBE_BIN" ]]; then
     exit 0
 fi
 
+# Every probe is a docker exec into this container. If it is stopped or unhealthy, every
+# exec fails, every file reads as corrupt, and two such runs would clear the strike limit
+# and hand --remediate an entire library to delete. Abort before probing anything.
+check_container_health "$FFPROBE_CONTAINER" "${DOCKER_TIMEOUT:-30}" "Arr Corruption Scan"
+
 CORRUPTION_SCAN_STATE_FILE="${CORRUPTION_SCAN_STATE_FILE:-$DATA_DIR/corruption_scan_state.tsv}"
 mkdir -p "$(dirname "$CORRUPTION_SCAN_STATE_FILE")"
 touch "$CORRUPTION_SCAN_STATE_FILE"
 
 CORRUPTION_SCAN_STRIKES_FILE="${CORRUPTION_SCAN_STRIKES_FILE:-$DATA_DIR/corruption_scan_strikes.tsv}"
 CORRUPTION_SCAN_STRIKE_LIMIT="${CORRUPTION_SCAN_STRIKE_LIMIT:-2}"
+CORRUPTION_SCAN_MAX_CORRUPT_PCT="${CORRUPTION_SCAN_MAX_CORRUPT_PCT:-10}"
+CORRUPTION_SCAN_MAX_CONSECUTIVE="${CORRUPTION_SCAN_MAX_CONSECUTIVE:-15}"
+CORRUPTION_SCAN_GUARD_MIN_SCANNED="${CORRUPTION_SCAN_GUARD_MIN_SCANNED:-20}"
 mkdir -p "$(dirname "$CORRUPTION_SCAN_STRIKES_FILE")"
 touch "$CORRUPTION_SCAN_STRIKES_FILE"
 
@@ -208,6 +304,19 @@ reset_scan_strikes() {
     [[ -n "$current" && "$current" != "0" ]] && set_scan_strikes "$1" 0
 }
 
+# Bails out of the whole run without committing anything. Safe to call at any point before
+# the commit phase: strikes are queued in memory until then, so an abort leaves the strike
+# file exactly as the previous run left it and deletes nothing.
+abort_scan() {
+    local why="$1"
+    error "Corruption scan ABORTED — $why"
+    error "No strikes recorded and nothing remediated this run — the library was not trusted."
+    [[ -n "${FRESH_CLEAN_TMP:-}" ]] && rm -f "$FRESH_CLEAN_TMP"
+    notify "Corruption scan aborted on $(hostname) ($MY_ID) — $why. Nothing deleted." \
+        "Arr Corruption Scan" "warning"
+    exit 1
+}
+
 # Per-arr API shape differences — everything else in the scan/strike/remediate loop below is
 # identical between Sonarr and Radarr.
 declare -A ARR_FILE_ENDPOINT=(   [sonarr]="episodefile" [radarr]="moviefile" )
@@ -229,6 +338,8 @@ if [[ "$SHOW_STATUS" == true ]]; then
     echo "$ICON_GEAR    State file:        $CORRUPTION_SCAN_STATE_FILE"
     echo "$ICON_GEAR    Strike limit:      $CORRUPTION_SCAN_STRIKE_LIMIT"
     echo "$ICON_GEAR    Remediate:         $REMEDIATE"
+    echo "$ICON_GEAR    Corrupt ceiling:   ${CORRUPTION_SCAN_MAX_CORRUPT_PCT}% of scanned (min ${CORRUPTION_SCAN_GUARD_MIN_SCANNED} scanned)"
+    echo "$ICON_GEAR    Consecutive trip:  $CORRUPTION_SCAN_MAX_CONSECUTIVE"
     echo "$ICON_GEAR    Scan limit:        ${SCAN_LIMIT:-unlimited} (per arr)"
     echo "$ICON_GEAR    Path filter:       ${PATH_FILTER:-none}"
     echo "━━━━━━━━━━━━━━━━━━━━━━━"
@@ -264,13 +375,36 @@ ffprobe_translate_path() {
 # Probes one file. Echoes "clean" or "corrupt:<reason>". Never trusts a truncated/garbled
 # stderr as automatically corrupt — only a real non-empty ffprobe stderr counts.
 probe_file() {
-    local host_path="$1" container_path output
+    local host_path="$1" container_path output rc
     container_path=$(ffprobe_translate_path "$host_path") || { echo "unmapped"; return; }
     output=$(docker exec "$FFPROBE_CONTAINER" "$FFPROBE_BIN" -v error "$container_path" 2>&1)
+    rc=$?
+
+    # docker exec writes its own failures to the same stream ffprobe uses, so a stopped
+    # container or an unreachable daemon is otherwise indistinguishable from a corrupt
+    # header. A stopped container exits 1 with a daemon message; a missing binary exits
+    # 127 — neither is evidence about the file, so both must be caught.
+    if (( rc >= 125 )) \
+        || [[ "$output" == "Error response from daemon:"* \
+        || "$output" == "Cannot connect to the Docker daemon"* \
+        || "$output" == "error during connect:"* ]]; then
+        echo "probe_error:${output//$'\n'/ }"
+        return
+    fi
+
     if [[ -z "$output" ]]; then
         echo "clean"
-    else
+    elif (( rc != 0 )); then
+        # ffprobe could not parse the file — EBML header parsing failed, moov atom not found,
+        # contradictionary STSC and STCO. This is the only class that may be remediated.
         echo "corrupt:${output//$'\n'/ }"
+    else
+        # Exit 0 with stderr output: a recoverable muxing complaint, most commonly
+        # "Referenced QT chapter track not found", which many recent .mp4 releases emit and
+        # which says nothing about playability. Equating any stderr with corruption is what
+        # produced 103 "corrupt" files on 2026-08-23 — 28 of 43 newly scanned Radarr items.
+        # Reported for visibility, never strike-tracked, never remediated.
+        echo "suspect:${output//$'\n'/ }"
     fi
 }
 
@@ -349,7 +483,7 @@ TOTAL_SCANNED=0
 TOTAL_CORRUPT=0
 TOTAL_REMEDIATED=0
 TOTAL_REMEDIATE_FAILED=0
-declare -A ARR_SCANNED ARR_SKIPPED_CACHED ARR_SKIPPED_UNMAPPED ARR_CORRUPT ARR_STRIKE_HELD ARR_REMEDIATED ARR_REMEDIATE_FAILED
+declare -A ARR_SCANNED ARR_SKIPPED_CACHED ARR_SKIPPED_UNMAPPED ARR_CORRUPT ARR_SUSPECT ARR_PROBE_ERRORS ARR_STRIKE_HELD ARR_REMEDIATED ARR_REMEDIATE_FAILED
 
 for arr in sonarr radarr; do
     url_var="${arr^^}_URL"; key_var="${arr^^}_API_KEY"
@@ -439,6 +573,12 @@ for arr in sonarr radarr; do
     STRIKE_HELD=0
     REMEDIATED=0
     REMEDIATE_FAILED=0
+    PROBE_ERRORS=0
+    SUSPECT_COUNT=0
+    CONSECUTIVE_BAD=0
+    QUEUE_PATH=()
+    QUEUE_STRIKES=()
+    QUEUE_ITEM=()
 
     FRESH_CLEAN_TMP=$(mktemp)
 
@@ -473,23 +613,82 @@ for arr in sonarr radarr; do
             continue
         fi
 
+        # A docker-level failure is not evidence about the file. Count it, never queue it.
+        if [[ "$result" == probe_error:* ]]; then
+            (( PROBE_ERRORS++ ))
+            (( CONSECUTIVE_BAD++ ))
+            warn "  ? $host_path — probe failed, NOT counted as corrupt: ${result#probe_error:}"
+            if (( CONSECUTIVE_BAD >= CORRUPTION_SCAN_MAX_CONSECUTIVE )); then
+                abort_scan "$CONSECUTIVE_BAD files in a row failed to probe cleanly"
+            fi
+            continue
+        fi
+
         if [[ "$result" == "clean" ]]; then
+            CONSECUTIVE_BAD=0
             reset_scan_strikes "$host_path"
             echo -e "${host_path}\t${stamp}" >> "$FRESH_CLEAN_TMP"
             [[ "$ENABLE_LOGGING" == true ]] && echo "  $ICON_SUCCESS $host_path"
             continue
         fi
 
-        # corrupt:<reason>
+        # A successful probe that merely warned. Proves the container is alive, so it clears
+        # the consecutive-failure tripwire, but it never becomes a strike.
+        if [[ "$result" == suspect:* ]]; then
+            CONSECUTIVE_BAD=0
+            (( SUSPECT_COUNT++ ))
+            [[ "$ENABLE_LOGGING" == true ]] && warn "  ~ $host_path — ffprobe warning (exit 0), NOT corrupt: ${result#suspect:}"
+            continue
+        fi
+
+        # corrupt:<reason> — queued, NOT committed. Nothing reaches the strike file and nothing
+        # is deleted until this arr has been fully probed and the guards below have passed. A
+        # container that dies mid-scan makes every remaining file read as corrupt, and a delete
+        # cannot be undone — so the destructive half has to wait until the corrupt rate for the
+        # whole run is known. 2026-08-23: one Jellyfin restart produced 103 false positives.
         reason="${result#corrupt:}"
         (( CORRUPT_COUNT++ ))
-        strikes=$(increment_scan_strikes "$host_path")
+        (( CONSECUTIVE_BAD++ ))
+
+        prev_strikes=$(get_scan_strikes "$host_path")
+        prev_strikes="${prev_strikes//[^0-9]/}"
+        strikes=$(( ${prev_strikes:-0} + 1 ))
+
+        QUEUE_PATH+=("$host_path")
+        QUEUE_STRIKES+=("$strikes")
+        QUEUE_ITEM+=("$item")
+
         echo "  $ICON_ERROR CORRUPT: $host_path (strike $strikes/$CORRUPTION_SCAN_STRIKE_LIMIT)"
         [[ "$ENABLE_LOGGING" == true ]] && echo "      $reason"
 
-        if [[ "$REMEDIATE" != true ]]; then
+        if (( CONSECUTIVE_BAD >= CORRUPTION_SCAN_MAX_CONSECUTIVE )); then
-            continue
+            abort_scan "$CONSECUTIVE_BAD files in a row failed to probe cleanly"
         fi
+    done < <(echo "$ITEMS" | jq -c '.[]')
+
+    # ━━━ False-positive guards — run before anything is committed ━━━
+    if (( CORRUPT_COUNT > 0 )); then
+        # The pre-flight check only proves the container was up when the scan started.
+        # Re-check now: a mid-scan death is exactly what this guard exists to catch.
+        check_container_health "$FFPROBE_CONTAINER" "${DOCKER_TIMEOUT:-30}" "Arr Corruption Scan"
+
+        if (( SCANNED >= CORRUPTION_SCAN_GUARD_MIN_SCANNED )); then
+            corrupt_pct=$(( CORRUPT_COUNT * 100 / SCANNED ))
+            if (( corrupt_pct >= CORRUPTION_SCAN_MAX_CORRUPT_PCT )); then
+                abort_scan "$CORRUPT_COUNT of $SCANNED probed files (${corrupt_pct}%) read as corrupt — at or above the ${CORRUPTION_SCAN_MAX_CORRUPT_PCT}% ceiling"
+            fi
+        fi
+    fi
+
+    # ━━━ Guards passed — commit strikes, then remediate whatever reached the limit ━━━
+    for _q in "${!QUEUE_PATH[@]}"; do
+        host_path="${QUEUE_PATH[$_q]}"
+        strikes="${QUEUE_STRIKES[$_q]}"
+        item="${QUEUE_ITEM[$_q]}"
+
+        set_scan_strikes "$host_path" "$strikes"
+
+        [[ "$REMEDIATE" != true ]] && continue
 
         if (( strikes < CORRUPTION_SCAN_STRIKE_LIMIT )); then
             warn "  $host_path — strike $strikes/$CORRUPTION_SCAN_STRIKE_LIMIT, not yet remediating (needs repeat confirmation)"
@@ -498,6 +697,8 @@ for arr in sonarr radarr; do
         fi
         reset_scan_strikes "$host_path"
 
+        file_id=$(echo "$item" | jq -r '.file_id')
+        parent_id=$(echo "$item" | jq -r '.parent_id')
         title=$(echo "$item" | jq -r '.title')
 
         http_code=$(curl -sf -o /dev/null -w "%{http_code}" -X DELETE \
@@ -532,7 +733,7 @@ for arr in sonarr radarr; do
             warn "  $title — deleted and verified, but search trigger returned HTTP $search_code"
             (( REMEDIATE_FAILED++ ))
         fi
-    done < <(echo "$ITEMS" | jq -c '.[]')
+    done
 
     merge_clean_state "$FRESH_CLEAN_TMP"
     rm -f "$FRESH_CLEAN_TMP"
@@ -541,6 +742,8 @@ for arr in sonarr radarr; do
     ARR_SKIPPED_CACHED[$arr]=$SKIPPED_CACHED
     ARR_SKIPPED_UNMAPPED[$arr]=$SKIPPED_UNMAPPED
     ARR_CORRUPT[$arr]=$CORRUPT_COUNT
+    ARR_SUSPECT[$arr]=$SUSPECT_COUNT
+    ARR_PROBE_ERRORS[$arr]=$PROBE_ERRORS
     ARR_STRIKE_HELD[$arr]=$STRIKE_HELD
     ARR_REMEDIATED[$arr]=$REMEDIATED
     ARR_REMEDIATE_FAILED[$arr]=$REMEDIATE_FAILED
@@ -565,6 +768,8 @@ for arr in sonarr radarr; do
     echo "    $ICON_SUCCESS Skipped (cached):    ${ARR_SKIPPED_CACHED[$arr]}"
     echo "    $ICON_WARN    Skipped (unmapped):  ${ARR_SKIPPED_UNMAPPED[$arr]}"
     echo "    $ICON_ERROR   Corrupt found:       ${ARR_CORRUPT[$arr]}"
+    echo "    $ICON_WARN    Warnings (exit 0):   ${ARR_SUSPECT[$arr]} (reported, never remediated)"
+    echo "    $ICON_WARN    Probe errors:        ${ARR_PROBE_ERRORS[$arr]} (not counted as corrupt)"
     if [[ "$REMEDIATE" == true ]]; then
         echo "    $ICON_WARN    Held (strikes):      ${ARR_STRIKE_HELD[$arr]}"
         echo "    $ICON_SUCCESS Remediated:          ${ARR_REMEDIATED[$arr]}"

Arrs_Stack/arr_download_orphan_cleaner.sh

@@ -39,7 +39,8 @@
 #                age gate an entry the arr cannot even name is not going to import: if the
 #                title is in the library and monitored, clearing it lets the arr search a
 #                copy it can actually parse; if it is not in the library, nothing is
-#                tracking it and it is dead weight either way. Guarded — see safeguard 6
+#                tracking it and it is dead weight either way. Guarded — see Non-Empty
+#                Library Requirement below
 #   HELD       — IMPORTABLE entries the arr keeps refusing (XEM-blocked, season-span
 #                files), and everything skipped by a guard → report only, human call
 #
@@ -47,33 +48,131 @@
 # there is no way to tell tracked from orphaned, and guessing means deleting active imports.
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Classify Before Acting
+#   Every entry is placed in exactly one class before anything is deleted, and each class
+#   has its own justification. Nothing is removed because it merely looked unwanted — it is
+#   removed because it matched a category whose deletion rationale is written down above.
+#
+# The Queue Is the Source of Truth
+#   Tracked-vs-orphaned is decided by the arr's own queue, never inferred from filenames or
+#   timestamps. If the queue cannot be read, the arr is skipped entirely rather than
+#   falling back to a weaker signal — a guess here deletes an active import.
+#
+# Deleting Is Recoverable, Deleting Wrong Is Not
+#   The classes that get deleted are ones the arr can re-acquire: junk it could never
+#   import, content the library already has, and entries it cannot even name. Anything
+#   whose loss would be permanent or ambiguous is held and reported for a human instead.
+#
+# Abnormal Volume Means Broken Input
+#   The delete cap exists because the realistic failure mode is bad input, not bad logic —
+#   a partial queue fetch classifies live downloads as orphans, and the only visible
+#   symptom is an unusually large delete total. The cap turns that into a stop.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   1. DOWNLOAD_ORPHAN_CLEANER_ENABLED master toggle
+# Root Enforcement
-#   2. Download dir must exist and live under /mnt/ — refuses to walk anything else
+#   Download folders are written by container users; removing them requires root.
-#   3. Queue fetch must succeed (see above)
+#
-#   4. Age gate — nothing under DOWNLOAD_ORPHAN_AGE days is touched
+# Lock Acquisition
-#   5. Deletion only for JUNK, parse-verified REDUNDANT, and UNMATCHED — never for
+#   acquire_lock "wait" with an EXIT trap releasing all locks, so an interrupted run never
-#      IMPORTABLE or anything a guard has held
+#   strands a lock and the daily orchestrator is never silently skipped.
-#   6. UNMATCHED deletes require the arr to report a non-empty library. An empty or
+#
-#      restoring database answers every parse with "no match", which would condemn the
+# Host Detection
-#      entire download dir; the library is queried directly rather than inferred from the
+#   detect_hosts() runs before any HOST*_-prefixed download dir is resolved.
-#      run's own matches, since a small batch that is legitimately all-unmatched is normal
+#
-#      once daily runs have caught up and would otherwise read as a broken database
+# DOWNLOAD_ORPHAN_CLEANER_ENABLED Toggle
-#   7. Run total over DOWNLOAD_ORPHAN_MAX_DELETE_GB aborts the delete pass and notifies —
+#   Master switch — exits cleanly when disabled.
-#      a queue fetch that returned partial data would classify live downloads as orphans,
+#
-#      and an abnormally large delete total is the visible symptom of exactly that
+# Download Path Restriction
-#      (--i-know-what-im-doing overrides, e.g. for a first run against a known backlog)
+#   The download dir must exist and live under /mnt/. Anything else is refused rather
+#   than walked, so a blank or malformed path can never point the scan at the filesystem
+#   root or a system directory.
+#
+# Queue Fetch Hard Gate
+#   The arr is skipped entirely if its queue cannot be read. Without the queue there is no
+#   way to distinguish tracked from orphaned, and guessing deletes active imports.
+#
+# Age Gate
+#   Nothing under DOWNLOAD_ORPHAN_AGE days is touched, so an entry mid-import is never a
+#   deletion candidate regardless of how it classifies.
+#
+# Deletion Class Restriction
+#   Only JUNK, parse-verified REDUNDANT and UNMATCHED are deleted. IMPORTABLE entries and
+#   anything a guard has held are reported, never removed.
+#
+# Non-Empty Library Requirement
+#   UNMATCHED deletions require the arr to report a non-empty library. An empty or
+#   restoring database answers every parse with "no match", which would condemn the whole
+#   download dir. The library is queried directly rather than inferred from this run's own
+#   match rate — a small batch that is legitimately all-unmatched is normal once daily runs
+#   have caught up, and would otherwise read as a broken database.
+#
+# Delete Volume Cap
+#   A run total over DOWNLOAD_ORPHAN_MAX_DELETE_GB aborts the delete pass and notifies.
+#   --i-know-what-im-doing overrides it for a deliberate first run against a known backlog.
+#
+# Dry Run Support
+#   --dry-run classifies everything and reports, deleting and importing nothing.
 #
 # ==============================================================================================
-# USAGE
+# CONFIGURATION
+# ==============================================================================================
+#
+# host*.conf (resolved per host after detect_hosts())
+#
+#   HOST*_SONARR_DOWNLOAD_DIR / HOST*_RADARR_DOWNLOAD_DIR
+#       Host-side path to the arr's completed-download folder. Absent means that arr's
+#       cleanup is skipped, not an error.
+#
+#   HOST*_SONARR_DOWNLOAD_CONTAINER_DIR / HOST*_RADARR_DOWNLOAD_CONTAINER_DIR
+#       The same folder as the arr container sees it — used when triggering the
+#       DownloadedEpisodesScan / DownloadedMoviesScan path.
+#
+#   SONARR_URL / SONARR_API_KEY / RADARR_URL / RADARR_API_KEY
+#       Aliased by detect_hosts(). A missing URL or key skips that arr.
+#
+# master.conf
+#
+#   DOWNLOAD_ORPHAN_CLEANER_ENABLED
+#       Master toggle (default: true)
+#
+#   DOWNLOAD_ORPHAN_AGE
+#       Days before an entry is eligible at all — younger entries may be mid-import
+#       (default: 7)
+#
+#   DOWNLOAD_ORPHAN_MIN_VIDEO_MB
+#       An entry with no video file above this size is JUNK (default: 50). Sonarr/Radarr only.
+#
+#   DOWNLOAD_ORPHAN_MIN_AUDIO_MB
+#       The same test for Lidarr (default: 2). Separate because a 50M floor would mark
+#       every album folder as JUNK — single tracks rarely reach it.
+#
+#   DOWNLOAD_ORPHAN_KEEP_MARKER
+#       A file with this name inside a download folder pins it — the folder is never
+#       classified or deleted (default: .vv-keep). For lossless rips the library holds
+#       only at lower quality, which REDUNDANT would otherwise sweep.
+#
+#   DOWNLOAD_ORPHAN_MAX_DELETE_GB
+#       Per-run delete budget in GB (default: 100). A backlog above this is drained
+#       safest-first (JUNK, then REDUNDANT, then UNMATCHED) up to the budget, and the
+#       remainder is deferred to the next run rather than aborting the pass.
+#
+#   SONARR_EXTENSIONS / RADARR_EXTENSIONS
+#       Video extensions used to decide whether an entry contains real media
+#
+# ==============================================================================================
+# RUNTIME MODES
 # ==============================================================================================
 #
 #   arr_download_orphan_cleaner.sh                          — daily orchestrator entry
 #   arr_download_orphan_cleaner.sh --dry-run                — classify and report only
 #   arr_download_orphan_cleaner.sh --status                 — show config and exit
-#   arr_download_orphan_cleaner.sh --i-know-what-im-doing   — bypass MAX_DELETE_GB cap
+#   arr_download_orphan_cleaner.sh --i-know-what-im-doing   — bypass MAX_DELETE_GB budget
 #
 # ==============================================================================================
 
@@ -109,6 +208,8 @@ fi
 
 DOWNLOAD_ORPHAN_AGE="${DOWNLOAD_ORPHAN_AGE:-7}"
 DOWNLOAD_ORPHAN_MIN_VIDEO_MB="${DOWNLOAD_ORPHAN_MIN_VIDEO_MB:-50}"
+DOWNLOAD_ORPHAN_MIN_AUDIO_MB="${DOWNLOAD_ORPHAN_MIN_AUDIO_MB:-2}"
+DOWNLOAD_ORPHAN_KEEP_MARKER="${DOWNLOAD_ORPHAN_KEEP_MARKER:-.vv-keep}"
 DOWNLOAD_ORPHAN_MAX_DELETE_GB="${DOWNLOAD_ORPHAN_MAX_DELETE_GB:-100}"
 
 if [[ "${SHOW_STATUS:-false}" == true ]]; then
@@ -116,9 +217,10 @@ if [[ "${SHOW_STATUS:-false}" == true ]]; then
     echo "$ICON_HOST    Identity:        $MY_ID ($LOCAL_SERVER_NAME)"
     echo "$ICON_GEAR    Enabled:         ${DOWNLOAD_ORPHAN_CLEANER_ENABLED}"
     echo "$ICON_TIME    Age gate:        ${DOWNLOAD_ORPHAN_AGE}d"
-    echo "$ICON_DISK    Junk threshold:  ${DOWNLOAD_ORPHAN_MIN_VIDEO_MB}M"
+    echo "$ICON_DISK    Junk threshold:  ${DOWNLOAD_ORPHAN_MIN_VIDEO_MB}M video / ${DOWNLOAD_ORPHAN_MIN_AUDIO_MB}M audio"
     echo "$ICON_SHIELD  Delete cap:      ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G"
-    for arr in SONARR RADARR; do
+    echo "$ICON_SHIELD  Keep marker:     ${DOWNLOAD_ORPHAN_KEEP_MARKER}"
+    for arr in SONARR RADARR LIDARR; do
         dir_var="${MY_ID}_${arr}_DOWNLOAD_DIR"
         echo "$ICON_CLEAN   ${arr}: ${!dir_var:-<not configured>}"
     done
@@ -133,13 +235,16 @@ AGE_CUTOFF=$(( $(date +%s) - DOWNLOAD_ORPHAN_AGE * 86400 ))
 TOTAL_DELETED=0
 TOTAL_DELETED_MB=0
 TOTAL_HELD=0
+TOTAL_DEFERRED=0
+TOTAL_KEPT=0
 TOTAL_SCANS=0
 
 echo "━━━━━ $ICON_CLEAN DOWNLOAD ORPHAN CLEANER ━━━━━"
 echo "$ICON_HOST  Identity: $MY_ID ($LOCAL_SERVER_NAME)"
 [[ "$DRY_RUN" == true ]] && echo "$ICON_SKIP  DRY RUN — nothing will be deleted or imported"
 
-for arr in sonarr radarr; do
+for arr in sonarr radarr lidarr; do
+    api_ver="v3"; [[ "$arr" == "lidarr" ]] && api_ver="v1"
     url_var="${arr^^}_URL"; key_var="${arr^^}_API_KEY"
     arr_url="${!url_var:-}"; arr_key="${!key_var:-}"
     dir_var="${MY_ID}_${arr^^}_DOWNLOAD_DIR"
@@ -159,24 +264,38 @@ for arr in sonarr radarr; do
     echo "━━━ $ICON_SYNC ${arr^} — $dl_dir ━━━"
 
     ver_var="${arr^^}_VERSION_MAJOR"
-    check_arr_version "$arr_url" "$arr_key" "v3" "${!ver_var}" "${arr^}" || {
+    check_arr_version "$arr_url" "$arr_key" "$api_ver" "${!ver_var}" "${arr^}" || {
         warn "${arr^} version check failed — skipping this arr"
         continue
     }
-
+    # min_mb is per-arr because the JUNK test is "contains no real media file". A 50MB floor
-    if [[ "$arr" == "sonarr" ]]; then
+    # is right for video and catastrophic for audio — most single tracks never reach it, so
+    # every music folder would classify as JUNK and be deleted regardless of import state.
+    case "$arr" in
+        sonarr)
             queue_endpoint="queue?pageSize=1000&includeUnknownSeriesItems=true"
             exts_var="SONARR_EXTENSIONS"
             scan_command="DownloadedEpisodesScan"
             library_endpoint="series"
-    else
+            min_mb="$DOWNLOAD_ORPHAN_MIN_VIDEO_MB"
+            ;;
+        radarr)
             queue_endpoint="queue?pageSize=1000&includeUnknownMovieItems=true"
             exts_var="RADARR_EXTENSIONS"
             scan_command="DownloadedMoviesScan"
             library_endpoint="movie"
-    fi
+            min_mb="$DOWNLOAD_ORPHAN_MIN_VIDEO_MB"
+            ;;
+        lidarr)
+            queue_endpoint="queue?pageSize=1000&includeUnknownArtistItems=true"
+            exts_var="LIDARR_EXTENSIONS"
+            scan_command="DownloadedAlbumsScan"
+            library_endpoint="artist"
+            min_mb="$DOWNLOAD_ORPHAN_MIN_AUDIO_MB"
+            ;;
+    esac
 
-    QUEUE_JSON=$(arr_api "$arr_url" "$arr_key" "v3" "$queue_endpoint" "${arr^}") || {
+    QUEUE_JSON=$(arr_api "$arr_url" "$arr_key" "$api_ver" "$queue_endpoint" "${arr^}") || {
         error "${arr^} queue fetch failed — cannot tell tracked from orphaned, skipping this arr"
         continue
     }
@@ -194,11 +313,21 @@ for arr in sonarr radarr; do
     SCAN_PATHS=()
     UNMATCHED_PATHS=()
     UNMATCHED_SIZES=()
-    arr_tracked=0; arr_recent=0; arr_held=0; arr_delete_mb=0
+    arr_tracked=0; arr_recent=0; arr_held=0; arr_delete_mb=0; arr_kept=0
 
     while IFS= read -r entry; do
         base="${entry##*/}"
 
+        # An operator keep-marker outranks every verdict below. Needed because REDUNDANT only
+        # asks "does the library hold this album", not "at what quality" — a lossless rip whose
+        # library copy is MP3 is redundant by that test and would be swept on the next run.
+        # The marker is a file inside the folder rather than a conf list so it survives renames
+        # and cannot drift out of sync with what is actually on disk.
+        if [[ -e "$entry/$DOWNLOAD_ORPHAN_KEEP_MARKER" ]]; then
+            arr_kept=$((arr_kept + 1))
+            continue
+        fi
+
         if [[ -n "${PROTECTED[$base]:-}" ]]; then
             arr_tracked=$((arr_tracked + 1))
             continue
@@ -210,18 +339,30 @@ for arr in sonarr radarr; do
             continue
         fi
 
-        has_video=false
+        # JUNK means "holds no real media". That verdict is only as good as the extension
+        # list, and a missing extension turns real content into a delete — 2026-08-21 the
+        # audio list had no "wv", which classified 23 folders of WavPack lossless (1.5G per
+        # file) as junk. So a folder with large files that are merely *unrecognised* is held
+        # for review, never deleted; only a folder with nothing big in it at all is junk.
+        has_media=false
+        big_unknown=0
         while IFS= read -r f; do
             if has_extension "$f" "${arr_exts[@]}"; then
-                has_video=true
+                has_media=true
                 break
             fi
-        done < <(find "$entry" -type f -size +"${DOWNLOAD_ORPHAN_MIN_VIDEO_MB}"M 2>/dev/null)
+            big_unknown=$((big_unknown + 1))
+        done < <(find "$entry" -type f -size +"${min_mb}"M 2>/dev/null)
 
-        size_mb=$(du -sm "$entry" 2>/dev/null | cut -f1)
+        size_mb=$(dir_size_mb "$entry") || size_mb=0
-        size_mb=${size_mb:-0}
 
-        if [[ "$has_video" == false ]]; then
+        if [[ "$has_media" == false ]] && (( big_unknown > 0 )); then
+            warn "  no recognised media, but $big_unknown large file(s) of unknown type — holding: $base"
+            arr_held=$((arr_held + 1))
+            continue
+        fi
+
+        if [[ "$has_media" == false ]]; then
             DELETE_PATHS+=("$entry")
             DELETE_SIZES+=("$size_mb")
             DELETE_LABELS+=("JUNK")
@@ -230,20 +371,43 @@ for arr in sonarr radarr; do
         fi
 
         enc_title=$(jq -rn --arg t "$base" '$t|@uri')
-        parse=$(arr_api "$arr_url" "$arr_key" "v3" "parse?title=${enc_title}" "${arr^}") || {
+        parse=$(arr_api "$arr_url" "$arr_key" "$api_ver" "parse?title=${enc_title}" "${arr^}") || {
             warn "  parse failed for: $base — holding"
             arr_held=$((arr_held + 1))
             continue
         }
 
-        if [[ "$arr" == "sonarr" ]]; then
+        case "$arr" in
+            sonarr)
                 matched=$(echo "$parse" | jq '(.series != null) and ((.episodes | length) > 0)')
                 missing=$(echo "$parse" | jq '[.episodes[]? | select(.hasFile == false)] | length')
-        else
+                ;;
+            radarr)
                 # Radarr's parse never populates hasFile — movieFileId is the reliable signal
                 matched=$(echo "$parse" | jq '.movie != null')
                 missing=$(echo "$parse" | jq 'if (.movie.movieFileId // 0) > 0 then 0 else 1 end')
+                ;;
+            lidarr)
+                # Lidarr's parse returns albums with statistics:null, so the track count has
+                # to be read back from album/{id} — the same shape of gap as Radarr's hasFile.
+                matched=$(echo "$parse" | jq '(.artist != null) and ((.albums | length) > 0)')
+                missing=1
+                if [[ "$matched" == true ]]; then
+                    album_id=$(echo "$parse" | jq -r '.albums[0].id // empty')
+                    if [[ -z "$album_id" ]]; then
+                        warn "  parse matched but returned no album id: $base — holding"
+                        arr_held=$((arr_held + 1))
+                        continue
                     fi
+                    album_json=$(arr_api "$arr_url" "$arr_key" "$api_ver" "album/$album_id" "${arr^}") || {
+                        warn "  album lookup failed for: $base — holding"
+                        arr_held=$((arr_held + 1))
+                        continue
+                    }
+                    missing=$(echo "$album_json" | jq 'if ((.statistics.trackFileCount // 0) > 0) then 0 else 1 end')
+                fi
+                ;;
+        esac
 
         if [[ "$matched" != true ]]; then
             UNMATCHED_PATHS+=("$entry")
@@ -266,7 +430,7 @@ for arr in sonarr radarr; do
     # fails on a small batch that is legitimately all-unmatched, which is the normal case
     # once daily runs have caught up.
     if (( ${#UNMATCHED_PATHS[@]} > 0 )); then
-        library_count=$(arr_api "$arr_url" "$arr_key" "v3" "$library_endpoint" "${arr^}" | jq 'length' 2>/dev/null)
+        library_count=$(arr_api "$arr_url" "$arr_key" "$api_ver" "$library_endpoint" "${arr^}" | jq 'length' 2>/dev/null)
         if [[ ! "$library_count" =~ ^[0-9]+$ ]] || (( library_count == 0 )); then
             warn "  ${arr^}: library reports ${library_count:-no} titles — cannot trust 'no match', holding ${#UNMATCHED_PATHS[@]} unmatched"
             arr_held=$((arr_held + ${#UNMATCHED_PATHS[@]}))
@@ -280,25 +444,58 @@ for arr in sonarr radarr; do
         fi
     fi
 
-    if (( arr_delete_mb / 1024 > DOWNLOAD_ORPHAN_MAX_DELETE_GB )) && [[ "$I_KNOW" != true ]]; then
+    # The cap is a per-run risk budget, not a reason to do nothing. Aborting the whole pass
-        error "${arr^}: delete total $((arr_delete_mb / 1024))G exceeds cap of ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G — aborting delete pass"
+    # once the backlog exceeds it is self-defeating: the backlog can never shrink below the
-        notify "${arr^} download orphan delete total $((arr_delete_mb / 1024))G exceeds ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G cap on $(hostname) — possible partial queue data, nothing deleted. Re-run with --i-know-what-im-doing if legitimate." \
+    # cap on its own, so every later run aborts too and the pool fills anyway (exactly how
+    # 347G accumulated here by 2026-08-21). Delete in ascending order of risk instead, stop
+    # at the cap, and defer the rest to the next run so a backlog drains over days.
+    #
+    # Live downloads are already protected by DOWNLOAD_ORPHAN_AGE, not by this cap — anything
+    # in flight is younger than the age gate and never reaches classification. That is what
+    # makes draining safe: the partial-queue-data case the cap was written for cannot put a
+    # still-downloading entry in these arrays.
+    cap_mb=$((DOWNLOAD_ORPHAN_MAX_DELETE_GB * 1024))
+    cap_active=true
+    [[ "$I_KNOW" == true || "$DRY_RUN" == true ]] && cap_active=false
+
+    arr_deferred=0; arr_deferred_mb=0; arr_run_mb=0
+
+    if [[ "$cap_active" == true ]] && (( arr_delete_mb > cap_mb )); then
+        warn "  ${arr^}: $((arr_delete_mb / 1024))G classified vs ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G cap — deleting safest-first up to the cap, deferring the rest"
+        notify "${arr^} download orphan backlog is $((arr_delete_mb / 1024))G on $(hostname), above the ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G per-run cap. Draining safest-first; the remainder follows on later runs. Re-run with --i-know-what-im-doing to clear it in one pass." \
             "Download Orphan Cleaner" "warning"
-        unset PROTECTED
+    fi
+
+    # JUNK first (no media at all), then REDUNDANT (parse-verified already in the library),
+    # then UNMATCHED last — it rests on "the arr does not know this title", the weakest of
+    # the three signals, so it is the first thing the cap defers.
+    for pass in JUNK REDUNDANT UNMATCHED; do
+        for i in "${!DELETE_PATHS[@]}"; do
+            [[ "${DELETE_LABELS[$i]}" == "$pass" ]] || continue
+            entry="${DELETE_PATHS[$i]}"
+
+            if [[ "$cap_active" == true ]] && (( arr_run_mb + DELETE_SIZES[i] > cap_mb )); then
+                arr_deferred=$((arr_deferred + 1))
+                arr_deferred_mb=$((arr_deferred_mb + DELETE_SIZES[i]))
                 continue
             fi
 
-    for i in "${!DELETE_PATHS[@]}"; do
-        entry="${DELETE_PATHS[$i]}"
             if [[ "$DRY_RUN" == true ]]; then
                 echo "  $ICON_SKIP would delete [${DELETE_LABELS[$i]}]: ${entry##*/} (${DELETE_SIZES[$i]}M)"
             else
                 echo "  $ICON_TRASH deleting [${DELETE_LABELS[$i]}]: ${entry##*/} (${DELETE_SIZES[$i]}M)"
                 rm -rf "$entry"
             fi
+            arr_run_mb=$((arr_run_mb + DELETE_SIZES[i]))
             TOTAL_DELETED=$((TOTAL_DELETED + 1))
             TOTAL_DELETED_MB=$((TOTAL_DELETED_MB + DELETE_SIZES[i]))
         done
+    done
+
+    if (( arr_deferred > 0 )); then
+        echo "  $ICON_WARN ${arr^}: deferred $arr_deferred entries ($((arr_deferred_mb / 1024))G) to the next run — cap reached"
+        TOTAL_DEFERRED=$((TOTAL_DEFERRED + arr_deferred))
+    fi
 
     for base in "${SCAN_PATHS[@]}"; do
         if [[ -z "$container_dir" ]]; then
@@ -318,8 +515,9 @@ for arr in sonarr radarr; do
         fi
     done
 
-    echo "  $ICON_SUMMARY ${arr^}: $arr_tracked tracked, $arr_recent recent, ${#DELETE_PATHS[@]} deleted ($((arr_delete_mb / 1024))G), ${#SCAN_PATHS[@]} import scans, $arr_held held"
+    echo "  $ICON_SUMMARY ${arr^}: $arr_tracked tracked, $arr_kept kept, $arr_recent recent, $((${#DELETE_PATHS[@]} - arr_deferred)) deleted ($((arr_run_mb / 1024))G), $arr_deferred deferred ($((arr_deferred_mb / 1024))G), ${#SCAN_PATHS[@]} import scans, $arr_held held"
     TOTAL_HELD=$((TOTAL_HELD + arr_held))
+    TOTAL_KEPT=$((TOTAL_KEPT + arr_kept))
     unset PROTECTED
 done
 
@@ -328,6 +526,8 @@ echo "━━━━━ $ICON_DONE SUMMARY ━━━━━"
 echo "$ICON_TRASH   Deleted:       $TOTAL_DELETED ($((TOTAL_DELETED_MB / 1024))G)"
 echo "$ICON_RUN     Import scans:  $TOTAL_SCANS"
 echo "$ICON_WARN    Held:          $TOTAL_HELD"
+echo "$ICON_SKIP    Deferred:      $TOTAL_DEFERRED"
+echo "$ICON_SHIELD  Kept (marker): $TOTAL_KEPT"
 
 # Held alone never notifies — there is always something awaiting review, and on a daily
 # schedule that would be a notification every morning saying nothing happened.

Arrs_Stack/arr_full_rescan.sh

@@ -20,22 +20,91 @@
 # and running them concurrently would just contend for the same disk I/O for no benefit.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Lidarr, then Sonarr, then Radarr — strictly sequential. Per arr:
+#
+#   1. Reachability
+#        → check_api; unreachable skips this arr only
+#
+#   2. Already-scanning check
+#        → a rescan already active (manual, or another script) means skip rather than
+#          stack a second full-disk walk on top of it
+#
+#   3. Capture the before count
+#        → tracked file count read from the arr's own stats
+#
+#   4. Trigger the rescan command
+#        → RescanFolders (Lidarr) / RescanSeries (Sonarr) / RescanMovie (Radarr)
+#
+#   5. Poll to completion
+#        → bounded by ARR_FULL_RESCAN_TIMEOUT
+#
+#   6. Report the delta
+#        → before vs after tracked count, so drift that was corrected is visible
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Proactive, Not Reactive
+#   check_tracked_count_floor() catches stat drift reactively, at the moment some other
+#   script is about to act on bad numbers. This job exists so that drift is corrected on a
+#   schedule instead of being discovered by whichever cleanup happens to trip over it first.
+#
+# Sequential by Design
+#   Each rescan is a full-disk walk. Running three concurrently contends for the same
+#   spindles and finishes no sooner, so the arrs are never parallelised — the slowness is
+#   accepted deliberately rather than optimised into I/O thrash.
+#
+# Never Stack a Scan
+#   An already-running rescan is left alone rather than duplicated. A second concurrent
+#   walk of the same library doubles the I/O cost and returns nothing the first will not.
+#
+# Per-Arr Isolation
+#   One arr being down, slow, or already scanning must never prevent the other two from
+#   being reconciled. Partial coverage beats a skipped run.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root required       — required for consistency across Arrs_Stack/, no direct fs writes here
+# Root Enforcement
-#   acquire_lock "wait" — waits for a prior run to finish rather than skipping or colliding;
+#   Kept for consistency across Arrs_Stack/ — this script makes no direct filesystem writes.
-#                         a full-library rescan across three arrs can run long, worth queuing
+#
-#                         behind rather than silently no-op'ing
+# Lock Acquisition
-#   Reachability check  — check_api before touching an arr; unreachable → skip that arr only
+#   acquire_lock "wait" — waits for a prior run rather than skipping or colliding. A full
-#   Active-rescan check — skips triggering a NEW rescan if one's already active on that arr
+#   rescan across three arrs runs long and is worth queuing behind, not silently dropping.
-#                         (manual trigger, another script) — never stacks a duplicate scan,
+#
-#                         see Tools/arr_rescan_monitor.sh for catching up that arr's cache
+# Host Detection
-#                         once the pre-existing scan finishes instead of waiting for next week
+#   detect_hosts() aliases each arr's URL and API key.
-#   Sequential only     — never runs two arrs' rescans in parallel; each is a heavy full-disk
+#
-#                         walk and concurrent walks would just contend for the same disk I/O
+# jq Dependency Check
-#   Per-arr isolation   — one arr failing, timing out, or being skipped never blocks the others
+#   Fails fast if jq is missing. Both the before/after tracked counts and the command
-#   --dry-run mode      — reports which arrs would be rescanned, triggers nothing
+#   payload are built with jq — without it the counts read empty and every delta would be
+#   reported as if nothing changed.
+#
+# Reachability Check
+#   check_api before touching an arr; unreachable skips that arr only.
+#
+# Active-Rescan Check
+#   Skips triggering a new rescan if one is already active on that arr, so a duplicate
+#   full-disk walk is never stacked. See Tools/arr_rescan_monitor.sh for catching that
+#   arr's cache up once the pre-existing scan finishes, rather than waiting a week.
+#
+# Sequential Only
+#   Two arrs' rescans never run in parallel.
+#
+# Per-Arr Isolation
+#   One arr failing, timing out, or being skipped never blocks the others.
+#
+# Timeout Bound
+#   ARR_FULL_RESCAN_TIMEOUT caps the wait per arr, so a rescan that never completes cannot
+#   hold the weekly window open indefinitely.
+#
+# Dry Run Support
+#   --dry-run reports which arrs would be rescanned and triggers nothing.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -72,6 +141,14 @@ if [[ "$EUID" -ne 0 ]]; then
     exit 1
 fi
 
+# Both the tracked-count reads and the command payload are built with jq — without it the
+# counts read empty and every arr would report a zero delta as if nothing had drifted.
+if ! command -v jq >/dev/null 2>&1; then
+    error "jq not found — required for JSON parsing"
+    notify "Arr full rescan failed on $(hostname) — jq not installed" "Arr Full Rescan" "warning"
+    exit 1
+fi
+
 acquire_lock "wait"
 
 detect_hosts

Arrs_Stack/arr_sync.sh

@@ -44,7 +44,7 @@
 # ==============================================================================================
 #
 # Remote API Access — Cache-First, SSH Fallback
-#   If conf_sync.sh has populated /tmp/.cache/vv/d/ and
+#   If conf_sync.sh has populated /tmp/varaverk/conf/ and
 #   load_config.sh has sourced it, HOST*_<ARR>_API_KEY vars are available
 #   in the environment. Remote functions use them to call the arr API
 #   directly over Tailscale (no SSH, no remote shell). If the cached key

Arrs_Stack/lidarr_cleanup.sh

@@ -88,7 +88,6 @@
 #   jq + curl validation — exits if either tool missing
 #   ARR_DOCKER_TIMEOUT   — container checks protected against daemon hangs (script-local, not common.sh's DOCKER_TIMEOUT)
 #   Duplicate detection  — temp file of tracked paths, grep before delete
-#   platform_require_cmd  — notify script validated before use
 #   Silent by default    — orphans/junk warn(), clean library logs silently
 #
 # ==============================================================================================
@@ -328,7 +327,7 @@ ARTIST_RESPONSE=$(arr_get_tracked_data "lidarr" "$LIDARR_URL" "$LIDARR_API_KEY"
 }
 
 ARTIST_IDS=$(echo "$ARTIST_RESPONSE" | jq -r '.[].id' 2>/dev/null)
-ARTIST_COUNT=$(echo "$ARTIST_IDS" | grep -c "[0-9]" 2>/dev/null || echo 0)
+ARTIST_COUNT=$(echo "$ARTIST_IDS" | grep -c "[0-9]" 2>/dev/null || true)
 
 # Safety Layer 4 — artist count > 0
 if [[ "$ARTIST_COUNT" -eq 0 ]]; then
@@ -458,6 +457,30 @@ NOW=$(date +%s)
 TO_DELETE_FILE="$TMP_DIR/to_delete_paths.txt"
 > "$TO_DELETE_FILE"
 
+# ── Orphan strikes ────────────────────────────────────────────────────────────────────────────
+# Same contract as radarr_cleanup.sh: a file must classify for deletion on
+# LIDARR_ORPHAN_STRIKE_LIMIT consecutive runs before it is removed. Covers the partial
+# classification failure that is too small to trip the tracked-count floor above. The file is
+# rebuilt from each run rather than edited, which is what prunes it.
+LIDARR_ORPHAN_STRIKE_LIMIT="${LIDARR_ORPHAN_STRIKE_LIMIT:-2}"
+STRIKES_FILE="${LIDARR_ORPHAN_STRIKES_FILE:-$DB_DIR/lidarr_orphan_strikes.tsv}"
+mkdir -p "$(dirname "$STRIKES_FILE")" 2>/dev/null || true
+touch "$STRIKES_FILE" 2>/dev/null || true
+STRIKES_NEW="$TMP_DIR/strikes_new.tsv"
+> "$STRIKES_NEW"
+HELD_COUNT=0
+HELD_BYTES=0
+
+orphan_strike_ok() {
+    local path="$1" prev strikes
+    prev=$(wd_state_get "$path" "$STRIKES_FILE"); prev="${prev//[^0-9]/}"
+    strikes=$(( ${prev:-0} + 1 ))
+    printf '%s:%s\n' "$path" "$strikes" >> "$STRIKES_NEW"
+    (( strikes >= LIDARR_ORPHAN_STRIKE_LIMIT )) && return 0
+    warn "  strike $strikes/$LIDARR_ORPHAN_STRIKE_LIMIT — not removing yet: $path"
+    return 1
+}
+
 while read -r FILE_SIZE FILE_CTIME filepath; do
     [[ -z "$filepath" ]] && continue
     FILE_CTIME="${FILE_CTIME%%.*}"
@@ -494,12 +517,14 @@ while read -r FILE_SIZE FILE_CTIME filepath; do
         warn "$ICON_TRASH ORPHAN: $filepath"
         (( ORPHAN_COUNT++ ))
         ORPHAN_BYTES=$(( ORPHAN_BYTES + FILE_SIZE ))
-        echo "$filepath" >> "$TO_DELETE_FILE"
+        if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
+        printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
     else
         log "JUNK: $filepath"
         (( JUNK_COUNT++ ))
         JUNK_BYTES=$(( JUNK_BYTES + FILE_SIZE ))
-        echo "$filepath" >> "$TO_DELETE_FILE"
+        if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
+        printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
     fi
 
 # -printf gets size + mtime directly from find's own stat() during the walk, instead of a
@@ -507,13 +532,43 @@ while read -r FILE_SIZE FILE_CTIME filepath; do
 # 4.3ms), since find already has to stat() every entry anyway to know it's -type f.
 done < <(find "$LIDARR_MUSIC_ROOT" -type f -printf '%s %C@ %p\n' 2>/dev/null)
 
-TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES ))
+# Eligible, not classified: a file still serving its strikes is an orphan but is not queued this
-TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT ))
+# run, so it must not appear in the denominator the budget reports against.
+TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES - HELD_BYTES ))
+TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT - HELD_COUNT ))
+
+# Rebuilt, never edited. Skipped on a dry run: a preview that advanced real counters would make
+# the next real run delete a run early.
+if [[ "$DRY_RUN" == false ]]; then
+    mv "$STRIKES_NEW" "$STRIKES_FILE" 2>/dev/null || warn "Could not update $STRIKES_FILE"
+fi
 
 # ==============================================================================================
 # ━━━ Safety Layer 7 — Deletion Size Threshold ━━━
 # ==============================================================================================
-check_delete_size_threshold "$TOTAL_DELETE_BYTES" "$LIDARR_MAX_DELETE_GB" "Lidarr Cleanup"
+# A per-run budget, not a veto — see apply_delete_budget() in common.sh. The ceiling still caps
+# any single run; it just no longer deadlocks on a backlog larger than itself.
+BUDGET_FILE="$TMP_DIR/to_delete_budgeted.txt"
+
+if [[ "$I_KNOW" == true ]]; then
+    warn "OVERRIDE — --i-know-what-im-doing active, per-run budget not applied"
+    cut -d"$(printf '\t')" -f3- "$TO_DELETE_FILE" > "$BUDGET_FILE"
+    _BUDGET_KEPT_COUNT=$TOTAL_REMOVED; _BUDGET_KEPT_BYTES=$TOTAL_DELETE_BYTES
+    _BUDGET_DEFERRED_COUNT=0; _BUDGET_DEFERRED_BYTES=0; _BUDGET_STUCK=""
+else
+    apply_delete_budget "$TO_DELETE_FILE" "$BUDGET_FILE" "$LIDARR_MAX_DELETE_GB"
+    if [[ -n "$_BUDGET_STUCK" ]]; then
+        error "Single file exceeds the ${LIDARR_MAX_DELETE_GB}GB budget on its own — nothing removed this run"
+        error "  $_BUDGET_STUCK"
+        error "Raise LIDARR_MAX_DELETE_GB or clear this one with --i-know-what-im-doing"
+        notify "Lidarr cleanup stalled on $(hostname) — one file exceeds the ${LIDARR_MAX_DELETE_GB}GB budget" \
+            "Lidarr Cleanup" "warning"
+    elif [[ "$_BUDGET_DEFERRED_COUNT" -gt 0 ]]; then
+        warn "Budget ${LIDARR_MAX_DELETE_GB}GB — removing $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED ($(format_bytes "$_BUDGET_KEPT_BYTES")), deferring $_BUDGET_DEFERRED_COUNT ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) to the next run"
+        notify "Lidarr cleanup removed $(format_bytes "$_BUDGET_KEPT_BYTES") of $(format_bytes "$TOTAL_DELETE_BYTES") on $(hostname) — $_BUDGET_DEFERRED_COUNT file(s) deferred" \
+            "Lidarr Cleanup" "normal"
+    fi
+fi
 
 # ── Execute Deletions ─────────────────────────────────────────────────────────────────────────
 # All safety layers passed — delete orphans and junk. Reuses TO_DELETE_FILE from the
@@ -522,7 +577,7 @@ if [[ "$DRY_RUN" == false ]]; then
     while IFS= read -r filepath; do
         [[ -z "$filepath" ]] && continue
         rm -f "$filepath" 2>/dev/null || error "Failed to delete: $filepath"
-    done < "$TO_DELETE_FILE"
+    done < "$BUDGET_FILE"
 
     info "Cleaning up empty folders..."
     find "$LIDARR_MUSIC_ROOT" -mindepth 1 -type d -empty -delete 2>/dev/null
@@ -547,6 +602,10 @@ echo "$ICON_SHIELD  Protected:       $PROTECTED_COUNT files (cover art, metadata
 echo "$ICON_TRASH   Orphans:         $ORPHAN_COUNT files ($ORPHAN_HUMAN)"
 echo "$ICON_TRASH   Junk:            $JUNK_COUNT files ($JUNK_HUMAN)"
 echo "$ICON_SKIP    Recent skipped:  $RECENT_COUNT files (under ${LIDARR_ORPHAN_AGE} days)"
+[[ "${HELD_COUNT:-0}" -gt 0 ]] && \
+  echo "$ICON_SKIP    Held (strikes):  $HELD_COUNT files ($(format_bytes "$HELD_BYTES")) — under ${LIDARR_ORPHAN_STRIKE_LIMIT} consecutive runs"
+[[ "${_BUDGET_DEFERRED_COUNT:-0}" -gt 0 ]] && \
+  echo "$ICON_SKIP    Deferred:        $_BUDGET_DEFERRED_COUNT files ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) — over the ${LIDARR_MAX_DELETE_GB}GB run budget"
 echo "$ICON_TIME    Duration:        $(format_duration $(( END - START )))"
 echo ""
 
@@ -555,8 +614,10 @@ if [[ "$DRY_RUN" == true ]]; then
 elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
     echo "$ICON_DONE Clean — nothing to remove"
 else
-    warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
+    # What was actually removed, not what was classified. With strikes and a budget in force those
-    notify "Lidarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"         "Lidarr Cleanup" "warning"
+    # differ, and reporting the classification as the outcome is the oldest bug shape here.
+    warn "$ICON_DONE Removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED eligible files ($(format_bytes "$_BUDGET_KEPT_BYTES"))"
+    notify "Lidarr cleanup on $(hostname) — removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED eligible files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"         "Lidarr Cleanup" "warning"
     # Notify Emby to clean missing files — removes ghost entries immediately
     notify_emby_scan
 fi

Arrs_Stack/lidarr_duplicate_artist_cleanup.sh

@@ -52,6 +52,56 @@
 #   output. Only ambiguous (overlapping-album) pairs produce a notification.
 #
 # ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Required by the container interaction and state writes.
+#
+# Lock Acquisition
+#   acquire_lock prevents overlapping runs racing on the same artist IDs.
+#
+# Host Detection
+#   detect_hosts() aliases LIDARR_URL / LIDARR_API_KEY.
+#
+# jq Dependency Check
+#   Fails fast if jq is missing — duplicate detection and every file-count read
+#   depend on it, and an absent jq would evaluate counts to empty and make every
+#   artist look like a zero-file phantom.
+#
+# API Reachability + Version Gate
+#   check_api then check_arr_version against LIDARR_VERSION_MAJOR before any read.
+#
+# Empty Library Abort
+#   A response of 0 artists aborts. An empty list is indistinguishable from
+#   "no duplicates" and must never be read as a clean result.
+#
+# Tracked-Count Floor
+#   check_tracked_count_floor against the baseline shared with lidarr_cleanup.sh.
+#   A library-wide desync — mid full-rescan, for example — makes trackFileCount read
+#   far below reality for many artists at once. Confirmed 2026-07-16: without this,
+#   both sides of a genuinely-real duplicate (ROMES) read as 0-file phantoms and the
+#   wrong one would have been deleted. Deliberately reuses lidarr_cleanup.sh's own
+#   baseline so every script depending on tracked counts shares one answer to "is
+#   Lidarr's data trustworthy right now" rather than forming a separate opinion.
+#
+# Files Never Deleted
+#   Removal passes deleteFiles=false. Only the phantom Lidarr entry is dropped;
+#   nothing on disk is touched, so a wrong call costs a re-add, not media.
+#
+# Phantom-Only Deletion
+#   Only the zero-file side of a duplicate pair is ever removed. If both sides hold
+#   files, or neither does unambiguously, the pair is flagged for manual review
+#   instead — the script never picks a winner between two real artists.
+#
+# Manual Review Reporting
+#   Flagged pairs are named in the summary and notification so an ambiguous
+#   duplicate surfaces as a decision to make rather than disappearing silently.
+#
+# Dry Run Support
+#   --dry-run reports every deletion and exclusion and performs none.
+#
+# ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
 #

Arrs_Stack/playback_aware_lidarr_discovery.sh

@@ -16,7 +16,7 @@
 # Goal: 0–5 meaningful Lidarr adds per week, not bulk imports.
 #
 # ==============================================================================================
-# FLOW
+# OPERATIONAL MODEL
 # ==============================================================================================
 #
 #   1. Fetch play completions from Emby activity log (last LOOKBACK_DAYS days)
@@ -58,11 +58,26 @@
 #   Max adds:  LIDARR_DISCOVERY_MAX_ADDS (default 5) caps each stage
 #
 # ==============================================================================================
-# REQUIREMENTS
+# CONFIGURATION
 # ==============================================================================================
 #
-#   Last.fm API key — required for both stages
+# host*.conf (aliased by detect_hosts())
-#   Configure HOST*_LASTFM_API_KEY in host*.conf
+#
+#   HOST*_LASTFM_API_KEY   Required for both stages — without it the run exits cleanly
+#                          rather than adding anything unscored.
+#   LIDARR_URL / LIDARR_API_KEY   Target arr
+#   EMBY_URL / EMBY_API_KEY       Play history source for seed selection
+#
+# master.conf
+#
+#   LIDARR_DISCOVERY_THRESHOLD          Score required to accept a candidate (0-100)
+#   LIDARR_DISCOVERY_LOOKBACK_DAYS      Emby play history window
+#   LIDARR_DISCOVERY_MIN_PLAYS          Min plays in the window before an artist is evaluated
+#   LIDARR_DISCOVERY_USER_CAP_PCT       Max % of the play score any one user can contribute,
+#                                       so a single heavy listener cannot drive the library
+#   LIDARR_DISCOVERY_MAX_ADDS           Hard cap on artists added per run
+#   LIDARR_DISCOVERY_REJECT_COOLDOWN    Days before a rejected artist is re-evaluated
+#   LIDARR_DISCOVERY_HISTORY            Decision history DB — accepted and rejected
 #
 # ==============================================================================================
 # DESIGN PRINCIPLES
@@ -526,7 +541,7 @@ echo "━━━ $ICON_SYNC Existing Libraries ━━━"
 # fetch when it's stale, and waits out an active rescan before either.
 LIDARR_ARTISTS_JSON=$(arr_get_tracked_data "lidarr" "$LIDARR_URL" "$LIDARR_API_KEY" "v1") || { error "Could not fetch Lidarr artists"; exit 1; }
 LIDARR_NAMES=$(echo "$LIDARR_ARTISTS_JSON" | jq -r '.[].artistName' 2>/dev/null)
-LIDARR_COUNT=$(echo "$LIDARR_NAMES" | grep -c . 2>/dev/null || echo 0)
+LIDARR_COUNT=$(echo "$LIDARR_NAMES" | grep -c . 2>/dev/null || true)
 log "$LIDARR_COUNT artists in Lidarr"
 
 EMBY_LIBRARY_JSON=$(emby_api "Items?IncludeItemTypes=MusicAlbum&Recursive=true&Fields=AlbumArtists&Limit=10000") || {
@@ -534,7 +549,7 @@ EMBY_LIBRARY_JSON=$(emby_api "Items?IncludeItemTypes=MusicAlbum&Recursive=true&F
     EMBY_ARTIST_NAMES=""
 }
 EMBY_ARTIST_NAMES=$(echo "$EMBY_LIBRARY_JSON" | jq -r '.Items[] | .AlbumArtists[]?.Name' 2>/dev/null)
-EMBY_ARTIST_COUNT=$(echo "$EMBY_ARTIST_NAMES" | grep -c . 2>/dev/null || echo 0)
+EMBY_ARTIST_COUNT=$(echo "$EMBY_ARTIST_NAMES" | grep -c . 2>/dev/null || true)
 log "$EMBY_ARTIST_COUNT album artists in Emby library"
 
 # MusicBrainz's canonical name for some artists (e.g. "blink‐182") uses a Unicode

Arrs_Stack/playback_aware_radarr_discovery.sh

@@ -16,7 +16,7 @@
 # Goal: 0–5 meaningful Radarr adds per run, not bulk imports.
 #
 # ==============================================================================================
-# FLOW
+# OPERATIONAL MODEL
 # ==============================================================================================
 #
 #   1. Fetch recently watched movies from Emby (SEED_LIBRARIES, last LOOKBACK_DAYS days)
@@ -56,12 +56,28 @@
 #   Max adds:  RADARR_DISCOVERY_MAX_ADDS (default 5)
 #
 # ==============================================================================================
-# REQUIREMENTS
+# CONFIGURATION
 # ==============================================================================================
 #
-#   TMDB API key — required for Stage 2 recommendations
+# host*.conf (aliased by detect_hosts())
-#   Configure HOST*_TMDB_API_KEY in host*.conf
+#
-#   Free key at: https://www.themoviedb.org/settings/api
+#   HOST*_TMDB_API_KEY   Required for Stage 2 recommendations. Free key at
+#                        https://www.themoviedb.org/settings/api — without it the run
+#                        exits cleanly rather than adding anything unscored.
+#   RADARR_URL / RADARR_API_KEY   Target arr
+#   EMBY_URL / EMBY_API_KEY       Play history source for seed selection
+#
+# master.conf
+#
+#   RADARR_DISCOVERY_THRESHOLD          Score required to accept a candidate (0-100)
+#   RADARR_DISCOVERY_LOOKBACK_DAYS      Emby watch history window
+#   RADARR_DISCOVERY_MAX_SEEDS          Max seed movies taken from Stage 1
+#   RADARR_DISCOVERY_MAX_ADDS           Hard cap on movies added per run
+#   RADARR_DISCOVERY_MIN_VOTE_COUNT     Min TMDB votes for a candidate to be considered
+#   RADARR_DISCOVERY_MIN_RATING         Min TMDB vote_average × 10
+#   RADARR_DISCOVERY_REJECT_COOLDOWN    Days before a rejected movie is re-evaluated
+#   RADARR_DISCOVERY_SEED_LIBRARIES     Emby libraries to draw seed movies from
+#   RADARR_DISCOVERY_HISTORY            Decision history DB — accepted and rejected
 #
 # ==============================================================================================
 # DESIGN PRINCIPLES

Arrs_Stack/playback_aware_sonarr_discovery.sh

@@ -17,7 +17,7 @@
 # Goal: 0–3 meaningful Sonarr adds per run, not bulk imports.
 #
 # ==============================================================================================
-# FLOW
+# OPERATIONAL MODEL
 # ==============================================================================================
 #
 #   1. Fetch all Series from Emby SONARR_EMBY_LIBRARIES — build TMDB+TVDB index
@@ -63,12 +63,29 @@
 #   Max adds:  SONARR_DISCOVERY_MAX_ADDS (default 3) — TV is a larger commitment than movies
 #
 # ==============================================================================================
-# REQUIREMENTS
+# CONFIGURATION
 # ==============================================================================================
 #
-#   TMDB API key — required for Stage 2 recommendations and external_ids lookup
+# host*.conf (aliased by detect_hosts())
-#   Configure HOST*_TMDB_API_KEY in host*.conf
+#
-#   Free key at: https://www.themoviedb.org/settings/api
+#   HOST*_TMDB_API_KEY   Required for Stage 2 recommendations and external_ids lookup.
+#                        Free key at https://www.themoviedb.org/settings/api — without it
+#                        the run exits cleanly rather than adding anything unscored.
+#   SONARR_URL / SONARR_API_KEY   Target arr
+#   EMBY_URL / EMBY_API_KEY       Play history source for seed selection
+#
+# master.conf
+#
+#   SONARR_DISCOVERY_THRESHOLD          Score required to accept a candidate (0-100)
+#   SONARR_DISCOVERY_LOOKBACK_DAYS      Emby episode play history window
+#   SONARR_DISCOVERY_MAX_SEEDS          Max seed series taken from Stage 1
+#   SONARR_DISCOVERY_MAX_ADDS           Hard cap on shows added per run
+#   SONARR_DISCOVERY_MIN_VOTE_COUNT     Min TMDB votes for a candidate to be considered
+#   SONARR_DISCOVERY_MIN_RATING         Min TMDB vote_average × 10
+#   SONARR_DISCOVERY_REJECT_COOLDOWN    Days before a rejected show is re-evaluated
+#   SONARR_DISCOVERY_USER_EPISODE_CAP   Max episodes one user contributes to seed volume
+#   SONARR_DISCOVERY_MONITOR_MODE       Sonarr monitor mode on add
+#   SONARR_DISCOVERY_HISTORY            Decision history DB — accepted and rejected
 #
 # ==============================================================================================
 # DESIGN PRINCIPLES

Arrs_Stack/radarr_classification_scan.sh

@@ -15,8 +15,11 @@
 #   FORWARD  — a movie classified as anime/kids is sitting outside its dedicated root
 #   REVERSE  — a movie sitting inside the kids/anime root doesn't match that classification
 #
-# Report-only. No files are moved and no Radarr API writes happen — this is a detection
+# Report-only by default — no files are moved and no Radarr API writes happen unless a
-# tool. Every rule below was validated against this library's real data before being
+# mode flag is given. Pass --move to relocate forward misplacements, or --remove-junk to
+# delete and import-exclude bad-metadata entries (see OPERATIONAL MODEL below); without
+# those flags this is purely a detection tool. Every rule below was validated against
+# this library's real data before being
 # adopted (see master.conf comments above the curated lists) — this is not a generic
 # genre-matcher, it's tuned specifically against the false-positive traps that showed up
 # when testing looser rules (documented per-rule below).
@@ -52,13 +55,38 @@
 #   blocklist and likely nothing legitimate to redownload under that exact TMDb match.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Report pass (always):
+#   check_api → check_arr_version → arr_get_tracked_data (cache-first, one call)
+#   → classify every movie → report FORWARD, REVERSE and JUNK findings
+#
+# Remove-junk pass (--remove-junk, runs first when combined with --move):
+#   One entry at a time, halting on the first failure.
+#   DELETE with deleteFiles=false and addImportExclusion=true — the Radarr entry is
+#   removed and blocked from re-adding, files on disk are never touched. is_junk
+#   requires hasFile == false, so there is no file behind these entries anyway.
+#   Verified by re-fetching and requiring a 404 before counting as removed.
+#
+# Move pass (--move):
+#   One movie at a time, verified after each.
+#   hasFile == true  → moveFiles=true, then poll the async MoveMovie command to
+#                      "completed" (bounded by RADARR_MOVE_POLL_TIMEOUT) before the
+#                      DB-field check — the DB flips instantly while the physical
+#                      move is still queued.
+#   hasFile == false → correct rootFolderPath/path and trigger MoviesSearch instead;
+#                      there is nothing to move.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
-# Report, Don't Act
+# Report by Default, Act Only on Request
-#   This script never calls Radarr's write API and never touches a file. Every finding is
+#   A bare run never calls Radarr's write API and never touches a file — every finding is
-#   a candidate for a human decision — moving media and re-pointing Radarr's tracking is a
+#   just a candidate. Acting on them requires an explicit --move or --remove-junk flag, so
-#   separate, deliberate follow-up action, not something this scan does automatically.
+#   the scan can be scheduled and re-run freely while the curated lists are being tuned
+#   without any risk of it rearranging the library on its own.
 #
 # Curated Lists, Not Bare Genre/Cert Matching
 #   Every signal used here failed at least once as a bare/standalone check during rule
@@ -72,6 +100,71 @@
 #   if the shared cache is warm) regardless of library size.
 #
 # ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Required by the container interaction and state writes.
+#
+# Lock Acquisition
+#   acquire_lock, plus acquire_lock "wait" around the write passes with an EXIT trap
+#   releasing all locks, so an interrupted run never leaves a lock behind.
+#
+# Host Detection
+#   detect_hosts() aliases RADARR_URL / RADARR_API_KEY / the root literals.
+#
+# curl + jq Dependency Check
+#   Fails fast if either is missing — every classification signal is parsed with jq.
+#
+# Report-Only Default
+#   No write happens without --move or --remove-junk.
+#
+# Required Var Check
+#   require_var on RADARR_URL and RADARR_API_KEY before any request.
+#
+# API Reachability + Version Gate
+#   check_api then check_arr_version against RADARR_VERSION_MAJOR. A major version bump
+#   can move or rename the fields every rule depends on, so a mismatch aborts rather
+#   than classifying against an unknown schema.
+#
+# Empty Library Abort
+#   A response of 0 movies aborts — an empty list is indistinguishable from a clean
+#   library and would otherwise report success during an API fault.
+#
+# Unconfigured Root Skip
+#   A blank kids/anime root skips that category's checks rather than comparing paths
+#   against an empty string.
+#
+# Files Never Deleted
+#   Junk removal passes deleteFiles=false. Only the Radarr entry is removed, and
+#   addImportExclusion=true stops it being re-added. is_junk additionally requires
+#   hasFile == false, so these entries have nothing on disk in the first place.
+#
+# One At A Time, Stop On First Failure
+#   Both write passes process one entry at a time and halt on the first failure rather
+#   than continuing through the library.
+#
+# Post-Write Verification
+#   Removal is confirmed by re-fetching and requiring a 404. Moves are confirmed by
+#   re-fetching and checking rootFolderPath and hasFile. The API response alone is
+#   never treated as proof.
+#
+# Async Move Completion Polling
+#   moveFiles=true flips the DB instantly while the physical move is a separate async
+#   MoveMovie command. Each move polls its own command to "completed" (bounded by
+#   RADARR_MOVE_POLL_TIMEOUT) before the DB-field check, so a batch cannot report
+#   everything moved while files are still queued at the old path.
+#
+# Junk Vote Threshold
+#   RADARR_JUNK_MIN_VOTES gates junk detection alongside hasFile == false and a null
+#   imdbId. All three must hold — a thin-metadata entry that actually has a file, or
+#   has an IMDb ID, is never treated as junk.
+#
+# Post-Write Cache Refresh
+#   The tracked-data cache is refreshed after writes so no other arr script reads a
+#   stale rootFolderPath or a movie that no longer exists.
+#
+# ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
 #

Arrs_Stack/radarr_cleanup.sh

@@ -98,7 +98,6 @@
 #   jq + curl validation — exits if either tool missing
 #   ARR_DOCKER_TIMEOUT   — container checks protected against daemon hangs (script-local, not common.sh's DOCKER_TIMEOUT)
 #   notify_emby_scan()   — triggers Emby clean after deletion
-#   platform_require_cmd  — notify script validated before use
 #   Silent by default    — orphans/junk warn(), clean library logs silently
 #
 # ==============================================================================================
@@ -315,7 +314,7 @@ MOVIES_RESPONSE=$(arr_get_tracked_data "radarr" "$RADARR_URL" "$RADARR_API_KEY"
 }
 
 MOVIE_IDS=$(echo "$MOVIES_RESPONSE" | jq -r '.[].id' 2>/dev/null)
-MOVIE_COUNT=$(echo "$MOVIE_IDS" | grep -c "." 2>/dev/null || echo 0)
+MOVIE_COUNT=$(echo "$MOVIE_IDS" | grep -c "." 2>/dev/null || true)
 
 # Safety Layer 4 — movie count > 0
 if [[ "$MOVIE_COUNT" -eq 0 ]]; then
@@ -451,9 +450,46 @@ NOW=$(date +%s)
 # just delete them directly instead of re-walking and re-classifying every SCAN_ROOTS entry a
 # second time (2026-07-17) — the size-threshold check below needs to know the total before
 # deleting anything, not before knowing what to delete.
+# Carries size and ctime alongside the path now, because the budget pass below has to order by
+# age and stop at a byte ceiling — neither of which a bare path list can answer.
 TO_DELETE_FILE="$TMP_DIR/to_delete_paths.txt"
 > "$TO_DELETE_FILE"
 
+# ── Orphan strikes ────────────────────────────────────────────────────────────────────────────
+# A file must classify for deletion on RADARR_ORPHAN_STRIKE_LIMIT consecutive runs before it is
+# actually removed. Gate 6 already refuses a run whose tracked count collapsed; this covers the
+# partial failure underneath that threshold — one root folder failing to enumerate makes its
+# movies look orphaned while the overall percentage still looks fine, and a transient fault will
+# not reproduce on the next run.
+#
+# The file is REBUILT from this run's classifications rather than edited in place, which is what
+# prunes it: anything that stopped being an orphan simply is not written again, so a file that
+# Radarr re-adopts loses its strikes without needing a reset pass to find it.
+#
+# Keyed by host path, which is why this could not have worked before 2026-08-26 — wd_state_set
+# built a regex from the key, and a release tag like [Bluray-1080p] holds the reversed range 1-0,
+# so every write truncated the store to one line. See common.sh.
+RADARR_ORPHAN_STRIKE_LIMIT="${RADARR_ORPHAN_STRIKE_LIMIT:-2}"
+STRIKES_FILE="${RADARR_ORPHAN_STRIKES_FILE:-$DB_DIR/radarr_orphan_strikes.tsv}"
+mkdir -p "$(dirname "$STRIKES_FILE")" 2>/dev/null || true
+touch "$STRIKES_FILE" 2>/dev/null || true
+STRIKES_NEW="$TMP_DIR/strikes_new.tsv"
+> "$STRIKES_NEW"
+HELD_COUNT=0
+HELD_BYTES=0
+
+# Records this run's strike for a file and says whether it has served enough of them.
+# Returns 0 when the file may be deleted, 1 when it is still accruing.
+orphan_strike_ok() {
+    local path="$1" prev strikes
+    prev=$(wd_state_get "$path" "$STRIKES_FILE"); prev="${prev//[^0-9]/}"
+    strikes=$(( ${prev:-0} + 1 ))
+    printf '%s:%s\n' "$path" "$strikes" >> "$STRIKES_NEW"
+    (( strikes >= RADARR_ORPHAN_STRIKE_LIMIT )) && return 0
+    warn "  strike $strikes/$RADARR_ORPHAN_STRIKE_LIMIT — not removing yet: $path"
+    return 1
+}
+
 while read -r FILE_SIZE FILE_CTIME filepath; do
     [[ -z "$filepath" ]] && continue
     FILE_CTIME="${FILE_CTIME%%.*}"
@@ -488,12 +524,14 @@ while read -r FILE_SIZE FILE_CTIME filepath; do
         warn "$ICON_TRASH ORPHAN: $filepath"
         (( ORPHAN_COUNT++ ))
         ORPHAN_BYTES=$(( ORPHAN_BYTES + FILE_SIZE ))
-        echo "$filepath" >> "$TO_DELETE_FILE"
+        if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
+        printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
     else
         log "JUNK: $filepath"
         (( JUNK_COUNT++ ))
         JUNK_BYTES=$(( JUNK_BYTES + FILE_SIZE ))
-        echo "$filepath" >> "$TO_DELETE_FILE"
+        if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
+        printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
     fi
 
 # -printf gets size + mtime directly from find's own stat() during the walk, instead of a
@@ -505,13 +543,94 @@ done < <(
     done | sort -u
 )
 
-TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES ))
+# Eligible, not classified. A file still serving its strikes was counted as an orphan above — it
-TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT ))
+# is one — but it is not going to be deleted this run, so it must not appear in the denominator
+# the budget reports against or the run claims to have skipped work it never queued.
+TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES - HELD_BYTES ))
+TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT - HELD_COUNT ))
+
+# Rebuilt, never edited: a path absent from this run is absent from the file, so a file Radarr
+# re-adopts drops its strikes with no reset pass needed. Skipped on a dry run — a preview that
+# advanced real strike counters would make the next real run delete a run early.
+if [[ "$DRY_RUN" == false ]]; then
+    mv "$STRIKES_NEW" "$STRIKES_FILE" 2>/dev/null || warn "Could not update $STRIKES_FILE"
+fi
 
 # ==============================================================================================
 # ━━━ Safety Layer 7 — Deletion Size Threshold ━━━
 # ==============================================================================================
-check_delete_size_threshold "$TOTAL_DELETE_BYTES" "$RADARR_MAX_DELETE_GB" "Radarr Cleanup"
+# The ceiling is a per-run budget, not a veto. It still means what it always meant — no single run
+# removes more than RADARR_MAX_DELETE_GB — but a backlog larger than the ceiling now drains over
+# consecutive nights instead of failing the orchestrator forever on a queue it cannot clear.
+# ── AI note (AI_ASSIST_CLEANUP) ───────────────────────────────────────────────────────────────
+# Describes the shape of what was classified. It decides nothing: the eligible set, the budget and
+# the strikes are all settled above and none of them read this. Switch AI_ASSIST_CLEANUP off and
+# the run removes exactly the same files — the log just loses a paragraph.
+#
+# ctime clustering is the signal worth surfacing. A normal upgrade cycle dribbles in over weeks; a
+# lump sharing one narrow ctime window with mtimes spread across months is a bulk write-back, which
+# is what a partnership merge against a partner holding older copies produces. That distinction
+# took a person an evening on 2026-08-26 and is the whole reason this note exists.
+if [[ "$ORPHAN_COUNT" -gt 0 ]] && [[ -s "$TO_DELETE_FILE" ]]; then
+    _ai_ev=$(awk -F'\t' '
+        { n++; bytes += $1
+          c = int($2)
+          if (cmin == 0 || c < cmin) cmin = c
+          if (c > cmax) cmax = c
+          bucket[int(c / 21600)]++ }
+        END {
+            for (b in bucket) if (bucket[b] > top) { top = bucket[b] }
+            printf "files=%d bytes_gb=%.1f ctime_span_hours=%.1f largest_6h_ctime_bucket=%d\n",
+                   n, bytes/1073741824, (cmax-cmin)/3600, top
+        }' "$TO_DELETE_FILE")
+    _ai_mt=$(cut -d"$(printf '\t')" -f3 "$TO_DELETE_FILE" | head -8 \
+             | while IFS= read -r p; do [[ -f "$p" ]] && \
+                 printf '%s  %s\n' "$(stat -c %y "$p" 2>/dev/null | cut -c1-7)" "$(basename "$p")"; done)
+
+    _ai_note=$(ai_assist_note AI_ASSIST_CLEANUP "You are looking at files an automated media-library cleanup has classified for deletion on an Unraid server. They are files on disk that the Radarr database no longer references.
+
+EVIDENCE
+$_ai_ev
+sample (modification month, then path):
+$_ai_mt
+
+A normal quality-upgrade cycle produces orphans whose ctimes are spread out over weeks, because each upgrade happens on its own day. A bulk event - a sync or restore writing files back onto this host - produces orphans sharing one narrow ctime window while their modification times stay spread across months, because the copy preserves modification time but resets ctime.
+
+In no more than three sentences, say which of those two this looks like and name the numbers above that support it. Do not recommend an action. Do not speculate beyond the evidence given.") || _ai_note=""
+
+    if [[ -n "$_ai_note" ]]; then
+        echo ""
+        echo "━━━ $ICON_GEAR AI note on this classification ━━━"
+        printf '%s\n' "$_ai_note"
+    fi
+    unset _ai_ev _ai_mt
+fi
+
+BUDGET_FILE="$TMP_DIR/to_delete_budgeted.txt"
+
+if [[ "$I_KNOW" == true ]]; then
+    warn "OVERRIDE — --i-know-what-im-doing active, per-run budget not applied"
+    cut -d"$(printf '\t')" -f3- "$TO_DELETE_FILE" > "$BUDGET_FILE"
+    _BUDGET_KEPT_COUNT=$TOTAL_REMOVED; _BUDGET_KEPT_BYTES=$TOTAL_DELETE_BYTES
+    _BUDGET_DEFERRED_COUNT=0; _BUDGET_DEFERRED_BYTES=0; _BUDGET_STUCK=""
+else
+    apply_delete_budget "$TO_DELETE_FILE" "$BUDGET_FILE" "$RADARR_MAX_DELETE_GB"
+
+    if [[ -n "$_BUDGET_STUCK" ]]; then
+        # One file larger than the whole budget can never fit, so it would be re-found and
+        # re-deferred every night. Name it rather than loop on it silently.
+        error "Single file exceeds the ${RADARR_MAX_DELETE_GB}GB budget on its own — nothing removed this run"
+        error "  $_BUDGET_STUCK"
+        error "Raise RADARR_MAX_DELETE_GB or clear this one with --i-know-what-im-doing"
+        notify "Radarr cleanup stalled on $(hostname) — one file exceeds the ${RADARR_MAX_DELETE_GB}GB budget" \
+            "Radarr Cleanup" "warning"
+    elif [[ "$_BUDGET_DEFERRED_COUNT" -gt 0 ]]; then
+        warn "Budget ${RADARR_MAX_DELETE_GB}GB — removing $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED ($(format_bytes "$_BUDGET_KEPT_BYTES")), deferring $_BUDGET_DEFERRED_COUNT ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) to the next run"
+        warn "Oldest first — the deferred files are the newest and are re-evaluated tomorrow"
+        notify "Radarr cleanup removed $(format_bytes "$_BUDGET_KEPT_BYTES") of $(format_bytes "$TOTAL_DELETE_BYTES") on $(hostname) — $_BUDGET_DEFERRED_COUNT file(s) deferred to the next run" \
+            "Radarr Cleanup" "normal"
+    fi
+fi
 
 # ── Execute Deletions ─────────────────────────────────────────────────────────────────────────
 # Reuses TO_DELETE_FILE from the classification pass above instead of re-walking and
@@ -520,7 +639,7 @@ if [[ "$DRY_RUN" == false ]]; then
     while IFS= read -r filepath; do
         [[ -z "$filepath" ]] && continue
         rm -f "$filepath" 2>/dev/null || error "Failed to delete: $filepath"
-    done < "$TO_DELETE_FILE"
+    done < "$BUDGET_FILE"
 
     info "Cleaning up empty folders..."
     for host_path in "${SCAN_ROOTS[@]}"; do
@@ -546,6 +665,10 @@ echo "$ICON_SHIELD  Protected:       $PROTECTED_COUNT files (artwork, subtitles,
 echo "$ICON_TRASH   Orphans:         $ORPHAN_COUNT files ($ORPHAN_HUMAN)"
 echo "$ICON_TRASH   Junk:            $JUNK_COUNT files ($JUNK_HUMAN)"
 echo "$ICON_SKIP    Recent skipped:  $RECENT_COUNT files (under ${RADARR_ORPHAN_AGE} days)"
+[[ "${HELD_COUNT:-0}" -gt 0 ]] && \
+  echo "$ICON_SKIP    Held (strikes):  $HELD_COUNT files ($(format_bytes "$HELD_BYTES")) — under ${RADARR_ORPHAN_STRIKE_LIMIT} consecutive runs"
+[[ "${_BUDGET_DEFERRED_COUNT:-0}" -gt 0 ]] && \
+  echo "$ICON_SKIP    Deferred:        $_BUDGET_DEFERRED_COUNT files ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) — over the ${RADARR_MAX_DELETE_GB}GB run budget"
 echo "$ICON_TIME    Duration:        $(format_duration $(( END - START )))"
 echo ""
 
@@ -554,8 +677,10 @@ if [[ "$DRY_RUN" == true ]]; then
 elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
     echo "$ICON_DONE Clean — nothing to remove"
 else
-    warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
+    # What was actually removed, not what was classified. With a budget in force those differ, and
-    notify "Radarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
+    # reporting the classification as the outcome is the oldest bug shape in this codebase.
+    warn "$ICON_DONE Removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED classified files ($(format_bytes "$_BUDGET_KEPT_BYTES"))"
+    notify "Radarr cleanup on $(hostname) — removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED classified files ($(format_bytes "$_BUDGET_KEPT_BYTES"))$([[ "${_BUDGET_DEFERRED_COUNT:-0}" -gt 0 ]] && echo ", $_BUDGET_DEFERRED_COUNT deferred")" \
         "Radarr Cleanup" "warning"
     # Notify Emby to clean missing files — removes ghost entries immediately
     notify_emby_scan

Arrs_Stack/sonarr_classification_scan.sh

@@ -61,9 +61,15 @@
 # no Sonarr equivalent for the same reason.
 #
 # ==============================================================================================
-# MOVE MODE (--move)
+# OPERATIONAL MODEL
 # ==============================================================================================
 #
+# Report pass (always):
+#   check_api → check_arr_version → arr_get_tracked_data (cache-first, one call)
+#   → classify every series → report FORWARD and REVERSE disagreements → exit
+#
+# Move pass (--move only), described in detail below:
+#
 # Acts on FORWARD misplacements (classified anime/kids, sitting in the wrong root) and on
 # REVERSE-KIDS leaks (adult certification sitting in the kids root — moved back to
 # SONARR_GENERAL_ROOT). Does NOT act on REVERSE-ANIME leaks — those are genuine judgment
@@ -93,6 +99,70 @@
 #   of library size. Refreshed after --move writes so no other script reads stale data.
 #
 # ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Required by the container interaction and state writes.
+#
+# Lock Acquisition
+#   acquire_lock prevents a scheduled run overlapping a manual --move.
+#
+# Host Detection
+#   detect_hosts() aliases SONARR_URL / SONARR_API_KEY / the root literals.
+#
+# curl + jq Dependency Check
+#   Fails fast if either is missing — every classification signal is parsed with jq,
+#   and a missing jq would evaluate each signal to empty and classify nothing.
+#
+# Report-Only Default
+#   Nothing is written to Sonarr without --move. The scan is safe to schedule and
+#   safe to run repeatedly while tuning the curated lists.
+#
+# Required Var Check
+#   require_var on SONARR_URL and SONARR_API_KEY before any request.
+#
+# API Reachability + Version Gate
+#   check_api then check_arr_version against SONARR_VERSION_MAJOR. A major version
+#   bump can move or rename the fields every rule here depends on, so a mismatch
+#   aborts rather than classifying against an unknown schema.
+#
+# Empty Library Abort
+#   A response of 0 series aborts. An empty list is indistinguishable from "nothing
+#   is misplaced" and would otherwise report a clean library during an API fault.
+#
+# Unconfigured Root Skip
+#   A blank SONARR_GENERAL_ROOT / KIDS_ROOT / ANIME_ROOT skips that category's
+#   checks rather than erroring — a host with no dedicated root is a valid setup,
+#   and a blank value must never be compared against as if it were a real path.
+#
+# One At A Time, Stop On First Failure
+#   Series are moved individually and the batch halts on the first failure rather
+#   than continuing. A misclassified root or a failing move is a condition to
+#   review, not to repeat across the library.
+#
+# Async Move Completion Polling
+#   moveFiles=true flips the DB instantly while the physical move is a separate
+#   async MoveSeries command Sonarr drains one at a time. Each move locates its own
+#   command and polls it to "completed" (bounded by SONARR_MOVE_POLL_TIMEOUT) before
+#   anything else is checked. Without this a batch reports every series moved while
+#   the files are still queued at the old path — and downstream orphan cleanup can
+#   act on that gap.
+#
+# Post-Move Re-Verification
+#   The PUT response is never trusted. The series is re-fetched and both
+#   rootFolderPath and episodeFileCount are confirmed against expectations before
+#   the move counts as successful.
+#
+# Reverse-Anime Leaks Excluded From Moves
+#   Deliberate style placements (Western/Chinese animation grouped with anime by
+#   choice) legitimately sit in the anime root. Those are reported, never moved.
+#
+# Post-Write Cache Refresh
+#   The tracked-data cache is refreshed after --move writes so no other arr script
+#   reads a stale rootFolderPath.
+#
+# ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
 #
@@ -249,6 +319,31 @@ if [[ "$ENABLE_LOGGING" == true ]]; then
         "  [\(if .forward_anime_miss then "FORWARD-ANIME" elif .forward_kids_miss then "FORWARD-KIDS" elif .reverse_anime_leak then "REVERSE-ANIME" elif .reverse_kids_leak then "REVERSE-KIDS" else "?" end)] \(.title) (root: \(.rootFolderPath), network: \(.network // "n/a"), cert: \(.certification // "n/a"))"'
 fi
 
+# ── The reverse leaks, written down ───────────────────────────────────────────────────────────
+# This block's own header says it does not act on REVERSE-ANIME leaks because they are genuine
+# judgement calls. That is right, and it is also why they are the one result worth persisting:
+# every other bucket either self-resolves or is acted on by --move, while these accumulate as a
+# number in a summary nobody can do anything with. Seventeen of them hid two live-action dramas
+# filed under anime for as long as the count stayed a count.
+#
+# Written as the script's own verdict so anything reading it — the triage that reads this next —
+# inherits the classification rather than computing a second opinion from the same metadata.
+# Report-only: this records what was found, it does not change what happens to any of it.
+if [[ -n "${STATE_DIR:-}" ]] && [[ "$DRY_RUN" != true ]]; then
+    _review_file="$STATE_DIR/arr_classification_review.json"
+    echo "$RESULTS" | jq -c --arg host "$MY_ID" --argjson ts "$(date +%s)" '
+        { host: $host, ts: $ts, arr: "sonarr",
+          reverse_anime: [ .[] | select(.reverse_anime_leak) |
+            { title, root: .rootFolderPath, network: (.network // ""), cert: (.certification // ""),
+              lang: (.originalLanguage.name // .originalLanguage // ""), id: .id } ],
+          reverse_kids:  [ .[] | select(.reverse_kids_leak) |
+            { title, root: .rootFolderPath, network: (.network // ""), cert: (.certification // "") } ] }
+    ' > "$_review_file" 2>/dev/null \
+      && log "$ICON_GEAR Review list written — $REVERSE_ANIME_COUNT anime leak(s) for triage" \
+      || warn "Could not write $_review_file — triage will have nothing to read"
+    unset _review_file
+fi
+
 # ==============================================================================================
 # ━━━ Summary ━━━
 # ==============================================================================================

Arrs_Stack/sonarr_cleanup.sh

@@ -92,7 +92,6 @@
 #   jq + curl validation — exits if either tool missing
 #   ARR_DOCKER_TIMEOUT   — container checks protected against daemon hangs (script-local, not common.sh's DOCKER_TIMEOUT)
 #   notify_emby_scan()   — triggers Emby clean after deletion
-#   platform_require_cmd  — notify script validated before use
 #   Silent by default    — orphans/junk warn(), clean library logs silently
 #
 # ==============================================================================================
@@ -309,7 +308,7 @@ SERIES_RESPONSE=$(arr_get_tracked_data "sonarr" "$SONARR_URL" "$SONARR_API_KEY"
 }
 
 SERIES_IDS=$(echo "$SERIES_RESPONSE" | jq -r '.[].id' 2>/dev/null)
-SERIES_COUNT=$(echo "$SERIES_IDS" | grep -c "." 2>/dev/null || echo 0)
+SERIES_COUNT=$(echo "$SERIES_IDS" | grep -c "." 2>/dev/null || true)
 
 # Safety Layer 4 — series count > 0
 if [[ "$SERIES_COUNT" -eq 0 ]]; then
@@ -441,6 +440,30 @@ NOW=$(date +%s)
 TO_DELETE_FILE="$TMP_DIR/to_delete_paths.txt"
 > "$TO_DELETE_FILE"
 
+# ── Orphan strikes ────────────────────────────────────────────────────────────────────────────
+# Same contract as radarr_cleanup.sh: a file must classify for deletion on
+# SONARR_ORPHAN_STRIKE_LIMIT consecutive runs before it is removed. Covers the partial
+# classification failure that is too small to trip the tracked-count floor above. The file is
+# rebuilt from each run rather than edited, which is what prunes it.
+SONARR_ORPHAN_STRIKE_LIMIT="${SONARR_ORPHAN_STRIKE_LIMIT:-2}"
+STRIKES_FILE="${SONARR_ORPHAN_STRIKES_FILE:-$DB_DIR/sonarr_orphan_strikes.tsv}"
+mkdir -p "$(dirname "$STRIKES_FILE")" 2>/dev/null || true
+touch "$STRIKES_FILE" 2>/dev/null || true
+STRIKES_NEW="$TMP_DIR/strikes_new.tsv"
+> "$STRIKES_NEW"
+HELD_COUNT=0
+HELD_BYTES=0
+
+orphan_strike_ok() {
+    local path="$1" prev strikes
+    prev=$(wd_state_get "$path" "$STRIKES_FILE"); prev="${prev//[^0-9]/}"
+    strikes=$(( ${prev:-0} + 1 ))
+    printf '%s:%s\n' "$path" "$strikes" >> "$STRIKES_NEW"
+    (( strikes >= SONARR_ORPHAN_STRIKE_LIMIT )) && return 0
+    warn "  strike $strikes/$SONARR_ORPHAN_STRIKE_LIMIT — not removing yet: $path"
+    return 1
+}
+
 while read -r FILE_SIZE FILE_CTIME filepath; do
     [[ -z "$filepath" ]] && continue
     FILE_CTIME="${FILE_CTIME%%.*}"
@@ -475,12 +498,14 @@ while read -r FILE_SIZE FILE_CTIME filepath; do
         warn "$ICON_TRASH ORPHAN: $filepath"
         (( ORPHAN_COUNT++ ))
         ORPHAN_BYTES=$(( ORPHAN_BYTES + FILE_SIZE ))
-        echo "$filepath" >> "$TO_DELETE_FILE"
+        if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
+        printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
     else
         log "JUNK: $filepath"
         (( JUNK_COUNT++ ))
         JUNK_BYTES=$(( JUNK_BYTES + FILE_SIZE ))
-        echo "$filepath" >> "$TO_DELETE_FILE"
+        if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
+        printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
     fi
 
 # -printf gets size + mtime directly from find's own stat() during the walk, instead of a
@@ -492,13 +517,43 @@ done < <(
     done | sort -u
 )
 
-TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES ))
+# Eligible, not classified: a file still serving its strikes is an orphan but is not queued this
-TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT ))
+# run, so it must not appear in the denominator the budget reports against.
+TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES - HELD_BYTES ))
+TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT - HELD_COUNT ))
+
+# Rebuilt, never edited. Skipped on a dry run: a preview that advanced real counters would make
+# the next real run delete a run early.
+if [[ "$DRY_RUN" == false ]]; then
+    mv "$STRIKES_NEW" "$STRIKES_FILE" 2>/dev/null || warn "Could not update $STRIKES_FILE"
+fi
 
 # ==============================================================================================
 # ━━━ Safety Layer 7 — Deletion Size Threshold ━━━
 # ==============================================================================================
-check_delete_size_threshold "$TOTAL_DELETE_BYTES" "$SONARR_MAX_DELETE_GB" "Sonarr Cleanup"
+# A per-run budget, not a veto — see apply_delete_budget() in common.sh. The ceiling still caps
+# any single run; it just no longer deadlocks on a backlog larger than itself.
+BUDGET_FILE="$TMP_DIR/to_delete_budgeted.txt"
+
+if [[ "$I_KNOW" == true ]]; then
+    warn "OVERRIDE — --i-know-what-im-doing active, per-run budget not applied"
+    cut -d"$(printf '\t')" -f3- "$TO_DELETE_FILE" > "$BUDGET_FILE"
+    _BUDGET_KEPT_COUNT=$TOTAL_REMOVED; _BUDGET_KEPT_BYTES=$TOTAL_DELETE_BYTES
+    _BUDGET_DEFERRED_COUNT=0; _BUDGET_DEFERRED_BYTES=0; _BUDGET_STUCK=""
+else
+    apply_delete_budget "$TO_DELETE_FILE" "$BUDGET_FILE" "$SONARR_MAX_DELETE_GB"
+    if [[ -n "$_BUDGET_STUCK" ]]; then
+        error "Single file exceeds the ${SONARR_MAX_DELETE_GB}GB budget on its own — nothing removed this run"
+        error "  $_BUDGET_STUCK"
+        error "Raise SONARR_MAX_DELETE_GB or clear this one with --i-know-what-im-doing"
+        notify "Sonarr cleanup stalled on $(hostname) — one file exceeds the ${SONARR_MAX_DELETE_GB}GB budget" \
+            "Sonarr Cleanup" "warning"
+    elif [[ "$_BUDGET_DEFERRED_COUNT" -gt 0 ]]; then
+        warn "Budget ${SONARR_MAX_DELETE_GB}GB — removing $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED ($(format_bytes "$_BUDGET_KEPT_BYTES")), deferring $_BUDGET_DEFERRED_COUNT ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) to the next run"
+        notify "Sonarr cleanup removed $(format_bytes "$_BUDGET_KEPT_BYTES") of $(format_bytes "$TOTAL_DELETE_BYTES") on $(hostname) — $_BUDGET_DEFERRED_COUNT file(s) deferred" \
+            "Sonarr Cleanup" "normal"
+    fi
+fi
 
 # ── Execute Deletions ─────────────────────────────────────────────────────────────────────────
 # Reuses TO_DELETE_FILE from the classification pass above instead of re-walking and
@@ -507,7 +562,7 @@ if [[ "$DRY_RUN" == false ]]; then
     while IFS= read -r filepath; do
         [[ -z "$filepath" ]] && continue
         rm -f "$filepath" 2>/dev/null || error "Failed to delete: $filepath"
-    done < "$TO_DELETE_FILE"
+    done < "$BUDGET_FILE"
 
     info "Cleaning up empty folders..."
     for host_path in "${SCAN_ROOTS[@]}"; do
@@ -533,6 +588,10 @@ echo "$ICON_SHIELD  Protected:       $PROTECTED_COUNT files (artwork, subtitles,
 echo "$ICON_TRASH   Orphans:         $ORPHAN_COUNT files ($ORPHAN_HUMAN)"
 echo "$ICON_TRASH   Junk:            $JUNK_COUNT files ($JUNK_HUMAN)"
 echo "$ICON_SKIP    Recent skipped:  $RECENT_COUNT files (under ${SONARR_ORPHAN_AGE} days)"
+[[ "${HELD_COUNT:-0}" -gt 0 ]] && \
+  echo "$ICON_SKIP    Held (strikes):  $HELD_COUNT files ($(format_bytes "$HELD_BYTES")) — under ${SONARR_ORPHAN_STRIKE_LIMIT} consecutive runs"
+[[ "${_BUDGET_DEFERRED_COUNT:-0}" -gt 0 ]] && \
+  echo "$ICON_SKIP    Deferred:        $_BUDGET_DEFERRED_COUNT files ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) — over the ${SONARR_MAX_DELETE_GB}GB run budget"
 echo "$ICON_TIME    Duration:        $(format_duration $(( END - START )))"
 echo ""
 
@@ -541,8 +600,10 @@ if [[ "$DRY_RUN" == true ]]; then
 elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
     echo "$ICON_DONE Clean — nothing to remove"
 else
-    warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
+    # What was actually removed, not what was classified. With strikes and a budget in force those
-    notify "Sonarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
+    # differ, and reporting the classification as the outcome is the oldest bug shape here.
+    warn "$ICON_DONE Removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED eligible files ($(format_bytes "$_BUDGET_KEPT_BYTES"))"
+    notify "Sonarr cleanup on $(hostname) — removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED eligible files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
         "Sonarr Cleanup" "warning"
     # Notify Emby to clean missing files — removes ghost entries immediately
     notify_emby_scan

Arrs_Stack/start_webhook_listener.sh

@@ -67,13 +67,38 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   WEBHOOK_PORT=0 gate  — exits cleanly before any setup if the listener is disabled
+# Root Enforcement
-#   acquire_lock "continuous" — exits 0 cleanly if a healthy instance is already running,
+#   Writes /var/log/varaverk and persists the generated secret into master.conf.
-#                          instead of relaunching into a port conflict
+#
-#   Secret auto-generate — WEBHOOK_SECRET generated via openssl rand if empty;
+# WEBHOOK_PORT=0 Gate
-#                          persisted to master.conf immediately so restarts reuse it
+#   Exits cleanly before any setup if the listener is disabled.
-#   Shared secret gate   — webhook URL must include ?key=<WEBHOOK_SECRET>;
+#
-#                          requests without a valid key are rejected by the Node.js server
+# node Presence Check
+#   node is exec'd at the end of this script. Checking up front fails with a clear
+#   reason at array start rather than an exec error buried in the log after setup.
+#
+# openssl Presence Check
+#   Checked before attempting to generate a secret, on the path that needs it.
+#
+# Lock Acquisition
+#   acquire_lock "continuous" exits 0 cleanly if a healthy instance is already
+#   listening, instead of relaunching into an EADDRINUSE port conflict. An array
+#   stop/start without a full reboot leaves the old node process alive, and without
+#   this array_started.sh would log a false failure.
+#
+# Secret Auto-Generation
+#   WEBHOOK_SECRET generated via openssl rand if empty, and persisted to master.conf
+#   so restarts reuse it.
+#
+# Secret Persistence Verification
+#   The write-back is confirmed by re-reading master.conf. If it did not land, the
+#   secret would exist only in this process and be regenerated on the next start,
+#   silently invalidating the key already registered in the arrs — so this aborts
+#   loudly rather than starting with a secret that will not survive a restart.
+#
+# Shared Secret Gate
+#   The webhook URL must include ?key=<WEBHOOK_SECRET>; requests without a valid key
+#   are rejected by the Node.js server.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -104,11 +129,25 @@ ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
 
 source "$ECOSYSTEM_ROOT/load_config.sh"
 
+if [[ "$EUID" -ne 0 ]]; then
+    error "Must be run as root"
+    exit 1
+fi
+
 [[ "${WEBHOOK_PORT:-0}" -eq 0 ]] && {
     echo "[webhook] WEBHOOK_PORT=0 — listener disabled"
     exit 0
 }
 
+# node is exec'd at the end of this script — checking here fails with a clear reason at
+# array start instead of an exec error buried in the log after all the setup has run.
+if ! command -v node >/dev/null 2>&1; then
+    error "node not found — required to run webhook_listener.js"
+    notify "Webhook listener failed to start on $(hostname) — node not installed" \
+        "Webhook Listener" "warning"
+    exit 1
+fi
+
 # Skip gracefully if a healthy instance is already listening — otherwise a
 # restart that doesn't kill the old node process (array stop/start without a
 # full reboot) hits EADDRINUSE and array_started.sh logs a false failure.
@@ -116,10 +155,29 @@ acquire_lock "continuous"
 
 # ── Auto-generate secret if not yet set ─────────────────────────────────────
 if [[ -z "${WEBHOOK_SECRET:-}" ]]; then
+    if ! command -v openssl >/dev/null 2>&1; then
+        error "openssl not found — cannot generate WEBHOOK_SECRET"
+        notify "Webhook listener failed to start on $(hostname) — openssl not installed" \
+            "Webhook Listener" "warning"
+        exit 1
+    fi
+
     GENERATED=$(openssl rand -hex 32)
     MASTER_CONF="$ECOSYSTEM_ROOT/Configurations/master.conf"
     sed -i "s/WEBHOOK_SECRET=\"\"/WEBHOOK_SECRET=\"$GENERATED\"/" "$MASTER_CONF"
     WEBHOOK_SECRET="$GENERATED"
+
+    # If the sed didn't match, the secret only exists in this process. The listener would
+    # come up, then regenerate a different secret on the next start — silently invalidating
+    # the key already registered in the arrs. Fail loudly instead.
+    if ! grep -q "WEBHOOK_SECRET=\"$GENERATED\"" "$MASTER_CONF" 2>/dev/null; then
+        error "Generated WEBHOOK_SECRET but could not persist it to $MASTER_CONF"
+        error "Set WEBHOOK_SECRET manually — a non-persisted secret changes on every restart"
+        notify "Webhook secret not persisted on $(hostname) — set WEBHOOK_SECRET manually" \
+            "Webhook Listener" "warning"
+        exit 1
+    fi
+
     echo "[webhook] Generated WEBHOOK_SECRET — run Tools/webhook_setup.sh to register in arrs"
 fi
 

Arrs_Stack/upgrade_webhook_handler.sh

@@ -15,6 +15,30 @@
 # begin searching — a search it will never win because we already have it.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Invoked per import by webhook_listener.js with <arr_type> <item_path>:
+#
+#   1. Validate
+#        → arr_type must be sonarr|radarr|lidarr; item_path must exist and be a safe
+#          absolute path
+#
+#   2. Resolve arr specifics
+#        → API port, API version and rescan command for that arr type
+#
+#   3. Discover remote nodes
+#        → discover_remote_nodes(); no remotes configured means exit cleanly
+#
+#   4. Per remote node, independently:
+#        a. Resolve its Tailscale IP — unresolvable skips that node
+#        b. rsync the single item to the same absolute path (--no-delete)
+#        c. Skip the rescan if rsync failed — never scan a partial file
+#        d. Trigger the arr's refresh command, cache-first API key with SSH fallback
+#
+# One failing node is counted and skipped; the rest still receive the upgrade.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -38,13 +62,47 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Arg validation      — exits with usage message if arr_type or item_path missing
+# Root Enforcement
-#   Path existence      — exits if item_path is not a directory on disk
+#   rsync runs over SSH as root and writes to root@remote at the same absolute path.
-#   Tailscale resolution — skips a node if its Tailscale IP cannot be resolved
+#
-#   rsync exit check    — rescan is only triggered if rsync succeeded; a failed
+# Argument Validation
-#                         transfer does not cause the remote arr to scan a partial file
+#   Exits with a usage message if arr_type or item_path is missing, and rejects an
-#   SSH fallback        — if no cached API key, falls back to SSH to read config.xml
+#   arr_type outside sonarr|radarr|lidarr rather than defaulting to one.
-#                         on the remote rather than failing the rescan step
+#
+# Path Existence
+#   Exits if item_path is not a directory on disk.
+#
+# Item Path Depth Guard
+#   item_path must be an absolute path at least three levels deep. It arrives from the
+#   arr's webhook payload and is rsynced to the same path on the partner, so a truncated
+#   or malformed value would push a system directory — or the filesystem root — onto the
+#   remote. The existence check alone does not catch this, because / is a directory.
+#
+# No Lock — Deliberate
+#   This is an event handler invoked per import by webhook_listener.js. Concurrent
+#   upgrades are normal and expected. A default lock would silently drop overlapping
+#   events, and a waiting lock would queue them behind a slow transfer, so neither is
+#   used: each invocation rsyncs a different item path and they do not contend.
+#
+# Tailscale Resolution
+#   Skips a node if its Tailscale IP cannot be resolved, rather than attempting the
+#   transfer against an unresolved or stale address.
+#
+# rsync Exit Check
+#   The rescan is only triggered if rsync succeeded. A failed transfer never causes the
+#   remote arr to scan a partial file into its library.
+#
+# No Delete on Push
+#   rsync runs with --no-delete. This pushes one upgraded item; it is not a mirror, and
+#   must never remove content on the partner that this run does not know about.
+#
+# SSH Fallback
+#   If no cached API key is available, falls back to SSH to read config.xml on the
+#   remote rather than failing the rescan step.
+#
+# Per-Node Isolation
+#   One unreachable or failing node is counted and skipped; the remaining nodes still
+#   receive the upgrade.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -90,6 +148,21 @@ ITEM_PATH="${2:-}"
 
 [[ -d "$ITEM_PATH" ]] || { echo "Path not found: $ITEM_PATH" >&2; exit 1; }
 
+if [[ "$EUID" -ne 0 ]]; then
+    echo "Must be run as root" >&2
+    exit 1
+fi
+
+# ITEM_PATH is rsynced to root@remote at the same absolute path. It arrives from the arr's
+# webhook payload, so a malformed or truncated value would push a system directory — or the
+# filesystem root — onto the partner. -d alone does not catch that: / is a directory.
+_depth="${ITEM_PATH//[^\/]/}"
+if [[ "$ITEM_PATH" != /* || "${#_depth}" -lt 3 ]]; then
+    echo "Refusing unsafe item path: '$ITEM_PATH' — expected an absolute path at least 3 levels deep" >&2
+    exit 1
+fi
+unset _depth
+
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "$SCRIPT_DIR/../load_config.sh"
 detect_hosts

Deployment/Manual-Deployment.md

@@ -0,0 +1,225 @@
+# ━━━━━ DEPLOYMENT — Manual ━━━━━
+
+Procedures and flag reference for the schema layer.
+For overview see README-Deployment.md. For per-script detail see the script headers.
+
+---
+
+## ━━━ PROCEDURES ━━━
+
+### Adding a New Configuration Variable
+
+The single most common task here, and the one with the quietest failure mode if done wrong.
+
+```bash
+# 1. Add it to the template — this is the versioned schema
+#    Shared threshold/toggle  → Deployment/master.conf.template
+#    Per-host credential/path → Deployment/host.conf.template  (use the HOSTN_ prefix)
+
+# 2. Add it to your own live conf so you can test immediately
+#    Configurations/master.conf   or   Configurations/host1.conf
+
+# 3. Use it in the script, with a safe default
+#    [[ "${NEW_THRESHOLD:-50}" -gt ... ]]
+
+# 4. Commit the script change and the template change TOGETHER
+git add Deployment/master.conf.template Watchdogs/System/storage_watchdog.sh
+git commit -m "..."
+```
+
+**Step 1 is the one that gets skipped.** A script merged without its template entry works on
+the machine it was written on and nowhere else — the variable is simply empty on every other
+node, and the script takes whatever branch an empty value produces. Nothing errors.
+
+Placement rule, same as everywhere in the ecosystem:
+
+| Kind | Goes in |
+|------|---------|
+| Threshold, toggle, profile, job list | `master.conf.template` |
+| Credential, path, container name, per-host identity | `host.conf.template` |
+
+---
+
+### Previewing a Schema Change Before It Lands
+
+```bash
+bash Deployment/conf_upgrade.sh \
+    --template Deployment/master.conf.template \
+    --target   Configurations/master.conf \
+    --dry-run
+```
+
+Prints the full ADDED / REMOVED / KEPT report and writes nothing. No root required — safe to
+run as any user, on any target, at any time.
+
+Read the **REMOVED** list carefully. A key showing as REMOVED means it is in your conf but no
+longer in the template — either genuinely deprecated, or someone forgot step 1 above and the
+next pull will drop a setting you still rely on.
+
+---
+
+### Applying a Schema Change by Hand
+
+Normally automatic via `git_pull_execute.sh`. To run it manually:
+
+```bash
+bash Deployment/conf_upgrade.sh \
+    --template Deployment/master.conf.template \
+    --target   Configurations/master.conf \
+    --backup
+```
+
+`--backup` writes `master.conf.bak` first. Use it — see The Recovery Gap in the README for
+why the backup matters more here than it looks.
+
+For a host conf, the template must have its prefix resolved first, exactly as
+`git_pull_execute.sh` does it:
+
+```bash
+TMPL=$(mktemp)
+sed "s/HOSTN_/HOST1_/g; s/REMOTE_ID/HOST2/g" Deployment/host.conf.template > "$TMPL"
+bash Deployment/conf_upgrade.sh --template "$TMPL" --target Configurations/host1.conf --backup
+rm -f "$TMPL"
+```
+
+Merging the raw template without substituting `HOSTN_` would add 140 new `HOSTN_*` keys
+alongside your real `HOST1_*` ones, and mark every real key as REMOVED.
+
+---
+
+### Populating Credentials on a New Host
+
+```bash
+bash Deployment/conf_populate.sh --dry-run     # always first
+bash Deployment/conf_populate.sh
+```
+
+Reads from the services actually running on this host and fills **empty** fields only.
+Existing values are never touched without `--overwrite`.
+
+What it detects: arr API keys and ports from each `config.xml`, root folders from the arr
+rootFolder API, path maps from docker volume mounts, SABnzbd/slskd/qBittorrent credentials
+from their own config files, Emby/Jellyfin containers and ports, the boot device transport,
+and the default-gateway NIC.
+
+Then pushes the updated conf to partners via `conf_sync.sh` so they hold the fresh keys
+immediately. `--no-push` skips that.
+
+**If a field stays empty after a run,** check the output for an ambiguity warning:
+
+```
+WARN: Container prefix 'authelia' is ambiguous — matches: Authelia-Secondary Authelia
+WARN:   Refusing to guess. Set the container name manually in host*.conf.
+```
+
+That is working as intended. Set it by hand and re-run.
+
+---
+
+### After Rotating an API Key
+
+```bash
+bash Deployment/conf_populate.sh --overwrite --dry-run
+bash Deployment/conf_populate.sh --overwrite
+```
+
+`--overwrite` replaces detected fields even when already set. This is the intended use for it
+— rotating an arr key, rebuilding a container, or repointing at a moved service.
+
+Note it overwrites **every** detected field, not just the rotated one. Run the dry-run first
+and read the list.
+
+---
+
+### Rebuilding a Wiped Node
+
+The templates are what make this possible without copying another machine's credentials.
+
+```bash
+# 1. Clone the repo — templates come with it, confs do not (gitignored)
+# 2. Seed the confs from the templates
+cp Deployment/master.conf.template Configurations/master.conf
+sed "s/HOSTN_/HOST2_/g; s/REMOTE_ID/HOST1/g" Deployment/host.conf.template > Configurations/host2.conf
+
+# 3. Fill in what can be detected automatically
+bash Deployment/conf_populate.sh
+
+# 4. Fill in the rest by hand — anything conf_populate cannot see:
+#    hostnames, DDNS containers, fallback tier lists, sync share lists,
+#    partner credentials, Discord webhook
+```
+
+From then on, `git_pull_execute.sh` keeps the conf in step with the template automatically.
+
+---
+
+## ━━━ FLAG REFERENCE ━━━
+
+### `conf_upgrade.sh`
+
+| Flag | Required | What it does |
+|------|----------|-------------|
+| `--template <file>` | yes | Source of structure and new keys |
+| `--target <file>` | yes | Existing conf — source of real values, always preserved |
+| `--dry-run` | | Print the change report, write nothing. No root needed. |
+| `--backup` | | Write `<target>.bak` before installing |
+
+Takes no other flags. It does not source `load_config.sh`, so `--log` and `--status` do not
+exist here — see the script header for why that is deliberate.
+
+### `conf_populate.sh`
+
+| Flag | What it does |
+|------|-------------|
+| `--dry-run` | Show what would be written, truncated. Changes nothing. |
+| `--overwrite` | Replace detected fields even when already set |
+| `--no-push` | Skip pushing the updated conf to partners |
+| `--log` | Verbose per-field output |
+
+---
+
+## ━━━ TROUBLESHOOTING ━━━
+
+### A variable is empty on one node but set on another
+
+The template entry is missing. Confirm:
+
+```bash
+grep -n "MY_VARIABLE" Deployment/master.conf.template Deployment/host.conf.template
+```
+
+No hit means the variable was added to a conf directly and never to the template, so it has
+never reached any other node. Add it to the template; the next pull propagates it.
+
+### `conf_upgrade` reports a key as REMOVED that is still in use
+
+Same cause, opposite direction — the key is in your conf and in a script, but not in the
+template. Add it to the template before the next pull drops it.
+
+### The conf came back with different permissions
+
+It should not — the target's mode and owner are copied onto the staged file before the
+rename. If they did change, check that the target existed before the run: `chmod --reference`
+silently no-ops against a missing file.
+
+### A conf edit disappeared after a pull
+
+Expected if the key is not in the template. `conf_upgrade` keeps values for keys that exist in
+both; a key present only in your conf is classified REMOVED and dropped. Add it to the
+template.
+
+### `conf_populate` skipped a field
+
+Either the service is not running, its config file was unreadable, or the container name
+prefix was ambiguous. The last case prints an explicit warning with the full match list —
+set that value by hand.
+
+---
+
+## ━━━ WHAT THIS FOLDER DOES NOT DO ━━━
+
+- **It does not create `Configurations/`.** `conf_upgrade.sh` aborts if the target does not
+  exist rather than creating a partial conf. Seeding a new node is the manual step above.
+- **It does not sync confs between hosts.** That is `System_Essentials/conf_sync.sh`.
+- **It does not validate values.** It reconciles *structure*. A threshold set to nonsense
+  merges through untouched — the consuming script owns validation.

Deployment/README-Deployment.md

@@ -0,0 +1,211 @@
+# ━━━━━ DEPLOYMENT ━━━━━
+
+The schema layer. `Configurations/*.conf` holds every value the ecosystem runs on — and is
+gitignored, because it holds credentials. This folder holds the **templates** those confs are
+built from, and the two scripts that keep the confs in step with them.
+
+Two files that are versioned, and two scripts that reconcile the unversioned confs against
+them:
+
+```
+  Deployment/master.conf.template     349 vars   ← the versioned schema
+  Deployment/host.conf.template       140 vars   ← per-host schema, HOSTN_-prefixed
+  Deployment/conf_upgrade.sh                     ← template → conf, values preserved
+  Deployment/conf_populate.sh                    ← running services → conf, empty fields only
+```
+
+> **The templates are the only versioned record of what configuration exists.** Nothing else
+> in git knows that a variable is supposed to be there.
+
+---
+
+## ━━━ THE PROBLEM THAT BUILT THIS ━━━
+
+**Config Holds Secrets, So Config Cannot Be Committed**
+`master.conf` and `host*.conf` contain API keys, passwords, SSH key paths and personal
+hostnames. They are gitignored, along with their `.bak` files:
+
+```
+.gitignore:4   Configurations/host*.conf
+.gitignore:5   Configurations/master.conf
+.gitignore:6   Configurations/*.bak
+```
+
+That is correct and non-negotiable. But it creates a problem: if the confs are not in git,
+then **git has no idea a new setting was ever added.** A script that starts reading
+`NEW_THRESHOLD` works on the machine where it was developed and silently fails everywhere
+else, because no other node's conf has that key.
+
+**A New Node Would Start With Nothing**
+Without a versioned schema there is no way to stand up a second server, or rebuild a wiped
+one, except by hand-copying a conf from a machine that already works — which means copying
+its credentials too.
+
+**Hand-Editing Confs Across Nodes Does Not Scale**
+Two servers, ~490 variables between them. Adding a setting by hand means editing it on every
+node, in the right section, with the right default, without disturbing the values already
+there. Miss one and the failure surfaces days later as a script behaving differently on one
+host.
+
+---
+
+## ━━━ WHAT THIS FOLDER DOES ━━━
+
+### 🔀  Schema Merge — `conf_upgrade.sh`
+
+Merges a template into an existing conf while preserving **every value the user has already
+set**. Runs automatically from `git_pull_execute.sh` after every single pull.
+
+```
+  Key in template only  →  ADDED     placeholder/default, filled in once
+  Key in conf only      →  REMOVED   deprecated in this version
+  Key in both           →  KEPT      the conf's value always wins
+  Comments, blank lines →  from the template — structure follows the new version
+```
+
+That last rule is what makes it safe to run unattended forever: the template supplies
+*structure and new keys*, never settings. Your values cannot be overwritten by a pull.
+
+The live confs currently match their templates exactly — 349 and 140 variables — which is
+what a working merge looks like.
+
+### 🔎  Credential Discovery — `conf_populate.sh`
+
+Reads settings out of the services actually running on this host and writes them into the
+host conf: arr API keys from each `config.xml`, ports from real docker port bindings, paths
+from real volume mounts, SABnzbd/slskd/qBittorrent credentials from their own config files.
+
+Only fills **empty** fields unless `--overwrite`. Manual — it is not scheduled anywhere.
+
+---
+
+## ━━━ HOW A CHANGE REACHES EVERY NODE ━━━
+
+```
+You add a variable
+    │
+    └── edit Deployment/master.conf.template   ← the versioned schema
+            │
+            git push
+            │
+            └── every node: git_pull_execute.sh
+                    │
+                    └── conf_upgrade.sh --template ... --target ... --backup
+                            │
+                            ADDED   → new key appears with the template default
+                            KEPT    → every existing value untouched
+                            REMOVED → deprecated keys dropped
+```
+
+**This is the rule that follows from it, and it is not optional:**
+
+> Any conf variable change — add, remove, or rename — must update
+> `Deployment/master.conf.template` and `Deployment/host.conf.template` **in the same pass**
+> as the script change that uses it.
+
+A script merged without its template entry works only on the machine it was written on.
+Nothing errors; the variable is simply empty everywhere else, and the script takes whatever
+branch an empty value leads to.
+
+---
+
+## ━━━ THE `HOSTN_` PLACEHOLDER ━━━
+
+`host.conf.template` is written with a generic prefix — 149 occurrences of `HOSTN_`:
+
+```bash
+HOSTN_SONARR_URL=""
+HOSTN_SONARR_API_KEY=""
+```
+
+`git_pull_execute.sh` substitutes the real identity before merging, so keys match the target:
+
+```bash
+sed "s/HOSTN_/${MY_ID}_/g; s/REMOTE_ID/${REMOTE_ID}/g" host.conf.template > "$TMPL_RESOLVED"
+```
+
+One template therefore serves every host. HOST1 merges it as `HOST1_*`, HOST2 as `HOST2_*`,
+and a third node would work with no template change at all.
+
+---
+
+## ━━━ SCRIPTS IN THIS FOLDER ━━━
+
+| Script | Role | When It Runs |
+|--------|------|-------------|
+| `conf_upgrade.sh` | Merge template into conf — structure forward, values preserved | Automatically, after every `git pull` |
+| `conf_populate.sh` | Detect settings from running services into the host conf | Manually — onboarding, or after a key rotation |
+| `migrate_data_layout.sh` | Move everything persisted into the rooted `data/` tree | Once per host, manually. Idempotent. |
+
+### 📦  Data layout migration — `migrate_data_layout.sh`
+
+`conf_upgrade.sh` adds keys the template has and the installation does not; it never rewrites a
+value you already have. That is exactly what you want from it, and exactly why it cannot perform
+a layout migration — the paths being moved are *existing* keys, so their values would keep
+pointing at the old layout forever while the new directory variables sat beside them unused.
+
+So this rewrites those values and moves the files to match. Both halves or neither.
+
+```bash
+Deployment/migrate_data_layout.sh --dry-run    # always first
+Deployment/migrate_data_layout.sh
+```
+
+It refuses to run while a job from *this* installation is active — scoped to the installation's
+own path, because `pgrep` is system-wide and a box running both a production checkout and a
+development clone will otherwise always look busy. `--force` overrides.
+
+Each host runs it itself: `data/` is gitignored, so a restructure travels as code and conf while
+the files stay where they are. See `data/README.md` for the resulting layout.
+
+| Template | Role |
+|----------|------|
+| `master.conf.template` | Shared schema — thresholds, toggles, profiles, orchestrator job lists |
+| `host.conf.template` | Per-host schema — credentials, paths, container names. `HOSTN_`-prefixed |
+
+---
+
+## ━━━ SAFEGUARDS WORTH KNOWING ━━━
+
+**The install is atomic.** `conf_upgrade.sh` stages the merged conf beside the target and
+installs it with a rename, never a copy. A `cp` truncates the live conf and writes into it —
+and every watchdog sources `load_config.sh` on every run, so anything reading during that
+window would get a partial conf with empty path variables. The temp file is staged in the
+target's own directory deliberately: `/tmp` is rootfs while the confs are on flash, and a
+cross-device `mv` degrades to copy-then-unlink, which is the exact torn write being avoided.
+
+**Dry run needs no privilege, writing does.** `--dry-run` prints the full ADDED / REMOVED /
+KEPT report and is useful to anyone. Installing over a conf under `/boot` requires root.
+
+**`conf_upgrade.sh` sources nothing — deliberately.** No `load_config.sh`, no `common.sh`.
+It is the tool that repairs the conf `load_config.sh` depends on, so it has to work when that
+conf is broken, partial, or missing keys. That is also why it uses plain `echo` rather than
+`log()`, and why it has no `acquire_lock` — concurrency is handled by the atomic rename
+instead, and since the merge is idempotent, last-writer-wins is identical to running once.
+
+**`conf_populate.sh` refuses to guess a container.** An ambiguous name prefix skips the field
+rather than picking the first match. Writing the wrong container name is worse than writing
+nothing: an empty field is visibly incomplete and gets fixed, a wrong one silently points the
+whole stack at the wrong instance. This host has a live example — `authelia` prefix-matches
+both `Authelia` (9091) and `Authelia-Secondary` (9092).
+
+---
+
+## ━━━ THE RECOVERY GAP ━━━
+
+Confs are gitignored, **and so are their `.bak` files**. There is no versioned history to
+revert to, and the single `.bak` slot is overwritten by whoever writes next:
+
+| file | modified | its `.bak` |
+|---|---|---|
+| `master.conf` | Jul 28 18:52 | Jul 28 18:52 |
+| `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** |
+
+A bad write to `master.conf` currently falls back to a file that may predate weeks of edits.
+Worth knowing before hand-editing a conf, and the reason `--backup` exists on `conf_upgrade.sh`
+at all.
+
+Moving `Configurations/` into a private repo would make `git diff` and `git revert` the
+recovery mechanism and give the history for free. That overlaps the existing GitHub-mirror
+TODO, which is blocked on the same question — see `Notes_AI-Design.md`, where it also blocks
+AI-assisted conf writes.

Deployment/conf_populate.sh

@@ -10,7 +10,7 @@
 # EMPTY fields, never overwrites existing values unless --overwrite is passed.
 #
 # After populating, pushes the updated conf to all partners via conf_sync.sh
-# so they have the fresh keys in their /tmp/.cache/vv/d/ cache immediately.
+# so they have the fresh keys in their /tmp/varaverk/conf/ cache immediately.
 #
 # ==============================================================================================
 # AUTO-DETECTED FIELDS
@@ -55,6 +55,97 @@
 #   GITEA_CONTAINER           fuzzy match from docker ps
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Resolve this host's conf from MY_ID
+#   2. Per service, locate its container and read its own config file:
+#        arrs      → config.xml via the container's /config volume mount
+#        SABnzbd   → sabnzbd.ini
+#        slskd     → config.yml
+#        qBit      → qBittorrent.conf (plaintext WebUI credentials only)
+#        Emby/JF   → docker port bindings and /transcode mount
+#   3. Write each value only if the conf field is EMPTY, unless --overwrite
+#   4. Push the updated conf to partners via conf_sync.sh, unless --no-push
+#
+# Container names are resolved by _resolve_container(): an exact name match wins, otherwise
+# a prefix match must be unambiguous or the field is skipped.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Never Overwrite a Human's Value
+#   Only empty fields are populated. A value already in the conf was either set deliberately
+#   or populated from a service that has since changed — either way, the file wins over
+#   detection. --overwrite exists for deliberate re-sync after a key rotation.
+#
+# Read From the Service, Not From Assumption
+#   Every value comes out of the service's own config file or docker metadata — ports from
+#   actual port bindings, paths from actual volume mounts. Nothing is derived from naming
+#   convention where the real value is readable.
+#
+# Refuse to Guess a Container
+#   An ambiguous prefix skips the field rather than picking one. Writing the wrong container
+#   name is worse than writing nothing: an empty field is visibly incomplete and gets fixed,
+#   while a wrong one silently points the whole stack at the wrong instance. This host has a
+#   live example — "authelia" prefix-matches both Authelia and Authelia-Secondary.
+#
+# Push Immediately After Populating
+#   Fresh credentials go to partners right away rather than waiting for the next scheduled
+#   conf sync, so a partner is never authenticating with a key this host has already rotated.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Reads service config files owned by container users and writes the host conf.
+#
+# Host Detection
+#   detect_hosts() resolves MY_ID, which selects which host conf is written. Populating the
+#   wrong host's conf would write this machine's credentials into a partner's file.
+#
+# Conf Existence Guard
+#   Aborts if the resolved host conf does not exist, rather than creating a partial one.
+#
+# Empty-Field-Only Writes
+#   Existing values are preserved unless --overwrite is passed explicitly.
+#
+# Container Ambiguity Guard
+#   _resolve_container() refuses a prefix matching more than one container, warning with the
+#   full match list. Exact name matches short-circuit and are never treated as ambiguous.
+#
+# Missing Service Tolerance
+#   A service that is not installed on this host is skipped with a log line. Absence is a
+#   valid configuration, not a failure.
+#
+# Map Block Validation
+#   Associative-array entries are only inserted when the target map actually exists in the
+#   conf; a missing map warns and skips instead of appending an orphaned entry.
+#
+# Dry Run Support
+#   --dry-run reports every value it would write, truncated, and writes nothing.
+#
+# Credential Truncation in Output
+#   Detected secrets are printed truncated (first 8 chars) so a populate run can be pasted
+#   into a log or issue without leaking full API keys.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# Reads and writes: Configurations/<hostid>.conf and Configurations/master.conf
+#
+# master.conf
+#
+#   DOCKER_APPDATA_BASE
+#       Fallback appdata root when a container exposes no /config mount to read from.
+#
+# Everything else this script touches is a field it populates rather than one it consumes —
+# see AUTO-DETECTED FIELDS above for the full list.
+#
+# ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================
 #
@@ -67,7 +158,11 @@
 # ==============================================================================================
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-source "$SCRIPT_DIR/../../../load_config.sh"
+# Deployment/ sits one level under the repo root, not three. The old ../../../ resolved to
+# /boot/config on a flash install and /mnt/user on an appdata one — outside the repo either
+# way, so this sourced nothing and every helper below was "command not found". Onboard Step 11
+# has been failing on that since it was written.
+source "$SCRIPT_DIR/../load_config.sh"
 SCRIPTS_ROOT="$SCRIPTS_DIR"
 
 OVERWRITE=false
@@ -174,11 +269,41 @@ _set_conf_map_entry() {
     (( UPDATED++ ))
 }
 
+# ── Helper: resolve a container name from a prefix, refusing ambiguity ────────
+# grep -m1 silently returns whichever name docker happens to list first. On this very host
+# "^authelia" matches both Authelia (9091, primary) and Authelia-Secondary (9092), and -m1
+# picks the secondary — writing the wrong container into the conf that every other script
+# then trusts. An exact match wins outright; otherwise a prefix match must be unambiguous.
+# Same rule detect_hosts() applies to host identity: exactly one candidate, or none.
+_resolve_container() {
+    local pattern="$1"
+    local all exact matches count
+
+    all=$(docker ps -a --format '{{.Names}}' 2>/dev/null)
+    [[ -z "$all" ]] && return 1
+
+    # Exact name match short-circuits — "Authelia" is not ambiguous with "Authelia-Secondary"
+    exact=$(printf '%s\n' "$all" | grep -ixm1 -- "$pattern")
+    [[ -n "$exact" ]] && { echo "$exact"; return 0; }
+
+    matches=$(printf '%s\n' "$all" | grep -i -- "^${pattern}")
+    count=$(printf '%s\n' "$matches" | grep -c .)
+
+    if [[ "$count" -gt 1 ]]; then
+        warn "Container prefix '${pattern}' is ambiguous — matches: $(printf '%s' "$matches" | tr '\n' ' ')"
+        warn "  Refusing to guess. Set the container name manually in host*.conf."
+        return 1
+    fi
+
+    [[ "$count" -eq 1 ]] && { printf '%s\n' "$matches"; return 0; }
+    return 1
+}
+
 # ── Helper: find arr config dir via docker volume mount ───────────────────────
 _arr_config_dir() {
     local pattern="$1"
     local container_name
-    container_name=$(docker ps -a --format '{{.Names}}' 2>/dev/null | grep -im1 "^${pattern}")
+    container_name=$(_resolve_container "$pattern") || return 1
     [[ -z "$container_name" ]] && return 1
 
     local config_path
@@ -239,7 +364,7 @@ _set_conf_var "${MY_ID}_SSH_KEY" "/root/.ssh/${owner}_rsync_automation"        "
 
 for arr in radarr sonarr lidarr; do
     arr_upper="${arr^^}"
-    arr_container=$(docker ps -a --format '{{.Names}}' 2>/dev/null | grep -im1 "^${arr}")
+    arr_container=$(_resolve_container "$arr")
     config_dir=$(_arr_config_dir "$arr") || {
         log "${arr_upper}: no running container found — skipping"
         continue
@@ -341,7 +466,7 @@ qbit_dir=$(_arr_config_dir "qbittorrent") && {
 transcode_dir=""
 
 for pattern in "emby" "jellyfin"; do
-    container=$(docker ps -a --format '{{.Names}}' 2>/dev/null | grep -im1 "^${pattern}")
+    container=$(_resolve_container "$pattern") || continue
     [[ -z "$container" ]] && continue
 
     case "$pattern" in

Deployment/conf_upgrade.sh

@@ -19,9 +19,104 @@
 # associative arrays (declare -A).
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Validate both --template and --target exist
+#   2. Parse each into keys, handling scalars, indexed arrays and declare -A
+#   3. Classify every key: ADDED (template only) / REMOVED (target only) / KEPT (both)
+#   4. Emit the merged result — template structure, target values — to a temp file
+#      staged in the target's own directory
+#   5. --dry-run stops here and prints the report
+#   6. --backup copies the current target to .bak
+#   7. Install by atomic rename over the target
+#
+# Called automatically by git_pull_execute.sh after every pull, for master.conf and this
+# host's own host*.conf. The host template is HOSTN_-prefixed and the caller substitutes
+# the real MY_ID before merging.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# The User's Value Always Wins
+#   For any key present in both files, the target's value is kept and the template's is
+#   discarded. The template supplies structure and new keys, never settings. This is what
+#   makes the upgrade safe to run unattended after every single pull.
+#
+# Structure Follows the Template
+#   Comments, ordering and blank lines come from the template, so an upgraded conf reads
+#   like the current version rather than accumulating layers of old formatting.
+#
+# Standalone by Design — Do Not Add load_config.sh
+#   This script sources nothing. It is the tool that repairs the conf that load_config.sh
+#   depends on, so it has to work when that conf is broken, partial, or missing keys.
+#   Sourcing load_config.sh here would make the repair tool fail in exactly the situation
+#   it exists for. That is also why it uses plain echo instead of log()/error(), and why
+#   there is no acquire_lock — common.sh is not available to it.
+#
+# Atomic Install, Never In-Place
+#   The merged conf is renamed over the target, not copied into it. Every watchdog sources
+#   load_config.sh on every run; a cp would truncate master.conf and write into it, and
+#   anything reading during that window gets a partial conf with empty path variables.
+#
+# Concurrency Handled by Atomicity, Not a Lock
+#   Two concurrent runs against the same target cannot corrupt it — each stages its own
+#   temp file and the rename is atomic, so the last writer simply wins. Since the merge is
+#   idempotent, that outcome is identical to running once.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
+# The Host Slot Must Be Declared, Never Inferred
+#   host.conf.template ships HOSTN_ placeholders; a live host conf uses HOST1_ / HOST2_. Key
+#   matching is literal, so merging the raw template against a real host conf classifies every
+#   existing key as deprecated — KEPT 0 — and the install drops every credential in the file.
+#   Verified on HOST1 2026-08-02: it would have removed HOST1_RADARR_API_KEY,
+#   HOST1_EMBY_API_KEY, HOST1_NPM_PASS and 137 others.
+#
+#   --host-slot HOST<n> is how a caller says which slot the template is being resolved for, and
+#   it works for any slot — HOST1, HOST2, and whatever a third server would be. The slot is
+#   still never inferred from the target and applied silently: the caller declares it, and the
+#   target is read only to contradict a wrong answer. A declared slot that disagrees with the
+#   target's own keys aborts, because substituting for the wrong slot destroys the file just as
+#   thoroughly as not substituting at all. Without the flag, a template containing HOSTN is
+#   refused exactly as before.
+#
+#   Both cases are substituted. HOSTN_ covers the key prefixes, bare HOSTN appears in section
+#   comments, and lowercase hostn is a real value — the hostn-appdata rsync profile keys. A
+#   substitution handling only HOSTN_ leaves a conf carrying a profile named hostn-appdata that
+#   nothing references.
+#
+# A Target Owning Two Slots Is Refused
+#   A host conf describes exactly one server. Finding both HOST1_ and HOST2_ key definitions in
+#   one target means it is not the file it claims to be, so the slot cross-check has nothing
+#   trustworthy to compare against and the run aborts rather than picking one.
+#
+# Total Mismatch Is Refused
+#   Keeping nothing from a populated conf is never a real upgrade; it means the two files do
+#   not describe the same thing. KEPT 0 with a non-empty REMOVED list aborts. This is the
+#   general net behind the HOSTN check — the failure mode is silent and total, so it fires in
+#   --dry-run as well, putting the warning in the report itself.
+#
+# Root Required to Write
+#   Installing over the target requires root. --dry-run deliberately does not, so the
+#   change report can be previewed by anyone.
+#
+# Atomic Rename
+#   The temp file is created in the target's own directory so the rename stays within one
+#   filesystem. On Unraid /tmp is rootfs while the confs live on flash, and a cross-device
+#   mv degrades to copy-then-unlink — precisely the torn write this avoids.
+#
+# Permissions Preserved
+#   mktemp creates 0600; the target's existing mode and owner are copied onto the temp file
+#   before it is installed, so a conf does not come back with different permissions.
+#
+# Temp File Cleanup
+#   An EXIT trap removes the staged file on any early exit, and is cleared once the rename
+#   has succeeded so the trap cannot delete the installed conf.
+#
 # Dry-run Mode
 #   --dry-run prints the full change report (ADDED / REMOVED / KEPT) then exits
 #   without writing anything. Always preview before applying to production confs.
@@ -34,10 +129,6 @@
 #   Both --template and --target are validated before any parsing begins.
 #   Missing files abort immediately with a clear error.
 #
-# Atomic Write
-#   Merged output is written to a tempfile first, then copied to the target.
-#   A partial write cannot corrupt the original.
-#
 # ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
@@ -46,6 +137,9 @@
 #
 #   --template  <file>    New version conf file (source of structure and defaults)
 #   --target    <file>    Existing user conf (source of real values — always preserved)
+#   --host-slot <HOSTn>   Resolve HOSTN/hostn placeholders to this slot before merging.
+#                         Required for host.conf.template; meaningless for master.conf.template,
+#                         which has no placeholders. Must match the target's own slot.
 #   --dry-run             Show what would change without writing
 #   --backup              Write a .bak copy of target before modifying
 #
@@ -62,6 +156,11 @@
 # conf_upgrade.sh --template Deployment/master.conf.template --target Configurations/master.conf
 #     Apply the upgrade in-place with no backup.
 #
+# conf_upgrade.sh --template Deployment/host.conf.template --target Configurations/host1.conf \
+#                 --host-slot HOST1 --dry-run
+#     Preview a host conf upgrade. HOSTN/hostn are resolved to HOST1/host1 first. Swap in HOST2
+#     and host2.conf for the other server — the template is the same file for every slot.
+#
 # ==============================================================================================
 
 set -uo pipefail
@@ -72,11 +171,14 @@ TEMPLATE=""
 TARGET=""
 DRY_RUN=false
 BACKUP=false
+HOST_SLOT=""
+_RESOLVED_TMPL=""   # set only when --host-slot triggers a substitution; cleaned on exit
 
 while [[ $# -gt 0 ]]; do
     case "$1" in
         --template)  TEMPLATE="$2";  shift 2 ;;
         --target)    TARGET="$2";    shift 2 ;;
+        --host-slot) HOST_SLOT="$2"; shift 2 ;;
         --dry-run)   DRY_RUN=true;   shift   ;;
         --backup)    BACKUP=true;    shift   ;;
         *) echo "Unknown option: $1" >&2; exit 1 ;;
@@ -88,6 +190,80 @@ done
 [[ -f "$TEMPLATE" ]] || { echo "Error: template not found: $TEMPLATE" >&2; exit 1; }
 [[ -f "$TARGET"   ]] || { echo "Error: target not found: $TARGET"     >&2; exit 1; }
 
+# ── Host slot resolution ─────────────────────────────────────────────────────────────────────
+#
+# host.conf.template ships HOSTN_ placeholders; a live host conf uses HOST1_ / HOST2_. Key
+# matching below is literal, so merging the raw template against a real host conf classifies
+# EVERY existing key as deprecated and every template key as new — KEPT 0, and the install
+# would drop every credential in the file.
+#
+# --host-slot is how a caller declares which slot the template is for. The slot is never
+# inferred from the target and silently applied: the caller states it, and the target is used
+# only to contradict a wrong answer. Substituting for the wrong slot is the same catastrophe as
+# not substituting at all, so a declared slot that disagrees with the target is refused.
+#
+# Both cases matter. HOSTN_ covers the 159 key prefixes; bare HOSTN appears in section comments,
+# and lowercase hostn is a real value — the hostn-appdata rsync profile keys. A substitution
+# that only handles HOSTN_ leaves a live conf with a profile named hostn-appdata that nothing
+# references, which is what the pull script did before this flag existed.
+_target_slots=$(grep -oE '^[[:space:]]*HOST[0-9]+_' "$TARGET" 2>/dev/null \
+                | grep -oE 'HOST[0-9]+' | sort -u)
+_target_slot=$(echo "$_target_slots" | head -1)
+if [[ $(echo "$_target_slots" | grep -c .) -gt 1 ]]; then
+    echo "Error: '$TARGET' defines keys for more than one host slot:" >&2
+    echo "         $(echo "$_target_slots" | tr '\n' ' ')" >&2
+    echo "       A host conf owns exactly one slot. Refusing rather than picking one." >&2
+    exit 1
+fi
+
+if [[ -n "$HOST_SLOT" ]]; then
+    if ! [[ "$HOST_SLOT" =~ ^HOST[0-9]+$ ]]; then
+        echo "Error: --host-slot must be HOST<n> (e.g. HOST1, HOST2) — got '$HOST_SLOT'" >&2
+        exit 1
+    fi
+    if [[ -n "$_target_slot" && "$_target_slot" != "$HOST_SLOT" ]]; then
+        echo "Error: --host-slot says $HOST_SLOT but '$TARGET' defines ${_target_slot}_ keys." >&2
+        echo "       Substituting for the wrong slot removes every key the target actually has," >&2
+        echo "       credentials included. Refusing." >&2
+        exit 1
+    fi
+    if grep -q -e 'HOSTN' -e 'hostn' "$TEMPLATE" 2>/dev/null; then
+        _lower=$(echo "$HOST_SLOT" | tr '[:upper:]' '[:lower:]')
+        _RESOLVED_TMPL="$(mktemp)"
+        trap '[[ -n "${_RESOLVED_TMPL:-}" ]] && rm -f "$_RESOLVED_TMPL"' EXIT
+        sed -e "s/HOSTN/${HOST_SLOT}/g" -e "s/hostn/${_lower}/g" "$TEMPLATE" > "$_RESOLVED_TMPL"
+        TEMPLATE="$_RESOLVED_TMPL"
+        echo "  Resolved HOSTN → ${HOST_SLOT} for this host"
+    fi
+elif grep -q 'HOSTN' "$TEMPLATE" 2>/dev/null; then
+    # No slot declared and the template is still generic — the original refusal, unchanged.
+    if [[ -n "$_target_slot" ]]; then
+        _lower=$(echo "$_target_slot" | tr '[:upper:]' '[:lower:]')
+        echo "Error: template still contains HOSTN placeholders, but the target uses ${_target_slot}_." >&2
+        echo "       Merging as-is would classify all ${_target_slot}_ keys as deprecated and remove" >&2
+        echo "       them — including every credential. Declare the slot:" >&2
+        echo "" >&2
+        echo "         $0 --template $TEMPLATE --target $TARGET --host-slot ${_target_slot} --dry-run" >&2
+        echo "" >&2
+        exit 1
+    fi
+fi
+
+# ── Block-parsing patterns ───────────────────────────────────────────────────────────────────
+# All three walkers below share these. A single-line array — KEY=(a b c) — both opens and
+# closes on one line. Matching only the opener leaves the walker inside a block it never
+# leaves, so every key until the next standalone ")" becomes invisible: skipped by
+# _collect_stats, dropped from HOST_MAP by _parse_target, and emitted from the template
+# instead of the user's conf by _write_merged. _ARR_ONELINE_RE is what stops that.
+
+_ARR_DECL_RE='^[[:space:]]*declare[[:space:]]+-[a-zA-Z]+[[:space:]]+([A-Z0-9_]+)[[:space:]]*=\('
+_ARR_OPEN_RE='^[[:space:]]*([A-Z0-9_]+)[[:space:]]*=\('
+_ARR_CLOSE_RE='^[[:space:]]*\)[[:space:]]*(#.*)?$'
+# The [^#]* is deliberate: it forces the closing ")" to appear before any comment, so an
+# opener like  FOO=(  # see note (here)  is not mistaken for a complete single-line array.
+_ARR_ONELINE_RE='=\([^#]*\)[[:space:]]*(#.*)?$'
+_SCALAR_RE='^[[:space:]]*([A-Z0-9_]+)[[:space:]]*='
+
 # ── Parse target → KEY → full definition block ───────────────────────────────────────────────
 
 declare -A HOST_MAP   # KEY → complete definition line(s) from user's conf
@@ -99,19 +275,23 @@ _parse_target() {
         if [[ "$in_block" == true ]]; then
             cur_block+="$line"$'\n'
             # Closing ) — optional trailing whitespace and comment
-            if [[ "$line" =~ ^[[:space:]]*\)[[:space:]]*(#.*)?$ ]]; then
+            if [[ "$line" =~ $_ARR_CLOSE_RE ]]; then
                 HOST_MAP["$cur_key"]="$cur_block"
                 in_block=false; cur_key=""; cur_block=""
             fi
         else
-            # declare -A KEY=(
+            # declare -A KEY=(   /   KEY=(
-            if [[ "$line" =~ ^[[:space:]]*declare[[:space:]]+-[a-zA-Z]+[[:space:]]+([A-Z0-9_]+)[[:space:]]*=\( ]]; then
+            if [[ "$line" =~ $_ARR_DECL_RE ]] || [[ "$line" =~ $_ARR_OPEN_RE ]]; then
-                cur_key="${BASH_REMATCH[1]}"; in_block=true; cur_block="$line"$'\n'
+                cur_key="${BASH_REMATCH[1]}"
-            # KEY=(
+                # KEY=(a b c) — opens and closes on one line, never enter block mode
-            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*=\( ]]; then
+                if [[ "$line" =~ $_ARR_ONELINE_RE ]]; then
-                cur_key="${BASH_REMATCH[1]}"; in_block=true; cur_block="$line"$'\n'
+                    HOST_MAP["$cur_key"]="$line"$'\n'
+                    cur_key=""
+                else
+                    in_block=true; cur_block="$line"$'\n'
+                fi
             # KEY=value  (simple scalar)
-            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*= ]]; then
+            elif [[ "$line" =~ $_SCALAR_RE ]]; then
                 HOST_MAP["${BASH_REMATCH[1]}"]="$line"$'\n'
             fi
         fi
@@ -128,7 +308,7 @@ _collect_stats() {
 
     while IFS= read -r line || [[ -n "$line" ]]; do
         if [[ "$in_block" == true ]]; then
-            if [[ "$line" =~ ^[[:space:]]*\)[[:space:]]*(#.*)?$ ]]; then
+            if [[ "$line" =~ $_ARR_CLOSE_RE ]]; then
                 in_block=false
                 TMPL_SEEN["$cur_key"]=1
                 if [[ -n "${HOST_MAP[$cur_key]+_}" ]]; then KEPT+=("$cur_key")
@@ -136,11 +316,17 @@ _collect_stats() {
                 cur_key=""
             fi
         else
-            if [[ "$line" =~ ^[[:space:]]*declare[[:space:]]+-[a-zA-Z]+[[:space:]]+([A-Z0-9_]+)[[:space:]]*=\( ]]; then
+            if [[ "$line" =~ $_ARR_DECL_RE ]] || [[ "$line" =~ $_ARR_OPEN_RE ]]; then
-                cur_key="${BASH_REMATCH[1]}"; in_block=true
+                cur_key="${BASH_REMATCH[1]}"
-            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*=\( ]]; then
+                if [[ "$line" =~ $_ARR_ONELINE_RE ]]; then
-                cur_key="${BASH_REMATCH[1]}"; in_block=true
+                    TMPL_SEEN["$cur_key"]=1
-            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*= ]]; then
+                    if [[ -n "${HOST_MAP[$cur_key]+_}" ]]; then KEPT+=("$cur_key")
+                    else                                        ADDED+=("$cur_key"); fi
+                    cur_key=""
+                else
+                    in_block=true
+                fi
+            elif [[ "$line" =~ $_SCALAR_RE ]]; then
                 local k="${BASH_REMATCH[1]}"
                 TMPL_SEEN["$k"]=1
                 if [[ -n "${HOST_MAP[$k]+_}" ]]; then KEPT+=("$k")
@@ -163,18 +349,23 @@ _write_merged() {
     while IFS= read -r line || [[ -n "$line" ]]; do
         if [[ "$in_block" == true ]]; then
             cur_block+="$line"$'\n'
-            if [[ "$line" =~ ^[[:space:]]*\)[[:space:]]*(#.*)?$ ]]; then
+            if [[ "$line" =~ $_ARR_CLOSE_RE ]]; then
                 in_block=false
                 if [[ -n "${HOST_MAP[$cur_key]+_}" ]]; then printf '%s' "${HOST_MAP[$cur_key]}"
                 else                                        printf '%s' "$cur_block"; fi
                 cur_key=""; cur_block=""
             fi
         else
-            if [[ "$line" =~ ^[[:space:]]*declare[[:space:]]+-[a-zA-Z]+[[:space:]]+([A-Z0-9_]+)[[:space:]]*=\( ]]; then
+            if [[ "$line" =~ $_ARR_DECL_RE ]] || [[ "$line" =~ $_ARR_OPEN_RE ]]; then
-                cur_key="${BASH_REMATCH[1]}"; in_block=true; cur_block="$line"$'\n'
+                cur_key="${BASH_REMATCH[1]}"
-            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*=\( ]]; then
+                if [[ "$line" =~ $_ARR_ONELINE_RE ]]; then
-                cur_key="${BASH_REMATCH[1]}"; in_block=true; cur_block="$line"$'\n'
+                    if [[ -n "${HOST_MAP[$cur_key]+_}" ]]; then printf '%s' "${HOST_MAP[$cur_key]}"
-            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*= ]]; then
+                    else                                        printf '%s\n' "$line"; fi
+                    cur_key=""
+                else
+                    in_block=true; cur_block="$line"$'\n'
+                fi
+            elif [[ "$line" =~ $_SCALAR_RE ]]; then
                 local k="${BASH_REMATCH[1]}"
                 if [[ -n "${HOST_MAP[$k]+_}" ]]; then printf '%s' "${HOST_MAP[$k]}"
                 else                                  printf '%s\n' "$line"; fi
@@ -208,6 +399,24 @@ fi
 
 echo "  KEPT    ${#KEPT[@]} existing vars — your values preserved"
 
+# ── Guard: total mismatch ────────────────────────────────────────────────────────────────────
+#
+# Keeping nothing from a populated conf is never a real upgrade — it is the signature of the
+# two files not describing the same thing (wrong template, wrong target, unsubstituted
+# placeholders). The HOSTN check above catches the known cause; this catches the rest, because
+# the failure mode is silent and total: every value in the file is replaced by a default.
+# Deliberately fires in --dry-run too, so the report itself carries the warning.
+if [[ ${#KEPT[@]} -eq 0 && ${#REMOVED[@]} -gt 0 ]]; then
+    echo "────────────────────────────────────────────────────────────────────────"
+    echo ""
+    echo "Error: refusing — this would keep NOTHING and remove all ${#REMOVED[@]} existing keys." >&2
+    echo "       A genuine upgrade preserves values; keeping zero means the template and the" >&2
+    echo "       target do not describe the same conf. Check that --template matches --target" >&2
+    echo "       and that any HOSTN placeholders were substituted for this host's slot." >&2
+    echo "" >&2
+    exit 1
+fi
+
 if [[ ${#ADDED[@]} -eq 0 && ${#REMOVED[@]} -eq 0 ]]; then
     echo "  Already up to date — no changes needed."
     echo "────────────────────────────────────────────────────────────────────────"
@@ -225,15 +434,36 @@ if [[ "$DRY_RUN" == true ]]; then
     exit 0
 fi
 
-TMPOUT="$(mktemp)"
+# Checked here rather than at the top: --dry-run is a read-only report and is useful to
-trap 'rm -f "$TMPOUT"' EXIT
+# anyone, but installing over a conf under /boot needs root. Plain echo because this script
+# deliberately does not source common.sh — see the header.
+if [[ "$EUID" -ne 0 ]]; then
+    echo "ERROR: writing '$TARGET' requires root (use --dry-run to preview as any user)" >&2
+    exit 1
+fi
+
+# Staged beside the target, not in /tmp. mv is only atomic within one filesystem, and on
+# Unraid /tmp is rootfs while the confs live on flash — a cross-device mv silently degrades
+# to copy-then-unlink, which is exactly the torn write this is meant to prevent.
+TMPOUT="$(mktemp "${TARGET}.XXXXXX")"
+trap 'rm -f "$TMPOUT"; [[ -n "${_RESOLVED_TMPL:-}" ]] && rm -f "$_RESOLVED_TMPL"' EXIT
+
+# mktemp creates 0600; carry the target's existing mode/owner across so the installed conf
+# does not come back with different permissions than it went in with.
+chmod --reference="$TARGET" "$TMPOUT" 2>/dev/null || true
+chown --reference="$TARGET" "$TMPOUT" 2>/dev/null || true
 
 _write_merged > "$TMPOUT"
 
 if [[ "$BACKUP" == true ]]; then
-    cp "$TARGET" "${TARGET}.bak"
+    cp -a "$TARGET" "${TARGET}.bak"
     echo "Backup: ${TARGET}.bak"
 fi
 
-cp "$TMPOUT" "$TARGET"
+# Atomic install. cp would truncate the live conf and write into it, leaving a window where
+# anything sourcing load_config.sh reads a half-written master.conf — every watchdog does
+# that constantly. A rename swaps the inode: readers get the old file or the new one.
+mv -f "$TMPOUT" "$TARGET"
+trap - EXIT
+[[ -n "$_RESOLVED_TMPL" ]] && rm -f "$_RESOLVED_TMPL"
 echo "Updated: $TARGET"

Deployment/host.conf.template

@@ -67,6 +67,7 @@
 #   CERTIFICATE MONITOR     domains checked for SSL expiry
 #   SMART HEALTH            drives to skip in SMART monitoring
 #   ZFS REPORT              pools to exclude from ZFS health report
+#   PCIe AER QUIET          dead PCIe devices removed at array start to stop AER log spam
 #
 #   ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
 #   RESOURCE MANAGER        containers paused/stopped under memory pressure
@@ -94,7 +95,7 @@
 
 # ━━━ Identity ━━━
 # HOSTN hostname lives in master.conf (not a credential — safe for all servers).
-# SSH key used for all server-to-server operations — rsync, failover, conf sync.
+# SSH key used for all server-to-server operations — rsync, fallback, conf sync.
 # Convention: /root/.ssh/<hostname-lowercase-no-unraid-prefix>_rsync_automation
 # Must be in /root/.ssh/ and authorised in the partner's /root/.ssh/authorized_keys.
 # Run Partnership/ssh_setup.sh to generate the key and copy it to the partner.
@@ -112,6 +113,11 @@
     HOSTN_EMBY_CONTAINER="Emby"
     HOSTN_EMBY_URL="http://localhost:8096"
     HOSTN_EMBY_API_KEY=""   # Emby Dashboard → API Keys → + New Key
+    HOSTN_EMBY_PUBLIC_URL="" # e.g. https://media.example.com/emby — browser-reachable base, used
+                            # to build image URLs that render in the WebGUI. Deliberately separate
+                            # from HOSTN_EMBY_URL: that one is for server-side API calls and is
+                            # usually localhost, which resolves to the wrong machine in a browser.
+                            # Empty = features that need an image quietly go without one.
 
 # ━━━ Jellyfin ━━━
     HOSTN_JELLYFIN_CONTAINER="Jellyfin"
@@ -123,6 +129,17 @@
 # Create in Gitea: Settings → Applications → Generate Token → scope: write:user
     HOSTN_GITEA_API_TOKEN=""
 
+# ━━━ Bug Reports ━━━
+# Only used when BUG_REPORT_LOCAL_ENABLED=true in master.conf. Reports then go to this Gitea
+# instead of GitHub — and stay there, so they do not reach the Varaverk maintainer.
+#
+# Reached locally or over Tailscale, so the token never crosses the public proxy and no Authelia
+# bypass is needed. It is a credential and lives here rather than master.conf for that reason;
+# it is never shipped, and the settings UI masks it.
+    HOSTN_BUG_REPORT_URL=""      # e.g. http://gitea:3000 or the tailnet name
+    HOSTN_BUG_REPORT_REPO=""     # owner/repo
+    HOSTN_BUG_REPORT_TOKEN=""    # Gitea API token with issue-write on that repo
+
 # ━━━ Notifications ━━━
 # Discord webhook — leave blank to disable.
     HOSTN_DISCORD_WEBHOOK=""
@@ -138,6 +155,23 @@
         # "Authelia|9091"
     )
 
+# Shares rsynced to the mirror during onboard Step 1e, BEFORE the auth containers are created.
+# This is the only rsync an onboard performs — media is never seeded here.
+# Profile is inferred from the directory basename, so Critical-Data resolves to critical-data:
+# a clean copy with the auth containers stopped on both sides. Do not point this at a share
+# whose profile keeps databases running; a dirty copy of MariaDB or Redis is worse than none,
+# because the container starts, reads Up, and restarts a dead database behind it.
+    HOSTN_PARTNERSHIP_PROVISION_SHARES=(
+        "/mnt/user/appdata-Fallback/Critical-Data"   # critical-data profile — the auth stack
+    )
+
+# Containers that belong in "<PartnerShort>-Fallback" on the mirror rather than in a mirrored
+# copy of this host's folder layout — the ones that exist there only to cover this host going
+# dark. Everything else the onboard deploys is filed onto the same shelf it occupies here
+# (Arrs Stack, Networking, Databases…), because it runs on the mirror continuously.
+# Empty is the normal state: leave it empty unless a container is genuinely failover-only.
+    HOSTN_PARTNERSHIP_FALLBACK_ONLY=()
+
 # XML templates pushed to mirror during onboard — auth stack.
 # Dependencies (databases) must come before apps that depend on them.
     HOSTN_PARTNERSHIP_AUTH_STACK=(
@@ -453,6 +487,8 @@
     HOSTN_RADARR_ANIME_ROOT=""   # rootFolderPath literal, as reported by Radarr API — leave blank if no dedicated anime root
     HOSTN_RADARR_DOWNLOAD_DIR=""            # host path of the completed-downloads folder Radarr imports from (e.g. "/mnt/cache/Temp_Storage/SABnzbd/Completed/Movies") — blank disables the download orphan cleaner for Radarr
     HOSTN_RADARR_DOWNLOAD_CONTAINER_DIR=""  # same folder as Radarr's container sees it (e.g. "/downloads/Completed/Movies") — needed to trigger import scans on held folders
+    HOSTN_LIDARR_DOWNLOAD_DIR=""            # host path of the completed-downloads folder Lidarr imports from (e.g. "/mnt/cache/Temp_Storage/SABnzbd/Completed/Music") — blank disables the download orphan cleaner for Lidarr
+    HOSTN_LIDARR_DOWNLOAD_CONTAINER_DIR=""  # same folder as Lidarr's container sees it (e.g. "/downloads/Completed/Music") — needed to trigger import scans on held folders
 
     declare -A HOSTN_RADARR_PATH_MAP=(
         # ["/movies"]="/mnt/user/Movies"
@@ -510,6 +546,19 @@
         "sda"   # boot USB — SMART not meaningful on flash drives
     )
 
+# ━━━ PCIe AER Quiet ━━━
+# PCI addresses removed from the bus at array start so dead hardware stops spamming
+# correctable AER errors into syslog. Full DDDD:BB:DD.F form — find them with:
+#   grep -o "from [0-9a-f:.]*" /var/log/syslog | sort | uniq -c | sort -rn
+#
+# Only devices bound to vfio-pci or to no driver at all are eligible. Anything with a
+# live driver is refused, so a mistyped address cannot pull an HBA or NIC out from
+# under a running system. Devices claimed by a running VM are refused too.
+# Gated by PCIE_QUIET_ENABLED in master.conf. Empty list = no-op.
+    HOSTN_PCIE_QUIET_DEVICES=(
+        # "0000:03:00.0"
+    )
+
 # ━━━ ZFS Report ━━━
     HOSTN_ZFS_REPORT_IGNORE_POOLS=(
         # "disk5"
@@ -569,8 +618,29 @@
     HOSTN_LLDAP_USER="admin"                    # lldap admin username
     HOSTN_LLDAP_PASS=""                         # lldap admin password
 
+
+# ==============================================================================================
+# ── Ollama / AI ───────────────────────────────────────────────────────────────────────────────
+# ==============================================================================================
+# Per-host because only some nodes actually have a GPU. A node with an empty OLLAMA_URL is not
+# an error — it falls through to the resolver and uses another node's Ollama over Tailscale.
+
+# ━━━ Ollama ━━━
+    HOSTN_OLLAMA_URL=""                          # e.g. http://localhost:11434 — empty if no local Ollama
+    HOSTN_OLLAMA_GPU_UUID=""                     # pins Ollama to one card on multi-GPU hosts
+    HOSTN_OLLAMA_MODEL="hf.co/unsloth/Qwen3-14B-GGUF:IQ4_XS"  # generation — must fully offload; see README-AI.md
+
+# ━━━ Web search ━━━
+# Per-host because one is an address on this network and the other is a credential. Only the
+# General Chat profile can use these — it is the profile that cannot change anything, which is
+# why it is the one allowed to look outside. Off until AI_WEB_SEARCH_ENABLED says otherwise.
+    HOSTN_DEGOOG_URL=""                          # e.g. http://localhost:4444 — self-hosted, no key, /api/search
+    HOSTN_SEARXNG_URL=""                         # e.g. http://localhost:8888 — needs format: [json] in its settings.yml
+    HOSTN_WEB_SEARCH_API_KEY=""                  # brave or tavily; unused when the provider is searxng
+    HOSTN_OLLAMA_EMBED_MODEL="nomic-embed-text"  # embeddings — the generation model cannot embed
+
 # ━━━ Authelia ━━━
-    HOSTN_AUTHELIA_CONFIG="/mnt/user/appdata/Authelia/configuration.yml"
+    HOSTN_AUTHELIA_CONFIG="/mnt/user/appdata-Fallback/Critical-Data/Authelia/configuration.yml"
     HOSTN_AUTHELIA_CONTAINER="Authelia"
 
 # ==============================================================================================

Deployment/master.conf.template

@@ -115,12 +115,77 @@
 # scripts, state, and data are all available before the array mounts.
 # Both directories are created automatically if they don't exist.
 #
-#   DATA_DIR   — historical logs, statistics, discovery histories, blocklists
+# DATA_DIR is the one on-disk root. Everything Varaverk persists lives under it, in a
-#   STATE_DIR  — runtime state files for all scripts (watchdogs, fallback, transcode, etc.)
+# subdirectory named for what the files are. Move DATA_DIR and the whole tree follows.
-#                Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no /boot/config root.
+#
+# It used to be two roots plus two strays: DATA_DIR beside State_Files/ as siblings, with the
+# conf-cache backup off in SCRIPTS_DIR/.cache/vv/d and the arr cache backups loose in DATA_DIR's
+# root. Nothing was wrong with any one of those decisions; together they meant no single place
+# answered "what does Varaverk keep on disk". State is data — it is the data that happens to
+# describe right now — so it belongs under the same root as the rest.
+#
+#   db/     — statistics, histories, counters, blocklists. Things that accumulate.
+#   state/  — runtime state for every script: watchdogs, fallback, transcode, setup.
+#             Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no repo root.
+#   ai/     — the retrieval index, operator memory, token ledger, filed bugs, saved chats.
+#   cache/  — persistent backups of the tmpfs caches, and ONLY those. A file belongs here when
+#             losing it costs a re-fetch and nothing else; anything that is a source of truth
+#             belongs in db/ or state/.
+#   logs/   — retained log output. Live logging still goes to LOG_DIR (/var/log/varaverk).
+#
+# STATE_DIR keeps its name and changes only its value, which is why this restructure did not
+# touch the 15 conf entries, 18 shell paths and 23 PHP paths that build on it.
+#
+# The tmpfs caches are NOT here and must not be moved here — see VV_CACHE_ROOT below. These are
+# on flash; those are read every second by the WebGUI and rewritten by the hundred megabytes.
     DATA_DIR="/boot/config/plugins/varaverk/data"
-    STATE_DIR="/boot/config/plugins/varaverk/State_Files"
+    DB_DIR="${DATA_DIR}/db"
-    PERSISTENT_CONF_CACHE="/boot/config/plugins/varaverk/.cache/vv/d"
+    STATE_DIR="${DATA_DIR}/state"
+    AI_DATA_DIR="${DATA_DIR}/ai"
+    CACHE_BACKUP_DIR="${DATA_DIR}/cache"
+    LOG_ARCHIVE_DIR="${DATA_DIR}/logs"
+    BACKUP_DIR="${DATA_DIR}/Backups"
+    CONF_BACKUP_DIR="${BACKUP_DIR}/Confs"        # 0700 — holds credentials
+    PERSISTENT_CONF_CACHE="${CACHE_BACKUP_DIR}/conf"
+    ARR_CACHE_BACKUP_DIR="${CACHE_BACKUP_DIR}/arr"
+
+# ── Cache Roots ──
+# Everything Varaverk keeps in RAM, under one root, defined once.
+#
+# These used to be four literals in load_config.sh and three more in the PHP layer, spread over
+# /tmp/vv_cache, /tmp/arr_cache, /tmp/.cache/vv/d, /tmp/.cache/vv/ai and /tmp/varaverk_ai_jobs —
+# five naming schemes, and no single place that could tell you what Varaverk had in tmpfs. PHP
+# could not read load_config.sh, so it restated the paths it needed and the two layers were kept
+# in agreement by hand. They live here because master.conf is the one file both layers actually
+# read: bash sources it, and the PHP conf parser resolves ${VAR} against it the same way.
+#
+# ONE ROOT, NOT ONE DIRECTORY. The subdirectories are deliberately separate and must stay that
+# way — they have genuinely different rules:
+#
+#   conf/  holds partner credentials, is chmod 700, and IS snapshotted to PERSISTENT_CONF_CACHE
+#          so it survives a reboot.
+#   ai/    holds non-secret token counters that are worthless when stale and must NOT be
+#          preserved across a reboot — losing them means "not collected here" until the next
+#          sync, which is the honest answer.
+#   jobs/  holds in-flight work handed off to a detached worker — an AI answer being generated,
+#          a container action being applied. chmod 700: each is readable by anyone who can guess
+#          its token, which is why the tokens are random_bytes and not sequential.
+#   arr/   is the large one (100s of MB) and is restored from a DATA_DIR backup on demand.
+#   api/   is the WebGUI payload cache, and is the only one safe to delete at any moment.
+#
+# Collapsing those into a single directory would give the credentials the token cache's
+# persistence rules, or the reverse. The win here is one definition, not one folder.
+#
+# Point VV_CACHE_ROOT somewhere else and everything follows. It must be on a filesystem that is
+# cleared or safe to clear on boot — every consumer treats a missing cache as a cold start, and
+# nothing here is a source of truth for anything.
+    VV_CACHE_ROOT="/tmp/varaverk"
+    VV_CACHE_DIR="${VV_CACHE_ROOT}/api"          # WebGUI payload cache — monitor, arrs, ai
+    CONF_RAM_CACHE_DIR="${VV_CACHE_ROOT}/conf"   # partner host*.conf — chmod 700, snapshotted
+    ARR_CACHE_DIR="${VV_CACHE_ROOT}/arr"         # arr payloads — restored from DATA_DIR on demand
+    AI_TOKEN_CACHE_DIR="${VV_CACHE_ROOT}/ai"     # partner token ledgers — never preserved
+    AI_JOB_DIR="${VV_CACHE_ROOT}/jobs/ai"        # in-flight AI answers — chmod 700
+    DOCKER_JOB_DIR="${VV_CACHE_ROOT}/jobs/docker" # in-flight container actions — chmod 700
 
 # ── Version Parity ──
 # Controls behaviour when local and remote unRAID versions differ.
@@ -132,14 +197,21 @@
 # ── Conf Sync ──
 # conf_sync.sh pulls partner host confs into /tmp/.vv/ RAM cache on array start and after
 # any conf save — makes partner vars (HOST2_*, etc.) available without touching git.
-    CONF_SYNC_ENABLED=true
+# Ships OFF for the same reason as RSYNC_ENABLED: a node seeded from this template has not yet
+# been paired with anyone, and pulling a partner's confs before the partnership exists is either
+# a no-op or a surprise. The onboard flow arms it once the pairing is real.
+    CONF_SYNC_ENABLED=false
 
 # ── Arr Sync ──
 # arr_sync.sh syncs Lidarr/Sonarr/Radarr libraries across all nodes bidirectionally.
 # Runs before rsync — all nodes agree on tracked library before files are transferred.
 # Remote API keys are read live from each node's config.xml via SSH — never stored here.
-    ARR_SYNC_ENABLED=true
+# Ships OFF. Bidirectional means a fresh node with an empty library is an equal party to the
-    ARR_SYNC_BLOCKLIST="${DATA_DIR}/arr_sync_blocklist.tsv"
+# merge, and "all nodes agree on tracked library" is not something you want decided by the node
+# that knows nothing. Onboard Step 9 runs arr_sync explicitly during bootstrap; this gate is for
+# the unattended runs afterwards.
+    ARR_SYNC_ENABLED=false
+    ARR_SYNC_BLOCKLIST="${DB_DIR}/arr_sync_blocklist.tsv"
     ARR_SYNC_CONNECT_TIMEOUT=10     # seconds — SSH connect timeout per node
     ARR_SYNC_API_TIMEOUT=60         # seconds — curl timeout for library fetches
     DOCKER_APPDATA_BASE="/mnt/user/appdata"
@@ -201,8 +273,10 @@
     VARAVERK_SETUP_FILE="$STATE_DIR/varaverk_setup.db"
 
 # Tailscale removal on offboard.
-    PARTNERSHIP_REMOVE_TAILSCALE=true       # remove mirror from Tailscale tailnet on offboard
+    PARTNERSHIP_REMOVE_TAILSCALE=false      # opt-in — removes the mirror from the tailnet on offboard.
-                                            # false = skip removal (manual or testing)
+                                            # OFF by default: it needs a TAILSCALE_API_KEY, which
+                                            # expires and must be reissued, and a feature that
+                                            # silently no-ops on a stale key should not be opt-out.
 
 # Tailscale API — required when PARTNERSHIP_REMOVE_TAILSCALE=true.
 # Stays in shared conf — only owner uses it, and owner is always running this script.
@@ -273,7 +347,16 @@
     GITEA_CONTAINER="Gitea"
     GITEA_REPO_PATH=""                  # e.g. YourUser/Varaverk.git
     GITEA_DOMAIN=""                         # e.g. git.yourdomain.com — requires NPM + DNS
-    TARGET_DIR="/mnt/user/appdata/Varaverk"
+# Derived, never a literal. This is the directory git_pull_execute.sh pulls into, and it is the
+# one host-specific path that used to live in master.conf as a fixed string — which made the
+# shared conf unshareable: pushing it to a partner that had migrated its install repointed that
+# partner's pull target at the old location, and the next pull built a second copy there while
+# the live install ran somewhere else.
+#
+# SCRIPTS_DIR is exported by load_config.sh from its own location before this file is sourced, so
+# this resolves to wherever the install actually is, on every node, in either storage mode, with
+# nothing to keep in step. DATA_DIR and STATE_DIR below already worked this way.
+    TARGET_DIR="${SCRIPTS_DIR}"
     GITEA_SSH_KEY="/root/.ssh/unraid_gitea"
     SSH_PORT=221                            # Gitea SSH port (default 22, Gitea often uses 221/222)
     GITEA_HTTP_PORT=3000                    # Gitea web/API port — used by gitea_ssh_setup.sh
@@ -289,16 +372,18 @@
 # Scripts launched by array_started.sh when the array comes online.
 # Launched in order — each as a background process.
 # One-shot scripts (ramdisk, syslog, fpm, inotify, network) run and exit naturally.
-# Continuous scripts (failover) run until array stops.
+# Continuous scripts (fallback) run until array stops.
 # Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via
 # watchdog_orchestrator.sh — NOT launched here.
     ARRAY_START_SCRIPTS=(
+        "Plugin/unraid/System_Essentials/pcie_aer_quiet.sh"         # drop AER-spamming dead hardware — runs first so later logs stay readable
+        "Plugin/unraid/System_Essentials/unraid_api_key_renew.sh"   # re-register Varaverk API key — registry is ephemeral
         "System_Essentials/conf_sync.sh"              # pull partner confs + push own conf into /tmp/.vv/ RAM cache
         "System_Essentials/conf_cache_restore.sh"     # load partner confs from persistent backup if conf_sync couldn't reach partner
         "Transcodes/ramdisk_setup.sh"                 # creates ramdisk + symlink before Emby starts
-        "unRAID_Essentials/docker_syslog_filter.sh"   # suppress veth noise before logs fill
+        "System_Essentials/docker_syslog_filter.sh"   # suppress veth noise before logs fill
-        "unRAID_Essentials/php_fpm_max_children.sh"   # WebGUI performance tuning
+        "Plugin/unraid/System_Essentials/php_fpm_max_children.sh"   # WebGUI performance tuning
-        "unRAID_Essentials/inotify_tuning.sh"         # bump inotify limits — containers miss events if exhausted
+        "System_Essentials/inotify_tuning.sh"         # bump inotify limits — containers miss events if exhausted
         "Docker_Essentials/docker_network_connect.sh" # ensure networks exist + connect containers
         "Arrs_Stack/arr_cache_prefill.sh"               # warm Lidarr/Sonarr/Radarr tracked-data caches before anything reads them cold
         "Arrs_Stack/start_webhook_listener.sh"             # arr upgrade webhook listener — continuous
@@ -311,10 +396,10 @@
 # Order matters: user scripts first (prevents new ops), then data movement, then containers.
     ARRAY_STOP_SCRIPTS=(
         "System_Essentials/conf_cache_save.sh"          # snapshot partner conf RAM cache → /boot before anything stops
-        "unRAID_Essentials/user_scripts_stop.sh"        # stop background scripts before they start new ops
+        "Plugin/unraid/System_Essentials/user_scripts_stop.sh"        # stop background scripts before they start new ops
         "Fallback/fallback.sh --stop"                   # gracefully stop fallback (not caught by user_scripts_stop)
-        "unRAID_Essentials/rsync_stop.sh --rsync-only"  # kill rsync; skip container recovery (handled below)
+        "System_Essentials/rsync_stop.sh --rsync-only"  # kill rsync; skip container recovery (handled below)
-        "unRAID_Essentials/mover_stop.sh"               # stop mover after rsync (they conflict on same files)
+        "Plugin/unraid/System_Essentials/mover_stop.sh"               # stop mover after rsync (they conflict on same files)
         "Docker_Essentials/docker_container_stop.sh"    # stop all containers last
     )
 
@@ -338,8 +423,14 @@
         "Watchdogs/resource_watchdog.sh"   # reduce system pressure before healing attempts
         "Watchdogs/docker_watchdog.sh"     # heal containers with freed resources
         "Watchdogs/system_watchdog.sh"     # system component health — storage + webgui
+        "Plugin/unraid/System_Essentials/unraid_api_key_renew.sh"   # re-register API key if registry lost it
+        "Plugin/unraid/Tools/ai_repair_sweep.sh"  # read what the last cycle logged; off unless AI_REPAIR_ENABLED
         "Watchdogs/stability_watchdog.sh"  # reboot if all else fails — last line of defense
     )
+# The repair sweep sits ahead of stability deliberately, which is the one exception to "stability
+# last". It reads the previous cycle's logs and may correct the very misconfiguration stability
+# would otherwise reboot for — a wrong port is not fixed by restarting the machine. It is bounded
+# by AI_PROBE_TIMEOUT, exits 0 in every case, and does nothing at all unless AI_REPAIR_ENABLED.
 
 # ━━━ System Watchdog ━━━
 # system_watchdog.sh runs SYSTEM_WATCHDOG_SCRIPTS sequentially each cycle.
@@ -399,9 +490,9 @@
         "Arrs_Stack/sonarr_cleanup.sh"                     # remove orphaned TV files
         "Arrs_Stack/radarr_cleanup.sh"                     # remove orphaned movie files
         # Daily, not weekly — DOWNLOAD_ORPHAN_AGE gates each folder on its own mtime, so
-        # frequency costs nothing in safety, but a weekly run clears a whole week of
+        # frequency costs nothing in safety, while a weekly run makes each pass a whole
-        # newly-eligible orphans at once and can exceed DOWNLOAD_ORPHAN_MAX_DELETE_GB, which
+        # week of newly-eligible orphans at once and far more likely to spill past
-        # aborts the pass entirely and rolls the backlog into an even larger next run.
+        # DOWNLOAD_ORPHAN_MAX_DELETE_GB into a deferred remainder.
         "Arrs_Stack/arr_download_orphan_cleaner.sh"        # sweep orphaned completed downloads out of the SAB Completed folders — deletes junk + already-imported leftovers, triggers import scans for genuinely-missing content
         # Runs after cleanup, not before — frees disk space from orphans/junk first, so the
         # searches triggered below (for relocated hasFile=false/episodeFileCount=0 entries)
@@ -412,8 +503,24 @@
         "Arrs_Stack/lidarr_missing_art.sh"                 # fetch missing album/artist artwork (HOST1 only — self-guards)
         "Arrs_Stack/radarr_tmdb_removed.sh"                # remove movies dropped from TMDb
         "Arrs_Stack/sonarr_tvdb_removed.sh"                # remove series dropped from TVDB
+        # Before the two docker jobs below, deliberately: they pull images and restart
+        # containers, so a container can legitimately not exist for a moment while they run.
+        # Reading docker in its settled overnight state keeps a rebuild from costing a strike.
+        # Three strikes at this cadence means three days genuinely absent before conf is edited.
+        "Tools/conf_container_prune.sh"              # drop containers that no longer exist from the conf lists
         "Docker_Essentials/docker_update.sh"          # pull container image updates before restart
         "Docker_Essentials/docker_daily_restart.sh"   # daily container restarts — runs last
+        # Last, and after git_pull_execute.sh which must always run first. Daily rather than with
+        # the Sunday cert monitor: CERT_HISTORY_STRIKES counts passes, so a weekly cadence would
+        # make five strikes five weeks. One NPM list call; it is what gives the Certs tab its
+        # per-domain renewal and failure counts.
+        "Plugin/unraid/Tools/cert_history.sh"         # record cert renewals, failures and age per domain
+        "Plugin/unraid/Tools/npm_access_stats.sh"    # aggregate NPM per-host access logs into request and byte totals
+        # After both of the above, because it reads what they write — the uptime history and the
+        # per-host request totals are two of the four things it reasons from. Daily rather than
+        # hourly: everything it files is a condition that has already lasted hours by the time
+        # AUTH_SWEEP_DOWN_MIN lets it through, so a faster cadence would find nothing new.
+        "Plugin/unraid/Tools/auth_sweep.sh"          # file findings for hosts that are not serving, and hostnames Authelia is not protecting
     )
 
 # Pull latest images for DAILY_RESTART_CONTAINERS before the daily restart.
@@ -424,7 +531,7 @@
 # Media shares synced daily by daily_sync_maintenance.sh.
 # Defined per-host in host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
 # Mesh model: every node pushes every media share. rsync has no --delete so pushes are additive.
-# arr_sync (union) ensures all arr libraries converge first. arr_cleanup removes true orphans.
+# arr_sync (union) ensures all arr libraries converge first. the arr cleanups remove true orphans.
 # Any node can download content to any share — it propagates to all nodes on the next cycle.
 # Adding HOST3: list every media share in HOST3_DAILY_SYNC_SHARES. No ownership to track.
 # These shares use DEFAULT_RSYNC_OPTS — no profile entry needed.
@@ -439,9 +546,9 @@
 # syncs WEEKLY_SYNC_SHARES → restarts → then iterates WEEKLY_MAINTENANCE_SCRIPTS.
 # Schedule: 30 2 * * 0 (Sunday 2:30am)
     WEEKLY_MAINTENANCE_SCRIPTS=(
+        "Docker_Essentials/docker_update.sh --weekly"   # pull latest images for WEEKLY_RESTART_CONTAINERS before restart
         "Docker_Essentials/docker_weekly_restart.sh"    # weekly container restarts after sync
-        "Docker_Essentials/docker_update_remaining.sh"  # pull updates for all other containers
+        "System_Essentials/clear_logs.sh"               # purge aged logs — Sunday only, low priority
-        "unRAID_Essentials/clear_logs.sh"               # purge aged logs — Sunday only, low priority
         "Arrs_Stack/arr_full_rescan.sh"                 # full disk↔DB reconciliation for Lidarr/Sonarr/Radarr — keeps tracked stats honest, runs before discovery so it works off fresh data
         "Arrs_Stack/arr_corruption_scan.sh --remediate"      # ffprobe-based corruption sweep of Sonarr's tracked files — deletes+re-searches only after CORRUPTION_SCAN_STRIKE_LIMIT consecutive hits on the same file
         "Arrs_Stack/playback_aware_lidarr_discovery.sh"      # behavior-driven music discovery using weekly Emby playback history
@@ -466,8 +573,8 @@
 # than restarting them a second time. A file older than DOCKER_UPDATE_REBUILT_STALE_HOURS is
 # treated as untrustworthy (docker_update.sh likely didn't run, or didn't run recently) — deleted,
 # and every container in that tier restarts normally, same as if the file never existed.
-    DOCKER_UPDATE_REBUILT_DAILY_FILE="$DATA_DIR/docker_update_rebuilt_daily.list"
+    DOCKER_UPDATE_REBUILT_DAILY_FILE="${DB_DIR}/docker_update_rebuilt_daily.list"
-    DOCKER_UPDATE_REBUILT_WEEKLY_FILE="$DATA_DIR/docker_update_rebuilt_weekly.list"
+    DOCKER_UPDATE_REBUILT_WEEKLY_FILE="${DB_DIR}/docker_update_rebuilt_weekly.list"
     DOCKER_UPDATE_REBUILT_STALE_HOURS=12
 
 # Shares synced during the weekly maintenance window — defined per host in host*.conf.
@@ -486,7 +593,7 @@
 # monthly_maintenance.sh fires only when BOTH gates pass:
 #   1. Server uptime >= MONTHLY_UPTIME_THRESHOLD_DAYS days
 #   2. Last run was >= MONTHLY_RUN_INTERVAL_DAYS days ago (or never run)
-# Cron: 0 3 * * * (daily 3am check — script self-gates, calling daily is safe)
+# Cron: 0 0 15 * * (15th of the month, midnight — script self-gates, so a spare run is safe)
 # NOT in WATCHDOG_ORCHESTRATOR_SCRIPTS — has its own cron entry.
 # Add scripts that require a long-stable settled system — scrubs, extended drive tests.
 # State file on /boot/config — survives reboots (interval gate independent of uptime gate).
@@ -507,6 +614,7 @@
         "Monitors/zfs_memory_snapshot.sh"       # ZFS pool health + ARC + Docker memory snapshot
         "Monitors/smart_health.sh"              # drive SMART attributes — reallocated, pending, temp
         "Monitors/cert_monitor.sh"              # SSL certificate expiry for all configured domains
+        "Monitors/uptime_report.sh"             # domains down now, and any that were not 100% this week
         "Monitors/backup_verify.sh"             # rsync mirror integrity via independent MD5 checksums
         "Monitors/bandwidth_monitor.sh"         # weekly rsync transfer totals and per-share breakdown
         "Monitors/emby_session_report.sh"       # Emby usage — streams, users, library, transcode ratio
@@ -528,7 +636,7 @@
 #   Fine grained control — disable specific orchestrators while keeping others
 #   Use when: rebuilding secondary, testing, per-window bandwidth management
 #
-# Example — HOST2 data rebuild:
+# Example — rebuilding a partner's data:
 #   RSYNC_ENABLED=true                ← rsync works, individual scripts run fine
 #   CRITICAL_RSYNC_ENABLED=true       ← 30min auth stack sync still runs
 #   INTERMEDIATE_RSYNC_ENABLED=false  ← skip 4h arr/mid-day rsync during rebuild
@@ -536,15 +644,37 @@
 #   WEEKLY_RSYNC_ENABLED=true         ← Emby + Critical-Data still sync (NVMe)
 #   MONTHLY_RSYNC_ENABLED=true        ← monthly_maintenance.sh rsync section
 #   FALLBACK_RSYNC_ENABLED=true       ← handback writeback still works
+#   MEDIA_SEED_ENABLED=false          ← the onboard first-fill never starts
+#
+# MEDIA_SEED_ENABLED is the odd one out: it does not gate a schedule, it gates a single
+# multi-week transfer. Rsync/media_seed.sh pushes every DAILY_SYNC_SHARES entry to a newly
+# onboarded partner — on HOST1 that is ~28 TB against BW_LIMIT, so weeks. Turn it off when the
+# partner is going to be filled some other way (a physically moved disk, an existing library),
+# and onboard will finish without ever starting it. Turning it back on does not start anything
+# by itself; dispatch it from the Partnership tab or run the script.
+#
+#   NOTE ON THE DEFAULT: this one ships OFF, and unlike Tier 1 it is not armed by a successful
+#   onboard — Step 9c arms RSYNC_ENABLED, CONF_SYNC_ENABLED and ARR_SYNC_ENABLED, and
+#   deliberately not this. Seeding a partner is a decision about weeks of bandwidth and tens of
+#   terabytes of disk, and the node that has just been onboarded is exactly the node nobody has
+#   checked the free space on yet. Arm it when you have decided the partner should be filled by
+#   rsync rather than by hand. Turning it on is one switch; noticing it was on is not.
 #   → Run individual: bash Rsync/rsync.sh /mnt/user/Movies
 #   → When ready: INTERMEDIATE_RSYNC_ENABLED=true  DAILY_RSYNC_ENABLED=true
-    RSYNC_ENABLED=true                  # Tier 1 — global gate, overrides everything below
+#   NOTE ON THE DEFAULT: Tier 1 ships OFF. This template is what a brand-new node is seeded from
+#   (Manual-Deployment.md — "Rebuilding a Wiped Node"), and a node that has just been stood up has
+#   empty shares and an unverified partner. Defaulting the global gate ON meant a fresh install
+#   started transferring — with --delete in the mix — before anyone had looked at it. Tier 2 stays
+#   ON so the structure is intact the moment you do arm Tier 1; this mirrors HOST1's live posture.
+#   Changing this default cannot affect an existing node: conf_upgrade KEEPS values already set.
+    RSYNC_ENABLED=false                 # Tier 1 — global gate, overrides everything below
     CRITICAL_RSYNC_ENABLED=true         # Tier 2 — critical_sync_maintenance.sh rsync section
     INTERMEDIATE_RSYNC_ENABLED=true     # Tier 2 — intermediate_sync_maintenance.sh rsync section
     DAILY_RSYNC_ENABLED=true            # Tier 2 — daily_sync_maintenance.sh rsync section
     WEEKLY_RSYNC_ENABLED=true           # Tier 2 — weekly_sync_maintenance.sh rsync section
     MONTHLY_RSYNC_ENABLED=true          # Tier 2 — monthly_maintenance.sh rsync section
     FALLBACK_RSYNC_ENABLED=true         # Tier 2 — fallback.sh writeback jobs on handback
+    MEDIA_SEED_ENABLED=false            # Tier 2 — Rsync/media_seed.sh, the onboard first-fill
 
 # ━━━ Download Webhook ━━━
 # Immediate push to remote nodes on every Sonarr/Radarr/Lidarr Download event.
@@ -568,6 +698,13 @@
                                     # lowered from 23 on 2026-07-16 — daily maintenance jobs alone now
                                     # take ~4h; 19h cap leaves room for them to still run same-day
                                     # before the next 1am fire even if a share hits the cap
+    RSYNC_MIN_FREE_GB=50            # warn if any disk backing the remote share is under this
+                                    # (0 disables). Online is not writable: a full destination
+                                    # makes rsync stall rather than fail. Warns, never aborts.
+    CONF_PRUNE_STRIKE_LIMIT=3       # consecutive runs a container must be absent from docker
+                                    # before its name is removed from the conf lists. A
+                                    # rebuild makes a container vanish briefly; one miss
+                                    # must never be enough to edit conf.
     CRITICAL_CONTAINER_NAMES=()     # containers stopped on REMOTE before rsync — profiles override
     DELAYED_CONTAINERS=()           # containers needing delay before starting — profiles override
     CONTAINER_DELAY=5               # seconds before starting delayed containers
@@ -578,10 +715,11 @@
 # --partial      keep partial file on interrupted transfer so next run resumes, not re-transfers
 # --timeout      kill stalled transfers instead of hanging indefinitely
 # --numeric-ids  use UIDs/GIDs numerically — prevents ownership mismatches between servers
-# --delete intentionally omitted — arr_cleanup.sh enforces media truth post-rsync.
+# --delete intentionally omitted — the per-arr cleanups (lidarr/sonarr/radarr_cleanup.sh)
+# enforce media truth post-rsync.
 # Media shares use this default; rsync spreads files only, never removes them.
 # Note: --no-whole-file removed — redundant over SSH (delta transfer is already the default).
-# Note: arr_cleanup only catches true orphans under the union model — intentional removals
+# Note: the arr cleanups only catch true orphans under the union model — intentional removals
 #       require arr_sync.sh --blocklist-add first, then manual file deletion.
     DEFAULT_RSYNC_OPTS=(-av --info=progress2 --human-readable --bwlimit="$BW_LIMIT" --inplace --partial --timeout=60 --numeric-ids)
 
@@ -669,7 +807,25 @@
     declare -A PROFILE_EXCLUDE_DIRS=(
         [arrs_stack]="logs *.tmp"
         [critical-data]="logs *.tmp *.log nginx/temp nginx/cache __pycache__ notification.txt"
-        [critical-fallback]="logs *.tmp *.log nginx/temp nginx/cache __pycache__ notification.txt *.db-wal *.db-shm"
+        # appendonlydir and mariadb-bin.* are excluded for the same reason as *.db-wal: they are
+        # written continuously by a database this profile copies WHILE IT RUNS. The exclusion had
+        # SQLite's WAL and nothing for Redis or MariaDB, and both broke the mirror on 2026-08-17.
+        #
+        # The failure is not a torn file, it is a surviving pointer to a missing one. The manifest
+        # and the index are a few hundred bytes and land instantly; the data files they name are
+        # large and are still in flight. The mirror ends up holding a promise nothing can keep:
+        #   Redis   appendonly.aof.manifest → appendonly.aof.30.base.rdb   (absent) → exit 1
+        #   MariaDB mariadb-bin.index, 265  → mariadb-bin.000265           (absent) → Aborting
+        #
+        # Excluded rather than repaired: neither is needed for the engine to START. Redis rebuilds
+        # an empty AOF, MariaDB begins a new binlog sequence. On a fallback mirror that is the
+        # right trade — those files exist for persistence and replication, and a database that
+        # starts empty beats one that cannot start at all.
+        #
+        # NOT a guarantee that a live-copied InnoDB is consistent. It happened to survive both
+        # times. The honest fix is a DB-layer dump synced as a file; this stops the observed
+        # breakage without pretending to be that.
+        [critical-fallback]="logs *.tmp *.log nginx/temp nginx/cache __pycache__ notification.txt *.db-wal *.db-shm appendonlydir mariadb-bin.*"
         [important-data]="logs *.tmp"
         [emby]="logs transcodes cache crash*"
     )
@@ -704,7 +860,34 @@
 
     EXTERNAL_IP="8.8.8.8"
     FALLBACK_CHECK_INTERVAL=30      # seconds between fallback state checks
-    FALLBACK_HANDBACK_STRIKES=3     # consecutive healthy checks before initiating handback (3×30s = 90s)
+                                    # Cost measured 2026-08-22: ~15ms CPU and 784 bytes per cycle
+                                    # per host (~2 MB/day, half of it across the mesh). The 2s of
+                                    # wall time per cycle is ping waiting, not work.
+    FALLBACK_HANDBACK_STRIKES=4     # consecutive healthy checks before initiating handback
+                                    # 4×30s = 2min. Raising this costs nothing — it requires more
+                                    # of the checks that already happen, it does not add any.
+                                    # Any failed check resets the counter, so the window is a
+                                    # genuine "stayed up this long", not "was up this often".
+# Entry is deliberately unguarded — one false from ping_remote enters FALLBACK, with no strike
+# count, because every second of a real outage is downtime and DDNS TTL is racing it. The
+# confidence comes from the probe itself instead: see FALLBACK_PROBE_* below and _ping_retry()
+# in common.sh.
+    FALLBACK_PROBE_RETRIES=3        # reachability attempts before declaring a host down
+    FALLBACK_PROBE_RETRY_DELAY=2    # seconds between those attempts
+                                    # Success returns on the first try, so a healthy cycle costs
+                                    # nothing extra. Only a failure pays: 3×4s + 2×2s = 16s worst
+                                    # case, which must stay under FALLBACK_CHECK_INTERVAL.
+    FALLBACK_DRY_RUN=false          # run the daemon in PREVIEW mode, permanently
+                                    # Survives array start, unlike a --dry-run typed at a shell:
+                                    # array_started.sh launches with no arguments, so a hand-run
+                                    # preview is replaced by the LIVE daemon at the next boot.
+                                    # Decides nothing, starts nothing, moves no DNS, sends no
+                                    # notification — it only reports what it WOULD have done.
+                                    # Set true to observe real outages before arming for real.
+    FALLBACK_LOG_MAX_MB=5           # size cap for data/logs/fallback.log before one rotation
+                                    # Event-only output (the default, no --log) is a few lines
+                                    # per incident and holds years; --log fills this in about a
+                                    # fortnight and keeps the previous one as fallback.log.1
     FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
     FALLBACK_ENABLED=false          # set true once both servers are configured and paired
                                     # false = suppresses "not running" warnings in status scripts
@@ -821,7 +1004,7 @@
 # Restart loop protection — prevents watchdog from endlessly restarting a broken container
     WATCHDOG_CONTAINER_RESTART_LIMIT=3
     WATCHDOG_CONTAINER_RESTART_WINDOW=1     # rolling window in hours
-    WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
+    WATCHDOG_CONTAINER_RESTART_LOG="${DB_DIR}/container_restart_history.db"
 
 # Notification batching — one summary per cycle instead of one ping per event
     WATCHDOG_BATCH_NOTIFY=true
@@ -897,7 +1080,7 @@
 # Read by sunday_morning_coffee_report.sh for weekly peak/avg/warning summary.
     INOTIFY_WARN_PCT=80             # warn if inotify instances exceed this % of limit
     PHP_FPM_WARN_PCT=80             # warn if php-fpm workers exceed this % of max_children
-    TUNING_MONITOR_LOG="$DATA_DIR/system_tuning_history.db"
+    TUNING_MONITOR_LOG="${DB_DIR}/system_tuning_history.db"
     TUNING_LOG_RETENTION=30         # days before old entries are purged
 
 # ━━━ Reboot ━━━
@@ -919,6 +1102,19 @@
 # log lines per hour that have no diagnostic value. Filter removes them at source.
     FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
 
+# ━━━ PCIe AER Quiet ━━━
+# Master gate for pcie_aer_quiet.sh — removes dead PCIe hardware from the bus at
+# array start so it stops flooding syslog with correctable AER errors.
+#
+# Correctable means the link recovered, so the errors are harmless — but the kernel
+# logs every one. Removing the device ends it at the source. Unlike pci=noaer this
+# keeps uncorrectable AER reporting alive on every other device, which the AI repair
+# triage relies on to tell a real fault from this noise.
+#
+# Devices are listed per host in host*.conf as HOST*_PCIE_QUIET_DEVICES.
+# Off by default — turn on only after filling in that list for this host.
+    PCIE_QUIET_ENABLED=false
+
 # ━━━ PHP-FPM ━━━
 # Higher max_children allows more concurrent PHP requests to the unRAID WebGUI.
 # Default is very low — increasing it prevents WebGUI slowdowns under load.
@@ -1071,17 +1267,19 @@
     LIDARR_MAX_DELETE_GB=5          # require --i-know-what-im-doing if deletion exceeds this
     LIDARR_MIN_TRACKED_PCT=80       # abort if tracked count drops below this % of last run
                                     # protects against API returning partial data on a bad day
-    LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count"
+    LIDARR_TRACKED_COUNT_FILE="${DB_DIR}/lidarr_tracked.count"
+    LIDARR_ORPHAN_STRIKES_FILE="${DB_DIR}/lidarr_orphan_strikes.tsv"  # consecutive-classification counts, keyed by host path
+    LIDARR_ORPHAN_STRIKE_LIMIT=2    # consecutive runs a file must classify before it is removed
     LIDARR_IMPORT_SCAN_TIMEOUT=600  # seconds to wait for pre-flight import scan
 
 # Lidarr tracked-data cache — shared by lidarr_cleanup.sh, lidarr_duplicate_artist_cleanup.sh,
-# lidarr_missing_art.sh, lidarr_release_fixer.sh, and lidarr_cache_prefill.sh. See
+# lidarr_missing_art.sh, lidarr_release_fixer.sh, and arr_cache_prefill.sh. See
 # lidarr_get_tracked_data() in common.sh for the fresh/stale/rescan-active branching logic.
-    LIDARR_CACHE_FILE="$DATA_DIR/lidarr_tracked_cache.json"
+    LIDARR_CACHE_FILE="${ARR_CACHE_BACKUP_DIR}/lidarr_tracked_cache.json"
-    LIDARR_RESCAN_DURATION_DB="$DATA_DIR/lidarr_rescan_duration.db"
+    LIDARR_RESCAN_DURATION_DB="${DB_DIR}/lidarr_rescan_duration.db"
     LIDARR_CACHE_MAX_AGE_DAYS=1     # force a live refresh (or rescan-aware wait) past this age
     ARR_PREFILL_WAIT_MINUTES=10     # array-start prefill: how long to retry reaching each arr
-    LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
+    LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma" "wv" "ape" "aiff" "aif" "alac" "dsf" "dff" "mpc" "tta")
     LIDARR_PROTECTED_PATTERNS=(
         # Metadata
         "*.nfo" "*.tbn"
@@ -1100,7 +1298,7 @@
     LIDARR_ART_RETRIES=2                # download retry attempts per image
     LIDARR_ART_SLEEP_BETWEEN=0.2        # seconds between fanart.tv API calls
     LIDARR_ART_RECHECK_DAYS=30          # days before re-querying art that upstream didn't have
-    LIDARR_ART_MISS_CACHE="${DATA_DIR}/lidarr_art_miss_cache.tsv"   # negative cache — art upstream has never had
+    LIDARR_ART_MISS_CACHE="${DB_DIR}/lidarr_art_miss_cache.tsv"   # negative cache — art upstream has never had
 #   HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in host*.conf
 
 # Lidarr discovery settings (playback_aware_lidarr_discovery.sh)
@@ -1110,7 +1308,7 @@
     LIDARR_DISCOVERY_USER_CAP_PCT=35      # max % any single user can contribute to play score (prevents one listener dominating)
     LIDARR_DISCOVERY_MAX_ADDS=5           # max artists to add per run — quality over bulk
     LIDARR_DISCOVERY_REJECT_COOLDOWN=30   # days before re-evaluating a rejected artist
-    LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db"
+    LIDARR_DISCOVERY_HISTORY="${DB_DIR}/lidarr_discovery_history.db"
 
 # Sonarr discovery settings (playback_aware_sonarr_discovery.sh)
     SONARR_DISCOVERY_THRESHOLD=52         # score to accept candidate (0-100)
@@ -1122,7 +1320,7 @@
     SONARR_DISCOVERY_REJECT_COOLDOWN=60   # days before re-evaluating a rejected show
     SONARR_DISCOVERY_USER_EPISODE_CAP=8   # max episodes any one user contributes to seed volume score
     SONARR_DISCOVERY_MONITOR_MODE="all"    # Sonarr monitor mode on add: all | future | first | latest | none
-    SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db"
+    SONARR_DISCOVERY_HISTORY="${DB_DIR}/sonarr_discovery_history.db"
 
 # Radarr discovery shared settings
     RADARR_DISCOVERY_THRESHOLD=52         # score to accept candidate (0-100) — lower than Lidarr since diverse seeds rarely overlap
@@ -1133,7 +1331,7 @@
     RADARR_DISCOVERY_MIN_RATING=60        # min TMDB vote_average × 10 (60 = 6.0/10)
     RADARR_DISCOVERY_REJECT_COOLDOWN=60   # days before re-evaluating a rejected movie
     RADARR_DISCOVERY_SEED_LIBRARIES=("Movies")  # Emby libraries to draw seed movies from
-    RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db"
+    RADARR_DISCOVERY_HISTORY="${DB_DIR}/radarr_discovery_history.db"
 
 # Emby → arr sync library allowlists
 # Only these Emby library names will be considered by the sync tools.
@@ -1146,19 +1344,30 @@
     SONARR_MAX_DELETE_GB=10         # require --i-know-what-im-doing if deletion exceeds this
     SONARR_MIN_TRACKED_PCT=80       # abort if tracked count drops below this % of last run
                                     # protects against API returning partial data on a bad day
-    SONARR_TRACKED_COUNT_FILE="$DATA_DIR/sonarr_tracked.count"
+    SONARR_TRACKED_COUNT_FILE="${DB_DIR}/sonarr_tracked.count"
+    SONARR_ORPHAN_STRIKES_FILE="${DB_DIR}/sonarr_orphan_strikes.tsv"  # consecutive-classification counts, keyed by host path
+    SONARR_ORPHAN_STRIKE_LIMIT=2    # consecutive runs a file must classify before it is removed
     SONARR_IMPORT_SCAN_TIMEOUT=600  # seconds to wait for pre-flight import scan
     SONARR_MOVE_POLL_TIMEOUT=3600   # seconds to wait for a single async MoveSeries command to
                                     # reach "completed" — generous because a large series can sit
                                     # queued behind other moves already in progress, not just its
                                     # own copy time
-    CORRUPTION_SCAN_STATE_FILE="${DATA_DIR}/corruption_scan_state.tsv"  # clean-file skip-cache
+    CORRUPTION_SCAN_STATE_FILE="${DB_DIR}/corruption_scan_state.tsv"  # clean-file skip-cache
-    CORRUPTION_SCAN_STRIKES_FILE="${DATA_DIR}/corruption_scan_strikes.tsv"  # consecutive corrupt-detection counts, keyed by host path
+    CORRUPTION_SCAN_STRIKES_FILE="${DB_DIR}/corruption_scan_strikes.tsv"  # consecutive corrupt-detection counts, keyed by host path
     CORRUPTION_SCAN_STRIKE_LIMIT=2  # consecutive corrupt detections (across separate scan runs)
                                     # required before --remediate deletes+re-searches — guards
                                     # against a one-off ffprobe hiccup (mid-write file, NFS blip)
                                     # triggering an unnecessary delete. Resets to 0 the moment a
                                     # file probes clean again.
+    CORRUPTION_SCAN_MAX_CORRUPT_PCT=10   # abort the run, committing nothing, if this share of
+                                    # newly-scanned files reads as corrupt. A healthy library sits
+                                    # near zero; a high rate means the detector is wrong, not the
+                                    # library. Only counts ffprobe exit != 0.
+    CORRUPTION_SCAN_MAX_CONSECUTIVE=15   # abort after this many files in a row fail to probe
+                                    # cleanly — catches the ffprobe container dying mid-scan,
+                                    # which the pre-flight health check cannot see.
+    CORRUPTION_SCAN_GUARD_MIN_SCANNED=20  # below this many newly-scanned files the percentage
+                                    # ceiling is not applied — too small a sample to judge.
     SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov")
     SONARR_PROTECTED_PATTERNS=(
         # Subtitles
@@ -1184,7 +1393,13 @@
     RADARR_MAX_DELETE_GB=30         # require --i-know-what-im-doing if deletion exceeds this
     RADARR_MIN_TRACKED_PCT=80       # abort if tracked count drops below this % of last run
                                     # protects against API returning partial data on a bad day
-    RADARR_TRACKED_COUNT_FILE="$DATA_DIR/radarr_tracked.count"
+    RADARR_TRACKED_COUNT_FILE="${DB_DIR}/radarr_tracked.count"
+    RADARR_ORPHAN_STRIKES_FILE="${DB_DIR}/radarr_orphan_strikes.tsv"  # consecutive-classification counts, keyed by host path
+    RADARR_ORPHAN_STRIKE_LIMIT=2    # consecutive runs a file must classify for deletion before it is
+                                    # removed. Gate 6 already catches an API returning far too few
+                                    # tracked files; this catches the partial failure too small to trip
+                                    # that percentage — one root folder failing to enumerate makes its
+                                    # movies look orphaned, and a transient one will not repeat.
     RADARR_IMPORT_SCAN_TIMEOUT=600  # seconds to wait for pre-flight import scan
     RADARR_MOVE_POLL_TIMEOUT=3600   # seconds to wait for a single async MoveMovie command to
                                     # reach "completed" — mirrors SONARR_MOVE_POLL_TIMEOUT
@@ -1205,10 +1420,17 @@
 # Per-host dirs: HOST*_SONARR_DOWNLOAD_DIR / HOST*_RADARR_DOWNLOAD_DIR (+ _CONTAINER_DIR).
     DOWNLOAD_ORPHAN_CLEANER_ENABLED=true
     DOWNLOAD_ORPHAN_AGE=7               # days — entries younger than this may be mid-import, never touched
-    DOWNLOAD_ORPHAN_MIN_VIDEO_MB=50     # no video file above this = junk (par2 debris, samples, dead archives)
+    DOWNLOAD_ORPHAN_KEEP_MARKER=".vv-keep"  # a file by this name inside a download folder pins it —
-    DOWNLOAD_ORPHAN_MAX_DELETE_GB=100   # abort delete pass over this — a partial queue fetch would classify
+                                        # never classified, never deleted. For lossless rips the
-                                        # live downloads as orphans, and a huge total is that failure's symptom;
+                                        # library holds only as MP3, which REDUNDANT would sweep
-                                        # --i-know-what-im-doing overrides for known backlogs
+    DOWNLOAD_ORPHAN_MIN_VIDEO_MB=50     # no video file above this = junk (par2 debris, samples, dead archives) — Sonarr/Radarr
+    DOWNLOAD_ORPHAN_MIN_AUDIO_MB=2      # same test for Lidarr — a 50M floor would mark every album folder
+                                        # as junk, since single tracks rarely reach it
+    DOWNLOAD_ORPHAN_MAX_DELETE_GB=100   # per-run delete budget — a backlog above this drains
+                                        # safest-first (JUNK, REDUNDANT, then UNMATCHED) up to
+                                        # the budget and defers the rest to the next run.
+                                        # Live downloads are protected by DOWNLOAD_ORPHAN_AGE,
+                                        # not by this; --i-know-what-im-doing clears it in one pass
 
 # ━━━ Arr Content Classification (radarr/sonarr_classification_scan.sh) ━━━
 #
@@ -1261,7 +1483,7 @@
 
 # ━━━ Arr Failed/Stalled Recovery ━━━
 # Auto blocklist + re-search failed imports and stalled downloads.
-# Runs every 6 hours — schedule: 0 */6 * * *
+# Runs every 4 hours via INTERMEDIATE_MAINTENANCE_SCRIPTS — no cron entry of its own.
 #
 # Targets five problem types:
 #   importFailed   — downloaded but arr couldn't import
@@ -1338,7 +1560,7 @@
 
 # Daily statistics log — read by weekly_health_digest.sh for transcode summary.
     TRANSCODE_STATE_FILE="$STATE_DIR/transcode_state.db"
-    TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
+    TRANSCODE_DAILY_LOG="${DB_DIR}/transcode_daily.db"
     TRANSCODE_LOG_RETENTION=90      # days before old entries purged
 
     TRANSCODE_CHECK_EMBY=true
@@ -1351,6 +1573,23 @@
 # Defined per host in host*.conf — Emby container names and keys differ per server:
 #   HOST1_TRANSCODE_SERVERS / HOST2_TRANSCODE_SERVERS
 
+# ==============================================================================================
+# ── AUTH STACK ────────────────────────────────────────────────────────────────────────────────
+# ==============================================================================================
+
+# ━━━ Auth Stack ━━━
+# Which identity stack this mesh runs behind its protected hostnames. The Auth tab reads this to
+# decide which panels to draw and which endpoints to call — it is a routing switch, not a
+# migration. Changing it does not move users, groups or rules between stacks; stand the new one
+# up first, then point this at it.
+#
+# Mesh-wide rather than per host: auth is the one service the partnership treats as shared, owned
+# by the mesh owner and consumed by everyone else, so two nodes disagreeing about which stack is
+# in force would mean two different answers to "who is this person".
+# "authelia_lldap" — Authelia for access rules, lldap for users and groups. Fully implemented.
+# "authentik"      — single stack for both. NOT IMPLEMENTED YET; the tab says so instead of drawing panels that cannot work.
+    AUTH_STACK="authelia_lldap"
+
 # ==============================================================================================
 # ── MONITORS ──────────────────────────────────────────────────────────────────────────────────
 # ==============================================================================================
@@ -1364,6 +1603,62 @@
     CERT_CRIT_DAYS=7            # critical alert within this many days
     CERT_TIMEOUT=10             # seconds per domain before giving up
 
+# ── Certificate History ──
+# Tools/cert_history.sh records every certificate NPM holds — first seen, renewals, failures —
+# into DB_DIR/cert_history.json, which is what the Certs tab draws its per-domain cards from.
+# A domain found expired on this many consecutive passes stops being counted as working. It is
+# not deleted: the history of something that broke is the reason the file is kept.
+    CERT_HISTORY_STRIKES=5      # expired passes before a domain is retired from the active list
+
+# ── Uptime Probe ──
+# Tools/uptime_probe.sh checks every hostname NPM serves, once a minute, from outside the proxy.
+# NPM's access log only describes hosts somebody visited; this is what watches the ones nobody
+# does — which is where a quietly broken container hides.
+#
+# A 302 to the Authelia portal or a 401 counts as UP. The question is whether the server is there,
+# and an auth redirect is proof that it is; counting it as down would mark every protected host on
+# this mesh permanently offline.
+#
+# Probes carry the User-Agent Varaverk-Uptime/1.0, which npm_access_stats.sh excludes from its
+# request counts. Fifty thousand self-inflicted requests a day would otherwise bury real traffic.
+    UPTIME_PROBE_ENABLED=true   # master switch for the per-minute probe
+    UPTIME_PROBE_TIMEOUT=8      # seconds per domain before it counts as down
+    UPTIME_PROBE_LIST_TTL=300   # seconds to reuse the domain list from NPM before re-reading it
+
+# ── Auth Sweep ──
+# Tools/auth_sweep.sh asks the two questions the Auth tab answers about one host, about every host,
+# and files what it finds as findings on the AI tab. Reports only — it starts nothing and rewrites
+# nothing, because every remedy here (start a container, edit a rule, change a default policy) is a
+# decision rather than a correction.
+#
+#   proxy_down   a host below UPTIME_MIN that has been failing longer than DOWN_MIN. The time gate
+#                is what keeps a reboot from filing a finding for every hostname on the machine.
+#   access_open  an Authelia instance whose default policy lets through every hostname its rules do
+#                not decide. Filed once per instance, not once per hostname — they all have the
+#                same single fix, and one finding per name is twenty-two copies of one sentence.
+#
+# The access half reads whichever Authelia each proxy host actually points at, which is not always
+# the one HOST*_AUTHELIA_CONFIG names — this installation runs two.
+    AUTH_SWEEP_ENABLED=true      # master switch
+    AUTH_SWEEP_UPTIME_MIN=96     # 24h percentage below which a host becomes a candidate
+    AUTH_SWEEP_DOWN_MIN=120      # minutes it must have been failing before anything is filed
+    AUTH_SWEEP_ACCESS_CHECK=true # run the "is it actually protected" half at all
+
+# ── Cert Triage ──
+# Tools/cert_triage.sh reads certbot's own logs and names why renewals failed. cert_history.sh
+# counts failures by noticing an expiry in the past; this reads the reason.
+#
+# Counts runs, not lines: one log file is one certbot invocation, and one failure writes its
+# reason into the ACME response, the traceback and certbot's summary, so line counting reports it
+# three times and inflates whichever category is most verbose.
+#
+# The log directory is found from the NPM container's own mount. Set CERT_TRIAGE_LOG_DIR only if
+# that lookup cannot work. Both bounds exist because this is reachable from a page request and the
+# directory here is 639 MB across a thousand rotated files.
+    CERT_TRIAGE_FILES=40         # rotated logs to read, newest first by rotation suffix
+    CERT_TRIAGE_MAX_BYTES=262144 # bytes read from the end of each — a run's reason is always last
+    CERT_TRIAGE_LOG_DIR=""       # empty = find it from the NPM container
+
 # ━━━ Backup Verify ━━━
 # Verifies rsync mirror health by comparing random file checksums between servers.
 # Catches silent corruption or incomplete syncs that rsync itself wouldn't detect.
@@ -1371,6 +1666,10 @@
 #   HOST1_BACKUP_VERIFY_SHARES / HOST2_BACKUP_VERIFY_SHARES
     BACKUP_VERIFY_SAMPLE=10         # random files to check per share
     BACKUP_VERIFY_MIN_SIZE=1M       # minimum file size to include in sample
+    BACKUP_VERIFY_MD5_TIMEOUT_MAX=600  # ceiling for one remote checksum. The per-file budget
+                                    # scales with size (~50MB/s); this caps it. A fixed
+                                    # connect-sized timeout killed multi-GB checksums and
+                                    # the empty result was then reported as MISSING.
 
 # ━━━ SMART Health ━━━
 # Monitors drive SMART attributes — discovers all drives via /dev/sd* and /dev/nvme*.
@@ -1383,7 +1682,7 @@
 
 # ━━━ ZFS Memory Snapshot ━━━
 # Weekly ZFS pool health and memory diagnostic report — informational only.
-    ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log"
+    ZFS_REPORT_LOG="${LOG_ARCHIVE_DIR}/zfs-weekly-health.log"
     ZFS_REPORT_ARC_WARN_PCT=90      # warn if ARC using more than this % of its max
     ZFS_REPORT_ARC_FREE_WARN_GB=10  # warn if ARC headroom (max - current) drops below this GB
     ZFS_REPORT_AVAIL_WARN_GB=20     # warn if less than this GB available on ZFS pool
@@ -1394,15 +1693,15 @@
 # ━━━ Bandwidth Monitor ━━━
 # Called automatically by rsync.sh after each sync — one bounded write per run.
 # Tracks transfer size, duration and profile per sync for weekly summary reporting.
-    BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
+    BANDWIDTH_LOG="${DB_DIR}/bandwidth_history.db"
     BANDWIDTH_LOG_RETENTION=90      # days before old entries purged
     BANDWIDTH_WARN_GB=50            # flag syncs larger than this in weekly report
 
 # Stats files — written by cleanup and recovery scripts, read by coffee report.
 # All in DATA_DIR — array always running when these are written.
-    ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db"     # lidarr/sonarr/radarr orphan stats
+    ARR_CLEANUP_STATS="${DB_DIR}/arr_cleanup_stats.db"     # lidarr/sonarr/radarr orphan stats
-    ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db"   # blocklist + re-search stats
+    ARR_RECOVERY_STATS="${DB_DIR}/arr_recovery_stats.db"   # blocklist + re-search stats
-    ARR_RECOVERY_FAILURE_COUNTS="$DATA_DIR/arr_recovery_failure_counts.db"  # per-item chronic-failure tracking
+    ARR_RECOVERY_FAILURE_COUNTS="${DB_DIR}/arr_recovery_failure_counts.db"  # per-item chronic-failure tracking
 
 # ━━━ Health Digest ━━━
 # Aggregated system health summary — reads existing state files, no new writes.
@@ -1490,7 +1789,7 @@
 # ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
 # ==============================================================================================
 # Single-pass system health check — last line of defense before a crash.
-# Called by watchdog_orchestrator.sh every minute. NOT started by array_started.sh.
+# Called by watchdog_orchestrator.sh every 15 minutes. NOT started by array_started.sh.
 # Re-sources config at each orchestrator run — config changes take effect immediately.
 #
 # ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
@@ -1512,9 +1811,17 @@
     SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
     DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
     DOCKER_WATCHDOG_INTENTIONAL_FILE="$STATE_DIR/docker_intentional_stops.db"
+    WATCHDOG_MUTE_FILE="$STATE_DIR/watchdog_mutes.db"
     SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
     SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
 
+# ━━━ Timed Mutes ━━━
+# A mute silences one container for a bounded time and then expires on its own. It exists because
+# every other exemption here is permanent — SCAN_IGNORE is a conf edit, an intentional stop lasts
+# until cleared — so a temporary problem gets a permanent workaround that nobody revisits.
+# Capped so "temporary" is enforced rather than intended: a mute longer than this is refused.
+    WATCHDOG_MUTE_MAX_HOURS=8
+
 # ━━━ Strike and Reboot Loop Settings ━━━
 # Strike system: a check must fail this many consecutive cycles before action is taken.
 # Single spikes (one bad reading) are ignored — sustained problems trigger reboot.
@@ -1592,6 +1899,280 @@
     SYS_WATCHDOG_ABORT_ON_PARITY=false           # aborting parity better than crashing mid-check
     SYS_WATCHDOG_ABORT_ON_MOVER=false            # aborting move better than crashing mid-move
 
+# ==============================================================================================
+# ── AI / RAG ──────────────────────────────────────────────────────────────────────────────────
+# ━━━ Conf Backups ━━━
+# Every write through the plugin copies the conf aside first, to CONF_BACKUP_DIR with an
+# ISO-8601 stamp. The confs are gitignored, so that directory is the whole recovery path — there
+# is no history to revert to. A backup that cannot be taken cancels the write.
+# Oldest are pruned past this count, per conf file.
+    CONF_BACKUP_RETAIN=30
+
+# ==============================================================================================
+# Varaverk works exactly as well with AI off as with it on. Nothing below is required for any
+# script to function — every feature that can lean on AI has a complete non-AI path, and the
+# AI path is an enhancement layered on top. If Ollama is unreachable, callers proceed without it.
+#
+# AI_ENABLED is necessary but not sufficient. Every feature stays individually off until it has
+# earned it — narration for months before anything is allowed near a decision.
+
+# ━━━ AI Master Switch ━━━
+# Which node runs the model, the retrieval index and the bug store. Every other node borrows it
+# over the mesh, so a box without a GPU still gets the assistant — it just does not get the AI tab.
+# Defaults to host1 when unset or malformed: whoever builds the mesh is host1.
+    AI_OWNER_HOST="host1"
+
+# Fail-closed: anything other than the literal "true" means off.
+    AI_ENABLED=false
+    AI_CONNECT_TIMEOUT=5            # seconds — probe when resolving which node has Ollama
+    AI_REQUEST_TIMEOUT=240          # seconds — must clear a cold model load
+    AI_RESOLVE_CACHE_TTL=300        # seconds — don't re-probe the mesh every invocation
+    AI_MAX_RETRIES=1                # AI is enhancement; do not retry hard
+
+# ━━━ AI Retrieval Index ━━━
+# The RAG index over this repo's own headers and documentation. Regenerable in minutes and
+# gitignored — it is derived data, never a source of truth.
+#
+# Only git-tracked files are ever indexed. Configurations/, State_Files/ and data/ are
+# gitignored, which is what makes it structurally impossible for a credential to reach the
+# index: the files holding them were never in the repo. Do not "improve" this to a filesystem
+# walk — an embedded secret cannot be rotated out of a vector.
+    AI_INDEX_DB="${AI_DATA_DIR}/ai_index.db"
+    AI_INDEX_BATCH=32               # chunks per embed request
+# A pull is the only thing that changes tracked files on a server, so it is the only moment the
+# index can go stale — and staleness is invisible in the answers, which keep citing the old
+# text with full confidence. Incremental: unchanged files are skipped, a no-op run is ~66ms.
+# Also gated on AI_ENABLED, and ai_index.sh refuses on its own unless that is true.
+    AI_INDEX_ON_PULL=true           # re-index after a git pull that changed tracked files
+    AI_SEARCH_K=8                   # chunks retrieved per query
+    AI_SEARCH_PER_FILE=3            # cap per file so one document cannot fill the context
+
+# ━━━ AI Memory ━━━
+# A small operator-maintained file the assistant is given at the start of every conversation:
+# who you are, how this install is set up, decisions already made, things it should stop asking.
+#
+# Injected into the prompt, never indexed. It lives under DATA_DIR, which is gitignored — that
+# is deliberate and load-bearing. Indexing it would embed a file that changes constantly, and
+# vector similarity is the wrong way to retrieve "things I was told to remember"; it also keeps
+# personal notes out of a repository that gets pushed.
+#
+# The character cap is a context budget, not a style guide. At 16384 the retrieved passages,
+# the model's reasoning and the conversation history are already competing; memory takes its
+# share off the top of every single turn, so keep it short and factual.
+# Two slots, and which one a fact lands in decides how much authority it carries.
+#
+#   assisted  written by the operator. The prompt tells the model to prefer it over retrieved
+#             passages — a human asserting a fact about their own machine outranks a doc that
+#             may be stale. This is the file the AI tab edits.
+#   learned   proposed by the assistant, accepted by the operator. A hint only: retrieval
+#             overrules it, and it is trimmed first when the budget bites. Deliberately the
+#             weaker seat, because memory the model writes AND the prompt ranks above the source
+#             code would let one wrong conclusion restate itself forever.
+#
+# AI_MEMORY_FILE is the pre-split name and is still honoured: while the assisted file does not
+# exist, the legacy path is read instead, so an upgrade loses nothing.
+    AI_MEMORY_ASSISTED_FILE="${AI_DATA_DIR}/mem_assisted.md"
+    AI_MEMORY_LEARNED_FILE="${AI_DATA_DIR}/mem_learned.md"
+    AI_MEMORY_FILE="${AI_DATA_DIR}/ai_memory.md"     # legacy — read only if the assisted file is absent
+
+# Which profiles each slot is given. "*" is all of them. Narrowing is how a General Chat question
+# about bash syntax stops carrying this machine's PCIe topology and disk serials.
+    AI_MEMORY_ASSISTED_PROFILES="*"
+    AI_MEMORY_LEARNED_PROFILES="*"
+
+    AI_MEMORY_MAX_CHARS=4000        # ~1000 tokens — the budget for both slots together
+# A ceiling on the learned slot alone, well under the total. A store that grows on its own would
+# otherwise end up occupying the whole budget, and the operator's own memory is what would get
+# truncated away — the exact inversion of which one matters.
+    AI_MEMORY_LEARNED_MAX_CHARS=1200
+
+# ━━━ AI Learned Memory ━━━
+# Lets the assistant propose facts worth keeping. It never writes memory directly: a candidate is
+# filed in data/ai/mem_proposals/ and the operator accepts or dismisses it, exactly as findings
+# work. Accepted text lands in the learned slot, which the prompt ranks BELOW retrieval.
+#
+# The reason for the indirection: memory rides on every future prompt. A model allowed to write
+# its own — and told to prefer it over the passages — would restate a wrong conclusion forever,
+# reading its own claim back as evidence. A bad proposal must cost one dismissal, not that.
+#
+# Costs nothing while off: the instruction that asks for a candidate is only added to the prompt
+# when this is true, so a disabled feature is genuinely absent rather than merely ignored.
+    AI_MEMORY_LEARN_ENABLED=false
+
+# Writes accepted candidates without asking. Cannot outrank its parent — with proposing off this
+# does nothing. Leave it false until the proposals have proven good for a while.
+    AI_MEMORY_LEARN_AUTO_ACCEPT=false
+
+# ━━━ AI Stored Conversations ━━━
+# How many past conversations the AI tab and the Monitor tab's AI row keep. One JSON file per
+# conversation under DATA_DIR/ai_chats, saved automatically when a turn completes; the oldest
+# drop off once the count is exceeded.
+#
+# A cap rather than a retention age. These are read by picking one out of a short list, and a
+# list you have to scroll is a list you stop using — the useful window is the handful of things
+# you were recently working on, which is a count, not a date.
+#
+# Not indexed and never retrieved into a prompt on their own. A stored chat only re-enters the
+# model's context when the operator explicitly reopens it, and it is re-validated per message on
+# the way in, exactly as live history is.
+    AI_CHAT_HISTORY_MAX=10          # conversations kept — clamped to 1-50
+
+# ━━━ AI Token Accounting ━━━
+# One row per completed turn, appended by whichever path ran it — the WebGUI worker and the
+# ai_query.sh CLI both write here, so the totals are not silently the tab's alone. Format is
+# pipe-delimited to match the other data/*.db files:
+#
+#   date|time|host|profile|source|prompt_tokens|completion_tokens|tok_s
+#
+# The host column records where the turn ran, not where it is read. Each host keeps its own
+# data/ and nothing syncs it, so a host only ever sees its own rows — the column is there so
+# the file is already shaped right if the partner payload fetch is ever extended to carry it.
+# A column added later cannot be backfilled.
+#
+# Retention is by row count rather than age: pruning is considered only when the file passes a
+# size threshold, so an ordinary turn costs one stat() and an append.
+    AI_TOKEN_DB="${AI_DATA_DIR}/ai_token_history.db"
+    AI_TOKEN_RETAIN_ROWS=20000      # oldest rows dropped past this — years of ordinary use
+
+# AI/ai_token_sync.sh pulled each partner's ledger into a tmpfs cache so the fleet total was a
+# fleet total. The mesh shares one AI now: every turn generates on the owner and is recorded
+# there against the node that asked, so the owner's own ledger already holds the whole fleet and
+# a partner ledger is empty by construction. Left off the schedule rather than deleted — the
+# reader still merges any cache from before the change, which describes real turns.
+    AI_TOKEN_SYNC_ENABLED=false
+
+
+# ━━━ AI Feature Toggles ━━━
+# Tier 1 is narration — it cannot change a decision. Tier 2 adds context to a decision a script
+# already made. Tier 3 assists a human. Enable in that order, and give each one weeks.
+    AI_ASSIST_REPORTS=false         # tier 1 — digest / coffee report narration
+    AI_ASSIST_WATCHDOG=false        # tier 2 — file a finding when a watchdog counter passes its limit (needs AI_REPAIR_ENABLED)
+    AI_ASSIST_DISCOVERY=false       # tier 2 — discovery / classification judgement calls
+    AI_ASSIST_CLEANUP=false         # tier 2 — orphan and stuck-import triage. Describes the shape of a
+                                    # classification in the log; decides nothing. Off = identical deletions.
+    AI_ASSIST_TIMEOUT=45            # seconds any single assist may take. An assist that can stall a
+                                    # nightly cleanup is not an assist — it is silently skipped past this.
+    AI_ASSIST_ONBOARD=false         # tier 3 — onboarding / settings assistance
+
+# ━━━ AI Repair ━━━
+# Reads the log of a job that has finished, turns known error shapes into findings, and offers
+# or applies a repair. Two switches, because detecting and repairing are separate things to
+# trust.
+#
+# AI_REPAIR_ENABLED on its own reads, files findings and offers fixes, and writes nothing. Run
+# it there first — long enough to read what it finds and disagree with some of it.
+#
+# AI_REPAIR_AUTOFIX_ENABLED is what allows a value to be written unattended, and only ever a
+# value a probe has answered on. Never a toggle: whether something should be switched on is a
+# decision about intent, and a probe cannot prove intent the way it can prove a port answers.
+    AI_REPAIR_ENABLED=false
+    AI_REPAIR_AUTOFIX_ENABLED=false
+
+# ━━━ AI Repair Findings ━━━
+# Misconfigurations found in this installation, as opposed to defects in Varaverk — those go to
+# ai_bugs. A finding is open until the configuration is right, and closes itself when the probe
+# that proved the fault starts passing. Closed ones are kept for a while, because "this happened
+# before and here is what fixed it" is worth more than the disk. Open findings are never pruned:
+# an unresolved problem does not stop mattering because it is old.
+    AI_FINDING_RETAIN_DAYS=90
+# Seconds a single probe may take. Nothing is written to conf that has not answered a probe, so
+# this is the budget for proving a candidate — kept short because a sweep may try several, and
+# an address worth switching to answers quickly or is not worth switching to.
+    AI_PROBE_TIMEOUT=4
+# Tell the operator when a finding needs them. Only findings the sweep cannot resolve itself are
+# announced, one notification for all of them rather than one each, and each is announced once —
+# it stays quiet through every later pass until the fault changes or gets worse. Delivery is
+# whatever notify() is set up to use: NOTIFY_UNRAID and the host's Discord webhook.
+#
+# On by default, unlike the two switches above. Those gate reading and writing, which are things
+# to be trusted first. This gates telling someone, which is the reason for having looked.
+    AI_REPAIR_NOTIFY_ENABLED=true
+
+# ━━━ AI Repair: what it reads ━━━
+# Beyond Varaverk's own logs and the arrs' health endpoints.
+#
+# The system log catches what Varaverk cannot see about itself — a disk throwing I/O errors, a
+# filesystem going read-only, a PCIe link retraining every two minutes. Container restarts and
+# OOM kills are deliberately left to docker_watchdog, which already notifies on them.
+#
+# Container logs catch the opposite blind spot: docker_watchdog watches whether a container is
+# up and answering, which a container that has been unable to write to its database all day
+# passes perfectly. Only environment faults are matched — disk full, read-only filesystem,
+# corrupt database, fd limits, expired certificates — because those strings come from libc, the
+# kernel and SQLite and mean the same thing in all fifty containers. Anything app-specific
+# belongs in that app's own health endpoint.
+#
+# Both are bounded by time (since the last pass) and by a line cap, so a flood costs one pass.
+# Tools/ai_log_check.sh replays this host's real logs against the patterns — run it after
+# changing any of them.
+    AI_REPAIR_SYSLOG_ENABLED=true
+    AI_REPAIR_SYSLOG_MAX_LINES=4000
+    AI_REPAIR_CONTAINER_LOGS_ENABLED=true
+    AI_REPAIR_CONTAINER_LOG_LINES=400
+
+# ━━━ AI Web Search ━━━
+# General Chat only. Every other profile either reads this installation or changes it; chat holds
+# no capability at all, and that is exactly why searching is the one thing it may do — a read that
+# leaves the house is safe on the profile that cannot act on what it finds. The Varaverk assistant
+# deliberately does not get it: its contract is that answers come from this installation's own
+# documents, and a web result there is an answer that looks sourced and is not.
+#
+# Off by default, and not because it is dangerous. Searching sends the operator's question to
+# something outside this house, and that is their decision to make rather than a default to
+# inherit. Nothing turns it on.
+#
+# Asked for per turn as well — there is a checkbox on the AI tab, and a question is only searched
+# when it is ticked. A question about this machine hands off to the assistant before the search
+# would run, so it never reaches the internet even with the box ticked.
+#
+# Provider: degoog | searxng | brave | tavily
+#   degoog   self-hosted, no key — set HOST*_DEGOOG_URL. Aggregates several engines and returns
+#            them merged. The default, and the only one verified against a live service here
+#   searxng  self-hosted, no key, no third party — set HOST*_SEARXNG_URL, and enable format: [json]
+#            in its own settings.yml, which the default image ships with off
+#   brave    HOST*_WEB_SEARCH_API_KEY, free tier available
+#   tavily   HOST*_WEB_SEARCH_API_KEY
+    AI_WEB_SEARCH_ENABLED=false
+    AI_WEB_SEARCH_PROVIDER=degoog
+    AI_WEB_SEARCH_RESULTS=4
+    AI_WEB_SEARCH_TIMEOUT=6
+
+# ━━━ AI Chat Caution ━━━
+# General Chat cannot see this installation, and web search gives it confident-sounding material
+# about the outside world. The danger is the overlap: a question about THIS machine, answered
+# from a general page about Unraid, reads exactly like an answer about this machine.
+#
+# Detection is grammatical rather than topical — possessives, "this box", state questions, "what
+# happened last night" — because the topics are unbounded and the grammar is not. When it fires,
+# chat keeps its caution and defers instead of answering from the web.
+#
+# Pipe-separated extra phrases for what grammar misses: a nickname for the box, a share name,
+# anything that in practice means "mine". Matched as literal text, not as patterns.
+    AI_CHAT_MY_SYSTEM_PHRASES=""
+
+# ━━━ Bug Reports ━━━
+# A bug the assistant files is written here and goes nowhere until the owner sends it. Nothing is
+# ever transmitted automatically — the report is shown in full, read-only, and sending is a
+# second, separate press.
+#
+# Two destinations, and they are not a fallback chain. Fetching code from several mirrors is
+# harmless because they all serve the same thing; sending a report is not, because the
+# destinations are different people. Local ships OFF so an install that has configured nothing
+# reports upstream rather than silently into a tracker nobody reads.
+#
+# LOCAL ON  — reports go to your own Gitea (see HOSTN_BUG_REPORT_* in host*.conf) and stay there.
+#             They do NOT reach the Varaverk maintainer. Turn it on if you want your own backlog.
+# LOCAL OFF — reports open a prefilled GitHub issue you submit under your own account.
+    BUG_REPORT_LOCAL_ENABLED=false
+    BUG_REPORT_GITHUB_REPO="FailedProxy/Varaverk"
+
+# ━━━ AI Conf Write Access ━━━
+# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
+# never credentials, never a container name. An empty whitelist means no writes regardless of
+# the toggle.
+    AI_CONF_WRITE_ENABLED=false
+    AI_CONF_WRITE_KEYS=()
+
 # ==============================================================================================
 # ──────────────────────── End Of User Variables ───────────────────────────────────────────────
 # ==============================================================================================

Deployment/migrate_data_layout.sh

@@ -0,0 +1,310 @@
+#!/bin/bash
+# ==============================================================================================
+# ============================== DATA LAYOUT MIGRATION =========================================
+# ==============================================================================================
+#
+# PURPOSE
+# ─────────────────────────────────────────────────────────────────────────────
+# One-time move of everything Varaverk persists into a single rooted tree under DATA_DIR.
+#
+#   State_Files/              →  data/state/
+#   data/*.db|.count|.tsv|.list → data/db/
+#   data/ai_*                 →  data/ai/
+#   data/*_tracked_cache.json →  data/cache/arr/
+#   data/*.log                →  data/logs/
+#   SCRIPTS_DIR/.cache/vv/d/  →  data/cache/conf/
+#
+# ==============================================================================================
+# WHY THIS EXISTS SEPARATELY FROM conf_upgrade
+# ==============================================================================================
+#
+# conf_upgrade adds keys the template has and the installation does not; it never rewrites a
+# value the operator already has, which is exactly the behaviour you want from it and exactly
+# why it cannot perform this migration. The paths being moved are existing keys — STATE_DIR,
+# BANDWIDTH_LOG, AI_INDEX_DB and two dozen more — so their values would keep pointing at the old
+# layout forever while the new directory variables sat beside them unused.
+#
+# So this rewrites those values, then moves the files to match. Both halves, or neither: a conf
+# pointing at a directory the data is not in is worse than not having started.
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Two halves, in order: move the files, then rewrite the conf keys that point at them. Doing it
+# the other way round would leave every path variable naming a location nothing had reached yet,
+# and any script that ran in between would create the old layout again underneath the new one.
+#
+# Idempotent. A path already under DATA_DIR is left alone, so a re-run after a partial migration
+# finishes the job rather than moving things twice or failing on what is already done.
+#
+# One-time by intent, not by a marker file. There is no "already migrated" flag — the check is
+# whether each individual path is already where it belongs, which is also what makes an
+# interrupted run safe to repeat.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Existing keys are rewritten, which is why conf_upgrade cannot do this.
+#   conf_upgrade adds keys the template has and the installation does not, and never rewrites a
+#   value the operator already holds — correct for it, and exactly why it is the wrong tool here.
+#   STATE_DIR, BANDWIDTH_LOG, AI_INDEX_DB and two dozen more are existing keys whose values must
+#   change, or they would go on naming the old layout forever while the new directory variables
+#   sat beside them unused.
+#
+# Move, never copy-and-hope.
+#   The data being relocated is the only copy — statistics, histories, the AI index, arr caches.
+#   Everything is moved and the source is gone afterwards, so there is no second location that
+#   might still be written to by something that missed the change.
+#
+# The conf rewrite is the last thing, and the riskiest thing.
+#   Until it happens the installation still works from the old layout. That ordering means an
+#   abort partway through leaves a system that runs, rather than one whose paths point at
+#   nothing.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Idempotent
+#   Every step tests before acting. A second run reports "already migrated" and changes nothing,
+#   which matters because the natural instinct after a partial failure is to run it again.
+#
+# Moves, never copies-and-deletes
+#   mv within one filesystem is atomic per file, so a reader either sees the file at the old
+#   path or the new one — never a half-written copy at both. Nothing is deleted; if a file
+#   cannot be moved it is reported and left exactly where it is.
+#
+# Conf is backed up before it is rewritten
+#   master.conf.bak-<stamp>, next to the original, same convention conf_upgrade uses.
+#
+# Refuses to run while the orchestrators might be writing
+#   A watchdog that sourced conf before the rewrite and writes state after the move would put a
+#   file back at the old path. The window is seconds and the damage is one stale file, but the
+#   check costs nothing and the failure is silent otherwise.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# This script reads conf to find the old locations and rewrites conf to record the new ones. It
+# is the one script here whose purpose is to change these values rather than obey them.
+#
+#   Read to locate what moves
+#     STATE_DIR, BANDWIDTH_LOG, AI_INDEX_DB, AI_MEMORY_FILE, AI_TOKEN_DB, ARR_CLEANUP_STATS,
+#     ARR_SYNC_BLOCKLIST, CORRUPTION_SCAN_STATE_FILE, LIDARR_CACHE_FILE, ZFS_REPORT_LOG and the
+#     rest of the per-script path keys — roughly two dozen in total.
+#
+#   Written as the new roots
+#     DATA_DIR and the directories beneath it: DB_DIR, STATE_DIR, AI_DATA_DIR,
+#     CACHE_BACKUP_DIR, ARR_CACHE_BACKUP_DIR, CONF_CACHE_BACKUP_DIR, LOG_ARCHIVE_DIR.
+#
+#   Every rewritten value is expressed as ${DB_DIR}/… rather than an absolute path, so a later
+#   storage-mode migration moves them again by changing one variable.
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# migrate_data_layout.sh --dry-run     Show what would move. Changes nothing. Do this first.
+# migrate_data_layout.sh               Perform the migration.
+# migrate_data_layout.sh --force       Skip the running-orchestrator check.
+#
+# ==============================================================================================
+
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONF="$ROOT/Configurations/master.conf"
+
+DRY_RUN=false
+FORCE=false
+for a in "$@"; do
+    case "$a" in
+        --dry-run) DRY_RUN=true ;;
+        --force)   FORCE=true ;;
+        *) echo "Unknown argument: $a" >&2; exit 2 ;;
+    esac
+done
+
+[[ -f "$CONF" ]] || { echo "[FATAL] master.conf not found at $CONF" >&2; exit 1; }
+
+# Resolve the roots the same way load_config.sh will after this runs.
+SCRIPTS_DIR="$ROOT"
+DATA_DIR="$(grep -m1 -E '^\s*DATA_DIR=' "$CONF" | cut -d'"' -f2)"
+DATA_DIR="${DATA_DIR//\$\{SCRIPTS_DIR\}/$SCRIPTS_DIR}"
+DATA_DIR="${DATA_DIR:-$ROOT/data}"
+
+OLD_STATE="$ROOT/State_Files"
+OLD_CONFCACHE="$ROOT/.cache/vv/d"
+
+DB_DIR="$DATA_DIR/db"
+STATE_DIR="$DATA_DIR/state"
+AI_DATA_DIR="$DATA_DIR/ai"
+CACHE_BACKUP_DIR="$DATA_DIR/cache"
+ARR_CACHE_BACKUP_DIR="$CACHE_BACKUP_DIR/arr"
+CONF_CACHE_BACKUP_DIR="$CACHE_BACKUP_DIR/conf"
+LOG_ARCHIVE_DIR="$DATA_DIR/logs"
+
+moved=0; skipped=0; failed=0
+
+say()  { printf '  %s\n' "$*"; }
+step() { printf '\n━━━ %s ━━━\n' "$*"; }
+
+# ── Guard: orchestrators mid-run ──────────────────────────────────────────────
+# Scoped to THIS installation's path, not to the script names. pgrep is system-wide, and a box
+# running both a production checkout and a development clone will always have one of them busy —
+# matching on "Orchestrators/" alone made a dev migration abort because prod was mid-cycle, which
+# is a process that cannot touch this tree's data at all. The full path is what distinguishes
+# them, so that is what is matched.
+if [[ "$FORCE" == false && "$DRY_RUN" == false ]]; then
+    running=$(pgrep -fa "$ROOT/(Orchestrators|Watchdogs|Media|Rsync)/" 2>/dev/null \
+              | grep -v "$$" | grep -v migrate_data_layout || true)
+    if [[ -n "$running" ]]; then
+        echo "[ABORT] A job from this installation is running — it may rewrite state mid-move:" >&2
+        echo "$running" >&2
+        echo "Wait for it to finish, or re-run with --force if you are sure." >&2
+        exit 1
+    fi
+fi
+
+# ── Move one path ─────────────────────────────────────────────────────────────
+move() {
+    local src="$1" dstdir="$2" base
+    base="$(basename "$src")"
+    [[ -e "$src" ]] || return 0
+    if [[ -e "$dstdir/$base" ]]; then
+        say "skip   $base — already at ${dstdir#$DATA_DIR/}/"
+        ((skipped++)); return 0
+    fi
+    if [[ "$DRY_RUN" == true ]]; then
+        say "would  $base → ${dstdir#$DATA_DIR/}/"
+        ((moved++)); return 0
+    fi
+    mkdir -p "$dstdir" 2>/dev/null
+    if mv "$src" "$dstdir/$base" 2>/dev/null; then
+        say "moved  $base → ${dstdir#$DATA_DIR/}/"
+        ((moved++))
+    else
+        say "FAILED $base — left in place"
+        ((failed++))
+    fi
+}
+
+# ── 1. Rewrite the conf values conf_upgrade cannot ────────────────────────────
+step "Step 1: master.conf path values"
+if grep -q 'STATE_DIR="\${DATA_DIR}/state"' "$CONF"; then
+    say "already migrated — no conf changes needed"
+else
+    if [[ "$DRY_RUN" == false ]]; then
+        cp "$CONF" "${CONF}.bak-$(date +%Y%m%d%H%M%S)"
+        sed -i -E \
+            -e 's|^(\s*STATE_DIR=)".*"|\1"${DATA_DIR}/state"|' \
+            -e 's|^(\s*PERSISTENT_CONF_CACHE=)".*"|\1"${CACHE_BACKUP_DIR}/conf"|' \
+            "$CONF"
+        for v in ARR_SYNC_BLOCKLIST DOCKER_UPDATE_REBUILT_DAILY_FILE DOCKER_UPDATE_REBUILT_WEEKLY_FILE \
+                 WATCHDOG_CONTAINER_RESTART_LOG TUNING_MONITOR_LOG LIDARR_TRACKED_COUNT_FILE \
+                 LIDARR_RESCAN_DURATION_DB LIDARR_ART_MISS_CACHE LIDARR_DISCOVERY_HISTORY \
+                 SONARR_DISCOVERY_HISTORY RADARR_DISCOVERY_HISTORY SONARR_TRACKED_COUNT_FILE \
+                 CORRUPTION_SCAN_STATE_FILE CORRUPTION_SCAN_STRIKES_FILE RADARR_TRACKED_COUNT_FILE \
+                 TRANSCODE_DAILY_LOG BANDWIDTH_LOG ARR_CLEANUP_STATS ARR_RECOVERY_STATS \
+                 ARR_RECOVERY_FAILURE_COUNTS; do
+            sed -i -E "s|^(\s*${v}=\")\\\$\{?DATA_DIR\}?/|\1\${DB_DIR}/|" "$CONF"
+        done
+        for v in AI_INDEX_DB AI_MEMORY_FILE AI_TOKEN_DB; do
+            sed -i -E "s|^(\s*${v}=\")\\\$\{?DATA_DIR\}?/|\1\${AI_DATA_DIR}/|" "$CONF"
+        done
+        sed -i -E 's|^(\s*LIDARR_CACHE_FILE=")\$\{?DATA_DIR\}?/|\1${ARR_CACHE_BACKUP_DIR}/|' "$CONF"
+        sed -i -E 's|^(\s*ZFS_REPORT_LOG=")\$\{?DATA_DIR\}?/|\1${LOG_ARCHIVE_DIR}/|' "$CONF"
+        say "rewritten — backup kept beside it"
+    else
+        say "would rewrite STATE_DIR, PERSISTENT_CONF_CACHE and 25 file paths"
+    fi
+fi
+
+# ── 2. Build the tree ─────────────────────────────────────────────────────────
+step "Step 2: directory tree"
+for d in "$DB_DIR" "$STATE_DIR" "$AI_DATA_DIR" "$ARR_CACHE_BACKUP_DIR" "$LOG_ARCHIVE_DIR"; do
+    if [[ -d "$d" ]]; then say "exists ${d#$DATA_DIR/}"
+    elif [[ "$DRY_RUN" == true ]]; then say "would create ${d#$DATA_DIR/}"
+    else mkdir -p "$d" && say "created ${d#$DATA_DIR/}"
+    fi
+done
+# The conf cache carries partner credentials and keeps its restrictive mode.
+if [[ "$DRY_RUN" == false ]]; then
+    mkdir -p "$CONF_CACHE_BACKUP_DIR" && chmod 700 "$CONF_CACHE_BACKUP_DIR"
+    say "created cache/conf (0700)"
+fi
+
+# ── 3. State files ────────────────────────────────────────────────────────────
+step "Step 3: State_Files → data/state"
+if [[ -d "$OLD_STATE" ]]; then
+    shopt -s nullglob dotglob
+    for f in "$OLD_STATE"/*; do move "$f" "$STATE_DIR"; done
+    shopt -u nullglob dotglob
+    if [[ "$DRY_RUN" == false && -d "$OLD_STATE" ]]; then
+        rmdir "$OLD_STATE" 2>/dev/null && say "removed empty State_Files/" \
+            || say "State_Files/ not empty — left in place, inspect it"
+    fi
+else
+    say "no State_Files/ — nothing to do"
+fi
+
+# ── 4. Sort the data root ─────────────────────────────────────────────────────
+step "Step 4: sort data/ into subfolders"
+classify() {
+    local f="$1" base; base="$(basename "$f")"
+    case "$base" in
+        ai_*)                     move "$f" "$AI_DATA_DIR" ;;
+        *_tracked_cache.json)     move "$f" "$ARR_CACHE_BACKUP_DIR" ;;
+        *.log)                    move "$f" "$LOG_ARCHIVE_DIR" ;;
+        *.db|*.count|*.tsv|*.list|*.json) move "$f" "$DB_DIR" ;;
+        *)                        say "leave  $base — unclassified, left in data/" ;;
+    esac
+}
+
+# Two passes, sidecars first. A SQLite database is three files, and the -wal holds committed
+# transactions that have not been checkpointed into the .db yet. Move the .db first and any
+# process that opens it during the gap sees a database with no write-ahead log, creates a fresh
+# one at the old path, and everything still in the old -wal is lost when it is moved over the
+# top. Sidecars ahead of their base closes that ordering: the worst case becomes a database
+# opened without its log still sitting beside it, which SQLite handles.
+shopt -s nullglob
+for f in "$DATA_DIR"/*-wal "$DATA_DIR"/*-shm; do
+    [[ -d "$f" ]] && continue
+    classify "$f"
+done
+for f in "$DATA_DIR"/*; do
+    # Skip the destinations by name, not everything that happens to be a directory. ai_bugs/ and
+    # ai_chats/ are directories that belong under ai/, and an earlier version of this skipped
+    # every directory outright — which moved neither, silently, while the PHP layer had already
+    # been repointed at the new location. They were empty at the time; that was luck, not design.
+    case "$(basename "$f")" in
+        db|state|ai|cache|logs) continue ;;
+        *-wal|*-shm)            continue ;;
+    esac
+    classify "$f"
+done
+shopt -u nullglob
+
+# ── 5. Conf cache backup ──────────────────────────────────────────────────────
+step "Step 5: conf cache backup"
+if [[ -d "$OLD_CONFCACHE" ]]; then
+    shopt -s nullglob dotglob
+    for f in "$OLD_CONFCACHE"/*; do move "$f" "$CONF_CACHE_BACKUP_DIR"; done
+    shopt -u nullglob dotglob
+    [[ "$DRY_RUN" == false ]] && rmdir "$OLD_CONFCACHE" "$ROOT/.cache/vv" "$ROOT/.cache" 2>/dev/null
+else
+    say "no $OLD_CONFCACHE — nothing to do"
+fi
+
+# ── Summary ───────────────────────────────────────────────────────────────────
+printf '\n━━━━━ SUMMARY ━━━━━\n'
+printf '  %-10s %s\n' "moved:"   "$moved"
+printf '  %-10s %s\n' "skipped:" "$skipped"
+printf '  %-10s %s\n' "failed:"  "$failed"
+[[ "$DRY_RUN" == true ]] && printf '\n  DRY RUN — nothing was changed.\n'
+[[ "$failed" -gt 0 ]] && exit 1
+exit 0

Docker_Essentials/Manual-Docker_Essentials.md

@@ -94,20 +94,54 @@ HOST1_NETWORK_CONNECT_CONTAINERS=(
 
 ```bash
 # master.conf
-DAILY_CONTAINER_UPDATES=true     # enable/disable daily image pull
+DAILY_CONTAINER_UPDATES=true      # enable/disable the daily image pull
                                   # docker_daily_restart.sh still runs regardless
                                   # update and restart are independent
 
-WEEKLY_REMAINING_UPDATES=true    # enable/disable weekly remainder pull + prune
+WEEKLY_CONTAINER_UPDATES=true     # enable/disable the weekly image pull
-                                  # to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS
+                                  # docker_weekly_restart.sh still runs regardless
+
+MONTHLY_REMAINING_UPDATES=true    # enable/disable the monthly remainder pull
+                                  # to disable: set false, or remove
+                                  # "docker_update.sh --remainder" from
+                                  # MONTHLY_MAINTENANCE_SCRIPTS
 ```
 
-`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list
+There is one update script, `docker_update.sh`, with three modes. Each reuses the restart
-used by `docker_daily_restart.sh`. No second list to maintain.
+list it pairs with, so there is no second list to maintain:
 
-`docker_update_remaining.sh` derives its target list automatically:
+| Mode | Targets | Runs |
-all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`.
+|------|---------|------|
-Everything gets updated at least once per week with no explicit configuration.
+| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` |
+| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` |
+| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` |
+
+Remainder mode needs no configuration at all. It takes every **running** container and
+subtracts:
+
+```
+  DAILY_RESTART_CONTAINERS                          already updated daily
+  WEEKLY_RESTART_CONTAINERS                         already updated weekly
+  PROFILE_CRITICAL_CONTAINER_NAMES[emby]            updated inline by the weekly sync window
+  PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]   updated inline by the weekly sync window
+  FALLBACK_<REMOTE>_TIER1..4                        owned by the remote's update cycle
+```
+
+Stopped containers are never targeted in any mode — pulling for a stopped container adds
+nothing, and it was most likely stopped deliberately.
+
+The fallback exclusion is a correctness rule, not an optimisation. This server only runs
+those containers during a fallback; the remote owns their version. If remainder updated them
+independently and a handback then occurred, the remote's older image could meet data written
+by the newer one.
+
+**Ordering matters.** The update always runs *before* its matching restart so the restart
+lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh`
+rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at
+creation time and would never pick up the new digest) and records it in
+`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers
+rather than restarting them a second time — and discards the file as stale if it is older
+than `DOCKER_UPDATE_REBUILT_STALE_HOURS`.
 
 ---
 
@@ -142,12 +176,22 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
 # Shared — applies to both servers
 
 # ── Container updates ──────────────────────────────────────────────────
-DAILY_CONTAINER_UPDATES=true
+DAILY_CONTAINER_UPDATES=true         # daily pull (DAILY_RESTART_CONTAINERS)
-WEEKLY_REMAINING_UPDATES=true
+WEEKLY_CONTAINER_UPDATES=true        # weekly pull (WEEKLY_RESTART_CONTAINERS)
+MONTHLY_REMAINING_UPDATES=true       # monthly pull (everything else)
+
+# Handoff between update and restart — written by docker_update.sh,
+# read by the matching restart script so it skips containers already
+# rebuilt onto a new image this run.
+DOCKER_UPDATE_REBUILT_DAILY_FILE
+DOCKER_UPDATE_REBUILT_WEEKLY_FILE
+DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all
 
 # ── Retry behaviour (shared by restart scripts) ────────────────────────
 RETRY_COUNT=3                        # retry attempts before marking failed
 SLEEP=5                              # seconds between retry attempts
+CONTAINER_DELAY                      # seconds between a dependency and its dependents
+RESTART_VERIFY_WAIT=3                # settle time before verifying a restart stuck
 
 # Watchdog thresholds (CPU, memory, HTTP, restart loop):
 # → see Watchdogs/Manual-Watchdogs.md
@@ -211,7 +255,24 @@ All scripts support these standard flags:
 summary block: identity, duration, counts, and a status line. Per-container detail only
 appears with `--log`. Warnings and errors are always visible regardless of `--log`.
 
-### `docker_update.sh --remainder`
+### Script-specific flags
-Switches to remainder mode — updates all running containers not in the managed daily/weekly
+
-lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers
+| Script | Flag | What it does |
-that haven't been updated recently.
+|--------|------|-------------|
+| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. |
+| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. |
+| `media_cleaner.sh` *(Media/)* | `<anime\|media>` | Required positional profile — there is no default. |
+
+`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by
+`daily_sync_maintenance.sh` before the daily restart.
+
+### Exit codes
+
+| Code | Meaning |
+|------|---------|
+| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) |
+| `1` | One or more containers failed — the summary names them |
+
+`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason
+to abort the restart that follows, which simply proceeds on the existing image. The failure
+is reported in the summary.

Docker_Essentials/README-Docker_Essentials.md

@@ -178,12 +178,28 @@ windows so any downtime from restarts is absorbed by the window that's already h
 
 ---
 
-### 🔄  Image Currency — `docker_update.sh` + `docker_update_remaining.sh`
+### 🔄  Image Currency — `docker_update.sh`
 
-Keeps all container images current without manual intervention. Daily updates for the
+One script, three modes — there is no separate remainder script.
-auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
+
-remainder pass for everything else — derives the target list automatically from `docker ps`
+| Mode | Targets | Runs |
-minus what was already updated, so there is no second list to maintain.
+|------|---------|------|
+| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, **before** `docker_daily_restart.sh` |
+| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, **before** `docker_weekly_restart.sh` |
+| `--remainder` | Everything running that is in neither list | Monthly, via `monthly_maintenance.sh` |
+
+The update always runs *before* its matching restart, so the restart lands on the freshly
+pulled image. Reversing that order would restart onto the old image and leave the new one
+sitting unused until the next window.
+
+Each mode reuses the restart list it pairs with rather than maintaining its own — add a
+container to `DAILY_RESTART_CONTAINERS` once and it gets both the restart and the image pull.
+Remainder mode needs no list at all: it derives its targets from `docker ps` minus the daily
+list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers.
+
+Fallback containers are deliberately excluded from remainder mode. This server only runs them
+during a fallback; the remote owns their version. Updating them here would risk the remote's
+older image meeting data written by a newer one after a handback.
 
 ---
 
@@ -229,7 +245,7 @@ full coordination model between all four watchdogs.
 |--------|------|-------------|
 | `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
 | `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
-| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly remainder pass |
+| `docker_update.sh` | Container image updates — three modes (default / `--weekly` / `--remainder`) | Daily and weekly before each restart; monthly for the remainder |
 | `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
 | `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
 | `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
@@ -257,8 +273,13 @@ Daily maintenance window (1am):
 
 Weekly maintenance window (2:30am Sunday):
     weekly_sync_maintenance.sh
-        ├── docker_weekly_restart.sh ──── restart less-critical services
+        ├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS)
-        └── docker_update.sh --remainder ─ update containers not in managed lists
+        └── docker_weekly_restart.sh ──── restart onto the fresh image
+
+Monthly maintenance window:
+    monthly_maintenance.sh
+        ├── docker_update.sh --remainder  update everything not in the managed lists
+        └── Tools/docker_prune_images.sh --all
 
 Critical maintenance window (every 30min):
     critical_sync_maintenance.sh
@@ -266,5 +287,39 @@ Critical maintenance window (every 30min):
 
 Array stopping:
     array_stopping.sh
-        └── docker_container_stop.sh ──── ordered graceful shutdown
+        └── docker_container_stop.sh ──── ordered graceful shutdown, verified per container
 ```
+
+---
+
+## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+Every script here talks to the Docker daemon, so they share the same protections. Each
+script's own header documents its full set; these are the ones worth knowing folder-wide.
+
+**Daemon health is checked, not assumed.** A hung daemon returns an empty container list,
+which is indistinguishable from "no containers running". Without the check,
+`docker_container_stop.sh` would report a clean shutdown that never happened, and
+`docker_update.sh --remainder` would report "nothing to update" while doing nothing.
+
+**Every docker call is timeout-wrapped.** A wedged daemon cannot stall a maintenance window
+or hold a lock open. The one deliberate exception is `docker pull` — a large image
+legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer.
+
+**State is respected.** Running containers get restarted; stopped ones stay stopped. A
+stopped container was almost certainly stopped on purpose, and none of these scripts has the
+authority to overrule that.
+
+**Restarts are verified, not assumed.** After each restart the container is re-checked once
+it has had time to settle. A container that starts and immediately crashes is recorded as a
+failure and notified — a restart that did not stick is never reported as success.
+
+**Dependency ordering is shared with the watchdog.** Restarts follow
+`HOST*_WATCHDOG_DEPENDENCIES`, with `CONTAINER_DELAY` between a dependency and its dependents,
+so a dependent is never brought up while what it needs is still initialising.
+
+**Locks prevent overlap.** Long windows can outlast their interval;
+`downloaders_reset.sh` uses wait-mode because it runs every 30 minutes and the previous pass
+may still be finishing.
+
+---

Docker_Essentials/docker_container_stop.sh

@@ -68,6 +68,15 @@
 # Docker Presence Check
 #   Verifies docker binary exists before execution.
 #
+# Docker Daemon Check
+#   Verifies the daemon is responsive before enumerating containers. A hung
+#   daemon returns an empty container list, which would otherwise be read as
+#   "nothing to stop" and pass a shutdown that never happened.
+#
+# Host Detection
+#   detect_hosts() identifies which server is running the script and sets
+#   MY_ID / LOCAL_SERVER_NAME for logging and notifications.
+#
 # Lock Acquisition
 #   acquire_lock prevents overlapping runs (e.g. array_stopping firing twice).
 #
@@ -140,10 +149,20 @@ fi
 
 detect_hosts
 
-[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no containers will be stopped"
-
 DOCKER_TIMEOUT=30
 DOCKER_STOP_TIMEOUT=30    # grace period for SIGTERM before docker sends SIGKILL internally
+
+# A hung daemon makes docker ps return nothing — indistinguishable from "no containers
+# running", which would silently report a clean shutdown that never happened.
+if ! timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
+    error "Docker daemon not responding — cannot verify container shutdown"
+    notify "Container stop aborted on $(hostname) ($MY_ID) — Docker daemon not responding" \
+        "Docker Container Stop" "warning"
+    exit 1
+fi
+
+[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no containers will be stopped"
+
 _RETRY_COUNT="${RETRY_COUNT:-3}"
 
 log "$ICON_GEAR Config: retries=$_RETRY_COUNT sleep=${SLEEP:-5}s grace=${DOCKER_STOP_TIMEOUT}s cmd-timeout=${DOCKER_TIMEOUT}s"
@@ -152,7 +171,7 @@ log "$ICON_GEAR Config: retries=$_RETRY_COUNT sleep=${SLEEP:-5}s grace=${DOCKER_
 # ━━━ Status ━━━
 # ==============================================================================================
 if [[ "$SHOW_STATUS" == true ]]; then
-    mapfile -t RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
+    mapfile -t RUNNING < <(timeout "$DOCKER_TIMEOUT" docker ps --format '{{.Names}}' 2>/dev/null | sort)
     echo ""
     echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
     echo "$ICON_HOST       Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
@@ -166,7 +185,7 @@ fi
 # ==============================================================================================
 # ━━━ Stop Containers ━━━
 # ==============================================================================================
-mapfile -t RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
+mapfile -t RUNNING < <(timeout "$DOCKER_TIMEOUT" docker ps --format '{{.Names}}' 2>/dev/null | sort)
 
 echo ""
 echo "━━━ $ICON_CONTAINERS Docker Container Stop — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
@@ -187,7 +206,7 @@ FAILED=()
 for container in "${RUNNING[@]}"; do
     [[ -z "$container" ]] && continue
     c_start=$(date +%s)
-    c_image=$(docker inspect --format '{{.Config.Image}}' "$container" 2>/dev/null || echo "unknown")
+    c_image=$(timeout "$DOCKER_TIMEOUT" docker inspect --format '{{.Config.Image}}' "$container" 2>/dev/null || echo "unknown")
     log "━━━ $ICON_CONTAINERS $container ($c_image) ━━━"
 
     if [[ "$DRY_RUN" == true ]]; then

Docker_Essentials/docker_daily_restart.sh

@@ -14,6 +14,33 @@
 # so there is no second list to maintain.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Containers are processed one at a time in dependency-safe order:
+#
+#   1. Build restart order
+#        → build_restart_order() sorts DAILY_RESTART_CONTAINERS by WATCHDOG_DEPENDENCIES
+#
+#   2. Skip anything docker_update.sh already rebuilt this run
+#        → a rebuild onto a new image already restarted it moments ago
+#
+#   3. Inspect container state
+#        missing  → skip, not an error
+#        stopped  → skip, stopped state is respected
+#        running  → restart
+#
+#   4. Restart with retry
+#        → retry_docker wraps each attempt in a timeout, up to RETRY_COUNT
+#
+#   5. Verify it stayed running
+#        → verify_running() settles for RESTART_VERIFY_WAIT then checks State.Running
+#        → a container that crashes immediately is marked failed and notified
+#
+#   6. Prune dangling images
+#        → restarts swap onto new images, leaving the old ones dangling
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -37,6 +64,27 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
+# Root Enforcement
+#   Docker operations require root privileges.
+#
+# Docker Presence Check
+#   Verifies the docker binary exists before execution. Notifies on absence —
+#   a missing binary during the maintenance window is worth knowing about.
+#
+# Docker Daemon Check
+#   Verifies the daemon is responsive before any restart work. Every container
+#   would otherwise fail its inspect and be logged as an unknown-status failure,
+#   burying one daemon fault under a list of bogus per-container errors.
+#
+# Host Detection
+#   detect_hosts() identifies which server is running the script and aliases
+#   HOST*_DAILY_RESTART_CONTAINERS and HOST*_WATCHDOG_DEPENDENCIES to the
+#   correct host's values.
+#
+# Empty List Guard
+#   Exits cleanly with a pointer to the relevant conf key if
+#   DAILY_RESTART_CONTAINERS is unconfigured for this host.
+#
 # Dependency Ordering
 #   Containers restart in dependency-safe order using HOST*_WATCHDOG_DEPENDENCIES.
 #   CONTAINER_DELAY seconds between dependency restart and dependent restart gives
@@ -53,6 +101,11 @@
 #   cannot cause this script to hang indefinitely. Timed-out commands retry
 #   per RETRY_COUNT before marking as failed.
 #
+# Stale Rebuild-List Guard
+#   The rebuilt-container list written by docker_update.sh is discarded if older
+#   than DOCKER_UPDATE_REBUILT_STALE_HOURS. A stale file would otherwise suppress
+#   real restarts based on an update run that never happened today.
+#
 # Lock Acquisition
 #   acquire_lock() prevents concurrent execution if a previous run is still active.
 #
@@ -134,6 +187,14 @@ fi
 # detect_hosts() sets MY_ID and aliases HOST*_DAILY_RESTART_CONTAINERS → DAILY_RESTART_CONTAINERS
 detect_hosts
 
+# Without this, a hung daemon fails every container's inspect individually and the summary
+# reports a list of unknown-status failures instead of the one fault that caused them.
+if ! timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
+    error "Docker daemon not responding — skipping daily restart"
+    notify "Daily restart skipped on $(hostname) — Docker daemon not responding" "Docker Daily Restart" "warning"
+    exit 1
+fi
+
 if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
     warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
     warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
@@ -213,7 +274,7 @@ LAST_RESTARTED=""
 for container in "${ORDERED_RESTART[@]}"; do
     [[ -z "$container" ]] && continue
     c_start=$(date +%s)
-    c_image=$(docker inspect --format '{{.Config.Image}}' "$container" 2>/dev/null || echo "unknown")
+    c_image=$(timeout "$DOCKER_TIMEOUT" docker inspect --format '{{.Config.Image}}' "$container" 2>/dev/null || echo "unknown")
     log "━━━ $ICON_CONTAINERS $container ($c_image) ━━━"
 
     if ! timeout "$DOCKER_TIMEOUT" docker inspect "$container" &>/dev/null; then
@@ -242,7 +303,6 @@ for container in "${ORDERED_RESTART[@]}"; do
                 RESTARTED+=("$container")
             else
                 if retry_docker docker restart "$container"; then
-                    [[ "${RESTART_VERIFY_WAIT:-3}" -gt 0 ]] && sleep "${RESTART_VERIFY_WAIT:-3}"
                     if verify_running "$container"; then
                         echo "$ICON_STARTED $container restarted and running in $(format_duration $(( $(date +%s) - c_start ))) ✅"
                         RESTARTED+=("$container")
@@ -283,7 +343,7 @@ if [[ "$DRY_RUN" == true ]]; then
     warn "DRY RUN — would prune dangling images"
     PRUNED_SUMMARY="(dry run)"
 else
-    PRUNED_OUTPUT=$(docker image prune -f 2>&1)
+    PRUNED_OUTPUT=$(timeout "$DOCKER_TIMEOUT" docker image prune -f 2>&1)
     [[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/  /'
     PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
 fi

Docker_Essentials/docker_network_connect.sh

@@ -51,6 +51,12 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
+# Root Enforcement
+#   Docker network operations require root privileges.
+#
+# Docker Presence Check
+#   Verifies the docker binary exists before any network operations.
+#
 # Lock Acquisition
 #   Prevents concurrent execution via acquire_lock(). Safe to call from
 #   array start hooks or manually without risk of overlap.
@@ -71,8 +77,10 @@
 #   Warns and exits cleanly if NETWORK_CONNECT_NETWORKS or
 #   NETWORK_CONNECT_CONTAINERS are unconfigured.
 #
-# Command Validation
+# Missing Container Tolerance
-#   Validates unRAID notify script before use.
+#   A configured container that does not exist yet warns and is skipped rather
+#   than failing the run. This script executes early at array start, before
+#   every container has necessarily been created.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -130,13 +138,12 @@ fi
 # detect_hosts() sets MY_ID and aliases HOST*_NETWORK_CONNECT_* arrays
 detect_hosts
 
-# Validate unRAID notify script — used for network creation alerts
-
 # Docker daemon check — network operations are useless if daemon is hung
 DOCKER_TIMEOUT=15
 if ! timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
     error "Docker daemon not responding — cannot manage networks"
-    notify "docker_network_connect failed on $(hostname) — Docker daemon not responding"         "Network Connect" "warning"
+    notify "docker_network_connect failed on $(hostname) — Docker daemon not responding" \
+        "Network Connect" "warning"
     exit 1
 fi
 
@@ -291,7 +298,8 @@ if [[ "$DRY_RUN" == true ]]; then
     warn "DRY RUN — no changes made"
 elif [[ ${#FAILED[@]} -gt 0 ]]; then
     echo "$ICON_ERROR Status: SOME OPERATIONS FAILED"
-    notify "Docker network connect failed on $(hostname) — ${FAILED[*]}"         "Network Connect" "warning"
+    notify "Docker network connect failed on $(hostname) — ${FAILED[*]}" \
+        "Network Connect" "warning"
 elif [[ ${#NETWORKS_CREATED[@]} -gt 0 ]]; then
     warn "Networks recreated — ${NETWORKS_CREATED[*]} — unRAID update likely wiped them"
 else

Docker_Essentials/docker_update.sh

@@ -74,6 +74,28 @@
 # Root Enforcement
 #   Docker operations require root privileges.
 #
+# Docker Presence Check
+#   Verifies the docker binary exists before execution.
+#
+# Docker Daemon Check
+#   Verifies the daemon is responsive before container discovery. Remainder mode
+#   derives its entire target list from docker ps — against a hung daemon that
+#   returns empty and the run silently reports "no containers to update".
+#
+# Timeout Protection
+#   Inspect, discovery and image-query commands are wrapped in a timeout so a
+#   hung daemon cannot stall the maintenance window. docker pull is deliberately
+#   NOT wrapped — a large image legitimately takes longer than any sane timeout,
+#   and killing it mid-layer wastes the transfer.
+#
+# Empty List Guards
+#   Each mode exits cleanly with a pointer to the relevant conf key when its
+#   container list is unconfigured for this host.
+#
+# Rebuild Failure Fallback
+#   A container that fails to rebuild is excluded from the rebuilt-list handoff
+#   file, so the follow-up restart script still gives it a normal restart pass.
+#
 # DAILY_CONTAINER_UPDATES / WEEKLY_CONTAINER_UPDATES Toggles
 #   Each mode exits cleanly when disabled. Restart scripts run regardless —
 #   update and restart are independent operations.
@@ -192,6 +214,14 @@ fi
 
 detect_hosts
 
+# Remainder mode builds its whole target list from docker ps — a hung daemon returns
+# empty and the run would report "no containers to update" instead of failing.
+if ! timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
+    error "Docker daemon not responding — skipping image updates"
+    notify "Docker update skipped on $(hostname) — Docker daemon not responding" "Docker Update" "warning"
+    exit 1
+fi
+
 # ==============================================================================================
 # ━━━ Container Discovery ━━━
 # ==============================================================================================
@@ -234,7 +264,7 @@ if [[ "$REMAINDER_MODE" == true ]]; then
     done
     unset _tier _tier_var _tier_arr _c
 
-    mapfile -t _all_running < <(docker ps --format '{{.Names}}' | sort)
+    mapfile -t _all_running < <(timeout "$DOCKER_TIMEOUT" docker ps --format '{{.Names}}' | sort)
     TARGET_CONTAINERS=()
     for _c in "${_all_running[@]}"; do
         [[ -z "${_exclude[$_c]+x}" ]] && TARGET_CONTAINERS+=("$_c")
@@ -328,13 +358,13 @@ for container in "${TARGET_CONTAINERS[@]}"; do
     [[ -z "$container" ]] && continue
     log "━━━ $ICON_CONTAINERS $container ━━━"
 
-    if ! docker inspect "$container" &>/dev/null; then
+    if ! timeout "$DOCKER_TIMEOUT" docker inspect "$container" &>/dev/null; then
         warn "$container — not found, skipping"
         SKIPPED+=("$container")
         continue
     fi
 
-    IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
+    IMAGE=$(timeout "$DOCKER_TIMEOUT" docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
     if [[ -z "$IMAGE" ]]; then
         warn "$container — could not determine image, skipping"
         SKIPPED+=("$container")
@@ -352,8 +382,8 @@ for container in "${TARGET_CONTAINERS[@]}"; do
     # Capture the image ID the container is currently running on, and the
     # image ID :latest points to before the pull. After pulling, we rebuild if
     # either a new digest landed OR the container is behind what :latest is now.
-    CONTAINER_IMAGE_ID=$(docker inspect "$container" --format='{{.Image}}' 2>/dev/null || echo "")
+    CONTAINER_IMAGE_ID=$(timeout "$DOCKER_TIMEOUT" docker inspect "$container" --format='{{.Image}}' 2>/dev/null || echo "")
-    OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
+    OLD_ID=$(timeout "$DOCKER_TIMEOUT" docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
 
     log "$ICON_SYNC  Pulling $IMAGE..."
     if [[ "$ENABLE_LOGGING" == "true" ]]; then
@@ -363,7 +393,7 @@ for container in "${TARGET_CONTAINERS[@]}"; do
         docker pull "$IMAGE" >/dev/null 2>&1
         _pull_rc=$?
     fi
-    NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
+    NEW_ID=$(timeout "$DOCKER_TIMEOUT" docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
 
     if [[ $_pull_rc -eq 0 ]]; then
         _pull_new=$(  [[ -n "$OLD_ID" && "$OLD_ID" != "$NEW_ID" ]] && echo true || echo false)
@@ -436,9 +466,9 @@ if [[ "$DRY_RUN" == true ]]; then
     PRUNED_SUMMARY="(dry run)"
 else
     for _old_id in "${OLD_IMAGE_IDS[@]}"; do
-        docker rmi "$_old_id" >/dev/null 2>&1 || true
+        timeout "$DOCKER_TIMEOUT" docker rmi "$_old_id" >/dev/null 2>&1 || true
     done
-    PRUNED_OUTPUT=$(docker image prune -f 2>&1)
+    PRUNED_OUTPUT=$(timeout "$DOCKER_TIMEOUT" docker image prune -f 2>&1)
     [[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/  /'
     PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
 fi

Docker_Essentials/docker_weekly_restart.sh

@@ -16,6 +16,33 @@
 # stopped → leave, missing → skip. Container state is always respected.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Identical to docker_daily_restart.sh, against WEEKLY_RESTART_CONTAINERS:
+#
+#   1. Build restart order
+#        → build_restart_order() sorts WEEKLY_RESTART_CONTAINERS by WATCHDOG_DEPENDENCIES
+#
+#   2. Skip anything docker_update.sh --weekly already rebuilt this run
+#        → a rebuild onto a new image already restarted it moments ago
+#
+#   3. Inspect container state
+#        missing  → skip, not an error
+#        stopped  → skip, stopped state is respected
+#        running  → restart
+#
+#   4. Restart with retry
+#        → retry_docker wraps each attempt in a timeout, up to RETRY_COUNT
+#
+#   5. Verify it stayed running
+#        → verify_running() settles for RESTART_VERIFY_WAIT then checks State.Running
+#        → a container that crashes immediately is marked failed and notified
+#
+#   6. Prune dangling images
+#        → restarts swap onto new images, leaving the old ones dangling
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -39,6 +66,26 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
+# Root Enforcement
+#   Docker operations require root privileges.
+#
+# Docker Presence Check
+#   Verifies the docker binary exists before execution. Notifies on absence.
+#
+# Docker Daemon Check
+#   Verifies the daemon is responsive before any restart work. Every container
+#   would otherwise fail its inspect and be logged as an unknown-status failure,
+#   burying one daemon fault under a list of bogus per-container errors.
+#
+# Host Detection
+#   detect_hosts() identifies which server is running the script and aliases
+#   HOST*_WEEKLY_RESTART_CONTAINERS and HOST*_WATCHDOG_DEPENDENCIES to the
+#   correct host's values.
+#
+# Empty List Guard
+#   Exits cleanly with a pointer to the relevant conf key if
+#   WEEKLY_RESTART_CONTAINERS is unconfigured for this host.
+#
 # Dependency Ordering
 #   Containers restart in dependency-safe order using HOST*_WATCHDOG_DEPENDENCIES.
 #   CONTAINER_DELAY seconds between dependency restart and dependent restart.
@@ -51,10 +98,10 @@
 #   All docker commands wrapped in a 30 second timeout. A hung Docker daemon
 #   cannot cause this script to hang indefinitely.
 #
-# Host Detection
+# Stale Rebuild-List Guard
-#   detect_hosts() identifies which server is running the script and aliases
+#   The rebuilt-container list written by docker_update.sh --weekly is discarded
-#   HOST*_WEEKLY_RESTART_CONTAINERS and HOST*_WATCHDOG_DEPENDENCIES to the
+#   if older than DOCKER_UPDATE_REBUILT_STALE_HOURS. A stale file would otherwise
-#   correct host's values.
+#   suppress real restarts based on an update run that never happened this week.
 #
 # Lock Acquisition
 #   acquire_lock() prevents concurrent execution.
@@ -84,6 +131,11 @@
 #   CONTAINER_DELAY
 #       Seconds to wait after restarting a dependency before starting its dependents
 #
+#   RESTART_VERIFY_WAIT
+#       Seconds verify_running() waits after docker restart before checking the
+#       container is running. Gives the process time to initialise before the
+#       state is sampled. (default: 3)
+#
 #   DOCKER_UPDATE_REBUILT_WEEKLY_FILE / DOCKER_UPDATE_REBUILT_STALE_HOURS
 #       List of containers docker_update.sh --weekly already rebuilt onto a new image
 #       this run — read here so they're not restarted a second time. Discarded as
@@ -132,6 +184,14 @@ fi
 # detect_hosts() sets MY_ID and aliases HOST*_WEEKLY_RESTART_CONTAINERS → WEEKLY_RESTART_CONTAINERS
 detect_hosts
 
+# Without this, a hung daemon fails every container's inspect individually and the summary
+# reports a list of unknown-status failures instead of the one fault that caused them.
+if ! timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
+    error "Docker daemon not responding — skipping weekly restart"
+    notify "Weekly restart skipped on $(hostname) — Docker daemon not responding" "Docker Weekly Restart" "warning"
+    exit 1
+fi
+
 if [[ ${#WEEKLY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
     warn "WEEKLY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
     warn "Check HOST*_WEEKLY_RESTART_CONTAINERS in host*.conf"
@@ -211,7 +271,7 @@ LAST_RESTARTED=""
 for container in "${ORDERED_RESTART[@]}"; do
     [[ -z "$container" ]] && continue
     c_start=$(date +%s)
-    c_image=$(docker inspect --format '{{.Config.Image}}' "$container" 2>/dev/null || echo "unknown")
+    c_image=$(timeout "$DOCKER_TIMEOUT" docker inspect --format '{{.Config.Image}}' "$container" 2>/dev/null || echo "unknown")
     log "━━━ $ICON_CONTAINERS $container ($c_image) ━━━"
 
     if ! timeout "$DOCKER_TIMEOUT" docker inspect "$container" &>/dev/null; then
@@ -281,7 +341,7 @@ if [[ "$DRY_RUN" == true ]]; then
     warn "DRY RUN — would prune dangling images"
     PRUNED_SUMMARY="(dry run)"
 else
-    PRUNED_OUTPUT=$(docker image prune -f 2>&1)
+    PRUNED_OUTPUT=$(timeout "$DOCKER_TIMEOUT" docker image prune -f 2>&1)
     [[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/  /'
     PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
 fi

Docker_Essentials/downloaders_reset.sh

@@ -61,6 +61,15 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
+# Root Enforcement
+#   Failed-import purging deletes directories owned by container users.
+#
+# Dependency Check
+#   Verifies curl and jq exist before any API work. jq backs the slskd connection
+#   probe — without it the probe returns false forever, the script burns its full
+#   60 second reconnect wait, then skips every slskd section as "disconnected".
+#   A missing dependency is reported as itself rather than as a phantom outage.
+#
 # Active Transfer Protection
 #   slskd: skips users with InProgress or Queued transfers before any removal.
 #   SABnzbd: age threshold enforced before deletion.
@@ -70,6 +79,19 @@
 #   Each section validates its downloader URL before API calls. Missing or
 #   unreachable downloaders skip without affecting other sections.
 #
+# No Downloaders Guard
+#   Exits cleanly when none of SLSKD_URL, SABNZBD_URL or QBIT_URL are set for
+#   this host — nothing configured is not an error.
+#
+# Timeout Protection
+#   Every curl carries --max-time. An unresponsive downloader cannot stall the
+#   30 minute maintenance cycle or overlap the next run.
+#
+# Deletion Scope Limit
+#   qBittorrent removals pass deleteFiles=false — the torrent record is dropped
+#   but files on disk are left for the arrs to manage. This script never deletes
+#   media.
+#
 # Host Detection
 #   detect_hosts() identifies which server is running the script and aliases
 #   all HOST*_SLSKD_*, HOST*_SABNZBD_*, and HOST*_QBIT_* vars to the correct
@@ -140,6 +162,17 @@ fi
 # Lock first — wait mode since this runs every 30min and previous may still be finishing
 acquire_lock "wait"
 
+# jq backs the slskd connection probe. Missing, the probe never returns true and slskd
+# looks permanently disconnected — a 60s wait followed by silently skipped sections.
+for _dep in curl jq; do
+    if ! command -v "$_dep" &>/dev/null; then
+        error "$_dep not found — required for downloader API calls"
+        notify "Downloaders reset failed on $(hostname) — $_dep not installed" "Downloaders Reset" "warning"
+        exit 1
+    fi
+done
+unset _dep
+
 # detect_hosts() sets MY_ID and aliases all HOST*_SLSKD_*, HOST*_SABNZBD_*, HOST*_QBIT_* vars
 detect_hosts
 
@@ -250,7 +283,7 @@ if [[ -n "$SLSKD_URL" ]] && [[ -n "$SLSKD_API_KEY" ]] && [[ "$SLSKD_CONNECTED" =
         IDS=$(echo "$SEARCHES" | tr '{' '\n' | \
             grep '"isComplete":true' | grep '"searchText":' | \
             grep -o '"id":"[^"]*"' | sed 's/"id":"//;s/"//')
-        COUNT=$(echo "$IDS" | grep -c . 2>/dev/null || echo 0)
+        COUNT=$(echo "$IDS" | grep -c . 2>/dev/null || true)
         COUNT="${COUNT//[^0-9]/}"; COUNT="${COUNT:-0}"
 
         if [[ "$COUNT" -eq 0 ]]; then
@@ -306,7 +339,7 @@ if [[ -n "$SLSKD_URL" ]] && [[ -n "$SLSKD_API_KEY" ]] && [[ "$SLSKD_CONNECTED" =
         if [[ -z "$USERNAMES" ]]; then
             success "No transfer records found ✅"
         else
-            USER_COUNT=$(echo "$USERNAMES" | grep -c . 2>/dev/null || echo 0)
+            USER_COUNT=$(echo "$USERNAMES" | grep -c . 2>/dev/null || true)
             log "Found $USER_COUNT user(s) with transfer records"
             SUCCESS=0; SKIPPED=0; FAIL=0
             while IFS= read -r USER; do
@@ -383,7 +416,7 @@ if [[ -n "$SLSKD_FAILED_IMPORTS_DIR" ]]; then
     else
         OLD_IMPORTS=$(find "$SLSKD_FAILED_IMPORTS_DIR" \
             -mindepth 1 -maxdepth 1 -mtime +"${DOWNLOADER_RETENTION_DAYS}")
-        IMPORT_COUNT=$(echo "$OLD_IMPORTS" | grep -c . 2>/dev/null || echo 0)
+        IMPORT_COUNT=$(echo "$OLD_IMPORTS" | grep -c . 2>/dev/null || true)
         IMPORT_COUNT="${IMPORT_COUNT//[^0-9]/}"; IMPORT_COUNT="${IMPORT_COUNT:-0}"
 
         if [[ "$IMPORT_COUNT" -eq 0 ]]; then
@@ -426,7 +459,7 @@ if [[ -n "$SABNZBD_URL" ]] && [[ -n "$SABNZBD_API_KEY" ]]; then
         if [[ -z "$COMPLETED_IDS" ]]; then
             success "No completed history found ✅"
         else
-            HIST_TOTAL=$(echo "$COMPLETED_IDS" | grep -c . 2>/dev/null || echo 0)
+            HIST_TOTAL=$(echo "$COMPLETED_IDS" | grep -c . 2>/dev/null || true)
             log "Found $HIST_TOTAL completed history entries"
             DELETED=0; SKIPPED=0
             while IFS= read -r NZO_ID; do
@@ -474,7 +507,7 @@ if [[ -n "$SABNZBD_URL" ]] && [[ -n "$SABNZBD_API_KEY" ]]; then
         if [[ -z "$FAILED_IDS" ]]; then
             success "No failed history found ✅"
         else
-            FAILED_TOTAL=$(echo "$FAILED_IDS" | grep -c . 2>/dev/null || echo 0)
+            FAILED_TOTAL=$(echo "$FAILED_IDS" | grep -c . 2>/dev/null || true)
             log "Found $FAILED_TOTAL failed history entries"
             DELETED=0; SKIPPED=0
             while IFS= read -r NZO_ID; do
@@ -524,7 +557,7 @@ if [[ -n "$SABNZBD_URL" ]] && [[ -n "$SABNZBD_API_KEY" ]]; then
         if [[ -z "$STALLED_IDS" ]]; then
             success "No stalled queue items found ✅"
         else
-            QUEUE_TOTAL=$(echo "$STALLED_IDS" | grep -c . 2>/dev/null || echo 0)
+            QUEUE_TOTAL=$(echo "$STALLED_IDS" | grep -c . 2>/dev/null || true)
             log "Found $QUEUE_TOTAL queue item(s) — checking status"
             DELETED=0; SKIPPED=0
             while IFS= read -r NZO_ID; do
@@ -585,7 +618,7 @@ if [[ -n "$QBIT_URL" ]] && [[ -n "$QBIT_USERNAME" ]]; then
             -H "Cookie: $QBIT_COOKIE" 2>/dev/null)
 
         NOW=$(date +%s)
-        TORRENT_TOTAL=$(echo "$TORRENTS" | tr '}' '\n' | grep -c '"hash"' 2>/dev/null || echo 0)
+        TORRENT_TOTAL=$(echo "$TORRENTS" | tr '}' '\n' | grep -c '"hash"' 2>/dev/null || true)
         log "Found $TORRENT_TOTAL torrent(s) — applying age/ratio filter"
         DELETED=0; SKIPPED=0
 

Fallback/Manual-Fallback.md

@@ -105,18 +105,36 @@ internet connectivity. Most hosts leave this empty.
 ---
 
 ```
-FALLBACK_HOST*_COVERS_HOST*_TIER1=(...)
+FALLBACK_HOST*_TIER1=(...)
-FALLBACK_HOST*_COVERS_HOST*_TIER2=(...)
+FALLBACK_HOST*_TIER2=(...)
-FALLBACK_HOST*_COVERS_HOST*_TIER3=(...)
+FALLBACK_HOST*_TIER3=(...)
-FALLBACK_HOST*_COVERS_HOST*_TIER4=(...)
+FALLBACK_HOST*_TIER4=(...)
 ```
-Containers this host starts for the remote host when the remote is down. TIER1 starts
-immediately. TIER2–4 activate after the corresponding delay thresholds.
 
-Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}`
+**Each host declares its OWN services, in its OWN conf.** `FALLBACK_HOST2_TIER1` lives in
+`host2.conf` and lists HOST2's vital containers — it is not a list HOST1 maintains.
 
-The DDNS container for the remote's domain must be the first entry in TIER1 — DNS
+At runtime `fallback.sh` reads its **partner's** list:
-coverage before anything else.
+
+```bash
+get_tier_containers() {
+    local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}"     # note: REMOTE_ID, not MY_ID
+```
+
+So HOST1, covering HOST2, reads `FALLBACK_HOST2_TIER1` — a variable defined in `host2.conf`
+and delivered to HOST1 through the partner conf cache (`conf_sync.sh`), because sparse
+checkout means HOST1 never pulls `host2.conf` from git.
+
+This is why the naming is what it is. The alternative — each host keeping a copy of its
+partner's container list — would need editing on both machines every time either one changed
+a service, and the two copies would silently diverge. Declaring once, on the host that owns
+the services, means a host is always the authority on what covering it requires.
+
+TIER1 starts immediately. TIER2–4 activate after their delay thresholds
+(`HOST*_TIER2_DELAY` and friends, also in that host's own conf).
+
+The DDNS container for that host's domain must be the **first entry in TIER1** — DNS coverage
+before anything else.
 
 ---
 
@@ -181,15 +199,15 @@ FALLBACK_TEST_HANDBACK_WAIT=300
 
 ---
 
-### host2.conf — HOST2 covering HOST1
+### host1.conf — HOST1's own services (started by HOST2 when HOST1 is down)
 
 ```bash
-HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
+HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
 
-FALLBACK_HOST2_STOP_ON_NO_NET=()
+FALLBACK_HOST1_STOP_ON_NO_NET=()
 
 # Tier 1 — immediate (vital services + Live TV)
-FALLBACK_HOST2_COVERS_HOST1_TIER1=(
+FALLBACK_HOST1_TIER1=(
     "Gmer4Lfe.com-DDNS"         # ALWAYS FIRST — DNS coverage before anything else
     "Emby"                       # media server — people are watching
     "NginxProxyManager"          # reverse proxy — all external access routes through this
@@ -205,7 +223,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER1=(
 )
 
 # Tier 2 — after 4 hours (shared productivity services)
-FALLBACK_HOST2_COVERS_HOST1_TIER2=(
+FALLBACK_HOST1_TIER2=(
     "Postgres-NextCloud"         # must start before NextCloud
     "NextCloud"
     "PostgreSQL-Immich"          # must start before Immich
@@ -214,7 +232,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER2=(
 )
 
 # Tier 3 — after 12 hours (secondary services)
-FALLBACK_HOST2_COVERS_HOST1_TIER3=(
+FALLBACK_HOST1_TIER3=(
     "Organizrv2-Gmer4Lfe"
     "AdGuard-Home"
     "UptimeKuma"
@@ -223,7 +241,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER3=(
 )
 
 # Tier 4 — after 24 hours (arrs + downloaders)
-FALLBACK_HOST2_COVERS_HOST1_TIER4=(
+FALLBACK_HOST1_TIER4=(
     "Sonarr-Gmer4Lfe"
     "Radarr-Gmer4Lfe"
     "Lidarr-Gmer4Lfe"
@@ -259,22 +277,22 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
 
 ---
 
-### host1.conf — HOST1 covering HOST2
+### host2.conf — HOST2's own services (started by HOST1 when HOST2 is down)
 
 ```bash
-HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
+HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
 
-FALLBACK_HOST1_STOP_ON_NO_NET=()
+FALLBACK_HOST2_STOP_ON_NO_NET=()
 
 # Tier 1 — immediate (HOST2's vital services)
-FALLBACK_HOST1_COVERS_HOST2_TIER1=(
+FALLBACK_HOST2_TIER1=(
     "Gmer4Lfe.us-DDNS"          # ALWAYS FIRST
     # HOST2's Tier 1 services — fill per HOST2's stack
 )
 
-FALLBACK_HOST1_COVERS_HOST2_TIER2=(...)
+FALLBACK_HOST2_TIER2=(...)
-FALLBACK_HOST1_COVERS_HOST2_TIER3=(...)
+FALLBACK_HOST2_TIER3=(...)
-FALLBACK_HOST1_COVERS_HOST2_TIER4=(...)
+FALLBACK_HOST2_TIER4=(...)
 
 # Tier delays for HOST2 outage
 HOST2_TIER2_DELAY=240
@@ -296,7 +314,7 @@ FALLBACK_HOST2_WRITEBACK_TIER1=(
 Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata)
 
 > In a shell where load_config.sh is not sourced, use the full path:
-> `/boot/config/plugins/varaverk/State_Files/fallback_state.db` (internal storage mode)
+> `/boot/config/plugins/varaverk/data/state/fallback_state.db` (internal storage mode)
 
 ```
 state=NORMAL            # NORMAL | FALLBACK | NO_INTERNET | DARK
@@ -492,7 +510,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
 
 1. Create the container on the covering server (stopped), with volume mounts pointing at
    the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
-2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf
+2. Add the container name to `FALLBACK_<THAT-HOST>_TIER*` in **that host's own** conf
    in the appropriate tier position (dependency ordering — databases before apps)
 3. Verify: `fallback.sh --status` shows the container in the expected tier list
 4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid

Fallback/README-Fallback.md

@@ -115,7 +115,7 @@ Both servers run `fallback.sh` independently as a continuous background process.
 makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:
 
 ```bash
-ping REMOTE_TAILSCALE_IP     # is the other server reachable?
+ping "$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")"   # is the other server reachable?
 ping EXTERNAL_IP             # do I have internet? (default: 8.8.8.8)
 ```
 
@@ -220,8 +220,8 @@ determines which server is local and which is remote at runtime, then selects th
 container arrays and tier delays from config via MY_ID.
 
 ```
-HOST2 covers HOST1:   FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf)
+HOST2 covers HOST1:   FALLBACK_HOST1_TIER* (in host2.conf)
-HOST1 covers HOST2:   FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf)
+HOST1 covers HOST2:   FALLBACK_HOST2_TIER* (in host1.conf)
 ```
 
 Both servers run identical scripts. MY_ID selects the correct arrays. No hostname

Fallback/coverage_deploy.sh

@@ -0,0 +1,351 @@
+#!/bin/bash
+# ══════════════════════════════════════════════════════════════════════════════════════════════
+# PURPOSE
+#   Put the containers this host has marked for fallback coverage onto the partner, so that the
+#   partner can actually start them during an outage — and take them off again on request.
+#
+# OPERATIONAL MODEL
+#   fallback.sh covers a host by running `docker start <name>` on the partner. It never creates
+#   anything. So a name in FALLBACK_<me>_TIER* is a promise that only holds if the partner already
+#   has that container built. Measured 2026-08-23: all 12 of HOST1's covered containers were absent
+#   from HOST2, meaning every tier would have failed on the first real outage while the UI showed
+#   coverage as configured. This script is what closes that gap.
+#
+#   Push and remove are separate, deliberate actions, never a side effect of saving the tier list.
+#   Editing coverage is a cheap config write; deploying a dozen containers onto another machine is
+#   not, and the two should not share a button.
+#
+# DESIGN PRINCIPLES
+#   Deployed, then verified STOPPED.
+#       A container built here and left running on the partner would be a second live instance of
+#       NextCloud, Gitea or PostgreSQL_Immich against the same data while this host is healthy.
+#       That is the danger_rsync_live_database_appdata failure with worse odds. Every deploy is
+#       followed by a stop and a re-inspect, and a container that will not stay stopped is an
+#       error, not a warning.
+#
+#   Remove takes the container AND its appdata.
+#       Operator decision 2026-08-23: the button is explicit, so a removal should leave nothing
+#       behind to reason about later. The risk it accepts is narrow and worth naming — if the
+#       partner ever covered for us, ITS appdata is the newer copy and is what a handback rsyncs
+#       home. The NORMAL-state gate below closes the live-failover window; what it cannot see is
+#       a handback that partially failed and then returned to NORMAL, so the UI says so before
+#       asking.
+#
+#       Two guards on the deletion itself: only paths under /mnt/*/appdata* are ever touched, and
+#       a bind of the appdata ROOT is refused outright — a container mounting /mnt/user/appdata
+#       would otherwise turn one removal into wiping every application on the partner.
+#
+#   Refuses to run unless fallback state is NORMAL.
+#       Pushing or removing containers mid-outage edits the thing currently keeping services up.
+#
+#   Coverage names are resolved to templates by <Name>, not by filename.
+#       my-Foo.xml routinely holds a container called something else. Matching on the filename
+#       silently pushes the wrong template, or nothing at all.
+#
+# OPERATIONAL SAFEGUARDS
+#   Only in NORMAL state. FALLBACK_STATE_FILE is read before anything is pushed or removed, and
+#   any other state refuses the action. A push during a live failover would deploy a second copy
+#   of a container the partner is currently running on our behalf; a remove would delete the one
+#   doing the covering.
+#
+#   --status is exempt from that gate, because it only reports. Refusing to answer "what is
+#   deployed over there" during a failover would withhold the information precisely when it is
+#   most wanted.
+#
+#   Every deploy is verified stopped, and a container that will not stay stopped is an error
+#   rather than a warning — see DESIGN PRINCIPLES. A second live instance against the same data
+#   is the failure this whole script exists inside.
+#
+#   Push and remove are explicit modes with no default. Running the script with no flag does
+#   nothing; neither action can be reached by accident, and neither is a side effect of editing
+#   the tier list.
+#
+#   --dry-run works in every mode and touches nothing on either host — no container is built,
+#   started, stopped or removed, and no template is written or deleted.
+#
+#   Remove deletes the container's appdata on the partner as well. That is deliberate and is the
+#   most destructive thing here; the NORMAL-state gate above is what keeps it away from a
+#   partner that is mid-handback.
+#
+# CONFIGURATION
+#   master.conf
+#     FALLBACK_<HOST>_TIER1..N   the covered container names — what --push deploys and --status
+#                                reports on. This script reads that list; it never edits it.
+#
+#   host*.conf
+#     FALLBACK_STATE_FILE        overrides where fallback.sh's state is read from. Defaults to
+#                                STATE_DIR/fallback_state.db. A missing file reads as NORMAL,
+#                                which is the correct default on a host where fallback has never
+#                                run.
+#
+# RUNTIME MODES
+#   coverage_deploy.sh --push            deploy every covered container onto the partner (stopped)
+#   coverage_deploy.sh --remove          stop, remove, and delete the pushed template on the partner
+#   coverage_deploy.sh --status          report, per covered container, whether it exists there
+#   any mode supports --dry-run
+#
+# DEPENDS ON
+#   Plugin/<platform>/Partnership/containers.sh   deploy_container_from_xml(), GPU transform
+#   FALLBACK_<me>_TIER1-4                          the coverage list this acts on
+# ══════════════════════════════════════════════════════════════════════════════════════════════
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/../load_config.sh"
+source "$SCRIPT_DIR/../Plugin/$PLATFORM/Partnership/containers.sh"
+
+SSH_TIMEOUT="${SSH_TIMEOUT:-15}"
+MODE=""
+DRY_RUN="${DRY_RUN:-false}"
+
+for arg in "$@"; do
+    case "$arg" in
+        --push)    MODE="push" ;;
+        --remove)  MODE="remove" ;;
+        --status)  MODE="status" ;;
+        --dry-run) DRY_RUN=true ;;
+    esac
+done
+
+if [[ -z "$MODE" ]]; then
+    error "No mode given — use --push, --remove or --status"
+    exit 1
+fi
+
+detect_hosts
+
+if [[ -z "$REMOTE_ID" || "$REMOTE_SERVER_NAME" == "unknown" ]]; then
+    error "No partner configured — nothing to push to"
+    exit 1
+fi
+
+# ── Gate: only with fallback idle ─────────────────────────────────────────────────────────────
+# Read rather than assumed. A missing state file means fallback has never run, which is idle
+# enough; a file that says anything other than NORMAL means services are in motion right now.
+FALLBACK_STATE_FILE="${FALLBACK_STATE_FILE:-${STATE_DIR}/fallback_state.db}"
+_fb_state="NORMAL"
+if [[ -f "$FALLBACK_STATE_FILE" ]]; then
+    _fb_state=$(grep -m1 '^state=' "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
+    _fb_state="${_fb_state:-NORMAL}"
+fi
+if [[ "$_fb_state" != "NORMAL" && "$MODE" != "status" ]]; then
+    error "Fallback state is $_fb_state, not NORMAL — refusing to $MODE"
+    error "Changing what the partner holds while a failover is live edits the thing keeping services up."
+    exit 1
+fi
+
+# ── The coverage list ─────────────────────────────────────────────────────────────────────────
+COVERED=()
+for _t in 1 2 3 4; do
+    _var="FALLBACK_${MY_ID}_TIER${_t}[@]"
+    for _c in "${!_var}"; do
+        [[ -n "$_c" ]] && COVERED+=("$_c")
+    done
+done
+
+if [[ ${#COVERED[@]} -eq 0 ]]; then
+    warn "No containers are covered in FALLBACK_${MY_ID}_TIER1-4 — nothing to do"
+    exit 0
+fi
+
+log "$ICON_FALLBACK Coverage: ${#COVERED[@]} container(s) for $REMOTE_SERVER_NAME to start during an outage"
+
+resolve_remote_ip
+MIRROR="$REMOTE_SERVER_NAME"
+MIRROR_IP="$REMOTE_SERVER"
+_key_var="${MY_ID}_SSH_KEY"
+MIRROR_SSH_KEY="${!_key_var}"
+
+if [[ ! -f "$MIRROR_SSH_KEY" ]]; then
+    error "SSH key $MIRROR_SSH_KEY not found — cannot reach $MIRROR"
+    exit 1
+fi
+
+# ── name -> template ──────────────────────────────────────────────────────────────────────────
+# Matched on the <Name> element. Filenames lie often enough that trusting them would push the
+# wrong container without saying so.
+xml_for_container() {
+    local want="$1" f n
+    for f in "$TEMPLATES_DIR"/*.xml; do
+        [[ -f "$f" ]] || continue
+        n=$(awk 'match($0,/<Name>([^<]+)<\/Name>/,a){print a[1];exit}' "$f")
+        [[ "$n" == "$want" ]] && { echo "$f"; return 0; }
+    done
+    return 1
+}
+
+remote_has_container() {
+    timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
+        -o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
+        "docker inspect $(printf '%q' "$1") >/dev/null 2>&1" 2>/dev/null
+}
+
+remote_state_of() {
+    timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
+        -o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
+        "docker inspect -f '{{.State.Status}}' $(printf '%q' "$1") 2>/dev/null" 2>/dev/null
+}
+
+OK=0; FAIL=0; SKIP=0
+
+case "$MODE" in
+
+status)
+    # Written as a cache as well as printed. The assistant's fallback_state block cannot afford an
+    # SSH round trip per container mid-question, so it reads this file and reports its AGE — a stale
+    # answer stated as stale is useful, stated as current it is the exact failure this feature
+    # exists to prevent.
+    _present="" _missing=""
+    for c in "${COVERED[@]}"; do
+        if remote_has_container "$c"; then
+            _st=$(remote_state_of "$c")
+            printf '  %-28s on %s (%s)\n' "$c" "$MIRROR" "$_st"
+            _present+="\"$c\":\"${_st:-unknown}\","
+            OK=$((OK+1))
+        else
+            printf '  %-28s MISSING on %s — docker start would fail\n' "$c" "$MIRROR"
+            _missing+="\"$c\","
+            FAIL=$((FAIL+1))
+        fi
+    done
+    log "$ICON_FALLBACK Coverage present: $OK · missing: $FAIL"
+
+    mkdir -p "$VV_CACHE_ROOT/api" 2>/dev/null || mkdir -p /tmp/varaverk/api 2>/dev/null
+    _cache="${VV_CACHE_ROOT:-/tmp/varaverk}/api/fallback_presence.json"
+    # Written atomically — a half-written cache read mid-question would report containers as
+    # missing that are merely unparsed.
+    printf '{"present":{%s},"missing":[%s],"partner":"%s","checked":%s}\n' \
+        "${_present%,}" "${_missing%,}" "$MIRROR" "$(date +%s)" > "$_cache.tmp" \
+        && mv -f "$_cache.tmp" "$_cache"
+
+    [[ "$FAIL" -gt 0 ]] && exit 2 || exit 0
+    ;;
+
+push)
+    # Networks first — a container whose network is absent is created and then cannot start,
+    # which is the failure that read as "auth 0/8, arr 0/5" during onboarding.
+    _nets=()
+    for c in "${COVERED[@]}"; do
+        x=$(xml_for_container "$c") || continue
+        net=$(sed -n 's/.*<Network>\([^<]*\)<\/Network>.*/\1/p' "$x" 2>/dev/null | head -1)
+        net="${net//[[:space:]]/}"
+        # br* is host hardware. wg* is a WireGuard-backed bridge whose meaning does NOT travel:
+        # recreating it on the partner as a plain bridge yields a network that exists, starts its
+        # containers, and routes their traffic OUTSIDE the tunnel. ChannelTube rides wg0 here.
+        case "$net" in
+            ''|bridge|host|none|br[0-9]*) continue ;;
+            wg[0-9]*)
+                warn "$c uses $net — a WireGuard-backed network. NOT created on $MIRROR: a plain"
+                warn "  bridge of the same name would route its traffic outside the tunnel. Build the"
+                warn "  matching tunnel there first, or drop $c from coverage."
+                continue ;;
+        esac
+        _seen=false
+        for n in "${_nets[@]}"; do [[ "$n" == "$net" ]] && { _seen=true; break; }; done
+        [[ "$_seen" == false ]] && _nets+=("$net")
+    done
+    for net in "${_nets[@]}"; do
+        driver=$(timeout "${DOCKER_TIMEOUT:-30}" docker network inspect "$net" --format '{{.Driver}}' 2>/dev/null)
+        if [[ "$driver" != "bridge" ]]; then
+            warn "Network $net is '${driver:-absent}' here, not bridge — create it on $MIRROR by hand"
+            continue
+        fi
+        if [[ "$DRY_RUN" == true ]]; then
+            warn "DRY RUN — would ensure network $net on $MIRROR"
+            continue
+        fi
+        timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
+            -o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
+            "docker network inspect $(printf '%q' "$net") >/dev/null 2>&1 \
+             || docker network create --driver bridge $(printf '%q' "$net") >/dev/null" 2>/dev/null \
+          && log "  network $net ready on $MIRROR" \
+          || warn "  could not ensure network $net on $MIRROR"
+    done
+
+    for c in "${COVERED[@]}"; do
+        x=$(xml_for_container "$c") || {
+            warn "$c — no template in $TEMPLATES_DIR names it; skipped"
+            SKIP=$((SKIP+1)); continue
+        }
+        if ! deploy_container_from_xml "$x" "$MIRROR_IP" "$MIRROR_SSH_KEY"; then
+            error "$c — deploy failed"
+            FAIL=$((FAIL+1)); continue
+        fi
+        if [[ "$DRY_RUN" == true ]]; then OK=$((OK+1)); continue; fi
+
+        # Deployed containers must not run here. Stop, then re-inspect — a stop that did not take
+        # is the one outcome that silently duplicates a live service against shared data.
+        timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
+            -o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
+            "docker stop $(printf '%q' "$c") >/dev/null 2>&1" 2>/dev/null
+        st=$(remote_state_of "$c")
+        if [[ "$st" == "running" ]]; then
+            error "$c is RUNNING on $MIRROR after deploy and would not stop — stop it there before continuing"
+            FAIL=$((FAIL+1))
+        else
+            log "  $c deployed and ${st:-stopped} on $MIRROR ✅"
+            OK=$((OK+1))
+        fi
+    done
+    log "$ICON_FALLBACK Push complete — deployed $OK · failed $FAIL · skipped $SKIP"
+    [[ "$FAIL" -gt 0 ]] && exit 1 || exit 0
+    ;;
+
+remove)
+    for c in "${COVERED[@]}"; do
+        if ! remote_has_container "$c"; then
+            log "  $c not on $MIRROR — nothing to remove"
+            SKIP=$((SKIP+1)); continue
+        fi
+        # Binds are read BEFORE the container goes — once it is removed there is nothing left to
+        # enumerate, and a path list gathered afterwards would silently be empty.
+        _binds=$(timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
+            -o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
+            "docker inspect --format '{{range .HostConfig.Binds}}{{println .}}{{end}}' $(printf '%q' "$c") 2>/dev/null \
+             | awk -F: '{print \$1}'" 2>/dev/null)
+
+        _wipe=()
+        while IFS= read -r _p; do
+            [[ -z "$_p" ]] && continue
+            # Only appdata, and never an appdata root. /mnt/user/appdata as a bind would make one
+            # container removal delete every application on the partner.
+            [[ "$_p" =~ ^/mnt/[^/]+/appdata[^/]*/.+ ]] || {
+                [[ "$_p" =~ ^/mnt/[^/]+/appdata[^/]*/?$ ]] && \
+                    warn "  $c binds the appdata ROOT ($_p) — refusing to delete it"
+                continue
+            }
+            _wipe+=("$_p")
+        done <<< "$_binds"
+
+        if [[ "$DRY_RUN" == true ]]; then
+            warn "DRY RUN — would stop and remove $c on $MIRROR"
+            for _p in "${_wipe[@]}"; do warn "  DRY RUN — would delete appdata $_p on $MIRROR"; done
+            OK=$((OK+1)); continue
+        fi
+        x=$(xml_for_container "$c") && xml_name=$(basename "$x") || xml_name=""
+        if timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
+             -o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
+             "docker stop $(printf '%q' "$c") >/dev/null 2>&1; \
+              docker rm $(printf '%q' "$c") >/dev/null 2>&1; \
+              ${xml_name:+rm -f ${TEMPLATES_DIR}/$(printf '%q' "$xml_name");} \
+              ! docker inspect $(printf '%q' "$c") >/dev/null 2>&1" 2>/dev/null; then
+            log "  $c removed from $MIRROR ✅"
+            for _p in "${_wipe[@]}"; do
+                if timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
+                     -o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
+                     "rm -rf -- $(printf '%q' "$_p") && ! [ -e $(printf '%q' "$_p") ]" 2>/dev/null; then
+                    log "    appdata deleted on $MIRROR: $_p"
+                else
+                    warn "    could not delete appdata on $MIRROR: $_p"
+                    FAIL=$((FAIL+1))
+                fi
+            done
+            OK=$((OK+1))
+        else
+            error "$c — removal failed or it still exists on $MIRROR"
+            FAIL=$((FAIL+1))
+        fi
+    done
+    log "$ICON_FALLBACK Remove complete — removed $OK · failed $FAIL · skipped $SKIP"
+    [[ "$FAIL" -gt 0 ]] && exit 1 || exit 0
+    ;;
+esac

Fallback/fallback.sh

@@ -89,7 +89,50 @@
 #
 # FALLBACK_ENABLED Gate
 #   Exits cleanly when disabled — safe to run on servers being rebuilt without
-#   triggering spurious fallback actions.
+#   triggering spurious fallback actions. Fail-closed: anything that is not exactly
+#   "true" counts as disabled, so a malformed toggle cannot grant this script DDNS
+#   authority and cross-server container control by accident.
+#
+# Docker Presence Check
+#   Verifies the docker binary exists before the state machine starts.
+#
+# Remote IP Resolution
+#   resolve_remote_ip() must resolve the partner before any SSH operation, so a
+#   remote command can never be issued against an unresolved or stale address.
+#
+# Asymmetric Failover / Handback
+#   Entering FALLBACK is immediate — a down partner means users are already affected.
+#   Returning requires FALLBACK_HANDBACK_STRIKES consecutive remote-up checks. The
+#   asymmetry is deliberate: protecting fast costs a few minutes of redundant
+#   coverage, handing back fast on a flapping partner costs a second outage.
+#
+# DDNS Excluded From Tier Loops
+#   The Tier 1 stop and start loops explicitly skip any container that is also in
+#   REMOTE_DDNS_CONTAINERS. DDNS is sequenced by the handoff and cutover steps alone,
+#   so ordinary tier processing can never move DNS at the wrong moment.
+#
+# Writeback Delay Gate
+#   Tier writeback rsync only runs once the outage has exceeded that tier's writeback
+#   delay. A brief blip does not trigger a full data writeback, which would cost more
+#   than the outage it is compensating for.
+#
+# FALLBACK_RSYNC_ENABLED Gate
+#   Writeback is skipped entirely when disabled, and the skip is announced rather than
+#   silent — containers still hand back, but nobody is left assuming data moved.
+#
+# Play State Sync Before Cutover
+#   Handback retries play_state_sync up to PLAY_SYNC_HANDBACK_RETRIES times before DNS
+#   cuts over, so users land on current watch state. Exhausting retries warns and
+#   proceeds — stale resume positions are not worth holding DNS on a downed service.
+#
+# Partnership Suspend Abort
+#   If partnership goes inactive mid-fallback, _abort_fallback_containers() stops the
+#   fallback containers and returns to NORMAL rather than leaving this host serving a
+#   partner it is no longer paired with.
+#
+# Container Verify Wait
+#   CONTAINER_VERIFY_WAIT seconds elapse after each start before the running check, so
+#   a container that starts and immediately crashes is caught rather than counted up.
 #
 # Version Parity Check
 #   Refuses handback if remote unRAID version doesn't match. A mismatch may
@@ -271,8 +314,55 @@ if [[ "$EUID" -ne 0 ]]; then
     exit 1
 fi
 
-# FALLBACK_ENABLED gate — exits cleanly when disabled (e.g. HOST2 being rebuilt)
+
-if [[ "${FALLBACK_ENABLED:-false}" == false ]]; then
+# ── Persistent dry-run ────────────────────────────────────────────────────────────────────────
+# Conf-driven, not argument-driven, deliberately. array_started.sh launches every entry as a bare
+# `bash script.sh &` with no arguments, so a --dry-run typed at a terminal survives exactly until
+# the next array start — and then the LIVE daemon comes up in its place, silently, which is the
+# one transition nobody would be watching for.
+#
+# Setting it here means the mode is a property of the install rather than of how the process
+# happened to be started: array start, the Fallback tab's button, and a hand-run all agree.
+#
+# OR, never override: --dry-run on the command line still wins over a conf that says false, so an
+# ad-hoc preview against a live install needs no conf edit.
+if [[ "${FALLBACK_DRY_RUN:-false}" == "true" ]]; then
+    DRY_RUN=true
+fi
+
+# ── Persistent log ────────────────────────────────────────────────────────────────────────────
+# array_started.sh launches every entry as a bare `bash script.sh &` with no redirection, so this
+# daemon's output has never been captured anywhere: /var/log/varaverk has a directory for every
+# other script family and none for Fallback. A month of dry-run observation would have persisted
+# nothing at all.
+#
+# /var/log is a 128 MB tmpfs on Unraid — RAM, and cleared on reboot — so the log goes to
+# LOG_ARCHIVE_DIR, which follows DATA_DIR onto real storage.
+#
+# Only when stdout is not a terminal. Run by hand, output still goes to the terminal exactly as
+# before; run by array_started or the Fallback tab's button, it lands in the file. A plain append
+# redirect rather than `tee` through process substitution: no extra child to outlive, and nothing
+# for the shutdown trap to race.
+FALLBACK_LOG="${LOG_ARCHIVE_DIR:-${DATA_DIR:-/tmp}/logs}/fallback.log"
+if [[ ! -t 1 ]]; then
+    mkdir -p "$(dirname "$FALLBACK_LOG")" 2>/dev/null
+    # One rotation, sized rather than line-counted — the whole point of this log is a long run,
+    # and _orch_trim_log()'s 1000-line cap would discard weeks of it. Event-only output (no
+    # --log) is a few lines per incident, so this holds years; --log fills it in about a fortnight
+    # and then keeps the most recent fortnight plus the one before it.
+    _fb_max=$(( ${FALLBACK_LOG_MAX_MB:-5} * 1048576 ))
+    if [[ -f "$FALLBACK_LOG" ]] && (( $(stat -c %s "$FALLBACK_LOG" 2>/dev/null || echo 0) > _fb_max )); then
+        mv -f "$FALLBACK_LOG" "${FALLBACK_LOG}.1" 2>/dev/null
+    fi
+    exec >> "$FALLBACK_LOG" 2>&1
+    echo ""
+    echo "═══ fallback.sh started $(date '+%Y-%m-%d %H:%M:%S') — dry_run=${DRY_RUN} pid=$$ ═══"
+fi
+# FALLBACK_ENABLED gate — exits cleanly when disabled.
+# Fail-closed: anything that isn't exactly "true" disables fallback. Matching only the
+# literal "false" would let a typo ("no", "0", "FALSE") hand this script DDNS authority
+# and cross-server container control on a toggle nobody meant to set.
+if [[ "${FALLBACK_ENABLED:-false}" != "true" ]]; then
     warn "FALLBACK_ENABLED=false — fallback monitoring disabled"
     warn "Set FALLBACK_ENABLED=true in master.conf when both servers are ready"
     exit 0
@@ -289,8 +379,6 @@ detect_hosts
 require_partnership
 resolve_remote_ip
 
-# Validate unRAID notify script — used throughout for state change notifications
-
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no container or DDNS changes will be made"
 
 # Timeout for all docker and SSH docker commands
@@ -324,6 +412,22 @@ state_set() {
 }
 
 state_init() {
+    # A dry run must not leave the host believing it failed over. state_set() writes
+    # unconditionally, and this file survives reboots and is what the real daemon — and the
+    # Monitor and Fallback cards — read to decide what is happening. A --dry-run walk through
+    # FAILOVER would have written state=FALLBACK, the tier flags and the strike counter into it
+    # for real, and nothing would have put them back.
+    #
+    # Copied rather than merely redirected, so the preview still starts from the live state and
+    # can advance through tiers exactly as a real run would. The copy lands in the RAM cache and
+    # dies with the reboot.
+    if [[ "$DRY_RUN" == true ]]; then
+        local live="$FALLBACK_STATE_FILE"
+        FALLBACK_STATE_FILE="${VV_CACHE_ROOT:-/tmp/varaverk}/fallback_state.dryrun.$$"
+        mkdir -p "$(dirname "$FALLBACK_STATE_FILE")"
+        if [[ -f "$live" ]]; then cp -f "$live" "$FALLBACK_STATE_FILE"; else : > "$FALLBACK_STATE_FILE"; fi
+        warn "DRY RUN — state writes redirected to $FALLBACK_STATE_FILE (live state untouched)"
+    fi
     mkdir -p "$(dirname "$FALLBACK_STATE_FILE")"
     [[ ! -f "$FALLBACK_STATE_FILE" ]] && touch "$FALLBACK_STATE_FILE"
     [[ -z "$(state_get state)" ]]                  && state_set state "NORMAL"
@@ -715,8 +819,16 @@ run_handback() {
                 for job in "${jobs[@]}"; do
                     [[ -z "$job" ]] && continue
                     log "Syncing: $job"
-                    [[ "$DRY_RUN" == false ]] && bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$job" \
+                    # if/else, not A && B || C. In the shorthand a REAL run whose rsync exits
-                        || warn "DRY RUN — would rsync: $job"
+                    # non-zero falls through to the || branch and logs "DRY RUN — would rsync",
+                    # so a failed Tier writeback reported itself as a preview and the real
+                    # failure went unsaid. The Tier 1 block below always had this right.
+                    if [[ "$DRY_RUN" == false ]]; then
+                        bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$job" \
+                            || error "Tier $tier writeback FAILED: $job"
+                    else
+                        warn "DRY RUN — would rsync: $job"
+                    fi
                 done
             else
                 log "Tier $tier writeback skipped — outage ${outage_minutes}min < ${threshold}min"
@@ -860,6 +972,19 @@ echo "  $ICON_NET  Remote IP: $REMOTE_SERVER"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 
 FALLBACK_RUNNING=true
+
+# The dry-run state copy is per-PID and would otherwise accumulate one file per preview run.
+# EXIT as well as the signals, because a dry run is usually ended with Ctrl-C or --stop but can
+# also just fall out of the loop.
+dryrun_state_cleanup() {
+    [[ "$DRY_RUN" == true && "$FALLBACK_STATE_FILE" == *".dryrun."* ]] && rm -f "$FALLBACK_STATE_FILE"
+    return 0
+}
+# Must call _release_all_locks too. acquire_lock() registers its own EXIT trap, and bash keeps
+# exactly one per signal — a bare `trap ... EXIT` here silently replaced it and orphaned
+# fallback.lock, which is the precise failure the _LOCK_FILES registry in common.sh was built to
+# stop. The signal trap only needs `exit 0`; that fires EXIT, which does both jobs.
+trap 'dryrun_state_cleanup; _release_all_locks' EXIT
 trap 'FALLBACK_RUNNING=false; warn "Fallback received shutdown signal — stopping cleanly"; exit 0' \
     SIGTERM SIGINT
 

Fallback/fallback_test.sh

@@ -41,9 +41,23 @@
 # ==============================================================================================
 #
 # iptables Safety Trap
-#   The DROP rule is removed via trap on ANY exit — normal completion, crash, error,
+#   The DROP rule is removed via an EXIT trap that fires on normal completion, error
-#   ctrl-c. Remote connectivity is always restored regardless of test outcome.
+#   exit, script crash, ctrl-c (SIGINT) and SIGTERM — verified, not assumed. Remote
-#   You cannot accidentally leave the remote permanently blocked.
+#   connectivity is restored regardless of test outcome.
+#
+# Stale Rule Sweep
+#   The trap above cannot cover SIGKILL or a power cut, which are the only ways a DROP
+#   rule survives the test. One stranded that way makes fallback.sh see the partner as
+#   permanently down and hold FALLBACK indefinitely, so pre-flight clears any leftover
+#   rule before doing anything else — including before the reachability check, which
+#   would otherwise fail and blame the network for the test's own residue.
+#
+# Root Enforcement
+#   iptables and container control require root.
+#
+# iptables Presence Check
+#   platform_require_cmd confirms iptables exists before the test begins — there is no
+#   point entering a connectivity simulation that cannot simulate anything.
 #
 # FALLBACK_ENABLED Gate
 #   Aborts if FALLBACK_ENABLED=false. Testing a disabled fallback system is
@@ -77,13 +91,13 @@
 #   FALLBACK_TEST_BLOCK_WAIT
 #       Seconds to wait in Phase 3 for fallback.sh to detect the outage.
 #       Must be > FALLBACK_CHECK_INTERVAL + buffer. At 30s interval: use ≥60s.
-#       (default: 60)
+#       (shipped default: 150)
 #
 #   FALLBACK_TEST_HANDBACK_WAIT
 #       Seconds to wait in Phase 6 for fallback.sh to complete handback.
 #       Must cover: FALLBACK_HANDBACK_STRIKES × FALLBACK_CHECK_INTERVAL + rsync
 #       duration + container start time. At 3 strikes × 30s + ~2min rsync +
-#       ~1min container start: use ≥240s. (default: 300)
+#       ~1min container start: use ≥240s. (shipped default: 360)
 #
 # ==============================================================================================
 # RUNTIME MODES
@@ -104,6 +118,11 @@
 # fallback_test.sh --log
 #     Verbose output on every check in every phase.
 #
+# fallback_test.sh --stop
+#     Stop a running test. SIGTERM only — never SIGKILL, because only this script's EXIT
+#     trap removes the iptables DROP rule it installed. Also sweeps a rule stranded by an
+#     earlier SIGKILL or power cut.
+#
 # ==============================================================================================
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -112,6 +131,58 @@ source "$SCRIPT_DIR/../load_config.sh"
 
 parse_args "$@"
 
+# ── Stop mode — runs before acquire_lock so we can target the holding instance ────────────────
+#
+# SIGTERM ONLY, and deliberately no SIGKILL escalation — the opposite of fallback.sh --stop.
+# A running test holds an iptables DROP rule against the partner, and the only thing that removes
+# it is this script's own EXIT trap. SIGKILL does not run traps, so force-killing a test strands
+# the rule: the partner stays invisible, fallback.sh reads that as a permanent outage and holds
+# FALLBACK indefinitely. A test that will not die is a worse outcome than a test still running,
+# so this reports the stranded rule and the command to clear it rather than causing one.
+if [[ " ${PARSED_ARGS[*]:-} " == *" --stop "* ]]; then
+    LOCKFILE="${LOCK_DIR}/fallback_test.lock"
+    if [[ ! -f "$LOCKFILE" ]]; then
+        log "No fallback_test.sh lock found — not running"
+        exit 0
+    fi
+    lock_content=$(cat "$LOCKFILE" 2>/dev/null)
+    target_pid="${lock_content%%:*}"
+    if [[ -z "$target_pid" ]] || ! kill -0 "$target_pid" 2>/dev/null; then
+        warn "Stale lock — fallback_test.sh not running (PID ${target_pid:-unknown} gone) — clearing"
+        rm -f "$LOCKFILE"
+        # A stale lock is exactly the SIGKILL/power-cut case, so the rule may still be in place.
+        if iptables -C OUTPUT -d "${REMOTE_SERVER:-0.0.0.0}" -j DROP 2>/dev/null; then
+            warn "Stranded iptables DROP rule found for $REMOTE_SERVER — removing"
+            iptables -D OUTPUT -d "$REMOTE_SERVER" -j DROP 2>/dev/null \
+                && warn "Stranded rule removed — remote connectivity restored ✅" \
+                || error "Could not remove stranded rule — run: iptables -D OUTPUT -d $REMOTE_SERVER -j DROP"
+        fi
+        exit 0
+    fi
+    warn "Stopping fallback_test.sh (PID $target_pid) — SIGTERM so its trap clears the iptables rule..."
+    kill -TERM "$target_pid" 2>/dev/null || true
+    waited=0
+    while kill -0 "$target_pid" 2>/dev/null && [[ "$waited" -lt 30 ]]; do
+        sleep 1
+        (( waited++ )) || true
+    done
+    if kill -0 "$target_pid" 2>/dev/null; then
+        error "fallback_test.sh (PID $target_pid) did not exit within 30s"
+        error "NOT force-killing — SIGKILL would strand the iptables DROP rule on $REMOTE_SERVER"
+        error "Wait, or clear manually: iptables -D OUTPUT -d $REMOTE_SERVER -j DROP"
+        exit 1
+    fi
+    warn "Stopped: fallback_test.sh (PID $target_pid) ✅"
+    if iptables -C OUTPUT -d "${REMOTE_SERVER:-0.0.0.0}" -j DROP 2>/dev/null; then
+        error "iptables DROP rule for $REMOTE_SERVER survived the stop — removing"
+        iptables -D OUTPUT -d "$REMOTE_SERVER" -j DROP 2>/dev/null \
+            && warn "Rule removed ✅" || error "Could not remove — run it by hand"
+    else
+        log "No iptables DROP rule remains for $REMOTE_SERVER ✅"
+    fi
+    exit 0
+fi
+
 FALLBACK_SCRIPT="$SCRIPT_DIR/fallback.sh"
 DOCKER_TIMEOUT=15
 
@@ -139,6 +210,28 @@ cleanup() {
 
 trap cleanup EXIT
 
+# ── Stale rule sweep — the one case the trap above cannot cover ───────────────────────────────
+# The EXIT trap fires on normal exit, error, ctrl-c and SIGTERM, but not on SIGKILL or a power
+# cut. A DROP rule stranded that way makes fallback.sh see the partner as permanently down and
+# sit in FALLBACK indefinitely — so clear any leftover from a previous run before starting.
+_clear_stale_block() {
+    [[ -z "${REMOTE_SERVER:-}" ]] && return
+    local removed=0
+    while iptables -C OUTPUT -d "$REMOTE_SERVER" -j DROP 2>/dev/null; do
+        if [[ "$DRY_RUN" == true ]]; then
+            warn "DRY RUN — would remove stale iptables block on $REMOTE_SERVER"
+            return
+        fi
+        iptables -D OUTPUT -d "$REMOTE_SERVER" -j DROP 2>/dev/null || break
+        (( removed++ ))
+    done
+    if [[ "$removed" -gt 0 ]]; then
+        warn "$ICON_SHIELD Removed $removed stale iptables block(s) on $REMOTE_SERVER from a previous run"
+        notify "Fallback test cleared $removed stale iptables block(s) on $(hostname) — a previous test was killed before cleanup" \
+            "Fallback Test" "warning"
+    fi
+}
+
 # ==============================================================================================
 # ━━━ Setup ━━━
 # ==============================================================================================
@@ -149,7 +242,8 @@ if [[ "$EUID" -ne 0 ]]; then
 fi
 
 # FALLBACK_ENABLED gate — no point testing if fallback is disabled
-if [[ "${FALLBACK_ENABLED:-false}" == false ]]; then
+# Fail-closed, matching fallback.sh — anything not exactly "true" counts as disabled.
+if [[ "${FALLBACK_ENABLED:-false}" != "true" ]]; then
     warn "FALLBACK_ENABLED=false — fallback test aborted"
     warn "Enable fallback in master.conf before running this test"
     exit 0
@@ -238,6 +332,11 @@ echo "━━━━━━━━━━━━━━━━━━━━━━━━
 echo ""
 echo "━━━ $ICON_SHIELD Phase 1 — Pre-flight ━━━"
 
+# Must run before the reachability check below — a stale DROP rule from a killed run makes
+# the partner look unreachable, and the test would abort blaming the network for its own
+# leftover.
+_clear_stale_block
+
 # Remote reachable
 if ping_remote; then
     log "$REMOTE_SERVER_NAME is reachable"
@@ -283,6 +382,27 @@ else
     warn "No state file found — assuming NORMAL (first run)"
 fi
 
+# fallback.sh must actually be RUNNING, not merely enabled
+#
+# Every phase after this one waits for the daemon to change state. FALLBACK_ENABLED=true says
+# it is allowed to run; it does not say array_started.sh launched it, or that it is still alive.
+# Without this the test passes pre-flight, drops a real iptables rule on the partner, waits
+# FALLBACK_TEST_BLOCK_WAIT for a transition nothing is there to make, and fails Phase 3 blaming
+# fallback detection. Only the EXIT trap gets connectivity back.
+#
+# In --dry-run nothing is blocked and nothing is waited on, so a dead daemon is worth saying but
+# not worth aborting for — the walkthrough still shows the operator the shape of the run.
+if pgrep -f "Fallback/fallback\.sh" >/dev/null 2>&1; then
+    log "fallback.sh daemon is running"
+elif [[ "$DRY_RUN" == true ]]; then
+    warn "fallback.sh is NOT running — a real test would abort here"
+else
+    error "fallback.sh is not running — nothing would detect the outage this test creates"
+    error "Start it with array_started.sh, or run with --dry-run to walk the phases"
+    phase_fail "Pre-flight"
+    exit 1
+fi
+
 # Tier 1 containers configured
 if [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then
     error "No Tier 1 containers configured for $MY_ID → $REMOTE_ID"

Kernel/decision_engine.sh

@@ -15,6 +15,20 @@
 # Currently paired with: Arrs_Stack/playback_aware_lidarr_discovery.sh
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# A library, not a program. Consumers source it and drive the pipeline themselves:
+#
+#   1. score_candidate()      — sum the four weighted component scores
+#   2. apply_temporal_decay() — reduce by one unit per 30 days of age, floored at 0
+#   3. is_duplicate_candidate() — check the consumer's own history file
+#   4. make_decision()        — ACCEPT or REJECT against the consumer's threshold
+#
+# Every input is supplied by the caller and every output is returned to it. The engine holds
+# no state between calls, reads no config, and never acts on its own verdict.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -38,6 +52,60 @@
 #   the consumer's concern — the engine never acts on its own verdict.
 #
 # ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# No Side Effects — Structural, Not Incidental
+#   Writes no files, makes no API calls, deletes nothing, starts nothing. This is the
+#   safeguard: a scoring mistake here can only ever produce a wrong number, never a wrong
+#   action. Keep it that way — the moment this library acquires a side effect, every consumer
+#   inherits it silently.
+#
+# No Root, No Lock, No detect_hosts — Deliberate
+#   Correct for a sourced library and should not be "fixed" to match the executable scripts.
+#   There is no state for a lock to protect, no privileged operation to gate, and no
+#   host-specific config to alias. It runs entirely inside the caller's process.
+#
+# Caller Owns the Verdict
+#   ACCEPT/REJECT is a return value, not an instruction. Nothing here can cause a candidate
+#   to be added, removed, or downloaded — the consumer decides what a verdict means.
+#
+# Literal Duplicate Matching
+#   is_duplicate_candidate() matches with grep -Fx, treating the candidate as data rather
+#   than a pattern. Artist and title strings routinely contain regex metacharacters, and a
+#   false positive here silently discards a genuinely new candidate.
+#
+# Decay Floors at Zero
+#   apply_temporal_decay() clamps at 0, so an old signal can never become a negative score
+#   that drags an otherwise-passing candidate below threshold.
+#
+# Integer Arithmetic Throughout
+#   All scoring is integer. No floating point means no locale-dependent decimal parsing and
+#   no rounding drift between hosts.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# None, by design. The engine reads no conf file and no environment variable.
+#
+# Thresholds, weights and strictness profiles live with the consumer — see
+# LIDARR_DISCOVERY_* / SONARR_DISCOVERY_* / RADARR_DISCOVERY_* in master.conf. That is what
+# keeps the core domain-agnostic: music strictness cannot leak into TV intake, because the
+# engine never learns which domain it is scoring for.
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# None — this file is sourced, never executed:
+#
+#   source "$SCRIPT_DIR/../Kernel/decision_engine.sh"
+#
+# It has no argument parsing, no --dry-run and no --status, because it takes no action that
+# a dry run could suppress.
+#
+# ==============================================================================================
 # FUNCTIONS
 # ==============================================================================================
 #
@@ -111,7 +179,12 @@ is_duplicate_candidate() {
     local candidate="$1"
     local history_file="$2"
 
-    grep -qi "^${candidate}$" "$history_file" 2>/dev/null
+    # -F -x, not "^$" anchors: the candidate is data, not a pattern. Interpolating it into a
+    # regex makes every metacharacter in an artist or title active — "R.E.M." matches "RxExMy",
+    # and an unbalanced bracket makes grep error out entirely. Either way the caller reads the
+    # result as "already seen" and silently skips something genuinely new. -F disables regex,
+    # -x anchors the whole line, which is exactly what the anchors were reaching for.
+    grep -qiFx -- "$candidate" "$history_file" 2>/dev/null
 }
 
 # ── make_decision ─────────────────────────────────────────────────────────────

Manual.md

@@ -3,11 +3,23 @@
 Getting a fresh two-server ecosystem running from scratch.  
 For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.
 
+**Ten steps, and they're in this order for a reason.** Each one assumes the previous one
+actually worked — not that you ran it, that it *worked*. Every step below ends with a way to
+check, and skipping those checks is how you end up three steps later debugging the wrong
+thing entirely.
+
+The steps that hurt most when rushed are **3** (naming) and **8** (testing failover). Step 3
+because renaming anything afterwards means chasing it through every conf, and step 8 because
+untested failover isn't redundancy — it's a guess you haven't checked yet.
+
+Budget an evening. It is not hard, but it is not five minutes either.
+
 ---
 
 ## ━━━ BEFORE YOU START ━━━
 
-Things that must be in place before you touch any scripts.
+None of this is optional and none of it is Varaverk's job to install. Get these in place
+first — every step after here assumes they're already true.
 
 ---
 
@@ -157,6 +169,21 @@ Replace these with your actual unRAID server hostnames. These must match exactly
 `detect_hosts()` compares the running server's hostname against these two values to
 know which server it is on. Everything else in the ecosystem flows from this.
 
+**Get this wrong and nothing works, but nothing errors either.** If neither value matches,
+`MY_ID` is never set, and every script that depends on it either exits early or resolves
+`${MY_ID}_SOMETHING` to an empty variable and quietly takes the wrong branch. Copy the value
+straight out of `hostname` on each box rather than typing what you think it is:
+
+```bash
+hostname     # run this on each server, paste the exact output
+```
+
+> **The 15-character trap.** unRAID truncates the Server Name to 15 characters for NetBIOS.
+> If your name is longer, what you set in the WebGUI and what `hostname` returns are two
+> different strings. `detect_hosts()` has a fallback that matches a truncated 15-char
+> hostname against a longer configured value — but only when the live hostname is
+> *exactly* 15 characters. Don't rely on it. Use the real `hostname` output.
+
 ---
 
 ### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────
@@ -206,15 +233,27 @@ HOST1_CRITICAL_SYNC_SHARES=(
     "/mnt/user/appdata-Fallback/Critical-Data|critical-data"
 )
 
-# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate)
+# Containers to start when HOST1 goes down (Tier 1 = immediate)
-HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=()   # HOST2's own covers for HOST1, and vice versa
+# Lives in host1.conf. Named for the host being COVERED, not the host doing the covering.
-HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=(
+FALLBACK_HOST1_TIER1=(
     "Emby"
     "VaultWarden"
     "NginxProxyManager"
 )
 ```
 
+**Read that naming carefully — it catches people.** The variable is
+`FALLBACK_${REMOTE_ID}_TIER${N}`, and `REMOTE_ID` is *the host that went down*. So
+`FALLBACK_HOST1_TIER1` is the list HOST2 reads when HOST1 is unreachable. It defines
+"what HOST1 needs covering", not "what HOST2 runs".
+
+It lives in `host1.conf` for the same reason — HOST1 owns the description of its own
+service stack. HOST2 receives it through the conf cache rather than keeping its own
+opinion about what HOST1 runs. One list, one owner, no drift.
+
+Tiers 2–4 activate progressively after `HOST1_TIER2_DELAY` etc., so a brief outage never
+drags the entire remote stack across.
+
 For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
 and `Manual-Watchdogs.md`.
 
@@ -455,6 +494,18 @@ bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log
 The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
 detect it and start Tier 1 containers, then unblocks and waits for handback.
 
+**Do not skip this step.** Everything up to here you can verify by reading output. Failover
+is the one part you cannot confirm by looking at it — a typo in a tier list, a container name
+that doesn't exist on the other side, a DDNS container that was renamed six months ago: all
+of it sits there looking completely fine until the moment it's needed.
+
+The test costs you twenty minutes and no downtime. The alternative is finding out at 2am,
+during the outage, when you have exactly one chance to get it right.
+
+**Watch the handback as closely as the failover.** Coming back is the harder half — it has to
+stop the remote's DDNS, sync the data back, start the local containers, and only then bring
+local DDNS up. Failover starting correctly tells you nothing about whether handback does.
+
 See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
 how to interpret the output.
 
@@ -651,6 +702,14 @@ Each subsystem has a README with the design decisions and a Manual with the conf
 reference and troubleshooting. The `--status` flag on any script shows the current
 configuration and state.
 
+**Turn things on one at a time and give each one a few days.** Everything below is off by
+default on a fresh install, and that's deliberate — a stack where six new subsystems went
+live the same night is a stack where you have no idea which one to blame. Enable, watch it
+through a full daily cycle, then enable the next.
+
+Anything that deletes files — the arr cleanup scripts especially — gets a `--dry-run --log`
+first. Read the list. Every time, not just the first time.
+
 ```
 Watchdogs/        → README-Watchdogs.md        configure memory limits, container lists
 Media/            → README-Media.md             enable arr cleanup, discovery scripts

Media/Manual-Media.md

@@ -86,10 +86,25 @@ MEDIA_FILE_PATTERNS=(
 ### Play State Sync
 
 ```bash
-PLAY_STATE_SYNC_ENABLED=true          # toggle entire sync
+PLAY_SYNC_ENABLED=true                # toggle entire sync
-PLAY_STATE_SYNC_LOOKBACK_DAYS=30      # history window for played items
+PLAY_SYNC_REMOTE=true                 # sync across hosts via Tailscale
+                                      # false = local servers only (this host's Emby + Jellyfin)
+PLAY_SYNC_TYPES="Movie,Episode"       # item types to sync — Audio excluded, music library too large
+PLAY_SYNC_FAV_TYPES="MusicArtist,MusicAlbum,Movie,Series"   # favourites, union sync, never unmarks
+
+PLAY_SYNC_PROBE=true                  # skip per-item work when nothing changed since last run
+PLAY_SYNC_PROBE_MAX_AGE_HOURS=24      # force a full comparison when the fingerprint is older
+
+PLAY_SYNC_HANDBACK_RETRIES=5          # fallback handback: attempts before DNS cutover proceeds
+PLAY_SYNC_HANDBACK_RETRY_DELAY=60     # seconds between those attempts
 ```
 
+> **There is no date/lookback filter, and one must not be re-added.** An earlier version
+> gated on played-date; it was removed once the real cost was measured — the 30-minute
+> runtime was fork overhead per item, not API volume or item count. The fix was jq epoch
+> parsing plus the response-hash probe below. Re-introducing a date window would reduce
+> correctness (older items silently stop syncing) without meaningfully reducing runtime.
+
 Emby and Jellyfin servers configured per-host:
 
 ```bash
@@ -194,13 +209,24 @@ Show every file examined, not just those removed.
 ### play_state_sync.sh
 
 `play_state_sync.sh`
-Sync played/unplayed state and resume positions from local Emby to the remote Emby.
+Sync played/unplayed state, resume positions and favourites across **every** configured Emby
-Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced.
+and Jellyfin server — not just local→remote. Servers are discovered from every
+`HOST*_TRANSCODE_SERVERS` entry, with remote hosts' localhost URLs rewritten to their
+Tailscale IP. Newest `LastPlayedDate` wins; state only ever moves forward, never clears.
+
+No age filter — every matched item is considered on every run. The **change probe** is what
+keeps that cheap: the raw API responses are hashed and compared against the fingerprint from
+the last successful run, and per-item processing is skipped entirely when nothing moved.
+Fetches still happen every run, so nothing can be missed by the probe.
 
 `play_state_sync.sh --full`
-Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be
+Bypass the change probe and force the full per-item comparison even when the fingerprint
-slow on large libraries. Use after a new Emby install or database restore to rebuild
+matches. Use after a new Emby install or database restore, or when debugging a sync that
-full play history.
+appears to be skipping work it should be doing.
+
+`play_state_sync.sh --wait`
+Wait for an in-progress run instead of exiting. For manual runs that would otherwise be
+skipped by the scheduled every-30-minute pass. Used by `fallback.sh` during handback.
 
 `play_state_sync.sh --dry-run`
 Show what would be synced without writing any state.

Media/README-Media.md

@@ -100,3 +100,43 @@ syncs between them.
 | `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
 | `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
 | `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
+
+---
+
+## ━━━ THE ctime INVARIANT — READ BEFORE CHANGING PERMISSIONS ━━━
+
+`media_shares_permissions.sh` applies every pass **conditionally** — it touches only entries
+whose owner or mode is actually wrong. That is not an optimisation, and it must stay that way.
+
+`chown` and `chmod` rewrite an inode's ctime **even when the value does not change**. A
+blanket pass would therefore restamp every file in the library every night.
+
+The arr cleanup scripts (`sonarr_cleanup.sh`, `radarr_cleanup.sh`, `lidarr_cleanup.sh`) gate
+orphan deletion on ctime. mtime cannot substitute: an import preserves the release's original
+timestamp, so mtime says nothing about when a file arrived here. Measured 2026-07-27 — of 400
+files imported that week, **all 400 had mtimes over 7 days old, one of them 9613 days.**
+
+So:
+
+```
+  blanket chown/chmod  →  every ctime resets to today
+                       →  no file ever appears older than *_ORPHAN_AGE
+                       →  orphan collection silently stops
+                       →  nothing errors, nothing warns, disk just fills
+```
+
+**The failure is invisible.** No script fails, no notification fires. The only symptom is
+orphans quietly accumulating until a pool fills — which is exactly how the 755 GB / 89%-full
+cache pool incident happened.
+
+Two rules follow, and both are load-bearing:
+
+1. **`media_shares_permissions.sh` passes stay conditional.** Making any of them unconditional
+   breaks orphan collection ecosystem-wide.
+2. **`Tools/bulk_permissions_repair.sh` is unconditional by design** — it exists to repair
+   known-wrong paths where correctness beats preserving a clock. That is precisely why it is a
+   manual, targeted tool and not scheduled. Pointing it at a whole media root pauses orphan
+   collection there for `*_ORPHAN_AGE` days.
+
+Both scripts' headers carry this warning too. If you are reading this because you are about to
+"simplify" the permissions job, this is the thing that breaks.

Media/media_cleaner.sh

@@ -64,11 +64,45 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   acquire_lock "wait"  — wait if previous run still active
+# Root Enforcement
-#   detect_hosts()       — correct folder lists per host via MY_ID aliases
+#   Media files are owned by container users; deleting them requires root.
-#   Empty array guards   — warns and exits cleanly if no folders or patterns configured
+#
-#   Folder existence     — skips missing folders with warning, continues others
+# Profile Required
-#   platform_require_cmd  — notify script validated before use
+#   Exits with usage if no profile is given. There is no default profile — an
+#   unspecified profile must never fall through to cleaning something.
+#
+# Lock Acquisition
+#   acquire_lock "wait" — waits for a previous run to finish rather than
+#   skipping, so a long anime pass does not cause the media pass to be dropped.
+#
+# Host Detection
+#   detect_hosts() aliases HOST*_ANIME_CLEAN_FOLDERS / HOST*_MEDIA_CLEAN_FOLDERS
+#   to the correct host's values.
+#
+# Empty Array Guards
+#   Exits cleanly if the resolved folder list or pattern list is empty. An empty
+#   pattern list would otherwise build a find with no -iname terms and match
+#   every file in the tree.
+#
+# Clean Path Depth Guard
+#   Every folder must be an absolute path at least three levels deep before it is
+#   scanned. The patterns include *.sh, *.zip, *.rar and *.exe, so a truncated
+#   entry like /mnt/user — which passes an existence check — would delete
+#   matching files across every share on the array.
+#
+# Folder Existence
+#   Missing folders are skipped with a warning; remaining folders still process.
+#
+# Explicit Pattern List
+#   Only patterns named in ANIME_FILE_PATTERNS / MEDIA_FILE_PATTERNS are removed.
+#   The script never infers intent from file size, age, or location.
+#
+# Count Before Delete
+#   Matching files are counted first; a folder with zero matches short-circuits
+#   before any rm is constructed.
+#
+# Dry Run Support
+#   --dry-run lists every file that would be deleted and removes nothing.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -205,6 +239,19 @@ for FOLDER in "${CLEAN_FOLDERS[@]}"; do
     FOLDER_NAME=$(basename "$FOLDER")
     echo "━━━ $ICON_CLEAN $FOLDER_NAME ━━━"
 
+    # The pattern list includes *.sh, *.zip, *.rar and *.exe. A truncated entry such as
+    # /mnt/user passes the -d check below and would sweep every share on the array, so
+    # require an absolute path at least three levels deep before scanning anything.
+    _depth="${FOLDER//[^\/]/}"
+    if [[ -z "$FOLDER" || "$FOLDER" != /* || "${#_depth}" -lt 3 ]]; then
+        error "Refusing to clean unsafe path: '${FOLDER:-empty}' — expected an absolute path at least 3 levels deep"
+        notify "Media cleaner ($PROFILE) refused unsafe path on $(hostname): '${FOLDER:-empty}'" \
+            "Media Cleaner" "warning"
+        FAILED+=("${FOLDER_NAME:-empty}")
+        echo ""
+        continue
+    fi
+
     if [[ ! -d "$FOLDER" ]]; then
         warn "$FOLDER_NAME not found — skipping"
         SKIPPED+=("$FOLDER_NAME")

Media/media_shares_permissions.sh

@@ -14,6 +14,30 @@
 # or unRAID environment resets after updates.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# For each share in MEDIA_PERMISSION_SHARES:
+#
+#   1. Path safety and existence
+#        → unsafe or missing paths are refused or skipped, never scanned
+#
+#   2. Count wrong ownership (diagnostic)
+#        → find ! -user / ! -group — the number reported as "corrected"
+#
+#   3. Ownership pass — only if the count is non-zero
+#        → chown PERMISSIONS_OWNER on non-matching entries only
+#
+#   4. Directory mode pass
+#        → chmod PERMISSIONS_DIR_MODE on directories not already at that mode
+#
+#   5. File mode pass
+#        → chmod PERMISSIONS_FILE_MODE on files not already at that mode
+#        → "No such file" errors ignored: volatile dirs (Emby transcodes) race
+#
+# Every pass is conditional by design — see Conditional Passes below.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -35,16 +59,53 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   acquire_lock "wait"  — wait if previous run still active (large share scans)
+# Root Enforcement
-#   detect_hosts()       — correct share list per host via MY_ID aliases
+#   chown to an arbitrary owner requires root.
-#   Empty array guard    — warns and exits cleanly if no shares configured
+#
-#   Folder existence     — skips missing shares with warning, continues others
+# Lock Acquisition
-#   Separate passes      — directories and files chmod'd separately for correctness
+#   acquire_lock "wait" — waits rather than skipping. Share scans are long, and
-#   Conditional passes   — only entries whose owner/mode is actually wrong are touched.
+#   this runs first in the daily window; skipping it would let arr cleanup run
-#                          chown/chmod restamp ctime even when the value doesn't change,
+#   against uncorrected ownership.
-#                          and the arr cleanups gate orphan deletion on ctime
+#
-#   platform_require_cmd  — notify script validated before use
+# Host Detection
-#   Silent by default    — only failures produce output, success is silent
+#   detect_hosts() aliases HOST*_MEDIA_PERMISSION_SHARES to this host's shares.
+#
+# Empty Array Guard
+#   Exits cleanly if no shares are configured for this host.
+#
+# Share Path Depth Guard
+#   Every share must be an absolute path at least three levels deep before it is
+#   scanned. A truncated entry like /mnt/user passes an existence check and would
+#   chown and chmod every share on the array — which, because chown/chmod restamp
+#   ctime, would erase the age signal the arr cleanups depend on across the whole
+#   library in a single run.
+#
+# Folder Existence
+#   Missing shares are skipped with a warning; remaining shares still process.
+#
+# Separate Passes
+#   Directories and files are chmod'd in separate passes — directories need the
+#   execute bit for traversal, media files must not have it.
+#
+# Conditional Passes
+#   Only entries whose owner or mode is actually wrong are touched. This is not
+#   an optimisation: chown/chmod rewrite an inode's ctime even when the value is
+#   unchanged, so a blanket pass would restamp every file nightly and destroy
+#   ctime as an age signal. The arr cleanups gate orphan deletion on ctime, and
+#   mtime cannot substitute — imports preserve the release's original timestamp.
+#   Making any pass unconditional silently stops orphan collection.
+#
+# Transcode Race Tolerance
+#   "No such file or directory" errors from the file pass are ignored. Volatile
+#   directories such as Emby transcodes delete files mid-scan; that is expected,
+#   not a permissions failure.
+#
+# Dry Run Support
+#   --dry-run counts the dirs, files and ownership entries that would change and
+#   modifies nothing.
+#
+# Silent by Default
+#   Only failures and diagnostics produce output; a clean run is quiet.
 #
 # Diagnostic — high corrected count on every run means a container has wrong PUID/PGID:
 #   Correct values on unRAID: PUID=99 (nobody) PGID=100 (users)
@@ -153,6 +214,18 @@ PERMISSIONS_GROUP="${PERMISSIONS_OWNER##*:}"
 for SHARE in "${MEDIA_PERMISSION_SHARES[@]}"; do
     SHARE_NAME=$(basename "$SHARE")
 
+    # A truncated entry such as /mnt/user passes the -d check below and would chown/chmod
+    # every share on the array. Because chown/chmod restamp ctime, that would erase the age
+    # signal the arr cleanups gate orphan deletion on — across the whole library, in one run.
+    _depth="${SHARE//[^\/]/}"
+    if [[ -z "$SHARE" || "$SHARE" != /* || "${#_depth}" -lt 3 ]]; then
+        error "Refusing to touch unsafe path: '${SHARE:-empty}' — expected an absolute path at least 3 levels deep"
+        notify "Media permissions refused unsafe path on $(hostname): '${SHARE:-empty}'" \
+            "Media Permissions" "warning"
+        FAILED+=("${SHARE_NAME:-empty}")
+        continue
+    fi
+
     if [[ ! -d "$SHARE" ]]; then
         warn "$SHARE_NAME not found — skipping"
         SKIPPED+=("$SHARE_NAME")

Media/play_state_sync.sh

@@ -18,7 +18,7 @@
 #   Audio    → MusicBrainz Track ID
 #
 # ==============================================================================================
-# SYNC LOGIC
+# OPERATIONAL MODEL
 # ==============================================================================================
 #
 # For each matched item across ≥2 servers:
@@ -55,21 +55,71 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   PLAY_SYNC_ENABLED gate   — exits cleanly when disabled; no partial runs
+# Root Enforcement
-#   PARTNERSHIP gate         — skips remote sync when partnership is inactive
+#   The probe fingerprint is written under STATE_DIR, which is not user-writable.
-#   Per-server reachability  — unreachable servers are skipped individually;
+#   Without root the fingerprint silently fails to persist and the change probe
-#                              one offline server does not abort the entire sync
+#   never suppresses anything.
-#   User match required      — a user missing from a server is skipped for that
+#
-#                              server; no cross-account state pollution
+# jq Dependency Check
-#   acquire_lock             — prevents concurrent runs from racing on the same
+#   Exits if jq is missing. All API response parsing and the epoch comparisons
-#                              items during the 30-minute critical window
+#   depend on it — without jq every comparison would silently evaluate empty.
+#
+# PLAY_SYNC_ENABLED Gate
+#   Exits cleanly when disabled; no partial runs.
+#
+# PLAY_SYNC_REMOTE Gate
+#   When false, only this host's own servers are synced. Remote hosts are skipped
+#   before any network call is attempted.
+#
+# Partnership Gate
+#   Remote hosts are skipped when PARTNERSHIP_ENABLED=false. Local Emby↔Jellyfin
+#   sync still runs — a dormant partnership does not disable local work.
+#
+# Tailscale Resolution Guard
+#   A remote host whose Tailscale IP cannot be resolved is skipped rather than
+#   contacted at its literal localhost URL, which would otherwise point the sync
+#   at this host's own server and cross-contaminate state.
+#
+# Placeholder Credential Guard
+#   Servers whose API key is empty or still a placeholder are dropped from the
+#   list before any request is made.
+#
+# Per-Server Reachability
+#   Unreachable servers are skipped individually; one offline server does not
+#   abort the entire sync.
+#
+# User Match Required
+#   A user missing from a server is skipped for that server. State is never
+#   written to an unrelated account that happens to exist there.
+#
+# Forward-Only Writes
+#   The sync only pushes state forward — it never clears a Played flag or resets
+#   a resume position. The worst outcome of a bad comparison is a no-op, not
+#   erased watch history.
+#
+# Lock Acquisition
+#   acquire_lock prevents concurrent runs racing on the same items during the
+#   30 minute critical window. --wait switches from skip to wait for manual runs.
+#
+# Probe Staleness Ceiling
+#   PLAY_SYNC_PROBE_MAX_AGE_HOURS forces a full comparison regardless of the
+#   hash. Fetches happen every run either way, so the probe can only skip
+#   per-item processing — it can never cause a change to be missed outright.
+#
+# Dry Run Support
+#   --dry-run performs all comparisons and writes no state.
 #
 # ==============================================================================================
-# CONFIGURATION (host*.conf, aliased by detect_hosts)
+# CONFIGURATION
 # ==============================================================================================
 #
+# host*.conf
+#
 #   HOST*_TRANSCODE_SERVERS  "Name|URL|APIKey|type" entries per host (emby/jellyfin)
-#                            All hosts are discovered automatically — no extra config needed.
+#                            Read directly for every HOST[0-9]+ defined — this script
+#                            deliberately does NOT call detect_hosts(), because it needs
+#                            every host's servers, not just this one's. Self is identified
+#                            by comparing HOST* values against hostname -s.
 #                            Remote host URLs have localhost rewritten to their Tailscale IP.
 #
 # master.conf
@@ -103,6 +153,10 @@
 # play_state_sync.sh --full
 #     Bypass the change probe — always run the full comparison.
 #
+# play_state_sync.sh --wait
+#     Wait for an in-progress run to finish instead of exiting. For manual runs
+#     that would otherwise be skipped by the every-30-minute scheduled pass.
+#
 # play_state_sync.sh --log
 #     Verbose output — show each item comparison.
 #
@@ -129,6 +183,13 @@ parse_args "${_FILTERED[@]}"
 # ==============================================================================================
 # ━━━ Setup ━━━
 # ==============================================================================================
+# The probe fingerprint lives under STATE_DIR — without root it silently fails to persist
+# and the change probe can never suppress a run.
+if [[ "$EUID" -ne 0 ]]; then
+    error "Must be run as root"
+    exit 1
+fi
+
 [[ "${PLAY_SYNC_ENABLED:-true}" != "true" ]] && echo "Play state sync disabled" && exit 0
 
 SYNC_TYPES="${PLAY_SYNC_TYPES:-Movie,Episode}"

Monitors/Manual-Monitors.md

@@ -230,7 +230,7 @@ data — it just doesn't trigger a notification for that condition.
 | State File | Source | What It Shows |
 |-----------|--------|---------------|
 | `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
-| `SYS_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
+| `DOCKER_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
 | `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts |
 | `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/stability_watchdog.sh` | Active system watchdog strikes |
 | `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |

Monitors/README-Monitors.md

@@ -153,7 +153,7 @@ Daily 8am:
   weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE
                           ── reads ──────────► WATCHDOG_STATE_FILE
                           ── reads ──────────► SYS_WATCHDOG_STATE_FILE
-                          ── reads ──────────► SYS_WATCHDOG_FAILED_FILE
+                          ── reads ──────────► DOCKER_WATCHDOG_FAILED_FILE
                           ── reads ──────────► BANDWIDTH_LOG
                           ── reads ──────────► TRANSCODE_DAILY_LOG
                           ── reads ──────────► TUNING_MONITOR_LOG

Monitors/backup_verify.sh

@@ -19,6 +19,24 @@
 # to HOST*_DAILY_SYNC_SHARES. Both aliased by detect_hosts().
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Pre-flight — connectivity to the remote, remote array mounted, version parity
+#   2. Resolve the share list (BACKUP_VERIFY_SHARES, else DAILY_SYNC_SHARES)
+#   3. Per share:
+#        a. Randomly sample BACKUP_VERIFY_SAMPLE files above BACKUP_VERIFY_MIN_SIZE
+#        b. Compute each file's MD5 locally
+#        c. Compute the same file's MD5 on the remote over SSH
+#        d. Classify: MATCH | MISMATCH | MISSING
+#   4. Report per-share and overall counts; notify on any MISMATCH and on
+#      significant MISSING counts
+#
+# Sampling rather than full verification is deliberate — a complete checksum of every
+# mirrored file would take longer than the interval between runs. Random sampling over
+# a weekly cadence surfaces systematic corruption without ever reading the whole library.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -56,9 +74,6 @@
 # SSH Timeout
 #   SSH_TIMEOUT caps all SSH calls. One hung connection does not block the run.
 #
-# Notification Validated
-#   platform_require_cmd confirms the notify script is present before use.
-#
 # ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
@@ -107,6 +122,7 @@ source "$SCRIPT_DIR/../load_config.sh"
 parse_args "$@"
 
 SSH_TIMEOUT=15
+BACKUP_VERIFY_MD5_TIMEOUT_MAX="${BACKUP_VERIFY_MD5_TIMEOUT_MAX:-600}"
 
 # ==============================================================================================
 # ━━━ Setup ━━━
@@ -213,6 +229,7 @@ TOTAL_CHECKED=0
 TOTAL_MATCH=0
 TOTAL_MISMATCH=0
 TOTAL_MISSING=0
+TOTAL_UNVERIFIED=0
 SHARES_WITH_ISSUES=()
 
 for share in "${VERIFY_SHARES[@]}"; do
@@ -250,6 +267,7 @@ for share in "${VERIFY_SHARES[@]}"; do
     SHARE_MATCH=0
     SHARE_MISMATCH=0
     SHARE_MISSING=0
+    SHARE_UNVERIFIED=0
 
     for local_file in "${SAMPLE_FILES[@]}"; do
         [[ -z "$local_file" ]] && continue
@@ -261,19 +279,49 @@ for share in "${VERIFY_SHARES[@]}"; do
             continue
         fi
 
-        # Remote checksum via SSH — timeout protected
+        # The path is interpolated into a remote shell command, so it must be escaped for
-        remote_md5=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+        # reuse as one word. A bare '$local_file' inside single quotes breaks on the first
+        # apostrophe — "Frieren - Beyond Journey's End" ended the quote early, md5sum fell
+        # back to reading stdin, and the empty-input hash d41d8cd9... was reported as a
+        # MISMATCH against a file that is byte-identical on the remote.
+        printf -v remote_q '%q' "$local_file"
+
+        # Existence and content are separate questions. Asking them together means a slow
+        # checksum is indistinguishable from an absent file.
+        remote_exists=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
             -o ConnectTimeout="$SSH_TIMEOUT" \
             -o StrictHostKeyChecking=no \
             root@"$REMOTE_SERVER" \
-            "md5sum '$local_file' 2>/dev/null | awk '{print \$1}'" 2>/dev/null)
+            "test -f $remote_q && echo yes" 2>/dev/null </dev/null)
 
         (( TOTAL_CHECKED++ ))
 
-        if [[ -z "$remote_md5" ]]; then
+        if [[ "$remote_exists" != "yes" ]]; then
             warn "$ICON_ERROR MISSING: $(basename "$local_file")"
             (( SHARE_MISSING++ ))
             (( TOTAL_MISSING++ ))
+            continue
+        fi
+
+        # md5sum of a multi-GB file cannot finish inside a connect-sized timeout. Budget by
+        # size — a 5.9GB file needs ~30s and was being killed at 15s, then counted MISSING
+        # even though it was present and correct.
+        local_size=$(stat -c%s "$local_file" 2>/dev/null || echo 0)
+        md5_timeout=$(( local_size / 52428800 + SSH_TIMEOUT ))
+        (( md5_timeout > BACKUP_VERIFY_MD5_TIMEOUT_MAX )) && md5_timeout=$BACKUP_VERIFY_MD5_TIMEOUT_MAX
+
+        remote_md5=$(timeout "$md5_timeout" ssh -i "$SSH_KEY" \
+            -o ConnectTimeout="$SSH_TIMEOUT" \
+            -o StrictHostKeyChecking=no \
+            root@"$REMOTE_SERVER" \
+            "md5sum $remote_q 2>/dev/null | awk '{print \$1}'" 2>/dev/null </dev/null)
+
+        if [[ -z "$remote_md5" ]]; then
+            # Present but unreadable within budget. Reporting this as a mismatch or a miss
+            # would be a claim the run did not earn.
+            warn "$ICON_WARN UNVERIFIED (checksum timed out after ${md5_timeout}s): $(basename "$local_file")"
+            (( SHARE_UNVERIFIED++ ))
+            (( TOTAL_UNVERIFIED++ ))
         elif [[ "$local_md5" == "$remote_md5" ]]; then
             log "MATCH: $(basename "$local_file")"
             (( SHARE_MATCH++ ))
@@ -288,8 +336,8 @@ for share in "${VERIFY_SHARES[@]}"; do
     done
 
     # Per-share result — only visible if issues found
-    if [[ "$SHARE_MISMATCH" -gt 0 || "$SHARE_MISSING" -gt 0 ]]; then
+    if [[ "$SHARE_MISMATCH" -gt 0 || "$SHARE_MISSING" -gt 0 || "$SHARE_UNVERIFIED" -gt 0 ]]; then
-        warn "$SHARE_NAME — match: $SHARE_MATCH  missing: $SHARE_MISSING  mismatch: $SHARE_MISMATCH"
+        warn "$SHARE_NAME — match: $SHARE_MATCH  missing: $SHARE_MISSING  mismatch: $SHARE_MISMATCH  unverified: $SHARE_UNVERIFIED"
         SHARES_WITH_ISSUES+=("$SHARE_NAME")
     else
         echo "$SHARE_NAME — all $SHARE_MATCH files match ✅"
@@ -310,10 +358,11 @@ echo "$ICON_VERIFY  Checked:   $TOTAL_CHECKED files"
 echo "$ICON_TIME    Duration:  $(format_duration $(( END - START )))"
 echo ""
 
-if [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 ]]; then
+if [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 || "$TOTAL_UNVERIFIED" -gt 0 ]]; then
     echo "$ICON_SUCCESS Match:      $TOTAL_MATCH"
     warn "Missing:    $TOTAL_MISSING"
     [[ "$TOTAL_MISMATCH" -gt 0 ]] && echo "$ICON_ERROR   Mismatch:   $TOTAL_MISMATCH"
+    [[ "$TOTAL_UNVERIFIED" -gt 0 ]] && warn "Unverified: $TOTAL_UNVERIFIED (present, checksum timed out)"
 fi
 
 if [[ "$DRY_RUN" == true ]]; then
@@ -322,6 +371,8 @@ elif [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 ]]; then
     echo "$ICON_ERROR Status: ISSUES FOUND — ${#SHARES_WITH_ISSUES[@]} share(s) need attention: ${SHARES_WITH_ISSUES[*]}"
     notify "Backup verify FAILED on $(hostname) → $REMOTE_SERVER_NAME — mismatches: $TOTAL_MISMATCH missing: $TOTAL_MISSING — shares: ${SHARES_WITH_ISSUES[*]}" \
         "Backup Verify" "warning"
+elif [[ "$TOTAL_UNVERIFIED" -gt 0 ]]; then
+    warn "Status: $TOTAL_MATCH verified, $TOTAL_UNVERIFIED could not be checksummed in time — NOT a clean run"
 else
     echo "$ICON_DONE Status: all $TOTAL_CHECKED files match across ${#VERIFY_SHARES[@]} shares ✅"
 fi

Monitors/bandwidth_monitor.sh

@@ -17,6 +17,24 @@
 # last 7 days activity timeline, any transfers or days exceeding BANDWIDTH_WARN_GB.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Log mode (--log-transfer), called by rsync.sh after every sync:
+#   1. Append one line: YYYY-MM-DD|HH:MM|profile|duration|status|bytes
+#   2. Trim entries older than BANDWIDTH_LOG_RETENTION days
+#   One bounded write per rsync run — never grows without limit, never rewrites history.
+#
+# Report mode (default), scheduled weekly:
+#   1. Read the accumulated log
+#   2. Aggregate per profile — run count, total bytes, average duration, failures
+#   3. Build a 7-day activity timeline
+#   4. Flag any single transfer or any single day exceeding BANDWIDTH_WARN_GB
+#
+# The two modes never run together: logging is a side effect of rsync, reporting is a
+# scheduled read. Report mode never writes to the log.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -46,9 +64,6 @@
 # Log Directory Guard
 #   Creates the log directory if it doesn't exist. Exits cleanly if unwritable.
 #
-# Notification Validated
-#   platform_require_cmd confirms the notify script is present before use.
-#
 # ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================

Monitors/cert_monitor.sh

@@ -15,6 +15,23 @@
 # separate message lists all CRITICAL domains. Not one notification per domain.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Validate openssl is present (platform_require_cmd) — without it nothing can be checked
+#   2. Per domain in CERT_MONITOR_DOMAINS:
+#        a. Open a real TLS connection with openssl s_client
+#        b. Parse notAfter from the served certificate
+#        c. Compute days remaining
+#        d. Classify: HEALTHY (silent) | WARNING (≤ CERT_WARN_DAYS)
+#                     | CRITICAL (≤ CERT_CRIT_DAYS) | FAILED (no connect / no parse)
+#   3. Batch by severity — one notification listing all WARNING domains, a separate
+#      one listing all CRITICAL domains
+#
+# A domain that fails to connect is reported as FAILED rather than assumed healthy or
+# assumed expired — an unreachable host and an expiring cert are different problems.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -44,8 +61,10 @@
 #   CERT_TIMEOUT caps each openssl connection attempt. One unreachable domain
 #   does not block the remaining domains.
 #
-# Notification Validated
+# openssl Validated
-#   platform_require_cmd confirms openssl and notify script are present before use.
+#   platform_require_cmd confirms openssl is present before any domain is checked — every
+#   check depends on it, so a missing binary is reported as itself rather than as every
+#   domain failing. The notify script is validated separately by the platform adapter.
 #
 # ==============================================================================================
 # CONFIGURATION

Monitors/emby_session_report.sh

@@ -18,6 +18,22 @@
 # configuration issue. Silent on clean runs.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Reachability — Emby API responding; unreachable exits cleanly rather than
+#      reporting an empty library as a real result
+#   2. Server info and uptime
+#   3. Active sessions — count, and the transcode-to-direct-play ratio
+#   4. Library counts — movies, episodes, songs
+#   5. Activity history over the last EMBY_REPORT_DAYS
+#   6. Top EMBY_REPORT_TOP_N items and most active users
+#   7. Ramdisk transcode status, read from the shared transcode state
+#
+# Every figure is queried fresh. The only notification is the transcode-ratio warning;
+# everything else is report output.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -54,9 +70,6 @@
 #   detect_hosts() aliases HOST*_EMBY_URL and HOST*_EMBY_API_KEY → EMBY_URL / EMBY_API_KEY.
 #   Each server reports on its own Emby instance automatically.
 #
-# Notification Validated
-#   platform_require_cmd confirms the notify script is present before use.
-#
 # ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================

Monitors/mesh_monitor.sh

@@ -53,6 +53,13 @@
 #   collect_hosts() populates ALL_HOST_IDS — if no HOST* vars are defined the
 #   output sections iterate over an empty array and exit cleanly.
 #
+# No Root, No Lock, No detect_hosts — Deliberate
+#   This is the one script in the ecosystem that intentionally omits all three, and
+#   they should not be added. It writes nothing, so there is no state for a lock to
+#   protect and no privileged operation to justify a root gate. It reports on every
+#   node rather than acting as one, so detect_hosts() would narrow it to this host's
+#   aliases — the opposite of what it is for. Every HOST* var is read directly instead.
+#
 # ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================

Monitors/smart_health.sh

@@ -19,6 +19,26 @@
 # from master.conf if dynamix.cfg is not found.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Validate smartctl is present (platform_require_cmd)
+#   2. Resolve temperature thresholds — dynamix.cfg first, master.conf as fallback
+#   3. Enumerate drives, skipping anything in SMART_IGNORE_DRIVES
+#   4. Per drive, read live SMART attributes and evaluate:
+#        overall status              FAILED           → critical
+#        Reallocated_Sector_Ct  > 0                   → concerning
+#        Current_Pending_Sector > 0                   → concerning
+#        Offline_Uncorrectable  > 0                   → critical
+#        Temperature_Celsius    vs warn/crit thresholds
+#        Power_On_Hours                               → informational only
+#      NVMe drives expose different attribute names — detected and mapped automatically.
+#   5. Report; notify only when something crosses a threshold. Silent when all pass.
+#
+# Read-only throughout — this queries attributes the drive already maintains and never
+# starts a self-test. Running one is smart_long_test.sh's job.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -55,8 +75,9 @@
 #   Reads hot/max/hotssd/maxssd from dynamix.cfg so smart_health.sh and unRAID's
 #   dashboard use the same thresholds. Falls back to master.conf values if not found.
 #
-# Notifications Validated
+# smartctl Validated
-#   platform_require_cmd confirms smartctl and notify script are present before use.
+#   platform_require_cmd confirms smartctl is present before any drive is queried. The
+#   notify script is validated separately by the platform adapter.
 #
 # ==============================================================================================
 # CONFIGURATION

Monitors/system_tuning_monitor.sh

@@ -18,6 +18,22 @@
 # WebGUI slowdowns or timeouts under load.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Each run snapshots:
+#   inotify: instances in use vs INOTIFY_MAX_INSTANCES kernel limit.
+#            Top 5 consumers by instance count. Warns above INOTIFY_WARN_PCT.
+#   php-fpm: active workers vs PHP_MAX_CHILDREN limit.
+#            Warns above PHP_FPM_WARN_PCT.
+#
+# Log line format (one per run, trimmed to TUNING_LOG_RETENTION days):
+#   DATE|TIME|INOTIFY_USED|INOTIFY_LIMIT|INOTIFY_PCT|INOTIFY_WARN|
+#   PHPFPM_ACTIVE|PHPFPM_MAX|PHPFPM_PCT|PHPFPM_WARN
+# INOTIFY_WARN and PHPFPM_WARN are 1/0 flags. weekly_health_digest.sh counts
+# warnings over the week to show trend severity.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -36,22 +52,6 @@
 #   signal — routine snapshots below the threshold produce nothing.
 #
 # ==============================================================================================
-# OPERATIONAL MODEL
-# ==============================================================================================
-#
-# Each run snapshots:
-#   inotify: instances in use vs INOTIFY_MAX_INSTANCES kernel limit.
-#            Top 5 consumers by instance count. Warns above INOTIFY_WARN_PCT.
-#   php-fpm: active workers vs PHP_MAX_CHILDREN limit.
-#            Warns above PHP_FPM_WARN_PCT.
-#
-# Log line format (one per run, trimmed to TUNING_LOG_RETENTION days):
-#   DATE|TIME|INOTIFY_USED|INOTIFY_LIMIT|INOTIFY_PCT|INOTIFY_WARN|
-#   PHPFPM_ACTIVE|PHPFPM_MAX|PHPFPM_PCT|PHPFPM_WARN
-# INOTIFY_WARN and PHPFPM_WARN are 1/0 flags. weekly_health_digest.sh counts
-# warnings over the week to show trend severity.
-#
-# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
@@ -65,9 +65,6 @@
 #   Trim uses tmp file + mv — partial writes during log rotation cannot corrupt
 #   the accumulated history.
 #
-# Notification Validated
-#   platform_require_cmd confirms the notify script is present before use.
-#
 # ==============================================================================================
 # STATE FILES
 # ==============================================================================================
@@ -212,7 +209,7 @@ fi
 # ==============================================================================================
 PHPFPM_MAX="${PHP_MAX_CHILDREN:-250}"
 
-PHPFPM_ACTIVE=$(ps aux 2>/dev/null | grep -c "php-fpm: pool" || echo 0)
+PHPFPM_ACTIVE=$(ps aux 2>/dev/null | grep -c "php-fpm: pool" || true)
 PHPFPM_ACTIVE="${PHPFPM_ACTIVE//[^0-9]/}"
 PHPFPM_ACTIVE="${PHPFPM_ACTIVE:-0}"
 

Monitors/uptime_report.sh

@@ -0,0 +1,93 @@
+#!/bin/bash
+# ==============================================================================================
+# ================================== Uptime Report =============================================
+# ==============================================================================================
+#
+# PURPOSE
+# ─────────────────────────────────────────────────────────────────────────────
+# The weekly read of what Tools/uptime_probe.sh has been recording every minute: anything down
+# right now, and anything that was not perfect over the last seven days. Runs in the Sunday
+# Morning Coffee Report.
+#
+# Silent on a clean week. A report that always says something is a report nobody reads, so this
+# prints nothing and notifies nothing when every domain was 100%.
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# uptime_probe.php --report exits 1 when it has something to say and 0 when it does not, so the
+# decision to notify is the exit code rather than this script parsing the text it just printed.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Silence is the normal output.
+#   A report that always says something is a report nobody reads. A perfect week prints nothing
+#   and notifies nothing, so anything that does appear in the Sunday report is worth the glance.
+#
+# The exit code is the decision, not the text.
+#   uptime_probe.php --report exits 1 when it has something to say and 0 when it does not. This
+#   script never parses the output it just printed to work out whether to notify — a report whose
+#   wording changed would otherwise silently stop notifying.
+#
+# It reads; it never probes.
+#   The measurements are already taken, once a minute, by Tools/uptime_probe.sh. Re-probing at
+#   report time would describe Sunday morning rather than the week being reported on.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Read-only. Reads the stored history and prints; records nothing, and cannot alter the data it
+# is reporting on.
+#
+# UPTIME_PROBE_ENABLED gates the whole run — with the probe off there is no history worth
+# reporting, and this says nothing rather than reporting an empty week as a perfect one.
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# uptime_report.sh    the weekly read. Silent when every domain was 100%.
+#
+# Called from COFFEE_REPORT_SCRIPTS; takes no arguments and has no other mode. For live figures
+# or a per-domain table, use Tools/uptime_probe.sh --status.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# UPTIME_PROBE_ENABLED   nothing here runs when the probe is switched off
+#
+# ==============================================================================================
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/../load_config.sh"
+
+acquire_lock
+detect_hosts
+
+if [[ "${UPTIME_PROBE_ENABLED:-true}" == "false" ]]; then
+    log "$ICON_GEAR Uptime probe disabled — nothing to report"
+    exit 0
+fi
+
+REPORT="$(php "$SCRIPT_DIR/../Plugin/unraid/Tools/uptime_probe.php" --report 2>/dev/null)"
+RC=$?
+
+if [[ $RC -eq 0 || -z "$REPORT" ]]; then
+    echo "$ICON_DONE All monitored domains at 100% this week ✅"
+    exit 0
+fi
+
+echo "$REPORT"
+
+DOWN_COUNT=$(grep -c "DOWN" <<< "$REPORT" || true)
+if [[ "$DOWN_COUNT" -gt 0 ]]; then
+    notify "$DOWN_COUNT domain(s) currently unreachable on $(hostname)" "Uptime" "warning"
+else
+    notify "Some domains had downtime this week on $(hostname)" "Uptime" "normal"
+fi
+exit 0

Monitors/weekly_health_digest.sh

@@ -14,24 +14,6 @@
 # into a single digest. Reads only — writes nothing, changes nothing.
 #
 # ==============================================================================================
-# DESIGN PRINCIPLES
-# ==============================================================================================
-#
-# Aggregator, Not Generator
-#   This script reads state files that other scripts maintain. It never produces
-#   health data itself — it only presents what is already there. Each source
-#   script remains responsible for its own state; this script is the envelope.
-#
-# Profile-Driven Notification
-#   The cron schedule never changes. The DIGEST_PROFILE in master.conf controls
-#   when notifications actually send — switching from daily noise to weekly
-#   summaries is a one-line conf change, not a cron edit.
-#
-# Read-Only, No Side Effects
-#   Writes nothing, changes nothing, triggers nothing. Safe to run at any time
-#   for a health snapshot without affecting any running service or state file.
-#
-# ==============================================================================================
 # OPERATIONAL MODEL
 # ==============================================================================================
 #
@@ -58,6 +40,24 @@
 #   RAMDISK_PATH / TRANSCODE_LINK — current transcode location and usage
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Aggregator, Not Generator
+#   This script reads state files that other scripts maintain. It never produces
+#   health data itself — it only presents what is already there. Each source
+#   script remains responsible for its own state; this script is the envelope.
+#
+# Profile-Driven Notification
+#   The cron schedule never changes. The DIGEST_PROFILE in master.conf controls
+#   when notifications actually send — switching from daily noise to weekly
+#   summaries is a one-line conf change, not a cron edit.
+#
+# Read-Only, No Side Effects
+#   Writes nothing, changes nothing, triggers nothing. Safe to run at any time
+#   for a health snapshot without affecting any running service or state file.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
@@ -76,8 +76,10 @@
 #   smart profile produces no output and no notification when nothing worth
 #   reporting is found.
 #
-# Notifications Validated
+# openssl Validated — Non-Fatal
-#   platform_require_cmd confirms notify and openssl are present before use.
+#   platform_require_cmd checks openssl and, unlike the other monitors, only warns if it
+#   is missing: the SSL section is skipped and the rest of the digest still runs. The
+#   notify script is validated separately by the platform adapter.
 #
 # ==============================================================================================
 # CONFIGURATION

Monitors/zfs_memory_snapshot.sh

@@ -15,6 +15,24 @@
 # comparison. In --dry-run mode, console only — nothing written to the log.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Five report sections (each skips gracefully if its data source is unavailable):
+#
+#   ZFS pool health   — status, state, errors per pool. Pools in
+#                       ZFS_REPORT_IGNORE_POOLS excluded from the report
+#                       (still fully monitored by unRAID — report-only exclusion).
+#   ARC statistics    — current ARC vs max, metadata pressure, hit rate.
+#                       Warns if ARC utilisation exceeds ZFS_REPORT_ARC_WARN_PCT, or if
+#                       ARC headroom (max - current) drops below ZFS_REPORT_ARC_FREE_WARN_GB.
+#   Memory status     — total, free, available RAM (informational only — see note below).
+#                       Warns if available < ZFS_REPORT_AVAIL_WARN_GB.
+#   Docker memory     — top ZFS_REPORT_DOCKER_TOP containers by memory usage.
+#                       Useful for spotting containers approaching watchdog limits.
+#   Kernel pressure   — vmstat snapshot (3 samples).
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -42,24 +60,6 @@
 #   reclaimable cache) is still checked separately as a true system-pressure signal.
 #
 # ==============================================================================================
-# OPERATIONAL MODEL
-# ==============================================================================================
-#
-# Five report sections (each skips gracefully if its data source is unavailable):
-#
-#   ZFS pool health   — status, state, errors per pool. Pools in
-#                       ZFS_REPORT_IGNORE_POOLS excluded from the report
-#                       (still fully monitored by unRAID — report-only exclusion).
-#   ARC statistics    — current ARC vs max, metadata pressure, hit rate.
-#                       Warns if ARC utilisation exceeds ZFS_REPORT_ARC_WARN_PCT, or if
-#                       ARC headroom (max - current) drops below ZFS_REPORT_ARC_FREE_WARN_GB.
-#   Memory status     — total, free, available RAM (informational only — see note below).
-#                       Warns if available < ZFS_REPORT_AVAIL_WARN_GB.
-#   Docker memory     — top ZFS_REPORT_DOCKER_TOP containers by memory usage.
-#                       Useful for spotting containers approaching watchdog limits.
-#   Kernel pressure   — vmstat snapshot (3 samples).
-#
-# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
@@ -79,9 +79,6 @@
 # Docker Stats Timeout
 #   DOCKER_TIMEOUT caps docker stats calls. A hung daemon does not block the report.
 #
-# Notification Validated
-#   platform_require_cmd confirms the notify script is present before use.
-#
 # ==============================================================================================
 # STATE FILES
 # ==============================================================================================
@@ -264,8 +261,17 @@ echo "━━━ $ICON_ZFS ARC Statistics ━━━"
 if [[ ! -f /proc/spl/kstat/zfs/arcstats ]]; then
     warn "ZFS arcstats not available — skipping ARC section"
 else
-    ARC_MAX=$(cat /sys/module/zfs/parameters/zfs_arc_max 2>/dev/null || \
+    # zfs_arc_max reads 0 when it has been left at the default, which is a value rather than a
-              awk '/^c_max / {print $3}' /proc/spl/kstat/zfs/arcstats)
+    # failure — so the || fallback never fires for the case that actually needs it, exactly like a
+    # grep -c that prints 0 and exits 1. Zero here would reach the ARC_PCT division below, and awk
+    # treats division by zero as fatal: it prints nothing, ARC_PCT comes back empty, and the whole
+    # ARC section reports blanks. c_max is the cap the kernel is really enforcing either way.
+    ARC_MAX=$(cat /sys/module/zfs/parameters/zfs_arc_max 2>/dev/null || echo 0)
+    ARC_MAX="${ARC_MAX//[^0-9]/}"
+    if [[ "${ARC_MAX:-0}" -eq 0 ]]; then
+        ARC_MAX=$(awk '/^c_max / {print $3}' /proc/spl/kstat/zfs/arcstats 2>/dev/null)
+        ARC_MAX="${ARC_MAX:-0}"
+    fi
     ARC_SIZE=$(awk '/^size / {print $3}' /proc/spl/kstat/zfs/arcstats)
     ARC_META_USED=$(awk '/^arc_meta_used / {print $3}' /proc/spl/kstat/zfs/arcstats)
 

Notes_AI-Design.md

@@ -0,0 +1,559 @@
+# Varaverk AI Integration — Design Notes
+
+**Status: retrieval is built; integration is not.** As of 2026-08-02 the `AI_*` and
+`HOST*_OLLAMA_*` variables exist in both confs and both templates, and `AI/` holds a working
+index and query path — see the RAG section at the end of this document and `AI/README-AI.md`.
+
+Everything else below remains design only. **No Varaverk script consults AI.** Every
+`AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false with an empty whitelist, and
+host resolution across the mesh is specified but not implemented. Originally captured
+2026-08-01 so the reasoning survives.
+
+Ollama itself *is* installed, tuned and verified on HOST1 — `qwen2.5-coder:14b` for
+generation, `nomic-embed-text` for embeddings, 16k context, pinned to the RTX 3080. See
+Hardware Budget for measured numbers. That is the substrate, not the integration.
+
+**Origin:** the RTX 3080 was freed when the Windows gaming VM was retired. It is bound to the
+`nvidia` driver, not `vfio` — not reserved for passthrough, so there is no VM contention to
+design around. Two goals at once: somewhere to learn local LLMs, and something Varaverk can
+genuinely use.
+
+**Build order** — deliberately lowest-risk first. Each stage must be boring before the next
+one starts:
+
+1. Chat assistant / settings helper / onboarding assistant — a wrong answer costs nothing
+2. Watchdog and discovery context — a wrong answer costs a bad suggestion, still gated
+3. Cleanup and sync decision aid — closest to destructive, last to be trusted
+
+The regular script is always the backup, at every stage.
+
+---
+
+## The governing principle
+
+**Varaverk works exactly as well with AI off as with it on.**
+
+Every script — including scripts written after this lands — is designed and hardened
+without AI first. AI is added afterwards as enhancement, never as a dependency. A script
+that cannot do its job when `AI_ENABLED=false` is a broken script, not an AI feature.
+
+This is the constraint everything else in this document answers to. If a design decision
+makes AI load-bearing, that decision is wrong.
+
+Corollary: AI never makes a destructive decision. The session that produced the current
+safeguard layer (depth guards, strike thresholds, verification-after-write) exists because
+config values and scan results feed `rm -rf`, `chown -R`, and rsync `--delete`. AI advises
+at the points where a script currently stops and defers to a human. The deterministic guard
+still pulls the trigger.
+
+---
+
+## Two independent off-switches
+
+| switch | meaning | source |
+|---|---|---|
+| `AI_ENABLED` | intent — do we want AI at all | `master.conf` |
+| resolver result | availability — is there a reachable host | runtime probe |
+
+**Both must produce the identical code path when off.** A caller that gets "no AI" from
+either must run its normal, non-AI logic — not a degraded variant, not a skipped step.
+
+`AI_ENABLED` follows the fail-closed idiom standardised across the ecosystem:
+
+```bash
+[[ "${AI_ENABLED:-false}" != "true" ]] && <normal path>
+```
+
+Not `== false`. Anything that isn't exactly `true` means off, so a typo can never switch AI
+on. (This is the same bug that was fixed in `fallback.sh` — see its `FALLBACK_ENABLED` gate.)
+
+---
+
+## Configuration schema
+
+Follows the existing rule: **thresholds and toggles → `master.conf`; hardware, paths,
+container names and per-host identity → `host*.conf`.**
+
+### `host*.conf` — per-host, because only one node actually has the GPU
+
+```bash
+# ━━━ Ollama / AI ━━━
+    HOST1_OLLAMA_URL="http://localhost:11434"   # empty on nodes without a local Ollama
+    HOST1_OLLAMA_CONTAINER="Ollama"             # for docker_watchdog / restart lists
+    HOST1_OLLAMA_GPU_UUID="GPU-309357d8-2a13-09e0-84ac-fcfdcdf5c626"
+    HOST1_OLLAMA_MODEL="qwen2.5-coder:14b"      # generation
+    HOST1_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — qwen cannot embed
+```
+
+A node with an empty `OLLAMA_URL` is not an error — it falls through to the resolver and
+uses the mesh. HOST2 gets the section with blanks, exactly like the NPM/lldap credentials
+it is already waiting on.
+
+### `master.conf` — shared behaviour
+
+```bash
+# ━━━ AI ━━━
+    AI_ENABLED=false                # master switch — fail-closed, != "true" means off
+    AI_CONNECT_TIMEOUT=5            # probe timeout when resolving a host
+    AI_REQUEST_TIMEOUT=240          # must clear a cold load — measured 1m45s after tuning.
+                                    # KEEP_ALIVE=-1 means this only bites after a restart,
+                                    # but a first call that times out is the worst first
+                                    # impression a caller can have. Re-measure against a
+                                    # real RAG query before fixing this number.
+    AI_RESOLVE_CACHE_TTL=300        # don't re-probe the mesh on every script invocation
+    AI_MAX_RETRIES=1                # AI is enhancement — do not retry hard
+
+    # Per-feature toggles — enable narration long before enabling decision aid
+    AI_ASSIST_REPORTS=false         # tier 1 — digest / coffee report narration
+    AI_ASSIST_WATCHDOG=false        # tier 2 — context on a flagged condition
+    AI_ASSIST_DISCOVERY=false       # tier 2 — discovery / classification judgment calls
+    AI_ASSIST_CLEANUP=false         # tier 2 — HELD orphans, stuck-import triage
+    AI_ASSIST_ONBOARD=false         # tier 3 — onboarding / settings assistance
+
+    # Conf writes — separate switch, off by default, see Conf Write Access below
+    AI_CONF_WRITE_ENABLED=false
+    AI_CONF_WRITE_KEYS=()           # explicit whitelist; never paths or credentials
+```
+
+**Per-feature toggles are load-bearing, not decoration.** They are what lets AI narrate the
+weekly digest for months before it is ever allowed near a cleanup decision. `AI_ENABLED` is
+necessary but not sufficient — every feature stays individually off until it has earned it.
+
+> When these land, `Deployment/master.conf.template` and `Deployment/host.conf.template`
+> must be updated in the same pass. That rule is not optional in this repo.
+
+---
+
+## Host resolution
+
+AI runs on the owner's node only. Remote mesh nodes reach it over Tailscale. No remote node
+needs a GPU, a model, or an Ollama container — only the resolver.
+
+`resolve_ollama_host()` mirrors the existing Gitea locator in `git_pull_execute.sh`:
+
+```
+Ollama answering on localhost:11434?      → use it            (owner's node)
+  else discover_remote_nodes()
+       → resolve_tailscale_ip(node)
+       → probe each :11434                → first responsive wins
+  else                                    → no AI host        (== AI_ENABLED=false)
+```
+
+Helpers already exist in `common.sh`: `discover_remote_nodes()` (767),
+`resolve_tailscale_ip()` (812), `check_connectivity()` (866).
+
+**Probe the API, not the container.** The Gitea locator checks `docker ps`. Do not copy that
+here — a container can be up while the model is unloaded, still pulling, or wedged. Probe
+`/api/tags`. Same principle written into `network_watchdog.sh`'s design principles:
+*verify the path, not the process*.
+
+**Cache the resolution** in `/tmp` state, like the arr cache. A cleanup script should not pay
+a Tailscale round-trip to discover AI it may never call.
+
+### Known consequence
+
+AI lives on HOST1, so **during a fallback — HOST1 down, HOST2 covering — the mesh has no AI.**
+That is precisely when a triage assistant would be most useful. Accepted: a second GPU on
+HOST2 is a lot of hardware for that window, and AI is enhancement-only by design. Worth
+knowing rather than discovering.
+
+---
+
+## Where AI is allowed to act
+
+Ranked by how much damage a wrong answer does.
+
+**Tier 1 — narration and summary (safe, do first)**
+- Sunday morning coffee report — turn metrics into prose
+- `weekly_health_digest.sh` — summarise, highlight what changed
+- Explain *why* a container is crash-looping from its logs
+
+**Tier 2 — triage and context on an existing flag (the real value)**
+Places where a script already detects something and stops:
+- `system_watchdog` / `stability_watchdog` flags a condition → AI adds context, correlates
+  with recent logs, suggests likely cause
+- Sonarr stuck-import triage — the "matched by series ID" recipe is textbook LLM work
+- `HELD` entries from `arr_download_orphan_cleaner.sh`
+- `reverse-anime-leak` from the classification scans — currently report-only *because* it is
+  a judgment call. That is exactly the shape AI suits.
+
+**Tier 3 — assisted configuration (needs the guardrails below)**
+- Onboarding a new host — the main motivation for conf write access
+- AI-assisted settings tuning: rsync profiles, fallback tiers, auth stack
+
+**Never**
+- Deciding what to delete
+- Choosing a path for any destructive operation
+- Anything that bypasses a strike counter, age gate, or verification step
+
+---
+
+## Conf write access
+
+Wanted mainly for onboarding and assisted settings. This is the highest-risk item here.
+
+**Current state: there is no recovery path.**
+
+```
+.gitignore:4  Configurations/host*.conf
+.gitignore:5  Configurations/master.conf
+.gitignore:6  Configurations/*.bak
+```
+
+Confs are gitignored — no git history to revert to — and so are the `.bak` files, so the
+backup is not versioned either. The only fallback is a single `.bak` slot written by
+`conf_upgrade`, and it goes stale immediately:
+
+| file | modified | its `.bak` |
+|---|---|---|
+| `master.conf` | Jul 28 18:52 | Jul 28 18:52 |
+| `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** |
+
+A bad write to `master.conf` currently falls back to a file that may predate a month of edits.
+**Fix this before any AI writes anything.**
+
+### Required before conf-write ships
+
+1. **Key whitelist, not file access.** Thresholds and toggles only — `*_WARN_GB`,
+   `*_STRIKE_LIMIT`, `*_ENABLED`, retention days. Never a path, never a credential, never a
+   container list. A wrong threshold is recoverable; a wrong path is what the depth guards
+   exist to catch.
+2. **Timestamped backups, plural** — `master.conf.2026-08-01T21:00`, retained. Not one
+   clobbered slot.
+3. **Validate before commit** — `bash -n` the candidate, then confirm `load_config.sh`
+   sources it cleanly. Never install a conf that has not been proven to parse.
+4. **Diff always logged.** An AI conf change should be at least as visible as a container
+   restart.
+5. **Lock against concurrent readers** — never rewrite a conf while scripts are mid-run.
+
+**Consider un-ignoring `Configurations/` into a private repo.** Then `git diff` and
+`git revert` become the recovery mechanism and the history is free. This overlaps the
+existing GitHub-mirror TODO, which is already blocked on the same question.
+
+---
+
+## Security
+
+Ollama has **no authentication of any kind**, and its API includes `DELETE /api/delete`
+(wipe models) and `POST /api/pull` (fill the disk). It currently binds `0.0.0.0:11434` with
+`OLLAMA_ORIGINS=*` — reachable from the entire LAN, not just Tailscale.
+
+The design only needs loopback (owner) plus the Tailscale interface (mesh). `0.0.0.0` is
+strictly wider than required, for no benefit. Restrict to loopback + Tailscale, or use
+Tailscale ACLs to allow only mesh nodes. Node-level ACLs fit the mesh model better than
+app-level auth Ollama cannot provide anyway.
+
+**Done 2026-08-01:** `/ext-varaverk` is now mounted `ro` (was `rw` into live prod).
+Verified on the running container — `rw=false`.
+
+**Still open:** the LAN exposure above. Deliberately not folded into the tuning rebuild,
+since bind-address versus Tailscale ACL is a decision rather than a setting.
+
+---
+
+## Hardware budget
+
+RTX 3080, 10 GB, pinned to Ollama by UUID — isolated from the Quadro P2000 that Emby
+transcodes on. Do not let AI onto the P2000.
+
+**Tuned and measured 2026-08-01.** These are observed values, not estimates.
+
+| | before | after |
+|---|---|---|
+| `OLLAMA_NUM_PARALLEL` | 2 | **1** |
+| `OLLAMA_KV_CACHE_TYPE` | f16 | **q8_0** |
+| `OLLAMA_CONTEXT_LENGTH` | 4096 | **16384** |
+| `OLLAMA_FLASH_ATTENTION` | false | **true** |
+| VRAM used | 9298 MiB (91%) | **8811 MiB (86%)** |
+| warm latency | 2.6s | **1.85s** |
+| cold load | 28.6s | **1m45s** |
+
+`/api/ps` confirms `ctx=16384` — the increase is real, not just an env var.
+
+**4× the context for less VRAM than before.** Flash Attention plus the quantized KV cache
+more than paid for the increase. Cold load got much slower, which is irrelevant while
+`OLLAMA_KEEP_ALIVE=-1` pins both models — but it is felt after any container restart.
+
+### Flash Attention is mandatory, not optional
+
+`OLLAMA_KV_CACHE_TYPE=q8_0` **will not load** without it:
+
+```
+llama_init_from_model: V cache quantization requires flash_attn
+llama-server process no longer running: exit status 1
+```
+
+Quantized K/V cache requires Flash Attention. Supported on Ampere and newer; the 3080
+qualifies. If the KV cache type is ever changed back toward a quantized value, Flash
+Attention must be on or the model silently fails to load and every call errors.
+
+### Tuning order still matters
+
+If these are ever re-tuned from defaults, the order is load-bearing — raising context first
+at high utilisation will OOM:
+
+1. `OLLAMA_NUM_PARALLEL` → 1 (each slot multiplies KV cache)
+2. `OLLAMA_FLASH_ATTENTION` → true (prerequisite for the next step)
+3. `OLLAMA_KV_CACHE_TYPE` → q8_0 (roughly halves KV memory)
+4. *then* `OLLAMA_CONTEXT_LENGTH` upward
+
+32k was considered and rejected — projected ~1000 MiB of KV, leaving under 450 MiB headroom.
+16k is the comfortable ceiling for a 14B on this card.
+
+### Concurrency
+
+**You cannot have 14B + long context + real parallelism on 10 GB.** Pick two.
+`OLLAMA_MAX_QUEUE=512` means excess requests queue rather than fail, and at ~2s responses,
+two or three users serialised is barely noticeable. Multi-user hits are expected to be rare.
+So: parallelism stays at 1, the queue absorbs bursts, and the VRAM goes to **context** —
+which is what RAG actually needs.
+
+### Applying template changes
+
+**Unraid's "Apply" does not reliably recreate the container.** Observed 2026-08-01: the
+template was saved correctly but the container was only *restarted*, so the env vars never
+took effect — `Created` stayed unchanged while `Started` advanced. Env changes require a
+remove-and-recreate.
+
+Force it with Unraid's own script:
+
+```bash
+/usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container Ollama
+docker start Ollama      # rebuild stops it — Ollama is not in unraid-autostart
+```
+
+Verify with `docker inspect Ollama --format '{{.Created}}'` — the timestamp must move.
+Checking the env vars alone is not enough; a restart leaves the old ones in place and
+looks like nothing happened.
+
+---
+
+## RAG
+
+No Python on Unraid, and none needed. Everything required is already present:
+`sqlite3 3.53`, `jq 1.8`, `node 22`, `php 8.4`, `awk`.
+
+- **chunks + vectors** → SQLite, one table
+- **similarity** → cosine in PHP or Node; milliseconds at this corpus size, no vector DB
+  container needed
+- **embed + generate** → Ollama HTTP, same `curl` pattern as every other integration here
+
+Note `qwen2.5-coder` returns `501 — does not support embeddings`. Embedding is
+`nomic-embed-text`'s job. Batch embedding works (n inputs → n vectors in one call) and is
+required — indexing 50k lines one HTTP call at a time is not viable.
+
+---
+
+### The corpus — and why its shape matters more than its size
+
+As of 2026-08-01, after the header audit and the per-folder documentation pass:
+
+| Layer | Size | What it answers |
+|-------|------|-----------------|
+| Script headers | 115 files × 6 sections = **690 chunks**, 14,685 lines | "What does *this script* do, and why that way" |
+| Folder docs | 18 `README-*.md` + 13 `Manual-*.md` | "How does this *group* work" / "how do I do the thing" |
+| Top-level | `README.md`, `Manual.md` | "What is this system" |
+| Conf templates | 2,175 lines, **~55% comment** | The schema, self-describing |
+| Bash bodies | 51,166 lines | Implementation — index last, lowest weight |
+
+Markdown total: **13,516 lines.** Still small enough that cosine over the whole set is
+milliseconds.
+
+**Chunking is already solved, and the audit is what solved it.** Every script carries
+`PURPOSE / OPERATIONAL MODEL / DESIGN PRINCIPLES / OPERATIONAL SAFEGUARDS / CONFIGURATION /
+RUNTIME MODES` — **115 of 115, no exceptions.** Split on `^# SECTION NAME$` and every chunk is
+a semantically coherent unit by construction. The single worst failure mode in naive RAG —
+a fixed-size window cutting mid-thought and embedding two half-ideas as one vector — cannot
+happen here. Median header is 117 lines, so a section lands around 130–200 tokens: comfortably
+inside `nomic-embed-text`'s window, no sub-splitting needed.
+
+**Store the section name as a column, not just as chunk text.** This is the highest-value
+thing the audit bought and it should not be thrown away at index time. Section type is a free
+metadata filter, so retrieval can route before it computes similarity:
+
+| Question shape | Filter to |
+|----------------|-----------|
+| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
+| "what variable controls X" | `CONFIGURATION` |
+| "does this take --dry-run" | `RUNTIME MODES` |
+| "why is it built this way" | `DESIGN PRINCIPLES` |
+| "what does this script do" | `PURPOSE` |
+
+Hybrid retrieval essentially for free, because every chunk already has a type.
+
+Suggested table shape:
+
+```sql
+CREATE TABLE vv_chunks (
+    id        INTEGER PRIMARY KEY,
+    path      TEXT NOT NULL,     -- repo-relative
+    kind      TEXT NOT NULL,     -- header | readme | manual | template | body
+    section   TEXT,              -- PURPOSE, OPERATIONAL SAFEGUARDS, ... (NULL for md/body)
+    heading   TEXT,              -- md ## heading, for doc chunks
+    content   TEXT NOT NULL,
+    vector    BLOB NOT NULL,     -- 768 float32
+    indexed   INTEGER NOT NULL   -- epoch; re-embed on mtime change only
+);
+```
+
+### Why this corpus is worth more than an equivalent pile of code
+
+A model can read `mover_stop.sh` and describe what it does. What it *cannot* derive from any
+amount of source is that a thing was done deliberately. The audit wrote those down:
+
+- the API cache writers are lockless and unprivileged **on purpose** — regenerable within a
+  minute, every consumer has a live fallback
+- `removeCompletedDownloads` / `removeFailedDownloads` both true is **intended**, not an
+  oversight
+- the arr cleanup ctime gate depends on `media_shares_permissions.sh` staying conditional —
+  reverting either silently stops orphan collection
+- `mesh_monitor.sh`, `adapter.sh`, `decision_engine.sh`, `containers.sh` and
+  `api_cache_writer.sh` carry no root check and no lock **by design** — each documents why in
+  its own header (libraries that must not `exit`, read-only probes, or regenerable output
+  with a live fallback)
+
+Without those in the index, the most likely contribution from an AI assistant reviewing this
+repo is a confident regression: *"I notice this script lacks a lock."* Weight
+`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS` heavily for any suggest-a-change flow —
+they are the guardrails against the assistant helpfully undoing a decision.
+
+### Indexing is safe by default — keep it that way
+
+`Configurations/*.conf` is gitignored; `Deployment/*.template` is tracked and carries all the
+explanatory comments. The corpus therefore describes the full schema while structurally
+**never containing a credential**, because the credential-bearing files were never in the repo
+to begin with.
+
+Treat that as a deliberate boundary, not a happy accident:
+
+- **index tracked files only** — never walk `Configurations/` or `data/`
+- a live conf value that the model genuinely needs should arrive through a *tool call* at
+  query time, subject to the same redaction rules as everything else in the Security section,
+  not be baked into a vector at index time
+- an embedded secret is unrevocable in a way a logged one is not — there is no rotation story
+  for a value already averaged into a 768-dim float
+
+### Known gap — the PHP layer is not covered
+
+78 PHP files under `Plugin/unraid/`; **2** carry a `PURPOSE` block. The entire web UI —
+`pages/`, `api/`, `include/` — is effectively invisible to retrieval.
+
+Consequence: any "AI helper per Varaverk page" feature has this as a hard prerequisite. A
+page-scoped assistant that cannot retrieve the page's own logic is worse than no assistant.
+
+`include/` is the high-value subset to do first — 16 files, and both the pages and the API
+endpoints route through the same `vv_*()` builders, so documenting it once covers both
+callers. This is a follow-on pass, not a blocker for indexing bash.
+
+---
+
+## Scheduled AI
+
+Same orchestrator tiers as everything else, gated on `AI_ENABLED` plus a reachable host.
+Natural fits: weekly digest narration, a periodic pass over `HELD`/report-only findings that
+have accumulated, post-incident summaries after a watchdog event.
+
+Must obey the existing tier discipline — an AI job that fails or times out is a non-fatal
+step like any other, and never blocks the rest of its tier.
+
+---
+
+## UI
+
+- **Dedicated AI page** in the plugin.
+- **Persistent conversation across pages.** A floating widget is not required — a chat column
+  is fine. The constraint is that Unraid's WebGUI is multi-page PHP with full reloads and no
+  SPA shell, so persistence means conversation state lives server-side keyed by session, with
+  the client re-hydrating per page.
+- **Per-page AI helpers** — contextual assistance scoped to whatever that page is about.
+
+---
+
+## "Too bad we can't just run the LLM inside Varaverk and cut out Ollama"
+
+There is a real answer: you don't cut Ollama out, you **absorb it**.
+
+Ollama does non-trivial work — model lifecycle, GPU scheduling, keep-alive, batching, an HTTP
+API. Reimplementing that in bash is not a good trade. But Varaverk already manages containers
+better than most things manage containers. Ollama becomes just another managed container:
+
+- add to `HOST*_WATCHDOG_CONTAINERS` so `docker_watchdog.sh` keeps it healthy
+- add to a restart list so it gets the same proactive treatment as everything else
+- give it a fallback tier if AI should survive a host outage
+- let `docker_update.sh` handle its image updates
+
+That is more Varaverk-native than embedding a model runtime would be, and it costs nothing
+new — the machinery already exists and was audited this session.
+
+---
+
+## Open questions
+
+- Un-ignore `Configurations/` into a private repo for conf history? (blocks conf-write, and
+  overlaps the GitHub-mirror TODO)
+- Does the AI page need auth separate from the Unraid WebGUI, given remote mesh members?
+- Retention/privacy for conversation history — logs may contain paths, container names,
+  possibly credentials pasted by a user.
+- Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the
+  serialisation? Defer until an actual problem is felt.
+
+---
+
+## RAG — built 2026-08-02
+
+Retrieval is live. `AI/` holds the implementation; `AI/README-AI.md` documents it in full. What
+follows is only what changed relative to the plan recorded above.
+
+**Corpus is larger than estimated.** ~2,950 chunks across ~180 files, not the 690 header chunks
+projected. Sub-chunking is why — see below.
+
+**Named-paragraph sub-chunking was necessary, and was not in the plan.** Section-level chunks
+alone were too coarse. `rsync.sh` documents fourteen safeguards in one 2.8k-char
+`OPERATIONAL SAFEGUARDS` block; a query about one of them scored 0.558, below unrelated chunks,
+because the other thirteen dominated the vector. Splitting on the named-paragraph titles the
+header convention already uses took the same query to 0.718 and first place. The parent section
+name is carried onto each sub-chunk so routing still works.
+
+**Two chunker bugs worth remembering.** The last section in a header (RUNTIME MODES in bash,
+DEPENDS ON in a page) ran to EOF and swept up every unrelated comment in the file —
+`scheduler.php` alone produced an 11k-char chunk of unrelated inline comments. And title
+detection must require the *next* line to be indented; without that, any wrapped prose line
+became a spurious chunk boundary mid-sentence.
+
+**Section routing is a boost, not a filter.** Intent detection is a heuristic and must not be
+able to exclude the chunk holding the answer. `--section=` forces a hard filter when wanted.
+
+**Vectors arrive pre-normalised.** `nomic-embed-text` returns L2-normalised vectors (measured
+norm 1.0000001), so cosine is a plain dot product. No normalising step, no magnitude cache.
+
+**`node:sqlite` over a native module.** Still flagged experimental, chosen because it needs no
+native compilation on Unraid. Acceptable because the index is disposable — if a Node upgrade
+breaks it, rebuild takes minutes. PHP reads the same float32 blobs with `unpack('f*', $blob)`
+when the UI needs them.
+
+**Retrieval quality, measured.** 9/10 top-3 hit rate on known-answer questions; the tenth had
+the answer at ranks 2 and 3, so 10/10 for answer-present-in-context at k=6. Full retrieval plus
+generation runs about 43s warm.
+
+### The finding that validated the whole thing
+
+First real end-to-end question asked which variable controls the mover's grace window. The model
+answered `MOVER_STOP_TIMEOUT`, "defaults to 30 seconds", citing `mover_stop.sh › CONFIGURATION`.
+Variable correct; the 30 was wrong — the real value is 300. The model was quoting the header
+verbatim. **The header was stale.**
+
+A sweep for the same pattern found six stale `(default: N)` claims across the repo — all
+corrected in the same pass. This is the operating principle for the folder:
+
+> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
+> wrong, check the cited source before blaming the model.
+
+It also means the index is a documentation-drift detector, not only a question-answering tool.
+
+### Still deliberately not built
+
+Nothing consults this. Every `AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false
+with an empty key whitelist, and no watchdog, cleanup or fallback path calls it. Host resolution
+across the Tailscale mesh is specified above but not implemented — `ai_index.sh` and
+`ai_query.sh` currently require a local `HOST*_OLLAMA_URL` and fail with a clear message when it
+is empty, rather than silently probing the mesh.

Orchestrators/README-Orchestrators.md

@@ -538,7 +538,7 @@ Runs on both servers; `detect_hosts()` determines which direction each sync goes
 #    Drive temperature exit codes respected — skip share or abort all on CRIT.
 #
 # 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
-#    media_management.sh → permissions + cleaners + arr cleanup
+#    Media/ + Arrs_Stack/ scripts → permissions, cleaners, arr cleanup, classification
 #    docker_daily_restart.sh → nightly container restarts
 # ─────────────────────────────────────────────────────────────────────────────
 ```
@@ -557,12 +557,12 @@ Runs on both servers; `detect_hosts()` determines which direction each sync goes
 #
 # HOST1 runs this script at 1am:
 #   → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
-#   → media_management.sh on HOST1's shares
+#   → Media/ + Arrs_Stack/ scripts on HOST1's shares
 #   → docker_daily_restart.sh on HOST1's containers
 #
 # HOST2 runs this script at 1am:
 #   → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
-#   → media_management.sh on HOST2's shares
+#   → Media/ + Arrs_Stack/ scripts on HOST2's shares
 #   → docker_daily_restart.sh on HOST2's containers
 # ─────────────────────────────────────────────────────────────────────────────
 ```

Orchestrators/array_started.sh

@@ -55,13 +55,33 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check            — all launched scripts require root
+# Root Enforcement
-#   acquire_lock          — prevents duplicate array start launches
+#   Every script launched here requires root.
-#   detect_hosts()        — MY_ID in notifications
+#
-#   platform_require_cmd  — notify validated before use
+# Lock Acquisition
-#   chmod +x auto-fix     — non-executable scripts fixed before launch
+#   acquire_lock prevents duplicate array start launches. Unraid can fire the array
-#   Full path on failure  — shows exact path for debugging
+#   start hook more than once, and a second pass would re-launch continuous scripts
-#   notify on failures    — alert if any script fails to launch
+#   that are already running.
+#
+# Host Detection
+#   detect_hosts() sets MY_ID for notifications.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if ARRAY_START_SCRIPTS is empty. An empty
+#   list would silently bring the array up with no ramdisk, no network setup, no
+#   watchdogs and no fallback — while reporting a clean start.
+#
+# Executable Auto-Fix
+#   Non-executable scripts are chmod +x'd before launch. A permission bit lost to a
+#   git checkout or a file copy should not silently disable a boot-time component.
+#
+# Full Path on Failure
+#   Failures report the exact resolved path, so a missing script is immediately
+#   distinguishable from a script that ran and failed.
+#
+# Failure Notification
+#   Any script that fails to launch raises a notification — array start is unattended,
+#   so a silent failure here would only surface much later as a missing service.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -109,6 +129,16 @@ acquire_lock
 
 detect_hosts
 
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#ARRAY_START_SCRIPTS[@]} -eq 0 ]]; then
+    error "ARRAY_START_SCRIPTS is empty — no array start scripts will run"
+    error "Check ARRAY_START_SCRIPTS in master.conf"
+    notify "array start scripts skipped on $(hostname) ($MY_ID) — ARRAY_START_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
+
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — scripts will not be launched"
 
 # ==============================================================================================
@@ -236,16 +266,8 @@ echo "$ICON_SUCCESS Launched: ${#JOB_PASS[@]}"
 echo "$ICON_TIME    Duration: $(format_duration $(( END - START )))"
 echo ""
 
-if [[ "$DRY_RUN" == true ]]; then
+# The configured list is the denominator — a script the conf names but that never launched is
-    warn "DRY RUN — no scripts launched"
+# skipped, not absent, and only shows up if something counts it.
-elif [[ ${#JOB_FAIL[@]} -gt 0 ]]; then
+JOB_COUNT="${#ARRAY_START_SCRIPTS[@]}"
-    warn "Status: ${#JOB_FAIL[@]} script(s) failed — ${JOB_FAIL[*]}"
+orchestrator_summary "ARRAY START" "$START" "Array Start"
-    notify "Array start on $(hostname) ($MY_ID) — ${#JOB_FAIL[@]} script(s) failed: ${JOB_FAIL[*]}" \
+exit $?
-        "Array Start" "warning"
-else
-    echo "$ICON_DONE Status: all ${#JOB_PASS[@]} script(s) launched ✅"
-fi
-echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-
-[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
-exit 0

Orchestrators/array_stopping.sh

@@ -40,11 +40,28 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check            — all stop scripts require root
+# Root Enforcement
-#   acquire_lock          — prevents concurrent array stop runs
+#   Every stop script launched here requires root.
-#   detect_hosts()        — MY_ID in notifications and logs
+#
-#   platform_require_cmd  — notify validated before use
+# Lock Acquisition
-#   notify on failures    — alert if any stop script fails
+#   acquire_lock prevents concurrent array stop runs. Two overlapping shutdown
+#   sequences would fight over the same containers.
+#
+# Host Detection
+#   detect_hosts() sets MY_ID for notifications and logs.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if ARRAY_STOP_SCRIPTS is empty. An empty
+#   list means the array stops without saving the conf cache or gracefully stopping
+#   containers — the failure would only be discovered at the next boot.
+#
+# Non-Fatal Steps
+#   A failing stop script is recorded and the remaining ones still run. Abandoning the
+#   shutdown sequence partway would leave more state unsaved than continuing does.
+#
+# Failure Notification
+#   Any failing stop script raises a notification. Shutdown is unattended and its
+#   failures are invisible until they cause a problem on the way back up.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -97,6 +114,16 @@ fi
 
 detect_hosts
 
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#ARRAY_STOP_SCRIPTS[@]} -eq 0 ]]; then
+    error "ARRAY_STOP_SCRIPTS is empty — no array stop scripts will run"
+    error "Check ARRAY_STOP_SCRIPTS in master.conf"
+    notify "array stop scripts skipped on $(hostname) ($MY_ID) — ARRAY_STOP_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
+
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no stop scripts will be executed"
 
 # ==============================================================================================
@@ -163,18 +190,6 @@ echo "$ICON_TIME    Duration: $(format_duration $(( END - START )))"
 [[ ${#JOB_FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR   Failed:   ${JOB_FAIL[*]}"
 echo ""
 
-if [[ "$DRY_RUN" == true ]]; then
+JOB_COUNT="$STEP"
-    warn "DRY RUN — no changes made"
+orchestrator_summary "ARRAY STOP" "$START" "Array Stop"
-elif [[ ${#JOB_FAIL[@]} -eq 0 ]]; then
+exit $?
-    echo "$ICON_DONE Status: all $STEP step(s) complete ✅"
-    notify "Array stop complete on $(hostname) ($MY_ID) — $STEP step(s) done" \
-        "Array Stop" "normal"
-else
-    warn "Status: ${#JOB_FAIL[@]} step(s) failed — ${JOB_FAIL[*]}"
-    notify "Array stop on $(hostname) ($MY_ID) — ${#JOB_FAIL[@]} step(s) failed: ${JOB_FAIL[*]}" \
-        "Array Stop" "warning"
-fi
-echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-
-[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
-exit 0

Orchestrators/critical_sync_maintenance.sh

@@ -44,11 +44,31 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check            — rsync and container stop/start require root
+# Root Enforcement
-#   acquire_lock "strict" — no pile-up; skip cycle if prior run still active
+#   rsync over SSH and container stop/start both require root.
-#   detect_hosts()        — MY_ID and REMOTE_ID for routing and logs
+#
-#   resolve_remote_ip     — confirms remote reachability before any transfer
+# Lock Acquisition
-#   RSYNC_ENABLED gate    — global kill switch respected before any rsync call
+#   acquire_lock in strict mode — a cycle is skipped rather than queued if the previous
+#   one is still running. At a 30-minute cadence, queuing would let a slow sync stack
+#   windows behind it.
+#
+# Host Detection
+#   detect_hosts() sets MY_ID and REMOTE_ID for routing and logs.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if CRITICAL_MAINTENANCE_SCRIPTS is empty —
+#   a silently empty critical tier would stop downloader resets and play-state sync
+#   while still reporting success every 30 minutes.
+#
+# Remote IP Resolution
+#   resolve_remote_ip confirms the partner is reachable before any transfer is attempted.
+#
+# RSYNC_ENABLED Gate
+#   The global kill switch is respected before any rsync call, so disabling rsync
+#   ecosystem-wide genuinely stops it here too.
+#
+# Non-Fatal Steps
+#   A failing job is recorded and the rest of the tier still runs.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -98,6 +118,16 @@ fi
 acquire_lock "strict"
 
 detect_hosts
+
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
+    error "CRITICAL_MAINTENANCE_SCRIPTS is empty — no critical maintenance scripts will run"
+    error "Check CRITICAL_MAINTENANCE_SCRIPTS in master.conf"
+    notify "critical maintenance scripts skipped on $(hostname) ($MY_ID) — CRITICAL_MAINTENANCE_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
 resolve_remote_ip
 
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
@@ -214,9 +244,28 @@ if [[ "${PARTNERSHIP_ENABLED:-false}" == true ]]; then
     PARTNER_DRY=""
     [[ "$DRY_RUN" == true ]] && PARTNER_DRY="--dry-run"
 
+    # A successful rsync proves the partner answered; a failed one is evidence it did not. But
+    # rsync being switched off is neither — and it used to be read as "unseen", so the offline
+    # counter climbed every 30 minutes toward the 30-day auto-offboard on a partnership whose
+    # only fault was that RSYNC_ENABLED=false. That is how a deliberately paused sync ends up
+    # dismantling the partnership it was paused for. With no rsync attempt there is nothing to
+    # report, so the check runs without touching the counter either way.
+    # Tier 2 counts as "switched off" here exactly as much as Tier 1 does. The guard below used
+    # to test RSYNC_ENABLED alone, but it is CRITICAL_RSYNC_ENABLED that governs whether this
+    # orchestrator attempts an rsync at all — so with Tier 1 open and Tier 2 closed, no transfer
+    # was attempted, RSYNC_OK stayed false, and the run fell through to --remote-unseen and
+    # incremented the counter every 30 minutes against a partner that was answering fine.
+    #
+    # Onboard Step 1d now leaves precisely that posture on purpose — Tier 1 open so provisioning
+    # can run, every Tier 2 gate closed so nothing is scheduled. A freshly onboarded, perfectly
+    # healthy partnership would have auto-offboarded itself 30 days later.
     if [[ "$RSYNC_OK" == true ]]; then
         bash "$SCRIPT_DIR/../Partnership/partnership_manager.sh" \
             --check --remote-seen $PARTNER_DRY
+    elif [[ "${RSYNC_ENABLED:-false}" != true || "${CRITICAL_RSYNC_ENABLED:-false}" != true ]]; then
+        echo "Critical rsync gated off — partnership check runs, offline counter untouched"
+        bash "$SCRIPT_DIR/../Partnership/partnership_manager.sh" \
+            --check $PARTNER_DRY
     else
         bash "$SCRIPT_DIR/../Partnership/partnership_manager.sh" \
             --check --remote-unseen $PARTNER_DRY
@@ -231,23 +280,7 @@ fi
 END=$(date +%s)
 DURATION=$(format_duration $(( END - START )))
 
-TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
+# Standard ending, quiet mode — 30-min cadence, so a healthy cycle stays one line.
-
+[[ ${#PASS[@]} -gt 0 ]] && echo "Synced: ${PASS[*]}"
-# Minimal one-liner when healthy — 30-min cadence, keep it quiet. Full detail on failure.
+orchestrator_summary "CRITICAL SYNC" "$START" "Critical Sync" quiet
-if [[ "$TOTAL_FAIL" -gt 0 ]]; then
+exit $?
-    echo ""
-    echo "━━━━━ $ICON_SUMMARY CRITICAL SYNC SUMMARY ━━━━━"
-    echo "$ICON_HOST   Identity: $MY_ID ($LOCAL_SERVER_NAME)"
-    echo "$ICON_TIME   Duration: $DURATION"
-    [[ ${#PASS[@]} -gt 0 ]] && echo "Synced:      ${PASS[*]}"
-    [[ ${#FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR  Failed shares: ${FAIL[*]}"
-    [[ ${#JOB_FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR  Failed jobs:   ${JOB_FAIL[*]}"
-    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-    notify "Critical sync failed on $(hostname) ($MY_ID) — ${FAIL[*]} ${JOB_FAIL[*]}" \
-        "Critical Sync" "warning"
-    exit 1
-else
-    echo "Critical sync complete — $MY_ID — ${DURATION} — ${#PASS[@]} share(s), ${#JOB_PASS[@]} job(s)"
-fi
-
-exit 0

Orchestrators/daily_sync_maintenance.sh

@@ -59,13 +59,40 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check           — rsync and docker operations require root
+# Root Enforcement
-#   acquire_lock         — prevents concurrent daily windows
+#   rsync and docker operations require root.
-#   detect_hosts()       — aliases correct per-host share lists
+#
-#   check_connectivity   — verified before any rsync
+# Lock Acquisition
-#   check_remote_rootfs  — aborts rsync if remote rootfs nearly full
+#   acquire_lock prevents two daily windows overlapping — the window is long and a
-#   Non-fatal jobs       — a failed job logs and continues; remaining jobs still run
+#   second pass would contend for the same shares and containers.
-#   notify on failure    — successful daily run produces no notification
+#
+# Host Detection
+#   detect_hosts() aliases the correct per-host share and script lists.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if DAILY_MAINTENANCE_SCRIPTS is empty. This
+#   is the largest tier in the ecosystem — an empty list would silently skip git pull,
+#   permissions, cleaners, arr cleanup and docker updates while reporting a clean run.
+#
+# Connectivity Check
+#   check_connectivity is verified before any rsync is attempted.
+#
+# Remote Rootfs Check
+#   check_remote_rootfs aborts rsync if the remote rootfs is nearly full, rather than
+#   pushing data to a partner that cannot hold it.
+#
+# Drive Temperature Escalation
+#   rsync.sh's exit code is honoured per share: exit 1 (temp WARN) skips that share and
+#   continues; exit 2 (temp CRITICAL) sets ABORT_ALL_SYNCS so every remaining share in
+#   the window is skipped and a notification is raised. Continuing to hammer drives that
+#   are already too hot is how a thermal warning becomes a dead disk.
+#
+# Non-Fatal Jobs
+#   A failed job is logged and the remaining jobs still run. Partial completion of a
+#   maintenance window beats abandoning it at the first error.
+#
+# Quiet on Success
+#   A successful daily run produces no notification — only failures surface.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -123,6 +150,16 @@ if ! command -v docker &>/dev/null; then
 fi
 
 detect_hosts
+
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#DAILY_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
+    error "DAILY_MAINTENANCE_SCRIPTS is empty — no daily maintenance scripts will run"
+    error "Check DAILY_MAINTENANCE_SCRIPTS in master.conf"
+    notify "daily maintenance scripts skipped on $(hostname) ($MY_ID) — DAILY_MAINTENANCE_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
 resolve_remote_ip
 
 acquire_lock
@@ -317,11 +354,6 @@ WINDOW_END=$(date +%s)
 # ━━━ Summary ━━━
 # ==============================================================================================
 echo ""
-echo "━━━━━ $ICON_SUMMARY DAILY MAINTENANCE SUMMARY ━━━━━"
-echo "$ICON_HOST   Identity: $MY_ID ($LOCAL_SERVER_NAME)"
-echo "$ICON_TIME   Window:   $(date -d @"$WINDOW_START" '+%Y-%m-%d %H:%M:%S') → $(date -d @"$WINDOW_END" '+%H:%M:%S')"
-echo "$ICON_TIME   Duration: $(format_duration $(( WINDOW_END - WINDOW_START )))"
-echo ""
 
 echo "$ICON_SYNC  Shares ($SHARE_COUNT):"
 for entry in "${SHARE_TIMES[@]}"; do
@@ -345,16 +377,7 @@ if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
     echo ""
 fi
 
-TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
+# Standard ending — derives skipped from SHARE_COUNT, so a run with rsync gated off reports
-
+# PARTIAL instead of "all complete".
-if [[ "$TOTAL_FAIL" -gt 0 ]]; then
+orchestrator_summary "DAILY MAINTENANCE" "$WINDOW_START" "Daily Maintenance"
-    warn "Status: $TOTAL_FAIL failure(s)"
+exit $?
-    notify "Daily maintenance completed with failures on $(hostname) ($MY_ID) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
-        "Daily Maintenance" "warning"
-    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-    exit 1
-else
-    echo "$ICON_DONE Status: all complete — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
-    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-    exit 0
-fi

Orchestrators/intermediate_sync_maintenance.sh

@@ -12,7 +12,8 @@
 # OPERATIONAL MODEL
 # ==============================================================================================
 #
-#   1. conf_sync.sh --pull-only          — refresh partner conf cache in RAM (/tmp/.cache/vv/d/)
+#   1. conf_sync.sh                      — refresh partner conf cache in RAM (/tmp/varaverk/conf/),
+#                                          both directions: pull theirs, push ours
 #   2. arr_sync.sh                       — sync Lidarr/Sonarr/Radarr libraries across all nodes
 #   3. Rsync window (optional)           — INTERMEDIATE_SYNC_SHARES, if any configured
 #   4. INTERMEDIATE_MAINTENANCE_SCRIPTS  — artwork fetch and any future 4-hour jobs
@@ -40,13 +41,36 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check           — scripts called here require root
+# Root Enforcement
-#   acquire_lock         — prevents concurrent intermediate windows
+#   Every script called from here requires root.
-#   detect_hosts()       — aliases correct per-host share lists
+#
-#   check_connectivity   — verified before any rsync (skipped if no shares)
+# Lock Acquisition
-#   check_remote_rootfs  — aborts rsync if remote rootfs nearly full
+#   acquire_lock prevents concurrent intermediate windows.
-#   Non-fatal jobs       — a failed arr_sync warns but does not block rsync or artwork fetch
+#
-#   Minimal on success   — runs 6x/day; full breakdown only on failure or --log
+# Host Detection
+#   detect_hosts() aliases the correct per-host share lists.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if INTERMEDIATE_MAINTENANCE_SCRIPTS is empty,
+#   rather than running six no-op windows a day that all report success.
+#
+# Connectivity Check
+#   check_connectivity is verified before any rsync, and skipped entirely when no shares
+#   are configured — there is nothing to reach a partner for.
+#
+# Remote Rootfs Check
+#   check_remote_rootfs aborts rsync if the remote rootfs is nearly full.
+#
+# Drive Temperature Escalation
+#   rsync.sh's exit code is honoured per share: exit 1 skips that share, exit 2 aborts
+#   every remaining sync in the window and notifies.
+#
+# Non-Fatal Jobs
+#   A failing arr_sync warns but does not block rsync or the artwork fetch that follow it.
+#
+# Minimal on Success
+#   Runs six times a day, so the full breakdown only prints on failure or with --log.
+#   A quiet run is the normal outcome and should not fill the log.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -101,6 +125,16 @@ fi
 acquire_lock
 
 detect_hosts
+
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#INTERMEDIATE_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
+    error "INTERMEDIATE_MAINTENANCE_SCRIPTS is empty — no intermediate maintenance scripts will run"
+    error "Check INTERMEDIATE_MAINTENANCE_SCRIPTS in master.conf"
+    notify "intermediate maintenance scripts skipped on $(hostname) ($MY_ID) — INTERMEDIATE_MAINTENANCE_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
 resolve_remote_ip
 
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
@@ -149,23 +183,27 @@ echo ""
 echo "━━━ $ICON_GEAR Intermediate Sync — $MY_ID — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
 
 # ==============================================================================================
-# ━━━ Conf Pull ━━━
+# ━━━ Conf Sync ━━━
 # ==============================================================================================
 echo ""
-echo "━━━ $ICON_GEAR Conf Pull ━━━"
+echo "━━━ $ICON_GEAR Conf Sync ━━━"
 
+# Full sync, not --pull-only. The push half was written as an event-driven fast path for the
+# conf-save hook, but no such hook was ever built — so outside array start nothing pushed this
+# host's conf to its partners at all, and a partner's copy of our conf stayed at whatever it was
+# when we last rebooted. Pull alone kept our view of them fresh while their view of us decayed.
 CONF_SYNC_SCRIPT="$ECOSYSTEM_ROOT/System_Essentials/conf_sync.sh"
 if [[ ! -f "$CONF_SYNC_SCRIPT" ]]; then
     warn "conf_sync.sh not found — skipping partner conf refresh"
 else
-    _conf_args=("--pull-only")
+    _conf_args=()
     [[ "$DRY_RUN" == true ]] && _conf_args+=("--dry-run")
     if bash "$CONF_SYNC_SCRIPT" "${_conf_args[@]}"; then
         echo "Partner conf cache refreshed ✅"
-        JOB_PASS+=("conf_sync.sh --pull-only")
+        JOB_PASS+=("conf_sync.sh")
     else
-        warn "Partner conf pull failed — cache may be stale"
+        warn "Partner conf sync failed — cache may be stale"
-        JOB_FAIL+=("conf_sync.sh --pull-only")
+        JOB_FAIL+=("conf_sync.sh")
     fi
     unset _conf_args
 fi
@@ -319,20 +357,8 @@ if [[ "$SHOW_FULL" == true ]]; then
     fi
 fi
 
-if [[ "$DRY_RUN" == true ]]; then
+# Standard ending, quiet mode — 4-hour cadence, so an OK cycle is one parseable line and
-    warn "DRY RUN — no changes made"
+# anything skipped or failed expands to the full block on its own.
-elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
+_mode=quiet; [[ "$ENABLE_LOGGING" == true ]] && _mode=full
-    if [[ "$SHOW_FULL" == true ]]; then
+orchestrator_summary "INTERMEDIATE SYNC" "$WINDOW_START" "Intermediate Sync" "$_mode"
-        echo "$ICON_DONE Status: all complete ✅ — ${#JOB_PASS[@]} job(s) run, ${#PASS[@]}/$SHARE_COUNT share(s) synced"
+exit $?
-    else
-        echo "$ICON_DONE Intermediate sync — ${#JOB_PASS[@]} job(s), ${#PASS[@]}/$SHARE_COUNT share(s) ($(format_duration $(( WINDOW_END - WINDOW_START ))))"
-    fi
-else
-    warn "Status: $TOTAL_FAIL failure(s)"
-    notify "Intermediate sync failed on $(hostname) ($MY_ID) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
-        "Intermediate Sync" "warning"
-fi
-[[ "$SHOW_FULL" == true ]] && echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-
-[[ "$TOTAL_FAIL" -gt 0 ]] && exit 1
-exit 0

Orchestrators/monthly_maintenance.sh

@@ -45,12 +45,34 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check      — ZFS scrub, SMART tests require root
+# Root Enforcement
-#   acquire_lock    — prevents concurrent monthly runs
+#   ZFS scrub and SMART tests require root.
-#   detect_hosts()  — MY_ID in notifications and logs
+#
-#   Uptime gate     — MONTHLY_UPTIME_THRESHOLD_DAYS must be met
+# Lock Acquisition
-#   Interval gate   — MONTHLY_RUN_INTERVAL_DAYS since last run must be met
+#   acquire_lock prevents concurrent monthly runs. These are long jobs — a scrub can run
-#   --force flag    — bypasses both gates for manual override
+#   for hours — and two at once would double the I/O cost for no benefit.
+#
+# Host Detection
+#   detect_hosts() sets MY_ID for notifications and logs.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if MONTHLY_MAINTENANCE_SCRIPTS is empty. A
+#   monthly job that silently does nothing is the hardest kind to notice missing.
+#
+# Uptime Gate
+#   MONTHLY_UPTIME_THRESHOLD_DAYS must be met before the run proceeds. Heavy full-disk
+#   work immediately after a boot competes with everything else still starting up.
+#
+# Interval Gate
+#   MONTHLY_RUN_INTERVAL_DAYS since the last successful run must have elapsed. The
+#   schedule fires more often than the work should actually happen, so the gate — not
+#   the cron entry — is what defines the real cadence.
+#
+# Force Override
+#   --force bypasses both gates for a deliberate manual run.
+#
+# Non-Fatal Steps
+#   A failing job is recorded and the remaining jobs still run.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -113,6 +135,16 @@ acquire_lock
 
 detect_hosts
 
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#MONTHLY_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
+    error "MONTHLY_MAINTENANCE_SCRIPTS is empty — no monthly maintenance scripts will run"
+    error "Check MONTHLY_MAINTENANCE_SCRIPTS in master.conf"
+    notify "monthly maintenance scripts skipped on $(hostname) ($MY_ID) — MONTHLY_MAINTENANCE_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
+
 [[ "$DRY_RUN" == true ]]   && warn "DRY RUN — no scripts will be executed"
 [[ "$FORCE_RUN" == true ]] && warn "FORCE — uptime and interval gates bypassed"
 
@@ -275,18 +307,8 @@ echo "$ICON_TIME    Duration: $(format_duration $(( END - START )))"
 [[ ${#JOB_FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR   Failed:   ${JOB_FAIL[*]}"
 echo ""
 
-if [[ "$DRY_RUN" == true ]]; then
+# STEP is what this orchestrator expected to run, so it is the denominator that makes a skipped
-    warn "DRY RUN — no changes made"
+# step visible rather than absent.
-elif [[ ${#JOB_FAIL[@]} -eq 0 ]]; then
+JOB_COUNT="$STEP"
-    echo "$ICON_DONE Status: all $STEP step(s) complete ✅"
+orchestrator_summary "MONTHLY MAINTENANCE" "$START" "Monthly Maintenance"
-    notify "Monthly maintenance complete on $(hostname) ($MY_ID) — $STEP step(s) done" \
+exit $?
-        "Monthly Maintenance" "normal"
-else
-    warn "Status: ${#JOB_FAIL[@]} step(s) failed — ${JOB_FAIL[*]}"
-    notify "Monthly maintenance on $(hostname) ($MY_ID) — ${#JOB_FAIL[@]} step(s) failed: ${JOB_FAIL[*]}" \
-        "Monthly Maintenance" "warning"
-fi
-echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-
-[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
-exit 0

Orchestrators/sunday_morning_coffee_report.sh

@@ -39,12 +39,33 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check        — required for the docker/system reads used in the report
+# Root Enforcement
-#   acquire_lock      — prevents overlapping weekly runs
+#   Required for the docker and system reads the report is built from.
-#   detect_hosts()    — MY_ID in banner and summary
+#
-#   Non-fatal steps   — a failed script is logged; remaining scripts still run
+# Lock Acquisition
-#   Flag pass-through — --dry-run and --log forwarded to all child scripts
+#   acquire_lock prevents overlapping weekly runs.
-#   notify() on failure — pushed only outside --dry-run, matching the runtime-mode contract below
+#
+# Host Detection
+#   detect_hosts() sets MY_ID for the banner and summary.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if COFFEE_REPORT_SCRIPTS is empty. A report
+#   that silently contains nothing still arrives looking like a report.
+#
+# Read-Only by Composition
+#   Every child here is a reporting script. This orchestrator changes nothing itself —
+#   it only sequences reads and assembles their output.
+#
+# Non-Fatal Steps
+#   A failing script is logged and the remaining ones still run, so one unavailable
+#   subsystem costs a section of the report rather than the whole thing.
+#
+# Flag Pass-Through
+#   --dry-run and --log are forwarded to every child script.
+#
+# Notification Contract
+#   notify() fires on failure only outside --dry-run, matching the runtime-mode contract
+#   below — a dry run never sends anything outward.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -77,6 +98,9 @@ ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
 
 source "$ECOSYSTEM_ROOT/load_config.sh"
 
+# Timed from here so the standard summary can report a real duration; this report had none.
+REPORT_START=$(date +%s)
+
 parse_args "$@"
 
 # ==============================================================================================
@@ -91,6 +115,16 @@ acquire_lock
 
 detect_hosts
 
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#COFFEE_REPORT_SCRIPTS[@]} -eq 0 ]]; then
+    error "COFFEE_REPORT_SCRIPTS is empty — no coffee report scripts will run"
+    error "Check COFFEE_REPORT_SCRIPTS in master.conf"
+    notify "coffee report scripts skipped on $(hostname) ($MY_ID) — COFFEE_REPORT_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
+
 # ==============================================================================================
 # ━━━ Helpers ━━━
 # ==============================================================================================
@@ -153,6 +187,19 @@ echo "━━━━━ ☕ COFFEE REPORT SUMMARY ━━━━━"
 echo "🖥️  Host:    $MY_ID"
 echo "⏱️  Done:    $(date '+%Y-%m-%d %H:%M:%S')"
 
+# Anything the Scheduler troubleshooter concluded was a Varaverk defect rather than a setting.
+# Counted here because a finding filed on a Tuesday and read on a Tuesday is a finding nobody
+# acts on — this report is the weekly moment the operator is actually looking. Count only: the
+# detail lives on the AI tab, and a report that reprints every open bug stops being skimmable.
+# Silent when there are none, and silent when AI is off, so a host without it reads the same as
+# it always has.
+if [[ "${AI_ENABLED:-false}" == true && -d "$DATA_DIR/ai_bugs" ]]; then
+    _ai_bugs_open=$(grep -l '"open": true' "$DATA_DIR"/ai_bugs/*.json 2>/dev/null | wc -l)
+    if [[ "${_ai_bugs_open:-0}" -gt 0 ]]; then
+        echo "🐞  AI bugs: $_ai_bugs_open open — see the AI tab"
+    fi
+fi
+
 if [[ ${#JOB_PASS[@]} -gt 0 ]]; then
     echo "✅  Passed:  ${JOB_PASS[*]}"
 fi
@@ -160,10 +207,9 @@ if [[ ${#JOB_FAIL[@]} -gt 0 ]]; then
     echo "❌  Failed:  ${JOB_FAIL[*]}"
 fi
 
-if [[ ${#JOB_FAIL[@]} -gt 0 && "$DRY_RUN" != true ]]; then
+# Standard ending. The configured section list is the denominator, so a report that quietly
-    notify "Sunday coffee report had failures on $(hostname) ($MY_ID) — ${JOB_FAIL[*]}" \
+# stopped producing one of its sections reads as skipped rather than simply not appearing.
-        "Sunday Morning Coffee Report" "warning"
+JOB_COUNT="${#SUNDAY_REPORT_SCRIPTS[@]:-0}"
-fi
+[[ "$JOB_COUNT" -eq 0 ]] && JOB_COUNT=$(( ${#JOB_PASS[@]} + ${#JOB_FAIL[@]} ))
-
+orchestrator_summary "SUNDAY MORNING COFFEE REPORT" "$REPORT_START" "Sunday Morning Coffee Report"
-[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
+exit $?
-exit 0

Orchestrators/transcode_management.sh

@@ -50,12 +50,37 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check      — mount and docker operations require root
+# Root Enforcement
-#   acquire_lock    — prevents concurrent 7-minute cycles overlapping
+#   Mount and docker operations require root.
-#   detect_hosts()  — aliases RAMDISK_PATH, TRANSCODE_SSD, RAMDISK_WARN_GB per host
+#
-#   --dry-run       — passed through to every script in TRANSCODE_MANAGEMENT_SCRIPTS
+# Lock Acquisition
-#   Exit code       — worst exit code across all scripts returned to cron
+#   acquire_lock prevents concurrent 7-minute cycles overlapping. Cleanup and the manager
-#   notify()        — pushed on failure, skipped in --dry-run
+#   both touch the same ramdisk, and two cycles at once could have one deleting files
+#   while the other is measuring usage to decide whether to flip.
+#
+# Host Detection
+#   detect_hosts() aliases RAMDISK_PATH, TRANSCODE_SSD and RAMDISK_WARN_GB per host.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if TRANSCODE_MANAGEMENT_SCRIPTS is empty —
+#   without it the ramdisk would silently stop being cleaned or flipped, and the first
+#   symptom would be a full ramdisk stalling playback.
+#
+# Ordering Is Load-Bearing
+#   Cleanup runs before the manager so the manager measures real active-session usage
+#   rather than usage inflated by stale files. Reversing them would trigger flips that
+#   a cleanup two seconds later would have made unnecessary.
+#
+# Dry Run Propagation
+#   --dry-run is passed through to every script in TRANSCODE_MANAGEMENT_SCRIPTS.
+#
+# Any-Failure Exit Code
+#   Exits 1 if any child failed, 0 otherwise — the individual exit codes are not
+#   propagated, only whether anything failed. A failure in an early child is therefore
+#   never masked by a later success.
+#
+# Notification Contract
+#   notify() fires on failure and is skipped in --dry-run.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -112,6 +137,20 @@ fi
 # detect_hosts() sets MY_ID and aliases all HOST*_TRANSCODE_* vars
 detect_hosts
 
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+# This orchestrator never timed itself, so its summary could not report a duration. Set before
+# any work so the figure means the cycle, not the tail of it.
+CYCLE_START=$(date +%s)
+
+if [[ ${#TRANSCODE_MANAGEMENT_SCRIPTS[@]} -eq 0 ]]; then
+    error "TRANSCODE_MANAGEMENT_SCRIPTS is empty — no transcode management scripts will run"
+    error "Check TRANSCODE_MANAGEMENT_SCRIPTS in master.conf"
+    notify "transcode management scripts skipped on $(hostname) ($MY_ID) — TRANSCODE_MANAGEMENT_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
+
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing through to child scripts"
 
 # ==============================================================================================
@@ -188,15 +227,9 @@ done
 # ==============================================================================================
 # ━━━ Summary — minimal one-liner by default (7-min cadence — keep it quiet when healthy) ━━━
 # ==============================================================================================
-if [[ "${#JOB_FAIL[@]}" -eq 0 ]]; then
+# Quiet by default — 7-min cadence. Anything failed or skipped expands on its own.
-    echo "$ICON_SUCCESS Transcode cycle — ${#JOB_PASS[@]}/${#TRANSCODE_MANAGEMENT_SCRIPTS[@]} passed"
+JOB_COUNT="${#TRANSCODE_MANAGEMENT_SCRIPTS[@]}"
-else
+orchestrator_summary "TRANSCODE CYCLE" "${CYCLE_START:-$(date +%s)}" "Transcode Management" quiet
-    error "Transcode cycle — failed: ${JOB_FAIL[*]}"
-    if [[ "$DRY_RUN" != true ]]; then
-        notify "Transcode management failure on $(hostname) ($MY_ID) — ${JOB_FAIL[*]}" \
-            "Transcode Management" "warning"
-    fi
-fi
 
 # ==============================================================================================
 # ━━━ Exit ━━━

Orchestrators/watchdog_orchestrator.sh

@@ -49,11 +49,34 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check       — watchdog operations require root
+# Root Enforcement
-#   acquire_lock     — strict; no pile-up if prior cycle still active
+#   Every watchdog launched from here requires root. Failing once at the top gives one
-#   detect_hosts()   — MY_ID in logs and notifications
+#   clear error instead of the same permission failure repeated per child.
-#   Array check      — exits early if /mnt/user is not shfs-mounted
+#
-#   Startup grace    — WATCHDOG_STARTUP_GRACE respected before any checks
+# Lock Acquisition
+#   acquire_lock in strict mode — if the previous cycle is still running, this one exits
+#   rather than queuing. At a 15-minute cadence a waiting lock would pile up cycles
+#   behind a slow watchdog and eventually run them all at once.
+#
+# Host Detection
+#   detect_hosts() sets MY_ID for logs and notifications.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if WATCHDOG_ORCHESTRATOR_SCRIPTS is empty.
+#   Without it the cycle reports "0/0 passed" and exits 0 every cycle — indistinguishable
+#   from a healthy run, while nothing at all is being monitored.
+#
+# Array Check
+#   Exits early if /mnt/user is not shfs-mounted. Watchdogs that inspect shares would
+#   otherwise read an unmounted array as missing data and act on it.
+#
+# Startup Grace
+#   WATCHDOG_STARTUP_GRACE is respected before any checks run, so containers still
+#   initialising after boot are not judged as unhealthy.
+#
+# Non-Fatal Steps
+#   run_orch_child() records a failing or missing watchdog and continues. One broken
+#   watchdog never suppresses the rest of the chain.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -104,6 +127,16 @@ acquire_lock
 
 detect_hosts
 
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]} -eq 0 ]]; then
+    error "WATCHDOG_ORCHESTRATOR_SCRIPTS is empty — no watchdogs will run"
+    error "Check WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf"
+    notify "watchdogs skipped on $(hostname) ($MY_ID) — WATCHDOG_ORCHESTRATOR_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
+
 log "$ICON_GEAR Config: grace=${WATCHDOG_STARTUP_GRACE}s heartbeat=${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true}/${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}hr scripts=${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"
 log "$ICON_WATCHDOG Order: $(for s in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do printf '%s ' "${s##*/}"; done)"
 
@@ -208,21 +241,15 @@ fi
 # ==============================================================================================
 # ━━━ Summary — minimal one-liner by default, full breakdown on failure or --log ━━━
 # ==============================================================================================
+# Per-script detail only when there is something to read; the standard block carries the rest.
 if [[ "${#JOB_FAIL[@]}" -gt 0 || "$ENABLE_LOGGING" == true ]]; then
-    echo ""
-    echo "━━━━━ $ICON_SUMMARY WATCHDOG CYCLE — $MY_ID — $(date '+%H:%M:%S') ━━━━━"
     for p in "${JOB_PASS[@]}"; do log "  $ICON_DONE  $p"; done
     for f in "${JOB_FAIL[@]}"; do error "  $ICON_ERROR $f"; done
-    echo "$ICON_TIME  Duration: $(format_duration $DURATION)"
-    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-else
-    echo "$ICON_DONE Watchdog cycle — ${#JOB_PASS[@]}/${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]} passed ($(format_duration $DURATION))"
 fi
 
-if [[ "${#JOB_FAIL[@]}" -gt 0 ]]; then
+# Quiet by default at a 15-min cadence. The configured script list is the denominator, so a
-    notify "Watchdog cycle failure on $(hostname) ($MY_ID) — ${JOB_FAIL[*]}" \
+# watchdog that silently stopped running one of its checks shows up as skipped.
-        "Watchdog Orchestrator" "warning"
+JOB_COUNT="${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"
-    exit 1
+_mode=quiet; [[ "$ENABLE_LOGGING" == true ]] && _mode=full
-fi
+orchestrator_summary "WATCHDOG CYCLE" "$CYCLE_START" "Watchdog Orchestrator" "$_mode"
-
+exit $?
-exit 0

Orchestrators/weekly_sync_maintenance.sh

@@ -40,15 +40,35 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   Root check            — stop/start containers and rsync require root
+# Root Enforcement
-#   acquire_lock          — prevents concurrent weekly windows
+#   Container stop/start and rsync both require root.
-#   detect_hosts()        — MY_ID in banner, summary, and notifications
+#
-#   check_connectivity    — verifies remote before any remote operations
+# Lock Acquisition
-#   check_remote_rootfs   — aborts rsync if remote rootfs nearly full
+#   acquire_lock prevents concurrent weekly windows. This window stops Emby and the auth
-#   DOCKER_TIMEOUT        — all docker calls protected
+#   stack — two overlapping runs would fight over the same critical containers.
-#   SSH_TIMEOUT           — all SSH calls protected
+#
-#   platform_require_cmd  — notify validated before use
+# Host Detection
-#   Silent on success     — runs weekly; only failures warrant notification
+#   detect_hosts() sets MY_ID for the banner, summary and notifications.
+#
+# Empty Job List Guard
+#   Exits with an error and a notification if WEEKLY_MAINTENANCE_SCRIPTS is empty, rather
+#   than taking the weekly outage window and doing nothing with it.
+#
+# Connectivity Check
+#   check_connectivity verifies the remote before any remote operation is attempted.
+#
+# Remote Rootfs Check
+#   check_remote_rootfs aborts rsync if the remote rootfs is nearly full.
+#
+# Timeout Protection
+#   DOCKER_TIMEOUT bounds every docker call and SSH_TIMEOUT every SSH call, so neither a
+#   hung daemon nor an unresponsive partner can hold the weekly window open indefinitely.
+#
+# Non-Fatal Steps
+#   A failing job is recorded and the remaining jobs still run.
+#
+# Silent on Success
+#   Runs weekly; only failures warrant a notification.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -109,6 +129,16 @@ if ! command -v docker &>/dev/null; then
 fi
 
 detect_hosts
+
+# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
+# from a healthy run. Fail loudly instead of silently doing no work.
+if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
+    error "WEEKLY_MAINTENANCE_SCRIPTS is empty — no weekly maintenance scripts will run"
+    error "Check WEEKLY_MAINTENANCE_SCRIPTS in master.conf"
+    notify "weekly maintenance scripts skipped on $(hostname) ($MY_ID) — WEEKLY_MAINTENANCE_SCRIPTS is empty" \
+        "$(basename "$0" .sh)" "warning"
+    exit 1
+fi
 resolve_remote_ip
 
 WINDOW_START=$(date +%s)
@@ -372,18 +402,14 @@ WINDOW_END=$(date +%s)
 # ==============================================================================================
 # ━━━ Summary ━━━
 # ==============================================================================================
+# Per-unit detail first — the standard block that follows carries the verdict and the counts, not
+# the names, and knowing WHICH share failed is the whole point of reading a log.
 echo ""
-echo "━━━━━ $ICON_SUMMARY WEEKLY SYNC MAINTENANCE SUMMARY ━━━━━"
-echo "$ICON_HOST   Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
-echo "$ICON_TIME   Window:    $(date -d @"$WINDOW_START" '+%Y-%m-%d %H:%M:%S') → $(date -d @"$WINDOW_END" '+%H:%M:%S')"
-echo "$ICON_TIME   Duration:  $(format_duration $(( WINDOW_END - WINDOW_START )))"
 echo "$ICON_GEAR   Updates:   local=${WEEKLY_SYNC_UPDATES:-false} remote=${WEEKLY_SYNC_UPDATES_REMOTE:-false}"
 echo ""
-
 echo "$ICON_SYNC  Sync jobs ($SHARE_COUNT):"
 for job in "${PASS[@]}"; do echo "  $ICON_DONE  $job"; done
 for job in "${FAIL[@]}"; do echo "  $ICON_ERROR $job"; done
-echo "  Passed: ${#PASS[@]}  Failed: ${#FAIL[@]}"
 
 if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
     echo ""
@@ -392,19 +418,7 @@ if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
     for job in "${JOB_FAIL[@]}"; do echo "  $ICON_ERROR $job"; done
 fi
 
-TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
+# Standard ending. Derives skipped from SHARE_COUNT vs what actually ran, so a gated-off section
-
+# can no longer read as success — this is the run that printed "all complete — 0 shares synced".
-echo ""
+orchestrator_summary "WEEKLY SYNC MAINTENANCE" "$WINDOW_START" "Weekly Maintenance"
-if [[ "$DRY_RUN" == true ]]; then
+exit $?
-    warn "DRY RUN — no changes made"
-elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
-    echo "$ICON_DONE Status: all complete ✅ — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
-else
-    warn "Status: $TOTAL_FAIL failure(s)"
-    notify "Weekly maintenance failed on $(hostname) ($MY_ID) — sync: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
-        "Weekly Maintenance" "warning"
-fi
-echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-
-[[ "$TOTAL_FAIL" -gt 0 ]] && exit 1
-exit 0

Partnership/README-Partnership.md

@@ -144,6 +144,8 @@ independently at 4-hour cadence.
 | `partnership_transfer.sh` | Transfer ownership from current owner to current mirror | Via `partnership_manager.sh --transfer`; owner only |
 | `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
 | `onboard_cancel.sh` | Remove SSH keys in one or both directions, clear setup flags | During cancelled or failed onboard; manual cleanup |
+| `gitea_ssh_setup.sh` | Generate a Gitea keypair and register it via the Gitea API | Called during onboard; manually when re-keying a server |
+| `share_setup.sh` | Create missing Unraid shares on the mirror from the owner's sync lists | Called during onboard; safe to re-run — existing shares are never modified |
 
 ---
 

Partnership/gitea_ssh_setup.sh

@@ -41,6 +41,31 @@
 #        Skipped if entry already exists — use --force to overwrite
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Separate Key From the Rsync Key
+#   Gitea authentication uses its own keypair rather than reusing the rsync automation key.
+#   The two have different lifecycles and different blast radii — re-keying Gitea should
+#   never break cross-host rsync, and revoking a partner's rsync access should not lock this
+#   host out of its own repository.
+#
+# Idempotent Registration
+#   Both generation and registration are skipped when already satisfied, and registration is
+#   matched on the public key itself rather than on a label. A key registered under a
+#   different title is still the same key, and re-registering it would leave duplicate
+#   entries accumulating in Gitea on every re-run.
+#
+# Locate, Do Not Assume
+#   The API endpoint is resolved at runtime — container-local first, then GITEA_DOMAIN. Same
+#   reasoning as git_pull_execute.sh: Gitea's address changes with container restarts and
+#   migrations, so hardcoding it guarantees an eventual break.
+#
+# Force Is Explicit
+#   Regeneration invalidates the key already registered in Gitea, so it requires --force. A
+#   bare re-run can never cost this host its repository access.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #

Partnership/onboard_cancel.sh

@@ -8,6 +8,102 @@
 # Removes SSH keys between HOST1 and HOST2 in the specified direction.
 # Safe to run at any phase. Clears related setup.db flags.
 #
+# The escape hatch for a half-finished onboard: partnership setup is multi-phase, and an
+# attempt abandoned midway leaves keys installed and phase flags set. This unwinds that so
+# onboarding can be started cleanly rather than resumed from an unknown state.
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Per direction requested:
+#
+#   h1 (HOST1 → HOST2)
+#     1. SSH to HOST2 and delete HOST1's public key line from its authorized_keys
+#     2. Clear HOST2's phase / key-ready flags from the setup db
+#     3. Delete the local HOST1 key pair
+#
+#   h2 (HOST2 → HOST1)
+#     Remove HOST2's public key from HOST1's authorized_keys, identifying the key by
+#     HOST2's hostname in the key comment.
+#
+#   both — run each direction in turn.
+#
+# Key removal is matched on the key blob or hostname comment, never on line number.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Safe at Any Phase
+#   Onboarding is multi-phase and can fail anywhere in it. This runs against whatever state
+#   exists rather than requiring a known starting point — a missing key or an absent flag is
+#   a no-op, not an error. Cancelling twice is harmless.
+#
+# Direction Is Explicit
+#   Removing a key is not symmetric: it breaks authentication for whichever side loses it.
+#   The direction must be stated, and defaults to h1 (this host's own outbound key) rather
+#   than to both, so an unqualified run cannot sever the partner's access to you.
+#
+# Unwind, Do Not Repair
+#   The job is to return to a clean pre-onboard state so onboarding can be re-run from the
+#   top. It deliberately does not attempt to salvage or resume a partial setup — a known
+#   empty state is worth more than a guessed-at partial one.
+#
+# Keys and Flags Together
+#   Removing the key without clearing the setup-db flags would leave onboarding believing a
+#   phase had completed. Both are cleared in the same pass for that reason.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Deletes key pairs from /root/.ssh and edits authorized_keys on both ends as root.
+#
+# Lock Acquisition
+#   acquire_lock prevents this racing an in-progress onboard, which would otherwise be
+#   installing the very keys this is removing.
+#
+# Host Detection
+#   detect_hosts() resolves MY_ID / REMOTE_ID so the direction flags map to real hosts.
+#
+# Direction Default
+#   Defaults to h1 — never removes the partner's inbound key unless explicitly asked.
+#
+# Targeted Key Removal
+#   authorized_keys lines are matched by key blob or hostname comment. Nothing is removed
+#   positionally, so an unrelated key can never be deleted because it sat on a given line.
+#
+# SSH Timeout
+#   The remote edit is wrapped in SSH_TIMEOUT — an unreachable partner fails fast rather
+#   than hanging a cancel that still has local cleanup to do.
+#
+# Idempotent
+#   Absent keys and absent flags are skipped silently. Re-running is safe.
+#
+# Dry Run Support
+#   --dry-run reports every key and flag it would remove, and removes none.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   SSH_TIMEOUT
+#       Bounds the remote authorized_keys edit.
+#
+# host*.conf
+#
+#   SSH_KEY
+#       Key used to reach the partner, and the local pair deleted in the h1 direction.
+#
+#   HOST* — hostnames, used to identify which key comment belongs to which side.
+#
+# Setup state lives in the platform setup db (platform_setup_db_path) — the phase and
+# key-ready flags cleared here are the same ones partnership_onboard.sh sets.
+#
 # ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================
@@ -85,14 +181,36 @@ if [[ "$DIRECTION" == "h1" || "$DIRECTION" == "both" ]]; then
             warn "DRY RUN — would remove HOST1 key from $MIRROR:/root/.ssh/authorized_keys"
             H1_DONE=true
         else
-            timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+            # `\|…|d`, not `|…|d`. sed only accepts a custom address delimiter when it is
+            # introduced by a backslash; the bare form is a syntax error — "unknown command: `|'".
+            # A delimiter other than / is still required, because the key blob is base64 and
+            # routinely contains /.
+            #
+            # The error went to 2>/dev/null and `echo ok` ran anyway, so this reported
+            # "key removed ✅" on every run while removing nothing, and a cancelled onboard left
+            # HOST1's key live on the mirror. Report on what the remote actually did instead.
+            _cancel_out=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
                 -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
-                "sed -i \"|${KEY_BLOB}|d\" /root/.ssh/authorized_keys 2>/dev/null
+                "sed -i \"\\|${KEY_BLOB}|d\" /root/.ssh/authorized_keys || { echo sed-failed; exit 1; }
-                 sed -i \"/^${MIRROR_ID}_PHASE\|^${MIRROR_ID}_KEY_READY/d\" $(platform_setup_db_path) 2>/dev/null
+                 chmod 600 /root/.ssh/authorized_keys 2>/dev/null
-                 echo ok" 2>/dev/null | grep -q ok && {
+                 sed -i \"/^${MIRROR_ID}_PHASE/d; /^${MIRROR_ID}_KEY_READY/d\" $(platform_setup_db_path) 2>/dev/null
+                 grep -qF '${KEY_BLOB}' /root/.ssh/authorized_keys 2>/dev/null && echo still-present || echo ok" 2>/dev/null)
+            case "$_cancel_out" in
+                *ok*)
                     echo "HOST1 key removed from $MIRROR authorized_keys ✅"
                     H1_DONE=true
-            } || warn "Could not SSH to $MIRROR — remove HOST1 key there manually"
+                    ;;
+                *still-present*)
+                    warn "HOST1 key still present in $MIRROR authorized_keys — remove it there manually"
+                    ;;
+                *sed-failed*)
+                    warn "Could not edit authorized_keys on $MIRROR — remove HOST1 key there manually"
+                    ;;
+                *)
+                    warn "Could not SSH to $MIRROR — remove HOST1 key there manually"
+                    ;;
+            esac
+            unset _cancel_out
         fi
     fi
 
@@ -132,7 +250,9 @@ if [[ "$DIRECTION" == "h2" || "$DIRECTION" == "both" ]]; then
         warn "DRY RUN — would remove $MIRROR_SHORT key from $AUTH_KEYS"
         H2_DONE=true
     else
-        sed -i "/${MIRROR_SHORT}/Id" "$AUTH_KEYS" && {
+        # chmod after: sed -i rewrites via a temp file, which lands under the umask and can
+        # leave the file 0666. sshd StrictModes then silently refuses every key in it.
+        sed -i "/${MIRROR_SHORT}/Id" "$AUTH_KEYS" && chmod 600 "$AUTH_KEYS" && {
             echo "$MIRROR key removed from HOST1 authorized_keys ✅"
             H2_DONE=true
         } || warn "Failed to remove $MIRROR key from HOST1 authorized_keys"

Partnership/partnership_manager.sh

@@ -79,8 +79,25 @@
 # SSH_TIMEOUT on all remote calls
 #   Every ssh/scp call is timeout-protected. No operation hangs on an unreachable peer.
 #
+# No acquire_lock — Deliberate
+#   This file is dual-role: an executable dispatcher AND a library that
+#   partnership_offboard.sh and partnership_transfer.sh source with
+#   PARTNERSHIP_LIB_MODE=1. A top-level acquire_lock would fire for every sourcing
+#   script, and --check can re-invoke this file as itself (bash "$0" --offboard),
+#   which a strict lock would deadlock. Concurrency is handled per-write with flock
+#   instead. Do not "fix" this to match the single-role scripts.
+#
 # flock on state writes
-#   Prevents concurrent state file corruption from overlapping --check cycles.
+#   write_state_file() serialises full state-file rewrites, and the --check counter
+#   updates are flocked separately on their own lock file. The offline counter is a
+#   read-modify-write: without the lock two overlapping --check cycles both read N and
+#   both write N+1, silently losing an increment and pushing the auto-offboard threshold
+#   past its configured window. The last_seen_remote sed is inside the same lock because
+#   it edits a file write_state_file() rewrites wholesale from other paths.
+#
+#   Note the redirection must sit INSIDE a command substitution — "$( ... ) 201>file"
+#   attaches the descriptor to the assignment rather than the subshell, and flock then
+#   fails with "Bad file descriptor" while the unlocked write proceeds anyway.
 #
 # SIGTERM trap on grace period sleep
 #   Offboard grace period is interruptible — Ctrl-C aborts cleanly.
@@ -334,16 +351,53 @@ EOF
     ) 200>"${file}.lock"
 }
 
+# Deliver the partnership state to the partner, at the path the partner actually reads, under
+# both names its UI looks for.
+#
+# This used to be one scp to "root@ip:$local_file" — the LOCAL absolute path reused verbatim on
+# the remote. That only works while both hosts install to the same place. HOST1 is on flash at
+# /boot/config/plugins/varaverk and HOST2 is in appdata mode at /mnt/user/appdata/Varaverk, so
+# the copy went to a directory HOST2 does not read and, more often, does not have — and the
+# failure surfaced as "will propagate on next sync", which nothing does.
+#
+# Both names, because the page resolves one file per node: partnership_<that node's hostname>.db.
+# The mirror needs partnership_<mirror>.db for its own card and partnership_<owner>.db for the
+# owner's. Nothing on the mirror writes the first one during an owner-driven onboard, which is
+# why a mirror that was fully onboarded still rendered as having no partnership at all.
+# The content is symmetric — state/owner/mirror/onboarded — so one file serves as both.
 push_state_to_remote() {
     local local_file="$1" remote_ip="$2" ssh_key="$3"
     if [[ "$DRY_RUN" == true ]]; then
         warn "DRY RUN — would push state file to remote"
         return 0
     fi
-    timeout "$SSH_TIMEOUT" scp -i "$ssh_key" -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes \
+
-        "$local_file" "root@${remote_ip}:${local_file}" 2>/dev/null && \
+    # Resolve where the partner keeps its state, from the partner. varaverk.cfg names its
-        echo "State file pushed to remote ✅" || \
+    # SCRIPTS_DIR; absent it, the flash default is the right guess for a stock install.
-        warn "Could not push state file to remote — will propagate on next sync"
+    local remote_sd remote_state_dir
+    remote_sd=$(timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" -o ConnectTimeout="$SSH_TIMEOUT" \
+        -o BatchMode=yes -o StrictHostKeyChecking=no root@"$remote_ip" \
+        'grep -oP "(?<=SCRIPTS_DIR=\")[^\"]+" /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null' \
+        2>/dev/null | tr -d '\r')
+    [[ -z "$remote_sd" ]] && remote_sd="/boot/config/plugins/varaverk"
+    remote_state_dir="${remote_sd}/data/state"
+
+    local rc=0
+    local name
+    for name in "partnership_${REMOTE_SERVER_NAME}.db" "partnership_${LOCAL_SERVER_NAME}.db"; do
+        timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" -o ConnectTimeout="$SSH_TIMEOUT" \
+            -o BatchMode=yes -o StrictHostKeyChecking=no root@"$remote_ip" \
+            "mkdir -p '$remote_state_dir' && cat > '${remote_state_dir}/${name}'" \
+            < "$local_file" 2>/dev/null || rc=1
+    done
+
+    if [[ $rc -eq 0 ]]; then
+        echo "State file pushed to $REMOTE_SERVER_NAME:${remote_state_dir} ✅"
+        return 0
+    fi
+    warn "Could not push state to $REMOTE_SERVER_NAME:${remote_state_dir} — it will keep showing"
+    warn "  no partnership until this succeeds. Nothing retries this on a schedule."
+    return 1
 }
 
 read_remote_state() {
@@ -450,6 +504,7 @@ do_ssh_key_revocation() {
                 "grep -v '${our_comment}' /root/.ssh/authorized_keys \
                  > /root/.ssh/authorized_keys.tmp 2>/dev/null \
                  && mv /root/.ssh/authorized_keys.tmp /root/.ssh/authorized_keys \
+                 && chmod 600 /root/.ssh/authorized_keys \
                  && echo removed" 2>/dev/null | grep -q removed; then
                 echo "Our pubkey revoked from $REMOTE_SERVER_NAME ✅"
                 SSH_REVOKE_REMOTE_OK=true
@@ -473,9 +528,16 @@ do_ssh_key_revocation() {
         SSH_REVOKE_LOCAL_OK=true
     elif [[ -f /root/.ssh/authorized_keys ]]; then
         if grep -q "@${REMOTE_SERVER_NAME}" /root/.ssh/authorized_keys 2>/dev/null; then
+            # chmod after the mv, every time. `>` creates the temp file under the shell's umask
+            # and `mv` keeps the NEW file's mode, so this rewrite left authorized_keys 0666 on a
+            # filesystem that permits it. sshd's StrictModes then refuses every key in it without
+            # saying so to the client — the key is present, byte-correct, and inert, and the next
+            # onboard's SSH step fails with nothing in any Varaverk log to explain it. Only
+            # /var/log/syslog knows: "Authentication refused: bad ownership or modes".
             if grep -v "@${REMOTE_SERVER_NAME}" /root/.ssh/authorized_keys \
                 > /root/.ssh/authorized_keys.tmp 2>/dev/null && \
-                mv /root/.ssh/authorized_keys.tmp /root/.ssh/authorized_keys; then
+                mv /root/.ssh/authorized_keys.tmp /root/.ssh/authorized_keys && \
+                chmod 600 /root/.ssh/authorized_keys; then
                 echo "$REMOTE_SERVER_NAME pubkey revoked locally ✅"
                 SSH_REVOKE_LOCAL_OK=true
             else
@@ -515,6 +577,10 @@ gather_partner_fallback_containers() {
 
 # Start this server's own parked containers after partnership ends.
 start_own_stack() {
+    # Returns non-zero if any container failed. It used to return whatever the loop's last
+    # docker start happened to produce, so a caller checking it learned nothing — and the
+    # offboard summary just printed "Step 6 — Own stack: started" either way.
+    local _rc=0
     echo ""
     echo "━━━ $ICON_START Restart Own Stack ━━━"
     if [[ ${#PARTNERSHIP_OWN_CONTAINERS[@]} -eq 0 ]]; then
@@ -531,14 +597,20 @@ start_own_stack() {
             echo "$container started ✅"
         else
             warn "$container failed to start — check manually"
+            _rc=1
         fi
     done
+    return "$_rc"
 }
 
 # Remove partnership containers on this server + their appdata bind-mount paths.
 # Appdata paths collected via docker inspect BEFORE removal — inspect fails on removed containers.
 # Safety gate: only paths matching /mnt/*/appdata* are deleted.
 cleanup_partner_containers() {
+    # Returns non-zero if any container or appdata path could not be removed. Previously the
+    # exit status was whatever the trailing while-loop produced, so "Step 5 — Local cleanup: ✅"
+    # was printed over a container that failed to remove.
+    local _rc=0
     declare -a containers=()
     gather_partner_fallback_containers containers
 
@@ -569,8 +641,12 @@ cleanup_partner_containers() {
         if timeout "${DOCKER_TIMEOUT:-30}" docker inspect "$container" >/dev/null 2>&1; then
             timeout "${DOCKER_TIMEOUT:-30}" docker stop "$container" >/dev/null 2>&1 || true
             _PM_TRAP_STOPPED+=("$container")
-            timeout "${DOCKER_TIMEOUT:-30}" docker rm   "$container" >/dev/null 2>&1 && \
+            if timeout "${DOCKER_TIMEOUT:-30}" docker rm "$container" >/dev/null 2>&1; then
-                echo "$container removed ✅" || warn "$container rm failed"
+                echo "$container removed ✅"
+            else
+                warn "$container rm failed"
+                _rc=1
+            fi
         else
             log "$container not found — skipping"
         fi
@@ -583,8 +659,14 @@ cleanup_partner_containers() {
             warn "  DRY RUN — would rm -rf $path"
             continue
         fi
-        rm -rf "$path" && echo "  Appdata removed: $path ✅" || warn "  Failed to remove: $path"
+        if rm -rf "$path"; then
+            echo "  Appdata removed: $path ✅"
+        else
+            warn "  Failed to remove: $path"
+            _rc=1
+        fi
     done <<< "$all_appdata_paths"
+    return "$_rc"
 }
 
 # SSH to mirror — remove all containers named *-${OWNER_SHORT} (owner's deployed containers)
@@ -592,6 +674,10 @@ cleanup_partner_containers() {
 # Appdata paths collected via SSH docker inspect before removal, then deleted via SSH.
 # Safety gate: only paths matching /mnt/*/appdata* are deleted on the remote.
 cleanup_owner_containers_on_mirror() {
+    # Returns non-zero if any remote removal failed, so the caller can report Step 7 honestly
+    # rather than from MIRROR_REACHABLE — which only says the mirror answered, not that the
+    # containers on it are gone.
+    local _rc=0
     local mirror_ip="$1"
     local owner_short
     owner_short=$(derive_short_name "$OWNER")
@@ -622,24 +708,31 @@ cleanup_owner_containers_on_mirror() {
             "docker inspect --format '{{range .HostConfig.Binds}}{{println .}}{{end}}' '$container' 2>/dev/null \
              | awk -F: '{print \$1}' | grep '^/mnt/.*/appdata'" 2>/dev/null)
 
-        timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+        if timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
             -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$mirror_ip" \
             "docker stop '$container' >/dev/null 2>&1
              docker rm   '$container' >/dev/null 2>&1 && echo removed" 2>/dev/null | \
-            grep -q removed && \
+            grep -q removed; then
-            echo "$container removed from $MIRROR ✅" || \
+            echo "$container removed from $MIRROR ✅"
+        else
             warn "Failed to remove $container from $MIRROR"
+            _rc=1
+        fi
 
         # Delete appdata on remote after container removal
         while IFS= read -r path; do
             [[ -z "$path" ]] && continue
-            timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+            if timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
                 -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$mirror_ip" \
-                "rm -rf '$path' && echo removed" 2>/dev/null | grep -q removed && \
+                "rm -rf '$path' && echo removed" 2>/dev/null | grep -q removed; then
-                echo "  Appdata removed on $MIRROR: $path ✅" || \
+                echo "  Appdata removed on $MIRROR: $path ✅"
+            else
                 warn "  Failed to remove appdata on $MIRROR: $path"
+                _rc=1
+            fi
         done <<< "$appdata_paths"
     done <<< "$container_list"
+    return "$_rc"
 }
 
 # SSH to mirror — start mirror's own parked containers.
@@ -661,18 +754,23 @@ start_mirror_own_stack() {
     fi
 
     log "Restarting own stack on $MIRROR: ${mirror_own[*]}"
+    local _rc=0
     for container in "${mirror_own[@]}"; do
         if [[ "$DRY_RUN" == true ]]; then
             warn "DRY RUN — would start $container on $MIRROR"
             continue
         fi
-        timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+        if timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
             -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$mirror_ip" \
             "docker start '$container' >/dev/null 2>&1 && echo started" 2>/dev/null | \
-            grep -q started && \
+            grep -q started; then
-            echo "$container started on $MIRROR ✅" || \
+            echo "$container started on $MIRROR ✅"
+        else
             warn "$container failed to start on $MIRROR — check manually"
+            _rc=1
+        fi
     done
+    return "$_rc"
 }
 
 # Create the mirror's Emby admin account on the owner's deployed Emby.
@@ -849,28 +947,70 @@ check_both_healthy() {
     return 0
 }
 
+# Returns non-zero when the mirror did NOT leave with current state, so the caller can say so.
+# The whole point of this step is the guarantee in the summary — "mirror leaves with current
+# Critical-Data" — and it used to print that unconditionally at the end of the function. An
+# offboard with RSYNC_ENABLED=false logged "rsync globally disabled, skipping all syncs"
+# immediately followed by "Final sync complete — mirror has current state ✅", and the summary
+# scored Step 2 as a pass. The mirror left with whatever it happened to have.
 do_final_sync() {
     log "Running final critical sync..."
+    local _synced=0 _failed=0
     if [[ "$DRY_RUN" == false ]]; then
-        if [[ "${#CRITICAL_SYNC_SHARES[@]}" -gt 0 ]]; then
+        # Tier 1 only, deliberately. rsync.sh honours RSYNC_ENABLED and nothing else — the Tier 2
+        # gates belong to the orchestrators, so a direct call like this one runs whether or not
+        # CRITICAL_RSYNC_ENABLED is set. Checking Tier 2 here would refuse a final sync that works
+        # perfectly well, and the offboard's whole reason for existing is to get current auth data
+        # onto the mirror before the keys go.
+        #
+        # What was wrong was the accounting below it: a share was counted synced whenever Tier 1
+        # happened to be on, with rsync.sh's exit status ignored entirely. A transfer that failed
+        # on a full disk, an offline share or a refused connection still reported "Final sync ✅
+        # — mirror has current state", which is the one claim in this script somebody acts on.
+        local _gate_ok=true
+        if [[ "${RSYNC_ENABLED:-true}" != true ]]; then
+            warn "RSYNC_ENABLED=false — Tier 1 stops every rsync, no share can be sent"
+            _gate_ok=false
+        fi
+
+        if [[ "$_gate_ok" != true ]]; then
+            _failed=1
+        elif [[ "${#CRITICAL_SYNC_SHARES[@]}" -gt 0 ]]; then
             for _share in "${CRITICAL_SYNC_SHARES[@]}"; do
                 [[ -z "$_share" ]] && continue
                 local _path="${_share%%|*}"
                 local _profile="${_share##*|}"
+                # Count on what rsync.sh returned, now that a closed gate can no longer reach here.
                 if [[ "$_path" == "$_profile" ]]; then
                     bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$_path" --log
                 else
                     bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$_path" \
                         --profile="$_profile" --log
                 fi
+                if [[ $? -eq 0 ]]; then (( _synced++ )) || true; else (( _failed++ )) || true; fi
             done
         else
             warn "CRITICAL_SYNC_SHARES is empty — skipping final sync (configure in host*.conf)"
+            _failed=1
         fi
     else
-        warn "DRY RUN — would run final critical sync (${#CRITICAL_SYNC_SHARES[@]:-hardcoded} shares)"
+        # ${#ARR[@]} and :- cannot be combined — bash rejects the whole word as "bad substitution",
+        # so this line aborted Step 2 of every offboard --dry-run with a shell error instead of
+        # printing. A count of an unset array is already 0, which is the only default needed.
+        warn "DRY RUN — would run final critical sync (${#CRITICAL_SYNC_SHARES[@]} shares)"
+        return 0
     fi
-    warn "Final sync complete — mirror has current state ✅"
+
+    if [[ "$_failed" -gt 0 ]]; then
+        warn "Final sync did NOT complete — $MIRROR leaves with whatever state it already had"
+        [[ "${RSYNC_ENABLED:-true}" != true ]] && \
+            warn "  RSYNC_ENABLED=false — the Tier 1 gate stopped it before any share was sent"
+        [[ "${CRITICAL_RSYNC_ENABLED:-true}" != true ]] && \
+            warn "  CRITICAL_RSYNC_ENABLED=false — the Tier 2 gate stopped it"
+        return 1
+    fi
+    echo "Final sync complete — $_synced share(s) sent, mirror has current state ✅"
+    return 0
 }
 
 # Safe master.conf modification with error handling — appends the key if not already present,
@@ -1028,17 +1168,33 @@ fi
 # ==============================================================================================
 if [[ "$MODE" == "check" ]]; then
 
-    # Update last_seen_remote and offline counter based on rsync outcome
+    # Update last_seen_remote and offline counter based on rsync outcome.
+    #
+    # Both branches are flocked. The counter is a read-modify-write, so two overlapping
+    # --check cycles would otherwise both read N and both write N+1 — silently losing an
+    # increment and pushing the auto-offboard threshold further out than configured. The
+    # sed on the state file is included because write_state_file() flocks the same file
+    # from other code paths, and an unsynchronised sed -i can land mid-rewrite.
     if [[ "$REMOTE_SEEN" == true ]]; then
+        (
+            flock -x 201
             echo "0" > "$OFFLINE_COUNTER"
             if [[ -f "$LOCAL_STATE_FILE" ]]; then
                 sed -i "s|^last_seen_remote=.*|last_seen_remote=$(date '+%Y-%m-%d %H:%M:%S')|" \
                     "$LOCAL_STATE_FILE" 2>/dev/null
             fi
+        ) 201>"${OFFLINE_COUNTER}.lock"
     elif [[ "$REMOTE_UNSEEN" == true ]]; then
-        OFFLINE_COUNT=$(cat "$OFFLINE_COUNTER" 2>/dev/null || echo 0)
+        # Redirection must live INSIDE the substitution — "$( ... ) 201>file" attaches the fd
-        OFFLINE_COUNT=$(( OFFLINE_COUNT + 1 ))
+        # to the assignment, not to the subshell doing the work, and flock then fails with
-        echo "$OFFLINE_COUNT" > "$OFFLINE_COUNTER"
+        # "Bad file descriptor" while the increment silently proceeds unlocked.
+        OFFLINE_COUNT=$( {
+            flock -x 201
+            _c=$(cat "$OFFLINE_COUNTER" 2>/dev/null || echo 0)
+            _c=$(( _c + 1 ))
+            echo "$_c" > "$OFFLINE_COUNTER"
+            echo "$_c"
+        } 201>"${OFFLINE_COUNTER}.lock" )
 
         # Auto-offboard threshold: threshold_days × 48 intervals/day (every 30min)
         THRESHOLD_INTERVALS=$(( ${PARTNERSHIP_OFFLINE_THRESHOLD:-30} * 48 ))

Partnership/partnership_offboard.sh

@@ -13,28 +13,108 @@
 # OPERATIONAL MODEL
 # ==============================================================================================
 #
-# OWNER PATH (10 steps)
+# OWNER PATH (11 steps)
 #   Step 1:  Stop rsync         — halt any running sync before state changes
 #   Step 2:  Final sync         — mirror leaves with current Critical-Data state
 #   Step 3:  Reconfigure WebUIs — mirror's auth WebUIs → localhost
-#   Step 4:  Disable sync       — CRITICAL_RSYNC_ENABLED=false in master.conf
+#   Step 4:  Disarm sync gates  — RSYNC/CRITICAL_RSYNC/CONF_SYNC/ARR_SYNC=false in master.conf,
+#                                 the exact inverse of onboard Step 9c
 #   Step 5:  Local cleanup      — remove fallback coverage containers + appdata
 #   Step 6:  Restart own stack  — bring up owner's own parked containers
 #   Step 7:  Remote cleanup     — remove auth/arr stack + fallback containers from mirror
 #   Step 8:  Restart mirror     — bring up mirror's own parked containers
-#   Step 9:  Revocation         — Emby admin, SSH keys
+#   Step 9:  Emby revocation    — remove the mirror's Emby admin while Emby is still reachable
 #   Step 10: Write state        — INACTIVE locally + pushed to mirror, mirror blocklisted
-#            Tailscale          — grace window then device removal (after state written)
+#            Tailscale          — grace deadline recorded, device removed after it expires
+#   Step 11: SSH revocation     — keys, both directions. Genuinely last: it is the step that
+#                                 removes the access every step above depends on
 #
 # MIRROR PATH (8 steps)
 #   Step 1: Stop rsync          — halt any running sync
 #   Step 2: Reconfigure WebUIs  — local auth WebUIs → localhost
 #   Step 3: Remote stack clean  — remove owner-deployed containers locally (auth/arr stack)
 #   Step 4: Fallback cleanup    — remove fallback coverage containers
-#   Step 5: Disable sync        — CRITICAL_RSYNC_ENABLED=false in master.conf
+#   Step 5: Disarm sync gates   — same four gates as the owner path
 #   Step 6: Revoke Emby admin   — remove own admin account from local Emby instance
 #   Step 7: Restart own stack   — bring up own parked containers
-#   Step 8: SSH revocation      — revoke keys both directions, write state, signal owner
+#   Step 8: Finalise            — write INACTIVE, clear phase flags, signal owner, THEN revoke
+#                                 keys. Revocation is last because the signal needs the key
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Sync Before Severing
+#   The final sync (owner step 2) runs before any state change, so the mirror leaves with
+#   current Critical-Data rather than a snapshot from whenever the last scheduled sync
+#   happened. Once keys are revoked there is no second chance to move data.
+#
+# Stop the Sync First
+#   Step 1 on both paths halts rsync before anything else. A sync running through a
+#   partnership teardown would be writing to a partner that is having its access removed
+#   underneath it.
+#
+# Revoke Last, Not First
+#   SSH keys and Emby admin are revoked at the end. Every earlier step needs working remote
+#   access — revoking up front would strand the remaining cleanup on the far side and leave
+#   the mirror holding containers nobody can remove.
+#
+# Both Sides Land Somewhere Valid
+#   Each path restarts the host's own parked containers before finishing. Offboarding must
+#   leave two working standalone servers, not one working server and one stripped of the
+#   coverage it was relying on.
+#
+# Role Detected, Not Declared
+#   Owner and mirror run different sequences, and the role is derived rather than passed in.
+#   A human choosing the wrong path would run the owner's remote-cleanup steps against a
+#   server that never deployed anything.
+#
+# Blocklist Is the Enforcement
+#   Writing INACTIVE state is not enough on its own — a stale cron or a script mid-flight
+#   could still attempt a sync. The mirror is blocklisted so rsync.sh refuses it outright,
+#   independently of whatever any config still says.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Container removal, conf edits and SSH key revocation all require root.
+#
+# Lock Acquisition
+#   acquire_lock "strict" — an offboard is not resumable partway, so a second instance is
+#   refused rather than queued behind the first.
+#
+# Host Detection
+#   detect_hosts() resolves MY_ID / REMOTE_ID, which the role detection builds on.
+#
+# Docker Presence Check
+#   Verified before any container removal is attempted.
+#
+# Ordered Teardown
+#   The step sequence is the safeguard — sync, then reconfigure, then remove, then restart,
+#   then revoke, then record. Reordering breaks the guarantees above.
+#
+# Own Stack Restored
+#   Parked containers are brought back up on both sides before the run completes.
+#
+# State Written Both Ends
+#   INACTIVE is written locally and pushed to the mirror, so neither side is left believing
+#   a partnership is still active.
+#
+# Partner Blocklisted
+#   The mirror is added to the partnership blocklist, which rsync.sh checks and refuses on —
+#   stale access cannot survive the offboard.
+#
+# Tailscale Grace Window Is a Deadline, Not a Sleep
+#   Device removal happens after state is written, not before, so the final state push
+#   cannot be cut off by removing its own transport. The grace period itself is recorded to
+#   STATE_DIR/tailscale_removal_due.db and the offboard returns. It used to sleep
+#   PARTNERSHIP_GRACE_HOURS inline — six hours by default — holding the lock and its job record
+#   open the whole time, reporting "running", and blocking any re-onboard behind it.
+#
+# Dry Run Support
+#   --dry-run walks the full sequence reporting each step without executing any.
 #
 # ==============================================================================================
 # CONFIGURATION
@@ -81,6 +161,19 @@ source "$SCRIPTS_ROOT/Plugin/$PLATFORM/Partnership/containers.sh"
 
 # ── Parse flags ───────────────────────────────────────────────────────────────────────────────
 REASON="manual"
+STEP_DISABLE_RSYNC_OK=true   # both paths report it; only the mirror path re-initialised it
+TAILSCALE_REMOVED=false      # set only when remove_tailscale_device actually succeeds
+
+# Owner-path step outcomes. Every one of these was a hardcoded ✅ in the summary, or derived from
+# MIRROR_REACHABLE — which says the mirror answered a ping, not that the work on it succeeded.
+# An offboard that failed to remove a single container still reported a clean teardown.
+STEP_LOCAL_CLEANUP_OK=true
+STEP_OWN_STACK_OK=true
+STEP_REMOTE_CLEANUP_OK=true      # or "skipped" when the mirror is unreachable
+STEP_MIRROR_STACK_OK=true        # or "skipped"
+STEP_STATE_WRITE_OK=true
+STEP_STATE_PUSH_OK=false     # INACTIVE actually delivered to the mirror, or "skipped"
+STEP_SETUP_PUSH_OK=false     # cleared phase flags delivered to the mirror, or "skipped"
 FILTERED_ARGS=()
 
 for arg in "$@"; do
@@ -141,6 +234,57 @@ echo "  Reason:  $REASON"
 echo ""
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permanent changes will be made"
 
+# ==============================================================================================
+# ── HELPER: disarm every sync gate a completed onboard armed ─────────────────────────────────
+#
+# The exact inverse of partnership_onboard.sh Step 9c, which sets RSYNC_ENABLED,
+# CONF_SYNC_ENABLED and ARR_SYNC_ENABLED to true on a completed onboard. Offboard used to turn
+# off CRITICAL_RSYNC_ENABLED and nothing else, which left the far more consequential Tier 1
+# RSYNC_ENABLED switched on with no partner to sync to.
+#
+# That asymmetry read as safe and was not. Turning off one Tier 2 gate looks like "sync is off"
+# in the summary, while INTERMEDIATE, DAILY, WEEKLY and FALLBACK rsync all stayed live behind an
+# open Tier 1 — and CONF_SYNC_ENABLED kept a 4-hourly job reaching for a partner that had just
+# been removed, failing and notifying each time.
+#
+# CRITICAL_RSYNC_ENABLED stays in the list. It is a Tier 2 gate and closing Tier 1 already stops
+# it, but leaving it true would misreport the state to anyone reading the conf rather than the
+# tier logic.
+#
+# FALLBACK_ENABLED is deliberately NOT here. Onboard does not arm it, so offboard has no business
+# disarming it — it is the operator's switch, and the summary says so rather than moving it.
+# ==============================================================================================
+_VV_SYNC_GATES=(RSYNC_ENABLED CRITICAL_RSYNC_ENABLED CONF_SYNC_ENABLED ARR_SYNC_ENABLED)
+
+# ── Stop any running rsync, and say honestly whether it worked ────────────────────────────────
+#
+# Both offboard paths called "$SCRIPTS_ROOT/Rsync/rsync_stop.sh". That file has never existed —
+# rsync_stop.sh lives in System_Essentials/. With stderr sent to /dev/null the "No such file"
+# went unseen, and the line below it printed "Rsync stopped ✅" unconditionally, so every
+# offboard ever run reported stopping an rsync it had not touched. On the owner path the failed
+# exit also set STEP_STOP_OK=false, which is why the summary said ❌ two lines under a ✅.
+#
+# One helper, one path, and the outcome is the return value.
+stop_rsync_now() {
+    local script="$SCRIPTS_ROOT/System_Essentials/rsync_stop.sh"
+    if [[ ! -f "$script" ]]; then
+        warn "rsync_stop.sh not found at $script"
+        return 1
+    fi
+    bash "$script" --rsync-only
+}
+
+_disarm_sync_gates() {
+    local gate rc=0 conf="$SCRIPTS_ROOT/Configurations/master.conf"
+    for gate in "${_VV_SYNC_GATES[@]}"; do
+        # set_conf_bool, not update_master_conf: the latter rewrites the whole line and would
+        # strip the trailing comment that explains what each tier gates. Same helper onboard
+        # arms with, so arming and disarming are one operation in two directions.
+        set_conf_bool "$gate" "false" "$conf" || rc=1
+    done
+    return "$rc"
+}
+
 # ==============================================================================================
 # ── HELPER: revoke own admin account from local Emby instance ────────────────────────────────
 #
@@ -232,8 +376,12 @@ if [[ "$AM_MIRROR" == true ]]; then
     echo "━━━ $ICON_STOP Step 1/8 — Stop Rsync ━━━"
 
     if [[ "$DRY_RUN" == false ]]; then
-        bash "$SCRIPTS_ROOT/Rsync/rsync_stop.sh" --rsync-only 2>/dev/null || true
+        if stop_rsync_now; then
             echo "Rsync stopped ✅"
+        else
+            warn "Could not stop rsync — a transfer may still be running into $OWNER"
+            STEP_STOP_RSYNC_OK=false
+        fi
     else
         warn "DRY RUN — would stop rsync"
     fi
@@ -262,16 +410,14 @@ if [[ "$AM_MIRROR" == true ]]; then
 
     cleanup_partner_containers || STEP_FALLBACK_CLEANUP_OK=false
 
-    # ── Step 5: Disable critical sync ─────────────────────────────────────────────────────────
+    # ── Step 5: Disarm the sync gates ─────────────────────────────────────────────────────────
     echo ""
-    echo "━━━ $ICON_GEAR Step 5/8 — Disable Critical Sync ━━━"
+    echo "━━━ $ICON_GEAR Step 5/8 — Disarm Sync Gates ━━━"
 
     if [[ "$DRY_RUN" == false ]]; then
-        update_master_conf "CRITICAL_RSYNC_ENABLED" "false" && \
+        _disarm_sync_gates || STEP_DISABLE_RSYNC_OK=false
-            warn "CRITICAL_RSYNC_ENABLED=false ✅" || \
-            { warn "Failed to update CRITICAL_RSYNC_ENABLED"; STEP_DISABLE_RSYNC_OK=false; }
     else
-        warn "DRY RUN — would set CRITICAL_RSYNC_ENABLED=false"
+        warn "DRY RUN — would disarm ${_VV_SYNC_GATES[*]}"
     fi
 
     # ── Step 6: Revoke Emby admin locally ─────────────────────────────────────────────────────
@@ -284,19 +430,31 @@ if [[ "$AM_MIRROR" == true ]]; then
     echo ""
     echo "━━━ $ICON_SHIELD Step 8/8 — SSH Revocation + State ━━━"
 
-    do_ssh_key_revocation "${OWNER_IP:-}"
+    # State first, keys last — same ordering the owner path needed. Revocation used to run here,
-
+    # before the push below, so the mirror destroyed the key and then tried to tell the owner it
+    # had left using that key. The owner never heard, and the notify promised it would "finalise
+    # on next check" — a check that now had no way in.
     NOW=$(date '+%Y-%m-%d %H:%M:%S')
     if [[ "$DRY_RUN" == false ]]; then
         write_state_file "$LOCAL_STATE_FILE" \
             "INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
         echo "Local state: INACTIVE ✅"
         add_to_blocklist "$OWNER" "$REASON"
+
+        # Inverse of onboard's write_onboard_phase. MIRROR_ID is this host on this path, and the
+        # flags are named for the mirror on both sides, so the same names clear here.
+        _setup_db="$(platform_setup_db_path)"
+        for _flag in "${MIRROR_ID}_PHASE1_DONE" "${MIRROR_ID}_PHASE2_DONE" "${MIRROR_ID}_KEY_READY"; do
+            clear_state_var "$_setup_db" "$_flag"
+        done
+        echo "Onboard phase flags cleared ✅"
+        unset _setup_db _flag
     else
-        warn "DRY RUN — would write INACTIVE state and blocklist $OWNER"
+        warn "DRY RUN — would write INACTIVE state, clear phase flags and blocklist $OWNER"
     fi
 
     if [[ "$OWNER_REACHABLE" == true ]]; then
+        platform_push_setup_state 2>/dev/null || warn "Could not push cleared setup state to $OWNER"
         push_state_to_remote "$LOCAL_STATE_FILE" "$OWNER_IP" "$OWNER_SSH_KEY"
         notify "Partnership offboard requested by $MIRROR — $OWNER will finalise on next check" \
             "Partnership" "normal"
@@ -304,6 +462,9 @@ if [[ "$AM_MIRROR" == true ]]; then
         warn "$OWNER unreachable — state written locally, owner will see it when reachable"
     fi
 
+    # Last, for the reason above: everything before it needs the key.
+    do_ssh_key_revocation "${OWNER_IP:-}"
+
     # ── Summary ───────────────────────────────────────────────────────────────────────────────
     END=$(date +%s)
     echo ""
@@ -329,7 +490,7 @@ if [[ "$AM_MIRROR" == true ]]; then
     echo "  Step 2 — WebUIs:           $(_ok "$STEP_WEBUI_OK")"
     echo "  Step 3 — Stack cleanup:    $(_ok "$STEP_STACK_CLEANUP_OK")"
     echo "  Step 4 — Fallback cleanup: $(_ok "$STEP_FALLBACK_CLEANUP_OK")"
-    echo "  Step 5 — Disable sync:     $(_ok "$STEP_DISABLE_RSYNC_OK")"
+    echo "  Step 5 — Sync gates:       $(_ok "$STEP_DISABLE_RSYNC_OK") (${_VV_SYNC_GATES[*]} → false)"
     echo "  Step 6 — Emby revoke:      $(_ok "$STEP_EMBY_OK")"
     echo "  Step 7 — Own stack:        started"
     echo "  Step 8 — Keys revoked:     $(_revoke_status)"
@@ -368,8 +529,14 @@ echo ""
 echo "━━━ $ICON_STOP Step 1/10 — Stop Rsync ━━━"
 
 if [[ "$DRY_RUN" == false ]]; then
-    bash "$SCRIPTS_ROOT/Rsync/rsync_stop.sh" --rsync-only 2>/dev/null || STEP_STOP_OK=false
+    if stop_rsync_now; then
         echo "Rsync stopped ✅"
+    else
+        warn "Could not stop rsync — a transfer may still be running into $MIRROR"
+        warn "  Everything below changes state while data is still moving, which is the one"
+        warn "  ordering this step exists to prevent. Check: ps -ef | grep rsync"
+        STEP_STOP_OK=false
+    fi
 else
     warn "DRY RUN — would stop rsync"
 fi
@@ -402,25 +569,24 @@ else
     (( WEBUI_FAILURES++ ))
 fi
 
-# ── Step 4: Disable critical sync ─────────────────────────────────────────────────────────────
+# ── Step 4: Disarm the sync gates ─────────────────────────────────────────────────────────────
 echo ""
-echo "━━━ $ICON_GEAR Step 4/10 — Disable Critical Sync ━━━"
+echo "━━━ $ICON_GEAR Step 4/10 — Disarm Sync Gates ━━━"
 
 if [[ "$DRY_RUN" == false ]]; then
-    update_master_conf "CRITICAL_RSYNC_ENABLED" "false"
+    _disarm_sync_gates || STEP_DISABLE_RSYNC_OK=false
-    warn "CRITICAL_RSYNC_ENABLED=false ✅"
 else
-    warn "DRY RUN — would set CRITICAL_RSYNC_ENABLED=false"
+    warn "DRY RUN — would disarm ${_VV_SYNC_GATES[*]}"
 fi
 
 # ── Step 5: Local container cleanup ───────────────────────────────────────────────────────────
 echo ""
 echo "━━━ $ICON_CONTAINERS Step 5/10 — Local Container Cleanup ━━━"
 
-cleanup_partner_containers
+cleanup_partner_containers || STEP_LOCAL_CLEANUP_OK=false
 
 # ── Step 6: Restart own stack ─────────────────────────────────────────────────────────────────
-start_own_stack
+start_own_stack || STEP_OWN_STACK_OK=false
 
 # ── Step 7: Remote container cleanup ──────────────────────────────────────────────────────────
 echo ""
@@ -428,64 +594,160 @@ echo "━━━ $ICON_CONTAINERS Step 7/10 — Remote Container Cleanup ━━
 
 if [[ "$MIRROR_REACHABLE" == true ]]; then
     # Remove auth/arr stack containers deployed during onboard (by config array)
-    cleanup_deployed_stack_on_remote "$MIRROR_IP" "$MIRROR_SSH_KEY"
+    cleanup_deployed_stack_on_remote "$MIRROR_IP" "$MIRROR_SSH_KEY" || STEP_REMOTE_CLEANUP_OK=false
     # Remove fallback coverage containers (by *-owner_short naming pattern)
-    cleanup_owner_containers_on_mirror "$MIRROR_IP"
+    cleanup_owner_containers_on_mirror "$MIRROR_IP" || STEP_REMOTE_CLEANUP_OK=false
 else
     warn "$MIRROR unreachable — remote container cleanup skipped"
     warn "Run 'partnership_offboard.sh' on $MIRROR to clean up manually"
+    STEP_REMOTE_CLEANUP_OK=skipped
 fi
 
 # ── Step 8: Restart mirror's own stack ────────────────────────────────────────────────────────
 echo ""
 echo "━━━ $ICON_START Step 8/10 — Restart Mirror Stack ━━━"
 
-[[ "$MIRROR_REACHABLE" == true ]] && start_mirror_own_stack "$MIRROR_IP"
+if [[ "$MIRROR_REACHABLE" == true ]]; then
+    start_mirror_own_stack "$MIRROR_IP" || STEP_MIRROR_STACK_OK=false
+else
+    STEP_MIRROR_STACK_OK=skipped
+fi
 
-# ── Step 9: Revocation (Emby + SSH) ──────────────────────────────────────────────────────────
+# ── Step 9: Emby revocation ───────────────────────────────────────────────────────────────────
 echo ""
-echo "━━━ $ICON_SHIELD Step 9/10 — Revocation ━━━"
+echo "━━━ $ICON_SHIELD Step 9/11 — Emby Revocation ━━━"
 
-# Emby admin — before SSH key revocation while Emby still reachable
+# Before SSH key revocation, while Emby is still reachable
 [[ "$MIRROR_REACHABLE" == true ]] && revoke_emby_admin "$MIRROR_IP"
 
-# SSH key revocation — mutual, both directions; must run while Tailscale still active
-do_ssh_key_revocation "${MIRROR_IP:-}"
-
 # ── Step 10: Write state, push to mirror, blocklist ───────────────────────────────────────────
-# State is written after container cleanup and SSH revocation so that:
+# Ahead of SSH revocation, which is now Step 11.
-#   • Re-running after a crash between steps 5–9 restarts from scratch (no early-exit on INACTIVE)
+#
-#   • --check sees INACTIVE during the Tailscale grace sleep and does not re-trigger offboard
+# "State Written Both Ends" is the guarantee this step exists for — neither side left believing
+# the partnership is active. It could not deliver it: revocation used to run here in Step 9 and
+# then this push authenticated with the key it had just destroyed, so every offboard ended
+# "Could not push state file to remote — will propagate on next sync". There is no next sync;
+# the gates are closed and the keys are gone. The mirror was left reading ACTIVE for ever.
+#
+# The original ordering note said state must follow revocation so a crash between steps 5–9
+# re-runs from scratch rather than early-exiting on INACTIVE. Writing it one step later than the
+# cleanup preserves that — the cleanup is still done before any state is recorded — while
+# putting the push back inside the window where it can actually reach the mirror.
 echo ""
-echo "━━━ $ICON_GEAR Step 10/10 — Write State ━━━"
+echo "━━━ $ICON_GEAR Step 10/11 — Write State ━━━"
 
 NOW=$(date '+%Y-%m-%d %H:%M:%S')
 
 if [[ "$DRY_RUN" == false ]]; then
-    write_state_file "$LOCAL_STATE_FILE" \
+    # Checked, because this is the record every other host and every later --check reads. A
-        "INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
+    # failed write here leaves both sides believing the partnership is still active while the
+    # summary says INACTIVE — the one line in the teardown that must not be assumed.
+    if write_state_file "$LOCAL_STATE_FILE" \
+        "INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"; then
         echo "Local state: INACTIVE ✅"
+    else
+        error "Failed to write local state file — $MIRROR may still look ACTIVE here"
+        STEP_STATE_WRITE_OK=false
+    fi
     add_to_blocklist "$MIRROR" "$REASON"
-    [[ "$MIRROR_REACHABLE" == true ]] && \
+
-        push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"
+    # The inverse of onboard's write_onboard_phase, which had none. Without this a completed
+    # offboard leaves the state file reading INACTIVE beside HOST*_PHASE1_DONE / _PHASE2_DONE
+    # still set — and the setup checklist and partnership card read the flags, not the state
+    # file, so a torn-down partnership went on presenting itself as fully onboarded.
+    #
+    # KEY_READY goes too: it means "a key is generated and waiting to be installed", which stops
+    # being true the moment Step 11 revokes both sides.
+    _setup_db="$(platform_setup_db_path)"
+    for _flag in "${MIRROR_ID}_PHASE1_DONE" "${MIRROR_ID}_PHASE2_DONE" "${MIRROR_ID}_KEY_READY"; do
+        clear_state_var "$_setup_db" "$_flag"
+    done
+    echo "Onboard phase flags cleared ✅"
+    unset _setup_db _flag
+
+    if [[ "$MIRROR_REACHABLE" == true ]]; then
+        # Two separate pushes, and the summary used to report only the second. A run where the
+        # setup.db push failed and the state-file push succeeded printed "Could not push state
+        # file to remote" in the body and "Pushed to mirror: ✅" in the summary — describing
+        # different files with the same words. They are tracked apart now, because they fail
+        # apart: the mirror can be told the partnership is INACTIVE while keeping the phase
+        # flags that make its own UI still claim a finished onboard.
+        #
+        # Pushed after the flags are cleared, so the mirror receives the cleared file rather than
+        # the version that still claimed a finished onboard.
+        if platform_push_setup_state 2>/dev/null; then
+            STEP_SETUP_PUSH_OK=true
+        else
+            warn "Could not push cleared phase flags to $MIRROR — its wizard and partnership"
+            warn "  card will keep showing a completed onboard until it is reinstalled or pulled"
+        fi
+        if push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"; then
+            STEP_STATE_PUSH_OK=true
+        else
+            warn "Could not push INACTIVE to $MIRROR — it will keep reading ACTIVE until told otherwise"
+        fi
+    else
+        STEP_STATE_PUSH_OK=skipped
+        STEP_SETUP_PUSH_OK=skipped
+    fi
 else
     warn "DRY RUN — would write INACTIVE state, blocklist $MIRROR, push to remote"
+    STEP_STATE_PUSH_OK=true
+    STEP_SETUP_PUSH_OK=true
 fi
 
+# ── Step 11: SSH key revocation ───────────────────────────────────────────────────────────────
+# Genuinely last. Every step above needs working remote access — this is the one that takes it
+# away, so it goes after the final thing that has to reach the mirror.
+echo ""
+echo "━━━ $ICON_SHIELD Step 11/11 — SSH Key Revocation ━━━"
+do_ssh_key_revocation "${MIRROR_IP:-}"
+
 # Tailscale removal — after state written so --check does not re-trigger offboard during grace sleep
-if [[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]]; then
+if [[ "${PARTNERSHIP_REMOVE_TAILSCALE:-false}" == true ]]; then
     echo ""
     echo "━━━ $ICON_NET Tailscale Separation ━━━"
-    if [[ "$MIRROR_REACHABLE" == true ]]; then
+    # The grace period is recorded as a deadline, not slept through.
-        grace_seconds=$(( ${PARTNERSHIP_GRACE_HOURS:-6} * 3600 ))
+    #
-        warn "Waiting ${PARTNERSHIP_GRACE_HOURS:-6}hr grace — mirror can collect backups..."
+    # This used to `sleep $((PARTNERSHIP_GRACE_HOURS * 3600))` inline — six hours by default —
+    # holding the offboard's lock and its job record open the whole time, showing "running" to
+    # every status reader, and blocking any re-onboard behind the lock. Worse, the sleep ran
+    # even when removal was going to be a no-op: TAILSCALE_API_KEY and TAILSCALE_TAILNET are
+    # both empty here, so the six hours bought nothing at all.
+    #
+    # The offboard's own work is finished by this point. Writing the deadline lets the teardown
+    # complete now and leaves the removal to whoever reads the file — and makes the wait
+    # visible and cancellable instead of buried in a sleeping process.
+    _grace_h="${PARTNERSHIP_GRACE_HOURS:-6}"
+    if [[ "$_grace_h" -gt 0 ]] && [[ "$MIRROR_REACHABLE" == true ]]; then
+        _due=$(( $(date +%s) + _grace_h * 3600 ))
         if [[ "$DRY_RUN" == false ]]; then
-            trap 'warn "Offboard interrupted during grace sleep"; exit 0' SIGTERM SIGINT
+            printf 'host=%s\ndue=%s\ndue_human=%s\nreason=%s\n' \
-            sleep "$grace_seconds"
+                "$MIRROR" "$_due" "$(date -d "@$_due" '+%Y-%m-%d %H:%M:%S')" "$REASON" \
-            trap - SIGTERM SIGINT
+                > "${STATE_DIR}/tailscale_removal_due.db"
+        fi
+        warn "Grace period: $MIRROR stays on the tailnet until $(date -d "@$_due" '+%Y-%m-%d %H:%M') — recorded, not slept"
+        # No CLI entry point removes it yet, and there is deliberately no invented one here:
+        # remove_tailscale_device() is a partnership_manager.sh function with no --mode of its
+        # own, and it no-ops without credentials regardless. Say what is true.
+        if [[ -z "${TAILSCALE_API_KEY:-}" || -z "${TAILSCALE_TAILNET:-}" ]]; then
+            warn "Automatic removal is not possible — TAILSCALE_API_KEY/TAILSCALE_TAILNET are unset; remove it in the Tailscale admin console"
+        else
+            warn "Removal after that is not yet automated — remove it in the Tailscale admin console"
+        fi
+        TAILSCALE_REMOVED=deferred
+        unset _grace_h _due
+    else
+        # No grace configured, or the mirror is already unreachable — remove now.
+        #
+        # Outcome recorded, not assumed. remove_tailscale_device returns 1 when TAILSCALE_API_KEY
+        # or TAILSCALE_TAILNET is unset — it warns "skipping Tailscale removal" and the summary
+        # went on to report "removed ✅" anyway, so an offboard that left the device on the
+        # tailnet said it had taken it off. Neither key is configured here, so that was every run.
+        if remove_tailscale_device "$MIRROR"; then
+            TAILSCALE_REMOVED=true
         fi
     fi
-    remove_tailscale_device "$MIRROR"
 fi
 
 # Backup handover notification
@@ -524,24 +786,60 @@ _revoke_status() {
 echo "  Step 1  — Stop rsync:     $(_ok "$STEP_STOP_OK")"
 echo "  Step 2  — Final sync:     $(_ok "$STEP_SYNC_OK")"
 echo "  Step 3  — WebUI failures: $WEBUI_FAILURES"
-echo "  Step 4  — Disable sync:   ✅"
+echo "  Step 4  — Sync gates:     $(_ok "$STEP_DISABLE_RSYNC_OK") (${_VV_SYNC_GATES[*]} → false)"
-echo "  Step 5  — Local cleanup:  ✅"
+echo "  Step 5  — Local cleanup:  $(_ok "$STEP_LOCAL_CLEANUP_OK")"
-echo "  Step 6  — Own stack:      started"
+echo "  Step 6  — Own stack:      $( [[ "$STEP_OWN_STACK_OK" == true ]] && echo "started ✅" || echo "⚠️  check warnings above" )"
-echo "  Step 7  — Remote cleanup: $( [[ "$MIRROR_REACHABLE" == true ]] && echo "✅" || echo "skipped (unreachable)" )"
+echo "  Step 7  — Remote cleanup: $( [[ "$STEP_REMOTE_CLEANUP_OK" == skipped ]] && echo "skipped (unreachable)" || _ok "$STEP_REMOTE_CLEANUP_OK" )"
-echo "  Step 8  — Mirror stack:   $( [[ "$MIRROR_REACHABLE" == true ]] && echo "started" || echo "skipped (unreachable)" )"
+echo "  Step 8  — Mirror stack:   $( [[ "$STEP_MIRROR_STACK_OK" == skipped ]] && echo "skipped (unreachable)" || { [[ "$STEP_MIRROR_STACK_OK" == true ]] && echo "started ✅" || echo "⚠️  check warnings above"; } )"
-echo "  Step 9  — Keys revoked:   $(_revoke_status)"
+echo "  Step 9  — Emby revoked:   $(_ok "${STEP_EMBY_OK:-true}")"
-echo "  Step 10 — State:          INACTIVE ✅"
+echo "  Step 10 — State:          $( [[ "$STEP_STATE_WRITE_OK" == true ]] && echo "INACTIVE ✅" || echo "⚠️  WRITE FAILED — still looks ACTIVE here" )"
+echo "  Step 10 — INACTIVE pushed:  $( [[ "$STEP_STATE_PUSH_OK" == skipped ]] && echo "skipped (unreachable)" || { [[ "$STEP_STATE_PUSH_OK" == true ]] && echo "✅" || echo "⚠️  $MIRROR still reads ACTIVE"; } )"
+echo "  Step 10 — Phase flags pushed: $( [[ "$STEP_SETUP_PUSH_OK" == skipped ]] && echo "skipped (unreachable)" || { [[ "$STEP_SETUP_PUSH_OK" == true ]] && echo "✅" || echo "⚠️  $MIRROR still shows a finished onboard"; } )"
+echo "  Step 11 — Keys revoked:   $(_revoke_status)"
 echo ""
 echo "  Blocklist: $MIRROR blocked — re-onboard to permit access again ✅"
-[[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]] && \
+if [[ "${PARTNERSHIP_REMOVE_TAILSCALE:-false}" == true ]]; then
-    echo "  Tailscale:   $MIRROR removed ✅"
+    case "$TAILSCALE_REMOVED" in
+        true)     echo "  Tailscale:   $MIRROR removed ✅" ;;
+        deferred) echo "  Tailscale:   $MIRROR kept until the grace period expires — see ${STATE_DIR}/tailscale_removal_due.db" ;;
+        *)        echo "  Tailscale:   $MIRROR NOT removed ⚠ — still on the tailnet (needs TAILSCALE_API_KEY + TAILSCALE_TAILNET)" ;;
+    esac
+fi
+# Named because it is the one partnership switch neither onboard nor offboard moves, so it
+# survives an offboard still true and there is nothing else that would ever mention it.
+[[ "${FALLBACK_ENABLED:-false}" == true ]] && \
+    echo "  FALLBACK_ENABLED is still true — nothing left to fail over to, turn it off yourself"
 echo ""
 echo "  $MIRROR leaves with:"
-echo "    ✓ Current auth config (final sync)"
+# Each line reads the step that produced it. "✓ Current auth config (final sync)" was a
-echo "    ✓ Auth WebUIs → localhost"
+# literal, so an offboard whose Step 2 had just warned "Final sync did NOT complete" still
+# closed by telling the operator the mirror held current auth data. That is the one claim here
+# somebody might act on — it is the difference between a partner that can stand alone and one
+# carrying a stale copy of the auth stack.
+if [[ "$STEP_SYNC_OK" == true ]]; then
+    echo "    ✓ Current auth config (final sync)"
+else
+    echo "    ✗ Auth config NOT synced — it keeps whatever it already had (see Step 2)"
+fi
+if [[ "${WEBUI_FAILURES:-0}" -eq 0 ]]; then
+    echo "    ✓ Auth WebUIs → localhost"
+else
+    echo "    ✗ ${WEBUI_FAILURES} auth WebUI(s) still point at $LOCAL_SERVER_NAME — fix by hand there"
+fi
 echo "    ✓ ${PARTNERSHIP_GRACE_HOURS:-6}hr to collect backups"
 echo ""
-[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
+
-    warn "$ICON_DONE DONE — clean separation complete ✅"
+# The verdict is derived, never asserted. "DONE — clean separation complete ✅" printed
+# unconditionally, under a summary that had already shown two failed steps.
+if [[ "$DRY_RUN" == true ]]; then
+    warn "DRY RUN — no changes made"
+elif [[ "$STEP_STOP_OK" == true && "$STEP_SYNC_OK" == true \
+        && "$STEP_STATE_PUSH_OK" != false && "$STEP_SETUP_PUSH_OK" != false \
+        && "${WEBUI_FAILURES:-0}" -eq 0 ]]; then
+    echo "$ICON_DONE DONE — clean separation complete ✅"
+else
+    warn "$ICON_DONE Offboard finished with unresolved steps — the partnership is ended, but"
+    warn "  the ❌ lines above did not happen. Re-read them before re-onboarding."
+fi
 echo "━━━━━━━━━━━━━━━━━━━━━━━"
 exit 0

Partnership/partnership_onboard.sh

@@ -22,6 +22,12 @@
 #   Step 1:  SSH key setup         — generate keypair, install on mirror, update conf
 #   Step 1b: Docker network        — ensure varaverk docker network exists on mirror
 #   Step 1c: Share setup           — create missing Unraid shares on mirror (pool-aware, idempotent)
+#   Step 1c2:Permit mirror         — drop the mirror from the partnership blocklist a previous
+#                                    offboard wrote, or Step 1e's rsync is refused
+#   Step 1d: Sync gates            — Tier 1 + CONF_SYNC + ARR_SYNC on, every Tier 2 rsync gate
+#                                    off. Here, not at the end, because Step 1e is an rsync
+#   Step 1e: Auth appdata          — rsync PARTNERSHIP_PROVISION_SHARES to the mirror BEFORE the
+#                                    containers that read it exist. The only rsync an onboard does
 #   Step 2:  Stop mirror auth      — stop mirror's existing auth containers before replacing
 #   Step 3:  Deploy auth stack     — push XMLs, pull images, create + start on mirror
 #                                    Mariadb/Redis health-checked before Authelia deploys
@@ -32,10 +38,19 @@
 #   Step 8:  Partnership onboard   — configure WebUIs → owner IP, write state, Emby
 #   Step 9:  Arr bootstrap         — bidirectional library sync (arr_sync.sh)
 #   Step 9b: Webhook setup         — register download webhook in arrs on both servers
-#   Step 9c: Media seed            — rsync all DAILY_SYNC_SHARES to mirror (--seed)
+#   Step 9e: Webhook listener      — start listener on mirror (runs continuously, no reboot needed)
-#                                    prevents arrs treating every file as missing after bootstrap
-#   Step 9d: Webhook listener      — start listener on mirror (runs continuously, no reboot needed)
 #   Step 10: Conf push             — push master.conf + setup state to all listed hosts
+#   Step 11: Service discovery     — conf_populate.sh on the mirror, last, once the stacks it
+#                                    would discover are actually deployed there
+#   Step 12: Container grouping    — reproduce this host's folder layout on the mirror (Arrs
+#                                    Stack, Networking, Databases…) for the containers deployed
+#                                    there; only unfiled or PARTNERSHIP_FALLBACK_ONLY entries go
+#                                    to "<OwnerShort>-Fallback". Icon resolved here and passed
+#                                    over: the mirror has no Emby key
+#
+#   Phase 3 (media seed) is NOT part of the above. Onboard ends at Phase 2 — connected, running,
+#   and saying so. Seeding is a separate operator-triggered phase; see --phase3-only. With
+#   MEDIA_SEED_ENABLED=false there is no Phase 3 and the model is two phases.
 #
 # ==============================================================================================
 # DESIGN PRINCIPLES
@@ -150,16 +165,27 @@
 # Partnership/partnership_onboard.sh --skip-webhook-setup
 #     Skip webhook registration in arrs (Step 9b)
 #
-# Partnership/partnership_onboard.sh --skip-media-seed
+# Partnership/partnership_onboard.sh --phase3-only
-#     Skip initial media share rsync to mirror (Step 9c)
+#     OWNER: media seed only. Refuses unless Phase 2 is done and MEDIA_SEED_ENABLED is true.
-#     Use when mirror already has files or you want to seed manually
+#     Sets the seeding gate posture (Tier 1 open, every Tier 2 gate closed) and dispatches
+#     Rsync/media_seed.sh detached. Returns in seconds; the seed runs for weeks.
+#
+# Partnership/partnership_onboard.sh --skip-appdata-provision
+#     Skip the pre-container auth appdata rsync (Step 1e)
+#     The mirror's auth containers then start against whatever is already there
 #
 # Partnership/partnership_onboard.sh --skip-webhook-listener
-#     Skip starting webhook listener on mirror (Step 9d)
+#     Skip starting webhook listener on mirror (Step 9e)
 #     Listener will start automatically on next array restart
 #
+# Partnership/partnership_onboard.sh --no-arm
+#     Leave every sync gate as it is (Step 1d). Step 1e then has no Tier 1 and cannot provision.
+#     Use when onboarding a node you want to keep inert — a rebuild test, or a mirror whose
+#     shares are not populated yet.
+#
 # Partnership/partnership_onboard.sh --phase1-only
-#     OWNER only: SSH key exchange + conf push. Safe to run before HOST2 has Varaverk.
+#     OWNER only: SSH key exchange, conf push, docker network, partner conf cache.
+#     Safe to run before HOST2 has Varaverk — all of it needs docker and SSH, not the plugin.
 #     Writes HOST2_PHASE1_DONE=true to varaverk_setup.db.
 #
 # Partnership/partnership_onboard.sh --phase2-only
@@ -184,10 +210,13 @@ SKIP_ARR_STACK=false
 SKIP_SERVICES_STACK=false
 SKIP_ARR_SYNC=false
 SKIP_WEBHOOK_SETUP=false
-SKIP_MEDIA_SEED=false
 SKIP_WEBHOOK_LISTENER=false
-PHASE1_ONLY=false   # OWNER: SSH + conf push only (HOST2 not yet installed)
+PHASE1_ONLY=false   # OWNER: SSH + conf push only (mirror not yet installed)
+                    # MIRROR: SSH key install only, no owner notification
 PHASE2_ONLY=false   # OWNER: containers/arr/onboard only (triggered by HOST2 after it onboards)
+PHASE3_ONLY=false   # OWNER: media seed only — a separate decision, see the PHASE 3 block
+SKIP_ARM=false      # leave the sync gates as they are — see Step 1d
+SKIP_APPDATA_PROVISION=false   # skip the pre-container auth appdata sync — see Step 1e
 FILTERED_ARGS=()
 
 for arg in "$@"; do
@@ -199,10 +228,12 @@ for arg in "$@"; do
         --skip-services-stack)   SKIP_SERVICES_STACK=true ;;
         --skip-arr-sync)         SKIP_ARR_SYNC=true ;;
         --skip-webhook-setup)    SKIP_WEBHOOK_SETUP=true ;;
-        --skip-media-seed)       SKIP_MEDIA_SEED=true ;;
+        --skip-appdata-provision) SKIP_APPDATA_PROVISION=true ;;
         --skip-webhook-listener) SKIP_WEBHOOK_LISTENER=true ;;
         --phase1-only)     PHASE1_ONLY=true ;;
         --phase2-only)     PHASE2_ONLY=true; SKIP_SSH=true ;;
+        --phase3-only)     PHASE3_ONLY=true; SKIP_SSH=true ;;
+        --no-arm)          SKIP_ARM=true ;;
         *)                 FILTERED_ARGS+=("$arg") ;;
     esac
 done
@@ -238,9 +269,15 @@ write_onboard_phase() {
     local state_file="$(platform_setup_db_path)"
     [[ "$DRY_RUN" == true ]] && { warn "DRY RUN — would write ${key}=true"; return 0; }
     set_state_var "$state_file" "$key" "true"
-    platform_push_setup_state
+    # The push now reports failure, so it is warned about rather than returned. This function's
+    # contract is "the phase is recorded here" — the flag is on local disk either way, and an
+    # unreachable partner must not make a completed phase look like it did not happen.
+    platform_push_setup_state \
+        || warn "Phase flag ${key} written locally but not delivered to the partner"
+    return 0
 }
 
+
 echo ""
 echo "━━━ $ICON_FALLBACK Partnership Onboard — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
 echo ""
@@ -297,14 +334,47 @@ if [[ "$AM_MIRROR" == true ]]; then
     echo "  Mirror sets up SSH keys, then notifies Owner to run Phase 2."
     echo ""
 
+    # Tested before attempted, the same guard the owner's phase 1 has had all along.
+    #
+    # The key step is a TERMINAL step by design — ssh_setup.sh runs ssh-copy-id, which prompts for
+    # the owner's root password on a first install. Once the operator has done that in a terminal,
+    # pressing ▶ Onboard ran the whole thing again: another ssh-copy-id, this time from the WebGUI
+    # with no TTY to answer the prompt, which fails and aborts the run at Step 1 — so the button
+    # whose entire job is Step 2 could never reach it.
+    #
+    # Working SSH is the actual precondition, not "have we run the setup script". If it already
+    # works there is nothing to install, whichever route installed it.
+    OWNER_IP_PRE=$(resolve_tailscale_ip "$OWNER" 2>/dev/null || true)
     if [[ "$SKIP_SSH" == true ]]; then
         warn "Skipping SSH setup (--skip-ssh)"
+    elif [[ -n "$OWNER_IP_PRE" ]] && timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+            -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+            root@"$OWNER_IP_PRE" exit 0 2>/dev/null; then
+        echo "SSH to $OWNER already works ✅ — key already installed, skipping setup"
     elif bash "$SCRIPT_DIR/ssh_setup.sh" "${EXTRA_FLAGS[@]}"; then
         echo "SSH key ready ✅"
     else
         error "SSH key setup failed"
+        error "Install the key from a terminal on this host — ssh-copy-id needs $OWNER's password,"
+        error "and a WebGUI button has no way to answer that prompt:"
+        error "    bash $SCRIPTS_ROOT/Partnership/partnership_onboard.sh --phase1-only"
         exit 1
     fi
+    unset OWNER_IP_PRE
+
+    # Stop after the key when asked. ssh_setup.sh runs ssh-copy-id, which prompts for the
+    # owner's root password on a first install — answerable in a terminal, never from the
+    # WebGUI button, which is why the mirror's panel sends the operator to a terminal for
+    # exactly this step and nothing more. The flag was parsed but only ever honoured on the
+    # owner path, so a mirror asked for phase 1 silently ran the whole thing.
+    if [[ "$PHASE1_ONLY" == true ]]; then
+        echo ""
+        echo "━━━━━ $ICON_SUMMARY MIRROR PHASE 1 COMPLETE ━━━━━"
+        echo "  SSH key: ready"
+        echo "  Next:    press ▶ Onboard on the Partnership tab to notify $OWNER"
+        echo "━━━━━━━━━━━━━━━━━━━━━━━"
+        exit 0
+    fi
 
     echo ""
     echo "━━━ Step 2/2 — Notify Owner to Run Phase 2 ━━━"
@@ -318,17 +388,77 @@ if [[ "$AM_MIRROR" == true ]]; then
         OWNER_SCRIPTS_DIR=$(resolve_remote_scripts_dir "$OWNER_IP")
 
         if [[ "$DRY_RUN" == true ]]; then
-            warn "DRY RUN — would SSH to $OWNER ($OWNER_IP) and trigger Phase 2"
+            warn "DRY RUN — would SSH to $OWNER ($OWNER_IP) and trigger Phase 2 via run_job.sh"
-            PHASE2_TRIGGERED=true
-        elif timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
-            -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$OWNER_IP" \
-            "nohup bash '${OWNER_SCRIPTS_DIR}/Partnership/partnership_onboard.sh' --phase2-only > /tmp/vv_phase2_onboard.log 2>&1 & echo triggered" \
-            2>/dev/null | grep -q triggered; then
-            echo "Phase 2 triggered on $OWNER ✅"
-            log "Watch progress on $OWNER: tail -f /tmp/vv_phase2_onboard.log"
             PHASE2_TRIGGERED=true
         else
+            # Launched through run_job.sh, the same path cron and api/run.php use, so Phase 2
+            # gets a stat file, a job log and a Scheduler entry on the owner. It used to be a
+            # bare `nohup … > /tmp/vv_phase2_onboard.log`, which ran fine and left the owner's
+            # entire half of onboarding invisible to its own WebGUI — no job record, nothing
+            # under /var/log/varaverk, nothing for the operator to look at when asking why
+            # pressing Onboard here appeared to do nothing.
+            #
+            # setsid, not bare nohup: the job must lead its own process group so api/stop.php
+            # can signal the whole tree. api/run.php carries the same note for the same reason.
+            #
+            # Reported triggered only after the stat file proves run_job.sh actually started.
+            # The old `& echo triggered` printed unconditionally — it would have claimed success
+            # for a path that does not exist on the owner, which is exactly the failure mode a
+            # mirror in a different storage mode hits.
+            # Its own timeout, not SSH_TIMEOUT: the remote waits for the runner to prove itself,
+            # and 15s would cut that short and report a healthy launch as a failure.
+            _phase2_out=$(timeout 40 ssh -i "$SSH_KEY" \
+                -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$OWNER_IP" \
+                "bash -s -- $(printf '%q' "$OWNER_SCRIPTS_DIR")" 2>/dev/null <<'PHASE2_TRIGGER'
+sd="$1"
+runner="$sd/Plugin/unraid/run_job.sh"
+script="$sd/Partnership/partnership_onboard.sh"
+stat_file="/var/log/varaverk/Partnership/partnership_onboard.json"
+
+[ -f "$runner" ] || { echo "missing-runner:$runner"; exit 1; }
+[ -f "$script" ] || { echo "missing-script:$script"; exit 1; }
+
+# Absolute, not "newer than the file we saw a moment ago". A previous run's stat file rewritten
+# inside the same second would compare equal and read as a failed launch.
+t0=$(date +%s)
+
+setsid nohup bash "$runner" "Partnership/partnership_onboard.sh" "$script" \
+    --manual --phase2-only >/dev/null 2>&1 </dev/null &
+
+# run_job.sh writes the stat file before it execs the script, so a stat file dated at or after
+# the moment we launched is proof the runner is alive — rather than proof the ssh call returned.
+for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
+    sleep 1
+    [ -f "$stat_file" ] || continue
+    now=$(date -r "$stat_file" +%s 2>/dev/null || echo 0)
+    [ "$now" -ge "$t0" ] && { echo triggered; exit 0; }
+done
+echo start-failed
+exit 1
+PHASE2_TRIGGER
+            )
+            # Failure patterns are matched first, and they echo a path back. An owner whose
+            # SCRIPTS_DIR happened to contain the word "triggered" would otherwise satisfy a
+            # leading *triggered* glob and report success for a launch that never happened.
+            case "$_phase2_out" in
+                missing-runner:*)
+                    error "Phase 2 not started — no run_job.sh at ${_phase2_out#missing-runner:} on $OWNER"
+                    ;;
+                missing-script:*)
+                    error "Phase 2 not started — no partnership_onboard.sh at ${_phase2_out#missing-script:} on $OWNER"
+                    ;;
+                start-failed)
+                    error "Phase 2 launch on $OWNER did not produce a job record — check run_job.sh there"
+                    ;;
+                triggered)
+                    echo "Phase 2 triggered on $OWNER ✅"
+                    log "Watch on $OWNER: Scheduler tab, or tail -f /var/log/varaverk/Partnership/partnership_onboard.log"
+                    PHASE2_TRIGGERED=true
+                    ;;
+                *)
                     warn "Could not auto-trigger Phase 2 on $OWNER"
+                    ;;
+            esac
         fi
     else
         warn "Cannot resolve $OWNER Tailscale IP"
@@ -356,10 +486,129 @@ MIRROR_IP=$(resolve_tailscale_ip "$MIRROR")
 log "Mirror: $MIRROR ($MIRROR_IP)"
 [[ "$PHASE1_ONLY" == true ]] && log "Mode: Phase 1 only (SSH + conf push)"
 [[ "$PHASE2_ONLY" == true ]] && log "Mode: Phase 2 only (containers + arr + onboard)"
+[[ "$PHASE3_ONLY" == true ]] && log "Mode: Phase 3 only (media seed)"
 echo ""
 
+# ==============================================================================================
+# ── PHASE 3: MEDIA SEED ───────────────────────────────────────────────────────────────────────
+#
+# The partnership is finished at the end of Phase 2 — connected, running, and saying so. Phase 3
+# is not part of it. It is the separate decision to fill the partner's disks by rsync, and it is
+# operator-triggered because a first seed of a full library is weeks of transfer and tens of
+# terabytes of somebody else's free space.
+#
+# The seed used to be a step inside Phase 2. Inline it held the onboard open for the entire
+# transfer, which meant the phase-2 flag, the conf push, discovery and container grouping all
+# waited on it, and both hosts described an unfinished onboard over a partnership that was
+# already live. Detaching it fixed the blocking but left the seed starting on its own, which is
+# still the wrong default: the machine decided to move 28 TB because an onboard succeeded.
+#
+# With MEDIA_SEED_ENABLED=false there is no Phase 3 at all and the model is two phases. That is
+# the point of the toggle — not a seed that runs and does nothing, an absent phase.
+# ==============================================================================================
+if [[ "$PHASE3_ONLY" == true ]]; then
+    echo ""
+    echo "━━━━━ $ICON_SYNC PHASE 3 — MEDIA SEED — $MY_ID → $MIRROR ━━━━━"
+    echo ""
+
+    _master_conf="$SCRIPTS_ROOT/Configurations/master.conf"
+    _seed_gate=$(grep -m1 -E '^[[:space:]]*MEDIA_SEED_ENABLED=' "$_master_conf" 2>/dev/null \
+                 | cut -d= -f2- | cut -d'#' -f1 | tr -d '"'"'" | tr -d '[:space:]')
+
+    if [[ -n "$_seed_gate" && "$_seed_gate" != "true" ]]; then
+        error "MEDIA_SEED_ENABLED is '$_seed_gate' — there is no Phase 3 on this host"
+        error "  Onboarding is two phases while seeding is off. Arm the toggle in master.conf"
+        error "  if you want the partner's library filled by rsync."
+        exit 1
+    fi
+
+    _setup_db="$(platform_setup_db_path)"
+    if ! grep -q "^${MIRROR_ID}_PHASE2_DONE=true" "$_setup_db" 2>/dev/null; then
+        error "$MIRROR has not completed Phase 2 — seeding a partner that is not onboarded"
+        error "  would push media at a host with no containers to serve it. Finish Phase 2 first."
+        exit 1
+    fi
+    unset _setup_db
+
+    # Seeding posture, set before the transfer rather than after it. Tier 1 has to be open or
+    # rsync.sh refuses every share; every Tier 2 gate is closed so the scheduled orchestrators
+    # are not competing for the same link and the same disks for the next several weeks. This is
+    # also the posture the seed leaves behind — global on, per-orchestrator off — so finishing
+    # Phase 3 does not quietly hand the schedule back.
+    echo "━━━ Gate posture ━━━"
+    if [[ "$DRY_RUN" == true ]]; then
+        warn "DRY RUN — would set RSYNC_ENABLED true and every Tier 2 rsync gate false"
+    else
+        cp -a "$_master_conf" "${_master_conf}.bak-phase3-$(date +%Y%m%d-%H%M%S)"
+        set_conf_bool RSYNC_ENABLED "true" "$_master_conf" \
+            || { error "Could not open Tier 1 — the seed would move nothing"; exit 1; }
+        for _gate in CRITICAL_RSYNC_ENABLED INTERMEDIATE_RSYNC_ENABLED DAILY_RSYNC_ENABLED \
+                     WEEKLY_RSYNC_ENABLED MONTHLY_RSYNC_ENABLED FALLBACK_RSYNC_ENABLED; do
+            set_conf_bool "$_gate" "false" "$_master_conf" \
+                || warn "Could not close $_gate — a scheduled sync may run beside the seed"
+        done
+        unset _gate
+        echo "  Tier 1 open, every Tier 2 gate closed ✅"
+    fi
+    unset _master_conf _seed_gate
+
+    echo ""
+    echo "━━━ Dispatch ━━━"
+    _seed_script="$SCRIPTS_ROOT/Rsync/media_seed.sh"
+    _runner="$SCRIPTS_ROOT/Plugin/$PLATFORM/run_job.sh"
+    if [[ "$DRY_RUN" == true ]]; then
+        warn "DRY RUN — would dispatch Rsync/media_seed.sh"
+        exit 0
+    fi
+    if [[ ! -f "$_seed_script" || ! -f "$_runner" ]]; then
+        error "Seed runner not found at $_seed_script"
+        exit 1
+    fi
+
+    # setsid, not nohup + &: the seed must outlive this script and must not share its process
+    # group, or stopping Phase 3 takes the transfer down with it.
+    _seed_flags=()
+    [[ "$ENABLE_LOGGING" == true ]] && _seed_flags+=(--log)
+    setsid bash "$_runner" "Rsync/media_seed.sh" "$_seed_script" "${_seed_flags[@]}" \
+        >/dev/null 2>&1 < /dev/null &
+    disown 2>/dev/null || true
+
+    # Report the record, not the launch. run_job.sh writes its stat file before running the
+    # script, so a fresh "running" record is the difference between dispatched and attempted.
+    _seed_stat="/var/log/varaverk/Rsync/media_seed.json"
+    _dispatched=false
+    for _i in $(seq 1 10); do
+        if [[ -f "$_seed_stat" ]] \
+           && grep -q '"status":"running"' "$_seed_stat" 2>/dev/null \
+           && (( $(date +%s) - $(stat -c %Y "$_seed_stat") < 60 )); then
+            _dispatched=true
+            break
+        fi
+        sleep 1
+    done
+
+    echo ""
+    echo "━━━━━ $ICON_SUMMARY PHASE 3 SUMMARY ━━━━━"
+    if [[ "$_dispatched" == true ]]; then
+        echo "  Seed:     running as job Rsync/media_seed.sh ✅"
+        echo "  Shares:   ${#DAILY_SYNC_SHARES[@]}"
+        echo "  Gates:    Tier 1 open · every Tier 2 closed"
+        echo ""
+        echo "  A first full seed runs for days. Watch it on the Partnership tab; the"
+        echo "  partnership itself has been live since Phase 2 and does not depend on this."
+        echo "━━━━━━━━━━━━━━━━━━━━━━━"
+        exit 0
+    fi
+    error "  Seed did not start — check /var/log/varaverk/Rsync/media_seed.log"
+    echo "━━━━━━━━━━━━━━━━━━━━━━━"
+    exit 1
+fi
+
 STEP_SSH_OK=false
 STEP_NETWORK_OK=false
+LOCAL_SETUP_OK=true      # partnership_manager --local-only; the summary claimed done ✅ regardless
+PHASE1_NET_OK=false      # Phase 1 only — network created on the mirror before any deploy
+PHASE1_CACHE_OK=false    # Phase 1 only — our conf pushed into the mirror's RAM cache
 STEP_STOP_AUTH_OK=true
 STEP_AUTH_OK=true
 AUTH_DEPLOYED=0
@@ -375,8 +624,6 @@ SERVICES_FAILED=0
 ONBOARD_OK=false
 ARR_SYNC_OK=false
 WEBHOOK_SETUP_OK=false
-MEDIA_SEED_OK=false
-MEDIA_SEED_COUNT=0
 WEBHOOK_LISTENER_OK=false
 MASTER_PUSH_OK=false
 
@@ -443,15 +690,17 @@ if [[ "$PHASE1_ONLY" == true ]]; then
         # UI will show "key ready, install manually" state via HOST2_KEY_READY flag.
         echo ""
         echo "━━━ Phase 1 — HOST1 Local Setup (SSH pending) ━━━"
-        bash "$SCRIPT_DIR/partnership_manager.sh" --onboard --local-only "${EXTRA_FLAGS[@]}" || \
+        if ! bash "$SCRIPT_DIR/partnership_manager.sh" --onboard --local-only "${EXTRA_FLAGS[@]}"; then
+            LOCAL_SETUP_OK=false
             warn "Local setup had issues — check partnership_manager.sh output above"
+        fi
 
         END=$(date +%s)
         echo ""
         echo "━━━━━ $ICON_SUMMARY PHASE 1 — SSH PENDING ━━━━━"
         echo "  SSH keys:    key generated ✅ — NOT yet installed on $MIRROR ⚠"
         echo "  Conf push:   skipped (needs SSH access to $MIRROR)"
-        echo "  HOST1 setup: done ✅"
+        echo "  HOST1 setup: $( [[ "$LOCAL_SETUP_OK" == true ]] && echo "done ✅" || echo "⚠️  had issues — see above" )"
         echo "  Duration:    $(format_duration $(( END - START )))"
         echo ""
         echo "  ACTION NEEDED: install the key on $MIRROR:"
@@ -482,11 +731,72 @@ if [[ "$PHASE1_ONLY" == true ]]; then
         fi
     fi
 
+    # ── Phase 1 — Docker network on the mirror ────────────────────────────────────────────────
+    # Here, not only in Step 1b, because a --phase1-only run exits above and never reaches it.
+    # The mirror needs docker, not Varaverk, so this works before the plugin is installed — and
+    # creating the network now means it is in place long before the first container is deployed
+    # against it. Deploying against a missing network is what left twelve containers stuck in
+    # `Created`, so the earliest safe moment is the right one.
+    echo ""
+    echo "━━━ Phase 1 — Docker Network ($MIRROR) ━━━"
+    if [[ "$DRY_RUN" == true ]]; then
+        warn "DRY RUN — would create any networks the stack templates reference on $MIRROR"
+        PHASE1_NET_OK=true
+    elif ensure_stack_networks_on_remote "$MIRROR_IP" "$MIRROR_SSH_KEY"; then
+        PHASE1_NET_OK=true
+    else
+        warn "Network prep incomplete on $MIRROR — Step 1b retries this during Phase 2"
+    fi
+
+    # ── Phase 1 — Partner conf cache ──────────────────────────────────────────────────────────
+    # CONF_SYNC_ENABLED is armed here rather than waiting for Step 1d. It is the safe one of the
+    # three gates — it moves no data, it copies each side's host*.conf into the other's tmpfs so
+    # partner vars resolve — and the moment SSH works is the moment that should start. Leaving it
+    # until Phase 2 meant conf_sync.sh, which sources the conf fresh in its own process, exited
+    # on a closed gate every time it was called before then.
+    #
+    # Push always; pull only if the mirror actually has a conf yet. Before HOST2 installs Varaverk
+    # there is nothing to pull, and an unconditional pull would count a failure and notify about
+    # a condition that is simply "the mirror is not installed". The pull lands on the re-run after the
+    # install — the same --phase1-only --skip-ssh the operator uses to push conf again.
+    echo ""
+    echo "━━━ Phase 1 — Partner Conf Cache ━━━"
+    _conf_sync_script="$SCRIPTS_ROOT/System_Essentials/conf_sync.sh"
+    if [[ "$DRY_RUN" == true ]]; then
+        warn "DRY RUN — would arm CONF_SYNC_ENABLED and cache confs with $MIRROR"
+        PHASE1_CACHE_OK=true
+    elif [[ ! -f "$_conf_sync_script" ]]; then
+        warn "conf_sync.sh not found — skipping partner conf cache"
+    else
+        set_conf_bool CONF_SYNC_ENABLED "true" "$SCRIPTS_ROOT/Configurations/master.conf" \
+            || warn "Could not arm CONF_SYNC_ENABLED — cache step may no-op"
+
+        if bash "$_conf_sync_script" --push-only; then
+            PHASE1_CACHE_OK=true
+        else
+            warn "Could not push our conf to $MIRROR"
+        fi
+
+        _mirror_sd=$(resolve_remote_scripts_dir "$MIRROR_IP" "$MIRROR_SSH_KEY" "no")
+        if timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" \
+             -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+             root@"$MIRROR_IP" \
+             "[ -f '${_mirror_sd}/Configurations/${MIRROR_ID,,}.conf' ]" 2>/dev/null; then
+            bash "$_conf_sync_script" --pull-only || warn "Could not pull ${MIRROR_ID,,}.conf from $MIRROR"
+        else
+            echo "  $MIRROR has no ${MIRROR_ID,,}.conf yet — it will cache once Varaverk is installed there"
+        fi
+        unset _mirror_sd
+    fi
+    unset _conf_sync_script
+
     # HOST1 local setup — runs immediately without needing HOST2
     echo ""
     echo "━━━ Phase 1 — HOST1 Local Setup ━━━"
-    bash "$SCRIPT_DIR/partnership_manager.sh" --onboard --local-only "${EXTRA_FLAGS[@]}" || \
+    if ! bash "$SCRIPT_DIR/partnership_manager.sh" --onboard --local-only "${EXTRA_FLAGS[@]}"; then
+        LOCAL_SETUP_OK=false
         warn "Local setup had issues — check partnership_manager.sh output above"
+    fi
 
     [[ "$DRY_RUN" == false ]] && write_onboard_phase "$MIRROR_ID" 1
 
@@ -495,11 +805,29 @@ if [[ "$PHASE1_ONLY" == true ]]; then
     echo "━━━━━ $ICON_SUMMARY PHASE 1 COMPLETE ━━━━━"
     echo "  SSH keys:     $( [[ "$STEP_SSH_OK" == true ]] && echo "ready ✅" || echo "skipped" )"
     echo "  Conf push:    $( [[ "$CONF_PUSH_OK" == true ]] && echo "done ✅" || echo "⚠ manual needed" )"
-    echo "  HOST1 setup:  done ✅"
+    echo "  Network:      $( [[ "$PHASE1_NET_OK" == true ]] && echo "ready on $MIRROR ✅" || echo "⚠ Step 1b will retry" )"
+    echo "  Conf cache:   $( [[ "$PHASE1_CACHE_OK" == true ]] && echo "pushed to $MIRROR ✅" || echo "⚠ not cached" )"
+    echo "  HOST1 setup:  $( [[ "$LOCAL_SETUP_OK" == true ]] && echo "done ✅" || echo "⚠️  had issues — see above" )"
     echo "  Duration:     $(format_duration $(( END - START )))"
     echo ""
     echo "  HOST1 is fully set up. HOST2 ($MIRROR) can now install the Varaverk plugin."
-    echo "  The wizard will detect the pushed conf and take the correct path."
+    # The push no longer waits for a Varaverk install on the far side. It resolves the partner's
+    # conf directory across both layouts and creates the internal one if neither exists, so the
+    # conf arrives BEFORE the plugin — which is the order that makes it useful. The .plg only
+    # seeds master.conf from the template when none is present, so what lands here survives the
+    # install and the wizard reads its identity straight out of it.
+    if [[ "$CONF_PUSH_OK" == true ]]; then
+        echo "  master.conf is on $MIRROR — the wizard will find it and take the partner path,"
+        echo "  already knowing $MY_ID and $MIRROR_ID. If the operator picks flash storage there,"
+        echo "  storage_migrate.sh moves the conf to appdata with the rest of the install."
+    else
+        echo "  master.conf was NOT delivered. Phase 1 seeds it into a bare host, so this is a"
+        echo "  real failure, not the pre-install state — check SSH and that /boot is writable"
+        echo "  on $MIRROR. Retry with:"
+        echo "    • bash Partnership/partnership_onboard.sh --phase1-only --skip-ssh"
+        echo "    • or push from Scheduler → master.conf → Save Conf"
+        echo "    • or, once the plugin is installed, 'Pull from HOST1' on HOST2's Setup tab"
+    fi
     echo "  When HOST2 completes its onboard, it will automatically trigger Phase 2 here."
     echo "━━━━━━━━━━━━━━━━━━━━━━━"
     exit 0
@@ -510,18 +838,45 @@ fi
 echo ""
 echo "━━━ Step 1b — Docker Network (Mirror) ━━━"
 
-_net_script="${SCRIPTS_ROOT}/Docker_Essentials/docker_network_connect.sh"
+# Two halves, and the first is the one that matters for a fresh mirror.
+#
+# ensure_stack_networks_on_remote reads the networks out of the XMLs this onboard is about to
+# push and creates any that are missing on the mirror. It does not consult the mirror's conf,
+# because on a fresh node that array is the template default — a single commented-out entry —
+# and an empty list is indistinguishable from "no networks needed". The result was every
+# container in both stacks created against a network that did not exist.
+#
+# docker_network_connect.sh still runs afterwards: it is what *connects* the mirror's own
+# listed containers to its own listed networks, which is a different job and remains the
+# mirror's to declare.
+if ! ensure_stack_networks_on_remote "$MIRROR_IP" "$MIRROR_SSH_KEY"; then
+    warn "One or more stack networks could not be prepared on $MIRROR — deploys below may fail"
+fi
+
+# Resolve the path on the mirror, not here. This used SCRIPTS_ROOT — the OWNER's install path —
+# to name a script it then runs over SSH on the MIRROR. That only holds while both hosts install
+# to the same place, and an appdata-mode mirror is at /mnt/user/appdata/Varaverk, so bash was
+# handed a path that does not exist there and Step 1b failed on every appdata install. The
+# warning even printed the owner's path and told the operator to go check it on the mirror.
+#
+# Same fix as Steps 11 and 12, which already resolve the remote layout this way.
 if [[ "$DRY_RUN" == true ]]; then
     warn "DRY RUN — would run docker_network_connect.sh on $MIRROR"
     STEP_NETWORK_OK=true
-elif timeout 60 ssh -i "$MIRROR_SSH_KEY" \
+else
-        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
+    _mirror_sd=$(resolve_remote_scripts_dir "$MIRROR_IP" "$MIRROR_SSH_KEY" "no")
-        "bash '$_net_script'" 2>/dev/null; then
+    _net_script="${_mirror_sd}/Docker_Essentials/docker_network_connect.sh"
+    if timeout 60 ssh -i "$MIRROR_SSH_KEY" \
+            -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+            root@"$MIRROR_IP" \
+            "[ -f '$_net_script' ] || { echo missing; exit 127; }; bash '$_net_script'" 2>/dev/null; then
         echo "Docker network ready on $MIRROR ✅"
         STEP_NETWORK_OK=true
-else
+    else
         warn "docker_network_connect.sh failed on $MIRROR — containers may fail if network is missing"
         warn "Check ${_net_script} on $MIRROR and re-run with --skip-ssh if needed"
+    fi
+    unset _mirror_sd
 fi
 
 # ── Step 1c: Share setup ─────────────────────────────────────────────────────────────────────
@@ -536,6 +891,135 @@ else
     bash "$SCRIPT_DIR/share_setup.sh"
 fi
 
+# ── Step 1c2: Permit the mirror ───────────────────────────────────────────────────────────────
+# A previous offboard blocklists the mirror, and rsync.sh refuses any host on that list outright
+# — deliberately, so a stale cron cannot keep syncing to a partner that has been removed.
+#
+# The blocklist was cleared inside Step 8, which was fine while every rsync in the onboard came
+# after it. Step 1e does not: it runs before the containers are deployed, which is the whole
+# point of it. On a re-onboard that ordering means the provisioning sync is refused, the auth
+# stack comes up against empty directories, and the step built to prevent exactly that reports
+# a failure nobody can explain from its own output.
+#
+# Un-blocking here is not a loosening. The offboard's own summary says "re-onboard to permit
+# access again" — an onboard is the event that permits it, and this is where the onboard starts
+# needing it. Step 8 still runs its own clear; removing an absent entry is a no-op.
+echo ""
+echo "━━━ Step 1c2 — Permit $MIRROR ━━━"
+
+UNBLOCK_OK=true
+_blocklist="${PARTNERSHIP_BLOCKLIST_FILE:-${STATE_DIR}/partnership_blocklist.db}"
+if [[ ! -f "$_blocklist" ]] || ! grep -q "^${MIRROR}|" "$_blocklist" 2>/dev/null; then
+    echo "  $MIRROR is not blocklisted ✅"
+elif [[ "$DRY_RUN" == true ]]; then
+    warn "DRY RUN — would remove $MIRROR from the partnership blocklist"
+else
+    sed -i "/^${MIRROR}|/d" "$_blocklist" 2>/dev/null
+    if grep -q "^${MIRROR}|" "$_blocklist" 2>/dev/null; then
+        error "Could not remove $MIRROR from $_blocklist — rsync.sh will refuse it"
+        UNBLOCK_OK=false
+    else
+        echo "  Removed $MIRROR from the blocklist ✅"
+    fi
+fi
+unset _blocklist
+
+# ── Step 1d: Sync gates ───────────────────────────────────────────────────────────────────────
+# This was Step 9c, after the containers were already deployed. It has to run here instead,
+# because Step 1e below is itself an rsync and Tier 1 stops every rsync — arming afterwards
+# meant the provisioning sync exited cleanly having moved nothing.
+#
+# master.conf.template ships a fresh node inert. A successful Phase 1 is what makes Tier 1,
+# CONF_SYNC and ARR_SYNC true; without this the defaults were a one-way door and somebody had
+# to remember to hand-edit three toggles on the machine where forgetting looks exactly like
+# everything working.
+#
+# Tier 2 is now set false rather than "left as configured". Tier 1 opening is what makes the
+# scheduled jobs live, and a node that was onboarded ten seconds ago is not a node anyone has
+# checked yet — free space, share layout, what the partner actually holds. Arriving connected
+# and idle is the useful state; the operator arms each orchestrator when they mean to.
+# MEDIA_SEED_ENABLED is not touched here at all — see Step 13.
+#
+# Owner only — the mirror receives these values in the Step 10 push rather than deciding.
+ARM_OK=true
+_VV_ARM_ON=(RSYNC_ENABLED CONF_SYNC_ENABLED ARR_SYNC_ENABLED)
+_VV_ARM_OFF=(CRITICAL_RSYNC_ENABLED INTERMEDIATE_RSYNC_ENABLED DAILY_RSYNC_ENABLED
+             WEEKLY_RSYNC_ENABLED MONTHLY_RSYNC_ENABLED FALLBACK_RSYNC_ENABLED)
+echo ""
+echo "━━━ $ICON_GEAR Step 1d — Sync Gates ━━━"
+
+if [[ "$SKIP_ARM" == true ]]; then
+    echo "  --no-arm — leaving the sync gates as they are"
+    warn "  Step 1e needs Tier 1 open; with it closed the auth appdata will not be provisioned"
+elif [[ "$AM_OWNER" != true ]]; then
+    echo "  mirror — the owner's push decides these"
+elif [[ "$DRY_RUN" == true ]]; then
+    warn "DRY RUN — would set ${_VV_ARM_ON[*]} true and ${_VV_ARM_OFF[*]} false"
+else
+    _master_conf="$SCRIPTS_ROOT/Configurations/master.conf"
+    if [[ ! -f "$_master_conf" ]]; then
+        warn "master.conf not found at $_master_conf — gates left closed"
+        ARM_OK=false
+    else
+        cp -a "$_master_conf" "${_master_conf}.bak-arm-$(date +%Y%m%d-%H%M%S)"
+        for _gate in "${_VV_ARM_ON[@]}";  do set_conf_bool "$_gate" "true"  "$_master_conf" || ARM_OK=false; done
+        for _gate in "${_VV_ARM_OFF[@]}"; do set_conf_bool "$_gate" "false" "$_master_conf" || ARM_OK=false; done
+        unset _gate
+        echo "  Tier 1 open:  ${_VV_ARM_ON[*]}"
+        echo "  Tier 2 off:   every scheduled rsync — arm them yourself when you are ready"
+    fi
+    unset _master_conf
+fi
+
+# ── Step 1e: Provision the auth stack's appdata ───────────────────────────────────────────────
+# The one rsync an onboard actually needs, and it has to land before Step 3 creates the
+# containers that read it. Deploying first meant Authelia, Lldap, NPM and both databases came up
+# on the mirror against empty directories and initialised themselves from nothing — so the
+# mirror's auth stack was a fresh install wearing the owner's container names.
+#
+# The critical-data profile, not critical-fallback. critical-data stops the auth containers on
+# both sides for the duration, which costs the owner its auth for about a minute at 588 MB, and
+# buys a consistent copy. critical-fallback is the dirty variant that keeps auth running, and a
+# dirty copy is exactly what broke Redis and MariaDB on the mirror before: the manifest and the
+# binlog index were copied without the files they name, and MariaDB hides that — the container
+# reads Up while mysqld_safe restarts the dead database every few seconds.
+#
+# Media is not seeded here or anywhere in Phase 2. This share is 588 MB and the containers do
+# not start correctly without it; a media library is terabytes and nothing waits on it.
+echo ""
+echo "━━━ $ICON_SYNC Step 1e — Auth Appdata Provision ━━━"
+
+APPDATA_PROVISION_OK=false
+if [[ "$SKIP_APPDATA_PROVISION" == true ]]; then
+    warn "Skipping (--skip-appdata-provision) — the mirror's auth stack will start empty"
+elif [[ "${#PARTNERSHIP_PROVISION_SHARES[@]}" -eq 0 ]]; then
+    warn "PARTNERSHIP_PROVISION_SHARES is empty — nothing to provision"
+    warn "  Set it in host${MY_ID: -1}.conf, or the mirror's auth stack starts from nothing"
+elif [[ "$DRY_RUN" == true ]]; then
+    warn "DRY RUN — would sync ${PARTNERSHIP_PROVISION_SHARES[*]} to $MIRROR"
+    APPDATA_PROVISION_OK=true
+else
+    _prov_rc=0
+    for _share in "${PARTNERSHIP_PROVISION_SHARES[@]}"; do
+        echo "  Provisioning: $_share"
+        _prov_flags=()
+        [[ "$ENABLE_LOGGING" == true ]] && _prov_flags+=(--log)
+        if ! bash "$SCRIPTS_ROOT/Rsync/rsync.sh" "$_share" "${_prov_flags[@]}"; then
+            warn "  Failed: $_share"
+            _prov_rc=1
+        fi
+    done
+    unset _share _prov_flags
+    if [[ "$_prov_rc" -eq 0 ]]; then
+        echo "Auth appdata provisioned to $MIRROR ✅"
+        APPDATA_PROVISION_OK=true
+    else
+        warn "Auth appdata NOT fully provisioned — the containers below will start against"
+        warn "  whatever is already on $MIRROR, which on a fresh node is nothing"
+    fi
+    unset _prov_rc
+fi
+
 # ── Step 2: Stop mirror's existing auth stack ─────────────────────────────────────────────────
 echo ""
 echo "━━━ Step 2 — Stop Mirror Auth Stack ━━━"
@@ -672,46 +1156,11 @@ else
 fi
 unset _webhook_script
 
-# ── Step 9c: Seed media shares on mirror ─────────────────────────────────────────────────────
+# ── Step 9e: Start webhook listener on mirror ─────────────────────────────────────────────────
-# arr_sync.sh bootstrapped the databases — mirror's arrs now know about all content but
-# have no files on disk. Without this rsync, every imported item looks missing and arrs
-# will immediately queue searches. --seed skips the empty-remote guard and does a clean push.
-echo ""
-echo "━━━ Step 9c — Media Share Seed ━━━"
-
-if [[ "$SKIP_MEDIA_SEED" == true ]]; then
-    warn "Skipping (--skip-media-seed)"
-elif [[ "${#DAILY_SYNC_SHARES[@]}" -eq 0 ]]; then
-    warn "DAILY_SYNC_SHARES empty for $MY_ID — skipping media seed"
-    warn "Configure HOST${MY_ID: -1}_DAILY_SYNC_SHARES in host${MY_ID: -1}.conf and run Rsync/rsync.sh --seed manually"
-else
-    echo "  Seeding ${#DAILY_SYNC_SHARES[@]} share(s) to $MIRROR — this may take a while"
-    _rsync_script="$SCRIPTS_ROOT/Rsync/rsync.sh"
-    _seed_flags=(--seed)
-    [[ "$DRY_RUN"  == true ]] && _seed_flags+=(--dry-run)
-    [[ "$ENABLE_LOGGING" == true ]] && _seed_flags+=(--log)
-    for _share in "${DAILY_SYNC_SHARES[@]}"; do
-        echo "  Seeding: $_share"
-        if bash "$_rsync_script" "$_share" "${_seed_flags[@]}"; then
-            (( MEDIA_SEED_COUNT++ )) || true
-        else
-            warn "  Seed failed for $_share — re-run: Rsync/rsync.sh $_share --seed"
-        fi
-    done
-    if [[ "$MEDIA_SEED_COUNT" -gt 0 ]]; then
-        echo "Media seed complete — ${MEDIA_SEED_COUNT}/${#DAILY_SYNC_SHARES[@]} share(s) ✅"
-        MEDIA_SEED_OK=true
-    else
-        warn "Media seed: no shares completed — check errors above"
-    fi
-    unset _rsync_script _seed_flags _share
-fi
-
-# ── Step 9d: Start webhook listener on mirror ─────────────────────────────────────────────────
 # Listener is in ARRAY_START_SCRIPTS so it starts on next boot, but the mirror's array is
 # already running — kick it now so events are captured immediately after onboard.
 echo ""
-echo "━━━ Step 9d — Webhook Listener (Mirror) ━━━"
+echo "━━━ Step 9e — Webhook Listener (Mirror) ━━━"
 
 _listener_script="$SCRIPTS_ROOT/Arrs_Stack/start_webhook_listener.sh"
 if [[ "$SKIP_WEBHOOK_LISTENER" == true ]]; then
@@ -756,6 +1205,133 @@ else
     fi
 fi
 
+# ── Step 11: Service discovery on the mirror ──────────────────────────────────────────────────
+# Deliberately last. conf_populate.sh fills host*.conf from what it can actually find running —
+# arr API keys, container names, URLs — and until Step 3 and Step 5 deployed the auth and arr
+# stacks there was nothing on the mirror to find. The wizard runs it during first-run setup,
+# which on a fresh mirror is precisely the moment the machine is still empty, so everything it
+# could have discovered was discovered as absent.
+#
+# No --overwrite: it only fills blanks, so anything the operator set by hand survives. --no-push
+# because Step 10 above has just pushed conf; letting discovery push again would race it.
+echo ""
+echo "━━━ $ICON_GEAR Step 11 — Service Discovery ($MIRROR) ━━━"
+
+POPULATE_OK=false
+# MIRROR_IP, not MIRROR_REACHABLE — the latter is partnership_offboard.sh's variable and does not
+# exist in this script, so the test was always true against an empty string and Step 11 reported
+# "skipped (unreachable)" on a mirror it had just deployed twelve containers to.
+if [[ -z "${MIRROR_IP:-}" ]]; then
+    warn "$MIRROR has no resolved IP — skipping discovery, run Deployment/conf_populate.sh there later"
+    POPULATE_OK=skipped
+elif [[ "$DRY_RUN" == true ]]; then
+    warn "DRY RUN — would run conf_populate.sh --no-push on $MIRROR"
+    POPULATE_OK=true
+else
+    _mirror_sd=$(resolve_remote_scripts_dir "$MIRROR_IP" "$MIRROR_SSH_KEY" "no")
+    _pop_script="${_mirror_sd}/Deployment/conf_populate.sh"
+    if timeout 180 ssh -i "$MIRROR_SSH_KEY" \
+         -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+         root@"$MIRROR_IP" \
+         "[ -f '$_pop_script' ] || { echo missing; exit 127; }; bash '$_pop_script' --no-push" 2>/dev/null; then
+        echo "Discovery complete on $MIRROR ✅"
+        POPULATE_OK=true
+    else
+        warn "Discovery failed on $MIRROR — run $_pop_script there by hand"
+    fi
+    unset _mirror_sd _pop_script
+fi
+
+# ── Step 12: Group our containers on the mirror ───────────────────────────────────────────────
+# The mirror now runs a dozen containers that are ours, scattered among its own. This files them
+# under one folder named after us — "<OwnerShort>-Fallback" — matching the convention the owner
+# already keeps for the mirror's containers.
+#
+# The icon is resolved HERE and passed over, not looked up there. It comes from the closest Emby
+# user to our own name, and the mirror has neither our Emby key nor necessarily an Emby at all —
+# so a lookup on that side would find nothing and the folder would come up blank.
+#
+# Not fatal in any direction: folder.view3 absent on the mirror is a clean skip, and a folder
+# without a picture is still a folder.
+echo ""
+echo "━━━ $ICON_GEAR Step 12 — Container Grouping ($MIRROR) ━━━"
+
+FOLDER_OK=false
+if [[ -z "${MIRROR_IP:-}" ]]; then
+    warn "$MIRROR has no resolved IP — skipping container grouping"
+    FOLDER_OK=skipped
+else
+    mapfile -t _deployed < <(deployed_stack_container_names)
+    _deployed_csv=$(IFS=,; echo "${_deployed[*]}")
+    if [[ -z "$_deployed_csv" ]]; then
+        log "No stack templates resolved to container names — nothing to group"
+        FOLDER_OK=skipped
+    elif [[ "$DRY_RUN" == true ]]; then
+        warn "DRY RUN — would create ${MY_ID}-named fallback folder on $MIRROR with: $_deployed_csv"
+        FOLDER_OK=true
+    else
+        _ff_local="$SCRIPTS_ROOT/Plugin/$PLATFORM/Tools/fallback_folder.php"
+        _mf_local="$SCRIPTS_ROOT/Plugin/$PLATFORM/Tools/mirror_folders.php"
+        _icon=$(php "$_ff_local" --host="$MY_ID" --icon-only 2>/dev/null || true)
+        [[ -z "$_icon" ]] && log "No icon resolved for $MY_ID — folder will be created without one"
+
+        _mirror_sd=$(resolve_remote_scripts_dir "$MIRROR_IP" "$MIRROR_SSH_KEY" "no")
+        _ff_remote="${_mirror_sd}/Plugin/${PLATFORM}/Tools/fallback_folder.php"
+        _mf_remote="${_mirror_sd}/Plugin/${PLATFORM}/Tools/mirror_folders.php"
+
+        # ── 12a: reproduce our own folder layout on the mirror ────────────────────────────────
+        # The deployed stacks run on the mirror continuously — they are not failover coverage —
+        # so they belong on the same shelves they occupy here: Sonarr in "Arrs Stack", NPM and
+        # Lldap in "Networking", the databases in "Databases". Filing all of them under
+        # "<Owner>-Fallback", which is what this step used to do, records whose they are and
+        # nothing about what they do.
+        #
+        # The plan is computed here, where the owner's folder.view3 layout lives, and applied
+        # there. Whatever the layout does not account for comes back as "unfiled" and is what
+        # the fallback folder is actually for.
+        _unfiled="$_deployed_csv"
+        if [[ -f "$_mf_local" ]]; then
+            # Comma-joined: the flag takes a CSV, and "${arr[*]}" would join on spaces — which
+            # also happen to appear inside folder names, so the plan must never be reflowed.
+            _fbonly=$(IFS=,; printf '%s' "${PARTNERSHIP_FALLBACK_ONLY[*]:-}")
+            _plan=$(php "$_mf_local" --export --containers="$_deployed_csv" \
+                        --fallback-only="$_fbonly" 2>/dev/null)
+            if [[ -n "$_plan" ]]; then
+                if printf '%s' "$_plan" | timeout 60 ssh -i "$MIRROR_SSH_KEY" \
+                     -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+                     root@"$MIRROR_IP" \
+                     "[ -f '$_mf_remote' ] || exit 127; php '$_mf_remote' --import" 2>/dev/null; then
+                    _unfiled=$(printf '%s' "$_plan" | php -r \
+                        'echo implode(",", json_decode(stream_get_contents(STDIN),true)["unfiled"] ?? []);' 2>/dev/null)
+                else
+                    warn "Could not mirror the folder layout to $MIRROR — falling back to one folder"
+                fi
+            fi
+            unset _plan _fbonly
+        fi
+
+        # ── 12b: the fallback folder gets only what is genuinely fallback ─────────────────────
+        # Unfiled containers plus anything named in PARTNERSHIP_FALLBACK_ONLY. Empty is the
+        # normal, correct outcome when every deployed container has a home in the layout.
+        if [[ -z "$_unfiled" ]]; then
+            echo "  Folder layout mirrored — nothing left for ${MY_ID}-Fallback ✅"
+            FOLDER_OK=true
+        elif timeout 60 ssh -i "$MIRROR_SSH_KEY" \
+             -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+             root@"$MIRROR_IP" \
+             "[ -f '$_ff_remote' ] || { echo missing; exit 127; }
+              php '$_ff_remote' --host=$(printf '%q' "$MY_ID") \
+                  --containers=$(printf '%q' "$_unfiled") \
+                  --icon=$(printf '%q' "$_icon")" 2>/dev/null; then
+            FOLDER_OK=true
+        else
+            warn "Could not group containers on $MIRROR — run $_ff_remote there by hand"
+        fi
+        unset _ff_local _ff_remote _mf_local _mf_remote _mirror_sd _icon _unfiled
+    fi
+    unset _deployed _deployed_csv
+fi
+
 # ── Write Phase 2 completion state ────────────────────────────────────────────────────────────
 [[ "$ONBOARD_OK" == true && "$DRY_RUN" == false ]] && write_onboard_phase "$MIRROR_ID" 2
 
@@ -773,6 +1349,9 @@ _skip() { [[ "$1" == true ]] && echo "skipped" || echo "$(_ok "$2")"; }
 
 echo "  Step 1  — SSH keys:         $(_skip "$SKIP_SSH"             "$STEP_SSH_OK")"
 echo "  Step 1b — Docker network:   $(_ok "$STEP_NETWORK_OK")"
+echo "  Step 1c2— Permit mirror:    $(_ok "$UNBLOCK_OK")"
+echo "  Step 1d — Sync gates:       $( [[ "$SKIP_ARM" == true ]] && echo "skipped (--no-arm)" || { [[ "$AM_OWNER" != true ]] && echo "mirror — set by owner" || _ok "$ARM_OK"; } )"
+echo "  Step 1e — Auth appdata:     $( [[ "$SKIP_APPDATA_PROVISION" == true ]] && echo "skipped" || _ok "$APPDATA_PROVISION_OK" )"
 echo "  Step 2  — Stop auth:        $(_skip "$SKIP_AUTH_STACK"       "$STEP_STOP_AUTH_OK")"
 echo "  Step 3  — Auth stack:       $( [[ "$SKIP_AUTH_STACK" == true ]] && echo "skipped" || echo "${AUTH_DEPLOYED} deployed, ${AUTH_FAILED} failed" )"
 echo "  Step 4  — Stop arr:         $(_skip "$SKIP_ARR_STACK"        "$STEP_STOP_ARR_OK")"
@@ -782,9 +1361,10 @@ echo "  Step 7  — Services stack:   $( [[ "$SKIP_SERVICES_STACK" == true ]] &&
 echo "  Step 8  — Onboard:          $(_ok "$ONBOARD_OK")"
 echo "  Step 9  — Arr bootstrap:    $( [[ "$SKIP_ARR_SYNC" == true || "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$ARR_SYNC_OK")" )"
 echo "  Step 9b — Webhook setup:    $(_skip "$SKIP_WEBHOOK_SETUP"    "$WEBHOOK_SETUP_OK")"
-echo "  Step 9c — Media seed:       $( [[ "$SKIP_MEDIA_SEED" == true ]] && echo "skipped" || echo "${MEDIA_SEED_COUNT}/${#DAILY_SYNC_SHARES[@]} shares $(_ok "$MEDIA_SEED_OK")" )"
+echo "  Step 9e — Webhook listener: $(_skip "$SKIP_WEBHOOK_LISTENER" "$WEBHOOK_LISTENER_OK")"
-echo "  Step 9d — Webhook listener: $(_skip "$SKIP_WEBHOOK_LISTENER" "$WEBHOOK_LISTENER_OK")"
 echo "  Step 10 — Conf push:        $( [[ "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$MASTER_PUSH_OK")" )"
+echo "  Step 11 — Discovery:        $( [[ "$POPULATE_OK" == skipped ]] && echo "skipped (unreachable)" || _ok "$POPULATE_OK" )"
+echo "  Step 12 — Grouping:         $( [[ "$FOLDER_OK" == skipped ]] && echo "skipped" || _ok "$FOLDER_OK" )"
 echo ""
 
 if [[ "$ONBOARD_OK" == true ]]; then

Partnership/partnership_transfer.sh

@@ -28,6 +28,31 @@
 #   Step 5:  Write State          — ACTIVE written locally and pushed to new mirror
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Config Changes Hands, Data Does Not
+#   No containers move and no appdata is copied. A transfer rewrites who is authoritative
+#   and where WebUIs point; the sync direction reverses naturally on the next fallback or
+#   critical-sync cycle. Keeping data out of the transfer is what makes it cheap enough to
+#   be reversible.
+#
+# Owner Initiates, Always
+#   Only the current owner can run this. The owner holds the authoritative config, so a
+#   mirror-initiated transfer would be writing ownership state it does not own — and if both
+#   sides ran it, neither would be owner.
+#
+# Prove Health Before Swapping
+#   Both servers must pass consecutive health checks first. Handing ownership to a partner
+#   that is unhealthy converts a recoverable situation into an outage with the authoritative
+#   side on the weaker host.
+#
+# Roles Swap Atomically
+#   Owner and mirror are two ends of one relationship, not independent flags. Any window
+#   where both believe they are owner — or neither does — is worse than the transfer simply
+#   failing, so the swap is written as one transition rather than two updates.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #

Partnership/share_setup.sh

@@ -25,7 +25,96 @@
 # Remote admin assigns disk sets from the Unraid UI after onboarding.
 #
 # ==============================================================================================
-# USAGE
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Resolve the mirror and its Tailscale IP — unresolvable aborts before any remote call
+#   2. Detect the remote's appdata cache pool from its own appdata.cfg (default: cache)
+#   3. For each path across the owner's daily/weekly/critical/intermediate share lists:
+#        a. Extract the top-level Unraid share name
+#        b. Remote already has a .cfg for it? → skip, never modify
+#        c. Otherwise use the LOCAL .cfg as a template:
+#             - substitute the remote's detected pool
+#             - clear disk include/exclude (disk layouts differ per server)
+#        d. Write the .cfg to remote /boot/config/shares/ and mkdir -p the share directory
+#        e. Sub-paths (e.g. appdata-Fallback/Critical-Data) get their subdir created after
+#           the top-level share exists
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Create Only, Never Modify
+#   An existing remote .cfg is always left alone. The remote admin may have deliberately
+#   tuned a share's pool, allocation or disk set — this script has no way to tell an
+#   intentional setting from a stale one, so it never overwrites.
+#
+# Disk Layout Is Not Portable
+#   Include/exclude lists are cleared rather than copied, because the two servers have
+#   different disks. Copying the owner's disk set onto a mirror with a different array
+#   would produce a share pointing at disks that do not exist.
+#
+# Detect the Pool, Do Not Assume It
+#   The remote's cache pool name is read from its own appdata.cfg rather than hardcoded or
+#   copied from local. Pool names differ per server and a wrong one silently lands appdata
+#   on the array.
+#
+# Media Shares Land Array-Only
+#   Shares with shareUseCache=no are created without a pool assignment. Disk sets are the
+#   remote admin's call from the Unraid UI after onboarding.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Reads $SSH_KEY from /root/.ssh and the local /boot/config/shares/*.cfg, and writes share
+#   configs onto the mirror as root.
+#
+# Lock Acquisition
+#   acquire_lock prevents concurrent runs. Two instances could both observe a share as
+#   missing and race to create it.
+#
+# Host Detection
+#   detect_hosts() resolves REMOTE_SERVER_NAME — the mirror this script targets.
+#
+# Mirror Resolution Guard
+#   Aborts if REMOTE_SERVER_NAME is unset or its Tailscale IP cannot be resolved, before any
+#   remote command is attempted.
+#
+# Existing Share Protection
+#   A remote .cfg that already exists is never touched — see Create Only above.
+#
+# SSH Timeouts and BatchMode
+#   Every remote call uses ConnectTimeout and BatchMode=yes, so an unreachable or
+#   password-prompting mirror fails fast instead of hanging the onboarding run.
+#
+# Pool Fallback
+#   An undetectable remote pool defaults to "cache" rather than writing an empty pool name
+#   into the share config.
+#
+# Dry Run Support
+#   --dry-run reports every share it would create and writes nothing.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# host*.conf (aliased by detect_hosts())
+#
+#   HOST*_DAILY_SYNC_SHARES / _WEEKLY_ / _CRITICAL_ / _INTERMEDIATE_
+#       The share lists this script reads. Any path appearing in one of these on the owner
+#       is a share the mirror is expected to have.
+#
+#   SSH_KEY
+#       Key used for every remote call. Written by ssh_setup.sh.
+#
+# master.conf
+#
+#   HOST* — hostnames, used to resolve the mirror via detect_hosts()
+#
+# ==============================================================================================
+# RUNTIME MODES
 # ==============================================================================================
 #
 # share_setup.sh
@@ -40,9 +129,21 @@ set -uo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "$SCRIPT_DIR/../load_config.sh"
-detect_hosts
 parse_args "$@"
 
+# ==============================================================================================
+# ━━━ Setup ━━━
+# ==============================================================================================
+# Reads $SSH_KEY from /root/.ssh, reads local /boot/config/shares/*.cfg, and writes share
+# configs onto the mirror as root.
+[[ "$EUID" -ne 0 ]] && { error "Must be run as root"; exit 1; }
+
+# Writes share .cfg files to the mirror. Two concurrent runs could both see a share as
+# missing and race to create it.
+acquire_lock
+
+detect_hosts
+
 # ── Resolve mirror ────────────────────────────────────────────────────────────────────────────
 
 [[ -z "${REMOTE_SERVER_NAME:-}" ]] && { error "Cannot determine mirror hostname — check HOST* in master.conf"; exit 1; }

Partnership/ssh_setup.sh

@@ -2,40 +2,143 @@
 # ==============================================================================================
 # ============================= SSH Setup ======================================================
 # ==============================================================================================
+#
+# PURPOSE
+# ─────────────────────────────────────────────────────────────────────────────
 # Generates the SSH keypair for rsync automation and installs it on the remote server.
+# Every cross-host operation in the ecosystem — rsync, conf sync, fallback container
+# control, the upgrade webhook — authenticates with this key. If it is missing or broken,
+# the mesh silently degrades to single-host.
+#
 # Key named after this server: hostname lowercased, unraid- prefix stripped.
 #   unRAID-Gmer4Lfe  → gmer4lfe_rsync_automation
-#   unRAID-Jayred365 → jayred365_rsync_automation
+#   unRAID-Jayred36  → jayred36_rsync_automation
-# Idempotent — skips generation if key already exists (use --force to regenerate).
-# Updates host*.conf with key path on success.
 #
-# ── MODES ─────────────────────────────────────────────────────────────────────────────────────
+# The name comes from the OS hostname, never from Tailscale. HOST2 answers to both
-#   (default)       — generate key if missing, copy to remote, update conf
+# `unRAID-Jayred36` (hostname -s — Unraid truncates Server Name to the 15-char NetBIOS limit)
-#   --force         — regenerate key even if it exists, re-copy to remote
+# and `unraid-jayred365` (its Tailscale peer name), and only the first one decides this filename.
-#   --validate      — test SSH auth to remote, track strikes, notify at limit
+# api/setup.php derives the same path independently when it pulls master.conf, from the same
-#   --status        — show key state, fingerprint, remote connectivity
+# source — so a key created under the Tailscale spelling is a key neither of them will find.
-#   --dry-run       — preview without creating, copying, or updating conf
+#
+# Idempotent — skips generation if the key already exists (--force to regenerate).
+# Updates host*.conf with the key path on success.
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Setup (default):
+#   1. Key exists? → skip generation unless --force
+#   2. Generate keypair, named from this host
+#   3. Copy the public key to the remote's authorized_keys
+#   4. Verify authentication actually works before claiming success
+#   5. Write the key path into host*.conf
+#
+# Validate (--validate), called during partnership --check cycles:
+#   Remote unreachable on Tailscale → network problem, NOT counted as a strike
+#   Remote reachable but SSH auth fails → key problem, strike incremented
+#   Clean connectivity for SSH_STRIKE_RESET_HRS → strikes reset automatically
+#   At SSH_MAX_STRIKES → notify and return exit 2 so the caller can escalate
 #
-# ── STRIKE SYSTEM (--validate) ────────────────────────────────────────────────────────────────
-# Called during partnership --check cycles to detect broken SSH auth.
-# Tracks consecutive SSH auth failures — not network unreachability.
-# Remote Tailscale IP unreachable = network issue → not counted as SSH strike.
-# Remote reachable but SSH auth fails = key issue → strike incremented.
-# Strikes reset automatically after SSH_STRIKE_RESET_HRS of clean connectivity.
-# At SSH_MAX_STRIKES: notify + return exit 2 (caller can escalate).
 #   State: DATA_DIR/ssh_strikes_{REMOTE_SERVER_NAME}.db
 #
-# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
+# ==============================================================================================
-#   SSH_MAX_STRIKES       — consecutive failures before notifying (default 5)
+# DESIGN PRINCIPLES
-#   SSH_STRIKE_RESET_HRS  — hours since last failure before counter resets (default 24)
+# ==============================================================================================
+#
+# Distinguish Unreachable From Unauthorised
+#   The strike system counts SSH auth failures only. A partner that is simply offline is a
+#   network condition, and counting it would fire a key-rotation alarm every time the remote
+#   reboots. Only "I can reach you but you will not let me in" is a key problem.
+#
+# Idempotent by Default, Destructive Only on Request
+#   A bare run never replaces an existing key. Regenerating invalidates every authorized_keys
+#   entry the old key was in — including on hosts this script is not talking to right now —
+#   so it requires --force explicitly.
+#
+# Verify Before Recording
+#   The key path is written into host*.conf only after authentication has been proven to
+#   work. Recording a key that does not authenticate would leave every downstream script
+#   pointed at a credential that silently fails.
+#
+# Strikes Reset on Recovery
+#   Counters clear themselves after a period of clean connectivity, so a transient outage
+#   does not accumulate toward an alarm across unrelated weeks.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Reads and writes /root/.ssh and installs keys on the remote as root.
+#
+# Lock Acquisition
+#   acquire_lock prevents concurrent runs. Two instances generating or copying keys at once
+#   could leave authorized_keys holding a key whose private half was already replaced.
+#
+# Host Detection
+#   detect_hosts() resolves MY_ID / REMOTE_ID for key naming and remote targeting.
+#
+# Existing Key Protection
+#   Generation is skipped when a key is present. Overwriting requires --force.
+#
+# Auth Verified Before Conf Write
+#   host*.conf is updated only after a successful authentication test.
+#
+# Network-vs-Auth Discrimination
+#   Unreachable remotes never increment the strike counter — see Design Principles.
+#
+# Strike Ceiling
+#   SSH_MAX_STRIKES bounds how long a genuinely broken key goes unreported, and exit 2 lets
+#   the caller decide whether that is escalation-worthy.
+#
+# Local-Only Escape Hatch
+#   --local-only generates the key without touching the remote, for onboarding a partner
+#   that is not reachable yet.
+#
+# Dry Run Support
+#   --dry-run previews generation, copy and conf update without performing any.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   SSH_MAX_STRIKES
+#       Consecutive SSH auth failures before notifying (default: 5)
+#
+#   SSH_STRIKE_RESET_HRS
+#       Hours of clean connectivity before the strike counter resets (default: 24)
+#
+# host*.conf
+#
+#   HOST*_SSH_KEY
+#       Written by this script on success. Read by rsync.sh, conf_sync.sh, fallback.sh and
+#       upgrade_webhook_handler.sh — every cross-host operation depends on it.
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# ssh_setup.sh
+#     Initial setup — generate if missing, copy to remote, update conf. Idempotent.
+#
+# ssh_setup.sh --force
+#     Regenerate the key even if one exists, and re-copy to the remote.
+#
+# ssh_setup.sh --validate
+#     Health check with strike tracking. Exit 2 at the strike limit.
+#
+# ssh_setup.sh --status
+#     Show key state, fingerprint, and remote connectivity. Then exit.
+#
+# ssh_setup.sh --local-only
+#     Generate the key locally and skip the remote copy.
+#
+# ssh_setup.sh --dry-run / --log
+#     Supported by every mode above.
 #
-# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
-#   Partnership/ssh_setup.sh                — initial setup (idempotent)
-#   Partnership/ssh_setup.sh --force        — regenerate + re-copy
-#   Partnership/ssh_setup.sh --validate     — health check + strike tracking
-#   Partnership/ssh_setup.sh --status       — show key and connectivity state
-#   Partnership/ssh_setup.sh --local-only   — generate key locally, skip remote copy
-#   Any mode supports --dry-run and --log
 # ==============================================================================================
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -352,6 +455,22 @@ else
     if ssh-copy-id -i "$SSH_PUB_PATH" -o ConnectTimeout="${SSH_TIMEOUT:-15}" \
         root@"$REMOTE_SERVER" 2>/dev/null; then
         echo "Public key installed on $REMOTE_SERVER_NAME ✅"
+
+        # Tighten the file ssh-copy-id just wrote, or the key it installed is inert.
+        #
+        # On Unraid /root/.ssh is a symlink to /boot/config/ssh/root, and a file created there
+        # comes out 0666. sshd runs StrictModes by default and refuses to read a group- or
+        # world-writable authorized_keys — it logs "Authentication refused: bad ownership or
+        # modes" and falls through to password auth, which BatchMode then declines.
+        #
+        # The failure is silent from this end: ssh-copy-id succeeds and reports success, the key
+        # is present and byte-correct, and every later connection is still refused. Observed
+        # exactly that — key installed at 07:15:12, onboard refused at 07:15:43, blobs identical.
+        ssh -o ConnectTimeout="${SSH_TIMEOUT:-15}" -o BatchMode=yes -o StrictHostKeyChecking=no \
+            -i "$SSH_KEY_PATH" root@"$REMOTE_SERVER" \
+            'chmod 700 "$(dirname "$(readlink -f ~/.ssh/authorized_keys)")" 2>/dev/null
+             chmod 600 "$(readlink -f ~/.ssh/authorized_keys)" 2>/dev/null' 2>/dev/null \
+          || warn "Could not tighten authorized_keys on $REMOTE_SERVER_NAME — if auth is refused, chmod 600 it there"
     else
         error "ssh-copy-id failed — check that:"
         error "  1. Remote server is reachable: tailscale status"

Plugin/build.sh

@@ -26,6 +26,73 @@
 #   rebuild cron when the array mounts. Scripts stay on appdata (git clone).
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Resolve version — first argument, else today's date as YYYY.MM.DD
+#   2. Verify makepkg exists (Slackware tooling — this only runs on unRAID)
+#   3. Stage Plugin/unraid/ into the install layout the package expects
+#   4. makepkg the stage into Plugin/dist/<name>-<version>-noarch-1.txz
+#   5. sha256sum the result and write the .sha256 alongside it
+#   6. Rewrite <!ENTITY version> and <!ENTITY sha256> in varaverk.plg so the .plg
+#      always points at the package just built
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Release Path, Not the Dev Path
+#   Day-to-day work uses plugin_setup.sh, which symlinks the source directory so edits are
+#   live immediately. This produces the artifact unRAID reinstalls from flash on every boot.
+#   Keeping the two separate is what allows fast iteration without shipping half-finished
+#   files into a package.
+#
+# The .plg Always Matches the Package
+#   Version and checksum are written into varaverk.plg in the same run that produces the
+#   .txz. A .plg pointing at a checksum it was not built against fails to install with a
+#   mismatch error, so the two are never updated independently.
+#
+# Date Version by Default
+#   An omitted version becomes today's date. Builds are therefore always ordered and always
+#   unique without anyone maintaining a counter.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# No Root — Deliberate
+#   Writes only into Plugin/dist/ and rewrites varaverk.plg inside the repo. It installs
+#   nothing and touches nothing under /usr or /boot, so it needs no privilege. Requiring
+#   root would mean building release artifacts as root for no reason.
+#
+# makepkg Presence Check
+#   Aborts immediately if makepkg is missing, with a note that this runs on unRAID. The
+#   Slackware toolchain is not present on a normal dev machine and a partial build would be
+#   worse than a clear refusal.
+#
+# Checksum Written From the Real Artifact
+#   The sha256 is computed from the .txz that was just produced, never assumed or carried
+#   forward, so the .plg cannot advertise a checksum for a different build.
+#
+# Staged Build
+#   The package is assembled from a staging directory rather than from the live source
+#   tree, so an in-progress edit cannot end up inside a release artifact.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# None — this is a build tool, not a runtime script. It reads no conf and calls no
+# load_config.sh.
+#
+# Inputs are positional and derived:
+#   $1                  version string (default: today, YYYY.MM.DD)
+#   Plugin/unraid/      source tree that gets packaged
+#   Plugin/varaverk.plg rewritten in place with the new version and checksum
+#
+# Output: Plugin/dist/<name>-<version>-noarch-1.txz and its .sha256
+#
+# ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================
 #

Plugin/plugin_setup.sh

@@ -14,9 +14,52 @@
 # immediately without re-running this script.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Detect the OS from /etc/*-version markers
+#   2. Resolve Plugin/<os>/ as the source — missing means abort before any change
+#   3. Inspect the web server's plugin location:
+#        symlink already        → remove it, it will be recreated
+#        real directory         → refuse, this is an existing install to clean up by hand
+#        absent                 → proceed
+#   4. Create the symlink
+#
+# After this, every edit under Plugin/<os>/ is live immediately — that is the whole point,
+# and why this is the development path rather than build.sh's packaged one.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Symlink, Not Copy
+#   The web server points at the repo rather than holding its own copy, so there is exactly
+#   one source of truth for the plugin's PHP. A copy would drift the moment anyone edited
+#   either side, and the drift would only surface as a UI behaving unlike the code.
+#
+# Run Once, By Hand
+#   Never scheduled. It performs a one-time structural change to where the web server looks;
+#   putting that on a timer would mean an unattended job could recreate a symlink over a
+#   deliberate manual install.
+#
+# Refuse, Do Not Replace
+#   A real directory at the target is left alone and the script exits. That directory is a
+#   genuine installation, and silently deleting it to make room for a symlink would discard
+#   an install this script did not create.
+#
+# OS Detected, Not Assumed
+#   The source directory comes from the detected OS, so the same script works unchanged on a
+#   second platform once Plugin/<os>/ exists — matching the adapter's approach.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
+# No Root — Deliberate
+#   Run by hand during install, when the operator already has whatever privilege the web
+#   server's plugin path requires. The guards below are structural rather than privilege
+#   based: the dangerous outcome here is clobbering an existing install, not lacking rights.
+#
 # Idempotent Symlink
 #   Removes any existing symlink at the target before recreating it. Safe to
 #   re-run after a repo move without leaving stale paths.

Plugin/unraid/Partnership/containers.sh

@@ -23,6 +23,82 @@
 #   - Deployed stack tracking via _STACK_DEPLOYED / _STACK_FAILED counters
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Platform Adapter, Same Contract as adapter.sh
+#   Container deployment is Unraid-specific — CA XML templates, dockerMan paths, the
+#   templates-user directory. Confining it here means the partnership scripts contain no
+#   Unraid knowledge and a second platform is a new Partnership/containers.sh, not edits
+#   scattered through onboard and offboard.
+#
+# Deploy From the Template, Not a Copy
+#   Containers are created from the CA XML the operator already maintains, so a partnership
+#   deployment produces the same container the Unraid UI would. Hand-built docker run lines
+#   would drift from the template the moment anyone edited it in the UI.
+#
+# GPU Detection Cached Per Session
+#   The remote GPU type is probed once and reused. Onboarding deploys several containers and
+#   each would otherwise repeat the same SSH round-trip to learn an answer that cannot change
+#   mid-run.
+#
+# Count Outcomes, Do Not Abort
+#   Failures increment _STACK_FAILED rather than exiting. A stack deployment that fails on
+#   one container should report which one and continue — the caller owns whether a partial
+#   stack is acceptable, and it is the only side with the context to decide.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# No Root, No Lock, No detect_hosts — Deliberate
+#   Sourced by partnership_onboard.sh and partnership_offboard.sh, both of which already
+#   enforce root and hold their own strict locks. Re-checking here would be redundant, and
+#   taking a lock would deadlock against the caller's. Do not add them.
+#
+# Caller Scope Is the Contract
+#   Functions read MIRROR, MIRROR_IP, MIRROR_SSH_KEY, SSH_TIMEOUT, DRY_RUN and SCRIPTS_ROOT
+#   from the caller. That coupling is deliberate — it keeps one definition of who the mirror
+#   is — but it means these functions are only valid inside the partnership scripts and
+#   cannot be sourced standalone.
+#
+# DRY_RUN Honoured Throughout
+#   Every deploy and cleanup path checks the caller's DRY_RUN, so a dry-run onboard makes no
+#   remote container changes.
+#
+# SSH Timeouts on Every Remote Call
+#   All remote operations use the caller's SSH_TIMEOUT — an unreachable mirror cannot hang
+#   an onboard partway through a stack deployment.
+#
+# Template Existence Checked
+#   A missing CA XML is counted as a failure for that container rather than producing a
+#   container built from nothing.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# No config vars of its own. Inputs come from the calling script's scope (see above).
+#
+# Platform paths it owns:
+#
+#   /boot/config/plugins/dockerMan/templates-user
+#       Unraid CA template directory. Source of every container definition deployed here.
+#
+# The container lists themselves live in host*.conf as HOST*_PARTNERSHIP_AUTH_STACK and
+# HOST*_PARTNERSHIP_ARR_STACK — read by the partnership scripts, passed in as arguments.
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# None — sourced, never executed:
+#
+#   source "$SCRIPTS_ROOT/Plugin/$PLATFORM/Partnership/containers.sh"
+#
+# No argument parsing and no flags. Dry-run behaviour comes from the caller's DRY_RUN.
+#
+# ==============================================================================================
 
 TEMPLATES_DIR="/boot/config/plugins/dockerMan/templates-user"
 
@@ -98,7 +174,12 @@ transform_xml_for_gpu() {
     local src_xml="$1" gpu_type="$2"
 
     # Detect GPU-aware XMLs — new style (--gpus "device=) or old style (--runtime=nvidia)
-    if ! grep -qE '--gpus[[:space:]]+"device=|--runtime=nvidia|NVIDIA_VISIBLE_DEVICES' "$src_xml" 2>/dev/null; then
+    # -- before the pattern is load-bearing: it begins with "--", so without it grep parses the
+    # pattern as an OPTION, exits 2, and the ! makes this branch always true — the function then
+    # returned the XML untransformed every single time, for every GPU type, with 2>/dev/null
+    # swallowing "invalid option". Every container onboarded to a mirror kept the owner NVIDIA
+    # device UUID and could not start on Intel or AMD hardware. Found 2026-08-23.
+    if ! grep -qE -- '--gpus[[:space:]]+"device=|--runtime=nvidia|NVIDIA_VISIBLE_DEVICES' "$src_xml" 2>/dev/null; then
         echo "$src_xml"
         return 0
     fi
@@ -203,7 +284,10 @@ deploy_container_from_xml() {
     [[ "$_transformed_xml" != "$xml_file" ]] && _gpu_tmp="$_transformed_xml"
     xml_file="$_transformed_xml"
 
-    local name repo network extra privileged
+    local name repo network extra privileged webui icon
+    # WebUI and Icon become Unraid labels below — see the docker create line for why.
+    webui=$(      awk 'match($0,/<WebUI>([^<]*)<\/WebUI>/,      a){print a[1];exit}' "$xml_file")
+    icon=$(       awk 'match($0,/<Icon>([^<]*)<\/Icon>/,        a){print a[1];exit}' "$xml_file")
     name=$(       awk 'match($0,/<Name>([^<]+)<\/Name>/,        a){print a[1];exit}' "$xml_file")
     repo=$(       awk 'match($0,/<Repository>([^<]+)<\/Repository>/,a){print a[1];exit}' "$xml_file")
     network=$(    awk 'match($0,/<Network>([^<]+)<\/Network>/,   a){print a[1];exit}' "$xml_file")
@@ -242,7 +326,19 @@ deploy_container_from_xml() {
         printf "docker stop %q 2>/dev/null || true\n" "$name"
         printf "docker rm   %q 2>/dev/null || true\n" "$name"
         echo ""
+        # Unraid's Docker Manager decides what it owns by label, not by template presence. The
+        # XML is SCPed to the mirror's templates-user above, but without these three the WebGUI
+        # lists the container as third-party: no Edit button, no WebUI link, no icon — the
+        # operator can see it running and cannot do anything with it.
+        #
+        # The values go in verbatim, placeholders and all: Unraid stores the literal
+        # "http://[IP]:[PORT:8989]/..." form in the label and substitutes at render time, so
+        # resolving them here would produce a link that stops being right the moment the
+        # container's port mapping changes.
         printf "docker create --name %q --restart=unless-stopped" "$name"
+        printf " --label %q" "net.unraid.docker.managed=dockerman"
+        [[ -n "$webui"             ]] && printf " --label %q" "net.unraid.docker.webui=${webui}"
+        [[ -n "$icon"              ]] && printf " --label %q" "net.unraid.docker.icon=${icon}"
         [[ -n "$network"           ]] && printf " --network=%q"  "$network"
         [[ "$privileged" == "true" ]] && printf " --privileged"
         [[ -n "$extra"             ]] && printf " %s" "$extra"
@@ -340,6 +436,112 @@ deploy_xml_stack() {
     done
 }
 
+# ==============================================================================================
+# ── Ensure the networks our pushed templates reference exist on a remote host ─────────────────
+#
+# The owner deploys the mirror's containers from the owner's own XMLs, and those XMLs name a
+# network. `docker create` fails outright if that network is missing, so the network has to
+# exist on the mirror before any stack is deployed.
+#
+# This used to be left entirely to docker_network_connect.sh running on the mirror, which
+# iterates the *mirror's* NETWORK_CONNECT_NETWORKS. host.conf.template ships that array with its
+# only entry commented out, so on a fresh node it is empty — nothing was created, and every
+# container in both stacks was created against a network that did not exist and could never
+# start. Twelve containers stuck in `Created`, reported as "0 deployed, 8 failed" and
+# "0 deployed, 5 failed" as though each container had its own problem.
+#
+# The owner knows what it is about to push, so it derives the requirement from the templates
+# rather than trusting the mirror's conf to have been filled in.
+#
+# Only bridge networks are created. br0 and friends are ipvlan/macvlan bound to real host
+# hardware — the parent interface cannot be inferred from here, and guessing one would attach
+# the mirror's containers to the wrong segment.
+#
+# Usage: ensure_stack_networks_on_remote "$MIRROR_IP" "$MIRROR_SSH_KEY"
+# ==============================================================================================
+# ==============================================================================================
+# ── Container names this onboard deploys, read from the templates it deploys them from ────────
+#
+# Echoes one name per line. The <Name> element is the same value deploy_xml_stack() passes to
+# `docker create --name`, so this is the deployed set by construction rather than by asking the
+# mirror what it ended up with — which would also pick up whatever the mirror already ran.
+#
+# Usage: mapfile -t names < <(deployed_stack_container_names)
+# ==============================================================================================
+deployed_stack_container_names() {
+    local -a xml_names=()
+    [[ ${#PARTNERSHIP_AUTH_STACK[@]}     -gt 0 ]] && xml_names+=("${PARTNERSHIP_AUTH_STACK[@]}")
+    [[ ${#PARTNERSHIP_ARR_STACK[@]}      -gt 0 ]] && xml_names+=("${PARTNERSHIP_ARR_STACK[@]}")
+    [[ ${#PARTNERSHIP_SERVICES_STACK[@]} -gt 0 ]] && xml_names+=("${PARTNERSHIP_SERVICES_STACK[@]}")
+
+    local xml_name xml_file cname
+    for xml_name in "${xml_names[@]}"; do
+        [[ -z "$xml_name" ]] && continue
+        xml_file="${TEMPLATES_DIR}/${xml_name}"
+        [[ -f "$xml_file" ]] || continue
+        cname=$(awk 'match($0,/<Name>([^<]+)<\/Name>/,a){print a[1];exit}' "$xml_file")
+        [[ -n "$cname" ]] && echo "$cname"
+    done
+}
+
+ensure_stack_networks_on_remote() {
+    local remote_ip="$1" ssh_key="$2"
+    local -a xml_names=() nets=()
+    [[ ${#PARTNERSHIP_AUTH_STACK[@]}     -gt 0 ]] && xml_names+=("${PARTNERSHIP_AUTH_STACK[@]}")
+    [[ ${#PARTNERSHIP_ARR_STACK[@]}      -gt 0 ]] && xml_names+=("${PARTNERSHIP_ARR_STACK[@]}")
+    [[ ${#PARTNERSHIP_SERVICES_STACK[@]} -gt 0 ]] && xml_names+=("${PARTNERSHIP_SERVICES_STACK[@]}")
+
+    local xml_name xml_file net n seen
+    for xml_name in "${xml_names[@]}"; do
+        [[ -z "$xml_name" ]] && continue
+        xml_file="${TEMPLATES_DIR}/${xml_name}"
+        [[ -f "$xml_file" ]] || continue
+        net=$(sed -n 's/.*<Network>\([^<]*\)<\/Network>.*/\1/p' "$xml_file" 2>/dev/null | head -1)
+        net="${net//[[:space:]]/}"
+        # Built-ins exist on every host; br* is host hardware, handled above.
+        case "$net" in ''|bridge|host|none|br[0-9]*) continue ;; esac
+        seen=false
+        for n in "${nets[@]}"; do [[ "$n" == "$net" ]] && { seen=true; break; }; done
+        [[ "$seen" == false ]] && nets+=("$net")
+    done
+
+    if [[ ${#nets[@]} -eq 0 ]]; then
+        log "No custom networks referenced by the pushed templates"
+        return 0
+    fi
+
+    local rc=0 driver
+    for net in "${nets[@]}"; do
+        driver=$(timeout "${DOCKER_TIMEOUT:-30}" docker network inspect "$net" \
+                 --format '{{.Driver}}' 2>/dev/null)
+        if [[ -z "$driver" ]]; then
+            warn "$net is referenced by a pushed template but does not exist here either — skipping"
+            rc=1
+            continue
+        fi
+        if [[ "$driver" != "bridge" ]]; then
+            warn "$net is $driver here, not bridge — create it on $MIRROR by hand, its parent interface is host-specific"
+            rc=1
+            continue
+        fi
+        if [[ "$DRY_RUN" == true ]]; then
+            warn "DRY RUN — would ensure network '$net' (bridge) exists on $MIRROR"
+            continue
+        fi
+        if timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
+             -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+             root@"$remote_ip" \
+             "docker network inspect $(printf '%q' "$net") >/dev/null 2>&1 \
+              || docker network create --driver bridge $(printf '%q' "$net") >/dev/null" 2>/dev/null; then
+            echo "  network $net (bridge) ready on $MIRROR ✅"
+        else
+            warn "  could not ensure network $net on $MIRROR — its containers will not start"
+            rc=1
+        fi
+    done
+    return "$rc"
+}
+
 # ==============================================================================================
 # ── Remove owner-deployed containers from a remote host ──────────────────────────────────────
 #
@@ -350,6 +552,7 @@ deploy_xml_stack() {
 # ==============================================================================================
 cleanup_deployed_stack_on_remote() {
     local remote_ip="$1" ssh_key="$2"
+    local _rc=0
     local -a xml_names=()
     [[ ${#PARTNERSHIP_AUTH_STACK[@]}     -gt 0 ]] && xml_names+=("${PARTNERSHIP_AUTH_STACK[@]}")
     [[ ${#PARTNERSHIP_ARR_STACK[@]}      -gt 0 ]] && xml_names+=("${PARTNERSHIP_ARR_STACK[@]}")
@@ -395,13 +598,20 @@ cleanup_deployed_stack_on_remote() {
 
         while IFS= read -r path; do
             [[ -z "$path" ]] && continue
-            timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
+            if timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
                 -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$remote_ip" \
-                "rm -rf '$path' && echo removed" 2>/dev/null | grep -q removed && \
+                "rm -rf '$path' && echo removed" 2>/dev/null | grep -q removed; then
-                echo "  Appdata removed on $MIRROR: $path ✅" || \
+                echo "  Appdata removed on $MIRROR: $path ✅"
+            else
                 warn "  Failed to remove appdata on $MIRROR: $path"
+                _rc=1
+            fi
         done <<< "$appdata_paths"
     done
+    # Only appdata failures are counted. The container branch above cannot tell "removal failed"
+    # from "already gone" — both produce no `removed` echo — and an offboard re-run on a
+    # half-finished teardown is a normal case, so treating that as failure would cry wolf.
+    return "$_rc"
 }
 
 # ==============================================================================================
@@ -414,6 +624,12 @@ cleanup_deployed_stack_on_remote() {
 cleanup_deployed_stack_locally() {
     local owner_ip="$1" ssh_key="$2"
     local -a xml_names=()
+    # Callers write `cleanup_deployed_stack_locally … || STEP_STACK_CLEANUP_OK=false`, so the
+    # exit status is what the offboard summary prints. Every removal below warns and carries on
+    # — one container that will not die must not abandon the rest of the stack — which meant the
+    # function ended on a `done` and could only ever return 0. Step 3 reported ✅ even when every
+    # docker rm and every rm -rf had failed. Failures are collected here and reported at the end.
+    local _rc=0
 
     if [[ -n "$owner_ip" ]]; then
         local -a auth_arr arr_arr
@@ -437,8 +653,13 @@ cleanup_deployed_stack_locally() {
     fi
 
     if [[ ${#xml_names[@]} -eq 0 ]]; then
-        log "Could not read deployed stack from owner — skipping auth/arr/services cleanup"
+        # Not a success. OWNER_REACHABLE only means a probe answered — the three SSH reads above
-        return 0
+        # can still time out or come back empty, and then nothing was cleaned. The caller's own
+        # unreachable-owner branch sets STEP_STACK_CLEANUP_OK=false for exactly this situation,
+        # so returning 0 here made the summary claim a cleanup that never ran.
+        warn "Could not read deployed stack from owner — auth/arr/services cleanup did not run"
+        warn "Containers will remain — re-run when the owner answers over SSH"
+        return 1
     fi
 
     local _local_short
@@ -475,17 +696,28 @@ cleanup_deployed_stack_locally() {
                 "$cname" 2>/dev/null | awk -F: '{print $1}' | grep '^/mnt/.*/appdata')
             timeout "${DOCKER_TIMEOUT:-30}" docker stop "$cname" >/dev/null 2>&1 || true
             _PM_TRAP_STOPPED+=("$cname")
-            timeout "${DOCKER_TIMEOUT:-30}" docker rm   "$cname" >/dev/null 2>&1 && \
+            if timeout "${DOCKER_TIMEOUT:-30}" docker rm "$cname" >/dev/null 2>&1; then
-                echo "  $cname removed ✅" || warn "  $cname rm failed"
+                echo "  $cname removed ✅"
+            else
+                warn "  $cname rm failed"
+                _rc=1
+            fi
         else
             log "  $cname not found locally — skipping"
         fi
 
         while IFS= read -r path; do
             [[ -z "$path" ]] && continue
-            rm -rf "$path" && echo "  Appdata removed: $path ✅" || warn "  Failed to remove: $path"
+            if rm -rf "$path"; then
+                echo "  Appdata removed: $path ✅"
+            else
+                warn "  Failed to remove: $path"
+                _rc=1
+            fi
         done <<< "$appdata_paths"
     done
+
+    return "$_rc"
 }
 
 # ==============================================================================================
@@ -502,11 +734,17 @@ reconfigure_webui() {
         return 0
     fi
 
+    # Match <Name>Authelia</Name>, not "Authelia". Unraid writes the container name as bare XML
+    # text, so the quoted form matched nothing in any template on any host — which is why every
+    # offboard ended with four "template not found ... WebUI needs manual reconfiguration"
+    # warnings and left the mirror's auth WebUIs pointing at the owner it had just left.
+    #
+    # xargs -r so an empty first grep does not run the second one against the whole directory.
     local template
     template=$(timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
         -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$remote_ip" \
         "grep -rl '<WebUI>' '$TEMPLATES_DIR/' 2>/dev/null | \
-         xargs grep -l '\"$container\"' 2>/dev/null | head -1" 2>/dev/null)
+         xargs -r grep -l '<Name>$container</Name>' 2>/dev/null | head -1" 2>/dev/null)
 
     if [[ -z "$template" ]]; then
         warn "$container template not found on $label — WebUI needs manual reconfiguration"

Plugin/unraid/README-unraid.md

@@ -0,0 +1,323 @@
+# ━━━━━ PLUGIN / UNRAID ━━━━━
+
+The Unraid platform layer. Everything in this folder exists to translate one specific
+operating system into the vocabulary Varaverk speaks — and to put a face on it in the Unraid
+WebGUI.
+
+Varaverk itself knows nothing about Unraid. No script outside this folder references
+`/etc/rc.d`, `/boot/config/plugins/dynamix`, `emhttp`, the mover, or a `.page` file. They call
+`platform_*()` and read `$STATE_DIR`; this folder decides what those mean on Unraid.
+
+```
+  adapter.sh              24 platform_*() functions — the entire OS contract
+  Varaverk.page           WebGUI menu entry (Tasks:95)
+  VaraverkSettings.page   WebGUI settings entry
+  event/                  Unraid array lifecycle hooks
+  pages/ api/ include/    the three-layer web UI
+  System_Essentials/      Unraid-specific system scripts
+  Tools/                  Unraid-specific operator tools
+  Watchdogs/System/       Unraid-specific watchdog (WebGUI health)
+  Partnership/            Unraid-specific container deploy/cleanup
+```
+
+> Swapping platforms means writing `Plugin/truenas/adapter.sh` implementing the same 24
+> function names. `load_config.sh` sources `Plugin/$PLATFORM/adapter.sh` — nothing else in
+> the codebase changes.
+
+---
+
+## ━━━ THE PROBLEM THAT BUILT THIS ━━━
+
+**Every NAS OS Answers the Same Questions Differently**
+"Is the array mounted?" "Restart this service." "What are the disk temperature thresholds?"
+"Send a notification." Every platform has an answer, and every answer is different — a
+different path, a different init system, a different config format.
+
+Scattering those differences through 115 scripts would mean every script carrying an `if
+unraid ... elif truenas` branch, and every new platform touching all of them.
+
+**Varaverk Vars Are Not OS Vars**
+Varaverk works in terms of `STATE_DIR`, `SCRIPTS_DIR`, `DOCKER_APPDATA_BASE`,
+`WATCHDOG_CONTAINERS`. Unraid works in terms of `/boot/config/plugins/`, `emhttp`,
+`/mnt/user`, dockerMan XML templates, and `dynamix.cfg`. Something has to map one onto the
+other, in exactly one place, or the two vocabularies leak into each other.
+
+**A Headless Ecosystem Is Hard to Trust**
+115 scripts running unattended produce a lot of state nobody can see. Without a UI the only
+way to know what the watchdogs did last night is to read logs. The WebGUI exists so the
+system can be inspected without SSH.
+
+---
+
+## ━━━ THE ADAPTER — WHERE OS MEETS VARAVERK ━━━
+
+`adapter.sh` is the whole contract. 24 functions, sourced by `load_config.sh` after
+`common.sh`, so every script in the ecosystem has them.
+
+The most-called ones, by usage across the repo:
+
+| Function | Calls | What it hides |
+|----------|-------|---------------|
+| `platform_require_cmd` | 28 | Verifying a platform binary exists and is executable |
+| `platform_restart_service` | 13 | `/etc/rc.d/rc.<name>` init scripts |
+| `platform_is_service_running` | 13 | Unraid's service process conventions |
+| `platform_is_mover_running` | 10 | The Unraid mover — no other OS has one |
+| `platform_get_os_version` | 10 | Where the version string lives |
+| `platform_setup_db_path` | 9 | Where wizard/setup state persists |
+| `platform_storage_path` | 7 | `/mnt/user` as the user-share root |
+
+Plus: disk states and temperature thresholds from `dynamix.cfg`, parity/maintenance detection,
+native notifications through the dynamix notify script, container rebuild from dockerMan XML
+templates, and the WebGUI install path.
+
+**The contract is strict, and it is what makes the layer work:**
+
+- **Every function returns 0/1 and never calls `exit`.** A missing capability is information
+  the caller needs, not a decision the adapter makes — a watchdog may want to skip where an
+  installer wants to abort.
+- **Value-producing functions write to stdout**, captured with `$()`. Status travels in the
+  exit code. Keeping those separate is what lets callers use them in conditionals.
+- **It reports; it does not remediate.** No retries, no escalation, no notification policy.
+  Burying those here would make identical calls behave differently per platform.
+
+---
+
+## ━━━ THE WEB UI — THREE LAYERS ━━━
+
+```
+  pages/*.php      what you look at        ── polls ─────►  api/
+  api/*.php        what the page fetches   ── requires ──►  include/
+  include/*.php    the actual logic — vv_*() functions
+```
+
+**Most pages hold no logic at all.** Eight of the eleven are pure view — markup, CSS and a
+poll loop, with every value arriving as JSON from `api/`. Only `auth.php`, `monitor.php` and
+`scheduler.php` require an `include/` file directly, and then only to server-render their
+initial state; their mutations still go through endpoints.
+
+That is why the layer split holds: a page and its endpoint cannot disagree about what a value
+means, because the page does not compute it. Adding a metric means adding it in `include/`
+once, and the endpoint, the cache writer, and the page all pick it up together.
+
+| Layer | Files | Role |
+|-------|-------|------|
+| `pages/` | 11 | One per WebGUI tab — monitor, docker, arrs, fallback, watchdog, rsync, partnership, scheduler, auth, settings, setup |
+| `api/` | 50 | JSON endpoints the pages poll, plus action endpoints (run a script, stop a job, toggle a flag) |
+| `include/` | 27 | Shared builders and helpers — `vv_monitor_*`, `vv_arrs_*`, `vv_docker_*`, config read/write, auth |
+
+**The Monitor board is declared, not laid out.** `include/monitor_board.php` holds one array
+naming every card on the Monitor grid, its width and its order, and generates the whole layout
+from it — the column ladder, the span clamps at each width, the row-height cap, and the
+compensation when a conditional card is absent. Column counts are 8/4/2/1 and spans are 1/2/4/8,
+which is what lets the board re-cut itself at any width with no holes and no hand-placed card.
+Breakpoints are arithmetic over `VV_MON_CARD_FLOOR`, never chosen by eye, and the row-height cap
+is conditional on the same arithmetic — it applies only where a rung fits one screen.
+`pages/monitor.php` carries the card bodies and nothing about where they go; moving a card is
+moving a line in that array. The page cross-checks the declaration against the cards that
+actually rendered and says so in the browser if they disagree.
+
+**Caching.** Several endpoints serve from `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs) rather than hitting live
+APIs on every page view, refreshed by `Tools/api_cache_writer.sh`. `?live=1` bypasses the
+cache. A missing or unparseable cache always falls back to a live call, so the cache can never
+be the reason a page fails to load.
+
+Anything expensive belongs in that one collection rather than in each consumer. `vv_ai_stats()`
+costs about a second — mostly waiting on Ollama and `nvidia-smi` — and the AI tab was paying it
+every 30 seconds per open tab; it is now written once to the `ai` cache and read by the tab, the
+Monitor row and the dock alike. Polling faster cannot make a figure newer, it only decides how
+soon a page notices the writer's update.
+
+> `Tools/api_cache_writer.php` rebuilds the monitor payload **independently** of
+> `api/monitor.php`, and is what the page normally reads — the endpoint only assembles one on a
+> cache miss. A key added to the endpoint and not to the writer leaves its card loading forever
+> on every ordinary page load and working only on the request that happens to miss.
+
+**One widget, rendered three times.** `include/ai_chat.php` owns the conversation surface —
+transcript, composer, profile and history pickers, source viewer, stored-chat list — and the AI
+tab, the Monitor tab's AI row and the Scheduler's right-hand panel all construct it. Every id is
+composed from a prefix so instances can coexist, and each tears down the previous holder of its
+prefix, because Unraid swaps tab content by AJAX without unloading the old page's JavaScript.
+
+A placement chooses how much room it gets and which extra controls it needs. It does not choose
+the shape, because a chat that rearranges itself per tab is three components wearing one name.
+The layout is fixed everywhere:
+
+- **Banner** — title, then `Medium`/`Large` and `Shortcuts` at the right.
+- **Transcript**.
+- **Composer**, then one row grouped by what each control acts on: the profile chip at the left
+  for what is *answering*, `New` `Saved` `Ask` centred for what you do to the *conversation*, `⤢`
+  at the right for what you do to the *window*.
+
+`⤢` and `Medium`/`Large` are deliberately separate: whether the conversation is expanded is an
+action taken constantly, how much expanded is worth is a preference set once. The shortcuts list
+is subtracted from the transcript rather than added above it, so opening it never changes the
+window's total height — which matters on the Scheduler, where that height is a fixed share of a
+panel that clips.
+
+The Scheduler panel was exactly that until it was folded in, and it proved the cost: a thread
+there died on reload while the other two were saved, and its bar had drifted into a different
+shape from the same control everywhere else. What was genuinely particular to it became options
+rather than a second implementation — `scope` as a function for a subject that follows the open
+view, `beforeSend` for the one reply that is recorded rather than asked, `think` for reasoning on
+diagnosis only, and `setHeights()` for a placement sized as a share of a panel, which is not a
+number that exists until layout has run.
+
+**Timezone.** `include/config.php` adopts Unraid's own `timeZone` from `ident.cfg` for the whole
+PHP layer. PHP here has no `date.timezone` and therefore ran in UTC while the server ran local,
+so every date this layer produced was offset from every date the shell layer wrote — and the two
+are compared constantly. `/etc/php.ini` is the wrong fix twice over: `/etc` is a RAM filesystem
+so the edit dies at reboot, and it would retimezone every other PHP application on the box.
+
+---
+
+## ━━━ WHAT GUARDS THE API LAYER ━━━
+
+Two layers, both supplied by the platform, neither implemented in this plugin.
+
+**1. nginx session auth.** `/etc/nginx/nginx.conf` applies `satisfy any; allow unix:; deny all;
+auth_request /auth-request.php;` to everything it serves, `/plugins/` included. A request
+without a valid WebGUI session never reaches PHP — it is redirected to the login page.
+
+**2. Unraid's CSRF prepend.** `/etc/php.ini` sets
+`auto_prepend_file=/usr/local/emhttp/webGui/include/local_prepend.php`, which runs before the
+first line of any endpoint and, for **every POST**, requires a valid token as either a
+`csrf_token` body field or an `X-CSRF-Token` header. On a mismatch it logs to syslog and
+`exit`s. php-fpm inherits this — there is no pool override in `/etc/php-fpm.d/www.conf`.
+
+Anything that clears both can do everything this plugin can do: stop the array, power off the
+host, write `master.conf` and push it to every partner, create an lldap user, write and
+schedule a root-run script. That is the same trust level as the rest of the WebGUI, and it is
+the intended model.
+
+### The two consequences that actually matter
+
+**GET is not covered.** The prepend inspects POSTs only. So any action that *changes something*
+must be POST — over GET it would run with no token check at all. Four actions were GET and have
+been converted: `create_api_key.php`, `arrs.php?action=refresh_remote`, `cert.php?action=run`
+and `setup.php?action=ssh_generate`. **When adding an endpoint the rule is simply: if it changes
+state, it is POST.** Read-only GETs (`monitor`, `board`, `checklist`, the browse pickers,
+`api_test.php`) are bounded by the session alone, which is correct for reads.
+
+**The webGUI's token injector is jQuery-only.** `$.ajaxPrefilter` in dynamix's
+`HeadInlineJS.php` appends `csrf_token` to jQuery POSTs. Varaverk's pages use native `fetch()`,
+which it does not touch — so a `fetch()` POST carries no token and the prepend kills it
+silently: `csrf_terminate()` exits with an empty body, `r.json()` throws on the empty response,
+and the page's own `.catch()` swallows it. There is no error anywhere except a syslog line.
+
+`Varaverk.page` therefore installs a small `window.fetch` shim, inline and above the tab
+content, that attaches `X-CSRF-Token` to same-origin `/plugins/varaverk/` requests. The header
+form is deliberate: it works for `FormData`, `URLSearchParams` and raw JSON bodies alike, so no
+call site has to know about it and a new one cannot forget it.  A cross-origin page cannot set
+a custom header without a preflight it will fail, which is what makes it a real defence rather
+than a formality.
+
+**Do not construct POSTs outside `fetch()`** — a raw `XMLHttpRequest` or a generated form —
+without adding the token yourself; the shim only wraps `fetch`. A plain `<form method="POST">`
+is already covered by a different platform mechanism: dynamix's `BodyInlineJS.php` appends a
+hidden `csrf_token` input to every form on the page, which is what makes `VaraverkSettings.page`
+work without any of this.
+
+### Never send a POST as multipart/form-data
+
+**`FormData` is banned in this plugin.** A `multipart/form-data` POST to the plugin API *hangs
+and never completes* on this host — confirmed 2026-08-02.
+
+The failure is completely silent, which is what makes it dangerous:
+
+- The browser sends it correctly, valid token and all (verified in the Network tab).
+- The Network row shows **no status code at all** — not 403, not 500. It never completes.
+- Server side there is nothing: no CSRF termination in syslog, no fatal in `/var/log/phplog`,
+  and no output from a log statement that is the literal first line of the endpoint.
+
+Use `URLSearchParams`. `fetch` sets `application/x-www-form-urlencoded` for it automatically,
+and it has the same `append()` / `set()` API, so it is a drop-in for any payload of strings:
+
+```js
+const fd = new URLSearchParams();      // NOT new FormData()
+fd.append('action', 'save');
+fetch(url, { method: 'POST', body: fd });
+```
+
+The correlation across the plugin was exact: every page using `URLSearchParams` worked, every
+page using `FormData` hung. All 21 call sites were converted in one pass. Suspected to date from
+the Unraid 7.3.1→7.3.2 upgrade; root cause in nginx/php-fpm was never identified, only the
+workaround. If a POST ever hangs with no status code and leaves no server-side trace whatsoever,
+**suspect the encoding first** — not CSRF, not auth, not the endpoint.
+
+### api/webhook.php is dead code
+
+It receives arr download events — but an arr has no WebGUI session and no CSRF token, so it is
+blocked by *both* layers above and cannot be called at all. The live path is
+`Arrs_Stack/webhook_listener.js` on `WEBHOOK_PORT`, running outside nginx entirely and
+validating `WEBHOOK_SECRET` itself. `api/webhook.php` cannot be made to work by adding a secret
+check — it would have to be served from outside `/plugins/` first. Removing it is the other
+reasonable option.
+
+Two endpoints read wider than the rest and are worth knowing about if the session boundary ever
+moves: `api/api_test.php` returns an API key prefix and the live GraphQL schema, and the browse
+actions in `api/import_script.php` and `api/manual_sync.php` list directory names anywhere on
+either host. All are read-only and none return file contents.
+
+---
+
+## ━━━ ARRAY LIFECYCLE HOOKS ━━━
+
+`event/` plugs Varaverk into Unraid's own array lifecycle. These are how the ecosystem starts
+and stops with the array rather than with the OS.
+
+| Hook | Fires | Runs |
+|------|-------|------|
+| `disks_mounted/array_start_jobs` | Array started | `Orchestrators/array_started.sh` → `ARRAY_START_SCRIPTS` |
+| `disks_mounted/rebuild_cron` | Array started | Regenerates `varaverk.cron` from the scheduler config |
+| `unmounting_disks/array_stop_jobs` | Array stopping | `Orchestrators/array_stopping.sh` → `ARRAY_STOP_SCRIPTS` |
+
+Array-start rather than boot is the correct trigger: in flash storage mode `SCRIPTS_DIR` lives
+in appdata, which does not exist until the array mounts.
+
+---
+
+## ━━━ THE PLATFORM-SPECIFIC SCRIPT MIRRORS ━━━
+
+These folders mirror the top-level Varaverk structure, but hold work that **cannot** be
+platform-agnostic. A script lives here rather than in the root folder when it manipulates
+Unraid itself.
+
+| Folder | Contents | Why it is platform-specific |
+|--------|----------|----------------------------|
+| `System_Essentials/` | 4 scripts | Mover, User Scripts plugin, PHP-FPM pool, unraid-api registry — all Unraid subsystems |
+| `Tools/` | 5 files | Share `.cfg` recreation, storage-mode migration, API cache writers |
+| `Watchdogs/System/` | `webgui_watchdog.sh` | emhttp / nginx / php-fpm health — there is no generic "WebGUI" |
+| `Partnership/` | `containers.sh` | Container deploy from dockerMan CA XML templates |
+
+Each has its own README where the group is large enough to need one; single-script folders are
+documented in the script header.
+
+---
+
+## ━━━ HOW THIS FOLDER IS INSTALLED ━━━
+
+Two paths, deliberately different:
+
+**Development** — `Plugin/plugin_setup.sh` symlinks this directory into the WebGUI plugin
+location. Every edit is live immediately. This is the normal working mode.
+
+**Release** — `Plugin/build.sh` packages this directory into a Slackware `.txz` and rewrites
+the version and checksum in `varaverk.plg`. That is what Unraid reinstalls from flash on every
+boot.
+
+In flash storage mode `git_pull_execute.sh` syncs `Plugin/` back to `/boot/` after each pull,
+so the WebGUI always serves current PHP even though `SCRIPTS_DIR` points at appdata.
+
+---
+
+## ━━━ ADDING A NEW PLATFORM ━━━
+
+1. Create `Plugin/<platform>/adapter.sh` implementing all 24 `platform_*()` functions.
+2. Set `PLATFORM=<platform>` in `varaverk.cfg`.
+
+That is the whole contract for the script ecosystem. `load_config.sh` sources
+`Plugin/$PLATFORM/adapter.sh` and every script keeps working unchanged.
+
+The WebGUI is separate and optional — it is Unraid-specific by nature (`.page` files, emhttp
+conventions). A new platform can run the entire ecosystem headless with only the adapter, and
+add a UI later in whatever form that platform expects.

Plugin/unraid/System_Essentials/README-System_Essentials.md

@@ -0,0 +1,134 @@
+# ━━━━━ PLUGIN / UNRAID / SYSTEM ESSENTIALS ━━━━━
+
+Four scripts that touch Unraid subsystems directly. They live here rather than in the
+top-level `System_Essentials/` because each one manipulates something that only exists on
+Unraid — the mover, the User Scripts plugin, the PHP-FPM pool serving emhttp, and the
+unraid-api service registry.
+
+There is no platform-agnostic version of any of them. On TrueNAS there is no mover to stop.
+
+| Script | Runs | Role |
+|--------|------|------|
+| `php_fpm_max_children.sh` | Array **start** | Raise the PHP-FPM worker ceiling so the WebGUI stays responsive |
+| `unraid_api_key_renew.sh` | Array **start** + every **15 min** | Re-register Varaverk's key in the ephemeral unraid-api registry |
+| `user_scripts_stop.sh` | Array **stop** | Stop User Scripts processes before the array goes down |
+| `mover_stop.sh` | Array **stop** | Stop the mover gracefully before rsync or reboot |
+
+All four are wired through `ARRAY_START_SCRIPTS` / `ARRAY_STOP_SCRIPTS` in `master.conf`.
+
+`unraid_api_key_renew.sh` is additionally listed in `WATCHDOG_ORCHESTRATOR_SCRIPTS`, so it also
+re-runs every 15 minutes. That is deliberate: the registry it repairs can be cleared by a
+service restart at any time, not only across a reboot, and the script is idempotent.
+
+---
+
+## ━━━ WHY EACH ONE EXISTS ━━━
+
+### 🐘  `php_fpm_max_children.sh` — the WebGUI gets slow, not broken
+
+Unraid's WebGUI runs through PHP-FPM, and the stock `pm.max_children` is very low (4–8). Under
+real load — several browser tabs polling, Docker operations running, a dashboard open — every
+worker saturates and new requests queue. The WebGUI becomes slow or stops answering, while
+nothing is actually wrong with the server.
+
+Runs at array start because the setting **does not survive an Unraid update** — the OS
+replaces the pool config. Idempotent: already at the target value means no write and no
+restart, so a clean boot is silent.
+
+### 🔑  `unraid_api_key_renew.sh` — the registry is ephemeral
+
+Varaverk's enhanced monitoring authenticates against unraid-api. That registry is cleared by
+OS updates and service restarts, so a key that worked yesterday can simply be gone.
+
+Re-registers unconditionally every array start rather than only when missing, because the
+failure it repairs is precisely *a key present in the conf but absent from the registry* —
+checking the conf would not detect it. Writes the resulting key into `HOST*_UNRAID_API_KEY`.
+
+### 🛑  `user_scripts_stop.sh` — stop new work before shutting down
+
+The User Scripts plugin spawns background processes that Varaverk does not own. During a
+shutdown sequence those can still be starting operations while everything else is being torn
+down.
+
+Runs **first** in `ARRAY_STOP_SCRIPTS` for that reason — stop new work before stopping the
+things it would work on. Reports each process by **script name**, not just PID, because
+"stopping 4 processes" tells an operator nothing they can act on.
+
+Matching is scoped strictly to processes the User Scripts plugin spawned. A broad pattern
+would catch Varaverk's own scripts — including, during a reboot, the very script doing the
+stopping.
+
+### 💾  `mover_stop.sh` — mover and rsync must not overlap
+
+The mover relocates files between cache and array. rsync reads those same paths. Both running
+at once can produce a corrupt or half-moved snapshot on the remote.
+
+Ordering in the stop sequence is deliberate: `rsync_stop.sh` runs **before** `mover_stop.sh`,
+because both write to the same paths and stopping the reader first is the safer order.
+
+Sends a wall warning first, then SIGTERM with a configurable window, then SIGKILL only if
+ignored. The mover is mid-file-move by definition — giving it the chance to finish the current
+file is the difference between a stopped transfer and a half-moved file.
+
+An absent mover exits 0. Callers use this as a precondition ("ensure the mover is not
+running"), so treating "already stopped" as failure would abort every reboot on a quiet
+system.
+
+---
+
+## ━━━ WHERE THEY SIT IN THE LIFECYCLE ━━━
+
+```
+Array starting
+    │
+    ├── php_fpm_max_children.sh ──── WebGUI can handle the load that's about to arrive
+    └── unraid_api_key_renew.sh ──── monitoring can authenticate again
+
+        ... normal operation ...
+
+Array stopping
+    │
+    ├── user_scripts_stop.sh ─────── stop new work first
+    ├── fallback.sh --stop
+    ├── rsync_stop.sh ────────────── stop the reader
+    ├── mover_stop.sh ────────────── then the writer — same paths
+    └── docker_container_stop.sh ─── containers last, verified one at a time
+```
+
+`mover_stop.sh` is also called directly by `server_reboot.sh`, and both stop scripts appear in
+the User Scripts master template for manual use.
+
+---
+
+## ━━━ SAFEGUARDS ━━━
+
+All four enforce root, take a lock, and support `--dry-run`. Specific to this group:
+
+**Graceful before forced.** `mover_stop.sh` and `user_scripts_stop.sh` both use SIGTERM with a
+window before SIGKILL. Neither kills first.
+
+**Absent target is success.** No mover running, no User Scripts running — both exit 0. These
+are preconditions, not commands that must find something to do.
+
+**Idempotent where it matters.** `php_fpm_max_children.sh` writes and restarts only when the
+value is actually wrong, so array start does not restart PHP-FPM every single boot.
+
+**Targeted process matching.** `user_scripts_stop.sh` matches only what the User Scripts
+plugin spawned — see above for why a broad pattern is dangerous here specifically.
+
+**Stop only, never start.** Neither stop script has a counterpart that restarts what it
+stopped. Unraid's own schedule owns when the mover runs; these only remove it from the picture
+for a window.
+
+---
+
+## ━━━ CONFIGURATION ━━━
+
+| Variable | Used by | Purpose |
+|----------|---------|---------|
+| `HOST*_UNRAID_API_KEY` | `unraid_api_key_renew.sh` | Written at array start and every 15 min — output, not input |
+| `MOVER_STOP_TIMEOUT` | `mover_stop.sh` | SIGTERM grace window before escalating |
+| `PHP_MAX_CHILDREN` | `php_fpm_max_children.sh` | Target worker ceiling |
+
+See each script's `CONFIGURATION` header section for the authoritative list — and
+`Deployment/master.conf.template`, which is the versioned schema for all of them.

Plugin/unraid/System_Essentials/mover_stop.sh

@@ -29,6 +29,28 @@
 # used as a last resort and may leave a file split across cache and array.
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Warn the Room First
+#   A wall message goes out before the mover is signalled. The mover moves other people's
+#   data between cache and array; anyone watching a transfer deserves to know why it stopped.
+#
+# Graceful First, Forced Last
+#   SIGTERM with a configurable window, then SIGKILL only if it is ignored. The mover is
+#   mid-file-move by definition — giving it the chance to finish the current file and exit
+#   cleanly is the difference between a stopped transfer and a half-moved file.
+#
+# Not Running Is Success
+#   An absent mover exits 0. Callers use this as a precondition ("ensure the mover is not
+#   running"), not as a command that must find something to kill — treating "already stopped"
+#   as failure would abort every reboot on a quiet system.
+#
+# Stop, Never Start
+#   This script has no counterpart that restarts the mover. Unraid's own schedule owns when
+#   the mover runs; this only ever removes it from the picture for a window.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
@@ -52,7 +74,7 @@
 # master.conf
 #
 #   MOVER_STOP_TIMEOUT
-#       Seconds between wall warning and SIGTERM. (default: 30)
+#       Seconds between wall warning and SIGTERM. (shipped default: 300)
 #
 # ==============================================================================================
 # RUNTIME MODES

Plugin/unraid/System_Essentials/pcie_aer_quiet.sh

@@ -0,0 +1,320 @@
+#!/bin/bash
+# ==============================================================================================
+# ============================== PCIe AER Quiet ================================================
+# ==============================================================================================
+#
+# PURPOSE
+# ─────────────────────────────────────────────────────────────────────────────
+# Removes listed PCIe devices from the bus at array start so dead hardware stops
+# flooding syslog with correctable AER errors. Run once via ARRAY_START_SCRIPTS.
+# Idempotent — silent when the devices are already absent.
+#
+# Some onboard controllers throw endless correctable PCIe errors. Correctable
+# means the link recovered and nothing was lost, but the kernel logs every one,
+# and on HOST1 that was ~180 lines/day drowning the syslog. Masking the error in
+# the device's AER registers hides it; `pci=noaer` silences the whole machine
+# and costs uncorrectable reporting on every device. Removing the device ends it
+# at the source and leaves AER fully intact everywhere else.
+#
+# This only makes sense for hardware nothing is using. The guards below refuse
+# anything else — see OPERATIONAL SAFEGUARDS.
+#
+# A reboot re-enumerates the devices, which is why this reapplies every array
+# start rather than being a one-time step. It is also the undo: clear the conf
+# list and reboot.
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# One pass over PCIE_QUIET_DEVICES at array start. Each address is looked up under
+# /sys/bus/pci/devices, checked against the guards below, and removed through the kernel's own
+# per-device remove attribute. Nothing is masked and no kernel parameter is set.
+#
+# Idempotent, and silent when there is nothing to do. An address that is already gone is not an
+# error — that is the normal state on every array start after the first within one boot.
+#
+# Reapplied every array start rather than once, because a reboot re-enumerates the bus and the
+# devices come back. That is also the undo: clear PCIE_QUIET_DEVICES and reboot.
+#
+# PCIE_QUIET_ENABLED gates the whole run before any device is touched.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Refuse Anything In Use
+#   A PCI address is one typo away from the HBA holding the array. The script
+#   removes a device only when the kernel shows it bound to vfio-pci or to no
+#   driver at all. A device with a real driver is skipped loudly, never removed.
+#
+# Address Shape Validated Before Any Write
+#   The address goes into a sysfs path. It is matched against the full
+#   DDDD:BB:DD.F form before being used to build one.
+#
+# Idempotent Presence Check
+#   A device already gone is not an error — it is the desired state. Silent skip.
+#
+# Apply Sequence (per device)
+#   1. Validate the address shape
+#   2. Absent from sysfs → already done, skip silently
+#   3. Bound to a driver other than vfio-pci → refuse, warn, continue
+#   4. Claimed by a running VM → refuse, warn, continue
+#   5. Write 1 to the device's sysfs remove node
+#   6. Verify it left the bus
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Required
+#   Writing to /sys/bus/pci/devices requires root.
+#
+# Driver Guard
+#   Only vfio-pci-bound or unbound devices are eligible. This is what stops a
+#   mistyped address from pulling the SAS controller, a NIC or an NVMe drive out
+#   from under a running system.
+#
+# Running VM Guard
+#   A vfio-pci device may be passed through to a live VM. Removing it would rip
+#   the device away mid-flight. Running domains are checked for the address.
+#
+# Single Instance Lock
+#   acquire_lock prevents concurrent runs at array start.
+#
+# Disabled By Default
+#   PCIE_QUIET_ENABLED is false and the device list is empty until an operator
+#   fills it in per host. An empty list exits silently, so a host that has no
+#   such hardware runs a no-op.
+#
+# Silent on Success
+#   Runs every boot — no noise when the devices are already gone.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   PCIE_QUIET_ENABLED
+#       Master gate. Nothing is removed while false. (default: false)
+#
+# host*.conf
+#
+#   HOST*_PCIE_QUIET_DEVICES
+#       PCI addresses to remove, full DDDD:BB:DD.F form. (default: empty)
+#       Aliased to PCIE_QUIET_DEVICES by detect_hosts.
+#
+#       HOST1 example — both are onboard, stubbed to vfio, used by nothing:
+#           "0000:03:00.0"   ASMedia ASM1143 USB 3.1 controller
+#           "0000:04:00.0"   Intel Wireless 8265
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# pcie_aer_quiet.sh
+#     Remove every eligible listed device. Silent if all are already absent.
+#
+# pcie_aer_quiet.sh --dry-run
+#     Show what would be removed, refused or skipped. No sysfs write.
+#
+# pcie_aer_quiet.sh --status
+#     Show each listed device, its driver, and whether it is present.
+#
+# pcie_aer_quiet.sh --log
+#     Verbose output showing each guard decision.
+#
+# ==============================================================================================
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+source "$SCRIPT_DIR/../../../load_config.sh"
+
+parse_args "$@"
+
+# ==============================================================================================
+# ━━━ Setup ━━━
+# ==============================================================================================
+if [[ "$EUID" -ne 0 ]]; then
+    error "Must be run as root — removing PCI devices requires root"
+    exit 1
+fi
+
+acquire_lock
+
+detect_hosts
+
+[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no devices will be removed"
+
+PCI_DEVICE_ROOT="/sys/bus/pci/devices"
+PCI_ADDR_PATTERN='^[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-9a-f]$'
+
+# ── Which driver currently owns the device, or "none" ────────────────────────────────────────
+pcie_driver_of() {
+    local addr="$1" link
+    link=$(readlink "$PCI_DEVICE_ROOT/$addr/driver" 2>/dev/null) || { echo "none"; return 0; }
+    [[ -n "$link" ]] && basename "$link" || echo "none"
+}
+
+# ── Is the address passed through to a VM that is running right now? ─────────────────────────
+# Returns the domain name on stdout when claimed, empty when free.
+pcie_claimed_by_vm() {
+    local addr="$1"
+    command -v virsh >/dev/null 2>&1 || return 0
+
+    local domain bus slot func xml_addr dom
+    domain="${addr%%:*}"
+    bus="${addr#*:}"; bus="${bus%%:*}"
+    slot="${addr##*:}"; slot="${slot%%.*}"
+    func="${addr##*.}"
+
+    xml_addr="domain='0x${domain}' bus='0x${bus}' slot='0x${slot}' function='0x${func}'"
+
+    while read -r dom; do
+        [[ -z "$dom" ]] && continue
+        if virsh dumpxml "$dom" 2>/dev/null | awk '/<hostdev/,/<\/hostdev>/' | \
+           grep -qF "$xml_addr"; then
+            echo "$dom"
+            return 0
+        fi
+    done < <(virsh list --name --state-running 2>/dev/null)
+}
+
+# ==============================================================================================
+# ━━━ Status ━━━
+# ==============================================================================================
+if [[ "$SHOW_STATUS" == true ]]; then
+    echo ""
+    echo "━━━━━ $ICON_SUMMARY PCIe AER QUIET STATUS ━━━━━"
+    echo "$ICON_HOST   Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
+    echo "$ICON_GEAR   Enabled:   $PCIE_QUIET_ENABLED"
+    echo "$ICON_PLUGIN Devices:   ${#PCIE_QUIET_DEVICES[@]} listed"
+    echo ""
+
+    if [[ ${#PCIE_QUIET_DEVICES[@]} -eq 0 ]]; then
+        echo "  $ICON_SKIP No devices configured — nothing to do on this host"
+    else
+        for addr in "${PCIE_QUIET_DEVICES[@]}"; do
+            if [[ ! "$addr" =~ $PCI_ADDR_PATTERN ]]; then
+                echo "  $ICON_ERROR $addr — malformed address"
+            elif [[ ! -e "$PCI_DEVICE_ROOT/$addr" ]]; then
+                echo "  $ICON_SUCCESS $addr — already absent from the bus ✅"
+            else
+                echo "  $ICON_WARN   $addr — present, driver: $(pcie_driver_of "$addr")"
+            fi
+        done
+    fi
+    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+    exit 0
+fi
+
+# ==============================================================================================
+# ━━━ Gates ━━━
+# ==============================================================================================
+if [[ "$PCIE_QUIET_ENABLED" != true ]]; then
+    log "$ICON_SKIP PCIE_QUIET_ENABLED is false — nothing removed"
+    exit 0
+fi
+
+if [[ ${#PCIE_QUIET_DEVICES[@]} -eq 0 ]]; then
+    log "$ICON_SKIP No PCIe devices configured for $MY_ID — nothing to do"
+    exit 0
+fi
+
+# ==============================================================================================
+# ━━━ Remove ━━━
+# ==============================================================================================
+START=$(date +%s)
+
+REMOVED=0
+ALREADY=0
+REFUSED=0
+
+for addr in "${PCIE_QUIET_DEVICES[@]}"; do
+
+    # ── Shape ────────────────────────────────────────────────────────────────────────────────
+    if [[ ! "$addr" =~ $PCI_ADDR_PATTERN ]]; then
+        error "Malformed PCI address, refusing: '$addr' (expected DDDD:BB:DD.F)"
+        REFUSED=$(( REFUSED + 1 ))
+        continue
+    fi
+
+    # ── Present? ─────────────────────────────────────────────────────────────────────────────
+    if [[ ! -e "$PCI_DEVICE_ROOT/$addr" ]]; then
+        log "$ICON_SUCCESS $addr already absent"
+        ALREADY=$(( ALREADY + 1 ))
+        continue
+    fi
+
+    # ── Driver guard ─────────────────────────────────────────────────────────────────────────
+    DRIVER=$(pcie_driver_of "$addr")
+    if [[ "$DRIVER" != "vfio-pci" && "$DRIVER" != "none" ]]; then
+        error "$addr is bound to '$DRIVER' — refusing to remove a device in use"
+        error "Only vfio-pci-bound or unbound devices are eligible. Stub it first, or remove it from PCIE_QUIET_DEVICES."
+        notify "PCIe quiet refused $addr on $(hostname) ($MY_ID) — bound to $DRIVER" \
+            "PCIe AER Quiet" "warning"
+        REFUSED=$(( REFUSED + 1 ))
+        continue
+    fi
+
+    # ── Running VM guard ─────────────────────────────────────────────────────────────────────
+    CLAIMED_BY=$(pcie_claimed_by_vm "$addr")
+    if [[ -n "$CLAIMED_BY" ]]; then
+        error "$addr is passed through to running VM '$CLAIMED_BY' — refusing to remove"
+        notify "PCIe quiet refused $addr on $(hostname) ($MY_ID) — in use by VM $CLAIMED_BY" \
+            "PCIe AER Quiet" "warning"
+        REFUSED=$(( REFUSED + 1 ))
+        continue
+    fi
+
+    # ── Apply ────────────────────────────────────────────────────────────────────────────────
+    if [[ "$DRY_RUN" == true ]]; then
+        warn "DRY RUN — would remove $addr (driver: $DRIVER)"
+        continue
+    fi
+
+    log "Removing $addr (driver: $DRIVER)..."
+    if ! echo 1 > "$PCI_DEVICE_ROOT/$addr/remove" 2>/dev/null; then
+        error "Write to $PCI_DEVICE_ROOT/$addr/remove failed"
+        REFUSED=$(( REFUSED + 1 ))
+        continue
+    fi
+
+    # ── Verify ───────────────────────────────────────────────────────────────────────────────
+    if [[ -e "$PCI_DEVICE_ROOT/$addr" ]]; then
+        error "$addr still present after remove — kernel refused"
+        REFUSED=$(( REFUSED + 1 ))
+        continue
+    fi
+
+    echo "Removed $addr ✅"
+    REMOVED=$(( REMOVED + 1 ))
+done
+
+END=$(date +%s)
+
+# ==============================================================================================
+# ━━━ Summary ━━━
+# ==============================================================================================
+if [[ $REMOVED -eq 0 && $REFUSED -eq 0 ]]; then
+    log "$ICON_SUCCESS All ${ALREADY} configured device(s) already absent"
+    exit 0
+fi
+
+echo ""
+echo "━━━━━ $ICON_SUMMARY PCIe AER QUIET SUMMARY ━━━━━"
+echo "$ICON_HOST   Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
+echo "$ICON_SUCCESS Removed:   $REMOVED"
+echo "$ICON_SKIP  Already:   $ALREADY"
+[[ $REFUSED -gt 0 ]] && echo "$ICON_ERROR Refused:   $REFUSED"
+echo "$ICON_TIME   Duration:  $(format_duration $(( END - START )))"
+echo ""
+if [[ $REFUSED -gt 0 ]]; then
+    echo "$ICON_WARN Status: completed with refusals ⚠️"
+else
+    echo "$ICON_DONE Status: done ✅"
+fi
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+
+exit 0

Plugin/unraid/System_Essentials/php_fpm_max_children.sh

@@ -18,6 +18,19 @@
 # total. Too high wastes RAM; too low causes slowdowns.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Read the current pm.max_children from the PHP-FPM pool config
+#   2. Already at or above the target → exit silently, no write, no restart
+#   3. Otherwise rewrite the value and restart PHP-FPM via the adapter
+#   4. Verify PHP-FPM came back up
+#
+# Runs at array start, before the WebGUI sees real load. The setting does not survive an
+# unRAID update — the OS replaces the pool config — which is why this reapplies every boot
+# rather than being a one-time install step.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #

Plugin/unraid/System_Essentials/unraid_api_key_renew.sh

@@ -14,6 +14,20 @@
 # page always reflects the live key value.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Check whether a Varaverk key already exists in the unraid-api registry
+#   2. Create or overwrite it — the registry is ephemeral, so re-registering is the norm
+#   3. Write the resulting key into this host's conf, replacing any previous value
+#   4. Push the key into each partner's OWN conf, at the path their varaverk.cfg reports
+#   5. Report whether the key was created, refreshed, or unchanged
+#
+# Runs at array start. The registry does not survive OS updates or an unraid-api restart,
+# which is why this re-registers unconditionally rather than only when the key is missing —
+# a key present in the conf but absent from the registry is the exact failure it repairs.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -36,6 +50,33 @@
 #   Conf file check — aborts before any writes if the host conf is missing
 #   dry-run mode    — shows what would happen without touching anything
 #
+#   Remote path discovery
+#       The partner's conf path comes from resolve_remote_scripts_dir(), which reads their
+#       varaverk.cfg, so a partner in appdata storage mode is found. The path was hardcoded
+#       to the flash plugin directory, which is wrong for any such partner.
+#
+#   Remote target must exist
+#       The pushed script refuses to create the conf and reports the path it looked at.
+#       resolve_remote_scripts_dir() falls back to our own SCRIPTS_DIR when the probe fails,
+#       and appending an API key to a merely plausible path is how the hardcoded version
+#       failed without saying so.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# host*.conf
+#
+#   HOST*_UNRAID_API_KEY
+#       Written by this script every array start. Read by the plugin's PHP for enhanced
+#       monitoring. Treated as output, not input — an existing value is always replaced,
+#       because a conf value that no longer matches the registry is precisely the broken
+#       state this exists to fix.
+#
+# Platform-owned:
+#
+#   The unraid-api service registry — ephemeral, cleared by OS updates and service restarts.
+#
 # ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================
@@ -56,6 +97,10 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "$SCRIPT_DIR/../../../load_config.sh"
 
 parse_args "$@"
+
+# Rewrites the API key into host*.conf and registers it with the unraid-api service.
+[[ "$EUID" -ne 0 ]] && { error "Must be run as root"; exit 1; }
+
 acquire_lock
 detect_hosts
 
@@ -102,7 +147,14 @@ if [[ -n "$KEY" ]]; then
     PREVIEW="${KEY:0:8}...${KEY: -4}"
     # Always sync registry key → conf, even if the key was already there.
     # Conf gets wiped on git pull / conf regeneration without touching the registry.
-    CONF_HAS_KEY=$(grep -oP "(?<=^\s*${VAR_NAME}=\")[^\"]*" "$CONF_FILE" 2>/dev/null || true)
+    # Sourced, not pattern-matched. This was a grep -oP with a variable-length lookbehind, which
+    # Unraid's grep is ugrep and rejects outright — "length of lookbehind assertion is not
+    # limited", rc 2, swallowed by the || true. CONF_HAS_KEY was therefore always empty, the
+    # comparison below never matched, and this script logged "conf is stale — syncing" and
+    # rewrote the same key into the conf every fifteen minutes since it was written.
+    # Sourcing also means the value is read the way bash reads it, escapes and all.
+    CONF_HAS_KEY=$(bash -c 'source "$1" >/dev/null 2>&1 || exit 0; printf "%s" "${!2-}"' \
+                   _ "$CONF_FILE" "$VAR_NAME" 2>/dev/null || true)
     if [[ "$CONF_HAS_KEY" == "$KEY" ]]; then
         echo "API key valid ✅ — $VAR_NAME = $PREVIEW"
         log "Key in registry and conf — no action needed"
@@ -154,6 +206,9 @@ if [[ -z "$SSH_KEY" ]]; then
     exit 0
 fi
 
+# resolve_remote_scripts_dir() reads this; every inline timeout below already uses 10.
+SSH_TIMEOUT=10
+
 for host_var in $(compgen -v | grep -E '^HOST[0-9]+$'); do
     partner_host="${!host_var}"
     [[ -z "$partner_host" ]] && continue
@@ -163,16 +218,27 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$'); do
     partner_ip=$(resolve_tailscale_ip "$partner_host" 2>/dev/null || true)
     [[ -z "$partner_ip" ]] && { log "Cannot resolve IP for $partner_host — skipping"; continue; }
 
-    # Target is the partner's OWN conf on their machine
+    # Target is the partner's OWN conf on their machine. Their SCRIPTS_DIR is read from their
-    partner_conf="/boot/config/plugins/varaverk/Configurations/${partner_slot}.conf"
+    # varaverk.cfg rather than assumed — this was hardcoded to the flash plugin path, so a
+    # partner in appdata storage mode had its key appended under a directory that does not
+    # exist there. HOST2 has run in appdata mode since it was installed.
+    partner_sd=$(resolve_remote_scripts_dir "$partner_ip" "$SSH_KEY" "no")
+    partner_conf="${partner_sd}/Configurations/${partner_slot}.conf"
     tmp=$(mktemp /tmp/vv_kp_XXXXXX.sh)
     remote="/tmp/vv_kp_${RANDOM}.sh"
     chmod 700 "$tmp"
 
-    # Key stays in the temp file — never appears in SSH command args
+    # Key stays in the temp file — never appears in SSH command args.
+    # The conf must already exist: resolve_remote_scripts_dir falls back to our own SCRIPTS_DIR
+    # when the probe fails, and appending a key to a path that is merely plausible is how the
+    # hardcoded version failed silently. Report the path instead of guessing.
     cat > "$tmp" <<PUSHSCRIPT
 #!/bin/sh
 target='${partner_conf}'
+if [ ! -f "\$target" ]; then
+  echo "missing:\$target"
+  exit 1
+fi
 if grep -q "\b${VAR_NAME}\b" "\$target" 2>/dev/null; then
   sed -i 's|^\(\\s*${VAR_NAME}\\s*=\\s*\)"[^"]*"|\1"${KEY}"|' "\$target"
 else
@@ -183,13 +249,14 @@ PUSHSCRIPT
 
     if timeout 10 scp -i "$SSH_KEY" -o ConnectTimeout=10 -o BatchMode=yes \
         -o StrictHostKeyChecking=no "$tmp" "root@${partner_ip}:${remote}" 2>/dev/null; then
-        if timeout 10 ssh -i "$SSH_KEY" -o ConnectTimeout=10 -o BatchMode=yes \
+        push_out=$(timeout 10 ssh -i "$SSH_KEY" -o ConnectTimeout=10 -o BatchMode=yes \
             -o StrictHostKeyChecking=no "root@${partner_ip}" \
-            "bash '${remote}'; rc=\$?; rm -f '${remote}'; exit \$rc" 2>/dev/null | grep -q ok; then
+            "bash '${remote}'; rc=\$?; rm -f '${remote}'; exit \$rc" 2>/dev/null)
-            echo "Key pushed to $partner_host ✅"
+        case "$push_out" in
-        else
+            *ok*)      echo "Key pushed to $partner_host ✅" ;;
-            warn "Key push to $partner_host failed — they can create their own copy"
+            missing:*) warn "Key push to $partner_host failed — no conf at ${push_out#missing:}" ;;
-        fi
+            *)         warn "Key push to $partner_host failed — they can create their own copy" ;;
+        esac
     else
         warn "SCP to $partner_host failed — skipping"
     fi

Plugin/unraid/System_Essentials/user_scripts_stop.sh

@@ -32,6 +32,25 @@
 #   tree mid-execution.
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Name the Scripts, Not Just the PIDs
+#   Output resolves each process to the script it is running. "Stopping 4 processes" tells
+#   an operator nothing they can act on; "stopping mover_tuning, preclear" tells them
+#   exactly what they are about to lose and whether they should wait.
+#
+# Targeted, Never Blanket
+#   Only processes spawned by the User Scripts plugin are matched. A broad pattern would
+#   catch unrelated shells and Varaverk's own scripts — including, during a reboot sequence,
+#   the very script doing the stopping.
+#
+# Shutdown Step and Manual Tool, Same Behaviour
+#   Called automatically by server_reboot.sh and usable by hand on a stuck script. It takes
+#   no "reboot mode" — the correct action is identical either way, and a mode flag would be
+#   a second code path that only ever runs unattended.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
@@ -51,6 +70,21 @@
 #   No processes running = log() only, no visible output.
 #
 # ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# No config vars. Targets are discovered from running processes rather than from a list,
+# because the User Scripts plugin owns what exists and any list here would go stale.
+#
+# Platform-owned:
+#
+#   The unRAID User Scripts plugin's script directory and the processes it spawns. Matching
+#   is scoped to those — see Targeted, Never Blanket above for why that matters during a
+#   reboot sequence.
+#
+# Called by server_reboot.sh as a shutdown step; also safe to run by hand.
+#
+# ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================
 #

Plugin/unraid/Tools/README-Tools.md

@@ -0,0 +1,135 @@
+# ━━━━━ PLUGIN / UNRAID / TOOLS ━━━━━
+
+Platform-specific tooling for the WebGUI and the Unraid installation itself. Two of these keep
+the UI fast, two are operator tools for specific recovery and migration situations.
+
+They live here rather than in the top-level `Tools/` because each one manipulates something
+Unraid-specific — share `.cfg` files, the plugin's own PHP payload builders, or the
+installation's storage location.
+
+| File | Type | Runs |
+|------|------|------|
+| `api_cache_writer.sh` + `.php` | UI cache | **Every minute** via cron |
+| `remote_arr_cache_writer.sh` | UI cache | **Every 2 hours** via cron |
+| `recreate_shares.sh` | Recovery | Manual — after a rebuild or fresh install |
+| `storage_migrate.sh` | Migration | Manual — switching storage mode |
+
+---
+
+## ━━━ THE CACHE WRITERS — WHY PAGES LOAD INSTANTLY ━━━
+
+The monitor and arrs pages present a lot of state: per-core CPU, memory, network, GPUs,
+containers, VMs, transcodes, arr library counts. Building that live on every page view means
+dozens of API calls and `docker inspect` runs per refresh, from a WebGUI that is already the
+first thing to slow down under load.
+
+Instead, both are written to `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs — RAM-speed reads, cleared on reboot) and
+the pages serve from there.
+
+### ⚡  `api_cache_writer.sh` — local payloads, every minute
+
+A deliberate one-line shim: `php api_cache_writer.php`. All logic is in the PHP because the
+payload builders (`vv_monitor_*`, `vv_arrs_*`) are the **same functions the live API endpoints
+call**. A bash reimplementation would be a second version of the same payload, free to drift.
+
+Nothing here is allowed to break the UI:
+
+- A failed run simply leaves the cache unrefreshed — pages fall back to live calls. Slower,
+  still correct.
+- A missing or unparseable cache is treated as absent, never as empty data.
+- `?live=1` bypasses the cache entirely.
+- Runs unprivileged with no lock — a torn cache file is replaced within 60 seconds, and every
+  reader already has the live fallback.
+
+### 🌐  `remote_arr_cache_writer.sh` — partner payloads, every 2 hours
+
+SSHes to each partner and calls `vv_arrs_local_node()` **on their** PHP stack, caching the
+result locally as `arrs_remote_<hostid>.json`.
+
+The partner builds its own payload rather than this host querying the partner's arr APIs
+directly. That matters: the partner already has working local URLs and API keys for its own
+arrs, so no cross-host credentials and no path mapping are involved. This host never holds
+keys for a remote's arrs.
+
+A partner whose Tailscale IP will not resolve is skipped, not fatal — the arrs page shows what
+it has and falls back for the rest.
+
+---
+
+## ━━━ THE OPERATOR TOOLS ━━━
+
+### 🗂️  `recreate_shares.sh` — after a rebuild
+
+Recreates share **directories** on the correct disks by reading `/boot/config/shares/*.cfg`
+and honouring each share's `shareInclude` disk list.
+
+The `.cfg` files are Unraid's own record of what a share is and which disks it spans — they
+are the source of truth, not a list Varaverk maintains. Typically needed on a host after a
+full disk replacement or fresh install, where the config survived but the folders did not.
+
+**Create only, never delete.** A directory that already exists is left alone. This runs at a
+moment when the operator's picture of what should exist may be out of date, and removing
+anything on that basis is how a recovery step becomes a data-loss step.
+
+**Refuses to run without `/mnt/user` mounted.** Creating share directories against an
+unmounted array writes them into the underlying root filesystem, where they then shadow the
+real shares once the array does mount.
+
+### 📦  `storage_migrate.sh` — moving the whole installation
+
+Relocates Varaverk between the two storage modes:
+
+| Mode | `SCRIPTS_DIR` | Trade-off |
+|------|---------------|-----------|
+| Internal | `/boot/config/plugins/varaverk` | Direct git pull/push. Available before the array mounts. |
+| Flash | `/mnt/user/appdata/Varaverk` | Preserves USB flash lifetime. **Array must be started** for Varaverk to function at all. |
+
+This is the most destructive script in the folder — `rsync --delete`, `cp -a`, `rm -rf` — and
+it rewrites the pointers everything else derives from:
+
+```
+  varaverk.cfg   SCRIPTS_DIR                    ← the authoritative path
+  master.conf    TARGET_DIR
+  host*.conf     HOST*_STORAGE_MODE_INTERNAL
+  varaverk.cron  regenerated so job paths follow the new SCRIPTS_DIR
+```
+
+`STATE_DIR`, `DATA_DIR`, `PERSISTENT_CONF_CACHE` and every orchestrator job path are built
+from `SCRIPTS_DIR`. Changing storage mode moves all of them at once, which is why the cron is
+**regenerated** rather than edited.
+
+In flash mode, `git_pull_execute.sh` syncs `Plugin/` back to `/boot/` after each pull, so the
+WebGUI keeps serving current PHP even though the scripts live in appdata.
+
+---
+
+## ━━━ SAFEGUARDS ━━━
+
+**The cache writers are deliberately unprivileged and lockless.** Their output is regenerable
+within a minute and every consumer has a live fallback. There is no privileged operation to
+gate and no state worth locking. This is documented in their headers so it is not "corrected"
+later.
+
+**The operator tools are the opposite.** Both enforce root, take a lock, and support
+`--dry-run`. `storage_migrate.sh` additionally exits cleanly if already in the requested mode,
+and removes the old location only once the new one is confirmed in place.
+
+**`recreate_shares.sh` will not act on an unmounted array** — see above; this is the guard
+that prevents shadow directories.
+
+---
+
+## ━━━ RELATIONSHIP TO THE WEB UI ━━━
+
+```
+  pages/monitor.php ─┐
+                     ├─► include/monitor.php  (vv_monitor_*)
+  api/monitor.php ───┘            ▲
+                                  │
+  Tools/api_cache_writer.php ─────┘   same builders → cache can't disagree with live
+        │
+        └─► /tmp/varaverk/api/monitor.json ──► served by api/monitor.php unless ?live=1
+```
+
+Adding a metric means adding it in `include/` once. The page, the live endpoint, and the cache
+all pick it up together.

Plugin/unraid/Tools/ai_explain_check.sh

@@ -0,0 +1,177 @@
+#!/bin/bash
+# ==============================================================================================
+# =============================== AI Explain Check =============================================
+# ==============================================================================================
+#
+# PURPOSE
+#   Runs every case in ai_explain_fixtures.txt through the worker's --explain mode and checks the
+#   routing it reports against what the fixture says it should be. Catches a guard that has been
+#   undone by a later guard, which is the failure this subsystem keeps producing.
+#
+# OPERATIONAL MODEL
+#   Never calls the model. --explain stops on the line where deterministic assembly ends, so a
+#   full pass costs about a second per case and returns the same answer every time. There is no
+#   Ollama dependency, no token spend, and no flaky wording to chase.
+#
+#   Not scheduled and deliberately not in any orchestrator. This is a development check — it runs
+#   when the routing changes, not every night. Nothing on the running system depends on it.
+#
+# DESIGN PRINCIPLES
+#   Asserts routing, never wording.
+#       Which capabilities a profile holds and which evidence was attached are decided before the
+#       model is asked anything. Asserting on generated prose would fail for reasons that tell
+#       nobody anything and the check would be ignored within a fortnight.
+#
+#   A malformed assertion fails loudly rather than passing quietly.
+#       An unrecognised key is an error, not a skip. A typo in an assertion that silently passes
+#       is worse than no assertion, because the line still reads as covered.
+#
+# OPERATIONAL SAFEGUARDS
+#   Read-only. Runs fixtures through --explain and compares; writes nothing and changes no state.
+#
+#   Exits non-zero on any failure, so it can gate a commit.
+#
+# RUNTIME MODES
+#   ai_explain_check.sh            check every fixture
+#   ai_explain_check.sh --verbose  print the full explain report for each case
+#   ai_explain_check.sh <pattern>  only cases whose question matches the pattern
+#
+# DEPENDS ON
+#   Plugin/unraid/Tools/ai_chat_worker.php   --explain mode
+#   Plugin/unraid/Tools/ai_explain_fixtures.txt
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+
+set -uo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+WORKER="$HERE/ai_chat_worker.php"
+FIXTURES="$HERE/ai_explain_fixtures.txt"
+
+VERBOSE=false
+FILTER=""
+for a in "$@"; do
+    case "$a" in
+        --verbose) VERBOSE=true ;;
+        *)         FILTER="$a" ;;
+    esac
+done
+
+[[ -f "$WORKER"   ]] || { echo "missing worker: $WORKER"; exit 2; }
+[[ -f "$FIXTURES" ]] || { echo "missing fixtures: $FIXTURES"; exit 2; }
+
+PASS=0; FAIL=0; SKIP=0
+FAILED_LINES=()
+
+trim() { local s="$1"; s="${s#"${s%%[![:space:]]*}"}"; s="${s%"${s##*[![:space:]]}"}"; printf '%s' "$s"; }
+
+# One field out of the explain report. Everything it reads is a fixed label printed by --explain.
+field() {
+    local report="$1" label="$2"
+    printf '%s' "$report" | grep -m1 -E "^ *$label " | sed -E "s/^ *$label +//" | sed -E 's/ +$//'
+}
+
+lineno=0
+while IFS= read -r raw || [[ -n "$raw" ]]; do
+    lineno=$((lineno + 1))
+    line="$(trim "$raw")"
+    [[ -z "$line" || "$line" == \#* ]] && continue
+
+    IFS='|' read -r q prof scope kind expect <<< "$line"
+    q="$(trim "$q")"; prof="$(trim "$prof")"; scope="$(trim "$scope")"
+    kind="$(trim "$kind")"; expect="$(trim "$expect")"
+    [[ -z "$q" ]] && continue
+
+    if [[ -n "$FILTER" && "$q" != *"$FILTER"* ]]; then SKIP=$((SKIP + 1)); continue; fi
+
+    report="$(php "$WORKER" --explain "$q" "$prof" "$scope" "$kind" 2>&1)"
+    if [[ $? -ne 0 ]]; then
+        echo "✗ line $lineno: explain failed — $q"
+        printf '%s\n' "$report" | head -3 | sed 's/^/    /'
+        FAIL=$((FAIL + 1)); FAILED_LINES+=("$lineno"); continue
+    fi
+
+    # PROFILE prints "asked -> used (escalated)" on a handoff and just the profile otherwise.
+    got_profile="$(printf '%s' "$report" | grep -m1 '^PROFILE' | sed -E 's/^PROFILE +//')"
+    got_profile="${got_profile##*-> }"
+    got_profile="$(printf '%s' "$got_profile" | sed -E 's/ *\(.*\)$//')"
+    got_caps="$(printf '%s' "$report" | grep -m1 '^CAPS' | sed -E 's/^CAPS +//')"
+    got_target="$(field "$report" 'named target')"
+    got_run="$(field "$report" 'run outcome')"
+    got_diag="$(field "$report" 'diagnostic')"
+    got_attached="$(printf '%s' "$report" | sed -n '/^ATTACHED/,/^$/p' | tail -n +2 \
+                    | awk 'NF {print $1}' | tr '\n' ',' )"
+
+    problems=()
+    for assert in $expect; do
+        key="${assert%%=*}"; want="${assert#*=}"
+        case "$key" in
+            profile)
+                [[ "$got_profile" == "$want" ]] || problems+=("profile: want $want, got $got_profile") ;;
+            target)
+                if [[ "$want" == "none" ]]; then
+                    [[ "$got_target" == "(none resolved)" ]] || problems+=("target: want none, got $got_target")
+                else
+                    [[ "$got_target" == "$want" ]] || problems+=("target: want $want, got $got_target")
+                fi ;;
+            run)
+                exp=$([[ "$want" == "yes" ]] && echo YES || echo no)
+                [[ "$got_run" == "$exp" ]] || problems+=("run outcome: want $exp, got $got_run") ;;
+            diag)
+                exp=$([[ "$want" == "yes" ]] && echo YES || echo no)
+                [[ "$got_diag" == "$exp" ]] || problems+=("diagnostic: want $exp, got $got_diag") ;;
+            caps)
+                # none        holds nothing at all
+                # only:a,b    holds exactly these and nothing else
+                # a           holds at least this
+                #
+                # only: exists because General Chat's guarantee is not "it can search" but "search
+                # is the only thing it can do". A presence check would still pass on the day
+                # something else is granted there, which is the day the check was written for.
+                if [[ "$want" == "none" ]]; then
+                    [[ "$got_caps" == "(none"* ]] || problems+=("caps: want none, got $got_caps")
+                elif [[ "$want" == only:* ]]; then
+                    want_set="${want#only:}"
+                    got_set="$(printf '%s' "$got_caps" | tr -d ' ' | tr ',' '\n' | sort | paste -sd, -)"
+                    exp_set="$(printf '%s' "$want_set" | tr -d ' ' | tr ',' '\n' | sort | paste -sd, -)"
+                    [[ "$got_set" == "$exp_set" ]] \
+                        || problems+=("caps: want exactly [$exp_set], got [$got_set]")
+                else
+                    [[ "$got_caps" == *"$want"* ]] || problems+=("caps: want $want in [$got_caps]")
+                fi ;;
+            has)
+                IFS=',' read -ra want_keys <<< "$want"
+                for w in "${want_keys[@]}"; do
+                    [[ ",$got_attached" == *",$w,"* ]] || problems+=("missing attachment: $w")
+                done ;;
+            hasnt)
+                IFS=',' read -ra bad_keys <<< "$want"
+                for b in "${bad_keys[@]}"; do
+                    [[ ",$got_attached" == *",$b,"* ]] && problems+=("attached but must not be: $b")
+                done ;;
+            *)
+                problems+=("unknown assertion '$key' — typo, or a key this checker does not know") ;;
+        esac
+    done
+
+    if [[ ${#problems[@]} -eq 0 ]]; then
+        PASS=$((PASS + 1))
+        printf '\033[32m✓\033[0m %-6s %s\n' "$prof" "$q"
+    else
+        FAIL=$((FAIL + 1)); FAILED_LINES+=("$lineno")
+        printf '\033[31m✗\033[0m %-6s %s\n' "$prof" "$q"
+        for p in "${problems[@]}"; do echo "      $p"; done
+    fi
+
+    [[ "$VERBOSE" == true ]] && printf '%s\n\n' "$report" | sed 's/^/    /'
+done < "$FIXTURES"
+
+echo
+echo "─────────────────────────────────────────────"
+printf 'passed %d   failed %d' "$PASS" "$FAIL"
+[[ $SKIP -gt 0 ]] && printf '   skipped %d' "$SKIP"
+echo
+if [[ $FAIL -gt 0 ]]; then
+    echo "failing fixture lines: ${FAILED_LINES[*]}"
+    exit 1
+fi
+exit 0