Complete the header template across Partnership, Kernel, Deployment and Plugin

Finishes the pass: every script now documents its safeguards, and the deliberate absences
in the sourced libraries are recorded so they are not "corrected" later.

Partnership/gitea_ssh_setup.sh

@@ -41,6 +41,31 @@
 #        Skipped if entry already exists — use --force to overwrite
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Separate Key From the Rsync Key
+#   Gitea authentication uses its own keypair rather than reusing the rsync automation key.
+#   The two have different lifecycles and different blast radii — re-keying Gitea should
+#   never break cross-host rsync, and revoking a partner's rsync access should not lock this
+#   host out of its own repository.
+#
+# Idempotent Registration
+#   Both generation and registration are skipped when already satisfied, and registration is
+#   matched on the public key itself rather than on a label. A key registered under a
+#   different title is still the same key, and re-registering it would leave duplicate
+#   entries accumulating in Gitea on every re-run.
+#
+# Locate, Do Not Assume
+#   The API endpoint is resolved at runtime — container-local first, then GITEA_DOMAIN. Same
+#   reasoning as git_pull_execute.sh: Gitea's address changes with container restarts and
+#   migrations, so hardcoding it guarantees an eventual break.
+#
+# Force Is Explicit
+#   Regeneration invalidates the key already registered in Gitea, so it requires --force. A
+#   bare re-run can never cost this host its repository access.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #

Partnership/onboard_cancel.sh

@@ -8,6 +8,102 @@
 # Removes SSH keys between HOST1 and HOST2 in the specified direction.
 # Safe to run at any phase. Clears related setup.db flags.
 #
+# The escape hatch for a half-finished onboard: partnership setup is multi-phase, and an
+# attempt abandoned midway leaves keys installed and phase flags set. This unwinds that so
+# onboarding can be started cleanly rather than resumed from an unknown state.
+#
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Per direction requested:
+#
+#   h1 (HOST1 → HOST2)
+#     1. SSH to HOST2 and delete HOST1's public key line from its authorized_keys
+#     2. Clear HOST2's phase / key-ready flags from the setup db
+#     3. Delete the local HOST1 key pair
+#
+#   h2 (HOST2 → HOST1)
+#     Remove HOST2's public key from HOST1's authorized_keys, identifying the key by
+#     HOST2's hostname in the key comment.
+#
+#   both — run each direction in turn.
+#
+# Key removal is matched on the key blob or hostname comment, never on line number.
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Safe at Any Phase
+#   Onboarding is multi-phase and can fail anywhere in it. This runs against whatever state
+#   exists rather than requiring a known starting point — a missing key or an absent flag is
+#   a no-op, not an error. Cancelling twice is harmless.
+#
+# Direction Is Explicit
+#   Removing a key is not symmetric: it breaks authentication for whichever side loses it.
+#   The direction must be stated, and defaults to h1 (this host's own outbound key) rather
+#   than to both, so an unqualified run cannot sever the partner's access to you.
+#
+# Unwind, Do Not Repair
+#   The job is to return to a clean pre-onboard state so onboarding can be re-run from the
+#   top. It deliberately does not attempt to salvage or resume a partial setup — a known
+#   empty state is worth more than a guessed-at partial one.
+#
+# Keys and Flags Together
+#   Removing the key without clearing the setup-db flags would leave onboarding believing a
+#   phase had completed. Both are cleared in the same pass for that reason.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Deletes key pairs from /root/.ssh and edits authorized_keys on both ends as root.
+#
+# Lock Acquisition
+#   acquire_lock prevents this racing an in-progress onboard, which would otherwise be
+#   installing the very keys this is removing.
+#
+# Host Detection
+#   detect_hosts() resolves MY_ID / REMOTE_ID so the direction flags map to real hosts.
+#
+# Direction Default
+#   Defaults to h1 — never removes the partner's inbound key unless explicitly asked.
+#
+# Targeted Key Removal
+#   authorized_keys lines are matched by key blob or hostname comment. Nothing is removed
+#   positionally, so an unrelated key can never be deleted because it sat on a given line.
+#
+# SSH Timeout
+#   The remote edit is wrapped in SSH_TIMEOUT — an unreachable partner fails fast rather
+#   than hanging a cancel that still has local cleanup to do.
+#
+# Idempotent
+#   Absent keys and absent flags are skipped silently. Re-running is safe.
+#
+# Dry Run Support
+#   --dry-run reports every key and flag it would remove, and removes none.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   SSH_TIMEOUT
+#       Bounds the remote authorized_keys edit.
+#
+# host*.conf
+#
+#   SSH_KEY
+#       Key used to reach the partner, and the local pair deleted in the h1 direction.
+#
+#   HOST* — hostnames, used to identify which key comment belongs to which side.
+#
+# Setup state lives in the platform setup db (platform_setup_db_path) — the phase and
+# key-ready flags cleared here are the same ones partnership_onboard.sh sets.
+#
 # ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================

Partnership/partnership_offboard.sh

@@ -37,6 +37,79 @@
 #   Step 8: SSH revocation      — revoke keys both directions, write state, signal owner
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Sync Before Severing
+#   The final sync (owner step 2) runs before any state change, so the mirror leaves with
+#   current Critical-Data rather than a snapshot from whenever the last scheduled sync
+#   happened. Once keys are revoked there is no second chance to move data.
+#
+# Stop the Sync First
+#   Step 1 on both paths halts rsync before anything else. A sync running through a
+#   partnership teardown would be writing to a partner that is having its access removed
+#   underneath it.
+#
+# Revoke Last, Not First
+#   SSH keys and Emby admin are revoked at the end. Every earlier step needs working remote
+#   access — revoking up front would strand the remaining cleanup on the far side and leave
+#   the mirror holding containers nobody can remove.
+#
+# Both Sides Land Somewhere Valid
+#   Each path restarts the host's own parked containers before finishing. Offboarding must
+#   leave two working standalone servers, not one working server and one stripped of the
+#   coverage it was relying on.
+#
+# Role Detected, Not Declared
+#   Owner and mirror run different sequences, and the role is derived rather than passed in.
+#   A human choosing the wrong path would run the owner's remote-cleanup steps against a
+#   server that never deployed anything.
+#
+# Blocklist Is the Enforcement
+#   Writing INACTIVE state is not enough on its own — a stale cron or a script mid-flight
+#   could still attempt a sync. The mirror is blocklisted so rsync.sh refuses it outright,
+#   independently of whatever any config still says.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Container removal, conf edits and SSH key revocation all require root.
+#
+# Lock Acquisition
+#   acquire_lock "strict" — an offboard is not resumable partway, so a second instance is
+#   refused rather than queued behind the first.
+#
+# Host Detection
+#   detect_hosts() resolves MY_ID / REMOTE_ID, which the role detection builds on.
+#
+# Docker Presence Check
+#   Verified before any container removal is attempted.
+#
+# Ordered Teardown
+#   The step sequence is the safeguard — sync, then reconfigure, then remove, then restart,
+#   then revoke, then record. Reordering breaks the guarantees above.
+#
+# Own Stack Restored
+#   Parked containers are brought back up on both sides before the run completes.
+#
+# State Written Both Ends
+#   INACTIVE is written locally and pushed to the mirror, so neither side is left believing
+#   a partnership is still active.
+#
+# Partner Blocklisted
+#   The mirror is added to the partnership blocklist, which rsync.sh checks and refuses on —
+#   stale access cannot survive the offboard.
+#
+# Tailscale Grace Window
+#   Device removal happens after state is written, not before, so the final state push
+#   cannot be cut off by removing its own transport.
+#
+# Dry Run Support
+#   --dry-run walks the full sequence reporting each step without executing any.
+#
+# ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
 #

Partnership/partnership_transfer.sh

@@ -28,6 +28,31 @@
 #   Step 5:  Write State          — ACTIVE written locally and pushed to new mirror
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Config Changes Hands, Data Does Not
+#   No containers move and no appdata is copied. A transfer rewrites who is authoritative
+#   and where WebUIs point; the sync direction reverses naturally on the next fallback or
+#   critical-sync cycle. Keeping data out of the transfer is what makes it cheap enough to
+#   be reversible.
+#
+# Owner Initiates, Always
+#   Only the current owner can run this. The owner holds the authoritative config, so a
+#   mirror-initiated transfer would be writing ownership state it does not own — and if both
+#   sides ran it, neither would be owner.
+#
+# Prove Health Before Swapping
+#   Both servers must pass consecutive health checks first. Handing ownership to a partner
+#   that is unhealthy converts a recoverable situation into an outage with the authoritative
+#   side on the weaker host.
+#
+# Roles Swap Atomically
+#   Owner and mirror are two ends of one relationship, not independent flags. Any window
+#   where both believe they are owner — or neither does — is worse than the transfer simply
+#   failing, so the swap is written as one transition rather than two updates.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #

Partnership/share_setup.sh

@@ -25,7 +25,96 @@
 # Remote admin assigns disk sets from the Unraid UI after onboarding.
 #
 # ==============================================================================================
-# USAGE
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Resolve the mirror and its Tailscale IP — unresolvable aborts before any remote call
+#   2. Detect the remote's appdata cache pool from its own appdata.cfg (default: cache)
+#   3. For each path across the owner's daily/weekly/critical/intermediate share lists:
+#        a. Extract the top-level Unraid share name
+#        b. Remote already has a .cfg for it? → skip, never modify
+#        c. Otherwise use the LOCAL .cfg as a template:
+#             - substitute the remote's detected pool
+#             - clear disk include/exclude (disk layouts differ per server)
+#        d. Write the .cfg to remote /boot/config/shares/ and mkdir -p the share directory
+#        e. Sub-paths (e.g. appdata-Fallback/Critical-Data) get their subdir created after
+#           the top-level share exists
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Create Only, Never Modify
+#   An existing remote .cfg is always left alone. The remote admin may have deliberately
+#   tuned a share's pool, allocation or disk set — this script has no way to tell an
+#   intentional setting from a stale one, so it never overwrites.
+#
+# Disk Layout Is Not Portable
+#   Include/exclude lists are cleared rather than copied, because the two servers have
+#   different disks. Copying the owner's disk set onto a mirror with a different array
+#   would produce a share pointing at disks that do not exist.
+#
+# Detect the Pool, Do Not Assume It
+#   The remote's cache pool name is read from its own appdata.cfg rather than hardcoded or
+#   copied from local. Pool names differ per server and a wrong one silently lands appdata
+#   on the array.
+#
+# Media Shares Land Array-Only
+#   Shares with shareUseCache=no are created without a pool assignment. Disk sets are the
+#   remote admin's call from the Unraid UI after onboarding.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Reads $SSH_KEY from /root/.ssh and the local /boot/config/shares/*.cfg, and writes share
+#   configs onto the mirror as root.
+#
+# Lock Acquisition
+#   acquire_lock prevents concurrent runs. Two instances could both observe a share as
+#   missing and race to create it.
+#
+# Host Detection
+#   detect_hosts() resolves REMOTE_SERVER_NAME — the mirror this script targets.
+#
+# Mirror Resolution Guard
+#   Aborts if REMOTE_SERVER_NAME is unset or its Tailscale IP cannot be resolved, before any
+#   remote command is attempted.
+#
+# Existing Share Protection
+#   A remote .cfg that already exists is never touched — see Create Only above.
+#
+# SSH Timeouts and BatchMode
+#   Every remote call uses ConnectTimeout and BatchMode=yes, so an unreachable or
+#   password-prompting mirror fails fast instead of hanging the onboarding run.
+#
+# Pool Fallback
+#   An undetectable remote pool defaults to "cache" rather than writing an empty pool name
+#   into the share config.
+#
+# Dry Run Support
+#   --dry-run reports every share it would create and writes nothing.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# host*.conf (aliased by detect_hosts())
+#
+#   HOST*_DAILY_SYNC_SHARES / _WEEKLY_ / _CRITICAL_ / _INTERMEDIATE_
+#       The share lists this script reads. Any path appearing in one of these on the owner
+#       is a share the mirror is expected to have.
+#
+#   SSH_KEY
+#       Key used for every remote call. Written by ssh_setup.sh.
+#
+# master.conf
+#
+#   HOST* — hostnames, used to resolve the mirror via detect_hosts()
+#
+# ==============================================================================================
+# RUNTIME MODES
 # ==============================================================================================
 #
 # share_setup.sh
@@ -40,9 +129,21 @@ set -uo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "$SCRIPT_DIR/../load_config.sh"
-detect_hosts
 parse_args "$@"
 
+# ==============================================================================================
+# ━━━ Setup ━━━
+# ==============================================================================================
+# Reads $SSH_KEY from /root/.ssh, reads local /boot/config/shares/*.cfg, and writes share
+# configs onto the mirror as root.
+[[ "$EUID" -ne 0 ]] && { error "Must be run as root"; exit 1; }
+
+# Writes share .cfg files to the mirror. Two concurrent runs could both see a share as
+# missing and race to create it.
+acquire_lock
+
+detect_hosts
+
 # ── Resolve mirror ────────────────────────────────────────────────────────────────────────────
 
 [[ -z "${REMOTE_SERVER_NAME:-}" ]] && { error "Cannot determine mirror hostname — check HOST* in master.conf"; exit 1; }

Partnership/ssh_setup.sh

@@ -2,40 +2,137 @@
 # ==============================================================================================
 # ============================= SSH Setup ======================================================
 # ==============================================================================================
+#
+# PURPOSE
+# ─────────────────────────────────────────────────────────────────────────────
 # Generates the SSH keypair for rsync automation and installs it on the remote server.
+# Every cross-host operation in the ecosystem — rsync, conf sync, fallback container
+# control, the upgrade webhook — authenticates with this key. If it is missing or broken,
+# the mesh silently degrades to single-host.
+#
 # Key named after this server: hostname lowercased, unraid- prefix stripped.
-#   unRAID-Gmer4Lfe → gmer4lfe_rsync_automation
+#   unRAID-Gmer4Lfe  → gmer4lfe_rsync_automation
 #   unRAID-Jayred365 → jayred365_rsync_automation
-# Idempotent — skips generation if key already exists (use --force to regenerate).
-# Updates host*.conf with key path on success.
 #
-# ── MODES ─────────────────────────────────────────────────────────────────────────────────────
-#   (default)       — generate key if missing, copy to remote, update conf
-#   --force         — regenerate key even if it exists, re-copy to remote
-#   --validate      — test SSH auth to remote, track strikes, notify at limit
-#   --status        — show key state, fingerprint, remote connectivity
-#   --dry-run       — preview without creating, copying, or updating conf
+# Idempotent — skips generation if the key already exists (--force to regenerate).
+# Updates host*.conf with the key path on success.
 #
-# ── STRIKE SYSTEM (--validate) ────────────────────────────────────────────────────────────────
-# Called during partnership --check cycles to detect broken SSH auth.
-# Tracks consecutive SSH auth failures — not network unreachability.
-# Remote Tailscale IP unreachable = network issue → not counted as SSH strike.
-# Remote reachable but SSH auth fails = key issue → strike incremented.
-# Strikes reset automatically after SSH_STRIKE_RESET_HRS of clean connectivity.
-# At SSH_MAX_STRIKES: notify + return exit 2 (caller can escalate).
-# State: DATA_DIR/ssh_strikes_{REMOTE_SERVER_NAME}.db
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
 #
-# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
-#   SSH_MAX_STRIKES       — consecutive failures before notifying (default 5)
-#   SSH_STRIKE_RESET_HRS  — hours since last failure before counter resets (default 24)
+# Setup (default):
+#   1. Key exists? → skip generation unless --force
+#   2. Generate keypair, named from this host
+#   3. Copy the public key to the remote's authorized_keys
+#   4. Verify authentication actually works before claiming success
+#   5. Write the key path into host*.conf
+#
+# Validate (--validate), called during partnership --check cycles:
+#   Remote unreachable on Tailscale → network problem, NOT counted as a strike
+#   Remote reachable but SSH auth fails → key problem, strike incremented
+#   Clean connectivity for SSH_STRIKE_RESET_HRS → strikes reset automatically
+#   At SSH_MAX_STRIKES → notify and return exit 2 so the caller can escalate
+#
+#   State: DATA_DIR/ssh_strikes_{REMOTE_SERVER_NAME}.db
+#
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Distinguish Unreachable From Unauthorised
+#   The strike system counts SSH auth failures only. A partner that is simply offline is a
+#   network condition, and counting it would fire a key-rotation alarm every time the remote
+#   reboots. Only "I can reach you but you will not let me in" is a key problem.
+#
+# Idempotent by Default, Destructive Only on Request
+#   A bare run never replaces an existing key. Regenerating invalidates every authorized_keys
+#   entry the old key was in — including on hosts this script is not talking to right now —
+#   so it requires --force explicitly.
+#
+# Verify Before Recording
+#   The key path is written into host*.conf only after authentication has been proven to
+#   work. Recording a key that does not authenticate would leave every downstream script
+#   pointed at a credential that silently fails.
+#
+# Strikes Reset on Recovery
+#   Counters clear themselves after a period of clean connectivity, so a transient outage
+#   does not accumulate toward an alarm across unrelated weeks.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# Root Enforcement
+#   Reads and writes /root/.ssh and installs keys on the remote as root.
+#
+# Lock Acquisition
+#   acquire_lock prevents concurrent runs. Two instances generating or copying keys at once
+#   could leave authorized_keys holding a key whose private half was already replaced.
+#
+# Host Detection
+#   detect_hosts() resolves MY_ID / REMOTE_ID for key naming and remote targeting.
+#
+# Existing Key Protection
+#   Generation is skipped when a key is present. Overwriting requires --force.
+#
+# Auth Verified Before Conf Write
+#   host*.conf is updated only after a successful authentication test.
+#
+# Network-vs-Auth Discrimination
+#   Unreachable remotes never increment the strike counter — see Design Principles.
+#
+# Strike Ceiling
+#   SSH_MAX_STRIKES bounds how long a genuinely broken key goes unreported, and exit 2 lets
+#   the caller decide whether that is escalation-worthy.
+#
+# Local-Only Escape Hatch
+#   --local-only generates the key without touching the remote, for onboarding a partner
+#   that is not reachable yet.
+#
+# Dry Run Support
+#   --dry-run previews generation, copy and conf update without performing any.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   SSH_MAX_STRIKES
+#       Consecutive SSH auth failures before notifying (default: 5)
+#
+#   SSH_STRIKE_RESET_HRS
+#       Hours of clean connectivity before the strike counter resets (default: 24)
+#
+# host*.conf
+#
+#   HOST*_SSH_KEY
+#       Written by this script on success. Read by rsync.sh, conf_sync.sh, fallback.sh and
+#       upgrade_webhook_handler.sh — every cross-host operation depends on it.
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# ssh_setup.sh
+#     Initial setup — generate if missing, copy to remote, update conf. Idempotent.
+#
+# ssh_setup.sh --force
+#     Regenerate the key even if one exists, and re-copy to the remote.
+#
+# ssh_setup.sh --validate
+#     Health check with strike tracking. Exit 2 at the strike limit.
+#
+# ssh_setup.sh --status
+#     Show key state, fingerprint, and remote connectivity. Then exit.
+#
+# ssh_setup.sh --local-only
+#     Generate the key locally and skip the remote copy.
+#
+# ssh_setup.sh --dry-run / --log
+#     Supported by every mode above.
 #
-# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
-#   Partnership/ssh_setup.sh                — initial setup (idempotent)
-#   Partnership/ssh_setup.sh --force        — regenerate + re-copy
-#   Partnership/ssh_setup.sh --validate     — health check + strike tracking
-#   Partnership/ssh_setup.sh --status       — show key and connectivity state
-#   Partnership/ssh_setup.sh --local-only   — generate key locally, skip remote copy
-#   Any mode supports --dry-run and --log
 # ==============================================================================================
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"