Varaverk/Partnership/README-Partnership.md

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🤝 PARTNERSHIP

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Managed lifecycle for a two-server partnership — setup, ongoing operation, and clean separation. One server owns the shared services. The other mirrors them and benefits from them. Every phase of the relationship has the same engineering discipline as the rest of the ecosystem.

This folder exists because a clean exit should be as easy as a clean setup. The partnership is not a permanent commitment. --offboard works from either server at any time. Everything the mirror needs to run independently is already there. The only thing that stops on separation is the sync — and that's intentional.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

---

🔴 Two Servers, One Auth Stack, No Clean Way to Share It

The auth stack — NginxProxyManager, LLDAP, Authelia, MariaDB, Redis — runs on HOST1. HOST2 serves its own domain to its own household. It needs its own auth. But maintaining two independent auth stacks means double the work: two places to add users, two places to update proxy rules, two places to renew certs, two configurations that inevitably drift apart. One change on HOST1 has to be manually replicated to HOST2 — or it isn't, and the configurations diverge silently.

The real cost isn't the initial setup. It's the maintenance burden that accumulates over months — every new user, every proxy rule change, every config update applied in one place and forgotten in the other.

The fix: one auth stack with a managed mirror. HOST1 owns the configuration. HOST2 runs a warm copy that stays current via 30-minute sync. HOST2's operator makes zero auth management decisions — clicking an auth container opens HOST1's WebUI via Tailscale. Changes happen there, propagate to HOST2 in 30 minutes. One place to manage everything for both households.

---

🔴 No Structure Around the Relationship Itself

Setting up the mirror was a manual process. SSH in, reconfigure container WebUI URLs one by one, copy auth config, verify connectivity, update state tracking. No defined sequence. No dry-run capability. No verification that each step worked. If something went wrong midway, the mirror was in an inconsistent state with no clear way to understand what had and hadn't been done. Offboard was worse — it involves stopping a sync that's been running for months, making a final copy of data, reconfiguring WebUIs back to local addresses, removing Tailscale access, and notifying both servers. A manual process with that many steps, taken under pressure, leaves one or both parties in a bad state.

The fix: partnership_manager.sh with explicit modes for each lifecycle phase. Each mode is a defined sequence. Every step is verified. Dry-run shows exactly what will happen before anything changes. State files make the current relationship status unambiguous from either server.

---

🔴 No Safe Way to Check If the Other Server Has Gone Away

After months of operation, HOST2 goes quiet. The sync starts failing. The offline counter increments. But nothing actually happens — the ecosystem just keeps failing the same sync, incrementing the same counter, sending the same notifications. Without a defined threshold and an automated response, "partner gone for 30 days" looks exactly like "partner gone for 3 years."

The fix: PARTNERSHIP_OFFLINE_THRESHOLD. After this many days of missed sync cycles, both servers independently auto-offboard. HOST1 removes HOST2 from Tailscale, disables critical sync, writes INACTIVE state. HOST2 — if it eventually comes back — reads HOST1's INACTIVE state and cleans up its own side. The relationship is formally ended from both sides without anyone needing to be present.

---

🔴 Ownership Transfer Had No Safe Path

The arrangement was always intended to be flexible — HOST1 owns the auth stack now, but circumstances change. Swapping ownership manually meant reconfiguring WebUIs on both servers, swapping sync direction, updating master.conf on both, and hoping the sequence was correct. A misstep — like flipping sync direction before the final sync completed — leaves both servers with different auth configurations and no clear source of truth.

The fix: --transfer with a required confirmation string, a consecutive health check system, and a strict sequence. The confirmation string cannot be typed accidentally. Health strikes require both servers to be healthy on multiple consecutive checks before the transfer begins. A final sync in the current direction runs before anything is flipped.

---

━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Onboarding (partnership_onboard.sh) — one-time setup run from both servers. Generates SSH keys, installs the auth stack and arr stack on the mirror from XML templates, configures WebUI redirects to the owner, and bootstraps the arr library. Role is detected automatically — no flags needed to declare which side you are.

Offboarding (partnership_offboard.sh) — handles clean separation from either role. Owner path: final sync, WebUI reconfigure, remote container + appdata cleanup (auth/arr stack by XML array, fallback coverage by naming), SSH key revocation, Tailscale removal. Mirror path: local WebUI reconfigure, remove owner-deployed containers locally, disable sync, revoke Emby admin, SSH key revocation, signal owner. Called by partnership_manager.sh --offboard but runnable directly.

Lifecycle management (partnership_manager.sh) — dispatcher and monitor.

• Manually: --onboard, --offboard (delegates to offboard script), --transfer, --status, --unblock
• Automatically: --check called every 30 minutes by critical_sync_maintenance.sh

SSH management (ssh_setup.sh) — generates the keypair for rsync automation, installs it on the remote, and tracks auth failures with a configurable strike system. Called by partnership_onboard.sh but runnable independently for validation and re-keying.

---

━━━ RELATIONSHIP TO OTHER FOLDERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Orchestrators/ — critical_sync_maintenance.sh calls partnership_manager.sh --check every 30 minutes, passing --remote-seen or --remote-unseen based on whether the rsync to the partner succeeded. The rsync outcome is the connectivity signal — no separate ping needed.

Rsync/ — Critical-Data rsync keeps the auth stack appdata current on the mirror (NPM rules, Authelia config, LLDAP database, certs). Partnership manages the relationship; Rsync delivers the actual data. On offboard, rsync_stop.sh --rsync-only stops any running rsync before the final sync runs.

Media/ — arr_sync.sh bootstraps the mirror's arr library during onboard, ensuring both servers have each other's full library from day one. Ongoing arr sync runs independently at 4-hour cadence.

---

━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Script	Role	When It Runs
partnership_onboard.sh	One-time setup — SSH keys, stack deploy, arr bootstrap	Manually, once per server per partnership
partnership_offboard.sh	Clean separation — both paths, both roles	Via partnership_manager.sh --offboard; or directly
partnership_manager.sh	Dispatcher + monitor — onboard WebUIs, health check, transfer, status	--check every 30min; all other modes manually
ssh_setup.sh	SSH key generation, remote install, auth validation	Called by onboard; manually for re-keying or validation

---

━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

INITIAL SETUP (run once)
─────────────────────────────────────────────────────────────────────────────
HOST2 (mirror) runs:
  partnership_onboard.sh
    └─ ssh_setup.sh                   generates keypair, copies to owner

HOST1 (owner) runs:
  partnership_onboard.sh
    ├─ ssh_setup.sh                   generates keypair, copies to mirror
    ├─ [stop mirror auth stack]       PARTNERSHIP_REPLACE_CONTAINERS via SSH
    ├─ deploy_container_from_xml()    pushes auth XMLs to mirror + starts containers
    │   └─ wait_for_container_healthy()  Mariadb/Redis health-checked before Authelia
    ├─ [stop mirror arr stack]        PARTNERSHIP_ARR_REPLACE_CONTAINERS via SSH
    ├─ deploy_container_from_xml()    pushes arr XMLs to mirror + starts containers
    ├─ partnership_manager.sh --onboard  reconfigures WebUIs, writes ACTIVE state
    └─ arr_sync.sh                    bootstraps full library on both servers

ONGOING OPERATION (every 30min)
─────────────────────────────────────────────────────────────────────────────
critical_sync_maintenance.sh
  ├─ Critical-Data rsync              keeps auth appdata current on mirror
  └─ partnership_manager.sh --check   reads state files, tracks offline counter
      ├─ --remote-seen path           rsync succeeded → reset counter
      └─ --remote-unseen path         rsync failed → increment counter → auto-offboard at threshold

OFFBOARD (manual or auto)
─────────────────────────────────────────────────────────────────────────────
partnership_manager.sh --offboard
  └─ partnership_offboard.sh (exec'd — holds own lock)
      ├─ [owner-initiated]    stop rsync → final sync → reconfigure mirror WebUIs
      │                       → disable critical sync → write INACTIVE state
      │                       → local fallback cleanup → restart own stack
      │                       → remote: remove auth/arr stack + fallback containers + appdata
      │                       → restart mirror stack → Emby revoke → SSH key revocation
      │                       → Tailscale removal after grace window
      └─ [mirror-initiated]   stop rsync → reconfigure own WebUIs
                              → remove owner-deployed containers locally (reads owner's stack arrays)
                              → remove local fallback containers → disable critical sync
                              → revoke own Emby admin → restart own stack
                              → SSH key revocation → write INACTIVE → signal owner