# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🤝  PARTNERSHIP
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Managed lifecycle for a two-server partnership — setup, ongoing operation,
and clean separation.** One server owns the shared services. The other mirrors
them and benefits from them. Every phase of the relationship has the same
engineering discipline as the rest of the ecosystem.

> **This folder exists because a clean exit should be as easy as a clean setup.**
> The partnership is not a permanent commitment. `--offboard` works from either
> server at any time. Everything the mirror needs to run independently is already
> there. The only thing that stops on separation is the sync — and that's intentional.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

---

### 🔴  Two Servers, One Auth Stack, No Clean Way to Share It

The auth stack — NginxProxyManager, LLDAP, Authelia, MariaDB, Redis — runs on
HOST1. HOST2 serves its own domain to its own household. It needs its own auth.
But maintaining two independent auth stacks means double the work: two places to
add users, two places to update proxy rules, two places to renew certs, two
configurations that inevitably drift apart. One change on HOST1 has to be manually
replicated to HOST2 — or it isn't, and the configurations diverge silently.

The real cost isn't the initial setup. It's the maintenance burden that accumulates
over months — every new user, every proxy rule change, every config update applied
in one place and forgotten in the other.

**The fix:** one auth stack with a managed mirror. HOST1 owns the configuration.
HOST2 runs a warm copy that stays current via 30-minute sync. HOST2's operator makes
zero auth management decisions — clicking an auth container opens HOST1's WebUI via
Tailscale. Changes happen there, propagate to HOST2 in 30 minutes. One place to
manage everything for both households.

---

### 🔴  No Structure Around the Relationship Itself

Setting up the mirror was a manual process. SSH in, reconfigure container WebUI
URLs one by one, copy auth config, verify connectivity, update state tracking. No
defined sequence. No dry-run capability. No verification that each step worked. If
something went wrong midway, the mirror was in an inconsistent state with no clear
way to understand what had and hadn't been done. Offboard was worse — it involves
stopping a sync that's been running for months, making a final copy of data,
reconfiguring WebUIs back to local addresses, removing Tailscale access, and
notifying both servers. A manual process with that many steps, taken under pressure,
leaves one or both parties in a bad state.

**The fix:** `partnership_manager.sh` with explicit modes for each lifecycle phase.
Each mode is a defined sequence. Every step is verified. Dry-run shows exactly what
will happen before anything changes. State files make the current relationship status
unambiguous from either server.

---

### 🔴  No Safe Way to Check If the Other Server Has Gone Away

After months of operation, HOST2 goes quiet. The sync starts failing. The offline
counter increments. But nothing actually happens — the ecosystem just keeps failing
the same sync, incrementing the same counter, sending the same notifications.
Without a defined threshold and an automated response, "partner gone for 30 days"
looks exactly like "partner gone for 3 years."

**The fix:** `PARTNERSHIP_OFFLINE_THRESHOLD`. After this many days of missed sync
cycles, both servers independently auto-offboard. HOST1 removes HOST2 from Tailscale,
disables critical sync, writes INACTIVE state. HOST2 — if it eventually comes back —
reads HOST1's INACTIVE state and cleans up its own side. The relationship is formally
ended from both sides without anyone needing to be present.

---

### 🔴  Ownership Transfer Had No Safe Path

The arrangement was always intended to be flexible — HOST1 owns the auth stack now,
but circumstances change. Swapping ownership manually meant reconfiguring WebUIs on
both servers, swapping sync direction, updating master.conf on both, and hoping the
sequence was correct. A misstep — like flipping sync direction before the final sync
completed — leaves both servers with different auth configurations and no clear source
of truth.

**The fix:** `--transfer` with a required confirmation string, a consecutive health
check system, and a strict sequence. The confirmation string cannot be typed
accidentally. Health strikes require both servers to be healthy on multiple
consecutive checks before the transfer begins. A final sync in the current direction
runs before anything is flipped.

---

## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Onboarding** (`partnership_onboard.sh`) — one-time setup run from both servers.
Generates SSH keys, installs the auth stack and arr stack on the mirror from XML
templates, configures WebUI redirects to the owner, and bootstraps the arr library.
Role is detected automatically — no flags needed to declare which side you are.

**Offboarding** (`partnership_offboard.sh`) — handles clean separation from either
role. Owner path: final sync, WebUI reconfigure, remote container + appdata cleanup
(auth/arr stack by XML array, fallback coverage by naming), SSH key revocation,
Tailscale removal. Mirror path: local WebUI reconfigure, remove owner-deployed
containers locally, disable sync, revoke Emby admin, SSH key revocation, signal owner.
Called by `partnership_manager.sh --offboard` but runnable directly.

**Lifecycle management** (`partnership_manager.sh`) — dispatcher and monitor.
- Manually: `--onboard`, `--offboard` (delegates to offboard script), `--transfer`, `--status`, `--unblock`
- Automatically: `--check` called every 30 minutes by `critical_sync_maintenance.sh`

**SSH management** (`ssh_setup.sh`) — generates the keypair for rsync automation,
installs it on the remote, and tracks auth failures with a configurable strike system.
Called by `partnership_onboard.sh` but runnable independently for validation and
re-keying.

---

## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Orchestrators/** — `critical_sync_maintenance.sh` calls `partnership_manager.sh --check`
every 30 minutes, passing `--remote-seen` or `--remote-unseen` based on whether the
rsync to the partner succeeded. The rsync outcome is the connectivity signal — no
separate ping needed.

**Rsync/** — Critical-Data rsync keeps the auth stack appdata current on the mirror
(NPM rules, Authelia config, LLDAP database, certs). Partnership manages the
relationship; Rsync delivers the actual data. On offboard, `rsync_stop.sh --rsync-only`
stops any running rsync before the final sync runs.

**Media/** — `arr_sync.sh` bootstraps the mirror's arr library during onboard, ensuring
both servers have each other's full library from day one. Ongoing arr sync runs
independently at 4-hour cadence.

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

| Script | Role | When It Runs |
|--------|------|--------------|
| `partnership_onboard.sh` | One-time setup — SSH keys, stack deploy, arr bootstrap | Manually, once per server per partnership |
| `partnership_offboard.sh` | Clean separation — both paths, both roles | Via `partnership_manager.sh --offboard`; or directly |
| `partnership_manager.sh` | Dispatcher + monitor — onboard WebUIs, health check, transfer, status | `--check` every 30min; all other modes manually |
| `partnership_transfer.sh` | Transfer ownership from current owner to current mirror | Via `partnership_manager.sh --transfer`; owner only |
| `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
| `onboard_cancel.sh` | Remove SSH keys in one or both directions, clear setup flags | During cancelled or failed onboard; manual cleanup |
| `gitea_ssh_setup.sh` | Generate a Gitea keypair and register it via the Gitea API | Called during onboard; manually when re-keying a server |
| `share_setup.sh` | Create missing Unraid shares on the mirror from the owner's sync lists | Called during onboard; safe to re-run — existing shares are never modified |

---

## ━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
INITIAL SETUP (run once)
─────────────────────────────────────────────────────────────────────────────
HOST2 (mirror) runs:
  partnership_onboard.sh
    └─ ssh_setup.sh                   generates keypair, copies to owner

HOST1 (owner) runs:
  partnership_onboard.sh
    ├─ ssh_setup.sh                   generates keypair, copies to mirror
    ├─ [stop mirror auth stack]       PARTNERSHIP_REPLACE_CONTAINERS via SSH
    ├─ deploy_container_from_xml()    pushes auth XMLs to mirror + starts containers
    │   └─ wait_for_container_healthy()  Mariadb/Redis health-checked before Authelia
    ├─ [stop mirror arr stack]        PARTNERSHIP_ARR_REPLACE_CONTAINERS via SSH
    ├─ deploy_container_from_xml()    pushes arr XMLs to mirror + starts containers
    ├─ partnership_manager.sh --onboard  reconfigures WebUIs, writes ACTIVE state
    └─ arr_sync.sh                    bootstraps full library on both servers

ONGOING OPERATION (every 30min)
─────────────────────────────────────────────────────────────────────────────
critical_sync_maintenance.sh
  ├─ Critical-Data rsync              keeps auth appdata current on mirror
  └─ partnership_manager.sh --check   reads state files, tracks offline counter
      ├─ --remote-seen path           rsync succeeded → reset counter
      └─ --remote-unseen path         rsync failed → increment counter → auto-offboard at threshold

OFFBOARD (manual or auto)
─────────────────────────────────────────────────────────────────────────────
partnership_manager.sh --offboard
  └─ partnership_offboard.sh (exec'd — holds own lock)
      ├─ [owner-initiated]    stop rsync → final sync → reconfigure mirror WebUIs
      │                       → disable critical sync → write INACTIVE state
      │                       → local fallback cleanup → restart own stack
      │                       → remote: remove auth/arr stack + fallback containers + appdata
      │                       → restart mirror stack → Emby revoke → SSH key revocation
      │                       → Tailscale removal after grace window
      └─ [mirror-initiated]   stop rsync → reconfigure own WebUIs
                              → remove owner-deployed containers locally (reads owner's stack arrays)
                              → remove local fallback containers → disable critical sync
                              → revoke own Emby admin → restart own stack
                              → SSH key revocation → write INACTIVE → signal owner
```
