Varaverk/Partnership/README-Partnership.md

Partnership

Manages the relationship lifecycle between two unRAID servers. Handles onboarding a new mirror, clean separation when someone wants to leave, and transferring ownership when the arrangement changes.

This script exists because the partnership between two servers has a lifecycle — and that lifecycle deserves the same engineering discipline as everything else in the ecosystem. A clean exit should be as easy as a clean setup.

---

The Relationship Model

Two servers. One owns the shared services. One mirrors them.

HOST1 (owner)       — source of truth
  Auth stack        ← all changes made here
  NPM proxy rules   ← created here, mirrored to mirror
  Certs             ← renewed here, mirrored to mirror
  Emby              ← runs here, dirty-synced every 15min
  Master.conf       ← source of truth, git push propagates

HOST2 (mirror)      — warm copy
  Auth containers   ← running, serving his domain
  NPM               ← running owner's config
  Certs             ← current, mirrored every 15min
  WebUIs            ← click any auth container → opens owner's WebUI via Tailscale
  Never makes changes directly ← overwritten on next sync

HOST1 is always the owner. HOST2 is always the mirror. PARTNERSHIP_OWNER_HOST="HOST1" in Master.conf — flips to "HOST2" only after --transfer. Everything derives from HOST1/HOST2 — no duplicate IP or key configuration needed.

The mirror never needs to think about auth management. His Docker UI opens, he clicks an auth container, lands on the owner's WebUI automatically via Tailscale. Changes made there sync to his server in 15 minutes. He benefits from every improvement the owner makes without doing anything.

The owner never needs to think about the mirror's server. Everything propagates automatically. Critical-Data syncs every 15 minutes via critical_sync_maintenance.sh. Certs stay current. Config stays consistent.

---

What Makes It Work

Three things make the mirror transparent to end users:

1. Auth stack warm on both servers

NPM, LLDAP, Authelia, certs — all running on both servers simultaneously. When traffic hits the mirror's domain, auth is already ready. No cold start, no broken auth window.

2. WebUIs pointing to owner

Every auth container on the mirror has its WebUI URL configured to point at the owner's Tailscale IP. From the mirror operator's perspective: click container, get owner's UI. From the owner's perspective: one place to manage everything.

3. Tailscale-only communication

All traffic between servers goes through Tailscale. Encrypted, no open ports, no VPN configuration. The WebUI redirect works because Tailscale keeps both servers permanently connected. HOST1 and HOST2 hostnames must match their exact Tailscale device names — already enforced by the ecosystem.

---

State Files

Partnership state is tracked in two files on /boot/config — survives reboots, available before the array starts, minimal flash wear.

/boot/config/partnership_HOST1.db   ← HOST1 writes only
/boot/config/partnership_HOST2.db   ← HOST2 writes only

Each server writes only its own file. State propagates via SSH — no rsync needed. critical_sync_maintenance.sh calls partnership_manage.sh --check every 15 minutes, which SSHes to read the remote file and act on any changes.

This is how deferred offboard works: HOST2 offboards while HOST1 is temporarily unreachable → HOST2 writes its state file → next time HOST1 can reach HOST2 → reads HOST2's state → finalises from its side automatically.

A per-server offline counter (/boot/config/partnership_offline_days.db) tracks consecutive missed 15-minute cycles. After PARTNERSHIP_OFFLINE_THRESHOLD days either server is unreachable, both independently auto-offboard.

---

The Lifecycle

Onboard — Setting Up a New Mirror

Run --onboard from HOST1 (owner) to establish the relationship:

# Always dry-run first
partnership_manage.sh --onboard --dry-run

# Live onboard
partnership_manage.sh --onboard

What happens:

1. Pre-flight — both servers healthy, Tailscale connected
2. SSH to HOST2 — reconfigure each auth container's WebUI URL to HOST1's Tailscale IP
3. Verify WebUI connectivity — curl each WebUI, confirm reachable
4. Write state files — partnership_HOST1.db=ACTIVE, push to HOST2
5. Reset offline counter
6. Notify both servers — partnership active

After onboard:

Mirror opens Docker UI
→ clicks NginxProxyManager
→ lands on owner's NPM WebUI
→ makes changes there
→ 15min later synced to his server
→ he never touched his own NPM

---

Offboard — Clean Separation

Either server can initiate offboard. The process differs slightly depending on who runs it.

# Always dry-run first
partnership_manage.sh --offboard --dry-run

# Live offboard
partnership_manage.sh --offboard

HOST1 (owner) initiates offboard:

1. 10 second countdown
2. Stop any running rsync (rsync_stop.sh --rsync-only)
3. Final sync — Critical-Data and Emby pushed one last time
4. SSH to HOST2 — reconfigure auth WebUIs back to localhost
5. Disable CRITICAL_RSYNC_ENABLED=false in Master.conf
6. Write partnership_HOST1.db=INACTIVE, push to HOST2
7. Wait PARTNERSHIP_GRACE_HOURS (6hr) — mirror has access to collect anything
8. Remove HOST2 from Tailscale tailnet
9. Notify HOST2

HOST2 (mirror) initiates offboard:

1. Reconfigure own auth WebUIs → localhost
2. Write partnership_HOST2.db=INACTIVE
3. SSH to HOST1 — push state file (or write locally if HOST1 unreachable)
4. Notify HOST1 — "mirror has requested offboard"

HOST1 finalises on next --check cycle:

5. Reads HOST2's state file — sees INACTIVE
6. Runs final sync — HOST2 leaves with current state
7. Disables critical sync
8. Writes partnership_HOST1.db=INACTIVE
9. Waits grace period → removes HOST2 from Tailscale

What HOST2 leaves with:

Current auth config        ✓  — final sync was clean
His own DDNS + updater     ✓  — always was his, unaffected
His own Git mirror         ✓  — full ecosystem, always mirrored
His anime source of truth  ✓  — always was his
Auth WebUIs → localhost    ✓  — managing his own auth now
Full independence           ✓  — just stop the sync
Certs valid ~60-90 days    ✓  — cert_monitor.sh warns at 30 days

Tailscale access and backup window both expire at the same time (PARTNERSHIP_GRACE_HOURS=6). Keeping backups accessible beyond Tailscale removal is meaningless — they expire together by design.

---

Auto-Offboard — 30 Day Offline Threshold

If either server is unreachable for PARTNERSHIP_OFFLINE_THRESHOLD days (default: 30), the other server auto-offboards. Works both directions:

HOST1 sees HOST2 offline 30 days:
  AM_OWNER → full offboard from owner side
  Tailscale removal ✅ (API call works regardless)
  
HOST2 sees HOST1 offline 30 days:
  AM_MIRROR → mirror offboard
  Reconfigures own WebUIs → localhost ✅
  Writes HOST2.db=INACTIVE ✅
  Fully independent immediately ✅
  When HOST1 comes back → sees INACTIVE → cleans its side ✅

This handles the "partner disappeared" scenario without manual intervention. 30 days is long enough to cover extended outages, short enough that you're not waiting forever for resolution.

---

Transfer — Flipping Ownership

Run --transfer from HOST1 when ownership needs to change:

# Always dry-run first — this is significant
partnership_manage.sh --transfer --dry-run

# Live transfer — requires explicit confirmation string
partnership_manage.sh --transfer --confirm=i-understand-this-transfers-ownership

The confirmation string is long and specific by design. You cannot type it accidentally.

What happens:

1. Display current and future ownership clearly
2. Confirmation string check
3. Health strike system — both servers must pass PARTNERSHIP_TRANSFER_STRIKES consecutive checks (max PARTNERSHIP_TRANSFER_MAX_ATTEMPTS attempts before giving up)
4. Final sync in current direction — new mirror leaves with current state
5. Reconfigure HOST2 WebUIs → new owner's Tailscale IP
6. Reconfigure HOST1 WebUIs → localhost (now manages directly)
7. Flip PARTNERSHIP_OWNER_HOST in Master.conf on both servers
8. Write updated state files
9. Notify both servers

After transfer:

Before:  HOST1 = owner, HOST2 = mirror
After:   HOST2 = owner, HOST1 = mirror

HOST2 now:
  Makes all auth changes      ← source of truth
  Pushes Critical-Data sync   ← every 15min
  Manages NPM, certs, LLDAP   ← directly

HOST1 now:
  Receives sync               ← warm mirror
  WebUIs point to HOST2       ← transparent
  Never makes changes directly

---

Check — Automated State Monitor

Called automatically by critical_sync_maintenance.sh every 15 minutes. Never run manually.

# Called internally — do not run manually
partnership_manage.sh --check --remote-seen    # rsync succeeded, HOST2 reachable
partnership_manage.sh --check --remote-unseen  # rsync failed, HOST2 unreachable

What it does each cycle:

• --remote-seen: resets offline counter, updates last_seen_remote timestamp
• --remote-unseen: increments offline counter, checks auto-offboard threshold
• SSHes to remote, reads remote state file
• Both ACTIVE → silent, healthy ✅
• Remote INACTIVE → owner finalises offboard, mirror cleans up
• Threshold exceeded → auto-offboard

---

Role-Based Access

HOST1 (owner):    --onboard ✅  --offboard ✅  --transfer ✅  --status ✅
HOST2 (mirror):   --onboard ❌  --offboard ✅  --transfer ❌  --status ✅

HOST2 is blocked from onboard and transfer by design — ownership is granted not taken. Either server can initiate offboard — clean exit is always available to both parties.

---

Configuration

All configuration in Master.conf under ── PARTNERSHIP ──, placed immediately after HOST CONFIGURATION.

PARTNERSHIP_ENABLED=false
PARTNERSHIP_OWNER_HOST="HOST1"          # "HOST1" or "HOST2" — flips on --transfer

# Auth containers reconfigured on onboard/offboard
# Format: "ContainerName|WebUIPort"
PARTNERSHIP_AUTH_WEBUIS=(
    "NginxProxyManager|81"
    "Lldap-Gmer4Lfe|17170"
    "Authelia|9091"
    "Authelia-Secondary|9092"
)

# Paths on owner that mirror should collect during grace window
PARTNERSHIP_MIRROR_BACKUPS=(
    # "/mnt/user/appdata-Failover/Jayred365-Emby"
)

# Timing
PARTNERSHIP_GRACE_HOURS=6               # hours after offboard before Tailscale removal
                                        # backup access also expires at this time
PARTNERSHIP_OFFLINE_THRESHOLD=30        # days unreachable before auto-offboard (both directions)
PARTNERSHIP_REMOVE_TAILSCALE=true       # remove mirror from Tailscale on offboard

# Tailscale API — required for PARTNERSHIP_REMOVE_TAILSCALE=true
TAILSCALE_API_KEY=""                    # tskey-api-...
TAILSCALE_TAILNET=""                    # your tailnet name

# Transfer safety
PARTNERSHIP_TRANSFER_CONFIRM="i-understand-this-transfers-ownership"
PARTNERSHIP_TRANSFER_STRIKES=3
PARTNERSHIP_TRANSFER_MAX_ATTEMPTS=20

# Onboard
PARTNERSHIP_ONBOARD_VERIFY=true
PARTNERSHIP_ONBOARD_NOTIFY=true
PARTNERSHIP_SYNC_INTERVAL=15            # minutes — informational, actual schedule in cron

---

critical_sync_maintenance.sh

The partnership check runs inside critical_sync_maintenance.sh — the 15-minute orchestrator that also handles Critical-Data and Emby failover syncs.

Orchestrators/critical_sync_maintenance.sh
Schedule: */15 * * * *

Execution order:
  1. Critical-Data rsync    ← auth stack sync (15min)
  2. emby-failover rsync    ← dirty Emby sync (15min)
  3. partnership --check    ← state check, informed by rsync outcome

The rsync outcome directly informs the partnership check — if rsync succeeded, HOST2 was reachable and the offline counter resets. If rsync failed, the counter increments.

Controlled by CRITICAL_RSYNC_ENABLED in Master.conf — set to false automatically on offboard.

---

Initial Setup Requirements

Before --onboard can run:

1. Tailscale connected on both servers

HOST1 and HOST2 hostnames must match their exact Tailscale device names — already enforced by the ecosystem since HOST1 and HOST2 in Master.conf are used for Tailscale IP resolution everywhere.

2. SSH keys configured — passwordless both directions

ssh -i /root/.ssh/your_key root@[HOST2-tailscale-ip] "hostname"

3. Auth containers exist on HOST2

Containers must exist before onboard. They can be stopped — onboard only reconfigures WebUI URLs. Critical-Data sync brings them live config.

4. Critical-Data rsync profile configured

The critical-data rsync profile must be set up in Master.conf. This is what keeps the mirror current after onboard. Configured in CRITICAL_SYNC_SHARES.

5. Tailscale API key configured (if PARTNERSHIP_REMOVE_TAILSCALE=true)

https://login.tailscale.com/admin/settings/keys
Scope: Devices write

---

Independence — Always One Stop Away

The partnership is designed with a clean exit built in from day one.

To stop the relationship — from either server:

partnership_manage.sh --offboard

That's it. The mirror's containers keep running. His domain keeps working. His DDNS keeps pointing to him. His git repo has the full ecosystem. His certs run for another 60-90 days.

What the owner keeps after separation:

His hardware           ← always was his
His media shares       ← his source of truth shares stay his
His services           ← unaffected
His domain             ← unaffected
His Emby users         ← continue as before

What the owner loses:

Offsite backup         ← was the mirror's hardware
Geographic redundancy  ← can't script that, it was the person

The arrangement was always mutual. Both parties contributed something the scripts couldn't provide — hardware in a different location on a different power utility. That's irreplaceable. Everything else in the ecosystem can be reconfigured.

---

Troubleshooting

WebUI still pointing to old IP after onboard:

• Check template was found: partnership_manage.sh --status
• Some containers may need a restart to pick up the new WebUI URL
• Verify with --status on both servers

Transfer health checks failing:

• Both arrays must be fully started
• Both Docker daemons must be responding
• Tailscale must be connected on both servers
• Max attempts: PARTNERSHIP_TRANSFER_MAX_ATTEMPTS (default 20) before giving up

Mirror's domain broken after offboard:

• Auth WebUIs reconfigured to localhost — mirror manages auth directly now
• Verify auth containers are running: docker ps
• Check cert expiry: run cert_monitor.sh

State mismatch between servers:

• Run --status on both servers to see both state files
• If one shows ACTIVE and other INACTIVE — run --offboard to resolve
• If HOST2 unreachable — HOST2 will self-resolve on next --check when reachable

Auto-offboard triggered unexpectedly:

• Check /boot/config/partnership_offline_days.db for counter value
• Extended outage or Tailscale connectivity issue may have incremented counter
• Re-onboard if partnership should continue: --onboard

---

Design Notes

Why HOST1/HOST2 instead of PARTNERSHIP_OWNER/MIRROR?

HOST1 and HOST2 are already defined in Master.conf with SSH keys, Tailscale names, and all connection details. Duplicating them as PARTNERSHIP_OWNER and PARTNERSHIP_MIRROR would require maintaining the same values twice. A single PARTNERSHIP_OWNER_HOST var flips ownership direction — everything else derives from the existing HOST1/HOST2 configuration.

Why does transfer require a confirmation string?

Transferring ownership touches Master.conf on both servers, reconfigures WebUIs, and flips sync direction. The confirmation string is the same philosophy as --i-know-what-im-doing in the arr cleanup scripts — make accidental execution impossible, not just unlikely.

Why do Tailscale removal and backup access expire at the same time?

Keeping backups accessible beyond Tailscale removal is meaningless — if the mirror can't reach the owner's server via Tailscale, he can't access the backups anyway. PARTNERSHIP_GRACE_HOURS controls both simultaneously. One var, consistent behavior, no misleading "30 days available" when access is gone in 6 hours.

Why does either server auto-offboard after 30 days offline?

30 consecutive days of missed sync cycles means the relationship has effectively ended regardless of intent. The auto-offboard makes the state official and cleans up both sides without requiring manual intervention from a server that may genuinely be gone. Each server acts independently — they don't need to coordinate to offboard.

Why does WebUI reconfiguration use the unRAID template system?

unRAID stores container WebUI URLs in /boot/config/plugins/dockerMan/templates-user/. Editing templates is the correct way to change WebUI URLs — it persists across container restarts and array reboots. Direct Docker label manipulation would be lost on the next container recreation.

Why is Tailscale WebUI left on the mirror's local server?

Tailscale WebUI shows that server's network state from its own perspective. When diagnosing connectivity issues between the servers, you need to see the mirror's network view — not the owner's. It's the one WebUI that genuinely needs to stay local.