Varaverk/Partnership/Manual-Partnership.md

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🤝 PARTNERSHIP — Manual

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config reference, setup procedures, and operational how-tos. For architecture see README-Partnership.md. For per-script detail see script headers.

---

━━━ THE RELATIONSHIP MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HOST1 (owner)              HOST2 (mirror)
────────────────           ────────────────────────────
Source of truth            Warm copy — always current
Auth stack config          Auth containers running
NPM proxy rules            NPM serving mirror's domain
LLDAP users                LLDAP — same users
Authelia policies          Authelia — same policies
Certs                      Certs — mirrored, valid
Emby                       Emby — dirty-synced every 15min

Changes made here    ──→   Propagated every 15 minutes
WebUI management     ──→   Redirected to HOST1 via Tailscale
Config → git push    ──→   Received via git pull on next cycle

Mirror's daily experience: open unRAID Docker UI, click NginxProxyManager, browser opens HOST1's NPM WebUI automatically via Tailscale. Make a proxy rule change. 15 minutes later it's live on HOST2. The mirror operator never manages auth directly — every WebUI redirect is transparent and automatic.

Owner's daily experience: manage auth as normal. Opens NPM → adds a proxy rule → 15min later live on both servers. Never needs to SSH to HOST2 or think about HOST2 during normal operation.

---

━━━ SETUP PREREQUISITES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Before partnership_onboard.sh can run, five things must be in place:

1. Tailscale Connected on Both Servers

# Verify HOST2 is visible from HOST1:
tailscale ip -4 unRAID-Jayred365    # returns HOST2's Tailscale IP

# Verify reachability:
tailscale ping unRAID-Jayred365

Hostnames must match their exact Tailscale device names — enforced already via REMOTE_SERVER_NAME in master.conf.

2. Auth Containers Exist on HOST2

# Containers must exist — they can be stopped:
ssh root@[HOST2-ip] "docker inspect NginxProxyManager --format '{{.State.Status}}'"
# Expected: created, exited, or running — NOT "no such container"

partnership_onboard.sh will stop these before deploying the owner's auth stack. List them in HOST2_PARTNERSHIP_REPLACE_CONTAINERS.

3. XML Templates Exist on HOST1

ls /boot/config/plugins/dockerMan/templates-user/my-NginxProxyManager.xml
ls /boot/config/plugins/dockerMan/templates-user/my-Authelia.xml
# etc — one XML per container listed in HOST1_PARTNERSHIP_AUTH_STACK

These are copied to HOST2 during onboard. The same XMLs that Unraid's Docker Manager uses — no extra configuration needed on HOST2 after deploy.

4. Critical-Data rsync Profile Configured

# master.conf — must include the auth stack appdata path:
CRITICAL_SYNC_SHARES=(
    "/mnt/user/appdata-Fallback/Critical-Data"    # auth config, certs, NPM rules
)

This is what keeps HOST2 current after onboard. Without it, onboard succeeds but the mirror's auth stack drifts from the owner's within hours.

5. Tailscale API Key (if PARTNERSHIP_REMOVE_TAILSCALE=true)

# master.conf
PARTNERSHIP_REMOVE_TAILSCALE=true
TAILSCALE_API_KEY="tskey-api-..."   # from tailscale.com/admin/settings/keys
TAILSCALE_TAILNET="example.github"  # your tailnet name

# Required scope: Devices write

---

━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

master.conf

# ── Partnership Gate ───────────────────────────────────────────────────────
PARTNERSHIP_ENABLED=false           # set true once both servers are configured
PARTNERSHIP_OWNER_HOST="HOST1"      # "HOST1" or "HOST2" — flips on --transfer

# ── Tailscale Removal ──────────────────────────────────────────────────────
PARTNERSHIP_REMOVE_TAILSCALE=true
TAILSCALE_API_KEY=""                # tskey-api-... from tailscale.com/admin/settings/keys
TAILSCALE_TAILNET=""                # your tailnet name (e.g. example.github)

# ── Timing ─────────────────────────────────────────────────────────────────
PARTNERSHIP_GRACE_HOURS=6           # hours before Tailscale removal after offboard
                                    # backup access also expires at this time
PARTNERSHIP_OFFLINE_THRESHOLD=30    # days unreachable before auto-offboard
PARTNERSHIP_SYNC_INTERVAL=15        # informational — actual schedule in cron

# ── Transfer Safety ────────────────────────────────────────────────────────
PARTNERSHIP_TRANSFER_CONFIRM="i-understand-this-transfers-ownership"
PARTNERSHIP_TRANSFER_STRIKES=3      # consecutive health checks required
PARTNERSHIP_TRANSFER_MAX_ATTEMPTS=20

# ── Onboard Behaviour ──────────────────────────────────────────────────────
PARTNERSHIP_ONBOARD_VERIFY=true     # curl-verify each WebUI after onboard
PARTNERSHIP_ONBOARD_NOTIFY=true     # notify both servers on successful onboard

master_host1.conf (owner side)

# Containers whose WebUI URLs are redirected to owner's Tailscale IP on onboard.
# Restored to localhost on offboard.
# Format: "ContainerName|WebUIPort"
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
    "NginxProxyManager|81"
    "Lldap-Gmer4Lfe|17170"
    "Authelia|9091"
    "Authelia-Secondary|9092"
)

# XML templates pushed to mirror during onboard (auth stack).
# ORDER MATTERS: database dependencies must come before Authelia.
# Mariadb/Redis are health-checked after deploy before continuing.
HOST1_PARTNERSHIP_AUTH_STACK=(
    "my-Mariadb-Authelia.xml"
    "my-Mariadb-Authelia-Secondary.xml"
    "my-Redis-Authelia.xml"
    "my-Redis-Authelia-Secondary.xml"
    "my-Authelia.xml"
    "my-Authelia-Secondary.xml"
    "my-NginxProxyManager.xml"
    "my-Lldap-Gmer4Lfe.xml"
)

# XML templates pushed to mirror during onboard (arr stack).
# Leave empty to skip arr stack deploy entirely.
HOST1_PARTNERSHIP_ARR_STACK=(
    # "my-Sonarr.xml"
    # "my-Radarr.xml"
    # "my-Lidarr.xml"
    # "my-Prowlarr.xml"
)

# Paths mirror can collect during grace window after offboard.
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
    # "/mnt/user/appdata-Fallback/Jayred365-Emby"
)

# Containers parked on this server when partnership is active.
# Stopped on onboard (owner deploys auth stack instead), restarted on offboard.
HOST1_PARTNERSHIP_OWN_CONTAINERS=(
    # "Emby"
    # "NginxProxyManager"
)

master_host2.conf (mirror side)

# Containers whose WebUI URLs are redirected on onboard.
# Usually left empty on mirror — owner's AUTH_WEBUIS drives the redirect.
HOST2_PARTNERSHIP_AUTH_WEBUIS=(
    # "NginxProxyManager|81"
)

# Auth containers to stop on this server before owner deploys auth stack.
HOST2_PARTNERSHIP_REPLACE_CONTAINERS=(
    "NginxProxyManager"
    "Authelia"
    "Authelia-Secondary"
    "Mariadb-Authelia"
    "Mariadb-Authelia-Secondary"
    "Redis-Authelia"
    "Redis-Authelia-Secondary"
    "Lldap"
)

# Arr containers to stop on this server before owner deploys arr stack.
HOST2_PARTNERSHIP_ARR_REPLACE_CONTAINERS=(
    # "Sonarr"
    # "Radarr"
    # "Lidarr"
    # "Prowlarr"
)

# Containers parked on this server when partnership is active.
# Stopped on onboard, restarted on offboard.
HOST2_PARTNERSHIP_OWN_CONTAINERS=(
    # "Emby"
    # "NginxProxyManager"
)

# Emby admin provisioning — owner reads these during onboard.
HOST2_PARTNERSHIP_EMBY_ADMIN_USER=""
HOST2_PARTNERSHIP_EMBY_ADMIN_PASS=""

---

━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Establishing a Partnership (First-Time Setup)

# Step 1: Run on HOST2 (mirror) first — generates SSH key only
Partnership/partnership_onboard.sh

# Step 2: Run on HOST1 (owner) — completes setup remotely
Partnership/partnership_onboard.sh --dry-run    # review first
Partnership/partnership_onboard.sh

# Step 3: Verify from either server
Partnership/partnership_manager.sh --status

HOST2 must run first to generate its SSH key so HOST1 can reach it during Step 2. If SSH is already configured, use --skip-ssh on both sides.

Skip flags for partial re-runs (if something failed midway):

--skip-ssh          # SSH already set up
--skip-auth-stack   # auth stack already deployed (skips Steps 3-4)
--skip-arr-stack    # arr stack already deployed (skips Steps 5-6)
--skip-arr-sync     # arr sync not needed yet (arrs not live on mirror)

---

Offboarding

Either server can initiate. Always dry-run first.

# From either server — review the sequence
Partnership/partnership_manager.sh --offboard --dry-run

# Live offboard — 10-second countdown before executing
Partnership/partnership_manager.sh --offboard

Owner-initiated (HOST1 runs --offboard):

1. Stop any running rsync
2. Final sync — mirror leaves with current auth config and certs
3. Reconfigure mirror WebUIs → localhost
4. Disable critical rsync (CRITICAL_RSYNC_ENABLED=false)
5. Write INACTIVE state on both servers, blocklist mirror
6. Local fallback cleanup — remove partner containers + appdata from HOST1
7. Restart own stack (HOST1's own parked containers)
8. Remote cleanup — remove auth/arr stack containers + appdata from mirror; remove fallback containers
9. Restart mirror's own stack
10. Revoke Emby admin, SSH key revocation (both directions), Tailscale removal

Mirror-initiated (HOST2 runs --offboard):

1. Stop any running rsync
2. Reconfigure own WebUIs → localhost (immediately independent)
3. Remove owner-deployed containers locally (reads owner's auth/arr stack arrays via SSH)
4. Remove local fallback coverage containers
5. Disable critical rsync (CRITICAL_RSYNC_ENABLED=false)
6. Revoke own Emby admin account from local Emby instance
7. Restart own stack (HOST2's own parked containers)
8. SSH key revocation, write INACTIVE state locally, push state to HOST1 if reachable

HOST1 finalises its own side on the next --check cycle after seeing HOST2's INACTIVE state.

What the mirror leaves with after offboard:

• Its own parked containers restarted (from PARTNERSHIP_OWN_CONTAINERS)
• Own arr media library (arr appdata is in appdata-Fallback/Arrs_Stack/ and is retained)
• Full git mirror of the ecosystem
• Auth WebUIs pointing to localhost — ready to set up own independent auth

---

Transferring Ownership

Transfers auth stack ownership from current owner to mirror. Only the current owner can initiate. Both servers must be healthy.

# Always dry-run first — shows health check results and master.conf changes
Partnership/partnership_manager.sh --transfer --dry-run

# Live transfer — requires exact confirmation string
Partnership/partnership_manager.sh --transfer --confirm=i-understand-this-transfers-ownership

Transfer sequence:

1. Display current and future ownership clearly
2. Require exact confirmation string
3. Both servers pass PARTNERSHIP_TRANSFER_STRIKES consecutive health checks
4. Final sync in current direction (current owner → current mirror)
5. Reconfigure WebUI templates on both servers
6. Flip PARTNERSHIP_OWNER_HOST in master.conf on both servers
7. Write updated state files and notify

After transfer, HOST2 is the owner. Run --transfer from HOST2 to transfer back.

---

Re-onboarding After Offboard

Former partners are blocklisted on offboard. Remove from blocklist first:

# Check what's on the blocklist
Partnership/partnership_manager.sh --status

# Remove the block (use exact hostname shown in status)
Partnership/partnership_manager.sh --unblock unRAID-Jayred365

# Then re-onboard normally
Partnership/partnership_onboard.sh --dry-run
Partnership/partnership_onboard.sh

---

Adding a New Container to the Auth Stack

1. Create the XML template on HOST1 (/boot/config/plugins/dockerMan/templates-user/my-NewContainer.xml)
2. Add the XML filename to HOST1_PARTNERSHIP_AUTH_STACK in master_host1.conf
   • If it has a database dependency, put the dep earlier in the array
3. Add the container name to HOST2_PARTNERSHIP_REPLACE_CONTAINERS in master_host2.conf
4. Re-run the auth stack portion:

   Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync
   

---

SSH Key Validation and Re-keying

# Check current key state and remote connectivity
Partnership/ssh_setup.sh --status

# Validate SSH auth (tracks strikes)
Partnership/ssh_setup.sh --validate

# Regenerate key and re-copy to remote (if key is compromised or expired)
Partnership/ssh_setup.sh --force

Strike tracking: SSH_MAX_STRIKES consecutive auth failures → notify. Network unreachability (Tailscale down) does not count as a strike. Counter resets after SSH_STRIKE_RESET_HRS of clean connectivity.

---

━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

partnership_onboard.sh

Flag	Effect
--dry-run	Preview all steps without making changes
--log	Verbose per-step output
--skip-ssh	Skip SSH key setup (Step 1) — key already in place
--skip-auth-stack	Skip auth stop + deploy (Steps 3-4) — already done
--skip-arr-stack	Skip arr stop + deploy (Steps 5-6) — not needed or already done
--skip-arr-sync	Skip arr library bootstrap (Step 8) — arrs not live yet

partnership_offboard.sh

Called automatically by partnership_manager.sh --offboard. Can also be run directly.

Flag	Effect
--dry-run	Preview all steps without making changes
--log	Verbose per-step output
--reason=<string>	Tag the offboard reason in state file and blocklist (default: manual)

partnership_manager.sh

Flag	Effect
--dry-run	Show the complete sequence without executing
--log	Verbose per-operation output
--onboard	Establish mirror relationship (owner only)
--offboard	Clean separation (either server) — 10s countdown
--transfer --confirm=...	Flip ownership (owner only)
--check --remote-seen|--remote-unseen	15-min health check (called by orchestrator)
--status	Show state files, blocklist, SSH key status
--unblock <hostname>	Remove hostname from blocklist

ssh_setup.sh

Flag	Effect
--force	Regenerate key even if it exists, re-copy to remote
--validate	Test SSH auth, track strikes, notify at limit
--status	Show key path, fingerprint, remote connectivity
--dry-run	Preview without creating or copying

---

━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Auth Stack Deploy Fails on Onboard

# Check that XML files exist on HOST1:
ls /boot/config/plugins/dockerMan/templates-user/my-Authelia.xml

# Check that mirror is reachable via Tailscale:
tailscale ping unRAID-Jayred365

# Check that SSH key is working:
Partnership/ssh_setup.sh --validate

# Re-run auth stack only:
Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync

If Mariadb/Redis deployed but Authelia still fails: the health-wait timed out (60s default). Authelia needs Mariadb to be fully initialized, which can take longer on first boot. Re-run with --skip-auth-stack removed — docker start on an already-created container will retry cleanly.

---

WebUI Still Pointing to Old IP After Onboard

# Check template was updated on HOST2:
ssh root@[HOST2-ip] "grep -i 'tailscale\|[HOST1-IP]' \
    /boot/config/plugins/dockerMan/templates-user/my-NginxProxyManager.xml"

# Check status output:
Partnership/partnership_manager.sh --status

# Some containers need a restart to pick up new WebUI URL:
ssh root@[HOST2-ip] "docker restart NginxProxyManager"

# Verify Tailscale routing:
# From HOST2, should reach HOST1's NPM:
curl http://[HOST1-tailscale-ip]:81

---

Transfer Health Checks Failing

# Both arrays must be fully started:
ls /mnt/user                           # should show share directories on both servers

# Both Docker daemons responding:
docker ps                              # should return a list, not hang

# Tailscale connected on both servers:
tailscale status                       # remote peer should show online

# Health check attempts before giving up: PARTNERSHIP_TRANSFER_MAX_ATTEMPTS (default 20)

---

Mirror's Domain Broken After Offboard

The owner-deployed auth stack has been removed from HOST2 as part of offboard cleanup. HOST2's own parked containers (from PARTNERSHIP_OWN_CONTAINERS) are restarted automatically, but if HOST2 had no pre-existing auth stack of its own, it needs one set up.

# Check what's running:
docker ps

# Verify own parked containers came back up:
# (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in master_host2.conf)

# If you need a fresh auth stack, deploy from HOST2's own XML templates:
docker create ... && docker start NginxProxyManager   # etc.

# Check cert expiry — auto-renewal no longer happens via HOST1:
cert_monitor.sh --dry-run

If the offboard was unexpected or incomplete, check state first:

Partnership/partnership_manager.sh --status

---

State Mismatch Between Servers

# Check both state files:
Partnership/partnership_manager.sh --status    # shows both sides via SSH

# If one shows ACTIVE and other INACTIVE:
#   The INACTIVE side has already offboarded.
#   Run --offboard on the ACTIVE side to sync the state.

# If HOST2 unreachable:
#   HOST2 self-resolves on next --check when reachable.
#   HOST1 reads INACTIVE state → finalises from owner side automatically.

---

Auto-Offboard Triggered Unexpectedly

# Check the offline counter:
cat /boot/config/partnership_offline_days.db

# Extended Tailscale outage may have incremented the counter.
# Check Tailscale peer visibility:
tailscale status

# If partnership should continue — re-onboard (first unblock the partner):
Partnership/partnership_manager.sh --unblock unRAID-Jayred365
Partnership/partnership_manager.sh --onboard --dry-run
Partnership/partnership_manager.sh --onboard

---

━━━ STATE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

/boot/config/partnership_HOST1.db   # HOST1 writes only
/boot/config/partnership_HOST2.db   # HOST2 writes only
/boot/config/partnership_blocklist.db  # hostname|timestamp|reason

# Example state file:
state=ACTIVE
last_updated=2026-05-14 03:00:00
last_seen_remote=2026-05-14 03:00:00
owner_host=HOST1
offline_days=0

Each server writes only its own state file. The other server reads via SSH. State propagates through SSH reads — no rsync, no shared filesystem. /boot/config — survives reboots, available before array starts, minimal flash wear. flock on all writes — prevents concurrent corruption from overlapping --check cycles.

---

━━━ ROLE-BASED ACCESS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

                    HOST1 (owner)    HOST2 (mirror)
--onboard                ✅               ❌     ownership is granted, not taken
--offboard               ✅               ✅     clean exit available to both parties
--transfer               ✅               ❌     owner only — mirror cannot self-promote
--check                  ✅               ✅     both servers monitor state
--status                 ✅               ✅     status is always available
--unblock                ✅               ✅     either server can clear its own blocklist

AM_OWNER / AM_MIRROR flags set by detect_hosts() from PARTNERSHIP_OWNER_HOST. All routing decisions use these flags — no hostname string comparisons. Mirror attempting --onboard or --transfer: blocked with a clear error message.

---

━━━ DESIGN NOTES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Why HOST1/HOST2 instead of PARTNERSHIP_OWNER/MIRROR? HOST1 and HOST2 are already defined in master.conf — SSH keys, Tailscale names, all connection details. A single PARTNERSHIP_OWNER_HOST var flips ownership. Duplicating as PARTNERSHIP_OWNER and PARTNERSHIP_MIRROR would require maintaining identical values in two places. One source of truth for the host definition.

Why does transfer require a confirmation string? Transfer touches master.conf on both servers, reconfigures WebUIs, and flips sync direction. A misstep mid-sequence leaves both servers with different auth configs and no clear source of truth. The confirmation string makes accidental execution impossible, not just unlikely.

Why do Tailscale removal and backup access expire at the same time? If the mirror can't reach the owner's server via Tailscale, the backup is also unreachable. PARTNERSHIP_GRACE_HOURS controls both — one var, consistent behaviour. No misleading "data available for 30 days" when access is gone in 6 hours.

Why does either server auto-offboard after 30 days offline? 30 consecutive days of missed sync cycles means the relationship has effectively ended regardless of intent. Auto-offboard makes the state official. Each server acts independently — no coordination required to finalise.

Why are auth/arr stack containers removed on offboard? The auth stack running on the mirror is the owner's stack — deployed from the owner's XML templates, managed by the owner. On offboard, the mirror restores its own parked containers (PARTNERSHIP_OWN_CONTAINERS) and sets up independent auth from scratch. Nothing from the owner's deployment lingers. Appdata is deleted alongside containers so there's no stale config left behind. Both fallback coverage containers (named *-Owner) and the owner-deployed stack (from PARTNERSHIP_AUTH_STACK / PARTNERSHIP_ARR_STACK) are removed — the mirror gets a clean slate.

Why does XML ordering matter for the auth stack? Authelia requires MariaDB and Redis to be running and accepting connections before it starts. deploy_container_from_xml() starts each container immediately after creating it. If Authelia is deployed before MariaDB, it will fail to connect and may not recover automatically. The array order in HOST1_PARTNERSHIP_AUTH_STACK is enforced by convention — database containers first, then applications.