50 KiB
50 KiB
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🤝 PARTNERSHIP
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Manages the full lifecycle of a two-server partnership — setup, daily operation, clean separation, and ownership transfer. One server owns the shared services. The other mirrors them and benefits from them. The relationship has a beginning, an ongoing operation, and an end — and every phase deserves the same engineering discipline as the rest of the ecosystem.
This script exists because a clean exit should be as easy as a clean setup. The partnership is not a permanent commitment.
--offboardworks from either server at any time. Everything HOST2 needs to run independently is already on HOST2. The only thing that stops working immediately is the sync — and that's intentional.
━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔴 Two Servers, One Auth Stack, No Clean Way to Share It
The auth stack — NginxProxyManager, LLDAP, Authelia, MariaDB, Redis — runs on HOST1. HOST2 serves his own domain to his own household. He needs his own auth. But maintaining two independent auth stacks means double the work: two places to add users, two places to update proxy rules, two places to renew certs, two configurations that inevitably drift apart. One change made on HOST1 has to be manually replicated to HOST2 — or it isn't, and the configurations diverge silently.
The real cost isn't the initial setup. It's the maintenance burden that accumulates over months — every new user, every proxy rule change, every cert renewal, every configuration update applied in one place and forgotten in the other.
The fix: one auth stack with a managed mirror. HOST1 owns the configuration. HOST2 runs a warm copy that stays current via 15-minute sync. HOST2's operator makes zero auth management decisions — he clicks an auth container, gets HOST1's WebUI via Tailscale, changes happen there, propagate to his server in 15 minutes. One place to manage everything for both households.
🔴 No Structure Around the Relationship Itself
Setting up the mirror was a manual process. SSH in, reconfigure container WebUI URLs one by one, copy auth config, verify connectivity, update state tracking. No defined sequence. No dry-run capability. No verification that each step worked. If something went wrong midway, the mirror was in an inconsistent state with no clear way to understand what had and hadn't been done.
The same problem on offboard — worse, actually. Because offboard involves stopping a sync that's been running for months, making a final copy of data, reconfiguring WebUIs back to local addresses, removing Tailscale access, and notifying both servers. A manual process with this many steps, taken under pressure at the moment of separation, is the kind of process that leaves one or both parties in a bad state.
The fix: partnership_manage.sh with explicit modes for each lifecycle phase. Each mode
is a defined sequence. Every step is verified. Dry-run shows exactly what will happen
before anything changes. State files make the current relationship status unambiguous
from either server.
🔴 No Safe Way to Check If the Other Server Has Gone Away
After months of operation, HOST2 goes quiet. Maybe a hardware failure. Maybe the person moved. Maybe a forgotten reboot that never completed. The sync starts failing. The offline counter increments. But nothing actually happens — the ecosystem just keeps failing the same sync, incrementing the same counter, sending the same notifications.
Without a defined threshold and an automated response, "partner gone for 30 days" looks exactly like "partner gone for 3 years" from the ecosystem's perspective. Both require manual intervention to clean up.
The fix: PARTNERSHIP_OFFLINE_THRESHOLD. After this many days of missed sync cycles,
both servers independently auto-offboard. HOST1 removes HOST2 from Tailscale, disables
critical sync, writes INACTIVE state. HOST2 — if it eventually comes back — reads HOST1's
INACTIVE state and cleans up its own side. The relationship is formally ended from both
sides without anyone needing to be physically present.
🔴 Ownership Transfer Had No Safe Path
The arrangement was always intended to be flexible — HOST1 owns the auth stack now, but circumstances change. HOST2 operator might want to take ownership. Or HOST1 might move and HOST2's location becomes more appropriate for the primary. Swapping ownership manually meant reconfiguring WebUIs on both servers, swapping sync direction, updating master.conf on both servers, and hoping the sequence was correct.
A misstep midway through — like flipping sync direction before the final sync completed — could leave both servers with different auth configurations and no clear source of truth.
The fix: --transfer with a required confirmation string, a consecutive health check
system, and a strict sequence. The confirmation string is as long and specific as
--i-know-what-im-doing in the arr cleanup scripts — you cannot type it accidentally.
Health strikes require both servers to be healthy on multiple consecutive checks before
the transfer begins. The sequence guarantees a final sync in the current direction before
anything is flipped.
━━━ THE RELATIONSHIP MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HOST1 (owner) HOST2 (mirror)
──────────────── ────────────────────────────
Source of truth Warm copy — always current
Auth stack config Auth containers running
NPM proxy rules NPM serving his domain
LLDAP users LLDAP — same users
Authelia policies Authelia — same policies
Certs Certs — mirrored, valid
Emby Emby — dirty-synced every 15min
Changes made here ──→ Propagated every 15 minutes
WebUI management ──→ Redirected to HOST1 via Tailscale
Config → git push ──→ Received via git pull on next cycle
── What the Mirror's Daily Experience Looks Like ───────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# HOST2 operator opens the unRAID Docker UI.
# Clicks NginxProxyManager.
# Browser opens HOST1's NPM WebUI automatically (via Tailscale redirect).
# Makes a proxy rule change.
# 15 minutes later it's live on HOST2.
#
# HOST2 operator never:
# Opens HOST2's NPM directly (WebUI points to HOST1)
# Manages LLDAP users on HOST2 (WebUI points to HOST1)
# Touches Authelia config on HOST2 (WebUI points to HOST1)
# Renews HOST2's certs manually (Critical-Data sync handles it)
#
# HOST2 operator always benefits from:
# Every proxy rule HOST1 adds
# Every user HOST1 adds to LLDAP
# Every security policy HOST1 configures
# Every cert HOST1 renews
# ─────────────────────────────────────────────────────────────────────────────
── What the Owner's Daily Experience Looks Like ────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# HOST1 operator manages auth as normal:
# Opens NPM → adds a proxy rule → 15min later live on both servers
# Opens LLDAP → adds a user → 15min later exists on both servers
# Renews a cert → 15min later current on both servers
#
# HOST1 operator never needs to:
# SSH to HOST2 to apply changes
# Coordinate with HOST2 operator on auth changes
# Know what HOST2's auth config currently looks like
# Think about HOST2 at all during normal operation
# ─────────────────────────────────────────────────────────────────────────────
── What Makes the Redirect Transparent ─────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Three things work together to make the auth stack transparent to HOST2:
#
# 1. Auth stack warm on both servers always
# NPM, LLDAP, Authelia, MariaDB, Redis — all running on HOST2 continuously.
# When HOST2's domain gets traffic, auth is ready immediately.
# No failover delay, no cold start.
#
# 2. WebUI URLs pointing to HOST1
# unRAID stores WebUI URLs in the Docker template.
# partnership_manage.sh --onboard edits these templates on HOST2.
# "NginxProxyManager|81" → URL becomes http://[HOST1-tailscale-ip]:81
# HOST2 operator clicks the container → gets HOST1's WebUI.
# All changes land on HOST1, propagate to HOST2 automatically.
#
# 3. Tailscale permanent connection
# Both servers permanently connected via Tailscale encrypted mesh.
# The WebUI redirect works because Tailscale IP is always reachable.
# No VPN configuration needed. No open ports.
# ─────────────────────────────────────────────────────────────────────────────
── The One WebUI That Stays Local ──────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Tailscale WebUI is left pointing to HOST2's own local interface.
# When diagnosing connectivity issues between servers you need to see
# HOST2's network state from its own perspective — not HOST1's view.
# It is the one WebUI that genuinely needs to stay local.
# Every other auth container WebUI redirects to HOST1.
# ─────────────────────────────────────────────────────────────────────────────
━━━ STATE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# ─────────────────────────────────────────────────────────────────────────────
# Each server writes ONLY its own state file. Neither server writes the other's.
# State propagates via SSH reads — no rsync, no shared filesystem.
#
# /boot/config/partnership_HOST1.db ← HOST1 writes only, HOST2 reads via SSH
# /boot/config/partnership_HOST2.db ← HOST2 writes only, HOST1 reads via SSH
#
# /boot/config — survives reboots, available before array starts, minimal flash wear.
# flock on all writes — prevents concurrent corruption from overlapping --check cycles.
#
# Example state file content:
state=ACTIVE
last_updated=2026-04-14 03:00:00
last_seen_remote=2026-04-14 03:00:00
owner_host=HOST1
offline_days=0
# ─────────────────────────────────────────────────────────────────────────────
Why this design enables deferred offboard:
HOST2 offboards while HOST1 is temporarily unreachable — writes its own state file to
INACTIVE. When HOST1 can next reach HOST2 (next --check cycle), it reads the INACTIVE
state and finalises the offboard from its side automatically. No coordination required.
No message passing. Each server acts on what it can observe.
━━━ THE LIFECYCLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🟢 --onboard — Establishing the Partnership
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Run from HOST1 (owner) only. Establishes the mirror relationship. After onboard, HOST2's auth WebUIs redirect to HOST1 and Critical-Data sync begins keeping them current.
Always dry-run first. Onboard modifies container templates on HOST2 and verifies WebUI connectivity. The dry-run shows exactly which templates would be changed and what the redirect URLs would become — review before committing.
── Onboard Sequence ─────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Step 1 — Pre-flight checks
# Both servers healthy (array started, Docker responding)
# Tailscale connected — HOST2 reachable via Tailscale IP
# unRAID version parity — both servers on compatible versions
# HOST2 auth containers exist (may be stopped — that's fine)
# → Any failure: abort, explain what failed, no changes made
#
# Step 2 — Reconfigure HOST2 WebUI templates
# For each container in PARTNERSHIP_AUTH_WEBUIS:
# Find template in /boot/config/plugins/dockerMan/templates-user/
# Update WebUI URL → http://[HOST1-TAILSCALE-IP]:PORT
# Container restart not needed — template change takes effect on next open
#
# Step 3 — Verify WebUI connectivity (if PARTNERSHIP_ONBOARD_VERIFY=true)
# curl each configured WebUI via HOST1's Tailscale IP
# Confirms Tailscale routing is working and the service is responding
# A redirect that can't be reached is worse than no redirect
#
# Step 4 — Write state files
# Write partnership_HOST1.db=ACTIVE locally
# SSH to HOST2 → push state, write partnership_HOST2.db=ACTIVE
# Reset offline counter on both sides
#
# Step 5 — Notify (if PARTNERSHIP_ONBOARD_NOTIFY=true)
# Notification to both servers: partnership active
# ─────────────────────────────────────────────────────────────────────────────
── Configuration ────────────────────────────────────────────────────────────
# master_host1.conf (or master_host2.conf for HOST2 side of the relationship)
# ─────────────────────────────────────────────────────────────────────────────
# Containers whose WebUI URLs are reconfigured on onboard.
# Format: "ContainerName|WebUIPort"
# The port here is what gets redirected — matches the container's WebUI port.
#
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
"NginxProxyManager|81" # NPM admin interface — all proxy rules managed here
"Lldap-Gmer4Lfe|17170" # user directory — all user management here
"Authelia|9091" # SSO authentication — policy management here
"Authelia-Secondary|9092" # secondary Authelia instance
)
# Paths the mirror should collect during the offboard grace window.
# Files here are accessible to HOST2 for PARTNERSHIP_GRACE_HOURS after offboard.
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
# "/mnt/user/appdata-Failover/Jayred365-Emby" # HOST2's Emby backup on HOST1
)
── Usage ───────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Always dry-run first — shows exact template changes and redirect URLs
# ─────────────────────────────────────────────────────────────────────────────
partnership_manage.sh --onboard --dry-run
# ─────────────────────────────────────────────────────────────────────────────
# Live onboard — owner (HOST1) only
# ─────────────────────────────────────────────────────────────────────────────
partnership_manage.sh --onboard
# Verbose — show each pre-flight check and template change
partnership_manage.sh --onboard --log
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔴 --offboard — Clean Separation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Either server can initiate offboard. The process differs slightly depending on who runs it — but both paths end with both servers fully independent. The mirror leaves with current auth config, working certs, and full control of its own services.
Always dry-run first. Offboard stops the sync, reconfigures WebUIs, and removes Tailscale access. The dry-run shows exactly what would happen, in order, before any of it does.
── Owner-Initiated Offboard (HOST1 runs --offboard) ───────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# 1. 10-second countdown shown — Ctrl-C to abort if you changed your mind
#
# 2. Stop any running rsync
# rsync_stop.sh --rsync-only — stops the rsync but leaves container recovery
# to docker_watchdog.sh. Nothing else is interrupted.
#
# 3. Final sync
# Critical-Data pushed to HOST2 one last time — HOST2 leaves with current state
# Emby failover sync pushed — HOST2 has current watch states
# This is the cleanest possible state for HOST2 to inherit.
#
# 4. Reconfigure HOST2 WebUI templates → localhost
# SSH to HOST2 → update each auth container WebUI URL → http://localhost:PORT
# HOST2 operator now clicks NPM → gets HOST2's own NPM
# HOST2 manages its own auth from this point forward
#
# 5. Disable critical sync
# Write CRITICAL_RSYNC_ENABLED=false to master.conf
# Critical-Data sync stops at next 15-minute cycle
#
# 6. Write state files
# partnership_HOST1.db=INACTIVE
# SSH to HOST2 → partnership_HOST2.db=INACTIVE
#
# 7. Grace period — PARTNERSHIP_GRACE_HOURS (default 6 hours)
# HOST2's data on HOST1 remains accessible for this window
# HOST2 can collect anything it needs via Tailscale during this time
# Both access and Tailscale removal happen at the same moment (step 8)
#
# 8. Remove HOST2 from Tailscale tailnet
# API call to revoke HOST2's device — encrypted connection ends
# Backup access ends at the same time by design — no misleading window
#
# 9. Notify HOST2 — "offboard complete, access removed in Xh"
# ─────────────────────────────────────────────────────────────────────────────
── Mirror-Initiated Offboard (HOST2 runs --offboard) ──────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# HOST2 can offboard at any time, even if HOST1 is temporarily unreachable.
# The deferred finalisation mechanism ensures HOST1 catches up automatically.
#
# 1. Reconfigure own WebUI templates → localhost
# All auth containers now point to HOST2's own services
# HOST2 manages its own auth immediately — no waiting for HOST1
#
# 2. Write partnership_HOST2.db=INACTIVE locally
#
# 3. SSH to HOST1 (if reachable) → push state file notification
# HOST1 reads INACTIVE → begins owner offboard sequence (steps 3-9 above)
# HOST1 runs final sync → sends current state to HOST2 one more time
# HOST1 handles Tailscale removal and grace period
#
# 4. If HOST1 unreachable — deferred finalisation
# HOST2 has already reconfigured to localhost — fully independent now
# STATE file written — HOST1 will read it on next --check when reachable
# Next time HOST1 runs --check and can reach HOST2: sees INACTIVE → finalises
# ─────────────────────────────────────────────────────────────────────────────
── What HOST2 Keeps After Offboard ────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# What HOST2 leaves with:
Current auth config ✓ final sync was clean — NPM rules, users, policies
Own DDNS + updater ✓ always was his, unaffected by partnership
Own Git mirror ✓ full ecosystem — git pull always ran on HOST2
Own anime source of truth ✓ Anime_Shows / Anime_Movies — always was his
Auth WebUIs → localhost ✓ managing own auth directly from this point
Full independence ✓ just stop the sync — ecosystem continues
Certs valid ~60-90 days ✓ cert_monitor.sh warns at 30 days — time to renew
# What HOST1 keeps after offboard:
His hardware ✓ unaffected
His media shares ✓ Movies, Tv_Shows, Music — his source of truth
His services ✓ running on HOST1 as always
His domain ✓ DNS unaffected
# What HOST1 loses:
Offsite backup ✗ was HOST2's hardware in a different location
Geographic redundancy ✗ cannot be scripted — that was the person
# ─────────────────────────────────────────────────────────────────────────────
# The arrangement was always mutual. Both parties provided something the scripts
# cannot replace — hardware on a different power utility in a different location.
# Everything else can be reconfigured. That cannot.
── Usage ───────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Available from either server — always dry-run first
# ─────────────────────────────────────────────────────────────────────────────
partnership_manage.sh --offboard --dry-run # show the complete sequence, no changes
partnership_manage.sh --offboard # live offboard — 10s countdown before starting
partnership_manage.sh --offboard --log # verbose — show each step as it executes
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔄 --transfer — Flipping Ownership
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Transfers ownership from HOST1 to HOST2 (or back after a previous transfer). After transfer, HOST2 becomes the auth source of truth and HOST1 receives syncs. Only the current owner can initiate transfer.
This is the most significant operation in the lifecycle. It reconfigures both servers, flips sync direction, and changes master.conf permanently on both sides. The confirmation string and health strike system exist to prevent accidents.
── Why This Is Protected ───────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Transfer involves:
# Modifying master.conf on both servers simultaneously
# Flipping sync direction (HOST1 → HOST2 becomes HOST2 → HOST1)
# Reconfiguring WebUI templates on both servers
# Making HOST2 the authoritative source for auth decisions
#
# A partial transfer — if interrupted mid-sequence — leaves both servers with
# different auth configurations and no clear source of truth. The health strike
# system prevents starting a transfer when either server is degraded.
# The confirmation string prevents accidental execution entirely.
# ─────────────────────────────────────────────────────────────────────────────
── Transfer Sequence ───────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# 1. Display current and future ownership clearly
# "Current: HOST1 → Mirror: HOST2"
# "After: HOST2 → Mirror: HOST1"
# No ambiguity about what is being confirmed.
#
# 2. Confirmation string check
# Must type exactly: i-understand-this-transfers-ownership
# Long and specific by design — cannot be typed accidentally.
#
# 3. Health strike system
# Both servers must pass PARTNERSHIP_TRANSFER_STRIKES consecutive health checks.
# Max PARTNERSHIP_TRANSFER_MAX_ATTEMPTS attempts (default 20) before aborting.
# Each strike: check both arrays started, Docker daemons responding, Tailscale connected.
# Protects against transferring during a degraded state.
#
# 4. Final sync in current direction
# Push Current-Data from current owner (HOST1) to current mirror (HOST2).
# New mirror (HOST1) starts with current state from the outgoing owner.
#
# 5. Reconfigure WebUI templates on HOST2
# HOST2 WebUIs → localhost (HOST2 now manages its own auth directly)
#
# 6. Reconfigure WebUI templates on HOST1
# HOST1 WebUIs → HOST2's Tailscale IP (HOST1 now redirects to HOST2)
#
# 7. Flip PARTNERSHIP_OWNER_HOST in master.conf on BOTH servers
# "HOST1" → "HOST2" written to both servers via SSH
#
# 8. Write updated state files on both servers
#
# 9. Notify both servers — transfer complete, new roles shown
# ─────────────────────────────────────────────────────────────────────────────
── After Transfer ───────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Before transfer: HOST1 = owner, HOST2 = mirror
# After transfer: HOST2 = owner, HOST1 = mirror
#
# HOST2 now:
# All auth changes made here directly
# Critical-Data pushed → to HOST1 every 15 minutes
# NPM, LLDAP, Authelia — HOST2's own instances are authoritative
# HOST1's WebUIs redirect to HOST2 via Tailscale
#
# HOST1 now:
# Receives Critical-Data sync from HOST2
# Auth WebUIs point to HOST2 — clicking NPM opens HOST2's NPM
# Never makes auth configuration changes directly
#
# To transfer back — run --transfer from HOST2 (the new owner)
# PARTNERSHIP_OWNER_HOST="HOST2" after first transfer, "HOST1" after second
# ─────────────────────────────────────────────────────────────────────────────
── Usage ───────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Dry run — ALWAYS first. Shows the full sequence, health check results,
# and what master.conf changes would be made on both servers.
# ─────────────────────────────────────────────────────────────────────────────
partnership_manage.sh --transfer --dry-run
# ─────────────────────────────────────────────────────────────────────────────
# Live transfer — owner only — requires exact confirmation string
# ─────────────────────────────────────────────────────────────────────────────
partnership_manage.sh --transfer --confirm=i-understand-this-transfers-ownership
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔁 --check — Automated State Monitor
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Called automatically by critical_sync_maintenance.sh every 15 minutes. Never run
manually. Silent when healthy — produces output only when state changes or thresholds
are hit.
# Called internally — do not run manually:
partnership_manage.sh --check --remote-seen # rsync succeeded, remote reachable
partnership_manage.sh --check --remote-unseen # rsync failed, remote unreachable
── What It Does Each Cycle ─────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# --remote-seen path (rsync succeeded → HOST2 was reachable):
# Reset offline counter to 0
# Update last_seen_remote timestamp
# SSH to read remote state file
# Both ACTIVE → silent ✅
# Remote INACTIVE → begin offboard finalisation from owner side
#
# --remote-unseen path (rsync failed → HOST2 unreachable):
# Increment offline counter by 1
# Counter × 15min = approximate offline duration
# At PARTNERSHIP_OFFLINE_THRESHOLD days: auto-offboard begins
# Notify when counter hits threshold — not on every missed cycle
#
# Why the rsync outcome drives this:
# The rsync that runs before --check provides a reliable connectivity signal.
# If rsync to HOST2 succeeded, HOST2 is reachable — reset counter.
# If rsync failed, HOST2 may be unreachable — increment counter.
# No separate ping needed — the sync attempt is the connectivity test.
# ─────────────────────────────────────────────────────────────────────────────
── Auto-Offboard ────────────────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# After this many days of consecutive missed sync cycles, auto-offboard begins.
# Works both directions independently — neither server needs the other to act.
#
PARTNERSHIP_OFFLINE_THRESHOLD=30 # days — each missed cycle = ~15min
#
# HOST1 sees HOST2 offline for 30 days:
# AM_OWNER → full owner-initiated offboard
# Tailscale removal via API (API works regardless of HOST2 reachability)
# Critical sync disabled
# State: INACTIVE
#
# HOST2 sees HOST1 offline for 30 days:
# AM_MIRROR → mirror-initiated offboard
# Reconfigure own WebUIs → localhost
# Write HOST2.db=INACTIVE
# Fully independent immediately
# When HOST1 eventually comes back → reads INACTIVE → cleans its side
#
# 30 days covers:
# Extended hardware failure (people do fix things eventually)
# Someone moving and being offline for weeks
# But not: a permanent departure
# ─────────────────────────────────────────────────────────────────────────────
━━━ ROLE-BASED ACCESS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# ─────────────────────────────────────────────────────────────────────────────
# Which server can run which mode:
#
HOST1 (owner) HOST2 (mirror)
--onboard ✅ ❌ ownership is granted, not taken
--offboard ✅ ✅ clean exit available to both parties
--transfer ✅ ❌ owner only — mirror cannot self-promote
--check ✅ ✅ both servers monitor state
--status ✅ ✅ status is always available
#
# AM_OWNER / AM_MIRROR flags set by detect_hosts() from PARTNERSHIP_OWNER_HOST.
# All routing decisions use these flags — no hostname string comparisons.
# Mirror attempting --onboard or --transfer: blocked with a clear error message.
# ─────────────────────────────────────────────────────────────────────────────
━━━ INTEGRATION WITH CRITICAL SYNC ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# ─────────────────────────────────────────────────────────────────────────────
# Partnership check runs inside critical_sync_maintenance.sh — the 15-minute
# orchestrator. The rsync outcome directly informs the partnership check.
#
# Schedule: */15 * * * *
#
# Execution order in critical_sync_maintenance.sh:
# 1. Critical-Data rsync → auth stack sync
# 2. emby-failover rsync → dirty Emby sync
# 3. partnership --check → state check, informed by rsync results
# → --remote-seen if rsync succeeded (HOST2 reachable)
# → --remote-unseen if rsync failed (HOST2 unreachable)
#
# When critical sync is disabled (CRITICAL_RSYNC_ENABLED=false):
# partnership --check is skipped automatically
# Offline counter does not increment from a sync that wasn't attempted
# ─────────────────────────────────────────────────────────────────────────────
━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# ── Partnership Gate ───────────────────────────────────────────────────────
PARTNERSHIP_ENABLED=false # set true once both servers are configured
PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer
# ── Tailscale Removal ──────────────────────────────────────────────────────
PARTNERSHIP_REMOVE_TAILSCALE=true # remove mirror from tailnet on offboard
TAILSCALE_API_KEY="" # tskey-api-... from tailscale.com/admin/settings/keys
TAILSCALE_TAILNET="" # your tailnet name (e.g. example.github)
# required when PARTNERSHIP_REMOVE_TAILSCALE=true
# ── Timing ─────────────────────────────────────────────────────────────────
PARTNERSHIP_GRACE_HOURS=6 # hours before Tailscale removal after offboard
# backup access also expires at this time — by design
PARTNERSHIP_OFFLINE_THRESHOLD=30 # days unreachable before auto-offboard (both servers)
PARTNERSHIP_SYNC_INTERVAL=15 # informational — actual schedule in cron
# ── Transfer Safety ────────────────────────────────────────────────────────
PARTNERSHIP_TRANSFER_CONFIRM="i-understand-this-transfers-ownership"
# must type this exactly to proceed — long by design
PARTNERSHIP_TRANSFER_STRIKES=3 # consecutive health checks required before transfer
PARTNERSHIP_TRANSFER_MAX_ATTEMPTS=20 # max health attempts before giving up
# ── Onboard Behaviour ──────────────────────────────────────────────────────
PARTNERSHIP_ONBOARD_VERIFY=true # curl each WebUI after reconfigure to verify routing
PARTNERSHIP_ONBOARD_NOTIFY=true # notify both servers on successful onboard
# master_host1.conf / master_host2.conf
# ─────────────────────────────────────────────────────────────────────────────
# Containers whose WebUI URLs are reconfigured on onboard/offboard
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
"NginxProxyManager|81"
"Lldap-Gmer4Lfe|17170"
"Authelia|9091"
"Authelia-Secondary|9092"
)
# Paths accessible to HOST2 during grace window after offboard
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
# "/mnt/user/appdata-Failover/Jayred365-Emby"
)
━━━ INITIAL SETUP REQUIREMENTS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Before --onboard can run, five things must be in place:
1. Tailscale Connected on Both Servers
# Verify HOST2 is visible from HOST1:
tailscale ip -4 unRAID-Jayred365 # should return HOST2's Tailscale IP
# Verify HOST1 is visible from HOST2:
tailscale ip -4 unRAID-Gmer4Lfe # should return HOST1's Tailscale IP
# Verify reachability:
tailscale ping unRAID-Jayred365
HOST1 and HOST2 hostnames must match their exact Tailscale device names — already
enforced by the ecosystem since REMOTE_SERVER_NAME is used everywhere.
2. SSH Keys — Passwordless Both Directions
# From HOST1 — should print HOST2's hostname without a password prompt:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"
# From HOST2 — should print HOST1's hostname without a password prompt:
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"
3. Auth Containers Exist on HOST2
# Containers must exist — they can be stopped:
ssh root@[HOST2-ip] "docker inspect NginxProxyManager --format '{{.State.Status}}'"
# Expected: created, exited, or running — NOT "no such container"
# Template files must exist for WebUI reconfiguration:
ssh root@[HOST2-ip] "ls /boot/config/plugins/dockerMan/templates-user/"
4. Critical-Data rsync Profile Configured
# master.conf — CRITICAL_SYNC_SHARES must include the auth stack path:
CRITICAL_SYNC_SHARES=(
"/mnt/user/appdata-Failover/Critical-Data" # auth config — NPM, Authelia, LLDAP, certs
)
# This is what keeps HOST2 current after onboard
5. Tailscale API Key (if removing from tailnet on offboard)
# master.conf
PARTNERSHIP_REMOVE_TAILSCALE=true
TAILSCALE_API_KEY="tskey-api-..." # from tailscale.com/admin/settings/keys
TAILSCALE_TAILNET="example.github" # your tailnet name
# Required scope: Devices write
# Generate at: https://login.tailscale.com/admin/settings/keys
━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔴 WebUI Still Pointing to Old IP After Onboard
Check template was found and modified:
partnership_manage.sh --status shows current WebUI redirect state
ssh HOST2 "cat /boot/config/plugins/dockerMan/templates-user/NginxProxyManager.xml"
should show HOST1's Tailscale IP
Some containers need a restart to pick up new WebUI URL:
docker restart NginxProxyManager on HOST2 — picks up template change
Verify Tailscale routing is working:
curl http://[HOST1-tailscale-ip]:81 from HOST2 — should reach HOST1's NPM
🔴 Transfer Health Checks Failing
Both arrays must be fully started:
ls /mnt/user should show share directories on both servers
Both Docker daemons responding:
docker ps should return a list, not hang
Tailscale connected on both servers:
tailscale status should show the remote peer as online
Max attempts before giving up: PARTNERSHIP_TRANSFER_MAX_ATTEMPTS (default 20)
Each attempt waits for next health window — may take several minutes total
🔴 Mirror's Domain Broken After Offboard
Auth WebUIs reconfigured to localhost — mirror manages auth directly now.
The containers are running and functional — they just no longer redirect to HOST1.
Verify auth containers running:
docker ps | grep -E "NPM|Authelia|LLDAP"
Check cert expiry — certs won't auto-renew anymore via HOST1:
cert_monitor.sh --dry-run shows days remaining on each domain's cert
Set up cert auto-renewal on HOST2 independently if not already configured.
🔴 State Mismatch Between Servers
Run --status on both servers to see both state files:
partnership_manage.sh --status on HOST1
partnership_manage.sh --status on HOST2
If one shows ACTIVE and other shows INACTIVE:
The INACTIVE side has already offboarded
Run --offboard on the ACTIVE side to sync the state
If HOST2 unreachable:
HOST2 will self-resolve on next --check when it becomes reachable
HOST1 reads HOST2's INACTIVE state → finalises from owner side automatically
🔴 Auto-Offboard Triggered Unexpectedly
Check the offline counter:
cat /boot/config/partnership_offline_days.db
Extended Tailscale connectivity issue may have incremented the counter:
tailscale status verify HOST2 appears in HOST1's peer list
If partnership should continue — re-onboard:
partnership_manage.sh --onboard --dry-run # verify setup is correct
partnership_manage.sh --onboard # re-establish partnership
Counter resets to 0 on successful --onboard.
━━━ DESIGN NOTES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Why HOST1/HOST2 instead of PARTNERSHIP_OWNER/MIRROR?
HOST1 and HOST2 are already defined in master.conf — SSH keys, Tailscale names,
all connection details. A single PARTNERSHIP_OWNER_HOST var flips ownership.
Duplicating everything as PARTNERSHIP_OWNER and PARTNERSHIP_MIRROR would require
maintaining identical values in two places. One source of truth for the host
definition, one var to flip direction.
Why does transfer require a confirmation string?
Same philosophy as --i-know-what-im-doing in the arr cleanup scripts.
Transfer touches master.conf on both servers, reconfigures WebUIs, and flips
sync direction. The confirmation string makes accidental execution impossible,
not just unlikely.
Why do Tailscale removal and backup access expire at the same time?
If the mirror can't reach the owner's server via Tailscale, the backup is also
unreachable. PARTNERSHIP_GRACE_HOURS controls both — one var, consistent behaviour,
no misleading "data available for 30 days" when access is gone in 6 hours.
Why does either server auto-offboard after 30 days offline?
30 consecutive days of missed sync cycles means the relationship has effectively
ended regardless of intent. Auto-offboard makes the state official. Each server
acts independently — they don't need to coordinate to offboard.
Why does WebUI reconfiguration use the unRAID template system?
unRAID stores WebUI URLs in /boot/config/plugins/dockerMan/templates-user/.
Editing templates persists across container restarts and array reboots.
Direct Docker label manipulation is lost on next container recreation.