945 lines
50 KiB
Markdown
945 lines
50 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
# 🤝 PARTNERSHIP
|
||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
**Manages the full lifecycle of a two-server partnership — setup, daily operation,
|
||
clean separation, and ownership transfer.** One server owns the shared services.
|
||
The other mirrors them and benefits from them. The relationship has a beginning,
|
||
an ongoing operation, and an end — and every phase deserves the same engineering
|
||
discipline as the rest of the ecosystem.
|
||
|
||
> **This script exists because a clean exit should be as easy as a clean setup.**
|
||
> The partnership is not a permanent commitment. `--offboard` works from either server
|
||
> at any time. Everything HOST2 needs to run independently is already on HOST2.
|
||
> The only thing that stops working immediately is the sync — and that's intentional.
|
||
|
||
---
|
||
|
||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
---
|
||
|
||
### 🔴 Two Servers, One Auth Stack, No Clean Way to Share It
|
||
|
||
The auth stack — NginxProxyManager, LLDAP, Authelia, MariaDB, Redis — runs on HOST1.
|
||
HOST2 serves his own domain to his own household. He needs his own auth. But maintaining
|
||
two independent auth stacks means double the work: two places to add users, two places to
|
||
update proxy rules, two places to renew certs, two configurations that inevitably drift
|
||
apart. One change made on HOST1 has to be manually replicated to HOST2 — or it isn't,
|
||
and the configurations diverge silently.
|
||
|
||
The real cost isn't the initial setup. It's the maintenance burden that accumulates over
|
||
months — every new user, every proxy rule change, every cert renewal, every configuration
|
||
update applied in one place and forgotten in the other.
|
||
|
||
The fix: one auth stack with a managed mirror. HOST1 owns the configuration. HOST2 runs
|
||
a warm copy that stays current via 15-minute sync. HOST2's operator makes zero auth
|
||
management decisions — he clicks an auth container, gets HOST1's WebUI via Tailscale,
|
||
changes happen there, propagate to his server in 15 minutes. One place to manage
|
||
everything for both households.
|
||
|
||
---
|
||
|
||
### 🔴 No Structure Around the Relationship Itself
|
||
|
||
Setting up the mirror was a manual process. SSH in, reconfigure container WebUI URLs
|
||
one by one, copy auth config, verify connectivity, update state tracking. No defined
|
||
sequence. No dry-run capability. No verification that each step worked. If something
|
||
went wrong midway, the mirror was in an inconsistent state with no clear way to
|
||
understand what had and hadn't been done.
|
||
|
||
The same problem on offboard — worse, actually. Because offboard involves stopping a sync
|
||
that's been running for months, making a final copy of data, reconfiguring WebUIs back
|
||
to local addresses, removing Tailscale access, and notifying both servers. A manual
|
||
process with this many steps, taken under pressure at the moment of separation,
|
||
is the kind of process that leaves one or both parties in a bad state.
|
||
|
||
The fix: `partnership_manage.sh` with explicit modes for each lifecycle phase. Each mode
|
||
is a defined sequence. Every step is verified. Dry-run shows exactly what will happen
|
||
before anything changes. State files make the current relationship status unambiguous
|
||
from either server.
|
||
|
||
---
|
||
|
||
### 🔴 No Safe Way to Check If the Other Server Has Gone Away
|
||
|
||
After months of operation, HOST2 goes quiet. Maybe a hardware failure. Maybe the
|
||
person moved. Maybe a forgotten reboot that never completed. The sync starts failing.
|
||
The offline counter increments. But nothing actually happens — the ecosystem just keeps
|
||
failing the same sync, incrementing the same counter, sending the same notifications.
|
||
|
||
Without a defined threshold and an automated response, "partner gone for 30 days"
|
||
looks exactly like "partner gone for 3 years" from the ecosystem's perspective.
|
||
Both require manual intervention to clean up.
|
||
|
||
The fix: `PARTNERSHIP_OFFLINE_THRESHOLD`. After this many days of missed sync cycles,
|
||
both servers independently auto-offboard. HOST1 removes HOST2 from Tailscale, disables
|
||
critical sync, writes INACTIVE state. HOST2 — if it eventually comes back — reads HOST1's
|
||
INACTIVE state and cleans up its own side. The relationship is formally ended from both
|
||
sides without anyone needing to be physically present.
|
||
|
||
---
|
||
|
||
### 🔴 Ownership Transfer Had No Safe Path
|
||
|
||
The arrangement was always intended to be flexible — HOST1 owns the auth stack now, but
|
||
circumstances change. HOST2 operator might want to take ownership. Or HOST1 might move
|
||
and HOST2's location becomes more appropriate for the primary. Swapping ownership manually
|
||
meant reconfiguring WebUIs on both servers, swapping sync direction, updating master.conf
|
||
on both servers, and hoping the sequence was correct.
|
||
|
||
A misstep midway through — like flipping sync direction before the final sync completed —
|
||
could leave both servers with different auth configurations and no clear source of truth.
|
||
|
||
The fix: `--transfer` with a required confirmation string, a consecutive health check
|
||
system, and a strict sequence. The confirmation string is as long and specific as
|
||
`--i-know-what-im-doing` in the arr cleanup scripts — you cannot type it accidentally.
|
||
Health strikes require both servers to be healthy on multiple consecutive checks before
|
||
the transfer begins. The sequence guarantees a final sync in the current direction before
|
||
anything is flipped.
|
||
|
||
---
|
||
|
||
## ━━━ THE RELATIONSHIP MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```
|
||
HOST1 (owner) HOST2 (mirror)
|
||
──────────────── ────────────────────────────
|
||
Source of truth Warm copy — always current
|
||
Auth stack config Auth containers running
|
||
NPM proxy rules NPM serving his domain
|
||
LLDAP users LLDAP — same users
|
||
Authelia policies Authelia — same policies
|
||
Certs Certs — mirrored, valid
|
||
Emby Emby — dirty-synced every 15min
|
||
|
||
Changes made here ──→ Propagated every 15 minutes
|
||
WebUI management ──→ Redirected to HOST1 via Tailscale
|
||
Config → git push ──→ Received via git pull on next cycle
|
||
```
|
||
|
||
---
|
||
|
||
### ── What the Mirror's Daily Experience Looks Like ───────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# HOST2 operator opens the unRAID Docker UI.
|
||
# Clicks NginxProxyManager.
|
||
# Browser opens HOST1's NPM WebUI automatically (via Tailscale redirect).
|
||
# Makes a proxy rule change.
|
||
# 15 minutes later it's live on HOST2.
|
||
#
|
||
# HOST2 operator never:
|
||
# Opens HOST2's NPM directly (WebUI points to HOST1)
|
||
# Manages LLDAP users on HOST2 (WebUI points to HOST1)
|
||
# Touches Authelia config on HOST2 (WebUI points to HOST1)
|
||
# Renews HOST2's certs manually (Critical-Data sync handles it)
|
||
#
|
||
# HOST2 operator always benefits from:
|
||
# Every proxy rule HOST1 adds
|
||
# Every user HOST1 adds to LLDAP
|
||
# Every security policy HOST1 configures
|
||
# Every cert HOST1 renews
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── What the Owner's Daily Experience Looks Like ────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# HOST1 operator manages auth as normal:
|
||
# Opens NPM → adds a proxy rule → 15min later live on both servers
|
||
# Opens LLDAP → adds a user → 15min later exists on both servers
|
||
# Renews a cert → 15min later current on both servers
|
||
#
|
||
# HOST1 operator never needs to:
|
||
# SSH to HOST2 to apply changes
|
||
# Coordinate with HOST2 operator on auth changes
|
||
# Know what HOST2's auth config currently looks like
|
||
# Think about HOST2 at all during normal operation
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── What Makes the Redirect Transparent ─────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Three things work together to make the auth stack transparent to HOST2:
|
||
#
|
||
# 1. Auth stack warm on both servers always
|
||
# NPM, LLDAP, Authelia, MariaDB, Redis — all running on HOST2 continuously.
|
||
# When HOST2's domain gets traffic, auth is ready immediately.
|
||
# No failover delay, no cold start.
|
||
#
|
||
# 2. WebUI URLs pointing to HOST1
|
||
# unRAID stores WebUI URLs in the Docker template.
|
||
# partnership_manage.sh --onboard edits these templates on HOST2.
|
||
# "NginxProxyManager|81" → URL becomes http://[HOST1-tailscale-ip]:81
|
||
# HOST2 operator clicks the container → gets HOST1's WebUI.
|
||
# All changes land on HOST1, propagate to HOST2 automatically.
|
||
#
|
||
# 3. Tailscale permanent connection
|
||
# Both servers permanently connected via Tailscale encrypted mesh.
|
||
# The WebUI redirect works because Tailscale IP is always reachable.
|
||
# No VPN configuration needed. No open ports.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── The One WebUI That Stays Local ──────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Tailscale WebUI is left pointing to HOST2's own local interface.
|
||
# When diagnosing connectivity issues between servers you need to see
|
||
# HOST2's network state from its own perspective — not HOST1's view.
|
||
# It is the one WebUI that genuinely needs to stay local.
|
||
# Every other auth container WebUI redirects to HOST1.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ STATE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Each server writes ONLY its own state file. Neither server writes the other's.
|
||
# State propagates via SSH reads — no rsync, no shared filesystem.
|
||
#
|
||
# /boot/config/partnership_HOST1.db ← HOST1 writes only, HOST2 reads via SSH
|
||
# /boot/config/partnership_HOST2.db ← HOST2 writes only, HOST1 reads via SSH
|
||
#
|
||
# /boot/config — survives reboots, available before array starts, minimal flash wear.
|
||
# flock on all writes — prevents concurrent corruption from overlapping --check cycles.
|
||
#
|
||
# Example state file content:
|
||
state=ACTIVE
|
||
last_updated=2026-04-14 03:00:00
|
||
last_seen_remote=2026-04-14 03:00:00
|
||
owner_host=HOST1
|
||
offline_days=0
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
**Why this design enables deferred offboard:**
|
||
HOST2 offboards while HOST1 is temporarily unreachable — writes its own state file to
|
||
INACTIVE. When HOST1 can next reach HOST2 (next `--check` cycle), it reads the INACTIVE
|
||
state and finalises the offboard from its side automatically. No coordination required.
|
||
No message passing. Each server acts on what it can observe.
|
||
|
||
---
|
||
|
||
## ━━━ THE LIFECYCLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🟢 --onboard — Establishing the Partnership
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Run from HOST1 (owner) only. Establishes the mirror relationship. After onboard,
|
||
HOST2's auth WebUIs redirect to HOST1 and Critical-Data sync begins keeping them current.
|
||
|
||
> **Always dry-run first.** Onboard modifies container templates on HOST2 and verifies
|
||
> WebUI connectivity. The dry-run shows exactly which templates would be changed and
|
||
> what the redirect URLs would become — review before committing.
|
||
|
||
---
|
||
|
||
### ── Onboard Sequence ─────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Step 1 — Pre-flight checks
|
||
# Both servers healthy (array started, Docker responding)
|
||
# Tailscale connected — HOST2 reachable via Tailscale IP
|
||
# unRAID version parity — both servers on compatible versions
|
||
# HOST2 auth containers exist (may be stopped — that's fine)
|
||
# → Any failure: abort, explain what failed, no changes made
|
||
#
|
||
# Step 2 — Reconfigure HOST2 WebUI templates
|
||
# For each container in PARTNERSHIP_AUTH_WEBUIS:
|
||
# Find template in /boot/config/plugins/dockerMan/templates-user/
|
||
# Update WebUI URL → http://[HOST1-TAILSCALE-IP]:PORT
|
||
# Container restart not needed — template change takes effect on next open
|
||
#
|
||
# Step 3 — Verify WebUI connectivity (if PARTNERSHIP_ONBOARD_VERIFY=true)
|
||
# curl each configured WebUI via HOST1's Tailscale IP
|
||
# Confirms Tailscale routing is working and the service is responding
|
||
# A redirect that can't be reached is worse than no redirect
|
||
#
|
||
# Step 4 — Write state files
|
||
# Write partnership_HOST1.db=ACTIVE locally
|
||
# SSH to HOST2 → push state, write partnership_HOST2.db=ACTIVE
|
||
# Reset offline counter on both sides
|
||
#
|
||
# Step 5 — Notify (if PARTNERSHIP_ONBOARD_NOTIFY=true)
|
||
# Notification to both servers: partnership active
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Configuration ────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master_host1.conf (or master_host2.conf for HOST2 side of the relationship)
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Containers whose WebUI URLs are reconfigured on onboard.
|
||
# Format: "ContainerName|WebUIPort"
|
||
# The port here is what gets redirected — matches the container's WebUI port.
|
||
#
|
||
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
|
||
"NginxProxyManager|81" # NPM admin interface — all proxy rules managed here
|
||
"Lldap-Gmer4Lfe|17170" # user directory — all user management here
|
||
"Authelia|9091" # SSO authentication — policy management here
|
||
"Authelia-Secondary|9092" # secondary Authelia instance
|
||
)
|
||
|
||
# Paths the mirror should collect during the offboard grace window.
|
||
# Files here are accessible to HOST2 for PARTNERSHIP_GRACE_HOURS after offboard.
|
||
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
|
||
# "/mnt/user/appdata-Failover/Jayred365-Emby" # HOST2's Emby backup on HOST1
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Always dry-run first — shows exact template changes and redirect URLs
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
partnership_manage.sh --onboard --dry-run
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Live onboard — owner (HOST1) only
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
partnership_manage.sh --onboard
|
||
|
||
# Verbose — show each pre-flight check and template change
|
||
partnership_manage.sh --onboard --log
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔴 --offboard — Clean Separation
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Either server can initiate offboard. The process differs slightly depending on who
|
||
runs it — but both paths end with both servers fully independent. The mirror leaves
|
||
with current auth config, working certs, and full control of its own services.
|
||
|
||
> **Always dry-run first.** Offboard stops the sync, reconfigures WebUIs, and
|
||
> removes Tailscale access. The dry-run shows exactly what would happen, in order,
|
||
> before any of it does.
|
||
|
||
---
|
||
|
||
### ── Owner-Initiated Offboard (HOST1 runs --offboard) ───────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# 1. 10-second countdown shown — Ctrl-C to abort if you changed your mind
|
||
#
|
||
# 2. Stop any running rsync
|
||
# rsync_stop.sh --rsync-only — stops the rsync but leaves container recovery
|
||
# to docker_watchdog.sh. Nothing else is interrupted.
|
||
#
|
||
# 3. Final sync
|
||
# Critical-Data pushed to HOST2 one last time — HOST2 leaves with current state
|
||
# Emby failover sync pushed — HOST2 has current watch states
|
||
# This is the cleanest possible state for HOST2 to inherit.
|
||
#
|
||
# 4. Reconfigure HOST2 WebUI templates → localhost
|
||
# SSH to HOST2 → update each auth container WebUI URL → http://localhost:PORT
|
||
# HOST2 operator now clicks NPM → gets HOST2's own NPM
|
||
# HOST2 manages its own auth from this point forward
|
||
#
|
||
# 5. Disable critical sync
|
||
# Write CRITICAL_RSYNC_ENABLED=false to master.conf
|
||
# Critical-Data sync stops at next 15-minute cycle
|
||
#
|
||
# 6. Write state files
|
||
# partnership_HOST1.db=INACTIVE
|
||
# SSH to HOST2 → partnership_HOST2.db=INACTIVE
|
||
#
|
||
# 7. Grace period — PARTNERSHIP_GRACE_HOURS (default 6 hours)
|
||
# HOST2's data on HOST1 remains accessible for this window
|
||
# HOST2 can collect anything it needs via Tailscale during this time
|
||
# Both access and Tailscale removal happen at the same moment (step 8)
|
||
#
|
||
# 8. Remove HOST2 from Tailscale tailnet
|
||
# API call to revoke HOST2's device — encrypted connection ends
|
||
# Backup access ends at the same time by design — no misleading window
|
||
#
|
||
# 9. Notify HOST2 — "offboard complete, access removed in Xh"
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Mirror-Initiated Offboard (HOST2 runs --offboard) ──────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# HOST2 can offboard at any time, even if HOST1 is temporarily unreachable.
|
||
# The deferred finalisation mechanism ensures HOST1 catches up automatically.
|
||
#
|
||
# 1. Reconfigure own WebUI templates → localhost
|
||
# All auth containers now point to HOST2's own services
|
||
# HOST2 manages its own auth immediately — no waiting for HOST1
|
||
#
|
||
# 2. Write partnership_HOST2.db=INACTIVE locally
|
||
#
|
||
# 3. SSH to HOST1 (if reachable) → push state file notification
|
||
# HOST1 reads INACTIVE → begins owner offboard sequence (steps 3-9 above)
|
||
# HOST1 runs final sync → sends current state to HOST2 one more time
|
||
# HOST1 handles Tailscale removal and grace period
|
||
#
|
||
# 4. If HOST1 unreachable — deferred finalisation
|
||
# HOST2 has already reconfigured to localhost — fully independent now
|
||
# STATE file written — HOST1 will read it on next --check when reachable
|
||
# Next time HOST1 runs --check and can reach HOST2: sees INACTIVE → finalises
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── What HOST2 Keeps After Offboard ────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# What HOST2 leaves with:
|
||
Current auth config ✓ final sync was clean — NPM rules, users, policies
|
||
Own DDNS + updater ✓ always was his, unaffected by partnership
|
||
Own Git mirror ✓ full ecosystem — git pull always ran on HOST2
|
||
Own anime source of truth ✓ Anime_Shows / Anime_Movies — always was his
|
||
Auth WebUIs → localhost ✓ managing own auth directly from this point
|
||
Full independence ✓ just stop the sync — ecosystem continues
|
||
Certs valid ~60-90 days ✓ cert_monitor.sh warns at 30 days — time to renew
|
||
|
||
# What HOST1 keeps after offboard:
|
||
His hardware ✓ unaffected
|
||
His media shares ✓ Movies, Tv_Shows, Music — his source of truth
|
||
His services ✓ running on HOST1 as always
|
||
His domain ✓ DNS unaffected
|
||
|
||
# What HOST1 loses:
|
||
Offsite backup ✗ was HOST2's hardware in a different location
|
||
Geographic redundancy ✗ cannot be scripted — that was the person
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# The arrangement was always mutual. Both parties provided something the scripts
|
||
# cannot replace — hardware on a different power utility in a different location.
|
||
# Everything else can be reconfigured. That cannot.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Available from either server — always dry-run first
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
partnership_manage.sh --offboard --dry-run # show the complete sequence, no changes
|
||
|
||
partnership_manage.sh --offboard # live offboard — 10s countdown before starting
|
||
partnership_manage.sh --offboard --log # verbose — show each step as it executes
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔄 --transfer — Flipping Ownership
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Transfers ownership from HOST1 to HOST2 (or back after a previous transfer). After
|
||
transfer, HOST2 becomes the auth source of truth and HOST1 receives syncs.
|
||
Only the current owner can initiate transfer.
|
||
|
||
> **This is the most significant operation in the lifecycle.** It reconfigures both
|
||
> servers, flips sync direction, and changes master.conf permanently on both sides.
|
||
> The confirmation string and health strike system exist to prevent accidents.
|
||
|
||
---
|
||
|
||
### ── Why This Is Protected ───────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Transfer involves:
|
||
# Modifying master.conf on both servers simultaneously
|
||
# Flipping sync direction (HOST1 → HOST2 becomes HOST2 → HOST1)
|
||
# Reconfiguring WebUI templates on both servers
|
||
# Making HOST2 the authoritative source for auth decisions
|
||
#
|
||
# A partial transfer — if interrupted mid-sequence — leaves both servers with
|
||
# different auth configurations and no clear source of truth. The health strike
|
||
# system prevents starting a transfer when either server is degraded.
|
||
# The confirmation string prevents accidental execution entirely.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Transfer Sequence ───────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# 1. Display current and future ownership clearly
|
||
# "Current: HOST1 → Mirror: HOST2"
|
||
# "After: HOST2 → Mirror: HOST1"
|
||
# No ambiguity about what is being confirmed.
|
||
#
|
||
# 2. Confirmation string check
|
||
# Must type exactly: i-understand-this-transfers-ownership
|
||
# Long and specific by design — cannot be typed accidentally.
|
||
#
|
||
# 3. Health strike system
|
||
# Both servers must pass PARTNERSHIP_TRANSFER_STRIKES consecutive health checks.
|
||
# Max PARTNERSHIP_TRANSFER_MAX_ATTEMPTS attempts (default 20) before aborting.
|
||
# Each strike: check both arrays started, Docker daemons responding, Tailscale connected.
|
||
# Protects against transferring during a degraded state.
|
||
#
|
||
# 4. Final sync in current direction
|
||
# Push Current-Data from current owner (HOST1) to current mirror (HOST2).
|
||
# New mirror (HOST1) starts with current state from the outgoing owner.
|
||
#
|
||
# 5. Reconfigure WebUI templates on HOST2
|
||
# HOST2 WebUIs → localhost (HOST2 now manages its own auth directly)
|
||
#
|
||
# 6. Reconfigure WebUI templates on HOST1
|
||
# HOST1 WebUIs → HOST2's Tailscale IP (HOST1 now redirects to HOST2)
|
||
#
|
||
# 7. Flip PARTNERSHIP_OWNER_HOST in master.conf on BOTH servers
|
||
# "HOST1" → "HOST2" written to both servers via SSH
|
||
#
|
||
# 8. Write updated state files on both servers
|
||
#
|
||
# 9. Notify both servers — transfer complete, new roles shown
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── After Transfer ───────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Before transfer: HOST1 = owner, HOST2 = mirror
|
||
# After transfer: HOST2 = owner, HOST1 = mirror
|
||
#
|
||
# HOST2 now:
|
||
# All auth changes made here directly
|
||
# Critical-Data pushed → to HOST1 every 15 minutes
|
||
# NPM, LLDAP, Authelia — HOST2's own instances are authoritative
|
||
# HOST1's WebUIs redirect to HOST2 via Tailscale
|
||
#
|
||
# HOST1 now:
|
||
# Receives Critical-Data sync from HOST2
|
||
# Auth WebUIs point to HOST2 — clicking NPM opens HOST2's NPM
|
||
# Never makes auth configuration changes directly
|
||
#
|
||
# To transfer back — run --transfer from HOST2 (the new owner)
|
||
# PARTNERSHIP_OWNER_HOST="HOST2" after first transfer, "HOST1" after second
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Dry run — ALWAYS first. Shows the full sequence, health check results,
|
||
# and what master.conf changes would be made on both servers.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
partnership_manage.sh --transfer --dry-run
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Live transfer — owner only — requires exact confirmation string
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
partnership_manage.sh --transfer --confirm=i-understand-this-transfers-ownership
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔁 --check — Automated State Monitor
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Called automatically by `critical_sync_maintenance.sh` every 15 minutes. Never run
|
||
manually. Silent when healthy — produces output only when state changes or thresholds
|
||
are hit.
|
||
|
||
```bash
|
||
# Called internally — do not run manually:
|
||
partnership_manage.sh --check --remote-seen # rsync succeeded, remote reachable
|
||
partnership_manage.sh --check --remote-unseen # rsync failed, remote unreachable
|
||
```
|
||
|
||
---
|
||
|
||
### ── What It Does Each Cycle ─────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# --remote-seen path (rsync succeeded → HOST2 was reachable):
|
||
# Reset offline counter to 0
|
||
# Update last_seen_remote timestamp
|
||
# SSH to read remote state file
|
||
# Both ACTIVE → silent ✅
|
||
# Remote INACTIVE → begin offboard finalisation from owner side
|
||
#
|
||
# --remote-unseen path (rsync failed → HOST2 unreachable):
|
||
# Increment offline counter by 1
|
||
# Counter × 15min = approximate offline duration
|
||
# At PARTNERSHIP_OFFLINE_THRESHOLD days: auto-offboard begins
|
||
# Notify when counter hits threshold — not on every missed cycle
|
||
#
|
||
# Why the rsync outcome drives this:
|
||
# The rsync that runs before --check provides a reliable connectivity signal.
|
||
# If rsync to HOST2 succeeded, HOST2 is reachable — reset counter.
|
||
# If rsync failed, HOST2 may be unreachable — increment counter.
|
||
# No separate ping needed — the sync attempt is the connectivity test.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Auto-Offboard ────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# After this many days of consecutive missed sync cycles, auto-offboard begins.
|
||
# Works both directions independently — neither server needs the other to act.
|
||
#
|
||
PARTNERSHIP_OFFLINE_THRESHOLD=30 # days — each missed cycle = ~15min
|
||
#
|
||
# HOST1 sees HOST2 offline for 30 days:
|
||
# AM_OWNER → full owner-initiated offboard
|
||
# Tailscale removal via API (API works regardless of HOST2 reachability)
|
||
# Critical sync disabled
|
||
# State: INACTIVE
|
||
#
|
||
# HOST2 sees HOST1 offline for 30 days:
|
||
# AM_MIRROR → mirror-initiated offboard
|
||
# Reconfigure own WebUIs → localhost
|
||
# Write HOST2.db=INACTIVE
|
||
# Fully independent immediately
|
||
# When HOST1 eventually comes back → reads INACTIVE → cleans its side
|
||
#
|
||
# 30 days covers:
|
||
# Extended hardware failure (people do fix things eventually)
|
||
# Someone moving and being offline for weeks
|
||
# But not: a permanent departure
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ ROLE-BASED ACCESS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Which server can run which mode:
|
||
#
|
||
HOST1 (owner) HOST2 (mirror)
|
||
--onboard ✅ ❌ ownership is granted, not taken
|
||
--offboard ✅ ✅ clean exit available to both parties
|
||
--transfer ✅ ❌ owner only — mirror cannot self-promote
|
||
--check ✅ ✅ both servers monitor state
|
||
--status ✅ ✅ status is always available
|
||
#
|
||
# AM_OWNER / AM_MIRROR flags set by detect_hosts() from PARTNERSHIP_OWNER_HOST.
|
||
# All routing decisions use these flags — no hostname string comparisons.
|
||
# Mirror attempting --onboard or --transfer: blocked with a clear error message.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ INTEGRATION WITH CRITICAL SYNC ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Partnership check runs inside critical_sync_maintenance.sh — the 15-minute
|
||
# orchestrator. The rsync outcome directly informs the partnership check.
|
||
#
|
||
# Schedule: */15 * * * *
|
||
#
|
||
# Execution order in critical_sync_maintenance.sh:
|
||
# 1. Critical-Data rsync → auth stack sync
|
||
# 2. emby-failover rsync → dirty Emby sync
|
||
# 3. partnership --check → state check, informed by rsync results
|
||
# → --remote-seen if rsync succeeded (HOST2 reachable)
|
||
# → --remote-unseen if rsync failed (HOST2 unreachable)
|
||
#
|
||
# When critical sync is disabled (CRITICAL_RSYNC_ENABLED=false):
|
||
# partnership --check is skipped automatically
|
||
# Offline counter does not increment from a sync that wasn't attempted
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
# ── Partnership Gate ───────────────────────────────────────────────────────
|
||
PARTNERSHIP_ENABLED=false # set true once both servers are configured
|
||
PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer
|
||
|
||
# ── Tailscale Removal ──────────────────────────────────────────────────────
|
||
PARTNERSHIP_REMOVE_TAILSCALE=true # remove mirror from tailnet on offboard
|
||
TAILSCALE_API_KEY="" # tskey-api-... from tailscale.com/admin/settings/keys
|
||
TAILSCALE_TAILNET="" # your tailnet name (e.g. example.github)
|
||
# required when PARTNERSHIP_REMOVE_TAILSCALE=true
|
||
|
||
# ── Timing ─────────────────────────────────────────────────────────────────
|
||
PARTNERSHIP_GRACE_HOURS=6 # hours before Tailscale removal after offboard
|
||
# backup access also expires at this time — by design
|
||
PARTNERSHIP_OFFLINE_THRESHOLD=30 # days unreachable before auto-offboard (both servers)
|
||
PARTNERSHIP_SYNC_INTERVAL=15 # informational — actual schedule in cron
|
||
|
||
# ── Transfer Safety ────────────────────────────────────────────────────────
|
||
PARTNERSHIP_TRANSFER_CONFIRM="i-understand-this-transfers-ownership"
|
||
# must type this exactly to proceed — long by design
|
||
PARTNERSHIP_TRANSFER_STRIKES=3 # consecutive health checks required before transfer
|
||
PARTNERSHIP_TRANSFER_MAX_ATTEMPTS=20 # max health attempts before giving up
|
||
|
||
# ── Onboard Behaviour ──────────────────────────────────────────────────────
|
||
PARTNERSHIP_ONBOARD_VERIFY=true # curl each WebUI after reconfigure to verify routing
|
||
PARTNERSHIP_ONBOARD_NOTIFY=true # notify both servers on successful onboard
|
||
|
||
# master_host1.conf / master_host2.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
# Containers whose WebUI URLs are reconfigured on onboard/offboard
|
||
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
|
||
"NginxProxyManager|81"
|
||
"Lldap-Gmer4Lfe|17170"
|
||
"Authelia|9091"
|
||
"Authelia-Secondary|9092"
|
||
)
|
||
|
||
# Paths accessible to HOST2 during grace window after offboard
|
||
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
|
||
# "/mnt/user/appdata-Failover/Jayred365-Emby"
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ INITIAL SETUP REQUIREMENTS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Before `--onboard` can run, five things must be in place:
|
||
|
||
---
|
||
|
||
### 1. Tailscale Connected on Both Servers
|
||
|
||
```bash
|
||
# Verify HOST2 is visible from HOST1:
|
||
tailscale ip -4 unRAID-Jayred365 # should return HOST2's Tailscale IP
|
||
|
||
# Verify HOST1 is visible from HOST2:
|
||
tailscale ip -4 unRAID-Gmer4Lfe # should return HOST1's Tailscale IP
|
||
|
||
# Verify reachability:
|
||
tailscale ping unRAID-Jayred365
|
||
```
|
||
|
||
HOST1 and HOST2 hostnames must match their exact Tailscale device names — already
|
||
enforced by the ecosystem since `REMOTE_SERVER_NAME` is used everywhere.
|
||
|
||
---
|
||
|
||
### 2. SSH Keys — Passwordless Both Directions
|
||
|
||
```bash
|
||
# From HOST1 — should print HOST2's hostname without a password prompt:
|
||
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"
|
||
|
||
# From HOST2 — should print HOST1's hostname without a password prompt:
|
||
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"
|
||
```
|
||
|
||
---
|
||
|
||
### 3. Auth Containers Exist on HOST2
|
||
|
||
```bash
|
||
# Containers must exist — they can be stopped:
|
||
ssh root@[HOST2-ip] "docker inspect NginxProxyManager --format '{{.State.Status}}'"
|
||
# Expected: created, exited, or running — NOT "no such container"
|
||
|
||
# Template files must exist for WebUI reconfiguration:
|
||
ssh root@[HOST2-ip] "ls /boot/config/plugins/dockerMan/templates-user/"
|
||
```
|
||
|
||
---
|
||
|
||
### 4. Critical-Data rsync Profile Configured
|
||
|
||
```bash
|
||
# master.conf — CRITICAL_SYNC_SHARES must include the auth stack path:
|
||
CRITICAL_SYNC_SHARES=(
|
||
"/mnt/user/appdata-Failover/Critical-Data" # auth config — NPM, Authelia, LLDAP, certs
|
||
)
|
||
# This is what keeps HOST2 current after onboard
|
||
```
|
||
|
||
---
|
||
|
||
### 5. Tailscale API Key (if removing from tailnet on offboard)
|
||
|
||
```bash
|
||
# master.conf
|
||
PARTNERSHIP_REMOVE_TAILSCALE=true
|
||
TAILSCALE_API_KEY="tskey-api-..." # from tailscale.com/admin/settings/keys
|
||
TAILSCALE_TAILNET="example.github" # your tailnet name
|
||
|
||
# Required scope: Devices write
|
||
# Generate at: https://login.tailscale.com/admin/settings/keys
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
---
|
||
|
||
### 🔴 WebUI Still Pointing to Old IP After Onboard
|
||
|
||
```
|
||
Check template was found and modified:
|
||
partnership_manage.sh --status shows current WebUI redirect state
|
||
ssh HOST2 "cat /boot/config/plugins/dockerMan/templates-user/NginxProxyManager.xml"
|
||
should show HOST1's Tailscale IP
|
||
|
||
Some containers need a restart to pick up new WebUI URL:
|
||
docker restart NginxProxyManager on HOST2 — picks up template change
|
||
|
||
Verify Tailscale routing is working:
|
||
curl http://[HOST1-tailscale-ip]:81 from HOST2 — should reach HOST1's NPM
|
||
```
|
||
|
||
---
|
||
|
||
### 🔴 Transfer Health Checks Failing
|
||
|
||
```
|
||
Both arrays must be fully started:
|
||
ls /mnt/user should show share directories on both servers
|
||
|
||
Both Docker daemons responding:
|
||
docker ps should return a list, not hang
|
||
|
||
Tailscale connected on both servers:
|
||
tailscale status should show the remote peer as online
|
||
|
||
Max attempts before giving up: PARTNERSHIP_TRANSFER_MAX_ATTEMPTS (default 20)
|
||
Each attempt waits for next health window — may take several minutes total
|
||
```
|
||
|
||
---
|
||
|
||
### 🔴 Mirror's Domain Broken After Offboard
|
||
|
||
```
|
||
Auth WebUIs reconfigured to localhost — mirror manages auth directly now.
|
||
The containers are running and functional — they just no longer redirect to HOST1.
|
||
|
||
Verify auth containers running:
|
||
docker ps | grep -E "NPM|Authelia|LLDAP"
|
||
|
||
Check cert expiry — certs won't auto-renew anymore via HOST1:
|
||
cert_monitor.sh --dry-run shows days remaining on each domain's cert
|
||
|
||
Set up cert auto-renewal on HOST2 independently if not already configured.
|
||
```
|
||
|
||
---
|
||
|
||
### 🔴 State Mismatch Between Servers
|
||
|
||
```
|
||
Run --status on both servers to see both state files:
|
||
partnership_manage.sh --status on HOST1
|
||
partnership_manage.sh --status on HOST2
|
||
|
||
If one shows ACTIVE and other shows INACTIVE:
|
||
The INACTIVE side has already offboarded
|
||
Run --offboard on the ACTIVE side to sync the state
|
||
|
||
If HOST2 unreachable:
|
||
HOST2 will self-resolve on next --check when it becomes reachable
|
||
HOST1 reads HOST2's INACTIVE state → finalises from owner side automatically
|
||
```
|
||
|
||
---
|
||
|
||
### 🔴 Auto-Offboard Triggered Unexpectedly
|
||
|
||
```
|
||
Check the offline counter:
|
||
cat /boot/config/partnership_offline_days.db
|
||
|
||
Extended Tailscale connectivity issue may have incremented the counter:
|
||
tailscale status verify HOST2 appears in HOST1's peer list
|
||
|
||
If partnership should continue — re-onboard:
|
||
partnership_manage.sh --onboard --dry-run # verify setup is correct
|
||
partnership_manage.sh --onboard # re-establish partnership
|
||
|
||
Counter resets to 0 on successful --onboard.
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ DESIGN NOTES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```
|
||
Why HOST1/HOST2 instead of PARTNERSHIP_OWNER/MIRROR?
|
||
HOST1 and HOST2 are already defined in master.conf — SSH keys, Tailscale names,
|
||
all connection details. A single PARTNERSHIP_OWNER_HOST var flips ownership.
|
||
Duplicating everything as PARTNERSHIP_OWNER and PARTNERSHIP_MIRROR would require
|
||
maintaining identical values in two places. One source of truth for the host
|
||
definition, one var to flip direction.
|
||
|
||
Why does transfer require a confirmation string?
|
||
Same philosophy as --i-know-what-im-doing in the arr cleanup scripts.
|
||
Transfer touches master.conf on both servers, reconfigures WebUIs, and flips
|
||
sync direction. The confirmation string makes accidental execution impossible,
|
||
not just unlikely.
|
||
|
||
Why do Tailscale removal and backup access expire at the same time?
|
||
If the mirror can't reach the owner's server via Tailscale, the backup is also
|
||
unreachable. PARTNERSHIP_GRACE_HOURS controls both — one var, consistent behaviour,
|
||
no misleading "data available for 30 days" when access is gone in 6 hours.
|
||
|
||
Why does either server auto-offboard after 30 days offline?
|
||
30 consecutive days of missed sync cycles means the relationship has effectively
|
||
ended regardless of intent. Auto-offboard makes the state official. Each server
|
||
acts independently — they don't need to coordinate to offboard.
|
||
|
||
Why does WebUI reconfiguration use the unRAID template system?
|
||
unRAID stores WebUI URLs in /boot/config/plugins/dockerMan/templates-user/.
|
||
Editing templates persists across container restarts and array reboots.
|
||
Direct Docker label manipulation is lost on next container recreation.
|
||
``` |