padded partnership set up
This commit is contained in:
@@ -0,0 +1,438 @@
|
||||
# Partnership
|
||||
|
||||
Manages the relationship lifecycle between two unRAID servers. Handles onboarding a new mirror, clean separation when someone wants to leave, and transferring ownership when the arrangement changes.
|
||||
|
||||
> This script exists because the partnership between two servers has a lifecycle — and that lifecycle deserves the same engineering discipline as everything else in the ecosystem. A clean exit should be as easy as a clean setup.
|
||||
|
||||
---
|
||||
|
||||
## The Relationship Model
|
||||
|
||||
Two servers. One owns the shared services. One mirrors them.
|
||||
|
||||
```
|
||||
HOST1 (owner) — source of truth
|
||||
Auth stack ← all changes made here
|
||||
NPM proxy rules ← created here, mirrored to mirror
|
||||
Certs ← renewed here, mirrored to mirror
|
||||
Emby ← runs here, dirty-synced every 15min
|
||||
Master.conf ← source of truth, git push propagates
|
||||
|
||||
HOST2 (mirror) — warm copy
|
||||
Auth containers ← running, serving his domain
|
||||
NPM ← running owner's config
|
||||
Certs ← current, mirrored every 15min
|
||||
WebUIs ← click any auth container → opens owner's WebUI via Tailscale
|
||||
Never makes changes directly ← overwritten on next sync
|
||||
```
|
||||
|
||||
**HOST1 is always the owner. HOST2 is always the mirror.**
|
||||
`PARTNERSHIP_OWNER_HOST="HOST1"` in Master.conf — flips to `"HOST2"` only after `--transfer`.
|
||||
Everything derives from HOST1/HOST2 — no duplicate IP or key configuration needed.
|
||||
|
||||
**The mirror never needs to think about auth management.** His Docker UI opens, he clicks an auth container, lands on the owner's WebUI automatically via Tailscale. Changes made there sync to his server in 15 minutes. He benefits from every improvement the owner makes without doing anything.
|
||||
|
||||
**The owner never needs to think about the mirror's server.** Everything propagates automatically. Critical-Data syncs every 15 minutes via `critical_sync_maintenance.sh`. Certs stay current. Config stays consistent.
|
||||
|
||||
---
|
||||
|
||||
## What Makes It Work
|
||||
|
||||
Three things make the mirror transparent to end users:
|
||||
|
||||
**1. Auth stack warm on both servers**
|
||||
|
||||
NPM, LLDAP, Authelia, certs — all running on both servers simultaneously. When traffic hits the mirror's domain, auth is already ready. No cold start, no broken auth window.
|
||||
|
||||
**2. WebUIs pointing to owner**
|
||||
|
||||
Every auth container on the mirror has its WebUI URL configured to point at the owner's Tailscale IP. From the mirror operator's perspective: click container, get owner's UI. From the owner's perspective: one place to manage everything.
|
||||
|
||||
**3. Tailscale-only communication**
|
||||
|
||||
All traffic between servers goes through Tailscale. Encrypted, no open ports, no VPN configuration. The WebUI redirect works because Tailscale keeps both servers permanently connected. HOST1 and HOST2 hostnames must match their exact Tailscale device names — already enforced by the ecosystem.
|
||||
|
||||
---
|
||||
|
||||
## State Files
|
||||
|
||||
Partnership state is tracked in two files on `/boot/config` — survives reboots, available before the array starts, minimal flash wear.
|
||||
|
||||
```
|
||||
/boot/config/partnership_HOST1.db ← HOST1 writes only
|
||||
/boot/config/partnership_HOST2.db ← HOST2 writes only
|
||||
```
|
||||
|
||||
Each server writes **only its own file**. State propagates via SSH — no rsync needed. `critical_sync_maintenance.sh` calls `partnership_manage.sh --check` every 15 minutes, which SSHes to read the remote file and act on any changes.
|
||||
|
||||
**This is how deferred offboard works:** HOST2 offboards while HOST1 is temporarily unreachable → HOST2 writes its state file → next time HOST1 can reach HOST2 → reads HOST2's state → finalises from its side automatically.
|
||||
|
||||
A per-server offline counter (`/boot/config/partnership_offline_days.db`) tracks consecutive missed 15-minute cycles. After `PARTNERSHIP_OFFLINE_THRESHOLD` days either server is unreachable, both independently auto-offboard.
|
||||
|
||||
---
|
||||
|
||||
## The Lifecycle
|
||||
|
||||
### Onboard — Setting Up a New Mirror
|
||||
|
||||
Run `--onboard` from HOST1 (owner) to establish the relationship:
|
||||
|
||||
```bash
|
||||
# Always dry-run first
|
||||
partnership_manage.sh --onboard --dry-run
|
||||
|
||||
# Live onboard
|
||||
partnership_manage.sh --onboard
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. Pre-flight — both servers healthy, Tailscale connected
|
||||
2. SSH to HOST2 — reconfigure each auth container's WebUI URL to HOST1's Tailscale IP
|
||||
3. Verify WebUI connectivity — curl each WebUI, confirm reachable
|
||||
4. Write state files — `partnership_HOST1.db=ACTIVE`, push to HOST2
|
||||
5. Reset offline counter
|
||||
6. Notify both servers — partnership active
|
||||
|
||||
**After onboard:**
|
||||
```
|
||||
Mirror opens Docker UI
|
||||
→ clicks NginxProxyManager
|
||||
→ lands on owner's NPM WebUI
|
||||
→ makes changes there
|
||||
→ 15min later synced to his server
|
||||
→ he never touched his own NPM
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Offboard — Clean Separation
|
||||
|
||||
**Either server can initiate offboard.** The process differs slightly depending on who runs it.
|
||||
|
||||
```bash
|
||||
# Always dry-run first
|
||||
partnership_manage.sh --offboard --dry-run
|
||||
|
||||
# Live offboard
|
||||
partnership_manage.sh --offboard
|
||||
```
|
||||
|
||||
#### HOST1 (owner) initiates offboard:
|
||||
|
||||
1. 10 second countdown
|
||||
2. Stop any running rsync (`rsync_stop.sh --rsync-only`)
|
||||
3. Final sync — Critical-Data and Emby pushed one last time
|
||||
4. SSH to HOST2 — reconfigure auth WebUIs back to localhost
|
||||
5. Disable `CRITICAL_RSYNC_ENABLED=false` in Master.conf
|
||||
6. Write `partnership_HOST1.db=INACTIVE`, push to HOST2
|
||||
7. Wait `PARTNERSHIP_GRACE_HOURS` (6hr) — mirror has access to collect anything
|
||||
8. Remove HOST2 from Tailscale tailnet
|
||||
9. Notify HOST2
|
||||
|
||||
#### HOST2 (mirror) initiates offboard:
|
||||
|
||||
1. Reconfigure own auth WebUIs → localhost
|
||||
2. Write `partnership_HOST2.db=INACTIVE`
|
||||
3. SSH to HOST1 — push state file (or write locally if HOST1 unreachable)
|
||||
4. Notify HOST1 — "mirror has requested offboard"
|
||||
|
||||
**HOST1 finalises on next `--check` cycle:**
|
||||
|
||||
5. Reads HOST2's state file — sees INACTIVE
|
||||
6. Runs final sync — HOST2 leaves with current state
|
||||
7. Disables critical sync
|
||||
8. Writes `partnership_HOST1.db=INACTIVE`
|
||||
9. Waits grace period → removes HOST2 from Tailscale
|
||||
|
||||
#### What HOST2 leaves with:
|
||||
|
||||
```
|
||||
Current auth config ✓ — final sync was clean
|
||||
His own DDNS + updater ✓ — always was his, unaffected
|
||||
His own Git mirror ✓ — full ecosystem, always mirrored
|
||||
His anime source of truth ✓ — always was his
|
||||
Auth WebUIs → localhost ✓ — managing his own auth now
|
||||
Full independence ✓ — just stop the sync
|
||||
Certs valid ~60-90 days ✓ — cert_monitor.sh warns at 30 days
|
||||
```
|
||||
|
||||
**Tailscale access and backup window both expire at the same time** (`PARTNERSHIP_GRACE_HOURS=6`). Keeping backups accessible beyond Tailscale removal is meaningless — they expire together by design.
|
||||
|
||||
---
|
||||
|
||||
### Auto-Offboard — 30 Day Offline Threshold
|
||||
|
||||
If either server is unreachable for `PARTNERSHIP_OFFLINE_THRESHOLD` days (default: 30), the other server auto-offboards. Works both directions:
|
||||
|
||||
```
|
||||
HOST1 sees HOST2 offline 30 days:
|
||||
AM_OWNER → full offboard from owner side
|
||||
Tailscale removal ✅ (API call works regardless)
|
||||
|
||||
HOST2 sees HOST1 offline 30 days:
|
||||
AM_MIRROR → mirror offboard
|
||||
Reconfigures own WebUIs → localhost ✅
|
||||
Writes HOST2.db=INACTIVE ✅
|
||||
Fully independent immediately ✅
|
||||
When HOST1 comes back → sees INACTIVE → cleans its side ✅
|
||||
```
|
||||
|
||||
This handles the "partner disappeared" scenario without manual intervention. 30 days is long enough to cover extended outages, short enough that you're not waiting forever for resolution.
|
||||
|
||||
---
|
||||
|
||||
### Transfer — Flipping Ownership
|
||||
|
||||
Run `--transfer` from HOST1 when ownership needs to change:
|
||||
|
||||
```bash
|
||||
# Always dry-run first — this is significant
|
||||
partnership_manage.sh --transfer --dry-run
|
||||
|
||||
# Live transfer — requires explicit confirmation string
|
||||
partnership_manage.sh --transfer --confirm=i-understand-this-transfers-ownership
|
||||
```
|
||||
|
||||
**The confirmation string is long and specific by design.** You cannot type it accidentally.
|
||||
|
||||
**What happens:**
|
||||
1. Display current and future ownership clearly
|
||||
2. Confirmation string check
|
||||
3. Health strike system — both servers must pass `PARTNERSHIP_TRANSFER_STRIKES` consecutive checks (max `PARTNERSHIP_TRANSFER_MAX_ATTEMPTS` attempts before giving up)
|
||||
4. Final sync in current direction — new mirror leaves with current state
|
||||
5. Reconfigure HOST2 WebUIs → new owner's Tailscale IP
|
||||
6. Reconfigure HOST1 WebUIs → localhost (now manages directly)
|
||||
7. Flip `PARTNERSHIP_OWNER_HOST` in Master.conf on both servers
|
||||
8. Write updated state files
|
||||
9. Notify both servers
|
||||
|
||||
**After transfer:**
|
||||
```
|
||||
Before: HOST1 = owner, HOST2 = mirror
|
||||
After: HOST2 = owner, HOST1 = mirror
|
||||
|
||||
HOST2 now:
|
||||
Makes all auth changes ← source of truth
|
||||
Pushes Critical-Data sync ← every 15min
|
||||
Manages NPM, certs, LLDAP ← directly
|
||||
|
||||
HOST1 now:
|
||||
Receives sync ← warm mirror
|
||||
WebUIs point to HOST2 ← transparent
|
||||
Never makes changes directly
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Check — Automated State Monitor
|
||||
|
||||
Called automatically by `critical_sync_maintenance.sh` every 15 minutes. Never run manually.
|
||||
|
||||
```bash
|
||||
# Called internally — do not run manually
|
||||
partnership_manage.sh --check --remote-seen # rsync succeeded, HOST2 reachable
|
||||
partnership_manage.sh --check --remote-unseen # rsync failed, HOST2 unreachable
|
||||
```
|
||||
|
||||
**What it does each cycle:**
|
||||
- `--remote-seen`: resets offline counter, updates `last_seen_remote` timestamp
|
||||
- `--remote-unseen`: increments offline counter, checks auto-offboard threshold
|
||||
- SSHes to remote, reads remote state file
|
||||
- Both ACTIVE → silent, healthy ✅
|
||||
- Remote INACTIVE → owner finalises offboard, mirror cleans up
|
||||
- Threshold exceeded → auto-offboard
|
||||
|
||||
---
|
||||
|
||||
## Role-Based Access
|
||||
|
||||
```
|
||||
HOST1 (owner): --onboard ✅ --offboard ✅ --transfer ✅ --status ✅
|
||||
HOST2 (mirror): --onboard ❌ --offboard ✅ --transfer ❌ --status ✅
|
||||
```
|
||||
|
||||
HOST2 is blocked from onboard and transfer by design — ownership is granted not taken. Either server can initiate offboard — clean exit is always available to both parties.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
All configuration in `Master.conf` under `── PARTNERSHIP ──`, placed immediately after HOST CONFIGURATION.
|
||||
|
||||
```bash
|
||||
PARTNERSHIP_ENABLED=false
|
||||
PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer
|
||||
|
||||
# Auth containers reconfigured on onboard/offboard
|
||||
# Format: "ContainerName|WebUIPort"
|
||||
PARTNERSHIP_AUTH_WEBUIS=(
|
||||
"NginxProxyManager|81"
|
||||
"Lldap-Gmer4Lfe|17170"
|
||||
"Authelia|9091"
|
||||
"Authelia-Secondary|9092"
|
||||
)
|
||||
|
||||
# Paths on owner that mirror should collect during grace window
|
||||
PARTNERSHIP_MIRROR_BACKUPS=(
|
||||
# "/mnt/user/appdata-Failover/Jayred365-Emby"
|
||||
)
|
||||
|
||||
# Timing
|
||||
PARTNERSHIP_GRACE_HOURS=6 # hours after offboard before Tailscale removal
|
||||
# backup access also expires at this time
|
||||
PARTNERSHIP_OFFLINE_THRESHOLD=30 # days unreachable before auto-offboard (both directions)
|
||||
PARTNERSHIP_REMOVE_TAILSCALE=true # remove mirror from Tailscale on offboard
|
||||
|
||||
# Tailscale API — required for PARTNERSHIP_REMOVE_TAILSCALE=true
|
||||
TAILSCALE_API_KEY="" # tskey-api-...
|
||||
TAILSCALE_TAILNET="" # your tailnet name
|
||||
|
||||
# Transfer safety
|
||||
PARTNERSHIP_TRANSFER_CONFIRM="i-understand-this-transfers-ownership"
|
||||
PARTNERSHIP_TRANSFER_STRIKES=3
|
||||
PARTNERSHIP_TRANSFER_MAX_ATTEMPTS=20
|
||||
|
||||
# Onboard
|
||||
PARTNERSHIP_ONBOARD_VERIFY=true
|
||||
PARTNERSHIP_ONBOARD_NOTIFY=true
|
||||
PARTNERSHIP_SYNC_INTERVAL=15 # minutes — informational, actual schedule in cron
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## critical_sync_maintenance.sh
|
||||
|
||||
The partnership check runs inside `critical_sync_maintenance.sh` — the 15-minute orchestrator that also handles Critical-Data and Emby failover syncs.
|
||||
|
||||
```
|
||||
Orchestrators/critical_sync_maintenance.sh
|
||||
Schedule: */15 * * * *
|
||||
|
||||
Execution order:
|
||||
1. Critical-Data rsync ← auth stack sync (15min)
|
||||
2. emby-failover rsync ← dirty Emby sync (15min)
|
||||
3. partnership --check ← state check, informed by rsync outcome
|
||||
```
|
||||
|
||||
The rsync outcome directly informs the partnership check — if rsync succeeded, HOST2 was reachable and the offline counter resets. If rsync failed, the counter increments.
|
||||
|
||||
Controlled by `CRITICAL_RSYNC_ENABLED` in Master.conf — set to `false` automatically on offboard.
|
||||
|
||||
---
|
||||
|
||||
## Initial Setup Requirements
|
||||
|
||||
Before `--onboard` can run:
|
||||
|
||||
**1. Tailscale connected on both servers**
|
||||
|
||||
HOST1 and HOST2 hostnames must match their exact Tailscale device names — already enforced by the ecosystem since `HOST1` and `HOST2` in Master.conf are used for Tailscale IP resolution everywhere.
|
||||
|
||||
**2. SSH keys configured — passwordless both directions**
|
||||
|
||||
```bash
|
||||
ssh -i /root/.ssh/your_key root@[HOST2-tailscale-ip] "hostname"
|
||||
```
|
||||
|
||||
**3. Auth containers exist on HOST2**
|
||||
|
||||
Containers must exist before onboard. They can be stopped — onboard only reconfigures WebUI URLs. Critical-Data sync brings them live config.
|
||||
|
||||
**4. Critical-Data rsync profile configured**
|
||||
|
||||
The `critical-data` rsync profile must be set up in Master.conf. This is what keeps the mirror current after onboard. Configured in `CRITICAL_SYNC_SHARES`.
|
||||
|
||||
**5. Tailscale API key configured** (if `PARTNERSHIP_REMOVE_TAILSCALE=true`)
|
||||
|
||||
```
|
||||
https://login.tailscale.com/admin/settings/keys
|
||||
Scope: Devices write
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Independence — Always One Stop Away
|
||||
|
||||
The partnership is designed with a clean exit built in from day one.
|
||||
|
||||
**To stop the relationship — from either server:**
|
||||
```bash
|
||||
partnership_manage.sh --offboard
|
||||
```
|
||||
|
||||
That's it. The mirror's containers keep running. His domain keeps working. His DDNS keeps pointing to him. His git repo has the full ecosystem. His certs run for another 60-90 days.
|
||||
|
||||
**What the owner keeps after separation:**
|
||||
```
|
||||
His hardware ← always was his
|
||||
His media shares ← his source of truth shares stay his
|
||||
His services ← unaffected
|
||||
His domain ← unaffected
|
||||
His Emby users ← continue as before
|
||||
```
|
||||
|
||||
**What the owner loses:**
|
||||
```
|
||||
Offsite backup ← was the mirror's hardware
|
||||
Geographic redundancy ← can't script that, it was the person
|
||||
```
|
||||
|
||||
The arrangement was always mutual. Both parties contributed something the scripts couldn't provide — hardware in a different location on a different power utility. That's irreplaceable. Everything else in the ecosystem can be reconfigured.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**WebUI still pointing to old IP after onboard:**
|
||||
- Check template was found: `partnership_manage.sh --status`
|
||||
- Some containers may need a restart to pick up the new WebUI URL
|
||||
- Verify with `--status` on both servers
|
||||
|
||||
**Transfer health checks failing:**
|
||||
- Both arrays must be fully started
|
||||
- Both Docker daemons must be responding
|
||||
- Tailscale must be connected on both servers
|
||||
- Max attempts: `PARTNERSHIP_TRANSFER_MAX_ATTEMPTS` (default 20) before giving up
|
||||
|
||||
**Mirror's domain broken after offboard:**
|
||||
- Auth WebUIs reconfigured to localhost — mirror manages auth directly now
|
||||
- Verify auth containers are running: `docker ps`
|
||||
- Check cert expiry: run `cert_monitor.sh`
|
||||
|
||||
**State mismatch between servers:**
|
||||
- Run `--status` on both servers to see both state files
|
||||
- If one shows ACTIVE and other INACTIVE — run `--offboard` to resolve
|
||||
- If HOST2 unreachable — HOST2 will self-resolve on next `--check` when reachable
|
||||
|
||||
**Auto-offboard triggered unexpectedly:**
|
||||
- Check `/boot/config/partnership_offline_days.db` for counter value
|
||||
- Extended outage or Tailscale connectivity issue may have incremented counter
|
||||
- Re-onboard if partnership should continue: `--onboard`
|
||||
|
||||
---
|
||||
|
||||
## Design Notes
|
||||
|
||||
**Why HOST1/HOST2 instead of PARTNERSHIP_OWNER/MIRROR?**
|
||||
|
||||
HOST1 and HOST2 are already defined in Master.conf with SSH keys, Tailscale names, and all connection details. Duplicating them as PARTNERSHIP_OWNER and PARTNERSHIP_MIRROR would require maintaining the same values twice. A single `PARTNERSHIP_OWNER_HOST` var flips ownership direction — everything else derives from the existing HOST1/HOST2 configuration.
|
||||
|
||||
**Why does transfer require a confirmation string?**
|
||||
|
||||
Transferring ownership touches Master.conf on both servers, reconfigures WebUIs, and flips sync direction. The confirmation string is the same philosophy as `--i-know-what-im-doing` in the arr cleanup scripts — make accidental execution impossible, not just unlikely.
|
||||
|
||||
**Why do Tailscale removal and backup access expire at the same time?**
|
||||
|
||||
Keeping backups accessible beyond Tailscale removal is meaningless — if the mirror can't reach the owner's server via Tailscale, he can't access the backups anyway. `PARTNERSHIP_GRACE_HOURS` controls both simultaneously. One var, consistent behavior, no misleading "30 days available" when access is gone in 6 hours.
|
||||
|
||||
**Why does either server auto-offboard after 30 days offline?**
|
||||
|
||||
30 consecutive days of missed sync cycles means the relationship has effectively ended regardless of intent. The auto-offboard makes the state official and cleans up both sides without requiring manual intervention from a server that may genuinely be gone. Each server acts independently — they don't need to coordinate to offboard.
|
||||
|
||||
**Why does WebUI reconfiguration use the unRAID template system?**
|
||||
|
||||
unRAID stores container WebUI URLs in `/boot/config/plugins/dockerMan/templates-user/`. Editing templates is the correct way to change WebUI URLs — it persists across container restarts and array reboots. Direct Docker label manipulation would be lost on the next container recreation.
|
||||
|
||||
**Why is Tailscale WebUI left on the mirror's local server?**
|
||||
|
||||
Tailscale WebUI shows that server's network state from its own perspective. When diagnosing connectivity issues between the servers, you need to see the mirror's network view — not the owner's. It's the one WebUI that genuinely needs to stay local.
|
||||
Reference in New Issue
Block a user