Gmer4Lfe
5e4f510a42
critical_sync_maintenance.sh runs every 30 minutes (*/30 cron). Three references in README.md and Partnership/README-Partnership.md still said 15 minutes from an earlier schedule.
193 lines
12 KiB
Markdown
193 lines
12 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 🤝 PARTNERSHIP
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Managed lifecycle for a two-server partnership — setup, ongoing operation,
|
|
and clean separation.** One server owns the shared services. The other mirrors
|
|
them and benefits from them. Every phase of the relationship has the same
|
|
engineering discipline as the rest of the ecosystem.
|
|
|
|
> **This folder exists because a clean exit should be as easy as a clean setup.**
|
|
> The partnership is not a permanent commitment. `--offboard` works from either
|
|
> server at any time. Everything the mirror needs to run independently is already
|
|
> there. The only thing that stops on separation is the sync — and that's intentional.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### 🔴 Two Servers, One Auth Stack, No Clean Way to Share It
|
|
|
|
The auth stack — NginxProxyManager, LLDAP, Authelia, MariaDB, Redis — runs on
|
|
HOST1. HOST2 serves its own domain to its own household. It needs its own auth.
|
|
But maintaining two independent auth stacks means double the work: two places to
|
|
add users, two places to update proxy rules, two places to renew certs, two
|
|
configurations that inevitably drift apart. One change on HOST1 has to be manually
|
|
replicated to HOST2 — or it isn't, and the configurations diverge silently.
|
|
|
|
The real cost isn't the initial setup. It's the maintenance burden that accumulates
|
|
over months — every new user, every proxy rule change, every config update applied
|
|
in one place and forgotten in the other.
|
|
|
|
**The fix:** one auth stack with a managed mirror. HOST1 owns the configuration.
|
|
HOST2 runs a warm copy that stays current via 30-minute sync. HOST2's operator makes
|
|
zero auth management decisions — clicking an auth container opens HOST1's WebUI via
|
|
Tailscale. Changes happen there, propagate to HOST2 in 30 minutes. One place to
|
|
manage everything for both households.
|
|
|
|
---
|
|
|
|
### 🔴 No Structure Around the Relationship Itself
|
|
|
|
Setting up the mirror was a manual process. SSH in, reconfigure container WebUI
|
|
URLs one by one, copy auth config, verify connectivity, update state tracking. No
|
|
defined sequence. No dry-run capability. No verification that each step worked. If
|
|
something went wrong midway, the mirror was in an inconsistent state with no clear
|
|
way to understand what had and hadn't been done. Offboard was worse — it involves
|
|
stopping a sync that's been running for months, making a final copy of data,
|
|
reconfiguring WebUIs back to local addresses, removing Tailscale access, and
|
|
notifying both servers. A manual process with that many steps, taken under pressure,
|
|
leaves one or both parties in a bad state.
|
|
|
|
**The fix:** `partnership_manager.sh` with explicit modes for each lifecycle phase.
|
|
Each mode is a defined sequence. Every step is verified. Dry-run shows exactly what
|
|
will happen before anything changes. State files make the current relationship status
|
|
unambiguous from either server.
|
|
|
|
---
|
|
|
|
### 🔴 No Safe Way to Check If the Other Server Has Gone Away
|
|
|
|
After months of operation, HOST2 goes quiet. The sync starts failing. The offline
|
|
counter increments. But nothing actually happens — the ecosystem just keeps failing
|
|
the same sync, incrementing the same counter, sending the same notifications.
|
|
Without a defined threshold and an automated response, "partner gone for 30 days"
|
|
looks exactly like "partner gone for 3 years."
|
|
|
|
**The fix:** `PARTNERSHIP_OFFLINE_THRESHOLD`. After this many days of missed sync
|
|
cycles, both servers independently auto-offboard. HOST1 removes HOST2 from Tailscale,
|
|
disables critical sync, writes INACTIVE state. HOST2 — if it eventually comes back —
|
|
reads HOST1's INACTIVE state and cleans up its own side. The relationship is formally
|
|
ended from both sides without anyone needing to be present.
|
|
|
|
---
|
|
|
|
### 🔴 Ownership Transfer Had No Safe Path
|
|
|
|
The arrangement was always intended to be flexible — HOST1 owns the auth stack now,
|
|
but circumstances change. Swapping ownership manually meant reconfiguring WebUIs on
|
|
both servers, swapping sync direction, updating master.conf on both, and hoping the
|
|
sequence was correct. A misstep — like flipping sync direction before the final sync
|
|
completed — leaves both servers with different auth configurations and no clear source
|
|
of truth.
|
|
|
|
**The fix:** `--transfer` with a required confirmation string, a consecutive health
|
|
check system, and a strict sequence. The confirmation string cannot be typed
|
|
accidentally. Health strikes require both servers to be healthy on multiple
|
|
consecutive checks before the transfer begins. A final sync in the current direction
|
|
runs before anything is flipped.
|
|
|
|
---
|
|
|
|
## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Onboarding** (`partnership_onboard.sh`) — one-time setup run from both servers.
|
|
Generates SSH keys, installs the auth stack and arr stack on the mirror from XML
|
|
templates, configures WebUI redirects to the owner, and bootstraps the arr library.
|
|
Role is detected automatically — no flags needed to declare which side you are.
|
|
|
|
**Offboarding** (`partnership_offboard.sh`) — handles clean separation from either
|
|
role. Owner path: final sync, WebUI reconfigure, remote container + appdata cleanup
|
|
(auth/arr stack by XML array, fallback coverage by naming), SSH key revocation,
|
|
Tailscale removal. Mirror path: local WebUI reconfigure, remove owner-deployed
|
|
containers locally, disable sync, revoke Emby admin, SSH key revocation, signal owner.
|
|
Called by `partnership_manager.sh --offboard` but runnable directly.
|
|
|
|
**Lifecycle management** (`partnership_manager.sh`) — dispatcher and monitor.
|
|
- Manually: `--onboard`, `--offboard` (delegates to offboard script), `--transfer`, `--status`, `--unblock`
|
|
- Automatically: `--check` called every 30 minutes by `critical_sync_maintenance.sh`
|
|
|
|
**SSH management** (`ssh_setup.sh`) — generates the keypair for rsync automation,
|
|
installs it on the remote, and tracks auth failures with a configurable strike system.
|
|
Called by `partnership_onboard.sh` but runnable independently for validation and
|
|
re-keying.
|
|
|
|
---
|
|
|
|
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Orchestrators/** — `critical_sync_maintenance.sh` calls `partnership_manager.sh --check`
|
|
every 30 minutes, passing `--remote-seen` or `--remote-unseen` based on whether the
|
|
rsync to the partner succeeded. The rsync outcome is the connectivity signal — no
|
|
separate ping needed.
|
|
|
|
**Rsync/** — Critical-Data rsync keeps the auth stack appdata current on the mirror
|
|
(NPM rules, Authelia config, LLDAP database, certs). Partnership manages the
|
|
relationship; Rsync delivers the actual data. On offboard, `rsync_stop.sh --rsync-only`
|
|
stops any running rsync before the final sync runs.
|
|
|
|
**Media/** — `arr_sync.sh` bootstraps the mirror's arr library during onboard, ensuring
|
|
both servers have each other's full library from day one. Ongoing arr sync runs
|
|
independently at 4-hour cadence.
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
| Script | Role | When It Runs |
|
|
|--------|------|--------------|
|
|
| `partnership_onboard.sh` | One-time setup — SSH keys, stack deploy, arr bootstrap | Manually, once per server per partnership |
|
|
| `partnership_offboard.sh` | Clean separation — both paths, both roles | Via `partnership_manager.sh --offboard`; or directly |
|
|
| `partnership_manager.sh` | Dispatcher + monitor — onboard WebUIs, health check, transfer, status | `--check` every 30min; all other modes manually |
|
|
| `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
|
|
|
|
---
|
|
|
|
## ━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```
|
|
INITIAL SETUP (run once)
|
|
─────────────────────────────────────────────────────────────────────────────
|
|
HOST2 (mirror) runs:
|
|
partnership_onboard.sh
|
|
└─ ssh_setup.sh generates keypair, copies to owner
|
|
|
|
HOST1 (owner) runs:
|
|
partnership_onboard.sh
|
|
├─ ssh_setup.sh generates keypair, copies to mirror
|
|
├─ [plugin install on mirror] FolderView3 if configured
|
|
├─ [stop mirror auth stack] PARTNERSHIP_REPLACE_CONTAINERS via SSH
|
|
├─ deploy_container_from_xml() pushes auth XMLs to mirror + starts containers
|
|
│ └─ wait_for_container_healthy() Mariadb/Redis health-checked before Authelia
|
|
├─ [stop mirror arr stack] PARTNERSHIP_ARR_REPLACE_CONTAINERS via SSH
|
|
├─ deploy_container_from_xml() pushes arr XMLs to mirror + starts containers
|
|
├─ partnership_manager.sh --onboard reconfigures WebUIs, writes ACTIVE state
|
|
└─ arr_sync.sh bootstraps full library on both servers
|
|
|
|
ONGOING OPERATION (every 30min)
|
|
─────────────────────────────────────────────────────────────────────────────
|
|
critical_sync_maintenance.sh
|
|
├─ Critical-Data rsync keeps auth appdata current on mirror
|
|
└─ partnership_manager.sh --check reads state files, tracks offline counter
|
|
├─ --remote-seen path rsync succeeded → reset counter
|
|
└─ --remote-unseen path rsync failed → increment counter → auto-offboard at threshold
|
|
|
|
OFFBOARD (manual or auto)
|
|
─────────────────────────────────────────────────────────────────────────────
|
|
partnership_manager.sh --offboard
|
|
└─ partnership_offboard.sh (exec'd — holds own lock)
|
|
├─ [owner-initiated] stop rsync → final sync → reconfigure mirror WebUIs
|
|
│ → disable critical sync → write INACTIVE state
|
|
│ → local fallback cleanup → restart own stack
|
|
│ → remote: remove auth/arr stack + fallback containers + appdata
|
|
│ → restart mirror stack → Emby revoke → SSH key revocation
|
|
│ → Tailscale removal after grace window
|
|
└─ [mirror-initiated] stop rsync → reconfigure own WebUIs
|
|
→ remove owner-deployed containers locally (reads owner's stack arrays)
|
|
→ remove local fallback containers → disable critical sync
|
|
→ revoke own Emby admin → restart own stack
|
|
→ SSH key revocation → write INACTIVE → signal owner
|
|
```
|