audit echo vs log across all scripts — outcomes always visible, verbose for per-item loops
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,192 @@
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# 🤝 PARTNERSHIP
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
**Managed lifecycle for a two-server partnership — setup, ongoing operation,
|
||||
and clean separation.** One server owns the shared services. The other mirrors
|
||||
them and benefits from them. Every phase of the relationship has the same
|
||||
engineering discipline as the rest of the ecosystem.
|
||||
|
||||
> **This folder exists because a clean exit should be as easy as a clean setup.**
|
||||
> The partnership is not a permanent commitment. `--offboard` works from either
|
||||
> server at any time. Everything the mirror needs to run independently is already
|
||||
> there. The only thing that stops on separation is the sync — and that's intentional.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Two Servers, One Auth Stack, No Clean Way to Share It
|
||||
|
||||
The auth stack — NginxProxyManager, LLDAP, Authelia, MariaDB, Redis — runs on
|
||||
HOST1. HOST2 serves its own domain to its own household. It needs its own auth.
|
||||
But maintaining two independent auth stacks means double the work: two places to
|
||||
add users, two places to update proxy rules, two places to renew certs, two
|
||||
configurations that inevitably drift apart. One change on HOST1 has to be manually
|
||||
replicated to HOST2 — or it isn't, and the configurations diverge silently.
|
||||
|
||||
The real cost isn't the initial setup. It's the maintenance burden that accumulates
|
||||
over months — every new user, every proxy rule change, every config update applied
|
||||
in one place and forgotten in the other.
|
||||
|
||||
**The fix:** one auth stack with a managed mirror. HOST1 owns the configuration.
|
||||
HOST2 runs a warm copy that stays current via 30-minute sync. HOST2's operator makes
|
||||
zero auth management decisions — clicking an auth container opens HOST1's WebUI via
|
||||
Tailscale. Changes happen there, propagate to HOST2 in 30 minutes. One place to
|
||||
manage everything for both households.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 No Structure Around the Relationship Itself
|
||||
|
||||
Setting up the mirror was a manual process. SSH in, reconfigure container WebUI
|
||||
URLs one by one, copy auth config, verify connectivity, update state tracking. No
|
||||
defined sequence. No dry-run capability. No verification that each step worked. If
|
||||
something went wrong midway, the mirror was in an inconsistent state with no clear
|
||||
way to understand what had and hadn't been done. Offboard was worse — it involves
|
||||
stopping a sync that's been running for months, making a final copy of data,
|
||||
reconfiguring WebUIs back to local addresses, removing Tailscale access, and
|
||||
notifying both servers. A manual process with that many steps, taken under pressure,
|
||||
leaves one or both parties in a bad state.
|
||||
|
||||
**The fix:** `partnership_manager.sh` with explicit modes for each lifecycle phase.
|
||||
Each mode is a defined sequence. Every step is verified. Dry-run shows exactly what
|
||||
will happen before anything changes. State files make the current relationship status
|
||||
unambiguous from either server.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 No Safe Way to Check If the Other Server Has Gone Away
|
||||
|
||||
After months of operation, HOST2 goes quiet. The sync starts failing. The offline
|
||||
counter increments. But nothing actually happens — the ecosystem just keeps failing
|
||||
the same sync, incrementing the same counter, sending the same notifications.
|
||||
Without a defined threshold and an automated response, "partner gone for 30 days"
|
||||
looks exactly like "partner gone for 3 years."
|
||||
|
||||
**The fix:** `PARTNERSHIP_OFFLINE_THRESHOLD`. After this many days of missed sync
|
||||
cycles, both servers independently auto-offboard. HOST1 removes HOST2 from Tailscale,
|
||||
disables critical sync, writes INACTIVE state. HOST2 — if it eventually comes back —
|
||||
reads HOST1's INACTIVE state and cleans up its own side. The relationship is formally
|
||||
ended from both sides without anyone needing to be present.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Ownership Transfer Had No Safe Path
|
||||
|
||||
The arrangement was always intended to be flexible — HOST1 owns the auth stack now,
|
||||
but circumstances change. Swapping ownership manually meant reconfiguring WebUIs on
|
||||
both servers, swapping sync direction, updating master.conf on both, and hoping the
|
||||
sequence was correct. A misstep — like flipping sync direction before the final sync
|
||||
completed — leaves both servers with different auth configurations and no clear source
|
||||
of truth.
|
||||
|
||||
**The fix:** `--transfer` with a required confirmation string, a consecutive health
|
||||
check system, and a strict sequence. The confirmation string cannot be typed
|
||||
accidentally. Health strikes require both servers to be healthy on multiple
|
||||
consecutive checks before the transfer begins. A final sync in the current direction
|
||||
runs before anything is flipped.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
**Onboarding** (`partnership_onboard.sh`) — one-time setup run from both servers.
|
||||
Generates SSH keys, installs the auth stack and arr stack on the mirror from XML
|
||||
templates, configures WebUI redirects to the owner, and bootstraps the arr library.
|
||||
Role is detected automatically — no flags needed to declare which side you are.
|
||||
|
||||
**Offboarding** (`partnership_offboard.sh`) — handles clean separation from either
|
||||
role. Owner path: final sync, WebUI reconfigure, remote container + appdata cleanup
|
||||
(auth/arr stack by XML array, fallback coverage by naming), SSH key revocation,
|
||||
Tailscale removal. Mirror path: local WebUI reconfigure, remove owner-deployed
|
||||
containers locally, disable sync, revoke Emby admin, SSH key revocation, signal owner.
|
||||
Called by `partnership_manager.sh --offboard` but runnable directly.
|
||||
|
||||
**Lifecycle management** (`partnership_manager.sh`) — dispatcher and monitor.
|
||||
- Manually: `--onboard`, `--offboard` (delegates to offboard script), `--transfer`, `--status`, `--unblock`
|
||||
- Automatically: `--check` called every 30 minutes by `critical_sync_maintenance.sh`
|
||||
|
||||
**SSH management** (`ssh_setup.sh`) — generates the keypair for rsync automation,
|
||||
installs it on the remote, and tracks auth failures with a configurable strike system.
|
||||
Called by `partnership_onboard.sh` but runnable independently for validation and
|
||||
re-keying.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
**Orchestrators/** — `critical_sync_maintenance.sh` calls `partnership_manager.sh --check`
|
||||
every 30 minutes, passing `--remote-seen` or `--remote-unseen` based on whether the
|
||||
rsync to the partner succeeded. The rsync outcome is the connectivity signal — no
|
||||
separate ping needed.
|
||||
|
||||
**Rsync/** — Critical-Data rsync keeps the auth stack appdata current on the mirror
|
||||
(NPM rules, Authelia config, LLDAP database, certs). Partnership manages the
|
||||
relationship; Rsync delivers the actual data. On offboard, `rsync_stop.sh --rsync-only`
|
||||
stops any running rsync before the final sync runs.
|
||||
|
||||
**Media/** — `arr_sync.sh` bootstraps the mirror's arr library during onboard, ensuring
|
||||
both servers have each other's full library from day one. Ongoing arr sync runs
|
||||
independently at 4-hour cadence.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|--------------|
|
||||
| `partnership_onboard.sh` | One-time setup — SSH keys, stack deploy, arr bootstrap | Manually, once per server per partnership |
|
||||
| `partnership_offboard.sh` | Clean separation — both paths, both roles | Via `partnership_manager.sh --offboard`; or directly |
|
||||
| `partnership_manager.sh` | Dispatcher + monitor — onboard WebUIs, health check, transfer, status | `--check` every 30min; all other modes manually |
|
||||
| `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```
|
||||
INITIAL SETUP (run once)
|
||||
─────────────────────────────────────────────────────────────────────────────
|
||||
HOST2 (mirror) runs:
|
||||
partnership_onboard.sh
|
||||
└─ ssh_setup.sh generates keypair, copies to owner
|
||||
|
||||
HOST1 (owner) runs:
|
||||
partnership_onboard.sh
|
||||
├─ ssh_setup.sh generates keypair, copies to mirror
|
||||
├─ [plugin install on mirror] FolderView3 if configured
|
||||
├─ [stop mirror auth stack] PARTNERSHIP_REPLACE_CONTAINERS via SSH
|
||||
├─ deploy_container_from_xml() pushes auth XMLs to mirror + starts containers
|
||||
│ └─ wait_for_container_healthy() Mariadb/Redis health-checked before Authelia
|
||||
├─ [stop mirror arr stack] PARTNERSHIP_ARR_REPLACE_CONTAINERS via SSH
|
||||
├─ deploy_container_from_xml() pushes arr XMLs to mirror + starts containers
|
||||
├─ partnership_manager.sh --onboard reconfigures WebUIs, writes ACTIVE state
|
||||
└─ arr_sync.sh bootstraps full library on both servers
|
||||
|
||||
ONGOING OPERATION (every 30min)
|
||||
─────────────────────────────────────────────────────────────────────────────
|
||||
critical_sync_maintenance.sh
|
||||
├─ Critical-Data rsync keeps auth appdata current on mirror
|
||||
└─ partnership_manager.sh --check reads state files, tracks offline counter
|
||||
├─ --remote-seen path rsync succeeded → reset counter
|
||||
└─ --remote-unseen path rsync failed → increment counter → auto-offboard at threshold
|
||||
|
||||
OFFBOARD (manual or auto)
|
||||
─────────────────────────────────────────────────────────────────────────────
|
||||
partnership_manager.sh --offboard
|
||||
└─ partnership_offboard.sh (exec'd — holds own lock)
|
||||
├─ [owner-initiated] stop rsync → final sync → reconfigure mirror WebUIs
|
||||
│ → disable critical sync → write INACTIVE state
|
||||
│ → local fallback cleanup → restart own stack
|
||||
│ → remote: remove auth/arr stack + fallback containers + appdata
|
||||
│ → restart mirror stack → Emby revoke → SSH key revocation
|
||||
│ → Tailscale removal after grace window
|
||||
└─ [mirror-initiated] stop rsync → reconfigure own WebUIs
|
||||
→ remove owner-deployed containers locally (reads owner's stack arrays)
|
||||
→ remove local fallback containers → disable critical sync
|
||||
→ revoke own Emby admin → restart own stack
|
||||
→ SSH key revocation → write INACTIVE → signal owner
|
||||
```
|
||||
Reference in New Issue
Block a user