Varaverk/Partnership/Manual-Partnership.md

# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🤝  PARTNERSHIP — Manual
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Config reference, setup procedures, and operational how-tos.
For architecture see README-Partnership.md. For per-script detail see script headers.

---

## ━━━ THE RELATIONSHIP MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
HOST1 (owner)              HOST2 (mirror)
────────────────           ────────────────────────────
Source of truth            Warm copy — always current
Auth stack config          Auth containers running
NPM proxy rules            NPM serving mirror's domain
LLDAP users                LLDAP — same users
Authelia policies          Authelia — same policies
Certs                      Certs — mirrored, valid
Emby                       Emby — dirty-synced every 15min

Changes made here    ──→   Propagated every 15 minutes
WebUI management     ──→   Redirected to HOST1 via Tailscale
Config → git push    ──→   Received via git pull on next cycle
```

**Mirror's daily experience:** open unRAID Docker UI, click NginxProxyManager,
browser opens HOST1's NPM WebUI automatically via Tailscale. Make a proxy rule
change. 15 minutes later it's live on HOST2. The mirror operator never manages
auth directly — every WebUI redirect is transparent and automatic.

**Owner's daily experience:** manage auth as normal. Opens NPM → adds a proxy
rule → 15min later live on both servers. Never needs to SSH to HOST2 or think
about HOST2 during normal operation.

---

## ━━━ SETUP PREREQUISITES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Before `partnership_onboard.sh` can run, five things must be in place:

### 1. Tailscale Connected on Both Servers

```bash
# Verify HOST2 is visible from HOST1:
tailscale ip -4 unRAID-Jayred365    # returns HOST2's Tailscale IP

# Verify reachability:
tailscale ping unRAID-Jayred365
```

Hostnames must match their exact Tailscale device names — enforced already via
`REMOTE_SERVER_NAME` in master.conf.

### 2. Auth Containers Exist on HOST2

```bash
# Containers must exist — they can be stopped:
ssh root@[HOST2-ip] "docker inspect NginxProxyManager --format '{{.State.Status}}'"
# Expected: created, exited, or running — NOT "no such container"
```

`partnership_onboard.sh` will stop these before deploying the owner's auth stack.
List them in `HOST2_PARTNERSHIP_REPLACE_CONTAINERS`.

### 3. XML Templates Exist on HOST1

```bash
ls /boot/config/plugins/dockerMan/templates-user/my-NginxProxyManager.xml
ls /boot/config/plugins/dockerMan/templates-user/my-Authelia.xml
# etc — one XML per container listed in HOST1_PARTNERSHIP_AUTH_STACK
```

These are copied to HOST2 during onboard. The same XMLs that Unraid's Docker
Manager uses — no extra configuration needed on HOST2 after deploy.

### 4. Critical-Data rsync Profile Configured

```bash
# master.conf — must include the auth stack appdata path:
CRITICAL_SYNC_SHARES=(
    "/mnt/user/appdata-Fallback/Critical-Data"    # auth config, certs, NPM rules
)
```

This is what keeps HOST2 current after onboard. Without it, onboard succeeds but
the mirror's auth stack drifts from the owner's within hours.

### 5. Tailscale API Key (if PARTNERSHIP_REMOVE_TAILSCALE=true)

```bash
# master.conf
PARTNERSHIP_REMOVE_TAILSCALE=true
TAILSCALE_API_KEY="tskey-api-..."   # from tailscale.com/admin/settings/keys
TAILSCALE_TAILNET="example.github"  # your tailnet name

# Required scope: Devices write
```

---

## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

### master.conf

```bash
# ── Partnership Gate ───────────────────────────────────────────────────────
PARTNERSHIP_ENABLED=false           # set true once both servers are configured
PARTNERSHIP_OWNER_HOST="HOST1"      # "HOST1" or "HOST2" — flips on --transfer

# ── Tailscale Removal ──────────────────────────────────────────────────────
PARTNERSHIP_REMOVE_TAILSCALE=true
TAILSCALE_API_KEY=""                # tskey-api-... from tailscale.com/admin/settings/keys
TAILSCALE_TAILNET=""                # your tailnet name (e.g. example.github)

# ── Timing ─────────────────────────────────────────────────────────────────
PARTNERSHIP_GRACE_HOURS=6           # hours before Tailscale removal after offboard
                                    # backup access also expires at this time
PARTNERSHIP_OFFLINE_THRESHOLD=30    # days unreachable before auto-offboard
PARTNERSHIP_SYNC_INTERVAL=15        # informational — actual schedule in cron

# ── Transfer Safety ────────────────────────────────────────────────────────
PARTNERSHIP_TRANSFER_CONFIRM="i-understand-this-transfers-ownership"
PARTNERSHIP_TRANSFER_STRIKES=3      # consecutive health checks required
PARTNERSHIP_TRANSFER_MAX_ATTEMPTS=20

# ── Onboard Behaviour ──────────────────────────────────────────────────────
PARTNERSHIP_ONBOARD_VERIFY=true     # curl-verify each WebUI after onboard
PARTNERSHIP_ONBOARD_NOTIFY=true     # notify both servers on successful onboard
```

### master_host1.conf (owner side)

```bash
# Containers whose WebUI URLs are redirected to owner's Tailscale IP on onboard.
# Restored to localhost on offboard.
# Format: "ContainerName|WebUIPort"
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
    "NginxProxyManager|81"
    "Lldap-Gmer4Lfe|17170"
    "Authelia|9091"
    "Authelia-Secondary|9092"
)

# XML templates pushed to mirror during onboard (auth stack).
# ORDER MATTERS: database dependencies must come before Authelia.
# Mariadb/Redis are health-checked after deploy before continuing.
HOST1_PARTNERSHIP_AUTH_STACK=(
    "my-Mariadb-Authelia.xml"
    "my-Mariadb-Authelia-Secondary.xml"
    "my-Redis-Authelia.xml"
    "my-Redis-Authelia-Secondary.xml"
    "my-Authelia.xml"
    "my-Authelia-Secondary.xml"
    "my-NginxProxyManager.xml"
    "my-Lldap-Gmer4Lfe.xml"
)

# XML templates pushed to mirror during onboard (arr stack).
# Leave empty to skip arr stack deploy entirely.
HOST1_PARTNERSHIP_ARR_STACK=(
    # "my-Sonarr.xml"
    # "my-Radarr.xml"
    # "my-Lidarr.xml"
    # "my-Prowlarr.xml"
)

# Paths mirror can collect during grace window after offboard.
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
    # "/mnt/user/appdata-Fallback/Jayred365-Emby"
)

# Containers parked on this server when partnership is active.
# Stopped on onboard (owner deploys auth stack instead), restarted on offboard.
HOST1_PARTNERSHIP_OWN_CONTAINERS=(
    # "Emby"
    # "NginxProxyManager"
)
```

### master_host2.conf (mirror side)

```bash
# Containers whose WebUI URLs are redirected on onboard.
# Usually left empty on mirror — owner's AUTH_WEBUIS drives the redirect.
HOST2_PARTNERSHIP_AUTH_WEBUIS=(
    # "NginxProxyManager|81"
)

# Auth containers to stop on this server before owner deploys auth stack.
HOST2_PARTNERSHIP_REPLACE_CONTAINERS=(
    "NginxProxyManager"
    "Authelia"
    "Authelia-Secondary"
    "Mariadb-Authelia"
    "Mariadb-Authelia-Secondary"
    "Redis-Authelia"
    "Redis-Authelia-Secondary"
    "Lldap"
)

# Arr containers to stop on this server before owner deploys arr stack.
HOST2_PARTNERSHIP_ARR_REPLACE_CONTAINERS=(
    # "Sonarr"
    # "Radarr"
    # "Lidarr"
    # "Prowlarr"
)

# Containers parked on this server when partnership is active.
# Stopped on onboard, restarted on offboard.
HOST2_PARTNERSHIP_OWN_CONTAINERS=(
    # "Emby"
    # "NginxProxyManager"
)

# Emby admin provisioning — owner reads these during onboard.
HOST2_PARTNERSHIP_EMBY_ADMIN_USER=""
HOST2_PARTNERSHIP_EMBY_ADMIN_PASS=""
```

---

## ━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

### Establishing a Partnership (First-Time Setup)

```bash
# Step 1: Run on HOST2 (mirror) first — generates SSH key only
Partnership/partnership_onboard.sh

# Step 2: Run on HOST1 (owner) — completes setup remotely
Partnership/partnership_onboard.sh --dry-run    # review first
Partnership/partnership_onboard.sh

# Step 3: Verify from either server
Partnership/partnership_manager.sh --status
```

HOST2 must run first to generate its SSH key so HOST1 can reach it during Step 2.
If SSH is already configured, use `--skip-ssh` on both sides.

**Skip flags for partial re-runs** (if something failed midway):
```bash
--skip-ssh          # SSH already set up
--skip-auth-stack   # auth stack already deployed (skips Steps 3-4)
--skip-arr-stack    # arr stack already deployed (skips Steps 5-6)
--skip-arr-sync     # arr sync not needed yet (arrs not live on mirror)
```

---

### Offboarding

Either server can initiate. Always dry-run first.

```bash
# From either server — review the sequence
Partnership/partnership_manager.sh --offboard --dry-run

# Live offboard — 10-second countdown before executing
Partnership/partnership_manager.sh --offboard
```

**Owner-initiated** (HOST1 runs `--offboard`):
1. Stop any running rsync
2. Final sync — mirror leaves with current auth config and certs
3. Reconfigure mirror WebUIs → localhost
4. Disable critical rsync (`CRITICAL_RSYNC_ENABLED=false`)
5. Write INACTIVE state on both servers, blocklist mirror
6. Local fallback cleanup — remove partner containers + appdata from HOST1
7. Restart own stack (HOST1's own parked containers)
8. Remote cleanup — remove auth/arr stack containers + appdata from mirror; remove fallback containers
9. Restart mirror's own stack
10. Revoke Emby admin, SSH key revocation (both directions), Tailscale removal

**Mirror-initiated** (HOST2 runs `--offboard`):
1. Stop any running rsync
2. Reconfigure own WebUIs → localhost (immediately independent)
3. Remove owner-deployed containers locally (reads owner's auth/arr stack arrays via SSH)
4. Remove local fallback coverage containers
5. Disable critical rsync (`CRITICAL_RSYNC_ENABLED=false`)
6. Revoke own Emby admin account from local Emby instance
7. Restart own stack (HOST2's own parked containers)
8. SSH key revocation, write INACTIVE state locally, push state to HOST1 if reachable

HOST1 finalises its own side on the next `--check` cycle after seeing HOST2's INACTIVE state.

**What the mirror leaves with after offboard:**
- Its own parked containers restarted (from `PARTNERSHIP_OWN_CONTAINERS`)
- Own arr media library (arr appdata is in `appdata-Fallback/Arrs_Stack/` and is retained)
- Full git mirror of the ecosystem
- Auth WebUIs pointing to localhost — ready to set up own independent auth

---

### Transferring Ownership

Transfers auth stack ownership from current owner to mirror. Only the current owner
can initiate. Both servers must be healthy.

```bash
# Always dry-run first — shows health check results and master.conf changes
Partnership/partnership_manager.sh --transfer --dry-run

# Live transfer — requires exact confirmation string
Partnership/partnership_manager.sh --transfer --confirm=i-understand-this-transfers-ownership
```

Transfer sequence:
1. Display current and future ownership clearly
2. Require exact confirmation string
3. Both servers pass `PARTNERSHIP_TRANSFER_STRIKES` consecutive health checks
4. Final sync in current direction (current owner → current mirror)
5. Reconfigure WebUI templates on both servers
6. Flip `PARTNERSHIP_OWNER_HOST` in master.conf on both servers
7. Write updated state files and notify

After transfer, HOST2 is the owner. Run `--transfer` from HOST2 to transfer back.

---

### Re-onboarding After Offboard

Former partners are blocklisted on offboard. Remove from blocklist first:

```bash
# Check what's on the blocklist
Partnership/partnership_manager.sh --status

# Remove the block (use exact hostname shown in status)
Partnership/partnership_manager.sh --unblock unRAID-Jayred365

# Then re-onboard normally
Partnership/partnership_onboard.sh --dry-run
Partnership/partnership_onboard.sh
```

---

### Adding a New Container to the Auth Stack

1. Create the XML template on HOST1 (`/boot/config/plugins/dockerMan/templates-user/my-NewContainer.xml`)
2. Add the XML filename to `HOST1_PARTNERSHIP_AUTH_STACK` in `master_host1.conf`
   - If it has a database dependency, put the dep earlier in the array
3. Add the container name to `HOST2_PARTNERSHIP_REPLACE_CONTAINERS` in `master_host2.conf`
4. Re-run the auth stack portion:
   ```bash
   Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync
   ```

---

### SSH Key Validation and Re-keying

```bash
# Check current key state and remote connectivity
Partnership/ssh_setup.sh --status

# Validate SSH auth (tracks strikes)
Partnership/ssh_setup.sh --validate

# Regenerate key and re-copy to remote (if key is compromised or expired)
Partnership/ssh_setup.sh --force
```

Strike tracking: `SSH_MAX_STRIKES` consecutive auth failures → notify.
Network unreachability (Tailscale down) does not count as a strike.
Counter resets after `SSH_STRIKE_RESET_HRS` of clean connectivity.

---

## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

### partnership_onboard.sh

| Flag | Effect |
|------|--------|
| `--dry-run` | Preview all steps without making changes |
| `--log` | Verbose per-step output |
| `--skip-ssh` | Skip SSH key setup (Step 1) — key already in place |
| `--skip-auth-stack` | Skip auth stop + deploy (Steps 3-4) — already done |
| `--skip-arr-stack` | Skip arr stop + deploy (Steps 5-6) — not needed or already done |
| `--skip-arr-sync` | Skip arr library bootstrap (Step 8) — arrs not live yet |

### partnership_offboard.sh

Called automatically by `partnership_manager.sh --offboard`. Can also be run directly.

| Flag | Effect |
|------|--------|
| `--dry-run` | Preview all steps without making changes |
| `--log` | Verbose per-step output |
| `--reason=<string>` | Tag the offboard reason in state file and blocklist (default: `manual`) |

### partnership_manager.sh

| Flag | Effect |
|------|--------|
| `--dry-run` | Show the complete sequence without executing |
| `--log` | Verbose per-operation output |
| `--onboard` | Establish mirror relationship (owner only) |
| `--offboard` | Clean separation (either server) — 10s countdown |
| `--transfer --confirm=...` | Flip ownership (owner only) |
| `--check --remote-seen\|--remote-unseen` | 15-min health check (called by orchestrator) |
| `--status` | Show state files, blocklist, SSH key status |
| `--unblock <hostname>` | Remove hostname from blocklist |

### ssh_setup.sh

| Flag | Effect |
|------|--------|
| `--force` | Regenerate key even if it exists, re-copy to remote |
| `--validate` | Test SSH auth, track strikes, notify at limit |
| `--status` | Show key path, fingerprint, remote connectivity |
| `--dry-run` | Preview without creating or copying |

---

## ━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

### Auth Stack Deploy Fails on Onboard

```bash
# Check that XML files exist on HOST1:
ls /boot/config/plugins/dockerMan/templates-user/my-Authelia.xml

# Check that mirror is reachable via Tailscale:
tailscale ping unRAID-Jayred365

# Check that SSH key is working:
Partnership/ssh_setup.sh --validate

# Re-run auth stack only:
Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync
```

If Mariadb/Redis deployed but Authelia still fails: the health-wait timed out
(60s default). Authelia needs Mariadb to be fully initialized, which can take longer
on first boot. Re-run with `--skip-auth-stack` removed — `docker start` on an
already-created container will retry cleanly.

---

### WebUI Still Pointing to Old IP After Onboard

```bash
# Check template was updated on HOST2:
ssh root@[HOST2-ip] "grep -i 'tailscale\|[HOST1-IP]' \
    /boot/config/plugins/dockerMan/templates-user/my-NginxProxyManager.xml"

# Check status output:
Partnership/partnership_manager.sh --status

# Some containers need a restart to pick up new WebUI URL:
ssh root@[HOST2-ip] "docker restart NginxProxyManager"

# Verify Tailscale routing:
# From HOST2, should reach HOST1's NPM:
curl http://[HOST1-tailscale-ip]:81
```

---

### Transfer Health Checks Failing

```bash
# Both arrays must be fully started:
ls /mnt/user                           # should show share directories on both servers

# Both Docker daemons responding:
docker ps                              # should return a list, not hang

# Tailscale connected on both servers:
tailscale status                       # remote peer should show online

# Health check attempts before giving up: PARTNERSHIP_TRANSFER_MAX_ATTEMPTS (default 20)
```

---

### Mirror's Domain Broken After Offboard

The owner-deployed auth stack has been removed from HOST2 as part of offboard cleanup.
HOST2's own parked containers (from `PARTNERSHIP_OWN_CONTAINERS`) are restarted
automatically, but if HOST2 had no pre-existing auth stack of its own, it needs one set up.

```bash
# Check what's running:
docker ps

# Verify own parked containers came back up:
# (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in master_host2.conf)

# If you need a fresh auth stack, deploy from HOST2's own XML templates:
docker create ... && docker start NginxProxyManager   # etc.

# Check cert expiry — auto-renewal no longer happens via HOST1:
cert_monitor.sh --dry-run
```

If the offboard was unexpected or incomplete, check state first:
```bash
Partnership/partnership_manager.sh --status
```

---

### State Mismatch Between Servers

```bash
# Check both state files:
Partnership/partnership_manager.sh --status    # shows both sides via SSH

# If one shows ACTIVE and other INACTIVE:
#   The INACTIVE side has already offboarded.
#   Run --offboard on the ACTIVE side to sync the state.

# If HOST2 unreachable:
#   HOST2 self-resolves on next --check when reachable.
#   HOST1 reads INACTIVE state → finalises from owner side automatically.
```

---

### Auto-Offboard Triggered Unexpectedly

```bash
# Check the offline counter:
cat /boot/config/partnership_offline_days.db

# Extended Tailscale outage may have incremented the counter.
# Check Tailscale peer visibility:
tailscale status

# If partnership should continue — re-onboard (first unblock the partner):
Partnership/partnership_manager.sh --unblock unRAID-Jayred365
Partnership/partnership_manager.sh --onboard --dry-run
Partnership/partnership_manager.sh --onboard
```

---

## ━━━ STATE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```bash
/boot/config/partnership_HOST1.db   # HOST1 writes only
/boot/config/partnership_HOST2.db   # HOST2 writes only
/boot/config/partnership_blocklist.db  # hostname|timestamp|reason

# Example state file:
state=ACTIVE
last_updated=2026-05-14 03:00:00
last_seen_remote=2026-05-14 03:00:00
owner_host=HOST1
offline_days=0
```

Each server writes only its own state file. The other server reads via SSH.
State propagates through SSH reads — no rsync, no shared filesystem.
`/boot/config` — survives reboots, available before array starts, minimal flash wear.
flock on all writes — prevents concurrent corruption from overlapping --check cycles.

---

## ━━━ ROLE-BASED ACCESS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
                    HOST1 (owner)    HOST2 (mirror)
--onboard                ✅               ❌     ownership is granted, not taken
--offboard               ✅               ✅     clean exit available to both parties
--transfer               ✅               ❌     owner only — mirror cannot self-promote
--check                  ✅               ✅     both servers monitor state
--status                 ✅               ✅     status is always available
--unblock                ✅               ✅     either server can clear its own blocklist
```

AM_OWNER / AM_MIRROR flags set by `detect_hosts()` from `PARTNERSHIP_OWNER_HOST`.
All routing decisions use these flags — no hostname string comparisons.
Mirror attempting `--onboard` or `--transfer`: blocked with a clear error message.

---

## ━━━ DESIGN NOTES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Why HOST1/HOST2 instead of PARTNERSHIP_OWNER/MIRROR?**
HOST1 and HOST2 are already defined in master.conf — SSH keys, Tailscale names,
all connection details. A single `PARTNERSHIP_OWNER_HOST` var flips ownership.
Duplicating as `PARTNERSHIP_OWNER` and `PARTNERSHIP_MIRROR` would require maintaining
identical values in two places. One source of truth for the host definition.

**Why does transfer require a confirmation string?**
Transfer touches master.conf on both servers, reconfigures WebUIs, and flips sync
direction. A misstep mid-sequence leaves both servers with different auth configs and
no clear source of truth. The confirmation string makes accidental execution impossible,
not just unlikely.

**Why do Tailscale removal and backup access expire at the same time?**
If the mirror can't reach the owner's server via Tailscale, the backup is also
unreachable. `PARTNERSHIP_GRACE_HOURS` controls both — one var, consistent behaviour.
No misleading "data available for 30 days" when access is gone in 6 hours.

**Why does either server auto-offboard after 30 days offline?**
30 consecutive days of missed sync cycles means the relationship has effectively ended
regardless of intent. Auto-offboard makes the state official. Each server acts
independently — no coordination required to finalise.

**Why are auth/arr stack containers removed on offboard?**
The auth stack running on the mirror is the owner's stack — deployed from the owner's
XML templates, managed by the owner. On offboard, the mirror restores its own parked
containers (`PARTNERSHIP_OWN_CONTAINERS`) and sets up independent auth from scratch.
Nothing from the owner's deployment lingers. Appdata is deleted alongside containers
so there's no stale config left behind. Both fallback coverage containers (named
`*-Owner`) and the owner-deployed stack (from `PARTNERSHIP_AUTH_STACK` / `PARTNERSHIP_ARR_STACK`)
are removed — the mirror gets a clean slate.

**Why does XML ordering matter for the auth stack?**
Authelia requires MariaDB and Redis to be running and accepting connections before it
starts. `deploy_container_from_xml()` starts each container immediately after creating
it. If Authelia is deployed before MariaDB, it will fail to connect and may not recover
automatically. The array order in `HOST1_PARTNERSHIP_AUTH_STACK` is enforced by
convention — database containers first, then applications.