feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,623 @@
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# 🤝 PARTNERSHIP — Manual
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Config reference, setup procedures, and operational how-tos.
|
||||
For architecture see README-Partnership.md. For per-script detail see script headers.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE RELATIONSHIP MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```
|
||||
HOST1 (owner) HOST2 (mirror)
|
||||
──────────────── ────────────────────────────
|
||||
Source of truth Warm copy — always current
|
||||
Auth stack config Auth containers running
|
||||
NPM proxy rules NPM serving mirror's domain
|
||||
LLDAP users LLDAP — same users
|
||||
Authelia policies Authelia — same policies
|
||||
Certs Certs — mirrored, valid
|
||||
Emby Emby — dirty-synced every 15min
|
||||
|
||||
Changes made here ──→ Propagated every 15 minutes
|
||||
WebUI management ──→ Redirected to HOST1 via Tailscale
|
||||
Config → git push ──→ Received via git pull on next cycle
|
||||
```
|
||||
|
||||
**Mirror's daily experience:** open unRAID Docker UI, click NginxProxyManager,
|
||||
browser opens HOST1's NPM WebUI automatically via Tailscale. Make a proxy rule
|
||||
change. 15 minutes later it's live on HOST2. The mirror operator never manages
|
||||
auth directly — every WebUI redirect is transparent and automatic.
|
||||
|
||||
**Owner's daily experience:** manage auth as normal. Opens NPM → adds a proxy
|
||||
rule → 15min later live on both servers. Never needs to SSH to HOST2 or think
|
||||
about HOST2 during normal operation.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SETUP PREREQUISITES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Before `partnership_onboard.sh` can run, five things must be in place:
|
||||
|
||||
### 1. Tailscale Connected on Both Servers
|
||||
|
||||
```bash
|
||||
# Verify HOST2 is visible from HOST1:
|
||||
tailscale ip -4 unRAID-Jayred365 # returns HOST2's Tailscale IP
|
||||
|
||||
# Verify reachability:
|
||||
tailscale ping unRAID-Jayred365
|
||||
```
|
||||
|
||||
Hostnames must match their exact Tailscale device names — enforced already via
|
||||
`REMOTE_SERVER_NAME` in master.conf.
|
||||
|
||||
### 2. Auth Containers Exist on HOST2
|
||||
|
||||
```bash
|
||||
# Containers must exist — they can be stopped:
|
||||
ssh root@[HOST2-ip] "docker inspect NginxProxyManager --format '{{.State.Status}}'"
|
||||
# Expected: created, exited, or running — NOT "no such container"
|
||||
```
|
||||
|
||||
`partnership_onboard.sh` will stop these before deploying the owner's auth stack.
|
||||
List them in `HOST2_PARTNERSHIP_REPLACE_CONTAINERS`.
|
||||
|
||||
### 3. XML Templates Exist on HOST1
|
||||
|
||||
```bash
|
||||
ls /boot/config/plugins/dockerMan/templates-user/my-NginxProxyManager.xml
|
||||
ls /boot/config/plugins/dockerMan/templates-user/my-Authelia.xml
|
||||
# etc — one XML per container listed in HOST1_PARTNERSHIP_AUTH_STACK
|
||||
```
|
||||
|
||||
These are copied to HOST2 during onboard. The same XMLs that Unraid's Docker
|
||||
Manager uses — no extra configuration needed on HOST2 after deploy.
|
||||
|
||||
### 4. Critical-Data rsync Profile Configured
|
||||
|
||||
```bash
|
||||
# master.conf — must include the auth stack appdata path:
|
||||
CRITICAL_SYNC_SHARES=(
|
||||
"/mnt/user/appdata-Fallback/Critical-Data" # auth config, certs, NPM rules
|
||||
)
|
||||
```
|
||||
|
||||
This is what keeps HOST2 current after onboard. Without it, onboard succeeds but
|
||||
the mirror's auth stack drifts from the owner's within hours.
|
||||
|
||||
### 5. Tailscale API Key (if PARTNERSHIP_REMOVE_TAILSCALE=true)
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
PARTNERSHIP_REMOVE_TAILSCALE=true
|
||||
TAILSCALE_API_KEY="tskey-api-..." # from tailscale.com/admin/settings/keys
|
||||
TAILSCALE_TAILNET="example.github" # your tailnet name
|
||||
|
||||
# Required scope: Devices write
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
### master.conf
|
||||
|
||||
```bash
|
||||
# ── Partnership Gate ───────────────────────────────────────────────────────
|
||||
PARTNERSHIP_ENABLED=false # set true once both servers are configured
|
||||
PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer
|
||||
|
||||
# ── Tailscale Removal ──────────────────────────────────────────────────────
|
||||
PARTNERSHIP_REMOVE_TAILSCALE=true
|
||||
TAILSCALE_API_KEY="" # tskey-api-... from tailscale.com/admin/settings/keys
|
||||
TAILSCALE_TAILNET="" # your tailnet name (e.g. example.github)
|
||||
|
||||
# ── Timing ─────────────────────────────────────────────────────────────────
|
||||
PARTNERSHIP_GRACE_HOURS=6 # hours before Tailscale removal after offboard
|
||||
# backup access also expires at this time
|
||||
PARTNERSHIP_OFFLINE_THRESHOLD=30 # days unreachable before auto-offboard
|
||||
PARTNERSHIP_SYNC_INTERVAL=15 # informational — actual schedule in cron
|
||||
|
||||
# ── Transfer Safety ────────────────────────────────────────────────────────
|
||||
PARTNERSHIP_TRANSFER_CONFIRM="i-understand-this-transfers-ownership"
|
||||
PARTNERSHIP_TRANSFER_STRIKES=3 # consecutive health checks required
|
||||
PARTNERSHIP_TRANSFER_MAX_ATTEMPTS=20
|
||||
|
||||
# ── Onboard Behaviour ──────────────────────────────────────────────────────
|
||||
PARTNERSHIP_ONBOARD_VERIFY=true # curl-verify each WebUI after onboard
|
||||
PARTNERSHIP_ONBOARD_NOTIFY=true # notify both servers on successful onboard
|
||||
```
|
||||
|
||||
### master_host1.conf (owner side)
|
||||
|
||||
```bash
|
||||
# Containers whose WebUI URLs are redirected to owner's Tailscale IP on onboard.
|
||||
# Restored to localhost on offboard.
|
||||
# Format: "ContainerName|WebUIPort"
|
||||
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
|
||||
"NginxProxyManager|81"
|
||||
"Lldap-Gmer4Lfe|17170"
|
||||
"Authelia|9091"
|
||||
"Authelia-Secondary|9092"
|
||||
)
|
||||
|
||||
# XML templates pushed to mirror during onboard (auth stack).
|
||||
# ORDER MATTERS: database dependencies must come before Authelia.
|
||||
# Mariadb/Redis are health-checked after deploy before continuing.
|
||||
HOST1_PARTNERSHIP_AUTH_STACK=(
|
||||
"my-Mariadb-Authelia.xml"
|
||||
"my-Mariadb-Authelia-Secondary.xml"
|
||||
"my-Redis-Authelia.xml"
|
||||
"my-Redis-Authelia-Secondary.xml"
|
||||
"my-Authelia.xml"
|
||||
"my-Authelia-Secondary.xml"
|
||||
"my-NginxProxyManager.xml"
|
||||
"my-Lldap-Gmer4Lfe.xml"
|
||||
)
|
||||
|
||||
# XML templates pushed to mirror during onboard (arr stack).
|
||||
# Leave empty to skip arr stack deploy entirely.
|
||||
HOST1_PARTNERSHIP_ARR_STACK=(
|
||||
# "my-Sonarr.xml"
|
||||
# "my-Radarr.xml"
|
||||
# "my-Lidarr.xml"
|
||||
# "my-Prowlarr.xml"
|
||||
)
|
||||
|
||||
# Paths mirror can collect during grace window after offboard.
|
||||
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
|
||||
# "/mnt/user/appdata-Fallback/Jayred365-Emby"
|
||||
)
|
||||
|
||||
# Containers parked on this server when partnership is active.
|
||||
# Stopped on onboard (owner deploys auth stack instead), restarted on offboard.
|
||||
HOST1_PARTNERSHIP_OWN_CONTAINERS=(
|
||||
# "Emby"
|
||||
# "NginxProxyManager"
|
||||
)
|
||||
```
|
||||
|
||||
### master_host2.conf (mirror side)
|
||||
|
||||
```bash
|
||||
# Containers whose WebUI URLs are redirected on onboard.
|
||||
# Usually left empty on mirror — owner's AUTH_WEBUIS drives the redirect.
|
||||
HOST2_PARTNERSHIP_AUTH_WEBUIS=(
|
||||
# "NginxProxyManager|81"
|
||||
)
|
||||
|
||||
# Auth containers to stop on this server before owner deploys auth stack.
|
||||
HOST2_PARTNERSHIP_REPLACE_CONTAINERS=(
|
||||
"NginxProxyManager"
|
||||
"Authelia"
|
||||
"Authelia-Secondary"
|
||||
"Mariadb-Authelia"
|
||||
"Mariadb-Authelia-Secondary"
|
||||
"Redis-Authelia"
|
||||
"Redis-Authelia-Secondary"
|
||||
"Lldap"
|
||||
)
|
||||
|
||||
# Arr containers to stop on this server before owner deploys arr stack.
|
||||
HOST2_PARTNERSHIP_ARR_REPLACE_CONTAINERS=(
|
||||
# "Sonarr"
|
||||
# "Radarr"
|
||||
# "Lidarr"
|
||||
# "Prowlarr"
|
||||
)
|
||||
|
||||
# Containers parked on this server when partnership is active.
|
||||
# Stopped on onboard, restarted on offboard.
|
||||
HOST2_PARTNERSHIP_OWN_CONTAINERS=(
|
||||
# "Emby"
|
||||
# "NginxProxyManager"
|
||||
)
|
||||
|
||||
# Emby admin provisioning — owner reads these during onboard.
|
||||
HOST2_PARTNERSHIP_EMBY_ADMIN_USER=""
|
||||
HOST2_PARTNERSHIP_EMBY_ADMIN_PASS=""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
### Establishing a Partnership (First-Time Setup)
|
||||
|
||||
```bash
|
||||
# Step 1: Run on HOST2 (mirror) first — generates SSH key only
|
||||
Partnership/partnership_onboard.sh
|
||||
|
||||
# Step 2: Run on HOST1 (owner) — completes setup remotely
|
||||
Partnership/partnership_onboard.sh --dry-run # review first
|
||||
Partnership/partnership_onboard.sh
|
||||
|
||||
# Step 3: Verify from either server
|
||||
Partnership/partnership_manager.sh --status
|
||||
```
|
||||
|
||||
HOST2 must run first to generate its SSH key so HOST1 can reach it during Step 2.
|
||||
If SSH is already configured, use `--skip-ssh` on both sides.
|
||||
|
||||
**Skip flags for partial re-runs** (if something failed midway):
|
||||
```bash
|
||||
--skip-ssh # SSH already set up
|
||||
--skip-auth-stack # auth stack already deployed (skips Steps 3-4)
|
||||
--skip-arr-stack # arr stack already deployed (skips Steps 5-6)
|
||||
--skip-arr-sync # arr sync not needed yet (arrs not live on mirror)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Offboarding
|
||||
|
||||
Either server can initiate. Always dry-run first.
|
||||
|
||||
```bash
|
||||
# From either server — review the sequence
|
||||
Partnership/partnership_manager.sh --offboard --dry-run
|
||||
|
||||
# Live offboard — 10-second countdown before executing
|
||||
Partnership/partnership_manager.sh --offboard
|
||||
```
|
||||
|
||||
**Owner-initiated** (HOST1 runs `--offboard`):
|
||||
1. Stop any running rsync
|
||||
2. Final sync — mirror leaves with current auth config and certs
|
||||
3. Reconfigure mirror WebUIs → localhost
|
||||
4. Disable critical rsync (`CRITICAL_RSYNC_ENABLED=false`)
|
||||
5. Write INACTIVE state on both servers, blocklist mirror
|
||||
6. Local fallback cleanup — remove partner containers + appdata from HOST1
|
||||
7. Restart own stack (HOST1's own parked containers)
|
||||
8. Remote cleanup — remove auth/arr stack containers + appdata from mirror; remove fallback containers
|
||||
9. Restart mirror's own stack
|
||||
10. Revoke Emby admin, SSH key revocation (both directions), Tailscale removal
|
||||
|
||||
**Mirror-initiated** (HOST2 runs `--offboard`):
|
||||
1. Stop any running rsync
|
||||
2. Reconfigure own WebUIs → localhost (immediately independent)
|
||||
3. Remove owner-deployed containers locally (reads owner's auth/arr stack arrays via SSH)
|
||||
4. Remove local fallback coverage containers
|
||||
5. Disable critical rsync (`CRITICAL_RSYNC_ENABLED=false`)
|
||||
6. Revoke own Emby admin account from local Emby instance
|
||||
7. Restart own stack (HOST2's own parked containers)
|
||||
8. SSH key revocation, write INACTIVE state locally, push state to HOST1 if reachable
|
||||
|
||||
HOST1 finalises its own side on the next `--check` cycle after seeing HOST2's INACTIVE state.
|
||||
|
||||
**What the mirror leaves with after offboard:**
|
||||
- Its own parked containers restarted (from `PARTNERSHIP_OWN_CONTAINERS`)
|
||||
- Own arr media library (arr appdata is in `appdata-Fallback/Arrs_Stack/` and is retained)
|
||||
- Full git mirror of the ecosystem
|
||||
- Auth WebUIs pointing to localhost — ready to set up own independent auth
|
||||
|
||||
---
|
||||
|
||||
### Transferring Ownership
|
||||
|
||||
Transfers auth stack ownership from current owner to mirror. Only the current owner
|
||||
can initiate. Both servers must be healthy.
|
||||
|
||||
```bash
|
||||
# Always dry-run first — shows health check results and master.conf changes
|
||||
Partnership/partnership_manager.sh --transfer --dry-run
|
||||
|
||||
# Live transfer — requires exact confirmation string
|
||||
Partnership/partnership_manager.sh --transfer --confirm=i-understand-this-transfers-ownership
|
||||
```
|
||||
|
||||
Transfer sequence:
|
||||
1. Display current and future ownership clearly
|
||||
2. Require exact confirmation string
|
||||
3. Both servers pass `PARTNERSHIP_TRANSFER_STRIKES` consecutive health checks
|
||||
4. Final sync in current direction (current owner → current mirror)
|
||||
5. Reconfigure WebUI templates on both servers
|
||||
6. Flip `PARTNERSHIP_OWNER_HOST` in master.conf on both servers
|
||||
7. Write updated state files and notify
|
||||
|
||||
After transfer, HOST2 is the owner. Run `--transfer` from HOST2 to transfer back.
|
||||
|
||||
---
|
||||
|
||||
### Re-onboarding After Offboard
|
||||
|
||||
Former partners are blocklisted on offboard. Remove from blocklist first:
|
||||
|
||||
```bash
|
||||
# Check what's on the blocklist
|
||||
Partnership/partnership_manager.sh --status
|
||||
|
||||
# Remove the block (use exact hostname shown in status)
|
||||
Partnership/partnership_manager.sh --unblock unRAID-Jayred365
|
||||
|
||||
# Then re-onboard normally
|
||||
Partnership/partnership_onboard.sh --dry-run
|
||||
Partnership/partnership_onboard.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Adding a New Container to the Auth Stack
|
||||
|
||||
1. Create the XML template on HOST1 (`/boot/config/plugins/dockerMan/templates-user/my-NewContainer.xml`)
|
||||
2. Add the XML filename to `HOST1_PARTNERSHIP_AUTH_STACK` in `master_host1.conf`
|
||||
- If it has a database dependency, put the dep earlier in the array
|
||||
3. Add the container name to `HOST2_PARTNERSHIP_REPLACE_CONTAINERS` in `master_host2.conf`
|
||||
4. Re-run the auth stack portion:
|
||||
```bash
|
||||
Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### SSH Key Validation and Re-keying
|
||||
|
||||
```bash
|
||||
# Check current key state and remote connectivity
|
||||
Partnership/ssh_setup.sh --status
|
||||
|
||||
# Validate SSH auth (tracks strikes)
|
||||
Partnership/ssh_setup.sh --validate
|
||||
|
||||
# Regenerate key and re-copy to remote (if key is compromised or expired)
|
||||
Partnership/ssh_setup.sh --force
|
||||
```
|
||||
|
||||
Strike tracking: `SSH_MAX_STRIKES` consecutive auth failures → notify.
|
||||
Network unreachability (Tailscale down) does not count as a strike.
|
||||
Counter resets after `SSH_STRIKE_RESET_HRS` of clean connectivity.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
### partnership_onboard.sh
|
||||
|
||||
| Flag | Effect |
|
||||
|------|--------|
|
||||
| `--dry-run` | Preview all steps without making changes |
|
||||
| `--log` | Verbose per-step output |
|
||||
| `--skip-ssh` | Skip SSH key setup (Step 1) — key already in place |
|
||||
| `--skip-auth-stack` | Skip auth stop + deploy (Steps 3-4) — already done |
|
||||
| `--skip-arr-stack` | Skip arr stop + deploy (Steps 5-6) — not needed or already done |
|
||||
| `--skip-arr-sync` | Skip arr library bootstrap (Step 8) — arrs not live yet |
|
||||
|
||||
### partnership_offboard.sh
|
||||
|
||||
Called automatically by `partnership_manager.sh --offboard`. Can also be run directly.
|
||||
|
||||
| Flag | Effect |
|
||||
|------|--------|
|
||||
| `--dry-run` | Preview all steps without making changes |
|
||||
| `--log` | Verbose per-step output |
|
||||
| `--reason=<string>` | Tag the offboard reason in state file and blocklist (default: `manual`) |
|
||||
|
||||
### partnership_manager.sh
|
||||
|
||||
| Flag | Effect |
|
||||
|------|--------|
|
||||
| `--dry-run` | Show the complete sequence without executing |
|
||||
| `--log` | Verbose per-operation output |
|
||||
| `--onboard` | Establish mirror relationship (owner only) |
|
||||
| `--offboard` | Clean separation (either server) — 10s countdown |
|
||||
| `--transfer --confirm=...` | Flip ownership (owner only) |
|
||||
| `--check --remote-seen\|--remote-unseen` | 15-min health check (called by orchestrator) |
|
||||
| `--status` | Show state files, blocklist, SSH key status |
|
||||
| `--unblock <hostname>` | Remove hostname from blocklist |
|
||||
|
||||
### ssh_setup.sh
|
||||
|
||||
| Flag | Effect |
|
||||
|------|--------|
|
||||
| `--force` | Regenerate key even if it exists, re-copy to remote |
|
||||
| `--validate` | Test SSH auth, track strikes, notify at limit |
|
||||
| `--status` | Show key path, fingerprint, remote connectivity |
|
||||
| `--dry-run` | Preview without creating or copying |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
### Auth Stack Deploy Fails on Onboard
|
||||
|
||||
```bash
|
||||
# Check that XML files exist on HOST1:
|
||||
ls /boot/config/plugins/dockerMan/templates-user/my-Authelia.xml
|
||||
|
||||
# Check that mirror is reachable via Tailscale:
|
||||
tailscale ping unRAID-Jayred365
|
||||
|
||||
# Check that SSH key is working:
|
||||
Partnership/ssh_setup.sh --validate
|
||||
|
||||
# Re-run auth stack only:
|
||||
Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync
|
||||
```
|
||||
|
||||
If Mariadb/Redis deployed but Authelia still fails: the health-wait timed out
|
||||
(60s default). Authelia needs Mariadb to be fully initialized, which can take longer
|
||||
on first boot. Re-run with `--skip-auth-stack` removed — `docker start` on an
|
||||
already-created container will retry cleanly.
|
||||
|
||||
---
|
||||
|
||||
### WebUI Still Pointing to Old IP After Onboard
|
||||
|
||||
```bash
|
||||
# Check template was updated on HOST2:
|
||||
ssh root@[HOST2-ip] "grep -i 'tailscale\|[HOST1-IP]' \
|
||||
/boot/config/plugins/dockerMan/templates-user/my-NginxProxyManager.xml"
|
||||
|
||||
# Check status output:
|
||||
Partnership/partnership_manager.sh --status
|
||||
|
||||
# Some containers need a restart to pick up new WebUI URL:
|
||||
ssh root@[HOST2-ip] "docker restart NginxProxyManager"
|
||||
|
||||
# Verify Tailscale routing:
|
||||
# From HOST2, should reach HOST1's NPM:
|
||||
curl http://[HOST1-tailscale-ip]:81
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Transfer Health Checks Failing
|
||||
|
||||
```bash
|
||||
# Both arrays must be fully started:
|
||||
ls /mnt/user # should show share directories on both servers
|
||||
|
||||
# Both Docker daemons responding:
|
||||
docker ps # should return a list, not hang
|
||||
|
||||
# Tailscale connected on both servers:
|
||||
tailscale status # remote peer should show online
|
||||
|
||||
# Health check attempts before giving up: PARTNERSHIP_TRANSFER_MAX_ATTEMPTS (default 20)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Mirror's Domain Broken After Offboard
|
||||
|
||||
The owner-deployed auth stack has been removed from HOST2 as part of offboard cleanup.
|
||||
HOST2's own parked containers (from `PARTNERSHIP_OWN_CONTAINERS`) are restarted
|
||||
automatically, but if HOST2 had no pre-existing auth stack of its own, it needs one set up.
|
||||
|
||||
```bash
|
||||
# Check what's running:
|
||||
docker ps
|
||||
|
||||
# Verify own parked containers came back up:
|
||||
# (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in master_host2.conf)
|
||||
|
||||
# If you need a fresh auth stack, deploy from HOST2's own XML templates:
|
||||
docker create ... && docker start NginxProxyManager # etc.
|
||||
|
||||
# Check cert expiry — auto-renewal no longer happens via HOST1:
|
||||
cert_monitor.sh --dry-run
|
||||
```
|
||||
|
||||
If the offboard was unexpected or incomplete, check state first:
|
||||
```bash
|
||||
Partnership/partnership_manager.sh --status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### State Mismatch Between Servers
|
||||
|
||||
```bash
|
||||
# Check both state files:
|
||||
Partnership/partnership_manager.sh --status # shows both sides via SSH
|
||||
|
||||
# If one shows ACTIVE and other INACTIVE:
|
||||
# The INACTIVE side has already offboarded.
|
||||
# Run --offboard on the ACTIVE side to sync the state.
|
||||
|
||||
# If HOST2 unreachable:
|
||||
# HOST2 self-resolves on next --check when reachable.
|
||||
# HOST1 reads INACTIVE state → finalises from owner side automatically.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Auto-Offboard Triggered Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check the offline counter:
|
||||
cat /boot/config/partnership_offline_days.db
|
||||
|
||||
# Extended Tailscale outage may have incremented the counter.
|
||||
# Check Tailscale peer visibility:
|
||||
tailscale status
|
||||
|
||||
# If partnership should continue — re-onboard (first unblock the partner):
|
||||
Partnership/partnership_manager.sh --unblock unRAID-Jayred365
|
||||
Partnership/partnership_manager.sh --onboard --dry-run
|
||||
Partnership/partnership_manager.sh --onboard
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ STATE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
/boot/config/partnership_HOST1.db # HOST1 writes only
|
||||
/boot/config/partnership_HOST2.db # HOST2 writes only
|
||||
/boot/config/partnership_blocklist.db # hostname|timestamp|reason
|
||||
|
||||
# Example state file:
|
||||
state=ACTIVE
|
||||
last_updated=2026-05-14 03:00:00
|
||||
last_seen_remote=2026-05-14 03:00:00
|
||||
owner_host=HOST1
|
||||
offline_days=0
|
||||
```
|
||||
|
||||
Each server writes only its own state file. The other server reads via SSH.
|
||||
State propagates through SSH reads — no rsync, no shared filesystem.
|
||||
`/boot/config` — survives reboots, available before array starts, minimal flash wear.
|
||||
flock on all writes — prevents concurrent corruption from overlapping --check cycles.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ ROLE-BASED ACCESS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```
|
||||
HOST1 (owner) HOST2 (mirror)
|
||||
--onboard ✅ ❌ ownership is granted, not taken
|
||||
--offboard ✅ ✅ clean exit available to both parties
|
||||
--transfer ✅ ❌ owner only — mirror cannot self-promote
|
||||
--check ✅ ✅ both servers monitor state
|
||||
--status ✅ ✅ status is always available
|
||||
--unblock ✅ ✅ either server can clear its own blocklist
|
||||
```
|
||||
|
||||
AM_OWNER / AM_MIRROR flags set by `detect_hosts()` from `PARTNERSHIP_OWNER_HOST`.
|
||||
All routing decisions use these flags — no hostname string comparisons.
|
||||
Mirror attempting `--onboard` or `--transfer`: blocked with a clear error message.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ DESIGN NOTES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
**Why HOST1/HOST2 instead of PARTNERSHIP_OWNER/MIRROR?**
|
||||
HOST1 and HOST2 are already defined in master.conf — SSH keys, Tailscale names,
|
||||
all connection details. A single `PARTNERSHIP_OWNER_HOST` var flips ownership.
|
||||
Duplicating as `PARTNERSHIP_OWNER` and `PARTNERSHIP_MIRROR` would require maintaining
|
||||
identical values in two places. One source of truth for the host definition.
|
||||
|
||||
**Why does transfer require a confirmation string?**
|
||||
Transfer touches master.conf on both servers, reconfigures WebUIs, and flips sync
|
||||
direction. A misstep mid-sequence leaves both servers with different auth configs and
|
||||
no clear source of truth. The confirmation string makes accidental execution impossible,
|
||||
not just unlikely.
|
||||
|
||||
**Why do Tailscale removal and backup access expire at the same time?**
|
||||
If the mirror can't reach the owner's server via Tailscale, the backup is also
|
||||
unreachable. `PARTNERSHIP_GRACE_HOURS` controls both — one var, consistent behaviour.
|
||||
No misleading "data available for 30 days" when access is gone in 6 hours.
|
||||
|
||||
**Why does either server auto-offboard after 30 days offline?**
|
||||
30 consecutive days of missed sync cycles means the relationship has effectively ended
|
||||
regardless of intent. Auto-offboard makes the state official. Each server acts
|
||||
independently — no coordination required to finalise.
|
||||
|
||||
**Why are auth/arr stack containers removed on offboard?**
|
||||
The auth stack running on the mirror is the owner's stack — deployed from the owner's
|
||||
XML templates, managed by the owner. On offboard, the mirror restores its own parked
|
||||
containers (`PARTNERSHIP_OWN_CONTAINERS`) and sets up independent auth from scratch.
|
||||
Nothing from the owner's deployment lingers. Appdata is deleted alongside containers
|
||||
so there's no stale config left behind. Both fallback coverage containers (named
|
||||
`*-Owner`) and the owner-deployed stack (from `PARTNERSHIP_AUTH_STACK` / `PARTNERSHIP_ARR_STACK`)
|
||||
are removed — the mirror gets a clean slate.
|
||||
|
||||
**Why does XML ordering matter for the auth stack?**
|
||||
Authelia requires MariaDB and Redis to be running and accepting connections before it
|
||||
starts. `deploy_container_from_xml()` starts each container immediately after creating
|
||||
it. If Authelia is deployed before MariaDB, it will fail to connect and may not recover
|
||||
automatically. The array order in `HOST1_PARTNERSHIP_AUTH_STACK` is enforced by
|
||||
convention — database containers first, then applications.
|
||||
+136
-889
File diff suppressed because it is too large
Load Diff
+269
-337
@@ -1,93 +1,187 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Partnership Manager ========================================
|
||||
# ============================= Partnership Manager ============================================
|
||||
# ==============================================================================================
|
||||
# Manages the relationship lifecycle between two unRAID servers.
|
||||
# HOST1 is always the owner (source of truth). HOST2 is always the mirror.
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Manages the full lifecycle of a two-server partnership — onboard, daily health
|
||||
# monitoring, offboard, and ownership transfer. Called by partnership_onboard.sh
|
||||
# during initial setup, and by critical_sync_maintenance.sh every 15 minutes for
|
||||
# the --check mode. All other modes are run manually.
|
||||
#
|
||||
# PARTNERSHIP_OWNER_HOST flips to "HOST2" after a successful --transfer.
|
||||
# AM_OWNER / AM_MIRROR flags (set by detect_hosts) control all routing —
|
||||
# no hostname string comparisons anywhere in this script.
|
||||
#
|
||||
# ── MODES ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# --onboard ← owner only — establish mirror relationship
|
||||
# Reconfigures HOST2 auth WebUIs → HOST1 Tailscale IP
|
||||
# HOST2 clicks NPM → gets HOST1's NPM automatically
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# --offboard ← either server — clean separation
|
||||
# Mirror-initiated: writes INACTIVE state, reconfigures own WebUIs → localhost
|
||||
# Owner finalises on next --check: final sync, Tailscale removal
|
||||
# Owner-initiated: final sync, reconfigures mirror WebUIs, Tailscale removal
|
||||
# Both leave with current state, clean exit ✅
|
||||
# --onboard (owner only)
|
||||
# Reconfigures mirror auth WebUIs → owner's Tailscale IP
|
||||
# Mirror operator clicks NPM → gets owner's NPM via Tailscale automatically
|
||||
# Writes ACTIVE state on both servers
|
||||
#
|
||||
# --transfer ← owner only — flip ownership between servers
|
||||
# Requires confirmation string + consecutive health strikes
|
||||
# Reconfigures both servers, flips PARTNERSHIP_OWNER_HOST in master.conf
|
||||
# --offboard (either server)
|
||||
# Mirror-initiated: reconfigures own WebUIs → localhost, writes INACTIVE state
|
||||
# Owner finalises on next --check: final sync, Tailscale removal
|
||||
# Owner-initiated: final sync, reconfigures mirror WebUIs → localhost, removes
|
||||
# mirror containers + appdata, SSH key revocation, Tailscale removal
|
||||
# Both leave with clean state ✅
|
||||
#
|
||||
# --check ← called by critical_sync_maintenance.sh every 15min
|
||||
# Reads both state files via SSH
|
||||
# Detects offboard requests → finalises from owner side
|
||||
# Updates last_seen_remote timestamp
|
||||
# Increments offline counter → auto-offboards after threshold
|
||||
# Silent when healthy ← never noisy on clean runs
|
||||
# --transfer (owner only)
|
||||
# Reconfigures both servers, flips PARTNERSHIP_OWNER_HOST in master.conf
|
||||
# Requires confirmation string + consecutive health check passes
|
||||
#
|
||||
# --status ← either server — show current state, both sides
|
||||
# --check (called every 15min by critical_sync_maintenance.sh)
|
||||
# --remote-seen: rsync succeeded → reset offline counter, read remote state
|
||||
# --remote-unseen: rsync failed → increment offline counter → auto-offboard at threshold
|
||||
# Silent when healthy — never noisy on clean runs
|
||||
#
|
||||
# ── STATE FILES ───────────────────────────────────────────────────────────────────────────────
|
||||
# On /boot/config — survives reboots, available before array starts:
|
||||
# /boot/config/partnership_HOST1.db ← HOST1 writes only
|
||||
# /boot/config/partnership_HOST2.db ← HOST2 writes only
|
||||
# Propagated via SSH — no rsync needed
|
||||
# --status (either server)
|
||||
# Show state files from both servers, blocklist, SSH key status
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — all operations require root
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Role enforcement — mirror cannot run owner-only modes
|
||||
# AM_OWNER / AM_MIRROR — all routing via these flags, not hostname strings
|
||||
# version parity — onboard checks both servers match unRAID version
|
||||
# remote docker daemon — onboard verifies remote daemon responsive
|
||||
# SSH_TIMEOUT — all SSH calls timeout-protected
|
||||
# flock on state writes — prevents concurrent state file corruption
|
||||
# SIGTERM trap — grace period sleep interruptible
|
||||
# Silent by default — only warns/errors produce output (--check is always silent healthy)
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_PARTNERSHIP_AUTH_WEBUIS — containers reconfigured on onboard/offboard
|
||||
# HOST*_PARTNERSHIP_MIRROR_BACKUPS — paths available after offboard
|
||||
# All aliased by detect_hosts() — script uses PARTNERSHIP_AUTH_WEBUIS etc.
|
||||
# Deferred offboard
|
||||
# Either server can offboard without the other being reachable. The initiating
|
||||
# server writes its state immediately and becomes independent. The other server
|
||||
# reads the INACTIVE state on its next --check and finalises automatically.
|
||||
# No message passing, no coordination required.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# PARTNERSHIP_ENABLED — global enable gate
|
||||
# PARTNERSHIP_OWNER_HOST — "HOST1" or "HOST2" — flips on --transfer
|
||||
# PARTNERSHIP_REMOVE_TAILSCALE — remove mirror from tailnet on offboard
|
||||
# PARTNERSHIP_GRACE_HOURS — hours before Tailscale removal after offboard
|
||||
# PARTNERSHIP_OFFLINE_THRESHOLD — days unreachable before auto-offboard
|
||||
# PARTNERSHIP_TRANSFER_CONFIRM — exact string required for --transfer
|
||||
# PARTNERSHIP_TRANSFER_STRIKES — consecutive health checks required
|
||||
# PARTNERSHIP_TRANSFER_MAX_ATTEMPTS — max attempts before giving up
|
||||
# PARTNERSHIP_ONBOARD_VERIFY — verify WebUI connectivity after onboard
|
||||
# PARTNERSHIP_ONBOARD_NOTIFY — notify both servers on onboard completion
|
||||
# PARTNERSHIP_SYNC_INTERVAL — informational — actual schedule in cron
|
||||
# TAILSCALE_API_KEY / TAILSCALE_TAILNET — required when PARTNERSHIP_REMOVE_TAILSCALE=true
|
||||
# Appdata cleanup on offboard
|
||||
# Partner containers are stopped and removed. Their appdata bind-mount paths
|
||||
# (collected via docker inspect before removal) are also deleted. Safety gate:
|
||||
# only paths matching /mnt/*/appdata* are deleted — media and config shares
|
||||
# outside the appdata tree are never touched.
|
||||
#
|
||||
# ── BLOCKLIST ─────────────────────────────────────────────────────────────────────────────────
|
||||
# After offboard, the former partner's hostname is written to:
|
||||
# /boot/config/partnership_blocklist.db (format: hostname|timestamp|reason)
|
||||
# Blocklist is application-layer; Tailscale removal is network-layer
|
||||
# Both happen on offboard. The blocklist prevents re-onboard until explicitly
|
||||
# cleared with --unblock. Tailscale removal ends encrypted access at the network
|
||||
# level. Grace period controls both simultaneously — one var, consistent behaviour.
|
||||
#
|
||||
# --onboard is hard-blocked if the remote is on the blocklist — exits with error.
|
||||
# --check silently skips remote state reads for blocklisted hosts (no noise every 15min).
|
||||
# --unblock <hostname> removes an entry to permit re-onboarding.
|
||||
# --status shows the full blocklist.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# The blocklist persists until explicitly cleared — surviving reboots, array restarts,
|
||||
# and Tailscale reconnections. Tailscale removal is a separate step at the network layer;
|
||||
# the blocklist is the application-layer guard.
|
||||
# Root check
|
||||
# All operations require root.
|
||||
#
|
||||
# Role enforcement
|
||||
# Mirror cannot run --onboard or --transfer. Blocked with a clear error message.
|
||||
#
|
||||
# Version parity check
|
||||
# --onboard verifies both servers are on compatible unRAID versions.
|
||||
#
|
||||
# SSH_TIMEOUT on all remote calls
|
||||
# Every ssh/scp call is timeout-protected. No operation hangs on an unreachable peer.
|
||||
#
|
||||
# flock on state writes
|
||||
# Prevents concurrent state file corruption from overlapping --check cycles.
|
||||
#
|
||||
# SIGTERM trap on grace period sleep
|
||||
# Offboard grace period is interruptible — Ctrl-C aborts cleanly.
|
||||
#
|
||||
# Silent by default
|
||||
# --check produces no output when both servers are healthy. Only state changes
|
||||
# and threshold crossings produce output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# /boot/config/partnership_HOST1.db — HOST1 writes, HOST2 reads via SSH
|
||||
# /boot/config/partnership_HOST2.db — HOST2 writes, HOST1 reads via SSH
|
||||
# /boot/config/partnership_blocklist.db — hostname|timestamp|reason, persists until cleared
|
||||
#
|
||||
# On /boot/config — survives reboots, available before array starts, minimal flash wear.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# PARTNERSHIP_ENABLED
|
||||
# Global gate — set true once both servers are configured (default: false)
|
||||
#
|
||||
# PARTNERSHIP_OWNER_HOST
|
||||
# "HOST1" or "HOST2" — flips on --transfer (default: "HOST1")
|
||||
#
|
||||
# PARTNERSHIP_REMOVE_TAILSCALE
|
||||
# Remove mirror from tailnet on offboard (default: true)
|
||||
#
|
||||
# PARTNERSHIP_GRACE_HOURS
|
||||
# Hours before Tailscale removal after offboard — backup access also expires then (default: 6)
|
||||
#
|
||||
# PARTNERSHIP_OFFLINE_THRESHOLD
|
||||
# Days of missed sync cycles before auto-offboard triggers (default: 30)
|
||||
#
|
||||
# PARTNERSHIP_TRANSFER_CONFIRM
|
||||
# Exact string required for --transfer (default: "i-understand-this-transfers-ownership")
|
||||
#
|
||||
# PARTNERSHIP_TRANSFER_STRIKES
|
||||
# Consecutive health checks required before transfer proceeds (default: 3)
|
||||
#
|
||||
# PARTNERSHIP_TRANSFER_MAX_ATTEMPTS
|
||||
# Max health check attempts before giving up (default: 20)
|
||||
#
|
||||
# PARTNERSHIP_ONBOARD_VERIFY
|
||||
# Curl-verify each WebUI after reconfigure to confirm Tailscale routing works (default: true)
|
||||
#
|
||||
# PARTNERSHIP_ONBOARD_NOTIFY
|
||||
# Notify both servers on successful onboard (default: true)
|
||||
#
|
||||
# PARTNERSHIP_SYNC_INTERVAL
|
||||
# Informational — actual schedule is in cron (default: 15)
|
||||
#
|
||||
# TAILSCALE_API_KEY / TAILSCALE_TAILNET
|
||||
# Required when PARTNERSHIP_REMOVE_TAILSCALE=true
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_PARTNERSHIP_AUTH_WEBUIS
|
||||
# Containers reconfigured on onboard/offboard. Format: "ContainerName|WebUIPort"
|
||||
# Aliased by detect_hosts() → PARTNERSHIP_AUTH_WEBUIS
|
||||
#
|
||||
# HOST*_PARTNERSHIP_MIRROR_BACKUPS
|
||||
# Paths accessible to the partner during grace window after offboard.
|
||||
# Aliased by detect_hosts() → PARTNERSHIP_MIRROR_BACKUPS
|
||||
#
|
||||
# HOST*_PARTNERSHIP_OWN_CONTAINERS
|
||||
# Containers parked here during partnership, restarted on offboard.
|
||||
# Aliased by detect_hosts() → PARTNERSHIP_OWN_CONTAINERS
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# partnership_manager.sh --onboard
|
||||
# Establish mirror relationship — owner only
|
||||
#
|
||||
# partnership_manager.sh --offboard
|
||||
# Clean separation — either server. 10-second countdown before executing.
|
||||
#
|
||||
# partnership_manager.sh --offboard --dry-run
|
||||
# Show the complete offboard sequence without executing
|
||||
#
|
||||
# partnership_manager.sh --transfer --confirm=i-understand-this-transfers-ownership
|
||||
# Flip ownership — owner only. Requires exact confirmation string.
|
||||
#
|
||||
# partnership_manager.sh --check --remote-seen|--remote-unseen
|
||||
# Called by critical_sync_maintenance.sh every 15min — do not run manually
|
||||
#
|
||||
# partnership_manager.sh --status
|
||||
# Show state files, blocklist, SSH key status from both servers
|
||||
#
|
||||
# partnership_manager.sh --unblock <hostname>
|
||||
# Remove hostname from blocklist to permit re-onboarding
|
||||
#
|
||||
# Any mode supports --dry-run and --log
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# partnership_manage.sh --onboard
|
||||
# partnership_manage.sh --offboard
|
||||
# partnership_manage.sh --transfer --confirm=i-understand-this-transfers-ownership
|
||||
# partnership_manage.sh --check --remote-seen|--remote-unseen
|
||||
# partnership_manage.sh --status
|
||||
# partnership_manage.sh --unblock <hostname>
|
||||
# Any mode supports --dry-run and --log
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -152,10 +246,12 @@ MIRROR_ID=$( [[ "$OWNER_ID" == "HOST1" ]] && echo "HOST2" || echo "HOST1" )
|
||||
|
||||
OWNER="${!OWNER_ID}" # hostname string
|
||||
MIRROR="${!MIRROR_ID}"
|
||||
OWNER_SSH_KEY_VAR="${OWNER_ID}_SSH_KEY"
|
||||
MIRROR_SSH_KEY_VAR="${MIRROR_ID}_SSH_KEY"
|
||||
OWNER_SSH_KEY="${!OWNER_SSH_KEY_VAR}"
|
||||
MIRROR_SSH_KEY="${!MIRROR_SSH_KEY_VAR}"
|
||||
# SSH_KEY (set by detect_hosts) is this server's own private key.
|
||||
# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
|
||||
# With sparse checkout, each server only has its own master_host{N}.conf — the other server's
|
||||
# key path is never available here. Use SSH_KEY for all outbound SSH regardless of mode.
|
||||
MIRROR_SSH_KEY="$SSH_KEY"
|
||||
OWNER_SSH_KEY="$SSH_KEY"
|
||||
|
||||
AM_OWNER=false
|
||||
AM_MIRROR=false
|
||||
@@ -169,34 +265,36 @@ OWNER_STATE_FILE="/boot/config/partnership_${OWNER}.db"
|
||||
MIRROR_STATE_FILE="/boot/config/partnership_${MIRROR}.db"
|
||||
OFFLINE_COUNTER="/boot/config/partnership_offline_days.db"
|
||||
|
||||
if [[ -z "$MODE" ]]; then
|
||||
error "No mode specified"
|
||||
echo "Usage:"
|
||||
echo " partnership_manage.sh --onboard"
|
||||
echo " partnership_manage.sh --offboard"
|
||||
echo " partnership_manage.sh --transfer --confirm=..."
|
||||
echo " partnership_manage.sh --check --remote-seen|--remote-unseen"
|
||||
echo " partnership_manage.sh --status"
|
||||
echo " partnership_manage.sh --unblock <hostname>"
|
||||
exit 1
|
||||
if [[ "${PARTNERSHIP_LIB_MODE:-}" != "1" ]]; then
|
||||
if [[ -z "$MODE" ]]; then
|
||||
error "No mode specified"
|
||||
echo "Usage:"
|
||||
echo " partnership_manager.sh --onboard"
|
||||
echo " partnership_manager.sh --offboard"
|
||||
echo " partnership_manager.sh --transfer --confirm=..."
|
||||
echo " partnership_manager.sh --check --remote-seen|--remote-unseen"
|
||||
echo " partnership_manager.sh --status"
|
||||
echo " partnership_manager.sh --unblock <hostname>"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Role-based access control
|
||||
if [[ "$AM_MIRROR" == true ]]; then
|
||||
case "$MODE" in
|
||||
onboard|transfer)
|
||||
error "Only the owner ($OWNER / $OWNER_ID) can run --$MODE"
|
||||
error "Run from $OWNER or use --offboard to separate cleanly"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
|
||||
# Lock for all modes except --check (frequent) and --offboard (offboard script holds its own)
|
||||
[[ "$MODE" != "check" && "$MODE" != "offboard" ]] && acquire_lock "strict"
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
fi
|
||||
|
||||
# Role-based access control
|
||||
if [[ "$AM_MIRROR" == true ]]; then
|
||||
case "$MODE" in
|
||||
onboard|transfer)
|
||||
error "Only the owner ($OWNER / $OWNER_ID) can run --$MODE"
|
||||
error "Run from $OWNER or use --offboard to separate cleanly"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
|
||||
# Lock for all modes except --check (check is called frequently, lock would pile up)
|
||||
[[ "$MODE" != "check" ]] && acquire_lock "strict"
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER FUNCTIONS ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
@@ -678,19 +776,39 @@ start_own_stack() {
|
||||
done
|
||||
}
|
||||
|
||||
# Remove partnership containers on this server.
|
||||
# Remove partnership containers on this server + their appdata bind-mount paths.
|
||||
# Uses FolderView3 folder if enabled (precise list), else falls back to FALLBACK_*_COVERS_* config.
|
||||
# Appdata paths collected via docker inspect BEFORE removal — inspect fails on removed containers.
|
||||
# Safety gate: only paths matching /mnt/*/appdata* are deleted.
|
||||
cleanup_partner_containers() {
|
||||
local folder_name="$1"
|
||||
declare -a containers=()
|
||||
gather_partner_fallback_containers containers
|
||||
|
||||
if [[ ${#containers[@]} -eq 0 ]]; then
|
||||
log "No partner containers found to remove"
|
||||
if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]]; then
|
||||
folderview3_remove_partner_folder "$folder_name"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Collect appdata paths BEFORE any removal — inspect fails once container is gone
|
||||
local all_appdata_paths=""
|
||||
for container in "${containers[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
if timeout "${DOCKER_TIMEOUT:-30}" docker inspect "$container" >/dev/null 2>&1; then
|
||||
local cpaths
|
||||
cpaths=$(docker inspect --format '{{range .HostConfig.Binds}}{{println .}}{{end}}' \
|
||||
"$container" 2>/dev/null | awk -F: '{print $1}' | grep '^/mnt/.*/appdata')
|
||||
[[ -n "$cpaths" ]] && all_appdata_paths+=$'\n'"$cpaths"
|
||||
fi
|
||||
done
|
||||
|
||||
# Remove containers
|
||||
if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]]; then
|
||||
folderview3_remove_partner_folder "$folder_name"
|
||||
else
|
||||
declare -a containers=()
|
||||
gather_partner_fallback_containers containers
|
||||
if [[ ${#containers[@]} -eq 0 ]]; then
|
||||
log "No partner containers found to remove"
|
||||
return 0
|
||||
fi
|
||||
for container in "${containers[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
@@ -706,9 +824,22 @@ cleanup_partner_containers() {
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# Delete appdata after containers are gone
|
||||
while IFS= read -r path; do
|
||||
[[ -z "$path" ]] && continue
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn " DRY RUN — would rm -rf $path"
|
||||
continue
|
||||
fi
|
||||
rm -rf "$path" && log " Appdata removed: $path ✅" || warn " Failed to remove: $path"
|
||||
done <<< "$all_appdata_paths"
|
||||
}
|
||||
|
||||
# SSH to mirror — remove all containers named *-${OWNER_SHORT} (owner's deployed containers).
|
||||
# SSH to mirror — remove all containers named *-${OWNER_SHORT} (owner's deployed containers)
|
||||
# and their appdata bind-mount paths.
|
||||
# Appdata paths collected via SSH docker inspect before removal, then deleted via SSH.
|
||||
# Safety gate: only paths matching /mnt/*/appdata* are deleted on the remote.
|
||||
cleanup_owner_containers_on_mirror() {
|
||||
local mirror_ip="$1"
|
||||
local owner_short
|
||||
@@ -716,7 +847,7 @@ cleanup_owner_containers_on_mirror() {
|
||||
|
||||
log "Removing owner-deployed containers from $MIRROR..."
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would remove *-${owner_short} containers from $MIRROR"
|
||||
warn "DRY RUN — would remove *-${owner_short} containers + appdata from $MIRROR"
|
||||
return 0
|
||||
fi
|
||||
|
||||
@@ -732,6 +863,14 @@ cleanup_owner_containers_on_mirror() {
|
||||
|
||||
while IFS= read -r container; do
|
||||
[[ -z "$container" ]] && continue
|
||||
|
||||
# Collect appdata paths before removal
|
||||
local appdata_paths
|
||||
appdata_paths=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" root@"$mirror_ip" \
|
||||
"docker inspect --format '{{range .HostConfig.Binds}}{{println .}}{{end}}' '$container' 2>/dev/null \
|
||||
| awk -F: '{print \$1}' | grep '^/mnt/.*/appdata'" 2>/dev/null)
|
||||
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" root@"$mirror_ip" \
|
||||
"docker stop '$container' >/dev/null 2>&1
|
||||
@@ -739,6 +878,16 @@ cleanup_owner_containers_on_mirror() {
|
||||
grep -q removed && \
|
||||
log "$container removed from $MIRROR ✅" || \
|
||||
warn "Failed to remove $container from $MIRROR"
|
||||
|
||||
# Delete appdata on remote after container removal
|
||||
while IFS= read -r path; do
|
||||
[[ -z "$path" ]] && continue
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" root@"$mirror_ip" \
|
||||
"rm -rf '$path' && echo removed" 2>/dev/null | grep -q removed && \
|
||||
log " Appdata removed on $MIRROR: $path ✅" || \
|
||||
warn " Failed to remove appdata on $MIRROR: $path"
|
||||
done <<< "$appdata_paths"
|
||||
done <<< "$container_list"
|
||||
}
|
||||
|
||||
@@ -983,6 +1132,11 @@ update_master_conf() {
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Library-mode guard — source only, skip all mode dispatch ─────────────────────────────────
|
||||
# partnership_offboard.sh sources this file with PARTNERSHIP_LIB_MODE=1 to get helper
|
||||
# functions without triggering any mode execution.
|
||||
[[ "${PARTNERSHIP_LIB_MODE:-}" == "1" ]] && { return 0 2>/dev/null || exit 0; }
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Unblock ━━━
|
||||
# ==============================================================================================
|
||||
@@ -1166,8 +1320,8 @@ if [[ "$MODE" == "check" ]]; then
|
||||
REMOTE_CONTENT=$(read_remote_state "$REMOTE_IP" "$SSH_KEY" "$REMOTE_STATE_FILE")
|
||||
if [[ -z "$REMOTE_CONTENT" ]]; then
|
||||
# IP resolved but SSH returned nothing — could be auth failure, not just missing file
|
||||
if [[ -f "$SCRIPT_DIR/../Initial_run/ssh_setup.sh" ]]; then
|
||||
bash "$SCRIPT_DIR/../Initial_run/ssh_setup.sh" --validate 2>/dev/null || true
|
||||
if [[ -f "$SCRIPT_DIR/ssh_setup.sh" ]]; then
|
||||
bash "$SCRIPT_DIR/ssh_setup.sh" --validate 2>/dev/null || true
|
||||
fi
|
||||
log "Partnership check — remote state file not found"
|
||||
exit 0
|
||||
@@ -1383,229 +1537,7 @@ fi
|
||||
# ━━━ Offboard ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$MODE" == "offboard" ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_FALLBACK Offboard — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
|
||||
# Check already offboarded
|
||||
if [[ -f "$LOCAL_STATE_FILE" ]]; then
|
||||
CURRENT_STATE=$(read_state_file "$LOCAL_STATE_FILE" "state")
|
||||
if [[ "$CURRENT_STATE" == "INACTIVE" ]]; then
|
||||
warn "Partnership already INACTIVE — use --status to verify both servers agree"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Mirror-initiated offboard ─────────────────────────────────────────────────────────────
|
||||
if [[ "$AM_MIRROR" == true ]]; then
|
||||
warn "$MIRROR_ID ($MIRROR) is initiating offboard"
|
||||
warn "Local auth WebUIs will be reconfigured → localhost"
|
||||
warn "$OWNER_ID ($OWNER) will finalise on its next --check cycle"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
echo ""
|
||||
echo "You have 10 seconds to cancel (Ctrl+C)..."
|
||||
sleep 10
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Reconfigure Local WebUIs → localhost ━━━"
|
||||
reconfigure_local_webuis "localhost"
|
||||
|
||||
# Remove owner's deployed containers + restart own stack
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Cleanup Partner Containers ━━━"
|
||||
PARTNER_FOLDER_NAME=$(derive_partner_folder_name "$OWNER")
|
||||
cleanup_partner_containers "$PARTNER_FOLDER_NAME"
|
||||
start_own_stack
|
||||
|
||||
NOW=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
write_state_file "$LOCAL_STATE_FILE" \
|
||||
"INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
|
||||
log "Local state: INACTIVE ✅"
|
||||
|
||||
[[ "$DRY_RUN" == false ]] && add_to_blocklist "$OWNER" "$REASON"
|
||||
|
||||
OWNER_IP=$(resolve_tailscale_ip "$OWNER")
|
||||
if [[ -n "$OWNER_IP" ]]; then
|
||||
push_state_to_remote "$LOCAL_STATE_FILE" "$OWNER_IP" "$MIRROR_SSH_KEY"
|
||||
notify "Partnership offboard requested by $MIRROR — $OWNER will finalise on next check" \
|
||||
"Partnership" "normal"
|
||||
else
|
||||
warn "$OWNER unreachable — state written locally, owner will see it when reachable"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY OFFBOARD SUMMARY ━━━━━"
|
||||
echo " Your WebUIs: reconfigured → localhost ✅"
|
||||
echo " State: INACTIVE ✅"
|
||||
echo " Blocklist: $OWNER blocked — re-onboard to permit access again ✅"
|
||||
echo " Owner: will finalise + final sync on next --check ✅"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Owner-initiated offboard ──────────────────────────────────────────────────────────────
|
||||
warn "Offboarding $MIRROR_ID ($MIRROR) from partnership"
|
||||
warn "Final sync will run — mirror leaves with current state"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
echo ""
|
||||
echo "You have 10 seconds to cancel (Ctrl+C)..."
|
||||
sleep 10
|
||||
echo "Proceeding..."
|
||||
fi
|
||||
|
||||
resolve_remote_ip
|
||||
|
||||
# Stop any running rsync first
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Stop Running Rsync ━━━"
|
||||
bash "$SCRIPT_DIR/../Rsync/rsync_stop.sh" --rsync-only 2>/dev/null || true
|
||||
|
||||
# Final sync
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Final Sync ━━━"
|
||||
do_final_sync
|
||||
|
||||
MIRROR_IP=$(resolve_tailscale_ip "$MIRROR")
|
||||
MIRROR_REACHABLE=false
|
||||
[[ -n "$MIRROR_IP" ]] && MIRROR_REACHABLE=true
|
||||
|
||||
# Reconfigure mirror WebUIs → localhost
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Reconfigure Mirror WebUIs → localhost ━━━"
|
||||
|
||||
WEBUI_FAILURES=0
|
||||
if [[ "$MIRROR_REACHABLE" == true ]]; then
|
||||
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
|
||||
[[ -z "$entry" ]] && continue
|
||||
container="${entry%%|*}"
|
||||
port="${entry##*|}"
|
||||
reconfigure_webui "$container" "$port" "localhost" \
|
||||
"$MIRROR_SSH_KEY" "$MIRROR_IP" "$MIRROR" || (( WEBUI_FAILURES++ ))
|
||||
done
|
||||
else
|
||||
warn "$MIRROR unreachable — WebUI reconfiguration skipped"
|
||||
warn "$MIRROR will reconfigure its own WebUIs when it sees INACTIVE state on --check"
|
||||
(( WEBUI_FAILURES++ ))
|
||||
fi
|
||||
|
||||
# Disable critical rsync
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Disable Critical Sync ━━━"
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
update_master_conf "CRITICAL_RSYNC_ENABLED" "false"
|
||||
warn "CRITICAL_RSYNC_ENABLED=false ✅"
|
||||
else
|
||||
warn "DRY RUN — would set CRITICAL_RSYNC_ENABLED=false"
|
||||
fi
|
||||
|
||||
# Write state files
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Write State ━━━"
|
||||
NOW=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
|
||||
write_state_file "$LOCAL_STATE_FILE" \
|
||||
"INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
|
||||
log "Local state: INACTIVE ✅"
|
||||
|
||||
[[ "$DRY_RUN" == false ]] && add_to_blocklist "$MIRROR" "$REASON"
|
||||
|
||||
if [[ "$MIRROR_REACHABLE" == true ]]; then
|
||||
push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"
|
||||
fi
|
||||
|
||||
# Local: remove fallback-coverage containers for mirror, restart own stack
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Local Container Cleanup ━━━"
|
||||
PARTNER_FOLDER_NAME=$(derive_partner_folder_name "$MIRROR")
|
||||
cleanup_partner_containers "$PARTNER_FOLDER_NAME"
|
||||
start_own_stack
|
||||
|
||||
# Remote: remove owner's deployed containers from mirror, restart mirror's own stack
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Remote Container Cleanup ━━━"
|
||||
if [[ "$MIRROR_REACHABLE" == true ]]; then
|
||||
cleanup_owner_containers_on_mirror "$MIRROR_IP"
|
||||
start_mirror_own_stack "$MIRROR_IP"
|
||||
else
|
||||
warn "$MIRROR unreachable — remote container cleanup skipped"
|
||||
warn "Run 'partnership_manager.sh --offboard' on $MIRROR to clean up manually"
|
||||
fi
|
||||
|
||||
# Emby admin revocation — before SSH key revocation while Emby still reachable
|
||||
[[ "$MIRROR_REACHABLE" == true ]] && revoke_emby_admin "$MIRROR_IP"
|
||||
|
||||
# SSH key revocation — mutual, both directions
|
||||
# Must run before Tailscale removal (SSH needs network) and after state is pushed
|
||||
SSH_REVOKE_REMOTE_OK=false
|
||||
SSH_REVOKE_LOCAL_OK=false
|
||||
do_ssh_key_revocation "${MIRROR_IP:-}"
|
||||
|
||||
# Tailscale removal
|
||||
if [[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_NET Tailscale Separation ━━━"
|
||||
if [[ "$MIRROR_REACHABLE" == true ]]; then
|
||||
grace_seconds=$(( ${PARTNERSHIP_GRACE_HOURS:-6} * 3600 ))
|
||||
warn "Waiting ${PARTNERSHIP_GRACE_HOURS:-6}hr grace — mirror can collect backups..."
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
trap 'warn "Offboard interrupted during grace sleep"; exit 0' SIGTERM SIGINT
|
||||
sleep "$grace_seconds"
|
||||
trap - SIGTERM SIGINT
|
||||
fi
|
||||
fi
|
||||
remove_tailscale_device "$MIRROR"
|
||||
fi
|
||||
|
||||
# Backup notification
|
||||
if [[ ${#PARTNERSHIP_MIRROR_BACKUPS[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_DISK Backup Handover ━━━"
|
||||
log "Backups available for $MIRROR:"
|
||||
for path in "${PARTNERSHIP_MIRROR_BACKUPS[@]}"; do
|
||||
[[ -z "$path" ]] && continue
|
||||
echo " $path"
|
||||
done
|
||||
notify "$MIRROR offboard complete — backups available for ${PARTNERSHIP_GRACE_HOURS:-6}hr. Tailscale access expires then." \
|
||||
"Partnership" "warning"
|
||||
fi
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY OFFBOARD SUMMARY ━━━━━"
|
||||
echo " Owner: $OWNER_ID ($OWNER)"
|
||||
echo " Mirror: $MIRROR_ID ($MIRROR)"
|
||||
echo " Final sync: complete ✅"
|
||||
echo " WebUI failures: $WEBUI_FAILURES"
|
||||
echo " Critical rsync: disabled ✅"
|
||||
echo " State: INACTIVE ✅"
|
||||
echo " Blocklist: $MIRROR blocked — re-onboard to permit access again ✅"
|
||||
_revoke_status() {
|
||||
if [[ "${SSH_REVOKE_REMOTE_OK:-false}" == true ]] && [[ "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
|
||||
echo "both directions ✅"
|
||||
elif [[ "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
|
||||
echo "local only ✅ — remote failed (revoke manually on $MIRROR)"
|
||||
else
|
||||
echo "⚠️ failed — check warnings above"
|
||||
fi
|
||||
}
|
||||
echo " Keys revoked: $(_revoke_status)"
|
||||
[[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]] && \
|
||||
echo " FolderView3: ${PARTNER_FOLDER_NAME:-} cleaned ✅"
|
||||
[[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]] && \
|
||||
echo " Tailscale: $MIRROR removed ✅"
|
||||
echo ""
|
||||
echo " $MIRROR leaves with:"
|
||||
echo " ✓ Current auth config (final sync)"
|
||||
echo " ✓ Auth WebUIs → localhost"
|
||||
echo " ✓ ${PARTNERSHIP_GRACE_HOURS:-6}hr to collect backups"
|
||||
echo " ✓ Full ecosystem — just stop the sync"
|
||||
echo ""
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
|
||||
warn "$ICON_DONE DONE — clean separation complete ✅"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
exec bash "$SCRIPT_DIR/partnership_offboard.sh" "${FILTERED_ARGS[@]}" --reason="$REASON"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
|
||||
Executable
+678
@@ -0,0 +1,678 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Partnership Offboard ===========================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Cleanly ends a partnership. Role is detected automatically — run on either server.
|
||||
# Owner path runs the full sequence including remote cleanup and final sync.
|
||||
# Mirror path handles the local side and signals the owner to complete its own cleanup.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# OWNER PATH (10 steps)
|
||||
# Step 1: Stop rsync — halt any running sync before state changes
|
||||
# Step 2: Final sync — mirror leaves with current Critical-Data state
|
||||
# Step 3: Reconfigure WebUIs — mirror's auth WebUIs → localhost
|
||||
# Step 4: Disable sync — CRITICAL_RSYNC_ENABLED=false in master.conf
|
||||
# Step 5: Write state — INACTIVE locally + pushed to mirror, mirror blocklisted
|
||||
# Step 6: Local cleanup — remove fallback coverage containers + appdata
|
||||
# Step 7: Restart own stack — bring up owner's own parked containers
|
||||
# Step 8: Remote cleanup — remove auth/arr stack + fallback containers from mirror
|
||||
# Step 9: Restart mirror — bring up mirror's own parked containers
|
||||
# Step 10: Revocation — Emby admin, SSH keys, Tailscale device
|
||||
#
|
||||
# MIRROR PATH (8 steps)
|
||||
# Step 1: Stop rsync — halt any running sync
|
||||
# Step 2: Reconfigure WebUIs — local auth WebUIs → localhost
|
||||
# Step 3: Remote stack clean — remove owner-deployed containers locally (auth/arr stack)
|
||||
# Step 4: Fallback cleanup — remove fallback coverage containers
|
||||
# Step 5: Disable sync — CRITICAL_RSYNC_ENABLED=false in master.conf
|
||||
# Step 6: Revoke Emby admin — remove own admin account from local Emby instance
|
||||
# Step 7: Restart own stack — bring up own parked containers
|
||||
# Step 8: SSH revocation — revoke keys both directions, write state, signal owner
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_PARTNERSHIP_AUTH_STACK
|
||||
# Auth container XMLs to push during onboard — used on offboard to identify what
|
||||
# to remove. Owner's PARTNERSHIP_AUTH_STACK determines which containers get removed
|
||||
# from the mirror on both owner-initiated and mirror-initiated offboard.
|
||||
#
|
||||
# HOST*_PARTNERSHIP_ARR_STACK
|
||||
# Arr container XMLs — same cleanup logic as auth stack.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Partnership/partnership_offboard.sh
|
||||
# Full offboard — role detected automatically
|
||||
#
|
||||
# Partnership/partnership_offboard.sh --dry-run
|
||||
# Preview all steps without executing
|
||||
#
|
||||
# Partnership/partnership_offboard.sh --log
|
||||
# Verbose per-step output
|
||||
#
|
||||
# Partnership/partnership_offboard.sh --reason=<string>
|
||||
# Tag the offboard reason in state file and blocklist (default: manual)
|
||||
# Called by partnership_manager.sh --offboard (reason passed through)
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SCRIPTS_ROOT="$SCRIPT_DIR/.."
|
||||
TEMPLATES_DIR="/boot/config/plugins/dockerMan/templates-user"
|
||||
SSH_TIMEOUT=15
|
||||
|
||||
source "$SCRIPTS_ROOT/load_config.sh"
|
||||
|
||||
# ── Parse flags ───────────────────────────────────────────────────────────────────────────────
|
||||
REASON="manual"
|
||||
FILTERED_ARGS=()
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--reason=*) REASON="${arg#--reason=}" ;;
|
||||
*) FILTERED_ARGS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
parse_args "${FILTERED_ARGS[@]}"
|
||||
|
||||
# ── Source partnership_manager.sh for shared helpers ──────────────────────────────────────────
|
||||
# PARTNERSHIP_LIB_MODE=1 skips mode dispatch — functions are defined, nothing is executed.
|
||||
PARTNERSHIP_LIB_MODE=1 source "$SCRIPT_DIR/partnership_manager.sh"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
[[ "$EUID" -ne 0 ]] && { error "Must be run as root"; exit 1; }
|
||||
|
||||
detect_hosts
|
||||
|
||||
OWNER_ID="${PARTNERSHIP_OWNER_HOST:-HOST1}"
|
||||
MIRROR_ID=$( [[ "$OWNER_ID" == "HOST1" ]] && echo "HOST2" || echo "HOST1" )
|
||||
OWNER="${!OWNER_ID}"
|
||||
MIRROR="${!MIRROR_ID}"
|
||||
MIRROR_SSH_KEY="$SSH_KEY"
|
||||
OWNER_SSH_KEY="$SSH_KEY"
|
||||
|
||||
AM_OWNER=false
|
||||
AM_MIRROR=false
|
||||
[[ "$MY_ID" == "$OWNER_ID" ]] && AM_OWNER=true
|
||||
[[ "$MY_ID" == "$MIRROR_ID" ]] && AM_MIRROR=true
|
||||
|
||||
LOCAL_STATE_FILE="/boot/config/partnership_${LOCAL_SERVER_NAME}.db"
|
||||
REMOTE_STATE_FILE="/boot/config/partnership_${REMOTE_SERVER_NAME}.db"
|
||||
OWNER_STATE_FILE="/boot/config/partnership_${OWNER}.db"
|
||||
MIRROR_STATE_FILE="/boot/config/partnership_${MIRROR}.db"
|
||||
OFFLINE_COUNTER="/boot/config/partnership_offline_days.db"
|
||||
|
||||
acquire_lock "strict"
|
||||
|
||||
# Check already offboarded
|
||||
if [[ -f "$LOCAL_STATE_FILE" ]]; then
|
||||
CURRENT_STATE=$(read_state_file "$LOCAL_STATE_FILE" "state")
|
||||
if [[ "$CURRENT_STATE" == "INACTIVE" ]]; then
|
||||
warn "Partnership already INACTIVE — use partnership_manager.sh --status to verify both servers agree"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
START=$(date +%s)
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_FALLBACK Partnership Offboard — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo ""
|
||||
echo " Role: $( [[ "$AM_OWNER" == true ]] && echo "OWNER" || echo "MIRROR" )"
|
||||
echo " This: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo " Partner: $( [[ "$AM_OWNER" == true ]] && echo "$MIRROR_ID ($MIRROR)" || echo "$OWNER_ID ($OWNER)" )"
|
||||
echo " Reason: $REASON"
|
||||
echo ""
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permanent changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER: remove owner-deployed containers from a remote host ───────────────────────────────
|
||||
#
|
||||
# Uses PARTNERSHIP_AUTH_STACK + PARTNERSHIP_ARR_STACK arrays (owner's conf) to derive
|
||||
# container names from local XML templates. SSHes to remote to stop, remove, and delete
|
||||
# appdata. Appdata paths are collected via docker inspect before removal so they aren't
|
||||
# lost once the container is gone. Safety gate: only /mnt/*/appdata* paths are deleted.
|
||||
# ==============================================================================================
|
||||
cleanup_deployed_stack_on_remote() {
|
||||
local remote_ip="$1" ssh_key="$2"
|
||||
local -a xml_names=()
|
||||
[[ ${#PARTNERSHIP_AUTH_STACK[@]} -gt 0 ]] && xml_names+=("${PARTNERSHIP_AUTH_STACK[@]}")
|
||||
[[ ${#PARTNERSHIP_ARR_STACK[@]} -gt 0 ]] && xml_names+=("${PARTNERSHIP_ARR_STACK[@]}")
|
||||
|
||||
if [[ ${#xml_names[@]} -eq 0 ]]; then
|
||||
log "No auth/arr stack arrays configured — skipping deployed stack cleanup"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log "Removing owner-deployed containers (auth/arr stacks) from $MIRROR..."
|
||||
for xml_name in "${xml_names[@]}"; do
|
||||
[[ -z "$xml_name" ]] && continue
|
||||
local xml_file="${TEMPLATES_DIR}/${xml_name}"
|
||||
if [[ ! -f "$xml_file" ]]; then
|
||||
warn " $xml_name not found in local $TEMPLATES_DIR — skipping"
|
||||
continue
|
||||
fi
|
||||
|
||||
local cname
|
||||
cname=$(awk 'match($0,/<Name>([^<]+)<\/Name>/,a){print a[1];exit}' "$xml_file")
|
||||
[[ -z "$cname" ]] && continue
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn " DRY RUN — would stop + rm $cname on $MIRROR"
|
||||
warn " DRY RUN — would delete appdata for $cname on $MIRROR"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Collect appdata paths via docker inspect before removal
|
||||
local appdata_paths
|
||||
appdata_paths=$(timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$remote_ip" \
|
||||
"docker inspect --format '{{range .HostConfig.Binds}}{{println .}}{{end}}' '$cname' 2>/dev/null \
|
||||
| awk -F: '{print \$1}' | grep '^/mnt/.*/appdata'" 2>/dev/null)
|
||||
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" root@"$remote_ip" \
|
||||
"docker stop '$cname' >/dev/null 2>&1
|
||||
docker rm '$cname' >/dev/null 2>&1 && echo removed" 2>/dev/null | \
|
||||
grep -q removed && \
|
||||
log " $cname removed from $MIRROR ✅" || \
|
||||
log " $cname not found on $MIRROR — skipping"
|
||||
|
||||
while IFS= read -r path; do
|
||||
[[ -z "$path" ]] && continue
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" root@"$remote_ip" \
|
||||
"rm -rf '$path' && echo removed" 2>/dev/null | grep -q removed && \
|
||||
log " Appdata removed on $MIRROR: $path ✅" || \
|
||||
warn " Failed to remove appdata on $MIRROR: $path"
|
||||
done <<< "$appdata_paths"
|
||||
done
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER: remove owner-deployed containers locally (mirror-initiated offboard) ─────────────
|
||||
#
|
||||
# SSHes to owner to read PARTNERSHIP_AUTH_STACK + PARTNERSHIP_ARR_STACK, then uses the
|
||||
# local templates-user/ copies (SCPed there during onboard) to get container names and
|
||||
# appdata paths. Appdata collected before removal. Skips gracefully if owner unreachable.
|
||||
# ==============================================================================================
|
||||
cleanup_deployed_stack_locally() {
|
||||
local owner_ip="$1" ssh_key="$2"
|
||||
local -a xml_names=()
|
||||
|
||||
if [[ -n "$owner_ip" ]]; then
|
||||
local -a auth_arr arr_arr
|
||||
mapfile -t auth_arr < <(timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$owner_ip" \
|
||||
"source '$SCRIPTS_ROOT/load_config.sh' 2>/dev/null
|
||||
detect_hosts 2>/dev/null
|
||||
printf '%s\n' \"\${PARTNERSHIP_AUTH_STACK[@]:-}\"" 2>/dev/null | grep -v '^$')
|
||||
mapfile -t arr_arr < <(timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$owner_ip" \
|
||||
"source '$SCRIPTS_ROOT/load_config.sh' 2>/dev/null
|
||||
detect_hosts 2>/dev/null
|
||||
printf '%s\n' \"\${PARTNERSHIP_ARR_STACK[@]:-}\"" 2>/dev/null | grep -v '^$')
|
||||
xml_names=("${auth_arr[@]}" "${arr_arr[@]}")
|
||||
fi
|
||||
|
||||
if [[ ${#xml_names[@]} -eq 0 ]]; then
|
||||
log "Could not read deployed stack from owner — skipping auth/arr cleanup"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log "Removing owner-deployed containers (auth/arr stacks) locally..."
|
||||
for xml_name in "${xml_names[@]}"; do
|
||||
[[ -z "$xml_name" ]] && continue
|
||||
local xml_file="${TEMPLATES_DIR}/${xml_name}"
|
||||
if [[ ! -f "$xml_file" ]]; then
|
||||
warn " $xml_name not found locally — skipping"
|
||||
continue
|
||||
fi
|
||||
|
||||
local cname
|
||||
cname=$(awk 'match($0,/<Name>([^<]+)<\/Name>/,a){print a[1];exit}' "$xml_file")
|
||||
[[ -z "$cname" ]] && continue
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn " DRY RUN — would stop + rm $cname"
|
||||
warn " DRY RUN — would delete appdata for $cname"
|
||||
continue
|
||||
fi
|
||||
|
||||
local appdata_paths=""
|
||||
if timeout "${DOCKER_TIMEOUT:-30}" docker inspect "$cname" >/dev/null 2>&1; then
|
||||
appdata_paths=$(docker inspect \
|
||||
--format '{{range .HostConfig.Binds}}{{println .}}{{end}}' \
|
||||
"$cname" 2>/dev/null | awk -F: '{print $1}' | grep '^/mnt/.*/appdata')
|
||||
timeout "${DOCKER_TIMEOUT:-30}" docker stop "$cname" >/dev/null 2>&1 || true
|
||||
timeout "${DOCKER_TIMEOUT:-30}" docker rm "$cname" >/dev/null 2>&1 && \
|
||||
log " $cname removed ✅" || warn " $cname rm failed"
|
||||
else
|
||||
log " $cname not found locally — skipping"
|
||||
fi
|
||||
|
||||
while IFS= read -r path; do
|
||||
[[ -z "$path" ]] && continue
|
||||
rm -rf "$path" && log " Appdata removed: $path ✅" || warn " Failed to remove: $path"
|
||||
done <<< "$appdata_paths"
|
||||
done
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER: revoke own admin account from local Emby instance ────────────────────────────────
|
||||
#
|
||||
# Mirror-initiated path only. Called before start_own_stack so Emby is still running.
|
||||
# Uses local EMBY_API_KEY and the mirror's own short name as the username to delete.
|
||||
# ==============================================================================================
|
||||
revoke_local_emby_admin() {
|
||||
local emby_port="${PARTNERSHIP_EMBY_PORT:-8096}"
|
||||
local emby_url="http://127.0.0.1:${emby_port}"
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_EMBY Emby Admin Revocation ━━━"
|
||||
|
||||
if [[ "${PARTNERSHIP_PROVISION_EMBY_ADMIN:-false}" != true ]]; then
|
||||
log "PARTNERSHIP_PROVISION_EMBY_ADMIN=false — skipping"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -z "${EMBY_API_KEY:-}" ]]; then
|
||||
warn "EMBY_API_KEY not set — skipping local Emby admin revocation"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# The account to revoke is this server's own short name (the mirror user's account)
|
||||
local username="${PARTNERSHIP_EMBY_ADMIN_USER:-$(derive_short_name "$LOCAL_SERVER_NAME")}"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would delete Emby admin '$username' at $emby_url"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log "Looking up Emby user '$username' at $emby_url..."
|
||||
|
||||
local users_json user_id
|
||||
users_json=$(curl -sf --max-time 15 \
|
||||
-H "X-Emby-Authorization: MediaBrowser Token=\"$EMBY_API_KEY\"" \
|
||||
"${emby_url}/Users" 2>/dev/null)
|
||||
|
||||
user_id=$(echo "$users_json" | \
|
||||
grep -o "\"Id\":\"[^\"]*\"[^}]*\"Name\":\"${username}\"" | \
|
||||
grep -o '"Id":"[^"]*"' | cut -d'"' -f4 | head -1)
|
||||
|
||||
if [[ -z "$user_id" ]]; then
|
||||
warn "Emby user '$username' not found at $emby_url — may already be removed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local del_code
|
||||
del_code=$(curl -sf --max-time 15 -w "%{http_code}" -o /dev/null \
|
||||
-X DELETE \
|
||||
-H "X-Emby-Authorization: MediaBrowser Token=\"$EMBY_API_KEY\"" \
|
||||
"${emby_url}/Users/${user_id}" 2>/dev/null)
|
||||
|
||||
if [[ "$del_code" == "200" ]] || [[ "$del_code" == "204" ]] || [[ "$del_code" == "404" ]]; then
|
||||
log "Emby admin '$username' removed ✅"
|
||||
else
|
||||
warn "Failed to delete Emby user '$username' (HTTP $del_code) — remove manually"
|
||||
fi
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── MIRROR PATH ───────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
if [[ "$AM_MIRROR" == true ]]; then
|
||||
warn "$MIRROR_ID ($MIRROR) is initiating offboard"
|
||||
warn "Owner ($OWNER) will see INACTIVE state on its next --check cycle and finalize"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
echo ""
|
||||
echo "You have 10 seconds to cancel (Ctrl+C)..."
|
||||
sleep 10
|
||||
fi
|
||||
|
||||
OWNER_IP=$(resolve_tailscale_ip "$OWNER")
|
||||
OWNER_REACHABLE=false
|
||||
[[ -n "$OWNER_IP" ]] && OWNER_REACHABLE=true
|
||||
|
||||
STEP_STOP_RSYNC_OK=true
|
||||
STEP_WEBUI_OK=true
|
||||
STEP_STACK_CLEANUP_OK=true
|
||||
STEP_FALLBACK_CLEANUP_OK=true
|
||||
STEP_DISABLE_RSYNC_OK=true
|
||||
STEP_EMBY_OK=true
|
||||
SSH_REVOKE_REMOTE_OK=false
|
||||
SSH_REVOKE_LOCAL_OK=false
|
||||
|
||||
# ── Step 1: Stop rsync ────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Step 1/8 — Stop Rsync ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
bash "$SCRIPTS_ROOT/Rsync/rsync_stop.sh" --rsync-only 2>/dev/null || true
|
||||
log "Rsync stopped ✅"
|
||||
else
|
||||
warn "DRY RUN — would stop rsync"
|
||||
fi
|
||||
|
||||
# ── Step 2: Reconfigure local WebUIs → localhost ──────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Step 2/8 — Reconfigure Local WebUIs → localhost ━━━"
|
||||
|
||||
reconfigure_local_webuis "localhost" || STEP_WEBUI_OK=false
|
||||
|
||||
# ── Step 3: Remove owner-deployed containers (auth/arr stack) locally ─────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Step 3/8 — Remove Owner-Deployed Containers ━━━"
|
||||
|
||||
if [[ "$OWNER_REACHABLE" == true ]]; then
|
||||
cleanup_deployed_stack_locally "$OWNER_IP" "$OWNER_SSH_KEY" || STEP_STACK_CLEANUP_OK=false
|
||||
else
|
||||
warn "Owner unreachable — cannot read deployed stack list"
|
||||
warn "Auth/arr containers will remain — remove manually or re-run when owner is reachable"
|
||||
STEP_STACK_CLEANUP_OK=false
|
||||
fi
|
||||
|
||||
# ── Step 4: Remove fallback coverage containers ───────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Step 4/8 — Fallback Container Cleanup ━━━"
|
||||
|
||||
PARTNER_FOLDER_NAME=$(derive_partner_folder_name "$OWNER")
|
||||
cleanup_partner_containers "$PARTNER_FOLDER_NAME" || STEP_FALLBACK_CLEANUP_OK=false
|
||||
|
||||
# ── Step 5: Disable critical sync ─────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Step 5/8 — Disable Critical Sync ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
update_master_conf "CRITICAL_RSYNC_ENABLED" "false" && \
|
||||
warn "CRITICAL_RSYNC_ENABLED=false ✅" || \
|
||||
{ warn "Failed to update CRITICAL_RSYNC_ENABLED"; STEP_DISABLE_RSYNC_OK=false; }
|
||||
else
|
||||
warn "DRY RUN — would set CRITICAL_RSYNC_ENABLED=false"
|
||||
fi
|
||||
|
||||
# ── Step 6: Revoke Emby admin locally ─────────────────────────────────────────────────────
|
||||
revoke_local_emby_admin || STEP_EMBY_OK=false
|
||||
|
||||
# ── Step 7: Restart own stack ─────────────────────────────────────────────────────────────
|
||||
start_own_stack
|
||||
|
||||
# ── Step 8: SSH key revocation, write state, signal owner ────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_SHIELD Step 8/8 — SSH Revocation + State ━━━"
|
||||
|
||||
do_ssh_key_revocation "${OWNER_IP:-}"
|
||||
|
||||
NOW=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
write_state_file "$LOCAL_STATE_FILE" \
|
||||
"INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
|
||||
log "Local state: INACTIVE ✅"
|
||||
add_to_blocklist "$OWNER" "$REASON"
|
||||
else
|
||||
warn "DRY RUN — would write INACTIVE state and blocklist $OWNER"
|
||||
fi
|
||||
|
||||
if [[ "$OWNER_REACHABLE" == true ]]; then
|
||||
push_state_to_remote "$LOCAL_STATE_FILE" "$OWNER_IP" "$OWNER_SSH_KEY"
|
||||
notify "Partnership offboard requested by $MIRROR — $OWNER will finalise on next check" \
|
||||
"Partnership" "normal"
|
||||
else
|
||||
warn "$OWNER unreachable — state written locally, owner will see it when reachable"
|
||||
fi
|
||||
|
||||
# ── Summary ───────────────────────────────────────────────────────────────────────────────
|
||||
END=$(date +%s)
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY OFFBOARD SUMMARY (Mirror) ━━━━━"
|
||||
echo " Mirror: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo " Owner: $OWNER_ID ($OWNER)"
|
||||
echo " Reason: $REASON"
|
||||
echo " Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
_ok() { [[ "$1" == true ]] && echo "✅" || echo "❌"; }
|
||||
_skip() { [[ "$1" == true ]] && echo "skipped" || echo "$(_ok "$2")"; }
|
||||
_revoke_status() {
|
||||
if [[ "${SSH_REVOKE_REMOTE_OK:-false}" == true && "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
|
||||
echo "both directions ✅"
|
||||
elif [[ "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
|
||||
echo "local only ✅ — remote failed (revoke manually on $OWNER)"
|
||||
else
|
||||
echo "⚠️ failed — check warnings above"
|
||||
fi
|
||||
}
|
||||
|
||||
echo " Step 1 — Stop rsync: $(_ok "$STEP_STOP_RSYNC_OK")"
|
||||
echo " Step 2 — WebUIs: $(_ok "$STEP_WEBUI_OK")"
|
||||
echo " Step 3 — Stack cleanup: $(_ok "$STEP_STACK_CLEANUP_OK")"
|
||||
echo " Step 4 — Fallback cleanup: $(_ok "$STEP_FALLBACK_CLEANUP_OK")"
|
||||
echo " Step 5 — Disable sync: $(_ok "$STEP_DISABLE_RSYNC_OK")"
|
||||
echo " Step 6 — Emby revoke: $(_ok "$STEP_EMBY_OK")"
|
||||
echo " Step 7 — Own stack: started"
|
||||
echo " Step 8 — Keys revoked: $(_revoke_status)"
|
||||
echo ""
|
||||
echo " State: INACTIVE ✅"
|
||||
echo " Blocklist: $OWNER blocked ✅"
|
||||
echo " Owner: will finalise + final sync on next --check"
|
||||
echo ""
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
|
||||
warn "$ICON_DONE DONE — mirror separation complete ✅"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── OWNER PATH ────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
warn "Offboarding $MIRROR_ID ($MIRROR) from partnership"
|
||||
warn "Final sync will run — mirror leaves with current state"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
echo ""
|
||||
echo "You have 10 seconds to cancel (Ctrl+C)..."
|
||||
sleep 10
|
||||
echo "Proceeding..."
|
||||
fi
|
||||
|
||||
WEBUI_FAILURES=0
|
||||
SSH_REVOKE_REMOTE_OK=false
|
||||
SSH_REVOKE_LOCAL_OK=false
|
||||
|
||||
# ── Step 1: Stop rsync ────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Step 1/10 — Stop Rsync ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
bash "$SCRIPTS_ROOT/Rsync/rsync_stop.sh" --rsync-only 2>/dev/null || true
|
||||
log "Rsync stopped ✅"
|
||||
else
|
||||
warn "DRY RUN — would stop rsync"
|
||||
fi
|
||||
|
||||
# ── Step 2: Final sync ────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Step 2/10 — Final Sync ━━━"
|
||||
|
||||
do_final_sync
|
||||
|
||||
# ── Step 3: Reconfigure mirror WebUIs → localhost ─────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Step 3/10 — Reconfigure Mirror WebUIs → localhost ━━━"
|
||||
|
||||
MIRROR_IP=$(resolve_tailscale_ip "$MIRROR")
|
||||
MIRROR_REACHABLE=false
|
||||
[[ -n "$MIRROR_IP" ]] && MIRROR_REACHABLE=true
|
||||
|
||||
if [[ "$MIRROR_REACHABLE" == true ]]; then
|
||||
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
|
||||
[[ -z "$entry" ]] && continue
|
||||
container="${entry%%|*}"
|
||||
port="${entry##*|}"
|
||||
reconfigure_webui "$container" "$port" "localhost" \
|
||||
"$MIRROR_SSH_KEY" "$MIRROR_IP" "$MIRROR" || (( WEBUI_FAILURES++ ))
|
||||
done
|
||||
else
|
||||
warn "$MIRROR unreachable — WebUI reconfiguration skipped"
|
||||
warn "$MIRROR will reconfigure its own WebUIs when it sees INACTIVE state on --check"
|
||||
(( WEBUI_FAILURES++ ))
|
||||
fi
|
||||
|
||||
# ── Step 4: Disable critical sync ─────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Step 4/10 — Disable Critical Sync ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
update_master_conf "CRITICAL_RSYNC_ENABLED" "false"
|
||||
warn "CRITICAL_RSYNC_ENABLED=false ✅"
|
||||
else
|
||||
warn "DRY RUN — would set CRITICAL_RSYNC_ENABLED=false"
|
||||
fi
|
||||
|
||||
# ── Step 5: Write state, push to mirror, blocklist ────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Step 5/10 — Write State ━━━"
|
||||
|
||||
NOW=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
write_state_file "$LOCAL_STATE_FILE" \
|
||||
"INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
|
||||
log "Local state: INACTIVE ✅"
|
||||
add_to_blocklist "$MIRROR" "$REASON"
|
||||
[[ "$MIRROR_REACHABLE" == true ]] && \
|
||||
push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"
|
||||
else
|
||||
warn "DRY RUN — would write INACTIVE state, blocklist $MIRROR, push to remote"
|
||||
fi
|
||||
|
||||
# ── Step 6: Local container cleanup ───────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Step 6/10 — Local Container Cleanup ━━━"
|
||||
|
||||
PARTNER_FOLDER_NAME=$(derive_partner_folder_name "$MIRROR")
|
||||
cleanup_partner_containers "$PARTNER_FOLDER_NAME"
|
||||
|
||||
# ── Step 7: Restart own stack ─────────────────────────────────────────────────────────────────
|
||||
start_own_stack
|
||||
|
||||
# ── Step 8: Remote container cleanup ──────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Step 8/10 — Remote Container Cleanup ━━━"
|
||||
|
||||
if [[ "$MIRROR_REACHABLE" == true ]]; then
|
||||
# Remove auth/arr stack containers deployed during onboard (by config array)
|
||||
cleanup_deployed_stack_on_remote "$MIRROR_IP" "$MIRROR_SSH_KEY"
|
||||
# Remove fallback coverage containers (by *-owner_short naming pattern)
|
||||
cleanup_owner_containers_on_mirror "$MIRROR_IP"
|
||||
else
|
||||
warn "$MIRROR unreachable — remote container cleanup skipped"
|
||||
warn "Run 'partnership_manager.sh --offboard' on $MIRROR to clean up manually"
|
||||
fi
|
||||
|
||||
# ── Step 9: Restart mirror's own stack ────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_START Step 9/10 — Restart Mirror Stack ━━━"
|
||||
|
||||
[[ "$MIRROR_REACHABLE" == true ]] && start_mirror_own_stack "$MIRROR_IP"
|
||||
|
||||
# ── Step 10: Revocation ───────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_SHIELD Step 10/10 — Revocation ━━━"
|
||||
|
||||
# Emby admin — before SSH key revocation while Emby still reachable
|
||||
[[ "$MIRROR_REACHABLE" == true ]] && revoke_emby_admin "$MIRROR_IP"
|
||||
|
||||
# SSH key revocation — mutual, both directions; must run while Tailscale still active
|
||||
do_ssh_key_revocation "${MIRROR_IP:-}"
|
||||
|
||||
# Tailscale removal — after SSH revocation, guard with grace window
|
||||
if [[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_NET Tailscale Separation ━━━"
|
||||
if [[ "$MIRROR_REACHABLE" == true ]]; then
|
||||
grace_seconds=$(( ${PARTNERSHIP_GRACE_HOURS:-6} * 3600 ))
|
||||
warn "Waiting ${PARTNERSHIP_GRACE_HOURS:-6}hr grace — mirror can collect backups..."
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
trap 'warn "Offboard interrupted during grace sleep"; exit 0' SIGTERM SIGINT
|
||||
sleep "$grace_seconds"
|
||||
trap - SIGTERM SIGINT
|
||||
fi
|
||||
fi
|
||||
remove_tailscale_device "$MIRROR"
|
||||
fi
|
||||
|
||||
# Backup handover notification
|
||||
if [[ ${#PARTNERSHIP_MIRROR_BACKUPS[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_DISK Backup Handover ━━━"
|
||||
log "Backups available for $MIRROR:"
|
||||
for path in "${PARTNERSHIP_MIRROR_BACKUPS[@]}"; do
|
||||
[[ -z "$path" ]] && continue
|
||||
echo " $path"
|
||||
done
|
||||
notify "$MIRROR offboard complete — backups available for ${PARTNERSHIP_GRACE_HOURS:-6}hr. Tailscale access expires then." \
|
||||
"Partnership" "warning"
|
||||
fi
|
||||
|
||||
# ── Summary ───────────────────────────────────────────────────────────────────────────────────
|
||||
END=$(date +%s)
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY OFFBOARD SUMMARY (Owner) ━━━━━"
|
||||
echo " Owner: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo " Mirror: $MIRROR_ID ($MIRROR)"
|
||||
echo " Reason: $REASON"
|
||||
echo " Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
_ok() { [[ "$1" == true ]] && echo "✅" || echo "❌"; }
|
||||
_revoke_status() {
|
||||
if [[ "${SSH_REVOKE_REMOTE_OK:-false}" == true && "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
|
||||
echo "both directions ✅"
|
||||
elif [[ "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
|
||||
echo "local only ✅ — remote failed (revoke manually on $MIRROR)"
|
||||
else
|
||||
echo "⚠️ failed — check warnings above"
|
||||
fi
|
||||
}
|
||||
|
||||
echo " Step 1 — Stop rsync: ✅"
|
||||
echo " Step 2 — Final sync: ✅"
|
||||
echo " Step 3 — WebUI failures: $WEBUI_FAILURES"
|
||||
echo " Step 4 — Disable sync: ✅"
|
||||
echo " Step 5 — State: INACTIVE ✅"
|
||||
echo " Step 6 — Local cleanup: ✅"
|
||||
echo " Step 7 — Own stack: started"
|
||||
echo " Step 8 — Remote cleanup: $( [[ "$MIRROR_REACHABLE" == true ]] && echo "✅" || echo "skipped (unreachable)" )"
|
||||
echo " Step 9 — Mirror stack: $( [[ "$MIRROR_REACHABLE" == true ]] && echo "started" || echo "skipped (unreachable)" )"
|
||||
echo " Step 10 — Keys revoked: $(_revoke_status)"
|
||||
echo ""
|
||||
echo " Blocklist: $MIRROR blocked — re-onboard to permit access again ✅"
|
||||
[[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]] && \
|
||||
echo " FolderView3: ${PARTNER_FOLDER_NAME:-} cleaned ✅"
|
||||
[[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]] && \
|
||||
echo " Tailscale: $MIRROR removed ✅"
|
||||
echo ""
|
||||
echo " $MIRROR leaves with:"
|
||||
echo " ✓ Current auth config (final sync)"
|
||||
echo " ✓ Auth WebUIs → localhost"
|
||||
echo " ✓ ${PARTNERSHIP_GRACE_HOURS:-6}hr to collect backups"
|
||||
echo ""
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
|
||||
warn "$ICON_DONE DONE — clean separation complete ✅"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
Executable
+640
@@ -0,0 +1,640 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Partnership Onboard ============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Runs once on both servers to establish a new partnership. Role is detected
|
||||
# automatically via detect_hosts() — no flags needed to declare which side you are.
|
||||
# Run on the mirror first (generates its SSH key), then on the owner to complete
|
||||
# setup remotely.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# MIRROR PATH (1 step)
|
||||
# Step 1: SSH key setup — generate keypair, copy to owner, update conf
|
||||
# Owner completes the rest remotely. Mirror is done.
|
||||
#
|
||||
# OWNER PATH (8 steps)
|
||||
# Step 1: SSH key setup — generate keypair, install on mirror, update conf
|
||||
# Step 2: Plugin install — FolderView3 and required plugins on mirror
|
||||
# Step 3: Stop mirror auth — stop mirror's existing auth containers before replacing
|
||||
# Step 4: Deploy auth stack — push XMLs, pull images, create + start on mirror
|
||||
# Mariadb/Redis health-checked before Authelia deploys
|
||||
# Step 5: Stop mirror arr — stop mirror's existing arr containers before replacing
|
||||
# Step 6: Deploy arr stack — push arr XMLs, pull images, create + start on mirror
|
||||
# Step 7: Partnership onboard — configure WebUIs → owner IP, write state, FolderView3, Emby
|
||||
# Step 8: Arr bootstrap — bidirectional library sync (arr_sync.sh)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Credentials never in SSH command strings
|
||||
# Auth stack containers hold API keys, DB passwords, etc. The deploy script is written
|
||||
# locally, SCPed to the remote, and executed there. Command-line args are never used
|
||||
# to pass credentials — they'd appear in `ps` output and shell history on both servers.
|
||||
#
|
||||
# XML templates are the single source of truth for deployed containers
|
||||
# The owner's templates-user/ XMLs define every container deployed on the mirror.
|
||||
# The same XMLs that Unraid's Docker Manager uses are what get SCPed — the mirror's
|
||||
# Docker Manager can manage the containers after onboard without additional config.
|
||||
#
|
||||
# Dependency ordering in the auth stack is owner-enforced
|
||||
# PARTNERSHIP_AUTH_STACK order matters: Mariadb and Redis must come before Authelia.
|
||||
# The array is ordered correctly in master_host1.conf. After each Mariadb/Redis deploy,
|
||||
# the script waits for the container to be healthy before continuing. This is a remote
|
||||
# health check — the container must be running (or report healthy) before the next
|
||||
# dependent is deployed.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check
|
||||
# All operations run as root — SSH key management, docker operations, conf updates.
|
||||
#
|
||||
# SSH timeout on all remote calls
|
||||
# Every ssh/scp call uses SSH_TIMEOUT. No operation hangs indefinitely on a
|
||||
# slow or unreachable mirror.
|
||||
#
|
||||
# --dry-run shows exact actions without executing
|
||||
# Every step prints what it would do. SCP, deploy, plugin install, arr sync —
|
||||
# all dry-run safe.
|
||||
#
|
||||
# Step skip flags for partial re-runs
|
||||
# --skip-ssh, --skip-auth-stack, --skip-arr-stack, --skip-arr-sync allow
|
||||
# resuming after a partial failure without re-running completed steps.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_PARTNERSHIP_AUTH_STACK
|
||||
# XML filenames (from this server's templates-user/) to push and deploy on the
|
||||
# mirror as its auth stack. Order matters: database deps before Authelia.
|
||||
# Aliased by detect_hosts() → PARTNERSHIP_AUTH_STACK
|
||||
#
|
||||
# HOST*_PARTNERSHIP_REPLACE_CONTAINERS
|
||||
# Containers to stop on the mirror before deploying the auth stack.
|
||||
# Defined in the MIRROR's own conf (master_host*.conf on HOST2) — never in HOST1's conf.
|
||||
# Read live from the mirror via SSH during Step 3 (sources mirror's load_config.sh at
|
||||
# the same $SCRIPTS_ROOT path — convention: both servers use the same repo location).
|
||||
# Leave empty on HOST2 if no conflicting containers exist (fresh mirror: nothing to stop).
|
||||
# Aliased by detect_hosts() → PARTNERSHIP_REPLACE_CONTAINERS (on the mirror)
|
||||
#
|
||||
# HOST*_PARTNERSHIP_ARR_STACK
|
||||
# XML filenames to push and deploy on the mirror as its arr stack.
|
||||
# Leave empty to skip arr stack deploy.
|
||||
# Aliased by detect_hosts() → PARTNERSHIP_ARR_STACK
|
||||
#
|
||||
# HOST*_PARTNERSHIP_ARR_REPLACE_CONTAINERS
|
||||
# Arr containers to stop on the mirror before deploying the arr stack.
|
||||
# Same rule as PARTNERSHIP_REPLACE_CONTAINERS: defined in mirror's own conf, never HOST1's.
|
||||
# Aliased by detect_hosts() → PARTNERSHIP_ARR_REPLACE_CONTAINERS (on the mirror)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Partnership/partnership_onboard.sh
|
||||
# Full onboard — role detected automatically
|
||||
#
|
||||
# Partnership/partnership_onboard.sh --dry-run
|
||||
# Preview all steps without making changes
|
||||
#
|
||||
# Partnership/partnership_onboard.sh --log
|
||||
# Verbose per-step output
|
||||
#
|
||||
# Partnership/partnership_onboard.sh --skip-ssh
|
||||
# Skip SSH key setup (key already in place)
|
||||
#
|
||||
# Partnership/partnership_onboard.sh --skip-auth-stack
|
||||
# Skip auth stack stop + deploy (Steps 3-4)
|
||||
#
|
||||
# Partnership/partnership_onboard.sh --skip-arr-stack
|
||||
# Skip arr stack stop + deploy (Steps 5-6)
|
||||
#
|
||||
# Partnership/partnership_onboard.sh --skip-arr-sync
|
||||
# Skip arr library bootstrap (Step 8)
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SCRIPTS_ROOT="$SCRIPT_DIR/.."
|
||||
TEMPLATES_DIR="/boot/config/plugins/dockerMan/templates-user"
|
||||
SSH_TIMEOUT=15
|
||||
|
||||
source "$SCRIPTS_ROOT/load_config.sh"
|
||||
|
||||
# ── Parse flags ───────────────────────────────────────────────────────────────────────────────
|
||||
SKIP_SSH=false
|
||||
SKIP_AUTH_STACK=false
|
||||
SKIP_ARR_STACK=false
|
||||
SKIP_ARR_SYNC=false
|
||||
FILTERED_ARGS=()
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--skip-ssh) SKIP_SSH=true ;;
|
||||
--skip-auth-stack) SKIP_AUTH_STACK=true ;;
|
||||
--skip-arr-stack) SKIP_ARR_STACK=true ;;
|
||||
--skip-arr-sync) SKIP_ARR_SYNC=true ;;
|
||||
*) FILTERED_ARGS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
parse_args "${FILTERED_ARGS[@]}"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
[[ "$EUID" -ne 0 ]] && { error "Must be run as root"; exit 1; }
|
||||
|
||||
detect_hosts
|
||||
|
||||
OWNER_ID="${PARTNERSHIP_OWNER_HOST:-HOST1}"
|
||||
MIRROR_ID=$( [[ "$OWNER_ID" == "HOST1" ]] && echo "HOST2" || echo "HOST1" )
|
||||
OWNER="${!OWNER_ID}"
|
||||
MIRROR="${!MIRROR_ID}"
|
||||
# SSH_KEY (set by detect_hosts) is this server's own private key.
|
||||
# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
|
||||
# HOST{N}_SSH_KEY lives in master_host{N}.conf — with sparse checkout, the other server's
|
||||
# conf is never present here. Always use SSH_KEY (local private key) for outbound SSH.
|
||||
MIRROR_SSH_KEY="$SSH_KEY"
|
||||
|
||||
AM_OWNER=false
|
||||
AM_MIRROR=false
|
||||
[[ "$MY_ID" == "$OWNER_ID" ]] && AM_OWNER=true
|
||||
[[ "$MY_ID" == "$MIRROR_ID" ]] && AM_MIRROR=true
|
||||
|
||||
EXTRA_FLAGS=()
|
||||
[[ "$DRY_RUN" == true ]] && EXTRA_FLAGS+=("--dry-run")
|
||||
[[ "$LOG_MODE" == true ]] && EXTRA_FLAGS+=("--log")
|
||||
|
||||
START=$(date +%s)
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_FALLBACK Partnership Onboard — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo ""
|
||||
echo " Role: $( [[ "$AM_OWNER" == true ]] && echo "OWNER" || echo "MIRROR" )"
|
||||
echo " This: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo " Partner: $( [[ "$AM_OWNER" == true ]] && echo "$MIRROR_ID ($MIRROR)" || echo "$OWNER_ID ($OWNER)" )"
|
||||
echo ""
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permanent changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER: deploy a container from a local Unraid template XML to a remote host ─────────────
|
||||
#
|
||||
# Parses Port / Path / Variable Config entries from the XML, SCPs the template and a
|
||||
# self-contained deploy script to the remote, executes it, then cleans up both sides.
|
||||
# Credentials are never passed as SSH command-line args — they stay in the SCPed script.
|
||||
# ==============================================================================================
|
||||
deploy_container_from_xml() {
|
||||
local xml_file="$1" remote_ip="$2" ssh_key="$3"
|
||||
local xml_name
|
||||
xml_name=$(basename "$xml_file")
|
||||
|
||||
# Extract top-level fields
|
||||
local name repo network extra privileged
|
||||
name=$( awk 'match($0,/<Name>([^<]+)<\/Name>/, a){print a[1];exit}' "$xml_file")
|
||||
repo=$( awk 'match($0,/<Repository>([^<]+)<\/Repository>/,a){print a[1];exit}' "$xml_file")
|
||||
network=$( awk 'match($0,/<Network>([^<]+)<\/Network>/, a){print a[1];exit}' "$xml_file")
|
||||
extra=$( awk 'match($0,/<ExtraParams>([^<]*)<\/ExtraParams>/,a){print a[1];exit}' "$xml_file")
|
||||
privileged=$( awk 'match($0,/<Privileged>([^<]+)<\/Privileged>/,a){print a[1];exit}' "$xml_file")
|
||||
|
||||
if [[ -z "$name" || -z "$repo" ]]; then
|
||||
warn " Cannot parse Name/Repository from $xml_name — skipping"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log "Deploying $name..."
|
||||
|
||||
# SCP the XML so Unraid Docker Manager recognises and can manage the container
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
timeout "$SSH_TIMEOUT" scp -i "$ssh_key" -o ConnectTimeout="$SSH_TIMEOUT" \
|
||||
"$xml_file" "root@${remote_ip}:${TEMPLATES_DIR}/${xml_name}" 2>/dev/null || {
|
||||
warn " SCP failed for $xml_name — skipping $name"
|
||||
return 1
|
||||
}
|
||||
else
|
||||
warn " DRY RUN — would SCP $xml_name → $MIRROR:${TEMPLATES_DIR}/"
|
||||
fi
|
||||
|
||||
# Build a self-contained deploy script locally.
|
||||
# Writing to a temp file keeps credentials out of SSH command strings.
|
||||
local tmp_script
|
||||
tmp_script=$(mktemp /tmp/deploy_XXXXXX.sh)
|
||||
chmod 600 "$tmp_script"
|
||||
|
||||
{
|
||||
echo "#!/bin/bash"
|
||||
echo "set -e"
|
||||
echo ""
|
||||
printf "docker pull %q 2>/dev/null || true\n" "$repo"
|
||||
printf "docker stop %q 2>/dev/null || true\n" "$name"
|
||||
printf "docker rm %q 2>/dev/null || true\n" "$name"
|
||||
echo ""
|
||||
printf "docker create --name %q --restart=unless-stopped" "$name"
|
||||
[[ -n "$network" ]] && printf " --network=%q" "$network"
|
||||
[[ "$privileged" == "true" ]] && printf " --privileged"
|
||||
[[ -n "$extra" ]] && printf " %s" "$extra"
|
||||
|
||||
# Port mappings → -p host:container/proto
|
||||
awk '/Type="Port"/ {
|
||||
match($0, /Target="([^"]+)"/, t)
|
||||
match($0, /Mode="([^"]+)"/, m)
|
||||
match($0, />([^<]+)<\/Config>/, v)
|
||||
if (t[1] != "" && v[1] != "") {
|
||||
proto = (m[1] == "udp") ? "udp" : "tcp"
|
||||
printf " -p %s:%s/%s", v[1], t[1], proto
|
||||
}
|
||||
}' "$xml_file"
|
||||
|
||||
# Volume mappings → -v 'host:container:mode'
|
||||
awk 'BEGIN{q=sprintf("%c",39)} /Type="Path"/ {
|
||||
match($0, /Target="([^"]+)"/, t)
|
||||
match($0, /Mode="([^"]+)"/, m)
|
||||
match($0, />([^<]+)<\/Config>/, v)
|
||||
if (t[1] != "" && v[1] != "") {
|
||||
mode = (m[1] == "ro") ? "ro" : "rw"
|
||||
printf " -v %s%s:%s:%s%s", q, v[1], t[1], mode, q
|
||||
}
|
||||
}' "$xml_file"
|
||||
|
||||
# Environment variables → -e 'KEY=VALUE' (single-quoted to protect $ and special chars)
|
||||
awk 'BEGIN{q=sprintf("%c",39)} /Type="Variable"/ {
|
||||
match($0, /Target="([^"]+)"/, t)
|
||||
match($0, />([^<]+)<\/Config>/, v)
|
||||
if (t[1] != "" && v[1] != "") {
|
||||
printf " -e %s%s=%s%s", q, t[1], v[1], q
|
||||
}
|
||||
}' "$xml_file"
|
||||
|
||||
printf " %q\n" "$repo"
|
||||
echo ""
|
||||
printf "docker start %q && echo 'deployed:%s'\n" "$name" "$name"
|
||||
} > "$tmp_script"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn " DRY RUN — would deploy $name on $MIRROR"
|
||||
rm -f "$tmp_script"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# SCP deploy script → remote, execute, clean up both sides
|
||||
local remote_script="/tmp/deploy_${name//[^a-zA-Z0-9_]/_}.sh"
|
||||
|
||||
if timeout "$SSH_TIMEOUT" scp -i "$ssh_key" -o ConnectTimeout="$SSH_TIMEOUT" \
|
||||
"$tmp_script" "root@${remote_ip}:${remote_script}" 2>/dev/null && \
|
||||
timeout 120 ssh -i "$ssh_key" -o ConnectTimeout="$SSH_TIMEOUT" root@"$remote_ip" \
|
||||
"bash '$remote_script' 2>&1; rc=\$?; rm -f '$remote_script'; exit \$rc" 2>/dev/null | \
|
||||
grep -q "deployed:${name}"; then
|
||||
log " $name deployed ✅"
|
||||
rm -f "$tmp_script"
|
||||
return 0
|
||||
else
|
||||
warn " $name deployment failed — check $MIRROR manually"
|
||||
rm -f "$tmp_script"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER: wait for a container on the remote to be healthy/running ─────────────────────────
|
||||
#
|
||||
# Polls docker inspect on the remote. Prefers the health status if a healthcheck is defined;
|
||||
# falls back to the running state for containers with no healthcheck. Non-fatal after timeout
|
||||
# — Authelia may take time to fully initialize but the deploy itself succeeded.
|
||||
# ==============================================================================================
|
||||
wait_for_container_healthy() {
|
||||
local name="$1" remote_ip="$2" ssh_key="$3"
|
||||
local max_wait=60 interval=5 elapsed=0
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && return 0
|
||||
|
||||
log " Waiting for $name to be ready..."
|
||||
while (( elapsed < max_wait )); do
|
||||
local status
|
||||
status=$(timeout "$SSH_TIMEOUT" ssh -i "$ssh_key" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$remote_ip" \
|
||||
"h=\$(docker inspect --format '{{.State.Health.Status}}' '$name' 2>/dev/null)
|
||||
r=\$(docker inspect --format '{{.State.Running}}' '$name' 2>/dev/null)
|
||||
echo \${h:-\$r}" 2>/dev/null)
|
||||
|
||||
case "$status" in
|
||||
healthy|true)
|
||||
log " $name ready ✅"
|
||||
return 0
|
||||
;;
|
||||
starting|unhealthy|false|"")
|
||||
sleep "$interval"
|
||||
(( elapsed += interval ))
|
||||
;;
|
||||
*)
|
||||
sleep "$interval"
|
||||
(( elapsed += interval ))
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
warn " $name not confirmed healthy after ${max_wait}s — continuing (may affect dependents)"
|
||||
return 0
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER: stop containers on the mirror by reading its own conf via SSH ────────────────────
|
||||
#
|
||||
# SSHes to the mirror, sources its load_config.sh at the same $SCRIPTS_ROOT path (both servers
|
||||
# use the same convention), and reads the named config array from the mirror's own conf.
|
||||
# HOST2's container list stays in HOST2's master_host2.conf — not duplicated in HOST1's conf.
|
||||
# Fails gracefully if scripts aren't present yet or the array is empty (nothing to stop).
|
||||
#
|
||||
# deploy_container_from_xml() already stops/removes containers with the same name as what's
|
||||
# being deployed. This step handles containers with DIFFERENT names that conflict.
|
||||
# ==============================================================================================
|
||||
stop_mirror_stack() {
|
||||
local config_var="$1" label="$2"
|
||||
local -a to_stop=()
|
||||
|
||||
mapfile -t to_stop < <(
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
|
||||
"source '$SCRIPTS_ROOT/load_config.sh' 2>/dev/null
|
||||
detect_hosts 2>/dev/null
|
||||
printf '%s\n' \"\${${config_var}[@]:-}\"" 2>/dev/null | grep -v '^$'
|
||||
)
|
||||
|
||||
if [[ ${#to_stop[@]} -eq 0 ]]; then
|
||||
log "No $label containers to stop on $MIRROR — skipping"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log "Stopping $label on $MIRROR: ${to_stop[*]}"
|
||||
for container in "${to_stop[@]}"; do
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn " DRY RUN — would stop + rm $container on $MIRROR"
|
||||
continue
|
||||
fi
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" root@"$MIRROR_IP" \
|
||||
"docker stop '$container' 2>/dev/null
|
||||
docker rm '$container' 2>/dev/null && echo removed" 2>/dev/null | \
|
||||
grep -q removed && \
|
||||
log " $container removed ✅" || \
|
||||
log " $container not found on $MIRROR — skipping"
|
||||
done
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER: deploy a stack of XMLs to the mirror, health-checking db deps between batches ────
|
||||
#
|
||||
# Sets globals _STACK_DEPLOYED and _STACK_FAILED rather than printing to stdout.
|
||||
# This avoids the process-substitution capture problem: warn() writes to stdout, so any
|
||||
# read -r X Y < <(func) would capture warn output as the count values.
|
||||
# ==============================================================================================
|
||||
_STACK_DEPLOYED=0
|
||||
_STACK_FAILED=0
|
||||
|
||||
deploy_xml_stack() {
|
||||
local -n xml_array_ref="$1"
|
||||
_STACK_DEPLOYED=0
|
||||
_STACK_FAILED=0
|
||||
|
||||
for xml_name in "${xml_array_ref[@]}"; do
|
||||
local xml_file="${TEMPLATES_DIR}/${xml_name}"
|
||||
if [[ ! -f "$xml_file" ]]; then
|
||||
warn "$xml_name not found in $TEMPLATES_DIR — skipping"
|
||||
(( _STACK_FAILED++ ))
|
||||
continue
|
||||
fi
|
||||
|
||||
# Extract container name to use for health-wait matching
|
||||
local cname
|
||||
cname=$(awk 'match($0,/<Name>([^<]+)<\/Name>/,a){print a[1];exit}' "$xml_file")
|
||||
|
||||
if deploy_container_from_xml "$xml_file" "$MIRROR_IP" "$MIRROR_SSH_KEY"; then
|
||||
(( _STACK_DEPLOYED++ ))
|
||||
# Health-check database deps before continuing — they must be ready before
|
||||
# Authelia/app containers that depend on them can start cleanly.
|
||||
if [[ -n "$cname" ]] && echo "$cname" | grep -qiE 'mariadb|redis|postgres|mysql'; then
|
||||
wait_for_container_healthy "$cname" "$MIRROR_IP" "$MIRROR_SSH_KEY"
|
||||
fi
|
||||
else
|
||||
(( _STACK_FAILED++ ))
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── MIRROR PATH ───────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
if [[ "$AM_MIRROR" == true ]]; then
|
||||
echo "━━━ Step 1/1 — SSH Key Setup (Mirror) ━━━"
|
||||
echo ""
|
||||
echo " Mirror only needs SSH keys ready."
|
||||
echo " Owner ($OWNER) completes the rest remotely."
|
||||
echo ""
|
||||
|
||||
if [[ "$SKIP_SSH" == true ]]; then
|
||||
warn "Skipping SSH setup (--skip-ssh)"
|
||||
elif bash "$SCRIPT_DIR/ssh_setup.sh" "${EXTRA_FLAGS[@]}"; then
|
||||
log "SSH key ready ✅"
|
||||
else
|
||||
error "SSH key setup failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY MIRROR SETUP COMPLETE ━━━━━"
|
||||
echo " SSH key: ready"
|
||||
echo " Next: Run Partnership/partnership_onboard.sh on $OWNER to complete setup"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── OWNER PATH ────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
MIRROR_IP=$(resolve_tailscale_ip "$MIRROR")
|
||||
[[ -z "$MIRROR_IP" ]] && { error "Cannot resolve $MIRROR Tailscale IP — is Tailscale running?"; exit 1; }
|
||||
log "Mirror: $MIRROR ($MIRROR_IP)"
|
||||
echo ""
|
||||
|
||||
STEP_SSH_OK=false
|
||||
STEP_PLUGINS_OK=true
|
||||
STEP_STOP_AUTH_OK=true
|
||||
STEP_AUTH_OK=true
|
||||
AUTH_DEPLOYED=0
|
||||
AUTH_FAILED=0
|
||||
STEP_STOP_ARR_OK=true
|
||||
STEP_ARR_OK=true
|
||||
ARR_DEPLOYED=0
|
||||
ARR_FAILED=0
|
||||
ONBOARD_OK=false
|
||||
ARR_SYNC_OK=false
|
||||
|
||||
# ── Step 1: SSH ───────────────────────────────────────────────────────────────────────────────
|
||||
echo "━━━ Step 1/8 — SSH Key Setup ━━━"
|
||||
|
||||
if [[ "$SKIP_SSH" == true ]]; then
|
||||
warn "Skipping (--skip-ssh)"
|
||||
STEP_SSH_OK=true
|
||||
elif bash "$SCRIPT_DIR/ssh_setup.sh" "${EXTRA_FLAGS[@]}"; then
|
||||
log "SSH keys ready ✅"
|
||||
STEP_SSH_OK=true
|
||||
else
|
||||
error "SSH key setup failed — aborting"
|
||||
error "Re-run or use --skip-ssh if key is already set up"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Step 2: Plugins ───────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ Step 2/8 — Plugin Install on Mirror ━━━"
|
||||
|
||||
if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]] && [[ -n "${PARTNERSHIP_FOLDERVIEW3_URL:-}" ]]; then
|
||||
FV3_PRESENT=$(timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
|
||||
"test -d /usr/local/emhttp/plugins/folder.view3 && echo yes" 2>/dev/null)
|
||||
|
||||
if [[ "$FV3_PRESENT" == "yes" ]]; then
|
||||
log "FolderView3 already installed on $MIRROR ✅"
|
||||
elif [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would install FolderView3 on $MIRROR"
|
||||
else
|
||||
log "Installing FolderView3 on $MIRROR..."
|
||||
timeout 60 ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" root@"$MIRROR_IP" \
|
||||
"plugin install '$PARTNERSHIP_FOLDERVIEW3_URL' 2>/dev/null && echo installed" \
|
||||
2>/dev/null | grep -q installed && \
|
||||
log "FolderView3 installed ✅" || {
|
||||
warn "FolderView3 install failed — install manually from Community Applications"
|
||||
STEP_PLUGINS_OK=false
|
||||
}
|
||||
fi
|
||||
else
|
||||
log "FolderView3 not configured — skipping"
|
||||
fi
|
||||
|
||||
# ── Step 3: Stop mirror's existing auth stack ─────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ Step 3/8 — Stop Mirror Auth Stack ━━━"
|
||||
|
||||
if [[ "$SKIP_AUTH_STACK" == true ]]; then
|
||||
warn "Skipping (--skip-auth-stack)"
|
||||
else
|
||||
stop_mirror_stack "PARTNERSHIP_REPLACE_CONTAINERS" "auth stack"
|
||||
fi
|
||||
|
||||
# ── Step 4: Deploy auth stack on mirror ───────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ Step 4/8 — Deploy Auth Stack on Mirror ━━━"
|
||||
|
||||
if [[ "$SKIP_AUTH_STACK" == true ]]; then
|
||||
warn "Skipping (--skip-auth-stack)"
|
||||
elif [[ ${#PARTNERSHIP_AUTH_STACK[@]} -eq 0 ]]; then
|
||||
warn "PARTNERSHIP_AUTH_STACK not set in ${MY_ID} conf — skipping auth stack deploy"
|
||||
warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to master_host${MY_ID: -1}.conf"
|
||||
STEP_AUTH_OK=false
|
||||
else
|
||||
deploy_xml_stack PARTNERSHIP_AUTH_STACK
|
||||
AUTH_DEPLOYED=$_STACK_DEPLOYED
|
||||
AUTH_FAILED=$_STACK_FAILED
|
||||
log "Auth stack: $AUTH_DEPLOYED deployed, $AUTH_FAILED failed"
|
||||
[[ "$AUTH_FAILED" -gt 0 ]] && STEP_AUTH_OK=false
|
||||
fi
|
||||
|
||||
# ── Step 5: Stop mirror's existing arr stack ──────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ Step 5/8 — Stop Mirror Arr Stack ━━━"
|
||||
|
||||
if [[ "$SKIP_ARR_STACK" == true ]]; then
|
||||
warn "Skipping (--skip-arr-stack)"
|
||||
elif [[ ${#PARTNERSHIP_ARR_STACK[@]} -eq 0 ]]; then
|
||||
log "PARTNERSHIP_ARR_STACK not configured — skipping arr stack deploy"
|
||||
SKIP_ARR_STACK=true
|
||||
else
|
||||
stop_mirror_stack "PARTNERSHIP_ARR_REPLACE_CONTAINERS" "arr stack"
|
||||
fi
|
||||
|
||||
# ── Step 6: Deploy arr stack on mirror ───────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ Step 6/8 — Deploy Arr Stack on Mirror ━━━"
|
||||
|
||||
if [[ "$SKIP_ARR_STACK" == true ]]; then
|
||||
warn "Skipping (--skip-arr-stack)"
|
||||
else
|
||||
deploy_xml_stack PARTNERSHIP_ARR_STACK
|
||||
ARR_DEPLOYED=$_STACK_DEPLOYED
|
||||
ARR_FAILED=$_STACK_FAILED
|
||||
log "Arr stack: $ARR_DEPLOYED deployed, $ARR_FAILED failed"
|
||||
[[ "$ARR_FAILED" -gt 0 ]] && STEP_ARR_OK=false
|
||||
fi
|
||||
|
||||
# ── Step 7: Partnership onboard ───────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ Step 7/8 — Partnership Onboard ━━━"
|
||||
|
||||
if bash "$SCRIPTS_ROOT/Partnership/partnership_manager.sh" --onboard "${EXTRA_FLAGS[@]}"; then
|
||||
log "Partnership onboard complete ✅"
|
||||
ONBOARD_OK=true
|
||||
else
|
||||
error "Partnership onboard failed"
|
||||
ONBOARD_OK=false
|
||||
fi
|
||||
|
||||
# ── Step 8: Arr library bootstrap ─────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ Step 8/8 — Arr Library Bootstrap ━━━"
|
||||
|
||||
if [[ "$ONBOARD_OK" == false ]]; then
|
||||
warn "Skipping — onboard did not complete"
|
||||
elif [[ "$SKIP_ARR_SYNC" == true ]]; then
|
||||
warn "Skipping (--skip-arr-sync)"
|
||||
elif [[ ! -f "$SCRIPTS_ROOT/Media/arr_sync.sh" ]]; then
|
||||
warn "arr_sync.sh not found — run Media/arr_sync.sh manually once arrs are live"
|
||||
elif bash "$SCRIPTS_ROOT/Media/arr_sync.sh" "${EXTRA_FLAGS[@]}"; then
|
||||
log "Arr bootstrap complete ✅"
|
||||
ARR_SYNC_OK=true
|
||||
else
|
||||
warn "Arr sync had errors — partnership still valid"
|
||||
warn "Re-run Media/arr_sync.sh once all arr containers are live"
|
||||
fi
|
||||
|
||||
# ── Summary ───────────────────────────────────────────────────────────────────────────────────
|
||||
END=$(date +%s)
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY ONBOARD SUMMARY ━━━━━"
|
||||
echo " Owner: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo " Mirror: $MIRROR ($MIRROR_IP)"
|
||||
echo " Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
_ok() { [[ "$1" == true ]] && echo "✅" || echo "❌"; }
|
||||
_skip() { [[ "$1" == true ]] && echo "skipped" || echo "$(_ok "$2")"; }
|
||||
|
||||
echo " Step 1 — SSH keys: $(_skip "$SKIP_SSH" "$STEP_SSH_OK")"
|
||||
echo " Step 2 — Plugins: $(_ok "$STEP_PLUGINS_OK")"
|
||||
echo " Step 3 — Stop auth: $(_skip "$SKIP_AUTH_STACK" "$STEP_STOP_AUTH_OK")"
|
||||
echo " Step 4 — Auth stack: $( [[ "$SKIP_AUTH_STACK" == true ]] && echo "skipped" || echo "${AUTH_DEPLOYED} deployed, ${AUTH_FAILED} failed" )"
|
||||
echo " Step 5 — Stop arr: $(_skip "$SKIP_ARR_STACK" "$STEP_STOP_ARR_OK")"
|
||||
echo " Step 6 — Arr stack: $( [[ "$SKIP_ARR_STACK" == true ]] && echo "skipped" || echo "${ARR_DEPLOYED} deployed, ${ARR_FAILED} failed" )"
|
||||
echo " Step 7 — Onboard: $(_ok "$ONBOARD_OK")"
|
||||
echo " Step 8 — Arr bootstrap: $( [[ "$SKIP_ARR_SYNC" == true || "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$ARR_SYNC_OK")" )"
|
||||
echo ""
|
||||
|
||||
if [[ "$ONBOARD_OK" == true ]]; then
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
|
||||
log "$ICON_DONE DONE — partnership established ✅"
|
||||
log "Verify with: Partnership/partnership_manager.sh --status"
|
||||
else
|
||||
error "Setup incomplete — resolve errors above and re-run"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
[[ "$ONBOARD_OK" == false ]] && exit 1
|
||||
exit 0
|
||||
Executable
+379
@@ -0,0 +1,379 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= SSH Setup ======================================================
|
||||
# ==============================================================================================
|
||||
# Generates the SSH keypair for rsync automation and installs it on the remote server.
|
||||
# Key named after this server: hostname lowercased, unraid- prefix stripped.
|
||||
# unRAID-Gmer4Lfe → gmer4lfe_rsync_automation
|
||||
# unRAID-Jayred365 → jayred365_rsync_automation
|
||||
# Idempotent — skips generation if key already exists (use --force to regenerate).
|
||||
# Updates master_host*.conf with key path on success.
|
||||
#
|
||||
# ── MODES ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# (default) — generate key if missing, copy to remote, update conf
|
||||
# --force — regenerate key even if it exists, re-copy to remote
|
||||
# --validate — test SSH auth to remote, track strikes, notify at limit
|
||||
# --status — show key state, fingerprint, remote connectivity
|
||||
# --dry-run — preview without creating, copying, or updating conf
|
||||
#
|
||||
# ── STRIKE SYSTEM (--validate) ────────────────────────────────────────────────────────────────
|
||||
# Called during partnership --check cycles to detect broken SSH auth.
|
||||
# Tracks consecutive SSH auth failures — not network unreachability.
|
||||
# Remote Tailscale IP unreachable = network issue → not counted as SSH strike.
|
||||
# Remote reachable but SSH auth fails = key issue → strike incremented.
|
||||
# Strikes reset automatically after SSH_STRIKE_RESET_HRS of clean connectivity.
|
||||
# At SSH_MAX_STRIKES: notify + return exit 2 (caller can escalate).
|
||||
# State: DATA_DIR/ssh_strikes_{REMOTE_SERVER_NAME}.db
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# SSH_MAX_STRIKES — consecutive failures before notifying (default 5)
|
||||
# SSH_STRIKE_RESET_HRS — hours since last failure before counter resets (default 24)
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# Partnership/ssh_setup.sh — initial setup (idempotent)
|
||||
# Partnership/ssh_setup.sh --force — regenerate + re-copy
|
||||
# Partnership/ssh_setup.sh --validate — health check + strike tracking
|
||||
# Partnership/ssh_setup.sh --status — show key and connectivity state
|
||||
# Any mode supports --dry-run and --log
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SCRIPTS_ROOT="$SCRIPT_DIR/.."
|
||||
|
||||
source "$SCRIPTS_ROOT/load_config.sh"
|
||||
|
||||
# ── Parse --force before parse_args ───────────────────────────────────────────────────────────
|
||||
MODE="setup"
|
||||
FORCE=false
|
||||
FILTERED_ARGS=()
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--force) FORCE=true ;;
|
||||
--validate) MODE="validate" ;;
|
||||
--status) MODE="status" ;;
|
||||
*) FILTERED_ARGS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
parse_args "${FILTERED_ARGS[@]}"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
validate_unraid_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
detect_hosts
|
||||
|
||||
# ── Derive key name from hostname ─────────────────────────────────────────────────────────────
|
||||
# Strip unraid- prefix (case-insensitive) if present → gmer4lfe_rsync_automation
|
||||
SHORT_NAME="${LOCAL_SERVER_NAME,,}"
|
||||
[[ "${SHORT_NAME}" == unraid-* ]] && SHORT_NAME="${SHORT_NAME:7}"
|
||||
SSH_KEY_NAME="${SHORT_NAME}_rsync_automation"
|
||||
SSH_KEY_PATH="/root/.ssh/${SSH_KEY_NAME}"
|
||||
SSH_PUB_PATH="${SSH_KEY_PATH}.pub"
|
||||
|
||||
# ── Host conf path ────────────────────────────────────────────────────────────────────────────
|
||||
HOST_NUM="${MY_ID#HOST}" # "1" or "2"
|
||||
HOST_CONF="$SCRIPTS_ROOT/master_host${HOST_NUM}.conf"
|
||||
KEY_CONF_VAR="${MY_ID}_SSH_KEY"
|
||||
|
||||
# ── Strike state file ─────────────────────────────────────────────────────────────────────────
|
||||
SSH_STRIKE_FILE="$DATA_DIR/ssh_strikes_${REMOTE_SERVER_NAME}.db"
|
||||
|
||||
# ==============================================================================================
|
||||
# ─── HELPERS ──────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
test_ssh_auth() {
|
||||
local remote_ip="$1"
|
||||
timeout "${SSH_TIMEOUT:-15}" ssh \
|
||||
-i "$SSH_KEY_PATH" \
|
||||
-o BatchMode=yes \
|
||||
-o ConnectTimeout="${SSH_TIMEOUT:-15}" \
|
||||
-o StrictHostKeyChecking=no \
|
||||
root@"$remote_ip" "echo ok" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
read_strikes() {
|
||||
grep "^strikes=" "$SSH_STRIKE_FILE" 2>/dev/null | cut -d= -f2 || echo 0
|
||||
}
|
||||
|
||||
read_last_strike_epoch() {
|
||||
local ts
|
||||
ts=$(grep "^last_strike=" "$SSH_STRIKE_FILE" 2>/dev/null | cut -d= -f2)
|
||||
[[ -n "$ts" ]] && date -d "$ts" +%s 2>/dev/null || echo 0
|
||||
}
|
||||
|
||||
write_strike_file() {
|
||||
local strikes="$1" last_strike="${2:-}" last_success="${3:-}"
|
||||
mkdir -p "$(dirname "$SSH_STRIKE_FILE")"
|
||||
cat > "$SSH_STRIKE_FILE" <<EOF
|
||||
strikes=${strikes}
|
||||
last_strike=${last_strike}
|
||||
last_success=${last_success}
|
||||
updated=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
EOF
|
||||
}
|
||||
|
||||
update_conf_key_path() {
|
||||
if [[ ! -f "$HOST_CONF" ]]; then
|
||||
warn "$(basename "$HOST_CONF") not found — update ${KEY_CONF_VAR} manually to: $SSH_KEY_PATH"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local current
|
||||
current=$(grep "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null | \
|
||||
sed 's/.*="\?\([^"]*\)"\?.*/\1/')
|
||||
|
||||
if [[ "$current" == "$SSH_KEY_PATH" ]]; then
|
||||
log "${KEY_CONF_VAR} already correct in $(basename "$HOST_CONF")"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if grep -q "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null; then
|
||||
sed -i "s|^[[:space:]]*${KEY_CONF_VAR}=.*| ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\"|" "$HOST_CONF" && \
|
||||
log "${KEY_CONF_VAR} updated in $(basename "$HOST_CONF") ✅" || \
|
||||
warn "Failed to update ${KEY_CONF_VAR} in $(basename "$HOST_CONF") — update manually"
|
||||
else
|
||||
warn "${KEY_CONF_VAR} not found in $(basename "$HOST_CONF") — add manually:"
|
||||
warn " ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\""
|
||||
fi
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$MODE" == "status" ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY SSH SETUP STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_NET Remote: $REMOTE_SERVER_NAME"
|
||||
echo ""
|
||||
echo " Key name: $SSH_KEY_NAME"
|
||||
echo " Key path: $SSH_KEY_PATH"
|
||||
echo ""
|
||||
|
||||
if [[ -f "$SSH_KEY_PATH" ]]; then
|
||||
local_fp=$(ssh-keygen -lf "$SSH_KEY_PATH" 2>/dev/null || echo "unreadable")
|
||||
echo " $ICON_DONE Private key: present"
|
||||
echo " Fingerprint: $local_fp"
|
||||
else
|
||||
echo " $ICON_ERROR Private key: NOT found"
|
||||
fi
|
||||
|
||||
if [[ -f "$SSH_PUB_PATH" ]]; then
|
||||
echo " $ICON_DONE Public key: present"
|
||||
else
|
||||
echo " $ICON_ERROR Public key: NOT found"
|
||||
fi
|
||||
|
||||
# Check conf
|
||||
if [[ -f "$HOST_CONF" ]]; then
|
||||
current_conf=$(grep "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null | \
|
||||
sed 's/.*="\?\([^"]*\)"\?.*/\1/')
|
||||
if [[ "$current_conf" == "$SSH_KEY_PATH" ]]; then
|
||||
echo " $ICON_DONE Conf ($(basename "$HOST_CONF")): ${KEY_CONF_VAR} ✅"
|
||||
else
|
||||
echo " $ICON_ERROR Conf ($(basename "$HOST_CONF")): ${KEY_CONF_VAR}=${current_conf:-not set}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Remote connectivity
|
||||
echo ""
|
||||
REMOTE_IP=$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")
|
||||
if [[ -z "$REMOTE_IP" ]]; then
|
||||
echo " $ICON_ERROR Remote ($REMOTE_SERVER_NAME): Tailscale unreachable"
|
||||
elif [[ -f "$SSH_KEY_PATH" ]] && test_ssh_auth "$REMOTE_IP"; then
|
||||
echo " $ICON_DONE Remote ($REMOTE_SERVER_NAME [$REMOTE_IP]): SSH auth OK ✅"
|
||||
else
|
||||
echo " $ICON_ERROR Remote ($REMOTE_SERVER_NAME [$REMOTE_IP]): SSH auth FAILED"
|
||||
fi
|
||||
|
||||
# Strike state
|
||||
if [[ -f "$SSH_STRIKE_FILE" ]]; then
|
||||
echo ""
|
||||
echo " Strike state:"
|
||||
while IFS= read -r line; do
|
||||
echo " $line"
|
||||
done < "$SSH_STRIKE_FILE"
|
||||
fi
|
||||
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Validate (SSH health check + strike tracking) ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$MODE" == "validate" ]]; then
|
||||
MAX_STRIKES="${SSH_MAX_STRIKES:-5}"
|
||||
RESET_HRS="${SSH_STRIKE_RESET_HRS:-24}"
|
||||
|
||||
REMOTE_IP=$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")
|
||||
if [[ -z "$REMOTE_IP" ]]; then
|
||||
log "SSH validate — $REMOTE_SERVER_NAME Tailscale unreachable, not an SSH issue"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ! -f "$SSH_KEY_PATH" ]]; then
|
||||
warn "SSH validate — key not found at $SSH_KEY_PATH — run ssh_setup.sh to create"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NOW=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
NOW_EPOCH=$(date +%s)
|
||||
|
||||
if test_ssh_auth "$REMOTE_IP"; then
|
||||
# SSH works — reset strikes if they were > 0
|
||||
STRIKES=$(read_strikes)
|
||||
if [[ "$STRIKES" -gt 0 ]]; then
|
||||
write_strike_file 0 "" "$NOW"
|
||||
log "SSH validate — auth restored to $REMOTE_SERVER_NAME ✅ (strikes reset)"
|
||||
else
|
||||
log "SSH validate — $REMOTE_SERVER_NAME SSH auth OK ✅"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# SSH auth failed — check if strikes should reset first
|
||||
STRIKES=$(read_strikes)
|
||||
LAST_STRIKE_EPOCH=$(read_last_strike_epoch)
|
||||
RESET_SECS=$(( RESET_HRS * 3600 ))
|
||||
|
||||
if [[ "$STRIKES" -gt 0 ]] && \
|
||||
[[ "$LAST_STRIKE_EPOCH" -gt 0 ]] && \
|
||||
[[ $(( NOW_EPOCH - LAST_STRIKE_EPOCH )) -gt "$RESET_SECS" ]]; then
|
||||
warn "SSH validate — strike counter reset after ${RESET_HRS}hr gap"
|
||||
STRIKES=0
|
||||
fi
|
||||
|
||||
STRIKES=$(( STRIKES + 1 ))
|
||||
write_strike_file "$STRIKES" "$NOW" ""
|
||||
warn "SSH validate — $REMOTE_SERVER_NAME auth FAILED (strike $STRIKES/$MAX_STRIKES)"
|
||||
|
||||
if [[ "$STRIKES" -ge "$MAX_STRIKES" ]]; then
|
||||
error "SSH auth to $REMOTE_SERVER_NAME broken — ${STRIKES} consecutive failures"
|
||||
error "Repair: run 'Partnership/ssh_setup.sh --force' to regenerate and re-copy key"
|
||||
notify "SSH auth broken to $REMOTE_SERVER_NAME ($MY_ID) — ${STRIKES} strikes, manual repair needed. Run: Partnership/ssh_setup.sh --force" \
|
||||
"SSH Setup" "warning"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup (default) or --force ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR SSH Key Setup — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo ""
|
||||
echo " Key name: $SSH_KEY_NAME"
|
||||
echo " Key path: $SSH_KEY_PATH"
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no keys will be created or copied"
|
||||
|
||||
# ── Key generation ────────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Key Generation ━━━"
|
||||
|
||||
if [[ -f "$SSH_KEY_PATH" ]] && [[ "$FORCE" == false ]]; then
|
||||
local_fp=$(ssh-keygen -lf "$SSH_KEY_PATH" 2>/dev/null || echo "unreadable")
|
||||
log "Key already exists — skipping generation (--force to regenerate)"
|
||||
log " $local_fp"
|
||||
else
|
||||
if [[ "$FORCE" == true ]] && [[ -f "$SSH_KEY_PATH" ]]; then
|
||||
warn "Regenerating key (--force) — existing key will be replaced"
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
rm -f "$SSH_KEY_PATH" "$SSH_PUB_PATH"
|
||||
fi
|
||||
fi
|
||||
|
||||
warn "Generating ed25519 keypair: $SSH_KEY_PATH"
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
ssh-keygen -t ed25519 -N "" -f "$SSH_KEY_PATH" -C "${SSH_KEY_NAME}@${LOCAL_SERVER_NAME}" && \
|
||||
log "Keypair generated ✅" || {
|
||||
error "Failed to generate keypair"
|
||||
exit 1
|
||||
}
|
||||
chmod 600 "$SSH_KEY_PATH"
|
||||
chmod 644 "$SSH_PUB_PATH"
|
||||
else
|
||||
warn "DRY RUN — would generate: $SSH_KEY_PATH"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Update conf ───────────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Update master_host${HOST_NUM}.conf ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
update_conf_key_path
|
||||
else
|
||||
warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in master_host${HOST_NUM}.conf"
|
||||
fi
|
||||
|
||||
# ── Copy to remote ────────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_NET Copy Public Key to Remote ($REMOTE_SERVER_NAME) ━━━"
|
||||
|
||||
resolve_remote_ip
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would ssh-copy-id to root@$REMOTE_SERVER"
|
||||
warn "DRY RUN — would require password for root@$REMOTE_SERVER if key auth not yet set up"
|
||||
else
|
||||
warn "Installing public key on $REMOTE_SERVER_NAME ($REMOTE_SERVER)..."
|
||||
warn "(Password prompt for root@$REMOTE_SERVER is expected on first setup)"
|
||||
|
||||
if ssh-copy-id -i "$SSH_PUB_PATH" -o ConnectTimeout="${SSH_TIMEOUT:-15}" \
|
||||
root@"$REMOTE_SERVER" 2>/dev/null; then
|
||||
log "Public key installed on $REMOTE_SERVER_NAME ✅"
|
||||
else
|
||||
error "ssh-copy-id failed — check that:"
|
||||
error " 1. Remote server is reachable: tailscale status"
|
||||
error " 2. Password auth is enabled on remote: grep PasswordAuthentication /etc/ssh/sshd_config"
|
||||
error " 3. The correct password for root@$REMOTE_SERVER_NAME is used"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Verify ────────────────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_VERIFY Verify SSH Auth ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
if test_ssh_auth "$REMOTE_SERVER"; then
|
||||
log "SSH auth to $REMOTE_SERVER_NAME working ✅"
|
||||
# Reset any existing strikes
|
||||
if [[ -f "$SSH_STRIKE_FILE" ]]; then
|
||||
write_strike_file 0 "" "$(date '+%Y-%m-%d %H:%M:%S')"
|
||||
fi
|
||||
else
|
||||
warn "SSH auth test failed — key may need a moment to propagate"
|
||||
warn "Verify manually: ssh -i $SSH_KEY_PATH root@$REMOTE_SERVER"
|
||||
fi
|
||||
else
|
||||
warn "DRY RUN — would test: ssh -i $SSH_KEY_PATH root@$REMOTE_SERVER echo ok"
|
||||
fi
|
||||
|
||||
# ── Summary ───────────────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY SSH SETUP SUMMARY ━━━━━"
|
||||
echo " Key: $SSH_KEY_PATH"
|
||||
echo " Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)"
|
||||
echo " Conf: ${KEY_CONF_VAR} in master_host${HOST_NUM}.conf"
|
||||
echo ""
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
|
||||
warn "$ICON_DONE DONE — SSH key ready for rsync automation ✅"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
Reference in New Issue
Block a user