1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Partnership Offboard ===========================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Cleanly ends a partnership. Role is detected automatically — run on either server.
9	# Owner path runs the full sequence including remote cleanup and final sync.
10	# Mirror path handles the local side and signals the owner to complete its own cleanup.
11	#
12	# ==============================================================================================
13	# OPERATIONAL MODEL
14	# ==============================================================================================
15	#
16	# OWNER PATH (10 steps)
17	#   Step 1:  Stop rsync         — halt any running sync before state changes
18	#   Step 2:  Final sync         — mirror leaves with current Critical-Data state
19	#   Step 3:  Reconfigure WebUIs — mirror's auth WebUIs → localhost
20	#   Step 4:  Disable sync       — CRITICAL_RSYNC_ENABLED=false in master.conf
21	#   Step 5:  Local cleanup      — remove fallback coverage containers + appdata
22	#   Step 6:  Restart own stack  — bring up owner's own parked containers
23	#   Step 7:  Remote cleanup     — remove auth/arr stack + fallback containers from mirror
24	#   Step 8:  Restart mirror     — bring up mirror's own parked containers
25	#   Step 9:  Revocation         — Emby admin, SSH keys
26	#   Step 10: Write state        — INACTIVE locally + pushed to mirror, mirror blocklisted
27	#            Tailscale          — grace window then device removal (after state written)
28	#
29	# MIRROR PATH (8 steps)
30	#   Step 1: Stop rsync          — halt any running sync
31	#   Step 2: Reconfigure WebUIs  — local auth WebUIs → localhost
32	#   Step 3: Remote stack clean  — remove owner-deployed containers locally (auth/arr stack)
33	#   Step 4: Fallback cleanup    — remove fallback coverage containers
34	#   Step 5: Disable sync        — CRITICAL_RSYNC_ENABLED=false in master.conf
35	#   Step 6: Revoke Emby admin   — remove own admin account from local Emby instance
36	#   Step 7: Restart own stack   — bring up own parked containers
37	#   Step 8: SSH revocation      — revoke keys both directions, write state, signal owner
38	#
39	# ==============================================================================================
40	# CONFIGURATION
41	# ==============================================================================================
42	#
43	# host*.conf
44	#
45	#   HOST*_PARTNERSHIP_AUTH_STACK
46	#       Auth container XMLs to push during onboard — used on offboard to identify what
47	#       to remove. Owner's PARTNERSHIP_AUTH_STACK determines which containers get removed
48	#       from the mirror on both owner-initiated and mirror-initiated offboard.
49	#
50	#   HOST*_PARTNERSHIP_ARR_STACK
51	#       Arr container XMLs — same cleanup logic as auth stack.
52	#
53	#   HOST*_PARTNERSHIP_SERVICES_STACK
54	#       Shared services XMLs (Emby, Jellyfin, Seerr, SeerrFin) — same cleanup logic.
55	#
56	# ==============================================================================================
57	# RUNTIME MODES
58	# ==============================================================================================
59	#
60	# Partnership/partnership_offboard.sh
61	#     Full offboard — role detected automatically
62	#
63	# Partnership/partnership_offboard.sh --dry-run
64	#     Preview all steps without executing
65	#
66	# Partnership/partnership_offboard.sh --log
67	#     Verbose per-step output
68	#
69	# Partnership/partnership_offboard.sh --reason=<string>
70	#     Tag the offboard reason in state file and blocklist (default: manual)
71	#     Called by partnership_manager.sh --offboard (reason passed through)
72	#
73	# ==============================================================================================
74
75	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
76	SCRIPTS_ROOT="$SCRIPT_DIR/.."
77	SSH_TIMEOUT=15
78
79	source "$SCRIPTS_ROOT/load_config.sh"
80	source "$SCRIPTS_ROOT/Plugin/$PLATFORM/Partnership/containers.sh"
81
82	# ── Parse flags ───────────────────────────────────────────────────────────────────────────────
83	REASON="manual"
84	FILTERED_ARGS=()
85
86	for arg in "$@"; do
87	    case "$arg" in
88	        --reason=*) REASON="${arg#--reason=}" ;;
89	        *)          FILTERED_ARGS+=("$arg") ;;
90	    esac
91	done
92
93	parse_args "${FILTERED_ARGS[@]}"
94
95	# ── Source partnership_manager.sh for shared helpers ──────────────────────────────────────────
96	# PARTNERSHIP_LIB_MODE=1 skips mode dispatch — functions are defined, nothing is executed.
97	PARTNERSHIP_LIB_MODE=1 source "$SCRIPT_DIR/partnership_manager.sh"
98
99	# ==============================================================================================
100	# ━━━ Setup ━━━
101	# ==============================================================================================
102	[[ "$EUID" -ne 0 ]] && { error "Must be run as root"; exit 1; }
103
104	if ! command -v docker &>/dev/null; then
105	    error "Docker command not found"
106	    exit 1
107	fi
108
109	detect_hosts
110
111	OWNER_ID="${PARTNERSHIP_OWNER_HOST:-HOST1}"
112	MIRROR_ID=$( [[ "$OWNER_ID" == "HOST1" ]] && echo "HOST2" || echo "HOST1" )
113	OWNER="${!OWNER_ID}"
114	MIRROR="${!MIRROR_ID}"
115	MIRROR_SSH_KEY="$SSH_KEY"
116	OWNER_SSH_KEY="$SSH_KEY"
117
118	AM_OWNER=false
119	AM_MIRROR=false
120	[[ "$MY_ID" == "$OWNER_ID" ]] && AM_OWNER=true
121	[[ "$MY_ID" == "$MIRROR_ID" ]] && AM_MIRROR=true
122
123	LOCAL_STATE_FILE="${STATE_DIR}/partnership_${LOCAL_SERVER_NAME}.db"
124	REMOTE_STATE_FILE="${STATE_DIR}/partnership_${REMOTE_SERVER_NAME}.db"
125	OWNER_STATE_FILE="${STATE_DIR}/partnership_${OWNER}.db"
126	MIRROR_STATE_FILE="${STATE_DIR}/partnership_${MIRROR}.db"
127	OFFLINE_COUNTER="${STATE_DIR}/partnership_offline_days.db"
128
129	acquire_lock "strict"
130
131	trap _pm_trap_restart_stopped EXIT
132
133	# Check already offboarded
134	if [[ -f "$LOCAL_STATE_FILE" ]]; then
135	    CURRENT_STATE=$(read_state_file "$LOCAL_STATE_FILE" "state")
136	    if [[ "$CURRENT_STATE" == "INACTIVE" ]]; then
137	        warn "Partnership already INACTIVE — use partnership_manager.sh --status to verify both servers agree"
138	        exit 0
139	    fi
140	fi
141
142	START=$(date +%s)
143
144	echo ""
145	echo "━━━ $ICON_FALLBACK Partnership Offboard — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
146	echo ""
147	echo "  Role:    $( [[ "$AM_OWNER" == true ]] && echo "OWNER" || echo "MIRROR" )"
148	echo "  This:    $MY_ID ($LOCAL_SERVER_NAME)"
149	echo "  Partner: $( [[ "$AM_OWNER" == true ]] && echo "$MIRROR_ID ($MIRROR)" || echo "$OWNER_ID ($OWNER)" )"
150	echo "  Reason:  $REASON"
151	echo ""
152	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permanent changes will be made"
153
154	# ==============================================================================================
155	# ── HELPER: revoke own admin account from local Emby instance ────────────────────────────────
156	#
157	# Mirror-initiated path only. Called before start_own_stack so Emby is still running.
158	# Uses local EMBY_API_KEY and the mirror's own short name as the username to delete.
159	# ==============================================================================================
160	revoke_local_emby_admin() {
161	    local emby_port="${PARTNERSHIP_EMBY_PORT:-8096}"
162	    local emby_url="http://127.0.0.1:${emby_port}"
163
164	    echo ""
165	    echo "━━━ $ICON_EMBY Emby Admin Revocation ━━━"
166
167	    if [[ "${PARTNERSHIP_PROVISION_EMBY_ADMIN:-false}" != true ]]; then
168	        log "PARTNERSHIP_PROVISION_EMBY_ADMIN=false — skipping"
169	        return 0
170	    fi
171
172	    if [[ -z "${EMBY_API_KEY:-}" ]]; then
173	        warn "EMBY_API_KEY not set — skipping local Emby admin revocation"
174	        return 1
175	    fi
176
177	    # The account to revoke is this server's own short name (the mirror user's account)
178	    local username="${PARTNERSHIP_EMBY_ADMIN_USER:-$(derive_short_name "$LOCAL_SERVER_NAME")}"
179
180	    if [[ "$DRY_RUN" == true ]]; then
181	        warn "DRY RUN — would delete Emby admin '$username' at $emby_url"
182	        return 0
183	    fi
184
185	    log "Looking up Emby user '$username' at $emby_url..."
186
187	    local users_json user_id
188	    users_json=$(curl -sf --max-time 15 \
189	        -H "X-Emby-Authorization: MediaBrowser Token=\"$EMBY_API_KEY\"" \
190	        "${emby_url}/Users" 2>/dev/null)
191
192	    user_id=$(echo "$users_json" | \
193	        grep -o "\"Id\":\"[^\"]*\"[^}]*\"Name\":\"${username}\"" | \
194	        grep -o '"Id":"[^"]*"' | cut -d'"' -f4 | head -1)
195
196	    if [[ -z "$user_id" ]]; then
197	        warn "Emby user '$username' not found at $emby_url — may already be removed"
198	        return 0
199	    fi
200
201	    local del_code
202	    del_code=$(curl -sf --max-time 15 -w "%{http_code}" -o /dev/null \
203	        -X DELETE \
204	        -H "X-Emby-Authorization: MediaBrowser Token=\"$EMBY_API_KEY\"" \
205	        "${emby_url}/Users/${user_id}" 2>/dev/null)
206
207	    if [[ "$del_code" == "200" ]] || [[ "$del_code" == "204" ]] || [[ "$del_code" == "404" ]]; then
208	        log "Emby admin '$username' removed ✅"
209	    else
210	        warn "Failed to delete Emby user '$username' (HTTP $del_code) — remove manually"
211	    fi
212	}
213
214	# ==============================================================================================
215	# ── MIRROR PATH ───────────────────────────────────────────────────────────────────────────────
216	# ==============================================================================================
217	if [[ "$AM_MIRROR" == true ]]; then
218	    warn "$MIRROR_ID ($MIRROR) is initiating offboard"
219	    warn "Owner ($OWNER) will see INACTIVE state on its next --check cycle and finalize"
220
221	    if [[ "$DRY_RUN" == false ]]; then
222	        echo ""
223	        echo "You have 10 seconds to cancel (Ctrl+C)..."
224	        sleep 10
225	    fi
226
227	    OWNER_IP=$(resolve_tailscale_ip "$OWNER")
228	    OWNER_REACHABLE=false
229	    [[ -n "$OWNER_IP" ]] && OWNER_REACHABLE=true
230
231	    STEP_STOP_RSYNC_OK=true
232	    STEP_WEBUI_OK=true
233	    STEP_STACK_CLEANUP_OK=true
234	    STEP_FALLBACK_CLEANUP_OK=true
235	    STEP_DISABLE_RSYNC_OK=true
236	    STEP_EMBY_OK=true
237	    SSH_REVOKE_REMOTE_OK=false
238	    SSH_REVOKE_LOCAL_OK=false
239
240	    # ── Step 1: Stop rsync ────────────────────────────────────────────────────────────────────
241	    echo ""
242	    echo "━━━ $ICON_STOP Step 1/8 — Stop Rsync ━━━"
243
244	    if [[ "$DRY_RUN" == false ]]; then
245	        bash "$SCRIPTS_ROOT/Rsync/rsync_stop.sh" --rsync-only 2>/dev/null || true
246	        log "Rsync stopped ✅"
247	    else
248	        warn "DRY RUN — would stop rsync"
249	    fi
250
251	    # ── Step 2: Reconfigure local WebUIs → localhost ──────────────────────────────────────────
252	    echo ""
253	    echo "━━━ $ICON_CONTAINERS Step 2/8 — Reconfigure Local WebUIs → localhost ━━━"
254
255	    reconfigure_local_webuis "localhost" || STEP_WEBUI_OK=false
256
257	    # ── Step 3: Remove owner-deployed containers (auth/arr stack) locally ─────────────────────
258	    echo ""
259	    echo "━━━ $ICON_CONTAINERS Step 3/8 — Remove Owner-Deployed Containers ━━━"
260
261	    if [[ "$OWNER_REACHABLE" == true ]]; then
262	        cleanup_deployed_stack_locally "$OWNER_IP" "$OWNER_SSH_KEY" || STEP_STACK_CLEANUP_OK=false
263	    else
264	        warn "Owner unreachable — cannot read deployed stack list"
265	        warn "Auth/arr containers will remain — remove manually or re-run when owner is reachable"
266	        STEP_STACK_CLEANUP_OK=false
267	    fi
268
269	    # ── Step 4: Remove fallback coverage containers ───────────────────────────────────────────
270	    echo ""
271	    echo "━━━ $ICON_CONTAINERS Step 4/8 — Fallback Container Cleanup ━━━"
272
273	    cleanup_partner_containers || STEP_FALLBACK_CLEANUP_OK=false
274
275	    # ── Step 5: Disable critical sync ─────────────────────────────────────────────────────────
276	    echo ""
277	    echo "━━━ $ICON_GEAR Step 5/8 — Disable Critical Sync ━━━"
278
279	    if [[ "$DRY_RUN" == false ]]; then
280	        update_master_conf "CRITICAL_RSYNC_ENABLED" "false" && \
281	            warn "CRITICAL_RSYNC_ENABLED=false ✅" || \
282	            { warn "Failed to update CRITICAL_RSYNC_ENABLED"; STEP_DISABLE_RSYNC_OK=false; }
283	    else
284	        warn "DRY RUN — would set CRITICAL_RSYNC_ENABLED=false"
285	    fi
286
287	    # ── Step 6: Revoke Emby admin locally ─────────────────────────────────────────────────────
288	    revoke_local_emby_admin || STEP_EMBY_OK=false
289
290	    # ── Step 7: Restart own stack ─────────────────────────────────────────────────────────────
291	    start_own_stack
292
293	    # ── Step 8: SSH key revocation, write state, signal owner ────────────────────────────────
294	    echo ""
295	    echo "━━━ $ICON_SHIELD Step 8/8 — SSH Revocation + State ━━━"
296
297	    do_ssh_key_revocation "${OWNER_IP:-}"
298
299	    NOW=$(date '+%Y-%m-%d %H:%M:%S')
300	    if [[ "$DRY_RUN" == false ]]; then
301	        write_state_file "$LOCAL_STATE_FILE" \
302	            "INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
303	        echo "Local state: INACTIVE ✅"
304	        add_to_blocklist "$OWNER" "$REASON"
305	    else
306	        warn "DRY RUN — would write INACTIVE state and blocklist $OWNER"
307	    fi
308
309	    if [[ "$OWNER_REACHABLE" == true ]]; then
310	        push_state_to_remote "$LOCAL_STATE_FILE" "$OWNER_IP" "$OWNER_SSH_KEY"
311	        notify "Partnership offboard requested by $MIRROR — $OWNER will finalise on next check" \
312	            "Partnership" "normal"
313	    else
314	        warn "$OWNER unreachable — state written locally, owner will see it when reachable"
315	    fi
316
317	    # ── Summary ───────────────────────────────────────────────────────────────────────────────
318	    END=$(date +%s)
319	    echo ""
320	    echo "━━━━━ $ICON_SUMMARY OFFBOARD SUMMARY (Mirror) ━━━━━"
321	    echo "  Mirror:  $MY_ID ($LOCAL_SERVER_NAME)"
322	    echo "  Owner:   $OWNER_ID ($OWNER)"
323	    echo "  Reason:  $REASON"
324	    echo "  Duration: $(format_duration $(( END - START )))"
325	    echo ""
326	    _ok()   { [[ "$1" == true ]] && echo "✅" || echo "❌"; }
327	    _skip() { [[ "$1" == true ]] && echo "skipped" || echo "$(_ok "$2")"; }
328	    _revoke_status() {
329	        if   [[ "${SSH_REVOKE_REMOTE_OK:-false}" == true && "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
330	            echo "both directions ✅"
331	        elif [[ "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
332	            echo "local only ✅ — remote failed (revoke manually on $OWNER)"
333	        else
334	            echo "⚠  failed — check warnings above"
335	        fi
336	    }
337
338	    echo "  Step 1 — Stop rsync:       $(_ok "$STEP_STOP_RSYNC_OK")"
339	    echo "  Step 2 — WebUIs:           $(_ok "$STEP_WEBUI_OK")"
340	    echo "  Step 3 — Stack cleanup:    $(_ok "$STEP_STACK_CLEANUP_OK")"
341	    echo "  Step 4 — Fallback cleanup: $(_ok "$STEP_FALLBACK_CLEANUP_OK")"
342	    echo "  Step 5 — Disable sync:     $(_ok "$STEP_DISABLE_RSYNC_OK")"
343	    echo "  Step 6 — Emby revoke:      $(_ok "$STEP_EMBY_OK")"
344	    echo "  Step 7 — Own stack:        started"
345	    echo "  Step 8 — Keys revoked:     $(_revoke_status)"
346	    echo ""
347	    echo "  State:    INACTIVE ✅"
348	    echo "  Blocklist: $OWNER blocked ✅"
349	    echo "  Owner:    will finalise + final sync on next --check"
350	    echo ""
351	    [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
352	        warn "$ICON_DONE DONE — mirror separation complete ✅"
353	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
354	    exit 0
355	fi
356
357	# ==============================================================================================
358	# ── OWNER PATH ────────────────────────────────────────────────────────────────────────────────
359	# ==============================================================================================
360	warn "Offboarding $MIRROR_ID ($MIRROR) from partnership"
361	warn "Final sync will run — mirror leaves with current state"
362
363	if [[ "$DRY_RUN" == false ]]; then
364	    echo ""
365	    echo "You have 10 seconds to cancel (Ctrl+C)..."
366	    sleep 10
367	    echo "Proceeding..."
368	fi
369
370	WEBUI_FAILURES=0
371	STEP_STOP_OK=true
372	STEP_SYNC_OK=true
373	SSH_REVOKE_REMOTE_OK=false
374	SSH_REVOKE_LOCAL_OK=false
375
376	# ── Step 1: Stop rsync ────────────────────────────────────────────────────────────────────────
377	echo ""
378	echo "━━━ $ICON_STOP Step 1/10 — Stop Rsync ━━━"
379
380	if [[ "$DRY_RUN" == false ]]; then
381	    bash "$SCRIPTS_ROOT/Rsync/rsync_stop.sh" --rsync-only 2>/dev/null || STEP_STOP_OK=false
382	    log "Rsync stopped ✅"
383	else
384	    warn "DRY RUN — would stop rsync"
385	fi
386
387	# ── Step 2: Final sync ────────────────────────────────────────────────────────────────────────
388	echo ""
389	echo "━━━ $ICON_SYNC Step 2/10 — Final Sync ━━━"
390
391	do_final_sync || STEP_SYNC_OK=false
392
393	# ── Step 3: Reconfigure mirror WebUIs → localhost ─────────────────────────────────────────────
394	echo ""
395	echo "━━━ $ICON_CONTAINERS Step 3/10 — Reconfigure Mirror WebUIs → localhost ━━━"
396
397	MIRROR_IP=$(resolve_tailscale_ip "$MIRROR")
398	MIRROR_REACHABLE=false
399	[[ -n "$MIRROR_IP" ]] && MIRROR_REACHABLE=true
400
401	if [[ "$MIRROR_REACHABLE" == true ]]; then
402	    for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
403	        [[ -z "$entry" ]] && continue
404	        container="${entry%%|*}"
405	        port="${entry##*|}"
406	        reconfigure_webui "$container" "$port" "localhost" \
407	            "$MIRROR_SSH_KEY" "$MIRROR_IP" "$MIRROR" || (( WEBUI_FAILURES++ ))
408	    done
409	else
410	    warn "$MIRROR unreachable — WebUI reconfiguration skipped"
411	    warn "$MIRROR will reconfigure its own WebUIs when it sees INACTIVE state on --check"
412	    (( WEBUI_FAILURES++ ))
413	fi
414
415	# ── Step 4: Disable critical sync ─────────────────────────────────────────────────────────────
416	echo ""
417	echo "━━━ $ICON_GEAR Step 4/10 — Disable Critical Sync ━━━"
418
419	if [[ "$DRY_RUN" == false ]]; then
420	    update_master_conf "CRITICAL_RSYNC_ENABLED" "false"
421	    warn "CRITICAL_RSYNC_ENABLED=false ✅"
422	else
423	    warn "DRY RUN — would set CRITICAL_RSYNC_ENABLED=false"
424	fi
425
426	# ── Step 5: Local container cleanup ───────────────────────────────────────────────────────────
427	echo ""
428	echo "━━━ $ICON_CONTAINERS Step 5/10 — Local Container Cleanup ━━━"
429
430	cleanup_partner_containers
431
432	# ── Step 6: Restart own stack ─────────────────────────────────────────────────────────────────
433	start_own_stack
434
435	# ── Step 7: Remote container cleanup ──────────────────────────────────────────────────────────
436	echo ""
437	echo "━━━ $ICON_CONTAINERS Step 7/10 — Remote Container Cleanup ━━━"
438
439	if [[ "$MIRROR_REACHABLE" == true ]]; then
440	    # Remove auth/arr stack containers deployed during onboard (by config array)
441	    cleanup_deployed_stack_on_remote "$MIRROR_IP" "$MIRROR_SSH_KEY"
442	    # Remove fallback coverage containers (by *-owner_short naming pattern)
443	    cleanup_owner_containers_on_mirror "$MIRROR_IP"
444	else
445	    warn "$MIRROR unreachable — remote container cleanup skipped"
446	    warn "Run 'partnership_offboard.sh' on $MIRROR to clean up manually"
447	fi
448
449	# ── Step 8: Restart mirror's own stack ────────────────────────────────────────────────────────
450	echo ""
451	echo "━━━ $ICON_START Step 8/10 — Restart Mirror Stack ━━━"
452
453	[[ "$MIRROR_REACHABLE" == true ]] && start_mirror_own_stack "$MIRROR_IP"
454
455	# ── Step 9: Revocation (Emby + SSH) ──────────────────────────────────────────────────────────
456	echo ""
457	echo "━━━ $ICON_SHIELD Step 9/10 — Revocation ━━━"
458
459	# Emby admin — before SSH key revocation while Emby still reachable
460	[[ "$MIRROR_REACHABLE" == true ]] && revoke_emby_admin "$MIRROR_IP"
461
462	# SSH key revocation — mutual, both directions; must run while Tailscale still active
463	do_ssh_key_revocation "${MIRROR_IP:-}"
464
465	# ── Step 10: Write state, push to mirror, blocklist ───────────────────────────────────────────
466	# State is written after container cleanup and SSH revocation so that:
467	#   • Re-running after a crash between steps 5–9 restarts from scratch (no early-exit on INACTIVE)
468	#   • --check sees INACTIVE during the Tailscale grace sleep and does not re-trigger offboard
469	echo ""
470	echo "━━━ $ICON_GEAR Step 10/10 — Write State ━━━"
471
472	NOW=$(date '+%Y-%m-%d %H:%M:%S')
473
474	if [[ "$DRY_RUN" == false ]]; then
475	    write_state_file "$LOCAL_STATE_FILE" \
476	        "INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
477	    log "Local state: INACTIVE ✅"
478	    add_to_blocklist "$MIRROR" "$REASON"
479	    [[ "$MIRROR_REACHABLE" == true ]] && \
480	        push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"
481	else
482	    warn "DRY RUN — would write INACTIVE state, blocklist $MIRROR, push to remote"
483	fi
484
485	# Tailscale removal — after state written so --check does not re-trigger offboard during grace sleep
486	if [[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]]; then
487	    echo ""
488	    echo "━━━ $ICON_NET Tailscale Separation ━━━"
489	    if [[ "$MIRROR_REACHABLE" == true ]]; then
490	        grace_seconds=$(( ${PARTNERSHIP_GRACE_HOURS:-6} * 3600 ))
491	        warn "Waiting ${PARTNERSHIP_GRACE_HOURS:-6}hr grace — mirror can collect backups..."
492	        if [[ "$DRY_RUN" == false ]]; then
493	            trap 'warn "Offboard interrupted during grace sleep"; exit 0' SIGTERM SIGINT
494	            sleep "$grace_seconds"
495	            trap - SIGTERM SIGINT
496	        fi
497	    fi
498	    remove_tailscale_device "$MIRROR"
499	fi
500
501	# Backup handover notification
502	if [[ ${#PARTNERSHIP_MIRROR_BACKUPS[@]} -gt 0 ]]; then
503	    echo ""
504	    echo "━━━ $ICON_DISK Backup Handover ━━━"
505	    echo "Backups available for $MIRROR:"
506	    for path in "${PARTNERSHIP_MIRROR_BACKUPS[@]}"; do
507	        [[ -z "$path" ]] && continue
508	        echo "  $path"
509	    done
510	    notify "$MIRROR offboard complete — backups available for ${PARTNERSHIP_GRACE_HOURS:-6}hr. Tailscale access expires then." \
511	        "Partnership" "warning"
512	fi
513
514	# ── Summary ───────────────────────────────────────────────────────────────────────────────────
515	END=$(date +%s)
516	echo ""
517	echo "━━━━━ $ICON_SUMMARY OFFBOARD SUMMARY (Owner) ━━━━━"
518	echo "  Owner:    $MY_ID ($LOCAL_SERVER_NAME)"
519	echo "  Mirror:   $MIRROR_ID ($MIRROR)"
520	echo "  Reason:   $REASON"
521	echo "  Duration: $(format_duration $(( END - START )))"
522	echo ""
523	_ok()   { [[ "$1" == true ]] && echo "✅" || echo "❌"; }
524	_revoke_status() {
525	    if   [[ "${SSH_REVOKE_REMOTE_OK:-false}" == true && "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
526	        echo "both directions ✅"
527	    elif [[ "${SSH_REVOKE_LOCAL_OK:-false}" == true ]]; then
528	        echo "local only ✅ — remote failed (revoke manually on $MIRROR)"
529	    else
530	        echo "⚠  failed — check warnings above"
531	    fi
532	}
533
534	echo "  Step 1  — Stop rsync:     $(_ok "$STEP_STOP_OK")"
535	echo "  Step 2  — Final sync:     $(_ok "$STEP_SYNC_OK")"
536	echo "  Step 3  — WebUI failures: $WEBUI_FAILURES"
537	echo "  Step 4  — Disable sync:   ✅"
538	echo "  Step 5  — Local cleanup:  ✅"
539	echo "  Step 6  — Own stack:      started"
540	echo "  Step 7  — Remote cleanup: $( [[ "$MIRROR_REACHABLE" == true ]] && echo "✅" || echo "skipped (unreachable)" )"
541	echo "  Step 8  — Mirror stack:   $( [[ "$MIRROR_REACHABLE" == true ]] && echo "started" || echo "skipped (unreachable)" )"
542	echo "  Step 9  — Keys revoked:   $(_revoke_status)"
543	echo "  Step 10 — State:          INACTIVE ✅"
544	echo ""
545	echo "  Blocklist: $MIRROR blocked — re-onboard to permit access again ✅"
546	[[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]] && \
547	    echo "  Tailscale:   $MIRROR removed ✅"
548	echo ""
549	echo "  $MIRROR leaves with:"
550	echo "    ✓ Current auth config (final sync)"
551	echo "    ✓ Auth WebUIs → localhost"
552	echo "    ✓ ${PARTNERSHIP_GRACE_HOURS:-6}hr to collect backups"
553	echo ""
554	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
555	    warn "$ICON_DONE DONE — clean separation complete ✅"
556	echo "━━━━━━━━━━━━━━━━━━━━━━━"
557	exit 0
558


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Partnership Transfer ============================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Transfers ownership from the current owner to the current mirror. After
9	# transfer the roles are swapped: what was the mirror becomes the new owner,
10	# and what was the owner becomes the new mirror.
11	#
12	# No containers are moved — only config and WebUI targets are updated. Both
13	# servers remain in the partnership; the sync direction reverses on the next
14	# fallback.sh / critical_sync_maintenance.sh cycle.
15	#
16	# ==============================================================================================
17	# OPERATIONAL MODEL
18	# ==============================================================================================
19	#
20	# OWNER ONLY — mirror cannot run this script.
21	#
22	#   Step 1:  Health Verification  — both servers pass N consecutive health checks
23	#   Step 2:  Pre-transfer Sync    — final sync in current direction (owner → mirror)
24	#   Step 3:  Reconfigure WebUIs   — new owner WebUIs → localhost
25	#                                   new mirror WebUIs → new owner IP
26	#   Step 4:  Flip Ownership       — update PARTNERSHIP_OWNER_HOST in master.conf
27	#                                   on both servers
28	#   Step 5:  Write State          — ACTIVE written locally and pushed to new mirror
29	#
30	# ==============================================================================================
31	# OPERATIONAL SAFEGUARDS
32	# ==============================================================================================
33	#
34	# Owner-Only Enforcement
35	#   The script reads MY_ID from detect_hosts() and exits immediately if it is
36	#   not the current PARTNERSHIP_OWNER_HOST. The mirror cannot run a transfer.
37	#
38	# Explicit Confirmation String
39	#   Requires the exact passphrase from PARTNERSHIP_TRANSFER_CONFIRM via
40	#   --confirm=<value>. Without a matching string the transfer is cancelled
41	#   before any steps execute. Prevents accidental ownership changes.
42	#
43	# Active Partnership Guard
44	#   Reads the local state file and exits if the current state is INACTIVE.
45	#   A transfer without an active partnership has no defined outcome.
46	#
47	# Dual Health Verification
48	#   Both servers must pass PARTNERSHIP_TRANSFER_STRIKES consecutive health
49	#   checks before proceeding. A single failure resets the strike counter.
50	#   After PARTNERSHIP_TRANSFER_MAX_ATTEMPTS total attempts the transfer aborts.
51	#
52	# Pre-transfer Final Sync
53	#   A full sync in the current direction (owner → mirror) runs immediately
54	#   before roles flip. Ensures the mirror is current before it becomes the owner.
55	#
56	# ==============================================================================================
57	# CONFIGURATION
58	# ==============================================================================================
59	#
60	# master.conf
61	#
62	#   PARTNERSHIP_OWNER_HOST
63	#       Current owner host ID (e.g. "HOST1"). Updated on both servers after transfer.
64	#
65	#   PARTNERSHIP_TRANSFER_CONFIRM
66	#       Exact string required to confirm transfer (default: "i-understand-this-transfers-ownership").
67	#       Pass via --confirm=<value>.
68	#
69	#   PARTNERSHIP_TRANSFER_STRIKES
70	#       Consecutive health checks both servers must pass before transfer proceeds (default: 3).
71	#
72	#   PARTNERSHIP_TRANSFER_MAX_ATTEMPTS
73	#       Max health check attempts before giving up (default: 20).
74	#
75	#   CRITICAL_SYNC_SHARES
76	#       Array of "path|profile" or "path" entries for do_final_sync().
77	#
78	#   PARTNERSHIP_AUTH_WEBUIS
79	#       Array of "ContainerName|WebUIPort" entries reconfigured during transfer.
80	#
81	# ==============================================================================================
82	# RUNTIME MODES
83	# ==============================================================================================
84	#
85	# Partnership/partnership_transfer.sh --confirm=i-understand-this-transfers-ownership
86	#     Full transfer — owner detected automatically.
87	#
88	# Partnership/partnership_transfer.sh --confirm=i-understand-this-transfers-ownership --dry-run
89	#     Preview all steps without executing. Confirmation check is skipped in dry-run mode.
90	#
91	# Partnership/partnership_transfer.sh --confirm=i-understand-this-transfers-ownership --log
92	#     Verbose per-step output.
93	#
94	# ==============================================================================================
95
96	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
97	SCRIPTS_ROOT="$SCRIPT_DIR/.."
98	SSH_TIMEOUT=15
99
100	source "$SCRIPTS_ROOT/load_config.sh"
101
102	# ── Parse flags ───────────────────────────────────────────────────────────────────────────────
103	TRANSFER_CONFIRM_INPUT=""
104	FILTERED_ARGS=()
105
106	for arg in "$@"; do
107	    case "$arg" in
108	        --confirm=*)  TRANSFER_CONFIRM_INPUT="${arg#--confirm=}" ;;
109	        *)            FILTERED_ARGS+=("$arg") ;;
110	    esac
111	done
112
113	parse_args "${FILTERED_ARGS[@]}"
114
115	# ── Source partnership_manager.sh for shared helpers ──────────────────────────────────────────
116	# PARTNERSHIP_LIB_MODE=1 skips mode dispatch — functions are defined, nothing is executed.
117	PARTNERSHIP_LIB_MODE=1 source "$SCRIPT_DIR/partnership_manager.sh"
118
119	# ==============================================================================================
120	# ━━━ Setup ━━━
121	# ==============================================================================================
122	[[ "$EUID" -ne 0 ]] && { error "Must be run as root"; exit 1; }
123
124	if ! command -v docker &>/dev/null; then
125	    error "Docker command not found"
126	    exit 1
127	fi
128
129	detect_hosts
130
131	OWNER_ID="${PARTNERSHIP_OWNER_HOST:-HOST1}"
132	MIRROR_ID=$( [[ "$OWNER_ID" == "HOST1" ]] && echo "HOST2" || echo "HOST1" )
133	OWNER="${!OWNER_ID}"
134	MIRROR="${!MIRROR_ID}"
135	MIRROR_SSH_KEY="$SSH_KEY"
136	OWNER_SSH_KEY="$SSH_KEY"
137
138	AM_OWNER=false
139	AM_MIRROR=false
140	[[ "$MY_ID" == "$OWNER_ID" ]] && AM_OWNER=true
141	[[ "$MY_ID" == "$MIRROR_ID" ]] && AM_MIRROR=true
142
143	LOCAL_STATE_FILE="${STATE_DIR}/partnership_${LOCAL_SERVER_NAME}.db"
144	REMOTE_STATE_FILE="${STATE_DIR}/partnership_${REMOTE_SERVER_NAME}.db"
145
146	acquire_lock "strict"
147
148	# ==============================================================================================
149	# ━━━ Preflight ━━━
150	# ==============================================================================================
151	echo ""
152	echo "━━━ $ICON_FALLBACK Partnership Transfer — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
153	echo ""
154
155	if [[ "$AM_MIRROR" == true ]]; then
156	    error "Only the owner ($OWNER / $OWNER_ID) can run --transfer"
157	    error "Run from $OWNER, or use --offboard and re-onboard with roles swapped"
158	    exit 1
159	fi
160
161	if [[ -f "$LOCAL_STATE_FILE" ]]; then
162	    CURRENT_STATE=$(read_state_file "$LOCAL_STATE_FILE" "state")
163	    if [[ "$CURRENT_STATE" == "INACTIVE" ]]; then
164	        error "No active partnership — transfer requires an active partnership"
165	        error "If roles are already correct, check PARTNERSHIP_OWNER_HOST in master.conf"
166	        exit 1
167	    fi
168	fi
169
170	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permanent changes will be made"
171
172	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
173	echo "⚠  WARNING — OWNERSHIP TRANSFER"
174	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
175	echo "  Current owner:  $OWNER_ID ($OWNER)"
176	echo "  Current mirror: $MIRROR_ID ($MIRROR)"
177	echo ""
178	echo "  After transfer:"
179	echo "  New owner:      $MIRROR_ID ($MIRROR)"
180	echo "  New mirror:     $OWNER_ID ($OWNER)"
181	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
182
183	# ==============================================================================================
184	# ━━━ Confirmation ━━━
185	# ==============================================================================================
186	if [[ "$DRY_RUN" == false ]]; then
187	    if [[ -z "$TRANSFER_CONFIRM_INPUT" ]]; then
188	        echo ""
189	        echo "To proceed, pass exactly:"
190	        echo "  --confirm=${PARTNERSHIP_TRANSFER_CONFIRM}"
191	        echo ""
192	        error "Transfer cancelled — confirmation required"
193	        exit 1
194	    fi
195	    if [[ "$TRANSFER_CONFIRM_INPUT" != "$PARTNERSHIP_TRANSFER_CONFIRM" ]]; then
196	        error "Confirmation string does not match — transfer cancelled"
197	        exit 1
198	    fi
199	    log "Confirmation accepted"
200	else
201	    warn "DRY RUN — confirmation check skipped"
202	fi
203
204	# ==============================================================================================
205	# ━━━ Step 1: Health Verification ━━━
206	# ==============================================================================================
207	echo ""
208	echo "━━━ $ICON_SHIELD Step 1: Health Verification ━━━"
209	log "Both servers must pass ${PARTNERSHIP_TRANSFER_STRIKES} consecutive health checks"
210
211	STRIKES=0
212	ATTEMPTS=0
213	MAX_ATTEMPTS="${PARTNERSHIP_TRANSFER_MAX_ATTEMPTS:-20}"
214
215	while [[ "$STRIKES" -lt "$PARTNERSHIP_TRANSFER_STRIKES" ]]; do
216	    (( ATTEMPTS++ ))
217	    if [[ "$ATTEMPTS" -gt "$MAX_ATTEMPTS" ]]; then
218	        error "Health checks failed after $MAX_ATTEMPTS attempts — servers not stable"
219	        error "Transfer cancelled — try again when both servers are healthy"
220	        exit 1
221	    fi
222
223	    if check_both_healthy; then
224	        (( STRIKES++ ))
225	        log "Health check passed ($STRIKES/${PARTNERSHIP_TRANSFER_STRIKES})"
226	        [[ "$STRIKES" -lt "$PARTNERSHIP_TRANSFER_STRIKES" ]] && sleep 10
227	    else
228	        warn "Health check failed — resetting (attempt $ATTEMPTS/$MAX_ATTEMPTS)"
229	        STRIKES=0
230	        sleep 30
231	    fi
232	done
233	warn "Both servers healthy — proceeding ✅"
234
235	# Resolve IPs after health checks confirm reachability
236	MIRROR_IP=$(resolve_tailscale_ip "$MIRROR")
237	OWNER_IP=$(resolve_tailscale_ip "$OWNER")
238	[[ -z "$MIRROR_IP" ]] && { error "Cannot resolve mirror Tailscale IP"; exit 1; }
239
240	# Compute post-transfer roles
241	NEW_OWNER_ID="$MIRROR_ID"
242	NEW_MIRROR_ID="$OWNER_ID"
243	NEW_OWNER="$MIRROR"
244	NEW_MIRROR="$OWNER"
245	NEW_OWNER_IP="$MIRROR_IP"
246
247	# ==============================================================================================
248	# ━━━ Step 2: Pre-transfer Sync ━━━
249	# ==============================================================================================
250	echo ""
251	echo "━━━ $ICON_SYNC Step 2: Pre-transfer Sync ━━━"
252	do_final_sync
253
254	# ==============================================================================================
255	# ━━━ Step 3: Reconfigure WebUIs ━━━
256	# ==============================================================================================
257	echo ""
258	echo "━━━ $ICON_CONTAINERS Step 3: Reconfigure WebUIs ━━━"
259
260	WEBUI_FAILURES=0
261
262	# New owner (current mirror, HOST2) WebUIs → localhost — it now manages itself directly
263	log "New owner ($NEW_OWNER) WebUIs → localhost"
264	for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
265	    [[ -z "$entry" ]] && continue
266	    container="${entry%%|*}"
267	    port="${entry##*|}"
268	    reconfigure_webui "$container" "$port" "localhost" \
269	        "$SSH_KEY" "$MIRROR_IP" "$MIRROR" || (( WEBUI_FAILURES++ ))
270	done
271
272	# New mirror (us, HOST1) WebUIs → new owner IP — defers to new owner going forward
273	log "New mirror ($NEW_MIRROR) WebUIs → $NEW_OWNER_IP"
274	reconfigure_local_webuis "$NEW_OWNER_IP"
275	WEBUI_RC=$?
276	[[ "$WEBUI_RC" -gt 0 ]] && (( WEBUI_FAILURES += WEBUI_RC ))
277
278	# ==============================================================================================
279	# ━━━ Step 4: Flip Ownership in master.conf ━━━
280	# ==============================================================================================
281	echo ""
282	echo "━━━ $ICON_GEAR Step 4: Flip Ownership ━━━"
283
284	if [[ "$DRY_RUN" == false ]]; then
285	    update_master_conf "PARTNERSHIP_OWNER_HOST" "\"$NEW_OWNER_ID\""
286
287	    # Push updated master.conf to new owner so both servers agree immediately.
288	    # master.conf is shared — host-specific credentials live in host*.conf.
289	    _probe_cmd=$(platform_scripts_dir_probe_cmd)
290	    _REMOTE_SD=$(ssh -i "$SSH_KEY" -o ConnectTimeout=5 -o StrictHostKeyChecking=no \
291	        "root@${MIRROR_IP}" "$_probe_cmd" \
292	        2>/dev/null | tr -d '[:space:]')
293	    _REMOTE_SD="${_REMOTE_SD:-$SCRIPTS_DIR}"
294	    scp -i "$SSH_KEY" \
295	        -o ConnectTimeout="$SSH_TIMEOUT" \
296	        -o StrictHostKeyChecking=no \
297	        "$SCRIPTS_ROOT/Configurations/master.conf" \
298	        "root@${MIRROR_IP}:${_REMOTE_SD}/Configurations/master.conf" 2>/dev/null && \
299	        log "master.conf pushed to $NEW_OWNER ✅" || \
300	        error "Failed to push master.conf to $NEW_OWNER — set PARTNERSHIP_OWNER_HOST=\"$NEW_OWNER_ID\" manually"
301	else
302	    warn "DRY RUN — would set PARTNERSHIP_OWNER_HOST=$NEW_OWNER_ID on both servers"
303	fi
304
305	# ==============================================================================================
306	# ━━━ Step 5: Write State ━━━
307	# ==============================================================================================
308	echo ""
309	echo "━━━ $ICON_SYNC Step 5: Write State ━━━"
310
311	NOW=$(date '+%Y-%m-%d %H:%M:%S')
312	write_state_file "$LOCAL_STATE_FILE" "ACTIVE" "$NOW" "" "$LOCAL_SERVER_NAME" "transfer"
313	push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$SSH_KEY"
314
315	# ==============================================================================================
316	# ━━━ Summary ━━━
317	# ==============================================================================================
318	echo ""
319	echo "━━━━━ $ICON_SUMMARY TRANSFER SUMMARY ━━━━━"
320	echo "  New owner:       $NEW_OWNER_ID ($NEW_OWNER — $NEW_OWNER_IP)"
321	echo "  New mirror:      $NEW_MIRROR_ID ($NEW_MIRROR)"
322	echo "  WebUI failures:  $WEBUI_FAILURES"
323	echo "  Sync direction:  $NEW_OWNER → $NEW_MIRROR (next cycle)"
324	echo "  Ownership:       PARTNERSHIP_OWNER_HOST=$NEW_OWNER_ID"
325	echo ""
326	if [[ "$DRY_RUN" == true ]]; then
327	    warn "DRY RUN — no changes made"
328	else
329	    warn "$ICON_DONE DONE — ownership transferred to $NEW_OWNER_ID ($NEW_OWNER) ✅"
330	    echo "fallback.sh and critical_sync_maintenance.sh will adapt on next cycle"
331	    echo "No containers were moved — only config and WebUI targets updated"
332	    [[ "$WEBUI_FAILURES" -gt 0 ]] && \
333	        warn "$WEBUI_FAILURES WebUI(s) failed — check templates manually"
334	    notify "Partnership ownership transferred — new owner: $NEW_OWNER ($NEW_OWNER_ID)" \
335	        "Partnership" "normal"
336	fi
337	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
338	exit 0
339


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= SSH Setup ======================================================
4	# ==============================================================================================
5	# Generates the SSH keypair for rsync automation and installs it on the remote server.
6	# Key named after this server: hostname lowercased, unraid- prefix stripped.
7	#   unRAID-Gmer4Lfe → gmer4lfe_rsync_automation
8	#   unRAID-Jayred365 → jayred365_rsync_automation
9	# Idempotent — skips generation if key already exists (use --force to regenerate).
10	# Updates host*.conf with key path on success.
11	#
12	# ── MODES ─────────────────────────────────────────────────────────────────────────────────────
13	#   (default)       — generate key if missing, copy to remote, update conf
14	#   --force         — regenerate key even if it exists, re-copy to remote
15	#   --validate      — test SSH auth to remote, track strikes, notify at limit
16	#   --status        — show key state, fingerprint, remote connectivity
17	#   --dry-run       — preview without creating, copying, or updating conf
18	#
19	# ── STRIKE SYSTEM (--validate) ────────────────────────────────────────────────────────────────
20	# Called during partnership --check cycles to detect broken SSH auth.
21	# Tracks consecutive SSH auth failures — not network unreachability.
22	# Remote Tailscale IP unreachable = network issue → not counted as SSH strike.
23	# Remote reachable but SSH auth fails = key issue → strike incremented.
24	# Strikes reset automatically after SSH_STRIKE_RESET_HRS of clean connectivity.
25	# At SSH_MAX_STRIKES: notify + return exit 2 (caller can escalate).
26	# State: DATA_DIR/ssh_strikes_{REMOTE_SERVER_NAME}.db
27	#
28	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
29	#   SSH_MAX_STRIKES       — consecutive failures before notifying (default 5)
30	#   SSH_STRIKE_RESET_HRS  — hours since last failure before counter resets (default 24)
31	#
32	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
33	#   Partnership/ssh_setup.sh                — initial setup (idempotent)
34	#   Partnership/ssh_setup.sh --force        — regenerate + re-copy
35	#   Partnership/ssh_setup.sh --validate     — health check + strike tracking
36	#   Partnership/ssh_setup.sh --status       — show key and connectivity state
37	#   Any mode supports --dry-run and --log
38	# ==============================================================================================
39
40	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
41	SCRIPTS_ROOT="$SCRIPT_DIR/.."
42
43	source "$SCRIPTS_ROOT/load_config.sh"
44
45	# ── Parse --force before parse_args ───────────────────────────────────────────────────────────
46	MODE="setup"
47	FORCE=false
48	LOCAL_ONLY=false
49	FILTERED_ARGS=()
50
51	for arg in "$@"; do
52	    case "$arg" in
53	        --force)      FORCE=true ;;
54	        --validate)   MODE="validate" ;;
55	        --status)     MODE="status" ;;
56	        --local-only) LOCAL_ONLY=true ;;
57	        *)            FILTERED_ARGS+=("$arg") ;;
58	    esac
59	done
60
61	parse_args "${FILTERED_ARGS[@]}"
62
63	# ==============================================================================================
64	# ━━━ Setup ━━━
65	# ==============================================================================================
66	if [[ "$EUID" -ne 0 ]]; then
67	    error "Must be run as root"
68	    exit 1
69	fi
70
71	acquire_lock
72
73	detect_hosts
74
75	# ── Derive key name from hostname ─────────────────────────────────────────────────────────────
76	# Strip unraid- prefix (case-insensitive) if present → gmer4lfe_rsync_automation
77	SHORT_NAME="${LOCAL_SERVER_NAME,,}"
78	[[ "${SHORT_NAME}" == unraid-* ]] && SHORT_NAME="${SHORT_NAME:7}"
79	SSH_KEY_NAME="${SHORT_NAME}_rsync_automation"
80	SSH_KEY_PATH="/root/.ssh/${SSH_KEY_NAME}"
81	SSH_PUB_PATH="${SSH_KEY_PATH}.pub"
82
83	# ── Host conf path ────────────────────────────────────────────────────────────────────────────
84	HOST_NUM="${MY_ID#HOST}"   # "1" or "2"
85	HOST_CONF="$SCRIPTS_ROOT/Configurations/host${HOST_NUM}.conf"
86	KEY_CONF_VAR="${MY_ID}_SSH_KEY"
87
88	# ── Strike state file ─────────────────────────────────────────────────────────────────────────
89	SSH_STRIKE_FILE="$DATA_DIR/ssh_strikes_${REMOTE_SERVER_NAME}.db"
90
91	# ==============================================================================================
92	# ─── HELPERS ──────────────────────────────────────────────────────────────────────────────────
93	# ==============================================================================================
94
95	test_ssh_auth() {
96	    local remote_ip="$1"
97	    timeout "${SSH_TIMEOUT:-15}" ssh \
98	        -i "$SSH_KEY_PATH" \
99	        -o BatchMode=yes \
100	        -o ConnectTimeout="${SSH_TIMEOUT:-15}" \
101	        -o StrictHostKeyChecking=no \
102	        root@"$remote_ip" "echo ok" >/dev/null 2>&1
103	}
104
105	read_strikes() {
106	    grep "^strikes=" "$SSH_STRIKE_FILE" 2>/dev/null | cut -d= -f2 || echo 0
107	}
108
109	read_last_strike_epoch() {
110	    local ts
111	    ts=$(grep "^last_strike=" "$SSH_STRIKE_FILE" 2>/dev/null | cut -d= -f2)
112	    [[ -n "$ts" ]] && date -d "$ts" +%s 2>/dev/null || echo 0
113	}
114
115	write_strike_file() {
116	    local strikes="$1" last_strike="${2:-}" last_success="${3:-}"
117	    mkdir -p "$(dirname "$SSH_STRIKE_FILE")"
118	    cat > "$SSH_STRIKE_FILE" <<EOF
119	strikes=${strikes}
120	last_strike=${last_strike}
121	last_success=${last_success}
122	updated=$(date '+%Y-%m-%d %H:%M:%S')
123	EOF
124	}
125
126	update_conf_key_path() {
127	    if [[ ! -f "$HOST_CONF" ]]; then
128	        warn "$(basename "$HOST_CONF") not found — update ${KEY_CONF_VAR} manually to: $SSH_KEY_PATH"
129	        return 1
130	    fi
131
132	    local current
133	    current=$(grep "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null | \
134	              sed 's/.*="\?\([^"]*\)"\?.*/\1/')
135
136	    if [[ "$current" == "$SSH_KEY_PATH" ]]; then
137	        log "${KEY_CONF_VAR} already correct in $(basename "$HOST_CONF")"
138	        return 0
139	    fi
140
141	    if grep -q "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null; then
142	        sed -i "s|^[[:space:]]*${KEY_CONF_VAR}=.*|    ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\"|" "$HOST_CONF" && \
143	            log "${KEY_CONF_VAR} updated in $(basename "$HOST_CONF") ✅" || \
144	            warn "Failed to update ${KEY_CONF_VAR} in $(basename "$HOST_CONF") — update manually"
145	    else
146	        warn "${KEY_CONF_VAR} not found in $(basename "$HOST_CONF") — add manually:"
147	        warn "    ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\""
148	    fi
149	}
150
151	# ==============================================================================================
152	# ━━━ Status ━━━
153	# ==============================================================================================
154	if [[ "$MODE" == "status" ]]; then
155	    echo ""
156	    echo "━━━━━ $ICON_SUMMARY SSH SETUP STATUS ━━━━━"
157	    echo "$ICON_HOST  Identity: $MY_ID ($LOCAL_SERVER_NAME)"
158	    echo "$ICON_NET   Remote:   $REMOTE_SERVER_NAME"
159	    echo ""
160	    echo "  Key name: $SSH_KEY_NAME"
161	    echo "  Key path: $SSH_KEY_PATH"
162	    echo ""
163
164	    if [[ -f "$SSH_KEY_PATH" ]]; then
165	        local_fp=$(ssh-keygen -lf "$SSH_KEY_PATH" 2>/dev/null || echo "unreadable")
166	        echo "  $ICON_DONE  Private key: present"
167	        echo "         Fingerprint: $local_fp"
168	    else
169	        echo "  $ICON_ERROR Private key: NOT found"
170	    fi
171
172	    if [[ -f "$SSH_PUB_PATH" ]]; then
173	        echo "  $ICON_DONE  Public key:  present"
174	    else
175	        echo "  $ICON_ERROR Public key:  NOT found"
176	    fi
177
178	    # Check conf
179	    if [[ -f "$HOST_CONF" ]]; then
180	        current_conf=$(grep "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null | \
181	                       sed 's/.*="\?\([^"]*\)"\?.*/\1/')
182	        if [[ "$current_conf" == "$SSH_KEY_PATH" ]]; then
183	            echo "  $ICON_DONE  Conf ($(basename "$HOST_CONF")): ${KEY_CONF_VAR} ✅"
184	        else
185	            echo "  $ICON_ERROR Conf ($(basename "$HOST_CONF")): ${KEY_CONF_VAR}=${current_conf:-not set}"
186	        fi
187	    fi
188
189	    # Remote connectivity
190	    echo ""
191	    REMOTE_IP=$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")
192	    if [[ -z "$REMOTE_IP" ]]; then
193	        echo "  $ICON_ERROR Remote ($REMOTE_SERVER_NAME): Tailscale unreachable"
194	    elif [[ -f "$SSH_KEY_PATH" ]] && test_ssh_auth "$REMOTE_IP"; then
195	        echo "  $ICON_DONE  Remote ($REMOTE_SERVER_NAME [$REMOTE_IP]): SSH auth OK ✅"
196	    else
197	        echo "  $ICON_ERROR Remote ($REMOTE_SERVER_NAME [$REMOTE_IP]): SSH auth FAILED"
198	    fi
199
200	    # Strike state
201	    if [[ -f "$SSH_STRIKE_FILE" ]]; then
202	        echo ""
203	        echo "  Strike state:"
204	        while IFS= read -r line; do
205	            echo "    $line"
206	        done < "$SSH_STRIKE_FILE"
207	    fi
208
209	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
210	    exit 0
211	fi
212
213	# ==============================================================================================
214	# ━━━ Validate (SSH health check + strike tracking) ━━━
215	# ==============================================================================================
216	if [[ "$MODE" == "validate" ]]; then
217	    MAX_STRIKES="${SSH_MAX_STRIKES:-5}"
218	    RESET_HRS="${SSH_STRIKE_RESET_HRS:-24}"
219
220	    REMOTE_IP=$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")
221	    if [[ -z "$REMOTE_IP" ]]; then
222	        log "SSH validate — $REMOTE_SERVER_NAME Tailscale unreachable, not an SSH issue"
223	        exit 0
224	    fi
225
226	    if [[ ! -f "$SSH_KEY_PATH" ]]; then
227	        warn "SSH validate — key not found at $SSH_KEY_PATH — run ssh_setup.sh to create"
228	        exit 1
229	    fi
230
231	    NOW=$(date '+%Y-%m-%d %H:%M:%S')
232	    NOW_EPOCH=$(date +%s)
233
234	    if test_ssh_auth "$REMOTE_IP"; then
235	        # SSH works — reset strikes if they were > 0
236	        STRIKES=$(read_strikes)
237	        if [[ "$STRIKES" -gt 0 ]]; then
238	            write_strike_file 0 "" "$NOW"
239	            log "SSH validate — auth restored to $REMOTE_SERVER_NAME ✅ (strikes reset)"
240	        else
241	            log "SSH validate — $REMOTE_SERVER_NAME SSH auth OK ✅"
242	        fi
243	        exit 0
244	    fi
245
246	    # SSH auth failed — check if strikes should reset first
247	    STRIKES=$(read_strikes)
248	    LAST_STRIKE_EPOCH=$(read_last_strike_epoch)
249	    RESET_SECS=$(( RESET_HRS * 3600 ))
250
251	    if [[ "$STRIKES" -gt 0 ]] && \
252	       [[ "$LAST_STRIKE_EPOCH" -gt 0 ]] && \
253	       [[ $(( NOW_EPOCH - LAST_STRIKE_EPOCH )) -gt "$RESET_SECS" ]]; then
254	        warn "SSH validate — strike counter reset after ${RESET_HRS}hr gap"
255	        STRIKES=0
256	    fi
257
258	    STRIKES=$(( STRIKES + 1 ))
259	    write_strike_file "$STRIKES" "$NOW" ""
260	    warn "SSH validate — $REMOTE_SERVER_NAME auth FAILED (strike $STRIKES/$MAX_STRIKES)"
261
262	    if [[ "$STRIKES" -ge "$MAX_STRIKES" ]]; then
263	        error "SSH auth to $REMOTE_SERVER_NAME broken — ${STRIKES} consecutive failures"
264	        error "Repair: run 'Partnership/ssh_setup.sh --force' to regenerate and re-copy key"
265	        notify "SSH auth broken to $REMOTE_SERVER_NAME ($MY_ID) — ${STRIKES} strikes, manual repair needed. Run: Partnership/ssh_setup.sh --force" \
266	            "SSH Setup" "warning"
267	        exit 2
268	    fi
269
270	    exit 1
271	fi
272
273	# ==============================================================================================
274	# ━━━ Setup (default) or --force ━━━
275	# ==============================================================================================
276	echo ""
277	echo "━━━ $ICON_GEAR SSH Key Setup — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
278	echo ""
279	echo "  Key name: $SSH_KEY_NAME"
280	echo "  Key path: $SSH_KEY_PATH"
281
282	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no keys will be created or copied"
283
284	# ── Key generation ────────────────────────────────────────────────────────────────────────────
285	echo ""
286	echo "━━━ $ICON_GEAR Key Generation ━━━"
287
288	if [[ -f "$SSH_KEY_PATH" ]] && [[ "$FORCE" == false ]]; then
289	    local_fp=$(ssh-keygen -lf "$SSH_KEY_PATH" 2>/dev/null || echo "unreadable")
290	    echo "Key already exists — skipping generation (--force to regenerate)"
291	    echo "  $local_fp"
292	else
293	    if [[ "$FORCE" == true ]] && [[ -f "$SSH_KEY_PATH" ]]; then
294	        warn "Regenerating key (--force) — existing key will be replaced"
295	        if [[ "$DRY_RUN" == false ]]; then
296	            rm -f "$SSH_KEY_PATH" "$SSH_PUB_PATH"
297	        fi
298	    fi
299
300	    warn "Generating ed25519 keypair: $SSH_KEY_PATH"
301	    if [[ "$DRY_RUN" == false ]]; then
302	        ssh-keygen -t ed25519 -N "" -f "$SSH_KEY_PATH" -C "${SSH_KEY_NAME}@${LOCAL_SERVER_NAME}" && \
303	            log "Keypair generated ✅" || {
304	            error "Failed to generate keypair"
305	            exit 1
306	        }
307	        chmod 600 "$SSH_KEY_PATH"
308	        chmod 644 "$SSH_PUB_PATH"
309	    else
310	        warn "DRY RUN — would generate: $SSH_KEY_PATH"
311	    fi
312	fi
313
314	# ── Update conf ───────────────────────────────────────────────────────────────────────────────
315	echo ""
316	echo "━━━ $ICON_GEAR Update host${HOST_NUM}.conf ━━━"
317
318	if [[ "$DRY_RUN" == false ]]; then
319	    update_conf_key_path
320	else
321	    warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in host${HOST_NUM}.conf"
322	fi
323
324	# ── Copy to remote ────────────────────────────────────────────────────────────────────────────
325	if [[ "$LOCAL_ONLY" == true ]]; then
326	    echo ""
327	    echo "━━━━━ $ICON_SUMMARY SSH SETUP SUMMARY (local) ━━━━━"
328	    echo "  Key:    $SSH_KEY_PATH"
329	    echo "  Pubkey: $SSH_PUB_PATH"
330	    echo "  Conf:   ${KEY_CONF_VAR} in host${HOST_NUM}.conf"
331	    echo ""
332	    warn "Local setup complete — copy public key to remote manually:"
333	    warn "  ssh-copy-id -i $SSH_PUB_PATH root@<remote>"
334	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
335	    exit 0
336	fi
337
338	echo ""
339	echo "━━━ $ICON_NET Copy Public Key to Remote ($REMOTE_SERVER_NAME) ━━━"
340
341	resolve_remote_ip
342
343	if [[ "$DRY_RUN" == true ]]; then
344	    warn "DRY RUN — would ssh-copy-id to root@$REMOTE_SERVER"
345	    warn "DRY RUN — would require password for root@$REMOTE_SERVER if key auth not yet set up"
346	else
347	    warn "Installing public key on $REMOTE_SERVER_NAME ($REMOTE_SERVER)..."
348	    warn "(Password prompt for root@$REMOTE_SERVER is expected on first setup)"
349
350	    if ssh-copy-id -i "$SSH_PUB_PATH" -o ConnectTimeout="${SSH_TIMEOUT:-15}" \
351	        root@"$REMOTE_SERVER" 2>/dev/null; then
352	        echo "Public key installed on $REMOTE_SERVER_NAME ✅"
353	    else
354	        error "ssh-copy-id failed — check that:"
355	        error "  1. Remote server is reachable: tailscale status"
356	        error "  2. Password auth is enabled on remote: grep PasswordAuthentication /etc/ssh/sshd_config"
357	        error "  3. The correct password for root@$REMOTE_SERVER_NAME is used"
358	        exit 1
359	    fi
360	fi
361
362	# ── Verify ────────────────────────────────────────────────────────────────────────────────────
363	echo ""
364	echo "━━━ $ICON_VERIFY Verify SSH Auth ━━━"
365
366	if [[ "$DRY_RUN" == false ]]; then
367	    if test_ssh_auth "$REMOTE_SERVER"; then
368	        echo "SSH auth to $REMOTE_SERVER_NAME working ✅"
369	        if [[ -f "$SSH_STRIKE_FILE" ]]; then
370	            write_strike_file 0 "" "$(date '+%Y-%m-%d %H:%M:%S')"
371	        fi
372	    else
373	        warn "SSH auth test failed — key may need a moment to propagate"
374	        warn "Verify manually: ssh -i $SSH_KEY_PATH root@$REMOTE_SERVER"
375	    fi
376	else
377	    warn "DRY RUN — would test: ssh -i $SSH_KEY_PATH root@$REMOTE_SERVER echo ok"
378	fi
379
380	# ── Summary ───────────────────────────────────────────────────────────────────────────────────
381	echo ""
382	echo "━━━━━ $ICON_SUMMARY SSH SETUP SUMMARY ━━━━━"
383	echo "  Key:    $SSH_KEY_PATH"
384	echo "  Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)"
385	echo "  Conf:   ${KEY_CONF_VAR} in host${HOST_NUM}.conf"
386	echo ""
387	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
388	    warn "$ICON_DONE DONE — SSH key ready for rsync automation ✅"
389	echo "━━━━━━━━━━━━━━━━━━━━━━━"
390	exit 0
391


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Partnership Onboard ============================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Runs once on both servers to establish a new partnership. Role is detected
9	# automatically via detect_hosts() — no flags needed to declare which side you are.
10	# Run on the mirror first (generates its SSH key), then on the owner to complete
11	# setup remotely.
12	#
13	# ==============================================================================================
14	# OPERATIONAL MODEL
15	# ==============================================================================================
16	#
17	# MIRROR PATH (1 step)
18	#   Step 1: SSH key setup — generate keypair, copy to owner, update conf
19	#           Owner completes the rest remotely. Mirror is done.
20	#
21	# OWNER PATH (10 steps)
22	#   Step 1:  SSH key setup         — generate keypair, install on mirror, update conf
23	#   Step 2:  Stop mirror auth      — stop mirror's existing auth containers before replacing
24	#   Step 3:  Deploy auth stack     — push XMLs, pull images, create + start on mirror
25	#                                    Mariadb/Redis health-checked before Authelia deploys
26	#   Step 4:  Stop mirror arr       — stop mirror's existing arr containers before replacing
27	#   Step 5:  Deploy arr stack      — push arr XMLs, pull images, create + start on mirror
28	#   Step 6:  Stop mirror services  — stop mirror's existing services containers before replacing
29	#   Step 7:  Deploy services stack — push Emby/Jellyfin/Seerr XMLs, pull images, create + start
30	#   Step 8:  Partnership onboard   — configure WebUIs → owner IP, write state, Emby
31	#   Step 9:  Arr bootstrap         — bidirectional library sync (arr_sync.sh)
32	#   Step 10: Conf push             — push master.conf + setup state to all listed hosts
33	#
34	# ==============================================================================================
35	# DESIGN PRINCIPLES
36	# ==============================================================================================
37	#
38	# Credentials never in SSH command strings
39	#   Auth stack containers hold API keys, DB passwords, etc. The deploy script is written
40	#   locally, SCPed to the remote, and executed there. Command-line args are never used
41	#   to pass credentials — they'd appear in `ps` output and shell history on both servers.
42	#
43	# XML templates are the single source of truth for deployed containers
44	#   The owner's templates-user/ XMLs define every container deployed on the mirror.
45	#   The same XMLs that Unraid's Docker Manager uses are what get SCPed — the mirror's
46	#   Docker Manager can manage the containers after onboard without additional config.
47	#
48	# Dependency ordering in the auth stack is owner-enforced
49	#   PARTNERSHIP_AUTH_STACK order matters: Mariadb and Redis must come before Authelia.
50	#   The array is ordered correctly in host1.conf. After each Mariadb/Redis deploy,
51	#   the script waits for the container to be healthy before continuing. This is a remote
52	#   health check — the container must be running (or report healthy) before the next
53	#   dependent is deployed.
54	#
55	# ==============================================================================================
56	# OPERATIONAL SAFEGUARDS
57	# ==============================================================================================
58	#
59	# Root check
60	#   All operations run as root — SSH key management, docker operations, conf updates.
61	#
62	# SSH timeout on all remote calls
63	#   Every ssh/scp call uses SSH_TIMEOUT. No operation hangs indefinitely on a
64	#   slow or unreachable mirror.
65	#
66	# --dry-run shows exact actions without executing
67	#   Every step prints what it would do. SCP, deploy, plugin install, arr sync —
68	#   all dry-run safe.
69	#
70	# Step skip flags for partial re-runs
71	#   --skip-ssh, --skip-auth-stack, --skip-arr-stack, --skip-arr-sync allow
72	#   resuming after a partial failure without re-running completed steps.
73	#
74	# ==============================================================================================
75	# CONFIGURATION
76	# ==============================================================================================
77	#
78	# host*.conf
79	#
80	#   HOST*_PARTNERSHIP_AUTH_STACK
81	#       XML filenames (from this server's templates-user/) to push and deploy on the
82	#       mirror as its auth stack. Order matters: database deps before Authelia.
83	#       Aliased by detect_hosts() → PARTNERSHIP_AUTH_STACK
84	#
85	#   HOST*_PARTNERSHIP_REPLACE_CONTAINERS
86	#       Containers to stop on the mirror before deploying the auth stack.
87	#       Defined in the MIRROR's own conf (host*.conf on HOST2) — never in HOST1's conf.
88	#       Read live from the mirror via SSH during Step 3 (sources mirror's load_config.sh at
89	#       the same $SCRIPTS_ROOT path — convention: both servers use the same repo location).
90	#       Leave empty on HOST2 if no conflicting containers exist (fresh mirror: nothing to stop).
91	#       Aliased by detect_hosts() → PARTNERSHIP_REPLACE_CONTAINERS (on the mirror)
92	#
93	#   HOST*_PARTNERSHIP_ARR_STACK
94	#       XML filenames to push and deploy on the mirror as its arr stack.
95	#       Leave empty to skip arr stack deploy.
96	#       Aliased by detect_hosts() → PARTNERSHIP_ARR_STACK
97	#
98	#   HOST*_PARTNERSHIP_ARR_REPLACE_CONTAINERS
99	#       Arr containers to stop on the mirror before deploying the arr stack.
100	#       Same rule as PARTNERSHIP_REPLACE_CONTAINERS: defined in mirror's own conf, never HOST1's.
101	#       Aliased by detect_hosts() → PARTNERSHIP_ARR_REPLACE_CONTAINERS (on the mirror)
102	#
103	#   HOST*_PARTNERSHIP_SERVICES_STACK
104	#       XML filenames to push and deploy on the mirror as its shared services stack.
105	#       Includes Emby, Jellyfin, Seerr, SeerrFin. Leave empty to skip services stack deploy.
106	#       Aliased by detect_hosts() → PARTNERSHIP_SERVICES_STACK
107	#
108	#   HOST*_PARTNERSHIP_SERVICES_REPLACE_CONTAINERS
109	#       Services containers to stop on the mirror before deploying the services stack.
110	#       Same rule as PARTNERSHIP_REPLACE_CONTAINERS: defined in mirror's own conf, never HOST1's.
111	#       Aliased by detect_hosts() → PARTNERSHIP_SERVICES_REPLACE_CONTAINERS (on the mirror)
112	#
113	# ==============================================================================================
114	# RUNTIME MODES
115	# ==============================================================================================
116	#
117	# Partnership/partnership_onboard.sh
118	#     Full onboard — role detected automatically
119	#
120	# Partnership/partnership_onboard.sh --dry-run
121	#     Preview all steps without making changes
122	#
123	# Partnership/partnership_onboard.sh --log
124	#     Verbose per-step output
125	#
126	# Partnership/partnership_onboard.sh --skip-ssh
127	#     Skip SSH key setup (key already in place)
128	#
129	# Partnership/partnership_onboard.sh --skip-auth-stack
130	#     Skip auth stack stop + deploy (Steps 3-4)
131	#
132	# Partnership/partnership_onboard.sh --skip-arr-stack
133	#     Skip arr stack stop + deploy (Steps 4-5)
134	#
135	# Partnership/partnership_onboard.sh --skip-services-stack
136	#     Skip services stack stop + deploy (Steps 6-7)
137	#
138	# Partnership/partnership_onboard.sh --skip-arr-sync
139	#     Skip arr library bootstrap (Step 9)
140	#
141	# Partnership/partnership_onboard.sh --phase1-only
142	#     OWNER only: SSH key exchange + conf push. Safe to run before HOST2 has Varaverk.
143	#     Writes HOST2_PHASE1_DONE=true to varaverk_setup.db.
144	#
145	# Partnership/partnership_onboard.sh --phase2-only
146	#     OWNER only: container deploy + arr + onboard (skips SSH). Triggered automatically
147	#     by HOST2 after it completes its Mirror-path onboard. Can also be run manually.
148	#     Writes HOST2_PHASE2_DONE=true to varaverk_setup.db.
149	#
150	# ==============================================================================================
151
152	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
153	SCRIPTS_ROOT="$SCRIPT_DIR/.."
154	SSH_TIMEOUT=15
155
156	source "$SCRIPTS_ROOT/load_config.sh"
157	source "$SCRIPTS_ROOT/Plugin/$PLATFORM/Partnership/containers.sh"
158
159	# ── Parse flags ───────────────────────────────────────────────────────────────────────────────
160	SKIP_SSH=false
161	SKIP_AUTH_STACK=false
162	SKIP_ARR_STACK=false
163	SKIP_SERVICES_STACK=false
164	SKIP_ARR_SYNC=false
165	PHASE1_ONLY=false   # OWNER: SSH + conf push only (HOST2 not yet installed)
166	PHASE2_ONLY=false   # OWNER: containers/arr/onboard only (triggered by HOST2 after it onboards)
167	FILTERED_ARGS=()
168
169	for arg in "$@"; do
170	    case "$arg" in
171	        --skip-ssh)        SKIP_SSH=true ;;
172	        --skip-auth-stack) SKIP_AUTH_STACK=true ;;
173	        --skip-arr-stack)      SKIP_ARR_STACK=true ;;
174	        --skip-services-stack) SKIP_SERVICES_STACK=true ;;
175	        --skip-arr-sync)       SKIP_ARR_SYNC=true ;;
176	        --phase1-only)     PHASE1_ONLY=true ;;
177	        --phase2-only)     PHASE2_ONLY=true; SKIP_SSH=true ;;
178	        *)                 FILTERED_ARGS+=("$arg") ;;
179	    esac
180	done
181
182	parse_args "${FILTERED_ARGS[@]}"
183
184	# ==============================================================================================
185	# ━━━ Setup ━━━
186	# ==============================================================================================
187	[[ "$EUID" -ne 0 ]] && { error "Must be run as root"; exit 1; }
188
189	acquire_lock
190
191	if ! command -v docker &>/dev/null; then
192	    error "Docker command not found"
193	    exit 1
194	fi
195
196	detect_hosts
197
198	OWNER_ID="${PARTNERSHIP_OWNER_HOST:-HOST1}"
199	MIRROR_ID=$( [[ "$OWNER_ID" == "HOST1" ]] && echo "HOST2" || echo "HOST1" )
200	OWNER="${!OWNER_ID}"
201	MIRROR="${!MIRROR_ID}"
202	# SSH_KEY (set by detect_hosts) is this server's own private key.
203	# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
204	# HOST{N}_SSH_KEY lives in host{N}.conf — with sparse checkout, the other server's
205	# conf is never present here. Always use SSH_KEY (local private key) for outbound SSH.
206	MIRROR_SSH_KEY="$SSH_KEY"
207
208	AM_OWNER=false
209	AM_MIRROR=false
210	[[ "$MY_ID" == "$OWNER_ID" ]] && AM_OWNER=true
211	[[ "$MY_ID" == "$MIRROR_ID" ]] && AM_MIRROR=true
212
213	EXTRA_FLAGS=()
214	[[ "$DRY_RUN"  == true ]] && EXTRA_FLAGS+=("--dry-run")
215	[[ "$LOG_MODE" == true ]] && EXTRA_FLAGS+=("--log")
216
217	START=$(date +%s)
218
219	# ── Helper: write phase completion flag to setup.db + push to remotes ─────────────────────────
220	write_onboard_phase() {
221	    local target_id="$1" phase="$2"
222	    local key="${target_id}_PHASE${phase}_DONE"
223	    local state_file="$(platform_setup_db_path)"
224	    [[ "$DRY_RUN" == true ]] && { warn "DRY RUN — would write ${key}=true"; return 0; }
225	    if grep -q "^${key}=" "$state_file" 2>/dev/null; then
226	        sed -i "s|^${key}=.*|${key}=true|" "$state_file"
227	    else
228	        echo "${key}=true" >> "$state_file"
229	    fi
230	    platform_push_setup_state
231	}
232
233	echo ""
234	echo "━━━ $ICON_FALLBACK Partnership Onboard — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
235	echo ""
236	echo "  Role:    $( [[ "$AM_OWNER" == true ]] && echo "OWNER" || echo "MIRROR" )"
237	echo "  This:    $MY_ID ($LOCAL_SERVER_NAME)"
238	echo "  Partner: $( [[ "$AM_OWNER" == true ]] && echo "$MIRROR_ID ($MIRROR)" || echo "$OWNER_ID ($OWNER)" )"
239	echo ""
240	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permanent changes will be made"
241
242	# ==============================================================================================
243	# ── HELPER: stop containers on the mirror by reading its own conf via SSH ────────────────────
244	#
245	# SSHes to the mirror, sources its load_config.sh at the same $SCRIPTS_ROOT path (both servers
246	# use the same convention), and reads the named config array from the mirror's own conf.
247	# HOST2's container list stays in HOST2's host2.conf — not duplicated in HOST1's conf.
248	# Fails gracefully if scripts aren't present yet or the array is empty (nothing to stop).
249	#
250	# deploy_container_from_xml() already stops/removes containers with the same name as what's
251	# being deployed. This step handles containers with DIFFERENT names that conflict.
252	# ==============================================================================================
253	stop_mirror_stack() {
254	    local config_var="$1" label="$2"
255	    local -a to_stop=()
256
257	    mapfile -t to_stop < <(
258	        timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" \
259	            -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
260	            "source '$SCRIPTS_ROOT/load_config.sh' 2>/dev/null
261	             detect_hosts 2>/dev/null
262	             printf '%s\n' \"\${${config_var}[@]:-}\"" 2>/dev/null | grep -v '^$'
263	    )
264
265	    if [[ ${#to_stop[@]} -eq 0 ]]; then
266	        log "No $label containers to stop on $MIRROR — skipping"
267	        return 0
268	    fi
269
270	    log "Stopping $label on $MIRROR: ${to_stop[*]}"
271	    for container in "${to_stop[@]}"; do
272	        if [[ "$DRY_RUN" == true ]]; then
273	            warn "  DRY RUN — would stop + rm $container on $MIRROR"
274	            continue
275	        fi
276	        timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" \
277	            -o ConnectTimeout="$SSH_TIMEOUT" root@"$MIRROR_IP" \
278	            "docker stop '$container' 2>/dev/null
279	             docker rm   '$container' 2>/dev/null && echo removed" 2>/dev/null | \
280	            grep -q removed && \
281	            log "  $container removed ✅" || \
282	            log "  $container not found on $MIRROR — skipping"
283	    done
284	}
285
286	# ==============================================================================================
287	# ── MIRROR PATH ───────────────────────────────────────────────────────────────────────────────
288	# ==============================================================================================
289	if [[ "$AM_MIRROR" == true ]]; then
290	    echo "━━━ Step 1/2 — SSH Key Setup (Mirror) ━━━"
291	    echo ""
292	    echo "  Mirror sets up SSH keys, then notifies Owner to run Phase 2."
293	    echo ""
294
295	    if [[ "$SKIP_SSH" == true ]]; then
296	        warn "Skipping SSH setup (--skip-ssh)"
297	    elif bash "$SCRIPT_DIR/ssh_setup.sh" "${EXTRA_FLAGS[@]}"; then
298	        log "SSH key ready ✅"
299	    else
300	        error "SSH key setup failed"
301	        exit 1
302	    fi
303
304	    echo ""
305	    echo "━━━ Step 2/2 — Notify Owner to Run Phase 2 ━━━"
306	    echo ""
307
308	    OWNER_IP=$(resolve_tailscale_ip "$OWNER" 2>/dev/null || true)
309	    PHASE2_TRIGGERED=false
310
311	    if [[ -n "$OWNER_IP" ]]; then
312	        # Read OWNER's SCRIPTS_DIR via platform probe command — don't assume same path as mirror
313	        _probe_cmd=$(platform_scripts_dir_probe_cmd)
314	        OWNER_SCRIPTS_DIR=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
315	            -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$OWNER_IP" \
316	            "$_probe_cmd" 2>/dev/null | tr -d '[:space:]')
317	        OWNER_SCRIPTS_DIR="${OWNER_SCRIPTS_DIR:-$SCRIPTS_DIR}"
318
319	        if [[ "$DRY_RUN" == true ]]; then
320	            warn "DRY RUN — would SSH to $OWNER ($OWNER_IP) and trigger Phase 2"
321	            PHASE2_TRIGGERED=true
322	        elif timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
323	            -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$OWNER_IP" \
324	            "nohup bash '${OWNER_SCRIPTS_DIR}/Partnership/partnership_onboard.sh' --phase2-only > /tmp/vv_phase2_onboard.log 2>&1 & echo triggered" \
325	            2>/dev/null | grep -q triggered; then
326	            log "Phase 2 triggered on $OWNER ✅"
327	            log "Watch progress on $OWNER: tail -f /tmp/vv_phase2_onboard.log"
328	            PHASE2_TRIGGERED=true
329	        else
330	            warn "Could not auto-trigger Phase 2 on $OWNER"
331	        fi
332	    else
333	        warn "Cannot resolve $OWNER Tailscale IP"
334	    fi
335
336	    echo ""
337	    echo "━━━━━ $ICON_SUMMARY MIRROR SETUP COMPLETE ━━━━━"
338	    echo "  SSH key:        ready"
339	    echo "  Phase 2 on $OWNER: $( [[ "$PHASE2_TRIGGERED" == true ]] && echo "triggered ✅" || echo "needs manual trigger ⚠" )"
340	    if [[ "$PHASE2_TRIGGERED" == false ]]; then
341	        echo ""
342	        echo "  Run manually on $OWNER:"
343	        echo "    bash Partnership/partnership_onboard.sh --phase2-only"
344	    fi
345	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
346	    exit 0
347	fi
348
349	# ==============================================================================================
350	# ── OWNER PATH ────────────────────────────────────────────────────────────────────────────────
351	# ==============================================================================================
352
353	MIRROR_IP=$(resolve_tailscale_ip "$MIRROR")
354	[[ -z "$MIRROR_IP" ]] && { error "Cannot resolve $MIRROR Tailscale IP — is Tailscale running?"; exit 1; }
355	log "Mirror: $MIRROR ($MIRROR_IP)"
356	[[ "$PHASE1_ONLY" == true ]] && log "Mode: Phase 1 only (SSH + conf push)"
357	[[ "$PHASE2_ONLY" == true ]] && log "Mode: Phase 2 only (containers + arr + onboard)"
358	echo ""
359
360	STEP_SSH_OK=false
361	STEP_STOP_AUTH_OK=true
362	STEP_AUTH_OK=true
363	AUTH_DEPLOYED=0
364	AUTH_FAILED=0
365	STEP_STOP_ARR_OK=true
366	STEP_ARR_OK=true
367	ARR_DEPLOYED=0
368	ARR_FAILED=0
369	STEP_STOP_SERVICES_OK=true
370	STEP_SERVICES_OK=true
371	SERVICES_DEPLOYED=0
372	SERVICES_FAILED=0
373	ONBOARD_OK=false
374	ARR_SYNC_OK=false
375	MASTER_PUSH_OK=false
376
377	# ── Step 1: SSH ───────────────────────────────────────────────────────────────────────────────
378	# Skipped when --phase2-only (SSH was already done in Phase 1).
379	echo "━━━ Step 1 — SSH Key Setup ━━━"
380
381	if [[ "$SKIP_SSH" == true ]]; then
382	    warn "Skipping (--skip-ssh)"
383	    STEP_SSH_OK=true
384	elif [[ "$PHASE1_ONLY" == true ]]; then
385	    # Phase 1 in background: test if SSH already works first — avoids ssh-copy-id
386	    # hanging for a password prompt with no TTY.
387	    if timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
388	        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" exit 0 2>/dev/null; then
389	        log "SSH to $MIRROR already works ✅ — skipping key install"
390	        STEP_SSH_OK=true
391	    else
392	        # Key not yet on HOST2 — try ssh_setup.sh (works interactively, may fail in background)
393	        if bash "$SCRIPT_DIR/ssh_setup.sh" "${EXTRA_FLAGS[@]}"; then
394	            log "SSH keys ready ✅"
395	            STEP_SSH_OK=true
396	        else
397	            # Soft-fail: generate key locally if not present, then tell user to install manually
398	            warn "Could not install key on $MIRROR automatically (no terminal for password prompt)"
399	            if [[ -f "$SSH_KEY" ]]; then
400	                log "Local key exists at: $SSH_KEY"
401	            else
402	                bash "$SCRIPT_DIR/ssh_setup.sh" --key-only "${EXTRA_FLAGS[@]}" 2>/dev/null || true
403	            fi
404	            if [[ -f "${SSH_KEY}.pub" ]]; then
405	                echo ""
406	                echo "  Install this key on $MIRROR to complete SSH setup:"
407	                echo "  ┌─────────────────────────────────────────────────────"
408	                cat "${SSH_KEY}.pub" | sed 's/^/  │ /'
409	                echo "  └─────────────────────────────────────────────────────"
410	                echo "  Run on a terminal: ssh-copy-id -i ${SSH_KEY}.pub root@${MIRROR_IP}"
411	                echo "  Then click 'Push Conf' in the Partnership tab."
412	                # Write key-ready flag so UI can show the manual-install state
413	                [[ "$DRY_RUN" == false ]] && {
414	                    local kflag="${MIRROR_ID}_KEY_READY"
415	                    local _setup_f="$(platform_setup_db_path)"
416	                    grep -q "^${kflag}=" "$_setup_f" 2>/dev/null \
417	                        && sed -i "s|^${kflag}=.*|${kflag}=true|" "$_setup_f" \
418	                        || echo "${kflag}=true" >> "$_setup_f"
419	                }
420	            fi
421	            STEP_SSH_OK=false
422	        fi
423	    fi
424	elif bash "$SCRIPT_DIR/ssh_setup.sh" "${EXTRA_FLAGS[@]}"; then
425	    log "SSH keys ready ✅"
426	    STEP_SSH_OK=true
427	else
428	    error "SSH key setup failed — aborting"
429	    error "Re-run or use --skip-ssh if key is already set up"
430	    exit 1
431	fi
432
433	# ── Phase 1 exit point ────────────────────────────────────────────────────────────────────────
434	# --phase1-only: SSH + conf push is all HOST1 needs to do before HOST2 installs Varaverk.
435	# HOST2's wizard will detect the pushed master.conf + state file and take the correct path.
436	if [[ "$PHASE1_ONLY" == true ]]; then
437	    if [[ "$STEP_SSH_OK" == false ]]; then
438	        # SSH key not yet installed on HOST2 — can't push conf, but local setup still runs.
439	        # UI will show "key ready, install manually" state via HOST2_KEY_READY flag.
440	        echo ""
441	        echo "━━━ Phase 1 — HOST1 Local Setup (SSH pending) ━━━"
442	        bash "$SCRIPT_DIR/partnership_manager.sh" --onboard --local-only "${EXTRA_FLAGS[@]}" || \
443	            warn "Local setup had issues — check partnership_manager.sh output above"
444
445	        END=$(date +%s)
446	        echo ""
447	        echo "━━━━━ $ICON_SUMMARY PHASE 1 — SSH PENDING ━━━━━"
448	        echo "  SSH keys:    key generated ✅ — NOT yet installed on $MIRROR ⚠"
449	        echo "  Conf push:   skipped (needs SSH access to $MIRROR)"
450	        echo "  HOST1 setup: done ✅"
451	        echo "  Duration:    $(format_duration $(( END - START )))"
452	        echo ""
453	        echo "  ACTION NEEDED: install the key on $MIRROR:"
454	        echo "    ssh-copy-id -i ${SSH_KEY}.pub root@${MIRROR_IP}"
455	        echo "  Then click 'Push Conf' in Partnership tab, or run:"
456	        echo "    bash Partnership/partnership_onboard.sh --phase1-only --skip-ssh"
457	        echo "━━━━━━━━━━━━━━━━━━━━━━━"
458	        exit 0
459	    fi
460
461	    echo ""
462	    echo "━━━ Phase 1 — Conf Push ━━━"
463
464	    CONF_PUSH_OK=false
465	    if [[ "$DRY_RUN" == true ]]; then
466	        warn "DRY RUN — would push master.conf + state file to $MIRROR"
467	        CONF_PUSH_OK=true
468	    else
469	        push_output=$(platform_push_conf)
470	        push_rc=$?
471	        [[ -n "$push_output" ]] && echo "$push_output"
472	        platform_push_setup_state
473	        if [[ $push_rc -eq 0 ]]; then
474	            log "Conf push complete ✅"
475	            CONF_PUSH_OK=true
476	        else
477	            warn "Conf push had failures — retry via Scheduler → master.conf → Save Conf"
478	        fi
479	    fi
480
481	    # HOST1 local setup — runs immediately without needing HOST2
482	    echo ""
483	    echo "━━━ Phase 1 — HOST1 Local Setup ━━━"
484	    bash "$SCRIPT_DIR/partnership_manager.sh" --onboard --local-only "${EXTRA_FLAGS[@]}" || \
485	        warn "Local setup had issues — check partnership_manager.sh output above"
486
487	    [[ "$DRY_RUN" == false ]] && write_onboard_phase "$MIRROR_ID" 1
488
489	    END=$(date +%s)
490	    echo ""
491	    echo "━━━━━ $ICON_SUMMARY PHASE 1 COMPLETE ━━━━━"
492	    echo "  SSH keys:     $( [[ "$STEP_SSH_OK" == true ]] && echo "ready ✅" || echo "skipped" )"
493	    echo "  Conf push:    $( [[ "$CONF_PUSH_OK" == true ]] && echo "done ✅" || echo "⚠ manual needed" )"
494	    echo "  HOST1 setup:  done ✅"
495	    echo "  Duration:     $(format_duration $(( END - START )))"
496	    echo ""
497	    echo "  HOST1 is fully set up. HOST2 ($MIRROR) can now install the Varaverk plugin."
498	    echo "  The wizard will detect the pushed conf and take the correct path."
499	    echo "  When HOST2 completes its onboard, it will automatically trigger Phase 2 here."
500	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
501	    exit 0
502	fi
503
504	# ── Step 2: Stop mirror's existing auth stack ─────────────────────────────────────────────────
505	echo ""
506	echo "━━━ Step 2 — Stop Mirror Auth Stack ━━━"
507
508	if [[ "$SKIP_AUTH_STACK" == true ]]; then
509	    warn "Skipping (--skip-auth-stack)"
510	else
511	    stop_mirror_stack "PARTNERSHIP_REPLACE_CONTAINERS" "auth stack"
512	fi
513
514	# ── Step 4: Deploy auth stack on mirror ───────────────────────────────────────────────────────
515	echo ""
516	echo "━━━ Step 3 — Deploy Auth Stack on Mirror ━━━"
517
518	if [[ "$SKIP_AUTH_STACK" == true ]]; then
519	    warn "Skipping (--skip-auth-stack)"
520	elif [[ ${#PARTNERSHIP_AUTH_STACK[@]} -eq 0 ]]; then
521	    warn "PARTNERSHIP_AUTH_STACK not set in ${MY_ID} conf — skipping auth stack deploy"
522	    warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to host${MY_ID: -1}.conf"
523	    STEP_AUTH_OK=false
524	else
525	    deploy_xml_stack PARTNERSHIP_AUTH_STACK
526	    AUTH_DEPLOYED=$_STACK_DEPLOYED
527	    AUTH_FAILED=$_STACK_FAILED
528	    echo "Auth stack: $AUTH_DEPLOYED deployed, $AUTH_FAILED failed"
529	    [[ "$AUTH_FAILED" -gt 0 ]] && STEP_AUTH_OK=false
530	fi
531
532	# ── Step 5: Stop mirror's existing arr stack ──────────────────────────────────────────────────
533	echo ""
534	echo "━━━ Step 4 — Stop Mirror Arr Stack ━━━"
535
536	if [[ "$SKIP_ARR_STACK" == true ]]; then
537	    warn "Skipping (--skip-arr-stack)"
538	elif [[ ${#PARTNERSHIP_ARR_STACK[@]} -eq 0 ]]; then
539	    log "PARTNERSHIP_ARR_STACK not configured — skipping arr stack deploy"
540	    SKIP_ARR_STACK=true
541	else
542	    stop_mirror_stack "PARTNERSHIP_ARR_REPLACE_CONTAINERS" "arr stack"
543	fi
544
545	# ── Step 5: Deploy arr stack on mirror ───────────────────────────────────────────────────────
546	echo ""
547	echo "━━━ Step 5 — Deploy Arr Stack on Mirror ━━━"
548
549	if [[ "$SKIP_ARR_STACK" == true ]]; then
550	    warn "Skipping (--skip-arr-stack)"
551	else
552	    deploy_xml_stack PARTNERSHIP_ARR_STACK
553	    ARR_DEPLOYED=$_STACK_DEPLOYED
554	    ARR_FAILED=$_STACK_FAILED
555	    echo "Arr stack: $ARR_DEPLOYED deployed, $ARR_FAILED failed"
556	    [[ "$ARR_FAILED" -gt 0 ]] && STEP_ARR_OK=false
557	fi
558
559	# ── Step 6: Stop mirror's existing services stack ─────────────────────────────────────────────
560	echo ""
561	echo "━━━ Step 6 — Stop Mirror Services Stack ━━━"
562
563	if [[ "$SKIP_SERVICES_STACK" == true ]]; then
564	    warn "Skipping (--skip-services-stack)"
565	elif [[ ${#PARTNERSHIP_SERVICES_STACK[@]} -eq 0 ]]; then
566	    log "PARTNERSHIP_SERVICES_STACK not configured — skipping services stack deploy"
567	    SKIP_SERVICES_STACK=true
568	else
569	    stop_mirror_stack "PARTNERSHIP_SERVICES_REPLACE_CONTAINERS" "services stack"
570	fi
571
572	# ── Step 7: Deploy services stack on mirror ───────────────────────────────────────────────────
573	echo ""
574	echo "━━━ Step 7 — Deploy Services Stack on Mirror ━━━"
575
576	if [[ "$SKIP_SERVICES_STACK" == true ]]; then
577	    warn "Skipping (--skip-services-stack)"
578	else
579	    deploy_xml_stack PARTNERSHIP_SERVICES_STACK
580	    SERVICES_DEPLOYED=$_STACK_DEPLOYED
581	    SERVICES_FAILED=$_STACK_FAILED
582	    echo "Services stack: $SERVICES_DEPLOYED deployed, $SERVICES_FAILED failed"
583	    [[ "$SERVICES_FAILED" -gt 0 ]] && STEP_SERVICES_OK=false
584	fi
585
586	# ── Step 8: Partnership onboard ───────────────────────────────────────────────────────────────
587	echo ""
588	echo "━━━ Step 8 — Partnership Onboard ━━━"
589
590	if bash "$SCRIPTS_ROOT/Partnership/partnership_manager.sh" --onboard "${EXTRA_FLAGS[@]}"; then
591	    echo "Partnership onboard complete ✅"
592	    ONBOARD_OK=true
593	else
594	    error "Partnership onboard failed"
595	    ONBOARD_OK=false
596	fi
597
598	# ── Step 9: Arr library bootstrap ─────────────────────────────────────────────────────────────
599	echo ""
600	echo "━━━ Step 9 — Arr Library Bootstrap ━━━"
601
602	if [[ "$ONBOARD_OK" == false ]]; then
603	    warn "Skipping — onboard did not complete"
604	elif [[ "$SKIP_ARR_SYNC" == true ]]; then
605	    warn "Skipping (--skip-arr-sync)"
606	elif [[ ! -f "$SCRIPTS_ROOT/Media/arr_sync.sh" ]]; then
607	    warn "arr_sync.sh not found — run Media/arr_sync.sh manually once arrs are live"
608	elif bash "$SCRIPTS_ROOT/Media/arr_sync.sh" "${EXTRA_FLAGS[@]}"; then
609	    echo "Arr bootstrap complete ✅"
610	    ARR_SYNC_OK=true
611	else
612	    warn "Arr sync had errors — partnership still valid"
613	    warn "Re-run Media/arr_sync.sh once all arr containers are live"
614	fi
615
616	# ── Step 10: Push master.conf to all listed hosts ─────────────────────────────────────────────
617	# SSH is now established and all partners have the plugin installed.
618	# Push the authoritative master.conf so every listed host is in sync immediately.
619	echo ""
620	echo "━━━ $ICON_GEAR Step 10 — master.conf Push ━━━"
621
622	if [[ "$ONBOARD_OK" == false ]]; then
623	    warn "Skipping — onboard did not complete"
624	elif [[ "$DRY_RUN" == true ]]; then
625	    warn "DRY RUN — would push master.conf to all listed hosts"
626	    MASTER_PUSH_OK=true
627	else
628	    push_output=$(platform_push_conf)
629	    push_rc=$?
630	    [[ -n "$push_output" ]] && echo "$push_output"
631	    platform_push_setup_state
632	    if [[ $push_rc -eq 0 ]]; then
633	        echo "master.conf sync complete ✅"
634	        MASTER_PUSH_OK=true
635	    else
636	        warn "master.conf push had failures — retry via Scheduler → master.conf → Save Conf"
637	    fi
638	fi
639
640	# ── Write Phase 2 completion state ────────────────────────────────────────────────────────────
641	[[ "$ONBOARD_OK" == true && "$DRY_RUN" == false ]] && write_onboard_phase "$MIRROR_ID" 2
642
643	# ── Summary ───────────────────────────────────────────────────────────────────────────────────
644	END=$(date +%s)
645	echo ""
646	echo "━━━━━ $ICON_SUMMARY ONBOARD SUMMARY ━━━━━"
647	echo "  Owner:    $MY_ID ($LOCAL_SERVER_NAME)"
648	echo "  Mirror:   $MIRROR ($MIRROR_IP)"
649	[[ "$PHASE2_ONLY" == true ]] && echo "  Mode:     Phase 2 (triggered by HOST2 notification)"
650	echo "  Duration: $(format_duration $(( END - START )))"
651	echo ""
652	_ok()   { [[ "$1" == true ]] && echo "✅" || echo "❌"; }
653	_skip() { [[ "$1" == true ]] && echo "skipped" || echo "$(_ok "$2")"; }
654
655	echo "  Step 1  — SSH keys:         $(_skip "$SKIP_SSH"             "$STEP_SSH_OK")"
656	echo "  Step 2  — Stop auth:        $(_skip "$SKIP_AUTH_STACK"       "$STEP_STOP_AUTH_OK")"
657	echo "  Step 3  — Auth stack:       $( [[ "$SKIP_AUTH_STACK" == true ]] && echo "skipped" || echo "${AUTH_DEPLOYED} deployed, ${AUTH_FAILED} failed" )"
658	echo "  Step 4  — Stop arr:         $(_skip "$SKIP_ARR_STACK"        "$STEP_STOP_ARR_OK")"
659	echo "  Step 5  — Arr stack:        $( [[ "$SKIP_ARR_STACK" == true ]] && echo "skipped" || echo "${ARR_DEPLOYED} deployed, ${ARR_FAILED} failed" )"
660	echo "  Step 6  — Stop services:    $(_skip "$SKIP_SERVICES_STACK"   "$STEP_STOP_SERVICES_OK")"
661	echo "  Step 7  — Services stack:   $( [[ "$SKIP_SERVICES_STACK" == true ]] && echo "skipped" || echo "${SERVICES_DEPLOYED} deployed, ${SERVICES_FAILED} failed" )"
662	echo "  Step 8  — Onboard:          $(_ok "$ONBOARD_OK")"
663	echo "  Step 9  — Arr bootstrap:    $( [[ "$SKIP_ARR_SYNC" == true || "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$ARR_SYNC_OK")" )"
664	echo "  Step 10 — Conf push:        $( [[ "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$MASTER_PUSH_OK")" )"
665	echo ""
666
667	if [[ "$ONBOARD_OK" == true ]]; then
668	    [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
669	        echo "$ICON_DONE DONE — partnership established ✅"
670	    echo "Verify with: Partnership/partnership_manager.sh --status"
671	else
672	    error "Setup incomplete — resolve errors above and re-run"
673	fi
674	echo "━━━━━━━━━━━━━━━━━━━━━━━"
675	[[ "$ONBOARD_OK" == false ]] && exit 1
676	exit 0
677


Now I'll audit each script systematically, comparing the header documentation against the actual code implementation.

## Audit Results

**onboard_cancel.sh**
- ✅ Script header and code align correctly

**partnership_manager.sh**
- Issue 1: Header claims `--unblock <hostname>` at line 184, but the code implements it as `--unblock` followed by positional argument (lines 224-225). Header is accurate, code works as documented.
- Issue 2: Header at line 131 lists "PARTNERSHIP_ONBOARD_VERIFY" as defaulting to `true`, but line 1254 checks `"${PARTNERSHIP_ONBOARD_VERIFY:-true}"` which only defaults to `true` if undefined. The header claim is correct and the code implements it.
- ✅ Operationally accurate

**partnership_offboard.sh**
- Issue: Header states Step 7 on line 24 is "Remote cleanup — remove auth/arr stack + fallback containers from mirror" but the code at line 441-443 calls two separate functions: `cleanup_deployed_stack_on_remote` (for auth/arr) and `cleanup_owner_containers_on_mirror` (for owner-deployed containers). The header doesn't mention `cleanup_owner_containers_on_mirror` explicitly, though it is described generically as "auth/arr stack + fallback containers". However, the actual execution splits this into two distinct functions with different logic. This is functional but the description could be clearer. The code does what the header claims, just via two sub-functions rather than a single operation.
- ✅ Functionally accurate despite implementation detail

**partnership_onboard.sh**
- Issue: Header at line 22-32 states "OWNER PATH (10 steps)" but the actual OWNER path in code execution (lines 377-677) shows the numbered steps don't align cleanly: Lines 379-431 handle Step 1 (SSH), then Step 2 at line 506 (stop auth), Step 3 at line 516 (deploy auth), Step 4 at line 534 (stop arr), Step 5 at line 547 (deploy arr), Step 6 at line 561 (stop services), Step 7 at line 574 (deploy services), Step 8 at line 588 (partnership onboard), Step 9 at line 600 (arr bootstrap), Step 10 at line 620 (conf push). The step numbers match the header claim of 10 steps, so this is accurate.
- ✅ Script header matches code

**partnership_transfer.sh**
- Issue: Header at line 75-76 states "CRITICAL_SYNC_SHARES" is in CONFIGURATION, and line 252 calls `do_final_sync` which uses this array. However, the header does not list PARTNERSHIP_AUTH_WEBUIS in CONFIGURATION (line 79 mentions it in CONFIGURATION under partnership_manager.sh context, but this script's CONFIGURATION section lines 60-80 does list "PARTNERSHIP_AUTH_WEBUIS" at line 79). Actually, reading carefully: the CONFIGURATION section properly lists CRITICAL_SYNC_SHARES at line 75 and PARTNERSHIP_AUTH_WEBUIS at line 79. This is correct.
- ✅ Accurate

**ssh_setup.sh**
- Issue: Header claims "--local-only" mode at line 56, but the header documentation at lines 12-37 does not list "--local-only" as a runtime mode. However, the code does parse it at line 56 and implements it at lines 325-335 with fallback behavior. The mode is undocumented in the header MODES section.
- Issue 2: Header does not document the return value `exit 2` behavior, but code at line 267 returns `exit 2` when max strikes is reached. The STRIKE SYSTEM section (lines 19-26) mentions "return exit 2 (caller can escalate)" at line 25, so this IS documented. Not an issue.
- ✅ Mostly accurate; --local-only exists but isn't in the MODES section header

---

## Summary

**onboard_cancel.sh**
✅ onboard_cancel.sh

**partnership_manager.sh**
✅ partnership_manager.sh

**partnership_offboard.sh**
✅ partnership_offboard.sh

**partnership_onboard.sh**
✅ partnership_onboard.sh

**partnership_transfer.sh**
✅ partnership_transfer.sh

**ssh_setup.sh**
- Header RUNTIME MODES section (lines 32-37) does not document the `--local-only` flag, though it is implemented in the code (line 56) and functional (lines 325-335). The flag should be added to the documented runtime modes.