Audit every bash script in these directories against its own header documentation. For each script, read the full file and check:
1. Does PURPOSE match what the code actually does?
2. Does CONFIGURATION list all config vars the script actually uses (missing vars, or vars listed but never referenced)?
3. Does RUNTIME MODES / flags match what's actually parsed (missing flags, or flags documented but not implemented)?
4. Are there OPERATIONAL SAFEGUARDS documented that aren't in the code?
5. Anything the code does that the header doesn't mention?

Scripts to audit:
- /boot/config/plugins/varaverk/Fallback/fallback.sh
- /boot/config/plugins/varaverk/Fallback/fallback_test.sh
- /boot/config/plugins/varaverk/Kernel/decision_engine.sh
- /boot/config/plugins/varaverk/Rsync/rsync.sh
- /boot/config/plugins/varaverk/load_config.sh
- /boot/config/plugins/varaverk/common.sh
- /boot/config/plugins/varaverk/git_pull_execute.sh

Report ONLY real discrepancies — not style opinions. Format: filename, then a short bullet list of issues. If a script's header matches its code exactly, just write "✅ filename". Be specific: quote the header claim and what the code actually does differently.


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.
- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch

update-config keybindings-help verify code-review simplify fewer-permission-prompts loop schedule claude-api run init review security-review


I'll audit each bash script against its header documentation. Let me start by reading all the scripts.


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Fallback Test ==============================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Controlled simulation of the fallback lifecycle. Validates the entire sequence
9	# without waiting for a real outage. Contains no fallback logic — exercises the
10	# real fallback.sh via an iptables DROP rule on the remote Tailscale IP.
11	#
12	# Run during a maintenance window. Users will experience a brief service
13	# interruption. Use --dry-run to walk through all phases without real changes.
14	#
15	# ==============================================================================================
16	# OPERATIONAL MODEL
17	# ==============================================================================================
18	#
19	# Phase 1 — Pre-flight         Both servers reachable, Docker daemons healthy,
20	#                               version parity, fallback.sh exists, state NORMAL
21	# Phase 2 — Block Remote        iptables DROP rule added — remote appears unreachable
22	# Phase 3 — Fallback Detection  Wait FALLBACK_TEST_BLOCK_WAIT for fallback.sh to
23	#                               detect the outage and enter FALLBACK state
24	# Phase 4 — Container Start     Verify Tier 1 containers started locally
25	# Phase 5 — Restore             iptables rule removed — remote reachable again
26	# Phase 6 — Handback            Wait FALLBACK_TEST_HANDBACK_WAIT for fallback.sh to
27	#                               complete full handback and return to NORMAL
28	# Phase 7 — Container Handback  Verify Tier 1 containers stopped locally
29	# Report  — Full pass/fail per phase with timing
30	#
31	# ==============================================================================================
32	# DESIGN PRINCIPLES
33	# ==============================================================================================
34	#
35	# Test Harness Only
36	#   Contains zero fallback logic. All fallback is exercised through fallback.sh.
37	#   Any change to fallback.sh is automatically reflected in the test result.
38	#
39	# ==============================================================================================
40	# OPERATIONAL SAFEGUARDS
41	# ==============================================================================================
42	#
43	# iptables Safety Trap
44	#   The DROP rule is removed via trap on ANY exit — normal completion, crash, error,
45	#   ctrl-c. Remote connectivity is always restored regardless of test outcome.
46	#   You cannot accidentally leave the remote permanently blocked.
47	#
48	# FALLBACK_ENABLED Gate
49	#   Aborts if FALLBACK_ENABLED=false. Testing a disabled fallback system is
50	#   misleading and potentially destructive.
51	#
52	# State Must Be NORMAL
53	#   Pre-flight fails if state is not NORMAL. Running a test during an actual
54	#   fallback event would interfere with the real event.
55	#
56	# Version Parity Check
57	#   Pre-flight verifies unRAID version parity before any iptables rules are
58	#   added. A mismatch makes the test result unreliable.
59	#
60	# Remote Docker Daemon Check
61	#   Pre-flight confirms remote Docker daemon is responsive before Phase 2.
62	#
63	# Lock Acquisition
64	#   acquire_lock() prevents concurrent test runs. Running two tests simultaneously
65	#   would produce conflicting iptables rules and unreliable results.
66	#
67	# Host Detection
68	#   detect_hosts() resolves MY_ID / REMOTE_ID from master.conf at startup.
69	#   Exits if the host cannot be identified — prevents testing on an unknown machine.
70	#
71	# ==============================================================================================
72	# CONFIGURATION
73	# ==============================================================================================
74	#
75	# master.conf
76	#
77	#   FALLBACK_TEST_BLOCK_WAIT
78	#       Seconds to wait in Phase 3 for fallback.sh to detect the outage.
79	#       Must be > FALLBACK_CHECK_INTERVAL + buffer. At 30s interval: use ≥60s.
80	#       (default: 60)
81	#
82	#   FALLBACK_TEST_HANDBACK_WAIT
83	#       Seconds to wait in Phase 6 for fallback.sh to complete handback.
84	#       Must cover: FALLBACK_HANDBACK_STRIKES × FALLBACK_CHECK_INTERVAL + rsync
85	#       duration + container start time. At 3 strikes × 30s + ~2min rsync +
86	#       ~1min container start: use ≥240s. (default: 300)
87	#
88	# ==============================================================================================
89	# RUNTIME MODES
90	# ==============================================================================================
91	#
92	# fallback_test.sh --dry-run
93	#     Walk through all 7 phases with output but no iptables changes and no
94	#     container starts/stops. ALWAYS run this before a live test.
95	#
96	# fallback_test.sh
97	#     Full live test — real iptables DROP rule, real container lifecycle.
98	#     Users will experience a brief service interruption. Run during a
99	#     maintenance window.
100	#
101	# fallback_test.sh --status
102	#     Show current fallback state and test timing configuration. No test run.
103	#
104	# fallback_test.sh --log
105	#     Verbose output on every check in every phase.
106	#
107	# ==============================================================================================
108
109	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
110
111	source "$SCRIPT_DIR/../load_config.sh"
112
113	parse_args "$@"
114
115	FALLBACK_SCRIPT="$SCRIPT_DIR/fallback.sh"
116	DOCKER_TIMEOUT=15
117
118	# ==============================================================================================
119	# ── SAFETY TRAP — always remove iptables rule on exit ─────────────────────────────────────────
120	# ==============================================================================================
121	# Fires on normal exit, error exit, ctrl-c, and script crashes.
122	# Remote connectivity is ALWAYS restored regardless of test outcome.
123
124	IPTABLES_RULE_ACTIVE=false
125
126	cleanup() {
127	    if [[ "$IPTABLES_RULE_ACTIVE" == true ]]; then
128	        echo ""
129	        warn "$ICON_SHIELD Cleanup — removing iptables block on $REMOTE_SERVER..."
130	        if [[ "$DRY_RUN" == false ]]; then
131	            iptables -D OUTPUT -d "$REMOTE_SERVER" -j DROP 2>/dev/null
132	            IPTABLES_RULE_ACTIVE=false
133	            warn "iptables rule removed — remote connectivity restored"
134	        else
135	            warn "DRY RUN — would remove iptables rule"
136	        fi
137	    fi
138	}
139
140	trap cleanup EXIT
141
142	# ==============================================================================================
143	# ━━━ Setup ━━━
144	# ==============================================================================================
145
146	if [[ "$EUID" -ne 0 ]]; then
147	    error "Must be run as root"
148	    exit 1
149	fi
150
151	# FALLBACK_ENABLED gate — no point testing if fallback is disabled
152	if [[ "${FALLBACK_ENABLED:-false}" == false ]]; then
153	    warn "FALLBACK_ENABLED=false — fallback test aborted"
154	    warn "Enable fallback in master.conf before running this test"
155	    exit 0
156	fi
157
158	acquire_lock   # strict single instance — modifies iptables and containers
159
160	if ! command -v docker &>/dev/null; then
161	    error "Docker command not found"
162	    exit 1
163	fi
164
165	detect_hosts
166	resolve_remote_ip
167
168	# Validate commands used by this script
169	platform_require_cmd \
170	    "$(which iptables 2>/dev/null || echo /sbin/iptables)" \
171	    "--version" "iptables" \
172	    "iptables" || { error "iptables not found — required for connectivity simulation"; exit 1; }
173
174
175	if [[ ! -f "$FALLBACK_SCRIPT" ]]; then
176	    error "fallback.sh not found at $FALLBACK_SCRIPT"
177	    exit 1
178	fi
179	log "fallback.sh found at $FALLBACK_SCRIPT"
180	log "$ICON_GEAR Config: remote=${REMOTE_SERVER_NAME} (${REMOTE_SERVER}) fallback-script=${FALLBACK_SCRIPT}"
181
182	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no iptables rules or container changes will be made"
183
184	# ==============================================================================================
185	# ━━━ Status ━━━
186	# ==============================================================================================
187	if [[ "$SHOW_STATUS" == true ]]; then
188	    local_ver=$(platform_get_os_version 2>/dev/null || echo "unknown")
189
190	    echo ""
191	    echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
192	    echo "$ICON_HOST      My ID:         $MY_ID ($LOCAL_SERVER_NAME)"
193	    echo "$ICON_HOST      Remote ID:     $REMOTE_ID ($REMOTE_SERVER_NAME — $REMOTE_SERVER)"
194	    echo "$ICON_GEAR      OS ver:        $local_ver"
195	    echo "$ICON_FALLBACK  Block wait:    ${FALLBACK_TEST_BLOCK_WAIT}s"
196	    echo "$ICON_FALLBACK  Handback wait: ${FALLBACK_TEST_HANDBACK_WAIT}s"
197	    echo "$ICON_FALLBACK  Check interval: ${FALLBACK_CHECK_INTERVAL}s"
198	    echo "$ICON_FALLBACK  Handback strikes: ${FALLBACK_HANDBACK_STRIKES}"
199	    echo "$ICON_GEAR      Dry Run:       $DRY_RUN"
200
201	    if [[ -f "$FALLBACK_STATE_FILE" ]]; then
202	        CURRENT_STATE=$(grep "^state=" "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
203	        echo "$ICON_FALLBACK  Current state: ${CURRENT_STATE:-unknown}"
204	    else
205	        echo "$ICON_FALLBACK  Current state: no state file"
206	    fi
207
208	    # Show Tier 1 containers for this host
209	    TIER1_VAR="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1"
210	    eval "TIER1_CONTAINERS=(\"\${${TIER1_VAR}[@]:-}\")"
211	    echo "$ICON_CONTAINERS Tier 1 to test: ${TIER1_CONTAINERS[*]:-none configured}"
212	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
213	    exit 0
214	fi
215
216	# ==============================================================================================
217	# ── PHASE TRACKING ────────────────────────────────────────────────────────────────────────────
218	# ==============================================================================================
219	PHASES_PASS=()
220	PHASES_FAIL=()
221	TOTAL_START=$(date +%s)
222
223	phase_pass() { PHASES_PASS+=("$1"); warn "$ICON_DONE  Phase: $1 — PASSED ✅"; }
224	phase_fail() { PHASES_FAIL+=("$1"); error "Phase: $1 — FAILED ❌"; }
225
226	# Get Tier 1 containers for this server's fallback responsibility
227	TIER1_VAR="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1"
228	eval "TIER1_CONTAINERS=(\"\${${TIER1_VAR}[@]:-}\")"
229
230	# ==============================================================================================
231	# ━━━ Phase 1 — Pre-flight ━━━
232	# ==============================================================================================
233	echo ""
234	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
235	echo "  $ICON_SHIELD FALLBACK TEST — $(date '+%Y-%m-%d %H:%M:%S')"
236	echo "  $ICON_HOST $MY_ID ($LOCAL_SERVER_NAME) → $REMOTE_ID ($REMOTE_SERVER_NAME)"
237	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
238	echo ""
239	echo "━━━ $ICON_SHIELD Phase 1 — Pre-flight ━━━"
240
241	# Remote reachable
242	if ping_remote; then
243	    log "$REMOTE_SERVER_NAME is reachable"
244	else
245	    error "$REMOTE_SERVER_NAME is not reachable — cannot run test"
246	    phase_fail "Pre-flight"
247	    exit 1
248	fi
249
250	# Internet reachable
251	if ping_internet; then
252	    log "Internet is reachable"
253	else
254	    error "No internet connectivity — cannot run test"
255	    phase_fail "Pre-flight"
256	    exit 1
257	fi
258
259	# Version parity — test may produce misleading results on mismatch
260	if ! check_os_version_parity; then
261	    error "unRAID version mismatch — test aborted to prevent misleading results"
262	    phase_fail "Pre-flight"
263	    exit 1
264	fi
265
266	# Remote Docker daemon — must be responsive before test manipulates containers
267	if ! check_remote_docker_daemon; then
268	    error "Remote Docker daemon not responsive — cannot run test"
269	    phase_fail "Pre-flight"
270	    exit 1
271	fi
272
273	# Fallback state must be NORMAL before test
274	if [[ -f "$FALLBACK_STATE_FILE" ]]; then
275	    CURRENT_STATE=$(grep "^state=" "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
276	    if [[ "$CURRENT_STATE" != "NORMAL" ]]; then
277	        error "Fallback state is $CURRENT_STATE — must be NORMAL before running test"
278	        phase_fail "Pre-flight"
279	        exit 1
280	    fi
281	    log "Fallback state is NORMAL"
282	else
283	    warn "No state file found — assuming NORMAL (first run)"
284	fi
285
286	# Tier 1 containers configured
287	if [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then
288	    error "No Tier 1 containers configured for $MY_ID → $REMOTE_ID"
289	    error "Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in host*.conf"
290	    phase_fail "Pre-flight"
291	    exit 1
292	fi
293	log "Tier 1 containers: ${TIER1_CONTAINERS[*]}"
294
295	phase_pass "Pre-flight"
296
297	# ==============================================================================================
298	# ━━━ Phase 2 — Block Remote Connectivity ━━━
299	# ==============================================================================================
300	echo ""
301	echo "━━━ $ICON_PING Phase 2 — Block Remote Connectivity ━━━"
302	warn "Adding iptables rule — dropping all traffic to $REMOTE_SERVER ($REMOTE_SERVER_NAME)"
303
304	if [[ "$DRY_RUN" == false ]]; then
305	    iptables -I OUTPUT -d "$REMOTE_SERVER" -j DROP
306	    IPTABLES_RULE_ACTIVE=true
307
308	    # Verify block is working
309	    sleep 2
310	    if ! ping -c1 -W2 "$REMOTE_SERVER" &>/dev/null; then
311	        log "Connectivity block confirmed — ping to remote fails as expected"
312	        phase_pass "Block Remote"
313	    else
314	        error "iptables rule did not block connectivity — ping still succeeds"
315	        phase_fail "Block Remote"
316	        exit 1
317	    fi
318	else
319	    warn "DRY RUN — would block $REMOTE_SERVER with iptables DROP rule"
320	    phase_pass "Block Remote"
321	fi
322
323	# ==============================================================================================
324	# ━━━ Phase 3 — Fallback Detection ━━━
325	# ==============================================================================================
326	echo ""
327	echo "━━━ $ICON_FALLBACK Phase 3 — Fallback Detection ━━━"
328	warn "Waiting ${FALLBACK_TEST_BLOCK_WAIT}s for fallback.sh to detect outage..."
329	log "fallback.sh check interval: ${FALLBACK_CHECK_INTERVAL}s"
330
331	if [[ "$DRY_RUN" == false ]]; then
332	    sleep "$FALLBACK_TEST_BLOCK_WAIT"
333
334	    if [[ -f "$FALLBACK_STATE_FILE" ]]; then
335	        NEW_STATE=$(grep "^state=" "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
336	        if [[ "$NEW_STATE" == "FALLBACK" ]]; then
337	            log "State changed to FALLBACK — outage detected correctly ✅"
338	            phase_pass "Fallback Detection"
339	        else
340	            error "State is $NEW_STATE — expected FALLBACK after ${FALLBACK_TEST_BLOCK_WAIT}s"
341	            warn "Is fallback.sh running? Check User Scripts plugin"
342	            phase_fail "Fallback Detection"
343	        fi
344	    else
345	        error "No state file found after wait — fallback.sh may not be running"
346	        phase_fail "Fallback Detection"
347	    fi
348	else
349	    warn "DRY RUN — would wait ${FALLBACK_TEST_BLOCK_WAIT}s then check for FALLBACK state"
350	    phase_pass "Fallback Detection"
351	fi
352
353	# ==============================================================================================
354	# ━━━ Phase 4 — Container Start Verification ━━━
355	# ==============================================================================================
356	echo ""
357	echo "━━━ $ICON_CONTAINERS Phase 4 — Tier 1 Containers Started Locally ━━━"
358	log "Checking Tier 1 containers: ${TIER1_CONTAINERS[*]}"
359
360	if [[ "$DRY_RUN" == false ]]; then
361	    CONTAINERS_OK=true
362	    for container in "${TIER1_CONTAINERS[@]}"; do
363	        [[ -z "$container" ]] && continue
364	        STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' \
365	            "$container" 2>/dev/null)
366	        if [[ "$STATUS" == "true" ]]; then
367	            log "$ICON_RUNNING $container is running locally ✅"
368	        else
369	            error "$ICON_NOT_RUNNING $container is NOT running locally"
370	            CONTAINERS_OK=false
371	        fi
372	    done
373
374	    if [[ "$CONTAINERS_OK" == true ]]; then
375	        phase_pass "Container Start"
376	    else
377	        phase_fail "Container Start"
378	    fi
379	else
380	    warn "DRY RUN — would verify these Tier 1 containers started: ${TIER1_CONTAINERS[*]}"
381	    phase_pass "Container Start"
382	fi
383
384	# ==============================================================================================
385	# ━━━ Phase 5 — Restore Remote Connectivity ━━━
386	# ==============================================================================================
387	echo ""
388	echo "━━━ $ICON_PING Phase 5 — Restore Remote Connectivity ━━━"
389	warn "Removing iptables block — $REMOTE_SERVER_NAME becomes reachable again"
390
391	if [[ "$DRY_RUN" == false ]]; then
392	    iptables -D OUTPUT -d "$REMOTE_SERVER" -j DROP 2>/dev/null
393	    IPTABLES_RULE_ACTIVE=false
394
395	    sleep 3
396	    if ping_remote; then
397	        log "$REMOTE_SERVER_NAME is reachable again ✅"
398	        phase_pass "Restore Connectivity"
399	    else
400	        error "$REMOTE_SERVER_NAME still unreachable after removing iptables rule"
401	        phase_fail "Restore Connectivity"
402	    fi
403	else
404	    warn "DRY RUN — would remove iptables rule"
405	    phase_pass "Restore Connectivity"
406	fi
407
408	# ==============================================================================================
409	# ━━━ Phase 6 — Handback ━━━
410	# ==============================================================================================
411	echo ""
412	echo "━━━ $ICON_FALLBACK Phase 6 — Handback ━━━"
413	warn "Waiting ${FALLBACK_TEST_HANDBACK_WAIT}s for fallback.sh to complete handback..."
414	log "Requires $FALLBACK_HANDBACK_STRIKES consecutive checks at ${FALLBACK_CHECK_INTERVAL}s"
415	log "Minimum handback time: $(( FALLBACK_HANDBACK_STRIKES * FALLBACK_CHECK_INTERVAL ))s"
416
417	if [[ "$DRY_RUN" == false ]]; then
418	    sleep "$FALLBACK_TEST_HANDBACK_WAIT"
419
420	    if [[ -f "$FALLBACK_STATE_FILE" ]]; then
421	        FINAL_STATE=$(grep "^state=" "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
422	        if [[ "$FINAL_STATE" == "NORMAL" ]]; then
423	            log "State returned to NORMAL — handback completed ✅"
424	            phase_pass "Handback"
425	        else
426	            error "State is $FINAL_STATE — expected NORMAL after ${FALLBACK_TEST_HANDBACK_WAIT}s"
427	            warn "Handback may still be in progress — check fallback.sh output"
428	            phase_fail "Handback"
429	        fi
430	    else
431	        error "No state file found"
432	        phase_fail "Handback"
433	    fi
434	else
435	    warn "DRY RUN — would wait ${FALLBACK_TEST_HANDBACK_WAIT}s then verify NORMAL state"
436	    phase_pass "Handback"
437	fi
438
439	# ==============================================================================================
440	# ━━━ Phase 7 — Container Handback Verification ━━━
441	# ==============================================================================================
442	echo ""
443	echo "━━━ $ICON_CONTAINERS Phase 7 — Tier 1 Containers Stopped Locally ━━━"
444	log "Verifying Tier 1 containers returned to $REMOTE_SERVER_NAME"
445
446	if [[ "$DRY_RUN" == false ]]; then
447	    HANDBACK_OK=true
448	    for container in "${TIER1_CONTAINERS[@]}"; do
449	        [[ -z "$container" ]] && continue
450	        STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' \
451	            "$container" 2>/dev/null)
452	        if [[ "$STATUS" != "true" ]]; then
453	            log "$ICON_NOT_RUNNING $container stopped locally — handed back ✅"
454	        else
455	            error "$ICON_RUNNING $container still running locally — handback may have failed"
456	            HANDBACK_OK=false
457	        fi
458	    done
459
460	    if [[ "$HANDBACK_OK" == true ]]; then
461	        phase_pass "Container Handback"
462	    else
463	        phase_fail "Container Handback"
464	    fi
465	else
466	    warn "DRY RUN — would verify Tier 1 containers stopped locally after handback"
467	    phase_pass "Container Handback"
468	fi
469
470	TOTAL_END=$(date +%s)
471
472	# ==============================================================================================
473	# ━━━ Test Report ━━━
474	# ==============================================================================================
475	echo ""
476	echo "━━━━━ $ICON_SUMMARY FALLBACK TEST REPORT ━━━━━"
477	echo "$ICON_HOST    My ID:    $MY_ID ($LOCAL_SERVER_NAME)"
478	echo "$ICON_HOST    Remote:   $REMOTE_ID ($REMOTE_SERVER_NAME)"
479	echo "$ICON_TIME    Duration: $(format_duration $((TOTAL_END - TOTAL_START)))"
480	echo ""
481	echo "  Phase Results:"
482	for phase in "${PHASES_PASS[@]}"; do
483	    echo "    $ICON_SUCCESS $phase"
484	done
485	for phase in "${PHASES_FAIL[@]}"; do
486	    echo "    $ICON_ERROR $phase"
487	done
488	echo ""
489
490	PASS_COUNT=${#PHASES_PASS[@]}
491	FAIL_COUNT=${#PHASES_FAIL[@]}
492	TOTAL_PHASES=$(( PASS_COUNT + FAIL_COUNT ))
493
494	if [[ "$DRY_RUN" == true ]]; then
495	    warn "DRY RUN — no changes made"
496	elif [[ "$FAIL_COUNT" -eq 0 ]]; then
497	    warn "$ICON_DONE ALL $TOTAL_PHASES PHASES PASSED"
498	    notify "Fallback test PASSED on $(hostname) — all $TOTAL_PHASES phases completed" \
499	        "Fallback Test" "normal"
500	else
501	    error "$FAIL_COUNT/$TOTAL_PHASES PHASES FAILED"
502	    notify "Fallback test FAILED on $(hostname) — $FAIL_COUNT/$TOTAL_PHASES phases failed: ${PHASES_FAIL[*]}" \
503	        "Fallback Test" "warning"
504	fi
505	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
506
507	[[ "$FAIL_COUNT" -gt 0 ]] && exit 1
508	exit 0


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Decision Engine ============================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Behavior-driven scoring kernel for media automation. Sourced by consumer
9	# scripts — not run directly. Evaluates candidates, applies weighted scoring,
10	# temporal decay, and deduplication, then returns a verdict.
11	#
12	# Does NOT download media, search indexers, or manage applications. Has no
13	# side effects — no file writes, no API calls, no deletions.
14	#
15	# Currently paired with: Media/playback_aware_lidarr_discovery.sh
16	#
17	# ==============================================================================================
18	# DESIGN PRINCIPLES
19	# ==============================================================================================
20	#
21	# Domain-Agnostic Core
22	#   The engine itself has no knowledge of music vs TV vs movies. Consumers
23	#   define thresholds, weights, and strictness profiles — the engine just
24	#   scores and decides. This prevents cross-domain bias pollution: music
25	#   strictness cannot contaminate TV intake logic.
26	#
27	#   Lidarr consumers  → highly selective, quality-first discovery
28	#   Sonarr consumers  → balanced, family-aware episodic intake
29	#   Radarr consumers  → broader flexibility with intelligent filtering
30	#
31	# Temporal Decay
32	#   Old behavioral signals lose influence over time (one decay unit per 30
33	#   days). Prevents permanent genre lock-in, historical bias accumulation,
34	#   and dead-user score dominance.
35	#
36	# Consumer Owns the Decision
37	#   The engine returns ACCEPT/REJECT/SCORE. What happens next is entirely
38	#   the consumer's concern — the engine never acts on its own verdict.
39	#
40	# ==============================================================================================
41	# FUNCTIONS
42	# ==============================================================================================
43	#
44	#   score_candidate user_score popularity_score recency_score quality_score
45	#       Returns TOTAL_SCORE (integer sum). Consumer defines actual weight values.
46	#
47	#   evaluate_threshold score minimum
48	#       Returns 0 (pass) or 1 (fail). Used as: if evaluate_threshold ...
49	#
50	#   apply_temporal_decay score age_days
51	#       Returns adjusted score. Subtracts (age_days / 30), floor at 0.
52	#
53	#   is_duplicate_candidate candidate history_file
54	#       Returns 0 (duplicate found in history_file) or 1 (not found).
55	#
56	#   make_decision score threshold
57	#       Returns "ACCEPT" or "REJECT". Calls evaluate_threshold internally.
58	#
59	# ==============================================================================================
60
61	# ── score_candidate ───────────────────────────────────────────────────────────
62	score_candidate() {
63
64	    local user_score="${1:-0}"
65	    local popularity_score="${2:-0}"
66	    local recency_score="${3:-0}"
67	    local quality_score="${4:-0}"
68
69	    TOTAL_SCORE=$(( \
70	        user_score + \
71	        popularity_score + \
72	        recency_score + \
73	        quality_score \
74	    ))
75
76	    echo "$TOTAL_SCORE"
77	}
78
79	# ── evaluate_threshold ───────────────────────────────────────────────────────
80	evaluate_threshold() {
81
82	    local score="$1"
83	    local minimum="$2"
84
85	    if (( score >= minimum )); then
86	        return 0
87	    fi
88
89	    return 1
90	}
91
92	# ── apply_temporal_decay ─────────────────────────────────────────────────────
93	apply_temporal_decay() {
94
95	    local score="$1"
96	    local age_days="$2"
97
98	    local decay=$(( age_days / 30 ))
99
100	    local adjusted=$(( score - decay ))
101
102	    (( adjusted < 0 )) && adjusted=0
103
104	    echo "$adjusted"
105	}
106
107	# ── is_duplicate_candidate ───────────────────────────────────────────────────
108	is_duplicate_candidate() {
109
110	    local candidate="$1"
111	    local history_file="$2"
112
113	    grep -qi "^${candidate}$" "$history_file" 2>/dev/null
114	}
115
116	# ── make_decision ─────────────────────────────────────────────────────────────
117	make_decision() {
118
119	    local score="$1"
120	    local threshold="$2"
121
122	    if evaluate_threshold "$score" "$threshold"; then
123	        echo "ACCEPT"
124	    else
125	        echo "REJECT"
126	    fi
127	}


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Rsync Core Script ==============================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Core rsync engine for the two-server ecosystem. Called per share or per
9	# appdata profile by orchestrators (daily_sync_maintenance, weekly_sync_maintenance,
10	# critical_sync_maintenance) and directly for manual or scheduled dirty syncs.
11	#
12	# Profile is inferred from the directory basename (lowercased). Override with
13	# --profile=name for explicit selection. If no profile matches, global defaults
14	# from master.conf apply and no containers are stopped.
15	#
16	# After each sync, logs transfer data to bandwidth_monitor.sh for the weekly
17	# bandwidth report. Silent on success — only failures produce visible output.
18	#
19	# ==============================================================================================
20	# OPERATIONAL MODEL
21	# ==============================================================================================
22	#
23	# Profiles define per-share behavior:
24	#   PROFILE_CRITICAL_CONTAINER_NAMES  — containers stopped on both servers before sync
25	#   PROFILE_DELAYED_CONTAINERS        — containers with a delay before restart after sync
26	#   PROFILE_CONTAINER_DELAY           — seconds before delayed containers start
27	#   PROFILE_RSYNC_OPTS                — rsync flags (does not inherit DEFAULT_RSYNC_OPTS)
28	#   PROFILE_BW_LIMIT                  — bandwidth limit in KB/s
29	#   PROFILE_RETRY_COUNT               — retry attempts on failure
30	#   PROFILE_SLEEP                     — seconds between retry attempts
31	#   PROFILE_EXCLUDE_DIRS              — paths excluded from transfer
32	#   PROFILE_REMOTE_RESTART_CONTAINERS — containers restarted on remote after dirty sync
33	#                                       Was running → restart. Was stopped → leave stopped.
34	#
35	# Two-tier rsync enable/disable:
36	#   Tier 1: RSYNC_ENABLED=false → all rsync stops immediately (checked by this script)
37	#   Tier 2: per-orchestrator flag (DAILY_RSYNC_ENABLED etc.) → checked by caller
38	#
39	# Bandwidth logging: after each sync, logs profile/duration/status/bytes to
40	# bandwidth_monitor.sh --log-transfer. Bytes captured from rsync --stats via awk
41	# using version-stable field names.
42	#
43	# ==============================================================================================
44	# OPERATIONAL SAFEGUARDS
45	# ==============================================================================================
46	#
47	# Global Rsync Gate
48	#   check_rsync_enabled() — RSYNC_ENABLED=false exits cleanly before any operation.
49	#
50	# Partnership Blocklist
51	#   Refuses to sync if REMOTE_SERVER_NAME appears in the partnership blocklist.
52	#   Written at offboard — prevents stale access after a partnership ends.
53	#
54	# Version Parity
55	#   check_os_version_parity — refuses sync if servers on incompatible unRAID versions.
56	#
57	# Remote Health Pre-flights
58	#   check_connectivity() — Tailscale IP reachable before any SSH
59	#   check_remote_rootfs() — aborts if remote rootfs exceeds ROOTFS_WARN_PCT
60	#   check_remote_share() — aborts if target directory missing or empty on remote
61	#   check_remote_disks() — verifies all backing disks online on remote
62	#
63	# Drive Temperature Check
64	#   check_local_disk_temps() — runs before any transfer. Exit 1 = skip this profile,
65	#   exit 2 = abort all remaining syncs (CRITICAL temperature).
66	#
67	# Remote Docker Daemon Check
68	#   check_remote_docker_daemon — verified before any container stop/start operations.
69	#   If daemon unresponsive: container operations skipped, rsync proceeds without stopping.
70	#
71	# Per-Profile Concurrency Lock
72	#   acquire_rsync_lock() — per-profile lock prevents parallel runs of the same profile.
73	#   Global concurrent limit prevents too many simultaneous rsync processes.
74	#
75	# Notification Validated
76	#   platform_require_cmd confirms the notify script is present before use.
77	#
78	# ==============================================================================================
79	# CONFIGURATION
80	# ==============================================================================================
81	#
82	# master.conf
83	#
84	#   RSYNC_ENABLED
85	#       Global on/off toggle for all rsync operations. (default: true)
86	#
87	#   DEFAULT_RSYNC_OPTS
88	#       Base rsync flags for unproiled shares. Note: --delete is intentionally absent —
89	#       media shares spread files only, arr cleanup scripts own deletions. Profile-specific
90	#       opts set --delete explicitly where needed.
91	#
92	#   BW_LIMIT
93	#       Default bandwidth cap in KB/s when no PROFILE_BW_LIMIT is set. (default: 0 = unlimited)
94	#
95	#   RETRY_COUNT
96	#       Default retry attempts on rsync failure. (default: 3)
97	#
98	#   SLEEP
99	#       Default seconds between retry attempts. (default: 60)
100	#
101	#   ROOTFS_WARN_PCT
102	#       Abort threshold for remote rootfs percentage full. (default: 75)
103	#
104	#   PROFILES["profile_KEY"]
105	#       Profile definitions — one entry per PROFILE_* key per profile name.
106	#       See OPERATIONAL MODEL above for all supported keys.
107	#
108	#   BANDWIDTH_LOG / BANDWIDTH_WARN_GB
109	#       Shared with bandwidth_monitor.sh — set once, used by both.
110	#
111	# ==============================================================================================
112	# RUNTIME MODES
113	# ==============================================================================================
114	#
115	# rsync.sh /path/to/share
116	#     Sync the given path to the remote. Profile inferred from directory basename.
117	#
118	# rsync.sh /path/to/share --profile=name
119	#     Sync with explicit profile override — bypasses basename inference.
120	#
121	# rsync.sh /path/to/share --dry-run
122	#     Run all pre-flight checks and show what rsync would transfer. No transfer,
123	#     no container stops.
124	#
125	# rsync.sh /path/to/share --status
126	#     Show resolved profile, remote identity, and configuration. Then exit.
127	#
128	# rsync.sh /path/to/share --log
129	#     Verbose output throughout — every decision logged.
130	#
131	# ==============================================================================================
132
133	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
134
135	source "$SCRIPT_DIR/../load_config.sh"
136
137	# ── Separate positional directory arg from flags ───────────────────────────────────────────────
138	DIRECTORY=""
139	PROFILE_OVERRIDE=""
140	RAW_ARGS=()
141
142	for ARG in "$@"; do
143	    case "$ARG" in
144	        --profile=*) PROFILE_OVERRIDE="${ARG#--profile=}" ;;
145	        --*|*=*)     RAW_ARGS+=("$ARG") ;;
146	        *)           [[ -z "$DIRECTORY" ]] && DIRECTORY="$ARG" || RAW_ARGS+=("$ARG") ;;
147	    esac
148	done
149
150	parse_args "${RAW_ARGS[@]}"
151
152	[[ -z "$DIRECTORY" ]] && {
153	    error "No directory specified"
154	    error "Usage: rsync.sh <dir> [--dry-run] [--log] [--profile=name]"
155	    exit 1
156	}
157
158	# ==============================================================================================
159	# ━━━ Setup ━━━
160	# ==============================================================================================
161	if [[ "$EUID" -ne 0 ]]; then
162	    error "Must be run as root"
163	    exit 1
164	fi
165
166	acquire_lock
167
168	if ! command -v docker &>/dev/null; then
169	    error "Docker command not found"
170	    exit 1
171	fi
172
173
174	detect_hosts
175
176	# Tier 1 global gate — Tier 2 (per-orchestrator) checked by caller
177	if ! check_rsync_enabled; then
178	    warn "RSYNC_ENABLED=false — exiting cleanly"
179	    exit 0
180	fi
181
182	# Blocklist gate — refuse to sync with a partner blocked after offboard
183	BLOCKLIST_FILE="${PARTNERSHIP_BLOCKLIST_FILE:-${STATE_DIR}/partnership_blocklist.db}"
184	if [[ -f "$BLOCKLIST_FILE" ]] && grep -q "^${REMOTE_SERVER_NAME}|" "$BLOCKLIST_FILE" 2>/dev/null; then
185	    error "Rsync blocked — $REMOTE_SERVER_NAME is on the partnership blocklist"
186	    error "Re-onboard the partnership to restore access: partnership_manager.sh --onboard"
187	    exit 1
188	fi
189
190	resolve_remote_ip
191
192	# ── Profile inference ─────────────────────────────────────────────────────────────────────────
193	if [[ -n "$PROFILE_OVERRIDE" ]]; then
194	    PROFILE_NAME="$PROFILE_OVERRIDE"
195	    log "Profile override: $PROFILE_NAME"
196	else
197	    PROFILE_NAME=$(basename "$DIRECTORY" | tr '[:upper:]' '[:lower:]')
198	    log "Profile inferred: $PROFILE_NAME"
199	fi
200
201	# Acquire per-profile lock and check global concurrent limit
202	acquire_rsync_lock "$PROFILE_NAME"
203
204	# Tee all output to a live log file for the Varaverk UI
205	VV_LIVE_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.log"
206	VV_LAST_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.last.log"
207	: > "$VV_LIVE_LOG"
208	exec 1> >(tee -a "$VV_LIVE_LOG") 2>&1
209
210	# ── Load profile settings ─────────────────────────────────────────────────────────────────────
211	BW_LIMIT=${PROFILE_BW_LIMIT[$PROFILE_NAME]:-$BW_LIMIT}
212	RETRY_COUNT=${PROFILE_RETRY_COUNT[$PROFILE_NAME]:-$RETRY_COUNT}
213	SLEEP=${PROFILE_SLEEP[$PROFILE_NAME]:-$SLEEP}
214	CONTAINER_DELAY=${PROFILE_CONTAINER_DELAY[$PROFILE_NAME]:-$CONTAINER_DELAY}
215
216	read -r -a CRITICAL_CONTAINER_NAMES       <<< "${PROFILE_CRITICAL_CONTAINER_NAMES[$PROFILE_NAME]:-}"
217	read -r -a DELAYED_CONTAINERS             <<< "${PROFILE_DELAYED_CONTAINERS[$PROFILE_NAME]:-}"
218	read -r -a EXCLUDE_DIRS                   <<< "${PROFILE_EXCLUDE_DIRS[$PROFILE_NAME]:-}"
219	read -r -a REMOTE_RESTART_CONTAINERS      <<< "${PROFILE_REMOTE_RESTART_CONTAINERS[$PROFILE_NAME]:-}"
220
221	# Local containers use same names as remote (mirrored naming scheme)
222	LOCAL_CRITICAL_CONTAINER_NAMES=("${CRITICAL_CONTAINER_NAMES[@]}")
223
224	log "$ICON_GEAR Config: profile=${PROFILE_NAME} bw-limit=${BW_LIMIT}KB/s retry=${RETRY_COUNT} sleep=${SLEEP}s container-delay=${CONTAINER_DELAY}s"
225	log "$ICON_GEAR Containers: critical=${CRITICAL_CONTAINER_NAMES[*]:-none} delayed=${DELAYED_CONTAINERS[*]:-none} remote-restart=${REMOTE_RESTART_CONTAINERS[*]:-none}"
226
227	[[ "$SHOW_STATUS" == true ]] && show_status && exit 0
228	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
229
230	# ==============================================================================================
231	# ━━━ Pre-flight Checks ━━━
232	# ==============================================================================================
233	echo ""
234	echo "━━━ $ICON_SHIELD Pre-flight ━━━"
235
236	# Disk temp — before touching remote or moving data
237	# Exit 1 = skip this profile | Exit 2 = abort all remaining profiles
238	check_local_disk_temps
239	TEMP_RESULT=$?
240	if [[ "$TEMP_RESULT" -eq 2 ]]; then
241	    error "Drive temps CRITICAL — aborting all remaining syncs"
242	    exit 2
243	elif [[ "$TEMP_RESULT" -eq 1 ]]; then
244	    warn "Drive temps high — skipping profile [$PROFILE_NAME]"
245	    exit 1
246	else
247	    log "Drive temps OK — $TEMP_CHECK_RESULT"
248	fi
249
250	# Version parity — refuse if servers on incompatible unRAID versions
251	check_os_version_parity || exit 1
252
253	check_connectivity
254	check_remote_rootfs
255	check_remote_share "$DIRECTORY"
256	check_remote_disks "$DIRECTORY"
257
258	# Remote Docker daemon — check before attempting container operations
259	if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]] || [[ ${#REMOTE_RESTART_CONTAINERS[@]} -gt 0 ]]; then
260	    check_remote_docker_daemon || {
261	        warn "Remote Docker daemon unresponsive — skipping container operations"
262	        warn "Proceeding with rsync only — containers will not be stopped or restarted"
263	        CRITICAL_CONTAINER_NAMES=()
264	        LOCAL_CRITICAL_CONTAINER_NAMES=()
265	        REMOTE_RESTART_CONTAINERS=()
266	    }
267	fi
268
269	# ==============================================================================================
270	# ━━━ Stop Containers ━━━
271	# ==============================================================================================
272	if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 || ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]]; then
273	    echo ""
274	    echo "━━━ $ICON_STOP $ICON_CONTAINERS Stop Containers ━━━"
275	    if [[ "$DRY_RUN" == true ]]; then
276	        for c in "${LOCAL_CRITICAL_CONTAINER_NAMES[@]:-}"; do
277	            [[ -n "$c" ]] && warn "DRY RUN — would stop local: $c"
278	        done
279	        for c in "${CRITICAL_CONTAINER_NAMES[@]:-}"; do
280	            [[ -n "$c" ]] && warn "DRY RUN — would stop remote: $c"
281	        done
282	    else
283	        # Local first — flush local databases before pushing
284	        stop_local_containers
285	        # Remote next — prevent writes while receiving
286	        stop_containers
287	    fi
288	fi
289
290	# ==============================================================================================
291	# ━━━ Transfer ━━━
292	# ==============================================================================================
293	echo ""
294	echo "━━━ $ICON_SYNC Transfer ━━━"
295	echo "$ICON_RUN  Source:   $DIRECTORY"
296	echo "$ICON_NET  Remote:   $REMOTE_SERVER:$DIRECTORY"
297	echo "$ICON_GEAR Profile:  $PROFILE_NAME"
298	echo "$ICON_HOST Identity: $MY_ID → $REMOTE_ID"
299	echo ""
300
301	get_rsync_opts
302
303	# Append profile excludes
304	for ex in "${EXCLUDE_DIRS[@]:-}"; do
305	    [[ -n "$ex" ]] && RSYNC_OPTS+=(--exclude="$ex")
306	done
307
308	# Add --stats to capture bytes transferred for bandwidth logging
309	RSYNC_OPTS+=(--stats)
310
311	[[ "$DRY_RUN" == true ]] && RSYNC_OPTS+=("--dry-run")
312
313	START=$(date +%s)
314	RSYNC_SUCCESS=false
315	BYTES_TRANSFERRED=0
316	ATTEMPT=0
317
318	for (( ATTEMPT=1; ATTEMPT<=RETRY_COUNT; ATTEMPT++ )); do
319	    log "$ICON_RETRY Attempt $ATTEMPT of $RETRY_COUNT..."
320	    echo "$ICON_SYNC  Rsync running — this may take a while..."
321
322	    RSYNC_OUTPUT=$(rsync "${RSYNC_OPTS[@]}" \
323	        -e "ssh -i $SSH_KEY -T -o Compression=no -o IPQoS=throughput" \
324	        "$DIRECTORY" "root@${REMOTE_SERVER}:$(dirname "$DIRECTORY")/" 2>&1)
325
326	    RSYNC_EXIT=$?
327
328	    if [[ "$RSYNC_EXIT" -eq 0 ]]; then
329	        # Parse bytes transferred from --stats output
330	        BYTES_TRANSFERRED=$(echo "$RSYNC_OUTPUT" | \
331	            awk '/Total transferred file size:/{gsub(/,/,"",$NF); gsub(/[^0-9]/,"",$NF); print $NF+0}')
332	        BYTES_TRANSFERRED="${BYTES_TRANSFERRED:-0}"
333
334	        log "$ICON_DONE Rsync complete — $BYTES_TRANSFERRED bytes transferred"
335	        RSYNC_SUCCESS=true
336	        break
337	    else
338	        warn "$ICON_RETRY Rsync failed (attempt $ATTEMPT/$RETRY_COUNT)"
339	        log "Exit code: $RSYNC_EXIT"
340	        if [[ "$ATTEMPT" -lt "$RETRY_COUNT" ]]; then
341	            log "Retrying in ${SLEEP}s..."
342	            sleep "$SLEEP"
343	        fi
344	    fi
345	done
346
347	# ==============================================================================================
348	# ━━━ Start Containers ━━━
349	# ==============================================================================================
350	if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 || ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]]; then
351	    echo ""
352	    echo "━━━ $ICON_START $ICON_CONTAINERS Start Containers ━━━"
353	    if [[ "$DRY_RUN" == true ]]; then
354	        for c in "${CRITICAL_CONTAINER_NAMES[@]:-}"; do
355	            [[ -n "$c" ]] && warn "DRY RUN — would start remote: $c"
356	        done
357	        for c in "${LOCAL_CRITICAL_CONTAINER_NAMES[@]:-}"; do
358	            [[ -n "$c" ]] && warn "DRY RUN — would start local: $c"
359	        done
360	    else
361	        # Remote first — can be coming up while local restarts
362	        start_containers
363	        # Local next
364	        start_local_containers
365	    fi
366	fi
367
368	# ==============================================================================================
369	# ━━━ Remote Restart (dirty sync profiles) ━━━
370	# ==============================================================================================
371	# For dirty sync profiles (critical-fallback, emby-fallback) — restart containers on remote
372	# that were running before sync so they pick up config changes from the dirty sync window.
373	# Was running → restart. Was stopped → leave stopped.
374	if [[ ${#REMOTE_RESTART_CONTAINERS[@]} -gt 0 ]]; then
375	    echo ""
376	    echo "━━━ $ICON_START $ICON_CONTAINERS Remote Restart (post dirty sync) ━━━"
377	    log "Restarting configured containers on $REMOTE_SERVER_NAME..."
378
379	    for container in "${REMOTE_RESTART_CONTAINERS[@]}"; do
380	        [[ -z "$container" ]] && continue
381
382	        # Check if container was running before sync (still tracked via RUNNING_CONTAINERS)
383	        WAS_RUNNING=false
384	        for prev in "${RUNNING_CONTAINERS[@]:-}"; do
385	            [[ "$prev" == "$container" ]] && WAS_RUNNING=true && break
386	        done
387
388	        if [[ "$WAS_RUNNING" == false ]]; then
389	            # Not in stop list — check current remote state
390	            REMOTE_STATUS=$(timeout 15 ssh -i "$SSH_KEY" \
391	                -o ConnectTimeout=10 root@"$REMOTE_SERVER" \
392	                "docker inspect -f '{{.State.Running}}' $container 2>/dev/null" 2>/dev/null)
393	            [[ "$REMOTE_STATUS" != "true" ]] && \
394	                log "$container not running on $REMOTE_SERVER_NAME — skipping remote restart" && \
395	                continue
396	        fi
397
398	        if [[ "$DRY_RUN" == true ]]; then
399	            warn "DRY RUN — would restart $container on $REMOTE_SERVER_NAME"
400	            continue
401	        fi
402
403	        timeout 15 ssh -i "$SSH_KEY" -o ConnectTimeout=10 root@"$REMOTE_SERVER" \
404	            "docker restart $container" >/dev/null 2>&1 && \
405	            log "$ICON_STARTED $container restarted on $REMOTE_SERVER_NAME ✅" || \
406	            warn "Failed to restart $container on $REMOTE_SERVER_NAME"
407	    done
408	fi
409
410	END=$(date +%s)
411	DURATION=$(( END - START ))
412
413	# ==============================================================================================
414	# ━━━ Bandwidth Logging ━━━
415	# ==============================================================================================
416	# Logs to bandwidth_monitor.sh — new format includes bytes transferred and warn flag.
417	# Only logs on actual runs (not dry-run) and only when bandwidth_monitor.sh exists.
418	BANDWIDTH_MONITOR="$SCRIPT_DIR/../Monitors/bandwidth_monitor.sh"
419
420	if [[ "$DRY_RUN" == false ]] && [[ -f "$BANDWIDTH_MONITOR" ]]; then
421	    STATUS="success"
422	    [[ "$RSYNC_SUCCESS" == false ]] && STATUS="failed"
423	    bash "$BANDWIDTH_MONITOR" --log-transfer \
424	        "$PROFILE_NAME" "$DURATION" "$STATUS" "$BYTES_TRANSFERRED"
425	    log "$ICON_BANDWIDTH Transfer logged to bandwidth monitor ($BYTES_TRANSFERRED bytes)"
426	fi
427
428	# ==============================================================================================
429	# ━━━ Summary ━━━
430	# ==============================================================================================
431	echo ""
432	echo "━━━━━ $ICON_SUMMARY RSYNC SUMMARY ━━━━━"
433	echo "$ICON_HOST    Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
434	echo "$ICON_RUN     Directory: $DIRECTORY"
435	echo "$ICON_GEAR    Profile:   $PROFILE_NAME"
436	echo "$ICON_TIME    Duration:  $(format_duration $DURATION)"
437	[[ "$BYTES_TRANSFERRED" -gt 0 ]] && \
438	    echo "$ICON_BANDWIDTH Transferred: $(awk "BEGIN {printf \"%.2fGB\", $BYTES_TRANSFERRED / 1073741824}")"
439
440	if [[ "$DRY_RUN" == true ]]; then
441	    warn "DRY RUN — no changes made"
442	elif [[ "$RSYNC_SUCCESS" == true ]]; then
443	    echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
444	else
445	    echo "$ICON_ERROR Status: FAILED after $RETRY_COUNT attempts"
446	    notify "Rsync FAILED — $DIRECTORY ($PROFILE_NAME) after $RETRY_COUNT attempts on $(hostname)" \
447	        "Rsync" "warning"
448	fi
449	echo "━━━━━━━━━━━━━━━━━━━━━━━"
450
451	# Flush tee and preserve log for UI — close both ends of the pipe so tee gets EOF
452	exec 1>&- 2>&-; wait
453	cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null
454	rm -f "$VV_LIVE_LOG"
455
456	[[ "$RSYNC_SUCCESS" == false ]] && [[ "$DRY_RUN" == false ]] && exit 1
457	exit 0