{"parentUuid":null,"isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":"Audit every bash script in these directories against its own header documentation. For each script, read the full file and check:\n1. Does PURPOSE match what the code actually does?\n2. Does CONFIGURATION list all config vars the script actually uses (missing vars, or vars listed but never referenced)?\n3. Does RUNTIME MODES / flags match what's actually parsed (missing flags, or flags documented but not implemented)?\n4. Are there OPERATIONAL SAFEGUARDS documented that aren't in the code?\n5. Anything the code does that the header doesn't mention?\n\nScripts to audit:\n- /boot/config/plugins/varaverk/Fallback/fallback.sh\n- /boot/config/plugins/varaverk/Fallback/fallback_test.sh\n- /boot/config/plugins/varaverk/Kernel/decision_engine.sh\n- /boot/config/plugins/varaverk/Rsync/rsync.sh\n- /boot/config/plugins/varaverk/load_config.sh\n- /boot/config/plugins/varaverk/common.sh\n- /boot/config/plugins/varaverk/git_pull_execute.sh\n\nReport ONLY real discrepancies — not style opinions. Format: filename, then a short bullet list of issues. If a script's header matches its code exactly, just write \"✅ filename\". Be specific: quote the header claim and what the code actually does differently."},"uuid":"bf04c0a8-2441-467c-8abb-9b9b2bc68490","timestamp":"2026-06-14T05:32:07.024Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"bf04c0a8-2441-467c-8abb-9b9b2bc68490","isSidechain":true,"agentId":"ab976a3a2d54c1360","attachment":{"type":"deferred_tools_delta","addedNames":["EnterWorktree","ExitWorktree","Monitor","TaskStop","WebFetch","WebSearch","mcp__claude_ai_Google_Drive__authenticate","mcp__claude_ai_Google_Drive__complete_authentication"],"addedLines":["EnterWorktree","ExitWorktree","Monitor","TaskStop","WebFetch","WebSearch","mcp__claude_ai_Google_Drive__authenticate","mcp__claude_ai_Google_Drive__complete_authentication"],"removedNames":[],"readdedNames":[]},"type":"attachment","uuid":"3ab20442-645d-437f-9327-d4f00798da82","timestamp":"2026-06-14T05:32:07.027Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"3ab20442-645d-437f-9327-d4f00798da82","isSidechain":true,"agentId":"ab976a3a2d54c1360","attachment":{"type":"skill_listing","content":"- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors (\"from now on when X\", \"each time X\", \"whenever X\", \"before/after X\") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions (\"allow X\", \"add permission\", \"move permission to\"), env vars (\"set X=Y\"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: \"allow npm commands\", \"add bq permission to global settings\", \"move permission to user settings\", \"set DEBUG=true\", \"when claude stops show X\". For simple settings like theme/model, suggest the /config command.\n- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: \"rebind ctrl+s\", \"add a chord shortcut\", \"change the submit key\", \"customize keybindings\".\n- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.\n- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.\n- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.\n- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.\n- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. \"check the deploy every 5 minutes\", \"keep running /babysit-prs\"). Do NOT invoke for one-off tasks.\n- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run (\"run this once at 3pm\", \"remind me to check X tomorrow\").\n- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.\nTRIGGER — read BEFORE opening the target file; don't skip because it \"looks like a one-liner\" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).\nSKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).\n- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).\n- init: Initialize a new CLAUDE.md file with codebase documentation\n- review: Review a pull request\n- security-review: Complete a security review of the pending changes on the current branch","skillCount":13,"isInitial":true,"names":["update-config","keybindings-help","verify","code-review","simplify","fewer-permission-prompts","loop","schedule","claude-api","run","init","review","security-review"]},"type":"attachment","uuid":"50d73c36-6b5e-44a8-94ff-221ec48b7913","timestamp":"2026-06-14T05:32:07.029Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"50d73c36-6b5e-44a8-94ff-221ec48b7913","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01KJvtia72nVTMza1m6EtBA8","type":"message","role":"assistant","content":[{"type":"text","text":"I'll audit each bash script against its header documentation. Let me start by reading all the scripts."}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2024,"cache_read_input_tokens":5697,"cache_creation":{"ephemeral_5m_input_tokens":2024,"ephemeral_1h_input_tokens":0},"output_tokens":4,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3TfU89TfBpzoqpshv","attributionAgent":"Explore","type":"assistant","uuid":"0af94090-015c-480d-a4ae-60c04ad7180b","timestamp":"2026-06-14T05:32:11.640Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"0af94090-015c-480d-a4ae-60c04ad7180b","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01KJvtia72nVTMza1m6EtBA8","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_011jFiq3uJ9e54rnB8WebUhX","name":"Read","input":{"file_path":"/boot/config/plugins/varaverk/Fallback/fallback.sh"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2024,"cache_read_input_tokens":5697,"cache_creation":{"ephemeral_5m_input_tokens":2024,"ephemeral_1h_input_tokens":0},"output_tokens":4,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3TfU89TfBpzoqpshv","attributionAgent":"Explore","type":"assistant","uuid":"92eb9a40-7b8b-4827-86f3-a145f2e85e0c","timestamp":"2026-06-14T05:32:12.076Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"92eb9a40-7b8b-4827-86f3-a145f2e85e0c","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01KJvtia72nVTMza1m6EtBA8","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01LQznyebrPzLdxyfkVoJ1Gc","name":"Read","input":{"file_path":"/boot/config/plugins/varaverk/Fallback/fallback_test.sh"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2024,"cache_read_input_tokens":5697,"cache_creation":{"ephemeral_5m_input_tokens":2024,"ephemeral_1h_input_tokens":0},"output_tokens":4,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3TfU89TfBpzoqpshv","attributionAgent":"Explore","type":"assistant","uuid":"d221fd15-e2a3-42e9-b60f-115af3849dfa","timestamp":"2026-06-14T05:32:12.141Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"d221fd15-e2a3-42e9-b60f-115af3849dfa","isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01LQznyebrPzLdxyfkVoJ1Gc","type":"tool_result","content":"1\t#!/bin/bash\n2\t# ==============================================================================================\n3\t# ================================= Fallback Test ==============================================\n4\t# ==============================================================================================\n5\t#\n6\t# PURPOSE\n7\t# ─────────────────────────────────────────────────────────────────────────────\n8\t# Controlled simulation of the fallback lifecycle. Validates the entire sequence\n9\t# without waiting for a real outage. Contains no fallback logic — exercises the\n10\t# real fallback.sh via an iptables DROP rule on the remote Tailscale IP.\n11\t#\n12\t# Run during a maintenance window. Users will experience a brief service\n13\t# interruption. Use --dry-run to walk through all phases without real changes.\n14\t#\n15\t# ==============================================================================================\n16\t# OPERATIONAL MODEL\n17\t# ==============================================================================================\n18\t#\n19\t# Phase 1 — Pre-flight Both servers reachable, Docker daemons healthy,\n20\t# version parity, fallback.sh exists, state NORMAL\n21\t# Phase 2 — Block Remote iptables DROP rule added — remote appears unreachable\n22\t# Phase 3 — Fallback Detection Wait FALLBACK_TEST_BLOCK_WAIT for fallback.sh to\n23\t# detect the outage and enter FALLBACK state\n24\t# Phase 4 — Container Start Verify Tier 1 containers started locally\n25\t# Phase 5 — Restore iptables rule removed — remote reachable again\n26\t# Phase 6 — Handback Wait FALLBACK_TEST_HANDBACK_WAIT for fallback.sh to\n27\t# complete full handback and return to NORMAL\n28\t# Phase 7 — Container Handback Verify Tier 1 containers stopped locally\n29\t# Report — Full pass/fail per phase with timing\n30\t#\n31\t# ==============================================================================================\n32\t# DESIGN PRINCIPLES\n33\t# ==============================================================================================\n34\t#\n35\t# Test Harness Only\n36\t# Contains zero fallback logic. All fallback is exercised through fallback.sh.\n37\t# Any change to fallback.sh is automatically reflected in the test result.\n38\t#\n39\t# ==============================================================================================\n40\t# OPERATIONAL SAFEGUARDS\n41\t# ==============================================================================================\n42\t#\n43\t# iptables Safety Trap\n44\t# The DROP rule is removed via trap on ANY exit — normal completion, crash, error,\n45\t# ctrl-c. Remote connectivity is always restored regardless of test outcome.\n46\t# You cannot accidentally leave the remote permanently blocked.\n47\t#\n48\t# FALLBACK_ENABLED Gate\n49\t# Aborts if FALLBACK_ENABLED=false. Testing a disabled fallback system is\n50\t# misleading and potentially destructive.\n51\t#\n52\t# State Must Be NORMAL\n53\t# Pre-flight fails if state is not NORMAL. Running a test during an actual\n54\t# fallback event would interfere with the real event.\n55\t#\n56\t# Version Parity Check\n57\t# Pre-flight verifies unRAID version parity before any iptables rules are\n58\t# added. A mismatch makes the test result unreliable.\n59\t#\n60\t# Remote Docker Daemon Check\n61\t# Pre-flight confirms remote Docker daemon is responsive before Phase 2.\n62\t#\n63\t# Lock Acquisition\n64\t# acquire_lock() prevents concurrent test runs. Running two tests simultaneously\n65\t# would produce conflicting iptables rules and unreliable results.\n66\t#\n67\t# Host Detection\n68\t# detect_hosts() resolves MY_ID / REMOTE_ID from master.conf at startup.\n69\t# Exits if the host cannot be identified — prevents testing on an unknown machine.\n70\t#\n71\t# ==============================================================================================\n72\t# CONFIGURATION\n73\t# ==============================================================================================\n74\t#\n75\t# master.conf\n76\t#\n77\t# FALLBACK_TEST_BLOCK_WAIT\n78\t# Seconds to wait in Phase 3 for fallback.sh to detect the outage.\n79\t# Must be > FALLBACK_CHECK_INTERVAL + buffer. At 30s interval: use ≥60s.\n80\t# (default: 60)\n81\t#\n82\t# FALLBACK_TEST_HANDBACK_WAIT\n83\t# Seconds to wait in Phase 6 for fallback.sh to complete handback.\n84\t# Must cover: FALLBACK_HANDBACK_STRIKES × FALLBACK_CHECK_INTERVAL + rsync\n85\t# duration + container start time. At 3 strikes × 30s + ~2min rsync +\n86\t# ~1min container start: use ≥240s. (default: 300)\n87\t#\n88\t# ==============================================================================================\n89\t# RUNTIME MODES\n90\t# ==============================================================================================\n91\t#\n92\t# fallback_test.sh --dry-run\n93\t# Walk through all 7 phases with output but no iptables changes and no\n94\t# container starts/stops. ALWAYS run this before a live test.\n95\t#\n96\t# fallback_test.sh\n97\t# Full live test — real iptables DROP rule, real container lifecycle.\n98\t# Users will experience a brief service interruption. Run during a\n99\t# maintenance window.\n100\t#\n101\t# fallback_test.sh --status\n102\t# Show current fallback state and test timing configuration. No test run.\n103\t#\n104\t# fallback_test.sh --log\n105\t# Verbose output on every check in every phase.\n106\t#\n107\t# ==============================================================================================\n108\t\n109\tSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n110\t\n111\tsource \"$SCRIPT_DIR/../load_config.sh\"\n112\t\n113\tparse_args \"$@\"\n114\t\n115\tFALLBACK_SCRIPT=\"$SCRIPT_DIR/fallback.sh\"\n116\tDOCKER_TIMEOUT=15\n117\t\n118\t# ==============================================================================================\n119\t# ── SAFETY TRAP — always remove iptables rule on exit ─────────────────────────────────────────\n120\t# ==============================================================================================\n121\t# Fires on normal exit, error exit, ctrl-c, and script crashes.\n122\t# Remote connectivity is ALWAYS restored regardless of test outcome.\n123\t\n124\tIPTABLES_RULE_ACTIVE=false\n125\t\n126\tcleanup() {\n127\t if [[ \"$IPTABLES_RULE_ACTIVE\" == true ]]; then\n128\t echo \"\"\n129\t warn \"$ICON_SHIELD Cleanup — removing iptables block on $REMOTE_SERVER...\"\n130\t if [[ \"$DRY_RUN\" == false ]]; then\n131\t iptables -D OUTPUT -d \"$REMOTE_SERVER\" -j DROP 2>/dev/null\n132\t IPTABLES_RULE_ACTIVE=false\n133\t warn \"iptables rule removed — remote connectivity restored\"\n134\t else\n135\t warn \"DRY RUN — would remove iptables rule\"\n136\t fi\n137\t fi\n138\t}\n139\t\n140\ttrap cleanup EXIT\n141\t\n142\t# ==============================================================================================\n143\t# ━━━ Setup ━━━\n144\t# ==============================================================================================\n145\t\n146\tif [[ \"$EUID\" -ne 0 ]]; then\n147\t error \"Must be run as root\"\n148\t exit 1\n149\tfi\n150\t\n151\t# FALLBACK_ENABLED gate — no point testing if fallback is disabled\n152\tif [[ \"${FALLBACK_ENABLED:-false}\" == false ]]; then\n153\t warn \"FALLBACK_ENABLED=false — fallback test aborted\"\n154\t warn \"Enable fallback in master.conf before running this test\"\n155\t exit 0\n156\tfi\n157\t\n158\tacquire_lock # strict single instance — modifies iptables and containers\n159\t\n160\tif ! command -v docker &>/dev/null; then\n161\t error \"Docker command not found\"\n162\t exit 1\n163\tfi\n164\t\n165\tdetect_hosts\n166\tresolve_remote_ip\n167\t\n168\t# Validate commands used by this script\n169\tplatform_require_cmd \\\n170\t \"$(which iptables 2>/dev/null || echo /sbin/iptables)\" \\\n171\t \"--version\" \"iptables\" \\\n172\t \"iptables\" || { error \"iptables not found — required for connectivity simulation\"; exit 1; }\n173\t\n174\t\n175\tif [[ ! -f \"$FALLBACK_SCRIPT\" ]]; then\n176\t error \"fallback.sh not found at $FALLBACK_SCRIPT\"\n177\t exit 1\n178\tfi\n179\tlog \"fallback.sh found at $FALLBACK_SCRIPT\"\n180\tlog \"$ICON_GEAR Config: remote=${REMOTE_SERVER_NAME} (${REMOTE_SERVER}) fallback-script=${FALLBACK_SCRIPT}\"\n181\t\n182\t[[ \"$DRY_RUN\" == true ]] && warn \"DRY RUN — no iptables rules or container changes will be made\"\n183\t\n184\t# ==============================================================================================\n185\t# ━━━ Status ━━━\n186\t# ==============================================================================================\n187\tif [[ \"$SHOW_STATUS\" == true ]]; then\n188\t local_ver=$(platform_get_os_version 2>/dev/null || echo \"unknown\")\n189\t\n190\t echo \"\"\n191\t echo \"━━━━━ $ICON_SUMMARY STATUS ━━━━━\"\n192\t echo \"$ICON_HOST My ID: $MY_ID ($LOCAL_SERVER_NAME)\"\n193\t echo \"$ICON_HOST Remote ID: $REMOTE_ID ($REMOTE_SERVER_NAME — $REMOTE_SERVER)\"\n194\t echo \"$ICON_GEAR OS ver: $local_ver\"\n195\t echo \"$ICON_FALLBACK Block wait: ${FALLBACK_TEST_BLOCK_WAIT}s\"\n196\t echo \"$ICON_FALLBACK Handback wait: ${FALLBACK_TEST_HANDBACK_WAIT}s\"\n197\t echo \"$ICON_FALLBACK Check interval: ${FALLBACK_CHECK_INTERVAL}s\"\n198\t echo \"$ICON_FALLBACK Handback strikes: ${FALLBACK_HANDBACK_STRIKES}\"\n199\t echo \"$ICON_GEAR Dry Run: $DRY_RUN\"\n200\t\n201\t if [[ -f \"$FALLBACK_STATE_FILE\" ]]; then\n202\t CURRENT_STATE=$(grep \"^state=\" \"$FALLBACK_STATE_FILE\" 2>/dev/null | cut -d= -f2)\n203\t echo \"$ICON_FALLBACK Current state: ${CURRENT_STATE:-unknown}\"\n204\t else\n205\t echo \"$ICON_FALLBACK Current state: no state file\"\n206\t fi\n207\t\n208\t # Show Tier 1 containers for this host\n209\t TIER1_VAR=\"FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1\"\n210\t eval \"TIER1_CONTAINERS=(\\\"\\${${TIER1_VAR}[@]:-}\\\")\"\n211\t echo \"$ICON_CONTAINERS Tier 1 to test: ${TIER1_CONTAINERS[*]:-none configured}\"\n212\t echo \"━━━━━━━━━━━━━━━━━━━━━━━\"\n213\t exit 0\n214\tfi\n215\t\n216\t# ==============================================================================================\n217\t# ── PHASE TRACKING ────────────────────────────────────────────────────────────────────────────\n218\t# ==============================================================================================\n219\tPHASES_PASS=()\n220\tPHASES_FAIL=()\n221\tTOTAL_START=$(date +%s)\n222\t\n223\tphase_pass() { PHASES_PASS+=(\"$1\"); warn \"$ICON_DONE Phase: $1 — PASSED ✅\"; }\n224\tphase_fail() { PHASES_FAIL+=(\"$1\"); error \"Phase: $1 — FAILED ❌\"; }\n225\t\n226\t# Get Tier 1 containers for this server's fallback responsibility\n227\tTIER1_VAR=\"FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1\"\n228\teval \"TIER1_CONTAINERS=(\\\"\\${${TIER1_VAR}[@]:-}\\\")\"\n229\t\n230\t# ==============================================================================================\n231\t# ━━━ Phase 1 — Pre-flight ━━━\n232\t# ==============================================================================================\n233\techo \"\"\n234\techo \"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\"\n235\techo \" $ICON_SHIELD FALLBACK TEST — $(date '+%Y-%m-%d %H:%M:%S')\"\n236\techo \" $ICON_HOST $MY_ID ($LOCAL_SERVER_NAME) → $REMOTE_ID ($REMOTE_SERVER_NAME)\"\n237\techo \"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\"\n238\techo \"\"\n239\techo \"━━━ $ICON_SHIELD Phase 1 — Pre-flight ━━━\"\n240\t\n241\t# Remote reachable\n242\tif ping_remote; then\n243\t log \"$REMOTE_SERVER_NAME is reachable\"\n244\telse\n245\t error \"$REMOTE_SERVER_NAME is not reachable — cannot run test\"\n246\t phase_fail \"Pre-flight\"\n247\t exit 1\n248\tfi\n249\t\n250\t# Internet reachable\n251\tif ping_internet; then\n252\t log \"Internet is reachable\"\n253\telse\n254\t error \"No internet connectivity — cannot run test\"\n255\t phase_fail \"Pre-flight\"\n256\t exit 1\n257\tfi\n258\t\n259\t# Version parity — test may produce misleading results on mismatch\n260\tif ! check_os_version_parity; then\n261\t error \"unRAID version mismatch — test aborted to prevent misleading results\"\n262\t phase_fail \"Pre-flight\"\n263\t exit 1\n264\tfi\n265\t\n266\t# Remote Docker daemon — must be responsive before test manipulates containers\n267\tif ! check_remote_docker_daemon; then\n268\t error \"Remote Docker daemon not responsive — cannot run test\"\n269\t phase_fail \"Pre-flight\"\n270\t exit 1\n271\tfi\n272\t\n273\t# Fallback state must be NORMAL before test\n274\tif [[ -f \"$FALLBACK_STATE_FILE\" ]]; then\n275\t CURRENT_STATE=$(grep \"^state=\" \"$FALLBACK_STATE_FILE\" 2>/dev/null | cut -d= -f2)\n276\t if [[ \"$CURRENT_STATE\" != \"NORMAL\" ]]; then\n277\t error \"Fallback state is $CURRENT_STATE — must be NORMAL before running test\"\n278\t phase_fail \"Pre-flight\"\n279\t exit 1\n280\t fi\n281\t log \"Fallback state is NORMAL\"\n282\telse\n283\t warn \"No state file found — assuming NORMAL (first run)\"\n284\tfi\n285\t\n286\t# Tier 1 containers configured\n287\tif [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then\n288\t error \"No Tier 1 containers configured for $MY_ID → $REMOTE_ID\"\n289\t error \"Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in host*.conf\"\n290\t phase_fail \"Pre-flight\"\n291\t exit 1\n292\tfi\n293\tlog \"Tier 1 containers: ${TIER1_CONTAINERS[*]}\"\n294\t\n295\tphase_pass \"Pre-flight\"\n296\t\n297\t# ==============================================================================================\n298\t# ━━━ Phase 2 — Block Remote Connectivity ━━━\n299\t# ==============================================================================================\n300\techo \"\"\n301\techo \"━━━ $ICON_PING Phase 2 — Block Remote Connectivity ━━━\"\n302\twarn \"Adding iptables rule — dropping all traffic to $REMOTE_SERVER ($REMOTE_SERVER_NAME)\"\n303\t\n304\tif [[ \"$DRY_RUN\" == false ]]; then\n305\t iptables -I OUTPUT -d \"$REMOTE_SERVER\" -j DROP\n306\t IPTABLES_RULE_ACTIVE=true\n307\t\n308\t # Verify block is working\n309\t sleep 2\n310\t if ! ping -c1 -W2 \"$REMOTE_SERVER\" &>/dev/null; then\n311\t log \"Connectivity block confirmed — ping to remote fails as expected\"\n312\t phase_pass \"Block Remote\"\n313\t else\n314\t error \"iptables rule did not block connectivity — ping still succeeds\"\n315\t phase_fail \"Block Remote\"\n316\t exit 1\n317\t fi\n318\telse\n319\t warn \"DRY RUN — would block $REMOTE_SERVER with iptables DROP rule\"\n320\t phase_pass \"Block Remote\"\n321\tfi\n322\t\n323\t# ==============================================================================================\n324\t# ━━━ Phase 3 — Fallback Detection ━━━\n325\t# ==============================================================================================\n326\techo \"\"\n327\techo \"━━━ $ICON_FALLBACK Phase 3 — Fallback Detection ━━━\"\n328\twarn \"Waiting ${FALLBACK_TEST_BLOCK_WAIT}s for fallback.sh to detect outage...\"\n329\tlog \"fallback.sh check interval: ${FALLBACK_CHECK_INTERVAL}s\"\n330\t\n331\tif [[ \"$DRY_RUN\" == false ]]; then\n332\t sleep \"$FALLBACK_TEST_BLOCK_WAIT\"\n333\t\n334\t if [[ -f \"$FALLBACK_STATE_FILE\" ]]; then\n335\t NEW_STATE=$(grep \"^state=\" \"$FALLBACK_STATE_FILE\" 2>/dev/null | cut -d= -f2)\n336\t if [[ \"$NEW_STATE\" == \"FALLBACK\" ]]; then\n337\t log \"State changed to FALLBACK — outage detected correctly ✅\"\n338\t phase_pass \"Fallback Detection\"\n339\t else\n340\t error \"State is $NEW_STATE — expected FALLBACK after ${FALLBACK_TEST_BLOCK_WAIT}s\"\n341\t warn \"Is fallback.sh running? Check User Scripts plugin\"\n342\t phase_fail \"Fallback Detection\"\n343\t fi\n344\t else\n345\t error \"No state file found after wait — fallback.sh may not be running\"\n346\t phase_fail \"Fallback Detection\"\n347\t fi\n348\telse\n349\t warn \"DRY RUN — would wait ${FALLBACK_TEST_BLOCK_WAIT}s then check for FALLBACK state\"\n350\t phase_pass \"Fallback Detection\"\n351\tfi\n352\t\n353\t# ==============================================================================================\n354\t# ━━━ Phase 4 — Container Start Verification ━━━\n355\t# ==============================================================================================\n356\techo \"\"\n357\techo \"━━━ $ICON_CONTAINERS Phase 4 — Tier 1 Containers Started Locally ━━━\"\n358\tlog \"Checking Tier 1 containers: ${TIER1_CONTAINERS[*]}\"\n359\t\n360\tif [[ \"$DRY_RUN\" == false ]]; then\n361\t CONTAINERS_OK=true\n362\t for container in \"${TIER1_CONTAINERS[@]}\"; do\n363\t [[ -z \"$container\" ]] && continue\n364\t STATUS=$(timeout \"$DOCKER_TIMEOUT\" docker inspect -f '{{.State.Running}}' \\\n365\t \"$container\" 2>/dev/null)\n366\t if [[ \"$STATUS\" == \"true\" ]]; then\n367\t log \"$ICON_RUNNING $container is running locally ✅\"\n368\t else\n369\t error \"$ICON_NOT_RUNNING $container is NOT running locally\"\n370\t CONTAINERS_OK=false\n371\t fi\n372\t done\n373\t\n374\t if [[ \"$CONTAINERS_OK\" == true ]]; then\n375\t phase_pass \"Container Start\"\n376\t else\n377\t phase_fail \"Container Start\"\n378\t fi\n379\telse\n380\t warn \"DRY RUN — would verify these Tier 1 containers started: ${TIER1_CONTAINERS[*]}\"\n381\t phase_pass \"Container Start\"\n382\tfi\n383\t\n384\t# ==============================================================================================\n385\t# ━━━ Phase 5 — Restore Remote Connectivity ━━━\n386\t# ==============================================================================================\n387\techo \"\"\n388\techo \"━━━ $ICON_PING Phase 5 — Restore Remote Connectivity ━━━\"\n389\twarn \"Removing iptables block — $REMOTE_SERVER_NAME becomes reachable again\"\n390\t\n391\tif [[ \"$DRY_RUN\" == false ]]; then\n392\t iptables -D OUTPUT -d \"$REMOTE_SERVER\" -j DROP 2>/dev/null\n393\t IPTABLES_RULE_ACTIVE=false\n394\t\n395\t sleep 3\n396\t if ping_remote; then\n397\t log \"$REMOTE_SERVER_NAME is reachable again ✅\"\n398\t phase_pass \"Restore Connectivity\"\n399\t else\n400\t error \"$REMOTE_SERVER_NAME still unreachable after removing iptables rule\"\n401\t phase_fail \"Restore Connectivity\"\n402\t fi\n403\telse\n404\t warn \"DRY RUN — would remove iptables rule\"\n405\t phase_pass \"Restore Connectivity\"\n406\tfi\n407\t\n408\t# ==============================================================================================\n409\t# ━━━ Phase 6 — Handback ━━━\n410\t# ==============================================================================================\n411\techo \"\"\n412\techo \"━━━ $ICON_FALLBACK Phase 6 — Handback ━━━\"\n413\twarn \"Waiting ${FALLBACK_TEST_HANDBACK_WAIT}s for fallback.sh to complete handback...\"\n414\tlog \"Requires $FALLBACK_HANDBACK_STRIKES consecutive checks at ${FALLBACK_CHECK_INTERVAL}s\"\n415\tlog \"Minimum handback time: $(( FALLBACK_HANDBACK_STRIKES * FALLBACK_CHECK_INTERVAL ))s\"\n416\t\n417\tif [[ \"$DRY_RUN\" == false ]]; then\n418\t sleep \"$FALLBACK_TEST_HANDBACK_WAIT\"\n419\t\n420\t if [[ -f \"$FALLBACK_STATE_FILE\" ]]; then\n421\t FINAL_STATE=$(grep \"^state=\" \"$FALLBACK_STATE_FILE\" 2>/dev/null | cut -d= -f2)\n422\t if [[ \"$FINAL_STATE\" == \"NORMAL\" ]]; then\n423\t log \"State returned to NORMAL — handback completed ✅\"\n424\t phase_pass \"Handback\"\n425\t else\n426\t error \"State is $FINAL_STATE — expected NORMAL after ${FALLBACK_TEST_HANDBACK_WAIT}s\"\n427\t warn \"Handback may still be in progress — check fallback.sh output\"\n428\t phase_fail \"Handback\"\n429\t fi\n430\t else\n431\t error \"No state file found\"\n432\t phase_fail \"Handback\"\n433\t fi\n434\telse\n435\t warn \"DRY RUN — would wait ${FALLBACK_TEST_HANDBACK_WAIT}s then verify NORMAL state\"\n436\t phase_pass \"Handback\"\n437\tfi\n438\t\n439\t# ==============================================================================================\n440\t# ━━━ Phase 7 — Container Handback Verification ━━━\n441\t# ==============================================================================================\n442\techo \"\"\n443\techo \"━━━ $ICON_CONTAINERS Phase 7 — Tier 1 Containers Stopped Locally ━━━\"\n444\tlog \"Verifying Tier 1 containers returned to $REMOTE_SERVER_NAME\"\n445\t\n446\tif [[ \"$DRY_RUN\" == false ]]; then\n447\t HANDBACK_OK=true\n448\t for container in \"${TIER1_CONTAINERS[@]}\"; do\n449\t [[ -z \"$container\" ]] && continue\n450\t STATUS=$(timeout \"$DOCKER_TIMEOUT\" docker inspect -f '{{.State.Running}}' \\\n451\t \"$container\" 2>/dev/null)\n452\t if [[ \"$STATUS\" != \"true\" ]]; then\n453\t log \"$ICON_NOT_RUNNING $container stopped locally — handed back ✅\"\n454\t else\n455\t error \"$ICON_RUNNING $container still running locally — handback may have failed\"\n456\t HANDBACK_OK=false\n457\t fi\n458\t done\n459\t\n460\t if [[ \"$HANDBACK_OK\" == true ]]; then\n461\t phase_pass \"Container Handback\"\n462\t else\n463\t phase_fail \"Container Handback\"\n464\t fi\n465\telse\n466\t warn \"DRY RUN — would verify Tier 1 containers stopped locally after handback\"\n467\t phase_pass \"Container Handback\"\n468\tfi\n469\t\n470\tTOTAL_END=$(date +%s)\n471\t\n472\t# ==============================================================================================\n473\t# ━━━ Test Report ━━━\n474\t# ==============================================================================================\n475\techo \"\"\n476\techo \"━━━━━ $ICON_SUMMARY FALLBACK TEST REPORT ━━━━━\"\n477\techo \"$ICON_HOST My ID: $MY_ID ($LOCAL_SERVER_NAME)\"\n478\techo \"$ICON_HOST Remote: $REMOTE_ID ($REMOTE_SERVER_NAME)\"\n479\techo \"$ICON_TIME Duration: $(format_duration $((TOTAL_END - TOTAL_START)))\"\n480\techo \"\"\n481\techo \" Phase Results:\"\n482\tfor phase in \"${PHASES_PASS[@]}\"; do\n483\t echo \" $ICON_SUCCESS $phase\"\n484\tdone\n485\tfor phase in \"${PHASES_FAIL[@]}\"; do\n486\t echo \" $ICON_ERROR $phase\"\n487\tdone\n488\techo \"\"\n489\t\n490\tPASS_COUNT=${#PHASES_PASS[@]}\n491\tFAIL_COUNT=${#PHASES_FAIL[@]}\n492\tTOTAL_PHASES=$(( PASS_COUNT + FAIL_COUNT ))\n493\t\n494\tif [[ \"$DRY_RUN\" == true ]]; then\n495\t warn \"DRY RUN — no changes made\"\n496\telif [[ \"$FAIL_COUNT\" -eq 0 ]]; then\n497\t warn \"$ICON_DONE ALL $TOTAL_PHASES PHASES PASSED\"\n498\t notify \"Fallback test PASSED on $(hostname) — all $TOTAL_PHASES phases completed\" \\\n499\t \"Fallback Test\" \"normal\"\n500\telse\n501\t error \"$FAIL_COUNT/$TOTAL_PHASES PHASES FAILED\"\n502\t notify \"Fallback test FAILED on $(hostname) — $FAIL_COUNT/$TOTAL_PHASES phases failed: ${PHASES_FAIL[*]}\" \\\n503\t \"Fallback Test\" \"warning\"\n504\tfi\n505\techo \"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\"\n506\t\n507\t[[ \"$FAIL_COUNT\" -gt 0 ]] && exit 1\n508\texit 0"}]},"uuid":"c3eb9676-c02a-44e1-8c9b-252e3f519d57","timestamp":"2026-06-14T05:32:12.151Z","sourceToolAssistantUUID":"d221fd15-e2a3-42e9-b60f-115af3849dfa","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"c3eb9676-c02a-44e1-8c9b-252e3f519d57","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01KJvtia72nVTMza1m6EtBA8","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01LW56UYpUBLR69BiaAZ3imt","name":"Read","input":{"file_path":"/boot/config/plugins/varaverk/Kernel/decision_engine.sh"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2024,"cache_read_input_tokens":5697,"cache_creation":{"ephemeral_5m_input_tokens":2024,"ephemeral_1h_input_tokens":0},"output_tokens":4,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3TfU89TfBpzoqpshv","attributionAgent":"Explore","type":"assistant","uuid":"d5933a6d-f045-4ed7-97ab-4897475f465a","timestamp":"2026-06-14T05:32:12.185Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"d5933a6d-f045-4ed7-97ab-4897475f465a","isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01LW56UYpUBLR69BiaAZ3imt","type":"tool_result","content":"1\t#!/bin/bash\n2\t# ==============================================================================================\n3\t# ================================= Decision Engine ============================================\n4\t# ==============================================================================================\n5\t#\n6\t# PURPOSE\n7\t# ─────────────────────────────────────────────────────────────────────────────\n8\t# Behavior-driven scoring kernel for media automation. Sourced by consumer\n9\t# scripts — not run directly. Evaluates candidates, applies weighted scoring,\n10\t# temporal decay, and deduplication, then returns a verdict.\n11\t#\n12\t# Does NOT download media, search indexers, or manage applications. Has no\n13\t# side effects — no file writes, no API calls, no deletions.\n14\t#\n15\t# Currently paired with: Media/playback_aware_lidarr_discovery.sh\n16\t#\n17\t# ==============================================================================================\n18\t# DESIGN PRINCIPLES\n19\t# ==============================================================================================\n20\t#\n21\t# Domain-Agnostic Core\n22\t# The engine itself has no knowledge of music vs TV vs movies. Consumers\n23\t# define thresholds, weights, and strictness profiles — the engine just\n24\t# scores and decides. This prevents cross-domain bias pollution: music\n25\t# strictness cannot contaminate TV intake logic.\n26\t#\n27\t# Lidarr consumers → highly selective, quality-first discovery\n28\t# Sonarr consumers → balanced, family-aware episodic intake\n29\t# Radarr consumers → broader flexibility with intelligent filtering\n30\t#\n31\t# Temporal Decay\n32\t# Old behavioral signals lose influence over time (one decay unit per 30\n33\t# days). Prevents permanent genre lock-in, historical bias accumulation,\n34\t# and dead-user score dominance.\n35\t#\n36\t# Consumer Owns the Decision\n37\t# The engine returns ACCEPT/REJECT/SCORE. What happens next is entirely\n38\t# the consumer's concern — the engine never acts on its own verdict.\n39\t#\n40\t# ==============================================================================================\n41\t# FUNCTIONS\n42\t# ==============================================================================================\n43\t#\n44\t# score_candidate user_score popularity_score recency_score quality_score\n45\t# Returns TOTAL_SCORE (integer sum). Consumer defines actual weight values.\n46\t#\n47\t# evaluate_threshold score minimum\n48\t# Returns 0 (pass) or 1 (fail). Used as: if evaluate_threshold ...\n49\t#\n50\t# apply_temporal_decay score age_days\n51\t# Returns adjusted score. Subtracts (age_days / 30), floor at 0.\n52\t#\n53\t# is_duplicate_candidate candidate history_file\n54\t# Returns 0 (duplicate found in history_file) or 1 (not found).\n55\t#\n56\t# make_decision score threshold\n57\t# Returns \"ACCEPT\" or \"REJECT\". Calls evaluate_threshold internally.\n58\t#\n59\t# ==============================================================================================\n60\t\n61\t# ── score_candidate ───────────────────────────────────────────────────────────\n62\tscore_candidate() {\n63\t\n64\t local user_score=\"${1:-0}\"\n65\t local popularity_score=\"${2:-0}\"\n66\t local recency_score=\"${3:-0}\"\n67\t local quality_score=\"${4:-0}\"\n68\t\n69\t TOTAL_SCORE=$(( \\\n70\t user_score + \\\n71\t popularity_score + \\\n72\t recency_score + \\\n73\t quality_score \\\n74\t ))\n75\t\n76\t echo \"$TOTAL_SCORE\"\n77\t}\n78\t\n79\t# ── evaluate_threshold ───────────────────────────────────────────────────────\n80\tevaluate_threshold() {\n81\t\n82\t local score=\"$1\"\n83\t local minimum=\"$2\"\n84\t\n85\t if (( score >= minimum )); then\n86\t return 0\n87\t fi\n88\t\n89\t return 1\n90\t}\n91\t\n92\t# ── apply_temporal_decay ─────────────────────────────────────────────────────\n93\tapply_temporal_decay() {\n94\t\n95\t local score=\"$1\"\n96\t local age_days=\"$2\"\n97\t\n98\t local decay=$(( age_days / 30 ))\n99\t\n100\t local adjusted=$(( score - decay ))\n101\t\n102\t (( adjusted < 0 )) && adjusted=0\n103\t\n104\t echo \"$adjusted\"\n105\t}\n106\t\n107\t# ── is_duplicate_candidate ───────────────────────────────────────────────────\n108\tis_duplicate_candidate() {\n109\t\n110\t local candidate=\"$1\"\n111\t local history_file=\"$2\"\n112\t\n113\t grep -qi \"^${candidate}$\" \"$history_file\" 2>/dev/null\n114\t}\n115\t\n116\t# ── make_decision ─────────────────────────────────────────────────────────────\n117\tmake_decision() {\n118\t\n119\t local score=\"$1\"\n120\t local threshold=\"$2\"\n121\t\n122\t if evaluate_threshold \"$score\" \"$threshold\"; then\n123\t echo \"ACCEPT\"\n124\t else\n125\t echo \"REJECT\"\n126\t fi\n127\t}"}]},"uuid":"fb734b7e-2616-40b0-8c05-69d41e760f87","timestamp":"2026-06-14T05:32:12.207Z","sourceToolAssistantUUID":"d5933a6d-f045-4ed7-97ab-4897475f465a","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"fb734b7e-2616-40b0-8c05-69d41e760f87","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01KJvtia72nVTMza1m6EtBA8","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_0174FazUeyWtdGFACjXD6jbL","name":"Read","input":{"file_path":"/boot/config/plugins/varaverk/Rsync/rsync.sh"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2024,"cache_read_input_tokens":5697,"cache_creation":{"ephemeral_5m_input_tokens":2024,"ephemeral_1h_input_tokens":0},"output_tokens":4,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3TfU89TfBpzoqpshv","attributionAgent":"Explore","type":"assistant","uuid":"dac09318-f367-414d-aba2-516a01211130","timestamp":"2026-06-14T05:32:12.303Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"dac09318-f367-414d-aba2-516a01211130","isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_0174FazUeyWtdGFACjXD6jbL","type":"tool_result","content":"1\t#!/bin/bash\n2\t# ==============================================================================================\n3\t# ============================= Rsync Core Script ==============================================\n4\t# ==============================================================================================\n5\t#\n6\t# PURPOSE\n7\t# ─────────────────────────────────────────────────────────────────────────────\n8\t# Core rsync engine for the two-server ecosystem. Called per share or per\n9\t# appdata profile by orchestrators (daily_sync_maintenance, weekly_sync_maintenance,\n10\t# critical_sync_maintenance) and directly for manual or scheduled dirty syncs.\n11\t#\n12\t# Profile is inferred from the directory basename (lowercased). Override with\n13\t# --profile=name for explicit selection. If no profile matches, global defaults\n14\t# from master.conf apply and no containers are stopped.\n15\t#\n16\t# After each sync, logs transfer data to bandwidth_monitor.sh for the weekly\n17\t# bandwidth report. Silent on success — only failures produce visible output.\n18\t#\n19\t# ==============================================================================================\n20\t# OPERATIONAL MODEL\n21\t# ==============================================================================================\n22\t#\n23\t# Profiles define per-share behavior:\n24\t# PROFILE_CRITICAL_CONTAINER_NAMES — containers stopped on both servers before sync\n25\t# PROFILE_DELAYED_CONTAINERS — containers with a delay before restart after sync\n26\t# PROFILE_CONTAINER_DELAY — seconds before delayed containers start\n27\t# PROFILE_RSYNC_OPTS — rsync flags (does not inherit DEFAULT_RSYNC_OPTS)\n28\t# PROFILE_BW_LIMIT — bandwidth limit in KB/s\n29\t# PROFILE_RETRY_COUNT — retry attempts on failure\n30\t# PROFILE_SLEEP — seconds between retry attempts\n31\t# PROFILE_EXCLUDE_DIRS — paths excluded from transfer\n32\t# PROFILE_REMOTE_RESTART_CONTAINERS — containers restarted on remote after dirty sync\n33\t# Was running → restart. Was stopped → leave stopped.\n34\t#\n35\t# Two-tier rsync enable/disable:\n36\t# Tier 1: RSYNC_ENABLED=false → all rsync stops immediately (checked by this script)\n37\t# Tier 2: per-orchestrator flag (DAILY_RSYNC_ENABLED etc.) → checked by caller\n38\t#\n39\t# Bandwidth logging: after each sync, logs profile/duration/status/bytes to\n40\t# bandwidth_monitor.sh --log-transfer. Bytes captured from rsync --stats via awk\n41\t# using version-stable field names.\n42\t#\n43\t# ==============================================================================================\n44\t# OPERATIONAL SAFEGUARDS\n45\t# ==============================================================================================\n46\t#\n47\t# Global Rsync Gate\n48\t# check_rsync_enabled() — RSYNC_ENABLED=false exits cleanly before any operation.\n49\t#\n50\t# Partnership Blocklist\n51\t# Refuses to sync if REMOTE_SERVER_NAME appears in the partnership blocklist.\n52\t# Written at offboard — prevents stale access after a partnership ends.\n53\t#\n54\t# Version Parity\n55\t# check_os_version_parity — refuses sync if servers on incompatible unRAID versions.\n56\t#\n57\t# Remote Health Pre-flights\n58\t# check_connectivity() — Tailscale IP reachable before any SSH\n59\t# check_remote_rootfs() — aborts if remote rootfs exceeds ROOTFS_WARN_PCT\n60\t# check_remote_share() — aborts if target directory missing or empty on remote\n61\t# check_remote_disks() — verifies all backing disks online on remote\n62\t#\n63\t# Drive Temperature Check\n64\t# check_local_disk_temps() — runs before any transfer. Exit 1 = skip this profile,\n65\t# exit 2 = abort all remaining syncs (CRITICAL temperature).\n66\t#\n67\t# Remote Docker Daemon Check\n68\t# check_remote_docker_daemon — verified before any container stop/start operations.\n69\t# If daemon unresponsive: container operations skipped, rsync proceeds without stopping.\n70\t#\n71\t# Per-Profile Concurrency Lock\n72\t# acquire_rsync_lock() — per-profile lock prevents parallel runs of the same profile.\n73\t# Global concurrent limit prevents too many simultaneous rsync processes.\n74\t#\n75\t# Notification Validated\n76\t# platform_require_cmd confirms the notify script is present before use.\n77\t#\n78\t# ==============================================================================================\n79\t# CONFIGURATION\n80\t# ==============================================================================================\n81\t#\n82\t# master.conf\n83\t#\n84\t# RSYNC_ENABLED\n85\t# Global on/off toggle for all rsync operations. (default: true)\n86\t#\n87\t# DEFAULT_RSYNC_OPTS\n88\t# Base rsync flags for unproiled shares. Note: --delete is intentionally absent —\n89\t# media shares spread files only, arr cleanup scripts own deletions. Profile-specific\n90\t# opts set --delete explicitly where needed.\n91\t#\n92\t# BW_LIMIT\n93\t# Default bandwidth cap in KB/s when no PROFILE_BW_LIMIT is set. (default: 0 = unlimited)\n94\t#\n95\t# RETRY_COUNT\n96\t# Default retry attempts on rsync failure. (default: 3)\n97\t#\n98\t# SLEEP\n99\t# Default seconds between retry attempts. (default: 60)\n100\t#\n101\t# ROOTFS_WARN_PCT\n102\t# Abort threshold for remote rootfs percentage full. (default: 75)\n103\t#\n104\t# PROFILES[\"profile_KEY\"]\n105\t# Profile definitions — one entry per PROFILE_* key per profile name.\n106\t# See OPERATIONAL MODEL above for all supported keys.\n107\t#\n108\t# BANDWIDTH_LOG / BANDWIDTH_WARN_GB\n109\t# Shared with bandwidth_monitor.sh — set once, used by both.\n110\t#\n111\t# ==============================================================================================\n112\t# RUNTIME MODES\n113\t# ==============================================================================================\n114\t#\n115\t# rsync.sh /path/to/share\n116\t# Sync the given path to the remote. Profile inferred from directory basename.\n117\t#\n118\t# rsync.sh /path/to/share --profile=name\n119\t# Sync with explicit profile override — bypasses basename inference.\n120\t#\n121\t# rsync.sh /path/to/share --dry-run\n122\t# Run all pre-flight checks and show what rsync would transfer. No transfer,\n123\t# no container stops.\n124\t#\n125\t# rsync.sh /path/to/share --status\n126\t# Show resolved profile, remote identity, and configuration. Then exit.\n127\t#\n128\t# rsync.sh /path/to/share --log\n129\t# Verbose output throughout — every decision logged.\n130\t#\n131\t# ==============================================================================================\n132\t\n133\tSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n134\t\n135\tsource \"$SCRIPT_DIR/../load_config.sh\"\n136\t\n137\t# ── Separate positional directory arg from flags ───────────────────────────────────────────────\n138\tDIRECTORY=\"\"\n139\tPROFILE_OVERRIDE=\"\"\n140\tRAW_ARGS=()\n141\t\n142\tfor ARG in \"$@\"; do\n143\t case \"$ARG\" in\n144\t --profile=*) PROFILE_OVERRIDE=\"${ARG#--profile=}\" ;;\n145\t --*|*=*) RAW_ARGS+=(\"$ARG\") ;;\n146\t *) [[ -z \"$DIRECTORY\" ]] && DIRECTORY=\"$ARG\" || RAW_ARGS+=(\"$ARG\") ;;\n147\t esac\n148\tdone\n149\t\n150\tparse_args \"${RAW_ARGS[@]}\"\n151\t\n152\t[[ -z \"$DIRECTORY\" ]] && {\n153\t error \"No directory specified\"\n154\t error \"Usage: rsync.sh [--dry-run] [--log] [--profile=name]\"\n155\t exit 1\n156\t}\n157\t\n158\t# ==============================================================================================\n159\t# ━━━ Setup ━━━\n160\t# ==============================================================================================\n161\tif [[ \"$EUID\" -ne 0 ]]; then\n162\t error \"Must be run as root\"\n163\t exit 1\n164\tfi\n165\t\n166\tacquire_lock\n167\t\n168\tif ! command -v docker &>/dev/null; then\n169\t error \"Docker command not found\"\n170\t exit 1\n171\tfi\n172\t\n173\t\n174\tdetect_hosts\n175\t\n176\t# Tier 1 global gate — Tier 2 (per-orchestrator) checked by caller\n177\tif ! check_rsync_enabled; then\n178\t warn \"RSYNC_ENABLED=false — exiting cleanly\"\n179\t exit 0\n180\tfi\n181\t\n182\t# Blocklist gate — refuse to sync with a partner blocked after offboard\n183\tBLOCKLIST_FILE=\"${PARTNERSHIP_BLOCKLIST_FILE:-${STATE_DIR}/partnership_blocklist.db}\"\n184\tif [[ -f \"$BLOCKLIST_FILE\" ]] && grep -q \"^${REMOTE_SERVER_NAME}|\" \"$BLOCKLIST_FILE\" 2>/dev/null; then\n185\t error \"Rsync blocked — $REMOTE_SERVER_NAME is on the partnership blocklist\"\n186\t error \"Re-onboard the partnership to restore access: partnership_manager.sh --onboard\"\n187\t exit 1\n188\tfi\n189\t\n190\tresolve_remote_ip\n191\t\n192\t# ── Profile inference ─────────────────────────────────────────────────────────────────────────\n193\tif [[ -n \"$PROFILE_OVERRIDE\" ]]; then\n194\t PROFILE_NAME=\"$PROFILE_OVERRIDE\"\n195\t log \"Profile override: $PROFILE_NAME\"\n196\telse\n197\t PROFILE_NAME=$(basename \"$DIRECTORY\" | tr '[:upper:]' '[:lower:]')\n198\t log \"Profile inferred: $PROFILE_NAME\"\n199\tfi\n200\t\n201\t# Acquire per-profile lock and check global concurrent limit\n202\tacquire_rsync_lock \"$PROFILE_NAME\"\n203\t\n204\t# Tee all output to a live log file for the Varaverk UI\n205\tVV_LIVE_LOG=\"/tmp/unraid_locks/rsync_${PROFILE_NAME}.log\"\n206\tVV_LAST_LOG=\"/tmp/unraid_locks/rsync_${PROFILE_NAME}.last.log\"\n207\t: > \"$VV_LIVE_LOG\"\n208\texec 1> >(tee -a \"$VV_LIVE_LOG\") 2>&1\n209\t\n210\t# ── Load profile settings ─────────────────────────────────────────────────────────────────────\n211\tBW_LIMIT=${PROFILE_BW_LIMIT[$PROFILE_NAME]:-$BW_LIMIT}\n212\tRETRY_COUNT=${PROFILE_RETRY_COUNT[$PROFILE_NAME]:-$RETRY_COUNT}\n213\tSLEEP=${PROFILE_SLEEP[$PROFILE_NAME]:-$SLEEP}\n214\tCONTAINER_DELAY=${PROFILE_CONTAINER_DELAY[$PROFILE_NAME]:-$CONTAINER_DELAY}\n215\t\n216\tread -r -a CRITICAL_CONTAINER_NAMES <<< \"${PROFILE_CRITICAL_CONTAINER_NAMES[$PROFILE_NAME]:-}\"\n217\tread -r -a DELAYED_CONTAINERS <<< \"${PROFILE_DELAYED_CONTAINERS[$PROFILE_NAME]:-}\"\n218\tread -r -a EXCLUDE_DIRS <<< \"${PROFILE_EXCLUDE_DIRS[$PROFILE_NAME]:-}\"\n219\tread -r -a REMOTE_RESTART_CONTAINERS <<< \"${PROFILE_REMOTE_RESTART_CONTAINERS[$PROFILE_NAME]:-}\"\n220\t\n221\t# Local containers use same names as remote (mirrored naming scheme)\n222\tLOCAL_CRITICAL_CONTAINER_NAMES=(\"${CRITICAL_CONTAINER_NAMES[@]}\")\n223\t\n224\tlog \"$ICON_GEAR Config: profile=${PROFILE_NAME} bw-limit=${BW_LIMIT}KB/s retry=${RETRY_COUNT} sleep=${SLEEP}s container-delay=${CONTAINER_DELAY}s\"\n225\tlog \"$ICON_GEAR Containers: critical=${CRITICAL_CONTAINER_NAMES[*]:-none} delayed=${DELAYED_CONTAINERS[*]:-none} remote-restart=${REMOTE_RESTART_CONTAINERS[*]:-none}\"\n226\t\n227\t[[ \"$SHOW_STATUS\" == true ]] && show_status && exit 0\n228\t[[ \"$DRY_RUN\" == true ]] && warn \"DRY RUN — no changes will be made\"\n229\t\n230\t# ==============================================================================================\n231\t# ━━━ Pre-flight Checks ━━━\n232\t# ==============================================================================================\n233\techo \"\"\n234\techo \"━━━ $ICON_SHIELD Pre-flight ━━━\"\n235\t\n236\t# Disk temp — before touching remote or moving data\n237\t# Exit 1 = skip this profile | Exit 2 = abort all remaining profiles\n238\tcheck_local_disk_temps\n239\tTEMP_RESULT=$?\n240\tif [[ \"$TEMP_RESULT\" -eq 2 ]]; then\n241\t error \"Drive temps CRITICAL — aborting all remaining syncs\"\n242\t exit 2\n243\telif [[ \"$TEMP_RESULT\" -eq 1 ]]; then\n244\t warn \"Drive temps high — skipping profile [$PROFILE_NAME]\"\n245\t exit 1\n246\telse\n247\t log \"Drive temps OK — $TEMP_CHECK_RESULT\"\n248\tfi\n249\t\n250\t# Version parity — refuse if servers on incompatible unRAID versions\n251\tcheck_os_version_parity || exit 1\n252\t\n253\tcheck_connectivity\n254\tcheck_remote_rootfs\n255\tcheck_remote_share \"$DIRECTORY\"\n256\tcheck_remote_disks \"$DIRECTORY\"\n257\t\n258\t# Remote Docker daemon — check before attempting container operations\n259\tif [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]] || [[ ${#REMOTE_RESTART_CONTAINERS[@]} -gt 0 ]]; then\n260\t check_remote_docker_daemon || {\n261\t warn \"Remote Docker daemon unresponsive — skipping container operations\"\n262\t warn \"Proceeding with rsync only — containers will not be stopped or restarted\"\n263\t CRITICAL_CONTAINER_NAMES=()\n264\t LOCAL_CRITICAL_CONTAINER_NAMES=()\n265\t REMOTE_RESTART_CONTAINERS=()\n266\t }\n267\tfi\n268\t\n269\t# ==============================================================================================\n270\t# ━━━ Stop Containers ━━━\n271\t# ==============================================================================================\n272\tif [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 || ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]]; then\n273\t echo \"\"\n274\t echo \"━━━ $ICON_STOP $ICON_CONTAINERS Stop Containers ━━━\"\n275\t if [[ \"$DRY_RUN\" == true ]]; then\n276\t for c in \"${LOCAL_CRITICAL_CONTAINER_NAMES[@]:-}\"; do\n277\t [[ -n \"$c\" ]] && warn \"DRY RUN — would stop local: $c\"\n278\t done\n279\t for c in \"${CRITICAL_CONTAINER_NAMES[@]:-}\"; do\n280\t [[ -n \"$c\" ]] && warn \"DRY RUN — would stop remote: $c\"\n281\t done\n282\t else\n283\t # Local first — flush local databases before pushing\n284\t stop_local_containers\n285\t # Remote next — prevent writes while receiving\n286\t stop_containers\n287\t fi\n288\tfi\n289\t\n290\t# ==============================================================================================\n291\t# ━━━ Transfer ━━━\n292\t# ==============================================================================================\n293\techo \"\"\n294\techo \"━━━ $ICON_SYNC Transfer ━━━\"\n295\techo \"$ICON_RUN Source: $DIRECTORY\"\n296\techo \"$ICON_NET Remote: $REMOTE_SERVER:$DIRECTORY\"\n297\techo \"$ICON_GEAR Profile: $PROFILE_NAME\"\n298\techo \"$ICON_HOST Identity: $MY_ID → $REMOTE_ID\"\n299\techo \"\"\n300\t\n301\tget_rsync_opts\n302\t\n303\t# Append profile excludes\n304\tfor ex in \"${EXCLUDE_DIRS[@]:-}\"; do\n305\t [[ -n \"$ex\" ]] && RSYNC_OPTS+=(--exclude=\"$ex\")\n306\tdone\n307\t\n308\t# Add --stats to capture bytes transferred for bandwidth logging\n309\tRSYNC_OPTS+=(--stats)\n310\t\n311\t[[ \"$DRY_RUN\" == true ]] && RSYNC_OPTS+=(\"--dry-run\")\n312\t\n313\tSTART=$(date +%s)\n314\tRSYNC_SUCCESS=false\n315\tBYTES_TRANSFERRED=0\n316\tATTEMPT=0\n317\t\n318\tfor (( ATTEMPT=1; ATTEMPT<=RETRY_COUNT; ATTEMPT++ )); do\n319\t log \"$ICON_RETRY Attempt $ATTEMPT of $RETRY_COUNT...\"\n320\t echo \"$ICON_SYNC Rsync running — this may take a while...\"\n321\t\n322\t RSYNC_OUTPUT=$(rsync \"${RSYNC_OPTS[@]}\" \\\n323\t -e \"ssh -i $SSH_KEY -T -o Compression=no -o IPQoS=throughput\" \\\n324\t \"$DIRECTORY\" \"root@${REMOTE_SERVER}:$(dirname \"$DIRECTORY\")/\" 2>&1)\n325\t\n326\t RSYNC_EXIT=$?\n327\t\n328\t if [[ \"$RSYNC_EXIT\" -eq 0 ]]; then\n329\t # Parse bytes transferred from --stats output\n330\t BYTES_TRANSFERRED=$(echo \"$RSYNC_OUTPUT\" | \\\n331\t awk '/Total transferred file size:/{gsub(/,/,\"\",$NF); gsub(/[^0-9]/,\"\",$NF); print $NF+0}')\n332\t BYTES_TRANSFERRED=\"${BYTES_TRANSFERRED:-0}\"\n333\t\n334\t log \"$ICON_DONE Rsync complete — $BYTES_TRANSFERRED bytes transferred\"\n335\t RSYNC_SUCCESS=true\n336\t break\n337\t else\n338\t warn \"$ICON_RETRY Rsync failed (attempt $ATTEMPT/$RETRY_COUNT)\"\n339\t log \"Exit code: $RSYNC_EXIT\"\n340\t if [[ \"$ATTEMPT\" -lt \"$RETRY_COUNT\" ]]; then\n341\t log \"Retrying in ${SLEEP}s...\"\n342\t sleep \"$SLEEP\"\n343\t fi\n344\t fi\n345\tdone\n346\t\n347\t# ==============================================================================================\n348\t# ━━━ Start Containers ━━━\n349\t# ==============================================================================================\n350\tif [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 || ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]]; then\n351\t echo \"\"\n352\t echo \"━━━ $ICON_START $ICON_CONTAINERS Start Containers ━━━\"\n353\t if [[ \"$DRY_RUN\" == true ]]; then\n354\t for c in \"${CRITICAL_CONTAINER_NAMES[@]:-}\"; do\n355\t [[ -n \"$c\" ]] && warn \"DRY RUN — would start remote: $c\"\n356\t done\n357\t for c in \"${LOCAL_CRITICAL_CONTAINER_NAMES[@]:-}\"; do\n358\t [[ -n \"$c\" ]] && warn \"DRY RUN — would start local: $c\"\n359\t done\n360\t else\n361\t # Remote first — can be coming up while local restarts\n362\t start_containers\n363\t # Local next\n364\t start_local_containers\n365\t fi\n366\tfi\n367\t\n368\t# ==============================================================================================\n369\t# ━━━ Remote Restart (dirty sync profiles) ━━━\n370\t# ==============================================================================================\n371\t# For dirty sync profiles (critical-fallback, emby-fallback) — restart containers on remote\n372\t# that were running before sync so they pick up config changes from the dirty sync window.\n373\t# Was running → restart. Was stopped → leave stopped.\n374\tif [[ ${#REMOTE_RESTART_CONTAINERS[@]} -gt 0 ]]; then\n375\t echo \"\"\n376\t echo \"━━━ $ICON_START $ICON_CONTAINERS Remote Restart (post dirty sync) ━━━\"\n377\t log \"Restarting configured containers on $REMOTE_SERVER_NAME...\"\n378\t\n379\t for container in \"${REMOTE_RESTART_CONTAINERS[@]}\"; do\n380\t [[ -z \"$container\" ]] && continue\n381\t\n382\t # Check if container was running before sync (still tracked via RUNNING_CONTAINERS)\n383\t WAS_RUNNING=false\n384\t for prev in \"${RUNNING_CONTAINERS[@]:-}\"; do\n385\t [[ \"$prev\" == \"$container\" ]] && WAS_RUNNING=true && break\n386\t done\n387\t\n388\t if [[ \"$WAS_RUNNING\" == false ]]; then\n389\t # Not in stop list — check current remote state\n390\t REMOTE_STATUS=$(timeout 15 ssh -i \"$SSH_KEY\" \\\n391\t -o ConnectTimeout=10 root@\"$REMOTE_SERVER\" \\\n392\t \"docker inspect -f '{{.State.Running}}' $container 2>/dev/null\" 2>/dev/null)\n393\t [[ \"$REMOTE_STATUS\" != \"true\" ]] && \\\n394\t log \"$container not running on $REMOTE_SERVER_NAME — skipping remote restart\" && \\\n395\t continue\n396\t fi\n397\t\n398\t if [[ \"$DRY_RUN\" == true ]]; then\n399\t warn \"DRY RUN — would restart $container on $REMOTE_SERVER_NAME\"\n400\t continue\n401\t fi\n402\t\n403\t timeout 15 ssh -i \"$SSH_KEY\" -o ConnectTimeout=10 root@\"$REMOTE_SERVER\" \\\n404\t \"docker restart $container\" >/dev/null 2>&1 && \\\n405\t log \"$ICON_STARTED $container restarted on $REMOTE_SERVER_NAME ✅\" || \\\n406\t warn \"Failed to restart $container on $REMOTE_SERVER_NAME\"\n407\t done\n408\tfi\n409\t\n410\tEND=$(date +%s)\n411\tDURATION=$(( END - START ))\n412\t\n413\t# ==============================================================================================\n414\t# ━━━ Bandwidth Logging ━━━\n415\t# ==============================================================================================\n416\t# Logs to bandwidth_monitor.sh — new format includes bytes transferred and warn flag.\n417\t# Only logs on actual runs (not dry-run) and only when bandwidth_monitor.sh exists.\n418\tBANDWIDTH_MONITOR=\"$SCRIPT_DIR/../Monitors/bandwidth_monitor.sh\"\n419\t\n420\tif [[ \"$DRY_RUN\" == false ]] && [[ -f \"$BANDWIDTH_MONITOR\" ]]; then\n421\t STATUS=\"success\"\n422\t [[ \"$RSYNC_SUCCESS\" == false ]] && STATUS=\"failed\"\n423\t bash \"$BANDWIDTH_MONITOR\" --log-transfer \\\n424\t \"$PROFILE_NAME\" \"$DURATION\" \"$STATUS\" \"$BYTES_TRANSFERRED\"\n425\t log \"$ICON_BANDWIDTH Transfer logged to bandwidth monitor ($BYTES_TRANSFERRED bytes)\"\n426\tfi\n427\t\n428\t# ==============================================================================================\n429\t# ━━━ Summary ━━━\n430\t# ==============================================================================================\n431\techo \"\"\n432\techo \"━━━━━ $ICON_SUMMARY RSYNC SUMMARY ━━━━━\"\n433\techo \"$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)\"\n434\techo \"$ICON_RUN Directory: $DIRECTORY\"\n435\techo \"$ICON_GEAR Profile: $PROFILE_NAME\"\n436\techo \"$ICON_TIME Duration: $(format_duration $DURATION)\"\n437\t[[ \"$BYTES_TRANSFERRED\" -gt 0 ]] && \\\n438\t echo \"$ICON_BANDWIDTH Transferred: $(awk \"BEGIN {printf \\\"%.2fGB\\\", $BYTES_TRANSFERRED / 1073741824}\")\"\n439\t\n440\tif [[ \"$DRY_RUN\" == true ]]; then\n441\t warn \"DRY RUN — no changes made\"\n442\telif [[ \"$RSYNC_SUCCESS\" == true ]]; then\n443\t echo \"$ICON_DONE Status: $ICON_SUCCESS DONE\"\n444\telse\n445\t echo \"$ICON_ERROR Status: FAILED after $RETRY_COUNT attempts\"\n446\t notify \"Rsync FAILED — $DIRECTORY ($PROFILE_NAME) after $RETRY_COUNT attempts on $(hostname)\" \\\n447\t \"Rsync\" \"warning\"\n448\tfi\n449\techo \"━━━━━━━━━━━━━━━━━━━━━━━\"\n450\t\n451\t# Flush tee and preserve log for UI — close both ends of the pipe so tee gets EOF\n452\texec 1>&- 2>&-; wait\n453\tcp \"$VV_LIVE_LOG\" \"$VV_LAST_LOG\" 2>/dev/null\n454\trm -f \"$VV_LIVE_LOG\"\n455\t\n456\t[[ \"$RSYNC_SUCCESS\" == false ]] && [[ \"$DRY_RUN\" == false ]] && exit 1\n457\texit 0"}]},"uuid":"dfb9c9ac-4107-4a4d-b59d-87cd07dbd456","timestamp":"2026-06-14T05:32:12.327Z","sourceToolAssistantUUID":"dac09318-f367-414d-aba2-516a01211130","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"92eb9a40-7b8b-4827-86f3-a145f2e85e0c","isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_011jFiq3uJ9e54rnB8WebUhX","type":"tool_result","content":"1\t#!/bin/bash\n2\t# ==============================================================================================\n3\t# ================================= Fallback ===================================================\n4\t# ==============================================================================================\n5\t#\n6\t# PURPOSE\n7\t# ─────────────────────────────────────────────────────────────────────────────\n8\t# Mutual container fallback between two unRAID servers. Runs continuously as a\n9\t# background process — started at array start by array_started.sh. Each server\n10\t# runs this script independently with no coordination between servers.\n11\t#\n12\t# All decisions are based solely on two pings per cycle: remote Tailscale IP\n13\t# reachable + internet reachable. No SSH signaling, no shared state, no election\n14\t# algorithm. Each server acts entirely from its own network perspective.\n15\t#\n16\t# Never requires human intervention during normal fallback and handback.\n17\t# Stop only via `fallback.sh --stop` — do NOT kill directly (state file may corrupt).\n18\t#\n19\t# ==============================================================================================\n20\t# OPERATIONAL MODEL\n21\t# ==============================================================================================\n22\t#\n23\t# Each cycle (every FALLBACK_CHECK_INTERVAL seconds):\n24\t# 1. Ping remote Tailscale IP — is the other server reachable?\n25\t# 2. Ping internet (EXTERNAL_IP) — do I have internet?\n26\t# 3. Determine state from the two results\n27\t# 4. Take the action for that state\n28\t#\n29\t# States:\n30\t# NORMAL — remote up, internet up — own containers, own DDNS on, silent\n31\t# FALLBACK — remote down, internet up — start tier containers, DDNS over\n32\t# NO_INTERNET — internet down — stop own DDNS immediately, wait\n33\t# DARK — remote down AND internet down — same actions as NO_INTERNET\n34\t#\n35\t# FALLBACK escalation (tiered by outage duration):\n36\t# Tier 1 — immediate — vital services + Live TV (cannot wait)\n37\t# Tier 2 — after HOST*_TIER2_DELAY (default 4hr) — shared productivity\n38\t# Tier 3 — after HOST*_TIER3_DELAY (default 12hr) — secondary services\n39\t# Tier 4 — after HOST*_TIER4_DELAY (default 24hr) — arrs + downloaders\n40\t#\n41\t# Handback (when remote returns after FALLBACK):\n42\t# 1. Strike confirmation — FALLBACK_HANDBACK_STRIKES consecutive remote-up checks\n43\t# 2. Pre-flight checks — version parity, remote array, remote Docker daemon\n44\t# 3. Staged reverse: Tier 4→3→2 — Emby stays on covering server throughout\n45\t# Each tier: stop local → rsync writeback → start on remote\n46\t# 4. DDNS handoff — stop remote DDNS before Tier 1 goes down\n47\t# 5. Tier 1 handback — stop local vital services, rsync writeback, start remote\n48\t# 6. Start remote DDNS — DNS cuts back ONLY after containers confirmed running\n49\t# 7. Return to NORMAL\n50\t#\n51\t# ==============================================================================================\n52\t# DESIGN PRINCIPLES\n53\t# ==============================================================================================\n54\t#\n55\t# Pure Ping-Only Detection\n56\t# No SSH signaling, no election algorithm, no shared state between servers.\n57\t# Each server makes all decisions from its own vantage point. A server that\n58\t# cannot reach the remote and has internet acts — no coordination needed.\n59\t#\n60\t# DDNS Absolute Control\n61\t# This script is the sole authority over DDNS containers. Network state\n62\t# returning is not permission to start DDNS — only the completion of the\n63\t# full handback sequence grants that. Eliminates the split-brain DNS window\n64\t# where two servers update the same domain with different IPs simultaneously.\n65\t#\n66\t# Staged Reverse-Tier Handback\n67\t# Emby stays on the covering server serving users while Tiers 4→3→2 hand\n68\t# back. The user impact window (Emby down) is only the Tier 1 rsync\n69\t# duration — typically minutes. Users are served continuously until the\n70\t# final step.\n71\t#\n72\t# Strike Confirmation\n73\t# Handback requires FALLBACK_HANDBACK_STRIKES consecutive remote-up checks.\n74\t# Brief network recovery during an ongoing outage would otherwise trigger a\n75\t# failed handback: containers stop on the covering server, remote goes down\n76\t# again mid-rsync. Strikes prevent this.\n77\t#\n78\t# MY_ID-Based Routing\n79\t# All tier arrays, DDNS lists, and writeback paths are selected via MY_ID\n80\t# set by detect_hosts() — not hostname string comparison. The same script\n81\t# and same config file handle both directions symmetrically.\n82\t#\n83\t# ==============================================================================================\n84\t# OPERATIONAL SAFEGUARDS\n85\t# ==============================================================================================\n86\t#\n87\t# Root Enforcement\n88\t# Docker and SSH operations require root.\n89\t#\n90\t# FALLBACK_ENABLED Gate\n91\t# Exits cleanly when disabled — safe to run on servers being rebuilt without\n92\t# triggering spurious fallback actions.\n93\t#\n94\t# Version Parity Check\n95\t# Refuses handback if remote unRAID version doesn't match. A mismatch may\n96\t# mean the remote is not fully ready or on an incompatible version.\n97\t#\n98\t# Remote Docker Daemon Check\n99\t# SSH-checks the remote Docker daemon is responding before any remote commands.\n100\t# Reachable on Tailscale does not mean Docker is ready.\n101\t#\n102\t# Container Verification\n103\t# After every local_start(), verifies the container came up and stayed running.\n104\t# Catches crashes-on-start before declaring fallback successful.\n105\t#\n106\t# Timeout Protection\n107\t# All docker commands (DOCKER_TIMEOUT=15s) and SSH commands (SSH_TIMEOUT=10s)\n108\t# wrapped in timeouts. A hung daemon does not block the state machine.\n109\t#\n110\t# Single Instance Lock\n111\t# acquire_lock() prevents two instances running simultaneously. Both would\n112\t# modify containers and DDNS independently — conflict is certain.\n113\t#\n114\t# Host Detection\n115\t# detect_hosts() resolves MY_ID / REMOTE_ID from master.conf at startup.\n116\t# Exits if the host cannot be identified — prevents running on an unknown machine.\n117\t#\n118\t# Partnership Gate\n119\t# When FALLBACK_PARTNERSHIP_REQUIRED=true, fallback is only permitted when the\n120\t# local partnership DB reports an active partnership. If partnership goes inactive,\n121\t# a grace timer starts. After FALLBACK_PARTNERSHIP_SUSPEND_AFTER minutes the loop\n122\t# suspends: no new FALLBACK is entered, and any live fallback containers are\n123\t# stopped cleanly. Resumes automatically when partnership becomes active again.\n124\t# Set FALLBACK_PARTNERSHIP_REQUIRED=false to run fallback without a partnership.\n125\t#\n126\t# Silent by Default\n127\t# State transitions: warn() — always visible. Healthy routine cycles: log()\n128\t# — suppressed unless --log. Produces no output on clean cycles.\n129\t#\n130\t# ==============================================================================================\n131\t# STATE FILES\n132\t# ==============================================================================================\n133\t#\n134\t# FALLBACK_STATE_FILE — /boot/config/fallback_state.db (survives reboots)\n135\t# Keys: state, fallback_start, handback_strikes, tier2_started,\n136\t# tier3_started, tier4_started, partnership_suspended, partner_lost_at.\n137\t# Lives on /boot/ intentionally — if the server was in FALLBACK when it\n138\t# rebooted, it resumes FALLBACK on restart.\n139\t#\n140\t# ==============================================================================================\n141\t# CONFIGURATION\n142\t# ==============================================================================================\n143\t#\n144\t# host*.conf\n145\t#\n146\t# HOST*_DDNS_CONTAINERS\n147\t# DDNS containers this host manages — stopped on internet loss, started\n148\t# as the last step of handback\n149\t#\n150\t# FALLBACK_HOST*_STOP_ON_NO_NET\n151\t# Containers stopped when this host loses internet\n152\t#\n153\t# FALLBACK_HOST*_COVERS_HOST*_TIER1–4\n154\t# Containers this host starts for the remote when remote is down, by tier.\n155\t# Variable pattern: FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}\n156\t#\n157\t# HOST*_TIER2_DELAY / TIER3_DELAY / TIER4_DELAY\n158\t# Minutes after FALLBACK entry before activating each tier.\n159\t# References the remote host's ID: HOST1_TIER2_DELAY for HOST1 outage.\n160\t# (defaults: 240 / 720 / 1440)\n161\t#\n162\t# HOST*_TIER1_WRITEBACK_DELAY\n163\t# Minimum outage minutes before Tier 1 writeback runs. Skip for short\n164\t# outages where primary state is cleaner. (default: 60)\n165\t#\n166\t# FALLBACK_HOST*_WRITEBACK_TIER1–4\n167\t# Paths rsynced back to remote on handback, per tier.\n168\t# Tier 4 auto-uses HOST*_DAILY_SYNC_SHARES — no separate list needed.\n169\t# Variable pattern: FALLBACK_${REMOTE_ID}_WRITEBACK_TIER${N}\n170\t#\n171\t# master.conf\n172\t#\n173\t# FALLBACK_ENABLED\n174\t# false = exit cleanly (e.g. remote server being rebuilt). (default: false)\n175\t#\n176\t# FALLBACK_PARTNERSHIP_REQUIRED\n177\t# true = fallback is gated on an active partnership — suspends when no active\n178\t# partner is found after FALLBACK_PARTNERSHIP_SUSPEND_AFTER minutes.\n179\t# false = fallback runs standalone regardless of partnership state. (default: true)\n180\t#\n181\t# FALLBACK_PARTNERSHIP_SUSPEND_AFTER\n182\t# Minutes without an active partnership before fallback suspends itself.\n183\t# Grace period: fallback continues normally during this window.\n184\t# 0 = suspend immediately when partnership goes inactive. (default: 120)\n185\t#\n186\t# FALLBACK_CHECK_INTERVAL\n187\t# Seconds between connectivity checks. (default: 30)\n188\t#\n189\t# FALLBACK_HANDBACK_STRIKES\n190\t# Consecutive remote-up checks required before handback begins. (default: 3)\n191\t#\n192\t# FALLBACK_STATE_FILE\n193\t# State file path — /boot/config/fallback_state.db — survives reboots.\n194\t#\n195\t# FALLBACK_RSYNC_ENABLED\n196\t# Gate for writeback rsync jobs during handback. (default: true)\n197\t#\n198\t# EXTERNAL_IP\n199\t# IP pinged for internet check. (default: 8.8.8.8)\n200\t#\n201\t# ==============================================================================================\n202\t# RUNTIME MODES\n203\t# ==============================================================================================\n204\t#\n205\t# fallback.sh\n206\t# Normal start — continuous loop. Started by array_started.sh at array start.\n207\t#\n208\t# fallback.sh --stop\n209\t# Gracefully stop the running instance (SIGTERM → wait 10s → SIGKILL).\n210\t# Used by array_stopping.sh during array shutdown.\n211\t#\n212\t# fallback.sh --dry-run\n213\t# Preview state changes without starting or stopping containers or DDNS.\n214\t#\n215\t# fallback.sh --status\n216\t# Show current state, identity, DDNS containers, outage duration if in\n217\t# FALLBACK, tier flags, strike counter. Then exit.\n218\t#\n219\t# fallback.sh --log\n220\t# Verbose output on every decision in every cycle.\n221\t#\n222\t# To stop: use `fallback.sh --stop`. Do NOT kill -9 — state file may corrupt if mid-write.\n223\t#\n224\t# ==============================================================================================\n225\t\n226\tSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n227\t\n228\tsource \"$SCRIPT_DIR/../load_config.sh\"\n229\t\n230\tparse_args \"$@\"\n231\t\n232\t# ── Stop mode — runs before acquire_lock so we can target the holding instance ────────────────\n233\tif [[ \" ${PARSED_ARGS[*]:-} \" == *\" --stop \"* ]]; then\n234\t LOCKFILE=\"${LOCK_DIR}/fallback.lock\"\n235\t if [[ ! -f \"$LOCKFILE\" ]]; then\n236\t log \"No fallback.sh lock found — not running\"\n237\t exit 0\n238\t fi\n239\t lock_content=$(cat \"$LOCKFILE\" 2>/dev/null)\n240\t target_pid=\"${lock_content%%:*}\"\n241\t if [[ -z \"$target_pid\" ]] || ! kill -0 \"$target_pid\" 2>/dev/null; then\n242\t warn \"Stale lock — fallback.sh not running (PID $target_pid gone) — clearing\"\n243\t rm -f \"$LOCKFILE\"\n244\t exit 0\n245\t fi\n246\t warn \"Stopping fallback.sh (PID $target_pid)...\"\n247\t kill -TERM \"$target_pid\" 2>/dev/null || true\n248\t waited=0\n249\t while kill -0 \"$target_pid\" 2>/dev/null && [[ \"$waited\" -lt 10 ]]; do\n250\t sleep 1\n251\t (( waited++ ))\n252\t done\n253\t if kill -0 \"$target_pid\" 2>/dev/null; then\n254\t warn \"SIGTERM ignored — sending SIGKILL to fallback.sh (PID $target_pid)\"\n255\t kill -KILL \"$target_pid\" 2>/dev/null || true\n256\t sleep 2\n257\t if kill -0 \"$target_pid\" 2>/dev/null; then\n258\t error \"Failed to kill fallback.sh (PID $target_pid)\"\n259\t exit 1\n260\t fi\n261\t warn \"Force-stopped: fallback.sh (PID $target_pid) ✅\"\n262\t else\n263\t warn \"Stopped: fallback.sh (PID $target_pid) ✅\"\n264\t fi\n265\t exit 0\n266\tfi\n267\t\n268\t# ==============================================================================================\n269\t# ━━━ Setup ━━━\n270\t# ==============================================================================================\n271\t\n272\tif [[ \"$EUID\" -ne 0 ]]; then\n273\t error \"Must be run as root\"\n274\t exit 1\n275\tfi\n276\t\n277\t# FALLBACK_ENABLED gate — exits cleanly when disabled (e.g. HOST2 being rebuilt)\n278\tif [[ \"${FALLBACK_ENABLED:-false}\" == false ]]; then\n279\t warn \"FALLBACK_ENABLED=false — fallback monitoring disabled\"\n280\t warn \"Set FALLBACK_ENABLED=true in master.conf when both servers are ready\"\n281\t exit 0\n282\tfi\n283\t\n284\tacquire_lock # strict single instance — notifies on collision\n285\t\n286\tif ! command -v docker &>/dev/null; then\n287\t error \"Docker command not found\"\n288\t exit 1\n289\tfi\n290\t\n291\tdetect_hosts\n292\tresolve_remote_ip\n293\t\n294\t# Validate unRAID notify script — used throughout for state change notifications\n295\t\n296\t[[ \"$DRY_RUN\" == true ]] && warn \"DRY RUN — no container or DDNS changes will be made\"\n297\t\n298\t# Timeout for all docker and SSH docker commands\n299\tDOCKER_TIMEOUT=15\n300\tSSH_TIMEOUT=10\n301\tCONTAINER_VERIFY_WAIT=5 # seconds after start before verifying container is up\n302\t\n303\tlog \"$ICON_GEAR Config: check-interval=${FALLBACK_CHECK_INTERVAL}s handback-strikes=${FALLBACK_HANDBACK_STRIKES} partnership-required=${FALLBACK_PARTNERSHIP_REQUIRED:-true} rsync-enabled=${FALLBACK_RSYNC_ENABLED:-true}\"\n304\tlog \"$ICON_GEAR Timeouts: docker=${DOCKER_TIMEOUT}s ssh=${SSH_TIMEOUT}s verify-wait=${CONTAINER_VERIFY_WAIT}s\"\n305\t\n306\t# ==============================================================================================\n307\t# ── STATE FILE HELPERS ────────────────────────────────────────────────────────────────────────\n308\t# ==============================================================================================\n309\t# State file on /boot/config — survives reboots.\n310\t# Format: key=value one per line.\n311\t# Keys: state, fallback_start, handback_strikes, tier2_started, tier3_started, tier4_started\n312\t\n313\tstate_get() {\n314\t grep \"^${1}=\" \"$FALLBACK_STATE_FILE\" 2>/dev/null | cut -d= -f2\n315\t}\n316\t\n317\tstate_set() {\n318\t local key=\"$1\" value=\"$2\"\n319\t if grep -q \"^${key}=\" \"$FALLBACK_STATE_FILE\" 2>/dev/null; then\n320\t # Escape sed replacement metacharacters: \\ first, then & and |\n321\t local safe=\"${value//\\\\/\\\\\\\\}\"; safe=\"${safe//&/\\\\&}\"; safe=\"${safe//|/\\\\|}\"\n322\t sed -i \"s|^${key}=.*|${key}=${safe}|\" \"$FALLBACK_STATE_FILE\"\n323\t else\n324\t echo \"${key}=${value}\" >> \"$FALLBACK_STATE_FILE\"\n325\t fi\n326\t}\n327\t\n328\tstate_init() {\n329\t mkdir -p \"$(dirname \"$FALLBACK_STATE_FILE\")\"\n330\t [[ ! -f \"$FALLBACK_STATE_FILE\" ]] && touch \"$FALLBACK_STATE_FILE\"\n331\t [[ -z \"$(state_get state)\" ]] && state_set state \"NORMAL\"\n332\t [[ -z \"$(state_get fallback_start)\" ]] && state_set fallback_start \"0\"\n333\t [[ -z \"$(state_get handback_strikes)\" ]] && state_set handback_strikes \"0\"\n334\t [[ -z \"$(state_get tier2_started)\" ]] && state_set tier2_started \"false\"\n335\t [[ -z \"$(state_get tier3_started)\" ]] && state_set tier3_started \"false\"\n336\t [[ -z \"$(state_get tier4_started)\" ]] && state_set tier4_started \"false\"\n337\t [[ -z \"$(state_get partnership_suspended)\" ]] && state_set partnership_suspended \"false\"\n338\t [[ -z \"$(state_get partner_lost_at)\" ]] && state_set partner_lost_at \"0\"\n339\t}\n340\t\n341\t# ==============================================================================================\n342\t# ── PARTNERSHIP GATE HELPERS ──────────────────────────────────────────────────────────────────\n343\t# ==============================================================================================\n344\t\n345\t# Returns 0 (true) if the local partnership DB reports an ACTIVE partnership.\n346\t# Reads /boot/config/partnership_.db — no subprocess, no SSH.\n347\tcheck_partnership_active() {\n348\t local state_file=\"${STATE_DIR}/partnership_${LOCAL_SERVER_NAME}.db\"\n349\t local state\n350\t state=$(grep \"^state=\" \"$state_file\" 2>/dev/null | cut -d= -f2)\n351\t [[ \"$state\" == \"ACTIVE\" ]]\n352\t}\n353\t\n354\t# Stop all tiers that were started during a fallback and clear their state flags.\n355\t# Called when the partnership gate suspends an in-progress fallback.\n356\t_abort_fallback_containers() {\n357\t for tier in 4 3 2 1; do\n358\t [[ \"$(state_get \"tier${tier}_started\")\" != \"true\" ]] && continue\n359\t warn \"Aborting Tier $tier containers — partnership suspended\"\n360\t local containers\n361\t read -r -a containers <<< \"$(get_tier_containers \"$tier\")\"\n362\t for container in \"${containers[@]}\"; do\n363\t [[ -n \"$container\" ]] && local_stop \"$container\"\n364\t done\n365\t state_set \"tier${tier}_started\" \"false\"\n366\t done\n367\t}\n368\t\n369\t# ==============================================================================================\n370\t# ── CONTAINER HELPERS ─────────────────────────────────────────────────────────────────────────\n371\t# ==============================================================================================\n372\t# All docker commands wrapped in DOCKER_TIMEOUT.\n373\t# All SSH docker commands wrapped in SSH_TIMEOUT.\n374\t# Verification after start — fallback is critical, confirm containers came up.\n375\t\n376\t# Start a container locally and verify it came up\n377\tlocal_start() {\n378\t local container=\"$1\"\n379\t [[ -z \"$container\" ]] && return\n380\t local status\n381\t status=$(timeout \"$DOCKER_TIMEOUT\" docker inspect -f '{{.State.Running}}' \\\n382\t \"$container\" 2>/dev/null)\n383\t if [[ \"$status\" == \"true\" ]]; then\n384\t log \"$container already running locally\"\n385\t return 0\n386\t fi\n387\t log \"$ICON_START Starting $container locally...\"\n388\t if [[ \"$DRY_RUN\" == true ]]; then\n389\t warn \"DRY RUN — would start $container locally\"\n390\t return 0\n391\t fi\n392\t if timeout \"$DOCKER_TIMEOUT\" docker start \"$container\" >/dev/null 2>&1; then\n393\t sleep \"$CONTAINER_VERIFY_WAIT\"\n394\t local post_status\n395\t post_status=$(timeout \"$DOCKER_TIMEOUT\" docker inspect -f '{{.State.Running}}' \\\n396\t \"$container\" 2>/dev/null)\n397\t if [[ \"$post_status\" == \"true\" ]]; then\n398\t log \"$ICON_STARTED $container started and running ✅\"\n399\t return 0\n400\t else\n401\t error \"$container started but crashed immediately\"\n402\t notify \"$container failed to stay running during fallback on $(hostname)\" \\\n403\t \"Fallback\" \"warning\"\n404\t return 1\n405\t fi\n406\t else\n407\t error \"Failed to start $container locally\"\n408\t notify \"Failed to start $container locally during fallback on $(hostname)\" \\\n409\t \"Fallback\" \"warning\"\n410\t return 1\n411\t fi\n412\t}\n413\t\n414\t# Stop a container locally\n415\tlocal_stop() {\n416\t local container=\"$1\"\n417\t [[ -z \"$container\" ]] && return\n418\t local status\n419\t status=$(timeout \"$DOCKER_TIMEOUT\" docker inspect -f '{{.State.Running}}' \\\n420\t \"$container\" 2>/dev/null)\n421\t if [[ \"$status\" != \"true\" ]]; then\n422\t log \"$container already stopped locally\"\n423\t return 0\n424\t fi\n425\t log \"$ICON_STOP Stopping $container locally...\"\n426\t if [[ \"$DRY_RUN\" == true ]]; then\n427\t warn \"DRY RUN — would stop $container locally\"\n428\t return 0\n429\t fi\n430\t timeout \"$DOCKER_TIMEOUT\" docker stop \"$container\" >/dev/null 2>&1 && \\\n431\t log \"$ICON_STOPPED $container stopped\" || \\\n432\t error \"Failed to stop $container locally\"\n433\t}\n434\t\n435\t# Start a container on remote via SSH and verify it came up\n436\tremote_start() {\n437\t local container=\"$1\"\n438\t [[ -z \"$container\" ]] && return\n439\t local status\n440\t status=$(timeout \"$SSH_TIMEOUT\" ssh -i \"$SSH_KEY\" -o ConnectTimeout=\"$SSH_TIMEOUT\" \\\n441\t root@\"$REMOTE_SERVER\" \\\n442\t \"timeout $DOCKER_TIMEOUT docker inspect -f '{{.State.Running}}' \\\n443\t $container 2>/dev/null\" 2>/dev/null)\n444\t if [[ \"$status\" == \"true\" ]]; then\n445\t log \"$container already running on $REMOTE_SERVER_NAME\"\n446\t return 0\n447\t fi\n448\t log \"$ICON_START Starting $container on $REMOTE_SERVER_NAME...\"\n449\t if [[ \"$DRY_RUN\" == true ]]; then\n450\t warn \"DRY RUN — would start $container on $REMOTE_SERVER_NAME\"\n451\t return 0\n452\t fi\n453\t if timeout \"$SSH_TIMEOUT\" ssh -i \"$SSH_KEY\" -o ConnectTimeout=\"$SSH_TIMEOUT\" \\\n454\t root@\"$REMOTE_SERVER\" \\\n455\t \"timeout $DOCKER_TIMEOUT docker start $container\" >/dev/null 2>&1; then\n456\t sleep \"$CONTAINER_VERIFY_WAIT\"\n457\t local post_status\n458\t post_status=$(timeout \"$SSH_TIMEOUT\" ssh -i \"$SSH_KEY\" \\\n459\t -o ConnectTimeout=\"$SSH_TIMEOUT\" root@\"$REMOTE_SERVER\" \\\n460\t \"timeout $DOCKER_TIMEOUT docker inspect -f '{{.State.Running}}' \\\n461\t $container 2>/dev/null\" 2>/dev/null)\n462\t if [[ \"$post_status\" == \"true\" ]]; then\n463\t log \"$ICON_STARTED $container started on $REMOTE_SERVER_NAME ✅\"\n464\t return 0\n465\t else\n466\t error \"$container started on $REMOTE_SERVER_NAME but crashed immediately\"\n467\t notify \"$container failed after start on $REMOTE_SERVER_NAME during handback on $(hostname)\" \\\n468\t \"Fallback\" \"warning\"\n469\t return 1\n470\t fi\n471\t else\n472\t error \"Failed to start $container on $REMOTE_SERVER_NAME\"\n473\t notify \"Failed to start $container on $REMOTE_SERVER_NAME during handback on $(hostname)\" \\\n474\t \"Fallback\" \"warning\"\n475\t return 1\n476\t fi\n477\t}\n478\t\n479\t# Stop a container on remote via SSH\n480\tremote_stop() {\n481\t local container=\"$1\"\n482\t [[ -z \"$container\" ]] && return\n483\t local status\n484\t status=$(timeout \"$SSH_TIMEOUT\" ssh -i \"$SSH_KEY\" -o ConnectTimeout=\"$SSH_TIMEOUT\" \\\n485\t root@\"$REMOTE_SERVER\" \\\n486\t \"timeout $DOCKER_TIMEOUT docker inspect -f '{{.State.Running}}' \\\n487\t $container 2>/dev/null\" 2>/dev/null)\n488\t if [[ \"$status\" != \"true\" ]]; then\n489\t log \"$container already stopped on $REMOTE_SERVER_NAME\"\n490\t return 0\n491\t fi\n492\t log \"$ICON_STOP Stopping $container on $REMOTE_SERVER_NAME...\"\n493\t if [[ \"$DRY_RUN\" == true ]]; then\n494\t warn \"DRY RUN — would stop $container on $REMOTE_SERVER_NAME\"\n495\t return 0\n496\t fi\n497\t timeout \"$SSH_TIMEOUT\" ssh -i \"$SSH_KEY\" -o ConnectTimeout=\"$SSH_TIMEOUT\" \\\n498\t root@\"$REMOTE_SERVER\" \\\n499\t \"timeout $DOCKER_TIMEOUT docker stop $container\" >/dev/null 2>&1 && \\\n500\t log \"$ICON_STOPPED $container stopped on $REMOTE_SERVER_NAME\" || \\\n501\t error \"Failed to stop $container on $REMOTE_SERVER_NAME\"\n502\t}\n503\t\n504\t# ==============================================================================================\n505\t# ── DDNS HELPERS ──────────────────────────────────────────────────────────────────────────────\n506\t# ==============================================================================================\n507\t# DDNS managed exclusively by this script — never by network state alone.\n508\t# Stopping DDNS on internet loss prevents split-brain DNS updates.\n509\t\n510\tlocal_ddns_stop() {\n511\t warn \"$ICON_NET Stopping local DDNS — internet lost\"\n512\t for container in \"${LOCAL_DDNS_CONTAINERS[@]}\"; do\n513\t local_stop \"$container\"\n514\t done\n515\t}\n516\t\n517\tlocal_ddns_start() {\n518\t warn \"$ICON_NET Starting local DDNS — handback complete, containers confirmed up\"\n519\t for container in \"${LOCAL_DDNS_CONTAINERS[@]}\"; do\n520\t local_start \"$container\"\n521\t done\n522\t}\n523\t\n524\tremote_ddns_stop() {\n525\t warn \"$ICON_NET Stopping remote DDNS on $REMOTE_SERVER_NAME — prevents split-brain during rsync\"\n526\t for container in \"${REMOTE_DDNS_CONTAINERS[@]}\"; do\n527\t remote_stop \"$container\"\n528\t done\n529\t}\n530\t\n531\t# ==============================================================================================\n532\t# ── TIERED FALLBACK HELPERS ───────────────────────────────────────────────────────────────────\n533\t# ==============================================================================================\n534\t# All routing uses MY_ID — not hostname string comparison.\n535\t# MY_ID set by detect_hosts() — \"HOST1\" or \"HOST2\" etc.\n536\t# \"This server covers the other\" — MY_ID selects which tier arrays to use.\n537\t\n538\tget_tier_containers() {\n539\t local tier=\"$1\"\n540\t local remote_id=\"${REMOTE_ID}\"\n541\t local var_name=\"FALLBACK_${MY_ID}_COVERS_${remote_id}_TIER${tier}\"\n542\t eval \"echo \\\"\\${${var_name}[@]:-}\\\"\"\n543\t}\n544\t\n545\tget_tier2_delay() {\n546\t local var_name=\"${REMOTE_ID}_TIER2_DELAY\"\n547\t echo \"${!var_name:-240}\"\n548\t}\n549\t\n550\tget_tier3_delay() {\n551\t local var_name=\"${REMOTE_ID}_TIER3_DELAY\"\n552\t echo \"${!var_name:-720}\"\n553\t}\n554\t\n555\tget_tier4_delay() {\n556\t local var_name=\"${REMOTE_ID}_TIER4_DELAY\"\n557\t echo \"${!var_name:-1440}\"\n558\t}\n559\t\n560\tget_tier1_writeback_delay() {\n561\t local var_name=\"${REMOTE_ID}_TIER1_WRITEBACK_DELAY\"\n562\t echo \"${!var_name:-60}\"\n563\t}\n564\t\n565\tget_writeback_jobs_for_tier() {\n566\t local tier=\"$1\"\n567\t local remote_id=\"${REMOTE_ID}\"\n568\t if [[ \"$tier\" -eq 4 ]]; then\n569\t # Tier 4 — daily sync shares of the remote + any extra writeback paths\n570\t local shares_var=\"${remote_id}_DAILY_SYNC_SHARES\"\n571\t local extra_var=\"FALLBACK_${remote_id}_WRITEBACK_TIER4\"\n572\t eval \"echo \\\"\\${${shares_var}[@]:-} \\${${extra_var}[@]:-}\\\"\"\n573\t else\n574\t local var_name=\"FALLBACK_${remote_id}_WRITEBACK_TIER${tier}\"\n575\t eval \"echo \\\"\\${${var_name}[@]:-}\\\"\"\n576\t fi\n577\t}\n578\t\n579\t# Select DDNS arrays based on MY_ID\n580\tset_ddns_arrays() {\n581\t local local_ddns_var=\"${MY_ID}_DDNS_CONTAINERS\"\n582\t local remote_ddns_var=\"${REMOTE_ID}_DDNS_CONTAINERS\"\n583\t eval \"LOCAL_DDNS_CONTAINERS=(\\\"\\${${local_ddns_var}[@]:-}\\\")\"\n584\t eval \"REMOTE_DDNS_CONTAINERS=(\\\"\\${${remote_ddns_var}[@]:-}\\\")\"\n585\t}\n586\t\n587\t# Select internet-loss stop list based on MY_ID\n588\tget_stop_on_no_net() {\n589\t local var_name=\"FALLBACK_${MY_ID}_STOP_ON_NO_NET\"\n590\t eval \"echo \\\"\\${${var_name}[@]:-}\\\"\"\n591\t}\n592\t\n593\t# ==============================================================================================\n594\t# ━━━ Status ━━━\n595\t# ==============================================================================================\n596\tif [[ \"$SHOW_STATUS\" == true ]]; then\n597\t state_init\n598\t set_ddns_arrays\n599\t CURRENT_STATE=$(state_get state)\n600\t FALLBACK_START_TS=$(state_get fallback_start)\n601\t TIER2=$(state_get tier2_started)\n602\t TIER3=$(state_get tier3_started)\n603\t TIER4=$(state_get tier4_started)\n604\t STRIKES=$(state_get handback_strikes)\n605\t\n606\t local_ver=$(platform_get_os_version 2>/dev/null || echo \"unknown\")\n607\t\n608\t echo \"\"\n609\t echo \"━━━━━ $ICON_SUMMARY FALLBACK STATUS ━━━━━\"\n610\t echo \"$ICON_HOST My ID: $MY_ID ($LOCAL_SERVER_NAME)\"\n611\t echo \"$ICON_HOST Remote ID: $REMOTE_ID ($REMOTE_SERVER_NAME — $REMOTE_SERVER)\"\n612\t echo \"$ICON_GEAR OS ver: $local_ver\"\n613\t echo \"$ICON_FALLBACK State: $CURRENT_STATE\"\n614\t echo \"$ICON_NET Local DDNS: ${LOCAL_DDNS_CONTAINERS[*]:-none}\"\n615\t echo \"$ICON_NET Remote DDNS: ${REMOTE_DDNS_CONTAINERS[*]:-none}\"\n616\t echo \"$ICON_TIME Interval: ${FALLBACK_CHECK_INTERVAL}s\"\n617\t echo \"$ICON_FALLBACK Strikes: $STRIKES / $FALLBACK_HANDBACK_STRIKES\"\n618\t\n619\t if [[ \"$CURRENT_STATE\" == \"FALLBACK\" && \"$FALLBACK_START_TS\" -gt 0 ]]; then\n620\t ELAPSED=$(( ($(date +%s) - FALLBACK_START_TS) / 60 ))\n621\t echo \"$ICON_TIME Outage: ${ELAPSED}min\"\n622\t echo \"$ICON_FALLBACK Tier 2: $TIER2 (delay: $(get_tier2_delay)min)\"\n623\t echo \"$ICON_FALLBACK Tier 3: $TIER3 (delay: $(get_tier3_delay)min)\"\n624\t echo \"$ICON_FALLBACK Tier 4: $TIER4 (delay: $(get_tier4_delay)min)\"\n625\t fi\n626\t\n627\t echo \"$ICON_GEAR Enabled: $FALLBACK_ENABLED\"\n628\t echo \"$ICON_GEAR Dry Run: $DRY_RUN\"\n629\t\n630\t if [[ \"${FALLBACK_PARTNERSHIP_REQUIRED:-true}\" == \"true\" ]]; then\n631\t SUSPENDED=$(state_get partnership_suspended)\n632\t PLOST=$(state_get partner_lost_at)\n633\t if [[ \"$SUSPENDED\" == \"true\" ]]; then\n634\t echo \"$ICON_GEAR Partnership: SUSPENDED (no active partnership)\"\n635\t elif [[ -n \"$PLOST\" && \"$PLOST\" != \"0\" ]]; then\n636\t PLOST_MIN=$(( ($(date +%s) - PLOST) / 60 ))\n637\t echo \"$ICON_GEAR Partnership: GRACE (${PLOST_MIN}min / ${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}min until suspend)\"\n638\t else\n639\t echo \"$ICON_GEAR Partnership: ACTIVE\"\n640\t fi\n641\t else\n642\t echo \"$ICON_GEAR Partnership: REQUIRED=false (standalone mode)\"\n643\t fi\n644\t\n645\t echo \"━━━━━━━━━━━━━━━━━━━━━━━\"\n646\t exit 0\n647\tfi\n648\t\n649\t# ==============================================================================================\n650\t# ── HANDBACK SEQUENCE ─────────────────────────────────────────────────────────────────────────\n651\t# ==============================================================================================\n652\t# Called when remote returns after FALLBACK state.\n653\t# CRITICAL sequencing — do not reorder without understanding the consequences.\n654\t# Each step must succeed before proceeding — aborts and retries next cycle on failure.\n655\t\n656\trun_handback() {\n657\t echo \"\"\n658\t echo \"━━━ $ICON_FALLBACK Handback Sequence — $(date '+%Y-%m-%d %H:%M:%S') ━━━\"\n659\t warn \"$REMOTE_SERVER_NAME has returned — beginning staged handback\"\n660\t\n661\t # ── Step 1: Pre-flight checks ────────────────────────────────────────────────────────────\n662\t echo \"\"\n663\t echo \"━━━ $ICON_SHIELD Pre-flight ━━━\"\n664\t\n665\t if ! check_os_version_parity; then\n666\t warn \"Version parity check failed — aborting handback, will retry next cycle\"\n667\t state_set handback_strikes 0\n668\t return 1\n669\t fi\n670\t\n671\t if ! check_remote_array; then\n672\t warn \"Remote array not ready — aborting handback, will retry next cycle\"\n673\t state_set handback_strikes 0\n674\t return 1\n675\t fi\n676\t\n677\t if ! check_remote_docker_daemon; then\n678\t warn \"Remote Docker daemon not ready — aborting handback, will retry next cycle\"\n679\t state_set handback_strikes 0\n680\t return 1\n681\t fi\n682\t\n683\t echo \" Pre-flight checks passed ✅\"\n684\t\n685\t local outage_minutes\n686\t outage_minutes=$(( ($(date +%s) - $(state_get fallback_start)) / 60 ))\n687\t local tier1_wb_delay t2_delay t3_delay t4_delay\n688\t tier1_wb_delay=$(get_tier1_writeback_delay)\n689\t t2_delay=$(get_tier2_delay)\n690\t t3_delay=$(get_tier3_delay)\n691\t t4_delay=$(get_tier4_delay)\n692\t\n693\t warn \"Outage duration: ${outage_minutes}min — staged handback Tier 4→3→2→1\"\n694\t\n695\t # ── Staged handback helper ────────────────────────────────────────────────────────────────\n696\t # Each tier: stop local containers → rsync writeback → start on remote.\n697\t # Skips tiers that were never activated. Emby (Tier 1) stays on fallback\n698\t # serving users until all higher tiers complete.\n699\t handback_tier() {\n700\t local tier=\"$1\" threshold=\"$2\" label=\"$3\"\n701\t\n702\t [[ \"$(state_get \"tier${tier}_started\")\" != \"true\" ]] && return 0\n703\t\n704\t echo \"\"\n705\t echo \"━━━ $ICON_FALLBACK Handback Tier $tier — $label ━━━\"\n706\t\n707\t local containers\n708\t read -r -a containers <<< \"$(get_tier_containers \"$tier\")\"\n709\t\n710\t echo \" Stopping local Tier $tier containers...\"\n711\t for container in \"${containers[@]}\"; do\n712\t [[ -n \"$container\" ]] && local_stop \"$container\"\n713\t done\n714\t\n715\t if ! check_rsync_enabled \"FALLBACK\"; then\n716\t warn \"FALLBACK_RSYNC_ENABLED=false — skipping Tier $tier writeback\"\n717\t else\n718\t local jobs\n719\t read -r -a jobs <<< \"$(get_writeback_jobs_for_tier \"$tier\")\"\n720\t if [[ ${#jobs[@]} -gt 0 && \"$outage_minutes\" -ge \"$threshold\" ]]; then\n721\t echo \" Rsync writeback — outage ${outage_minutes}min >= ${threshold}min\"\n722\t for job in \"${jobs[@]}\"; do\n723\t [[ -z \"$job\" ]] && continue\n724\t log \"Syncing: $job\"\n725\t [[ \"$DRY_RUN\" == false ]] && bash \"$SCRIPT_DIR/../Rsync/rsync.sh\" \"$job\" \\\n726\t || warn \"DRY RUN — would rsync: $job\"\n727\t done\n728\t else\n729\t log \"Tier $tier writeback skipped — outage ${outage_minutes}min < ${threshold}min\"\n730\t fi\n731\t fi\n732\t\n733\t echo \" Starting Tier $tier on $REMOTE_SERVER_NAME...\"\n734\t for container in \"${containers[@]}\"; do\n735\t [[ -n \"$container\" ]] && remote_start \"$container\"\n736\t done\n737\t\n738\t state_set \"tier${tier}_started\" \"false\"\n739\t warn \"$ICON_DONE Tier $tier handed back to $REMOTE_SERVER_NAME\"\n740\t }\n741\t\n742\t # ── Steps 2–4: Tiers 4→3→2 — Emby stays up throughout ──────────────────────────────────\n743\t handback_tier 4 \"$t4_delay\" \"media shares + edge cases\"\n744\t handback_tier 3 \"$t3_delay\" \"secondary services\"\n745\t handback_tier 2 \"$t2_delay\" \"productivity services\"\n746\t\n747\t # ── Step 5: DDNS handoff — immediately before Tier 1 goes down ──────────────────────────\n748\t # Moved here (not at start) so DNS keeps pointing at fallback during higher-tier handbacks.\n749\t # Prevents split-brain DNS during Tier 1 rsync window.\n750\t echo \"\"\n751\t echo \"━━━ $ICON_NET DDNS Handoff ━━━\"\n752\t remote_ddns_stop\n753\t sleep 5\n754\t\n755\t # ── Step 6: Tier 1 handback — vital services + Emby ─────────────────────────────────────\n756\t echo \"\"\n757\t echo \"━━━ $ICON_FALLBACK Handback Tier 1 — vital services ━━━\"\n758\t\n759\t read -r -a t1 <<< \"$(get_tier_containers 1)\"\n760\t\n761\t echo \" Stopping local Tier 1 containers...\"\n762\t for container in \"${t1[@]}\"; do\n763\t [[ -z \"$container\" ]] && continue\n764\t local is_ddns=false\n765\t for ddns in \"${REMOTE_DDNS_CONTAINERS[@]}\"; do\n766\t [[ \"$container\" == \"$ddns\" ]] && is_ddns=true && break\n767\t done\n768\t [[ \"$is_ddns\" == false ]] && local_stop \"$container\"\n769\t done\n770\t\n771\t if ! check_rsync_enabled \"FALLBACK\"; then\n772\t warn \"FALLBACK_RSYNC_ENABLED=false — skipping Tier 1 writeback\"\n773\t else\n774\t local t1_jobs\n775\t read -r -a t1_jobs <<< \"$(get_writeback_jobs_for_tier 1)\"\n776\t if [[ ${#t1_jobs[@]} -gt 0 && \"$outage_minutes\" -ge \"$tier1_wb_delay\" ]]; then\n777\t echo \" Rsync writeback — outage ${outage_minutes}min >= ${tier1_wb_delay}min\"\n778\t for job in \"${t1_jobs[@]}\"; do\n779\t [[ -z \"$job\" ]] && continue\n780\t log \"Syncing: $job\"\n781\t if [[ \"$DRY_RUN\" == false ]]; then\n782\t if [[ \"$(basename \"$job\")\" == \"Emby\" ]]; then\n783\t bash \"$SCRIPT_DIR/../Rsync/rsync.sh\" \"$job\" --profile=emby-fallback\n784\t else\n785\t bash \"$SCRIPT_DIR/../Rsync/rsync.sh\" \"$job\"\n786\t fi\n787\t else\n788\t warn \"DRY RUN — would rsync: $job\"\n789\t fi\n790\t done\n791\t else\n792\t log \"Tier 1 writeback skipped — outage ${outage_minutes}min < ${tier1_wb_delay}min\"\n793\t fi\n794\t fi\n795\t\n796\t echo \" Starting Tier 1 on $REMOTE_SERVER_NAME...\"\n797\t sleep 10 # give databases time to initialise before apps\n798\t for container in \"${t1[@]}\"; do\n799\t [[ -z \"$container\" ]] && continue\n800\t local is_ddns=false\n801\t for ddns in \"${REMOTE_DDNS_CONTAINERS[@]}\"; do\n802\t [[ \"$container\" == \"$ddns\" ]] && is_ddns=true && break\n803\t done\n804\t [[ \"$is_ddns\" == false ]] && remote_start \"$container\"\n805\t done\n806\t\n807\t # ── Step 7: DNS Cutover — final step ────────────────────────────────────────────────────\n808\t # DNS cuts over ONLY after Tier 1 containers confirmed up\n809\t echo \"\"\n810\t echo \"━━━ $ICON_NET DNS Cutover ━━━\"\n811\t sleep 5\n812\t for container in \"${REMOTE_DDNS_CONTAINERS[@]}\"; do\n813\t remote_start \"$container\"\n814\t done\n815\t warn \"$ICON_NET Remote DDNS started — DNS now points at $REMOTE_SERVER_NAME\"\n816\t\n817\t # ── Step 8: Return to NORMAL ─────────────────────────────────────────────────────────────\n818\t echo \"\"\n819\t state_set state \"NORMAL\"\n820\t state_set fallback_start \"0\"\n821\t state_set handback_strikes \"0\"\n822\t state_set tier2_started \"false\"\n823\t state_set tier3_started \"false\"\n824\t state_set tier4_started \"false\"\n825\t\n826\t # Restore local DDNS — may have been stopped during NO_INTERNET or DARK state before handback\n827\t local_ddns_start\n828\t\n829\t warn \"$ICON_DONE Handback complete — returned to NORMAL\"\n830\t notify \"Fallback handback complete on $(hostname) — $REMOTE_SERVER_NAME is back, all containers returned\" \\\n831\t \"Fallback\" \"normal\"\n832\t}\n833\t\n834\t# ==============================================================================================\n835\t# ━━━ Main State Machine ━━━\n836\t# ==============================================================================================\n837\tstate_init\n838\tset_ddns_arrays\n839\t\n840\techo \"\"\n841\techo \"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\"\n842\techo \" $ICON_FALLBACK FALLBACK — $(date '+%Y-%m-%d %H:%M:%S')\"\n843\techo \" $ICON_HOST $MY_ID ($LOCAL_SERVER_NAME) → monitoring $REMOTE_ID ($REMOTE_SERVER_NAME)\"\n844\techo \" $ICON_NET Remote IP: $REMOTE_SERVER\"\n845\techo \"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\"\n846\t\n847\tFALLBACK_RUNNING=true\n848\ttrap 'FALLBACK_RUNNING=false; warn \"Fallback received shutdown signal — stopping cleanly\"; exit 0' \\\n849\t SIGTERM SIGINT\n850\t\n851\twhile [[ \"$FALLBACK_RUNNING\" == true ]]; do\n852\t NOW=$(date +%s)\n853\t CURRENT_STATE=$(state_get state)\n854\t\n855\t # ── Connectivity checks ──────────────────────────────────────────────────────────────────\n856\t REMOTE_UP=false\n857\t INTERNET_UP=false\n858\t\n859\t ping_remote && REMOTE_UP=true\n860\t ping_internet && INTERNET_UP=true\n861\t\n862\t log \"Remote: $REMOTE_UP | Internet: $INTERNET_UP | State: $CURRENT_STATE\"\n863\t\n864\t # ── Partnership gate ──────────────────────────────────────────────────────────────────────\n865\t # When FALLBACK_PARTNERSHIP_REQUIRED=true, a fallback without an active partner is\n866\t # meaningless — who are we covering for? The gate suspends the loop after a grace\n867\t # period and aborts any live fallback containers. Clears automatically when partnership\n868\t # is restored.\n869\t if [[ \"${FALLBACK_PARTNERSHIP_REQUIRED:-true}\" == \"true\" ]]; then\n870\t if check_partnership_active; then\n871\t # Partnership healthy — clear any accumulated grace state\n872\t if [[ \"$(state_get partnership_suspended)\" == \"true\" ]]; then\n873\t warn \"Active partnership restored — re-enabling fallback monitoring\"\n874\t state_set partnership_suspended \"false\"\n875\t state_set partner_lost_at \"0\"\n876\t notify \"Fallback re-enabled on $(hostname) — active partnership restored\" \\\n877\t \"Fallback\" \"normal\"\n878\t elif [[ \"$(state_get partner_lost_at)\" != \"0\" ]]; then\n879\t log \"Partnership recovered within grace window — resetting timer\"\n880\t state_set partner_lost_at \"0\"\n881\t fi\n882\t else\n883\t PARTNER_LOST_AT=$(state_get partner_lost_at)\n884\t if [[ -z \"$PARTNER_LOST_AT\" || \"$PARTNER_LOST_AT\" == \"0\" ]]; then\n885\t warn \"No active partnership detected — grace timer started (${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}min before suspend)\"\n886\t state_set partner_lost_at \"$NOW\"\n887\t PARTNER_LOST_AT=$NOW\n888\t fi\n889\t\n890\t PARTNER_LOST_MIN=$(( (NOW - PARTNER_LOST_AT) / 60 ))\n891\t SUSPEND_AFTER=${FALLBACK_PARTNERSHIP_SUSPEND_AFTER:-120}\n892\t\n893\t if [[ \"$(state_get partnership_suspended)\" == \"true\" || \"$PARTNER_LOST_MIN\" -ge \"$SUSPEND_AFTER\" ]]; then\n894\t # Suspend — abort any live fallback containers, then skip this cycle entirely\n895\t if [[ \"$(state_get partnership_suspended)\" != \"true\" ]]; then\n896\t warn \"No active partnership for ${PARTNER_LOST_MIN}min — suspending fallback\"\n897\t state_set partnership_suspended \"true\"\n898\t notify \"Fallback suspended on $(hostname) — no active partnership for ${PARTNER_LOST_MIN}min\" \\\n899\t \"Fallback\" \"warning\"\n900\t fi\n901\t if [[ \"$CURRENT_STATE\" == \"FALLBACK\" ]]; then\n902\t warn \"Aborting in-progress fallback — partnership suspended\"\n903\t _abort_fallback_containers\n904\t state_set state \"NORMAL\"\n905\t state_set fallback_start \"0\"\n906\t state_set handback_strikes \"0\"\n907\t notify \"Fallback aborted on $(hostname) — no active partnership, containers stopped\" \\\n908\t \"Fallback\" \"warning\"\n909\t fi\n910\t log \"Partnership suspended — skipping fallback cycle\"\n911\t sleep \"$FALLBACK_CHECK_INTERVAL\" & wait $!\n912\t continue\n913\t else\n914\t log \"No active partnership — ${PARTNER_LOST_MIN}min elapsed / ${SUSPEND_AFTER}min until suspend\"\n915\t fi\n916\t fi\n917\t fi\n918\t\n919\t # ════════════════════════════════════════════════════════════════\n920\t # NORMAL STATE\n921\t # ════════════════════════════════════════════════════════════════\n922\t if [[ \"$CURRENT_STATE\" == \"NORMAL\" ]]; then\n923\t\n924\t if [[ \"$REMOTE_UP\" == true && \"$INTERNET_UP\" == true ]]; then\n925\t log \"$ICON_SUCCESS NORMAL — all systems up\"\n926\t\n927\t elif [[ \"$REMOTE_UP\" == false && \"$INTERNET_UP\" == true ]]; then\n928\t # Remote is down — enter FALLBACK immediately\n929\t echo \"\"\n930\t echo \"━━━ $ICON_FALLBACK Entering FALLBACK — $(date '+%Y-%m-%d %H:%M:%S') ━━━\"\n931\t warn \"$REMOTE_SERVER_NAME ($REMOTE_ID) is unreachable — internet is up — starting fallback\"\n932\t\n933\t state_set state \"FALLBACK\"\n934\t state_set fallback_start \"$NOW\"\n935\t state_set handback_strikes \"0\"\n936\t state_set tier2_started \"false\"\n937\t state_set tier3_started \"false\"\n938\t state_set tier4_started \"false\"\n939\t\n940\t echo \"\"\n941\t echo \"━━━ $ICON_START Tier 1 — Immediate ━━━\"\n942\t read -r -a tier1 <<< \"$(get_tier_containers 1)\"\n943\t for container in \"${tier1[@]}\"; do\n944\t [[ -n \"$container\" ]] && local_start \"$container\"\n945\t done\n946\t\n947\t notify \"FALLBACK started on $(hostname) — $REMOTE_SERVER_NAME is down — Tier 1 started\" \\\n948\t \"Fallback\" \"warning\"\n949\t\n950\t elif [[ \"$INTERNET_UP\" == false ]]; then\n951\t # Lost internet — enter NO_INTERNET\n952\t echo \"\"\n953\t echo \"━━━ $ICON_NET Entering NO_INTERNET — $(date '+%Y-%m-%d %H:%M:%S') ━━━\"\n954\t warn \"Internet connectivity lost — stopping local DDNS\"\n955\t\n956\t state_set state \"NO_INTERNET\"\n957\t local_ddns_stop\n958\t\n959\t # Stop containers configured to stop on internet loss\n960\t read -r -a stop_on_no_net <<< \"$(get_stop_on_no_net)\"\n961\t for container in \"${stop_on_no_net[@]}\"; do\n962\t [[ -n \"$container\" ]] && local_stop \"$container\"\n963\t done\n964\t\n965\t notify \"NO_INTERNET on $(hostname) — DDNS stopped, waiting for recovery\" \\\n966\t \"Fallback\" \"warning\"\n967\t fi\n968\t\n969\t # ════════════════════════════════════════════════════════════════\n970\t # FALLBACK STATE\n971\t # ════════════════════════════════════════════════════════════════\n972\t elif [[ \"$CURRENT_STATE\" == \"FALLBACK\" ]]; then\n973\t\n974\t FALLBACK_START_TS=$(state_get fallback_start)\n975\t ELAPSED_MIN=$(( (NOW - FALLBACK_START_TS) / 60 ))\n976\t\n977\t if [[ \"$REMOTE_UP\" == true && \"$INTERNET_UP\" == true ]]; then\n978\t # Remote returned — increment handback strikes\n979\t STRIKES=$(state_get handback_strikes)\n980\t STRIKES=$(( STRIKES + 1 ))\n981\t state_set handback_strikes \"$STRIKES\"\n982\t\n983\t warn \"$ICON_FALLBACK $REMOTE_SERVER_NAME is back — handback strike $STRIKES/$FALLBACK_HANDBACK_STRIKES\"\n984\t\n985\t if [[ \"$STRIKES\" -ge \"$FALLBACK_HANDBACK_STRIKES\" ]]; then\n986\t run_handback\n987\t fi\n988\t\n989\t elif [[ \"$INTERNET_UP\" == false ]]; then\n990\t # Lost internet during fallback — enter DARK\n991\t echo \"\"\n992\t echo \"━━━ $ICON_NET Entering DARK — $(date '+%Y-%m-%d %H:%M:%S') ━━━\"\n993\t warn \"Lost internet during fallback — entering DARK state\"\n994\t state_set state \"DARK\"\n995\t local_ddns_stop\n996\t notify \"DARK state on $(hostname) — lost internet during fallback\" \\\n997\t \"Fallback\" \"warning\"\n998\t\n999\t else\n1000\t # Still in FALLBACK state — check tier escalation\n1001\t state_set handback_strikes \"0\"\n1002\t\n1003\t TIER2_DELAY=$(get_tier2_delay)\n1004\t if [[ \"$(state_get tier2_started)\" == \"false\" && \\\n1005\t \"$ELAPSED_MIN\" -ge \"$TIER2_DELAY\" ]]; then\n1006\t echo \"\"\n1007\t echo \"━━━ $ICON_START Tier 2 — ${ELAPSED_MIN}min outage ━━━\"\n1008\t read -r -a tier2 <<< \"$(get_tier_containers 2)\"\n1009\t for container in \"${tier2[@]}\"; do\n1010\t [[ -n \"$container\" ]] && local_start \"$container\"\n1011\t done\n1012\t state_set tier2_started \"true\"\n1013\t notify \"Fallback Tier 2 started on $(hostname) — ${ELAPSED_MIN}min outage\" \\\n1014\t \"Fallback\" \"warning\"\n1015\t fi\n1016\t\n1017\t TIER3_DELAY=$(get_tier3_delay)\n1018\t if [[ \"$(state_get tier3_started)\" == \"false\" && \\\n1019\t \"$ELAPSED_MIN\" -ge \"$TIER3_DELAY\" ]]; then\n1020\t echo \"\"\n1021\t echo \"━━━ $ICON_START Tier 3 — ${ELAPSED_MIN}min outage ━━━\"\n1022\t read -r -a tier3 <<< \"$(get_tier_containers 3)\"\n1023\t for container in \"${tier3[@]}\"; do\n1024\t [[ -n \"$container\" ]] && local_start \"$container\"\n1025\t done\n1026\t state_set tier3_started \"true\"\n1027\t notify \"Fallback Tier 3 started on $(hostname) — ${ELAPSED_MIN}min outage\" \\\n1028\t \"Fallback\" \"warning\"\n1029\t fi\n1030\t\n1031\t TIER4_DELAY=$(get_tier4_delay)\n1032\t if [[ \"$(state_get tier4_started)\" == \"false\" && \\\n1033\t \"$ELAPSED_MIN\" -ge \"$TIER4_DELAY\" ]]; then\n1034\t echo \"\"\n1035\t echo \"━━━ $ICON_START Tier 4 — ${ELAPSED_MIN}min outage — full workflow ━━━\"\n1036\t read -r -a tier4 <<< \"$(get_tier_containers 4)\"\n1037\t for container in \"${tier4[@]}\"; do\n1038\t [[ -n \"$container\" ]] && local_start \"$container\"\n1039\t done\n1040\t state_set tier4_started \"true\"\n1041\t notify \"Fallback Tier 4 started on $(hostname) — ${ELAPSED_MIN}min outage — full workflow active\" \\\n1042\t \"Fallback\" \"warning\"\n1043\t fi\n1044\t\n1045\t log \"$ICON_FALLBACK FALLBACK — ${ELAPSED_MIN}min — T2:$(state_get tier2_started) T3:$(state_get tier3_started) T4:$(state_get tier4_started)\"\n1046\t fi\n1047\t\n1048\t # ════════════════════════════════════════════════════════════════\n1049\t # NO_INTERNET STATE\n1050\t # ════════════════════════════════════════════════════════════════\n1051\t elif [[ \"$CURRENT_STATE\" == \"NO_INTERNET\" ]]; then\n1052\t\n1053\t if [[ \"$INTERNET_UP\" == true ]]; then\n1054\t echo \"\"\n1055\t echo \"━━━ $ICON_NET Internet Recovered — $(date '+%Y-%m-%d %H:%M:%S') ━━━\"\n1056\t\n1057\t if [[ \"$REMOTE_UP\" == true ]]; then\n1058\t warn \"Remote is up — returning to NORMAL\"\n1059\t state_set state \"NORMAL\"\n1060\t local_ddns_start\n1061\t notify \"Internet recovered on $(hostname) — returning to NORMAL\" \\\n1062\t \"Fallback\" \"normal\"\n1063\t else\n1064\t warn \"Internet back but $REMOTE_SERVER_NAME still down — entering FALLBACK\"\n1065\t state_set state \"FALLBACK\"\n1066\t state_set fallback_start \"$NOW\"\n1067\t state_set handback_strikes \"0\"\n1068\t state_set tier2_started \"false\"\n1069\t state_set tier3_started \"false\"\n1070\t state_set tier4_started \"false\"\n1071\t\n1072\t echo \"\"\n1073\t echo \"━━━ $ICON_START Tier 1 — Immediate ━━━\"\n1074\t read -r -a tier1 <<< \"$(get_tier_containers 1)\"\n1075\t for container in \"${tier1[@]}\"; do\n1076\t [[ -n \"$container\" ]] && local_start \"$container\"\n1077\t done\n1078\t notify \"Internet recovered on $(hostname) but $REMOTE_SERVER_NAME still down — entering FALLBACK\" \\\n1079\t \"Fallback\" \"warning\"\n1080\t fi\n1081\t else\n1082\t log \"$ICON_NET NO_INTERNET — waiting for connectivity\"\n1083\t fi\n1084\t\n1085\t # ════════════════════════════════════════════════════════════════\n1086\t # DARK STATE\n1087\t # ════════════════════════════════════════════════════════════════\n1088\t elif [[ \"$CURRENT_STATE\" == \"DARK\" ]]; then\n1089\t\n1090\t if [[ \"$INTERNET_UP\" == true ]]; then\n1091\t echo \"\"\n1092\t echo \"━━━ $ICON_NET Emerging from DARK — $(date '+%Y-%m-%d %H:%M:%S') ━━━\"\n1093\t\n1094\t if [[ \"$REMOTE_UP\" == true ]]; then\n1095\t warn \"Remote up, internet up — transitioning through FALLBACK for handback\"\n1096\t # Was in FALLBACK state before DARK — route through FALLBACK state for handback\n1097\t state_set state \"FALLBACK\"\n1098\t state_set handback_strikes \"0\"\n1099\t else\n1100\t warn \"Internet back but $REMOTE_SERVER_NAME still down — entering FALLBACK\"\n1101\t state_set state \"FALLBACK\"\n1102\t state_set fallback_start \"$NOW\"\n1103\t state_set handback_strikes \"0\"\n1104\t state_set tier2_started \"false\"\n1105\t state_set tier3_started \"false\"\n1106\t state_set tier4_started \"false\"\n1107\t\n1108\t read -r -a tier1 <<< \"$(get_tier_containers 1)\"\n1109\t for container in \"${tier1[@]}\"; do\n1110\t [[ -n \"$container\" ]] && local_start \"$container\"\n1111\t done\n1112\t fi\n1113\t notify \"Emerging from DARK state on $(hostname)\" \"Fallback\" \"warning\"\n1114\t else\n1115\t log \"$ICON_FALLBACK DARK — no internet, no remote — waiting\"\n1116\t fi\n1117\t fi\n1118\t\n1119\t # ── Sleep until next check ───────────────────────────────────────────────────────────────\n1120\t log \"Next check in ${FALLBACK_CHECK_INTERVAL}s — $(date '+%H:%M:%S')\"\n1121\t sleep \"$FALLBACK_CHECK_INTERVAL\" &\n1122\t wait $!\n1123\t\n1124\tdone"}]},"uuid":"d7170069-dc41-44da-af16-ca26c1f83b6b","timestamp":"2026-06-14T05:32:12.352Z","sourceToolAssistantUUID":"92eb9a40-7b8b-4827-86f3-a145f2e85e0c","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"d7170069-dc41-44da-af16-ca26c1f83b6b","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01KJvtia72nVTMza1m6EtBA8","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_01HY6KBcUxC3h3Jnv73zGM5t","name":"Read","input":{"file_path":"/boot/config/plugins/varaverk/load_config.sh"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2024,"cache_read_input_tokens":5697,"cache_creation":{"ephemeral_5m_input_tokens":2024,"ephemeral_1h_input_tokens":0},"output_tokens":4,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3TfU89TfBpzoqpshv","attributionAgent":"Explore","type":"assistant","uuid":"ef60356d-a6e2-47fd-94bd-ae8d207678cb","timestamp":"2026-06-14T05:32:12.515Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"ef60356d-a6e2-47fd-94bd-ae8d207678cb","isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01HY6KBcUxC3h3Jnv73zGM5t","type":"tool_result","content":"1\t#!/bin/bash\n2\t# ==============================================================================================\n3\t# ================================= CONFIGURATION LOADER =======================================\n4\t# ==============================================================================================\n5\t# Single entry point for all configuration sourcing across the ecosystem.\n6\t# Every script sources this file instead of sourcing master.conf files directly.\n7\t#\n8\t# ── HOW IT WORKS ──────────────────────────────────────────────────────────────────────────────\n9\t# 1. Detects OS platform → PLATFORM=unraid|truenas|unknown; exports SCRIPTS_DIR\n10\t# 2. Sources master.conf (shared config — hostnames, thresholds, toggles, profiles, job lists)\n11\t# 3. Auto-discovers and sources all host*.conf files in the same directory\n12\t# Each host conf extends the shared profile arrays and adds host-specific credentials\n13\t# 4. Sources common.sh (shared functions — detect_hosts, logging, notifications etc.)\n14\t# 5. Sources Plugin//adapter.sh (platform_*() functions for OS-specific ops)\n15\t#\n16\t# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────\n17\t# Without this loader every script had to explicitly source each conf file:\n18\t# source master.conf\n19\t# source host1.conf\n20\t# source host2.conf\n21\t# source common.sh\n22\t#\n23\t# Adding a new server meant updating every script.\n24\t# With this loader — add host3.conf to the git repo and every server\n25\t# auto-discovers it on next git pull. Zero script changes required. Ever.\n26\t#\n27\t# ── ADDING A NEW SERVER ───────────────────────────────────────────────────────────────────────\n28\t# 1. Create Configurations/host3.conf following the same structure as HOST1/HOST2\n29\t# 2. Commit and push to git repo\n30\t# 3. All servers pull it automatically — no other changes needed\n31\t#\n32\t# ── USAGE IN SCRIPTS ──────────────────────────────────────────────────────────────────────────\n33\t# Replace the three source lines at the top of every script with:\n34\t#\n35\t# SCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n36\t# source \"$SCRIPT_DIR/../load_config.sh\"\n37\t#\n38\t# Scripts in subdirectories (Rsync/, Docker_Essentials/ etc.) use ../ to reach root.\n39\t# Scripts in root directory use ./ instead:\n40\t#\n41\t# SCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n42\t# source \"$SCRIPT_DIR/load_config.sh\"\n43\t#\n44\t# ── SPARSE CHECKOUT NOTE ──────────────────────────────────────────────────────────────────────\n45\t# Sparse checkout controls which host*.conf files each server receives.\n46\t# HOST1 only pulls Configurations/host1.conf — never HOST2's credentials.\n47\t# HOST2 only pulls Configurations/host2.conf — never HOST1's credentials.\n48\t# This loader sources whatever conf files ARE present — sparse checkout handles the rest.\n49\t# Both servers pull all non-credential conf files (Configurations/master.conf, common.sh, load_config.sh).\n50\t#\n51\t# ==============================================================================================\n52\t\n53\t# ━━━ Locate config root ━━━\n54\t# load_config.sh always lives in the repo root.\n55\t# Scripts call it from subdirectories using ../ — resolve to the actual root.\n56\t LOAD_CONFIG_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n57\t\n58\t# ━━━ Platform detection ━━━\n59\t# PLATFORM drives the adapter layer — Plugin//adapter.sh\n60\t# Each platform adapter provides the same function API; scripts stay OS-agnostic.\n61\t if [[ -f /etc/unraid-version ]]; then PLATFORM=\"unraid\"\n62\t elif [[ -f /etc/truenas ]]; then PLATFORM=\"truenas\"\n63\t else PLATFORM=\"unknown\"\n64\t fi\n65\t export PLATFORM SCRIPTS_DIR=\"$LOAD_CONFIG_DIR\"\n66\t\n67\t# ━━━ Source shared config ━━━\n68\t# master.conf must be sourced first — it declares the shared PROFILE_* arrays\n69\t# that Host confs extend. Sourcing host confs before master.conf would fail.\n70\t if [[ ! -f \"$LOAD_CONFIG_DIR/Configurations/master.conf\" ]]; then\n71\t echo \"[FATAL] master.conf not found at $LOAD_CONFIG_DIR/Configurations/master.conf\" >&2\n72\t echo \"[FATAL] Check TARGET_DIR and git pull status\" >&2\n73\t exit 1\n74\t fi\n75\t source \"$LOAD_CONFIG_DIR/Configurations/master.conf\"\n76\t\n77\t# ━━━ Auto-discover and source all host*.conf files ━━━\n78\t# Sorted for consistent load order — HOST1 before HOST2 before HOST3 etc.\n79\t# Each host conf extends the shared PROFILE_* arrays and adds host-specific vars.\n80\t# Missing files are silently skipped — sparse checkout intentionally withholds some.\n81\t# At least one host conf must be present or the ecosystem has no identity to work with.\n82\t _host_confs_loaded=0\n83\t declare -A _disk_conf_basenames=()\n84\t\n85\t # Use a sorted array glob — avoids word-splitting on paths with spaces\n86\t while IFS= read -r _conf; do\n87\t [[ -f \"$_conf\" ]] || continue\n88\t source \"$_conf\"\n89\t (( _host_confs_loaded++ ))\n90\t _disk_conf_basenames[\"$(basename \"$_conf\")\"]=1\n91\t [[ \"${ENABLE_LOGGING:-false}\" == \"true\" ]] && \\\n92\t echo \"[LOG] Loaded host config: $(basename \"$_conf\")\" >&2\n93\t done < <(printf '%s\\n' \"$LOAD_CONFIG_DIR/Configurations\"/host*.conf 2>/dev/null | sort)\n94\t\n95\t if [[ \"$_host_confs_loaded\" -eq 0 ]]; then\n96\t echo \"[FATAL] No host*.conf files found in $LOAD_CONFIG_DIR\" >&2\n97\t echo \"[FATAL] At least one host conf required — check git pull and sparse checkout\" >&2\n98\t exit 1\n99\t fi\n100\t\n101\t# ━━━ Source partner confs from /tmp cache ━━━\n102\t# conf_sync.sh pulls partner host*.conf files into /tmp/.vv/config/cached/.confs/\n103\t# on array start and after any conf save. Sourcing them here makes partner vars\n104\t# (HOST2_*, HOST3_*, …) available without committing credentials to the git repo\n105\t# or violating sparse checkout — partner confs live in RAM only, cleared on reboot.\n106\t# Confs already loaded from disk are skipped — disk copy is authoritative.\n107\t _VV_CONF_CACHE=\"/tmp/.vv/config/cached/.confs\"\n108\t if [[ -d \"$_VV_CONF_CACHE\" ]]; then\n109\t while IFS= read -r _conf; do\n110\t [[ -f \"$_conf\" ]] || continue\n111\t _conf_base=\"$(basename \"$_conf\")\"\n112\t # Skip if already sourced from disk\n113\t [[ -n \"${_disk_conf_basenames[$_conf_base]:-}\" ]] && continue\n114\t source \"$_conf\"\n115\t [[ \"${ENABLE_LOGGING:-false}\" == \"true\" ]] && \\\n116\t echo \"[LOG] Loaded cached partner config: $_conf_base\" >&2\n117\t done < <(printf '%s\\n' \"$_VV_CONF_CACHE\"/host*.conf 2>/dev/null | sort)\n118\t fi\n119\t unset _VV_CONF_CACHE _conf_base _disk_conf_basenames\n120\t\n121\t# ━━━ Source shared functions ━━━\n122\t# common.sh sourced last — it calls detect_hosts() which needs HOST* vars to be set.\n123\t if [[ ! -f \"$LOAD_CONFIG_DIR/common.sh\" ]]; then\n124\t echo \"[FATAL] common.sh not found at $LOAD_CONFIG_DIR/common.sh\" >&2\n125\t exit 1\n126\t fi\n127\t source \"$LOAD_CONFIG_DIR/common.sh\"\n128\t\n129\t# ━━━ Source platform adapter ━━━\n130\t# Provides platform_*() functions used by common.sh and scripts.\n131\t# Guard lets the ecosystem run before Plugin//adapter.sh exists.\n132\t _adapter=\"$LOAD_CONFIG_DIR/Plugin/$PLATFORM/adapter.sh\"\n133\t [[ -f \"$_adapter\" ]] && source \"$_adapter\"\n134\t\n135\t# ━━━ Derived path constants ━━━\n136\t# Centralised here so every script that sources load_config.sh has them without\n137\t# re-deriving from SCRIPTS_DIR or hardcoding /var/log or /tmp paths inline.\n138\t CONF_DIR=\"${SCRIPTS_DIR}/Configurations\"\n139\t LOG_DIR=\"/var/log/varaverk\"\n140\t VV_CACHE_DIR=\"/tmp/vv_cache\"\n141\t export CONF_DIR LOG_DIR VV_CACHE_DIR\n142\t\n143\t# ━━━ Cleanup ━━━\n144\t unset _conf _host_confs_loaded _adapter LOAD_CONFIG_DIR"}]},"uuid":"4063f158-aeb4-4575-a6ab-4c721ec80b6b","timestamp":"2026-06-14T05:32:12.524Z","sourceToolAssistantUUID":"ef60356d-a6e2-47fd-94bd-ae8d207678cb","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"4063f158-aeb4-4575-a6ab-4c721ec80b6b","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01KJvtia72nVTMza1m6EtBA8","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_014uZPQUnRBfJSPHuueozfip","name":"Read","input":{"file_path":"/boot/config/plugins/varaverk/common.sh"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2024,"cache_read_input_tokens":5697,"cache_creation":{"ephemeral_5m_input_tokens":2024,"ephemeral_1h_input_tokens":0},"output_tokens":4,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3TfU89TfBpzoqpshv","attributionAgent":"Explore","type":"assistant","uuid":"e60685fd-3658-425f-b029-03c8ce4eca6f","timestamp":"2026-06-14T05:32:12.675Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"e60685fd-3658-425f-b029-03c8ce4eca6f","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01KJvtia72nVTMza1m6EtBA8","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_013u1VkMb4pZiKYSTwMiAeuT","name":"Read","input":{"file_path":"/boot/config/plugins/varaverk/git_pull_execute.sh"},"caller":{"type":"direct"}}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":2024,"cache_read_input_tokens":5697,"cache_creation":{"ephemeral_5m_input_tokens":2024,"ephemeral_1h_input_tokens":0},"output_tokens":4,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3TfU89TfBpzoqpshv","attributionAgent":"Explore","type":"assistant","uuid":"2525452a-39c2-4b3d-8568-0158f90580f9","timestamp":"2026-06-14T05:32:12.743Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"2525452a-39c2-4b3d-8568-0158f90580f9","isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_013u1VkMb4pZiKYSTwMiAeuT","type":"tool_result","content":"1\t#!/bin/bash\n2\t# ==============================================================================================\n3\t# ================================= Git Pull & Execute =========================================\n4\t# ==============================================================================================\n5\t# Pulls the latest scripts from the Gitea repository via SSH.\n6\t# Lives at the repo root — sources load_config.sh from the same directory.\n7\t#\n8\t# ── WHAT THIS SCRIPT DOES ─────────────────────────────────────────────────────────────────────\n9\t# 1. Detects which server it's running on via detect_hosts() (MY_ID)\n10\t# 2. Configures sparse checkout to exclude other servers' credential files\n11\t# Each server only pulls its own host*.conf — never sees peer credentials\n12\t# 3. Pulls or clones latest scripts from Gitea\n13\t# 4. Sets executable permissions on all .sh files\n14\t#\n15\t# ── SPARSE CHECKOUT ───────────────────────────────────────────────────────────────────────────\n16\t# Sparse checkout ensures each server only receives its own host conf:\n17\t# HOST1 pulls: master.conf + host1.conf + all scripts\n18\t# HOST1 skips: host2.conf, host3.conf etc.\n19\t# HOST2 pulls: master.conf + host2.conf + all scripts\n20\t# HOST2 skips: host1.conf, host3.conf etc.\n21\t#\n22\t# Adding a new server:\n23\t# Create host3.conf in the repo\n24\t# All existing servers automatically exclude it on next pull\n25\t# New server gets only its own conf ✅\n26\t#\n27\t# ── GITEA LOCATION DETECTION ──────────────────────────────────────────────────────────────────\n28\t# Detects where Gitea is running at runtime — works through fallback:\n29\t# Gitea local → connects via local IP\n30\t# Gitea remote → connects via Tailscale IP\n31\t# Both fail → falls back to GITEA_DOMAIN if configured\n32\t#\n33\t# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────\n34\t# GITEA_CONTAINER — Docker container name for Gitea\n35\t# GITEA_REPO_PATH — repo path on Gitea (e.g. Varaverk/varaverk.git)\n36\t# GITEA_DOMAIN — public domain fallback (optional)\n37\t# TARGET_DIR — local path to clone/pull into\n38\t# GITEA_SSH_KEY — SSH key for Gitea authentication\n39\t# SSH_PORT — Gitea SSH port (often 221 or 222)\n40\t#\n41\t# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────\n42\t# git_pull_execute.sh — normal pull\n43\t# git_pull_execute.sh --dry-run — preview without making changes\n44\t# git_pull_execute.sh --log — verbose output\n45\t# git_pull_execute.sh --status — show config and exit\n46\t# ==============================================================================================\n47\t\n48\tSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n49\t\n50\t# Root level script — load_config.sh is in the same directory\n51\tsource \"$SCRIPT_DIR/load_config.sh\"\n52\t\n53\tparse_args \"$@\"\n54\t\n55\t# ==============================================================================================\n56\t# ━━━ Setup ━━━\n57\t# ==============================================================================================\n58\tif [[ \"$EUID\" -ne 0 ]]; then\n59\t error \"Must be run as root\"\n60\t exit 1\n61\tfi\n62\tacquire_lock\n63\t\n64\tif ! command -v docker &>/dev/null; then\n65\t error \"Docker command not found\"\n66\t exit 1\n67\tfi\n68\t\n69\t# detect_hosts() sets MY_ID — needed for sparse checkout configuration\n70\tdetect_hosts\n71\t\n72\t# ==============================================================================================\n73\t# ━━━ Locate Gitea ━━━\n74\t# ==============================================================================================\n75\techo \"\"\n76\techo \"━━━ $ICON_CONTAINERS Locate Gitea ━━━\"\n77\t\n78\tif docker ps --format \"{{.Names}}\" 2>/dev/null | grep -q \"^${GITEA_CONTAINER}$\"; then\n79\t # Gitea is running on this server — use local IP\n80\t GITEA_IP=$(hostname -I | awk '{print $1}')\n81\t log \"Gitea running locally — connecting via $GITEA_IP\"\n82\telse\n83\t # Gitea not running locally — find it on the remote server via Tailscale\n84\t log \"Gitea not running locally — checking remote server\"\n85\t GITEA_IP=$(resolve_tailscale_ip \"${REMOTE_SERVER_NAME}\")\n86\t if [[ -n \"$GITEA_IP\" ]]; then\n87\t echo \" Gitea on $REMOTE_SERVER_NAME — connecting via Tailscale $GITEA_IP\"\n88\t elif [[ -n \"${GITEA_DOMAIN:-}\" ]]; then\n89\t warn \"Tailscale resolution failed — falling back to $GITEA_DOMAIN\"\n90\t GITEA_IP=\"$GITEA_DOMAIN\"\n91\t else\n92\t error \"Cannot find Gitea — local: not running, Tailscale: failed, domain: not configured\"\n93\t notify \"Git pull failed on $(hostname) — cannot locate Gitea container\" \"Git Sync\" \"alert\"\n94\t exit 1\n95\t fi\n96\tfi\n97\t\n98\tREPO_SSH=\"git@${GITEA_IP}:${GITEA_REPO_PATH}\"\n99\t\n100\trequire_var REPO_SSH\n101\trequire_var TARGET_DIR\n102\trequire_var GITEA_SSH_KEY\n103\trequire_var SSH_PORT\n104\t\n105\t# ==============================================================================================\n106\t# ━━━ Status ━━━\n107\t# ==============================================================================================\n108\tif [[ \"$SHOW_STATUS\" == true ]]; then\n109\t echo \"\"\n110\t echo \"━━━━━ $ICON_SUMMARY STATUS ━━━━━\"\n111\t echo \"$ICON_NET Repo: $REPO_SSH\"\n112\t echo \"$ICON_GEAR Target: $TARGET_DIR\"\n113\t echo \"$ICON_GEAR SSH Key: $GITEA_SSH_KEY\"\n114\t echo \"$ICON_GEAR SSH Port: $SSH_PORT\"\n115\t echo \"$ICON_HOST My ID: $MY_ID ($LOCAL_SERVER_NAME)\"\n116\t echo \"$ICON_HOST Remote ID: $REMOTE_ID ($REMOTE_SERVER_NAME)\"\n117\t echo \"$ICON_NOTIFY Notify: unRAID=${NOTIFY_UNRAID:-false} Discord=$([[ -n \"${MY_DISCORD_WEBHOOK:-}\" ]] && echo enabled || echo disabled)\"\n118\t echo \"$ICON_GEAR Dry Run: $DRY_RUN\"\n119\t echo \"━━━━━━━━━━━━━━━━━━━━━━━\"\n120\t exit 0\n121\tfi\n122\t\n123\t[[ \"$DRY_RUN\" == true ]] && warn \"DRY RUN — no changes will be made\"\n124\t\n125\t# ==============================================================================================\n126\t# ━━━ Sparse Checkout Configuration ━━━\n127\t# ==============================================================================================\n128\t# Build the list of host*.conf files that belong to OTHER servers.\n129\t# This server pulls everything EXCEPT those files.\n130\t# MY_ID is set by detect_hosts() — e.g. \"HOST1\"\n131\t\n132\tconfigure_sparse_checkout() {\n133\t local repo_dir=\"$1\"\n134\t\n135\t log \"Configuring sparse checkout for $MY_ID...\"\n136\t\n137\t # Enable sparse checkout\n138\t git -C \"$repo_dir\" config core.sparseCheckout true 2>/dev/null\n139\t\n140\t # Build exclusion list — all host*.conf files except MY_ID's\n141\t local sparse_file=\"$repo_dir/.git/info/sparse-checkout\"\n142\t mkdir -p \"$(dirname \"$sparse_file\")\"\n143\t\n144\t # Start with: pull everything\n145\t echo \"/*\" > \"$sparse_file\"\n146\t\n147\t # Exclude each other server's conf file\n148\t # Find all host*.conf files present in the repo\n149\t local excluded=0\n150\t for conf_file in \"$repo_dir\"/host*.conf; do\n151\t [[ -f \"$conf_file\" ]] || continue\n152\t local conf_name\n153\t conf_name=$(basename \"$conf_file\")\n154\t\n155\t # Determine which HOST ID owns this conf by grepping its hostname var\n156\t # Pattern: HOST1=\"unRAID-...\" or HOST2=\"unRAID-...\"\n157\t local conf_host_id\n158\t conf_host_id=$(grep -m1 -oP '^\\s+HOST[0-9]+(?==)' \"$conf_file\" 2>/dev/null | tr -d ' ')\n159\t\n160\t if [[ -z \"$conf_host_id\" ]]; then\n161\t log \"Cannot determine HOST ID for $conf_name — including in pull (safe default)\"\n162\t continue\n163\t fi\n164\t\n165\t if [[ \"$conf_host_id\" != \"$MY_ID\" ]]; then\n166\t echo \"!$conf_name\" >> \"$sparse_file\"\n167\t log \"Sparse checkout: excluding $conf_name (belongs to $conf_host_id)\"\n168\t ((excluded++))\n169\t else\n170\t log \"Sparse checkout: including $conf_name (belongs to $MY_ID — this server)\"\n171\t fi\n172\t done\n173\t\n174\t if [[ \"$excluded\" -gt 0 ]]; then\n175\t echo \" Sparse checkout: excluding $excluded peer conf file(s) — credentials protected\"\n176\t else\n177\t log \"Sparse checkout: no peer conf files to exclude (single server or first run)\"\n178\t fi\n179\t}\n180\t\n181\t# ==============================================================================================\n182\t# ━━━ Git Sync ━━━\n183\t# ==============================================================================================\n184\techo \"\"\n185\techo \"━━━ $ICON_SYNC Git Sync ━━━\"\n186\techo \"$ICON_NET Repo: $REPO_SSH\"\n187\techo \"$ICON_GEAR Target: $TARGET_DIR\"\n188\techo \"\"\n189\t\n190\tSTART=$(date +%s)\n191\tSYNC_SUCCESS=false\n192\t\n193\tif [[ \"$DRY_RUN\" == true ]]; then\n194\t warn \"DRY RUN — would sync $REPO_SSH → $TARGET_DIR\"\n195\t warn \"DRY RUN — would configure sparse checkout for $MY_ID\"\n196\t warn \"DRY RUN — would exclude peer host*.conf files\"\n197\t SYNC_SUCCESS=true\n198\telse\n199\t mkdir -p \"$TARGET_DIR\"\n200\t git config --global --add safe.directory \"$TARGET_DIR\"\n201\t cd \"$TARGET_DIR\" || { error \"Cannot cd into $TARGET_DIR\"; exit 1; }\n202\t\n203\t if [[ -d \".git\" ]]; then\n204\t # ── Existing repository ──────────────────────────────────────────────\n205\t echo \" Existing repository — updating\"\n206\t\n207\t # Configure sparse checkout BEFORE pull\n208\t # Uses conf files already present from last pull to determine exclusions\n209\t configure_sparse_checkout \"$TARGET_DIR\"\n210\t\n211\t echo \" Pulling latest changes...\"\n212\t if GIT_SSH_COMMAND=\"ssh -i $GITEA_SSH_KEY -p $SSH_PORT\" git pull --ff-only; then\n213\t echo \" Git pull successful\"\n214\t SYNC_SUCCESS=true\n215\t else\n216\t # ff-only fails when local commits or tracked changes exist that can't\n217\t # fast-forward. Fail loudly — never silently destroy local work.\n218\t error \"Git pull failed — local changes conflict with remote (will not force-reset)\"\n219\t notify \"Git pull failed on $(hostname) — local changes conflict, manual resolve needed\" \"Git Sync\" \"alert\"\n220\t exit 1\n221\t fi\n222\t\n223\t else\n224\t # ── Fresh clone ──────────────────────────────────────────────────────\n225\t echo \" No repository found — cloning\"\n226\t\n227\t # Clone first — need the repo to exist before configuring sparse checkout\n228\t if GIT_SSH_COMMAND=\"ssh -i $GITEA_SSH_KEY -p $SSH_PORT\" git clone \"$REPO_SSH\" .; then\n229\t echo \" Clone successful\"\n230\t\n231\t # Configure sparse checkout after clone\n232\t # Now all host*.conf files are present — can detect exclusions\n233\t configure_sparse_checkout \"$TARGET_DIR\"\n234\t\n235\t # Apply sparse checkout — removes excluded files from working tree\n236\t echo \" Applying sparse checkout...\"\n237\t git read-tree -mu HEAD\n238\t echo \" Sparse checkout applied — peer credentials removed from working tree\"\n239\t\n240\t SYNC_SUCCESS=true\n241\t else\n242\t error \"Clone failed\"\n243\t notify \"Git clone failed on $(hostname) — check Gitea connectivity\" \"Git Sync\" \"alert\"\n244\t exit 1\n245\t fi\n246\t fi\n247\t\n248\t # ── Permissions ──────────────────────────────────────────────────────────\n249\t echo \"\"\n250\t echo \"━━━ $ICON_GEAR Permissions ━━━\"\n251\t log \"Setting executable permissions on all .sh files...\"\n252\t find \"$TARGET_DIR\" -type f -name \"*.sh\" -exec chmod +x {} \\;\n253\t echo \" Permissions set on .sh files\"\n254\t\n255\t # ── Flash mode: sync Plugin/ to /boot/ so the webUI picks up updates ─────\n256\t # In flash mode SCRIPTS_DIR is in appdata — Plugin/ lives in the repo there\n257\t # but Unraid serves PHP from /boot/. Sync after every pull to keep them in step.\n258\t _BOOT_DIR=$(platform_webui_install_path)\n259\t if [[ \"$TARGET_DIR\" != \"$_BOOT_DIR\" ]]; then\n260\t echo \"\"\n261\t echo \"━━━ $ICON_SYNC Flash mode: sync Plugin/ → /boot/ ━━━\"\n262\t if rsync -a --delete \"$TARGET_DIR/Plugin/\" \"$_BOOT_DIR/Plugin/\" 2>/dev/null; then\n263\t echo \" Plugin/ synced to /boot/ ✅\"\n264\t else\n265\t warn \"Plugin/ sync to /boot/ failed — webUI may be stale until next pull\"\n266\t fi\n267\t fi\n268\tfi\n269\t\n270\tEND=$(date +%s)\n271\t\n272\t# ==============================================================================================\n273\t# ━━━ Conf Upgrade ━━━\n274\t# ==============================================================================================\n275\t# Merges new conf structure into the live conf files after every pull.\n276\t# New keys → added with template defaults (user fills in once).\n277\t# Removed keys → dropped. Existing values → always preserved.\n278\t# Silent when already up to date — no overhead on unchanged pulls.\n279\t\n280\techo \"\"\n281\techo \"━━━ $ICON_GEAR Conf Upgrade ━━━\"\n282\t\n283\tUPGRADE_SCRIPT=\"$TARGET_DIR/Deployment/conf_upgrade.sh\"\n284\tCONF_DIR=\"$TARGET_DIR/Configurations\"\n285\t\n286\tif [[ ! -f \"$UPGRADE_SCRIPT\" ]]; then\n287\t log \"conf_upgrade.sh not found — skipping (pre-deployment-folder repo)\"\n288\telif [[ \"$DRY_RUN\" == true ]]; then\n289\t warn \"DRY RUN — would upgrade master.conf and ${MY_ID,,}.conf\"\n290\telif [[ \"$SYNC_SUCCESS\" == true ]]; then\n291\t _DRY=\"\"\n292\t\n293\t # master.conf\n294\t if [[ -f \"$CONF_DIR/master.conf.template\" && -f \"$CONF_DIR/master.conf\" ]]; then\n295\t bash \"$UPGRADE_SCRIPT\" \\\n296\t --template \"$CONF_DIR/master.conf.template\" \\\n297\t --target \"$CONF_DIR/master.conf\" \\\n298\t --backup $_DRY\n299\t else\n300\t warn \"master.conf.template or master.conf not found — skipping\"\n301\t fi\n302\t\n303\t # This server's host conf only — sparse checkout ensures we have it\n304\t HOST_CONF=\"$CONF_DIR/${MY_ID,,}.conf\"\n305\t if [[ -f \"$CONF_DIR/host.conf.template\" && -f \"$HOST_CONF\" ]]; then\n306\t # Template uses HOSTN_ as generic prefix. Substitute MY_ID before merging\n307\t # so keys match the target (HOST1_* or HOST2_*) and real values are preserved.\n308\t TMPL_RESOLVED=$(mktemp)\n309\t sed \"s/HOSTN_/${MY_ID}_/g; s/REMOTE_ID/${REMOTE_ID}/g\" \"$CONF_DIR/host.conf.template\" > \"$TMPL_RESOLVED\"\n310\t bash \"$UPGRADE_SCRIPT\" \\\n311\t --template \"$TMPL_RESOLVED\" \\\n312\t --target \"$HOST_CONF\" \\\n313\t --backup $_DRY\n314\t rm -f \"$TMPL_RESOLVED\"\n315\t else\n316\t warn \"${MY_ID,,}.conf or host.conf.template not found — skipping\"\n317\t fi\n318\tfi\n319\t\n320\t# ==============================================================================================\n321\t# ━━━ Summary ━━━\n322\t# ==============================================================================================\n323\techo \"\"\n324\techo \"━━━━━ $ICON_SUMMARY GIT SYNC SUMMARY ━━━━━\"\n325\techo \"$ICON_NET Repo: $REPO_SSH\"\n326\techo \"$ICON_GEAR Target: $TARGET_DIR\"\n327\techo \"$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)\"\n328\techo \"$ICON_LOCK Excluded: peer host*.conf files\"\n329\techo \"$ICON_TIME Duration: $(format_duration $((END - START)))\"\n330\tif [[ \"$DRY_RUN\" == true ]]; then\n331\t echo \"$ICON_WARN Status: DRY RUN — no changes made\"\n332\telif [[ \"$SYNC_SUCCESS\" == true ]]; then\n333\t echo \"$ICON_DONE Status: $ICON_SUCCESS DONE\"\n334\t notify \"Repository synced successfully on $(hostname)\" \"Git Sync\" \"normal\"\n335\telse\n336\t echo \"$ICON_ERROR Status: $ICON_ERROR FAILED\"\n337\tfi\n338\techo \"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\""}]},"uuid":"3e564d4a-3214-4279-9da9-21dd859cde7c","timestamp":"2026-06-14T05:32:12.752Z","sourceToolAssistantUUID":"2525452a-39c2-4b3d-8568-0158f90580f9","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"e60685fd-3658-425f-b029-03c8ce4eca6f","isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_014uZPQUnRBfJSPHuueozfip","type":"tool_result","content":"[Truncated: PARTIAL view — showing lines 1-1573 of 1926 total (26018 tokens, cap 25000). Call Read with offset=1574 limit=1573 for the next page, or Grep to find a specific section. Do NOT answer from this page alone if the answer may be further in the file.]\n\n1\t#!/bin/bash\n2\t# ==============================================================================================\n3\t# ================================= COMMON LIBRARY =============================================\n4\t# ==============================================================================================\n5\t# Shared functions used by every script in the ecosystem.\n6\t# Sourced automatically by load_config.sh — do not source directly.\n7\t#\n8\t# ── WHAT THIS FILE PROVIDES ───────────────────────────────────────────────────────────────────\n9\t# Icons — consistent visual language across all script output\n10\t# Output helpers — info, warn, error, success, log, notify\n11\t# Arg parser — unified --dry-run, --log, --status, KEY=VALUE handling\n12\t# Validation — require_var, validate_int\n13\t# Host detection — detect_hosts() identifies local/remote server, sets all aliases\n14\t# Connectivity — resolve_remote_ip, check_connectivity, ping_remote, ping_internet\n15\t# Health checks — local/remote array, docker daemon, rootfs, share, disk temps\n16\t# Disk verification — check_remote_disks() — XFS, ZFS, cache pools, nested paths\n17\t# Container mgmt — stop/start local and remote containers with state tracking\n18\t# Rsync helpers — get_rsync_opts, check_rsync_enabled\n19\t# Locking — acquire_lock, acquire_rsync_lock — concurrent execution protection\n20\t# Path translation — translate_path() — container → host path for arr cleanup scripts\n21\t# Arr utilities — check_arr_version() — API version safety gate\n22\t# Status display — show_status() — runtime config dump on --status flag\n23\t#\n24\t# ── CHANGELOG ─────────────────────────────────────────────────────────────────────────────────\n25\t#\n26\t# Version What changed\n27\t# ───────────────────────────────────────────────────────────────────────────────────────────\n28\t# v1.0 Initial stable framework — output helpers, arg parser, host detection\n29\t#\n30\t# v1.1 format_duration() moved here from daily_sync_maintenance.sh for shared use\n31\t# SSH_KEY naming collision resolved — gitea key renamed GITEA_SSH_KEY in master.conf\n32\t#\n33\t# v1.2 check_connectivity() added — fatal ping with Tailscale hint on failure\n34\t# Function header comment blocks standardised across all functions\n35\t#\n36\t# v1.3 check_remote_rootfs() — aborts rsync if remote rootfs exceeds ROOTFS_WARN\n37\t# check_remote_share() — aborts if target directory missing or empty on remote\n38\t# Both protect against rsync running when remote array is down\n39\t#\n40\t# v1.4 check_remote_disks() — verifies all physical disks backing a share are mounted\n41\t# Auto-discovers disk layout at runtime from disks.ini — no config required\n42\t# Aborts if any disk backing the share is offline or unmounted\n43\t#\n44\t# v1.5-2.9 Icon set expanded progressively — each operation has its own distinct icon\n45\t# All function output updated to use correct icon per context\n46\t# notify() added — unRAID native + Discord webhook notifications\n47\t# Script locking system added — acquire_lock(), acquire_rsync_lock()\n48\t# translate_path() added — container → host path for arr cleanup scripts\n49\t# check_arr_version() added — API version safety gate before arr operations\n50\t# check_api() added — pre-flight API reachability check\n51\t# ping_remote(), ping_internet() — non-fatal ping for fallback use\n52\t# check_local_array(), check_remote_array() — array health checks\n53\t# check_remote_docker() — Docker daemon health check\n54\t# get_unraid_temp_thresholds() — reads thresholds from dynamix.cfg\n55\t# check_local_disk_temps() — pre-rsync temp check with exit codes 0/1/2\n56\t# stop/start local containers added alongside existing remote variants\n57\t#\n58\t# v3.1 Three-file config split — master.conf + host1.conf + host2.conf\n59\t# load_config.sh introduced — auto-discovers all host*.conf files\n60\t# detect_hosts() rewritten — sets MY_ID/REMOTE_ID and aliases all HOST* vars\n61\t# All scripts now source load_config.sh instead of conf files directly\n62\t# Adding a new server = add host*.conf, zero script changes required\n63\t#\n64\t# v3.2 check_remote_disks() rewritten — three-tier detection:\n65\t# Tier 1: array disk paths (/mnt/disk*/sharename)\n66\t# Tier 2: ZFS pools with find -maxdepth 2 (catches nested cache paths)\n67\t# Tier 3: shfs fallback — verifies /mnt/user is mounted\n68\t# Fixes: nested paths like /mnt/cache/appdata-Fallback/Critical-Data\n69\t# Fixes: ZFS cache pools not detected when share is not at pool root\n70\t#\n71\t# v3.3 PROFILE_REMOTE_RESTART_CONTAINERS support added\n72\t# Dirty sync profiles (critical-fallback, emby-fallback) restart remote\n73\t# containers after sync if they were running before — picks up config changes\n74\t# Was stopped → stays stopped. Was running → gets restarted. ✅\n75\t#\n76\t# v3.5 notify_emby_scan() added — triggers Emby \"Clean Missing Files\" task\n77\t# after arr cleanup scripts delete orphaned files\n78\t# Emby immediately removes ghost entries — no user-facing file-not-found errors\n79\t# Called by lidarr/sonarr/radarr_cleanup.sh when files are deleted\n80\t#\n81\t#\n82\t# Three new safety functions added:\n83\t# check_os_version_parity() — refuses remote ops on version mismatch\n84\t# reads OS version via platform_get_os_version() / platform_os_version_probe_cmd()\n85\t# major mismatch → abort | minor mismatch → configurable warn/abort\n86\t# check_remote_docker_daemon() — verifies remote Docker daemon before\n87\t# issuing any remote container commands — strike system → skip/retry/exit\n88\t# validate_unraid_cmd() — verifies unRAID-specific commands exist and\n89\t# produce expected output before use — notifies and exits calling script\n90\t# if command changed or disappeared after upgrade — other scripts unaffected\n91\t#\n92\t# ── VERSION ───────────────────────────────────────────────────────────────────────────────────\n93\t# Current: v3.5\n94\t# ==============================================================================================\n95\t\n96\t# Cron runs with a minimal PATH (/usr/bin:/bin) that omits /usr/local/sbin where tailscale,\n97\t# and other Unraid tools live. Prepend it here so every sourcing script finds them.\n98\texport PATH=\"/usr/local/sbin:/usr/local/bin:$PATH\"\n99\t\n100\t# ==============================================================================================\n101\t# ── ICONS ─────────────────────────────────────────────────────────────────────────────────────\n102\t# ==============================================================================================\n103\t# Each icon has one job — do not reuse across different contexts.\n104\t# Adding a new icon: add to the appropriate group below with a comment describing its job.\n105\t\n106\t# System / Host\n107\tICON_HOST=\"🖥️\" # host detection and identity\n108\tICON_NET=\"🌐\" # network / IP resolution\n109\tICON_PING=\"📡\" # connectivity check\n110\tICON_GEAR=\"⚙️\" # setup section header / profile load\n111\t\n112\t# Health Checks\n113\tICON_DISK=\"💾\" # disk checks\n114\tICON_HEALTH=\"🩺\" # rootfs / share health checks\n115\tICON_SHIELD=\"🛡️\" # pre-flight section header\n116\t\n117\t# Containers\n118\tICON_CONTAINERS=\"📦\" # container section anchor\n119\tICON_STOP=\"⛔\" # stop command being issued\n120\tICON_STOPPED=\"🔴\" # container confirmed stopped\n121\tICON_START=\"▶️\" # start command being issued\n122\tICON_STARTED=\"💚\" # container confirmed started\n123\tICON_RUNNING=\"🟢\" # container already running when checked\n124\tICON_SKIP=\"⏭️\" # skipping — already running healthy instance\n125\tICON_PROTECTED=\"🔰\" # file/item is protected — never delete\n126\tICON_NOT_RUNNING=\"⭕\" # container already stopped when checked\n127\t\n128\t# Transfer\n129\tICON_SYNC=\"🔄\" # transfer section header\n130\tICON_RUN=\"🚀\" # sync starting / rsync attempt\n131\tICON_RETRY=\"🔁\" # retry attempt\n132\tICON_DONE=\"🏁\" # transfer complete\n133\t\n134\t# Summary\n135\tICON_SUMMARY=\"📋\" # summary section header\n136\tICON_TIME=\"⏱️\" # duration line\n137\t\n138\t# System Operations\n139\tICON_MOVER=\"🔃\" # mover operations\n140\tICON_REBOOT=\"⚡\" # scheduled server reboot\n141\tICON_REBOOT_SMART=\"🚨\" # smart conditional reboot triggered\n142\tICON_PLUGIN=\"🧩\" # user scripts plugin operations\n143\tICON_PHP=\"👥\" # PHP-FPM operations\n144\tICON_WEBGUI=\"💻\" # WebGUI / nginx / emhttp operations\n145\t\n146\t# Diagnostics\n147\tICON_ZFS=\"📊\" # ZFS ARC statistics\n148\tICON_MEM=\"🧠\" # memory status\n149\tICON_WATCHDOG=\"🐾\" # docker watchdog monitoring operations\n150\t\n151\t# Media Operations\n152\tICON_CLEAN=\"🧹\" # media cleaner operations\n153\tICON_TRASH=\"🗑️\" # files being deleted\n154\tICON_PERMS=\"🔐\" # permissions operation / section header\n155\tICON_UNLOCKED=\"🔓\" # permissions successfully applied to a share\n156\tICON_LOCK=\"🔏\" # script instance lock — acquired/released\n157\t\n158\t# Transcode Operations\n159\tICON_RAM=\"💨\" # ramdisk operations — fast ephemeral storage\n160\tICON_LINK=\"🔗\" # symlink state and management\n161\t\n162\t# fallback Operations\n163\tICON_FALLBACK=\"🔀\" # fallback state changes and operations\n164\t\n165\t# Docker Network Operations\n166\tICON_DOCKER_NET=\"🔌\" # Docker network connect operations\n167\t\n168\t# Security / Certificate Operations\n169\tICON_CERT=\"🔒\" # SSL certificate monitoring\n170\t\n171\t# Monitor Operations\n172\tICON_MONITOR=\"📈\" # monitoring section headers and general monitoring\n173\tICON_SMART=\"🔧\" # drive SMART health attribute monitoring\n174\tICON_BANDWIDTH=\"📶\" # bandwidth usage tracking and reporting\n175\tICON_DIGEST=\"📰\" # health digest — aggregated system summary\n176\tICON_EMBY=\"🎬\" # Emby media server session reporting\n177\tICON_VERIFY=\"✔️\" # backup verification — checksum comparison\n178\t\n179\t# Notifications\n180\tICON_NOTIFY=\"🔔\" # notification operations\n181\t\n182\t# Output\n183\tICON_INFO=\"ℹ️\"\n184\tICON_WARN=\"⚠️\"\n185\tICON_ERROR=\"❌\"\n186\tICON_SUCCESS=\"✅\"\n187\t\n188\t# ==============================================================================================\n189\t# ── OUTPUT HELPERS ────────────────────────────────────────────────────────────────────────────\n190\t# ==============================================================================================\n191\t# Standardised output functions used across all scripts.\n192\t#\n193\t# Output tiers:\n194\t# echo — always visible — summaries, status conclusions, section headers\n195\t# info() — always visible — status confirmations, reachability, check pass\n196\t# warn() — always visible — state transitions, warnings, important events\n197\t# error() — always visible — something broke\n198\t# success() — always visible — operation completed successfully\n199\t# log() — only with --log flag — per-item detail, internal checks\n200\t#\n201\t# All output goes to stdout — callers can redirect as needed.\n202\t\n203\tinfo() { echo \"$ICON_INFO [INFO] $*\"; }\n204\twarn() { echo \"$ICON_WARN [WARN] $*\"; }\n205\terror() { echo \"$ICON_ERROR [ERROR] $*\"; }\n206\tsuccess() { echo \"$ICON_DONE [OK] $*\"; }\n207\t\n208\tlog() {\n209\t [[ \"${ENABLE_LOGGING:-false}\" == true ]] && echo \"[LOG] $*\"\n210\t return 0\n211\t}\n212\t\n213\t# ==============================================================================================\n214\t# ── NOTIFICATION ──────────────────────────────────────────────────────────────────────────────\n215\t# ==============================================================================================\n216\t# Sends a notification via unRAID native system and/or Discord webhook.\n217\t# Both channels are optional and independently controlled:\n218\t# NOTIFY_UNRAID — shared toggle in master.conf\n219\t# MY_DISCORD_WEBHOOK — per-host in host*.conf, set by detect_hosts()\n220\t#\n221\t# Severity levels: normal, warning, alert\n222\t# Usage: notify \"message\" \"subject\" \"severity\"\n223\t\n224\tnotify() {\n225\t local message=\"$1\"\n226\t local subject=\"${2:-unRAID Notification}\"\n227\t local severity=\"${3:-normal}\"\n228\t\n229\t log \"$ICON_NOTIFY Sending notification: $subject — $message\"\n230\t\n231\t if [[ \"${NOTIFY_UNRAID:-false}\" == true ]]; then\n232\t if platform_send_os_notification \"$message\" \"$subject\" \"$severity\"; then\n233\t log \"$ICON_NOTIFY OS notification sent\"\n234\t else\n235\t log \"$ICON_NOTIFY OS notify not available — skipping\"\n236\t fi\n237\t fi\n238\t\n239\t # Uses MY_DISCORD_WEBHOOK — set by detect_hosts() from HOST*_DISCORD_WEBHOOK\n240\t if [[ -n \"${MY_DISCORD_WEBHOOK:-}\" ]]; then\n241\t local payload\n242\t payload=$(printf '{\"content\": \"%s — **%s**\\\\n%s\"}' \\\n243\t \"$ICON_NOTIFY\" \"$subject\" \"$message\")\n244\t if curl -s -H \"Content-Type: application/json\" \\\n245\t -d \"$payload\" \"$MY_DISCORD_WEBHOOK\" >/dev/null 2>&1; then\n246\t log \"$ICON_NOTIFY Discord notification sent\"\n247\t else\n248\t warn \"Discord notification failed — check HOST*_DISCORD_WEBHOOK in host*.conf\"\n249\t fi\n250\t fi\n251\t}\n252\t\n253\t# ==============================================================================================\n254\t# ── DURATION FORMATTER ────────────────────────────────────────────────────────────────────────\n255\t# ==============================================================================================\n256\t# Converts raw seconds into a human readable string.\n257\t# Examples: 47 → \"47s\" | 653 → \"10m53s\"\n258\t# Usage: format_duration $SECONDS\n259\t\n260\tformat_duration() {\n261\t local secs=$1\n262\t local days=$(( secs / 86400 ))\n263\t local hrs=$(( (secs % 86400) / 3600 ))\n264\t local mins=$(( (secs % 3600) / 60 ))\n265\t local rem=$(( secs % 60 ))\n266\t if [[ $days -gt 0 ]]; then echo \"${days}d ${hrs}h ${mins}m\"\n267\t elif [[ $hrs -gt 0 ]]; then echo \"${hrs}h ${mins}m\"\n268\t elif [[ $mins -gt 0 ]]; then echo \"${mins}m${rem}s\"\n269\t else echo \"${rem}s\"; fi\n270\t}\n271\t\n272\t# ==============================================================================================\n273\t# ── ARG PARSER ────────────────────────────────────────────────────────────────────────────────\n274\t# ==============================================================================================\n275\t# Processes all flags and key=value pairs passed to any script.\n276\t# Call early in every script: parse_args \"$@\"\n277\t#\n278\t# Supported flags:\n279\t# --dry-run | -n → DRY_RUN=true\n280\t# --log → ENABLE_LOGGING=true\n281\t# --no-log → ENABLE_LOGGING=false\n282\t# --status → SHOW_STATUS=true\n283\t# --help | -h → print usage and exit\n284\t#\n285\t# Supported key=value:\n286\t# LOG=true/false → toggle logging\n287\t# BW_LIMIT=5000 → override any declared master.conf variable for this run\n288\t#\n289\t# Unparsed positional args returned in PARSED_ARGS array.\n290\t\n291\tparse_args() {\n292\t ENABLE_LOGGING=${ENABLE_LOGGING:-false}\n293\t DRY_RUN=${DRY_RUN:-false}\n294\t SHOW_STATUS=${SHOW_STATUS:-false}\n295\t CLEAN_ARGS=()\n296\t\n297\t for ARG in \"$@\"; do\n298\t if [[ \"$ARG\" == *=* ]]; then\n299\t VAR=\"${ARG%%=*}\"\n300\t VAL=\"${ARG#*=}\"\n301\t case \"$VAR\" in\n302\t LOG)\n303\t [[ \"$VAL\" == \"true\" ]] && ENABLE_LOGGING=true\n304\t [[ \"$VAL\" == \"false\" ]] && ENABLE_LOGGING=false\n305\t ;;\n306\t *)\n307\t if declare -p \"$VAR\" &>/dev/null; then\n308\t printf -v \"$VAR\" '%s' \"$VAL\"\n309\t log \"Set $VAR=$VAL\"\n310\t else\n311\t warn \"Unknown variable: $VAR\"\n312\t fi\n313\t ;;\n314\t esac\n315\t else\n316\t case \"$ARG\" in\n317\t --dry-run|-n) DRY_RUN=true ;;\n318\t --log) ENABLE_LOGGING=true ;;\n319\t --no-log) ENABLE_LOGGING=false ;;\n320\t --status|--summary) SHOW_STATUS=true ;;\n321\t --help|-h)\n322\t echo \"Usage: script [--dry-run] [--log] [--status]\"\n323\t exit 0\n324\t ;;\n325\t *) CLEAN_ARGS+=(\"$ARG\") ;;\n326\t esac\n327\t fi\n328\t done\n329\t\n330\t PARSED_ARGS=(\"${CLEAN_ARGS[@]}\")\n331\t}\n332\t\n333\t# ==============================================================================================\n334\t# ── VALIDATION HELPERS ────────────────────────────────────────────────────────────────────────\n335\t# ==============================================================================================\n336\t\n337\t# Exits with error if a required variable is empty or unset.\n338\t# Usage: require_var VAR_NAME\n339\trequire_var() {\n340\t [[ -z \"${!1:-}\" ]] && error \"Missing required: $1\" && exit 1\n341\t}\n342\t\n343\t# Exits with error if a variable is not a valid positive integer.\n344\t# Usage: validate_int VAR_NAME \"$VAR_VALUE\"\n345\tvalidate_int() {\n346\t local name=\"$1\" value=\"$2\"\n347\t if [[ -z \"$value\" ]]; then\n348\t error \"$name is not set — check master.conf\"\n349\t exit 1\n350\t fi\n351\t if ! [[ \"$value\" =~ ^[0-9]+$ ]]; then\n352\t error \"$name must be a positive integer — got: '$value'\"\n353\t exit 1\n354\t fi\n355\t log \"$name validated: $value\"\n356\t}\n357\t\n358\t# ==============================================================================================\n359\t# ── HOST DETECTION ────────────────────────────────────────────────────────────────────────────\n360\t# ==============================================================================================\n361\t# Determines which server is local and which is remote by comparing hostname against\n362\t# all HOST* values discovered from host*.conf files.\n363\t#\n364\t# Sets:\n365\t# MY_ID — \"HOST1\" or \"HOST2\" (the role key, not the hostname)\n366\t# REMOTE_ID — the peer's role key\n367\t# LOCAL_SERVER_NAME — local hostname string (e.g. \"unRAID-Gmer4Lfe\")\n368\t# REMOTE_SERVER_NAME — remote hostname string\n369\t# SSH_KEY — SSH key for server-to-server operations\n370\t# REMOTE_STORAGE_PATH — remote's storage root from HOST*_STORAGE_PATH in host*.conf;\n371\t# falls back to platform_storage_path() for unconfigured pairs\n372\t#\n373\t# Also sets aliases for all host-specific arrays so scripts use unprefixed names:\n374\t# DAILY_SYNC_SHARES ← HOST*_DAILY_SYNC_SHARES\n375\t# INTERMEDIATE_SYNC_SHARES ← HOST*_INTERMEDIATE_SYNC_SHARES\n376\t# WEEKLY_SYNC_SHARES ← HOST*_WEEKLY_SYNC_SHARES\n377\t# CRITICAL_SYNC_SHARES ← HOST*_CRITICAL_SYNC_SHARES\n378\t# DAILY_RESTART_CONTAINERS ← HOST*_DAILY_RESTART_CONTAINERS\n379\t# WEEKLY_RESTART_CONTAINERS ← HOST*_WEEKLY_RESTART_CONTAINERS\n380\t# WATCHDOG_CONTAINERS ← HOST*_WATCHDOG_CONTAINERS (associative)\n381\t# WATCHDOG_CONTAINER_URLS ← HOST*_WATCHDOG_CONTAINER_URLS (associative)\n382\t# WATCHDOG_REQUIRED_CONTAINERS ← HOST*_WATCHDOG_REQUIRED_CONTAINERS\n383\t# WATCHDOG_SCAN_IGNORE ← HOST*_WATCHDOG_SCAN_IGNORE\n384\t# WATCHDOG_DEPENDENCIES ← HOST*_WATCHDOG_DEPENDENCIES (associative)\n385\t# WATCHDOG_APPDATA_SIZES ← HOST*_WATCHDOG_APPDATA_SIZES (associative)\n386\t# NETWORK_CONNECT_CONTAINERS ← HOST*_NETWORK_CONNECT_CONTAINERS\n387\t# NETWORK_CONNECT_NETWORKS ← HOST*_NETWORK_CONNECT_NETWORKS\n388\t# MEDIA_PERMISSION_SHARES ← HOST*_MEDIA_PERMISSION_SHARES\n389\t# ANIME_CLEAN_FOLDERS ← HOST*_ANIME_CLEAN_FOLDERS\n390\t# MEDIA_CLEAN_FOLDERS ← HOST*_MEDIA_CLEAN_FOLDERS\n391\t# CERT_MONITOR_DOMAINS ← HOST*_CERT_MONITOR_DOMAINS\n392\t# SMART_IGNORE_DRIVES ← HOST*_SMART_IGNORE_DRIVES\n393\t# ZFS_REPORT_IGNORE_POOLS ← HOST*_ZFS_REPORT_IGNORE_POOLS\n394\t# TRANSCODE_SSD ← HOST*_TRANSCODE_SSD\n395\t# TRANSCODE_SERVERS ← HOST*_TRANSCODE_SERVERS\n396\t# RAMDISK_SIZE ← HOST*_RAMDISK_SIZE\n397\t# RAMDISK_WARN_GB ← HOST*_RAMDISK_WARN_GB\n398\t# RAMDISK_LOW_GB ← HOST*_RAMDISK_LOW_GB\n399\t# BACKUP_VERIFY_SHARES ← HOST*_BACKUP_VERIFY_SHARES\n400\t# DDNS_CONTAINERS ← HOST*_DDNS_CONTAINERS\n401\t# PARTNERSHIP_AUTH_WEBUIS ← HOST*_PARTNERSHIP_AUTH_WEBUIS\n402\t# PARTNERSHIP_MIRROR_BACKUPS ← HOST*_PARTNERSHIP_MIRROR_BACKUPS\n403\t# MY_DISCORD_WEBHOOK ← HOST*_DISCORD_WEBHOOK\n404\t# EMBY_CONTAINER ← HOST*_EMBY_CONTAINER\n405\t# EMBY_URL ← HOST*_EMBY_URL\n406\t# EMBY_API_KEY ← HOST*_EMBY_API_KEY\n407\t# GITEA_API_TOKEN ← HOST*_GITEA_API_TOKEN\n408\t#\n409\t# Arr-specific vars set by detect_hosts() when needed by arr scripts:\n410\t# LIDARR_URL / LIDARR_API_KEY / LIDARR_MUSIC_ROOT (MY_ID only — Lidarr is HOST1 only)\n411\t# SONARR_URL / SONARR_API_KEY / SONARR_TV_ROOT\n412\t# RADARR_URL / RADARR_API_KEY / RADARR_MOVIES_ROOT\n413\t# SLSKD_URL / SLSKD_API_KEY / SLSKD_FAILED_IMPORTS_DIR (MY_ID only)\n414\t# SABNZBD_URL / SABNZBD_API_KEY (MY_ID only)\n415\t# QBIT_URL / QBIT_USERNAME / QBIT_PASSWORD (MY_ID only)\n416\t\n417\tdetect_hosts() {\n418\t local local_hostname\n419\t local_hostname=\"$(hostname)\"\n420\t\n421\t # ── Find MY_ID by matching hostname against all HOST* vars ────────────────\n422\t MY_ID=\"\"\n423\t local host_var host_val\n424\t\n425\t # Scan all HOST* vars that hold a hostname value\n426\t # HOST1, HOST2, HOST3 etc. — all defined in host*.conf files\n427\t for host_var in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do\n428\t host_val=\"${!host_var:-}\"\n429\t [[ -z \"$host_val\" ]] && continue\n430\t if [[ \"$local_hostname\" == \"$host_val\" ]]; then\n431\t MY_ID=\"$host_var\"\n432\t break\n433\t fi\n434\t done\n435\t\n436\t if [[ -z \"$MY_ID\" ]]; then\n437\t error \"Unknown host: $local_hostname\"\n438\t error \"Hostname must match a HOST* value in host*.conf\"\n439\t error \"Available: $(for h in HOST1 HOST2 HOST3 HOST4; do\n440\t [[ -n \"${!h:-}\" ]] && echo -n \"${!h} \"; done)\"\n441\t exit 1\n442\t fi\n443\t\n444\t # ── Find REMOTE_ID — first HOST* that isn't MY_ID ────────────────────────\n445\t REMOTE_ID=\"\"\n446\t for host_var in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do\n447\t [[ \"$host_var\" == \"$MY_ID\" ]] && continue\n448\t host_val=\"${!host_var:-}\"\n449\t [[ -z \"$host_val\" ]] && continue\n450\t REMOTE_ID=\"$host_var\"\n451\t break\n452\t done\n453\t\n454\t # ── Set host name strings ─────────────────────────────────────────────────\n455\t LOCAL_SERVER_NAME=\"${!MY_ID}\"\n456\t [[ -n \"$REMOTE_ID\" ]] && REMOTE_SERVER_NAME=\"${!REMOTE_ID}\" || REMOTE_SERVER_NAME=\"unknown\"\n457\t\n458\t # ── Set SSH key ───────────────────────────────────────────────────────────\n459\t local ssh_key_var=\"${MY_ID}_SSH_KEY\"\n460\t SSH_KEY=\"${!ssh_key_var:-}\"\n461\t if [[ -z \"$SSH_KEY\" ]]; then\n462\t error \"Missing SSH key: ${ssh_key_var} not set in host*.conf\"\n463\t exit 1\n464\t fi\n465\t\n466\t # ── Set scalar aliases ────────────────────────────────────────────────────\n467\t MY_DISCORD_WEBHOOK_VAR=\"${MY_ID}_DISCORD_WEBHOOK\"\n468\t MY_DISCORD_WEBHOOK=\"${!MY_DISCORD_WEBHOOK_VAR:-}\"\n469\t\n470\t OWNER_NAME_VAR=\"${MY_ID}_OWNER\"\n471\t OWNER_NAME=\"${!OWNER_NAME_VAR:-}\"\n472\t OWNER_EMAIL_VAR=\"${MY_ID}_OWNER_EMAIL\"\n473\t OWNER_EMAIL=\"${!OWNER_EMAIL_VAR:-}\"\n474\t\n475\t EMBY_CONTAINER_VAR=\"${MY_ID}_EMBY_CONTAINER\"\n476\t EMBY_CONTAINER=\"${!EMBY_CONTAINER_VAR:-}\"\n477\t EMBY_URL_VAR=\"${MY_ID}_EMBY_URL\"\n478\t EMBY_URL=\"${!EMBY_URL_VAR:-}\"\n479\t EMBY_API_KEY_VAR=\"${MY_ID}_EMBY_API_KEY\"\n480\t EMBY_API_KEY=\"${!EMBY_API_KEY_VAR:-}\"\n481\t\n482\t GITEA_API_TOKEN_VAR=\"${MY_ID}_GITEA_API_TOKEN\"\n483\t GITEA_API_TOKEN=\"${!GITEA_API_TOKEN_VAR:-}\"\n484\t\n485\t local _prov_var\n486\t for _prov_var in \\\n487\t PARTNERSHIP_PROVISION_EMBY_ADMIN \\\n488\t PARTNERSHIP_EMBY_ADMIN_USER \\\n489\t PARTNERSHIP_EMBY_ADMIN_PASS \\\n490\t PARTNERSHIP_EMBY_PORT; do\n491\t local _src=\"${MY_ID}_${_prov_var}\"\n492\t printf -v \"$_prov_var\" '%s' \"${!_src:-}\"\n493\t done\n494\t\n495\t # ── System watchdog check toggles — per-host ─────────────────────────────\n496\t # Aliased as unprefixed SYS_WATCHDOG_CHECK_* for use in system_watchdog.sh\n497\t local _wd_checks=(\n498\t SYS_WATCHDOG_NIC\n499\t SYS_WATCHDOG_CHECK_DOCKER_DAEMON SYS_WATCHDOG_CHECK_ROOTFS\n500\t SYS_WATCHDOG_CHECK_KERNEL_OOPS SYS_WATCHDOG_CHECK_FD\n501\t SYS_WATCHDOG_CHECK_BOOT SYS_WATCHDOG_CHECK_OOM\n502\t SYS_WATCHDOG_CHECK_RAM SYS_WATCHDOG_CHECK_LOG\n503\t SYS_WATCHDOG_CHECK_ARC SYS_WATCHDOG_CHECK_CPU_TEMP\n504\t SYS_WATCHDOG_CHECK_LOAD SYS_WATCHDOG_CHECK_ZOMBIES\n505\t SYS_WATCHDOG_CHECK_CONTAINERS SYS_WATCHDOG_CHECK_TMP\n506\t SYS_WATCHDOG_CHECK_MDSTAT SYS_WATCHDOG_CHECK_NETWORK\n507\t SYS_WATCHDOG_CHECK_SSHD SYS_WATCHDOG_CHECK_RUNAWAY\n508\t )\n509\t for _wd_var in \"${_wd_checks[@]}\"; do\n510\t local _wd_src=\"${MY_ID}_${_wd_var}\"\n511\t # Only alias if the host-specific var is set — preserves master.conf defaults\n512\t [[ -n \"${!_wd_src+x}\" ]] && eval \"${_wd_var}=\\\"\\${${_wd_src}}\\\"\"\n513\t done\n514\t\n515\t TRANSCODE_SSD_VAR=\"${MY_ID}_TRANSCODE_SSD\"\n516\t TRANSCODE_SSD=\"${!TRANSCODE_SSD_VAR:-}\"\n517\t RAMDISK_SIZE_VAR=\"${MY_ID}_RAMDISK_SIZE\"\n518\t RAMDISK_SIZE=\"${!RAMDISK_SIZE_VAR:-8G}\"\n519\t RAMDISK_WARN_GB_VAR=\"${MY_ID}_RAMDISK_WARN_GB\"\n520\t RAMDISK_WARN_GB=\"${!RAMDISK_WARN_GB_VAR:-6.8}\"\n521\t RAMDISK_LOW_GB_VAR=\"${MY_ID}_RAMDISK_LOW_GB\"\n522\t RAMDISK_LOW_GB=\"${!RAMDISK_LOW_GB_VAR:-5.5}\"\n523\t\n524\t # Arr credentials — MY_ID only (remote arr accessed via its own host)\n525\t LIDARR_URL_VAR=\"${MY_ID}_LIDARR_URL\"; LIDARR_URL=\"${!LIDARR_URL_VAR:-}\"\n526\t LIDARR_API_KEY_VAR=\"${MY_ID}_LIDARR_API_KEY\"; LIDARR_API_KEY=\"${!LIDARR_API_KEY_VAR:-}\"\n527\t LIDARR_MUSIC_ROOT_VAR=\"${MY_ID}_LIDARR_MUSIC_ROOT\"; LIDARR_MUSIC_ROOT=\"${!LIDARR_MUSIC_ROOT_VAR:-}\"\n528\t FANART_API_KEY_VAR=\"${MY_ID}_FANART_API_KEY\"; FANART_API_KEY=\"${!FANART_API_KEY_VAR:-}\"\n529\t LASTFM_API_KEY_VAR=\"${MY_ID}_LASTFM_API_KEY\"; LASTFM_API_KEY=\"${!LASTFM_API_KEY_VAR:-}\"\n530\t TMDB_API_KEY_VAR=\"${MY_ID}_TMDB_API_KEY\"; TMDB_API_KEY=\"${!TMDB_API_KEY_VAR:-}\"\n531\t\n532\t SONARR_URL_VAR=\"${MY_ID}_SONARR_URL\"; SONARR_URL=\"${!SONARR_URL_VAR:-}\"\n533\t SONARR_API_KEY_VAR=\"${MY_ID}_SONARR_API_KEY\"; SONARR_API_KEY=\"${!SONARR_API_KEY_VAR:-}\"\n534\t SONARR_TV_ROOT_VAR=\"${MY_ID}_SONARR_TV_ROOT\"; SONARR_TV_ROOT=\"${!SONARR_TV_ROOT_VAR:-}\"\n535\t\n536\t RADARR_URL_VAR=\"${MY_ID}_RADARR_URL\"; RADARR_URL=\"${!RADARR_URL_VAR:-}\"\n537\t RADARR_API_KEY_VAR=\"${MY_ID}_RADARR_API_KEY\"; RADARR_API_KEY=\"${!RADARR_API_KEY_VAR:-}\"\n538\t RADARR_MOVIES_ROOT_VAR=\"${MY_ID}_RADARR_MOVIES_ROOT\"; RADARR_MOVIES_ROOT=\"${!RADARR_MOVIES_ROOT_VAR:-}\"\n539\t\n540\t SLSKD_URL_VAR=\"${MY_ID}_SLSKD_URL\"; SLSKD_URL=\"${!SLSKD_URL_VAR:-}\"\n541\t SLSKD_API_KEY_VAR=\"${MY_ID}_SLSKD_API_KEY\"; SLSKD_API_KEY=\"${!SLSKD_API_KEY_VAR:-}\"\n542\t SLSKD_FAILED_IMPORTS_DIR_VAR=\"${MY_ID}_SLSKD_FAILED_IMPORTS_DIR\"\n543\t SLSKD_FAILED_IMPORTS_DIR=\"${!SLSKD_FAILED_IMPORTS_DIR_VAR:-}\"\n544\t\n545\t SABNZBD_URL_VAR=\"${MY_ID}_SABNZBD_URL\"; SABNZBD_URL=\"${!SABNZBD_URL_VAR:-}\"\n546\t SABNZBD_API_KEY_VAR=\"${MY_ID}_SABNZBD_API_KEY\"; SABNZBD_API_KEY=\"${!SABNZBD_API_KEY_VAR:-}\"\n547\t\n548\t QBIT_URL_VAR=\"${MY_ID}_QBIT_URL\"; QBIT_URL=\"${!QBIT_URL_VAR:-}\"\n549\t QBIT_USERNAME_VAR=\"${MY_ID}_QBIT_USERNAME\"; QBIT_USERNAME=\"${!QBIT_USERNAME_VAR:-}\"\n550\t QBIT_PASSWORD_VAR=\"${MY_ID}_QBIT_PASSWORD\"; QBIT_PASSWORD=\"${!QBIT_PASSWORD_VAR:-}\"\n551\t\n552\t # ── Set array aliases — indexed arrays ────────────────────────────────────\n553\t # Each eval copies the host-specific array into the unprefixed name scripts use\n554\t\n555\t _alias_array() {\n556\t local alias_name=\"$1\"\n557\t local source_var=\"${MY_ID}_${alias_name}\"\n558\t eval \"${alias_name}=(\\\"\\${${source_var}[@]}\\\")\"\n559\t }\n560\t\n561\t _alias_array \"DAILY_SYNC_SHARES\"\n562\t _alias_array \"PERSONAL_SHARES\"\n563\t _alias_array \"INTERMEDIATE_SYNC_SHARES\"\n564\t _alias_array \"WEEKLY_SYNC_SHARES\"\n565\t _alias_array \"CRITICAL_SYNC_SHARES\"\n566\t _alias_array \"BACKUP_VERIFY_SHARES\"\n567\t _alias_array \"DAILY_RESTART_CONTAINERS\"\n568\t _alias_array \"WEEKLY_RESTART_CONTAINERS\"\n569\t _alias_array \"WATCHDOG_REQUIRED_CONTAINERS\"\n570\t _alias_array \"WATCHDOG_SCAN_IGNORE\"\n571\t _alias_array \"NETWORK_CONNECT_CONTAINERS\"\n572\t _alias_array \"NETWORK_CONNECT_NETWORKS\"\n573\t _alias_array \"MEDIA_PERMISSION_SHARES\"\n574\t _alias_array \"ANIME_CLEAN_FOLDERS\"\n575\t _alias_array \"MEDIA_CLEAN_FOLDERS\"\n576\t _alias_array \"CERT_MONITOR_DOMAINS\"\n577\t _alias_array \"SMART_IGNORE_DRIVES\"\n578\t _alias_array \"ZFS_REPORT_IGNORE_POOLS\"\n579\t _alias_array \"TRANSCODE_SERVERS\"\n580\t _alias_array \"DDNS_CONTAINERS\"\n581\t _alias_array \"PARTNERSHIP_AUTH_WEBUIS\"\n582\t _alias_array \"PARTNERSHIP_MIRROR_BACKUPS\"\n583\t _alias_array \"PARTNERSHIP_OWN_CONTAINERS\"\n584\t _alias_array \"PARTNERSHIP_AUTH_STACK\"\n585\t _alias_array \"PARTNERSHIP_REPLACE_CONTAINERS\"\n586\t _alias_array \"PARTNERSHIP_ARR_STACK\"\n587\t _alias_array \"PARTNERSHIP_ARR_REPLACE_CONTAINERS\"\n588\t _alias_array \"PARTNERSHIP_SERVICES_STACK\"\n589\t _alias_array \"PARTNERSHIP_SERVICES_REPLACE_CONTAINERS\"\n590\t _alias_array \"RW_PAUSE_CONTAINERS\"\n591\t _alias_array \"RW_STOP_CONTAINERS\"\n592\t\n593\t # ── Set array aliases — associative arrays ────────────────────────────────\n594\t # Associative arrays cannot be copied with eval — must be rebuilt key by key\n595\t\n596\t _alias_assoc() {\n597\t local alias_name=\"$1\"\n598\t local source_name=\"${MY_ID}_${alias_name}\"\n599\t eval \"declare -gA ${alias_name}\"\n600\t local -n _assoc_src=\"${source_name}\" 2>/dev/null || return 0\n601\t local -n _assoc_dst=\"${alias_name}\"\n602\t local _assoc_key\n603\t for _assoc_key in \"${!_assoc_src[@]}\"; do\n604\t _assoc_dst[\"$_assoc_key\"]=\"${_assoc_src[$_assoc_key]}\"\n605\t done\n606\t }\n607\t\n608\t _alias_assoc \"WATCHDOG_CONTAINERS\"\n609\t _alias_assoc \"WATCHDOG_CONTAINER_URLS\"\n610\t _alias_assoc \"WATCHDOG_CONTAINER_API_CHECKS\"\n611\t _alias_assoc \"WATCHDOG_DEPENDENCIES\"\n612\t _alias_assoc \"WATCHDOG_APPDATA_SIZES\"\n613\t\n614\t # ── Remote storage path ───────────────────────────────────────────────────\n615\t # Resolved from REMOTE_ID's host*.conf so SSH commands use the correct path\n616\t # even when the remote is a different platform. Falls back to the local\n617\t # platform_storage_path() for unconfigured same-platform pairs.\n618\t local _rsp_var=\"${REMOTE_ID}_STORAGE_PATH\"\n619\t REMOTE_STORAGE_PATH=\"${!_rsp_var:-$(platform_storage_path)}\"\n620\t\n621\t # ── Output ────────────────────────────────────────────────────────────────\n622\t log \"$ICON_HOST Host: $LOCAL_SERVER_NAME → $REMOTE_SERVER_NAME\"\n623\t}\n624\t\n625\t# ==============================================================================================\n626\t# ── REMOTE IP RESOLUTION ──────────────────────────────────────────────────────────────────────\n627\t# ==============================================================================================\n628\t# Resolves the Tailscale IPv4 address of the remote server.\n629\t# Sets REMOTE_SERVER used by all subsequent SSH and rsync calls.\n630\t# Exits if resolution fails — Tailscale may be down or peer offline.\n631\t\n632\tresolve_remote_ip() {\n633\t log \"Resolving remote IP for $REMOTE_SERVER_NAME...\"\n634\t local ts_name=\"${REMOTE_SERVER_NAME,,}\"\n635\t local attempts=3\n636\t\n637\t for ((i=1; i<=attempts; i++)); do\n638\t # Direct lookup — works when MagicDNS short-name resolution is active\n639\t REMOTE_SERVER=$(tailscale ip -4 \"$ts_name\" 2>/dev/null)\n640\t\n641\t # Fallback — parse tailscale status for FQDN entries (e.g. hostname.tailXXXX.ts.net)\n642\t if [[ -z \"$REMOTE_SERVER\" ]]; then\n643\t REMOTE_SERVER=$(tailscale status 2>/dev/null | awk -v name=\"$ts_name\" '$2 ~ \"^\" name { print $1; exit }')\n644\t fi\n645\t\n646\t [[ -n \"$REMOTE_SERVER\" ]] && break\n647\t [[ $i -lt $attempts ]] && { warn \"Tailscale resolution attempt $i/$attempts failed — retrying in 5s...\"; sleep 5; }\n648\t done\n649\t\n650\t if [[ -z \"$REMOTE_SERVER\" ]]; then\n651\t error \"Failed to resolve Tailscale IP for $REMOTE_SERVER_NAME after $attempts attempts\"\n652\t error \"Check: tailscale status | grep $ts_name\"\n653\t exit 1\n654\t fi\n655\t info \"$ICON_NET Remote IP: $REMOTE_SERVER\"\n656\t}\n657\t\n658\t# Resolve any hostname to a Tailscale IPv4 — tries direct lookup, falls back to status parse.\n659\t# Usage: ip=$(resolve_tailscale_ip \"hostname\") — returns empty string on failure.\n660\tresolve_tailscale_ip() {\n661\t local hostname=\"${1,,}\"\n662\t local ip\n663\t ip=$(tailscale ip -4 \"$hostname\" 2>/dev/null)\n664\t [[ -z \"$ip\" ]] && \\\n665\t ip=$(tailscale status 2>/dev/null | awk -v name=\"$hostname\" '$2 ~ \"^\" name { print $1; exit }')\n666\t echo \"$ip\"\n667\t}\n668\t\n669\t# ==============================================================================================\n670\t# ── CONNECTIVITY CHECKS ───────────────────────────────────────────────────────────────────────\n671\t# ==============================================================================================\n672\t\n673\t# Fatal connectivity check — used by rsync and other scripts that must abort if unreachable.\n674\t# For fallback use ping_remote() which returns status without exiting.\n675\tcheck_connectivity() {\n676\t log \"Checking connectivity to $REMOTE_SERVER...\"\n677\t if ! ping -c1 -W3 \"$REMOTE_SERVER\" &>/dev/null; then\n678\t error \"$ICON_PING Remote $REMOTE_SERVER ($REMOTE_SERVER_NAME) is unreachable\"\n679\t info \"Hint: tailscale status | grep $REMOTE_SERVER_NAME\"\n680\t exit 1\n681\t fi\n682\t info \"$ICON_PING $REMOTE_SERVER_NAME is reachable\"\n683\t}\n684\t\n685\t# Non-fatal ping — used by fallback.sh which handles its own state machine.\n686\t# Returns 0 if reachable, 1 if not — does NOT exit.\n687\tping_remote() {\n688\t ping -c2 -W3 \"$REMOTE_SERVER\" &>/dev/null\n689\t}\n690\t\n691\t# Non-fatal external connectivity check — used by fallback.sh.\n692\t# Returns 0 if internet reachable, 1 if not — does NOT exit.\n693\tping_internet() {\n694\t ping -c2 -W3 \"${EXTERNAL_IP:-8.8.8.8}\" &>/dev/null\n695\t}\n696\t\n697\t# ==============================================================================================\n698\t# ── RSYNC GATE CHECK ──────────────────────────────────────────────────────────────────────────\n699\t# ==============================================================================================\n700\t# Two-tier rsync enable/disable system.\n701\t# Tier 1: RSYNC_ENABLED — global gate, overrides everything\n702\t# Tier 2: orchestrator-specific flag — fine-grained control per window\n703\t#\n704\t# Usage:\n705\t# check_rsync_enabled \"CRITICAL\" ← checks RSYNC_ENABLED + CRITICAL_RSYNC_ENABLED\n706\t# check_rsync_enabled \"INTERMEDIATE\" ← checks RSYNC_ENABLED + INTERMEDIATE_RSYNC_ENABLED\n707\t# check_rsync_enabled \"DAILY\" ← checks RSYNC_ENABLED + DAILY_RSYNC_ENABLED\n708\t# check_rsync_enabled \"WEEKLY\" ← checks RSYNC_ENABLED + WEEKLY_RSYNC_ENABLED\n709\t# check_rsync_enabled \"FALLBACK\" ← checks RSYNC_ENABLED + FALLBACK_RSYNC_ENABLED\n710\t# check_rsync_enabled ← checks RSYNC_ENABLED only (direct rsync.sh call)\n711\t#\n712\t# Returns: 0 = enabled, proceed | 1 = disabled, skip cleanly\n713\t\n714\tcheck_rsync_enabled() {\n715\t local orchestrator=\"${1:-}\"\n716\t\n717\t # Tier 1 — global gate\n718\t if [[ \"${RSYNC_ENABLED:-true}\" == false ]]; then\n719\t warn \"RSYNC_ENABLED=false — rsync globally disabled, skipping all syncs\"\n720\t return 1\n721\t fi\n722\t\n723\t # Tier 2 — per-orchestrator\n724\t if [[ -n \"$orchestrator\" ]]; then\n725\t local var_name=\"${orchestrator}_RSYNC_ENABLED\"\n726\t local var_value=\"${!var_name:-true}\"\n727\t if [[ \"$var_value\" == false ]]; then\n728\t info \"${var_name}=false — rsync disabled for $orchestrator orchestrator\"\n729\t info \"All other $orchestrator jobs will still run normally\"\n730\t return 1\n731\t fi\n732\t fi\n733\t\n734\t return 0\n735\t}\n736\t\n737\t# ==============================================================================================\n738\t# ── PLATFORM SERVICE STATE ────────────────────────────────────────────────────────────────────\n739\t# ==============================================================================================\n740\t# Check whether Docker and VM Manager are enabled on this host.\n741\t# Delegates to platform_is_service_enabled() from the platform adapter.\n742\t\n743\tis_docker_enabled() {\n744\t platform_is_service_enabled docker\n745\t}\n746\t\n747\tis_vm_manager_enabled() {\n748\t platform_is_service_enabled libvirt\n749\t}\n750\t\n751\t# ==============================================================================================\n752\t# ── LOCAL HEALTH CHECKS ───────────────────────────────────────────────────────────────────────\n753\t# ==============================================================================================\n754\t\n755\t# Verifies local storage is mounted and has shares.\n756\t# Non-fatal — returns status for caller to decide.\n757\t# Used by fallback before starting remote containers locally.\n758\tcheck_local_array() {\n759\t log \"Checking local array...\"\n760\t if ! platform_storage_healthy; then\n761\t error \"$ICON_DISK Local array is not started — storage not mounted\"\n762\t return 1\n763\t fi\n764\t local _storage_path file_count\n765\t _storage_path=$(platform_storage_path)\n766\t file_count=$(ls \"$_storage_path\" 2>/dev/null | wc -l)\n767\t if [[ \"$file_count\" -eq 0 ]]; then\n768\t error \"$ICON_DISK Local array appears empty — shares may not be available\"\n769\t return 1\n770\t fi\n771\t log \"Local array is healthy\"\n772\t return 0\n773\t}\n774\t\n775\t# ==============================================================================================\n776\t# ── REMOTE HEALTH CHECKS ─────────────────────────────────────────────────────────────────────\n777\t# ==============================================================================================\n778\t\n779\t# Verifies remote storage is mounted via SSH.\n780\t# Non-fatal — returns status. Used before handback rsync.\n781\t# Syncing to remote with no array fills rootfs rapidly.\n782\tcheck_remote_array() {\n783\t log \"Checking remote array on $REMOTE_SERVER_NAME...\"\n784\t local _spath result\n785\t _spath=\"$REMOTE_STORAGE_PATH\"\n786\t result=$(ssh -i \"$SSH_KEY\" -o ConnectTimeout=10 root@\"$REMOTE_SERVER\" \\\n787\t \"mountpoint -q '$_spath' && echo yes || echo no\" 2>/dev/null)\n788\t if [[ \"$result\" != \"yes\" ]]; then\n789\t error \"$ICON_DISK Remote array not started on $REMOTE_SERVER_NAME\"\n790\t return 1\n791\t fi\n792\t log \"Remote array is healthy\"\n793\t return 0\n794\t}\n795\t\n796\t# Verifies remote Docker daemon is responding.\n797\t# Non-fatal — returns status. A hung daemon means container commands silently fail.\n798\tcheck_remote_docker() {\n799\t log \"Checking remote Docker daemon on $REMOTE_SERVER_NAME...\"\n800\t if ! ssh -i \"$SSH_KEY\" -o ConnectTimeout=10 root@\"$REMOTE_SERVER\" \\\n801\t \"timeout 10 docker ps\" >/dev/null 2>&1; then\n802\t error \"$ICON_CONTAINERS Remote Docker daemon not responding on $REMOTE_SERVER_NAME\"\n803\t return 1\n804\t fi\n805\t log \"Remote Docker daemon is healthy\"\n806\t return 0\n807\t}\n808\t\n809\t# Aborts if remote rootfs (/) usage is at or above ROOTFS_WARN threshold.\n810\t# Fatal — exits the calling script.\n811\t# When remote array is down rsync writes land on rootfs and fill it rapidly.\n812\tcheck_remote_rootfs() {\n813\t log \"Checking remote rootfs usage...\"\n814\t local remote_usage\n815\t remote_usage=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n816\t \"df / --output=pcent | tail -1 | tr -d ' %'\" 2>/dev/null)\n817\t if [[ -z \"$remote_usage\" ]]; then\n818\t error \"Could not retrieve rootfs usage from $REMOTE_SERVER_NAME\"\n819\t exit 1\n820\t fi\n821\t if [[ \"$remote_usage\" -ge \"${ROOTFS_WARN:-75}\" ]]; then\n822\t error \"$ICON_HEALTH Remote rootfs ${remote_usage}% — threshold ${ROOTFS_WARN:-75}%\"\n823\t exit 1\n824\t fi\n825\t info \"$ICON_HEALTH Remote rootfs: ${remote_usage}% (threshold: ${ROOTFS_WARN:-75}%)\"\n826\t}\n827\t\n828\t# Verifies target directory exists and is not empty on remote.\n829\t# Fatal — exits the calling script.\n830\t# Usage: check_remote_share \"/mnt/user/Movies\"\n831\tcheck_remote_share() {\n832\t local dir=\"$1\"\n833\t log \"Checking remote share: $dir...\"\n834\t\n835\t local share_exists\n836\t share_exists=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n837\t \"[[ -d '$dir' ]] && echo yes || echo no\" 2>/dev/null)\n838\t if [[ \"$share_exists\" != \"yes\" ]]; then\n839\t error \"$ICON_HEALTH Remote share does not exist: $dir\"\n840\t exit 1\n841\t fi\n842\t\n843\t local share_empty\n844\t share_empty=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n845\t \"[[ -z \\\"\\$(ls -A '$dir' 2>/dev/null)\\\" ]] && echo yes || echo no\" 2>/dev/null)\n846\t if [[ \"$share_empty\" == \"yes\" ]]; then\n847\t warn \"$ICON_HEALTH Remote share exists but is empty: $dir — aborting to protect data\"\n848\t exit 1\n849\t fi\n850\t\n851\t info \"$ICON_HEALTH Remote share verified: $dir\"\n852\t}\n853\t\n854\t# ==============================================================================================\n855\t# ── DISK TEMPERATURE CHECKS ───────────────────────────────────────────────────────────────────\n856\t# ==============================================================================================\n857\t\n858\t# Returns 0 (true) if device is non-rotational (SSD/NVMe), 1 (false) if HDD.\n859\t# Checks /sys/block//queue/rotational — 0=SSD, 1=HDD.\n860\tis_ssd() {\n861\t local base\n862\t base=$(basename \"$1\")\n863\t [[ \"$(cat \"/sys/block/$base/queue/rotational\" 2>/dev/null)\" == \"0\" ]]\n864\t}\n865\t\n866\t# Sets globals: UNRAID_DISK_HOT UNRAID_DISK_MAX UNRAID_SSD_HOT UNRAID_SSD_MAX\n867\t# Falls back to master.conf values (SMART_TEMP_WARN/CRIT) if platform returns defaults.\n868\tget_unraid_temp_thresholds() {\n869\t read -r UNRAID_DISK_HOT UNRAID_DISK_MAX UNRAID_SSD_HOT UNRAID_SSD_MAX \\\n870\t <<< \"$(platform_get_temp_thresholds)\"\n871\t UNRAID_DISK_HOT=\"${UNRAID_DISK_HOT:-${SMART_TEMP_WARN:-45}}\"\n872\t UNRAID_DISK_MAX=\"${UNRAID_DISK_MAX:-${SMART_TEMP_CRIT:-55}}\"\n873\t UNRAID_SSD_HOT=\"${UNRAID_SSD_HOT:-60}\"\n874\t UNRAID_SSD_MAX=\"${UNRAID_SSD_MAX:-70}\"\n875\t}\n876\t\n877\t# Checks local disk temperatures before rsync.\n878\t# Reads temps and rotational flag from /var/local/emhttp/disks.ini.\n879\t# Uses unRAID's own thresholds from dynamix.cfg via get_unraid_temp_thresholds().\n880\t#\n881\t# Returns:\n882\t# 0 = all temps OK — proceed\n883\t# 1 = warn threshold exceeded — skip this profile only\n884\t# 2 = critical threshold exceeded — abort all remaining profiles\n885\t#\n886\t# Sets TEMP_CHECK_RESULT with human readable summary for logging.\n887\tcheck_local_disk_temps() {\n888\t get_unraid_temp_thresholds\n889\t\n890\t local disks_ini_content\n891\t if ! disks_ini_content=$(platform_get_disk_states); then\n892\t warn \"disk state unavailable — skipping temp check\"\n893\t TEMP_CHECK_RESULT=\"temp check skipped (disk state unavailable)\"\n894\t return 0\n895\t fi\n896\t\n897\t local worst_result=0\n898\t local hot_drives=()\n899\t local crit_drives=()\n900\t local current_name=\"\"\n901\t local current_device=\"\"\n902\t local current_rotational=\"\"\n903\t local current_temp=\"\"\n904\t\n905\t check_drive() {\n906\t [[ -z \"$current_name\" ]] || [[ -z \"$current_temp\" ]] && return\n907\t [[ \"$current_temp\" -eq 0 ]] && return # spun down — skip\n908\t\n909\t local warn_thresh crit_thresh\n910\t if [[ \"$current_rotational\" == \"0\" ]]; then\n911\t warn_thresh=\"$UNRAID_SSD_HOT\"\n912\t crit_thresh=\"$UNRAID_SSD_MAX\"\n913\t else\n914\t warn_thresh=\"$UNRAID_DISK_HOT\"\n915\t crit_thresh=\"$UNRAID_DISK_MAX\"\n916\t fi\n917\t\n918\t if [[ \"$current_temp\" -ge \"$crit_thresh\" ]]; then\n919\t crit_drives+=(\"${current_name}(${current_device}):${current_temp}°C≥${crit_thresh}°C\")\n920\t [[ $worst_result -lt 2 ]] && worst_result=2\n921\t elif [[ \"$current_temp\" -ge \"$warn_thresh\" ]]; then\n922\t hot_drives+=(\"${current_name}(${current_device}):${current_temp}°C≥${warn_thresh}°C\")\n923\t [[ $worst_result -lt 1 ]] && worst_result=1\n924\t fi\n925\t }\n926\t\n927\t while IFS= read -r ini_line; do\n928\t if echo \"$ini_line\" | grep -qE '^\\[\"(disk[0-9]+|parity[0-9]?|cache[0-9]?)\"\\]'; then\n929\t check_drive\n930\t current_name=$(echo \"$ini_line\" | grep -o '\"[^\"]*\"' | head -1 | tr -d '\"')\n931\t current_device=\"\"\n932\t current_rotational=\"1\" # default HDD\n933\t current_temp=\"\"\n934\t elif echo \"$ini_line\" | grep -q '^device='; then\n935\t current_device=$(echo \"$ini_line\" | cut -d= -f2 | tr -d '\"')\n936\t elif echo \"$ini_line\" | grep -q '^rotational='; then\n937\t current_rotational=$(echo \"$ini_line\" | cut -d= -f2 | tr -d '\"')\n938\t elif echo \"$ini_line\" | grep -q '^temp='; then\n939\t current_temp=$(echo \"$ini_line\" | cut -d= -f2 | tr -d '\"')\n940\t current_temp=\"${current_temp//[^0-9]/}\"\n941\t current_temp=\"${current_temp:-0}\"\n942\t fi\n943\t done <<< \"$disks_ini_content\"\n944\t check_drive # process last drive\n945\t\n946\t if [[ ${#crit_drives[@]} -gt 0 ]]; then\n947\t TEMP_CHECK_RESULT=\"CRITICAL temps: ${crit_drives[*]}\"\n948\t error \"Drive temp CRITICAL — aborting all remaining syncs: ${crit_drives[*]}\"\n949\t notify \"Rsync aborted on $(hostname) — drive temp CRITICAL: ${crit_drives[*]}\" \"Rsync Temp Check\" \"warning\"\n950\t return 2\n951\t elif [[ ${#hot_drives[@]} -gt 0 ]]; then\n952\t TEMP_CHECK_RESULT=\"HOT drives: ${hot_drives[*]}\"\n953\t warn \"Drive temp WARNING — skipping this profile: ${hot_drives[*]}\"\n954\t notify \"Rsync profile skipped on $(hostname) — drive temp WARNING: ${hot_drives[*]}\" \"Rsync Temp Check\" \"normal\"\n955\t return 1\n956\t else\n957\t TEMP_CHECK_RESULT=\"all normal (HDD warn:${UNRAID_DISK_HOT}°C crit:${UNRAID_DISK_MAX}°C SSD warn:${UNRAID_SSD_HOT}°C crit:${UNRAID_SSD_MAX}°C)\"\n958\t return 0\n959\t fi\n960\t}\n961\t\n962\t# ==============================================================================================\n963\t# ── REMOTE DISK VERIFICATION ──────────────────────────────────────────────────────────────────\n964\t# ==============================================================================================\n965\t# Verifies all disks backing a share are healthy on the remote server.\n966\t# Auto-detects filesystem type from disks.ini — handles XFS, ZFS, and cache pools.\n967\t# No configuration required — discovers disk layout at runtime.\n968\t#\n969\t# Three-tier detection handles all share locations:\n970\t# Tier 1: Array disk paths — /mnt/disk*/sharename (XFS or ZFS per-disk)\n971\t# Tier 2: ZFS standalone pools — finds nested paths up to 2 levels deep\n972\t# catches /mnt/cache/appdata-Fallback/Critical-Data correctly\n973\t# Tier 3: shfs fallback — if share exists under /mnt/user, resolve backing pool\n974\t# or verify shfs itself is mounted if pool can't be determined\n975\t#\n976\t# Fatal — exits the calling script if any disk is offline.\n977\t# Usage: check_remote_disks \"/mnt/user/Movies\"\n978\t\n979\tcheck_remote_disks() {\n980\t local dir=\"$1\"\n981\t local share_name\n982\t share_name=$(basename \"$dir\")\n983\t\n984\t info \"$ICON_DISK Checking disks backing $share_name on $REMOTE_SERVER_NAME...\"\n985\t\n986\t # Read remote disk state for fsType mapping\n987\t local disks_ini_content _disk_states_path\n988\t _disk_states_path=$(platform_disk_states_path)\n989\t disks_ini_content=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n990\t \"cat '$_disk_states_path' 2>/dev/null\" 2>/dev/null)\n991\t\n992\t if [[ -z \"$disks_ini_content\" ]]; then\n993\t error \"$ICON_DISK Cannot read disk state from $REMOTE_SERVER_NAME\"\n994\t exit 1\n995\t fi\n996\t\n997\t # Tier 1 — array disk paths (/mnt/disk*/sharename)\n998\t local backing_disks\n999\t backing_disks=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1000\t \"ls -d /mnt/disk*/$share_name 2>/dev/null | awk -F/ '{print \\$3}'\" 2>/dev/null)\n1001\t\n1002\t # Tier 2 — ZFS standalone pools, including nested paths\n1003\t local zfs_pool_paths\n1004\t zfs_pool_paths=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1005\t \"zpool list -H -o name 2>/dev/null | while read pool; do\n1006\t [[ -d \\\"/mnt/\\${pool}/$share_name\\\" ]] && echo \\\"\\$pool\\\" && continue\n1007\t find \\\"/mnt/\\${pool}\\\" -maxdepth 2 -name '$share_name' -type d 2>/dev/null | \\\n1008\t grep -q . && echo \\\"\\$pool\\\"\n1009\t done | sort -u\" 2>/dev/null)\n1010\t\n1011\t # Tier 3 — storage root fallback\n1012\t local _spath=\"$REMOTE_STORAGE_PATH\"\n1013\t if [[ -z \"$backing_disks\" ]] && [[ -z \"$zfs_pool_paths\" ]]; then\n1014\t local on_user\n1015\t on_user=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1016\t \"find '$_spath' -maxdepth 3 -name '$share_name' -type d 2>/dev/null | head -1\" \\\n1017\t 2>/dev/null)\n1018\t if [[ -n \"$on_user\" ]]; then\n1019\t local actual_path\n1020\t actual_path=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1021\t \"readlink -f '$on_user' 2>/dev/null || echo '$on_user'\" 2>/dev/null)\n1022\t local pool_name\n1023\t pool_name=$(echo \"$actual_path\" | awk -F/ '{print $3}')\n1024\t if [[ -n \"$pool_name\" ]] && [[ \"$pool_name\" != \"user\" ]]; then\n1025\t zfs_pool_paths=\"$pool_name\"\n1026\t info \"$ICON_DISK $share_name found at $actual_path on $REMOTE_SERVER_NAME\"\n1027\t else\n1028\t # Can't determine pool — verify storage root itself is mounted\n1029\t local shfs_ok\n1030\t shfs_ok=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1031\t \"mountpoint -q '$_spath' && echo yes || echo no\" 2>/dev/null)\n1032\t if [[ \"$shfs_ok\" == \"yes\" ]]; then\n1033\t success \"All disks backing $share_name are online ✅ (via storage root)\"\n1034\t return 0\n1035\t fi\n1036\t fi\n1037\t fi\n1038\t fi\n1039\t\n1040\t if [[ -z \"$backing_disks\" ]] && [[ -z \"$zfs_pool_paths\" ]]; then\n1041\t error \"$ICON_DISK No disks found backing $share_name on $REMOTE_SERVER_NAME\"\n1042\t exit 1\n1043\t fi\n1044\t\n1045\t local all_ok=true\n1046\t\n1047\t # Check array disks (XFS or ZFS single-disk-in-array)\n1048\t if [[ -n \"$backing_disks\" ]]; then\n1049\t while IFS= read -r disk_name; do\n1050\t [[ -z \"$disk_name\" ]] && continue\n1051\t\n1052\t local fs_type\n1053\t fs_type=$(echo \"$disks_ini_content\" | awk -F= -v disk=\"$disk_name\" '\n1054\t /^\\[\"'\"'\"'?/ { current=substr($0,3,length($0)-4) }\n1055\t current==disk && /^fsType=/ { print $2; exit }\n1056\t ' | tr -d '\"')\n1057\t fs_type=\"${fs_type:-xfs}\"\n1058\t\n1059\t if [[ \"$fs_type\" == \"zfs\" ]]; then\n1060\t local pool_health\n1061\t pool_health=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1062\t \"zpool list -H -o health '$disk_name' 2>/dev/null\" 2>/dev/null)\n1063\t if [[ \"$pool_health\" == \"ONLINE\" ]]; then\n1064\t info \"$ICON_DISK $disk_name (ZFS) $ICON_RUNNING — $share_name ONLINE\"\n1065\t else\n1066\t error \"$ICON_DISK $disk_name (ZFS) $ICON_STOPPED — pool ${pool_health:-offline}\"\n1067\t all_ok=false\n1068\t fi\n1069\t else\n1070\t local mounted\n1071\t mounted=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1072\t \"mountpoint -q '/mnt/$disk_name' && echo yes || echo no\" 2>/dev/null)\n1073\t if [[ \"$mounted\" == \"yes\" ]]; then\n1074\t info \"$ICON_DISK $disk_name (XFS) $ICON_RUNNING — $share_name present\"\n1075\t else\n1076\t error \"$ICON_DISK $disk_name (XFS) $ICON_STOPPED — not mounted\"\n1077\t all_ok=false\n1078\t fi\n1079\t fi\n1080\t done <<< \"$backing_disks\"\n1081\t fi\n1082\t\n1083\t # Check ZFS standalone pools (cache, gaming, media-servers etc.)\n1084\t if [[ -n \"$zfs_pool_paths\" ]]; then\n1085\t while IFS= read -r pool_name; do\n1086\t [[ -z \"$pool_name\" ]] && continue\n1087\t local pool_health\n1088\t pool_health=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1089\t \"zpool list -H -o health '$pool_name' 2>/dev/null\" 2>/dev/null)\n1090\t if [[ \"$pool_health\" == \"ONLINE\" ]]; then\n1091\t info \"$ICON_DISK $pool_name (ZFS pool) $ICON_RUNNING — $share_name ONLINE\"\n1092\t else\n1093\t error \"$ICON_DISK $pool_name (ZFS pool) $ICON_STOPPED — pool ${pool_health:-offline}\"\n1094\t all_ok=false\n1095\t fi\n1096\t done <<< \"$zfs_pool_paths\"\n1097\t fi\n1098\t\n1099\t if [[ \"$all_ok\" == false ]]; then\n1100\t error \"One or more disks backing $share_name are offline on $REMOTE_SERVER_NAME\"\n1101\t exit 1\n1102\t fi\n1103\t success \"All disks backing $share_name are online ✅\"\n1104\t}\n1105\t\n1106\t# ==============================================================================================\n1107\t# ── CONTAINER MANAGEMENT ──────────────────────────────────────────────────────────────────────\n1108\t# ==============================================================================================\n1109\t# Four functions — stop/start on both local and remote.\n1110\t# State tracking: only containers that WERE running get restarted.\n1111\t# Was stopped → stays stopped. Was running → gets restarted. ✅\n1112\t\n1113\t# Tracks which remote containers were running before stop — used by start_containers()\n1114\tRUNNING_CONTAINERS=()\n1115\t\n1116\t# ── Docker command helpers (used by docker_daily_restart, docker_weekly_restart, docker_update) ──\n1117\t\n1118\t# Default timeouts — scripts can override before calling.\n1119\tDOCKER_TIMEOUT=${DOCKER_TIMEOUT:-30} # seconds before docker command is killed\n1120\tRESTART_VERIFY_WAIT=${RESTART_VERIFY_WAIT:-5} # seconds to wait before checking state post-restart\n1121\t\n1122\t# Run a docker command with a hard timeout. Returns 124 on timeout, else docker's exit code.\n1123\tdocker_cmd() {\n1124\t timeout \"$DOCKER_TIMEOUT\" \"$@\"\n1125\t local exit_code=$?\n1126\t if [[ \"$exit_code\" -eq 124 ]]; then\n1127\t error \"Docker command timed out after ${DOCKER_TIMEOUT}s: $*\"\n1128\t return 1\n1129\t fi\n1130\t return \"$exit_code\"\n1131\t}\n1132\t\n1133\t# Verify a container is still running after a restart.\n1134\t# Waits RESTART_VERIFY_WAIT seconds to let it settle, then checks State.Running.\n1135\tverify_running() {\n1136\t local container=\"$1\"\n1137\t sleep \"$RESTART_VERIFY_WAIT\"\n1138\t local state\n1139\t state=$(docker inspect -f '{{.State.Running}}' \"$container\" 2>/dev/null)\n1140\t if [[ \"$state\" != \"true\" ]]; then\n1141\t error \"$container failed to stay running after restart — may have crashed\"\n1142\t return 1\n1143\t fi\n1144\t return 0\n1145\t}\n1146\t\n1147\t# Retry a docker command up to RETRY_COUNT times, sleeping SLEEP seconds between attempts.\n1148\tretry_docker() {\n1149\t local attempt=1\n1150\t while [[ \"$attempt\" -le \"$RETRY_COUNT\" ]]; do\n1151\t log \"$ICON_RETRY Attempt $attempt of $RETRY_COUNT: $*\"\n1152\t if docker_cmd \"$@\"; then\n1153\t log \"Succeeded on attempt $attempt\"\n1154\t return 0\n1155\t else\n1156\t warn \"Attempt $attempt failed\"\n1157\t (( attempt++ ))\n1158\t [[ \"$attempt\" -le \"$RETRY_COUNT\" ]] && sleep \"$SLEEP\"\n1159\t fi\n1160\t done\n1161\t error \"Command failed after $RETRY_COUNT attempts: $*\"\n1162\t return 1\n1163\t}\n1164\t\n1165\t# ── Emby API helper ───────────────────────────────────────────────────────────\n1166\t# General Emby REST call. Uses EMBY_URL and EMBY_API_KEY (aliased by detect_hosts).\n1167\t# Usage: emby_api [timeout_seconds]\n1168\t# Prints response body on 200; returns 1 on HTTP error or curl failure.\n1169\temby_api() {\n1170\t local endpoint=\"$1\" max_time=\"${2:-30}\"\n1171\t local response http_code body\n1172\t response=$(curl -sf --max-time \"$max_time\" \\\n1173\t -H \"X-Emby-Token: $EMBY_API_KEY\" \\\n1174\t -w \"\\n%{http_code}\" \\\n1175\t \"${EMBY_URL}/${endpoint}\" 2>/dev/null)\n1176\t http_code=$(echo \"$response\" | tail -1)\n1177\t body=$(echo \"$response\" | head -n -1)\n1178\t [[ \"$http_code\" != \"200\" ]] && { error \"Emby API HTTP $http_code: $endpoint\"; return 1; }\n1179\t echo \"$body\"\n1180\t}\n1181\t\n1182\t# Stop containers on the REMOTE server via SSH.\n1183\t# Reads CRITICAL_CONTAINER_NAMES — set from profile arrays by rsync.sh.\n1184\t# Tracks running state in RUNNING_CONTAINERS for restart after sync.\n1185\tstop_containers() {\n1186\t if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -eq 0 ]] || \\\n1187\t [[ \"${CRITICAL_CONTAINER_NAMES[*]}\" == \"\" ]]; then\n1188\t log \"No remote containers configured for this profile, skipping remote stop.\"\n1189\t return\n1190\t fi\n1191\t echo \"━━━ $ICON_STOP $ICON_CONTAINERS Containers ━━━\"\n1192\t info \"Stopping remote containers...\"\n1193\t RUNNING_CONTAINERS=()\n1194\t for c in \"${CRITICAL_CONTAINER_NAMES[@]}\"; do\n1195\t [[ -z \"$c\" ]] && continue\n1196\t local status\n1197\t status=$(ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \\\n1198\t \"docker inspect -f '{{.State.Running}}' $c 2>/dev/null\" 2>/dev/null || echo \"unknown\")\n1199\t if [[ \"$status\" == \"true\" ]]; then\n1200\t echo \"$ICON_STOP Stopping remote $c...\"\n1201\t RUNNING_CONTAINERS+=(\"$c\")\n1202\t if ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \"docker stop $c\" >/dev/null; then\n1203\t echo \"$ICON_STOPPED $c stopped\"\n1204\t else\n1205\t error \"Failed to stop remote $c\"\n1206\t fi\n1207\t elif [[ \"$status\" == \"false\" ]]; then\n1208\t echo \"$ICON_NOT_RUNNING $c is not running — skipping\"\n1209\t else\n1210\t log \"$c not found on remote — skipping\"\n1211\t fi\n1212\t done\n1213\t}\n1214\t\n1215\t# Tracks which local containers were running before stop — used by start_local_containers()\n1216\tLOCAL_RUNNING_CONTAINERS=()\n1217\t\n1218\t# Stop containers on the LOCAL server.\n1219\t# Reads LOCAL_CRITICAL_CONTAINER_NAMES — set from profile arrays by rsync.sh.\n1220\t# Flushes local databases cleanly before pushing data to remote.\n1221\tstop_local_containers() {\n1222\t if [[ ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -eq 0 ]] || \\\n1223\t [[ \"${LOCAL_CRITICAL_CONTAINER_NAMES[*]}\" == \"\" ]]; then\n1224\t log \"No local containers configured for this profile, skipping local stop.\"\n1225\t return\n1226\t fi\n1227\t info \"Stopping local containers...\"\n1228\t LOCAL_RUNNING_CONTAINERS=()\n1229\t for c in \"${LOCAL_CRITICAL_CONTAINER_NAMES[@]}\"; do\n1230\t [[ -z \"$c\" ]] && continue\n1231\t local status\n1232\t status=$(docker inspect -f '{{.State.Running}}' \"$c\" 2>/dev/null || echo \"unknown\")\n1233\t if [[ \"$status\" == \"true\" ]]; then\n1234\t echo \"$ICON_STOP Stopping local $c...\"\n1235\t LOCAL_RUNNING_CONTAINERS+=(\"$c\")\n1236\t if docker stop \"$c\" >/dev/null; then\n1237\t echo \"$ICON_STOPPED $c stopped\"\n1238\t else\n1239\t error \"Failed to stop local $c\"\n1240\t fi\n1241\t elif [[ \"$status\" == \"false\" ]]; then\n1242\t echo \"$ICON_NOT_RUNNING $c is not running — skipping\"\n1243\t else\n1244\t log \"$c not found locally — skipping\"\n1245\t fi\n1246\t done\n1247\t}\n1248\t\n1249\t# Restart containers on the REMOTE server that were running before sync.\n1250\t# Reads RUNNING_CONTAINERS set by stop_containers().\n1251\t# Respects DELAYED_CONTAINERS — waits CONTAINER_DELAY seconds before starting them.\n1252\tstart_containers() {\n1253\t if [[ ${#RUNNING_CONTAINERS[@]} -eq 0 ]]; then\n1254\t log \"No remote containers to restart.\"\n1255\t return\n1256\t fi\n1257\t info \"Starting remote containers...\"\n1258\t for c in \"${RUNNING_CONTAINERS[@]}\"; do\n1259\t [[ -z \"$c\" ]] && continue\n1260\t local needs_delay=false\n1261\t for d in \"${DELAYED_CONTAINERS[@]}\"; do\n1262\t [[ \"$c\" == \"$d\" ]] && needs_delay=true && break\n1263\t done\n1264\t if [[ \"$needs_delay\" == true ]]; then\n1265\t info \"Waiting ${CONTAINER_DELAY}s before starting $c...\"\n1266\t sleep \"$CONTAINER_DELAY\"\n1267\t fi\n1268\t echo \"$ICON_START Starting remote $c...\"\n1269\t if ssh -i \"$SSH_KEY\" root@\"$REMOTE_SERVER\" \"docker start $c\" >/dev/null 2>&1; then\n1270\t echo \"$ICON_STARTED $c started\"\n1271\t else\n1272\t error \"Failed to start remote $c — start manually if needed\"\n1273\t fi\n1274\t done\n1275\t}\n1276\t\n1277\t# Restart containers on the LOCAL server that were running before sync.\n1278\t# Reads LOCAL_RUNNING_CONTAINERS set by stop_local_containers().\n1279\t# Respects DELAYED_CONTAINERS and CONTAINER_DELAY same as remote.\n1280\tstart_local_containers() {\n1281\t if [[ ${#LOCAL_RUNNING_CONTAINERS[@]} -eq 0 ]]; then\n1282\t log \"No local containers to restart.\"\n1283\t return\n1284\t fi\n1285\t info \"Starting local containers...\"\n1286\t for c in \"${LOCAL_RUNNING_CONTAINERS[@]}\"; do\n1287\t [[ -z \"$c\" ]] && continue\n1288\t local needs_delay=false\n1289\t for d in \"${DELAYED_CONTAINERS[@]}\"; do\n1290\t [[ \"$c\" == \"$d\" ]] && needs_delay=true && break\n1291\t done\n1292\t if [[ \"$needs_delay\" == true ]]; then\n1293\t info \"Waiting ${CONTAINER_DELAY}s before starting local $c...\"\n1294\t sleep \"$CONTAINER_DELAY\"\n1295\t fi\n1296\t echo \"$ICON_START Starting local $c...\"\n1297\t if docker start \"$c\" >/dev/null 2>&1; then\n1298\t echo \"$ICON_STARTED $c started\"\n1299\t else\n1300\t error \"Failed to start local $c — start manually if needed\"\n1301\t fi\n1302\t done\n1303\t}\n1304\t\n1305\t# ==============================================================================================\n1306\t# ── RSYNC OPTIONS ─────────────────────────────────────────────────────────────────────────────\n1307\t# ==============================================================================================\n1308\t# Loads rsync options for the current profile.\n1309\t# Falls back to DEFAULT_RSYNC_OPTS from master.conf if no profile match found.\n1310\t# Profile opts do NOT inherit from defaults — list all desired flags explicitly in master.conf.\n1311\t\n1312\tget_rsync_opts() {\n1313\t if [[ -n \"${PROFILE_RSYNC_OPTS[$PROFILE_NAME]:-}\" ]]; then\n1314\t read -r -a RSYNC_OPTS <<< \"${PROFILE_RSYNC_OPTS[$PROFILE_NAME]}\"\n1315\t log \"Using profile rsync opts for $PROFILE_NAME: ${RSYNC_OPTS[*]}\"\n1316\t else\n1317\t RSYNC_OPTS=(\"${DEFAULT_RSYNC_OPTS[@]}\")\n1318\t log \"Using default rsync opts: ${RSYNC_OPTS[*]}\"\n1319\t fi\n1320\t}\n1321\t\n1322\t# ==============================================================================================\n1323\t# ── SCRIPT LOCKING ────────────────────────────────────────────────────────────────────────────\n1324\t# ==============================================================================================\n1325\t# Prevents multiple instances of the same script running simultaneously.\n1326\t# All lock files live in /tmp/unraid_locks/ — auto-cleared on reboot.\n1327\t#\n1328\t# Modes:\n1329\t# strict — exit immediately if already running (default)\n1330\t# wait — wait LOCK_WAIT_TIMEOUT seconds then exit if still locked\n1331\t# continuous — for long-running scripts: skip gracefully if healthy instance running\n1332\t#\n1333\t# Features:\n1334\t# Stale lock detection — if lock PID is dead, clears and proceeds\n1335\t# PID reuse protection — lock stores PID:scriptname, validates both\n1336\t# Age warning — warns if lock older than LOCK_WARN_AGE (skipped for continuous)\n1337\t# EXIT trap — lock always released on exit, crash, or kill signal\n1338\t\n1339\tLOCK_DIR=\"/tmp/unraid_locks\"\n1340\t\n1341\t# Ensure DATA_DIR and STATE_DIR exist — created here so every script that sources common.sh\n1342\t# can safely write to either without checking first.\n1343\tif [[ -n \"${DATA_DIR:-}\" ]] && [[ ! -d \"$DATA_DIR\" ]]; then\n1344\t mkdir -p \"$DATA_DIR\" 2>/dev/null || true\n1345\tfi\n1346\tif [[ -n \"${STATE_DIR:-}\" ]] && [[ ! -d \"$STATE_DIR\" ]]; then\n1347\t mkdir -p \"$STATE_DIR\" 2>/dev/null || true\n1348\tfi\n1349\t\n1350\tRSYNC_COUNT_FILE=\"$LOCK_DIR/rsync_active_count\"\n1351\tRSYNC_MAX_CONCURRENT=3\n1352\tLOCK_WARN_AGE=300 # seconds — warn if lock older than this (5min default)\n1353\tLOCK_WAIT_TIMEOUT=30 # seconds — how long \"wait\" mode waits before giving up\n1354\t\n1355\t# Registry of all lock files acquired by this process — released together on exit.\n1356\t# Prevents the single-trap-per-acquire problem: each new acquire_lock/acquire_rsync_lock\n1357\t# call used to overwrite the EXIT trap, orphaning the previous lock file.\n1358\tdeclare -ga _LOCK_FILES=()\n1359\t_release_all_locks() {\n1360\t local lf\n1361\t for lf in \"${_LOCK_FILES[@]}\"; do\n1362\t [[ -f \"$lf\" ]] && rm -f \"$lf\"\n1363\t done\n1364\t}\n1365\t_register_lock() {\n1366\t _LOCK_FILES+=(\"$1\")\n1367\t trap '_release_all_locks' EXIT\n1368\t}\n1369\t\n1370\t# Internal — script name used as lock identifier\n1371\t_lock_name() {\n1372\t basename \"${BASH_SOURCE[1]:-$0}\" .sh\n1373\t}\n1374\t\n1375\t# Internal — lock file path for this script\n1376\t_lock_file() {\n1377\t echo \"$LOCK_DIR/${1:-$(_lock_name)}.lock\"\n1378\t}\n1379\t\n1380\t# Internal — release lock on exit (registered via EXIT trap)\n1381\t_release_on_exit() {\n1382\t local lockfile=\"$1\"\n1383\t [[ -f \"$lockfile\" ]] && rm -f \"$lockfile\"\n1384\t}\n1385\t\n1386\t# Acquire exclusive lock for this script.\n1387\t# Usage: acquire_lock [strict|wait|continuous]\n1388\tacquire_lock() {\n1389\t local mode=\"${1:-strict}\"\n1390\t local script_name\n1391\t script_name=$(basename \"${BASH_SOURCE[1]:-$0}\" .sh)\n1392\t local lockfile\n1393\t lockfile=\"$(_lock_file \"$script_name\")\"\n1394\t\n1395\t mkdir -p \"$LOCK_DIR\"\n1396\t\n1397\t if [[ -f \"$lockfile\" ]]; then\n1398\t local lock_content existing_pid locked_name\n1399\t lock_content=$(cat \"$lockfile\" 2>/dev/null)\n1400\t existing_pid=\"${lock_content%%:*}\"\n1401\t locked_name=\"${lock_content##*:}\"\n1402\t\n1403\t # Stale lock — PID dead\n1404\t if [[ -z \"$existing_pid\" ]] || ! kill -0 \"$existing_pid\" 2>/dev/null; then\n1405\t warn \"Stale lock detected for $script_name (PID $existing_pid gone) — clearing\"\n1406\t rm -f \"$lockfile\"\n1407\t\n1408\t # PID reuse — PID alive but belongs to different process\n1409\t elif [[ \"$locked_name\" != \"$script_name\" ]]; then\n1410\t warn \"Lock PID $existing_pid reused by different process ($locked_name ≠ $script_name) — clearing\"\n1411\t rm -f \"$lockfile\"\n1412\t\n1413\t else\n1414\t # Lock is genuinely active — check age\n1415\t local lock_age\n1416\t lock_age=$(( $(date +%s) - $(stat -c %Y \"$lockfile\" 2>/dev/null || echo 0) ))\n1417\t if [[ \"$lock_age\" -gt \"$LOCK_WARN_AGE\" ]] && [[ \"$mode\" != \"continuous\" ]]; then\n1418\t warn \"$script_name has been running for ${lock_age}s — may be stuck (PID $existing_pid)\"\n1419\t fi\n1420\t\n1421\t if [[ \"$mode\" == \"continuous\" ]]; then\n1422\t log \"$ICON_SKIP $script_name already running healthy (PID $existing_pid) — skipping\"\n1423\t exit 0\n1424\t elif [[ \"$mode\" == \"wait\" ]]; then\n1425\t info \"Another instance of $script_name is running — waiting up to ${LOCK_WAIT_TIMEOUT}s\"\n1426\t local waited=0\n1427\t while [[ -f \"$lockfile\" ]] && [[ \"$waited\" -lt \"$LOCK_WAIT_TIMEOUT\" ]]; do\n1428\t sleep 1\n1429\t ((waited++))\n1430\t lock_content=$(cat \"$lockfile\" 2>/dev/null)\n1431\t existing_pid=\"${lock_content%%:*}\"\n1432\t locked_name=\"${lock_content##*:}\"\n1433\t if [[ -z \"$existing_pid\" ]] || ! kill -0 \"$existing_pid\" 2>/dev/null; then\n1434\t warn \"Lock became stale while waiting — clearing\"\n1435\t rm -f \"$lockfile\"\n1436\t break\n1437\t elif [[ \"$locked_name\" != \"$script_name\" ]]; then\n1438\t warn \"Lock PID reused while waiting — clearing\"\n1439\t rm -f \"$lockfile\"\n1440\t break\n1441\t fi\n1442\t done\n1443\t if [[ -f \"$lockfile\" ]]; then\n1444\t error \"$script_name still locked after ${LOCK_WAIT_TIMEOUT}s — exiting\"\n1445\t exit 1\n1446\t fi\n1447\t else\n1448\t error \"Another instance of $script_name is already running (PID $existing_pid) — exiting\"\n1449\t case \"$script_name\" in\n1450\t fallback|transcode_management|daily_sync_maintenance|stability_watchdog)\n1451\t notify \"$script_name lock collision on $(hostname) — concurrent instance detected\" \"$script_name\" \"warning\"\n1452\t ;;\n1453\t esac\n1454\t exit 1\n1455\t fi\n1456\t fi\n1457\t fi\n1458\t\n1459\t # Acquire lock — store PID:scriptname to prevent PID reuse false positives\n1460\t echo \"$$:$script_name\" > \"$lockfile\"\n1461\t _register_lock \"$lockfile\"\n1462\t log \"$ICON_LOCK Lock acquired: $script_name (PID $$)\"\n1463\t}\n1464\t\n1465\t# Acquire per-profile rsync lock + enforce global concurrent limit.\n1466\t# Prevents same profile running twice and limits total concurrent rsync instances.\n1467\t# Usage: acquire_rsync_lock \"$PROFILE_NAME\"\n1468\tacquire_rsync_lock() {\n1469\t local profile=\"$1\"\n1470\t local profile_lock\n1471\t profile_lock=\"$(_lock_file \"rsync_${profile}\")\"\n1472\t\n1473\t mkdir -p \"$LOCK_DIR\"\n1474\t\n1475\t # Per-profile lock — same profile cannot run twice\n1476\t if [[ -f \"$profile_lock\" ]]; then\n1477\t local lock_content existing_pid locked_name\n1478\t lock_content=$(cat \"$profile_lock\" 2>/dev/null)\n1479\t existing_pid=\"${lock_content%%:*}\"\n1480\t locked_name=\"${lock_content##*:}\"\n1481\t if [[ -n \"$existing_pid\" ]] && kill -0 \"$existing_pid\" 2>/dev/null && \\\n1482\t [[ \"$locked_name\" == \"rsync_${profile}\" ]]; then\n1483\t error \"rsync profile '$profile' is already running (PID $existing_pid) — exiting\"\n1484\t exit 1\n1485\t else\n1486\t warn \"Stale rsync lock for profile '$profile' — clearing\"\n1487\t rm -f \"$profile_lock\"\n1488\t fi\n1489\t fi\n1490\t\n1491\t # Global concurrent limit — validate count against actual live locks\n1492\t local current_count=0\n1493\t if [[ -f \"$RSYNC_COUNT_FILE\" ]]; then\n1494\t current_count=$(cat \"$RSYNC_COUNT_FILE\" 2>/dev/null || echo 0)\n1495\t local actual_count=0\n1496\t for lf in \"$LOCK_DIR\"/rsync_*.lock; do\n1497\t [[ -f \"$lf\" ]] || continue\n1498\t local lpid lf_content\n1499\t lf_content=$(cat \"$lf\" 2>/dev/null)\n1500\t lpid=\"${lf_content%%:*}\"\n1501\t kill -0 \"$lpid\" 2>/dev/null && ((actual_count++))\n1502\t done\n1503\t if [[ \"$actual_count\" -ne \"$current_count\" ]]; then\n1504\t log \"rsync count corrected: $current_count → $actual_count\"\n1505\t current_count=$actual_count\n1506\t echo \"$current_count\" > \"$RSYNC_COUNT_FILE\"\n1507\t fi\n1508\t fi\n1509\t\n1510\t if [[ \"$current_count\" -ge \"$RSYNC_MAX_CONCURRENT\" ]]; then\n1511\t error \"Maximum concurrent rsync limit ($RSYNC_MAX_CONCURRENT) reached — exiting\"\n1512\t info \"Active rsync locks: $(ls \"$LOCK_DIR\"/rsync_*.lock 2>/dev/null | \\\n1513\t xargs -I{} basename {} .lock | tr '\\n' ' ')\"\n1514\t exit 1\n1515\t fi\n1516\t\n1517\t # Acquire profile lock and increment counter\n1518\t echo \"$$:rsync_${profile}\" > \"$profile_lock\"\n1519\t echo $(( current_count + 1 )) > \"$RSYNC_COUNT_FILE\"\n1520\t _register_lock \"$profile_lock\"\n1521\t log \"$ICON_LOCK rsync lock acquired: profile '$profile' (PID $$, active: $(( current_count + 1 ))/$RSYNC_MAX_CONCURRENT)\"\n1522\t}\n1523\t\n1524\t# Internal — release rsync lock and decrement global counter on exit\n1525\t_release_rsync_on_exit() {\n1526\t local profile_lock=\"$1\"\n1527\t [[ -f \"$profile_lock\" ]] && rm -f \"$profile_lock\"\n1528\t if [[ -f \"$RSYNC_COUNT_FILE\" ]]; then\n1529\t local count\n1530\t count=$(cat \"$RSYNC_COUNT_FILE\" 2>/dev/null || echo 1)\n1531\t count=$(( count - 1 ))\n1532\t [[ \"$count\" -lt 0 ]] && count=0\n1533\t echo \"$count\" > \"$RSYNC_COUNT_FILE\"\n1534\t fi\n1535\t}\n1536\t\n1537\t# ==============================================================================================\n1538\t# ── PATH TRANSLATION ──────────────────────────────────────────────────────────────────────────\n1539\t# ==============================================================================================\n1540\t# Translates container-internal file paths to host paths for arr cleanup scripts.\n1541\t# Arr APIs return file paths as seen inside the container — scripts need host paths to scan.\n1542\t#\n1543\t# Uses ARR_PATH_MAP associative array — must be declared before calling.\n1544\t# Longest-match wins — prevents partial path collisions.\n1545\t#\n1546\t# Usage:\n1547\t# declare -A ARR_PATH_MAP=([\"/ext-music\"]=\"/mnt/user/Music-New\")\n1548\t# translate_path \"/ext-music/Artist/Album/track.flac\"\n1549\t# Returns: /mnt/user/Music-New/Artist/Album/track.flac\n1550\t#\n1551\t# If no match found — returns path unchanged (safe fallback).\n1552\t\n1553\ttranslate_path() {\n1554\t local api_path=\"$1\"\n1555\t local best_match=\"\"\n1556\t local best_len=0\n1557\t\n1558\t for container_path in \"${!ARR_PATH_MAP[@]}\"; do\n1559\t if [[ \"$api_path\" == \"$container_path\"* ]]; then\n1560\t if [[ \"${#container_path}\" -gt \"$best_len\" ]]; then\n1561\t best_match=\"$container_path\"\n1562\t best_len=\"${#container_path}\"\n1563\t fi\n1564\t fi\n1565\t done\n1566\t\n1567\t if [[ -n \"$best_match\" ]]; then\n1568\t echo \"${ARR_PATH_MAP[$best_match]}${api_path#$best_match}\"\n1569\t else\n1570\t echo \"$api_path\" # no match — return unchanged\n1571\t fi\n1572\t}\n1573\t"}]},"uuid":"4fad9708-a691-4740-b582-eac75cbbe580","timestamp":"2026-06-14T05:32:12.839Z","sourceToolAssistantUUID":"e60685fd-3658-425f-b029-03c8ce4eca6f","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"4fad9708-a691-4740-b582-eac75cbbe580","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01SCu2NP68Me7th2NAwyH4X6","type":"message","role":"assistant","content":[{"type":"text","text":"Now let me examine the common.sh file more thoroughly to check the rest of it:"}],"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":4989,"cache_creation_input_tokens":60615,"cache_read_input_tokens":7721,"cache_creation":{"ephemeral_5m_input_tokens":60615,"ephemeral_1h_input_tokens":0},"output_tokens":1,"service_tier":"standard","inference_geo":"not_available"},"diagnostics":null},"requestId":"req_011Cc2c3fQu4GmmuynUqrmjt","attributionAgent":"Explore","type":"assistant","uuid":"17552295-da9e-40e8-b533-f7d5c6cc5f6c","timestamp":"2026-06-14T05:32:15.776Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"17552295-da9e-40e8-b533-f7d5c6cc5f6c","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_01SCu2NP68Me7th2NAwyH4X6","type":"message","role":"assistant","content":[{"type":"tool_use","id":"toolu_013dxWXKsWYDxbNrk2nxgbDL","name":"Read","input":{"file_path":"/boot/config/plugins/varaverk/common.sh","offset":1574,"limit":1573},"caller":{"type":"direct"}}],"stop_reason":"tool_use","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":4989,"cache_creation_input_tokens":60615,"cache_read_input_tokens":7721,"output_tokens":121,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":60615},"inference_geo":"not_available","iterations":[{"input_tokens":4989,"output_tokens":121,"cache_read_input_tokens":7721,"cache_creation_input_tokens":60615,"cache_creation":{"ephemeral_5m_input_tokens":60615,"ephemeral_1h_input_tokens":0},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cc2c3fQu4GmmuynUqrmjt","attributionAgent":"Explore","type":"assistant","uuid":"0154131e-d0a9-42d6-86db-12212ede3cd7","timestamp":"2026-06-14T05:32:16.093Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"0154131e-d0a9-42d6-86db-12212ede3cd7","isSidechain":true,"promptId":"4ede2c2c-ae62-435a-a0fb-4f8d63fa41ab","agentId":"ab976a3a2d54c1360","type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_013dxWXKsWYDxbNrk2nxgbDL","type":"tool_result","content":"1574\t# ==============================================================================================\n1575\t# ── ARR VERSION CHECK ─────────────────────────────────────────────────────────────────────────\n1576\t# ==============================================================================================\n1577\t# Verifies arr major version matches tested version in master.conf.\n1578\t# Exits if version doesn't match — prevents running against untested API structure.\n1579\t# Warns and proceeds if version endpoint is unreachable.\n1580\t#\n1581\t# Usage:\n1582\t# check_arr_version \"$RADARR_URL\" \"$RADARR_API_KEY\" \"v3\" \"$RADARR_VERSION_MAJOR\" \"Radarr\"\n1583\t#\n1584\t# Arguments:\n1585\t# $1 = arr base URL\n1586\t# $2 = API key\n1587\t# $3 = API path prefix (v3 or v1)\n1588\t# $4 = expected major version number from master.conf\n1589\t# $5 = arr name for error messages\n1590\t\n1591\tcheck_arr_version() {\n1592\t local url=\"$1\"\n1593\t local api_key=\"$2\"\n1594\t local api_prefix=\"$3\"\n1595\t local expected_major=\"$4\"\n1596\t local arr_name=\"${5:-Arr}\"\n1597\t\n1598\t local status_response version major_version\n1599\t\n1600\t status_response=$(curl -sf --max-time 10 \\\n1601\t -H \"X-Api-Key: $api_key\" \\\n1602\t \"${url}/api/${api_prefix}/system/status\" 2>/dev/null)\n1603\t\n1604\t if [[ -z \"$status_response\" ]]; then\n1605\t warn \"$arr_name version check failed — could not reach system/status endpoint\"\n1606\t warn \"Proceeding without version verification — monitor for API errors\"\n1607\t return 0\n1608\t fi\n1609\t\n1610\t version=$(echo \"$status_response\" | \\\n1611\t grep -o '\"version\": *\"[^\"]*\"' | \\\n1612\t grep -o '[0-9][^\"]*' | head -1)\n1613\t\n1614\t if [[ -z \"$version\" ]]; then\n1615\t warn \"$arr_name version check failed — could not parse version from response\"\n1616\t warn \"Proceeding without version verification — monitor for API errors\"\n1617\t return 0\n1618\t fi\n1619\t\n1620\t major_version=\"${version%%.*}\"\n1621\t\n1622\t if [[ \"$major_version\" == \"$expected_major\" ]]; then\n1623\t success \"$arr_name version: $version (major $major_version — tested ✅)\"\n1624\t return 0\n1625\t else\n1626\t error \"$arr_name version mismatch — running v${major_version}, tested against v${expected_major}\"\n1627\t error \"The API endpoint structure may have changed — exiting to protect your library\"\n1628\t error \"Update ${arr_name^^}_VERSION_MAJOR in master.conf after verifying the script works with v${major_version}\"\n1629\t notify \"$arr_name version mismatch on $(hostname) — running v${major_version}, script tested against v${expected_major}\" \"$arr_name Cleanup\" \"warning\"\n1630\t exit 1\n1631\t fi\n1632\t}\n1633\t\n1634\t# ==============================================================================================\n1635\t# ── EMBY LIBRARY SCAN ─────────────────────────────────────────────────────────────────────────\n1636\t# ==============================================================================================\n1637\t# Triggers Emby \"Clean Missing Files\" scheduled task after arr cleanup scripts delete files.\n1638\t# Without this Emby keeps showing deleted files as ghost entries until its next scheduled scan.\n1639\t# User clicks a ghost entry → file not found error. This prevents that entirely.\n1640\t#\n1641\t# Uses EMBY_URL and EMBY_API_KEY — aliased by detect_hosts() from HOST*_EMBY_* vars.\n1642\t# Skips cleanly if Emby is not configured on this host (empty URL or API key).\n1643\t#\n1644\t# Emby task discovery:\n1645\t# GET /ScheduledTasks → find task where Name contains \"Clean Missing\"\n1646\t# POST /ScheduledTasks/Running/{taskId} → trigger it\n1647\t#\n1648\t# Non-fatal — logs warning if task cannot be triggered but does not exit.\n1649\t# Clean Missing Files runs across all libraries at once — one call handles\n1650\t# music, TV, and movies together regardless of which arr triggered the cleanup.\n1651\t#\n1652\t# Usage: notify_emby_scan (no args — always triggers Clean Missing Files)\n1653\t\n1654\tnotify_emby_scan() {\n1655\t # Skip if Emby not configured on this host\n1656\t if [[ -z \"${EMBY_URL:-}\" ]] || [[ -z \"${EMBY_API_KEY:-}\" ]]; then\n1657\t log \"Emby not configured on $MY_ID — skipping library scan notification\"\n1658\t return 0\n1659\t fi\n1660\t\n1661\t log \"Notifying Emby to clean missing files...\"\n1662\t\n1663\t # Get all scheduled tasks\n1664\t local tasks_response\n1665\t tasks_response=$(curl -sf --max-time 15 \\\n1666\t -H \"X-Api-Key: $EMBY_API_KEY\" \\\n1667\t \"${EMBY_URL}/ScheduledTasks\" 2>/dev/null)\n1668\t\n1669\t if [[ -z \"$tasks_response\" ]]; then\n1670\t warn \"Could not reach Emby scheduled tasks API at $EMBY_URL — skipping scan\"\n1671\t return 0\n1672\t fi\n1673\t\n1674\t # Find the \"Clean Missing Files\" task ID\n1675\t local task_id\n1676\t task_id=$(echo \"$tasks_response\" | \\\n1677\t grep -o '\"Id\":\"[^\"]*\",\"Name\":\"[^\"]*Clean Missing[^\"]*\"' | \\\n1678\t grep -o '\"Id\":\"[^\"]*\"' | \\\n1679\t sed 's/\"Id\":\"//;s/\"//' | head -1)\n1680\t\n1681\t # Fallback — try \"Scan Media Library\" if Clean Missing not found\n1682\t if [[ -z \"$task_id\" ]]; then\n1683\t task_id=$(echo \"$tasks_response\" | \\\n1684\t grep -o '\"Id\":\"[^\"]*\",\"Name\":\"[^\"]*Scan Media Library[^\"]*\"' | \\\n1685\t grep -o '\"Id\":\"[^\"]*\"' | \\\n1686\t sed 's/\"Id\":\"//;s/\"//' | head -1)\n1687\t [[ -n \"$task_id\" ]] && log \"Clean Missing Files not found — using Scan Media Library\"\n1688\t fi\n1689\t\n1690\t if [[ -z \"$task_id\" ]]; then\n1691\t warn \"Could not find Emby Clean Missing Files or Scan Media Library task\"\n1692\t warn \"Check Emby scheduled tasks — ghost entries will persist until next Emby scan\"\n1693\t return 0\n1694\t fi\n1695\t\n1696\t log \"Found Emby task ID: $task_id\"\n1697\t\n1698\t # Trigger the task\n1699\t local http_code\n1700\t http_code=$(curl -sf --max-time 15 \\\n1701\t -X POST \\\n1702\t -H \"X-Api-Key: $EMBY_API_KEY\" \\\n1703\t -w \"%{http_code}\" \\\n1704\t -o /dev/null \\\n1705\t \"${EMBY_URL}/ScheduledTasks/Running/${task_id}\" 2>/dev/null)\n1706\t\n1707\t if [[ \"$http_code\" == \"204\" ]] || [[ \"$http_code\" == \"200\" ]]; then\n1708\t warn \"$ICON_EMBY Emby Clean Missing Files triggered — ghost entries will be removed\"\n1709\t else\n1710\t warn \"Emby task trigger returned HTTP $http_code — ghost entries may persist\"\n1711\t warn \"Check Emby dashboard — manually run Clean Missing Files if needed\"\n1712\t fi\n1713\t}\n1714\t\n1715\t# ==============================================================================================\n1716\t# ── API REACHABILITY CHECK ────────────────────────────────────────────────────────────────────\n1717\t# ==============================================================================================\n1718\t# Pre-flight check — verifies an API endpoint is reachable before attempting operations.\n1719\t# Non-fatal — returns 0 if reachable, 1 if not. Caller decides whether to exit.\n1720\t#\n1721\t# Usage: check_api \"http://localhost:8989\" \"Sonarr\" || exit 1\n1722\t\n1723\tcheck_api() {\n1724\t local url=\"$1\"\n1725\t local service=\"${2:-API}\"\n1726\t local timeout=\"${3:-10}\"\n1727\t\n1728\t if curl -sf --max-time \"$timeout\" \"$url\" >/dev/null 2>&1; then\n1729\t log \"$service API reachable: $url\"\n1730\t return 0\n1731\t else\n1732\t error \"$service API not reachable: $url\"\n1733\t return 1\n1734\t fi\n1735\t}\n1736\t\n1737\t# ==============================================================================================\n1738\t# ── UNRAID VERSION PARITY CHECK ───────────────────────────────────────────────────────────────\n1739\t# ==============================================================================================\n1740\t# Verifies local and remote servers are running compatible unRAID versions before any\n1741\t# remote operation. Version mismatches can mean changed APIs, commands, or behaviours\n1742\t# that silently break remote container operations, rsync, or fallback logic.\n1743\t#\n1744\t# Reads platform OS version on both sides via adapter.\n1745\t#\n1746\t# Mismatch behaviour (UNRAID_VERSION_MISMATCH_ACTION in master.conf):\n1747\t# \"warn\" — log warning and continue (default for minor/patch differences)\n1748\t# \"abort\" — exit the calling script (default for major version differences)\n1749\t# Major version mismatch always aborts regardless of setting\n1750\t#\n1751\t# Usage: check_os_version_parity || exit 1\n1752\t\n1753\tcheck_os_version_parity() {\n1754\t local local_version remote_version\n1755\t\n1756\t # Read local version\n1757\t local_version=$(platform_get_os_version 2>/dev/null)\n1758\t if [[ -z \"$local_version\" ]]; then\n1759\t warn \"Cannot read local OS version — skipping version parity check\"\n1760\t return 0\n1761\t fi\n1762\t\n1763\t # Read remote version via SSH\n1764\t local _probe_cmd\n1765\t _probe_cmd=$(platform_os_version_probe_cmd)\n1766\t remote_version=$(ssh -i \"$SSH_KEY\" -o ConnectTimeout=10 root@\"$REMOTE_SERVER\" \\\n1767\t \"$_probe_cmd\" 2>/dev/null)\n1768\t if [[ -z \"$remote_version\" ]]; then\n1769\t warn \"Cannot read remote OS version from $REMOTE_SERVER_NAME — skipping parity check\"\n1770\t return 0\n1771\t fi\n1772\t\n1773\t log \"Version parity: local=$local_version remote=$remote_version\"\n1774\t\n1775\t if [[ \"$local_version\" == \"$remote_version\" ]]; then\n1776\t log \"OS versions match: $local_version ✅\"\n1777\t return 0\n1778\t fi\n1779\t\n1780\t # Parse major versions\n1781\t local local_major remote_major\n1782\t local_major=\"${local_version%%.*}\"\n1783\t remote_major=\"${remote_version%%.*}\"\n1784\t\n1785\t if [[ \"$local_major\" != \"$remote_major\" ]]; then\n1786\t error \"OS MAJOR version mismatch — local: $local_version remote: $remote_version\"\n1787\t error \"Major version differences may break remote APIs, commands, and behaviours\"\n1788\t error \"Update both servers to the same major version before running remote operations\"\n1789\t notify \"OS major version mismatch on $(hostname) — local: $local_version remote: $remote_version — remote operations aborted\" \\\n1790\t \"Version Parity\" \"warning\"\n1791\t return 1\n1792\t fi\n1793\t\n1794\t # Minor/patch mismatch — action depends on config\n1795\t local action=\"${UNRAID_VERSION_MISMATCH_ACTION:-warn}\"\n1796\t warn \"OS version mismatch — local: $local_version remote: $remote_version\"\n1797\t\n1798\t if [[ \"$action\" == \"abort\" ]]; then\n1799\t error \"UNRAID_VERSION_MISMATCH_ACTION=abort — refusing to continue\"\n1800\t notify \"OS version mismatch on $(hostname) — local: $local_version remote: $remote_version — aborted\" \\\n1801\t \"Version Parity\" \"warning\"\n1802\t return 1\n1803\t fi\n1804\t\n1805\t warn \"UNRAID_VERSION_MISMATCH_ACTION=warn — continuing despite mismatch (monitor for issues)\"\n1806\t return 0\n1807\t}\n1808\t\n1809\t# ==============================================================================================\n1810\t# ── REMOTE DOCKER DAEMON CHECK ────────────────────────────────────────────────────────────────\n1811\t# ==============================================================================================\n1812\t# Verifies the remote Docker daemon is responsive before issuing any remote container commands.\n1813\t# A hung remote daemon causes docker commands to hang or silently fail — checking first\n1814\t# prevents cascading failures and gives clear diagnostic output.\n1815\t#\n1816\t# Strike system:\n1817\t# Each consecutive failed check adds a strike to REMOTE_DOCKER_STRIKES\n1818\t# Below REMOTE_DOCKER_STRIKE_LIMIT → warn and return 1 (caller skips operation)\n1819\t# At limit → notify critical and return 1 (caller should exit or escalate)\n1820\t# Strikes reset when daemon recovers\n1821\t#\n1822\t# Usage: check_remote_docker_daemon || return 1\n1823\t\n1824\tREMOTE_DOCKER_STRIKES=0\n1825\t\n1826\tcheck_remote_docker_daemon() {\n1827\t local timeout=\"${1:-10}\"\n1828\t\n1829\t if ssh -i \"$SSH_KEY\" -o ConnectTimeout=\"$timeout\" root@\"$REMOTE_SERVER\" \\\n1830\t \"timeout $timeout docker info\" >/dev/null 2>&1; then\n1831\t # Daemon healthy — clear strikes if we had issues\n1832\t if [[ \"$REMOTE_DOCKER_STRIKES\" -gt 0 ]]; then\n1833\t info \"Remote Docker daemon on $REMOTE_SERVER_NAME recovered — clearing strikes\"\n1834\t REMOTE_DOCKER_STRIKES=0\n1835\t fi\n1836\t log \"Remote Docker daemon healthy on $REMOTE_SERVER_NAME\"\n1837\t return 0\n1838\t fi\n1839\t\n1840\t REMOTE_DOCKER_STRIKES=$(( REMOTE_DOCKER_STRIKES + 1 ))\n1841\t local limit=\"${REMOTE_DOCKER_STRIKE_LIMIT:-3}\"\n1842\t warn \"Remote Docker daemon not responding on $REMOTE_SERVER_NAME (strike $REMOTE_DOCKER_STRIKES/$limit)\"\n1843\t\n1844\t if [[ \"$REMOTE_DOCKER_STRIKES\" -ge \"$limit\" ]]; then\n1845\t error \"Remote Docker daemon unresponsive on $REMOTE_SERVER_NAME after $limit consecutive checks\"\n1846\t error \"Skipping all remote container operations — check Docker on $REMOTE_SERVER_NAME\"\n1847\t notify \"Remote Docker daemon unresponsive on $REMOTE_SERVER_NAME — remote operations skipped on $(hostname)\" \"Remote Docker Daemon\" \"warning\"\n1848\t fi\n1849\t\n1850\t return 1\n1851\t}\n1852\t\n1853\t# ==============================================================================================\n1854\t# ── UNRAID COMMAND VALIDATION ─────────────────────────────────────────────────────────────────\n1855\t# ==============================================================================================\n1856\t# Verifies a unRAID-specific command exists and produces expected output before use.\n1857\t# Protects against commands moving, changing format, or disappearing after upgrades.\n1858\t#\n1859\t# If validation fails — notifies and returns 1. Caller exits its own section only.\n1860\t# Other scripts that pass validation are unaffected.\n1861\t#\n1862\t# Usage:\n1863\t# validate_unraid_cmd # \"/usr/local/emhttp/plugins/dynamix/scripts/notify\" # \"--help\" # \"Usage\" # \"unRAID notify script\"\n1864\t#\n1865\t# Arguments:\n1866\t# $1 = full path to command\n1867\t# $2 = test argument to pass (use \"\" for no argument)\n1868\t# $3 = expected string in output (use \"\" to skip output check)\n1869\t# $4 = human readable name for error messages\n1870\t\n1871\tvalidate_unraid_cmd() {\n1872\t local cmd_path=\"$1\"\n1873\t local test_arg=\"$2\"\n1874\t local expected_output=\"$3\"\n1875\t local cmd_name=\"${4:-$1}\"\n1876\t\n1877\t # Check command exists and is executable\n1878\t if [[ ! -x \"$cmd_path\" ]]; then\n1879\t error \"unRAID command not found or not executable: $cmd_path\"\n1880\t error \"$cmd_name may have moved or been removed — check after recent unRAID upgrade\"\n1881\t notify \"$cmd_name not found on $(hostname) at $cmd_path — check after unRAID upgrade\" \"Command Validation\" \"warning\"\n1882\t return 1\n1883\t fi\n1884\t\n1885\t # Check output matches expected pattern if provided\n1886\t if [[ -n \"$expected_output\" ]]; then\n1887\t local actual_output\n1888\t actual_output=$(timeout 10 \"$cmd_path\" $test_arg 2>&1 || true)\n1889\t if ! echo \"$actual_output\" | grep -q \"$expected_output\"; then\n1890\t error \"$cmd_name output format changed — expected '$expected_output' not found\"\n1891\t error \"Command may have changed after unRAID upgrade — review $cmd_path\"\n1892\t notify \"$cmd_name output format changed on $(hostname) — may need script update after unRAID upgrade\" \"Command Validation\" \"warning\"\n1893\t return 1\n1894\t fi\n1895\t fi\n1896\t\n1897\t log \"$cmd_name validated: $cmd_path ✅\"\n1898\t return 0\n1899\t}\n1900\t\n1901\t# ==============================================================================================\n1902\t# ── STATUS DISPLAY ────────────────────────────────────────────────────────────────────────────\n1903\t# ==============================================================================================\n1904\t# Prints current runtime configuration — triggered by --status flag in any script.\n1905\t# Useful for verifying detect_hosts() resolved correctly and profile loaded as expected.\n1906\t\n1907\tshow_status() {\n1908\t local local_ver\n1909\t local_ver=$(platform_get_os_version 2>/dev/null || echo \"unknown\")\n1910\t echo \"━━━━━ $ICON_SUMMARY STATUS ━━━━━\"\n1911\t echo \"Local: $LOCAL_SERVER_NAME\"\n1912\t echo \"Remote: $REMOTE_SERVER_NAME\"\n1913\t echo \"IP: ${REMOTE_SERVER:-not resolved}\"\n1914\t echo \"My ID: ${MY_ID:-not set}\"\n1915\t echo \"Remote ID: ${REMOTE_ID:-not set}\"\n1916\t echo \"OS ver: $local_ver\"\n1917\t echo \"Profile: ${PROFILE_NAME:-n/a}\"\n1918\t\n1919\t echo \"DryRun: ${DRY_RUN:-false}\"\n1920\t echo \"Logging: ${ENABLE_LOGGING:-false}\"\n1921\t echo \"SSH Key: ${SSH_KEY:-not set}\"\n1922\t echo \"Containers: ${CRITICAL_CONTAINER_NAMES[*]:-n/a}\"\n1923\t echo \"Delayed: ${DELAYED_CONTAINERS[*]:-n/a}\"\n1924\t echo \"Excludes: ${EXCLUDE_DIRS[*]:-n/a}\"\n1925\t echo \"━━━━━━━━━━━━━━━━━━━━━━━\"\n1926\t}"}]},"uuid":"863d5a10-4cb2-4ad7-8cb4-16534cc57ed7","timestamp":"2026-06-14T05:32:16.107Z","sourceToolAssistantUUID":"0154131e-d0a9-42d6-86db-12212ede3cd7","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"} {"parentUuid":"863d5a10-4cb2-4ad7-8cb4-16534cc57ed7","isSidechain":true,"agentId":"ab976a3a2d54c1360","message":{"model":"claude-haiku-4-5-20251001","id":"msg_018P3VCDQYH3S5YEu7LBzTay","type":"message","role":"assistant","content":[{"type":"text","text":"Now I'll compile my audit findings:\n\n## Audit Report: Bash Script Header Discrepancies\n\n### /boot/config/plugins/varaverk/Fallback/fallback.sh\n- **CONFIGURATION issue**: Header documents `FALLBACK_HOST*_TIER2_DELAY / TIER3_DELAY / TIER4_DELAY` as \"References the remote host's ID\" but the code actually uses `get_tier2_delay()` etc which call `${REMOTE_ID}_TIER2_DELAY` — this is correct behavior but the phrasing in the header is confusing and inaccurate. Header should say \"Keyed by REMOTE_ID: e.g., HOST2_TIER2_DELAY when local server is covering HOST2\"\n\n- **CONFIGURATION issue**: Header lists `FALLBACK_HOST*_WRITEBACK_TIER1–4` with variable pattern `FALLBACK_${REMOTE_ID}_WRITEBACK_TIER${N}` but the actual code at lines 565-576 shows Tier 4 specially uses `HOST*_DAILY_SYNC_SHARES` (line 570: `local shares_var=\"${remote_id}_DAILY_SYNC_SHARES\"`). Header is missing this detail that Tier 4 auto-uses DAILY_SYNC_SHARES.\n\n### ✅ /boot/config/plugins/varaverk/Fallback/fallback_test.sh\nHeader and code match perfectly.\n\n### ✅ /boot/config/plugins/varaverk/Kernel/decision_engine.sh\nHeader and code match perfectly.\n\n### /boot/config/plugins/varaverk/Rsync/rsync.sh\n- **CONFIGURATION issue**: Header CONFIGURATION section lists `PROFILE_RSYNC_OPTS` saying \"does not inherit DEFAULT_RSYNC_OPTS\" but this is explained only in comments. Code at line 1317 in common.sh confirms this: `read -r -a RSYNC_OPTS <<< \"${PROFILE_RSYNC_OPTS[$PROFILE_NAME]}\"` uses profile opts directly without merging defaults. Header is correct but could be clearer.\n\n- **RUNTIME MODES issue**: Header documents `--log` flag (line 128) but the code at line 228 shows it's processed by `parse_args` from common.sh which sets `ENABLE_LOGGING=true`. This is actually implemented correctly, just indirectly through common.sh.\n\n### ✅ /boot/config/plugins/varaverk/load_config.sh\nHeader and code match perfectly. The loader documentation is comprehensive and accurate.\n\n### ✅ /boot/config/plugins/varaverk/common.sh\nExtensive header documentation at lines 1-94 matches all code. All output helpers, arg parser, validation helpers, host detection, and connectivity functions are documented and implemented correctly.\n\n### ✅ /boot/config/plugins/varaverk/git_pull_execute.sh\nHeader and code match perfectly.\n\n---\n\n## Summary\n\n**Real discrepancies found: 2**\n\n1. **fallback.sh line 159**: Header claims `HOST*_TIER2_DELAY / TIER3_DELAY / TIER4_DELAY` \"References the remote host's ID\" but doesn't clearly explain this means the variable is named with the REMOTE_ID prefix (e.g., `HOST2_TIER2_DELAY` not `FALLBACK_HOST2_TIER2_DELAY`).\n\n2. **fallback.sh lines 565-576**: Header omits that Tier 4 writeback automatically includes `HOST*_DAILY_SYNC_SHARES` without requiring a separate `FALLBACK_${REMOTE_ID}_WRITEBACK_TIER4` list—this is a real feature that should be documented in CONFIGURATION."}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens":5,"cache_creation_input_tokens":10540,"cache_read_input_tokens":68336,"output_tokens":894,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":10540},"inference_geo":"not_available","iterations":[{"input_tokens":5,"output_tokens":894,"cache_read_input_tokens":68336,"cache_creation_input_tokens":10540,"cache_creation":{"ephemeral_5m_input_tokens":10540,"ephemeral_1h_input_tokens":0},"type":"message"}],"speed":"standard"},"diagnostics":null},"requestId":"req_011Cc2c3uFqvR9ULeHyy8S2q","attributionAgent":"Explore","type":"assistant","uuid":"349b042d-107f-464c-86d8-17a0c2c7547f","timestamp":"2026-06-14T05:32:27.044Z","userType":"external","entrypoint":"cli","cwd":"/boot/config/plugins/varaverk","sessionId":"685e6c5b-62c1-40bd-9cfd-2c9f7e15c50f","version":"2.1.177","gitBranch":"HEAD","slug":"mutable-sniffing-naur"}