# ━━━━━ FALLBACK — Manual ━━━━━ Config reference, procedures, operational workflows. For overview see README-Fallback.md. For per-script detail see script headers. --- ## ━━━ CONFIGURATION — master.conf ━━━ ``` FALLBACK_ENABLED=true ``` Enable or disable the entire fallback system. `false` = exit cleanly on startup — no monitoring, no container actions. Set false when a server is being rebuilt or fallback is temporarily suspended. Must be explicitly enabled when both servers are ready. **(default: false)** --- ``` FALLBACK_CHECK_INTERVAL=30 ``` Seconds between connectivity checks (ping remote + ping internet). Shorter = faster detection and more pings. With `FALLBACK_HANDBACK_STRIKES=3` at 30s intervals: fallback detected in ≤30s, handback requires 90s of continuous remote-up before beginning. **(default: 30)** --- ``` FALLBACK_HANDBACK_STRIKES=3 ``` Consecutive remote-up checks required before handback begins. Prevents false triggers from brief network recovery during an ongoing outage. 3 strikes × 30s = 90s continuous up required before handback sequence starts. **(default: 3)** --- ``` FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db" ``` Path to the persistent state file. In `$STATE_DIR` — survives reboots whether storage mode is internal (boot device) or appdata (array). If the server was in FALLBACK state when it rebooted, it resumes FALLBACK on restart rather than assuming everything is normal. --- ``` FALLBACK_RSYNC_ENABLED=true ``` Gate for writeback rsync jobs during handback. Set `false` to hand back without syncing any data — useful when the primary still has reliable last-known-good state and writeback would be counterproductive. **(default: true)** --- ``` EXTERNAL_IP=8.8.8.8 ``` IP pinged to verify internet connectivity. **(default: 8.8.8.8)** --- ``` FALLBACK_TEST_BLOCK_WAIT=60 ``` Seconds fallback_test.sh waits in Phase 3 for fallback.sh to detect the simulated outage. Must be greater than `FALLBACK_CHECK_INTERVAL` plus a buffer. At 30s interval: use ≥60s. **(default: 60)** --- ``` FALLBACK_TEST_HANDBACK_WAIT=300 ``` Seconds fallback_test.sh waits in Phase 6 for fallback.sh to complete the full handback sequence. Must cover: strike confirmation window + pre-flight time + rsync duration + container start time. At 3 strikes × 30s + ~2min rsync + ~1min container start: use ≥240s. **(default: 300)** --- ## ━━━ CONFIGURATION — host*.conf ━━━ ``` HOST*_DDNS_CONTAINERS=("...") ``` DDNS containers managed by this host — the containers that update this server's DNS records. Stopped when internet is lost. Started as the very last step of handback after all primary containers are confirmed running. ```bash HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS") HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS") ``` --- ``` FALLBACK_HOST*_STOP_ON_NO_NET=(...) ``` Containers to stop when this host loses internet. Services that are meaningless without internet connectivity. Most hosts leave this empty. --- ``` FALLBACK_HOST*_COVERS_HOST*_TIER1=(...) FALLBACK_HOST*_COVERS_HOST*_TIER2=(...) FALLBACK_HOST*_COVERS_HOST*_TIER3=(...) FALLBACK_HOST*_COVERS_HOST*_TIER4=(...) ``` Containers this host starts for the remote host when the remote is down. TIER1 starts immediately. TIER2–4 activate after the corresponding delay thresholds. Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}` The DDNS container for the remote's domain must be the first entry in TIER1 — DNS coverage before anything else. --- ``` HOST*_TIER2_DELAY=240 HOST*_TIER3_DELAY=720 HOST*_TIER4_DELAY=1440 ``` Minutes after FALLBACK state entry before activating each tier. Variable references the **remote** host's ID — delays for a HOST1 outage use `HOST1_TIER*_DELAY`. ```bash HOST1_TIER2_DELAY=240 # 4 hours — covers most ISP and power issues HOST1_TIER3_DELAY=720 # 12 hours — genuine extended outage HOST1_TIER4_DELAY=1440 # 24 hours — arrs only worth starting at this threshold ``` --- ``` HOST*_TIER1_WRITEBACK_DELAY=60 ``` Minimum outage duration in minutes before Tier 1 writeback runs on handback. Outages shorter than this skip writeback entirely — the primary's last-known-good state is more reliable than a short period of activity on the covering server. ```bash HOST1_TIER1_WRITEBACK_DELAY=60 # skip writeback for outages under 60 minutes ``` --- ``` FALLBACK_HOST*_WRITEBACK_TIER1=(...) FALLBACK_HOST*_WRITEBACK_TIER2=(...) FALLBACK_HOST*_WRITEBACK_TIER3=(...) ``` Paths rsynced back to the remote on handback, per tier. Writeback runs only if the outage exceeded the tier's activation delay. Variable pattern: `FALLBACK_${REMOTE_ID}_WRITEBACK_TIER${N}` **Tier 4 writeback** auto-uses the remote's `HOST*_DAILY_SYNC_SHARES` — the same list daily_sync_maintenance.sh uses, in the opposite direction. No separate TIER4 list needed. --- ## ━━━ FULL CONFIGURATION REFERENCE BLOCKS ━━━ ### master.conf ```bash FALLBACK_ENABLED=true FALLBACK_CHECK_INTERVAL=30 FALLBACK_HANDBACK_STRIKES=3 FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db" FALLBACK_RSYNC_ENABLED=true EXTERNAL_IP=8.8.8.8 FALLBACK_TEST_BLOCK_WAIT=60 FALLBACK_TEST_HANDBACK_WAIT=300 ``` --- ### host2.conf — HOST2 covering HOST1 ```bash HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS") FALLBACK_HOST2_STOP_ON_NO_NET=() # Tier 1 — immediate (vital services + Live TV) FALLBACK_HOST2_COVERS_HOST1_TIER1=( "Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else "Emby" # media server — people are watching "NginxProxyManager" # reverse proxy — all external access routes through this "Lldap-Gmer4Lfe" # user directory — already warm, verify and keep "Mariadb-Authelia" # auth database — already warm, verify and keep "Redis-Authelia" # auth session cache — already warm, verify and keep "Authelia" # SSO — already warm, serving users already "VaultWarden" # passwords — people lock themselves out without this "Dispatcharr" # Live TV scheduler — people are watching right now "Dispatcharr-Basic" "Dispatcharr-Iptv-Users" "ErsatzTV-Emby" # channel schedule builder ) # Tier 2 — after 4 hours (shared productivity services) FALLBACK_HOST2_COVERS_HOST1_TIER2=( "Postgres-NextCloud" # must start before NextCloud "NextCloud" "PostgreSQL-Immich" # must start before Immich "Immich-Gmer4Lfe" "Jellyseerr" ) # Tier 3 — after 12 hours (secondary services) FALLBACK_HOST2_COVERS_HOST1_TIER3=( "Organizrv2-Gmer4Lfe" "AdGuard-Home" "UptimeKuma" "Gitea" "Collabora-CODE" ) # Tier 4 — after 24 hours (arrs + downloaders) FALLBACK_HOST2_COVERS_HOST1_TIER4=( "Sonarr-Gmer4Lfe" "Radarr-Gmer4Lfe" "Lidarr-Gmer4Lfe" "Prowlarr-Gmer4Lfe" "SABnzbd-Gmer4Lfe" "qBittorrent-Gmer4Lfe" "LidaTube" "Pinchflat" ) # Tier delays for HOST1 outage HOST1_TIER2_DELAY=240 HOST1_TIER3_DELAY=720 HOST1_TIER4_DELAY=1440 HOST1_TIER1_WRITEBACK_DELAY=60 # Writeback paths — synced back to HOST1 on handback FALLBACK_HOST1_WRITEBACK_TIER1=( "/mnt/user/appdata-Fallback/Critical-Data" # auth stack — Authelia + NPM + certs "/mnt/user/Media_Server/Emby" # Emby userdata — watch history, playstates ) FALLBACK_HOST1_WRITEBACK_TIER2=( "/mnt/user/appdata-Fallback/Important-Data" # NextCloud + Postgres + Immich ) FALLBACK_HOST1_WRITEBACK_TIER3=( "/mnt/user/appdata-Fallback/Gmer4Lfe" # secondary appdata accumulated changes ) # Tier 4 writeback uses HOST1_DAILY_SYNC_SHARES automatically — no list needed here. ``` --- ### host1.conf — HOST1 covering HOST2 ```bash HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS") FALLBACK_HOST1_STOP_ON_NO_NET=() # Tier 1 — immediate (HOST2's vital services) FALLBACK_HOST1_COVERS_HOST2_TIER1=( "Gmer4Lfe.us-DDNS" # ALWAYS FIRST # HOST2's Tier 1 services — fill per HOST2's stack ) FALLBACK_HOST1_COVERS_HOST2_TIER2=(...) FALLBACK_HOST1_COVERS_HOST2_TIER3=(...) FALLBACK_HOST1_COVERS_HOST2_TIER4=(...) # Tier delays for HOST2 outage HOST2_TIER2_DELAY=240 HOST2_TIER3_DELAY=720 HOST2_TIER4_DELAY=1440 HOST2_TIER1_WRITEBACK_DELAY=60 FALLBACK_HOST2_WRITEBACK_TIER1=( # HOST2's critical data paths ) # Tier 4 writeback uses HOST2_DAILY_SYNC_SHARES automatically. ``` --- ## ━━━ STATE FILE REFERENCE ━━━ Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata) > In a shell where load_config.sh is not sourced, use the full path: > `/boot/config/plugins/varaverk/State_Files/fallback_state.db` (internal storage mode) ``` state=NORMAL # NORMAL | FALLBACK | NO_INTERNET | DARK fallback_start=0 # epoch timestamp when FALLBACK began (0 = not in FALLBACK) handback_strikes=0 # consecutive remote-up checks accumulated toward handback tier2_started=false # whether Tier 2 containers started this event tier3_started=false # whether Tier 3 containers started tier4_started=false # whether Tier 4 containers started ``` View state: `fallback.sh --status` (preferred — parsed output) Raw file: `cat "$STATE_DIR/fallback_state.db"` (requires STATE_DIR set, or use full path) The file is managed exclusively by fallback.sh. Do not edit it while fallback.sh is running — the next cycle will overwrite your changes. Use the Manual State Reset procedure (below) when fallback.sh is not running. --- ## ━━━ INITIAL SETUP ━━━ Before fallback.sh can work on both servers: ### 1. Tailscale Connected on Both Servers ```bash # Verify from HOST1 tailscale ip -4 unRAID-Jayred365 tailscale ping unRAID-Jayred365 # Verify from HOST2 tailscale ip -4 unRAID-Gmer4Lfe tailscale ping unRAID-Gmer4Lfe ``` ### 2. SSH Keys Configured — No Password Prompt ```bash # From HOST1 — should print HOST2's hostname without password prompt ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname" # From HOST2 — should print HOST1's hostname ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname" ``` ### 3. Container Names Match Fallback containers must exist (but stopped) on the covering server, with volume mounts pointing at mirrored share paths. ```bash # Verify on HOST2 that HOST1's container exists (stopped is expected) ssh root@[HOST2-tailscale-ip] "docker inspect Emby --format '{{.State.Status}}'" # Expected: created or exited — NOT "no such container" ``` ### 4. Critical Data Mirrored These shares must exist on HOST2 with current data from HOST1 before fallback is needed: ``` /mnt/user/appdata-Fallback/Critical-Data # auth stack — NPM, LLDAP, Authelia, certs /mnt/user/appdata-Fallback/Important-Data # NextCloud + Postgres + Immich /mnt/user/Media_Server/Emby # Emby userdata — watch history, playstates /mnt/user/appdata-Fallback/Gmer4Lfe # server appdata ``` ```bash # Verify data is current — check modification times ssh root@[HOST2-tailscale-ip] "ls -la /mnt/user/appdata-Fallback/Critical-Data/" ``` Sync is maintained continuously by `daily_sync_maintenance.sh` critical-data profile. ### 5. DDNS TTL Set to 1 Minute Set in your DDNS provider settings. Higher TTL means users continue hitting the old IP for longer after fallback. At 5-minute TTL, users can be hitting a downed server for up to 5 minutes before DNS switches. ### 6. Both Servers Running fallback.sh Fallback only works in one direction if only one server is running the script. Both servers must be running it continuously for mutual coverage. ```bash # Verify fallback.sh is running pgrep -f "fallback.sh" # Check the state file cat "$STATE_DIR/fallback_state.db" ``` Start via User Scripts plugin on both servers. --- ## ━━━ MONITORING ━━━ **Sunday coffee report** (`sunday_morning_coffee_report.sh`) — Fallback section shows current state, outage duration if not NORMAL, Tailscale reachability, and whether fallback.sh is running. **Weekly health digest** (`weekly_health_digest.sh`) — reads the state file. If `DIGEST_SMART_ON_FALLBACK=true` and state is not NORMAL, it sends a notification even in smart mode — a non-NORMAL state at digest time needs attention. **Direct check:** ```bash fallback.sh --status # full state snapshot cat "$STATE_DIR/fallback_state.db" # raw state file ``` --- ## ━━━ PROCEDURES ━━━ ### Running the Fallback Test > This starts and stops real containers on both servers. Users will experience a brief > service interruption. Always run `--dry-run` first. ```bash # Step 1 — verify all phases without touching anything fallback_test.sh --dry-run # Step 2 — schedule maintenance window, then run live fallback_test.sh # Step 3 — check state after test completes fallback.sh --status cat "$STATE_DIR/fallback_state.db" ``` If the test doesn't complete cleanly, the state file may be left in FALLBACK. The iptables safety trap in fallback_test.sh removes the DROP rule on any exit, so remote connectivity is always restored. Use the Manual State Reset procedure below if the state file is stuck. ### Manual State Reset Use when the state file is stuck in a non-NORMAL state after testing, a failed handback, or killing fallback.sh directly (not via User Scripts Abort). **Before resetting, verify the situation is actually safe to reset:** ```bash # 1. Right containers running on the right server docker ps | grep -E "Emby|VaultWarden|NginxProxyManager" # 2. DDNS pointing at the correct server nslookup Gmer4Lfe.com 8.8.8.8 nslookup Gmer4Lfe.us 8.8.8.8 # 3. Both servers visible on Tailscale tailscale ping [remote-tailscale-ip] # 4. No actual fallback in progress (remote is genuinely up and stable) ping -c 5 [remote-tailscale-ip] ``` **Stop fallback.sh first (via User Scripts Abort), then reset:** ```bash # Set STATE_DIR (or source load_config.sh to get it from the environment) source /boot/config/plugins/varaverk/load_config.sh # View current state cat "$STATE_DIR/fallback_state.db" # Write a clean NORMAL state cat > "$STATE_DIR/fallback_state.db" << 'EOF' state=NORMAL fallback_start=0 handback_strikes=0 tier2_started=false tier3_started=false tier4_started=false EOF # Verify the write cat "$STATE_DIR/fallback_state.db" ``` Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its next cycle. > **Warning:** Do NOT reset during an actual fallback event. fallback.sh will think > everything is normal and stop covering the remote — services go offline until the next > detection cycle catches it again. ### Adding a Container to a Tier 1. Create the container on the covering server (stopped), with volume mounts pointing at the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server) 2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf in the appropriate tier position (dependency ordering — databases before apps) 3. Verify: `fallback.sh --status` shows the container in the expected tier list 4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid --- ## ━━━ TROUBLESHOOTING ━━━ ### Fallback Not Triggering ``` Is fallback.sh running on the covering server? → User Scripts plugin → check status of the fallback script → pgrep -f "fallback.sh" Is Tailscale connected? → tailscale status (should show the remote peer) Can this server reach the remote Tailscale IP? → ping [remote-tailscale-ip] What does fallback.sh report? → fallback.sh --status → cat "$STATE_DIR/fallback_state.db" ``` ### Handback Not Completing ``` Is the primary's array fully started? → ls /mnt/user (should show share directories) Is Docker responding on the primary? → docker ps (should return a list, not hang) Is rootfs nearly full? (pre-flight checks this) → df / Is rsync running and stuck? → pgrep rsync → A stalled rsync blocks handback. Let fallback.sh retry next cycle. ``` ### DDNS Not Cutting Over ``` Is the DDNS container running on the covering server? → docker ps | grep DDNS What TTL is the DNS record set to? → nslookup Gmer4Lfe.com 8.8.8.8 (check TTL in response) → High TTL = slow propagation Is the DDNS provider accepting updates? → docker logs [ddns-container] --tail 50 ``` ### Split Brain — Both DDNS Running ``` This should not happen if the handback sequence completed correctly. If it has happened: 1. Check both servers for running DDNS containers HOST1: docker ps | grep DDNS HOST2: docker ps | grep DDNS 2. Stop the duplicate (the one on the covering server) docker stop [duplicate-ddns-container] 3. Understand the state before resetting fallback.sh --status cat "$STATE_DIR/fallback_state.db" 4. Perform Manual State Reset above on the server in a bad state 5. Restart fallback.sh via User Scripts plugin ``` --- ## ━━━ OUTPUT TIERS ━━━ Both scripts have two output levels controlled by `--log`. **fallback.sh** — daemon, runs continuously. Without `--log`, only state transitions, warnings, errors, and the startup banner are visible. Per-cycle detail (ping results, state evaluation) is suppressed — the daemon runs every 15–30 seconds and clean cycles produce no output by design. State transitions are always visible because they are the events that matter. Use `--log` when debugging why the state machine is or is not acting. **fallback_test.sh** — one-shot test harness. Without `--log`, phase headers, pass/fail results, and the final test report are always visible. Per-container checks within each phase are suppressed. With `--log`, every check in every phase is shown. Warnings and errors are always visible regardless of `--log`. --- ## ━━━ FLAG REFERENCE ━━━ ### fallback.sh `fallback.sh` Normal start — continuous loop. Started automatically by `array_started.sh` at array start. **Do NOT stop by killing the process** — state file may be left inconsistent. Stop via `fallback.sh --stop` only. `fallback.sh --dry-run` Walk through one full cycle showing what would happen based on current network state. No containers started or stopped. No DDNS changes. Use to verify configuration before relying on it. `fallback.sh --status` Show current state, identity, DDNS containers, check interval, handback strikes, and (if in FALLBACK) outage duration and tier activation status. Use this first to understand current state before any manual intervention. `fallback.sh --log` Verbose output on every decision in every cycle — ping results, state evaluation, tier checks. Use when debugging why the state machine is or is not acting as expected. --- ### fallback_test.sh `fallback_test.sh --dry-run` Walk through all 7 phases with full output. No iptables rules added. No container starts or stops. **Always run this before a live test** — confirms timing configuration is correct and phases would pass before committing to real changes. `fallback_test.sh` Full live test — real iptables DROP rule, real container lifecycle. Users will experience a brief service interruption. Run during a maintenance window. The iptables safety trap removes the DROP rule on any exit (normal completion, crash, ctrl-c). `fallback_test.sh --status` Show current fallback state, test timing configuration, and the Tier 1 containers that would be tested. No test run. `fallback_test.sh --log` Verbose output on every check in every phase.