Varaverk/Fallback/Manual-Fallback.md

━━━━━ FALLBACK — Manual ━━━━━

Config reference, procedures, operational workflows. For overview see README-Fallback.md. For per-script detail see script headers.

---

━━━ CONFIGURATION — master.conf ━━━

FALLBACK_ENABLED=true

Enable or disable the entire fallback system. false = exit cleanly on startup — no monitoring, no container actions. Set false when a server is being rebuilt or fallback is temporarily suspended. Must be explicitly enabled when both servers are ready. (default: false)

---

FALLBACK_CHECK_INTERVAL=30

Seconds between connectivity checks (ping remote + ping internet). Shorter = faster detection and more pings. With FALLBACK_HANDBACK_STRIKES=3 at 30s intervals: fallback detected in ≤30s, handback requires 90s of continuous remote-up before beginning. (default: 30)

---

FALLBACK_HANDBACK_STRIKES=3

Consecutive remote-up checks required before handback begins. Prevents false triggers from brief network recovery during an ongoing outage. 3 strikes × 30s = 90s continuous up required before handback sequence starts. (default: 3)

---

FALLBACK_STATE_FILE=/boot/config/fallback_state.db

Path to the persistent state file. Lives on /boot/ intentionally — survives reboots. If the server was in FALLBACK state when it rebooted, it resumes FALLBACK on restart rather than assuming everything is normal.

---

FALLBACK_RSYNC_ENABLED=true

Gate for writeback rsync jobs during handback. Set false to hand back without syncing any data — useful when the primary still has reliable last-known-good state and writeback would be counterproductive. (default: true)

---

EXTERNAL_IP=8.8.8.8

IP pinged to verify internet connectivity. (default: 8.8.8.8)

---

FALLBACK_TEST_BLOCK_WAIT=60

Seconds fallback_test.sh waits in Phase 3 for fallback.sh to detect the simulated outage. Must be greater than FALLBACK_CHECK_INTERVAL plus a buffer. At 30s interval: use ≥60s. (default: 60)

---

FALLBACK_TEST_HANDBACK_WAIT=300

Seconds fallback_test.sh waits in Phase 6 for fallback.sh to complete the full handback sequence. Must cover: strike confirmation window + pre-flight time + rsync duration + container start time. At 3 strikes × 30s + ~2min rsync + ~1min container start: use ≥240s. (default: 300)

---

━━━ CONFIGURATION — host*.conf ━━━

HOST*_DDNS_CONTAINERS=("...")

DDNS containers managed by this host — the containers that update this server's DNS records. Stopped when internet is lost. Started as the very last step of handback after all primary containers are confirmed running.

HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")

---

FALLBACK_HOST*_STOP_ON_NO_NET=(...)

Containers to stop when this host loses internet. Services that are meaningless without internet connectivity. Most hosts leave this empty.

---

FALLBACK_HOST*_COVERS_HOST*_TIER1=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER2=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER3=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER4=(...)

Containers this host starts for the remote host when the remote is down. TIER1 starts immediately. TIER2–4 activate after the corresponding delay thresholds.

Variable pattern: FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}

The DDNS container for the remote's domain must be the first entry in TIER1 — DNS coverage before anything else.

---

HOST*_TIER2_DELAY=240
HOST*_TIER3_DELAY=720
HOST*_TIER4_DELAY=1440

Minutes after FALLBACK state entry before activating each tier. Variable references the remote host's ID — delays for a HOST1 outage use HOST1_TIER*_DELAY.

HOST1_TIER2_DELAY=240     # 4 hours — covers most ISP and power issues
HOST1_TIER3_DELAY=720     # 12 hours — genuine extended outage
HOST1_TIER4_DELAY=1440    # 24 hours — arrs only worth starting at this threshold

---

HOST*_TIER1_WRITEBACK_DELAY=60

Minimum outage duration in minutes before Tier 1 writeback runs on handback. Outages shorter than this skip writeback entirely — the primary's last-known-good state is more reliable than a short period of activity on the covering server.

HOST1_TIER1_WRITEBACK_DELAY=60    # skip writeback for outages under 60 minutes

---

FALLBACK_HOST*_WRITEBACK_TIER1=(...)
FALLBACK_HOST*_WRITEBACK_TIER2=(...)
FALLBACK_HOST*_WRITEBACK_TIER3=(...)

Paths rsynced back to the remote on handback, per tier. Writeback runs only if the outage exceeded the tier's activation delay.

Variable pattern: FALLBACK_${REMOTE_ID}_WRITEBACK_TIER${N}

Tier 4 writeback auto-uses the remote's HOST*_DAILY_SYNC_SHARES — the same list daily_sync_maintenance.sh uses, in the opposite direction. No separate TIER4 list needed.

---

━━━ FULL CONFIGURATION REFERENCE BLOCKS ━━━

master.conf

FALLBACK_ENABLED=true
FALLBACK_CHECK_INTERVAL=30
FALLBACK_HANDBACK_STRIKES=3
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
FALLBACK_RSYNC_ENABLED=true
EXTERNAL_IP=8.8.8.8
FALLBACK_TEST_BLOCK_WAIT=60
FALLBACK_TEST_HANDBACK_WAIT=300

---

host2.conf — HOST2 covering HOST1

HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")

FALLBACK_HOST2_STOP_ON_NO_NET=()

# Tier 1 — immediate (vital services + Live TV)
FALLBACK_HOST2_COVERS_HOST1_TIER1=(
    "Gmer4Lfe.com-DDNS"         # ALWAYS FIRST — DNS coverage before anything else
    "Emby"                       # media server — people are watching
    "NginxProxyManager"          # reverse proxy — all external access routes through this
    "Lldap-Gmer4Lfe"             # user directory — already warm, verify and keep
    "Mariadb-Authelia"           # auth database — already warm, verify and keep
    "Redis-Authelia"             # auth session cache — already warm, verify and keep
    "Authelia"                   # SSO — already warm, serving users already
    "VaultWarden"                # passwords — people lock themselves out without this
    "Dispatcharr"                # Live TV scheduler — people are watching right now
    "Dispatcharr-Basic"
    "Dispatcharr-Iptv-Users"
    "ErsatzTV-Emby"              # channel schedule builder
)

# Tier 2 — after 4 hours (shared productivity services)
FALLBACK_HOST2_COVERS_HOST1_TIER2=(
    "Postgres-NextCloud"         # must start before NextCloud
    "NextCloud"
    "PostgreSQL-Immich"          # must start before Immich
    "Immich-Gmer4Lfe"
    "Jellyseerr"
)

# Tier 3 — after 12 hours (secondary services)
FALLBACK_HOST2_COVERS_HOST1_TIER3=(
    "Organizrv2-Gmer4Lfe"
    "AdGuard-Home"
    "UptimeKuma"
    "Gitea"
    "Collabora-CODE"
)

# Tier 4 — after 24 hours (arrs + downloaders)
FALLBACK_HOST2_COVERS_HOST1_TIER4=(
    "Sonarr-Gmer4Lfe"
    "Radarr-Gmer4Lfe"
    "Lidarr-Gmer4Lfe"
    "Prowlarr-Gmer4Lfe"
    "SABnzbd-Gmer4Lfe"
    "qBittorrent-Gmer4Lfe"
    "LidaTube"
    "Pinchflat"
)

# Tier delays for HOST1 outage
HOST1_TIER2_DELAY=240
HOST1_TIER3_DELAY=720
HOST1_TIER4_DELAY=1440
HOST1_TIER1_WRITEBACK_DELAY=60

# Writeback paths — synced back to HOST1 on handback
FALLBACK_HOST1_WRITEBACK_TIER1=(
    "/mnt/user/appdata-Failover/Critical-Data"    # auth stack — Authelia + NPM + certs
    "/mnt/user/Media_Server/Emby"                 # Emby userdata — watch history, playstates
)

FALLBACK_HOST1_WRITEBACK_TIER2=(
    "/mnt/user/appdata-Failover/Important-Data"   # NextCloud + Postgres + Immich
)

FALLBACK_HOST1_WRITEBACK_TIER3=(
    "/mnt/user/appdata-Failover/Gmer4Lfe"         # secondary appdata accumulated changes
)

# Tier 4 writeback uses HOST1_DAILY_SYNC_SHARES automatically — no list needed here.

---

host1.conf — HOST1 covering HOST2

HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")

FALLBACK_HOST1_STOP_ON_NO_NET=()

# Tier 1 — immediate (HOST2's vital services)
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
    "Gmer4Lfe.us-DDNS"          # ALWAYS FIRST
    # HOST2's Tier 1 services — fill per HOST2's stack
)

FALLBACK_HOST1_COVERS_HOST2_TIER2=(...)
FALLBACK_HOST1_COVERS_HOST2_TIER3=(...)
FALLBACK_HOST1_COVERS_HOST2_TIER4=(...)

# Tier delays for HOST2 outage
HOST2_TIER2_DELAY=240
HOST2_TIER3_DELAY=720
HOST2_TIER4_DELAY=1440
HOST2_TIER1_WRITEBACK_DELAY=60

FALLBACK_HOST2_WRITEBACK_TIER1=(
    # HOST2's critical data paths
)

# Tier 4 writeback uses HOST2_DAILY_SYNC_SHARES automatically.

---

━━━ STATE FILE REFERENCE ━━━

Location: /boot/config/fallback_state.db (survives reboots)

state=NORMAL            # NORMAL | FALLBACK | NO_INTERNET | DARK
fallback_start=0        # epoch timestamp when FALLBACK began (0 = not in FALLBACK)
handback_strikes=0      # consecutive remote-up checks accumulated toward handback
tier2_started=false     # whether Tier 2 containers started this event
tier3_started=false     # whether Tier 3 containers started
tier4_started=false     # whether Tier 4 containers started

View state: cat /boot/config/fallback_state.db
Check state: fallback.sh --status

The file is managed exclusively by fallback.sh. Do not edit it while fallback.sh is running — the next cycle will overwrite your changes. Use the Manual State Reset procedure (below) when fallback.sh is not running.

---

━━━ INITIAL SETUP ━━━

Before fallback.sh can work on both servers:

1. Tailscale Connected on Both Servers

# Verify from HOST1
tailscale ip -4 unRAID-Jayred365
tailscale ping unRAID-Jayred365

# Verify from HOST2
tailscale ip -4 unRAID-Gmer4Lfe
tailscale ping unRAID-Gmer4Lfe

2. SSH Keys Configured — No Password Prompt

# From HOST1 — should print HOST2's hostname without password prompt
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"

# From HOST2 — should print HOST1's hostname
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"

3. Container Names Match

Fallback containers must exist (but stopped) on the covering server, with volume mounts pointing at mirrored share paths.

# Verify on HOST2 that HOST1's container exists (stopped is expected)
ssh root@[HOST2-tailscale-ip] "docker inspect Emby --format '{{.State.Status}}'"
# Expected: created or exited — NOT "no such container"

4. Critical Data Mirrored

These shares must exist on HOST2 with current data from HOST1 before failover is needed:

/mnt/user/appdata-Failover/Critical-Data    # auth stack — NPM, LLDAP, Authelia, certs
/mnt/user/appdata-Failover/Important-Data   # NextCloud + Postgres + Immich
/mnt/user/Media_Server/Emby                 # Emby userdata — watch history, playstates
/mnt/user/appdata-Failover/Gmer4Lfe         # server appdata

# Verify data is current — check modification times
ssh root@[HOST2-tailscale-ip] "ls -la /mnt/user/appdata-Failover/Critical-Data/"

Sync is maintained continuously by daily_sync_maintenance.sh critical-data profile.

5. DDNS TTL Set to 1 Minute

Set in your DDNS provider settings. Higher TTL means users continue hitting the old IP for longer after failover. At 5-minute TTL, users can be hitting a downed server for up to 5 minutes before DNS switches.

6. Both Servers Running fallback.sh

Fallback only works in one direction if only one server is running the script. Both servers must be running it continuously for mutual coverage.

# Verify fallback.sh is running
pgrep -f "fallback.sh"

# Check the state file
cat /boot/config/fallback_state.db

Start via User Scripts plugin on both servers.

---

━━━ MONITORING ━━━

Sunday coffee report (sunday_morning_coffee_report.sh) — Fallback section shows current state, outage duration if not NORMAL, Tailscale reachability, and whether fallback.sh is running.

Weekly health digest (weekly_health_digest.sh) — reads the state file. If DIGEST_SMART_ON_FAILOVER=true and state is not NORMAL, it sends a notification even in smart mode — a non-NORMAL state at digest time needs attention.

Direct check:

fallback.sh --status          # full state snapshot
cat /boot/config/fallback_state.db    # raw state file

---

━━━ PROCEDURES ━━━

Running the Failover Test

This starts and stops real containers on both servers. Users will experience a brief service interruption. Always run --dry-run first.

# Step 1 — verify all phases without touching anything
fallback_test.sh --dry-run

# Step 2 — schedule maintenance window, then run live
fallback_test.sh

# Step 3 — check state after test completes
fallback.sh --status
cat /boot/config/fallback_state.db

If the test doesn't complete cleanly, the state file may be left in FALLBACK. The iptables safety trap in fallback_test.sh removes the DROP rule on any exit, so remote connectivity is always restored. Use the Manual State Reset procedure below if the state file is stuck.

Manual State Reset

Use when the state file is stuck in a non-NORMAL state after testing, a failed handback, or killing fallback.sh directly (not via User Scripts Abort).

Before resetting, verify the situation is actually safe to reset:

# 1. Right containers running on the right server
docker ps | grep -E "Emby|VaultWarden|NginxProxyManager"

# 2. DDNS pointing at the correct server
nslookup Gmer4Lfe.com 8.8.8.8
nslookup Gmer4Lfe.us 8.8.8.8

# 3. Both servers visible on Tailscale
tailscale ping [remote-tailscale-ip]

# 4. No actual fallback in progress (remote is genuinely up and stable)
ping -c 5 [remote-tailscale-ip]

Stop fallback.sh first (via User Scripts Abort), then reset:

# View current state
cat /boot/config/fallback_state.db

# Write a clean NORMAL state
cat > /boot/config/fallback_state.db << 'EOF'
state=NORMAL
fallback_start=0
handback_strikes=0
tier2_started=false
tier3_started=false
tier4_started=false
EOF

# Verify the write
cat /boot/config/fallback_state.db

Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its next cycle.

Warning: Do NOT reset during an actual fallback event. fallback.sh will think everything is normal and stop covering the remote — services go offline until the next detection cycle catches it again.

Adding a Container to a Tier

1. Create the container on the covering server (stopped), with volume mounts pointing at the mirrored share path (e.g. /mnt/user/Movies must exist on the covering server)
2. Add the container name to FALLBACK_HOST*_COVERS_HOST*_TIER* in host*.conf in the appropriate tier position (dependency ordering — databases before apps)
3. Verify: fallback.sh --status shows the container in the expected tier list
4. Run fallback_test.sh --dry-run to confirm the full configuration is valid

---

━━━ TROUBLESHOOTING ━━━

Fallback Not Triggering

Is fallback.sh running on the covering server?
  → User Scripts plugin → check status of the fallback script
  → pgrep -f "fallback.sh"

Is Tailscale connected?
  → tailscale status   (should show the remote peer)

Can this server reach the remote Tailscale IP?
  → ping [remote-tailscale-ip]

What does fallback.sh report?
  → fallback.sh --status
  → cat /boot/config/fallback_state.db

Handback Not Completing

Is the primary's array fully started?
  → ls /mnt/user   (should show share directories)

Is Docker responding on the primary?
  → docker ps   (should return a list, not hang)

Is rootfs nearly full? (pre-flight checks this)
  → df /

Is rsync running and stuck?
  → pgrep rsync
  → A stalled rsync blocks handback. Let fallback.sh retry next cycle.

DDNS Not Cutting Over

Is the DDNS container running on the covering server?
  → docker ps | grep DDNS

What TTL is the DNS record set to?
  → nslookup Gmer4Lfe.com 8.8.8.8   (check TTL in response)
  → High TTL = slow propagation

Is the DDNS provider accepting updates?
  → docker logs [ddns-container] --tail 50

Split Brain — Both DDNS Running

This should not happen if the handback sequence completed correctly.
If it has happened:

1. Check both servers for running DDNS containers
   HOST1:  docker ps | grep DDNS
   HOST2:  docker ps | grep DDNS

2. Stop the duplicate (the one on the covering server)
   docker stop [duplicate-ddns-container]

3. Understand the state before resetting
   fallback.sh --status
   cat /boot/config/fallback_state.db

4. Perform Manual State Reset above on the server in a bad state

5. Restart fallback.sh via User Scripts plugin

---

━━━ OUTPUT TIERS ━━━

Both scripts have two output levels controlled by --log.

fallback.sh — daemon, runs continuously. Without --log, only state transitions, warnings, errors, and the startup banner are visible. Per-cycle detail (ping results, state evaluation) is suppressed — the daemon runs every 15–30 seconds and clean cycles produce no output by design. State transitions are always visible because they are the events that matter. Use --log when debugging why the state machine is or is not acting.

fallback_test.sh — one-shot test harness. Without --log, phase headers, pass/fail results, and the final test report are always visible. Per-container checks within each phase are suppressed. With --log, every check in every phase is shown. Warnings and errors are always visible regardless of --log.

---

━━━ FLAG REFERENCE ━━━

fallback.sh

fallback.sh
Normal start — continuous loop. Start via User Scripts plugin or array_started.sh. Do NOT stop by killing the process — state file may be left inconsistent. Stop via User Scripts Abort only.

fallback.sh --dry-run
Walk through one full cycle showing what would happen based on current network state. No containers started or stopped. No DDNS changes. Use to verify configuration before relying on it.

fallback.sh --status
Show current state, identity, DDNS containers, check interval, handback strikes, and (if in FALLBACK) outage duration and tier activation status. Use this first to understand current state before any manual intervention.

fallback.sh --log
Verbose output on every decision in every cycle — ping results, state evaluation, tier checks. Use when debugging why the state machine is or is not acting as expected.

---

fallback_test.sh

fallback_test.sh --dry-run
Walk through all 7 phases with full output. No iptables rules added. No container starts or stops. Always run this before a live test — confirms timing configuration is correct and phases would pass before committing to real changes.

fallback_test.sh
Full live test — real iptables DROP rule, real container lifecycle. Users will experience a brief service interruption. Run during a maintenance window. The iptables safety trap removes the DROP rule on any exit (normal completion, crash, ctrl-c).

fallback_test.sh --status
Show current fallback state, test timing configuration, and the Tier 1 containers that would be tested. No test run.

fallback_test.sh --log
Verbose output on every check in every phase.