FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8c5667b1dfec3a9caeaab53da2bbfaf201206933
Varaverk/Failover/README-Failover.md
T

17 KiB
Raw Blame History

Failover

Mutual container failover between two unRAID servers. When one server goes down, the other automatically starts its containers. When it comes back, everything hands back cleanly with data synced and DNS cutting over at exactly the right moment.

This system was built from scratch and refined through a year of production testing before being standardised into this ecosystem. The DDNS sequencing and handback order were the hardest parts to get right — the logic is documented here so it's never lost.

The Setup

Two fully independent unRAID servers connected via Tailscale:

HOST1 — unRAID-Gmer4Lfe (Primary)
  Location:   Local
  Owns:       Gmer4Lfe.com DDNS
  Runs:       Full service stack
  
HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
  Location:   Remote — 50 miles away
  Owns:       Gmer4Lfe.us DDNS
  Runs:       Its own service stack + mirrors HOST1 data

Both servers run failover.sh as a background task continuously. Neither server knows what the other is doing — they only know what they can ping from their own network perspective.

How It Works

Every FAILOVER_CHECK_INTERVAL seconds (default: 120s) each server makes two pings:

1. Ping remote server (Tailscale IP)
2. Ping internet (8.8.8.8)

The combination of those two results determines the current state and what action to take. That's it. No SSH signaling between servers, no shared state files, no coordination — pure autonomous decision making based on observable facts.

States

NORMAL

Remote: reachable   Internet: reachable

Both servers running normally. Each server runs its own containers. DDNS on — pointing DNS at this server's IP. Silent operation.

FAILOVER

Remote: unreachable   Internet: reachable

The remote server is down but this server has internet. Start the remote server's containers locally. The remote's DDNS container is started first — DNS starts pointing at this server immediately. Failover is additive — your own containers keep running, remote containers are added on top.

NO_INTERNET

Internet: unreachable   (remote state unknown)

This server has lost internet. Stop own DDNS immediately — no point updating DNS records when you can't reach the outside world and it would send conflicting updates. Do not start remote containers — there's no internet to serve them on. Wait for recovery.

DARK

Remote: unreachable   Internet: unreachable

Both pings fail. Same actions as NO_INTERNET — can't determine if remote is truly down or just unreachable through the same outage affecting your internet. Conservative approach: stop DDNS, wait.

DDNS — The Critical Part

This took a year to get right. Do not change the sequencing.

Each server owns one DDNS container. The script controls when each DDNS runs — the network state never auto-starts DDNS.

NORMAL:
  HOST1 DDNS (Gmer4Lfe.com)  → ON   — always on while HOST1 has internet
  HOST2 DDNS (Gmer4Lfe.us)   → ON   — always on while HOST2 has internet

HOST1 loses internet:
  HOST1 DDNS                  → OFF  — immediately stopped
  HOST2 DDNS                  → stays ON (unaffected)
  
HOST2 detects HOST1 is down:
  HOST1 DDNS (on HOST2)       → ON   — HOST2 starts it as Tier 1 action
  DNS now points at HOST2's IP
  
HOST1 returns — handback:
  HOST1 DDNS (on HOST2)       → OFF  — stopped FIRST before anything else
  ... rsync runs ...
  HOST1 containers start on HOST1
  HOST1 DDNS (on HOST1)       → ON   — started LAST after containers confirmed up

Why DDNS never auto-starts on internet return:

If HOST1 lost internet and its DDNS auto-started when internet returned, you'd have both HOST1 and HOST2 running the same DDNS simultaneously — pointing DNS at two different IPs at the same time. DNS TTL is 1 minute — users would get routed randomly between servers. This is called split brain and it causes exactly the kind of intermittent failures that are hard to diagnose.

The script is the sole authority over when DDNS starts. Network state coming back is not permission to start DDNS. Only the completion of the full handback sequence is.

One domain per server, one DDNS active per domain, always:

Gmer4Lfe.com → runs on HOST1 normally, moves to HOST2 during HOST1 outage
Gmer4Lfe.us  → runs on HOST2 normally, moves to HOST1 during HOST2 outage

Tiered Failover

Not every service needs to start immediately when the other server goes down. Starting the full stack on a secondary server wastes resources for short outages — most are resolved in minutes.

Tier 1 — Immediate (0 min)
  Remote DDNS                ← DNS coverage first
  Emby                       ← media server, people are watching
  NginxProxyManager          ← reverse proxy, everything routes through this
  Lldap, Mariadb, Redis      ← auth stack, required by everything proxied
  Authelia x2                ← authentication
  VaultWarden                ← passwords, needed immediately
  Dispatcharr x3             ← Live TV, people are watching right now
  ErsatzTV                   ← Live TV scheduling

Tier 2 — After HOST1_TIER2_DELAY minutes (default: 120min)
  NextCloud + Postgres        ← file access
  Immich + PostgreSQL         ← photos
  Jellyseerr                  ← media requests

Tier 3 — After HOST1_TIER3_DELAY minutes (default: 360min)
  Organizrv2                  ← dashboard
  AdGuard-Home                ← DNS filtering
  UptimeKuma                  ← monitoring
  Gitea                       ← git server
  Collabora-CODE              ← document editing

Tier 4 — After HOST1_TIER4_DELAY minutes (default: 1080min / 18hr)
  Full arr stack              ← Sonarr, Radarr, Lidarr, Prowlarr etc.
  Downloaders                 ← SABnzbd, Qbittorrent, LidaTube, Pinchflat
  24hr+ outage = full workflow continuity

Live TV is Tier 1 because people are watching. You cannot tell a household mid-game that their live TV will be back in 2 hours.

Arrs and downloaders are Tier 4 because they generate significant I/O and have minimal writeback on handback — if HOST1 comes back before 18 hours, the arrs never started on HOST2 and there's nothing to sync back.

Handback Sequence

When HOST1 returns after being down, the handback must happen in exactly this order:

1. Strike confirmation
   — FAILOVER_HANDBACK_STRIKES consecutive remote-up checks
   — prevents handing back during a brief network blip
   — 2 strikes × 120s = 4 minute confirmation window

2. Pre-flight checks
   — Remote array is started
   — Remote Docker daemon is responding
   — Remote rootfs is not nearly full
   — Abort if any check fails — retry next cycle

3. Stop remote DDNS FIRST
   — DNS stops updating before anything moves
   — Prevents split brain during the transition window
   — This is the most critical ordering step

4. Stop remote containers
   — Clean state before rsync
   — No competing writes during transfer
   — Containers are only down during the rsync window
   — This minimises user disruption

5. Rsync writeback
   — Full bandwidth available — DDNS stopped, containers stopped
   — Only critical data synced back:
       appdata-Failover/Critical-Data   (auth stack)
       appdata-Failover/Important-Data  (NextCloud + Postgres)
       appdata-Failover/Emby            (userdata, playstates)
       appdata-Failover/Gmer4Lfe        (server appdata)
   — Media files skipped — already on HOST1, never moved
   — Downloads skipped — start fresh is cleaner

6. Start local containers
   — Dependencies respected — databases before apps
   — Brief pause to let databases initialise before dependents start

7. Start local DDNS LAST
   — DNS only cuts back after containers are confirmed up
   — Users hit HOST1 only after it's actually ready to serve them

8. Return to NORMAL
   — State file reset
   — Tier flags cleared
   — Next cycle confirms everything is healthy

Why containers stop before rsync:

Earlier versions synced while containers were still running on the remote. This caused:

  • Rsync competing with active container I/O — slower transfers
  • Files changing mid-transfer — potential inconsistency
  • Database writes during sync — dirty state on handback

Stopping containers first means rsync gets a clean static source at full bandwidth. The window where containers are down is the rsync duration only — typically minutes.

Mutual Failover — Both Directions

The same script handles both directions. detect_hosts() in common.sh determines which server is local and which is remote at runtime, then selects the correct arrays from Master.conf.

HOST2 covers HOST1 (HOST1 goes down):
  Uses: FAILOVER_HOST2_RUNS_FOR_HOST1_* arrays
  Tiers configured by: HOST1_TIER*_DELAY variables

HOST1 covers HOST2 (HOST2 goes down):
  Uses: FAILOVER_HOST1_RUNS_FOR_HOST2_* arrays
  Tiers configured by: HOST2_TIER*_DELAY variables

Both servers run identical scripts. The configuration in Master.conf controls what each server does for the other.

Configuration

All configuration in Master.conf under the ── FAILOVER ── section.

# Core timing
EXTERNAL_IP="8.8.8.8"              # internet ping target
FAILOVER_CHECK_INTERVAL=120         # seconds between checks
FAILOVER_HANDBACK_STRIKES=2         # confirmations before handback
FAILOVER_STATE_FILE="/boot/config/failover_state.db"

# DDNS ownership — one per server, script controlled exclusively
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com")
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us")

# What HOST2 runs for HOST1 (tiered)
FAILOVER_HOST2_RUNS_FOR_HOST1_IMMEDIATE=(...)
FAILOVER_HOST2_RUNS_FOR_HOST1_2HR=(...)
FAILOVER_HOST2_RUNS_FOR_HOST1_6HR=(...)
FAILOVER_HOST2_RUNS_FOR_HOST1_18HR=(...)

# What HOST1 runs for HOST2 (tiered)
FAILOVER_HOST1_RUNS_FOR_HOST2_IMMEDIATE=(...)
# ...

# Tier delays — configurable per host, in minutes
HOST1_TIER2_DELAY=120
HOST1_TIER3_DELAY=360
HOST1_TIER4_DELAY=1080
HOST2_TIER2_DELAY=120
HOST2_TIER3_DELAY=360
HOST2_TIER4_DELAY=1080

# Writeback jobs on handback
FAILOVER_HOST1_WRITEBACK=(
    "/mnt/user/appdata-Failover/Critical-Data"
    "/mnt/user/appdata-Failover/Important-Data"
    "/mnt/user/appdata-Failover/Emby"
    "/mnt/user/appdata-Failover/Gmer4Lfe"
)

Initial Setup Requirements

Before failover.sh can run on both servers:

1. Tailscale connected on both servers

# Verify on HOST1
tailscale ip -4 unRAID-Jayred365   # should return HOST2's Tailscale IP

# Verify on HOST2
tailscale ip -4 unRAID-Gmer4Lfe   # should return HOST1's Tailscale IP

2. SSH keys configured

HOST1 must be able to SSH to HOST2 without a password, and vice versa:

# From HOST1
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"

# From HOST2
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"

3. Container names match

Failover containers must exist on the server that will start them. If HOST2 starts Emby for HOST1, the Emby container must be created (but stopped) on HOST2 with its volume mounts pointing to the mirrored data.

4. Data mirrored

Critical appdata synced to the remote server before failover is needed — not after. The daily rsync profiles keep this current:

appdata-Failover/Critical-Data    → auth stack
appdata-Failover/Important-Data   → NextCloud + Postgres
appdata-Failover/Emby             → Emby userdata
appdata-Failover/Gmer4Lfe         → server appdata

5. DDNS TTL set to 1 minute

In your DDNS provider settings. Higher TTL means users continue hitting the old IP for longer after failover. 1 minute is the minimum most providers allow — it means worst-case 1 minute of disruption.

6. Both servers running failover.sh

Both servers must be running the script simultaneously. Failover only works in one direction if only one server is running it.

Scripts

failover.sh

The main state machine. Run as a background task at array start on both servers.

# Scheduled as: At Startup of Array (Background Script)
/mnt/user/appdata/unraid_scripts/Failover/failover.sh

# Check current state without restarting the loop
/mnt/user/appdata/unraid_scripts/Failover/failover.sh --status

# Test logic without touching containers
/mnt/user/appdata/unraid_scripts/Failover/failover.sh --dry-run --log

To stop: Click Abort in the User Scripts plugin. Do NOT kill the process directly — the state file may be left inconsistent. Use Tools/failover_state_reset.sh to recover from a stuck state.

failover_test.sh

Controlled simulation of the full failover lifecycle. Validates everything works before you need it.

# Always dry run first
/mnt/user/appdata/unraid_scripts/Failover/failover_test.sh --dry-run

# Live test — run during maintenance window
/mnt/user/appdata/unraid_scripts/Failover/failover_test.sh

What it does:

  1. Verifies both servers reachable and state is NORMAL
  2. Adds iptables rule blocking all traffic to remote IP
  3. Waits FAILOVER_TEST_BLOCK_WAIT seconds for failover.sh to detect outage
  4. Verifies FAILOVER state and Tier 1 containers started
  5. Removes iptables rule — remote becomes reachable again
  6. Waits FAILOVER_TEST_HANDBACK_WAIT seconds for handback
  7. Verifies containers returned and state is NORMAL
  8. Full pass/fail report per phase

Safety trap: The iptables rule is removed via trap on ANY exit — crash, error, ctrl-c, or normal completion. Remote connectivity is always restored regardless of test outcome.

⚠️ Run during a maintenance window. Real containers start and stop during the test — users will experience a brief interruption. Schedule it for 3am or a quiet period.

Timing configuration:

FAILOVER_TEST_BLOCK_WAIT=150     # must be > FAILOVER_CHECK_INTERVAL + buffer
FAILOVER_TEST_HANDBACK_WAIT=360  # covers strikes × interval + rsync time

State File

The state file at /boot/config/failover_state.db persists across reboots — it's on /boot/ not /tmp/. This means the script remembers what state it was in before a reboot and can resume correctly.

state=NORMAL
failover_start=0
handback_strikes=0
tier2_started=false
tier3_started=false
tier4_started=false
last_reset=2026-04-14 03:00:00

If the state file gets stuck in a non-NORMAL state after testing or a failed handback, use Tools/failover_state_reset.sh to reset it manually after verifying both servers are in their correct states.

Monitoring

Monitors/weekly_health_digest.sh reads the failover state file and includes it in the weekly digest. If DIGEST_SMART_ON_FAILOVER=true and the state is not NORMAL, the digest sends a notification even in smart mode.

failover.sh --status gives an instant snapshot of current state, active tier flags, outage duration, and handback strike count.

What Gets Written Back on Handback

Data Written back Reason
Emby userdata / playstates ✅ Yes Small, important — watch history
Auth stack data ✅ Yes Authelia sessions, LLDAP data
NextCloud data ✅ Yes File changes during outage
Emby metadata ✅ Yes Any metadata scraped during outage
Media files ❌ No Already on HOST1, never moved
Downloads ❌ No Start fresh — cleaner than partial state
Arr databases ❌ No (if under 18hr) Arrs never started, nothing to sync

The minimal writeback is intentional and by design. The goal is to get HOST1 back to the state it was in before the outage plus the delta of what changed during it — not to sync everything.

Troubleshooting

Failover not triggering:

  • Is failover.sh running on HOST2? Check User Scripts plugin
  • Is HOST2's Tailscale connected and can it ping HOST1?
  • Check the state file — what state is HOST2 in?

Handback not completing:

  • Is HOST1's array fully started?
  • Is Docker responding on HOST1? docker ps should work
  • Is HOST1's rootfs below ROOTFS_WARN?
  • Check rsync writeback jobs — a stalled rsync blocks handback

DDNS not cutting over:

  • Check the DDNS container is actually running on the covering server
  • Check DNS TTL — if set high users won't see the cutover for a while
  • Check your DDNS provider — are updates being accepted?

State file stuck:

  • Use Tools/failover_state_reset.sh --status to see what's in it
  • Verify both servers manually — right containers on right server, DDNS correct
  • Run Tools/failover_state_reset.sh to reset

Split brain (both DDNS running):

  • This should not happen if the handback sequence is followed
  • Check both servers — one should have DDNS stopped
  • Manually stop the duplicate DDNS container
  • Reset the state file and restart failover.sh
Reference in New Issue View Git Blame Copy Permalink