469 lines
18 KiB
Markdown
469 lines
18 KiB
Markdown
# Failover
|
||
|
||
Mutual container failover between two unRAID servers. When one server goes down, the other automatically starts its containers. When it comes back, everything hands back cleanly with data synced and DNS cutting over at exactly the right moment.
|
||
|
||
> This system was built from scratch and refined through a year of production testing before being standardised into this ecosystem. The DDNS sequencing and handback order were the hardest parts to get right — the logic is documented here so it's never lost.
|
||
|
||
---
|
||
|
||
## The Setup
|
||
|
||
Two fully independent unRAID servers connected via Tailscale:
|
||
|
||
```
|
||
HOST1 — unRAID-Gmer4Lfe (Primary)
|
||
Hardware: Threadripper 1950X, 128GB RAM
|
||
Owns: Gmer4Lfe.com DDNS
|
||
Runs: Full service stack
|
||
|
||
HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
|
||
Hardware: Intel i5 10th gen — completely different hardware
|
||
Location: Remote — 50 miles away
|
||
Owns: Gmer4Lfe.us DDNS
|
||
Runs: Its own service stack + mirrors HOST1 critical data
|
||
```
|
||
|
||
**Hardware doesn't need to match.** Everything is accessed through `/mnt/user/` — unRAID's fused share layer. HOST1 has a Threadripper with multiple ZFS pools. HOST2 has a completely different CPU, fewer drives, different layout. The failover containers on HOST2 mount `/mnt/user/Movies` and see the same data because the share names match and the data is mirrored. The hardware underneath is irrelevant.
|
||
|
||
**What needs to match between servers:**
|
||
- Share names (`/mnt/user/Movies` = `/mnt/user/Movies`)
|
||
- Container names for shared failover services
|
||
- Docker custom network names (so NPM can reach containers by name, not IP)
|
||
|
||
**Normal operation — HOST2 is essentially passive:**
|
||
|
||
HOST1 is the source of truth. It runs everything — all arrs, all downloads, all active services. HOST2 keeps its media mirror current via nightly rsync and waits. All media is mirrored so when Emby starts on HOST2 during failover it has the exact same library, same metadata, same watch history. No separate library, no separate database, no user-visible difference.
|
||
|
||
Don't run arrs on both servers simultaneously — two instances writing to the same share causes conflicts and duplicate downloads. HOST2's arrs only start at Tier 4 (18hr+ outage) when genuine workflow continuity is needed.
|
||
|
||
Both servers run `failover.sh` as a background task continuously. Neither server knows what the other is doing — they only know what they can ping from their own network perspective.
|
||
|
||
---
|
||
|
||
## How It Works
|
||
|
||
Every `FAILOVER_CHECK_INTERVAL` seconds (default: 120s) each server makes two pings:
|
||
|
||
```
|
||
1. Ping remote server (Tailscale IP)
|
||
2. Ping internet (8.8.8.8)
|
||
```
|
||
|
||
The combination of those two results determines the current state and what action to take. That's it. No SSH signaling between servers, no shared state files, no coordination — pure autonomous decision making based on observable facts.
|
||
|
||
---
|
||
|
||
## States
|
||
|
||
### NORMAL
|
||
|
||
```
|
||
Remote: reachable Internet: reachable
|
||
```
|
||
|
||
Both servers running normally. Each server runs its own containers. DDNS on — pointing DNS at this server's IP. Silent operation.
|
||
|
||
### FAILOVER
|
||
|
||
```
|
||
Remote: unreachable Internet: reachable
|
||
```
|
||
|
||
The remote server is down but this server has internet. Start the remote server's containers locally. The remote's DDNS container is started first — DNS starts pointing at this server immediately. Failover is **additive** — your own containers keep running, remote containers are added on top.
|
||
|
||
### NO_INTERNET
|
||
|
||
```
|
||
Internet: unreachable (remote state unknown)
|
||
```
|
||
|
||
This server has lost internet. Stop own DDNS immediately — no point updating DNS records when you can't reach the outside world and it would send conflicting updates. Do not start remote containers — there's no internet to serve them on. Wait for recovery.
|
||
|
||
### DARK
|
||
|
||
```
|
||
Remote: unreachable Internet: unreachable
|
||
```
|
||
|
||
Both pings fail. Same actions as NO_INTERNET — can't determine if remote is truly down or just unreachable through the same outage affecting your internet. Conservative approach: stop DDNS, wait.
|
||
|
||
---
|
||
|
||
## DDNS — The Critical Part
|
||
|
||
**This took a year to get right. Do not change the sequencing.**
|
||
|
||
Each server owns one DDNS container. The script controls when each DDNS runs — the network state never auto-starts DDNS.
|
||
|
||
```
|
||
NORMAL:
|
||
HOST1 DDNS (Gmer4Lfe.com) → ON — always on while HOST1 has internet
|
||
HOST2 DDNS (Gmer4Lfe.us) → ON — always on while HOST2 has internet
|
||
|
||
HOST1 loses internet:
|
||
HOST1 DDNS → OFF — immediately stopped
|
||
HOST2 DDNS → stays ON (unaffected)
|
||
|
||
HOST2 detects HOST1 is down:
|
||
HOST1 DDNS (on HOST2) → ON — HOST2 starts it as Tier 1 action
|
||
DNS now points at HOST2's IP
|
||
|
||
HOST1 returns — handback:
|
||
HOST1 DDNS (on HOST2) → OFF — stopped FIRST before anything else
|
||
... rsync runs ...
|
||
HOST1 containers start on HOST1
|
||
HOST1 DDNS (on HOST1) → ON — started LAST after containers confirmed up
|
||
```
|
||
|
||
**Why DDNS never auto-starts on internet return:**
|
||
|
||
If HOST1 lost internet and its DDNS auto-started when internet returned, you'd have both HOST1 and HOST2 running the same DDNS simultaneously — pointing DNS at two different IPs at the same time. DNS TTL is 1 minute — users would get routed randomly between servers. This is called split brain and it causes exactly the kind of intermittent failures that are hard to diagnose.
|
||
|
||
The script is the sole authority over when DDNS starts. Network state coming back is not permission to start DDNS. Only the completion of the full handback sequence is.
|
||
|
||
**One domain per server, one DDNS active per domain, always:**
|
||
|
||
```
|
||
Gmer4Lfe.com → runs on HOST1 normally, moves to HOST2 during HOST1 outage
|
||
Gmer4Lfe.us → runs on HOST2 normally, moves to HOST1 during HOST2 outage
|
||
```
|
||
|
||
---
|
||
|
||
## Tiered Failover
|
||
|
||
Not every service needs to start immediately when the other server goes down. Starting the full stack on a secondary server wastes resources for short outages — most are resolved in minutes.
|
||
|
||
```
|
||
Tier 1 — Immediate (0 min)
|
||
Remote DDNS ← DNS coverage first
|
||
Emby ← media server, people are watching
|
||
NginxProxyManager ← reverse proxy, everything routes through this
|
||
Lldap, Mariadb, Redis ← auth stack, required by everything proxied
|
||
Authelia x2 ← authentication
|
||
VaultWarden ← passwords, needed immediately
|
||
Dispatcharr x3 ← Live TV, people are watching right now
|
||
ErsatzTV ← Live TV scheduling
|
||
|
||
Tier 2 — After HOST1_TIER2_DELAY minutes (default: 120min)
|
||
NextCloud + Postgres ← file access
|
||
Immich + PostgreSQL ← photos
|
||
Jellyseerr ← media requests
|
||
|
||
Tier 3 — After HOST1_TIER3_DELAY minutes (default: 360min)
|
||
Organizrv2 ← dashboard
|
||
AdGuard-Home ← DNS filtering
|
||
UptimeKuma ← monitoring
|
||
Gitea ← git server
|
||
Collabora-CODE ← document editing
|
||
|
||
Tier 4 — After HOST1_TIER4_DELAY minutes (default: 1080min / 18hr)
|
||
Full arr stack ← Sonarr, Radarr, Lidarr, Prowlarr etc.
|
||
Downloaders ← SABnzbd, Qbittorrent, LidaTube, Pinchflat
|
||
24hr+ outage = full workflow continuity
|
||
```
|
||
|
||
**Live TV is Tier 1 because people are watching.** You cannot tell a household mid-game that their live TV will be back in 2 hours.
|
||
|
||
**Arrs and downloaders are Tier 4** because they generate significant I/O and have minimal writeback on handback — if HOST1 comes back before 18 hours, the arrs never started on HOST2 and there's nothing to sync back.
|
||
|
||
---
|
||
|
||
## Handback Sequence
|
||
|
||
When HOST1 returns after being down, the handback must happen in exactly this order:
|
||
|
||
```
|
||
1. Strike confirmation
|
||
— FAILOVER_HANDBACK_STRIKES consecutive remote-up checks
|
||
— prevents handing back during a brief network blip
|
||
— 2 strikes × 120s = 4 minute confirmation window
|
||
|
||
2. Pre-flight checks
|
||
— Remote array is started
|
||
— Remote Docker daemon is responding
|
||
— Remote rootfs is not nearly full
|
||
— Abort if any check fails — retry next cycle
|
||
|
||
3. Stop remote DDNS FIRST
|
||
— DNS stops updating before anything moves
|
||
— Prevents split brain during the transition window
|
||
— This is the most critical ordering step
|
||
|
||
4. Stop remote containers
|
||
— Clean state before rsync
|
||
— No competing writes during transfer
|
||
— Containers are only down during the rsync window
|
||
— This minimises user disruption
|
||
|
||
5. Rsync writeback
|
||
— Full bandwidth available — DDNS stopped, containers stopped
|
||
— Only critical data synced back:
|
||
appdata-Failover/Critical-Data (auth stack)
|
||
appdata-Failover/Important-Data (NextCloud + Postgres)
|
||
appdata-Failover/Emby (userdata, playstates)
|
||
appdata-Failover/Gmer4Lfe (server appdata)
|
||
— Media files skipped — already on HOST1, never moved
|
||
— Downloads skipped — start fresh is cleaner
|
||
|
||
6. Start local containers
|
||
— Dependencies respected — databases before apps
|
||
— Brief pause to let databases initialise before dependents start
|
||
|
||
7. Start local DDNS LAST
|
||
— DNS only cuts back after containers are confirmed up
|
||
— Users hit HOST1 only after it's actually ready to serve them
|
||
|
||
8. Return to NORMAL
|
||
— State file reset
|
||
— Tier flags cleared
|
||
— Next cycle confirms everything is healthy
|
||
```
|
||
|
||
**Why containers stop before rsync:**
|
||
|
||
Earlier versions synced while containers were still running on the remote. This caused:
|
||
- Rsync competing with active container I/O — slower transfers
|
||
- Files changing mid-transfer — potential inconsistency
|
||
- Database writes during sync — dirty state on handback
|
||
|
||
Stopping containers first means rsync gets a clean static source at full bandwidth. The window where containers are down is the rsync duration only — typically minutes.
|
||
|
||
---
|
||
|
||
## Mutual Failover — Both Directions
|
||
|
||
The same script handles both directions. `detect_hosts()` in `common.sh` determines which server is local and which is remote at runtime, then selects the correct arrays from `Master.conf`.
|
||
|
||
```
|
||
HOST2 covers HOST1 (HOST1 goes down):
|
||
Uses: FAILOVER_HOST2_RUNS_FOR_HOST1_* arrays
|
||
Tiers configured by: HOST1_TIER*_DELAY variables
|
||
|
||
HOST1 covers HOST2 (HOST2 goes down):
|
||
Uses: FAILOVER_HOST1_RUNS_FOR_HOST2_* arrays
|
||
Tiers configured by: HOST2_TIER*_DELAY variables
|
||
```
|
||
|
||
Both servers run identical scripts. The configuration in `Master.conf` controls what each server does for the other.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
All configuration in `Master.conf` under the `── FAILOVER ──` section.
|
||
|
||
```bash
|
||
# Core timing
|
||
EXTERNAL_IP="8.8.8.8" # internet ping target
|
||
FAILOVER_CHECK_INTERVAL=120 # seconds between checks
|
||
FAILOVER_HANDBACK_STRIKES=2 # confirmations before handback
|
||
FAILOVER_STATE_FILE="/boot/config/failover_state.db"
|
||
|
||
# DDNS ownership — one per server, script controlled exclusively
|
||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com")
|
||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us")
|
||
|
||
# What HOST2 runs for HOST1 (tiered)
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_IMMEDIATE=(...)
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_2HR=(...)
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_6HR=(...)
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_18HR=(...)
|
||
|
||
# What HOST1 runs for HOST2 (tiered)
|
||
FAILOVER_HOST1_RUNS_FOR_HOST2_IMMEDIATE=(...)
|
||
# ...
|
||
|
||
# Tier delays — configurable per host, in minutes
|
||
HOST1_TIER2_DELAY=120
|
||
HOST1_TIER3_DELAY=360
|
||
HOST1_TIER4_DELAY=1080
|
||
HOST2_TIER2_DELAY=120
|
||
HOST2_TIER3_DELAY=360
|
||
HOST2_TIER4_DELAY=1080
|
||
|
||
# Writeback jobs on handback
|
||
FAILOVER_HOST1_WRITEBACK=(
|
||
"/mnt/user/appdata-Failover/Critical-Data"
|
||
"/mnt/user/appdata-Failover/Important-Data"
|
||
"/mnt/user/appdata-Failover/Emby"
|
||
"/mnt/user/appdata-Failover/Gmer4Lfe"
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## Initial Setup Requirements
|
||
|
||
Before `failover.sh` can run on both servers:
|
||
|
||
**1. Tailscale connected on both servers**
|
||
```bash
|
||
# Verify on HOST1
|
||
tailscale ip -4 unRAID-Jayred365 # should return HOST2's Tailscale IP
|
||
|
||
# Verify on HOST2
|
||
tailscale ip -4 unRAID-Gmer4Lfe # should return HOST1's Tailscale IP
|
||
```
|
||
|
||
**2. SSH keys configured**
|
||
|
||
HOST1 must be able to SSH to HOST2 without a password, and vice versa:
|
||
```bash
|
||
# From HOST1
|
||
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"
|
||
|
||
# From HOST2
|
||
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"
|
||
```
|
||
|
||
**3. Container names match**
|
||
|
||
Failover containers must exist on the server that will start them. If HOST2 starts `Emby` for HOST1, the `Emby` container must be created (but stopped) on HOST2 with its volume mounts pointing to the mirrored data.
|
||
|
||
**4. Data mirrored**
|
||
|
||
Critical appdata synced to the remote server before failover is needed — not after. The daily rsync profiles keep this current:
|
||
```
|
||
appdata-Failover/Critical-Data → auth stack
|
||
appdata-Failover/Important-Data → NextCloud + Postgres
|
||
appdata-Failover/Emby → Emby userdata
|
||
appdata-Failover/Gmer4Lfe → server appdata
|
||
```
|
||
|
||
**5. DDNS TTL set to 1 minute**
|
||
|
||
In your DDNS provider settings. Higher TTL means users continue hitting the old IP for longer after failover. 1 minute is the minimum most providers allow — it means worst-case 1 minute of disruption.
|
||
|
||
**6. Both servers running failover.sh**
|
||
|
||
Both servers must be running the script simultaneously. Failover only works in one direction if only one server is running it.
|
||
|
||
---
|
||
|
||
## Scripts
|
||
|
||
### `failover.sh`
|
||
|
||
The main state machine. Run as a background task at array start on both servers.
|
||
|
||
```bash
|
||
# Scheduled as: At Startup of Array (Background Script)
|
||
/mnt/user/appdata/unraid_scripts/Failover/failover.sh
|
||
|
||
# Check current state without restarting the loop
|
||
/mnt/user/appdata/unraid_scripts/Failover/failover.sh --status
|
||
|
||
# Test logic without touching containers
|
||
/mnt/user/appdata/unraid_scripts/Failover/failover.sh --dry-run --log
|
||
```
|
||
|
||
**To stop:** Click Abort in the User Scripts plugin. Do NOT kill the process directly — the state file may be left inconsistent. Use `Tools/failover_state_reset.sh` to recover from a stuck state.
|
||
|
||
---
|
||
|
||
### `failover_test.sh`
|
||
|
||
Controlled simulation of the full failover lifecycle. Validates everything works before you need it.
|
||
|
||
```bash
|
||
# Always dry run first
|
||
/mnt/user/appdata/unraid_scripts/Failover/failover_test.sh --dry-run
|
||
|
||
# Live test — run during maintenance window
|
||
/mnt/user/appdata/unraid_scripts/Failover/failover_test.sh
|
||
```
|
||
|
||
**What it does:**
|
||
|
||
1. Verifies both servers reachable and state is NORMAL
|
||
2. Adds iptables rule blocking all traffic to remote IP
|
||
3. Waits `FAILOVER_TEST_BLOCK_WAIT` seconds for `failover.sh` to detect outage
|
||
4. Verifies FAILOVER state and Tier 1 containers started
|
||
5. Removes iptables rule — remote becomes reachable again
|
||
6. Waits `FAILOVER_TEST_HANDBACK_WAIT` seconds for handback
|
||
7. Verifies containers returned and state is NORMAL
|
||
8. Full pass/fail report per phase
|
||
|
||
**Safety trap:** The iptables rule is removed via `trap` on ANY exit — crash, error, ctrl-c, or normal completion. Remote connectivity is always restored regardless of test outcome.
|
||
|
||
**⚠️ Run during a maintenance window.** Real containers start and stop during the test — users will experience a brief interruption. Schedule it for 3am or a quiet period.
|
||
|
||
**Timing configuration:**
|
||
```bash
|
||
FAILOVER_TEST_BLOCK_WAIT=150 # must be > FAILOVER_CHECK_INTERVAL + buffer
|
||
FAILOVER_TEST_HANDBACK_WAIT=360 # covers strikes × interval + rsync time
|
||
```
|
||
|
||
---
|
||
|
||
## State File
|
||
|
||
The state file at `/boot/config/failover_state.db` persists across reboots — it's on `/boot/` not `/tmp/`. This means the script remembers what state it was in before a reboot and can resume correctly.
|
||
|
||
```
|
||
state=NORMAL
|
||
failover_start=0
|
||
handback_strikes=0
|
||
tier2_started=false
|
||
tier3_started=false
|
||
tier4_started=false
|
||
last_reset=2026-04-14 03:00:00
|
||
```
|
||
|
||
If the state file gets stuck in a non-NORMAL state after testing or a failed handback, use `Tools/failover_state_reset.sh` to reset it manually after verifying both servers are in their correct states.
|
||
|
||
---
|
||
|
||
## Monitoring
|
||
|
||
`Monitors/weekly_health_digest.sh` reads the failover state file and includes it in the weekly digest. If `DIGEST_SMART_ON_FAILOVER=true` and the state is not NORMAL, the digest sends a notification even in smart mode.
|
||
|
||
`failover.sh --status` gives an instant snapshot of current state, active tier flags, outage duration, and handback strike count.
|
||
|
||
---
|
||
|
||
## What Gets Written Back on Handback
|
||
|
||
| Data | Written back | Reason |
|
||
|------|-------------|--------|
|
||
| Emby userdata / playstates | ✅ Yes | Small, important — watch history |
|
||
| Auth stack data | ✅ Yes | Authelia sessions, LLDAP data |
|
||
| NextCloud data | ✅ Yes | File changes during outage |
|
||
| Emby metadata | ✅ Yes | Any metadata scraped during outage |
|
||
| Media files | ❌ No | Already on HOST1, never moved |
|
||
| Downloads | ❌ No | Start fresh — cleaner than partial state |
|
||
| Arr databases | ❌ No (if under 18hr) | Arrs never started, nothing to sync |
|
||
|
||
The minimal writeback is intentional and by design. The goal is to get HOST1 back to the state it was in before the outage plus the delta of what changed during it — not to sync everything.
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
**Failover not triggering:**
|
||
- Is `failover.sh` running on HOST2? Check User Scripts plugin
|
||
- Is HOST2's Tailscale connected and can it ping HOST1?
|
||
- Check the state file — what state is HOST2 in?
|
||
|
||
**Handback not completing:**
|
||
- Is HOST1's array fully started?
|
||
- Is Docker responding on HOST1? `docker ps` should work
|
||
- Is HOST1's rootfs below `ROOTFS_WARN`?
|
||
- Check rsync writeback jobs — a stalled rsync blocks handback
|
||
|
||
**DDNS not cutting over:**
|
||
- Check the DDNS container is actually running on the covering server
|
||
- Check DNS TTL — if set high users won't see the cutover for a while
|
||
- Check your DDNS provider — are updates being accepted?
|
||
|
||
**State file stuck:**
|
||
- Use `Tools/failover_state_reset.sh --status` to see what's in it
|
||
- Verify both servers manually — right containers on right server, DDNS correct
|
||
- Run `Tools/failover_state_reset.sh` to reset
|
||
|
||
**Split brain (both DDNS running):**
|
||
- This should not happen if the handback sequence is followed
|
||
- Check both servers — one should have DDNS stopped
|
||
- Manually stop the duplicate DDNS container
|
||
- Reset the state file and restart `failover.sh` |