1165 lines
56 KiB
Markdown
1165 lines
56 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
# 🔀 FAILOVER
|
||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
**Mutual automatic failover between two fully independent unRAID servers.**
|
||
When one server goes down the other starts its containers, cuts over DNS, and keeps
|
||
users online. When it comes back everything hands back in the correct sequence — DDNS
|
||
first off, rsync, containers, DDNS last on — so users only hit the returning server
|
||
after it is actually ready to serve them.
|
||
|
||
> **Built from scratch. Refined through a year of production testing.** The DDNS
|
||
> sequencing and handback order were the hardest parts to get right. Both directions
|
||
> are exercised regularly with `failover_test.sh`. The logic is documented here in
|
||
> full so it is never lost and never has to be rediscovered.
|
||
|
||
---
|
||
|
||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Running a self-hosted media and productivity stack means being the operator. When
|
||
something goes down — power cut, ISP failure, hardware fault — it is your problem.
|
||
Cloud services have entire infrastructure teams and redundant data centers. A home
|
||
unRAID server has one power outlet, one ISP, and one point of failure.
|
||
|
||
The specific problems that led to building this:
|
||
|
||
---
|
||
|
||
### 🔴 A Single Point of Failure for an Entire Household
|
||
|
||
HOST1 runs Emby, NginxProxyManager, Authelia, NextCloud, VaultWarden, and every
|
||
other service the household uses daily. When HOST1 goes down — even briefly — all of
|
||
those services go down with it. Everyone in the household loses access simultaneously.
|
||
The one place everything lives is the one thing that can fail.
|
||
|
||
The fix: a second server with mirrored critical data and the ability to start any of
|
||
HOST1's containers. The second server covers the first automatically. From a user's
|
||
perspective, a brief interruption and then everything is back — not hours of downtime.
|
||
|
||
---
|
||
|
||
### 🔴 DNS Cutting Over Before the Server Was Ready
|
||
|
||
Early attempts at failover started the containers on the covering server and then
|
||
updated DNS. The problem: DNS propagation happens in under a minute. Users hit the
|
||
new IP before Emby had finished starting, before Authelia had loaded its sessions,
|
||
before NPM had loaded its proxy configurations. Emby reconnection attempts failed.
|
||
Users who were mid-stream got dropped and couldn't reconnect. The auth wall was up
|
||
before the service behind it was.
|
||
|
||
The fix: warm standby for the auth stack. NPM, LLDAP, and Authelia run actively on
|
||
both servers at all times — not cold-started at failover. When DNS cuts over, auth
|
||
is already running and ready. The 30-60 second dead zone after failover disappeared.
|
||
|
||
---
|
||
|
||
### 🔴 Split Brain DNS During Handback
|
||
|
||
When HOST1 returned after being down, the obvious thing to do was: start HOST1's
|
||
containers, then switch DNS back. The problem: between "start containers" and "DNS
|
||
switches" there is a window where HOST1's DDNS container and HOST2's DDNS container
|
||
are both running, both updating the same domain record with different IPs. Users
|
||
during that window get routed randomly between two servers — some to the primary that
|
||
has fresh data, some to the covering server that might be mid-cleanup. Authentication
|
||
sessions don't transfer between servers. This is split brain and it causes exactly
|
||
the intermittent failures that are hardest to diagnose.
|
||
|
||
The fix: stop DDNS on the covering server first, before anything else moves. Nothing
|
||
starts on HOST1 until HOST2's DDNS is confirmed stopped. DNS is always owned by
|
||
exactly one server. There is no window where two DDNS containers are updating the
|
||
same record.
|
||
|
||
---
|
||
|
||
### 🔴 Rsync Running Into Active Container I/O
|
||
|
||
Early versions synced data back to HOST1 while containers were still running on HOST2
|
||
— the logic being "minimise downtime, sync while active." The problem: rsync competing
|
||
with active container I/O is slower. Files changing mid-transfer can cause
|
||
inconsistency. Database writes during sync risk dirty state on the returned server.
|
||
The shorter-downtime approach paradoxically created a higher risk of problems on
|
||
handback.
|
||
|
||
The fix: stop containers before syncing. The window where containers are down is only
|
||
the rsync duration — typically a few minutes. Clean static source at full bandwidth.
|
||
Predictable, consistent handback state every time.
|
||
|
||
---
|
||
|
||
### 🔴 No Way to Validate the System Before Needing It
|
||
|
||
A failover system that has never been tested is not a failover system — it is a hope.
|
||
The only honest test is to actually pull the plug and see what happens. But doing that
|
||
on a live production server means real users experience real downtime, and if something
|
||
is misconfigured the downtime is extended while you debug it.
|
||
|
||
The fix: `failover_test.sh`. A controlled simulation that uses an iptables rule to
|
||
make the remote server appear unreachable — triggering the full failover sequence
|
||
without actually taking anything offline. The test validates every phase: detection,
|
||
container start, handback, container stop, DNS return. A safety trap removes the
|
||
iptables rule on any exit — crash, error, ctrl-c, or clean completion. The remote is
|
||
always reachable after the test regardless of what happened during it.
|
||
|
||
---
|
||
|
||
## ━━━ THE SETUP ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```
|
||
HOST1 — unRAID-Gmer4Lfe
|
||
Hardware: Threadripper 1950X, 128GB RAM, ZFS cache pools
|
||
Location: Primary site
|
||
DDNS: Gmer4Lfe.com
|
||
Role: Primary — full service stack + source of truth for Movies/Shows/Music
|
||
|
||
HOST2 — unRAID-Jayred365
|
||
Hardware: Intel i5 10th gen, 64GB RAM — completely different hardware
|
||
Location: Remote — 50 miles away
|
||
DDNS: Gmer4Lfe.us
|
||
Role: Secondary — own stack + covers HOST1 + mirrors critical data
|
||
```
|
||
|
||
**Hardware does not need to match.** Everything is accessed through `/mnt/user/` —
|
||
unRAID's fused share layer. HOST1 has a Threadripper with ZFS pools. HOST2 has a
|
||
completely different CPU, fewer drives, different layout. Failover containers on HOST2
|
||
mount `/mnt/user/Movies` and see mirrored data because the share names match. The
|
||
hardware underneath is irrelevant.
|
||
|
||
**What must match between servers:**
|
||
|
||
```
|
||
Share names /mnt/user/Movies must exist on both servers (mirrored data)
|
||
Container names "Emby" on HOST2 must be the container HOST2 starts for HOST1
|
||
Network names Docker custom networks must have matching names for NPM routing
|
||
```
|
||
|
||
---
|
||
|
||
### ── Split Source of Truth — No Conflicts ────────────────────────────────────
|
||
|
||
Both servers run arr instances simultaneously with zero conflict — because they manage
|
||
completely different shares:
|
||
|
||
```
|
||
HOST1 owns source of truth for: HOST2 owns source of truth for:
|
||
/mnt/user/Movies (Radarr) /mnt/user/Anime_Movies (his Radarr)
|
||
/mnt/user/Tv_Shows (Sonarr) /mnt/user/Anime_Shows (his Sonarr)
|
||
/mnt/user/Music (Lidarr)
|
||
|
||
Each server mirrors the other's shares:
|
||
HOST1 ← pulls Anime from HOST2
|
||
HOST2 ← pulls Movies/Shows/Music from HOST1
|
||
```
|
||
|
||
**The rule:** never run two instances of the same arr against the same share
|
||
simultaneously. Different arrs managing different shares is fine. Scheduling keeps
|
||
them clean even within the same window:
|
||
|
||
```bash
|
||
# master.conf scheduling notes
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# HOST2 arrs: midnight → noon managing and downloading anime
|
||
# HOST1 Tdarr: 12:30 → 23:00 transcoding anime from HOST2's share
|
||
#
|
||
# They operate in non-overlapping windows even on the same share.
|
||
# HOST1 is processing content that HOST2 already downloaded.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Auth Stack — Warm on Both Servers ───────────────────────────────────────
|
||
|
||
NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them
|
||
running to serve his users through his domain every day — this is not a failover-only
|
||
configuration. HOST1 is source of truth: all changes (proxy rules, user accounts,
|
||
certs, Authelia policies) mirror to HOST2 every 15 minutes via critical sync.
|
||
|
||
```
|
||
Running warm on both servers always:
|
||
NginxProxyManager ← serving both domains continuously
|
||
LLDAP ← authenticating all users continuously
|
||
Authelia ← protecting all services continuously
|
||
Certs ← mirrored, valid, already loaded
|
||
|
||
Started from stopped only at failover:
|
||
Emby ← media server (Tier 1 — immediate)
|
||
Remote DDNS ← DNS updater for the covered domain (Tier 1 — immediate)
|
||
Dispatcharr x3 ← Live TV schedulers (Tier 1 — people are watching)
|
||
...and tiered services beyond Tier 1
|
||
```
|
||
|
||
Running warm eliminates the 30-60 second dead zone where auth is coming up after DNS
|
||
has already cut over. By the time users hit the covering server, auth is already ready.
|
||
|
||
---
|
||
|
||
## ━━━ HOW IT WORKS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Both servers run `failover.sh` independently as a continuous background process. They
|
||
do not coordinate with each other. There is no SSH signaling between servers, no shared
|
||
state file, no election algorithm. Each server makes all of its decisions based entirely
|
||
on two pings from its own network perspective:
|
||
|
||
```bash
|
||
# Every FAILOVER_CHECK_INTERVAL seconds (default 120s):
|
||
ping -c 1 -W 3 $REMOTE_TAILSCALE_IP # Is the other server reachable?
|
||
ping -c 1 -W 3 $EXTERNAL_IP # Do I have internet? (default: 8.8.8.8)
|
||
```
|
||
|
||
The combination of those two answers determines the current state. That is the entire
|
||
input to the decision logic. Pure, observable facts from each server's own vantage point.
|
||
|
||
---
|
||
|
||
## ━━━ STATES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
### ✅ NORMAL
|
||
|
||
```
|
||
Remote ping: PASS Internet ping: PASS
|
||
```
|
||
|
||
Both servers running normally. Each server runs its own containers. Own DDNS ON —
|
||
pointing DNS at this server's IP. Silent operation — no output on clean cycles.
|
||
|
||
---
|
||
|
||
### 🔴 FAILOVER
|
||
|
||
```
|
||
Remote ping: FAIL Internet ping: PASS
|
||
```
|
||
|
||
The remote server is unreachable but this server has internet. The remote's containers
|
||
need to be started locally — users are hitting DNS records that point to a server that's
|
||
down. Failover is **additive** — own containers keep running, remote containers are
|
||
added on top. DDNS for the remote domain starts immediately (Tier 1) so DNS cuts over
|
||
within TTL window (1 minute).
|
||
|
||
---
|
||
|
||
### 🟡 NO_INTERNET
|
||
|
||
```
|
||
Internet ping: FAIL (remote state irrelevant)
|
||
```
|
||
|
||
This server has lost internet connectivity. Stop own DDNS immediately — updating DNS
|
||
records without internet access sends conflicting updates and could route users to an
|
||
unreachable server. Do not start remote containers — there is no internet to serve
|
||
them on. Wait for recovery.
|
||
|
||
---
|
||
|
||
### ⚫ DARK
|
||
|
||
```
|
||
Remote ping: FAIL Internet ping: FAIL
|
||
```
|
||
|
||
Both pings fail. Cannot determine if the remote is truly down or if the same outage
|
||
affecting internet connectivity is making it unreachable. Conservative approach: same
|
||
actions as NO_INTERNET — stop own DDNS, wait. Do not start remote containers.
|
||
|
||
---
|
||
|
||
## ━━━ DDNS — THE CRITICAL PART ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
> **This took a year to get right. Do not change the sequencing.**
|
||
|
||
The DDNS rules are absolute. The script is the sole authority over when any DDNS
|
||
container starts. Network state returning is not permission to start DDNS. Only the
|
||
completion of the full handback sequence grants that permission.
|
||
|
||
---
|
||
|
||
### ── The Rules ────────────────────────────────────────────────────────────────
|
||
|
||
```
|
||
ONE DOMAIN → ONE DDNS ACTIVE → AT ALL TIMES
|
||
|
||
Gmer4Lfe.com → HOST1's DDNS normally → HOST2's DDNS during HOST1 outage
|
||
Gmer4Lfe.us → HOST2's DDNS normally → HOST1's DDNS during HOST2 outage
|
||
|
||
Own DDNS: ON when this server has internet. OFF when internet is lost.
|
||
Remote DDNS: ON as Tier 1 failover action. OFF as first handback action.
|
||
Auto-start: NEVER — DDNS never starts automatically on internet return.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Why Auto-Start Is Forbidden ─────────────────────────────────────────────
|
||
|
||
If HOST1 lost internet and its DDNS auto-started when internet returned, there is a
|
||
window where both HOST1 and HOST2 are running the same DDNS — each updating the same
|
||
domain record with different IPs. DNS TTL is 1 minute. During that window users get
|
||
routed randomly between two servers. Authentication sessions do not transfer between
|
||
servers. This is split brain.
|
||
|
||
Split brain from double-DDNS is exactly the class of failure that produces the most
|
||
confusing symptoms — intermittent auth failures, users sometimes getting through and
|
||
sometimes not, no clear error state anywhere. The fix is absolute: the script controls
|
||
when DDNS starts. Network state coming back is not permission.
|
||
|
||
---
|
||
|
||
### ── Full DDNS Lifecycle — HOST1 Outage ─────────────────────────────────────
|
||
|
||
```
|
||
Normal operation:
|
||
HOST1 DDNS (Gmer4Lfe.com) → ON always on while HOST1 has internet
|
||
HOST2 DDNS (Gmer4Lfe.us) → ON always on while HOST2 has internet
|
||
|
||
HOST1 loses internet:
|
||
HOST1 DDNS → OFF stopped immediately — no internet, no updates
|
||
HOST2 DDNS → ON unaffected, HOST2 still has internet
|
||
|
||
HOST2 detects HOST1 is down (FAILOVER state):
|
||
HOST1 DDNS (on HOST2) → ON Tier 1 action — DNS now points at HOST2's IP
|
||
Both HOST2 DDNS containers → ON own domain + covering HOST1's domain
|
||
|
||
HOST1 returns (handback sequence):
|
||
HOST1 DDNS (on HOST2) → OFF FIRST action before anything else
|
||
[rsync writeback runs]
|
||
[HOST1 containers start]
|
||
HOST1 DDNS (on HOST1) → ON LAST action — only after containers confirmed up
|
||
|
||
Normal operation restored:
|
||
HOST1 DDNS (Gmer4Lfe.com) → ON back on HOST1
|
||
HOST2 DDNS (Gmer4Lfe.us) → ON back to normal
|
||
```
|
||
|
||
DDNS stops on HOST2 before rsync starts. DDNS starts on HOST1 after containers are
|
||
confirmed running. There is a brief window where neither DDNS is updating the record —
|
||
this is intentional. DNS TTL caches the last value. During the rsync + container start
|
||
window, cached DNS still routes users to HOST2 where the containers are still running.
|
||
By the time the cache expires, HOST1's DDNS has started and the record points at HOST1.
|
||
|
||
---
|
||
|
||
## ━━━ TIERED FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Not every service needs to start immediately. Starting the full stack on a secondary
|
||
server wastes resources for short outages — most power blips and brief ISP issues
|
||
resolve in minutes. The tier system starts only what is needed for the actual outage
|
||
duration.
|
||
|
||
---
|
||
|
||
### ── Tier 1 — Immediate (0 minutes) ─────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# These start the moment FAILOVER state is entered.
|
||
# Vital services + services people are actively using right now.
|
||
#
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1=(
|
||
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
|
||
"Emby" # media server — people are watching
|
||
"NginxProxyManager" # reverse proxy — all external access routes through this
|
||
"Lldap-Gmer4Lfe" # user directory — already warm, verify and keep
|
||
"Mariadb-Authelia" # auth database — already warm, verify and keep
|
||
"Redis-Authelia" # auth session cache — already warm, verify and keep
|
||
"Authelia" # SSO — already warm, serving users already
|
||
"VaultWarden" # passwords — needed immediately, people lock themselves out
|
||
"Dispatcharr" # Live TV scheduler — people are watching right now
|
||
"Dispatcharr-Basic" # secondary scheduler
|
||
"Dispatcharr-Iptv-Users" # tertiary scheduler
|
||
"ErsatzTV-Emby" # channel schedule builder
|
||
)
|
||
#
|
||
# Live TV is Tier 1 because you cannot tell a household mid-game that their
|
||
# live TV will be back in 2 hours. The disruption is unacceptable.
|
||
# Passwords are Tier 1 because people lock themselves out of everything else
|
||
# if VaultWarden is unavailable — it becomes the blocking failure.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Tier 2 — After HOST*_TIER2_DELAY (default: 4 hours) ────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Shared productivity services. Not immediately needed for media access but
|
||
# actively used throughout the day. 4 hours is long enough to resolve most
|
||
# ISP and power issues without starting services unnecessarily.
|
||
#
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER2=(
|
||
"NextCloud" # file sync and sharing
|
||
"Postgres-NextCloud" # NextCloud's database — must start before NextCloud
|
||
"Immich-Gmer4Lfe" # photo library
|
||
"PostgreSQL-Immich" # Immich's database — same dependency
|
||
"Jellyseerr" # media request management
|
||
)
|
||
HOST1_TIER2_DELAY=240 # minutes — 4 hours
|
||
```
|
||
|
||
---
|
||
|
||
### ── Tier 3 — After HOST*_TIER3_DELAY (default: 12 hours) ──────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Secondary services — useful but not critical for daily operation.
|
||
# 12 hours covers most genuine outage scenarios while avoiding unnecessary
|
||
# resource usage on HOST2 for short-to-medium disruptions.
|
||
#
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER3=(
|
||
"Organizrv2-Gmer4Lfe" # dashboard
|
||
"AdGuard-Home" # DNS filtering
|
||
"UptimeKuma" # uptime monitoring
|
||
"Gitea" # git server
|
||
"Collabora-CODE" # document editing for NextCloud
|
||
)
|
||
HOST1_TIER3_DELAY=720 # minutes — 12 hours
|
||
```
|
||
|
||
---
|
||
|
||
### ── Tier 4 — After HOST*_TIER4_DELAY (default: 24 hours) ──────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Full workflow continuity — arrs and downloaders.
|
||
# These generate significant I/O and have writeback requirements on handback.
|
||
# 24 hours is the threshold where it becomes worthwhile to start them.
|
||
#
|
||
# Arrs_Stack only starts if HOST1 has genuinely been down for 24+ hours.
|
||
# Below this threshold: arrs never started on HOST2 → nothing to sync back.
|
||
# Above this threshold: meaningful downloads accumulated → full Tier 4 writeback.
|
||
#
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER4=(
|
||
"Sonarr-Gmer4Lfe" # TV show management
|
||
"Radarr-Gmer4Lfe" # movie management
|
||
"Lidarr-Gmer4Lfe" # music management
|
||
"Prowlarr-Gmer4Lfe" # indexer management
|
||
"SABnzbd-Gmer4Lfe" # usenet downloader
|
||
"qBittorrent-Gmer4Lfe" # torrent downloader
|
||
"LidaTube" # YouTube music
|
||
"Pinchflat" # YouTube video
|
||
)
|
||
HOST1_TIER4_DELAY=1440 # minutes — 24 hours
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ HANDBACK SEQUENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
When the remote server returns after a FAILOVER event, handback must happen in exactly
|
||
this order. Every step has a reason. Do not reorder.
|
||
|
||
---
|
||
|
||
### ── Step 1 — Strike Confirmation ────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Network blips happen. A server can be reachable for one ping cycle and then
|
||
# drop again. If handback triggered on the first successful ping, a brief
|
||
# network recovery during an ongoing outage would cause a failed handback —
|
||
# containers stop on the covering server, rsync starts, remote goes down again
|
||
# mid-transfer. That is worse than not handing back at all.
|
||
#
|
||
# Strike confirmation requires N consecutive successful remote pings before
|
||
# handback begins. With FAILOVER_CHECK_INTERVAL=120s and HANDBACK_STRIKES=2:
|
||
# → remote must be up continuously for 4 minutes before handback starts
|
||
# → eliminates false triggers from brief network recovery
|
||
#
|
||
FAILOVER_HANDBACK_STRIKES=2 # consecutive remote-up checks required
|
||
FAILOVER_CHECK_INTERVAL=120 # seconds between checks
|
||
#
|
||
# 2 strikes × 120s = 4 minute confirmation window before handback begins
|
||
```
|
||
|
||
---
|
||
|
||
### ── Step 2 — Pre-flight Checks ──────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Before starting the handback sequence, verify the remote is actually ready.
|
||
# Reachable on Tailscale does not mean the array is started and Docker is
|
||
# running — the server might still be booting.
|
||
#
|
||
# Pre-flight checks (any failure → abort this cycle, retry next):
|
||
# ✓ Version parity — both servers on compatible unRAID versions
|
||
# ✓ Remote array — /mnt/user is mounted on remote
|
||
# ✓ Remote Docker — daemon is responding within DOCKER_TIMEOUT
|
||
# ✓ Remote rootfs — not nearly full (headroom for rsync temp files)
|
||
#
|
||
# If any pre-flight fails: abort handback, reset strike counter, retry.
|
||
# Better to wait another cycle than to start a handback that will fail midway.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Step 3 — Stop Remote DDNS FIRST ─────────────────────────────────────────
|
||
|
||
```
|
||
This is the most critical ordering step in the entire sequence.
|
||
|
||
Remote DDNS stops before any containers stop, before rsync starts,
|
||
before HOST1 containers start, before HOST1 DDNS starts.
|
||
|
||
During the gap between remote DDNS stopping and HOST1 DDNS starting:
|
||
DNS TTL caches the last value → users still routed to HOST2
|
||
HOST2 containers still running → users are still served
|
||
No split brain window exists because one DDNS stopped before
|
||
the other started. The transition is clean.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Step 4 — Stop Remote Containers ────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Containers stop on the covering server before rsync begins.
|
||
# Clean static source. No competing I/O. No database writes during transfer.
|
||
# Full bandwidth available to rsync.
|
||
#
|
||
# The user impact window (containers down) is the rsync duration only — typically
|
||
# minutes for critical data. Media files are never synced back — they were never
|
||
# moved. Downloads are never synced back — start fresh is cleaner.
|
||
#
|
||
# Containers stop in the correct dependency order:
|
||
# Dependents before dependencies (Authelia before MariaDB)
|
||
# Same WATCHDOG_DEPENDENCIES ordering used by docker_watchdog.sh
|
||
```
|
||
|
||
---
|
||
|
||
### ── Step 5 — Tiered Rsync Writeback ─────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# What gets written back depends on how long the outage lasted.
|
||
# Short outages skip writeback entirely — the primary's last-known-good state
|
||
# is more reliable than a few minutes of activity on the covering server.
|
||
#
|
||
# Under 60min → skip all writeback (primary state is cleaner)
|
||
# Under 4hr → skip Tier 2+ writeback
|
||
# Under 12hr → skip Tier 3+ writeback
|
||
# Under 24hr → skip Tier 4 writeback (arrs never started, nothing to sync)
|
||
# Over 24hr → full tiered writeback (meaningful delta accumulated)
|
||
#
|
||
# Real-world outage profile:
|
||
# Power blip 2-10min → no writeback, clean restart (most common)
|
||
# ISP issue 10-60min → no writeback, clean restart
|
||
# Real outage 3-6hr → Tier 1+2 writeback, skip 3+4
|
||
# Extended 24hr+ → full tiered writeback
|
||
|
||
FAILOVER_HOST1_WRITEBACK_TIER1=(
|
||
"/mnt/user/appdata-Failover/Critical-Data" # auth stack — Authelia + NPM + certs
|
||
"/mnt/user/Media_Server/Emby" # Emby userdata — watch history, playstates
|
||
)
|
||
|
||
FAILOVER_HOST1_WRITEBACK_TIER2=(
|
||
"/mnt/user/appdata-Failover/Important-Data" # NextCloud + Postgres + Immich
|
||
)
|
||
|
||
FAILOVER_HOST1_WRITEBACK_TIER3=(
|
||
"/mnt/user/appdata-Failover/Gmer4Lfe" # secondary appdata accumulated changes
|
||
)
|
||
|
||
# Tier 4 writeback uses HOST2_DAILY_SYNC_SHARES automatically — same list
|
||
# daily_sync_maintenance.sh uses, in the opposite direction.
|
||
# HOST2 was running HOST1's arrs against mirrored media — that content needs
|
||
# to sync back before HOST1's own arrs restart.
|
||
# No duplicate list needed — uses the same config, reversed direction.
|
||
|
||
# What never gets written back:
|
||
# Own media shares (Movies, Tv_Shows, Music) — already on HOST1, never moved
|
||
# Downloads — start fresh is cleaner than partial download state
|
||
```
|
||
|
||
---
|
||
|
||
### ── Step 6 — Start Remote Containers ───────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Containers start on HOST1 in dependency-safe order.
|
||
# Same WATCHDOG_DEPENDENCIES ordering — databases before applications.
|
||
# CONTAINER_DELAY seconds between dependency start and dependent start.
|
||
# Each container verified running after a settle period before the next starts.
|
||
# A container that starts and immediately crashes is caught and reported.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Step 7 — Start Remote DDNS LAST ─────────────────────────────────────────
|
||
|
||
```
|
||
DNS cuts back to HOST1 ONLY after all containers are confirmed running.
|
||
|
||
Users hit HOST1 only after HOST1 is actually ready to serve them.
|
||
|
||
The handback sequence completes. State file resets to NORMAL.
|
||
Tier flags cleared. Next cycle confirms everything is healthy.
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ MUTUAL FAILOVER — BOTH DIRECTIONS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
The same `failover.sh` script handles both directions without any code changes.
|
||
`detect_hosts()` in `common.sh` determines which server is local and which is remote
|
||
at runtime, then selects the correct container arrays and tier delays from `master.conf`.
|
||
|
||
```bash
|
||
# master.conf — symmetric configuration for both directions
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
# HOST2 covers HOST1 (HOST1 goes down):
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1=(...) # what HOST2 starts for HOST1
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER2=(...)
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER3=(...)
|
||
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER4=(...)
|
||
HOST1_TIER2_DELAY=240 # delays for HOST1 outage — from HOST1's perspective
|
||
HOST1_TIER3_DELAY=720
|
||
HOST1_TIER4_DELAY=1440
|
||
|
||
# HOST1 covers HOST2 (HOST2 goes down):
|
||
FAILOVER_HOST1_RUNS_FOR_HOST2_TIER1=(...) # what HOST1 starts for HOST2
|
||
FAILOVER_HOST1_RUNS_FOR_HOST2_TIER2=(...)
|
||
HOST2_TIER2_DELAY=240 # delays for HOST2 outage — from HOST2's perspective
|
||
HOST2_TIER3_DELAY=720
|
||
HOST2_TIER4_DELAY=1440
|
||
|
||
# Writeback — what each server syncs back on handback
|
||
FAILOVER_HOST1_WRITEBACK_TIER1=(...) # what HOST2 syncs back to HOST1
|
||
FAILOVER_HOST2_WRITEBACK_TIER1=(...) # what HOST1 syncs back to HOST2
|
||
```
|
||
|
||
Both servers run identical scripts with identical configuration. The configuration
|
||
controls what each server does. The script reads `MY_ID` from `detect_hosts()` and
|
||
selects the correct arrays. No hostname comparisons anywhere in the script.
|
||
|
||
---
|
||
|
||
## ━━━ INDEPENDENCE — ALWAYS ONE RSYNC STOP AWAY ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
HOST2 is designed to be fully independent if needed. If HOST2 ever wants to fully
|
||
separate from HOST1: stop HOST1 pushing data. Any changes HOST2 makes to his own
|
||
data stick permanently. His server becomes fully independent immediately — no script
|
||
changes, no migration, no data movement required.
|
||
|
||
The ecosystem supports this by design. Everything HOST2 runs is self-contained.
|
||
The failover and rsync scripts are configuration-driven — stopping the rsync job
|
||
is the entire separation process.
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔀 failover.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
The main state machine. Runs continuously as a background process on both servers,
|
||
started at array start by `array_start.sh`. Makes all decisions autonomously from
|
||
two pings. Never requires human intervention during normal failover and handback.
|
||
|
||
```bash
|
||
# Started automatically at array start via array_start.sh
|
||
# Runs continuously until array stops (SIGTERM → clean shutdown)
|
||
# Interval: FAILOVER_CHECK_INTERVAL=120 (2 minutes)
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Normal operation — started by array_start.sh, runs continuously.
|
||
# You do not need to run this manually under normal circumstances.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover.sh
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Status — show current state at a glance:
|
||
# • Current state (NORMAL / FAILOVER / NO_INTERNET / DARK)
|
||
# • Which tier containers are active
|
||
# • Outage duration if in FAILOVER
|
||
# • Handback strike count if remote has returned
|
||
# • DDNS containers currently running on this server
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover.sh --status
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Dry run — walk through one full cycle without starting or stopping anything.
|
||
# Shows what the script would do based on current network state.
|
||
# Useful for verifying configuration before relying on it.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover.sh --dry-run
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Verbose — full detail on every decision made in each cycle.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover.sh --log
|
||
```
|
||
|
||
> **To stop:** Click Abort in the User Scripts plugin. Do NOT kill the process directly
|
||
> — the state file may be left inconsistent. If the state gets stuck, use
|
||
> `failover_state_reset.sh` to recover.
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🧪 failover_test.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Controlled simulation of the full failover lifecycle. Validates the entire sequence
|
||
without waiting for a real outage. Contains no failover logic itself — exercises
|
||
the real `failover.sh` through an iptables block. Any change to `failover.sh`
|
||
is automatically reflected in the test.
|
||
|
||
> **⚠️ This starts and stops real containers on both servers. Users will experience
|
||
> a brief service interruption. Run during a maintenance window or quiet period.
|
||
> Use `--dry-run` first to walk through all phases without making any changes.**
|
||
|
||
---
|
||
|
||
### ── Test Phases ─────────────────────────────────────────────────────────────
|
||
|
||
```
|
||
Phase 1 — Pre-flight
|
||
Verify both servers reachable via Tailscale
|
||
Verify Docker daemons responding on both servers
|
||
Verify version parity between servers
|
||
Verify failover.sh exists and is running on remote
|
||
Verify current state is NORMAL on both servers
|
||
→ Any failure: abort test before touching anything
|
||
|
||
Phase 2 — Block Remote
|
||
Add iptables rule dropping all traffic to remote Tailscale IP
|
||
From this server's perspective: remote is now unreachable
|
||
failover.sh on this server will see the remote as down on next check
|
||
|
||
Phase 3 — Failover Detection
|
||
Wait FAILOVER_TEST_BLOCK_WAIT seconds for failover.sh to detect outage
|
||
Verify this server entered FAILOVER state
|
||
Verify Tier 1 containers started on this server
|
||
|
||
Phase 4 — Container Verification
|
||
Check each Tier 1 container is actually running
|
||
Report any that failed to start
|
||
|
||
Phase 5 — Restore Remote
|
||
Remove iptables rule — remote becomes reachable again
|
||
(Safety trap also removes rule on any exit — crash, ctrl-c, error)
|
||
|
||
Phase 6 — Handback Wait
|
||
Wait FAILOVER_TEST_HANDBACK_WAIT seconds for failover.sh to:
|
||
Confirm remote up (strike confirmation window)
|
||
Complete pre-flight checks
|
||
Stop remote DDNS
|
||
Stop remote containers
|
||
Run rsync writeback (if applicable)
|
||
Start local containers
|
||
Start local DDNS
|
||
Return to NORMAL
|
||
|
||
Phase 7 — Handback Verification
|
||
Verify state returned to NORMAL
|
||
Verify Tier 1 containers stopped on this server (handed back)
|
||
Verify DDNS is correct on both servers
|
||
|
||
Phase 8 — Report
|
||
Full pass/fail per phase with timing
|
||
Clear indication of what failed if any phase did not pass
|
||
```
|
||
|
||
---
|
||
|
||
### ── Safeguards ──────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# IPTABLES SAFETY TRAP — the most important safeguard.
|
||
# The iptables rule that blocks remote connectivity is added in Phase 2.
|
||
# It is removed via a trap on ANY exit condition:
|
||
# Normal completion → rule removed
|
||
# Script crashes → rule removed
|
||
# Error in any phase → rule removed
|
||
# ctrl-c → rule removed
|
||
#
|
||
# Remote connectivity is always restored regardless of test outcome.
|
||
# You cannot accidentally leave the remote permanently blocked.
|
||
#
|
||
# FAILOVER_ENABLED gate — aborts if FAILOVER_ENABLED=false in master.conf.
|
||
# Running a failover test when failover is disabled would be confusing and
|
||
# potentially destructive. The gate prevents this.
|
||
#
|
||
# Version parity check — pre-flight verifies both servers are on compatible
|
||
# unRAID versions before any iptables rules are added.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Timing Configuration ────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# These must be set correctly for the test to pass.
|
||
# Too short: test fails because failover.sh hasn't had time to complete its work.
|
||
# Too long: test runs unnecessarily long.
|
||
#
|
||
FAILOVER_TEST_BLOCK_WAIT=150 # must be > FAILOVER_CHECK_INTERVAL + buffer
|
||
# failover.sh checks every 120s — 150s gives
|
||
# one full cycle plus 30s margin
|
||
|
||
FAILOVER_TEST_HANDBACK_WAIT=360 # must cover: strike confirmation window
|
||
# + pre-flight time
|
||
# + rsync duration (critical data only)
|
||
# + container start time
|
||
# 2 strikes × 120s + ~2min rsync + ~1min start
|
||
# = ~6 minutes = 360 seconds
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Dry run — ALWAYS run this first.
|
||
# Walks through all 8 phases with full output but no iptables changes and
|
||
# no container starts/stops. Verifies configuration and timing before
|
||
# committing to a live test.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover_test.sh --dry-run
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Live test — run during maintenance window.
|
||
# Full sequence with real iptables rules and real container lifecycle.
|
||
# Users will experience a brief service interruption — schedule accordingly.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover_test.sh
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Status — show current failover state and test configuration without running.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover_test.sh --status
|
||
|
||
# Verbose — full detail on every check in every phase
|
||
failover_test.sh --log
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔧 failover_state_reset.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Manual recovery tool. Resets the state file to NORMAL and clears all tier flags.
|
||
Use when the state file is stuck in a non-NORMAL state after testing, a failed
|
||
handback, or killing `failover.sh` mid-cycle.
|
||
|
||
> **Does NOT start or stop containers — state file only.** After reset, `failover.sh`
|
||
> will resume from NORMAL on its next cycle. Verify that containers are actually in
|
||
> the right state before resetting — the state file should reflect reality.
|
||
|
||
---
|
||
|
||
### ── When to Use This ────────────────────────────────────────────────────────
|
||
|
||
```
|
||
After failover_test.sh didn't complete cleanly
|
||
→ state file left in FAILOVER but containers are actually back to normal
|
||
|
||
After a failed handback
|
||
→ state shows FAILOVER but remote is back up and containers are split
|
||
|
||
After killing failover.sh directly (not via User Scripts Abort)
|
||
→ state is unknown, cycle was interrupted mid-operation
|
||
|
||
After a dev/debug session
|
||
→ state left in a non-NORMAL state from testing
|
||
```
|
||
|
||
---
|
||
|
||
### ── Safety Checks ───────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# VERIFY BEFORE RESETTING:
|
||
# ✓ Right containers running on the right server
|
||
# ✓ DDNS pointing at the correct server (check with nslookup Gmer4Lfe.com)
|
||
# ✓ No actual failover in progress (remote is genuinely up)
|
||
# ✓ Both servers can see each other (tailscale ping)
|
||
#
|
||
# Resetting during an actual failover causes failover.sh to think everything
|
||
# is normal and stop covering the remote — services go offline until the next
|
||
# detection cycle catches it again.
|
||
#
|
||
# The script warns if failover.sh is currently running when you attempt reset.
|
||
# It does not block you — but heed the warning. If failover.sh is running and
|
||
# you reset mid-cycle, the next cycle will start from a clean NORMAL state
|
||
# regardless of what was in progress.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Status — show current state file contents without changing anything.
|
||
# Also shows whether failover.sh is currently running.
|
||
# Use this first to understand the situation before resetting.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover_state_reset.sh --status
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Interactive reset — prompts for "YES" before proceeding.
|
||
# Default mode — use when running manually at a terminal.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover_state_reset.sh
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Dry run — show current state and what would be written, without writing.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover_state_reset.sh --dry-run
|
||
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Force — non-interactive, no confirmation prompt.
|
||
# For use in scripts or when running without a terminal.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
failover_state_reset.sh --force
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ STATE FILE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```bash
|
||
# /boot/config/failover_state.db
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Lives on /boot/ — survives reboots intentionally.
|
||
# If a server was in FAILOVER state when it rebooted, it should resume
|
||
# FAILOVER on restart — not assume everything is NORMAL again.
|
||
#
|
||
state=NORMAL # current state machine state
|
||
failover_start=0 # epoch timestamp when FAILOVER began (0 = not in FAILOVER)
|
||
handback_strikes=0 # consecutive remote-up checks accumulated toward handback
|
||
tier2_started=false # whether Tier 2 containers have been started this event
|
||
tier3_started=false # whether Tier 3 containers have been started
|
||
tier4_started=false # whether Tier 4 containers have been started
|
||
last_reset=2026-04-14 03:00:00
|
||
reset_by=HOST1 # which server last reset the state (set by failover_state_reset.sh)
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ INITIAL SETUP REQUIREMENTS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Before `failover.sh` can work on both servers:
|
||
|
||
---
|
||
|
||
### 1. Tailscale Connected on Both Servers
|
||
|
||
```bash
|
||
# Verify from HOST1 — should return HOST2's Tailscale IP
|
||
tailscale ip -4 unRAID-Jayred365
|
||
|
||
# Verify from HOST2 — should return HOST1's Tailscale IP
|
||
tailscale ip -4 unRAID-Gmer4Lfe
|
||
|
||
# Test reachability from HOST1
|
||
tailscale ping unRAID-Jayred365
|
||
```
|
||
|
||
---
|
||
|
||
### 2. SSH Keys Configured — No Password Prompt
|
||
|
||
```bash
|
||
# From HOST1 — should print HOST2's hostname without asking for password
|
||
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"
|
||
|
||
# From HOST2 — should print HOST1's hostname without asking for password
|
||
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"
|
||
```
|
||
|
||
---
|
||
|
||
### 3. Container Names Match
|
||
|
||
Failover containers must be created (but stopped) on the server that will run them.
|
||
If HOST2 will start `Emby` for HOST1, the `Emby` container must exist on HOST2 with
|
||
its volume mounts pointing at the mirrored share paths.
|
||
|
||
```bash
|
||
# Verify the container exists on HOST2 (stopped is expected)
|
||
ssh root@[HOST2-ip] "docker inspect Emby --format '{{.State.Status}}'"
|
||
# Expected: created or exited — not "no such container"
|
||
```
|
||
|
||
---
|
||
|
||
### 4. Critical Data Mirrored Before Failover is Needed
|
||
|
||
```bash
|
||
# These shares must exist on HOST2 with current data from HOST1:
|
||
/mnt/user/appdata-Failover/Critical-Data # auth stack
|
||
/mnt/user/appdata-Failover/Important-Data # NextCloud + Postgres
|
||
/mnt/user/Media_Server/Emby # Emby userdata
|
||
/mnt/user/appdata-Failover/Gmer4Lfe # server appdata
|
||
|
||
# Verify data is current — check modification times
|
||
ssh root@[HOST2-ip] "ls -la /mnt/user/appdata-Failover/Critical-Data/"
|
||
```
|
||
|
||
---
|
||
|
||
### 5. DDNS TTL Set to 1 Minute
|
||
|
||
In your DDNS provider settings. Higher TTL means users continue hitting the old IP
|
||
for longer after failover. 1 minute is the minimum most providers allow. At 5 minute
|
||
TTL, users can be hitting a downed server for up to 5 minutes before DNS switches.
|
||
|
||
---
|
||
|
||
### 6. Both Servers Running failover.sh
|
||
|
||
Failover only works in one direction if only one server is running the script.
|
||
For mutual coverage, both servers must be running it continuously.
|
||
|
||
```bash
|
||
# Verify failover.sh is running on both servers
|
||
pgrep -f "failover.sh"
|
||
|
||
# Check the state file directly
|
||
cat /boot/config/failover_state.db
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ MONITORING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
**Sunday coffee report** (`sunday_morning_coffee_report.sh`) includes a Failover section
|
||
showing current state, outage duration if not NORMAL, Tailscale reachability, and
|
||
whether the failover.sh process is running.
|
||
|
||
**Weekly health digest** (`weekly_health_digest.sh`) reads the state file. If
|
||
`DIGEST_SMART_ON_FAILOVER=true` and state is not NORMAL, it sends a notification
|
||
even in smart mode — a non-NORMAL state at digest time needs attention.
|
||
|
||
**Direct status check:**
|
||
|
||
```bash
|
||
failover.sh --status # full state snapshot from current server
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
---
|
||
|
||
### 🔴 Failover Not Triggering
|
||
|
||
```
|
||
Is failover.sh actually running on the covering server?
|
||
→ User Scripts plugin → check status of the failover script
|
||
|
||
Is Tailscale connected on the covering server?
|
||
→ tailscale status (should show the remote peer)
|
||
|
||
Can the covering server reach the remote's Tailscale IP?
|
||
→ ping [remote-tailscale-ip]
|
||
|
||
What state is the covering server in?
|
||
→ failover.sh --status (should show FAILOVER if remote is down)
|
||
→ cat /boot/config/failover_state.db
|
||
```
|
||
|
||
---
|
||
|
||
### 🔴 Handback Not Completing
|
||
|
||
```
|
||
Is HOST1's array fully started?
|
||
→ ls /mnt/user (should show share directories)
|
||
|
||
Is Docker responding on HOST1?
|
||
→ docker ps (should return a list, not hang)
|
||
|
||
Is HOST1's rootfs below the warn threshold?
|
||
→ df / (rootfs nearly full blocks handback pre-flight)
|
||
|
||
Is rsync running and stuck?
|
||
→ pgrep rsync (a stalled rsync blocks the handback sequence)
|
||
→ rsync_stop.sh to clear it, then let failover.sh retry
|
||
```
|
||
|
||
---
|
||
|
||
### 🔴 DDNS Not Cutting Over
|
||
|
||
```
|
||
Is the DDNS container actually running on the covering server?
|
||
→ docker ps | grep DDNS
|
||
|
||
What TTL is your DNS record set to?
|
||
→ nslookup Gmer4Lfe.com 8.8.8.8 (check TTL in response)
|
||
→ High TTL means slow propagation
|
||
|
||
Is your DDNS provider accepting updates?
|
||
→ Check DDNS container logs: docker logs [ddns-container] --tail 50
|
||
```
|
||
|
||
---
|
||
|
||
### 🔴 Split Brain (Both DDNS Running)
|
||
|
||
```
|
||
This should not happen if the handback sequence completed correctly.
|
||
If it has happened:
|
||
|
||
1. Check both servers — which DDNS containers are running where
|
||
HOST1: docker ps | grep DDNS
|
||
HOST2: docker ps | grep DDNS
|
||
|
||
2. Manually stop the duplicate
|
||
docker stop [duplicate-ddns-container]
|
||
|
||
3. Reset the state file on the server in a bad state
|
||
failover_state_reset.sh --status (understand the situation)
|
||
failover_state_reset.sh --force (after verifying it is safe)
|
||
|
||
4. Restart failover.sh (it will have been stopped by the state reset)
|
||
Start via User Scripts plugin
|
||
```
|
||
|
||
---
|
||
|
||
### 🔴 State File Stuck in FAILOVER After Testing
|
||
|
||
```
|
||
Normal — this is what failover_state_reset.sh is for.
|
||
|
||
1. Verify everything is actually back to normal:
|
||
- Right containers on right server
|
||
- DDNS correct (nslookup Gmer4Lfe.com)
|
||
- Both servers visible on Tailscale
|
||
|
||
2. Reset the state file:
|
||
failover_state_reset.sh --status (confirm what you're resetting)
|
||
failover_state_reset.sh (interactive — prompts YES)
|
||
|
||
failover.sh will resume from NORMAL on its next cycle.
|
||
``` |