# ━━━━━ FALLBACK ━━━━━

Mutual automatic fallback between two independent unRAID servers. When one goes down the
other starts its containers, cuts over DNS, and keeps users online. When it comes back
everything hands back in the correct sequence — covering DDNS stops, containers stop, rsync
writeback runs, containers start on the primary, primary DDNS starts last — so users hit the
returning server only after it's actually ready.

> **Built from scratch. Refined through a year of production testing.** The DDNS sequencing
> and handback order were the hardest parts to get right. Both directions are exercised
> regularly with `fallback_test.sh`.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━

Running a self-hosted stack means being the operator. These are the specific problems
that drove this build:

**A Single Point of Failure for an Entire Household**  
HOST1 runs Emby, NPM, Authelia, NextCloud, VaultWarden, and every service the household
uses daily. When HOST1 goes down — even briefly — all of those services go down with it.
The fix: a second server with mirrored critical data that covers the first automatically.
From a user's perspective, a brief interruption and then everything is back.

**DNS Cutting Over Before the Server Was Ready**  
Early attempts started containers on the covering server then updated DNS. Problem: DNS
propagated in under a minute. Users hit the new IP before Emby had finished starting,
before Authelia had loaded its sessions, before NPM had loaded its proxy configurations.
The fix: warm standby for the auth stack. NPM, LLDAP, and Authelia run actively on both
servers at all times. When DNS cuts over, auth is already running and ready.

**Split Brain DNS During Handback**  
When HOST1 returned, the obvious sequence was: start HOST1 containers, then switch DNS
back. Problem: between "start containers" and "DNS switches" both servers' DDNS containers
were running, both updating the same domain with different IPs. Users got routed randomly
between servers — intermittent auth failures, no clear error state anywhere.  
The fix: stop DDNS on the covering server first, before anything else moves. There is
never a window where two DDNS containers update the same record.

**Rsync Running Into Active Container I/O**  
Syncing data back while containers were still running — to minimise downtime — produced
slower transfers, potential file inconsistency, and database dirty state risk.  
The fix: stop containers before syncing. The outage window is only the rsync duration —
typically minutes. Clean static source at full bandwidth, predictable state every time.

**No Way to Validate the System Before Needing It**  
A fallback system that has never been tested is not a fallback system — it is a hope.  
The fix: `fallback_test.sh` — a controlled simulation using an iptables DROP rule to make
the remote appear unreachable, triggering the full sequence without taking anything offline.
A safety trap removes the rule on any exit — crash, error, ctrl-c, or clean completion.

---

## ━━━ THE TWO-SERVER SETUP ━━━

```
HOST1 — unRAID-Gmer4Lfe
  Hardware:   Threadripper 1950X, 128GB RAM, ZFS cache pools
  Location:   Primary site
  DDNS:       Gmer4Lfe.com
  Role:       Primary — full service stack, arr_sync union for all media shares

HOST2 — unRAID-Jayred365
  Hardware:   Intel i5 10th gen, 64GB RAM
  Location:   Remote — 50 miles away
  DDNS:       Gmer4Lfe.us
  Role:       Secondary — own stack + covers HOST1 + mirrors critical data
```

**Hardware does not need to match.** Everything is accessed through `/mnt/user/` — unRAID's
fused share layer. HOST1 has a Threadripper with ZFS pools. HOST2 has completely different
hardware. Fallback containers on HOST2 mount `/mnt/user/Movies` and see mirrored data
because the share names match. The hardware underneath is irrelevant.

**What must match between servers:**

```
Share names         /mnt/user/Movies must exist on both servers (mirrored data)
Container names     "Emby" on HOST2 must be the container HOST2 starts for HOST1
Network names       Docker custom networks must match for NPM routing to work
```

### Union Library — arr_sync.sh Is the Source of Truth

Both servers run arr instances simultaneously. arr_sync.sh keeps all databases in
bidirectional union before every rsync cycle — either server can download anything
and it propagates to the other:

```
HOST1 runs:  Radarr (Movies), Sonarr (Tv_Shows), Lidarr (Music)
HOST2 runs:  Radarr (Anime_Movies), Sonarr (Anime_Shows)

All arr databases merged via arr_sync. rsync spreads files additively.
Either server downloads to any share — the other catches up on the next cycle.
```

When HOST2 runs HOST1's arr containers during a Tier 4 fallback, those arrs already
know the full library — arr_sync kept them in union while both servers were online.

### Auth Stack — Warm on Both Servers

NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running
to serve his own users daily — this is not a fallback-only configuration. HOST1 is source of
truth: all changes mirror to HOST2 every 30 minutes via critical sync.

When DNS cuts over, auth is already running on the covering server. The 30-60 second dead
zone where auth is coming up after DNS has already switched does not exist.

---

## ━━━ HOW IT WORKS ━━━

Both servers run `fallback.sh` independently as a continuous background process. Each server
makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:

```bash
ping "$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")"   # is the other server reachable?
ping EXTERNAL_IP             # do I have internet? (default: 8.8.8.8)
```

No SSH signaling between servers. No shared state file. No election algorithm. Each server
acts entirely from its own network perspective.

**States:**

| State | Remote | Internet | Action |
|-------|--------|----------|--------|
| NORMAL | up | up | Silent — own containers, own DDNS on |
| FALLBACK | down | up | Start tier containers, cut DDNS over |
| NO_INTERNET | — | down | Stop own DDNS immediately, wait |
| DARK | down | down | Same as NO_INTERNET — cannot determine cause |

---

## ━━━ DDNS — THE CRITICAL PART ━━━

> **This took a year to get right. Do not change the sequencing.**

`fallback.sh` is the sole authority over when any DDNS container starts. Network state
returning is not permission to start DDNS. Only completion of the full handback sequence
grants that permission.

```
ONE DOMAIN → ONE DDNS ACTIVE → AT ALL TIMES

Gmer4Lfe.com → HOST1's DDNS normally → HOST2's DDNS during HOST1 outage
Gmer4Lfe.us  → HOST2's DDNS normally → HOST1's DDNS during HOST2 outage

Own DDNS:     ON when this server has internet. OFF when internet is lost.
Remote DDNS:  ON as Tier 1 fallback action. OFF as FIRST handback action.
Auto-start:   NEVER — DDNS never starts automatically on internet return.
```

**Why auto-start is forbidden:** If HOST1 lost internet and its DDNS auto-started when
internet returned, there is a window where both servers are updating the same domain with
different IPs. Users get routed randomly — some to the primary with fresh data, some to
the covering server. Authentication sessions don't transfer between servers. This is split
brain and produces the most confusing symptoms: intermittent auth failures with no clear
error state anywhere.

The brief gap where neither DDNS is updating the record is intentional. DNS TTL caches the
last value. During the rsync + container start window, cached DNS still routes users to the
covering server where containers are still running. By the time the cache expires, the
primary's DDNS has started and the record points at the right server.

---

## ━━━ TIERED FALLBACK ━━━

Starting the full stack for a 5-minute power blip wastes resources — most brief outages
resolve before Tier 2 would even activate. Tiers start only what is needed for the actual
outage duration.

| Tier | Delay | Coverage | Why This Timing |
|------|-------|----------|-----------------|
| Tier 1 | Immediate | Vital services + Live TV | People are watching — cannot wait 4 hours |
| Tier 2 | 4hr (HOST*_TIER2_DELAY) | NextCloud, Immich, Jellyseerr | 4hr covers most ISP and power events |
| Tier 3 | 12hr (HOST*_TIER3_DELAY) | Dashboard, AdGuard, Git, Collabora | Secondary — useful but not daily-critical |
| Tier 4 | 24hr (HOST*_TIER4_DELAY) | Arrs + downloaders | Significant I/O — only worth starting at 24hr |

Tier 1 always includes the remote domain's DDNS container as the first entry — DNS
coverage happens before any other container starts.

For the actual container lists and tier delay values, see `Manual-Fallback.md`.

---

## ━━━ HANDBACK SEQUENCE ━━━

When the remote server returns after a FALLBACK event. Every step has a reason.
Do not reorder.

1. **Strike confirmation** — FALLBACK_HANDBACK_STRIKES consecutive remote-up checks before
   handback begins. Prevents false triggers from brief network recovery.

2. **Pre-flight checks** — version parity, remote array mounted, remote Docker daemon up.
   Any failure aborts and retries next cycle.

3. **Staged reverse handback: Tier 4 → 3 → 2** — Emby and vital services stay on the
   covering server serving users throughout this phase. Each tier: stop local containers
   → rsync writeback (if outage exceeded tier threshold) → start on remote.

4. **DDNS handoff** — stop remote DDNS immediately before Tier 1 goes down. This prevents
   split brain during the Tier 1 rsync window.

5. **Tier 1 handback** — stop local vital services, rsync writeback, start on remote.

6. **Start remote DDNS last** — DNS cuts back to the primary only after all containers
   are confirmed running.

7. **Return to NORMAL** — state file reset, own DDNS restored if it was stopped.

---

## ━━━ MUTUAL FALLBACK — BOTH DIRECTIONS ━━━

The same `fallback.sh` handles both directions without any code changes. `detect_hosts()`
determines which server is local and which is remote at runtime, then selects the correct
container arrays and tier delays from config via MY_ID.

```
HOST2 covers HOST1:   FALLBACK_HOST1_TIER* (in host2.conf)
HOST1 covers HOST2:   FALLBACK_HOST2_TIER* (in host1.conf)
```

Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
comparisons anywhere in the script code.

---

## ━━━ INDEPENDENCE — ALWAYS ONE RSYNC STOP AWAY ━━━

HOST2 is designed to be fully independent if needed. If HOST2 ever wants to separate from
HOST1: stop HOST1 pushing data. Any changes HOST2 makes to his own data stick permanently.
His server becomes fully independent immediately — no script changes, no migration, no data
movement required. The fallback and rsync scripts are configuration-driven.

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━

| Script | Role | When It Runs |
|--------|------|--------------|
| `fallback.sh` | Continuous state machine — monitors remote, manages fallback and handback | Continuously (started by `array_started.sh`) |
| `fallback_test.sh` | 7-phase test harness — validates the entire fallback lifecycle via iptables simulation | On demand — maintenance window only |

---

## ━━━ HOW THE SCRIPTS RELATE ━━━

```
array_started.sh
      │
      └── starts fallback.sh (continuous loop)
                │
                ├── Every FALLBACK_CHECK_INTERVAL seconds:
                │     ping remote, ping internet
                │     → determine state → act on containers + DDNS
                │
                └── Test path:
                          │
                    fallback_test.sh
                          │
                          ├── Phase 1: pre-flight — both servers ready
                          ├── Phase 2: iptables DROP rule → remote appears down
                          ├── Phase 3: wait for fallback.sh to detect → FALLBACK state
                          ├── Phase 4: verify Tier 1 containers started locally
                          ├── Phase 5: remove DROP rule → remote reachable again
                          ├── Phase 6: wait for fallback.sh to complete handback → NORMAL
                          └── Phase 7: verify Tier 1 containers stopped locally
```

`fallback_test.sh` contains no fallback logic. It exercises the real `fallback.sh` through
connectivity manipulation. Any change to `fallback.sh` is automatically reflected in the
test result.
