Varaverk/Fallback/Manual-Fallback.md

# ━━━━━ FALLBACK — Manual ━━━━━

Config reference, procedures, operational workflows.
For overview see README-Fallback.md. For per-script detail see script headers.

---

## ━━━ CONFIGURATION — master.conf ━━━

```
FALLBACK_ENABLED=true
```
Enable or disable the entire fallback system. `false` = exit cleanly on startup — no
monitoring, no container actions. Set false when a server is being rebuilt or fallback
is temporarily suspended. Must be explicitly enabled when both servers are ready.
**(default: false)**

---

```
FALLBACK_CHECK_INTERVAL=30
```
Seconds between connectivity checks (ping remote + ping internet). Shorter = faster
detection and more pings. With `FALLBACK_HANDBACK_STRIKES=3` at 30s intervals:
fallback detected in ≤30s, handback requires 90s of continuous remote-up before
beginning. **(default: 30)**

---

```
FALLBACK_HANDBACK_STRIKES=3
```
Consecutive remote-up checks required before handback begins. Prevents false triggers
from brief network recovery during an ongoing outage. 3 strikes × 30s = 90s continuous
up required before handback sequence starts. **(default: 3)**

---

```
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
```
Path to the persistent state file. Lives on `/boot/` intentionally — survives reboots.
If the server was in FALLBACK state when it rebooted, it resumes FALLBACK on restart
rather than assuming everything is normal.

---

```
FALLBACK_RSYNC_ENABLED=true
```
Gate for writeback rsync jobs during handback. Set `false` to hand back without syncing
any data — useful when the primary still has reliable last-known-good state and writeback
would be counterproductive. **(default: true)**

---

```
EXTERNAL_IP=8.8.8.8
```
IP pinged to verify internet connectivity. **(default: 8.8.8.8)**

---

```
FALLBACK_TEST_BLOCK_WAIT=60
```
Seconds fallback_test.sh waits in Phase 3 for fallback.sh to detect the simulated
outage. Must be greater than `FALLBACK_CHECK_INTERVAL` plus a buffer. At 30s interval:
use ≥60s. **(default: 60)**

---

```
FALLBACK_TEST_HANDBACK_WAIT=300
```
Seconds fallback_test.sh waits in Phase 6 for fallback.sh to complete the full handback
sequence. Must cover: strike confirmation window + pre-flight time + rsync duration +
container start time. At 3 strikes × 30s + ~2min rsync + ~1min container start: use ≥240s.
**(default: 300)**

---

## ━━━ CONFIGURATION — master_host*.conf ━━━

```
HOST*_DDNS_CONTAINERS=("...")
```
DDNS containers managed by this host — the containers that update this server's DNS
records. Stopped when internet is lost. Started as the very last step of handback after
all primary containers are confirmed running.

```bash
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
```

---

```
FALLBACK_HOST*_STOP_ON_NO_NET=(...)
```
Containers to stop when this host loses internet. Services that are meaningless without
internet connectivity. Most hosts leave this empty.

---

```
FALLBACK_HOST*_COVERS_HOST*_TIER1=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER2=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER3=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER4=(...)
```
Containers this host starts for the remote host when the remote is down. TIER1 starts
immediately. TIER2–4 activate after the corresponding delay thresholds.

Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}`

The DDNS container for the remote's domain must be the first entry in TIER1 — DNS
coverage before anything else.

---

```
HOST*_TIER2_DELAY=240
HOST*_TIER3_DELAY=720
HOST*_TIER4_DELAY=1440
```
Minutes after FALLBACK state entry before activating each tier. Variable references the
**remote** host's ID — delays for a HOST1 outage use `HOST1_TIER*_DELAY`.

```bash
HOST1_TIER2_DELAY=240     # 4 hours — covers most ISP and power issues
HOST1_TIER3_DELAY=720     # 12 hours — genuine extended outage
HOST1_TIER4_DELAY=1440    # 24 hours — arrs only worth starting at this threshold
```

---

```
HOST*_TIER1_WRITEBACK_DELAY=60
```
Minimum outage duration in minutes before Tier 1 writeback runs on handback. Outages
shorter than this skip writeback entirely — the primary's last-known-good state is more
reliable than a short period of activity on the covering server.

```bash
HOST1_TIER1_WRITEBACK_DELAY=60    # skip writeback for outages under 60 minutes
```

---

```
FALLBACK_HOST*_WRITEBACK_TIER1=(...)
FALLBACK_HOST*_WRITEBACK_TIER2=(...)
FALLBACK_HOST*_WRITEBACK_TIER3=(...)
```
Paths rsynced back to the remote on handback, per tier. Writeback runs only if the
outage exceeded the tier's activation delay.

Variable pattern: `FALLBACK_${REMOTE_ID}_WRITEBACK_TIER${N}`

**Tier 4 writeback** auto-uses the remote's `HOST*_DAILY_SYNC_SHARES` — the same list
daily_sync_maintenance.sh uses, in the opposite direction. No separate TIER4 list needed.

---

## ━━━ FULL CONFIGURATION REFERENCE BLOCKS ━━━

### master.conf

```bash
FALLBACK_ENABLED=true
FALLBACK_CHECK_INTERVAL=30
FALLBACK_HANDBACK_STRIKES=3
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
FALLBACK_RSYNC_ENABLED=true
EXTERNAL_IP=8.8.8.8
FALLBACK_TEST_BLOCK_WAIT=60
FALLBACK_TEST_HANDBACK_WAIT=300
```

---

### master_host2.conf — HOST2 covering HOST1

```bash
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")

FALLBACK_HOST2_STOP_ON_NO_NET=()

# Tier 1 — immediate (vital services + Live TV)
FALLBACK_HOST2_COVERS_HOST1_TIER1=(
    "Gmer4Lfe.com-DDNS"         # ALWAYS FIRST — DNS coverage before anything else
    "Emby"                       # media server — people are watching
    "NginxProxyManager"          # reverse proxy — all external access routes through this
    "Lldap-Gmer4Lfe"             # user directory — already warm, verify and keep
    "Mariadb-Authelia"           # auth database — already warm, verify and keep
    "Redis-Authelia"             # auth session cache — already warm, verify and keep
    "Authelia"                   # SSO — already warm, serving users already
    "VaultWarden"                # passwords — people lock themselves out without this
    "Dispatcharr"                # Live TV scheduler — people are watching right now
    "Dispatcharr-Basic"
    "Dispatcharr-Iptv-Users"
    "ErsatzTV-Emby"              # channel schedule builder
)

# Tier 2 — after 4 hours (shared productivity services)
FALLBACK_HOST2_COVERS_HOST1_TIER2=(
    "Postgres-NextCloud"         # must start before NextCloud
    "NextCloud"
    "PostgreSQL-Immich"          # must start before Immich
    "Immich-Gmer4Lfe"
    "Jellyseerr"
)

# Tier 3 — after 12 hours (secondary services)
FALLBACK_HOST2_COVERS_HOST1_TIER3=(
    "Organizrv2-Gmer4Lfe"
    "AdGuard-Home"
    "UptimeKuma"
    "Gitea"
    "Collabora-CODE"
)

# Tier 4 — after 24 hours (arrs + downloaders)
FALLBACK_HOST2_COVERS_HOST1_TIER4=(
    "Sonarr-Gmer4Lfe"
    "Radarr-Gmer4Lfe"
    "Lidarr-Gmer4Lfe"
    "Prowlarr-Gmer4Lfe"
    "SABnzbd-Gmer4Lfe"
    "qBittorrent-Gmer4Lfe"
    "LidaTube"
    "Pinchflat"
)

# Tier delays for HOST1 outage
HOST1_TIER2_DELAY=240
HOST1_TIER3_DELAY=720
HOST1_TIER4_DELAY=1440
HOST1_TIER1_WRITEBACK_DELAY=60

# Writeback paths — synced back to HOST1 on handback
FALLBACK_HOST1_WRITEBACK_TIER1=(
    "/mnt/user/appdata-Failover/Critical-Data"    # auth stack — Authelia + NPM + certs
    "/mnt/user/Media_Server/Emby"                 # Emby userdata — watch history, playstates
)

FALLBACK_HOST1_WRITEBACK_TIER2=(
    "/mnt/user/appdata-Failover/Important-Data"   # NextCloud + Postgres + Immich
)

FALLBACK_HOST1_WRITEBACK_TIER3=(
    "/mnt/user/appdata-Failover/Gmer4Lfe"         # secondary appdata accumulated changes
)

# Tier 4 writeback uses HOST1_DAILY_SYNC_SHARES automatically — no list needed here.
```

---

### master_host1.conf — HOST1 covering HOST2

```bash
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")

FALLBACK_HOST1_STOP_ON_NO_NET=()

# Tier 1 — immediate (HOST2's vital services)
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
    "Gmer4Lfe.us-DDNS"          # ALWAYS FIRST
    # HOST2's Tier 1 services — fill per HOST2's stack
)

FALLBACK_HOST1_COVERS_HOST2_TIER2=(...)
FALLBACK_HOST1_COVERS_HOST2_TIER3=(...)
FALLBACK_HOST1_COVERS_HOST2_TIER4=(...)

# Tier delays for HOST2 outage
HOST2_TIER2_DELAY=240
HOST2_TIER3_DELAY=720
HOST2_TIER4_DELAY=1440
HOST2_TIER1_WRITEBACK_DELAY=60

FALLBACK_HOST2_WRITEBACK_TIER1=(
    # HOST2's critical data paths
)

# Tier 4 writeback uses HOST2_DAILY_SYNC_SHARES automatically.
```

---

## ━━━ STATE FILE REFERENCE ━━━

Location: `/boot/config/fallback_state.db` (survives reboots)

```
state=NORMAL            # NORMAL | FALLBACK | NO_INTERNET | DARK
fallback_start=0        # epoch timestamp when FALLBACK began (0 = not in FALLBACK)
handback_strikes=0      # consecutive remote-up checks accumulated toward handback
tier2_started=false     # whether Tier 2 containers started this event
tier3_started=false     # whether Tier 3 containers started
tier4_started=false     # whether Tier 4 containers started
```

View state: `cat /boot/config/fallback_state.db`  
Check state: `fallback.sh --status`

The file is managed exclusively by fallback.sh. Do not edit it while fallback.sh is
running — the next cycle will overwrite your changes. Use the Manual State Reset
procedure (below) when fallback.sh is not running.

---

## ━━━ INITIAL SETUP ━━━

Before fallback.sh can work on both servers:

### 1. Tailscale Connected on Both Servers

```bash
# Verify from HOST1
tailscale ip -4 unRAID-Jayred365
tailscale ping unRAID-Jayred365

# Verify from HOST2
tailscale ip -4 unRAID-Gmer4Lfe
tailscale ping unRAID-Gmer4Lfe
```

### 2. SSH Keys Configured — No Password Prompt

```bash
# From HOST1 — should print HOST2's hostname without password prompt
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"

# From HOST2 — should print HOST1's hostname
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"
```

### 3. Container Names Match

Fallback containers must exist (but stopped) on the covering server, with volume mounts
pointing at mirrored share paths.

```bash
# Verify on HOST2 that HOST1's container exists (stopped is expected)
ssh root@[HOST2-tailscale-ip] "docker inspect Emby --format '{{.State.Status}}'"
# Expected: created or exited — NOT "no such container"
```

### 4. Critical Data Mirrored

These shares must exist on HOST2 with current data from HOST1 before failover is needed:

```
/mnt/user/appdata-Failover/Critical-Data    # auth stack — NPM, LLDAP, Authelia, certs
/mnt/user/appdata-Failover/Important-Data   # NextCloud + Postgres + Immich
/mnt/user/Media_Server/Emby                 # Emby userdata — watch history, playstates
/mnt/user/appdata-Failover/Gmer4Lfe         # server appdata
```

```bash
# Verify data is current — check modification times
ssh root@[HOST2-tailscale-ip] "ls -la /mnt/user/appdata-Failover/Critical-Data/"
```

Sync is maintained continuously by `daily_sync_maintenance.sh` critical-data profile.

### 5. DDNS TTL Set to 1 Minute

Set in your DDNS provider settings. Higher TTL means users continue hitting the old IP
for longer after failover. At 5-minute TTL, users can be hitting a downed server for up
to 5 minutes before DNS switches.

### 6. Both Servers Running fallback.sh

Fallback only works in one direction if only one server is running the script. Both
servers must be running it continuously for mutual coverage.

```bash
# Verify fallback.sh is running
pgrep -f "fallback.sh"

# Check the state file
cat /boot/config/fallback_state.db
```

Start via User Scripts plugin on both servers.

---

## ━━━ MONITORING ━━━

**Sunday coffee report** (`sunday_morning_coffee_report.sh`) — Fallback section shows
current state, outage duration if not NORMAL, Tailscale reachability, and whether
fallback.sh is running.

**Weekly health digest** (`weekly_health_digest.sh`) — reads the state file. If
`DIGEST_SMART_ON_FAILOVER=true` and state is not NORMAL, it sends a notification even
in smart mode — a non-NORMAL state at digest time needs attention.

**Direct check:**

```bash
fallback.sh --status          # full state snapshot
cat /boot/config/fallback_state.db    # raw state file
```

---

## ━━━ PROCEDURES ━━━

### Running the Failover Test

> This starts and stops real containers on both servers. Users will experience a brief
> service interruption. Always run `--dry-run` first.

```bash
# Step 1 — verify all phases without touching anything
fallback_test.sh --dry-run

# Step 2 — schedule maintenance window, then run live
fallback_test.sh

# Step 3 — check state after test completes
fallback.sh --status
cat /boot/config/fallback_state.db
```

If the test doesn't complete cleanly, the state file may be left in FALLBACK. The
iptables safety trap in fallback_test.sh removes the DROP rule on any exit, so remote
connectivity is always restored. Use the Manual State Reset procedure below if the state
file is stuck.

### Manual State Reset

Use when the state file is stuck in a non-NORMAL state after testing, a failed handback,
or killing fallback.sh directly (not via User Scripts Abort).

**Before resetting, verify the situation is actually safe to reset:**

```bash
# 1. Right containers running on the right server
docker ps | grep -E "Emby|VaultWarden|NginxProxyManager"

# 2. DDNS pointing at the correct server
nslookup Gmer4Lfe.com 8.8.8.8
nslookup Gmer4Lfe.us 8.8.8.8

# 3. Both servers visible on Tailscale
tailscale ping [remote-tailscale-ip]

# 4. No actual fallback in progress (remote is genuinely up and stable)
ping -c 5 [remote-tailscale-ip]
```

**Stop fallback.sh first (via User Scripts Abort), then reset:**

```bash
# View current state
cat /boot/config/fallback_state.db

# Write a clean NORMAL state
cat > /boot/config/fallback_state.db << 'EOF'
state=NORMAL
fallback_start=0
handback_strikes=0
tier2_started=false
tier3_started=false
tier4_started=false
EOF

# Verify the write
cat /boot/config/fallback_state.db
```

Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its next cycle.

> **Warning:** Do NOT reset during an actual fallback event. fallback.sh will think
> everything is normal and stop covering the remote — services go offline until the next
> detection cycle catches it again.

### Adding a Container to a Tier

1. Create the container on the covering server (stopped), with volume mounts pointing at
   the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in master_host*.conf
   in the appropriate tier position (dependency ordering — databases before apps)
3. Verify: `fallback.sh --status` shows the container in the expected tier list
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid

---

## ━━━ TROUBLESHOOTING ━━━

### Fallback Not Triggering

```
Is fallback.sh running on the covering server?
  → User Scripts plugin → check status of the fallback script
  → pgrep -f "fallback.sh"

Is Tailscale connected?
  → tailscale status   (should show the remote peer)

Can this server reach the remote Tailscale IP?
  → ping [remote-tailscale-ip]

What does fallback.sh report?
  → fallback.sh --status
  → cat /boot/config/fallback_state.db
```

### Handback Not Completing

```
Is the primary's array fully started?
  → ls /mnt/user   (should show share directories)

Is Docker responding on the primary?
  → docker ps   (should return a list, not hang)

Is rootfs nearly full? (pre-flight checks this)
  → df /

Is rsync running and stuck?
  → pgrep rsync
  → A stalled rsync blocks handback. Let fallback.sh retry next cycle.
```

### DDNS Not Cutting Over

```
Is the DDNS container running on the covering server?
  → docker ps | grep DDNS

What TTL is the DNS record set to?
  → nslookup Gmer4Lfe.com 8.8.8.8   (check TTL in response)
  → High TTL = slow propagation

Is the DDNS provider accepting updates?
  → docker logs [ddns-container] --tail 50
```

### Split Brain — Both DDNS Running

```
This should not happen if the handback sequence completed correctly.
If it has happened:

1. Check both servers for running DDNS containers
   HOST1:  docker ps | grep DDNS
   HOST2:  docker ps | grep DDNS

2. Stop the duplicate (the one on the covering server)
   docker stop [duplicate-ddns-container]

3. Understand the state before resetting
   fallback.sh --status
   cat /boot/config/fallback_state.db

4. Perform Manual State Reset above on the server in a bad state

5. Restart fallback.sh via User Scripts plugin
```

---

## ━━━ OUTPUT TIERS ━━━

Both scripts have two output levels controlled by `--log`.

**fallback.sh** — daemon, runs continuously. Without `--log`, only state transitions,
warnings, errors, and the startup banner are visible. Per-cycle detail (ping results,
state evaluation) is suppressed — the daemon runs every 15–30 seconds and clean cycles
produce no output by design. State transitions are always visible because they are the
events that matter. Use `--log` when debugging why the state machine is or is not
acting.

**fallback_test.sh** — one-shot test harness. Without `--log`, phase headers, pass/fail
results, and the final test report are always visible. Per-container checks within each
phase are suppressed. With `--log`, every check in every phase is shown. Warnings and
errors are always visible regardless of `--log`.

---

## ━━━ FLAG REFERENCE ━━━

### fallback.sh

`fallback.sh`  
Normal start — continuous loop. Start via User Scripts plugin or array_start.sh. **Do NOT
stop by killing the process** — state file may be left inconsistent. Stop via User Scripts
Abort only.

`fallback.sh --dry-run`  
Walk through one full cycle showing what would happen based on current network state. No
containers started or stopped. No DDNS changes. Use to verify configuration before relying
on it.

`fallback.sh --status`  
Show current state, identity, DDNS containers, check interval, handback strikes, and
(if in FALLBACK) outage duration and tier activation status. Use this first to understand
current state before any manual intervention.

`fallback.sh --log`  
Verbose output on every decision in every cycle — ping results, state evaluation, tier
checks. Use when debugging why the state machine is or is not acting as expected.

---

### fallback_test.sh

`fallback_test.sh --dry-run`  
Walk through all 7 phases with full output. No iptables rules added. No container
starts or stops. **Always run this before a live test** — confirms timing configuration
is correct and phases would pass before committing to real changes.

`fallback_test.sh`  
Full live test — real iptables DROP rule, real container lifecycle. Users will experience
a brief service interruption. Run during a maintenance window. The iptables safety trap
removes the DROP rule on any exit (normal completion, crash, ctrl-c).

`fallback_test.sh --status`  
Show current fallback state, test timing configuration, and the Tier 1 containers that
would be tested. No test run.

`fallback_test.sh --log`  
Verbose output on every check in every phase.