feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,613 @@
|
||||
# ━━━━━ FALLBACK — Manual ━━━━━
|
||||
|
||||
Config reference, procedures, operational workflows.
|
||||
For overview see README-Fallback.md. For per-script detail see script headers.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONFIGURATION — master.conf ━━━
|
||||
|
||||
```
|
||||
FALLBACK_ENABLED=true
|
||||
```
|
||||
Enable or disable the entire fallback system. `false` = exit cleanly on startup — no
|
||||
monitoring, no container actions. Set false when a server is being rebuilt or fallback
|
||||
is temporarily suspended. Must be explicitly enabled when both servers are ready.
|
||||
**(default: false)**
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_CHECK_INTERVAL=30
|
||||
```
|
||||
Seconds between connectivity checks (ping remote + ping internet). Shorter = faster
|
||||
detection and more pings. With `FALLBACK_HANDBACK_STRIKES=3` at 30s intervals:
|
||||
fallback detected in ≤30s, handback requires 90s of continuous remote-up before
|
||||
beginning. **(default: 30)**
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_HANDBACK_STRIKES=3
|
||||
```
|
||||
Consecutive remote-up checks required before handback begins. Prevents false triggers
|
||||
from brief network recovery during an ongoing outage. 3 strikes × 30s = 90s continuous
|
||||
up required before handback sequence starts. **(default: 3)**
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
|
||||
```
|
||||
Path to the persistent state file. Lives on `/boot/` intentionally — survives reboots.
|
||||
If the server was in FALLBACK state when it rebooted, it resumes FALLBACK on restart
|
||||
rather than assuming everything is normal.
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_RSYNC_ENABLED=true
|
||||
```
|
||||
Gate for writeback rsync jobs during handback. Set `false` to hand back without syncing
|
||||
any data — useful when the primary still has reliable last-known-good state and writeback
|
||||
would be counterproductive. **(default: true)**
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
EXTERNAL_IP=8.8.8.8
|
||||
```
|
||||
IP pinged to verify internet connectivity. **(default: 8.8.8.8)**
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_TEST_BLOCK_WAIT=60
|
||||
```
|
||||
Seconds fallback_test.sh waits in Phase 3 for fallback.sh to detect the simulated
|
||||
outage. Must be greater than `FALLBACK_CHECK_INTERVAL` plus a buffer. At 30s interval:
|
||||
use ≥60s. **(default: 60)**
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_TEST_HANDBACK_WAIT=300
|
||||
```
|
||||
Seconds fallback_test.sh waits in Phase 6 for fallback.sh to complete the full handback
|
||||
sequence. Must cover: strike confirmation window + pre-flight time + rsync duration +
|
||||
container start time. At 3 strikes × 30s + ~2min rsync + ~1min container start: use ≥240s.
|
||||
**(default: 300)**
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONFIGURATION — master_host*.conf ━━━
|
||||
|
||||
```
|
||||
HOST*_DDNS_CONTAINERS=("...")
|
||||
```
|
||||
DDNS containers managed by this host — the containers that update this server's DNS
|
||||
records. Stopped when internet is lost. Started as the very last step of handback after
|
||||
all primary containers are confirmed running.
|
||||
|
||||
```bash
|
||||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_HOST*_STOP_ON_NO_NET=(...)
|
||||
```
|
||||
Containers to stop when this host loses internet. Services that are meaningless without
|
||||
internet connectivity. Most hosts leave this empty.
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER1=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER2=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER3=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER4=(...)
|
||||
```
|
||||
Containers this host starts for the remote host when the remote is down. TIER1 starts
|
||||
immediately. TIER2–4 activate after the corresponding delay thresholds.
|
||||
|
||||
Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}`
|
||||
|
||||
The DDNS container for the remote's domain must be the first entry in TIER1 — DNS
|
||||
coverage before anything else.
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
HOST*_TIER2_DELAY=240
|
||||
HOST*_TIER3_DELAY=720
|
||||
HOST*_TIER4_DELAY=1440
|
||||
```
|
||||
Minutes after FALLBACK state entry before activating each tier. Variable references the
|
||||
**remote** host's ID — delays for a HOST1 outage use `HOST1_TIER*_DELAY`.
|
||||
|
||||
```bash
|
||||
HOST1_TIER2_DELAY=240 # 4 hours — covers most ISP and power issues
|
||||
HOST1_TIER3_DELAY=720 # 12 hours — genuine extended outage
|
||||
HOST1_TIER4_DELAY=1440 # 24 hours — arrs only worth starting at this threshold
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
HOST*_TIER1_WRITEBACK_DELAY=60
|
||||
```
|
||||
Minimum outage duration in minutes before Tier 1 writeback runs on handback. Outages
|
||||
shorter than this skip writeback entirely — the primary's last-known-good state is more
|
||||
reliable than a short period of activity on the covering server.
|
||||
|
||||
```bash
|
||||
HOST1_TIER1_WRITEBACK_DELAY=60 # skip writeback for outages under 60 minutes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_HOST*_WRITEBACK_TIER1=(...)
|
||||
FALLBACK_HOST*_WRITEBACK_TIER2=(...)
|
||||
FALLBACK_HOST*_WRITEBACK_TIER3=(...)
|
||||
```
|
||||
Paths rsynced back to the remote on handback, per tier. Writeback runs only if the
|
||||
outage exceeded the tier's activation delay.
|
||||
|
||||
Variable pattern: `FALLBACK_${REMOTE_ID}_WRITEBACK_TIER${N}`
|
||||
|
||||
**Tier 4 writeback** auto-uses the remote's `HOST*_DAILY_SYNC_SHARES` — the same list
|
||||
daily_sync_maintenance.sh uses, in the opposite direction. No separate TIER4 list needed.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FULL CONFIGURATION REFERENCE BLOCKS ━━━
|
||||
|
||||
### master.conf
|
||||
|
||||
```bash
|
||||
FALLBACK_ENABLED=true
|
||||
FALLBACK_CHECK_INTERVAL=30
|
||||
FALLBACK_HANDBACK_STRIKES=3
|
||||
FALLBACK_STATE_FILE=/boot/config/fallback_state.db
|
||||
FALLBACK_RSYNC_ENABLED=true
|
||||
EXTERNAL_IP=8.8.8.8
|
||||
FALLBACK_TEST_BLOCK_WAIT=60
|
||||
FALLBACK_TEST_HANDBACK_WAIT=300
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### master_host2.conf — HOST2 covering HOST1
|
||||
|
||||
```bash
|
||||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||||
|
||||
FALLBACK_HOST2_STOP_ON_NO_NET=()
|
||||
|
||||
# Tier 1 — immediate (vital services + Live TV)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
||||
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
|
||||
"Emby" # media server — people are watching
|
||||
"NginxProxyManager" # reverse proxy — all external access routes through this
|
||||
"Lldap-Gmer4Lfe" # user directory — already warm, verify and keep
|
||||
"Mariadb-Authelia" # auth database — already warm, verify and keep
|
||||
"Redis-Authelia" # auth session cache — already warm, verify and keep
|
||||
"Authelia" # SSO — already warm, serving users already
|
||||
"VaultWarden" # passwords — people lock themselves out without this
|
||||
"Dispatcharr" # Live TV scheduler — people are watching right now
|
||||
"Dispatcharr-Basic"
|
||||
"Dispatcharr-Iptv-Users"
|
||||
"ErsatzTV-Emby" # channel schedule builder
|
||||
)
|
||||
|
||||
# Tier 2 — after 4 hours (shared productivity services)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER2=(
|
||||
"Postgres-NextCloud" # must start before NextCloud
|
||||
"NextCloud"
|
||||
"PostgreSQL-Immich" # must start before Immich
|
||||
"Immich-Gmer4Lfe"
|
||||
"Jellyseerr"
|
||||
)
|
||||
|
||||
# Tier 3 — after 12 hours (secondary services)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER3=(
|
||||
"Organizrv2-Gmer4Lfe"
|
||||
"AdGuard-Home"
|
||||
"UptimeKuma"
|
||||
"Gitea"
|
||||
"Collabora-CODE"
|
||||
)
|
||||
|
||||
# Tier 4 — after 24 hours (arrs + downloaders)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER4=(
|
||||
"Sonarr-Gmer4Lfe"
|
||||
"Radarr-Gmer4Lfe"
|
||||
"Lidarr-Gmer4Lfe"
|
||||
"Prowlarr-Gmer4Lfe"
|
||||
"SABnzbd-Gmer4Lfe"
|
||||
"qBittorrent-Gmer4Lfe"
|
||||
"LidaTube"
|
||||
"Pinchflat"
|
||||
)
|
||||
|
||||
# Tier delays for HOST1 outage
|
||||
HOST1_TIER2_DELAY=240
|
||||
HOST1_TIER3_DELAY=720
|
||||
HOST1_TIER4_DELAY=1440
|
||||
HOST1_TIER1_WRITEBACK_DELAY=60
|
||||
|
||||
# Writeback paths — synced back to HOST1 on handback
|
||||
FALLBACK_HOST1_WRITEBACK_TIER1=(
|
||||
"/mnt/user/appdata-Failover/Critical-Data" # auth stack — Authelia + NPM + certs
|
||||
"/mnt/user/Media_Server/Emby" # Emby userdata — watch history, playstates
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER2=(
|
||||
"/mnt/user/appdata-Failover/Important-Data" # NextCloud + Postgres + Immich
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER3=(
|
||||
"/mnt/user/appdata-Failover/Gmer4Lfe" # secondary appdata accumulated changes
|
||||
)
|
||||
|
||||
# Tier 4 writeback uses HOST1_DAILY_SYNC_SHARES automatically — no list needed here.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### master_host1.conf — HOST1 covering HOST2
|
||||
|
||||
```bash
|
||||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||||
|
||||
FALLBACK_HOST1_STOP_ON_NO_NET=()
|
||||
|
||||
# Tier 1 — immediate (HOST2's vital services)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
|
||||
"Gmer4Lfe.us-DDNS" # ALWAYS FIRST
|
||||
# HOST2's Tier 1 services — fill per HOST2's stack
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER2=(...)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER3=(...)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER4=(...)
|
||||
|
||||
# Tier delays for HOST2 outage
|
||||
HOST2_TIER2_DELAY=240
|
||||
HOST2_TIER3_DELAY=720
|
||||
HOST2_TIER4_DELAY=1440
|
||||
HOST2_TIER1_WRITEBACK_DELAY=60
|
||||
|
||||
FALLBACK_HOST2_WRITEBACK_TIER1=(
|
||||
# HOST2's critical data paths
|
||||
)
|
||||
|
||||
# Tier 4 writeback uses HOST2_DAILY_SYNC_SHARES automatically.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ STATE FILE REFERENCE ━━━
|
||||
|
||||
Location: `/boot/config/fallback_state.db` (survives reboots)
|
||||
|
||||
```
|
||||
state=NORMAL # NORMAL | FALLBACK | NO_INTERNET | DARK
|
||||
fallback_start=0 # epoch timestamp when FALLBACK began (0 = not in FALLBACK)
|
||||
handback_strikes=0 # consecutive remote-up checks accumulated toward handback
|
||||
tier2_started=false # whether Tier 2 containers started this event
|
||||
tier3_started=false # whether Tier 3 containers started
|
||||
tier4_started=false # whether Tier 4 containers started
|
||||
```
|
||||
|
||||
View state: `cat /boot/config/fallback_state.db`
|
||||
Check state: `fallback.sh --status`
|
||||
|
||||
The file is managed exclusively by fallback.sh. Do not edit it while fallback.sh is
|
||||
running — the next cycle will overwrite your changes. Use the Manual State Reset
|
||||
procedure (below) when fallback.sh is not running.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ INITIAL SETUP ━━━
|
||||
|
||||
Before fallback.sh can work on both servers:
|
||||
|
||||
### 1. Tailscale Connected on Both Servers
|
||||
|
||||
```bash
|
||||
# Verify from HOST1
|
||||
tailscale ip -4 unRAID-Jayred365
|
||||
tailscale ping unRAID-Jayred365
|
||||
|
||||
# Verify from HOST2
|
||||
tailscale ip -4 unRAID-Gmer4Lfe
|
||||
tailscale ping unRAID-Gmer4Lfe
|
||||
```
|
||||
|
||||
### 2. SSH Keys Configured — No Password Prompt
|
||||
|
||||
```bash
|
||||
# From HOST1 — should print HOST2's hostname without password prompt
|
||||
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"
|
||||
|
||||
# From HOST2 — should print HOST1's hostname
|
||||
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"
|
||||
```
|
||||
|
||||
### 3. Container Names Match
|
||||
|
||||
Fallback containers must exist (but stopped) on the covering server, with volume mounts
|
||||
pointing at mirrored share paths.
|
||||
|
||||
```bash
|
||||
# Verify on HOST2 that HOST1's container exists (stopped is expected)
|
||||
ssh root@[HOST2-tailscale-ip] "docker inspect Emby --format '{{.State.Status}}'"
|
||||
# Expected: created or exited — NOT "no such container"
|
||||
```
|
||||
|
||||
### 4. Critical Data Mirrored
|
||||
|
||||
These shares must exist on HOST2 with current data from HOST1 before failover is needed:
|
||||
|
||||
```
|
||||
/mnt/user/appdata-Failover/Critical-Data # auth stack — NPM, LLDAP, Authelia, certs
|
||||
/mnt/user/appdata-Failover/Important-Data # NextCloud + Postgres + Immich
|
||||
/mnt/user/Media_Server/Emby # Emby userdata — watch history, playstates
|
||||
/mnt/user/appdata-Failover/Gmer4Lfe # server appdata
|
||||
```
|
||||
|
||||
```bash
|
||||
# Verify data is current — check modification times
|
||||
ssh root@[HOST2-tailscale-ip] "ls -la /mnt/user/appdata-Failover/Critical-Data/"
|
||||
```
|
||||
|
||||
Sync is maintained continuously by `daily_sync_maintenance.sh` critical-data profile.
|
||||
|
||||
### 5. DDNS TTL Set to 1 Minute
|
||||
|
||||
Set in your DDNS provider settings. Higher TTL means users continue hitting the old IP
|
||||
for longer after failover. At 5-minute TTL, users can be hitting a downed server for up
|
||||
to 5 minutes before DNS switches.
|
||||
|
||||
### 6. Both Servers Running fallback.sh
|
||||
|
||||
Fallback only works in one direction if only one server is running the script. Both
|
||||
servers must be running it continuously for mutual coverage.
|
||||
|
||||
```bash
|
||||
# Verify fallback.sh is running
|
||||
pgrep -f "fallback.sh"
|
||||
|
||||
# Check the state file
|
||||
cat /boot/config/fallback_state.db
|
||||
```
|
||||
|
||||
Start via User Scripts plugin on both servers.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ MONITORING ━━━
|
||||
|
||||
**Sunday coffee report** (`sunday_morning_coffee_report.sh`) — Fallback section shows
|
||||
current state, outage duration if not NORMAL, Tailscale reachability, and whether
|
||||
fallback.sh is running.
|
||||
|
||||
**Weekly health digest** (`weekly_health_digest.sh`) — reads the state file. If
|
||||
`DIGEST_SMART_ON_FAILOVER=true` and state is not NORMAL, it sends a notification even
|
||||
in smart mode — a non-NORMAL state at digest time needs attention.
|
||||
|
||||
**Direct check:**
|
||||
|
||||
```bash
|
||||
fallback.sh --status # full state snapshot
|
||||
cat /boot/config/fallback_state.db # raw state file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ PROCEDURES ━━━
|
||||
|
||||
### Running the Failover Test
|
||||
|
||||
> This starts and stops real containers on both servers. Users will experience a brief
|
||||
> service interruption. Always run `--dry-run` first.
|
||||
|
||||
```bash
|
||||
# Step 1 — verify all phases without touching anything
|
||||
fallback_test.sh --dry-run
|
||||
|
||||
# Step 2 — schedule maintenance window, then run live
|
||||
fallback_test.sh
|
||||
|
||||
# Step 3 — check state after test completes
|
||||
fallback.sh --status
|
||||
cat /boot/config/fallback_state.db
|
||||
```
|
||||
|
||||
If the test doesn't complete cleanly, the state file may be left in FALLBACK. The
|
||||
iptables safety trap in fallback_test.sh removes the DROP rule on any exit, so remote
|
||||
connectivity is always restored. Use the Manual State Reset procedure below if the state
|
||||
file is stuck.
|
||||
|
||||
### Manual State Reset
|
||||
|
||||
Use when the state file is stuck in a non-NORMAL state after testing, a failed handback,
|
||||
or killing fallback.sh directly (not via User Scripts Abort).
|
||||
|
||||
**Before resetting, verify the situation is actually safe to reset:**
|
||||
|
||||
```bash
|
||||
# 1. Right containers running on the right server
|
||||
docker ps | grep -E "Emby|VaultWarden|NginxProxyManager"
|
||||
|
||||
# 2. DDNS pointing at the correct server
|
||||
nslookup Gmer4Lfe.com 8.8.8.8
|
||||
nslookup Gmer4Lfe.us 8.8.8.8
|
||||
|
||||
# 3. Both servers visible on Tailscale
|
||||
tailscale ping [remote-tailscale-ip]
|
||||
|
||||
# 4. No actual fallback in progress (remote is genuinely up and stable)
|
||||
ping -c 5 [remote-tailscale-ip]
|
||||
```
|
||||
|
||||
**Stop fallback.sh first (via User Scripts Abort), then reset:**
|
||||
|
||||
```bash
|
||||
# View current state
|
||||
cat /boot/config/fallback_state.db
|
||||
|
||||
# Write a clean NORMAL state
|
||||
cat > /boot/config/fallback_state.db << 'EOF'
|
||||
state=NORMAL
|
||||
fallback_start=0
|
||||
handback_strikes=0
|
||||
tier2_started=false
|
||||
tier3_started=false
|
||||
tier4_started=false
|
||||
EOF
|
||||
|
||||
# Verify the write
|
||||
cat /boot/config/fallback_state.db
|
||||
```
|
||||
|
||||
Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its next cycle.
|
||||
|
||||
> **Warning:** Do NOT reset during an actual fallback event. fallback.sh will think
|
||||
> everything is normal and stop covering the remote — services go offline until the next
|
||||
> detection cycle catches it again.
|
||||
|
||||
### Adding a Container to a Tier
|
||||
|
||||
1. Create the container on the covering server (stopped), with volume mounts pointing at
|
||||
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
|
||||
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in master_host*.conf
|
||||
in the appropriate tier position (dependency ordering — databases before apps)
|
||||
3. Verify: `fallback.sh --status` shows the container in the expected tier list
|
||||
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
|
||||
|
||||
---
|
||||
|
||||
## ━━━ TROUBLESHOOTING ━━━
|
||||
|
||||
### Fallback Not Triggering
|
||||
|
||||
```
|
||||
Is fallback.sh running on the covering server?
|
||||
→ User Scripts plugin → check status of the fallback script
|
||||
→ pgrep -f "fallback.sh"
|
||||
|
||||
Is Tailscale connected?
|
||||
→ tailscale status (should show the remote peer)
|
||||
|
||||
Can this server reach the remote Tailscale IP?
|
||||
→ ping [remote-tailscale-ip]
|
||||
|
||||
What does fallback.sh report?
|
||||
→ fallback.sh --status
|
||||
→ cat /boot/config/fallback_state.db
|
||||
```
|
||||
|
||||
### Handback Not Completing
|
||||
|
||||
```
|
||||
Is the primary's array fully started?
|
||||
→ ls /mnt/user (should show share directories)
|
||||
|
||||
Is Docker responding on the primary?
|
||||
→ docker ps (should return a list, not hang)
|
||||
|
||||
Is rootfs nearly full? (pre-flight checks this)
|
||||
→ df /
|
||||
|
||||
Is rsync running and stuck?
|
||||
→ pgrep rsync
|
||||
→ A stalled rsync blocks handback. Let fallback.sh retry next cycle.
|
||||
```
|
||||
|
||||
### DDNS Not Cutting Over
|
||||
|
||||
```
|
||||
Is the DDNS container running on the covering server?
|
||||
→ docker ps | grep DDNS
|
||||
|
||||
What TTL is the DNS record set to?
|
||||
→ nslookup Gmer4Lfe.com 8.8.8.8 (check TTL in response)
|
||||
→ High TTL = slow propagation
|
||||
|
||||
Is the DDNS provider accepting updates?
|
||||
→ docker logs [ddns-container] --tail 50
|
||||
```
|
||||
|
||||
### Split Brain — Both DDNS Running
|
||||
|
||||
```
|
||||
This should not happen if the handback sequence completed correctly.
|
||||
If it has happened:
|
||||
|
||||
1. Check both servers for running DDNS containers
|
||||
HOST1: docker ps | grep DDNS
|
||||
HOST2: docker ps | grep DDNS
|
||||
|
||||
2. Stop the duplicate (the one on the covering server)
|
||||
docker stop [duplicate-ddns-container]
|
||||
|
||||
3. Understand the state before resetting
|
||||
fallback.sh --status
|
||||
cat /boot/config/fallback_state.db
|
||||
|
||||
4. Perform Manual State Reset above on the server in a bad state
|
||||
|
||||
5. Restart fallback.sh via User Scripts plugin
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FLAG REFERENCE ━━━
|
||||
|
||||
### fallback.sh
|
||||
|
||||
`fallback.sh`
|
||||
Normal start — continuous loop. Start via User Scripts plugin or array_start.sh. **Do NOT
|
||||
stop by killing the process** — state file may be left inconsistent. Stop via User Scripts
|
||||
Abort only.
|
||||
|
||||
`fallback.sh --dry-run`
|
||||
Walk through one full cycle showing what would happen based on current network state. No
|
||||
containers started or stopped. No DDNS changes. Use to verify configuration before relying
|
||||
on it.
|
||||
|
||||
`fallback.sh --status`
|
||||
Show current state, identity, DDNS containers, check interval, handback strikes, and
|
||||
(if in FALLBACK) outage duration and tier activation status. Use this first to understand
|
||||
current state before any manual intervention.
|
||||
|
||||
`fallback.sh --log`
|
||||
Verbose output on every decision in every cycle — ping results, state evaluation, tier
|
||||
checks. Use when debugging why the state machine is or is not acting as expected.
|
||||
|
||||
---
|
||||
|
||||
### fallback_test.sh
|
||||
|
||||
`fallback_test.sh --dry-run`
|
||||
Walk through all 7 phases with full output. No iptables rules added. No container
|
||||
starts or stops. **Always run this before a live test** — confirms timing configuration
|
||||
is correct and phases would pass before committing to real changes.
|
||||
|
||||
`fallback_test.sh`
|
||||
Full live test — real iptables DROP rule, real container lifecycle. Users will experience
|
||||
a brief service interruption. Run during a maintenance window. The iptables safety trap
|
||||
removes the DROP rule on any exit (normal completion, crash, ctrl-c).
|
||||
|
||||
`fallback_test.sh --status`
|
||||
Show current fallback state, test timing configuration, and the Tier 1 containers that
|
||||
would be tested. No test run.
|
||||
|
||||
`fallback_test.sh --log`
|
||||
Verbose output on every check in every phase.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,275 @@
|
||||
# ━━━━━ FALLBACK ━━━━━
|
||||
|
||||
Mutual automatic failover between two independent unRAID servers. When one goes down the
|
||||
other starts its containers, cuts over DNS, and keeps users online. When it comes back
|
||||
everything hands back in the correct sequence — covering DDNS stops, containers stop, rsync
|
||||
writeback runs, containers start on the primary, primary DDNS starts last — so users hit the
|
||||
returning server only after it's actually ready.
|
||||
|
||||
> **Built from scratch. Refined through a year of production testing.** The DDNS sequencing
|
||||
> and handback order were the hardest parts to get right. Both directions are exercised
|
||||
> regularly with `fallback_test.sh`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
Running a self-hosted stack means being the operator. These are the specific problems
|
||||
that drove this build:
|
||||
|
||||
**A Single Point of Failure for an Entire Household**
|
||||
HOST1 runs Emby, NPM, Authelia, NextCloud, VaultWarden, and every service the household
|
||||
uses daily. When HOST1 goes down — even briefly — all of those services go down with it.
|
||||
The fix: a second server with mirrored critical data that covers the first automatically.
|
||||
From a user's perspective, a brief interruption and then everything is back.
|
||||
|
||||
**DNS Cutting Over Before the Server Was Ready**
|
||||
Early attempts started containers on the covering server then updated DNS. Problem: DNS
|
||||
propagated in under a minute. Users hit the new IP before Emby had finished starting,
|
||||
before Authelia had loaded its sessions, before NPM had loaded its proxy configurations.
|
||||
The fix: warm standby for the auth stack. NPM, LLDAP, and Authelia run actively on both
|
||||
servers at all times. When DNS cuts over, auth is already running and ready.
|
||||
|
||||
**Split Brain DNS During Handback**
|
||||
When HOST1 returned, the obvious sequence was: start HOST1 containers, then switch DNS
|
||||
back. Problem: between "start containers" and "DNS switches" both servers' DDNS containers
|
||||
were running, both updating the same domain with different IPs. Users got routed randomly
|
||||
between servers — intermittent auth failures, no clear error state anywhere.
|
||||
The fix: stop DDNS on the covering server first, before anything else moves. There is
|
||||
never a window where two DDNS containers update the same record.
|
||||
|
||||
**Rsync Running Into Active Container I/O**
|
||||
Syncing data back while containers were still running — to minimise downtime — produced
|
||||
slower transfers, potential file inconsistency, and database dirty state risk.
|
||||
The fix: stop containers before syncing. The outage window is only the rsync duration —
|
||||
typically minutes. Clean static source at full bandwidth, predictable state every time.
|
||||
|
||||
**No Way to Validate the System Before Needing It**
|
||||
A failover system that has never been tested is not a failover system — it is a hope.
|
||||
The fix: `fallback_test.sh` — a controlled simulation using an iptables DROP rule to make
|
||||
the remote appear unreachable, triggering the full sequence without taking anything offline.
|
||||
A safety trap removes the rule on any exit — crash, error, ctrl-c, or clean completion.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE TWO-SERVER SETUP ━━━
|
||||
|
||||
```
|
||||
HOST1 — unRAID-Gmer4Lfe
|
||||
Hardware: Threadripper 1950X, 128GB RAM, ZFS cache pools
|
||||
Location: Primary site
|
||||
DDNS: Gmer4Lfe.com
|
||||
Role: Primary — full service stack + source of truth for Movies/Shows/Music
|
||||
|
||||
HOST2 — unRAID-Jayred365
|
||||
Hardware: Intel i5 10th gen, 64GB RAM
|
||||
Location: Remote — 50 miles away
|
||||
DDNS: Gmer4Lfe.us
|
||||
Role: Secondary — own stack + covers HOST1 + mirrors critical data
|
||||
```
|
||||
|
||||
**Hardware does not need to match.** Everything is accessed through `/mnt/user/` — unRAID's
|
||||
fused share layer. HOST1 has a Threadripper with ZFS pools. HOST2 has completely different
|
||||
hardware. Fallback containers on HOST2 mount `/mnt/user/Movies` and see mirrored data
|
||||
because the share names match. The hardware underneath is irrelevant.
|
||||
|
||||
**What must match between servers:**
|
||||
|
||||
```
|
||||
Share names /mnt/user/Movies must exist on both servers (mirrored data)
|
||||
Container names "Emby" on HOST2 must be the container HOST2 starts for HOST1
|
||||
Network names Docker custom networks must match for NPM routing to work
|
||||
```
|
||||
|
||||
### Split Source of Truth — No Conflicts
|
||||
|
||||
Both servers run arr instances simultaneously with zero conflict — they manage completely
|
||||
different shares:
|
||||
|
||||
```
|
||||
HOST1 owns: Movies (Radarr), Tv_Shows (Sonarr), Music (Lidarr)
|
||||
HOST2 owns: Anime_Movies (his Radarr), Anime_Shows (his Sonarr)
|
||||
|
||||
Each server mirrors the other's shares continuously via rsync.
|
||||
```
|
||||
|
||||
The rule: never run two arr instances against the same share simultaneously. Different arrs
|
||||
managing different shares is fine. When HOST2 runs HOST1's arrs during a Tier 4 fallback,
|
||||
HOST1's Tdarr is not running (HOST1 is down) — no conflict.
|
||||
|
||||
### Auth Stack — Warm on Both Servers
|
||||
|
||||
NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running
|
||||
to serve his own users daily — this is not a fallback-only configuration. HOST1 is source of
|
||||
truth: all changes mirror to HOST2 every 15 minutes via critical sync.
|
||||
|
||||
When DNS cuts over, auth is already running on the covering server. The 30-60 second dead
|
||||
zone where auth is coming up after DNS has already switched does not exist.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW IT WORKS ━━━
|
||||
|
||||
Both servers run `fallback.sh` independently as a continuous background process. Each server
|
||||
makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:
|
||||
|
||||
```bash
|
||||
ping REMOTE_TAILSCALE_IP # is the other server reachable?
|
||||
ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8)
|
||||
```
|
||||
|
||||
No SSH signaling between servers. No shared state file. No election algorithm. Each server
|
||||
acts entirely from its own network perspective.
|
||||
|
||||
**States:**
|
||||
|
||||
| State | Remote | Internet | Action |
|
||||
|-------|--------|----------|--------|
|
||||
| NORMAL | up | up | Silent — own containers, own DDNS on |
|
||||
| FALLBACK | down | up | Start tier containers, cut DDNS over |
|
||||
| NO_INTERNET | — | down | Stop own DDNS immediately, wait |
|
||||
| DARK | down | down | Same as NO_INTERNET — cannot determine cause |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ DDNS — THE CRITICAL PART ━━━
|
||||
|
||||
> **This took a year to get right. Do not change the sequencing.**
|
||||
|
||||
`fallback.sh` is the sole authority over when any DDNS container starts. Network state
|
||||
returning is not permission to start DDNS. Only completion of the full handback sequence
|
||||
grants that permission.
|
||||
|
||||
```
|
||||
ONE DOMAIN → ONE DDNS ACTIVE → AT ALL TIMES
|
||||
|
||||
Gmer4Lfe.com → HOST1's DDNS normally → HOST2's DDNS during HOST1 outage
|
||||
Gmer4Lfe.us → HOST2's DDNS normally → HOST1's DDNS during HOST2 outage
|
||||
|
||||
Own DDNS: ON when this server has internet. OFF when internet is lost.
|
||||
Remote DDNS: ON as Tier 1 fallback action. OFF as FIRST handback action.
|
||||
Auto-start: NEVER — DDNS never starts automatically on internet return.
|
||||
```
|
||||
|
||||
**Why auto-start is forbidden:** If HOST1 lost internet and its DDNS auto-started when
|
||||
internet returned, there is a window where both servers are updating the same domain with
|
||||
different IPs. Users get routed randomly — some to the primary with fresh data, some to
|
||||
the covering server. Authentication sessions don't transfer between servers. This is split
|
||||
brain and produces the most confusing symptoms: intermittent auth failures with no clear
|
||||
error state anywhere.
|
||||
|
||||
The brief gap where neither DDNS is updating the record is intentional. DNS TTL caches the
|
||||
last value. During the rsync + container start window, cached DNS still routes users to the
|
||||
covering server where containers are still running. By the time the cache expires, the
|
||||
primary's DDNS has started and the record points at the right server.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ TIERED FALLBACK ━━━
|
||||
|
||||
Starting the full stack for a 5-minute power blip wastes resources — most brief outages
|
||||
resolve before Tier 2 would even activate. Tiers start only what is needed for the actual
|
||||
outage duration.
|
||||
|
||||
| Tier | Delay | Coverage | Why This Timing |
|
||||
|------|-------|----------|-----------------|
|
||||
| Tier 1 | Immediate | Vital services + Live TV | People are watching — cannot wait 4 hours |
|
||||
| Tier 2 | 4hr (HOST*_TIER2_DELAY) | NextCloud, Immich, Jellyseerr | 4hr covers most ISP and power events |
|
||||
| Tier 3 | 12hr (HOST*_TIER3_DELAY) | Dashboard, AdGuard, Git, Collabora | Secondary — useful but not daily-critical |
|
||||
| Tier 4 | 24hr (HOST*_TIER4_DELAY) | Arrs + downloaders | Significant I/O — only worth starting at 24hr |
|
||||
|
||||
Tier 1 always includes the remote domain's DDNS container as the first entry — DNS
|
||||
coverage happens before any other container starts.
|
||||
|
||||
For the actual container lists and tier delay values, see `Manual-Fallback.md`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HANDBACK SEQUENCE ━━━
|
||||
|
||||
When the remote server returns after a FALLBACK event. Every step has a reason.
|
||||
Do not reorder.
|
||||
|
||||
1. **Strike confirmation** — FALLBACK_HANDBACK_STRIKES consecutive remote-up checks before
|
||||
handback begins. Prevents false triggers from brief network recovery.
|
||||
|
||||
2. **Pre-flight checks** — version parity, remote array mounted, remote Docker daemon up.
|
||||
Any failure aborts and retries next cycle.
|
||||
|
||||
3. **Staged reverse handback: Tier 4 → 3 → 2** — Emby and vital services stay on the
|
||||
covering server serving users throughout this phase. Each tier: stop local containers
|
||||
→ rsync writeback (if outage exceeded tier threshold) → start on remote.
|
||||
|
||||
4. **DDNS handoff** — stop remote DDNS immediately before Tier 1 goes down. This prevents
|
||||
split brain during the Tier 1 rsync window.
|
||||
|
||||
5. **Tier 1 handback** — stop local vital services, rsync writeback, start on remote.
|
||||
|
||||
6. **Start remote DDNS last** — DNS cuts back to the primary only after all containers
|
||||
are confirmed running.
|
||||
|
||||
7. **Return to NORMAL** — state file reset, own DDNS restored if it was stopped.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ MUTUAL FALLBACK — BOTH DIRECTIONS ━━━
|
||||
|
||||
The same `fallback.sh` handles both directions without any code changes. `detect_hosts()`
|
||||
determines which server is local and which is remote at runtime, then selects the correct
|
||||
container arrays and tier delays from config via MY_ID.
|
||||
|
||||
```
|
||||
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in master_host2.conf)
|
||||
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in master_host1.conf)
|
||||
```
|
||||
|
||||
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
|
||||
comparisons anywhere in the script code.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ INDEPENDENCE — ALWAYS ONE RSYNC STOP AWAY ━━━
|
||||
|
||||
HOST2 is designed to be fully independent if needed. If HOST2 ever wants to separate from
|
||||
HOST1: stop HOST1 pushing data. Any changes HOST2 makes to his own data stick permanently.
|
||||
His server becomes fully independent immediately — no script changes, no migration, no data
|
||||
movement required. The fallback and rsync scripts are configuration-driven.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|--------------|
|
||||
| `fallback.sh` | Continuous state machine — monitors remote, manages fallback and handback | Continuously (started by `array_start.sh`) |
|
||||
| `fallback_test.sh` | 7-phase test harness — validates the entire fallback lifecycle via iptables simulation | On demand — maintenance window only |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
array_start.sh
|
||||
│
|
||||
└── starts fallback.sh (continuous loop)
|
||||
│
|
||||
├── Every FALLBACK_CHECK_INTERVAL seconds:
|
||||
│ ping remote, ping internet
|
||||
│ → determine state → act on containers + DDNS
|
||||
│
|
||||
└── Test path:
|
||||
│
|
||||
fallback_test.sh
|
||||
│
|
||||
├── Phase 1: pre-flight — both servers ready
|
||||
├── Phase 2: iptables DROP rule → remote appears down
|
||||
├── Phase 3: wait for fallback.sh to detect → FALLBACK state
|
||||
├── Phase 4: verify Tier 1 containers started locally
|
||||
├── Phase 5: remove DROP rule → remote reachable again
|
||||
├── Phase 6: wait for fallback.sh to complete handback → NORMAL
|
||||
└── Phase 7: verify Tier 1 containers stopped locally
|
||||
```
|
||||
|
||||
`fallback_test.sh` contains no fallback logic. It exercises the real `fallback.sh` through
|
||||
connectivity manipulation. Any change to `fallback.sh` is automatically reflected in the
|
||||
test result.
|
||||
+89
-42
@@ -1,54 +1,101 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Failover Test ==============================================
|
||||
# ================================= Fallback Test ==============================================
|
||||
# ==============================================================================================
|
||||
# Controlled simulation of the failover lifecycle — validates the entire failover sequence
|
||||
# without waiting for a real outage.
|
||||
#
|
||||
# ── WHAT THIS SCRIPT IS ───────────────────────────────────────────────────────────────────────
|
||||
# A test harness only — contains no failover logic.
|
||||
# All failover logic lives in fallback.sh and is exercised by this test.
|
||||
# Any changes to fallback.sh are automatically reflected here.
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Controlled simulation of the fallback lifecycle. Validates the entire sequence
|
||||
# without waiting for a real outage. Contains no fallback logic — exercises the
|
||||
# real fallback.sh via an iptables DROP rule on the remote Tailscale IP.
|
||||
#
|
||||
# ── TEST SEQUENCE ─────────────────────────────────────────────────────────────────────────────
|
||||
# Phase 1 — Pre-flight verify both servers reachable, daemons healthy,
|
||||
# version parity, fallback.sh exists, state is NORMAL
|
||||
# Phase 2 — Block Remote iptables rule drops all traffic to remote IP
|
||||
# Phase 3 — Fallback Detection wait for fallback.sh to detect outage and enter FALLBACK
|
||||
# Phase 4 — Container Start verify Tier 1 failover containers started locally
|
||||
# Phase 5 — Restore remove iptables rule, remote becomes reachable
|
||||
# Phase 6 — Handback wait for fallback.sh to complete handback to NORMAL
|
||||
# Phase 7 — Container Handback verify Tier 1 containers stopped locally after handback
|
||||
# Phase 8 — Report full pass/fail summary per phase
|
||||
# Run during a maintenance window. Users will experience a brief service
|
||||
# interruption. Use --dry-run to walk through all phases without real changes.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# FALLBACK_ENABLED gate — aborts if fallback monitoring is disabled
|
||||
# iptables safety trap — rule ALWAYS removed on exit (crash, error, ctrl-c, normal)
|
||||
# remote connectivity always restored regardless of outcome
|
||||
# Version parity check — pre-flight verifies both servers on compatible unRAID versions
|
||||
# Remote Docker daemon — pre-flight verifies remote daemon is responsive
|
||||
# DOCKER_TIMEOUT — all docker calls protected against daemon hangs
|
||||
# MY_ID-based routing — tier containers selected via MY_ID not hostname comparison
|
||||
# Command validation — iptables and notify validated before use
|
||||
# Dry-run safe — full sequence walkthrough without touching iptables or containers
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── WARNING ───────────────────────────────────────────────────────────────────────────────────
|
||||
# ⚠️ This script starts and stops REAL containers on both servers.
|
||||
# Run during a maintenance window — users will experience a brief service interruption.
|
||||
# Use --dry-run to walk through the sequence without any real changes.
|
||||
# Phase 1 — Pre-flight Both servers reachable, Docker daemons healthy,
|
||||
# version parity, fallback.sh exists, state NORMAL
|
||||
# Phase 2 — Block Remote iptables DROP rule added — remote appears unreachable
|
||||
# Phase 3 — Fallback Detection Wait FALLBACK_TEST_BLOCK_WAIT for fallback.sh to
|
||||
# detect the outage and enter FALLBACK state
|
||||
# Phase 4 — Container Start Verify Tier 1 containers started locally
|
||||
# Phase 5 — Restore iptables rule removed — remote reachable again
|
||||
# Phase 6 — Handback Wait FALLBACK_TEST_HANDBACK_WAIT for fallback.sh to
|
||||
# complete full handback and return to NORMAL
|
||||
# Phase 7 — Container Handback Verify Tier 1 containers stopped locally
|
||||
# Report — Full pass/fail per phase with timing
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# FALLBACK_TEST_BLOCK_WAIT — seconds to wait for fallback.sh to detect outage
|
||||
# FALLBACK_TEST_HANDBACK_WAIT — seconds to wait for fallback.sh to complete handback
|
||||
# FALLBACK_CHECK_INTERVAL — check interval of the running fallback.sh (informational)
|
||||
# FALLBACK_HANDBACK_STRIKES — strikes required before handback (informational)
|
||||
# FALLBACK_STATE_FILE — state file path to read current state
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Test Harness Only
|
||||
# Contains zero fallback logic. All fallback is exercised through fallback.sh.
|
||||
# Any change to fallback.sh is automatically reflected in the test result.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# iptables Safety Trap
|
||||
# The DROP rule is removed via trap on ANY exit — normal completion, crash, error,
|
||||
# ctrl-c. Remote connectivity is always restored regardless of test outcome.
|
||||
# You cannot accidentally leave the remote permanently blocked.
|
||||
#
|
||||
# FALLBACK_ENABLED Gate
|
||||
# Aborts if FALLBACK_ENABLED=false. Testing a disabled fallback system is
|
||||
# misleading and potentially destructive.
|
||||
#
|
||||
# State Must Be NORMAL
|
||||
# Pre-flight fails if state is not NORMAL. Running a test during an actual
|
||||
# fallback event would interfere with the real event.
|
||||
#
|
||||
# Version Parity Check
|
||||
# Pre-flight verifies unRAID version parity before any iptables rules are
|
||||
# added. A mismatch makes the test result unreliable.
|
||||
#
|
||||
# Remote Docker Daemon Check
|
||||
# Pre-flight confirms remote Docker daemon is responsive before Phase 2.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# FALLBACK_TEST_BLOCK_WAIT
|
||||
# Seconds to wait in Phase 3 for fallback.sh to detect the outage.
|
||||
# Must be > FALLBACK_CHECK_INTERVAL + buffer. At 30s interval: use ≥60s.
|
||||
# (default: 60)
|
||||
#
|
||||
# FALLBACK_TEST_HANDBACK_WAIT
|
||||
# Seconds to wait in Phase 6 for fallback.sh to complete handback.
|
||||
# Must cover: FALLBACK_HANDBACK_STRIKES × FALLBACK_CHECK_INTERVAL + rsync
|
||||
# duration + container start time. At 3 strikes × 30s + ~2min rsync +
|
||||
# ~1min container start: use ≥240s. (default: 300)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# fallback_test.sh --dry-run
|
||||
# Walk through all 7 phases with output but no iptables changes and no
|
||||
# container starts/stops. ALWAYS run this before a live test.
|
||||
#
|
||||
# fallback_test.sh
|
||||
# Full live test — real iptables DROP rule, real container lifecycle.
|
||||
# Users will experience a brief service interruption. Run during a
|
||||
# maintenance window.
|
||||
#
|
||||
# fallback_test.sh --status
|
||||
# Show current fallback state and test timing configuration. No test run.
|
||||
#
|
||||
# fallback_test.sh --log
|
||||
# Verbose output on every check in every phase.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# fallback_test.sh — run full test sequence
|
||||
# fallback_test.sh --dry-run — walk through all phases without changes
|
||||
# fallback_test.sh --status — show current fallback state and test config
|
||||
# fallback_test.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user