656 lines
20 KiB
Markdown
656 lines
20 KiB
Markdown
# ━━━━━ FALLBACK — Manual ━━━━━
|
||
|
||
Config reference, procedures, operational workflows.
|
||
For overview see README-Fallback.md. For per-script detail see script headers.
|
||
|
||
---
|
||
|
||
## ━━━ CONFIGURATION — master.conf ━━━
|
||
|
||
```
|
||
FALLBACK_ENABLED=true
|
||
```
|
||
Enable or disable the entire fallback system. `false` = exit cleanly on startup — no
|
||
monitoring, no container actions. Set false when a server is being rebuilt or fallback
|
||
is temporarily suspended. Must be explicitly enabled when both servers are ready.
|
||
**(default: false)**
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_CHECK_INTERVAL=30
|
||
```
|
||
Seconds between connectivity checks (ping remote + ping internet). Shorter = faster
|
||
detection and more pings. With `FALLBACK_HANDBACK_STRIKES=3` at 30s intervals:
|
||
fallback detected in ≤30s, handback requires 90s of continuous remote-up before
|
||
beginning. **(default: 30)**
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_HANDBACK_STRIKES=3
|
||
```
|
||
Consecutive remote-up checks required before handback begins. Prevents false triggers
|
||
from brief network recovery during an ongoing outage. 3 strikes × 30s = 90s continuous
|
||
up required before handback sequence starts. **(default: 3)**
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
|
||
```
|
||
Path to the persistent state file. In `$STATE_DIR` — survives reboots whether storage
|
||
mode is internal (boot device) or appdata (array). If the server was in FALLBACK state
|
||
when it rebooted, it resumes FALLBACK on restart rather than assuming everything is normal.
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_RSYNC_ENABLED=true
|
||
```
|
||
Gate for writeback rsync jobs during handback. Set `false` to hand back without syncing
|
||
any data — useful when the primary still has reliable last-known-good state and writeback
|
||
would be counterproductive. **(default: true)**
|
||
|
||
---
|
||
|
||
```
|
||
EXTERNAL_IP=8.8.8.8
|
||
```
|
||
IP pinged to verify internet connectivity. **(default: 8.8.8.8)**
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_TEST_BLOCK_WAIT=60
|
||
```
|
||
Seconds fallback_test.sh waits in Phase 3 for fallback.sh to detect the simulated
|
||
outage. Must be greater than `FALLBACK_CHECK_INTERVAL` plus a buffer. At 30s interval:
|
||
use ≥60s. **(default: 60)**
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_TEST_HANDBACK_WAIT=300
|
||
```
|
||
Seconds fallback_test.sh waits in Phase 6 for fallback.sh to complete the full handback
|
||
sequence. Must cover: strike confirmation window + pre-flight time + rsync duration +
|
||
container start time. At 3 strikes × 30s + ~2min rsync + ~1min container start: use ≥240s.
|
||
**(default: 300)**
|
||
|
||
---
|
||
|
||
## ━━━ CONFIGURATION — host*.conf ━━━
|
||
|
||
```
|
||
HOST*_DDNS_CONTAINERS=("...")
|
||
```
|
||
DDNS containers managed by this host — the containers that update this server's DNS
|
||
records. Stopped when internet is lost. Started as the very last step of handback after
|
||
all primary containers are confirmed running.
|
||
|
||
```bash
|
||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||
```
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_HOST*_STOP_ON_NO_NET=(...)
|
||
```
|
||
Containers to stop when this host loses internet. Services that are meaningless without
|
||
internet connectivity. Most hosts leave this empty.
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_HOST*_TIER1=(...)
|
||
FALLBACK_HOST*_TIER2=(...)
|
||
FALLBACK_HOST*_TIER3=(...)
|
||
FALLBACK_HOST*_TIER4=(...)
|
||
```
|
||
|
||
**Each host declares its OWN services, in its OWN conf.** `FALLBACK_HOST2_TIER1` lives in
|
||
`host2.conf` and lists HOST2's vital containers — it is not a list HOST1 maintains.
|
||
|
||
At runtime `fallback.sh` reads its **partner's** list:
|
||
|
||
```bash
|
||
get_tier_containers() {
|
||
local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}" # note: REMOTE_ID, not MY_ID
|
||
```
|
||
|
||
So HOST1, covering HOST2, reads `FALLBACK_HOST2_TIER1` — a variable defined in `host2.conf`
|
||
and delivered to HOST1 through the partner conf cache (`conf_sync.sh`), because sparse
|
||
checkout means HOST1 never pulls `host2.conf` from git.
|
||
|
||
This is why the naming is what it is. The alternative — each host keeping a copy of its
|
||
partner's container list — would need editing on both machines every time either one changed
|
||
a service, and the two copies would silently diverge. Declaring once, on the host that owns
|
||
the services, means a host is always the authority on what covering it requires.
|
||
|
||
TIER1 starts immediately. TIER2–4 activate after their delay thresholds
|
||
(`HOST*_TIER2_DELAY` and friends, also in that host's own conf).
|
||
|
||
The DDNS container for that host's domain must be the **first entry in TIER1** — DNS coverage
|
||
before anything else.
|
||
|
||
---
|
||
|
||
```
|
||
HOST*_TIER2_DELAY=240
|
||
HOST*_TIER3_DELAY=720
|
||
HOST*_TIER4_DELAY=1440
|
||
```
|
||
Minutes after FALLBACK state entry before activating each tier. Variable references the
|
||
**remote** host's ID — delays for a HOST1 outage use `HOST1_TIER*_DELAY`.
|
||
|
||
```bash
|
||
HOST1_TIER2_DELAY=240 # 4 hours — covers most ISP and power issues
|
||
HOST1_TIER3_DELAY=720 # 12 hours — genuine extended outage
|
||
HOST1_TIER4_DELAY=1440 # 24 hours — arrs only worth starting at this threshold
|
||
```
|
||
|
||
---
|
||
|
||
```
|
||
HOST*_TIER1_WRITEBACK_DELAY=60
|
||
```
|
||
Minimum outage duration in minutes before Tier 1 writeback runs on handback. Outages
|
||
shorter than this skip writeback entirely — the primary's last-known-good state is more
|
||
reliable than a short period of activity on the covering server.
|
||
|
||
```bash
|
||
HOST1_TIER1_WRITEBACK_DELAY=60 # skip writeback for outages under 60 minutes
|
||
```
|
||
|
||
---
|
||
|
||
```
|
||
FALLBACK_HOST*_WRITEBACK_TIER1=(...)
|
||
FALLBACK_HOST*_WRITEBACK_TIER2=(...)
|
||
FALLBACK_HOST*_WRITEBACK_TIER3=(...)
|
||
```
|
||
Paths rsynced back to the remote on handback, per tier. Writeback runs only if the
|
||
outage exceeded the tier's activation delay.
|
||
|
||
Variable pattern: `FALLBACK_${REMOTE_ID}_WRITEBACK_TIER${N}`
|
||
|
||
**Tier 4 writeback** auto-uses the remote's `HOST*_DAILY_SYNC_SHARES` — the same list
|
||
daily_sync_maintenance.sh uses, in the opposite direction. No separate TIER4 list needed.
|
||
|
||
---
|
||
|
||
## ━━━ FULL CONFIGURATION REFERENCE BLOCKS ━━━
|
||
|
||
### master.conf
|
||
|
||
```bash
|
||
FALLBACK_ENABLED=true
|
||
FALLBACK_CHECK_INTERVAL=30
|
||
FALLBACK_HANDBACK_STRIKES=3
|
||
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
|
||
FALLBACK_RSYNC_ENABLED=true
|
||
EXTERNAL_IP=8.8.8.8
|
||
FALLBACK_TEST_BLOCK_WAIT=60
|
||
FALLBACK_TEST_HANDBACK_WAIT=300
|
||
```
|
||
|
||
---
|
||
|
||
### host1.conf — HOST1's own services (started by HOST2 when HOST1 is down)
|
||
|
||
```bash
|
||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||
|
||
FALLBACK_HOST1_STOP_ON_NO_NET=()
|
||
|
||
# Tier 1 — immediate (vital services + Live TV)
|
||
FALLBACK_HOST1_TIER1=(
|
||
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
|
||
"Emby" # media server — people are watching
|
||
"NginxProxyManager" # reverse proxy — all external access routes through this
|
||
"Lldap" # user directory — already warm, verify and keep
|
||
"Mariadb-Authelia" # auth database — already warm, verify and keep
|
||
"Redis-Authelia" # auth session cache — already warm, verify and keep
|
||
"Authelia" # SSO — already warm, serving users already
|
||
"VaultWarden" # passwords — people lock themselves out without this
|
||
"Dispatcharr" # Live TV scheduler — people are watching right now
|
||
"Dispatcharr-Basic"
|
||
"Dispatcharr-Iptv-Users"
|
||
"ErsatzTV-Emby" # channel schedule builder
|
||
)
|
||
|
||
# Tier 2 — after 4 hours (shared productivity services)
|
||
FALLBACK_HOST1_TIER2=(
|
||
"Postgres-NextCloud" # must start before NextCloud
|
||
"NextCloud"
|
||
"PostgreSQL-Immich" # must start before Immich
|
||
"Immich-Gmer4Lfe"
|
||
"Jellyseerr"
|
||
)
|
||
|
||
# Tier 3 — after 12 hours (secondary services)
|
||
FALLBACK_HOST1_TIER3=(
|
||
"Organizrv2-Gmer4Lfe"
|
||
"AdGuard-Home"
|
||
"UptimeKuma"
|
||
"Gitea"
|
||
"Collabora-CODE"
|
||
)
|
||
|
||
# Tier 4 — after 24 hours (arrs + downloaders)
|
||
FALLBACK_HOST1_TIER4=(
|
||
"Sonarr-Gmer4Lfe"
|
||
"Radarr-Gmer4Lfe"
|
||
"Lidarr-Gmer4Lfe"
|
||
"Prowlarr-Gmer4Lfe"
|
||
"SABnzbd-Gmer4Lfe"
|
||
"qBittorrent-Gmer4Lfe"
|
||
"LidaTube"
|
||
"Pinchflat"
|
||
)
|
||
|
||
# Tier delays for HOST1 outage
|
||
HOST1_TIER2_DELAY=240
|
||
HOST1_TIER3_DELAY=720
|
||
HOST1_TIER4_DELAY=1440
|
||
HOST1_TIER1_WRITEBACK_DELAY=60
|
||
|
||
# Writeback paths — synced back to HOST1 on handback
|
||
FALLBACK_HOST1_WRITEBACK_TIER1=(
|
||
"/mnt/user/appdata-Fallback/Critical-Data" # auth stack — Authelia + NPM + certs
|
||
"/mnt/user/Media_Server/Emby" # Emby userdata — watch history, playstates
|
||
)
|
||
|
||
FALLBACK_HOST1_WRITEBACK_TIER2=(
|
||
"/mnt/user/appdata-Fallback/Important-Data" # NextCloud + Postgres + Immich
|
||
)
|
||
|
||
FALLBACK_HOST1_WRITEBACK_TIER3=(
|
||
"/mnt/user/appdata-Fallback/Gmer4Lfe" # secondary appdata accumulated changes
|
||
)
|
||
|
||
# Tier 4 writeback uses HOST1_DAILY_SYNC_SHARES automatically — no list needed here.
|
||
```
|
||
|
||
---
|
||
|
||
### host2.conf — HOST2's own services (started by HOST1 when HOST2 is down)
|
||
|
||
```bash
|
||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||
|
||
FALLBACK_HOST2_STOP_ON_NO_NET=()
|
||
|
||
# Tier 1 — immediate (HOST2's vital services)
|
||
FALLBACK_HOST2_TIER1=(
|
||
"Gmer4Lfe.us-DDNS" # ALWAYS FIRST
|
||
# HOST2's Tier 1 services — fill per HOST2's stack
|
||
)
|
||
|
||
FALLBACK_HOST2_TIER2=(...)
|
||
FALLBACK_HOST2_TIER3=(...)
|
||
FALLBACK_HOST2_TIER4=(...)
|
||
|
||
# Tier delays for HOST2 outage
|
||
HOST2_TIER2_DELAY=240
|
||
HOST2_TIER3_DELAY=720
|
||
HOST2_TIER4_DELAY=1440
|
||
HOST2_TIER1_WRITEBACK_DELAY=60
|
||
|
||
FALLBACK_HOST2_WRITEBACK_TIER1=(
|
||
# HOST2's critical data paths
|
||
)
|
||
|
||
# Tier 4 writeback uses HOST2_DAILY_SYNC_SHARES automatically.
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ STATE FILE REFERENCE ━━━
|
||
|
||
Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata)
|
||
|
||
> In a shell where load_config.sh is not sourced, use the full path:
|
||
> `/boot/config/plugins/varaverk/data/state/fallback_state.db` (internal storage mode)
|
||
|
||
```
|
||
state=NORMAL # NORMAL | FALLBACK | NO_INTERNET | DARK
|
||
fallback_start=0 # epoch timestamp when FALLBACK began (0 = not in FALLBACK)
|
||
handback_strikes=0 # consecutive remote-up checks accumulated toward handback
|
||
tier2_started=false # whether Tier 2 containers started this event
|
||
tier3_started=false # whether Tier 3 containers started
|
||
tier4_started=false # whether Tier 4 containers started
|
||
```
|
||
|
||
View state: `fallback.sh --status` (preferred — parsed output)
|
||
Raw file: `cat "$STATE_DIR/fallback_state.db"` (requires STATE_DIR set, or use full path)
|
||
|
||
The file is managed exclusively by fallback.sh. Do not edit it while fallback.sh is
|
||
running — the next cycle will overwrite your changes. Use the Manual State Reset
|
||
procedure (below) when fallback.sh is not running.
|
||
|
||
---
|
||
|
||
## ━━━ INITIAL SETUP ━━━
|
||
|
||
Before fallback.sh can work on both servers:
|
||
|
||
### 1. Tailscale Connected on Both Servers
|
||
|
||
```bash
|
||
# Verify from HOST1
|
||
tailscale ip -4 unRAID-Jayred365
|
||
tailscale ping unRAID-Jayred365
|
||
|
||
# Verify from HOST2
|
||
tailscale ip -4 unRAID-Gmer4Lfe
|
||
tailscale ping unRAID-Gmer4Lfe
|
||
```
|
||
|
||
### 2. SSH Keys Configured — No Password Prompt
|
||
|
||
```bash
|
||
# From HOST1 — should print HOST2's hostname without password prompt
|
||
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname"
|
||
|
||
# From HOST2 — should print HOST1's hostname
|
||
ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname"
|
||
```
|
||
|
||
### 3. Container Names Match
|
||
|
||
Fallback containers must exist (but stopped) on the covering server, with volume mounts
|
||
pointing at mirrored share paths.
|
||
|
||
```bash
|
||
# Verify on HOST2 that HOST1's container exists (stopped is expected)
|
||
ssh root@[HOST2-tailscale-ip] "docker inspect Emby --format '{{.State.Status}}'"
|
||
# Expected: created or exited — NOT "no such container"
|
||
```
|
||
|
||
### 4. Critical Data Mirrored
|
||
|
||
These shares must exist on HOST2 with current data from HOST1 before fallback is needed:
|
||
|
||
```
|
||
/mnt/user/appdata-Fallback/Critical-Data # auth stack — NPM, LLDAP, Authelia, certs
|
||
/mnt/user/appdata-Fallback/Important-Data # NextCloud + Postgres + Immich
|
||
/mnt/user/Media_Server/Emby # Emby userdata — watch history, playstates
|
||
/mnt/user/appdata-Fallback/Gmer4Lfe # server appdata
|
||
```
|
||
|
||
```bash
|
||
# Verify data is current — check modification times
|
||
ssh root@[HOST2-tailscale-ip] "ls -la /mnt/user/appdata-Fallback/Critical-Data/"
|
||
```
|
||
|
||
Sync is maintained continuously by `daily_sync_maintenance.sh` critical-data profile.
|
||
|
||
### 5. DDNS TTL Set to 1 Minute
|
||
|
||
Set in your DDNS provider settings. Higher TTL means users continue hitting the old IP
|
||
for longer after fallback. At 5-minute TTL, users can be hitting a downed server for up
|
||
to 5 minutes before DNS switches.
|
||
|
||
### 6. Both Servers Running fallback.sh
|
||
|
||
Fallback only works in one direction if only one server is running the script. Both
|
||
servers must be running it continuously for mutual coverage.
|
||
|
||
```bash
|
||
# Verify fallback.sh is running
|
||
pgrep -f "fallback.sh"
|
||
|
||
# Check the state file
|
||
cat "$STATE_DIR/fallback_state.db"
|
||
```
|
||
|
||
Start via User Scripts plugin on both servers.
|
||
|
||
---
|
||
|
||
## ━━━ MONITORING ━━━
|
||
|
||
**Sunday coffee report** (`sunday_morning_coffee_report.sh`) — Fallback section shows
|
||
current state, outage duration if not NORMAL, Tailscale reachability, and whether
|
||
fallback.sh is running.
|
||
|
||
**Weekly health digest** (`weekly_health_digest.sh`) — reads the state file. If
|
||
`DIGEST_SMART_ON_FALLBACK=true` and state is not NORMAL, it sends a notification even
|
||
in smart mode — a non-NORMAL state at digest time needs attention.
|
||
|
||
**Direct check:**
|
||
|
||
```bash
|
||
fallback.sh --status # full state snapshot
|
||
cat "$STATE_DIR/fallback_state.db" # raw state file
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ PROCEDURES ━━━
|
||
|
||
### Running the Fallback Test
|
||
|
||
> This starts and stops real containers on both servers. Users will experience a brief
|
||
> service interruption. Always run `--dry-run` first.
|
||
|
||
```bash
|
||
# Step 1 — verify all phases without touching anything
|
||
fallback_test.sh --dry-run
|
||
|
||
# Step 2 — schedule maintenance window, then run live
|
||
fallback_test.sh
|
||
|
||
# Step 3 — check state after test completes
|
||
fallback.sh --status
|
||
cat "$STATE_DIR/fallback_state.db"
|
||
```
|
||
|
||
If the test doesn't complete cleanly, the state file may be left in FALLBACK. The
|
||
iptables safety trap in fallback_test.sh removes the DROP rule on any exit, so remote
|
||
connectivity is always restored. Use the Manual State Reset procedure below if the state
|
||
file is stuck.
|
||
|
||
### Manual State Reset
|
||
|
||
Use when the state file is stuck in a non-NORMAL state after testing, a failed handback,
|
||
or killing fallback.sh directly (not via User Scripts Abort).
|
||
|
||
**Before resetting, verify the situation is actually safe to reset:**
|
||
|
||
```bash
|
||
# 1. Right containers running on the right server
|
||
docker ps | grep -E "Emby|VaultWarden|NginxProxyManager"
|
||
|
||
# 2. DDNS pointing at the correct server
|
||
nslookup Gmer4Lfe.com 8.8.8.8
|
||
nslookup Gmer4Lfe.us 8.8.8.8
|
||
|
||
# 3. Both servers visible on Tailscale
|
||
tailscale ping [remote-tailscale-ip]
|
||
|
||
# 4. No actual fallback in progress (remote is genuinely up and stable)
|
||
ping -c 5 [remote-tailscale-ip]
|
||
```
|
||
|
||
**Stop fallback.sh first (via User Scripts Abort), then reset:**
|
||
|
||
```bash
|
||
# Set STATE_DIR (or source load_config.sh to get it from the environment)
|
||
source /boot/config/plugins/varaverk/load_config.sh
|
||
|
||
# View current state
|
||
cat "$STATE_DIR/fallback_state.db"
|
||
|
||
# Write a clean NORMAL state
|
||
cat > "$STATE_DIR/fallback_state.db" << 'EOF'
|
||
state=NORMAL
|
||
fallback_start=0
|
||
handback_strikes=0
|
||
tier2_started=false
|
||
tier3_started=false
|
||
tier4_started=false
|
||
EOF
|
||
|
||
# Verify the write
|
||
cat "$STATE_DIR/fallback_state.db"
|
||
```
|
||
|
||
Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its next cycle.
|
||
|
||
> **Warning:** Do NOT reset during an actual fallback event. fallback.sh will think
|
||
> everything is normal and stop covering the remote — services go offline until the next
|
||
> detection cycle catches it again.
|
||
|
||
### Adding a Container to a Tier
|
||
|
||
1. Create the container on the covering server (stopped), with volume mounts pointing at
|
||
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
|
||
2. Add the container name to `FALLBACK_<THAT-HOST>_TIER*` in **that host's own** conf
|
||
in the appropriate tier position (dependency ordering — databases before apps)
|
||
3. Verify: `fallback.sh --status` shows the container in the expected tier list
|
||
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
|
||
|
||
---
|
||
|
||
## ━━━ TROUBLESHOOTING ━━━
|
||
|
||
### Fallback Not Triggering
|
||
|
||
```
|
||
Is fallback.sh running on the covering server?
|
||
→ User Scripts plugin → check status of the fallback script
|
||
→ pgrep -f "fallback.sh"
|
||
|
||
Is Tailscale connected?
|
||
→ tailscale status (should show the remote peer)
|
||
|
||
Can this server reach the remote Tailscale IP?
|
||
→ ping [remote-tailscale-ip]
|
||
|
||
What does fallback.sh report?
|
||
→ fallback.sh --status
|
||
→ cat "$STATE_DIR/fallback_state.db"
|
||
```
|
||
|
||
### Handback Not Completing
|
||
|
||
```
|
||
Is the primary's array fully started?
|
||
→ ls /mnt/user (should show share directories)
|
||
|
||
Is Docker responding on the primary?
|
||
→ docker ps (should return a list, not hang)
|
||
|
||
Is rootfs nearly full? (pre-flight checks this)
|
||
→ df /
|
||
|
||
Is rsync running and stuck?
|
||
→ pgrep rsync
|
||
→ A stalled rsync blocks handback. Let fallback.sh retry next cycle.
|
||
```
|
||
|
||
### DDNS Not Cutting Over
|
||
|
||
```
|
||
Is the DDNS container running on the covering server?
|
||
→ docker ps | grep DDNS
|
||
|
||
What TTL is the DNS record set to?
|
||
→ nslookup Gmer4Lfe.com 8.8.8.8 (check TTL in response)
|
||
→ High TTL = slow propagation
|
||
|
||
Is the DDNS provider accepting updates?
|
||
→ docker logs [ddns-container] --tail 50
|
||
```
|
||
|
||
### Split Brain — Both DDNS Running
|
||
|
||
```
|
||
This should not happen if the handback sequence completed correctly.
|
||
If it has happened:
|
||
|
||
1. Check both servers for running DDNS containers
|
||
HOST1: docker ps | grep DDNS
|
||
HOST2: docker ps | grep DDNS
|
||
|
||
2. Stop the duplicate (the one on the covering server)
|
||
docker stop [duplicate-ddns-container]
|
||
|
||
3. Understand the state before resetting
|
||
fallback.sh --status
|
||
cat "$STATE_DIR/fallback_state.db"
|
||
|
||
4. Perform Manual State Reset above on the server in a bad state
|
||
|
||
5. Restart fallback.sh via User Scripts plugin
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ OUTPUT TIERS ━━━
|
||
|
||
Both scripts have two output levels controlled by `--log`.
|
||
|
||
**fallback.sh** — daemon, runs continuously. Without `--log`, only state transitions,
|
||
warnings, errors, and the startup banner are visible. Per-cycle detail (ping results,
|
||
state evaluation) is suppressed — the daemon runs every 15–30 seconds and clean cycles
|
||
produce no output by design. State transitions are always visible because they are the
|
||
events that matter. Use `--log` when debugging why the state machine is or is not
|
||
acting.
|
||
|
||
**fallback_test.sh** — one-shot test harness. Without `--log`, phase headers, pass/fail
|
||
results, and the final test report are always visible. Per-container checks within each
|
||
phase are suppressed. With `--log`, every check in every phase is shown. Warnings and
|
||
errors are always visible regardless of `--log`.
|
||
|
||
---
|
||
|
||
## ━━━ FLAG REFERENCE ━━━
|
||
|
||
### fallback.sh
|
||
|
||
`fallback.sh`
|
||
Normal start — continuous loop. Started automatically by `array_started.sh` at array start.
|
||
**Do NOT stop by killing the process** — state file may be left inconsistent. Stop via
|
||
`fallback.sh --stop` only.
|
||
|
||
`fallback.sh --dry-run`
|
||
Walk through one full cycle showing what would happen based on current network state. No
|
||
containers started or stopped. No DDNS changes. Use to verify configuration before relying
|
||
on it.
|
||
|
||
`fallback.sh --status`
|
||
Show current state, identity, DDNS containers, check interval, handback strikes, and
|
||
(if in FALLBACK) outage duration and tier activation status. Use this first to understand
|
||
current state before any manual intervention.
|
||
|
||
`fallback.sh --log`
|
||
Verbose output on every decision in every cycle — ping results, state evaluation, tier
|
||
checks. Use when debugging why the state machine is or is not acting as expected.
|
||
|
||
---
|
||
|
||
### fallback_test.sh
|
||
|
||
`fallback_test.sh --dry-run`
|
||
Walk through all 7 phases with full output. No iptables rules added. No container
|
||
starts or stops. **Always run this before a live test** — confirms timing configuration
|
||
is correct and phases would pass before committing to real changes.
|
||
|
||
`fallback_test.sh`
|
||
Full live test — real iptables DROP rule, real container lifecycle. Users will experience
|
||
a brief service interruption. Run during a maintenance window. The iptables safety trap
|
||
removes the DROP rule on any exit (normal completion, crash, ctrl-c).
|
||
|
||
`fallback_test.sh --status`
|
||
Show current fallback state, test timing configuration, and the Tier 1 containers that
|
||
would be tested. No test run.
|
||
|
||
`fallback_test.sh --log`
|
||
Verbose output on every check in every phase.
|