rename: failover → fallback throughout codebase — this system is Fallback, state is FALLBACK; preserve conventional failover comparisons

This commit is contained in:
Gmer4Lfe
2026-06-01 20:04:17 -04:00
parent 65f6bd7b8a
commit cb7d7a688d
62 changed files with 85 additions and 85 deletions
View File
+3 -3
View File
@@ -43,7 +43,7 @@
# #
# Version Ownership # Version Ownership
# Fallback containers are excluded from remainder mode. This server only runs # Fallback containers are excluded from remainder mode. This server only runs
# them during a failover. The remote server owns their version — if remainder # them during a fallback. The remote server owns their version — if remainder
# updates them independently and a handback occurs, the remote's older image # updates them independently and a handback occurs, the remote's older image
# may not handle data written by the newer version. # may not handle data written by the newer version.
# #
@@ -72,7 +72,7 @@
# #
# Fallback Exclusion # Fallback Exclusion
# Remainder mode excludes containers owned by the remote server's update cycle # Remainder mode excludes containers owned by the remote server's update cycle
# to prevent version divergence across the failover boundary. # to prevent version divergence across the fallback boundary.
# #
# Running-Only Filter # Running-Only Filter
# Stopped containers excluded from remainder mode — intentionally down. # Stopped containers excluded from remainder mode — intentionally down.
@@ -179,7 +179,7 @@ if [[ "$REMAINDER_MODE" == true ]]; then
unset _weekly_str _weekly_arr unset _weekly_str _weekly_arr
# Fallback coverage containers — owned by the remote server's update cycle. # Fallback coverage containers — owned by the remote server's update cycle.
# This server runs them during failover but should never update them independently. # This server runs them during fallback but should never update them independently.
# Updating them here risks version divergence: if remote's writeback after handback # Updating them here risks version divergence: if remote's writeback after handback
# encounters data written by a newer version, it may not handle it correctly. # encounters data written by a newer version, it may not handle it correctly.
for _tier in 1 2 3 4; do for _tier in 1 2 3 4; do
View File
View File
View File
+4 -4
View File
@@ -352,7 +352,7 @@ ssh root@[HOST2-tailscale-ip] "docker inspect Emby --format '{{.State.Status}}'"
### 4. Critical Data Mirrored ### 4. Critical Data Mirrored
These shares must exist on HOST2 with current data from HOST1 before failover is needed: These shares must exist on HOST2 with current data from HOST1 before fallback is needed:
``` ```
/mnt/user/appdata-Fallback/Critical-Data # auth stack — NPM, LLDAP, Authelia, certs /mnt/user/appdata-Fallback/Critical-Data # auth stack — NPM, LLDAP, Authelia, certs
@@ -371,7 +371,7 @@ Sync is maintained continuously by `daily_sync_maintenance.sh` critical-data pro
### 5. DDNS TTL Set to 1 Minute ### 5. DDNS TTL Set to 1 Minute
Set in your DDNS provider settings. Higher TTL means users continue hitting the old IP Set in your DDNS provider settings. Higher TTL means users continue hitting the old IP
for longer after failover. At 5-minute TTL, users can be hitting a downed server for up for longer after fallback. At 5-minute TTL, users can be hitting a downed server for up
to 5 minutes before DNS switches. to 5 minutes before DNS switches.
### 6. Both Servers Running fallback.sh ### 6. Both Servers Running fallback.sh
@@ -398,7 +398,7 @@ current state, outage duration if not NORMAL, Tailscale reachability, and whethe
fallback.sh is running. fallback.sh is running.
**Weekly health digest** (`weekly_health_digest.sh`) — reads the state file. If **Weekly health digest** (`weekly_health_digest.sh`) — reads the state file. If
`DIGEST_SMART_ON_FAILOVER=true` and state is not NORMAL, it sends a notification even `DIGEST_SMART_ON_FALLBACK=true` and state is not NORMAL, it sends a notification even
in smart mode — a non-NORMAL state at digest time needs attention. in smart mode — a non-NORMAL state at digest time needs attention.
**Direct check:** **Direct check:**
@@ -412,7 +412,7 @@ cat /boot/config/fallback_state.db # raw state file
## ━━━ PROCEDURES ━━━ ## ━━━ PROCEDURES ━━━
### Running the Failover Test ### Running the Fallback Test
> This starts and stops real containers on both servers. Users will experience a brief > This starts and stops real containers on both servers. Users will experience a brief
> service interruption. Always run `--dry-run` first. > service interruption. Always run `--dry-run` first.
+2 -2
View File
@@ -1,6 +1,6 @@
# ━━━━━ FALLBACK ━━━━━ # ━━━━━ FALLBACK ━━━━━
Mutual automatic failover between two independent unRAID servers. When one goes down the Mutual automatic fallback between two independent unRAID servers. When one goes down the
other starts its containers, cuts over DNS, and keeps users online. When it comes back other starts its containers, cuts over DNS, and keeps users online. When it comes back
everything hands back in the correct sequence — covering DDNS stops, containers stop, rsync everything hands back in the correct sequence — covering DDNS stops, containers stop, rsync
writeback runs, containers start on the primary, primary DDNS starts last — so users hit the writeback runs, containers start on the primary, primary DDNS starts last — so users hit the
@@ -45,7 +45,7 @@ The fix: stop containers before syncing. The outage window is only the rsync dur
typically minutes. Clean static source at full bandwidth, predictable state every time. typically minutes. Clean static source at full bandwidth, predictable state every time.
**No Way to Validate the System Before Needing It** **No Way to Validate the System Before Needing It**
A failover system that has never been tested is not a failover system — it is a hope. A fallback system that has never been tested is not a fallback system — it is a hope.
The fix: `fallback_test.sh` — a controlled simulation using an iptables DROP rule to make The fix: `fallback_test.sh` — a controlled simulation using an iptables DROP rule to make
the remote appear unreachable, triggering the full sequence without taking anything offline. the remote appear unreachable, triggering the full sequence without taking anything offline.
A safety trap removes the rule on any exit — crash, error, ctrl-c, or clean completion. A safety trap removes the rule on any exit — crash, error, ctrl-c, or clean completion.
Regular → Executable
View File
Regular → Executable
View File
+2 -2
View File
@@ -415,7 +415,7 @@ cat /boot/config/fallback_state.db
--- ---
## ━━━ STEP 8: TEST FAILOVER ━━━ ## ━━━ STEP 8: TEST FALLBACK ━━━
Before relying on the system, confirm it actually triggers. `fallback_test.sh` Before relying on the system, confirm it actually triggers. `fallback_test.sh`
simulates an outage using `iptables` — no real downtime, no real data changes. simulates an outage using `iptables` — no real downtime, no real data changes.
@@ -601,7 +601,7 @@ The remote server is unreachable on the first check. Common causes:
```bash ```bash
# Confirm which state fallback is in # Confirm which state fallback is in
cat /boot/config/fallback_state.db cat /boot/config/fallback_state.db
# If stuck in FAILOVER after remote comes back: reset state # If stuck in FALLBACK after remote comes back: reset state
bash Tools/fallback_state_reset.sh bash Tools/fallback_state_reset.sh
``` ```
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
+1 -1
View File
@@ -83,7 +83,7 @@ drive SMART attributes, ZFS pool state, ARC statistics, kernel memory pressure.
Monitors/ ← observes and reports (this folder) Monitors/ ← observes and reports (this folder)
Docker_Essentials/ ← acts on containers (docker_watchdog starts/stops) Docker_Essentials/ ← acts on containers (docker_watchdog starts/stops)
unRAID_Essentials/ ← acts on the server (stability_watchdog, inotify_tuning) unRAID_Essentials/ ← acts on the server (stability_watchdog, inotify_tuning)
Fallback/ ← acts on the full stack (failover, handback) Fallback/ ← acts on the full stack (fallback, handback)
Rsync/ ← calls bandwidth_monitor.sh (auto-logs each sync) Rsync/ ← calls bandwidth_monitor.sh (auto-logs each sync)
``` ```
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
View File
Regular → Executable
+1 -1
View File
@@ -198,7 +198,7 @@ FINDINGS=() # notable but not critical
ISSUES=() # need attention ISSUES=() # need attention
DIGEST_LINES=() # full report lines DIGEST_LINES=() # full report lines
# ── Failover State ──────────────────────────────────────────────────────────────────────────── # ── fallback State ────────────────────────────────────────────────────────────────────────────
if [[ -f "$FALLBACK_STATE_FILE" ]]; then if [[ -f "$FALLBACK_STATE_FILE" ]]; then
FALLBACK_STATE=$(grep "^state=" "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2) FALLBACK_STATE=$(grep "^state=" "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
if [[ -n "$FALLBACK_STATE" ]]; then if [[ -n "$FALLBACK_STATE" ]]; then
Regular → Executable
View File
+1 -1
View File
@@ -18,7 +18,7 @@ claude
. fix failover strike list timing, maybe 30 seconds. them a t 90 seconds 3 stike triggers. just gotta test buffer. never had the strike system . fix fallback strike list timing, maybe 30 seconds. them a t 90 seconds 3 stike triggers. just gotta test buffer. never had the strike system
. verify silent toggle switches back on good notifications . verify silent toggle switches back on good notifications
View File
+3 -3
View File
@@ -658,7 +658,7 @@ DAILY_MAINTENANCE_SCRIPTS=(
--- ---
### ── Relationship to Failover Writeback ────────────────────────────────────── ### ── Relationship to Fallback Writeback ──────────────────────────────────────
```bash ```bash
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
@@ -668,7 +668,7 @@ DAILY_MAINTENANCE_SCRIPTS=(
# Normal (daily_sync_maintenance.sh): # Normal (daily_sync_maintenance.sh):
# HOST1 → pushes Movies, Tv_Shows → HOST2 # HOST1 → pushes Movies, Tv_Shows → HOST2
# #
# Tier 4 failover writeback (HOST1 returns after 24hr+ outage): # Tier 4 fallback writeback (HOST1 returns after 24hr+ outage):
# HOST2 → pushes Movies, Tv_Shows → HOST1 # HOST2 → pushes Movies, Tv_Shows → HOST1
# (HOST2 was running HOST1's arrs and accumulated content) # (HOST2 was running HOST1's arrs and accumulated content)
# #
@@ -710,7 +710,7 @@ maintenance block before the 7am coffee report.
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# Containers stop BEFORE sync — clean static source, full bandwidth. # Containers stop BEFORE sync — clean static source, full bandwidth.
# Containers start AFTER sync — on fresh data, in dependency order. # Containers start AFTER sync — on fresh data, in dependency order.
# DDNS and failover continue running throughout — only managed containers stop. # DDNS and fallback continue running throughout — only managed containers stop.
# #
# 1. Pre-flight checks — connectivity, remote Docker daemon, remote rootfs # 1. Pre-flight checks — connectivity, remote Docker daemon, remote rootfs
# 2. Stop local containers — Emby + auth stack stopped on this server # 2. Stop local containers — Emby + auth stack stopped on this server
View File
View File
View File
View File
View File
View File
+3 -3
View File
@@ -582,7 +582,7 @@ derive_partner_folder_name() {
local hostname="$1" local hostname="$1"
local short="${hostname,,}" local short="${hostname,,}"
[[ "$short" == unraid-* ]] && short="${short:7}" [[ "$short" == unraid-* ]] && short="${short:7}"
# Capitalise first char for readability: jayred365 → Jayred365-Failover # Capitalise first char for readability: jayred365 → Jayred365-fallback
echo "${short^}-Fallback" echo "${short^}-Fallback"
} }
@@ -733,7 +733,7 @@ folderview3_remove_partner_folder() {
return 0 return 0
} }
# Gather all partner failover containers for this server (all tiers) # Gather all partner fallback containers for this server (all tiers)
gather_partner_fallback_containers() { gather_partner_fallback_containers() {
local out_var="$1" local out_var="$1"
eval "${out_var}=()" eval "${out_var}=()"
@@ -1570,7 +1570,7 @@ if (vv_write_conf_raw('master.conf', \$master)) {
warn "DRY RUN — would write ACTIVE state and push to remote" warn "DRY RUN — would write ACTIVE state and push to remote"
fi fi
# FolderView3 — create partner folder with this server's failover containers for remote # FolderView3 — create partner folder with this server's fallback containers for remote
if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]]; then if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]]; then
echo "" echo ""
echo "━━━ $ICON_CONTAINERS FolderView3 Integration ━━━" echo "━━━ $ICON_CONTAINERS FolderView3 Integration ━━━"
Regular → Executable
View File
+24 -24
View File
@@ -35,7 +35,7 @@ User Scripts Enhanced — Kurotaku
``` ```
1. Every script entry → Run in Background: YES 1. Every script entry → Run in Background: YES
Without background mode the plugin UI freezes waiting for the script to complete. Without background mode the plugin UI freezes waiting for the script to complete.
Long-running scripts and continuous loops (failover, watchdogs) require background. Long-running scripts and continuous loops (fallback, watchdogs) require background.
There are no exceptions — set background on everything. There are no exceptions — set background on everything.
2. Run as root: YES 2. Run as root: YES
@@ -66,7 +66,7 @@ Create these categories in User Scripts Enhanced before adding any entries:
``` ```
Orchestrators — scheduled maintenance windows and startup launcher Orchestrators — scheduled maintenance windows and startup launcher
Failover — mutual failover state machine and test tools Fallback — mutual fallback state machine and test tools
Monitors — weekly reports, health checks, trend tracking Monitors — weekly reports, health checks, trend tracking
Rsync — manual per-profile sync triggers Rsync — manual per-profile sync triggers
Docker_Essentials — container management tools (manual use) Docker_Essentials — container management tools (manual use)
@@ -119,12 +119,12 @@ ramdisk_setup.sh — creates /mnt/ramdisk_transcodes and symlink before
docker_network_connect.sh— ensures custom networks exist + containers connected docker_network_connect.sh— ensures custom networks exist + containers connected
stability_watchdog.sh ← last-resort — three-tier server stability watchdog stability_watchdog.sh ← last-resort — three-tier server stability watchdog
docker_watchdog.sh ← continuous — two-tier container healing watchdog docker_watchdog.sh ← continuous — two-tier container healing watchdog
fallback.sh ← continuous — mutual failover state machine fallback.sh ← continuous — mutual fallback state machine
``` ```
**After running:** **After running:**
One-shot scripts exit cleanly. Three continuous processes run in the background until One-shot scripts exit cleanly. Three continuous processes run in the background until
the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|failover"` the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|fallback"`
--- ---
@@ -291,7 +291,7 @@ in one orchestrated sequence.
--- ---
### ── rsync_emby_failover ───────────────────────────────────────────────────── ### ── rsync_emby_fallback ─────────────────────────────────────────────────────
> **Superseded by `critical_sync_maintenance`.** Delete this User Script from the > **Superseded by `critical_sync_maintenance`.** Delete this User Script from the
> plugin. The emby-fallback dirty sync now runs as step 2 inside > plugin. The emby-fallback dirty sync now runs as step 2 inside
@@ -413,7 +413,7 @@ Background: YES
``` ```
**What it does:** **What it does:**
Reads state files from across the ecosystem — watchdog strikes, failover state, skip Reads state files from across the ecosystem — watchdog strikes, fallback state, skip
list, bandwidth history, transcode stats, cert expiry — and compiles them into a list, bandwidth history, transcode stats, cert expiry — and compiles them into a
digest notification. Reads only, writes nothing. digest notification. Reads only, writes nothing.
@@ -421,7 +421,7 @@ digest notification. Reads only, writes nothing.
``` ```
always — sends every run regardless of content (daily digest, even when healthy) always — sends every run regardless of content (daily digest, even when healthy)
smart — sends only when something worth reporting is found smart — sends only when something worth reporting is found
triggers: watchdog strikes, non-NORMAL failover, cert warnings, bandwidth spikes triggers: watchdog strikes, non-NORMAL fallback, cert warnings, bandwidth spikes
weekly — sends once per week on DIGEST_DAY (Sunday), silent all other days weekly — sends once per week on DIGEST_DAY (Sunday), silent all other days
``` ```
@@ -642,7 +642,7 @@ that survive rsync updates without parser changes.
--- ---
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🔄 FAILOVER ## 🔄 FALLBACK
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> **fallback.sh is started by array_started.sh — not scheduled separately.** > **fallback.sh is started by array_started.sh — not scheduled separately.**
@@ -650,11 +650,11 @@ that survive rsync updates without parser changes.
--- ---
### ── failover_status ───────────────────────────────────────────────────────── ### ── fallback_status ─────────────────────────────────────────────────────────
``` ```
Category: Failover Category: Fallback
Name: failover_status Name: fallback_status
Schedule: None — run manually Schedule: None — run manually
Background: NO (runs and exits — status check) Background: NO (runs and exits — status check)
``` ```
@@ -665,21 +665,21 @@ Background: NO (runs and exits — status check)
``` ```
**What it does:** **What it does:**
Shows current failover state at a glance without affecting the running loop. Shows current fallback state at a glance without affecting the running loop.
State (NORMAL/FAILOVER/NO_INTERNET/DARK). Which tier containers are active. State (NORMAL/FALLBACK/NO_INTERNET/DARK). Which tier containers are active.
Outage duration if in FAILOVER. Handback strike count if remote has returned. Outage duration if in FALLBACK. Handback strike count if remote has returned.
DDNS containers currently running. Remote server Tailscale visibility. DDNS containers currently running. Remote server Tailscale visibility.
**Use before any manual failover intervention.** Understanding the current state **Use before any manual fallback intervention.** Understanding the current state
before acting prevents making things worse. before acting prevents making things worse.
--- ---
### ── failover_test ──────────────────────────────────────────────────────────── ### ── fallback_test ────────────────────────────────────────────────────────────
``` ```
Category: Failover Category: Fallback
Name: failover_test Name: fallback_test
Schedule: None — run manually (during a planned maintenance window) Schedule: None — run manually (during a planned maintenance window)
Background: YES — test takes several minutes Background: YES — test takes several minutes
``` ```
@@ -690,7 +690,7 @@ Background: YES — test takes several minutes
``` ```
**What it does:** **What it does:**
Validates the full failover lifecycle — detection, Tier 1 container start, handback, Validates the full fallback lifecycle — detection, Tier 1 container start, handback,
container stop, DNS return — using an iptables rule to simulate remote unreachability. container stop, DNS return — using an iptables rule to simulate remote unreachability.
No real outage. Safety trap removes the iptables rule on any exit condition including No real outage. Safety trap removes the iptables rule on any exit condition including
ctrl-c and crashes — remote is always reachable after the test. ctrl-c and crashes — remote is always reachable after the test.
@@ -702,7 +702,7 @@ ctrl-c and crashes — remote is always reachable after the test.
**For a live test** (real container lifecycle, brief service interruption): **For a live test** (real container lifecycle, brief service interruption):
Change `--dry-run` to nothing and run during a maintenance window. Change `--dry-run` to nothing and run during a maintenance window.
**8 phases:** pre-flight → block remote → detect FAILOVER → verify Tier 1 started **8 phases:** pre-flight → block remote → detect FALLBACK → verify Tier 1 started
→ restore remote → wait for handback → verify NORMAL restored → report pass/fail. → restore remote → wait for handback → verify NORMAL restored → report pass/fail.
--- ---
@@ -711,7 +711,7 @@ Change `--dry-run` to nothing and run during a maintenance window.
> **Not yet built.** `fallback_state_reset.sh` is a planned utility — not in `Fallback/` yet. > **Not yet built.** `fallback_state_reset.sh` is a planned utility — not in `Fallback/` yet.
> >
> **Manual workaround** when state file is stuck in FAILOVER after a failed test or > **Manual workaround** when state file is stuck in FALLBACK after a failed test or
> forced kill of fallback.sh: > forced kill of fallback.sh:
> ```bash > ```bash
> # Verify first — right containers on right server, DDNS correct, fallback.sh not running > # Verify first — right containers on right server, DDNS correct, fallback.sh not running
@@ -721,7 +721,7 @@ Change `--dry-run` to nothing and run during a maintenance window.
> Resets state only — does NOT start or stop any containers. > Resets state only — does NOT start or stop any containers.
**When you need this:** **When you need this:**
State file stuck in FAILOVER after a test that didn't complete cleanly. After killing State file stuck in FALLBACK after a test that didn't complete cleanly. After killing
fallback.sh directly (not via plugin Abort button). After a failed handback. fallback.sh directly (not via plugin Abort button). After a failed handback.
**Verify before resetting:** **Verify before resetting:**
@@ -1615,7 +1615,7 @@ Wrong startup setup (old approach):
docker_syslog_filter — At Startup of Array docker_syslog_filter — At Startup of Array
stability_watchdog — Every 15 Minutes via watchdog_orchestrator stability_watchdog — Every 15 Minutes via watchdog_orchestrator
docker_watchdog — At Startup of Array docker_watchdog — At Startup of Array
failover — At Startup of Array fallback — At Startup of Array
(unpredictable order, no coordination) (unpredictable order, no coordination)
``` ```
@@ -1680,7 +1680,7 @@ array_start --dry-run verify all startup scripts found and sequenced co
transcode_management --status confirm ramdisk mounted and symlink pointing at ramdisk transcode_management --status confirm ramdisk mounted and symlink pointing at ramdisk
critical_sync_maintenance verify Critical-Data sync completes to HOST2 critical_sync_maintenance verify Critical-Data sync completes to HOST2
daily_sync_maintenance --dry-run verify remote connectivity and share list daily_sync_maintenance --dry-run verify remote connectivity and share list
failover_status confirm NORMAL state on both servers fallback_status confirm NORMAL state on both servers
docker_watchdog_status confirm required containers all running docker_watchdog_status confirm required containers all running
stability_watchdog_status confirm no threshold alerts on first check stability_watchdog_status confirm no threshold alerts on first check
``` ```
+23 -23
View File
@@ -5,7 +5,7 @@
**A complete self-healing, self-maintaining, mutually-redundant two-server unRAID **A complete self-healing, self-maintaining, mutually-redundant two-server unRAID
ecosystem.** One codebase runs on both servers. Both servers run their own lives ecosystem.** One codebase runs on both servers. Both servers run their own lives
independently. When one goes down the other covers it — not because of conventional independently. When one goes down the other covers it — not because of conventional
failover infrastructure, but because both servers already have copies of everything fallback infrastructure, but because both servers already have copies of everything
that matters and both know how to run it. that matters and both know how to run it.
> **This is not conventional failover.** There is no primary and standby. There is no > **This is not conventional failover.** There is no primary and standby. There is no
@@ -38,7 +38,7 @@ HOST2 — unRAID-Jayred365
**Hardware does not need to match.** Everything is accessed through `/mnt/user/` **Hardware does not need to match.** Everything is accessed through `/mnt/user/`
unRAID's unified share layer. HOST1 has a Threadripper with ZFS. HOST2 has a completely unRAID's unified share layer. HOST1 has a Threadripper with ZFS. HOST2 has a completely
different CPU and fewer drives. A failover container on HOST2 mounts `/mnt/user/Movies` different CPU and fewer drives. A fallback container on HOST2 mounts `/mnt/user/Movies`
and sees mirrored data because the share names match. The hardware underneath is and sees mirrored data because the share names match. The hardware underneath is
irrelevant. irrelevant.
@@ -60,7 +60,7 @@ break silently.
# #
# The convention: "no suffix = this container participates in the shared stack" # The convention: "no suffix = this container participates in the shared stack"
# #
Emby # the shared Emby — runs on HOST1, picked up by HOST2 on failover Emby # the shared Emby — runs on HOST1, picked up by HOST2 on fallback
NginxProxyManager # shared NPM — runs on both, same config via sync NginxProxyManager # shared NPM — runs on both, same config via sync
Lldap-Gmer4Lfe # shared LLDAP — same user directory, both servers Lldap-Gmer4Lfe # shared LLDAP — same user directory, both servers
Authelia # shared auth — both servers, same config Authelia # shared auth — both servers, same config
@@ -93,7 +93,7 @@ codebase work on both servers without modification.
Immich-Gmer4Lfe # Gmer4Lfe's photo library — personal Immich-Gmer4Lfe # Gmer4Lfe's photo library — personal
``` ```
User-specific containers are covered during failover (see Tiered Failover below) but User-specific containers are covered during fallback (see Tiered Fallback below) but
are distinct from the shared stack. They are never confused with each other because are distinct from the shared stack. They are never confused with each other because
the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts targeting the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts targeting
`Emby-Gmer4Lfe` are targeting different containers for different purposes. `Emby-Gmer4Lfe` are targeting different containers for different purposes.
@@ -113,7 +113,7 @@ the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts tar
# → looks for exactly those names in docker ps # → looks for exactly those names in docker ps
# #
# A container named differently than expected is not found — no error, no restart, # A container named differently than expected is not found — no error, no restart,
# no failover coverage. It silently does not happen. # no fallback coverage. It silently does not happen.
# Keep names consistent on both servers for shared containers. # Keep names consistent on both servers for shared containers.
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
``` ```
@@ -235,11 +235,11 @@ where determines whether you need encryption and whether sync failure matters.
--- ---
### ── Shared Failover Shares — Sync Required ───────────────────────────────── ### ── Shared Fallback Shares — Sync Required ─────────────────────────────────
```bash ```bash
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# These shares enable failover — HOST2 must have current copies for the covered # These shares enable fallback — HOST2 must have current copies for the covered
# services to work correctly when they start on HOST2. # services to work correctly when they start on HOST2.
# #
/mnt/user/Media_Server/Emby # Emby userdata — watch history, playstates /mnt/user/Media_Server/Emby # Emby userdata — watch history, playstates
@@ -248,7 +248,7 @@ where determines whether you need encryption and whether sync failure matters.
/mnt/user/Tv_Shows # media library content /mnt/user/Tv_Shows # media library content
/mnt/user/Music # media library content /mnt/user/Music # media library content
# #
# Sync failure here = failover runs but with stale data (acceptable) or missing # Sync failure here = fallback runs but with stale data (acceptable) or missing
# data (problematic). These sync every night via daily_sync_maintenance.sh. # data (problematic). These sync every night via daily_sync_maintenance.sh.
# Critical-Data and Emby also sync every 30 minutes via critical_sync_maintenance.sh. # Critical-Data and Emby also sync every 30 minutes via critical_sync_maintenance.sh.
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
@@ -256,12 +256,12 @@ where determines whether you need encryption and whether sync failure matters.
--- ---
### ── Non-Failover Shares — Sync as Backup, No Encryption Required ─────────── ### ── Non-Fallback Shares — Sync as Backup, No Encryption Required ───────────
```bash ```bash
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# Some shares don't participate in failover — their container only runs on one # Some shares don't participate in fallback — their container only runs on one
# server and is covered during failover from scratch (or not at all). # server and is covered during fallback from scratch (or not at all).
# These shares can sync as offsite backup without encryption. # These shares can sync as offsite backup without encryption.
# Content is not sensitive — anyone with access to HOST2 could read it. # Content is not sensitive — anyone with access to HOST2 could read it.
# #
@@ -323,7 +323,7 @@ Two fully independent servers, each running their own household's services.
Each server has its own domain, its own users, its own media library management. Each server has its own domain, its own users, its own media library management.
Each server is the primary for its own services every single day. Each server is the primary for its own services every single day.
The "failover" component is additive: The "fallback" component is additive:
When HOST1 goes down → HOST2 starts HOST1's containers on top of its own When HOST1 goes down → HOST2 starts HOST1's containers on top of its own
When HOST2 comes back → HOST1's containers hand back cleanly When HOST2 comes back → HOST1's containers hand back cleanly
HOST2 never stopped running its own services during this entire time HOST2 never stopped running its own services during this entire time
@@ -378,7 +378,7 @@ This is not standby. This is mutual coverage between peers.
--- ---
## ━━━ HOW FAILOVER ACTUALLY WORKS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ HOW FALLBACK ACTUALLY WORKS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
`fallback.sh` runs on both servers as a continuous background process. Every 2 minutes `fallback.sh` runs on both servers as a continuous background process. Every 2 minutes
it pings two things: the other server's Tailscale IP, and `8.8.8.8`. The combination it pings two things: the other server's Tailscale IP, and `8.8.8.8`. The combination
@@ -387,7 +387,7 @@ of those two answers determines the current state.
``` ```
Remote ping PASS + Internet PASS → NORMAL — both servers running independently Remote ping PASS + Internet PASS → NORMAL — both servers running independently
Remote ping FAIL + Internet PASS → FAILOVER — remote is down, start covering it Remote ping FAIL + Internet PASS → FALLBACK — remote is down, start covering it
Tier 1 immediately: DDNS + Emby + Auth + VaultWarden + Live TV + Gitea Tier 1 immediately: DDNS + Emby + Auth + VaultWarden + Live TV + Gitea
Tier 2 at 4 hours: NextCloud + Immich + Jellyseerr Tier 2 at 4 hours: NextCloud + Immich + Jellyseerr
Tier 3 at 12 hours: AdGuard + Uptime monitoring Tier 3 at 12 hours: AdGuard + Uptime monitoring
@@ -410,12 +410,12 @@ DDNS last. Split-brain DNS is prevented by design.
## ━━━ THE AUTH STACK — WARM ON BOTH SERVERS ALWAYS ━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ THE AUTH STACK — WARM ON BOTH SERVERS ALWAYS ━━━━━━━━━━━━━━━━━━━━━━━━━━
NPM, LLDAP, Authelia, MariaDB, Redis — all running actively on both servers at all NPM, LLDAP, Authelia, MariaDB, Redis — all running actively on both servers at all
times. Not cold standby. Not started at failover. Running right now, serving each times. Not cold standby. Not started at fallback. Running right now, serving each
server's own domain. server's own domain.
```bash ```bash
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# Why this eliminates the failover dead zone: # Why this eliminates the fallback dead zone:
# #
# Cold standby approach: HOST1 goes down → DNS switches to HOST2 → HOST2 starts # Cold standby approach: HOST1 goes down → DNS switches to HOST2 → HOST2 starts
# auth containers → 30-60 seconds before auth is ready → users hit the auth wall # auth containers → 30-60 seconds before auth is ready → users hit the auth wall
@@ -454,7 +454,7 @@ Source of truth:
Config auth stack config ──→ mirrored every 30 min Config auth stack config ──→ mirrored every 30 min
Emby state watch history, playstates──→ dirty sync every 30 min Emby state watch history, playstates──→ dirty sync every 30 min
Failover coverage: Fallback coverage:
If HOST1 down: — starts Emby, VaultWarden, If HOST1 down: — starts Emby, VaultWarden,
Dispatcharr, all arr stack Dispatcharr, all arr stack
serves Gmer4Lfe.com via DDNS serves Gmer4Lfe.com via DDNS
@@ -489,7 +489,7 @@ varaverk/
├── Rsync/ ← Core rsync script — called per share with profile ├── Rsync/ ← Core rsync script — called per share with profile
│ README: README-Rsync_Setup.md │ README: README-Rsync_Setup.md
├── Fallback/ ← Mutual automatic failover — continuous background process ├── Fallback/ ← Mutual automatic fallback — continuous background process
│ README: README-Fallback.md │ README: README-Fallback.md
├── Watchdogs/ ← All watchdog scripts: resource, docker, system, stability ├── Watchdogs/ ← All watchdog scripts: resource, docker, system, stability
@@ -564,7 +564,7 @@ Array start (Varaverk event hook):
→ php_fpm_max_children.sh WebGUI tuning before first request → php_fpm_max_children.sh WebGUI tuning before first request
→ ramdisk_setup.sh create ramdisk before Emby starts → ramdisk_setup.sh create ramdisk before Emby starts
→ docker_network_connect.sh connect containers to extra networks → docker_network_connect.sh connect containers to extra networks
→ fallback.sh continuous — mutual failover state machine → fallback.sh continuous — mutual fallback state machine
Every 15 minutes: Every 15 minutes:
watchdog_orchestrator.sh fires each watchdog in sequence watchdog_orchestrator.sh fires each watchdog in sequence
@@ -782,9 +782,9 @@ If you're setting this up from scratch on two servers:
4. Run initial sync to populate HOST2 4. Run initial sync to populate HOST2
bash daily_sync_maintenance.sh --log bash daily_sync_maintenance.sh --log
5. Verify failover works before relying on it 5. Verify fallback works before relying on it
bash failover_test.sh --dry-run first bash fallback_test.sh --dry-run first
bash failover_test.sh when ready — uses iptables simulation, no real downtime bash fallback_test.sh when ready — uses iptables simulation, no real downtime
``` ```
--- ---
@@ -802,7 +802,7 @@ If you're setting this up from scratch on two servers:
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles | | [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard | | [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler | | [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
| [README-Tools.md](Tools/README-Tools.md) | failover_state_reset, skip list manager, emby_database_repair, container export | | [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
| [README-Unraid_Essentials.md](unRAID_Essentials/README-Unraid_Essentials.md) | `git_pull_execute.sh`, `server_reboot.sh`, `mover_stop.sh`, `user_scripts_stop.sh` | | [README-Unraid_Essentials.md](unRAID_Essentials/README-Unraid_Essentials.md) | `git_pull_execute.sh`, `server_reboot.sh`, `mover_stop.sh`, `user_scripts_stop.sh` |
--- ---
+2 -2
View File
@@ -68,7 +68,7 @@ Terminal access — unRAID UI → Tools → Terminal, or SSH
Optional but recommended: Optional but recommended:
``` ```
Gitea (Docker container on HOST1) — self-hosted git for the script repository Gitea (Docker container on HOST1) — self-hosted git for the script repository
Working Emby installation — for transcode management and failover Working Emby installation — for transcode management and fallback
``` ```
--- ---
@@ -366,7 +366,7 @@ PROFILES["emby-fallback_REMOTE_RESTART_CONTAINERS"]=("Emby")
- Why: WAL is checkpointed on clean shutdown — safe to copy everything - Why: WAL is checkpointed on clean shutdown — safe to copy everything
- Result: HOST2 gets a gold-standard Emby state once per week - Result: HOST2 gets a gold-standard Emby state once per week
The two profiles work together. emby-fallback keeps HOST2 current for immediate failover. The two profiles work together. emby-fallback keeps HOST2 current for immediate fallback.
emby gives HOST2 full fidelity once per week. Neither alone is sufficient. emby gives HOST2 full fidelity once per week. Neither alone is sufficient.
--- ---
Regular → Executable
View File
+1 -1
View File
@@ -572,7 +572,7 @@ docker_watchdog
Memory-monitored containers Memory-monitored containers
Recent restart history + skip list Recent restart history + skip list
failover (fallback.sh) fallback (fallback.sh)
Current state (NORMAL / FALLBACK / HANDBACK) Current state (NORMAL / FALLBACK / HANDBACK)
Tier flags and timestamps Tier flags and timestamps
Remote Tailscale visibility Remote Tailscale visibility
+5 -5
View File
@@ -13,8 +13,8 @@ the scheduled maintenance model. These scripts sit ready for the moment you need
## ━━━ THE PROBLEM THAT BUILT THIS ━━━ ## ━━━ THE PROBLEM THAT BUILT THIS ━━━
**Failover State Stuck After Testing** **Fallback State Stuck After Testing**
Run a failover test, something exits uncleanly, state file shows `FALLBACK`. Run a fallback test, something exits uncleanly, state file shows `FALLBACK`.
`fallback.sh` resumes and reads FALLBACK — starts containers it shouldn't, makes `fallback.sh` resumes and reads FALLBACK — starts containers it shouldn't, makes
decisions based on a state that doesn't reflect reality. Manual recovery means decisions based on a state that doesn't reflect reality. Manual recovery means
knowing the exact file format and every field to reset. At 2am after a failed test. knowing the exact file format and every field to reset. At 2am after a failed test.
@@ -90,7 +90,7 @@ unRAID_Essentials/ ← regular system maintenance — scheduled
Docker_Essentials/ ← regular container management — scheduled Docker_Essentials/ ← regular container management — scheduled
Monitors/ ← regular health reporting — scheduled Monitors/ ← regular health reporting — scheduled
Orchestrators/ ← regular maintenance windows — scheduled Orchestrators/ ← regular maintenance windows — scheduled
Fallback/ ← automated failover/handback — event-driven Fallback/ ← automated fallback/handback — event-driven
Tools/ ← situational utilities — run when needed Tools/ ← situational utilities — run when needed
``` ```
@@ -116,7 +116,7 @@ The relationship is one-way: Tools act on state that other scripts have written.
| Script | What It Fixes | When to Run | | Script | What It Fixes | When to Run |
|--------|--------------|-------------| |--------|--------------|-------------|
| `fallback_state_reset.sh` | State file stuck in FALLBACK after test or failed handback | After failover testing or manual intervention | | `fallback_state_reset.sh` | State file stuck in FALLBACK after test or failed handback | After fallback testing or manual intervention |
| `watchdog_skip_list_manager.sh` | Container stuck on watchdog skip list after fixing root cause | After fixing a container that hit the restart loop limit | | `watchdog_skip_list_manager.sh` | Container stuck on watchdog skip list after fixing root cause | After fixing a container that hit the restart loop limit |
| `bulk_permissions_repair.sh` | Files owned by wrong user after admin copy or bad container config | When arr operations fail due to permissions | | `bulk_permissions_repair.sh` | Files owned by wrong user after admin copy or bad container config | When arr operations fail due to permissions |
| `container_data_export.sh` | Need a clean backup before a risky container update or migration | Before major updates, appdata migrations, or container removals | | `container_data_export.sh` | Need a clean backup before a risky container update or migration | Before major updates, appdata migrations, or container removals |
@@ -143,7 +143,7 @@ Situation arises
┌──────────────────────────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────────────────────────────┐
│ Tools/ Run directly when needed │ │ Tools/ Run directly when needed │
│ │ │ │
│ fallback_state_reset.sh ◄── after failover test / failed handback│ │ fallback_state_reset.sh ◄── after fallback test / failed handback│
│ watchdog_skip_list_manager ◄── after fixing a crash-looping container│ │ watchdog_skip_list_manager ◄── after fixing a crash-looping container│
│ bulk_permissions_repair ◄── wrong ownership after copy or rsync │ │ bulk_permissions_repair ◄── wrong ownership after copy or rsync │
│ container_data_export ◄── before a risky update or migration │ │ container_data_export ◄── before a risky update or migration │
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
+4 -4
View File
@@ -1,12 +1,12 @@
#!/bin/bash #!/bin/bash
# ============================================================================================== # ==============================================================================================
# ============================= Failover State Reset =========================================== # ============================= fallback State Reset ===========================================
# ============================================================================================== # ==============================================================================================
# #
# PURPOSE # PURPOSE
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# Resets the fallback state file to NORMAL and clears all tier flags. Use when # Resets the fallback state file to NORMAL and clears all tier flags. Use when
# the state file is stuck after failover testing, a failed handback, manual # the state file is stuck after fallback testing, a failed handback, manual
# intervention, or fallback.sh being killed mid-cycle. # intervention, or fallback.sh being killed mid-cycle.
# #
# Writes a fresh state file with: # Writes a fresh state file with:
@@ -17,8 +17,8 @@
# resumes from NORMAL on its next cycle. # resumes from NORMAL on its next cycle.
# #
# WARNING: Only run when you have verified the stack is actually in a normal # WARNING: Only run when you have verified the stack is actually in a normal
# state — right containers on the right server, DDNS correct, no active failover # state — right containers on the right server, DDNS correct, no active fallback
# in progress. Resetting state during a real failover causes fallback.sh to stop # in progress. Resetting state during a real fallback causes fallback.sh to stop
# covering the remote server until the next detection cycle. # covering the remote server until the next detection cycle.
# #
# ============================================================================================== # ==============================================================================================
Regular → Executable
View File
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
+5 -5
View File
@@ -48,7 +48,7 @@
# translate_path() added — container → host path for arr cleanup scripts # translate_path() added — container → host path for arr cleanup scripts
# check_arr_version() added — API version safety gate before arr operations # check_arr_version() added — API version safety gate before arr operations
# check_api() added — pre-flight API reachability check # check_api() added — pre-flight API reachability check
# ping_remote(), ping_internet() — non-fatal ping for failover use # ping_remote(), ping_internet() — non-fatal ping for fallback use
# check_local_array(), check_remote_array() — array health checks # check_local_array(), check_remote_array() — array health checks
# check_remote_docker() — Docker daemon health check # check_remote_docker() — Docker daemon health check
# get_unraid_temp_thresholds() — reads thresholds from dynamix.cfg # get_unraid_temp_thresholds() — reads thresholds from dynamix.cfg
@@ -159,7 +159,7 @@ ICON_LOCK="🔏" # script instance lock — acquired/released
ICON_RAM="💨" # ramdisk operations — fast ephemeral storage ICON_RAM="💨" # ramdisk operations — fast ephemeral storage
ICON_LINK="🔗" # symlink state and management ICON_LINK="🔗" # symlink state and management
# Failover Operations # fallback Operations
ICON_FALLBACK="🔀" # fallback state changes and operations ICON_FALLBACK="🔀" # fallback state changes and operations
# Docker Network Operations # Docker Network Operations
@@ -656,7 +656,7 @@ resolve_tailscale_ip() {
# ============================================================================================== # ==============================================================================================
# Fatal connectivity check — used by rsync and other scripts that must abort if unreachable. # Fatal connectivity check — used by rsync and other scripts that must abort if unreachable.
# For failover use ping_remote() which returns status without exiting. # For fallback use ping_remote() which returns status without exiting.
check_connectivity() { check_connectivity() {
log "Checking connectivity to $REMOTE_SERVER..." log "Checking connectivity to $REMOTE_SERVER..."
if ! ping -c1 -W3 "$REMOTE_SERVER" &>/dev/null; then if ! ping -c1 -W3 "$REMOTE_SERVER" &>/dev/null; then
@@ -739,7 +739,7 @@ is_vm_manager_enabled() {
# Verifies local /mnt/user is mounted and has shares. # Verifies local /mnt/user is mounted and has shares.
# Non-fatal — returns status for caller to decide. # Non-fatal — returns status for caller to decide.
# Used by failover before starting remote containers locally. # Used by fallback before starting remote containers locally.
check_local_array() { check_local_array() {
log "Checking local array..." log "Checking local array..."
if ! mountpoint -q /mnt/user 2>/dev/null; then if ! mountpoint -q /mnt/user 2>/dev/null; then
@@ -1660,7 +1660,7 @@ check_api() {
# ============================================================================================== # ==============================================================================================
# Verifies local and remote servers are running compatible unRAID versions before any # Verifies local and remote servers are running compatible unRAID versions before any
# remote operation. Version mismatches can mean changed APIs, commands, or behaviours # remote operation. Version mismatches can mean changed APIs, commands, or behaviours
# that silently break remote container operations, rsync, or failover logic. # that silently break remote container operations, rsync, or fallback logic.
# #
# Reads /etc/unraid-version on both sides — format: version="7.2.3" # Reads /etc/unraid-version on both sides — format: version="7.2.3"
# #
+1 -1
View File
@@ -25,7 +25,7 @@
# New server gets only its own conf ✅ # New server gets only its own conf ✅
# #
# ── GITEA LOCATION DETECTION ────────────────────────────────────────────────────────────────── # ── GITEA LOCATION DETECTION ──────────────────────────────────────────────────────────────────
# Detects where Gitea is running at runtime — works through failover: # Detects where Gitea is running at runtime — works through fallback:
# Gitea local → connects via local IP # Gitea local → connects via local IP
# Gitea remote → connects via Tailscale IP # Gitea remote → connects via Tailscale IP
# Both fail → falls back to GITEA_DOMAIN if configured # Both fail → falls back to GITEA_DOMAIN if configured
Regular → Executable
View File
Regular → Executable
View File
View File
Regular → Executable
View File