Correct Fallback tier variable naming and document rsync merge-run

The fallback docs used an older COVERS naming in the wrong conf file, and rsync's merge-run
mode — the only path that carries --delete — was undocumented entirely.
This commit is contained in:
Gmer4Lfe
2026-08-01 23:05:30 -04:00
parent 8a2707ee37
commit 53aa72d38c
3 changed files with 115 additions and 28 deletions
+42 -24
View File
@@ -105,18 +105,36 @@ internet connectivity. Most hosts leave this empty.
--- ---
``` ```
FALLBACK_HOST*_COVERS_HOST*_TIER1=(...) FALLBACK_HOST*_TIER1=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER2=(...) FALLBACK_HOST*_TIER2=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER3=(...) FALLBACK_HOST*_TIER3=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER4=(...) FALLBACK_HOST*_TIER4=(...)
``` ```
Containers this host starts for the remote host when the remote is down. TIER1 starts
immediately. TIER24 activate after the corresponding delay thresholds.
Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}` **Each host declares its OWN services, in its OWN conf.** `FALLBACK_HOST2_TIER1` lives in
`host2.conf` and lists HOST2's vital containers — it is not a list HOST1 maintains.
The DDNS container for the remote's domain must be the first entry in TIER1 — DNS At runtime `fallback.sh` reads its **partner's** list:
coverage before anything else.
```bash
get_tier_containers() {
local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}" # note: REMOTE_ID, not MY_ID
```
So HOST1, covering HOST2, reads `FALLBACK_HOST2_TIER1` — a variable defined in `host2.conf`
and delivered to HOST1 through the partner conf cache (`conf_sync.sh`), because sparse
checkout means HOST1 never pulls `host2.conf` from git.
This is why the naming is what it is. The alternative — each host keeping a copy of its
partner's container list — would need editing on both machines every time either one changed
a service, and the two copies would silently diverge. Declaring once, on the host that owns
the services, means a host is always the authority on what covering it requires.
TIER1 starts immediately. TIER24 activate after their delay thresholds
(`HOST*_TIER2_DELAY` and friends, also in that host's own conf).
The DDNS container for that host's domain must be the **first entry in TIER1** — DNS coverage
before anything else.
--- ---
@@ -181,15 +199,15 @@ FALLBACK_TEST_HANDBACK_WAIT=300
--- ---
### host2.conf — HOST2 covering HOST1 ### host1.conf — HOST1's own services (started by HOST2 when HOST1 is down)
```bash ```bash
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS") HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
FALLBACK_HOST2_STOP_ON_NO_NET=() FALLBACK_HOST1_STOP_ON_NO_NET=()
# Tier 1 — immediate (vital services + Live TV) # Tier 1 — immediate (vital services + Live TV)
FALLBACK_HOST2_COVERS_HOST1_TIER1=( FALLBACK_HOST1_TIER1=(
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else "Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
"Emby" # media server — people are watching "Emby" # media server — people are watching
"NginxProxyManager" # reverse proxy — all external access routes through this "NginxProxyManager" # reverse proxy — all external access routes through this
@@ -205,7 +223,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER1=(
) )
# Tier 2 — after 4 hours (shared productivity services) # Tier 2 — after 4 hours (shared productivity services)
FALLBACK_HOST2_COVERS_HOST1_TIER2=( FALLBACK_HOST1_TIER2=(
"Postgres-NextCloud" # must start before NextCloud "Postgres-NextCloud" # must start before NextCloud
"NextCloud" "NextCloud"
"PostgreSQL-Immich" # must start before Immich "PostgreSQL-Immich" # must start before Immich
@@ -214,7 +232,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER2=(
) )
# Tier 3 — after 12 hours (secondary services) # Tier 3 — after 12 hours (secondary services)
FALLBACK_HOST2_COVERS_HOST1_TIER3=( FALLBACK_HOST1_TIER3=(
"Organizrv2-Gmer4Lfe" "Organizrv2-Gmer4Lfe"
"AdGuard-Home" "AdGuard-Home"
"UptimeKuma" "UptimeKuma"
@@ -223,7 +241,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER3=(
) )
# Tier 4 — after 24 hours (arrs + downloaders) # Tier 4 — after 24 hours (arrs + downloaders)
FALLBACK_HOST2_COVERS_HOST1_TIER4=( FALLBACK_HOST1_TIER4=(
"Sonarr-Gmer4Lfe" "Sonarr-Gmer4Lfe"
"Radarr-Gmer4Lfe" "Radarr-Gmer4Lfe"
"Lidarr-Gmer4Lfe" "Lidarr-Gmer4Lfe"
@@ -259,22 +277,22 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
--- ---
### host1.conf — HOST1 covering HOST2 ### host2.conf — HOST2's own services (started by HOST1 when HOST2 is down)
```bash ```bash
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS") HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
FALLBACK_HOST1_STOP_ON_NO_NET=() FALLBACK_HOST2_STOP_ON_NO_NET=()
# Tier 1 — immediate (HOST2's vital services) # Tier 1 — immediate (HOST2's vital services)
FALLBACK_HOST1_COVERS_HOST2_TIER1=( FALLBACK_HOST2_TIER1=(
"Gmer4Lfe.us-DDNS" # ALWAYS FIRST "Gmer4Lfe.us-DDNS" # ALWAYS FIRST
# HOST2's Tier 1 services — fill per HOST2's stack # HOST2's Tier 1 services — fill per HOST2's stack
) )
FALLBACK_HOST1_COVERS_HOST2_TIER2=(...) FALLBACK_HOST2_TIER2=(...)
FALLBACK_HOST1_COVERS_HOST2_TIER3=(...) FALLBACK_HOST2_TIER3=(...)
FALLBACK_HOST1_COVERS_HOST2_TIER4=(...) FALLBACK_HOST2_TIER4=(...)
# Tier delays for HOST2 outage # Tier delays for HOST2 outage
HOST2_TIER2_DELAY=240 HOST2_TIER2_DELAY=240
@@ -492,7 +510,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
1. Create the container on the covering server (stopped), with volume mounts pointing at 1. Create the container on the covering server (stopped), with volume mounts pointing at
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server) the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf 2. Add the container name to `FALLBACK_<THAT-HOST>_TIER*` in **that host's own** conf
in the appropriate tier position (dependency ordering — databases before apps) in the appropriate tier position (dependency ordering — databases before apps)
3. Verify: `fallback.sh --status` shows the container in the expected tier list 3. Verify: `fallback.sh --status` shows the container in the expected tier list
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid 4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
+3 -3
View File
@@ -115,7 +115,7 @@ Both servers run `fallback.sh` independently as a continuous background process.
makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds: makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:
```bash ```bash
ping REMOTE_TAILSCALE_IP # is the other server reachable? ping "$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")" # is the other server reachable?
ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8) ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8)
``` ```
@@ -220,8 +220,8 @@ determines which server is local and which is remote at runtime, then selects th
container arrays and tier delays from config via MY_ID. container arrays and tier delays from config via MY_ID.
``` ```
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf) HOST2 covers HOST1: FALLBACK_HOST1_TIER* (in host2.conf)
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf) HOST1 covers HOST2: FALLBACK_HOST2_TIER* (in host1.conf)
``` ```
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
+70 -1
View File
@@ -607,7 +607,7 @@ DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace"
BW_LIMIT=0 # KB/s, 0 = unlimited BW_LIMIT=0 # KB/s, 0 = unlimited
RETRY_COUNT=3 RETRY_COUNT=3
SLEEP=60 # seconds between retries SLEEP=60 # seconds between retries
ROOTFS_WARN_PCT=75 # abort if remote rootfs above this % ROOTFS_WARN=75 # abort if remote rootfs above this %
# Shared with Monitors/ # Shared with Monitors/
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db" BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
@@ -696,3 +696,72 @@ name doesn't match any profile key, or when testing a specific profile.
rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
rsync.sh /mnt/user/Media_Server/Emby --profile=emby rsync.sh /mnt/user/Media_Server/Emby --profile=emby
``` ```
### --seed
Skip the empty-remote-share guard. Normally `check_remote_share()` aborts when the target
directory on the remote is missing or empty, because that usually means the array is not
mounted and syncing into it would write onto the underlying root filesystem. `--seed` is the
deliberate exception, for the first push of a genuinely new share.
```bash
rsync.sh /mnt/user/New_Share --seed
```
### --merge-run
Bidirectional merge. Two passes:
```
Pass 1 pull remote → local, with --ignore-existing
Pass 2 push local → remote, with --delete
```
Local ends up authoritative: the remote may contribute files the local lacks, but never a
competing version of a file the local already has. Also triggered automatically when the
pre-scan detects ≥75% directory overlap with the remote.
**`--delete` is applied only if pass 1 completed.** Its entire justification is "local is now
the authoritative superset", and a pull that was capped by `RSYNC_MAX_RUNTIME_HOURS` or failed
outright means it is not — the remote still holds content the local never received. On an
incomplete pull the push runs **without** `--delete`, warns, and notifies. Local content still
propagates; nothing remote-unique is destroyed; the delete happens on a later run whose pull
actually succeeded.
> This was a real defect, fixed 2026-08-01. `--delete` used to be unconditional, so a capped
> or failed pull would permanently delete remote content the pull had not yet retrieved — and
> the timeout branch's own comment promised it would "resume the pull next run", which was
> impossible once the data was gone. If you are changing this function, that interlock is the
> part that matters.
---
## ━━━ SOURCE PATH GUARDS ━━━
`rsync.sh` pushes to `root@remote` at the **same absolute path**, and `--merge-run` adds
`--delete`. The source argument is therefore a remote-side hazard, not just a local one, so
two guards run before anything else:
| Guard | Rejects |
|-------|---------|
| Absolute path, ≥3 components | `/`, `/mnt`, `/mnt/user`, empty |
| Directory must exist locally | typo'd paths |
Every real job path clears both — see `HOST*_DAILY_SYNC_SHARES` and friends, all of which are
`/mnt/user/<Share>` or deeper.
Without the existence check, a mistyped path under `--merge-run` would be *created* locally by
the pull pass, filled from the remote, then pushed back with `--delete`.
---
## ━━━ RUNTIME CEILING ━━━
No single transfer attempt may exceed `RSYNC_MAX_RUNTIME_HOURS` (default 23). On timeout the
transfer is terminated and resumes on the next scheduled run.
This is only safe because `--partial` is in `DEFAULT_RSYNC_OPTS` — a terminated transfer picks
up where it left off rather than restarting. The cap exists so one huge or stuck transfer
cannot hold its per-profile lock indefinitely and starve every other profile of a turn.
Both `--merge-run` passes are capped independently.