Correct Fallback tier variable naming and document rsync merge-run
The fallback docs used an older COVERS naming in the wrong conf file, and rsync's merge-run mode — the only path that carries --delete — was undocumented entirely.
This commit is contained in:
Gmer4Lfe
+42
-24
@@ -105,18 +105,36 @@ internet connectivity. Most hosts leave this empty.
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER1=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER2=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER3=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER4=(...)
|
||||
FALLBACK_HOST*_TIER1=(...)
|
||||
FALLBACK_HOST*_TIER2=(...)
|
||||
FALLBACK_HOST*_TIER3=(...)
|
||||
FALLBACK_HOST*_TIER4=(...)
|
||||
```
|
||||
Containers this host starts for the remote host when the remote is down. TIER1 starts
|
||||
immediately. TIER2–4 activate after the corresponding delay thresholds.
|
||||
|
||||
Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}`
|
||||
**Each host declares its OWN services, in its OWN conf.** `FALLBACK_HOST2_TIER1` lives in
|
||||
`host2.conf` and lists HOST2's vital containers — it is not a list HOST1 maintains.
|
||||
|
||||
The DDNS container for the remote's domain must be the first entry in TIER1 — DNS
|
||||
coverage before anything else.
|
||||
At runtime `fallback.sh` reads its **partner's** list:
|
||||
|
||||
```bash
|
||||
get_tier_containers() {
|
||||
local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}" # note: REMOTE_ID, not MY_ID
|
||||
```
|
||||
|
||||
So HOST1, covering HOST2, reads `FALLBACK_HOST2_TIER1` — a variable defined in `host2.conf`
|
||||
and delivered to HOST1 through the partner conf cache (`conf_sync.sh`), because sparse
|
||||
checkout means HOST1 never pulls `host2.conf` from git.
|
||||
|
||||
This is why the naming is what it is. The alternative — each host keeping a copy of its
|
||||
partner's container list — would need editing on both machines every time either one changed
|
||||
a service, and the two copies would silently diverge. Declaring once, on the host that owns
|
||||
the services, means a host is always the authority on what covering it requires.
|
||||
|
||||
TIER1 starts immediately. TIER2–4 activate after their delay thresholds
|
||||
(`HOST*_TIER2_DELAY` and friends, also in that host's own conf).
|
||||
|
||||
The DDNS container for that host's domain must be the **first entry in TIER1** — DNS coverage
|
||||
before anything else.
|
||||
|
||||
---
|
||||
|
||||
@@ -181,15 +199,15 @@ FALLBACK_TEST_HANDBACK_WAIT=300
|
||||
|
||||
---
|
||||
|
||||
### host2.conf — HOST2 covering HOST1
|
||||
### host1.conf — HOST1's own services (started by HOST2 when HOST1 is down)
|
||||
|
||||
```bash
|
||||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||||
|
||||
FALLBACK_HOST2_STOP_ON_NO_NET=()
|
||||
FALLBACK_HOST1_STOP_ON_NO_NET=()
|
||||
|
||||
# Tier 1 — immediate (vital services + Live TV)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
||||
FALLBACK_HOST1_TIER1=(
|
||||
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
|
||||
"Emby" # media server — people are watching
|
||||
"NginxProxyManager" # reverse proxy — all external access routes through this
|
||||
@@ -205,7 +223,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
||||
)
|
||||
|
||||
# Tier 2 — after 4 hours (shared productivity services)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER2=(
|
||||
FALLBACK_HOST1_TIER2=(
|
||||
"Postgres-NextCloud" # must start before NextCloud
|
||||
"NextCloud"
|
||||
"PostgreSQL-Immich" # must start before Immich
|
||||
@@ -214,7 +232,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER2=(
|
||||
)
|
||||
|
||||
# Tier 3 — after 12 hours (secondary services)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER3=(
|
||||
FALLBACK_HOST1_TIER3=(
|
||||
"Organizrv2-Gmer4Lfe"
|
||||
"AdGuard-Home"
|
||||
"UptimeKuma"
|
||||
@@ -223,7 +241,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER3=(
|
||||
)
|
||||
|
||||
# Tier 4 — after 24 hours (arrs + downloaders)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER4=(
|
||||
FALLBACK_HOST1_TIER4=(
|
||||
"Sonarr-Gmer4Lfe"
|
||||
"Radarr-Gmer4Lfe"
|
||||
"Lidarr-Gmer4Lfe"
|
||||
@@ -259,22 +277,22 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
|
||||
|
||||
---
|
||||
|
||||
### host1.conf — HOST1 covering HOST2
|
||||
### host2.conf — HOST2's own services (started by HOST1 when HOST2 is down)
|
||||
|
||||
```bash
|
||||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||||
|
||||
FALLBACK_HOST1_STOP_ON_NO_NET=()
|
||||
FALLBACK_HOST2_STOP_ON_NO_NET=()
|
||||
|
||||
# Tier 1 — immediate (HOST2's vital services)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
|
||||
FALLBACK_HOST2_TIER1=(
|
||||
"Gmer4Lfe.us-DDNS" # ALWAYS FIRST
|
||||
# HOST2's Tier 1 services — fill per HOST2's stack
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER2=(...)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER3=(...)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER4=(...)
|
||||
FALLBACK_HOST2_TIER2=(...)
|
||||
FALLBACK_HOST2_TIER3=(...)
|
||||
FALLBACK_HOST2_TIER4=(...)
|
||||
|
||||
# Tier delays for HOST2 outage
|
||||
HOST2_TIER2_DELAY=240
|
||||
@@ -492,7 +510,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
|
||||
|
||||
1. Create the container on the covering server (stopped), with volume mounts pointing at
|
||||
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
|
||||
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf
|
||||
2. Add the container name to `FALLBACK_<THAT-HOST>_TIER*` in **that host's own** conf
|
||||
in the appropriate tier position (dependency ordering — databases before apps)
|
||||
3. Verify: `fallback.sh --status` shows the container in the expected tier list
|
||||
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
|
||||
|
||||
@@ -115,7 +115,7 @@ Both servers run `fallback.sh` independently as a continuous background process.
|
||||
makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:
|
||||
|
||||
```bash
|
||||
ping REMOTE_TAILSCALE_IP # is the other server reachable?
|
||||
ping "$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")" # is the other server reachable?
|
||||
ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8)
|
||||
```
|
||||
|
||||
@@ -220,8 +220,8 @@ determines which server is local and which is remote at runtime, then selects th
|
||||
container arrays and tier delays from config via MY_ID.
|
||||
|
||||
```
|
||||
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf)
|
||||
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf)
|
||||
HOST2 covers HOST1: FALLBACK_HOST1_TIER* (in host2.conf)
|
||||
HOST1 covers HOST2: FALLBACK_HOST2_TIER* (in host1.conf)
|
||||
```
|
||||
|
||||
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
|
||||
|
||||
+70
-1
@@ -607,7 +607,7 @@ DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace"
|
||||
BW_LIMIT=0 # KB/s, 0 = unlimited
|
||||
RETRY_COUNT=3
|
||||
SLEEP=60 # seconds between retries
|
||||
ROOTFS_WARN_PCT=75 # abort if remote rootfs above this %
|
||||
ROOTFS_WARN=75 # abort if remote rootfs above this %
|
||||
|
||||
# Shared with Monitors/
|
||||
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
|
||||
@@ -696,3 +696,72 @@ name doesn't match any profile key, or when testing a specific profile.
|
||||
rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
|
||||
rsync.sh /mnt/user/Media_Server/Emby --profile=emby
|
||||
```
|
||||
|
||||
### --seed
|
||||
|
||||
Skip the empty-remote-share guard. Normally `check_remote_share()` aborts when the target
|
||||
directory on the remote is missing or empty, because that usually means the array is not
|
||||
mounted and syncing into it would write onto the underlying root filesystem. `--seed` is the
|
||||
deliberate exception, for the first push of a genuinely new share.
|
||||
|
||||
```bash
|
||||
rsync.sh /mnt/user/New_Share --seed
|
||||
```
|
||||
|
||||
### --merge-run
|
||||
|
||||
Bidirectional merge. Two passes:
|
||||
|
||||
```
|
||||
Pass 1 pull remote → local, with --ignore-existing
|
||||
Pass 2 push local → remote, with --delete
|
||||
```
|
||||
|
||||
Local ends up authoritative: the remote may contribute files the local lacks, but never a
|
||||
competing version of a file the local already has. Also triggered automatically when the
|
||||
pre-scan detects ≥75% directory overlap with the remote.
|
||||
|
||||
**`--delete` is applied only if pass 1 completed.** Its entire justification is "local is now
|
||||
the authoritative superset", and a pull that was capped by `RSYNC_MAX_RUNTIME_HOURS` or failed
|
||||
outright means it is not — the remote still holds content the local never received. On an
|
||||
incomplete pull the push runs **without** `--delete`, warns, and notifies. Local content still
|
||||
propagates; nothing remote-unique is destroyed; the delete happens on a later run whose pull
|
||||
actually succeeded.
|
||||
|
||||
> This was a real defect, fixed 2026-08-01. `--delete` used to be unconditional, so a capped
|
||||
> or failed pull would permanently delete remote content the pull had not yet retrieved — and
|
||||
> the timeout branch's own comment promised it would "resume the pull next run", which was
|
||||
> impossible once the data was gone. If you are changing this function, that interlock is the
|
||||
> part that matters.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SOURCE PATH GUARDS ━━━
|
||||
|
||||
`rsync.sh` pushes to `root@remote` at the **same absolute path**, and `--merge-run` adds
|
||||
`--delete`. The source argument is therefore a remote-side hazard, not just a local one, so
|
||||
two guards run before anything else:
|
||||
|
||||
| Guard | Rejects |
|
||||
|-------|---------|
|
||||
| Absolute path, ≥3 components | `/`, `/mnt`, `/mnt/user`, empty |
|
||||
| Directory must exist locally | typo'd paths |
|
||||
|
||||
Every real job path clears both — see `HOST*_DAILY_SYNC_SHARES` and friends, all of which are
|
||||
`/mnt/user/<Share>` or deeper.
|
||||
|
||||
Without the existence check, a mistyped path under `--merge-run` would be *created* locally by
|
||||
the pull pass, filled from the remote, then pushed back with `--delete`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RUNTIME CEILING ━━━
|
||||
|
||||
No single transfer attempt may exceed `RSYNC_MAX_RUNTIME_HOURS` (default 23). On timeout the
|
||||
transfer is terminated and resumes on the next scheduled run.
|
||||
|
||||
This is only safe because `--partial` is in `DEFAULT_RSYNC_OPTS` — a terminated transfer picks
|
||||
up where it left off rather than restarting. The cap exists so one huge or stuck transfer
|
||||
cannot hold its per-profile lock indefinitely and starve every other profile of a turn.
|
||||
|
||||
Both `--merge-run` passes are capped independently.
|
||||
|
||||
Reference in New Issue
Block a user