diff --git a/Fallback/Manual-Fallback.md b/Fallback/Manual-Fallback.md index ed7108c..691e1d7 100644 --- a/Fallback/Manual-Fallback.md +++ b/Fallback/Manual-Fallback.md @@ -105,18 +105,36 @@ internet connectivity. Most hosts leave this empty. --- ``` -FALLBACK_HOST*_COVERS_HOST*_TIER1=(...) -FALLBACK_HOST*_COVERS_HOST*_TIER2=(...) -FALLBACK_HOST*_COVERS_HOST*_TIER3=(...) -FALLBACK_HOST*_COVERS_HOST*_TIER4=(...) +FALLBACK_HOST*_TIER1=(...) +FALLBACK_HOST*_TIER2=(...) +FALLBACK_HOST*_TIER3=(...) +FALLBACK_HOST*_TIER4=(...) ``` -Containers this host starts for the remote host when the remote is down. TIER1 starts -immediately. TIER2–4 activate after the corresponding delay thresholds. -Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}` +**Each host declares its OWN services, in its OWN conf.** `FALLBACK_HOST2_TIER1` lives in +`host2.conf` and lists HOST2's vital containers — it is not a list HOST1 maintains. -The DDNS container for the remote's domain must be the first entry in TIER1 — DNS -coverage before anything else. +At runtime `fallback.sh` reads its **partner's** list: + +```bash +get_tier_containers() { + local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}" # note: REMOTE_ID, not MY_ID +``` + +So HOST1, covering HOST2, reads `FALLBACK_HOST2_TIER1` — a variable defined in `host2.conf` +and delivered to HOST1 through the partner conf cache (`conf_sync.sh`), because sparse +checkout means HOST1 never pulls `host2.conf` from git. + +This is why the naming is what it is. The alternative — each host keeping a copy of its +partner's container list — would need editing on both machines every time either one changed +a service, and the two copies would silently diverge. Declaring once, on the host that owns +the services, means a host is always the authority on what covering it requires. + +TIER1 starts immediately. TIER2–4 activate after their delay thresholds +(`HOST*_TIER2_DELAY` and friends, also in that host's own conf). + +The DDNS container for that host's domain must be the **first entry in TIER1** — DNS coverage +before anything else. --- @@ -181,15 +199,15 @@ FALLBACK_TEST_HANDBACK_WAIT=300 --- -### host2.conf — HOST2 covering HOST1 +### host1.conf — HOST1's own services (started by HOST2 when HOST1 is down) ```bash -HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS") +HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS") -FALLBACK_HOST2_STOP_ON_NO_NET=() +FALLBACK_HOST1_STOP_ON_NO_NET=() # Tier 1 — immediate (vital services + Live TV) -FALLBACK_HOST2_COVERS_HOST1_TIER1=( +FALLBACK_HOST1_TIER1=( "Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else "Emby" # media server — people are watching "NginxProxyManager" # reverse proxy — all external access routes through this @@ -205,7 +223,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER1=( ) # Tier 2 — after 4 hours (shared productivity services) -FALLBACK_HOST2_COVERS_HOST1_TIER2=( +FALLBACK_HOST1_TIER2=( "Postgres-NextCloud" # must start before NextCloud "NextCloud" "PostgreSQL-Immich" # must start before Immich @@ -214,7 +232,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER2=( ) # Tier 3 — after 12 hours (secondary services) -FALLBACK_HOST2_COVERS_HOST1_TIER3=( +FALLBACK_HOST1_TIER3=( "Organizrv2-Gmer4Lfe" "AdGuard-Home" "UptimeKuma" @@ -223,7 +241,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER3=( ) # Tier 4 — after 24 hours (arrs + downloaders) -FALLBACK_HOST2_COVERS_HOST1_TIER4=( +FALLBACK_HOST1_TIER4=( "Sonarr-Gmer4Lfe" "Radarr-Gmer4Lfe" "Lidarr-Gmer4Lfe" @@ -259,22 +277,22 @@ FALLBACK_HOST1_WRITEBACK_TIER3=( --- -### host1.conf — HOST1 covering HOST2 +### host2.conf — HOST2's own services (started by HOST1 when HOST2 is down) ```bash -HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS") +HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS") -FALLBACK_HOST1_STOP_ON_NO_NET=() +FALLBACK_HOST2_STOP_ON_NO_NET=() # Tier 1 — immediate (HOST2's vital services) -FALLBACK_HOST1_COVERS_HOST2_TIER1=( +FALLBACK_HOST2_TIER1=( "Gmer4Lfe.us-DDNS" # ALWAYS FIRST # HOST2's Tier 1 services — fill per HOST2's stack ) -FALLBACK_HOST1_COVERS_HOST2_TIER2=(...) -FALLBACK_HOST1_COVERS_HOST2_TIER3=(...) -FALLBACK_HOST1_COVERS_HOST2_TIER4=(...) +FALLBACK_HOST2_TIER2=(...) +FALLBACK_HOST2_TIER3=(...) +FALLBACK_HOST2_TIER4=(...) # Tier delays for HOST2 outage HOST2_TIER2_DELAY=240 @@ -492,7 +510,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n 1. Create the container on the covering server (stopped), with volume mounts pointing at the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server) -2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf +2. Add the container name to `FALLBACK__TIER*` in **that host's own** conf in the appropriate tier position (dependency ordering — databases before apps) 3. Verify: `fallback.sh --status` shows the container in the expected tier list 4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid diff --git a/Fallback/README-Fallback.md b/Fallback/README-Fallback.md index eb9aced..8241299 100644 --- a/Fallback/README-Fallback.md +++ b/Fallback/README-Fallback.md @@ -115,7 +115,7 @@ Both servers run `fallback.sh` independently as a continuous background process. makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds: ```bash -ping REMOTE_TAILSCALE_IP # is the other server reachable? +ping "$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")" # is the other server reachable? ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8) ``` @@ -220,8 +220,8 @@ determines which server is local and which is remote at runtime, then selects th container arrays and tier delays from config via MY_ID. ``` -HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf) -HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf) +HOST2 covers HOST1: FALLBACK_HOST1_TIER* (in host2.conf) +HOST1 covers HOST2: FALLBACK_HOST2_TIER* (in host1.conf) ``` Both servers run identical scripts. MY_ID selects the correct arrays. No hostname diff --git a/Rsync/Manual-Rsync.md b/Rsync/Manual-Rsync.md index 48a7ff5..44299fd 100644 --- a/Rsync/Manual-Rsync.md +++ b/Rsync/Manual-Rsync.md @@ -607,7 +607,7 @@ DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace" BW_LIMIT=0 # KB/s, 0 = unlimited RETRY_COUNT=3 SLEEP=60 # seconds between retries -ROOTFS_WARN_PCT=75 # abort if remote rootfs above this % +ROOTFS_WARN=75 # abort if remote rootfs above this % # Shared with Monitors/ BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db" @@ -696,3 +696,72 @@ name doesn't match any profile key, or when testing a specific profile. rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data rsync.sh /mnt/user/Media_Server/Emby --profile=emby ``` + +### --seed + +Skip the empty-remote-share guard. Normally `check_remote_share()` aborts when the target +directory on the remote is missing or empty, because that usually means the array is not +mounted and syncing into it would write onto the underlying root filesystem. `--seed` is the +deliberate exception, for the first push of a genuinely new share. + +```bash +rsync.sh /mnt/user/New_Share --seed +``` + +### --merge-run + +Bidirectional merge. Two passes: + +``` + Pass 1 pull remote → local, with --ignore-existing + Pass 2 push local → remote, with --delete +``` + +Local ends up authoritative: the remote may contribute files the local lacks, but never a +competing version of a file the local already has. Also triggered automatically when the +pre-scan detects ≥75% directory overlap with the remote. + +**`--delete` is applied only if pass 1 completed.** Its entire justification is "local is now +the authoritative superset", and a pull that was capped by `RSYNC_MAX_RUNTIME_HOURS` or failed +outright means it is not — the remote still holds content the local never received. On an +incomplete pull the push runs **without** `--delete`, warns, and notifies. Local content still +propagates; nothing remote-unique is destroyed; the delete happens on a later run whose pull +actually succeeded. + +> This was a real defect, fixed 2026-08-01. `--delete` used to be unconditional, so a capped +> or failed pull would permanently delete remote content the pull had not yet retrieved — and +> the timeout branch's own comment promised it would "resume the pull next run", which was +> impossible once the data was gone. If you are changing this function, that interlock is the +> part that matters. + +--- + +## ━━━ SOURCE PATH GUARDS ━━━ + +`rsync.sh` pushes to `root@remote` at the **same absolute path**, and `--merge-run` adds +`--delete`. The source argument is therefore a remote-side hazard, not just a local one, so +two guards run before anything else: + +| Guard | Rejects | +|-------|---------| +| Absolute path, ≥3 components | `/`, `/mnt`, `/mnt/user`, empty | +| Directory must exist locally | typo'd paths | + +Every real job path clears both — see `HOST*_DAILY_SYNC_SHARES` and friends, all of which are +`/mnt/user/` or deeper. + +Without the existence check, a mistyped path under `--merge-run` would be *created* locally by +the pull pass, filled from the remote, then pushed back with `--delete`. + +--- + +## ━━━ RUNTIME CEILING ━━━ + +No single transfer attempt may exceed `RSYNC_MAX_RUNTIME_HOURS` (default 23). On timeout the +transfer is terminated and resumes on the next scheduled run. + +This is only safe because `--partial` is in `DEFAULT_RSYNC_OPTS` — a terminated transfer picks +up where it left off rather than restarting. The cap exists so one huge or stuck transfer +cannot hold its per-profile lock indefinitely and starve every other profile of a turn. + +Both `--merge-run` passes are capped independently.