FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added write back timing checks correlastes with tiers timing

Browse Source
This commit is contained in:
FailedProxy
2026-04-19 15:45:15 -04:00
parent ed9e927c41
commit 4399c81b6e
5 changed files with 407 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Failover/README-Failover.md
+48 -3
View File
@@ -18,6 +18,7 @@ HOST1 — unRAID-Gmer4Lfe (Primary)
HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
Hardware: Intel i5 10th gen — completely different hardware
RAM: 64GB
Location: Remote — 50 miles away
Owns: Gmer4Lfe.us DDNS
Runs: Its own service stack + mirrors HOST1 critical data
@@ -30,11 +31,55 @@ HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
- Container names for shared failover services
- Docker custom network names (so NPM can reach containers by name, not IP)
**Normal operation — HOST2 is essentially passive:**
**Split source of truth — each server owns different shares:**
HOST1 is the source of truth. It runs everything — all arrs, all downloads, all active services. HOST2 keeps its media mirror current via nightly rsync and waits. All media is mirrored so when Emby starts on HOST2 during failover it has the exact same library, same metadata, same watch history. No separate library, no separate database, no user-visible difference.
Both servers run arrs simultaneously — no conflict because they manage completely different shares:
Don't run arrs on both servers simultaneously — two instances writing to the same share causes conflicts and duplicate downloads. HOST2's arrs only start at Tier 4 (18hr+ outage) when genuine workflow continuity is needed.
```
HOST1 arrs — source of truth for: HOST2 arrs — source of truth for:
Movies (Radarr) Anime_Movies (his Radarr)
Tv_Shows (Sonarr) Anime_Shows (his Sonarr)
Music (Lidarr)
Each mirrors the other's shares in the opposite direction.
HOST1 mirrors anime FROM HOST2.
HOST2 mirrors movies/shows/music FROM HOST1.
```
Scheduling keeps them clean even within the same share window:
```
HOST2 arrs: midnight → noon managing and downloading anime
HOST1 Tdarr: 12:30 → 23:00 transcoding anime, syncs as source of truth
```
**The rule:** never run two instances of the same arr against the same share simultaneously. Different arrs managing different shares is perfectly fine.
At Tier 4 failover (18hr+ outage) each server's arr copies activate to cover the other's shares — only when the truth holder has genuinely been down long enough.
**Auth stack — runs warm on both servers simultaneously:**
NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running to serve his users through his domain every day. HOST1 is source of truth — all changes mirror to HOST2 every 15 minutes. Certs, proxy rules, user accounts, Authelia policies — all current on both servers at all times.
Early testing ran the auth stack cold on failover. Results: 30-60 seconds of broken authentication after DNS cut over. Emby clients hit HOST2 before auth was ready — reconnects failed, streams died. Running warm eliminates this window entirely.
**What actually starts from stopped on failover (HOST1 goes down):**
```
Emby ← starts from stopped on HOST2
DDNS updater ← HOST1's domain updater starts on HOST2
Already running — verified healthy, NOT cold started:
NPM — serving his domain continuously
LLDAP — authenticating his users continuously
Authelia — protecting his services continuously
Certs — mirrored, valid, already loaded
```
DNS cuts over in 1 minute. Auth is already ready. Library transcode users reconnect seamlessly through buffer. Live TV and direct play users notice and need to resume — the known, accepted tradeoff.
**Independence — always one rsync stop away:**
If HOST2 ever wants to fully separate: stop HOST1 pushing. Changes he makes stick permanently. His server becomes fully independent immediately. No script changes, no migration — just stop the rsync job. The ecosystem supports this by design.
Both servers run `failover.sh` as a background task continuously. Neither server knows what the other is doing — they only know what they can ping from their own network perspective.
Failover/failover.sh
+81 -21
View File
@@ -49,7 +49,11 @@
# 2. Pre-flight checks — remote array started, Docker healthy, rootfs not full
# 3. Stop remote DDNS first — prevents split brain DNS during rsync
# 4. Stop remote containers — clean state, no dirty writes during rsync
# 5. Rsync writeback — full bandwidth, clean source, minimal data
# 5. Tiered rsync writeback — skip if outage under threshold (short outages = cleaner to skip)
# Tier 1: skip if under HOST1_TIER1_WRITEBACK_DELAY (60min default)
# Tier 2: skip if under HOST1_TIER2_DELAY
# Tier 3: skip if under HOST1_TIER3_DELAY
# Tier 4: skip if under HOST*_TIER4_DELAY — opposing daily sync shares + edge cases
# 6. Start local containers — confirmed up before DNS cuts over
# 7. Start local DDNS — DNS cuts back ONLY after containers confirmed up
# 8. Return to NORMAL
@@ -283,11 +287,41 @@ get_tier4_delay() {
echo "$HOST1_TIER4_DELAY" || echo "$HOST2_TIER4_DELAY"
}
get_writeback_jobs() {
get_tier1_writeback_delay() {
[[ "$LOCAL_SERVER_NAME" == "$HOST2" ]] && \
echo "$HOST1_TIER1_WRITEBACK_DELAY" || echo "$HOST2_TIER1_WRITEBACK_DELAY"
}
get_writeback_jobs_for_tier() {
local tier="$1"
if [[ "$LOCAL_SERVER_NAME" == "$HOST2" ]]; then
echo "${FAILOVER_HOST1_WRITEBACK[@]}"
# HOST2 is covering HOST1 — write back HOST1's data
case "$tier" in
1) echo "${FAILOVER_HOST1_WRITEBACK_TIER1[@]}" ;;
2) echo "${FAILOVER_HOST1_WRITEBACK_TIER2[@]}" ;;
3) echo "${FAILOVER_HOST1_WRITEBACK_TIER3[@]}" ;;
4)
# Tier 4 — push HOST1's daily sync shares back (opposing orch list)
# These are HOST1's source-of-truth shares that HOST2's arrs managed during outage
# Plus any edge case paths defined in FAILOVER_HOST1_WRITEBACK_TIER4
echo "${HOST1_DAILY_SYNC_SHARES[@]}"
echo "${FAILOVER_HOST1_WRITEBACK_TIER4[@]}"
;;
esac
else
echo "${FAILOVER_HOST2_WRITEBACK[@]}"
# HOST1 is covering HOST2 — write back HOST2's data
case "$tier" in
1) echo "${FAILOVER_HOST2_WRITEBACK_TIER1[@]}" ;;
2) echo "${FAILOVER_HOST2_WRITEBACK_TIER2[@]}" ;;
3) echo "${FAILOVER_HOST2_WRITEBACK_TIER3[@]}" ;;
4)
# Tier 4 — push HOST2's daily sync shares back (opposing orch list)
# These are HOST2's source-of-truth shares that HOST1's arrs managed during outage
# Plus any edge case paths defined in FAILOVER_HOST2_WRITEBACK_TIER4
echo "${HOST2_DAILY_SYNC_SHARES[@]}"
echo "${FAILOVER_HOST2_WRITEBACK_TIER4[@]}"
;;
esac
fi
}
@@ -416,27 +450,53 @@ run_handback() {
success "All failover containers stopped locally"
# ── Step 4: Rsync writeback ──────────────────────────────────────────────────────────────
# Full bandwidth, clean source, minimal data
# Only critical appdata synced back — media files and downloads skipped
# Tiered writeback with skip window — short outages do not benefit from writeback.
# Emby syncs every 30min dirty (live container). Clean sync runs nightly at 2:30am.
# After a short outage HOST1's clean nightly state is more reliable than HOST2's
# dirty sync accumulation — skip writeback entirely for short outages.
#
# Tier 1 — skip if under HOST1_TIER1_WRITEBACK_DELAY (default 60min)
# Tier 2 — skip if under HOST1_TIER2_DELAY (reused — if Tier 2 never started, skip)
# Tier 3 — skip if under HOST1_TIER3_DELAY (reused — same logic)
# Tier 4 — always writeback — 18hr+ means meaningful delta accumulated
echo ""
echo "━━━ $ICON_SYNC Rsync Writeback ━━━"
local writeback_jobs
read -r -a writeback_jobs <<< "$(get_writeback_jobs)"
local outage_minutes=$(( ($(date +%s) - $(state_get failover_start)) / 60 ))
local tier1_wb_delay
tier1_wb_delay=$(get_tier1_writeback_delay)
if [[ ${#writeback_jobs[@]} -eq 0 ]]; then
info "No writeback jobs configured — skipping rsync"
else
for job in "${writeback_jobs[@]}"; do
[[ -z "$job" ]] && continue
info "Syncing: $job"
if [[ "$DRY_RUN" == false ]]; then
bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$job"
else
warn "DRY RUN — would rsync: $job"
fi
done
fi
info "Outage duration: ${outage_minutes}min"
run_writeback_tier() {
local tier="$1"
local threshold="$2"
local label="$3"
local jobs
read -r -a jobs <<< "$(get_writeback_jobs_for_tier "$tier")"
[[ ${#jobs[@]} -eq 0 ]] && return
if [[ "$outage_minutes" -ge "$threshold" ]]; then
info "Tier $tier writeback ($label) — outage ${outage_minutes}min >= ${threshold}min"
for job in "${jobs[@]}"; do
[[ -z "$job" ]] && continue
info "Syncing: $job"
if [[ "$DRY_RUN" == false ]]; then
bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$job"
else
warn "DRY RUN — would rsync: $job"
fi
done
else
info "Tier $tier writeback skipped — outage ${outage_minutes}min < ${threshold}min threshold"
info "Primary has cleaner state — no writeback needed"
fi
}
run_writeback_tier 1 "$tier1_wb_delay" "Emby + auth stack"
run_writeback_tier 2 "$HOST1_TIER2_DELAY" "NextCloud + Immich"
run_writeback_tier 3 "$HOST1_TIER3_DELAY" "secondary services"
run_writeback_tier 4 "$(get_tier4_delay)" "media shares + edge cases"
success "Writeback complete"