added write back timing checks correlastes with tiers timing

2026-04-19 15:45:15 -04:00
parent ed9e927c41
commit 4399c81b6e
5 changed files with 407 additions and 79 deletions
@@ -18,6 +18,7 @@ HOST1 — unRAID-Gmer4Lfe (Primary)

 HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
  Hardware:   Intel i5 10th gen — completely different hardware
+  RAM:        64GB
  Location:   Remote — 50 miles away
  Owns:       Gmer4Lfe.us DDNS
  Runs:       Its own service stack + mirrors HOST1 critical data
@@ -30,11 +31,55 @@ HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
 - Container names for shared failover services
 - Docker custom network names (so NPM can reach containers by name, not IP)

-**Normal operation — HOST2 is essentially passive:**
+**Split source of truth — each server owns different shares:**

-HOST1 is the source of truth. It runs everything — all arrs, all downloads, all active services. HOST2 keeps its media mirror current via nightly rsync and waits. All media is mirrored so when Emby starts on HOST2 during failover it has the exact same library, same metadata, same watch history. No separate library, no separate database, no user-visible difference.
+Both servers run arrs simultaneously — no conflict because they manage completely different shares:

-Don't run arrs on both servers simultaneously — two instances writing to the same share causes conflicts and duplicate downloads. HOST2's arrs only start at Tier 4 (18hr+ outage) when genuine workflow continuity is needed.
+```
+HOST1 arrs — source of truth for:    HOST2 arrs — source of truth for:
+  Movies    (Radarr)                   Anime_Movies  (his Radarr)
+  Tv_Shows  (Sonarr)                   Anime_Shows   (his Sonarr)
+  Music     (Lidarr)
+
+Each mirrors the other's shares in the opposite direction.
+HOST1 mirrors anime FROM HOST2.
+HOST2 mirrors movies/shows/music FROM HOST1.
+```
+
+Scheduling keeps them clean even within the same share window:
+```
+HOST2 arrs:    midnight → noon     managing and downloading anime
+HOST1 Tdarr:   12:30   → 23:00    transcoding anime, syncs as source of truth
+```
+
+**The rule:** never run two instances of the same arr against the same share simultaneously. Different arrs managing different shares is perfectly fine.
+
+At Tier 4 failover (18hr+ outage) each server's arr copies activate to cover the other's shares — only when the truth holder has genuinely been down long enough.
+
+**Auth stack — runs warm on both servers simultaneously:**
+
+NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running to serve his users through his domain every day. HOST1 is source of truth — all changes mirror to HOST2 every 15 minutes. Certs, proxy rules, user accounts, Authelia policies — all current on both servers at all times.
+
+Early testing ran the auth stack cold on failover. Results: 30-60 seconds of broken authentication after DNS cut over. Emby clients hit HOST2 before auth was ready — reconnects failed, streams died. Running warm eliminates this window entirely.
+
+**What actually starts from stopped on failover (HOST1 goes down):**
+
+```
+Emby          ← starts from stopped on HOST2
+DDNS updater  ← HOST1's domain updater starts on HOST2
+
+Already running — verified healthy, NOT cold started:
+  NPM      — serving his domain continuously
+  LLDAP    — authenticating his users continuously
+  Authelia — protecting his services continuously
+  Certs    — mirrored, valid, already loaded
+```
+
+DNS cuts over in 1 minute. Auth is already ready. Library transcode users reconnect seamlessly through buffer. Live TV and direct play users notice and need to resume — the known, accepted tradeoff.
+
+**Independence — always one rsync stop away:**
+
+If HOST2 ever wants to fully separate: stop HOST1 pushing. Changes he makes stick permanently. His server becomes fully independent immediately. No script changes, no migration — just stop the rsync job. The ecosystem supports this by design.

 Both servers run `failover.sh` as a background task continuously. Neither server knows what the other is doing — they only know what they can ping from their own network perspective.

@@ -49,7 +49,11 @@
 #   2. Pre-flight checks       — remote array started, Docker healthy, rootfs not full
 #   3. Stop remote DDNS first  — prevents split brain DNS during rsync
 #   4. Stop remote containers  — clean state, no dirty writes during rsync
-#   5. Rsync writeback         — full bandwidth, clean source, minimal data
+#   5. Tiered rsync writeback  — skip if outage under threshold (short outages = cleaner to skip)
+#                                Tier 1: skip if under HOST1_TIER1_WRITEBACK_DELAY (60min default)
+#                                Tier 2: skip if under HOST1_TIER2_DELAY
+#                                Tier 3: skip if under HOST1_TIER3_DELAY
+#                                Tier 4: skip if under HOST*_TIER4_DELAY — opposing daily sync shares + edge cases
 #   6. Start local containers  — confirmed up before DNS cuts over
 #   7. Start local DDNS        — DNS cuts back ONLY after containers confirmed up
 #   8. Return to NORMAL
@@ -283,11 +287,41 @@ get_tier4_delay() {
        echo "$HOST1_TIER4_DELAY" || echo "$HOST2_TIER4_DELAY"
 }

-get_writeback_jobs() {
+get_tier1_writeback_delay() {
+    [[ "$LOCAL_SERVER_NAME" == "$HOST2" ]] && \
+        echo "$HOST1_TIER1_WRITEBACK_DELAY" || echo "$HOST2_TIER1_WRITEBACK_DELAY"
+}
+
+get_writeback_jobs_for_tier() {
+    local tier="$1"
    if [[ "$LOCAL_SERVER_NAME" == "$HOST2" ]]; then
-        echo "${FAILOVER_HOST1_WRITEBACK[@]}"
+        # HOST2 is covering HOST1 — write back HOST1's data
+        case "$tier" in
+            1) echo "${FAILOVER_HOST1_WRITEBACK_TIER1[@]}" ;;
+            2) echo "${FAILOVER_HOST1_WRITEBACK_TIER2[@]}" ;;
+            3) echo "${FAILOVER_HOST1_WRITEBACK_TIER3[@]}" ;;
+            4)
+                # Tier 4 — push HOST1's daily sync shares back (opposing orch list)
+                # These are HOST1's source-of-truth shares that HOST2's arrs managed during outage
+                # Plus any edge case paths defined in FAILOVER_HOST1_WRITEBACK_TIER4
+                echo "${HOST1_DAILY_SYNC_SHARES[@]}"
+                echo "${FAILOVER_HOST1_WRITEBACK_TIER4[@]}"
+                ;;
+        esac
    else
-        echo "${FAILOVER_HOST2_WRITEBACK[@]}"
+        # HOST1 is covering HOST2 — write back HOST2's data
+        case "$tier" in
+            1) echo "${FAILOVER_HOST2_WRITEBACK_TIER1[@]}" ;;
+            2) echo "${FAILOVER_HOST2_WRITEBACK_TIER2[@]}" ;;
+            3) echo "${FAILOVER_HOST2_WRITEBACK_TIER3[@]}" ;;
+            4)
+                # Tier 4 — push HOST2's daily sync shares back (opposing orch list)
+                # These are HOST2's source-of-truth shares that HOST1's arrs managed during outage
+                # Plus any edge case paths defined in FAILOVER_HOST2_WRITEBACK_TIER4
+                echo "${HOST2_DAILY_SYNC_SHARES[@]}"
+                echo "${FAILOVER_HOST2_WRITEBACK_TIER4[@]}"
+                ;;
+        esac
    fi
 }

@@ -416,27 +450,53 @@ run_handback() {
    success "All failover containers stopped locally"

    # ── Step 4: Rsync writeback ──────────────────────────────────────────────────────────────
-    # Full bandwidth, clean source, minimal data
-    # Only critical appdata synced back — media files and downloads skipped
+    # Tiered writeback with skip window — short outages do not benefit from writeback.
+    # Emby syncs every 30min dirty (live container). Clean sync runs nightly at 2:30am.
+    # After a short outage HOST1's clean nightly state is more reliable than HOST2's
+    # dirty sync accumulation — skip writeback entirely for short outages.
+    #
+    # Tier 1 — skip if under HOST1_TIER1_WRITEBACK_DELAY (default 60min)
+    # Tier 2 — skip if under HOST1_TIER2_DELAY (reused — if Tier 2 never started, skip)
+    # Tier 3 — skip if under HOST1_TIER3_DELAY (reused — same logic)
+    # Tier 4 — always writeback — 18hr+ means meaningful delta accumulated
    echo ""
    echo "━━━ $ICON_SYNC Rsync Writeback ━━━"

-    local writeback_jobs
-    read -r -a writeback_jobs <<< "$(get_writeback_jobs)"
+    local outage_minutes=$(( ($(date +%s) - $(state_get failover_start)) / 60 ))
+    local tier1_wb_delay
+    tier1_wb_delay=$(get_tier1_writeback_delay)

-    if [[ ${#writeback_jobs[@]} -eq 0 ]]; then
-        info "No writeback jobs configured — skipping rsync"
-    else
-        for job in "${writeback_jobs[@]}"; do
-            [[ -z "$job" ]] && continue
-            info "Syncing: $job"
-            if [[ "$DRY_RUN" == false ]]; then
-                bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$job"
-            else
-                warn "DRY RUN — would rsync: $job"
-            fi
-        done
-    fi
+    info "Outage duration: ${outage_minutes}min"
+
+    run_writeback_tier() {
+        local tier="$1"
+        local threshold="$2"
+        local label="$3"
+        local jobs
+        read -r -a jobs <<< "$(get_writeback_jobs_for_tier "$tier")"
+        [[ ${#jobs[@]} -eq 0 ]] && return
+
+        if [[ "$outage_minutes" -ge "$threshold" ]]; then
+            info "Tier $tier writeback ($label) — outage ${outage_minutes}min >= ${threshold}min"
+            for job in "${jobs[@]}"; do
+                [[ -z "$job" ]] && continue
+                info "Syncing: $job"
+                if [[ "$DRY_RUN" == false ]]; then
+                    bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$job"
+                else
+                    warn "DRY RUN — would rsync: $job"
+                fi
+            done
+        else
+            info "Tier $tier writeback skipped — outage ${outage_minutes}min < ${threshold}min threshold"
+            info "Primary has cleaner state — no writeback needed"
+        fi
+    }
+
+    run_writeback_tier 1 "$tier1_wb_delay"        "Emby + auth stack"
+    run_writeback_tier 2 "$HOST1_TIER2_DELAY"     "NextCloud + Immich"
+    run_writeback_tier 3 "$HOST1_TIER3_DELAY"     "secondary services"
+    run_writeback_tier 4 "$(get_tier4_delay)"     "media shares + edge cases"

    success "Writeback complete"

@@ -115,7 +115,7 @@

 # ━━━ Rsync Defaults ━━━
 # Global fallback values used when no profile match is found for a directory.
-# Shares in DAILY_SYNC_SHARES always use these globals — no profile is defined for them.
+# Shares in HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES always use these globals — no profile is defined for them.
 # Appdata shares (Arrs_Stack, Critical-Data etc.) match profiles by directory basename.
 # If a profile key exists in a PROFILE_* array that value overrides the global.
 # If a profile key is missing the global below is used as the fallback.
@@ -143,24 +143,40 @@

 # ━━━ Daily Sync Shares ━━━
 # Media shares synced once daily by Orchestrators/daily_sync.sh.
-# These shares have no profile entry — all use DEFAULT_RSYNC_OPTS above.
-# Add or remove paths here to control what syncs each night.
+# Each server only syncs the shares it is source of truth for — direction is automatic.
+# detect_hosts() determines which server is running and picks the correct list.
+#
+# HOST1 pushes its truth shares TO HOST2.
+# HOST2 pushes its truth shares TO HOST1.
+# Never both pushing the same share — one server is always the truth holder.
+#
+# These shares use DEFAULT_RSYNC_OPTS — no profile entry needed.
 # For shares needing custom bandwidth or container stops — create a profile below instead.
-DAILY_SYNC_SHARES=(
-    /mnt/user/Anime_Movies-Old
-    /mnt/user/Anime_Shows-Old
-    /mnt/user/Anime_Shows
+
+# HOST1 truth shares — pushed from HOST1 to HOST2 nightly
+HOST1_DAILY_SYNC_SHARES=(
    /mnt/user/Books
    /mnt/user/Intros
    /mnt/user/Kids_Movies
    /mnt/user/Kids_Tv_Shows
    /mnt/user/Movies
+    /mnt/user/Music
    /mnt/user/Music_Videos
    /mnt/user/Nextcloud
    /mnt/user/stand-up_comedy
+    /mnt/user/Sports
    /mnt/user/Tv_Shows
 )

+# HOST2 truth shares — pushed from HOST2 to HOST1 nightly
+# HOST2 is source of truth for anime — his arrs manage these shares
+HOST2_DAILY_SYNC_SHARES=(
+    /mnt/user/Anime_Movies
+    /mnt/user/Anime_Movies-Old
+    /mnt/user/Anime_Shows
+    /mnt/user/Anime_Shows-Old
+)
+
 # ━━━ Personal Encrypted Shares ━━━
 # Personal shares synced to the remote server for offsite backup.
 # These are independent of the failover container stack — data backup only.
@@ -535,27 +551,84 @@ HOST2_TIER4_DELAY=1080
 # Containers are stopped before this runs — clean source, no competing writes.
 # Full bandwidth available — DDNS stopped, containers stopped, nothing competing.
 #
-# Priority:
-#   Critical — Emby userdata/playstates (small, fast, important)
-#   Critical — Auth stack data
-#   Skip     — Media files (already on primary, never moved)
-#   Skip     — Downloads (start fresh — cleaner than syncing partial state)
+# ── WRITEBACK SKIP WINDOW ─────────────────────────────────────────────────────────────────────
+# Short outages do not benefit from writeback — the covering server accumulated dirty
+# or minimal data not worth writing over the primary's cleaner state.
 #
-# Format: "/path/to/source" — matched to rsync profile by directory basename
+# Emby syncs every 30min from a live running container (dirty sync).
+# A clean full sync runs nightly at 2:30am with Emby stopped.
+# After a short outage HOST1's nightly clean state is more reliable
+# than HOST2's dirty 30min sync data — skip writeback entirely.
+#
+# Real world outage profile:
+#   2-10 minutes  — power blip, most common → skip writeback
+#   10-60 minutes — ISP issue, router restart → skip writeback
+#   1hr+          — actual problem → writeback worthwhile
+#   18hr+         — Tier 4 activated → always writeback
+#
+# Tier 1 writeback delay — separate from Tier 1 start delay (Tier 1 always starts immediately)
+# Tier 2+ reuse their existing TIER_DELAY vars — if containers started, time passed = writeback warranted
+# Tier 4 writeback — skips if under HOST*_TIER4_DELAY (same threshold as container start)
+#   Only runs if outage was long enough to activate Tier 4 containers
+#   Automatically uses the opposing host's daily sync share list (HOST*_DAILY_SYNC_SHARES)
+#   FAILOVER_HOST*_WRITEBACK_TIER4 is for edge cases only — empty by default

-# HOST1 writeback — run by HOST2 during HOST1 handback
-FAILOVER_HOST1_WRITEBACK=(
-    "/mnt/user/appdata-Failover/Critical-Data"  # auth stack — Authelia, Mariadb, Redis, LLDAP, NPM
-    "/mnt/user/appdata-Failover/Important-Data" # NextCloud + Postgres
+HOST1_TIER1_WRITEBACK_DELAY=60  # minutes — skip Tier 1 writeback if outage under this
+HOST2_TIER1_WRITEBACK_DELAY=60  # minutes — skip Tier 1 writeback if outage under this
+# Tier 2 writeback threshold = HOST1_TIER2_DELAY (reused)
+# Tier 3 writeback threshold = HOST1_TIER3_DELAY (reused)
+# Tier 4 always writebacks — no threshold
+
+# ── WRITEBACK JOB LISTS ───────────────────────────────────────────────────────────────────────
+# Organised by tier — writeback runs per tier based on outage duration
+# Priority:
+#   Tier 1 — Emby userdata, auth stack — small, fast, most important
+#   Tier 2 — NextCloud, Immich — user files that may have changed
+#   Tier 3 — secondary services
+#   Tier 4 — always runs if Tier 4 activated — arrs accumulated meaningful state
+#   Skip   — media files (already on primary, never moved)
+#   Skip   — downloads (start fresh — cleaner than syncing partial state)
+
+# HOST1 writeback tiers — run by HOST2 during HOST1 handback
+FAILOVER_HOST1_WRITEBACK_TIER1=(
    "/mnt/user/appdata-Failover/Emby"           # Emby userdata, playstates, metadata
-    "/mnt/user/appdata-Failover/Gmer4Lfe"       # server specific appdata — Organizr, UptimeKuma
-#   "/mnt/user/appdata-Failover/Arrs_Stack"     # skip — arrs start fresh on handback
+    "/mnt/user/appdata-Failover/Critical-Data"  # auth stack — Authelia, Mariadb, Redis, LLDAP, NPM
 )

-# HOST2 writeback — run by HOST1 during HOST2 handback
-FAILOVER_HOST2_WRITEBACK=(
-    # "/mnt/user/appdata-Failover/Jayred365"    # HOST2 specific appdata
-    # "container-placeholder"
+FAILOVER_HOST1_WRITEBACK_TIER2=(
+    "/mnt/user/appdata-Failover/Important-Data" # NextCloud + Postgres
+)
+
+FAILOVER_HOST1_WRITEBACK_TIER3=(
+    "/mnt/user/appdata-Failover/Gmer4Lfe"       # server specific appdata — Organizr, UptimeKuma
+)
+
+FAILOVER_HOST1_WRITEBACK_TIER4=(
+    # Edge case paths outside of normal HOST1_DAILY_SYNC_SHARES
+    # Tier 4 automatically syncs HOST1_DAILY_SYNC_SHARES back — add extras here only
+    # Examples: a share that exists but isn't in the daily sync list,
+    #           a special appdata path only relevant after extended outage
+    # "/mnt/user/SomeSpecialShare"
+)
+
+# HOST2 writeback tiers — run by HOST1 during HOST2 handback
+FAILOVER_HOST2_WRITEBACK_TIER1=(
+    # "/mnt/user/appdata-Failover/Jayred365-Emby"
+    # "/mnt/user/appdata-Failover/Jayred365-Critical"
+)
+
+FAILOVER_HOST2_WRITEBACK_TIER2=(
+    # "/mnt/user/appdata-Failover/Jayred365-Important"
+)
+
+FAILOVER_HOST2_WRITEBACK_TIER3=(
+    # "/mnt/user/appdata-Failover/Jayred365"
+)
+
+FAILOVER_HOST2_WRITEBACK_TIER4=(
+    # Edge case paths outside of normal HOST2_DAILY_SYNC_SHARES
+    # Tier 4 automatically syncs HOST2_DAILY_SYNC_SHARES back — add extras here only
+    # "/mnt/user/SomeSpecialShare"
 )

 # ==============================================================================================
@@ -893,13 +966,13 @@ MEDIA_MAINTENANCE_JOBS=(
 #   and new sessions land on SSD permanently for that container run.

    RAMDISK_PATH="/mnt/ramdisk_transcodes"  # tmpfs mount point created at array start
-    RAMDISK_SIZE="8G"                       # ceiling — tmpfs only uses RAM actually needed
+    RAMDISK_SIZE="10G"                       # ceiling — tmpfs only uses RAM actually needed
    TRANSCODE_LINK="/mnt/ram-transcode"      # symlink Emby points at — location never changes
    TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback location

 # Usage thresholds in GB — hysteresis gap between WARN and LOW prevents flip-flop
-    RAMDISK_WARN_GB=6.8     # flip symlink to SSD at or above this usage
-    RAMDISK_LOW_GB=5.5      # flip symlink back to ramdisk when usage drops here
+    RAMDISK_WARN_GB=8.8     # flip symlink to SSD at or above this usage
+    RAMDISK_LOW_GB=6.5      # flip symlink back to ramdisk when usage drops here
    RAMDISK_SSD_MIN_GB=20   # minimum free GB on SSD required before allowing flip to SSD

 # Cleanup age thresholds — files must be older than these AND not open by any process
@@ -953,9 +1026,10 @@ CERT_MONITOR_DOMAINS=(
 # ━━━ Backup Verify ━━━
 # Verifies the rsync mirror is healthy by comparing random file checksums between servers.
 # Uses existing SSH keys — no additional configuration needed beyond the share list.
-# Leave BACKUP_VERIFY_SHARES empty to automatically use DAILY_SYNC_SHARES as the target list.
+# Leave BACKUP_VERIFY_SHARES empty to automatically use the local host's daily sync shares
+# (HOST1_DAILY_SYNC_SHARES or HOST2_DAILY_SYNC_SHARES based on detect_hosts()).
 BACKUP_VERIFY_SHARES=(
-    # leave empty to use DAILY_SYNC_SHARES automatically
+    # leave empty to use host-specific daily sync shares automatically
 )
    BACKUP_VERIFY_SAMPLE=10     # number of files to randomly sample per share per run
    BACKUP_VERIFY_MIN_SIZE=1M   # skip files smaller than this — avoids tiny junk files
@@ -2,9 +2,18 @@
 # -----------------------------------------------------------------------------------------------
 # --------------------------------- Daily Sync Orchestrator ------------------------------------
 # -----------------------------------------------------------------------------------------------
-# Runs all bulk media shares sequentially. Scheduled via unRAID User Scripts plugin at 1am.
+# Runs all media shares sequentially in the correct direction for the local server.
+# Each server pushes its own source-of-truth shares to the remote — direction is automatic.
+#
+# HOST1 pushes: its truth shares (Movies, Tv_Shows, Music etc.) + personal → HOST2
+# HOST2 pushes: its truth shares (Anime_Shows, Anime_Movies etc.) + personal → HOST1
+#
+# detect_hosts() determines which server is running the script at runtime.
+# Share lists are configured per host in Master.conf — no script changes needed
+# to add, remove, or reconfigure shares.
+#
 # Per-share rsync handled by rsync.sh — this script tracks pass/fail and total time only.
-# Add or remove shares in Master.conf under DAILY_SYNC_SHARES.
+# Scheduled via unRAID User Scripts plugin at 1am on both servers.
 # -----------------------------------------------------------------------------------------------

 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -35,21 +44,28 @@ check_connectivity
 check_remote_rootfs

 # -----------------------------------------------------------------------------------------------
-# Tracking
+# Build share list — host-specific truth shares + personal shares
+# Each server only syncs the shares it is source of truth for
+# Personal shares appended after media shares
 # -----------------------------------------------------------------------------------------------
 PASS=()
 FAIL=()
 SHARE_TIMES=()
 TOTAL_START=$(date +%s)

-# Build combined share list — media shares + personal shares for this host
-ALL_SHARES=("${DAILY_SYNC_SHARES[@]}")
+ALL_SHARES=()

 if [[ "$LOCAL_SERVER_NAME" == "$HOST1" ]]; then
+    for share in "${HOST1_DAILY_SYNC_SHARES[@]}"; do
+        [[ -n "$share" ]] && ALL_SHARES+=("$share")
+    done
    for share in "${HOST1_PERSONAL_SHARES[@]}"; do
        [[ -n "$share" ]] && ALL_SHARES+=("$share")
    done
 elif [[ "$LOCAL_SERVER_NAME" == "$HOST2" ]]; then
+    for share in "${HOST2_DAILY_SYNC_SHARES[@]}"; do
+        [[ -n "$share" ]] && ALL_SHARES+=("$share")
+    done
    for share in "${HOST2_PERSONAL_SHARES[@]}"; do
        [[ -n "$share" ]] && ALL_SHARES+=("$share")
    done
@@ -4,6 +4,51 @@ A modular, git-managed automation ecosystem for two unRAID servers. One configur

 ---

+## What This Is
+
+Two friends. Hardware they already own. A mutual agreement to help each other.
+
+This ecosystem delivers practical high availability and offsite backup between two unRAID servers at zero additional hardware cost. It is not enterprise-grade clustering. It does not promise zero disruption. What it delivers is **minimal disruption** — and for a homelab, that's genuinely good enough.
+
+**The alternative:**
+```
+Proxmox cluster       — minimum 3 nodes
+Ceph storage          — dedicated hardware
+Enterprise networking — specialised equipment
+Offsite infrastructure — at least 3 servers minimum
+Cost                  — thousands in hardware, ongoing licensing
+```
+
+**This ecosystem:**
+```
+Two friends who already run unRAID
+Hardware they already own
+Mutual agreement — each covers the other
+Cost: $0 beyond what they were already running
+```
+
+**The honest user experience during failover:**
+```
+Library transcode users  — buffer absorbs the cutover, usually seamless
+Library direct play      — will notice, needs to resume
+Live TV direct play      — no buffer, notices every time, has to resume
+Watch history            — worst case 15-30min window behind
+
+Could force transcode on Live TV to get the buffer benefit —
+not worth the permanent CPU/RAM overhead for an event that rarely happens.
+Live TV disruption is the known, accepted tradeoff.
+```
+
+**The real world math:**
+
+How often does a homelab server actually go down? How many of those times are during peak viewing? How many viewers notice vs just resume? Is the occasional minor annoyance worth thousands in enterprise hardware?
+
+For most people — absolutely not. This ecosystem exists for those people.
+
+**Minimal disruption, not no disruption. Free, mutual, and genuinely useful.**
+
+---
+
 ## The Goal

 A self-hosted infrastructure that runs itself.
@@ -27,15 +72,18 @@ HOST1 — unRAID-Gmer4Lfe
  Storage:    Multiple ZFS pools + cache
  Domain:     Gmer4Lfe.com
  Runs:       Full media stack, auth, live TV, arrs, downloaders
-              arr master — source of truth for shared library
+              Source of truth: Movies, Tv_Shows, Music
+              Tdarr — runs anime transcoding (stronger hardware)

 HOST2 — unRAID-Jayred365 (Buddy server)
  Hardware:   Intel i5 10th gen — completely different hardware
  Storage:    Different disk count, different pool layout
+  RAM:        64GB
  Location:   Remote — 50 miles from HOST1
  Domain:     Gmer4Lfe.us (his own domain, his own services)
  Runs:       His own independent Emby, his own containers, his own users
-              Access to shared media library via mirrored shares
+              Source of truth: Anime_Shows, Anime_Movies
+              His arrs manage anime — syncs to HOST1 as mirror

 Network:    Tailscale — encrypted tunnel between both servers
 Repo:       Self-hosted Gitea on HOST1
@@ -47,45 +95,114 @@ Deployment: git pull on either server → both stay current
 ```
 Normal operation:
  HOST1  — Gmer4Lfe.com — full stack, live TV, arrs, everything
+             Source of truth for all config, DNS, auth, certs
  HOST2  — Gmer4Lfe.us  — his Emby, his containers, his domain, his users
-           Both independently serving media from the mirrored shared library
+             Mirror of HOST1's auth stack — changes on HOST1 propagate here
 ```

-**When HOST2 goes down:**
+**The shared production Emby:**
+
+All users — from both servers — connect to one Emby instance running on HOST1. HOST1 has the stronger hardware and better bandwidth. Both domains route through NPM to the same Emby. This is intentional — one library, one watch history, one set of users.
+
 ```
-HOST1 detects HOST2 unreachable + internet up
-→ Starts HOST2's DDNS (Gmer4Lfe.us) on HOST1 — his domain stays live
-→ Starts HOST2's specific containers — his services keep running on HOST1
-→ HOST1's own services keep running unaffected — failover is additive
-→ Waits for HOST2 to come back stable (FAILOVER_HANDBACK_STRIKES checks)
-→ Hands everything back — his containers, his DDNS — returns to NORMAL
+Gmer4Lfe.com → NPM → Emby   (my users)
+Gmer4Lfe.us  → NPM → Emby   (his users)
+Both point at the same container on HOST1
+```
+
+**Auth stack — runs on both servers simultaneously:**
+
+NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running to serve his users through his domain in normal operation. HOST1 is source of truth — all changes made there, mirrored to HOST2 every 15 minutes. One username and password works across all services on both domains. Group-based access controls what each user can see.
+
+```
+Change made on HOST1 LLDAP → mirrored to HOST2 → works on both domains
+Cert renewed on HOST1       → mirrored to HOST2 → valid on both domains
+NPM rule added on HOST1     → mirrored to HOST2 → routes on both servers
+```
+
+**Why the auth stack runs on both — a year of testing:**
+
+Early versions started the auth stack from cold on failover. Testing showed 30-60 seconds of broken authentication after DNS cut over — Emby clients hit HOST2 before LLDAP, Authelia, and NPM were ready. For library transcode users with buffer this was invisible. For Live TV and direct play users this meant the stream died and reconnect failed, forcing a manual exit and resume.
+
+Running the auth stack warm on HOST2 at all times eliminates this window entirely. When failover triggers, the auth stack is already running and ready. Only two containers actually start from stopped:
+
+```
+Emby          ← starts from stopped on HOST2
+DDNS updater  ← HOST1's domain updater starts on HOST2
+
+Everything else in Tier 1 is verified healthy — not cold started:
+NPM, LLDAP, Authelia, certs — already running, already warm
 ```

 **When HOST1 goes down:**
 ```
 HOST2 detects HOST1 unreachable + internet up
-→ Starts HOST1's DDNS (Gmer4Lfe.com) on HOST2 — my domain stays live
-→ Starts HOST1's Tier 1 containers — Emby, auth, live TV immediately
-→ Tiers 2/3/4 escalate if outage extends beyond configured delays
-→ HOST2's own services keep running unaffected — failover is additive
-→ Waits for HOST1 to come back stable and hands everything back
+→ Starts HOST1's DDNS updater — Gmer4Lfe.com now points at HOST2
+→ Starts Emby — only real container starting from stopped
+→ Auth stack already running — users authenticate immediately
+→ DNS TTL 1 minute — cutover fast
+→ Library transcode users — buffer absorbs, usually seamless
+→ Live TV / direct play users — will notice, need to resume
+→ Watch history — worst case 15-30min behind (mirror interval)
+→ Tiers 2/3/4 escalate if outage extends
+→ HOST2's own services keep running unaffected throughout
+```
+
+**When HOST2 goes down:**
+```
+HOST1 detects HOST2 unreachable + internet up
+→ Starts HOST2's DDNS updater — Gmer4Lfe.us now points at HOST1
+→ Starts his Emby — his users served from HOST1
+→ His auth stack starts from stopped (his server is down)
+   HOST1 has mirrored auth data — starts with current state
+→ Tiers escalate if outage extends
+→ HOST1's own services keep running unaffected throughout
+```
+
+**Handback — waits for stability:**
+```
+Remote returns → FAILOVER_HANDBACK_STRIKES consecutive stable checks
+→ Prevents handing back during a brief network blip
+→ Stops remote DDNS first — no split brain during transition
+→ Rsync writeback — userdata, watch states, auth changes during outage
+→ Starts containers on correct server
+→ Starts DDNS last — only after containers confirmed healthy
+→ Returns to NORMAL
 ```

 **The hardware doesn't need to match.** `/mnt/user/` abstracts everything. A share called `Movies` is `/mnt/user/Movies` on both servers regardless of what drives or pools back it. rsync syncs the content. Container mounts use the same path. The hardware underneath is irrelevant.

-**The shared media library:**
+**The shared media library and split source of truth:**

-All media syncs from HOST1 to HOST2 nightly. HOST1 is the arr master and source of truth — all downloads, all library management happens there. HOST2 accesses the mirrored library with his own Emby instance in normal operation, and serves it via HOST1's failover Emby when HOST1 is down.
+Each server owns specific shares as source of truth — managed by their own arrs. The other server mirrors those shares and treats them as read only in normal operation.

 ```
-HOST1 /mnt/user/Movies  →  rsync nightly  →  HOST2 /mnt/user/Movies
+HOST1 source of truth:            HOST2 source of truth:
+  Movies    ← Radarr               Anime_Shows   ← his Sonarr
+  Tv_Shows  ← Sonarr               Anime_Movies  ← his Radarr
+  Music     ← Lidarr

-HOST1's Emby      → /mnt/user/Movies  ← always
-HOST2's Emby      → /mnt/user/Movies  ← his own instance, same library
-Failover Emby     → /mnt/user/Movies  ← HOST1's Emby running on HOST2 during outage
+HOST1 mirrors ← HOST2 anime        HOST2 mirrors ← HOST1 movies/shows/music
 ```

-Don't run arrs on both simultaneously — two instances writing to the same share causes conflicts. HOST2's arrs only start at Tier 4 (18hr+ outage) when full workflow continuity is genuinely needed.
+Both servers run arrs simultaneously — no conflict because they manage completely different shares. Scheduling keeps them further separated:
+
+```
+HOST2 arrs:   midnight → noon      downloading and managing anime
+HOST1 Tdarr:  12:30   → 23:00     transcoding anime, syncs as source of truth
+```
+
+**The rule is not "don't run arrs on both servers" — it's:**
+**never run two instances of the same arr against the same share simultaneously.**
+
+Each Emby instance on both servers can access all media — movies, shows, anime — because everything is mirrored in both directions. HOST1 mirrors anime from HOST2. HOST2 mirrors movies and shows from HOST1.
+
+```
+HOST1 /mnt/user/Movies      →  rsync  →  HOST2 /mnt/user/Movies
+HOST2 /mnt/user/Anime_Shows →  rsync  →  HOST1 /mnt/user/Anime_Shows
+```
+
+At Tier 4 failover (18hr+ outage) each server's arr copies spin up to cover the other's shares — but only when the truth holder has genuinely been down long enough to need it.

 **Container naming convention:**

@@ -331,6 +448,22 @@ failover.sh           — infrastructure level, other server covers

 **--dry-run everywhere.** Every script supports `--dry-run`. Test before you schedule.

+**Hardware utilization without script changes.** Any unRAID system can participate regardless of hardware. The stronger server runs the heavier jobs — not because the scripts force it, but because app scheduling and rsync timing make it happen naturally. No script changes needed to reconfigure who does what:
+
+```
+HOST1 — Threadripper, 128GB
+  Tdarr transcoding: 12:30 → 23:00    ← heavy job on stronger hardware
+  
+HOST2 — i5, 64GB
+  His arrs: midnight → noon            ← lighter load, scheduled around Tdarr
+  Benefits from HOST1's transcoding
+  without the CPU/RAM cost
+```
+
+Change a schedule in an app. Adjust an rsync timing. The ecosystem adapts. The scripts never need to know which server is doing what — they just do their jobs on whatever server they're running on. This makes the ecosystem applicable to any two unRAID servers in any configuration — not just this specific hardware pairing.
+
+**Mutually beneficial, not one-way.** Both servers contribute what their hardware does best. Both benefit from what the other provides. The arrangement works because it's genuinely useful to both sides — not because one server is just a passive backup for the other.
+
 ---

 ## Scheduling Overview