added write back timing checks correlastes with tiers timing

This commit is contained in:
2026-04-19 15:45:15 -04:00
parent ed9e927c41
commit 4399c81b6e
5 changed files with 407 additions and 79 deletions
+48 -3
View File
@@ -18,6 +18,7 @@ HOST1 — unRAID-Gmer4Lfe (Primary)
HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
Hardware: Intel i5 10th gen — completely different hardware
RAM: 64GB
Location: Remote — 50 miles away
Owns: Gmer4Lfe.us DDNS
Runs: Its own service stack + mirrors HOST1 critical data
@@ -30,11 +31,55 @@ HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
- Container names for shared failover services
- Docker custom network names (so NPM can reach containers by name, not IP)
**Normal operation — HOST2 is essentially passive:**
**Split source of truth — each server owns different shares:**
HOST1 is the source of truth. It runs everything — all arrs, all downloads, all active services. HOST2 keeps its media mirror current via nightly rsync and waits. All media is mirrored so when Emby starts on HOST2 during failover it has the exact same library, same metadata, same watch history. No separate library, no separate database, no user-visible difference.
Both servers run arrs simultaneously — no conflict because they manage completely different shares:
Don't run arrs on both servers simultaneously — two instances writing to the same share causes conflicts and duplicate downloads. HOST2's arrs only start at Tier 4 (18hr+ outage) when genuine workflow continuity is needed.
```
HOST1 arrs — source of truth for: HOST2 arrs — source of truth for:
Movies (Radarr) Anime_Movies (his Radarr)
Tv_Shows (Sonarr) Anime_Shows (his Sonarr)
Music (Lidarr)
Each mirrors the other's shares in the opposite direction.
HOST1 mirrors anime FROM HOST2.
HOST2 mirrors movies/shows/music FROM HOST1.
```
Scheduling keeps them clean even within the same share window:
```
HOST2 arrs: midnight → noon managing and downloading anime
HOST1 Tdarr: 12:30 → 23:00 transcoding anime, syncs as source of truth
```
**The rule:** never run two instances of the same arr against the same share simultaneously. Different arrs managing different shares is perfectly fine.
At Tier 4 failover (18hr+ outage) each server's arr copies activate to cover the other's shares — only when the truth holder has genuinely been down long enough.
**Auth stack — runs warm on both servers simultaneously:**
NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running to serve his users through his domain every day. HOST1 is source of truth — all changes mirror to HOST2 every 15 minutes. Certs, proxy rules, user accounts, Authelia policies — all current on both servers at all times.
Early testing ran the auth stack cold on failover. Results: 30-60 seconds of broken authentication after DNS cut over. Emby clients hit HOST2 before auth was ready — reconnects failed, streams died. Running warm eliminates this window entirely.
**What actually starts from stopped on failover (HOST1 goes down):**
```
Emby ← starts from stopped on HOST2
DDNS updater ← HOST1's domain updater starts on HOST2
Already running — verified healthy, NOT cold started:
NPM — serving his domain continuously
LLDAP — authenticating his users continuously
Authelia — protecting his services continuously
Certs — mirrored, valid, already loaded
```
DNS cuts over in 1 minute. Auth is already ready. Library transcode users reconnect seamlessly through buffer. Live TV and direct play users notice and need to resume — the known, accepted tradeoff.
**Independence — always one rsync stop away:**
If HOST2 ever wants to fully separate: stop HOST1 pushing. Changes he makes stick permanently. His server becomes fully independent immediately. No script changes, no migration — just stop the rsync job. The ecosystem supports this by design.
Both servers run `failover.sh` as a background task continuously. Neither server knows what the other is doing — they only know what they can ping from their own network perspective.