FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added write back timing checks correlastes with tiers timing

Browse Source
This commit is contained in:
FailedProxy
2026-04-19 15:45:15 -04:00
parent ed9e927c41
commit 4399c81b6e
5 changed files with 407 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+155 -22
View File
@@ -4,6 +4,51 @@ A modular, git-managed automation ecosystem for two unRAID servers. One configur
---
## What This Is
Two friends. Hardware they already own. A mutual agreement to help each other.
This ecosystem delivers practical high availability and offsite backup between two unRAID servers at zero additional hardware cost. It is not enterprise-grade clustering. It does not promise zero disruption. What it delivers is **minimal disruption** — and for a homelab, that's genuinely good enough.
**The alternative:**
```
Proxmox cluster — minimum 3 nodes
Ceph storage — dedicated hardware
Enterprise networking — specialised equipment
Offsite infrastructure — at least 3 servers minimum
Cost — thousands in hardware, ongoing licensing
```
**This ecosystem:**
```
Two friends who already run unRAID
Hardware they already own
Mutual agreement — each covers the other
Cost: $0 beyond what they were already running
```
**The honest user experience during failover:**
```
Library transcode users — buffer absorbs the cutover, usually seamless
Library direct play — will notice, needs to resume
Live TV direct play — no buffer, notices every time, has to resume
Watch history — worst case 15-30min window behind
Could force transcode on Live TV to get the buffer benefit —
not worth the permanent CPU/RAM overhead for an event that rarely happens.
Live TV disruption is the known, accepted tradeoff.
```
**The real world math:**
How often does a homelab server actually go down? How many of those times are during peak viewing? How many viewers notice vs just resume? Is the occasional minor annoyance worth thousands in enterprise hardware?
For most people — absolutely not. This ecosystem exists for those people.
**Minimal disruption, not no disruption. Free, mutual, and genuinely useful.**
---
## The Goal
A self-hosted infrastructure that runs itself.
@@ -27,15 +72,18 @@ HOST1 — unRAID-Gmer4Lfe
Storage: Multiple ZFS pools + cache
Domain: Gmer4Lfe.com
Runs: Full media stack, auth, live TV, arrs, downloaders
arr master — source of truth for shared library
Source of truth: Movies, Tv_Shows, Music
Tdarr — runs anime transcoding (stronger hardware)
HOST2 — unRAID-Jayred365 (Buddy server)
Hardware: Intel i5 10th gen — completely different hardware
Storage: Different disk count, different pool layout
RAM: 64GB
Location: Remote — 50 miles from HOST1
Domain: Gmer4Lfe.us (his own domain, his own services)
Runs: His own independent Emby, his own containers, his own users
Access to shared media library via mirrored shares
Source of truth: Anime_Shows, Anime_Movies
His arrs manage anime — syncs to HOST1 as mirror
Network: Tailscale — encrypted tunnel between both servers
Repo: Self-hosted Gitea on HOST1
@@ -47,45 +95,114 @@ Deployment: git pull on either server → both stay current
```
Normal operation:
HOST1 — Gmer4Lfe.com — full stack, live TV, arrs, everything
Source of truth for all config, DNS, auth, certs
HOST2 — Gmer4Lfe.us — his Emby, his containers, his domain, his users
Both independently serving media from the mirrored shared library
Mirror of HOST1's auth stack — changes on HOST1 propagate here
```
**When HOST2 goes down:**
**The shared production Emby:**
All users — from both servers — connect to one Emby instance running on HOST1. HOST1 has the stronger hardware and better bandwidth. Both domains route through NPM to the same Emby. This is intentional — one library, one watch history, one set of users.
```
HOST1 detects HOST2 unreachable + internet up
→ Starts HOST2's DDNS (Gmer4Lfe.us) on HOST1 — his domain stays live
→ Starts HOST2's specific containers — his services keep running on HOST1
→ HOST1's own services keep running unaffected — failover is additive
→ Waits for HOST2 to come back stable (FAILOVER_HANDBACK_STRIKES checks)
→ Hands everything back — his containers, his DDNS — returns to NORMAL
Gmer4Lfe.com → NPM → Emby (my users)
Gmer4Lfe.us → NPM → Emby (his users)
Both point at the same container on HOST1
```
**Auth stack — runs on both servers simultaneously:**
NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running to serve his users through his domain in normal operation. HOST1 is source of truth — all changes made there, mirrored to HOST2 every 15 minutes. One username and password works across all services on both domains. Group-based access controls what each user can see.
```
Change made on HOST1 LLDAP → mirrored to HOST2 → works on both domains
Cert renewed on HOST1 → mirrored to HOST2 → valid on both domains
NPM rule added on HOST1 → mirrored to HOST2 → routes on both servers
```
**Why the auth stack runs on both — a year of testing:**
Early versions started the auth stack from cold on failover. Testing showed 30-60 seconds of broken authentication after DNS cut over — Emby clients hit HOST2 before LLDAP, Authelia, and NPM were ready. For library transcode users with buffer this was invisible. For Live TV and direct play users this meant the stream died and reconnect failed, forcing a manual exit and resume.
Running the auth stack warm on HOST2 at all times eliminates this window entirely. When failover triggers, the auth stack is already running and ready. Only two containers actually start from stopped:
```
Emby ← starts from stopped on HOST2
DDNS updater ← HOST1's domain updater starts on HOST2
Everything else in Tier 1 is verified healthy — not cold started:
NPM, LLDAP, Authelia, certs — already running, already warm
```
**When HOST1 goes down:**
```
HOST2 detects HOST1 unreachable + internet up
→ Starts HOST1's DDNS (Gmer4Lfe.com) on HOST2 — my domain stays live
→ Starts HOST1's Tier 1 containers — Emby, auth, live TV immediately
→ Tiers 2/3/4 escalate if outage extends beyond configured delays
→ HOST2's own services keep running unaffected — failover is additive
→ Waits for HOST1 to come back stable and hands everything back
→ Starts HOST1's DDNS updater — Gmer4Lfe.com now points at HOST2
→ Starts Emby — only real container starting from stopped
→ Auth stack already running — users authenticate immediately
→ DNS TTL 1 minute — cutover fast
→ Library transcode users — buffer absorbs, usually seamless
→ Live TV / direct play users — will notice, need to resume
→ Watch history — worst case 15-30min behind (mirror interval)
→ Tiers 2/3/4 escalate if outage extends
→ HOST2's own services keep running unaffected throughout
```
**When HOST2 goes down:**
```
HOST1 detects HOST2 unreachable + internet up
→ Starts HOST2's DDNS updater — Gmer4Lfe.us now points at HOST1
→ Starts his Emby — his users served from HOST1
→ His auth stack starts from stopped (his server is down)
HOST1 has mirrored auth data — starts with current state
→ Tiers escalate if outage extends
→ HOST1's own services keep running unaffected throughout
```
**Handback — waits for stability:**
```
Remote returns → FAILOVER_HANDBACK_STRIKES consecutive stable checks
→ Prevents handing back during a brief network blip
→ Stops remote DDNS first — no split brain during transition
→ Rsync writeback — userdata, watch states, auth changes during outage
→ Starts containers on correct server
→ Starts DDNS last — only after containers confirmed healthy
→ Returns to NORMAL
```
**The hardware doesn't need to match.** `/mnt/user/` abstracts everything. A share called `Movies` is `/mnt/user/Movies` on both servers regardless of what drives or pools back it. rsync syncs the content. Container mounts use the same path. The hardware underneath is irrelevant.
**The shared media library:**
**The shared media library and split source of truth:**
All media syncs from HOST1 to HOST2 nightly. HOST1 is the arr master and source of truth — all downloads, all library management happens there. HOST2 accesses the mirrored library with his own Emby instance in normal operation, and serves it via HOST1's failover Emby when HOST1 is down.
Each server owns specific shares as source of truth — managed by their own arrs. The other server mirrors those shares and treats them as read only in normal operation.
```
HOST1 /mnt/user/Movies → rsync nightly → HOST2 /mnt/user/Movies
HOST1 source of truth: HOST2 source of truth:
Movies ← Radarr Anime_Shows ← his Sonarr
Tv_Shows ← Sonarr Anime_Movies ← his Radarr
Music ← Lidarr
HOST1's Emby → /mnt/user/Movies ← always
HOST2's Emby → /mnt/user/Movies ← his own instance, same library
Failover Emby → /mnt/user/Movies ← HOST1's Emby running on HOST2 during outage
HOST1 mirrors ← HOST2 anime HOST2 mirrors ← HOST1 movies/shows/music
```
Don't run arrs on both simultaneously — two instances writing to the same share causes conflicts. HOST2's arrs only start at Tier 4 (18hr+ outage) when full workflow continuity is genuinely needed.
Both servers run arrs simultaneously — no conflict because they manage completely different shares. Scheduling keeps them further separated:
```
HOST2 arrs: midnight → noon downloading and managing anime
HOST1 Tdarr: 12:30 → 23:00 transcoding anime, syncs as source of truth
```
**The rule is not "don't run arrs on both servers" — it's:**
**never run two instances of the same arr against the same share simultaneously.**
Each Emby instance on both servers can access all media — movies, shows, anime — because everything is mirrored in both directions. HOST1 mirrors anime from HOST2. HOST2 mirrors movies and shows from HOST1.
```
HOST1 /mnt/user/Movies → rsync → HOST2 /mnt/user/Movies
HOST2 /mnt/user/Anime_Shows → rsync → HOST1 /mnt/user/Anime_Shows
```
At Tier 4 failover (18hr+ outage) each server's arr copies spin up to cover the other's shares — but only when the truth holder has genuinely been down long enough to need it.
**Container naming convention:**
@@ -331,6 +448,22 @@ failover.sh — infrastructure level, other server covers
**--dry-run everywhere.** Every script supports `--dry-run`. Test before you schedule.
**Hardware utilization without script changes.** Any unRAID system can participate regardless of hardware. The stronger server runs the heavier jobs — not because the scripts force it, but because app scheduling and rsync timing make it happen naturally. No script changes needed to reconfigure who does what:
```
HOST1 — Threadripper, 128GB
Tdarr transcoding: 12:30 → 23:00 ← heavy job on stronger hardware
HOST2 — i5, 64GB
His arrs: midnight → noon ← lighter load, scheduled around Tdarr
Benefits from HOST1's transcoding
without the CPU/RAM cost
```
Change a schedule in an app. Adjust an rsync timing. The ecosystem adapts. The scripts never need to know which server is doing what — they just do their jobs on whatever server they're running on. This makes the ecosystem applicable to any two unRAID servers in any configuration — not just this specific hardware pairing.
**Mutually beneficial, not one-way.** Both servers contribute what their hardware does best. Both benefit from what the other provides. The arrangement works because it's genuinely useful to both sides — not because one server is just a passive backup for the other.
---
## Scheduling Overview