FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fixed needed readme's over the writeback change

Browse Source
This commit is contained in:
FailedProxy
2026-04-19 16:20:22 -04:00
parent 4399c81b6e
commit af14a5ca2c
3 changed files with 112 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Failover/README-Failover.md
+56 -19
View File
@@ -62,20 +62,26 @@ NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs
Early testing ran the auth stack cold on failover. Results: 30-60 seconds of broken authentication after DNS cut over. Emby clients hit HOST2 before auth was ready — reconnects failed, streams died. Running warm eliminates this window entirely. Early testing ran the auth stack cold on failover. Results: 30-60 seconds of broken authentication after DNS cut over. Emby clients hit HOST2 before auth was ready — reconnects failed, streams died. Running warm eliminates this window entirely.
**What actually starts from stopped on failover (HOST1 goes down):** **What actually starts from stopped — both directions:**
``` ```
Emby ← starts from stopped on HOST2 Tier 1 — immediate:
DDNS updater ← HOST1's domain updater starts on HOST2 Emby ← starts from stopped on covering server
DDNS updater ← remote domain updater starts on covering server
Already running — verified healthy, NOT cold started: Already running — verified healthy, NOT cold started:
NPM — serving his domain continuously NPM — serving both domains continuously
LLDAP — authenticating his users continuously LLDAP — authenticating all users continuously
Authelia — protecting his services continuously Authelia — protecting all services continuously
Certs — mirrored, valid, already loaded Certs — mirrored, valid, already loaded on both servers
Tier 4 — 18hr+ outage:
Arrs_Stack ← starts from stopped on covering server
mirrored Arrs_Stack appdata — starts in known good state
not running in normal operation on covering server
``` ```
DNS cuts over in 1 minute. Auth is already ready. Library transcode users reconnect seamlessly through buffer. Live TV and direct play users notice and need to resume — the known, accepted tradeoff. DNS cuts over in 1 minute. Auth is already ready. Library transcode users reconnect seamlessly through buffer. Live TV and direct play users notice and need to resume — the known, accepted tradeoff. Watch history worst case 15-30min behind — in reality a show or two, movies may be unaffected given their length.
**Independence — always one rsync stop away:** **Independence — always one rsync stop away:**
@@ -470,17 +476,48 @@ If the state file gets stuck in a non-NORMAL state after testing or a failed han
## What Gets Written Back on Handback ## What Gets Written Back on Handback
| Data | Written back | Reason | | Data | Written back | Condition |
|------|-------------|--------| |------|-------------|-----------|
| Emby userdata / playstates | ✅ Yes | Small, important — watch history | | Emby userdata / playstates | ✅ If outage > TIER1_WRITEBACK_DELAY (60min) | Small, important — watch history |
| Auth stack data | ✅ Yes | Authelia sessions, LLDAP data | | Auth stack data | ✅ If outage > TIER1_WRITEBACK_DELAY (60min) | Authelia sessions, LLDAP data |
| NextCloud data | ✅ Yes | File changes during outage | | NextCloud data | ✅ If outage > TIER2_DELAY (120min) | File changes during outage |
| Emby metadata | ✅ Yes | Any metadata scraped during outage | | Secondary appdata | ✅ If outage > TIER3_DELAY (360min) | Any accumulated changes |
| Media files | ❌ No | Already on HOST1, never moved | | Opposing host's media shares | ✅ If outage > TIER4_DELAY (1080min) | HOST1_DAILY_SYNC_SHARES or HOST2_DAILY_SYNC_SHARES — same list daily_sync.sh uses, opposite direction |
| Downloads | ❌ No | Start fresh — cleaner than partial state | | Edge case paths | ✅ If outage > TIER4_DELAY | Anything in FAILOVER_HOST*_WRITEBACK_TIER4 |
| Arr databases | ❌ No (if under 18hr) | Arrs never started, nothing to sync | | Media files (own truth shares) | ❌ Never | Already on primary, never moved |
| Downloads | ❌ Never | Start fresh — cleaner than partial state |
The minimal writeback is intentional and by design. The goal is to get HOST1 back to the state it was in before the outage plus the delta of what changed during it — not to sync everything. **Writeback skip window — short outages skip writeback entirely:**
Emby syncs every 30 minutes from a live running container (dirty sync). A clean full sync runs nightly at 2:30am with Emby stopped. After a short outage, the primary's clean nightly state is more reliable than what accumulated on the covering server — skip writeback entirely.
```
Under 60min → skip Tier 1-3 writeback — primary has cleaner data
Under 120min → skip Tier 2-3 writeback
Under 360min → skip Tier 3 writeback
Under 1080min → skip Tier 4 writeback — arrs never started, nothing to sync back
Over 1080min → full tiered writeback — meaningful delta accumulated
```
Real world outage profile:
```
Power blip 2-10min → no writeback, clean restart (most common)
ISP issue 10-60min → no writeback, clean restart
Real outage 3hr → Tier 1+2 writeback, skip 3+4
Extended 18hr+ → full tiered writeback including media shares
```
**Tier 4 media writeback — opposing orch list:**
At Tier 4, the covering server has been running the other server's arrs against the mirrored media shares. New content downloaded during the outage needs to sync back before the primary's own arrs restart. Rather than maintaining a duplicate share list, Tier 4 writeback automatically uses the opposing host's `HOST*_DAILY_SYNC_SHARES` — the same list `daily_sync.sh` uses, but in the opposite direction.
```
Normal (daily_sync.sh): HOST1 → pushes Movies, Tv_Shows → HOST2
Tier 4 writeback: HOST2 → pushes Movies, Tv_Shows → HOST1 (opposite direction)
Same list, correct direction for the situation, no duplication.
```
Additional edge case paths outside the daily sync list can be added to `FAILOVER_HOST*_WRITEBACK_TIER4` in Master.conf.
--- ---
Orchestrators/README-Orchestrators.md
+35 -10
View File
@@ -59,38 +59,63 @@ Every cycle `transcode_management.sh` records stats to `/boot/config/transcode_d
### `daily_sync.sh` ### `daily_sync.sh`
Syncs all bulk media shares to the remote server sequentially. Scheduled once daily. Syncs each server's source-of-truth media shares to the remote server sequentially. Each server only pushes the shares it owns — direction and share list are automatic based on which server is running the script.
```bash ```bash
# Scheduled as: 0 1 * * * (1am daily) # Scheduled as: 0 1 * * * (1am daily — on both servers)
/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
``` ```
**Bidirectional — same script, correct direction automatically:**
```
HOST1 runs daily_sync.sh → pushes HOST1_DAILY_SYNC_SHARES → TO HOST2
Movies, Tv_Shows, Music, Books etc. — HOST1 is source of truth
HOST2 runs daily_sync.sh → pushes HOST2_DAILY_SYNC_SHARES → TO HOST1
Anime_Shows, Anime_Movies — HOST2 is source of truth
```
`detect_hosts()` determines which server is local at runtime and selects the correct share list. No script changes needed to reconfigure who syncs what — only `Master.conf` changes required.
**What it does:** **What it does:**
1. Detects local and remote server via `detect_hosts()` 1. Detects local server via `detect_hosts()` — determines HOST1 or HOST2
2. Resolves remote Tailscale IP 2. Resolves remote Tailscale IP
3. Runs a single pre-flight check — connectivity + remote rootfs 3. Runs a single pre-flight check — connectivity + remote rootfs
4. Iterates through every share in `DAILY_SYNC_SHARES` 4. Builds share list from `HOST1_DAILY_SYNC_SHARES` or `HOST2_DAILY_SYNC_SHARES`
5. Calls `Rsync/rsync.sh` for each share 5. Appends personal shares (`HOST1_PERSONAL_SHARES` or `HOST2_PERSONAL_SHARES`)
6. Tracks pass/fail and duration per share 6. Calls `Rsync/rsync.sh` for each share
7. Reports a combined summary 7. Tracks pass/fail and duration per share
8. Reports a combined summary
**Why one pre-flight check upfront:** **Why one pre-flight check upfront:**
Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast before attempting 12 shares. Individual share existence and disk checks still run per-share inside `rsync.sh`. Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast. Individual share existence and disk checks still run per-share inside `rsync.sh`.
**Configuration:** **Configuration:**
```bash ```bash
# Master.conf — add or remove paths to control what syncs nightly # Master.conf — per-host share lists
DAILY_SYNC_SHARES=( # HOST1 truth shares — pushed from HOST1 to HOST2 nightly
HOST1_DAILY_SYNC_SHARES=(
/mnt/user/Movies /mnt/user/Movies
/mnt/user/Tv_Shows /mnt/user/Tv_Shows
/mnt/user/Music
# ...
)
# HOST2 truth shares — pushed from HOST2 to HOST1 nightly
HOST2_DAILY_SYNC_SHARES=(
/mnt/user/Anime_Shows /mnt/user/Anime_Shows
/mnt/user/Anime_Movies
# ... # ...
) )
``` ```
These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call `rsync.sh` directly on a separate schedule instead. These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call `rsync.sh` directly on a separate schedule instead.
**Relationship to failover writeback:**
The same share lists are used by `failover.sh` for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 18hr+ and HOST2's arrs downloaded new content, Tier 4 writeback pushes `HOST1_DAILY_SYNC_SHARES` FROM HOST2 back TO HOST1. No duplicate configuration needed.
**Example output:** **Example output:**
``` ```
━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━ ━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━
README.md
+21 -13
View File
@@ -30,9 +30,10 @@ Cost: $0 beyond what they were already running
**The honest user experience during failover:** **The honest user experience during failover:**
``` ```
Library transcode users — buffer absorbs the cutover, usually seamless Library transcode users — buffer absorbs the cutover, usually seamless
Library direct play — will notice, needs to resume Library direct play — may notice, needs to resume
Live TV direct play — no buffer, notices every time, has to resume Live TV direct play — no buffer, notices every time, has to resume
Watch history — worst case 15-30min window behind Watch history — worst case 15-30min window behind
reality: a show or two, movies likely unaffected
Could force transcode on Live TV to get the buffer benefit — Could force transcode on Live TV to get the buffer benefit —
not worth the permanent CPU/RAM overhead for an event that rarely happens. not worth the permanent CPU/RAM overhead for an event that rarely happens.
@@ -124,26 +125,32 @@ NPM rule added on HOST1 → mirrored to HOST2 → routes on both servers
Early versions started the auth stack from cold on failover. Testing showed 30-60 seconds of broken authentication after DNS cut over — Emby clients hit HOST2 before LLDAP, Authelia, and NPM were ready. For library transcode users with buffer this was invisible. For Live TV and direct play users this meant the stream died and reconnect failed, forcing a manual exit and resume. Early versions started the auth stack from cold on failover. Testing showed 30-60 seconds of broken authentication after DNS cut over — Emby clients hit HOST2 before LLDAP, Authelia, and NPM were ready. For library transcode users with buffer this was invisible. For Live TV and direct play users this meant the stream died and reconnect failed, forcing a manual exit and resume.
Running the auth stack warm on HOST2 at all times eliminates this window entirely. When failover triggers, the auth stack is already running and ready. Only two containers actually start from stopped: Running the auth stack warm on both servers at all times eliminates this window entirely. When failover triggers, the auth stack is already running and ready. What actually starts from stopped:
``` ```
Emby ← starts from stopped on HOST2 Tier 1 — immediate:
DDNS updater ← HOST1's domain updater starts on HOST2 Emby ← starts from stopped on covering server
DDNS updater ← remote domain updater starts on covering server
Everything else in Tier 1 is verified healthy — not cold started: Everything else verified healthy — NOT cold started:
NPM, LLDAP, Authelia, certs — already running, already warm NPM, LLDAP, Authelia, certs — already running, already warm on both servers
Tier 4 — 18hr+ outage:
Arrs_Stack ← starts from stopped on covering server
has mirrored Arrs_Stack appdata — starts in known good state
``` ```
**When HOST1 goes down:** **When HOST1 goes down:**
``` ```
HOST2 detects HOST1 unreachable + internet up HOST2 detects HOST1 unreachable + internet up
→ Starts HOST1's DDNS updater — Gmer4Lfe.com now points at HOST2 → Starts HOST1's DDNS updater — Gmer4Lfe.com now points at HOST2
→ Starts Emby — only real container starting from stopped → Starts Emby — only real Tier 1 container starting from stopped
→ Auth stack already running — users authenticate immediately → Auth stack already running on HOST2 — users authenticate immediately
→ DNS TTL 1 minute — cutover fast → DNS TTL 1 minute — cutover fast
→ Library transcode users — buffer absorbs, usually seamless → Library transcode users — buffer absorbs, usually seamless
→ Live TV / direct play users — will notice, need to resume → Live TV / direct play users — will notice, need to resume
→ Watch history — worst case 15-30min behind (mirror interval) → Watch history — worst case 15-30min behind, reality a show or two
movies may be unaffected given their length
→ Tiers 2/3/4 escalate if outage extends → Tiers 2/3/4 escalate if outage extends
→ HOST2's own services keep running unaffected throughout → HOST2's own services keep running unaffected throughout
``` ```
@@ -153,9 +160,10 @@ HOST2 detects HOST1 unreachable + internet up
HOST1 detects HOST2 unreachable + internet up HOST1 detects HOST2 unreachable + internet up
→ Starts HOST2's DDNS updater — Gmer4Lfe.us now points at HOST1 → Starts HOST2's DDNS updater — Gmer4Lfe.us now points at HOST1
→ Starts his Emby — his users served from HOST1 → Starts his Emby — his users served from HOST1
→ His auth stack starts from stopped (his server is down) → His auth stack already running on HOST1 — serving his domain continuously
HOST1 has mirrored auth data — starts with current state → Tiers 2/3 escalate if outage extends
→ Tiers escalate if outage extends → Tier 4 (18hr+): his Arrs_Stack starts from stopped on HOST1
mirrored appdata — starts in known good state
→ HOST1's own services keep running unaffected throughout → HOST1's own services keep running unaffected throughout
``` ```
@@ -240,7 +248,7 @@ HOST2_PERSONAL_SHARES=(
) )
``` ```
`daily_sync.sh` automatically picks up the personal shares for the local host and syncs them alongside the media shares. `daily_sync.sh` automatically picks up the personal shares for the local host and syncs them alongside that host's truth shares.
**Encrypting personal shares — unRAID 7 ZFS:** **Encrypting personal shares — unRAID 7 ZFS:**