From c54039392dc814461fe11f900d2c3bb9328cc862 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 1 Aug 2026 23:47:50 -0400 Subject: [PATCH] Rewrite README and Manual front pages in project voice The repo front page read like generic documentation rather than a record of why any of this exists. Adds the origin story, the current honest state, real failure history, and fixes drift in platform_*() naming, the fallback tier variable pattern, and the Plugin/unraid script locations. --- Manual.md | 67 +++++++++++++++++++++++++++-- README.md | 123 ++++++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 173 insertions(+), 17 deletions(-) diff --git a/Manual.md b/Manual.md index fa2d509..139525d 100644 --- a/Manual.md +++ b/Manual.md @@ -3,11 +3,23 @@ Getting a fresh two-server ecosystem running from scratch. For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers. +**Ten steps, and they're in this order for a reason.** Each one assumes the previous one +actually worked — not that you ran it, that it *worked*. Every step below ends with a way to +check, and skipping those checks is how you end up three steps later debugging the wrong +thing entirely. + +The steps that hurt most when rushed are **3** (naming) and **8** (testing failover). Step 3 +because renaming anything afterwards means chasing it through every conf, and step 8 because +untested failover isn't redundancy — it's a guess you haven't checked yet. + +Budget an evening. It is not hard, but it is not five minutes either. + --- ## ━━━ BEFORE YOU START ━━━ -Things that must be in place before you touch any scripts. +None of this is optional and none of it is Varaverk's job to install. Get these in place +first — every step after here assumes they're already true. --- @@ -157,6 +169,21 @@ Replace these with your actual unRAID server hostnames. These must match exactly `detect_hosts()` compares the running server's hostname against these two values to know which server it is on. Everything else in the ecosystem flows from this. +**Get this wrong and nothing works, but nothing errors either.** If neither value matches, +`MY_ID` is never set, and every script that depends on it either exits early or resolves +`${MY_ID}_SOMETHING` to an empty variable and quietly takes the wrong branch. Copy the value +straight out of `hostname` on each box rather than typing what you think it is: + +```bash +hostname # run this on each server, paste the exact output +``` + +> **The 15-character trap.** unRAID truncates the Server Name to 15 characters for NetBIOS. +> If your name is longer, what you set in the WebGUI and what `hostname` returns are two +> different strings. `detect_hosts()` has a fallback that matches a truncated 15-char +> hostname against a longer configured value — but only when the live hostname is +> *exactly* 15 characters. Don't rely on it. Use the real `hostname` output. + --- ### ── 3b. host1.conf — HOST1 identity ──────────────────────────────────────── @@ -206,15 +233,27 @@ HOST1_CRITICAL_SYNC_SHARES=( "/mnt/user/appdata-Fallback/Critical-Data|critical-data" ) -# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate) -HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=() # HOST2's own covers for HOST1, and vice versa -HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=( +# Containers to start when HOST1 goes down (Tier 1 = immediate) +# Lives in host1.conf. Named for the host being COVERED, not the host doing the covering. +FALLBACK_HOST1_TIER1=( "Emby" "VaultWarden" "NginxProxyManager" ) ``` +**Read that naming carefully — it catches people.** The variable is +`FALLBACK_${REMOTE_ID}_TIER${N}`, and `REMOTE_ID` is *the host that went down*. So +`FALLBACK_HOST1_TIER1` is the list HOST2 reads when HOST1 is unreachable. It defines +"what HOST1 needs covering", not "what HOST2 runs". + +It lives in `host1.conf` for the same reason — HOST1 owns the description of its own +service stack. HOST2 receives it through the conf cache rather than keeping its own +opinion about what HOST1 runs. One list, one owner, no drift. + +Tiers 2–4 activate progressively after `HOST1_TIER2_DELAY` etc., so a brief outage never +drags the entire remote stack across. + For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`, and `Manual-Watchdogs.md`. @@ -455,6 +494,18 @@ bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to detect it and start Tier 1 containers, then unblocks and waits for handback. +**Do not skip this step.** Everything up to here you can verify by reading output. Failover +is the one part you cannot confirm by looking at it — a typo in a tier list, a container name +that doesn't exist on the other side, a DDNS container that was renamed six months ago: all +of it sits there looking completely fine until the moment it's needed. + +The test costs you twenty minutes and no downtime. The alternative is finding out at 2am, +during the outage, when you have exactly one chance to get it right. + +**Watch the handback as closely as the failover.** Coming back is the harder half — it has to +stop the remote's DDNS, sync the data back, start the local containers, and only then bring +local DDNS up. Failover starting correctly tells you nothing about whether handback does. + See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and how to interpret the output. @@ -651,6 +702,14 @@ Each subsystem has a README with the design decisions and a Manual with the conf reference and troubleshooting. The `--status` flag on any script shows the current configuration and state. +**Turn things on one at a time and give each one a few days.** Everything below is off by +default on a fresh install, and that's deliberate — a stack where six new subsystems went +live the same night is a stack where you have no idea which one to blame. Enable, watch it +through a full daily cycle, then enable the next. + +Anything that deletes files — the arr cleanup scripts especially — gets a `--dry-run --log` +first. Read the list. Every time, not just the first time. + ``` Watchdogs/ → README-Watchdogs.md configure memory limits, container lists Media/ → README-Media.md enable arr cleanup, discovery scripts diff --git a/README.md b/README.md index f923241..99f9a3b 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,51 @@ correct adapter. The scripts themselves never branch on the OS. --- +## ━━━ HOW THIS GOT HERE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + +**One server. One script. Three lines.** + +That's genuinely where this started. Something needed doing on a schedule, so I wrote three +lines and moved on. + +Then it was two servers, 50 miles apart, and about **60 scripts** with nothing holding them +together. Every one standalone. Every one with its own hardcoded paths, its own container +names, its own idea of how to log something or send a notification. + +**And it worked.** That's the part worth saying. The stack stayed up, the media kept playing, +the backups kept running. It was not broken. + +It was just *miserable to change.* + +Every adjustment meant finding every script that touched the thing, on both machines, and +editing each one the same way — then hoping I'd found them all. Miss one and it didn't error. +It just quietly kept doing the old thing until I noticed weeks later. Two servers meant two +copies of every mistake, and the two copies drifted apart a little more every time I touched +them. + +So it got rebuilt, in roughly this order: + +``` +git stop editing live scripts on two machines and calling it version control +common.sh one place for log, notify, locking, host detection, retries +master.conf thresholds and toggles — shared, one copy +host*.conf credentials, paths, container names — per server +load_config.sh sources all of it, in the right order, at the top of every script +``` + +Then the long part: refactoring 60 standalone scripts into that shape, one at a time, until +every value lived in a conf file and every shared behaviour lived in `common.sh`. Change the +retry count once, every script gets it. Rename a container in one place, everything follows. + +That is the whole idea, and it's why the structure looks the way it does. The three-file +config model and the shared library aren't design patterns picked out of a book — they're +the specific answer to "I have 60 scripts on two machines and I'm scared to touch any of +them." + +That became Varaverk. + +--- + ## ━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ This is a live system, not a finished product. HOST1 has been running the whole stack @@ -57,7 +102,7 @@ HOST1 — unRAID-Gmer4Lfe HOST2 — unRAID-Jayred365 Hardware: Intel i5 10th gen, 64GB RAM - Location: Remote — different building, different power utility + Location: Remote — 50 miles away, different power utility and ISP Domain: Gmer4Lfe.us Arrs: Anime_Shows, Anime_Movies — union with HOST1 via arr_sync His own Emby — for his household @@ -173,6 +218,10 @@ as sync targets and mount points — a mismatch is a broken path. ## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +Two households, one library, and a hard line between "our media" and "your stuff." Everything +in this section comes down to that split — media is pooled and shared freely, personal data +never is, and no script gets to blur the two. + ### ── Common Libraries — Shared Pool ───────────────────────────────────────── Common media like Movies, Music, and TV Shows are a **shared pool**. Every container @@ -257,8 +306,13 @@ across separate shares per-user or per-application. ## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -Not everything needs to sync. Not everything should sync. Understanding what goes -where determines whether you need encryption and whether sync failure matters. +Not everything needs to sync. Not everything *should*. Two households means some of this +data is mine, some of it is his, and some of it belongs to both of us — and treating all +three the same way is how you end up either leaking someone's photos or losing the thing you +actually needed a copy of. + +Where a piece of data lands in this table decides two things: whether it needs encryption, +and whether a failed sync is something to fix tonight or something to shrug at. --- @@ -504,6 +558,15 @@ Monitoring: ## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +Folders are grouped by **what a script does to the system**, not by which service it happens +to talk to. There's no `Sonarr/` folder — cleaning up after Sonarr is media work, watching +whether its container is alive is watchdog work, and those two things belong next to their +own kind. + +Every folder has its own README explaining the group, and every script has a header +explaining itself. If you're looking for one specific thing, the README reference near the +bottom of this page is the faster route. + ``` varaverk/ │ @@ -563,6 +626,16 @@ varaverk/ ## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━ +This is the part that fixed the 60-script problem. Three files, one rule: + +**Thresholds and toggles go in `master.conf`. Credentials, paths and container names go in +`host*.conf`.** If you find yourself about to hardcode a value in a script, it belongs in one +of these instead — that's the whole discipline, and everything else follows from it. + +Sparse checkout means each server only ever receives its own `host*.conf`. HOST1 never has +HOST2's credentials sitting on disk, and vice versa. The partner's conf arrives at runtime +through the RAM cache, which is why `conf_sync.sh` exists. + ```bash # ───────────────────────────────────────────────────────────────────────────── # All three conf files live in Configurations/. load_config.sh sources them. @@ -642,7 +715,8 @@ Every 4 hours: 2:30am Sunday: weekly_sync_maintenance.sh stop → update images → clean sync → start → weekly restarts - + playback_aware_lidarr/radarr/sonarr_discovery.sh (behavior-driven arr discovery) + + playback_aware_{lidarr,radarr,sonarr}_discovery.sh + (behaviour-driven discovery from real playback history) 8am daily: weekly_health_digest.sh DIGEST_PROFILE controls when it actually notifies @@ -714,8 +788,9 @@ restart lands on the new image. Change the order and things break quietly rather ## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -Problems are handled at the most appropriate level. Scripts escalate only when the -level below them has failed: +Nothing here jumps straight to the big hammer. Each layer only gets involved because the one +below it already tried and failed — and the layer holding the biggest hammer, the one that +can reboot the box, is the most reluctant of all: ``` Problem: Emby container using 22GB RAM (limit: 18GB) @@ -833,7 +908,12 @@ CPU-based) instead of PaddleOCR. Same plugin, no VRAM. ## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -If you're setting this up from scratch on two servers: +If you're setting this up from scratch on two servers, the order below is not a suggestion. +Steps 2 and 3 are cheap now and expensive later — renaming a container or a share after the +confs, the fallback tiers and the sync profiles all reference it means touching every one of +them, and missing one doesn't error. + +Do the naming first. Everything after it assumes the names are already right. ``` 1. Follow Manual.md — end-to-end setup guide @@ -854,6 +934,11 @@ If you're setting this up from scratch on two servers: bash fallback_test.sh when ready — uses iptables simulation, no real downtime ``` +**Do step 5.** Untested failover isn't redundancy, it's a belief. `fallback_test.sh` simulates +the outage with iptables rather than actually pulling the other server down, so there's no +excuse not to run it — and finding out your tier lists have a typo is a lot nicer on a Tuesday +afternoon than at 2am when the power's actually out. + --- ## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ @@ -976,15 +1061,18 @@ Every one of these cost me something to learn. Reading them here is cheaper. # --dry-run --log first on a new system. README-Media.md has the full procedure. # 9. Scripts never call OS commands directly. -# All platform-specific operations go through plugin_*() functions defined in +# All platform-specific operations go through platform_*() functions defined in # Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all # isolated in the adapter. This is what makes the codebase portable. # Adding a new OS = writing one adapter file, not patching dozens of scripts. -# 10. System_Essentials/ is platform-agnostic at the script level. -# Scripts like server_reboot.sh, mover_stop.sh, and webgui_watchdog.sh call -# adapter functions (platform_is_mover_running, platform_restart_service, etc.) -# The adapter handles what those mean on the actual OS. The scripts are clean. +# 10. Top-level System_Essentials/ is platform-agnostic. Plugin/unraid/ is not. +# server_reboot.sh lives at the top level and calls adapter functions only — +# platform_is_mover_running, platform_restart_service. It never names an OS. +# Scripts that manipulate an Unraid subsystem directly — mover_stop.sh, +# webgui_watchdog.sh, php_fpm_max_children.sh — live under Plugin/unraid/ +# instead, because there is no generic version of them to write. +# Which folder a script is in tells you whether it survives a platform change. # ───────────────────────────────────────────────────────────────────────────── ``` @@ -996,4 +1084,13 @@ Every one of these cost me something to learn. Reading them here is cheaper. Name: Varaverk Repository: git@192.168.50.2:FailedProxy/Varaverk.git Install path: /boot/config/plugins/varaverk -``` \ No newline at end of file +``` + +--- + +This is a home media stack for two households. It is not an enterprise product, it doesn't +want to be one, and it makes trade-offs no enterprise would accept — because the thing it's +actually optimising for is that nobody in either house has to know any of this exists. + +The measure of success here isn't uptime percentages. It's that the power goes out 50 miles +away, and the show keeps playing. \ No newline at end of file