FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite README and Manual front pages in project voice

Browse Source 
The repo front page read like generic documentation rather than a record of
why any of this exists. Adds the origin story, the current honest state, real
failure history, and fixes drift in platform_*() naming, the fallback tier
variable pattern, and the Plugin/unraid script locations.
This commit is contained in:
Gmer4Lfe
2026-08-01 23:47:50 -04:00
parent 671bf2352b
commit c54039392d
2 changed files with 173 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Manual.md
+63 -4
View File
@@ -3,11 +3,23 @@
Getting a fresh two-server ecosystem running from scratch.
For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.
**Ten steps, and they're in this order for a reason.** Each one assumes the previous one
actually worked — not that you ran it, that it *worked*. Every step below ends with a way to
check, and skipping those checks is how you end up three steps later debugging the wrong
thing entirely.
The steps that hurt most when rushed are **3** (naming) and **8** (testing failover). Step 3
because renaming anything afterwards means chasing it through every conf, and step 8 because
untested failover isn't redundancy — it's a guess you haven't checked yet.
Budget an evening. It is not hard, but it is not five minutes either.
---
## ━━━ BEFORE YOU START ━━━
Things that must be in place before you touch any scripts.
None of this is optional and none of it is Varaverk's job to install. Get these in place
first — every step after here assumes they're already true.
---
@@ -157,6 +169,21 @@ Replace these with your actual unRAID server hostnames. These must match exactly
`detect_hosts()` compares the running server's hostname against these two values to
know which server it is on. Everything else in the ecosystem flows from this.
**Get this wrong and nothing works, but nothing errors either.** If neither value matches,
`MY_ID` is never set, and every script that depends on it either exits early or resolves
`${MY_ID}_SOMETHING` to an empty variable and quietly takes the wrong branch. Copy the value
straight out of `hostname` on each box rather than typing what you think it is:
```bash
hostname # run this on each server, paste the exact output
```
> **The 15-character trap.** unRAID truncates the Server Name to 15 characters for NetBIOS.
> If your name is longer, what you set in the WebGUI and what `hostname` returns are two
> different strings. `detect_hosts()` has a fallback that matches a truncated 15-char
> hostname against a longer configured value — but only when the live hostname is
> *exactly* 15 characters. Don't rely on it. Use the real `hostname` output.
---
### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────
@@ -206,15 +233,27 @@ HOST1_CRITICAL_SYNC_SHARES=(
"/mnt/user/appdata-Fallback/Critical-Data|critical-data"
)
# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate)
HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=() # HOST2's own covers for HOST1, and vice versa
HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=(
# Containers to start when HOST1 goes down (Tier 1 = immediate)
# Lives in host1.conf. Named for the host being COVERED, not the host doing the covering.
FALLBACK_HOST1_TIER1=(
"Emby"
"VaultWarden"
"NginxProxyManager"
)
```
**Read that naming carefully — it catches people.** The variable is
`FALLBACK_${REMOTE_ID}_TIER${N}`, and `REMOTE_ID` is *the host that went down*. So
`FALLBACK_HOST1_TIER1` is the list HOST2 reads when HOST1 is unreachable. It defines
"what HOST1 needs covering", not "what HOST2 runs".
It lives in `host1.conf` for the same reason — HOST1 owns the description of its own
service stack. HOST2 receives it through the conf cache rather than keeping its own
opinion about what HOST1 runs. One list, one owner, no drift.
Tiers 2–4 activate progressively after `HOST1_TIER2_DELAY` etc., so a brief outage never
drags the entire remote stack across.
For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
and `Manual-Watchdogs.md`.
@@ -455,6 +494,18 @@ bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log
The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
detect it and start Tier 1 containers, then unblocks and waits for handback.
**Do not skip this step.** Everything up to here you can verify by reading output. Failover
is the one part you cannot confirm by looking at it — a typo in a tier list, a container name
that doesn't exist on the other side, a DDNS container that was renamed six months ago: all
of it sits there looking completely fine until the moment it's needed.
The test costs you twenty minutes and no downtime. The alternative is finding out at 2am,
during the outage, when you have exactly one chance to get it right.
**Watch the handback as closely as the failover.** Coming back is the harder half — it has to
stop the remote's DDNS, sync the data back, start the local containers, and only then bring
local DDNS up. Failover starting correctly tells you nothing about whether handback does.
See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
how to interpret the output.
@@ -651,6 +702,14 @@ Each subsystem has a README with the design decisions and a Manual with the conf
reference and troubleshooting. The `--status` flag on any script shows the current
configuration and state.
**Turn things on one at a time and give each one a few days.** Everything below is off by
default on a fresh install, and that's deliberate — a stack where six new subsystems went
live the same night is a stack where you have no idea which one to blame. Enable, watch it
through a full daily cycle, then enable the next.
Anything that deletes files — the arr cleanup scripts especially — gets a `--dry-run --log`
first. Read the list. Every time, not just the first time.
```
Watchdogs/ → README-Watchdogs.md configure memory limits, container lists
Media/ → README-Media.md enable arr cleanup, discovery scripts
README.md
+109 -12
View File
@@ -23,6 +23,51 @@ correct adapter. The scripts themselves never branch on the OS.
---
## ━━━ HOW THIS GOT HERE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
**One server. One script. Three lines.**
That's genuinely where this started. Something needed doing on a schedule, so I wrote three
lines and moved on.
Then it was two servers, 50 miles apart, and about **60 scripts** with nothing holding them
together. Every one standalone. Every one with its own hardcoded paths, its own container
names, its own idea of how to log something or send a notification.
**And it worked.** That's the part worth saying. The stack stayed up, the media kept playing,
the backups kept running. It was not broken.
It was just *miserable to change.*
Every adjustment meant finding every script that touched the thing, on both machines, and
editing each one the same way — then hoping I'd found them all. Miss one and it didn't error.
It just quietly kept doing the old thing until I noticed weeks later. Two servers meant two
copies of every mistake, and the two copies drifted apart a little more every time I touched
them.
So it got rebuilt, in roughly this order:
```
git stop editing live scripts on two machines and calling it version control
common.sh one place for log, notify, locking, host detection, retries
master.conf thresholds and toggles — shared, one copy
host*.conf credentials, paths, container names — per server
load_config.sh sources all of it, in the right order, at the top of every script
```
Then the long part: refactoring 60 standalone scripts into that shape, one at a time, until
every value lived in a conf file and every shared behaviour lived in `common.sh`. Change the
retry count once, every script gets it. Rename a container in one place, everything follows.
That is the whole idea, and it's why the structure looks the way it does. The three-file
config model and the shared library aren't design patterns picked out of a book — they're
the specific answer to "I have 60 scripts on two machines and I'm scared to touch any of
them."
That became Varaverk.
---
## ━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
This is a live system, not a finished product. HOST1 has been running the whole stack
@@ -57,7 +102,7 @@ HOST1 — unRAID-Gmer4Lfe
HOST2 — unRAID-Jayred365
Hardware: Intel i5 10th gen, 64GB RAM
Location: Remote — different building, different power utility
Location: Remote — 50 miles away, different power utility and ISP
Domain: Gmer4Lfe.us
Arrs: Anime_Shows, Anime_Movies — union with HOST1 via arr_sync
His own Emby — for his household
@@ -173,6 +218,10 @@ as sync targets and mount points — a mismatch is a broken path.
## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Two households, one library, and a hard line between "our media" and "your stuff." Everything
in this section comes down to that split — media is pooled and shared freely, personal data
never is, and no script gets to blur the two.
### ── Common Libraries — Shared Pool ─────────────────────────────────────────
Common media like Movies, Music, and TV Shows are a **shared pool**. Every container
@@ -257,8 +306,13 @@ across separate shares per-user or per-application.
## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Not everything needs to sync. Not everything should sync. Understanding what goes
where determines whether you need encryption and whether sync failure matters.
Not everything needs to sync. Not everything *should*. Two households means some of this
data is mine, some of it is his, and some of it belongs to both of us — and treating all
three the same way is how you end up either leaking someone's photos or losing the thing you
actually needed a copy of.
Where a piece of data lands in this table decides two things: whether it needs encryption,
and whether a failed sync is something to fix tonight or something to shrug at.
---
@@ -504,6 +558,15 @@ Monitoring:
## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Folders are grouped by **what a script does to the system**, not by which service it happens
to talk to. There's no `Sonarr/` folder — cleaning up after Sonarr is media work, watching
whether its container is alive is watchdog work, and those two things belong next to their
own kind.
Every folder has its own README explaining the group, and every script has a header
explaining itself. If you're looking for one specific thing, the README reference near the
bottom of this page is the faster route.
```
varaverk/
│
@@ -563,6 +626,16 @@ varaverk/
## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━
This is the part that fixed the 60-script problem. Three files, one rule:
**Thresholds and toggles go in `master.conf`. Credentials, paths and container names go in
`host*.conf`.** If you find yourself about to hardcode a value in a script, it belongs in one
of these instead — that's the whole discipline, and everything else follows from it.
Sparse checkout means each server only ever receives its own `host*.conf`. HOST1 never has
HOST2's credentials sitting on disk, and vice versa. The partner's conf arrives at runtime
through the RAM cache, which is why `conf_sync.sh` exists.
```bash
# ─────────────────────────────────────────────────────────────────────────────
# All three conf files live in Configurations/. load_config.sh sources them.
@@ -642,7 +715,8 @@ Every 4 hours:
2:30am Sunday:
weekly_sync_maintenance.sh stop → update images → clean sync → start → weekly restarts
+ playback_aware_lidarr/radarr/sonarr_discovery.sh (behavior-driven arr discovery)
+ playback_aware_{lidarr,radarr,sonarr}_discovery.sh
(behaviour-driven discovery from real playback history)
8am daily:
weekly_health_digest.sh DIGEST_PROFILE controls when it actually notifies
@@ -714,8 +788,9 @@ restart lands on the new image. Change the order and things break quietly rather
## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Problems are handled at the most appropriate level. Scripts escalate only when the
level below them has failed:
Nothing here jumps straight to the big hammer. Each layer only gets involved because the one
below it already tried and failed — and the layer holding the biggest hammer, the one that
can reboot the box, is the most reluctant of all:
```
Problem: Emby container using 22GB RAM (limit: 18GB)
@@ -833,7 +908,12 @@ CPU-based) instead of PaddleOCR. Same plugin, no VRAM.
## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
If you're setting this up from scratch on two servers:
If you're setting this up from scratch on two servers, the order below is not a suggestion.
Steps 2 and 3 are cheap now and expensive later — renaming a container or a share after the
confs, the fallback tiers and the sync profiles all reference it means touching every one of
them, and missing one doesn't error.
Do the naming first. Everything after it assumes the names are already right.
```
1. Follow Manual.md — end-to-end setup guide
@@ -854,6 +934,11 @@ If you're setting this up from scratch on two servers:
bash fallback_test.sh when ready — uses iptables simulation, no real downtime
```
**Do step 5.** Untested failover isn't redundancy, it's a belief. `fallback_test.sh` simulates
the outage with iptables rather than actually pulling the other server down, so there's no
excuse not to run it — and finding out your tier lists have a typo is a lot nicer on a Tuesday
afternoon than at 2am when the power's actually out.
---
## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
@@ -976,15 +1061,18 @@ Every one of these cost me something to learn. Reading them here is cheaper.
# --dry-run --log first on a new system. README-Media.md has the full procedure.
# 9. Scripts never call OS commands directly.
# All platform-specific operations go through plugin_*() functions defined in
# All platform-specific operations go through platform_*() functions defined in
# Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all
# isolated in the adapter. This is what makes the codebase portable.
# Adding a new OS = writing one adapter file, not patching dozens of scripts.
# 10. System_Essentials/ is platform-agnostic at the script level.
# Scripts like server_reboot.sh, mover_stop.sh, and webgui_watchdog.sh call
# adapter functions (platform_is_mover_running, platform_restart_service, etc.)
# The adapter handles what those mean on the actual OS. The scripts are clean.
# 10. Top-level System_Essentials/ is platform-agnostic. Plugin/unraid/ is not.
# server_reboot.sh lives at the top level and calls adapter functions only —
# platform_is_mover_running, platform_restart_service. It never names an OS.
# Scripts that manipulate an Unraid subsystem directly — mover_stop.sh,
# webgui_watchdog.sh, php_fpm_max_children.sh — live under Plugin/unraid/
# instead, because there is no generic version of them to write.
# Which folder a script is in tells you whether it survives a platform change.
# ─────────────────────────────────────────────────────────────────────────────
```
@@ -997,3 +1085,12 @@ Name: Varaverk
Repository: git@192.168.50.2:FailedProxy/Varaverk.git
Install path: /boot/config/plugins/varaverk
```
---
This is a home media stack for two households. It is not an enterprise product, it doesn't
want to be one, and it makes trade-offs no enterprise would accept — because the thing it's
actually optimising for is that nobody in either house has to know any of this exists.
The measure of success here isn't uptime percentages. It's that the power goes out 50 miles
away, and the show keeps playing.