Compare commits

..
2 Commits
Author SHA1 Message Date
Gmer4Lfe c54039392d Rewrite README and Manual front pages in project voice
The repo front page read like generic documentation rather than a record of
why any of this exists. Adds the origin story, the current honest state, real
failure history, and fixes drift in platform_*() naming, the fallback tier
variable pattern, and the Plugin/unraid script locations.
2026-08-01 23:47:50 -04:00
Gmer4Lfe 671bf2352b Give the README a voice and bring it current
It is the front page and it read like a spec sheet. Adds the incidents the safeguards came
from, an honest note about what is actually switched on, and the layers built since.
2026-08-01 23:34:34 -04:00
2 changed files with 288 additions and 33 deletions
+63 -4
View File
@@ -3,11 +3,23 @@
Getting a fresh two-server ecosystem running from scratch. Getting a fresh two-server ecosystem running from scratch.
For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers. For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.
**Ten steps, and they're in this order for a reason.** Each one assumes the previous one
actually worked — not that you ran it, that it *worked*. Every step below ends with a way to
check, and skipping those checks is how you end up three steps later debugging the wrong
thing entirely.
The steps that hurt most when rushed are **3** (naming) and **8** (testing failover). Step 3
because renaming anything afterwards means chasing it through every conf, and step 8 because
untested failover isn't redundancy — it's a guess you haven't checked yet.
Budget an evening. It is not hard, but it is not five minutes either.
--- ---
## ━━━ BEFORE YOU START ━━━ ## ━━━ BEFORE YOU START ━━━
Things that must be in place before you touch any scripts. None of this is optional and none of it is Varaverk's job to install. Get these in place
first — every step after here assumes they're already true.
--- ---
@@ -157,6 +169,21 @@ Replace these with your actual unRAID server hostnames. These must match exactly
`detect_hosts()` compares the running server's hostname against these two values to `detect_hosts()` compares the running server's hostname against these two values to
know which server it is on. Everything else in the ecosystem flows from this. know which server it is on. Everything else in the ecosystem flows from this.
**Get this wrong and nothing works, but nothing errors either.** If neither value matches,
`MY_ID` is never set, and every script that depends on it either exits early or resolves
`${MY_ID}_SOMETHING` to an empty variable and quietly takes the wrong branch. Copy the value
straight out of `hostname` on each box rather than typing what you think it is:
```bash
hostname # run this on each server, paste the exact output
```
> **The 15-character trap.** unRAID truncates the Server Name to 15 characters for NetBIOS.
> If your name is longer, what you set in the WebGUI and what `hostname` returns are two
> different strings. `detect_hosts()` has a fallback that matches a truncated 15-char
> hostname against a longer configured value — but only when the live hostname is
> *exactly* 15 characters. Don't rely on it. Use the real `hostname` output.
--- ---
### ── 3b. host1.conf — HOST1 identity ──────────────────────────────────────── ### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────
@@ -206,15 +233,27 @@ HOST1_CRITICAL_SYNC_SHARES=(
"/mnt/user/appdata-Fallback/Critical-Data|critical-data" "/mnt/user/appdata-Fallback/Critical-Data|critical-data"
) )
# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate) # Containers to start when HOST1 goes down (Tier 1 = immediate)
HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=() # HOST2's own covers for HOST1, and vice versa # Lives in host1.conf. Named for the host being COVERED, not the host doing the covering.
HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=( FALLBACK_HOST1_TIER1=(
"Emby" "Emby"
"VaultWarden" "VaultWarden"
"NginxProxyManager" "NginxProxyManager"
) )
``` ```
**Read that naming carefully — it catches people.** The variable is
`FALLBACK_${REMOTE_ID}_TIER${N}`, and `REMOTE_ID` is *the host that went down*. So
`FALLBACK_HOST1_TIER1` is the list HOST2 reads when HOST1 is unreachable. It defines
"what HOST1 needs covering", not "what HOST2 runs".
It lives in `host1.conf` for the same reason — HOST1 owns the description of its own
service stack. HOST2 receives it through the conf cache rather than keeping its own
opinion about what HOST1 runs. One list, one owner, no drift.
Tiers 24 activate progressively after `HOST1_TIER2_DELAY` etc., so a brief outage never
drags the entire remote stack across.
For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`, For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
and `Manual-Watchdogs.md`. and `Manual-Watchdogs.md`.
@@ -455,6 +494,18 @@ bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log
The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
detect it and start Tier 1 containers, then unblocks and waits for handback. detect it and start Tier 1 containers, then unblocks and waits for handback.
**Do not skip this step.** Everything up to here you can verify by reading output. Failover
is the one part you cannot confirm by looking at it — a typo in a tier list, a container name
that doesn't exist on the other side, a DDNS container that was renamed six months ago: all
of it sits there looking completely fine until the moment it's needed.
The test costs you twenty minutes and no downtime. The alternative is finding out at 2am,
during the outage, when you have exactly one chance to get it right.
**Watch the handback as closely as the failover.** Coming back is the harder half — it has to
stop the remote's DDNS, sync the data back, start the local containers, and only then bring
local DDNS up. Failover starting correctly tells you nothing about whether handback does.
See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
how to interpret the output. how to interpret the output.
@@ -651,6 +702,14 @@ Each subsystem has a README with the design decisions and a Manual with the conf
reference and troubleshooting. The `--status` flag on any script shows the current reference and troubleshooting. The `--status` flag on any script shows the current
configuration and state. configuration and state.
**Turn things on one at a time and give each one a few days.** Everything below is off by
default on a fresh install, and that's deliberate — a stack where six new subsystems went
live the same night is a stack where you have no idea which one to blame. Enable, watch it
through a full daily cycle, then enable the next.
Anything that deletes files — the arr cleanup scripts especially — gets a `--dry-run --log`
first. Read the list. Every time, not just the first time.
``` ```
Watchdogs/ → README-Watchdogs.md configure memory limits, container lists Watchdogs/ → README-Watchdogs.md configure memory limits, container lists
Media/ → README-Media.md enable arr cleanup, discovery scripts Media/ → README-Media.md enable arr cleanup, discovery scripts
+224 -28
View File
@@ -23,6 +23,72 @@ correct adapter. The scripts themselves never branch on the OS.
--- ---
## ━━━ HOW THIS GOT HERE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
**One server. One script. Three lines.**
That's genuinely where this started. Something needed doing on a schedule, so I wrote three
lines and moved on.
Then it was two servers, 50 miles apart, and about **60 scripts** with nothing holding them
together. Every one standalone. Every one with its own hardcoded paths, its own container
names, its own idea of how to log something or send a notification.
**And it worked.** That's the part worth saying. The stack stayed up, the media kept playing,
the backups kept running. It was not broken.
It was just *miserable to change.*
Every adjustment meant finding every script that touched the thing, on both machines, and
editing each one the same way — then hoping I'd found them all. Miss one and it didn't error.
It just quietly kept doing the old thing until I noticed weeks later. Two servers meant two
copies of every mistake, and the two copies drifted apart a little more every time I touched
them.
So it got rebuilt, in roughly this order:
```
git stop editing live scripts on two machines and calling it version control
common.sh one place for log, notify, locking, host detection, retries
master.conf thresholds and toggles — shared, one copy
host*.conf credentials, paths, container names — per server
load_config.sh sources all of it, in the right order, at the top of every script
```
Then the long part: refactoring 60 standalone scripts into that shape, one at a time, until
every value lived in a conf file and every shared behaviour lived in `common.sh`. Change the
retry count once, every script gets it. Rename a container in one place, everything follows.
That is the whole idea, and it's why the structure looks the way it does. The three-file
config model and the shared library aren't design patterns picked out of a book — they're
the specific answer to "I have 60 scripts on two machines and I'm scared to touch any of
them."
That became Varaverk.
---
## ━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
This is a live system, not a finished product. HOST1 has been running the whole stack
unattended for months. HOST2 is mid-rebuild, so some of what's described below is written
and tested but currently switched off:
```
FALLBACK_ENABLED=true ← running
PARTNERSHIP_ENABLED=true ← running
RSYNC_ENABLED=false ← off until HOST2 is back
CONF_SYNC_ENABLED=false ← off until HOST2 is back
```
Everything here is built and audited. The two-server half is waiting on the second server.
If a section reads like it's describing something that already happens every night — on
HOST1, it is. Across both hosts, it will be.
I'd rather tell you that than have you find out by reading the toggles.
---
## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
``` ```
@@ -36,7 +102,7 @@ HOST1 — unRAID-Gmer4Lfe
HOST2 — unRAID-Jayred365 HOST2 — unRAID-Jayred365
Hardware: Intel i5 10th gen, 64GB RAM Hardware: Intel i5 10th gen, 64GB RAM
Location: Remote — different building, different power utility Location: Remote — 50 miles away, different power utility and ISP
Domain: Gmer4Lfe.us Domain: Gmer4Lfe.us
Arrs: Anime_Shows, Anime_Movies — union with HOST1 via arr_sync Arrs: Anime_Shows, Anime_Movies — union with HOST1 via arr_sync
His own Emby — for his household His own Emby — for his household
@@ -76,7 +142,7 @@ break silently.
Dispatcharr # shared Live TV scheduler Dispatcharr # shared Live TV scheduler
``` ```
When `fallback.sh` on HOST2 reads `FALLBACK_HOST2_COVERS_HOST1_TIER1=(..."Emby"...)`, When `fallback.sh` on HOST2 reads `FALLBACK_HOST1_TIER1=(..."Emby"...)`,
it looks for a container called exactly `Emby` and starts it. HOST2 must have a it looks for a container called exactly `Emby` and starts it. HOST2 must have a
container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
its volume mounts pointing at the mirrored share paths. This is what makes one its volume mounts pointing at the mirrored share paths. This is what makes one
@@ -113,7 +179,7 @@ the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts tar
# The entire ecosystem uses container names as identifiers. # The entire ecosystem uses container names as identifiers.
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia") # docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
# → looks for exactly "NginxProxyManager" on the running server # → looks for exactly "NginxProxyManager" on the running server
# fallback.sh: FALLBACK_HOST2_COVERS_HOST1_TIER1=("Emby" "VaultWarden") # fallback.sh: FALLBACK_HOST1_TIER1=("Emby" "VaultWarden")
# → looks for exactly "Emby" and "VaultWarden" on HOST2 # → looks for exactly "Emby" and "VaultWarden" on HOST2
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia") # docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
# → looks for exactly those names in docker ps # → looks for exactly those names in docker ps
@@ -152,6 +218,10 @@ as sync targets and mount points — a mismatch is a broken path.
## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Two households, one library, and a hard line between "our media" and "your stuff." Everything
in this section comes down to that split — media is pooled and shared freely, personal data
never is, and no script gets to blur the two.
### ── Common Libraries — Shared Pool ───────────────────────────────────────── ### ── Common Libraries — Shared Pool ─────────────────────────────────────────
Common media like Movies, Music, and TV Shows are a **shared pool**. Every container Common media like Movies, Music, and TV Shows are a **shared pool**. Every container
@@ -226,9 +296,9 @@ across separate shares per-user or per-application.
# arr_sync.sh guarantees both arrs agree on what exists before files move. # arr_sync.sh guarantees both arrs agree on what exists before files move.
# rsync then spreads the files to make storage match what arrs already know. # rsync then spreads the files to make storage match what arrs already know.
# #
# arr_cleanup.sh (daily) removes true orphans — files no arr tracks. # The per-arr cleanups (daily) remove true orphans — files no arr tracks.
# arr_cleanup uses the union model too: a file is only an orphan if # They use the union model too: a file is only an orphan if neither arr
# neither arr on either server has it indexed. # on either server has it indexed.
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
``` ```
@@ -236,8 +306,13 @@ across separate shares per-user or per-application.
## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Not everything needs to sync. Not everything should sync. Understanding what goes Not everything needs to sync. Not everything *should*. Two households means some of this
where determines whether you need encryption and whether sync failure matters. data is mine, some of it is his, and some of it belongs to both of us — and treating all
three the same way is how you end up either leaking someone's photos or losing the thing you
actually needed a copy of.
Where a piece of data lands in this table decides two things: whether it needs encryption,
and whether a failed sync is something to fix tonight or something to shrug at.
--- ---
@@ -311,14 +386,18 @@ where determines whether you need encryption and whether sync failure matters.
## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> **Understanding this distinction prevents a lot of confusion.** > **Get this wrong and nothing else in here will make sense.**
Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a Every guide you'll find describes the same thing: a primary, a standby, shared storage or
primary server and a standby server with shared storage or synchronous replication. synchronous replication. The standby sits there doing nothing, existing only for the day the
The standby exists only to take over when the primary fails. It runs nothing primary dies. Pacemaker, DRBD, Proxmox HA — all built for a datacenter where downtime is
independently. Its purpose is 100% failover coverage with no service interruption. measured in dollars per second.
**This is not that.** **This is not that.** Nobody is on call. There is no shared storage, because there is no
datacenter — there are two houses. The standby isn't idle, because it isn't a standby: it's
somebody's actual media server, and they're watching it right now.
The bar is not five nines. The bar is *nobody notices their show stopped working.*
--- ---
@@ -479,6 +558,15 @@ Monitoring:
## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Folders are grouped by **what a script does to the system**, not by which service it happens
to talk to. There's no `Sonarr/` folder — cleaning up after Sonarr is media work, watching
whether its container is alive is watchdog work, and those two things belong next to their
own kind.
Every folder has its own README explaining the group, and every script has a header
explaining itself. If you're looking for one specific thing, the README reference near the
bottom of this page is the faster route.
``` ```
varaverk/ varaverk/
@@ -538,6 +626,16 @@ varaverk/
## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━ ## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━
This is the part that fixed the 60-script problem. Three files, one rule:
**Thresholds and toggles go in `master.conf`. Credentials, paths and container names go in
`host*.conf`.** If you find yourself about to hardcode a value in a script, it belongs in one
of these instead — that's the whole discipline, and everything else follows from it.
Sparse checkout means each server only ever receives its own `host*.conf`. HOST1 never has
HOST2's credentials sitting on disk, and vice versa. The partner's conf arrives at runtime
through the RAM cache, which is why `conf_sync.sh` exists.
```bash ```bash
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# All three conf files live in Configurations/. load_config.sh sources them. # All three conf files live in Configurations/. load_config.sh sources them.
@@ -617,7 +715,8 @@ Every 4 hours:
2:30am Sunday: 2:30am Sunday:
weekly_sync_maintenance.sh stop → update images → clean sync → start → weekly restarts weekly_sync_maintenance.sh stop → update images → clean sync → start → weekly restarts
+ playback_aware_lidarr/radarr/sonarr_discovery.sh (behavior-driven arr discovery) + playback_aware_{lidarr,radarr,sonarr}_discovery.sh
(behaviour-driven discovery from real playback history)
8am daily: 8am daily:
weekly_health_digest.sh DIGEST_PROFILE controls when it actually notifies weekly_health_digest.sh DIGEST_PROFILE controls when it actually notifies
@@ -656,26 +755,42 @@ Throughout the day:
rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares) rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares)
media_shares_permissions.sh (fix ownership before arr cleanup) media_shares_permissions.sh (fix ownership before arr cleanup)
media_cleaner.sh anime + media (remove scene junk) media_cleaner.sh anime + media (remove scene junk)
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files) lidarr_release_fixer.sh (fix wrong MusicBrainz editions first)
docker_daily_restart.sh (NPM, Authelia, Dispatcharr fresh start) lidarr_duplicate_artist_cleanup.sh (drop phantom zero-file duplicates)
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files — library side)
arr_download_orphan_cleaner.sh (remove orphans — download side)
radarr/sonarr_classification_scan.sh --move (anime/kids sitting in the wrong root)
docker_update.sh → docker_daily_restart.sh (pull first, then restart onto it)
2:30am Sunday: 2:30am Sunday:
Stop Emby + auth stack both sides Stop Emby + auth stack both sides
Pull updated Docker images (zero extra downtime — already stopped) Pull updated Docker images (zero extra downtime — already stopped)
rsync Emby + Critical-Data clean mirror rsync Emby + Critical-Data clean mirror
Start containers on new images Start containers on new images
docker_weekly_restart.sh (NextCloud, AdGuard, Immich) docker_update.sh --weekly → docker_weekly_restart.sh
arr_corruption_scan.sh (ffprobe every tracked video)
arr_full_rescan.sh (force a real disk↔database reconcile)
7am Sunday: 7am Sunday:
sunday_morning_coffee_report.sh (everything that happened this week) sunday_morning_coffee_report.sh (everything that happened this week)
Monthly (self-gated on 30-day uptime):
ZFS scrub, SMART long tests
docker_update.sh --remainder (everything not in the managed lists)
``` ```
The ordering in the 1am block is not arbitrary. Permissions run before the cleanups because
the cleanups need correct ownership to delete. The release fixer runs before the Lidarr
cleanup so it isn't cleaning up files it's about to fix. Updates run before restarts so the
restart lands on the new image. Change the order and things break quietly rather than loudly.
--- ---
## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Problems are handled at the most appropriate level. Scripts escalate only when the Nothing here jumps straight to the big hammer. Each layer only gets involved because the one
level below them has failed: below it already tried and failed — and the layer holding the biggest hammer, the one that
can reboot the box, is the most reluctant of all:
``` ```
Problem: Emby container using 22GB RAM (limit: 18GB) Problem: Emby container using 22GB RAM (limit: 18GB)
@@ -793,7 +908,12 @@ CPU-based) instead of PaddleOCR. Same plugin, no VRAM.
## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
If you're setting this up from scratch on two servers: If you're setting this up from scratch on two servers, the order below is not a suggestion.
Steps 2 and 3 are cheap now and expensive later — renaming a container or a share after the
confs, the fallback tiers and the sync profiles all reference it means touching every one of
them, and missing one doesn't error.
Do the naming first. Everything after it assumes the names are already right.
``` ```
1. Follow Manual.md — end-to-end setup guide 1. Follow Manual.md — end-to-end setup guide
@@ -814,6 +934,11 @@ If you're setting this up from scratch on two servers:
bash fallback_test.sh when ready — uses iptables simulation, no real downtime bash fallback_test.sh when ready — uses iptables simulation, no real downtime
``` ```
**Do step 5.** Untested failover isn't redundancy, it's a belief. `fallback_test.sh` simulates
the outage with iptables rather than actually pulling the other server down, so there's no
excuse not to run it — and finding out your tier lists have a typo is a lot nicer on a Tuesday
afternoon than at 2am when the power's actually out.
--- ---
## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
@@ -828,15 +953,74 @@ If you're setting this up from scratch on two servers:
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback | | [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles | | [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard | | [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
| [README-Arrs_Stack.md](Arrs_Stack/README-Arrs_Stack.md) | Union model, orphan cleanup, classification scans, corruption scan, download-orphan cleaner, discovery |
| [README-Watchdogs.md](Watchdogs/README-Watchdogs.md) | The four tiers, strike systems, reboot rate limiting, cross-watchdog coordination |
| [README-Rsync.md](Rsync/README-Rsync.md) | Profiles, pre-flight checks, merge-run, source path guards, runtime ceiling |
| [README-Deployment.md](Deployment/README-Deployment.md) | The conf templates — the only versioned record of what configuration exists |
| [README-Kernel.md](Kernel/README-Kernel.md) | The scoring kernel consumers source for discovery decisions |
| [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages | | [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
| [README-unraid.md](Plugin/unraid/README-unraid.md) | The Unraid platform layer — adapter contract, three-layer web UI, array lifecycle hooks |
| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler | | [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
| [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export | | [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
Every folder has a `README-*.md` for *why it exists* and most have a `Manual-*.md` for
*how to operate it*. Every script has a header block covering purpose, operational model,
design principles, safeguards, configuration and runtime modes — if you want to know what a
script does, the top of the file is the authoritative answer, not this page.
---
## ━━━ THINGS THAT ACTUALLY WENT WRONG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Most of the safeguards in this repo exist because something bit me first. If you're wondering
why a script checks something that looks paranoid, it's usually because it didn't, once.
**The 755 GB nobody was looking at.**
Every cleanup script walked the *library* side. Nothing walked the *download* side. Completed
downloads the arrs had quietly stopped tracking piled up in SABnzbd's Completed folders —
oldest from 2022 — until the cache pool hit 89%. That's `arr_download_orphan_cleaner.sh`.
**Lidarr said it had 23% of my music.**
1,004 of 1,357 artists reporting a fraction of their real track count, with every file sitting
right there on disk and no scan running. Organic scans only touch files involved in an import,
so an untouched library just drifts. That's `arr_full_rescan.sh`.
**"Full House" moved. Except it didn't.**
Sonarr's API reported `episodeFileCount:192` at the new path while the actual 75 GB was still
at the old one — `moveFiles=true` flips the database instantly, but the physical move is a
separate async command draining one at a time behind ~20 others. Every classification move now
polls its own command to completion before believing the database.
**mtime lies about when a file arrived.**
Measured on one week's imports: 400 of 400 files had mtimes over 7 days old. One was 9,613
days old. An import preserves the release's original timestamp, so orphan detection has to
gate on ctime — which means the nightly permissions job must only touch files that are
actually wrong, because `chown` restamps ctime even when it changes nothing. Make that pass
unconditional and orphan collection silently stops. Nothing errors. The pool just fills.
**A rsync merge pass ran for 9.7 days.**
Now every transfer attempt is capped, and resumes rather than restarting — which only works
because `--partial` is in the defaults.
**R.E.M. was already in the library. Apparently.**
The duplicate check interpolated the artist name straight into a `grep` pattern, so `R.E.M.`
happily matched a stored `RxExMy`. Music is full of regex metacharacters. It's a literal
match now.
**"Craig of the Creek" is not anime.**
TheTVDB tags it with the Anime genre anyway, which is why the anime signal needs corroboration
from language or network before it counts. In the other direction, "The Pussycat Dolls
Present" is a real MTV show with zero votes and no IMDb ID — which is why the junk-detection
heuristic that works on movies is deliberately *not* ported to TV.
**`php -S` silently drops POST bodies** on Unraid's PHP 8.4. The webhook listener is Node.js
for exactly that reason, and no other.
--- ---
## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Things that are different from what you might expect: Every one of these cost me something to learn. Reading them here is cheaper.
```bash ```bash
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
@@ -877,15 +1061,18 @@ Things that are different from what you might expect:
# --dry-run --log first on a new system. README-Media.md has the full procedure. # --dry-run --log first on a new system. README-Media.md has the full procedure.
# 9. Scripts never call OS commands directly. # 9. Scripts never call OS commands directly.
# All platform-specific operations go through plugin_*() functions defined in # All platform-specific operations go through platform_*() functions defined in
# Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all # Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all
# isolated in the adapter. This is what makes the codebase portable. # isolated in the adapter. This is what makes the codebase portable.
# Adding a new OS = writing one adapter file, not patching dozens of scripts. # Adding a new OS = writing one adapter file, not patching dozens of scripts.
# 10. System_Essentials/ is platform-agnostic at the script level. # 10. Top-level System_Essentials/ is platform-agnostic. Plugin/unraid/ is not.
# Scripts like server_reboot.sh, mover_stop.sh, and webgui_watchdog.sh call # server_reboot.sh lives at the top level and calls adapter functions only —
# adapter functions (platform_is_mover_running, platform_restart_service, etc.) # platform_is_mover_running, platform_restart_service. It never names an OS.
# The adapter handles what those mean on the actual OS. The scripts are clean. # Scripts that manipulate an Unraid subsystem directly — mover_stop.sh,
# webgui_watchdog.sh, php_fpm_max_children.sh — live under Plugin/unraid/
# instead, because there is no generic version of them to write.
# Which folder a script is in tells you whether it survives a platform change.
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
``` ```
@@ -895,6 +1082,15 @@ Things that are different from what you might expect:
``` ```
Name: Varaverk Name: Varaverk
Repository: git@192.168.50.2:Varaverk/varaverk.git Repository: git@192.168.50.2:FailedProxy/Varaverk.git
Install path: /boot/config/plugins/varaverk Install path: /boot/config/plugins/varaverk
``` ```
---
This is a home media stack for two households. It is not an enterprise product, it doesn't
want to be one, and it makes trade-offs no enterprise would accept — because the thing it's
actually optimising for is that nobody in either house has to know any of this exists.
The measure of success here isn't uptime percentages. It's that the power goes out 50 miles
away, and the show keeps playing.