Compare commits
2
Commits
1ed6b92118
...
c54039392d
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c54039392d | ||
|
|
671bf2352b |
@@ -3,11 +3,23 @@
|
||||
Getting a fresh two-server ecosystem running from scratch.
|
||||
For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.
|
||||
|
||||
**Ten steps, and they're in this order for a reason.** Each one assumes the previous one
|
||||
actually worked — not that you ran it, that it *worked*. Every step below ends with a way to
|
||||
check, and skipping those checks is how you end up three steps later debugging the wrong
|
||||
thing entirely.
|
||||
|
||||
The steps that hurt most when rushed are **3** (naming) and **8** (testing failover). Step 3
|
||||
because renaming anything afterwards means chasing it through every conf, and step 8 because
|
||||
untested failover isn't redundancy — it's a guess you haven't checked yet.
|
||||
|
||||
Budget an evening. It is not hard, but it is not five minutes either.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ BEFORE YOU START ━━━
|
||||
|
||||
Things that must be in place before you touch any scripts.
|
||||
None of this is optional and none of it is Varaverk's job to install. Get these in place
|
||||
first — every step after here assumes they're already true.
|
||||
|
||||
---
|
||||
|
||||
@@ -157,6 +169,21 @@ Replace these with your actual unRAID server hostnames. These must match exactly
|
||||
`detect_hosts()` compares the running server's hostname against these two values to
|
||||
know which server it is on. Everything else in the ecosystem flows from this.
|
||||
|
||||
**Get this wrong and nothing works, but nothing errors either.** If neither value matches,
|
||||
`MY_ID` is never set, and every script that depends on it either exits early or resolves
|
||||
`${MY_ID}_SOMETHING` to an empty variable and quietly takes the wrong branch. Copy the value
|
||||
straight out of `hostname` on each box rather than typing what you think it is:
|
||||
|
||||
```bash
|
||||
hostname # run this on each server, paste the exact output
|
||||
```
|
||||
|
||||
> **The 15-character trap.** unRAID truncates the Server Name to 15 characters for NetBIOS.
|
||||
> If your name is longer, what you set in the WebGUI and what `hostname` returns are two
|
||||
> different strings. `detect_hosts()` has a fallback that matches a truncated 15-char
|
||||
> hostname against a longer configured value — but only when the live hostname is
|
||||
> *exactly* 15 characters. Don't rely on it. Use the real `hostname` output.
|
||||
|
||||
---
|
||||
|
||||
### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────
|
||||
@@ -206,15 +233,27 @@ HOST1_CRITICAL_SYNC_SHARES=(
|
||||
"/mnt/user/appdata-Fallback/Critical-Data|critical-data"
|
||||
)
|
||||
|
||||
# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate)
|
||||
HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=() # HOST2's own covers for HOST1, and vice versa
|
||||
HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
||||
# Containers to start when HOST1 goes down (Tier 1 = immediate)
|
||||
# Lives in host1.conf. Named for the host being COVERED, not the host doing the covering.
|
||||
FALLBACK_HOST1_TIER1=(
|
||||
"Emby"
|
||||
"VaultWarden"
|
||||
"NginxProxyManager"
|
||||
)
|
||||
```
|
||||
|
||||
**Read that naming carefully — it catches people.** The variable is
|
||||
`FALLBACK_${REMOTE_ID}_TIER${N}`, and `REMOTE_ID` is *the host that went down*. So
|
||||
`FALLBACK_HOST1_TIER1` is the list HOST2 reads when HOST1 is unreachable. It defines
|
||||
"what HOST1 needs covering", not "what HOST2 runs".
|
||||
|
||||
It lives in `host1.conf` for the same reason — HOST1 owns the description of its own
|
||||
service stack. HOST2 receives it through the conf cache rather than keeping its own
|
||||
opinion about what HOST1 runs. One list, one owner, no drift.
|
||||
|
||||
Tiers 2–4 activate progressively after `HOST1_TIER2_DELAY` etc., so a brief outage never
|
||||
drags the entire remote stack across.
|
||||
|
||||
For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
|
||||
and `Manual-Watchdogs.md`.
|
||||
|
||||
@@ -455,6 +494,18 @@ bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log
|
||||
The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
|
||||
detect it and start Tier 1 containers, then unblocks and waits for handback.
|
||||
|
||||
**Do not skip this step.** Everything up to here you can verify by reading output. Failover
|
||||
is the one part you cannot confirm by looking at it — a typo in a tier list, a container name
|
||||
that doesn't exist on the other side, a DDNS container that was renamed six months ago: all
|
||||
of it sits there looking completely fine until the moment it's needed.
|
||||
|
||||
The test costs you twenty minutes and no downtime. The alternative is finding out at 2am,
|
||||
during the outage, when you have exactly one chance to get it right.
|
||||
|
||||
**Watch the handback as closely as the failover.** Coming back is the harder half — it has to
|
||||
stop the remote's DDNS, sync the data back, start the local containers, and only then bring
|
||||
local DDNS up. Failover starting correctly tells you nothing about whether handback does.
|
||||
|
||||
See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
|
||||
how to interpret the output.
|
||||
|
||||
@@ -651,6 +702,14 @@ Each subsystem has a README with the design decisions and a Manual with the conf
|
||||
reference and troubleshooting. The `--status` flag on any script shows the current
|
||||
configuration and state.
|
||||
|
||||
**Turn things on one at a time and give each one a few days.** Everything below is off by
|
||||
default on a fresh install, and that's deliberate — a stack where six new subsystems went
|
||||
live the same night is a stack where you have no idea which one to blame. Enable, watch it
|
||||
through a full daily cycle, then enable the next.
|
||||
|
||||
Anything that deletes files — the arr cleanup scripts especially — gets a `--dry-run --log`
|
||||
first. Read the list. Every time, not just the first time.
|
||||
|
||||
```
|
||||
Watchdogs/ → README-Watchdogs.md configure memory limits, container lists
|
||||
Media/ → README-Media.md enable arr cleanup, discovery scripts
|
||||
|
||||
@@ -23,6 +23,72 @@ correct adapter. The scripts themselves never branch on the OS.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THIS GOT HERE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
**One server. One script. Three lines.**
|
||||
|
||||
That's genuinely where this started. Something needed doing on a schedule, so I wrote three
|
||||
lines and moved on.
|
||||
|
||||
Then it was two servers, 50 miles apart, and about **60 scripts** with nothing holding them
|
||||
together. Every one standalone. Every one with its own hardcoded paths, its own container
|
||||
names, its own idea of how to log something or send a notification.
|
||||
|
||||
**And it worked.** That's the part worth saying. The stack stayed up, the media kept playing,
|
||||
the backups kept running. It was not broken.
|
||||
|
||||
It was just *miserable to change.*
|
||||
|
||||
Every adjustment meant finding every script that touched the thing, on both machines, and
|
||||
editing each one the same way — then hoping I'd found them all. Miss one and it didn't error.
|
||||
It just quietly kept doing the old thing until I noticed weeks later. Two servers meant two
|
||||
copies of every mistake, and the two copies drifted apart a little more every time I touched
|
||||
them.
|
||||
|
||||
So it got rebuilt, in roughly this order:
|
||||
|
||||
```
|
||||
git stop editing live scripts on two machines and calling it version control
|
||||
common.sh one place for log, notify, locking, host detection, retries
|
||||
master.conf thresholds and toggles — shared, one copy
|
||||
host*.conf credentials, paths, container names — per server
|
||||
load_config.sh sources all of it, in the right order, at the top of every script
|
||||
```
|
||||
|
||||
Then the long part: refactoring 60 standalone scripts into that shape, one at a time, until
|
||||
every value lived in a conf file and every shared behaviour lived in `common.sh`. Change the
|
||||
retry count once, every script gets it. Rename a container in one place, everything follows.
|
||||
|
||||
That is the whole idea, and it's why the structure looks the way it does. The three-file
|
||||
config model and the shared library aren't design patterns picked out of a book — they're
|
||||
the specific answer to "I have 60 scripts on two machines and I'm scared to touch any of
|
||||
them."
|
||||
|
||||
That became Varaverk.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
This is a live system, not a finished product. HOST1 has been running the whole stack
|
||||
unattended for months. HOST2 is mid-rebuild, so some of what's described below is written
|
||||
and tested but currently switched off:
|
||||
|
||||
```
|
||||
FALLBACK_ENABLED=true ← running
|
||||
PARTNERSHIP_ENABLED=true ← running
|
||||
RSYNC_ENABLED=false ← off until HOST2 is back
|
||||
CONF_SYNC_ENABLED=false ← off until HOST2 is back
|
||||
```
|
||||
|
||||
Everything here is built and audited. The two-server half is waiting on the second server.
|
||||
If a section reads like it's describing something that already happens every night — on
|
||||
HOST1, it is. Across both hosts, it will be.
|
||||
|
||||
I'd rather tell you that than have you find out by reading the toggles.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```
|
||||
@@ -36,7 +102,7 @@ HOST1 — unRAID-Gmer4Lfe
|
||||
|
||||
HOST2 — unRAID-Jayred365
|
||||
Hardware: Intel i5 10th gen, 64GB RAM
|
||||
Location: Remote — different building, different power utility
|
||||
Location: Remote — 50 miles away, different power utility and ISP
|
||||
Domain: Gmer4Lfe.us
|
||||
Arrs: Anime_Shows, Anime_Movies — union with HOST1 via arr_sync
|
||||
His own Emby — for his household
|
||||
@@ -76,7 +142,7 @@ break silently.
|
||||
Dispatcharr # shared Live TV scheduler
|
||||
```
|
||||
|
||||
When `fallback.sh` on HOST2 reads `FALLBACK_HOST2_COVERS_HOST1_TIER1=(..."Emby"...)`,
|
||||
When `fallback.sh` on HOST2 reads `FALLBACK_HOST1_TIER1=(..."Emby"...)`,
|
||||
it looks for a container called exactly `Emby` and starts it. HOST2 must have a
|
||||
container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
|
||||
its volume mounts pointing at the mirrored share paths. This is what makes one
|
||||
@@ -113,7 +179,7 @@ the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts tar
|
||||
# The entire ecosystem uses container names as identifiers.
|
||||
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
|
||||
# → looks for exactly "NginxProxyManager" on the running server
|
||||
# fallback.sh: FALLBACK_HOST2_COVERS_HOST1_TIER1=("Emby" "VaultWarden")
|
||||
# fallback.sh: FALLBACK_HOST1_TIER1=("Emby" "VaultWarden")
|
||||
# → looks for exactly "Emby" and "VaultWarden" on HOST2
|
||||
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
|
||||
# → looks for exactly those names in docker ps
|
||||
@@ -152,6 +218,10 @@ as sync targets and mount points — a mismatch is a broken path.
|
||||
|
||||
## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Two households, one library, and a hard line between "our media" and "your stuff." Everything
|
||||
in this section comes down to that split — media is pooled and shared freely, personal data
|
||||
never is, and no script gets to blur the two.
|
||||
|
||||
### ── Common Libraries — Shared Pool ─────────────────────────────────────────
|
||||
|
||||
Common media like Movies, Music, and TV Shows are a **shared pool**. Every container
|
||||
@@ -226,9 +296,9 @@ across separate shares per-user or per-application.
|
||||
# arr_sync.sh guarantees both arrs agree on what exists before files move.
|
||||
# rsync then spreads the files to make storage match what arrs already know.
|
||||
#
|
||||
# arr_cleanup.sh (daily) removes true orphans — files no arr tracks.
|
||||
# arr_cleanup uses the union model too: a file is only an orphan if
|
||||
# neither arr on either server has it indexed.
|
||||
# The per-arr cleanups (daily) remove true orphans — files no arr tracks.
|
||||
# They use the union model too: a file is only an orphan if neither arr
|
||||
# on either server has it indexed.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
@@ -236,8 +306,13 @@ across separate shares per-user or per-application.
|
||||
|
||||
## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Not everything needs to sync. Not everything should sync. Understanding what goes
|
||||
where determines whether you need encryption and whether sync failure matters.
|
||||
Not everything needs to sync. Not everything *should*. Two households means some of this
|
||||
data is mine, some of it is his, and some of it belongs to both of us — and treating all
|
||||
three the same way is how you end up either leaking someone's photos or losing the thing you
|
||||
actually needed a copy of.
|
||||
|
||||
Where a piece of data lands in this table decides two things: whether it needs encryption,
|
||||
and whether a failed sync is something to fix tonight or something to shrug at.
|
||||
|
||||
---
|
||||
|
||||
@@ -311,14 +386,18 @@ where determines whether you need encryption and whether sync failure matters.
|
||||
|
||||
## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
> **Understanding this distinction prevents a lot of confusion.**
|
||||
> **Get this wrong and nothing else in here will make sense.**
|
||||
|
||||
Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a
|
||||
primary server and a standby server with shared storage or synchronous replication.
|
||||
The standby exists only to take over when the primary fails. It runs nothing
|
||||
independently. Its purpose is 100% failover coverage with no service interruption.
|
||||
Every guide you'll find describes the same thing: a primary, a standby, shared storage or
|
||||
synchronous replication. The standby sits there doing nothing, existing only for the day the
|
||||
primary dies. Pacemaker, DRBD, Proxmox HA — all built for a datacenter where downtime is
|
||||
measured in dollars per second.
|
||||
|
||||
**This is not that.**
|
||||
**This is not that.** Nobody is on call. There is no shared storage, because there is no
|
||||
datacenter — there are two houses. The standby isn't idle, because it isn't a standby: it's
|
||||
somebody's actual media server, and they're watching it right now.
|
||||
|
||||
The bar is not five nines. The bar is *nobody notices their show stopped working.*
|
||||
|
||||
---
|
||||
|
||||
@@ -479,6 +558,15 @@ Monitoring:
|
||||
|
||||
## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Folders are grouped by **what a script does to the system**, not by which service it happens
|
||||
to talk to. There's no `Sonarr/` folder — cleaning up after Sonarr is media work, watching
|
||||
whether its container is alive is watchdog work, and those two things belong next to their
|
||||
own kind.
|
||||
|
||||
Every folder has its own README explaining the group, and every script has a header
|
||||
explaining itself. If you're looking for one specific thing, the README reference near the
|
||||
bottom of this page is the faster route.
|
||||
|
||||
```
|
||||
varaverk/
|
||||
│
|
||||
@@ -538,6 +626,16 @@ varaverk/
|
||||
|
||||
## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
This is the part that fixed the 60-script problem. Three files, one rule:
|
||||
|
||||
**Thresholds and toggles go in `master.conf`. Credentials, paths and container names go in
|
||||
`host*.conf`.** If you find yourself about to hardcode a value in a script, it belongs in one
|
||||
of these instead — that's the whole discipline, and everything else follows from it.
|
||||
|
||||
Sparse checkout means each server only ever receives its own `host*.conf`. HOST1 never has
|
||||
HOST2's credentials sitting on disk, and vice versa. The partner's conf arrives at runtime
|
||||
through the RAM cache, which is why `conf_sync.sh` exists.
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# All three conf files live in Configurations/. load_config.sh sources them.
|
||||
@@ -617,7 +715,8 @@ Every 4 hours:
|
||||
|
||||
2:30am Sunday:
|
||||
weekly_sync_maintenance.sh stop → update images → clean sync → start → weekly restarts
|
||||
+ playback_aware_lidarr/radarr/sonarr_discovery.sh (behavior-driven arr discovery)
|
||||
+ playback_aware_{lidarr,radarr,sonarr}_discovery.sh
|
||||
(behaviour-driven discovery from real playback history)
|
||||
|
||||
8am daily:
|
||||
weekly_health_digest.sh DIGEST_PROFILE controls when it actually notifies
|
||||
@@ -656,26 +755,42 @@ Throughout the day:
|
||||
rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares)
|
||||
media_shares_permissions.sh (fix ownership before arr cleanup)
|
||||
media_cleaner.sh anime + media (remove scene junk)
|
||||
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files)
|
||||
docker_daily_restart.sh (NPM, Authelia, Dispatcharr fresh start)
|
||||
lidarr_release_fixer.sh (fix wrong MusicBrainz editions first)
|
||||
lidarr_duplicate_artist_cleanup.sh (drop phantom zero-file duplicates)
|
||||
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files — library side)
|
||||
arr_download_orphan_cleaner.sh (remove orphans — download side)
|
||||
radarr/sonarr_classification_scan.sh --move (anime/kids sitting in the wrong root)
|
||||
docker_update.sh → docker_daily_restart.sh (pull first, then restart onto it)
|
||||
|
||||
2:30am Sunday:
|
||||
Stop Emby + auth stack both sides
|
||||
Pull updated Docker images (zero extra downtime — already stopped)
|
||||
rsync Emby + Critical-Data clean mirror
|
||||
Start containers on new images
|
||||
docker_weekly_restart.sh (NextCloud, AdGuard, Immich)
|
||||
docker_update.sh --weekly → docker_weekly_restart.sh
|
||||
arr_corruption_scan.sh (ffprobe every tracked video)
|
||||
arr_full_rescan.sh (force a real disk↔database reconcile)
|
||||
|
||||
7am Sunday:
|
||||
sunday_morning_coffee_report.sh (everything that happened this week)
|
||||
|
||||
Monthly (self-gated on 30-day uptime):
|
||||
ZFS scrub, SMART long tests
|
||||
docker_update.sh --remainder (everything not in the managed lists)
|
||||
```
|
||||
|
||||
The ordering in the 1am block is not arbitrary. Permissions run before the cleanups because
|
||||
the cleanups need correct ownership to delete. The release fixer runs before the Lidarr
|
||||
cleanup so it isn't cleaning up files it's about to fix. Updates run before restarts so the
|
||||
restart lands on the new image. Change the order and things break quietly rather than loudly.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Problems are handled at the most appropriate level. Scripts escalate only when the
|
||||
level below them has failed:
|
||||
Nothing here jumps straight to the big hammer. Each layer only gets involved because the one
|
||||
below it already tried and failed — and the layer holding the biggest hammer, the one that
|
||||
can reboot the box, is the most reluctant of all:
|
||||
|
||||
```
|
||||
Problem: Emby container using 22GB RAM (limit: 18GB)
|
||||
@@ -793,7 +908,12 @@ CPU-based) instead of PaddleOCR. Same plugin, no VRAM.
|
||||
|
||||
## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
If you're setting this up from scratch on two servers:
|
||||
If you're setting this up from scratch on two servers, the order below is not a suggestion.
|
||||
Steps 2 and 3 are cheap now and expensive later — renaming a container or a share after the
|
||||
confs, the fallback tiers and the sync profiles all reference it means touching every one of
|
||||
them, and missing one doesn't error.
|
||||
|
||||
Do the naming first. Everything after it assumes the names are already right.
|
||||
|
||||
```
|
||||
1. Follow Manual.md — end-to-end setup guide
|
||||
@@ -814,6 +934,11 @@ If you're setting this up from scratch on two servers:
|
||||
bash fallback_test.sh when ready — uses iptables simulation, no real downtime
|
||||
```
|
||||
|
||||
**Do step 5.** Untested failover isn't redundancy, it's a belief. `fallback_test.sh` simulates
|
||||
the outage with iptables rather than actually pulling the other server down, so there's no
|
||||
excuse not to run it — and finding out your tier lists have a typo is a lot nicer on a Tuesday
|
||||
afternoon than at 2am when the power's actually out.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
@@ -828,15 +953,74 @@ If you're setting this up from scratch on two servers:
|
||||
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
|
||||
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
|
||||
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
|
||||
| [README-Arrs_Stack.md](Arrs_Stack/README-Arrs_Stack.md) | Union model, orphan cleanup, classification scans, corruption scan, download-orphan cleaner, discovery |
|
||||
| [README-Watchdogs.md](Watchdogs/README-Watchdogs.md) | The four tiers, strike systems, reboot rate limiting, cross-watchdog coordination |
|
||||
| [README-Rsync.md](Rsync/README-Rsync.md) | Profiles, pre-flight checks, merge-run, source path guards, runtime ceiling |
|
||||
| [README-Deployment.md](Deployment/README-Deployment.md) | The conf templates — the only versioned record of what configuration exists |
|
||||
| [README-Kernel.md](Kernel/README-Kernel.md) | The scoring kernel consumers source for discovery decisions |
|
||||
| [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
|
||||
| [README-unraid.md](Plugin/unraid/README-unraid.md) | The Unraid platform layer — adapter contract, three-layer web UI, array lifecycle hooks |
|
||||
| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
|
||||
| [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
|
||||
|
||||
Every folder has a `README-*.md` for *why it exists* and most have a `Manual-*.md` for
|
||||
*how to operate it*. Every script has a header block covering purpose, operational model,
|
||||
design principles, safeguards, configuration and runtime modes — if you want to know what a
|
||||
script does, the top of the file is the authoritative answer, not this page.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THINGS THAT ACTUALLY WENT WRONG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Most of the safeguards in this repo exist because something bit me first. If you're wondering
|
||||
why a script checks something that looks paranoid, it's usually because it didn't, once.
|
||||
|
||||
**The 755 GB nobody was looking at.**
|
||||
Every cleanup script walked the *library* side. Nothing walked the *download* side. Completed
|
||||
downloads the arrs had quietly stopped tracking piled up in SABnzbd's Completed folders —
|
||||
oldest from 2022 — until the cache pool hit 89%. That's `arr_download_orphan_cleaner.sh`.
|
||||
|
||||
**Lidarr said it had 23% of my music.**
|
||||
1,004 of 1,357 artists reporting a fraction of their real track count, with every file sitting
|
||||
right there on disk and no scan running. Organic scans only touch files involved in an import,
|
||||
so an untouched library just drifts. That's `arr_full_rescan.sh`.
|
||||
|
||||
**"Full House" moved. Except it didn't.**
|
||||
Sonarr's API reported `episodeFileCount:192` at the new path while the actual 75 GB was still
|
||||
at the old one — `moveFiles=true` flips the database instantly, but the physical move is a
|
||||
separate async command draining one at a time behind ~20 others. Every classification move now
|
||||
polls its own command to completion before believing the database.
|
||||
|
||||
**mtime lies about when a file arrived.**
|
||||
Measured on one week's imports: 400 of 400 files had mtimes over 7 days old. One was 9,613
|
||||
days old. An import preserves the release's original timestamp, so orphan detection has to
|
||||
gate on ctime — which means the nightly permissions job must only touch files that are
|
||||
actually wrong, because `chown` restamps ctime even when it changes nothing. Make that pass
|
||||
unconditional and orphan collection silently stops. Nothing errors. The pool just fills.
|
||||
|
||||
**A rsync merge pass ran for 9.7 days.**
|
||||
Now every transfer attempt is capped, and resumes rather than restarting — which only works
|
||||
because `--partial` is in the defaults.
|
||||
|
||||
**R.E.M. was already in the library. Apparently.**
|
||||
The duplicate check interpolated the artist name straight into a `grep` pattern, so `R.E.M.`
|
||||
happily matched a stored `RxExMy`. Music is full of regex metacharacters. It's a literal
|
||||
match now.
|
||||
|
||||
**"Craig of the Creek" is not anime.**
|
||||
TheTVDB tags it with the Anime genre anyway, which is why the anime signal needs corroboration
|
||||
from language or network before it counts. In the other direction, "The Pussycat Dolls
|
||||
Present" is a real MTV show with zero votes and no IMDb ID — which is why the junk-detection
|
||||
heuristic that works on movies is deliberately *not* ported to TV.
|
||||
|
||||
**`php -S` silently drops POST bodies** on Unraid's PHP 8.4. The webhook listener is Node.js
|
||||
for exactly that reason, and no other.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Things that are different from what you might expect:
|
||||
Every one of these cost me something to learn. Reading them here is cheaper.
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
@@ -877,15 +1061,18 @@ Things that are different from what you might expect:
|
||||
# --dry-run --log first on a new system. README-Media.md has the full procedure.
|
||||
|
||||
# 9. Scripts never call OS commands directly.
|
||||
# All platform-specific operations go through plugin_*() functions defined in
|
||||
# All platform-specific operations go through platform_*() functions defined in
|
||||
# Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all
|
||||
# isolated in the adapter. This is what makes the codebase portable.
|
||||
# Adding a new OS = writing one adapter file, not patching dozens of scripts.
|
||||
|
||||
# 10. System_Essentials/ is platform-agnostic at the script level.
|
||||
# Scripts like server_reboot.sh, mover_stop.sh, and webgui_watchdog.sh call
|
||||
# adapter functions (platform_is_mover_running, platform_restart_service, etc.)
|
||||
# The adapter handles what those mean on the actual OS. The scripts are clean.
|
||||
# 10. Top-level System_Essentials/ is platform-agnostic. Plugin/unraid/ is not.
|
||||
# server_reboot.sh lives at the top level and calls adapter functions only —
|
||||
# platform_is_mover_running, platform_restart_service. It never names an OS.
|
||||
# Scripts that manipulate an Unraid subsystem directly — mover_stop.sh,
|
||||
# webgui_watchdog.sh, php_fpm_max_children.sh — live under Plugin/unraid/
|
||||
# instead, because there is no generic version of them to write.
|
||||
# Which folder a script is in tells you whether it survives a platform change.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
@@ -895,6 +1082,15 @@ Things that are different from what you might expect:
|
||||
|
||||
```
|
||||
Name: Varaverk
|
||||
Repository: git@192.168.50.2:Varaverk/varaverk.git
|
||||
Repository: git@192.168.50.2:FailedProxy/Varaverk.git
|
||||
Install path: /boot/config/plugins/varaverk
|
||||
```
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
This is a home media stack for two households. It is not an enterprise product, it doesn't
|
||||
want to be one, and it makes trade-offs no enterprise would accept — because the thing it's
|
||||
actually optimising for is that nobody in either house has to know any of this exists.
|
||||
|
||||
The measure of success here isn't uptime percentages. It's that the power goes out 50 miles
|
||||
away, and the show keeps playing.
|
||||
Reference in New Issue
Block a user