Compare commits
2
Commits
1ed6b92118
...
c54039392d
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c54039392d | ||
|
|
671bf2352b |
@@ -3,11 +3,23 @@
|
|||||||
Getting a fresh two-server ecosystem running from scratch.
|
Getting a fresh two-server ecosystem running from scratch.
|
||||||
For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.
|
For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.
|
||||||
|
|
||||||
|
**Ten steps, and they're in this order for a reason.** Each one assumes the previous one
|
||||||
|
actually worked — not that you ran it, that it *worked*. Every step below ends with a way to
|
||||||
|
check, and skipping those checks is how you end up three steps later debugging the wrong
|
||||||
|
thing entirely.
|
||||||
|
|
||||||
|
The steps that hurt most when rushed are **3** (naming) and **8** (testing failover). Step 3
|
||||||
|
because renaming anything afterwards means chasing it through every conf, and step 8 because
|
||||||
|
untested failover isn't redundancy — it's a guess you haven't checked yet.
|
||||||
|
|
||||||
|
Budget an evening. It is not hard, but it is not five minutes either.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ BEFORE YOU START ━━━
|
## ━━━ BEFORE YOU START ━━━
|
||||||
|
|
||||||
Things that must be in place before you touch any scripts.
|
None of this is optional and none of it is Varaverk's job to install. Get these in place
|
||||||
|
first — every step after here assumes they're already true.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -157,6 +169,21 @@ Replace these with your actual unRAID server hostnames. These must match exactly
|
|||||||
`detect_hosts()` compares the running server's hostname against these two values to
|
`detect_hosts()` compares the running server's hostname against these two values to
|
||||||
know which server it is on. Everything else in the ecosystem flows from this.
|
know which server it is on. Everything else in the ecosystem flows from this.
|
||||||
|
|
||||||
|
**Get this wrong and nothing works, but nothing errors either.** If neither value matches,
|
||||||
|
`MY_ID` is never set, and every script that depends on it either exits early or resolves
|
||||||
|
`${MY_ID}_SOMETHING` to an empty variable and quietly takes the wrong branch. Copy the value
|
||||||
|
straight out of `hostname` on each box rather than typing what you think it is:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
hostname # run this on each server, paste the exact output
|
||||||
|
```
|
||||||
|
|
||||||
|
> **The 15-character trap.** unRAID truncates the Server Name to 15 characters for NetBIOS.
|
||||||
|
> If your name is longer, what you set in the WebGUI and what `hostname` returns are two
|
||||||
|
> different strings. `detect_hosts()` has a fallback that matches a truncated 15-char
|
||||||
|
> hostname against a longer configured value — but only when the live hostname is
|
||||||
|
> *exactly* 15 characters. Don't rely on it. Use the real `hostname` output.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────
|
### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────
|
||||||
@@ -206,15 +233,27 @@ HOST1_CRITICAL_SYNC_SHARES=(
|
|||||||
"/mnt/user/appdata-Fallback/Critical-Data|critical-data"
|
"/mnt/user/appdata-Fallback/Critical-Data|critical-data"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate)
|
# Containers to start when HOST1 goes down (Tier 1 = immediate)
|
||||||
HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=() # HOST2's own covers for HOST1, and vice versa
|
# Lives in host1.conf. Named for the host being COVERED, not the host doing the covering.
|
||||||
HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
FALLBACK_HOST1_TIER1=(
|
||||||
"Emby"
|
"Emby"
|
||||||
"VaultWarden"
|
"VaultWarden"
|
||||||
"NginxProxyManager"
|
"NginxProxyManager"
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Read that naming carefully — it catches people.** The variable is
|
||||||
|
`FALLBACK_${REMOTE_ID}_TIER${N}`, and `REMOTE_ID` is *the host that went down*. So
|
||||||
|
`FALLBACK_HOST1_TIER1` is the list HOST2 reads when HOST1 is unreachable. It defines
|
||||||
|
"what HOST1 needs covering", not "what HOST2 runs".
|
||||||
|
|
||||||
|
It lives in `host1.conf` for the same reason — HOST1 owns the description of its own
|
||||||
|
service stack. HOST2 receives it through the conf cache rather than keeping its own
|
||||||
|
opinion about what HOST1 runs. One list, one owner, no drift.
|
||||||
|
|
||||||
|
Tiers 2–4 activate progressively after `HOST1_TIER2_DELAY` etc., so a brief outage never
|
||||||
|
drags the entire remote stack across.
|
||||||
|
|
||||||
For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
|
For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
|
||||||
and `Manual-Watchdogs.md`.
|
and `Manual-Watchdogs.md`.
|
||||||
|
|
||||||
@@ -455,6 +494,18 @@ bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log
|
|||||||
The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
|
The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
|
||||||
detect it and start Tier 1 containers, then unblocks and waits for handback.
|
detect it and start Tier 1 containers, then unblocks and waits for handback.
|
||||||
|
|
||||||
|
**Do not skip this step.** Everything up to here you can verify by reading output. Failover
|
||||||
|
is the one part you cannot confirm by looking at it — a typo in a tier list, a container name
|
||||||
|
that doesn't exist on the other side, a DDNS container that was renamed six months ago: all
|
||||||
|
of it sits there looking completely fine until the moment it's needed.
|
||||||
|
|
||||||
|
The test costs you twenty minutes and no downtime. The alternative is finding out at 2am,
|
||||||
|
during the outage, when you have exactly one chance to get it right.
|
||||||
|
|
||||||
|
**Watch the handback as closely as the failover.** Coming back is the harder half — it has to
|
||||||
|
stop the remote's DDNS, sync the data back, start the local containers, and only then bring
|
||||||
|
local DDNS up. Failover starting correctly tells you nothing about whether handback does.
|
||||||
|
|
||||||
See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
|
See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
|
||||||
how to interpret the output.
|
how to interpret the output.
|
||||||
|
|
||||||
@@ -651,6 +702,14 @@ Each subsystem has a README with the design decisions and a Manual with the conf
|
|||||||
reference and troubleshooting. The `--status` flag on any script shows the current
|
reference and troubleshooting. The `--status` flag on any script shows the current
|
||||||
configuration and state.
|
configuration and state.
|
||||||
|
|
||||||
|
**Turn things on one at a time and give each one a few days.** Everything below is off by
|
||||||
|
default on a fresh install, and that's deliberate — a stack where six new subsystems went
|
||||||
|
live the same night is a stack where you have no idea which one to blame. Enable, watch it
|
||||||
|
through a full daily cycle, then enable the next.
|
||||||
|
|
||||||
|
Anything that deletes files — the arr cleanup scripts especially — gets a `--dry-run --log`
|
||||||
|
first. Read the list. Every time, not just the first time.
|
||||||
|
|
||||||
```
|
```
|
||||||
Watchdogs/ → README-Watchdogs.md configure memory limits, container lists
|
Watchdogs/ → README-Watchdogs.md configure memory limits, container lists
|
||||||
Media/ → README-Media.md enable arr cleanup, discovery scripts
|
Media/ → README-Media.md enable arr cleanup, discovery scripts
|
||||||
|
|||||||
@@ -23,6 +23,72 @@ correct adapter. The scripts themselves never branch on the OS.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## ━━━ HOW THIS GOT HERE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
**One server. One script. Three lines.**
|
||||||
|
|
||||||
|
That's genuinely where this started. Something needed doing on a schedule, so I wrote three
|
||||||
|
lines and moved on.
|
||||||
|
|
||||||
|
Then it was two servers, 50 miles apart, and about **60 scripts** with nothing holding them
|
||||||
|
together. Every one standalone. Every one with its own hardcoded paths, its own container
|
||||||
|
names, its own idea of how to log something or send a notification.
|
||||||
|
|
||||||
|
**And it worked.** That's the part worth saying. The stack stayed up, the media kept playing,
|
||||||
|
the backups kept running. It was not broken.
|
||||||
|
|
||||||
|
It was just *miserable to change.*
|
||||||
|
|
||||||
|
Every adjustment meant finding every script that touched the thing, on both machines, and
|
||||||
|
editing each one the same way — then hoping I'd found them all. Miss one and it didn't error.
|
||||||
|
It just quietly kept doing the old thing until I noticed weeks later. Two servers meant two
|
||||||
|
copies of every mistake, and the two copies drifted apart a little more every time I touched
|
||||||
|
them.
|
||||||
|
|
||||||
|
So it got rebuilt, in roughly this order:
|
||||||
|
|
||||||
|
```
|
||||||
|
git stop editing live scripts on two machines and calling it version control
|
||||||
|
common.sh one place for log, notify, locking, host detection, retries
|
||||||
|
master.conf thresholds and toggles — shared, one copy
|
||||||
|
host*.conf credentials, paths, container names — per server
|
||||||
|
load_config.sh sources all of it, in the right order, at the top of every script
|
||||||
|
```
|
||||||
|
|
||||||
|
Then the long part: refactoring 60 standalone scripts into that shape, one at a time, until
|
||||||
|
every value lived in a conf file and every shared behaviour lived in `common.sh`. Change the
|
||||||
|
retry count once, every script gets it. Rename a container in one place, everything follows.
|
||||||
|
|
||||||
|
That is the whole idea, and it's why the structure looks the way it does. The three-file
|
||||||
|
config model and the shared library aren't design patterns picked out of a book — they're
|
||||||
|
the specific answer to "I have 60 scripts on two machines and I'm scared to touch any of
|
||||||
|
them."
|
||||||
|
|
||||||
|
That became Varaverk.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
This is a live system, not a finished product. HOST1 has been running the whole stack
|
||||||
|
unattended for months. HOST2 is mid-rebuild, so some of what's described below is written
|
||||||
|
and tested but currently switched off:
|
||||||
|
|
||||||
|
```
|
||||||
|
FALLBACK_ENABLED=true ← running
|
||||||
|
PARTNERSHIP_ENABLED=true ← running
|
||||||
|
RSYNC_ENABLED=false ← off until HOST2 is back
|
||||||
|
CONF_SYNC_ENABLED=false ← off until HOST2 is back
|
||||||
|
```
|
||||||
|
|
||||||
|
Everything here is built and audited. The two-server half is waiting on the second server.
|
||||||
|
If a section reads like it's describing something that already happens every night — on
|
||||||
|
HOST1, it is. Across both hosts, it will be.
|
||||||
|
|
||||||
|
I'd rather tell you that than have you find out by reading the toggles.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -36,7 +102,7 @@ HOST1 — unRAID-Gmer4Lfe
|
|||||||
|
|
||||||
HOST2 — unRAID-Jayred365
|
HOST2 — unRAID-Jayred365
|
||||||
Hardware: Intel i5 10th gen, 64GB RAM
|
Hardware: Intel i5 10th gen, 64GB RAM
|
||||||
Location: Remote — different building, different power utility
|
Location: Remote — 50 miles away, different power utility and ISP
|
||||||
Domain: Gmer4Lfe.us
|
Domain: Gmer4Lfe.us
|
||||||
Arrs: Anime_Shows, Anime_Movies — union with HOST1 via arr_sync
|
Arrs: Anime_Shows, Anime_Movies — union with HOST1 via arr_sync
|
||||||
His own Emby — for his household
|
His own Emby — for his household
|
||||||
@@ -76,7 +142,7 @@ break silently.
|
|||||||
Dispatcharr # shared Live TV scheduler
|
Dispatcharr # shared Live TV scheduler
|
||||||
```
|
```
|
||||||
|
|
||||||
When `fallback.sh` on HOST2 reads `FALLBACK_HOST2_COVERS_HOST1_TIER1=(..."Emby"...)`,
|
When `fallback.sh` on HOST2 reads `FALLBACK_HOST1_TIER1=(..."Emby"...)`,
|
||||||
it looks for a container called exactly `Emby` and starts it. HOST2 must have a
|
it looks for a container called exactly `Emby` and starts it. HOST2 must have a
|
||||||
container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
|
container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
|
||||||
its volume mounts pointing at the mirrored share paths. This is what makes one
|
its volume mounts pointing at the mirrored share paths. This is what makes one
|
||||||
@@ -113,7 +179,7 @@ the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts tar
|
|||||||
# The entire ecosystem uses container names as identifiers.
|
# The entire ecosystem uses container names as identifiers.
|
||||||
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
|
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
|
||||||
# → looks for exactly "NginxProxyManager" on the running server
|
# → looks for exactly "NginxProxyManager" on the running server
|
||||||
# fallback.sh: FALLBACK_HOST2_COVERS_HOST1_TIER1=("Emby" "VaultWarden")
|
# fallback.sh: FALLBACK_HOST1_TIER1=("Emby" "VaultWarden")
|
||||||
# → looks for exactly "Emby" and "VaultWarden" on HOST2
|
# → looks for exactly "Emby" and "VaultWarden" on HOST2
|
||||||
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
|
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
|
||||||
# → looks for exactly those names in docker ps
|
# → looks for exactly those names in docker ps
|
||||||
@@ -152,6 +218,10 @@ as sync targets and mount points — a mismatch is a broken path.
|
|||||||
|
|
||||||
## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
Two households, one library, and a hard line between "our media" and "your stuff." Everything
|
||||||
|
in this section comes down to that split — media is pooled and shared freely, personal data
|
||||||
|
never is, and no script gets to blur the two.
|
||||||
|
|
||||||
### ── Common Libraries — Shared Pool ─────────────────────────────────────────
|
### ── Common Libraries — Shared Pool ─────────────────────────────────────────
|
||||||
|
|
||||||
Common media like Movies, Music, and TV Shows are a **shared pool**. Every container
|
Common media like Movies, Music, and TV Shows are a **shared pool**. Every container
|
||||||
@@ -226,9 +296,9 @@ across separate shares per-user or per-application.
|
|||||||
# arr_sync.sh guarantees both arrs agree on what exists before files move.
|
# arr_sync.sh guarantees both arrs agree on what exists before files move.
|
||||||
# rsync then spreads the files to make storage match what arrs already know.
|
# rsync then spreads the files to make storage match what arrs already know.
|
||||||
#
|
#
|
||||||
# arr_cleanup.sh (daily) removes true orphans — files no arr tracks.
|
# The per-arr cleanups (daily) remove true orphans — files no arr tracks.
|
||||||
# arr_cleanup uses the union model too: a file is only an orphan if
|
# They use the union model too: a file is only an orphan if neither arr
|
||||||
# neither arr on either server has it indexed.
|
# on either server has it indexed.
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -236,8 +306,13 @@ across separate shares per-user or per-application.
|
|||||||
|
|
||||||
## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
Not everything needs to sync. Not everything should sync. Understanding what goes
|
Not everything needs to sync. Not everything *should*. Two households means some of this
|
||||||
where determines whether you need encryption and whether sync failure matters.
|
data is mine, some of it is his, and some of it belongs to both of us — and treating all
|
||||||
|
three the same way is how you end up either leaking someone's photos or losing the thing you
|
||||||
|
actually needed a copy of.
|
||||||
|
|
||||||
|
Where a piece of data lands in this table decides two things: whether it needs encryption,
|
||||||
|
and whether a failed sync is something to fix tonight or something to shrug at.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -311,14 +386,18 @@ where determines whether you need encryption and whether sync failure matters.
|
|||||||
|
|
||||||
## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
> **Understanding this distinction prevents a lot of confusion.**
|
> **Get this wrong and nothing else in here will make sense.**
|
||||||
|
|
||||||
Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a
|
Every guide you'll find describes the same thing: a primary, a standby, shared storage or
|
||||||
primary server and a standby server with shared storage or synchronous replication.
|
synchronous replication. The standby sits there doing nothing, existing only for the day the
|
||||||
The standby exists only to take over when the primary fails. It runs nothing
|
primary dies. Pacemaker, DRBD, Proxmox HA — all built for a datacenter where downtime is
|
||||||
independently. Its purpose is 100% failover coverage with no service interruption.
|
measured in dollars per second.
|
||||||
|
|
||||||
**This is not that.**
|
**This is not that.** Nobody is on call. There is no shared storage, because there is no
|
||||||
|
datacenter — there are two houses. The standby isn't idle, because it isn't a standby: it's
|
||||||
|
somebody's actual media server, and they're watching it right now.
|
||||||
|
|
||||||
|
The bar is not five nines. The bar is *nobody notices their show stopped working.*
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -479,6 +558,15 @@ Monitoring:
|
|||||||
|
|
||||||
## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
Folders are grouped by **what a script does to the system**, not by which service it happens
|
||||||
|
to talk to. There's no `Sonarr/` folder — cleaning up after Sonarr is media work, watching
|
||||||
|
whether its container is alive is watchdog work, and those two things belong next to their
|
||||||
|
own kind.
|
||||||
|
|
||||||
|
Every folder has its own README explaining the group, and every script has a header
|
||||||
|
explaining itself. If you're looking for one specific thing, the README reference near the
|
||||||
|
bottom of this page is the faster route.
|
||||||
|
|
||||||
```
|
```
|
||||||
varaverk/
|
varaverk/
|
||||||
│
|
│
|
||||||
@@ -538,6 +626,16 @@ varaverk/
|
|||||||
|
|
||||||
## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━
|
## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
This is the part that fixed the 60-script problem. Three files, one rule:
|
||||||
|
|
||||||
|
**Thresholds and toggles go in `master.conf`. Credentials, paths and container names go in
|
||||||
|
`host*.conf`.** If you find yourself about to hardcode a value in a script, it belongs in one
|
||||||
|
of these instead — that's the whole discipline, and everything else follows from it.
|
||||||
|
|
||||||
|
Sparse checkout means each server only ever receives its own `host*.conf`. HOST1 never has
|
||||||
|
HOST2's credentials sitting on disk, and vice versa. The partner's conf arrives at runtime
|
||||||
|
through the RAM cache, which is why `conf_sync.sh` exists.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
# All three conf files live in Configurations/. load_config.sh sources them.
|
# All three conf files live in Configurations/. load_config.sh sources them.
|
||||||
@@ -617,7 +715,8 @@ Every 4 hours:
|
|||||||
|
|
||||||
2:30am Sunday:
|
2:30am Sunday:
|
||||||
weekly_sync_maintenance.sh stop → update images → clean sync → start → weekly restarts
|
weekly_sync_maintenance.sh stop → update images → clean sync → start → weekly restarts
|
||||||
+ playback_aware_lidarr/radarr/sonarr_discovery.sh (behavior-driven arr discovery)
|
+ playback_aware_{lidarr,radarr,sonarr}_discovery.sh
|
||||||
|
(behaviour-driven discovery from real playback history)
|
||||||
|
|
||||||
8am daily:
|
8am daily:
|
||||||
weekly_health_digest.sh DIGEST_PROFILE controls when it actually notifies
|
weekly_health_digest.sh DIGEST_PROFILE controls when it actually notifies
|
||||||
@@ -656,26 +755,42 @@ Throughout the day:
|
|||||||
rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares)
|
rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares)
|
||||||
media_shares_permissions.sh (fix ownership before arr cleanup)
|
media_shares_permissions.sh (fix ownership before arr cleanup)
|
||||||
media_cleaner.sh anime + media (remove scene junk)
|
media_cleaner.sh anime + media (remove scene junk)
|
||||||
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files)
|
lidarr_release_fixer.sh (fix wrong MusicBrainz editions first)
|
||||||
docker_daily_restart.sh (NPM, Authelia, Dispatcharr fresh start)
|
lidarr_duplicate_artist_cleanup.sh (drop phantom zero-file duplicates)
|
||||||
|
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files — library side)
|
||||||
|
arr_download_orphan_cleaner.sh (remove orphans — download side)
|
||||||
|
radarr/sonarr_classification_scan.sh --move (anime/kids sitting in the wrong root)
|
||||||
|
docker_update.sh → docker_daily_restart.sh (pull first, then restart onto it)
|
||||||
|
|
||||||
2:30am Sunday:
|
2:30am Sunday:
|
||||||
Stop Emby + auth stack both sides
|
Stop Emby + auth stack both sides
|
||||||
Pull updated Docker images (zero extra downtime — already stopped)
|
Pull updated Docker images (zero extra downtime — already stopped)
|
||||||
rsync Emby + Critical-Data clean mirror
|
rsync Emby + Critical-Data clean mirror
|
||||||
Start containers on new images
|
Start containers on new images
|
||||||
docker_weekly_restart.sh (NextCloud, AdGuard, Immich)
|
docker_update.sh --weekly → docker_weekly_restart.sh
|
||||||
|
arr_corruption_scan.sh (ffprobe every tracked video)
|
||||||
|
arr_full_rescan.sh (force a real disk↔database reconcile)
|
||||||
|
|
||||||
7am Sunday:
|
7am Sunday:
|
||||||
sunday_morning_coffee_report.sh (everything that happened this week)
|
sunday_morning_coffee_report.sh (everything that happened this week)
|
||||||
|
|
||||||
|
Monthly (self-gated on 30-day uptime):
|
||||||
|
ZFS scrub, SMART long tests
|
||||||
|
docker_update.sh --remainder (everything not in the managed lists)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The ordering in the 1am block is not arbitrary. Permissions run before the cleanups because
|
||||||
|
the cleanups need correct ownership to delete. The release fixer runs before the Lidarr
|
||||||
|
cleanup so it isn't cleaning up files it's about to fix. Updates run before restarts so the
|
||||||
|
restart lands on the new image. Change the order and things break quietly rather than loudly.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
Problems are handled at the most appropriate level. Scripts escalate only when the
|
Nothing here jumps straight to the big hammer. Each layer only gets involved because the one
|
||||||
level below them has failed:
|
below it already tried and failed — and the layer holding the biggest hammer, the one that
|
||||||
|
can reboot the box, is the most reluctant of all:
|
||||||
|
|
||||||
```
|
```
|
||||||
Problem: Emby container using 22GB RAM (limit: 18GB)
|
Problem: Emby container using 22GB RAM (limit: 18GB)
|
||||||
@@ -793,7 +908,12 @@ CPU-based) instead of PaddleOCR. Same plugin, no VRAM.
|
|||||||
|
|
||||||
## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
If you're setting this up from scratch on two servers:
|
If you're setting this up from scratch on two servers, the order below is not a suggestion.
|
||||||
|
Steps 2 and 3 are cheap now and expensive later — renaming a container or a share after the
|
||||||
|
confs, the fallback tiers and the sync profiles all reference it means touching every one of
|
||||||
|
them, and missing one doesn't error.
|
||||||
|
|
||||||
|
Do the naming first. Everything after it assumes the names are already right.
|
||||||
|
|
||||||
```
|
```
|
||||||
1. Follow Manual.md — end-to-end setup guide
|
1. Follow Manual.md — end-to-end setup guide
|
||||||
@@ -814,6 +934,11 @@ If you're setting this up from scratch on two servers:
|
|||||||
bash fallback_test.sh when ready — uses iptables simulation, no real downtime
|
bash fallback_test.sh when ready — uses iptables simulation, no real downtime
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Do step 5.** Untested failover isn't redundancy, it's a belief. `fallback_test.sh` simulates
|
||||||
|
the outage with iptables rather than actually pulling the other server down, so there's no
|
||||||
|
excuse not to run it — and finding out your tier lists have a typo is a lot nicer on a Tuesday
|
||||||
|
afternoon than at 2am when the power's actually out.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
@@ -828,15 +953,74 @@ If you're setting this up from scratch on two servers:
|
|||||||
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
|
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
|
||||||
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
|
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
|
||||||
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
|
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
|
||||||
|
| [README-Arrs_Stack.md](Arrs_Stack/README-Arrs_Stack.md) | Union model, orphan cleanup, classification scans, corruption scan, download-orphan cleaner, discovery |
|
||||||
|
| [README-Watchdogs.md](Watchdogs/README-Watchdogs.md) | The four tiers, strike systems, reboot rate limiting, cross-watchdog coordination |
|
||||||
|
| [README-Rsync.md](Rsync/README-Rsync.md) | Profiles, pre-flight checks, merge-run, source path guards, runtime ceiling |
|
||||||
|
| [README-Deployment.md](Deployment/README-Deployment.md) | The conf templates — the only versioned record of what configuration exists |
|
||||||
|
| [README-Kernel.md](Kernel/README-Kernel.md) | The scoring kernel consumers source for discovery decisions |
|
||||||
| [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
|
| [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
|
||||||
|
| [README-unraid.md](Plugin/unraid/README-unraid.md) | The Unraid platform layer — adapter contract, three-layer web UI, array lifecycle hooks |
|
||||||
| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
|
| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
|
||||||
| [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
|
| [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
|
||||||
|
|
||||||
|
Every folder has a `README-*.md` for *why it exists* and most have a `Manual-*.md` for
|
||||||
|
*how to operate it*. Every script has a header block covering purpose, operational model,
|
||||||
|
design principles, safeguards, configuration and runtime modes — if you want to know what a
|
||||||
|
script does, the top of the file is the authoritative answer, not this page.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THINGS THAT ACTUALLY WENT WRONG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
Most of the safeguards in this repo exist because something bit me first. If you're wondering
|
||||||
|
why a script checks something that looks paranoid, it's usually because it didn't, once.
|
||||||
|
|
||||||
|
**The 755 GB nobody was looking at.**
|
||||||
|
Every cleanup script walked the *library* side. Nothing walked the *download* side. Completed
|
||||||
|
downloads the arrs had quietly stopped tracking piled up in SABnzbd's Completed folders —
|
||||||
|
oldest from 2022 — until the cache pool hit 89%. That's `arr_download_orphan_cleaner.sh`.
|
||||||
|
|
||||||
|
**Lidarr said it had 23% of my music.**
|
||||||
|
1,004 of 1,357 artists reporting a fraction of their real track count, with every file sitting
|
||||||
|
right there on disk and no scan running. Organic scans only touch files involved in an import,
|
||||||
|
so an untouched library just drifts. That's `arr_full_rescan.sh`.
|
||||||
|
|
||||||
|
**"Full House" moved. Except it didn't.**
|
||||||
|
Sonarr's API reported `episodeFileCount:192` at the new path while the actual 75 GB was still
|
||||||
|
at the old one — `moveFiles=true` flips the database instantly, but the physical move is a
|
||||||
|
separate async command draining one at a time behind ~20 others. Every classification move now
|
||||||
|
polls its own command to completion before believing the database.
|
||||||
|
|
||||||
|
**mtime lies about when a file arrived.**
|
||||||
|
Measured on one week's imports: 400 of 400 files had mtimes over 7 days old. One was 9,613
|
||||||
|
days old. An import preserves the release's original timestamp, so orphan detection has to
|
||||||
|
gate on ctime — which means the nightly permissions job must only touch files that are
|
||||||
|
actually wrong, because `chown` restamps ctime even when it changes nothing. Make that pass
|
||||||
|
unconditional and orphan collection silently stops. Nothing errors. The pool just fills.
|
||||||
|
|
||||||
|
**A rsync merge pass ran for 9.7 days.**
|
||||||
|
Now every transfer attempt is capped, and resumes rather than restarting — which only works
|
||||||
|
because `--partial` is in the defaults.
|
||||||
|
|
||||||
|
**R.E.M. was already in the library. Apparently.**
|
||||||
|
The duplicate check interpolated the artist name straight into a `grep` pattern, so `R.E.M.`
|
||||||
|
happily matched a stored `RxExMy`. Music is full of regex metacharacters. It's a literal
|
||||||
|
match now.
|
||||||
|
|
||||||
|
**"Craig of the Creek" is not anime.**
|
||||||
|
TheTVDB tags it with the Anime genre anyway, which is why the anime signal needs corroboration
|
||||||
|
from language or network before it counts. In the other direction, "The Pussycat Dolls
|
||||||
|
Present" is a real MTV show with zero votes and no IMDb ID — which is why the junk-detection
|
||||||
|
heuristic that works on movies is deliberately *not* ported to TV.
|
||||||
|
|
||||||
|
**`php -S` silently drops POST bodies** on Unraid's PHP 8.4. The webhook listener is Node.js
|
||||||
|
for exactly that reason, and no other.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
Things that are different from what you might expect:
|
Every one of these cost me something to learn. Reading them here is cheaper.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
@@ -877,15 +1061,18 @@ Things that are different from what you might expect:
|
|||||||
# --dry-run --log first on a new system. README-Media.md has the full procedure.
|
# --dry-run --log first on a new system. README-Media.md has the full procedure.
|
||||||
|
|
||||||
# 9. Scripts never call OS commands directly.
|
# 9. Scripts never call OS commands directly.
|
||||||
# All platform-specific operations go through plugin_*() functions defined in
|
# All platform-specific operations go through platform_*() functions defined in
|
||||||
# Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all
|
# Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all
|
||||||
# isolated in the adapter. This is what makes the codebase portable.
|
# isolated in the adapter. This is what makes the codebase portable.
|
||||||
# Adding a new OS = writing one adapter file, not patching dozens of scripts.
|
# Adding a new OS = writing one adapter file, not patching dozens of scripts.
|
||||||
|
|
||||||
# 10. System_Essentials/ is platform-agnostic at the script level.
|
# 10. Top-level System_Essentials/ is platform-agnostic. Plugin/unraid/ is not.
|
||||||
# Scripts like server_reboot.sh, mover_stop.sh, and webgui_watchdog.sh call
|
# server_reboot.sh lives at the top level and calls adapter functions only —
|
||||||
# adapter functions (platform_is_mover_running, platform_restart_service, etc.)
|
# platform_is_mover_running, platform_restart_service. It never names an OS.
|
||||||
# The adapter handles what those mean on the actual OS. The scripts are clean.
|
# Scripts that manipulate an Unraid subsystem directly — mover_stop.sh,
|
||||||
|
# webgui_watchdog.sh, php_fpm_max_children.sh — live under Plugin/unraid/
|
||||||
|
# instead, because there is no generic version of them to write.
|
||||||
|
# Which folder a script is in tells you whether it survives a platform change.
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -895,6 +1082,15 @@ Things that are different from what you might expect:
|
|||||||
|
|
||||||
```
|
```
|
||||||
Name: Varaverk
|
Name: Varaverk
|
||||||
Repository: git@192.168.50.2:Varaverk/varaverk.git
|
Repository: git@192.168.50.2:FailedProxy/Varaverk.git
|
||||||
Install path: /boot/config/plugins/varaverk
|
Install path: /boot/config/plugins/varaverk
|
||||||
```
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
This is a home media stack for two households. It is not an enterprise product, it doesn't
|
||||||
|
want to be one, and it makes trade-offs no enterprise would accept — because the thing it's
|
||||||
|
actually optimising for is that nobody in either house has to know any of this exists.
|
||||||
|
|
||||||
|
The measure of success here isn't uptime percentages. It's that the power goes out 50 miles
|
||||||
|
away, and the show keeps playing.
|
||||||
Reference in New Issue
Block a user