Give the README a voice and bring it current
It is the front page and it read like a spec sheet. Adds the incidents the safeguards came from, an honest note about what is actually switched on, and the layers built since.
This commit is contained in:
@@ -23,6 +23,27 @@ correct adapter. The scripts themselves never branch on the OS.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## ━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
This is a live system, not a finished product. HOST1 has been running the whole stack
|
||||||
|
unattended for months. HOST2 is mid-rebuild, so some of what's described below is written
|
||||||
|
and tested but currently switched off:
|
||||||
|
|
||||||
|
```
|
||||||
|
FALLBACK_ENABLED=true ← running
|
||||||
|
PARTNERSHIP_ENABLED=true ← running
|
||||||
|
RSYNC_ENABLED=false ← off until HOST2 is back
|
||||||
|
CONF_SYNC_ENABLED=false ← off until HOST2 is back
|
||||||
|
```
|
||||||
|
|
||||||
|
Everything here is built and audited. The two-server half is waiting on the second server.
|
||||||
|
If a section reads like it's describing something that already happens every night — on
|
||||||
|
HOST1, it is. Across both hosts, it will be.
|
||||||
|
|
||||||
|
I'd rather tell you that than have you find out by reading the toggles.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -76,7 +97,7 @@ break silently.
|
|||||||
Dispatcharr # shared Live TV scheduler
|
Dispatcharr # shared Live TV scheduler
|
||||||
```
|
```
|
||||||
|
|
||||||
When `fallback.sh` on HOST2 reads `FALLBACK_HOST2_COVERS_HOST1_TIER1=(..."Emby"...)`,
|
When `fallback.sh` on HOST2 reads `FALLBACK_HOST1_TIER1=(..."Emby"...)`,
|
||||||
it looks for a container called exactly `Emby` and starts it. HOST2 must have a
|
it looks for a container called exactly `Emby` and starts it. HOST2 must have a
|
||||||
container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
|
container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
|
||||||
its volume mounts pointing at the mirrored share paths. This is what makes one
|
its volume mounts pointing at the mirrored share paths. This is what makes one
|
||||||
@@ -113,7 +134,7 @@ the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts tar
|
|||||||
# The entire ecosystem uses container names as identifiers.
|
# The entire ecosystem uses container names as identifiers.
|
||||||
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
|
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
|
||||||
# → looks for exactly "NginxProxyManager" on the running server
|
# → looks for exactly "NginxProxyManager" on the running server
|
||||||
# fallback.sh: FALLBACK_HOST2_COVERS_HOST1_TIER1=("Emby" "VaultWarden")
|
# fallback.sh: FALLBACK_HOST1_TIER1=("Emby" "VaultWarden")
|
||||||
# → looks for exactly "Emby" and "VaultWarden" on HOST2
|
# → looks for exactly "Emby" and "VaultWarden" on HOST2
|
||||||
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
|
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
|
||||||
# → looks for exactly those names in docker ps
|
# → looks for exactly those names in docker ps
|
||||||
@@ -226,9 +247,9 @@ across separate shares per-user or per-application.
|
|||||||
# arr_sync.sh guarantees both arrs agree on what exists before files move.
|
# arr_sync.sh guarantees both arrs agree on what exists before files move.
|
||||||
# rsync then spreads the files to make storage match what arrs already know.
|
# rsync then spreads the files to make storage match what arrs already know.
|
||||||
#
|
#
|
||||||
# arr_cleanup.sh (daily) removes true orphans — files no arr tracks.
|
# The per-arr cleanups (daily) remove true orphans — files no arr tracks.
|
||||||
# arr_cleanup uses the union model too: a file is only an orphan if
|
# They use the union model too: a file is only an orphan if neither arr
|
||||||
# neither arr on either server has it indexed.
|
# on either server has it indexed.
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -311,14 +332,18 @@ where determines whether you need encryption and whether sync failure matters.
|
|||||||
|
|
||||||
## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
> **Understanding this distinction prevents a lot of confusion.**
|
> **Get this wrong and nothing else in here will make sense.**
|
||||||
|
|
||||||
Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a
|
Every guide you'll find describes the same thing: a primary, a standby, shared storage or
|
||||||
primary server and a standby server with shared storage or synchronous replication.
|
synchronous replication. The standby sits there doing nothing, existing only for the day the
|
||||||
The standby exists only to take over when the primary fails. It runs nothing
|
primary dies. Pacemaker, DRBD, Proxmox HA — all built for a datacenter where downtime is
|
||||||
independently. Its purpose is 100% failover coverage with no service interruption.
|
measured in dollars per second.
|
||||||
|
|
||||||
**This is not that.**
|
**This is not that.** Nobody is on call. There is no shared storage, because there is no
|
||||||
|
datacenter — there are two houses. The standby isn't idle, because it isn't a standby: it's
|
||||||
|
somebody's actual media server, and they're watching it right now.
|
||||||
|
|
||||||
|
The bar is not five nines. The bar is *nobody notices their show stopped working.*
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -656,20 +681,35 @@ Throughout the day:
|
|||||||
rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares)
|
rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares)
|
||||||
media_shares_permissions.sh (fix ownership before arr cleanup)
|
media_shares_permissions.sh (fix ownership before arr cleanup)
|
||||||
media_cleaner.sh anime + media (remove scene junk)
|
media_cleaner.sh anime + media (remove scene junk)
|
||||||
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files)
|
lidarr_release_fixer.sh (fix wrong MusicBrainz editions first)
|
||||||
docker_daily_restart.sh (NPM, Authelia, Dispatcharr fresh start)
|
lidarr_duplicate_artist_cleanup.sh (drop phantom zero-file duplicates)
|
||||||
|
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files — library side)
|
||||||
|
arr_download_orphan_cleaner.sh (remove orphans — download side)
|
||||||
|
radarr/sonarr_classification_scan.sh --move (anime/kids sitting in the wrong root)
|
||||||
|
docker_update.sh → docker_daily_restart.sh (pull first, then restart onto it)
|
||||||
|
|
||||||
2:30am Sunday:
|
2:30am Sunday:
|
||||||
Stop Emby + auth stack both sides
|
Stop Emby + auth stack both sides
|
||||||
Pull updated Docker images (zero extra downtime — already stopped)
|
Pull updated Docker images (zero extra downtime — already stopped)
|
||||||
rsync Emby + Critical-Data clean mirror
|
rsync Emby + Critical-Data clean mirror
|
||||||
Start containers on new images
|
Start containers on new images
|
||||||
docker_weekly_restart.sh (NextCloud, AdGuard, Immich)
|
docker_update.sh --weekly → docker_weekly_restart.sh
|
||||||
|
arr_corruption_scan.sh (ffprobe every tracked video)
|
||||||
|
arr_full_rescan.sh (force a real disk↔database reconcile)
|
||||||
|
|
||||||
7am Sunday:
|
7am Sunday:
|
||||||
sunday_morning_coffee_report.sh (everything that happened this week)
|
sunday_morning_coffee_report.sh (everything that happened this week)
|
||||||
|
|
||||||
|
Monthly (self-gated on 30-day uptime):
|
||||||
|
ZFS scrub, SMART long tests
|
||||||
|
docker_update.sh --remainder (everything not in the managed lists)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The ordering in the 1am block is not arbitrary. Permissions run before the cleanups because
|
||||||
|
the cleanups need correct ownership to delete. The release fixer runs before the Lidarr
|
||||||
|
cleanup so it isn't cleaning up files it's about to fix. Updates run before restarts so the
|
||||||
|
restart lands on the new image. Change the order and things break quietly rather than loudly.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
@@ -828,15 +868,74 @@ If you're setting this up from scratch on two servers:
|
|||||||
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
|
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
|
||||||
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
|
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
|
||||||
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
|
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
|
||||||
|
| [README-Arrs_Stack.md](Arrs_Stack/README-Arrs_Stack.md) | Union model, orphan cleanup, classification scans, corruption scan, download-orphan cleaner, discovery |
|
||||||
|
| [README-Watchdogs.md](Watchdogs/README-Watchdogs.md) | The four tiers, strike systems, reboot rate limiting, cross-watchdog coordination |
|
||||||
|
| [README-Rsync.md](Rsync/README-Rsync.md) | Profiles, pre-flight checks, merge-run, source path guards, runtime ceiling |
|
||||||
|
| [README-Deployment.md](Deployment/README-Deployment.md) | The conf templates — the only versioned record of what configuration exists |
|
||||||
|
| [README-Kernel.md](Kernel/README-Kernel.md) | The scoring kernel consumers source for discovery decisions |
|
||||||
| [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
|
| [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
|
||||||
|
| [README-unraid.md](Plugin/unraid/README-unraid.md) | The Unraid platform layer — adapter contract, three-layer web UI, array lifecycle hooks |
|
||||||
| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
|
| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
|
||||||
| [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
|
| [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
|
||||||
|
|
||||||
|
Every folder has a `README-*.md` for *why it exists* and most have a `Manual-*.md` for
|
||||||
|
*how to operate it*. Every script has a header block covering purpose, operational model,
|
||||||
|
design principles, safeguards, configuration and runtime modes — if you want to know what a
|
||||||
|
script does, the top of the file is the authoritative answer, not this page.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THINGS THAT ACTUALLY WENT WRONG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
Most of the safeguards in this repo exist because something bit me first. If you're wondering
|
||||||
|
why a script checks something that looks paranoid, it's usually because it didn't, once.
|
||||||
|
|
||||||
|
**The 755 GB nobody was looking at.**
|
||||||
|
Every cleanup script walked the *library* side. Nothing walked the *download* side. Completed
|
||||||
|
downloads the arrs had quietly stopped tracking piled up in SABnzbd's Completed folders —
|
||||||
|
oldest from 2022 — until the cache pool hit 89%. That's `arr_download_orphan_cleaner.sh`.
|
||||||
|
|
||||||
|
**Lidarr said it had 23% of my music.**
|
||||||
|
1,004 of 1,357 artists reporting a fraction of their real track count, with every file sitting
|
||||||
|
right there on disk and no scan running. Organic scans only touch files involved in an import,
|
||||||
|
so an untouched library just drifts. That's `arr_full_rescan.sh`.
|
||||||
|
|
||||||
|
**"Full House" moved. Except it didn't.**
|
||||||
|
Sonarr's API reported `episodeFileCount:192` at the new path while the actual 75 GB was still
|
||||||
|
at the old one — `moveFiles=true` flips the database instantly, but the physical move is a
|
||||||
|
separate async command draining one at a time behind ~20 others. Every classification move now
|
||||||
|
polls its own command to completion before believing the database.
|
||||||
|
|
||||||
|
**mtime lies about when a file arrived.**
|
||||||
|
Measured on one week's imports: 400 of 400 files had mtimes over 7 days old. One was 9,613
|
||||||
|
days old. An import preserves the release's original timestamp, so orphan detection has to
|
||||||
|
gate on ctime — which means the nightly permissions job must only touch files that are
|
||||||
|
actually wrong, because `chown` restamps ctime even when it changes nothing. Make that pass
|
||||||
|
unconditional and orphan collection silently stops. Nothing errors. The pool just fills.
|
||||||
|
|
||||||
|
**A rsync merge pass ran for 9.7 days.**
|
||||||
|
Now every transfer attempt is capped, and resumes rather than restarting — which only works
|
||||||
|
because `--partial` is in the defaults.
|
||||||
|
|
||||||
|
**R.E.M. was already in the library. Apparently.**
|
||||||
|
The duplicate check interpolated the artist name straight into a `grep` pattern, so `R.E.M.`
|
||||||
|
happily matched a stored `RxExMy`. Music is full of regex metacharacters. It's a literal
|
||||||
|
match now.
|
||||||
|
|
||||||
|
**"Craig of the Creek" is not anime.**
|
||||||
|
TheTVDB tags it with the Anime genre anyway, which is why the anime signal needs corroboration
|
||||||
|
from language or network before it counts. In the other direction, "The Pussycat Dolls
|
||||||
|
Present" is a real MTV show with zero votes and no IMDb ID — which is why the junk-detection
|
||||||
|
heuristic that works on movies is deliberately *not* ported to TV.
|
||||||
|
|
||||||
|
**`php -S` silently drops POST bodies** on Unraid's PHP 8.4. The webhook listener is Node.js
|
||||||
|
for exactly that reason, and no other.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
Things that are different from what you might expect:
|
Every one of these cost me something to learn. Reading them here is cheaper.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
@@ -895,6 +994,6 @@ Things that are different from what you might expect:
|
|||||||
|
|
||||||
```
|
```
|
||||||
Name: Varaverk
|
Name: Varaverk
|
||||||
Repository: git@192.168.50.2:Varaverk/varaverk.git
|
Repository: git@192.168.50.2:FailedProxy/Varaverk.git
|
||||||
Install path: /boot/config/plugins/varaverk
|
Install path: /boot/config/plugins/varaverk
|
||||||
```
|
```
|
||||||
Reference in New Issue
Block a user