From 671bf2352b077352017820733ecdeb90147183e2 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 1 Aug 2026 23:34:34 -0400 Subject: [PATCH] Give the README a voice and bring it current It is the front page and it read like a spec sheet. Adds the incidents the safeguards came from, an honest note about what is actually switched on, and the layers built since. --- README.md | 131 +++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 115 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index fa2230d..f923241 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,27 @@ correct adapter. The scripts themselves never branch on the OS. --- +## ━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + +This is a live system, not a finished product. HOST1 has been running the whole stack +unattended for months. HOST2 is mid-rebuild, so some of what's described below is written +and tested but currently switched off: + +``` +FALLBACK_ENABLED=true ← running +PARTNERSHIP_ENABLED=true ← running +RSYNC_ENABLED=false ← off until HOST2 is back +CONF_SYNC_ENABLED=false ← off until HOST2 is back +``` + +Everything here is built and audited. The two-server half is waiting on the second server. +If a section reads like it's describing something that already happens every night — on +HOST1, it is. Across both hosts, it will be. + +I'd rather tell you that than have you find out by reading the toggles. + +--- + ## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ``` @@ -76,7 +97,7 @@ break silently. Dispatcharr # shared Live TV scheduler ``` -When `fallback.sh` on HOST2 reads `FALLBACK_HOST2_COVERS_HOST1_TIER1=(..."Emby"...)`, +When `fallback.sh` on HOST2 reads `FALLBACK_HOST1_TIER1=(..."Emby"...)`, it looks for a container called exactly `Emby` and starts it. HOST2 must have a container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with its volume mounts pointing at the mirrored share paths. This is what makes one @@ -113,7 +134,7 @@ the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts tar # The entire ecosystem uses container names as identifiers. # docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia") # → looks for exactly "NginxProxyManager" on the running server -# fallback.sh: FALLBACK_HOST2_COVERS_HOST1_TIER1=("Emby" "VaultWarden") +# fallback.sh: FALLBACK_HOST1_TIER1=("Emby" "VaultWarden") # → looks for exactly "Emby" and "VaultWarden" on HOST2 # docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia") # → looks for exactly those names in docker ps @@ -226,9 +247,9 @@ across separate shares per-user or per-application. # arr_sync.sh guarantees both arrs agree on what exists before files move. # rsync then spreads the files to make storage match what arrs already know. # -# arr_cleanup.sh (daily) removes true orphans — files no arr tracks. -# arr_cleanup uses the union model too: a file is only an orphan if -# neither arr on either server has it indexed. +# The per-arr cleanups (daily) remove true orphans — files no arr tracks. +# They use the union model too: a file is only an orphan if neither arr +# on either server has it indexed. # ───────────────────────────────────────────────────────────────────────────── ``` @@ -311,14 +332,18 @@ where determines whether you need encryption and whether sync failure matters. ## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -> **Understanding this distinction prevents a lot of confusion.** +> **Get this wrong and nothing else in here will make sense.** -Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a -primary server and a standby server with shared storage or synchronous replication. -The standby exists only to take over when the primary fails. It runs nothing -independently. Its purpose is 100% failover coverage with no service interruption. +Every guide you'll find describes the same thing: a primary, a standby, shared storage or +synchronous replication. The standby sits there doing nothing, existing only for the day the +primary dies. Pacemaker, DRBD, Proxmox HA — all built for a datacenter where downtime is +measured in dollars per second. -**This is not that.** +**This is not that.** Nobody is on call. There is no shared storage, because there is no +datacenter — there are two houses. The standby isn't idle, because it isn't a standby: it's +somebody's actual media server, and they're watching it right now. + +The bar is not five nines. The bar is *nobody notices their show stopped working.* --- @@ -656,20 +681,35 @@ Throughout the day: rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares) media_shares_permissions.sh (fix ownership before arr cleanup) media_cleaner.sh anime + media (remove scene junk) - lidarr/sonarr/radarr_cleanup.sh (remove orphaned files) - docker_daily_restart.sh (NPM, Authelia, Dispatcharr fresh start) + lidarr_release_fixer.sh (fix wrong MusicBrainz editions first) + lidarr_duplicate_artist_cleanup.sh (drop phantom zero-file duplicates) + lidarr/sonarr/radarr_cleanup.sh (remove orphaned files — library side) + arr_download_orphan_cleaner.sh (remove orphans — download side) + radarr/sonarr_classification_scan.sh --move (anime/kids sitting in the wrong root) + docker_update.sh → docker_daily_restart.sh (pull first, then restart onto it) 2:30am Sunday: Stop Emby + auth stack both sides Pull updated Docker images (zero extra downtime — already stopped) rsync Emby + Critical-Data clean mirror Start containers on new images - docker_weekly_restart.sh (NextCloud, AdGuard, Immich) + docker_update.sh --weekly → docker_weekly_restart.sh + arr_corruption_scan.sh (ffprobe every tracked video) + arr_full_rescan.sh (force a real disk↔database reconcile) 7am Sunday: sunday_morning_coffee_report.sh (everything that happened this week) + +Monthly (self-gated on 30-day uptime): + ZFS scrub, SMART long tests + docker_update.sh --remainder (everything not in the managed lists) ``` +The ordering in the 1am block is not arbitrary. Permissions run before the cleanups because +the cleanups need correct ownership to delete. The release fixer runs before the Lidarr +cleanup so it isn't cleaning up files it's about to fix. Updates run before restarts so the +restart lands on the new image. Change the order and things break quietly rather than loudly. + --- ## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ @@ -828,15 +868,74 @@ If you're setting this up from scratch on two servers: | [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback | | [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles | | [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard | +| [README-Arrs_Stack.md](Arrs_Stack/README-Arrs_Stack.md) | Union model, orphan cleanup, classification scans, corruption scan, download-orphan cleaner, discovery | +| [README-Watchdogs.md](Watchdogs/README-Watchdogs.md) | The four tiers, strike systems, reboot rate limiting, cross-watchdog coordination | +| [README-Rsync.md](Rsync/README-Rsync.md) | Profiles, pre-flight checks, merge-run, source path guards, runtime ceiling | +| [README-Deployment.md](Deployment/README-Deployment.md) | The conf templates — the only versioned record of what configuration exists | +| [README-Kernel.md](Kernel/README-Kernel.md) | The scoring kernel consumers source for discovery decisions | | [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages | +| [README-unraid.md](Plugin/unraid/README-unraid.md) | The Unraid platform layer — adapter contract, three-layer web UI, array lifecycle hooks | | [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler | | [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export | +Every folder has a `README-*.md` for *why it exists* and most have a `Manual-*.md` for +*how to operate it*. Every script has a header block covering purpose, operational model, +design principles, safeguards, configuration and runtime modes — if you want to know what a +script does, the top of the file is the authoritative answer, not this page. + +--- + +## ━━━ THINGS THAT ACTUALLY WENT WRONG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + +Most of the safeguards in this repo exist because something bit me first. If you're wondering +why a script checks something that looks paranoid, it's usually because it didn't, once. + +**The 755 GB nobody was looking at.** +Every cleanup script walked the *library* side. Nothing walked the *download* side. Completed +downloads the arrs had quietly stopped tracking piled up in SABnzbd's Completed folders — +oldest from 2022 — until the cache pool hit 89%. That's `arr_download_orphan_cleaner.sh`. + +**Lidarr said it had 23% of my music.** +1,004 of 1,357 artists reporting a fraction of their real track count, with every file sitting +right there on disk and no scan running. Organic scans only touch files involved in an import, +so an untouched library just drifts. That's `arr_full_rescan.sh`. + +**"Full House" moved. Except it didn't.** +Sonarr's API reported `episodeFileCount:192` at the new path while the actual 75 GB was still +at the old one — `moveFiles=true` flips the database instantly, but the physical move is a +separate async command draining one at a time behind ~20 others. Every classification move now +polls its own command to completion before believing the database. + +**mtime lies about when a file arrived.** +Measured on one week's imports: 400 of 400 files had mtimes over 7 days old. One was 9,613 +days old. An import preserves the release's original timestamp, so orphan detection has to +gate on ctime — which means the nightly permissions job must only touch files that are +actually wrong, because `chown` restamps ctime even when it changes nothing. Make that pass +unconditional and orphan collection silently stops. Nothing errors. The pool just fills. + +**A rsync merge pass ran for 9.7 days.** +Now every transfer attempt is capped, and resumes rather than restarting — which only works +because `--partial` is in the defaults. + +**R.E.M. was already in the library. Apparently.** +The duplicate check interpolated the artist name straight into a `grep` pattern, so `R.E.M.` +happily matched a stored `RxExMy`. Music is full of regex metacharacters. It's a literal +match now. + +**"Craig of the Creek" is not anime.** +TheTVDB tags it with the Anime genre anyway, which is why the anime signal needs corroboration +from language or network before it counts. In the other direction, "The Pussycat Dolls +Present" is a real MTV show with zero votes and no IMDb ID — which is why the junk-detection +heuristic that works on movies is deliberately *not* ported to TV. + +**`php -S` silently drops POST bodies** on Unraid's PHP 8.4. The webhook listener is Node.js +for exactly that reason, and no other. + --- ## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -Things that are different from what you might expect: +Every one of these cost me something to learn. Reading them here is cheaper. ```bash # ───────────────────────────────────────────────────────────────────────────── @@ -895,6 +994,6 @@ Things that are different from what you might expect: ``` Name: Varaverk -Repository: git@192.168.50.2:Varaverk/varaverk.git +Repository: git@192.168.50.2:FailedProxy/Varaverk.git Install path: /boot/config/plugins/varaverk ``` \ No newline at end of file