Give the README a voice and bring it current

It is the front page and it read like a spec sheet. Adds the incidents the safeguards came from, an honest note about what is actually switched on, and the layers built since.
2026-08-01 23:34:34 -04:00
parent 1ed6b92118
commit 671bf2352b
1 changed files with 115 additions and 16 deletions
@@ -23,6 +23,27 @@ correct adapter. The scripts themselves never branch on the OS.

 ---

+## ━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+This is a live system, not a finished product. HOST1 has been running the whole stack
+unattended for months. HOST2 is mid-rebuild, so some of what's described below is written
+and tested but currently switched off:
+
+```
+FALLBACK_ENABLED=true        ← running
+PARTNERSHIP_ENABLED=true     ← running
+RSYNC_ENABLED=false          ← off until HOST2 is back
+CONF_SYNC_ENABLED=false      ← off until HOST2 is back
+```
+
+Everything here is built and audited. The two-server half is waiting on the second server.
+If a section reads like it's describing something that already happens every night — on
+HOST1, it is. Across both hosts, it will be.
+
+I'd rather tell you that than have you find out by reading the toggles.
+
+---
+
 ## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

 ```
@@ -76,7 +97,7 @@ break silently.
    Dispatcharr             # shared Live TV scheduler
 ```

-When `fallback.sh` on HOST2 reads `FALLBACK_HOST2_COVERS_HOST1_TIER1=(..."Emby"...)`,
+When `fallback.sh` on HOST2 reads `FALLBACK_HOST1_TIER1=(..."Emby"...)`,
 it looks for a container called exactly `Emby` and starts it. HOST2 must have a
 container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
 its volume mounts pointing at the mirrored share paths. This is what makes one
@@ -113,7 +134,7 @@ the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts tar
 # The entire ecosystem uses container names as identifiers.
 # docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
 #   → looks for exactly "NginxProxyManager" on the running server
-# fallback.sh: FALLBACK_HOST2_COVERS_HOST1_TIER1=("Emby" "VaultWarden")
+# fallback.sh: FALLBACK_HOST1_TIER1=("Emby" "VaultWarden")
 #   → looks for exactly "Emby" and "VaultWarden" on HOST2
 # docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
 #   → looks for exactly those names in docker ps
@@ -226,9 +247,9 @@ across separate shares per-user or per-application.
 #   arr_sync.sh guarantees both arrs agree on what exists before files move.
 #   rsync then spreads the files to make storage match what arrs already know.
 #
-# arr_cleanup.sh (daily) removes true orphans — files no arr tracks.
-# arr_cleanup uses the union model too: a file is only an orphan if
-# neither arr on either server has it indexed.
+# The per-arr cleanups (daily) remove true orphans — files no arr tracks.
+# They use the union model too: a file is only an orphan if neither arr
+# on either server has it indexed.
 # ─────────────────────────────────────────────────────────────────────────────
 ```

@@ -311,14 +332,18 @@ where determines whether you need encryption and whether sync failure matters.

 ## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

-> **Understanding this distinction prevents a lot of confusion.**
+> **Get this wrong and nothing else in here will make sense.**

-Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a
-primary server and a standby server with shared storage or synchronous replication.
-The standby exists only to take over when the primary fails. It runs nothing
-independently. Its purpose is 100% failover coverage with no service interruption.
+Every guide you'll find describes the same thing: a primary, a standby, shared storage or
+synchronous replication. The standby sits there doing nothing, existing only for the day the
+primary dies. Pacemaker, DRBD, Proxmox HA — all built for a datacenter where downtime is
+measured in dollars per second.

-**This is not that.**
+**This is not that.** Nobody is on call. There is no shared storage, because there is no
+datacenter — there are two houses. The standby isn't idle, because it isn't a standby: it's
+somebody's actual media server, and they're watching it right now.
+
+The bar is not five nines. The bar is *nobody notices their show stopped working.*

 ---

@@ -656,20 +681,35 @@ Throughout the day:
  rsync Anime_Shows, Anime_Movies → HOST1      (HOST2 pushes its truth shares)
  media_shares_permissions.sh                  (fix ownership before arr cleanup)
  media_cleaner.sh anime + media               (remove scene junk)
-  lidarr/sonarr/radarr_cleanup.sh              (remove orphaned files)
-  docker_daily_restart.sh                      (NPM, Authelia, Dispatcharr fresh start)
+  lidarr_release_fixer.sh                      (fix wrong MusicBrainz editions first)
+  lidarr_duplicate_artist_cleanup.sh           (drop phantom zero-file duplicates)
+  lidarr/sonarr/radarr_cleanup.sh              (remove orphaned files — library side)
+  arr_download_orphan_cleaner.sh               (remove orphans — download side)
+  radarr/sonarr_classification_scan.sh --move  (anime/kids sitting in the wrong root)
+  docker_update.sh → docker_daily_restart.sh   (pull first, then restart onto it)

 2:30am Sunday:
  Stop Emby + auth stack both sides
  Pull updated Docker images (zero extra downtime — already stopped)
  rsync Emby + Critical-Data clean mirror
  Start containers on new images
-  docker_weekly_restart.sh                     (NextCloud, AdGuard, Immich)
+  docker_update.sh --weekly → docker_weekly_restart.sh
+  arr_corruption_scan.sh                       (ffprobe every tracked video)
+  arr_full_rescan.sh                           (force a real disk↔database reconcile)

 7am Sunday:
  sunday_morning_coffee_report.sh              (everything that happened this week)
+
+Monthly (self-gated on 30-day uptime):
+  ZFS scrub, SMART long tests
+  docker_update.sh --remainder                 (everything not in the managed lists)
 ```

+The ordering in the 1am block is not arbitrary. Permissions run before the cleanups because
+the cleanups need correct ownership to delete. The release fixer runs before the Lidarr
+cleanup so it isn't cleaning up files it's about to fix. Updates run before restarts so the
+restart lands on the new image. Change the order and things break quietly rather than loudly.
+
 ---

 ## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
@@ -828,15 +868,74 @@ If you're setting this up from scratch on two servers:
 | [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
 | [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
 | [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
+| [README-Arrs_Stack.md](Arrs_Stack/README-Arrs_Stack.md) | Union model, orphan cleanup, classification scans, corruption scan, download-orphan cleaner, discovery |
+| [README-Watchdogs.md](Watchdogs/README-Watchdogs.md) | The four tiers, strike systems, reboot rate limiting, cross-watchdog coordination |
+| [README-Rsync.md](Rsync/README-Rsync.md) | Profiles, pre-flight checks, merge-run, source path guards, runtime ceiling |
+| [README-Deployment.md](Deployment/README-Deployment.md) | The conf templates — the only versioned record of what configuration exists |
+| [README-Kernel.md](Kernel/README-Kernel.md) | The scoring kernel consumers source for discovery decisions |
 | [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
+| [README-unraid.md](Plugin/unraid/README-unraid.md) | The Unraid platform layer — adapter contract, three-layer web UI, array lifecycle hooks |
 | [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
 | [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |

+Every folder has a `README-*.md` for *why it exists* and most have a `Manual-*.md` for
+*how to operate it*. Every script has a header block covering purpose, operational model,
+design principles, safeguards, configuration and runtime modes — if you want to know what a
+script does, the top of the file is the authoritative answer, not this page.
+
+---
+
+## ━━━ THINGS THAT ACTUALLY WENT WRONG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+Most of the safeguards in this repo exist because something bit me first. If you're wondering
+why a script checks something that looks paranoid, it's usually because it didn't, once.
+
+**The 755 GB nobody was looking at.**
+Every cleanup script walked the *library* side. Nothing walked the *download* side. Completed
+downloads the arrs had quietly stopped tracking piled up in SABnzbd's Completed folders —
+oldest from 2022 — until the cache pool hit 89%. That's `arr_download_orphan_cleaner.sh`.
+
+**Lidarr said it had 23% of my music.**
+1,004 of 1,357 artists reporting a fraction of their real track count, with every file sitting
+right there on disk and no scan running. Organic scans only touch files involved in an import,
+so an untouched library just drifts. That's `arr_full_rescan.sh`.
+
+**"Full House" moved. Except it didn't.**
+Sonarr's API reported `episodeFileCount:192` at the new path while the actual 75 GB was still
+at the old one — `moveFiles=true` flips the database instantly, but the physical move is a
+separate async command draining one at a time behind ~20 others. Every classification move now
+polls its own command to completion before believing the database.
+
+**mtime lies about when a file arrived.**
+Measured on one week's imports: 400 of 400 files had mtimes over 7 days old. One was 9,613
+days old. An import preserves the release's original timestamp, so orphan detection has to
+gate on ctime — which means the nightly permissions job must only touch files that are
+actually wrong, because `chown` restamps ctime even when it changes nothing. Make that pass
+unconditional and orphan collection silently stops. Nothing errors. The pool just fills.
+
+**A rsync merge pass ran for 9.7 days.**
+Now every transfer attempt is capped, and resumes rather than restarting — which only works
+because `--partial` is in the defaults.
+
+**R.E.M. was already in the library. Apparently.**
+The duplicate check interpolated the artist name straight into a `grep` pattern, so `R.E.M.`
+happily matched a stored `RxExMy`. Music is full of regex metacharacters. It's a literal
+match now.
+
+**"Craig of the Creek" is not anime.**
+TheTVDB tags it with the Anime genre anyway, which is why the anime signal needs corroboration
+from language or network before it counts. In the other direction, "The Pussycat Dolls
+Present" is a real MTV show with zero votes and no IMDb ID — which is why the junk-detection
+heuristic that works on movies is deliberately *not* ported to TV.
+
+**`php -S` silently drops POST bodies** on Unraid's PHP 8.4. The webhook listener is Node.js
+for exactly that reason, and no other.
+
 ---

 ## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

-Things that are different from what you might expect:
+Every one of these cost me something to learn. Reading them here is cheaper.

 ```bash
 # ─────────────────────────────────────────────────────────────────────────────
@@ -895,6 +994,6 @@ Things that are different from what you might expect:

 ```
 Name:         Varaverk
-Repository:   git@192.168.50.2:Varaverk/varaverk.git
+Repository:   git@192.168.50.2:FailedProxy/Varaverk.git
 Install path: /boot/config/plugins/varaverk
 ```