Varaverk/Arrs_Stack/README-Arrs_Stack.md

━━━━━ ARRS STACK ━━━━━

Lifecycle management for the arr suite (Lidarr, Sonarr, Radarr) across a multi-server ecosystem. Library sync so every node tracks the same content. Orphan cleanup against live arr APIs so deleted content actually leaves disk. Release correction so wrong MusicBrainz editions don't silently block Lidarr imports. Emby notified automatically after every deletion. Failed downloads recovered overnight. Artwork fetched continuously. Database hygiene for dropped upstream entries. Quality upgrades propagate to all nodes immediately via webhook. Weekly discovery adds new content based on what you actually play and watch.

These scripts permanently delete files and modify arr databases. Orphan cleanup is protected by multiple safety layers that must all pass before anything is touched — but dry runs and log review are still the right first step on any new system or after any configuration change. See Manual-Arrs_Stack.md for the safe testing procedure.

---

━━━ THE PROBLEMS THAT BUILT THIS ━━━

Deleted Shows and Removed Albums Still on Disk When you remove a series from Sonarr and the delete command fails — permission issue, container wasn't running, path mismatch — the files stay permanently. Over years on an active library this accumulates significantly. The fix: arr cleanup scripts query the live API for every tracked file path, walk the disk, and delete anything absent from the API response that's old enough to be past the import window.

Wrong MusicBrainz Release Edition Silently Blocking Lidarr Imports Lidarr selects one specific release edition per album using a MusicBrainz release ID. When it picks the wrong edition (Brazil CD instead of US CD, Japan Digital instead of standard), the track IDs don't match what's embedded in the files. RescanFolders reports "Importing 0 tracks" even with perfectly tagged, complete files present. No error — just silence. The fix: lidarr_release_fixer.sh reads the MUSICBRAINZ_ALBUMID tag from each file, finds the matching release in Lidarr's known releases, switches the selection, and queues a RefreshArtist. Runs before cleanup so corrected albums are imported before the orphan scan ever sees them.

Emby Showing Ghost Entries After Cleanup After arr cleanup deletes files, Emby still shows them until its next scheduled scan — potentially hours later. Users see broken entries that produce "file not found" errors. The fix: notify_emby_scan() is called automatically after every deletion. Triggers Emby's "Clean Missing Files" task immediately.

No Safety Net on Deletion Size A misconfigured root path — pointing cleanup at the wrong directory — means the API returns zero tracked files for a root that actually contains thousands. Every file walks as an orphan. Everything gets deleted. This is the catastrophic failure mode. The fix: LIDARR/SONARR/RADARR_MAX_DELETE_GB — if total deletion size exceeds the limit, the script stops and requires --i-know-what-im-doing to proceed. The flag name is long and annoying by design. It cannot be added by accident.

Failed Downloads Accumulating Silently Import failures and stalled downloads sit in arr queues indefinitely. Without intervention they occupy queue slots, block new searches, and the item never gets downloaded. Checking queues manually across three arrs is tedious. The fix: arrs_failed_stalled_recovery.sh inspects all arr queues, blocklists the bad release, removes it, and triggers a re-search. Runs daily.

---

━━━ WHAT THIS FOLDER DOES ━━━

Library Sync arr_sync.sh — full-mesh arr library sync across all nodes. Every node syncs with every other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the actual files.

Release Correction lidarr_release_fixer.sh — reads MUSICBRAINZ_ALBUMID from FLAC and MP3 files, matches against Lidarr's known releases per album, switches monitored=true to the correct edition, queues RefreshArtist. Runs before lidarr_cleanup.sh daily. Handles both FLAC (vorbis comment block) and MP3 (ID3v2 TXXX frame).

Orphan Cleanup lidarr_cleanup.sh, sonarr_cleanup.sh, radarr_cleanup.sh — API-verified orphan removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven safety layers, automatic Emby notification after deletion.

Database Hygiene radarr_tmdb_removed.sh, sonarr_tvdb_removed.sh — remove entries that upstream databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in arrs and can never be monitored or downloaded. Most are announced-but-never-released entries. Files are kept by default — most have none.

Library Recovery arrs_failed_stalled_recovery.sh — detect and recover failed imports and stalled downloads across all arrs. Blocklists the bad release and triggers a re-search — hands- free overnight recovery.

Library Enrichment lidarr_missing_art.sh — fetch missing album and artist artwork from fanart.tv and fallback sources. Never overwrites existing files.

Upgrade Propagation start_webhook_listener.sh — Node.js HTTP server that receives Sonarr/Radarr/Lidarr OnUpgrade webhooks. Continuous; started at array start. Writes to /var/log/varaverk/upgrade_webhook.log. upgrade_webhook_handler.sh — triggered by the webhook listener. Pushes the upgraded item folder to every remote node immediately, then triggers an arr library rescan on each remote so the upgraded file is accepted without triggering a redundant quality search.

Discovery playback_aware_lidarr_discovery.sh — behavior-driven music discovery. Scores your Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to Lidarr. 0–5 meaningful adds per week. playback_aware_radarr_discovery.sh — behavior-driven movie discovery. Scores recently watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr. playback_aware_sonarr_discovery.sh — behavior-driven TV discovery. Scores recently watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds.

---

━━━ EXECUTION ORDER ━━━

Daily via daily_sync_maintenance.sh (DAILY_MAINTENANCE_SCRIPTS):

1. lidarr_release_fixer.sh   — correct wrong release editions before cleanup sees them
2. lidarr_cleanup.sh         — orphan removal (music)
3. sonarr_cleanup.sh         — orphan removal (TV)
4. radarr_cleanup.sh         — orphan removal (movies)
5. lidarr_missing_art.sh     — fetch missing artwork (HOST1 only)
6. radarr_tmdb_removed.sh    — remove TMDb-dropped movies
7. sonarr_tvdb_removed.sh    — remove TVDB-dropped series

Note: media_shares_permissions.sh and media_cleaner.sh run before these from Media/ — permissions and junk removal must complete first.

Every 30 min + 4hr via orchestrators (CRITICAL/INTERMEDIATE_MAINTENANCE_SCRIPTS):

arrs_failed_stalled_recovery.sh   — failed/stalled queue recovery
arr_sync.sh                       — library sync across all nodes

Weekly via weekly_sync_maintenance.sh (WEEKLY_MAINTENANCE_SCRIPTS):

playback_aware_lidarr_discovery.sh   — score play history → Last.fm similar → Lidarr
playback_aware_radarr_discovery.sh   — score watch history → TMDB recommendations → Radarr
playback_aware_sonarr_discovery.sh   — score episode history → TMDB TV → Sonarr

Continuous (started by array_started.sh):

start_webhook_listener.sh   — Node.js webhook server; dispatches upgrade_webhook_handler.sh

On every arr upgrade (triggered by webhook):

upgrade_webhook_handler.sh  — push upgraded folder to all remote nodes + trigger arr rescan

Why release fixer before cleanup: the fixer corrects Lidarr's release selection so files get imported. If cleanup ran first, a correctable album could accumulate age toward the orphan threshold before the fixer had a chance to fix it.

Why arr_sync before rsync: once arrs agree on what to track, rsync spreads the actual files. An upgrade on one node — new tracked path, old path no longer in API — gets cleaned by arr_cleanup on all nodes after the next sync cycle.

---

━━━ HOST AWARENESS ━━━

Scripts run on both servers via detect_hosts(), which aliases all HOST*_ prefixed vars to their unprefixed names at runtime. No manual HOST1/HOST2 comparisons exist in any script.

arr_sync.sh keeps all arr databases in bidirectional union — either server can download to any share. Arr cleanup uses the union model: a file is only an orphan if the arr on this host doesn't have it indexed. Arr scripts check the aliased URL — if empty (arr not configured on this host), they exit cleanly with no action.

lidarr_release_fixer.sh and lidarr_missing_art.sh exit cleanly on hosts without Lidarr configured — no HOST1_LIDARR_URL means nothing runs.

---

━━━ SCRIPTS IN THIS FOLDER ━━━

Script	Role	When It Runs
arr_sync.sh	Full-mesh arr library sync — all nodes track the same content	Every 4hr + weekly before rsync
lidarr_release_fixer.sh	Fix wrong MusicBrainz release editions so files get imported	Daily before lidarr_cleanup
lidarr_cleanup.sh	Delete orphaned music files not tracked by Lidarr	Daily
sonarr_cleanup.sh	Delete orphaned TV files not tracked by Sonarr	Daily
radarr_cleanup.sh	Delete orphaned movie files not tracked by Radarr	Daily
arrs_failed_stalled_recovery.sh	Auto-recover failed imports and stalled downloads	Every 30 min / daily
lidarr_missing_art.sh	Fetch missing album and artist artwork	Daily (HOST1 only)
radarr_tmdb_removed.sh	Remove movies dropped from TMDb	Daily
sonarr_tvdb_removed.sh	Remove series dropped from TVDB	Daily
start_webhook_listener.sh	Node.js webhook server — receive arr OnUpgrade and dispatch handler	Continuous
upgrade_webhook_handler.sh	Push upgraded item folder to remote nodes + trigger arr rescan	On each arr upgrade
playback_aware_lidarr_discovery.sh	Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr	Weekly
playback_aware_radarr_discovery.sh	Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr	Weekly
playback_aware_sonarr_discovery.sh	Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr	Weekly
arr_download_orphan_cleaner.sh	Clear orphaned completed downloads out of the SABnzbd Completed folders	Daily
sonarr_classification_scan.sh	Detect series sitting in the wrong root (anime / kids / general); --move acts	Daily
radarr_classification_scan.sh	Same for movies, plus junk-metadata detection via --remove-junk	Daily
lidarr_duplicate_artist_cleanup.sh	Remove phantom zero-file duplicate artists; flag real ones for review	Daily
arr_cache_prefill.sh	Warm the shared tracked-data cache so consumers never read cold	Array start + every 4hr
arr_corruption_scan.sh	ffprobe every tracked video for corrupt headers; --remediate deletes + re-searches	Weekly
arr_full_rescan.sh	Force a real disk↔database reconciliation on all three arrs	Weekly

---

━━━ THE NEWER LAYERS ━━━

The original folder was "delete what the arrs no longer track". These were added as distinct failure modes surfaced — each exists because something went wrong that the cleanups could not have caught.

🗑️ Download-Side Orphans — arr_download_orphan_cleaner.sh

Every cleanup script here walks the library side. Nothing walked the download side — so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed folders indefinitely. Discovered as 755 GB of orphaned TV downloads, oldest from 2022, filling the cache pool to 89%.

Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr is skipped entirely, because without it there is no way to tell an active import from an orphan. A run total over DOWNLOAD_ORPHAN_MAX_DELETE_GB aborts — an abnormally large delete is the visible symptom of a partial queue fetch.

🎭 Wrong-Root Detection — sonarr_classification_scan.sh + radarr_classification_scan.sh

Overseerr lets any user request content into the wrong root folder — kids shows into general TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre, certification, network/studio, original language) and report where the computed classification disagrees with the folder the item actually sits in.

Report-only by default. --move acts on forward misplacements and adult-content-in-kids-root leaks. It deliberately does not move non-anime content out of the anime root — deliberate style placements (Western animation grouped with anime by choice) are genuine judgment calls.

Both poll the arr's async move command to completion before verifying, because moveFiles=true flips the database instantly while the physical move is still queued behind others.

🎨 Phantom Artists — lidarr_duplicate_artist_cleanup.sh

MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one holding nothing. Removes only the zero-file side, with deleteFiles=false so nothing on disk is touched. Pairs where both sides hold files are flagged for review, never auto-resolved.

Gated on the tracked-count floor shared with lidarr_cleanup.sh — during a library-wide desync both sides of a real duplicate can read as zero-file phantoms.

🩺 Corruption + Reconciliation — arr_corruption_scan.sh + arr_full_rescan.sh

arr_corruption_scan.sh ffprobes tracked video files for corrupt headers. Report-only unless --remediate, which deletes the file record and triggers an explicit re-search. Requires repeat detections across separate runs before acting, so a transient probe failure cannot delete a healthy file.

arr_full_rescan.sh forces a genuine disk↔database reconciliation. Organic scans only touch files involved in an import, so an untouched library silently drifts — confirmed when Lidarr reported ~23% of its true track count for 1,004 of 1,357 artists with no scan running and every file present on disk.

⚡ Cache Warmth — arr_cache_prefill.sh

Populates the shared tracked-data cache at array start and every 4 hours, so consumers never pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still writes through on a cold cache.

---

━━━ HOW THE SCRIPTS RELATE ━━━

Every 4hr / weekly (arr sync before rsync):
  arr_sync.sh ──────────────── syncs tracked IDs across all nodes
        │                      union model: any node adds → all nodes get it
        │
        └── then rsync spreads the actual files to all nodes
        └── then arr_cleanup removes orphans on all nodes (old paths, removed content)

Daily maintenance window:
  [Media/media_shares_permissions.sh + media_cleaner.sh run first — from Media/]
        │
        ▼
  lidarr_release_fixer.sh ────── reads MBID tag → switches release in Lidarr → RefreshArtist
        │ (corrected albums get imported before cleanup scans for orphans)
        ▼
  lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans
  sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans
  radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans
        │
        └── each cleanup → notify_emby_scan() → Emby removes ghost entries

Daily recovery:
  arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search

Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
  playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr
  playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr
  playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr
        │
        └── each discovery script fires arr search immediately after successful add

Continuous (started by array_started.sh):
  start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
        │ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
        └── upgrade_webhook_handler.sh
              ├── rsync upgraded folder → all remote nodes immediately
              └── trigger arr library rescan on each remote (accept new file, no quality search)

Ad-hoc enrichment:
  lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
  radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
  sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion