Varaverk/Arrs_Stack/README-Arrs_Stack.md

# ━━━━━ ARRS STACK ━━━━━

Lifecycle management for the arr suite (Lidarr, Sonarr, Radarr) across a multi-server
ecosystem. Library sync so every node tracks the same content. Orphan cleanup against live
arr APIs so deleted content actually leaves disk. Release correction so wrong MusicBrainz
editions don't silently block Lidarr imports. Emby notified automatically after every
deletion. Failed downloads recovered overnight. Artwork fetched continuously. Database
hygiene for dropped upstream entries. Quality upgrades propagate to all nodes immediately
via webhook. Weekly discovery adds new content based on what you actually play and watch.

> **These scripts permanently delete files and modify arr databases.** Orphan cleanup is
> protected by multiple safety layers that must all pass before anything is touched — but
> dry runs and log review are still the right first step on any new system or after any
> configuration change. See Manual-Arrs_Stack.md for the safe testing procedure.

---

## ━━━ THE PROBLEMS THAT BUILT THIS ━━━

**Deleted Shows and Removed Albums Still on Disk**
When you remove a series from Sonarr and the delete command fails — permission issue,
container wasn't running, path mismatch — the files stay permanently. Over years on an
active library this accumulates significantly.
The fix: arr cleanup scripts query the live API for every tracked file path, walk the
disk, and delete anything absent from the API response that's old enough to be past the
import window.

**Wrong MusicBrainz Release Edition Silently Blocking Lidarr Imports**
Lidarr selects one specific release edition per album using a MusicBrainz release ID.
When it picks the wrong edition (Brazil CD instead of US CD, Japan Digital instead of
standard), the track IDs don't match what's embedded in the files. RescanFolders reports
"Importing 0 tracks" even with perfectly tagged, complete files present. No error — just
silence.
The fix: `lidarr_release_fixer.sh` reads the MUSICBRAINZ_ALBUMID tag from each file,
finds the matching release in Lidarr's known releases, switches the selection, and queues
a RefreshArtist. Runs before cleanup so corrected albums are imported before the orphan
scan ever sees them.

**Emby Showing Ghost Entries After Cleanup**
After arr cleanup deletes files, Emby still shows them until its next scheduled scan —
potentially hours later. Users see broken entries that produce "file not found" errors.
The fix: `notify_emby_scan()` is called automatically after every deletion. Triggers
Emby's "Clean Missing Files" task immediately.

**No Safety Net on Deletion Size**
A misconfigured root path — pointing cleanup at the wrong directory — means the API
returns zero tracked files for a root that actually contains thousands. Every file walks
as an orphan. Everything gets deleted. This is the catastrophic failure mode.
The fix: `LIDARR/SONARR/RADARR_MAX_DELETE_GB` — if total deletion size exceeds the
limit, the script stops and requires `--i-know-what-im-doing` to proceed. The flag name
is long and annoying by design. It cannot be added by accident.

**Failed Downloads Accumulating Silently**
Import failures and stalled downloads sit in arr queues indefinitely. Without
intervention they occupy queue slots, block new searches, and the item never gets
downloaded. Checking queues manually across three arrs is tedious.
The fix: `arrs_failed_stalled_recovery.sh` inspects all arr queues, blocklists the bad
release, removes it, and triggers a re-search. Runs daily.

---

## ━━━ WHAT THIS FOLDER DOES ━━━

**Library Sync**
`arr_sync.sh` — full-mesh arr library sync across all nodes. Every node syncs with every
other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the
actual files.

**Release Correction**
`lidarr_release_fixer.sh` — reads MUSICBRAINZ_ALBUMID from FLAC and MP3 files, matches
against Lidarr's known releases per album, switches monitored=true to the correct
edition, queues RefreshArtist. Runs before lidarr_cleanup.sh daily. Handles both FLAC
(vorbis comment block) and MP3 (ID3v2 TXXX frame).

**Orphan Cleanup**
`lidarr_cleanup.sh`, `sonarr_cleanup.sh`, `radarr_cleanup.sh` — API-verified orphan
removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven
safety layers, automatic Emby notification after deletion.

**Database Hygiene**
`radarr_tmdb_removed.sh`, `sonarr_tvdb_removed.sh` — remove entries that upstream
databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in
arrs and can never be monitored or downloaded. Most are announced-but-never-released
entries. Files are kept by default — most have none.

**Library Recovery**
`arrs_failed_stalled_recovery.sh` — detect and recover failed imports and stalled
downloads across all arrs. Blocklists the bad release and triggers a re-search — hands-
free overnight recovery.

**Library Enrichment**
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
fallback sources. Never overwrites existing files.

**Upgrade Propagation**
`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr
OnUpgrade webhooks. Continuous; started at array start. Writes to
`/var/log/varaverk/upgrade_webhook.log`.
`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded
item folder to every remote node immediately, then triggers an arr library rescan on
each remote so the upgraded file is accepted without triggering a redundant quality search.

**Discovery**
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
Lidarr. 0–5 meaningful adds per week.
`playback_aware_radarr_discovery.sh` — behavior-driven movie discovery. Scores recently
watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr.
`playback_aware_sonarr_discovery.sh` — behavior-driven TV discovery. Scores recently
watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds
top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds.

---

## ━━━ EXECUTION ORDER ━━━

**Daily via `daily_sync_maintenance.sh` (DAILY_MAINTENANCE_SCRIPTS):**

```
1. lidarr_release_fixer.sh   — correct wrong release editions before cleanup sees them
2. lidarr_cleanup.sh         — orphan removal (music)
3. sonarr_cleanup.sh         — orphan removal (TV)
4. radarr_cleanup.sh         — orphan removal (movies)
5. lidarr_missing_art.sh     — fetch missing artwork (HOST1 only)
6. radarr_tmdb_removed.sh    — remove TMDb-dropped movies
7. sonarr_tvdb_removed.sh    — remove TVDB-dropped series
```

Note: `media_shares_permissions.sh` and `media_cleaner.sh` run before these from
`Media/` — permissions and junk removal must complete first.

**Every 30 min + 4hr via orchestrators (CRITICAL/INTERMEDIATE_MAINTENANCE_SCRIPTS):**

```
arrs_failed_stalled_recovery.sh   — failed/stalled queue recovery
arr_sync.sh                       — library sync across all nodes
```

**Weekly via `weekly_sync_maintenance.sh` (WEEKLY_MAINTENANCE_SCRIPTS):**

```
playback_aware_lidarr_discovery.sh   — score play history → Last.fm similar → Lidarr
playback_aware_radarr_discovery.sh   — score watch history → TMDB recommendations → Radarr
playback_aware_sonarr_discovery.sh   — score episode history → TMDB TV → Sonarr
```

**Continuous (started by `array_started.sh`):**

```
start_webhook_listener.sh   — Node.js webhook server; dispatches upgrade_webhook_handler.sh
```

**On every arr upgrade (triggered by webhook):**

```
upgrade_webhook_handler.sh  — push upgraded folder to all remote nodes + trigger arr rescan
```

**Why release fixer before cleanup:** the fixer corrects Lidarr's release selection so
files get imported. If cleanup ran first, a correctable album could accumulate age toward
the orphan threshold before the fixer had a chance to fix it.

**Why arr_sync before rsync:** once arrs agree on what to track, rsync spreads the actual
files. An upgrade on one node — new tracked path, old path no longer in API — gets
cleaned by arr_cleanup on all nodes after the next sync cycle.

---

## ━━━ HOST AWARENESS ━━━

Scripts run on both servers via `detect_hosts()`, which aliases all `HOST*_` prefixed vars
to their unprefixed names at runtime. No manual `HOST1`/`HOST2` comparisons exist in any
script.

`arr_sync.sh` keeps all arr databases in bidirectional union — either server can download
to any share. Arr cleanup uses the union model: a file is only an orphan if the arr on
this host doesn't have it indexed. Arr scripts check the aliased URL — if empty (arr not
configured on this host), they exit cleanly with no action.

`lidarr_release_fixer.sh` and `lidarr_missing_art.sh` exit cleanly on hosts without
Lidarr configured — no HOST1_LIDARR_URL means nothing runs.

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━

| Script | Role | When It Runs |
|--------|------|--------------|
| `arr_sync.sh` | Full-mesh arr library sync — all nodes track the same content | Every 4hr + weekly before rsync |
| `lidarr_release_fixer.sh` | Fix wrong MusicBrainz release editions so files get imported | Daily before lidarr_cleanup |
| `lidarr_cleanup.sh` | Delete orphaned music files not tracked by Lidarr | Daily |
| `sonarr_cleanup.sh` | Delete orphaned TV files not tracked by Sonarr | Daily |
| `radarr_cleanup.sh` | Delete orphaned movie files not tracked by Radarr | Daily |
| `arrs_failed_stalled_recovery.sh` | Auto-recover failed imports and stalled downloads | Every 30 min / daily |
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Daily (HOST1 only) |
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Daily |
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Daily |
| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous |
| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade |
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
| `arr_download_orphan_cleaner.sh` | Clear orphaned completed downloads out of the SABnzbd Completed folders | Daily |
| `sonarr_classification_scan.sh` | Detect series sitting in the wrong root (anime / kids / general); `--move` acts | Daily |
| `radarr_classification_scan.sh` | Same for movies, plus junk-metadata detection via `--remove-junk` | Daily |
| `lidarr_duplicate_artist_cleanup.sh` | Remove phantom zero-file duplicate artists; flag real ones for review | Daily |
| `arr_cache_prefill.sh` | Warm the shared tracked-data cache so consumers never read cold | Array start + every 4hr |
| `arr_corruption_scan.sh` | ffprobe every tracked video for corrupt headers; `--remediate` deletes + re-searches | Weekly |
| `arr_full_rescan.sh` | Force a real disk↔database reconciliation on all three arrs | Weekly |

---

## ━━━ THE NEWER LAYERS ━━━

The original folder was "delete what the arrs no longer track". These were added as distinct
failure modes surfaced — each exists because something went wrong that the cleanups could not
have caught.

### 🗑️  Download-Side Orphans — `arr_download_orphan_cleaner.sh`

Every cleanup script here walks the **library** side. Nothing walked the **download** side —
so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed
folders indefinitely. Discovered as **755 GB of orphaned TV downloads, oldest from 2022**,
filling the cache pool to 89%.

Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and
acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr
is skipped entirely, because without it there is no way to tell an active import from an
orphan. A run total over `DOWNLOAD_ORPHAN_MAX_DELETE_GB` aborts — an abnormally large delete
is the visible symptom of a partial queue fetch.

### 🎭  Wrong-Root Detection — `sonarr_classification_scan.sh` + `radarr_classification_scan.sh`

Overseerr lets any user request content into the wrong root folder — kids shows into general
TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre,
certification, network/studio, original language) and report where the computed classification
disagrees with the folder the item actually sits in.

Report-only by default. `--move` acts on forward misplacements and adult-content-in-kids-root
leaks. It deliberately does **not** move non-anime content out of the anime root — deliberate
style placements (Western animation grouped with anime by choice) are genuine judgment calls.

Both poll the arr's async move command to completion before verifying, because `moveFiles=true`
flips the database instantly while the physical move is still queued behind others.

### 🎨  Phantom Artists — `lidarr_duplicate_artist_cleanup.sh`

MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one
holding nothing. Removes only the zero-file side, with `deleteFiles=false` so nothing on disk
is touched. Pairs where both sides hold files are flagged for review, never auto-resolved.

Gated on the tracked-count floor shared with `lidarr_cleanup.sh` — during a library-wide desync
both sides of a real duplicate can read as zero-file phantoms.

### 🩺  Corruption + Reconciliation — `arr_corruption_scan.sh` + `arr_full_rescan.sh`

`arr_corruption_scan.sh` ffprobes tracked video files for corrupt headers. Report-only unless
`--remediate`, which deletes the file record and triggers an explicit re-search. Requires
repeat detections across separate runs before acting, so a transient probe failure cannot
delete a healthy file.

`arr_full_rescan.sh` forces a genuine disk↔database reconciliation. Organic scans only touch
files involved in an import, so an untouched library silently drifts — confirmed when Lidarr
reported **~23% of its true track count** for 1,004 of 1,357 artists with no scan running and
every file present on disk.

### ⚡  Cache Warmth — `arr_cache_prefill.sh`

Populates the shared tracked-data cache at array start and every 4 hours, so consumers never
pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still
writes through on a cold cache.

---

## ━━━ HOW THE SCRIPTS RELATE ━━━

```
Every 4hr / weekly (arr sync before rsync):
  arr_sync.sh ──────────────── syncs tracked IDs across all nodes
        │                      union model: any node adds → all nodes get it
        │
        └── then rsync spreads the actual files to all nodes
        └── then arr_cleanup removes orphans on all nodes (old paths, removed content)

Daily maintenance window:
  [Media/media_shares_permissions.sh + media_cleaner.sh run first — from Media/]
        │
        ▼
  lidarr_release_fixer.sh ────── reads MBID tag → switches release in Lidarr → RefreshArtist
        │ (corrected albums get imported before cleanup scans for orphans)
        ▼
  lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans
  sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans
  radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans
        │
        └── each cleanup → notify_emby_scan() → Emby removes ghost entries

Daily recovery:
  arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search

Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
  playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr
  playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr
  playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr
        │
        └── each discovery script fires arr search immediately after successful add

Continuous (started by array_started.sh):
  start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
        │ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
        └── upgrade_webhook_handler.sh
              ├── rsync upgraded folder → all remote nodes immediately
              └── trigger arr library rescan on each remote (accept new file, no quality search)

Ad-hoc enrichment:
  lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
  radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
  sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion
```