Split Media/ docs into Media/ and Arrs_Stack/ to match folder reorganization
Media README and Manual now cover only the 3 remaining scripts (permissions, cleaner, play_state_sync). Arrs_Stack README and Manual cover all arr stack scripts including lidarr_release_fixer. Fixed stale --skip-strike-list reference in flag docs.
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,250 @@
|
||||
# ━━━━━ ARRS STACK ━━━━━
|
||||
|
||||
Lifecycle management for the arr suite (Lidarr, Sonarr, Radarr) across a multi-server
|
||||
ecosystem. Library sync so every node tracks the same content. Orphan cleanup against live
|
||||
arr APIs so deleted content actually leaves disk. Release correction so wrong MusicBrainz
|
||||
editions don't silently block Lidarr imports. Emby notified automatically after every
|
||||
deletion. Failed downloads recovered overnight. Artwork fetched continuously. Database
|
||||
hygiene for dropped upstream entries. Quality upgrades propagate to all nodes immediately
|
||||
via webhook. Weekly discovery adds new content based on what you actually play and watch.
|
||||
|
||||
> **These scripts permanently delete files and modify arr databases.** Orphan cleanup is
|
||||
> protected by multiple safety layers that must all pass before anything is touched — but
|
||||
> dry runs and log review are still the right first step on any new system or after any
|
||||
> configuration change. See Manual-Arrs_Stack.md for the safe testing procedure.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEMS THAT BUILT THIS ━━━
|
||||
|
||||
**Deleted Shows and Removed Albums Still on Disk**
|
||||
When you remove a series from Sonarr and the delete command fails — permission issue,
|
||||
container wasn't running, path mismatch — the files stay permanently. Over years on an
|
||||
active library this accumulates significantly.
|
||||
The fix: arr cleanup scripts query the live API for every tracked file path, walk the
|
||||
disk, and delete anything absent from the API response that's old enough to be past the
|
||||
import window.
|
||||
|
||||
**Wrong MusicBrainz Release Edition Silently Blocking Lidarr Imports**
|
||||
Lidarr selects one specific release edition per album using a MusicBrainz release ID.
|
||||
When it picks the wrong edition (Brazil CD instead of US CD, Japan Digital instead of
|
||||
standard), the track IDs don't match what's embedded in the files. RescanFolders reports
|
||||
"Importing 0 tracks" even with perfectly tagged, complete files present. No error — just
|
||||
silence.
|
||||
The fix: `lidarr_release_fixer.sh` reads the MUSICBRAINZ_ALBUMID tag from each file,
|
||||
finds the matching release in Lidarr's known releases, switches the selection, and queues
|
||||
a RefreshArtist. Runs before cleanup so corrected albums are imported before the orphan
|
||||
scan ever sees them.
|
||||
|
||||
**Emby Showing Ghost Entries After Cleanup**
|
||||
After arr cleanup deletes files, Emby still shows them until its next scheduled scan —
|
||||
potentially hours later. Users see broken entries that produce "file not found" errors.
|
||||
The fix: `notify_emby_scan()` is called automatically after every deletion. Triggers
|
||||
Emby's "Clean Missing Files" task immediately.
|
||||
|
||||
**No Safety Net on Deletion Size**
|
||||
A misconfigured root path — pointing cleanup at the wrong directory — means the API
|
||||
returns zero tracked files for a root that actually contains thousands. Every file walks
|
||||
as an orphan. Everything gets deleted. This is the catastrophic failure mode.
|
||||
The fix: `LIDARR/SONARR/RADARR_MAX_DELETE_GB` — if total deletion size exceeds the
|
||||
limit, the script stops and requires `--i-know-what-im-doing` to proceed. The flag name
|
||||
is long and annoying by design. It cannot be added by accident.
|
||||
|
||||
**Failed Downloads Accumulating Silently**
|
||||
Import failures and stalled downloads sit in arr queues indefinitely. Without
|
||||
intervention they occupy queue slots, block new searches, and the item never gets
|
||||
downloaded. Checking queues manually across three arrs is tedious.
|
||||
The fix: `arrs_failed_stalled_recovery.sh` inspects all arr queues, blocklists the bad
|
||||
release, removes it, and triggers a re-search. Runs daily.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
**Library Sync**
|
||||
`arr_sync.sh` — full-mesh arr library sync across all nodes. Every node syncs with every
|
||||
other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the
|
||||
actual files.
|
||||
|
||||
**Release Correction**
|
||||
`lidarr_release_fixer.sh` — reads MUSICBRAINZ_ALBUMID from FLAC and MP3 files, matches
|
||||
against Lidarr's known releases per album, switches monitored=true to the correct
|
||||
edition, queues RefreshArtist. Runs before lidarr_cleanup.sh daily. Handles both FLAC
|
||||
(vorbis comment block) and MP3 (ID3v2 TXXX frame).
|
||||
|
||||
**Orphan Cleanup**
|
||||
`lidarr_cleanup.sh`, `sonarr_cleanup.sh`, `radarr_cleanup.sh` — API-verified orphan
|
||||
removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven
|
||||
safety layers, automatic Emby notification after deletion.
|
||||
|
||||
**Database Hygiene**
|
||||
`radarr_tmdb_removed.sh`, `sonarr_tvdb_removed.sh` — remove entries that upstream
|
||||
databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in
|
||||
arrs and can never be monitored or downloaded. Most are announced-but-never-released
|
||||
entries. Files are kept by default — most have none.
|
||||
|
||||
**Library Recovery**
|
||||
`arrs_failed_stalled_recovery.sh` — detect and recover failed imports and stalled
|
||||
downloads across all arrs. Blocklists the bad release and triggers a re-search — hands-
|
||||
free overnight recovery.
|
||||
|
||||
**Library Enrichment**
|
||||
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
|
||||
fallback sources. Never overwrites existing files.
|
||||
|
||||
**Upgrade Propagation**
|
||||
`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr
|
||||
OnUpgrade webhooks. Continuous; started at array start. Writes to
|
||||
`/var/log/varaverk/upgrade_webhook.log`.
|
||||
`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded
|
||||
item folder to every remote node immediately, then triggers an arr library rescan on
|
||||
each remote so the upgraded file is accepted without triggering a redundant quality search.
|
||||
|
||||
**Discovery**
|
||||
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
|
||||
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
|
||||
Lidarr. 0–5 meaningful adds per week.
|
||||
`playback_aware_radarr_discovery.sh` — behavior-driven movie discovery. Scores recently
|
||||
watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr.
|
||||
`playback_aware_sonarr_discovery.sh` — behavior-driven TV discovery. Scores recently
|
||||
watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds
|
||||
top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ EXECUTION ORDER ━━━
|
||||
|
||||
**Daily via `daily_sync_maintenance.sh` (DAILY_MAINTENANCE_SCRIPTS):**
|
||||
|
||||
```
|
||||
1. lidarr_release_fixer.sh — correct wrong release editions before cleanup sees them
|
||||
2. lidarr_cleanup.sh — orphan removal (music)
|
||||
3. sonarr_cleanup.sh — orphan removal (TV)
|
||||
4. radarr_cleanup.sh — orphan removal (movies)
|
||||
5. lidarr_missing_art.sh — fetch missing artwork (HOST1 only)
|
||||
6. radarr_tmdb_removed.sh — remove TMDb-dropped movies
|
||||
7. sonarr_tvdb_removed.sh — remove TVDB-dropped series
|
||||
```
|
||||
|
||||
Note: `media_shares_permissions.sh` and `media_cleaner.sh` run before these from
|
||||
`Media/` — permissions and junk removal must complete first.
|
||||
|
||||
**Every 30 min + 4hr via orchestrators (CRITICAL/INTERMEDIATE_MAINTENANCE_SCRIPTS):**
|
||||
|
||||
```
|
||||
arrs_failed_stalled_recovery.sh — failed/stalled queue recovery
|
||||
arr_sync.sh — library sync across all nodes
|
||||
```
|
||||
|
||||
**Weekly via `weekly_sync_maintenance.sh` (WEEKLY_MAINTENANCE_SCRIPTS):**
|
||||
|
||||
```
|
||||
playback_aware_lidarr_discovery.sh — score play history → Last.fm similar → Lidarr
|
||||
playback_aware_radarr_discovery.sh — score watch history → TMDB recommendations → Radarr
|
||||
playback_aware_sonarr_discovery.sh — score episode history → TMDB TV → Sonarr
|
||||
```
|
||||
|
||||
**Continuous (started by `array_started.sh`):**
|
||||
|
||||
```
|
||||
start_webhook_listener.sh — Node.js webhook server; dispatches upgrade_webhook_handler.sh
|
||||
```
|
||||
|
||||
**On every arr upgrade (triggered by webhook):**
|
||||
|
||||
```
|
||||
upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan
|
||||
```
|
||||
|
||||
**Why release fixer before cleanup:** the fixer corrects Lidarr's release selection so
|
||||
files get imported. If cleanup ran first, a correctable album could accumulate age toward
|
||||
the orphan threshold before the fixer had a chance to fix it.
|
||||
|
||||
**Why arr_sync before rsync:** once arrs agree on what to track, rsync spreads the actual
|
||||
files. An upgrade on one node — new tracked path, old path no longer in API — gets
|
||||
cleaned by arr_cleanup on all nodes after the next sync cycle.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOST AWARENESS ━━━
|
||||
|
||||
Scripts run on both servers via `detect_hosts()`, which aliases all `HOST*_` prefixed vars
|
||||
to their unprefixed names at runtime. No manual `HOST1`/`HOST2` comparisons exist in any
|
||||
script.
|
||||
|
||||
`arr_sync.sh` keeps all arr databases in bidirectional union — either server can download
|
||||
to any share. Arr cleanup uses the union model: a file is only an orphan if the arr on
|
||||
this host doesn't have it indexed. Arr scripts check the aliased URL — if empty (arr not
|
||||
configured on this host), they exit cleanly with no action.
|
||||
|
||||
`lidarr_release_fixer.sh` and `lidarr_missing_art.sh` exit cleanly on hosts without
|
||||
Lidarr configured — no HOST1_LIDARR_URL means nothing runs.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|--------------|
|
||||
| `arr_sync.sh` | Full-mesh arr library sync — all nodes track the same content | Every 4hr + weekly before rsync |
|
||||
| `lidarr_release_fixer.sh` | Fix wrong MusicBrainz release editions so files get imported | Daily before lidarr_cleanup |
|
||||
| `lidarr_cleanup.sh` | Delete orphaned music files not tracked by Lidarr | Daily |
|
||||
| `sonarr_cleanup.sh` | Delete orphaned TV files not tracked by Sonarr | Daily |
|
||||
| `radarr_cleanup.sh` | Delete orphaned movie files not tracked by Radarr | Daily |
|
||||
| `arrs_failed_stalled_recovery.sh` | Auto-recover failed imports and stalled downloads | Every 30 min / daily |
|
||||
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Daily (HOST1 only) |
|
||||
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Daily |
|
||||
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Daily |
|
||||
| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous |
|
||||
| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade |
|
||||
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
|
||||
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
|
||||
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
Every 4hr / weekly (arr sync before rsync):
|
||||
arr_sync.sh ──────────────── syncs tracked IDs across all nodes
|
||||
│ union model: any node adds → all nodes get it
|
||||
│
|
||||
└── then rsync spreads the actual files to all nodes
|
||||
└── then arr_cleanup removes orphans on all nodes (old paths, removed content)
|
||||
|
||||
Daily maintenance window:
|
||||
[Media/media_shares_permissions.sh + media_cleaner.sh run first — from Media/]
|
||||
│
|
||||
▼
|
||||
lidarr_release_fixer.sh ────── reads MBID tag → switches release in Lidarr → RefreshArtist
|
||||
│ (corrected albums get imported before cleanup scans for orphans)
|
||||
▼
|
||||
lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans
|
||||
sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans
|
||||
radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans
|
||||
│
|
||||
└── each cleanup → notify_emby_scan() → Emby removes ghost entries
|
||||
|
||||
Daily recovery:
|
||||
arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search
|
||||
|
||||
Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
|
||||
playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr
|
||||
playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr
|
||||
playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr
|
||||
│
|
||||
└── each discovery script fires arr search immediately after successful add
|
||||
|
||||
Continuous (started by array_started.sh):
|
||||
start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
|
||||
│ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
|
||||
└── upgrade_webhook_handler.sh
|
||||
├── rsync upgraded folder → all remote nodes immediately
|
||||
└── trigger arr library rescan on each remote (accept new file, no quality search)
|
||||
|
||||
Ad-hoc enrichment:
|
||||
lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
|
||||
radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
|
||||
sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion
|
||||
```
|
||||
Reference in New Issue
Block a user