Varaverk/Media/README-Media.md

# ━━━━━ MEDIA ━━━━━

Library health, consistency, sync, and behavior-driven discovery for a multi-server arr
stack. Correct permissions so arrs can manage files. Junk removal so orphan detection
isn't confused by scene debris. Library sync so every node tracks the same content.
Orphan cleanup against live arr APIs so deleted content actually leaves disk. Emby
notified automatically after every deletion. Weekly discovery adds new music, movies,
and TV shows based on what you actually play — no manual browsing required.

> **These scripts permanently delete files.** The arr cleanup scripts are protected by
> multiple safety layers that must all pass before anything is touched — but dry runs and
> log review are still the right first step on any new system or after any configuration
> change. The testing procedure in Manual-Media.md exists for a reason.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━

**Files Owned by Root That Arrs Can't Touch**  
Download clients without explicit PUID/PGID write files owned by root. Arrs running as
`nobody:users` cannot rename, move, or delete them. Import fails. Upgrade attempts fail.
The failure is subtle — arr shows the file as managed but can't touch it. You only
discover this when an upgrade is requested and the old version refuses to delete.  
The fix: `media_shares_permissions.sh` applies correct ownership daily. Even if a
container is misconfigured, the window is at most 24 hours.

**Scene Junk Confusing Orphan Detection**  
Scene releases include `.sfv`, `.nfo`, `.rar`, `.sample` files alongside the actual media.
After extraction and import these are worthless — but they're not tracked by any arr.
They look like orphans. Processing them as orphans means the cleanup output is full of
noise, making it hard to spot actual orphaned media.  
The fix: `media_cleaner.sh` runs before any arr cleanup and removes all known junk
patterns first. By the time arr cleanup runs, every untracked file is actual media.

**Deleted Shows and Removed Albums Still on Disk**  
When you remove a series from Sonarr and the delete command fails — permission issue,
container wasn't running, path mismatch — the files stay permanently. Over years on an
active library this accumulates significantly.  
The fix: arr cleanup scripts query the live API for every tracked file path, walk the
disk, and delete anything absent from the API response that's old enough to be past the
import window.

**Emby Showing Ghost Entries After Cleanup**  
After arr cleanup deletes files, Emby still shows them until its next scheduled scan —
potentially hours later. Users see broken entries that produce "file not found" errors.  
The fix: `notify_emby_scan()` is called automatically after every deletion. Triggers
Emby's "Clean Missing Files" task immediately.

**No Safety Net on Deletion Size**  
A misconfigured root path — pointing cleanup at the wrong directory — means the API
returns zero tracked files for a root that actually contains thousands. Every file walks
as an orphan. Everything gets deleted. This is the catastrophic failure mode.  
The fix: `LIDARR/SONARR/RADARR_MAX_DELETE_GB` — if total deletion size exceeds the
limit, the script stops and requires `--i-know-what-im-doing` to proceed. The flag name
is long and annoying by design. It cannot be added by accident.

---

## ━━━ WHAT THIS FOLDER DOES ━━━

Scripts divide into five functional areas:

**Library Foundation**  
`media_shares_permissions.sh` — normalize ownership and permissions daily. Runs first in
every maintenance window because arr cleanup depends on correct ownership to delete files.

**Junk Removal**  
`media_cleaner.sh` — remove scene debris and tool artifacts before orphan scan. Runs
second, before any arr cleanup, so orphan detection only encounters actual media files.

**Library Sync**  
`arr_sync.sh` — full-mesh arr library sync across all nodes. Every node syncs with every
other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the
actual files. The architectural shift from file-first sync to arr-first sync.

**Orphan Cleanup**  
`lidarr_cleanup.sh`, `sonarr_cleanup.sh`, `radarr_cleanup.sh` — API-verified orphan
removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven
safety layers, automatic Emby notification after deletion.

**Database Hygiene**  
`radarr_tmdb_removed.sh`, `sonarr_tvdb_removed.sh` — remove entries that upstream
databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in
arrs and can never be monitored or downloaded. Most are announced-but-never-released
entries. Files are kept by default — most have none.

**Library Enrichment**  
`arrs_failed_stalled_recovery.sh` — detect and recover failed imports and stalled
downloads across all arrs. Blocklists the bad release and triggers a re-search — hands-
free overnight recovery.  
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
fallback sources. Never overwrites existing files.

**Discovery**  
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
Lidarr. 0–5 meaningful adds per week.  
`playback_aware_radarr_discovery.sh` — behavior-driven movie discovery. Scores recently
watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr.  
`playback_aware_sonarr_discovery.sh` — behavior-driven TV discovery. Scores recently
watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds
top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds.

---

## ━━━ EXECUTION ORDER ━━━

Scripts run in multiple contexts — not all are part of the daily maintenance window:

**Daily via `media_management.sh` (MEDIA_MAINTENANCE_JOBS):**

```
1. media_shares_permissions.sh   — permissions first — arr cleanup depends on this
2. media_cleaner.sh anime        — junk before orphan scan
3. media_cleaner.sh media
4. lidarr_cleanup.sh             — after permissions + clean
5. sonarr_cleanup.sh
6. radarr_cleanup.sh
```

**Weekly arr sync (before rsync in weekly_sync_maintenance.sh):**

```
arr_sync.sh    — arrs agree on library → rsync then spreads the files
```

**Daily recovery (separate schedule — 5am or every 6hr):**

```
arrs_failed_stalled_recovery.sh
```

**Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS in master.conf):**

```
playback_aware_lidarr_discovery.sh   — score play history → Last.fm similar → add to Lidarr
playback_aware_radarr_discovery.sh   — score watch history → TMDB recommendations → add to Radarr
playback_aware_sonarr_discovery.sh   — score episode history → TMDB TV recommendations → add to Sonarr
```

**Ad-hoc or separate schedule:**

```
lidarr_missing_art.sh       — fetch missing artwork
radarr_tmdb_removed.sh      — weekly cleanup of TMDb-dropped entries
sonarr_tvdb_removed.sh      — weekly cleanup of TVDB-dropped entries
```

**Why permissions before everything else:** arr cleanup needs `nobody:users` ownership to
delete files. If a file is `root:root`, deletion fails silently — the file looks processed
but stays on disk.

**Why junk before arr cleanup:** junk files are not tracked by any arr — they look like
orphans. Removing them first means orphan detection only finds actual media. Cleaner
output, more accurate detection.

**Why arr_sync before rsync:** once arrs agree on what to track, rsync spreads the actual
files. An upgrade on one node — new tracked path, old path no longer in API — gets
cleaned by arr_cleanup on all nodes after the next sync cycle.

---

## ━━━ HOST AWARENESS ━━━

Scripts run on both servers but only operate on the shares and arrs that belong to that
server. `detect_hosts()` aliases all `HOST*_` prefixed vars to their unprefixed names.

```
HOST1 — source of truth for:        HOST2 — source of truth for:
  Movies    (Radarr)                   Anime_Movies  (his Radarr)
  Tv_Shows  (Sonarr)                   Anime_Shows   (his Sonarr)
  Music     (Lidarr)
```

Permissions and cleaner scripts run on both servers against their own shares. Arr cleanup
scripts check the aliased URL — if empty (arr not configured on this host), they exit
cleanly with no action. No manual `HOST1`/`HOST2` comparisons exist in any script.

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━

| Script | Role | When It Runs |
|--------|------|--------------|
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily via `media_management.sh` |
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily via `media_management.sh` |
| `arr_sync.sh` | Full-mesh arr library sync — all nodes track the same content | Weekly before rsync |
| `lidarr_cleanup.sh` | Delete orphaned music files not tracked by Lidarr | Daily via `media_management.sh` |
| `sonarr_cleanup.sh` | Delete orphaned TV files not tracked by Sonarr | Daily via `media_management.sh` |
| `radarr_cleanup.sh` | Delete orphaned movie files not tracked by Radarr | Daily via `media_management.sh` |
| `arrs_failed_stalled_recovery.sh` | Auto-recover failed imports and stalled downloads | Daily (5am or every 6hr) |
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Ad-hoc or separate schedule |
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Ad-hoc or weekly |
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Ad-hoc or weekly |
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly via `weekly_sync_maintenance.sh` |
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly via `weekly_sync_maintenance.sh` |
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly via `weekly_sync_maintenance.sh` |

---

## ━━━ HOW THE SCRIPTS RELATE ━━━

```
Weekly arr sync (before rsync):
  arr_sync.sh ──────────────── syncs tracked IDs across all nodes
        │                      union model: any node adds → all nodes get it
        │
        └── then rsync spreads the actual files to all nodes
        └── then arr_cleanup removes orphans on all nodes (old paths, removed content)

Daily maintenance window (media_management.sh):
  media_shares_permissions.sh
        │ (permissions correct — arr can now delete files)
        ▼
  media_cleaner.sh (anime + media)
        │ (junk removed — orphan scan finds only actual media)
        ▼
  lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans
  sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans
  radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans
        │
        └── each cleanup → notify_emby_scan() → Emby removes ghost entries

Daily recovery:
  arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search

Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
  playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr
  playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr
  playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr
        │
        └── each discovery script fires arr search immediately after successful add

Ad-hoc enrichment:
  lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
  radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
  sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion
```