Varaverk/Media/README-Media.md

━━━━━ MEDIA ━━━━━

Library health, consistency, sync, and behavior-driven discovery for a multi-server arr stack. Correct permissions so arrs can manage files. Junk removal so orphan detection isn't confused by scene debris. Library sync so every node tracks the same content. Orphan cleanup against live arr APIs so deleted content actually leaves disk. Emby notified automatically after every deletion. Watch state synced across Emby/Jellyfin every 30 minutes. Quality upgrades propagate to all nodes immediately via webhook. Weekly discovery adds new music, movies, and TV shows based on what you actually play.

These scripts permanently delete files. The arr cleanup scripts are protected by multiple safety layers that must all pass before anything is touched — but dry runs and log review are still the right first step on any new system or after any configuration change. The testing procedure in Manual-Media.md exists for a reason.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━

Files Owned by Root That Arrs Can't Touch
Download clients without explicit PUID/PGID write files owned by root. Arrs running as nobody:users cannot rename, move, or delete them. Import fails. Upgrade attempts fail. The failure is subtle — arr shows the file as managed but can't touch it. You only discover this when an upgrade is requested and the old version refuses to delete.
The fix: media_shares_permissions.sh applies correct ownership daily. Even if a container is misconfigured, the window is at most 24 hours.

Scene Junk Confusing Orphan Detection
Scene releases include .sfv, .nfo, .rar, .sample files alongside the actual media. After extraction and import these are worthless — but they're not tracked by any arr. They look like orphans. Processing them as orphans means the cleanup output is full of noise, making it hard to spot actual orphaned media.
The fix: media_cleaner.sh runs before any arr cleanup and removes all known junk patterns first. By the time arr cleanup runs, every untracked file is actual media.

Deleted Shows and Removed Albums Still on Disk
When you remove a series from Sonarr and the delete command fails — permission issue, container wasn't running, path mismatch — the files stay permanently. Over years on an active library this accumulates significantly.
The fix: arr cleanup scripts query the live API for every tracked file path, walk the disk, and delete anything absent from the API response that's old enough to be past the import window.

Emby Showing Ghost Entries After Cleanup
After arr cleanup deletes files, Emby still shows them until its next scheduled scan — potentially hours later. Users see broken entries that produce "file not found" errors.
The fix: notify_emby_scan() is called automatically after every deletion. Triggers Emby's "Clean Missing Files" task immediately.

No Safety Net on Deletion Size
A misconfigured root path — pointing cleanup at the wrong directory — means the API returns zero tracked files for a root that actually contains thousands. Every file walks as an orphan. Everything gets deleted. This is the catastrophic failure mode.
The fix: LIDARR/SONARR/RADARR_MAX_DELETE_GB — if total deletion size exceeds the limit, the script stops and requires --i-know-what-im-doing to proceed. The flag name is long and annoying by design. It cannot be added by accident.

---

━━━ WHAT THIS FOLDER DOES ━━━

Scripts divide into five functional areas:

Library Foundation
media_shares_permissions.sh — normalize ownership and permissions daily. Runs first in every maintenance window because arr cleanup depends on correct ownership to delete files.

Junk Removal
media_cleaner.sh — remove scene debris and tool artifacts before orphan scan. Runs second, before any arr cleanup, so orphan detection only encounters actual media files.

Library Sync
arr_sync.sh — full-mesh arr library sync across all nodes. Every node syncs with every other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the actual files. The architectural shift from file-first sync to arr-first sync.

Orphan Cleanup
lidarr_cleanup.sh, sonarr_cleanup.sh, radarr_cleanup.sh — API-verified orphan removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven safety layers, automatic Emby notification after deletion.

Database Hygiene
radarr_tmdb_removed.sh, sonarr_tvdb_removed.sh — remove entries that upstream databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in arrs and can never be monitored or downloaded. Most are announced-but-never-released entries. Files are kept by default — most have none.

Library Enrichment
arrs_failed_stalled_recovery.sh — detect and recover failed imports and stalled downloads across all arrs. Blocklists the bad release and triggers a re-search — hands- free overnight recovery.
lidarr_missing_art.sh — fetch missing album and artist artwork from fanart.tv and fallback sources. Never overwrites existing files.

Play State Sync
play_state_sync.sh — syncs watched/played state and resume positions across all configured Emby and Jellyfin servers. Newest timestamp wins. Runs every 30 minutes via critical_sync_maintenance.sh.

Upgrade Propagation
start_webhook_listener.sh — Node.js HTTP server that receives Sonarr/Radarr/Lidarr OnUpgrade webhooks. Continuous; started at array start. Writes to /var/log/varaverk/upgrade_webhook.log.
upgrade_webhook_handler.sh — triggered by the webhook listener. Pushes the upgraded item folder to every remote node immediately, then triggers an arr library rescan on each remote so the upgraded file is accepted without triggering a redundant quality search.

Discovery
playback_aware_lidarr_discovery.sh — behavior-driven music discovery. Scores your Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to Lidarr. 0–5 meaningful adds per week.
playback_aware_radarr_discovery.sh — behavior-driven movie discovery. Scores recently watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr.
playback_aware_sonarr_discovery.sh — behavior-driven TV discovery. Scores recently watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds.

---

━━━ EXECUTION ORDER ━━━

Scripts run in multiple contexts — not all are part of the daily maintenance window:

Daily via media_management.sh (MEDIA_MAINTENANCE_JOBS):

1. media_shares_permissions.sh   — permissions first — arr cleanup depends on this
2. media_cleaner.sh anime        — junk before orphan scan
3. media_cleaner.sh media
4. lidarr_cleanup.sh             — after permissions + clean
5. sonarr_cleanup.sh
6. radarr_cleanup.sh

Weekly arr sync (before rsync in weekly_sync_maintenance.sh):

arr_sync.sh    — arrs agree on library → rsync then spreads the files

Daily recovery (separate schedule — 5am or every 6hr):

arrs_failed_stalled_recovery.sh

Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS in master.conf):

playback_aware_lidarr_discovery.sh   — score play history → Last.fm similar → add to Lidarr
playback_aware_radarr_discovery.sh   — score watch history → TMDB recommendations → add to Radarr
playback_aware_sonarr_discovery.sh   — score episode history → TMDB TV recommendations → add to Sonarr

Every 30 min via critical_sync_maintenance.sh (CRITICAL_MAINTENANCE_SCRIPTS):

play_state_sync.sh          — sync watched/resume state across Emby + Jellyfin

Continuous (started by array_started.sh):

start_webhook_listener.sh   — Node.js webhook server; dispatch upgrade_webhook_handler.sh

On every arr upgrade (triggered by webhook):

upgrade_webhook_handler.sh  — push upgraded folder to all remote nodes + trigger arr rescan

Ad-hoc or separate schedule:

lidarr_missing_art.sh       — fetch missing artwork
radarr_tmdb_removed.sh      — weekly cleanup of TMDb-dropped entries
sonarr_tvdb_removed.sh      — weekly cleanup of TVDB-dropped entries

Why permissions before everything else: arr cleanup needs nobody:users ownership to delete files. If a file is root:root, deletion fails silently — the file looks processed but stays on disk.

Why junk before arr cleanup: junk files are not tracked by any arr — they look like orphans. Removing them first means orphan detection only finds actual media. Cleaner output, more accurate detection.

Why arr_sync before rsync: once arrs agree on what to track, rsync spreads the actual files. An upgrade on one node — new tracked path, old path no longer in API — gets cleaned by arr_cleanup on all nodes after the next sync cycle.

---

━━━ HOST AWARENESS ━━━

Scripts run on both servers via detect_hosts(), which aliases all HOST*_ prefixed vars to their unprefixed names at runtime. No manual HOST1/HOST2 comparisons exist in any script.

arr_sync.sh keeps all arr databases in bidirectional union — either server can download to any share. Arr cleanup uses the union model: a file is only an orphan if neither arr on either server has it indexed. Arr scripts check the aliased URL — if empty (arr not configured on this host), they exit cleanly with no action.

Permissions and cleaner scripts run locally against each server's own shares, defined in HOST*_MEDIA_PERMISSION_SHARES and HOST*_MEDIA_CLEAN_FOLDERS in host*.conf.

---

━━━ SCRIPTS IN THIS FOLDER ━━━

Script	Role	When It Runs
media_shares_permissions.sh	Apply nobody:users ownership + correct permissions to all media shares	Daily via media_management.sh
media_cleaner.sh	Remove junk files (two profiles: anime + media)	Daily via media_management.sh
arr_sync.sh	Full-mesh arr library sync — all nodes track the same content	Weekly before rsync
lidarr_cleanup.sh	Delete orphaned music files not tracked by Lidarr	Daily via media_management.sh
sonarr_cleanup.sh	Delete orphaned TV files not tracked by Sonarr	Daily via media_management.sh
radarr_cleanup.sh	Delete orphaned movie files not tracked by Radarr	Daily via media_management.sh
arrs_failed_stalled_recovery.sh	Auto-recover failed imports and stalled downloads	Daily (5am or every 6hr)
lidarr_missing_art.sh	Fetch missing album and artist artwork	Ad-hoc or separate schedule
radarr_tmdb_removed.sh	Remove movies dropped from TMDb	Ad-hoc or weekly
sonarr_tvdb_removed.sh	Remove series dropped from TVDB	Ad-hoc or weekly
play_state_sync.sh	Sync watched/played state + resume positions across Emby + Jellyfin	Every 30 min via critical_sync_maintenance.sh
start_webhook_listener.sh	Node.js webhook server — receive arr OnUpgrade and dispatch handler	Continuous (started by array_started.sh)
upgrade_webhook_handler.sh	Push upgraded item folder to remote nodes + trigger arr rescan	On each arr upgrade (via webhook)
playback_aware_lidarr_discovery.sh	Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr	Weekly via weekly_sync_maintenance.sh
playback_aware_radarr_discovery.sh	Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr	Weekly via weekly_sync_maintenance.sh
playback_aware_sonarr_discovery.sh	Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr	Weekly via weekly_sync_maintenance.sh

---

━━━ HOW THE SCRIPTS RELATE ━━━

Weekly arr sync (before rsync):
  arr_sync.sh ──────────────── syncs tracked IDs across all nodes
        │                      union model: any node adds → all nodes get it
        │
        └── then rsync spreads the actual files to all nodes
        └── then arr_cleanup removes orphans on all nodes (old paths, removed content)

Daily maintenance window (media_management.sh):
  media_shares_permissions.sh
        │ (permissions correct — arr can now delete files)
        ▼
  media_cleaner.sh (anime + media)
        │ (junk removed — orphan scan finds only actual media)
        ▼
  lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans
  sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans
  radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans
        │
        └── each cleanup → notify_emby_scan() → Emby removes ghost entries

Daily recovery:
  arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search

Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
  playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr
  playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr
  playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr
        │
        └── each discovery script fires arr search immediately after successful add

Every 30 min (critical_sync_maintenance.sh CRITICAL_MAINTENANCE_SCRIPTS):
  play_state_sync.sh ─── newest timestamp wins → watched/resume state synced
                          across all configured Emby + Jellyfin servers

Continuous (started by array_started.sh):
  start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
        │ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
        └── upgrade_webhook_handler.sh
              ├── rsync upgraded folder → all remote nodes immediately
              └── trigger arr library rescan on each remote (accept new file, no quality search)

Ad-hoc enrichment:
  lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
  radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
  sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion