Split Media/ docs into Media/ and Arrs_Stack/ to match folder reorganization
Media README and Manual now cover only the 3 remaining scripts (permissions, cleaner, play_state_sync). Arrs_Stack README and Manual cover all arr stack scripts including lidarr_release_fixer. Fixed stale --skip-strike-list reference in flag docs.
This commit is contained in:
Gmer4Lfe
+29
-208
@@ -1,181 +1,73 @@
|
||||
# ━━━━━ MEDIA ━━━━━
|
||||
|
||||
Library health, consistency, sync, and behavior-driven discovery for a multi-server arr
|
||||
stack. Correct permissions so arrs can manage files. Junk removal so orphan detection
|
||||
isn't confused by scene debris. Library sync so every node tracks the same content.
|
||||
Orphan cleanup against live arr APIs so deleted content actually leaves disk. Emby
|
||||
notified automatically after every deletion. Watch state synced across Emby/Jellyfin
|
||||
every 30 minutes. Quality upgrades propagate to all nodes immediately via webhook.
|
||||
Weekly discovery adds new music, movies, and TV shows based on what you actually play.
|
||||
Foundation-level media library management — permissions, junk removal, and play state
|
||||
sync. These three scripts run before and independently of arr stack operations.
|
||||
|
||||
> **These scripts permanently delete files.** The arr cleanup scripts are protected by
|
||||
> multiple safety layers that must all pass before anything is touched — but dry runs and
|
||||
> log review are still the right first step on any new system or after any configuration
|
||||
> change. The testing procedure in Manual-Media.md exists for a reason.
|
||||
For arr stack scripts (orphan cleanup, release fixer, sync, discovery, webhooks) see
|
||||
`Arrs_Stack/README-Arrs_Stack.md`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
## ━━━ THE PROBLEMS THAT BUILT THIS ━━━
|
||||
|
||||
**Files Owned by Root That Arrs Can't Touch**
|
||||
**Files Owned by Root That Arrs Can't Touch**
|
||||
Download clients without explicit PUID/PGID write files owned by root. Arrs running as
|
||||
`nobody:users` cannot rename, move, or delete them. Import fails. Upgrade attempts fail.
|
||||
The failure is subtle — arr shows the file as managed but can't touch it. You only
|
||||
discover this when an upgrade is requested and the old version refuses to delete.
|
||||
discover this when an upgrade is requested and the old version refuses to delete.
|
||||
The fix: `media_shares_permissions.sh` applies correct ownership daily. Even if a
|
||||
container is misconfigured, the window is at most 24 hours.
|
||||
|
||||
**Scene Junk Confusing Orphan Detection**
|
||||
**Scene Junk Confusing Orphan Detection**
|
||||
Scene releases include `.sfv`, `.nfo`, `.rar`, `.sample` files alongside the actual media.
|
||||
After extraction and import these are worthless — but they're not tracked by any arr.
|
||||
They look like orphans. Processing them as orphans means the cleanup output is full of
|
||||
noise, making it hard to spot actual orphaned media.
|
||||
noise, making it hard to spot actual orphaned media.
|
||||
The fix: `media_cleaner.sh` runs before any arr cleanup and removes all known junk
|
||||
patterns first. By the time arr cleanup runs, every untracked file is actual media.
|
||||
|
||||
**Deleted Shows and Removed Albums Still on Disk**
|
||||
When you remove a series from Sonarr and the delete command fails — permission issue,
|
||||
container wasn't running, path mismatch — the files stay permanently. Over years on an
|
||||
active library this accumulates significantly.
|
||||
The fix: arr cleanup scripts query the live API for every tracked file path, walk the
|
||||
disk, and delete anything absent from the API response that's old enough to be past the
|
||||
import window.
|
||||
|
||||
**Emby Showing Ghost Entries After Cleanup**
|
||||
After arr cleanup deletes files, Emby still shows them until its next scheduled scan —
|
||||
potentially hours later. Users see broken entries that produce "file not found" errors.
|
||||
The fix: `notify_emby_scan()` is called automatically after every deletion. Triggers
|
||||
Emby's "Clean Missing Files" task immediately.
|
||||
|
||||
**No Safety Net on Deletion Size**
|
||||
A misconfigured root path — pointing cleanup at the wrong directory — means the API
|
||||
returns zero tracked files for a root that actually contains thousands. Every file walks
|
||||
as an orphan. Everything gets deleted. This is the catastrophic failure mode.
|
||||
The fix: `LIDARR/SONARR/RADARR_MAX_DELETE_GB` — if total deletion size exceeds the
|
||||
limit, the script stops and requires `--i-know-what-im-doing` to proceed. The flag name
|
||||
is long and annoying by design. It cannot be added by accident.
|
||||
**Watch State Diverging Across Servers**
|
||||
With two Emby servers, played status and resume positions diverge — a film marked watched
|
||||
on HOST1 shows as unwatched on HOST2. Two users on different servers get different
|
||||
continue-watching rows.
|
||||
The fix: `play_state_sync.sh` syncs watched/played state and resume positions every 30
|
||||
minutes. Newest timestamp wins. Both servers always reflect the same play history.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
Scripts divide into five functional areas:
|
||||
|
||||
**Library Foundation**
|
||||
**Library Foundation**
|
||||
`media_shares_permissions.sh` — normalize ownership and permissions daily. Runs first in
|
||||
every maintenance window because arr cleanup depends on correct ownership to delete files.
|
||||
|
||||
**Junk Removal**
|
||||
**Junk Removal**
|
||||
`media_cleaner.sh` — remove scene debris and tool artifacts before orphan scan. Runs
|
||||
second, before any arr cleanup, so orphan detection only encounters actual media files.
|
||||
before any arr cleanup so orphan detection only encounters actual media files.
|
||||
|
||||
**Library Sync**
|
||||
`arr_sync.sh` — full-mesh arr library sync across all nodes. Every node syncs with every
|
||||
other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the
|
||||
actual files. The architectural shift from file-first sync to arr-first sync.
|
||||
|
||||
**Orphan Cleanup**
|
||||
`lidarr_cleanup.sh`, `sonarr_cleanup.sh`, `radarr_cleanup.sh` — API-verified orphan
|
||||
removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven
|
||||
safety layers, automatic Emby notification after deletion.
|
||||
|
||||
**Database Hygiene**
|
||||
`radarr_tmdb_removed.sh`, `sonarr_tvdb_removed.sh` — remove entries that upstream
|
||||
databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in
|
||||
arrs and can never be monitored or downloaded. Most are announced-but-never-released
|
||||
entries. Files are kept by default — most have none.
|
||||
|
||||
**Library Enrichment**
|
||||
`arrs_failed_stalled_recovery.sh` — detect and recover failed imports and stalled
|
||||
downloads across all arrs. Blocklists the bad release and triggers a re-search — hands-
|
||||
free overnight recovery.
|
||||
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
|
||||
fallback sources. Never overwrites existing files.
|
||||
|
||||
**Play State Sync**
|
||||
**Play State Sync**
|
||||
`play_state_sync.sh` — syncs watched/played state and resume positions across all
|
||||
configured Emby and Jellyfin servers. Newest timestamp wins. Runs every 30 minutes
|
||||
via critical_sync_maintenance.sh.
|
||||
|
||||
**Upgrade Propagation**
|
||||
`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr
|
||||
OnUpgrade webhooks. Continuous; started at array start. Writes to
|
||||
`/var/log/varaverk/upgrade_webhook.log`.
|
||||
`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded
|
||||
item folder to every remote node immediately, then triggers an arr library rescan on
|
||||
each remote so the upgraded file is accepted without triggering a redundant quality search.
|
||||
|
||||
**Discovery**
|
||||
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
|
||||
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
|
||||
Lidarr. 0–5 meaningful adds per week.
|
||||
`playback_aware_radarr_discovery.sh` — behavior-driven movie discovery. Scores recently
|
||||
watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr.
|
||||
`playback_aware_sonarr_discovery.sh` — behavior-driven TV discovery. Scores recently
|
||||
watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds
|
||||
top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ EXECUTION ORDER ━━━
|
||||
|
||||
Scripts run in multiple contexts — not all are part of the daily maintenance window:
|
||||
|
||||
**Daily via `media_management.sh` (MEDIA_MAINTENANCE_JOBS):**
|
||||
**Daily via `daily_sync_maintenance.sh` (DAILY_MAINTENANCE_SCRIPTS — runs first):**
|
||||
|
||||
```
|
||||
1. media_shares_permissions.sh — permissions first — arr cleanup depends on this
|
||||
2. media_cleaner.sh anime — junk before orphan scan
|
||||
3. media_cleaner.sh media
|
||||
4. lidarr_cleanup.sh — after permissions + clean
|
||||
5. sonarr_cleanup.sh
|
||||
6. radarr_cleanup.sh
|
||||
```
|
||||
|
||||
**Weekly arr sync (before rsync in weekly_sync_maintenance.sh):**
|
||||
|
||||
```
|
||||
arr_sync.sh — arrs agree on library → rsync then spreads the files
|
||||
```
|
||||
|
||||
**Daily recovery (separate schedule — 5am or every 6hr):**
|
||||
|
||||
```
|
||||
arrs_failed_stalled_recovery.sh
|
||||
```
|
||||
|
||||
**Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS in master.conf):**
|
||||
|
||||
```
|
||||
playback_aware_lidarr_discovery.sh — score play history → Last.fm similar → add to Lidarr
|
||||
playback_aware_radarr_discovery.sh — score watch history → TMDB recommendations → add to Radarr
|
||||
playback_aware_sonarr_discovery.sh — score episode history → TMDB TV recommendations → add to Sonarr
|
||||
```
|
||||
These complete before any Arrs_Stack/ scripts run.
|
||||
|
||||
**Every 30 min via `critical_sync_maintenance.sh` (CRITICAL_MAINTENANCE_SCRIPTS):**
|
||||
|
||||
```
|
||||
play_state_sync.sh — sync watched/resume state across Emby + Jellyfin
|
||||
```
|
||||
|
||||
**Continuous (started by `array_started.sh`):**
|
||||
|
||||
```
|
||||
start_webhook_listener.sh — Node.js webhook server; dispatch upgrade_webhook_handler.sh
|
||||
```
|
||||
|
||||
**On every arr upgrade (triggered by webhook):**
|
||||
|
||||
```
|
||||
upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan
|
||||
```
|
||||
|
||||
**Ad-hoc or separate schedule:**
|
||||
|
||||
```
|
||||
lidarr_missing_art.sh — fetch missing artwork
|
||||
radarr_tmdb_removed.sh — weekly cleanup of TMDb-dropped entries
|
||||
sonarr_tvdb_removed.sh — weekly cleanup of TVDB-dropped entries
|
||||
play_state_sync.sh — sync watched/resume state across Emby + Jellyfin
|
||||
```
|
||||
|
||||
**Why permissions before everything else:** arr cleanup needs `nobody:users` ownership to
|
||||
@@ -186,96 +78,25 @@ but stays on disk.
|
||||
orphans. Removing them first means orphan detection only finds actual media. Cleaner
|
||||
output, more accurate detection.
|
||||
|
||||
**Why arr_sync before rsync:** once arrs agree on what to track, rsync spreads the actual
|
||||
files. An upgrade on one node — new tracked path, old path no longer in API — gets
|
||||
cleaned by arr_cleanup on all nodes after the next sync cycle.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOST AWARENESS ━━━
|
||||
|
||||
Scripts run on both servers via `detect_hosts()`, which aliases all `HOST*_` prefixed vars
|
||||
to their unprefixed names at runtime. No manual `HOST1`/`HOST2` comparisons exist in any script.
|
||||
|
||||
arr_sync.sh keeps all arr databases in bidirectional union — either server can download
|
||||
to any share. Arr cleanup uses the union model: a file is only an orphan if neither arr
|
||||
on either server has it indexed. Arr scripts check the aliased URL — if empty (arr not
|
||||
configured on this host), they exit cleanly with no action.
|
||||
to their unprefixed names at runtime.
|
||||
|
||||
Permissions and cleaner scripts run locally against each server's own shares, defined
|
||||
in `HOST*_MEDIA_PERMISSION_SHARES` and `HOST*_MEDIA_CLEAN_FOLDERS` in host*.conf.
|
||||
|
||||
`play_state_sync.sh` reads both servers' Emby/Jellyfin endpoints from host*.conf and
|
||||
syncs between them.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|--------------|
|
||||
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily via `media_management.sh` |
|
||||
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily via `media_management.sh` |
|
||||
| `arr_sync.sh` | Full-mesh arr library sync — all nodes track the same content | Weekly before rsync |
|
||||
| `lidarr_cleanup.sh` | Delete orphaned music files not tracked by Lidarr | Daily via `media_management.sh` |
|
||||
| `sonarr_cleanup.sh` | Delete orphaned TV files not tracked by Sonarr | Daily via `media_management.sh` |
|
||||
| `radarr_cleanup.sh` | Delete orphaned movie files not tracked by Radarr | Daily via `media_management.sh` |
|
||||
| `arrs_failed_stalled_recovery.sh` | Auto-recover failed imports and stalled downloads | Daily (5am or every 6hr) |
|
||||
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Ad-hoc or separate schedule |
|
||||
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Ad-hoc or weekly |
|
||||
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Ad-hoc or weekly |
|
||||
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min via `critical_sync_maintenance.sh` |
|
||||
| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous (started by `array_started.sh`) |
|
||||
| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade (via webhook) |
|
||||
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly via `weekly_sync_maintenance.sh` |
|
||||
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly via `weekly_sync_maintenance.sh` |
|
||||
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly via `weekly_sync_maintenance.sh` |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
Weekly arr sync (before rsync):
|
||||
arr_sync.sh ──────────────── syncs tracked IDs across all nodes
|
||||
│ union model: any node adds → all nodes get it
|
||||
│
|
||||
└── then rsync spreads the actual files to all nodes
|
||||
└── then arr_cleanup removes orphans on all nodes (old paths, removed content)
|
||||
|
||||
Daily maintenance window (media_management.sh):
|
||||
media_shares_permissions.sh
|
||||
│ (permissions correct — arr can now delete files)
|
||||
▼
|
||||
media_cleaner.sh (anime + media)
|
||||
│ (junk removed — orphan scan finds only actual media)
|
||||
▼
|
||||
lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans
|
||||
sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans
|
||||
radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans
|
||||
│
|
||||
└── each cleanup → notify_emby_scan() → Emby removes ghost entries
|
||||
|
||||
Daily recovery:
|
||||
arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search
|
||||
|
||||
Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
|
||||
playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr
|
||||
playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr
|
||||
playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr
|
||||
│
|
||||
└── each discovery script fires arr search immediately after successful add
|
||||
|
||||
Every 30 min (critical_sync_maintenance.sh CRITICAL_MAINTENANCE_SCRIPTS):
|
||||
play_state_sync.sh ─── newest timestamp wins → watched/resume state synced
|
||||
across all configured Emby + Jellyfin servers
|
||||
|
||||
Continuous (started by array_started.sh):
|
||||
start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
|
||||
│ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
|
||||
└── upgrade_webhook_handler.sh
|
||||
├── rsync upgraded folder → all remote nodes immediately
|
||||
└── trigger arr library rescan on each remote (accept new file, no quality search)
|
||||
|
||||
Ad-hoc enrichment:
|
||||
lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
|
||||
radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
|
||||
sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion
|
||||
```
|
||||
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
|
||||
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
|
||||
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
|
||||
|
||||
Reference in New Issue
Block a user