diff --git a/Arrs_Stack/Manual-Arrs_Stack.md b/Arrs_Stack/Manual-Arrs_Stack.md new file mode 100644 index 0000000..777035b --- /dev/null +++ b/Arrs_Stack/Manual-Arrs_Stack.md @@ -0,0 +1,670 @@ +# ━━━━━ ARRS STACK — Manual ━━━━━ + +Config reference, procedures, operational workflows. +For overview see README-Arrs_Stack.md. For per-script detail see script headers. + +--- + +## ━━━ ARR CLEANUP — FILE CLASSIFICATION ━━━ + +Every file found on disk during an arr cleanup run falls into exactly one category: + +``` +TRACKED → arr API returned this exact path → leave it alone +PROTECTED → matches ARR_PROTECTED_PATTERNS → never delete +ORPHAN → media extension, not tracked, old enough → delete +JUNK → not a media extension, not protected → delete (any age) +RECENT → not tracked, under ARR_ORPHAN_AGE days → skip (may be mid-import) +``` + +**Why protected patterns are critical:** arrs generate artwork (`*.jpg`), metadata +(`*.nfo`), and subtitles/lyrics that do NOT appear in the tracked file API response. +Without protection, these would be classified as orphans and deleted — removing cover art +from every album, every movie poster, every TV show thumbnail. Requires a full rescan +to recover. Never remove artwork extensions from protected patterns. + +--- + +## ━━━ ARR CLEANUP — SAFETY LAYERS ━━━ + +All 7 layers must pass before any file is touched. There is no way to push through a +failed safety check without the explicit override flag. + +``` +1. Container running + healthy — a stopped container has an empty API +2. API reachable — no API = no tracked file list = everything looks orphaned +3. API version matches — major version must match tested version in master.conf +4. Item count > 0 — no artists/series/movies = something is wrong with DB +5. Tracked file count > 0 — empty response = everything would be deleted +6. Tracked count >= MIN_TRACKED_PCT — dramatic drop from last run = abort and alert +7. Deletion size < MAX_DELETE_GB — last line of defense against misconfigured root path +``` + +Layer 7 is the catastrophic failure prevention. A misconfigured root path — pointing +cleanup at the wrong directory — means the API returns zero tracked files for a root +that actually contains thousands. Everything walks as an orphan. Everything gets deleted. +`LIDARR/SONARR/RADARR_MAX_DELETE_GB` requires `--i-know-what-im-doing` to proceed past it. + +--- + +## ━━━ CONFIGURATION — master.conf ━━━ + +### Lidarr Cleanup Thresholds + +```bash +LIDARR_ORPHAN_AGE=7 # days — files newer than this are RECENT (mid-import window) +LIDARR_MIN_TRACKED_PCT=80 # abort if API returns < 80% of last known count +LIDARR_MAX_DELETE_GB=5 # require --i-know-what-im-doing above this +LIDARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan +LIDARR_VERSION_MAJOR=3 # expected Lidarr major version (API safety check) +LIDARR_LOCK_WARN_AGE=3600 # 1hr — large libraries take time, not stuck +LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma") +LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc") +LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count" # persistent baseline +ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db" # read by coffee report +``` + +--- + +### Lidarr Release Fixer + +```bash +LIDARR_RELEASE_FIXER_ENABLED=true # set false to disable without removing from job list +``` + +No additional thresholds — uses existing `LIDARR_URL`, `LIDARR_API_KEY`, +`LIDARR_MUSIC_ROOT`, `LIDARR_PATH_MAP`, and `LIDARR_VERSION_MAJOR` from host*.conf and +master.conf. Reads FLAC (vorbis comment block type 4) and MP3 (ID3v2 TXXX frame) tags. + +--- + +### Sonarr Cleanup Thresholds + +```bash +SONARR_ORPHAN_AGE=7 +SONARR_MAX_DELETE_GB=50 +SONARR_IMPORT_SCAN_TIMEOUT=600 +SONARR_VERSION_MAJOR=4 +SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov") +SONARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa") +``` + +Note: `*.ts` IS in extensions — transport stream is used for Live TV recordings tracked +by Sonarr. Orphaned `.ts` recordings should be cleaned like any other orphaned episode. + +--- + +### Radarr Cleanup Thresholds + +```bash +RADARR_ORPHAN_AGE=7 +RADARR_MAX_DELETE_GB=50 +RADARR_IMPORT_SCAN_TIMEOUT=600 +RADARR_VERSION_MAJOR=6 +RADARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "wmv" "mov") +RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa") +``` + +--- + +### Arr Sync + +```bash +ARR_SYNC_ENABLED=true +ARR_SYNC_BLOCKLIST="$DATA_DIR/arr_sync_blocklist.tsv" # tombstone file +ARR_SYNC_CONNECT_TIMEOUT=10 # SSH connect timeout in seconds +ARR_SYNC_API_TIMEOUT=60 # curl API call timeout in seconds +DOCKER_APPDATA_BASE=/mnt/user/appdata +ARR_SYNC_LIDARR_PORT=8686 +ARR_SYNC_SONARR_PORT=8989 +ARR_SYNC_RADARR_PORT=7878 +``` + +--- + +### Arr Recovery + +```bash +ARR_IMPORT_RECOVERY_AGE=6 # hours — items newer than this are skipped +SONARR_VERSION_MAJOR=4 +RADARR_VERSION_MAJOR=6 +LIDARR_VERSION_MAJOR=3 +ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db" # read by coffee report +``` + +--- + +### TMDb / TVDB Removed + +```bash +RADARR_DROPPED_ADD_EXCLUSION=true # add removed movies to Radarr import exclusion +SONARR_DROPPED_ADD_EXCLUSION=true # add removed series to Sonarr import exclusion +``` + +--- + +### Lidarr Missing Art + +```bash +FANART_API_KEY="your-fanart-tv-api-key" +LASTFM_API_KEY="your-lastfm-api-key" +LIDARR_ART_MIN_SIZE=5000 # minimum valid download size in bytes +LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs +LIDARR_ART_RETRIES=2 # download retry attempts per image +LIDARR_ART_SLEEP_BETWEEN=1 # seconds between fanart.tv API calls (rate limit) +``` + +--- + +### Lidarr Discovery + +```bash +LIDARR_DISCOVERY_THRESHOLD=70 # minimum score for Stage 1 seeds and Stage 2 adds +LIDARR_DISCOVERY_LOOKBACK_DAYS=7 # Emby play history window in days +LIDARR_DISCOVERY_MIN_PLAYS=3 # min plays before an artist is evaluated as a seed +LIDARR_DISCOVERY_MAX_ADDS=5 # max seeds (Stage 1) and max adds (Stage 2) per run +LIDARR_DISCOVERY_USER_CAP_PCT=35 # max % any one user contributes to play weight +LIDARR_DISCOVERY_REJECT_COOLDOWN=30 # days before re-evaluating a Stage 2 reject +LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db" +``` + +Requires `HOST*_LASTFM_API_KEY` in `host*.conf`. + +--- + +### Radarr Discovery + +```bash +RADARR_DISCOVERY_THRESHOLD=52 # minimum score to add a candidate +RADARR_DISCOVERY_LOOKBACK_DAYS=30 # Emby watch history window in days +RADARR_DISCOVERY_MAX_SEEDS=5 # max seed movies from Stage 1 +RADARR_DISCOVERY_MAX_ADDS=5 # max movies to add per run +RADARR_DISCOVERY_MIN_VOTE_COUNT=100 # min TMDB votes for a candidate +RADARR_DISCOVERY_MIN_RATING=60 # min TMDB vote_average × 10 (60 = 6.0/10) +RADARR_DISCOVERY_REJECT_COOLDOWN=60 # days before re-evaluating a rejected movie +RADARR_DISCOVERY_SEED_LIBRARIES=("Movies") +RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db" +``` + +Requires `HOST*_TMDB_API_KEY` in `host*.conf`. + +--- + +### Sonarr Discovery + +```bash +SONARR_DISCOVERY_THRESHOLD=52 # minimum score to add a candidate +SONARR_DISCOVERY_LOOKBACK_DAYS=14 # Emby episode history window in days +SONARR_DISCOVERY_MAX_SEEDS=5 # max seed series from Stage 1 +SONARR_DISCOVERY_MAX_ADDS=3 # max shows to add per run (TV is a larger commitment) +SONARR_DISCOVERY_MIN_VOTE_COUNT=50 # min TMDB votes for a candidate +SONARR_DISCOVERY_MIN_RATING=65 # min TMDB vote_average × 10 (65 = 6.5/10) +SONARR_DISCOVERY_REJECT_COOLDOWN=60 # days before re-evaluating a rejected show +SONARR_DISCOVERY_USER_EPISODE_CAP=8 # max episodes per user in seed scoring +SONARR_DISCOVERY_MONITOR_MODE="all" # "all" = all seasons monitored; "future" = upcoming only +SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db" +``` + +Requires `HOST*_TMDB_API_KEY` in `host*.conf`. + +> **MONITOR_MODE note:** Use `"all"` (default) to have Sonarr search all existing seasons +> after adding a show. `"future"` only marks upcoming seasons as monitored — shows where +> all seasons have already aired will appear unmonitored and Sonarr will not search for them. + +--- + +### Upgrade Webhook + +```bash +WEBHOOK_PORT=7821 # 0 = disable listener +WEBHOOK_SECRET="" # auto-generated on first start if empty +``` + +When `WEBHOOK_PORT=0`, `start_webhook_listener.sh` exits cleanly and no listener starts. +When `WEBHOOK_SECRET` is empty, a 32-byte hex secret is generated on first start and +written back to `master.conf`. Run `Tools/webhook_setup.sh` to register the URL in arrs. + +Webhook log: `/var/log/varaverk/upgrade_webhook.log` + +--- + +## ━━━ CONFIGURATION — host*.conf ━━━ + +```bash +# Arr connection details — must match arr settings exactly +HOST1_LIDARR_URL="http://192.168.50.2:8686" +HOST1_LIDARR_API_KEY="..." +HOST1_LIDARR_MUSIC_ROOT="/mnt/user/Music" +HOST1_LIDARR_PATH_MAP="" # container→host path translation if needed + +HOST1_SONARR_URL="http://192.168.50.2:8989" +HOST1_SONARR_API_KEY="..." +HOST1_SONARR_TV_ROOT="/mnt/user/Tv_Shows" +HOST1_SONARR_PATH_MAP="" + +HOST1_RADARR_URL="http://192.168.50.2:7878" +HOST1_RADARR_API_KEY="..." +HOST1_RADARR_MOVIES_ROOT="/mnt/user/Movies" +HOST1_RADARR_PATH_MAP="" + +HOST1_EMBY_URL="http://192.168.50.2:8096" +HOST1_EMBY_API_KEY="..." +``` + +> **LIDARR/SONARR/RADARR_MUSIC/TV/MOVIES_ROOT must exactly match the Root Folder path in +> the arr's own settings.** Arr UI → Settings → Media Management → Root Folders. +> A mismatch means every file on disk looks untracked — all appear as orphans. +> MAX_DELETE_GB is the only thing standing between a path mismatch and losing your library. + +--- + +## ━━━ SAFE TESTING PROCEDURE ━━━ + +> **The arr cleanup scripts permanently delete files.** There is no recycle bin, no undo. +> Follow this procedure on first use, after any root path change, after any API key change, +> and after any significant arr library change. + +### Step 1 — Dry Run With Full Logging + +```bash +lidarr_release_fixer.sh --dry-run --log +lidarr_cleanup.sh --dry-run --log +sonarr_cleanup.sh --dry-run --log +radarr_cleanup.sh --dry-run --log +``` + +### Step 2 — Review the Output + +``` +Release fixer: + Are the "would fix" albums expected? + → Wrong release UUIDs switching to correct ones is expected behaviour + → If everything is already correct, nothing to do + +Cleanup: + Are TRACKED files the ones you expect? + → Known arr-managed files should show as TRACKED + → If they show as ORPHAN, the root path is wrong — STOP + + Is the ORPHAN count reasonable? + → Healthy cleanup removes dozens to hundreds, not tens of thousands + → Large count = stop, investigate root path before proceeding + + Are PROTECTED patterns working? + → Artwork (*.jpg) and subtitles (*.srt) must show as PROTECTED + → If they show as ORPHAN, check PROTECTED_PATTERNS config + + Are RECENT files being correctly skipped? + → Files downloaded in the last 7 days should show as RECENT, not ORPHAN +``` + +### Step 3 — Check Numbers if Something Looks Wrong + +```bash +# Root path mismatch? Compare these: +# Lidarr UI: Settings → Media Management → Root Folders +# Sonarr UI: Settings → Media Management → Root Folders +# Radarr UI: Settings → Media Management → Root Folders +# Must exactly match LIDARR_MUSIC_ROOT / SONARR_TV_ROOT / RADARR_MOVIES_ROOT + +# Is the arr running? +docker ps | grep -E "Lidarr|Sonarr|Radarr" +``` + +### Step 4 — Run Live + +```bash +# Only after dry run review passes. +lidarr_release_fixer.sh +lidarr_cleanup.sh +sonarr_cleanup.sh +radarr_cleanup.sh +``` + +### Step 5 — Verify in Arr UI + +``` +Library count — should not have dropped significantly +Missing files — check if any monitored content shows as missing +Emby library — should show no ghost entries (notify_emby_scan handles this automatically) +``` + +--- + +## ━━━ PROCEDURES ━━━ + +### Adding a New Arr + +```bash +# 1. Copy radarr_cleanup.sh as template +cp Arrs_Stack/radarr_cleanup.sh Arrs_Stack/readarr_cleanup.sh + +# 2. Replace RADARR_ prefix with READARR_ throughout +# Update API endpoint, tracked file API path, extension list, protected patterns + +# 3. Add to host*.conf +HOST1_READARR_URL="http://192.168.50.2:8787" +HOST1_READARR_API_KEY="your-api-key" +HOST1_READARR_BOOKS_ROOT="/mnt/user/Books" + +# 4. Add thresholds to master.conf +READARR_ORPHAN_AGE=7 +READARR_MAX_DELETE_GB=50 +READARR_EXTENSIONS=("epub" "pdf" "mobi" "azw3" "cbz" "cbr") +READARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo") + +# 5. Add to DAILY_MAINTENANCE_SCRIPTS in master.conf +"Arrs_Stack/readarr_cleanup.sh" +``` + +Run `--dry-run --log` before scheduling. + +--- + +### Managing the Arr Sync Blocklist + +```bash +# Add item to blocklist (removes from all arrs + tombstones the ID) +arr_sync.sh --blocklist-add lidarr "reason" +arr_sync.sh --blocklist-add sonarr "reason" +arr_sync.sh --blocklist-add radarr "reason" + +# Remove from blocklist (un-tombstones the ID — does NOT re-add to arrs) +arr_sync.sh --blocklist-remove lidarr + +# View all blocklisted IDs +arr_sync.sh --blocklist-list +``` + +`--blocklist-add` is the only destructive operation — it simultaneously: +1. Writes the tombstone entry to the blocklist TSV file +2. Deletes the item from the local arr API (no file deletion) +3. SSHes each remote node and deletes from their arr API + +Files become orphans on all nodes — arr_cleanup removes them on the next run. + +--- + +## ━━━ TROUBLESHOOTING ━━━ + +### Arr Cleanup Deleting Files It Shouldn't + +``` +1. Check the protected patterns — artwork and subtitles must be listed + LIDARR_PROTECTED_PATTERNS / SONARR_PROTECTED_PATTERNS / RADARR_PROTECTED_PATTERNS + +2. Check the root path matches arr settings exactly + Run: lidarr_cleanup.sh --status (shows configured root path) + Compare: Lidarr UI → Settings → Media Management → Root Folders + +3. Check if files are truly orphaned + Run: lidarr_cleanup.sh --dry-run --log + Look for the specific file — verify it shows ORPHAN, not PROTECTED or TRACKED +``` + +### Lidarr Files Present But Not Importing + +``` +Symptom: album folder exists with properly tagged files, Lidarr shows 0 tracks imported. +Cause: Lidarr selected the wrong MusicBrainz release edition. The file's MBID doesn't + match Lidarr's foreignReleaseId, so track ID lookup fails. + +Fix: + lidarr_release_fixer.sh --dry-run --log + → Shows which albums would be corrected and which release UUIDs would change + + lidarr_release_fixer.sh + → Applies corrections and queues RefreshArtist for affected artists + +If the file MBID doesn't match any Lidarr release (shows "MBID not in list"): + → File was tagged from a source Lidarr doesn't know about (unofficial, re-tagged) + → Manual import or re-tag with correct MBID +``` + +### Arr Cleanup Aborting at Safety Layer 6 (Tracked Count Drop) + +``` +API returned far fewer tracked files than last run. +Possible causes: + - Arr database was recently rebuilt from scratch + - Large manual library removal + - Path map mismatch after arr migration + +If intentional (library intentionally reduced): + Delete LIDARR_TRACKED_COUNT_FILE to reset the baseline + Run cleanup once — it will establish a new baseline + +If unintentional: + Investigate before proceeding — the arr may have a problem +``` + +### Arr Sync Not Picking Up New Content + +``` +Is ARR_SYNC_ENABLED=true in master.conf? + +Can this host SSH to the remote without password? + → ssh -i [SSH_KEY] root@[remote-tailscale-ip] "hostname" + +Is the arr accessible on the remote? + → arr_sync.sh --status (shows each node's arr reachability) + → arr_sync.sh --log (verbose output per-node, per-arr) + +Is the item in the blocklist? + → arr_sync.sh --blocklist-list +``` + +### Emby Still Showing Ghost Entries After Cleanup + +``` +notify_emby_scan() is called automatically after every arr cleanup deletion. +If ghosts persist: + 1. Is Emby's API responding? + curl -s "http://[emby-ip]:8096/System/Info/Public" + + 2. Is EMBY_URL / EMBY_API_KEY correct in host*.conf? + Run: sonarr_cleanup.sh --status (shows Emby config) + + 3. Trigger manually in Emby: + Library → Manage Library → Clean Missing Files +``` + +--- + +## ━━━ OUTPUT TIERS ━━━ + +All scripts have two output levels controlled by `--log`. + +Without `--log`, each script processes silently and always concludes with a summary +block: identity, duration, counts (files removed, items added, arrs cleaned), and a +status line. Warnings and errors are always visible. + +With `--log`, per-item detail appears: individual titles being processed, API query +progress, per-node sync results, and per-file examination output. Use this when +debugging unexpected results or validating configuration before the first scheduled run. + +Dry-run output follows the same tiers — `--dry-run` alone shows the summary of what +would happen; `--dry-run --log` shows the full per-item preview list. + +--- + +## ━━━ FLAG REFERENCE ━━━ + +### lidarr_release_fixer.sh + +`lidarr_release_fixer.sh` +Read MUSICBRAINZ_ALBUMID from FLAC/MP3 files, correct Lidarr release selection, queue RefreshArtist. + +`lidarr_release_fixer.sh --dry-run` +Show which albums would be corrected and the before/after release UUIDs. No API writes. + +`lidarr_release_fixer.sh --status` +Show configuration, Lidarr URL, enabled state. + +`lidarr_release_fixer.sh --log` +Verbose per-album output — shows every album checked, not just those corrected. + +--- + +### lidarr_cleanup.sh / sonarr_cleanup.sh / radarr_cleanup.sh + +`[script] --dry-run --log` +Preview every classification decision. **Always run this first.** See Safe Testing Procedure. + +`[script]` +Live run — deletes confirmed orphans and junk, triggers Emby clean. + +`[script] --log` +Live run with verbose per-file output. + +`[script] --status` +Show configuration, API status, tracked file count, and last run stats. + +`[script] --i-know-what-im-doing` +Bypass the MAX_DELETE_GB size threshold. Required when deletion exceeds the configured +limit. Long flag name is intentional — cannot be added accidentally. + +`[script] --skip-age-check` +Bypass the ORPHAN_AGE age check. Deletes RECENT files too — files that are under the +age threshold. Use when you know recent downloads are actually orphans. + +`[script] --i-know-what-im-doing --skip-age-check` +**NUCLEAR MODE** — age check and size threshold both bypassed. Deletes on first pass. +Use when you want a clean one-pass wipe of everything the arr doesn't track. +No recovery possible after deletion. + +--- + +### arrs_failed_stalled_recovery.sh + +`arrs_failed_stalled_recovery.sh` +Check all configured arrs for failed imports and stalled downloads. Blocklist + remove + +re-search for each problem item. + +`arrs_failed_stalled_recovery.sh --dry-run` +Show what would be actioned per arr without making any changes. + +`arrs_failed_stalled_recovery.sh --status` +Show configuration, arr reachability, and last recovery stats. + +`arrs_failed_stalled_recovery.sh --log` +Verbose output per item per arr. + +--- + +### arr_sync.sh + +`arr_sync.sh` +Sync all arr types across all configured nodes. + +`arr_sync.sh --dry-run` +Show what would be added/removed on each node without making changes. + +`arr_sync.sh --status` +Show node configuration, arr reachability, and blocklist count. + +`arr_sync.sh --log` +Verbose per-node, per-arr output. + +`arr_sync.sh --blocklist-add [arr] [id] "[reason]"` +Remove item from all arrs and tombstone the ID. See Procedures above. + +`arr_sync.sh --blocklist-remove [arr] [id]` +Remove tombstone — does NOT re-add item to arrs. + +`arr_sync.sh --blocklist-list` +Show all tombstoned IDs. + +--- + +### radarr_tmdb_removed.sh / sonarr_tvdb_removed.sh + +`[script]` +Remove records for entries with status="deleted" (dropped from upstream database). +Files are kept. Import exclusion is added. + +`[script] --delete-files` +Also delete associated files from disk. Most dropped entries have no files — they were +announced movies/series that were never downloaded. + +`[script] --dry-run` +Preview what would be removed without making changes. + +`[script] --status` +Show arr connection status and current count of dropped entries. + +`[script] --log` +Verbose per-entry output. + +--- + +### lidarr_missing_art.sh + +`lidarr_missing_art.sh` +Fetch all missing album and artist artwork from fanart.tv and fallback sources. +Never overwrites existing files. + +`lidarr_missing_art.sh --dry-run` +Show what would be downloaded without writing any files. + +`lidarr_missing_art.sh --status` +Show configuration and API key status. + +`lidarr_missing_art.sh --log` +Verbose per-album, per-artist output. + +--- + +### playback_aware_lidarr_discovery.sh + +`playback_aware_lidarr_discovery.sh` +Score Emby play history, run Last.fm getSimilar on top artists, add candidates above +threshold to Lidarr. Triggers ArtistSearch immediately after each successful add. + +`playback_aware_lidarr_discovery.sh --dry-run` +Score and rank all Stage 1 seeds and Stage 2 candidates. No Lidarr API calls. No writes +to history file. Shows exactly what would be added and at what score. + +`playback_aware_lidarr_discovery.sh --status` +Show config values, history file path and size, and API key status. + +`playback_aware_lidarr_discovery.sh --log` +Verbose per-artist scoring output for both stages. + +--- + +### playback_aware_radarr_discovery.sh + +`playback_aware_radarr_discovery.sh` +Score recently watched Emby movies, run TMDB recommendations on seeds, add candidates +above threshold to Radarr. Triggers MoviesSearch immediately after each successful add. + +`playback_aware_radarr_discovery.sh --dry-run` +Score and rank all Stage 1 seeds and Stage 2 candidates. No Radarr API calls. No writes +to history file. + +`playback_aware_radarr_discovery.sh --status` +Show config values, history file path and size, and API key status. + +`playback_aware_radarr_discovery.sh --log` +Verbose per-movie scoring output for both stages. + +--- + +### playback_aware_sonarr_discovery.sh + +`playback_aware_sonarr_discovery.sh` +Score recently watched Emby series (weighted by user diversity), run TMDB TV +recommendations on seeds, add candidates above threshold to Sonarr. Triggers SeriesSearch +immediately after each successful add. + +`playback_aware_sonarr_discovery.sh --dry-run` +Score and rank all Stage 1 seeds and Stage 2 candidates. No Sonarr API calls. No writes +to history file. + +`playback_aware_sonarr_discovery.sh --status` +Show config values, history file path and size, and API key status. + +`playback_aware_sonarr_discovery.sh --log` +Verbose per-series scoring output — shows user diversity, recency, and volume scores per +seed; breadth, rating, and votes scores per candidate. diff --git a/Arrs_Stack/README-Arrs_Stack.md b/Arrs_Stack/README-Arrs_Stack.md new file mode 100644 index 0000000..7fe062f --- /dev/null +++ b/Arrs_Stack/README-Arrs_Stack.md @@ -0,0 +1,250 @@ +# ━━━━━ ARRS STACK ━━━━━ + +Lifecycle management for the arr suite (Lidarr, Sonarr, Radarr) across a multi-server +ecosystem. Library sync so every node tracks the same content. Orphan cleanup against live +arr APIs so deleted content actually leaves disk. Release correction so wrong MusicBrainz +editions don't silently block Lidarr imports. Emby notified automatically after every +deletion. Failed downloads recovered overnight. Artwork fetched continuously. Database +hygiene for dropped upstream entries. Quality upgrades propagate to all nodes immediately +via webhook. Weekly discovery adds new content based on what you actually play and watch. + +> **These scripts permanently delete files and modify arr databases.** Orphan cleanup is +> protected by multiple safety layers that must all pass before anything is touched — but +> dry runs and log review are still the right first step on any new system or after any +> configuration change. See Manual-Arrs_Stack.md for the safe testing procedure. + +--- + +## ━━━ THE PROBLEMS THAT BUILT THIS ━━━ + +**Deleted Shows and Removed Albums Still on Disk** +When you remove a series from Sonarr and the delete command fails — permission issue, +container wasn't running, path mismatch — the files stay permanently. Over years on an +active library this accumulates significantly. +The fix: arr cleanup scripts query the live API for every tracked file path, walk the +disk, and delete anything absent from the API response that's old enough to be past the +import window. + +**Wrong MusicBrainz Release Edition Silently Blocking Lidarr Imports** +Lidarr selects one specific release edition per album using a MusicBrainz release ID. +When it picks the wrong edition (Brazil CD instead of US CD, Japan Digital instead of +standard), the track IDs don't match what's embedded in the files. RescanFolders reports +"Importing 0 tracks" even with perfectly tagged, complete files present. No error — just +silence. +The fix: `lidarr_release_fixer.sh` reads the MUSICBRAINZ_ALBUMID tag from each file, +finds the matching release in Lidarr's known releases, switches the selection, and queues +a RefreshArtist. Runs before cleanup so corrected albums are imported before the orphan +scan ever sees them. + +**Emby Showing Ghost Entries After Cleanup** +After arr cleanup deletes files, Emby still shows them until its next scheduled scan — +potentially hours later. Users see broken entries that produce "file not found" errors. +The fix: `notify_emby_scan()` is called automatically after every deletion. Triggers +Emby's "Clean Missing Files" task immediately. + +**No Safety Net on Deletion Size** +A misconfigured root path — pointing cleanup at the wrong directory — means the API +returns zero tracked files for a root that actually contains thousands. Every file walks +as an orphan. Everything gets deleted. This is the catastrophic failure mode. +The fix: `LIDARR/SONARR/RADARR_MAX_DELETE_GB` — if total deletion size exceeds the +limit, the script stops and requires `--i-know-what-im-doing` to proceed. The flag name +is long and annoying by design. It cannot be added by accident. + +**Failed Downloads Accumulating Silently** +Import failures and stalled downloads sit in arr queues indefinitely. Without +intervention they occupy queue slots, block new searches, and the item never gets +downloaded. Checking queues manually across three arrs is tedious. +The fix: `arrs_failed_stalled_recovery.sh` inspects all arr queues, blocklists the bad +release, removes it, and triggers a re-search. Runs daily. + +--- + +## ━━━ WHAT THIS FOLDER DOES ━━━ + +**Library Sync** +`arr_sync.sh` — full-mesh arr library sync across all nodes. Every node syncs with every +other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the +actual files. + +**Release Correction** +`lidarr_release_fixer.sh` — reads MUSICBRAINZ_ALBUMID from FLAC and MP3 files, matches +against Lidarr's known releases per album, switches monitored=true to the correct +edition, queues RefreshArtist. Runs before lidarr_cleanup.sh daily. Handles both FLAC +(vorbis comment block) and MP3 (ID3v2 TXXX frame). + +**Orphan Cleanup** +`lidarr_cleanup.sh`, `sonarr_cleanup.sh`, `radarr_cleanup.sh` — API-verified orphan +removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven +safety layers, automatic Emby notification after deletion. + +**Database Hygiene** +`radarr_tmdb_removed.sh`, `sonarr_tvdb_removed.sh` — remove entries that upstream +databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in +arrs and can never be monitored or downloaded. Most are announced-but-never-released +entries. Files are kept by default — most have none. + +**Library Recovery** +`arrs_failed_stalled_recovery.sh` — detect and recover failed imports and stalled +downloads across all arrs. Blocklists the bad release and triggers a re-search — hands- +free overnight recovery. + +**Library Enrichment** +`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and +fallback sources. Never overwrites existing files. + +**Upgrade Propagation** +`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr +OnUpgrade webhooks. Continuous; started at array start. Writes to +`/var/log/varaverk/upgrade_webhook.log`. +`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded +item folder to every remote node immediately, then triggers an arr library rescan on +each remote so the upgraded file is accepted without triggering a redundant quality search. + +**Discovery** +`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your +Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to +Lidarr. 0–5 meaningful adds per week. +`playback_aware_radarr_discovery.sh` — behavior-driven movie discovery. Scores recently +watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr. +`playback_aware_sonarr_discovery.sh` — behavior-driven TV discovery. Scores recently +watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds +top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds. + +--- + +## ━━━ EXECUTION ORDER ━━━ + +**Daily via `daily_sync_maintenance.sh` (DAILY_MAINTENANCE_SCRIPTS):** + +``` +1. lidarr_release_fixer.sh — correct wrong release editions before cleanup sees them +2. lidarr_cleanup.sh — orphan removal (music) +3. sonarr_cleanup.sh — orphan removal (TV) +4. radarr_cleanup.sh — orphan removal (movies) +5. lidarr_missing_art.sh — fetch missing artwork (HOST1 only) +6. radarr_tmdb_removed.sh — remove TMDb-dropped movies +7. sonarr_tvdb_removed.sh — remove TVDB-dropped series +``` + +Note: `media_shares_permissions.sh` and `media_cleaner.sh` run before these from +`Media/` — permissions and junk removal must complete first. + +**Every 30 min + 4hr via orchestrators (CRITICAL/INTERMEDIATE_MAINTENANCE_SCRIPTS):** + +``` +arrs_failed_stalled_recovery.sh — failed/stalled queue recovery +arr_sync.sh — library sync across all nodes +``` + +**Weekly via `weekly_sync_maintenance.sh` (WEEKLY_MAINTENANCE_SCRIPTS):** + +``` +playback_aware_lidarr_discovery.sh — score play history → Last.fm similar → Lidarr +playback_aware_radarr_discovery.sh — score watch history → TMDB recommendations → Radarr +playback_aware_sonarr_discovery.sh — score episode history → TMDB TV → Sonarr +``` + +**Continuous (started by `array_started.sh`):** + +``` +start_webhook_listener.sh — Node.js webhook server; dispatches upgrade_webhook_handler.sh +``` + +**On every arr upgrade (triggered by webhook):** + +``` +upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan +``` + +**Why release fixer before cleanup:** the fixer corrects Lidarr's release selection so +files get imported. If cleanup ran first, a correctable album could accumulate age toward +the orphan threshold before the fixer had a chance to fix it. + +**Why arr_sync before rsync:** once arrs agree on what to track, rsync spreads the actual +files. An upgrade on one node — new tracked path, old path no longer in API — gets +cleaned by arr_cleanup on all nodes after the next sync cycle. + +--- + +## ━━━ HOST AWARENESS ━━━ + +Scripts run on both servers via `detect_hosts()`, which aliases all `HOST*_` prefixed vars +to their unprefixed names at runtime. No manual `HOST1`/`HOST2` comparisons exist in any +script. + +`arr_sync.sh` keeps all arr databases in bidirectional union — either server can download +to any share. Arr cleanup uses the union model: a file is only an orphan if the arr on +this host doesn't have it indexed. Arr scripts check the aliased URL — if empty (arr not +configured on this host), they exit cleanly with no action. + +`lidarr_release_fixer.sh` and `lidarr_missing_art.sh` exit cleanly on hosts without +Lidarr configured — no HOST1_LIDARR_URL means nothing runs. + +--- + +## ━━━ SCRIPTS IN THIS FOLDER ━━━ + +| Script | Role | When It Runs | +|--------|------|--------------| +| `arr_sync.sh` | Full-mesh arr library sync — all nodes track the same content | Every 4hr + weekly before rsync | +| `lidarr_release_fixer.sh` | Fix wrong MusicBrainz release editions so files get imported | Daily before lidarr_cleanup | +| `lidarr_cleanup.sh` | Delete orphaned music files not tracked by Lidarr | Daily | +| `sonarr_cleanup.sh` | Delete orphaned TV files not tracked by Sonarr | Daily | +| `radarr_cleanup.sh` | Delete orphaned movie files not tracked by Radarr | Daily | +| `arrs_failed_stalled_recovery.sh` | Auto-recover failed imports and stalled downloads | Every 30 min / daily | +| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Daily (HOST1 only) | +| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Daily | +| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Daily | +| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous | +| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade | +| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly | +| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly | +| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly | + +--- + +## ━━━ HOW THE SCRIPTS RELATE ━━━ + +``` +Every 4hr / weekly (arr sync before rsync): + arr_sync.sh ──────────────── syncs tracked IDs across all nodes + │ union model: any node adds → all nodes get it + │ + └── then rsync spreads the actual files to all nodes + └── then arr_cleanup removes orphans on all nodes (old paths, removed content) + +Daily maintenance window: + [Media/media_shares_permissions.sh + media_cleaner.sh run first — from Media/] + │ + ▼ + lidarr_release_fixer.sh ────── reads MBID tag → switches release in Lidarr → RefreshArtist + │ (corrected albums get imported before cleanup scans for orphans) + ▼ + lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans + sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans + radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans + │ + └── each cleanup → notify_emby_scan() → Emby removes ghost entries + +Daily recovery: + arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search + +Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS): + playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr + playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr + playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr + │ + └── each discovery script fires arr search immediately after successful add + +Continuous (started by array_started.sh): + start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT + │ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET + └── upgrade_webhook_handler.sh + ├── rsync upgraded folder → all remote nodes immediately + └── trigger arr library rescan on each remote (accept new file, no quality search) + +Ad-hoc enrichment: + lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv + radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion + sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion +``` diff --git a/Media/Manual-Media.md b/Media/Manual-Media.md index 27c0add..8693cbc 100644 --- a/Media/Manual-Media.md +++ b/Media/Manual-Media.md @@ -1,8 +1,11 @@ # ━━━━━ MEDIA — Manual ━━━━━ -Config reference, procedures, operational workflows. +Config reference, procedures, operational workflows for Media/ scripts. For overview see README-Media.md. For per-script detail see script headers. +For arr stack config and procedures (cleanup, release fixer, sync, discovery) see +`Arrs_Stack/Manual-Arrs_Stack.md`. + --- ## ━━━ PERMISSIONS MODEL ━━━ @@ -32,48 +35,6 @@ Once fixed, this script corrects 0 files per run — it becomes a pure daily fai --- -## ━━━ ARR CLEANUP — FILE CLASSIFICATION ━━━ - -Every file found on disk during an arr cleanup run falls into exactly one category: - -``` -TRACKED → arr API returned this exact path → leave it alone -PROTECTED → matches ARR_PROTECTED_PATTERNS → never delete -ORPHAN → media extension, not tracked, old enough → delete -JUNK → not a media extension, not protected → delete (any age) -RECENT → not tracked, under ARR_ORPHAN_AGE days → skip (may be mid-import) -``` - -**Why protected patterns are critical:** arrs generate artwork (`*.jpg`), metadata -(`*.nfo`), and subtitles/lyrics that do NOT appear in the tracked file API response. -Without protection, these would be classified as orphans and deleted — removing cover art -from every album, every movie poster, every TV show thumbnail. Requires a full rescan -to recover. Never remove artwork extensions from protected patterns. - ---- - -## ━━━ ARR CLEANUP — SAFETY LAYERS ━━━ - -All 7 layers must pass before any file is touched. There is no way to push through a -failed safety check without the explicit override flag. - -``` -1. Container running + healthy — a stopped container has an empty API -2. API reachable — no API = no tracked file list = everything looks orphaned -3. API version matches — major version must match tested version in master.conf -4. Item count > 0 — no artists/series/movies = something is wrong with DB -5. Tracked file count > 0 — empty response = everything would be deleted -6. Tracked count >= MIN_TRACKED_PCT — dramatic drop from last run = abort and alert -7. Deletion size < MAX_DELETE_GB — last line of defense against misconfigured root path -``` - -Layer 7 is the catastrophic failure prevention. A misconfigured root path — pointing -cleanup at the wrong directory — means the API returns zero tracked files for a root -that actually contains thousands. Everything walks as an orphan. Everything gets deleted. -`LIDARR/SONARR/RADARR_MAX_DELETE_GB` requires `--i-know-what-im-doing` to proceed past it. - ---- - ## ━━━ CONFIGURATION — master.conf ━━━ ### Permissions @@ -113,167 +74,15 @@ MEDIA_FILE_PATTERNS=( ) ``` -> **DO NOT add patterns that match media you want to keep:** -> `*.mkv *.mp4 *.avi *.m4v` — video files -> `*.flac *.mp3 *.m4a` — audio files -> `*.srt *.sub *.ass` — subtitle files (Bazarr managed) -> `*.jpg *.png` — artwork +> **DO NOT add patterns that match media you want to keep:** +> `*.mkv *.mp4 *.avi *.m4v` — video files +> `*.flac *.mp3 *.m4a` — audio files +> `*.srt *.sub *.ass` — subtitle files (Bazarr managed) +> `*.jpg *.png` — artwork > Always use `--dry-run` when adding new patterns. --- -### Lidarr Cleanup Thresholds - -```bash -LIDARR_ORPHAN_AGE=7 # days — files newer than this are RECENT (mid-import window) -LIDARR_MIN_TRACKED_PCT=80 # abort if API returns < 80% of last known count -LIDARR_MAX_DELETE_GB=50 # require --i-know-what-im-doing above this -LIDARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan -LIDARR_VERSION_MAJOR=3 # expected Lidarr major version (API safety check) -LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma") -LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc") -LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count" # persistent baseline -ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db" # read by coffee report -``` - ---- - -### Sonarr Cleanup Thresholds - -```bash -SONARR_ORPHAN_AGE=7 -SONARR_MAX_DELETE_GB=50 -SONARR_IMPORT_SCAN_TIMEOUT=600 -SONARR_VERSION_MAJOR=4 -SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov") -SONARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa") -``` - -Note: `*.ts` IS in extensions — transport stream is used for Live TV recordings tracked -by Sonarr. Orphaned `.ts` recordings should be cleaned like any other orphaned episode. - ---- - -### Radarr Cleanup Thresholds - -```bash -RADARR_ORPHAN_AGE=7 -RADARR_MAX_DELETE_GB=50 -RADARR_IMPORT_SCAN_TIMEOUT=600 -RADARR_VERSION_MAJOR=6 -RADARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "wmv" "mov") -RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa") -``` - ---- - -### Arr Sync - -```bash -ARR_SYNC_ENABLED=true -ARR_SYNC_BLOCKLIST="$DATA_DIR/arr_sync_blocklist.tsv" # tombstone file -ARR_SYNC_CONNECT_TIMEOUT=10 # SSH connect timeout in seconds -ARR_SYNC_API_TIMEOUT=60 # curl API call timeout in seconds -DOCKER_APPDATA_BASE=/mnt/user/appdata -ARR_SYNC_LIDARR_PORT=8686 -ARR_SYNC_SONARR_PORT=8989 -ARR_SYNC_RADARR_PORT=7878 -``` - ---- - -### Arr Recovery - -```bash -ARR_IMPORT_RECOVERY_AGE=6 # hours — items newer than this are skipped -SONARR_VERSION_MAJOR=4 -RADARR_VERSION_MAJOR=6 -LIDARR_VERSION_MAJOR=3 -ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db" # read by coffee report -``` - ---- - -### TMDb / TVDB Removed - -```bash -RADARR_DROPPED_ADD_EXCLUSION=true # add removed movies to Radarr import exclusion -SONARR_DROPPED_ADD_EXCLUSION=true # add removed series to Sonarr import exclusion -``` - ---- - -### Lidarr Missing Art - -```bash -FANART_API_KEY="your-fanart-tv-api-key" -LASTFM_API_KEY="your-lastfm-api-key" -LIDARR_ART_MIN_SIZE=5000 # minimum valid download size in bytes -LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs -LIDARR_ART_RETRIES=2 # download retry attempts per image -LIDARR_ART_SLEEP_BETWEEN=1 # seconds between fanart.tv API calls (rate limit) -``` - ---- - -### Lidarr Discovery - -```bash -LIDARR_DISCOVERY_THRESHOLD=70 # minimum score for Stage 1 seeds and Stage 2 adds -LIDARR_DISCOVERY_LOOKBACK_DAYS=7 # Emby play history window in days -LIDARR_DISCOVERY_MIN_PLAYS=3 # min plays before an artist is evaluated as a seed -LIDARR_DISCOVERY_MAX_ADDS=5 # max seeds (Stage 1) and max adds (Stage 2) per run -LIDARR_DISCOVERY_USER_CAP_PCT=35 # max % any one user contributes to play weight -LIDARR_DISCOVERY_REJECT_COOLDOWN=30 # days before re-evaluating a Stage 2 reject -LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db" -``` - -Requires `HOST*_LASTFM_API_KEY` in `host*.conf`. - ---- - -### Radarr Discovery - -```bash -RADARR_DISCOVERY_THRESHOLD=52 # minimum score to add a candidate -RADARR_DISCOVERY_LOOKBACK_DAYS=30 # Emby watch history window in days -RADARR_DISCOVERY_MAX_SEEDS=5 # max seed movies from Stage 1 -RADARR_DISCOVERY_MAX_ADDS=5 # max movies to add per run -RADARR_DISCOVERY_MIN_VOTE_COUNT=100 # min TMDB votes for a candidate -RADARR_DISCOVERY_MIN_RATING=60 # min TMDB vote_average × 10 (60 = 6.0/10) -RADARR_DISCOVERY_REJECT_COOLDOWN=60 # days before re-evaluating a rejected movie -RADARR_DISCOVERY_SEED_LIBRARIES=("Movies") # Emby libraries to draw seeds from -RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db" -``` - -Requires `HOST*_TMDB_API_KEY` in `host*.conf`. - ---- - -### Sonarr Discovery - -```bash -SONARR_DISCOVERY_THRESHOLD=52 # minimum score to add a candidate -SONARR_DISCOVERY_LOOKBACK_DAYS=14 # Emby episode history window in days -SONARR_DISCOVERY_MAX_SEEDS=5 # max seed series from Stage 1 -SONARR_DISCOVERY_MAX_ADDS=3 # max shows to add per run (TV is a larger commitment) -SONARR_DISCOVERY_MIN_VOTE_COUNT=50 # min TMDB votes for a candidate -SONARR_DISCOVERY_MIN_RATING=65 # min TMDB vote_average × 10 (65 = 6.5/10) -SONARR_DISCOVERY_REJECT_COOLDOWN=60 # days before re-evaluating a rejected show -SONARR_DISCOVERY_USER_EPISODE_CAP=8 # max episodes per user in seed scoring -SONARR_DISCOVERY_MONITOR_MODE="all" # "all" = all seasons monitored; "future" = upcoming only -SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db" -# SONARR_EMBY_LIBRARIES is shared with emby_to_sonarr_sync — see Orchestrator Job Order -``` - -Requires `HOST*_TMDB_API_KEY` in `host*.conf`. - -> **MONITOR_MODE note:** Use `"all"` (default) to have Sonarr search all existing seasons -> after adding a show. `"future"` only marks upcoming seasons as monitored — shows where -> all seasons have already aired will appear unmonitored and Sonarr will not search for them. - ---- - ### Play State Sync ```bash @@ -293,41 +102,8 @@ HOST1_JELLYFIN_API_KEY="" --- -### Upgrade Webhook - -```bash -# master.conf -WEBHOOK_PORT=7821 # 0 = disable listener -WEBHOOK_SECRET="" # auto-generated on first start if empty -``` - -When `WEBHOOK_PORT=0`, `start_webhook_listener.sh` exits cleanly and no listener starts. -When `WEBHOOK_SECRET` is empty, a 32-byte hex secret is generated on first start and -written back to `master.conf`. Run `Tools/webhook_setup.sh` to register the URL in arrs. - -Webhook log: `/var/log/varaverk/upgrade_webhook.log` - ---- - -### Orchestrator Job Order - -```bash -MEDIA_MAINTENANCE_JOBS=( - "Media/media_shares_permissions.sh" # 1. permissions — always first - "Media/media_cleaner.sh anime" # 2. junk removal — before orphan scan - "Media/media_cleaner.sh media" # 3. - "Arrs_Stack/lidarr_cleanup.sh" # 4. arr cleanup — after permissions + clean - "Arrs_Stack/sonarr_cleanup.sh" # 5. - "Arrs_Stack/radarr_cleanup.sh" # 6. -) -``` - ---- - ## ━━━ CONFIGURATION — host*.conf ━━━ -### host1.conf - ```bash # Shares this server applies permissions to HOST1_MEDIA_PERMISSION_SHARES=( @@ -356,224 +132,6 @@ HOST1_MEDIA_CLEAN_FOLDERS=( "/mnt/user/stand-up_comedy" "/mnt/user/Tv_Shows" ) - -# Arr connection details — must match arr settings exactly -HOST1_LIDARR_URL="http://192.168.50.2:8686" -HOST1_LIDARR_API_KEY="..." -HOST1_LIDARR_MUSIC_ROOT="/mnt/user/Music" -HOST1_LIDARR_PATH_MAP="" # container→host path translation if needed - -HOST1_SONARR_URL="http://192.168.50.2:8989" -HOST1_SONARR_API_KEY="..." -HOST1_SONARR_TV_ROOT="/mnt/user/Tv_Shows" -HOST1_SONARR_PATH_MAP="" - -HOST1_RADARR_URL="http://192.168.50.2:7878" -HOST1_RADARR_API_KEY="..." -HOST1_RADARR_MOVIES_ROOT="/mnt/user/Movies" -HOST1_RADARR_PATH_MAP="" - -HOST1_EMBY_URL="http://192.168.50.2:8096" -HOST1_EMBY_API_KEY="..." -``` - -> **LIDARR/SONARR/RADARR_MUSIC/TV/MOVIES_ROOT must exactly match the Root Folder path in -> the arr's own settings.** Arr UI → Settings → Media Management → Root Folders. -> A mismatch means every file on disk looks untracked — all appear as orphans. -> MAX_DELETE_GB is the only thing standing between a path mismatch and losing your library. - ---- - -## ━━━ SAFE TESTING PROCEDURE ━━━ - -> **The arr cleanup scripts permanently delete files.** There is no recycle bin, no undo. -> Follow this procedure on first use, after any root path change, after any API key change, -> and after any significant arr library change. - -### Step 1 — Dry Run With Full Logging - -```bash -lidarr_cleanup.sh --dry-run --log -sonarr_cleanup.sh --dry-run --log -radarr_cleanup.sh --dry-run --log -``` - -### Step 2 — Review the Output - -``` -Are TRACKED files the ones you expect? - → Known arr-managed files should show as TRACKED - → If they show as ORPHAN, the root path is wrong — STOP - -Is the ORPHAN count reasonable? - → Healthy cleanup removes dozens to hundreds, not tens of thousands - → Large count = stop, investigate root path before proceeding - -Are PROTECTED patterns working? - → Artwork (*.jpg) and subtitles (*.srt) must show as PROTECTED - → If they show as ORPHAN, check PROTECTED_PATTERNS config - -Are RECENT files being correctly skipped? - → Files downloaded in the last 7 days should show as RECENT, not ORPHAN -``` - -### Step 3 — Check Numbers if Something Looks Wrong - -```bash -# Root path mismatch? Compare these: -# Lidarr UI: Settings → Media Management → Root Folders -# Sonarr UI: Settings → Media Management → Root Folders -# Radarr UI: Settings → Media Management → Root Folders -# Must exactly match LIDARR_MUSIC_ROOT / SONARR_TV_ROOT / RADARR_MOVIES_ROOT - -# Is the arr running? -docker ps | grep -E "Lidarr|Sonarr|Radarr" - -# Library scan not complete? -# Trigger manual scan in arr UI and wait for completion -``` - -### Step 4 — Run Live - -```bash -# Only after dry run review passes. -lidarr_cleanup.sh -sonarr_cleanup.sh -radarr_cleanup.sh -``` - -### Step 5 — Verify in Arr UI - -``` -Library count — should not have dropped significantly - healthy cleanup removes a small number, not a large percentage -Missing files — check if any monitored content shows as missing -Emby library — should show no ghost entries (notify_emby_scan handles this automatically) -``` - ---- - -## ━━━ PROCEDURES ━━━ - -### Adding a New Arr - -```bash -# 1. Copy radarr_cleanup.sh as template -cp radarr_cleanup.sh readarr_cleanup.sh - -# 2. Replace RADARR_ prefix with READARR_ throughout -# Update API endpoint, tracked file API path, extension list, protected patterns - -# 3. Add to host*.conf -HOST1_READARR_URL="http://192.168.50.2:8787" -HOST1_READARR_API_KEY="your-api-key" -HOST1_READARR_BOOKS_ROOT="/mnt/user/Books" - -# 4. Add thresholds to master.conf -READARR_ORPHAN_AGE=7 -READARR_MAX_DELETE_GB=50 -READARR_EXTENSIONS=("epub" "pdf" "mobi" "azw3" "cbz" "cbr") -READARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo") - -# 5. Add to MEDIA_MAINTENANCE_JOBS in master.conf -MEDIA_MAINTENANCE_JOBS=( - ...existing jobs... - "Media/readarr_cleanup.sh" -) -``` - -media_management.sh picks it up automatically. No orchestrator changes needed. -Run `--dry-run --log` before scheduling. - ---- - -### Managing the Arr Sync Blocklist - -```bash -# Add item to blocklist (removes from all arrs + tombstones the ID) -arr_sync.sh --blocklist-add lidarr "reason" -arr_sync.sh --blocklist-add sonarr "reason" -arr_sync.sh --blocklist-add radarr "reason" - -# Remove from blocklist (un-tombstones the ID — does NOT re-add to arrs) -arr_sync.sh --blocklist-remove lidarr - -# View all blocklisted IDs -arr_sync.sh --blocklist-list -``` - -`--blocklist-add` is the only destructive operation — it simultaneously: -1. Writes the tombstone entry to the blocklist TSV file -2. Deletes the item from the local arr API (no file deletion) -3. SSHes each remote node and deletes from their arr API - -Files become orphans on all nodes — arr_cleanup removes them on the next run. - ---- - -## ━━━ TROUBLESHOOTING ━━━ - -### Arr Cleanup Deleting Files It Shouldn't - -``` -1. Check the protected patterns — artwork and subtitles must be listed - LIDARR_PROTECTED_PATTERNS / SONARR_PROTECTED_PATTERNS / RADARR_PROTECTED_PATTERNS - -2. Check the root path matches arr settings exactly - Run: lidarr_cleanup.sh --status (shows configured root path) - Compare: Lidarr UI → Settings → Media Management → Root Folders - -3. Check if files are truly orphaned - Run: lidarr_cleanup.sh --dry-run --log - Look for the specific file — verify it shows ORPHAN, not PROTECTED or TRACKED -``` - -### Arr Cleanup Aborting at Safety Layer 6 (Tracked Count Drop) - -``` -API returned far fewer tracked files than last run. -Possible causes: - - Arr database was recently rebuilt from scratch - - Large manual library removal - - Path map mismatch after arr migration - -If intentional (library intentionally reduced): - Delete LIDARR_TRACKED_COUNT_FILE to reset the baseline - Run cleanup once — it will establish a new baseline - -If unintentional: - Investigate before proceeding — the arr may have a problem -``` - -### Arr Sync Not Picking Up New Content - -``` -Is ARR_SYNC_ENABLED=true in master.conf? - -Can this host SSH to the remote without password? - → ssh -i [SSH_KEY] root@[remote-tailscale-ip] "hostname" - -Is the arr accessible on the remote? - → arr_sync.sh --status (shows each node's arr reachability) - → arr_sync.sh --log (verbose output per-node, per-arr) - -Is the item in the blocklist? - → arr_sync.sh --blocklist-list -``` - -### Emby Still Showing Ghost Entries After Cleanup - -``` -notify_emby_scan() is called automatically after every arr cleanup deletion. -If ghosts persist: - 1. Is Emby's API responding? - curl -s "http://[emby-ip]:8096/System/Info/Public" - - 2. Is EMBY_URL / EMBY_API_KEY correct in host*.conf? - Run: sonarr_cleanup.sh --status (shows Emby config) - - 3. Trigger manually in Emby: - Library → Manage Library → Clean Missing Files ``` --- @@ -582,16 +140,11 @@ If ghosts persist: All scripts have two output levels controlled by `--log`. -Without `--log`, each script processes silently and always concludes with a summary -block: identity, duration, counts (files removed, items added, arrs cleaned), and a -status line. Warnings and errors are always visible. +Without `--log`, each script processes silently and concludes with a summary block. +Warnings and errors are always visible. -With `--log`, per-item detail appears: individual titles being processed, API query -progress, per-node sync results, and per-file examination output. Use this when -debugging unexpected results or validating configuration before the first scheduled run. - -Dry-run output follows the same tiers — `--dry-run` alone shows the summary of what -would happen; `--dry-run --log` shows the full per-item preview list. +With `--log`, per-item detail appears — individual shares being processed, file counts, +per-server sync results. --- @@ -599,18 +152,18 @@ would happen; `--dry-run --log` shows the full per-item preview list. ### media_shares_permissions.sh -`media_shares_permissions.sh` +`media_shares_permissions.sh` Apply correct ownership and permissions to all configured media shares. Safe to run manually at any time — idempotent, only changes what's wrong. -`media_shares_permissions.sh --dry-run` +`media_shares_permissions.sh --dry-run` Show how many files and directories would be corrected per share. If unexpectedly large, check container PUID/PGID settings first (PUID=99 PGID=100). -`media_shares_permissions.sh --status` +`media_shares_permissions.sh --status` Show configured share list and ownership of the share roots. -`media_shares_permissions.sh --log` +`media_shares_permissions.sh --log` Show ownership correction count per share and per file (verbose). > On large libraries this runs 20-30 minutes. This is expected — millions of files with @@ -620,204 +173,40 @@ Show ownership correction count per share and per file (verbose). ### media_cleaner.sh -`media_cleaner.sh anime` +`media_cleaner.sh anime` Remove junk files from anime share folders using ANIME_FILE_PATTERNS. -`media_cleaner.sh media` +`media_cleaner.sh media` Remove junk files from media share folders using MEDIA_FILE_PATTERNS. -`media_cleaner.sh [profile] --dry-run` +`media_cleaner.sh [profile] --dry-run` Show what would be deleted without removing anything. Always run first when adding new patterns or folders. -`media_cleaner.sh [profile] --status` +`media_cleaner.sh [profile] --status` Show folder list and file patterns for the profile. -`media_cleaner.sh [profile] --log` +`media_cleaner.sh [profile] --log` Show every file examined, not just those removed. --- -### lidarr_cleanup.sh / sonarr_cleanup.sh / radarr_cleanup.sh - -`[script] --dry-run --log` -Preview every classification decision. **Always run this first.** See Safe Testing Procedure. - -`[script]` -Live run — deletes confirmed orphans and junk, triggers Emby clean. - -`[script] --log` -Live run with verbose per-file output. - -`[script] --status` -Show configuration, API status, tracked file count, and last run stats. - -`[script] --i-know-what-im-doing` -Bypass the MAX_DELETE_GB size threshold. Required when deletion exceeds the configured -limit. Long flag name is intentional — cannot be added accidentally. - -`[script] --skip-strike-list` -Bypass the ORPHAN_AGE age check. Deletes RECENT files too — files that are under the -age threshold. Use when you know recent downloads are actually orphans. - -`[script] --i-know-what-im-doing --skip-strike-list` -**NUCLEAR MODE** — age check and size threshold both bypassed. Deletes on first pass. -Use when you want a clean one-pass wipe of everything the arr doesn't track. -No recovery possible after deletion. - ---- - ### play_state_sync.sh -`play_state_sync.sh` +`play_state_sync.sh` Sync played/unplayed state and resume positions from local Emby to the remote Emby. -Only items played within PLAY_SYNC_DAYS are synced. +Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced. -`play_state_sync.sh --full` -Ignore PLAY_SYNC_DAYS — sync all played items regardless of age. May be slow on large -libraries. Use after a new Emby install or database restore to rebuild full play history. +`play_state_sync.sh --full` +Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be +slow on large libraries. Use after a new Emby install or database restore to rebuild +full play history. -`play_state_sync.sh --dry-run` +`play_state_sync.sh --dry-run` Show what would be synced without writing any state. -`play_state_sync.sh --status` +`play_state_sync.sh --status` Show configured servers, reachability, and user counts. -`play_state_sync.sh --log` +`play_state_sync.sh --log` Verbose output — show each item comparison. - ---- - -### arrs_failed_stalled_recovery.sh - -`arrs_failed_stalled_recovery.sh` -Check all configured arrs for failed imports and stalled downloads. Blocklist + remove + -re-search for each problem item. - -`arrs_failed_stalled_recovery.sh --dry-run` -Show what would be actioned per arr without making any changes. - -`arrs_failed_stalled_recovery.sh --status` -Show configuration, arr reachability, and last recovery stats. - -`arrs_failed_stalled_recovery.sh --log` -Verbose output per item per arr. - ---- - -### arr_sync.sh - -`arr_sync.sh` -Sync all arr types across all configured nodes. - -`arr_sync.sh --dry-run` -Show what would be added/removed on each node without making changes. - -`arr_sync.sh --status` -Show node configuration, arr reachability, and blocklist count. - -`arr_sync.sh --log` -Verbose per-node, per-arr output. - -`arr_sync.sh --blocklist-add [arr] [id] "[reason]"` -Remove item from all arrs and tombstone the ID. See Procedures above. - -`arr_sync.sh --blocklist-remove [arr] [id]` -Remove tombstone — does NOT re-add item to arrs. - -`arr_sync.sh --blocklist-list` -Show all tombstoned IDs. - ---- - -### radarr_tmdb_removed.sh / sonarr_tvdb_removed.sh - -`[script]` -Remove records for entries with status="deleted" (dropped from upstream database). -Files are kept. Import exclusion is added. - -`[script] --delete-files` -Also delete associated files from disk. Most dropped entries have no files — they were -announced movies/series that were never downloaded. - -`[script] --dry-run` -Preview what would be removed without making changes. - -`[script] --status` -Show arr connection status and current count of dropped entries. - -`[script] --log` -Verbose per-entry output. - ---- - -### lidarr_missing_art.sh - -`lidarr_missing_art.sh` -Fetch all missing album and artist artwork from fanart.tv and fallback sources. -Never overwrites existing files. - -`lidarr_missing_art.sh --dry-run` -Show what would be downloaded without writing any files. - -`lidarr_missing_art.sh --status` -Show configuration and API key status. - -`lidarr_missing_art.sh --log` -Verbose per-album, per-artist output. - ---- - -### playback_aware_lidarr_discovery.sh - -`playback_aware_lidarr_discovery.sh` -Score Emby play history, run Last.fm getSimilar on top artists, add candidates above -threshold to Lidarr. Triggers ArtistSearch immediately after each successful add. - -`playback_aware_lidarr_discovery.sh --dry-run` -Score and rank all Stage 1 seeds and Stage 2 candidates. No Lidarr API calls. No writes -to history file. Shows exactly what would be added and at what score. - -`playback_aware_lidarr_discovery.sh --status` -Show config values, history file path and size, and API key status. - -`playback_aware_lidarr_discovery.sh --log` -Verbose per-artist scoring output for both stages. - ---- - -### playback_aware_radarr_discovery.sh - -`playback_aware_radarr_discovery.sh` -Score recently watched Emby movies, run TMDB recommendations on seeds, add candidates -above threshold to Radarr. Triggers MoviesSearch immediately after each successful add. - -`playback_aware_radarr_discovery.sh --dry-run` -Score and rank all Stage 1 seeds and Stage 2 candidates. No Radarr API calls. No writes -to history file. - -`playback_aware_radarr_discovery.sh --status` -Show config values, history file path and size, and API key status. - -`playback_aware_radarr_discovery.sh --log` -Verbose per-movie scoring output for both stages. - ---- - -### playback_aware_sonarr_discovery.sh - -`playback_aware_sonarr_discovery.sh` -Score recently watched Emby series (weighted by user diversity), run TMDB TV -recommendations on seeds, add candidates above threshold to Sonarr. Triggers SeriesSearch -immediately after each successful add. - -`playback_aware_sonarr_discovery.sh --dry-run` -Score and rank all Stage 1 seeds and Stage 2 candidates. No Sonarr API calls. No writes -to history file. - -`playback_aware_sonarr_discovery.sh --status` -Show config values, history file path and size, and API key status. - -`playback_aware_sonarr_discovery.sh --log` -Verbose per-series scoring output — shows user diversity, recency, and volume scores per -seed; breadth, rating, and votes scores per candidate. diff --git a/Media/README-Media.md b/Media/README-Media.md index 3f701b9..57ad3a1 100644 --- a/Media/README-Media.md +++ b/Media/README-Media.md @@ -1,181 +1,73 @@ # ━━━━━ MEDIA ━━━━━ -Library health, consistency, sync, and behavior-driven discovery for a multi-server arr -stack. Correct permissions so arrs can manage files. Junk removal so orphan detection -isn't confused by scene debris. Library sync so every node tracks the same content. -Orphan cleanup against live arr APIs so deleted content actually leaves disk. Emby -notified automatically after every deletion. Watch state synced across Emby/Jellyfin -every 30 minutes. Quality upgrades propagate to all nodes immediately via webhook. -Weekly discovery adds new music, movies, and TV shows based on what you actually play. +Foundation-level media library management — permissions, junk removal, and play state +sync. These three scripts run before and independently of arr stack operations. -> **These scripts permanently delete files.** The arr cleanup scripts are protected by -> multiple safety layers that must all pass before anything is touched — but dry runs and -> log review are still the right first step on any new system or after any configuration -> change. The testing procedure in Manual-Media.md exists for a reason. +For arr stack scripts (orphan cleanup, release fixer, sync, discovery, webhooks) see +`Arrs_Stack/README-Arrs_Stack.md`. --- -## ━━━ THE PROBLEM THAT BUILT THIS ━━━ +## ━━━ THE PROBLEMS THAT BUILT THIS ━━━ -**Files Owned by Root That Arrs Can't Touch** +**Files Owned by Root That Arrs Can't Touch** Download clients without explicit PUID/PGID write files owned by root. Arrs running as `nobody:users` cannot rename, move, or delete them. Import fails. Upgrade attempts fail. The failure is subtle — arr shows the file as managed but can't touch it. You only -discover this when an upgrade is requested and the old version refuses to delete. +discover this when an upgrade is requested and the old version refuses to delete. The fix: `media_shares_permissions.sh` applies correct ownership daily. Even if a container is misconfigured, the window is at most 24 hours. -**Scene Junk Confusing Orphan Detection** +**Scene Junk Confusing Orphan Detection** Scene releases include `.sfv`, `.nfo`, `.rar`, `.sample` files alongside the actual media. After extraction and import these are worthless — but they're not tracked by any arr. They look like orphans. Processing them as orphans means the cleanup output is full of -noise, making it hard to spot actual orphaned media. +noise, making it hard to spot actual orphaned media. The fix: `media_cleaner.sh` runs before any arr cleanup and removes all known junk patterns first. By the time arr cleanup runs, every untracked file is actual media. -**Deleted Shows and Removed Albums Still on Disk** -When you remove a series from Sonarr and the delete command fails — permission issue, -container wasn't running, path mismatch — the files stay permanently. Over years on an -active library this accumulates significantly. -The fix: arr cleanup scripts query the live API for every tracked file path, walk the -disk, and delete anything absent from the API response that's old enough to be past the -import window. - -**Emby Showing Ghost Entries After Cleanup** -After arr cleanup deletes files, Emby still shows them until its next scheduled scan — -potentially hours later. Users see broken entries that produce "file not found" errors. -The fix: `notify_emby_scan()` is called automatically after every deletion. Triggers -Emby's "Clean Missing Files" task immediately. - -**No Safety Net on Deletion Size** -A misconfigured root path — pointing cleanup at the wrong directory — means the API -returns zero tracked files for a root that actually contains thousands. Every file walks -as an orphan. Everything gets deleted. This is the catastrophic failure mode. -The fix: `LIDARR/SONARR/RADARR_MAX_DELETE_GB` — if total deletion size exceeds the -limit, the script stops and requires `--i-know-what-im-doing` to proceed. The flag name -is long and annoying by design. It cannot be added by accident. +**Watch State Diverging Across Servers** +With two Emby servers, played status and resume positions diverge — a film marked watched +on HOST1 shows as unwatched on HOST2. Two users on different servers get different +continue-watching rows. +The fix: `play_state_sync.sh` syncs watched/played state and resume positions every 30 +minutes. Newest timestamp wins. Both servers always reflect the same play history. --- ## ━━━ WHAT THIS FOLDER DOES ━━━ -Scripts divide into five functional areas: - -**Library Foundation** +**Library Foundation** `media_shares_permissions.sh` — normalize ownership and permissions daily. Runs first in every maintenance window because arr cleanup depends on correct ownership to delete files. -**Junk Removal** +**Junk Removal** `media_cleaner.sh` — remove scene debris and tool artifacts before orphan scan. Runs -second, before any arr cleanup, so orphan detection only encounters actual media files. +before any arr cleanup so orphan detection only encounters actual media files. -**Library Sync** -`arr_sync.sh` — full-mesh arr library sync across all nodes. Every node syncs with every -other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the -actual files. The architectural shift from file-first sync to arr-first sync. - -**Orphan Cleanup** -`lidarr_cleanup.sh`, `sonarr_cleanup.sh`, `radarr_cleanup.sh` — API-verified orphan -removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven -safety layers, automatic Emby notification after deletion. - -**Database Hygiene** -`radarr_tmdb_removed.sh`, `sonarr_tvdb_removed.sh` — remove entries that upstream -databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in -arrs and can never be monitored or downloaded. Most are announced-but-never-released -entries. Files are kept by default — most have none. - -**Library Enrichment** -`arrs_failed_stalled_recovery.sh` — detect and recover failed imports and stalled -downloads across all arrs. Blocklists the bad release and triggers a re-search — hands- -free overnight recovery. -`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and -fallback sources. Never overwrites existing files. - -**Play State Sync** +**Play State Sync** `play_state_sync.sh` — syncs watched/played state and resume positions across all configured Emby and Jellyfin servers. Newest timestamp wins. Runs every 30 minutes via critical_sync_maintenance.sh. -**Upgrade Propagation** -`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr -OnUpgrade webhooks. Continuous; started at array start. Writes to -`/var/log/varaverk/upgrade_webhook.log`. -`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded -item folder to every remote node immediately, then triggers an arr library rescan on -each remote so the upgraded file is accepted without triggering a redundant quality search. - -**Discovery** -`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your -Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to -Lidarr. 0–5 meaningful adds per week. -`playback_aware_radarr_discovery.sh` — behavior-driven movie discovery. Scores recently -watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr. -`playback_aware_sonarr_discovery.sh` — behavior-driven TV discovery. Scores recently -watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds -top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds. - --- ## ━━━ EXECUTION ORDER ━━━ -Scripts run in multiple contexts — not all are part of the daily maintenance window: - -**Daily via `media_management.sh` (MEDIA_MAINTENANCE_JOBS):** +**Daily via `daily_sync_maintenance.sh` (DAILY_MAINTENANCE_SCRIPTS — runs first):** ``` 1. media_shares_permissions.sh — permissions first — arr cleanup depends on this 2. media_cleaner.sh anime — junk before orphan scan 3. media_cleaner.sh media -4. lidarr_cleanup.sh — after permissions + clean -5. sonarr_cleanup.sh -6. radarr_cleanup.sh ``` -**Weekly arr sync (before rsync in weekly_sync_maintenance.sh):** - -``` -arr_sync.sh — arrs agree on library → rsync then spreads the files -``` - -**Daily recovery (separate schedule — 5am or every 6hr):** - -``` -arrs_failed_stalled_recovery.sh -``` - -**Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS in master.conf):** - -``` -playback_aware_lidarr_discovery.sh — score play history → Last.fm similar → add to Lidarr -playback_aware_radarr_discovery.sh — score watch history → TMDB recommendations → add to Radarr -playback_aware_sonarr_discovery.sh — score episode history → TMDB TV recommendations → add to Sonarr -``` +These complete before any Arrs_Stack/ scripts run. **Every 30 min via `critical_sync_maintenance.sh` (CRITICAL_MAINTENANCE_SCRIPTS):** ``` -play_state_sync.sh — sync watched/resume state across Emby + Jellyfin -``` - -**Continuous (started by `array_started.sh`):** - -``` -start_webhook_listener.sh — Node.js webhook server; dispatch upgrade_webhook_handler.sh -``` - -**On every arr upgrade (triggered by webhook):** - -``` -upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan -``` - -**Ad-hoc or separate schedule:** - -``` -lidarr_missing_art.sh — fetch missing artwork -radarr_tmdb_removed.sh — weekly cleanup of TMDb-dropped entries -sonarr_tvdb_removed.sh — weekly cleanup of TVDB-dropped entries +play_state_sync.sh — sync watched/resume state across Emby + Jellyfin ``` **Why permissions before everything else:** arr cleanup needs `nobody:users` ownership to @@ -186,96 +78,25 @@ but stays on disk. orphans. Removing them first means orphan detection only finds actual media. Cleaner output, more accurate detection. -**Why arr_sync before rsync:** once arrs agree on what to track, rsync spreads the actual -files. An upgrade on one node — new tracked path, old path no longer in API — gets -cleaned by arr_cleanup on all nodes after the next sync cycle. - --- ## ━━━ HOST AWARENESS ━━━ Scripts run on both servers via `detect_hosts()`, which aliases all `HOST*_` prefixed vars -to their unprefixed names at runtime. No manual `HOST1`/`HOST2` comparisons exist in any script. - -arr_sync.sh keeps all arr databases in bidirectional union — either server can download -to any share. Arr cleanup uses the union model: a file is only an orphan if neither arr -on either server has it indexed. Arr scripts check the aliased URL — if empty (arr not -configured on this host), they exit cleanly with no action. +to their unprefixed names at runtime. Permissions and cleaner scripts run locally against each server's own shares, defined in `HOST*_MEDIA_PERMISSION_SHARES` and `HOST*_MEDIA_CLEAN_FOLDERS` in host*.conf. +`play_state_sync.sh` reads both servers' Emby/Jellyfin endpoints from host*.conf and +syncs between them. + --- ## ━━━ SCRIPTS IN THIS FOLDER ━━━ | Script | Role | When It Runs | |--------|------|--------------| -| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily via `media_management.sh` | -| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily via `media_management.sh` | -| `arr_sync.sh` | Full-mesh arr library sync — all nodes track the same content | Weekly before rsync | -| `lidarr_cleanup.sh` | Delete orphaned music files not tracked by Lidarr | Daily via `media_management.sh` | -| `sonarr_cleanup.sh` | Delete orphaned TV files not tracked by Sonarr | Daily via `media_management.sh` | -| `radarr_cleanup.sh` | Delete orphaned movie files not tracked by Radarr | Daily via `media_management.sh` | -| `arrs_failed_stalled_recovery.sh` | Auto-recover failed imports and stalled downloads | Daily (5am or every 6hr) | -| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Ad-hoc or separate schedule | -| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Ad-hoc or weekly | -| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Ad-hoc or weekly | -| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min via `critical_sync_maintenance.sh` | -| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous (started by `array_started.sh`) | -| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade (via webhook) | -| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly via `weekly_sync_maintenance.sh` | -| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly via `weekly_sync_maintenance.sh` | -| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly via `weekly_sync_maintenance.sh` | - ---- - -## ━━━ HOW THE SCRIPTS RELATE ━━━ - -``` -Weekly arr sync (before rsync): - arr_sync.sh ──────────────── syncs tracked IDs across all nodes - │ union model: any node adds → all nodes get it - │ - └── then rsync spreads the actual files to all nodes - └── then arr_cleanup removes orphans on all nodes (old paths, removed content) - -Daily maintenance window (media_management.sh): - media_shares_permissions.sh - │ (permissions correct — arr can now delete files) - ▼ - media_cleaner.sh (anime + media) - │ (junk removed — orphan scan finds only actual media) - ▼ - lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans - sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans - radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans - │ - └── each cleanup → notify_emby_scan() → Emby removes ghost entries - -Daily recovery: - arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search - -Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS): - playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr - playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr - playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr - │ - └── each discovery script fires arr search immediately after successful add - -Every 30 min (critical_sync_maintenance.sh CRITICAL_MAINTENANCE_SCRIPTS): - play_state_sync.sh ─── newest timestamp wins → watched/resume state synced - across all configured Emby + Jellyfin servers - -Continuous (started by array_started.sh): - start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT - │ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET - └── upgrade_webhook_handler.sh - ├── rsync upgraded folder → all remote nodes immediately - └── trigger arr library rescan on each remote (accept new file, no quality search) - -Ad-hoc enrichment: - lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv - radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion - sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion -``` +| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first | +| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup | +| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |