Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
75f2a4e3fd | ||
|
|
cf180c1179 |
@@ -0,0 +1,670 @@
|
|||||||
|
# ━━━━━ ARRS STACK — Manual ━━━━━
|
||||||
|
|
||||||
|
Config reference, procedures, operational workflows.
|
||||||
|
For overview see README-Arrs_Stack.md. For per-script detail see script headers.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ ARR CLEANUP — FILE CLASSIFICATION ━━━
|
||||||
|
|
||||||
|
Every file found on disk during an arr cleanup run falls into exactly one category:
|
||||||
|
|
||||||
|
```
|
||||||
|
TRACKED → arr API returned this exact path → leave it alone
|
||||||
|
PROTECTED → matches ARR_PROTECTED_PATTERNS → never delete
|
||||||
|
ORPHAN → media extension, not tracked, old enough → delete
|
||||||
|
JUNK → not a media extension, not protected → delete (any age)
|
||||||
|
RECENT → not tracked, under ARR_ORPHAN_AGE days → skip (may be mid-import)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why protected patterns are critical:** arrs generate artwork (`*.jpg`), metadata
|
||||||
|
(`*.nfo`), and subtitles/lyrics that do NOT appear in the tracked file API response.
|
||||||
|
Without protection, these would be classified as orphans and deleted — removing cover art
|
||||||
|
from every album, every movie poster, every TV show thumbnail. Requires a full rescan
|
||||||
|
to recover. Never remove artwork extensions from protected patterns.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ ARR CLEANUP — SAFETY LAYERS ━━━
|
||||||
|
|
||||||
|
All 7 layers must pass before any file is touched. There is no way to push through a
|
||||||
|
failed safety check without the explicit override flag.
|
||||||
|
|
||||||
|
```
|
||||||
|
1. Container running + healthy — a stopped container has an empty API
|
||||||
|
2. API reachable — no API = no tracked file list = everything looks orphaned
|
||||||
|
3. API version matches — major version must match tested version in master.conf
|
||||||
|
4. Item count > 0 — no artists/series/movies = something is wrong with DB
|
||||||
|
5. Tracked file count > 0 — empty response = everything would be deleted
|
||||||
|
6. Tracked count >= MIN_TRACKED_PCT — dramatic drop from last run = abort and alert
|
||||||
|
7. Deletion size < MAX_DELETE_GB — last line of defense against misconfigured root path
|
||||||
|
```
|
||||||
|
|
||||||
|
Layer 7 is the catastrophic failure prevention. A misconfigured root path — pointing
|
||||||
|
cleanup at the wrong directory — means the API returns zero tracked files for a root
|
||||||
|
that actually contains thousands. Everything walks as an orphan. Everything gets deleted.
|
||||||
|
`LIDARR/SONARR/RADARR_MAX_DELETE_GB` requires `--i-know-what-im-doing` to proceed past it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ CONFIGURATION — master.conf ━━━
|
||||||
|
|
||||||
|
### Lidarr Cleanup Thresholds
|
||||||
|
|
||||||
|
```bash
|
||||||
|
LIDARR_ORPHAN_AGE=7 # days — files newer than this are RECENT (mid-import window)
|
||||||
|
LIDARR_MIN_TRACKED_PCT=80 # abort if API returns < 80% of last known count
|
||||||
|
LIDARR_MAX_DELETE_GB=5 # require --i-know-what-im-doing above this
|
||||||
|
LIDARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan
|
||||||
|
LIDARR_VERSION_MAJOR=3 # expected Lidarr major version (API safety check)
|
||||||
|
LIDARR_LOCK_WARN_AGE=3600 # 1hr — large libraries take time, not stuck
|
||||||
|
LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
|
||||||
|
LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc")
|
||||||
|
LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count" # persistent baseline
|
||||||
|
ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db" # read by coffee report
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Lidarr Release Fixer
|
||||||
|
|
||||||
|
```bash
|
||||||
|
LIDARR_RELEASE_FIXER_ENABLED=true # set false to disable without removing from job list
|
||||||
|
```
|
||||||
|
|
||||||
|
No additional thresholds — uses existing `LIDARR_URL`, `LIDARR_API_KEY`,
|
||||||
|
`LIDARR_MUSIC_ROOT`, `LIDARR_PATH_MAP`, and `LIDARR_VERSION_MAJOR` from host*.conf and
|
||||||
|
master.conf. Reads FLAC (vorbis comment block type 4) and MP3 (ID3v2 TXXX frame) tags.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Sonarr Cleanup Thresholds
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SONARR_ORPHAN_AGE=7
|
||||||
|
SONARR_MAX_DELETE_GB=50
|
||||||
|
SONARR_IMPORT_SCAN_TIMEOUT=600
|
||||||
|
SONARR_VERSION_MAJOR=4
|
||||||
|
SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov")
|
||||||
|
SONARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa")
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: `*.ts` IS in extensions — transport stream is used for Live TV recordings tracked
|
||||||
|
by Sonarr. Orphaned `.ts` recordings should be cleaned like any other orphaned episode.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Radarr Cleanup Thresholds
|
||||||
|
|
||||||
|
```bash
|
||||||
|
RADARR_ORPHAN_AGE=7
|
||||||
|
RADARR_MAX_DELETE_GB=50
|
||||||
|
RADARR_IMPORT_SCAN_TIMEOUT=600
|
||||||
|
RADARR_VERSION_MAJOR=6
|
||||||
|
RADARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "wmv" "mov")
|
||||||
|
RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Arr Sync
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ARR_SYNC_ENABLED=true
|
||||||
|
ARR_SYNC_BLOCKLIST="$DATA_DIR/arr_sync_blocklist.tsv" # tombstone file
|
||||||
|
ARR_SYNC_CONNECT_TIMEOUT=10 # SSH connect timeout in seconds
|
||||||
|
ARR_SYNC_API_TIMEOUT=60 # curl API call timeout in seconds
|
||||||
|
DOCKER_APPDATA_BASE=/mnt/user/appdata
|
||||||
|
ARR_SYNC_LIDARR_PORT=8686
|
||||||
|
ARR_SYNC_SONARR_PORT=8989
|
||||||
|
ARR_SYNC_RADARR_PORT=7878
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Arr Recovery
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ARR_IMPORT_RECOVERY_AGE=6 # hours — items newer than this are skipped
|
||||||
|
SONARR_VERSION_MAJOR=4
|
||||||
|
RADARR_VERSION_MAJOR=6
|
||||||
|
LIDARR_VERSION_MAJOR=3
|
||||||
|
ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db" # read by coffee report
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### TMDb / TVDB Removed
|
||||||
|
|
||||||
|
```bash
|
||||||
|
RADARR_DROPPED_ADD_EXCLUSION=true # add removed movies to Radarr import exclusion
|
||||||
|
SONARR_DROPPED_ADD_EXCLUSION=true # add removed series to Sonarr import exclusion
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Lidarr Missing Art
|
||||||
|
|
||||||
|
```bash
|
||||||
|
FANART_API_KEY="your-fanart-tv-api-key"
|
||||||
|
LASTFM_API_KEY="your-lastfm-api-key"
|
||||||
|
LIDARR_ART_MIN_SIZE=5000 # minimum valid download size in bytes
|
||||||
|
LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs
|
||||||
|
LIDARR_ART_RETRIES=2 # download retry attempts per image
|
||||||
|
LIDARR_ART_SLEEP_BETWEEN=1 # seconds between fanart.tv API calls (rate limit)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Lidarr Discovery
|
||||||
|
|
||||||
|
```bash
|
||||||
|
LIDARR_DISCOVERY_THRESHOLD=70 # minimum score for Stage 1 seeds and Stage 2 adds
|
||||||
|
LIDARR_DISCOVERY_LOOKBACK_DAYS=7 # Emby play history window in days
|
||||||
|
LIDARR_DISCOVERY_MIN_PLAYS=3 # min plays before an artist is evaluated as a seed
|
||||||
|
LIDARR_DISCOVERY_MAX_ADDS=5 # max seeds (Stage 1) and max adds (Stage 2) per run
|
||||||
|
LIDARR_DISCOVERY_USER_CAP_PCT=35 # max % any one user contributes to play weight
|
||||||
|
LIDARR_DISCOVERY_REJECT_COOLDOWN=30 # days before re-evaluating a Stage 2 reject
|
||||||
|
LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db"
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires `HOST*_LASTFM_API_KEY` in `host*.conf`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Radarr Discovery
|
||||||
|
|
||||||
|
```bash
|
||||||
|
RADARR_DISCOVERY_THRESHOLD=52 # minimum score to add a candidate
|
||||||
|
RADARR_DISCOVERY_LOOKBACK_DAYS=30 # Emby watch history window in days
|
||||||
|
RADARR_DISCOVERY_MAX_SEEDS=5 # max seed movies from Stage 1
|
||||||
|
RADARR_DISCOVERY_MAX_ADDS=5 # max movies to add per run
|
||||||
|
RADARR_DISCOVERY_MIN_VOTE_COUNT=100 # min TMDB votes for a candidate
|
||||||
|
RADARR_DISCOVERY_MIN_RATING=60 # min TMDB vote_average × 10 (60 = 6.0/10)
|
||||||
|
RADARR_DISCOVERY_REJECT_COOLDOWN=60 # days before re-evaluating a rejected movie
|
||||||
|
RADARR_DISCOVERY_SEED_LIBRARIES=("Movies")
|
||||||
|
RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db"
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Sonarr Discovery
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SONARR_DISCOVERY_THRESHOLD=52 # minimum score to add a candidate
|
||||||
|
SONARR_DISCOVERY_LOOKBACK_DAYS=14 # Emby episode history window in days
|
||||||
|
SONARR_DISCOVERY_MAX_SEEDS=5 # max seed series from Stage 1
|
||||||
|
SONARR_DISCOVERY_MAX_ADDS=3 # max shows to add per run (TV is a larger commitment)
|
||||||
|
SONARR_DISCOVERY_MIN_VOTE_COUNT=50 # min TMDB votes for a candidate
|
||||||
|
SONARR_DISCOVERY_MIN_RATING=65 # min TMDB vote_average × 10 (65 = 6.5/10)
|
||||||
|
SONARR_DISCOVERY_REJECT_COOLDOWN=60 # days before re-evaluating a rejected show
|
||||||
|
SONARR_DISCOVERY_USER_EPISODE_CAP=8 # max episodes per user in seed scoring
|
||||||
|
SONARR_DISCOVERY_MONITOR_MODE="all" # "all" = all seasons monitored; "future" = upcoming only
|
||||||
|
SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db"
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
|
||||||
|
|
||||||
|
> **MONITOR_MODE note:** Use `"all"` (default) to have Sonarr search all existing seasons
|
||||||
|
> after adding a show. `"future"` only marks upcoming seasons as monitored — shows where
|
||||||
|
> all seasons have already aired will appear unmonitored and Sonarr will not search for them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Upgrade Webhook
|
||||||
|
|
||||||
|
```bash
|
||||||
|
WEBHOOK_PORT=7821 # 0 = disable listener
|
||||||
|
WEBHOOK_SECRET="" # auto-generated on first start if empty
|
||||||
|
```
|
||||||
|
|
||||||
|
When `WEBHOOK_PORT=0`, `start_webhook_listener.sh` exits cleanly and no listener starts.
|
||||||
|
When `WEBHOOK_SECRET` is empty, a 32-byte hex secret is generated on first start and
|
||||||
|
written back to `master.conf`. Run `Tools/webhook_setup.sh` to register the URL in arrs.
|
||||||
|
|
||||||
|
Webhook log: `/var/log/varaverk/upgrade_webhook.log`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ CONFIGURATION — host*.conf ━━━
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Arr connection details — must match arr settings exactly
|
||||||
|
HOST1_LIDARR_URL="http://192.168.50.2:8686"
|
||||||
|
HOST1_LIDARR_API_KEY="..."
|
||||||
|
HOST1_LIDARR_MUSIC_ROOT="/mnt/user/Music"
|
||||||
|
HOST1_LIDARR_PATH_MAP="" # container→host path translation if needed
|
||||||
|
|
||||||
|
HOST1_SONARR_URL="http://192.168.50.2:8989"
|
||||||
|
HOST1_SONARR_API_KEY="..."
|
||||||
|
HOST1_SONARR_TV_ROOT="/mnt/user/Tv_Shows"
|
||||||
|
HOST1_SONARR_PATH_MAP=""
|
||||||
|
|
||||||
|
HOST1_RADARR_URL="http://192.168.50.2:7878"
|
||||||
|
HOST1_RADARR_API_KEY="..."
|
||||||
|
HOST1_RADARR_MOVIES_ROOT="/mnt/user/Movies"
|
||||||
|
HOST1_RADARR_PATH_MAP=""
|
||||||
|
|
||||||
|
HOST1_EMBY_URL="http://192.168.50.2:8096"
|
||||||
|
HOST1_EMBY_API_KEY="..."
|
||||||
|
```
|
||||||
|
|
||||||
|
> **LIDARR/SONARR/RADARR_MUSIC/TV/MOVIES_ROOT must exactly match the Root Folder path in
|
||||||
|
> the arr's own settings.** Arr UI → Settings → Media Management → Root Folders.
|
||||||
|
> A mismatch means every file on disk looks untracked — all appear as orphans.
|
||||||
|
> MAX_DELETE_GB is the only thing standing between a path mismatch and losing your library.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SAFE TESTING PROCEDURE ━━━
|
||||||
|
|
||||||
|
> **The arr cleanup scripts permanently delete files.** There is no recycle bin, no undo.
|
||||||
|
> Follow this procedure on first use, after any root path change, after any API key change,
|
||||||
|
> and after any significant arr library change.
|
||||||
|
|
||||||
|
### Step 1 — Dry Run With Full Logging
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lidarr_release_fixer.sh --dry-run --log
|
||||||
|
lidarr_cleanup.sh --dry-run --log
|
||||||
|
sonarr_cleanup.sh --dry-run --log
|
||||||
|
radarr_cleanup.sh --dry-run --log
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2 — Review the Output
|
||||||
|
|
||||||
|
```
|
||||||
|
Release fixer:
|
||||||
|
Are the "would fix" albums expected?
|
||||||
|
→ Wrong release UUIDs switching to correct ones is expected behaviour
|
||||||
|
→ If everything is already correct, nothing to do
|
||||||
|
|
||||||
|
Cleanup:
|
||||||
|
Are TRACKED files the ones you expect?
|
||||||
|
→ Known arr-managed files should show as TRACKED
|
||||||
|
→ If they show as ORPHAN, the root path is wrong — STOP
|
||||||
|
|
||||||
|
Is the ORPHAN count reasonable?
|
||||||
|
→ Healthy cleanup removes dozens to hundreds, not tens of thousands
|
||||||
|
→ Large count = stop, investigate root path before proceeding
|
||||||
|
|
||||||
|
Are PROTECTED patterns working?
|
||||||
|
→ Artwork (*.jpg) and subtitles (*.srt) must show as PROTECTED
|
||||||
|
→ If they show as ORPHAN, check PROTECTED_PATTERNS config
|
||||||
|
|
||||||
|
Are RECENT files being correctly skipped?
|
||||||
|
→ Files downloaded in the last 7 days should show as RECENT, not ORPHAN
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3 — Check Numbers if Something Looks Wrong
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Root path mismatch? Compare these:
|
||||||
|
# Lidarr UI: Settings → Media Management → Root Folders
|
||||||
|
# Sonarr UI: Settings → Media Management → Root Folders
|
||||||
|
# Radarr UI: Settings → Media Management → Root Folders
|
||||||
|
# Must exactly match LIDARR_MUSIC_ROOT / SONARR_TV_ROOT / RADARR_MOVIES_ROOT
|
||||||
|
|
||||||
|
# Is the arr running?
|
||||||
|
docker ps | grep -E "Lidarr|Sonarr|Radarr"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4 — Run Live
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Only after dry run review passes.
|
||||||
|
lidarr_release_fixer.sh
|
||||||
|
lidarr_cleanup.sh
|
||||||
|
sonarr_cleanup.sh
|
||||||
|
radarr_cleanup.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 5 — Verify in Arr UI
|
||||||
|
|
||||||
|
```
|
||||||
|
Library count — should not have dropped significantly
|
||||||
|
Missing files — check if any monitored content shows as missing
|
||||||
|
Emby library — should show no ghost entries (notify_emby_scan handles this automatically)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ PROCEDURES ━━━
|
||||||
|
|
||||||
|
### Adding a New Arr
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Copy radarr_cleanup.sh as template
|
||||||
|
cp Arrs_Stack/radarr_cleanup.sh Arrs_Stack/readarr_cleanup.sh
|
||||||
|
|
||||||
|
# 2. Replace RADARR_ prefix with READARR_ throughout
|
||||||
|
# Update API endpoint, tracked file API path, extension list, protected patterns
|
||||||
|
|
||||||
|
# 3. Add to host*.conf
|
||||||
|
HOST1_READARR_URL="http://192.168.50.2:8787"
|
||||||
|
HOST1_READARR_API_KEY="your-api-key"
|
||||||
|
HOST1_READARR_BOOKS_ROOT="/mnt/user/Books"
|
||||||
|
|
||||||
|
# 4. Add thresholds to master.conf
|
||||||
|
READARR_ORPHAN_AGE=7
|
||||||
|
READARR_MAX_DELETE_GB=50
|
||||||
|
READARR_EXTENSIONS=("epub" "pdf" "mobi" "azw3" "cbz" "cbr")
|
||||||
|
READARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo")
|
||||||
|
|
||||||
|
# 5. Add to DAILY_MAINTENANCE_SCRIPTS in master.conf
|
||||||
|
"Arrs_Stack/readarr_cleanup.sh"
|
||||||
|
```
|
||||||
|
|
||||||
|
Run `--dry-run --log` before scheduling.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Managing the Arr Sync Blocklist
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Add item to blocklist (removes from all arrs + tombstones the ID)
|
||||||
|
arr_sync.sh --blocklist-add lidarr <musicbrainz-artist-id> "reason"
|
||||||
|
arr_sync.sh --blocklist-add sonarr <tvdb-series-id> "reason"
|
||||||
|
arr_sync.sh --blocklist-add radarr <tmdb-movie-id> "reason"
|
||||||
|
|
||||||
|
# Remove from blocklist (un-tombstones the ID — does NOT re-add to arrs)
|
||||||
|
arr_sync.sh --blocklist-remove lidarr <id>
|
||||||
|
|
||||||
|
# View all blocklisted IDs
|
||||||
|
arr_sync.sh --blocklist-list
|
||||||
|
```
|
||||||
|
|
||||||
|
`--blocklist-add` is the only destructive operation — it simultaneously:
|
||||||
|
1. Writes the tombstone entry to the blocklist TSV file
|
||||||
|
2. Deletes the item from the local arr API (no file deletion)
|
||||||
|
3. SSHes each remote node and deletes from their arr API
|
||||||
|
|
||||||
|
Files become orphans on all nodes — arr_cleanup removes them on the next run.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ TROUBLESHOOTING ━━━
|
||||||
|
|
||||||
|
### Arr Cleanup Deleting Files It Shouldn't
|
||||||
|
|
||||||
|
```
|
||||||
|
1. Check the protected patterns — artwork and subtitles must be listed
|
||||||
|
LIDARR_PROTECTED_PATTERNS / SONARR_PROTECTED_PATTERNS / RADARR_PROTECTED_PATTERNS
|
||||||
|
|
||||||
|
2. Check the root path matches arr settings exactly
|
||||||
|
Run: lidarr_cleanup.sh --status (shows configured root path)
|
||||||
|
Compare: Lidarr UI → Settings → Media Management → Root Folders
|
||||||
|
|
||||||
|
3. Check if files are truly orphaned
|
||||||
|
Run: lidarr_cleanup.sh --dry-run --log
|
||||||
|
Look for the specific file — verify it shows ORPHAN, not PROTECTED or TRACKED
|
||||||
|
```
|
||||||
|
|
||||||
|
### Lidarr Files Present But Not Importing
|
||||||
|
|
||||||
|
```
|
||||||
|
Symptom: album folder exists with properly tagged files, Lidarr shows 0 tracks imported.
|
||||||
|
Cause: Lidarr selected the wrong MusicBrainz release edition. The file's MBID doesn't
|
||||||
|
match Lidarr's foreignReleaseId, so track ID lookup fails.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
lidarr_release_fixer.sh --dry-run --log
|
||||||
|
→ Shows which albums would be corrected and which release UUIDs would change
|
||||||
|
|
||||||
|
lidarr_release_fixer.sh
|
||||||
|
→ Applies corrections and queues RefreshArtist for affected artists
|
||||||
|
|
||||||
|
If the file MBID doesn't match any Lidarr release (shows "MBID not in list"):
|
||||||
|
→ File was tagged from a source Lidarr doesn't know about (unofficial, re-tagged)
|
||||||
|
→ Manual import or re-tag with correct MBID
|
||||||
|
```
|
||||||
|
|
||||||
|
### Arr Cleanup Aborting at Safety Layer 6 (Tracked Count Drop)
|
||||||
|
|
||||||
|
```
|
||||||
|
API returned far fewer tracked files than last run.
|
||||||
|
Possible causes:
|
||||||
|
- Arr database was recently rebuilt from scratch
|
||||||
|
- Large manual library removal
|
||||||
|
- Path map mismatch after arr migration
|
||||||
|
|
||||||
|
If intentional (library intentionally reduced):
|
||||||
|
Delete LIDARR_TRACKED_COUNT_FILE to reset the baseline
|
||||||
|
Run cleanup once — it will establish a new baseline
|
||||||
|
|
||||||
|
If unintentional:
|
||||||
|
Investigate before proceeding — the arr may have a problem
|
||||||
|
```
|
||||||
|
|
||||||
|
### Arr Sync Not Picking Up New Content
|
||||||
|
|
||||||
|
```
|
||||||
|
Is ARR_SYNC_ENABLED=true in master.conf?
|
||||||
|
|
||||||
|
Can this host SSH to the remote without password?
|
||||||
|
→ ssh -i [SSH_KEY] root@[remote-tailscale-ip] "hostname"
|
||||||
|
|
||||||
|
Is the arr accessible on the remote?
|
||||||
|
→ arr_sync.sh --status (shows each node's arr reachability)
|
||||||
|
→ arr_sync.sh --log (verbose output per-node, per-arr)
|
||||||
|
|
||||||
|
Is the item in the blocklist?
|
||||||
|
→ arr_sync.sh --blocklist-list
|
||||||
|
```
|
||||||
|
|
||||||
|
### Emby Still Showing Ghost Entries After Cleanup
|
||||||
|
|
||||||
|
```
|
||||||
|
notify_emby_scan() is called automatically after every arr cleanup deletion.
|
||||||
|
If ghosts persist:
|
||||||
|
1. Is Emby's API responding?
|
||||||
|
curl -s "http://[emby-ip]:8096/System/Info/Public"
|
||||||
|
|
||||||
|
2. Is EMBY_URL / EMBY_API_KEY correct in host*.conf?
|
||||||
|
Run: sonarr_cleanup.sh --status (shows Emby config)
|
||||||
|
|
||||||
|
3. Trigger manually in Emby:
|
||||||
|
Library → Manage Library → Clean Missing Files
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ OUTPUT TIERS ━━━
|
||||||
|
|
||||||
|
All scripts have two output levels controlled by `--log`.
|
||||||
|
|
||||||
|
Without `--log`, each script processes silently and always concludes with a summary
|
||||||
|
block: identity, duration, counts (files removed, items added, arrs cleaned), and a
|
||||||
|
status line. Warnings and errors are always visible.
|
||||||
|
|
||||||
|
With `--log`, per-item detail appears: individual titles being processed, API query
|
||||||
|
progress, per-node sync results, and per-file examination output. Use this when
|
||||||
|
debugging unexpected results or validating configuration before the first scheduled run.
|
||||||
|
|
||||||
|
Dry-run output follows the same tiers — `--dry-run` alone shows the summary of what
|
||||||
|
would happen; `--dry-run --log` shows the full per-item preview list.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ FLAG REFERENCE ━━━
|
||||||
|
|
||||||
|
### lidarr_release_fixer.sh
|
||||||
|
|
||||||
|
`lidarr_release_fixer.sh`
|
||||||
|
Read MUSICBRAINZ_ALBUMID from FLAC/MP3 files, correct Lidarr release selection, queue RefreshArtist.
|
||||||
|
|
||||||
|
`lidarr_release_fixer.sh --dry-run`
|
||||||
|
Show which albums would be corrected and the before/after release UUIDs. No API writes.
|
||||||
|
|
||||||
|
`lidarr_release_fixer.sh --status`
|
||||||
|
Show configuration, Lidarr URL, enabled state.
|
||||||
|
|
||||||
|
`lidarr_release_fixer.sh --log`
|
||||||
|
Verbose per-album output — shows every album checked, not just those corrected.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### lidarr_cleanup.sh / sonarr_cleanup.sh / radarr_cleanup.sh
|
||||||
|
|
||||||
|
`[script] --dry-run --log`
|
||||||
|
Preview every classification decision. **Always run this first.** See Safe Testing Procedure.
|
||||||
|
|
||||||
|
`[script]`
|
||||||
|
Live run — deletes confirmed orphans and junk, triggers Emby clean.
|
||||||
|
|
||||||
|
`[script] --log`
|
||||||
|
Live run with verbose per-file output.
|
||||||
|
|
||||||
|
`[script] --status`
|
||||||
|
Show configuration, API status, tracked file count, and last run stats.
|
||||||
|
|
||||||
|
`[script] --i-know-what-im-doing`
|
||||||
|
Bypass the MAX_DELETE_GB size threshold. Required when deletion exceeds the configured
|
||||||
|
limit. Long flag name is intentional — cannot be added accidentally.
|
||||||
|
|
||||||
|
`[script] --skip-age-check`
|
||||||
|
Bypass the ORPHAN_AGE age check. Deletes RECENT files too — files that are under the
|
||||||
|
age threshold. Use when you know recent downloads are actually orphans.
|
||||||
|
|
||||||
|
`[script] --i-know-what-im-doing --skip-age-check`
|
||||||
|
**NUCLEAR MODE** — age check and size threshold both bypassed. Deletes on first pass.
|
||||||
|
Use when you want a clean one-pass wipe of everything the arr doesn't track.
|
||||||
|
No recovery possible after deletion.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### arrs_failed_stalled_recovery.sh
|
||||||
|
|
||||||
|
`arrs_failed_stalled_recovery.sh`
|
||||||
|
Check all configured arrs for failed imports and stalled downloads. Blocklist + remove +
|
||||||
|
re-search for each problem item.
|
||||||
|
|
||||||
|
`arrs_failed_stalled_recovery.sh --dry-run`
|
||||||
|
Show what would be actioned per arr without making any changes.
|
||||||
|
|
||||||
|
`arrs_failed_stalled_recovery.sh --status`
|
||||||
|
Show configuration, arr reachability, and last recovery stats.
|
||||||
|
|
||||||
|
`arrs_failed_stalled_recovery.sh --log`
|
||||||
|
Verbose output per item per arr.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### arr_sync.sh
|
||||||
|
|
||||||
|
`arr_sync.sh`
|
||||||
|
Sync all arr types across all configured nodes.
|
||||||
|
|
||||||
|
`arr_sync.sh --dry-run`
|
||||||
|
Show what would be added/removed on each node without making changes.
|
||||||
|
|
||||||
|
`arr_sync.sh --status`
|
||||||
|
Show node configuration, arr reachability, and blocklist count.
|
||||||
|
|
||||||
|
`arr_sync.sh --log`
|
||||||
|
Verbose per-node, per-arr output.
|
||||||
|
|
||||||
|
`arr_sync.sh --blocklist-add [arr] [id] "[reason]"`
|
||||||
|
Remove item from all arrs and tombstone the ID. See Procedures above.
|
||||||
|
|
||||||
|
`arr_sync.sh --blocklist-remove [arr] [id]`
|
||||||
|
Remove tombstone — does NOT re-add item to arrs.
|
||||||
|
|
||||||
|
`arr_sync.sh --blocklist-list`
|
||||||
|
Show all tombstoned IDs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### radarr_tmdb_removed.sh / sonarr_tvdb_removed.sh
|
||||||
|
|
||||||
|
`[script]`
|
||||||
|
Remove records for entries with status="deleted" (dropped from upstream database).
|
||||||
|
Files are kept. Import exclusion is added.
|
||||||
|
|
||||||
|
`[script] --delete-files`
|
||||||
|
Also delete associated files from disk. Most dropped entries have no files — they were
|
||||||
|
announced movies/series that were never downloaded.
|
||||||
|
|
||||||
|
`[script] --dry-run`
|
||||||
|
Preview what would be removed without making changes.
|
||||||
|
|
||||||
|
`[script] --status`
|
||||||
|
Show arr connection status and current count of dropped entries.
|
||||||
|
|
||||||
|
`[script] --log`
|
||||||
|
Verbose per-entry output.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### lidarr_missing_art.sh
|
||||||
|
|
||||||
|
`lidarr_missing_art.sh`
|
||||||
|
Fetch all missing album and artist artwork from fanart.tv and fallback sources.
|
||||||
|
Never overwrites existing files.
|
||||||
|
|
||||||
|
`lidarr_missing_art.sh --dry-run`
|
||||||
|
Show what would be downloaded without writing any files.
|
||||||
|
|
||||||
|
`lidarr_missing_art.sh --status`
|
||||||
|
Show configuration and API key status.
|
||||||
|
|
||||||
|
`lidarr_missing_art.sh --log`
|
||||||
|
Verbose per-album, per-artist output.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### playback_aware_lidarr_discovery.sh
|
||||||
|
|
||||||
|
`playback_aware_lidarr_discovery.sh`
|
||||||
|
Score Emby play history, run Last.fm getSimilar on top artists, add candidates above
|
||||||
|
threshold to Lidarr. Triggers ArtistSearch immediately after each successful add.
|
||||||
|
|
||||||
|
`playback_aware_lidarr_discovery.sh --dry-run`
|
||||||
|
Score and rank all Stage 1 seeds and Stage 2 candidates. No Lidarr API calls. No writes
|
||||||
|
to history file. Shows exactly what would be added and at what score.
|
||||||
|
|
||||||
|
`playback_aware_lidarr_discovery.sh --status`
|
||||||
|
Show config values, history file path and size, and API key status.
|
||||||
|
|
||||||
|
`playback_aware_lidarr_discovery.sh --log`
|
||||||
|
Verbose per-artist scoring output for both stages.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### playback_aware_radarr_discovery.sh
|
||||||
|
|
||||||
|
`playback_aware_radarr_discovery.sh`
|
||||||
|
Score recently watched Emby movies, run TMDB recommendations on seeds, add candidates
|
||||||
|
above threshold to Radarr. Triggers MoviesSearch immediately after each successful add.
|
||||||
|
|
||||||
|
`playback_aware_radarr_discovery.sh --dry-run`
|
||||||
|
Score and rank all Stage 1 seeds and Stage 2 candidates. No Radarr API calls. No writes
|
||||||
|
to history file.
|
||||||
|
|
||||||
|
`playback_aware_radarr_discovery.sh --status`
|
||||||
|
Show config values, history file path and size, and API key status.
|
||||||
|
|
||||||
|
`playback_aware_radarr_discovery.sh --log`
|
||||||
|
Verbose per-movie scoring output for both stages.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### playback_aware_sonarr_discovery.sh
|
||||||
|
|
||||||
|
`playback_aware_sonarr_discovery.sh`
|
||||||
|
Score recently watched Emby series (weighted by user diversity), run TMDB TV
|
||||||
|
recommendations on seeds, add candidates above threshold to Sonarr. Triggers SeriesSearch
|
||||||
|
immediately after each successful add.
|
||||||
|
|
||||||
|
`playback_aware_sonarr_discovery.sh --dry-run`
|
||||||
|
Score and rank all Stage 1 seeds and Stage 2 candidates. No Sonarr API calls. No writes
|
||||||
|
to history file.
|
||||||
|
|
||||||
|
`playback_aware_sonarr_discovery.sh --status`
|
||||||
|
Show config values, history file path and size, and API key status.
|
||||||
|
|
||||||
|
`playback_aware_sonarr_discovery.sh --log`
|
||||||
|
Verbose per-series scoring output — shows user diversity, recency, and volume scores per
|
||||||
|
seed; breadth, rating, and votes scores per candidate.
|
||||||
@@ -0,0 +1,250 @@
|
|||||||
|
# ━━━━━ ARRS STACK ━━━━━
|
||||||
|
|
||||||
|
Lifecycle management for the arr suite (Lidarr, Sonarr, Radarr) across a multi-server
|
||||||
|
ecosystem. Library sync so every node tracks the same content. Orphan cleanup against live
|
||||||
|
arr APIs so deleted content actually leaves disk. Release correction so wrong MusicBrainz
|
||||||
|
editions don't silently block Lidarr imports. Emby notified automatically after every
|
||||||
|
deletion. Failed downloads recovered overnight. Artwork fetched continuously. Database
|
||||||
|
hygiene for dropped upstream entries. Quality upgrades propagate to all nodes immediately
|
||||||
|
via webhook. Weekly discovery adds new content based on what you actually play and watch.
|
||||||
|
|
||||||
|
> **These scripts permanently delete files and modify arr databases.** Orphan cleanup is
|
||||||
|
> protected by multiple safety layers that must all pass before anything is touched — but
|
||||||
|
> dry runs and log review are still the right first step on any new system or after any
|
||||||
|
> configuration change. See Manual-Arrs_Stack.md for the safe testing procedure.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THE PROBLEMS THAT BUILT THIS ━━━
|
||||||
|
|
||||||
|
**Deleted Shows and Removed Albums Still on Disk**
|
||||||
|
When you remove a series from Sonarr and the delete command fails — permission issue,
|
||||||
|
container wasn't running, path mismatch — the files stay permanently. Over years on an
|
||||||
|
active library this accumulates significantly.
|
||||||
|
The fix: arr cleanup scripts query the live API for every tracked file path, walk the
|
||||||
|
disk, and delete anything absent from the API response that's old enough to be past the
|
||||||
|
import window.
|
||||||
|
|
||||||
|
**Wrong MusicBrainz Release Edition Silently Blocking Lidarr Imports**
|
||||||
|
Lidarr selects one specific release edition per album using a MusicBrainz release ID.
|
||||||
|
When it picks the wrong edition (Brazil CD instead of US CD, Japan Digital instead of
|
||||||
|
standard), the track IDs don't match what's embedded in the files. RescanFolders reports
|
||||||
|
"Importing 0 tracks" even with perfectly tagged, complete files present. No error — just
|
||||||
|
silence.
|
||||||
|
The fix: `lidarr_release_fixer.sh` reads the MUSICBRAINZ_ALBUMID tag from each file,
|
||||||
|
finds the matching release in Lidarr's known releases, switches the selection, and queues
|
||||||
|
a RefreshArtist. Runs before cleanup so corrected albums are imported before the orphan
|
||||||
|
scan ever sees them.
|
||||||
|
|
||||||
|
**Emby Showing Ghost Entries After Cleanup**
|
||||||
|
After arr cleanup deletes files, Emby still shows them until its next scheduled scan —
|
||||||
|
potentially hours later. Users see broken entries that produce "file not found" errors.
|
||||||
|
The fix: `notify_emby_scan()` is called automatically after every deletion. Triggers
|
||||||
|
Emby's "Clean Missing Files" task immediately.
|
||||||
|
|
||||||
|
**No Safety Net on Deletion Size**
|
||||||
|
A misconfigured root path — pointing cleanup at the wrong directory — means the API
|
||||||
|
returns zero tracked files for a root that actually contains thousands. Every file walks
|
||||||
|
as an orphan. Everything gets deleted. This is the catastrophic failure mode.
|
||||||
|
The fix: `LIDARR/SONARR/RADARR_MAX_DELETE_GB` — if total deletion size exceeds the
|
||||||
|
limit, the script stops and requires `--i-know-what-im-doing` to proceed. The flag name
|
||||||
|
is long and annoying by design. It cannot be added by accident.
|
||||||
|
|
||||||
|
**Failed Downloads Accumulating Silently**
|
||||||
|
Import failures and stalled downloads sit in arr queues indefinitely. Without
|
||||||
|
intervention they occupy queue slots, block new searches, and the item never gets
|
||||||
|
downloaded. Checking queues manually across three arrs is tedious.
|
||||||
|
The fix: `arrs_failed_stalled_recovery.sh` inspects all arr queues, blocklists the bad
|
||||||
|
release, removes it, and triggers a re-search. Runs daily.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||||
|
|
||||||
|
**Library Sync**
|
||||||
|
`arr_sync.sh` — full-mesh arr library sync across all nodes. Every node syncs with every
|
||||||
|
other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the
|
||||||
|
actual files.
|
||||||
|
|
||||||
|
**Release Correction**
|
||||||
|
`lidarr_release_fixer.sh` — reads MUSICBRAINZ_ALBUMID from FLAC and MP3 files, matches
|
||||||
|
against Lidarr's known releases per album, switches monitored=true to the correct
|
||||||
|
edition, queues RefreshArtist. Runs before lidarr_cleanup.sh daily. Handles both FLAC
|
||||||
|
(vorbis comment block) and MP3 (ID3v2 TXXX frame).
|
||||||
|
|
||||||
|
**Orphan Cleanup**
|
||||||
|
`lidarr_cleanup.sh`, `sonarr_cleanup.sh`, `radarr_cleanup.sh` — API-verified orphan
|
||||||
|
removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven
|
||||||
|
safety layers, automatic Emby notification after deletion.
|
||||||
|
|
||||||
|
**Database Hygiene**
|
||||||
|
`radarr_tmdb_removed.sh`, `sonarr_tvdb_removed.sh` — remove entries that upstream
|
||||||
|
databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in
|
||||||
|
arrs and can never be monitored or downloaded. Most are announced-but-never-released
|
||||||
|
entries. Files are kept by default — most have none.
|
||||||
|
|
||||||
|
**Library Recovery**
|
||||||
|
`arrs_failed_stalled_recovery.sh` — detect and recover failed imports and stalled
|
||||||
|
downloads across all arrs. Blocklists the bad release and triggers a re-search — hands-
|
||||||
|
free overnight recovery.
|
||||||
|
|
||||||
|
**Library Enrichment**
|
||||||
|
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
|
||||||
|
fallback sources. Never overwrites existing files.
|
||||||
|
|
||||||
|
**Upgrade Propagation**
|
||||||
|
`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr
|
||||||
|
OnUpgrade webhooks. Continuous; started at array start. Writes to
|
||||||
|
`/var/log/varaverk/upgrade_webhook.log`.
|
||||||
|
`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded
|
||||||
|
item folder to every remote node immediately, then triggers an arr library rescan on
|
||||||
|
each remote so the upgraded file is accepted without triggering a redundant quality search.
|
||||||
|
|
||||||
|
**Discovery**
|
||||||
|
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
|
||||||
|
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
|
||||||
|
Lidarr. 0–5 meaningful adds per week.
|
||||||
|
`playback_aware_radarr_discovery.sh` — behavior-driven movie discovery. Scores recently
|
||||||
|
watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr.
|
||||||
|
`playback_aware_sonarr_discovery.sh` — behavior-driven TV discovery. Scores recently
|
||||||
|
watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds
|
||||||
|
top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ EXECUTION ORDER ━━━
|
||||||
|
|
||||||
|
**Daily via `daily_sync_maintenance.sh` (DAILY_MAINTENANCE_SCRIPTS):**
|
||||||
|
|
||||||
|
```
|
||||||
|
1. lidarr_release_fixer.sh — correct wrong release editions before cleanup sees them
|
||||||
|
2. lidarr_cleanup.sh — orphan removal (music)
|
||||||
|
3. sonarr_cleanup.sh — orphan removal (TV)
|
||||||
|
4. radarr_cleanup.sh — orphan removal (movies)
|
||||||
|
5. lidarr_missing_art.sh — fetch missing artwork (HOST1 only)
|
||||||
|
6. radarr_tmdb_removed.sh — remove TMDb-dropped movies
|
||||||
|
7. sonarr_tvdb_removed.sh — remove TVDB-dropped series
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: `media_shares_permissions.sh` and `media_cleaner.sh` run before these from
|
||||||
|
`Media/` — permissions and junk removal must complete first.
|
||||||
|
|
||||||
|
**Every 30 min + 4hr via orchestrators (CRITICAL/INTERMEDIATE_MAINTENANCE_SCRIPTS):**
|
||||||
|
|
||||||
|
```
|
||||||
|
arrs_failed_stalled_recovery.sh — failed/stalled queue recovery
|
||||||
|
arr_sync.sh — library sync across all nodes
|
||||||
|
```
|
||||||
|
|
||||||
|
**Weekly via `weekly_sync_maintenance.sh` (WEEKLY_MAINTENANCE_SCRIPTS):**
|
||||||
|
|
||||||
|
```
|
||||||
|
playback_aware_lidarr_discovery.sh — score play history → Last.fm similar → Lidarr
|
||||||
|
playback_aware_radarr_discovery.sh — score watch history → TMDB recommendations → Radarr
|
||||||
|
playback_aware_sonarr_discovery.sh — score episode history → TMDB TV → Sonarr
|
||||||
|
```
|
||||||
|
|
||||||
|
**Continuous (started by `array_started.sh`):**
|
||||||
|
|
||||||
|
```
|
||||||
|
start_webhook_listener.sh — Node.js webhook server; dispatches upgrade_webhook_handler.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**On every arr upgrade (triggered by webhook):**
|
||||||
|
|
||||||
|
```
|
||||||
|
upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why release fixer before cleanup:** the fixer corrects Lidarr's release selection so
|
||||||
|
files get imported. If cleanup ran first, a correctable album could accumulate age toward
|
||||||
|
the orphan threshold before the fixer had a chance to fix it.
|
||||||
|
|
||||||
|
**Why arr_sync before rsync:** once arrs agree on what to track, rsync spreads the actual
|
||||||
|
files. An upgrade on one node — new tracked path, old path no longer in API — gets
|
||||||
|
cleaned by arr_cleanup on all nodes after the next sync cycle.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ HOST AWARENESS ━━━
|
||||||
|
|
||||||
|
Scripts run on both servers via `detect_hosts()`, which aliases all `HOST*_` prefixed vars
|
||||||
|
to their unprefixed names at runtime. No manual `HOST1`/`HOST2` comparisons exist in any
|
||||||
|
script.
|
||||||
|
|
||||||
|
`arr_sync.sh` keeps all arr databases in bidirectional union — either server can download
|
||||||
|
to any share. Arr cleanup uses the union model: a file is only an orphan if the arr on
|
||||||
|
this host doesn't have it indexed. Arr scripts check the aliased URL — if empty (arr not
|
||||||
|
configured on this host), they exit cleanly with no action.
|
||||||
|
|
||||||
|
`lidarr_release_fixer.sh` and `lidarr_missing_art.sh` exit cleanly on hosts without
|
||||||
|
Lidarr configured — no HOST1_LIDARR_URL means nothing runs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||||
|
|
||||||
|
| Script | Role | When It Runs |
|
||||||
|
|--------|------|--------------|
|
||||||
|
| `arr_sync.sh` | Full-mesh arr library sync — all nodes track the same content | Every 4hr + weekly before rsync |
|
||||||
|
| `lidarr_release_fixer.sh` | Fix wrong MusicBrainz release editions so files get imported | Daily before lidarr_cleanup |
|
||||||
|
| `lidarr_cleanup.sh` | Delete orphaned music files not tracked by Lidarr | Daily |
|
||||||
|
| `sonarr_cleanup.sh` | Delete orphaned TV files not tracked by Sonarr | Daily |
|
||||||
|
| `radarr_cleanup.sh` | Delete orphaned movie files not tracked by Radarr | Daily |
|
||||||
|
| `arrs_failed_stalled_recovery.sh` | Auto-recover failed imports and stalled downloads | Every 30 min / daily |
|
||||||
|
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Daily (HOST1 only) |
|
||||||
|
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Daily |
|
||||||
|
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Daily |
|
||||||
|
| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous |
|
||||||
|
| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade |
|
||||||
|
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
|
||||||
|
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
|
||||||
|
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||||
|
|
||||||
|
```
|
||||||
|
Every 4hr / weekly (arr sync before rsync):
|
||||||
|
arr_sync.sh ──────────────── syncs tracked IDs across all nodes
|
||||||
|
│ union model: any node adds → all nodes get it
|
||||||
|
│
|
||||||
|
└── then rsync spreads the actual files to all nodes
|
||||||
|
└── then arr_cleanup removes orphans on all nodes (old paths, removed content)
|
||||||
|
|
||||||
|
Daily maintenance window:
|
||||||
|
[Media/media_shares_permissions.sh + media_cleaner.sh run first — from Media/]
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
lidarr_release_fixer.sh ────── reads MBID tag → switches release in Lidarr → RefreshArtist
|
||||||
|
│ (corrected albums get imported before cleanup scans for orphans)
|
||||||
|
▼
|
||||||
|
lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans
|
||||||
|
sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans
|
||||||
|
radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans
|
||||||
|
│
|
||||||
|
└── each cleanup → notify_emby_scan() → Emby removes ghost entries
|
||||||
|
|
||||||
|
Daily recovery:
|
||||||
|
arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search
|
||||||
|
|
||||||
|
Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
|
||||||
|
playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr
|
||||||
|
playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr
|
||||||
|
playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr
|
||||||
|
│
|
||||||
|
└── each discovery script fires arr search immediately after successful add
|
||||||
|
|
||||||
|
Continuous (started by array_started.sh):
|
||||||
|
start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
|
||||||
|
│ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
|
||||||
|
└── upgrade_webhook_handler.sh
|
||||||
|
├── rsync upgraded folder → all remote nodes immediately
|
||||||
|
└── trigger arr library rescan on each remote (accept new file, no quality search)
|
||||||
|
|
||||||
|
Ad-hoc enrichment:
|
||||||
|
lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
|
||||||
|
radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
|
||||||
|
sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion
|
||||||
|
```
|
||||||
@@ -1418,6 +1418,7 @@
|
|||||||
# ━━━ State Files ━━━
|
# ━━━ State Files ━━━
|
||||||
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
|
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
|
||||||
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
|
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
|
||||||
|
DOCKER_WATCHDOG_INTENTIONAL_FILE="$STATE_DIR/docker_intentional_stops.db"
|
||||||
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
|
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
|
||||||
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
|
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
|
||||||
|
|
||||||
|
|||||||
@@ -1404,6 +1404,7 @@
|
|||||||
# ━━━ State Files ━━━
|
# ━━━ State Files ━━━
|
||||||
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
|
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
|
||||||
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
|
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
|
||||||
|
DOCKER_WATCHDOG_INTENTIONAL_FILE="$STATE_DIR/docker_intentional_stops.db"
|
||||||
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
|
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
|
||||||
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
|
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
|
||||||
|
|
||||||
|
|||||||
+12
-623
@@ -1,8 +1,11 @@
|
|||||||
# ━━━━━ MEDIA — Manual ━━━━━
|
# ━━━━━ MEDIA — Manual ━━━━━
|
||||||
|
|
||||||
Config reference, procedures, operational workflows.
|
Config reference, procedures, operational workflows for Media/ scripts.
|
||||||
For overview see README-Media.md. For per-script detail see script headers.
|
For overview see README-Media.md. For per-script detail see script headers.
|
||||||
|
|
||||||
|
For arr stack config and procedures (cleanup, release fixer, sync, discovery) see
|
||||||
|
`Arrs_Stack/Manual-Arrs_Stack.md`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ PERMISSIONS MODEL ━━━
|
## ━━━ PERMISSIONS MODEL ━━━
|
||||||
@@ -32,48 +35,6 @@ Once fixed, this script corrects 0 files per run — it becomes a pure daily fai
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ ARR CLEANUP — FILE CLASSIFICATION ━━━
|
|
||||||
|
|
||||||
Every file found on disk during an arr cleanup run falls into exactly one category:
|
|
||||||
|
|
||||||
```
|
|
||||||
TRACKED → arr API returned this exact path → leave it alone
|
|
||||||
PROTECTED → matches ARR_PROTECTED_PATTERNS → never delete
|
|
||||||
ORPHAN → media extension, not tracked, old enough → delete
|
|
||||||
JUNK → not a media extension, not protected → delete (any age)
|
|
||||||
RECENT → not tracked, under ARR_ORPHAN_AGE days → skip (may be mid-import)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Why protected patterns are critical:** arrs generate artwork (`*.jpg`), metadata
|
|
||||||
(`*.nfo`), and subtitles/lyrics that do NOT appear in the tracked file API response.
|
|
||||||
Without protection, these would be classified as orphans and deleted — removing cover art
|
|
||||||
from every album, every movie poster, every TV show thumbnail. Requires a full rescan
|
|
||||||
to recover. Never remove artwork extensions from protected patterns.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ━━━ ARR CLEANUP — SAFETY LAYERS ━━━
|
|
||||||
|
|
||||||
All 7 layers must pass before any file is touched. There is no way to push through a
|
|
||||||
failed safety check without the explicit override flag.
|
|
||||||
|
|
||||||
```
|
|
||||||
1. Container running + healthy — a stopped container has an empty API
|
|
||||||
2. API reachable — no API = no tracked file list = everything looks orphaned
|
|
||||||
3. API version matches — major version must match tested version in master.conf
|
|
||||||
4. Item count > 0 — no artists/series/movies = something is wrong with DB
|
|
||||||
5. Tracked file count > 0 — empty response = everything would be deleted
|
|
||||||
6. Tracked count >= MIN_TRACKED_PCT — dramatic drop from last run = abort and alert
|
|
||||||
7. Deletion size < MAX_DELETE_GB — last line of defense against misconfigured root path
|
|
||||||
```
|
|
||||||
|
|
||||||
Layer 7 is the catastrophic failure prevention. A misconfigured root path — pointing
|
|
||||||
cleanup at the wrong directory — means the API returns zero tracked files for a root
|
|
||||||
that actually contains thousands. Everything walks as an orphan. Everything gets deleted.
|
|
||||||
`LIDARR/SONARR/RADARR_MAX_DELETE_GB` requires `--i-know-what-im-doing` to proceed past it.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ━━━ CONFIGURATION — master.conf ━━━
|
## ━━━ CONFIGURATION — master.conf ━━━
|
||||||
|
|
||||||
### Permissions
|
### Permissions
|
||||||
@@ -122,158 +83,6 @@ MEDIA_FILE_PATTERNS=(
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Lidarr Cleanup Thresholds
|
|
||||||
|
|
||||||
```bash
|
|
||||||
LIDARR_ORPHAN_AGE=7 # days — files newer than this are RECENT (mid-import window)
|
|
||||||
LIDARR_MIN_TRACKED_PCT=80 # abort if API returns < 80% of last known count
|
|
||||||
LIDARR_MAX_DELETE_GB=50 # require --i-know-what-im-doing above this
|
|
||||||
LIDARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan
|
|
||||||
LIDARR_VERSION_MAJOR=3 # expected Lidarr major version (API safety check)
|
|
||||||
LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
|
|
||||||
LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc")
|
|
||||||
LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count" # persistent baseline
|
|
||||||
ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db" # read by coffee report
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Sonarr Cleanup Thresholds
|
|
||||||
|
|
||||||
```bash
|
|
||||||
SONARR_ORPHAN_AGE=7
|
|
||||||
SONARR_MAX_DELETE_GB=50
|
|
||||||
SONARR_IMPORT_SCAN_TIMEOUT=600
|
|
||||||
SONARR_VERSION_MAJOR=4
|
|
||||||
SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov")
|
|
||||||
SONARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa")
|
|
||||||
```
|
|
||||||
|
|
||||||
Note: `*.ts` IS in extensions — transport stream is used for Live TV recordings tracked
|
|
||||||
by Sonarr. Orphaned `.ts` recordings should be cleaned like any other orphaned episode.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Radarr Cleanup Thresholds
|
|
||||||
|
|
||||||
```bash
|
|
||||||
RADARR_ORPHAN_AGE=7
|
|
||||||
RADARR_MAX_DELETE_GB=50
|
|
||||||
RADARR_IMPORT_SCAN_TIMEOUT=600
|
|
||||||
RADARR_VERSION_MAJOR=6
|
|
||||||
RADARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "wmv" "mov")
|
|
||||||
RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa")
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Arr Sync
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ARR_SYNC_ENABLED=true
|
|
||||||
ARR_SYNC_BLOCKLIST="$DATA_DIR/arr_sync_blocklist.tsv" # tombstone file
|
|
||||||
ARR_SYNC_CONNECT_TIMEOUT=10 # SSH connect timeout in seconds
|
|
||||||
ARR_SYNC_API_TIMEOUT=60 # curl API call timeout in seconds
|
|
||||||
DOCKER_APPDATA_BASE=/mnt/user/appdata
|
|
||||||
ARR_SYNC_LIDARR_PORT=8686
|
|
||||||
ARR_SYNC_SONARR_PORT=8989
|
|
||||||
ARR_SYNC_RADARR_PORT=7878
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Arr Recovery
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ARR_IMPORT_RECOVERY_AGE=6 # hours — items newer than this are skipped
|
|
||||||
SONARR_VERSION_MAJOR=4
|
|
||||||
RADARR_VERSION_MAJOR=6
|
|
||||||
LIDARR_VERSION_MAJOR=3
|
|
||||||
ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db" # read by coffee report
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### TMDb / TVDB Removed
|
|
||||||
|
|
||||||
```bash
|
|
||||||
RADARR_DROPPED_ADD_EXCLUSION=true # add removed movies to Radarr import exclusion
|
|
||||||
SONARR_DROPPED_ADD_EXCLUSION=true # add removed series to Sonarr import exclusion
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Lidarr Missing Art
|
|
||||||
|
|
||||||
```bash
|
|
||||||
FANART_API_KEY="your-fanart-tv-api-key"
|
|
||||||
LASTFM_API_KEY="your-lastfm-api-key"
|
|
||||||
LIDARR_ART_MIN_SIZE=5000 # minimum valid download size in bytes
|
|
||||||
LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs
|
|
||||||
LIDARR_ART_RETRIES=2 # download retry attempts per image
|
|
||||||
LIDARR_ART_SLEEP_BETWEEN=1 # seconds between fanart.tv API calls (rate limit)
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Lidarr Discovery
|
|
||||||
|
|
||||||
```bash
|
|
||||||
LIDARR_DISCOVERY_THRESHOLD=70 # minimum score for Stage 1 seeds and Stage 2 adds
|
|
||||||
LIDARR_DISCOVERY_LOOKBACK_DAYS=7 # Emby play history window in days
|
|
||||||
LIDARR_DISCOVERY_MIN_PLAYS=3 # min plays before an artist is evaluated as a seed
|
|
||||||
LIDARR_DISCOVERY_MAX_ADDS=5 # max seeds (Stage 1) and max adds (Stage 2) per run
|
|
||||||
LIDARR_DISCOVERY_USER_CAP_PCT=35 # max % any one user contributes to play weight
|
|
||||||
LIDARR_DISCOVERY_REJECT_COOLDOWN=30 # days before re-evaluating a Stage 2 reject
|
|
||||||
LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db"
|
|
||||||
```
|
|
||||||
|
|
||||||
Requires `HOST*_LASTFM_API_KEY` in `host*.conf`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Radarr Discovery
|
|
||||||
|
|
||||||
```bash
|
|
||||||
RADARR_DISCOVERY_THRESHOLD=52 # minimum score to add a candidate
|
|
||||||
RADARR_DISCOVERY_LOOKBACK_DAYS=30 # Emby watch history window in days
|
|
||||||
RADARR_DISCOVERY_MAX_SEEDS=5 # max seed movies from Stage 1
|
|
||||||
RADARR_DISCOVERY_MAX_ADDS=5 # max movies to add per run
|
|
||||||
RADARR_DISCOVERY_MIN_VOTE_COUNT=100 # min TMDB votes for a candidate
|
|
||||||
RADARR_DISCOVERY_MIN_RATING=60 # min TMDB vote_average × 10 (60 = 6.0/10)
|
|
||||||
RADARR_DISCOVERY_REJECT_COOLDOWN=60 # days before re-evaluating a rejected movie
|
|
||||||
RADARR_DISCOVERY_SEED_LIBRARIES=("Movies") # Emby libraries to draw seeds from
|
|
||||||
RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db"
|
|
||||||
```
|
|
||||||
|
|
||||||
Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Sonarr Discovery
|
|
||||||
|
|
||||||
```bash
|
|
||||||
SONARR_DISCOVERY_THRESHOLD=52 # minimum score to add a candidate
|
|
||||||
SONARR_DISCOVERY_LOOKBACK_DAYS=14 # Emby episode history window in days
|
|
||||||
SONARR_DISCOVERY_MAX_SEEDS=5 # max seed series from Stage 1
|
|
||||||
SONARR_DISCOVERY_MAX_ADDS=3 # max shows to add per run (TV is a larger commitment)
|
|
||||||
SONARR_DISCOVERY_MIN_VOTE_COUNT=50 # min TMDB votes for a candidate
|
|
||||||
SONARR_DISCOVERY_MIN_RATING=65 # min TMDB vote_average × 10 (65 = 6.5/10)
|
|
||||||
SONARR_DISCOVERY_REJECT_COOLDOWN=60 # days before re-evaluating a rejected show
|
|
||||||
SONARR_DISCOVERY_USER_EPISODE_CAP=8 # max episodes per user in seed scoring
|
|
||||||
SONARR_DISCOVERY_MONITOR_MODE="all" # "all" = all seasons monitored; "future" = upcoming only
|
|
||||||
SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db"
|
|
||||||
# SONARR_EMBY_LIBRARIES is shared with emby_to_sonarr_sync — see Orchestrator Job Order
|
|
||||||
```
|
|
||||||
|
|
||||||
Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
|
|
||||||
|
|
||||||
> **MONITOR_MODE note:** Use `"all"` (default) to have Sonarr search all existing seasons
|
|
||||||
> after adding a show. `"future"` only marks upcoming seasons as monitored — shows where
|
|
||||||
> all seasons have already aired will appear unmonitored and Sonarr will not search for them.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Play State Sync
|
### Play State Sync
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -293,41 +102,8 @@ HOST1_JELLYFIN_API_KEY=""
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Upgrade Webhook
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# master.conf
|
|
||||||
WEBHOOK_PORT=7821 # 0 = disable listener
|
|
||||||
WEBHOOK_SECRET="" # auto-generated on first start if empty
|
|
||||||
```
|
|
||||||
|
|
||||||
When `WEBHOOK_PORT=0`, `start_webhook_listener.sh` exits cleanly and no listener starts.
|
|
||||||
When `WEBHOOK_SECRET` is empty, a 32-byte hex secret is generated on first start and
|
|
||||||
written back to `master.conf`. Run `Tools/webhook_setup.sh` to register the URL in arrs.
|
|
||||||
|
|
||||||
Webhook log: `/var/log/varaverk/upgrade_webhook.log`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Orchestrator Job Order
|
|
||||||
|
|
||||||
```bash
|
|
||||||
MEDIA_MAINTENANCE_JOBS=(
|
|
||||||
"Media/media_shares_permissions.sh" # 1. permissions — always first
|
|
||||||
"Media/media_cleaner.sh anime" # 2. junk removal — before orphan scan
|
|
||||||
"Media/media_cleaner.sh media" # 3.
|
|
||||||
"Arrs_Stack/lidarr_cleanup.sh" # 4. arr cleanup — after permissions + clean
|
|
||||||
"Arrs_Stack/sonarr_cleanup.sh" # 5.
|
|
||||||
"Arrs_Stack/radarr_cleanup.sh" # 6.
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ━━━ CONFIGURATION — host*.conf ━━━
|
## ━━━ CONFIGURATION — host*.conf ━━━
|
||||||
|
|
||||||
### host1.conf
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Shares this server applies permissions to
|
# Shares this server applies permissions to
|
||||||
HOST1_MEDIA_PERMISSION_SHARES=(
|
HOST1_MEDIA_PERMISSION_SHARES=(
|
||||||
@@ -356,224 +132,6 @@ HOST1_MEDIA_CLEAN_FOLDERS=(
|
|||||||
"/mnt/user/stand-up_comedy"
|
"/mnt/user/stand-up_comedy"
|
||||||
"/mnt/user/Tv_Shows"
|
"/mnt/user/Tv_Shows"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Arr connection details — must match arr settings exactly
|
|
||||||
HOST1_LIDARR_URL="http://192.168.50.2:8686"
|
|
||||||
HOST1_LIDARR_API_KEY="..."
|
|
||||||
HOST1_LIDARR_MUSIC_ROOT="/mnt/user/Music"
|
|
||||||
HOST1_LIDARR_PATH_MAP="" # container→host path translation if needed
|
|
||||||
|
|
||||||
HOST1_SONARR_URL="http://192.168.50.2:8989"
|
|
||||||
HOST1_SONARR_API_KEY="..."
|
|
||||||
HOST1_SONARR_TV_ROOT="/mnt/user/Tv_Shows"
|
|
||||||
HOST1_SONARR_PATH_MAP=""
|
|
||||||
|
|
||||||
HOST1_RADARR_URL="http://192.168.50.2:7878"
|
|
||||||
HOST1_RADARR_API_KEY="..."
|
|
||||||
HOST1_RADARR_MOVIES_ROOT="/mnt/user/Movies"
|
|
||||||
HOST1_RADARR_PATH_MAP=""
|
|
||||||
|
|
||||||
HOST1_EMBY_URL="http://192.168.50.2:8096"
|
|
||||||
HOST1_EMBY_API_KEY="..."
|
|
||||||
```
|
|
||||||
|
|
||||||
> **LIDARR/SONARR/RADARR_MUSIC/TV/MOVIES_ROOT must exactly match the Root Folder path in
|
|
||||||
> the arr's own settings.** Arr UI → Settings → Media Management → Root Folders.
|
|
||||||
> A mismatch means every file on disk looks untracked — all appear as orphans.
|
|
||||||
> MAX_DELETE_GB is the only thing standing between a path mismatch and losing your library.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ━━━ SAFE TESTING PROCEDURE ━━━
|
|
||||||
|
|
||||||
> **The arr cleanup scripts permanently delete files.** There is no recycle bin, no undo.
|
|
||||||
> Follow this procedure on first use, after any root path change, after any API key change,
|
|
||||||
> and after any significant arr library change.
|
|
||||||
|
|
||||||
### Step 1 — Dry Run With Full Logging
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lidarr_cleanup.sh --dry-run --log
|
|
||||||
sonarr_cleanup.sh --dry-run --log
|
|
||||||
radarr_cleanup.sh --dry-run --log
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2 — Review the Output
|
|
||||||
|
|
||||||
```
|
|
||||||
Are TRACKED files the ones you expect?
|
|
||||||
→ Known arr-managed files should show as TRACKED
|
|
||||||
→ If they show as ORPHAN, the root path is wrong — STOP
|
|
||||||
|
|
||||||
Is the ORPHAN count reasonable?
|
|
||||||
→ Healthy cleanup removes dozens to hundreds, not tens of thousands
|
|
||||||
→ Large count = stop, investigate root path before proceeding
|
|
||||||
|
|
||||||
Are PROTECTED patterns working?
|
|
||||||
→ Artwork (*.jpg) and subtitles (*.srt) must show as PROTECTED
|
|
||||||
→ If they show as ORPHAN, check PROTECTED_PATTERNS config
|
|
||||||
|
|
||||||
Are RECENT files being correctly skipped?
|
|
||||||
→ Files downloaded in the last 7 days should show as RECENT, not ORPHAN
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 3 — Check Numbers if Something Looks Wrong
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Root path mismatch? Compare these:
|
|
||||||
# Lidarr UI: Settings → Media Management → Root Folders
|
|
||||||
# Sonarr UI: Settings → Media Management → Root Folders
|
|
||||||
# Radarr UI: Settings → Media Management → Root Folders
|
|
||||||
# Must exactly match LIDARR_MUSIC_ROOT / SONARR_TV_ROOT / RADARR_MOVIES_ROOT
|
|
||||||
|
|
||||||
# Is the arr running?
|
|
||||||
docker ps | grep -E "Lidarr|Sonarr|Radarr"
|
|
||||||
|
|
||||||
# Library scan not complete?
|
|
||||||
# Trigger manual scan in arr UI and wait for completion
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 4 — Run Live
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Only after dry run review passes.
|
|
||||||
lidarr_cleanup.sh
|
|
||||||
sonarr_cleanup.sh
|
|
||||||
radarr_cleanup.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 5 — Verify in Arr UI
|
|
||||||
|
|
||||||
```
|
|
||||||
Library count — should not have dropped significantly
|
|
||||||
healthy cleanup removes a small number, not a large percentage
|
|
||||||
Missing files — check if any monitored content shows as missing
|
|
||||||
Emby library — should show no ghost entries (notify_emby_scan handles this automatically)
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ━━━ PROCEDURES ━━━
|
|
||||||
|
|
||||||
### Adding a New Arr
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. Copy radarr_cleanup.sh as template
|
|
||||||
cp radarr_cleanup.sh readarr_cleanup.sh
|
|
||||||
|
|
||||||
# 2. Replace RADARR_ prefix with READARR_ throughout
|
|
||||||
# Update API endpoint, tracked file API path, extension list, protected patterns
|
|
||||||
|
|
||||||
# 3. Add to host*.conf
|
|
||||||
HOST1_READARR_URL="http://192.168.50.2:8787"
|
|
||||||
HOST1_READARR_API_KEY="your-api-key"
|
|
||||||
HOST1_READARR_BOOKS_ROOT="/mnt/user/Books"
|
|
||||||
|
|
||||||
# 4. Add thresholds to master.conf
|
|
||||||
READARR_ORPHAN_AGE=7
|
|
||||||
READARR_MAX_DELETE_GB=50
|
|
||||||
READARR_EXTENSIONS=("epub" "pdf" "mobi" "azw3" "cbz" "cbr")
|
|
||||||
READARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo")
|
|
||||||
|
|
||||||
# 5. Add to MEDIA_MAINTENANCE_JOBS in master.conf
|
|
||||||
MEDIA_MAINTENANCE_JOBS=(
|
|
||||||
...existing jobs...
|
|
||||||
"Media/readarr_cleanup.sh"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
media_management.sh picks it up automatically. No orchestrator changes needed.
|
|
||||||
Run `--dry-run --log` before scheduling.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Managing the Arr Sync Blocklist
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Add item to blocklist (removes from all arrs + tombstones the ID)
|
|
||||||
arr_sync.sh --blocklist-add lidarr <musicbrainz-artist-id> "reason"
|
|
||||||
arr_sync.sh --blocklist-add sonarr <tvdb-series-id> "reason"
|
|
||||||
arr_sync.sh --blocklist-add radarr <tmdb-movie-id> "reason"
|
|
||||||
|
|
||||||
# Remove from blocklist (un-tombstones the ID — does NOT re-add to arrs)
|
|
||||||
arr_sync.sh --blocklist-remove lidarr <id>
|
|
||||||
|
|
||||||
# View all blocklisted IDs
|
|
||||||
arr_sync.sh --blocklist-list
|
|
||||||
```
|
|
||||||
|
|
||||||
`--blocklist-add` is the only destructive operation — it simultaneously:
|
|
||||||
1. Writes the tombstone entry to the blocklist TSV file
|
|
||||||
2. Deletes the item from the local arr API (no file deletion)
|
|
||||||
3. SSHes each remote node and deletes from their arr API
|
|
||||||
|
|
||||||
Files become orphans on all nodes — arr_cleanup removes them on the next run.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ━━━ TROUBLESHOOTING ━━━
|
|
||||||
|
|
||||||
### Arr Cleanup Deleting Files It Shouldn't
|
|
||||||
|
|
||||||
```
|
|
||||||
1. Check the protected patterns — artwork and subtitles must be listed
|
|
||||||
LIDARR_PROTECTED_PATTERNS / SONARR_PROTECTED_PATTERNS / RADARR_PROTECTED_PATTERNS
|
|
||||||
|
|
||||||
2. Check the root path matches arr settings exactly
|
|
||||||
Run: lidarr_cleanup.sh --status (shows configured root path)
|
|
||||||
Compare: Lidarr UI → Settings → Media Management → Root Folders
|
|
||||||
|
|
||||||
3. Check if files are truly orphaned
|
|
||||||
Run: lidarr_cleanup.sh --dry-run --log
|
|
||||||
Look for the specific file — verify it shows ORPHAN, not PROTECTED or TRACKED
|
|
||||||
```
|
|
||||||
|
|
||||||
### Arr Cleanup Aborting at Safety Layer 6 (Tracked Count Drop)
|
|
||||||
|
|
||||||
```
|
|
||||||
API returned far fewer tracked files than last run.
|
|
||||||
Possible causes:
|
|
||||||
- Arr database was recently rebuilt from scratch
|
|
||||||
- Large manual library removal
|
|
||||||
- Path map mismatch after arr migration
|
|
||||||
|
|
||||||
If intentional (library intentionally reduced):
|
|
||||||
Delete LIDARR_TRACKED_COUNT_FILE to reset the baseline
|
|
||||||
Run cleanup once — it will establish a new baseline
|
|
||||||
|
|
||||||
If unintentional:
|
|
||||||
Investigate before proceeding — the arr may have a problem
|
|
||||||
```
|
|
||||||
|
|
||||||
### Arr Sync Not Picking Up New Content
|
|
||||||
|
|
||||||
```
|
|
||||||
Is ARR_SYNC_ENABLED=true in master.conf?
|
|
||||||
|
|
||||||
Can this host SSH to the remote without password?
|
|
||||||
→ ssh -i [SSH_KEY] root@[remote-tailscale-ip] "hostname"
|
|
||||||
|
|
||||||
Is the arr accessible on the remote?
|
|
||||||
→ arr_sync.sh --status (shows each node's arr reachability)
|
|
||||||
→ arr_sync.sh --log (verbose output per-node, per-arr)
|
|
||||||
|
|
||||||
Is the item in the blocklist?
|
|
||||||
→ arr_sync.sh --blocklist-list
|
|
||||||
```
|
|
||||||
|
|
||||||
### Emby Still Showing Ghost Entries After Cleanup
|
|
||||||
|
|
||||||
```
|
|
||||||
notify_emby_scan() is called automatically after every arr cleanup deletion.
|
|
||||||
If ghosts persist:
|
|
||||||
1. Is Emby's API responding?
|
|
||||||
curl -s "http://[emby-ip]:8096/System/Info/Public"
|
|
||||||
|
|
||||||
2. Is EMBY_URL / EMBY_API_KEY correct in host*.conf?
|
|
||||||
Run: sonarr_cleanup.sh --status (shows Emby config)
|
|
||||||
|
|
||||||
3. Trigger manually in Emby:
|
|
||||||
Library → Manage Library → Clean Missing Files
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -582,16 +140,11 @@ If ghosts persist:
|
|||||||
|
|
||||||
All scripts have two output levels controlled by `--log`.
|
All scripts have two output levels controlled by `--log`.
|
||||||
|
|
||||||
Without `--log`, each script processes silently and always concludes with a summary
|
Without `--log`, each script processes silently and concludes with a summary block.
|
||||||
block: identity, duration, counts (files removed, items added, arrs cleaned), and a
|
Warnings and errors are always visible.
|
||||||
status line. Warnings and errors are always visible.
|
|
||||||
|
|
||||||
With `--log`, per-item detail appears: individual titles being processed, API query
|
With `--log`, per-item detail appears — individual shares being processed, file counts,
|
||||||
progress, per-node sync results, and per-file examination output. Use this when
|
per-server sync results.
|
||||||
debugging unexpected results or validating configuration before the first scheduled run.
|
|
||||||
|
|
||||||
Dry-run output follows the same tiers — `--dry-run` alone shows the summary of what
|
|
||||||
would happen; `--dry-run --log` shows the full per-item preview list.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -638,44 +191,16 @@ Show every file examined, not just those removed.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### lidarr_cleanup.sh / sonarr_cleanup.sh / radarr_cleanup.sh
|
|
||||||
|
|
||||||
`[script] --dry-run --log`
|
|
||||||
Preview every classification decision. **Always run this first.** See Safe Testing Procedure.
|
|
||||||
|
|
||||||
`[script]`
|
|
||||||
Live run — deletes confirmed orphans and junk, triggers Emby clean.
|
|
||||||
|
|
||||||
`[script] --log`
|
|
||||||
Live run with verbose per-file output.
|
|
||||||
|
|
||||||
`[script] --status`
|
|
||||||
Show configuration, API status, tracked file count, and last run stats.
|
|
||||||
|
|
||||||
`[script] --i-know-what-im-doing`
|
|
||||||
Bypass the MAX_DELETE_GB size threshold. Required when deletion exceeds the configured
|
|
||||||
limit. Long flag name is intentional — cannot be added accidentally.
|
|
||||||
|
|
||||||
`[script] --skip-strike-list`
|
|
||||||
Bypass the ORPHAN_AGE age check. Deletes RECENT files too — files that are under the
|
|
||||||
age threshold. Use when you know recent downloads are actually orphans.
|
|
||||||
|
|
||||||
`[script] --i-know-what-im-doing --skip-strike-list`
|
|
||||||
**NUCLEAR MODE** — age check and size threshold both bypassed. Deletes on first pass.
|
|
||||||
Use when you want a clean one-pass wipe of everything the arr doesn't track.
|
|
||||||
No recovery possible after deletion.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### play_state_sync.sh
|
### play_state_sync.sh
|
||||||
|
|
||||||
`play_state_sync.sh`
|
`play_state_sync.sh`
|
||||||
Sync played/unplayed state and resume positions from local Emby to the remote Emby.
|
Sync played/unplayed state and resume positions from local Emby to the remote Emby.
|
||||||
Only items played within PLAY_SYNC_DAYS are synced.
|
Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced.
|
||||||
|
|
||||||
`play_state_sync.sh --full`
|
`play_state_sync.sh --full`
|
||||||
Ignore PLAY_SYNC_DAYS — sync all played items regardless of age. May be slow on large
|
Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be
|
||||||
libraries. Use after a new Emby install or database restore to rebuild full play history.
|
slow on large libraries. Use after a new Emby install or database restore to rebuild
|
||||||
|
full play history.
|
||||||
|
|
||||||
`play_state_sync.sh --dry-run`
|
`play_state_sync.sh --dry-run`
|
||||||
Show what would be synced without writing any state.
|
Show what would be synced without writing any state.
|
||||||
@@ -685,139 +210,3 @@ Show configured servers, reachability, and user counts.
|
|||||||
|
|
||||||
`play_state_sync.sh --log`
|
`play_state_sync.sh --log`
|
||||||
Verbose output — show each item comparison.
|
Verbose output — show each item comparison.
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### arrs_failed_stalled_recovery.sh
|
|
||||||
|
|
||||||
`arrs_failed_stalled_recovery.sh`
|
|
||||||
Check all configured arrs for failed imports and stalled downloads. Blocklist + remove +
|
|
||||||
re-search for each problem item.
|
|
||||||
|
|
||||||
`arrs_failed_stalled_recovery.sh --dry-run`
|
|
||||||
Show what would be actioned per arr without making any changes.
|
|
||||||
|
|
||||||
`arrs_failed_stalled_recovery.sh --status`
|
|
||||||
Show configuration, arr reachability, and last recovery stats.
|
|
||||||
|
|
||||||
`arrs_failed_stalled_recovery.sh --log`
|
|
||||||
Verbose output per item per arr.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### arr_sync.sh
|
|
||||||
|
|
||||||
`arr_sync.sh`
|
|
||||||
Sync all arr types across all configured nodes.
|
|
||||||
|
|
||||||
`arr_sync.sh --dry-run`
|
|
||||||
Show what would be added/removed on each node without making changes.
|
|
||||||
|
|
||||||
`arr_sync.sh --status`
|
|
||||||
Show node configuration, arr reachability, and blocklist count.
|
|
||||||
|
|
||||||
`arr_sync.sh --log`
|
|
||||||
Verbose per-node, per-arr output.
|
|
||||||
|
|
||||||
`arr_sync.sh --blocklist-add [arr] [id] "[reason]"`
|
|
||||||
Remove item from all arrs and tombstone the ID. See Procedures above.
|
|
||||||
|
|
||||||
`arr_sync.sh --blocklist-remove [arr] [id]`
|
|
||||||
Remove tombstone — does NOT re-add item to arrs.
|
|
||||||
|
|
||||||
`arr_sync.sh --blocklist-list`
|
|
||||||
Show all tombstoned IDs.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### radarr_tmdb_removed.sh / sonarr_tvdb_removed.sh
|
|
||||||
|
|
||||||
`[script]`
|
|
||||||
Remove records for entries with status="deleted" (dropped from upstream database).
|
|
||||||
Files are kept. Import exclusion is added.
|
|
||||||
|
|
||||||
`[script] --delete-files`
|
|
||||||
Also delete associated files from disk. Most dropped entries have no files — they were
|
|
||||||
announced movies/series that were never downloaded.
|
|
||||||
|
|
||||||
`[script] --dry-run`
|
|
||||||
Preview what would be removed without making changes.
|
|
||||||
|
|
||||||
`[script] --status`
|
|
||||||
Show arr connection status and current count of dropped entries.
|
|
||||||
|
|
||||||
`[script] --log`
|
|
||||||
Verbose per-entry output.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### lidarr_missing_art.sh
|
|
||||||
|
|
||||||
`lidarr_missing_art.sh`
|
|
||||||
Fetch all missing album and artist artwork from fanart.tv and fallback sources.
|
|
||||||
Never overwrites existing files.
|
|
||||||
|
|
||||||
`lidarr_missing_art.sh --dry-run`
|
|
||||||
Show what would be downloaded without writing any files.
|
|
||||||
|
|
||||||
`lidarr_missing_art.sh --status`
|
|
||||||
Show configuration and API key status.
|
|
||||||
|
|
||||||
`lidarr_missing_art.sh --log`
|
|
||||||
Verbose per-album, per-artist output.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### playback_aware_lidarr_discovery.sh
|
|
||||||
|
|
||||||
`playback_aware_lidarr_discovery.sh`
|
|
||||||
Score Emby play history, run Last.fm getSimilar on top artists, add candidates above
|
|
||||||
threshold to Lidarr. Triggers ArtistSearch immediately after each successful add.
|
|
||||||
|
|
||||||
`playback_aware_lidarr_discovery.sh --dry-run`
|
|
||||||
Score and rank all Stage 1 seeds and Stage 2 candidates. No Lidarr API calls. No writes
|
|
||||||
to history file. Shows exactly what would be added and at what score.
|
|
||||||
|
|
||||||
`playback_aware_lidarr_discovery.sh --status`
|
|
||||||
Show config values, history file path and size, and API key status.
|
|
||||||
|
|
||||||
`playback_aware_lidarr_discovery.sh --log`
|
|
||||||
Verbose per-artist scoring output for both stages.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### playback_aware_radarr_discovery.sh
|
|
||||||
|
|
||||||
`playback_aware_radarr_discovery.sh`
|
|
||||||
Score recently watched Emby movies, run TMDB recommendations on seeds, add candidates
|
|
||||||
above threshold to Radarr. Triggers MoviesSearch immediately after each successful add.
|
|
||||||
|
|
||||||
`playback_aware_radarr_discovery.sh --dry-run`
|
|
||||||
Score and rank all Stage 1 seeds and Stage 2 candidates. No Radarr API calls. No writes
|
|
||||||
to history file.
|
|
||||||
|
|
||||||
`playback_aware_radarr_discovery.sh --status`
|
|
||||||
Show config values, history file path and size, and API key status.
|
|
||||||
|
|
||||||
`playback_aware_radarr_discovery.sh --log`
|
|
||||||
Verbose per-movie scoring output for both stages.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### playback_aware_sonarr_discovery.sh
|
|
||||||
|
|
||||||
`playback_aware_sonarr_discovery.sh`
|
|
||||||
Score recently watched Emby series (weighted by user diversity), run TMDB TV
|
|
||||||
recommendations on seeds, add candidates above threshold to Sonarr. Triggers SeriesSearch
|
|
||||||
immediately after each successful add.
|
|
||||||
|
|
||||||
`playback_aware_sonarr_discovery.sh --dry-run`
|
|
||||||
Score and rank all Stage 1 seeds and Stage 2 candidates. No Sonarr API calls. No writes
|
|
||||||
to history file.
|
|
||||||
|
|
||||||
`playback_aware_sonarr_discovery.sh --status`
|
|
||||||
Show config values, history file path and size, and API key status.
|
|
||||||
|
|
||||||
`playback_aware_sonarr_discovery.sh --log`
|
|
||||||
Verbose per-series scoring output — shows user diversity, recency, and volume scores per
|
|
||||||
seed; breadth, rating, and votes scores per candidate.
|
|
||||||
|
|||||||
+22
-201
@@ -1,21 +1,14 @@
|
|||||||
# ━━━━━ MEDIA ━━━━━
|
# ━━━━━ MEDIA ━━━━━
|
||||||
|
|
||||||
Library health, consistency, sync, and behavior-driven discovery for a multi-server arr
|
Foundation-level media library management — permissions, junk removal, and play state
|
||||||
stack. Correct permissions so arrs can manage files. Junk removal so orphan detection
|
sync. These three scripts run before and independently of arr stack operations.
|
||||||
isn't confused by scene debris. Library sync so every node tracks the same content.
|
|
||||||
Orphan cleanup against live arr APIs so deleted content actually leaves disk. Emby
|
|
||||||
notified automatically after every deletion. Watch state synced across Emby/Jellyfin
|
|
||||||
every 30 minutes. Quality upgrades propagate to all nodes immediately via webhook.
|
|
||||||
Weekly discovery adds new music, movies, and TV shows based on what you actually play.
|
|
||||||
|
|
||||||
> **These scripts permanently delete files.** The arr cleanup scripts are protected by
|
For arr stack scripts (orphan cleanup, release fixer, sync, discovery, webhooks) see
|
||||||
> multiple safety layers that must all pass before anything is touched — but dry runs and
|
`Arrs_Stack/README-Arrs_Stack.md`.
|
||||||
> log review are still the right first step on any new system or after any configuration
|
|
||||||
> change. The testing procedure in Manual-Media.md exists for a reason.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
## ━━━ THE PROBLEMS THAT BUILT THIS ━━━
|
||||||
|
|
||||||
**Files Owned by Root That Arrs Can't Touch**
|
**Files Owned by Root That Arrs Can't Touch**
|
||||||
Download clients without explicit PUID/PGID write files owned by root. Arrs running as
|
Download clients without explicit PUID/PGID write files owned by root. Arrs running as
|
||||||
@@ -33,149 +26,48 @@ noise, making it hard to spot actual orphaned media.
|
|||||||
The fix: `media_cleaner.sh` runs before any arr cleanup and removes all known junk
|
The fix: `media_cleaner.sh` runs before any arr cleanup and removes all known junk
|
||||||
patterns first. By the time arr cleanup runs, every untracked file is actual media.
|
patterns first. By the time arr cleanup runs, every untracked file is actual media.
|
||||||
|
|
||||||
**Deleted Shows and Removed Albums Still on Disk**
|
**Watch State Diverging Across Servers**
|
||||||
When you remove a series from Sonarr and the delete command fails — permission issue,
|
With two Emby servers, played status and resume positions diverge — a film marked watched
|
||||||
container wasn't running, path mismatch — the files stay permanently. Over years on an
|
on HOST1 shows as unwatched on HOST2. Two users on different servers get different
|
||||||
active library this accumulates significantly.
|
continue-watching rows.
|
||||||
The fix: arr cleanup scripts query the live API for every tracked file path, walk the
|
The fix: `play_state_sync.sh` syncs watched/played state and resume positions every 30
|
||||||
disk, and delete anything absent from the API response that's old enough to be past the
|
minutes. Newest timestamp wins. Both servers always reflect the same play history.
|
||||||
import window.
|
|
||||||
|
|
||||||
**Emby Showing Ghost Entries After Cleanup**
|
|
||||||
After arr cleanup deletes files, Emby still shows them until its next scheduled scan —
|
|
||||||
potentially hours later. Users see broken entries that produce "file not found" errors.
|
|
||||||
The fix: `notify_emby_scan()` is called automatically after every deletion. Triggers
|
|
||||||
Emby's "Clean Missing Files" task immediately.
|
|
||||||
|
|
||||||
**No Safety Net on Deletion Size**
|
|
||||||
A misconfigured root path — pointing cleanup at the wrong directory — means the API
|
|
||||||
returns zero tracked files for a root that actually contains thousands. Every file walks
|
|
||||||
as an orphan. Everything gets deleted. This is the catastrophic failure mode.
|
|
||||||
The fix: `LIDARR/SONARR/RADARR_MAX_DELETE_GB` — if total deletion size exceeds the
|
|
||||||
limit, the script stops and requires `--i-know-what-im-doing` to proceed. The flag name
|
|
||||||
is long and annoying by design. It cannot be added by accident.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||||
|
|
||||||
Scripts divide into five functional areas:
|
|
||||||
|
|
||||||
**Library Foundation**
|
**Library Foundation**
|
||||||
`media_shares_permissions.sh` — normalize ownership and permissions daily. Runs first in
|
`media_shares_permissions.sh` — normalize ownership and permissions daily. Runs first in
|
||||||
every maintenance window because arr cleanup depends on correct ownership to delete files.
|
every maintenance window because arr cleanup depends on correct ownership to delete files.
|
||||||
|
|
||||||
**Junk Removal**
|
**Junk Removal**
|
||||||
`media_cleaner.sh` — remove scene debris and tool artifacts before orphan scan. Runs
|
`media_cleaner.sh` — remove scene debris and tool artifacts before orphan scan. Runs
|
||||||
second, before any arr cleanup, so orphan detection only encounters actual media files.
|
before any arr cleanup so orphan detection only encounters actual media files.
|
||||||
|
|
||||||
**Library Sync**
|
|
||||||
`arr_sync.sh` — full-mesh arr library sync across all nodes. Every node syncs with every
|
|
||||||
other — union model, no hierarchy. Once arrs agree on what to track, rsync spreads the
|
|
||||||
actual files. The architectural shift from file-first sync to arr-first sync.
|
|
||||||
|
|
||||||
**Orphan Cleanup**
|
|
||||||
`lidarr_cleanup.sh`, `sonarr_cleanup.sh`, `radarr_cleanup.sh` — API-verified orphan
|
|
||||||
removal. Five classification categories (TRACKED/PROTECTED/ORPHAN/JUNK/RECENT), seven
|
|
||||||
safety layers, automatic Emby notification after deletion.
|
|
||||||
|
|
||||||
**Database Hygiene**
|
|
||||||
`radarr_tmdb_removed.sh`, `sonarr_tvdb_removed.sh` — remove entries that upstream
|
|
||||||
databases have dropped (TMDb/TVDB status="deleted"). These generate health warnings in
|
|
||||||
arrs and can never be monitored or downloaded. Most are announced-but-never-released
|
|
||||||
entries. Files are kept by default — most have none.
|
|
||||||
|
|
||||||
**Library Enrichment**
|
|
||||||
`arrs_failed_stalled_recovery.sh` — detect and recover failed imports and stalled
|
|
||||||
downloads across all arrs. Blocklists the bad release and triggers a re-search — hands-
|
|
||||||
free overnight recovery.
|
|
||||||
`lidarr_missing_art.sh` — fetch missing album and artist artwork from fanart.tv and
|
|
||||||
fallback sources. Never overwrites existing files.
|
|
||||||
|
|
||||||
**Play State Sync**
|
**Play State Sync**
|
||||||
`play_state_sync.sh` — syncs watched/played state and resume positions across all
|
`play_state_sync.sh` — syncs watched/played state and resume positions across all
|
||||||
configured Emby and Jellyfin servers. Newest timestamp wins. Runs every 30 minutes
|
configured Emby and Jellyfin servers. Newest timestamp wins. Runs every 30 minutes
|
||||||
via critical_sync_maintenance.sh.
|
via critical_sync_maintenance.sh.
|
||||||
|
|
||||||
**Upgrade Propagation**
|
|
||||||
`start_webhook_listener.sh` — Node.js HTTP server that receives Sonarr/Radarr/Lidarr
|
|
||||||
OnUpgrade webhooks. Continuous; started at array start. Writes to
|
|
||||||
`/var/log/varaverk/upgrade_webhook.log`.
|
|
||||||
`upgrade_webhook_handler.sh` — triggered by the webhook listener. Pushes the upgraded
|
|
||||||
item folder to every remote node immediately, then triggers an arr library rescan on
|
|
||||||
each remote so the upgraded file is accepted without triggering a redundant quality search.
|
|
||||||
|
|
||||||
**Discovery**
|
|
||||||
`playback_aware_lidarr_discovery.sh` — behavior-driven music discovery. Scores your
|
|
||||||
Emby play history, runs Last.fm getSimilar on top artists, adds the best matches to
|
|
||||||
Lidarr. 0–5 meaningful adds per week.
|
|
||||||
`playback_aware_radarr_discovery.sh` — behavior-driven movie discovery. Scores recently
|
|
||||||
watched movies, runs TMDB recommendations on seeds, adds top candidates to Radarr.
|
|
||||||
`playback_aware_sonarr_discovery.sh` — behavior-driven TV discovery. Scores recently
|
|
||||||
watched series weighted by user diversity, runs TMDB TV recommendations on seeds, adds
|
|
||||||
top shows to Sonarr. Multi-user design: one person binge-watching does not dominate seeds.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ EXECUTION ORDER ━━━
|
## ━━━ EXECUTION ORDER ━━━
|
||||||
|
|
||||||
Scripts run in multiple contexts — not all are part of the daily maintenance window:
|
**Daily via `daily_sync_maintenance.sh` (DAILY_MAINTENANCE_SCRIPTS — runs first):**
|
||||||
|
|
||||||
**Daily via `media_management.sh` (MEDIA_MAINTENANCE_JOBS):**
|
|
||||||
|
|
||||||
```
|
```
|
||||||
1. media_shares_permissions.sh — permissions first — arr cleanup depends on this
|
1. media_shares_permissions.sh — permissions first — arr cleanup depends on this
|
||||||
2. media_cleaner.sh anime — junk before orphan scan
|
2. media_cleaner.sh anime — junk before orphan scan
|
||||||
3. media_cleaner.sh media
|
3. media_cleaner.sh media
|
||||||
4. lidarr_cleanup.sh — after permissions + clean
|
|
||||||
5. sonarr_cleanup.sh
|
|
||||||
6. radarr_cleanup.sh
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Weekly arr sync (before rsync in weekly_sync_maintenance.sh):**
|
These complete before any Arrs_Stack/ scripts run.
|
||||||
|
|
||||||
```
|
|
||||||
arr_sync.sh — arrs agree on library → rsync then spreads the files
|
|
||||||
```
|
|
||||||
|
|
||||||
**Daily recovery (separate schedule — 5am or every 6hr):**
|
|
||||||
|
|
||||||
```
|
|
||||||
arrs_failed_stalled_recovery.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
**Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS in master.conf):**
|
|
||||||
|
|
||||||
```
|
|
||||||
playback_aware_lidarr_discovery.sh — score play history → Last.fm similar → add to Lidarr
|
|
||||||
playback_aware_radarr_discovery.sh — score watch history → TMDB recommendations → add to Radarr
|
|
||||||
playback_aware_sonarr_discovery.sh — score episode history → TMDB TV recommendations → add to Sonarr
|
|
||||||
```
|
|
||||||
|
|
||||||
**Every 30 min via `critical_sync_maintenance.sh` (CRITICAL_MAINTENANCE_SCRIPTS):**
|
**Every 30 min via `critical_sync_maintenance.sh` (CRITICAL_MAINTENANCE_SCRIPTS):**
|
||||||
|
|
||||||
```
|
```
|
||||||
play_state_sync.sh — sync watched/resume state across Emby + Jellyfin
|
play_state_sync.sh — sync watched/resume state across Emby + Jellyfin
|
||||||
```
|
|
||||||
|
|
||||||
**Continuous (started by `array_started.sh`):**
|
|
||||||
|
|
||||||
```
|
|
||||||
start_webhook_listener.sh — Node.js webhook server; dispatch upgrade_webhook_handler.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
**On every arr upgrade (triggered by webhook):**
|
|
||||||
|
|
||||||
```
|
|
||||||
upgrade_webhook_handler.sh — push upgraded folder to all remote nodes + trigger arr rescan
|
|
||||||
```
|
|
||||||
|
|
||||||
**Ad-hoc or separate schedule:**
|
|
||||||
|
|
||||||
```
|
|
||||||
lidarr_missing_art.sh — fetch missing artwork
|
|
||||||
radarr_tmdb_removed.sh — weekly cleanup of TMDb-dropped entries
|
|
||||||
sonarr_tvdb_removed.sh — weekly cleanup of TVDB-dropped entries
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Why permissions before everything else:** arr cleanup needs `nobody:users` ownership to
|
**Why permissions before everything else:** arr cleanup needs `nobody:users` ownership to
|
||||||
@@ -186,96 +78,25 @@ but stays on disk.
|
|||||||
orphans. Removing them first means orphan detection only finds actual media. Cleaner
|
orphans. Removing them first means orphan detection only finds actual media. Cleaner
|
||||||
output, more accurate detection.
|
output, more accurate detection.
|
||||||
|
|
||||||
**Why arr_sync before rsync:** once arrs agree on what to track, rsync spreads the actual
|
|
||||||
files. An upgrade on one node — new tracked path, old path no longer in API — gets
|
|
||||||
cleaned by arr_cleanup on all nodes after the next sync cycle.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ HOST AWARENESS ━━━
|
## ━━━ HOST AWARENESS ━━━
|
||||||
|
|
||||||
Scripts run on both servers via `detect_hosts()`, which aliases all `HOST*_` prefixed vars
|
Scripts run on both servers via `detect_hosts()`, which aliases all `HOST*_` prefixed vars
|
||||||
to their unprefixed names at runtime. No manual `HOST1`/`HOST2` comparisons exist in any script.
|
to their unprefixed names at runtime.
|
||||||
|
|
||||||
arr_sync.sh keeps all arr databases in bidirectional union — either server can download
|
|
||||||
to any share. Arr cleanup uses the union model: a file is only an orphan if neither arr
|
|
||||||
on either server has it indexed. Arr scripts check the aliased URL — if empty (arr not
|
|
||||||
configured on this host), they exit cleanly with no action.
|
|
||||||
|
|
||||||
Permissions and cleaner scripts run locally against each server's own shares, defined
|
Permissions and cleaner scripts run locally against each server's own shares, defined
|
||||||
in `HOST*_MEDIA_PERMISSION_SHARES` and `HOST*_MEDIA_CLEAN_FOLDERS` in host*.conf.
|
in `HOST*_MEDIA_PERMISSION_SHARES` and `HOST*_MEDIA_CLEAN_FOLDERS` in host*.conf.
|
||||||
|
|
||||||
|
`play_state_sync.sh` reads both servers' Emby/Jellyfin endpoints from host*.conf and
|
||||||
|
syncs between them.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||||
|
|
||||||
| Script | Role | When It Runs |
|
| Script | Role | When It Runs |
|
||||||
|--------|------|--------------|
|
|--------|------|--------------|
|
||||||
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily via `media_management.sh` |
|
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
|
||||||
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily via `media_management.sh` |
|
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
|
||||||
| `arr_sync.sh` | Full-mesh arr library sync — all nodes track the same content | Weekly before rsync |
|
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
|
||||||
| `lidarr_cleanup.sh` | Delete orphaned music files not tracked by Lidarr | Daily via `media_management.sh` |
|
|
||||||
| `sonarr_cleanup.sh` | Delete orphaned TV files not tracked by Sonarr | Daily via `media_management.sh` |
|
|
||||||
| `radarr_cleanup.sh` | Delete orphaned movie files not tracked by Radarr | Daily via `media_management.sh` |
|
|
||||||
| `arrs_failed_stalled_recovery.sh` | Auto-recover failed imports and stalled downloads | Daily (5am or every 6hr) |
|
|
||||||
| `lidarr_missing_art.sh` | Fetch missing album and artist artwork | Ad-hoc or separate schedule |
|
|
||||||
| `radarr_tmdb_removed.sh` | Remove movies dropped from TMDb | Ad-hoc or weekly |
|
|
||||||
| `sonarr_tvdb_removed.sh` | Remove series dropped from TVDB | Ad-hoc or weekly |
|
|
||||||
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min via `critical_sync_maintenance.sh` |
|
|
||||||
| `start_webhook_listener.sh` | Node.js webhook server — receive arr OnUpgrade and dispatch handler | Continuous (started by `array_started.sh`) |
|
|
||||||
| `upgrade_webhook_handler.sh` | Push upgraded item folder to remote nodes + trigger arr rescan | On each arr upgrade (via webhook) |
|
|
||||||
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly via `weekly_sync_maintenance.sh` |
|
|
||||||
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly via `weekly_sync_maintenance.sh` |
|
|
||||||
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly via `weekly_sync_maintenance.sh` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
|
||||||
|
|
||||||
```
|
|
||||||
Weekly arr sync (before rsync):
|
|
||||||
arr_sync.sh ──────────────── syncs tracked IDs across all nodes
|
|
||||||
│ union model: any node adds → all nodes get it
|
|
||||||
│
|
|
||||||
└── then rsync spreads the actual files to all nodes
|
|
||||||
└── then arr_cleanup removes orphans on all nodes (old paths, removed content)
|
|
||||||
|
|
||||||
Daily maintenance window (media_management.sh):
|
|
||||||
media_shares_permissions.sh
|
|
||||||
│ (permissions correct — arr can now delete files)
|
|
||||||
▼
|
|
||||||
media_cleaner.sh (anime + media)
|
|
||||||
│ (junk removed — orphan scan finds only actual media)
|
|
||||||
▼
|
|
||||||
lidarr_cleanup.sh ──────────── queries Lidarr API → walks /Music → deletes orphans
|
|
||||||
sonarr_cleanup.sh ──────────── queries Sonarr API → walks /Tv_Shows → deletes orphans
|
|
||||||
radarr_cleanup.sh ──────────── queries Radarr API → walks /Movies → deletes orphans
|
|
||||||
│
|
|
||||||
└── each cleanup → notify_emby_scan() → Emby removes ghost entries
|
|
||||||
|
|
||||||
Daily recovery:
|
|
||||||
arrs_failed_stalled_recovery.sh ── importFailed/stalled → blocklist → re-search
|
|
||||||
|
|
||||||
Weekly discovery (WEEKLY_MAINTENANCE_SCRIPTS):
|
|
||||||
playback_aware_lidarr_discovery.sh ─ Emby plays → Last.fm similar → top candidates → Lidarr
|
|
||||||
playback_aware_radarr_discovery.sh ─ Emby watches → TMDB recommendations → top candidates → Radarr
|
|
||||||
playback_aware_sonarr_discovery.sh ─ Emby episodes → TMDB TV recommendations → top candidates → Sonarr
|
|
||||||
│
|
|
||||||
└── each discovery script fires arr search immediately after successful add
|
|
||||||
|
|
||||||
Every 30 min (critical_sync_maintenance.sh CRITICAL_MAINTENANCE_SCRIPTS):
|
|
||||||
play_state_sync.sh ─── newest timestamp wins → watched/resume state synced
|
|
||||||
across all configured Emby + Jellyfin servers
|
|
||||||
|
|
||||||
Continuous (started by array_started.sh):
|
|
||||||
start_webhook_listener.sh ── Node.js HTTP server listens on WEBHOOK_PORT
|
|
||||||
│ arr OnUpgrade fires webhook → POST to http://HOST_LAN_IP:WEBHOOK_PORT/webhook?key=SECRET
|
|
||||||
└── upgrade_webhook_handler.sh
|
|
||||||
├── rsync upgraded folder → all remote nodes immediately
|
|
||||||
└── trigger arr library rescan on each remote (accept new file, no quality search)
|
|
||||||
|
|
||||||
Ad-hoc enrichment:
|
|
||||||
lidarr_missing_art.sh ─────── discovers missing artwork → fetches from fanart.tv
|
|
||||||
radarr_tmdb_removed.sh ────── status="deleted" → remove from Radarr + add exclusion
|
|
||||||
sonarr_tvdb_removed.sh ────── status="deleted" → remove from Sonarr + add exclusion
|
|
||||||
```
|
|
||||||
|
|||||||
@@ -7,14 +7,9 @@ claude
|
|||||||
need a script that check file locations. like if a kids movie is in movies. it needs to detect this and update the location in the arrs. so it would tell radarr to move shrek from movies to kids movies and trck the new file
|
need a script that check file locations. like if a kids movie is in movies. it needs to detect this and update the location in the arrs. so it would tell radarr to move shrek from movies to kids movies and trck the new file
|
||||||
|
|
||||||
|
|
||||||
look at a way to better determine when a container was manually shutdown for docker watchdog
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
we may need to add this to our preflight before lidarr cleanup. check if lidarr need corrected before
|
|
||||||
removal. since we have a strike list we can save results of previous scan then run this script then
|
|
||||||
continue. lidarr has been wierd for the last year. maybe we
|
|
||||||
────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -27,46 +22,6 @@ we may need to add this to our preflight before lidarr cleanup. check if lidarr
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
. fix fallback strike list timing, maybe 30 seconds. them a t 90 seconds 3 stike triggers. just gotta test buffer. never had the strike system
|
|
||||||
. verify silent toggle switches back on good notifications
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Saved. Here's where we land and what to pick up:
|
|
||||||
|
|
||||||
What happened: Lidarr was silently discarding valid files because it selected the wrong MusicBrainz release edition (Brazil CD, Japan Digital, etc.) while your files were tagged for a different one (US CD, Vinyl). The track IDs don't match across editions, so RescanFolders always returned "Importing 0 tracks." Not a file problem — a database mismatch. The 5GB threshold in lidarr_cleanup.sh was doing exactly what it should.
|
|
||||||
|
|
||||||
Fixed this session: 123 albums bulk-corrected (read MUSICBRAINZ_ALBUMID from FLAC tags, switched Lidarr's release selection, RefreshArtist). Full root rescan still running in the background.
|
|
||||||
|
|
||||||
Next session — pick up here:
|
|
||||||
1. Design the daily fixer script (Media/lidarr_release_fixer.sh) — FLAC + MP3 (ID3 TXXX) tag reading, same bulk-fix logic, runs from daily orchestrator. Strike system in lidarr_cleanup.sh already provides the protection window.
|
|
||||||
2. 42 albums where file MBID isn't in Lidarr at all (unofficial/re-tagged) — manual import or ignore.
|
|
||||||
3. 217 MP3 albums — need ID3 TXXX frame reader added to the fixer.
|
|
||||||
4. Check the full root rescan completed and see if zero-file count dropped further below 927.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -130,10 +130,11 @@
|
|||||||
# STATE FILES
|
# STATE FILES
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
#
|
#
|
||||||
# WATCHDOG_STATE_FILE — strike counts, daemon flags (STATE_DIR — survives reboots)
|
# WATCHDOG_STATE_FILE — strike counts, daemon flags (STATE_DIR — survives reboots)
|
||||||
# DOCKER_WATCHDOG_FAILED_FILE — container skip list (STATE_DIR — survives reboots)
|
# DOCKER_WATCHDOG_FAILED_FILE — container skip list (STATE_DIR — survives reboots)
|
||||||
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection (DATA_DIR)
|
# DOCKER_WATCHDOG_INTENTIONAL_FILE — intentional stops list (STATE_DIR — survives reboots)
|
||||||
# RW_STATE_FILE — read-only: resource_watchdog RAM emergency flag
|
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection (DATA_DIR)
|
||||||
|
# RW_STATE_FILE — read-only: resource_watchdog RAM emergency flag
|
||||||
#
|
#
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# CONFIGURATION
|
# CONFIGURATION
|
||||||
@@ -198,6 +199,10 @@
|
|||||||
# DOCKER_WATCHDOG_HEARTBEAT_HOURS
|
# DOCKER_WATCHDOG_HEARTBEAT_HOURS
|
||||||
# Hours between alive heartbeat log entries
|
# Hours between alive heartbeat log entries
|
||||||
#
|
#
|
||||||
|
# DOCKER_WATCHDOG_INTENTIONAL_FILE
|
||||||
|
# Path to intentional stops state file (STATE_DIR). Containers in this file
|
||||||
|
# are never restarted by the watchdog, regardless of exit code.
|
||||||
|
#
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# RUNTIME MODES
|
# RUNTIME MODES
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
@@ -210,13 +215,23 @@
|
|||||||
# happen based on current container states. Use to verify configuration.
|
# happen based on current container states. Use to verify configuration.
|
||||||
#
|
#
|
||||||
# docker_watchdog.sh --status
|
# docker_watchdog.sh --status
|
||||||
# Show strike counts, skip list contents, grace period status, RAM emergency
|
# Show strike counts, skip list contents, intentional stops, grace period status,
|
||||||
# deferral state, and last cycle timing. Then exit.
|
# RAM emergency deferral state, and last cycle timing. Then exit.
|
||||||
#
|
#
|
||||||
# docker_watchdog.sh --log
|
# docker_watchdog.sh --log
|
||||||
# Verbose output — full detail for every container checked and every decision.
|
# Verbose output — full detail for every container checked and every decision.
|
||||||
# Use to debug why a container is or is not being restarted.
|
# Use to debug why a container is or is not being restarted.
|
||||||
#
|
#
|
||||||
|
# docker_watchdog.sh --pause ContainerName
|
||||||
|
# Add ContainerName to the intentional stops list. Watchdog will not strike or
|
||||||
|
# restart it until it is seen running again or --resume is called. Persists across
|
||||||
|
# reboots. Use when stopping a required container for planned maintenance.
|
||||||
|
#
|
||||||
|
# docker_watchdog.sh --resume ContainerName
|
||||||
|
# Remove ContainerName from the intentional stops list. Normal watchdog monitoring
|
||||||
|
# resumes on the next cycle. The container is not started — it remains stopped
|
||||||
|
# until started manually.
|
||||||
|
#
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
@@ -225,6 +240,17 @@ source "$SCRIPT_DIR/../load_config.sh"
|
|||||||
|
|
||||||
parse_args "$@"
|
parse_args "$@"
|
||||||
|
|
||||||
|
# Extract watchdog-specific flags not handled by common parse_args
|
||||||
|
WATCHDOG_PAUSE_CONTAINER=""
|
||||||
|
WATCHDOG_RESUME_CONTAINER=""
|
||||||
|
for (( _wdi=0; _wdi<${#PARSED_ARGS[@]}; _wdi++ )); do
|
||||||
|
case "${PARSED_ARGS[$_wdi]}" in
|
||||||
|
--pause) ((_wdi++)); WATCHDOG_PAUSE_CONTAINER="${PARSED_ARGS[$_wdi]:-}" ;;
|
||||||
|
--resume) ((_wdi++)); WATCHDOG_RESUME_CONTAINER="${PARSED_ARGS[$_wdi]:-}" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
unset _wdi
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ━━━ Setup — runs once at start ━━━
|
# ━━━ Setup — runs once at start ━━━
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
@@ -256,7 +282,7 @@ log "$ICON_CONTAINERS Tier1: watched=${#WATCHDOG_CONTAINERS[@]} required=${#WAT
|
|||||||
|
|
||||||
# Ensure state files exist
|
# Ensure state files exist
|
||||||
touch "$WATCHDOG_STATE_FILE" "$WATCHDOG_CONTAINER_RESTART_LOG" \
|
touch "$WATCHDOG_STATE_FILE" "$WATCHDOG_CONTAINER_RESTART_LOG" \
|
||||||
"$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null
|
"$DOCKER_WATCHDOG_FAILED_FILE" "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null
|
||||||
|
|
||||||
# Timeout for all docker commands — configurable via WATCHDOG_DAEMON_TIMEOUT in master.conf
|
# Timeout for all docker commands — configurable via WATCHDOG_DAEMON_TIMEOUT in master.conf
|
||||||
DOCKER_TIMEOUT="${WATCHDOG_DAEMON_TIMEOUT:-20}"
|
DOCKER_TIMEOUT="${WATCHDOG_DAEMON_TIMEOUT:-20}"
|
||||||
@@ -270,6 +296,8 @@ if [[ "$SHOW_STATUS" == true ]]; then
|
|||||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||||
_watched="${!WATCHDOG_CONTAINERS[*]}"; echo "$ICON_CONTAINERS Watched: ${_watched:-none}"
|
_watched="${!WATCHDOG_CONTAINERS[*]}"; echo "$ICON_CONTAINERS Watched: ${_watched:-none}"
|
||||||
echo "$ICON_CONTAINERS Required: ${WATCHDOG_REQUIRED_CONTAINERS[*]:-none}"
|
echo "$ICON_CONTAINERS Required: ${WATCHDOG_REQUIRED_CONTAINERS[*]:-none}"
|
||||||
|
_intentional=$(cat "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
||||||
|
echo "$ICON_SKIP Intentional: ${_intentional:-none}"
|
||||||
echo "$ICON_WATCHDOG Scan all: $WATCHDOG_SCAN_ALL"
|
echo "$ICON_WATCHDOG Scan all: $WATCHDOG_SCAN_ALL"
|
||||||
echo "$ICON_WATCHDOG Ignore: ${WATCHDOG_SCAN_IGNORE[*]:-none}"
|
echo "$ICON_WATCHDOG Ignore: ${WATCHDOG_SCAN_IGNORE[*]:-none}"
|
||||||
echo "$ICON_WATCHDOG Schedule: every 15 min (cron via watchdog_orchestrator)"
|
echo "$ICON_WATCHDOG Schedule: every 15 min (cron via watchdog_orchestrator)"
|
||||||
@@ -286,6 +314,23 @@ if [[ "$SHOW_STATUS" == true ]]; then
|
|||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# ── Intentional stop management — --pause / --resume ─────────────────────────────────────────
|
||||||
|
if [[ -n "$WATCHDOG_PAUSE_CONTAINER" || -n "$WATCHDOG_RESUME_CONTAINER" ]]; then
|
||||||
|
if [[ -n "$WATCHDOG_PAUSE_CONTAINER" ]]; then
|
||||||
|
if grep -q "^${WATCHDOG_PAUSE_CONTAINER}$" "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null; then
|
||||||
|
warn "$WATCHDOG_PAUSE_CONTAINER already in intentional stops"
|
||||||
|
else
|
||||||
|
echo "$WATCHDOG_PAUSE_CONTAINER" >> "$DOCKER_WATCHDOG_INTENTIONAL_FILE"
|
||||||
|
success "$WATCHDOG_PAUSE_CONTAINER added to intentional stops — watchdog will not restart it"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if [[ -n "$WATCHDOG_RESUME_CONTAINER" ]]; then
|
||||||
|
sed -i "/^${WATCHDOG_RESUME_CONTAINER}$/d" "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null
|
||||||
|
success "$WATCHDOG_RESUME_CONTAINER removed from intentional stops — normal monitoring resumes next cycle"
|
||||||
|
fi
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ── HELPER FUNCTIONS ──────────────────────────────────────────────────────────────────────────
|
# ── HELPER FUNCTIONS ──────────────────────────────────────────────────────────────────────────
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
@@ -326,6 +371,19 @@ remove_from_skip_list() {
|
|||||||
warn "$1 recovered — removed from skip list ✅"
|
warn "$1 recovered — removed from skip list ✅"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Check if container is in the intentional stops list
|
||||||
|
is_intentional_stop() {
|
||||||
|
grep -q "^${1}$" "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
# Remove from intentional stops — called when container is seen running again
|
||||||
|
clear_intentional_stop() {
|
||||||
|
if grep -q "^${1}$" "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null; then
|
||||||
|
sed -i "/^${1}$/d" "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null
|
||||||
|
warn "$1 running — removed from intentional stops ✅"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
# Log a restart event to the rolling restart history file
|
# Log a restart event to the rolling restart history file
|
||||||
log_restart() {
|
log_restart() {
|
||||||
local container="$1"
|
local container="$1"
|
||||||
@@ -616,9 +674,11 @@ CYCLE_START=$(date +%s)
|
|||||||
[[ -n "$c" ]] && IGNORE_MAP["$c"]=1
|
[[ -n "$c" ]] && IGNORE_MAP["$c"]=1
|
||||||
done
|
done
|
||||||
|
|
||||||
# ── Skip list visibility ─────────────────────────────────────────────────────────────────
|
# ── Skip list and intentional stops visibility ───────────────────────────────────────────
|
||||||
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
||||||
[[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed"
|
[[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed"
|
||||||
|
_intentional_contents=$(cat "$DOCKER_WATCHDOG_INTENTIONAL_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
||||||
|
[[ -n "$_intentional_contents" ]] && warn "$ICON_SKIP Intentional stops: $_intentional_contents — watchdog will not restart these"
|
||||||
|
|
||||||
# ── Docker daemon health check — first check every run ──────────────────────────────────
|
# ── Docker daemon health check — first check every run ──────────────────────────────────
|
||||||
# If daemon is hung all container operations will fail — check first, skip run if down
|
# If daemon is hung all container operations will fail — check first, skip run if down
|
||||||
@@ -676,23 +736,41 @@ CYCLE_START=$(date +%s)
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "$STATUS" == "true" ]]; then
|
if [[ "$STATUS" == "true" ]]; then
|
||||||
# Running — clear any strikes
|
# Running — clear strikes and any intentional-stop flag set from a prior cycle
|
||||||
set_strikes "$container" 0 "$WATCHDOG_STATE_FILE"
|
set_strikes "$container" 0 "$WATCHDOG_STATE_FILE"
|
||||||
|
clear_intentional_stop "$container"
|
||||||
log "$ICON_RUNNING $container — running ✅"
|
log "$ICON_RUNNING $container — running ✅"
|
||||||
else
|
else
|
||||||
|
# Check intentional stops first — explicit operator instruction beats everything
|
||||||
|
if is_intentional_stop "$container"; then
|
||||||
|
log "$container — intentionally stopped (on pause list) — skipping"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Exit code 0 = cleanly stopped (docker stop, Unraid UI stop, clean shutdown).
|
||||||
|
# Don't strike or restart — operator almost certainly stopped it on purpose.
|
||||||
|
# Use --pause to make this permanent across reboots.
|
||||||
|
LAST_EXIT=$(timeout "$DOCKER_TIMEOUT" docker inspect -f \
|
||||||
|
'{{.State.ExitCode}}' "$container" 2>/dev/null)
|
||||||
|
if [[ "$LAST_EXIT" == "0" ]]; then
|
||||||
|
log "$container — stopped cleanly (exit 0) — treating as intentional; use --pause to suppress permanently"
|
||||||
|
set_strikes "$container" 0 "$WATCHDOG_STATE_FILE"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
STRIKES=$(get_strikes "$container" "$WATCHDOG_STATE_FILE")
|
STRIKES=$(get_strikes "$container" "$WATCHDOG_STATE_FILE")
|
||||||
STRIKES=$(( STRIKES + 1 ))
|
STRIKES=$(( STRIKES + 1 ))
|
||||||
set_strikes "$container" "$STRIKES" "$WATCHDOG_STATE_FILE"
|
set_strikes "$container" "$STRIKES" "$WATCHDOG_STATE_FILE"
|
||||||
warn "$container — not running (strike $STRIKES/$SYS_WATCHDOG_STRIKE_LIMIT)"
|
warn "$container — not running, exit ${LAST_EXIT} (strike $STRIKES/$SYS_WATCHDOG_STRIKE_LIMIT)"
|
||||||
((T1_WARNINGS++))
|
((T1_WARNINGS++))
|
||||||
|
|
||||||
if [[ "$STRIKES" -ge "$SYS_WATCHDOG_STRIKE_LIMIT" ]]; then
|
if [[ "$STRIKES" -ge "$SYS_WATCHDOG_STRIKE_LIMIT" ]]; then
|
||||||
result=0
|
result=0
|
||||||
safe_restart "$container" "required container down" || result=$?
|
safe_restart "$container" "required container down (exit ${LAST_EXIT})" || result=$?
|
||||||
case $result in
|
case $result in
|
||||||
0) set_strikes "$container" 0 "$WATCHDOG_STATE_FILE"
|
0) set_strikes "$container" 0 "$WATCHDOG_STATE_FILE"
|
||||||
((T1_RESTARTS++))
|
((T1_RESTARTS++))
|
||||||
queue_notify "$container was down and restarted on $(hostname)" "warning" ;;
|
queue_notify "$container was down (exit ${LAST_EXIT}) and restarted on $(hostname)" "warning" ;;
|
||||||
2) : ;; # Added to skip list — already notified
|
2) : ;; # Added to skip list — already notified
|
||||||
*) queue_notify "$container failed to restart on $(hostname)" "warning" ;;
|
*) queue_notify "$container failed to restart on $(hostname)" "warning" ;;
|
||||||
esac
|
esac
|
||||||
|
|||||||
Reference in New Issue
Block a user