Gmer4Lfe
5866097d6a
Every script that fetches the full Lidarr/Sonarr/Radarr tracked-library list now goes through arr_get_tracked_data() instead of hitting the API directly -- cache-first when fresh, live fetch as fallback when stale, waits out an active rescan before either. Per-item file data (trackFile/ episodefile/moviefile) stays live-only everywhere, since that's the actual disk-truth these scripts' decisions depend on and was never part of what's cached. Also adds arr_cache_prefill.sh to CRITICAL_MAINTENANCE_SCRIPTS (30min tier) with a short 1min wait ceiling, so the cache stays consistently fresh instead of only refreshing whenever some other script happens to write through. A full cache refresh for all three arrs measured at ~12s total live -- nothing like the multi-hour cost of an actual rescan.
764 lines
31 KiB
Bash
Executable File
764 lines
31 KiB
Bash
Executable File
#!/bin/bash
|
||
# ==============================================================================================
|
||
# =========================== Playback-Aware Lidarr Discovery ==================================
|
||
# ==============================================================================================
|
||
#
|
||
# PURPOSE
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Two-stage behavior-driven music discovery.
|
||
#
|
||
# Stage 1 — Score what you actually played this week. The top artists become
|
||
# high-quality seeds, not just anything that hit the minimum play count.
|
||
#
|
||
# Stage 2 — Run Last.fm artist.getSimilar on those seeds. Score the recommendations
|
||
# and add the top artists to Lidarr.
|
||
#
|
||
# Goal: 0–5 meaningful Lidarr adds per week, not bulk imports.
|
||
#
|
||
# ==============================================================================================
|
||
# FLOW
|
||
# ==============================================================================================
|
||
#
|
||
# 1. Fetch play completions from Emby activity log (last LOOKBACK_DAYS days)
|
||
# 2. Aggregate by artist — apply per-user influence cap to play weights
|
||
#
|
||
# ── Stage 1 ─────────────────────────────────────────────────────────────────
|
||
# 3. Score each played artist: user signal + recency + Last.fm popularity/quality
|
||
# 4. Take top MAX_ADDS by score → discovery seeds
|
||
#
|
||
# ── Stage 2 ─────────────────────────────────────────────────────────────────
|
||
# 5. For each seed, call Last.fm artist.getSimilar → collect candidates
|
||
# 6. Aggregate: affinity (weighted similarity × seed score) + breadth (distinct seeds)
|
||
# 7. Filter: already in Lidarr, already in Emby, placeholder artists, cooldown
|
||
# 8. Score candidates: affinity + breadth + popularity + quality
|
||
# 9. Take top MAX_ADDS above threshold → add to Lidarr
|
||
#
|
||
# ==============================================================================================
|
||
# SCORING MODEL
|
||
# ==============================================================================================
|
||
#
|
||
# Stage 1 — seed selection (max 100)
|
||
# user_score (0-40) — effective plays × 5, cap 40
|
||
# recency_score (0-30) — days since last play; today→30, older→less
|
||
# popularity_score (0-20) — Last.fm listeners
|
||
# quality_score (0-10) — Last.fm global playcount
|
||
#
|
||
# Stage 2 — candidate scoring (max 100)
|
||
# affinity_score (0-40) — seed_score × similarity, normalized; raw/20, cap 40
|
||
# breadth_score (0-30) — 1 seed→5, 2 seeds→18, 3+seeds→30
|
||
# popularity_score (0-20) — Last.fm listeners
|
||
# quality_score (0-10) — Last.fm global playcount
|
||
#
|
||
# Threshold: LIDARR_DISCOVERY_THRESHOLD (default 70) applied to both stages
|
||
# Max adds: LIDARR_DISCOVERY_MAX_ADDS (default 5) caps each stage
|
||
#
|
||
# ==============================================================================================
|
||
# REQUIREMENTS
|
||
# ==============================================================================================
|
||
#
|
||
# Last.fm API key — required for both stages
|
||
# Configure HOST*_LASTFM_API_KEY in host*.conf
|
||
#
|
||
# ==============================================================================================
|
||
# DESIGN PRINCIPLES
|
||
# ==============================================================================================
|
||
#
|
||
# Playback as Intent Signal
|
||
# What users actually listen to is a stronger signal than what they follow or
|
||
# own. The scoring model weights demonstrated listening behaviour — recency,
|
||
# play count, user breadth — over passive library membership.
|
||
#
|
||
# Selective by Design
|
||
# 0–5 adds per week is the target, not bulk imports. A high score threshold
|
||
# combined with MAX_ADDS ensures only high-confidence recommendations are
|
||
# acted on. Volume is not the goal — meaningful discovery is.
|
||
#
|
||
# Two-Stage Filtering
|
||
# Stage 1 rejects weak seeds before they drive Stage 2. Low-quality seeds
|
||
# produce low-quality similar-artist recommendations. Filtering at the seed
|
||
# stage improves the entire output, not just the top of the list.
|
||
#
|
||
# ==============================================================================================
|
||
# OPERATIONAL SAFEGUARDS
|
||
# ==============================================================================================
|
||
#
|
||
# Root Required
|
||
# acquire_lock and Lidarr API writes require root. Script exits cleanly if not root.
|
||
#
|
||
# Dry-Run Mode
|
||
# --dry-run scores and ranks all candidates but makes no Lidarr API calls and does not
|
||
# write to the history file. Safe to run at any time to preview what would be added.
|
||
#
|
||
# Add-Only
|
||
# Only adds artists to Lidarr. Never deletes or modifies existing entries.
|
||
#
|
||
# Cooldown Guard
|
||
# Candidates rejected this run are recorded in the history file and not re-evaluated
|
||
# until LIDARR_DISCOVERY_REJECT_COOLDOWN days have elapsed.
|
||
#
|
||
# ==============================================================================================
|
||
# STATE FILES
|
||
# ==============================================================================================
|
||
#
|
||
# LIDARR_DISCOVERY_HISTORY (default: $DATA_DIR/lidarr_discovery_history.db)
|
||
# Tracks added artists and rejected candidates with timestamps. Written after every
|
||
# non-dry-run. Enforces reject cooldown and prevents re-adding items added by previous
|
||
# runs. Safe to delete — next run starts fresh with no memory.
|
||
#
|
||
# ==============================================================================================
|
||
# CONFIGURATION (master.conf)
|
||
# ==============================================================================================
|
||
#
|
||
# LIDARR_DISCOVERY_THRESHOLD — minimum score for Stage 1 seeds and Stage 2 adds (default: 70)
|
||
# LIDARR_DISCOVERY_LOOKBACK_DAYS — Emby play history window in days (default: 7)
|
||
# LIDARR_DISCOVERY_MIN_PLAYS — min plays to be evaluated in Stage 1 (default: 3)
|
||
# LIDARR_DISCOVERY_MAX_ADDS — max seeds (Stage 1) and max adds (Stage 2) (default: 5)
|
||
# LIDARR_DISCOVERY_USER_CAP_PCT — max % any one user contributes to play weight (default: 35)
|
||
# LIDARR_DISCOVERY_REJECT_COOLDOWN — days before re-evaluating a Stage 2 reject (default: 30)
|
||
# LIDARR_DISCOVERY_HISTORY — history/state file path
|
||
#
|
||
# ==============================================================================================
|
||
# RUNTIME MODES
|
||
# ==============================================================================================
|
||
#
|
||
# playback_aware_lidarr_discovery.sh — normal run
|
||
# playback_aware_lidarr_discovery.sh --dry-run — score and rank, no Lidarr changes
|
||
# playback_aware_lidarr_discovery.sh --log — verbose output
|
||
# playback_aware_lidarr_discovery.sh --status — show config and exit
|
||
#
|
||
# Recommended schedule: weekly (WEEKLY_MAINTENANCE_SCRIPTS in master.conf)
|
||
#
|
||
# ==============================================================================================
|
||
|
||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
|
||
source "$SCRIPT_DIR/../load_config.sh"
|
||
source "$SCRIPT_DIR/../Kernel/decision_engine.sh"
|
||
|
||
parse_args "$@"
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Setup ━━━
|
||
# ==============================================================================================
|
||
if [[ "$EUID" -ne 0 ]]; then
|
||
error "Must be run as root"
|
||
exit 1
|
||
fi
|
||
|
||
acquire_lock
|
||
detect_hosts
|
||
|
||
if ! command -v jq >/dev/null 2>&1; then
|
||
error "jq not installed — required for API JSON parsing"
|
||
exit 1
|
||
fi
|
||
|
||
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
|
||
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
|
||
exit 1
|
||
fi
|
||
|
||
if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then
|
||
error "LIDARR_URL / LIDARR_API_KEY not configured — check host*.conf"
|
||
exit 1
|
||
fi
|
||
|
||
if [[ -z "${LASTFM_API_KEY:-}" ]]; then
|
||
error "LASTFM_API_KEY not configured — required for discovery"
|
||
error "Configure HOST*_LASTFM_API_KEY in host*.conf"
|
||
exit 1
|
||
fi
|
||
|
||
THRESHOLD="${LIDARR_DISCOVERY_THRESHOLD:-70}"
|
||
LOOKBACK_DAYS="${LIDARR_DISCOVERY_LOOKBACK_DAYS:-7}"
|
||
MIN_PLAYS="${LIDARR_DISCOVERY_MIN_PLAYS:-3}"
|
||
MAX_ADDS="${LIDARR_DISCOVERY_MAX_ADDS:-5}"
|
||
USER_CAP_PCT="${LIDARR_DISCOVERY_USER_CAP_PCT:-35}"
|
||
REJECT_COOLDOWN="${LIDARR_DISCOVERY_REJECT_COOLDOWN:-30}"
|
||
HISTORY_FILE="${LIDARR_DISCOVERY_HISTORY:-${DATA_DIR}/lidarr_discovery_history.db}"
|
||
|
||
log "$ICON_GEAR Config: threshold=${THRESHOLD} lookback=${LOOKBACK_DAYS}d min-plays=${MIN_PLAYS} max-adds=${MAX_ADDS} user-cap=${USER_CAP_PCT}% reject-cooldown=${REJECT_COOLDOWN}d"
|
||
|
||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no artists will be added to Lidarr"
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Status ━━━
|
||
# ==============================================================================================
|
||
if [[ "$SHOW_STATUS" == true ]]; then
|
||
echo ""
|
||
echo "━━━━━ $ICON_SUMMARY LIDARR DISCOVERY STATUS ━━━━━"
|
||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||
echo "$ICON_SYNC Emby: ${EMBY_URL}"
|
||
echo "$ICON_SYNC Lidarr: ${LIDARR_URL}"
|
||
echo "$ICON_GEAR Threshold: ${THRESHOLD} / 100 (both stages)"
|
||
echo "$ICON_TIME Lookback: ${LOOKBACK_DAYS} days"
|
||
echo "$ICON_GEAR Min plays: ${MIN_PLAYS}"
|
||
echo "$ICON_GEAR Max adds: ${MAX_ADDS} (seeds in Stage 1, adds in Stage 2)"
|
||
echo "$ICON_GEAR User cap: ${USER_CAP_PCT}% max per user (floor: 3 plays)"
|
||
echo "$ICON_GEAR Reject TTL: ${REJECT_COOLDOWN} days"
|
||
echo "$ICON_GEAR History file: ${HISTORY_FILE}"
|
||
echo "$ICON_GEAR Last.fm: configured"
|
||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||
exit 0
|
||
fi
|
||
|
||
# ==============================================================================================
|
||
# ── API HELPERS ───────────────────────────────────────────────────────────────────────────────
|
||
# ==============================================================================================
|
||
|
||
|
||
_lidarr_get() {
|
||
curl -sf --max-time 20 \
|
||
-H "X-Api-Key: $LIDARR_API_KEY" \
|
||
"${LIDARR_URL}/api/v1/${1}" 2>/dev/null
|
||
}
|
||
|
||
_lidarr_lookup() {
|
||
curl -sf --max-time 20 --get \
|
||
--data-urlencode "term=$1" \
|
||
-H "X-Api-Key: $LIDARR_API_KEY" \
|
||
"${LIDARR_URL}/api/v1/artist/lookup" 2>/dev/null
|
||
}
|
||
|
||
_lidarr_post() {
|
||
curl -sf --max-time 20 -X POST \
|
||
-H "X-Api-Key: $LIDARR_API_KEY" \
|
||
-H "Content-Type: application/json" \
|
||
-d "$1" \
|
||
"${LIDARR_URL}/api/v1/artist" 2>/dev/null
|
||
}
|
||
|
||
_lastfm_info() {
|
||
curl -sf --max-time 10 --get \
|
||
--data-urlencode "method=artist.getinfo" \
|
||
--data-urlencode "artist=$1" \
|
||
--data-urlencode "api_key=$LASTFM_API_KEY" \
|
||
--data-urlencode "format=json" \
|
||
"http://ws.audioscrobbler.com/2.0/" 2>/dev/null
|
||
}
|
||
|
||
_lastfm_similar() {
|
||
curl -sf --max-time 15 --get \
|
||
--data-urlencode "method=artist.getSimilar" \
|
||
--data-urlencode "artist=$1" \
|
||
--data-urlencode "limit=10" \
|
||
--data-urlencode "api_key=$LASTFM_API_KEY" \
|
||
--data-urlencode "format=json" \
|
||
"http://ws.audioscrobbler.com/2.0/" 2>/dev/null
|
||
}
|
||
|
||
# ==============================================================================================
|
||
# ── SCORING HELPERS ───────────────────────────────────────────────────────────────────────────
|
||
# ==============================================================================================
|
||
|
||
_user_score() {
|
||
local s=$(( $1 * 5 ))
|
||
(( s > 40 )) && s=40
|
||
echo "$s"
|
||
}
|
||
|
||
_recency_score() {
|
||
local days="$1"
|
||
if (( days == 0 )); then echo 30
|
||
elif (( days == 1 )); then echo 25
|
||
elif (( days == 2 )); then echo 20
|
||
elif (( days == 3 )); then echo 15
|
||
elif (( days == 4 )); then echo 10
|
||
elif (( days == 5 )); then echo 8
|
||
else echo 5
|
||
fi
|
||
}
|
||
|
||
_affinity_score() {
|
||
local s=$(( $1 / 20 ))
|
||
(( s > 40 )) && s=40
|
||
echo "$s"
|
||
}
|
||
|
||
_breadth_score() {
|
||
local seeds="$1"
|
||
if (( seeds >= 3 )); then echo 30
|
||
elif (( seeds == 2 )); then echo 18
|
||
else echo 5
|
||
fi
|
||
}
|
||
|
||
_popularity_score() {
|
||
local listeners="$1"
|
||
if (( listeners >= 5000000 )); then echo 20
|
||
elif (( listeners >= 1000000 )); then echo 15
|
||
elif (( listeners >= 500000 )); then echo 10
|
||
elif (( listeners >= 100000 )); then echo 5
|
||
else echo 2
|
||
fi
|
||
}
|
||
|
||
_quality_score() {
|
||
local playcount="$1"
|
||
if (( playcount >= 100000000 )); then echo 10
|
||
elif (( playcount >= 10000000 )); then echo 7
|
||
elif (( playcount >= 1000000 )); then echo 4
|
||
else echo 2
|
||
fi
|
||
}
|
||
|
||
_is_placeholder_artist() {
|
||
local a="${1,,}"
|
||
[[ "$a" =~ ^(va|various|various artists|unknown artist|unknown|soundtrack|original soundtrack|ost)$ ]]
|
||
}
|
||
|
||
_lfm_scores() {
|
||
local artist="$1"
|
||
local LFM_JSON lfm_listeners lfm_playcount
|
||
LFM_JSON=$(_lastfm_info "$artist")
|
||
if [[ -n "$LFM_JSON" ]] && echo "$LFM_JSON" | jq -e '.artist' >/dev/null 2>&1; then
|
||
lfm_listeners=$(echo "$LFM_JSON" | jq -r '.artist.stats.listeners // "0"' | tr -d ',')
|
||
lfm_playcount=$(echo "$LFM_JSON" | jq -r '.artist.stats.playcount // "0"' | tr -d ',')
|
||
lfm_listeners=${lfm_listeners//[^0-9]/}; lfm_listeners=${lfm_listeners:-0}
|
||
lfm_playcount=${lfm_playcount//[^0-9]/}; lfm_playcount=${lfm_playcount:-0}
|
||
echo "${lfm_listeners}|${lfm_playcount}"
|
||
else
|
||
echo "0|0"
|
||
fi
|
||
}
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Fetch Emby Play History ━━━
|
||
# ==============================================================================================
|
||
echo ""
|
||
echo "━━━━━ $ICON_SUMMARY Lidarr Discovery — $(date '+%Y-%m-%d %H:%M:%S') ━━━━━"
|
||
echo "$ICON_HOST $MY_ID ($LOCAL_SERVER_NAME) | lookback: ${LOOKBACK_DAYS}d | threshold: ${THRESHOLD}/100 | max: ${MAX_ADDS}"
|
||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no artists will be added"
|
||
|
||
echo ""
|
||
echo "━━━ $ICON_SYNC Emby Play History ━━━"
|
||
|
||
CUTOFF_ISO=$(date -d "${LOOKBACK_DAYS} days ago" '+%Y-%m-%dT%H:%M:%SZ')
|
||
TODAY_EPOCH=$(date +%s)
|
||
TODAY=$(date +%Y-%m-%d)
|
||
|
||
ACTIVITY_JSON=$(emby_api "System/ActivityLog/Entries?MinDate=${CUTOFF_ISO}&Limit=5000") || {
|
||
error "Could not fetch Emby activity log"
|
||
exit 1
|
||
}
|
||
|
||
declare -A ARTIST_PLAYS
|
||
declare -A ARTIST_LAST_PLAY
|
||
declare -A ARTIST_USER_PLAYS
|
||
declare -A ARTIST_EFFECTIVE # precomputed effective plays after user cap
|
||
TOTAL_PLAYS=0
|
||
|
||
while IFS='|' read -r play_date artist user_id; do
|
||
[[ -z "$artist" || "$artist" == "null" ]] && continue
|
||
_is_placeholder_artist "$artist" && continue
|
||
|
||
(( TOTAL_PLAYS++ ))
|
||
ARTIST_PLAYS["$artist"]=$(( ${ARTIST_PLAYS["$artist"]:-0} + 1 ))
|
||
ARTIST_USER_PLAYS["${artist}|${user_id}"]=$(( ${ARTIST_USER_PLAYS["${artist}|${user_id}"]:-0} + 1 ))
|
||
|
||
current="${ARTIST_LAST_PLAY["$artist"]:-}"
|
||
if [[ -z "$current" || "$play_date" > "$current" ]]; then
|
||
ARTIST_LAST_PLAY["$artist"]="$play_date"
|
||
fi
|
||
done < <(echo "$ACTIVITY_JSON" | jq -r '
|
||
.Items // [] | .[] |
|
||
select(.Name | test("has finished playing"; "i")) |
|
||
select(.Name | test(" - ")) |
|
||
select(.Name | test(", Ep[0-9]") | not) |
|
||
[
|
||
.Date,
|
||
(
|
||
.Name |
|
||
split(" has finished playing ")[1] |
|
||
split(" on ") | .[0:-1] | join(" on ") |
|
||
split(" - ")[0] | ltrimstr(" ") | rtrimstr(" ")
|
||
),
|
||
(.UserId // "unknown")
|
||
] | join("|")
|
||
' 2>/dev/null)
|
||
|
||
# Precompute effective plays (user cap applied) for all qualifying artists
|
||
for _a in "${!ARTIST_PLAYS[@]}"; do
|
||
_plays="${ARTIST_PLAYS["$_a"]}"
|
||
(( _plays < MIN_PLAYS )) && continue
|
||
_cap=$(( (_plays * USER_CAP_PCT + 99) / 100 ))
|
||
(( _cap < 3 )) && _cap=3
|
||
_eff=0
|
||
for _ukey in "${!ARTIST_USER_PLAYS[@]}"; do
|
||
[[ "$_ukey" == "${_a}|"* ]] || continue
|
||
_uc="${ARTIST_USER_PLAYS["$_ukey"]}"
|
||
(( _eff += _uc < _cap ? _uc : _cap ))
|
||
done
|
||
(( _eff == 0 )) && _eff="$_plays"
|
||
ARTIST_EFFECTIVE["$_a"]="$_eff"
|
||
done
|
||
|
||
UNIQUE_ARTISTS=${#ARTIST_PLAYS[@]}
|
||
QUALIFIED=$(( $(echo "${!ARTIST_EFFECTIVE[@]}" | wc -w) ))
|
||
log "$TOTAL_PLAYS play completions | $UNIQUE_ARTISTS unique artists | $QUALIFIED qualify (${MIN_PLAYS}+ plays)"
|
||
|
||
if [[ "$QUALIFIED" -eq 0 ]]; then
|
||
warn "No artists qualify this week — nothing to evaluate"
|
||
exit 0
|
||
fi
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Stage 1: Score Played Artists → Select Seeds ━━━
|
||
# ==============================================================================================
|
||
echo ""
|
||
echo "━━━ $ICON_GEAR Stage 1: Scoring Played Artists ━━━"
|
||
|
||
S1_SCORED=() # "score|artist"
|
||
|
||
for artist in "${!ARTIST_EFFECTIVE[@]}"; do
|
||
effective="${ARTIST_EFFECTIVE["$artist"]}"
|
||
last_played="${ARTIST_LAST_PLAY["$artist"]}"
|
||
last_epoch=$(date -d "$last_played" +%s 2>/dev/null || echo "$TODAY_EPOCH")
|
||
days_ago=$(( (TODAY_EPOCH - last_epoch) / 86400 ))
|
||
|
||
user_s=$(_user_score "$effective")
|
||
recency_s=$(_recency_score "$days_ago")
|
||
|
||
IFS='|' read -r lfm_listeners lfm_playcount <<< "$(_lfm_scores "$artist")"
|
||
popularity_s=$(_popularity_score "$lfm_listeners")
|
||
quality_s=$(_quality_score "$lfm_playcount")
|
||
|
||
total=$(score_candidate "$user_s" "$popularity_s" "$recency_s" "$quality_s")
|
||
total=$(apply_temporal_decay "$total" "$days_ago")
|
||
|
||
log " [${total}] $artist (plays: ${ARTIST_PLAYS["$artist"]}→${effective} | ${days_ago}d ago)"
|
||
S1_SCORED+=("${total}|${artist}")
|
||
done
|
||
|
||
IFS=$'\n' S1_SORTED=($(printf '%s\n' "${S1_SCORED[@]}" | sort -t'|' -k1 -rn 2>/dev/null))
|
||
|
||
echo ""
|
||
echo "━━━ $ICON_SUMMARY Stage 1 Results (top ${MAX_ADDS} seeds) ━━━"
|
||
|
||
SEEDS=()
|
||
SEED_SCORES=() # parallel array: score for each seed (used as Stage 2 weight)
|
||
|
||
for _entry in "${S1_SORTED[@]}"; do
|
||
(( ${#SEEDS[@]} >= MAX_ADDS )) && break
|
||
_score="${_entry%%|*}"
|
||
_artist="${_entry#*|}"
|
||
(( _score < THRESHOLD )) && break # sorted desc — below threshold means rest are too
|
||
SEEDS+=("$_artist")
|
||
SEED_SCORES+=("$_score")
|
||
printf " Seed [%3s] %s (plays: %s)\n" "$_score" "$_artist" "${ARTIST_PLAYS["$_artist"]}"
|
||
done
|
||
|
||
if [[ "${#SEEDS[@]}" -eq 0 ]]; then
|
||
warn "No played artists scored above ${THRESHOLD} this week — no seeds for discovery"
|
||
exit 0
|
||
fi
|
||
|
||
log "${#SEEDS[@]} seed(s) selected"
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Stage 2: Last.fm Similarity Discovery ━━━
|
||
# ==============================================================================================
|
||
echo ""
|
||
echo "━━━ $ICON_SYNC Stage 2: Last.fm Similarity Discovery ━━━"
|
||
|
||
declare -A CANDIDATE_SCORE
|
||
declare -A CANDIDATE_SEED_COUNT
|
||
declare -A CANDIDATE_LAST_SEED
|
||
|
||
for (( _i=0; _i<${#SEEDS[@]}; _i++ )); do
|
||
seed_artist="${SEEDS[$_i]}"
|
||
seed_score="${SEED_SCORES[$_i]}"
|
||
seed_last_play="${ARTIST_LAST_PLAY["$seed_artist"]}"
|
||
|
||
log " Seed: $seed_artist (score: $seed_score)"
|
||
|
||
SIMILAR_JSON=$(_lastfm_similar "$seed_artist")
|
||
if [[ -z "$SIMILAR_JSON" ]]; then
|
||
warn " Last.fm similar unavailable for: $seed_artist"
|
||
continue
|
||
fi
|
||
|
||
while IFS='|' read -r sim_name sim_match; do
|
||
[[ -z "$sim_name" || "$sim_name" == "null" ]] && continue
|
||
_is_placeholder_artist "$sim_name" && continue
|
||
[[ "${sim_name,,}" == "${seed_artist,,}" ]] && continue
|
||
|
||
sim_pct=$(echo "$sim_match" | awk '{printf "%d", $1 * 100 + 0.5}')
|
||
(( sim_pct < 1 )) && sim_pct=1
|
||
|
||
# Weight contribution by seed's Stage 1 score, not raw play count
|
||
contribution=$(( seed_score * sim_pct ))
|
||
CANDIDATE_SCORE["$sim_name"]=$(( ${CANDIDATE_SCORE["$sim_name"]:-0} + contribution ))
|
||
CANDIDATE_SEED_COUNT["$sim_name"]=$(( ${CANDIDATE_SEED_COUNT["$sim_name"]:-0} + 1 ))
|
||
|
||
existing_date="${CANDIDATE_LAST_SEED["$sim_name"]:-}"
|
||
if [[ -z "$existing_date" || "$seed_last_play" > "$existing_date" ]]; then
|
||
CANDIDATE_LAST_SEED["$sim_name"]="$seed_last_play"
|
||
fi
|
||
done < <(echo "$SIMILAR_JSON" | jq -r '
|
||
.similarartists.artist // [] | .[] |
|
||
[.name, .match] | join("|")
|
||
' 2>/dev/null)
|
||
done
|
||
|
||
CANDIDATE_COUNT=${#CANDIDATE_SCORE[@]}
|
||
log "$CANDIDATE_COUNT discovery candidates from Last.fm"
|
||
|
||
if [[ "$CANDIDATE_COUNT" -eq 0 ]]; then
|
||
warn "No candidates returned from Last.fm — check API key or seed artist names"
|
||
exit 0
|
||
fi
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Fetch Existing Libraries ━━━
|
||
# ==============================================================================================
|
||
echo ""
|
||
echo "━━━ $ICON_SYNC Existing Libraries ━━━"
|
||
|
||
# Cache-first — arr_get_tracked_data() serves the shared cache when it's fresh (kept current
|
||
# every 30min by arr_cache_prefill.sh in CRITICAL_MAINTENANCE_SCRIPTS), falls back to a live
|
||
# fetch when it's stale, and waits out an active rescan before either.
|
||
LIDARR_ARTISTS_JSON=$(arr_get_tracked_data "lidarr" "$LIDARR_URL" "$LIDARR_API_KEY" "v1") || { error "Could not fetch Lidarr artists"; exit 1; }
|
||
LIDARR_NAMES=$(echo "$LIDARR_ARTISTS_JSON" | jq -r '.[].artistName' 2>/dev/null)
|
||
LIDARR_COUNT=$(echo "$LIDARR_NAMES" | grep -c . 2>/dev/null || echo 0)
|
||
log "$LIDARR_COUNT artists in Lidarr"
|
||
|
||
EMBY_LIBRARY_JSON=$(emby_api "Items?IncludeItemTypes=MusicAlbum&Recursive=true&Fields=AlbumArtists&Limit=10000") || {
|
||
warn "Could not fetch Emby artist library — skipping Emby filter"
|
||
EMBY_ARTIST_NAMES=""
|
||
}
|
||
EMBY_ARTIST_NAMES=$(echo "$EMBY_LIBRARY_JSON" | jq -r '.Items[] | .AlbumArtists[]?.Name' 2>/dev/null)
|
||
EMBY_ARTIST_COUNT=$(echo "$EMBY_ARTIST_NAMES" | grep -c . 2>/dev/null || echo 0)
|
||
log "$EMBY_ARTIST_COUNT album artists in Emby library"
|
||
|
||
# MusicBrainz's canonical name for some artists (e.g. "blink‐182") uses a Unicode
|
||
# hyphen/dash rather than plain ASCII "-". Last.fm's candidate names are plain ASCII,
|
||
# so an exact-string match against Lidarr/Emby's names silently misses these artists
|
||
# every time — they never register as "already known" and get retried (and rejected
|
||
# as duplicates) on every future run. Normalize both sides before comparing.
|
||
_normalize_dashes() {
|
||
local n="$1"
|
||
n="${n//‐/-}" # U+2010 HYPHEN
|
||
n="${n//‑/-}" # U+2011 NON-BREAKING HYPHEN
|
||
n="${n//‒/-}" # U+2012 FIGURE DASH
|
||
n="${n//–/-}" # U+2013 EN DASH
|
||
n="${n//—/-}" # U+2014 EM DASH
|
||
echo "$n"
|
||
}
|
||
LIDARR_NAMES=$(_normalize_dashes "$LIDARR_NAMES")
|
||
EMBY_ARTIST_NAMES=$(_normalize_dashes "$EMBY_ARTIST_NAMES")
|
||
|
||
_in_lidarr() { echo "$LIDARR_NAMES" | grep -iq "^$(_normalize_dashes "$1")$"; }
|
||
_in_emby_library(){ [[ -n "$EMBY_ARTIST_NAMES" ]] && echo "$EMBY_ARTIST_NAMES" | grep -iq "^$(_normalize_dashes "$1")$"; }
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Score Stage 2 Candidates ━━━
|
||
# ==============================================================================================
|
||
echo ""
|
||
echo "━━━ $ICON_GEAR Stage 2: Scoring Candidates ━━━"
|
||
|
||
ACCEPT_LIST=()
|
||
REJECT_LIST=()
|
||
ALREADY_KNOWN=0
|
||
SKIP_COOLDOWN=0
|
||
|
||
for candidate in "${!CANDIDATE_SCORE[@]}"; do
|
||
raw_score="${CANDIDATE_SCORE["$candidate"]}"
|
||
seed_count="${CANDIDATE_SEED_COUNT["$candidate"]}"
|
||
last_seed_date="${CANDIDATE_LAST_SEED["$candidate"]:-}"
|
||
|
||
if _in_lidarr "$candidate"; then
|
||
(( ALREADY_KNOWN++ ))
|
||
log " $ICON_SKIP In Lidarr: $candidate"
|
||
continue
|
||
fi
|
||
|
||
if _in_emby_library "$candidate"; then
|
||
(( ALREADY_KNOWN++ ))
|
||
log " $ICON_SKIP In Emby: $candidate"
|
||
continue
|
||
fi
|
||
|
||
if [[ -f "$HISTORY_FILE" ]]; then
|
||
last_rejection=$(grep -i "^REJECT|${candidate}|" "$HISTORY_FILE" 2>/dev/null | tail -1 | cut -d'|' -f3)
|
||
if [[ -n "$last_rejection" ]]; then
|
||
last_epoch=$(date -d "$last_rejection" +%s 2>/dev/null || echo 0)
|
||
days_since=$(( (TODAY_EPOCH - last_epoch) / 86400 ))
|
||
if (( days_since < REJECT_COOLDOWN )); then
|
||
(( SKIP_COOLDOWN++ ))
|
||
log " $ICON_SKIP Cooldown (rejected ${days_since}d ago): $candidate"
|
||
continue
|
||
fi
|
||
fi
|
||
fi
|
||
|
||
if [[ -n "$last_seed_date" ]]; then
|
||
last_epoch=$(date -d "$last_seed_date" +%s 2>/dev/null || echo "$TODAY_EPOCH")
|
||
days_ago=$(( (TODAY_EPOCH - last_epoch) / 86400 ))
|
||
else
|
||
days_ago=0
|
||
fi
|
||
|
||
affinity_s=$(_affinity_score "$raw_score")
|
||
breadth_s=$(_breadth_score "$seed_count")
|
||
|
||
IFS='|' read -r lfm_listeners lfm_playcount <<< "$(_lfm_scores "$candidate")"
|
||
if (( lfm_listeners > 0 || lfm_playcount > 0 )); then
|
||
lfm_label="${lfm_listeners} lfm listeners"
|
||
popularity_s=$(_popularity_score "$lfm_listeners")
|
||
quality_s=$(_quality_score "$lfm_playcount")
|
||
else
|
||
lfm_label="not on Last.fm"
|
||
popularity_s=0
|
||
quality_s=0
|
||
fi
|
||
|
||
total=$(score_candidate "$affinity_s" "$popularity_s" "$breadth_s" "$quality_s")
|
||
total=$(apply_temporal_decay "$total" "$days_ago")
|
||
decision=$(make_decision "$total" "$THRESHOLD")
|
||
|
||
entry="${total}|${candidate}|${seed_count}|${affinity_s}|${lfm_label}"
|
||
[[ "$decision" == "ACCEPT" ]] && ACCEPT_LIST+=("$entry") || REJECT_LIST+=("$entry")
|
||
done
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Results ━━━
|
||
# ==============================================================================================
|
||
echo ""
|
||
echo "━━━ $ICON_SUMMARY Stage 2 Results (threshold: ${THRESHOLD}, max: ${MAX_ADDS}) ━━━"
|
||
|
||
IFS=$'\n' _ALL_ACCEPTS=($(printf '%s\n' "${ACCEPT_LIST[@]}" | sort -t'|' -k1 -rn 2>/dev/null))
|
||
IFS=$'\n' SORTED_REJECTS=($(printf '%s\n' "${REJECT_LIST[@]}" | sort -t'|' -k1 -rn 2>/dev/null))
|
||
|
||
# Cap to MAX_ADDS — overflow goes to reject history so they don't resurface for REJECT_COOLDOWN days
|
||
SORTED_ACCEPTS=("${_ALL_ACCEPTS[@]:0:$MAX_ADDS}")
|
||
for (( _i=MAX_ADDS; _i<${#_ALL_ACCEPTS[@]}; _i++ )); do
|
||
SORTED_REJECTS+=("${_ALL_ACCEPTS[$_i]}")
|
||
done
|
||
|
||
_print_row() {
|
||
local label="$1" entry="$2"
|
||
IFS='|' read -r score name seeds affinity lfm <<< "$entry"
|
||
printf " %-8s [%3s] %-40s seeds: %s | affinity: %2s | %s\n" \
|
||
"$label" "$score" "$name" "$seeds" "$affinity" "$lfm"
|
||
}
|
||
|
||
for entry in "${SORTED_ACCEPTS[@]}"; do [[ -n "$entry" ]] && _print_row "ACCEPT" "$entry"; done
|
||
for entry in "${SORTED_REJECTS[@]}"; do [[ -n "$entry" ]] && _print_row "REJECT" "$entry"; done
|
||
|
||
echo ""
|
||
echo " Already in library: $ALREADY_KNOWN | Cooldown: $SKIP_COOLDOWN"
|
||
echo " ACCEPT: ${#SORTED_ACCEPTS[@]} | REJECT: ${#SORTED_REJECTS[@]}"
|
||
|
||
if [[ "$DRY_RUN" == true ]]; then
|
||
warn "DRY RUN complete — run without --dry-run to add accepted artists"
|
||
exit 0
|
||
fi
|
||
|
||
if [[ "${#SORTED_ACCEPTS[@]}" -eq 0 ]]; then
|
||
log "No artists above threshold — nothing to add"
|
||
for entry in "${SORTED_REJECTS[@]}"; do
|
||
[[ -z "$entry" ]] && continue
|
||
IFS='|' read -r score name _ <<< "$entry"
|
||
echo "REJECT|${name}|${TODAY}" >> "$HISTORY_FILE" 2>/dev/null
|
||
done
|
||
exit 0
|
||
fi
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Add to Lidarr ━━━
|
||
# ==============================================================================================
|
||
echo ""
|
||
echo "━━━ $ICON_SYNC Adding to Lidarr ━━━"
|
||
|
||
LIDARR_ROOT=$(_lidarr_get "rootfolder" | jq -r 'first(.[] | select(.accessible == true)) | .path' 2>/dev/null)
|
||
if [[ -z "$LIDARR_ROOT" ]]; then error "Could not determine Lidarr root folder"; exit 1; fi
|
||
|
||
QUALITY_PROFILES=$(_lidarr_get "qualityprofile") || { error "Could not fetch quality profiles"; exit 1; }
|
||
METADATA_PROFILES=$(_lidarr_get "metadataprofile") || { error "Could not fetch metadata profiles"; exit 1; }
|
||
|
||
DEFAULT_QUALITY_ID=$(echo "$QUALITY_PROFILES" | jq -r '.[0].id' 2>/dev/null)
|
||
DEFAULT_METADATA_ID=$(echo "$METADATA_PROFILES" | jq -r '
|
||
first(.[] | select(.name | test("Standard"; "i")) | .id) // .[0].id' 2>/dev/null)
|
||
|
||
log "Quality profile: $DEFAULT_QUALITY_ID | Metadata profile: $DEFAULT_METADATA_ID"
|
||
|
||
ADDED=0
|
||
FAILED=0
|
||
|
||
for entry in "${SORTED_ACCEPTS[@]}"; do
|
||
[[ -z "$entry" ]] && continue
|
||
IFS='|' read -r score name seeds affinity lfm <<< "$entry"
|
||
|
||
LOOKUP=$(_lidarr_lookup "$name")
|
||
|
||
if [[ -z "$LOOKUP" ]] || echo "$LOOKUP" | jq -e '. == [] or . == null' >/dev/null 2>&1; then
|
||
warn " $ICON_WARN No match in Lidarr lookup: $name"
|
||
echo "FAIL|${name}|${TODAY}" >> "$HISTORY_FILE" 2>/dev/null
|
||
(( FAILED++ ))
|
||
continue
|
||
fi
|
||
|
||
ARTIST_DATA=$(echo "$LOOKUP" | jq '.[0]' 2>/dev/null)
|
||
MBID=$(echo "$ARTIST_DATA" | jq -r '.foreignArtistId // ""' 2>/dev/null)
|
||
LIDARR_NAME=$(echo "$ARTIST_DATA" | jq -r '.artistName // ""' 2>/dev/null)
|
||
|
||
if [[ -z "$MBID" ]]; then
|
||
warn " $ICON_WARN No MusicBrainz ID for: $name"
|
||
echo "FAIL|${name}|${TODAY}" >> "$HISTORY_FILE" 2>/dev/null
|
||
(( FAILED++ ))
|
||
continue
|
||
fi
|
||
|
||
PAYLOAD=$(echo "$ARTIST_DATA" | jq \
|
||
--arg root "$LIDARR_ROOT" \
|
||
--argjson qid "$DEFAULT_QUALITY_ID" \
|
||
--argjson mid "$DEFAULT_METADATA_ID" \
|
||
'. + {
|
||
rootFolderPath: $root,
|
||
qualityProfileId: $qid,
|
||
metadataProfileId: $mid,
|
||
monitored: true,
|
||
addOptions: {
|
||
monitor: "all",
|
||
searchForMissingAlbums: true
|
||
}
|
||
}' 2>/dev/null)
|
||
|
||
RESULT=$(_lidarr_post "$PAYLOAD")
|
||
|
||
if echo "$RESULT" | jq -e '.id' >/dev/null 2>&1; then
|
||
log " $ICON_DONE Added: $LIDARR_NAME (score: $score | seeds: $seeds)"
|
||
echo "ACCEPT|${name}|${TODAY}" >> "$HISTORY_FILE" 2>/dev/null
|
||
(( ADDED++ ))
|
||
else
|
||
warn " $ICON_WARN Failed to add: $name"
|
||
log " $(echo "$RESULT" | head -c 200)"
|
||
echo "FAIL|${name}|${TODAY}" >> "$HISTORY_FILE" 2>/dev/null
|
||
(( FAILED++ ))
|
||
fi
|
||
done
|
||
|
||
for entry in "${SORTED_REJECTS[@]}"; do
|
||
[[ -z "$entry" ]] && continue
|
||
IFS='|' read -r score name _ <<< "$entry"
|
||
echo "REJECT|${name}|${TODAY}" >> "$HISTORY_FILE" 2>/dev/null
|
||
done
|
||
|
||
# ==============================================================================================
|
||
# ━━━ Summary ━━━
|
||
# ==============================================================================================
|
||
echo ""
|
||
echo "━━━━━ $ICON_SUMMARY DISCOVERY COMPLETE ━━━━━"
|
||
echo " $ICON_DONE Added: $ADDED"
|
||
[[ "$FAILED" -gt 0 ]] && echo " $ICON_WARN Failed: $FAILED"
|
||
echo " $ICON_SKIP Rejected: ${#SORTED_REJECTS[@]}"
|
||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||
|
||
[[ "$ADDED" -gt 0 ]] && notify \
|
||
"$ADDED artist(s) added to Lidarr via discovery on $(hostname)" \
|
||
"Lidarr Discovery" "normal"
|
||
|
||
exit 0
|