Update headers on today's arr-caching changes
Comment-only. Headers on the scripts touched during today's caching work (cache-first fetches, write-through per-item cache, single-walk consolidation, movieFile-embedded fix) still described pre-change behavior. Also brought common.sh's top-level cache doc block current -- it was written for the single-consumer 2026-07-16 state and didn't mention the tmpfs move, the write guard, or the 15+ consumers that now go through it.
This commit is contained in:
@@ -10,6 +10,12 @@
|
||||
# rsync in the weekly sync window: once arrs agree on what to track, rsync
|
||||
# spreads the actual files.
|
||||
#
|
||||
# The LOCAL side of each sync (_local_library()) is cache-first (2026-07-17) — comes from
|
||||
# the shared tracked-data cache via arr_get_tracked_data(), fresh (kept warm every 30min by
|
||||
# arr_cache_prefill.sh), live fetch as fallback. The REMOTE side (_remote_library()) is
|
||||
# unaffected — that cache is per-host by design, so a remote node's library is always
|
||||
# fetched live here.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -20,6 +20,16 @@
|
||||
# Only escalates to the scary abort-and-notify when the count is genuinely low AND nothing
|
||||
# is actively rescanning.
|
||||
#
|
||||
# Cache-first, both layers (2026-07-17). The artist list itself comes from the shared
|
||||
# tracked-data cache via arr_get_tracked_data() — fresh (kept warm every 30min by
|
||||
# arr_cache_prefill.sh), live fetch as fallback. The per-artist trackFile walk below still
|
||||
# always fetches live (that's the actual disk-truth this script's delete decisions depend
|
||||
# on), but write-throughs its result to arr_item_cache_write() so lidarr_missing_art.sh,
|
||||
# running later in the same nightly window, can read it instead of repeating the same walk.
|
||||
# The filesystem is walked once per run, not twice — classification records which paths are
|
||||
# eligible for deletion as it goes, and the delete pass (once the size-threshold check below
|
||||
# passes) just acts on that list instead of re-walking and re-classifying the whole tree.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -24,6 +24,14 @@
|
||||
# tags or renames media files. Only writes missing artwork files to existing
|
||||
# album/artist directories.
|
||||
#
|
||||
# Cache-first, both layers (2026-07-17). The artist list comes from the shared tracked-data
|
||||
# cache via arr_get_tracked_data() — fresh (kept warm every 30min by arr_cache_prefill.sh),
|
||||
# live fetch as fallback. The per-artist track-file walk used to build the album→directory
|
||||
# map now reads lidarr_cleanup.sh's write-through cache first (arr_get_cached_items() —
|
||||
# lidarr_cleanup.sh runs earlier in the same nightly window and already does this exact
|
||||
# walk for its own cleanup decisions), falling back to its own live per-artist walk only if
|
||||
# that cache is missing or from outside the current window.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -34,6 +34,11 @@
|
||||
# RefreshArtist is batched — one per artist, even if multiple albums were fixed.
|
||||
# Lidarr handles the post-refresh rescan and import automatically.
|
||||
#
|
||||
# Cache-first artist list (2026-07-17) — comes from the shared tracked-data cache via
|
||||
# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch
|
||||
# as fallback. Everything below (per-album release lookups) still fetches live — that data
|
||||
# isn't part of what's cached.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -33,6 +33,11 @@
|
||||
# 8. Score candidates: affinity + breadth + popularity + quality
|
||||
# 9. Take top MAX_ADDS above threshold → add to Lidarr
|
||||
#
|
||||
# Step 7's "already in Lidarr" check reads the shared tracked-data cache via
|
||||
# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch —
|
||||
# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan
|
||||
# snapshot arr_full_rescan.sh just wrote.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# SCORING MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -33,6 +33,11 @@
|
||||
# 7. Score candidates: breadth + TMDB rating + vote count
|
||||
# 8. Take top MAX_ADDS above threshold → add to Radarr
|
||||
#
|
||||
# Step 6's "already in Radarr" check reads the shared tracked-data cache via
|
||||
# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch —
|
||||
# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan
|
||||
# snapshot arr_full_rescan.sh just wrote.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# SCORING MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -37,6 +37,11 @@
|
||||
# 11. Take top MAX_ADDS above threshold
|
||||
# 12. Get TVDB ID via TMDB external_ids → Sonarr lookup → add + trigger SeriesSearch
|
||||
#
|
||||
# Step 9's "already in Sonarr" check reads the shared tracked-data cache via
|
||||
# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch —
|
||||
# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan
|
||||
# snapshot arr_full_rescan.sh just wrote.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# SCORING MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -18,6 +18,17 @@
|
||||
# false-alarm abort. Mirrors the same fix built for lidarr_cleanup.sh 2026-07-16 after
|
||||
# a whole-library rescan there made trackFileCount read 22% of normal mid-scan.
|
||||
#
|
||||
# Cache-first movie list, no per-movie API calls at all (2026-07-17). The movie list comes
|
||||
# from the shared tracked-data cache via arr_get_tracked_data() — fresh (kept warm every
|
||||
# 30min by arr_cache_prefill.sh), live fetch as fallback. Radarr is structurally different
|
||||
# from Lidarr/Sonarr here: its movie list already embeds movieFile.path directly on every
|
||||
# hasFile=true entry (confirmed live, zero exceptions across the full library), so there's
|
||||
# no separate per-movie moviefile?movieId=X walk needed at all — what used to be up to 2896
|
||||
# individual API calls is now a jq filter over data already in hand. The filesystem is
|
||||
# walked once per run, not twice — classification records which paths are eligible for
|
||||
# deletion as it goes, and the delete pass (once the size-threshold check below passes)
|
||||
# just acts on that list instead of re-walking and re-classifying the whole tree.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -10,6 +10,10 @@
|
||||
# or downloaded. Most are announced-but-never-released films delisted before
|
||||
# release.
|
||||
#
|
||||
# Cache-first movie list (2026-07-17) — comes from the shared tracked-data cache via
|
||||
# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch
|
||||
# as fallback.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -19,6 +19,17 @@
|
||||
# lidarr_cleanup.sh 2026-07-16 after a whole-library rescan there made trackFileCount
|
||||
# read 22% of normal mid-scan.
|
||||
#
|
||||
# Cache-first, both layers (2026-07-17). The series list itself comes from the shared
|
||||
# tracked-data cache via arr_get_tracked_data() — fresh (kept warm every 30min by
|
||||
# arr_cache_prefill.sh), live fetch as fallback. The per-series episodefile walk below still
|
||||
# always fetches live (that's the actual disk-truth this script's delete decisions depend
|
||||
# on), but write-throughs its result to arr_item_cache_write() for any future script that
|
||||
# needs Sonarr's per-episode data — no second consumer exists yet, unlike Lidarr's
|
||||
# lidarr_missing_art.sh, but the data's there once one does. The filesystem is walked once
|
||||
# per run, not twice — classification records which paths are eligible for deletion as it
|
||||
# goes, and the delete pass (once the size-threshold check below passes) just acts on that
|
||||
# list instead of re-walking and re-classifying the whole tree.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -9,6 +9,10 @@
|
||||
# status="deleted" — they generate health errors and can never be monitored
|
||||
# or downloaded.
|
||||
#
|
||||
# Cache-first series list (2026-07-17) — comes from the shared tracked-data cache via
|
||||
# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch
|
||||
# as fallback.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -12,6 +12,10 @@
|
||||
# after Lidarr is set up, or any time you suspect gaps between what you listen
|
||||
# to and what Lidarr monitors.
|
||||
#
|
||||
# Cache-first Lidarr library fetch (2026-07-17) — comes from the shared tracked-data cache
|
||||
# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live
|
||||
# fetch as fallback.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# FLOW
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -11,6 +11,10 @@
|
||||
# Intended as a bootstrap / catch-up tool. Run after Radarr setup, after a
|
||||
# database wipe, or any time you suspect gaps between your library and Radarr.
|
||||
#
|
||||
# Cache-first Radarr library fetch (2026-07-17) — comes from the shared tracked-data cache
|
||||
# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live
|
||||
# fetch as fallback.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# FLOW
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -11,6 +11,10 @@
|
||||
# Intended as a bootstrap / catch-up tool. Run after Sonarr setup, after a
|
||||
# database wipe, or any time you suspect gaps between your library and Sonarr.
|
||||
#
|
||||
# Cache-first Sonarr library fetch (2026-07-17) — comes from the shared tracked-data cache
|
||||
# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live
|
||||
# fetch as fallback.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# FLOW
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -2239,26 +2239,41 @@ trigger_and_await_command() {
|
||||
# ── ARR TRACKED-DATA CACHE (Lidarr, Sonarr, Radarr) ──────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Shared cache for each arr's tracked-library data (Lidarr artists, Sonarr series, Radarr
|
||||
# movies). Multiple scripts across all three arrs need a reasonably-current snapshot of "what
|
||||
# does this arr think it has tracked." Hitting the live API fresh every time is wasteful, and
|
||||
# during an active rescan the live number is actively misleading — tracked counts dip and
|
||||
# recover as files are detached/re-verified one by one (confirmed 2026-07-16 on Lidarr: a
|
||||
# whole-library RescanFolders made trackFileCount read 22% of normal mid-scan, which is
|
||||
# exactly the false-alarm lidarr_cleanup.sh's count-drop guard is meant to catch, but a
|
||||
# genuine rescan isn't the "something's actually wrong" case that guard exists for — the same
|
||||
# risk applies to Sonarr's RescanSeries and Radarr's RescanMovie).
|
||||
# movies). Every script across all three arrs that needs "what does this arr think it has
|
||||
# tracked" goes through this — 15+ consumers as of 2026-07-17 (all three cleanup scripts, the
|
||||
# missing-art/release-fixer/duplicate-cleanup scripts, all three playback-aware discovery
|
||||
# scripts, the TVDB/TMDb-removed checks, arr_sync.sh's local side, and the emby_to_*_sync
|
||||
# bootstrap tools). Hitting the live API fresh every time is wasteful, and during an active
|
||||
# rescan the live number is actively misleading — tracked counts dip and recover as files are
|
||||
# detached/re-verified one by one (confirmed 2026-07-16 on Lidarr: a whole-library
|
||||
# RescanFolders made trackFileCount read 22% of normal mid-scan, which is exactly the
|
||||
# false-alarm lidarr_cleanup.sh's count-drop guard is meant to catch, but a genuine rescan
|
||||
# isn't the "something's actually wrong" case that guard exists for — the same risk applies to
|
||||
# Sonarr's RescanSeries and Radarr's RescanMovie).
|
||||
#
|
||||
# Built Lidarr-only first (2026-07-16), generalized the same day to cover all three arrs —
|
||||
# identical mechanism, keyed by arr_type ("lidarr"/"sonarr"/"radarr") so each arr's cache and
|
||||
# duration history stay separate.
|
||||
#
|
||||
# Storage (2026-07-17): primary copy lives on tmpfs (ARR_CACHE_DIR) — reads/writes never touch
|
||||
# the array disk, and losing it on reboot costs nothing since a full rebuild for all three arrs
|
||||
# measures ~12s live. A persistent backup on $DATA_DIR is kept in sync by every write and gets
|
||||
# transparently restored into tmpfs by arr_cache_age_seconds() the moment it notices tmpfs is
|
||||
# missing — so a cache that was fresh before a reboot reads as fresh after too.
|
||||
#
|
||||
# Write-through: any script that already does a live library-list fetch for its own purposes
|
||||
# writes the result here as a side effect via arr_cache_write() — no dedicated polling timer
|
||||
# needed. Arrs_Stack/arr_cache_prefill.sh closes the cold-boot gap by populating all three
|
||||
# caches once at array start, before anything else needs them.
|
||||
# needed. arr_cache_write() itself refuses to write while a rescan-type command is active for
|
||||
# that arr (2026-07-17) — protects every caller uniformly, not just arr_get_tracked_data().
|
||||
# Arrs_Stack/arr_cache_prefill.sh keeps the cache warm two ways: once at array start (10min
|
||||
# wait ceiling, closes the cold-boot gap) and again every 30min via CRITICAL_MAINTENANCE_SCRIPTS
|
||||
# (1min wait ceiling — a live fetch takes seconds, not the boot-time wait).
|
||||
#
|
||||
# Consumers should call arr_get_tracked_data() — never read a cache file directly. It handles
|
||||
# the fresh/stale-no-rescan/stale-rescan-active branching so no script reimplements it.
|
||||
#
|
||||
# This is the top-level LIBRARY cache only. A separate, shorter-lived cache exists for the
|
||||
# much more expensive per-item track/episode data — see "PER-ITEM TRACKED-FILE CACHE" below.
|
||||
# ==============================================================================================
|
||||
|
||||
# Rescan-type command names per arr — operations long/heavy enough that overlapping with one
|
||||
|
||||
Reference in New Issue
Block a user