Update headers on today's arr-caching changes

Comment-only. Headers on the scripts touched during today's caching work
(cache-first fetches, write-through per-item cache, single-walk
consolidation, movieFile-embedded fix) still described pre-change
behavior. Also brought common.sh's top-level cache doc block current --
it was written for the single-consumer 2026-07-16 state and didn't
mention the tmpfs move, the write guard, or the 15+ consumers that now
go through it.
This commit is contained in:
Gmer4Lfe
2026-07-17 01:08:46 -04:00
parent de2879bdee
commit bac1ef1c17
15 changed files with 111 additions and 10 deletions
+6
View File
@@ -10,6 +10,12 @@
# rsync in the weekly sync window: once arrs agree on what to track, rsync # rsync in the weekly sync window: once arrs agree on what to track, rsync
# spreads the actual files. # spreads the actual files.
# #
# The LOCAL side of each sync (_local_library()) is cache-first (2026-07-17) — comes from
# the shared tracked-data cache via arr_get_tracked_data(), fresh (kept warm every 30min by
# arr_cache_prefill.sh), live fetch as fallback. The REMOTE side (_remote_library()) is
# unaffected — that cache is per-host by design, so a remote node's library is always
# fetched live here.
#
# ============================================================================================== # ==============================================================================================
# OPERATIONAL MODEL # OPERATIONAL MODEL
# ============================================================================================== # ==============================================================================================
+10
View File
@@ -20,6 +20,16 @@
# Only escalates to the scary abort-and-notify when the count is genuinely low AND nothing # Only escalates to the scary abort-and-notify when the count is genuinely low AND nothing
# is actively rescanning. # is actively rescanning.
# #
# Cache-first, both layers (2026-07-17). The artist list itself comes from the shared
# tracked-data cache via arr_get_tracked_data() — fresh (kept warm every 30min by
# arr_cache_prefill.sh), live fetch as fallback. The per-artist trackFile walk below still
# always fetches live (that's the actual disk-truth this script's delete decisions depend
# on), but write-throughs its result to arr_item_cache_write() so lidarr_missing_art.sh,
# running later in the same nightly window, can read it instead of repeating the same walk.
# The filesystem is walked once per run, not twice — classification records which paths are
# eligible for deletion as it goes, and the delete pass (once the size-threshold check below
# passes) just acts on that list instead of re-walking and re-classifying the whole tree.
#
# ============================================================================================== # ==============================================================================================
# OPERATIONAL MODEL # OPERATIONAL MODEL
# ============================================================================================== # ==============================================================================================
+8
View File
@@ -24,6 +24,14 @@
# tags or renames media files. Only writes missing artwork files to existing # tags or renames media files. Only writes missing artwork files to existing
# album/artist directories. # album/artist directories.
# #
# Cache-first, both layers (2026-07-17). The artist list comes from the shared tracked-data
# cache via arr_get_tracked_data() — fresh (kept warm every 30min by arr_cache_prefill.sh),
# live fetch as fallback. The per-artist track-file walk used to build the album→directory
# map now reads lidarr_cleanup.sh's write-through cache first (arr_get_cached_items() —
# lidarr_cleanup.sh runs earlier in the same nightly window and already does this exact
# walk for its own cleanup decisions), falling back to its own live per-artist walk only if
# that cache is missing or from outside the current window.
#
# ============================================================================================== # ==============================================================================================
# DESIGN PRINCIPLES # DESIGN PRINCIPLES
# ============================================================================================== # ==============================================================================================
+5
View File
@@ -34,6 +34,11 @@
# RefreshArtist is batched — one per artist, even if multiple albums were fixed. # RefreshArtist is batched — one per artist, even if multiple albums were fixed.
# Lidarr handles the post-refresh rescan and import automatically. # Lidarr handles the post-refresh rescan and import automatically.
# #
# Cache-first artist list (2026-07-17) — comes from the shared tracked-data cache via
# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch
# as fallback. Everything below (per-album release lookups) still fetches live — that data
# isn't part of what's cached.
#
# ============================================================================================== # ==============================================================================================
# DESIGN PRINCIPLES # DESIGN PRINCIPLES
# ============================================================================================== # ==============================================================================================
@@ -33,6 +33,11 @@
# 8. Score candidates: affinity + breadth + popularity + quality # 8. Score candidates: affinity + breadth + popularity + quality
# 9. Take top MAX_ADDS above threshold → add to Lidarr # 9. Take top MAX_ADDS above threshold → add to Lidarr
# #
# Step 7's "already in Lidarr" check reads the shared tracked-data cache via
# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch —
# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan
# snapshot arr_full_rescan.sh just wrote.
#
# ============================================================================================== # ==============================================================================================
# SCORING MODEL # SCORING MODEL
# ============================================================================================== # ==============================================================================================
@@ -33,6 +33,11 @@
# 7. Score candidates: breadth + TMDB rating + vote count # 7. Score candidates: breadth + TMDB rating + vote count
# 8. Take top MAX_ADDS above threshold → add to Radarr # 8. Take top MAX_ADDS above threshold → add to Radarr
# #
# Step 6's "already in Radarr" check reads the shared tracked-data cache via
# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch —
# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan
# snapshot arr_full_rescan.sh just wrote.
#
# ============================================================================================== # ==============================================================================================
# SCORING MODEL # SCORING MODEL
# ============================================================================================== # ==============================================================================================
@@ -37,6 +37,11 @@
# 11. Take top MAX_ADDS above threshold # 11. Take top MAX_ADDS above threshold
# 12. Get TVDB ID via TMDB external_ids → Sonarr lookup → add + trigger SeriesSearch # 12. Get TVDB ID via TMDB external_ids → Sonarr lookup → add + trigger SeriesSearch
# #
# Step 9's "already in Sonarr" check reads the shared tracked-data cache via
# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch —
# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan
# snapshot arr_full_rescan.sh just wrote.
#
# ============================================================================================== # ==============================================================================================
# SCORING MODEL # SCORING MODEL
# ============================================================================================== # ==============================================================================================
+11
View File
@@ -18,6 +18,17 @@
# false-alarm abort. Mirrors the same fix built for lidarr_cleanup.sh 2026-07-16 after # false-alarm abort. Mirrors the same fix built for lidarr_cleanup.sh 2026-07-16 after
# a whole-library rescan there made trackFileCount read 22% of normal mid-scan. # a whole-library rescan there made trackFileCount read 22% of normal mid-scan.
# #
# Cache-first movie list, no per-movie API calls at all (2026-07-17). The movie list comes
# from the shared tracked-data cache via arr_get_tracked_data() — fresh (kept warm every
# 30min by arr_cache_prefill.sh), live fetch as fallback. Radarr is structurally different
# from Lidarr/Sonarr here: its movie list already embeds movieFile.path directly on every
# hasFile=true entry (confirmed live, zero exceptions across the full library), so there's
# no separate per-movie moviefile?movieId=X walk needed at all — what used to be up to 2896
# individual API calls is now a jq filter over data already in hand. The filesystem is
# walked once per run, not twice — classification records which paths are eligible for
# deletion as it goes, and the delete pass (once the size-threshold check below passes)
# just acts on that list instead of re-walking and re-classifying the whole tree.
#
# ============================================================================================== # ==============================================================================================
# OPERATIONAL MODEL # OPERATIONAL MODEL
# ============================================================================================== # ==============================================================================================
+4
View File
@@ -10,6 +10,10 @@
# or downloaded. Most are announced-but-never-released films delisted before # or downloaded. Most are announced-but-never-released films delisted before
# release. # release.
# #
# Cache-first movie list (2026-07-17) — comes from the shared tracked-data cache via
# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch
# as fallback.
#
# ============================================================================================== # ==============================================================================================
# OPERATIONAL MODEL # OPERATIONAL MODEL
# ============================================================================================== # ==============================================================================================
+11
View File
@@ -19,6 +19,17 @@
# lidarr_cleanup.sh 2026-07-16 after a whole-library rescan there made trackFileCount # lidarr_cleanup.sh 2026-07-16 after a whole-library rescan there made trackFileCount
# read 22% of normal mid-scan. # read 22% of normal mid-scan.
# #
# Cache-first, both layers (2026-07-17). The series list itself comes from the shared
# tracked-data cache via arr_get_tracked_data() — fresh (kept warm every 30min by
# arr_cache_prefill.sh), live fetch as fallback. The per-series episodefile walk below still
# always fetches live (that's the actual disk-truth this script's delete decisions depend
# on), but write-throughs its result to arr_item_cache_write() for any future script that
# needs Sonarr's per-episode data — no second consumer exists yet, unlike Lidarr's
# lidarr_missing_art.sh, but the data's there once one does. The filesystem is walked once
# per run, not twice — classification records which paths are eligible for deletion as it
# goes, and the delete pass (once the size-threshold check below passes) just acts on that
# list instead of re-walking and re-classifying the whole tree.
#
# ============================================================================================== # ==============================================================================================
# OPERATIONAL MODEL # OPERATIONAL MODEL
# ============================================================================================== # ==============================================================================================
+4
View File
@@ -9,6 +9,10 @@
# status="deleted" — they generate health errors and can never be monitored # status="deleted" — they generate health errors and can never be monitored
# or downloaded. # or downloaded.
# #
# Cache-first series list (2026-07-17) — comes from the shared tracked-data cache via
# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch
# as fallback.
#
# ============================================================================================== # ==============================================================================================
# OPERATIONAL MODEL # OPERATIONAL MODEL
# ============================================================================================== # ==============================================================================================
+4
View File
@@ -12,6 +12,10 @@
# after Lidarr is set up, or any time you suspect gaps between what you listen # after Lidarr is set up, or any time you suspect gaps between what you listen
# to and what Lidarr monitors. # to and what Lidarr monitors.
# #
# Cache-first Lidarr library fetch (2026-07-17) — comes from the shared tracked-data cache
# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live
# fetch as fallback.
#
# ============================================================================================== # ==============================================================================================
# FLOW # FLOW
# ============================================================================================== # ==============================================================================================
+4
View File
@@ -11,6 +11,10 @@
# Intended as a bootstrap / catch-up tool. Run after Radarr setup, after a # Intended as a bootstrap / catch-up tool. Run after Radarr setup, after a
# database wipe, or any time you suspect gaps between your library and Radarr. # database wipe, or any time you suspect gaps between your library and Radarr.
# #
# Cache-first Radarr library fetch (2026-07-17) — comes from the shared tracked-data cache
# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live
# fetch as fallback.
#
# ============================================================================================== # ==============================================================================================
# FLOW # FLOW
# ============================================================================================== # ==============================================================================================
+4
View File
@@ -11,6 +11,10 @@
# Intended as a bootstrap / catch-up tool. Run after Sonarr setup, after a # Intended as a bootstrap / catch-up tool. Run after Sonarr setup, after a
# database wipe, or any time you suspect gaps between your library and Sonarr. # database wipe, or any time you suspect gaps between your library and Sonarr.
# #
# Cache-first Sonarr library fetch (2026-07-17) — comes from the shared tracked-data cache
# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live
# fetch as fallback.
#
# ============================================================================================== # ==============================================================================================
# FLOW # FLOW
# ============================================================================================== # ==============================================================================================
+25 -10
View File
@@ -2239,26 +2239,41 @@ trigger_and_await_command() {
# ── ARR TRACKED-DATA CACHE (Lidarr, Sonarr, Radarr) ────────────────────────────────────────── # ── ARR TRACKED-DATA CACHE (Lidarr, Sonarr, Radarr) ──────────────────────────────────────────
# ============================================================================================== # ==============================================================================================
# Shared cache for each arr's tracked-library data (Lidarr artists, Sonarr series, Radarr # Shared cache for each arr's tracked-library data (Lidarr artists, Sonarr series, Radarr
# movies). Multiple scripts across all three arrs need a reasonably-current snapshot of "what # movies). Every script across all three arrs that needs "what does this arr think it has
# does this arr think it has tracked." Hitting the live API fresh every time is wasteful, and # tracked" goes through this — 15+ consumers as of 2026-07-17 (all three cleanup scripts, the
# during an active rescan the live number is actively misleading — tracked counts dip and # missing-art/release-fixer/duplicate-cleanup scripts, all three playback-aware discovery
# recover as files are detached/re-verified one by one (confirmed 2026-07-16 on Lidarr: a # scripts, the TVDB/TMDb-removed checks, arr_sync.sh's local side, and the emby_to_*_sync
# whole-library RescanFolders made trackFileCount read 22% of normal mid-scan, which is # bootstrap tools). Hitting the live API fresh every time is wasteful, and during an active
# exactly the false-alarm lidarr_cleanup.sh's count-drop guard is meant to catch, but a # rescan the live number is actively misleading — tracked counts dip and recover as files are
# genuine rescan isn't the "something's actually wrong" case that guard exists for — the same # detached/re-verified one by one (confirmed 2026-07-16 on Lidarr: a whole-library
# risk applies to Sonarr's RescanSeries and Radarr's RescanMovie). # RescanFolders made trackFileCount read 22% of normal mid-scan, which is exactly the
# false-alarm lidarr_cleanup.sh's count-drop guard is meant to catch, but a genuine rescan
# isn't the "something's actually wrong" case that guard exists for — the same risk applies to
# Sonarr's RescanSeries and Radarr's RescanMovie).
# #
# Built Lidarr-only first (2026-07-16), generalized the same day to cover all three arrs — # Built Lidarr-only first (2026-07-16), generalized the same day to cover all three arrs —
# identical mechanism, keyed by arr_type ("lidarr"/"sonarr"/"radarr") so each arr's cache and # identical mechanism, keyed by arr_type ("lidarr"/"sonarr"/"radarr") so each arr's cache and
# duration history stay separate. # duration history stay separate.
# #
# Storage (2026-07-17): primary copy lives on tmpfs (ARR_CACHE_DIR) — reads/writes never touch
# the array disk, and losing it on reboot costs nothing since a full rebuild for all three arrs
# measures ~12s live. A persistent backup on $DATA_DIR is kept in sync by every write and gets
# transparently restored into tmpfs by arr_cache_age_seconds() the moment it notices tmpfs is
# missing — so a cache that was fresh before a reboot reads as fresh after too.
#
# Write-through: any script that already does a live library-list fetch for its own purposes # Write-through: any script that already does a live library-list fetch for its own purposes
# writes the result here as a side effect via arr_cache_write() — no dedicated polling timer # writes the result here as a side effect via arr_cache_write() — no dedicated polling timer
# needed. Arrs_Stack/arr_cache_prefill.sh closes the cold-boot gap by populating all three # needed. arr_cache_write() itself refuses to write while a rescan-type command is active for
# caches once at array start, before anything else needs them. # that arr (2026-07-17) — protects every caller uniformly, not just arr_get_tracked_data().
# Arrs_Stack/arr_cache_prefill.sh keeps the cache warm two ways: once at array start (10min
# wait ceiling, closes the cold-boot gap) and again every 30min via CRITICAL_MAINTENANCE_SCRIPTS
# (1min wait ceiling — a live fetch takes seconds, not the boot-time wait).
# #
# Consumers should call arr_get_tracked_data() — never read a cache file directly. It handles # Consumers should call arr_get_tracked_data() — never read a cache file directly. It handles
# the fresh/stale-no-rescan/stale-rescan-active branching so no script reimplements it. # the fresh/stale-no-rescan/stale-rescan-active branching so no script reimplements it.
#
# This is the top-level LIBRARY cache only. A separate, shorter-lived cache exists for the
# much more expensive per-item track/episode data — see "PER-ITEM TRACKED-FILE CACHE" below.
# ============================================================================================== # ==============================================================================================
# Rescan-type command names per arr — operations long/heavy enough that overlapping with one # Rescan-type command names per arr — operations long/heavy enough that overlapping with one