From bac1ef1c175ad4a431eb1f94fab30318ba6610c0 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Fri, 17 Jul 2026 01:08:46 -0400 Subject: [PATCH] Update headers on today's arr-caching changes Comment-only. Headers on the scripts touched during today's caching work (cache-first fetches, write-through per-item cache, single-walk consolidation, movieFile-embedded fix) still described pre-change behavior. Also brought common.sh's top-level cache doc block current -- it was written for the single-consumer 2026-07-16 state and didn't mention the tmpfs move, the write guard, or the 15+ consumers that now go through it. --- Arrs_Stack/arr_sync.sh | 6 ++++ Arrs_Stack/lidarr_cleanup.sh | 10 ++++++ Arrs_Stack/lidarr_missing_art.sh | 8 +++++ Arrs_Stack/lidarr_release_fixer.sh | 5 +++ Arrs_Stack/playback_aware_lidarr_discovery.sh | 5 +++ Arrs_Stack/playback_aware_radarr_discovery.sh | 5 +++ Arrs_Stack/playback_aware_sonarr_discovery.sh | 5 +++ Arrs_Stack/radarr_cleanup.sh | 11 ++++++ Arrs_Stack/radarr_tmdb_removed.sh | 4 +++ Arrs_Stack/sonarr_cleanup.sh | 11 ++++++ Arrs_Stack/sonarr_tvdb_removed.sh | 4 +++ Tools/emby_to_lidarr_sync.sh | 4 +++ Tools/emby_to_radarr_sync.sh | 4 +++ Tools/emby_to_sonarr_sync.sh | 4 +++ common.sh | 35 +++++++++++++------ 15 files changed, 111 insertions(+), 10 deletions(-) diff --git a/Arrs_Stack/arr_sync.sh b/Arrs_Stack/arr_sync.sh index d00335d..5e6a6eb 100755 --- a/Arrs_Stack/arr_sync.sh +++ b/Arrs_Stack/arr_sync.sh @@ -10,6 +10,12 @@ # rsync in the weekly sync window: once arrs agree on what to track, rsync # spreads the actual files. # +# The LOCAL side of each sync (_local_library()) is cache-first (2026-07-17) — comes from +# the shared tracked-data cache via arr_get_tracked_data(), fresh (kept warm every 30min by +# arr_cache_prefill.sh), live fetch as fallback. The REMOTE side (_remote_library()) is +# unaffected — that cache is per-host by design, so a remote node's library is always +# fetched live here. +# # ============================================================================================== # OPERATIONAL MODEL # ============================================================================================== diff --git a/Arrs_Stack/lidarr_cleanup.sh b/Arrs_Stack/lidarr_cleanup.sh index 8a30957..3ded711 100755 --- a/Arrs_Stack/lidarr_cleanup.sh +++ b/Arrs_Stack/lidarr_cleanup.sh @@ -20,6 +20,16 @@ # Only escalates to the scary abort-and-notify when the count is genuinely low AND nothing # is actively rescanning. # +# Cache-first, both layers (2026-07-17). The artist list itself comes from the shared +# tracked-data cache via arr_get_tracked_data() — fresh (kept warm every 30min by +# arr_cache_prefill.sh), live fetch as fallback. The per-artist trackFile walk below still +# always fetches live (that's the actual disk-truth this script's delete decisions depend +# on), but write-throughs its result to arr_item_cache_write() so lidarr_missing_art.sh, +# running later in the same nightly window, can read it instead of repeating the same walk. +# The filesystem is walked once per run, not twice — classification records which paths are +# eligible for deletion as it goes, and the delete pass (once the size-threshold check below +# passes) just acts on that list instead of re-walking and re-classifying the whole tree. +# # ============================================================================================== # OPERATIONAL MODEL # ============================================================================================== diff --git a/Arrs_Stack/lidarr_missing_art.sh b/Arrs_Stack/lidarr_missing_art.sh index 7718d3b..1d89a94 100755 --- a/Arrs_Stack/lidarr_missing_art.sh +++ b/Arrs_Stack/lidarr_missing_art.sh @@ -24,6 +24,14 @@ # tags or renames media files. Only writes missing artwork files to existing # album/artist directories. # +# Cache-first, both layers (2026-07-17). The artist list comes from the shared tracked-data +# cache via arr_get_tracked_data() — fresh (kept warm every 30min by arr_cache_prefill.sh), +# live fetch as fallback. The per-artist track-file walk used to build the album→directory +# map now reads lidarr_cleanup.sh's write-through cache first (arr_get_cached_items() — +# lidarr_cleanup.sh runs earlier in the same nightly window and already does this exact +# walk for its own cleanup decisions), falling back to its own live per-artist walk only if +# that cache is missing or from outside the current window. +# # ============================================================================================== # DESIGN PRINCIPLES # ============================================================================================== diff --git a/Arrs_Stack/lidarr_release_fixer.sh b/Arrs_Stack/lidarr_release_fixer.sh index 84eff5e..f0f1001 100755 --- a/Arrs_Stack/lidarr_release_fixer.sh +++ b/Arrs_Stack/lidarr_release_fixer.sh @@ -34,6 +34,11 @@ # RefreshArtist is batched — one per artist, even if multiple albums were fixed. # Lidarr handles the post-refresh rescan and import automatically. # +# Cache-first artist list (2026-07-17) — comes from the shared tracked-data cache via +# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch +# as fallback. Everything below (per-album release lookups) still fetches live — that data +# isn't part of what's cached. +# # ============================================================================================== # DESIGN PRINCIPLES # ============================================================================================== diff --git a/Arrs_Stack/playback_aware_lidarr_discovery.sh b/Arrs_Stack/playback_aware_lidarr_discovery.sh index 1ee4296..9bbf874 100755 --- a/Arrs_Stack/playback_aware_lidarr_discovery.sh +++ b/Arrs_Stack/playback_aware_lidarr_discovery.sh @@ -33,6 +33,11 @@ # 8. Score candidates: affinity + breadth + popularity + quality # 9. Take top MAX_ADDS above threshold → add to Lidarr # +# Step 7's "already in Lidarr" check reads the shared tracked-data cache via +# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch — +# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan +# snapshot arr_full_rescan.sh just wrote. +# # ============================================================================================== # SCORING MODEL # ============================================================================================== diff --git a/Arrs_Stack/playback_aware_radarr_discovery.sh b/Arrs_Stack/playback_aware_radarr_discovery.sh index 08a0e6d..ff1fa8e 100755 --- a/Arrs_Stack/playback_aware_radarr_discovery.sh +++ b/Arrs_Stack/playback_aware_radarr_discovery.sh @@ -33,6 +33,11 @@ # 7. Score candidates: breadth + TMDB rating + vote count # 8. Take top MAX_ADDS above threshold → add to Radarr # +# Step 6's "already in Radarr" check reads the shared tracked-data cache via +# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch — +# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan +# snapshot arr_full_rescan.sh just wrote. +# # ============================================================================================== # SCORING MODEL # ============================================================================================== diff --git a/Arrs_Stack/playback_aware_sonarr_discovery.sh b/Arrs_Stack/playback_aware_sonarr_discovery.sh index 3c36c7f..1fede3b 100755 --- a/Arrs_Stack/playback_aware_sonarr_discovery.sh +++ b/Arrs_Stack/playback_aware_sonarr_discovery.sh @@ -37,6 +37,11 @@ # 11. Take top MAX_ADDS above threshold # 12. Get TVDB ID via TMDB external_ids → Sonarr lookup → add + trigger SeriesSearch # +# Step 9's "already in Sonarr" check reads the shared tracked-data cache via +# arr_get_tracked_data() (cache-first, live fallback, 2026-07-17) instead of a live fetch — +# this runs weekly right after arr_full_rescan.sh, so it's reading the genuine post-rescan +# snapshot arr_full_rescan.sh just wrote. +# # ============================================================================================== # SCORING MODEL # ============================================================================================== diff --git a/Arrs_Stack/radarr_cleanup.sh b/Arrs_Stack/radarr_cleanup.sh index 24e18bc..19f4e32 100755 --- a/Arrs_Stack/radarr_cleanup.sh +++ b/Arrs_Stack/radarr_cleanup.sh @@ -18,6 +18,17 @@ # false-alarm abort. Mirrors the same fix built for lidarr_cleanup.sh 2026-07-16 after # a whole-library rescan there made trackFileCount read 22% of normal mid-scan. # +# Cache-first movie list, no per-movie API calls at all (2026-07-17). The movie list comes +# from the shared tracked-data cache via arr_get_tracked_data() — fresh (kept warm every +# 30min by arr_cache_prefill.sh), live fetch as fallback. Radarr is structurally different +# from Lidarr/Sonarr here: its movie list already embeds movieFile.path directly on every +# hasFile=true entry (confirmed live, zero exceptions across the full library), so there's +# no separate per-movie moviefile?movieId=X walk needed at all — what used to be up to 2896 +# individual API calls is now a jq filter over data already in hand. The filesystem is +# walked once per run, not twice — classification records which paths are eligible for +# deletion as it goes, and the delete pass (once the size-threshold check below passes) +# just acts on that list instead of re-walking and re-classifying the whole tree. +# # ============================================================================================== # OPERATIONAL MODEL # ============================================================================================== diff --git a/Arrs_Stack/radarr_tmdb_removed.sh b/Arrs_Stack/radarr_tmdb_removed.sh index f4f902a..7d8c8bd 100755 --- a/Arrs_Stack/radarr_tmdb_removed.sh +++ b/Arrs_Stack/radarr_tmdb_removed.sh @@ -10,6 +10,10 @@ # or downloaded. Most are announced-but-never-released films delisted before # release. # +# Cache-first movie list (2026-07-17) — comes from the shared tracked-data cache via +# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch +# as fallback. +# # ============================================================================================== # OPERATIONAL MODEL # ============================================================================================== diff --git a/Arrs_Stack/sonarr_cleanup.sh b/Arrs_Stack/sonarr_cleanup.sh index 9dca58e..e726930 100755 --- a/Arrs_Stack/sonarr_cleanup.sh +++ b/Arrs_Stack/sonarr_cleanup.sh @@ -19,6 +19,17 @@ # lidarr_cleanup.sh 2026-07-16 after a whole-library rescan there made trackFileCount # read 22% of normal mid-scan. # +# Cache-first, both layers (2026-07-17). The series list itself comes from the shared +# tracked-data cache via arr_get_tracked_data() — fresh (kept warm every 30min by +# arr_cache_prefill.sh), live fetch as fallback. The per-series episodefile walk below still +# always fetches live (that's the actual disk-truth this script's delete decisions depend +# on), but write-throughs its result to arr_item_cache_write() for any future script that +# needs Sonarr's per-episode data — no second consumer exists yet, unlike Lidarr's +# lidarr_missing_art.sh, but the data's there once one does. The filesystem is walked once +# per run, not twice — classification records which paths are eligible for deletion as it +# goes, and the delete pass (once the size-threshold check below passes) just acts on that +# list instead of re-walking and re-classifying the whole tree. +# # ============================================================================================== # OPERATIONAL MODEL # ============================================================================================== diff --git a/Arrs_Stack/sonarr_tvdb_removed.sh b/Arrs_Stack/sonarr_tvdb_removed.sh index 24506ca..51c658a 100755 --- a/Arrs_Stack/sonarr_tvdb_removed.sh +++ b/Arrs_Stack/sonarr_tvdb_removed.sh @@ -9,6 +9,10 @@ # status="deleted" — they generate health errors and can never be monitored # or downloaded. # +# Cache-first series list (2026-07-17) — comes from the shared tracked-data cache via +# arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live fetch +# as fallback. +# # ============================================================================================== # OPERATIONAL MODEL # ============================================================================================== diff --git a/Tools/emby_to_lidarr_sync.sh b/Tools/emby_to_lidarr_sync.sh index eaaaec6..8bc6350 100755 --- a/Tools/emby_to_lidarr_sync.sh +++ b/Tools/emby_to_lidarr_sync.sh @@ -12,6 +12,10 @@ # after Lidarr is set up, or any time you suspect gaps between what you listen # to and what Lidarr monitors. # +# Cache-first Lidarr library fetch (2026-07-17) — comes from the shared tracked-data cache +# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live +# fetch as fallback. +# # ============================================================================================== # FLOW # ============================================================================================== diff --git a/Tools/emby_to_radarr_sync.sh b/Tools/emby_to_radarr_sync.sh index 1eff5e1..d6455d1 100755 --- a/Tools/emby_to_radarr_sync.sh +++ b/Tools/emby_to_radarr_sync.sh @@ -11,6 +11,10 @@ # Intended as a bootstrap / catch-up tool. Run after Radarr setup, after a # database wipe, or any time you suspect gaps between your library and Radarr. # +# Cache-first Radarr library fetch (2026-07-17) — comes from the shared tracked-data cache +# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live +# fetch as fallback. +# # ============================================================================================== # FLOW # ============================================================================================== diff --git a/Tools/emby_to_sonarr_sync.sh b/Tools/emby_to_sonarr_sync.sh index 2131918..878ef6e 100755 --- a/Tools/emby_to_sonarr_sync.sh +++ b/Tools/emby_to_sonarr_sync.sh @@ -11,6 +11,10 @@ # Intended as a bootstrap / catch-up tool. Run after Sonarr setup, after a # database wipe, or any time you suspect gaps between your library and Sonarr. # +# Cache-first Sonarr library fetch (2026-07-17) — comes from the shared tracked-data cache +# via arr_get_tracked_data(), fresh (kept warm every 30min by arr_cache_prefill.sh), live +# fetch as fallback. +# # ============================================================================================== # FLOW # ============================================================================================== diff --git a/common.sh b/common.sh index d208255..40de7c1 100755 --- a/common.sh +++ b/common.sh @@ -2239,26 +2239,41 @@ trigger_and_await_command() { # ── ARR TRACKED-DATA CACHE (Lidarr, Sonarr, Radarr) ────────────────────────────────────────── # ============================================================================================== # Shared cache for each arr's tracked-library data (Lidarr artists, Sonarr series, Radarr -# movies). Multiple scripts across all three arrs need a reasonably-current snapshot of "what -# does this arr think it has tracked." Hitting the live API fresh every time is wasteful, and -# during an active rescan the live number is actively misleading — tracked counts dip and -# recover as files are detached/re-verified one by one (confirmed 2026-07-16 on Lidarr: a -# whole-library RescanFolders made trackFileCount read 22% of normal mid-scan, which is -# exactly the false-alarm lidarr_cleanup.sh's count-drop guard is meant to catch, but a -# genuine rescan isn't the "something's actually wrong" case that guard exists for — the same -# risk applies to Sonarr's RescanSeries and Radarr's RescanMovie). +# movies). Every script across all three arrs that needs "what does this arr think it has +# tracked" goes through this — 15+ consumers as of 2026-07-17 (all three cleanup scripts, the +# missing-art/release-fixer/duplicate-cleanup scripts, all three playback-aware discovery +# scripts, the TVDB/TMDb-removed checks, arr_sync.sh's local side, and the emby_to_*_sync +# bootstrap tools). Hitting the live API fresh every time is wasteful, and during an active +# rescan the live number is actively misleading — tracked counts dip and recover as files are +# detached/re-verified one by one (confirmed 2026-07-16 on Lidarr: a whole-library +# RescanFolders made trackFileCount read 22% of normal mid-scan, which is exactly the +# false-alarm lidarr_cleanup.sh's count-drop guard is meant to catch, but a genuine rescan +# isn't the "something's actually wrong" case that guard exists for — the same risk applies to +# Sonarr's RescanSeries and Radarr's RescanMovie). # # Built Lidarr-only first (2026-07-16), generalized the same day to cover all three arrs — # identical mechanism, keyed by arr_type ("lidarr"/"sonarr"/"radarr") so each arr's cache and # duration history stay separate. # +# Storage (2026-07-17): primary copy lives on tmpfs (ARR_CACHE_DIR) — reads/writes never touch +# the array disk, and losing it on reboot costs nothing since a full rebuild for all three arrs +# measures ~12s live. A persistent backup on $DATA_DIR is kept in sync by every write and gets +# transparently restored into tmpfs by arr_cache_age_seconds() the moment it notices tmpfs is +# missing — so a cache that was fresh before a reboot reads as fresh after too. +# # Write-through: any script that already does a live library-list fetch for its own purposes # writes the result here as a side effect via arr_cache_write() — no dedicated polling timer -# needed. Arrs_Stack/arr_cache_prefill.sh closes the cold-boot gap by populating all three -# caches once at array start, before anything else needs them. +# needed. arr_cache_write() itself refuses to write while a rescan-type command is active for +# that arr (2026-07-17) — protects every caller uniformly, not just arr_get_tracked_data(). +# Arrs_Stack/arr_cache_prefill.sh keeps the cache warm two ways: once at array start (10min +# wait ceiling, closes the cold-boot gap) and again every 30min via CRITICAL_MAINTENANCE_SCRIPTS +# (1min wait ceiling — a live fetch takes seconds, not the boot-time wait). # # Consumers should call arr_get_tracked_data() — never read a cache file directly. It handles # the fresh/stale-no-rescan/stale-rescan-active branching so no script reimplements it. +# +# This is the top-level LIBRARY cache only. A separate, shorter-lived cache exists for the +# much more expensive per-item track/episode data — see "PER-ITEM TRACKED-FILE CACHE" below. # ============================================================================================== # Rescan-type command names per arr — operations long/heavy enough that overlapping with one