lidarr_cleanup.sh and sonarr_cleanup.sh already walk every artist/series
individually (trackFile/episodefile) for their own cleanup decisions --
that walk now also writes the raw per-item data through to a short-lived
tmpfs cache (arr_item_cache_write, 4h freshness, no persistent backup
since every consumer already has its own live fallback). lidarr_missing_art.sh
runs later in the same nightly window and now reads that cache first,
skipping its own redundant per-artist walk entirely on a hit. Sonarr side
is write-through only for now -- no second consumer exists yet, but the
data's there for whenever one does. Future consumers: arr_get_cached_items()
first, live per-item fetch as fallback, same pattern as these two.
Reads/writes now hit tmpfs (ARR_CACHE_DIR) instead of the array disk --
a full rebuild for all three arrs measures ~12s live, so there's no real
cost to losing it on reboot. The existing on-disk file becomes a backup
that arr_cache_write() keeps in sync on every write, and
arr_cache_age_seconds() transparently restores it into tmpfs the moment
it notices tmpfs is missing -- so a cache that was fresh before reboot
reads as fresh after too, closing the cold-start gap without needing a
dedicated restore step anywhere else.
A direct arr_cache_write() call mid-rescan wrote a partial snapshot that
looked like real data loss to every consumer of the cache. The guard now
lives in arr_cache_write() itself so every caller is protected, not just
arr_get_tracked_data(). arr_rescan_monitor.sh closes the resulting gap for
rescans triggered outside arr_full_rescan.sh's own trigger-and-wait path.
Shared cache/rescan-duration logic in common.sh now takes an arr_type
param instead of being Lidarr-specific, so Sonarr and Radarr cleanup
scripts get the same cache-first fetch + rescan-aware retry Lidarr had.
Avoids redundant full-library API calls across scripts run back to back,
and stops false failures when a fetch lands mid-rescan.
Shared cache (lidarr_get_tracked_data() in common.sh) so scripts stop
hitting Lidarr's live API for tracked counts every run, and stop
treating a mid-rescan dip as a genuine problem — a whole-library
RescanFolders legitimately makes trackFileCount read far below normal
while it re-verifies every file (confirmed 2026-07-16: 22% of normal
mid-scan). Cache reads fresh-if-recent, waits out an active rescan
(calibrated to that command's own historical duration, tracked per
command name since RescanFolders and DownloadedAlbumsScan take wildly
different amounts of time), then falls back to a stale cache rather
than hard-failing after a few strikes.
lidarr_cleanup.sh: no longer stacks a fresh DownloadedAlbumsScan on
top of one already running, and the tracked-count floor check now
waits out a genuine rescan instead of aborting on every overlap.
lidarr_duplicate_artist_cleanup.sh (new): finds case-insensitive
duplicate artist entries — same display name, different MusicBrainz
ID, added when a search/list-sync matches the wrong same-named artist.
Deletes the empty phantom side and blocks it from Import List
Exclusions, leaves genuinely-different-real-artists alone (checked by
album title overlap, deduped per-artist first so a legitimate reissue
under an artist's own catalog doesn't false-flag as cross-artist
overlap), and only notifies for the rare case where both sides have
real, overlapping content.
lidarr_cache_prefill.sh (new): warms the cache at array start so nothing
reads it cold after boot.
lidarr_missing_art.sh, lidarr_release_fixer.sh: write-through the cache
as a side effect of fetches they already needed for their own purposes.
Unraid truncates the Server Name to 15 chars. Depending on which form
ends up in master.conf's HOST* value (the truncated OS hostname, or a
longer name matching what Tailscale independently registered for that
peer), either host-identity detection or Tailscale IP resolution could
fail — hit live on HOST2 in both directions this session.
- common.sh detect_hosts(): was case-sensitive exact match only, with no
fallback and an exit 1 on failure — meaning every orchestrator/watchdog/
rsync script would hard-fail on a truncated-hostname host, not just the
web UI. Now case-insensitive, with a prefix-match fallback when the
local hostname is exactly 15 chars.
- common.sh resolve_tailscale_ip(): already did a bare regex prefix match
with zero ambiguity handling (pre-existing risk: e.g. server1/server10
could collide). Replaced with an explicit unambiguous-only check.
- config.php vv_detect_host() / vv_resolve_tailscale_ip(): same treatment,
kept as close a mirror of the bash logic as PHP allows.
All fallbacks require an EXACT prefix match (never fuzzy/percentage
similarity — considered and rejected, since names like server1/server2/
server3 would be dangerously similar under any generic similarity metric)
and require exactly one candidate to qualify; ambiguous matches are
treated as no match rather than guessed. Verified live against HOST1/
HOST2 in both master.conf configurations (short and long HOST2 value).
Codebase-wide audit pass: fixed real bugs (SSH hangs missing BatchMode,
local-outside-function no-ops, variable name collisions, a truncated
ratio calc, wrong state-dir path, DARK vs NO_INTERNET drift, and more),
then pulled logic that was duplicated across multiple scripts — arr
cleanup safety gates, docker restart ordering, container maintenance
stop/restart, watchdog state-file helpers, partnership role resolution,
cert expiry checks, remote node discovery, and TMDB discovery scoring —
into common.sh so each now has a single implementation.
Every orchestrator invoked its children differently — four near-duplicate
run_job() copies, a differently-shaped run_watchdog(), or plain inline bash
calls, each with its own take on path resolution, pass/fail naming, and
dry-run threading. Extracted one shared run_orch_child() into common.sh so
there's a single place to fix or extend this behavior going forward.
Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were
checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log
silently did nothing beyond basic logging on those two. Fixed to
$ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no
trailing exit, so their exit codes reflected whatever the last command
happened to return rather than actual success/failure. transcode_management.sh
had no failure notification and no summary at all. Also made
transcode_management.sh's two-script pipeline config-driven
(TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room
to extend it later without editing the orchestrator itself.
Emby 4.9.5 rejects X-Api-Key — notify_emby_scan() now uses X-Emby-Token
consistently with emby_api(). Radarr v6 changed DownloadedMoviesScan to
require a path parameter — both cleanup scripts now use ProcessMonitoredDownloads
which achieves the same pre-flight intent without a path.
All three *_RECOVERY vars were missing from detect_hosts() so the conf-file
values were never surfaced — LIDARR_RECOVERY defaulted to false even though
HOST1_LIDARR_RECOVERY=true, keeping Lidarr disabled in arrs_failed_stalled_recovery.
Adds require_partnership() to common.sh — exits cleanly when PARTNERSHIP_ENABLED=false.
Removes FALLBACK_PARTNERSHIP_REQUIRED toggle — partnership is now always required,
not optional. Cross-server scripts (rsync, conf sync, fallback, arr sync, play state,
backup verify) all call require_partnership after detect_hosts.
stability_watchdog.sh mover check now uses platform_is_mover_running()
instead of raw pgrep, matching server_reboot.sh.
check_unraid_version_parity() renamed to check_os_version_parity() —
the function internals were already platform-neutral after the adapter
refactor; the name was the last Unraid artifact.
OS version reads now go through platform_get_os_version() and
platform_os_version_probe_cmd() instead of grepping /etc/unraid-version directly.
STATE_DIR fallbacks to /boot/config removed — STATE_DIR is always set by
load_config.sh and the fallback encoded a platform-specific path.
Setup DB path references use platform_setup_db_path() instead of the
VARAVERK_SETUP_FILE/-/boot/config compound fallback.
DOCKER_APPDATA_BASE default removed from arr_sync.sh — the adapter sets it.
detect_hosts() now sets REMOTE_STORAGE_PATH from HOST*_STORAGE_PATH in the
remote's conf instead of calling platform_storage_path() locally. SSH commands
that check the remote's storage (check_remote_array, check_remote_disks,
check_both_healthy) use $REMOTE_STORAGE_PATH so they work correctly when the
remote is a different platform with a different storage root.
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.
New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.
Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
common.sh gains:
- docker_cmd() + verify_running() + retry_docker() — removed from all 3 Docker_Essentials
scripts where they were byte-for-byte duplicates
- emby_api(endpoint, [timeout=30]) — removed from 6 Media/Tools scripts that each defined
their own _emby_api() with the same curl/parse/error pattern; call sites renamed emby_api
- format_duration() extended with days/hours branch (was capped at minutes+seconds)
- notify() comment: scripts do not need to preflight the notify script via validate_unraid_cmd
Tailscale deduplication:
- arr_sync.sh: _resolve_node_ip() and inline block in _delete_remote_item() both replaced
with resolve_tailscale_ip() from common.sh
- git_pull_execute.sh: inline tailscale ip -4 replaced with resolve_tailscale_ip() (adds
the tailscale status fallback that was missing)
common.sh: add is_docker_enabled() and is_vm_manager_enabled() helpers
reading /boot/config/docker.cfg and /boot/config/domain.cfg.
docker_watchdog: exit cleanly if Docker not enabled in Unraid settings.
stability_watchdog: skip Docker daemon check and Docker container stop
if Docker not enabled; skip virsh VM shutdown if VM Manager not enabled.
server_reboot: skip VM shutdown and libvirt stop if VM Manager not enabled.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move SCRIPTS_DIR/DATA_DIR/STATE_DIR from appdata to /boot/config/plugins/varaverk
- All state files now in STATE_DIR (no more /tmp or /boot/config root writes)
- Bootstrap: Gitea-first clone with GitHub fallback, no array dependency
- varaverk.cfg seeded with Gitea connection settings
- .gitignore: add State_Files/, varaverk.cfg, varaverk-*.txz
- Partnership/transcode/fallback scripts use STATE_DIR variables
- PHP config.php: DATA_DIR/STATE_DIR constants, VV_SETUP_STATE_FILE dynamic
- deploy.sh PROD_ROOT updated to plugin flash dir
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cron runs with PATH=/usr/bin:/bin, which omits /usr/local/sbin where
tailscale (and other Unraid tools) live. resolve_remote_ip silently got
empty results from both tailscale calls, failing all 3 retry attempts
every scheduled run. Export the full path at source time so all scripts
find tailscale regardless of how they're invoked.
Scripts calling acquire_lock followed by acquire_rsync_lock (rsync.sh)
or a custom EXIT trap (resource_watchdog.sh) would overwrite the lock
release trap, leaving the first lock file behind on exit.
Replace per-acquire trap with a _LOCK_FILES registry. _register_lock()
appends each lock file to the array and sets a single _release_all_locks
trap. All acquired locks release together on exit regardless of how many
traps are subsequently set or overwritten.
resource_watchdog.sh: chain _release_all_locks into both its custom trap
and the state-persisted disarm path so the lock releases in all exit paths.
The first cron run of critical_sync_maintenance failed because Tailscale
had a transient moment at startup. resolve_remote_ip() now retries 3 times
with a 5s delay before giving up, making it resilient to brief daemon
hiccups without masking real peer-offline conditions.
master.conf:
- reorder ORCHESTRATORS section by run frequency; add missing entries
- remove heartbeat vars (watchdogs no longer run in continuous cycle)
- remove dead REMOTE_DOCKER_RETRY_WAIT and display-only watchdog interval vars
- move INTERMEDIATE_RSYNC_ENABLED into RSYNC two-tier block; reorder by frequency
common.sh:
- add missing info() and success() definitions — both were called throughout but never defined
- update output tier description to list all 5 functions
- fix double-icon in check_local_disk_temps() error/warn calls
- fix smashed curl/ssh command lines in notify_emby_scan() and check_remote_docker_daemon()
- update check_rsync_enabled() comment to include INTERMEDIATE, ordered by frequency
emby_to_radarr_sync.sh / emby_to_sonarr_sync.sh:
- wire up RADARR_EMBY_LIBRARIES / SONARR_EMBY_LIBRARIES; empty array now scans all libraries
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
Partnership/gitea_ssh_setup.sh: generates ed25519 keypair, registers public key
with Gitea API (POST /api/v1/user/keys), tests SSH connection, writes ~/.ssh/config
entry. Idempotent — skips steps already done. --force regenerates and re-registers.
Resolves Gitea endpoint from container IP first, falls back to GITEA_DOMAIN.
common.sh: alias HOST*_GITEA_API_TOKEN → GITEA_API_TOKEN in detect_hosts().
Add to detect_hosts() doc comment.
master.conf: add GITEA_HTTP_PORT=3000 for API endpoint construction.
host1.conf: add HOST1_GITEA_API_TOKEN (fill in from Gitea Settings → Applications).
Add my-Gitea.xml to HOST1_PARTNERSHIP_AUTH_STACK — onboard pushes it to HOST2.
host2.conf: add Gitea to FALLBACK_HOST2_COVERS_HOST1_TIER1 — starts immediately
when HOST1 goes down, making the source of truth reachable independently of
HOST1's auth stack.
host1.conf: HOST1_RAMDISK_SIZE 8G→10G, WARN_GB 6.8→8.5, LOW_GB 5.5→7.
ramdisk_stop.sh: new Tools/ script — redirects symlink to SSD before unmount
so Emby keeps writing during maintenance, auto-falls back to lazy umount
when only directory handles are open (no active writes).
common.sh: fix `info` call at line 595 (undefined) → `log`.
Tools docs: added ramdisk_stop.sh to README table, categories, diagram, and Manual.
- playback_aware_radarr_discovery.sh: new two-stage movie discovery
using recently watched Emby movies as seeds and TMDB recommendations
as the similarity engine; scores on recency + rating + vote count;
adds HOST*_TMDB_API_KEY alias to detect_hosts() in common.sh
- emby_to_sonarr_sync.sh: restrict to SONARR_EMBY_LIBRARIES allowlist
(Anime, Kids Shows, Stand-Up Comedy, TV shows) — excludes Youtube
and Recorded Sports libraries; add garbage title filter for multi-
season folder names (S01-S03) and bare bracket tags ([Prof])
- emby_to_radarr_sync.sh: add _is_episode_title filter to block anime
episode files stored in Movies library (underscore/dot patterns,
fansub bracket format, episode markers, codec metadata strings)
- emby_to_lidarr_sync.sh: add U+FFFD Unicode replacement character
filter alongside the existing ASCII ? filter for encoding corruption
- master.conf: RADARR_DISCOVERY_* settings, SONARR/RADARR_EMBY_LIBRARIES
allowlists, Radarr discovery added to WEEKLY_MAINTENANCE_SCRIPTS
- downloaders_reset: connection check block before slskd API sections;
triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
common.sh, master confs, and new Manual/README docs
Consistent naming with the watchdog family (system_watchdog, docker_watchdog).
File renamed, all RM_ config variables and internal references updated to RW_
across master.conf, master_host1.conf, common.sh, docker_watchdog.sh,
system_watchdog.sh, watchdog_orchestrator.sh, and sunday_morning_coffee_report.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduce a four-layer self-healing stack replacing the continuous-loop watchdogs:
- resource_manager.sh (new): single-pass pressure reduction layer; throttles
SABnzbd/qBit at level 1, docker-pauses background containers at level 2,
docker-stops optional containers and signals docker_watchdog to defer at
level 3; graduated recovery with hysteresis
- watchdog_orchestrator.sh (new, Orchestrators/): runs resource_manager →
docker_watchdog → system_watchdog in sequence; intended for per-minute cron
via User Scripts; startup grace, acquire_lock to prevent pile-up, heartbeat
- docker_watchdog.sh: de-looped to single-pass; daemon strikes persisted to
state file across runs; cross-script coordination reads RM_STATE_FILE instead
of SYS_WATCHDOG_STATE_FILE
- system_watchdog.sh: de-looped to single-pass; stripped of all container
management (shutdown_non_essential_containers removed); reboot-only last resort
- master.conf: removed system_watchdog and docker_watchdog from
ARRAY_START_SCRIPTS; added WATCHDOG ORCHESTRATOR and RESOURCE MANAGER sections
- master_host1.conf: added RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS arrays
- common.sh: aliased RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS via detect_hosts()
- continuous_scripts_status.sh: moved to Tools/ (preserved for future use)
- sunday_morning_coffee_report.sh: watchdog section updated to use state file
mtime checks instead of is_running; added Resource Manager subsection;
fixed mem_shutdown grep filter pointing to wrong state file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)
Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
dry runs)
master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)
master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS
Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Arr sync (new):
- Media/arr_sync.sh — full mesh bidirectional sync across all HOST* nodes
- Lidarr (MusicBrainz), Sonarr (TVDB), Radarr (TMDB) all handled in one script
- Remote API keys read live from config.xml via SSH — never stored in conf files
- Shared blocklist (DATA_DIR/arr_sync_blocklist.tsv) merged from all nodes at runtime
- Graceful skip if arr not configured locally or not reachable on a remote node
- --blocklist-add / --blocklist-remove / --blocklist-list management flags
- daily_sync_maintenance.sh — arr sync runs as explicit phase before rsync
- partnership_onboard.sh — Step 3 bootstraps merged library on both sides at onboard
- master.conf — ARR_SYNC_* config block, DOCKER_APPDATA_BASE
Rsync / cleanup:
- DEFAULT_RSYNC_OPTS — removed --delete; arr_cleanup.sh owns orphan enforcement
- lidarr_cleanup.sh — removed HOST1-only guard; runs on any node with Lidarr configured
Config architecture:
- HOST1/HOST2 hostnames moved from master_host*.conf → master.conf (not credentials)
- Sparse checkout now works correctly: each server only needs its own host conf
- detect_hosts() still resolves MY_ID + REMOTE_ID via master.conf hostname values
Bug fix:
- common.sh line 493 — watchdog toggle eval had broken quoting; all SYS_WATCHDOG_CHECK_*
globals were silently set to empty instead of their configured values
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
${!REMOTE_ID:-unknown} with REMOTE_ID="" is a fatal bash error that terminates
the entire function — not just the line. All subsequent aliases (EMBY_CONTAINER,
TRANSCODE_SSD, RAMDISK_LOW_GB, SSH_KEY) were never set, causing awk syntax errors
and "SSD fallback path missing" warnings in every transcode script invocation.
Guard with an explicit empty check instead of relying on the :- fallback,
which bash does not apply when the indirect expansion itself is invalid.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gmer4Lfe
FailedProxy