Everything a child printed was interleaved into the parent's log with forty
other scripts, separated only by human headings that map to no script id — so
scoped_log could answer for ten orchestrators and nothing else. tee rather than
a redirect, so the parent keeps receiving exactly what it did before and this
only adds a destination; PIPESTATUS[0] preserves the child's exit code through
the pipe. Trimmed to 1000 lines like run_job.sh does, because LOG_DIR is tmpfs
and an untrimmed chatty script spends RAM. Falls back to the original unpiped
call when the log cannot be opened — a child must never be denied a run because
somewhere to write about it could not be created.
run_job.sh records the ten jobs cron starts; the seventy entries inside the
*_SCRIPTS lists had none, and nothing said so. Consumers quietly got less — arr
cleanup stats fell through to a daily aggregate with no end time, and
troubleshoot answered "I cannot see a run record" for every script that is not
an orchestrator, which is nearly all of them and exactly the ones a question is
about. run_orch_child() already had the start time, exit code and severity
ladder; it just never wrote them down. Not run_job.sh itself: that also locks
and interval-gates, which would let a child skip itself or block on its parent.
Every existing exemption lasts until someone remembers to undo it, and nobody
does — Healarr has sat in a pressure list since it was uninstalled and seven
ignore entries name containers that are gone. A mute states when it ends and
then ends, capped by WATCHDOG_MUTE_MAX_HOURS so temporary is enforced rather
than intended. Applied where IGNORE_MAP is built, so all five check sites
inherit it, and shown with its countdown because an invisible suppression is
the thing being fixed.
run_job.sh already grades runs as ok/warn/error by exit code, so a job that exited 1
recorded a warn while its log showed a red failure line — and the log is the louder of
the two.
Finishes the pass: every script now documents its safeguards, and the deliberate absences
in the sourced libraries are recorded so they are not "corrected" later.
Acts on forward misplacements and reverse-kids-leak (adult content in
the kids root — not a judgment call, same reasoning as the Radarr
side), moving to the new SONARR_GENERAL_ROOT config var. Reverse-anime
-leak stays report-only — deliberate style placements like Castlevania
are common and valid there. episodeFileCount is Sonarr's hasFile
equivalent: series with 0 files get relocated + an immediate
SeriesSearch instead of a file move. One series at a time, verified
after each — matches the race condition found doing this by hand for
the Fate/Zero and Fate/Stay Night moves earlier in the session.
Unlike reverse-anime-leak (deliberate style placements like Castlevania
are common and valid there), reverse-kids-leak's signal is specifically
"adult certification with zero Family/Animation genre" — confirmed live
the 4 titles it catches (Addams Family, Saving Mr. Banks, Dark Shadows,
The DUFF) are all genuinely non-kids content. Moves them to the new
RADARR_GENERAL_ROOT config var (container-path literal for the general
Movies root, needed since RADARR_MOVIES_ROOT is a filesystem path).
Overseerr lets users request content into the wrong root folder; these
new report-only scans classify every tracked movie/series (anime, kids,
regular) from metadata alone and flag mismatches against the actual
root folder, in both directions. Rules were validated against real
library data before being adopted — see the header comments in each
script and the master.conf notes above the curated lists.
This turned out to be the actual dominant cost left in the cleanup
scripts' classification loop, hidden behind the stat/dirname forks fixed
earlier today -- called once per non-tracked file (most files in a
media library, since every protected sidecar counts). Measured: 47.3s
vs 1.65s for 10,000 calls (~28.6x), identical results confirmed via
spot check before switching. Only used by lidarr_cleanup.sh/
sonarr_cleanup.sh/radarr_cleanup.sh.
Comment-only. Headers on the scripts touched during today's caching work
(cache-first fetches, write-through per-item cache, single-walk
consolidation, movieFile-embedded fix) still described pre-change
behavior. Also brought common.sh's top-level cache doc block current --
it was written for the single-consumer 2026-07-16 state and didn't
mention the tmpfs move, the write guard, or the 15+ consumers that now
go through it.
lidarr_cleanup.sh and sonarr_cleanup.sh already walk every artist/series
individually (trackFile/episodefile) for their own cleanup decisions --
that walk now also writes the raw per-item data through to a short-lived
tmpfs cache (arr_item_cache_write, 4h freshness, no persistent backup
since every consumer already has its own live fallback). lidarr_missing_art.sh
runs later in the same nightly window and now reads that cache first,
skipping its own redundant per-artist walk entirely on a hit. Sonarr side
is write-through only for now -- no second consumer exists yet, but the
data's there for whenever one does. Future consumers: arr_get_cached_items()
first, live per-item fetch as fallback, same pattern as these two.
Reads/writes now hit tmpfs (ARR_CACHE_DIR) instead of the array disk --
a full rebuild for all three arrs measures ~12s live, so there's no real
cost to losing it on reboot. The existing on-disk file becomes a backup
that arr_cache_write() keeps in sync on every write, and
arr_cache_age_seconds() transparently restores it into tmpfs the moment
it notices tmpfs is missing -- so a cache that was fresh before reboot
reads as fresh after too, closing the cold-start gap without needing a
dedicated restore step anywhere else.
A direct arr_cache_write() call mid-rescan wrote a partial snapshot that
looked like real data loss to every consumer of the cache. The guard now
lives in arr_cache_write() itself so every caller is protected, not just
arr_get_tracked_data(). arr_rescan_monitor.sh closes the resulting gap for
rescans triggered outside arr_full_rescan.sh's own trigger-and-wait path.
Shared cache/rescan-duration logic in common.sh now takes an arr_type
param instead of being Lidarr-specific, so Sonarr and Radarr cleanup
scripts get the same cache-first fetch + rescan-aware retry Lidarr had.
Avoids redundant full-library API calls across scripts run back to back,
and stops false failures when a fetch lands mid-rescan.
Shared cache (lidarr_get_tracked_data() in common.sh) so scripts stop
hitting Lidarr's live API for tracked counts every run, and stop
treating a mid-rescan dip as a genuine problem — a whole-library
RescanFolders legitimately makes trackFileCount read far below normal
while it re-verifies every file (confirmed 2026-07-16: 22% of normal
mid-scan). Cache reads fresh-if-recent, waits out an active rescan
(calibrated to that command's own historical duration, tracked per
command name since RescanFolders and DownloadedAlbumsScan take wildly
different amounts of time), then falls back to a stale cache rather
than hard-failing after a few strikes.
lidarr_cleanup.sh: no longer stacks a fresh DownloadedAlbumsScan on
top of one already running, and the tracked-count floor check now
waits out a genuine rescan instead of aborting on every overlap.
lidarr_duplicate_artist_cleanup.sh (new): finds case-insensitive
duplicate artist entries — same display name, different MusicBrainz
ID, added when a search/list-sync matches the wrong same-named artist.
Deletes the empty phantom side and blocks it from Import List
Exclusions, leaves genuinely-different-real-artists alone (checked by
album title overlap, deduped per-artist first so a legitimate reissue
under an artist's own catalog doesn't false-flag as cross-artist
overlap), and only notifies for the rare case where both sides have
real, overlapping content.
lidarr_cache_prefill.sh (new): warms the cache at array start so nothing
reads it cold after boot.
lidarr_missing_art.sh, lidarr_release_fixer.sh: write-through the cache
as a side effect of fetches they already needed for their own purposes.
Unraid truncates the Server Name to 15 chars. Depending on which form
ends up in master.conf's HOST* value (the truncated OS hostname, or a
longer name matching what Tailscale independently registered for that
peer), either host-identity detection or Tailscale IP resolution could
fail — hit live on HOST2 in both directions this session.
- common.sh detect_hosts(): was case-sensitive exact match only, with no
fallback and an exit 1 on failure — meaning every orchestrator/watchdog/
rsync script would hard-fail on a truncated-hostname host, not just the
web UI. Now case-insensitive, with a prefix-match fallback when the
local hostname is exactly 15 chars.
- common.sh resolve_tailscale_ip(): already did a bare regex prefix match
with zero ambiguity handling (pre-existing risk: e.g. server1/server10
could collide). Replaced with an explicit unambiguous-only check.
- config.php vv_detect_host() / vv_resolve_tailscale_ip(): same treatment,
kept as close a mirror of the bash logic as PHP allows.
All fallbacks require an EXACT prefix match (never fuzzy/percentage
similarity — considered and rejected, since names like server1/server2/
server3 would be dangerously similar under any generic similarity metric)
and require exactly one candidate to qualify; ambiguous matches are
treated as no match rather than guessed. Verified live against HOST1/
HOST2 in both master.conf configurations (short and long HOST2 value).
Codebase-wide audit pass: fixed real bugs (SSH hangs missing BatchMode,
local-outside-function no-ops, variable name collisions, a truncated
ratio calc, wrong state-dir path, DARK vs NO_INTERNET drift, and more),
then pulled logic that was duplicated across multiple scripts — arr
cleanup safety gates, docker restart ordering, container maintenance
stop/restart, watchdog state-file helpers, partnership role resolution,
cert expiry checks, remote node discovery, and TMDB discovery scoring —
into common.sh so each now has a single implementation.
Every orchestrator invoked its children differently — four near-duplicate
run_job() copies, a differently-shaped run_watchdog(), or plain inline bash
calls, each with its own take on path resolution, pass/fail naming, and
dry-run threading. Extracted one shared run_orch_child() into common.sh so
there's a single place to fix or extend this behavior going forward.
Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were
checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log
silently did nothing beyond basic logging on those two. Fixed to
$ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no
trailing exit, so their exit codes reflected whatever the last command
happened to return rather than actual success/failure. transcode_management.sh
had no failure notification and no summary at all. Also made
transcode_management.sh's two-script pipeline config-driven
(TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room
to extend it later without editing the orchestrator itself.
Emby 4.9.5 rejects X-Api-Key — notify_emby_scan() now uses X-Emby-Token
consistently with emby_api(). Radarr v6 changed DownloadedMoviesScan to
require a path parameter — both cleanup scripts now use ProcessMonitoredDownloads
which achieves the same pre-flight intent without a path.
All three *_RECOVERY vars were missing from detect_hosts() so the conf-file
values were never surfaced — LIDARR_RECOVERY defaulted to false even though
HOST1_LIDARR_RECOVERY=true, keeping Lidarr disabled in arrs_failed_stalled_recovery.
Adds require_partnership() to common.sh — exits cleanly when PARTNERSHIP_ENABLED=false.
Removes FALLBACK_PARTNERSHIP_REQUIRED toggle — partnership is now always required,
not optional. Cross-server scripts (rsync, conf sync, fallback, arr sync, play state,
backup verify) all call require_partnership after detect_hosts.
stability_watchdog.sh mover check now uses platform_is_mover_running()
instead of raw pgrep, matching server_reboot.sh.
check_unraid_version_parity() renamed to check_os_version_parity() —
the function internals were already platform-neutral after the adapter
refactor; the name was the last Unraid artifact.
OS version reads now go through platform_get_os_version() and
platform_os_version_probe_cmd() instead of grepping /etc/unraid-version directly.
STATE_DIR fallbacks to /boot/config removed — STATE_DIR is always set by
load_config.sh and the fallback encoded a platform-specific path.
Setup DB path references use platform_setup_db_path() instead of the
VARAVERK_SETUP_FILE/-/boot/config compound fallback.
DOCKER_APPDATA_BASE default removed from arr_sync.sh — the adapter sets it.
detect_hosts() now sets REMOTE_STORAGE_PATH from HOST*_STORAGE_PATH in the
remote's conf instead of calling platform_storage_path() locally. SSH commands
that check the remote's storage (check_remote_array, check_remote_disks,
check_both_healthy) use $REMOTE_STORAGE_PATH so they work correctly when the
remote is a different platform with a different storage root.
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.
New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.
Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
common.sh gains:
- docker_cmd() + verify_running() + retry_docker() — removed from all 3 Docker_Essentials
scripts where they were byte-for-byte duplicates
- emby_api(endpoint, [timeout=30]) — removed from 6 Media/Tools scripts that each defined
their own _emby_api() with the same curl/parse/error pattern; call sites renamed emby_api
- format_duration() extended with days/hours branch (was capped at minutes+seconds)
- notify() comment: scripts do not need to preflight the notify script via validate_unraid_cmd
Tailscale deduplication:
- arr_sync.sh: _resolve_node_ip() and inline block in _delete_remote_item() both replaced
with resolve_tailscale_ip() from common.sh
- git_pull_execute.sh: inline tailscale ip -4 replaced with resolve_tailscale_ip() (adds
the tailscale status fallback that was missing)
common.sh: add is_docker_enabled() and is_vm_manager_enabled() helpers
reading /boot/config/docker.cfg and /boot/config/domain.cfg.
docker_watchdog: exit cleanly if Docker not enabled in Unraid settings.
stability_watchdog: skip Docker daemon check and Docker container stop
if Docker not enabled; skip virsh VM shutdown if VM Manager not enabled.
server_reboot: skip VM shutdown and libvirt stop if VM Manager not enabled.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move SCRIPTS_DIR/DATA_DIR/STATE_DIR from appdata to /boot/config/plugins/varaverk
- All state files now in STATE_DIR (no more /tmp or /boot/config root writes)
- Bootstrap: Gitea-first clone with GitHub fallback, no array dependency
- varaverk.cfg seeded with Gitea connection settings
- .gitignore: add State_Files/, varaverk.cfg, varaverk-*.txz
- Partnership/transcode/fallback scripts use STATE_DIR variables
- PHP config.php: DATA_DIR/STATE_DIR constants, VV_SETUP_STATE_FILE dynamic
- deploy.sh PROD_ROOT updated to plugin flash dir
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cron runs with PATH=/usr/bin:/bin, which omits /usr/local/sbin where
tailscale (and other Unraid tools) live. resolve_remote_ip silently got
empty results from both tailscale calls, failing all 3 retry attempts
every scheduled run. Export the full path at source time so all scripts
find tailscale regardless of how they're invoked.
Scripts calling acquire_lock followed by acquire_rsync_lock (rsync.sh)
or a custom EXIT trap (resource_watchdog.sh) would overwrite the lock
release trap, leaving the first lock file behind on exit.
Replace per-acquire trap with a _LOCK_FILES registry. _register_lock()
appends each lock file to the array and sets a single _release_all_locks
trap. All acquired locks release together on exit regardless of how many
traps are subsequently set or overwritten.
resource_watchdog.sh: chain _release_all_locks into both its custom trap
and the state-persisted disarm path so the lock releases in all exit paths.
The first cron run of critical_sync_maintenance failed because Tailscale
had a transient moment at startup. resolve_remote_ip() now retries 3 times
with a 5s delay before giving up, making it resilient to brief daemon
hiccups without masking real peer-offline conditions.
master.conf:
- reorder ORCHESTRATORS section by run frequency; add missing entries
- remove heartbeat vars (watchdogs no longer run in continuous cycle)
- remove dead REMOTE_DOCKER_RETRY_WAIT and display-only watchdog interval vars
- move INTERMEDIATE_RSYNC_ENABLED into RSYNC two-tier block; reorder by frequency
common.sh:
- add missing info() and success() definitions — both were called throughout but never defined
- update output tier description to list all 5 functions
- fix double-icon in check_local_disk_temps() error/warn calls
- fix smashed curl/ssh command lines in notify_emby_scan() and check_remote_docker_daemon()
- update check_rsync_enabled() comment to include INTERMEDIATE, ordered by frequency
emby_to_radarr_sync.sh / emby_to_sonarr_sync.sh:
- wire up RADARR_EMBY_LIBRARIES / SONARR_EMBY_LIBRARIES; empty array now scans all libraries
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
Partnership/gitea_ssh_setup.sh: generates ed25519 keypair, registers public key
with Gitea API (POST /api/v1/user/keys), tests SSH connection, writes ~/.ssh/config
entry. Idempotent — skips steps already done. --force regenerates and re-registers.
Resolves Gitea endpoint from container IP first, falls back to GITEA_DOMAIN.
common.sh: alias HOST*_GITEA_API_TOKEN → GITEA_API_TOKEN in detect_hosts().
Add to detect_hosts() doc comment.
master.conf: add GITEA_HTTP_PORT=3000 for API endpoint construction.
host1.conf: add HOST1_GITEA_API_TOKEN (fill in from Gitea Settings → Applications).
Add my-Gitea.xml to HOST1_PARTNERSHIP_AUTH_STACK — onboard pushes it to HOST2.
host2.conf: add Gitea to FALLBACK_HOST2_COVERS_HOST1_TIER1 — starts immediately
when HOST1 goes down, making the source of truth reachable independently of
HOST1's auth stack.
host1.conf: HOST1_RAMDISK_SIZE 8G→10G, WARN_GB 6.8→8.5, LOW_GB 5.5→7.
ramdisk_stop.sh: new Tools/ script — redirects symlink to SSD before unmount
so Emby keeps writing during maintenance, auto-falls back to lazy umount
when only directory handles are open (no active writes).
common.sh: fix `info` call at line 595 (undefined) → `log`.
Tools docs: added ramdisk_stop.sh to README table, categories, diagram, and Manual.
- playback_aware_radarr_discovery.sh: new two-stage movie discovery
using recently watched Emby movies as seeds and TMDB recommendations
as the similarity engine; scores on recency + rating + vote count;
adds HOST*_TMDB_API_KEY alias to detect_hosts() in common.sh
- emby_to_sonarr_sync.sh: restrict to SONARR_EMBY_LIBRARIES allowlist
(Anime, Kids Shows, Stand-Up Comedy, TV shows) — excludes Youtube
and Recorded Sports libraries; add garbage title filter for multi-
season folder names (S01-S03) and bare bracket tags ([Prof])
- emby_to_radarr_sync.sh: add _is_episode_title filter to block anime
episode files stored in Movies library (underscore/dot patterns,
fansub bracket format, episode markers, codec metadata strings)
- emby_to_lidarr_sync.sh: add U+FFFD Unicode replacement character
filter alongside the existing ASCII ? filter for encoding corruption
- master.conf: RADARR_DISCOVERY_* settings, SONARR/RADARR_EMBY_LIBRARIES
allowlists, Radarr discovery added to WEEKLY_MAINTENANCE_SCRIPTS
- downloaders_reset: connection check block before slskd API sections;
triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
common.sh, master confs, and new Manual/README docs
Consistent naming with the watchdog family (system_watchdog, docker_watchdog).
File renamed, all RM_ config variables and internal references updated to RW_
across master.conf, master_host1.conf, common.sh, docker_watchdog.sh,
system_watchdog.sh, watchdog_orchestrator.sh, and sunday_morning_coffee_report.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduce a four-layer self-healing stack replacing the continuous-loop watchdogs:
- resource_manager.sh (new): single-pass pressure reduction layer; throttles
SABnzbd/qBit at level 1, docker-pauses background containers at level 2,
docker-stops optional containers and signals docker_watchdog to defer at
level 3; graduated recovery with hysteresis
- watchdog_orchestrator.sh (new, Orchestrators/): runs resource_manager →
docker_watchdog → system_watchdog in sequence; intended for per-minute cron
via User Scripts; startup grace, acquire_lock to prevent pile-up, heartbeat
- docker_watchdog.sh: de-looped to single-pass; daemon strikes persisted to
state file across runs; cross-script coordination reads RM_STATE_FILE instead
of SYS_WATCHDOG_STATE_FILE
- system_watchdog.sh: de-looped to single-pass; stripped of all container
management (shutdown_non_essential_containers removed); reboot-only last resort
- master.conf: removed system_watchdog and docker_watchdog from
ARRAY_START_SCRIPTS; added WATCHDOG ORCHESTRATOR and RESOURCE MANAGER sections
- master_host1.conf: added RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS arrays
- common.sh: aliased RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS via detect_hosts()
- continuous_scripts_status.sh: moved to Tools/ (preserved for future use)
- sunday_morning_coffee_report.sh: watchdog section updated to use state file
mtime checks instead of is_running; added Resource Manager subsection;
fixed mem_shutdown grep filter pointing to wrong state file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gmer4Lfe