Codebase-wide audit pass: fixed real bugs (SSH hangs missing BatchMode,
local-outside-function no-ops, variable name collisions, a truncated
ratio calc, wrong state-dir path, DARK vs NO_INTERNET drift, and more),
then pulled logic that was duplicated across multiple scripts — arr
cleanup safety gates, docker restart ordering, container maintenance
stop/restart, watchdog state-file helpers, partnership role resolution,
cert expiry checks, remote node discovery, and TMDB discovery scoring —
into common.sh so each now has a single implementation.
TRANSCODE_LOG_RETENTION was documented as trimming the daily transcode log
but never referenced — log grew unbounded. TRANSCODE_ORPHAN_AGE was shown in
--status but never used, so empty session folders were deleted immediately
instead of after the documented grace period, risking a race against ffmpeg
creating a folder just before writing its first segment.
docker_watchdog.sh's daemon-health thresholds were only hardcoded fallbacks
despite comments claiming they were master.conf-configurable, and it
referenced a heartbeat feature that was never implemented (that's owned by
watchdog_orchestrator.sh, its caller) — added the three thresholds to
master.conf for real and removed the stale heartbeat claim.
Also added the missing HOSTN_PARTNERSHIP_SERVICES_STACK block to
host.conf.template (containers.sh already read it via detect_hosts, just
never had a template entry) and corrected play_state_sync.sh's doc comment
for PLAY_SYNC_TYPES' actual default.
Same audit as the orchestrator standardization pass (2a062e5), extended to
every remaining script. Found the same class of bug independently recurring:
ramdisk_stop.sh checked $LOG (nothing assigns it, should be $ENABLE_LOGGING),
partnership_onboard.sh checked $LOG_MODE (same issue), emby_session_report.sh
checked $TRANSCODE_PCT which was never computed so the high-transcode alert
could never fire, and storage_migrate.sh never called detect_hosts() so
$MY_ID was empty, silently breaking the post-migration host*.conf update.
partnership_manager.sh used `local` at top-level script scope (invalid outside
a function) and had two master.conf path references missing "Configurations/".
Along the way: several scripts (share_setup.sh, conf_sync.sh,
downloaders_reset.sh, transcode_cleanup.sh, transcode_manager.sh,
remote_arr_cache_writer.sh, upgrade_webhook_handler.sh) had no explicit
trailing exit code, so they always reported success regardless of real
failures. play_state_sync.sh was missing the partnership gate its own header
documented, so remote play-state sync ran even with PARTNERSHIP_ENABLED=false;
it also always exited 0 on sync errors. arr_profile_enforcer.sh and
webhook_setup.sh hand-rolled their own flag parsing instead of common.sh's
parse_args, so --log silently did nothing on either.
system_watchdog.sh was itself an un-standardized mini-orchestrator — converted
to the shared run_orch_child()/JOB_PASS/JOB_FAIL pattern, added the missing
failure notification, and fixed dry-run to pass --dry-run down to children
instead of skipping them outright. Also fixed a stale webgui_watchdog.sh path
in master.conf.template that would break system_watchdog.sh on any fresh
install.
Closed a sibling-drift gap: radarr_cleanup.sh and sonarr_cleanup.sh were
missing lidarr_cleanup.sh's tracked-count percentage-drop safety gate and its
"not configured on this host, skip cleanly" guard — both now match Lidarr's
7-gate model.
Every orchestrator invoked its children differently — four near-duplicate
run_job() copies, a differently-shaped run_watchdog(), or plain inline bash
calls, each with its own take on path resolution, pass/fail naming, and
dry-run threading. Extracted one shared run_orch_child() into common.sh so
there's a single place to fix or extend this behavior going forward.
Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were
checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log
silently did nothing beyond basic logging on those two. Fixed to
$ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no
trailing exit, so their exit codes reflected whatever the last command
happened to return rather than actual success/failure. transcode_management.sh
had no failure notification and no summary at all. Also made
transcode_management.sh's two-script pipeline config-driven
(TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room
to extend it later without editing the orchestrator itself.
Custom Scripts (the Scheduler page's inline editor) used to save into the
git-tracked Custom/ folder, so anything saved there would end up on GitHub.
They now live in /boot/config/plugins/user.scripts/Varaverk/Scripts, same
folder family as Unraid's own User Scripts plugin. Import Script lets you
browse the whole server and move an existing script in instead of only
creating new ones inline — always a move, never a copy, so no stray
duplicate is left where it came from.
findmnt returns 'flash/boot' for a ZFS dataset, not a /dev/* path.
lsblk -no pkname then fails, leaving transport as 'unknown'. Resolve
ZFS pools to a backing device via zpool before checking transport.
Unraid API returns fsUsed=0 when a disk's filesystem is unmounted (spun
down). disks.ini keeps the last-known value in KB even after spindown —
use it as fallback when isSpinning=false and fsUsed=0. Also remove the
mounted-only guard in the ini fallback path (vv_disk_entry) for the same
reason. Stale comments in user_script_plug-in.sh and partnership_manager.sh
also cleaned up.
Exit code 0 on a required container (docker stop, UI stop) now reads as
intentional — no strike, no restart. Non-zero exits still trigger the
existing strike → restart path.
Adds --pause / --resume management commands and a persistent intentional-
stops state file for maintenance windows where even the exit-code heuristic
isn't enough. Containers auto-cleared from the list when seen running again.
Media README and Manual now cover only the 3 remaining scripts (permissions, cleaner,
play_state_sync). Arrs_Stack README and Manual cover all arr stack scripts including
lidarr_release_fixer. Fixed stale --skip-strike-list reference in flag docs.
Media/ now holds only media-level scripts (cleaner, permissions, play_state_sync).
All arr management scripts (cleanup, discovery, sync, webhooks, release fixer) live in Arrs_Stack/.
Reads MUSICBRAINZ_ALBUMID from FLAC (vorbis block type 4) and MP3 (ID3v2 TXXX)
files, matches against Lidarr's known releases, switches monitored=true to the
correct one, and queues RefreshArtist. Runs before lidarr_cleanup.sh in the
daily job list so the strike system doesn't act on files that just needed a
release correction.
Unraid 7's React CSS overrides line-height: 1.5 on textareas to normal.
vvRestoreEditorPrefs() only called vvFontSize() for non-default sizes, so
the inline style that overrides Unraid's CSS was never set for the default
12px case — leaving Firefox to render at ~14px while our code calculated
positions at 18px, putting the cursor 2-3 lines above where clicked.
Now always calls vvFontSize() so the inline lineHeight is always forced.
Unraid quotes values (DOCKER_ENABLED="yes") and uses SERVICE="enable" for
libvirt — the old patterns matched nothing, so docker_watchdog always skipped.
Emby 4.9.5 rejects X-Api-Key — notify_emby_scan() now uses X-Emby-Token
consistently with emby_api(). Radarr v6 changed DownloadedMoviesScan to
require a path parameter — both cleanup scripts now use ProcessMonitoredDownloads
which achieves the same pre-flight intent without a path.
All three *_RECOVERY vars were missing from detect_hosts() so the conf-file
values were never surfaced — LIDARR_RECOVERY defaulted to false even though
HOST1_LIDARR_RECOVERY=true, keeping Lidarr disabled in arrs_failed_stalled_recovery.
Every script now has the established header format: PURPOSE with ─────── separator,
OPERATIONAL MODEL, DESIGN PRINCIPLES, OPERATIONAL SAFEGUARDS, CONFIGURATION, and
RUNTIME MODES — structured with full ====== banner sections throughout.
Orchestrators converted from compact ── inline format to full banners. Stale
emby-fallback and dirty sync references removed from Plugin/user_script_plug-in.sh.
docker_weekly_restart.sh was missing the trailing dangling prune that daily_restart has.
docker_update.sh rebuild failure message always named docker_daily_restart.sh regardless of mode.
docker_prune_images.sh --all added to monthly — the only scheduled path that removes tagged orphan images.
Favorites on any server propagate to all others; never unmarks.
Covers MusicArtist, MusicAlbum, Movie, Series (Audio tracks future).
Provider map extended to include FAV_TYPES alongside SYNC_TYPES,
with Series/Episode TVDB IDs namespaced to avoid collisions.
Retries up to PLAY_SYNC_HANDBACK_RETRIES times (default 5, 60s apart)
before giving up — one successful run catches all state regardless of
outage length, so users land on current watch state after DNS flips.
Emby now runs continuously on both hosts — play_state_sync via API
handles watch state reconciliation, making the 30-min dirty rsync
redundant. Handback runs play_state_sync --wait before DNS cutover
so users land on current state after a failover event.
- Emby removed from FALLBACK_HOST1_TIER1 (always running)
- Emby dirty sync removed from HOST1_CRITICAL_SYNC_SHARES
- Emby writeback removed from FALLBACK_HOST1_WRITEBACK_TIER1
- play_state_sync gains --wait flag (uses lock wait mode for handback)
- Tier 1 emby-fallback special case removed from handback writeback loop
- play_state_sync --wait added as Step 7 in handback, before DNS cutover
Single mixed Movie,Episode,Audio query caused Jellyfin to reorder items unpredictably,
pushing most episodes past the page limit and leaving PROV_LOOKUP empty — every
Emby→Jellyfin push silently failed with 'item not found'. Also removes the
ExcludeLocationTypes=Virtual param which caused the same corruption in JF.
PHP scheduler load at shutdown time can fail silently (errors suppressed); a failed PHP call meant the stop script never ran without any visible indication.
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.
PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.
conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.
fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.
README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
Adds require_partnership() to common.sh — exits cleanly when PARTNERSHIP_ENABLED=false.
Removes FALLBACK_PARTNERSHIP_REQUIRED toggle — partnership is now always required,
not optional. Cross-server scripts (rsync, conf sync, fallback, arr sync, play state,
backup verify) all call require_partnership after detect_hosts.
Writes partner confs from RAM cache to /boot/config/.cache/vv/d/ while
remote is offline, and removes the backup when remote comes back. Called
each minute via SYSTEM_WATCHDOG_SCRIPTS so crashes and power loss are
covered — not just graceful shutdowns.
All FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER* references updated to
FALLBACK_${REMOTE_ID}_TIER* across fallback_test.sh, partnership_manager.sh,
docker_update.sh, mesh_monitor.sh, and monitor.php. mesh_monitor.sh drops
the inner covering-host loop — tier data now lives in the covered host's own
conf so no cross-host scan is needed. monitor.php reads from the covered
host's conf file rather than the local host's.
Previously HOST1 defined what it would run for HOST2 (FALLBACK_HOST1_COVERS_HOST2_TIER*).
Now each host defines what it wants run when IT goes down (FALLBACK_HOST1_TIER*), and the
covering server reads the down host's conf via the RAM/persistent cache.
get_tier_containers() reads FALLBACK_${REMOTE_ID}_TIER* instead of
FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*. Tier data migrated to the correct host confs.
Writeback paths and delays were already REMOTE_ID-based — no change needed there.
conf_cache_save.sh runs first on array stop — snapshots partner confs from RAM
cache to /boot/config/varaverk/conf_bak/ before anything else shuts down.
conf_cache_restore.sh runs after conf_sync.sh on array start — if partner was
unreachable and RAM cache is incomplete, loads the backup into RAM then removes
it. Normal reboots: backup written, fresh pull succeeds, backup deleted unused.
Edge case (partner down at boot): backup fills the gap so fallback.sh has the
partner vars it needs to operate correctly.
Web files now served via symlink to the git repo so git pull changes survive
reboots without rebuilding the txz. Also includes: docker pull/rebuild/restart
with live log streaming, arr_profile_enforcer for Sonarr/Radarr quality
profiles, monitor page cache fix (background writer now in cron), and
ARR_KIDS/SONARR/RADARR profile name vars in master.conf.
Adds SSH key (hostname convention), arr path maps (docker volume mounts),
Authelia container + config path, boot device storage mode detection,
and master.conf HOST identity + Gitea container. Also fixes RADARR_MOVIE_ROOT
→ RADARR_MOVIES_ROOT to match the variable name used by all other scripts.
Live TV and Direct Stream sessions never appear in the transcoding
session list but do write segments to the ramdisk. Card now shows
the active segment count so the ramdisk usage is explained.
HLS segments are written atomically so lsof never sees them as open.
Cleanup now shows "Active" count (too-young files) which correctly
reflects Live TV / Direct Stream segments. Manager session count now
counts flat file prefixes in addition to subdirs.
Weekly sync window was pulling images but using docker start after rsync,
which ignores the new digest. Containers in the emby/critical-data profiles
(Emby, Mariadb, Redis) never actually landed on updated images.
docker_update_remaining.sh merged into docker_update.sh --remainder, which
already had better exclusion logic. Added WEEKLY_REMAINING_UPDATES toggle
and WEEKLY_RESTART_CONTAINERS exclusion to remainder mode.
Onboard now runs docker_network_connect.sh on mirror before deploying stacks.
Fix discrepancies found by auditing docs against headers:
- stability_watchdog.sh: wrong variable name (STRIKES → STRIKE_LIMIT) and wrong default (2hr → 12hr) in header
- Watchdogs manual: REBOOT_WINDOW_HRS example value was 2, should be 12
- Transcodes README: transcode_cleanup/manager table said "Every 3 min", should be "Every 7 minutes"
- Tools manual: fallback_state_reset.sh still marked "not yet built" — rewrite section to reflect current script
- Tools manual: claude_startup --setup flag doesn't exist; modes were inverted — fix and add --launch
- Tools README + manual: add docker_prune_images.sh (existed but undocumented)
- Tools manual: add play_state_sync.sh --full to flag reference
- Orchestrators README: arrs_failed_stalled_recovery runs via intermediate_sync (every 4hr), not standalone every 6hr
- Plugin README: add build.sh to scripts table
- Partnership README + manual: add partnership_transfer.sh and onboard_cancel.sh to script tables and flag reference
PHP reads master.conf values like STATE_DIR="${SCRIPTS_DIR}/State_Files"
literally — the ${SCRIPTS_DIR} token is never expanded, so any PHP code
using the returned STATE_DIR value got an invalid path. Two-pass resolution:
SCRIPTS_DIR first (from PHP constant), then remaining ${VAR} tokens using
the now-resolved var set (covers DATA_DIR-based paths like ARR_SYNC_BLOCKLIST).
Behavior fixes:
- docker_daily_restart.sh: implement RESTART_VERIFY_WAIT sleep before
verify_running() — variable was logged as config but never applied
- cert_monitor.sh: --log now shows expiry date and days remaining for
healthy domains in the per-domain summary; problems always show details
Header corrections:
- downloaders_reset.sh: fix frequency contradiction (15 min → 30 min)
- lidarr_cleanup.sh: Emby scan triggers on actual deletions, not every run
- ssh_setup.sh: document --local-only flag in USAGE section
- zfs_memory_snapshot.sh: clarify dry-run skips log write in PURPOSE
- ramdisk_stop.sh / ramdisk_setup.sh: document TRANSCODE_STATE_FILE var
- smart_long_test.sh: remove SMART_TEMP_WARN/CRIT — not used in code
stability_watchdog.sh mover check now uses platform_is_mover_running()
instead of raw pgrep, matching server_reboot.sh.
check_unraid_version_parity() renamed to check_os_version_parity() —
the function internals were already platform-neutral after the adapter
refactor; the name was the last Unraid artifact.
OS version reads now go through platform_get_os_version() and
platform_os_version_probe_cmd() instead of grepping /etc/unraid-version directly.
STATE_DIR fallbacks to /boot/config removed — STATE_DIR is always set by
load_config.sh and the fallback encoded a platform-specific path.
Setup DB path references use platform_setup_db_path() instead of the
VARAVERK_SETUP_FILE/-/boot/config compound fallback.
DOCKER_APPDATA_BASE default removed from arr_sync.sh — the adapter sets it.
detect_hosts() now sets REMOTE_STORAGE_PATH from HOST*_STORAGE_PATH in the
remote's conf instead of calling platform_storage_path() locally. SSH commands
that check the remote's storage (check_remote_array, check_remote_disks,
check_both_healthy) use $REMOTE_STORAGE_PATH so they work correctly when the
remote is a different platform with a different storage root.
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.
New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.
Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
host.conf.template: add AUTH STACK section (NPM/lldap/Authelia) so
conf_upgrade.sh stops stripping those keys from host*.conf at 1am.
setup wizard: storage mode selector auto-detects USB vs NVMe at first
run; user can override; triggers storage_migrate.sh when mode differs
from current SCRIPTS_DIR location.
claude_startup.sh: reads HOST*_STORAGE_MODE_INTERNAL to pick internal
or appdata path rather than always requiring array to be mounted.
- REMOTE_ID_TIER*_DELAY renamed to HOSTN_TIER*_DELAY — these are own-host outage timers
- HOSTN_RADARR_MOVIE_ROOT → HOSTN_RADARR_MOVIES_ROOT (matches common.sh)
- HOSTN_SYS_WATCHDOG_CHECK_DOCKER → CHECK_DOCKER_DAEMON, CHECK_TEMP → CHECK_CPU_TEMP
- Add REMOTE_ID substitution to the sed in git_pull_execute.sh so COVERS_REMOTE_ID
resolves to the actual remote host (e.g. FALLBACK_HOST1_COVERS_HOST2_TIER1)
- Add HOSTN_PARTNERSHIP_REPLACE_CONTAINERS, HOSTN_PARTNERSHIP_ARR_REPLACE_CONTAINERS,
and HOSTN_WATCHDOG_CONTAINER_API_CHECKS to template so they aren't removed each pull
Section order now matches master.conf: IDENTITY → PARTNERSHIP → RSYNC →
FALLBACK → DOCKER → MEDIA → ARR STACK → TRANSCODES → MONITORS →
RESOURCE MANAGER → SYSTEM WATCHDOG. Extracted PARTNERSHIP from IDENTITY
into its own top-level section. PERSONAL SHARES moved to sit directly
after DAILY SYNC SHARES. NIC var moved to top of SYSTEM WATCHDOG block.
STATE_DIR introduced, DATA_DIR path updated to flash, all state files
migrated from /tmp and /boot/config root to $STATE_DIR. TRANSCODE_STATE_FILE
and DOCKER_WATCHDOG_FAILED_FILE added; SYS_WATCHDOG_FAILED_FILE renamed to
DOCKER_WATCHDOG_FAILED_FILE to reflect what it actually tracks.
Gmer4Lfe