common.sh: add is_docker_enabled() and is_vm_manager_enabled() helpers
reading /boot/config/docker.cfg and /boot/config/domain.cfg.
docker_watchdog: exit cleanly if Docker not enabled in Unraid settings.
stability_watchdog: skip Docker daemon check and Docker container stop
if Docker not enabled; skip virsh VM shutdown if VM Manager not enabled.
server_reboot: skip VM shutdown and libvirt stop if VM Manager not enabled.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move SCRIPTS_DIR/DATA_DIR/STATE_DIR from appdata to /boot/config/plugins/varaverk
- All state files now in STATE_DIR (no more /tmp or /boot/config root writes)
- Bootstrap: Gitea-first clone with GitHub fallback, no array dependency
- varaverk.cfg seeded with Gitea connection settings
- .gitignore: add State_Files/, varaverk.cfg, varaverk-*.txz
- Partnership/transcode/fallback scripts use STATE_DIR variables
- PHP config.php: DATA_DIR/STATE_DIR constants, VV_SETUP_STATE_FILE dynamic
- deploy.sh PROD_ROOT updated to plugin flash dir
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cron runs with PATH=/usr/bin:/bin, which omits /usr/local/sbin where
tailscale (and other Unraid tools) live. resolve_remote_ip silently got
empty results from both tailscale calls, failing all 3 retry attempts
every scheduled run. Export the full path at source time so all scripts
find tailscale regardless of how they're invoked.
Scripts calling acquire_lock followed by acquire_rsync_lock (rsync.sh)
or a custom EXIT trap (resource_watchdog.sh) would overwrite the lock
release trap, leaving the first lock file behind on exit.
Replace per-acquire trap with a _LOCK_FILES registry. _register_lock()
appends each lock file to the array and sets a single _release_all_locks
trap. All acquired locks release together on exit regardless of how many
traps are subsequently set or overwritten.
resource_watchdog.sh: chain _release_all_locks into both its custom trap
and the state-persisted disarm path so the lock releases in all exit paths.
The first cron run of critical_sync_maintenance failed because Tailscale
had a transient moment at startup. resolve_remote_ip() now retries 3 times
with a 5s delay before giving up, making it resilient to brief daemon
hiccups without masking real peer-offline conditions.
master.conf:
- reorder ORCHESTRATORS section by run frequency; add missing entries
- remove heartbeat vars (watchdogs no longer run in continuous cycle)
- remove dead REMOTE_DOCKER_RETRY_WAIT and display-only watchdog interval vars
- move INTERMEDIATE_RSYNC_ENABLED into RSYNC two-tier block; reorder by frequency
common.sh:
- add missing info() and success() definitions — both were called throughout but never defined
- update output tier description to list all 5 functions
- fix double-icon in check_local_disk_temps() error/warn calls
- fix smashed curl/ssh command lines in notify_emby_scan() and check_remote_docker_daemon()
- update check_rsync_enabled() comment to include INTERMEDIATE, ordered by frequency
emby_to_radarr_sync.sh / emby_to_sonarr_sync.sh:
- wire up RADARR_EMBY_LIBRARIES / SONARR_EMBY_LIBRARIES; empty array now scans all libraries
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
Partnership/gitea_ssh_setup.sh: generates ed25519 keypair, registers public key
with Gitea API (POST /api/v1/user/keys), tests SSH connection, writes ~/.ssh/config
entry. Idempotent — skips steps already done. --force regenerates and re-registers.
Resolves Gitea endpoint from container IP first, falls back to GITEA_DOMAIN.
common.sh: alias HOST*_GITEA_API_TOKEN → GITEA_API_TOKEN in detect_hosts().
Add to detect_hosts() doc comment.
master.conf: add GITEA_HTTP_PORT=3000 for API endpoint construction.
host1.conf: add HOST1_GITEA_API_TOKEN (fill in from Gitea Settings → Applications).
Add my-Gitea.xml to HOST1_PARTNERSHIP_AUTH_STACK — onboard pushes it to HOST2.
host2.conf: add Gitea to FALLBACK_HOST2_COVERS_HOST1_TIER1 — starts immediately
when HOST1 goes down, making the source of truth reachable independently of
HOST1's auth stack.
host1.conf: HOST1_RAMDISK_SIZE 8G→10G, WARN_GB 6.8→8.5, LOW_GB 5.5→7.
ramdisk_stop.sh: new Tools/ script — redirects symlink to SSD before unmount
so Emby keeps writing during maintenance, auto-falls back to lazy umount
when only directory handles are open (no active writes).
common.sh: fix `info` call at line 595 (undefined) → `log`.
Tools docs: added ramdisk_stop.sh to README table, categories, diagram, and Manual.
- playback_aware_radarr_discovery.sh: new two-stage movie discovery
using recently watched Emby movies as seeds and TMDB recommendations
as the similarity engine; scores on recency + rating + vote count;
adds HOST*_TMDB_API_KEY alias to detect_hosts() in common.sh
- emby_to_sonarr_sync.sh: restrict to SONARR_EMBY_LIBRARIES allowlist
(Anime, Kids Shows, Stand-Up Comedy, TV shows) — excludes Youtube
and Recorded Sports libraries; add garbage title filter for multi-
season folder names (S01-S03) and bare bracket tags ([Prof])
- emby_to_radarr_sync.sh: add _is_episode_title filter to block anime
episode files stored in Movies library (underscore/dot patterns,
fansub bracket format, episode markers, codec metadata strings)
- emby_to_lidarr_sync.sh: add U+FFFD Unicode replacement character
filter alongside the existing ASCII ? filter for encoding corruption
- master.conf: RADARR_DISCOVERY_* settings, SONARR/RADARR_EMBY_LIBRARIES
allowlists, Radarr discovery added to WEEKLY_MAINTENANCE_SCRIPTS
- downloaders_reset: connection check block before slskd API sections;
triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
common.sh, master confs, and new Manual/README docs
Consistent naming with the watchdog family (system_watchdog, docker_watchdog).
File renamed, all RM_ config variables and internal references updated to RW_
across master.conf, master_host1.conf, common.sh, docker_watchdog.sh,
system_watchdog.sh, watchdog_orchestrator.sh, and sunday_morning_coffee_report.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduce a four-layer self-healing stack replacing the continuous-loop watchdogs:
- resource_manager.sh (new): single-pass pressure reduction layer; throttles
SABnzbd/qBit at level 1, docker-pauses background containers at level 2,
docker-stops optional containers and signals docker_watchdog to defer at
level 3; graduated recovery with hysteresis
- watchdog_orchestrator.sh (new, Orchestrators/): runs resource_manager →
docker_watchdog → system_watchdog in sequence; intended for per-minute cron
via User Scripts; startup grace, acquire_lock to prevent pile-up, heartbeat
- docker_watchdog.sh: de-looped to single-pass; daemon strikes persisted to
state file across runs; cross-script coordination reads RM_STATE_FILE instead
of SYS_WATCHDOG_STATE_FILE
- system_watchdog.sh: de-looped to single-pass; stripped of all container
management (shutdown_non_essential_containers removed); reboot-only last resort
- master.conf: removed system_watchdog and docker_watchdog from
ARRAY_START_SCRIPTS; added WATCHDOG ORCHESTRATOR and RESOURCE MANAGER sections
- master_host1.conf: added RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS arrays
- common.sh: aliased RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS via detect_hosts()
- continuous_scripts_status.sh: moved to Tools/ (preserved for future use)
- sunday_morning_coffee_report.sh: watchdog section updated to use state file
mtime checks instead of is_running; added Resource Manager subsection;
fixed mem_shutdown grep filter pointing to wrong state file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)
Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
dry runs)
master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)
master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS
Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Arr sync (new):
- Media/arr_sync.sh — full mesh bidirectional sync across all HOST* nodes
- Lidarr (MusicBrainz), Sonarr (TVDB), Radarr (TMDB) all handled in one script
- Remote API keys read live from config.xml via SSH — never stored in conf files
- Shared blocklist (DATA_DIR/arr_sync_blocklist.tsv) merged from all nodes at runtime
- Graceful skip if arr not configured locally or not reachable on a remote node
- --blocklist-add / --blocklist-remove / --blocklist-list management flags
- daily_sync_maintenance.sh — arr sync runs as explicit phase before rsync
- partnership_onboard.sh — Step 3 bootstraps merged library on both sides at onboard
- master.conf — ARR_SYNC_* config block, DOCKER_APPDATA_BASE
Rsync / cleanup:
- DEFAULT_RSYNC_OPTS — removed --delete; arr_cleanup.sh owns orphan enforcement
- lidarr_cleanup.sh — removed HOST1-only guard; runs on any node with Lidarr configured
Config architecture:
- HOST1/HOST2 hostnames moved from master_host*.conf → master.conf (not credentials)
- Sparse checkout now works correctly: each server only needs its own host conf
- detect_hosts() still resolves MY_ID + REMOTE_ID via master.conf hostname values
Bug fix:
- common.sh line 493 — watchdog toggle eval had broken quoting; all SYS_WATCHDOG_CHECK_*
globals were silently set to empty instead of their configured values
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
${!REMOTE_ID:-unknown} with REMOTE_ID="" is a fatal bash error that terminates
the entire function — not just the line. All subsequent aliases (EMBY_CONTAINER,
TRANSCODE_SSD, RAMDISK_LOW_GB, SSH_KEY) were never set, causing awk syntax errors
and "SSD fallback path missing" warnings in every transcode script invocation.
Guard with an explicit empty check instead of relying on the :- fallback,
which bash does not apply when the indirect expansion itself is invalid.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FailedProxy