The lookup grepped for a quoted container name that appears in no Unraid template, so every offboard reported four WebUIs it could not reconfigure. The final sync counted a share synced whenever Tier 1 was on, ignoring rsync.sh's status entirely.
The scp reused the local absolute path on the remote, so an appdata-mode mirror never received it, and nothing wrote the mirror's own state file at all — a fully onboarded mirror rendered as having no partnership.
grep -v > tmp && mv keeps the temp file's umask mode, so an offboard left authorized_keys 0666 and sshd StrictModes silently refused every key in it — including the one the next onboard installs.
Codebase-wide audit pass: fixed real bugs (SSH hangs missing BatchMode,
local-outside-function no-ops, variable name collisions, a truncated
ratio calc, wrong state-dir path, DARK vs NO_INTERNET drift, and more),
then pulled logic that was duplicated across multiple scripts — arr
cleanup safety gates, docker restart ordering, container maintenance
stop/restart, watchdog state-file helpers, partnership role resolution,
cert expiry checks, remote node discovery, and TMDB discovery scoring —
into common.sh so each now has a single implementation.
Same audit as the orchestrator standardization pass (2a062e5), extended to
every remaining script. Found the same class of bug independently recurring:
ramdisk_stop.sh checked $LOG (nothing assigns it, should be $ENABLE_LOGGING),
partnership_onboard.sh checked $LOG_MODE (same issue), emby_session_report.sh
checked $TRANSCODE_PCT which was never computed so the high-transcode alert
could never fire, and storage_migrate.sh never called detect_hosts() so
$MY_ID was empty, silently breaking the post-migration host*.conf update.
partnership_manager.sh used `local` at top-level script scope (invalid outside
a function) and had two master.conf path references missing "Configurations/".
Along the way: several scripts (share_setup.sh, conf_sync.sh,
downloaders_reset.sh, transcode_cleanup.sh, transcode_manager.sh,
remote_arr_cache_writer.sh, upgrade_webhook_handler.sh) had no explicit
trailing exit code, so they always reported success regardless of real
failures. play_state_sync.sh was missing the partnership gate its own header
documented, so remote play-state sync ran even with PARTNERSHIP_ENABLED=false;
it also always exited 0 on sync errors. arr_profile_enforcer.sh and
webhook_setup.sh hand-rolled their own flag parsing instead of common.sh's
parse_args, so --log silently did nothing on either.
system_watchdog.sh was itself an un-standardized mini-orchestrator — converted
to the shared run_orch_child()/JOB_PASS/JOB_FAIL pattern, added the missing
failure notification, and fixed dry-run to pass --dry-run down to children
instead of skipping them outright. Also fixed a stale webgui_watchdog.sh path
in master.conf.template that would break system_watchdog.sh on any fresh
install.
Closed a sibling-drift gap: radarr_cleanup.sh and sonarr_cleanup.sh were
missing lidarr_cleanup.sh's tracked-count percentage-drop safety gate and its
"not configured on this host, skip cleanly" guard — both now match Lidarr's
7-gate model.
Unraid API returns fsUsed=0 when a disk's filesystem is unmounted (spun
down). disks.ini keeps the last-known value in KB even after spindown —
use it as fallback when isSpinning=false and fsUsed=0. Also remove the
mounted-only guard in the ini fallback path (vv_disk_entry) for the same
reason. Stale comments in user_script_plug-in.sh and partnership_manager.sh
also cleaned up.
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.
PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.
conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.
fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.
README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
All FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER* references updated to
FALLBACK_${REMOTE_ID}_TIER* across fallback_test.sh, partnership_manager.sh,
docker_update.sh, mesh_monitor.sh, and monitor.php. mesh_monitor.sh drops
the inner covering-host loop — tier data now lives in the covered host's own
conf so no cross-host scan is needed. monitor.php reads from the covered
host's conf file rather than the local host's.
stability_watchdog.sh mover check now uses platform_is_mover_running()
instead of raw pgrep, matching server_reboot.sh.
check_unraid_version_parity() renamed to check_os_version_parity() —
the function internals were already platform-neutral after the adapter
refactor; the name was the last Unraid artifact.
OS version reads now go through platform_get_os_version() and
platform_os_version_probe_cmd() instead of grepping /etc/unraid-version directly.
STATE_DIR fallbacks to /boot/config removed — STATE_DIR is always set by
load_config.sh and the fallback encoded a platform-specific path.
Setup DB path references use platform_setup_db_path() instead of the
VARAVERK_SETUP_FILE/-/boot/config compound fallback.
DOCKER_APPDATA_BASE default removed from arr_sync.sh — the adapter sets it.
detect_hosts() now sets REMOTE_STORAGE_PATH from HOST*_STORAGE_PATH in the
remote's conf instead of calling platform_storage_path() locally. SSH commands
that check the remote's storage (check_remote_array, check_remote_disks,
check_both_healthy) use $REMOTE_STORAGE_PATH so they work correctly when the
remote is a different platform with a different storage root.
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.
New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.
Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
- Auth stack: fold cert monitor into Auth Stack page as fourth tab (Certs);
remove standalone cert page and top-level tab
- cert_monitor.sh: write JSON status cache to State_Files/cert_status.json
after each run; expose per-domain days/expiry via _CERT_DAYS/_CERT_EXPIRY globals
- api/cert.php: new — serves cached cert status; falls back to configured
domains as UNKN when no cache exists; POST action=run triggers live check
- arrs db fallbacks: vv_arr_cleanup_stats/discovery_stats/recovery_stats now
read from data/*.db files when log JSON files don't yet exist
- config.php vv_conf_vars(): unescape bash \$ → $ so passwords with dollar
signs read correctly from conf files
- host1.conf: fill in HOST1_NPM_USER/PASS and HOST1_LLDAP_USER/PASS
- Partnership adapter pattern: Unraid-specific container logic extracted to
Plugin/unraid/Partnership/; platform-agnostic structure stays in Partnership/
- First-run wizard: uniform multi-step flow for all hosts; HOST2 pull moved
to checklist; auto SSH keygen and API key creation on save
- api/checklist.php: live setup checklist with pull_master action
- Fullscreen toggle: hide Unraid header/menu; state persists via localStorage
- Move SCRIPTS_DIR/DATA_DIR/STATE_DIR from appdata to /boot/config/plugins/varaverk
- All state files now in STATE_DIR (no more /tmp or /boot/config root writes)
- Bootstrap: Gitea-first clone with GitHub fallback, no array dependency
- varaverk.cfg seeded with Gitea connection settings
- .gitignore: add State_Files/, varaverk.cfg, varaverk-*.txz
- Partnership/transcode/fallback scripts use STATE_DIR variables
- PHP config.php: DATA_DIR/STATE_DIR constants, VV_SETUP_STATE_FILE dynamic
- deploy.sh PROD_ROOT updated to plugin flash dir
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
flag_key was declared with 'local' at script top level, which is only
valid inside bash functions. This left flag_key empty, writing '=true'
(malformed) to setup.db instead of 'HOST1_LOCAL_DONE=true'.
Changed to a plain variable assignment.
When HOST1 completes its local setup (--onboard --local-only), enable the
partnership in master.conf so sync scripts start functioning immediately.
Pushes the updated master.conf to all listed hosts via vv_push_master_conf().
partnership_manager.sh:
- --local-only flag for --onboard: skips remote pre-flight, WebUI reconfig,
state push; runs FolderView3 folder creation + writes HOST1_LOCAL_DONE=true
to setup.db + pushes it; exits without writing ACTIVE (partnership not yet
established until Phase 2)
partnership_onboard.sh:
- Phase 1 exit now runs partnership_manager.sh --onboard --local-only after
SSH + conf push, so HOST1 is fully configured on its own side before Phase 2
include/partnership.php:
- vv_pt_nodes() reads HOST1_LOCAL_DONE from setup.db; exposes as local_done
on the self node (null for remote nodes)
pages/partnership.php:
- Self node card shows green "Local setup complete · waiting for partner"
badge when local_done=true
- Actions: "Complete HOST1 Setup" button appears when HOST1 hasn't run local
setup yet but HOST2 is already at phase 1 — lets user trigger it manually
- vvPtLocalSetup() top-level function calls partnership_manager.sh via run.php
Follows the same extraction pattern as partnership_offboard.sh.
partnership_manager.sh --transfer now exec-forwards to the new script.
Legacy inline block preserved under `if false` for reference.
Fixes in partnership_transfer.sh vs the original inline code:
- WebUI reconfiguration corrected: SSH to new owner (mirror) to set localhost;
reconfigure_local_webuis() on current server to point to new owner IP
- master.conf remote update uses scp push instead of self-SSH with wrong path
- SSH key uses $SSH_KEY directly (was using undefined {HOST*}_SSH_KEY expansion)
partnership_manager.sh:
- fix auto-offboard threshold: × 96 → × 48 (was triggering at 2× configured days)
- fix do_final_sync(): use CRITICAL_SYNC_SHARES if configured; fall back to hardcoded paths
- replace inline --check offboard blocks with background partnership_offboard.sh calls:
• owner path was missing remote container cleanup (steps 8+9) and blocklist
• mirror path was missing blocklist, SSH revocation, CRITICAL_RSYNC_ENABLED=false
• local grace_seconds outside a function (bash error) eliminated with the block removal
• 6hr Tailscale grace sleep was blocking the cron for the full grace window
partnership_offboard.sh:
- reorder owner path: state write moved from step 5 to step 10 (last) so re-running
after a mid-offboard crash restarts from scratch instead of exiting as INACTIVE;
state is now written before the Tailscale grace sleep so --check stops re-triggering
- add FolderView3 remote folder cleanup in step 7 (mirror's fallback folder persisted)
- track step 1+2 outcome with STEP_STOP_OK / STEP_SYNC_OK; summary now reflects actual
result instead of hardcoding ✅ regardless of dry-run or failure
- update header step list to match new step order
appdata-Failover → appdata-Fallback:
All .md files updated — scripts and confs already used appdata-Fallback.
Share was renamed; docs were the only stale references.
critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
Docker_Essentials README/downloaders_reset.sh, Fallback README,
weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.
rsync_emby_failover User Script superseded:
Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
Section in User Script Plug-in README marked for deletion from plugin.
transcode_management.sh: */3 → */7:
transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
Transcodes README/manual, Orchestrators README, User Script Plug-in README,
user_script_plug-in.sh all updated.
22 scripts used docker without verifying it's present. Consistent with the
pattern already established in Docker_Essentials scripts. Added after acquire_lock
and before detect_hosts in each script's setup section.
- downloaders_reset: connection check block before slskd API sections;
triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
common.sh, master confs, and new Manual/README docs
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)
Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
dry runs)
master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)
master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS
Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On offboard, neither server should retain SSH access to the other.
Keys are now revoked as the final step before Tailscale removal — after
all state pushes and syncs are complete so no SSH operation is cut short.
Revocation runs in both owner-initiated offboard and when --check
finalises a mirror-requested offboard. Both directions are handled:
- Our pubkey removed from remote's authorized_keys via SSH (identified
by key comment keyname@hostname set by ssh_setup.sh at keygen time)
- Remote's pubkey removed from local authorized_keys (matched by
@REMOTE_SERVER_NAME suffix — unique across the authorized_keys file)
Non-fatal: SSH failure logs a manual revocation instruction; local
cleanup still runs. Summary shows per-direction revocation status.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After offboard, the former partner's hostname is written to
/boot/config/partnership_blocklist.db. This blocks passive reconnection:
- --check skips the remote entirely (no auto-reconnect noise)
- rsync.sh refuses to sync with a blocklisted host
- --onboard warns about the previous offboard but proceeds,
and clears the block on success (onboard = deliberate intent)
New --unblock <hostname> mode removes an entry to permit re-onboarding
without running a full --onboard first. --status shows the full blocklist.
Blocklist file survives reboots (/boot/config) and Tailscale reconnections —
application-layer guard that complements Tailscale device removal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gmer4Lfe
FailedProxy