Commit Graph
31 Commits
Author SHA1 Message Date
Gmer4Lfe c4d4d8160d Replace Emby dirty-sync workaround with play_state_sync in handback
Emby now runs continuously on both hosts — play_state_sync via API
handles watch state reconciliation, making the 30-min dirty rsync
redundant. Handback runs play_state_sync --wait before DNS cutover
so users land on current state after a failover event.

- Emby removed from FALLBACK_HOST1_TIER1 (always running)
- Emby dirty sync removed from HOST1_CRITICAL_SYNC_SHARES
- Emby writeback removed from FALLBACK_HOST1_WRITEBACK_TIER1
- play_state_sync gains --wait flag (uses lock wait mode for handback)
- Tier 1 emby-fallback special case removed from handback writeback loop
- play_state_sync --wait added as Step 7 in handback, before DNS cutover
2026-06-22 23:14:36 -04:00
Gmer4Lfe bf3e7cc2c4 Storage-mode awareness pass + doc update for System_Essentials through Partnership
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.

PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.

conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.

fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.

README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
2026-06-19 19:32:39 -04:00
Gmer4Lfe 0564580605 Make PARTNERSHIP_ENABLED the authoritative gate for all cross-server operations
Adds require_partnership() to common.sh — exits cleanly when PARTNERSHIP_ENABLED=false.
Removes FALLBACK_PARTNERSHIP_REQUIRED toggle — partnership is now always required,
not optional. Cross-server scripts (rsync, conf sync, fallback, arr sync, play state,
backup verify) all call require_partnership after detect_hosts.
2026-06-19 18:27:41 -04:00
Gmer4Lfe 28d434b76c Propagate new fallback model to all remaining script references
All FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER* references updated to
FALLBACK_${REMOTE_ID}_TIER* across fallback_test.sh, partnership_manager.sh,
docker_update.sh, mesh_monitor.sh, and monitor.php. mesh_monitor.sh drops
the inner covering-host loop — tier data now lives in the covered host's own
conf so no cross-host scan is needed. monitor.php reads from the covered
host's conf file rather than the local host's.
2026-06-19 17:49:26 -04:00
Gmer4Lfe b27b2d62f5 Flip fallback coverage model: each host defines its own recovery profile
Previously HOST1 defined what it would run for HOST2 (FALLBACK_HOST1_COVERS_HOST2_TIER*).
Now each host defines what it wants run when IT goes down (FALLBACK_HOST1_TIER*), and the
covering server reads the down host's conf via the RAM/persistent cache.

get_tier_containers() reads FALLBACK_${REMOTE_ID}_TIER* instead of
FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*. Tier data migrated to the correct host confs.
Writeback paths and delays were already REMOTE_ID-based — no change needed there.
2026-06-19 17:44:53 -04:00
Gmer4Lfe 3964f6fb46 audit echo vs log across all scripts — outcomes always visible, verbose for per-item loops 2026-06-14 12:40:15 -04:00
Gmer4Lfe b4d6866f40 Polish before first GitHub push: consistent adapter usage and naming
stability_watchdog.sh mover check now uses platform_is_mover_running()
instead of raw pgrep, matching server_reboot.sh.

check_unraid_version_parity() renamed to check_os_version_parity() —
the function internals were already platform-neutral after the adapter
refactor; the name was the last Unraid artifact.
2026-06-14 01:30:16 -04:00
Gmer4Lfe ee5a07be1c Remove remaining OS-specific hardcodes from core scripts
OS version reads now go through platform_get_os_version() and
platform_os_version_probe_cmd() instead of grepping /etc/unraid-version directly.

STATE_DIR fallbacks to /boot/config removed — STATE_DIR is always set by
load_config.sh and the fallback encoded a platform-specific path.

Setup DB path references use platform_setup_db_path() instead of the
VARAVERK_SETUP_FILE/-/boot/config compound fallback.

DOCKER_APPDATA_BASE default removed from arr_sync.sh — the adapter sets it.
2026-06-14 01:22:02 -04:00
Gmer4Lfe 27bfc21cb0 Platform-agnostic refactor: eliminate OS-specific hardcodes from core scripts
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.

New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.

Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
2026-06-14 00:59:19 -04:00
Gmer4Lfe f10f09eeb1 Bug fixes, arr cleanup API-driven scan, shared JS formatters
Bug fixes:
- fallback.sh: escape sed metacharacters (\ & |) in state_set values
- common.sh: parse PID from lock file content correctly (handles pid:metadata format)
- unraid_api_key_renew.sh: fix path depth (../../../) and sync registry key to conf when stale
- stop.php: only clear pid/status if process is actually dead — D-state survives SIGKILL
- array_started.sh: check ARRAY_START_SCRIPTS empty before printing launch header

Arr cleanup:
- radarr_cleanup.sh / sonarr_cleanup.sh: fetch root folders from arr API instead of
  reverse-looking up the path map — handles multi-root-folder setups correctly

UI:
- varaverk.js: extract shared formatters (_relTime, _fmtBytes, _sz, _uptime, _gb, _tb, _n)
- arrs.php / partnership.php: use shared formatters, remove duplicates
- arrs.php / fallback.php: show error message on fetch failure instead of silent empty
- docker.php: disable rename input during request, restore original value on failure
- setup.php: abort controller timeout on detect fetch
- partnership.php: remove Re-run Phase 2 button opacity dimming
2026-06-12 21:41:11 -04:00
Gmer4Lfe 127b070f7b Verbose logging: add config dump log() calls to Monitors, Media, Tools, Rsync, Fallback 2026-06-04 21:37:53 -04:00
Gmer4Lfe 369a9e6c19 Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites
- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename)
- Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API
  for storage health, service management, mover, user scripts, notifications,
  disk temps, and platform command validation
- Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR,
  auto-source Plugin/$PLATFORM/adapter.sh after common.sh
- Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg,
  disks.ini, and validate_unraid_cmd calls with platform_*() functions across
  watchdogs, orchestrators, and System_Essentials scripts
- Update all documentation: rename refs, update webgui escalation logic,
  add platform adapter section to Plugin README, update main README with
  portability vision and corrected self-healing stack description
2026-06-04 18:14:34 -04:00
Gmer4Lfe f8325a7067 rename: fix remaining failover refs in fallback.sh, fallback_test.sh, array_started.sh, user_script_plug-in.sh 2026-06-01 20:05:02 -04:00
Gmer4Lfe cb7d7a688d rename: failover → fallback throughout codebase — this system is Fallback, state is FALLBACK; preserve conventional failover comparisons 2026-06-01 20:04:17 -04:00
Gmer4Lfe 65f6bd7b8a fix: fallback.sh — last User Scripts Abort ref → fallback.sh --stop 2026-06-01 20:00:22 -04:00
Gmer4Lfe e0ae80d2ea fix: remove all User Scripts dependencies — Varaverk plugin handles everything via event hooks, no User Scripts needed 2026-06-01 19:57:21 -04:00
Gmer4Lfe 98d2e6efa4 fix: remaining stale refs — boot drive path in Plugin Manual, array_started.sh typos in fallback and ramdisk_setup 2026-06-01 19:51:40 -04:00
Gmer4Lfe bc81e5a9e0 fix: fallback.sh — array_start.sh → array_started.sh typo 2026-06-01 19:41:55 -04:00
Gmer4LfeandClaude Sonnet 4.6 fb0530deba Consolidate all paths to plugin flash dir, fix watchdog 7.3 triggers
- Move SCRIPTS_DIR/DATA_DIR/STATE_DIR from appdata to /boot/config/plugins/varaverk
- All state files now in STATE_DIR (no more /tmp or /boot/config root writes)
- Bootstrap: Gitea-first clone with GitHub fallback, no array dependency
- varaverk.cfg seeded with Gitea connection settings
- .gitignore: add State_Files/, varaverk.cfg, varaverk-*.txz
- Partnership/transcode/fallback scripts use STATE_DIR variables
- PHP config.php: DATA_DIR/STATE_DIR constants, VV_SETUP_STATE_FILE dynamic
- deploy.sh PROD_ROOT updated to plugin flash dir

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 13:30:20 -04:00
Gmer4Lfe 0fe99cf2a9 rename: appdata/varaverk → appdata/Varaverk (capital V throughout) 2026-05-30 10:22:51 -04:00
Gmer4Lfe ee086f309e rename appdata-Failover→appdata-Fallback in docs; critical sync 15→30min; transcode 3→7min
appdata-Failover → appdata-Fallback:
  All .md files updated — scripts and confs already used appdata-Fallback.
  Share was renamed; docs were the only stale references.

critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
  critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
  Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
  Docker_Essentials README/downloaders_reset.sh, Fallback README,
  weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.

rsync_emby_failover User Script superseded:
  Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
  critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
  Section in User Script Plug-in README marked for deletion from plugin.

transcode_management.sh: */3 → */7:
  transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
  Transcodes README/manual, Orchestrators README, User Script Plug-in README,
  user_script_plug-in.sh all updated.
2026-05-22 22:22:21 -04:00
Gmer4Lfe 070013cb4f fix widespread naming errors across all docs — array_started, fallback, FALLBACK_
array_start.sh → array_started.sh (bulk rename — actual file has always been
array_started.sh; docs consistently used the wrong name across 9 files)

failover.sh → fallback.sh (bulk rename — script is Fallback/fallback.sh;
docs used the old name throughout)

FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1 → FALLBACK_HOST2_COVERS_HOST1_TIER1
(README.md — wrong variable name, actual vars use FALLBACK_ prefix and
COVERS_ not RUNS_FOR_)

docker_watchdog.sh attribution: "started by array_started.sh" → "called every
minute by watchdog_orchestrator.sh" (docker_watchdog is not in ARRAY_START_SCRIPTS)

server_reboot.sh shutdown sequence: step 7 "rc.docker stop" → calls
array_stopping.sh (user_scripts, fallback, rsync, mover, docker_container_stop)

Rsync manual ARRAY_START_SCRIPTS list: removed stability_watchdog and
docker_watchdog (now in watchdog_orchestrator); failover → fallback
2026-05-22 21:53:32 -04:00
Gmer4Lfe 8d8a4fd637 update stale media ownership language across all docs
arr_sync.sh bidirectional union replaced the old split-ownership model.
Old: HOST1 owns Movies/TV/Music, HOST2 owns Anime — each arr manages
different shares to avoid conflicts.
New: arr_sync.sh keeps all arr databases in union. Either server can
download to any share at any time. The union is the source of truth.

Auth stack references left intact — HOST1 still owns auth config
(one-way sync to HOST2). Only media library ownership language updated.

Files changed:
  README.md                          — server block + share annotations
  Fallback/README-Fallback.md        — role line + Split Source of Truth section
  Orchestrators/README-Orchestrators — daily sync share comments
  Rsync/Manual-Rsync.md              — DEFAULT_RSYNC_OPTS comment block
  Media/README-Media.md              — HOST AWARENESS section
2026-05-22 21:35:45 -04:00
Gmer4Lfe 95151c2278 Watchdogs/ folder + host conf rename
Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
2026-05-22 17:08:36 -04:00
Gmer4Lfe c1c12cbe73 Fallback echo/log audit pass — fallback.sh, fallback_test.sh + Manual
Remove setup banners from fallback.sh and fallback_test.sh. Add Host Detection
to fallback.sh OPERATIONAL SAFEGUARDS; add Lock Acquisition + Host Detection to
fallback_test.sh. Add output tiers section to Manual-Fallback.md documenting
daemon vs one-shot output behaviour.
2026-05-21 16:33:36 -04:00
Gmer4Lfe e932936acc Add docker availability check to all scripts that use docker commands
22 scripts used docker without verifying it's present. Consistent with the
pattern already established in Docker_Essentials scripts. Added after acquire_lock
and before detect_hosts in each script's setup section.
2026-05-20 18:35:34 -04:00
Gmer4Lfe e13f2fa14f feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections;
  triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
  gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
  Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
  common.sh, master confs, and new Manual/README docs
2026-05-19 20:00:10 -04:00
Gmer4Lfe aa6d165725 feat: add fallback.sh --stop for clean array shutdown
Fallback runs as a bare subprocess (cmdline: bash .../Fallback/fallback.sh),
not via /tmp/user.scripts, so user_scripts_stop.sh cannot find it.
Adding --stop mode closes that gap in the array shutdown sequence.

fallback.sh --stop: reads lock file (/tmp/unraid_locks/fallback.lock) to
get the running PID, sends SIGTERM (hits the existing trap that sets
FALLBACK_RUNNING=false), waits up to 10s, SIGKILLs if still alive.

master.conf: insert Fallback/fallback.sh --stop as step 2 in
ARRAY_STOP_SCRIPTS (after user_scripts_stop, before rsync_stop).

user_script_plug-in.sh: update ARRAY STOPPING block to show 5-step order
and add v2.5 changelog entry.
2026-05-18 22:04:07 -04:00
Gmer4LfeandClaude Sonnet 4.6 0ae31b5fa6 feat: Tailscale resolution hardening, partnership offboard completion, Emby provisioning
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
  parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
  is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
  PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)

Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
  own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
  problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
  if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
  matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
  or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
  for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
  cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
  dry runs)

master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)

master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS

Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 20:09:13 -04:00
Gmer4LfeandClaude Sonnet 4.6 b65f572367 feat: arr-native distributed media sync + config architecture fixes
Arr sync (new):
- Media/arr_sync.sh — full mesh bidirectional sync across all HOST* nodes
  - Lidarr (MusicBrainz), Sonarr (TVDB), Radarr (TMDB) all handled in one script
  - Remote API keys read live from config.xml via SSH — never stored in conf files
  - Shared blocklist (DATA_DIR/arr_sync_blocklist.tsv) merged from all nodes at runtime
  - Graceful skip if arr not configured locally or not reachable on a remote node
  - --blocklist-add / --blocklist-remove / --blocklist-list management flags
- daily_sync_maintenance.sh — arr sync runs as explicit phase before rsync
- partnership_onboard.sh — Step 3 bootstraps merged library on both sides at onboard
- master.conf — ARR_SYNC_* config block, DOCKER_APPDATA_BASE

Rsync / cleanup:
- DEFAULT_RSYNC_OPTS — removed --delete; arr_cleanup.sh owns orphan enforcement
- lidarr_cleanup.sh — removed HOST1-only guard; runs on any node with Lidarr configured

Config architecture:
- HOST1/HOST2 hostnames moved from master_host*.conf → master.conf (not credentials)
- Sparse checkout now works correctly: each server only needs its own host conf
- detect_hosts() still resolves MY_ID + REMOTE_ID via master.conf hostname values

Bug fix:
- common.sh line 493 — watchdog toggle eval had broken quoting; all SYS_WATCHDOG_CHECK_*
  globals were silently set to empty instead of their configured values

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 09:53:40 -04:00
Gmer4LfeandClaude Sonnet 4.6 009820e981 refactor: rename failover/HA → fallback across entire codebase
Removes all references to "failover" and "HA" (high availability)
terminology from variable names, config keys, state values, rsync
profile names, directory paths, and user-visible strings.

Mapping:
  FAILOVER_*              → FALLBACK_*
  FAILOVER_HOST*_RUNS_FOR → FALLBACK_HOST*_COVERS
  critical-failover       → critical-fallback
  emby-failover           → emby-fallback
  appdata-Failover/       → appdata-Fallback/
  "FAILOVER" state value  → "FALLBACK"
  failover_start key      → fallback_start
  Failover/ directory     → Fallback/
  failover.sh             → fallback.sh
  failover_state.db       → fallback_state.db
  -Failover folder suffix → -Fallback

State machine: NORMAL | FALLBACK | DARK (unchanged)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 19:28:21 -04:00