Commit Graph
44 Commits
Author SHA1 Message Date
Gmer4Lfe 260f0a61ce Wire up unused config vars found in follow-up audit pass
TRANSCODE_LOG_RETENTION was documented as trimming the daily transcode log
but never referenced — log grew unbounded. TRANSCODE_ORPHAN_AGE was shown in
--status but never used, so empty session folders were deleted immediately
instead of after the documented grace period, risking a race against ffmpeg
creating a folder just before writing its first segment.

docker_watchdog.sh's daemon-health thresholds were only hardcoded fallbacks
despite comments claiming they were master.conf-configurable, and it
referenced a heartbeat feature that was never implemented (that's owned by
watchdog_orchestrator.sh, its caller) — added the three thresholds to
master.conf for real and removed the stale heartbeat claim.

Also added the missing HOSTN_PARTNERSHIP_SERVICES_STACK block to
host.conf.template (containers.sh already read it via detect_hosts, just
never had a template entry) and corrected play_state_sync.sh's doc comment
for PLAY_SYNC_TYPES' actual default.
2026-07-03 17:45:20 -04:00
Gmer4Lfe 69189bbf18 Fix dead-variable and exit-code bugs found in codebase-wide audit
Same audit as the orchestrator standardization pass (2a062e5), extended to
every remaining script. Found the same class of bug independently recurring:
ramdisk_stop.sh checked $LOG (nothing assigns it, should be $ENABLE_LOGGING),
partnership_onboard.sh checked $LOG_MODE (same issue), emby_session_report.sh
checked $TRANSCODE_PCT which was never computed so the high-transcode alert
could never fire, and storage_migrate.sh never called detect_hosts() so
$MY_ID was empty, silently breaking the post-migration host*.conf update.
partnership_manager.sh used `local` at top-level script scope (invalid outside
a function) and had two master.conf path references missing "Configurations/".

Along the way: several scripts (share_setup.sh, conf_sync.sh,
downloaders_reset.sh, transcode_cleanup.sh, transcode_manager.sh,
remote_arr_cache_writer.sh, upgrade_webhook_handler.sh) had no explicit
trailing exit code, so they always reported success regardless of real
failures. play_state_sync.sh was missing the partnership gate its own header
documented, so remote play-state sync ran even with PARTNERSHIP_ENABLED=false;
it also always exited 0 on sync errors. arr_profile_enforcer.sh and
webhook_setup.sh hand-rolled their own flag parsing instead of common.sh's
parse_args, so --log silently did nothing on either.

system_watchdog.sh was itself an un-standardized mini-orchestrator — converted
to the shared run_orch_child()/JOB_PASS/JOB_FAIL pattern, added the missing
failure notification, and fixed dry-run to pass --dry-run down to children
instead of skipping them outright. Also fixed a stale webgui_watchdog.sh path
in master.conf.template that would break system_watchdog.sh on any fresh
install.

Closed a sibling-drift gap: radarr_cleanup.sh and sonarr_cleanup.sh were
missing lidarr_cleanup.sh's tracked-count percentage-drop safety gate and its
"not configured on this host, skip cleanly" guard — both now match Lidarr's
7-gate model.
2026-07-03 17:35:30 -04:00
Gmer4Lfe 3964f6fb46 audit echo vs log across all scripts — outcomes always visible, verbose for per-item loops 2026-06-14 12:40:15 -04:00
Gmer4Lfe df18657183 Fix transcode session detection for Live TV and Direct Stream
HLS segments are written atomically so lsof never sees them as open.
Cleanup now shows "Active" count (too-young files) which correctly
reflects Live TV / Direct Stream segments. Manager session count now
counts flat file prefixes in addition to subdirs.
2026-06-14 11:24:21 -04:00
Gmer4Lfe ba3eed39e3 Sync READMEs and manuals to current script headers
Fix discrepancies found by auditing docs against headers:
- stability_watchdog.sh: wrong variable name (STRIKES → STRIKE_LIMIT) and wrong default (2hr → 12hr) in header
- Watchdogs manual: REBOOT_WINDOW_HRS example value was 2, should be 12
- Transcodes README: transcode_cleanup/manager table said "Every 3 min", should be "Every 7 minutes"
- Tools manual: fallback_state_reset.sh still marked "not yet built" — rewrite section to reflect current script
- Tools manual: claude_startup --setup flag doesn't exist; modes were inverted — fix and add --launch
- Tools README + manual: add docker_prune_images.sh (existed but undocumented)
- Tools manual: add play_state_sync.sh --full to flag reference
- Orchestrators README: arrs_failed_stalled_recovery runs via intermediate_sync (every 4hr), not standalone every 6hr
- Plugin README: add build.sh to scripts table
- Partnership README + manual: add partnership_transfer.sh and onboard_cancel.sh to script tables and flag reference
2026-06-14 02:01:27 -04:00
Gmer4Lfe 181114aed5 Header/code audit fixes across all core scripts
Behavior fixes:
- docker_daily_restart.sh: implement RESTART_VERIFY_WAIT sleep before
  verify_running() — variable was logged as config but never applied
- cert_monitor.sh: --log now shows expiry date and days remaining for
  healthy domains in the per-domain summary; problems always show details

Header corrections:
- downloaders_reset.sh: fix frequency contradiction (15 min → 30 min)
- lidarr_cleanup.sh: Emby scan triggers on actual deletions, not every run
- ssh_setup.sh: document --local-only flag in USAGE section
- zfs_memory_snapshot.sh: clarify dry-run skips log write in PURPOSE
- ramdisk_stop.sh / ramdisk_setup.sh: document TRANSCODE_STATE_FILE var
- smart_long_test.sh: remove SMART_TEMP_WARN/CRIT — not used in code
2026-06-14 01:40:35 -04:00
Gmer4Lfe 27bfc21cb0 Platform-agnostic refactor: eliminate OS-specific hardcodes from core scripts
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.

New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.

Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
2026-06-14 00:59:19 -04:00
Gmer4Lfe 369a9e6c19 Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites
- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename)
- Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API
  for storage health, service management, mover, user scripts, notifications,
  disk temps, and platform command validation
- Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR,
  auto-source Plugin/$PLATFORM/adapter.sh after common.sh
- Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg,
  disks.ini, and validate_unraid_cmd calls with platform_*() functions across
  watchdogs, orchestrators, and System_Essentials scripts
- Update all documentation: rename refs, update webgui escalation logic,
  add platform adapter section to Plugin README, update main README with
  portability vision and corrected self-healing stack description
2026-06-04 18:14:34 -04:00
Gmer4LfeandClaude Sonnet 4.6 bc70ebe5ee Add verbose log() coverage across Docker_Essentials, unRAID_Essentials, Transcodes, and Orchestrators
- Config/threshold dumps at startup in every script (retry counts, timeouts, sizes, thresholds)
- Per-item detail in verbose: container images, timing per container/share/job, image ID diffs
- Orchestrators: watchdog cycle now logs array state, grace state, per-script timing; transcode_management shows ramdisk state before each cycle; critical_sync logs share list and maintenance scripts; coffee report logs server state at run time
- Summary counts replaced with names in verbose where previously only counts were shown

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-01 20:51:34 -04:00
Gmer4Lfe cb7d7a688d rename: failover → fallback throughout codebase — this system is Fallback, state is FALLBACK; preserve conventional failover comparisons 2026-06-01 20:04:17 -04:00
Gmer4Lfe e0ae80d2ea fix: remove all User Scripts dependencies — Varaverk plugin handles everything via event hooks, no User Scripts needed 2026-06-01 19:57:21 -04:00
Gmer4Lfe 98d2e6efa4 fix: remaining stale refs — boot drive path in Plugin Manual, array_started.sh typos in fallback and ramdisk_setup 2026-06-01 19:51:40 -04:00
Gmer4LfeandClaude Sonnet 4.6 cbf8593225 docs: remove bind-propagation=shared, document runc v1.3.5 crash
runc v1.3.5 (Docker 29.x / Unraid 7.3+) crashes on bind-propagation=shared
with any container — not GPU-specific. Ramdisk tmpfs is already MS_SHARED
at kernel level; propagation is automatic and the option is unnecessary.

- Remove bind-propagation=shared from all example Extra Parameters
- Update "Why shared Is Required" → "Mount Propagation — Unraid 7.3+"
- Update GPU troubleshooting section to cover the general runc crash
- Verify commands updated (no longer checking for Propagation: shared)
- Historical notes preserved explaining why it existed before

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 14:08:17 -04:00
Gmer4LfeandClaude Sonnet 4.6 fb0530deba Consolidate all paths to plugin flash dir, fix watchdog 7.3 triggers
- Move SCRIPTS_DIR/DATA_DIR/STATE_DIR from appdata to /boot/config/plugins/varaverk
- All state files now in STATE_DIR (no more /tmp or /boot/config root writes)
- Bootstrap: Gitea-first clone with GitHub fallback, no array dependency
- varaverk.cfg seeded with Gitea connection settings
- .gitignore: add State_Files/, varaverk.cfg, varaverk-*.txz
- Partnership/transcode/fallback scripts use STATE_DIR variables
- PHP config.php: DATA_DIR/STATE_DIR constants, VV_SETUP_STATE_FILE dynamic
- deploy.sh PROD_ROOT updated to plugin flash dir

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 13:30:20 -04:00
Gmer4Lfe fb051b60c1 Varaverk: FallBack + Watchdog tabs; plugin path restructure to Plugin/unraid/
- FallBack tab: per-node tier inventory + active fallback card with duration, tier, handback strikes, running container status
- Watchdog tab: live system health (RAM bar + thresholds, load, uptime, daemon), docker watchdog strikes + skip list + restart history, stability strikes + reboot log, resource pressure alert card, config inventory (mem limits, required, pause/stop lists)
- Swapped partnership/arrs tab order; FallBack between partnership and watchdog
- Plugin source tree moved from Plugin/usr/local/emhttp/plugins/varaverk/ to Plugin/unraid/
- Deployment/ conf templates added
2026-05-28 22:24:50 -04:00
Gmer4Lfe 59c2372fa1 Fix stale 3-minute transcode timing — corrected to 7 minutes
transcode_management.sh runs every 7 minutes. Six references across
README.md, README-Transcoding.md, Manual-Transcoding.md, transcode_manager.sh,
and transcode_cleanup.sh still said 3 minutes from before the schedule change.
2026-05-23 15:43:14 -04:00
Gmer4Lfe e215db2a57 Warn that VRAM exhaustion masks --gpus fixes
If PaddleOCR (or any GPU sidecar) holds all VRAM, --gpus will still
appear to fail even when configured correctly. Added explicit callout:
check nvidia-smi before concluding the GPU flag is wrong.
2026-05-23 14:30:33 -04:00
Gmer4Lfe c11e85d8be Show full GPU Extra Parameters string everywhere it appears
README-Transcoding now has a dedicated REQUIRED EXTRA PARAMETERS section
with both GPU and non-GPU variants and the Google warning up front.

Manual-Transcoding: two troubleshooting sections that showed mount-only
now show both GPU and non-GPU full strings. NVIDIA section quote style
fixed (--gpus "device=UUID" consistently).
2026-05-23 14:24:41 -04:00
Gmer4Lfe 1381540dab Document GPU setup, --gpus vs --runtime=nvidia, and VRAM starvation
Required Mount section now shows full Extra Parameters for both GPU and
non-GPU containers and explains why --gpus beats --runtime=nvidia at
setup time rather than burying it in Troubleshooting.

New troubleshooting entry documents the GPU memory starvation cascade:
OCR sidecar (PaddleOCR) holds VRAM → Jellyfin hard-fails, Emby falls
back to CPU silently. Emby's NVIDIA startup probe is one-shot — VRAM
must be free before restarting or NVIDIA stays disabled for the session.
Includes correct recovery sequence and Tesseract fallback for EmbyCredits.

README callout updated from two to three non-obvious requirements.
2026-05-23 14:11:00 -04:00
Gmer4Lfe 3b2501d091 docs(transcodes): note unRAID 7.2.5 regression context for NVIDIA conflict 2026-05-23 10:54:51 -04:00
Gmer4Lfe f992804739 docs(transcodes): document NVIDIA_VISIBLE_DEVICES + bind-propagation=shared conflict
--runtime=nvidia + NVIDIA_VISIBLE_DEVICES + bind-propagation=shared causes
"open /proc/self/mountinfo" init failure. Fix: --gpus device=UUID flag instead.
2026-05-23 10:46:58 -04:00
Gmer4Lfe 48374c1640 docs(transcodes): add dirty shutdown recovery to troubleshooting
Documents the known issue where an unclean shutdown leaves the --mount
bind-propagation entry in a broken state that prevents Emby from starting.
Recovery: remove mount → start → wait for load → re-add mount → restart.
2026-05-23 09:47:28 -04:00
Gmer4Lfe ee086f309e rename appdata-Failover→appdata-Fallback in docs; critical sync 15→30min; transcode 3→7min
appdata-Failover → appdata-Fallback:
  All .md files updated — scripts and confs already used appdata-Fallback.
  Share was renamed; docs were the only stale references.

critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
  critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
  Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
  Docker_Essentials README/downloaders_reset.sh, Fallback README,
  weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.

rsync_emby_failover User Script superseded:
  Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
  critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
  Section in User Script Plug-in README marked for deletion from plugin.

transcode_management.sh: */3 → */7:
  transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
  Transcodes README/manual, Orchestrators README, User Script Plug-in README,
  user_script_plug-in.sh all updated.
2026-05-22 22:22:21 -04:00
Gmer4Lfe 070013cb4f fix widespread naming errors across all docs — array_started, fallback, FALLBACK_
array_start.sh → array_started.sh (bulk rename — actual file has always been
array_started.sh; docs consistently used the wrong name across 9 files)

failover.sh → fallback.sh (bulk rename — script is Fallback/fallback.sh;
docs used the old name throughout)

FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1 → FALLBACK_HOST2_COVERS_HOST1_TIER1
(README.md — wrong variable name, actual vars use FALLBACK_ prefix and
COVERS_ not RUNS_FOR_)

docker_watchdog.sh attribution: "started by array_started.sh" → "called every
minute by watchdog_orchestrator.sh" (docker_watchdog is not in ARRAY_START_SCRIPTS)

server_reboot.sh shutdown sequence: step 7 "rc.docker stop" → calls
array_stopping.sh (user_scripts, fallback, rsync, mover, docker_container_stop)

Rsync manual ARRAY_START_SCRIPTS list: removed stability_watchdog and
docker_watchdog (now in watchdog_orchestrator); failover → fallback
2026-05-22 21:53:32 -04:00
Gmer4Lfe 95151c2278 Watchdogs/ folder + host conf rename
Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
2026-05-22 17:08:36 -04:00
Gmer4Lfe 588cde1cc4 Transcodes echo/log audit pass
ramdisk_setup.sh: path/size detail lines → log (already in log block above);
status done → echo. transcode_cleanup.sh: flip-back decision line and status
done → echo. transcode_manager.sh: clean-run single-line summary → echo.
Manual-Transcoding.md: added Output Tiers section.
2026-05-21 17:07:23 -04:00
Gmer4Lfe e932936acc Add docker availability check to all scripts that use docker commands
22 scripts used docker without verifying it's present. Consistent with the
pattern already established in Docker_Essentials scripts. Added after acquire_lock
and before detect_hosts in each script's setup section.
2026-05-20 18:35:34 -04:00
Gmer4Lfe e13f2fa14f feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections;
  triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
  gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
  Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
  common.sh, master confs, and new Manual/README docs
2026-05-19 20:00:10 -04:00
Gmer4LfeandClaude Sonnet 4.6 0ae31b5fa6 feat: Tailscale resolution hardening, partnership offboard completion, Emby provisioning
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
  parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
  is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
  PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)

Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
  own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
  problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
  if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
  matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
  or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
  for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
  cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
  dry runs)

master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)

master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS

Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 20:09:13 -04:00
FailedProxy ec7de648dc massive update. Master conf split, now modular with a load sceriprt to drive all configs to scripts. with unraid scpecific safeguard tests , and improved standardized ux. including dynamic host detect, who am i who else it there. EVERY SINGLE SCRIPT UPDATED. DEBATING THAT THIS IS ACUALLY V2 2026-05-03 17:16:49 -04:00
FailedProxy 6b36b8c07a fixed the double header in transcode manager 2026-04-20 18:20:26 -04:00
FailedProxy 1a16ca02a9 transcode manager now handles multiple media servers 2026-04-20 18:17:13 -04:00
FailedProxy e83473ab8e added safe guards to all scripts that need it. all reference comm.sh like it should be 2026-04-20 17:18:25 -04:00
FailedProxy 4103e1be65 icorp host profiles into scripts that needed it 2026-04-19 17:39:40 -04:00
FailedProxy bb709b2762 fixed in the way transcode is set up, all scripts in that folder adjusted 2026-04-19 12:36:44 -04:00
FailedProxy 2ee3d607fb fixed reADME AND USER SCRIPT PLUGIN 2026-04-19 12:12:49 -04:00
FailedProxy 8c5667b1df added all ai generated Readme files, added the last of the tools scripts 2026-04-18 15:40:41 -04:00
FailedProxy bccc490f04 new approach getting data for transcode manager 2026-04-15 20:13:59 -04:00
FailedProxy 950e32b6ac fixed live dectect in transcode manager 2026-04-15 18:54:50 -04:00
FailedProxy 01bc4cbcc2 added new ux to transcode manager 2026-04-15 18:49:54 -04:00
FailedProxy a3f2366ef8 transcode manager, symlink repair and more 2026-04-14 18:35:34 -04:00
FailedProxy ee107e5df7 fixed transcode cleanup 2026-04-11 10:37:14 -04:00
FailedProxy 7848b7c551 fixed an error in transcode manager 2026-04-11 09:01:15 -04:00
FailedProxy 70a58feee6 created transcode manager 2026-04-11 08:34:00 -04:00