Commit Graph
31 Commits
Author SHA1 Message Date
Gmer4Lfe 0621f66889 Share one AI across the mesh instead of confining it to the owner
Curated state copied to every node is state that can disagree, so the index,
the model and the shared memory stay on the owner and each node reaches them
over the SSH trust onboarding already builds. Chats stay on the node that had
them; memory and bug reports stay the owner's to write.
2026-08-20 19:53:28 -04:00
Gmer4Lfe 2fefb56b8c Say which revision the pull restarted into
The restart line named no version, so a log showing one gave no way to tell what
the rest of the run actually executed.
2026-08-08 13:23:16 -04:00
Gmer4Lfe 934c0b56be Make a pull that updates the puller take effect on that pull
git installs the new file by rename, so the running bash finishes on the inode
it started with and every step after the sync used the previous version.
2026-08-08 13:21:14 -04:00
Gmer4Lfe e0cf83c944 Let one host template upgrade any host's conf
The pull substituted only HOSTN_, so bare HOSTN in comments kept tripping
conf_upgrade's own guard and no host conf had upgraded since the guard landed.
2026-08-08 13:08:24 -04:00
Gmer4Lfe 9c8eb8a2bc Refresh the AI index after a pull that changed tracked files
AI_INDEX_ON_PULL was designed, defaulted false, and never read by anything —
so nothing has ever rebuilt the index automatically. A pull is the only thing
that changes tracked files on a server, which makes it the only moment the
index can go stale, and a timer would do nothing 23 times a day while still
drifting from the pull that matters.

Staleness is invisible in the answers: the index keeps returning the old text
with confident, correct-looking citations. Gated on pull success, on
AI_INDEX_ON_PULL and on AI_ENABLED, and never fatal — a git pull must not fail
because an embedding call timed out.
2026-08-03 18:26:13 -04:00
Gmer4Lfe f92ee4064b Add full banner headers to all scripts across the codebase
Every script now has the established header format: PURPOSE with ─────── separator,
OPERATIONAL MODEL, DESIGN PRINCIPLES, OPERATIONAL SAFEGUARDS, CONFIGURATION, and
RUNTIME MODES — structured with full ====== banner sections throughout.

Orchestrators converted from compact ── inline format to full banners. Stale
emby-fallback and dirty sync references removed from Plugin/user_script_plug-in.sh.
2026-06-26 18:50:05 -04:00
Gmer4Lfe 6153283de4 move conf templates and conf_populate to Deployment/; add rogue-prevention rules to CLAUDE.md 2026-06-14 13:18:08 -04:00
Gmer4Lfe 27bfc21cb0 Platform-agnostic refactor: eliminate OS-specific hardcodes from core scripts
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.

New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.

Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
2026-06-14 00:59:19 -04:00
Gmer4Lfe 60ff658181 Fix conf_upgrade template naming bugs that caused false ADDED/REMOVED on every nightly pull
- REMOTE_ID_TIER*_DELAY renamed to HOSTN_TIER*_DELAY — these are own-host outage timers
- HOSTN_RADARR_MOVIE_ROOT → HOSTN_RADARR_MOVIES_ROOT (matches common.sh)
- HOSTN_SYS_WATCHDOG_CHECK_DOCKER → CHECK_DOCKER_DAEMON, CHECK_TEMP → CHECK_CPU_TEMP
- Add REMOTE_ID substitution to the sed in git_pull_execute.sh so COVERS_REMOTE_ID
  resolves to the actual remote host (e.g. FALLBACK_HOST1_COVERS_HOST2_TIER1)
- Add HOSTN_PARTNERSHIP_REPLACE_CONTAINERS, HOSTN_PARTNERSHIP_ARR_REPLACE_CONTAINERS,
  and HOSTN_WATCHDOG_CONTAINER_API_CHECKS to template so they aren't removed each pull
2026-06-13 10:22:02 -04:00
Gmer4Lfe 3937d5e33b Consolidate conf templates into Configurations/ — remove stale Deployment/conf_templates/
Deployment/conf_templates/ had two stale files: host.conf.template was a copy of host1.conf
(not a HOSTN template), and master.conf had hardcoded hostnames and an old appdata DATA_DIR path.
The correct templates already exist in Configurations/ (master.conf.template, host.conf.template).

Updated varaverk.plg and git_pull_execute.sh to read from Configurations/ directly.
Removed TMPL_DIR variable from git_pull_execute.sh — CONF_DIR covers both.
2026-06-12 21:35:58 -04:00
Gmer4Lfe 9c3ace95a7 Auth stack certs tab, arrs db fallbacks, cert monitor cache, conf parser fix
- Auth stack: fold cert monitor into Auth Stack page as fourth tab (Certs);
  remove standalone cert page and top-level tab
- cert_monitor.sh: write JSON status cache to State_Files/cert_status.json
  after each run; expose per-domain days/expiry via _CERT_DAYS/_CERT_EXPIRY globals
- api/cert.php: new — serves cached cert status; falls back to configured
  domains as UNKN when no cache exists; POST action=run triggers live check
- arrs db fallbacks: vv_arr_cleanup_stats/discovery_stats/recovery_stats now
  read from data/*.db files when log JSON files don't yet exist
- config.php vv_conf_vars(): unescape bash \$ → $ so passwords with dollar
  signs read correctly from conf files
- host1.conf: fill in HOST1_NPM_USER/PASS and HOST1_LLDAP_USER/PASS
- Partnership adapter pattern: Unraid-specific container logic extracted to
  Plugin/unraid/Partnership/; platform-agnostic structure stays in Partnership/
- First-run wizard: uniform multi-step flow for all hosts; HOST2 pull moved
  to checklist; auto SSH keygen and API key creation on save
- api/checklist.php: live setup checklist with pull_master action
- Fullscreen toggle: hide Unraid header/menu; state persists via localStorage
2026-06-05 23:17:30 -04:00
Gmer4Lfe 437c525fbd git pull: replace reset --hard + clean -fd with --ff-only 2026-06-05 19:24:48 -04:00
Gmer4Lfe b5d33cd6a5 Bash common.sh: consolidate docker_cmd/retry_docker/verify_running/emby_api, fix tailscale dups
common.sh gains:
- docker_cmd() + verify_running() + retry_docker() — removed from all 3 Docker_Essentials
  scripts where they were byte-for-byte duplicates
- emby_api(endpoint, [timeout=30]) — removed from 6 Media/Tools scripts that each defined
  their own _emby_api() with the same curl/parse/error pattern; call sites renamed emby_api
- format_duration() extended with days/hours branch (was capped at minutes+seconds)
- notify() comment: scripts do not need to preflight the notify script via validate_unraid_cmd

Tailscale deduplication:
- arr_sync.sh: _resolve_node_ip() and inline block in _delete_remote_item() both replaced
  with resolve_tailscale_ip() from common.sh
- git_pull_execute.sh: inline tailscale ip -4 replaced with resolve_tailscale_ip() (adds
  the tailscale status fallback that was missing)
2026-06-04 17:00:50 -04:00
Gmer4Lfe 0cb429222a Add storage mode migration: internal NVMe vs USB flash, Settings tab, fix hardcoded /boot/ paths 2026-06-03 17:40:35 -04:00
Gmer4Lfe cb7d7a688d rename: failover → fallback throughout codebase — this system is Fallback, state is FALLBACK; preserve conventional failover comparisons 2026-06-01 20:04:17 -04:00
Gmer4Lfe da3d402f1a Revert "git_pull_execute: GitHub fallback when GITEA_REPO_PATH is empty"
This reverts commit 9a26478498.
2026-05-30 16:56:49 -04:00
Gmer4Lfe 9a26478498 git_pull_execute: GitHub fallback when GITEA_REPO_PATH is empty
End users leave GITEA_REPO_PATH="" in master.conf (template default).
Before hitting any Gitea detection logic, the script now checks for this
and does git pull --ff-only origin main directly — origin is already
GitHub since the .plg cloned from there.

Sparse checkout skipped on this path: conf files are gitignored so
nothing sensitive is in the repo to exclude.

Developer path (GITEA_REPO_PATH configured) is completely unchanged.
2026-05-30 16:55:25 -04:00
Gmer4Lfe 921900b23b rename appdata folder and repo from unraid_scripts to varaverk
- TARGET_DIR, DATA_DIR: /mnt/user/appdata/unraid_scripts → /mnt/user/appdata/varaverk
- GITEA_REPO_PATH: Varaverk/Unraid_Scripts.git → Varaverk/varaverk.git
- DEV_ROOT: .../Development/Unraid_Scripts → .../Development/varaverk
- deploy hook path updated in .claude/settings.json
- All script comments, docs, and manual paths updated throughout
2026-05-30 10:07:03 -04:00
Gmer4Lfe fb051b60c1 Varaverk: FallBack + Watchdog tabs; plugin path restructure to Plugin/unraid/
- FallBack tab: per-node tier inventory + active fallback card with duration, tier, handback strikes, running container status
- Watchdog tab: live system health (RAM bar + thresholds, load, uptime, daemon), docker watchdog strikes + skip list + restart history, stability strikes + reboot log, resource pressure alert card, config inventory (mem limits, required, pause/stop lists)
- Swapped partnership/arrs tab order; FallBack between partnership and watchdog
- Plugin source tree moved from Plugin/usr/local/emhttp/plugins/varaverk/ to Plugin/unraid/
- Deployment/ conf templates added
2026-05-28 22:24:50 -04:00
Gmer4Lfe 95151c2278 Watchdogs/ folder + host conf rename
Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
2026-05-22 17:08:36 -04:00
Gmer4Lfe 9ee8af1a71 git_pull_execute: remove setup banner 2026-05-21 17:29:25 -04:00
Gmer4Lfe e932936acc Add docker availability check to all scripts that use docker commands
22 scripts used docker without verifying it's present. Consistent with the
pattern already established in Docker_Essentials scripts. Added after acquire_lock
and before detect_hosts in each script's setup section.
2026-05-20 18:35:34 -04:00
Gmer4LfeandClaude Sonnet 4.6 f16c962ac0 feat: rename array orchestrators, add docker_update remainder mode
- Rename array_start.sh → array_started.sh, array_stop.sh → array_stopping.sh
  to clarify these are event-driven (array has started/is stopping), not imperative
- Update all references across 9 files (master.conf, user_script_plug-in.sh,
  watchdogs, continuous_scripts_status.sh, claude_startup.sh)
- Add --remainder mode to docker_update.sh: updates all running containers
  excluding daily containers, weekly sync-window containers (emby + critical-data),
  and fallback coverage containers (FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*)
  Fallback containers excluded because the remote server owns their version —
  independent updates risk writeback incompatibility on handback
- weekly_sync_maintenance.sh calls docker_update.sh --remainder as final step
- git_pull_execute.sh: add safe.directory config to fix dubious ownership error
  when running as root on a directory owned by uid 1000

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 22:12:08 -04:00
Gmer4LfeandClaude Sonnet 4.6 0ae31b5fa6 feat: Tailscale resolution hardening, partnership offboard completion, Emby provisioning
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
  parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
  is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
  PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)

Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
  own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
  problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
  if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
  matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
  or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
  for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
  cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
  dry runs)

master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)

master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS

Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 20:09:13 -04:00
Gmer4LfeandClaude Sonnet 4.6 b65f572367 feat: arr-native distributed media sync + config architecture fixes
Arr sync (new):
- Media/arr_sync.sh — full mesh bidirectional sync across all HOST* nodes
  - Lidarr (MusicBrainz), Sonarr (TVDB), Radarr (TMDB) all handled in one script
  - Remote API keys read live from config.xml via SSH — never stored in conf files
  - Shared blocklist (DATA_DIR/arr_sync_blocklist.tsv) merged from all nodes at runtime
  - Graceful skip if arr not configured locally or not reachable on a remote node
  - --blocklist-add / --blocklist-remove / --blocklist-list management flags
- daily_sync_maintenance.sh — arr sync runs as explicit phase before rsync
- partnership_onboard.sh — Step 3 bootstraps merged library on both sides at onboard
- master.conf — ARR_SYNC_* config block, DOCKER_APPDATA_BASE

Rsync / cleanup:
- DEFAULT_RSYNC_OPTS — removed --delete; arr_cleanup.sh owns orphan enforcement
- lidarr_cleanup.sh — removed HOST1-only guard; runs on any node with Lidarr configured

Config architecture:
- HOST1/HOST2 hostnames moved from master_host*.conf → master.conf (not credentials)
- Sparse checkout now works correctly: each server only needs its own host conf
- detect_hosts() still resolves MY_ID + REMOTE_ID via master.conf hostname values

Bug fix:
- common.sh line 493 — watchdog toggle eval had broken quoting; all SYS_WATCHDOG_CHECK_*
  globals were silently set to empty instead of their configured values

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 09:53:40 -04:00
Gmer4LfeandClaude Sonnet 4.6 5526bc1eca v2 post-migration bug fixes, lidarr_missing_art port, and Partnership SSH/FolderView3 enhancements
Bug fixes across the ecosystem after v1→v2 architecture migration and Unraid 7.2.5 upgrade:

- common.sh: fix _alias_array() phantom empty element (removed [@]:-} pattern), fix
  resolve_remote_ip() with Tailscale FQDN lowercase + awk fallback, add FANART/LASTFM
  key aliases in detect_hosts(), global [@]:-} sweep across 10+ scripts
- webgui_restart.sh: fix emhttp detection (pgrep emhttpd) and restart command for 7.2.5
  (/usr/local/sbin/emhttp stop && start — rc.emhttp removed in 7.2.5)
- bandwidth_monitor.sh, continuous_scripts_status.sh: fix 'local' keyword outside function
- backup_verify.sh: fix resolve_remote_ip() called before detect_hosts()
- Orchestrators: fix script display duplication bug in status output (${entry##*/})
- rsync_stop.sh, git_pull_execute.sh, partnership_manager.sh, coffee_report: lowercase all
  tailscale ip -4 call sites to match Tailscale's lowercase device names
- master_host1.conf: fix SSH key path (gmer4lfe_rsync_automation), add FANART/LASTFM keys
- master_host2.conf: add FANART/LASTFM API keys

New: Media/lidarr_missing_art.sh
- Full ecosystem port of standalone Lidarr artwork fetcher
- Fetches missing album art via fanart.tv + Last.fm APIs
- @tsv batch extraction: 1 jq call per API response vs N*albums (8050 albums in 24s)
- HOST guard (HOST1 only), --status, --dry-run, acquire_lock

New: Initial_run/ssh_setup.sh
- Generates {hostname}_rsync_automation ed25519 keypair (skip if exists, --force to regen)
- ssh-copy-id to remote via Tailscale IP, auto-updates master_host*.conf
- --validate mode: strike tracking (SSH_MAX_STRIKES, SSH_STRIKE_RESET_HRS),
  notify at limit — Tailscale-unreachable remote does NOT count as SSH strike

New: Initial_run/partnership_onboard.sh
- Orchestrator: ssh_setup.sh then partnership_manager.sh --onboard in one command

Partnership/partnership_manager.sh: FolderView3 integration
- Derive partner folder name at runtime (strip unraid- prefix case-insensitively)
- --onboard: create {Mirror}-Failover folder with failover tier containers
- --offboard (both paths): stop + rm containers in folder, remove JSON entry
- --check: calls ssh_setup.sh --validate when IP resolves but SSH state empty
- --status: shows FolderView3 folder state and containers inline

master.conf: SSH_MAX_STRIKES, SSH_STRIKE_RESET_HRS, PARTNERSHIP_FOLDERVIEW3,
PARTNERSHIP_FOLDERVIEW3_URL added to Partnership section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 18:20:23 -04:00
FailedProxy ec7de648dc massive update. Master conf split, now modular with a load sceriprt to drive all configs to scripts. with unraid scpecific safeguard tests , and improved standardized ux. including dynamic host detect, who am i who else it there. EVERY SINGLE SCRIPT UPDATED. DEBATING THAT THIS IS ACUALLY V2 2026-05-03 17:16:49 -04:00
FailedProxy e83473ab8e added safe guards to all scripts that need it. all reference comm.sh like it should be 2026-04-20 17:18:25 -04:00
FailedProxy b1d1bcc847 added dns lookup if local and tailscale fail for git 2026-04-19 17:43:49 -04:00
FailedProxy 4103e1be65 icorp host profiles into scripts that needed it 2026-04-19 17:39:40 -04:00
FailedProxy 6cc26c8fb9 added notify to all needed in unraid essentials 2026-04-09 17:20:09 -04:00