Commit Graph
19 Commits
Author SHA1 Message Date
Gmer4Lfe d8a77c700b Stop counting a gated-off critical rsync as an unreachable partner, and report the counter in the unit it is stored in
The guard tested Tier 1 only, so onboard Step 1d's posture — Tier 1 open, every Tier 2 closed — made a healthy partner climb toward auto-offboard every 30 minutes. The counter holds intervals, not days, and was rendered raw.
2026-08-17 11:39:19 -04:00
Gmer4Lfe 19ba1e723c Stop a paused rsync counting as an absent partner and auto-offboarding the partnership 2026-08-16 13:13:25 -04:00
Gmer4Lfe e8b114094a Bring script headers onto the template and close safeguard gaps
Headers claimed protections the code never had, and several destructive paths had no
guard against a collapsed config value.
2026-08-01 20:37:59 -04:00
Gmer4Lfe 2a062e5140 Standardize orchestrator child-script execution and logging
Every orchestrator invoked its children differently — four near-duplicate
run_job() copies, a differently-shaped run_watchdog(), or plain inline bash
calls, each with its own take on path resolution, pass/fail naming, and
dry-run threading. Extracted one shared run_orch_child() into common.sh so
there's a single place to fix or extend this behavior going forward.

Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were
checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log
silently did nothing beyond basic logging on those two. Fixed to
$ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no
trailing exit, so their exit codes reflected whatever the last command
happened to return rather than actual success/failure. transcode_management.sh
had no failure notification and no summary at all. Also made
transcode_management.sh's two-script pipeline config-driven
(TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room
to extend it later without editing the orchestrator itself.
2026-07-03 10:57:52 -04:00
Gmer4Lfe f92ee4064b Add full banner headers to all scripts across the codebase
Every script now has the established header format: PURPOSE with ─────── separator,
OPERATIONAL MODEL, DESIGN PRINCIPLES, OPERATIONAL SAFEGUARDS, CONFIGURATION, and
RUNTIME MODES — structured with full ====== banner sections throughout.

Orchestrators converted from compact ── inline format to full banners. Stale
emby-fallback and dirty sync references removed from Plugin/user_script_plug-in.sh.
2026-06-26 18:50:05 -04:00
Gmer4Lfe 1003bee72a Replace emby dirty sync references with play_state_sync in comments and docs 2026-06-26 17:56:28 -04:00
Gmer4Lfe 3964f6fb46 audit echo vs log across all scripts — outcomes always visible, verbose for per-item loops 2026-06-14 12:40:15 -04:00
Gmer4Lfe 27bfc21cb0 Platform-agnostic refactor: eliminate OS-specific hardcodes from core scripts
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.

New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.

Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
2026-06-14 00:59:19 -04:00
Gmer4Lfe 369a9e6c19 Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites
- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename)
- Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API
  for storage health, service management, mover, user scripts, notifications,
  disk temps, and platform command validation
- Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR,
  auto-source Plugin/$PLATFORM/adapter.sh after common.sh
- Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg,
  disks.ini, and validate_unraid_cmd calls with platform_*() functions across
  watchdogs, orchestrators, and System_Essentials scripts
- Update all documentation: rename refs, update webgui escalation logic,
  add platform adapter section to Plugin README, update main README with
  portability vision and corrected self-healing stack description
2026-06-04 18:14:34 -04:00
Gmer4LfeandClaude Sonnet 4.6 bc70ebe5ee Add verbose log() coverage across Docker_Essentials, unRAID_Essentials, Transcodes, and Orchestrators
- Config/threshold dumps at startup in every script (retry counts, timeouts, sizes, thresholds)
- Per-item detail in verbose: container images, timing per container/share/job, image ID diffs
- Orchestrators: watchdog cycle now logs array state, grace state, per-script timing; transcode_management shows ramdisk state before each cycle; critical_sync logs share list and maintenance scripts; coffee report logs server state at run time
- Summary counts replaced with names in verbose where previously only counts were shown

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-01 20:51:34 -04:00
Gmer4Lfe cb7d7a688d rename: failover → fallback throughout codebase — this system is Fallback, state is FALLBACK; preserve conventional failover comparisons 2026-06-01 20:04:17 -04:00
Gmer4Lfe ee086f309e rename appdata-Failover→appdata-Fallback in docs; critical sync 15→30min; transcode 3→7min
appdata-Failover → appdata-Fallback:
  All .md files updated — scripts and confs already used appdata-Fallback.
  Share was renamed; docs were the only stale references.

critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
  critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
  Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
  Docker_Essentials README/downloaders_reset.sh, Fallback README,
  weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.

rsync_emby_failover User Script superseded:
  Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
  critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
  Section in User Script Plug-in README marked for deletion from plugin.

transcode_management.sh: */3 → */7:
  transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
  Transcodes README/manual, Orchestrators README, User Script Plug-in README,
  user_script_plug-in.sh all updated.
2026-05-22 22:22:21 -04:00
Gmer4Lfe 95151c2278 Watchdogs/ folder + host conf rename
Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
2026-05-22 17:08:36 -04:00
Gmer4Lfe ef8ef436cf Orchestrators echo/log audit pass
Status conclusions, clean-cycle summaries, skipped-feature lines, and phase
completion results → echo across all nine orchestrator scripts. Per-share,
per-job, and per-script detail lines inside loops remain log. README-Orchestrators:
added Output Tiers section.
2026-05-21 17:25:29 -04:00
Gmer4LfeandClaude Sonnet 4.6 0ae31b5fa6 feat: Tailscale resolution hardening, partnership offboard completion, Emby provisioning
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
  parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
  is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
  PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)

Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
  own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
  problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
  if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
  matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
  or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
  for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
  cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
  dry runs)

master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)

master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS

Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 20:09:13 -04:00
Gmer4LfeandClaude Sonnet 4.6 009820e981 refactor: rename failover/HA → fallback across entire codebase
Removes all references to "failover" and "HA" (high availability)
terminology from variable names, config keys, state values, rsync
profile names, directory paths, and user-visible strings.

Mapping:
  FAILOVER_*              → FALLBACK_*
  FAILOVER_HOST*_RUNS_FOR → FALLBACK_HOST*_COVERS
  critical-failover       → critical-fallback
  emby-failover           → emby-fallback
  appdata-Failover/       → appdata-Fallback/
  "FAILOVER" state value  → "FALLBACK"
  failover_start key      → fallback_start
  Failover/ directory     → Fallback/
  failover.sh             → fallback.sh
  failover_state.db       → fallback_state.db
  -Failover folder suffix → -Fallback

State machine: NORMAL | FALLBACK | DARK (unchanged)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 19:28:21 -04:00
Gmer4LfeandClaude Sonnet 4.6 5526bc1eca v2 post-migration bug fixes, lidarr_missing_art port, and Partnership SSH/FolderView3 enhancements
Bug fixes across the ecosystem after v1→v2 architecture migration and Unraid 7.2.5 upgrade:

- common.sh: fix _alias_array() phantom empty element (removed [@]:-} pattern), fix
  resolve_remote_ip() with Tailscale FQDN lowercase + awk fallback, add FANART/LASTFM
  key aliases in detect_hosts(), global [@]:-} sweep across 10+ scripts
- webgui_restart.sh: fix emhttp detection (pgrep emhttpd) and restart command for 7.2.5
  (/usr/local/sbin/emhttp stop && start — rc.emhttp removed in 7.2.5)
- bandwidth_monitor.sh, continuous_scripts_status.sh: fix 'local' keyword outside function
- backup_verify.sh: fix resolve_remote_ip() called before detect_hosts()
- Orchestrators: fix script display duplication bug in status output (${entry##*/})
- rsync_stop.sh, git_pull_execute.sh, partnership_manager.sh, coffee_report: lowercase all
  tailscale ip -4 call sites to match Tailscale's lowercase device names
- master_host1.conf: fix SSH key path (gmer4lfe_rsync_automation), add FANART/LASTFM keys
- master_host2.conf: add FANART/LASTFM API keys

New: Media/lidarr_missing_art.sh
- Full ecosystem port of standalone Lidarr artwork fetcher
- Fetches missing album art via fanart.tv + Last.fm APIs
- @tsv batch extraction: 1 jq call per API response vs N*albums (8050 albums in 24s)
- HOST guard (HOST1 only), --status, --dry-run, acquire_lock

New: Initial_run/ssh_setup.sh
- Generates {hostname}_rsync_automation ed25519 keypair (skip if exists, --force to regen)
- ssh-copy-id to remote via Tailscale IP, auto-updates master_host*.conf
- --validate mode: strike tracking (SSH_MAX_STRIKES, SSH_STRIKE_RESET_HRS),
  notify at limit — Tailscale-unreachable remote does NOT count as SSH strike

New: Initial_run/partnership_onboard.sh
- Orchestrator: ssh_setup.sh then partnership_manager.sh --onboard in one command

Partnership/partnership_manager.sh: FolderView3 integration
- Derive partner folder name at runtime (strip unraid- prefix case-insensitively)
- --onboard: create {Mirror}-Failover folder with failover tier containers
- --offboard (both paths): stop + rm containers in folder, remove JSON entry
- --check: calls ssh_setup.sh --validate when IP resolves but SSH state empty
- --status: shows FolderView3 folder state and containers inline

master.conf: SSH_MAX_STRIKES, SSH_STRIKE_RESET_HRS, PARTNERSHIP_FOLDERVIEW3,
PARTNERSHIP_FOLDERVIEW3_URL added to Partnership section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 18:20:23 -04:00
FailedProxy ec7de648dc massive update. Master conf split, now modular with a load sceriprt to drive all configs to scripts. with unraid scpecific safeguard tests , and improved standardized ux. including dynamic host detect, who am i who else it there. EVERY SINGLE SCRIPT UPDATED. DEBATING THAT THIS IS ACUALLY V2 2026-05-03 17:16:49 -04:00
FailedProxy 94f90a4579 padded partnership set up 2026-04-29 20:09:28 -04:00