The guard tested Tier 1 only, so onboard Step 1d's posture — Tier 1 open, every Tier 2 closed — made a healthy partner climb toward auto-offboard every 30 minutes. The counter holds intervals, not days, and was rendered raw.
Finishes the pass: every script now documents its safeguards, and the deliberate absences
in the sourced libraries are recorded so they are not "corrected" later.
stop_local_containers() reads LOCAL_CRITICAL_CONTAINER_NAMES, which was
never set here — only rsync.sh sets it, in a child process that can't
propagate back. Local stop always no-op'd, so the rebuild-on-new-image
step never had anything to act on.
Codebase-wide audit pass: fixed real bugs (SSH hangs missing BatchMode,
local-outside-function no-ops, variable name collisions, a truncated
ratio calc, wrong state-dir path, DARK vs NO_INTERNET drift, and more),
then pulled logic that was duplicated across multiple scripts — arr
cleanup safety gates, docker restart ordering, container maintenance
stop/restart, watchdog state-file helpers, partnership role resolution,
cert expiry checks, remote node discovery, and TMDB discovery scoring —
into common.sh so each now has a single implementation.
Every orchestrator invoked its children differently — four near-duplicate
run_job() copies, a differently-shaped run_watchdog(), or plain inline bash
calls, each with its own take on path resolution, pass/fail naming, and
dry-run threading. Extracted one shared run_orch_child() into common.sh so
there's a single place to fix or extend this behavior going forward.
Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were
checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log
silently did nothing beyond basic logging on those two. Fixed to
$ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no
trailing exit, so their exit codes reflected whatever the last command
happened to return rather than actual success/failure. transcode_management.sh
had no failure notification and no summary at all. Also made
transcode_management.sh's two-script pipeline config-driven
(TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room
to extend it later without editing the orchestrator itself.
Media/ now holds only media-level scripts (cleaner, permissions, play_state_sync).
All arr management scripts (cleanup, discovery, sync, webhooks, release fixer) live in Arrs_Stack/.
Every script now has the established header format: PURPOSE with ─────── separator,
OPERATIONAL MODEL, DESIGN PRINCIPLES, OPERATIONAL SAFEGUARDS, CONFIGURATION, and
RUNTIME MODES — structured with full ====== banner sections throughout.
Orchestrators converted from compact ── inline format to full banners. Stale
emby-fallback and dirty sync references removed from Plugin/user_script_plug-in.sh.
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.
PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.
conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.
fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.
README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
Weekly sync window was pulling images but using docker start after rsync,
which ignores the new digest. Containers in the emby/critical-data profiles
(Emby, Mariadb, Redis) never actually landed on updated images.
docker_update_remaining.sh merged into docker_update.sh --remainder, which
already had better exclusion logic. Added WEEKLY_REMAINING_UPDATES toggle
and WEEKLY_RESTART_CONTAINERS exclusion to remainder mode.
Onboard now runs docker_network_connect.sh on mirror before deploying stacks.
Fix discrepancies found by auditing docs against headers:
- stability_watchdog.sh: wrong variable name (STRIKES → STRIKE_LIMIT) and wrong default (2hr → 12hr) in header
- Watchdogs manual: REBOOT_WINDOW_HRS example value was 2, should be 12
- Transcodes README: transcode_cleanup/manager table said "Every 3 min", should be "Every 7 minutes"
- Tools manual: fallback_state_reset.sh still marked "not yet built" — rewrite section to reflect current script
- Tools manual: claude_startup --setup flag doesn't exist; modes were inverted — fix and add --launch
- Tools README + manual: add docker_prune_images.sh (existed but undocumented)
- Tools manual: add play_state_sync.sh --full to flag reference
- Orchestrators README: arrs_failed_stalled_recovery runs via intermediate_sync (every 4hr), not standalone every 6hr
- Plugin README: add build.sh to scripts table
- Partnership README + manual: add partnership_transfer.sh and onboard_cancel.sh to script tables and flag reference
OS version reads now go through platform_get_os_version() and
platform_os_version_probe_cmd() instead of grepping /etc/unraid-version directly.
STATE_DIR fallbacks to /boot/config removed — STATE_DIR is always set by
load_config.sh and the fallback encoded a platform-specific path.
Setup DB path references use platform_setup_db_path() instead of the
VARAVERK_SETUP_FILE/-/boot/config compound fallback.
DOCKER_APPDATA_BASE default removed from arr_sync.sh — the adapter sets it.
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.
New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.
Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
Partner conf cache (/tmp/.vv/) was only refreshed at array start. If a partner
updated their conf mid-day, this host's RAM cache went stale until next reboot.
Intermediate orchestrator now pulls partner confs every 4 hours as step 1.
conf_sync.sh --pull-only: pulls partner confs into local cache, skips push
(push is already handled by conf_populate.sh --push-only on conf save).
- Config/threshold dumps at startup in every script (retry counts, timeouts, sizes, thresholds)
- Per-item detail in verbose: container images, timing per container/share/job, image ID diffs
- Orchestrators: watchdog cycle now logs array state, grace state, per-script timing; transcode_management shows ramdisk state before each cycle; critical_sync logs share list and maintenance scripts; coffee report logs server state at run time
- Summary counts replaced with names in verbose where previously only counts were shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Exits cleanly before startup grace check if /mnt/user is not mounted
as shfs. Prevents watchdogs running against Docker/storage that isn't
available, and blocks false-positive reboots when array is stopped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move SCRIPTS_DIR/DATA_DIR/STATE_DIR from appdata to /boot/config/plugins/varaverk
- All state files now in STATE_DIR (no more /tmp or /boot/config root writes)
- Bootstrap: Gitea-first clone with GitHub fallback, no array dependency
- varaverk.cfg seeded with Gitea connection settings
- .gitignore: add State_Files/, varaverk.cfg, varaverk-*.txz
- Partnership/transcode/fallback scripts use STATE_DIR variables
- PHP config.php: DATA_DIR/STATE_DIR constants, VV_SETUP_STATE_FILE dynamic
- deploy.sh PROD_ROOT updated to plugin flash dir
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scheduler UI:
- Arrange mode: drag scripts between orchs and reorder within arrays; right
panel shows unassigned script pool; Save Arrangement commits to master.conf
- + Folder: named collapsible subfolders for Custom Scripts stored in schedule.json
- Rsync children: hide Run/Dry Run/Log/location when orch is ON; show standalone
location + cron controls when orch is OFF; cron only fires when both filled
- Non-conf-managed children (transcode): toggles now show enabled when orch is on
- Right panel height sync: fix ResizeObserver feedback loop via align-self:flex-start
on left panel and left.offsetHeight in vvFitRight
- How do I use this: updated to cover arrange, folders, rsync standalone, transcode
New API endpoints:
- board.php, clearlock.php, movescript.php, rawconf.php, readscript.php
- reorderarray.php, rsync_standalone.php, savefolders.php
run.php / dryrun.php: accept optional --location= arg for standalone rsync calls
Previously ran daily at 3am with internal self-gating. Now scheduled directly
on the 15th — uptime and interval gates still apply; a skip means next month's 15th.
- Cron and Advanced button vertically aligned in their respective rows
- Verbose checkbox next to Log button — saves immediately, persists to schedule.json
- --log flag injected into cron lines, manual Run, and Dry Run when verbose enabled
- Coffee report rewritten as lean orchestrator over COFFEE_REPORT_SCRIPTS array
- COFFEE_REPORT_SCRIPTS added to master.conf (7 Sunday monitor scripts)
- Scheduler Advanced tab auto-discovers children via existing array detection
appdata-Failover → appdata-Fallback:
All .md files updated — scripts and confs already used appdata-Fallback.
Share was renamed; docs were the only stale references.
critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
Docker_Essentials README/downloaders_reset.sh, Fallback README,
weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.
rsync_emby_failover User Script superseded:
Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
Section in User Script Plug-in README marked for deletion from plugin.
transcode_management.sh: */3 → */7:
transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
Transcodes README/manual, Orchestrators README, User Script Plug-in README,
user_script_plug-in.sh all updated.
array_start.sh → array_started.sh (bulk rename — actual file has always been
array_started.sh; docs consistently used the wrong name across 9 files)
failover.sh → fallback.sh (bulk rename — script is Fallback/fallback.sh;
docs used the old name throughout)
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1 → FALLBACK_HOST2_COVERS_HOST1_TIER1
(README.md — wrong variable name, actual vars use FALLBACK_ prefix and
COVERS_ not RUNS_FOR_)
docker_watchdog.sh attribution: "started by array_started.sh" → "called every
minute by watchdog_orchestrator.sh" (docker_watchdog is not in ARRAY_START_SCRIPTS)
server_reboot.sh shutdown sequence: step 7 "rc.docker stop" → calls
array_stopping.sh (user_scripts, fallback, rsync, mover, docker_container_stop)
Rsync manual ARRAY_START_SCRIPTS list: removed stability_watchdog and
docker_watchdog (now in watchdog_orchestrator); failover → fallback
Failover/ was renamed to Fallback/ — all script paths and README links
updated across README.md, Orchestrators README, and User Script Plug-in README.
array_start.sh → array_started.sh in Orchestrators scripts table (wrong name).
fallback_state_reset.sh: script is referenced in User Script Plug-in README
but does not exist in Fallback/. Entry updated to document it as not yet built
and provides the manual workaround (direct state file write).
Note: /mnt/user/appdata-Failover/ share path references left unchanged —
that is the actual unRAID share name on the server, not the script folder.
arr_sync.sh bidirectional union replaced the old split-ownership model.
Old: HOST1 owns Movies/TV/Music, HOST2 owns Anime — each arr manages
different shares to avoid conflicts.
New: arr_sync.sh keeps all arr databases in union. Either server can
download to any share at any time. The union is the source of truth.
Auth stack references left intact — HOST1 still owns auth config
(one-way sync to HOST2). Only media library ownership language updated.
Files changed:
README.md — server block + share annotations
Fallback/README-Fallback.md — role line + Split Source of Truth section
Orchestrators/README-Orchestrators — daily sync share comments
Rsync/Manual-Rsync.md — DEFAULT_RSYNC_OPTS comment block
Media/README-Media.md — HOST AWARENESS section
Two-gate design: server must have ≥30 days uptime AND last run must
be ≥30 days ago. Both gates must pass before any scripts fire. Called
daily at 3am via cron — script self-gates, calling more often is safe.
State file on /boot/config (survives reboots): the interval gate is
independent of uptime. A reboot resets uptime but does not reset when
maintenance last ran — both gates must independently pass.
MONTHLY_MAINTENANCE_SCRIPTS added to master.conf in ORCHESTRATORS
section. zfs_pool_scrub.sh and smart_long_test.sh listed but commented
(neither script exists yet). Also commits mesh_monitor.sh move to
Monitors/ that was staged from prior session.
Supports --force to bypass both gates for manual runs.
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
watchdog_orchestrator.sh stays in Orchestrators/ — it's an orchestrator, not a watchdog.
watchdog_skip_list_manager.sh stays in Tools/ — it's a management utility.
Only the 4 watchdog scripts belong in Watchdogs/:
docker_watchdog.sh, resource_watchdog.sh, storage_watchdog.sh, system_watchdog.sh
Gmer4Lfe