Every orchestrator invoked its children differently — four near-duplicate
run_job() copies, a differently-shaped run_watchdog(), or plain inline bash
calls, each with its own take on path resolution, pass/fail naming, and
dry-run threading. Extracted one shared run_orch_child() into common.sh so
there's a single place to fix or extend this behavior going forward.
Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were
checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log
silently did nothing beyond basic logging on those two. Fixed to
$ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no
trailing exit, so their exit codes reflected whatever the last command
happened to return rather than actual success/failure. transcode_management.sh
had no failure notification and no summary at all. Also made
transcode_management.sh's two-script pipeline config-driven
(TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room
to extend it later without editing the orchestrator itself.
Every script now has the established header format: PURPOSE with ─────── separator,
OPERATIONAL MODEL, DESIGN PRINCIPLES, OPERATIONAL SAFEGUARDS, CONFIGURATION, and
RUNTIME MODES — structured with full ====== banner sections throughout.
Orchestrators converted from compact ── inline format to full banners. Stale
emby-fallback and dirty sync references removed from Plugin/user_script_plug-in.sh.
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.
New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.
Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
- Config/threshold dumps at startup in every script (retry counts, timeouts, sizes, thresholds)
- Per-item detail in verbose: container images, timing per container/share/job, image ID diffs
- Orchestrators: watchdog cycle now logs array state, grace state, per-script timing; transcode_management shows ramdisk state before each cycle; critical_sync logs share list and maintenance scripts; coffee report logs server state at run time
- Summary counts replaced with names in verbose where previously only counts were shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Exits cleanly before startup grace check if /mnt/user is not mounted
as shfs. Prevents watchdogs running against Docker/storage that isn't
available, and blocks false-positive reboots when array is stopped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move SCRIPTS_DIR/DATA_DIR/STATE_DIR from appdata to /boot/config/plugins/varaverk
- All state files now in STATE_DIR (no more /tmp or /boot/config root writes)
- Bootstrap: Gitea-first clone with GitHub fallback, no array dependency
- varaverk.cfg seeded with Gitea connection settings
- .gitignore: add State_Files/, varaverk.cfg, varaverk-*.txz
- Partnership/transcode/fallback scripts use STATE_DIR variables
- PHP config.php: DATA_DIR/STATE_DIR constants, VV_SETUP_STATE_FILE dynamic
- deploy.sh PROD_ROOT updated to plugin flash dir
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scheduler UI:
- Arrange mode: drag scripts between orchs and reorder within arrays; right
panel shows unassigned script pool; Save Arrangement commits to master.conf
- + Folder: named collapsible subfolders for Custom Scripts stored in schedule.json
- Rsync children: hide Run/Dry Run/Log/location when orch is ON; show standalone
location + cron controls when orch is OFF; cron only fires when both filled
- Non-conf-managed children (transcode): toggles now show enabled when orch is on
- Right panel height sync: fix ResizeObserver feedback loop via align-self:flex-start
on left panel and left.offsetHeight in vvFitRight
- How do I use this: updated to cover arrange, folders, rsync standalone, transcode
New API endpoints:
- board.php, clearlock.php, movescript.php, rawconf.php, readscript.php
- reorderarray.php, rsync_standalone.php, savefolders.php
run.php / dryrun.php: accept optional --location= arg for standalone rsync calls
watchdog_orchestrator.sh stays in Orchestrators/ — it's an orchestrator, not a watchdog.
watchdog_skip_list_manager.sh stays in Tools/ — it's a management utility.
Only the 4 watchdog scripts belong in Watchdogs/:
docker_watchdog.sh, resource_watchdog.sh, storage_watchdog.sh, system_watchdog.sh
Consistent naming with the watchdog family (system_watchdog, docker_watchdog).
File renamed, all RM_ config variables and internal references updated to RW_
across master.conf, master_host1.conf, common.sh, docker_watchdog.sh,
system_watchdog.sh, watchdog_orchestrator.sh, and sunday_morning_coffee_report.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduce a four-layer self-healing stack replacing the continuous-loop watchdogs:
- resource_manager.sh (new): single-pass pressure reduction layer; throttles
SABnzbd/qBit at level 1, docker-pauses background containers at level 2,
docker-stops optional containers and signals docker_watchdog to defer at
level 3; graduated recovery with hysteresis
- watchdog_orchestrator.sh (new, Orchestrators/): runs resource_manager →
docker_watchdog → system_watchdog in sequence; intended for per-minute cron
via User Scripts; startup grace, acquire_lock to prevent pile-up, heartbeat
- docker_watchdog.sh: de-looped to single-pass; daemon strikes persisted to
state file across runs; cross-script coordination reads RM_STATE_FILE instead
of SYS_WATCHDOG_STATE_FILE
- system_watchdog.sh: de-looped to single-pass; stripped of all container
management (shutdown_non_essential_containers removed); reboot-only last resort
- master.conf: removed system_watchdog and docker_watchdog from
ARRAY_START_SCRIPTS; added WATCHDOG ORCHESTRATOR and RESOURCE MANAGER sections
- master_host1.conf: added RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS arrays
- common.sh: aliased RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS via detect_hosts()
- continuous_scripts_status.sh: moved to Tools/ (preserved for future use)
- sunday_morning_coffee_report.sh: watchdog section updated to use state file
mtime checks instead of is_running; added Resource Manager subsection;
fixed mem_shutdown grep filter pointing to wrong state file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gmer4Lfe