Every orchestrator invoked its children differently — four near-duplicate
run_job() copies, a differently-shaped run_watchdog(), or plain inline bash
calls, each with its own take on path resolution, pass/fail naming, and
dry-run threading. Extracted one shared run_orch_child() into common.sh so
there's a single place to fix or extend this behavior going forward.
Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were
checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log
silently did nothing beyond basic logging on those two. Fixed to
$ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no
trailing exit, so their exit codes reflected whatever the last command
happened to return rather than actual success/failure. transcode_management.sh
had no failure notification and no summary at all. Also made
transcode_management.sh's two-script pipeline config-driven
(TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room
to extend it later without editing the orchestrator itself.
Exit code 0 on a required container (docker stop, UI stop) now reads as
intentional — no strike, no restart. Non-zero exits still trigger the
existing strike → restart path.
Adds --pause / --resume management commands and a persistent intentional-
stops state file for maintenance windows where even the exit-code heuristic
isn't enough. Containers auto-cleared from the list when seen running again.
Media/ now holds only media-level scripts (cleaner, permissions, play_state_sync).
All arr management scripts (cleanup, discovery, sync, webhooks, release fixer) live in Arrs_Stack/.
Reads MUSICBRAINZ_ALBUMID from FLAC (vorbis block type 4) and MP3 (ID3v2 TXXX)
files, matches against Lidarr's known releases, switches monitored=true to the
correct one, and queues RefreshArtist. Runs before lidarr_cleanup.sh in the
daily job list so the strike system doesn't act on files that just needed a
release correction.
docker_weekly_restart.sh was missing the trailing dangling prune that daily_restart has.
docker_update.sh rebuild failure message always named docker_daily_restart.sh regardless of mode.
docker_prune_images.sh --all added to monthly — the only scheduled path that removes tagged orphan images.
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.
PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.
conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.
fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.
README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
Adds require_partnership() to common.sh — exits cleanly when PARTNERSHIP_ENABLED=false.
Removes FALLBACK_PARTNERSHIP_REQUIRED toggle — partnership is now always required,
not optional. Cross-server scripts (rsync, conf sync, fallback, arr sync, play state,
backup verify) all call require_partnership after detect_hosts.
Writes partner confs from RAM cache to /boot/config/.cache/vv/d/ while
remote is offline, and removes the backup when remote comes back. Called
each minute via SYSTEM_WATCHDOG_SCRIPTS so crashes and power loss are
covered — not just graceful shutdowns.
conf_cache_save.sh runs first on array stop — snapshots partner confs from RAM
cache to /boot/config/varaverk/conf_bak/ before anything else shuts down.
conf_cache_restore.sh runs after conf_sync.sh on array start — if partner was
unreachable and RAM cache is incomplete, loads the backup into RAM then removes
it. Normal reboots: backup written, fresh pull succeeds, backup deleted unused.
Edge case (partner down at boot): backup fills the gap so fallback.sh has the
partner vars it needs to operate correctly.
Web files now served via symlink to the git repo so git pull changes survive
reboots without rebuilding the txz. Also includes: docker pull/rebuild/restart
with live log streaming, arr_profile_enforcer for Sonarr/Radarr quality
profiles, monitor page cache fix (background writer now in cron), and
ARR_KIDS/SONARR/RADARR profile name vars in master.conf.
Removed from DAILY_MAINTENANCE_SCRIPTS — critical orch already covers it
at a much tighter cadence. Play state stays in sync within 30 minutes of
any watched/resume activity on either server.
Syncs played/resume state bidirectionally across all configured Emby
and Jellyfin servers. Newest LastPlayedDate wins; users matched by
name (case-insensitive); unmatched users skipped cleanly.
Matching: IMDb/TMDB for movies, TVDB+s/e for episodes, MusicBrainz
track ID for audio. Also syncs resume positions for partial plays.
Config: PLAY_SYNC_ENABLED, PLAY_SYNC_DAYS (default 90), PLAY_SYNC_TYPES.
Modes: --dry-run, --status, --full (ignore day window), --log.
Wired into DAILY_MAINTENANCE_SCRIPTS after arr cleanup.
- Cron and Advanced button vertically aligned in their respective rows
- Verbose checkbox next to Log button — saves immediately, persists to schedule.json
- --log flag injected into cron lines, manual Run, and Dry Run when verbose enabled
- Coffee report rewritten as lean orchestrator over COFFEE_REPORT_SCRIPTS array
- COFFEE_REPORT_SCRIPTS added to master.conf (7 Sunday monitor scripts)
- Scheduler Advanced tab auto-discovers children via existing array detection
master.conf:
- reorder ORCHESTRATORS section by run frequency; add missing entries
- remove heartbeat vars (watchdogs no longer run in continuous cycle)
- remove dead REMOTE_DOCKER_RETRY_WAIT and display-only watchdog interval vars
- move INTERMEDIATE_RSYNC_ENABLED into RSYNC two-tier block; reorder by frequency
common.sh:
- add missing info() and success() definitions — both were called throughout but never defined
- update output tier description to list all 5 functions
- fix double-icon in check_local_disk_temps() error/warn calls
- fix smashed curl/ssh command lines in notify_emby_scan() and check_remote_docker_daemon()
- update check_rsync_enabled() comment to include INTERMEDIATE, ordered by frequency
emby_to_radarr_sync.sh / emby_to_sonarr_sync.sh:
- wire up RADARR_EMBY_LIBRARIES / SONARR_EMBY_LIBRARIES; empty array now scans all libraries
appdata-Failover → appdata-Fallback:
All .md files updated — scripts and confs already used appdata-Fallback.
Share was renamed; docs were the only stale references.
critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
Docker_Essentials README/downloaders_reset.sh, Fallback README,
weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.
rsync_emby_failover User Script superseded:
Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
Section in User Script Plug-in README marked for deletion from plugin.
transcode_management.sh: */3 → */7:
transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
Transcodes README/manual, Orchestrators README, User Script Plug-in README,
user_script_plug-in.sh all updated.
Two-gate design: server must have ≥30 days uptime AND last run must
be ≥30 days ago. Both gates must pass before any scripts fire. Called
daily at 3am via cron — script self-gates, calling more often is safe.
State file on /boot/config (survives reboots): the interval gate is
independent of uptime. A reboot resets uptime but does not reset when
maintenance last ran — both gates must independently pass.
MONTHLY_MAINTENANCE_SCRIPTS added to master.conf in ORCHESTRATORS
section. zfs_pool_scrub.sh and smart_long_test.sh listed but commented
(neither script exists yet). Also commits mesh_monitor.sh move to
Monitors/ that was staged from prior session.
Supports --force to bypass both gates for manual runs.
Checks:
1. Internet reachability (gates all remaining checks)
2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
3. Tailscale status — notify only, no auto-restart
4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart
Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle
Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
master.conf, host1.conf, host2.conf relocated from repo root to Configurations/.
load_config.sh updated to source from the new path — no other scripts need changes.
Note: sparse checkout rules on HOST1 and HOST2 need updating to use
Configurations/host1.conf and Configurations/host2.conf respectively.
Gmer4Lfe