Writes partner confs from RAM cache to /boot/config/.cache/vv/d/ while
remote is offline, and removes the backup when remote comes back. Called
each minute via SYSTEM_WATCHDOG_SCRIPTS so crashes and power loss are
covered — not just graceful shutdowns.
conf_cache_save.sh runs first on array stop — snapshots partner confs from RAM
cache to /boot/config/varaverk/conf_bak/ before anything else shuts down.
conf_cache_restore.sh runs after conf_sync.sh on array start — if partner was
unreachable and RAM cache is incomplete, loads the backup into RAM then removes
it. Normal reboots: backup written, fresh pull succeeds, backup deleted unused.
Edge case (partner down at boot): backup fills the gap so fallback.sh has the
partner vars it needs to operate correctly.
Web files now served via symlink to the git repo so git pull changes survive
reboots without rebuilding the txz. Also includes: docker pull/rebuild/restart
with live log streaming, arr_profile_enforcer for Sonarr/Radarr quality
profiles, monitor page cache fix (background writer now in cron), and
ARR_KIDS/SONARR/RADARR profile name vars in master.conf.
host.conf.template: add AUTH STACK section (NPM/lldap/Authelia) so
conf_upgrade.sh stops stripping those keys from host*.conf at 1am.
setup wizard: storage mode selector auto-detects USB vs NVMe at first
run; user can override; triggers storage_migrate.sh when mode differs
from current SCRIPTS_DIR location.
claude_startup.sh: reads HOST*_STORAGE_MODE_INTERNAL to pick internal
or appdata path rather than always requiring array to be mounted.
- REMOTE_ID_TIER*_DELAY renamed to HOSTN_TIER*_DELAY — these are own-host outage timers
- HOSTN_RADARR_MOVIE_ROOT → HOSTN_RADARR_MOVIES_ROOT (matches common.sh)
- HOSTN_SYS_WATCHDOG_CHECK_DOCKER → CHECK_DOCKER_DAEMON, CHECK_TEMP → CHECK_CPU_TEMP
- Add REMOTE_ID substitution to the sed in git_pull_execute.sh so COVERS_REMOTE_ID
resolves to the actual remote host (e.g. FALLBACK_HOST1_COVERS_HOST2_TIER1)
- Add HOSTN_PARTNERSHIP_REPLACE_CONTAINERS, HOSTN_PARTNERSHIP_ARR_REPLACE_CONTAINERS,
and HOSTN_WATCHDOG_CONTAINER_API_CHECKS to template so they aren't removed each pull
Section order now matches master.conf: IDENTITY → PARTNERSHIP → RSYNC →
FALLBACK → DOCKER → MEDIA → ARR STACK → TRANSCODES → MONITORS →
RESOURCE MANAGER → SYSTEM WATCHDOG. Extracted PARTNERSHIP from IDENTITY
into its own top-level section. PERSONAL SHARES moved to sit directly
after DAILY SYNC SHARES. NIC var moved to top of SYSTEM WATCHDOG block.
STATE_DIR introduced, DATA_DIR path updated to flash, all state files
migrated from /tmp and /boot/config root to $STATE_DIR. TRANSCODE_STATE_FILE
and DOCKER_WATCHDOG_FAILED_FILE added; SYS_WATCHDOG_FAILED_FILE renamed to
DOCKER_WATCHDOG_FAILED_FILE to reflect what it actually tracks.
Plugin/varaverk.plg: URL updated to github.com/FailedProxy/Varaverk
Configurations/host.conf.template:
Full host conf structure with HOSTN/hostn placeholders.
All personal values blank, all sections documented.
Covers identity, rsync, docker, fallback, media, monitors,
transcodes, arrs, system watchdog, resource manager.
Varaverk.page: checks if HOST1 is blank before rendering tabs.
If blank → shows setup wizard, returns early (tabs never render).
pages/setup.php: first-run wizard UI.
Auto-populates hostname from hostname -s.
Role selection: primary (HOST1) or partner (HOST2+).
Partner slot selector for HOST3+.
api/setup.php: handles wizard form POST.
Writes HOST1/HOST2 (and HOST3+) into master.conf preserving all
other content. Creates host*.conf from template with HOSTN/hostn
replaced and SSH key path pre-filled from hostname convention.
Never overwrites an existing host*.conf.
Plugin/varaverk.plg:
- Method="install update": git clone (fresh) or pull (update)
Validates Unraid, checks array is started via shfs mount, clones
to SCRIPTS_DIR (read from varaverk.cfg, default /mnt/user/appdata/Varaverk),
creates Plugin/unraid symlink, inits varaverk.cfg, bootstraps master.conf
from template on first install.
- Every boot: lightweight boot.sh recreates symlink if missing (no git ops).
- Method="remove": removes web symlink, leaves appdata intact.
Configurations/master.conf.template:
- Full master.conf with personal values blanked: HOST1/HOST2 empty,
GITEA_REPO_PATH empty, personal container names in profiles cleared,
FALLBACK_ENABLED=false, CRITICAL/DAILY rsync defaults corrected to true.
New users fill in HOST1, HOST2, then proceed to partnership setup.
Removed from DAILY_MAINTENANCE_SCRIPTS — critical orch already covers it
at a much tighter cadence. Play state stays in sync within 30 minutes of
any watched/resume activity on either server.
Syncs played/resume state bidirectionally across all configured Emby
and Jellyfin servers. Newest LastPlayedDate wins; users matched by
name (case-insensitive); unmatched users skipped cleanly.
Matching: IMDb/TMDB for movies, TVDB+s/e for episodes, MusicBrainz
track ID for audio. Also syncs resume positions for partial plays.
Config: PLAY_SYNC_ENABLED, PLAY_SYNC_DAYS (default 90), PLAY_SYNC_TYPES.
Modes: --dry-run, --status, --full (ignore day window), --log.
Wired into DAILY_MAINTENANCE_SCRIPTS after arr cleanup.
- Cron and Advanced button vertically aligned in their respective rows
- Verbose checkbox next to Log button — saves immediately, persists to schedule.json
- --log flag injected into cron lines, manual Run, and Dry Run when verbose enabled
- Coffee report rewritten as lean orchestrator over COFFEE_REPORT_SCRIPTS array
- COFFEE_REPORT_SCRIPTS added to master.conf (7 Sunday monitor scripts)
- Scheduler Advanced tab auto-discovers children via existing array detection
master.conf:
- reorder ORCHESTRATORS section by run frequency; add missing entries
- remove heartbeat vars (watchdogs no longer run in continuous cycle)
- remove dead REMOTE_DOCKER_RETRY_WAIT and display-only watchdog interval vars
- move INTERMEDIATE_RSYNC_ENABLED into RSYNC two-tier block; reorder by frequency
common.sh:
- add missing info() and success() definitions — both were called throughout but never defined
- update output tier description to list all 5 functions
- fix double-icon in check_local_disk_temps() error/warn calls
- fix smashed curl/ssh command lines in notify_emby_scan() and check_remote_docker_daemon()
- update check_rsync_enabled() comment to include INTERMEDIATE, ordered by frequency
emby_to_radarr_sync.sh / emby_to_sonarr_sync.sh:
- wire up RADARR_EMBY_LIBRARIES / SONARR_EMBY_LIBRARIES; empty array now scans all libraries
appdata-Failover → appdata-Fallback:
All .md files updated — scripts and confs already used appdata-Fallback.
Share was renamed; docs were the only stale references.
critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
Docker_Essentials README/downloaders_reset.sh, Fallback README,
weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.
rsync_emby_failover User Script superseded:
Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
Section in User Script Plug-in README marked for deletion from plugin.
transcode_management.sh: */3 → */7:
transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
Transcodes README/manual, Orchestrators README, User Script Plug-in README,
user_script_plug-in.sh all updated.
Two-gate design: server must have ≥30 days uptime AND last run must
be ≥30 days ago. Both gates must pass before any scripts fire. Called
daily at 3am via cron — script self-gates, calling more often is safe.
State file on /boot/config (survives reboots): the interval gate is
independent of uptime. A reboot resets uptime but does not reset when
maintenance last ran — both gates must independently pass.
MONTHLY_MAINTENANCE_SCRIPTS added to master.conf in ORCHESTRATORS
section. zfs_pool_scrub.sh and smart_long_test.sh listed but commented
(neither script exists yet). Also commits mesh_monitor.sh move to
Monitors/ that was staged from prior session.
Supports --force to bypass both gates for manual runs.
Checks:
1. Internet reachability (gates all remaining checks)
2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
3. Tailscale status — notify only, no auto-restart
4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart
Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle
Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
master.conf, host1.conf, host2.conf relocated from repo root to Configurations/.
load_config.sh updated to source from the new path — no other scripts need changes.
Note: sparse checkout rules on HOST1 and HOST2 need updating to use
Configurations/host1.conf and Configurations/host2.conf respectively.
Gmer4Lfe
FailedProxy