- Run button (blue) fires script immediately, opens log panel;
stamps log with manual run timestamp before exec
- Save moved to card footer — one button saves all cron expressions
for the orchestrator and all its children at once
- Toggle still auto-saves immediately on flip (enable/disable is instant)
- Removed onblur from cron inputs — cron only saves on explicit Save
- api/run.php: validates id, appends timestamp, execs script in background
- Per-job log files replace single jobs.log: each script writes to
/var/log/varaverk/<Category>/<script>.log mirroring the job ID
- api/log.php: serves last 200 lines of a job's log file with mtime
timestamp; supports POST ?clear=1 to truncate
- Scheduler page redesigned as full-width cards (one per orchestrator)
with Save + Log buttons; Log toggles an inline panel that polls
every 3 seconds and auto-scrolls — same pattern as unRAID User Scripts
- Children shown under Advanced with identical per-job log panels
- CSS: vv-sched-card, vv-log-panel, vv-log-pre, vv-btn-sm
Non-greedy (.*?) stopped at the first ) found, which could be inside
a comment like '# gracefully stop fallback (not caught by ...)'.
Anchoring the closing paren to start-of-line (^\s*\)) ensures it only
matches the array's actual closing paren, not ) inside comment text.
Orchestrators using dynamic dispatch (array_started, watchdog_orchestrator,
daily/weekly/etc maintenance) iterate \${VARNAME_SCRIPTS[@]} arrays defined
in master.conf. The previous regex only caught static hardcoded paths.
Two-strategy approach, merged and deduped:
1. Static: \$VAR/../Category/script.sh and \$VAR/Category/script.sh patterns
2. Dynamic: detect \${VARNAME[@]} references in the orch, parse that array
from master.conf — handles inline args and skips commented entries
Changes to master.conf arrays are reflected immediately on next page load.
Previous regex only matched literal bash/source calls with bare paths.
Orchestrators use shell variables like \$SCRIPT_DIR/../Transcodes/script.sh
and \$SCRIPTS_ROOT/Category/script.sh — the old pattern missed all of these.
New regex captures the Category/script.sh portion from either pattern,
requires at least one subdirectory (excludes load_config.sh and root-level
utilities), and validates each candidate against the filesystem.
transcode_management now correctly shows transcode_cleanup and transcode_manager.
Orchestrators using master.conf config arrays for dynamic dispatch still show
no children — correct, as those lists cannot be statically parsed.
/etc/cron.d/varaverk is RAM-based and wiped on every reboot. The
disks_mounted event fires early in boot (before Docker) with /boot
already readable, so schedule.json is available to restore the cron
file automatically without any user action.
/var/log/ is RAM-based on unRAID — wiped on reboot. Creating the log
directory in vv_cron_rebuild() ensures it exists before the first cron
job fires, without needing a separate boot script.
- Rename varaverk.page → Varaverk.page: unRAID nginx only routes URLs
starting with a capital letter (~^/[A-Z].*), lowercase caused 404
- Add require_once config.php to include/scheduler.php and include/docs.php:
both used SCRIPTS_DIR constant without including the file that defines it
- Replace hardcoded SCRIPTS_DIR with cfg-file-backed setting: reads from
/boot/config/plugins/varaverk/varaverk.cfg, falls back to default on first run
- Add Plugin Settings card to Config tab with scripts path field
- Add api/settings.php to write the cfg file (validates directory exists)
critical_sync_maintenance.sh runs every 30 minutes (*/30 cron).
Three references in README.md and Partnership/README-Partnership.md
still said 15 minutes from an earlier schedule.
transcode_management.sh runs every 7 minutes. Six references across
README.md, README-Transcoding.md, Manual-Transcoding.md, transcode_manager.sh,
and transcode_cleanup.sh still said 3 minutes from before the schedule change.
If PaddleOCR (or any GPU sidecar) holds all VRAM, --gpus will still
appear to fail even when configured correctly. Added explicit callout:
check nvidia-smi before concluding the GPU flag is wrong.
README-Transcoding now has a dedicated REQUIRED EXTRA PARAMETERS section
with both GPU and non-GPU variants and the Google warning up front.
Manual-Transcoding: two troubleshooting sections that showed mount-only
now show both GPU and non-GPU full strings. NVIDIA section quote style
fixed (--gpus "device=UUID" consistently).
Every search result shows --runtime=nvidia which conflicts with
bind-propagation=shared on unRAID 7.2.5+ and breaks on container
rebuilds. New prominent section explains the correct --gpus approach,
why the old method fails, and the VRAM starvation cascade.
Item #6 in Surprises updated to reference the new section; arr cleanup
renumbered to #8.
Required Mount section now shows full Extra Parameters for both GPU and
non-GPU containers and explains why --gpus beats --runtime=nvidia at
setup time rather than burying it in Troubleshooting.
New troubleshooting entry documents the GPU memory starvation cascade:
OCR sidecar (PaddleOCR) holds VRAM → Jellyfin hard-fails, Emby falls
back to CPU silently. Emby's NVIDIA startup probe is one-shot — VRAM
must be free before restarting or NVIDIA stays disabled for the session.
Includes correct recovery sequence and Tesseract fallback for EmbyCredits.
README callout updated from two to three non-obvious requirements.
Documents the known issue where an unclean shutdown leaves the --mount
bind-propagation entry in a broken state that prevents Emby from starting.
Recovery: remove mount → start → wait for load → re-add mount → restart.
Follows the same extraction pattern as partnership_offboard.sh.
partnership_manager.sh --transfer now exec-forwards to the new script.
Legacy inline block preserved under `if false` for reference.
Fixes in partnership_transfer.sh vs the original inline code:
- WebUI reconfiguration corrected: SSH to new owner (mirror) to set localhost;
reconfigure_local_webuis() on current server to point to new owner IP
- master.conf remote update uses scp push instead of self-SSH with wrong path
- SSH key uses $SSH_KEY directly (was using undefined {HOST*}_SSH_KEY expansion)
partnership_manager.sh:
- fix auto-offboard threshold: × 96 → × 48 (was triggering at 2× configured days)
- fix do_final_sync(): use CRITICAL_SYNC_SHARES if configured; fall back to hardcoded paths
- replace inline --check offboard blocks with background partnership_offboard.sh calls:
• owner path was missing remote container cleanup (steps 8+9) and blocklist
• mirror path was missing blocklist, SSH revocation, CRITICAL_RSYNC_ENABLED=false
• local grace_seconds outside a function (bash error) eliminated with the block removal
• 6hr Tailscale grace sleep was blocking the cron for the full grace window
partnership_offboard.sh:
- reorder owner path: state write moved from step 5 to step 10 (last) so re-running
after a mid-offboard crash restarts from scratch instead of exiting as INACTIVE;
state is now written before the Tailscale grace sleep so --check stops re-triggering
- add FolderView3 remote folder cleanup in step 7 (mirror's fallback folder persisted)
- track step 1+2 outcome with STEP_STOP_OK / STEP_SYNC_OK; summary now reflects actual
result instead of hardcoding ✅ regardless of dry-run or failure
- update header step list to match new step order
master.conf:
- reorder ORCHESTRATORS section by run frequency; add missing entries
- remove heartbeat vars (watchdogs no longer run in continuous cycle)
- remove dead REMOTE_DOCKER_RETRY_WAIT and display-only watchdog interval vars
- move INTERMEDIATE_RSYNC_ENABLED into RSYNC two-tier block; reorder by frequency
common.sh:
- add missing info() and success() definitions — both were called throughout but never defined
- update output tier description to list all 5 functions
- fix double-icon in check_local_disk_temps() error/warn calls
- fix smashed curl/ssh command lines in notify_emby_scan() and check_remote_docker_daemon()
- update check_rsync_enabled() comment to include INTERMEDIATE, ordered by frequency
emby_to_radarr_sync.sh / emby_to_sonarr_sync.sh:
- wire up RADARR_EMBY_LIBRARIES / SONARR_EMBY_LIBRARIES; empty array now scans all libraries
appdata-Failover → appdata-Fallback:
All .md files updated — scripts and confs already used appdata-Fallback.
Share was renamed; docs were the only stale references.
critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
Docker_Essentials README/downloaders_reset.sh, Fallback README,
weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.
rsync_emby_failover User Script superseded:
Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
Section in User Script Plug-in README marked for deletion from plugin.
transcode_management.sh: */3 → */7:
transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
Transcodes README/manual, Orchestrators README, User Script Plug-in README,
user_script_plug-in.sh all updated.
Update the Three-Tier Response and RAM Tiers sections to reflect the
resource_watchdog/stability_watchdog split: MEM_WARN_GB/MEM_SHUTDOWN_GB/
MEM_RECOVER_GB moved to RW_RAM_SOFT/MEDIUM/HARD/RECOVER_GB in resource_watchdog;
stability_watchdog now only holds SYS_WATCHDOG_MEM_GB (reboot trigger). Update
required containers reference from SYS_WATCHDOG_REQUIRED_CONTAINERS (removed)
to HOST*_WATCHDOG_REQUIRED_CONTAINERS in host*.conf.
array_start.sh → array_started.sh (bulk rename — actual file has always been
array_started.sh; docs consistently used the wrong name across 9 files)
failover.sh → fallback.sh (bulk rename — script is Fallback/fallback.sh;
docs used the old name throughout)
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1 → FALLBACK_HOST2_COVERS_HOST1_TIER1
(README.md — wrong variable name, actual vars use FALLBACK_ prefix and
COVERS_ not RUNS_FOR_)
docker_watchdog.sh attribution: "started by array_started.sh" → "called every
minute by watchdog_orchestrator.sh" (docker_watchdog is not in ARRAY_START_SCRIPTS)
server_reboot.sh shutdown sequence: step 7 "rc.docker stop" → calls
array_stopping.sh (user_scripts, fallback, rsync, mover, docker_container_stop)
Rsync manual ARRAY_START_SCRIPTS list: removed stability_watchdog and
docker_watchdog (now in watchdog_orchestrator); failover → fallback
Failover/ was renamed to Fallback/ — all script paths and README links
updated across README.md, Orchestrators README, and User Script Plug-in README.
array_start.sh → array_started.sh in Orchestrators scripts table (wrong name).
fallback_state_reset.sh: script is referenced in User Script Plug-in README
but does not exist in Fallback/. Entry updated to document it as not yet built
and provides the manual workaround (direct state file write).
Note: /mnt/user/appdata-Failover/ share path references left unchanged —
that is the actual unRAID share name on the server, not the script folder.
arr_sync.sh bidirectional union replaced the old split-ownership model.
Old: HOST1 owns Movies/TV/Music, HOST2 owns Anime — each arr manages
different shares to avoid conflicts.
New: arr_sync.sh keeps all arr databases in union. Either server can
download to any share at any time. The union is the source of truth.
Auth stack references left intact — HOST1 still owns auth config
(one-way sync to HOST2). Only media library ownership language updated.
Files changed:
README.md — server block + share annotations
Fallback/README-Fallback.md — role line + Split Source of Truth section
Orchestrators/README-Orchestrators — daily sync share comments
Rsync/Manual-Rsync.md — DEFAULT_RSYNC_OPTS comment block
Media/README-Media.md — HOST AWARENESS section
Old model: HOST1 owns Movies/TV/Music, HOST2 owns Anime — each arr
manages different shares to prevent conflicts.
New model: arr_sync.sh syncs all arr databases bidirectionally before
every rsync cycle. Either server can download to any share at any time.
The databases converge (union, not overwrite), then rsync spreads files
additively. Neither server owns a share. The union is the source of truth.
Also updates share name annotations and removes the "read-only mirror"
framing from the share naming section.
Two-gate design: server must have ≥30 days uptime AND last run must
be ≥30 days ago. Both gates must pass before any scripts fire. Called
daily at 3am via cron — script self-gates, calling more often is safe.
State file on /boot/config (survives reboots): the interval gate is
independent of uptime. A reboot resets uptime but does not reset when
maintenance last ran — both gates must independently pass.
MONTHLY_MAINTENANCE_SCRIPTS added to master.conf in ORCHESTRATORS
section. zfs_pool_scrub.sh and smart_long_test.sh listed but commented
(neither script exists yet). Also commits mesh_monitor.sh move to
Monitors/ that was staged from prior session.
Supports --force to bypass both gates for manual runs.
Checks:
1. Internet reachability (gates all remaining checks)
2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
3. Tailscale status — notify only, no auto-restart
4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart
Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle
Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
master.conf, host1.conf, host2.conf relocated from repo root to Configurations/.
load_config.sh updated to source from the new path — no other scripts need changes.
Note: sparse checkout rules on HOST1 and HOST2 need updating to use
Configurations/host1.conf and Configurations/host2.conf respectively.
README.md: header updated to VARAVERK. All "User Scripts plugin" references
replaced with "Arbiter". Project name section: "Working name: Arbiter" → "Varaverk".
Gitea corrected to Tier 1 (was shown as Tier 3, updated to Tier 1 this session).
master.conf: GITEA_REPO_PATH updated FailedProxy → Varaverk org name.
Script internals (user_scripts_stop.sh, script headers, other docs) reference
the /tmp/user.scripts working path which is Arbiter-specific — those will be
updated in a separate pass once Arbiter's working directory is confirmed.
Partnership/gitea_ssh_setup.sh: generates ed25519 keypair, registers public key
with Gitea API (POST /api/v1/user/keys), tests SSH connection, writes ~/.ssh/config
entry. Idempotent — skips steps already done. --force regenerates and re-registers.
Resolves Gitea endpoint from container IP first, falls back to GITEA_DOMAIN.
common.sh: alias HOST*_GITEA_API_TOKEN → GITEA_API_TOKEN in detect_hosts().
Add to detect_hosts() doc comment.
master.conf: add GITEA_HTTP_PORT=3000 for API endpoint construction.
host1.conf: add HOST1_GITEA_API_TOKEN (fill in from Gitea Settings → Applications).
Add my-Gitea.xml to HOST1_PARTNERSHIP_AUTH_STACK — onboard pushes it to HOST2.
host2.conf: add Gitea to FALLBACK_HOST2_COVERS_HOST1_TIER1 — starts immediately
when HOST1 goes down, making the source of truth reachable independently of
HOST1's auth stack.
host1.conf: HOST1_RAMDISK_SIZE 8G→10G, WARN_GB 6.8→8.5, LOW_GB 5.5→7.
ramdisk_stop.sh: new Tools/ script — redirects symlink to SSD before unmount
so Emby keeps writing during maintenance, auto-falls back to lazy umount
when only directory handles are open (no active writes).
common.sh: fix `info` call at line 595 (undefined) → `log`.
Tools docs: added ramdisk_stop.sh to README table, categories, diagram, and Manual.
watchdog_orchestrator.sh stays in Orchestrators/ — it's an orchestrator, not a watchdog.
watchdog_skip_list_manager.sh stays in Tools/ — it's a management utility.
Only the 4 watchdog scripts belong in Watchdogs/:
docker_watchdog.sh, resource_watchdog.sh, storage_watchdog.sh, system_watchdog.sh
Gmer4Lfe