Daily restarts create orphaned images that were never cleaned up.
Added docker image prune -f section matching docker_update_remaining.sh pattern.
Prune runs each cycle so orphans from the previous day's restart are removed daily
rather than waiting for the weekly docker_update_remaining.sh sweep.
Previously ran daily at 3am with internal self-gating. Now scheduled directly
on the 15th — uptime and interval gates still apply; a skip means next month's 15th.
- vv_script_children() now calls vv_script_suggested_cron() per child,
adding suggested_cron and suggested_label to each child entry
- Orch rows (non-event) and regular script child rows render a faint
.vv-cron-hint span to the left of the cron input showing the suggested
cron from the script's # Schedule: header
- Clicking the hint fills in the cron input (data-cron + onclick)
- Tooltip shows full "suggested_cron — label" string on hover
- .vv-cron-hint: color #444, brightens to #888 on hover
- Right panel title: "Suggested Schedules" → "Scheduler Information"
- Removed top hint paragraph; info now lives in the panel itself
- Added three collapsible info sections (open by default) at top of panel:
Controls — toggle, cron, run, dry run, log, stop, verbose, config
Orchestrators — enabled/disabled behavior, child cron rules, event triggers
Children & Advanced — toggle semantics, independent cron, rsync flag badge
- Bullet lists use CSS columns:2 with column-fill:balance for even distribution
- "Suggested Schedules" divider separates info from parsed schedule blocks
- Added vvToggleSug() for DOM-traversal-based accordion (no ID dependency)
- Rsync/rsync.sh children in Advanced now show as conf_flag type when the
parent orch controls a *_RSYNC_ENABLED tier flag; toggle writes true/false
to master.conf instead of comment/uncommenting a SCRIPTS array entry
- Added vv_conf_flag_value() and vv_conf_flag_set() helpers in scheduler.php
- Added api/flag_toggle.php endpoint (validates *_RSYNC_ENABLED pattern)
- Added api/stop.php: kills process group, sweeps stuck locks, updates stat
- vv-flag-badge CSS (amber, monospace) to distinguish from event/script rows
- vvSaveChild() routes conf_flag children to flag_toggle.php unconditionally
- vvApplyOrchState() keeps conf_flag toggle at real flag value; disables when orch off
- vvSaveAll() skips conf_flag children (immediate-save only)
When orch is ON (god mode):
- Children's toggle state is read from master.conf comment status
- Toggling a child comments/uncomments its line in the *_SCRIPTS array
- Child crons are suppressed in vv_cron_rebuild — orch is the sole trigger
- Children not found in any *_SCRIPTS array are shown disabled (read-only)
When orch is OFF:
- All children flip to off; schedule.json updated immediately
- A child with a cron value + enabled toggle gets its own independent cron entry
- A child with no cron does nothing when enabled
New: vv_conf_script_map() — cached per-request scan of master.conf arrays
New: vv_parse_conf_array_full() — includes commented entries (disabled scripts)
New: vv_conf_toggle_script() — comments/uncomments a script line in master.conf
New: api/conf_toggle.php — endpoint for child toggle → master.conf write
Two hardcoded event-triggered entries appear at the top of the scheduler
(array_started.sh / array_stopping.sh). They show an ⚡ Array Start /
⚡ Array Stop badge instead of a cron field and are enabled/disabled via
the normal toggle.
Static event scripts fire them via Unraid's event system:
- event/disks_mounted/array_start_jobs → runs in background (non-blocking)
- event/disks_unmounting/array_stop_jobs → runs foreground (blocks until done)
vv_cron_rebuild() skips @array_* entries so they never land in the cron file.
Scripts executed on each event are managed in master.conf via
ARRAY_START_SCRIPTS and ARRAY_STOP_SCRIPTS arrays.
Auto Scroll now pauses automatically when user scrolls up and resumes
when they reach the bottom — no separate Scroll Lock needed.
Lidarr discovery no longer hard-skips single-seed candidates; they
score low on breadth but can still reach threshold on merit.
Before update_cron was wired in, the plugin wrote directly to
/etc/cron.d/varaverk. That file persisted after the migration and
caused duplicate job firing (two watchdog instances at :00/:15/:30/:45).
The second instance hit the lock, exited with warn, and overwrote the
stat file — killing the running indicator while the first was still live.
vv_cron_rebuild() now unlinks the legacy file after every update_cron
call. Since rebuild_cron (boot event) calls vv_cron_rebuild(), the
cleanup runs on every reboot and every schedule save.
run_job.sh was overwriting the log on every cron fire, making it
impossible to see run history. Switch to append mode, print a
── YYYY-MM-DD HH:MM:SS ── separator before each run, and trim to
the last 1000 lines after each run to keep logs bounded.
Cron runs with PATH=/usr/bin:/bin, which omits /usr/local/sbin where
tailscale (and other Unraid tools) live. resolve_remote_ip silently got
empty results from both tailscale calls, failing all 3 retry attempts
every scheduled run. Export the full path at source time so all scripts
find tailscale regardless of how they're invoked.
Scripts calling acquire_lock followed by acquire_rsync_lock (rsync.sh)
or a custom EXIT trap (resource_watchdog.sh) would overwrite the lock
release trap, leaving the first lock file behind on exit.
Replace per-acquire trap with a _LOCK_FILES registry. _register_lock()
appends each lock file to the array and sets a single _release_all_locks
trap. All acquired locks release together on exit regardless of how many
traps are subsequently set or overwritten.
resource_watchdog.sh: chain _release_all_locks into both its custom trap
and the state-persisted disarm path so the lock releases in all exit paths.
The first cron run of critical_sync_maintenance failed because Tailscale
had a transient moment at startup. resolve_remote_ip() now retries 3 times
with a 5s delay before giving up, making it resilient to brief daemon
hiccups without masking real peer-offline conditions.
Add api/status.php — reads all scheduled jobs' stat files and returns a
{id: status} map. The scheduler page polls it every 3s and lights up the
dot on any job whose stat shows status=running, clearing it when done.
Replace single vvRunningId with vvRunningSet (Set) so multiple jobs can
show as running simultaneously. vvSetDot/vvClearDot handle per-job dots
and the header dot for the active job. Remove stale-count detection from
vvFetchRight — status poll is now the single source of truth.
Create varaverk.plg on flash (/boot/config/plugins/varaverk/varaverk.plg).
On boot, rebuild_cron copies it to /var/log/plugins/ so update_cron natively
picks up varaverk.cron and merges it into /etc/cron.d/root.
With the .plg in place the dev-mode direct-inject fallback in vv_cron_rebuild()
is no longer needed and has been removed.
dcron uses the filename in /etc/cron.d/ as the username, so a file named
'varaverk' ran jobs as a non-existent user and was silently ignored.
Switch to writing /boot/config/plugins/varaverk/varaverk.cron and calling
update_cron (which merges all plugin .cron files into /etc/cron.d/root).
In dev mode (no varaverk.plg registered), fall back to injecting our entries
directly into the live root crontab via crontab -c /etc/cron.d -.
run_job.sh: new wrapper called by scheduler cron for every job.
Writes /var/log/varaverk/<id>.json (status/start/end/exit/pid)
and /var/log/varaverk/<id>.log (latest run output, overwritten).
Maps exit 0→ok, 1→warn, 2+→error so watchdog "took action" runs
show warn not error.
scheduler: cron now calls bash run_job.sh instead of inline bash -c.
Added vv_job_stat_path(). LOG_DIR moved to config.php (shared).
monitor: vv_scripts_status() now reads /var/log/varaverk/*.json and
*/*.json — no User Scripts tmpScripts dependency at all.
Stale running detection via /proc/<pid> check.
Returns error_count in addition to running/ok/warn counts.
pages/monitor.php: added error pill (red ✗), error icon/color in
list, duration shown next to age for each script entry.
log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
All watchdog scripts now open with a consistent echo banner:
━━━ <icon> <Name> Watchdog — <timestamp> ━━━
Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.
Also: network_watchdog now logs "Network healthy ✅" on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
network_watchdog: Tailscale JSON uses "BackendState": "Running" (with space)
but grep pattern matched "BackendState":"Running" (no space) — always failed.
Switched to grep -qE with \s* to handle both forms.
system_watchdog: add echo banner at run start so its output is visually
separated from docker_watchdog in orchestrator logs. Previously appeared
as a continuation of the docker section since docker_watchdog has a banner
but system_watchdog did not.
Layout:
- Switch from three flex rows to single CSS grid (repeat(8,1fr)) — eliminates
gap-count discrepancy between rows; span 1/2/4/8 cards are now pixel-identical
across all rows regardless of how many cards share the row
New cards:
- Scripts card: last-run status (ok/warn/running/unknown), clickable filter pills,
7-row scroll, flex:1 matching Power/Parity
- Streams card: live per-second progress counters, color-coded play buttons
(blue=live, orange=transcoding, green=direct, yellow=paused), server badges
with counts, left-fill column layout (3 per col)
- Transcode card: NVMe indicator, active session count + scrollable list,
session rows with server icon + title + type + method
Dynamic thresholds:
- vv_disk_thresholds() reads /boot/config/plugins/dynamix/dynamix.cfg
- Disk utilization bars: green/orange/red based on warning/critical settings
- Temperature colors: hot/max for HDD, hotssd/maxssd for SSD/NVMe
- Thresholds update every poll cycle; changing Unraid settings takes effect automatically
Other:
- SSD path detection uses findmnt FSTYPE to exclude tmpfs/ramfs (fixes RAM
being reported as SSD transcode target)
- varaverk.css h3: add white-space:normal + overflow:hidden to neutralise
any Unraid global h3 styles that could force card width
- network_watchdog: remove ISSUES++ for Tailscale — it's notify-only by
design, no restart possible; cascading to system watchdog exit 1 was noise
- docker_watchdog: move banner to top of cycle so body errors appear under
the correct section header, not visually inside Resource Manager output
- CPU/GPU cards: flex-column so body div takes remaining height
- CPU canvas: flex:1 + min-height:36px instead of fixed 36px
- GPU: add vvGpuHistory + canvas (purple, same style as CPU); canvas flex:1
stretches to fill whatever space the row's tallest card leaves over
- External IP: switch from DDNS domain lookup to curl ifconfig.me with 5-min
cache; DDNS approach broke because host detection uses HOST1_NAME but conf
has HOST1, so domain was never resolved
- Tailscale IP: switch from hardcoded tailscale0 iface to `tailscale ip -4`;
interface name varies (tailscale0, tailscale1) across kernels/installs
media.php: Emby/Jellyfin/Plex session poller for Streams card
confform.php: conf file form helpers for Config tab
script.php: per-script run/log API endpoint for Scheduler tab
vv_network_stats() calls vv_conf_vars() and vv_detect_host() which are defined in
config.php. monitor.php had no require for it, causing a PHP fatal error on every
poll → empty JSON → all monitor cards stuck on "Loading...".
Monitor tab:
- CPU card: per-core bars (freq-colored), overall usage bar, rolling history chart
- Memory card: breakdown by system/VM/ZFS/Docker/free with progress bars
- Network card: LAN + ext + Tailscale IP display, live RX/TX chart with auto-scale
- Streams card: Emby/Jellyfin/Plex sessions with playback bar and server badges
- GPU card: adds power draw, encode%, decode% meters
- Transcode card: ramdisk vs SSD location pill, session count, flip history
- Canvas chart helpers: vvDrawChart, vvDrawNetChart, vvMeter, vvFmtBps, vvFmtGib
api/monitor.php: adds cpu, mem, net to response payload
- Add .vv-save-check span to each job row (orchestrators + children)
- CSS: fade-in-then-out animation (visible 1.2s, fades over 2s)
- vvFlashSaved(job): triggers the checkmark animation on a specific row
- vvFlashStatus(el, msg, ok): shows timed status text (was called but missing)
- vvSaveJob: now shows per-job checkmark on successful save
- vvSaveAll: shows checkmark on each job as its response arrives, plus
global "✓ All saved" / "✗ Some failed" status text below the button
Unraid's system crond is dcron (dillon's cron daemon 4.5), which reads
/etc/cron.d/ for system crontabs WITHOUT a username field, and refuses
to execute files that are group- or world-writable.
vv_cron_rebuild() was writing "SCHED root bash ..." (with username field)
and leaving the file world-writable (666), so dcron silently ignored every
entry. Fixed:
- Remove CRON_USER from the generated cron line format
- chmod 0600 the cron file after writing it
- Change suggestion title color from bright green (#6fcf97) to soft grey-blue (#7a9eb5)
- Fix vv_parse_user_script_template() block-detection regex — $ end-anchor
failed on lines ending with multibyte U+2500 ─ chars; removed anchor so
parser now returns all 22 blocks from user_script_plug-in.sh
- Cron and Advanced button vertically aligned in their respective rows
- Verbose checkbox next to Log button — saves immediately, persists to schedule.json
- --log flag injected into cron lines, manual Run, and Dry Run when verbose enabled
- Coffee report rewritten as lean orchestrator over COFFEE_REPORT_SCRIPTS array
- COFFEE_REPORT_SCRIPTS added to master.conf (7 Sunday monitor scripts)
- Scheduler Advanced tab auto-discovers children via existing array detection
Cron input now sits right of the script label in the same row.
Below each row: one-line description pulled live from the script header
(PURPOSE block or first meaningful comment line) — updates whenever
scripts are edited, no plugin changes needed.
Scripts list on left, persistent log panel on right. On screens ≤880px
falls back to stacked single-column. Right panel stays open until page
refresh or another selection. Running dot pulses green on the job row
and in the log header while script executes, clears when log goes stale.
All fetch() POSTs now send application/x-www-form-urlencoded with the
page-injected csrf_token, satisfying unRAID's auto_prepend CSRF check.
All PHP API handlers switched from php://input JSON to $_POST.
Also adds Dry Run button (orange, between Run and Log) that sets
DRY_RUN=1 in the script environment before executing.
- Run button (blue) fires script immediately, opens log panel;
stamps log with manual run timestamp before exec
- Save moved to card footer — one button saves all cron expressions
for the orchestrator and all its children at once
- Toggle still auto-saves immediately on flip (enable/disable is instant)
- Removed onblur from cron inputs — cron only saves on explicit Save
- api/run.php: validates id, appends timestamp, execs script in background
- Per-job log files replace single jobs.log: each script writes to
/var/log/varaverk/<Category>/<script>.log mirroring the job ID
- api/log.php: serves last 200 lines of a job's log file with mtime
timestamp; supports POST ?clear=1 to truncate
- Scheduler page redesigned as full-width cards (one per orchestrator)
with Save + Log buttons; Log toggles an inline panel that polls
every 3 seconds and auto-scrolls — same pattern as unRAID User Scripts
- Children shown under Advanced with identical per-job log panels
- CSS: vv-sched-card, vv-log-panel, vv-log-pre, vv-btn-sm
Non-greedy (.*?) stopped at the first ) found, which could be inside
a comment like '# gracefully stop fallback (not caught by ...)'.
Anchoring the closing paren to start-of-line (^\s*\)) ensures it only
matches the array's actual closing paren, not ) inside comment text.
Orchestrators using dynamic dispatch (array_started, watchdog_orchestrator,
daily/weekly/etc maintenance) iterate \${VARNAME_SCRIPTS[@]} arrays defined
in master.conf. The previous regex only caught static hardcoded paths.
Two-strategy approach, merged and deduped:
1. Static: \$VAR/../Category/script.sh and \$VAR/Category/script.sh patterns
2. Dynamic: detect \${VARNAME[@]} references in the orch, parse that array
from master.conf — handles inline args and skips commented entries
Changes to master.conf arrays are reflected immediately on next page load.
Previous regex only matched literal bash/source calls with bare paths.
Orchestrators use shell variables like \$SCRIPT_DIR/../Transcodes/script.sh
and \$SCRIPTS_ROOT/Category/script.sh — the old pattern missed all of these.
New regex captures the Category/script.sh portion from either pattern,
requires at least one subdirectory (excludes load_config.sh and root-level
utilities), and validates each candidate against the filesystem.
transcode_management now correctly shows transcode_cleanup and transcode_manager.
Orchestrators using master.conf config arrays for dynamic dispatch still show
no children — correct, as those lists cannot be statically parsed.
/etc/cron.d/varaverk is RAM-based and wiped on every reboot. The
disks_mounted event fires early in boot (before Docker) with /boot
already readable, so schedule.json is available to restore the cron
file automatically without any user action.
/var/log/ is RAM-based on unRAID — wiped on reboot. Creating the log
directory in vv_cron_rebuild() ensures it exists before the first cron
job fires, without needing a separate boot script.
- Rename varaverk.page → Varaverk.page: unRAID nginx only routes URLs
starting with a capital letter (~^/[A-Z].*), lowercase caused 404
- Add require_once config.php to include/scheduler.php and include/docs.php:
both used SCRIPTS_DIR constant without including the file that defines it
- Replace hardcoded SCRIPTS_DIR with cfg-file-backed setting: reads from
/boot/config/plugins/varaverk/varaverk.cfg, falls back to default on first run
- Add Plugin Settings card to Config tab with scripts path field
- Add api/settings.php to write the cfg file (validates directory exists)
critical_sync_maintenance.sh runs every 30 minutes (*/30 cron).
Three references in README.md and Partnership/README-Partnership.md
still said 15 minutes from an earlier schedule.
transcode_management.sh runs every 7 minutes. Six references across
README.md, README-Transcoding.md, Manual-Transcoding.md, transcode_manager.sh,
and transcode_cleanup.sh still said 3 minutes from before the schedule change.
If PaddleOCR (or any GPU sidecar) holds all VRAM, --gpus will still
appear to fail even when configured correctly. Added explicit callout:
check nvidia-smi before concluding the GPU flag is wrong.
README-Transcoding now has a dedicated REQUIRED EXTRA PARAMETERS section
with both GPU and non-GPU variants and the Google warning up front.
Manual-Transcoding: two troubleshooting sections that showed mount-only
now show both GPU and non-GPU full strings. NVIDIA section quote style
fixed (--gpus "device=UUID" consistently).
Every search result shows --runtime=nvidia which conflicts with
bind-propagation=shared on unRAID 7.2.5+ and breaks on container
rebuilds. New prominent section explains the correct --gpus approach,
why the old method fails, and the VRAM starvation cascade.
Item #6 in Surprises updated to reference the new section; arr cleanup
renumbered to #8.
Required Mount section now shows full Extra Parameters for both GPU and
non-GPU containers and explains why --gpus beats --runtime=nvidia at
setup time rather than burying it in Troubleshooting.
New troubleshooting entry documents the GPU memory starvation cascade:
OCR sidecar (PaddleOCR) holds VRAM → Jellyfin hard-fails, Emby falls
back to CPU silently. Emby's NVIDIA startup probe is one-shot — VRAM
must be free before restarting or NVIDIA stays disabled for the session.
Includes correct recovery sequence and Tesseract fallback for EmbyCredits.
README callout updated from two to three non-obvious requirements.
Documents the known issue where an unclean shutdown leaves the --mount
bind-propagation entry in a broken state that prevents Emby from starting.
Recovery: remove mount → start → wait for load → re-add mount → restart.
Follows the same extraction pattern as partnership_offboard.sh.
partnership_manager.sh --transfer now exec-forwards to the new script.
Legacy inline block preserved under `if false` for reference.
Fixes in partnership_transfer.sh vs the original inline code:
- WebUI reconfiguration corrected: SSH to new owner (mirror) to set localhost;
reconfigure_local_webuis() on current server to point to new owner IP
- master.conf remote update uses scp push instead of self-SSH with wrong path
- SSH key uses $SSH_KEY directly (was using undefined {HOST*}_SSH_KEY expansion)
partnership_manager.sh:
- fix auto-offboard threshold: × 96 → × 48 (was triggering at 2× configured days)
- fix do_final_sync(): use CRITICAL_SYNC_SHARES if configured; fall back to hardcoded paths
- replace inline --check offboard blocks with background partnership_offboard.sh calls:
• owner path was missing remote container cleanup (steps 8+9) and blocklist
• mirror path was missing blocklist, SSH revocation, CRITICAL_RSYNC_ENABLED=false
• local grace_seconds outside a function (bash error) eliminated with the block removal
• 6hr Tailscale grace sleep was blocking the cron for the full grace window
partnership_offboard.sh:
- reorder owner path: state write moved from step 5 to step 10 (last) so re-running
after a mid-offboard crash restarts from scratch instead of exiting as INACTIVE;
state is now written before the Tailscale grace sleep so --check stops re-triggering
- add FolderView3 remote folder cleanup in step 7 (mirror's fallback folder persisted)
- track step 1+2 outcome with STEP_STOP_OK / STEP_SYNC_OK; summary now reflects actual
result instead of hardcoding ✅ regardless of dry-run or failure
- update header step list to match new step order
master.conf:
- reorder ORCHESTRATORS section by run frequency; add missing entries
- remove heartbeat vars (watchdogs no longer run in continuous cycle)
- remove dead REMOTE_DOCKER_RETRY_WAIT and display-only watchdog interval vars
- move INTERMEDIATE_RSYNC_ENABLED into RSYNC two-tier block; reorder by frequency
common.sh:
- add missing info() and success() definitions — both were called throughout but never defined
- update output tier description to list all 5 functions
- fix double-icon in check_local_disk_temps() error/warn calls
- fix smashed curl/ssh command lines in notify_emby_scan() and check_remote_docker_daemon()
- update check_rsync_enabled() comment to include INTERMEDIATE, ordered by frequency
emby_to_radarr_sync.sh / emby_to_sonarr_sync.sh:
- wire up RADARR_EMBY_LIBRARIES / SONARR_EMBY_LIBRARIES; empty array now scans all libraries
Gmer4Lfe