Commit Graph
410 Commits
Author SHA1 Message Date
Gmer4Lfe 4832dd8d4e scheduler: replace hint paragraph with compact 3-column legend grid 2026-05-24 21:56:27 -04:00
Gmer4Lfe 655c520ae8 scheduler: add Array Starting / Array Stopping event triggers
Two hardcoded event-triggered entries appear at the top of the scheduler
(array_started.sh / array_stopping.sh). They show an  Array Start /
 Array Stop badge instead of a cron field and are enabled/disabled via
the normal toggle.

Static event scripts fire them via Unraid's event system:
- event/disks_mounted/array_start_jobs  → runs in background (non-blocking)
- event/disks_unmounting/array_stop_jobs → runs foreground (blocks until done)

vv_cron_rebuild() skips @array_* entries so they never land in the cron file.
Scripts executed on each event are managed in master.conf via
ARRAY_START_SCRIPTS and ARRAY_STOP_SCRIPTS arrays.
2026-05-24 21:54:13 -04:00
Gmer4Lfe 1540c62d3a scheduler: smart auto-scroll replaces separate scroll lock checkbox; lidarr: allow single-seed candidates
Auto Scroll now pauses automatically when user scrolls up and resumes
when they reach the bottom — no separate Scroll Lock needed.

Lidarr discovery no longer hard-skips single-seed candidates; they
score low on breadth but can still reach threshold on merit.
2026-05-24 21:40:03 -04:00
Gmer4Lfe 253d5bf2e1 fix(varaverk): remove legacy /etc/cron.d/varaverk on cron rebuild
Before update_cron was wired in, the plugin wrote directly to
/etc/cron.d/varaverk. That file persisted after the migration and
caused duplicate job firing (two watchdog instances at :00/:15/:30/:45).
The second instance hit the lock, exited with warn, and overwrote the
stat file — killing the running indicator while the first was still live.

vv_cron_rebuild() now unlinks the legacy file after every update_cron
call. Since rebuild_cron (boot event) calls vv_cron_rebuild(), the
cleanup runs on every reboot and every schedule save.
2026-05-24 21:27:38 -04:00
Gmer4Lfe bcb1956bbe feat(varaverk): append logs per run with timestamp separator
run_job.sh was overwriting the log on every cron fire, making it
impossible to see run history. Switch to append mode, print a
── YYYY-MM-DD HH:MM:SS ── separator before each run, and trim to
the last 1000 lines after each run to keep logs bounded.
2026-05-24 21:12:28 -04:00
gmer4lfe 7a61a56ee1 fix(common): prepend /usr/local/sbin to PATH for cron compatibility
Cron runs with PATH=/usr/bin:/bin, which omits /usr/local/sbin where
tailscale (and other Unraid tools) live. resolve_remote_ip silently got
empty results from both tailscale calls, failing all 3 retry attempts
every scheduled run. Export the full path at source time so all scripts
find tailscale regardless of how they're invoked.
2026-05-24 21:08:39 -04:00
gmer4lfe fc1fd27a3f fix(common): lock registry prevents EXIT trap overwrite orphaning locks
Scripts calling acquire_lock followed by acquire_rsync_lock (rsync.sh)
or a custom EXIT trap (resource_watchdog.sh) would overwrite the lock
release trap, leaving the first lock file behind on exit.

Replace per-acquire trap with a _LOCK_FILES registry. _register_lock()
appends each lock file to the array and sets a single _release_all_locks
trap. All acquired locks release together on exit regardless of how many
traps are subsequently set or overwritten.

resource_watchdog.sh: chain _release_all_locks into both its custom trap
and the state-persisted disarm path so the lock releases in all exit paths.
2026-05-24 21:02:59 -04:00
Gmer4Lfe 207dead909 monitor: darken transcode stream bar to deep orange (#e65100) 2026-05-24 20:20:42 -04:00
Gmer4Lfe 2128539450 style(monitor): Live TV bar blue, transcode orange, direct green 2026-05-24 20:16:59 -04:00
Gmer4Lfe 0f6154ad82 style(monitor): stream progress bars match parity blue (#1e3a5a) 2026-05-24 20:15:46 -04:00
Gmer4Lfe 0757d07ddf fix(varaverk): move varaverk.plg to /boot/config/plugins/ root (matches all other plugins) 2026-05-24 20:10:21 -04:00
Gmer4Lfe 376c7f7bc8 feat(scheduler): persist invert preference in localStorage 2026-05-24 20:00:00 -04:00
Gmer4Lfe 51fe2664b8 feat(scheduler): scroll lock on by default, persisted in localStorage 2026-05-24 19:58:44 -04:00
gmer4lfe b754842da2 fix(common): retry Tailscale IP resolution up to 3 times before failing
The first cron run of critical_sync_maintenance failed because Tailscale
had a transient moment at startup. resolve_remote_ip() now retries 3 times
with a 5s delay before giving up, making it resilient to brief daemon
hiccups without masking real peer-offline conditions.
2026-05-24 18:56:59 -04:00
Gmer4Lfe 7c14627bc3 feat(scheduler): live running indicators for cron-triggered jobs
Add api/status.php — reads all scheduled jobs' stat files and returns a
{id: status} map. The scheduler page polls it every 3s and lights up the
dot on any job whose stat shows status=running, clearing it when done.

Replace single vvRunningId with vvRunningSet (Set) so multiple jobs can
show as running simultaneously. vvSetDot/vvClearDot handle per-job dots
and the header dot for the active job. Remove stale-count detection from
vvFetchRight — status poll is now the single source of truth.
2026-05-24 18:46:45 -04:00
Gmer4Lfe 6600565dd5 feat(varaverk): add local .plg registration; remove dev-mode cron fallback
Create varaverk.plg on flash (/boot/config/plugins/varaverk/varaverk.plg).
On boot, rebuild_cron copies it to /var/log/plugins/ so update_cron natively
picks up varaverk.cron and merges it into /etc/cron.d/root.

With the .plg in place the dev-mode direct-inject fallback in vv_cron_rebuild()
is no longer needed and has been removed.
2026-05-24 18:38:43 -04:00
Gmer4Lfe 03e72b6a06 fix(scheduler): use Unraid plugin cron mechanism instead of /etc/cron.d/varaverk
dcron uses the filename in /etc/cron.d/ as the username, so a file named
'varaverk' ran jobs as a non-existent user and was silently ignored.

Switch to writing /boot/config/plugins/varaverk/varaverk.cron and calling
update_cron (which merges all plugin .cron files into /etc/cron.d/root).

In dev mode (no varaverk.plg registered), fall back to injecting our entries
directly into the live root crontab via crontab -c /etc/cron.d -.
2026-05-24 18:11:50 -04:00
Gmer4Lfe 57d55c647c varaverk: replace User Scripts dependency with native job tracking
run_job.sh: new wrapper called by scheduler cron for every job.
  Writes /var/log/varaverk/<id>.json (status/start/end/exit/pid)
  and /var/log/varaverk/<id>.log (latest run output, overwritten).
  Maps exit 0→ok, 1→warn, 2+→error so watchdog "took action" runs
  show warn not error.

scheduler: cron now calls bash run_job.sh instead of inline bash -c.
  Added vv_job_stat_path(). LOG_DIR moved to config.php (shared).

monitor: vv_scripts_status() now reads /var/log/varaverk/*.json and
  */*.json — no User Scripts tmpScripts dependency at all.
  Stale running detection via /proc/<pid> check.
  Returns error_count in addition to running/ok/warn counts.

pages/monitor.php: added error pill (red ✗), error icon/color in
  list, duration shown next to age for each script entry.
2026-05-24 17:51:31 -04:00
Gmer4Lfe ea9f39a803 watchdogs: make healthy-state outputs always visible
log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
2026-05-24 17:25:33 -04:00
Gmer4Lfe afd9129869 watchdogs: add echo banners to network/storage/stability
All watchdog scripts now open with a consistent echo banner:
  ━━━ <icon> <Name> Watchdog — <timestamp> ━━━

Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.

Also: network_watchdog now logs "Network healthy " on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
2026-05-24 17:20:55 -04:00
Gmer4Lfe b78fdddb04 fix(watchdogs): Tailscale false positive + system watchdog banner
network_watchdog: Tailscale JSON uses "BackendState": "Running" (with space)
but grep pattern matched "BackendState":"Running" (no space) — always failed.
Switched to grep -qE with \s* to handle both forms.

system_watchdog: add echo banner at run start so its output is visually
separated from docker_watchdog in orchestrator logs. Previously appeared
as a continuation of the docker section since docker_watchdog has a banner
but system_watchdog did not.
2026-05-24 17:11:54 -04:00
Gmer4Lfe 760fb4a3e2 varaverk: monitor tab overhaul — grid layout, dynamic thresholds, new cards
Layout:
- Switch from three flex rows to single CSS grid (repeat(8,1fr)) — eliminates
  gap-count discrepancy between rows; span 1/2/4/8 cards are now pixel-identical
  across all rows regardless of how many cards share the row

New cards:
- Scripts card: last-run status (ok/warn/running/unknown), clickable filter pills,
  7-row scroll, flex:1 matching Power/Parity
- Streams card: live per-second progress counters, color-coded play buttons
  (blue=live, orange=transcoding, green=direct, yellow=paused), server badges
  with counts, left-fill column layout (3 per col)
- Transcode card: NVMe indicator, active session count + scrollable list,
  session rows with server icon + title + type + method

Dynamic thresholds:
- vv_disk_thresholds() reads /boot/config/plugins/dynamix/dynamix.cfg
- Disk utilization bars: green/orange/red based on warning/critical settings
- Temperature colors: hot/max for HDD, hotssd/maxssd for SSD/NVMe
- Thresholds update every poll cycle; changing Unraid settings takes effect automatically

Other:
- SSD path detection uses findmnt FSTYPE to exclude tmpfs/ramfs (fixes RAM
  being reported as SSD transcode target)
- varaverk.css h3: add white-space:normal + overflow:hidden to neutralise
  any Unraid global h3 styles that could force card width
2026-05-24 16:56:50 -04:00
Gmer4Lfe f85493cf35 watchdogs: fix Tailscale error propagation and docker watchdog output order
- network_watchdog: remove ISSUES++ for Tailscale — it's notify-only by
  design, no restart possible; cascading to system watchdog exit 1 was noise
- docker_watchdog: move banner to top of cycle so body errors appear under
  the correct section header, not visually inside Resource Manager output
2026-05-24 16:56:32 -04:00
Gmer4Lfe 359b051e64 varaverk: System card (identity/time/uptime/buttons); merge Fallback+Partner into one card 2026-05-24 12:00:09 -04:00
Gmer4Lfe 725c371f21 varaverk: remove GPU history chart 2026-05-24 11:53:54 -04:00
Gmer4Lfe 6be9873f49 varaverk: fallback and partner cards — 250px min-width, matched flex:1 2026-05-24 11:50:51 -04:00
Gmer4Lfe 36831f0693 Revert "varaverk: top row cards equal width (flex:1 across all five)"
This reverts commit f35a0f2dd4.
2026-05-24 11:49:30 -04:00
Gmer4Lfe f35a0f2dd4 varaverk: top row cards equal width (flex:1 across all five) 2026-05-24 11:48:50 -04:00
Gmer4Lfe cead4eb46f varaverk: redesign Fallback card — host online/offline list, state line, active fallback containers with image 2026-05-24 11:47:08 -04:00
Gmer4Lfe 6bb44cae8b varaverk: Docker memory — use docker stats sum instead of raw cgroup (matches Unraid dashboard) 2026-05-24 11:41:59 -04:00
Gmer4Lfe 9e236e3f52 varaverk: fix Docker memory — use /sys/fs/cgroup/docker/memory.current (Unraid cgv2 path) 2026-05-24 11:39:41 -04:00
Gmer4Lfe c17be8d017 varaverk: add Partner card — known hosts, owner tags, Tailscale online status 2026-05-24 11:36:23 -04:00
Gmer4Lfe cbac3949e0 varaverk: network canvas 100→110px 2026-05-24 11:30:15 -04:00
Gmer4Lfe 3c26f8d767 varaverk: memory top procs — top 3, inline right of total on same line 2026-05-24 11:28:51 -04:00
Gmer4Lfe 397ec405a0 varaverk: memory card — top 5 procs by RSS as horizontal strip under total 2026-05-24 11:25:27 -04:00
Gmer4Lfe 475cb3e75c varaverk: memory card — top 3 usages by size in top-right corner 2026-05-24 11:21:43 -04:00
Gmer4Lfe 2cd0a8e66e varaverk: inline RX/TX legend with speed numbers; drop separate legend row 2026-05-24 11:20:11 -04:00
Gmer4Lfe 50614b9cbe varaverk: network canvas 52→100px 2026-05-24 11:16:49 -04:00
Gmer4Lfe 423a8e3e0a varaverk: CPU canvas 36→60px; add GPU history chart at 60px 2026-05-24 11:15:31 -04:00
Gmer4Lfe 496d70c137 Revert "varaverk: stretch CPU and GPU graphs to card bottom; add GPU history chart"
This reverts commit 660c4762f2.
2026-05-24 11:12:55 -04:00
Gmer4Lfe 660c4762f2 varaverk: stretch CPU and GPU graphs to card bottom; add GPU history chart
- CPU/GPU cards: flex-column so body div takes remaining height
- CPU canvas: flex:1 + min-height:36px instead of fixed 36px
- GPU: add vvGpuHistory + canvas (purple, same style as CPU); canvas flex:1
  stretches to fill whatever space the row's tallest card leaves over
2026-05-24 11:11:04 -04:00
Gmer4Lfe 4a22f3af66 varaverk: fix ext IP and Tailscale IP in network card
- External IP: switch from DDNS domain lookup to curl ifconfig.me with 5-min
  cache; DDNS approach broke because host detection uses HOST1_NAME but conf
  has HOST1, so domain was never resolved
- Tailscale IP: switch from hardcoded tailscale0 iface to `tailscale ip -4`;
  interface name varies (tailscale0, tailscale1) across kernels/installs
2026-05-24 11:07:15 -04:00
Gmer4Lfe b2e937526f varaverk: add media, confform, script API/include files (from previous session)
media.php: Emby/Jellyfin/Plex session poller for Streams card
confform.php: conf file form helpers for Config tab
script.php: per-script run/log API endpoint for Scheduler tab
2026-05-24 10:59:13 -04:00
Gmer4Lfe 279ac156c2 varaverk: fix monitor API — include config.php so vv_conf_vars/detect_host resolve
vv_network_stats() calls vv_conf_vars() and vv_detect_host() which are defined in
config.php. monitor.php had no require for it, causing a PHP fatal error on every
poll → empty JSON → all monitor cards stuck on "Loading...".
2026-05-24 10:55:13 -04:00
Gmer4Lfe ab3e28531e varaverk: monitor overhaul — CPU/memory/network/streams cards; scheduler polish
Monitor tab:
- CPU card: per-core bars (freq-colored), overall usage bar, rolling history chart
- Memory card: breakdown by system/VM/ZFS/Docker/free with progress bars
- Network card: LAN + ext + Tailscale IP display, live RX/TX chart with auto-scale
- Streams card: Emby/Jellyfin/Plex sessions with playback bar and server badges
- GPU card: adds power draw, encode%, decode% meters
- Transcode card: ramdisk vs SSD location pill, session count, flip history
- Canvas chart helpers: vvDrawChart, vvDrawNetChart, vvMeter, vvFmtBps, vvFmtGib

api/monitor.php: adds cpu, mem, net to response payload
2026-05-24 10:52:42 -04:00
Gmer4Lfe f7bef55e51 varaverk: green checkmark feedback on schedule save
- Add .vv-save-check span to each job row (orchestrators + children)
- CSS: fade-in-then-out animation (visible 1.2s, fades over 2s)
- vvFlashSaved(job): triggers the checkmark animation on a specific row
- vvFlashStatus(el, msg, ok): shows timed status text (was called but missing)
- vvSaveJob: now shows per-job checkmark on successful save
- vvSaveAll: shows checkmark on each job as its response arrives, plus
  global "✓ All saved" / "✗ Some failed" status text below the button
2026-05-23 21:07:15 -04:00
Gmer4Lfe 8383306831 varaverk: fix cron format and permissions for dcron
Unraid's system crond is dcron (dillon's cron daemon 4.5), which reads
/etc/cron.d/ for system crontabs WITHOUT a username field, and refuses
to execute files that are group- or world-writable.

vv_cron_rebuild() was writing "SCHED root bash ..." (with username field)
and leaving the file world-writable (666), so dcron silently ignored every
entry. Fixed:
  - Remove CRON_USER from the generated cron line format
  - chmod 0600 the cron file after writing it
2026-05-23 21:03:52 -04:00
Gmer4Lfe 0e13d8772d user_script_plug-in.sh: v2.6 full audit and update
- Remove ARRS FAILED/STALLED RECOVERY as standalone orchestrator — now in
  INTERMEDIATE_MAINTENANCE_SCRIPTS (called every 4 hours by intermediate_sync)
- Remove WEBGUI RESTART standalone block — webgui_watchdog.sh now runs via
  SYSTEM_WATCHDOG_SCRIPTS chain inside system_watchdog.sh (every minute)
- Remove CLEAR LOGS standalone block — now in WEEKLY_MAINTENANCE_SCRIPTS
  (called as part of Sunday weekly_sync_maintenance window)
- Remove SUNDAY MONITOR BLOCK individual cron entries — all monitors now run
  via COFFEE_REPORT_SCRIPTS inside sunday_morning_coffee_report.sh
- Add MONTHLY MAINTENANCE orchestrator block (0 3 * * * — self-gated to monthly)
- Update WATCHDOG ORCHESTRATOR to document SYSTEM_WATCHDOG_SCRIPTS chain
- Update INTERMEDIATE SYNC to document arrs_failed in INTERMEDIATE_MAINTENANCE_SCRIPTS
- Update WEEKLY SYNC to document playback_aware_*.sh in WEEKLY_MAINTENANCE_SCRIPTS
- Fix INITIAL RUN paths: Initial_run/ → Partnership/
- Add individual entries: storage_watchdog, network_watchdog, playback_aware_lidarr/
  radarr/sonarr, mesh_monitor, ramdisk_stop, emby_to_lidarr/radarr/sonarr,
  gitea_ssh_setup, partnership_offboard, partnership_transfer
- Update continuous_scripts_status path to Old_Arch_Still_Works/ with deprecation note
- Rewrite QUICK REFERENCE to match actual current schedule
2026-05-23 20:57:47 -04:00
Gmer4Lfe f6cb861f36 varaverk: suggestions panel color + parser regex fix
- Change suggestion title color from bright green (#6fcf97) to soft grey-blue (#7a9eb5)
- Fix vv_parse_user_script_template() block-detection regex — $ end-anchor
  failed on lines ending with multibyte U+2500 ─ chars; removed anchor so
  parser now returns all 22 blocks from user_script_plug-in.sh
2026-05-23 20:47:20 -04:00
Gmer4Lfe a1745a575b Scheduler UI polish + per-script verbose logging + coffee report as orchestrator
- Cron and Advanced button vertically aligned in their respective rows
- Verbose checkbox next to Log button — saves immediately, persists to schedule.json
- --log flag injected into cron lines, manual Run, and Dry Run when verbose enabled
- Coffee report rewritten as lean orchestrator over COFFEE_REPORT_SCRIPTS array
- COFFEE_REPORT_SCRIPTS added to master.conf (7 Sunday monitor scripts)
- Scheduler Advanced tab auto-discovers children via existing array detection
2026-05-23 20:05:26 -04:00