Compare commits

...
90 Commits
Author SHA1 Message Date
Gmer4Lfe 5fa7750046 Sonarr and Lidarr had the same deadlock and the same missing confirmation as Radarr, so give them the same budget and strikes 2026-08-26 21:20:05 -04:00
Gmer4Lfe 1838eed855 Give AI_ASSIST_CLEANUP the consumer it has never had: it describes the shape of a classification and decides nothing, so switching it off changes no deletion 2026-08-26 18:03:17 -04:00
Gmer4Lfe 96d8a5e3f0 Make a file earn its deletion over consecutive runs, so a partial classification failure too small to trip the tracked-count gate cannot remove anything 2026-08-26 17:58:42 -04:00
Gmer4Lfe 0431e720de A cap that aborts cannot drain a backlog bigger than itself, so make it a per-run budget and let the queue clear over consecutive nights 2026-08-26 17:55:21 -04:00
Gmer4Lfe 7f4921de49 Give the wizard an assistant scoped to the step you are actually stuck on, so a first-run question can say "this" and mean something 2026-08-26 17:31:52 -04:00
Gmer4Lfe d4c19baa32 Folding the Scheduler dock into the shared chat left its scope behind, so the troubleshooter has been diagnosing without the log tail for whatever was open 2026-08-26 17:25:28 -04:00
Gmer4Lfe d444fd8036 A path with an apostrophe broke the remote shell quoting and a big file outran the connect timeout, so intact backups were reported corrupt and absent 2026-08-25 21:30:57 -04:00
Gmer4Lfe d42b1e2dda ffprobe exit 0 with a chapter-track warning is not corruption; guard the delete path against a detector that is wrong at scale 2026-08-25 18:29:43 -04:00
Gmer4Lfe 9492dc4c39 dig names the resolver it could not reach in its error text, so a DNS timeout was scraped as the answer and restarted DDNS over nothing 2026-08-25 18:29:43 -04:00
Gmer4Lfe 172beca3c5 A state key is a file path, not a regex — a release tag like [Bluray-1080p] holds an invalid range, so grep bailed and the tempfile swap wiped every other entry 2026-08-25 18:29:43 -04:00
Gmer4Lfe f0c1289519 Let the file own why the row cap is conditional; the README states that it is 2026-08-25 17:21:20 -04:00
Gmer4Lfe 128172d3a8 The row counts below four columns were the naive division, not what the board does 2026-08-25 17:21:20 -04:00
Gmer4Lfe 820e8325a6 Docs: the row-height cap is conditional now, and the board's narrow-screen behaviour was nowhere in the page help 2026-08-25 17:15:20 -04:00
Gmer4Lfe 74c6a0f5eb Monitor: hold the System card to the Network card's height below eight columns, where the two stop sharing a row 2026-08-25 17:07:18 -04:00
Gmer4Lfe d289a9101c Monitor: cap the System card's Varaverk block so the host figures set the card's height, not the sum of both 2026-08-25 17:01:52 -04:00
Gmer4Lfe 9553ecb16a The two functions the row-overflow hatch added were not in EXPORTS 2026-08-25 17:01:10 -04:00
Gmer4Lfe fb104bf05a Monitor: a rung taller than one screen sizes rows to content, since the page scrolls there anyway and the cap only clipped cards 2026-08-25 16:54:19 -04:00
Gmer4Lfe f4adc31215 Close the last four headers, and say what the destructive tools refuse to do 2026-08-25 16:51:14 -04:00
Gmer4Lfe c6254f2342 The dev checkers documented themselves under headings nothing else uses 2026-08-25 16:45:39 -04:00
Gmer4Lfe d25b147a56 A CLI tool has runtime modes, not a request 2026-08-25 16:42:39 -04:00
Gmer4Lfe 1f9915f074 Fill the single missing section in three scripts that had everything else 2026-08-25 16:41:06 -04:00
Gmer4Lfe c0e7a97a56 Document what the cert and auth sweeps will not do 2026-08-25 16:39:01 -04:00
Gmer4Lfe 68c835f33b Document what the proxy and cert collectors guarantee 2026-08-25 16:37:25 -04:00
Gmer4Lfe 6b9fda76db Say how the spawned tools are actually invoked 2026-08-25 16:34:02 -04:00
Gmer4Lfe 1be259b66d Two AI includes documented themselves under names nothing else uses 2026-08-25 16:31:43 -04:00
Gmer4Lfe 2c3c88d067 Say what the include layer exports and what it reads from conf 2026-08-25 16:29:57 -04:00
Gmer4Lfe 3fc14d16bf A worker that is spawned has arguments, not runtime modes 2026-08-25 16:25:55 -04:00
Gmer4Lfe 4cc50889d6 Document the api layer's operational models, and fix two headers that had gone stale against the code 2026-08-25 16:25:55 -04:00
Gmer4Lfe 63e68740b6 Put the header sections back in the order the other 116 scripts use 2026-08-25 16:19:19 -04:00
Gmer4Lfe f2fddeb87d The reboot path's container stops were unbounded, and stopped a different list than it restored 2026-08-24 20:22:55 -04:00
Gmer4Lfe eda411c0be Give the PHP layer somewhere to record what it swallowed 2026-08-24 18:51:36 -04:00
Gmer4Lfe 492757ce70 Walk appdata once for the space check, and refuse to export when it cannot be measured 2026-08-24 18:46:26 -04:00
Gmer4Lfe b58282b3d6 Read a filesystem once where two probes were answering the same question 2026-08-24 18:46:26 -04:00
Gmer4Lfe 9da2760a42 A localStorage guard was covering the layout restore it wrapped 2026-08-24 18:42:34 -04:00
Gmer4Lfe 049f633667 Bound the watchdogs' docker calls — a hung daemon is what they exist to catch 2026-08-24 18:39:59 -04:00
Gmer4Lfe 84946ed0c6 clear_logs previewed on one size basis and cleared on another 2026-08-24 18:36:29 -04:00
Gmer4Lfe 514e13660c Give the bash side the disk helpers it never had, reporting the same unit as vv_df 2026-08-24 18:36:29 -04:00
Gmer4Lfe 9333334b7b Bound the two tailscale lookups that were missed when the others were 2026-08-24 18:33:40 -04:00
Gmer4Lfe 367d0203b8 Find guards wired onto a helper that can never return non-zero 2026-08-24 18:22:59 -04:00
Gmer4Lfe 37017882c7 Check the header standard now that it is written down rather than implied 2026-08-24 18:22:59 -04:00
Gmer4Lfe f116a1584e Offboard reported a stack cleanup that could not fail and, on one path, had not run 2026-08-24 18:22:59 -04:00
Gmer4Lfe 1a5bacce87 Point the webhook's dependency note at where the handler actually lives 2026-08-24 18:22:59 -04:00
Gmer4Lfe f6e5f7dcd6 Drop the remote-metrics endpoint nothing called and api/monitor.php already served 2026-08-24 18:22:59 -04:00
Gmer4Lfe 0f0c407458 Arrange and folder saves reported failure on every success, and split a move across two writes 2026-08-24 18:22:59 -04:00
Gmer4Lfe 354992c9e7 Keep a script's arguments and trailing comment when it moves between orchestrators 2026-08-24 18:22:59 -04:00
Gmer4Lfe 28987240aa Let a sync window take a whole share in one click
The window editor is path-first, which is exact but slow for the case that is nearly all of them.
The chips only offer whole shares; a subpath already in the window marks its share partial rather
than offering to widen it, and the path box stays the only way to express a subpath or a profile.
2026-08-23 17:16:24 -04:00
Gmer4Lfe fbf2b92167 Drop containers that no longer exist from the conf lists, after three days absent
A removed container leaves its name behind in every list that named it, and those names go on
being acted upon — a watchdog waiting for a container docker has never heard of. Strikes rather
than a single miss, because a rebuild makes a container vanish for a moment; and the lists naming
containers on the REMOTE are excluded, since pruning those against local docker would empty them.
2026-08-23 16:54:53 -04:00
Gmer4Lfe 206a119a4b Show each container's fallback tier on the Monitor board
The row's left border carries it rather than the status dot, which already means running or
stopped.
2026-08-23 16:38:58 -04:00
Gmer4Lfe 671e7ea5a4 Make fallback coverage something that can actually happen, and say so on the page
fallback.sh starts covered containers with docker start and never creates them, so a coverage list
the partner has never been sent is a promise nothing can keep — all twelve were missing. Adds the
push and remove paths, a readiness card that checks rather than infers, and the fallback state the
assistant needs to answer for it.
2026-08-23 16:38:58 -04:00
Gmer4Lfe d5cf3db2ec Close every orchestrator the same way, and make skipped work a visible outcome
A gated-off section left nothing failed, so the weekly could run for hours and report "all
complete" beside "0 shares synced"; skipped is now derived from what was expected rather than
self-reported, and the verdict degrades to PARTIAL instead of flattering.
2026-08-23 16:38:58 -04:00
Gmer4Lfe dc8823724d Let the GPU transform actually run when deploying to a partner
The detection pattern starts with --, so grep parsed it as an option and exited 2; the negation
made the early return always true and every XML was deployed untransformed, keeping the owner's
NVIDIA device UUID on hosts that have no such card.
2026-08-23 16:38:58 -04:00
Gmer4Lfe a392108562 Warn before a transfer when a disk backing the remote share is nearly full
Online is not writable: check_remote_disks confirmed each backing disk was mounted and healthy but
never how full it was, and rsync answers a full destination by stalling rather than failing.
2026-08-23 10:15:54 -04:00
Gmer4Lfe fd1ab58598 Declare the Monitor board in one place, so a card's width and position stop living in eighteen inline styles and four media queries 2026-08-23 00:44:30 -04:00
Gmer4Lfe bd13bdc6e1 Revert the Monitor column ladder for now; the layout work is paused, not abandoned 2026-08-22 23:53:27 -04:00
Gmer4Lfe bdf1bce4be Place the storage and array cards by span, not by column index, so they follow the grid instead of forcing it back to eight 2026-08-22 23:50:13 -04:00
Gmer4Lfe c5c0477653 Derive Monitor's column count from what the cards actually need, so no width forces a sideways scroll 2026-08-22 23:45:02 -04:00
Gmer4Lfe 76faf0f9f8 Let an installation add its own tabs without the repo having to name them 2026-08-22 16:37:37 -04:00
Gmer4Lfe 879109e55d Drain the orphan backlog safest-first instead of aborting, so a run over budget still makes progress 2026-08-22 16:37:31 -04:00
Gmer4Lfe 61b7d2a96a Partnership stacks are the owner's declaration, so a mirror must read them from the owner's conf 2026-08-22 14:03:57 -04:00
Gmer4Lfe 15d802ae3d Preview mode that survives a reboot, and a fallback log that survives one too 2026-08-22 13:52:34 -04:00
Gmer4Lfe ac17be2cd9 Say fallback, not failover — the synonym was silently matching nothing in three renderers, plus retrying probes and a 2-minute handback window 2026-08-22 13:43:30 -04:00
Gmer4Lfe f1ba418aa8 A dry run walked the failover branches and reached notify(), which had no dry-run gate 2026-08-22 09:53:53 -04:00
Gmer4Lfe b17cc3a09d Say on each node card whose containers the tiers are, since they are the other host's 2026-08-22 09:49:43 -04:00
Gmer4Lfe 1c38abe1d0 Fallback coverage: colour the wait, group by tier, and lock the stacks the partner already runs 2026-08-22 09:33:06 -04:00
Gmer4Lfe 5d42a28d34 Each assistant card remembers its own conversation instead of every card resuming the newest one 2026-08-22 08:46:35 -04:00
Gmer4Lfe e5f4ede3f2 Coverage and settings above the chat, tier delays and the probe target alongside them, coverage in three columns 2026-08-22 01:43:07 -04:00
Gmer4Lfe ce613bcf9d Rehome the failover coverage picker on the Fallback tab, five days after it was dropped from Partnership for belonging here 2026-08-22 01:32:08 -04:00
Gmer4Lfe 8ce8f0dfbe The Monitor fallback card described a state file without saying whether anything still writes it 2026-08-22 01:11:00 -04:00
Gmer4Lfe 13c221244e A partner's conf is in the RAM cache, not CONF_DIR, and a commented-out template entry is not configuration 2026-08-22 01:06:24 -04:00
Gmer4Lfe a89b704fa1 Watchdog and Monitor looked the partner up by exact key, and the remote probe passed a quote to cut as a filename 2026-08-22 00:59:24 -04:00
Gmer4Lfe 0f228020ef State-file age is time since the last transition, not staleness, and a preview's state is the one being decided 2026-08-22 00:52:38 -04:00
Gmer4Lfe 8454d3d0a2 A dry run does not refresh live state, so say that rather than showing its age as stale 2026-08-22 00:48:16 -04:00
Gmer4Lfe da15cc7531 Rebuild the Fallback tab around what the daemon is actually doing, not what its last state file said 2026-08-22 00:44:35 -04:00
Gmer4Lfe 5924aae1c0 Fallback read the peer list by exact key, so a partner whose tailnet name differs by one character rendered as UNREACHABLE 2026-08-22 00:34:33 -04:00
Gmer4Lfe 46239012b4 The dry-run cleanup trap replaced the lock-release trap and orphaned fallback.lock 2026-08-22 00:24:06 -04:00
Gmer4Lfe f8119f3786 Give the test harness a stop that cannot strand its iptables rule, and clean up the dry-run state copy 2026-08-22 00:20:40 -04:00
Gmer4Lfe 41c0598220 A fallback dry run wrote real reboot-surviving state, and a failed tier writeback reported itself as a dry run 2026-08-22 00:00:56 -04:00
Gmer4Lfe 387382ec9f Media Stack called every node local, and a host whose watchdogs never ran reported all clear 2026-08-21 22:50:09 -04:00
Gmer4Lfe 20998c8c31 The Streams scope was reset to this host on every page load, on a dashboard that is never reloaded by choice 2026-08-21 21:58:24 -04:00
Gmer4Lfe f5d6edd9d0 The mirrored folder layout never reached the container card, which reads Varaverk's own store and not folder.view3's 2026-08-21 21:50:22 -04:00
Gmer4Lfe 4808a57289 Monitor's partner card read phase flags that only ever describe the mirror, so the owner rendered as not provisioned on the mirror 2026-08-21 21:43:00 -04:00
Gmer4Lfe 96b907acbc Say what Varaverk itself is using on the System card
The card described the machine and said nothing about the thing whose dashboard
it is. Cache size matters most: VV_CACHE_ROOT is under /tmp, which on Unraid is
RAM, so it is shown against the rootfs percentage it counts against. Jobs are
locks with a live process — four lock files were sitting there and one job was
actually running.
2026-08-21 09:39:52 -04:00
Gmer4Lfe d9c37ac763 Add a checker for the two JS faults this plugin has actually shipped
A cross-scope identifier and a fetch chain ending in an empty catch both pass
php -l and node --check, and together they turned a ReferenceError into what
looked like a slow load for hours. --self-test asserts both detectors still
find a known fault, because a checker that silently stops working reports a
confident zero.
2026-08-21 09:32:04 -04:00
Gmer4Lfe 146be2d0ad Report every failed fetch instead of discarding it
Twenty-four fetch chains ended in an empty catch, which is not error handling
but error deletion: the request fails, nothing renders, nothing is logged, and
the surface sits on Loading forever. That is how the mesh chat's ReferenceError
read as a slow load for as long as it did.
2026-08-21 09:23:51 -04:00
Gmer4Lfe 906375dc23 Share the font map with the function that renders it
_vvNcStyle referenced a const declared inside vvNcRender, so with the
formatting toggle on every mesh transcript threw ReferenceError and drew
nothing. The plain-render early return skipped the line, which is why it looked
like a slow load rather than a broken one.
2026-08-21 09:05:21 -04:00
Gmer4Lfe e1cdf03d40 Say why mesh chat failed instead of retrying a minute later
Every fetch ended in an empty catch, so a failed channel list was
indistinguishable from a slow one: nothing rendered, nothing was logged, and
the next attempt was a backoff tick away. Errors now land in the card's status
line and the console, a failure retries in two seconds, and opening the pane
before the list has arrived asks for it rather than waiting.
2026-08-21 09:00:19 -04:00
Gmer4Lfe 8041da5424 Stop starting Unraid's Node CLI on every Partnership render
`unraid-api apikey` costs 1.98s and ran once per render just to answer whether
a key exists. Cached for a minute — still read from the key store, so a
deletion still shows, a minute later rather than instantly. Partner probes are
multiplexed at the same time.
2026-08-21 08:53:29 -04:00
Gmer4Lfe 7c4f881b57 Render an answer in full instead of folding it behind a control
The fold saved a scroll and cost a click on every long answer, and what it hid
was usually the part worth reading. Follow already handles holding your place
while an answer arrives.
2026-08-21 08:44:08 -04:00
Gmer4Lfe 81abc5adee Add a mesh scope to the Streams card
Local stays the default and the cheap path; mesh asks each partner for its own
sessions over SSH, live rather than cached, because a stream is true for
minutes and a cached one would be confidently wrong about the only thing the
card exists to say.
2026-08-21 08:38:37 -04:00
Gmer4Lfe 331b7b6b13 Resolve a peer from tailscale status before falling back to DNS
MagicDNS does not work on this mesh, so `tailscale ip -4` missed and blocked on
a system DNS lookup until it timed out: 5.04s per call against 0.010s for the
status parse, paid by every mesh operation on both hosts. Exact match first, so
no precision is traded for the speed.
2026-08-21 08:38:37 -04:00
115 changed files with 7292 additions and 982 deletions
+12
View File
@@ -43,3 +43,15 @@ Plugin/dist/
*.swp
*~
.vscode/
# ── Local-only plugin surfaces (per-installation, never pushed) ───────────────
# Varaverk.page discovers pages/local/*.php and registers each as a tab; api/local/ holds their
# endpoints. Both are symlinks into a store outside this repo, so what they contain belongs to
# one installation and is not part of the project. The tracked loader is deliberately generic —
# it names no page — so the public mirror never learns what a given server runs here.
#
# No trailing slash on either pattern. These paths are symlinks, not directories, and git treats
# a symlink as a blob — a "dir/" pattern does not match one, so the entries sat untracked rather
# than ignored, which is the same near-miss the *.bak rule above documents.
Plugin/unraid/pages/local
Plugin/unraid/api/local
+25
View File
@@ -30,6 +30,31 @@
# fetches its own data controls its own freshness rather than depending on the partner's cron.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# A missing file means unknown, never zero.
# The whole point of the tab reporting "not collected here" is that it is a different claim
# from "this partner spent nothing". If a partner is dark, unreachable or has never run a turn,
# there is simply no cache file, and the reader is expected to say so rather than render a 0
# that looks like a measurement.
#
# The reader pulls; nobody pushes.
# conf_sync.sh pushes as well, because a partner that cannot reach us still needs our
# credentials. Nothing here is needed by anyone else, so a host that wants fleet totals fetches
# them and owns its own freshness instead of depending on someone else's cron having run.
#
# RAM, not flash.
# The cache lands in tmpfs. It is a copy of a file that already exists on the partner and is
# rebuilt on the next pass, so writing it to flash would cost wear for something that is never
# worth surviving a reboot.
#
# Same shape as conf_sync.sh, deliberately.
# Resolve over Tailscale, scp one small file into a tmpfs cache, let a missing file mean
# unknown. A second transport pattern for a second small file would be a second set of
# failure modes to learn.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
+120 -8
View File
@@ -273,6 +273,9 @@ touch "$CORRUPTION_SCAN_STATE_FILE"
CORRUPTION_SCAN_STRIKES_FILE="${CORRUPTION_SCAN_STRIKES_FILE:-$DATA_DIR/corruption_scan_strikes.tsv}"
CORRUPTION_SCAN_STRIKE_LIMIT="${CORRUPTION_SCAN_STRIKE_LIMIT:-2}"
CORRUPTION_SCAN_MAX_CORRUPT_PCT="${CORRUPTION_SCAN_MAX_CORRUPT_PCT:-10}"
CORRUPTION_SCAN_MAX_CONSECUTIVE="${CORRUPTION_SCAN_MAX_CONSECUTIVE:-15}"
CORRUPTION_SCAN_GUARD_MIN_SCANNED="${CORRUPTION_SCAN_GUARD_MIN_SCANNED:-20}"
mkdir -p "$(dirname "$CORRUPTION_SCAN_STRIKES_FILE")"
touch "$CORRUPTION_SCAN_STRIKES_FILE"
@@ -301,6 +304,19 @@ reset_scan_strikes() {
[[ -n "$current" && "$current" != "0" ]] && set_scan_strikes "$1" 0
}
# Bails out of the whole run without committing anything. Safe to call at any point before
# the commit phase: strikes are queued in memory until then, so an abort leaves the strike
# file exactly as the previous run left it and deletes nothing.
abort_scan() {
local why="$1"
error "Corruption scan ABORTED — $why"
error "No strikes recorded and nothing remediated this run — the library was not trusted."
[[ -n "${FRESH_CLEAN_TMP:-}" ]] && rm -f "$FRESH_CLEAN_TMP"
notify "Corruption scan aborted on $(hostname) ($MY_ID) — $why. Nothing deleted." \
"Arr Corruption Scan" "warning"
exit 1
}
# Per-arr API shape differences — everything else in the scan/strike/remediate loop below is
# identical between Sonarr and Radarr.
declare -A ARR_FILE_ENDPOINT=( [sonarr]="episodefile" [radarr]="moviefile" )
@@ -322,6 +338,8 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo "$ICON_GEAR State file: $CORRUPTION_SCAN_STATE_FILE"
echo "$ICON_GEAR Strike limit: $CORRUPTION_SCAN_STRIKE_LIMIT"
echo "$ICON_GEAR Remediate: $REMEDIATE"
echo "$ICON_GEAR Corrupt ceiling: ${CORRUPTION_SCAN_MAX_CORRUPT_PCT}% of scanned (min ${CORRUPTION_SCAN_GUARD_MIN_SCANNED} scanned)"
echo "$ICON_GEAR Consecutive trip: $CORRUPTION_SCAN_MAX_CONSECUTIVE"
echo "$ICON_GEAR Scan limit: ${SCAN_LIMIT:-unlimited} (per arr)"
echo "$ICON_GEAR Path filter: ${PATH_FILTER:-none}"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
@@ -357,13 +375,36 @@ ffprobe_translate_path() {
# Probes one file. Echoes "clean" or "corrupt:<reason>". Never trusts a truncated/garbled
# stderr as automatically corrupt — only a real non-empty ffprobe stderr counts.
probe_file() {
local host_path="$1" container_path output
local host_path="$1" container_path output rc
container_path=$(ffprobe_translate_path "$host_path") || { echo "unmapped"; return; }
output=$(docker exec "$FFPROBE_CONTAINER" "$FFPROBE_BIN" -v error "$container_path" 2>&1)
rc=$?
# docker exec writes its own failures to the same stream ffprobe uses, so a stopped
# container or an unreachable daemon is otherwise indistinguishable from a corrupt
# header. A stopped container exits 1 with a daemon message; a missing binary exits
# 127 — neither is evidence about the file, so both must be caught.
if (( rc >= 125 )) \
|| [[ "$output" == "Error response from daemon:"* \
|| "$output" == "Cannot connect to the Docker daemon"* \
|| "$output" == "error during connect:"* ]]; then
echo "probe_error:${output//$'\n'/ }"
return
fi
if [[ -z "$output" ]]; then
echo "clean"
else
elif (( rc != 0 )); then
# ffprobe could not parse the file — EBML header parsing failed, moov atom not found,
# contradictionary STSC and STCO. This is the only class that may be remediated.
echo "corrupt:${output//$'\n'/ }"
else
# Exit 0 with stderr output: a recoverable muxing complaint, most commonly
# "Referenced QT chapter track not found", which many recent .mp4 releases emit and
# which says nothing about playability. Equating any stderr with corruption is what
# produced 103 "corrupt" files on 2026-08-23 — 28 of 43 newly scanned Radarr items.
# Reported for visibility, never strike-tracked, never remediated.
echo "suspect:${output//$'\n'/ }"
fi
}
@@ -442,7 +483,7 @@ TOTAL_SCANNED=0
TOTAL_CORRUPT=0
TOTAL_REMEDIATED=0
TOTAL_REMEDIATE_FAILED=0
declare -A ARR_SCANNED ARR_SKIPPED_CACHED ARR_SKIPPED_UNMAPPED ARR_CORRUPT ARR_STRIKE_HELD ARR_REMEDIATED ARR_REMEDIATE_FAILED
declare -A ARR_SCANNED ARR_SKIPPED_CACHED ARR_SKIPPED_UNMAPPED ARR_CORRUPT ARR_SUSPECT ARR_PROBE_ERRORS ARR_STRIKE_HELD ARR_REMEDIATED ARR_REMEDIATE_FAILED
for arr in sonarr radarr; do
url_var="${arr^^}_URL"; key_var="${arr^^}_API_KEY"
@@ -532,6 +573,12 @@ for arr in sonarr radarr; do
STRIKE_HELD=0
REMEDIATED=0
REMEDIATE_FAILED=0
PROBE_ERRORS=0
SUSPECT_COUNT=0
CONSECUTIVE_BAD=0
QUEUE_PATH=()
QUEUE_STRIKES=()
QUEUE_ITEM=()
FRESH_CLEAN_TMP=$(mktemp)
@@ -566,23 +613,82 @@ for arr in sonarr radarr; do
continue
fi
# A docker-level failure is not evidence about the file. Count it, never queue it.
if [[ "$result" == probe_error:* ]]; then
(( PROBE_ERRORS++ ))
(( CONSECUTIVE_BAD++ ))
warn " ? $host_path — probe failed, NOT counted as corrupt: ${result#probe_error:}"
if (( CONSECUTIVE_BAD >= CORRUPTION_SCAN_MAX_CONSECUTIVE )); then
abort_scan "$CONSECUTIVE_BAD files in a row failed to probe cleanly"
fi
continue
fi
if [[ "$result" == "clean" ]]; then
CONSECUTIVE_BAD=0
reset_scan_strikes "$host_path"
echo -e "${host_path}\t${stamp}" >> "$FRESH_CLEAN_TMP"
[[ "$ENABLE_LOGGING" == true ]] && echo " $ICON_SUCCESS $host_path"
continue
fi
# corrupt:<reason>
# A successful probe that merely warned. Proves the container is alive, so it clears
# the consecutive-failure tripwire, but it never becomes a strike.
if [[ "$result" == suspect:* ]]; then
CONSECUTIVE_BAD=0
(( SUSPECT_COUNT++ ))
[[ "$ENABLE_LOGGING" == true ]] && warn " ~ $host_path — ffprobe warning (exit 0), NOT corrupt: ${result#suspect:}"
continue
fi
# corrupt:<reason> — queued, NOT committed. Nothing reaches the strike file and nothing
# is deleted until this arr has been fully probed and the guards below have passed. A
# container that dies mid-scan makes every remaining file read as corrupt, and a delete
# cannot be undone — so the destructive half has to wait until the corrupt rate for the
# whole run is known. 2026-08-23: one Jellyfin restart produced 103 false positives.
reason="${result#corrupt:}"
(( CORRUPT_COUNT++ ))
strikes=$(increment_scan_strikes "$host_path")
(( CONSECUTIVE_BAD++ ))
prev_strikes=$(get_scan_strikes "$host_path")
prev_strikes="${prev_strikes//[^0-9]/}"
strikes=$(( ${prev_strikes:-0} + 1 ))
QUEUE_PATH+=("$host_path")
QUEUE_STRIKES+=("$strikes")
QUEUE_ITEM+=("$item")
echo " $ICON_ERROR CORRUPT: $host_path (strike $strikes/$CORRUPTION_SCAN_STRIKE_LIMIT)"
[[ "$ENABLE_LOGGING" == true ]] && echo " $reason"
if [[ "$REMEDIATE" != true ]]; then
continue
if (( CONSECUTIVE_BAD >= CORRUPTION_SCAN_MAX_CONSECUTIVE )); then
abort_scan "$CONSECUTIVE_BAD files in a row failed to probe cleanly"
fi
done < <(echo "$ITEMS" | jq -c '.[]')
# ━━━ False-positive guards — run before anything is committed ━━━
if (( CORRUPT_COUNT > 0 )); then
# The pre-flight check only proves the container was up when the scan started.
# Re-check now: a mid-scan death is exactly what this guard exists to catch.
check_container_health "$FFPROBE_CONTAINER" "${DOCKER_TIMEOUT:-30}" "Arr Corruption Scan"
if (( SCANNED >= CORRUPTION_SCAN_GUARD_MIN_SCANNED )); then
corrupt_pct=$(( CORRUPT_COUNT * 100 / SCANNED ))
if (( corrupt_pct >= CORRUPTION_SCAN_MAX_CORRUPT_PCT )); then
abort_scan "$CORRUPT_COUNT of $SCANNED probed files (${corrupt_pct}%) read as corrupt — at or above the ${CORRUPTION_SCAN_MAX_CORRUPT_PCT}% ceiling"
fi
fi
fi
# ━━━ Guards passed — commit strikes, then remediate whatever reached the limit ━━━
for _q in "${!QUEUE_PATH[@]}"; do
host_path="${QUEUE_PATH[$_q]}"
strikes="${QUEUE_STRIKES[$_q]}"
item="${QUEUE_ITEM[$_q]}"
set_scan_strikes "$host_path" "$strikes"
[[ "$REMEDIATE" != true ]] && continue
if (( strikes < CORRUPTION_SCAN_STRIKE_LIMIT )); then
warn " $host_path — strike $strikes/$CORRUPTION_SCAN_STRIKE_LIMIT, not yet remediating (needs repeat confirmation)"
@@ -591,6 +697,8 @@ for arr in sonarr radarr; do
fi
reset_scan_strikes "$host_path"
file_id=$(echo "$item" | jq -r '.file_id')
parent_id=$(echo "$item" | jq -r '.parent_id')
title=$(echo "$item" | jq -r '.title')
http_code=$(curl -sf -o /dev/null -w "%{http_code}" -X DELETE \
@@ -625,7 +733,7 @@ for arr in sonarr radarr; do
warn " $title — deleted and verified, but search trigger returned HTTP $search_code"
(( REMEDIATE_FAILED++ ))
fi
done < <(echo "$ITEMS" | jq -c '.[]')
done
merge_clean_state "$FRESH_CLEAN_TMP"
rm -f "$FRESH_CLEAN_TMP"
@@ -634,6 +742,8 @@ for arr in sonarr radarr; do
ARR_SKIPPED_CACHED[$arr]=$SKIPPED_CACHED
ARR_SKIPPED_UNMAPPED[$arr]=$SKIPPED_UNMAPPED
ARR_CORRUPT[$arr]=$CORRUPT_COUNT
ARR_SUSPECT[$arr]=$SUSPECT_COUNT
ARR_PROBE_ERRORS[$arr]=$PROBE_ERRORS
ARR_STRIKE_HELD[$arr]=$STRIKE_HELD
ARR_REMEDIATED[$arr]=$REMEDIATED
ARR_REMEDIATE_FAILED[$arr]=$REMEDIATE_FAILED
@@ -658,6 +768,8 @@ for arr in sonarr radarr; do
echo " $ICON_SUCCESS Skipped (cached): ${ARR_SKIPPED_CACHED[$arr]}"
echo " $ICON_WARN Skipped (unmapped): ${ARR_SKIPPED_UNMAPPED[$arr]}"
echo " $ICON_ERROR Corrupt found: ${ARR_CORRUPT[$arr]}"
echo " $ICON_WARN Warnings (exit 0): ${ARR_SUSPECT[$arr]} (reported, never remediated)"
echo " $ICON_WARN Probe errors: ${ARR_PROBE_ERRORS[$arr]} (not counted as corrupt)"
if [[ "$REMEDIATE" == true ]]; then
echo " $ICON_WARN Held (strikes): ${ARR_STRIKE_HELD[$arr]}"
echo " $ICON_SUCCESS Remediated: ${ARR_REMEDIATED[$arr]}"
+165 -53
View File
@@ -146,10 +146,21 @@
# (default: 7)
#
# DOWNLOAD_ORPHAN_MIN_VIDEO_MB
# An entry with no video file above this size is JUNK (default: 50)
# An entry with no video file above this size is JUNK (default: 50). Sonarr/Radarr only.
#
# DOWNLOAD_ORPHAN_MIN_AUDIO_MB
# The same test for Lidarr (default: 2). Separate because a 50M floor would mark
# every album folder as JUNK — single tracks rarely reach it.
#
# DOWNLOAD_ORPHAN_KEEP_MARKER
# A file with this name inside a download folder pins it — the folder is never
# classified or deleted (default: .vv-keep). For lossless rips the library holds
# only at lower quality, which REDUNDANT would otherwise sweep.
#
# DOWNLOAD_ORPHAN_MAX_DELETE_GB
# Abort the delete pass if the run total exceeds this (default: 100)
# Per-run delete budget in GB (default: 100). A backlog above this is drained
# safest-first (JUNK, then REDUNDANT, then UNMATCHED) up to the budget, and the
# remainder is deferred to the next run rather than aborting the pass.
#
# SONARR_EXTENSIONS / RADARR_EXTENSIONS
# Video extensions used to decide whether an entry contains real media
@@ -161,7 +172,7 @@
# arr_download_orphan_cleaner.sh — daily orchestrator entry
# arr_download_orphan_cleaner.sh --dry-run — classify and report only
# arr_download_orphan_cleaner.sh --status — show config and exit
# arr_download_orphan_cleaner.sh --i-know-what-im-doing — bypass MAX_DELETE_GB cap
# arr_download_orphan_cleaner.sh --i-know-what-im-doing — bypass MAX_DELETE_GB budget
#
# ==============================================================================================
@@ -197,6 +208,8 @@ fi
DOWNLOAD_ORPHAN_AGE="${DOWNLOAD_ORPHAN_AGE:-7}"
DOWNLOAD_ORPHAN_MIN_VIDEO_MB="${DOWNLOAD_ORPHAN_MIN_VIDEO_MB:-50}"
DOWNLOAD_ORPHAN_MIN_AUDIO_MB="${DOWNLOAD_ORPHAN_MIN_AUDIO_MB:-2}"
DOWNLOAD_ORPHAN_KEEP_MARKER="${DOWNLOAD_ORPHAN_KEEP_MARKER:-.vv-keep}"
DOWNLOAD_ORPHAN_MAX_DELETE_GB="${DOWNLOAD_ORPHAN_MAX_DELETE_GB:-100}"
if [[ "${SHOW_STATUS:-false}" == true ]]; then
@@ -204,9 +217,10 @@ if [[ "${SHOW_STATUS:-false}" == true ]]; then
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_GEAR Enabled: ${DOWNLOAD_ORPHAN_CLEANER_ENABLED}"
echo "$ICON_TIME Age gate: ${DOWNLOAD_ORPHAN_AGE}d"
echo "$ICON_DISK Junk threshold: ${DOWNLOAD_ORPHAN_MIN_VIDEO_MB}M"
echo "$ICON_DISK Junk threshold: ${DOWNLOAD_ORPHAN_MIN_VIDEO_MB}M video / ${DOWNLOAD_ORPHAN_MIN_AUDIO_MB}M audio"
echo "$ICON_SHIELD Delete cap: ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G"
for arr in SONARR RADARR; do
echo "$ICON_SHIELD Keep marker: ${DOWNLOAD_ORPHAN_KEEP_MARKER}"
for arr in SONARR RADARR LIDARR; do
dir_var="${MY_ID}_${arr}_DOWNLOAD_DIR"
echo "$ICON_CLEAN ${arr}: ${!dir_var:-<not configured>}"
done
@@ -221,13 +235,16 @@ AGE_CUTOFF=$(( $(date +%s) - DOWNLOAD_ORPHAN_AGE * 86400 ))
TOTAL_DELETED=0
TOTAL_DELETED_MB=0
TOTAL_HELD=0
TOTAL_DEFERRED=0
TOTAL_KEPT=0
TOTAL_SCANS=0
echo "━━━━━ $ICON_CLEAN DOWNLOAD ORPHAN CLEANER ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
[[ "$DRY_RUN" == true ]] && echo "$ICON_SKIP DRY RUN — nothing will be deleted or imported"
for arr in sonarr radarr; do
for arr in sonarr radarr lidarr; do
api_ver="v3"; [[ "$arr" == "lidarr" ]] && api_ver="v1"
url_var="${arr^^}_URL"; key_var="${arr^^}_API_KEY"
arr_url="${!url_var:-}"; arr_key="${!key_var:-}"
dir_var="${MY_ID}_${arr^^}_DOWNLOAD_DIR"
@@ -247,24 +264,38 @@ for arr in sonarr radarr; do
echo "━━━ $ICON_SYNC ${arr^}$dl_dir ━━━"
ver_var="${arr^^}_VERSION_MAJOR"
check_arr_version "$arr_url" "$arr_key" "v3" "${!ver_var}" "${arr^}" || {
check_arr_version "$arr_url" "$arr_key" "$api_ver" "${!ver_var}" "${arr^}" || {
warn "${arr^} version check failed — skipping this arr"
continue
}
# min_mb is per-arr because the JUNK test is "contains no real media file". A 50MB floor
# is right for video and catastrophic for audio — most single tracks never reach it, so
# every music folder would classify as JUNK and be deleted regardless of import state.
case "$arr" in
sonarr)
queue_endpoint="queue?pageSize=1000&includeUnknownSeriesItems=true"
exts_var="SONARR_EXTENSIONS"
scan_command="DownloadedEpisodesScan"
library_endpoint="series"
min_mb="$DOWNLOAD_ORPHAN_MIN_VIDEO_MB"
;;
radarr)
queue_endpoint="queue?pageSize=1000&includeUnknownMovieItems=true"
exts_var="RADARR_EXTENSIONS"
scan_command="DownloadedMoviesScan"
library_endpoint="movie"
min_mb="$DOWNLOAD_ORPHAN_MIN_VIDEO_MB"
;;
lidarr)
queue_endpoint="queue?pageSize=1000&includeUnknownArtistItems=true"
exts_var="LIDARR_EXTENSIONS"
scan_command="DownloadedAlbumsScan"
library_endpoint="artist"
min_mb="$DOWNLOAD_ORPHAN_MIN_AUDIO_MB"
;;
esac
if [[ "$arr" == "sonarr" ]]; then
queue_endpoint="queue?pageSize=1000&includeUnknownSeriesItems=true"
exts_var="SONARR_EXTENSIONS"
scan_command="DownloadedEpisodesScan"
library_endpoint="series"
else
queue_endpoint="queue?pageSize=1000&includeUnknownMovieItems=true"
exts_var="RADARR_EXTENSIONS"
scan_command="DownloadedMoviesScan"
library_endpoint="movie"
fi
QUEUE_JSON=$(arr_api "$arr_url" "$arr_key" "v3" "$queue_endpoint" "${arr^}") || {
QUEUE_JSON=$(arr_api "$arr_url" "$arr_key" "$api_ver" "$queue_endpoint" "${arr^}") || {
error "${arr^} queue fetch failed — cannot tell tracked from orphaned, skipping this arr"
continue
}
@@ -282,11 +313,21 @@ for arr in sonarr radarr; do
SCAN_PATHS=()
UNMATCHED_PATHS=()
UNMATCHED_SIZES=()
arr_tracked=0; arr_recent=0; arr_held=0; arr_delete_mb=0
arr_tracked=0; arr_recent=0; arr_held=0; arr_delete_mb=0; arr_kept=0
while IFS= read -r entry; do
base="${entry##*/}"
# An operator keep-marker outranks every verdict below. Needed because REDUNDANT only
# asks "does the library hold this album", not "at what quality" — a lossless rip whose
# library copy is MP3 is redundant by that test and would be swept on the next run.
# The marker is a file inside the folder rather than a conf list so it survives renames
# and cannot drift out of sync with what is actually on disk.
if [[ -e "$entry/$DOWNLOAD_ORPHAN_KEEP_MARKER" ]]; then
arr_kept=$((arr_kept + 1))
continue
fi
if [[ -n "${PROTECTED[$base]:-}" ]]; then
arr_tracked=$((arr_tracked + 1))
continue
@@ -298,18 +339,30 @@ for arr in sonarr radarr; do
continue
fi
has_video=false
# JUNK means "holds no real media". That verdict is only as good as the extension
# list, and a missing extension turns real content into a delete — 2026-08-21 the
# audio list had no "wv", which classified 23 folders of WavPack lossless (1.5G per
# file) as junk. So a folder with large files that are merely *unrecognised* is held
# for review, never deleted; only a folder with nothing big in it at all is junk.
has_media=false
big_unknown=0
while IFS= read -r f; do
if has_extension "$f" "${arr_exts[@]}"; then
has_video=true
has_media=true
break
fi
done < <(find "$entry" -type f -size +"${DOWNLOAD_ORPHAN_MIN_VIDEO_MB}"M 2>/dev/null)
big_unknown=$((big_unknown + 1))
done < <(find "$entry" -type f -size +"${min_mb}"M 2>/dev/null)
size_mb=$(du -sm "$entry" 2>/dev/null | cut -f1)
size_mb=${size_mb:-0}
size_mb=$(dir_size_mb "$entry") || size_mb=0
if [[ "$has_video" == false ]]; then
if [[ "$has_media" == false ]] && (( big_unknown > 0 )); then
warn " no recognised media, but $big_unknown large file(s) of unknown type — holding: $base"
arr_held=$((arr_held + 1))
continue
fi
if [[ "$has_media" == false ]]; then
DELETE_PATHS+=("$entry")
DELETE_SIZES+=("$size_mb")
DELETE_LABELS+=("JUNK")
@@ -318,20 +371,43 @@ for arr in sonarr radarr; do
fi
enc_title=$(jq -rn --arg t "$base" '$t|@uri')
parse=$(arr_api "$arr_url" "$arr_key" "v3" "parse?title=${enc_title}" "${arr^}") || {
parse=$(arr_api "$arr_url" "$arr_key" "$api_ver" "parse?title=${enc_title}" "${arr^}") || {
warn " parse failed for: $base — holding"
arr_held=$((arr_held + 1))
continue
}
if [[ "$arr" == "sonarr" ]]; then
matched=$(echo "$parse" | jq '(.series != null) and ((.episodes | length) > 0)')
missing=$(echo "$parse" | jq '[.episodes[]? | select(.hasFile == false)] | length')
else
# Radarr's parse never populates hasFile — movieFileId is the reliable signal
matched=$(echo "$parse" | jq '.movie != null')
missing=$(echo "$parse" | jq 'if (.movie.movieFileId // 0) > 0 then 0 else 1 end')
fi
case "$arr" in
sonarr)
matched=$(echo "$parse" | jq '(.series != null) and ((.episodes | length) > 0)')
missing=$(echo "$parse" | jq '[.episodes[]? | select(.hasFile == false)] | length')
;;
radarr)
# Radarr's parse never populates hasFile — movieFileId is the reliable signal
matched=$(echo "$parse" | jq '.movie != null')
missing=$(echo "$parse" | jq 'if (.movie.movieFileId // 0) > 0 then 0 else 1 end')
;;
lidarr)
# Lidarr's parse returns albums with statistics:null, so the track count has
# to be read back from album/{id} — the same shape of gap as Radarr's hasFile.
matched=$(echo "$parse" | jq '(.artist != null) and ((.albums | length) > 0)')
missing=1
if [[ "$matched" == true ]]; then
album_id=$(echo "$parse" | jq -r '.albums[0].id // empty')
if [[ -z "$album_id" ]]; then
warn " parse matched but returned no album id: $base — holding"
arr_held=$((arr_held + 1))
continue
fi
album_json=$(arr_api "$arr_url" "$arr_key" "$api_ver" "album/$album_id" "${arr^}") || {
warn " album lookup failed for: $base — holding"
arr_held=$((arr_held + 1))
continue
}
missing=$(echo "$album_json" | jq 'if ((.statistics.trackFileCount // 0) > 0) then 0 else 1 end')
fi
;;
esac
if [[ "$matched" != true ]]; then
UNMATCHED_PATHS+=("$entry")
@@ -354,7 +430,7 @@ for arr in sonarr radarr; do
# fails on a small batch that is legitimately all-unmatched, which is the normal case
# once daily runs have caught up.
if (( ${#UNMATCHED_PATHS[@]} > 0 )); then
library_count=$(arr_api "$arr_url" "$arr_key" "v3" "$library_endpoint" "${arr^}" | jq 'length' 2>/dev/null)
library_count=$(arr_api "$arr_url" "$arr_key" "$api_ver" "$library_endpoint" "${arr^}" | jq 'length' 2>/dev/null)
if [[ ! "$library_count" =~ ^[0-9]+$ ]] || (( library_count == 0 )); then
warn " ${arr^}: library reports ${library_count:-no} titles — cannot trust 'no match', holding ${#UNMATCHED_PATHS[@]} unmatched"
arr_held=$((arr_held + ${#UNMATCHED_PATHS[@]}))
@@ -368,26 +444,59 @@ for arr in sonarr radarr; do
fi
fi
if (( arr_delete_mb / 1024 > DOWNLOAD_ORPHAN_MAX_DELETE_GB )) && [[ "$I_KNOW" != true ]]; then
error "${arr^}: delete total $((arr_delete_mb / 1024))G exceeds cap of ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G — aborting delete pass"
notify "${arr^} download orphan delete total $((arr_delete_mb / 1024))G exceeds ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G cap on $(hostname) — possible partial queue data, nothing deleted. Re-run with --i-know-what-im-doing if legitimate." \
# The cap is a per-run risk budget, not a reason to do nothing. Aborting the whole pass
# once the backlog exceeds it is self-defeating: the backlog can never shrink below the
# cap on its own, so every later run aborts too and the pool fills anyway (exactly how
# 347G accumulated here by 2026-08-21). Delete in ascending order of risk instead, stop
# at the cap, and defer the rest to the next run so a backlog drains over days.
#
# Live downloads are already protected by DOWNLOAD_ORPHAN_AGE, not by this cap — anything
# in flight is younger than the age gate and never reaches classification. That is what
# makes draining safe: the partial-queue-data case the cap was written for cannot put a
# still-downloading entry in these arrays.
cap_mb=$((DOWNLOAD_ORPHAN_MAX_DELETE_GB * 1024))
cap_active=true
[[ "$I_KNOW" == true || "$DRY_RUN" == true ]] && cap_active=false
arr_deferred=0; arr_deferred_mb=0; arr_run_mb=0
if [[ "$cap_active" == true ]] && (( arr_delete_mb > cap_mb )); then
warn " ${arr^}: $((arr_delete_mb / 1024))G classified vs ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G cap — deleting safest-first up to the cap, deferring the rest"
notify "${arr^} download orphan backlog is $((arr_delete_mb / 1024))G on $(hostname), above the ${DOWNLOAD_ORPHAN_MAX_DELETE_GB}G per-run cap. Draining safest-first; the remainder follows on later runs. Re-run with --i-know-what-im-doing to clear it in one pass." \
"Download Orphan Cleaner" "warning"
unset PROTECTED
continue
fi
for i in "${!DELETE_PATHS[@]}"; do
entry="${DELETE_PATHS[$i]}"
if [[ "$DRY_RUN" == true ]]; then
echo " $ICON_SKIP would delete [${DELETE_LABELS[$i]}]: ${entry##*/} (${DELETE_SIZES[$i]}M)"
else
echo " $ICON_TRASH deleting [${DELETE_LABELS[$i]}]: ${entry##*/} (${DELETE_SIZES[$i]}M)"
rm -rf "$entry"
fi
TOTAL_DELETED=$((TOTAL_DELETED + 1))
TOTAL_DELETED_MB=$((TOTAL_DELETED_MB + DELETE_SIZES[i]))
# JUNK first (no media at all), then REDUNDANT (parse-verified already in the library),
# then UNMATCHED last — it rests on "the arr does not know this title", the weakest of
# the three signals, so it is the first thing the cap defers.
for pass in JUNK REDUNDANT UNMATCHED; do
for i in "${!DELETE_PATHS[@]}"; do
[[ "${DELETE_LABELS[$i]}" == "$pass" ]] || continue
entry="${DELETE_PATHS[$i]}"
if [[ "$cap_active" == true ]] && (( arr_run_mb + DELETE_SIZES[i] > cap_mb )); then
arr_deferred=$((arr_deferred + 1))
arr_deferred_mb=$((arr_deferred_mb + DELETE_SIZES[i]))
continue
fi
if [[ "$DRY_RUN" == true ]]; then
echo " $ICON_SKIP would delete [${DELETE_LABELS[$i]}]: ${entry##*/} (${DELETE_SIZES[$i]}M)"
else
echo " $ICON_TRASH deleting [${DELETE_LABELS[$i]}]: ${entry##*/} (${DELETE_SIZES[$i]}M)"
rm -rf "$entry"
fi
arr_run_mb=$((arr_run_mb + DELETE_SIZES[i]))
TOTAL_DELETED=$((TOTAL_DELETED + 1))
TOTAL_DELETED_MB=$((TOTAL_DELETED_MB + DELETE_SIZES[i]))
done
done
if (( arr_deferred > 0 )); then
echo " $ICON_WARN ${arr^}: deferred $arr_deferred entries ($((arr_deferred_mb / 1024))G) to the next run — cap reached"
TOTAL_DEFERRED=$((TOTAL_DEFERRED + arr_deferred))
fi
for base in "${SCAN_PATHS[@]}"; do
if [[ -z "$container_dir" ]]; then
echo " $ICON_WARN IMPORTABLE but ${cdir_var} not set — holding: $base"
@@ -406,8 +515,9 @@ for arr in sonarr radarr; do
fi
done
echo " $ICON_SUMMARY ${arr^}: $arr_tracked tracked, $arr_recent recent, ${#DELETE_PATHS[@]} deleted ($((arr_delete_mb / 1024))G), ${#SCAN_PATHS[@]} import scans, $arr_held held"
echo " $ICON_SUMMARY ${arr^}: $arr_tracked tracked, $arr_kept kept, $arr_recent recent, $((${#DELETE_PATHS[@]} - arr_deferred)) deleted ($((arr_run_mb / 1024))G), $arr_deferred deferred ($((arr_deferred_mb / 1024))G), ${#SCAN_PATHS[@]} import scans, $arr_held held"
TOTAL_HELD=$((TOTAL_HELD + arr_held))
TOTAL_KEPT=$((TOTAL_KEPT + arr_kept))
unset PROTECTED
done
@@ -416,6 +526,8 @@ echo "━━━━━ $ICON_DONE SUMMARY ━━━━━"
echo "$ICON_TRASH Deleted: $TOTAL_DELETED ($((TOTAL_DELETED_MB / 1024))G)"
echo "$ICON_RUN Import scans: $TOTAL_SCANS"
echo "$ICON_WARN Held: $TOTAL_HELD"
echo "$ICON_SKIP Deferred: $TOTAL_DEFERRED"
echo "$ICON_SHIELD Kept (marker): $TOTAL_KEPT"
# Held alone never notifies — there is always something awaiting review, and on a daily
# schedule that would be a notification every morning saying nothing happened.
Executable → Regular
+70 -8
View File
@@ -457,6 +457,30 @@ NOW=$(date +%s)
TO_DELETE_FILE="$TMP_DIR/to_delete_paths.txt"
> "$TO_DELETE_FILE"
# ── Orphan strikes ────────────────────────────────────────────────────────────────────────────
# Same contract as radarr_cleanup.sh: a file must classify for deletion on
# LIDARR_ORPHAN_STRIKE_LIMIT consecutive runs before it is removed. Covers the partial
# classification failure that is too small to trip the tracked-count floor above. The file is
# rebuilt from each run rather than edited, which is what prunes it.
LIDARR_ORPHAN_STRIKE_LIMIT="${LIDARR_ORPHAN_STRIKE_LIMIT:-2}"
STRIKES_FILE="${LIDARR_ORPHAN_STRIKES_FILE:-$DB_DIR/lidarr_orphan_strikes.tsv}"
mkdir -p "$(dirname "$STRIKES_FILE")" 2>/dev/null || true
touch "$STRIKES_FILE" 2>/dev/null || true
STRIKES_NEW="$TMP_DIR/strikes_new.tsv"
> "$STRIKES_NEW"
HELD_COUNT=0
HELD_BYTES=0
orphan_strike_ok() {
local path="$1" prev strikes
prev=$(wd_state_get "$path" "$STRIKES_FILE"); prev="${prev//[^0-9]/}"
strikes=$(( ${prev:-0} + 1 ))
printf '%s:%s\n' "$path" "$strikes" >> "$STRIKES_NEW"
(( strikes >= LIDARR_ORPHAN_STRIKE_LIMIT )) && return 0
warn " strike $strikes/$LIDARR_ORPHAN_STRIKE_LIMIT — not removing yet: $path"
return 1
}
while read -r FILE_SIZE FILE_CTIME filepath; do
[[ -z "$filepath" ]] && continue
FILE_CTIME="${FILE_CTIME%%.*}"
@@ -493,12 +517,14 @@ while read -r FILE_SIZE FILE_CTIME filepath; do
warn "$ICON_TRASH ORPHAN: $filepath"
(( ORPHAN_COUNT++ ))
ORPHAN_BYTES=$(( ORPHAN_BYTES + FILE_SIZE ))
echo "$filepath" >> "$TO_DELETE_FILE"
if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
else
log "JUNK: $filepath"
(( JUNK_COUNT++ ))
JUNK_BYTES=$(( JUNK_BYTES + FILE_SIZE ))
echo "$filepath" >> "$TO_DELETE_FILE"
if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
fi
# -printf gets size + mtime directly from find's own stat() during the walk, instead of a
@@ -506,13 +532,43 @@ while read -r FILE_SIZE FILE_CTIME filepath; do
# 4.3ms), since find already has to stat() every entry anyway to know it's -type f.
done < <(find "$LIDARR_MUSIC_ROOT" -type f -printf '%s %C@ %p\n' 2>/dev/null)
TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES ))
TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT ))
# Eligible, not classified: a file still serving its strikes is an orphan but is not queued this
# run, so it must not appear in the denominator the budget reports against.
TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES - HELD_BYTES ))
TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT - HELD_COUNT ))
# Rebuilt, never edited. Skipped on a dry run: a preview that advanced real counters would make
# the next real run delete a run early.
if [[ "$DRY_RUN" == false ]]; then
mv "$STRIKES_NEW" "$STRIKES_FILE" 2>/dev/null || warn "Could not update $STRIKES_FILE"
fi
# ==============================================================================================
# ━━━ Safety Layer 7 — Deletion Size Threshold ━━━
# ==============================================================================================
check_delete_size_threshold "$TOTAL_DELETE_BYTES" "$LIDARR_MAX_DELETE_GB" "Lidarr Cleanup"
# A per-run budget, not a veto — see apply_delete_budget() in common.sh. The ceiling still caps
# any single run; it just no longer deadlocks on a backlog larger than itself.
BUDGET_FILE="$TMP_DIR/to_delete_budgeted.txt"
if [[ "$I_KNOW" == true ]]; then
warn "OVERRIDE — --i-know-what-im-doing active, per-run budget not applied"
cut -d"$(printf '\t')" -f3- "$TO_DELETE_FILE" > "$BUDGET_FILE"
_BUDGET_KEPT_COUNT=$TOTAL_REMOVED; _BUDGET_KEPT_BYTES=$TOTAL_DELETE_BYTES
_BUDGET_DEFERRED_COUNT=0; _BUDGET_DEFERRED_BYTES=0; _BUDGET_STUCK=""
else
apply_delete_budget "$TO_DELETE_FILE" "$BUDGET_FILE" "$LIDARR_MAX_DELETE_GB"
if [[ -n "$_BUDGET_STUCK" ]]; then
error "Single file exceeds the ${LIDARR_MAX_DELETE_GB}GB budget on its own — nothing removed this run"
error " $_BUDGET_STUCK"
error "Raise LIDARR_MAX_DELETE_GB or clear this one with --i-know-what-im-doing"
notify "Lidarr cleanup stalled on $(hostname) — one file exceeds the ${LIDARR_MAX_DELETE_GB}GB budget" \
"Lidarr Cleanup" "warning"
elif [[ "$_BUDGET_DEFERRED_COUNT" -gt 0 ]]; then
warn "Budget ${LIDARR_MAX_DELETE_GB}GB — removing $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED ($(format_bytes "$_BUDGET_KEPT_BYTES")), deferring $_BUDGET_DEFERRED_COUNT ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) to the next run"
notify "Lidarr cleanup removed $(format_bytes "$_BUDGET_KEPT_BYTES") of $(format_bytes "$TOTAL_DELETE_BYTES") on $(hostname)$_BUDGET_DEFERRED_COUNT file(s) deferred" \
"Lidarr Cleanup" "normal"
fi
fi
# ── Execute Deletions ─────────────────────────────────────────────────────────────────────────
# All safety layers passed — delete orphans and junk. Reuses TO_DELETE_FILE from the
@@ -521,7 +577,7 @@ if [[ "$DRY_RUN" == false ]]; then
while IFS= read -r filepath; do
[[ -z "$filepath" ]] && continue
rm -f "$filepath" 2>/dev/null || error "Failed to delete: $filepath"
done < "$TO_DELETE_FILE"
done < "$BUDGET_FILE"
info "Cleaning up empty folders..."
find "$LIDARR_MUSIC_ROOT" -mindepth 1 -type d -empty -delete 2>/dev/null
@@ -546,6 +602,10 @@ echo "$ICON_SHIELD Protected: $PROTECTED_COUNT files (cover art, metadata
echo "$ICON_TRASH Orphans: $ORPHAN_COUNT files ($ORPHAN_HUMAN)"
echo "$ICON_TRASH Junk: $JUNK_COUNT files ($JUNK_HUMAN)"
echo "$ICON_SKIP Recent skipped: $RECENT_COUNT files (under ${LIDARR_ORPHAN_AGE} days)"
[[ "${HELD_COUNT:-0}" -gt 0 ]] && \
echo "$ICON_SKIP Held (strikes): $HELD_COUNT files ($(format_bytes "$HELD_BYTES")) — under ${LIDARR_ORPHAN_STRIKE_LIMIT} consecutive runs"
[[ "${_BUDGET_DEFERRED_COUNT:-0}" -gt 0 ]] && \
echo "$ICON_SKIP Deferred: $_BUDGET_DEFERRED_COUNT files ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) — over the ${LIDARR_MAX_DELETE_GB}GB run budget"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
@@ -554,8 +614,10 @@ if [[ "$DRY_RUN" == true ]]; then
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
echo "$ICON_DONE Clean — nothing to remove"
else
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
notify "Lidarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" "Lidarr Cleanup" "warning"
# What was actually removed, not what was classified. With strikes and a budget in force those
# differ, and reporting the classification as the outcome is the oldest bug shape here.
warn "$ICON_DONE Removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED eligible files ($(format_bytes "$_BUDGET_KEPT_BYTES"))"
notify "Lidarr cleanup on $(hostname) — removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED eligible files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" "Lidarr Cleanup" "warning"
# Notify Emby to clean missing files — removes ghost entries immediately
notify_emby_scan
fi
Executable → Regular
+134 -8
View File
@@ -450,9 +450,46 @@ NOW=$(date +%s)
# just delete them directly instead of re-walking and re-classifying every SCAN_ROOTS entry a
# second time (2026-07-17) — the size-threshold check below needs to know the total before
# deleting anything, not before knowing what to delete.
# Carries size and ctime alongside the path now, because the budget pass below has to order by
# age and stop at a byte ceiling — neither of which a bare path list can answer.
TO_DELETE_FILE="$TMP_DIR/to_delete_paths.txt"
> "$TO_DELETE_FILE"
# ── Orphan strikes ────────────────────────────────────────────────────────────────────────────
# A file must classify for deletion on RADARR_ORPHAN_STRIKE_LIMIT consecutive runs before it is
# actually removed. Gate 6 already refuses a run whose tracked count collapsed; this covers the
# partial failure underneath that threshold — one root folder failing to enumerate makes its
# movies look orphaned while the overall percentage still looks fine, and a transient fault will
# not reproduce on the next run.
#
# The file is REBUILT from this run's classifications rather than edited in place, which is what
# prunes it: anything that stopped being an orphan simply is not written again, so a file that
# Radarr re-adopts loses its strikes without needing a reset pass to find it.
#
# Keyed by host path, which is why this could not have worked before 2026-08-26 — wd_state_set
# built a regex from the key, and a release tag like [Bluray-1080p] holds the reversed range 1-0,
# so every write truncated the store to one line. See common.sh.
RADARR_ORPHAN_STRIKE_LIMIT="${RADARR_ORPHAN_STRIKE_LIMIT:-2}"
STRIKES_FILE="${RADARR_ORPHAN_STRIKES_FILE:-$DB_DIR/radarr_orphan_strikes.tsv}"
mkdir -p "$(dirname "$STRIKES_FILE")" 2>/dev/null || true
touch "$STRIKES_FILE" 2>/dev/null || true
STRIKES_NEW="$TMP_DIR/strikes_new.tsv"
> "$STRIKES_NEW"
HELD_COUNT=0
HELD_BYTES=0
# Records this run's strike for a file and says whether it has served enough of them.
# Returns 0 when the file may be deleted, 1 when it is still accruing.
orphan_strike_ok() {
local path="$1" prev strikes
prev=$(wd_state_get "$path" "$STRIKES_FILE"); prev="${prev//[^0-9]/}"
strikes=$(( ${prev:-0} + 1 ))
printf '%s:%s\n' "$path" "$strikes" >> "$STRIKES_NEW"
(( strikes >= RADARR_ORPHAN_STRIKE_LIMIT )) && return 0
warn " strike $strikes/$RADARR_ORPHAN_STRIKE_LIMIT — not removing yet: $path"
return 1
}
while read -r FILE_SIZE FILE_CTIME filepath; do
[[ -z "$filepath" ]] && continue
FILE_CTIME="${FILE_CTIME%%.*}"
@@ -487,12 +524,14 @@ while read -r FILE_SIZE FILE_CTIME filepath; do
warn "$ICON_TRASH ORPHAN: $filepath"
(( ORPHAN_COUNT++ ))
ORPHAN_BYTES=$(( ORPHAN_BYTES + FILE_SIZE ))
echo "$filepath" >> "$TO_DELETE_FILE"
if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
else
log "JUNK: $filepath"
(( JUNK_COUNT++ ))
JUNK_BYTES=$(( JUNK_BYTES + FILE_SIZE ))
echo "$filepath" >> "$TO_DELETE_FILE"
if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
fi
# -printf gets size + mtime directly from find's own stat() during the walk, instead of a
@@ -504,13 +543,94 @@ done < <(
done | sort -u
)
TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES ))
TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT ))
# Eligible, not classified. A file still serving its strikes was counted as an orphan above — it
# is one — but it is not going to be deleted this run, so it must not appear in the denominator
# the budget reports against or the run claims to have skipped work it never queued.
TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES - HELD_BYTES ))
TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT - HELD_COUNT ))
# Rebuilt, never edited: a path absent from this run is absent from the file, so a file Radarr
# re-adopts drops its strikes with no reset pass needed. Skipped on a dry run — a preview that
# advanced real strike counters would make the next real run delete a run early.
if [[ "$DRY_RUN" == false ]]; then
mv "$STRIKES_NEW" "$STRIKES_FILE" 2>/dev/null || warn "Could not update $STRIKES_FILE"
fi
# ==============================================================================================
# ━━━ Safety Layer 7 — Deletion Size Threshold ━━━
# ==============================================================================================
check_delete_size_threshold "$TOTAL_DELETE_BYTES" "$RADARR_MAX_DELETE_GB" "Radarr Cleanup"
# The ceiling is a per-run budget, not a veto. It still means what it always meant — no single run
# removes more than RADARR_MAX_DELETE_GB — but a backlog larger than the ceiling now drains over
# consecutive nights instead of failing the orchestrator forever on a queue it cannot clear.
# ── AI note (AI_ASSIST_CLEANUP) ───────────────────────────────────────────────────────────────
# Describes the shape of what was classified. It decides nothing: the eligible set, the budget and
# the strikes are all settled above and none of them read this. Switch AI_ASSIST_CLEANUP off and
# the run removes exactly the same files — the log just loses a paragraph.
#
# ctime clustering is the signal worth surfacing. A normal upgrade cycle dribbles in over weeks; a
# lump sharing one narrow ctime window with mtimes spread across months is a bulk write-back, which
# is what a partnership merge against a partner holding older copies produces. That distinction
# took a person an evening on 2026-08-26 and is the whole reason this note exists.
if [[ "$ORPHAN_COUNT" -gt 0 ]] && [[ -s "$TO_DELETE_FILE" ]]; then
_ai_ev=$(awk -F'\t' '
{ n++; bytes += $1
c = int($2)
if (cmin == 0 || c < cmin) cmin = c
if (c > cmax) cmax = c
bucket[int(c / 21600)]++ }
END {
for (b in bucket) if (bucket[b] > top) { top = bucket[b] }
printf "files=%d bytes_gb=%.1f ctime_span_hours=%.1f largest_6h_ctime_bucket=%d\n",
n, bytes/1073741824, (cmax-cmin)/3600, top
}' "$TO_DELETE_FILE")
_ai_mt=$(cut -d"$(printf '\t')" -f3 "$TO_DELETE_FILE" | head -8 \
| while IFS= read -r p; do [[ -f "$p" ]] && \
printf '%s %s\n' "$(stat -c %y "$p" 2>/dev/null | cut -c1-7)" "$(basename "$p")"; done)
_ai_note=$(ai_assist_note AI_ASSIST_CLEANUP "You are looking at files an automated media-library cleanup has classified for deletion on an Unraid server. They are files on disk that the Radarr database no longer references.
EVIDENCE
$_ai_ev
sample (modification month, then path):
$_ai_mt
A normal quality-upgrade cycle produces orphans whose ctimes are spread out over weeks, because each upgrade happens on its own day. A bulk event - a sync or restore writing files back onto this host - produces orphans sharing one narrow ctime window while their modification times stay spread across months, because the copy preserves modification time but resets ctime.
In no more than three sentences, say which of those two this looks like and name the numbers above that support it. Do not recommend an action. Do not speculate beyond the evidence given.") || _ai_note=""
if [[ -n "$_ai_note" ]]; then
echo ""
echo "━━━ $ICON_GEAR AI note on this classification ━━━"
printf '%s\n' "$_ai_note"
fi
unset _ai_ev _ai_mt
fi
BUDGET_FILE="$TMP_DIR/to_delete_budgeted.txt"
if [[ "$I_KNOW" == true ]]; then
warn "OVERRIDE — --i-know-what-im-doing active, per-run budget not applied"
cut -d"$(printf '\t')" -f3- "$TO_DELETE_FILE" > "$BUDGET_FILE"
_BUDGET_KEPT_COUNT=$TOTAL_REMOVED; _BUDGET_KEPT_BYTES=$TOTAL_DELETE_BYTES
_BUDGET_DEFERRED_COUNT=0; _BUDGET_DEFERRED_BYTES=0; _BUDGET_STUCK=""
else
apply_delete_budget "$TO_DELETE_FILE" "$BUDGET_FILE" "$RADARR_MAX_DELETE_GB"
if [[ -n "$_BUDGET_STUCK" ]]; then
# One file larger than the whole budget can never fit, so it would be re-found and
# re-deferred every night. Name it rather than loop on it silently.
error "Single file exceeds the ${RADARR_MAX_DELETE_GB}GB budget on its own — nothing removed this run"
error " $_BUDGET_STUCK"
error "Raise RADARR_MAX_DELETE_GB or clear this one with --i-know-what-im-doing"
notify "Radarr cleanup stalled on $(hostname) — one file exceeds the ${RADARR_MAX_DELETE_GB}GB budget" \
"Radarr Cleanup" "warning"
elif [[ "$_BUDGET_DEFERRED_COUNT" -gt 0 ]]; then
warn "Budget ${RADARR_MAX_DELETE_GB}GB — removing $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED ($(format_bytes "$_BUDGET_KEPT_BYTES")), deferring $_BUDGET_DEFERRED_COUNT ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) to the next run"
warn "Oldest first — the deferred files are the newest and are re-evaluated tomorrow"
notify "Radarr cleanup removed $(format_bytes "$_BUDGET_KEPT_BYTES") of $(format_bytes "$TOTAL_DELETE_BYTES") on $(hostname)$_BUDGET_DEFERRED_COUNT file(s) deferred to the next run" \
"Radarr Cleanup" "normal"
fi
fi
# ── Execute Deletions ─────────────────────────────────────────────────────────────────────────
# Reuses TO_DELETE_FILE from the classification pass above instead of re-walking and
@@ -519,7 +639,7 @@ if [[ "$DRY_RUN" == false ]]; then
while IFS= read -r filepath; do
[[ -z "$filepath" ]] && continue
rm -f "$filepath" 2>/dev/null || error "Failed to delete: $filepath"
done < "$TO_DELETE_FILE"
done < "$BUDGET_FILE"
info "Cleaning up empty folders..."
for host_path in "${SCAN_ROOTS[@]}"; do
@@ -545,6 +665,10 @@ echo "$ICON_SHIELD Protected: $PROTECTED_COUNT files (artwork, subtitles,
echo "$ICON_TRASH Orphans: $ORPHAN_COUNT files ($ORPHAN_HUMAN)"
echo "$ICON_TRASH Junk: $JUNK_COUNT files ($JUNK_HUMAN)"
echo "$ICON_SKIP Recent skipped: $RECENT_COUNT files (under ${RADARR_ORPHAN_AGE} days)"
[[ "${HELD_COUNT:-0}" -gt 0 ]] && \
echo "$ICON_SKIP Held (strikes): $HELD_COUNT files ($(format_bytes "$HELD_BYTES")) — under ${RADARR_ORPHAN_STRIKE_LIMIT} consecutive runs"
[[ "${_BUDGET_DEFERRED_COUNT:-0}" -gt 0 ]] && \
echo "$ICON_SKIP Deferred: $_BUDGET_DEFERRED_COUNT files ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) — over the ${RADARR_MAX_DELETE_GB}GB run budget"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
@@ -553,8 +677,10 @@ if [[ "$DRY_RUN" == true ]]; then
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
echo "$ICON_DONE Clean — nothing to remove"
else
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
notify "Radarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
# What was actually removed, not what was classified. With a budget in force those differ, and
# reporting the classification as the outcome is the oldest bug shape in this codebase.
warn "$ICON_DONE Removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED classified files ($(format_bytes "$_BUDGET_KEPT_BYTES"))"
notify "Radarr cleanup on $(hostname) — removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED classified files ($(format_bytes "$_BUDGET_KEPT_BYTES"))$([[ "${_BUDGET_DEFERRED_COUNT:-0}" -gt 0 ]] && echo ", $_BUDGET_DEFERRED_COUNT deferred")" \
"Radarr Cleanup" "warning"
# Notify Emby to clean missing files — removes ghost entries immediately
notify_emby_scan
Executable → Regular
+70 -8
View File
@@ -440,6 +440,30 @@ NOW=$(date +%s)
TO_DELETE_FILE="$TMP_DIR/to_delete_paths.txt"
> "$TO_DELETE_FILE"
# ── Orphan strikes ────────────────────────────────────────────────────────────────────────────
# Same contract as radarr_cleanup.sh: a file must classify for deletion on
# SONARR_ORPHAN_STRIKE_LIMIT consecutive runs before it is removed. Covers the partial
# classification failure that is too small to trip the tracked-count floor above. The file is
# rebuilt from each run rather than edited, which is what prunes it.
SONARR_ORPHAN_STRIKE_LIMIT="${SONARR_ORPHAN_STRIKE_LIMIT:-2}"
STRIKES_FILE="${SONARR_ORPHAN_STRIKES_FILE:-$DB_DIR/sonarr_orphan_strikes.tsv}"
mkdir -p "$(dirname "$STRIKES_FILE")" 2>/dev/null || true
touch "$STRIKES_FILE" 2>/dev/null || true
STRIKES_NEW="$TMP_DIR/strikes_new.tsv"
> "$STRIKES_NEW"
HELD_COUNT=0
HELD_BYTES=0
orphan_strike_ok() {
local path="$1" prev strikes
prev=$(wd_state_get "$path" "$STRIKES_FILE"); prev="${prev//[^0-9]/}"
strikes=$(( ${prev:-0} + 1 ))
printf '%s:%s\n' "$path" "$strikes" >> "$STRIKES_NEW"
(( strikes >= SONARR_ORPHAN_STRIKE_LIMIT )) && return 0
warn " strike $strikes/$SONARR_ORPHAN_STRIKE_LIMIT — not removing yet: $path"
return 1
}
while read -r FILE_SIZE FILE_CTIME filepath; do
[[ -z "$filepath" ]] && continue
FILE_CTIME="${FILE_CTIME%%.*}"
@@ -474,12 +498,14 @@ while read -r FILE_SIZE FILE_CTIME filepath; do
warn "$ICON_TRASH ORPHAN: $filepath"
(( ORPHAN_COUNT++ ))
ORPHAN_BYTES=$(( ORPHAN_BYTES + FILE_SIZE ))
echo "$filepath" >> "$TO_DELETE_FILE"
if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
else
log "JUNK: $filepath"
(( JUNK_COUNT++ ))
JUNK_BYTES=$(( JUNK_BYTES + FILE_SIZE ))
echo "$filepath" >> "$TO_DELETE_FILE"
if ! orphan_strike_ok "$filepath"; then (( HELD_COUNT++ )); HELD_BYTES=$(( HELD_BYTES + FILE_SIZE )); continue; fi
printf '%s\t%s\t%s\n' "$FILE_SIZE" "$FILE_CTIME" "$filepath" >> "$TO_DELETE_FILE"
fi
# -printf gets size + mtime directly from find's own stat() during the walk, instead of a
@@ -491,13 +517,43 @@ done < <(
done | sort -u
)
TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES ))
TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT ))
# Eligible, not classified: a file still serving its strikes is an orphan but is not queued this
# run, so it must not appear in the denominator the budget reports against.
TOTAL_DELETE_BYTES=$(( ORPHAN_BYTES + JUNK_BYTES - HELD_BYTES ))
TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT - HELD_COUNT ))
# Rebuilt, never edited. Skipped on a dry run: a preview that advanced real counters would make
# the next real run delete a run early.
if [[ "$DRY_RUN" == false ]]; then
mv "$STRIKES_NEW" "$STRIKES_FILE" 2>/dev/null || warn "Could not update $STRIKES_FILE"
fi
# ==============================================================================================
# ━━━ Safety Layer 7 — Deletion Size Threshold ━━━
# ==============================================================================================
check_delete_size_threshold "$TOTAL_DELETE_BYTES" "$SONARR_MAX_DELETE_GB" "Sonarr Cleanup"
# A per-run budget, not a veto — see apply_delete_budget() in common.sh. The ceiling still caps
# any single run; it just no longer deadlocks on a backlog larger than itself.
BUDGET_FILE="$TMP_DIR/to_delete_budgeted.txt"
if [[ "$I_KNOW" == true ]]; then
warn "OVERRIDE — --i-know-what-im-doing active, per-run budget not applied"
cut -d"$(printf '\t')" -f3- "$TO_DELETE_FILE" > "$BUDGET_FILE"
_BUDGET_KEPT_COUNT=$TOTAL_REMOVED; _BUDGET_KEPT_BYTES=$TOTAL_DELETE_BYTES
_BUDGET_DEFERRED_COUNT=0; _BUDGET_DEFERRED_BYTES=0; _BUDGET_STUCK=""
else
apply_delete_budget "$TO_DELETE_FILE" "$BUDGET_FILE" "$SONARR_MAX_DELETE_GB"
if [[ -n "$_BUDGET_STUCK" ]]; then
error "Single file exceeds the ${SONARR_MAX_DELETE_GB}GB budget on its own — nothing removed this run"
error " $_BUDGET_STUCK"
error "Raise SONARR_MAX_DELETE_GB or clear this one with --i-know-what-im-doing"
notify "Sonarr cleanup stalled on $(hostname) — one file exceeds the ${SONARR_MAX_DELETE_GB}GB budget" \
"Sonarr Cleanup" "warning"
elif [[ "$_BUDGET_DEFERRED_COUNT" -gt 0 ]]; then
warn "Budget ${SONARR_MAX_DELETE_GB}GB — removing $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED ($(format_bytes "$_BUDGET_KEPT_BYTES")), deferring $_BUDGET_DEFERRED_COUNT ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) to the next run"
notify "Sonarr cleanup removed $(format_bytes "$_BUDGET_KEPT_BYTES") of $(format_bytes "$TOTAL_DELETE_BYTES") on $(hostname)$_BUDGET_DEFERRED_COUNT file(s) deferred" \
"Sonarr Cleanup" "normal"
fi
fi
# ── Execute Deletions ─────────────────────────────────────────────────────────────────────────
# Reuses TO_DELETE_FILE from the classification pass above instead of re-walking and
@@ -506,7 +562,7 @@ if [[ "$DRY_RUN" == false ]]; then
while IFS= read -r filepath; do
[[ -z "$filepath" ]] && continue
rm -f "$filepath" 2>/dev/null || error "Failed to delete: $filepath"
done < "$TO_DELETE_FILE"
done < "$BUDGET_FILE"
info "Cleaning up empty folders..."
for host_path in "${SCAN_ROOTS[@]}"; do
@@ -532,6 +588,10 @@ echo "$ICON_SHIELD Protected: $PROTECTED_COUNT files (artwork, subtitles,
echo "$ICON_TRASH Orphans: $ORPHAN_COUNT files ($ORPHAN_HUMAN)"
echo "$ICON_TRASH Junk: $JUNK_COUNT files ($JUNK_HUMAN)"
echo "$ICON_SKIP Recent skipped: $RECENT_COUNT files (under ${SONARR_ORPHAN_AGE} days)"
[[ "${HELD_COUNT:-0}" -gt 0 ]] && \
echo "$ICON_SKIP Held (strikes): $HELD_COUNT files ($(format_bytes "$HELD_BYTES")) — under ${SONARR_ORPHAN_STRIKE_LIMIT} consecutive runs"
[[ "${_BUDGET_DEFERRED_COUNT:-0}" -gt 0 ]] && \
echo "$ICON_SKIP Deferred: $_BUDGET_DEFERRED_COUNT files ($(format_bytes "$_BUDGET_DEFERRED_BYTES")) — over the ${SONARR_MAX_DELETE_GB}GB run budget"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
@@ -540,8 +600,10 @@ if [[ "$DRY_RUN" == true ]]; then
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
echo "$ICON_DONE Clean — nothing to remove"
else
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
notify "Sonarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
# What was actually removed, not what was classified. With strikes and a budget in force those
# differ, and reporting the classification as the outcome is the oldest bug shape here.
warn "$ICON_DONE Removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED eligible files ($(format_bytes "$_BUDGET_KEPT_BYTES"))"
notify "Sonarr cleanup on $(hostname) — removed $_BUDGET_KEPT_COUNT of $TOTAL_REMOVED eligible files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
"Sonarr Cleanup" "warning"
# Notify Emby to clean missing files — removes ghost entries immediately
notify_emby_scan
+3 -1
View File
@@ -95,7 +95,7 @@
# ━━━ Identity ━━━
# HOSTN hostname lives in master.conf (not a credential — safe for all servers).
# SSH key used for all server-to-server operations — rsync, failover, conf sync.
# SSH key used for all server-to-server operations — rsync, fallback, conf sync.
# Convention: /root/.ssh/<hostname-lowercase-no-unraid-prefix>_rsync_automation
# Must be in /root/.ssh/ and authorised in the partner's /root/.ssh/authorized_keys.
# Run Partnership/ssh_setup.sh to generate the key and copy it to the partner.
@@ -487,6 +487,8 @@
HOSTN_RADARR_ANIME_ROOT="" # rootFolderPath literal, as reported by Radarr API — leave blank if no dedicated anime root
HOSTN_RADARR_DOWNLOAD_DIR="" # host path of the completed-downloads folder Radarr imports from (e.g. "/mnt/cache/Temp_Storage/SABnzbd/Completed/Movies") — blank disables the download orphan cleaner for Radarr
HOSTN_RADARR_DOWNLOAD_CONTAINER_DIR="" # same folder as Radarr's container sees it (e.g. "/downloads/Completed/Movies") — needed to trigger import scans on held folders
HOSTN_LIDARR_DOWNLOAD_DIR="" # host path of the completed-downloads folder Lidarr imports from (e.g. "/mnt/cache/Temp_Storage/SABnzbd/Completed/Music") — blank disables the download orphan cleaner for Lidarr
HOSTN_LIDARR_DOWNLOAD_CONTAINER_DIR="" # same folder as Lidarr's container sees it (e.g. "/downloads/Completed/Music") — needed to trigger import scans on held folders
declare -A HOSTN_RADARR_PATH_MAP=(
# ["/movies"]="/mnt/user/Movies"
+83 -11
View File
@@ -372,7 +372,7 @@
# Scripts launched by array_started.sh when the array comes online.
# Launched in order — each as a background process.
# One-shot scripts (ramdisk, syslog, fpm, inotify, network) run and exit naturally.
# Continuous scripts (failover) run until array stops.
# Continuous scripts (fallback) run until array stops.
# Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via
# watchdog_orchestrator.sh — NOT launched here.
ARRAY_START_SCRIPTS=(
@@ -490,9 +490,9 @@
"Arrs_Stack/sonarr_cleanup.sh" # remove orphaned TV files
"Arrs_Stack/radarr_cleanup.sh" # remove orphaned movie files
# Daily, not weekly — DOWNLOAD_ORPHAN_AGE gates each folder on its own mtime, so
# frequency costs nothing in safety, but a weekly run clears a whole week of
# newly-eligible orphans at once and can exceed DOWNLOAD_ORPHAN_MAX_DELETE_GB, which
# aborts the pass entirely and rolls the backlog into an even larger next run.
# frequency costs nothing in safety, while a weekly run makes each pass a whole
# week of newly-eligible orphans at once and far more likely to spill past
# DOWNLOAD_ORPHAN_MAX_DELETE_GB into a deferred remainder.
"Arrs_Stack/arr_download_orphan_cleaner.sh" # sweep orphaned completed downloads out of the SAB Completed folders — deletes junk + already-imported leftovers, triggers import scans for genuinely-missing content
# Runs after cleanup, not before — frees disk space from orphans/junk first, so the
# searches triggered below (for relocated hasFile=false/episodeFileCount=0 entries)
@@ -503,6 +503,11 @@
"Arrs_Stack/lidarr_missing_art.sh" # fetch missing album/artist artwork (HOST1 only — self-guards)
"Arrs_Stack/radarr_tmdb_removed.sh" # remove movies dropped from TMDb
"Arrs_Stack/sonarr_tvdb_removed.sh" # remove series dropped from TVDB
# Before the two docker jobs below, deliberately: they pull images and restart
# containers, so a container can legitimately not exist for a moment while they run.
# Reading docker in its settled overnight state keeps a rebuild from costing a strike.
# Three strikes at this cadence means three days genuinely absent before conf is edited.
"Tools/conf_container_prune.sh" # drop containers that no longer exist from the conf lists
"Docker_Essentials/docker_update.sh" # pull container image updates before restart
"Docker_Essentials/docker_daily_restart.sh" # daily container restarts — runs last
# Last, and after git_pull_execute.sh which must always run first. Daily rather than with
@@ -693,6 +698,13 @@
# lowered from 23 on 2026-07-16 — daily maintenance jobs alone now
# take ~4h; 19h cap leaves room for them to still run same-day
# before the next 1am fire even if a share hits the cap
RSYNC_MIN_FREE_GB=50 # warn if any disk backing the remote share is under this
# (0 disables). Online is not writable: a full destination
# makes rsync stall rather than fail. Warns, never aborts.
CONF_PRUNE_STRIKE_LIMIT=3 # consecutive runs a container must be absent from docker
# before its name is removed from the conf lists. A
# rebuild makes a container vanish briefly; one miss
# must never be enough to edit conf.
CRITICAL_CONTAINER_NAMES=() # containers stopped on REMOTE before rsync — profiles override
DELAYED_CONTAINERS=() # containers needing delay before starting — profiles override
CONTAINER_DELAY=5 # seconds before starting delayed containers
@@ -848,7 +860,34 @@
EXTERNAL_IP="8.8.8.8"
FALLBACK_CHECK_INTERVAL=30 # seconds between fallback state checks
FALLBACK_HANDBACK_STRIKES=3 # consecutive healthy checks before initiating handback (3×30s = 90s)
# Cost measured 2026-08-22: ~15ms CPU and 784 bytes per cycle
# per host (~2 MB/day, half of it across the mesh). The 2s of
# wall time per cycle is ping waiting, not work.
FALLBACK_HANDBACK_STRIKES=4 # consecutive healthy checks before initiating handback
# 4×30s = 2min. Raising this costs nothing — it requires more
# of the checks that already happen, it does not add any.
# Any failed check resets the counter, so the window is a
# genuine "stayed up this long", not "was up this often".
# Entry is deliberately unguarded — one false from ping_remote enters FALLBACK, with no strike
# count, because every second of a real outage is downtime and DDNS TTL is racing it. The
# confidence comes from the probe itself instead: see FALLBACK_PROBE_* below and _ping_retry()
# in common.sh.
FALLBACK_PROBE_RETRIES=3 # reachability attempts before declaring a host down
FALLBACK_PROBE_RETRY_DELAY=2 # seconds between those attempts
# Success returns on the first try, so a healthy cycle costs
# nothing extra. Only a failure pays: 3×4s + 2×2s = 16s worst
# case, which must stay under FALLBACK_CHECK_INTERVAL.
FALLBACK_DRY_RUN=false # run the daemon in PREVIEW mode, permanently
# Survives array start, unlike a --dry-run typed at a shell:
# array_started.sh launches with no arguments, so a hand-run
# preview is replaced by the LIVE daemon at the next boot.
# Decides nothing, starts nothing, moves no DNS, sends no
# notification — it only reports what it WOULD have done.
# Set true to observe real outages before arming for real.
FALLBACK_LOG_MAX_MB=5 # size cap for data/logs/fallback.log before one rotation
# Event-only output (the default, no --log) is a few lines
# per incident and holds years; --log fills this in about a
# fortnight and keeps the previous one as fallback.log.1
FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
FALLBACK_ENABLED=false # set true once both servers are configured and paired
# false = suppresses "not running" warnings in status scripts
@@ -1229,6 +1268,8 @@
LIDARR_MIN_TRACKED_PCT=80 # abort if tracked count drops below this % of last run
# protects against API returning partial data on a bad day
LIDARR_TRACKED_COUNT_FILE="${DB_DIR}/lidarr_tracked.count"
LIDARR_ORPHAN_STRIKES_FILE="${DB_DIR}/lidarr_orphan_strikes.tsv" # consecutive-classification counts, keyed by host path
LIDARR_ORPHAN_STRIKE_LIMIT=2 # consecutive runs a file must classify before it is removed
LIDARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan
# Lidarr tracked-data cache — shared by lidarr_cleanup.sh, lidarr_duplicate_artist_cleanup.sh,
@@ -1238,7 +1279,7 @@
LIDARR_RESCAN_DURATION_DB="${DB_DIR}/lidarr_rescan_duration.db"
LIDARR_CACHE_MAX_AGE_DAYS=1 # force a live refresh (or rescan-aware wait) past this age
ARR_PREFILL_WAIT_MINUTES=10 # array-start prefill: how long to retry reaching each arr
LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma" "wv" "ape" "aiff" "aif" "alac" "dsf" "dff" "mpc" "tta")
LIDARR_PROTECTED_PATTERNS=(
# Metadata
"*.nfo" "*.tbn"
@@ -1304,6 +1345,8 @@
SONARR_MIN_TRACKED_PCT=80 # abort if tracked count drops below this % of last run
# protects against API returning partial data on a bad day
SONARR_TRACKED_COUNT_FILE="${DB_DIR}/sonarr_tracked.count"
SONARR_ORPHAN_STRIKES_FILE="${DB_DIR}/sonarr_orphan_strikes.tsv" # consecutive-classification counts, keyed by host path
SONARR_ORPHAN_STRIKE_LIMIT=2 # consecutive runs a file must classify before it is removed
SONARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan
SONARR_MOVE_POLL_TIMEOUT=3600 # seconds to wait for a single async MoveSeries command to
# reach "completed" — generous because a large series can sit
@@ -1316,6 +1359,15 @@
# against a one-off ffprobe hiccup (mid-write file, NFS blip)
# triggering an unnecessary delete. Resets to 0 the moment a
# file probes clean again.
CORRUPTION_SCAN_MAX_CORRUPT_PCT=10 # abort the run, committing nothing, if this share of
# newly-scanned files reads as corrupt. A healthy library sits
# near zero; a high rate means the detector is wrong, not the
# library. Only counts ffprobe exit != 0.
CORRUPTION_SCAN_MAX_CONSECUTIVE=15 # abort after this many files in a row fail to probe
# cleanly — catches the ffprobe container dying mid-scan,
# which the pre-flight health check cannot see.
CORRUPTION_SCAN_GUARD_MIN_SCANNED=20 # below this many newly-scanned files the percentage
# ceiling is not applied — too small a sample to judge.
SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov")
SONARR_PROTECTED_PATTERNS=(
# Subtitles
@@ -1342,6 +1394,12 @@
RADARR_MIN_TRACKED_PCT=80 # abort if tracked count drops below this % of last run
# protects against API returning partial data on a bad day
RADARR_TRACKED_COUNT_FILE="${DB_DIR}/radarr_tracked.count"
RADARR_ORPHAN_STRIKES_FILE="${DB_DIR}/radarr_orphan_strikes.tsv" # consecutive-classification counts, keyed by host path
RADARR_ORPHAN_STRIKE_LIMIT=2 # consecutive runs a file must classify for deletion before it is
# removed. Gate 6 already catches an API returning far too few
# tracked files; this catches the partial failure too small to trip
# that percentage — one root folder failing to enumerate makes its
# movies look orphaned, and a transient one will not repeat.
RADARR_IMPORT_SCAN_TIMEOUT=600 # seconds to wait for pre-flight import scan
RADARR_MOVE_POLL_TIMEOUT=3600 # seconds to wait for a single async MoveMovie command to
# reach "completed" — mirrors SONARR_MOVE_POLL_TIMEOUT
@@ -1362,10 +1420,17 @@
# Per-host dirs: HOST*_SONARR_DOWNLOAD_DIR / HOST*_RADARR_DOWNLOAD_DIR (+ _CONTAINER_DIR).
DOWNLOAD_ORPHAN_CLEANER_ENABLED=true
DOWNLOAD_ORPHAN_AGE=7 # days — entries younger than this may be mid-import, never touched
DOWNLOAD_ORPHAN_MIN_VIDEO_MB=50 # no video file above this = junk (par2 debris, samples, dead archives)
DOWNLOAD_ORPHAN_MAX_DELETE_GB=100 # abort delete pass over this — a partial queue fetch would classify
# live downloads as orphans, and a huge total is that failure's symptom;
# --i-know-what-im-doing overrides for known backlogs
DOWNLOAD_ORPHAN_KEEP_MARKER=".vv-keep" # a file by this name inside a download folder pins it —
# never classified, never deleted. For lossless rips the
# library holds only as MP3, which REDUNDANT would sweep
DOWNLOAD_ORPHAN_MIN_VIDEO_MB=50 # no video file above this = junk (par2 debris, samples, dead archives) — Sonarr/Radarr
DOWNLOAD_ORPHAN_MIN_AUDIO_MB=2 # same test for Lidarr — a 50M floor would mark every album folder
# as junk, since single tracks rarely reach it
DOWNLOAD_ORPHAN_MAX_DELETE_GB=100 # per-run delete budget — a backlog above this drains
# safest-first (JUNK, REDUNDANT, then UNMATCHED) up to
# the budget and defers the rest to the next run.
# Live downloads are protected by DOWNLOAD_ORPHAN_AGE,
# not by this; --i-know-what-im-doing clears it in one pass
# ━━━ Arr Content Classification (radarr/sonarr_classification_scan.sh) ━━━
#
@@ -1601,6 +1666,10 @@
# HOST1_BACKUP_VERIFY_SHARES / HOST2_BACKUP_VERIFY_SHARES
BACKUP_VERIFY_SAMPLE=10 # random files to check per share
BACKUP_VERIFY_MIN_SIZE=1M # minimum file size to include in sample
BACKUP_VERIFY_MD5_TIMEOUT_MAX=600 # ceiling for one remote checksum. The per-file budget
# scales with size (~50MB/s); this caps it. A fixed
# connect-sized timeout killed multi-GB checksums and
# the empty result was then reported as MISSING.
# ━━━ SMART Health ━━━
# Monitors drive SMART attributes — discovers all drives via /dev/sd* and /dev/nvme*.
@@ -1979,7 +2048,10 @@
AI_ASSIST_REPORTS=false # tier 1 — digest / coffee report narration
AI_ASSIST_WATCHDOG=false # tier 2 — file a finding when a watchdog counter passes its limit (needs AI_REPAIR_ENABLED)
AI_ASSIST_DISCOVERY=false # tier 2 — discovery / classification judgement calls
AI_ASSIST_CLEANUP=false # tier 2 — orphan and stuck-import triage
AI_ASSIST_CLEANUP=false # tier 2 — orphan and stuck-import triage. Describes the shape of a
# classification in the log; decides nothing. Off = identical deletions.
AI_ASSIST_TIMEOUT=45 # seconds any single assist may take. An assist that can stall a
# nightly cleanup is not an assist — it is silently skipped past this.
AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance
# ━━━ AI Repair ━━━
+55
View File
@@ -28,6 +28,42 @@
# pointing at a directory the data is not in is worse than not having started.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Two halves, in order: move the files, then rewrite the conf keys that point at them. Doing it
# the other way round would leave every path variable naming a location nothing had reached yet,
# and any script that ran in between would create the old layout again underneath the new one.
#
# Idempotent. A path already under DATA_DIR is left alone, so a re-run after a partial migration
# finishes the job rather than moving things twice or failing on what is already done.
#
# One-time by intent, not by a marker file. There is no "already migrated" flag — the check is
# whether each individual path is already where it belongs, which is also what makes an
# interrupted run safe to repeat.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Existing keys are rewritten, which is why conf_upgrade cannot do this.
# conf_upgrade adds keys the template has and the installation does not, and never rewrites a
# value the operator already holds — correct for it, and exactly why it is the wrong tool here.
# STATE_DIR, BANDWIDTH_LOG, AI_INDEX_DB and two dozen more are existing keys whose values must
# change, or they would go on naming the old layout forever while the new directory variables
# sat beside them unused.
#
# Move, never copy-and-hope.
# The data being relocated is the only copy — statistics, histories, the AI index, arr caches.
# Everything is moved and the source is gone afterwards, so there is no second location that
# might still be written to by something that missed the change.
#
# The conf rewrite is the last thing, and the riskiest thing.
# Until it happens the installation still works from the old layout. That ordering means an
# abort partway through leaves a system that runs, rather than one whose paths point at
# nothing.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
@@ -49,6 +85,25 @@
# check costs nothing and the failure is silent otherwise.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# This script reads conf to find the old locations and rewrites conf to record the new ones. It
# is the one script here whose purpose is to change these values rather than obey them.
#
# Read to locate what moves
# STATE_DIR, BANDWIDTH_LOG, AI_INDEX_DB, AI_MEMORY_FILE, AI_TOKEN_DB, ARR_CLEANUP_STATS,
# ARR_SYNC_BLOCKLIST, CORRUPTION_SCAN_STATE_FILE, LIDARR_CACHE_FILE, ZFS_REPORT_LOG and the
# rest of the per-script path keys — roughly two dozen in total.
#
# Written as the new roots
# DATA_DIR and the directories beneath it: DB_DIR, STATE_DIR, AI_DATA_DIR,
# CACHE_BACKUP_DIR, ARR_CACHE_BACKUP_DIR, CONF_CACHE_BACKUP_DIR, LOG_ARCHIVE_DIR.
#
# Every rewritten value is expressed as ${DB_DIR}/… rather than an absolute path, so a later
# storage-mode migration moves them again by changing one variable.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
+351
View File
@@ -0,0 +1,351 @@
#!/bin/bash
# ══════════════════════════════════════════════════════════════════════════════════════════════
# PURPOSE
# Put the containers this host has marked for fallback coverage onto the partner, so that the
# partner can actually start them during an outage — and take them off again on request.
#
# OPERATIONAL MODEL
# fallback.sh covers a host by running `docker start <name>` on the partner. It never creates
# anything. So a name in FALLBACK_<me>_TIER* is a promise that only holds if the partner already
# has that container built. Measured 2026-08-23: all 12 of HOST1's covered containers were absent
# from HOST2, meaning every tier would have failed on the first real outage while the UI showed
# coverage as configured. This script is what closes that gap.
#
# Push and remove are separate, deliberate actions, never a side effect of saving the tier list.
# Editing coverage is a cheap config write; deploying a dozen containers onto another machine is
# not, and the two should not share a button.
#
# DESIGN PRINCIPLES
# Deployed, then verified STOPPED.
# A container built here and left running on the partner would be a second live instance of
# NextCloud, Gitea or PostgreSQL_Immich against the same data while this host is healthy.
# That is the danger_rsync_live_database_appdata failure with worse odds. Every deploy is
# followed by a stop and a re-inspect, and a container that will not stay stopped is an
# error, not a warning.
#
# Remove takes the container AND its appdata.
# Operator decision 2026-08-23: the button is explicit, so a removal should leave nothing
# behind to reason about later. The risk it accepts is narrow and worth naming — if the
# partner ever covered for us, ITS appdata is the newer copy and is what a handback rsyncs
# home. The NORMAL-state gate below closes the live-failover window; what it cannot see is
# a handback that partially failed and then returned to NORMAL, so the UI says so before
# asking.
#
# Two guards on the deletion itself: only paths under /mnt/*/appdata* are ever touched, and
# a bind of the appdata ROOT is refused outright — a container mounting /mnt/user/appdata
# would otherwise turn one removal into wiping every application on the partner.
#
# Refuses to run unless fallback state is NORMAL.
# Pushing or removing containers mid-outage edits the thing currently keeping services up.
#
# Coverage names are resolved to templates by <Name>, not by filename.
# my-Foo.xml routinely holds a container called something else. Matching on the filename
# silently pushes the wrong template, or nothing at all.
#
# OPERATIONAL SAFEGUARDS
# Only in NORMAL state. FALLBACK_STATE_FILE is read before anything is pushed or removed, and
# any other state refuses the action. A push during a live failover would deploy a second copy
# of a container the partner is currently running on our behalf; a remove would delete the one
# doing the covering.
#
# --status is exempt from that gate, because it only reports. Refusing to answer "what is
# deployed over there" during a failover would withhold the information precisely when it is
# most wanted.
#
# Every deploy is verified stopped, and a container that will not stay stopped is an error
# rather than a warning — see DESIGN PRINCIPLES. A second live instance against the same data
# is the failure this whole script exists inside.
#
# Push and remove are explicit modes with no default. Running the script with no flag does
# nothing; neither action can be reached by accident, and neither is a side effect of editing
# the tier list.
#
# --dry-run works in every mode and touches nothing on either host — no container is built,
# started, stopped or removed, and no template is written or deleted.
#
# Remove deletes the container's appdata on the partner as well. That is deliberate and is the
# most destructive thing here; the NORMAL-state gate above is what keeps it away from a
# partner that is mid-handback.
#
# CONFIGURATION
# master.conf
# FALLBACK_<HOST>_TIER1..N the covered container names — what --push deploys and --status
# reports on. This script reads that list; it never edits it.
#
# host*.conf
# FALLBACK_STATE_FILE overrides where fallback.sh's state is read from. Defaults to
# STATE_DIR/fallback_state.db. A missing file reads as NORMAL,
# which is the correct default on a host where fallback has never
# run.
#
# RUNTIME MODES
# coverage_deploy.sh --push deploy every covered container onto the partner (stopped)
# coverage_deploy.sh --remove stop, remove, and delete the pushed template on the partner
# coverage_deploy.sh --status report, per covered container, whether it exists there
# any mode supports --dry-run
#
# DEPENDS ON
# Plugin/<platform>/Partnership/containers.sh deploy_container_from_xml(), GPU transform
# FALLBACK_<me>_TIER1-4 the coverage list this acts on
# ══════════════════════════════════════════════════════════════════════════════════════════════
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
source "$SCRIPT_DIR/../Plugin/$PLATFORM/Partnership/containers.sh"
SSH_TIMEOUT="${SSH_TIMEOUT:-15}"
MODE=""
DRY_RUN="${DRY_RUN:-false}"
for arg in "$@"; do
case "$arg" in
--push) MODE="push" ;;
--remove) MODE="remove" ;;
--status) MODE="status" ;;
--dry-run) DRY_RUN=true ;;
esac
done
if [[ -z "$MODE" ]]; then
error "No mode given — use --push, --remove or --status"
exit 1
fi
detect_hosts
if [[ -z "$REMOTE_ID" || "$REMOTE_SERVER_NAME" == "unknown" ]]; then
error "No partner configured — nothing to push to"
exit 1
fi
# ── Gate: only with fallback idle ─────────────────────────────────────────────────────────────
# Read rather than assumed. A missing state file means fallback has never run, which is idle
# enough; a file that says anything other than NORMAL means services are in motion right now.
FALLBACK_STATE_FILE="${FALLBACK_STATE_FILE:-${STATE_DIR}/fallback_state.db}"
_fb_state="NORMAL"
if [[ -f "$FALLBACK_STATE_FILE" ]]; then
_fb_state=$(grep -m1 '^state=' "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
_fb_state="${_fb_state:-NORMAL}"
fi
if [[ "$_fb_state" != "NORMAL" && "$MODE" != "status" ]]; then
error "Fallback state is $_fb_state, not NORMAL — refusing to $MODE"
error "Changing what the partner holds while a failover is live edits the thing keeping services up."
exit 1
fi
# ── The coverage list ─────────────────────────────────────────────────────────────────────────
COVERED=()
for _t in 1 2 3 4; do
_var="FALLBACK_${MY_ID}_TIER${_t}[@]"
for _c in "${!_var}"; do
[[ -n "$_c" ]] && COVERED+=("$_c")
done
done
if [[ ${#COVERED[@]} -eq 0 ]]; then
warn "No containers are covered in FALLBACK_${MY_ID}_TIER1-4 — nothing to do"
exit 0
fi
log "$ICON_FALLBACK Coverage: ${#COVERED[@]} container(s) for $REMOTE_SERVER_NAME to start during an outage"
resolve_remote_ip
MIRROR="$REMOTE_SERVER_NAME"
MIRROR_IP="$REMOTE_SERVER"
_key_var="${MY_ID}_SSH_KEY"
MIRROR_SSH_KEY="${!_key_var}"
if [[ ! -f "$MIRROR_SSH_KEY" ]]; then
error "SSH key $MIRROR_SSH_KEY not found — cannot reach $MIRROR"
exit 1
fi
# ── name -> template ──────────────────────────────────────────────────────────────────────────
# Matched on the <Name> element. Filenames lie often enough that trusting them would push the
# wrong container without saying so.
xml_for_container() {
local want="$1" f n
for f in "$TEMPLATES_DIR"/*.xml; do
[[ -f "$f" ]] || continue
n=$(awk 'match($0,/<Name>([^<]+)<\/Name>/,a){print a[1];exit}' "$f")
[[ "$n" == "$want" ]] && { echo "$f"; return 0; }
done
return 1
}
remote_has_container() {
timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
-o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
"docker inspect $(printf '%q' "$1") >/dev/null 2>&1" 2>/dev/null
}
remote_state_of() {
timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
-o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
"docker inspect -f '{{.State.Status}}' $(printf '%q' "$1") 2>/dev/null" 2>/dev/null
}
OK=0; FAIL=0; SKIP=0
case "$MODE" in
status)
# Written as a cache as well as printed. The assistant's fallback_state block cannot afford an
# SSH round trip per container mid-question, so it reads this file and reports its AGE — a stale
# answer stated as stale is useful, stated as current it is the exact failure this feature
# exists to prevent.
_present="" _missing=""
for c in "${COVERED[@]}"; do
if remote_has_container "$c"; then
_st=$(remote_state_of "$c")
printf ' %-28s on %s (%s)\n' "$c" "$MIRROR" "$_st"
_present+="\"$c\":\"${_st:-unknown}\","
OK=$((OK+1))
else
printf ' %-28s MISSING on %s — docker start would fail\n' "$c" "$MIRROR"
_missing+="\"$c\","
FAIL=$((FAIL+1))
fi
done
log "$ICON_FALLBACK Coverage present: $OK · missing: $FAIL"
mkdir -p "$VV_CACHE_ROOT/api" 2>/dev/null || mkdir -p /tmp/varaverk/api 2>/dev/null
_cache="${VV_CACHE_ROOT:-/tmp/varaverk}/api/fallback_presence.json"
# Written atomically — a half-written cache read mid-question would report containers as
# missing that are merely unparsed.
printf '{"present":{%s},"missing":[%s],"partner":"%s","checked":%s}\n' \
"${_present%,}" "${_missing%,}" "$MIRROR" "$(date +%s)" > "$_cache.tmp" \
&& mv -f "$_cache.tmp" "$_cache"
[[ "$FAIL" -gt 0 ]] && exit 2 || exit 0
;;
push)
# Networks first — a container whose network is absent is created and then cannot start,
# which is the failure that read as "auth 0/8, arr 0/5" during onboarding.
_nets=()
for c in "${COVERED[@]}"; do
x=$(xml_for_container "$c") || continue
net=$(sed -n 's/.*<Network>\([^<]*\)<\/Network>.*/\1/p' "$x" 2>/dev/null | head -1)
net="${net//[[:space:]]/}"
# br* is host hardware. wg* is a WireGuard-backed bridge whose meaning does NOT travel:
# recreating it on the partner as a plain bridge yields a network that exists, starts its
# containers, and routes their traffic OUTSIDE the tunnel. ChannelTube rides wg0 here.
case "$net" in
''|bridge|host|none|br[0-9]*) continue ;;
wg[0-9]*)
warn "$c uses $net — a WireGuard-backed network. NOT created on $MIRROR: a plain"
warn " bridge of the same name would route its traffic outside the tunnel. Build the"
warn " matching tunnel there first, or drop $c from coverage."
continue ;;
esac
_seen=false
for n in "${_nets[@]}"; do [[ "$n" == "$net" ]] && { _seen=true; break; }; done
[[ "$_seen" == false ]] && _nets+=("$net")
done
for net in "${_nets[@]}"; do
driver=$(timeout "${DOCKER_TIMEOUT:-30}" docker network inspect "$net" --format '{{.Driver}}' 2>/dev/null)
if [[ "$driver" != "bridge" ]]; then
warn "Network $net is '${driver:-absent}' here, not bridge — create it on $MIRROR by hand"
continue
fi
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would ensure network $net on $MIRROR"
continue
fi
timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
-o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
"docker network inspect $(printf '%q' "$net") >/dev/null 2>&1 \
|| docker network create --driver bridge $(printf '%q' "$net") >/dev/null" 2>/dev/null \
&& log " network $net ready on $MIRROR" \
|| warn " could not ensure network $net on $MIRROR"
done
for c in "${COVERED[@]}"; do
x=$(xml_for_container "$c") || {
warn "$c — no template in $TEMPLATES_DIR names it; skipped"
SKIP=$((SKIP+1)); continue
}
if ! deploy_container_from_xml "$x" "$MIRROR_IP" "$MIRROR_SSH_KEY"; then
error "$c — deploy failed"
FAIL=$((FAIL+1)); continue
fi
if [[ "$DRY_RUN" == true ]]; then OK=$((OK+1)); continue; fi
# Deployed containers must not run here. Stop, then re-inspect — a stop that did not take
# is the one outcome that silently duplicates a live service against shared data.
timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
-o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
"docker stop $(printf '%q' "$c") >/dev/null 2>&1" 2>/dev/null
st=$(remote_state_of "$c")
if [[ "$st" == "running" ]]; then
error "$c is RUNNING on $MIRROR after deploy and would not stop — stop it there before continuing"
FAIL=$((FAIL+1))
else
log " $c deployed and ${st:-stopped} on $MIRROR"
OK=$((OK+1))
fi
done
log "$ICON_FALLBACK Push complete — deployed $OK · failed $FAIL · skipped $SKIP"
[[ "$FAIL" -gt 0 ]] && exit 1 || exit 0
;;
remove)
for c in "${COVERED[@]}"; do
if ! remote_has_container "$c"; then
log " $c not on $MIRROR — nothing to remove"
SKIP=$((SKIP+1)); continue
fi
# Binds are read BEFORE the container goes — once it is removed there is nothing left to
# enumerate, and a path list gathered afterwards would silently be empty.
_binds=$(timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
-o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
"docker inspect --format '{{range .HostConfig.Binds}}{{println .}}{{end}}' $(printf '%q' "$c") 2>/dev/null \
| awk -F: '{print \$1}'" 2>/dev/null)
_wipe=()
while IFS= read -r _p; do
[[ -z "$_p" ]] && continue
# Only appdata, and never an appdata root. /mnt/user/appdata as a bind would make one
# container removal delete every application on the partner.
[[ "$_p" =~ ^/mnt/[^/]+/appdata[^/]*/.+ ]] || {
[[ "$_p" =~ ^/mnt/[^/]+/appdata[^/]*/?$ ]] && \
warn " $c binds the appdata ROOT ($_p) — refusing to delete it"
continue
}
_wipe+=("$_p")
done <<< "$_binds"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would stop and remove $c on $MIRROR"
for _p in "${_wipe[@]}"; do warn " DRY RUN — would delete appdata $_p on $MIRROR"; done
OK=$((OK+1)); continue
fi
x=$(xml_for_container "$c") && xml_name=$(basename "$x") || xml_name=""
if timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
-o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
"docker stop $(printf '%q' "$c") >/dev/null 2>&1; \
docker rm $(printf '%q' "$c") >/dev/null 2>&1; \
${xml_name:+rm -f ${TEMPLATES_DIR}/$(printf '%q' "$xml_name");} \
! docker inspect $(printf '%q' "$c") >/dev/null 2>&1" 2>/dev/null; then
log " $c removed from $MIRROR"
for _p in "${_wipe[@]}"; do
if timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" -o ConnectTimeout="$SSH_TIMEOUT" \
-o BatchMode=yes -o StrictHostKeyChecking=no root@"$MIRROR_IP" \
"rm -rf -- $(printf '%q' "$_p") && ! [ -e $(printf '%q' "$_p") ]" 2>/dev/null; then
log " appdata deleted on $MIRROR: $_p"
else
warn " could not delete appdata on $MIRROR: $_p"
FAIL=$((FAIL+1))
fi
done
OK=$((OK+1))
else
error "$c — removal failed or it still exists on $MIRROR"
FAIL=$((FAIL+1))
fi
done
log "$ICON_FALLBACK Remove complete — removed $OK · failed $FAIL · skipped $SKIP"
[[ "$FAIL" -gt 0 ]] && exit 1 || exit 0
;;
esac
+84 -3
View File
@@ -314,6 +314,50 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
# ── Persistent dry-run ────────────────────────────────────────────────────────────────────────
# Conf-driven, not argument-driven, deliberately. array_started.sh launches every entry as a bare
# `bash script.sh &` with no arguments, so a --dry-run typed at a terminal survives exactly until
# the next array start — and then the LIVE daemon comes up in its place, silently, which is the
# one transition nobody would be watching for.
#
# Setting it here means the mode is a property of the install rather than of how the process
# happened to be started: array start, the Fallback tab's button, and a hand-run all agree.
#
# OR, never override: --dry-run on the command line still wins over a conf that says false, so an
# ad-hoc preview against a live install needs no conf edit.
if [[ "${FALLBACK_DRY_RUN:-false}" == "true" ]]; then
DRY_RUN=true
fi
# ── Persistent log ────────────────────────────────────────────────────────────────────────────
# array_started.sh launches every entry as a bare `bash script.sh &` with no redirection, so this
# daemon's output has never been captured anywhere: /var/log/varaverk has a directory for every
# other script family and none for Fallback. A month of dry-run observation would have persisted
# nothing at all.
#
# /var/log is a 128 MB tmpfs on Unraid — RAM, and cleared on reboot — so the log goes to
# LOG_ARCHIVE_DIR, which follows DATA_DIR onto real storage.
#
# Only when stdout is not a terminal. Run by hand, output still goes to the terminal exactly as
# before; run by array_started or the Fallback tab's button, it lands in the file. A plain append
# redirect rather than `tee` through process substitution: no extra child to outlive, and nothing
# for the shutdown trap to race.
FALLBACK_LOG="${LOG_ARCHIVE_DIR:-${DATA_DIR:-/tmp}/logs}/fallback.log"
if [[ ! -t 1 ]]; then
mkdir -p "$(dirname "$FALLBACK_LOG")" 2>/dev/null
# One rotation, sized rather than line-counted — the whole point of this log is a long run,
# and _orch_trim_log()'s 1000-line cap would discard weeks of it. Event-only output (no
# --log) is a few lines per incident, so this holds years; --log fills it in about a fortnight
# and then keeps the most recent fortnight plus the one before it.
_fb_max=$(( ${FALLBACK_LOG_MAX_MB:-5} * 1048576 ))
if [[ -f "$FALLBACK_LOG" ]] && (( $(stat -c %s "$FALLBACK_LOG" 2>/dev/null || echo 0) > _fb_max )); then
mv -f "$FALLBACK_LOG" "${FALLBACK_LOG}.1" 2>/dev/null
fi
exec >> "$FALLBACK_LOG" 2>&1
echo ""
echo "═══ fallback.sh started $(date '+%Y-%m-%d %H:%M:%S') — dry_run=${DRY_RUN} pid=$$ ═══"
fi
# FALLBACK_ENABLED gate — exits cleanly when disabled.
# Fail-closed: anything that isn't exactly "true" disables fallback. Matching only the
# literal "false" would let a typo ("no", "0", "FALSE") hand this script DDNS authority
@@ -368,6 +412,22 @@ state_set() {
}
state_init() {
# A dry run must not leave the host believing it failed over. state_set() writes
# unconditionally, and this file survives reboots and is what the real daemon — and the
# Monitor and Fallback cards — read to decide what is happening. A --dry-run walk through
# FAILOVER would have written state=FALLBACK, the tier flags and the strike counter into it
# for real, and nothing would have put them back.
#
# Copied rather than merely redirected, so the preview still starts from the live state and
# can advance through tiers exactly as a real run would. The copy lands in the RAM cache and
# dies with the reboot.
if [[ "$DRY_RUN" == true ]]; then
local live="$FALLBACK_STATE_FILE"
FALLBACK_STATE_FILE="${VV_CACHE_ROOT:-/tmp/varaverk}/fallback_state.dryrun.$$"
mkdir -p "$(dirname "$FALLBACK_STATE_FILE")"
if [[ -f "$live" ]]; then cp -f "$live" "$FALLBACK_STATE_FILE"; else : > "$FALLBACK_STATE_FILE"; fi
warn "DRY RUN — state writes redirected to $FALLBACK_STATE_FILE (live state untouched)"
fi
mkdir -p "$(dirname "$FALLBACK_STATE_FILE")"
[[ ! -f "$FALLBACK_STATE_FILE" ]] && touch "$FALLBACK_STATE_FILE"
[[ -z "$(state_get state)" ]] && state_set state "NORMAL"
@@ -759,8 +819,16 @@ run_handback() {
for job in "${jobs[@]}"; do
[[ -z "$job" ]] && continue
log "Syncing: $job"
[[ "$DRY_RUN" == false ]] && bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$job" \
|| warn "DRY RUN — would rsync: $job"
# if/else, not A && B || C. In the shorthand a REAL run whose rsync exits
# non-zero falls through to the || branch and logs "DRY RUN — would rsync",
# so a failed Tier writeback reported itself as a preview and the real
# failure went unsaid. The Tier 1 block below always had this right.
if [[ "$DRY_RUN" == false ]]; then
bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$job" \
|| error "Tier $tier writeback FAILED: $job"
else
warn "DRY RUN — would rsync: $job"
fi
done
else
log "Tier $tier writeback skipped — outage ${outage_minutes}min < ${threshold}min"
@@ -904,6 +972,19 @@ echo " $ICON_NET Remote IP: $REMOTE_SERVER"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
FALLBACK_RUNNING=true
# The dry-run state copy is per-PID and would otherwise accumulate one file per preview run.
# EXIT as well as the signals, because a dry run is usually ended with Ctrl-C or --stop but can
# also just fall out of the loop.
dryrun_state_cleanup() {
[[ "$DRY_RUN" == true && "$FALLBACK_STATE_FILE" == *".dryrun."* ]] && rm -f "$FALLBACK_STATE_FILE"
return 0
}
# Must call _release_all_locks too. acquire_lock() registers its own EXIT trap, and bash keeps
# exactly one per signal — a bare `trap ... EXIT` here silently replaced it and orphaned
# fallback.lock, which is the precise failure the _LOCK_FILES registry in common.sh was built to
# stop. The signal trap only needs `exit 0`; that fires EXIT, which does both jobs.
trap 'dryrun_state_cleanup; _release_all_locks' EXIT
trap 'FALLBACK_RUNNING=false; warn "Fallback received shutdown signal — stopping cleanly"; exit 0' \
SIGTERM SIGINT
@@ -1185,4 +1266,4 @@ while [[ "$FALLBACK_RUNNING" == true ]]; do
sleep "$FALLBACK_CHECK_INTERVAL" &
wait $!
done
done
+78
View File
@@ -118,6 +118,11 @@
# fallback_test.sh --log
# Verbose output on every check in every phase.
#
# fallback_test.sh --stop
# Stop a running test. SIGTERM only — never SIGKILL, because only this script's EXIT
# trap removes the iptables DROP rule it installed. Also sweeps a rule stranded by an
# earlier SIGKILL or power cut.
#
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -126,6 +131,58 @@ source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
# ── Stop mode — runs before acquire_lock so we can target the holding instance ────────────────
#
# SIGTERM ONLY, and deliberately no SIGKILL escalation — the opposite of fallback.sh --stop.
# A running test holds an iptables DROP rule against the partner, and the only thing that removes
# it is this script's own EXIT trap. SIGKILL does not run traps, so force-killing a test strands
# the rule: the partner stays invisible, fallback.sh reads that as a permanent outage and holds
# FALLBACK indefinitely. A test that will not die is a worse outcome than a test still running,
# so this reports the stranded rule and the command to clear it rather than causing one.
if [[ " ${PARSED_ARGS[*]:-} " == *" --stop "* ]]; then
LOCKFILE="${LOCK_DIR}/fallback_test.lock"
if [[ ! -f "$LOCKFILE" ]]; then
log "No fallback_test.sh lock found — not running"
exit 0
fi
lock_content=$(cat "$LOCKFILE" 2>/dev/null)
target_pid="${lock_content%%:*}"
if [[ -z "$target_pid" ]] || ! kill -0 "$target_pid" 2>/dev/null; then
warn "Stale lock — fallback_test.sh not running (PID ${target_pid:-unknown} gone) — clearing"
rm -f "$LOCKFILE"
# A stale lock is exactly the SIGKILL/power-cut case, so the rule may still be in place.
if iptables -C OUTPUT -d "${REMOTE_SERVER:-0.0.0.0}" -j DROP 2>/dev/null; then
warn "Stranded iptables DROP rule found for $REMOTE_SERVER — removing"
iptables -D OUTPUT -d "$REMOTE_SERVER" -j DROP 2>/dev/null \
&& warn "Stranded rule removed — remote connectivity restored ✅" \
|| error "Could not remove stranded rule — run: iptables -D OUTPUT -d $REMOTE_SERVER -j DROP"
fi
exit 0
fi
warn "Stopping fallback_test.sh (PID $target_pid) — SIGTERM so its trap clears the iptables rule..."
kill -TERM "$target_pid" 2>/dev/null || true
waited=0
while kill -0 "$target_pid" 2>/dev/null && [[ "$waited" -lt 30 ]]; do
sleep 1
(( waited++ )) || true
done
if kill -0 "$target_pid" 2>/dev/null; then
error "fallback_test.sh (PID $target_pid) did not exit within 30s"
error "NOT force-killing — SIGKILL would strand the iptables DROP rule on $REMOTE_SERVER"
error "Wait, or clear manually: iptables -D OUTPUT -d $REMOTE_SERVER -j DROP"
exit 1
fi
warn "Stopped: fallback_test.sh (PID $target_pid) ✅"
if iptables -C OUTPUT -d "${REMOTE_SERVER:-0.0.0.0}" -j DROP 2>/dev/null; then
error "iptables DROP rule for $REMOTE_SERVER survived the stop — removing"
iptables -D OUTPUT -d "$REMOTE_SERVER" -j DROP 2>/dev/null \
&& warn "Rule removed ✅" || error "Could not remove — run it by hand"
else
log "No iptables DROP rule remains for $REMOTE_SERVER"
fi
exit 0
fi
FALLBACK_SCRIPT="$SCRIPT_DIR/fallback.sh"
DOCKER_TIMEOUT=15
@@ -325,6 +382,27 @@ else
warn "No state file found — assuming NORMAL (first run)"
fi
# fallback.sh must actually be RUNNING, not merely enabled
#
# Every phase after this one waits for the daemon to change state. FALLBACK_ENABLED=true says
# it is allowed to run; it does not say array_started.sh launched it, or that it is still alive.
# Without this the test passes pre-flight, drops a real iptables rule on the partner, waits
# FALLBACK_TEST_BLOCK_WAIT for a transition nothing is there to make, and fails Phase 3 blaming
# fallback detection. Only the EXIT trap gets connectivity back.
#
# In --dry-run nothing is blocked and nothing is waited on, so a dead daemon is worth saying but
# not worth aborting for — the walkthrough still shows the operator the shape of the run.
if pgrep -f "Fallback/fallback\.sh" >/dev/null 2>&1; then
log "fallback.sh daemon is running"
elif [[ "$DRY_RUN" == true ]]; then
warn "fallback.sh is NOT running — a real test would abort here"
else
error "fallback.sh is not running — nothing would detect the outage this test creates"
error "Start it with array_started.sh, or run with --dry-run to walk the phases"
phase_fail "Pre-flight"
exit 1
fi
# Tier 1 containers configured
if [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then
error "No Tier 1 containers configured for $MY_ID$REMOTE_ID"
Executable → Regular
+46 -10
View File
@@ -122,6 +122,7 @@ source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
SSH_TIMEOUT=15
BACKUP_VERIFY_MD5_TIMEOUT_MAX="${BACKUP_VERIFY_MD5_TIMEOUT_MAX:-600}"
# ==============================================================================================
# ━━━ Setup ━━━
@@ -228,6 +229,7 @@ TOTAL_CHECKED=0
TOTAL_MATCH=0
TOTAL_MISMATCH=0
TOTAL_MISSING=0
TOTAL_UNVERIFIED=0
SHARES_WITH_ISSUES=()
for share in "${VERIFY_SHARES[@]}"; do
@@ -265,6 +267,7 @@ for share in "${VERIFY_SHARES[@]}"; do
SHARE_MATCH=0
SHARE_MISMATCH=0
SHARE_MISSING=0
SHARE_UNVERIFIED=0
for local_file in "${SAMPLE_FILES[@]}"; do
[[ -z "$local_file" ]] && continue
@@ -276,19 +279,49 @@ for share in "${VERIFY_SHARES[@]}"; do
continue
fi
# Remote checksum via SSH — timeout protected
remote_md5=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
# The path is interpolated into a remote shell command, so it must be escaped for
# reuse as one word. A bare '$local_file' inside single quotes breaks on the first
# apostrophe — "Frieren - Beyond Journey's End" ended the quote early, md5sum fell
# back to reading stdin, and the empty-input hash d41d8cd9... was reported as a
# MISMATCH against a file that is byte-identical on the remote.
printf -v remote_q '%q' "$local_file"
# Existence and content are separate questions. Asking them together means a slow
# checksum is indistinguishable from an absent file.
remote_exists=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
-o ConnectTimeout="$SSH_TIMEOUT" \
-o StrictHostKeyChecking=no \
root@"$REMOTE_SERVER" \
"md5sum '$local_file' 2>/dev/null | awk '{print \$1}'" 2>/dev/null)
"test -f $remote_q && echo yes" 2>/dev/null </dev/null)
(( TOTAL_CHECKED++ ))
if [[ -z "$remote_md5" ]]; then
if [[ "$remote_exists" != "yes" ]]; then
warn "$ICON_ERROR MISSING: $(basename "$local_file")"
(( SHARE_MISSING++ ))
(( TOTAL_MISSING++ ))
continue
fi
# md5sum of a multi-GB file cannot finish inside a connect-sized timeout. Budget by
# size — a 5.9GB file needs ~30s and was being killed at 15s, then counted MISSING
# even though it was present and correct.
local_size=$(stat -c%s "$local_file" 2>/dev/null || echo 0)
md5_timeout=$(( local_size / 52428800 + SSH_TIMEOUT ))
(( md5_timeout > BACKUP_VERIFY_MD5_TIMEOUT_MAX )) && md5_timeout=$BACKUP_VERIFY_MD5_TIMEOUT_MAX
remote_md5=$(timeout "$md5_timeout" ssh -i "$SSH_KEY" \
-o ConnectTimeout="$SSH_TIMEOUT" \
-o StrictHostKeyChecking=no \
root@"$REMOTE_SERVER" \
"md5sum $remote_q 2>/dev/null | awk '{print \$1}'" 2>/dev/null </dev/null)
if [[ -z "$remote_md5" ]]; then
# Present but unreadable within budget. Reporting this as a mismatch or a miss
# would be a claim the run did not earn.
warn "$ICON_WARN UNVERIFIED (checksum timed out after ${md5_timeout}s): $(basename "$local_file")"
(( SHARE_UNVERIFIED++ ))
(( TOTAL_UNVERIFIED++ ))
elif [[ "$local_md5" == "$remote_md5" ]]; then
log "MATCH: $(basename "$local_file")"
(( SHARE_MATCH++ ))
@@ -303,8 +336,8 @@ for share in "${VERIFY_SHARES[@]}"; do
done
# Per-share result — only visible if issues found
if [[ "$SHARE_MISMATCH" -gt 0 || "$SHARE_MISSING" -gt 0 ]]; then
warn "$SHARE_NAME — match: $SHARE_MATCH missing: $SHARE_MISSING mismatch: $SHARE_MISMATCH"
if [[ "$SHARE_MISMATCH" -gt 0 || "$SHARE_MISSING" -gt 0 || "$SHARE_UNVERIFIED" -gt 0 ]]; then
warn "$SHARE_NAME — match: $SHARE_MATCH missing: $SHARE_MISSING mismatch: $SHARE_MISMATCH unverified: $SHARE_UNVERIFIED"
SHARES_WITH_ISSUES+=("$SHARE_NAME")
else
echo "$SHARE_NAME — all $SHARE_MATCH files match ✅"
@@ -325,10 +358,11 @@ echo "$ICON_VERIFY Checked: $TOTAL_CHECKED files"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
if [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 ]]; then
echo "$ICON_SUCCESS Match: $TOTAL_MATCH"
warn "Missing: $TOTAL_MISSING"
[[ "$TOTAL_MISMATCH" -gt 0 ]] && echo "$ICON_ERROR Mismatch: $TOTAL_MISMATCH"
if [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 || "$TOTAL_UNVERIFIED" -gt 0 ]]; then
echo "$ICON_SUCCESS Match: $TOTAL_MATCH"
warn "Missing: $TOTAL_MISSING"
[[ "$TOTAL_MISMATCH" -gt 0 ]] && echo "$ICON_ERROR Mismatch: $TOTAL_MISMATCH"
[[ "$TOTAL_UNVERIFIED" -gt 0 ]] && warn "Unverified: $TOTAL_UNVERIFIED (present, checksum timed out)"
fi
if [[ "$DRY_RUN" == true ]]; then
@@ -337,6 +371,8 @@ elif [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 ]]; then
echo "$ICON_ERROR Status: ISSUES FOUND — ${#SHARES_WITH_ISSUES[@]} share(s) need attention: ${SHARES_WITH_ISSUES[*]}"
notify "Backup verify FAILED on $(hostname)$REMOTE_SERVER_NAME — mismatches: $TOTAL_MISMATCH missing: $TOTAL_MISSING — shares: ${SHARES_WITH_ISSUES[*]}" \
"Backup Verify" "warning"
elif [[ "$TOTAL_UNVERIFIED" -gt 0 ]]; then
warn "Status: $TOTAL_MATCH verified, $TOTAL_UNVERIFIED could not be checksummed in time — NOT a clean run"
else
echo "$ICON_DONE Status: all $TOTAL_CHECKED files match across ${#VERIFY_SHARES[@]} shares ✅"
fi
+17 -17
View File
@@ -18,6 +18,22 @@
# WebGUI slowdowns or timeouts under load.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Each run snapshots:
# inotify: instances in use vs INOTIFY_MAX_INSTANCES kernel limit.
# Top 5 consumers by instance count. Warns above INOTIFY_WARN_PCT.
# php-fpm: active workers vs PHP_MAX_CHILDREN limit.
# Warns above PHP_FPM_WARN_PCT.
#
# Log line format (one per run, trimmed to TUNING_LOG_RETENTION days):
# DATE|TIME|INOTIFY_USED|INOTIFY_LIMIT|INOTIFY_PCT|INOTIFY_WARN|
# PHPFPM_ACTIVE|PHPFPM_MAX|PHPFPM_PCT|PHPFPM_WARN
# INOTIFY_WARN and PHPFPM_WARN are 1/0 flags. weekly_health_digest.sh counts
# warnings over the week to show trend severity.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
@@ -36,22 +52,6 @@
# signal — routine snapshots below the threshold produce nothing.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Each run snapshots:
# inotify: instances in use vs INOTIFY_MAX_INSTANCES kernel limit.
# Top 5 consumers by instance count. Warns above INOTIFY_WARN_PCT.
# php-fpm: active workers vs PHP_MAX_CHILDREN limit.
# Warns above PHP_FPM_WARN_PCT.
#
# Log line format (one per run, trimmed to TUNING_LOG_RETENTION days):
# DATE|TIME|INOTIFY_USED|INOTIFY_LIMIT|INOTIFY_PCT|INOTIFY_WARN|
# PHPFPM_ACTIVE|PHPFPM_MAX|PHPFPM_PCT|PHPFPM_WARN
# INOTIFY_WARN and PHPFPM_WARN are 1/0 flags. weekly_health_digest.sh counts
# warnings over the week to show trend severity.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
@@ -259,4 +259,4 @@ echo "${DATE}|${TIME}|${INOTIFY_USED}|${INOTIFY_LIMIT}|${INOTIFY_PCT}|${INOTIFY_
>> "$TUNING_MONITOR_LOG"
echo "Snapshot written: inotify ${INOTIFY_PCT}% php-fpm ${PHPFPM_PCT}%"
log "Entry: ${DATE}|${TIME}|${INOTIFY_USED}/${INOTIFY_LIMIT}(${INOTIFY_PCT}%,warn=${INOTIFY_WARN})|${PHPFPM_ACTIVE}/${PHPFPM_MAX}(${PHPFPM_PCT}%,warn=${PHPFPM_WARN})"
log "Entry: ${DATE}|${TIME}|${INOTIFY_USED}/${INOTIFY_LIMIT}(${INOTIFY_PCT}%,warn=${INOTIFY_WARN})|${PHPFPM_ACTIVE}/${PHPFPM_MAX}(${PHPFPM_PCT}%,warn=${PHPFPM_WARN})"
+36
View File
@@ -20,6 +20,42 @@
# decision to notify is the exit code rather than this script parsing the text it just printed.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Silence is the normal output.
# A report that always says something is a report nobody reads. A perfect week prints nothing
# and notifies nothing, so anything that does appear in the Sunday report is worth the glance.
#
# The exit code is the decision, not the text.
# uptime_probe.php --report exits 1 when it has something to say and 0 when it does not. This
# script never parses the output it just printed to work out whether to notify — a report whose
# wording changed would otherwise silently stop notifying.
#
# It reads; it never probes.
# The measurements are already taken, once a minute, by Tools/uptime_probe.sh. Re-probing at
# report time would describe Sunday morning rather than the week being reported on.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Read-only. Reads the stored history and prints; records nothing, and cannot alter the data it
# is reporting on.
#
# UPTIME_PROBE_ENABLED gates the whole run — with the probe off there is no history worth
# reporting, and this says nothing rather than reporting an empty week as a perfect one.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# uptime_report.sh the weekly read. Silent when every domain was 100%.
#
# Called from COFFEE_REPORT_SCRIPTS; takes no arguments and has no other mode. For live figures
# or a per-domain table, use Tools/uptime_probe.sh --status.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
Executable → Regular
+19 -19
View File
@@ -14,24 +14,6 @@
# into a single digest. Reads only — writes nothing, changes nothing.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Aggregator, Not Generator
# This script reads state files that other scripts maintain. It never produces
# health data itself — it only presents what is already there. Each source
# script remains responsible for its own state; this script is the envelope.
#
# Profile-Driven Notification
# The cron schedule never changes. The DIGEST_PROFILE in master.conf controls
# when notifications actually send — switching from daily noise to weekly
# summaries is a one-line conf change, not a cron edit.
#
# Read-Only, No Side Effects
# Writes nothing, changes nothing, triggers nothing. Safe to run at any time
# for a health snapshot without affecting any running service or state file.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
@@ -58,6 +40,24 @@
# RAMDISK_PATH / TRANSCODE_LINK — current transcode location and usage
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Aggregator, Not Generator
# This script reads state files that other scripts maintain. It never produces
# health data itself — it only presents what is already there. Each source
# script remains responsible for its own state; this script is the envelope.
#
# Profile-Driven Notification
# The cron schedule never changes. The DIGEST_PROFILE in master.conf controls
# when notifications actually send — switching from daily noise to weekly
# summaries is a one-line conf change, not a cron edit.
#
# Read-Only, No Side Effects
# Writes nothing, changes nothing, triggers nothing. Safe to run at any time
# for a health snapshot without affecting any running service or state file.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
@@ -394,4 +394,4 @@ if [[ "$DRY_RUN" == true ]]; then
elif [[ "$SHOULD_SEND" == true ]]; then
notify "$NOTIFY_MSG" "Health Digest" "$NOTIFY_SEV"
echo "Digest sent"
fi
fi
Executable → Regular
+19 -19
View File
@@ -15,6 +15,24 @@
# comparison. In --dry-run mode, console only — nothing written to the log.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Five report sections (each skips gracefully if its data source is unavailable):
#
# ZFS pool health — status, state, errors per pool. Pools in
# ZFS_REPORT_IGNORE_POOLS excluded from the report
# (still fully monitored by unRAID — report-only exclusion).
# ARC statistics — current ARC vs max, metadata pressure, hit rate.
# Warns if ARC utilisation exceeds ZFS_REPORT_ARC_WARN_PCT, or if
# ARC headroom (max - current) drops below ZFS_REPORT_ARC_FREE_WARN_GB.
# Memory status — total, free, available RAM (informational only — see note below).
# Warns if available < ZFS_REPORT_AVAIL_WARN_GB.
# Docker memory — top ZFS_REPORT_DOCKER_TOP containers by memory usage.
# Useful for spotting containers approaching watchdog limits.
# Kernel pressure — vmstat snapshot (3 samples).
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
@@ -42,24 +60,6 @@
# reclaimable cache) is still checked separately as a true system-pressure signal.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Five report sections (each skips gracefully if its data source is unavailable):
#
# ZFS pool health — status, state, errors per pool. Pools in
# ZFS_REPORT_IGNORE_POOLS excluded from the report
# (still fully monitored by unRAID — report-only exclusion).
# ARC statistics — current ARC vs max, metadata pressure, hit rate.
# Warns if ARC utilisation exceeds ZFS_REPORT_ARC_WARN_PCT, or if
# ARC headroom (max - current) drops below ZFS_REPORT_ARC_FREE_WARN_GB.
# Memory status — total, free, available RAM (informational only — see note below).
# Warns if available < ZFS_REPORT_AVAIL_WARN_GB.
# Docker memory — top ZFS_REPORT_DOCKER_TOP containers by memory usage.
# Useful for spotting containers approaching watchdog limits.
# Kernel pressure — vmstat snapshot (3 samples).
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
@@ -390,4 +390,4 @@ fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
[[ ${#WARNINGS[@]} -gt 0 ]] && exit 1
exit 0
exit 0
+5 -13
View File
@@ -266,16 +266,8 @@ echo "$ICON_SUCCESS Launched: ${#JOB_PASS[@]}"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no scripts launched"
elif [[ ${#JOB_FAIL[@]} -gt 0 ]]; then
warn "Status: ${#JOB_FAIL[@]} script(s) failed — ${JOB_FAIL[*]}"
notify "Array start on $(hostname) ($MY_ID) — ${#JOB_FAIL[@]} script(s) failed: ${JOB_FAIL[*]}" \
"Array Start" "warning"
else
echo "$ICON_DONE Status: all ${#JOB_PASS[@]} script(s) launched ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
exit 0
# The configured list is the denominator — a script the conf names but that never launched is
# skipped, not absent, and only shows up if something counts it.
JOB_COUNT="${#ARRAY_START_SCRIPTS[@]}"
orchestrator_summary "ARRAY START" "$START" "Array Start"
exit $?
+3 -15
View File
@@ -190,18 +190,6 @@ echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
[[ ${#JOB_FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR Failed: ${JOB_FAIL[*]}"
echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ ${#JOB_FAIL[@]} -eq 0 ]]; then
echo "$ICON_DONE Status: all $STEP step(s) complete ✅"
notify "Array stop complete on $(hostname) ($MY_ID) — $STEP step(s) done" \
"Array Stop" "normal"
else
warn "Status: ${#JOB_FAIL[@]} step(s) failed — ${JOB_FAIL[*]}"
notify "Array stop on $(hostname) ($MY_ID) — ${#JOB_FAIL[@]} step(s) failed: ${JOB_FAIL[*]}" \
"Array Stop" "warning"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
exit 0
JOB_COUNT="$STEP"
orchestrator_summary "ARRAY STOP" "$START" "Array Stop"
exit $?
+4 -20
View File
@@ -280,23 +280,7 @@ fi
END=$(date +%s)
DURATION=$(format_duration $(( END - START )))
TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
# Minimal one-liner when healthy — 30-min cadence, keep it quiet. Full detail on failure.
if [[ "$TOTAL_FAIL" -gt 0 ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY CRITICAL SYNC SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Duration: $DURATION"
[[ ${#PASS[@]} -gt 0 ]] && echo "Synced: ${PASS[*]}"
[[ ${#FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR Failed shares: ${FAIL[*]}"
[[ ${#JOB_FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR Failed jobs: ${JOB_FAIL[*]}"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
notify "Critical sync failed on $(hostname) ($MY_ID) — ${FAIL[*]} ${JOB_FAIL[*]}" \
"Critical Sync" "warning"
exit 1
else
echo "Critical sync complete — $MY_ID${DURATION}${#PASS[@]} share(s), ${#JOB_PASS[@]} job(s)"
fi
exit 0
# Standard ending, quiet mode — 30-min cadence, so a healthy cycle stays one line.
[[ ${#PASS[@]} -gt 0 ]] && echo "Synced: ${PASS[*]}"
orchestrator_summary "CRITICAL SYNC" "$START" "Critical Sync" quiet
exit $?
+4 -18
View File
@@ -354,11 +354,6 @@ WINDOW_END=$(date +%s)
# ━━━ Summary ━━━
# ==============================================================================================
echo ""
echo "━━━━━ $ICON_SUMMARY DAILY MAINTENANCE SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Window: $(date -d @"$WINDOW_START" '+%Y-%m-%d %H:%M:%S')$(date -d @"$WINDOW_END" '+%H:%M:%S')"
echo "$ICON_TIME Duration: $(format_duration $(( WINDOW_END - WINDOW_START )))"
echo ""
echo "$ICON_SYNC Shares ($SHARE_COUNT):"
for entry in "${SHARE_TIMES[@]}"; do
@@ -382,16 +377,7 @@ if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
echo ""
fi
TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
if [[ "$TOTAL_FAIL" -gt 0 ]]; then
warn "Status: $TOTAL_FAIL failure(s)"
notify "Daily maintenance completed with failures on $(hostname) ($MY_ID) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
"Daily Maintenance" "warning"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
exit 1
else
echo "$ICON_DONE Status: all complete — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
# Standard ending — derives skipped from SHARE_COUNT, so a run with rsync gated off reports
# PARTIAL instead of "all complete".
orchestrator_summary "DAILY MAINTENANCE" "$WINDOW_START" "Daily Maintenance"
exit $?
+5 -17
View File
@@ -357,20 +357,8 @@ if [[ "$SHOW_FULL" == true ]]; then
fi
fi
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
if [[ "$SHOW_FULL" == true ]]; then
echo "$ICON_DONE Status: all complete ✅ — ${#JOB_PASS[@]} job(s) run, ${#PASS[@]}/$SHARE_COUNT share(s) synced"
else
echo "$ICON_DONE Intermediate sync — ${#JOB_PASS[@]} job(s), ${#PASS[@]}/$SHARE_COUNT share(s) ($(format_duration $(( WINDOW_END - WINDOW_START ))))"
fi
else
warn "Status: $TOTAL_FAIL failure(s)"
notify "Intermediate sync failed on $(hostname) ($MY_ID) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
"Intermediate Sync" "warning"
fi
[[ "$SHOW_FULL" == true ]] && echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
[[ "$TOTAL_FAIL" -gt 0 ]] && exit 1
exit 0
# Standard ending, quiet mode — 4-hour cadence, so an OK cycle is one parseable line and
# anything skipped or failed expands to the full block on its own.
_mode=quiet; [[ "$ENABLE_LOGGING" == true ]] && _mode=full
orchestrator_summary "INTERMEDIATE SYNC" "$WINDOW_START" "Intermediate Sync" "$_mode"
exit $?
+5 -15
View File
@@ -307,18 +307,8 @@ echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
[[ ${#JOB_FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR Failed: ${JOB_FAIL[*]}"
echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ ${#JOB_FAIL[@]} -eq 0 ]]; then
echo "$ICON_DONE Status: all $STEP step(s) complete ✅"
notify "Monthly maintenance complete on $(hostname) ($MY_ID) — $STEP step(s) done" \
"Monthly Maintenance" "normal"
else
warn "Status: ${#JOB_FAIL[@]} step(s) failed — ${JOB_FAIL[*]}"
notify "Monthly maintenance on $(hostname) ($MY_ID) — ${#JOB_FAIL[@]} step(s) failed: ${JOB_FAIL[*]}" \
"Monthly Maintenance" "warning"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
exit 0
# STEP is what this orchestrator expected to run, so it is the denominator that makes a skipped
# step visible rather than absent.
JOB_COUNT="$STEP"
orchestrator_summary "MONTHLY MAINTENANCE" "$START" "Monthly Maintenance"
exit $?
@@ -98,6 +98,9 @@ ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$ECOSYSTEM_ROOT/load_config.sh"
# Timed from here so the standard summary can report a real duration; this report had none.
REPORT_START=$(date +%s)
parse_args "$@"
# ==============================================================================================
@@ -204,10 +207,9 @@ if [[ ${#JOB_FAIL[@]} -gt 0 ]]; then
echo "❌ Failed: ${JOB_FAIL[*]}"
fi
if [[ ${#JOB_FAIL[@]} -gt 0 && "$DRY_RUN" != true ]]; then
notify "Sunday coffee report had failures on $(hostname) ($MY_ID) — ${JOB_FAIL[*]}" \
"Sunday Morning Coffee Report" "warning"
fi
[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
exit 0
# Standard ending. The configured section list is the denominator, so a report that quietly
# stopped producing one of its sections reads as skipped rather than simply not appearing.
JOB_COUNT="${#SUNDAY_REPORT_SCRIPTS[@]:-0}"
[[ "$JOB_COUNT" -eq 0 ]] && JOB_COUNT=$(( ${#JOB_PASS[@]} + ${#JOB_FAIL[@]} ))
orchestrator_summary "SUNDAY MORNING COFFEE REPORT" "$REPORT_START" "Sunday Morning Coffee Report"
exit $?
+7 -9
View File
@@ -139,6 +139,10 @@ detect_hosts
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
# from a healthy run. Fail loudly instead of silently doing no work.
# This orchestrator never timed itself, so its summary could not report a duration. Set before
# any work so the figure means the cycle, not the tail of it.
CYCLE_START=$(date +%s)
if [[ ${#TRANSCODE_MANAGEMENT_SCRIPTS[@]} -eq 0 ]]; then
error "TRANSCODE_MANAGEMENT_SCRIPTS is empty — no transcode management scripts will run"
error "Check TRANSCODE_MANAGEMENT_SCRIPTS in master.conf"
@@ -223,15 +227,9 @@ done
# ==============================================================================================
# ━━━ Summary — minimal one-liner by default (7-min cadence — keep it quiet when healthy) ━━━
# ==============================================================================================
if [[ "${#JOB_FAIL[@]}" -eq 0 ]]; then
echo "$ICON_SUCCESS Transcode cycle — ${#JOB_PASS[@]}/${#TRANSCODE_MANAGEMENT_SCRIPTS[@]} passed"
else
error "Transcode cycle — failed: ${JOB_FAIL[*]}"
if [[ "$DRY_RUN" != true ]]; then
notify "Transcode management failure on $(hostname) ($MY_ID) — ${JOB_FAIL[*]}" \
"Transcode Management" "warning"
fi
fi
# Quiet by default — 7-min cadence. Anything failed or skipped expands on its own.
JOB_COUNT="${#TRANSCODE_MANAGEMENT_SCRIPTS[@]}"
orchestrator_summary "TRANSCODE CYCLE" "${CYCLE_START:-$(date +%s)}" "Transcode Management" quiet
# ==============================================================================================
# ━━━ Exit ━━━
+7 -13
View File
@@ -241,21 +241,15 @@ fi
# ==============================================================================================
# ━━━ Summary — minimal one-liner by default, full breakdown on failure or --log ━━━
# ==============================================================================================
# Per-script detail only when there is something to read; the standard block carries the rest.
if [[ "${#JOB_FAIL[@]}" -gt 0 || "$ENABLE_LOGGING" == true ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY WATCHDOG CYCLE — $MY_ID$(date '+%H:%M:%S') ━━━━━"
for p in "${JOB_PASS[@]}"; do log " $ICON_DONE $p"; done
for f in "${JOB_FAIL[@]}"; do error " $ICON_ERROR $f"; done
echo "$ICON_TIME Duration: $(format_duration $DURATION)"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
else
echo "$ICON_DONE Watchdog cycle — ${#JOB_PASS[@]}/${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]} passed ($(format_duration $DURATION))"
fi
if [[ "${#JOB_FAIL[@]}" -gt 0 ]]; then
notify "Watchdog cycle failure on $(hostname) ($MY_ID) — ${JOB_FAIL[*]}" \
"Watchdog Orchestrator" "warning"
exit 1
fi
exit 0
# Quiet by default at a 15-min cadence. The configured script list is the denominator, so a
# watchdog that silently stopped running one of its checks shows up as skipped.
JOB_COUNT="${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"
_mode=quiet; [[ "$ENABLE_LOGGING" == true ]] && _mode=full
orchestrator_summary "WATCHDOG CYCLE" "$CYCLE_START" "Watchdog Orchestrator" "$_mode"
exit $?
+6 -22
View File
@@ -402,18 +402,14 @@ WINDOW_END=$(date +%s)
# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
# Per-unit detail first — the standard block that follows carries the verdict and the counts, not
# the names, and knowing WHICH share failed is the whole point of reading a log.
echo ""
echo "━━━━━ $ICON_SUMMARY WEEKLY SYNC MAINTENANCE SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Window: $(date -d @"$WINDOW_START" '+%Y-%m-%d %H:%M:%S')$(date -d @"$WINDOW_END" '+%H:%M:%S')"
echo "$ICON_TIME Duration: $(format_duration $(( WINDOW_END - WINDOW_START )))"
echo "$ICON_GEAR Updates: local=${WEEKLY_SYNC_UPDATES:-false} remote=${WEEKLY_SYNC_UPDATES_REMOTE:-false}"
echo ""
echo "$ICON_SYNC Sync jobs ($SHARE_COUNT):"
for job in "${PASS[@]}"; do echo " $ICON_DONE $job"; done
for job in "${FAIL[@]}"; do echo " $ICON_ERROR $job"; done
echo " Passed: ${#PASS[@]} Failed: ${#FAIL[@]}"
if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
echo ""
@@ -422,19 +418,7 @@ if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
for job in "${JOB_FAIL[@]}"; do echo " $ICON_ERROR $job"; done
fi
TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
echo "$ICON_DONE Status: all complete ✅ — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
else
warn "Status: $TOTAL_FAIL failure(s)"
notify "Weekly maintenance failed on $(hostname) ($MY_ID) — sync: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
"Weekly Maintenance" "warning"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
[[ "$TOTAL_FAIL" -gt 0 ]] && exit 1
exit 0
# Standard ending. Derives skipped from SHARE_COUNT vs what actually ran, so a gated-off section
# can no longer read as success — this is the run that printed "all complete — 0 shares synced".
orchestrator_summary "WEEKLY SYNC MAINTENANCE" "$WINDOW_START" "Weekly Maintenance"
exit $?
+33 -6
View File
@@ -174,7 +174,12 @@ transform_xml_for_gpu() {
local src_xml="$1" gpu_type="$2"
# Detect GPU-aware XMLs — new style (--gpus "device=) or old style (--runtime=nvidia)
if ! grep -qE '--gpus[[:space:]]+"device=|--runtime=nvidia|NVIDIA_VISIBLE_DEVICES' "$src_xml" 2>/dev/null; then
# -- before the pattern is load-bearing: it begins with "--", so without it grep parses the
# pattern as an OPTION, exits 2, and the ! makes this branch always true — the function then
# returned the XML untransformed every single time, for every GPU type, with 2>/dev/null
# swallowing "invalid option". Every container onboarded to a mirror kept the owner NVIDIA
# device UUID and could not start on Intel or AMD hardware. Found 2026-08-23.
if ! grep -qE -- '--gpus[[:space:]]+"device=|--runtime=nvidia|NVIDIA_VISIBLE_DEVICES' "$src_xml" 2>/dev/null; then
echo "$src_xml"
return 0
fi
@@ -619,6 +624,12 @@ cleanup_deployed_stack_on_remote() {
cleanup_deployed_stack_locally() {
local owner_ip="$1" ssh_key="$2"
local -a xml_names=()
# Callers write `cleanup_deployed_stack_locally … || STEP_STACK_CLEANUP_OK=false`, so the
# exit status is what the offboard summary prints. Every removal below warns and carries on
# — one container that will not die must not abandon the rest of the stack — which meant the
# function ended on a `done` and could only ever return 0. Step 3 reported ✅ even when every
# docker rm and every rm -rf had failed. Failures are collected here and reported at the end.
local _rc=0
if [[ -n "$owner_ip" ]]; then
local -a auth_arr arr_arr
@@ -642,8 +653,13 @@ cleanup_deployed_stack_locally() {
fi
if [[ ${#xml_names[@]} -eq 0 ]]; then
log "Could not read deployed stack from owner — skipping auth/arr/services cleanup"
return 0
# Not a success. OWNER_REACHABLE only means a probe answered — the three SSH reads above
# can still time out or come back empty, and then nothing was cleaned. The caller's own
# unreachable-owner branch sets STEP_STACK_CLEANUP_OK=false for exactly this situation,
# so returning 0 here made the summary claim a cleanup that never ran.
warn "Could not read deployed stack from owner — auth/arr/services cleanup did not run"
warn "Containers will remain — re-run when the owner answers over SSH"
return 1
fi
local _local_short
@@ -680,17 +696,28 @@ cleanup_deployed_stack_locally() {
"$cname" 2>/dev/null | awk -F: '{print $1}' | grep '^/mnt/.*/appdata')
timeout "${DOCKER_TIMEOUT:-30}" docker stop "$cname" >/dev/null 2>&1 || true
_PM_TRAP_STOPPED+=("$cname")
timeout "${DOCKER_TIMEOUT:-30}" docker rm "$cname" >/dev/null 2>&1 && \
echo " $cname removed ✅" || warn " $cname rm failed"
if timeout "${DOCKER_TIMEOUT:-30}" docker rm "$cname" >/dev/null 2>&1; then
echo " $cname removed ✅"
else
warn " $cname rm failed"
_rc=1
fi
else
log " $cname not found locally — skipping"
fi
while IFS= read -r path; do
[[ -z "$path" ]] && continue
rm -rf "$path" && echo " Appdata removed: $path" || warn " Failed to remove: $path"
if rm -rf "$path"; then
echo " Appdata removed: $path"
else
warn " Failed to remove: $path"
_rc=1
fi
done <<< "$appdata_paths"
done
return "$_rc"
}
# ==============================================================================================
+12 -1
View File
@@ -103,7 +103,18 @@ once, and the endpoint, the cache writer, and the page all pick it up together.
|-------|-------|------|
| `pages/` | 11 | One per WebGUI tab — monitor, docker, arrs, fallback, watchdog, rsync, partnership, scheduler, auth, settings, setup |
| `api/` | 50 | JSON endpoints the pages poll, plus action endpoints (run a script, stop a job, toggle a flag) |
| `include/` | 16 | Shared builders and helpers — `vv_monitor_*`, `vv_arrs_*`, `vv_docker_*`, config read/write, auth |
| `include/` | 27 | Shared builders and helpers — `vv_monitor_*`, `vv_arrs_*`, `vv_docker_*`, config read/write, auth |
**The Monitor board is declared, not laid out.** `include/monitor_board.php` holds one array
naming every card on the Monitor grid, its width and its order, and generates the whole layout
from it — the column ladder, the span clamps at each width, the row-height cap, and the
compensation when a conditional card is absent. Column counts are 8/4/2/1 and spans are 1/2/4/8,
which is what lets the board re-cut itself at any width with no holes and no hand-placed card.
Breakpoints are arithmetic over `VV_MON_CARD_FLOOR`, never chosen by eye, and the row-height cap
is conditional on the same arithmetic — it applies only where a rung fits one screen.
`pages/monitor.php` carries the card bodies and nothing about where they go; moving a card is
moving a line in that array. The page cross-checks the declaration against the cards that
actually rendered and says so in the browser if they disagree.
**Caching.** Several endpoints serve from `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs) rather than hitting live
APIs on every page view, refreshed by `Tools/api_cache_writer.sh`. `?live=1` bypasses the
@@ -24,6 +24,22 @@
# list and reboot.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# One pass over PCIE_QUIET_DEVICES at array start. Each address is looked up under
# /sys/bus/pci/devices, checked against the guards below, and removed through the kernel's own
# per-device remove attribute. Nothing is masked and no kernel parameter is set.
#
# Idempotent, and silent when there is nothing to do. An address that is already gone is not an
# error — that is the normal state on every array start after the first within one boot.
#
# Reapplied every array start rather than once, because a reboot re-enumerates the bus and the
# devices come back. That is also the undo: clear PCIE_QUIET_DEVICES and reboot.
#
# PCIE_QUIET_ENABLED gates the whole run before any device is touched.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
+14 -14
View File
@@ -18,6 +18,19 @@
# total. Too high wastes RAM; too low causes slowdowns.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# 1. Read the current pm.max_children from the PHP-FPM pool config
# 2. Already at or above the target → exit silently, no write, no restart
# 3. Otherwise rewrite the value and restart PHP-FPM via the adapter
# 4. Verify PHP-FPM came back up
#
# Runs at array start, before the WebGUI sees real load. The setting does not survive an
# unRAID update — the OS replaces the pool config — which is why this reapplies every boot
# rather than being a one-time install step.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
@@ -41,19 +54,6 @@
# 7. Read back config to confirm value applied
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# 1. Read the current pm.max_children from the PHP-FPM pool config
# 2. Already at or above the target → exit silently, no write, no restart
# 3. Otherwise rewrite the value and restart PHP-FPM via the adapter
# 4. Verify PHP-FPM came back up
#
# Runs at array start, before the WebGUI sees real load. The setting does not survive an
# unRAID update — the OS replaces the pool config — which is why this reapplies every boot
# rather than being a one-time install step.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
@@ -257,4 +257,4 @@ echo ""
echo "$ICON_DONE Status: done ✅"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
exit 0
+14 -14
View File
@@ -14,20 +14,6 @@
# page always reflects the live key value.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Self-Healing at Boot
# The unraid-api registry is ephemeral — OS updates and service restarts clear
# it without warning. Running at every array start means the key is always
# present after boot without any manual intervention.
#
# Conf Stays Current
# HOST*_UNRAID_API_KEY in the local host conf is updated after every renewal.
# The partnership page reads the conf — it always reflects the live key value
# without a separate sync step.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
@@ -42,6 +28,20 @@
# a key present in the conf but absent from the registry is the exact failure it repairs.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Self-Healing at Boot
# The unraid-api registry is ephemeral — OS updates and service restarts clear
# it without warning. Running at every array start means the key is always
# present after boot without any manual intervention.
#
# Conf Stays Current
# HOST*_UNRAID_API_KEY in the local host conf is updated after every renewal.
# The partnership page reads the conf — it always reflects the live key value
# without a separate sync step.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
+11
View File
@@ -422,6 +422,17 @@ if ($can('system_state')) {
}
}
// Fallback is dormant until it isn't, so "configured" and "would work" are unrelated — this is the
// only block that reports the second. Cheap: local conf and state file plus one cached presence
// read, never a network round trip.
if ($can('fallback_state')) {
$fb = vv_ai_fallback_state();
if ($fb !== '') {
$attached['fallback'] = 'readiness';
$diagBlock .= $fb;
}
}
// The troubleshooting profile gets the actual tail of the one log the operator is looking at,
// warnings and ordinary lines alike. The fleet-wide WARN/ERROR sweep above cannot answer "why
// did this one stop" — the last line a script printed before dying is usually not labelled.
+9 -6
View File
@@ -16,12 +16,7 @@
# Not scheduled and deliberately not in any orchestrator. This is a development check — it runs
# when the routing changes, not every night. Nothing on the running system depends on it.
#
# RUNTIME MODES
# ai_explain_check.sh check every fixture
# ai_explain_check.sh --verbose print the full explain report for each case
# ai_explain_check.sh <pattern> only cases whose question matches the pattern
#
# OPERATIONAL SAFEGUARDS
# DESIGN PRINCIPLES
# Asserts routing, never wording.
# Which capabilities a profile holds and which evidence was attached are decided before the
# model is asked anything. Asserting on generated prose would fail for reasons that tell
@@ -31,8 +26,16 @@
# An unrecognised key is an error, not a skip. A typo in an assertion that silently passes
# is worse than no assertion, because the line still reads as covered.
#
# OPERATIONAL SAFEGUARDS
# Read-only. Runs fixtures through --explain and compares; writes nothing and changes no state.
#
# Exits non-zero on any failure, so it can gate a commit.
#
# RUNTIME MODES
# ai_explain_check.sh check every fixture
# ai_explain_check.sh --verbose print the full explain report for each case
# ai_explain_check.sh <pattern> only cases whose question matches the pattern
#
# DEPENDS ON
# Plugin/unraid/Tools/ai_chat_worker.php --explain mode
# Plugin/unraid/Tools/ai_explain_fixtures.txt
+28 -4
View File
@@ -8,10 +8,34 @@
// Reads ai_log_fixtures.txt and this host's /var/log/syslog*. Files nothing, writes nothing,
// and calls no part of the sweep beyond vv_ai_syslog_findings() on lines it supplies itself.
//
// EXIT
// 0 when every fixture is recognised as written. Precision findings are reported but never
// fail the run: what a real syslog contains is a fact about the machine, not about the
// patterns, and a genuinely failing disk should not turn this into a red test.
// DESIGN PRINCIPLES
// Only recall can fail the run.
// A missed fixture is a fact about the patterns and is always a defect. A precision hit is
// a fact about this machine — a genuinely failing disk should not turn this red, and if it
// did, the honest fix would be to stop having a failing disk rather than to edit a pattern.
//
// Precision is replayed against real history, not a sample.
// The patterns that cause damage are the ones matching ordinary operation, and ordinary
// operation is exactly what a hand-written fixture file never contains. Only the machine's
// own syslog can show what a pattern fires on when nothing is wrong.
//
// The sweep is never invoked, only its matcher.
// vv_ai_syslog_findings() is called on lines this file supplies. Running the real sweep
// would file findings, and a test that has to be cleaned up afterwards stops being run.
//
// OPERATIONAL SAFEGUARDS
// Read-only. Reads ai_log_fixtures.txt and this host's /var/log/syslog*; files no finding,
// writes no store, and touches no conf beyond the enable flag.
//
// Exit 0 when every fixture is recognised as written. Precision findings are reported but
// never fail the run — see DESIGN PRINCIPLES.
//
// RUNTIME MODES
// php ai_log_check.php both checks
// php ai_log_check.php --recall fixtures only
// php ai_log_check.php --precision replay this host's syslog history only
//
// Not scheduled, and deliberately so. Run it after touching VV_AI_SYSLOG_PATTERNS.
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once dirname(__DIR__) . '/include/ai_repair.php';
+41
View File
@@ -15,6 +15,47 @@
#
# Run it after touching VV_AI_SYSLOG_PATTERNS. Nothing here writes: no findings are filed, no
# conf is read for anything but the enable flag, and the sweep is never invoked.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# A wrapper. The work is in ai_log_check.php, next to the sweep's own matcher — the patterns and
# vv_ai_syslog_findings() live in include/ai_repair.php, and a bash reimplementation of the
# matching would be a second set of regexes to keep in step with the first.
#
# Flags are forwarded verbatim; nothing is interpreted here.
#
# Not scheduled and in no orchestrator. This is a development check that runs when the patterns
# change, not on a timer — nothing on the running system depends on it.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Two checks, because the failure modes are opposite.
# Recall catches a pattern that stopped matching; precision catches one that matches too much.
# A single test would catch one and silently permit the other, and the second is the one that
# fills the findings store with noise until the operator stops reading it.
#
# Precision is measured against this machine's real history.
# A hand-written fixture file can show that a pattern matches what it should. Only a real
# syslog can show what it also matches when nothing is wrong.
#
# Only recall fails the run.
# What a real syslog contains is a fact about the machine, not about the patterns. A genuinely
# failing disk should not turn this red.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Read-only. No finding is filed, no store is written, and the repair sweep itself is never run
# — only its matcher, on lines this check supplies.
#
# Safe to run on a live host at any time, including one that is currently faulting. It observes
# the syslog it replays and changes nothing about it.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
+5
View File
@@ -42,6 +42,11 @@
// stdin is an untrusted stream from another process. A malformed or endless payload must
// fail as a bad request, not as an out-of-memory.
//
// RUNTIME MODES
// Not invoked by hand. include/ai_rpc.php opens an SSH session to the owner and runs this
// file with the request as JSON on stdin; the response is JSON on stdout. There are no flags
// and no arguments — the action, the profile and the payload all arrive in the request body.
//
// DEPENDS ON
// include/ai_actions.php vv_ai_dispatch() — the shared handlers
// include/config.php vv_ai_is_owner()
+8
View File
@@ -66,6 +66,13 @@
// VV_CACHE_DIR/arrs.json consumed by api/arrs.php
// stdout one timing line, captured into the job log
//
// RUNTIME MODES
// api_cache_writer.php
// Build both payloads and write them to the cache. Takes no arguments.
//
// Run every minute by Tools/api_cache_writer.sh, which is the cron entry — this file is the
// work, that one is the schedule, the lock and the log.
//
// DEPENDS ON
// include/monitor.php, include/common.php, include/unraid_api.php,
// include/vms.php, include/docker_folders.php, include/arrs.php
@@ -112,6 +119,7 @@ if (vv_ai_ui_on()) {
$monitor = [
'system' => vv_system_info(),
'varaverk' => vv_varaverk_state(),
'fallback' => vv_fallback_state(),
'fallback_active' => vv_fallback_active(),
'partner' => vv_partner_state(),
+32
View File
@@ -28,6 +28,38 @@
# finding for every hostname on the machine.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# It answers the questions nobody was going to ask.
# Both checks were already possible from the Auth tab, and both required somebody to open it
# and press a button on the right row. That is why one host here returned nothing but 5xx for
# months. A check that depends on being run is a check that is not running.
#
# Files findings; never applies remedies.
# The fixes are "start a container", "edit a rule", "change a default policy". Each of those
# is a judgement about intent, and a sweep that guessed at intent would be repairing a
# deliberate configuration as often as a broken one.
#
# Down is measured in time, not in samples.
# A host must have been failing for longer than AUTH_SWEEP_DOWN_MIN before anything is filed.
# Counting failed samples instead would file a finding for every hostname on the machine every
# time it reboots, and a report that cries wolf after every restart stops being read.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Read-only against the auth stack. Nothing here starts a container, edits a proxy host, changes
# a rule or alters a default policy — the two questions are answered by observation alone.
#
# --dry-run reports what it would file and writes no finding. --report only reads what is
# already filed, and is silent when clean, so it cannot manufacture noise for the Sunday report.
#
# Findings are filed, not notified. A sweep that raised an alert per host would be its own
# incident every time the array restarted.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
+34
View File
@@ -25,6 +25,40 @@
# own created_on, which is a real date; nothing else is back-filled.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# History is accumulated, never reconstructed.
# NPM holds only the present, so every past state this file knows about is one it observed at
# the time. Counts start at zero on first run and nothing is back-filled — first_seen is the
# single exception, seeded from NPM's own created_on because that is a real recorded date
# rather than an inference.
#
# A domain is retired on strikes, not on a single bad pass.
# A pass fails for a domain when it is absent from NPM's list or its expiry is already in the
# past — and NPM's list can come back short for reasons that have nothing to do with the
# certificate, such as an API hiccup or a restart mid-pass. CERT_HISTORY_STRIKES consecutive
# failures are required before a domain is retired, and it stays in the store afterwards, so
# one bad read can neither erase months of history nor hide a genuine expiry.
#
# The wrapper holds no logic.
# The work sits next to the NPM client it needs, because token handling lives in
# include/auth.php and a bash reimplementation would be a second copy of the thing most worth
# having only one of. Flags are forwarded verbatim.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# The store is replaced atomically: temp file, verified, then renamed over the original. A pass
# that fails partway leaves the previous history intact rather than a truncated file — this is
# the only record of what these certificates did, and there is no second copy to restore from.
#
# --dry-run reports every change it would make and writes nothing. --status only reads.
#
# Read-only against NPM. Certificates are observed; nothing here renews, deletes or edits one.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
+15
View File
@@ -40,6 +40,21 @@
// downstream of something else here, and listing it alongside its own cause invites fixing the
// symptom.
//
// OPERATIONAL SAFEGUARDS
// Read-only. Opens certbot's logs and nothing else — no certificate is requested, renewed or
// deleted here, and no log is rotated or truncated.
//
// Bounded by file count and by bytes per file, so a directory that has grown to 639 MB across
// 1001 files cannot turn a page load into an unbounded read. Only the tail of each log is
// examined, because a run explains its failure at the end rather than the beginning.
//
// An unrecognised failure is reported as unclassified, never folded into the nearest category.
// A tidy chart that is wrong sends the operator to fix a domain that was never broken.
//
// The log directory is discovered from the NPM container rather than assumed, so a container
// path change surfaces as "no logs found" instead of an empty triage that reads as "no
// failures".
//
// RUNTIME MODES
// cert_triage.php summary — categories, affected domains, and the causal reading
// cert_triage.php --json the same as JSON, for the Certs tab
+36
View File
@@ -30,6 +30,42 @@
# fails renewals for domains that have nothing wrong with them.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# One invocation is one data point.
# A single failure writes its reason three times — the ACME response, the traceback, and
# certbot's summary. Counting lines would report it as three and inflate whichever category
# happens to be the most verbose, which is the opposite of what triage is for.
#
# Causes are separated from consequences.
# Rate limiting is nearly always downstream: retries against a hostname with no DNS record
# exhaust the allowance, which then fails renewals for domains that have nothing wrong with
# them. Reporting the rate limit as the problem sends the operator to fix the wrong domains.
#
# Rotation suffix is the clock, not mtime.
# Every file in this directory carries the same mtime because they arrive as a synced set, so
# ordering by mtime returns an arbitrary answer that looks authoritative.
#
# Bounded by design, not by hope.
# 639 MB across 1001 files cannot be read in a page load. Only CERT_TRIAGE_FILES logs are
# opened, and only CERT_TRIAGE_MAX_BYTES from the end of each, because a failure explains
# itself at the end of the run rather than the start.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Read-only throughout. It opens certbot's logs and nothing else — no certificate is requested,
# renewed or deleted, and no log is rotated or truncated by anything here.
#
# Diagnosis only. This names why renewals failed; acting on that is the operator's, and
# cert_history.sh remains the thing that records what happened.
#
# The log directory is normally discovered from the NPM container rather than hardcoded, so a
# container path change surfaces as "no logs found" rather than as a silently empty triage.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
+32 -9
View File
@@ -13,16 +13,39 @@
// php Tools/conf_widget_check.php assertions, then the live summary
// php Tools/conf_widget_check.php --list every live field and its inferred control
//
// WHY IT ASSERTS AGAINST SNIPPETS AND NOT THE LIVE CONF
// The live conf is the thing being described, so it cannot also be the thing that proves the
// description right — an inference rule that silently stopped matching would keep passing as
// the conf drifted to suit it. The snippets are frozen copies of each convention as written,
// so a rule change that breaks one shows up here rather than as a wrong control on a page.
// DESIGN PRINCIPLES
// Assertions run against snippets, never against the live conf.
// The live conf is the thing being described, so it cannot also be the thing that proves
// the description right — an inference rule that silently stopped matching would keep
// passing as the conf drifted to suit it. The snippets are frozen copies of each
// convention as written, so a rule change that breaks one shows up here rather than as a
// wrong control on a page.
//
// WHAT AN INFERENCE IS NOT
// Consistent with confform.php, none of this validates. A number field carrying min and max is
// a courtesy to whoever is typing, not a promise the value is sensible — the consuming script
// still owns that question.
// An inference is a drawing decision, not a validation.
// Consistent with confform.php, none of this validates. A number field carrying min and
// max is a courtesy to whoever is typing, not a promise the value is sensible — the
// consuming script still owns that question.
//
// The live pass reports, it does not assert.
// What this host's master.conf infers to is a description of that file, not a verdict on
// it. Turning the live summary into pass/fail would make an unusual but legitimate
// setting look like a defect.
//
// OPERATIONAL SAFEGUARDS
// Read-only. Parses conf and reports; writes no conf, no store and no page.
//
// Exits non-zero only when a snippet assertion fails, so it can gate a commit without a real
// conf's contents ever being able to break the build.
//
// Never renders. It reports which control would be drawn; the drawing stays in confform.php,
// so this cannot disagree with the page by construction.
//
// RUNTIME MODES
// php Tools/conf_widget_check.php assertions, then the live summary
// php Tools/conf_widget_check.php --list every live field and its inferred control
//
// Hand-run. Not scheduled and in no orchestrator — run it after touching _vv_conf_widget(),
// after adding a conf convention, or when a setting draws as the wrong control.
//
// DEPENDS ON
// include/confform.php _vv_conf_parse_field_range(), vv_conf_key_is_secret()
+14 -1
View File
@@ -9,6 +9,19 @@
// Onboard calls this on the mirror after deploying the owner's stack, so those containers land
// somewhere that says whose they are instead of scattering into the mirror's own groups.
//
// OPERATIONAL MODEL
// Create-or-update, never replace. The named folder is found in folder.view3 if it exists and
// is amended in place; only its absence causes one to be created. Every other folder in the
// file, and every container already filed elsewhere, is left exactly as it was — this is one
// shelf in someone else's cupboard.
//
// Called during onboard on the mirror, after the owner's stack has been deployed there, so the
// containers exist by the time anything tries to file them.
//
// The icon is resolved separately from the folder and never gates it. --icon-only performs
// just that lookup and prints the URL, which is how onboard asks for it without writing
// anything.
//
// DESIGN PRINCIPLES
// The name comes from master.conf, never from a second source.
// HOST2="unRAID-Jayred36" → "Jayred36-Fallback". The convention already existed by hand as
@@ -39,7 +52,7 @@
// --dry-run prints the resulting folder and writes nothing.
// Icon resolution failing never blocks the folder: no image is a cosmetic loss, no folder is not.
//
// REQUEST
// RUNTIME MODES
// fallback_folder.php --host=HOST2 [--containers=A,B,C] [--icon=URL] [--dry-run]
// fallback_folder.php --host=HOST2 --icon-only resolve and print the icon URL, write nothing
//
+337
View File
@@ -0,0 +1,337 @@
#!/bin/bash
# ==============================================================================================
# ============================= WebGUI JavaScript Checker ======================================
# ==============================================================================================
#
# PURPOSE
# ==============================================================================================
# Finds the two JavaScript faults this plugin has actually shipped, neither of which any syntax
# check can see, because both are runtime behaviour:
#
# 1. An identifier declared inside one function and referenced from another. Throws
# ReferenceError on every call, kills the rest of the render, and passes php -l and
# node --check without complaint.
#
# 2. A fetch chain ending in an empty catch. Not error handling — error deletion. The request
# fails, nothing renders, nothing is logged, and the surface sits on "Loading…" forever.
#
# The two compound: on 2026-08-20 a cross-scope ReferenceError in the mesh chat was swallowed by
# an empty catch on every render. It presented as "the chat takes a minute to load" — the minute
# was the poller's backoff — and hours went into profiling PHP that was never slow. Once a catch
# reported the error, the fault named itself in one line.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# 1. Extract the JS from each <script> block in a PHP/page file
# 2. Strip PHP tags, comments, string literals, template-literal text and object keys
# 3. Cross-scope pass — declarations per function, then uses judged against them
# 4. Catch pass — an empty .catch() within 25 lines of a fetch()
#
# Reads only. Prints findings and exits non-zero when any are found, so it can gate a commit.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# A Checker That Cannot Fail Loudly Is Worse Than No Checker
# --self-test runs both detectors against a fixture carrying both faults and asserts they are
# found. It is not decoration. The first version of this scan reported a confident zero across
# the whole plugin because a regex was silently broken; the fixture is what catches that.
#
# Preprocessing Is Where The False Positives Die
# A raw scan produced ~1600 candidates, nearly all of them HTML attribute names, CSS keywords
# and English prose living inside template literals. Keeping only the ${...} expressions cut
# that to double digits. What remains after preprocessing is worth a human's attention.
#
# Report, Never Rewrite
# This prints file and line. It does not edit. A tool that silently "fixes" a false positive in
# a render path is a worse outcome than the fault it was hunting.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Escaped $ In Every Perl Character Class
# [A-Za-z_$] makes Perl interpolate $] — its version variable — and silently mangles the
# pattern into something that matches nothing. That mistake produced a false clean result twice
# while this was being written. Every class here writes \$, and --self-test would catch it
# returning.
#
# Known Parser Gaps, Stated Rather Than Hidden
# This is regex and brace counting, not a JavaScript parser. It over-reports and never
# under-reports, which is the safe direction: a clean run is meaningful, a dirty one needs a
# human. Preprocessing took the raw count from ~1600 to single digits by handling template
# literals, string literals, object keys, regex literals, destructuring, nested-function
# parameters and multi-declarator const.
#
# BASELINE as of 2026-08-21: three known false positives on a healthy tree —
# setup.php `to`, `id` in vvRenderOnboardPanel()
# Varaverk.page `s` in vvRenderMirrorOnboard()
# Each was read and confirmed harmless. Findings beyond those three are new and worth opening.
# If that baseline ever reaches zero, delete this paragraph rather than letting it rot.
#
# Non-Zero Exit On Findings
# So it can sit in front of a commit. A checker whose output has to be noticed by a human is a
# checker that stops being run.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# js_check.sh
# Scan every pages/*.php, include/*.php and Varaverk.page.
#
# js_check.sh --scope | --catches
# One pass only.
#
# js_check.sh --self-test
# Verify both detectors against the built-in fixture. Run this after editing this file.
#
# js_check.sh <file> [...]
# Scan named files instead of the whole plugin.
#
# ==============================================================================================
set -uo pipefail
PLUGIN_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
DO_SCOPE=true; DO_CATCH=true; SELF_TEST=false
FILES=()
for arg in "$@"; do
case "$arg" in
--scope) DO_CATCH=false ;;
--catches) DO_SCOPE=false ;;
--self-test) SELF_TEST=true ;;
-*) echo "Unknown option: $arg" >&2; exit 1 ;;
*) FILES+=("$arg") ;;
esac
done
# ── Preprocessor ──────────────────────────────────────────────────────────────────────────────
cat > "$WORK/prep.pl" <<'PREP'
# NOTE: every character class escapes $ as \$. Bare [A-Za-z_$] interpolates $] (Perl's version)
# and silently matches nothing — see OPERATIONAL SAFEGUARDS.
local $/; my $s = <>;
$s =~ s{<\?=.*?\?>}{0}gs;
$s =~ s{<\?php.*?\?>}{}gs;
$s =~ s{/\*.*?\*/}{}gs;
$s =~ s{//[^\n]*}{}g;
# Template literals carry HTML, CSS and prose. Only the ${...} expressions are code.
$s =~ s{`((?:[^`\\]|\\.)*)`}{ " " . join(" ", $1 =~ m{\$\{([^{}]*(?:\{[^{}]*\}[^{}]*)*)\}}g) . " " }ges;
$s =~ s{'(?:[^'\\\n]|\\.)*'}{""}g;
$s =~ s{"(?:[^"\\\n]|\\.)*"}{""}g;
# Regex literals. Only where one can legally begin — after ( = , : return — so division is not
# mistaken for a pattern. Their contents and flags are not identifiers: /\s*os|windows/i was
# reporting s, os and windows as undeclared variables.
$s =~ s{(=>\s*|[=(,:\[;?!]\s*|[&|]{2}\s*|\breturn\s+)/(?:[^/\\\n]|\\.)+/[gimsuy]*}{$1 0}g;
# Object-literal keys are not references. Anchored to { or , so a ternary keeps its operand.
$s =~ s/([{,]\s*)[A-Za-z_\$][A-Za-z0-9_\$]*\s*:/$1 /g;
print $s;
PREP
# ── Cross-scope detector ──────────────────────────────────────────────────────────────────────
cat > "$WORK/scope.awk" <<'SCOPE'
function fname(l, m) { if (match(l, /function[ \t]+[A-Za-z_$][A-Za-z0-9_$]*/)) { m=substr(l,RSTART,RLENGTH); sub(/function[ \t]+/,"",m); return m } return "" }
BEGIN {
# Built-ins and browser globals. Belt and braces — a parser slip that registers one of these as
# a local would otherwise report it in every other function in the file, which is exactly how
# Math appeared fifteen times while this was being written.
split("Math JSON Object Array String Number Boolean Date RegExp Error Promise Set Map WeakMap " \
"Symbol BigInt Intl console document window navigator location history screen localStorage " \
"sessionStorage fetch setTimeout setInterval clearTimeout clearInterval requestAnimationFrame " \
"parseInt parseFloat isNaN isFinite encodeURIComponent decodeURIComponent URLSearchParams " \
"FormData Headers Request Response AbortController CustomEvent Event Node Element " \
"getComputedStyle matchMedia structuredClone queueMicrotask btoa atob crypto performance " \
"globalThis undefined NaN Infinity ResizeObserver MutationObserver Uint32Array swal", g, " ")
for (gi in g) glob[g[gi]]=1
}
{ L[NR]=$0 }
END {
depth=0; cur=""
for (n=1; n<=NR; n++) {
line=L[n]
if (depth==0) { f=fname(line); if (f!="") cur=f }
scope = (cur=="" ? "(module)" : cur)
# Parameters of ANY function on this line, nested helpers included. Attributing a nested
# helper's params to the enclosing function is an approximation, and the right one: it can
# only suppress a report, never invent one, and nested params were the largest false-positive
# class in this codebase.
if (match(line, /function[ \t]*[A-Za-z0-9_$]*[ \t]*\([^)]*\)/)) {
pp=substr(line,RSTART,RLENGTH); sub(/.*\(/,"",pp); sub(/\).*/,"",pp)
c=split(pp,pa,","); for (k=1;k<=c;k++) { gsub(/[ \t]/,"",pa[k]); sub(/=.*/,"",pa[k]); if (pa[k]!="") decl[scope,pa[k]]=n } }
# Every declarator in the statement, not just the first: `const s = a, n = b` declares both,
# and reading only `s` left `n` looking undeclared wherever it was used.
if (match(line, /(const|let|var)[ \t]+/)) {
rest=substr(line,RSTART+RLENGTH)
# Destructuring binds every name inside the brackets: `for (const [folder, items] of …)`
# and `const { a, b } = obj`. Handled before the comma split, which cannot read them.
if (match(rest, /^[ \t]*[\[{][^\]}]*[\]}]/)) {
db=substr(rest,RSTART,RLENGTH)
while (match(db, /[A-Za-z_$][A-Za-z0-9_$]*/)) {
nm2=substr(db,RSTART,RLENGTH)
decl[scope,nm2]=n; if (scope=="(module)") mod[nm2]=n
db=substr(db,RSTART+RLENGTH) }
}
cc=split(rest,dl,",")
for (kk=1;kk<=cc;kk++) {
# The identifier must be followed by "=" or end the segment. Without that test a comma
# inside the initialiser — `const pct = Math.min(a, Math.round(b))` — makes the second
# argument look like a second declarator, and Math gets registered as a local.
if (match(dl[kk], /^[ \t]*[A-Za-z_$][A-Za-z0-9_$]*[ \t]*(=[^=]|=$|$)/)) {
nm=dl[kk]; sub(/^[ \t]*/,"",nm); sub(/[ \t]*=.*$/,"",nm); gsub(/[ \t]/,"",nm)
if (nm!="") { decl[scope,nm]=n; if (scope=="(module)") mod[nm]=n }
} else break # stop at the first non-declarator; the rest is an expression
}
}
tmp=line
while (match(tmp, /(const|let|var)[ \t]+[A-Za-z_$][A-Za-z0-9_$]*/)) {
d=substr(tmp,RSTART,RLENGTH); sub(/(const|let|var)[ \t]+/,"",d)
decl[scope,d]=n; if (scope=="(module)") mod[d]=n
tmp=substr(tmp,RSTART+RLENGTH) }
tmp=line
while (match(tmp, /\([^)]*\)[ \t]*=>/)) {
d=substr(tmp,RSTART,RLENGTH); gsub(/[()=>\t]/," ",d); gsub(/[{}\[\]]/," ",d)
c2=split(d,ap,","); for (k2=1;k2<=c2;k2++) { gsub(/[ \t]/,"",ap[k2]); sub(/=.*/,"",ap[k2]); if (ap[k2]!="") decl[scope,ap[k2]]=n }
tmp=substr(tmp,RSTART+RLENGTH) }
tmp=line
while (match(tmp, /[A-Za-z_$][A-Za-z0-9_$]*[ \t]*=>/)) {
d=substr(tmp,RSTART,RLENGTH); gsub(/[ \t=>]/,"",d); if (d!="") decl[scope,d]=n
tmp=substr(tmp,RSTART+RLENGTH) }
if (match(line, /for[ \t]*\([ \t]*(const|let|var)[ \t]+[A-Za-z_$][A-Za-z0-9_$]*/)) {
d=substr(line,RSTART,RLENGTH); sub(/.*[ \t]/,"",d); decl[scope,d]=n }
if (match(line, /catch[ \t]*\([ \t]*[A-Za-z_$][A-Za-z0-9_$]*/)) {
d=substr(line,RSTART,RLENGTH); sub(/.*\([ \t]*/,"",d); decl[scope,d]=n }
if (fname(line)!="") isfn[fname(line)]=1
for (i=1;i<=length(line);i++) { c=substr(line,i,1); if (c=="{") depth++; else if (c=="}") { depth--; if (depth<=0) { depth=0; cur="" } } }
}
depth=0; cur=""
for (n=1; n<=NR; n++) {
line=L[n]
if (depth==0) { f=fname(line); if (f!="") cur=f }
scope = (cur=="" ? "(module)" : cur)
if (scope!="(module)") {
tmp=line
while (match(tmp, /[A-Za-z_$][A-Za-z0-9_$]*/)) {
id=substr(tmp,RSTART,RLENGTH); pre=substr(tmp,RSTART-1,1)
tmp=substr(tmp,RSTART+RLENGTH)
if (pre=="." || id in isfn || id in mod || id in glob) continue
if ((scope,id) in decl) continue
for (o in decl) { split(o,q,SUBSEP); if (q[2]==id && q[1]!=scope && q[1]!="(module)") {
key=id SUBSEP scope; if (!(rep[key]++)) printf " line %d: %s used in %s() — declared only inside %s()\n", n, id, scope, q[1]; break } }
}
}
for (i=1;i<=length(line);i++) { c=substr(line,i,1); if (c=="{") depth++; else if (c=="}") { depth--; if (depth<=0) { depth=0; cur="" } } }
}
}
SCOPE
scan_scope() {
local f="$1"
awk '/<script/{p=1;next} /<\/script>/{p=0;print ""} p' "$f" | perl "$WORK/prep.pl" > "$WORK/x.js" 2>/dev/null
awk -f "$WORK/scope.awk" "$WORK/x.js" 2>/dev/null
}
# An empty catch within 25 lines of a fetch. Line distance rather than real chain parsing: the
# chains here are short, and a bare `catch {}` far from any request is usually a deliberate
# localStorage or execCommand guard, which this must not report.
scan_catches() {
local f="$1"
awk '/fetch\(|XMLHttpRequest/ { inf=NR }
/\.catch\(\s*\(\s*[a-z_]*\s*\)\s*=>\s*\{\s*\}\s*\)/ {
if (inf && NR-inf <= 25) printf " line %d: fetch chain ends in an empty catch\n", NR }' "$f"
}
# ── Self-test ─────────────────────────────────────────────────────────────────────────────────
if [[ "$SELF_TEST" == true ]]; then
cat > "$WORK/fixture.php" <<'FIX'
<script>
function outer() {
const paletteMap = { a: '#fff' };
return paletteMap;
}
function styler(x) {
// reads a const that lives in outer() — ReferenceError at runtime
if (paletteMap[x]) return paletteMap[x];
return '';
}
function loader() {
fetch('/api/thing')
.then(r => r.json())
.then(d => { render(d); })
.catch(() => {});
}
</script>
FIX
fails=0
echo "── self-test ────────────────────────────────────────────────"
if scan_scope "$WORK/fixture.php" | grep -q "paletteMap"; then
echo " cross-scope detector PASS"
else
echo " cross-scope detector FAIL — known fault not reported"; fails=1
fi
if scan_catches "$WORK/fixture.php" | grep -q "empty catch"; then
echo " empty-catch detector PASS"
else
echo " empty-catch detector FAIL — known fault not reported"; fails=1
fi
# A clean file must stay clean, or the detector is merely reporting everything.
cat > "$WORK/clean.php" <<'CLN'
<script>
const paletteMap = { a: '#fff' };
function styler(x) { return paletteMap[x] || ''; }
function loader() {
fetch('/api/thing').then(r => r.json()).then(d => { styler(d); })
.catch(e => report('thing', e));
}
</script>
CLN
if [[ -z "$(scan_scope "$WORK/clean.php")$(scan_catches "$WORK/clean.php")" ]]; then
echo " clean file stays clean PASS"
else
echo " clean file stays clean FAIL — false positive on correct code"; fails=1
fi
echo "─────────────────────────────────────────────────────────────"
[[ "$fails" -eq 0 ]] && echo "self-test OK" || echo "SELF-TEST FAILED — do not trust a clean scan"
exit "$fails"
fi
# ── Scan ──────────────────────────────────────────────────────────────────────────────────────
if [[ "${#FILES[@]}" -eq 0 ]]; then
while IFS= read -r p; do FILES+=("$p"); done < <(
find "$PLUGIN_DIR/pages" "$PLUGIN_DIR/include" -maxdepth 1 -name '*.php' 2>/dev/null
[[ -f "$PLUGIN_DIR/Varaverk.page" ]] && echo "$PLUGIN_DIR/Varaverk.page"
)
fi
found=0
for f in "${FILES[@]}"; do
[[ -f "$f" ]] || continue
grep -q "<script" "$f" 2>/dev/null || continue
out=""
[[ "$DO_SCOPE" == true ]] && out+="$(scan_scope "$f")"
if [[ "$DO_CATCH" == true ]]; then
c="$(scan_catches "$f")"
[[ -n "$c" ]] && out+=$'\n'"$c"
fi
out="$(echo "$out" | sed '/^$/d')"
if [[ -n "$out" ]]; then
echo "### $(basename "$f")"
echo "$out"
found=$(( found + $(echo "$out" | grep -c '^ line') ))
fi
done
echo
if [[ "$found" -eq 0 ]]; then
echo "clean — no cross-scope references, no silent fetch catches"
else
echo "$found finding(s). Cross-scope hits over-report: regex literals, nested-function"
echo "parameters, destructured for-of and multi-declarator const each read as undeclared."
echo "Read each before changing anything."
fi
exit $(( found > 0 ? 1 : 0 ))
@@ -44,6 +44,17 @@
// php mesh_traffic_sample.php append one sample per mesh peer
// php mesh_traffic_sample.php --show print what it would record, write nothing
//
// CONFIGURATION
// master.conf
// HOST1, HOST2, … the hostname of each node in the mesh. Every HOST<n> with a non-empty
// value becomes a name matched against tailscale's peer list; a peer that
// is not one of them is skipped, so a device that merely shares the
// tailnet is never recorded as mesh traffic. No HOST<n> set at all means
// there is no mesh to sample and the run exits without writing.
//
// Nothing is read from host*.conf. This samples the local tailscale daemon, so it needs to
// know which peers count and nothing about how to reach them.
//
// DEPENDS ON
// tailscale status --json the counters
// include/config.php DATA_DIR, vv_conf_vars() for the HOST* list
+146 -68
View File
@@ -11,7 +11,8 @@
// --export --containers=a,b,c on the OWNER. Reads folder.view3's docker.json, intersects
// each folder with the deployed list, prints a JSON plan.
// --import on the MIRROR. Reads that plan on stdin and upserts each
// folder by name into the mirror's own docker.json.
// folder by name into BOTH the mirror's folder.view3 docker.json
// and Varaverk's own docker_folders.json.
//
// Onboard Step 12 runs the pair. Nothing is assumed about the mirror's layout: folders it
// already has are matched by name and extended, never duplicated or replaced.
@@ -39,8 +40,13 @@
//
// Atomic write: .vv.tmp then rename(), so folder.view3 never reads a truncated file.
//
// Absent plugin is a clean skip, not an error. A mirror without folder.view3 installed has
// nowhere to put folders and that is not a failure of the onboard.
// Both stores or the layout is only half real. The Monitor container card reads Varaverk's
// docker_folders.json, never folder.view3's, and include/docker.php seeds it exactly once —
// when the file is absent. A mirror onboarded before this step existed had already been
// seeded, so writing only folder.view3 left the card showing the old single lump forever.
//
// Absent plugin is no longer a skip on import. folder.view3 is optional; Varaverk's own
// store is not, and it is the one the card reads. Export still needs the plugin and says so.
//
// Import trusts nothing about shape: every folder needs a non-empty name and an array of
// container names, and anything else in the payload is ignored rather than merged.
@@ -57,12 +63,38 @@
// "unfiled":[…]}
// import: one line per folder written, then a count
//
// RUNTIME MODES
// mirror_folders.php --export
// Print this host's folder layout as JSON, for the owner to compute a plan from.
//
// mirror_folders.php --import
// Read a layout on stdin and apply it to this host's folder.view3.
//
// mirror_folders.php --containers=<csv>
// Restrict the operation to these container names.
//
// mirror_folders.php --fallback-only=<csv>
// Place only these into the partner's fallback folder, leaving the rest where they are.
//
// --dry-run
// Compute and report the plan without writing folder.view3. Combines with any of the above.
//
// DEPENDS ON
// /boot/config/plugins/folder.view3/docker.json the layout both sides read
// /boot/config/plugins/folder.view3/docker.json the third-party layout, optional on import
// SCRIPTS_DIR/docker_folders.json Varaverk's own layout — what the card reads
// Tools/fallback_folder.php still owns the "<Owner>-Fallback" folder
// ═══════════════════════════════════════════════════════════════════════════════════════════════
$pluginDir = dirname(__DIR__);
require_once $pluginDir . '/include/config.php';
define('FV3_JSON', '/boot/config/plugins/folder.view3/docker.json');
// Varaverk's own store. Written as well as folder.view3's, because the Monitor container card
// reads THIS file and nothing else — include/docker.php imports folder.view3 exactly once, when
// this file does not yet exist. On a mirror onboarded before the layout step existed, that
// bootstrap had already happened, so the card went on showing one lump "<Owner>-Fallback"
// holding the entire auth and arr stacks while folder.view3 next to it showed the real shelves.
define('VV_STORE', SCRIPTS_DIR . '/docker_folders.json');
$opts = getopt('', ['export', 'import', 'containers:', 'fallback-only:', 'dry-run']);
@@ -72,15 +104,18 @@ $listArg = function (string $k) use ($opts): array {
};
// ── Load ─────────────────────────────────────────────────────────────────────────────────────
if (!file_exists(FV3_JSON)) {
// Export has nothing to read; import has nowhere to write. Both are clean no-ops.
if (isset($opts['export'])) { echo json_encode(['folders' => [], 'unfiled' => $listArg('containers')]) . "\n"; exit(0); }
echo "folder.view3 is not installed — nothing to do\n";
// folder.view3 is optional on the mirror. Export needs it and has nothing to say without it;
// import does not — Varaverk's own store is the one the card reads, and it is always writable.
$haveFv3 = file_exists(FV3_JSON);
if (!$haveFv3 && isset($opts['export'])) {
echo json_encode(['folders' => [], 'unfiled' => $listArg('containers')]) . "\n";
exit(0);
}
$fv3 = json_decode((string)@file_get_contents(FV3_JSON), true);
if (!is_array($fv3)) { fwrite(STDERR, "folder.view3 docker.json is unreadable\n"); exit(1); }
$fv3 = [];
if ($haveFv3) {
$fv3 = json_decode((string)@file_get_contents(FV3_JSON), true);
if (!is_array($fv3)) { fwrite(STDERR, "folder.view3 docker.json is unreadable\n"); exit(1); }
}
// ── Export ───────────────────────────────────────────────────────────────────────────────────
if (isset($opts['export'])) {
$deployed = $listArg('containers');
@@ -117,6 +152,7 @@ if (isset($opts['export'])) {
exit(0);
}
// ── Import ───────────────────────────────────────────────────────────────────────────────────
if (!isset($opts['import'])) {
fwrite(STDERR, "usage: mirror_folders.php --export --containers=… | --import\n");
@@ -130,71 +166,113 @@ if (!is_array($plan) || !isset($plan['folders']) || !is_array($plan['folders']))
exit(1);
}
$written = 0;
foreach ($plan['folders'] as $spec) {
if (!is_array($spec)) continue;
$name = trim((string)($spec['name'] ?? ''));
$cs = array_values(array_filter(array_map('trim', (array)($spec['containers'] ?? [])), 'strlen'));
if ($name === '' || !$cs) continue;
// Applied to both stores, so they cannot drift apart the way they already did once. Takes a
// store, returns the store with the plan folded in plus what changed — no writing, no printing,
// because the two callers report differently.
$applyPlan = function (array $store) use ($plan): array {
$written = 0;
$lines = [];
foreach ($plan['folders'] as $spec) {
if (!is_array($spec)) continue;
$name = trim((string)($spec['name'] ?? ''));
$cs = array_values(array_filter(array_map('trim', (array)($spec['containers'] ?? [])), 'strlen'));
if ($name === '' || !$cs) continue;
$targetId = null;
foreach ($fv3 as $id => $f) {
if (is_array($f) && strcasecmp((string)($f['name'] ?? ''), $name) === 0) { $targetId = $id; break; }
$targetId = null;
foreach ($store as $id => $f) {
if (is_array($f) && strcasecmp((string)($f['name'] ?? ''), $name) === 0) { $targetId = $id; break; }
}
$created = false;
if ($targetId === null) {
// folder.view3's own id shape: 20 chars of url-safe base64.
$targetId = substr(str_replace(['+', '/', '='], '', base64_encode(random_bytes(15))), 0, 20);
$store[$targetId] = ['name' => $name, 'icon' => '', 'settings' => ['', '', '1', '', '1', ''],
'regex' => '', 'containers' => [], 'containerImages' => []];
$created = true;
}
$store[$targetId]['name'] = $name;
// Only fill an icon that is missing — a mirror that has styled its own folder keeps its choice.
if (($spec['icon'] ?? '') !== '' && trim((string)($store[$targetId]['icon'] ?? '')) === '') {
$store[$targetId]['icon'] = (string)$spec['icon'];
}
$have = (array)($store[$targetId]['containers'] ?? []);
$lc = array_map('strtolower', array_map('strval', $have));
$added = 0;
foreach ($cs as $c) {
if (in_array(strtolower($c), $lc, true)) continue;
$have[] = $c; $lc[] = strtolower($c); $added++;
}
$store[$targetId]['containers'] = array_values($have);
$lines[] = sprintf(" %-7s %-22s +%d (%s)", $created ? 'create' : 'update', $name, $added, implode(', ', $cs));
$written++;
}
$created = false;
if ($targetId === null) {
// folder.view3's own id shape: 20 chars of url-safe base64.
$targetId = substr(str_replace(['+', '/', '='], '', base64_encode(random_bytes(15))), 0, 20);
$fv3[$targetId] = ['name' => $name, 'icon' => '', 'settings' => ['', '', '1', '', '1', ''],
'regex' => '', 'containers' => [], 'containerImages' => []];
$created = true;
// ── Take the filed containers back out of any "-Fallback" folder ─────────────────────────
// A container that now sits in "Arrs Stack" must not also sit in "Gmer4Lfe-Fallback". The
// fallback folder answers "what is this host covering for the owner", and an earlier Step 12
// dumped every deployed container into it — so it claimed the whole auth and arr stacks,
// which run here continuously and are not failover coverage at all.
//
// Only containers this plan just filed are removed. Anything the operator put in that folder
// by hand, or that the onboard filed there deliberately as fallback-only, is left alone.
$filed = [];
foreach ($plan['folders'] as $spec) {
foreach ((array)($spec['containers'] ?? []) as $c) $filed[strtolower(trim((string)$c))] = true;
}
$fv3[$targetId]['name'] = $name;
// Only fill an icon that is missing — a mirror that has styled its own folder keeps its choice.
if (($spec['icon'] ?? '') !== '' && trim((string)($fv3[$targetId]['icon'] ?? '')) === '') {
$fv3[$targetId]['icon'] = (string)$spec['icon'];
$pruned = 0;
foreach ($store as $id => $f) {
if (!is_array($f) || !preg_match('/-Fallback$/i', (string)($f['name'] ?? ''))) continue;
$keep = [];
foreach ((array)($f['containers'] ?? []) as $c) {
if (isset($filed[strtolower(trim((string)$c))])) { $pruned++; continue; }
$keep[] = $c;
}
$store[$id]['containers'] = array_values($keep);
}
$have = (array)($fv3[$targetId]['containers'] ?? []);
$lc = array_map('strtolower', array_map('strval', $have));
$added = 0;
foreach ($cs as $c) {
if (in_array(strtolower($c), $lc, true)) continue;
$have[] = $c; $lc[] = strtolower($c); $added++;
return [$store, $written, $pruned, $lines];
};
$dryRun = isset($opts['dry-run']);
// ── folder.view3's file ──────────────────────────────────────────────────────────────────────
$fvWritten = 0;
if ($haveFv3) {
[$fv3, $fvWritten, $fvPruned, $fvLines] = $applyPlan($fv3);
foreach ($fvLines as $l) echo $l . "\n";
if ($fvPruned) printf(" pruned %d container(s) from -Fallback folder(s) — they are filed properly now\n", $fvPruned);
if (($fvWritten || $fvPruned) && !$dryRun) {
$tmp = FV3_JSON . '.vv.tmp';
if (file_put_contents($tmp, json_encode($fv3, JSON_UNESCAPED_SLASHES)) === false || !rename($tmp, FV3_JSON)) {
fwrite(STDERR, "failed to write " . FV3_JSON . "\n"); exit(1);
}
printf(" %d folder(s) written to folder.view3 ✅\n", $fvWritten);
}
$fv3[$targetId]['containers'] = array_values($have);
printf(" %-7s %-22s +%d (%s)\n", $created ? 'create' : 'update', $name, $added, implode(', ', $cs));
$written++;
} else {
echo " folder.view3 not installed — Varaverk's own layout only\n";
}
// ── Take the filed containers back out of any "-Fallback" folder ─────────────────────────────
// A container that now sits in "Arrs Stack" must not also sit in "Gmer4Lfe-Fallback". The
// fallback folder answers "what is this host covering for the owner", and an earlier Step 12
// dumped every deployed container into it — so it claimed the whole auth and arr stacks, which
// run here continuously and are not failover coverage at all.
//
// Only containers this plan just filed are removed. Anything the operator put in that folder by
// hand, or that the onboard filed there deliberately as fallback-only, is left alone.
$filed = [];
foreach ($plan['folders'] as $spec) {
foreach ((array)($spec['containers'] ?? []) as $c) $filed[strtolower(trim((string)$c))] = true;
// ── Varaverk's own store ─────────────────────────────────────────────────────────────────────
// Seeded from folder.view3 when it does not exist yet, which is the same bootstrap
// include/docker.php performs — done here too so the very first import lands on a real layout
// rather than an empty file.
$vvStore = [];
if (file_exists(VV_STORE)) {
$vvStore = json_decode((string)@file_get_contents(VV_STORE), true);
if (!is_array($vvStore)) $vvStore = [];
} elseif ($haveFv3) {
$vvStore = $fv3;
}
$pruned = 0;
foreach ($fv3 as $id => $f) {
if (!is_array($f) || !preg_match('/-Fallback$/i', (string)($f['name'] ?? ''))) continue;
$keep = [];
foreach ((array)($f['containers'] ?? []) as $c) {
if (isset($filed[strtolower(trim((string)$c))])) { $pruned++; continue; }
$keep[] = $c;
[$vvStore, $vvWritten, $vvPruned, ] = $applyPlan($vvStore);
if (($vvWritten || $vvPruned) && !$dryRun) {
@mkdir(dirname(VV_STORE), 0755, true);
$tmp = VV_STORE . '.vv.tmp';
if (file_put_contents($tmp, json_encode($vvStore, JSON_UNESCAPED_SLASHES)) === false || !rename($tmp, VV_STORE)) {
fwrite(STDERR, "failed to write " . VV_STORE . "\n"); exit(1);
}
if ($pruned) $fv3[$id]['containers'] = array_values($keep);
}
if ($pruned) printf(" pruned %d container(s) from -Fallback folder(s) — they are filed properly now\n", $pruned);
printf(" %d folder(s) %s Varaverk's layout%s\n", $vvWritten,
$dryRun ? 'would be written to' : 'written to',
$vvPruned ? sprintf(" (%d unfiled from -Fallback)", $vvPruned) : '');
if (!$written && !$pruned) { echo " nothing to write\n"; exit(0); }
if (isset($opts['dry-run'])) { echo " DRY RUN — nothing written\n"; exit(0); }
$tmp = FV3_JSON . '.vv.tmp';
if (file_put_contents($tmp, json_encode($fv3, JSON_UNESCAPED_SLASHES)) === false || !rename($tmp, FV3_JSON)) {
fwrite(STDERR, "failed to write " . FV3_JSON . "\n"); exit(1);
}
printf(" %d folder(s) written ✅\n", $written);
if ($dryRun) echo " DRY RUN — nothing written\n";
if (!$fvWritten && !$vvWritten) echo " nothing to write\n";
+22 -3
View File
@@ -5,12 +5,25 @@
// vv_nc_deliver(), with the record on stdin.
//
// OPERATIONAL MODEL
// php node_chat_receive.php --channel=<id> record on stdin, one JSON object
// php node_chat_receive.php --flush retry this host's own undelivered spool
//
// Exit 0 means stored. The sender treats anything else as undelivered and spools for retry, so
// a non-zero exit here is a message that will arrive later rather than one that is lost.
//
// Two jobs in one file because they are two halves of the same contract: --channel takes what
// a partner is delivering now, --flush pushes what this host failed to deliver earlier. A node
// is both a receiver and a sender, and splitting them would mean two files that must agree on
// the same spool layout.
//
// DESIGN PRINCIPLES
// The exit code is the receipt, and it is the only one.
// There is no acknowledgement message and no reply body. The sender is an SSH command that
// already has an exit status, so inventing a second channel to say the same thing would
// give the two ways to disagree.
//
// Refusing is cheaper than being wrong.
// An unparseable record, an unknown channel or a failed write all exit non-zero and leave
// the message in the sender's spool. A retry costs one SSH round trip; a message accepted
// and dropped is gone with nothing recording that it existed.
//
// OPERATIONAL SAFEGUARDS
// Reached only over SSH with a key this mesh installed, so the caller already has root. This
// file therefore validates shape, not authority — there is no privilege here to protect that
@@ -22,6 +35,12 @@
// Storage is append-and-trim through vv_nc_append(), which is idempotent on message id — a
// retry of something that already landed is a no-op rather than a duplicate.
//
// RUNTIME MODES
// Not invoked by hand — the sending node's vv_nc_deliver() runs it over SSH.
//
// php node_chat_receive.php --channel=<id> store one JSON record, read from stdin
// php node_chat_receive.php --flush retry this host's own undelivered spool
//
// DEPENDS ON
// include/node_chat.php vv_nc_append(), vv_nc_channel(), vv_nc_flush_spool()
// ═══════════════════════════════════════════════════════════════════════════════════════════════
+36
View File
@@ -22,6 +22,42 @@
# passes are not counted. Running daily keeps that to whatever NPM rotates in a day.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Incremental by byte offset, never a re-read.
# 475 MB across 41 files cannot be parsed on a page load, and re-parsing what was already
# counted would make each pass slower than the last. Each run records where it stopped and
# resumes there, so cost tracks new traffic rather than total traffic.
#
# A rotated log is detected, not assumed.
# A file smaller than the offset already recorded for it means NPM rotated it beneath us, so
# that file's offset restarts at zero and a rotation is counted. The totals are cumulative and
# are never reset by it — what rotated out between two passes is simply not counted, which is
# why this runs often enough to keep that gap to whatever NPM rotates in a day.
#
# Varaverk's own probes are excluded.
# uptime_probe.sh requests every proxied host every minute under Varaverk-Uptime/1.0. Counting
# those would make the monitor the busiest client of everything it monitors.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Read-only against NPM. Access logs are read and never rotated, truncated or removed — that is
# NPM's business, and a stats collector that deleted its own inputs would be unrecoverable.
#
# --dry-run parses and reports without writing the store or advancing any offset, so a dry run
# leaves the next real pass with exactly the same work to do.
#
# --reset forgets offsets and totals deliberately. It starts from the current end of each log,
# so it discards history rather than double-counting it.
#
# The store is replaced atomically: written to a temp file, re-read and parsed to prove it is
# valid JSON, then renamed over the original. A pass that dies midway — or produces something
# unparseable — leaves the previous document untouched rather than a truncated one.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
+13 -13
View File
@@ -15,19 +15,6 @@
# Accepts --host=HOST2 to refresh a single host (used by the UI refresh button).
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Cache-First, Never Live on Page Load
# Remote arr APIs have non-trivial latency — calling them on every page view
# would make the arrs page slow and fragile. Writing to /tmp/vv_cache/ on a
# 2-hour schedule decouples page load time from network availability.
#
# Single-Host Refresh for UI
# The UI refresh button passes --host=HOSTN to update one host without waiting
# for the full 2-hour cycle. Keeps the cache fresh when a user requests it.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
@@ -44,6 +31,19 @@
# back to live calls when a file is missing or stale.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Cache-First, Never Live on Page Load
# Remote arr APIs have non-trivial latency — calling them on every page view
# would make the arrs page slow and fragile. Writing to /tmp/vv_cache/ on a
# 2-hour schedule decouples page load time from network availability.
#
# Single-Host Refresh for UI
# The UI refresh button passes --host=HOSTN to update one host without waiting
# for the full 2-hour cycle. Keeps the cache fresh when a user requests it.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
+40 -22
View File
@@ -5,31 +5,49 @@
// to reach it. Generated so the assistant can answer "how do I change X" with a path through
// the pages instead of an instruction to open master.conf.
//
// WHY THE ASSISTANT NEEDS THIS AT ALL
// The retrieval index reads git-tracked files. PHP body markup is not indexed and would be
// useless if it were — a page is a pile of divs, not a description of itself — so the assistant
// has never had any way to know the UI exists. It could name a conf key and nothing more.
// pages/readme/*.md is the one directory the chunker classifies as kind='ui', which is why the
// output lands there and not in docs/.
//
// WHY IT IS GENERATED
// A hand-written map is a second description of the pages, and the moment a card moves it
// starts lying with total confidence — which is worse than saying nothing, because the
// assistant will repeat it. Everything here is derived from the same registries the pages
// themselves are built from: VV_SCRIPT_CONF_SECTIONS for what the Scheduler shows per script,
// VV_UI_SECTION_SURFACES for the pages that show sections by subject, and the conf files for
// the settings and their controls.
//
// OPERATIONAL MODEL
// Hand-run, and re-run after adding a conf section, a script mapping or a settings surface.
// Writes exactly one file and nothing else.
// Reads the section registries and the conf files, resolves each setting to the page and card
// that renders it, and writes the whole map in one pass. Nothing is merged with what is
// already there — the output is derived entirely from the registries, so a stale entry cannot
// survive a rebuild.
//
// php Tools/ui_map_build.php write the map
// php Tools/ui_map_build.php --check report what it would change, write nothing
// DESIGN PRINCIPLES
// The assistant cannot see the UI any other way.
// The retrieval index reads git-tracked files. PHP body markup is not indexed and would be
// useless if it were — a page is a pile of divs, not a description of itself — so the
// assistant has never had any way to know the UI exists. It could name a conf key and
// nothing more. pages/readme/*.md is the one directory the chunker classifies as
// kind='ui', which is why the output lands there and not in docs/.
//
// Only sections that are genuinely reachable are listed. A section no page renders is reported
// at the end as unreachable rather than silently omitted — a setting with no route is a real
// finding, and the map is the only place that would notice.
// Generated, because a hand-written map lies with confidence.
// A second description of the pages starts being wrong the moment a card moves, and that
// is worse than saying nothing, because the assistant will repeat it. Everything here is
// derived from the same registries the pages themselves are built from:
// VV_SCRIPT_CONF_SECTIONS for what the Scheduler shows per script, VV_UI_SECTION_SURFACES
// for the pages that show sections by subject, and the conf files for the settings and
// their controls.
//
// An unreachable section is reported, never dropped.
// A section no page renders is listed at the end rather than silently omitted. A setting
// with no route through the UI is a real finding, and this map is the only thing that
// would ever notice.
//
// OPERATIONAL SAFEGUARDS
// Writes exactly one file, pages/readme/ui-map.md, and nothing else. No conf is modified, no
// page is touched, and the registries it reads are only read.
//
// --check reports what would change and writes nothing, so the map can be verified current in
// a commit without regenerating it.
//
// Generated output only. Nothing hand-edited belongs in ui-map.md — an edit there is lost on
// the next run, which is the correct behaviour for a derived file and the reason the header
// says so.
//
// RUNTIME MODES
// php Tools/ui_map_build.php write the map
// php Tools/ui_map_build.php --check report what it would change, write nothing
//
// Hand-run. Re-run after adding a conf section, a script mapping or a settings surface.
//
// DEPENDS ON
// include/confform.php the section registries, the parser, and the inferred controls
+33
View File
@@ -25,6 +25,39 @@
# that, this monitor becomes fifty thousand requests a day in the logs it reports on.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Probe what is served, not what was visited.
# The domain list comes from NPM's own proxy hosts, so a host nobody has requested is measured
# exactly like a busy one. Deriving the list from traffic would leave the quietest hosts — the
# ones most likely to be broken without anyone noticing — permanently unmonitored.
#
# The probe is excluded from the statistics it feeds.
# Every request carries Varaverk-Uptime/1.0, which npm_access_stats.sh filters out. Without
# that the monitor would be the largest single source of traffic in the logs it reports on,
# and every access figure would be measuring this script.
#
# The wrapper holds no logic.
# Flags are forwarded verbatim and nothing is interpreted here. Two places that both know what
# --dry-run means is two places that can disagree about it.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# UPTIME_PROBE_ENABLED gates the whole run. Off means no probes and no writes.
#
# Bounded per domain by UPTIME_PROBE_TIMEOUT, so an unresponsive host costs one timeout rather
# than stalling the pass — the probes run in parallel, so one slow domain never delays the rest.
#
# --dry-run probes and reports without writing. --status and --events read stored history and
# probe nothing, so neither can alter what it is describing.
#
# Read-only against NPM. The domain list is read; no proxy host, certificate or setting is
# touched by anything in this path.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
+70 -2
View File
@@ -92,6 +92,37 @@ function vvEscAttr(s) {
.replace(/</g,'&lt;').replace(/>/g,'&gt;');
}
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// What a failed fetch says.
//
// Every fetch in this plugin used to end in `.catch(() => {})` — 24 of them. That is not error
// handling, it is error deletion: the request fails, nothing renders, nothing is logged, and the
// surface either sits on "Loading…" forever or silently keeps showing stale numbers. The mesh chat
// spent an unknown amount of time "taking a minute to load" because a ReferenceError was thrown on
// every render and swallowed here; the fault named itself the moment a catch reported it.
//
// Console always, because a poller that drops one tick should not shout on screen. A target
// element when the caller has one, because a panel that will otherwise never fill has to say why.
//
// Global for the same reason vvEscHtml is: pages/*.php are included one at a time and each would
// otherwise carry its own copy, which is the arrangement that lets two of them drift.
function vvFetchErr(where, e, el) {
const msg = (e && e.message) ? e.message : String(e || 'request failed');
try { console.warn('[varaverk] ' + where + ' — ' + msg, e); } catch (_) {}
if (el) {
const n = (typeof el === 'string') ? document.getElementById(el) : el;
if (n) { n.textContent = where + ' failed: ' + msg; n.style.color = '#a05a2c'; }
}
}
// Throws on a non-2xx instead of handing HTML to JSON.parse. Unraid answers an expired session
// with a 302 to the login page, so without this the reported error is "Unexpected token '<'",
// which names the symptom and hides the cause.
function vvJson(r) {
if (!r.ok) throw new Error('HTTP ' + r.status);
return r.json();
}
// A URL about to be put in href/src or handed to window.open. Anything that is not plainly http,
// https or a site-relative path becomes empty — javascript: is the one that matters, and an
// allowlist is the only way to say that without chasing encodings. include/docs.php applies the
@@ -446,8 +477,42 @@ $validTabs = ['monitor', 'scheduler', 'watchdog', 'partnership', 'fallback', 'ar
$_vv_ai = vv_ai_owner_ui_on();
if ($_vv_ai) $validTabs[] = 'ai';
// ── Local pages ───────────────────────────────────────────────────────────────────────────────
// pages/local/ is gitignored, so whatever is in it belongs to this installation alone and never
// reaches the public mirror. This loader is the tracked half: a generic extension point that
// knows nothing about what it is loading.
//
// It exists because the alternative — a tracked `if (file_exists(pages/thing.php))` per private
// page — puts the name and purpose of every private page into the public repo, which defeats the
// point of keeping the page out of it.
//
// Discovered rather than configured: a conf key listing local pages would itself be a tracked
// file naming them, and an untracked one would be a second thing to keep in sync with the
// directory. The directory is the declaration.
//
// The label comes from a `// vv-local-page: Name` line in the first 2KB of the file, falling back
// to the capitalised id. Reading it out of the file keeps the page self-describing — nothing
// outside it has to be edited to add one.
$localPages = [];
foreach (glob("$pluginDir/pages/local/*.php") ?: [] as $_lp) {
$_id = basename($_lp, '.php');
// Ids are restricted and collisions rejected: $tab is user input that becomes an include
// path below, and a local page must never be able to shadow a real tab.
if (!preg_match('/^[a-z0-9][a-z0-9_-]{0,31}$/', $_id)) continue;
if (in_array($_id, $validTabs, true)) continue;
$_lbl = ucfirst($_id);
if (preg_match('/^\s*(?:\/\/|#)\s*vv-local-page:\s*(.+)$/m', (string)@file_get_contents($_lp, false, null, 0, 2048), $_m)) {
$_lbl = trim($_m[1]);
}
$localPages[$_id] = ['path' => $_lp, 'label' => $_lbl];
$validTabs[] = $_id;
}
unset($_lp, $_id, $_lbl, $_m);
if (!in_array($tab, $validTabs)) $tab = 'monitor';
$tabLabels = ['monitor' => 'Monitor', 'scheduler' => 'Scheduler', 'watchdog' => 'Watchdog', 'partnership' => 'Partnership', 'fallback' => 'FallBack', 'arrs' => 'Media Stack', 'rsync' => 'Rsync', 'auth' => 'Auth Stack', 'settings' => 'Settings', 'ai' => 'AI'];
foreach ($localPages as $_id => $_lp) $tabLabels[$_id] = $_lp['label'];
unset($_id, $_lp);
// Cache stamp for the stylesheet and script below. Both are served straight off the plugin
// directory at a path that never changes, so a browser holding an old copy keeps using it after
@@ -490,9 +555,12 @@ foreach (['css/varaverk.css', 'js/varaverk.js'] as $_vv_a) {
<!-- Tab content -->
<div id="vv-content">
<?php
$page = "$pluginDir/pages/$tab.php";
// Local pages resolve from the map built above, never by composing a path out of $tab —
// the map's keys were validated against a strict pattern, so nothing user-supplied
// reaches an include.
$page = isset($localPages[$tab]) ? $localPages[$tab]['path'] : "$pluginDir/pages/$tab.php";
if (file_exists($page)) include $page;
else echo "<p>Page not found: $tab</p>";
else echo "<p>Page not found: " . htmlspecialchars($tab, ENT_QUOTES) . "</p>";
?>
</div>
+1 -1
View File
@@ -170,7 +170,7 @@ if (is_dir(LOG_DIR)) {
if ($lastErr !== null)
$errors[] = ['script' => $script, 'line' => $lastErr, 'ts' => (int)$lf->getMTime()];
}
} catch (Exception $e) {}
} catch (Exception $e) { vv_log_error('api/board.php', 'log walk failed: ' . $e->getMessage()); }
usort($errors, fn($a, $b) => $b['ts'] - $a['ts']);
}
$out['errors'] = array_slice($errors, 0, 20);
+7
View File
@@ -5,6 +5,13 @@
// state, tier activation, handback strikes, covered container status — for the fallback
// tab's 30s poll.
//
// OPERATIONAL MODEL
// Computed fresh on every request, deliberately uncached. The inputs are small local state
// files that fallback.sh rewrites as it moves between states, so assembling them costs about
// nothing — and a cached fallback picture is the one kind of stale this tab must never serve.
// A page showing NORMAL because the answer was cached before the switch is worse than a page
// that took an extra moment to load.
//
// DESIGN PRINCIPLES
// Thin transport. Every judgement about what a state file means lives in
// include/fallback.php. This file exists to give the browser a URL.
+130
View File
@@ -0,0 +1,130 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Start and stop the fallback daemon, its dry-run preview, and the failover test harness, on
// this host or on a partner. The Fallback tab's only write surface.
//
// OPERATIONAL MODEL
// Every action shells out to the script that already owns the operation — fallback.sh --stop
// and fallback_test.sh --stop — rather than signalling PIDs from PHP. Those two know things
// this layer must not have to: which lock holds the process, how long to wait, and, for the
// test, that SIGKILL must never be used because only its EXIT trap removes the iptables rule
// it installed.
//
// DESIGN PRINCIPLES
// The scripts own stopping; this endpoint owns routing.
// Duplicating the escalation logic here would put a second, divergent implementation of
// "how to stop fallback safely" in a language that cannot run its traps.
//
// Start is dry-run only.
// The live daemon is started by array_started.sh at array start, which is the one context
// where the tier delays and the state file mean what they say. A button that launched a
// live failover monitor mid-session — against a partner mid-maintenance, from a page
// someone was reading — is not a button this page should have. Previewing is safe and is
// what the page is for.
//
// OPERATIONAL SAFEGUARDS
// POST only. Unraid's CSRF token injector is jQuery-only, so a native fetch() GET would fail
// silently anyway; making these POST means a link or prefetch cannot stop a daemon.
//
// host is matched against the configured host list, never used as a path or a shell word.
// The slot resolves to a hostname from conf, then to a Tailscale IP through the same
// unambiguous-prefix resolver the rest of the mesh uses. A value that does not name a
// configured host is refused before anything runs.
//
// setsid, not nohup+&, for the dry run.
// A backgrounded child stays in php-fpm's process group, and the group kill that ends a
// request takes it with it. setsid detaches it into its own session so it survives the
// response — the same fix the Scheduler's Stop button needed.
//
// Output is captured and returned, not discarded.
// These scripts report refusals in words — "did not exit within 30s", "NOT force-killing"
// — and a boolean would throw away the only explanation the operator gets.
//
// REQUEST
// POST action=stop|stop_test|start_dry|clear_lock host=<slot>
//
// RESPONSE
// {"ok":true,"output":string} action ran; output is the script's own report
// {"ok":false,"error":string} bad method, unknown action, or unresolvable host
//
// DEPENDS ON
// include/fallback.php vv_pt_peer_lookup(), vv_pt_ts_peers(), vv_pt_ssh()
// Fallback/fallback.sh --stop
// Fallback/fallback_test.sh --stop
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once dirname(__DIR__) . '/include/fallback.php';
header('Content-Type: application/json');
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
echo json_encode(['ok' => false, 'error' => 'POST only']);
exit;
}
$action = (string)($_POST['action'] ?? '');
$slot = strtolower((string)($_POST['host'] ?? ''));
$allowed = ['stop', 'stop_test', 'start_dry', 'clear_lock'];
if (!in_array($action, $allowed, true)) {
echo json_encode(['ok' => false, 'error' => 'Unknown action']);
exit;
}
$hosts = vv_fb_known_hosts();
if (!isset($hosts[$slot])) {
echo json_encode(['ok' => false, 'error' => 'Unknown host']);
exit;
}
$isMe = ($slot === vv_detect_host());
$scripts = rtrim(SCRIPTS_DIR, '/');
// The command, as the script that owns the operation would be invoked by hand.
$cmds = [
'stop' => 'bash ' . escapeshellarg("$scripts/Fallback/fallback.sh") . ' --stop 2>&1',
'stop_test' => 'bash ' . escapeshellarg("$scripts/Fallback/fallback_test.sh") . ' --stop 2>&1',
// setsid so it outlives this request. No redirect: fallback.sh writes its own persistent
// log when stdout is not a terminal, so an ad-hoc preview and the array-start daemon leave
// their record in the same file rather than one going to tmpfs and vanishing on reboot.
'start_dry' => 'setsid bash ' . escapeshellarg("$scripts/Fallback/fallback.sh")
. ' --dry-run --log > /dev/null 2>&1 < /dev/null & echo started',
'clear_lock' => 'rm -f /tmp/unraid_locks/fallback.lock /tmp/unraid_locks/fallback_test.lock && echo cleared',
];
if ($isMe) {
@mkdir('/tmp/varaverk', 0755, true);
$out = (string)shell_exec($cmds[$action]);
echo json_encode(['ok' => true, 'output' => trim($out)]);
exit;
}
// Remote: same command, same script, over the SSH this file's neighbours already use.
$tsPeers = vv_pt_ts_peers();
$ts = vv_pt_peer_lookup($tsPeers, $hosts[$slot]);
$ip = $ts['ip'] ?? null;
$myId = strtoupper(vv_detect_host());
$sshKey = vv_fb_scalar(vv_read_conf_raw(vv_detect_host() . '.conf'), $myId . '_SSH_KEY');
if (!$ip || !$sshKey) {
echo json_encode(['ok' => false, 'error' => 'Partner not resolvable — no Tailscale IP or no SSH key']);
exit;
}
// The remote's SCRIPTS_DIR is not this host's: appdata mode on one side and flash on the other
// is the normal case on this mesh, so ask the partner where it keeps them.
$remoteDir = trim((string)vv_pt_ssh($ip, $sshKey,
'sed -n \'s/^SCRIPTS_DIR="\(.*\)"$/\1/p\' /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null'));
if ($remoteDir === '') $remoteDir = '/boot/config/plugins/varaverk';
$remoteCmds = [
'stop' => "bash '$remoteDir/Fallback/fallback.sh' --stop 2>&1",
'stop_test' => "bash '$remoteDir/Fallback/fallback_test.sh' --stop 2>&1",
'start_dry' => "setsid bash '$remoteDir/Fallback/fallback.sh'"
. " --dry-run --log > /dev/null 2>&1 < /dev/null & echo started",
'clear_lock' => 'rm -f /tmp/unraid_locks/fallback.lock /tmp/unraid_locks/fallback_test.lock && echo cleared',
];
$out = vv_pt_ssh($ip, $sshKey, $remoteCmds[$action]);
echo json_encode(['ok' => true, 'output' => trim((string)$out)]);
+418
View File
@@ -0,0 +1,418 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Failover coverage: FALLBACK_<me>_TIER1-4 in this host's own conf — which of THIS host's
// containers the partner starts when this host goes dark, and in which delay tier.
//
// OPERATIONAL MODEL
// GET this host's containers plus current tier membership.
// POST action=cover tiers=<json {container: tier}> rewrites all four tier arrays.
//
// Originally the Coverage picker on the Partnership page (e8ee5b0), removed the same day in
// 1a836da — "they describe what the partner runs during an outage, which is the Fallback tab's
// subject" — and never rehomed, because that tab had nothing to host it. This is that card,
// rebuilt where it belongs, with the services half left behind: pushing XML templates to a
// mirror is an onboard concern, not a failover one.
//
// DESIGN PRINCIPLES
// Edit the array fallback.sh actually reads, not a parallel one.
// The coverage tiers already exist and already carry the timing. A second "what to fail
// over" list would be a second answer to the same question, and the two would drift.
//
// A host edits only its OWN tiers, and the page says so.
// FALLBACK_<host>_TIER* lives in that host's conf and describes what someone else runs for
// it. Sparse checkout means this host does not have the partner's host*.conf at all — only
// the read-only RAM cache conf_sync fills — so an editor for the partner's coverage would
// be writing to a cache that the next sync overwrites. Configure HOST2's coverage from
// HOST2. This is the same trap that left the Watchdog card reporting a partner's lists as
// empty when they were merely somewhere else.
//
// OPERATIONAL SAFEGUARDS
// POST only for writes, so Unraid's CSRF guard applies.
//
// Names are validated against containers this host runs, PLUS whatever the tiers already name.
// The conf legitimately holds entries for containers not present right now — removed,
// stopped, or renamed. Validating only against the running set would refuse to save a list
// the operator never touched. New names still have to be real; the guard is against
// inventing containers, not against keeping ones already recorded.
//
// A tier outside 1-4 is rejected, never clamped. Silently moving a container from tier 9 to
// tier 4 would give it a 24-hour delay nobody asked for.
//
// An absent array is refused, not appended. Writing a new block into an unknown position in a
// conf is how a setting ends up in the wrong section and stops being read.
//
// Writing an empty list is allowed — "cover nothing" is a legitimate choice and the only way
// to express it.
//
// REQUEST
// GET → current lists
// POST action=cover tiers={"Emby":1,...} → rewrite tiers 1-4
//
// RESPONSE
// {"ok":true,...} read payload, or {"ok":true,"counts":{...}} after a write
// {"ok":false,"error":string} validation or write failure, stated
//
// DEPENDS ON
// include/confform.php vv_conf_edit(), vv_conf_last_error(), vv_parse_conf_list()
// include/common.php vv_docker_containers()
// include/config.php vv_detect_host(), vv_read_conf_raw(), vv_push_master_conf()
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
require_once dirname(__DIR__) . '/include/fallback.php'; // vv_fb_proc()
require_once dirname(__DIR__) . '/include/confform.php';
require_once dirname(__DIR__) . '/include/common.php';
$hostId = vv_detect_host();
$hostUp = strtoupper($hostId);
$myConf = $hostId . '.conf';
$TIERS = [1, 2, 3, 4];
$tierVar = fn(int $t) => "FALLBACK_{$hostUp}_TIER{$t}";
// ── Read ─────────────────────────────────────────────────────────────────────────────────────
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
$raw = vv_read_conf_raw($myConf);
$mRaw = vv_read_conf_raw('master.conf');
$cover = [];
foreach ($TIERS as $t) {
// host*.conf first, master.conf second — installs that kept the tiers there still read.
$vals = vv_parse_conf_list($raw, $tierVar($t)) ?: vv_parse_conf_list($mRaw, $tierVar($t));
foreach ($vals as $c) { $c = trim($c); if ($c !== '') $cover[$c] = $t; }
}
$containers = [];
foreach (vv_docker_containers() as $c) {
$n = is_array($c) ? ($c['name'] ?? '') : (string)$c;
if ($n !== '') $containers[] = $n;
}
sort($containers, SORT_NATURAL | SORT_FLAG_CASE);
// Stacks are declared by the PARTNERSHIP OWNER and deployed to everyone, so on a mirror they
// are not in this host's conf at all — HOST2_PARTNERSHIP_AUTH_STACK is the shipped template,
// still commented out, while the eight auth containers it describes run there permanently
// because the owner put them there. Reading "this host's" stacks left the mirror's card
// showing every one of them as ordinary, selectable, uncovered.
//
// Owner's conf first, then this host's, unioned: on the owner the two are the same file, and
// a host that declares extras of its own still has them honoured. The owner's copy reaches a
// mirror through the conf_sync RAM cache, which vv_read_host_conf_raw() knows how to find.
$ownerSlot = strtolower(vv_parse_conf_scalar(vv_read_conf_raw('master.conf'), 'PARTNERSHIP_OWNER_HOST'));
$stackSrc = [];
foreach (array_unique(array_filter([$ownerSlot, $hostId])) as $slot) {
$stackSrc[strtoupper($slot)] = vv_read_host_conf_raw($slot);
}
$stackOf = [];
$byLower = [];
foreach ($containers as $n) $byLower[strtolower($n)] = $n;
foreach ($stackSrc as $id => $srcRaw) {
foreach ([
'auth' => "{$id}_PARTNERSHIP_AUTH_STACK",
'arrs' => "{$id}_PARTNERSHIP_ARR_STACK",
'services' => "{$id}_PARTNERSHIP_SERVICES_STACK",
] as $label => $var) {
foreach (vv_parse_conf_list($srcRaw, $var) as $xml) {
$n = preg_replace('/^my-|\.xml$/', '', trim($xml));
if ($n === '') continue;
// Only what this host actually runs. The owner's stack lists everything it
// deploys mesh-wide; a name with no container here is not "always up" here.
if (!isset($byLower[strtolower($n)])) continue;
$stackOf[$byLower[strtolower($n)]] = $label;
}
}
}
// Named in a tier but not installed here. Reported rather than filtered: a tier entry for a
// container that does not exist is a line fallback.sh fails on during an outage, which is
// the worst possible moment to find a typo.
$missing = [];
$have = array_map('strtolower', $containers);
foreach (array_keys($cover) as $n) if (!in_array(strtolower($n), $have, true)) $missing[] = $n;
echo json_encode([
'ok' => true,
'host' => $hostUp,
'containers' => $containers,
'cover' => (object)$cover,
'stacks' => (object)$stackOf,
'missing' => $missing,
'tier_vars' => array_map($tierVar, $TIERS),
]);
exit;
}
// ── Write ────────────────────────────────────────────────────────────────────────────────────
// ── Readiness: would a failover actually work right now ─────────────────────────────────────
// Every row is a deterministic check with a stated basis. The assistant on this page EXPLAINS
// these rows; it never produces them. A model must not be the thing that says failover is ready —
// that is precisely the class of answer this codebase keeps finding to be confidently wrong, and
// on 2026-08-23 the coverage card itself was the confidently wrong surface: 12 containers listed,
// none of them present on the partner.
//
// Rows are ordered by what breaks first, not by severity, so reading top to bottom follows the
// order a real outage would hit them.
if (($_POST['action'] ?? '') === 'readiness') {
$rows = [];
$add = function (string $id, string $label, string $verdict, string $detail, string $ask = '')
use (&$rows) {
// verdict: ok | warn | fail | unknown — unknown is never dressed up as ok
$rows[] = ['id' => $id, 'label' => $label, 'verdict' => $verdict,
'detail' => $detail, 'ask' => $ask];
};
$me = vv_detect_host();
$ME = strtoupper($me);
$conf = vv_read_conf_raw('master.conf');
$hc = vv_read_conf_raw($me . '.conf');
// 1. is fallback even armed
$fbEnabled = preg_match('/^\s*FALLBACK_ENABLED\s*=\s*"?(\w+)/m', $conf, $m) ? $m[1] : 'unset';
$add('enabled', 'Fallback armed',
$fbEnabled === 'true' ? 'ok' : 'fail',
'FALLBACK_ENABLED=' . $fbEnabled,
'FALLBACK_ENABLED is ' . $fbEnabled . ' — what does that mean for a real outage?');
// 2. current state — anything but NORMAL means it is already doing something
$stateFile = STATE_DIR . '/fallback_state.db';
$state = 'unknown';
if (is_readable($stateFile) && preg_match('/^state=(\S+)/m', (string)@file_get_contents($stateFile), $m)) {
$state = $m[1];
}
// Same rule the node cards use: no state file plus a live daemon means the node has simply
// never transitioned, which is health, not ignorance. Reading the file alone gives a healthy
// node the same verdict as one whose daemon is dead.
$daemon = function_exists('vv_fb_proc') ? (vv_fb_proc('fallback')['running'] ?? false) : false;
$inferred = false;
if ($state === 'unknown' && $daemon) { $state = 'NORMAL'; $inferred = true; }
$add('state', 'State is NORMAL',
$state === 'NORMAL' ? 'ok' : ($state === 'unknown' ? 'unknown' : 'warn'),
'state=' . $state . ($inferred ? ' (from the live daemon — never transitioned)' : ''),
'Fallback state is ' . $state . '. What does that mean and what should I check?');
// 3. coverage configured at all
$covered = [];
for ($t = 1; $t <= 4; $t++) {
foreach (vv_parse_conf_list($hc, "FALLBACK_{$ME}_TIER{$t}") as $c) $covered[] = $c;
}
$add('coverage', 'Containers are covered',
$covered ? 'ok' : 'fail',
$covered ? count($covered) . ' container(s) across the tiers' : 'no containers in any tier',
$covered
? 'Walk me through what happens if this host goes dark right now, tier by tier, with the delays.'
: 'Nothing is listed in my fallback tiers — what would happen if this host went dark?');
// 4. THE one that was silently false — does the partner actually hold them
$cache = (defined('VV_CACHE_ROOT') ? VV_CACHE_ROOT : '/tmp/varaverk') . '/api/fallback_presence.json';
if (!is_readable($cache)) {
$add('present', 'Partner has the containers', 'unknown',
'never checked — run the presence check',
'How do I find out whether the partner actually has my covered containers?');
} else {
$j = json_decode((string)@file_get_contents($cache), true);
$miss = (array)($j['missing'] ?? []);
$age = time() - (int)@filemtime($cache);
$when = $age < 3600 ? round($age / 60) . 'm ago' : round($age / 3600) . 'h ago';
$add('present', 'Partner has the containers',
$miss ? 'fail' : 'ok',
$miss ? count($miss) . ' of ' . count($covered) . ' missing (' . $when . '): '
. implode(', ', array_slice($miss, 0, 4)) . (count($miss) > 4 ? '…' : '')
: 'all ' . count($covered) . ' present (' . $when . ')',
$miss ? 'The partner is missing ' . implode(', ', array_slice($miss, 0, 6))
. '. What happens during a failover, and how do I fix it?' : '');
}
// 5. host-specific networks that cannot be recreated on the partner
$wg = [];
foreach ($covered as $c) {
foreach (glob('/boot/config/plugins/dockerMan/templates-user/*.xml') as $x) {
$t = @file_get_contents($x);
if ($t === false || strpos($t, "<Name>$c</Name>") === false) continue;
if (preg_match('~<Network>(wg\d+)</Network>~', $t, $m)) $wg[] = "$c ({$m[1]})";
break;
}
}
if ($wg) {
$add('wgnet', 'No tunnel-bound networks', 'warn',
implode(', ', $wg),
'Some covered containers use a WireGuard-backed network. Why can that not move to the partner?');
}
// 6. handback writeback — invisible until the day it matters
$wb = preg_match('/^\s*FALLBACK_RSYNC_ENABLED\s*=\s*"?(\w+)/m', $conf, $m) ? $m[1] : 'unset';
$add('writeback', 'Handback writeback', $wb === 'true' ? 'ok' : 'warn',
'FALLBACK_RSYNC_ENABLED=' . $wb,
'FALLBACK_RSYNC_ENABLED is ' . $wb . ' — what do I lose on handback?');
// Overall verdict is the worst row, never an average. One failed check is a failed failover.
$order = ['ok' => 0, 'warn' => 1, 'unknown' => 2, 'fail' => 3];
$worst = 'ok';
foreach ($rows as $r) if ($order[$r['verdict']] > $order[$worst]) $worst = $r['verdict'];
echo json_encode(['ok' => true, 'verdict' => $worst, 'rows' => $rows,
'summary' => $worst === 'ok'
? 'Every check passed'
: ($worst === 'fail' ? 'A failover would NOT work as configured'
: 'Failover is configured but something needs a look')]);
exit;
}
// ── Push / remove / status: what the PARTNER actually holds ──────────────────────────────────
// Coverage names a container; fallback.sh starts it with `docker start`, which fails unless the
// partner already has it built. Measured 2026-08-23: 12 of 12 covered containers were absent from
// the partner, so every tier would have failed on the first real outage. These three actions are
// how the card closes and inspects that gap.
//
// Deliberately NOT folded into `cover`. Saving a tier list is a cheap, reversible config write;
// deploying a dozen containers onto another machine is neither, and a stray click should not be
// able to do it.
$_covAction = $_POST['action'] ?? '';
if (in_array($_covAction, ['push', 'remove', 'deploy_status'], true)) {
$dir = rtrim(SCRIPTS_DIR, '/');
$script = $dir . '/Fallback/coverage_deploy.sh';
$runner = $dir . '/Plugin/unraid/run_job.sh';
if (!is_file($script)) {
echo json_encode(['ok' => false, 'error' => 'coverage_deploy.sh not found on this host']);
exit;
}
// Status is read-only and fast enough to answer inline; the two that change the partner are
// dispatched to run_job.sh so they get a job record, a log, and a UI surface like every other
// long operation here.
if ($_covAction === 'deploy_status') {
$out = [];
exec('timeout 120 /bin/bash ' . escapeshellarg($script) . ' --status 2>&1', $out, $rc);
$present = []; $missing = [];
foreach ($out as $line) {
if (preg_match('/^\s{2}(\S+)\s+MISSING on/', $line, $m)) $missing[] = $m[1];
elseif (preg_match('/^\s{2}(\S+)\s+on \S+ \((\w+)\)/', $line, $m)) $present[$m[1]] = $m[2];
}
echo json_encode([
'ok' => true,
'present' => $present,
'missing' => $missing,
// rc 2 means "ran fine, some are missing" — not a failure of the check itself.
'checked' => ($rc === 0 || $rc === 2),
]);
exit;
}
if (!is_file($runner)) {
echo json_encode(['ok' => false, 'error' => 'run_job.sh not found on this host']);
exit;
}
$flag = $_covAction === 'push' ? '--push' : '--remove';
$stat = '/var/log/varaverk/Fallback/coverage_deploy.json';
shell_exec('setsid /bin/bash ' . escapeshellarg($runner)
. ' ' . escapeshellarg('Fallback/coverage_deploy.sh')
. ' ' . escapeshellarg($script)
. ' --manual ' . escapeshellarg($flag)
. ' >/dev/null 2>&1 </dev/null &');
// Report what the record says, not that the command was issued — run_job.sh writes its stat
// file before running, so a live record is the difference between a job that started and one
// refused for already running, or killed by the NORMAL-state gate.
for ($i = 0; $i < 12; $i++) {
if (is_file($stat)) {
$j = json_decode((string)@file_get_contents($stat), true);
if (is_array($j) && ($j['status'] ?? '') === 'running' && time() - filemtime($stat) < 60) {
echo json_encode(['ok' => true, 'status' => 'running', 'action' => $_covAction]);
exit;
}
}
usleep(250000);
}
echo json_encode(['ok' => false,
'error' => 'Job did not report as running — check the Fallback log. It refuses to run unless fallback state is NORMAL.']);
exit;
}
if (($_POST['action'] ?? '') !== 'cover') {
echo json_encode(['ok' => false, 'error' => 'Unknown action']);
exit;
}
$known = [];
foreach (vv_docker_containers() as $c) {
$n = is_array($c) ? ($c['name'] ?? '') : (string)$c;
if ($n !== '') $known[strtolower($n)] = $n;
}
$existingRaw = vv_read_conf_raw($myConf);
foreach ($TIERS as $t) {
foreach (vv_parse_conf_list($existingRaw, $tierVar($t)) as $n) {
$n = trim($n);
if ($n !== '' && !isset($known[strtolower($n)])) $known[strtolower($n)] = $n;
}
}
// Stack containers refused here, not only greyed out in the picker. A disabled select is a
// courtesy to the operator, not a constraint on the endpoint.
//
// Sourced from the OWNER's conf as well as this host's, for the same reason the read path is: on
// a mirror the stack it runs is the owner's declaration, and checking only the local conf would
// have let a mirror assign a fallback tier to a container that never stops.
$ownerSlotW = strtolower(vv_parse_conf_scalar(vv_read_conf_raw('master.conf'), 'PARTNERSHIP_OWNER_HOST'));
$stackNames = [];
foreach (array_unique(array_filter([$ownerSlotW, $hostId])) as $slot) {
$srcRaw = vv_read_host_conf_raw($slot);
$id = strtoupper($slot);
foreach ([
"{$id}_PARTNERSHIP_AUTH_STACK",
"{$id}_PARTNERSHIP_ARR_STACK",
"{$id}_PARTNERSHIP_SERVICES_STACK",
] as $var) {
foreach (vv_parse_conf_list($srcRaw, $var) as $xml) {
$n = preg_replace('/^my-|\.xml$/', '', trim($xml));
if ($n !== '') $stackNames[strtolower($n)] = true;
}
}
}
$map = json_decode((string)($_POST['tiers'] ?? ''), true);
if (!is_array($map)) { echo json_encode(['ok' => false, 'error' => 'tiers must be an object']); exit; }
$byTier = array_fill_keys($TIERS, []);
foreach ($map as $name => $tier) {
$t = (int)$tier;
if (!in_array($t, $TIERS, true)) {
echo json_encode(['ok' => false, 'error' => "Tier $tier is not 1-4 (for $name)"]); exit;
}
if (!isset($known[strtolower((string)$name)])) {
echo json_encode(['ok' => false, 'error' => "No container named $name on this host"]); exit;
}
if (isset($stackNames[strtolower((string)$name)])) {
// Direction-neutral wording: on the owner this container is deployed TO the partner, on a
// mirror it was deployed HERE by the owner. Both mean the same thing for coverage — it
// runs on both nodes continuously, so there is nothing for a tier to start.
echo json_encode(['ok' => false,
'error' => "$name belongs to a partnership stack — it runs on both nodes continuously, so it cannot be given a fallback tier"]); exit;
}
$byTier[$t][] = $known[strtolower((string)$name)];
}
// Rewrites one `NAME=(` … `)` block in place, preserving the conf's leading indent.
$rewrite = function (string $cur, string $var, array $items): ?string {
$body = '';
foreach ($items as $i) $body .= " \"" . $i . "\"\n";
$pattern = '/^([ \t]*)' . preg_quote($var, '/') . '=\((?:[^)]*)\)/m';
if (!preg_match($pattern, $cur)) return null; // absent: refuse rather than append blind
return preg_replace_callback($pattern,
fn($m) => $m[1] . $var . "=(\n" . $body . $m[1] . ")", $cur, 1);
};
$ok = vv_conf_edit($myConf, function (string $cur) use ($byTier, $TIERS, $tierVar, $rewrite): ?string {
foreach ($TIERS as $t) {
$next = $rewrite($cur, $tierVar($t), $byTier[$t]);
if ($next === null) return null;
$cur = $next;
}
return $cur;
}, [], array_map($tierVar, $TIERS));
if (!$ok) { echo json_encode(['ok' => false, 'error' => vv_conf_last_error() ?: 'Write failed']); exit; }
// A partner holding the old list is a partner that will act on the old list.
vv_push_master_conf();
echo json_encode(['ok' => true, 'counts' => array_map('count', $byTier)]);
+23 -6
View File
@@ -4,12 +4,19 @@
// Active media sessions endpoint. Normalised now-playing across every Emby, Jellyfin and
// Plex instance configured for this host, for the monitor page's session panel.
//
// OPERATIONAL MODEL
// Local is the default and stays the cheap path: one call per media server configured on this
// host. Mesh adds one bounded SSH hop per partner and is only requested while the operator is
// looking at the mesh view, so a dashboard left open on the default costs exactly what it did
// before the scope existed.
//
// DESIGN PRINCIPLES
// Thin transport. Discovery, per-server API dialects and normalisation all live in
// include/media.php; this file only sets the content type and encodes the result.
//
// No parameters. Which servers to ask is derived from conf, not from the request, so the
// browser cannot point this endpoint at an arbitrary URL.
// The request chooses a scope, never a target. scope= selects local or mesh; which servers
// are asked, and which partners the mesh hop reaches, are both derived from conf. The browser
// can widen what it asks for, but it cannot point this endpoint at an arbitrary URL.
//
// OPERATIONAL SAFEGUARDS
// Bounded by the library's 3s per-request timeout.
@@ -25,15 +32,25 @@
// messages a client.
//
// REQUEST
// GET, no parameters
// GET this host's sessions
// GET ?scope=mesh every node's sessions, each row tagged with the host it is playing on
//
// RESPONSE
// vv_media_sessions() verbatim — a flat list of normalised sessions across all servers
// local {"scope":"local","sessions":[…],"server_names":[…],"server_count":N}
// mesh {"scope":"mesh","nodes":[…],"sessions":[…],"server_names":[…],"server_count":N}
//
// sessions is the normalised list; in the mesh scope each row also carries the host it is
// playing on. server_count counts media servers, not sessions. There is no error shape — an
// unreachable server contributes nothing; see OPERATIONAL SAFEGUARDS.
//
// DEPENDS ON
// include/media.php vv_media_sessions()
// include/media.php vv_media_sessions(), vv_media_sessions_mesh()
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
require_once dirname(__DIR__) . '/include/media.php';
echo json_encode(vv_media_sessions());
// Anything that is not the literal "mesh" is local. Fail-closed on the expensive path, matching
// how every other toggle in this plugin reads its value.
echo json_encode(($_GET['scope'] ?? '') === 'mesh'
? vv_media_sessions_mesh()
: vv_media_sessions() + ['scope' => 'local']);
+1
View File
@@ -112,6 +112,7 @@ if (vv_ai_ui_on()) {
echo json_encode([
'system' => vv_system_info(),
'varaverk' => vv_varaverk_state(),
'fallback' => vv_fallback_state(),
'fallback_active' => vv_fallback_active(),
'partner' => vv_partner_state(),
-66
View File
@@ -1,66 +0,0 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Remote node metrics endpoint. The monitor page's partner cards — CPU, memory, storage,
// uptime and container counts for every host other than this one.
//
// OPERATIONAL MODEL
// Split from monitor.php on cost, not on subject. Local metrics are cheap file reads;
// remote metrics are SSH round trips to every partner. Keeping them on separate URLs lets
// the page poll local stats often and remote stats rarely, and lets a dark partner slow
// only its own request. Served from a 1-hour cache by default; ?live bypasses it for the
// page's explicit refresh button.
//
// DESIGN PRINCIPLES
// The cache is the default and the live call is the exception.
// An hour is deliberately long. Partner hardware stats do not move fast enough to
// justify paying SSH latency on every page load, and the refresh button exists for the
// moment someone actually needs current numbers.
//
// Every payload carries its own timestamp.
// ts is written into the cached document, so the page can render the age rather than
// presenting hour-old numbers as current.
//
// The live path writes the cache too.
// A manual refresh benefits every subsequent visitor instead of being discarded.
//
// OPERATIONAL SAFEGUARDS
// Cache miss is distinguished from empty payload.
// vv_cache_read() returns null on a miss, expiry, or unparseable file, and the check is
// an explicit !== null. A legitimately empty result — the single-host case, where there
// are no remote hosts at all — is served from cache rather than being mistaken for a
// miss and forced onto the SSH path on every single poll.
//
// Read-only over SSH. The remote commands are stat collection only; nothing is started,
// stopped, or written on a partner.
//
// Unreachable partners degrade per node inside vv_remote_hosts_stats(), so one dark host
// cannot empty the other cards.
//
// HTTP caching is disabled even though the payload is cached server-side.
// The two are not the same lever. The server-side cache has an age the page can see and
// a bypass it can trigger; a browser or proxy cache has neither, and would defeat ?live
// entirely.
//
// REQUEST
// GET served from the 3600s cache when one is present
// GET ?live bypass the cache, collect fresh, and rewrite it
//
// RESPONSE
// {"remote_hosts":{…},"ts":epoch}
//
// DEPENDS ON
// include/monitor.php vv_remote_hosts_stats(), vv_cache_read(), vv_cache_write()
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
header('Cache-Control: no-cache, no-store');
require_once dirname(__DIR__) . '/include/monitor.php';
if (!isset($_GET['live'])) {
$cached = vv_cache_read('monitor_remote', 3600);
if ($cached !== null) { echo json_encode($cached); exit; }
}
$data = ['remote_hosts' => vv_remote_hosts_stats(), 'ts' => time()];
vv_cache_write('monitor_remote', $data);
echo json_encode($data);
+35 -9
View File
@@ -25,9 +25,18 @@
// of every orchestrator. That is a distinct intent from conf_toggle.php's commenting
// out — this removes the line, that disables it in place.
//
// Indentation is normalised on re-insertion.
// The moved line is rewritten as two spaces and the quoted path, so a script does not
// carry its old array's formatting into its new one.
// Entry text moves verbatim; only a fresh entry is written from the bare path.
// An entry may carry inline arguments ("Media/media_cleaner.sh anime"), a trailing
// comment and its own indentation. Regenerating the line from the script path drops
// all three — six live entries in master.conf carry arguments, and stripping them
// would leave media_cleaner.sh with no share and fallback.sh without --stop. The
// matched line is therefore carried across untouched, which is the same rule
// reorderarray.php follows. A script that was in no array is written fresh, indented
// to match the entries already in the target.
//
// One path may hold several entries, and they move together.
// "Media/media_cleaner.sh anime" and "… media" are two jobs sharing a path. Every
// match is collected and re-inserted, rather than collapsing them into one.
//
// A move that finds nothing to move still succeeds.
// The removal pass is best-effort; only a missing *target* is an error. A script that
@@ -115,17 +124,29 @@ if (!$lines) {
exit;
}
$scriptEsc = preg_quote($script, '/');
$removedLine = null;
$inArray = false;
$scriptEsc = preg_quote($script, '/');
$removedLines = [];
$inArray = false;
// Step 1: find and remove the script line from whatever array it is currently in.
// Step 1: find and remove the script's line(s) from whatever array they are in.
//
// The original text is carried across verbatim. An entry is not just a path — it may hold
// inline arguments ("Media/media_cleaner.sh anime"), a trailing comment, and the file's
// indentation, and regenerating the line from the bare path silently drops all three. Six
// live entries in master.conf carry arguments; a move that strips them would leave
// media_cleaner.sh with no share to clean and fallback.sh without --stop.
//
// reorderarray.php preserves entry text for exactly this reason. A move must not be the one
// operation that loses it.
//
// A path can legitimately appear more than once in the same array with different arguments,
// so every match is collected and re-inserted together rather than collapsing to one.
$newLines = [];
foreach ($lines as $line) {
if (preg_match('/^\s*[A-Z_]+_SCRIPTS\s*=\s*\(/', $line)) $inArray = true;
if ($inArray && preg_match('/^\s*\)\s*(?:#.*)?$/', $line) && !str_contains($line, '(')) $inArray = false;
if ($inArray && preg_match('/^\s*(?:#\s*)?"' . $scriptEsc . '(?:\s[^"]*)?"/', $line)) {
$removedLine = ' "' . $script . '"' . "\n"; // normalise indentation when re-inserting
$removedLines[] = $line;
continue; // drop from current location
}
$newLines[] = $line;
@@ -136,10 +157,15 @@ if ($toArray) {
$resultLines = [];
$inTarget = false;
$inserted = false;
$indent = ' '; // master.conf indents array entries eight spaces
foreach ($newLines as $line) {
if (preg_match('/^\s*' . preg_quote($toArray, '/') . '\s*=\s*\(/', $line)) $inTarget = true;
// Match the indentation the target array actually uses rather than assuming it.
if ($inTarget && preg_match('/^(\s+)(?:#\s*)?"/', $line, $im)) $indent = $im[1];
if ($inTarget && !$inserted && preg_match('/^\s*\)\s*(?:#.*)?$/', $line) && !str_contains($line, '(')) {
$resultLines[] = $removedLine ?? (' "' . $script . '"' . "\n");
foreach ($removedLines ?: [$indent . '"' . $script . '"' . "\n"] as $moved) {
$resultLines[] = $moved;
}
$inTarget = false;
$inserted = true;
}
+29 -6
View File
@@ -4,12 +4,19 @@
// The Partnership tab's mesh chat: read a channel, post to it, forget a message on this
// machine, and mark a channel read.
//
// REQUEST
// GET channels + this host's id + unread counts
// GET ?channel=<id> that channel's messages
// POST action=send channel=<id> text=… [color=#rrggbb] [font=mono|sans|serif]
// POST action=delete channel=<id> id=<msgid> local only
// POST action=read channel=<id> mark seen up to now
// OPERATIONAL MODEL
// The method is the routing. Anything that is not a POST is a read — channel list or one
// channel's messages — and every POST carries an action. That keeps the CSRF boundary and the
// read/write boundary on the same line, so a mutation cannot arrive un-covered by being
// spelled as a GET.
//
// Reads are computed per request, not cached. Unread counts come from walking the last 200
// messages of each channel against this host's read mark; the card polls on a slow cycle and
// the store is a small append log, so a cache would add a staleness class for no gain.
//
// Delivery is not part of the response's success. vv_nc_send() writes locally and spools for
// any partner that could not be reached, and the reply carries that spool depth as `queued`.
// A sleeping partner is a pending message, not a failed one.
//
// DESIGN PRINCIPLES
// Read marks are local and per channel. "Unread" is a fact about this operator at this
@@ -28,6 +35,22 @@
// locally and spooled for retry; saying "failed" over something that is stored and queued
// would be the wrong claim.
//
// REQUEST
// GET channels + this host's id + unread counts
// GET ?channel=<id> that channel's messages
// POST action=send channel=<id> text=… [color=#rrggbb] [font=mono|sans|serif]
// POST action=delete channel=<id> id=<msgid> local only
// POST action=read channel=<id> mark seen up to now
//
// RESPONSE
// {"ok":true,"me":<host>,"channels":[{…,"unread":N}],"hostnames":{id:name}} channel list
// {"ok":true,"me":<host>,"channel":<id>,"messages":[…],"last_read":<ts>} one channel
// {"ok":true,"msg":{…},"queued":N} send; N = spooled
// {"ok":true|false} delete
// {"ok":true} read
// {"ok":false,"error":"Unknown channel"|"Nothing to send"|"Could not store message"
// |"No message id"|"Unknown action"}
//
// DEPENDS ON
// include/node_chat.php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
+10
View File
@@ -5,6 +5,16 @@
// reachability, SSH trust, conf-sync state and the shared-service inventory — for the
// partnership tab's poll.
//
// OPERATIONAL MODEL
// One call, one document. The endpoint holds no logic and adds no cache of its own — it
// encodes whatever vv_partnership_all() returns, so any freshness policy is the library's and
// there is only one place it can be changed.
//
// The whole picture is assembled per request rather than exposed as separate endpoints for
// hosts, trust and services. Those answers are read together and judged against each other —
// a host that is reachable but has lost SSH trust is a different state from either fact alone,
// and splitting them would let the tab render a combination that never existed at one moment.
//
// DESIGN PRINCIPLES
// Thin transport. Host enumeration, SSH probing and trust evaluation live in
// include/partnership.php; this file only sets the content type and encodes.
+13
View File
@@ -4,6 +4,19 @@
// Script and document reader. Returns the full text of one .sh or .md file inside
// SCRIPTS_DIR — the source view behind the scheduler page's script viewer and the docs tab.
//
// OPERATIONAL MODEL
// Validate, resolve, read. The id is checked against a character class and an extension list
// before it is joined to SCRIPTS_DIR, so nothing reaches the filesystem that did not already
// look like a repo-relative path.
//
// The whole file is returned in one response — no ranges, no pagination. These are scripts and
// documents, not logs; the largest is a few hundred kilobytes, and a viewer that had to stitch
// pages together would be more machinery than the thing it displays.
//
// Every failure is a JSON body with ok:false, never an HTTP error code. The scheduler's viewer
// and the docs tab both parse the response before looking at anything else, so a 404 would
// surface as a parse failure rather than as "that file is not there".
//
// DESIGN PRINCIPLES
// Two extensions, one endpoint.
// Scripts and their READMEs are read the same way because they are read for the same
+1 -1
View File
@@ -76,7 +76,7 @@ try {
'dur' => isset($d['end']) ? max(0, (int)$d['end'] - (int)$d['start']) : 0,
];
}
} catch (Exception $e) {}
} catch (Exception $e) { vv_log_error('api/recent.php', 'run-log walk failed: ' . $e->getMessage()); }
usort($runs, fn($a, $b) => $b['start'] - $a['start']);
echo json_encode(['ok' => true, 'runs' => array_slice($runs, 0, 24)]);
+32
View File
@@ -91,6 +91,38 @@ require_once dirname(__DIR__) . '/include/monitor.php';
// ── Live sync log ─────────────────────────────────────────────────────────────
$action = $_GET['action'] ?? '';
// ── Share list, for the quick-select strip on the window editor ──────────────────────────────
// The editor is path-first — type or browse, then Add — which is exact but slow for the case that
// is nearly all of them: add one whole share. This answers "what shares exist here", so the strip
// can offer them as one click each.
//
// Read from /boot/config/shares/*.cfg rather than by listing /mnt/user, because a share is a
// declared thing: the directory can be absent on a share that has never been written to, and
// listing the mount would also invent entries for stray directories that are not shares at all.
if ($action === 'shares') {
$out = [];
foreach (glob('/boot/config/shares/*.cfg') ?: [] as $cfg) {
$name = basename($cfg, '.cfg');
$raw = (string) @file_get_contents($cfg);
$get = function (string $k) use ($raw): string {
return preg_match('/^' . $k . '="([^"]*)"/m', $raw, $m) ? $m[1] : '';
};
$path = '/mnt/user/' . $name;
$out[] = [
'name' => $name,
'path' => $path,
'comment' => $get('shareComment'),
'pool' => $get('shareCachePool'),
'cache' => $get('shareUseCache'),
// Present is not the same as declared — a share can exist in conf with no directory yet.
'exists' => is_dir($path),
];
}
usort($out, fn($a, $b) => strcasecmp($a['name'], $b['name']));
header('Content-Type: application/json');
echo json_encode(['ok' => true, 'shares' => $out]);
exit;
}
if ($action === 'rsync_log') {
$lockDir = '/tmp/unraid_locks';
$lines = [];
+1 -1
View File
@@ -123,7 +123,7 @@ if ($action === 'list_scripts') {
$label = str_replace('_', ' ', basename($rel, '.sh'));
$groups[$folder][] = ['id' => $rel, 'label' => $label];
}
} catch (Exception $e) {}
} catch (Exception $e) { vv_log_error('api/rsync_win_arrays.php', 'script tree walk failed: ' . $e->getMessage()); }
ksort($groups);
foreach ($groups as &$g) usort($g, fn($a, $b) => strcmp($a['id'], $b['id']));
echo json_encode(['ok' => true, 'groups' => $groups]);
+13 -1
View File
@@ -5,6 +5,16 @@
// resource, docker, system, storage, network and stability — together with the thresholds
// each one is judging against, for the watchdog tab's poll.
//
// OPERATIONAL MODEL
// Served from a 5-minute cache unless ?live is present. Assembling this payload reads every
// watchdog's state files and resolves every threshold out of master.conf, which is far more
// work than the tab's poll needs — the watchdogs themselves only run every 15 minutes, so a
// fresher answer would describe the same cycle.
//
// The cache is consulted before include/watchdog.php is even loaded, so a cache hit costs one
// file read and nothing else. ?live skips the read, recomputes, and writes the result back, so
// an explicit refresh also benefits the next visitor rather than being discarded.
//
// DESIGN PRINCIPLES
// Thin transport. State-file parsing and threshold resolution live in
// include/watchdog.php; this file only sets the content type and encodes.
@@ -15,7 +25,9 @@
// against, so the page never has to fetch the two independently and risk mismatching
// them across a conf edit.
//
// No parameters. Which watchdogs exist is fixed by the codebase, not by the request.
// The only parameter is a freshness override. ?live decides how old an answer may be, never
// what is in it — which watchdogs exist is fixed by the codebase, and no request can select,
// filter or widen the set.
//
// OPERATIONAL SAFEGUARDS
// Read-only. Nothing here clears a strike, lifts a skip-list entry, restarts a container,
+1 -1
View File
@@ -95,7 +95,7 @@
//
// DEPENDS ON
// include/config.php vv_conf_vars(), SCRIPTS_DIR
// Media/upgrade_webhook_handler.sh the backgrounded handler
// Arrs_Stack/upgrade_webhook_handler.sh the backgrounded handler
// master.conf DOWNLOAD_WEBHOOK_ENABLED
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
+121 -36
View File
@@ -190,7 +190,32 @@ body.vv-fullscreen #displaybox { padding-left: 1rem !important; padding-top: .5r
.vv-nb-settings { display: flex; align-items: center; gap: 8px; flex-wrap: wrap;
padding: 5px 10px; background: #101010; border-bottom: 1px solid #1a1a1a; }
/* ── Monitor: locked row height + scrollable cards ────────────────────────── */
/* ── Monitor: the board ───────────────────────────────────────────────────────
Only the half of the layout that is the same at every width lives here. Which cards exist,
how wide each one is, what order they sit in, how many columns the board has at this width
and how tall a row may be are all generated by include/monitor_board.php and arrive as
--vv-cols, --vv-sp and --vv-rowdiv. Do not put a column count or a breakpoint in this file;
they are arithmetic over the card floor, and that arithmetic has one home.
minmax(0, 1fr), not 1fr: bare 1fr means minmax(auto, 1fr), and that auto floor resolves to the
larger of min-content and min-width — one card holding an unbreakable string would push its
column past its share and drag the grid out of the container. The row axis carried this guard
for years; the column axis never did.
min-width: 0 on the cards for the same reason, overriding the 200px .vv-card floor the flex
layouts on other pages need. On this board the ladder is what guarantees a card's width, so a
card defending its own floor can only overflow the track the ladder just sized for it. */
#vv-monitor {
display: grid;
grid-template-columns: repeat(var(--vv-cols, 8), minmax(0, 1fr));
gap: 12px;
width: 100%;
box-sizing: border-box;
}
#vv-monitor > .vv-card {
grid-column: span var(--vv-sp, 1);
min-width: 0;
}
/* Cards on the monitor grid are flex columns — h3 pins, body scrolls */
#vv-monitor .vv-card {
@@ -210,32 +235,65 @@ body.vv-fullscreen #displaybox { padding-left: 1rem !important; padding-top: .5r
/* Hide scrollbars on any nested scrollable div inside monitor cards */
#vv-monitor .vv-card div::-webkit-scrollbar { display: none; }
/* System card, held to the Network card's height.
At eight columns these two share a row (system 1 + ups 1 + cpu 2 + memory 2 + network 2), so
the grid already stretches them to the same height and nothing here is needed. Below eight the
board re-cuts into rows of four: System lands in row 1 with UPS and CPU, Network in row 2 with
Memory, and once rows are sized to content (vv_mon_rung_overflows) the two are free to differ.
System is the taller — it carries a host block and a Varaverk block — so it is the one capped.
222px is Network measured rather than guessed: 2 border + 24 padding + 25 h3 + 61 header block
+ 110 canvas. The canvas is a fixed 110px, which is what makes the number worth writing down —
most of Network's height cannot drift. What can is the IP list (LAN/EXT/TS), roughly 18px a row,
so a host resolving fewer of the three leaves System slightly tall against it.
Two ids on the selectors, to outrank #vv-monitor .vv-card from the auto-rows hatch above. That
hatch lifts overflow off every card so content-sized rows can work; this card is the one place
the clamps have to go back on, or a max-height with overflow:visible would draw straight
through the card's own border. */
@media (max-width: 1383px), (max-height: 700px) {
#vv-monitor #vv-system { max-height: 222px; overflow: hidden; }
#vv-monitor #vv-system > div { overflow-y: auto; min-height: 0; }
}
/* Dynamic row heights capped per screen tier — rows size to content, never exceed the cap.
minmax(0, Xpx): track is content-driven but capped; align-items:stretch makes all cards
in a row fill the track, so short cards (Pools, Watchdog) match tall ones (Array).
Breakpoints are viewport height (after browser chrome), not screen height. */
Breakpoints are viewport height (after browser chrome), not screen height.
The divisor is --vv-rowdiv, not a literal 4. It used to be 4 because the board happened to be
four rows at eight columns, so the two numbers were the same by accident and only one of them
moved when the column count changed. monitor_board.php derives it as min(rows at this rung,
VV_MON_ROWS_PER_SCREEN): the cap means "no card taller than this fraction of the screen", and a
board short enough to fit divides by its own row count and fills the screen instead. */
/* ~720p (viewport ≤ 700px) */
@media (min-width: 481px) and (max-height: 700px) {
#vv-monitor { grid-auto-rows: minmax(0, calc((100vh - 160px) / 4)); }
}
/* ~1080p (viewport 7011100px) */
@media (min-width: 481px) and (min-height: 701px) and (max-height: 1100px) {
#vv-monitor { grid-auto-rows: minmax(0, calc((100vh - 240px) / 4)); }
@media (min-height: 701px) and (max-height: 1100px) {
#vv-monitor { grid-auto-rows: minmax(0, calc((100vh - 240px) / var(--vv-rowdiv, 4))); }
}
/* ~1440p (viewport 11011450px) — calibrated on 15" 1440p display */
@media (min-width: 481px) and (min-height: 1101px) and (max-height: 1450px) {
#vv-monitor { grid-auto-rows: minmax(0, calc((100vh - 335px) / 4)); }
@media (min-height: 1101px) and (max-height: 1450px) {
#vv-monitor { grid-auto-rows: minmax(0, calc((100vh - 335px) / var(--vv-rowdiv, 4))); }
}
/* ~4K (viewport > 1450px) */
@media (min-width: 481px) and (min-height: 1451px) {
#vv-monitor { grid-auto-rows: minmax(0, calc((100vh - 500px) / 4)); }
@media (min-height: 1451px) {
#vv-monitor { grid-auto-rows: minmax(0, calc((100vh - 500px) / var(--vv-rowdiv, 4))); }
}
/* Mobile: natural heights, let page scroll */
@media (max-width: 480px) {
#vv-monitor { grid-auto-rows: auto !important; }
#vv-monitor .vv-card { overflow: visible !important; }
/* Natural heights, page scrolls. Height is what makes a phone in landscape unusable, not width:
this hatch was keyed to max-width:480 alone, so portrait (393px wide) got content-sized rows and
worked, while landscape (851x393) missed it and inherited the four-row cap — 58px cards with
overflow:hidden and scrollbars disabled. Nothing was visible and nothing said so.
Height only now. The width half was a literal 695 that had to be kept in step with
VV_MON_CARD_FLOOR by hand; monitor_board.php emits these same three rules for every rung whose
row count exceeds VV_MON_ROWS_PER_SCREEN — see vv_mon_rung_overflows() — which covers that band
and the four-column one above it from the arithmetic instead. What is left here is the case the
ladder cannot see: a window wide enough for a rung that does fit a screen, on a screen too short
to give those rows a usable height. */
@media (max-height: 700px) {
#vv-monitor { grid-auto-rows: auto; }
#vv-monitor .vv-card { overflow: visible; }
#vv-monitor .vv-card > div { overflow-y: visible; min-height: auto; }
}
@@ -251,13 +309,17 @@ body.vv-fullscreen #displaybox { padding-left: 1rem !important; padding-top: .5r
default stays start so a card added later has to say so. */
#vv-monitor-ai {
display: grid;
grid-template-columns: repeat(8, 1fr);
grid-template-columns: repeat(8, minmax(0, 1fr));
gap: 12px;
margin-top: 12px;
width: 100%;
box-sizing: border-box;
align-items: start;
}
/* Same reason as the board above: the ladder guarantees the width, so a card defending the 200px
.vv-card floor can only overflow the track it was given. At the four-column rung's own minimum
(696px) a column is 160px, and that floor would have pushed this row out of the page. */
#vv-monitor-ai > .vv-card { min-width: 0; }
/* Explicit placement, two rows. The left column carries AI over Tokens; Conversations and the
Assistant span both rows, so the Assistant sets the height and the two stacked cards divide
@@ -354,8 +416,14 @@ body.vv-fullscreen #displaybox { padding-left: 1rem !important; padding-top: .5r
#vv-ai-tokens-card .vv-ai-tok-head:first-child { margin-top: 0; }
/* Medium width — 4 columns, mirroring #vv-monitor's own breakpoint. The 1/2/5 spans do not
survive the narrower grid: 5 of 4 would silently overflow the track. */
@media (max-width: 1024px) {
survive the narrower grid: 5 of 4 would silently overflow the track.
1383 and 695 below are the four- and two-column rungs from include/monitor_board.php, minus
one. This row keeps its own placement — the 1/2/5 split is not a power of two and does not want
to be — but it cannot keep its own breakpoints: at any width where these two disagree the board
above is four columns while this row is still eight, and the page visibly stops being one
board. If VV_MON_CARD_FLOOR moves, these move with it. */
@media (max-width: 1383px) {
#vv-monitor-ai { grid-template-columns: repeat(4, 1fr) !important; }
#vv-ai-stats-card { grid-column: 1 / span 1 !important; grid-row: 1 !important; }
#vv-ai-tokens-card { grid-column: 1 / span 1 !important; grid-row: 2 !important; }
@@ -365,10 +433,14 @@ body.vv-fullscreen #displaybox { padding-left: 1rem !important; padding-top: .5r
#vv-ai-assistant-card { grid-column: 1 / span 4 !important; grid-row: 3 !important; }
}
/* Phone — single column, natural heights, page scrolls. Matching heights is meaningless once
/* Narrow — single column, natural heights, page scrolls. Matching heights is meaningless once
the cards are stacked, and the inner scroll has to go with it: on a page that already scrolls,
a scroll region inside it is a trap for a thumb. */
@media (max-width: 480px) {
a scroll region inside it is a trap for a thumb.
695, where the board above drops to two columns, rather than the 480 this used to use. There is
no two-column layout for this row and there should not be: its 1/1/3/4 split at four columns is
already the narrowest arrangement in which the Assistant transcript is readable. */
@media (max-width: 695px) {
#vv-monitor-ai { grid-template-columns: 1fr !important; }
/* grid-row has to be released along with grid-column. The explicit rows above are the whole
reason Tokens sits under AI, and left in place on a single-column grid they would stack
@@ -389,14 +461,14 @@ body.vv-fullscreen #displaybox { padding-left: 1rem !important; padding-top: .5r
.vv-cpu-core { min-width: 7px !important; }
}
@media (max-width: 1024px) {
#vv-monitor { grid-template-columns: repeat(4, 1fr) !important; }
#vv-docker { grid-column: span 4 !important; }
/* Reset explicit placements so cards reflow in the 4-col grid */
#vv-docker-folders { grid-column: span 4 !important; }
#vv-parity-card { grid-column: auto !important; }
#vv-storage-card { grid-column: auto !important; }
#vv-array-card { grid-column: auto !important; }
/* The four-column rung. Nothing about placement is here any more: the column count and every
span come from the generated ladder, and the six !important overrides this block used to carry
existed only to out-shout the inline styles the cards no longer have. Two of them
(#vv-storage-card, #vv-array-card) were resetting hand-placed column indexes — 3/span 2 and
5/span 4 — which is the pair that made the grid invent implicit columns at any count but eight.
What is left is the one thing that genuinely belongs to width: at this rung the Streams card is
half the board rather than a quarter, and its right-hand chips no longer fit. */
@media (max-width: 1383px) {
/* Streams header: hide right chip group entirely, keep server badges + media type */
.vv-stream-right { display: none; }
}
@@ -441,10 +513,7 @@ body.vv-fullscreen #displaybox { padding-left: 1rem !important; padding-top: .5r
.vv-cpu-cores { gap: 2px !important; }
.vv-cpu-core { min-width: 6px !important; }
/* Monitor single-column — explicit placement cards need override too */
#vv-monitor { grid-template-columns: 1fr !important; }
#vv-monitor > .vv-card { grid-column: 1 / -1 !important; }
#vv-docker-folders { grid-column: 1 / -1 !important; }
/* The single-column rung is generated — see include/monitor_board.php. */
}
/* Shared footer (Save Schedule left, info right) — same min-height so log card ends level with script cards */
@@ -1167,6 +1236,19 @@ mark { background: #5d4037; color: #ffcc80; border-radius: 2px; }
.vv-chip-group-device, .vv-chip-group-res, .vv-chip-group-codec { display: contents; }
.vv-stream-empty { color: #555; font-style: italic; font-size: 12px; margin: 4px 0; }
.vv-stream-empty span { font-size: 11px; color: #444; }
/* Streams scope control. Deliberately named vv-strm-* rather than reusing a bare utility name:
Unraid Connect injects a global Tailwind layer into every page, and a class named after a
utility gets whatever that layer says. */
.vv-strm-tab { background: #111; border: 1px solid #242424; color: #555; font-size: 10px;
padding: 2px 10px; border-radius: 3px; cursor: pointer; text-transform: uppercase;
letter-spacing: .05em; }
.vv-strm-tab:hover { color: #bbb; border-color: #3a3a3a; }
.vv-strm-tab.on { color: #4caf50; border-color: #2d4a2d; background: #0d1a0d; }
/* The node a stream is playing on. Only rendered in mesh scope — in local scope every row is
this host and the label would be noise on every line. */
.vv-strm-node { font-size: 9px; color: #4a7a9f; border: 1px solid #24384a; background: #0d151c;
border-radius: 2px; padding: 0 4px; margin-left: 5px; white-space: nowrap; }
.vv-stream-row { margin-bottom: 10px; padding-bottom: 10px; border-bottom: 1px solid #282828; }
.vv-stream-row:last-child { margin-bottom: 0; padding-bottom: 0; border-bottom: none; }
.vv-stream-top { display: flex; align-items: center; gap: 6px; margin-bottom: 4px; }
@@ -1283,8 +1365,11 @@ code.vv-unknown-var { color: #ff9800; background: #1f130d; }
.vv-df-fname { flex: 1; font-size: 12px; color: #aaa; font-weight: 500;
white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.vv-df-folder-body { padding-left: 10px; padding-bottom: 3px; }
.vv-df-container { display: flex; align-items: center; gap: 7px; padding: 3px 6px;
cursor: pointer; border-radius: 3px; user-select: none; }
/* padding-left is 4px against a 2px border so a container with no fallback tier still lines up
with one that has a stripe — the border is always present, only its colour changes. */
.vv-df-container { display: flex; align-items: center; gap: 7px; padding: 3px 6px 3px 4px;
cursor: pointer; border-radius: 3px; user-select: none;
border-left: 2px solid transparent; }
.vv-df-container:hover { background: rgba(255,255,255,0.04); }
.vv-df-active { background: rgba(100,149,237,0.1) !important;
outline: 1px solid rgba(100,149,237,0.35);
+108
View File
@@ -1483,6 +1483,114 @@ function vv_ai_bug_report(array $b): string {
// Strictly read-only, and there is no counterpart that changes any of it. Knowing a container is
// down is what lets an explanation be about this machine instead of about Unraid in general;
// restarting it is a decision that belongs to a person looking at the screen.
// ══════════════════════════════════════════════════════════════════════════════════════════════
// Fallback readiness, for the assistant on the Fallback tab.
//
// Fallback differs from every other subsystem here in one way that shapes this whole function: it
// is DORMANT until it isn't. A watchdog leaves strikes and restarts to reason about; fallback
// leaves nothing at all until a real outage, so "looks fine" and "would work" are unrelated. On
// 2026-08-23 the coverage card showed 12 containers configured and every one of them was absent
// from the partner — a failover would have started nothing, and no surface said so.
//
// So this reports what would ACTUALLY happen, not what is configured to happen, and it is explicit
// about the difference between the two.
//
// Never blocks on the network. Partner presence costs an SSH round trip per container, which is far
// too slow for a question already waiting on a model, so it is read from the cache
// coverage_deploy.sh --status writes and reported WITH ITS AGE. A stale answer stated as stale is
// useful; a stale answer stated as current is the failure this whole feature exists to prevent.
// ══════════════════════════════════════════════════════════════════════════════════════════════
function vv_ai_fallback_state(): string {
$me = vv_detect_host();
if ($me === '') return '';
// vv_detect_host() returns the LOWERCASE slug (host1); the conf keys are uppercase
// (FALLBACK_HOST1_TIER1). Building the key from the slug as-is silently matched nothing and
// reported "NOTHING is covered" on a host with twelve covered containers — a confidently
// wrong answer, which is the one outcome this block must never produce.
$ME = strtoupper($me);
$conf = vv_read_conf_raw('master.conf');
if ($conf === '') return '';
$s = "FALLBACK READINESS (read-only — you cannot change any of it, and you must never tell the "
. "operator a failover will work unless the evidence below says so)\n";
// ── current state ───────────────────────────────────────────────────────────────────────
$stateFile = STATE_DIR . '/fallback_state.db';
$state = 'unknown'; $since = '';
if (is_readable($stateFile)) {
$raw = (string) @file_get_contents($stateFile);
if (preg_match('/^state=(\S+)/m', $raw, $m)) $state = $m[1];
if (preg_match('/^fallback_start=(\d+)/m', $raw, $m) && (int)$m[1] > 0) {
$since = ' since ' . date('Y-m-d H:i', (int) $m[1]);
}
$age = time() - (int) @filemtime($stateFile);
// The steady NORMAL path writes nothing, so an old mtime is not staleness — it is quiet.
$s .= "- state: $state$since (state file last written "
. ($age < 3600 ? round($age / 60) . ' minutes' : round($age / 86400) . ' days') . " ago; "
. "the NORMAL path writes nothing, so an old file means nothing has changed)\n";
} else {
// No file is not the same as not running: fallback.sh writes only on a transition.
$live = function_exists('vv_fb_proc') ? (vv_fb_proc('fallback')['running'] ?? false) : false;
$s .= $live
? "- state: NORMAL (inferred — the daemon is running and has never recorded a transition, "
. "so it has written no state file; this is health, not ignorance)\n"
: "- state: no state file AND no running daemon — fallback is not operating on this host\n";
}
foreach (['FALLBACK_ENABLED', 'FALLBACK_RSYNC_ENABLED'] as $k) {
if (preg_match('/^\s*' . $k . '\s*=\s*"?(\w+)"?/m', $conf, $m)) {
$s .= "- $k: {$m[1]}"
. ($k === 'FALLBACK_RSYNC_ENABLED' && $m[1] !== 'true'
? " <- handback writeback is OFF: anything the partner writes while covering "
. "for this host never comes home\n" : "\n");
}
}
// ── coverage, tier by tier, with the real delays ─────────────────────────────────────────
$hostConf = vv_read_conf_raw($me . '.conf');
$covered = [];
for ($t = 1; $t <= 4; $t++) {
$names = vv_parse_conf_list($hostConf, "FALLBACK_{$ME}_TIER{$t}");
if (!$names) continue;
$delay = '';
if ($t > 1 && preg_match('/^\s*' . $ME . '_TIER' . $t . '_DELAY\s*=\s*"?(\d+)/m', $hostConf, $m)) {
$delay = " after {$m[1]} minutes";
}
$s .= "- tier $t" . ($t === 1 ? ' (immediate)' : $delay) . ': ' . implode(', ', $names) . "\n";
foreach ($names as $n) $covered[] = $n;
}
if (!$covered) {
$s .= "- coverage: NOTHING is covered — a failover would start no containers at all\n";
return $s . "\n";
}
// ── does the partner actually have them ─────────────────────────────────────────────────
$cache = '/tmp/varaverk/api/fallback_presence.json';
if (is_readable($cache)) {
$j = json_decode((string) @file_get_contents($cache), true);
$age = time() - (int) @filemtime($cache);
$miss = (array) ($j['missing'] ?? []);
$have = array_keys((array) ($j['present'] ?? []));
$when = $age < 3600 ? round($age / 60) . ' minutes ago' : round($age / 3600) . ' hours ago';
if ($miss) {
$s .= "- ON THE PARTNER (checked $when): " . count($miss) . ' of ' . count($covered)
. " covered container(s) DO NOT EXIST there: " . implode(', ', $miss) . "\n"
. " fallback.sh starts a covered container with `docker start`; it never creates one, "
. "so each of those would fail during a real outage. Push them from the Fallback "
. "coverage card.\n";
} else {
$s .= "- ON THE PARTNER (checked $when): all " . count($have)
. " covered container(s) exist there\n";
}
} else {
$s .= "- ON THE PARTNER: not checked. Say so plainly — whether a failover would actually "
. "start anything is UNKNOWN until the coverage card's presence check runs.\n";
}
return $s . "\n";
}
function vv_ai_system_state(): string {
$p = '/tmp/varaverk/api/monitor.json';
if (!is_readable($p)) return '';
+43 -39
View File
@@ -209,8 +209,6 @@ function vv_ai_chat_assets(): void {
/* Folded answers cap at a readable height with a hard bottom edge rather than a fade a fade
over a code block reads as a rendering fault. */
.vv-ai-fold { max-height:420px; overflow:hidden; }
.vv-ai-more { display:block; margin:4px 0 2px; }
.vv-ai-last { display:flex; gap:10px; padding:0 2px 4px; }
.vv-ai-lnk { background:none; border:none; padding:0; font-size:10px; color:#4a4a4a;
@@ -945,6 +943,14 @@ vv_ai_profiles_script();
// Per instance, like the reasoning key beside it: the Monitor card and a tab's assistant are
// opened for different reasons and should not share one answer to "pick up where I left off".
const RESUME_KEY = 'vvAiResume:' + P;
// WHICH thread this card was last on, per card. The chat store is shared on purpose — a
// conversation started on the dashboard is the one you carry on in a tab — but "resume" was
// resolved as "the newest thread in this profile", and Fallback, Partnership and Monitor all
// pin the same profile. Opening any of them dragged all three onto whichever had been typed
// in last, so two cards could never hold two different conversations. The pool below is still
// the shared list; this only decides which of it this card comes back to.
const LAST_CHAT_KEY = 'vvAiLastChat:' + P;
const rememberChat = id => { try { id ? localStorage.setItem(LAST_CHAT_KEY, id) : localStorage.removeItem(LAST_CHAT_KEY); } catch (_) {} };
const seeThink = () => { const c = $('see-think'); return !!c && c.checked; };
// Paths ride in data attributes rather than an onclick. They come out of the index, and a
@@ -1002,9 +1008,6 @@ vv_ai_profiles_script();
}
const node = el(h + '</div>');
chatEl().appendChild(node);
// Measured after it is in the DOM — scrollHeight is 0 on a detached node, so a fold decided
// before appending would either never fire or fire on everything.
foldIfLong(node);
// Obeys Follow like every other write. This was the one place that did not, and it was the
// worst place for it: unticking the box to hold your place while an answer arrives, only to
// be dragged to the bottom the instant it lands.
@@ -1012,19 +1015,11 @@ vv_ai_profiles_script();
syncJump();
}
// A long answer buries the composer on a 15" panel, and the composer is where the next thing
// happens. Folded to a readable height with the control always visible — never hover-revealed,
// because the surfaces this runs on have no pointer near them.
const FOLD_PX = 420;
function foldIfLong(node) {
const body = node.querySelector('.vv-ai-body');
if (!body || body.scrollHeight <= FOLD_PX) return;
body.classList.add('vv-ai-fold');
body.insertAdjacentHTML('afterend',
`<button type="button" class="vv-ai-lnk vv-ai-more" data-more>Show the rest`
+ ` (${Math.round(body.scrollHeight / 20)} lines)</button>`);
}
// Answers render in full. They used to fold at 420px behind a "Show the rest" control, on the
// reasoning that a long answer buries the composer on a 15" panel — true, but the cost was
// paid on every answer to save a scroll, and the fold hid the part of a long answer that is
// usually the point. Scrolling is cheaper than a click plus a scroll, and Follow already
// handles keeping your place while an answer arrives.
function addError(msg) {
const p = $('pending'); if (p) p.remove();
chatEl().appendChild(el(`<div class="vv-ai-msg bot"><div class="vv-ai-role">Varaverk</div>`
@@ -1076,14 +1071,6 @@ vv_ai_profiles_script();
}
return;
}
const more = e.target.closest('[data-more]');
if (more) {
const body = more.previousElementSibling;
if (body) body.classList.remove('vv-ai-fold');
more.remove();
return;
}
const stog = e.target.closest('[data-src-toggle]');
if (stog) {
const box = stog.parentElement;
@@ -1376,6 +1363,15 @@ vv_ai_profiles_script();
profile: profile,
question: q,
history: JSON.stringify(sendable()),
// Where the caller is standing, so a question can say "this setting" and mean it. The
// backend whitelists it and passes it to the worker, which turns it into a location
// line; a page that sets no scope sends '' and the worker omits that line.
//
// Keep this here. The scope-aware dock in 185abdb sent it from the page's own ask call;
// 8eeb4c3 folded that dock into this component and the line did not come with it, so
// for the whole of that window Scheduler computed a scope, re-read it at send time, and
// the model never saw it. Nothing errored — the answer just arrived ungrounded.
scope: (typeof o.scope === 'function' ? o.scope() : (o.scope || '')),
kind: (PROFILES[profile].kind && kindEl) ? kindEl.value : '',
// A checkbox where the page offers one, otherwise whatever the page decides from the
// profile, otherwise on. The Scheduler reasons only when diagnosing: working out what a
@@ -1461,7 +1457,7 @@ vv_ai_profiles_script();
fetch(API, { method: 'POST', headers: POST_HEAD,
body: new URLSearchParams({ action: 'stop', token: t }) })
.then(r => r.json())
.catch(() => {});
.catch(e => vvFetchErr('stop generation', e));
// Nothing is rendered from the response. The poll already owns turning a terminal state into
// a message, and having two paths do it is how a turn ends up in the transcript twice.
}
@@ -1486,7 +1482,7 @@ vv_ai_profiles_script();
noteLine('Stopped before anything was written.');
}
fetch(API, { method: 'POST', headers: POST_HEAD,
body: new URLSearchParams({ action: 'clear', token }) }).catch(() => {});
body: new URLSearchParams({ action: 'clear', token }) }).catch(e => vvFetchErr('clear job', e));
finish();
save();
return;
@@ -1496,7 +1492,7 @@ vv_ai_profiles_script();
addAnswer(j);
messages.push({ role: 'assistant', content: j.answer });
fetch(API, { method: 'POST', headers: POST_HEAD,
body: new URLSearchParams({ action: 'clear', token }) }).catch(() => {});
body: new URLSearchParams({ action: 'clear', token }) }).catch(e => vvFetchErr('clear job', e));
finish();
save();
// The answer is passed so a page can react to what was said, not just that a turn
@@ -1539,8 +1535,8 @@ vv_ai_profiles_script();
messages: JSON.stringify(messages),
}) })
.then(r => r.json())
.then(d => { if (d.ok) { chatId = d.id; onChats(chatId); } })
.catch(() => {});
.then(d => { if (d.ok) { chatId = d.id; rememberChat(chatId); onChats(chatId); } })
.catch(e => vvFetchErr('save conversation', e));
}
@@ -1558,10 +1554,6 @@ vv_ai_profiles_script();
+ `<div class="vv-ai-body">${fmt(m.content)}</div>`
+ (m.offer ? offerHtml(m.offer, i) : '') + `</div>`);
c.appendChild(node);
// Folded here too, not only on arrival. A reopened thread carrying several long answers
// is exactly the case the fold exists for — it is what buries the composer on a 15"
// panel. Measured after the append, because scrollHeight is 0 on a detached node.
foldIfLong(node);
});
scroll();
syncLast();
@@ -1623,6 +1615,7 @@ vv_ai_profiles_script();
// end would send the model an empty history for a transcript full of context.
sendFrom = 0;
chatId = c.id || '';
rememberChat(chatId);
lastSources = [];
if (PROFILES[c.profile]) applyProfile(c.profile);
render();
@@ -1632,6 +1625,9 @@ vv_ai_profiles_script();
function newChat() {
messages = []; sendFrom = 0; chatId = ''; lastSources = [];
// Forget, so New means new on the next visit too. Without this the card would reopen the
// thread the operator had just deliberately stepped away from.
rememberChat('');
reset();
onChats('');
}
@@ -2144,9 +2140,17 @@ vv_ai_profiles_script();
const pool = want ? d.chats.filter(c => (c.profile || 'chat') === want) : d.chats;
if (!pool.length) return;
const last = pool.slice().sort((a, b) => (b.updated || 0) - (a.updated || 0))[0];
// This card's own last thread first, and only if it is still in the pool — deleted
// elsewhere, or moved to a profile this card does not show, both mean it is not ours to
// reopen. Newest-in-pool remains the answer for a first visit and for that case, which is
// what every card did unconditionally before.
let want_id = '';
try { want_id = localStorage.getItem(LAST_CHAT_KEY) || ''; } catch (_) {}
const mine = want_id ? pool.find(c => c.id === want_id) : null;
const last = mine || pool.slice().sort((a, b) => (b.updated || 0) - (a.updated || 0))[0];
if (last) loadChat(last.id);
}).catch(() => {});
}).catch(e => vvFetchErr('resume conversation', e));
}
const inst = {
@@ -2330,7 +2334,7 @@ vv_ai_profiles_script();
function load() {
fetch(API + '?action=chats').then(r => r.json())
.then(d => { if (d.ok) { rows = d.chats || []; render(); } })
.catch(() => {});
.catch(e => vvFetchErr('conversation list', e));
}
box.addEventListener('click', e => {
@@ -2353,7 +2357,7 @@ vv_ai_profiles_script();
// from the store, which would silently resurrect it on the next turn.
if (del.dataset.del === activeId && o.chat) o.chat.newChat();
load();
}).catch(() => {});
}).catch(e => vvFetchErr('delete conversation', e));
return;
}
const row = e.target.closest('.vv-ai-crow');
+42 -20
View File
@@ -8,31 +8,53 @@
// the prompt on its own. A proposal is filed; the operator accepts or dismisses it; accepted
// text lands in the learned memory slot, which the prompt explicitly ranks BELOW retrieval.
//
// WHY IT IS A PROPOSAL AND NOT A WRITE
// Memory is injected into every future prompt. A model that writes its own memory writes its
// own mistakes, and then reads them back as established fact — growing more confident on each
// turn while the actual source code says otherwise. The cost of a bad proposal has to be one
// dismissal, not a permanently poisoned prompt. This is the same two-gate shape the repair
// system uses, for the same reason: whether something should be remembered is intent, and a
// model cannot prove intent.
// DESIGN PRINCIPLES
// It is a proposal, never a write.
// Memory is injected into every future prompt. A model that writes its own memory writes
// its own mistakes, and then reads them back as established fact — growing more confident
// on each turn while the actual source code says otherwise. The cost of a bad proposal has
// to be one dismissal, not a permanently poisoned prompt. This is the same two-gate shape
// the repair system uses, for the same reason: whether something should be remembered is
// intent, and a model cannot prove intent.
//
// WHY DEDUP IS NOT THE MODEL'S JOB
// "Do I already know this" is a semantic comparison, and a 14B at IQ4_XS is confidently wrong
// at it often enough to matter — with every miss costing budget permanently. So dedup here is
// deterministic: normalise, then reject on exact match or containment in either direction
// against assisted memory, learned memory, and everything previously dismissed. It will let
// through a reworded duplicate; it will never silently drop something new, and that is the
// right way round for a store the operator reviews anyway.
// Dedup is deterministic, not the model's job.
// "Do I already know this" is a semantic comparison, and a 14B at IQ4_XS is confidently
// wrong at it often enough to matter — with every miss costing budget permanently. So
// dedup here is deterministic: normalise, then reject on exact match or containment in
// either direction against assisted memory, learned memory, and everything previously
// dismissed. It will let through a reworded duplicate; it will never silently drop
// something new, and that is the right way round for a store the operator reviews anyway.
//
// GATES
// AI_MEMORY_LEARN_ENABLED false — nothing is proposed, and the prompt gains nothing
// AI_MEMORY_LEARN_AUTO_ACCEPT false — accepted writes happen only when the operator says so
// The second cannot outrank the first: auto-accept with proposing off does nothing at all.
// OPERATIONAL SAFEGUARDS
// Two gates, and the second cannot outrank the first.
// AI_MEMORY_LEARN_ENABLED off means nothing is proposed at all, so auto-accept with
// proposing off does nothing. Neither defaults to on.
//
// Dismissed rows are kept, not deleted.
// They are the only thing that stops the same suggestion arriving again every night. A
// store that forgot its refusals would re-propose what the operator has already judged.
//
// Nothing here reaches the prompt directly. Accepted text lands in the learned memory slot,
// which the prompt ranks BELOW retrieval — so even an accepted mistake cannot outrank the
// source code it contradicts.
//
// EXPORTS
// Gates vv_ai_mem_learn_enabled(), vv_ai_mem_learn_auto()
// Dedup vv_ai_mem_norm(), vv_ai_mem_known(), vv_ai_mem_is_dup(), vv_ai_mem_is_ui_fact()
// Store vv_ai_mem_dir(), vv_ai_mem_propose(), vv_ai_mem_list(), vv_ai_mem_remove(),
// vv_ai_mem_append(), vv_ai_mem_write_row()
// Operator vv_ai_mem_action()
// — accept or dismiss one proposal. The only entry that changes what a future
// prompt will contain.
//
// CONFIGURATION
// master.conf
// AI_MEMORY_LEARN_ENABLED propose at all. Default false.
// AI_MEMORY_LEARN_AUTO_ACCEPT write accepted text without asking. Default false.
//
// STORE
// data/ai/mem_proposals/<id>.json — one file per proposal, mirroring the findings store.
// States: open | accepted | dismissed. Dismissed rows are KEPT, because they are what stops
// the same suggestion arriving again every night.
// States: open | accepted | dismissed.
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/ai.php';
+41 -36
View File
@@ -5,47 +5,51 @@
// shared chat include and every page that renders it — reads it from here instead of
// restating it.
//
// WHY THIS EXISTS
// A profile used to be defined in five places: history depth in api/ai.php, capabilities in
// include/ai.php, label/hint/depth again in the chat's JavaScript, a prompt branch in the
// worker, and a label map in pages/scheduler.php. They had already drifted — the JavaScript
// knew three profiles where PHP knew four, so the shared chat could not offer troubleshoot at
// all and the Scheduler dock hand-rolled its own labels to compensate. The include carried a
// comment telling the next person not to let the two tables diverge, which is a comment doing
// a data structure's job.
// DESIGN PRINCIPLES
// One definition, not five.
// A profile used to be defined in five places: history depth in api/ai.php, capabilities
// in include/ai.php, label/hint/depth again in the chat's JavaScript, a prompt branch in
// the worker, and a label map in pages/scheduler.php. They had already drifted — the
// JavaScript knew three profiles where PHP knew four, so the shared chat could not offer
// troubleshoot at all and the Scheduler dock hand-rolled its own labels to compensate. The
// include carried a comment telling the next person not to let the two tables diverge,
// which is a comment doing a data structure's job.
//
// WHAT LIVES HERE, AND WHAT DELIBERATELY DOES NOT
// Here: anything more than one file needs to agree on — the set of profiles, their labels and
// hints, history depth, capabilities, and whether a profile is offered as a button.
// What lives here is whatever more than one file must agree on.
// The set of profiles, their labels and hints, history depth, capabilities, and whether a
// profile is offered as a button.
//
// Not here: the system prompts. They are long, delicate, and have exactly one consumer, so
// moving them would be churn against the most sensitive text in the subsystem for no reduction
// in duplication. Tools/ai_chat_worker.php still owns them; it just keys off ids validated
// here rather than an if-chain that invents its own vocabulary.
// Not the system prompts. They are long, delicate, and have exactly one consumer, so
// moving them would be churn against the most sensitive text in the subsystem for no
// reduction in duplication. Tools/ai_chat_worker.php still owns them; it just keys off ids
// validated here rather than an if-chain that invents its own vocabulary.
//
// CAPABILITIES ARE PER PROFILE, NOT PER CAPABILITY
// The old table was inverted — capability => [profiles] — which reads well when adding a
// capability and badly when answering the question actually asked at runtime, which is always
// "what can this profile do". Same content, turned the right way round.
// Capabilities are stored per profile, not per capability.
// The old table was inverted — capability => [profiles] — which reads well when adding a
// capability and badly when answering the question actually asked at runtime, which is
// always "what can this profile do". Same content, turned the right way round.
//
// A profile is a contract plus a set of inputs, and the inputs are the half that has to be
// enforced rather than requested. The caps list is that half.
// A profile is a contract plus a set of inputs, and the inputs are the half that has to be
// enforced rather than requested. The caps list is that half.
//
// It exists because the alternative already failed. The same permissions used to live as a
// dozen `$profile === 'varaverk' || $profile === 'troubleshoot'` conditions spread across the
// worker, and answering "may chat ever be shown a log?" meant reading all of them. It could —
// a gate added for run-outcome questions granted it by omission, and the chat profile, whose
// entire value is that it has NOT been shown this installation, was one phrasing away from
// being handed a health sweep and 120 lines of log. Nothing about that was visible at the
// point of the mistake. Here it would have been one missing word on one line.
// OPERATIONAL SAFEGUARDS
// The capability list is the grant, and it is enforced in one place.
// The same permissions used to live as a dozen `$profile === 'varaverk' || $profile ===
// 'troubleshoot'` conditions spread across the worker, and answering "may chat ever be
// shown a log?" meant reading all of them. It could — a gate added for run-outcome
// questions granted it by omission, and the chat profile, whose entire value is that it
// has NOT been shown this installation, was one phrasing away from being handed a health
// sweep and 120 lines of log. Nothing about that was visible at the point of the mistake.
// Here it would have been one missing word on one line.
//
// A capability is permission, not need. varaverk holds 'health' but only attaches it when the
// question looks diagnostic; troubleshoot attaches it always. The gates decide whether an
// input is warranted, this decides whether it is allowed, and a gate can never widen the grant.
// A capability is permission, not need.
// varaverk holds 'health' but only attaches it when the question looks diagnostic;
// troubleshoot attaches it always. The gates decide whether an input is warranted, this
// decides whether it is allowed, and a gate can never widen the grant.
//
// chat holding an empty capability list is a guarantee, not an oversight. Anything added to it
// stops being general chat and becomes an assistant that sometimes lies about this
// installation.
// chat holding an empty capability list is a guarantee, not an oversight.
// Anything added to it stops being general chat and becomes an assistant that sometimes
// lies about this installation.
//
// EXPORTS
// vv_ai_profiles() the whole table
@@ -116,7 +120,7 @@ const VV_AI_PROFILES_DEF = [
// system_state is read-only and shared with repair. Both need to know a container is down
// or a pool is full to explain anything about this machine rather than about Unraid in
// general; neither gets a way to act on it, and only repair can change a setting.
'caps' => ['retrieve', 'health', 'system_state', 'run_evidence', 'scoped_log',
'caps' => ['retrieve', 'health', 'system_state', 'fallback_state', 'run_evidence', 'scoped_log',
'incidents', 'conf_lookup', 'file_bugs'],
],
// The only profile that may change a setting, and the only one not offered as a button.
@@ -136,7 +140,7 @@ const VV_AI_PROFILES_DEF = [
'hint' => 'Works through a finding with you, and can apply a fix you approve.',
'turns' => 3,
'ui' => false,
'caps' => ['retrieve', 'health', 'system_state', 'run_evidence', 'scoped_log', 'incidents',
'caps' => ['retrieve', 'health', 'system_state', 'fallback_state', 'run_evidence', 'scoped_log', 'incidents',
'conf_lookup', 'conf_write', 'probe', 'file_findings', 'phrasebook', 'past_fixes'],
],
];
@@ -148,6 +152,7 @@ const VV_AI_CAP_MEANING = [
'kind_filter' => 'the retrieval kind filter the page exposes',
'health' => 'live health sweep measured at question time — the AI subsystem only',
'system_state' => 'read-only view of the machine: hardware, containers, pools, array, UPS',
'fallback_state' => 'whether a failover would actually work: state, tiers, and whether the partner really has the covered containers',
'run_evidence' => 'run record and log tail for a script named in the question',
'scoped_log' => 'log tail for whatever the operator currently has open',
'incidents' => 'operator-written history of what previously went wrong with this thing',
+9
View File
@@ -64,6 +64,15 @@
// vv_ai_route() action → route for this node
// vv_ai_rpc() forward one action to the owner, return its response body
//
// CONFIGURATION
// host*.conf
// <HOST>_SSH_KEY private key this node authenticates to the owner with. Absent or not a
// file means no remote route exists, and vv_ai_route() says so rather than
// attempting a hop that cannot succeed.
// <OWNER> the owner's hostname, looked up by the id vv_ai_owner_host() returns —
// resolved to an address through vv_resolve_tailscale_ip(), never used as
// a hostname directly, because MagicDNS does not resolve across the tailnets.
//
// DEPENDS ON
// include/config.php vv_ai_owner_host(), vv_ai_is_owner(), vv_resolve_tailscale_ip()
// Tools/ai_rpc.php the far side — reached at the WebGUI symlink path
+25
View File
@@ -43,6 +43,31 @@
//
// Redacted before it is logged, like every other question in this subsystem.
//
// EXPORTS
// Settings vv_ai_web_enabled(), vv_ai_web_provider(), vv_ai_web_results_max(),
// vv_ai_web_timeout(), vv_ai_web_searx_url(), vv_ai_web_degoog_url(),
// vv_ai_web_api_key(), vv_ai_web_ready()
// — every one reads conf; none of them reaches the network.
// Providers vv_ai_web_degoog(), vv_ai_web_searxng(), vv_ai_web_brave(), vv_ai_web_tavily()
// — one per backend, each returning the same normalised shape.
// Transport vv_ai_web_http(), vv_ai_web_normalise()
// Entry vv_ai_web_search(), vv_ai_web_context()
// — what the rest of the subsystem calls. Everything above is reachable but only
// these two are meant to be: search() picks the provider, context() formats for
// the prompt.
//
// CONFIGURATION
// master.conf
// AI_WEB_SEARCH_ENABLED master switch; anything but "true" disables the whole subsystem
// AI_WEB_SEARCH_PROVIDER which backend vv_ai_web_search() dispatches to
// AI_WEB_SEARCH_RESULTS result cap handed to the provider
// AI_WEB_SEARCH_TIMEOUT per-request timeout, in seconds
//
// host*.conf — per host, because the endpoint and the key are this machine's
// <HOST>_SEARXNG_URL self-hosted SearXNG base URL
// <HOST>_DEGOOG_URL self-hosted degoogle/Whoogle base URL
// <HOST>_WEB_SEARCH_API_KEY Brave or Tavily key, whichever provider is selected
//
// DEPENDS ON
// include/ai.php vv_ai_redact(), vv_conf_vars()
// ═══════════════════════════════════════════════════════════════════════════════════════════════
+6
View File
@@ -460,6 +460,12 @@ function vv_arrs_all(): array {
$cacheFile = VV_CACHE_DIR . '/arrs_remote_' . $h . '.json';
if (file_exists($cacheFile)) {
$node = json_decode(file_get_contents($cacheFile), true) ?: [];
// The cache file was written by vv_arrs_local_node() ON THAT HOST, where the node is by
// definition local — so it arrives claiming 'local' => true and, re-emitted verbatim,
// made every node on every host report as this one. Ownership is decided here, by who
// is reading the file, not by who wrote it.
$node['local'] = false;
$node['host'] = $h;
$node['cached'] = true;
$node['cache_age'] = time() - (int)filemtime($cacheFile);
$result[] = $node;
+4 -1
View File
@@ -544,7 +544,10 @@ function vv_network_stats(): array {
}
// Tailscale IP — use `tailscale ip` CLI (interface name varies: tailscale0, tailscale1, etc.)
$tsIp = trim(shell_exec('tailscale ip -4 2>/dev/null | head -1') ?: '');
// Bounded like every other tailscale lookup in the tree. vv_network_stats() is served by
// api/monitor_fast.php, which the Monitor page polls once a second; an unbounded call here
// makes a slow tailscale silently drop that live cadence to whatever the CLI takes.
$tsIp = trim(shell_exec('timeout 2 tailscale ip -4 2>/dev/null | head -1') ?: '');
return [
'available' => true,
+1 -1
View File
@@ -249,7 +249,7 @@ const VV_SCRIPT_CONF_SECTIONS = [
'Arrs_Stack/arrs_failed_stalled_recovery.sh' => ['Arr Failed/Stalled Recovery', 'Arr Recovery Toggles'],
'Media/play_state_sync.sh' => ['Play State Sync', 'Play State Sync — Handback'],
'Orchestrators/sunday_morning_coffee_report.sh' => ['Sunday Morning Coffee Report'],
'Fallback/fallback_test.sh' => ['Failover Test'],
'Fallback/fallback_test.sh' => ['Fallback Test'],
'Rsync/rsync.sh' => ['Rsync Enable/Disable', 'Rsync Defaults',
'Rsync Profile System', 'Rsync Merge Auto-Promote'],
'System_Essentials/server_reboot.sh' => ['Reboot', 'Emby'],
+107 -14
View File
@@ -160,6 +160,56 @@ define('LOG_DIR', '/var/log/varaverk');
define('CUSTOM_SCRIPTS_DIR', $_vv_cfg['CUSTOM_SCRIPTS_DIR'] ?? '/boot/config/plugins/user.scripts/Varaverk/Scripts');
unset($_vv_cfg);
// ── Recording a failure this layer chose to survive ───────────────────────────────────────────
// The bash half writes a log for every run. The PHP half had nowhere to write at all — no
// error_log() call in 107 files — so an endpoint that caught an exception and carried on left no
// trace anywhere. Five directory walks do exactly that, and an unreadable directory renders as a
// legitimately empty result: the scheduler's script library shows "no scripts" whether you have
// none or the tree could not be read.
//
// This is the smallest thing that fixes it. It does not change any response contract; it records
// what was swallowed so the operator can find out why a page went empty.
//
// It lives in config.php because that is the one file every layer reaches — api/, include/ and
// pages/ all load it, several of them without ever loading common.php.
define('VV_PHP_LOG', LOG_DIR . '/php.log');
define('VV_PHP_LOG_MAX', 262144); // 256KB. /var/log is tmpfs on Unraid — this is RAM, so a
// repeating fault must not be able to grow without bound.
/**
* Record a failure that was caught and survived. Never throws, never echoes, never affects the
* response a logger that can break a request is worse than no logger.
*/
function vv_log_error(string $where, string $message): void {
try {
if (!is_dir(LOG_DIR)) return;
// Timestamps are local, matching every bash log — see "The clock this layer runs on".
$line = date('Y-m-d H:i:s') . ' ' . $where . ' '
. preg_replace('/\s+/', ' ', trim($message)) . "\n";
$size = @filesize(VV_PHP_LOG);
if ($size !== false && $size > VV_PHP_LOG_MAX) {
// Drop the older half rather than the whole file, so a fault that repeats every poll
// still leaves recent context instead of a log that empties itself at random.
$keep = @file_get_contents(VV_PHP_LOG, false, null, (int)(VV_PHP_LOG_MAX / 2));
if ($keep !== false) {
// Resume on a line boundary — slicing at a byte offset lands mid-line, and a
// half-written first entry is worse than one fewer entry.
$nl = strpos($keep, "\n");
$keep = $nl === false ? '' : substr($keep, $nl + 1);
@file_put_contents(VV_PHP_LOG, $keep, LOCK_EX);
}
}
$isNew = !file_exists(VV_PHP_LOG);
@file_put_contents(VV_PHP_LOG, $line, FILE_APPEND | LOCK_EX);
// Do not inherit the ambient umask. php-fpm's is loose enough to create this 0666, and a
// world-writable file under /var/log is the kind of small thing that is never noticed.
if ($isNew) @chmod(VV_PHP_LOG, 0644);
} catch (Throwable $_) {
return; // Deliberately silent: this is the last place an error should propagate from.
}
}
define('VV_SETUP_STATE_FILE', STATE_DIR . '/varaverk_setup.db');
// ── The two install layouts ───────────────────────────────────────────────────
@@ -416,28 +466,47 @@ function vv_get_hostname(bool $flush = false): string {
return $name;
}
// Mirror of common.sh resolve_tailscale_ip(): tries `tailscale ip -4` first (Tailscale manages
// the mapping so this survives IP changes), falls back to parsing `tailscale status` text.
// Mirror of common.sh resolve_tailscale_ip(): resolves a hostname to a Tailscale IPv4.
//
// `tailscale status` is asked first, and that ordering is the whole point. It used to try
// `tailscale ip -4` first, on the reasoning that Tailscale owns the mapping — true, but that call
// resolves through MagicDNS, and MagicDNS does not work here: these hosts sit on separate tailnets
// shared into each other, so the name misses and the call falls through to a system DNS lookup
// that times out. Measured at **5.04 seconds, every call**, against 0.010s for the status parse.
//
// Nine files resolve peers this way — the AI RPC, node_chat, the arr collector, the media mesh
// view — so that was five seconds added to every mesh operation on both hosts, quietly, for as
// long as the mesh has existed. Nothing looked broken; everything was just slow.
//
// Order is now: exact match, unambiguous prefix match, then the DNS path bounded to two seconds as
// a last resort. Exactness is not given up to get the speed — `tailscale status` carries the same
// mapping `tailscale ip` would return, and an exact name match against it is exactly as precise.
//
// Memoised per request: a page that resolves the same partner four times paid four lookups.
function vv_resolve_tailscale_ip(string $hostname): string {
$h = strtolower($hostname);
$ip = trim(shell_exec('tailscale ip -4 ' . escapeshellarg($h) . ' 2>/dev/null') ?: '');
if ($ip) return $ip;
static $cache = [];
$h = strtolower($hostname);
if (isset($cache[$h])) return $cache[$h];
// Fallback: unambiguous prefix match against tailscale status (either direction) — handles
// Unraid's 15-char NetBIOS hostname truncation vs. a longer name recorded in master.conf.
// Only accept the match when exactly one peer could qualify; never guess between multiple
// candidates that happen to share a prefix (e.g. server1/server10).
$out = shell_exec('tailscale status 2>/dev/null') ?: '';
$out = shell_exec('tailscale status 2>/dev/null') ?: '';
$matches = [];
foreach (explode("\n", $out) as $line) {
$cols = preg_split('/\s+/', trim($line));
if (!isset($cols[1])) continue;
$peerHost = strtolower(explode('.', $cols[1])[0]);
if (str_starts_with($peerHost, $h) || str_starts_with($h, $peerHost)) {
$matches[] = $cols[0];
}
if ($peerHost === $h) return $cache[$h] = $cols[0];
// Unambiguous prefix match, either direction — handles Unraid's 15-char NetBIOS hostname
// truncation against a longer name recorded in master.conf. Only accepted when exactly one
// peer could qualify; never a guess between candidates sharing a prefix (server1/server10).
if (str_starts_with($peerHost, $h) || str_starts_with($h, $peerHost)) $matches[] = $cols[0];
}
return count($matches) === 1 ? $matches[0] : '';
if (count($matches) === 1) return $cache[$h] = $matches[0];
// Only reached when status cannot decide. Bounded, because this is the path that blocks on DNS
// when MagicDNS is unavailable — which is the normal case on this mesh.
return $cache[$h] = trim(shell_exec('timeout 2 tailscale ip -4 ' . escapeshellarg($h) . ' 2>/dev/null') ?: '');
}
// Cached alongside the others: this reads master.conf in full and is called by vv_conf_vars() on
@@ -624,6 +693,30 @@ function vv_read_conf_raw(string $filename): string {
return file_exists($path) ? file_get_contents($path) : '';
}
// Read one HOST's conf, wherever that host's conf actually lives for this reader.
//
// Sparse checkout means a host only ever checks out its OWN host*.conf, so on HOST1 the file
// Configurations/host2.conf does not exist and never will. vv_read_conf_raw() looks only in
// CONF_DIR, so every caller that passed it a partner's slot silently got an empty string — and
// an empty conf parses into empty arrays, which render as "none". The Watchdog tab showed the
// partner's memory limits, required list, pause list, stop list and scan-ignore list as five
// empty sections, and the Fallback tab reported the partner's tiers as unconfigured, on a mesh
// where conf_sync had been delivering all of it into the RAM cache the whole time.
//
// Partner confs are pulled into VV_CONF_RAM_CACHE_DIR by System_Essentials/conf_sync.sh. Disk
// first for this host, RAM cache for anyone else; the disk copy is never preferred for a
// partner, because a stale Configurations/host2.conf left behind by an old install is exactly
// the file that must not win. An absent cache returns empty, same as before.
function vv_read_host_conf_raw(string $slot): string {
$slot = strtolower($slot);
if ($slot === vv_detect_host()) return vv_read_conf_raw($slot . '.conf');
$cached = rtrim(VV_CONF_RAM_CACHE_DIR, '/') . '/' . $slot . '.conf';
if (is_readable($cached)) return (string)@file_get_contents($cached);
return '';
}
function vv_write_conf_raw(string $filename, string $content): bool {
$path = CONF_DIR . '/' . $filename;
$tmp = $path . '.vv.tmp';
+1 -1
View File
@@ -121,7 +121,7 @@ function vv_ai_findings_strip(string $prefix, array $kinds, string $title): void
</div>`;
}).join('');
})
.catch(() => {});
.catch(e => vvFetchErr('repair findings', e));
}
// Delegated, and it reloads rather than mutating the row: the store is the truth and a card that
+24 -1
View File
@@ -19,7 +19,8 @@
//
// EXPORTS
// vv_container_webui() WebUI URL for one container, or empty
// vv_get_docker_folders() folder grouping in the legacy shape
// vv_container_tier_map() container name (lowercased) => fallback tier 1-4
// vv_get_docker_folders() folder grouping in the legacy shape, each container carrying 'tier'
//
// CONFIGURATION
// Inherits everything from docker.php — see that file's CONFIGURATION block.
@@ -54,11 +55,31 @@ function vv_container_webui(string $name, array $portMap): string {
return $url;
}
// Fallback tier for each container, from FALLBACK_<ME>_TIER1..4 in THIS host's own conf —
// that list is what the partner starts for us, so a lower tier means it comes back sooner.
// Keyed lowercase because conf spelling and docker's spelling of a name need not match.
function vv_container_tier_map(): array {
$me = strtoupper(vv_detect_host());
if ($me === 'UNKNOWN') return [];
$raw = vv_read_host_conf_raw(strtolower($me));
if ($raw === '') return [];
$map = [];
for ($t = 1; $t <= 4; $t++) {
foreach (vv_parse_conf_list($raw, "FALLBACK_{$me}_TIER{$t}") as $name) {
$key = strtolower(trim($name));
if ($key !== '' && !isset($map[$key])) $map[$key] = $t; // lowest tier wins a duplicate
}
}
return $map;
}
function vv_get_docker_folders(): array {
// Varaverk's own docker_folders.json is the primary store (see include/docker.php) —
// reading folder.view3's mirror directly here left this widget empty on any host
// without that optional third-party plugin installed.
$folderData = vv_dk_read_json();
$tierMap = vv_container_tier_map();
// One docker ps call: names, status, port mappings
$raw = shell_exec("docker ps -a --format '{{.Names}}\t{{.Status}}\t{{.Ports}}' 2>/dev/null") ?? '';
@@ -92,6 +113,7 @@ function vv_get_docker_folders(): array {
'running' => $running,
'status' => $status,
'webui' => vv_container_webui($cname, $portMap),
'tier' => $tierMap[strtolower($cname)] ?? null,
];
}
usort($containers, fn($a, $b) => $b['running'] <=> $a['running'] ?: strcmp($a['name'], $b['name']));
@@ -114,6 +136,7 @@ function vv_get_docker_folders(): array {
'running' => $running,
'status' => $status,
'webui' => vv_container_webui($cname, $portMap),
'tier' => $tierMap[strtolower($cname)] ?? null,
];
}
usort($ungrouped, fn($a, $b) => strcmp($a['name'], $b['name']));
+169 -3
View File
@@ -73,6 +73,8 @@ function vv_fb_parse_state(string $text): array {
'tier4_started' => false,
'partnership_suspended' => false,
'partner_lost_at' => 0,
// Set only when the verdict came from the daemon rather than the file — see vv_fb_all().
'inferred' => false,
];
foreach (explode("\n", $text) as $line) {
$line = trim($line);
@@ -115,6 +117,87 @@ function vv_fb_remote_running(string $ip, string $sshKey): array {
return array_values(array_filter(explode("\n", trim($out))));
}
// ── Daemon / process state ────────────────────────────────────────────────────
//
// The page had no way to say whether fallback.sh was running at all, which is the first thing
// anyone looking at this tab wants to know — every state below is written BY that daemon, so a
// stale NORMAL from a process that died days ago read exactly like a healthy one.
//
// Mode matters as much as liveness. A --dry-run instance takes the same `fallback` lock as the
// real daemon, so the lock alone cannot tell them apart; the cmdline can, and the per-PID
// dry-run state copy is a second confirmation.
function vv_fb_proc(string $lockName): array {
$lockFile = '/tmp/unraid_locks/' . $lockName . '.lock';
$out = ['running' => false, 'pid' => null, 'mode' => null, 'since' => null, 'stale_lock' => false];
if (!is_file($lockFile)) return $out;
$pid = (int)strtok((string)@file_get_contents($lockFile), ':');
// A lock whose PID is gone is not "running" — it is residue from a SIGKILL or a power cut,
// and saying so is the difference between "stop it" and "clear it".
if ($pid <= 0 || !is_dir("/proc/$pid")) {
$out['stale_lock'] = true;
$out['pid'] = $pid ?: null;
return $out;
}
$cmd = (string)@file_get_contents("/proc/$pid/cmdline");
$args = explode("\0", $cmd);
$out['running'] = true;
$out['pid'] = $pid;
$out['mode'] = in_array('--dry-run', $args, true) || in_array('-n', $args, true) ? 'dry-run' : 'live';
$st = @stat("/proc/$pid");
if ($st) $out['since'] = (int)$st['mtime'];
return $out;
}
// Per-host daemon state. Local reads /proc directly; a partner is asked over the same SSH the
// rest of this file already uses, in one call rather than three.
function vv_fb_remote_proc(string $ip, string $sshKey): array {
$cmd = 'for L in fallback fallback_test; do F=/tmp/unraid_locks/$L.lock; '
. 'if [ -f "$F" ]; then P=$(cut -d: -f1 "$F"); '
. 'if [ -d "/proc/$P" ]; then M=live; tr "\\0" " " < /proc/$P/cmdline | grep -q -- "--dry-run" && M=dry-run; '
. 'echo "$L:running:$P:$M"; else echo "$L:stale:$P:"; fi; else echo "$L:none::"; fi; done';
$out = vv_pt_ssh($ip, $sshKey, $cmd);
$res = ['fallback' => ['running' => false, 'pid' => null, 'mode' => null, 'stale_lock' => false],
'fallback_test' => ['running' => false, 'pid' => null, 'mode' => null, 'stale_lock' => false]];
foreach (explode("\n", trim((string)$out)) as $line) {
$p = explode(':', trim($line));
if (count($p) < 4 || !isset($res[$p[0]])) continue;
if ($p[1] === 'running') {
$res[$p[0]] = ['running' => true, 'pid' => (int)$p[2], 'mode' => $p[3] ?: 'live', 'stale_lock' => false];
} elseif ($p[1] === 'stale') {
$res[$p[0]]['stale_lock'] = true;
$res[$p[0]]['pid'] = (int)$p[2] ?: null;
}
}
return $res;
}
// A running dry run keeps its own state file — VV_CACHE_ROOT/fallback_state.dryrun.<pid> — and
// refreshes it every check interval, exactly as the live daemon would. Reading the LIVE file
// while a preview is running is how a healthy dry run came to render as UNKNOWN on a host that
// has simply never run fallback for real: the answer existed, in a file next to the one being
// read. The preview is shown as the preview, never merged into the live state.
function vv_fb_dryrun_state(int $pid): array {
$p = (VV_CACHE_ROOT ?: '/tmp/varaverk') . '/fallback_state.dryrun.' . $pid;
if (!is_file($p)) return ['state' => null, 'age' => null];
$s = vv_fb_parse_state((string)@file_get_contents($p));
$s['age'] = time() - (int)@filemtime($p);
return $s;
}
function vv_fb_remote_dryrun_state(string $ip, string $sshKey, int $pid): array {
$f = '/tmp/varaverk/fallback_state.dryrun.' . $pid;
$out = vv_pt_ssh($ip, $sshKey, "[ -f '$f' ] && { echo \"__age=\$(( \$(date +%s) - \$(stat -c %Y '$f') ))\"; cat '$f'; }");
if (trim((string)$out) === '') return ['state' => null, 'age' => null];
$age = null;
if (preg_match('/^__age=(\d+)/m', (string)$out, $m)) $age = (int)$m[1];
$s = vv_fb_parse_state((string)$out);
$s['age'] = $age;
return $s;
}
// ── Covers — what a node runs for the other when it's down ───────────────────
function vv_fb_covers(string $covering, string $remote, string $coveringRaw, string $remoteRaw): array {
@@ -156,7 +239,7 @@ function vv_fb_all(): array {
// Read all host conf raws upfront
$raws = [];
foreach (array_keys($hosts) as $slot) {
$raws[$slot] = vv_read_conf_raw($slot . '.conf');
$raws[$slot] = vv_read_host_conf_raw($slot); // partner conf lives in the RAM cache, not CONF_DIR
}
// SSH key — from local host conf
@@ -167,8 +250,10 @@ function vv_fb_all(): array {
$nodes = [];
foreach ($hosts as $slot => $hostname) {
$isMe = ($slot === $currentHost || $currentHost === 'unknown');
$tsLabel = strtolower($hostname);
$ts = $tsPeers[$tsLabel] ?? ['online' => null, 'active' => false, 'ip' => null];
// Exact-key only, until now — see vv_pt_peer_lookup(). This mesh's conf name and tailnet
// name differ by one character, so HOST2 missed every lookup and the page rendered a
// partner that was up the whole time as state=UNREACHABLE.
$ts = vv_pt_peer_lookup($tsPeers, $hostname);
$ip = $ts['ip'] ?? null;
// State
@@ -205,14 +290,85 @@ function vv_fb_all(): array {
break; // 2-node only
}
// Daemon liveness per node. Everything in $state was written by this process — without
// it a NORMAL left behind by a daemon that died days ago is indistinguishable from a
// NORMAL being refreshed every 30 seconds.
if ($isMe) {
$proc = vv_fb_proc('fallback');
$procTest = vv_fb_proc('fallback_test');
} elseif ($ip && $mySshKey && $ts['online']) {
$rp = vv_fb_remote_proc($ip, $mySshKey);
$proc = $rp['fallback'];
$procTest = $rp['fallback_test'];
} else {
$proc = ['running' => null, 'pid' => null, 'mode' => null, 'stale_lock' => false];
$procTest = ['running' => null, 'pid' => null, 'mode' => null, 'stale_lock' => false];
}
// When a preview is running, read what IT is deciding — kept beside the live file and
// refreshed on the same interval. Reported separately so the live state is never
// overwritten by a preview's opinion.
$preview = ['state' => null, 'age' => null];
if (($proc['mode'] ?? '') === 'dry-run' && !empty($proc['pid'])) {
$preview = $isMe
? vv_fb_dryrun_state((int)$proc['pid'])
: vv_fb_remote_dryrun_state($ip, $mySshKey, (int)$proc['pid']);
}
// ── A live daemon that has simply never transitioned ────────────────────────────────
// fallback.sh writes its state file ONLY on a transition; the steady NORMAL path writes
// nothing at all. So a node that has run cleanly since it was built has no file, and
// reading the file alone reports it as UNKNOWN — the same verdict given to a node whose
// daemon is dead. Those are opposite conditions and they were rendered identically.
//
// Observed on HOST2 2026-08-23: daemon live (pid 2208657, valid lock), tailscale and ssh
// both fine, no state file, card said UNKNOWN.
//
// The rule this page is built on — never claim healthy for a host you cannot verify — is
// kept: this host CAN be verified, just not from the file that was being consulted. The
// daemon holding a live lock is the evidence. reach['state_file'] still reports false,
// because there genuinely is no file; 'inferred' says where the verdict came from instead.
// Captured before the inference below rewrites it: reach[state_file] must keep answering
// "was there a file", not "do we have a verdict". Deriving it after inference made the
// card claim a state file existed on a host that has none.
$hadStateFile = ($state['state'] ?? 'UNKNOWN') !== 'UNKNOWN';
if (($state['state'] ?? 'UNKNOWN') === 'UNKNOWN' && ($proc['running'] ?? null) === true) {
$state['state'] = 'NORMAL';
$state['inferred'] = true;
}
// How fresh the state actually is. The daemon rewrites its file every check interval, so
// an age far past that interval means it is wedged even while the process still exists.
$stateAge = null;
if ($isMe) {
$sp = STATE_DIR . '/fallback_state.db';
if (is_file($sp)) $stateAge = time() - (int)@filemtime($sp);
}
// "Reachable" is three separate facts and one boolean hid which had failed.
$reach = [
'tailscale' => $isMe ? true : ($ts['online'] === true),
'ip' => $isMe ? null : $ip,
'ssh' => $isMe ? true : ($running !== [] || ($proc['running'] !== null)),
'state_file' => $hadStateFile,
];
$nodes[] = [
'slot' => $slot,
'id' => strtoupper($slot),
'hostname' => $hostname,
'is_me' => $isMe,
'ts_online' => $ts['online'],
'ts_ip' => $ip,
'state' => $state,
'state_age' => $stateAge,
'preview' => $preview,
'running' => $running,
'running_count' => count($running),
'proc' => $proc,
'proc_test' => $procTest,
'reach' => $reach,
'covers' => $covers,
];
}
@@ -222,9 +378,19 @@ function vv_fb_all(): array {
'fb_enabled' => $fbEnabled,
'partnership_enabled' => $ptEnabled,
'fb_rsync_enabled' => $rsyncEnabled,
'dry_run' => vv_fb_scalar($masterRaw, 'FALLBACK_DRY_RUN') === 'true',
'handback_req' => $handbackReq,
'check_interval' => $checkInterval,
'suspend_after' => $suspendAfter,
// Tier delays are per-host and live in this host's own conf, not master.conf — the save
// path has to name the right file per field or the write is refused as an absent key.
'my_conf' => vv_detect_host() . '.conf',
'tier_delays' => [
'tier2' => (int)(vv_fb_scalar($myRaw, strtoupper($currentHost) . '_TIER2_DELAY') ?: 240),
'tier3' => (int)(vv_fb_scalar($myRaw, strtoupper($currentHost) . '_TIER3_DELAY') ?: 720),
'tier4' => (int)(vv_fb_scalar($myRaw, strtoupper($currentHost) . '_TIER4_DELAY') ?: 1440),
],
'external_ip' => vv_fb_scalar($masterRaw, 'EXTERNAL_IP'),
'nodes' => $nodes,
];
}
+107
View File
@@ -258,6 +258,113 @@ function vv_media_sessions(): array {
];
}
// Who is watching, across the whole mesh.
//
// Fetched live rather than read from a cache, unlike vv_media_servers_mesh(). A server's version
// and CPU are true for hours; a stream is true for minutes, and a "now playing" card assembled
// from a two-hour-old file would be confidently wrong about the one thing it exists to say. The
// cost is only paid when the operator asks for the mesh view — the card polls local by default.
//
// Each partner answers about itself for the reason the arr and media-server collectors do: its
// Emby URL is http://localhost:8096, which is true there and meaningless here, and its API keys
// never leave it. Same transport as remote_arr_cache_writer.sh, at a faster cadence, so the
// connection is multiplexed and every hop is bounded.
//
// A partner that cannot be reached is reported as unreachable, never as zero streams. Those look
// identical in a total and mean opposite things.
function vv_media_sessions_mesh(): array {
$me = vv_detect_host();
$hosts = function_exists('vv_known_hosts') ? vv_known_hosts() : [$me => $me];
$nodes = [];
$sessions = [];
$names = [];
$count = 0;
foreach ($hosts as $h => $hostName) {
if ($h === $me) {
$local = vv_media_sessions();
$nodes[] = ['host' => $h, 'name' => $hostName, 'local' => true, 'reachable' => true,
'server_count' => $local['server_count'], 'stream_count' => count($local['sessions'])];
foreach ($local['sessions'] as $s) {
$s['_host'] = $h; $s['_hostName'] = $hostName; $s['_local'] = true;
$sessions[] = $s;
}
$names = array_merge($names, $local['server_names']);
$count += $local['server_count'];
continue;
}
$remote = vv_media_sessions_remote($h, $hostName);
$nodes[] = ['host' => $h, 'name' => $hostName, 'local' => false,
'reachable' => $remote !== null,
'server_count' => $remote['server_count'] ?? 0,
'stream_count' => $remote !== null ? count($remote['sessions']) : 0];
if ($remote === null) continue;
foreach ($remote['sessions'] as $s) {
$s['_host'] = $h; $s['_hostName'] = $hostName; $s['_local'] = false;
$sessions[] = $s;
}
$names = array_merge($names, $remote['server_names']);
$count += $remote['server_count'];
}
return [
'scope' => 'mesh',
'nodes' => $nodes,
'sessions' => $sessions,
'server_names' => $names,
'server_count' => $count,
];
}
// One partner's sessions, or null when it could not be asked.
//
// Null rather than an empty payload, deliberately: the caller has to be able to distinguish "that
// node has nobody watching" from "that node did not answer", and an empty array cannot carry the
// difference.
function vv_media_sessions_remote(string $host, string $hostName): ?array {
$vars = vv_conf_vars();
$sshKey = $vars[strtoupper(vv_detect_host()) . '_SSH_KEY'] ?? '';
if (!$sshKey || !is_file($sshKey)) return null;
$ip = vv_resolve_tailscale_ip($hostName);
if (!$ip) return null;
// The WebGUI symlink, which is where Unraid serves the plugin from on every node whatever its
// storage mode — the same path remote_arr_cache_writer.sh uses, and not a guess about where
// the partner installed itself.
$php = 'php -r ' . escapeshellarg(
"require_once '/usr/local/emhttp/plugins/varaverk/include/media.php';"
. " echo json_encode(vv_media_sessions());"
);
// ControlPersist because the card polls this every 12 seconds while the mesh view is open, and
// a fresh handshake per poll per partner is most of the cost. Timeouts are short: a dark
// partner must render as unreachable quickly, not hold the whole card.
$sock = rtrim(VV_CACHE_ROOT, '/') . '/ssh';
if (!is_dir($sock)) @mkdir($sock, 0700, true);
$cmd = 'timeout 8 ssh -i ' . escapeshellarg($sshKey)
. ' -o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=5'
. ' -o ControlMaster=auto -o ControlPersist=60s'
. ' -o ControlPath=' . escapeshellarg($sock . '/media-%h')
. ' root@' . escapeshellarg($ip) . ' ' . escapeshellarg($php) . ' 2>/dev/null';
$raw = shell_exec($cmd);
if (!$raw) return null;
$d = json_decode(trim($raw), true);
if (!is_array($d) || !isset($d['sessions'])) return null;
return [
'sessions' => is_array($d['sessions']) ? $d['sessions'] : [],
'server_names' => is_array($d['server_names'] ?? null) ? $d['server_names'] : [],
'server_count' => (int)($d['server_count'] ?? 0),
];
}
// ── What each media server is actually doing ──────────────────────────────────────────────────
// Two halves that only mean something together. The application knows its version, whether an
// update is waiting and who is watching; the container knows what that is costing in CPU, memory
+175 -6
View File
@@ -1,6 +1,4 @@
<?php
require_once __DIR__ . '/common.php';
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Monitor-page roll-ups that are not raw system metrics: partner reachability, fallback
@@ -55,6 +53,10 @@ require_once __DIR__ . '/common.php';
// DATA_DIR container_restart_history.db
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/common.php';
require_once __DIR__ . '/fallback.php'; // vv_fb_proc(), vv_fb_dryrun_state() — daemon liveness
require_once __DIR__ . '/partnership.php'; // vv_pt_peer_match() — tailnet name vs conf hostname
function vv_partner_state(): array {
$vars = vv_conf_vars();
$myHostId = strtoupper(vv_detect_host());
@@ -74,16 +76,38 @@ function vv_partner_state(): array {
$setupDb = vv_setup_state_read();
// HOST<n>_PHASE*_DONE always describes the MIRROR — it is the owner's record of how far it got
// provisioning the other side, so the owner's own slot has no such flag and never will. Read from
// the mirror that made the owner compute phase 0 and render "Not provisioned" on the very host
// that had just finished onboarding it. An ACTIVE partnership is the authority on whether the
// mesh is provisioned; the flags only say who did what to whom. Same rule as vv_pt_nodes().
$ptFile = STATE_DIR . '/partnership_' . vv_get_hostname() . '.db';
$ptRaw = [];
foreach ((is_readable($ptFile) ? file($ptFile) : []) ?: [] as $line) {
[$k, $v] = array_pad(explode('=', trim($line), 2), 2, '');
$ptRaw[trim($k)] = trim($v, "\"'");
}
$ptActive = ($ptRaw['state'] ?? '') === 'ACTIVE';
$ptMembers = array_filter([strtolower($ptRaw['owner'] ?? ''), strtolower($ptRaw['mirror'] ?? '')]);
$hosts = [];
foreach ($hostIds as $id) {
$hostname = $vars[$id] ?? '';
if (!$hostname) continue;
$isMe = ($id === $myHostId);
$isOwner = strcasecmp($id, $vars['PARTNERSHIP_OWNER_HOST'] ?? '') === 0;
$online = $isMe ? true : ($tsPeers[strtolower($hostname)] ?? null);
// Exact-key lookup, until now. The tailnet name and the conf hostname differ by one
// character on this mesh, so the partner card reported a host that was up as unknown.
// Same unambiguous-prefix rule as everywhere else — one candidate or none.
$peerKey = $isMe ? null : vv_pt_peer_match(array_keys($tsPeers), $hostname);
$online = $isMe ? true : ($peerKey !== null ? $tsPeers[$peerKey] : null);
$onboardPhase = $isMe ? null
: (($setupDb[$id . '_PHASE2_DONE'] ?? '') === 'true' ? 2
: (($setupDb[$id . '_PHASE1_DONE'] ?? '') === 'true' ? 1 : 0));
if ($onboardPhase !== null && $onboardPhase < 2 && $ptActive
&& in_array(strtolower($hostname), $ptMembers, true)) {
$onboardPhase = 2;
}
$hosts[] = [
'id' => $id,
'hostname' => $hostname,
@@ -103,6 +127,60 @@ function vv_partner_state(): array {
];
}
// Everything below the state file: whether anything is actually running, what it would do, and
// when it last changed its mind. The card previously showed "✓ Nominal · monitoring · 30s" from
// a state file alone — the same sentence whether a daemon was checking every 30 seconds or
// nothing had run for five days.
function vv_fb_card_extra(): array {
$me = vv_detect_host();
$vars = vv_conf_vars();
// Liveness and mode, read the same way the Fallback tab reads it.
$proc = function_exists('vv_fb_proc')
? vv_fb_proc('fallback')
: ['running' => null, 'pid' => null, 'mode' => null, 'stale_lock' => false];
// A dry run keeps its own state copy, so the live file legitimately stops changing while a
// preview runs. Reported so the card can say which file it is describing.
$preview = ['state' => null, 'age' => null];
if (($proc['mode'] ?? '') === 'dry-run' && !empty($proc['pid']) && function_exists('vv_fb_dryrun_state')) {
$preview = vv_fb_dryrun_state((int)$proc['pid']);
}
$sp = STATE_DIR . '/fallback_state.db';
$lastChg = is_file($sp) ? time() - (int)@filemtime($sp) : null;
// What this host would take on if the partner went dark, and what the partner would take on
// for us. Both are counts, not lists — the card is one column wide and the Fallback tab owns
// the detail. Tier lists always live in the COVERED host's conf.
$partner = null;
foreach (vv_known_hosts() as $slot => $hostname) {
if ($slot !== $me) { $partner = $slot; break; }
}
$weCover = $theyCover = null;
if ($partner !== null && function_exists('vv_read_host_conf_raw')) {
$pRaw = vv_read_host_conf_raw($partner);
$myRaw = vv_read_host_conf_raw($me);
$count = function (string $raw, string $id): int {
$n = 0;
for ($t = 1; $t <= 4; $t++) $n += count(vv_parse_bash_array($raw, "FALLBACK_{$id}_TIER{$t}"));
return $n;
};
$weCover = $count($pRaw, strtoupper($partner)); // partner's tiers = what we start for them
$theyCover = $count($myRaw, strtoupper($me)); // our tiers = what they start for us
}
return [
'daemon' => $proc,
'preview' => $preview,
'last_change' => $lastChg,
'partner_id' => $partner !== null ? strtoupper($partner) : null,
'we_cover' => $weCover,
'they_cover' => $theyCover,
'rsync_on_handback' => ($vars['FALLBACK_RSYNC_ENABLED'] ?? 'false') === 'true',
];
}
function vv_fallback_state(): array {
$vars = vv_conf_vars();
$enabled = ($vars['FALLBACK_ENABLED'] ?? 'false') === 'true';
@@ -112,7 +190,7 @@ function vv_fallback_state(): array {
$stateFile = STATE_DIR . '/fallback_state.db';
if (!file_exists($stateFile)) {
return ['state' => 'UNKNOWN', 'enabled' => $enabled, 'check_interval' => $interval,
return vv_fb_card_extra() + ['state' => 'UNKNOWN', 'enabled' => $enabled, 'check_interval' => $interval,
'handback_strikes' => 0, 'handback_strikes_required' => $reqStrikes,
'partnership_suspended' => false,
'partner_lost_at' => 0, 'partnership_suspend_after' => $suspendAfter];
@@ -122,9 +200,11 @@ function vv_fallback_state(): array {
[$k, $v] = array_pad(explode('=', trim($line), 2), 2, '');
$raw[trim($k)] = trim($v);
}
return [
return vv_fb_card_extra() + [
'state' => $raw['state'] ?? 'UNKNOWN',
'failover_start' => $raw['failover_start'] ?? '0',
// fallback_start, not failover_start. The state file has only ever written the former, so this
// read returned '0' every time and the Monitor card's outage duration never rendered.
'fallback_start' => $raw['fallback_start'] ?? '0',
'tier2_started' => $raw['tier2_started'] ?? 'false',
'tier3_started' => $raw['tier3_started'] ?? 'false',
'tier4_started' => $raw['tier4_started'] ?? 'false',
@@ -292,8 +372,27 @@ function vv_watchdog_summary(): array {
&& $rwLevel === 0 && $daemonStrikes === 0 && $oomCount === 0 && $reboots === 0
&& $npmStrikes === 0 && $nicState === 'up' && $sshdOk;
// "No strikes" and "never ran" are not the same fact. Every strike file read above defaults an
// absent file to zero, so a host where the watchdogs have never executed reports exactly like
// one they just swept clean — a green card standing in for no information at all. HOST2 showed
// that green for four days on a STATE_DIR holding no watchdog file whatsoever, because its
// schedule.json never had the orchestrator enabled.
//
// $healthy stays a verdict about strikes; the caller is told separately whether there was
// anything to form a verdict from.
$lastRun = 0;
foreach ([
'watchdog_orch_hb.count', 'container_watchdog_state.db', 'resource_watchdog_state.db',
'system_watchdog_state.db', 'storage_watchdog_state.db', 'network_watchdog_state.db',
] as $f) {
$p = STATE_DIR . '/' . $f;
if (is_file($p)) $lastRun = max($lastRun, (int)@filemtime($p));
}
return [
'healthy' => $healthy,
'last_run' => $lastRun ?: null,
'never_ran' => $lastRun === 0,
'ctr_strikes' => $ctrStrikes,
'rw_level' => $rwLevel,
'rw_paused' => array_values(array_filter(explode(',', $rw['rm_paused_containers'] ?? ''))),
@@ -329,6 +428,76 @@ function vv_watchdog_summary(): array {
];
}
// What Varaverk itself is occupying and doing, as opposed to what the machine is.
//
// The System card described the host — model, uptime, load — and said nothing about the thing
// whose dashboard it is. These are the figures that are Varaverk's own and that nothing else on
// the page reports.
//
// The cache size is the one worth watching. VV_CACHE_ROOT is /tmp/varaverk, and on Unraid /tmp is
// on rootfs, which is RAM — so this directory is memory, not disk, and the arr payload cache is
// most of it. master.conf says never to move these onto flash, which makes the size the thing to
// keep an eye on instead. Reported next to the rootfs percentage it consumes, because 187 MB
// means nothing without the ceiling it counts against.
//
// du rather than a recursive PHP walk: both roots are small and page-cached, measured at 3ms
// each, and this is assembled once a minute by the cache writer rather than per page load.
function vv_varaverk_state(): array {
$du = function (string $path): ?int {
if (!is_dir($path)) return null;
$out = shell_exec('du -sb ' . escapeshellarg($path) . ' 2>/dev/null');
return preg_match('/^(\d+)/', (string)$out, $m) ? (int)$m[1] : null;
};
// Locks whose process is still alive. A lock file alone does not mean a job is running —
// acquire_lock() clears one whose pid is gone, and three were sitting in LOCK_DIR from jobs
// that finished days ago. Counting files would have reported four jobs running and one
// actually was.
$running = [];
$stale = 0;
foreach ((array)@glob('/tmp/unraid_locks/*.lock') as $lock) {
$content = trim((string)@file_get_contents($lock));
if ($content === '') { $stale++; continue; }
[$pid, $name] = array_pad(explode(':', $content, 2), 2, '');
$pid = (int)$pid;
if ($pid > 1 && @posix_kill($pid, 0)) {
$running[] = $name !== '' ? $name : basename($lock, '.lock');
} else {
$stale++;
}
}
sort($running);
// Newest partner conf in the RAM cache. conf_sync fills it; an age climbing past its schedule
// means the mesh has stopped talking, which nothing else on this page would show.
$confAge = null;
foreach ((array)@glob(VV_CONF_RAM_CACHE_DIR . '/host*.conf') as $c) {
$m = @filemtime($c);
if ($m && ($confAge === null || (time() - $m) < $confAge)) $confAge = time() - $m;
}
// Storage mode as a fact about where this install actually is, not as the conf toggle's word
// for it — the toggle is what someone intended and the path is what happened.
//
// "internal", not "flash". /boot is the internal mode in Varaverk's own vocabulary and on this
// hardware it is a mirrored NVMe pool, not a USB stick; calling it flash on the dashboard
// would invite a write-wear worry that does not apply here.
$internal = str_starts_with(SCRIPTS_DIR, '/boot');
return [
'cache_root' => VV_CACHE_ROOT,
'cache_bytes' => $du(VV_CACHE_ROOT),
'data_bytes' => $du(DATA_DIR),
'scripts_dir' => SCRIPTS_DIR,
'storage' => $internal ? 'internal' : 'appdata',
'commit' => trim((string)shell_exec(
'git -C ' . escapeshellarg(SCRIPTS_DIR) . ' rev-parse --short HEAD 2>/dev/null')),
'jobs_running' => $running,
'locks_stale' => $stale,
'conf_age_sec' => $confAge,
];
}
function vv_scripts_status(): array {
$logDir = LOG_DIR;
$statFiles = array_merge(
+327
View File
@@ -0,0 +1,327 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// The Monitor board declared as data. One array below decides which cards exist, what width
// each one takes, and what order they sit in. Everything about the page's layout — the column
// ladder, the span clamps at each rung, the row-height cap, the compensation when a card is
// absent — is generated from that array by vv_mon_board_css().
//
// pages/monitor.php holds the card bodies and nothing about where they go.
//
// DESIGN PRINCIPLES
// Powers of two, all the way down.
// Column counts are 8, 4, 2, 1 and every span is 1, 2, 4 or 8. That pairing is what makes
// the board tile with no holes at every width without a single hand-placed card: 32 span
// units divide into 8 columns as 4 rows and into 4 as 8. A rung of 6 or 10 columns does
// not divide a span of 4, which is why the previous ladder needed a per-breakpoint
// override for every wide card and still left holes.
//
// Below four columns the arithmetic stops being a division. Spans clamp to the column
// count, so a span-4 card contributes 2 at two columns and 1 at one, and the board is 13
// rows and then 18 rather than the 16 and 32 a naive division predicts. It still tiles
// without holes, for a different reason: a clamped card fills its row outright.
//
// Breakpoints are arithmetic, not taste.
// A column count is viable exactly when the cards still fit:
//
// window >= N * card-floor + (N-1) * gap + wrap-padding
//
// vv_mon_rung_min() is that line. Change VV_MON_CARD_FLOOR and every breakpoint moves with
// it. No number in the generated CSS is chosen by eye.
//
// Order comes from this file, not from document order.
// Each card is emitted with a CSS `order`, which grid auto-placement honours. Moving a card
// on the board is moving a line in this array; the markup in pages/monitor.php never has to
// be cut and pasted, which is how the old board accumulated hand-placed columns.
//
// A card that disappears has to give its width to someone.
// Only one card on this board is conditional — the second GPU, hidden by JS on a one-GPU
// host. Absent, its row summed to 7 of 8 and the board carried a hole on HOST2 that nothing
// reported. 'absorbs' names the card whose width is taken over, and the JS toggles one class
// on the container to apply it.
//
// OPERATIONAL SAFEGUARDS
// Row sums are checked, not assumed. vv_mon_board_check() verifies every declared row adds to
// the top rung and that every span is a power of two; failures are handed to the page, which
// reports them in the browser rather than rendering a quietly broken board.
//
// The declaration and the markup are cross-checked at runtime. vv_mon_board_js() ships the
// declared ids to the page, which compares them against the cards actually present and flags
// either direction — a card in the markup nobody declared, or a declaration with no card.
//
// EXPORTS
// Declaration vv_mon_board(), vv_mon_board_cards(), vv_mon_board_total()
// — the one array every other function here reads. Change the layout by editing
// that declaration, never by editing what follows.
// Geometry vv_mon_rung_min(), vv_mon_rung_rows(), vv_mon_rung_rowdiv(),
// vv_mon_rung_overflows(), vv_mon_absent_class()
// — derive the column ladder and each card's span from the declaration.
// vv_mon_rung_overflows() is the one that asks whether a rung is already taller
// than one screen, which is when the row-height cap stops helping and starts
// clipping.
// Emitters vv_mon_board_css(), vv_mon_absorb_css(), vv_mon_autorows_css(),
// vv_mon_board_js()
// — vv_mon_autorows_css() is the hatch for an overflowing rung: rows sized to
// content, and the overflow clamps lifted off the cards so they can use it.
// Validation vv_mon_board_check()
// — the only one that reports rather than renders; see OPERATIONAL SAFEGUARDS.
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// The narrowest a card may be drawn. Every breakpoint is a function of this number — see
// vv_mon_rung_min(). 160 rather than the 200 the cards carried before: at 200 the 8-column rung
// needed 1704px, which put a 1720px half-screen 16px inside the margin and dropped a 1327px
// tablet to 4 columns. 160 moves that rung to 1384 and gives both real headroom.
const VV_MON_CARD_FLOOR = 160;
// Below three columns the constraint stops being the card and starts being the content: a label
// and its value need about 240px before they wrap into nonsense. A phone in portrait is 393px, so
// this is what keeps it at one column instead of two 180px cards.
const VV_MON_PHONE_FLOOR = 240;
const VV_MON_GAP = 12; // grid gap, must match the gap in #vv-monitor
const VV_MON_PAD = 20; // horizontal padding the Unraid page wrapper takes off the viewport
// The column ladder, widest first.
const VV_MON_RUNGS = [8, 4, 2, 1];
// Cap on how many rows may share one screen height — and the line between two behaviours, not one
// rule. A rung with no more rows than this fits a screen: the row-height cap divides the viewport
// by the rung's own row count, or by this when the rung has more, so the board fills the screen
// and no card grows past its share. A rung with MORE rows than this cannot fit however its rows
// are sized, so the cap is dropped there and rows size to content instead — see
// vv_mon_rung_overflows(). Capping a board that scrolls regardless buys nothing and clips every
// card to pay for it. This board is 4 rows at eight columns and 8 at four, so both cases are live.
const VV_MON_ROWS_PER_SCREEN = 4;
// ── The board ─────────────────────────────────────────────────────────────────────────────────
// Grouped by the row each group forms at the top rung. The grouping is documentation and a
// checkable invariant — every group must sum to 8 — not placement: cards are auto-placed in the
// flattened order below, which is what lets a narrower rung re-cut them into different rows.
function vv_mon_board(): array {
return [
// The host itself.
'Host' => [
['id' => 'vv-system', 'span' => 1],
['id' => 'vv-ups-card', 'span' => 1],
['id' => 'vv-cpu', 'span' => 2],
['id' => 'vv-memory', 'span' => 2],
['id' => 'vv-network', 'span' => 2],
],
// What is running on it, and who is covering it.
'Workload' => [
['id' => 'vv-scripts-card', 'span' => 1],
['id' => 'vv-fallback', 'span' => 1],
['id' => 'vv-partner', 'span' => 2],
['id' => 'vv-docker-folders', 'span' => 4],
],
// The media path, from the sync that feeds it to the sessions coming out of it.
'Media' => [
['id' => 'vv-rsync-card', 'span' => 1],
['id' => 'vv-gpu-card', 'span' => 1],
['id' => 'vv-gpu1-card', 'span' => 1],
// Takes the second GPU's column on a one-GPU host — see DESIGN PRINCIPLES.
['id' => 'vv-transcode', 'span' => 1, 'absorbs' => 'vv-gpu1-card'],
['id' => 'vv-streams', 'span' => 4],
],
// Storage and the things that watch it.
'Storage' => [
['id' => 'vv-watchdog-card', 'span' => 1],
['id' => 'vv-parity-card', 'span' => 1],
['id' => 'vv-storage-card', 'span' => 2],
['id' => 'vv-array-card', 'span' => 4],
],
];
}
// The board in placement order, one flat list.
function vv_mon_board_cards(): array {
$flat = [];
foreach (vv_mon_board() as $row => $cards) {
foreach ($cards as $card) {
$card['row'] = $row;
$flat[] = $card;
}
}
return $flat;
}
function vv_mon_board_total(): int {
$total = 0;
foreach (vv_mon_board_cards() as $card) $total += $card['span'];
return $total;
}
// The narrowest window at which $cols columns still hold cards of at least the floor width.
function vv_mon_rung_min(int $cols): int {
$floor = ($cols <= 2) ? VV_MON_PHONE_FLOOR : VV_MON_CARD_FLOOR;
return $cols * $floor + ($cols - 1) * VV_MON_GAP + VV_MON_PAD;
}
// How many rows the board occupies at a given column count, and the divisor the row-height cap
// uses there. They differ once the board is taller than one screen: at 4 columns the board is 8
// rows and the page is meant to scroll, with each card still a readable quarter-screen tall.
// Counted by walking the same sparse auto-placement the browser does, not as total spans over
// columns. Those two agree only while nothing is clamped: at two columns a span-4 card occupies
// two units, not four, so the board is 13 rows there and the division says 16.
function vv_mon_rung_rows(int $cols): int {
$row = 0;
$col = 0;
foreach (vv_mon_board_cards() as $card) {
$span = min($card['span'], $cols);
if ($col + $span > $cols) { $row++; $col = 0; }
$col += $span;
if ($col >= $cols) { $row++; $col = 0; }
}
return $col === 0 ? $row : $row + 1;
}
function vv_mon_rung_rowdiv(int $cols): int {
return min(vv_mon_rung_rows($cols), VV_MON_ROWS_PER_SCREEN);
}
// Whether this rung is already taller than one screen. The row-height cap means "no card taller
// than 1/VV_MON_ROWS_PER_SCREEN of the screen", which is only worth paying for on a rung that
// fits a screen — there it is what keeps the board from scrolling. On a rung with more rows than
// the cap the page scrolls no matter what, so the cap buys nothing and only cuts each card's
// content off. At four columns this board is eight rows: an 11" tablet in landscape was being
// given 126px cards on a board two screens tall.
function vv_mon_rung_overflows(int $cols): bool {
return vv_mon_rung_rows($cols) > VV_MON_ROWS_PER_SCREEN;
}
// Rows sized by content instead of by the cap, for a rung that overflows the screen anyway.
// Mirrors the phone hatch in css/varaverk.css: the cards have to give up overflow:hidden too, or
// they keep clipping at a height nothing is constraining any more.
function vv_mon_autorows_css(): string {
return '#vv-monitor{grid-auto-rows:auto;}'
. '#vv-monitor .vv-card{overflow:visible;}'
. '#vv-monitor .vv-card>div{overflow-y:visible;min-height:auto;}';
}
// The class the page puts on #vv-monitor when a card is not rendered, so the generated absorb
// rules can fire. Derived from the id on both sides — PHP writes the rule, JS writes the class —
// so the two can never drift apart by a typo.
function vv_mon_absent_class(string $id): string {
return 'vv-absent-' . $id;
}
// Problems with the declaration itself, as human-readable lines. Empty means the board is sound.
function vv_mon_board_check(): array {
$problems = [];
$top = VV_MON_RUNGS[0];
$seen = [];
foreach (vv_mon_board() as $row => $cards) {
$sum = 0;
foreach ($cards as $card) {
$span = $card['span'];
if ($span < 1 || ($span & ($span - 1)) !== 0) {
$problems[] = "{$card['id']}: span $span is not a power of two";
}
if ($span > $top) {
$problems[] = "{$card['id']}: span $span exceeds the $top-column board";
}
if (isset($seen[$card['id']])) {
$problems[] = "{$card['id']}: declared twice";
}
$seen[$card['id']] = true;
$sum += $span;
}
if ($sum !== $top) {
$problems[] = "row '$row' sums to $sum, not $top";
}
}
foreach (vv_mon_board_cards() as $card) {
if (isset($card['absorbs']) && !isset($seen[$card['absorbs']])) {
$problems[] = "{$card['id']}: absorbs '{$card['absorbs']}', which is not on the board";
}
}
return $problems;
}
// ── Generated CSS ─────────────────────────────────────────────────────────────────────────────
// Everything positional. The static half — display:grid, the gap, the row-height tiers — lives in
// css/varaverk.css and reads --vv-cols, --vv-sp and --vv-rowdiv from here.
function vv_mon_board_css(): string {
$cards = vv_mon_board_cards();
$top = VV_MON_RUNGS[0];
$out = '';
$out .= "#vv-monitor{--vv-cols:$top;--vv-rowdiv:" . vv_mon_rung_rowdiv($top) . ";}\n";
if (vv_mon_rung_overflows($top)) $out .= vv_mon_autorows_css() . "\n";
$order = 0;
foreach ($cards as $card) {
$order++;
$out .= "#{$card['id']}{order:$order;--vv-sp:{$card['span']};}\n";
}
$out .= vv_mon_absorb_css($top);
foreach (VV_MON_RUNGS as $cols) {
if ($cols === $top) continue;
// One below the next rung up: the band this column count owns ends where that one begins.
$max = vv_mon_rung_min($cols * 2) - 1;
$out .= "@media (max-width:{$max}px){";
$out .= "#vv-monitor{--vv-cols:$cols;--vv-rowdiv:" . vv_mon_rung_rowdiv($cols) . ";}";
if ($cols === 1) {
// Every span at once, and through the child selector rather than per id: it outranks
// the per-card rules above on specificity, so nothing can leave a span behind and
// make the grid invent an implicit column.
$out .= "#vv-monitor>.vv-card{--vv-sp:1;}";
} else {
$wide = [];
foreach ($cards as $card) {
if ($card['span'] > $cols) $wide[] = '#' . $card['id'];
}
if ($wide) $out .= implode(',', $wide) . "{--vv-sp:$cols;}";
}
if (vv_mon_rung_overflows($cols)) $out .= vv_mon_autorows_css();
$out .= vv_mon_absorb_css($cols);
$out .= "}\n";
}
return $out;
}
// Absorb rules for one rung. Higher specificity than both the per-card span and the one-column
// child selector, which is why every rung has to restate them — at one column an unclamped
// absorb would be the only span left above 1.
function vv_mon_absorb_css(int $cols): string {
$out = '';
foreach (vv_mon_board_cards() as $card) {
if (!isset($card['absorbs'])) continue;
$donor = null;
foreach (vv_mon_board_cards() as $other) {
if ($other['id'] === $card['absorbs']) { $donor = $other; break; }
}
if ($donor === null) continue;
$span = min($card['span'] + $donor['span'], $cols);
$cls = vv_mon_absent_class($donor['id']);
$out .= "#vv-monitor.$cls #{$card['id']}{--vv-sp:$span;}";
}
return $out;
}
// What the page's runtime guard needs: the declared ids in order, the absent-class names for the
// conditional cards, and any problem the declaration already knows about.
function vv_mon_board_js(): string {
$ids = [];
$absent = [];
foreach (vv_mon_board_cards() as $card) {
$ids[] = $card['id'];
if (isset($card['absorbs'])) {
$absent[$card['absorbs']] = vv_mon_absent_class($card['absorbs']);
}
}
return json_encode([
'ids' => $ids,
'absent' => $absent,
'problems' => vv_mon_board_check(),
], JSON_UNESCAPED_SLASHES);
}
+88 -20
View File
@@ -52,6 +52,16 @@
// vv_nc_send() store + deliver to the channel's other members
// vv_nc_delete_local() forget one message on this machine
// vv_nc_flush_spool() retry undelivered
//
// CONFIGURATION
// host*.conf
// <HOST>_SSH_KEY private key used to reach the channel's other members. Missing or not a
// file means delivery is skipped and the message is spooled — stored here
// and retried later, never reported as failed.
// <HOST> each member's hostname, looked up by the id in the channel definition.
//
// The store's own paths are defines, not conf: VV_NC_DIR under DATA_DIR/db, its .spool
// subdirectory, and VV_NC_KEEP — the per-channel message retention, fixed at 300.
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/config.php';
@@ -413,6 +423,14 @@ function vvNcToggleFmt() {
// Style string for a message. Closed set on both ends — the server validates the same values,
// so nothing here can emit a property the store did not agree to.
// Module scope, because two functions need it: _vvNcStyle() renders with it and vvNcRender() used
// to declare its own copy. The copy was a `const` inside vvNcRender, so the reference here — in a
// different function — resolved to nothing and threw ReferenceError on every render, but only with
// the formatting toggle on, since the plain-render early return below skips the line that uses it.
// Every mesh transcript therefore failed to draw for anyone who had formatting enabled, and the
// empty catch on the fetch swallowed the reason.
const _VV_NC_FONTS = { mono: 'monospace', sans: 'system-ui,sans-serif', serif: 'Georgia,serif' };
function _vvNcStyle(m, mine) {
// Reader wins over sender. Someone else's purple italic large is their idea of emphasis, and
// on a wall-mounted dashboard it is noise — this renders every message plain here without
@@ -423,7 +441,7 @@ function _vvNcStyle(m, mine) {
const st = m.style || {};
const sizes = { sm: '11px', lg: '14px' };
let s = `color:${st.color || (mine ? '#7a9a7a' : '#9aa')};`;
if (fonts[st.font]) s += `font-family:${fonts[st.font]};`;
if (_VV_NC_FONTS[st.font]) s += `font-family:${_VV_NC_FONTS[st.font]};`;
s += `font-size:${sizes[st.size] || '12px'};`;
if (st.bold) s += 'font-weight:700;';
if (st.italic) s += 'font-style:italic;';
@@ -438,11 +456,47 @@ function _vvNcWho(id) {
return full ? (_vvNc.hosts[id] || id) : id.toUpperCase();
}
// Whatever went wrong, said out loud in the card's own status line.
//
// Every fetch here used to end in `.catch(() => {})`. A failed channel list was therefore
// indistinguishable from a slow one: nothing appeared, nothing was logged, and the next attempt
// was a backoff tick away — up to ninety seconds — so a single failed request looked exactly like
// "the chat takes a minute to load". Silence plus a long retry is how a fast failure becomes a
// slow mystery.
function _vvNcErr(where, e) {
_vvNcErrShown = true;
const st = document.getElementById('vv-nc-status');
const msg = (e && e.message) ? e.message : String(e || 'failed');
if (st) { st.textContent = where + ': ' + msg; st.style.color = '#a05a2c'; }
try { console.warn('[mesh chat] ' + where, e); } catch (_) {}
}
// Cleared on any success. Deliberately not conditional on "was it showing an error" — comparing a
// computed style back to the hex that set it is a string match on the browser's rgb() rendering,
// which is the kind of check that works until it silently does not.
let _vvNcErrShown = false;
function _vvNcOk() {
if (!_vvNcErrShown) return;
_vvNcErrShown = false;
const st = document.getElementById('vv-nc-status');
if (st) { st.textContent = ''; st.style.color = '#3a3a3a'; }
}
// A failed attempt retries in two seconds rather than waiting for the next backoff tick. Recovery
// should track how long the failure lasted, not how long the poller happens to be sleeping.
function _vvNcRetrySoon() {
if (_vvNcTimer) clearTimeout(_vvNcTimer);
_vvNcTimer = setTimeout(() => { vvNcChans(); _vvNcSchedule(); }, 2000);
}
function vvNcChans() {
fetch('/plugins/varaverk/api/node_chat.php?_=' + Date.now())
.then(r => r.json())
.then(r => {
if (!r.ok) throw new Error('HTTP ' + r.status);
return r.json();
})
.then(d => {
if (!d.ok) return;
if (!d.ok) { _vvNcErr('channels', d.error || 'endpoint returned ok:false'); return; }
_vvNcOk();
_vvNc.me = d.me; _vvNc.chans = d.channels || []; _vvNc.hosts = d.hostnames || {};
const sel = document.getElementById('vv-nc-ch');
if (!sel) return;
@@ -457,17 +511,27 @@ function vvNcChans() {
const b = document.getElementById('vv-nc-badge');
if (b) { b.hidden = !tot || _vvNcMeshOn; b.textContent = '✉ ' + tot; }
})
.catch(() => {});
.catch(e => { _vvNcErr('channels', e); _vvNcRetrySoon(); });
}
function vvNcSelect(ch) { _vvNc.ch = ch; vvNcLoad(); }
function vvNcLoad() {
if (!_vvNc.ch) return;
// No channel yet means the list has not landed — ask for it rather than returning silently and
// leaving the transcript on "loading" until a backoff tick happens to fetch it. This is the path
// taken when the mesh pane is opened before the first channel list has arrived.
if (!_vvNc.ch) { vvNcChans(); return; }
fetch('/plugins/varaverk/api/node_chat.php?channel=' + encodeURIComponent(_vvNc.ch) + '&_=' + Date.now())
.then(r => r.json())
.then(d => { if (d.ok) { _vvNc.msgs = d.messages || []; _vvNc.lastRead = d.last_read || 0; vvNcRender(); } })
.catch(() => {});
.then(r => {
if (!r.ok) throw new Error('HTTP ' + r.status);
return r.json();
})
.then(d => {
if (!d.ok) { _vvNcErr('messages', d.error || 'endpoint returned ok:false'); return; }
_vvNcOk();
_vvNc.msgs = d.messages || []; _vvNc.lastRead = d.last_read || 0; vvNcRender();
})
.catch(e => { _vvNcErr('messages', e); });
}
function vvNcRender() {
@@ -482,7 +546,6 @@ function vvNcRender() {
+ (unreadOnly ? 'Nothing new.' : 'No messages yet — say hello.') + `</div>`;
return;
}
const fonts = { mono: 'monospace', sans: 'system-ui,sans-serif', serif: 'Georgia,serif' };
box.innerHTML = msgs.map(m => {
const mine = m.from === _vvNc.me;
const fresh = (m.ts || 0) > _vvNc.lastRead && !mine;
@@ -508,7 +571,7 @@ function vvNcRender() {
method: 'POST', headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: new URLSearchParams({ csrf_token: typeof csrf_token !== 'undefined' ? csrf_token : '',
action: 'read', channel: _vvNc.ch })
}).catch(() => {});
}).catch(e => _vvNcErr('read', e));
}
}
@@ -610,7 +673,7 @@ function vvNcDel(id) {
method: 'POST', headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: new URLSearchParams({ csrf_token: typeof csrf_token !== 'undefined' ? csrf_token : '',
action: 'delete', channel: _vvNc.ch, id })
}).then(() => vvNcLoad()).catch(() => {});
}).then(() => vvNcLoad()).catch(e => _vvNcErr('delete', e));
}
@@ -671,16 +734,21 @@ function vvNcInit(prefix, meshDefault) {
}
// Restore the expanded state before the first paint, so the card does not visibly resize.
try {
if (localStorage.getItem('vvNcBig:' + prefix) === '1') {
_vvNcBig = true;
_vvNcApplyBig();
// Bring the assistant along, so both halves start the session agreeing.
if (aiBtn && !aiBtn.classList.contains('vv-ai-grow-on')) {
_vvNcSyncing = true; aiBtn.click(); _vvNcSyncing = false;
}
//
// Only the storage read is guarded. The try used to wrap _vvNcApplyBig() and aiBtn.click()
// too, which meant a TypeError in either — or in any handler aiBtn.click() reaches — was
// caught here and read as "localStorage is blocked". The layout would silently fail to
// restore, with nothing in the console and no way to tell the two causes apart.
let _wantBig = false;
try { _wantBig = localStorage.getItem('vvNcBig:' + prefix) === '1'; } catch (_) {}
if (_wantBig) {
_vvNcBig = true;
_vvNcApplyBig();
// Bring the assistant along, so both halves start the session agreeing.
if (aiBtn && !aiBtn.classList.contains('vv-ai-grow-on')) {
_vvNcSyncing = true; aiBtn.click(); _vvNcSyncing = false;
}
} catch (_) {}
}
vvNcMode(want);
vvNcChans();
_vvNcSchedule();
+69 -19
View File
@@ -187,13 +187,58 @@ function vv_pt_ts_peers(): array {
return $peers;
}
// Resolve one host's peer record out of vv_pt_ts_peers().
//
// The tailnet name and the OS hostname are not the same string, and nothing keeps them in step:
// this mesh has master.conf saying "unRAID-Jayred36" while the tailnet device is
// "unraid-jayred365". An exact-key lookup finds nothing, and every consumer that did one showed
// a live partner as dark — the Partnership cards until that was fixed inline, and the Fallback
// page for as long as it has existed, where the miss became state=UNREACHABLE.
//
// Exact key first, then a single unambiguous prefix match in either direction. One candidate or
// none — server1 must never resolve to server10 because it happens to share a prefix, and this
// is never similarity scoring. Same rule as vv_resolve_tailscale_ip(), which applies it to
// `tailscale status` text rather than to the parsed peer array.
// The matching rule itself, separated from the value it looks up, because the peer map is not
// always the same shape: include/monitor.php keys hostname => bool while everything else keys
// hostname => array. Four call sites had each written their own exact-key lookup and all four
// rendered this mesh's partner as dark. Returns the matching KEY, or null.
function vv_pt_peer_match(array $peerNames, string $hostname): ?string {
$label = strtolower($hostname);
if (in_array($label, $peerNames, true)) return $label;
$cand = [];
foreach ($peerNames as $peerName) {
if (str_starts_with($peerName, $label) || str_starts_with($label, $peerName)) $cand[] = $peerName;
}
return count($cand) === 1 ? $cand[0] : null;
}
function vv_pt_peer_lookup(array $tsPeers, string $hostname): array {
$key = vv_pt_peer_match(array_keys($tsPeers), $hostname);
return $key !== null ? $tsPeers[$key] : ['online' => null, 'active' => false, 'ip' => null];
}
// ── SSH helper — run a single command on a remote host ────────────────────────
// Multiplexed, because this page makes several of these per render — a state read per partner,
// plus whatever a card asks for — and each one was paying a full handshake to a node across a real
// internet hop. Measured at about two seconds per call against 0.09s once a master is up.
//
// The socket lives in tmpfs so a reboot cannot inherit a stale one, and the path is kept short: a
// unix socket path is capped near 108 characters and ssh appends the user and host to this.
function vv_pt_ssh(string $ip, string $sshKey, string $cmd, int $timeout = 4): string {
if (!$ip || !$sshKey || !file_exists($sshKey)) return '';
$sock = rtrim(VV_CACHE_ROOT, '/') . '/ssh';
if (!is_dir($sock)) @mkdir($sock, 0700, true);
$full = sprintf(
'ssh -i %s -o ConnectTimeout=%d -o StrictHostKeyChecking=no -o BatchMode=yes root@%s %s 2>/dev/null',
escapeshellarg($sshKey), $timeout, escapeshellarg($ip), escapeshellarg($cmd)
'ssh -i %s -o ConnectTimeout=%d -o StrictHostKeyChecking=no -o BatchMode=yes'
. ' -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=%s'
. ' root@%s %s 2>/dev/null',
escapeshellarg($sshKey), $timeout, escapeshellarg($sock . '/pt-%h'),
escapeshellarg($ip), escapeshellarg($cmd)
);
return shell_exec($full) ?: '';
}
@@ -444,18 +489,7 @@ function vv_pt_nodes(): array {
//
// Same rule as vv_resolve_tailscale_ip(), and the same refusal: one candidate or none.
// server1 must never resolve to server10 because it happens to share a prefix.
$tsLabel = strtolower($hostname);
$ts = $tsPeers[$tsLabel] ?? null;
if ($ts === null) {
$cand = [];
foreach ($tsPeers as $peerName => $peer) {
if (str_starts_with($peerName, $tsLabel) || str_starts_with($tsLabel, $peerName)) {
$cand[] = $peer;
}
}
if (count($cand) === 1) $ts = $cand[0];
}
if ($ts === null) $ts = ['online' => null, 'active' => false, 'ip' => null];
$ts = vv_pt_peer_lookup($tsPeers, $hostname);
// Fallback state
$fbState = 'UNKNOWN';
@@ -513,11 +547,27 @@ function vv_pt_nodes(): array {
$apiKeySet = false;
$apiKeyPreview = '';
if ($isMe) {
$hn = trim((string)shell_exec("hostname -s 2>/dev/null | sed 's/^[Uu][Nn][Rr][Aa][Ii][Dd]-//'")) ?: 'Varaverk';
$keyName = 'Varaverk ' . $hn;
$apiOut = shell_exec('/usr/local/sbin/unraid-api apikey --name ' . escapeshellarg($keyName) . ' --json </dev/null 2>/dev/null');
$apiData = json_decode(trim($apiOut ?? ''), true);
if (is_array($apiData) && !empty($apiData['key'])) {
// Cached for a minute, because `unraid-api apikey` is not a file read — it starts
// Unraid's Node CLI, measured at 1.98s, and it ran on every render of this page. That
// was the single largest cost in assembling it, and this page is the one that opens
// the mesh chat, so it was two seconds in front of a conversation every time.
//
// Still reads the key store rather than the conf, so a key deleted in Unraid's own UI
// is still reflected — a minute later rather than instantly, which is the trade. The
// preview is eight characters of a key that changes when someone deliberately rotates
// it; nobody is watching it to the second.
$apiData = vv_cache_read('pt_apikey', 60);
if ($apiData === null) {
$hn = trim((string)shell_exec("hostname -s 2>/dev/null | sed 's/^[Uu][Nn][Rr][Aa][Ii][Dd]-//'")) ?: 'Varaverk';
$keyName = 'Varaverk ' . $hn;
$apiOut = shell_exec('/usr/local/sbin/unraid-api apikey --name ' . escapeshellarg($keyName) . ' --json </dev/null 2>/dev/null');
$decoded = json_decode(trim($apiOut ?? ''), true);
// Cached either way. A host with no key would otherwise pay the two seconds on
// every render forever, which is the case that needs the cache most.
$apiData = is_array($decoded) ? $decoded : [];
vv_cache_write('pt_apikey', $apiData);
}
if (!empty($apiData['key'])) {
$apiKeySet = true;
$apiKeyPreview = substr($apiData['key'], 0, 8) . '...' . substr($apiData['key'], -4);
}
+1 -1
View File
@@ -468,7 +468,7 @@ function vv_script_library(): array {
if (in_array($rel, $orchIds) || isset($confMap[$rel]) || isset($cardIds[$rel]) || isset($schedule[$rel])) continue;
$library[] = ['id' => $rel, 'label' => vv_pretty_label(basename($rel, '.sh'))];
}
} catch (Exception $e) {}
} catch (Exception $e) { vv_log_error('include/scheduler.php', 'script library walk failed: ' . $e->getMessage()); }
usort($library, fn($a, $b) => strcmp($a['id'], $b['id']));
return $library;
}
+21 -6
View File
@@ -138,7 +138,15 @@ function vv_wd_bash_assoc(string $raw, string $varname): array {
// declare -A VARNAME=( ["key"]=val ["key2"]=val2 )
if (!preg_match('/^\s*declare\s+-A\s+' . preg_quote($varname, '/') . '\s*=\s*\(\s*(.*?)\s*\)/ms', $raw, $m))
return [];
preg_match_all('/\["([^"]+)"\]\s*=\s*"?([^"\s\)]*)"?/', $m[1], $pairs);
// Strip comments before matching, from # to end of line. Without this a commented-out
// template entry — `# ["Emby"]=18432`, which is how every host*.conf ships — parses as a
// live memory limit. It never showed on this host because this host's entries are real; it
// appeared the moment a PARTNER's conf became readable and its whole commented block was
// read as configuration. Cut per line, so a trailing comment on a real entry still keeps it:
// `["Emby"]=24576 # 24GB — raised 2026-07-11` must survive. vv_parse_bash_array(), which
// the list form delegates to, has always done this.
$body = preg_replace('/#.*$/m', '', $m[1]);
preg_match_all('/\["([^"]+)"\]\s*=\s*"?([^"\s\)]*)"?/', $body, $pairs);
$out = [];
foreach ($pairs[1] as $i => $k) $out[$k] = $pairs[2][$i];
return $out;
@@ -366,8 +374,12 @@ function vv_wd_remote_data(string $ip, string $sshKey, string $restartLogPath):
. 'sf="$sd/data/state"; [ -d "$sf" ] || sf="$sd/State_Files"; '
. 'db="$sd/data/db"; [ -d "$db" ] || db="$sd/data"; '
. "printf 'UPTIME:%s\nLOAD:%s\nCORES:%s\nDAEMON:%s\nOOM:%s\nBASELINECOUNT:%s\nBASELINEAGE:%s\n---MEMINFO---\n%s\n---RW---\n%s\n---DOCK---\n%s\n---SKIP---\n%s\n---SYS---\n%s\n---REBOOT---\n%s\n---RESTART---\n%s\n---STORAGE---\n%s\n---NETWORK---\n%s\n' "
. '"$(cat /proc/uptime|cut -d\" \" -f1)" '
. '"$(cat /proc/loadavg|cut -d\" \" -f1)" '
// awk, not `cut -d" "`. This is a PHP SINGLE-quoted string, which does not process \" —
// so the shell received a literal backslash-quote, cut read the quote as a FILENAME
// ("cut: '\"': No such file or directory"), and UPTIME and LOAD came back empty while every
// quote-free field beside them parsed fine. awk needs no delimiter argument at all.
. '"$(awk \'{print $1}\' /proc/uptime)" '
. '"$(awk \'{print $1}\' /proc/loadavg)" '
. '"$(nproc)" '
. '"$(docker info >/dev/null 2>&1 && echo ok || echo err)" '
. '"$(cat "$sf/system_watchdog_oom.db" 2>/dev/null||echo 0)" '
@@ -550,10 +562,13 @@ function vv_wd_all(): array {
$nodes = [];
foreach ($hosts as $slot => $hostname) {
$isMe = ($slot === $currentHost || $currentHost === 'unknown');
$tsLabel = strtolower($hostname);
$ts = $tsPeers[$tsLabel] ?? ['online' => null, 'active' => false, 'ip' => null];
// Exact-key only, until now — the third copy of that lookup in this codebase and the
// third to render a live partner as dark. This mesh's conf name and tailnet name differ
// by one character, so HOST2 missed every time and its whole card read UNREACHABLE /
// "No data" while HOST2's own page showed the same watchdogs reporting OK.
$ts = vv_pt_peer_lookup($tsPeers, $hostname);
$ip = $ts['ip'] ?? null;
$raw = vv_read_conf_raw($slot . '.conf');
$raw = vv_read_host_conf_raw($slot); // partner conf lives in the RAM cache, not CONF_DIR
$remoteApiKey = vv_wd_scalar($raw, strtoupper($slot) . '_UNRAID_API_KEY');
+4 -4
View File
@@ -618,7 +618,7 @@ vv_ai_chat_markup('vv-ai', [
function loadBanner(live) {
fetch(API + '?action=stats' + (live ? '&live=1' : '')).then(r => r.json())
.then(d => { if (d.ok) renderBanner(d.stats); })
.catch(() => {});
.catch(e => vvFetchErr('AI banner', e));
}
// ── Bug reports ─────────────────────────────────────────────────────────
@@ -628,7 +628,7 @@ vv_ai_chat_markup('vv-ai', [
function loadBugs() {
fetch(API + '?action=bugs').then(r => r.json())
.then(d => { if (d.ok) renderBugs(d.bugs || []); })
.catch(() => {});
.catch(e => vvFetchErr('AI bug list', e));
}
function renderBugs(bugs) {
@@ -750,7 +750,7 @@ vv_ai_chat_markup('vv-ai', [
fetch(API, { method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8' },
body: new URLSearchParams({ action: 'bug_close', id, open: '0' }) })
.then(() => loadBugs()).catch(() => {});
.then(() => loadBugs()).catch(e => vvFetchErr('AI bug report', e));
};
// ── Findings & proposals ────────────────────────────────────────────────
@@ -1108,7 +1108,7 @@ vv_ai_chat_markup('vv-ai', [
function loadTokens() {
fetch(API + '?action=tokens').then(r => r.json())
.then(d => { if (d.ok) { tokData = d.tokens; renderTokens(); } })
.catch(() => {});
.catch(e => vvFetchErr('AI token ledger', e));
}
function renderTokens() {
File diff suppressed because it is too large Load Diff
+329 -47
View File
@@ -87,6 +87,7 @@
// api/ai.php the AI row's turns and conversation store
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once dirname(__DIR__) . '/include/monitor.php';
require_once dirname(__DIR__) . '/include/monitor_board.php';
require_once dirname(__DIR__) . '/include/ai_chat.php';
require_once dirname(__DIR__) . '/include/docs.php';
if (vv_ai_ui_on()) vv_ai_chat_assets();
@@ -104,14 +105,24 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-api-banner" style="display:none;border-radius:4px;padding:5px 10px;margin-bottom:8px;font-size:11px;"></div>
<div id="vv-monitor" style="display:grid;grid-template-columns:repeat(8,1fr);gap:12px;width:100%;box-sizing:border-box;">
<!-- Placement for every card below comes from include/monitor_board.php spans, order, the
column ladder and the row-height cap. Nothing here carries a grid-column of its own, which is
the point: an inline style outranks the whole stylesheet, so the cards that used to carry one
could only be overridden by shouting !important at them from four media queries. -->
<style><?= vv_mon_board_css() ?></style>
<!-- Row 1: System | Power | CPU | Memory | Network -->
<div class="vv-card" id="vv-system" style="grid-column:span 1;">
<div id="vv-board-warn" style="display:none;border-radius:4px;padding:5px 10px;margin-bottom:8px;font-size:11px;background:#2a1414;border:1px solid #5a1e1e;color:#e0a0a0;"></div>
<div id="vv-monitor">
<!-- Board group 'Host' membership and width are declared in include/monitor_board.php.
Listing them here too would be a second answer to the same question, and the one that
drifts: the group below is only what the board happens to render at eight columns. -->
<div class="vv-card" id="vv-system">
<div id="vv-system-body">Loading...</div>
</div>
<div class="vv-card" id="vv-ups-card" style="grid-column:span 1;">
<div class="vv-card" id="vv-ups-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<svg width="9" height="14" viewBox="0 0 11 18" fill="none" style="opacity:0.4;flex-shrink:0;">
@@ -123,7 +134,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-ups-body">Loading...</div>
</div>
<div class="vv-card" id="vv-cpu" style="grid-column:span 2;">
<div class="vv-card" id="vv-cpu">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="12" height="12" viewBox="0 0 12 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round"><rect x="3" y="3" width="6" height="6" rx="0.8"/><line x1="4.5" y1="3" x2="4.5" y2="1.2"/><line x1="7.5" y1="3" x2="7.5" y2="1.2"/><line x1="4.5" y1="9" x2="4.5" y2="10.8"/><line x1="7.5" y1="9" x2="7.5" y2="10.8"/><line x1="3" y1="4.5" x2="1.2" y2="4.5"/><line x1="3" y1="7.5" x2="1.2" y2="7.5"/><line x1="9" y1="4.5" x2="10.8" y2="4.5"/><line x1="9" y1="7.5" x2="10.8" y2="7.5"/></svg></span>
@@ -134,7 +145,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-cpu-body">Loading...</div>
</div>
<div class="vv-card" id="vv-memory" style="grid-column:span 2;">
<div class="vv-card" id="vv-memory">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="14" height="8" viewBox="0 0 14 8" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round"><rect x="0.6" y="2.5" width="12.8" height="3" rx="0.5"/><line x1="3" y1="2.5" x2="3" y2="0.8"/><line x1="5.5" y1="2.5" x2="5.5" y2="0.8"/><line x1="8" y1="2.5" x2="8" y2="0.8"/><line x1="10.5" y1="2.5" x2="10.5" y2="0.8"/><line x1="3" y1="5.5" x2="3" y2="7.2"/><line x1="5.5" y1="5.5" x2="5.5" y2="7.2"/><line x1="8" y1="5.5" x2="8" y2="7.2"/><line x1="10.5" y1="5.5" x2="10.5" y2="7.2"/></svg></span>
@@ -145,7 +156,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-memory-body">Loading...</div>
</div>
<div class="vv-card" id="vv-network" style="grid-column:span 2;">
<div class="vv-card" id="vv-network">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="13" height="12" viewBox="0 0 13 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round"><rect x="4.5" y="0.5" width="4" height="3" rx="0.7"/><rect x="0.5" y="8" width="4" height="3" rx="0.7"/><rect x="8.5" y="8" width="4" height="3" rx="0.7"/><line x1="6.5" y1="3.5" x2="6.5" y2="6"/><line x1="6.5" y1="6" x2="2.5" y2="6"/><line x1="2.5" y1="6" x2="2.5" y2="8"/><line x1="6.5" y1="6" x2="10.5" y2="6"/><line x1="10.5" y1="6" x2="10.5" y2="8"/></svg></span>
@@ -156,8 +167,8 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-network-body">Loading...</div>
</div>
<!-- Row 2: Scripts | Fallback | Partner | Containers & VMs -->
<div class="vv-card" id="vv-scripts-card" style="grid-column:span 1;">
<!-- Board group 'Workload' -->
<div class="vv-card" id="vv-scripts-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="12" height="12" viewBox="0 0 12 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><rect x="1" y="1" width="10" height="10" rx="1.5"/><line x1="1" y1="4.5" x2="11" y2="4.5"/><polyline points="3.5,7 5,8 3.5,9"/><line x1="6" y1="9" x2="8.5" y2="9"/></svg></span>
@@ -168,7 +179,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-scripts-body">Loading...</div>
</div>
<div class="vv-card" id="vv-fallback" style="grid-column:span 1;">
<div class="vv-card" id="vv-fallback">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="10" height="12" viewBox="0 0 10 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><path d="M5 0.8L9.2 2.8V6.2C9.2 9 5 11.2 5 11.2C5 11.2 0.8 9 0.8 6.2V2.8Z"/></svg></span>
@@ -178,7 +189,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-fallback-body">Loading...</div>
</div>
<div class="vv-card" id="vv-partner" style="grid-column:span 2;">
<div class="vv-card" id="vv-partner">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="14" height="10" viewBox="0 0 14 10" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round"><rect x="0.5" y="1.5" width="4" height="7" rx="0.8"/><rect x="9.5" y="1.5" width="4" height="7" rx="0.8"/><line x1="4.5" y1="5" x2="9.5" y2="5"/><circle cx="7" cy="5" r="1" fill="#888" stroke="none"/></svg></span>
@@ -189,7 +200,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-partner-body">Loading...</div>
</div>
<div class="vv-card" id="vv-docker-folders" style="grid-column:span 4;">
<div class="vv-card" id="vv-docker-folders">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="14" height="10" viewBox="0 0 14 10" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><rect x="1" y="0.5" width="12" height="2.5" rx="0.5"/><rect x="1" y="3.8" width="12" height="2.5" rx="0.5"/><rect x="1" y="7" width="12" height="2.5" rx="0.5"/></svg></span>
@@ -200,8 +211,8 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-docker-folders-body">Loading...</div>
</div>
<!-- Row 3: Rsync | GPU 0 | GPU 1 | Transcode | Streams -->
<div class="vv-card" id="vv-rsync-card" style="grid-column:span 1;">
<!-- Board group 'Media' -->
<div class="vv-card" id="vv-rsync-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="12" height="12" viewBox="0 0 12 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><polyline points="2,4 10,4 8,2"/><polyline points="10,8 2,8 4,10"/></svg></span>
@@ -211,7 +222,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-rsync-body">Loading...</div>
</div>
<div class="vv-card" id="vv-gpu-card" style="grid-column:span 1;">
<div class="vv-card" id="vv-gpu-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="15" height="10" viewBox="0 0 16 10" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><rect x="0.8" y="0.8" width="14.4" height="7" rx="1.2"/><rect x="2.5" y="2.5" width="3.5" height="3.5" rx="0.5"/><line x1="8" y1="3" x2="13" y2="3"/><line x1="8" y1="5" x2="11" y2="5"/><rect x="3" y="7.8" width="2" height="1.8" rx="0.3" stroke="none" fill="#666"/><rect x="6.5" y="7.8" width="2" height="1.8" rx="0.3" stroke="none" fill="#666"/><rect x="10" y="7.8" width="2" height="1.8" rx="0.3" stroke="none" fill="#666"/></svg></span>
@@ -222,7 +233,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-gpu-body">Loading...</div>
</div>
<div class="vv-card" id="vv-gpu1-card" style="grid-column:span 1;">
<div class="vv-card" id="vv-gpu1-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="15" height="10" viewBox="0 0 16 10" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><rect x="0.8" y="0.8" width="14.4" height="7" rx="1.2"/><rect x="2.5" y="2.5" width="3.5" height="3.5" rx="0.5"/><line x1="8" y1="3" x2="13" y2="3"/><line x1="8" y1="5" x2="11" y2="5"/><rect x="3" y="7.8" width="2" height="1.8" rx="0.3" stroke="none" fill="#666"/><rect x="6.5" y="7.8" width="2" height="1.8" rx="0.3" stroke="none" fill="#666"/><rect x="10" y="7.8" width="2" height="1.8" rx="0.3" stroke="none" fill="#666"/></svg></span>
@@ -233,7 +244,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-gpu1-body">Loading...</div>
</div>
<div class="vv-card" id="vv-transcode" style="grid-column:span 1;">
<div class="vv-card" id="vv-transcode">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="14" height="12" viewBox="0 0 14 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><rect x="0.5" y="1" width="4" height="10" rx="0.8"/><line x1="0.5" y1="3.5" x2="2.3" y2="3.5"/><line x1="0.5" y1="8.5" x2="2.3" y2="8.5"/><line x1="2.7" y1="3.5" x2="4.5" y2="3.5"/><line x1="2.7" y1="8.5" x2="4.5" y2="8.5"/><line x1="6" y1="6" x2="9.5" y2="6"/><polyline points="8.2,4.3 10.2,6 8.2,7.7"/><rect x="11" y="1" width="2.5" height="10" rx="0.5"/></svg></span>
@@ -243,18 +254,26 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-transcode-body">Loading...</div>
</div>
<div class="vv-card" id="vv-streams" style="grid-column:span 4;">
<!-- Scope control lives in the header rather than in the body because the body is replaced on
every poll, and a control that is re-rendered underneath a click is a control that loses
one. Hidden until a partner is known on a single-host install the two buttons would be
two names for the same view, which is how the Media Stack tab handles it too. -->
<div class="vv-card" id="vv-streams">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="11" height="12" viewBox="0 0 11 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><polygon points="1.5,1 1.5,11 10,6"/></svg></span>
Streams
</span>
<span id="vv-streams-scope" style="display:none;gap:4px;margin-left:auto;">
<button class="vv-strm-tab on" data-strmscope="local">This host</button>
<button class="vv-strm-tab" data-strmscope="mesh">Mesh</button>
</span>
</h3>
<div id="vv-streams-body">Loading...</div>
</div>
<!-- Row 4: Watchdog | Parity | Pools | Array -->
<div class="vv-card" id="vv-watchdog-card" style="grid-column:span 1;">
<!-- Board group 'Storage' -->
<div class="vv-card" id="vv-watchdog-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="14" height="9" viewBox="0 0 14 8" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round"><path d="M1 4C3.2 1 5 0.5 7 0.5C9 0.5 10.8 1 13 4C10.8 7 9 7.5 7 7.5C5 7.5 3.2 7 1 4Z"/><circle cx="7" cy="4" r="1.5"/></svg></span>
@@ -264,7 +283,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-watchdog-body">Loading...</div>
</div>
<div class="vv-card" id="vv-parity-card" style="grid-column:span 1;">
<div class="vv-card" id="vv-parity-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="12" height="12" viewBox="0 0 12 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round" stroke-linejoin="round"><circle cx="6" cy="6" r="5"/><polyline points="3.5,6 5.5,8 8.5,4"/></svg></span>
@@ -275,7 +294,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-parity-body">Loading...</div>
</div>
<div class="vv-card" id="vv-storage-card" style="grid-column:3/span 2;">
<div class="vv-card" id="vv-storage-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="12" height="12" viewBox="0 0 12 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round"><ellipse cx="6" cy="3" rx="4.5" ry="1.5"/><line x1="1.5" y1="3" x2="1.5" y2="9"/><line x1="10.5" y1="3" x2="10.5" y2="9"/><ellipse cx="6" cy="9" rx="4.5" ry="1.5"/></svg></span>
@@ -287,7 +306,7 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
<div id="vv-storage-body">Loading...</div>
</div>
<div class="vv-card" id="vv-array-card" style="grid-column:5/span 4;">
<div class="vv-card" id="vv-array-card">
<h3>
<span style="display:flex;align-items:center;gap:5px;">
<span class="vv-ico"><svg width="14" height="12" viewBox="0 0 14 12" fill="none" stroke="#aaa" stroke-width="1.1" stroke-linecap="round"><rect x="0.7" y="0.7" width="12.6" height="3" rx="0.8"/><rect x="0.7" y="4.5" width="12.6" height="3" rx="0.8"/><rect x="0.7" y="8.3" width="12.6" height="3" rx="0.8"/><circle cx="11.5" cy="2.2" r="0.8" fill="#888" stroke="none"/><circle cx="11.5" cy="6" r="0.8" fill="#888" stroke="none"/><circle cx="11.5" cy="9.8" r="0.8" fill="#888" stroke="none"/></svg></span>
@@ -301,6 +320,36 @@ $_vv_doc_vars = array_merge(vv_conf_vars(), ['SCRIPTS_DIR' => SCRIPTS_DIR]);
</div>
<script>
// The board is declared in one file and rendered in another, so the two can drift: a card added
// to the markup nobody declared gets order 0 and lands first, and a declaration with no card
// leaves the row it belonged to one span short. Neither shows as an error — the board just comes
// out subtly wrong — so both are checked here against what actually rendered, alongside anything
// vv_mon_board_check() already found wrong with the declaration itself.
window.VV_BOARD = <?= vv_mon_board_js() ?>;
(function vvBoardGuard() {
const grid = document.getElementById('vv-monitor');
const warn = document.getElementById('vv-board-warn');
if (!grid || !warn) return;
const rendered = Array.from(grid.children)
.filter(el => el.classList.contains('vv-card'))
.map(el => el.id);
const undeclared = rendered.filter(id => !window.VV_BOARD.ids.includes(id));
const missing = window.VV_BOARD.ids.filter(id => !rendered.includes(id));
const problems = window.VV_BOARD.problems.slice();
if (undeclared.length) problems.push('cards in the markup that the board does not declare: ' + undeclared.join(', '));
if (missing.length) problems.push('cards the board declares that did not render: ' + missing.join(', '));
if (!problems.length) return;
warn.textContent = 'Monitor board: ' + problems.join(' · ');
warn.style.display = 'block';
console.warn('[varaverk] monitor board', problems);
})();
</script>
<?php if (vv_ai_ui_on()): ?>
<!-- ── AI row its own grid, deliberately ──────────────────────────────────────
Same eight columns and the same gap, so it reads as the last row of the one above. It is a
@@ -941,6 +990,55 @@ function vvPollMonitor(live) {
: '—';
const _coreMeta = _threadInfo ? ` <span style="color:#444;">(${_threadInfo})</span>` : '';
// ── Varaverk's own footprint ────────────────────────────────────────────
// The rows above describe the machine. These describe the thing whose dashboard this is,
// and nothing else on the page reports them.
const vv = d.varaverk ?? {};
const _rootPct = d.watchdog?.stability?.rootfs_pct ?? null;
// The cache is RAM, not disk: VV_CACHE_ROOT is /tmp/varaverk and /tmp lives on rootfs, which
// on Unraid is a memory filesystem. Shown against the rootfs percentage because a size with
// no ceiling beside it is a number nobody can act on.
// Local formatters. vvFmt() is a GB formatter — feeding it megabytes prints "186 GB" — and
// vvRelTime() takes a timestamp where these carry an age in seconds. Both would have been
// silently wrong rather than broken.
const _vvSize = b => b == null ? '—'
: b >= 1073741824 ? (b / 1073741824).toFixed(1) + ' GB' : Math.round(b / 1048576) + ' MB';
const _vvAge = s => s == null ? '—'
: s < 90 ? s + 's' : s < 5400 ? Math.round(s / 60) + 'm'
: s < 172800 ? Math.round(s / 3600) + 'h' : Math.round(s / 86400) + 'd';
const _cacheColor = _rootPct == null ? '#888'
: _rootPct >= 90 ? '#f44336' : _rootPct >= 75 ? '#ff9800' : '#888';
// The suffix belongs to a real figure. "— in RAM" reads as a measurement that came back
// empty rather than one that was never taken.
const _cacheStr = vv.cache_bytes == null
? `<span style="color:#555;">not measured</span>`
: `${_vvSize(vv.cache_bytes)}<span style="color:#444;"> in RAM`
+ (_rootPct != null ? ` · rootfs ${_rootPct}%` : '') + `</span>`;
const _dataStr = vv.data_bytes == null
? `<span style="color:#555;">not measured</span>`
: `${_vvSize(vv.data_bytes)}<span style="color:#444;"> on ${vvEscHtml(vv.storage || '?')}</span>`;
// Jobs actually running — locks whose process is alive. A stale lock is not a running job
// and acquire_lock() clears it on the next run, so it is reported as an aside rather than
// as a fault.
const _jobs = vv.jobs_running ?? [];
const _jobStr = _jobs.length === 0
? `<span style="color:#555;">idle</span>`
: `<span style="color:#4caf50;">${_jobs.length} running</span>`
+ `<span style="color:#444;"> · ${vvEscHtml(_jobs.slice(0, 2).join(', '))}`
+ (_jobs.length > 2 ? ` +${_jobs.length - 2}` : '') + `</span>`;
const _staleStr = (vv.locks_stale ?? 0) > 0
? `<span style="color:#3a3a3a;"> · ${vv.locks_stale} stale lock${vv.locks_stale !== 1 ? 's' : ''}</span>` : '';
// Conf sync freshness. An age climbing past its schedule means the mesh has stopped talking,
// which no other row on this page would show.
const _confAge = vv.conf_age_sec;
const _confStr = _confAge == null ? `<span style="color:#555;">none cached</span>`
: `<span style="color:${_confAge > 172800 ? '#ff9800' : '#888'};">${_vvAge(_confAge)} ago</span>`;
document.getElementById('vv-system-body').innerHTML =
`<div style="display:flex;justify-content:space-between;align-items:flex-start;margin-bottom:8px;">
<div style="min-width:0;">
@@ -980,12 +1078,43 @@ function vvPollMonitor(live) {
<span style="color:#444;">Load</span> <span style="color:${_loadColor};">${_loadStr}</span>
<span style="color:#444;">Running</span> <span style="color:#888;">${_runningCtrs} ctr${_runningCtrs !== 1 ? 's' : ''}${_runningVMs > 0 ? ` · ${_runningVMs} VM` : ''}</span>
<span style="color:#444;">Version</span> <span style="color:#3a3a3a;">${ver}</span>
</div>
<!-- Varaverk's own figures, ruled off from the host's. Same grid so the labels line up,
separate block so it reads as a different subject rather than more of the same. -->
<div style="border-top:1px solid #1e1e1e;margin-top:8px;padding-top:7px;">
<div style="font-size:9px;color:#333;text-transform:uppercase;letter-spacing:.07em;margin-bottom:5px;">Varaverk</div>
<div style="display:grid;grid-template-columns:auto 1fr;gap:3px 8px;font-size:11px;">
<span style="color:#444;">Cache</span> <span style="color:${_cacheColor};">${_cacheStr}</span>
<span style="color:#444;">Data</span> <span style="color:#888;">${_dataStr}</span>
<span style="color:#444;">Jobs</span> <span>${_jobStr}${_staleStr}</span>
<span style="color:#444;">Conf</span> <span>${_confStr}</span>
<span style="color:#444;">Build</span> <span style="color:#3a3a3a;font-family:monospace;">${vvEscHtml(vv.commit || '—')}</span>
</div>
</div>`;
// ── Partner ──────────────────────────────────────────────────────────────
const pt = d.partner ?? {};
const ptHosts = pt.hosts ?? [];
const ptRemote = d.remote_hosts ?? {};
// The Streams scope control is offered only once there is a partner to combine with. On a
// single-host install "This host" and "Mesh" are two names for the same view, and the
// Media Stack tab hides its equivalent for the same reason.
const _strmScopeEl = document.getElementById('vv-streams-scope');
const _strmHasPeer = Object.keys(ptRemote).length > 0;
if (_strmScopeEl) _strmScopeEl.style.display = _strmHasPeer ? 'flex' : 'none';
// A remembered "mesh" outlives the partner that justified it — an offboard, or a browser
// carrying the preference to a single-host install. The control is hidden in that state,
// so there would be no way back: the card would sit in a mesh view of a mesh that is not
// there, with nothing on screen to say why. The scope follows the control.
if (!_strmHasPeer && vvStreamScope === 'mesh') {
vvStreamScope = 'local';
try { localStorage.setItem(VV_STRM_SCOPE_KEY, 'local'); } catch (_) {}
vvStreamScopeSync();
vvPollStreams();
}
const ptStatus = pt.enabled
? `<span style="color:#4caf50;">enabled</span> · sync every ${pt.sync_min}min`
: `<span style="color:#555;">disabled</span>`;
@@ -1117,7 +1246,7 @@ function vvPollMonitor(live) {
// ── State label + outage duration ──────────────────────────────────────
let stateExtra = '';
if (state === 'FALLBACK' || state === 'DARK') {
const start = parseInt(fb.failover_start ?? 0);
const start = parseInt(fb.fallback_start ?? 0);
if (start > 0) {
const sec = Math.floor(Date.now() / 1000) - start;
stateExtra = ` · ${Math.floor(sec/3600)}h ${Math.floor((sec%3600)/60)}m`;
@@ -1125,6 +1254,49 @@ function vvPollMonitor(live) {
}
fbHtml += `<div style="font-size:11px;color:${sColor};font-weight:500;margin-bottom:6px;">${sLabel}${stateExtra}</div>`;
// ── Daemon liveness — shown in EVERY state ─────────────────────────────
// The line above describes a file. This one describes whether anything is still writing
// it. Without it the card read "✓ Nominal · monitoring · 30s" on a host where nothing had
// run for five days: the state file said NORMAL, and nothing contradicted it.
const dae = fb.daemon ?? {};
let daeHtml;
if (dae.stale_lock) {
daeHtml = `<span style="color:#f44336;"> not running</span>
<span style="color:#444;"> · stale lock${dae.pid ? ' (PID ' + dae.pid + ')' : ''}</span>`;
} else if (dae.running === false) {
daeHtml = `<span style="color:#f44336;"> not running</span>
<span style="color:#444;"> · nothing is watching</span>`;
} else if (dae.running && dae.mode === 'dry-run') {
// A preview decides nothing and writes to its own state copy, so the live figures on
// this card stop moving while it runs. Said plainly rather than left to infer.
daeHtml = `<span style="color:#4a9eff;"> dry run</span>
<span style="color:#444;"> · preview only, PID ${dae.pid}</span>`;
} else if (dae.running) {
daeHtml = `<span style="color:#4caf50;"> watching</span>
<span style="color:#444;"> · every ${interval}s</span>`;
} else {
daeHtml = `<span style="color:#555;">— unknown</span>`;
}
fbHtml += `<div style="font-size:10px;margin-bottom:7px;">${daeHtml}</div>`;
// ── Mutual coverage ────────────────────────────────────────────────────
// Both directions, because they are configured in different files and are routinely
// asymmetric — this mesh currently has one side fully populated and the other empty,
// which is invisible if the card only reports its own.
const pid2 = fb.partner_id;
if (pid2) {
const we = fb.we_cover;
const they = fb.they_cover;
const num = (n, warn) => n === null || n === undefined
? `<span style="color:#333;">—</span>`
: `<span style="color:${n === 0 && warn ? '#ff9800' : '#888'};">${n}</span>`;
fbHtml += `<div style="display:grid;grid-template-columns:1fr auto;gap:1px 8px;font-size:10px;margin-bottom:6px;">
<span style="color:#444;">We cover ${vvEscHtml(pid2)}</span>${num(we, true)}
<span style="color:#444;">${vvEscHtml(pid2)} covers us</span>${num(they, false)}
</div>`;
if (we === 0) fbHtml += `<div style="font-size:9px;color:#5a4020;margin-bottom:6px;">No tiers set for ${vvEscHtml(pid2)} — we would start nothing</div>`;
}
// ── Handback strike progress ───────────────────────────────────────────
if (state === 'FALLBACK' && strikes > 0) {
const dots = Array.from({length: maxStrikes}, (_,i) =>
@@ -1137,7 +1309,7 @@ function vvPollMonitor(live) {
</div>`;
}
// ── Tier badges (FAILOVER only) ────────────────────────────────────────
// ── Tier badges (FALLBACK only) ────────────────────────────────────────
if (state === 'FALLBACK') {
const t2 = fb.tier2_started === 'true';
const t3 = fb.tier3_started === 'true';
@@ -1170,9 +1342,22 @@ function vvPollMonitor(live) {
// ── NORMAL quiet state ─────────────────────────────────────────────────
if (state === 'NORMAL' && !fbActive.length) {
let meta = `monitoring · ${interval}s`;
if (ptRequired) meta += ' · partnership gated';
fbHtml += `<div style="font-size:10px;color:#2a2a2a;">${meta}</div>`;
// "monitoring · 30s" used to live here and was the card's only footer — a claim about a
// running process made from a file. The daemon row above owns that claim now, so this
// reports the things the state file genuinely knows.
const bits = [];
const lc = fb.last_change;
if (lc !== null && lc !== undefined) {
const a = lc < 60 ? lc + 's' : lc < 3600 ? Math.floor(lc/60) + 'm'
: lc < 86400 ? Math.floor(lc/3600) + 'h' : Math.floor(lc/86400) + 'd';
bits.push(`no change in ${a}`);
}
if (ptRequired) bits.push('partnership gated');
// Only mentioned when OFF, and only here: handback writeback is the step that carries
// work done during an outage back to the recovered host, and losing it silently is the
// expensive half of a failover nobody notices until afterwards.
if (fb.rsync_on_handback === false) bits.push('<span style="color:#5a4020;">no handback rsync</span>');
if (bits.length) fbHtml += `<div style="font-size:9px;color:#2a2a2a;">${bits.join(' · ')}</div>`;
}
// ── NO_INTERNET / DARK detail ──────────────────────────────────────────
@@ -1189,6 +1374,12 @@ function vvPollMonitor(live) {
if (fbCard) {
fbCard.classList.remove('vv-accent-ok','vv-accent-warn','vv-accent-err');
if (!enabled || ptSuspended) fbCard.classList.add('vv-accent-warn');
// A green accent is a claim that this is covered. It is not, if nothing is running or
// the only thing running is a preview that would take no action.
else if (fb.daemon && (fb.daemon.running === false || fb.daemon.stale_lock))
fbCard.classList.add('vv-accent-err');
else if (fb.daemon && fb.daemon.mode === 'dry-run')
fbCard.classList.add('vv-accent-warn');
else if (state === 'NORMAL') fbCard.classList.add('vv-accent-ok');
else if (state === 'FALLBACK') fbCard.classList.add('vv-accent-err');
else if (state !== 'UNKNOWN') fbCard.classList.add('vv-accent-warn');
@@ -1389,8 +1580,16 @@ function vvPollMonitor(live) {
const issueCount = ctrNames.length + stabNames.length + storIssues
+ (daemonHit ? 1 : 0) + (oom > 0 ? 1 : 0)
+ (reboots > 0 ? 1 : 0) + (npmStrikes > 0 ? 1 : 0);
const bannerCls = healthy ? 'vv-banner-ok' : (reboots || oom || daemonHit ? 'vv-banner-err' : 'vv-banner-warn');
const bannerTxt = healthy ? '✓ All clear' : `⚠ ${issueCount} issue${issueCount !== 1 ? 's' : ''}`;
// A host whose watchdogs have never executed has no verdict to report. Every strike set
// reads absent state as zero, so "All clear" here would be standing in for no data —
// which is the failure this card's own header warns about. Said plainly instead.
const neverRan = wd.never_ran === true;
const bannerCls = neverRan ? 'vv-banner-warn'
: healthy ? 'vv-banner-ok'
: (reboots || oom || daemonHit ? 'vv-banner-err' : 'vv-banner-warn');
const bannerTxt = neverRan ? '○ Never run on this host'
: healthy ? '✓ All clear'
: `⚠ ${issueCount} issue${issueCount !== 1 ? 's' : ''}`;
let html = `<div class="vv-banner ${bannerCls}">${bannerTxt}`;
if (reboots > 0) html += `<span style="font-size:10px;">${reboots} reboot${reboots !== 1 ? 's' : ''}/12h</span>`;
@@ -1514,7 +1713,7 @@ function vvPollMonitor(live) {
const wdCard = document.getElementById('vv-watchdog-card');
if (wdCard) {
wdCard.classList.remove('vv-accent-ok','vv-accent-warn','vv-accent-err');
if (healthy) wdCard.classList.add('vv-accent-ok');
if (healthy && !neverRan) wdCard.classList.add('vv-accent-ok');
else if (reboots > 0 || oom > 0 || daemonHit) wdCard.classList.add('vv-accent-err');
else wdCard.classList.add('vv-accent-warn');
}
@@ -1858,14 +2057,17 @@ function vvPollMonitor(live) {
renderGpuCard(gpuList[0], 'vv-gpu-body', 'vv-gpu-label', 'GPU');
renderGpuCard(gpuList[1], 'vv-gpu1-body', 'vv-gpu1-label', 'GPU 1');
// Row 3 is 8 columns wide: Rsync(1) + GPU0(1) + GPU1(1) + Transcode(1) + Streams(4).
// On a single-GPU host the second card is hidden and Transcode widens back to 2 so the
// row still fills exactly — otherwise it would leave a one-column hole.
// On a single-GPU host the second GPU card is hidden, and its column has to go somewhere or
// the row is one span short at every rung. Which card absorbs it is declared on the board
// ('absorbs' => 'vv-gpu1-card'); all this does is say the card is gone. The width itself is
// set by generated CSS, per rung — this used to write style.gridColumn='span 2' directly,
// which an inline style made unclampable, so a one-GPU phone got a span of 2 in a
// one-column grid and scrolled sideways.
const gpu1Card = document.getElementById('vv-gpu1-card');
const transcodeCard = document.getElementById('vv-transcode');
const twoGpus = gpuList.length > 1;
if (gpu1Card) gpu1Card.style.display = twoGpus ? '' : 'none';
if (transcodeCard) transcodeCard.style.gridColumn = twoGpus ? 'span 1' : 'span 2';
const twoGpus = gpuList.length > 1;
if (gpu1Card) gpu1Card.style.display = twoGpus ? '' : 'none';
const gpu1Absent = window.VV_BOARD?.absent?.['vv-gpu1-card'];
if (gpu1Absent) document.getElementById('vv-monitor')?.classList.toggle(gpu1Absent, !twoGpus);
// ── Scripts ─────────────────────────────────────────────────────────────
vvLastScripts = d.scripts ?? {};
@@ -2073,7 +2275,7 @@ function vvLoadTokens() {
return fetch('/plugins/varaverk/api/ai.php?action=tokens')
.then(r => r.json())
.then(d => { if (d.ok) { vvTokData = d.tokens; vvRenderTokens(); } })
.catch(() => {});
.catch(e => vvFetchErr('token ledger', e));
}
function vvRenderTokens() {
@@ -2407,7 +2609,8 @@ function vvRenderStreams() {
if (sessions.length === 0) {
el.innerHTML = `<div class="vv-stream-servers"><div class="vv-stream-left">${badges}</div></div>`
+ '<p class="vv-stream-empty">Nothing playing</p>';
+ `<p class="vv-stream-empty">Nothing playing${vvStreamScope === 'mesh' ? ' anywhere in the mesh' : ''}</p>`
+ vvStreamNodeNotes();
return;
}
@@ -2437,7 +2640,12 @@ function vvRenderStreams() {
<div style="display:flex;justify-content:space-between;align-items:center;font-size:11px;margin-bottom:2px;">
<span style="color:${s.paused ? '#fdd835' : '#aaa'};white-space:nowrap;overflow:hidden;text-overflow:ellipsis;flex:1;">
<span style="color:${iconColor};">${icon}</span> ${vvEscHtml(s.title)}</span>
<span style="color:#444;font-size:10px;margin-left:6px;flex-shrink:0;">${vvEscHtml(s.server)}</span>
<span style="color:#444;font-size:10px;margin-left:6px;flex-shrink:0;">${vvEscHtml(s.server)}${
// Only in mesh scope: in local scope every row is this host, and the badge would be the
// same word on every line.
vvStreamScope === 'mesh' && s._hostName
? `<span class="vv-strm-node">${vvEscHtml(s._hostName)}</span>` : ''
}</span>
</div>
<div style="display:flex;justify-content:space-between;font-size:10px;color:#555;margin-bottom:3px;">
<span>${vvEscHtml(s.user)}</span>
@@ -2468,29 +2676,90 @@ function vvRenderStreams() {
<div class="vv-stream-left">${badges}${mtypeSection}</div>
<div class="vv-stream-right">${deviceSection}${resSection}${codecSection}</div>
</div>
<div style="display:flex;gap:10px;">${sCols.map(vvStreamCol).join('')}</div>${overflow}`;
<div style="display:flex;gap:10px;">${sCols.map(vvStreamCol).join('')}</div>${overflow}${vvStreamNodeNotes()}`;
}
// Local by default, and it stays the cheap path — one call per media server on this box. Mesh adds
// a bounded SSH hop per partner, paid only while the operator is looking at it.
// Remembered per browser, because which scope you want is a property of the screen in front of
// you: the wall dashboard wants the whole mesh, a laptop opened to check one box does not. A
// first visit — and anything unrecognised in storage — is local.
//
// Validated against the two real scopes rather than trusted. A stale or hand-edited key would
// otherwise put the card in a scope with no button to leave it by.
const VV_STRM_SCOPE_KEY = 'vv-streams-scope';
let vvStreamScope = 'local';
try {
const _s = localStorage.getItem(VV_STRM_SCOPE_KEY);
if (_s === 'local' || _s === 'mesh') vvStreamScope = _s;
} catch (_) {}
// The markup ships with "This host" lit, so this only ever has to move the highlight.
if (vvStreamScope !== 'local') vvStreamScopeSync();
function vvStreamScopeSync() {
document.querySelectorAll('#vv-streams [data-strmscope]').forEach(b =>
b.classList.toggle('on', b.dataset.strmscope === vvStreamScope));
}
let vvStreamNodes = [];
function vvPollStreams() {
return fetch('/plugins/varaverk/api/media.php')
const scope = vvStreamScope;
return fetch('/plugins/varaverk/api/media.php' + (scope === 'mesh' ? '?scope=mesh' : ''))
.then(r => r.json())
.then(d => {
// A reply that arrives after the operator switched scope describes the other view. Dropped
// rather than rendered: mesh rows landing in a local view is the kind of wrongness nobody
// reads as a bug, they just believe it.
if (scope !== vvStreamScope) return;
vvLastSessions = d.sessions ?? [];
vvLastStreamNames = d.server_names ?? [];
vvStreamServerCount = d.server_count ?? 0;
vvStreamNodes = d.nodes ?? [];
vvLastStreamPollAt = Math.floor(Date.now() / 1000);
const el = document.getElementById('vv-streams-body');
if (vvStreamServerCount === 0) {
el.innerHTML = '<p class="vv-stream-empty">No media servers detected.<br>'
+ '<span>Add EMBY_API_KEY / JELLYFIN_API_KEY / PLEX_TOKEN to master.conf to configure.</span></p>';
el.innerHTML = scope === 'mesh'
? '<p class="vv-stream-empty">No media servers anywhere in the mesh.</p>' + vvStreamNodeNotes()
: '<p class="vv-stream-empty">No media servers detected.<br>'
+ '<span>Add EMBY_API_KEY / JELLYFIN_API_KEY / PLEX_TOKEN to master.conf to configure.</span></p>';
return;
}
vvRenderStreams();
})
.catch(() => {});
.catch(e => vvFetchErr('media streams', e));
}
// An unreachable partner is stated, never folded into the totals as zero. "Nobody is watching
// there" and "that node did not answer" are opposite facts and look identical in a count.
function vvStreamNodeNotes() {
if (vvStreamScope !== 'mesh') return '';
const bad = vvStreamNodes.filter(n => !n.local && !n.reachable);
if (!bad.length) return '';
return bad.map(n =>
`<div style="font-size:10px;color:#a05a2c;margin-top:4px;">${vvEscHtml(n.name)} — unreachable, not counted</div>`
).join('');
}
// Delegated off the card, so it survives the body being replaced on every poll.
document.getElementById('vv-streams')?.addEventListener('click', ev => {
const btn = ev.target.closest('[data-strmscope]');
if (!btn) return;
const next = btn.dataset.strmscope;
if (next === vvStreamScope) return;
vvStreamScope = next;
try { localStorage.setItem(VV_STRM_SCOPE_KEY, next); } catch (_) {}
vvStreamScopeSync();
// Switching is a request to see the other view now, not at the next tick.
const el = document.getElementById('vv-streams-body');
if (el) el.innerHTML = '<p class="vv-stream-empty">Loading…</p>';
vvPollStreams();
});
// Guarded like the others — this one reaches out to every configured media server, so a wedged
// Emby is exactly the case where unguarded ticks would stack.
vvPollRunner(vvPollStreams, 12000);
@@ -2707,6 +2976,12 @@ function vvRenderDockerFolders(data) {
const stateColor = s => ({ running:'#4caf50', paused:'#ff9800' })[s] ?? '#444';
const stateLabel = s => ({ running:'Running', paused:'Paused', 'shut off':'Off', crashed:'Crashed' })[s] ?? s;
// Fallback tier stripe. Warm (comes back immediately) to cool (waits 24h), riding the row's
// left border rather than the status dot — the dot already means running vs stopped, and
// overloading it would make a stopped tier-1 container indistinguishable from a running one.
const tierColor = t => ({ 1:'#ff5252', 2:'#ffa726', 3:'#ffd54f', 4:'#4fc3f7' })[t] ?? '';
const tierWhen = t => ({ 1:'immediately', 2:'after 4h', 3:'after 12h', 4:'after 24h' })[t] ?? '';
// ── VMs section ─────────────────────────────────────────────────────────────
let html = '';
const vms = data.vms?.vms ?? [];
@@ -2765,7 +3040,14 @@ function vvRenderDockerFolders(data) {
class="vv-btn-sm vv-edit-btn"> Edit</button>
</div>`;
}
return `<div class="vv-df-container${active ? ' vv-df-active' : ''}"
// Tier arrives as an int from conf, but it is still payload — anything outside 1-4 gets no
// stripe rather than being interpolated into the style attribute.
const tNum = Number(c.tier);
const tOk = Number.isInteger(tNum) && tNum >= 1 && tNum <= 4;
const tSty = tOk ? ` style="border-left-color:${tierColor(tNum)};"` : '';
const tTip = tOk ? ` title="Fallback tier ${tNum} — partner starts this ${tierWhen(tNum)}"` : '';
return `<div class="vv-df-container${active ? ' vv-df-active' : ''}"${tSty}${tTip}
onclick="event.stopPropagation();vvToggleContainer('${sn}')">
<span class="vv-df-dot" style="background:${dot};${pulse}"></span>
<span class="vv-df-cname">${vvEscHtml(c.name)}</span>
+5 -3
View File
@@ -710,14 +710,16 @@ function _renderConfig(cfg) {
const fbColors = {
NORMAL: '#4caf50',
FAILOVER: '#f44336',
// FALLBACK — fallback.sh writes NORMAL|FALLBACK|NO_INTERNET|DARK and never 'FAILOVER', so
// this entry matched nothing and a covering host rendered as UNKNOWN grey.
FALLBACK: '#f44336',
NO_INTERNET: '#ff9800',
DARK: '#9e9e9e',
UNKNOWN: '#444',
};
const fbLabels = {
NORMAL: '✓ Nominal',
FAILOVER: '⚠ Failover',
FALLBACK: '⚠ Fallback active',
NO_INTERNET: '⚡ No internet',
DARK: '◌ Dark mode',
UNKNOWN: '— Unknown',
@@ -1274,7 +1276,7 @@ function vvPtLoad() {
fetch('/plugins/varaverk/api/partnership.php')
.then(r => r.json())
.then(_render)
.catch(() => {});
.catch(e => vvFetchErr('partnership status', e));
}
_vvPtReload = vvPtLoad; // expose to top-level toggle fns
@@ -10,6 +10,22 @@ a minute, so polling faster could not make it newer — it only decides how soon
**Cards you do not have do not appear.** No GPU, no UPS, no VMs, no partner — the card is absent
rather than showing zeros. An empty card would be a permanent reminder of nothing.
**The board re-flows on a narrow screen.** On a tablet or a phone the cards regroup into fewer,
wider columns rather than shrinking in place. Nothing is hidden and nothing is dropped — the same
cards are there in the same order, cut into different rows.
Below the full-width layout the page scrolls as a whole instead of each card scrolling inside
itself. A board that is taller than the screen has to scroll somewhere, and scrolling the page
once is better than being handed a screen of cards that are each too short to read.
**Cards in the same row share a height.** They line up along the bottom, so a card with little to
say carries empty space under it. That space means its neighbour is the taller card — not that a
reading is missing.
The System card is the one exception. On a narrow screen it stops after Version and the Varaverk
figures below scroll, so that the host itself — hostname, clock, array state, uptime — is what
you get without scrolling anything.
---
## Reference — System, power, CPU, memory, network
+89 -6
View File
@@ -40,6 +40,19 @@ require_once dirname(__DIR__) . '/include/confui.php';
require_once dirname(__DIR__) . '/include/ai_chat.php';
?>
<style>
/* ── Quick-add share chips ───────────────────────────────────────────────────────────────────
A chip is a share, not a path. Anything already covered by this window is shown dimmed and
inert rather than hidden a share missing from the strip would read as "not on this host",
which is a different and wrong statement. */
.vv-ry-chips { display:flex;flex-wrap:wrap;gap:3px;max-height:96px;overflow-y:auto;
padding:4px;background:#0a0a0a;border:1px solid #161616;border-radius:3px; }
.vv-ry-chip { font-size:10px;padding:2px 7px;border-radius:10px;cursor:pointer;
background:#0e1a2a;color:#7ab;border:1px solid #1e3a5a;white-space:nowrap;
user-select:none; }
.vv-ry-chip:hover { background:#14263d; }
.vv-ry-chip.in { background:#0d1a0d;color:#4a7a4a;border-color:#1f3a1f;cursor:default; }
.vv-ry-chip.part { background:#1a1408;color:#a08a4a;border-color:#3a2f1a; }
.vv-ry-chip.gone { opacity:.45; }
.vv-ry-card { background:#161616;border:1px solid #2a2a2a;border-radius:6px;padding:10px 12px;min-width:0; }
.vv-ry-sec { font-size:10px;font-weight:bold;color:#444;letter-spacing:.07em;text-transform:uppercase;margin-bottom:6px; }
.vv-ry-row { display:flex;justify-content:space-between;align-items:baseline;gap:6px;margin:2px 0; }
@@ -1074,7 +1087,7 @@ function vvRyLoad() {
fetch('/plugins/varaverk/api/rsync.php')
.then(r => r.json())
.then(_render)
.catch(() => {});
.catch(e => vvFetchErr('rsync config', e));
}
vvRyLoad();
@@ -1237,6 +1250,15 @@ function _vvRyBuildEditPanel(key) {
}
// Share add form
h += `<div style="margin-top:6px;" onclick="event.stopPropagation()">
<!-- Quick select. The path box below stays the precise tool subpaths like
/mnt/user/Media_Server/Emby and per-entry profiles can only be expressed there. This
strip only covers the common case: add one whole share, one click. -->
<div style="margin-bottom:5px;">
<div style="font-size:9px;color:#4a6a80;text-transform:uppercase;letter-spacing:.04em;margin-bottom:3px;">
Quick add a share <span style="color:#3a3a3a;text-transform:none;letter-spacing:0;"> uses the profile selected below</span>
</div>
<div id="vv-ry-echips-${key}" class="vv-ry-chips">loading…</div>
</div>
<div style="display:flex;gap:4px;margin-bottom:3px;">
<input id="vv-ry-epath-${key}" type="text" placeholder="/mnt/user/…" autocomplete="off"
style="background:#0d0d0d;border:1px solid #1e1e1e;border-radius:3px;color:#888;
@@ -1272,6 +1294,9 @@ function _vvRyBuildEditPanel(key) {
panel.innerHTML = h;
_vvRyLoadProfiles(key);
// After innerHTML, not before — the chip host does not exist until the panel is written, and
// the strip has to redraw on every rebuild so an entry added or removed re-marks its share.
vvRyRenderChips(key);
}
// ── Script mutations ──────────────────────────────────────────────────────────
@@ -1302,6 +1327,64 @@ function vvRyERemShare(key, idx) {
const s = _vvRyWinEState[key]?.shares; if (!s) return;
s.splice(idx, 1); _vvRyBuildEditPanel(key);
}
// ── Quick-add share chips ────────────────────────────────────────────────────────────────────
// Fetched once per page and reused by every window's editor; rebuilding 41 chips per panel render
// is cheap, another HTTP round trip per open is not.
let _vvRyShares = null;
async function _vvRyLoadShares() {
if (_vvRyShares) return _vvRyShares;
try {
const d = await (await fetch('/plugins/varaverk/api/rsync.php?action=shares')).json();
_vvRyShares = d.ok ? (d.shares || []) : [];
} catch (e) { _vvRyShares = []; }
return _vvRyShares;
}
// A share is "in" only when this window holds its exact path. A window holding a SUBPATH of it —
// /mnt/user/Media_Server/Emby against the Media_Server share — is marked partial instead, because
// adding the parent would silently widen a deliberately narrow entry.
function _vvRyChipState(share, entries) {
const p = (share.path || '').replace(/\/+$/, '');
for (const e of entries) {
const ep = (e.path || '').replace(/\/+$/, '');
if (ep === p) return 'in';
if (ep.startsWith(p + '/')) return 'part';
}
return '';
}
async function vvRyRenderChips(key) {
const host = document.getElementById('vv-ry-echips-' + key);
if (!host) return;
const shares = await _vvRyLoadShares();
const entries = (_vvRyWinEState[key] && _vvRyWinEState[key].shares) || [];
if (!shares.length) { host.innerHTML = '<span style="font-size:10px;color:#444;">no shares found</span>'; return; }
host.innerHTML = shares.map(function (s) {
const st = _vvRyChipState(s, entries);
const cls = 'vv-ry-chip' + (st ? ' ' + st : '') + (s.exists ? '' : ' gone');
const tip = st === 'in' ? 'already in this window'
: st === 'part' ? 'a subpath of this share is already here — adding the whole share would widen it'
: (s.comment || s.path) + (s.exists ? '' : ' (declared, but no directory yet)');
return '<span class="' + cls + '" data-path="' + vvEscAttr(s.path) + '" data-state="' + st + '" '
+ 'title="' + vvEscAttr(tip) + '">' + vvEscHtml(s.name) + '</span>';
}).join('');
// Delegated, and bound once — the chips are rebuilt on every render.
if (!host.dataset.bound) {
host.dataset.bound = '1';
host.addEventListener('click', function (ev) {
const c = ev.target.closest('.vv-ry-chip');
if (!c || c.dataset.state === 'in') return;
const pathEl = document.getElementById('vv-ry-epath-' + key);
if (pathEl) pathEl.value = c.dataset.path;
// Routed through the existing add so profile handling, validation and the state shape stay
// in exactly one place.
vvRyEAddShare(key);
});
}
}
function vvRyEAddShare(key) {
const path = document.getElementById('vv-ry-epath-' + key)?.value.trim();
const prof = document.getElementById('vv-ry-eprof-' + key)?.value.trim() || '';
@@ -1499,7 +1582,7 @@ function vvRpLoad() {
_vvRpFillForm(_vvRpSelected);
}
})
.catch(() => {});
.catch(e => vvFetchErr('rsync profiles', e));
}
function _vvRpPopulateSelector() {
@@ -1656,7 +1739,7 @@ function vvMsStop() {
fetch('/plugins/varaverk/api/manual_sync.php', { method: 'POST', body: fd })
.then(r => r.json())
.then(() => {})
.catch(() => {})
.catch(e => vvFetchErr('manual sync start', e))
.finally(() => { btn.textContent = '■ Stop'; btn.disabled = false; });
}
@@ -2018,7 +2101,7 @@ function _vvMsStartPoll(token, btn, stat, out, badge, stopBtn) {
setTimeout(() => { badge.style.display = 'none'; stat.textContent = ''; }, 6000);
}
})
.catch(() => {});
.catch(e => vvFetchErr('manual sync poll', e));
}, 1500);
}
@@ -2039,7 +2122,7 @@ function vvMsLoadRecent() {
fetch(_MS_API + '?action=recent')
.then(r => r.json())
.then(d => { _vvMsRecent = (d && d.rows) || []; _msRenderRecent(); })
.catch(() => {});
.catch(e => vvFetchErr('recent syncs', e));
}
function _msRenderRecent() {
@@ -2098,7 +2181,7 @@ document.addEventListener('click', async ev => {
body: new URLSearchParams(body) })
.then(r => r.json())
.then(() => vvMsLoadRecent())
.catch(() => {});
.catch(e => vvFetchErr('manual sync action', e));
});
// Fills the form and stops. Deliberately does not run: one click away from a transfer that may
+37 -16
View File
@@ -126,7 +126,7 @@ try {
if (in_array($_parts[0], $_repoSkip)) continue;
$_repoFiles[] = $_rel;
}
} catch (Exception $_re) {}
} catch (Exception $_re) { vv_log_error('pages/scheduler.php', 'repo file walk failed: ' . $_re->getMessage()); }
sort($_repoFiles);
// Docs tree: README and Manual with their per-module children
@@ -679,7 +679,7 @@ $runningScripts = array_unique($runningScripts);
<div class="vv-sug-body">
<pre class="vv-readme-body">Not enterprise HA. No SLAs, no quorum nodes, no guaranteed uptime don't put your billing system on this.
What it is: mutual automatic failover between two independent unRAID servers. When one goes down the other starts its containers, cuts over DNS, and keeps users online. When it comes back, everything hands back in the correct sequence DDNS stops, containers stop, rsync writeback runs, containers start on the primary, primary DDNS starts last so users hit the returning server only after it's actually ready.
What it is: mutual automatic fallback between two independent unRAID servers. When one goes down the other starts its containers, cuts over DNS, and keeps users online. When it comes back, everything hands back in the correct sequence DDNS stops, containers stop, rsync writeback runs, containers start on the primary, primary DDNS starts last so users hit the returning server only after it's actually ready.
Built from scratch. Refined through a year of production testing. The DDNS sequencing and handback order were the hardest parts to get right. Both directions are exercised regularly with fallback_test.sh.</pre>
</div>
@@ -1567,7 +1567,7 @@ function vvPollStatus() {
}
vvRunningSet = nowRunning;
})
.catch(() => {});
.catch(e => vvFetchErr('job status', e));
}
function vvStartStatusPoll() {
@@ -2785,7 +2785,7 @@ function vvBoardPoll() {
vvUpdateErrors(d.errors ?? []);
vvUpdatePartner(d.partner);
})
.catch(() => {});
.catch(e => vvFetchErr('script board', e));
}
function vvUpdateLocks(locks) {
@@ -4661,21 +4661,42 @@ async function vvSaveArrange() {
if (!arrayMap.has(primaryArray)) arrayMap.set(primaryArray, []);
});
let failed = false;
const fail = () => {
btn.textContent = 'Error!';
btn.style.color = '#f44336';
setTimeout(() => { btn.textContent = 'Save Arrangement'; btn.disabled = false; btn.style.color = ''; }, 2500);
};
// Phase 1 — relocations, one atomic call each.
//
// A cross-array move changes two arrays. Expressing it as two reorderarray.php calls means
// a failure between them leaves master.conf half-written: the script removed from its old
// orchestrator and never added to the new one, or present in both and running twice. There
// is no rollback, and the page still shows the intended arrangement, so the operator re-drags
// from a view that no longer matches the file.
//
// movescript.php does the whole relocation inside one guarded write, so it either happens or
// it does not. Phase 2 can then only get the ordering wrong, never the membership.
const moves = new Map(); // script → final array ('' = out of every orchestrator)
for (const p of vvArrangePending) {
if (p.fromArray === p.toArray) continue; // pure reorder — phase 2 owns it
moves.set(p.script, p.toArray || '');
}
for (const [script, toArray] of moves) {
const r = await vvPost('/plugins/varaverk/api/movescript.php', { script, to_array: toArray })
.catch(() => ({ ok: false }));
if (!r.ok) return fail();
}
// Phase 2 — order and enabled state, per array. Membership is already correct.
for (const [arrayName, scripts] of arrayMap) {
const r = await vvPost('/plugins/varaverk/api/reorderarray.php', {
array_name: arrayName,
scripts: JSON.stringify(scripts)
}).then(r => r.json()).catch(() => ({ ok: false }));
if (!r.ok) { failed = true; break; }
}).catch(() => ({ ok: false }));
if (!r.ok) return fail();
}
if (failed) {
btn.textContent = 'Error!';
btn.style.color = '#f44336';
setTimeout(() => { btn.textContent = 'Save Arrangement'; btn.disabled = false; btn.style.color = ''; }, 2500);
return;
}
location.reload();
}
@@ -4905,7 +4926,7 @@ async function _vvFolderDrop(e) {
const folders = vvGetCurrentFolders();
const r = await vvPost('/plugins/varaverk/api/savefolders.php', {
folders: JSON.stringify(folders)
}).then(r => r.json()).catch(() => ({ ok: false }));
}).catch(() => ({ ok: false }));
if (!r.ok) {
oldParent.appendChild(srcEl);
@@ -4967,7 +4988,7 @@ function vvPollSnapshot() {
const s = (d.fallback ?? 'UNKNOWN').toUpperCase();
fb.textContent = s;
fb.style.color = s === 'NORMAL' ? '#4caf50'
: s === 'FAILOVER' ? '#f44336'
: s === 'FALLBACK' ? '#f44336' // never 'FAILOVER' — see fallback.sh state_set
: s === 'NO_INTERNET' ? '#ff9800' : '#666';
}
const pt = document.getElementById('vv-snap-partner');
@@ -5062,7 +5083,7 @@ async function _vvDoCreateFolder(name, wrap) {
folders[name] = [];
const r = await vvPost('/plugins/varaverk/api/savefolders.php', {
folders: JSON.stringify(folders)
}).then(r => r.json()).catch(() => ({ ok: false }));
}).catch(() => ({ ok: false }));
if (!r.ok) { vvAlert('Failed to create folder.'); return; }
wrap.remove();
// Add folder group to DOM
+64 -1
View File
@@ -37,6 +37,12 @@
// Step 2: auto-populate + guide + checklist.
// master.conf pull (for partner servers) lives in the checklist, not here.
// The assistant is a bonus on this page, never a dependency: vv_ai_ui_on() is false on a node
// with AI off or no model reachable, which is the normal state of the fresh install this wizard
// exists to serve. The card simply is not rendered and setup proceeds exactly as before.
require_once dirname(__DIR__) . '/include/ai_chat.php';
if (vv_ai_ui_on()) vv_ai_chat_assets();
$detectedHostname = vv_get_hostname();
// ── Identity already on disk? ────────────────────────────────────────────────────────────────
@@ -313,6 +319,21 @@ hr.vv-hr { border: none; border-top: 1px solid #1e1e1e; margin: 22px 0; }
<div id="vv-onboard-panel"></div>
<div id="vv-done-banner"></div>
<?php if (vv_ai_ui_on()): ?>
<hr class="vv-hr">
<div id="vv-su-ai-card">
<?php vv_ai_chat_markup('vv-su-ai', [
'profile' => 'varaverk',
'compact' => true,
'title' => 'Setup assistant',
'height' => '260px',
'tall' => '460px',
'empty' => 'Stuck on a step? Ask what it wants and why — "what is the API key for", '
. '"why does SSH need the other host", "what does populate actually do".',
]); ?>
</div>
<?php endif; ?>
<div style="margin-top:18px;text-align:right;">
<a href="#" id="vv-exit-link" onclick="vvGoNext(event)"
style="font-size:11px;color:#3a3a3a;text-decoration:none;">Skip </a>
@@ -443,6 +464,7 @@ function vvShowStep2(redirect, apiKey) {
_vvRedirect = redirect || '?tab=scheduler';
document.getElementById('vv-step1').style.display = 'none';
document.getElementById('vv-step2').style.display = 'block';
vvSetupAiInit(); // card is visible now, so it can measure itself correctly
if (apiKey && apiKey.ok) {
const btn = document.getElementById('vv-key-btn');
const status = document.getElementById('vv-key-status');
@@ -662,7 +684,7 @@ function vvStartPhase2Watch(sinceMs) {
const done = (d.items || []).some(i => i.id === 'partnership' && i.ok);
if (done || Date.now() > deadline) stop();
})
.catch(() => {});
.catch(e => vvFetchErr('setup checklist', e));
};
// Redraw between checklist polls so the elapsed counter moves. Without it the panel is static
@@ -676,6 +698,47 @@ function vvStartPhase2Watch(sinceMs) {
_vvPartnerPoll = setInterval(check, 6000);
}
// ── Setup assistant ───────────────────────────────────────────────────────────
// Scoped to the step the operator is actually stuck on, so "what does this want" resolves without
// them naming it. The worker turns a non-empty scope into a line saying what is open in the
// WebGUI; the varaverk profile holds no scoped_log capability, so a checklist id never triggers a
// log lookup and cannot produce a "log missing" note for something that was never a script.
//
// Re-read at send time, not captured: the checklist re-polls while the page is open and the step
// they are on can change between opening the composer and pressing Ask.
//
// First not-ok, not-deferred item — deferred means "I have decided to skip this", which is not
// where they are stuck. Falls back to the bare page when the list is clean or has not loaded.
function vvSetupScope() {
const d = _vvLastChecklist;
if (!d || !Array.isArray(d.items)) return 'Setup';
const stuck = d.items.find(i => !i.ok && !i.deferred);
return stuck && stuck.id ? 'Setup/' + stuck.id : 'Setup';
}
// Mounted when Step 2 is revealed, not at parse time. This is the only page where the card starts
// inside a display:none block, and a chat that measured itself while hidden would come up wrong
// with nothing to correct it — the component carries no resize observer. Idempotent because
// vvShowStep2 is reachable more than once.
let _vvSetupAiUp = false;
function vvSetupAiInit() {
if (_vvSetupAiUp) return;
if (typeof VvAiChat !== 'function') return;
if (!document.getElementById('vv-su-ai-chat')) return; // AI off — card was never rendered
_vvSetupAiUp = true;
VvAiChat({
prefix: 'vv-su-ai',
profile: 'varaverk',
// Pinned, for the same reason the Partnership card pins it: resuming whatever thread was last
// touched anywhere in the UI could land a first-run operator mid-way through someone else's
// Scheduler conversation, on a card with no picker to get back from.
resumeProfile: 'varaverk',
scope: vvSetupScope,
scopeLabel: 'Setup',
empty: 'Stuck on a step? Ask what it wants and why.',
});
}
function vvLoadChecklist() {
fetch('/plugins/varaverk/api/checklist.php?_=' + Date.now())
.then(r => r.json()).then(d => {
+19
View File
@@ -35,6 +35,25 @@
# progress in the UI, and lets it be re-run or cancelled without touching onboard.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# One share at a time, in DAILY_SYNC_SHARES order, each handed to Rsync/rsync.sh --seed. The
# transfer itself, its bandwidth cap and its resume behaviour are all rsync.sh's — this script
# decides what to seed and in what order, never how.
#
# Sequential on purpose. The cap that matters is DEFAULT_RSYNC_OPTS' --bwlimit, which is a limit
# per rsync rather than per host; running shares in parallel would multiply it by the number of
# shares and saturate the link the rest of the ecosystem is still using.
#
# Refuses before it starts rather than partway. RSYNC_ENABLED must be true and DAILY_SYNC_SHARES
# must be non-empty, both checked up front — a multi-week transfer is the wrong place to
# discover that the global gate was off.
#
# Nothing waits on this. Phase 3 runs after the partnership is already established, so a seed
# that is still copying weeks later blocks no flag, no status card and no job record.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#

Some files were not shown because too many files have changed in this diff Show More