Deploying first meant Authelia, Lldap, NPM and both databases initialised themselves against empty directories. The gate arming moves to Step 1d for the same reason: the provisioning sync is an rsync, and Tier 1 stops every rsync. Tier 2 now ends the onboard off rather than as-found.
Step 1 printed 'Rsync stopped' over a file-not-found, and the closing block asserted a clean separation and a synced auth config regardless of what happened.
A first seed is ~28 TB behind a 12.5 MB/s bwlimit, and it ran inline as Step 9d, so the
phase-2 flag every status reader depends on was written only after it finished.
Phase 1 exists to hand the partner its identity before the install, but the push refused any
host without varaverk.cfg — so the conf could only ever reach a node that no longer needed it
to be told who it was. It now resolves the remote's conf directory across both install layouts
and creates the internal one when neither exists.
Both checks already answered their question on demand and both needed
somebody to press a button on the right row. One host here has returned
nothing but 5xx for months.
Filed as findings, which is the existing answer to a condition that
persists while nobody is looking. Grouped by cause rather than by
hostname: a default policy of bypass produced twenty-two findings that
were one sentence repeated, and they have one fix between them.
A low uptime figure, a refused login and a certificate that stopped
renewing all looked the same from the row: a number, with the reason
split across NPM, an Authelia config and the directory.
The why-check goes and looks — TCP to the forward target, HTTP through
the proxy, a second handshake with verification off to tell a broken
certificate from a broken service. Forward hosts are docker names that
only resolve on NPM's network, so an unresolvable one is redirected to
the container address and the substitution is reported; a check that
could not be made must never read as a check that failed.
The access simulator walks the rules the way Authelia does and shows the
ones it stepped over, reading whichever instance the chosen host points
at rather than the one conf names — there are two here.
Cert triage counts runs rather than log lines and orders by rotation
suffix rather than mtime, both of which change the answer.
NPM's access log only describes hosts somebody visited; the host most likely to be quietly broken
is the one nobody does. Probes carry a User-Agent npm_access_stats.sh drops — unmarked, this
monitor would be fifty thousand requests a day in the very logs it reports on.
NPM writes an access log per host and counts nothing, so 475 MB of logs held the only answer to
"is anything using this". Aggregated on a schedule and read from a few kB of JSON; the row also
now says whether an auth_request block is in front of the site, which nothing showed before.
Saving a host sent advanced_config as an empty string and reset http2, both HSTS flags, meta,
locations and enabled — so changing a port removed the auth_request block that puts Authelia in
front of the site. Twenty-five of the thirty-five hosts here carry one.
NPM knows what a certificate is today and nothing about what it was, so ten of them could fail
renewal for months — 1001 certbot runs, zero successes — without anything on any page saying so.
Counts start at zero and are only ever observed; only first_seen is seeded, from NPM's own date.
NPM answers a successful DELETE with the bare literal true, which vv_npm_raw() is typed to
return an array from — so deleting or toggling a proxy host has always thrown a TypeError
rather than doing the thing.
Authentik is the likely destination and the page had Authelia and lldap wired in at every
level, so the seam goes in now: the panels and every endpoint action route off one conf value,
and a stack that cannot be driven yet says so rather than drawing controls with nothing behind.
Opening the editor to fix a name and then having to close it to reach a password or a group
was three dialogs for one sitting; the row buttons stay for one-click access from the list.
lldap lets you edit five things about a user and one about a group; the page reached two of
them, so correcting a surname or a group's name still meant opening the container's own WebUI.
The block is rebuilt from the parsed model on every save and the parser discarded comments,
so one press of Save deleted the five ## lines that are the only thing in the file saying
what each rule is for — which inline editing was about to make far easier to press.
Most rules carry thirteen or fourteen domains, which in one table cell was a four-hundred
character run that answered no question at a glance; the group is what you are looking for
and it is now the heading.
The value was written as typed, bash expanded it when the read-back sourced the file, and
the guard rolled the whole write back with nothing on screen but "save failed" — which is
also what an empty value, a trailing space, and a stale API-key check had been doing.
Not set, rejected and unreachable all reached the page as one message about checking
credentials, which sends you to a password when the field is simply empty — as both of
HOST1's were, with no card on the page to fill them in from.
vv_is_ai_host() was `=== 'host1'`, which made a physical fact — that is where the
GPU is — look like a rule. AI_OWNER_HOST declares it, so the card can move to a
rebuilt host3 or a friend's spare.
The gate was also doing two jobs. Assistant docks and findings strips now ask
whether a model is reachable, so a node without a GPU gets them by borrowing;
the AI tab asks whether this is the owner, because that page carries the bug
reports, the index and the model configuration — the surface where the
vocabulary assumes you built the mesh.
Resolution is local, then owner, then anyone else declaring a model, pinned once
it answers. Pinned rather than re-derived per call: a mesh that re-decides every
request eventually decides differently mid-conversation, and a chat whose second
turn lands on another machine has no history there. Cleared only on a transport
failure, and only when there is somewhere else to go — a single-node mesh whose
model is down should say so, not report AI as unconfigured.
Identity now includes the host: "Bazarr is on the skip list" is a different fact
on each machine that says it, and without the host in the hash the second node
to report a subject would land on the first node's record — one machine's
dismissal silencing another's live fault.
Nine dismissed findings existed when this changed, so the write path migrates a
legacy id once, at the only moment it matters. Verified: a dismissal survives the
rename rather than resurfacing as new.
acted_by is stamped in apply_action, not set_state — the sweep calls set_state to
move a candidate to needs_operator, and crediting a machine for classifying is
not the same as recording who decided.
The five AI_ASSIST_ switches and the findings they produce were reachable only
from the AI tab, which is a long way from the page a finding is about.
Nothing was synchronised because nothing needed to be: findings are one file per
finding, and every surface is a view over that store with actions going to the
same endpoint. Acting on the Watchdog tab shows on the AI tab because they are
not two copies. The strip reloads rather than editing its own row, which is the
only way they could have drifted apart.
Each strip shows one page's kinds. Actions are whatever the server offers for
that row, so Move appears on media findings without this card knowing what a
move is.
Declared in PHP rather than in each bash header, against scriptinfo's usual rule
that the header next to the code is authoritative: it is authoritative about what
the script does, and an enhancement is something else reading its output, gated
by a flag the script has never heard of. A description of PHP inside a file that
cannot enforce it would drift the first time either changed.
Shown with its switch, so "there is an enhancement" and "it is running" are never
the same claim. Discovery is listed even though nothing acts on its output — it
takes the first accessible root folder with no regard for content, and the
classification scan is what notices that night. That relationship explains where
misfiled series come from and was written down nowhere.
The triage could say Narcos is filed under anime and offer nothing to do about
it. Findings that recorded a destination now carry a Move action, which the AI
tab renders as a button on its own — actions are read from the server, so no UI
knew this existed. Armed like Fix and Dismiss, because it is the only action in
that card that relocates files rather than settings and the undo is another move.
Offered only where the triage said misfiled, never for uncertain: three
narrowings before a file is touched. Re-reads the series from Sonarr first, so a
finding gone stale cannot undo a move made by hand in the meantime.
The scan's own header calls reverse-anime leaks "genuine judgment calls" and
reports them as a count. Seventeen of them concealed two live-action crime
dramas filed under anime for as long as nobody read the list. The scan now
persists that bucket as its own verdict, and the sweep asks the model to sort
only those eleven — not the eleven hundred it already classifies correctly.
Files findings for misfiled and uncertain only; donghua and anime-adjacent are
answers, not problems, and filing them would rebuild the undifferentiated list
this exists to break up. Moves nothing — --move stays a flag a human types.
The grid said repeat(5,1fr) against six windows, so Fallback sat alone on a
second row. The count now travels with the cards as --vv-win-n, so adding a
window widens the row instead of quietly starting another one. Grid rather than
flex keeps every card exactly the same width, and it steps 6→3→2 rather than
dropping straight to two — the middle step is the one a 15" panel lands on.
It covers the pipeline rather than only acquisition: the arrs, the jobs that run
over the same files, and the servers that play them. The label moves; the tab id,
file and URL do not, so bookmarks and the cache key keep working.
The settings card widens with it — Play State Sync, Media Permissions, Media
Cleaner, Emby and Jellyfin were reachable only through the Settings catch-all,
which is a long way to go for a setting named after the page you are on. Match
is whole-word, so "Intermediate Sync" is not caught by "media"; section count is
unchanged at 126 with the same 5 gaps, so nothing was taken from another tab.
/proc/net/dev counts octets; the field is named rx_bps, vvFmtBps renders Kb/s
and Mb/s, and the Monitor card prints it directly under the NIC link speed from
/sys/class/net/*/speed, which genuinely is megabits. So the one figure you would
read against the link understated traffic eightfold — 811 KB/s shown as "0.8
Mb/s" beside a 10 Gb/s link when it was 6.5. Converted at the source so the
field name stops lying; both consumers are bit-labelled or relative.
PHP defaults to UTC on Unraid while every bash script stamps local time, and the
two write into the same files — conf_changes.log was four hours out from every
log you would correlate it against. The parsing half was worse: vv_ai_syslog_ts()
reads local-time syslog lines through strtotime() under UTC, landing every event
four hours early, and the repair sweep bounds its scan to "since the last pass"
— so a fault that had just happened could read as four hours old and fall
outside the window. Also fixes date-string comparisons against bandwidth and
cleanup dbs, which bash writes with local dates.
The page multiplies its core count by the RW_LOAD_* multipliers to colour the
load bar, and resource_watchdog.sh computes its real thresholds from nproc. The
page preferred the API's physical core count — 16 against 32 threads — so it
turned amber at load 32 and red at 48 while the watchdog did not reach soft
pressure until 64 or medium until 96. A page whose whole job is reporting what
the watchdogs think was reporting a crisis about a machine they considered idle.
The remote path already sent the partner's own nproc, so both nodes now agree.
docker stats sums CPU across every logical processor, so Jellyfin's "227%" is
2.3 threads busy — 7% of a 32-thread host, which is what Unraid shows and what
the operator was reading. The card was printing the raw figure and colouring it
against thresholds meant for a percentage, so ordinary background work looked
like an emergency; I twice flagged Jellyfin as pegged on that basis and was
wrong both times. Divides by nproc, not the watchdog's `cores` — that field is
the physical 16 because it compares load averages, and borrowing it here would
have looked reasonable and doubled every reading.
A partner's Emby URL is http://localhost:8096 — true there, meaningless here —
so nobody queries a partner's media server directly. Each host reports its own
over the SSH call remote_arr_cache_writer.sh already makes, which keeps its API
keys on it and costs no extra connection. Local is the default: opening the tab
to check your own server should not first make you read past a partner's.
A partner with no cache is stated as not collected rather than omitted, because
an empty mesh view and a partner that is down look identical otherwise.
The counts need the entire library downloaded to compute six numbers — 16.0 MB
from Radarr and 5.5 MB from Sonarr, measured — and they were on the same
one-minute clock as queue depth and health, which pulled ~30 GB a day out of the
arrs to re-count records that had not changed. They now cache for fifteen
minutes against how often the numbers actually move: 7.6s to 3.0s per refresh.
Emby and Jellyfin get a card each: version, CPU, memory, uptime, streams, users
and transcodes, with update/restart flags. Two halves of one question — a server
answering happily at 145% CPU is a different situation from one at 8%, and
neither the app nor the container says so alone.
The media jobs row I just added formats durations with _dur(), which the rsync
page defines locally and js/varaverk.js does not export at all. The reference
threw inside _render(), the fetch's catch reported it as "Error loading arrs
status", and the whole page went blank — nothing to do with the API, which was
returning valid JSON throughout.
play_state_sync, media_shares_permissions and media_cleaner work on the same
files the arrs manage and were visible only by opening the Scheduler and reading
an orchestrator's log. Readable at all because run_orch_child() now writes a run
record and a per-script log for its children — this could not have been written
yesterday. Each card shows the sentence the script itself ended on rather than a
count re-derived here, since the three word their outcome differently and the
wording is the part worth reading.
Everything a child printed was interleaved into the parent's log with forty
other scripts, separated only by human headings that map to no script id — so
scoped_log could answer for ten orchestrators and nothing else. tee rather than
a redirect, so the parent keeps receiving exactly what it did before and this
only adds a destination; PIPESTATUS[0] preserves the child's exit code through
the pipe. Trimmed to 1000 lines like run_job.sh does, because LOG_DIR is tmpfs
and an untrimmed chatty script spends RAM. Falls back to the original unpiped
call when the log cannot be opened — a child must never be denied a run because
somewhere to write about it could not be created.
run_job.sh records the ten jobs cron starts; the seventy entries inside the
*_SCRIPTS lists had none, and nothing said so. Consumers quietly got less — arr
cleanup stats fell through to a daily aggregate with no end time, and
troubleshoot answered "I cannot see a run record" for every script that is not
an orchestrator, which is nearly all of them and exactly the ones a question is
about. run_orch_child() already had the start time, exit code and severity
ladder; it just never wrote them down. Not run_job.sh itself: that also locks
and interval-gates, which would let a child skip itself or block on its parent.
The arr cleanups are entries in DAILY_MAINTENANCE_SCRIPTS, which the orchestrator
invokes with plain bash, so they never get the run record the primary parse needs
— every payload has come from the daily aggregate db. That path read every column
except the byte counts, leaving orphans_sz at its "0B" default, which is
invisible at zero orphans and would have read "12 orphans (0B)" the first time
there were any. 'total' had no source there and no reader anywhere, so it is gone.
Arr Sync now reads "disabled" rather than "never run" when its switch is off.
Health messages go straight from each arr's API into innerHTML and quote things
nobody here controls — indexer names from Prowlarr, import-list names, release
titles. The page used no escaping at all. _sz() was also binary while labelling
its output TB, so /tv read 86.9 TB here and 95.6 TB on every other tab; it has
one caller and that caller is disk capacity, so it now matches the rest.
Retyping two paths and nine checkboxes correctly every time is where the
mistakes come from, and "one-off" described how a transfer is scheduled rather
than how often it is run. Pinned entries are exempt from the rotation — the
command used twice a year is both the most worth keeping and the first that ten
ordinary runs evict. Stored server-side, so the list is there from any screen,
and loading one fills the form and stops rather than running it.
--stats prints "Total transferred file size: 1,234,567 bytes", so $NF is the
unit word; stripping non-digits from it left an empty string and ""+0 is 0.
Every transfer ever logged recorded zero. Nothing looked broken — the history
filled with 1,423 correct rows all reading 0, the graph drew a flat line, and
DIGEST_SMART_ON_BANDWIDTH could never fire because 0 is never above 50GB.
Both parse sites now share one program, since fixing either alone would have
left the other silently wrong.
The preview ended in the literal text "…src host:dst", so the one thing worth
checking before pressing Run — the real source and destination, and which of
them has the trailing slash — was the one thing never shown. --delete sat
unmarked among nine checkboxes with no confirmation on a card that can remove
data on the far host. Confirm fires only for --delete without --dry-run, so it
stays worth reading.
last_sync reads the maintenance orchestrators' run records, and those do git
pull, permissions, cleaners and docker updates on schedule whether or not the
global gate is open — so the page reported a healthy 56-minute "daily sync"
while nothing had transferred since 16 July. Both cards also compared status
against 'success' while being fed 'ok', so every clean run rendered in the
failure colour. Adds the assistant, scoped to the tab.
Every existing exemption lasts until someone remembers to undo it, and nobody
does — Healarr has sat in a pressure list since it was uninstalled and seven
ignore entries name containers that are gone. A mute states when it ends and
then ends, capped by WATCHDOG_MUTE_MAX_HOURS so temporary is enforced rather
than intended. Applied where IGNORE_MAP is built, so all five check sites
inherit it, and shown with its countdown because an invisible suppression is
the thing being fixed.
Without it the page renders zeros, all-clear pills and green dots when the
orchestrator has stopped entirely — its healthiest appearance describing a
machine with no watchdogs, against a header promising absence is never drawn as
an all-clear. The heartbeat already existed; overdue is measured against the
cron schedule rather than an assumed fifteen minutes.
Gmer4Lfe