Both checks already answered their question on demand and both needed
somebody to press a button on the right row. One host here has returned
nothing but 5xx for months.
Filed as findings, which is the existing answer to a condition that
persists while nobody is looking. Grouped by cause rather than by
hostname: a default policy of bypass produced twenty-two findings that
were one sentence repeated, and they have one fix between them.
NPM's access log only describes hosts somebody visited; the host most likely to be quietly broken
is the one nobody does. Probes carry a User-Agent npm_access_stats.sh drops — unmarked, this
monitor would be fifty thousand requests a day in the very logs it reports on.
NPM writes an access log per host and counts nothing, so 475 MB of logs held the only answer to
"is anything using this". Aggregated on a schedule and read from a few kB of JSON; the row also
now says whether an auth_request block is in front of the site, which nothing showed before.
NPM knows what a certificate is today and nothing about what it was, so ten of them could fail
renewal for months — 1001 certbot runs, zero successes — without anything on any page saying so.
Counts start at zero and are only ever observed; only first_seen is seeded, from NPM's own date.
Authentik is the likely destination and the page had Authelia and lldap wired in at every
level, so the seam goes in now: the panels and every endpoint action route off one conf value,
and a stack that cannot be driven yet says so rather than drawing controls with nothing behind.
vv_is_ai_host() was `=== 'host1'`, which made a physical fact — that is where the
GPU is — look like a rule. AI_OWNER_HOST declares it, so the card can move to a
rebuilt host3 or a friend's spare.
The gate was also doing two jobs. Assistant docks and findings strips now ask
whether a model is reachable, so a node without a GPU gets them by borrowing;
the AI tab asks whether this is the owner, because that page carries the bug
reports, the index and the model configuration — the surface where the
vocabulary assumes you built the mesh.
Resolution is local, then owner, then anyone else declaring a model, pinned once
it answers. Pinned rather than re-derived per call: a mesh that re-decides every
request eventually decides differently mid-conversation, and a chat whose second
turn lands on another machine has no history there. Cleared only on a transport
failure, and only when there is somewhere else to go — a single-node mesh whose
model is down should say so, not report AI as unconfigured.
Every existing exemption lasts until someone remembers to undo it, and nobody
does — Healarr has sat in a pressure list since it was uninstalled and seven
ignore entries name containers that are gone. A mute states when it ends and
then ends, capped by WATCHDOG_MUTE_MAX_HOURS so temporary is enforced rather
than intended. Applied where IGNORE_MAP is built, so all five check sites
inherit it, and shown with its countdown because an invisible suppression is
the thing being fixed.
A fourth candidate source beside the arrs, the system log and container logs:
skip-listed containers, strikes past their limit, repeated restarts, unattended
reboots and sustained pressure become findings. The kind is deliberately not
conf-bound, so it cannot autofix by construction rather than by a switch — and
WATCHDOG_SCAN_IGNORE suppresses it, so a knowingly broken container stays quiet.
First consumer AI_ASSIST_WATCHDOG has ever had.
The operator already runs it, so this is the one provider that costs nothing to adopt: no key,
no card, no third party, and no new container. It aggregates several engines and returns them
merged, and its /api/search reply is already the shape everything downstream expects — title,
url, snippet, with content as a fallback for the roughly one result in six that has no snippet.
It is also now the only provider whose response has been read from a live service rather than
from documentation, which is why it is the default.
Search is the one capability that fits the profile holding none: every other capability either
reads this installation or changes it, and chat's whole contract is that it can do neither. The
assistant deliberately does not get it — its contract is that answers come from this install's
own documents, and a web result there is an answer that looks sourced and is not. A chat question
about this machine hands off to the assistant before the search would run, so it never reaches
the internet even with the box ticked.
Off by default, and not because it is dangerous: searching sends the operator's question outside
the house, which is theirs to decide. Asked for per turn as well as enabled in conf.
Provider-agnostic, as asked — searxng, brave, tavily. Only whichever is configured here can be
verified; all three read every field by name, so a shape that changes underneath yields no
results rather than wrong ones.
The explain fixtures asserted chat holds no capabilities at all, which is exactly the guarantee
worth keeping. caps=only: now states the set rather than its emptiness, so the check still fails
the day something else is granted there.
Triage could see what Varaverk wrote about itself and what the arrs said about themselves, and
nothing else — so a disk throwing I/O errors, a filesystem going read-only or a PCIe link
retraining every two minutes was invisible to the thing whose job is noticing. Container state
was already watched; no line any container actually wrote ever was.
Container patterns match the environment rather than the application. Fifty containers are fifty
programs with no shared vocabulary for malfunctioning, but an exact shared one for a full disk or
a corrupt SQLite file, because those strings come from libc and SQLite rather than from the app.
Both halves are checked by Tools/ai_log_check.sh, which is two tests because the failure modes
are opposite: fixtures for recall on faults this host has never had, and a replay of its real
logs for precision — 74,519 syslog lines and 79,193 container lines, matching only the PCIe
errors it genuinely has.
Severity was being graded from a hand-picked three fields, so every one of these would have been
filed as a warning however bad it was, and notified as one.
A finding nobody is told about is a finding nobody has, and the card added earlier only shows
them to someone who opens the tab. Only needs_operator is announced — an open finding may still
be repaired by the next pass — one notification for all of them, and each is announced once and
stays quiet until the fault changes or gets worse.
vv_notify() hands the message to common.sh's notify() rather than reimplementing the channels,
and calls detect_hosts() explicitly because load_config.sh deliberately does not: without it the
Unraid notification arrives and Discord silently never does. It also reports false when no
channel is switched on at all, since notify() exits 0 either way and a caller believing that
would mark a finding as told and never mention it again.
Notification text is folded to ASCII. Unraid's notifier dropped an em dash outright and left the
double space behind, which was found by sending one and reading what arrived.
Nine entries named paths nothing lives at — a directory that was renamed to System_Essentials,
and docker_update_remaining.sh which has never existed — and the API key renewal was absent
from both arrays it runs from on this host. A fresh install would have skipped all of it.
Nothing in Varaverk fires when a job ends, so this picks up run records that completed
since the last pass rather than adding a hook to forty scripts. Ahead of stability in the
cycle on purpose: a wrong port is not fixed by rebooting the machine.
A model can be confident a port should be 8686; a probe can report that 8686 answered, and
only the second is a fact. Detecting and repairing are also separate things to trust, so
the feature runs read-only until the write switch is turned on under it.
A bug stays open until the code changes and nothing here can close it; a finding closes
itself when the probe that proved the fault starts passing, so the two cannot share a
store. Acknowledging one is scoped to the value it was acknowledged at.
These files are gitignored, so a bad write had nothing to go back to; the escaping order
also stored any value containing a quote truncated but still parseable, which only a
read-back can catch.
The pull substituted only HOSTN_, so bare HOSTN in comments kept tripping
conf_upgrade's own guard and no host conf had upgraded since the guard landed.
AI_INDEX_ON_PULL was designed, defaulted false, and never read by anything —
so nothing has ever rebuilt the index automatically. A pull is the only thing
that changes tracked files on a server, which makes it the only moment the
index can go stale, and a timer would do nothing 23 times a day while still
drifting from the pull that matters.
Staleness is invisible in the answers: the index keeps returning the old text
with confident, correct-looking citations. Gated on pull success, on
AI_INDEX_ON_PULL and on AI_ENABLED, and never fatal — a git pull must not fail
because an embedding call timed out.
A small operator-written file handed to the model at the start of every
conversation — who you are, how this install is set up, what has already been
decided. Injected ahead of the retrieved passages and marked operator-authored
so it outranks anything they contradict, and never cited as a source.
Deliberately not indexed and deliberately under DATA_DIR: it changes
constantly, vector similarity is the wrong way to retrieve things you were
told to remember, and gitignoring it keeps personal notes out of a pushed
repository. The character cap is a context budget — this text costs its share
of 16k on every single turn.
Open-WebUI runs in its own container with no WebGUI session, and nginx applies
auth_request to everything it serves, so it cannot call the plugin API. Same
shape as the arr webhook listener: node outside nginx, shared secret, one
read-only verb. Serves chunks rather than answers because the caller already
has a model loaded.
Merging the raw host.conf.template against a live host conf matched HOSTN_
against HOST1_, classified all 140 real keys as deprecated and would have
removed every credential on the host. Also points the Ollama model default
at a tag that still exists.
ai_index.sh and ai_query.sh follow the usual conventions — fail-closed gate,
root check, lock, dry-run, status — with Node doing only the vector maths and
SQLite blobs, the same split api_cache_writer.sh uses for PHP.
AI_* and HOST*_OLLAMA_* land in both confs and both templates in this pass.
Everything ships off: AI_ENABLED false, every AI_ASSIST_* false, conf writes
disabled with an empty whitelist. Nothing in the ecosystem consults it.
The Authelia config default pointed at /mnt/user/appdata/Authelia, which does
not exist — the auth stack lives in the Critical-Data share so it gets synced.
Because conf_populate only fills empty fields, that wrong non-empty default
blocked auto-detection permanently and left the auth page's rules panel dead.
Also corrects three schedule comments that no longer matched varaverk.cron and
two references to scripts that have since been renamed or split.
On this host "authelia" matches both Authelia and Authelia-Secondary, and taking the first
match writes the wrong instance into the conf every other script then trusts.
A copy truncates the live conf and writes into it, so anything sourcing load_config.sh
during that window reads a partial file with empty path variables.
Gmer4Lfe