NPM knows what a certificate is today and nothing about what it was, so ten of them could fail
renewal for months — 1001 certbot runs, zero successes — without anything on any page saying so.
Counts start at zero and are only ever observed; only first_seen is seeded, from NPM's own date.
Authentik is the likely destination and the page had Authelia and lldap wired in at every
level, so the seam goes in now: the panels and every endpoint action route off one conf value,
and a stack that cannot be driven yet says so rather than drawing controls with nothing behind.
vv_is_ai_host() was `=== 'host1'`, which made a physical fact — that is where the
GPU is — look like a rule. AI_OWNER_HOST declares it, so the card can move to a
rebuilt host3 or a friend's spare.
The gate was also doing two jobs. Assistant docks and findings strips now ask
whether a model is reachable, so a node without a GPU gets them by borrowing;
the AI tab asks whether this is the owner, because that page carries the bug
reports, the index and the model configuration — the surface where the
vocabulary assumes you built the mesh.
Resolution is local, then owner, then anyone else declaring a model, pinned once
it answers. Pinned rather than re-derived per call: a mesh that re-decides every
request eventually decides differently mid-conversation, and a chat whose second
turn lands on another machine has no history there. Cleared only on a transport
failure, and only when there is somewhere else to go — a single-node mesh whose
model is down should say so, not report AI as unconfigured.
Every existing exemption lasts until someone remembers to undo it, and nobody
does — Healarr has sat in a pressure list since it was uninstalled and seven
ignore entries name containers that are gone. A mute states when it ends and
then ends, capped by WATCHDOG_MUTE_MAX_HOURS so temporary is enforced rather
than intended. Applied where IGNORE_MAP is built, so all five check sites
inherit it, and shown with its countdown because an invisible suppression is
the thing being fixed.
A fourth candidate source beside the arrs, the system log and container logs:
skip-listed containers, strikes past their limit, repeated restarts, unattended
reboots and sustained pressure become findings. The kind is deliberately not
conf-bound, so it cannot autofix by construction rather than by a switch — and
WATCHDOG_SCAN_IGNORE suppresses it, so a knowingly broken container stays quiet.
First consumer AI_ASSIST_WATCHDOG has ever had.
The operator already runs it, so this is the one provider that costs nothing to adopt: no key,
no card, no third party, and no new container. It aggregates several engines and returns them
merged, and its /api/search reply is already the shape everything downstream expects — title,
url, snippet, with content as a fallback for the roughly one result in six that has no snippet.
It is also now the only provider whose response has been read from a live service rather than
from documentation, which is why it is the default.
Search is the one capability that fits the profile holding none: every other capability either
reads this installation or changes it, and chat's whole contract is that it can do neither. The
assistant deliberately does not get it — its contract is that answers come from this install's
own documents, and a web result there is an answer that looks sourced and is not. A chat question
about this machine hands off to the assistant before the search would run, so it never reaches
the internet even with the box ticked.
Off by default, and not because it is dangerous: searching sends the operator's question outside
the house, which is theirs to decide. Asked for per turn as well as enabled in conf.
Provider-agnostic, as asked — searxng, brave, tavily. Only whichever is configured here can be
verified; all three read every field by name, so a shape that changes underneath yields no
results rather than wrong ones.
The explain fixtures asserted chat holds no capabilities at all, which is exactly the guarantee
worth keeping. caps=only: now states the set rather than its emptiness, so the check still fails
the day something else is granted there.
Triage could see what Varaverk wrote about itself and what the arrs said about themselves, and
nothing else — so a disk throwing I/O errors, a filesystem going read-only or a PCIe link
retraining every two minutes was invisible to the thing whose job is noticing. Container state
was already watched; no line any container actually wrote ever was.
Container patterns match the environment rather than the application. Fifty containers are fifty
programs with no shared vocabulary for malfunctioning, but an exact shared one for a full disk or
a corrupt SQLite file, because those strings come from libc and SQLite rather than from the app.
Both halves are checked by Tools/ai_log_check.sh, which is two tests because the failure modes
are opposite: fixtures for recall on faults this host has never had, and a replay of its real
logs for precision — 74,519 syslog lines and 79,193 container lines, matching only the PCIe
errors it genuinely has.
Severity was being graded from a hand-picked three fields, so every one of these would have been
filed as a warning however bad it was, and notified as one.
A finding nobody is told about is a finding nobody has, and the card added earlier only shows
them to someone who opens the tab. Only needs_operator is announced — an open finding may still
be repaired by the next pass — one notification for all of them, and each is announced once and
stays quiet until the fault changes or gets worse.
vv_notify() hands the message to common.sh's notify() rather than reimplementing the channels,
and calls detect_hosts() explicitly because load_config.sh deliberately does not: without it the
Unraid notification arrives and Discord silently never does. It also reports false when no
channel is switched on at all, since notify() exits 0 either way and a caller believing that
would mark a finding as told and never mention it again.
Notification text is folded to ASCII. Unraid's notifier dropped an em dash outright and left the
double space behind, which was found by sending one and reading what arrived.
Nine entries named paths nothing lives at — a directory that was renamed to System_Essentials,
and docker_update_remaining.sh which has never existed — and the API key renewal was absent
from both arrays it runs from on this host. A fresh install would have skipped all of it.
Nothing in Varaverk fires when a job ends, so this picks up run records that completed
since the last pass rather than adding a hook to forty scripts. Ahead of stability in the
cycle on purpose: a wrong port is not fixed by rebooting the machine.
A model can be confident a port should be 8686; a probe can report that 8686 answered, and
only the second is a fact. Detecting and repairing are also separate things to trust, so
the feature runs read-only until the write switch is turned on under it.
A bug stays open until the code changes and nothing here can close it; a finding closes
itself when the probe that proved the fault starts passing, so the two cannot share a
store. Acknowledging one is scoped to the value it was acknowledged at.
These files are gitignored, so a bad write had nothing to go back to; the escaping order
also stored any value containing a quote truncated but still parseable, which only a
read-back can catch.
AI_INDEX_ON_PULL was designed, defaulted false, and never read by anything —
so nothing has ever rebuilt the index automatically. A pull is the only thing
that changes tracked files on a server, which makes it the only moment the
index can go stale, and a timer would do nothing 23 times a day while still
drifting from the pull that matters.
Staleness is invisible in the answers: the index keeps returning the old text
with confident, correct-looking citations. Gated on pull success, on
AI_INDEX_ON_PULL and on AI_ENABLED, and never fatal — a git pull must not fail
because an embedding call timed out.
A small operator-written file handed to the model at the start of every
conversation — who you are, how this install is set up, what has already been
decided. Injected ahead of the retrieved passages and marked operator-authored
so it outranks anything they contradict, and never cited as a source.
Deliberately not indexed and deliberately under DATA_DIR: it changes
constantly, vector similarity is the wrong way to retrieve things you were
told to remember, and gitignoring it keeps personal notes out of a pushed
repository. The character cap is a context budget — this text costs its share
of 16k on every single turn.
Open-WebUI runs in its own container with no WebGUI session, and nginx applies
auth_request to everything it serves, so it cannot call the plugin API. Same
shape as the arr webhook listener: node outside nginx, shared secret, one
read-only verb. Serves chunks rather than answers because the caller already
has a model loaded.
ai_index.sh and ai_query.sh follow the usual conventions — fail-closed gate,
root check, lock, dry-run, status — with Node doing only the vector maths and
SQLite blobs, the same split api_cache_writer.sh uses for PHP.
AI_* and HOST*_OLLAMA_* land in both confs and both templates in this pass.
Everything ships off: AI_ENABLED false, every AI_ASSIST_* false, conf writes
disabled with an empty whitelist. Nothing in the ecosystem consults it.
The Authelia config default pointed at /mnt/user/appdata/Authelia, which does
not exist — the auth stack lives in the Critical-Data share so it gets synced.
Because conf_populate only fills empty fields, that wrong non-empty default
blocked auto-detection permanently and left the auth page's rules panel dead.
Also corrects three schedule comments that no longer matched varaverk.cron and
two references to scripts that have since been renamed or split.
docker_update.sh rebuilds (stop+recreate) any container whose image
changed, in every mode — but for daily/weekly that was always followed
by the restart script's own unconditional pass, stopping and starting
the same container twice back to back. docker_update.sh now records
which containers it rebuilt this run to a file; docker_daily_restart.sh
and docker_weekly_restart.sh read it and skip those specifically,
still restarting everything else as before. A file older than
DOCKER_UPDATE_REBUILT_STALE_HOURS (default 12) is discarded rather
than trusted, so a missed or failed update run can't suppress a
restart indefinitely.
Template was missing theme.mkv/theme.mp4 and all trailer patterns,
including trailer-*.* for Trailarr's hyphenated TV trailer naming
(trailer-2.mkv) — a fresh install would hit the same false-positive
orphan bug just fixed live on prod.
Healarr does the same job but crashes on a Go concurrency bug (unsynchronized
map access) whenever multiple corruption events land close together — confirmed
via its own crash log, not fixable from our side. Processing one file at a time
here sidesteps the whole bug class instead of trying to work around it.
Delete + explicit EpisodeSearch rather than relying on Sonarr's own background
missing-search cycle, since that skips unmonitored episodes and this shouldn't.
Mirrors the Sonarr fix — same MoveMovieService one-at-a-time drain architecture,
never confirmed live on the Radarr side but the DB-instant/move-async split
is identical, so the same batch-verification race applies.
Confirmed live: DB fields flip instantly on the moveFiles=true PUT but the
physical move drains async, one command at a time — a batch could report
every series moved while most were still sitting at the old path.
arr_cache_prefill moved from critical (30min) to intermediate (4h) —
nothing in the critical tier reads this cache, and every actual
consumer already tolerates staleness well past 4h. Classification
scans run in daily, after the cleanup scripts (not before) — cleanup
frees disk space from orphans/junk first, so the searches --move
triggers for undownloaded relocated entries have headroom for the
resulting grabs.
Overseerr lets users request content into the wrong root folder; these
new report-only scans classify every tracked movie/series (anime, kids,
regular) from metadata alone and flag mismatches against the actual
root folder, in both directions. Rules were validated against real
library data before being adopted — see the header comments in each
script and the master.conf notes above the curated lists.
Every script that fetches the full Lidarr/Sonarr/Radarr tracked-library
list now goes through arr_get_tracked_data() instead of hitting the API
directly -- cache-first when fresh, live fetch as fallback when stale,
waits out an active rescan before either. Per-item file data (trackFile/
episodefile/moviefile) stays live-only everywhere, since that's the
actual disk-truth these scripts' decisions depend on and was never part
of what's cached.
Also adds arr_cache_prefill.sh to CRITICAL_MAINTENANCE_SCRIPTS (30min
tier) with a short 1min wait ceiling, so the cache stays consistently
fresh instead of only refreshing whenever some other script happens to
write through. A full cache refresh for all three arrs measured at ~12s
total live -- nothing like the multi-hour cost of an actual rescan.
Organic scans only touch files actually involved in an import — an
artist/series/movie that already has files sitting untouched on disk
never gets its tracked-file stats refreshed on its own. Confirmed
2026-07-16: Lidarr reported ~23% of its true trackFileCount with no
scan running, for artists whose files were verified present and
readable the whole time. Downstream scripts trust these stats as
source of truth for the share, so drift needs to be caught before
someone notices a suspiciously low number.
Shared cache/rescan-duration logic in common.sh now takes an arr_type
param instead of being Lidarr-specific, so Sonarr and Radarr cleanup
scripts get the same cache-first fetch + rescan-aware retry Lidarr had.
Avoids redundant full-library API calls across scripts run back to back,
and stops false failures when a fetch lands mid-rescan.
Daily maintenance jobs alone now take ~4h with rsync disabled. 19h cap
leaves enough room in the 24h cycle for maintenance to still run
same-day before the next 1am fire, even if a share hits the cap.
Shared cache (lidarr_get_tracked_data() in common.sh) so scripts stop
hitting Lidarr's live API for tracked counts every run, and stop
treating a mid-rescan dip as a genuine problem — a whole-library
RescanFolders legitimately makes trackFileCount read far below normal
while it re-verifies every file (confirmed 2026-07-16: 22% of normal
mid-scan). Cache reads fresh-if-recent, waits out an active rescan
(calibrated to that command's own historical duration, tracked per
command name since RescanFolders and DownloadedAlbumsScan take wildly
different amounts of time), then falls back to a stale cache rather
than hard-failing after a few strikes.
lidarr_cleanup.sh: no longer stacks a fresh DownloadedAlbumsScan on
top of one already running, and the tracked-count floor check now
waits out a genuine rescan instead of aborting on every overlap.
lidarr_duplicate_artist_cleanup.sh (new): finds case-insensitive
duplicate artist entries — same display name, different MusicBrainz
ID, added when a search/list-sync matches the wrong same-named artist.
Deletes the empty phantom side and blocks it from Import List
Exclusions, leaves genuinely-different-real-artists alone (checked by
album title overlap, deduped per-artist first so a legitimate reissue
under an artist's own catalog doesn't false-flag as cross-artist
overlap), and only notifies for the rare case where both sides have
real, overlapping content.
lidarr_cache_prefill.sh (new): warms the cache at array start so nothing
reads it cold after boot.
lidarr_missing_art.sh, lidarr_release_fixer.sh: write-through the cache
as a side effect of fetches they already needed for their own purposes.
Gmer4Lfe