FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bb7c112a2d9cbb8b4f234affa325cd227ff7fc97
Varaverk/Plugin/unraid/include/ai_repair.php
T
Gmer4Lfe 2bda4cdaf8 Declare the AI owner, and let every node borrow its model over the mesh 
vv_is_ai_host() was `=== 'host1'`, which made a physical fact — that is where the
GPU is — look like a rule. AI_OWNER_HOST declares it, so the card can move to a
rebuilt host3 or a friend's spare.

The gate was also doing two jobs. Assistant docks and findings strips now ask
whether a model is reachable, so a node without a GPU gets them by borrowing;
the AI tab asks whether this is the owner, because that page carries the bug
reports, the index and the model configuration — the surface where the
vocabulary assumes you built the mesh.

Resolution is local, then owner, then anyone else declaring a model, pinned once
it answers. Pinned rather than re-derived per call: a mesh that re-decides every
request eventually decides differently mid-conversation, and a chat whose second
turn lands on another machine has no history there. Cleared only on a transport
failure, and only when there is somewhere else to go — a single-node mesh whose
model is down should say so, not report AI as unconfigured.
2026-08-14 22:54:26 -04:00

2423 lines
127 KiB
PHP
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Findings about this installation being misconfigured, and the record of what was done about
// them. A finding is "Emby is not answering at the address the conf gives", not "Varaverk has
// a bug" — the second is what ai_bugs holds.
//
// OPERATIONAL MODEL
// A sweep reads a completed run's log, deterministic patterns turn error lines into typed
// findings, and each finding is either repaired or handed to the operator. Findings persist
// because the repair may need something only a human can supply, and that conversation has to
// survive the page being closed.
//
// Four candidate sources: the arrs' own health, the system log, container logs, and the
// watchdog counters. The first three read text; the fourth reads state the watchdogs already
// maintain, and is the only one whose findings can never be repaired from here — see
// vv_ai_watchdog_findings() for why that is structural rather than a setting.
//
// WHY THIS IS NOT ai_bugs
// Same storage shape, different lifecycle, and the difference is the whole reason for a second
// store. A bug is open until Varaverk's code changes; nothing on this host can close it. A
// finding is open until this host's configuration is right, and the same probe that proved a
// fix can later prove the fault is gone — so findings close themselves and bugs cannot.
//
// Filing them together would mean a list where half the rows are actionable by the operator
// and half are actionable by whoever maintains the project, with no way to tell which is which
// except by reading them.
//
// DESIGN PRINCIPLES
// A finding names something specific, or it is not a finding.
// The point of the record is that something can be done about it. "The daily sync looked
// unhappy" is a feeling; "HOST1_EMBY_URL points at a host that refuses connections" is a
// finding. Triage that cannot resolve a target produces nothing rather than a vague row.
//
// For the conf-bound kinds that is a conf key, and it is still required — a proposal to
// edit a key that does not exist can never be actioned. For the rest it is whatever
// identifies the thing: an arr reporting "all lists are unavailable" is specific and
// actionable with no Varaverk key to change, because the action is in Radarr's own UI.
//
// Evidence is the log line, quoted.
// Same rule as ai_bugs, for the same reason: a finding that cannot show the line it came
// from cannot be checked, and this store is meant to be checkable.
//
// Identity is kind + subject + reference, not the message text.
// A port that has been wrong for a week is one finding seen 400 times, not 400 findings.
// Wording drifts as logs change; the thing being wrong does not. "Indexers unavailable:
// NzbNoob" becoming "NzbNoob, Miatrix" is the same finding getting worse.
//
// OPERATIONAL SAFEGUARDS
// A proposed value is recorded, never trusted.
// 'proposed' is what something might be changed to; 'proven' is whether a probe actually
// got an answer from it. Only proven values are ever written to conf, and the two fields
// are kept separate so a record cannot imply verification it did not have.
//
// Closing is evidence-driven, not time-driven.
// A finding closes when its probe passes or the operator dismisses it. It does not expire,
// because "we stopped seeing it in the log" is equally consistent with the job no longer
// running at all.
//
// Under data/ and therefore gitignored: these quote this installation's logs and name its
// hosts, ports and containers.
//
// EXPORTS
// vv_ai_findings_dir() the store
// vv_ai_finding_write() file or increment one finding
// vv_ai_findings_list() findings, newest activity first
// vv_ai_finding_get() one by id
// vv_ai_finding_close() mark resolved, with how
// vv_ai_finding_dismiss() operator says this is not a problem
// vv_ai_findings_for_chat() the open ones worth opening a conversation about
//
// CONFIGURATION
// AI_DATA_DIR findings live in ai_findings/ beneath it
// AI_FINDING_RETAIN_DAYS closed findings older than this are removed (default 90)
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/config.php';
require_once __DIR__ . '/ai.php';
// For vv_notify() only. A finding that cannot reach the operator is the one thing this file
// cannot do on its own, and reimplementing the channels here would be a second answer to a
// question common.sh already answers.
require_once __DIR__ . '/common.php';
// What a finding can be. The kind decides which repair is even conceivable, so an unknown kind
// is refused rather than stored as an untyped row nothing knows how to act on.
const VV_AI_FINDING_KINDS = [
'unreachable' => 'a configured address or port refused, timed out, or did not resolve',
'auth_rejected' => 'the endpoint answered, and rejected the credential',
'unknown_target' => 'a conf entry names a container or share that does not exist here',
'missing_value' => 'a conf key required by the job that ran is empty',
'arr_health' => 'an arr is reporting a problem about itself',
'system_fault' => 'the kernel reported a hardware or filesystem fault about this machine',
'container_fault' => 'a container is logging a fault about its own environment',
'media_misfiled' => 'a series is shelved somewhere its own metadata does not support',
'watchdog_strike' => 'a watchdog has counted something far enough to be worth a record',
];
// Which kinds are a statement about Varaverk's configuration, and which are a statement about
// something else that is nonetheless worth recording.
//
// The rule was originally "a finding names a conf key or it is not a finding", to stop the store
// filling with observations nobody could act on. What that rule was really protecting is that
// every finding identifies something specific and actionable — naming a conf key was the proxy,
// because at the time every source of findings was a conf problem.
//
// An arr reporting "all lists are unavailable" is specific and actionable, and there is no
// Varaverk key to change: the action is in Radarr's own UI. So the identity widens to a general
// reference, and the conf-key requirement narrows to the kinds it was written for. What has not
// changed is that a finding with nothing to point at is still refused.
const VV_AI_CONF_BOUND_KINDS = ['unreachable', 'auth_rejected', 'unknown_target', 'missing_value'];
function vv_ai_kind_is_conf_bound(string $kind): bool {
return in_array($kind, VV_AI_CONF_BOUND_KINDS, true);
}
// How a finding ended, when it ends.
const VV_AI_FINDING_STATES = [
'open' => 'seen, not yet acted on',
'needs_operator' => 'cannot be repaired here — the value is not derivable from this host',
'acknowledged' => 'the operator knows, and it stays quiet until the state it was acked at changes',
'fixed' => 'a proven value was written to conf',
'resolved' => 'the probe now passes; whatever was wrong is no longer wrong',
'dismissed' => 'the operator says this is not a problem, permanently',
];
// ── Why acknowledged is not dismissed ────────────────────────────────────────────────────────
// "I know critical rsync is off, stop telling me" and "this is never a problem" are different
// instructions, and collapsing them loses the half that matters. An acknowledgement is scoped to
// the state it was given in: CRITICAL_RSYNC_ENABLED being false is a deliberate choice today and
// a stale note the moment it goes true again.
//
// So an ack records what the key read when it was given, and expires when that changes. The
// finding comes back on its own, without the operator having to remember to look — which is the
// difference between a note and a silence.
// ── Toggles are the operator's, always ───────────────────────────────────────────────────────
// A repair may never enable or disable anything on its own. Not because it would get the value
// wrong — a boolean has only two — but because the value is not a fact to be discovered. Whether
// critical rsync should be on is a decision about intent, and a probe cannot prove intent the
// way it can prove that a port answers.
//
// The Fix action still writes it when the operator asks for it. What is forbidden is the
// unattended path choosing for them.
function vv_ai_conf_is_toggle(string $key): bool {
$v = strtolower(trim((string)(vv_conf_vars()[$key] ?? '')));
return $v === 'true' || $v === 'false';
}
// May the sweep write this without being asked? Two conditions, both required: a probe actually
// answered on the proposed value, and the key is not a toggle.
function vv_ai_finding_may_autofix(array $f): bool {
if (!vv_ai_repair_autofix_enabled()) return false;
if (empty($f['proven'])) return false;
if (($f['proposed'] ?? null) === null) return false;
if (vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))) return false;
// A third condition, for paths only. "Proven" means a probe answered, and every probe this
// has is a network probe — nothing in it can answer a filesystem question, so a path
// proposal reaches here carrying a proof that is about something else entirely. Requiring
// the directory to exist is the equivalent evidence, and it is the difference between
// pointing a cleanup at a real share and pointing it at a typo that will be created empty
// by the first script to write there.
//
// vv_conf_path_write_ok() still runs inside the writer underneath this. That one refuses
// what is dangerous; this one refuses what is merely unproven, which is a bar only the
// unattended path has to clear.
$proposed = (string)$f['proposed'];
if ($proposed !== '' && $proposed[0] === '/' && !file_exists($proposed)) return false;
return true;
}
// ── Severity is derived, never supplied ──────────────────────────────────────────────────────
// Same ladder run_job.sh records runs against — ok / warn / error — so a finding and the run it
// came from cannot describe the same event at two different volumes.
//
// The rule that matters: a finding whose key is a toggle can never be an error. Something not
// happening because it was switched off is the switch working. That is true whether the switch
// was flipped deliberately last month or by accident this morning, and the store cannot tell
// those apart — so it reports the fact and lets the operator supply the intent.
//
// Everything else takes its level from what the fault costs. A credential the endpoint rejected
// stops that integration dead; an address that does not answer might be a host still booting.
function vv_ai_finding_severity(array $f): string {
$key = (string)($f['conf_key'] ?? '');
// Deliberate-state findings never escalate, whatever their kind.
if ($key !== '' && vv_ai_conf_is_toggle($key)) return 'warn';
// An arr grades its own health and is the authority on it — "error" from Radarr means Radarr
// has stopped doing something, which is not a judgement to second-guess from out here.
if (($f['kind'] ?? '') === 'arr_health') {
return ($f['arr_type'] ?? '') === 'error' ? 'error' : 'warn';
}
// Same shape as arr_health, for the same reason: the level belongs to the pattern that
// matched, because "correctable PCIe error" and "I/O error on a disk" are the same kind of
// finding and nothing like the same news. Carried on the candidate rather than inferred here.
// Shared by both log-derived kinds — one field, because they mean exactly the same thing.
if (in_array($f['kind'] ?? '', ['system_fault', 'container_fault'], true)) {
return ($f['sys_level'] ?? '') === 'error' ? 'error' : 'warn';
}
return match ($f['kind'] ?? '') {
'auth_rejected' => 'error', // answered and refused — nothing gets through until fixed
'missing_value' => 'error', // configured to use something that was never supplied
'unreachable' => 'warn', // may be transient; the strike system is what escalates it
'unknown_target' => 'warn',
default => 'warn',
};
}
// ── Gates ────────────────────────────────────────────────────────────────────────────────────
// Two switches, because detecting and repairing are separate things to trust.
//
// AI_REPAIR_ENABLED alone gives a system that reads logs, files findings and offers fixes, and
// writes nothing. That is the state this should live in first — long enough to read what it
// found and disagree with some of it. A subsystem that starts by editing conf has to be believed
// before there is any evidence for believing it.
//
// AI_REPAIR_AUTOFIX_ENABLED is what lets a proven value be written without being asked, and it
// is meaningless on its own: nothing to write if nothing is looking. Both must be true, in the
// same layered way AI_ENABLED is necessary but never sufficient.
function vv_ai_repair_enabled(): bool {
if (!vv_ai_config()['enabled']) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_ENABLED'] ?? 'false'))) === 'true';
}
function vv_ai_repair_autofix_enabled(): bool {
if (!vv_ai_repair_enabled()) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_AUTOFIX_ENABLED'] ?? 'false'))) === 'true';
}
// The first consumer AI_ASSIST_WATCHDOG has ever had. It shipped with the rest of the AI_ASSIST_
// tier flags and nothing read it, which meant a switch that looked like it did something.
//
// Separate from AI_REPAIR_ENABLED rather than folded into it because the two answer different
// questions. Repair asks "may this read job logs and offer to change conf". This asks "may this
// have an opinion about what the watchdogs are counting" — a different subject, a different
// appetite for noise, and one you may well want off while repair stays on.
function vv_ai_assist_watchdog_enabled(): bool {
if (!vv_ai_repair_enabled()) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_ASSIST_WATCHDOG'] ?? 'false'))) === 'true';
}
function vv_ai_findings_dir(): string {
$d = AI_DATA_DIR . '/ai_findings';
if (!is_dir($d)) @mkdir($d, 0755, true);
return $d;
}
function vv_ai_finding_retain_days(): int {
$n = (int)(vv_conf_vars()['AI_FINDING_RETAIN_DAYS'] ?? 90);
return max(1, $n);
}
// kind + subject + reference. Deliberately not the message: the same wrong port produces slightly
// different log text as the software around it changes, and that must not mint a second record.
//
// The reference is the conf key for a conf-bound kind, and whatever else identifies the thing
// otherwise — for an arr health item, the check that raised it. "Indexers unavailable: NzbNoob"
// becomes "Indexers unavailable: NzbNoob, Miatrix" as more fail, and that is the same finding
// getting worse rather than a second one.
// Identity includes the host, because "Bazarr is on the skip list" is a different fact on each
// machine that says it. Without the host in the hash, the second node to report the same subject
// would land on the first node's record and overwrite it — and the operator would see one finding
// where two machines have the same problem, or worse, one machine's dismissal silencing another's
// live fault.
//
// $host defaults to this node, so every existing caller keeps working and local findings are
// unchanged in meaning. It is a parameter rather than always-local because a collected finding
// from a partner has to hash as that partner's, not as ours.
function vv_ai_finding_id(string $kind, string $subject, string $ref, string $host = ''): string {
$host = $host !== '' ? $host : vv_detect_host();
return substr(sha1(strtolower($host . '|' . $kind . '|' . $subject . '|' . $ref)), 0, 12);
}
// What the id was before the host joined the hash. Kept solely so a finding already on disk can be
// found once — see the migration in vv_ai_finding_write(). Nine dismissed findings existed when
// this changed, and a dismissal that silently expires is the one outcome this store must never
// produce: "stop telling me about this" has to outlast a refactor.
function vv_ai_finding_id_legacy(string $kind, string $subject, string $ref): string {
return substr(sha1(strtolower($kind . '|' . $subject . '|' . $ref)), 0, 12);
}
function vv_ai_finding_path(string $id): ?string {
if (!preg_match('/^[0-9a-f]{12}$/', $id)) return null;
return vv_ai_findings_dir() . '/' . $id . '.json';
}
function vv_ai_finding_get(string $id): ?array {
$p = vv_ai_finding_path($id);
if ($p === null || !is_file($p)) return null;
$r = json_decode((string)@file_get_contents($p), true);
return is_array($r) ? $r : null;
}
// Files a finding, or increments the one already describing this fault.
//
// $f expects: kind, subject, conf_key, conf_file, observed, evidence, source_log
// and optionally: proposed, proven, state, note
function vv_ai_finding_write(array $f): array {
$kind = (string)($f['kind'] ?? '');
$subject = trim((string)($f['subject'] ?? ''));
$confKey = trim((string)($f['conf_key'] ?? ''));
$evidence = trim((string)($f['evidence'] ?? ''));
// What identifies this finding. Conf-bound kinds are identified by their key; everything else
// supplies its own reference, and a finding with neither points at nothing and is refused.
$ref = trim((string)($f['ref'] ?? $confKey));
if (!isset(VV_AI_FINDING_KINDS[$kind])) return ['ok' => false, 'error' => 'unknown kind'];
if ($subject === '' || $ref === '') return ['ok' => false, 'error' => 'subject and a reference are required'];
if ($evidence === '') return ['ok' => false, 'error' => 'evidence required'];
if (vv_ai_kind_is_conf_bound($kind)) {
if ($confKey === '') return ['ok' => false, 'error' => 'conf_key required for this kind'];
// The key has to be a real shell identifier for the same reason the conf writer insists
// on it: a finding is a proposal to edit that key, and a malformed one cannot be actioned.
if (!vv_conf_key_valid($confKey)) return ['ok' => false, 'error' => 'malformed conf key'];
}
$state = (string)($f['state'] ?? 'open');
if (!isset(VV_AI_FINDING_STATES[$state])) $state = 'open';
$now = time();
$id = vv_ai_finding_id($kind, $subject, $ref);
$rec = [
'id' => $id,
// Which machine this is about. Written even on a single-host install, because the store
// outlives the topology — a finding filed today is still on disk when the second node
// arrives, and one without a host is a record nobody can place.
'host' => (string)($f['host'] ?? vv_detect_host()),
'kind' => $kind,
'subject' => mb_substr($subject, 0, 120),
'conf_key' => $confKey,
'conf_file' => (string)($f['conf_file'] ?? 'master.conf'),
// Secrets never enter this store. A finding about a rejected API key is about the key
// being wrong, and the wrong value is of no use to anyone reading the record later.
'observed' => vv_conf_key_is_secret($confKey) ? '<redacted>'
: mb_substr((string)($f['observed'] ?? ''), 0, 300),
'proposed' => isset($f['proposed']) && !vv_conf_key_is_secret($confKey)
? mb_substr((string)$f['proposed'], 0, 300) : null,
'proven' => (bool)($f['proven'] ?? false),
'state' => $state,
'evidence' => mb_substr(vv_ai_redact($evidence), 0, 1200),
'source_log' => mb_substr((string)($f['source_log'] ?? ''), 0, 200),
'note' => mb_substr((string)($f['note'] ?? ''), 0, 1000),
// Recomputed on every sighting rather than stored once: a key that becomes a toggle, or
// a toggle that is replaced by a real value, changes what this finding means.
'ref' => mb_substr($ref, 0, 120),
// Every field the grader consults has to be handed to it. It was given a hand-picked
// three, so a kind added later that grades on a fourth silently came out as a warning —
// which is how a disk throwing I/O errors would have been filed at the same level as a
// switched-off toggle, and notified as a warning rather than an alert.
'severity' => vv_ai_finding_severity(['kind' => $kind, 'conf_key' => $confKey,
'arr_type' => (string)($f['arr_type'] ?? ''),
'sys_level' => (string)($f['sys_level'] ?? '')]),
'host' => vv_detect_host(),
'first' => $now,
'last' => $now,
'seen' => 1,
'closed_at' => null,
// What the key read when the operator acknowledged it. Null unless acked; the ack
// expires the moment the live value stops matching this.
'ack_value' => null,
// A stable identity for the kinds whose evidence changes on every pass. Empty for the
// rest, which pin to a conf value or to the evidence itself. Stored rather than derived
// so an ack given in the browser pins to exactly what the sweep will compare against.
'pin' => mb_substr((string)($f['pin'] ?? ''), 0, 120),
// What this finding looked like when it was last announced. Carried across sightings
// below — a stamp that reset every fifteen minutes would be a notification every fifteen
// minutes, which is how an operator learns to ignore the channel.
'notified' => null,
];
$p = vv_ai_finding_path($id);
if ($p === null) return ['ok' => false, 'error' => 'bad id'];
// One-time rename from the pre-host id, done here rather than as a startup sweep because this
// is the only moment it matters: a finding is being written, and the question is whether this
// node has said it before. A migration that ran anywhere else would have to walk the whole
// store to answer a question only the write path asks.
//
// Idempotent by construction — after the rename the legacy path no longer exists, and a record
// already carrying the new id never looks.
if (!is_file($p)) {
$legacy = vv_ai_finding_path(vv_ai_finding_id_legacy($kind, $subject, $ref));
if ($legacy !== null && $legacy !== $p && is_file($legacy)) @rename($legacy, $p);
}
if (is_file($p)) {
$old = json_decode((string)@file_get_contents($p), true);
if (is_array($old)) {
$rec['first'] = $old['first'] ?? $now;
$rec['seen'] = (int)($old['seen'] ?? 0) + 1;
// A dismissed finding stays dismissed however many times the log repeats it —
// otherwise "this is fine, stop telling me" lasts exactly one cycle. A fixed one
// reopens, because seeing the fault again after a repair means the repair did not
// hold, which is the single most important thing this store can tell anyone.
if (($old['state'] ?? '') === 'dismissed') {
$rec['state'] = 'dismissed';
$rec['closed_at'] = $old['closed_at'] ?? null;
}
// An acknowledgement holds only while the thing acknowledged is still true. Compare
// what it is pinned to now against what it read when the ack was given: unchanged
// means stay quiet, changed means the note is stale and the finding comes back by
// itself. $rec carries this sighting's evidence, so a fault that has changed shape
// fails this comparison even when no conf key is involved.
if (($old['state'] ?? '') === 'acknowledged') {
$ackedAt = (string)($old['ack_value'] ?? '');
if ($ackedAt === vv_ai_finding_ack_pin($rec)) {
$rec['state'] = 'acknowledged';
$rec['ack_value'] = $ackedAt;
$rec['closed_at'] = $old['closed_at'] ?? null;
}
// Otherwise $rec keeps the state this sighting computed — it has reopened.
}
// Preserve an operator's note over a generated one.
if ($rec['note'] === '' && !empty($old['note'])) $rec['note'] = $old['note'];
// Carried, never recomputed. This record is rebuilt from scratch on every sighting,
// so anything not copied forward here is reset — and a reset announcement stamp
// means this finding is announced again on the next pass, and the one after that.
$rec['notified'] = $old['notified'] ?? null;
}
}
if (@file_put_contents($p, json_encode($rec, JSON_PRETTY_PRINT)) === false) {
return ['ok' => false, 'error' => 'write failed'];
}
return ['ok' => true, 'id' => $id, 'seen' => $rec['seen'], 'state' => $rec['state']];
}
// $states filters; empty means everything. Newest activity first, because a finding seen in the
// last cycle matters more than one that has been sitting fixed for a month.
function vv_ai_findings_list(array $states = ['open', 'needs_operator']): array {
$out = [];
foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) {
$r = json_decode((string)@file_get_contents($file), true);
if (!is_array($r)) continue;
if ($states && !in_array($r['state'] ?? 'open', $states, true)) continue;
$out[] = $r;
}
usort($out, fn($a, $b) => ($b['last'] ?? 0) <=> ($a['last'] ?? 0));
return $out;
}
function vv_ai_finding_set_state(string $id, string $state, string $note = ''): bool {
if (!isset(VV_AI_FINDING_STATES[$state])) return false;
$r = vv_ai_finding_get($id);
if ($r === null) return false;
$r['state'] = $state;
$r['closed_at'] = in_array($state, ['open', 'needs_operator'], true) ? null : time();
if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000);
// No attribution here on purpose. This setter is called by the sweep as well as by a person —
// it is how a candidate becomes needs_operator — and stamping every transition would credit a
// machine for deciding something it only classified. Attribution belongs to the answer, not to
// the bookkeeping, so vv_ai_finding_apply_action() records it and this does not.
$p = vv_ai_finding_path($id);
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
}
function vv_ai_finding_close(string $id, string $note = ''): bool {
return vv_ai_finding_set_state($id, 'resolved', $note);
}
function vv_ai_finding_dismiss(string $id, string $note = ''): bool {
return vv_ai_finding_set_state($id, 'dismissed', $note);
}
// What an acknowledgement is pinned to — the thing that has to stay the same for the ack to keep
// meaning what it meant.
//
// For a conf-bound finding that is the key's value, which is what the operator was looking at
// when they said "I know". A finding with no key had nothing to pin to and so compared '' with
// '' — every ack on an arr health item was silently permanent, which is dismiss wearing ack's
// label. Those pin to the shape of the fault instead: "indexers unavailable: NzbNoob" and
// "indexers unavailable: NzbNoob, Miatrix" are one finding getting worse, and an ack given for
// the first has not been given for the second.
// A finding may supply its own pin, and the log-derived kinds must. Their evidence carries a line
// count and the timestamp of the first matching line, so it is different on every single pass —
// pinning to it would expire an acknowledgement within fifteen minutes and re-announce a fault
// the operator had just said they knew about, forever. A PCIe controller that has thrown
// correctable errors since the machine was built is exactly that case.
//
// The pin those kinds supply is their identity: the fault class and the thing it is about. A
// fault that gets genuinely worse — correctable becoming uncorrectable — has a different class,
// so it is a different finding and announces on its own rather than hiding behind this one's ack.
function vv_ai_finding_ack_pin(array $f): string {
if (($f['pin'] ?? '') !== '') return (string)$f['pin'];
$key = (string)($f['conf_key'] ?? '');
if ($key !== '') return (string)(vv_conf_vars()[$key] ?? '');
return 'ev:' . substr(sha1((string)($f['evidence'] ?? '')), 0, 16);
}
// "I know about this — leave it, and tell me if it changes."
//
// Stamps what it is pinned to onto the record. Every later sighting compares against that stamp,
// so the acknowledgement covers this state and not the finding forever. Acking that critical
// rsync is off says nothing about critical rsync being on.
function vv_ai_finding_ack(string $id, string $note = ''): bool {
$r = vv_ai_finding_get($id);
if ($r === null) return false;
$r['state'] = 'acknowledged';
$r['ack_value'] = vv_ai_finding_ack_pin($r);
$r['closed_at'] = time();
if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000);
$p = vv_ai_finding_path($id);
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
}
// What the operator can do about a finding, and what each choice means. Returned rather than
// hardcoded in the UI so the chat and the page cannot offer different options for the same row,
// and enforced in vv_ai_finding_apply_action() so neither can act on one it was not offered.
//
// Fix appears for anything with a proposed value, toggle or not — the prohibition is on the
// sweep choosing, never on the operator choosing. Open rows also carry ack, dismiss and cancel,
// because "I know", "this is never a problem" and "not now" are all valid answers to being told
// something, and they are three different answers.
function vv_ai_finding_actions(array $f): array {
// A closed finding has one question left, and it is not the original one: was closing it
// right? Offering fix or ack on a row that is already dismissed is offering to decide
// something that has been decided. Reopen is here because dismiss is otherwise permanent —
// the write path keeps a dismissed finding dismissed however many times the fault recurs,
// so a mis-click would need someone editing JSON on disk to undo.
if (!in_array((string)($f['state'] ?? 'open'), ['open', 'needs_operator'], true)) {
return ['reopen' => 'Put it back on the list — either closing it was wrong, or it is back'];
}
$actions = [];
if (($f['proposed'] ?? null) !== null) {
$actions['fix'] = vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))
? 'Set ' . $f['conf_key'] . ' — a toggle, so this only ever happens because you asked'
: 'Write the proven value to ' . $f['conf_key'];
}
// Offered only when a destination was recorded, which is only for a series the triage called
// misfiled. "Uncertain" carries no move_to, so it gets the same buttons every other finding
// has and no way to act on a judgement nobody made.
if (($f['move_to'] ?? '') !== '' && (int)($f['move_id'] ?? 0) > 0) {
$actions['move'] = 'Move ' . ($f['subject'] ?? 'it') . ' to ' . $f['move_to']
. ' — relocates the files on disk';
}
$actions['ack'] = ($f['conf_key'] ?? '') !== ''
? 'Known and intended. Stays quiet until ' . $f['conf_key'] . ' changes'
: 'Known and intended. Stays quiet until the fault itself changes';
$actions['dismiss'] = 'Not a problem, ever. Stays closed even when it is seen again';
$actions['cancel'] = 'Leave it alone for now';
return $actions;
}
// Closed findings are kept for a while because "this happened before and here is what fixed it"
// is worth more than the disk it costs. Open ones are never pruned — an unresolved problem does
// not stop mattering because it is old.
function vv_ai_findings_prune(): int {
$cutoff = time() - (vv_ai_finding_retain_days() * 86400);
$n = 0;
foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) {
$r = json_decode((string)@file_get_contents($file), true);
if (!is_array($r)) continue;
if (in_array($r['state'] ?? 'open', ['open', 'needs_operator'], true)) continue;
if ((int)($r['closed_at'] ?? 0) > $cutoff) continue;
if (@unlink($file)) $n++;
}
return $n;
}
// ── The system's own log ─────────────────────────────────────────────────────────────────────
// Everything above this reads Varaverk's logs and asks the arrs about themselves. Neither can
// see a disk throwing I/O errors, a filesystem going read-only, or a PCIe link retraining every
// two minutes — and those are the faults that explain the ones Varaverk does notice.
//
// /var/log/syslog and not dmesg. Unraid's syslog carries the kernel ring buffer's lines already,
// prefixed with `kernel:`, and it carries a real date on every line. dmesg's ring buffer wraps,
// has no persistence across a reboot, and its -T timestamps are derived from uptime rather than
// recorded, which makes "since the last pass" unanswerable from it.
//
// What is deliberately NOT here:
// Container restarts, crash loops and OOMKilled — docker_watchdog owns those and notifies on
// them, and a second opinion arriving by a second channel is not more information.
// ZFS pool health — `zpool status` answers that exactly, and inferring it from log lines when
// the authoritative command is one call away would be guessing on purpose.
//
// A host-level OOM kill IS here, and does overlap docker_watchdog when the process killed was in
// a container. They are different halves: the watchdog reports that a container restarted, this
// reports what the kernel killed and that it was memory. The store folds repeats into one row.
const VV_AI_SYSLOG_PATTERNS = [
// Verified against this host's own syslog, which produced 109 of these in a day. Correctable
// means the link recovered, which is why it is a warning and not an error — but a device
// retraining continuously is failing slowly.
['re' => '/AER: (?:Multiple )?Corrected error.*?(?:from|received from) (?<subject>[0-9a-f:.]+)/i',
'level' => 'warn', 'what' => 'PCIe correctable errors'],
['re' => '/AER: (?:Multiple )?Correctable error message received from (?<subject>[0-9a-f:.]+)/i',
'level' => 'warn', 'what' => 'PCIe correctable errors'],
// The severity word is followed by a parenthetical often enough that requiring "error" to
// come straight after it misses the common form: "Uncorrected (Non-Fatal) error received
// from". Found by a fixture, not by reasoning about it.
['re' => '/AER: (?:Multiple )?(?:Uncorrected|Fatal|Uncorrectable)[^:]{0,20}? error.*?from (?<subject>[0-9a-f:.]+)/i',
'level' => 'error', 'what' => 'PCIe uncorrectable errors'],
// Block layer. blk_update_request is where a failed read or write surfaces with the device
// named, which is the line worth keeping — the ATA exception above it names a port, not a
// disk, and a port number is not something an operator can act on.
['re' => '/blk_update_request: (?:critical )?(?:I\/O|medium|target|nonexistent) error, dev (?<subject>[a-z0-9]+)/i',
'level' => 'error', 'what' => 'block I/O errors'],
['re' => '/Buffer I\/O error on dev (?<subject>[a-z0-9]+)/i',
'level' => 'error', 'what' => 'buffer I/O errors'],
// The SCSI layer writes the device in brackets — "sd 1:0:3:0: [sdo] tag#28 FAILED" — so the
// name is not followed by a colon the way it is everywhere else. Both forms accepted.
['re' => '/(?:\[(?<subject>sd[a-z]+|nvme\d+n\d+)\]|(?<subject2>sd[a-z]+|nvme\d+n\d+):).*?(?:unrecovered read error|medium error|rejecting I\/O to|failed command)/i',
'level' => 'error', 'what' => 'device errors'],
// Filesystems. Each names the device inside its own parentheses, which is the identity the
// finding is keyed on — one finding per filesystem, however many lines it emits.
['re' => '/XFS \((?<subject>[^)]+)\): (?:Metadata|Corruption|corruption|log I\/O error|writeback error|Internal error)/',
'level' => 'error', 'what' => 'XFS errors'],
['re' => '/BTRFS (?:error|critical) \(device (?<subject>[^)]+)\)/',
'level' => 'error', 'what' => 'BTRFS errors'],
['re' => '/EXT4-fs error \(device (?<subject>[^)]+)\)/',
'level' => 'error', 'what' => 'ext4 errors'],
['re' => '/(?:Remounting|remounting) filesystem read-only/',
'level' => 'error', 'what' => 'a filesystem went read-only', 'subject' => 'filesystem'],
// Memory. The process name is the subject, so "the kernel keeps killing shfs" is one finding
// rather than one per occurrence.
['re' => '/Out of memory: Killed process \d+ \((?<subject>[^)]+)\)/',
'level' => 'error', 'what' => 'out-of-memory kills'],
// The kernel saying it has broken. No subject to extract that means anything, so the finding
// is about the machine.
['re' => '/(?:kernel BUG at|general protection fault|Oops: |Kernel panic)/',
'level' => 'error', 'what' => 'kernel faults', 'subject' => 'kernel'],
];
function vv_ai_syslog_path(): string {
return '/var/log/syslog';
}
function vv_ai_syslog_enabled(): bool {
if (!vv_ai_repair_enabled()) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_SYSLOG_ENABLED'] ?? 'true'))) !== 'false';
}
function vv_ai_syslog_max_lines(): int {
$n = (int)(vv_conf_vars()['AI_REPAIR_SYSLOG_MAX_LINES'] ?? 4000);
return max(100, min(50000, $n));
}
// "Aug 9 21:46:07" — syslog's format carries no year, which is a real problem exactly once a
// year. Parsed against the current one, and anything landing more than a day in the future is
// read as last year's: on 1 January, December's lines would otherwise be stamped eleven months
// ahead and every one of them would look newer than the last sweep, forever.
//
// Returns null for a line that does not start with a timestamp, which is a continuation line and
// belongs to whatever preceded it.
function vv_ai_syslog_ts(string $line, ?int $now = null): ?int {
if (!preg_match('/^([A-Z][a-z]{2}\s+\d{1,2} \d{2}:\d{2}:\d{2})/', $line, $m)) return null;
$now = $now ?? time();
$ts = strtotime($m[1] . ' ' . date('Y', $now));
if ($ts === false) return null;
if ($ts > $now + 86400) {
$ts = strtotime($m[1] . ' ' . (int)(date('Y', $now) - 1));
if ($ts === false) return null;
}
return $ts;
}
// The tail of syslog, bounded twice: by the line cap and by the timestamp. The cap is applied
// first and is what stops a rotation, a boot, or a flood from turning one pass into a scan of
// the whole file.
function vv_ai_syslog_lines(int $since, ?int $maxLines = null, ?string $path = null): array {
$path = $path ?? vv_ai_syslog_path();
if (!is_readable($path)) return [];
$max = $maxLines ?? vv_ai_syslog_max_lines();
$out = []; $rc = 0;
exec('tail -n ' . (int)$max . ' ' . escapeshellarg($path) . ' 2>/dev/null', $out, $rc);
if ($rc !== 0) return [];
$now = time();
$kept = [];
foreach ($out as $line) {
$ts = vv_ai_syslog_ts($line, $now);
// A line with no timestamp cannot be placed in time. Kept only if the line before it was
// kept, since that is what a continuation is.
if ($ts === null) { if ($kept) $kept[] = $line; continue; }
if ($ts <= $since) continue;
$kept[] = $line;
}
return $kept;
}
// Log lines in, finding candidates out. One candidate per (pattern, subject) however many lines
// matched, with the count carried in the evidence — a disk that threw four hundred I/O errors in
// a quarter of an hour is one fault, and four hundred findings would be four hundred ways to
// miss it.
function vv_ai_syslog_findings(int $since, ?array $lines = null): array {
if (!vv_ai_syslog_enabled()) return [];
$lines = $lines ?? vv_ai_syslog_lines($since);
if (!$lines) return [];
$agg = [];
foreach ($lines as $line) {
foreach (VV_AI_SYSLOG_PATTERNS as $p) {
if (!preg_match($p['re'], $line, $m)) continue;
// subject2 is the second branch of an alternation. PHP refuses two groups with the
// same name in one pattern, so a pattern that can find its subject in either of two
// shapes has to name them apart and try both here.
$subject = trim((string)($m['subject'] ?? ''));
if ($subject === '') $subject = trim((string)($m['subject2'] ?? ''));
if ($subject === '') $subject = trim((string)($p['subject'] ?? ''));
if ($subject === '') $subject = 'system';
$key = $p['what'] . '|' . $subject;
if (!isset($agg[$key])) {
$agg[$key] = ['what' => $p['what'], 'subject' => $subject, 'level' => $p['level'],
'count' => 0, 'first_line' => trim($line)];
}
$agg[$key]['count']++;
// The worst level wins when two patterns describe the same subject.
if ($p['level'] === 'error') $agg[$key]['level'] = 'error';
break; // one pattern per line; the list is ordered most specific first
}
}
$found = [];
foreach ($agg as $a) {
$found[] = [
'kind' => 'system_fault',
'subject' => $a['subject'],
// Identity, not contents — the evidence below counts lines and quotes a timestamp,
// so it differs every pass and would expire an acknowledgement immediately.
'pin' => 'sys:' . $a['what'] . '|' . $a['subject'],
// What identifies it: the class of fault, not the message. The wording of a kernel
// line changes between releases and the fault does not.
'ref' => $a['what'],
'conf_key' => '',
'conf_file' => '',
'sys_level' => $a['level'],
'observed' => $a['count'] . ' since the last pass',
'evidence' => sprintf('%s on %s — %d line%s since the last pass. First: %s',
$a['what'], $a['subject'], $a['count'],
$a['count'] === 1 ? '' : 's',
mb_substr($a['first_line'], 0, 200)),
'source_log' => vv_ai_syslog_path(),
// Nothing here is repairable from conf, so it goes straight to the operator rather
// than sitting open waiting for a probe that will never run.
'state' => 'needs_operator',
];
}
return $found;
}
// ── What the containers are saying ───────────────────────────────────────────────────────────
// docker_watchdog watches container *state* — is it up, does its port answer, is it restarting.
// None of that reads a line the application wrote, so a container that is running perfectly and
// has been unable to write to its database for a day looks completely healthy from out there.
//
// The patterns are deliberately about the container's environment rather than its behaviour.
// Fifty containers run here and they are fifty different applications; there is no useful shared
// vocabulary for "this app is malfunctioning". There is an exact shared vocabulary for "the disk
// is full", "the filesystem is read-only" and "my database is corrupt", because those come from
// libc, the kernel and SQLite rather than from the application — the same string in every one of
// them. Anything app-specific belongs in that app's own health endpoint, which is where the arr
// checks already come from.
//
// Noise is the whole risk here. Every pattern below was run against the real logs of all fifty
// containers on this host before being kept; see Tools/ai_container_check.sh.
const VV_AI_CONTAINER_PATTERNS = [
['re' => '/no space left on device/i',
'level' => 'error', 'what' => 'disk full'],
['re' => '/read-only file ?system|Read-only file system/i',
'level' => 'error', 'what' => 'read-only filesystem'],
['re' => '/disk quota exceeded/i',
'level' => 'error', 'what' => 'disk quota exceeded'],
// SQLite's own wording. "database is locked" is deliberately absent: it is contention, it is
// transient, and the arrs emit it in normal operation.
['re' => '/database disk image is malformed|database or disk is full|file is (?:not a database|encrypted or is not a database)/i',
'level' => 'error', 'what' => 'database corruption'],
['re' => '/too many open files/i',
'level' => 'error', 'what' => 'file descriptor limit'],
['re' => '/certificate has expired|certificate is not yet valid|certificate verify failed/i',
'level' => 'error', 'what' => 'certificate problems'],
];
function vv_ai_container_logs_enabled(): bool {
if (!vv_ai_repair_enabled()) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_CONTAINER_LOGS_ENABLED'] ?? 'true'))) !== 'false';
}
function vv_ai_container_log_lines(): int {
$n = (int)(vv_conf_vars()['AI_REPAIR_CONTAINER_LOG_LINES'] ?? 400);
return max(50, min(5000, $n));
}
// Running containers only. A stopped one has nothing new to say, and its last words before it
// stopped are docker_watchdog's business.
function vv_ai_running_containers(): array {
$out = []; $rc = 0;
exec('docker ps --format {{.Names}} 2>/dev/null', $out, $rc);
if ($rc !== 0) return [];
return array_values(array_filter(array_map('trim', $out), fn($n) => $n !== ''));
}
// One container's log since the marker, bounded by both a time and a line count. --since alone
// is not a bound: a container that logged a million lines in the last quarter hour would hand
// back all of them.
function vv_ai_container_log(string $name, int $since, ?int $maxLines = null): array {
if (!preg_match('/^[A-Za-z0-9][A-Za-z0-9_.-]*$/', $name)) return [];
$max = $maxLines ?? vv_ai_container_log_lines();
// Both streams: applications disagree about which one an error belongs on, and several of
// these write everything to stdout.
$cmd = 'docker logs --since ' . escapeshellarg((string)max(1, $since))
. ' --tail ' . (int)$max . ' ' . escapeshellarg($name) . ' 2>&1';
$out = []; $rc = 0;
exec($cmd, $out, $rc);
return $rc === 0 ? $out : [];
}
// Same aggregation as the syslog pass: one candidate per (container, fault class), with the
// count in the evidence rather than a finding per line.
function vv_ai_container_findings(int $since, ?array $logsByContainer = null): array {
if ($logsByContainer === null && !vv_ai_container_logs_enabled()) return [];
$agg = [];
$names = $logsByContainer !== null ? array_keys($logsByContainer) : vv_ai_running_containers();
foreach ($names as $name) {
$lines = $logsByContainer !== null ? $logsByContainer[$name]
: vv_ai_container_log($name, $since);
foreach ($lines as $line) {
foreach (VV_AI_CONTAINER_PATTERNS as $p) {
if (!preg_match($p['re'], $line)) continue;
$key = $name . '|' . $p['what'];
if (!isset($agg[$key])) {
$agg[$key] = ['name' => $name, 'what' => $p['what'], 'level' => $p['level'],
'count' => 0, 'first_line' => trim($line)];
}
$agg[$key]['count']++;
break;
}
}
}
$found = [];
foreach ($agg as $a) {
$found[] = [
'kind' => 'container_fault',
'subject' => $a['name'],
'pin' => 'ctr:' . $a['what'] . '|' . $a['name'],
'ref' => $a['what'],
'conf_key' => '',
'conf_file' => '',
'sys_level' => $a['level'],
'observed' => $a['count'] . ' since the last pass',
'evidence' => sprintf('%s in %s — %d line%s since the last pass. First: %s',
$a['what'], $a['name'], $a['count'],
$a['count'] === 1 ? '' : 's',
mb_substr($a['first_line'], 0, 200)),
'source_log' => 'docker logs ' . $a['name'],
'state' => 'needs_operator',
];
}
return $found;
}
// ── What the watchdogs have counted ──────────────────────────────────────────────────────────
// The fourth candidate source. The other three read logs; this reads the counters the watchdogs
// keep, and turns the ones that have gone far enough into a record with a name on it.
//
// It reads vv_wd_all() rather than the state files, for the reason the Watchdog tab does: the
// meaning of a strike lives in include/watchdog.php, and a second parser here would be a second
// opinion about it. It also gets the thresholds in the same payload, so a count is always judged
// against the limit that was in force when it was read rather than one fetched separately.
//
// Deliberately not conf-bound. 'watchdog_strike' is absent from VV_AI_CONF_BOUND_KINDS, so these
// findings carry no conf key, so vv_ai_finding_may_autofix() can never return true for one. That
// is a structural guarantee rather than a switch: there is no setting that makes this able to
// change anything, and none can be added without also deciding which key it would write.
//
// What it does NOT do is explain. The text below is derived, not generated — no model is asked
// anything on this path. The reading is a click away in the troubleshoot profile, already scoped
// to the orchestrator's log by the Why? buttons on the Watchdog tab. Detection has to be worth
// believing before it is worth narrating, and a wrong cause stated fluently is worse than none.
const VV_AI_WD_PRESSURE_LEVELS = [2 => 'medium', 3 => 'hard'];
function vv_ai_watchdog_findings(?array $payload = null): array {
if ($payload === null && !vv_ai_assist_watchdog_enabled()) return [];
$wd = $payload ?? vv_cache_read('watchdog', 900);
if ($wd === null) {
require_once __DIR__ . '/watchdog.php';
$wd = vv_wd_all();
}
$cfg = $wd['cfg'] ?? [];
$found = [];
foreach ($wd['nodes'] ?? [] as $node) {
// This host only. A partner's counters are its own to file findings about, and the
// operator reading them here could not act on them from this side anyway.
if (empty($node['is_me'])) continue;
$st = $node['states'] ?? null;
if (!$st) continue;
$id = (string)($node['id'] ?? 'this host');
// The operator has already said they know about these. Suppressing here rather than at
// write time keeps them out of the pin store entirely — 7DaysToDie is the worked example:
// knowingly broken, deliberately ignored, and a finding about it every fifteen minutes
// would be exactly the crying-wolf this is supposed to avoid.
$ignore = array_flip((array)($node['config']['ignore'] ?? []));
$add = function (string $subject, string $ref, string $level, string $observed,
string $evidence) use (&$found) {
$found[] = [
'kind' => 'watchdog_strike',
'subject' => $subject,
'pin' => 'wd:' . $ref . '|' . $subject,
'ref' => $ref,
'conf_key' => '',
'conf_file' => '',
'sys_level' => $level,
'observed' => $observed,
'evidence' => $evidence,
// Every watchdog logs through the orchestrator; none keeps a log of its own.
'source_log' => 'Orchestrators/watchdog_orchestrator',
'state' => 'needs_operator',
];
};
// 1. Skip list — the terminal state. The watchdog tried its restart limit and stopped,
// which means nothing further is coming automatically. Always worth a record.
foreach ((array)($st['skiplist'] ?? []) as $name) {
if (isset($ignore[$name])) continue;
$add($name, 'skiplist', 'error', 'on the skip list',
sprintf('%s is on the docker watchdog skip list on %s. It was restarted up to '
. 'the limit of %d and did not stay up, so the watchdog has stopped trying '
. 'and will not restart it again without intervention.',
$name, $id, (int)($cfg['restart_limit'] ?? 3)));
}
// 2. Container check strikes at or past their limit. Below the limit the watchdog is
// still working the problem and a finding would be premature.
$lim = max(1, (int)($cfg['cpu_fail_lim'] ?? 2));
foreach ((array)($st['ctr_strikes'] ?? []) as $key => $count) {
$name = preg_replace('/_(cpu|http|api|docker)$/', '', (string)$key);
if (isset($ignore[$name]) || (int)$count < $lim) continue;
$check = preg_match('/_(cpu|http|api|docker)$/', (string)$key, $m) ? $m[1] : 'health';
$add($name, 'strike:' . $check, 'warn', $count . ' / ' . $lim,
sprintf('The docker watchdog has counted %d consecutive %s failures for %s on %s, '
. 'against a limit of %d.', (int)$count, $check, $name, $id, $lim));
}
// 3. Repeated restarts inside the 24h window. Distinct from a strike: the container is
// coming back each time, which is why nothing has escalated, and is also why this is
// easy to never notice.
$rc = [];
foreach ((array)($st['restarts'] ?? []) as $r) {
$n = (string)($r['name'] ?? '');
if ($n !== '') $rc[$n] = ($rc[$n] ?? 0) + 1;
}
$rlim = max(2, (int)($cfg['restart_limit'] ?? 3));
foreach ($rc as $name => $count) {
if (isset($ignore[$name]) || $count < $rlim) continue;
$add($name, 'restarts', 'warn', $count . ' in 24h',
sprintf('%s has been restarted %d times in the last 24 hours on %s. Each one '
. 'succeeded, so no strike escalated — but a container that needs restarting '
. 'this often is failing at something between restarts.', $name, $count, $id));
}
// 4. Any unattended reboot. Not thresholded: the machine restarting itself is worth
// saying once every time it happens, however normal the count is.
$reboots = (array)($st['reboots'] ?? []);
if ($reboots) {
$add($id, 'reboots', 'error', count($reboots) . ' / ' . (int)($cfg['reboot_limit'] ?? 3),
sprintf('The stability watchdog has rebooted %s %d time%s in the last %d hours, '
. 'against a limit of %d before it stops trying.',
$id, count($reboots), count($reboots) === 1 ? '' : 's',
(int)($cfg['reboot_window'] ?? 12), (int)($cfg['reboot_limit'] ?? 3)));
}
// 5. Sustained memory pressure. Level 1 is soft and self-clears constantly; from level 2
// the watchdog has acted on containers, and that is a thing that happened.
$lvl = (int)($st['rw_level'] ?? 0);
if (isset(VV_AI_WD_PRESSURE_LEVELS[$lvl])) {
$paused = array_values(array_filter((array)($st['rw_paused'] ?? [])));
$stopped = array_values(array_filter((array)($st['rw_stopped'] ?? [])));
$acted = array_merge(
$paused ? ['paused ' . implode(', ', $paused)] : [],
$stopped ? ['stopped ' . implode(', ', $stopped)] : []
);
$add($id, 'pressure', $lvl >= 3 ? 'error' : 'warn',
VV_AI_WD_PRESSURE_LEVELS[$lvl] . ' pressure',
sprintf('The resource watchdog is at %s pressure on %s%s.%s',
VV_AI_WD_PRESSURE_LEVELS[$lvl], $id,
$acted ? ' and has ' . implode(' and ', $acted) : '',
!empty($st['mem_shutdown']) ? ' The memory shutdown flag is set, so '
. 'docker_watchdog is deferring container restarts.' : ''));
}
}
return $found;
}
// ── What the AI adds to which script ──────────────────────────────────────────────────────────
// Declared here rather than in each script's header, and that is a deliberate departure from
// api/scriptinfo.php's usual rule that the header next to the code is authoritative. It is
// authoritative about what the script does. An enhancement is not the script's behaviour — it is
// something else reading the script's output afterwards, implemented in PHP, gated by a flag the
// script has never heard of. Writing it into the bash header would put a description of PHP inside
// a file that cannot enforce it, and the two would drift the first time either changed.
//
// A script may appear here without the enhancement being about its output. sonarr discovery is
// listed because the triage is its safety net: discovery takes the first accessible root folder
// (playback_aware_sonarr_discovery.sh:750) with no regard for whether the show is anime, kids or
// general, and the classification scan is what notices the next night. That relationship is real,
// it explains where misfiled series come from, and until now it was written down nowhere.
const VV_AI_SCRIPT_ENHANCEMENTS = [
'Arrs_Stack/sonarr_classification_scan.sh' => [
'name' => 'Classification triage',
'flag' => 'AI_ASSIST_DISCOVERY',
'what' => 'This scan reports reverse-anime leaks and deliberately does not act on them — '
. 'its own header calls them genuine judgement calls, because no metadata field '
. 'separates a misfile from a deliberate choice. The triage sorts that bucket into '
. 'misfiled, donghua, anime-adjacent or uncertain.',
'acts' => 'Files a finding for misfiled and uncertain only. A misfiled series carries a '
. 'Move action, which relocates the files — pressed by a person, never automatic.',
],
'Arrs_Stack/playback_aware_sonarr_discovery.sh' => [
'name' => 'Classification triage (safety net)',
'flag' => 'AI_ASSIST_DISCOVERY',
'what' => 'Discovery adds every show it finds to the first accessible root folder, without '
. 'classifying it. Anything it places wrongly is caught by the classification scan '
. 'that night, and the triage is what turns that scan\'s count into named titles.',
'acts' => 'Nothing here. The enhancement runs against the classification scan\'s output, '
. 'not against this script.',
],
'Orchestrators/watchdog_orchestrator.sh' => [
'name' => 'Watchdog findings',
'flag' => 'AI_ASSIST_WATCHDOG',
'what' => 'Turns the counters the watchdogs keep — skip lists, strikes past their limit, '
. 'repeated restarts, unattended reboots, sustained pressure — into named findings '
. 'instead of numbers on a card.',
'acts' => 'Files findings only. The kind is not conf-bound, so it cannot autofix by '
. 'construction rather than by a setting.',
],
];
function vv_ai_script_enhancements(string $id = ''): array {
if ($id === '') return VV_AI_SCRIPT_ENHANCEMENTS;
$id = ltrim(trim($id), '/');
if (!str_ends_with($id, '.sh')) $id .= '.sh';
$e = VV_AI_SCRIPT_ENHANCEMENTS[$id] ?? null;
if (!$e) return [];
// The flag state travels with it. An enhancement listed against a script without saying
// whether it is switched on is the same trap AI_ASSIST_WATCHDOG was for months: a name that
// looks like a feature and does nothing.
$e['enabled'] = strtolower(trim((string)(vv_conf_vars()[$e['flag']] ?? 'false'))) === 'true'
&& vv_ai_repair_enabled();
return $e;
}
// ── Classification triage ─────────────────────────────────────────────────────────────────────
// The one place in the media stack where a model beats the rule it is helping.
//
// sonarr_classification_scan.sh decides anime/kids/regular from metadata — genre, certification,
// network, original language — and that works: forward misses are zero. What it cannot decide is
// the reverse direction, which its own header calls "genuine judgment calls". A series sitting in
// the anime root with no anime signal is either misfiled or a deliberate choice, and no metadata
// field distinguishes those. So it reports a count, every night, and a count is not actionable:
// seventeen of them concealed two live-action crime dramas filed under anime for as long as
// nobody read the list.
//
// This asks only about that bucket — eleven titles, not the eleven hundred the scan handles — and
// only to sort them. It moves nothing. --move stays a flag a human types, because relocating media
// is the one action here that is tedious to undo and impossible to notice going wrong.
//
// Reads the scan's own verdict from the review file rather than re-deriving it. A second opinion
// computed from the same metadata would be the same answer with extra steps, and would drift from
// the script the first time either changed.
const VV_AI_TRIAGE_BUCKETS = ['misfiled', 'donghua', 'anime_adjacent', 'uncertain'];
function vv_ai_assist_discovery_enabled(): bool {
if (!vv_ai_repair_enabled()) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_ASSIST_DISCOVERY'] ?? 'false'))) === 'true';
}
function vv_ai_classification_review(): array {
$raw = @file_get_contents(STATE_DIR . '/arr_classification_review.json');
if ($raw === false) return [];
$d = json_decode($raw, true);
return is_array($d) ? $d : [];
}
// One call for the whole list, not one per title. Eleven separate requests would spend eleven
// model loads on a question that fits in a paragraph, and the model reads the set better than the
// items — "these six are all Chinese streaming platforms" is a judgement about the group.
function vv_ai_triage_classification(array $items, ?callable $ask = null): array {
if (!$items) return [];
$lines = [];
foreach ($items as $i => $x) {
$lines[] = sprintf('%d. %s — network: %s, certification: %s', $i + 1,
$x['title'] ?? '?', $x['network'] ?: 'unknown', $x['cert'] ?: 'unknown');
}
$prompt = "These television series are filed in a library folder reserved for anime. A metadata "
. "rule could not confirm any of them as anime, so each is either misfiled or a deliberate "
. "choice by the library's owner.\n\n"
. "Sort every one into exactly one bucket:\n"
. " misfiled — not animation at all, or animation with no plausible claim to the "
. "anime shelf. Live action belongs here.\n"
. " donghua — Chinese or Korean animation.\n"
. " anime_adjacent — Japanese-produced, or Western animation made in an anime style, where "
. "shelving it as anime is defensible.\n"
. " uncertain — you do not recognise it well enough to say.\n\n"
. "Answer with one line per series, exactly: NUMBER|BUCKET|a short reason.\n"
. "Use uncertain rather than guessing. Add nothing else.\n\n"
. implode("\n", $lines);
$reply = $ask ? $ask($prompt) : vv_ai_ask_model($prompt);
if ($reply === null) return [];
$out = [];
foreach (explode("\n", $reply) as $line) {
if (!preg_match('/^\s*(\d+)\s*\|\s*([a-z_]+)\s*\|\s*(.+)$/i', trim($line), $m)) continue;
$idx = (int)$m[1] - 1;
$b = strtolower(trim($m[2]));
if (!isset($items[$idx]) || !in_array($b, VV_AI_TRIAGE_BUCKETS, true)) continue;
$out[] = $items[$idx] + ['bucket' => $b, 'why' => mb_substr(trim($m[3]), 0, 160)];
}
return $out;
}
// Deliberately not streaming and deliberately not the chat worker's path: this is one bounded
// question with no conversation, no history and no capability grants, asked from a sweep that must
// exit 0 whatever happens. A failure returns null and the caller files nothing.
function vv_ai_ask_model(string $prompt, int $timeout = 120): ?string {
$cfg = vv_ai_config();
if (empty($cfg['enabled']) || empty($cfg['url'])) return null;
$ctx = stream_context_create(['http' => [
'method' => 'POST',
'header' => "Content-Type: application/json\r\n",
'content' => json_encode([
'model' => $cfg['model'],
'messages' => [['role' => 'user', 'content' => $prompt]],
'stream' => false,
'think' => false,
'options' => ['num_ctx' => 8192, 'temperature' => 0],
]),
'timeout' => $timeout,
'ignore_errors' => true,
]]);
$raw = @file_get_contents(rtrim($cfg['url'], '/') . '/api/chat', false, $ctx);
// Unpin on failure so the next resolution steps to another node that has a model. A refusal
// or a bad reply is not a failure of the endpoint — only not reaching it is, which is why
// this sits on the transport result and not on the parse below.
if ($raw === false) { vv_ai_model_failed($cfg['model_host'] ?? ''); return null; }
$d = json_decode($raw, true);
return $d['message']['content'] ?? null;
}
// Findings, one per title that is not defensible where it sits. donghua and anime_adjacent are
// answers, not problems — filing them would recreate the undifferentiated list this exists to
// break up. uncertain is filed, because "the model could not tell either" is worth knowing and is
// the honest outcome for an obscure title.
function vv_ai_classification_findings(): array {
if (!vv_ai_assist_discovery_enabled()) return [];
$vars = vv_conf_vars();
$myId = strtoupper(vv_detect_host());
$rev = vv_ai_classification_review();
$items = $rev['reverse_anime'] ?? [];
if (!$items) return [];
$found = [];
foreach (vv_ai_triage_classification($items) as $r) {
if (!in_array($r['bucket'], ['misfiled', 'uncertain'], true)) continue;
$title = (string)($r['title'] ?? '?');
// Only a misfiled series gets a destination. "Uncertain" means nobody knows where it
// belongs, and offering a move for it would turn a shrug into a button.
$target = $r['bucket'] === 'misfiled' ? (string)($vars[$myId . '_SONARR_GENERAL_ROOT'] ?? '') : '';
$found[] = [
'kind' => 'media_misfiled',
'subject' => $title,
'pin' => 'cls:' . $r['bucket'] . '|' . $title,
'ref' => 'classification',
'conf_key' => '',
'conf_file' => '',
'sys_level' => $r['bucket'] === 'misfiled' ? 'warn' : 'info',
'observed' => $r['bucket'],
'evidence' => sprintf('%s sits in %s. The metadata rule found no anime signal, and '
. 'the triage calls it %s — %s. Network %s, certification %s.%s',
$title, $r['root'] ?? 'the anime root', $r['bucket'],
$r['why'] ?? 'no reason given',
$r['network'] ?: 'unknown', $r['cert'] ?: 'unknown',
$target !== '' ? ' Moving it would relocate the files to ' . $target . '.'
: ' Nothing has been moved.'),
// What Accept would do. Carried on the finding rather than recomputed at press time,
// so the button acts on the judgement that was shown rather than on a fresh one.
'move_arr' => 'sonarr',
'move_id' => (int)($r['id'] ?? 0),
'move_from' => (string)($r['root'] ?? ''),
'move_to' => $target,
'source_log' => 'Arrs_Stack/sonarr_classification_scan',
'state' => 'needs_operator',
];
}
return $found;
}
// Relocates a series to another root folder, files and all, through Sonarr's own API.
//
// The one genuinely destructive thing this file can do, so it is deliberately awkward to reach:
// only from the 'move' action, which is only offered on a finding that recorded a destination,
// which only happens for a series the triage called misfiled. Three narrowings, each of which has
// to hold before a file is touched.
//
// moveFiles=true is the whole point — changing rootFolderPath without it leaves Sonarr pointing at
// a path where nothing lives, which is worse than the misfiling it was correcting.
//
// Re-reads the series from Sonarr rather than trusting the finding's copy. The finding may be
// hours old and the operator may have moved it by hand in the meantime; PUTting a stale record
// back would undo that silently.
function vv_ai_move_series(array $f): array {
$id = (int)($f['move_id'] ?? 0);
$to = trim((string)($f['move_to'] ?? ''));
if ($id <= 0 || $to === '') return ['ok' => false, 'error' => 'no destination recorded'];
require_once __DIR__ . '/arrs.php';
$arr = null;
foreach (vv_discover_arrs() as $node) {
foreach ($node['arrs'] as $a) if ($a['type'] === 'sonarr') { $arr = $a; break 2; }
}
if (!$arr) return ['ok' => false, 'error' => 'no sonarr configured'];
$cur = vv_arr_http($arr['url'], $arr['key'], '/api/v3/series/' . $id, 10);
if (!is_array($cur) || empty($cur['id'])) return ['ok' => false, 'error' => 'series not found'];
// Already where it should be — a no-op is success, not a failure, and saying so stops the
// finding being reopened forever by a fault that is already fixed.
if (($cur['rootFolderPath'] ?? '') === $to) {
return ['ok' => true, 'note' => 'already there'];
}
$cur['rootFolderPath'] = $to;
$ctx = stream_context_create(['http' => [
'method' => 'PUT',
'header' => "X-Api-Key: {$arr['key']}\r\nContent-Type: application/json\r\n",
'content' => json_encode($cur),
'timeout' => 30,
'ignore_errors' => true,
]]);
$res = @file_get_contents(rtrim($arr['url'], '/') . '/api/v3/series/' . $id . '?moveFiles=true',
false, $ctx);
if ($res === false) return ['ok' => false, 'error' => 'sonarr did not answer'];
$d = json_decode($res, true);
if (!is_array($d) || ($d['rootFolderPath'] ?? '') !== $to) {
return ['ok' => false, 'error' => 'sonarr did not accept the move'];
}
vv_ai_audit_move($f, $to);
return ['ok' => true];
}
// A move leaves no trace in conf_changes.log because it is not a conf change, and the arr's own
// history records it as an edit without saying who asked. Written here so "why did this series
// move" has an answer that names the finding.
function vv_ai_audit_move(array $f, string $to): void {
@file_put_contents(LOG_DIR . '/conf_changes.log',
sprintf("%s finding=%s move subject=%s from=%s to=%s ip=%s\n",
date('Y-m-d H:i:s'), $f['id'] ?? '?', $f['subject'] ?? '?',
$f['move_from'] ?? '?', $to, $_SERVER['REMOTE_ADDR'] ?? 'cli'),
FILE_APPEND | LOCK_EX);
}
// ── Reaching the operator ────────────────────────────────────────────────────────────────────
// A finding nobody is told about is a finding nobody has. The card on the AI tab shows them, but
// only to someone who opens the tab, and the point of this subsystem is that it works while
// nobody is looking.
//
// Only needs_operator is announced. An open finding may still be repaired by the next pass, and
// announcing one would be telling the operator about a problem that has already been handled by
// the time they read it. acknowledged and dismissed are the operator's own answers and are never
// announced at all.
//
// AI_REPAIR_NOTIFY_ENABLED defaults to true when absent, unlike the two switches above it. Those
// gate reading and writing, which are things to be trusted first; this gates telling someone,
// which is the point of having found anything. It still cannot fire unless AI_REPAIR_ENABLED is
// on, and notify() itself is subject to NOTIFY_UNRAID and the host's webhook.
function vv_ai_notify_enabled(): bool {
if (!vv_ai_repair_enabled()) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_NOTIFY_ENABLED'] ?? 'true'))) !== 'false';
}
// What has to change before this finding is worth mentioning twice.
//
// The same pin as an acknowledgement, plus the severity. That means a finding is announced once
// and then stays quiet — through every fifteen-minute pass, however many times it is seen — until
// either the thing it is about changes or it gets worse. "Indexers unavailable: NzbNoob" becoming
// "NzbNoob, Miatrix" is news; the same sentence for the ninth time is not.
function vv_ai_finding_notify_pin(array $f): string {
return vv_ai_finding_ack_pin($f) . '|' . (string)($f['severity'] ?? 'warn');
}
// One notification for everything that needs saying, not one per finding. Returns what it did so
// the sweep can log it and the tests can read it without a notification having to be sent.
function vv_ai_findings_announce(bool $dryRun = false): array {
$out = ['sent' => false, 'count' => 0, 'subject' => '', 'message' => '', 'ids' => []];
if (!vv_ai_notify_enabled()) return $out + ['skipped' => 'AI_REPAIR_NOTIFY_ENABLED is false'];
// Nothing switched on to receive it. Checked before anything is composed, and reported as a
// configuration state rather than a delivery failure: a failure is retried on the next pass
// and logged loudly, and neither is the right response to "no channel has been set up".
if (!vv_notify_available()) {
return $out + ['skipped' => 'no notification channel — NOTIFY_UNRAID is false and no webhook is set'];
}
$due = [];
foreach (vv_ai_findings_list(['needs_operator']) as $f) {
if (($f['notified'] ?? null) === vv_ai_finding_notify_pin($f)) continue;
$due[] = $f;
}
if (!$due) return $out;
// Worst first: if the message is truncated, the part that survives is the part that matters.
usort($due, fn($a, $b) => (($b['severity'] ?? '') === 'error' ? 1 : 0)
<=> (($a['severity'] ?? '') === 'error' ? 1 : 0));
$lines = [];
foreach (array_slice($due, 0, 3) as $f) {
// Evidence is a log excerpt and routinely spans lines. Collapsed here rather than left
// for vv_notify() to tidy on the way out: this string is also what gets logged and what
// the sweep summary returns, and only one of those three readers strips newlines.
// Collapsed before truncating, so 90 characters means 90 visible ones.
$ev = trim(preg_replace('/\s+/u', ' ', (string)($f['evidence'] ?? '')));
$lines[] = sprintf('%s - %s: %s', $f['subject'] ?? '?', $f['ref'] ?? '?',
mb_substr($ev, 0, 90));
}
if (count($due) > 3) $lines[] = sprintf('and %d more', count($due) - 3);
$n = count($due);
$anyErr = false;
foreach ($due as $f) if (($f['severity'] ?? '') === 'error') $anyErr = true;
$out['count'] = $n;
$out['subject'] = sprintf('Varaverk repair - %d finding%s need%s you', $n, $n === 1 ? '' : 's',
$n === 1 ? 's' : '');
// ' · ' rather than newlines: notify() builds its Discord payload by printf-ing into a JSON
// string literal, and a raw newline there produces a body the webhook rejects.
$out['message'] = implode(' | ', $lines);
$out['ids'] = array_column($due, 'id');
if ($dryRun) return $out;
$out['sent'] = vv_notify($out['message'], $out['subject'], $anyErr ? 'alert' : 'warning');
// Stamped only on a delivery that worked. A channel that is down should retry on the next
// pass rather than mark these as told and go quiet about them forever.
if ($out['sent']) {
foreach ($due as $f) {
$rec = vv_ai_finding_get($f['id']);
if ($rec === null) continue;
$rec['notified'] = vv_ai_finding_notify_pin($rec);
$p = vv_ai_finding_path($f['id']);
if ($p !== null) @file_put_contents($p, json_encode($rec, JSON_PRETTY_PRINT));
}
}
return $out;
}
// What the assistant should raise when a page loads: things that need the operator, newest
// first. Repaired findings are deliberately not here — a fix that worked is a log entry, not a
// conversation, and opening every session with a list of things that already went right is how
// an operator learns to close the panel without reading it.
function vv_ai_findings_for_chat(int $limit = 3): array {
return array_slice(vv_ai_findings_list(['needs_operator']), 0, max(1, $limit));
}
// ── The sweep ────────────────────────────────────────────────────────────────────────────────
// There is no post-run hook in Varaverk — nothing fires when a job finishes. Rather than add a
// call to forty scripts, this picks up run records that completed since the last pass. One entry
// in an orchestrator's list instead of forty edits, and it batches naturally.
//
// Runs that reported ok are read too. A container failing its HTTP check warns and leaves the
// watchdog exiting 0, so "only look at failures" would miss the whole class of fault this exists
// for: the job worked, and told you something is wrong.
function vv_ai_sweep_marker_path(): string {
return STATE_DIR . '/ai_repair_sweep.db';
}
function vv_ai_sweep_last(): int {
return (int)trim((string)@file_get_contents(vv_ai_sweep_marker_path()));
}
function vv_ai_sweep_mark(int $ts): void {
if (!is_dir(STATE_DIR)) @mkdir(STATE_DIR, 0755, true);
@file_put_contents(vv_ai_sweep_marker_path(), (string)$ts, LOCK_EX);
}
// Run records that finished after $since. A record still marked running is skipped rather than
// read half-written — it will be picked up on the pass after it finishes.
function vv_ai_recent_runs(int $since): array {
$out = [];
$base = realpath(LOG_DIR);
if ($base === false) return [];
foreach ((array)@glob($base . '/{,*/,*/*/,*/*/*/}*.json', GLOB_BRACE) as $path) {
$r = json_decode((string)@file_get_contents($path), true);
if (!is_array($r) || empty($r['id']) || ($r['status'] ?? '') === 'running') continue;
$end = (int)($r['end'] ?? 0);
if ($end <= $since) continue;
$log = preg_replace('/\.json$/', '.log', $path);
if (!is_file($log)) continue;
$out[] = ['id' => (string)$r['id'], 'status' => (string)($r['status'] ?? '?'),
'start' => (int)($r['start'] ?? 0), 'end' => $end, 'log' => $log];
}
usort($out, fn($a, $b) => $a['end'] <=> $b['end']);
return $out;
}
// The lines one run wrote, and only those. Logs are appended across runs, so a tail alone would
// re-read the previous run's output and re-report faults that have already been dealt with.
// Filtering on the leading timestamp scopes the evidence to the run being examined.
function vv_ai_run_log_lines(string $logPath, int $startTs, int $maxLines = 2000): array {
$out = []; $rc = 0;
@exec('tail -n ' . (int)$maxLines . ' ' . escapeshellarg($logPath) . ' 2>/dev/null', $out, $rc);
if ($rc !== 0) return [];
$kept = [];
foreach ($out as $line) {
if (preg_match('/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/', $line, $m)) {
// A line older than the run belongs to a previous one. Two seconds of slack because
// the record's start is stamped by the runner, not by the first line the job writes.
if (strtotime($m[1]) < $startTs - 2) continue;
}
$kept[] = $line;
}
return $kept;
}
// One pass. Returns a summary rather than logging it, so the caller decides what to record and
// the whole thing stays testable without a log to read afterwards.
//
// $dryRun does everything except write conf and move the marker — including probing, which is
// the point: it answers "what would this have done" with real evidence rather than a guess.
function vv_ai_repair_sweep(bool $dryRun = false): array {
if (!vv_ai_repair_enabled()) {
return ['ok' => false, 'error' => 'AI_REPAIR_ENABLED is not true', 'runs' => 0];
}
$started = time();
$since = vv_ai_sweep_last();
$runs = vv_ai_recent_runs($since);
$sum = ['ok' => true, 'runs' => count($runs), 'findings' => 0, 'fixed' => 0,
'needs_operator' => 0, 'resolved' => 0, 'quiet' => 0, 'details' => []];
// Candidates from four sources. Log triage is bounded to runs that finished since the last
// pass; the arrs are asked every time, because their health is a current state rather than
// something that appeared in a log once. Asking costs three local HTTP calls.
//
// The system log is bounded the same way the run logs are — everything since the marker —
// so a fault that has been shouting for an hour is counted once per pass rather than once
// per line, and stops producing candidates the moment it stops being logged.
//
// The watchdog counters are asked every time for the same reason the arrs are: a strike count
// is a current state, not something that appeared in a log once. It is also the cheapest of
// the four — one cached file read — and gated separately, on AI_ASSIST_WATCHDOG.
$candidates = vv_ai_arr_health_findings();
foreach (vv_ai_syslog_findings($since) as $c) $candidates[] = $c;
foreach (vv_ai_container_findings($since) as $c) $candidates[] = $c;
foreach (vv_ai_watchdog_findings() as $c) $candidates[] = $c;
// The only source here that asks the model anything. Gated separately on AI_ASSIST_DISCOVERY,
// and it does nothing at all unless the classification scan has left a review file — so on a
// host that never runs that scan this costs one failed file read.
foreach (vv_ai_classification_findings() as $c) $candidates[] = $c;
foreach ($runs as $run) {
$lines = vv_ai_run_log_lines($run['log'], $run['start']);
if (!$lines) continue;
$rel = ltrim(str_replace(realpath(LOG_DIR), '', $run['log']), '/');
foreach (vv_ai_triage_log($lines, $rel) as $c) $candidates[] = $c;
}
foreach ($candidates as $cand) {
$cand = vv_ai_probe_finding($cand);
$sum['findings']++;
// Write first, so a finding exists even if the repair below fails. A repair that
// errored without leaving a record is the one failure mode there is no way back from.
$w = vv_ai_finding_write($cand);
if (!($w['ok'] ?? false)) continue;
$id = $w['id'];
// Already acknowledged or dismissed — the operator has spoken, and re-fixing behind
// them would be the opposite of what an acknowledgement means.
if (in_array($w['state'] ?? '', ['acknowledged', 'dismissed'], true)) {
$sum['quiet']++;
continue;
}
if (($cand['state'] ?? '') === 'resolved') {
vv_ai_finding_close($id, (string)($cand['note'] ?? ''));
$sum['resolved']++;
continue;
}
if (vv_ai_finding_may_autofix($cand)) {
if ($dryRun) { $sum['details'][] = "would fix {$cand['conf_key']} → {$cand['proposed']}"; continue; }
$r = vv_ai_finding_apply_action($id, 'fix', 'Probed and written by the repair sweep.');
if ($r['ok'] ?? false) { $sum['fixed']++; $sum['details'][] = "fixed {$cand['conf_key']}"; }
else { $sum['needs_operator']++; vv_ai_finding_set_state($id, 'needs_operator', (string)($r['error'] ?? '')); }
continue;
}
if (($cand['state'] ?? '') === 'needs_operator') $sum['needs_operator']++;
}
// Marked only on a completed pass, and to when the pass began — a job that finished while
// this was running is then picked up next time instead of being skipped for having ended
// before a marker written at the end.
if (!$dryRun) vv_ai_sweep_mark($started);
// Announced after the marker rather than before, and over the whole store rather than only
// what this pass touched. A finding that reached needs_operator two passes ago and was never
// successfully delivered is still owed to the operator, and the pin is what stops that from
// meaning it is announced twice.
$sum['announced'] = vv_ai_findings_announce($dryRun);
return $sum;
}
// ── Answering a finding in words ─────────────────────────────────────────────────────────────
// The buttons are unambiguous by construction. This is for the other path — replying "yeah go
// ahead" in the chat that raised the finding — and it is matched here rather than asked of the
// model, because the model's answer would be a conf write and a wrong reading of "no, leave it"
// is not recoverable by apologising.
//
// Same shape as vv_ai_route_from_chat(): anchored patterns, most specific first, and anything
// unrecognised returns null so the assistant asks again instead of guessing. Two actions both
// matching is also null — "leave it, I know" and "leave it for now" differ by one clause and
// mean different things, so a phrase that supports both is not an instruction yet.
const VV_AI_ACTION_PATTERNS = [
// Acknowledge — "this is deliberate, stop telling me".
'ack' => [
'/\b(i|we) know\b/u',
'/\b(that|this|it)(?:\'s| is) (fine|expected|intentional|deliberate|on purpose)\b/u',
'/\bon purpose\b/u',
'/\b(aware|acknowledge|ack)\b/u',
'/\bmeant to be\b/u',
],
// Apply the proposed value.
'fix' => [
'/\bfix (it|that|this|them)?\b/u',
'/\b(go ahead|do it|apply|make the change|change it|update it|correct it)\b/u',
// A bare affirmative, as the whole message — "yes" answering "shall I fix it" is an
// instruction, "yes it looks wrong" is agreement about the diagnosis and nothing more.
// A trailing please is still bare.
'/\b(yes|yeah|yep|yup|sure|ok|okay)\b(\s*,?\s*please)?[\s,.!]*$/u',
'/\bplease do\b/u',
],
// Not now — no state written, it comes back next sweep.
'cancel' => [
// "leave it" is matched bare, not only as "leave it alone" / "leave it for now". The
// ambiguity this function is built to refuse — "leave it, I know" reading as both cancel
// and ack — did not actually arise with the longer forms, so that sentence resolved to
// ack and silenced the finding until the fault changed. The looser pattern is what makes
// the two readings collide and sends it back to be restated.
'/\b(not now|later|leave it\b|skip( it)?|cancel|ignore for now)\b/u',
'/\b(no|nope|nah)\b[\s,.!]*$/u',
'/\b(don\'?t|do not) (fix|touch|change|write|apply)\b/u',
],
// Never a problem. Deliberately narrow: this is the one answer that cannot expire on its
// own, so it is only read from a sentence that says so outright. Anything vaguer than these
// is meant to land on ack, which comes back by itself when the fault changes.
'dismiss' => [
'/\bdismiss\b/u',
'/\b(this|that|it)(?:\'s| is) not (a |an )?(problem|bug|issue|real)\b/u',
'/\bnever (a problem|an issue|report this)\b/u',
],
// Undo a close.
'reopen' => [
'/\breopen\b/u',
'/\bun-?dismiss\b/u',
],
];
// Returns one of the keys in VV_AI_ACTION_PATTERNS, or null when the reply does not clearly mean
// exactly one of them.
//
// Only call this when a finding is actually pending. A bare "yes" means fix in answer to "shall
// I fix it" and means nothing at all on its own, and the difference is context this function
// cannot see.
function vv_ai_finding_action_from_text(string $text): ?string {
$t = strtolower(trim($text));
if ($t === '') return null;
$t = preg_replace('/\s+/', ' ', $t);
$matched = [];
foreach (VV_AI_ACTION_PATTERNS as $action => $patterns) {
foreach ($patterns as $re) {
if (preg_match($re, $t)) { $matched[$action] = true; break; }
}
}
// Exactly one reading, or none. "leave it, I know" hits both ack and cancel; that is a
// sentence the operator should be asked to restate, not one to pick a winner from.
return count($matched) === 1 ? array_key_first($matched) : null;
}
// Carry out an answered action against a stored finding.
//
// Fix goes through the same guarded write path as everything else, and is the one place a
// toggle may be written — because reaching here means the operator asked for it by name. The
// unattended sweep never calls this.
// Records which node answered a finding and how. Separate from the action itself so every route
// through apply_action() is stamped the same way — a switch arm that forgot would be a finding
// with an outcome and no author, which is exactly the record the mesh needs and the hardest kind
// of gap to notice afterwards.
function vv_ai_finding_stamp_actor(string $id, string $action): bool {
$r = vv_ai_finding_get($id);
if ($r === null) return false;
$r['acted_by'] = vv_detect_host();
$r['acted_at'] = time();
$r['acted'] = $action;
$p = vv_ai_finding_path($id);
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
}
function vv_ai_finding_apply_action(string $id, string $action, string $note = ''): array {
$f = vv_ai_finding_get($id);
if ($f === null) return ['ok' => false, 'error' => 'no such finding'];
// Only what this finding actually offers, in the state it is actually in. The page renders
// its buttons from the same function, but a stale tab holds buttons the store has moved past
// — a row acked in one window is still showing Fix in another — and the endpoint is reachable
// without either. Checking here is what makes vv_ai_finding_actions() the authority rather
// than a suggestion.
if (!isset(vv_ai_finding_actions($f)[$action])) {
return ['ok' => false, 'error' => 'not offered for this finding: ' . $action];
}
// Stamped before the action runs, not after, so a move that half-succeeds still records who
// asked for it. 'cancel' is excluded because it is the button for changing your mind, and
// recording it would make "nobody did anything" look like a decision.
//
// Deliberately the node, not a person: this interface authenticates as root and has no user
// to name. In a mesh the useful question is which machine answered — one node dismissing what
// another raised is the case worth being able to reconstruct.
if ($action !== 'cancel') {
vv_ai_finding_stamp_actor($id, $action);
$f = vv_ai_finding_get($id) ?? $f;
}
switch ($action) {
case 'ack':
return ['ok' => vv_ai_finding_ack($id, $note), 'action' => 'ack'];
case 'dismiss':
return ['ok' => vv_ai_finding_dismiss($id, $note), 'action' => 'dismiss'];
// Back to open, never straight back to needs_operator: whether it still cannot be
// repaired here is the next sweep's finding to make, not a state to restore.
// The only action in this file that changes something outside conf. It relocates files on
// disk through Sonarr, which is why it is a button and not something a sweep decides:
// vv_ai_move_series() is reached from here and from nowhere else.
case 'move':
$r = vv_ai_move_series($f);
if (!($r['ok'] ?? false)) return $r + ['action' => 'move'];
// Closed rather than acked. Acked means "known and intended, stay quiet"; this one
// was acted on, and the record should say the fault is gone rather than tolerated.
vv_ai_finding_close($id, 'Moved to ' . ($f['move_to'] ?? '?') . '. ' . $note);
return ['ok' => true, 'action' => 'move', 'moved_to' => $f['move_to'] ?? ''];
case 'reopen':
return ['ok' => vv_ai_finding_set_state($id, 'open', $note), 'action' => 'reopen'];
case 'cancel':
// Deliberately writes nothing at all. "Not now" is not a state, it is the absence of
// one — recording it would make the finding look decided when it is still open.
return ['ok' => true, 'action' => 'cancel'];
case 'fix':
$proposed = $f['proposed'] ?? null;
if ($proposed === null) return ['ok' => false, 'error' => 'nothing proposed to write'];
$key = (string)$f['conf_key'];
$ok = vv_conf_write_changes([[
'file' => (string)($f['conf_file'] ?? 'master.conf'),
'key' => $key,
'value' => (string)$proposed,
'type' => 'scalar',
]]);
$wrote = !in_array(false, $ok, true);
if ($wrote) {
vv_ai_finding_set_state($id, 'fixed',
$note !== '' ? $note : 'Wrote ' . $key . ' at the operator\'s request.');
}
return ['ok' => $wrote, 'action' => 'fix',
'error' => $wrote ? null : 'conf write refused — see conf_changes.log'];
}
return ['ok' => false, 'error' => 'unknown action'];
}
// ── Resolving a log line back to a conf key ──────────────────────────────────────────────────
// By value wherever possible, by name only as a fallback.
//
// A log line usually contains the thing that failed — the URL that did not answer. That value
// came from a conf key, so searching the conf for which key holds it is an exact lookup with a
// definite answer. Guessing the key from the container's name is inference, and the failure mode
// is silent: HOST1_EMBY_URL and HOST1_EMBY_EXTERNAL_URL are both plausible for "Emby" and only
// one of them is the value that just failed.
//
// Same principle as resolve_tailscale_ip() refusing similarity matching for host identity: an
// exact match or an honest nothing.
// Every conf key whose value the given text starts with, longest first. Prefix rather than
// equality because a log reports the URL it actually called — the conf value plus an endpoint
// path — and the longest match is the most specific key that could have produced it.
function vv_ai_conf_keys_for_value(string $value): array {
$value = trim($value);
if ($value === '' || strlen($value) < 6) return [];
$hits = [];
foreach (vv_conf_vars() as $k => $v) {
$v = trim((string)$v);
if ($v === '' || strlen($v) < 6) continue;
if ($v === $value || str_starts_with($value, $v)) $hits[$k] = strlen($v);
}
arsort($hits);
return array_keys($hits);
}
// <HOSTID>_<SUBJECT>_<SUFFIX>, built literally and then checked for existence. Nothing is
// inferred: either the conf holds a key by exactly that name or this returns null.
function vv_ai_conf_key_for_subject(string $subject, string $suffix): ?string {
$norm = strtoupper(preg_replace('/[^A-Za-z0-9]+/', '_', trim($subject)));
if ($norm === '') return null;
$key = strtoupper(vv_detect_host()) . '_' . $norm . '_' . strtoupper($suffix);
return array_key_exists($key, vv_conf_vars()) ? $key : null;
}
// Which conf file a key lives in. A finding has to name the file it would be edited in, and
// host keys are not in master.conf.
function vv_ai_conf_file_for_key(string $key): string {
foreach (vv_get_conf_files() as $f) {
if (preg_match('/^\s*' . preg_quote($key, '/') . '\s*=/m', vv_read_conf_raw($f))) return $f;
}
return 'master.conf';
}
// ── Deterministic triage ─────────────────────────────────────────────────────────────────────
// Patterns are written against log formats that exist in this repo today, taken from the emit
// sites rather than imagined. A pattern that stops matching because its log line was reworded
// produces no finding, which is the safe direction — the alternative is a pattern loose enough
// to match anything, which fills the store with rows nobody can act on.
//
// No model runs here. "Connection refused" is not a judgement call, and a 14B model invoked
// after every cron job to notice it would be slower, costlier and less reliable than a regex.
// The model's job starts where these stop: explaining a finding, and talking the operator
// through the ones that cannot be repaired automatically.
const VV_AI_TRIAGE_PATTERNS = [
// docker_watchdog HTTP check — the richest signal, carrying both subject and failing URL.
[
're' => '/^(?P<subject>\S+) — not responding at (?P<observed>\S+) \(strike/u',
'kind' => 'unreachable',
'suffix' => 'URL',
],
// docker_watchdog API check, 401/403. The endpoint answered, so the address is right and
// the credential is not.
[
're' => '/^(?P<subject>\S+) — API check skipped \(HTTP (?:401|403)/u',
'kind' => 'auth_rejected',
'suffix' => 'API_KEY',
],
// Configured for an API check with nothing to authenticate with.
[
're' => '/^(?P<subject>\S+) — API check skipped \(no key configured\)/u',
'kind' => 'missing_value',
'suffix' => 'API_KEY',
],
[
're' => '/^Skipping (?P<subject>\S+) — placeholder API key/u',
'kind' => 'missing_value',
'suffix' => 'API_KEY',
],
[
're' => '/^(?P<subject>\S+) (?:—\s*)?API unreachable/u',
'kind' => 'unreachable',
'suffix' => 'URL',
],
];
// One log line in, at most one finding candidate out. Returns null for everything else, which is
// almost every line.
function vv_ai_triage_line(string $line): ?array {
// Strip the timestamp and level decoration the logger adds, so patterns can anchor on ^.
$body = preg_replace('/^\S+\s+\S+\s+(?:[^\[]*\[[A-Z]+\]\s*)?/u', '', rtrim($line));
$body = trim((string)$body);
if ($body === '') return null;
foreach (VV_AI_TRIAGE_PATTERNS as $p) {
if (!preg_match($p['re'], $body, $m)) continue;
$subject = trim($m['subject'] ?? '');
$observed = trim($m['observed'] ?? '');
if ($subject === '') continue;
// Value first, name second. Only one candidate key is accepted — two keys holding the
// same value means the log cannot say which one produced it, and picking either is the
// guess this whole approach exists to avoid.
$key = null;
if ($observed !== '') {
$byValue = vv_ai_conf_keys_for_value($observed);
if (count($byValue) === 1) $key = $byValue[0];
}
if ($key === null) $key = vv_ai_conf_key_for_subject($subject, $p['suffix']);
if ($key === null) continue; // nothing actionable — no row
return [
'kind' => $p['kind'],
'subject' => $subject,
'conf_key' => $key,
'conf_file' => vv_ai_conf_file_for_key($key),
'observed' => $observed !== '' ? $observed : (string)(vv_conf_vars()[$key] ?? ''),
'evidence' => $body,
];
}
return null;
}
// A whole log tail in, one candidate per distinct fault out. A job that retried twelve times
// produces twelve identical lines, and the store's dedupe would collapse them anyway — doing it
// here keeps the sweep from writing the same file twelve times in a row.
function vv_ai_triage_log(array $lines, string $sourceLog = ''): array {
$found = [];
foreach ($lines as $line) {
$c = vv_ai_triage_line((string)$line);
if ($c === null) continue;
$c['source_log'] = $sourceLog;
$found[vv_ai_finding_id($c['kind'], $c['subject'], $c['conf_key'])] = $c;
}
return array_values($found);
}
// ── The phrasebook ───────────────────────────────────────────────────────────────────────────
// What the operator said, what it was taken to mean, and whether that was right.
//
// The point is not to fine-tune anything. It is that "daily", said three times and meaning
// daily_sync_maintenance.sh all three, stops being an inference and becomes a lookup. This file
// is how a term earns that promotion, and everything it promotes is exact — the model keeps the
// language, the resolution stays deterministic. Same division as the rest of this subsystem.
//
// The corrections are the rows that matter. A resolution that was right confirms what was
// already believed; a resolution that was wrong, with what it should have been, is the only
// record of a mistake that would otherwise be repeated indefinitely.
//
// JSON Lines rather than the pipe-delimited shape the token ledger uses: that file holds numbers
// and a hostname, this one holds whatever the operator typed, and a delimiter that occurs in the
// data is not a delimiter. Append-only, one object per line, so a truncated write costs the last
// row rather than the corpus.
function vv_ai_phrasebook_path(): string {
return AI_DATA_DIR . '/ai_phrasebook.jsonl';
}
// $said what the operator actually wrote, verbatim but redacted
// $term the fragment that carried the meaning — "daily", "critical rsync", "emby key"
// $target what it resolved to: a conf key, a script id, an array name
// $kind conf_key | script | array | section | unknown
// $outcome accepted | corrected | rejected
// $correctedTo what it should have been, when the resolution was wrong
//
// $target and $correctedTo must be a canonical identifier — CONF_BACKUP_DIR, not
// "CONF_BACKUP_DIR=${DATA_DIR}/Backups/Confs" and not "the backups directory". Promotion works
// by counting how often a term resolved to the same thing, so a target described three different
// ways is three meanings, and the term never promotes. Learned the hard way while seeding this
// with real corrections: the same fix, written up three ways, taught nothing.
function vv_ai_phrase_record(string $said, string $term, string $target, string $kind,
string $outcome = 'accepted', string $correctedTo = ''): bool {
$said = trim($said);
$term = strtolower(trim($term));
if ($said === '' || $term === '') return false;
if (!in_array($outcome, ['accepted', 'corrected', 'rejected'], true)) return false;
if (!is_dir(AI_DATA_DIR)) @mkdir(AI_DATA_DIR, 0755, true);
// An operator asking to set a credential types the credential. This file is long-lived and
// read back for years; it is the last place a key should be preserved verbatim.
$row = [
'ts' => time(),
'said' => mb_substr(vv_ai_redact($said), 0, 500),
'term' => mb_substr($term, 0, 80),
'target' => mb_substr(trim($target), 0, 160),
'kind' => $kind,
'outcome' => $outcome,
];
if ($correctedTo !== '') $row['corrected_to'] = mb_substr(vv_ai_redact($correctedTo), 0, 160);
return @file_put_contents(vv_ai_phrasebook_path(),
json_encode($row, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE) . "\n",
FILE_APPEND | LOCK_EX) !== false;
}
function vv_ai_phrase_all(): array {
$out = [];
foreach ((array)@file(vv_ai_phrasebook_path(), FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES) as $line) {
$r = json_decode($line, true);
if (is_array($r) && isset($r['term'])) $out[] = $r;
}
return $out;
}
// Terms that have earned a deterministic mapping: seen at least $minSeen times, resolving to one
// target every time, and never corrected away from it.
//
// A single contradiction disqualifies the term outright rather than going with the majority. The
// whole value of promoting a term is that it stops being a guess — a term that meant two things
// is still a guess, and a confident wrong alias is worse than no alias, because nothing downstream
// will question it.
function vv_ai_phrase_aliases(int $minSeen = 3): array {
$seen = [];
foreach (vv_ai_phrase_all() as $r) {
$term = (string)$r['term'];
// A correction records both the wrong reading and the right one. The right one is what
// the term means; the wrong one is what it must never be promoted to again.
$target = ($r['outcome'] === 'corrected' && !empty($r['corrected_to']))
? (string)$r['corrected_to'] : (string)$r['target'];
if ($r['outcome'] === 'rejected' || $target === '') continue;
$seen[$term]['targets'][$target] = ($seen[$term]['targets'][$target] ?? 0) + 1;
$seen[$term]['kind'] = (string)($r['kind'] ?? 'unknown');
if ($r['outcome'] === 'corrected') $seen[$term]['wrong'][(string)$r['target']] = true;
}
$aliases = [];
foreach ($seen as $term => $d) {
if (count($d['targets']) !== 1) continue; // meant two things — still a guess
$target = array_key_first($d['targets']);
if (isset($d['wrong'][$target])) continue; // was itself corrected away from
if ($d['targets'][$target] < $minSeen) continue;
$aliases[$term] = ['target' => $target, 'kind' => $d['kind'], 'seen' => $d['targets'][$target]];
}
return $aliases;
}
// Exact alias hit for a term, or null. Deliberately not fuzzy: an alias exists precisely so that
// this lookup is certain, and a near-match would reintroduce the guessing it replaced.
function vv_ai_phrase_lookup(string $term, int $minSeen = 3): ?array {
return vv_ai_phrase_aliases($minSeen)[strtolower(trim($term))] ?? null;
}
// ── What the repair profile is given before it answers ───────────────────────────────────────
// Two records of what has already happened, assembled as plain text for the prompt.
//
// The phrasebook half is the operator's vocabulary — and the corrections matter more than the
// settled terms, because a term that has been corrected is one the assistant has already got
// wrong once and would otherwise get wrong again.
//
// The closed-findings half is this installation's history: the same fault, and what actually
// ended it. "Emby stopped answering last month and the address had changed" is worth more at the
// start of a repair conversation than any amount of reasoning from first principles.
//
// Bounded hard. This goes into a 16k context that retrieval and a log tail are also competing
// for, and an unbounded history would crowd out the evidence for the fault actually being
// discussed.
function vv_ai_repair_context(int $maxAliases = 20, int $maxFixes = 8): string {
$out = [];
$aliases = vv_ai_phrase_aliases();
if ($aliases) {
$lines = [];
foreach (array_slice($aliases, 0, $maxAliases, true) as $term => $a) {
$lines[] = ' "' . $term . '" means ' . $a['target'];
}
$out[] = "What the operator calls things:\n" . implode("\n", $lines);
}
$unsettled = vv_ai_phrase_unsettled();
if ($unsettled) {
$lines = [];
foreach (array_slice($unsettled, 0, $maxAliases, true) as $term => $c) {
$last = end($c);
$lines[] = ' "' . $term . '" was read as ' . ($last['took_it_as'] ?: '?')
. ' and meant ' . ($last['meant'] ?: '?');
}
$out[] = "Corrected before — do not repeat these:\n" . implode("\n", $lines);
}
$closed = vv_ai_findings_list(['fixed', 'resolved']);
if ($closed) {
$lines = [];
foreach (array_slice($closed, 0, $maxFixes) as $f) {
$lines[] = ' ' . ($f['subject'] ?? '?') . ' / ' . ($f['ref'] ?? $f['conf_key'] ?? '?')
. ' — ' . ($f['state'] ?? '?')
. (!empty($f['note']) ? ': ' . mb_substr((string)$f['note'], 0, 140) : '');
}
$out[] = "Already dealt with on this host:\n" . implode("\n", $lines);
}
$spellings = vv_ai_spellings();
if ($spellings) {
$lines = [];
foreach (array_slice($spellings, 0, $maxAliases, true) as $typo => $meant) {
$lines[] = ' "' . $typo . '" = "' . $meant . '"';
}
$out[] = "The operator types quickly; known shorthand:\n" . implode("\n", $lines);
}
return implode("\n\n", $out);
}
// ── Resolving what the operator named ────────────────────────────────────────────────────────
// "turn off the zfs scrub", "change the emby api key" — a phrase in, an exact target out, or an
// honest refusal with the candidates that were considered.
//
// Four layers, most certain first, and each either answers exactly or declines:
//
// 1. A learned alias. The operator has used this term before and it settled on one target.
// 2. The literal key. "HOST1_EMBY_URL" or "host1 emby url" is not a phrase to interpret.
// 3. Every conf key containing all of the significant words. Exactly one is an answer; more
// than one is a question.
// 4. Script ids from the orchestrator arrays, matched the same way.
//
// What this deliberately does not do is score similarity. There is no closest match, no edit
// distance, no "did you mean". The same reasoning as resolve_tailscale_ip() refusing to guess at
// host identity: the failure mode of a near-match is silent and confident, and here it would
// write to a key the operator never named. Ambiguity is returned as ambiguity, and the assistant
// asks — which is also how the phrasebook learns, because the answer to that question is a
// correction worth recording.
//
// Stop words exist because "the", "for" and "in" appear in a conf key somewhere and would make
// every phrase match everything.
const VV_AI_RESOLVE_STOPWORDS = [
'the','a','an','to','for','in','on','of','and','or','is','it','this','that','my','our',
'please','can','you','set','change','turn','make','update','put','value','key','conf','config',
'setting','settings','off','on_','now','back','again','me','we','let','lets',
];
// A key or script id broken into its own words. HOST1_EMBY_API_KEY is four words, and
// Tools/zfs_pool_scrub.sh is five — the path separator and the extension are word boundaries too.
function vv_ai_resolve_segments(string $name): array {
$n = strtolower(preg_replace('/\.sh$/', '', $name));
return array_values(array_filter(preg_split('/[^a-z0-9]+/', $n, -1, PREG_SPLIT_NO_EMPTY) ?: []));
}
function vv_ai_resolve_tokens(string $phrase): array {
$p = strtolower(trim($phrase));
$p = preg_replace('/[^a-z0-9_\s-]+/', ' ', $p);
$words = preg_split('/[\s_-]+/', $p, -1, PREG_SPLIT_NO_EMPTY) ?: [];
return array_values(array_filter($words,
fn($w) => strlen($w) > 1 && !in_array($w, VV_AI_RESOLVE_STOPWORDS, true)));
}
// Script ids named in any *_SCRIPTS array, so "zfs scrub" can resolve to Tools/zfs_pool_scrub.sh
// — which is the shape of request that has no conf key at all.
function vv_ai_resolve_script_ids(): array {
$ids = [];
foreach (vv_get_conf_files() as $f) {
if (preg_match_all('/^\s*#?\s*"([A-Za-z0-9_\/.-]+\.sh)(?:\s[^"]*)?"/m',
vv_read_conf_raw($f), $m)) {
foreach ($m[1] as $id) $ids[$id] = true;
}
}
return array_keys($ids);
}
// Returns:
// ok=true with target, kind and via — one certain answer
// ok=false with candidates — several, and the caller must ask
// ok=false with candidates empty — nothing recognised
function vv_ai_resolve_target(string $phrase): array {
$none = ['ok' => false, 'target' => null, 'kind' => null, 'via' => 'none', 'candidates' => []];
$tokens = vv_ai_resolve_tokens($phrase);
if (!$tokens) return $none;
// 1 — learned
$alias = vv_ai_phrase_lookup(strtolower(trim($phrase)));
if ($alias === null) {
// Also try the significant words alone, since "turn off ai repair" and "ai repair" are
// the same instruction with different framing.
$alias = vv_ai_phrase_lookup(implode(' ', $tokens));
}
if ($alias !== null) {
return ['ok' => true, 'target' => $alias['target'], 'kind' => $alias['kind'],
'via' => 'alias', 'candidates' => []];
}
$vars = vv_conf_vars();
// 2 — the literal key, however it was spaced or cased
$literal = strtoupper(implode('_', $tokens));
if (array_key_exists($literal, $vars)) {
return ['ok' => true, 'target' => $literal, 'kind' => 'conf_key',
'via' => 'exact', 'candidates' => []];
}
// 3 — conf keys whose own words include every significant word
//
// Whole segments, not substrings. Substring matching made "mov" resolve to
// MOVER_STOP_TIMEOUT with full confidence, which is exactly the near-match this is supposed
// to refuse: a short fragment that happens to be unique is not the operator naming a key.
$keyHits = [];
foreach (array_keys($vars) as $key) {
$segs = vv_ai_resolve_segments($key);
foreach ($tokens as $t) {
if (!in_array($t, $segs, true)) continue 2;
}
$keyHits[] = $key;
}
if (count($keyHits) === 1) {
return ['ok' => true, 'target' => $keyHits[0], 'kind' => 'conf_key',
'via' => 'match', 'candidates' => []];
}
// 4 — script ids, same rule
$scriptHits = [];
foreach (vv_ai_resolve_script_ids() as $id) {
$segs = vv_ai_resolve_segments($id);
foreach ($tokens as $t) {
if (!in_array($t, $segs, true)) continue 2;
}
$scriptHits[] = $id;
}
if (!$keyHits && count($scriptHits) === 1) {
return ['ok' => true, 'target' => $scriptHits[0], 'kind' => 'script',
'via' => 'match', 'candidates' => []];
}
$all = array_merge($keyHits, $scriptHits);
if (!$all) return $none;
// Several. Returned rather than ranked — picking one here is the guess this avoids.
sort($all);
return ['ok' => false, 'target' => null, 'kind' => null, 'via' => 'ambiguous',
'candidates' => array_slice($all, 0, 12)];
}
// ── Spellings ────────────────────────────────────────────────────────────────────────────────
// The operator types quickly and knows it: "haversync" for "have rsync", "as it to the list" for
// "add it". Recorded when the meaning was obvious in context, so the next occurrence is read
// rather than puzzled over.
//
// Promoted on first sighting, unlike a term alias, and the difference is deliberate. A term alias
// decides what gets written to conf, so it has to be earned by repetition. A spelling decides how
// a sentence is read, costs a misreading at worst, and is rarely made identically three times —
// requiring repetition would mean never learning any of them.
//
// Recorded only when the intent was actually clear. A guess written down here is worse than
// leaving it out, because it will be applied silently every time afterwards.
function vv_ai_spelling_record(string $said, string $typo, string $meant): bool {
$typo = strtolower(trim($typo));
$meant = trim($meant);
if ($typo === '' || $meant === '' || strcasecmp($typo, $meant) === 0) return false;
return vv_ai_phrase_record($said, $typo, $meant, 'spelling');
}
// What the operator most likely meant by a word, or null. Last writing wins, so a correction to
// an earlier reading simply supersedes it.
function vv_ai_spelling_lookup(string $word): ?string {
$word = strtolower(trim($word));
$hit = null;
foreach (vv_ai_phrase_all() as $r) {
if (($r['kind'] ?? '') !== 'spelling' || ($r['outcome'] ?? '') === 'rejected') continue;
if (($r['term'] ?? '') === $word) $hit = (string)$r['target'];
}
return $hit;
}
function vv_ai_spellings(): array {
$out = [];
foreach (vv_ai_phrase_all() as $r) {
if (($r['kind'] ?? '') !== 'spelling' || ($r['outcome'] ?? '') === 'rejected') continue;
$out[(string)$r['term']] = (string)$r['target'];
}
ksort($out);
return $out;
}
// Terms that have been corrected and have not yet earned promotion — what the assistant is still
// getting wrong, and the thing worth reading when asking why it keeps mistaking something.
function vv_ai_phrase_unsettled(): array {
$out = [];
foreach (vv_ai_phrase_all() as $r) {
if (($r['outcome'] ?? '') !== 'corrected') continue;
$out[(string)$r['term']][] = ['said' => $r['said'], 'took_it_as' => $r['target'],
'meant' => $r['corrected_to'] ?? '', 'ts' => $r['ts'] ?? 0];
}
return $out;
}
// ── What the arrs say about themselves ───────────────────────────────────────────────────────
// Sonarr, Radarr and Lidarr each publish a health endpoint listing what they believe is wrong,
// already structured and already graded. No log parsing, no pattern that goes stale when a
// message is reworded, and no guessing at severity — the arr is the authority on whether its own
// condition is an error or a warning.
//
// This is the one source here that needs no triage at all. Everything else in this file exists
// because logs are prose; these arrive as records.
//
// Note the API version differs: Lidarr is v1 where Sonarr and Radarr are v3. vv_discover_arrs()
// already carries it per arr, which is why this reads it rather than assuming.
function vv_ai_arr_health_findings(): array {
if (!function_exists('vv_discover_arrs')) {
require_once __DIR__ . '/arrs.php';
}
$host = vv_detect_host();
$found = [];
foreach (vv_discover_arrs() as $node) {
if (($node['host'] ?? '') !== $host) continue;
foreach ($node['arrs'] ?? [] as $arr) {
$type = ucfirst((string)($arr['type'] ?? ''));
$url = (string)($arr['url'] ?? '');
$key = (string)($arr['key'] ?? '');
$api = (string)($arr['api'] ?? 'v3');
if ($type === '' || $url === '' || $key === '') continue;
$items = vv_arr_http($url, $key, "/api/$api/health", vv_ai_probe_timeout());
// null is unreachable, which is a different finding and one the log triage already
// raises. An empty array is the arr saying it is fine, and must not be confused with
// not having been able to ask.
if (!is_array($items)) continue;
foreach ($items as $item) {
$source = trim((string)($item['source'] ?? ''));
$message = trim((string)($item['message'] ?? ''));
if ($source === '' || $message === '') continue;
$found[] = [
'kind' => 'arr_health',
'subject' => $type,
'ref' => $source,
'conf_key' => '',
'conf_file' => '',
'arr_type' => strtolower((string)($item['type'] ?? 'warning')),
'observed' => $message,
'evidence' => $type . ' › ' . $source . ': ' . $message,
'source_log' => $type . ' /api/' . $api . '/health',
// The arr's own documentation for this check, which is the actual next step
// for most of them and costs nothing to carry.
'note' => trim((string)($item['wikiUrl'] ?? '')),
];
}
}
break;
}
return $found;
}
// ── Proving a candidate ──────────────────────────────────────────────────────────────────────
// The guard the whole unattended path rests on: nothing is written that has not answered.
//
// A model can be confident that a port should be 8686. A probe can report that 8686 answered.
// Only the second is a fact, and only facts get written to conf without being asked. Everything
// a probe cannot settle becomes a conversation instead — which is not a lesser outcome, it is
// the honest one for a value that cannot be derived from this machine.
//
// Deliberately narrow. These check reachability and identity, never correctness of behaviour:
// that Lidarr answers on 8686 does not prove 8686 is the port you meant, only that something is
// listening there and calling itself Lidarr. Proving intent is not a probe's job.
function vv_ai_probe_timeout(): int {
return max(1, (int)(vv_conf_vars()['AI_PROBE_TIMEOUT'] ?? 4));
}
// Does this URL answer at all? Any HTTP status counts, including 401 — a refusal is proof that
// something is listening and speaking HTTP, which is exactly what an address probe is asking.
// Distinguishing "wrong address" from "wrong credential" is the point of having both kinds.
function vv_ai_probe_url(string $url): array {
if (!preg_match('#^https?://[^\s/$.?\#][^\s]*$#i', $url)) {
return ['ok' => false, 'reason' => 'not a url'];
}
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_NOBODY => true,
CURLOPT_TIMEOUT => vv_ai_probe_timeout(),
CURLOPT_CONNECTTIMEOUT => vv_ai_probe_timeout(),
CURLOPT_SSL_VERIFYPEER => false, // these are LAN and Tailscale endpoints, often self-signed
CURLOPT_SSL_VERIFYHOST => false,
]);
curl_exec($ch);
$code = (int)curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
$err = curl_error($ch);
curl_close($ch);
return $code > 0
? ['ok' => true, 'code' => $code]
: ['ok' => false, 'reason' => $err !== '' ? $err : 'no response'];
}
// Every container on this host, by exact name. The membership test for unknown_target findings,
// and one docker call rather than one per candidate.
function vv_ai_container_names(): array {
static $names = null;
if ($names !== null) return $names;
$out = [];
@exec('timeout ' . vv_ai_probe_timeout() . " docker ps -a --format '{{.Names}}' 2>/dev/null", $out, $rc);
return $names = ($rc === 0) ? array_values(array_filter(array_map('trim', $out))) : [];
}
// Candidate corrections for a URL whose host or port stopped answering.
//
// Only two transformations, both conservative: the same host on a port that some other conf key
// already uses, and the same port on a host some other conf key already names. Both draw
// exclusively from values already present in this installation's conf — nothing is invented, and
// a scan of the port range is deliberately not attempted. Finding *a* listening port is not the
// same as finding the right service, and a probe that accepts any answer would happily point
// Lidarr at Sonarr.
function vv_ai_url_candidates(string $observed): array {
$parts = @parse_url($observed);
if (!is_array($parts) || empty($parts['host'])) return [];
$hosts = $ports = [];
foreach (vv_conf_vars() as $k => $v) {
$v = trim((string)$v);
if (!preg_match('#^https?://#i', $v)) continue;
$p = @parse_url($v);
if (!is_array($p) || empty($p['host'])) continue;
$hosts[$p['host']] = true;
if (!empty($p['port'])) $ports[(int)$p['port']] = true;
}
$scheme = $parts['scheme'] ?? 'http';
$path = $parts['path'] ?? '';
$out = [];
foreach (array_keys($ports) as $port) {
$c = $scheme . '://' . $parts['host'] . ':' . $port . $path;
if ($c !== $observed) $out[$c] = true;
}
foreach (array_keys($hosts) as $host) {
$port = !empty($parts['port']) ? ':' . $parts['port'] : '';
$c = $scheme . '://' . $host . $port . $path;
if ($c !== $observed) $out[$c] = true;
}
return array_keys($out);
}
// Try to prove a correction for one finding. Returns the finding with 'proposed' and 'proven'
// filled in, or unchanged when nothing could be proven — which is the common case and not a
// failure.
//
// auth_rejected and missing_value are never proven here on purpose. A credential cannot be
// derived from this host by definition: if it could be read from somewhere, it would not be a
// credential. Those go straight to the operator.
function vv_ai_probe_finding(array $f): array {
$kind = (string)($f['kind'] ?? '');
if ($kind === 'unknown_target') {
$observed = (string)($f['observed'] ?? '');
// Exact membership only. A container name is a literal, and "close to an existing name"
// is how a repair renames the wrong thing.
if (in_array($observed, vv_ai_container_names(), true)) {
return $f; // it exists after all — nothing to correct
}
return $f + ['state' => 'needs_operator'];
}
if ($kind !== 'unreachable') {
// Nothing on this machine can supply a credential, so there is nothing to prove.
$f['state'] = 'needs_operator';
return $f;
}
$observed = (string)($f['observed'] ?? '');
// If the observed address answers now, the fault has cleared on its own — a host that was
// rebooting, most often. Recording a proposal here would repair something already working.
if (vv_ai_probe_url($observed)['ok']) {
$f['state'] = 'resolved';
$f['note'] = 'Answered when probed — the address was reachable again by the time this ran.';
return $f;
}
$answered = [];
foreach (vv_ai_url_candidates($observed) as $candidate) {
if (vv_ai_probe_url($candidate)['ok']) $answered[] = $candidate;
}
// Exactly one, or none. Two addresses answering means the probe cannot say which is the
// right one, and picking either is the guess this exists to prevent.
if (count($answered) === 1) {
$f['proposed'] = $answered[0];
$f['proven'] = true;
$f['note'] = 'Probed ' . $answered[0] . ' and it answered; ' . $observed . ' did not.';
return $f;
}
$f['proven'] = false;
$f['state'] = 'needs_operator';
$f['note'] = $answered
? 'Several addresses answered (' . implode(', ', $answered) . '), so none was written.'
: 'Nothing answered at ' . $observed . ', and no address in the conf answered either.';
return $f;
}
Reference in New Issue View Git Blame Copy Permalink