Gmer4Lfe
91357a2d03
A finding nobody is told about is a finding nobody has, and the card added earlier only shows them to someone who opens the tab. Only needs_operator is announced — an open finding may still be repaired by the next pass — one notification for all of them, and each is announced once and stays quiet until the fault changes or gets worse. vv_notify() hands the message to common.sh's notify() rather than reimplementing the channels, and calls detect_hosts() explicitly because load_config.sh deliberately does not: without it the Unraid notification arrives and Discord silently never does. It also reports false when no channel is switched on at all, since notify() exits 0 either way and a caller believing that would mark a finding as told and never mention it again. Notification text is folded to ASCII. Unraid's notifier dropped an em dash outright and left the double space behind, which was found by sending one and reading what arrived.
1562 lines
78 KiB
PHP
1562 lines
78 KiB
PHP
<?php
|
||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||
// PURPOSE
|
||
// Findings about this installation being misconfigured, and the record of what was done about
|
||
// them. A finding is "Emby is not answering at the address the conf gives", not "Varaverk has
|
||
// a bug" — the second is what ai_bugs holds.
|
||
//
|
||
// OPERATIONAL MODEL
|
||
// A sweep reads a completed run's log, deterministic patterns turn error lines into typed
|
||
// findings, and each finding is either repaired or handed to the operator. Findings persist
|
||
// because the repair may need something only a human can supply, and that conversation has to
|
||
// survive the page being closed.
|
||
//
|
||
// WHY THIS IS NOT ai_bugs
|
||
// Same storage shape, different lifecycle, and the difference is the whole reason for a second
|
||
// store. A bug is open until Varaverk's code changes; nothing on this host can close it. A
|
||
// finding is open until this host's configuration is right, and the same probe that proved a
|
||
// fix can later prove the fault is gone — so findings close themselves and bugs cannot.
|
||
//
|
||
// Filing them together would mean a list where half the rows are actionable by the operator
|
||
// and half are actionable by whoever maintains the project, with no way to tell which is which
|
||
// except by reading them.
|
||
//
|
||
// DESIGN PRINCIPLES
|
||
// A finding names something specific, or it is not a finding.
|
||
// The point of the record is that something can be done about it. "The daily sync looked
|
||
// unhappy" is a feeling; "HOST1_EMBY_URL points at a host that refuses connections" is a
|
||
// finding. Triage that cannot resolve a target produces nothing rather than a vague row.
|
||
//
|
||
// For the conf-bound kinds that is a conf key, and it is still required — a proposal to
|
||
// edit a key that does not exist can never be actioned. For the rest it is whatever
|
||
// identifies the thing: an arr reporting "all lists are unavailable" is specific and
|
||
// actionable with no Varaverk key to change, because the action is in Radarr's own UI.
|
||
//
|
||
// Evidence is the log line, quoted.
|
||
// Same rule as ai_bugs, for the same reason: a finding that cannot show the line it came
|
||
// from cannot be checked, and this store is meant to be checkable.
|
||
//
|
||
// Identity is kind + subject + reference, not the message text.
|
||
// A port that has been wrong for a week is one finding seen 400 times, not 400 findings.
|
||
// Wording drifts as logs change; the thing being wrong does not. "Indexers unavailable:
|
||
// NzbNoob" becoming "NzbNoob, Miatrix" is the same finding getting worse.
|
||
//
|
||
// OPERATIONAL SAFEGUARDS
|
||
// A proposed value is recorded, never trusted.
|
||
// 'proposed' is what something might be changed to; 'proven' is whether a probe actually
|
||
// got an answer from it. Only proven values are ever written to conf, and the two fields
|
||
// are kept separate so a record cannot imply verification it did not have.
|
||
//
|
||
// Closing is evidence-driven, not time-driven.
|
||
// A finding closes when its probe passes or the operator dismisses it. It does not expire,
|
||
// because "we stopped seeing it in the log" is equally consistent with the job no longer
|
||
// running at all.
|
||
//
|
||
// Under data/ and therefore gitignored: these quote this installation's logs and name its
|
||
// hosts, ports and containers.
|
||
//
|
||
// EXPORTS
|
||
// vv_ai_findings_dir() the store
|
||
// vv_ai_finding_write() file or increment one finding
|
||
// vv_ai_findings_list() findings, newest activity first
|
||
// vv_ai_finding_get() one by id
|
||
// vv_ai_finding_close() mark resolved, with how
|
||
// vv_ai_finding_dismiss() operator says this is not a problem
|
||
// vv_ai_findings_for_chat() the open ones worth opening a conversation about
|
||
//
|
||
// CONFIGURATION
|
||
// AI_DATA_DIR findings live in ai_findings/ beneath it
|
||
// AI_FINDING_RETAIN_DAYS closed findings older than this are removed (default 90)
|
||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||
require_once __DIR__ . '/config.php';
|
||
require_once __DIR__ . '/ai.php';
|
||
// For vv_notify() only. A finding that cannot reach the operator is the one thing this file
|
||
// cannot do on its own, and reimplementing the channels here would be a second answer to a
|
||
// question common.sh already answers.
|
||
require_once __DIR__ . '/common.php';
|
||
|
||
// What a finding can be. The kind decides which repair is even conceivable, so an unknown kind
|
||
// is refused rather than stored as an untyped row nothing knows how to act on.
|
||
const VV_AI_FINDING_KINDS = [
|
||
'unreachable' => 'a configured address or port refused, timed out, or did not resolve',
|
||
'auth_rejected' => 'the endpoint answered, and rejected the credential',
|
||
'unknown_target' => 'a conf entry names a container or share that does not exist here',
|
||
'missing_value' => 'a conf key required by the job that ran is empty',
|
||
'arr_health' => 'an arr is reporting a problem about itself',
|
||
];
|
||
|
||
// Which kinds are a statement about Varaverk's configuration, and which are a statement about
|
||
// something else that is nonetheless worth recording.
|
||
//
|
||
// The rule was originally "a finding names a conf key or it is not a finding", to stop the store
|
||
// filling with observations nobody could act on. What that rule was really protecting is that
|
||
// every finding identifies something specific and actionable — naming a conf key was the proxy,
|
||
// because at the time every source of findings was a conf problem.
|
||
//
|
||
// An arr reporting "all lists are unavailable" is specific and actionable, and there is no
|
||
// Varaverk key to change: the action is in Radarr's own UI. So the identity widens to a general
|
||
// reference, and the conf-key requirement narrows to the kinds it was written for. What has not
|
||
// changed is that a finding with nothing to point at is still refused.
|
||
const VV_AI_CONF_BOUND_KINDS = ['unreachable', 'auth_rejected', 'unknown_target', 'missing_value'];
|
||
|
||
function vv_ai_kind_is_conf_bound(string $kind): bool {
|
||
return in_array($kind, VV_AI_CONF_BOUND_KINDS, true);
|
||
}
|
||
|
||
// How a finding ended, when it ends.
|
||
const VV_AI_FINDING_STATES = [
|
||
'open' => 'seen, not yet acted on',
|
||
'needs_operator' => 'cannot be repaired here — the value is not derivable from this host',
|
||
'acknowledged' => 'the operator knows, and it stays quiet until the state it was acked at changes',
|
||
'fixed' => 'a proven value was written to conf',
|
||
'resolved' => 'the probe now passes; whatever was wrong is no longer wrong',
|
||
'dismissed' => 'the operator says this is not a problem, permanently',
|
||
];
|
||
|
||
// ── Why acknowledged is not dismissed ────────────────────────────────────────────────────────
|
||
// "I know critical rsync is off, stop telling me" and "this is never a problem" are different
|
||
// instructions, and collapsing them loses the half that matters. An acknowledgement is scoped to
|
||
// the state it was given in: CRITICAL_RSYNC_ENABLED being false is a deliberate choice today and
|
||
// a stale note the moment it goes true again.
|
||
//
|
||
// So an ack records what the key read when it was given, and expires when that changes. The
|
||
// finding comes back on its own, without the operator having to remember to look — which is the
|
||
// difference between a note and a silence.
|
||
|
||
// ── Toggles are the operator's, always ───────────────────────────────────────────────────────
|
||
// A repair may never enable or disable anything on its own. Not because it would get the value
|
||
// wrong — a boolean has only two — but because the value is not a fact to be discovered. Whether
|
||
// critical rsync should be on is a decision about intent, and a probe cannot prove intent the
|
||
// way it can prove that a port answers.
|
||
//
|
||
// The Fix action still writes it when the operator asks for it. What is forbidden is the
|
||
// unattended path choosing for them.
|
||
function vv_ai_conf_is_toggle(string $key): bool {
|
||
$v = strtolower(trim((string)(vv_conf_vars()[$key] ?? '')));
|
||
return $v === 'true' || $v === 'false';
|
||
}
|
||
|
||
// May the sweep write this without being asked? Two conditions, both required: a probe actually
|
||
// answered on the proposed value, and the key is not a toggle.
|
||
function vv_ai_finding_may_autofix(array $f): bool {
|
||
if (!vv_ai_repair_autofix_enabled()) return false;
|
||
if (empty($f['proven'])) return false;
|
||
if (($f['proposed'] ?? null) === null) return false;
|
||
if (vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))) return false;
|
||
|
||
// A third condition, for paths only. "Proven" means a probe answered, and every probe this
|
||
// has is a network probe — nothing in it can answer a filesystem question, so a path
|
||
// proposal reaches here carrying a proof that is about something else entirely. Requiring
|
||
// the directory to exist is the equivalent evidence, and it is the difference between
|
||
// pointing a cleanup at a real share and pointing it at a typo that will be created empty
|
||
// by the first script to write there.
|
||
//
|
||
// vv_conf_path_write_ok() still runs inside the writer underneath this. That one refuses
|
||
// what is dangerous; this one refuses what is merely unproven, which is a bar only the
|
||
// unattended path has to clear.
|
||
$proposed = (string)$f['proposed'];
|
||
if ($proposed !== '' && $proposed[0] === '/' && !file_exists($proposed)) return false;
|
||
|
||
return true;
|
||
}
|
||
|
||
// ── Severity is derived, never supplied ──────────────────────────────────────────────────────
|
||
// Same ladder run_job.sh records runs against — ok / warn / error — so a finding and the run it
|
||
// came from cannot describe the same event at two different volumes.
|
||
//
|
||
// The rule that matters: a finding whose key is a toggle can never be an error. Something not
|
||
// happening because it was switched off is the switch working. That is true whether the switch
|
||
// was flipped deliberately last month or by accident this morning, and the store cannot tell
|
||
// those apart — so it reports the fact and lets the operator supply the intent.
|
||
//
|
||
// Everything else takes its level from what the fault costs. A credential the endpoint rejected
|
||
// stops that integration dead; an address that does not answer might be a host still booting.
|
||
function vv_ai_finding_severity(array $f): string {
|
||
$key = (string)($f['conf_key'] ?? '');
|
||
|
||
// Deliberate-state findings never escalate, whatever their kind.
|
||
if ($key !== '' && vv_ai_conf_is_toggle($key)) return 'warn';
|
||
|
||
// An arr grades its own health and is the authority on it — "error" from Radarr means Radarr
|
||
// has stopped doing something, which is not a judgement to second-guess from out here.
|
||
if (($f['kind'] ?? '') === 'arr_health') {
|
||
return ($f['arr_type'] ?? '') === 'error' ? 'error' : 'warn';
|
||
}
|
||
|
||
return match ($f['kind'] ?? '') {
|
||
'auth_rejected' => 'error', // answered and refused — nothing gets through until fixed
|
||
'missing_value' => 'error', // configured to use something that was never supplied
|
||
'unreachable' => 'warn', // may be transient; the strike system is what escalates it
|
||
'unknown_target' => 'warn',
|
||
default => 'warn',
|
||
};
|
||
}
|
||
|
||
// ── Gates ────────────────────────────────────────────────────────────────────────────────────
|
||
// Two switches, because detecting and repairing are separate things to trust.
|
||
//
|
||
// AI_REPAIR_ENABLED alone gives a system that reads logs, files findings and offers fixes, and
|
||
// writes nothing. That is the state this should live in first — long enough to read what it
|
||
// found and disagree with some of it. A subsystem that starts by editing conf has to be believed
|
||
// before there is any evidence for believing it.
|
||
//
|
||
// AI_REPAIR_AUTOFIX_ENABLED is what lets a proven value be written without being asked, and it
|
||
// is meaningless on its own: nothing to write if nothing is looking. Both must be true, in the
|
||
// same layered way AI_ENABLED is necessary but never sufficient.
|
||
function vv_ai_repair_enabled(): bool {
|
||
if (!vv_ai_config()['enabled']) return false;
|
||
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_ENABLED'] ?? 'false'))) === 'true';
|
||
}
|
||
|
||
function vv_ai_repair_autofix_enabled(): bool {
|
||
if (!vv_ai_repair_enabled()) return false;
|
||
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_AUTOFIX_ENABLED'] ?? 'false'))) === 'true';
|
||
}
|
||
|
||
function vv_ai_findings_dir(): string {
|
||
$d = AI_DATA_DIR . '/ai_findings';
|
||
if (!is_dir($d)) @mkdir($d, 0755, true);
|
||
return $d;
|
||
}
|
||
|
||
function vv_ai_finding_retain_days(): int {
|
||
$n = (int)(vv_conf_vars()['AI_FINDING_RETAIN_DAYS'] ?? 90);
|
||
return max(1, $n);
|
||
}
|
||
|
||
// kind + subject + reference. Deliberately not the message: the same wrong port produces slightly
|
||
// different log text as the software around it changes, and that must not mint a second record.
|
||
//
|
||
// The reference is the conf key for a conf-bound kind, and whatever else identifies the thing
|
||
// otherwise — for an arr health item, the check that raised it. "Indexers unavailable: NzbNoob"
|
||
// becomes "Indexers unavailable: NzbNoob, Miatrix" as more fail, and that is the same finding
|
||
// getting worse rather than a second one.
|
||
function vv_ai_finding_id(string $kind, string $subject, string $ref): string {
|
||
return substr(sha1(strtolower($kind . '|' . $subject . '|' . $ref)), 0, 12);
|
||
}
|
||
|
||
function vv_ai_finding_path(string $id): ?string {
|
||
if (!preg_match('/^[0-9a-f]{12}$/', $id)) return null;
|
||
return vv_ai_findings_dir() . '/' . $id . '.json';
|
||
}
|
||
|
||
function vv_ai_finding_get(string $id): ?array {
|
||
$p = vv_ai_finding_path($id);
|
||
if ($p === null || !is_file($p)) return null;
|
||
$r = json_decode((string)@file_get_contents($p), true);
|
||
return is_array($r) ? $r : null;
|
||
}
|
||
|
||
// Files a finding, or increments the one already describing this fault.
|
||
//
|
||
// $f expects: kind, subject, conf_key, conf_file, observed, evidence, source_log
|
||
// and optionally: proposed, proven, state, note
|
||
function vv_ai_finding_write(array $f): array {
|
||
$kind = (string)($f['kind'] ?? '');
|
||
$subject = trim((string)($f['subject'] ?? ''));
|
||
$confKey = trim((string)($f['conf_key'] ?? ''));
|
||
$evidence = trim((string)($f['evidence'] ?? ''));
|
||
|
||
// What identifies this finding. Conf-bound kinds are identified by their key; everything else
|
||
// supplies its own reference, and a finding with neither points at nothing and is refused.
|
||
$ref = trim((string)($f['ref'] ?? $confKey));
|
||
|
||
if (!isset(VV_AI_FINDING_KINDS[$kind])) return ['ok' => false, 'error' => 'unknown kind'];
|
||
if ($subject === '' || $ref === '') return ['ok' => false, 'error' => 'subject and a reference are required'];
|
||
if ($evidence === '') return ['ok' => false, 'error' => 'evidence required'];
|
||
|
||
if (vv_ai_kind_is_conf_bound($kind)) {
|
||
if ($confKey === '') return ['ok' => false, 'error' => 'conf_key required for this kind'];
|
||
// The key has to be a real shell identifier for the same reason the conf writer insists
|
||
// on it: a finding is a proposal to edit that key, and a malformed one cannot be actioned.
|
||
if (!vv_conf_key_valid($confKey)) return ['ok' => false, 'error' => 'malformed conf key'];
|
||
}
|
||
|
||
$state = (string)($f['state'] ?? 'open');
|
||
if (!isset(VV_AI_FINDING_STATES[$state])) $state = 'open';
|
||
|
||
$now = time();
|
||
$id = vv_ai_finding_id($kind, $subject, $ref);
|
||
$rec = [
|
||
'id' => $id,
|
||
'kind' => $kind,
|
||
'subject' => mb_substr($subject, 0, 120),
|
||
'conf_key' => $confKey,
|
||
'conf_file' => (string)($f['conf_file'] ?? 'master.conf'),
|
||
// Secrets never enter this store. A finding about a rejected API key is about the key
|
||
// being wrong, and the wrong value is of no use to anyone reading the record later.
|
||
'observed' => vv_conf_key_is_secret($confKey) ? '<redacted>'
|
||
: mb_substr((string)($f['observed'] ?? ''), 0, 300),
|
||
'proposed' => isset($f['proposed']) && !vv_conf_key_is_secret($confKey)
|
||
? mb_substr((string)$f['proposed'], 0, 300) : null,
|
||
'proven' => (bool)($f['proven'] ?? false),
|
||
'state' => $state,
|
||
'evidence' => mb_substr(vv_ai_redact($evidence), 0, 1200),
|
||
'source_log' => mb_substr((string)($f['source_log'] ?? ''), 0, 200),
|
||
'note' => mb_substr((string)($f['note'] ?? ''), 0, 1000),
|
||
// Recomputed on every sighting rather than stored once: a key that becomes a toggle, or
|
||
// a toggle that is replaced by a real value, changes what this finding means.
|
||
'ref' => mb_substr($ref, 0, 120),
|
||
'severity' => vv_ai_finding_severity(['kind' => $kind, 'conf_key' => $confKey,
|
||
'arr_type' => (string)($f['arr_type'] ?? '')]),
|
||
'host' => vv_detect_host(),
|
||
'first' => $now,
|
||
'last' => $now,
|
||
'seen' => 1,
|
||
'closed_at' => null,
|
||
// What the key read when the operator acknowledged it. Null unless acked; the ack
|
||
// expires the moment the live value stops matching this.
|
||
'ack_value' => null,
|
||
// What this finding looked like when it was last announced. Carried across sightings
|
||
// below — a stamp that reset every fifteen minutes would be a notification every fifteen
|
||
// minutes, which is how an operator learns to ignore the channel.
|
||
'notified' => null,
|
||
];
|
||
|
||
$p = vv_ai_finding_path($id);
|
||
if ($p === null) return ['ok' => false, 'error' => 'bad id'];
|
||
|
||
if (is_file($p)) {
|
||
$old = json_decode((string)@file_get_contents($p), true);
|
||
if (is_array($old)) {
|
||
$rec['first'] = $old['first'] ?? $now;
|
||
$rec['seen'] = (int)($old['seen'] ?? 0) + 1;
|
||
// A dismissed finding stays dismissed however many times the log repeats it —
|
||
// otherwise "this is fine, stop telling me" lasts exactly one cycle. A fixed one
|
||
// reopens, because seeing the fault again after a repair means the repair did not
|
||
// hold, which is the single most important thing this store can tell anyone.
|
||
if (($old['state'] ?? '') === 'dismissed') {
|
||
$rec['state'] = 'dismissed';
|
||
$rec['closed_at'] = $old['closed_at'] ?? null;
|
||
}
|
||
// An acknowledgement holds only while the thing acknowledged is still true. Compare
|
||
// what it is pinned to now against what it read when the ack was given: unchanged
|
||
// means stay quiet, changed means the note is stale and the finding comes back by
|
||
// itself. $rec carries this sighting's evidence, so a fault that has changed shape
|
||
// fails this comparison even when no conf key is involved.
|
||
if (($old['state'] ?? '') === 'acknowledged') {
|
||
$ackedAt = (string)($old['ack_value'] ?? '');
|
||
if ($ackedAt === vv_ai_finding_ack_pin($rec)) {
|
||
$rec['state'] = 'acknowledged';
|
||
$rec['ack_value'] = $ackedAt;
|
||
$rec['closed_at'] = $old['closed_at'] ?? null;
|
||
}
|
||
// Otherwise $rec keeps the state this sighting computed — it has reopened.
|
||
}
|
||
// Preserve an operator's note over a generated one.
|
||
if ($rec['note'] === '' && !empty($old['note'])) $rec['note'] = $old['note'];
|
||
|
||
// Carried, never recomputed. This record is rebuilt from scratch on every sighting,
|
||
// so anything not copied forward here is reset — and a reset announcement stamp
|
||
// means this finding is announced again on the next pass, and the one after that.
|
||
$rec['notified'] = $old['notified'] ?? null;
|
||
}
|
||
}
|
||
|
||
if (@file_put_contents($p, json_encode($rec, JSON_PRETTY_PRINT)) === false) {
|
||
return ['ok' => false, 'error' => 'write failed'];
|
||
}
|
||
return ['ok' => true, 'id' => $id, 'seen' => $rec['seen'], 'state' => $rec['state']];
|
||
}
|
||
|
||
// $states filters; empty means everything. Newest activity first, because a finding seen in the
|
||
// last cycle matters more than one that has been sitting fixed for a month.
|
||
function vv_ai_findings_list(array $states = ['open', 'needs_operator']): array {
|
||
$out = [];
|
||
foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) {
|
||
$r = json_decode((string)@file_get_contents($file), true);
|
||
if (!is_array($r)) continue;
|
||
if ($states && !in_array($r['state'] ?? 'open', $states, true)) continue;
|
||
$out[] = $r;
|
||
}
|
||
usort($out, fn($a, $b) => ($b['last'] ?? 0) <=> ($a['last'] ?? 0));
|
||
return $out;
|
||
}
|
||
|
||
function vv_ai_finding_set_state(string $id, string $state, string $note = ''): bool {
|
||
if (!isset(VV_AI_FINDING_STATES[$state])) return false;
|
||
$r = vv_ai_finding_get($id);
|
||
if ($r === null) return false;
|
||
|
||
$r['state'] = $state;
|
||
$r['closed_at'] = in_array($state, ['open', 'needs_operator'], true) ? null : time();
|
||
if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000);
|
||
|
||
$p = vv_ai_finding_path($id);
|
||
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
|
||
}
|
||
|
||
function vv_ai_finding_close(string $id, string $note = ''): bool {
|
||
return vv_ai_finding_set_state($id, 'resolved', $note);
|
||
}
|
||
|
||
function vv_ai_finding_dismiss(string $id, string $note = ''): bool {
|
||
return vv_ai_finding_set_state($id, 'dismissed', $note);
|
||
}
|
||
|
||
// What an acknowledgement is pinned to — the thing that has to stay the same for the ack to keep
|
||
// meaning what it meant.
|
||
//
|
||
// For a conf-bound finding that is the key's value, which is what the operator was looking at
|
||
// when they said "I know". A finding with no key had nothing to pin to and so compared '' with
|
||
// '' — every ack on an arr health item was silently permanent, which is dismiss wearing ack's
|
||
// label. Those pin to the shape of the fault instead: "indexers unavailable: NzbNoob" and
|
||
// "indexers unavailable: NzbNoob, Miatrix" are one finding getting worse, and an ack given for
|
||
// the first has not been given for the second.
|
||
function vv_ai_finding_ack_pin(array $f): string {
|
||
$key = (string)($f['conf_key'] ?? '');
|
||
if ($key !== '') return (string)(vv_conf_vars()[$key] ?? '');
|
||
return 'ev:' . substr(sha1((string)($f['evidence'] ?? '')), 0, 16);
|
||
}
|
||
|
||
// "I know about this — leave it, and tell me if it changes."
|
||
//
|
||
// Stamps what it is pinned to onto the record. Every later sighting compares against that stamp,
|
||
// so the acknowledgement covers this state and not the finding forever. Acking that critical
|
||
// rsync is off says nothing about critical rsync being on.
|
||
function vv_ai_finding_ack(string $id, string $note = ''): bool {
|
||
$r = vv_ai_finding_get($id);
|
||
if ($r === null) return false;
|
||
|
||
$r['state'] = 'acknowledged';
|
||
$r['ack_value'] = vv_ai_finding_ack_pin($r);
|
||
$r['closed_at'] = time();
|
||
if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000);
|
||
|
||
$p = vv_ai_finding_path($id);
|
||
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
|
||
}
|
||
|
||
// What the operator can do about a finding, and what each choice means. Returned rather than
|
||
// hardcoded in the UI so the chat and the page cannot offer different options for the same row,
|
||
// and enforced in vv_ai_finding_apply_action() so neither can act on one it was not offered.
|
||
//
|
||
// Fix appears for anything with a proposed value, toggle or not — the prohibition is on the
|
||
// sweep choosing, never on the operator choosing. Open rows also carry ack, dismiss and cancel,
|
||
// because "I know", "this is never a problem" and "not now" are all valid answers to being told
|
||
// something, and they are three different answers.
|
||
function vv_ai_finding_actions(array $f): array {
|
||
// A closed finding has one question left, and it is not the original one: was closing it
|
||
// right? Offering fix or ack on a row that is already dismissed is offering to decide
|
||
// something that has been decided. Reopen is here because dismiss is otherwise permanent —
|
||
// the write path keeps a dismissed finding dismissed however many times the fault recurs,
|
||
// so a mis-click would need someone editing JSON on disk to undo.
|
||
if (!in_array((string)($f['state'] ?? 'open'), ['open', 'needs_operator'], true)) {
|
||
return ['reopen' => 'Put it back on the list — either closing it was wrong, or it is back'];
|
||
}
|
||
|
||
$actions = [];
|
||
|
||
if (($f['proposed'] ?? null) !== null) {
|
||
$actions['fix'] = vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))
|
||
? 'Set ' . $f['conf_key'] . ' — a toggle, so this only ever happens because you asked'
|
||
: 'Write the proven value to ' . $f['conf_key'];
|
||
}
|
||
|
||
$actions['ack'] = ($f['conf_key'] ?? '') !== ''
|
||
? 'Known and intended. Stays quiet until ' . $f['conf_key'] . ' changes'
|
||
: 'Known and intended. Stays quiet until the fault itself changes';
|
||
$actions['dismiss'] = 'Not a problem, ever. Stays closed even when it is seen again';
|
||
$actions['cancel'] = 'Leave it alone for now';
|
||
|
||
return $actions;
|
||
}
|
||
|
||
// Closed findings are kept for a while because "this happened before and here is what fixed it"
|
||
// is worth more than the disk it costs. Open ones are never pruned — an unresolved problem does
|
||
// not stop mattering because it is old.
|
||
function vv_ai_findings_prune(): int {
|
||
$cutoff = time() - (vv_ai_finding_retain_days() * 86400);
|
||
$n = 0;
|
||
foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) {
|
||
$r = json_decode((string)@file_get_contents($file), true);
|
||
if (!is_array($r)) continue;
|
||
if (in_array($r['state'] ?? 'open', ['open', 'needs_operator'], true)) continue;
|
||
if ((int)($r['closed_at'] ?? 0) > $cutoff) continue;
|
||
if (@unlink($file)) $n++;
|
||
}
|
||
return $n;
|
||
}
|
||
|
||
// ── Reaching the operator ────────────────────────────────────────────────────────────────────
|
||
// A finding nobody is told about is a finding nobody has. The card on the AI tab shows them, but
|
||
// only to someone who opens the tab, and the point of this subsystem is that it works while
|
||
// nobody is looking.
|
||
//
|
||
// Only needs_operator is announced. An open finding may still be repaired by the next pass, and
|
||
// announcing one would be telling the operator about a problem that has already been handled by
|
||
// the time they read it. acknowledged and dismissed are the operator's own answers and are never
|
||
// announced at all.
|
||
//
|
||
// AI_REPAIR_NOTIFY_ENABLED defaults to true when absent, unlike the two switches above it. Those
|
||
// gate reading and writing, which are things to be trusted first; this gates telling someone,
|
||
// which is the point of having found anything. It still cannot fire unless AI_REPAIR_ENABLED is
|
||
// on, and notify() itself is subject to NOTIFY_UNRAID and the host's webhook.
|
||
function vv_ai_notify_enabled(): bool {
|
||
if (!vv_ai_repair_enabled()) return false;
|
||
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_NOTIFY_ENABLED'] ?? 'true'))) !== 'false';
|
||
}
|
||
|
||
// What has to change before this finding is worth mentioning twice.
|
||
//
|
||
// The same pin as an acknowledgement, plus the severity. That means a finding is announced once
|
||
// and then stays quiet — through every fifteen-minute pass, however many times it is seen — until
|
||
// either the thing it is about changes or it gets worse. "Indexers unavailable: NzbNoob" becoming
|
||
// "NzbNoob, Miatrix" is news; the same sentence for the ninth time is not.
|
||
function vv_ai_finding_notify_pin(array $f): string {
|
||
return vv_ai_finding_ack_pin($f) . '|' . (string)($f['severity'] ?? 'warn');
|
||
}
|
||
|
||
// One notification for everything that needs saying, not one per finding. Returns what it did so
|
||
// the sweep can log it and the tests can read it without a notification having to be sent.
|
||
function vv_ai_findings_announce(bool $dryRun = false): array {
|
||
$out = ['sent' => false, 'count' => 0, 'subject' => '', 'message' => '', 'ids' => []];
|
||
if (!vv_ai_notify_enabled()) return $out + ['skipped' => 'AI_REPAIR_NOTIFY_ENABLED is false'];
|
||
|
||
// Nothing switched on to receive it. Checked before anything is composed, and reported as a
|
||
// configuration state rather than a delivery failure: a failure is retried on the next pass
|
||
// and logged loudly, and neither is the right response to "no channel has been set up".
|
||
if (!vv_notify_available()) {
|
||
return $out + ['skipped' => 'no notification channel — NOTIFY_UNRAID is false and no webhook is set'];
|
||
}
|
||
|
||
$due = [];
|
||
foreach (vv_ai_findings_list(['needs_operator']) as $f) {
|
||
if (($f['notified'] ?? null) === vv_ai_finding_notify_pin($f)) continue;
|
||
$due[] = $f;
|
||
}
|
||
if (!$due) return $out;
|
||
|
||
// Worst first: if the message is truncated, the part that survives is the part that matters.
|
||
usort($due, fn($a, $b) => (($b['severity'] ?? '') === 'error' ? 1 : 0)
|
||
<=> (($a['severity'] ?? '') === 'error' ? 1 : 0));
|
||
|
||
$lines = [];
|
||
foreach (array_slice($due, 0, 3) as $f) {
|
||
// Evidence is a log excerpt and routinely spans lines. Collapsed here rather than left
|
||
// for vv_notify() to tidy on the way out: this string is also what gets logged and what
|
||
// the sweep summary returns, and only one of those three readers strips newlines.
|
||
// Collapsed before truncating, so 90 characters means 90 visible ones.
|
||
$ev = trim(preg_replace('/\s+/u', ' ', (string)($f['evidence'] ?? '')));
|
||
$lines[] = sprintf('%s - %s: %s', $f['subject'] ?? '?', $f['ref'] ?? '?',
|
||
mb_substr($ev, 0, 90));
|
||
}
|
||
if (count($due) > 3) $lines[] = sprintf('and %d more', count($due) - 3);
|
||
|
||
$n = count($due);
|
||
$anyErr = false;
|
||
foreach ($due as $f) if (($f['severity'] ?? '') === 'error') $anyErr = true;
|
||
|
||
$out['count'] = $n;
|
||
$out['subject'] = sprintf('Varaverk repair - %d finding%s need%s you', $n, $n === 1 ? '' : 's',
|
||
$n === 1 ? 's' : '');
|
||
// ' · ' rather than newlines: notify() builds its Discord payload by printf-ing into a JSON
|
||
// string literal, and a raw newline there produces a body the webhook rejects.
|
||
$out['message'] = implode(' | ', $lines);
|
||
$out['ids'] = array_column($due, 'id');
|
||
|
||
if ($dryRun) return $out;
|
||
|
||
$out['sent'] = vv_notify($out['message'], $out['subject'], $anyErr ? 'alert' : 'warning');
|
||
|
||
// Stamped only on a delivery that worked. A channel that is down should retry on the next
|
||
// pass rather than mark these as told and go quiet about them forever.
|
||
if ($out['sent']) {
|
||
foreach ($due as $f) {
|
||
$rec = vv_ai_finding_get($f['id']);
|
||
if ($rec === null) continue;
|
||
$rec['notified'] = vv_ai_finding_notify_pin($rec);
|
||
$p = vv_ai_finding_path($f['id']);
|
||
if ($p !== null) @file_put_contents($p, json_encode($rec, JSON_PRETTY_PRINT));
|
||
}
|
||
}
|
||
return $out;
|
||
}
|
||
|
||
// What the assistant should raise when a page loads: things that need the operator, newest
|
||
// first. Repaired findings are deliberately not here — a fix that worked is a log entry, not a
|
||
// conversation, and opening every session with a list of things that already went right is how
|
||
// an operator learns to close the panel without reading it.
|
||
function vv_ai_findings_for_chat(int $limit = 3): array {
|
||
return array_slice(vv_ai_findings_list(['needs_operator']), 0, max(1, $limit));
|
||
}
|
||
|
||
// ── The sweep ────────────────────────────────────────────────────────────────────────────────
|
||
// There is no post-run hook in Varaverk — nothing fires when a job finishes. Rather than add a
|
||
// call to forty scripts, this picks up run records that completed since the last pass. One entry
|
||
// in an orchestrator's list instead of forty edits, and it batches naturally.
|
||
//
|
||
// Runs that reported ok are read too. A container failing its HTTP check warns and leaves the
|
||
// watchdog exiting 0, so "only look at failures" would miss the whole class of fault this exists
|
||
// for: the job worked, and told you something is wrong.
|
||
|
||
function vv_ai_sweep_marker_path(): string {
|
||
return STATE_DIR . '/ai_repair_sweep.db';
|
||
}
|
||
|
||
function vv_ai_sweep_last(): int {
|
||
return (int)trim((string)@file_get_contents(vv_ai_sweep_marker_path()));
|
||
}
|
||
|
||
function vv_ai_sweep_mark(int $ts): void {
|
||
if (!is_dir(STATE_DIR)) @mkdir(STATE_DIR, 0755, true);
|
||
@file_put_contents(vv_ai_sweep_marker_path(), (string)$ts, LOCK_EX);
|
||
}
|
||
|
||
// Run records that finished after $since. A record still marked running is skipped rather than
|
||
// read half-written — it will be picked up on the pass after it finishes.
|
||
function vv_ai_recent_runs(int $since): array {
|
||
$out = [];
|
||
$base = realpath(LOG_DIR);
|
||
if ($base === false) return [];
|
||
|
||
foreach ((array)@glob($base . '/{,*/,*/*/,*/*/*/}*.json', GLOB_BRACE) as $path) {
|
||
$r = json_decode((string)@file_get_contents($path), true);
|
||
if (!is_array($r) || empty($r['id']) || ($r['status'] ?? '') === 'running') continue;
|
||
|
||
$end = (int)($r['end'] ?? 0);
|
||
if ($end <= $since) continue;
|
||
|
||
$log = preg_replace('/\.json$/', '.log', $path);
|
||
if (!is_file($log)) continue;
|
||
|
||
$out[] = ['id' => (string)$r['id'], 'status' => (string)($r['status'] ?? '?'),
|
||
'start' => (int)($r['start'] ?? 0), 'end' => $end, 'log' => $log];
|
||
}
|
||
usort($out, fn($a, $b) => $a['end'] <=> $b['end']);
|
||
return $out;
|
||
}
|
||
|
||
// The lines one run wrote, and only those. Logs are appended across runs, so a tail alone would
|
||
// re-read the previous run's output and re-report faults that have already been dealt with.
|
||
// Filtering on the leading timestamp scopes the evidence to the run being examined.
|
||
function vv_ai_run_log_lines(string $logPath, int $startTs, int $maxLines = 2000): array {
|
||
$out = []; $rc = 0;
|
||
@exec('tail -n ' . (int)$maxLines . ' ' . escapeshellarg($logPath) . ' 2>/dev/null', $out, $rc);
|
||
if ($rc !== 0) return [];
|
||
|
||
$kept = [];
|
||
foreach ($out as $line) {
|
||
if (preg_match('/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/', $line, $m)) {
|
||
// A line older than the run belongs to a previous one. Two seconds of slack because
|
||
// the record's start is stamped by the runner, not by the first line the job writes.
|
||
if (strtotime($m[1]) < $startTs - 2) continue;
|
||
}
|
||
$kept[] = $line;
|
||
}
|
||
return $kept;
|
||
}
|
||
|
||
// One pass. Returns a summary rather than logging it, so the caller decides what to record and
|
||
// the whole thing stays testable without a log to read afterwards.
|
||
//
|
||
// $dryRun does everything except write conf and move the marker — including probing, which is
|
||
// the point: it answers "what would this have done" with real evidence rather than a guess.
|
||
function vv_ai_repair_sweep(bool $dryRun = false): array {
|
||
if (!vv_ai_repair_enabled()) {
|
||
return ['ok' => false, 'error' => 'AI_REPAIR_ENABLED is not true', 'runs' => 0];
|
||
}
|
||
|
||
$started = time();
|
||
$since = vv_ai_sweep_last();
|
||
$runs = vv_ai_recent_runs($since);
|
||
|
||
$sum = ['ok' => true, 'runs' => count($runs), 'findings' => 0, 'fixed' => 0,
|
||
'needs_operator' => 0, 'resolved' => 0, 'quiet' => 0, 'details' => []];
|
||
|
||
// Candidates from two sources. Log triage is bounded to runs that finished since the last
|
||
// pass; the arrs are asked every time, because their health is a current state rather than
|
||
// something that appeared in a log once. Asking costs three local HTTP calls.
|
||
$candidates = vv_ai_arr_health_findings();
|
||
|
||
foreach ($runs as $run) {
|
||
$lines = vv_ai_run_log_lines($run['log'], $run['start']);
|
||
if (!$lines) continue;
|
||
|
||
$rel = ltrim(str_replace(realpath(LOG_DIR), '', $run['log']), '/');
|
||
foreach (vv_ai_triage_log($lines, $rel) as $c) $candidates[] = $c;
|
||
}
|
||
|
||
foreach ($candidates as $cand) {
|
||
$cand = vv_ai_probe_finding($cand);
|
||
$sum['findings']++;
|
||
|
||
// Write first, so a finding exists even if the repair below fails. A repair that
|
||
// errored without leaving a record is the one failure mode there is no way back from.
|
||
$w = vv_ai_finding_write($cand);
|
||
if (!($w['ok'] ?? false)) continue;
|
||
$id = $w['id'];
|
||
|
||
// Already acknowledged or dismissed — the operator has spoken, and re-fixing behind
|
||
// them would be the opposite of what an acknowledgement means.
|
||
if (in_array($w['state'] ?? '', ['acknowledged', 'dismissed'], true)) {
|
||
$sum['quiet']++;
|
||
continue;
|
||
}
|
||
|
||
if (($cand['state'] ?? '') === 'resolved') {
|
||
vv_ai_finding_close($id, (string)($cand['note'] ?? ''));
|
||
$sum['resolved']++;
|
||
continue;
|
||
}
|
||
|
||
if (vv_ai_finding_may_autofix($cand)) {
|
||
if ($dryRun) { $sum['details'][] = "would fix {$cand['conf_key']} → {$cand['proposed']}"; continue; }
|
||
$r = vv_ai_finding_apply_action($id, 'fix', 'Probed and written by the repair sweep.');
|
||
if ($r['ok'] ?? false) { $sum['fixed']++; $sum['details'][] = "fixed {$cand['conf_key']}"; }
|
||
else { $sum['needs_operator']++; vv_ai_finding_set_state($id, 'needs_operator', (string)($r['error'] ?? '')); }
|
||
continue;
|
||
}
|
||
|
||
if (($cand['state'] ?? '') === 'needs_operator') $sum['needs_operator']++;
|
||
}
|
||
|
||
// Marked only on a completed pass, and to when the pass began — a job that finished while
|
||
// this was running is then picked up next time instead of being skipped for having ended
|
||
// before a marker written at the end.
|
||
if (!$dryRun) vv_ai_sweep_mark($started);
|
||
|
||
// Announced after the marker rather than before, and over the whole store rather than only
|
||
// what this pass touched. A finding that reached needs_operator two passes ago and was never
|
||
// successfully delivered is still owed to the operator, and the pin is what stops that from
|
||
// meaning it is announced twice.
|
||
$sum['announced'] = vv_ai_findings_announce($dryRun);
|
||
|
||
return $sum;
|
||
}
|
||
|
||
// ── Answering a finding in words ─────────────────────────────────────────────────────────────
|
||
// The buttons are unambiguous by construction. This is for the other path — replying "yeah go
|
||
// ahead" in the chat that raised the finding — and it is matched here rather than asked of the
|
||
// model, because the model's answer would be a conf write and a wrong reading of "no, leave it"
|
||
// is not recoverable by apologising.
|
||
//
|
||
// Same shape as vv_ai_route_from_chat(): anchored patterns, most specific first, and anything
|
||
// unrecognised returns null so the assistant asks again instead of guessing. Two actions both
|
||
// matching is also null — "leave it, I know" and "leave it for now" differ by one clause and
|
||
// mean different things, so a phrase that supports both is not an instruction yet.
|
||
const VV_AI_ACTION_PATTERNS = [
|
||
// Acknowledge — "this is deliberate, stop telling me".
|
||
'ack' => [
|
||
'/\b(i|we) know\b/u',
|
||
'/\b(that|this|it)(?:\'s| is) (fine|expected|intentional|deliberate|on purpose)\b/u',
|
||
'/\bon purpose\b/u',
|
||
'/\b(aware|acknowledge|ack)\b/u',
|
||
'/\bmeant to be\b/u',
|
||
],
|
||
// Apply the proposed value.
|
||
'fix' => [
|
||
'/\bfix (it|that|this|them)?\b/u',
|
||
'/\b(go ahead|do it|apply|make the change|change it|update it|correct it)\b/u',
|
||
// A bare affirmative, as the whole message — "yes" answering "shall I fix it" is an
|
||
// instruction, "yes it looks wrong" is agreement about the diagnosis and nothing more.
|
||
// A trailing please is still bare.
|
||
'/\b(yes|yeah|yep|yup|sure|ok|okay)\b(\s*,?\s*please)?[\s,.!]*$/u',
|
||
'/\bplease do\b/u',
|
||
],
|
||
// Not now — no state written, it comes back next sweep.
|
||
'cancel' => [
|
||
// "leave it" is matched bare, not only as "leave it alone" / "leave it for now". The
|
||
// ambiguity this function is built to refuse — "leave it, I know" reading as both cancel
|
||
// and ack — did not actually arise with the longer forms, so that sentence resolved to
|
||
// ack and silenced the finding until the fault changed. The looser pattern is what makes
|
||
// the two readings collide and sends it back to be restated.
|
||
'/\b(not now|later|leave it\b|skip( it)?|cancel|ignore for now)\b/u',
|
||
'/\b(no|nope|nah)\b[\s,.!]*$/u',
|
||
'/\b(don\'?t|do not) (fix|touch|change|write|apply)\b/u',
|
||
],
|
||
// Never a problem. Deliberately narrow: this is the one answer that cannot expire on its
|
||
// own, so it is only read from a sentence that says so outright. Anything vaguer than these
|
||
// is meant to land on ack, which comes back by itself when the fault changes.
|
||
'dismiss' => [
|
||
'/\bdismiss\b/u',
|
||
'/\b(this|that|it)(?:\'s| is) not (a |an )?(problem|bug|issue|real)\b/u',
|
||
'/\bnever (a problem|an issue|report this)\b/u',
|
||
],
|
||
// Undo a close.
|
||
'reopen' => [
|
||
'/\breopen\b/u',
|
||
'/\bun-?dismiss\b/u',
|
||
],
|
||
];
|
||
|
||
// Returns one of the keys in VV_AI_ACTION_PATTERNS, or null when the reply does not clearly mean
|
||
// exactly one of them.
|
||
//
|
||
// Only call this when a finding is actually pending. A bare "yes" means fix in answer to "shall
|
||
// I fix it" and means nothing at all on its own, and the difference is context this function
|
||
// cannot see.
|
||
function vv_ai_finding_action_from_text(string $text): ?string {
|
||
$t = strtolower(trim($text));
|
||
if ($t === '') return null;
|
||
$t = preg_replace('/\s+/', ' ', $t);
|
||
|
||
$matched = [];
|
||
foreach (VV_AI_ACTION_PATTERNS as $action => $patterns) {
|
||
foreach ($patterns as $re) {
|
||
if (preg_match($re, $t)) { $matched[$action] = true; break; }
|
||
}
|
||
}
|
||
|
||
// Exactly one reading, or none. "leave it, I know" hits both ack and cancel; that is a
|
||
// sentence the operator should be asked to restate, not one to pick a winner from.
|
||
return count($matched) === 1 ? array_key_first($matched) : null;
|
||
}
|
||
|
||
// Carry out an answered action against a stored finding.
|
||
//
|
||
// Fix goes through the same guarded write path as everything else, and is the one place a
|
||
// toggle may be written — because reaching here means the operator asked for it by name. The
|
||
// unattended sweep never calls this.
|
||
function vv_ai_finding_apply_action(string $id, string $action, string $note = ''): array {
|
||
$f = vv_ai_finding_get($id);
|
||
if ($f === null) return ['ok' => false, 'error' => 'no such finding'];
|
||
|
||
// Only what this finding actually offers, in the state it is actually in. The page renders
|
||
// its buttons from the same function, but a stale tab holds buttons the store has moved past
|
||
// — a row acked in one window is still showing Fix in another — and the endpoint is reachable
|
||
// without either. Checking here is what makes vv_ai_finding_actions() the authority rather
|
||
// than a suggestion.
|
||
if (!isset(vv_ai_finding_actions($f)[$action])) {
|
||
return ['ok' => false, 'error' => 'not offered for this finding: ' . $action];
|
||
}
|
||
|
||
switch ($action) {
|
||
case 'ack':
|
||
return ['ok' => vv_ai_finding_ack($id, $note), 'action' => 'ack'];
|
||
|
||
case 'dismiss':
|
||
return ['ok' => vv_ai_finding_dismiss($id, $note), 'action' => 'dismiss'];
|
||
|
||
// Back to open, never straight back to needs_operator: whether it still cannot be
|
||
// repaired here is the next sweep's finding to make, not a state to restore.
|
||
case 'reopen':
|
||
return ['ok' => vv_ai_finding_set_state($id, 'open', $note), 'action' => 'reopen'];
|
||
|
||
case 'cancel':
|
||
// Deliberately writes nothing at all. "Not now" is not a state, it is the absence of
|
||
// one — recording it would make the finding look decided when it is still open.
|
||
return ['ok' => true, 'action' => 'cancel'];
|
||
|
||
case 'fix':
|
||
$proposed = $f['proposed'] ?? null;
|
||
if ($proposed === null) return ['ok' => false, 'error' => 'nothing proposed to write'];
|
||
|
||
$key = (string)$f['conf_key'];
|
||
$ok = vv_conf_write_changes([[
|
||
'file' => (string)($f['conf_file'] ?? 'master.conf'),
|
||
'key' => $key,
|
||
'value' => (string)$proposed,
|
||
'type' => 'scalar',
|
||
]]);
|
||
$wrote = !in_array(false, $ok, true);
|
||
|
||
if ($wrote) {
|
||
vv_ai_finding_set_state($id, 'fixed',
|
||
$note !== '' ? $note : 'Wrote ' . $key . ' at the operator\'s request.');
|
||
}
|
||
return ['ok' => $wrote, 'action' => 'fix',
|
||
'error' => $wrote ? null : 'conf write refused — see conf_changes.log'];
|
||
}
|
||
return ['ok' => false, 'error' => 'unknown action'];
|
||
}
|
||
|
||
// ── Resolving a log line back to a conf key ──────────────────────────────────────────────────
|
||
// By value wherever possible, by name only as a fallback.
|
||
//
|
||
// A log line usually contains the thing that failed — the URL that did not answer. That value
|
||
// came from a conf key, so searching the conf for which key holds it is an exact lookup with a
|
||
// definite answer. Guessing the key from the container's name is inference, and the failure mode
|
||
// is silent: HOST1_EMBY_URL and HOST1_EMBY_EXTERNAL_URL are both plausible for "Emby" and only
|
||
// one of them is the value that just failed.
|
||
//
|
||
// Same principle as resolve_tailscale_ip() refusing similarity matching for host identity: an
|
||
// exact match or an honest nothing.
|
||
|
||
// Every conf key whose value the given text starts with, longest first. Prefix rather than
|
||
// equality because a log reports the URL it actually called — the conf value plus an endpoint
|
||
// path — and the longest match is the most specific key that could have produced it.
|
||
function vv_ai_conf_keys_for_value(string $value): array {
|
||
$value = trim($value);
|
||
if ($value === '' || strlen($value) < 6) return [];
|
||
|
||
$hits = [];
|
||
foreach (vv_conf_vars() as $k => $v) {
|
||
$v = trim((string)$v);
|
||
if ($v === '' || strlen($v) < 6) continue;
|
||
if ($v === $value || str_starts_with($value, $v)) $hits[$k] = strlen($v);
|
||
}
|
||
arsort($hits);
|
||
return array_keys($hits);
|
||
}
|
||
|
||
// <HOSTID>_<SUBJECT>_<SUFFIX>, built literally and then checked for existence. Nothing is
|
||
// inferred: either the conf holds a key by exactly that name or this returns null.
|
||
function vv_ai_conf_key_for_subject(string $subject, string $suffix): ?string {
|
||
$norm = strtoupper(preg_replace('/[^A-Za-z0-9]+/', '_', trim($subject)));
|
||
if ($norm === '') return null;
|
||
|
||
$key = strtoupper(vv_detect_host()) . '_' . $norm . '_' . strtoupper($suffix);
|
||
return array_key_exists($key, vv_conf_vars()) ? $key : null;
|
||
}
|
||
|
||
// Which conf file a key lives in. A finding has to name the file it would be edited in, and
|
||
// host keys are not in master.conf.
|
||
function vv_ai_conf_file_for_key(string $key): string {
|
||
foreach (vv_get_conf_files() as $f) {
|
||
if (preg_match('/^\s*' . preg_quote($key, '/') . '\s*=/m', vv_read_conf_raw($f))) return $f;
|
||
}
|
||
return 'master.conf';
|
||
}
|
||
|
||
// ── Deterministic triage ─────────────────────────────────────────────────────────────────────
|
||
// Patterns are written against log formats that exist in this repo today, taken from the emit
|
||
// sites rather than imagined. A pattern that stops matching because its log line was reworded
|
||
// produces no finding, which is the safe direction — the alternative is a pattern loose enough
|
||
// to match anything, which fills the store with rows nobody can act on.
|
||
//
|
||
// No model runs here. "Connection refused" is not a judgement call, and a 14B model invoked
|
||
// after every cron job to notice it would be slower, costlier and less reliable than a regex.
|
||
// The model's job starts where these stop: explaining a finding, and talking the operator
|
||
// through the ones that cannot be repaired automatically.
|
||
const VV_AI_TRIAGE_PATTERNS = [
|
||
// docker_watchdog HTTP check — the richest signal, carrying both subject and failing URL.
|
||
[
|
||
're' => '/^(?P<subject>\S+) — not responding at (?P<observed>\S+) \(strike/u',
|
||
'kind' => 'unreachable',
|
||
'suffix' => 'URL',
|
||
],
|
||
// docker_watchdog API check, 401/403. The endpoint answered, so the address is right and
|
||
// the credential is not.
|
||
[
|
||
're' => '/^(?P<subject>\S+) — API check skipped \(HTTP (?:401|403)/u',
|
||
'kind' => 'auth_rejected',
|
||
'suffix' => 'API_KEY',
|
||
],
|
||
// Configured for an API check with nothing to authenticate with.
|
||
[
|
||
're' => '/^(?P<subject>\S+) — API check skipped \(no key configured\)/u',
|
||
'kind' => 'missing_value',
|
||
'suffix' => 'API_KEY',
|
||
],
|
||
[
|
||
're' => '/^Skipping (?P<subject>\S+) — placeholder API key/u',
|
||
'kind' => 'missing_value',
|
||
'suffix' => 'API_KEY',
|
||
],
|
||
[
|
||
're' => '/^(?P<subject>\S+) (?:—\s*)?API unreachable/u',
|
||
'kind' => 'unreachable',
|
||
'suffix' => 'URL',
|
||
],
|
||
];
|
||
|
||
// One log line in, at most one finding candidate out. Returns null for everything else, which is
|
||
// almost every line.
|
||
function vv_ai_triage_line(string $line): ?array {
|
||
// Strip the timestamp and level decoration the logger adds, so patterns can anchor on ^.
|
||
$body = preg_replace('/^\S+\s+\S+\s+(?:[^\[]*\[[A-Z]+\]\s*)?/u', '', rtrim($line));
|
||
$body = trim((string)$body);
|
||
if ($body === '') return null;
|
||
|
||
foreach (VV_AI_TRIAGE_PATTERNS as $p) {
|
||
if (!preg_match($p['re'], $body, $m)) continue;
|
||
|
||
$subject = trim($m['subject'] ?? '');
|
||
$observed = trim($m['observed'] ?? '');
|
||
if ($subject === '') continue;
|
||
|
||
// Value first, name second. Only one candidate key is accepted — two keys holding the
|
||
// same value means the log cannot say which one produced it, and picking either is the
|
||
// guess this whole approach exists to avoid.
|
||
$key = null;
|
||
if ($observed !== '') {
|
||
$byValue = vv_ai_conf_keys_for_value($observed);
|
||
if (count($byValue) === 1) $key = $byValue[0];
|
||
}
|
||
if ($key === null) $key = vv_ai_conf_key_for_subject($subject, $p['suffix']);
|
||
if ($key === null) continue; // nothing actionable — no row
|
||
|
||
return [
|
||
'kind' => $p['kind'],
|
||
'subject' => $subject,
|
||
'conf_key' => $key,
|
||
'conf_file' => vv_ai_conf_file_for_key($key),
|
||
'observed' => $observed !== '' ? $observed : (string)(vv_conf_vars()[$key] ?? ''),
|
||
'evidence' => $body,
|
||
];
|
||
}
|
||
return null;
|
||
}
|
||
|
||
// A whole log tail in, one candidate per distinct fault out. A job that retried twelve times
|
||
// produces twelve identical lines, and the store's dedupe would collapse them anyway — doing it
|
||
// here keeps the sweep from writing the same file twelve times in a row.
|
||
function vv_ai_triage_log(array $lines, string $sourceLog = ''): array {
|
||
$found = [];
|
||
foreach ($lines as $line) {
|
||
$c = vv_ai_triage_line((string)$line);
|
||
if ($c === null) continue;
|
||
$c['source_log'] = $sourceLog;
|
||
$found[vv_ai_finding_id($c['kind'], $c['subject'], $c['conf_key'])] = $c;
|
||
}
|
||
return array_values($found);
|
||
}
|
||
|
||
// ── The phrasebook ───────────────────────────────────────────────────────────────────────────
|
||
// What the operator said, what it was taken to mean, and whether that was right.
|
||
//
|
||
// The point is not to fine-tune anything. It is that "daily", said three times and meaning
|
||
// daily_sync_maintenance.sh all three, stops being an inference and becomes a lookup. This file
|
||
// is how a term earns that promotion, and everything it promotes is exact — the model keeps the
|
||
// language, the resolution stays deterministic. Same division as the rest of this subsystem.
|
||
//
|
||
// The corrections are the rows that matter. A resolution that was right confirms what was
|
||
// already believed; a resolution that was wrong, with what it should have been, is the only
|
||
// record of a mistake that would otherwise be repeated indefinitely.
|
||
//
|
||
// JSON Lines rather than the pipe-delimited shape the token ledger uses: that file holds numbers
|
||
// and a hostname, this one holds whatever the operator typed, and a delimiter that occurs in the
|
||
// data is not a delimiter. Append-only, one object per line, so a truncated write costs the last
|
||
// row rather than the corpus.
|
||
|
||
function vv_ai_phrasebook_path(): string {
|
||
return AI_DATA_DIR . '/ai_phrasebook.jsonl';
|
||
}
|
||
|
||
// $said what the operator actually wrote, verbatim but redacted
|
||
// $term the fragment that carried the meaning — "daily", "critical rsync", "emby key"
|
||
// $target what it resolved to: a conf key, a script id, an array name
|
||
// $kind conf_key | script | array | section | unknown
|
||
// $outcome accepted | corrected | rejected
|
||
// $correctedTo what it should have been, when the resolution was wrong
|
||
//
|
||
// $target and $correctedTo must be a canonical identifier — CONF_BACKUP_DIR, not
|
||
// "CONF_BACKUP_DIR=${DATA_DIR}/Backups/Confs" and not "the backups directory". Promotion works
|
||
// by counting how often a term resolved to the same thing, so a target described three different
|
||
// ways is three meanings, and the term never promotes. Learned the hard way while seeding this
|
||
// with real corrections: the same fix, written up three ways, taught nothing.
|
||
function vv_ai_phrase_record(string $said, string $term, string $target, string $kind,
|
||
string $outcome = 'accepted', string $correctedTo = ''): bool {
|
||
$said = trim($said);
|
||
$term = strtolower(trim($term));
|
||
if ($said === '' || $term === '') return false;
|
||
if (!in_array($outcome, ['accepted', 'corrected', 'rejected'], true)) return false;
|
||
|
||
if (!is_dir(AI_DATA_DIR)) @mkdir(AI_DATA_DIR, 0755, true);
|
||
|
||
// An operator asking to set a credential types the credential. This file is long-lived and
|
||
// read back for years; it is the last place a key should be preserved verbatim.
|
||
$row = [
|
||
'ts' => time(),
|
||
'said' => mb_substr(vv_ai_redact($said), 0, 500),
|
||
'term' => mb_substr($term, 0, 80),
|
||
'target' => mb_substr(trim($target), 0, 160),
|
||
'kind' => $kind,
|
||
'outcome' => $outcome,
|
||
];
|
||
if ($correctedTo !== '') $row['corrected_to'] = mb_substr(vv_ai_redact($correctedTo), 0, 160);
|
||
|
||
return @file_put_contents(vv_ai_phrasebook_path(),
|
||
json_encode($row, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE) . "\n",
|
||
FILE_APPEND | LOCK_EX) !== false;
|
||
}
|
||
|
||
function vv_ai_phrase_all(): array {
|
||
$out = [];
|
||
foreach ((array)@file(vv_ai_phrasebook_path(), FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES) as $line) {
|
||
$r = json_decode($line, true);
|
||
if (is_array($r) && isset($r['term'])) $out[] = $r;
|
||
}
|
||
return $out;
|
||
}
|
||
|
||
// Terms that have earned a deterministic mapping: seen at least $minSeen times, resolving to one
|
||
// target every time, and never corrected away from it.
|
||
//
|
||
// A single contradiction disqualifies the term outright rather than going with the majority. The
|
||
// whole value of promoting a term is that it stops being a guess — a term that meant two things
|
||
// is still a guess, and a confident wrong alias is worse than no alias, because nothing downstream
|
||
// will question it.
|
||
function vv_ai_phrase_aliases(int $minSeen = 3): array {
|
||
$seen = [];
|
||
foreach (vv_ai_phrase_all() as $r) {
|
||
$term = (string)$r['term'];
|
||
// A correction records both the wrong reading and the right one. The right one is what
|
||
// the term means; the wrong one is what it must never be promoted to again.
|
||
$target = ($r['outcome'] === 'corrected' && !empty($r['corrected_to']))
|
||
? (string)$r['corrected_to'] : (string)$r['target'];
|
||
if ($r['outcome'] === 'rejected' || $target === '') continue;
|
||
|
||
$seen[$term]['targets'][$target] = ($seen[$term]['targets'][$target] ?? 0) + 1;
|
||
$seen[$term]['kind'] = (string)($r['kind'] ?? 'unknown');
|
||
if ($r['outcome'] === 'corrected') $seen[$term]['wrong'][(string)$r['target']] = true;
|
||
}
|
||
|
||
$aliases = [];
|
||
foreach ($seen as $term => $d) {
|
||
if (count($d['targets']) !== 1) continue; // meant two things — still a guess
|
||
$target = array_key_first($d['targets']);
|
||
if (isset($d['wrong'][$target])) continue; // was itself corrected away from
|
||
if ($d['targets'][$target] < $minSeen) continue;
|
||
$aliases[$term] = ['target' => $target, 'kind' => $d['kind'], 'seen' => $d['targets'][$target]];
|
||
}
|
||
return $aliases;
|
||
}
|
||
|
||
// Exact alias hit for a term, or null. Deliberately not fuzzy: an alias exists precisely so that
|
||
// this lookup is certain, and a near-match would reintroduce the guessing it replaced.
|
||
function vv_ai_phrase_lookup(string $term, int $minSeen = 3): ?array {
|
||
return vv_ai_phrase_aliases($minSeen)[strtolower(trim($term))] ?? null;
|
||
}
|
||
|
||
// ── What the repair profile is given before it answers ───────────────────────────────────────
|
||
// Two records of what has already happened, assembled as plain text for the prompt.
|
||
//
|
||
// The phrasebook half is the operator's vocabulary — and the corrections matter more than the
|
||
// settled terms, because a term that has been corrected is one the assistant has already got
|
||
// wrong once and would otherwise get wrong again.
|
||
//
|
||
// The closed-findings half is this installation's history: the same fault, and what actually
|
||
// ended it. "Emby stopped answering last month and the address had changed" is worth more at the
|
||
// start of a repair conversation than any amount of reasoning from first principles.
|
||
//
|
||
// Bounded hard. This goes into a 16k context that retrieval and a log tail are also competing
|
||
// for, and an unbounded history would crowd out the evidence for the fault actually being
|
||
// discussed.
|
||
function vv_ai_repair_context(int $maxAliases = 20, int $maxFixes = 8): string {
|
||
$out = [];
|
||
|
||
$aliases = vv_ai_phrase_aliases();
|
||
if ($aliases) {
|
||
$lines = [];
|
||
foreach (array_slice($aliases, 0, $maxAliases, true) as $term => $a) {
|
||
$lines[] = ' "' . $term . '" means ' . $a['target'];
|
||
}
|
||
$out[] = "What the operator calls things:\n" . implode("\n", $lines);
|
||
}
|
||
|
||
$unsettled = vv_ai_phrase_unsettled();
|
||
if ($unsettled) {
|
||
$lines = [];
|
||
foreach (array_slice($unsettled, 0, $maxAliases, true) as $term => $c) {
|
||
$last = end($c);
|
||
$lines[] = ' "' . $term . '" was read as ' . ($last['took_it_as'] ?: '?')
|
||
. ' and meant ' . ($last['meant'] ?: '?');
|
||
}
|
||
$out[] = "Corrected before — do not repeat these:\n" . implode("\n", $lines);
|
||
}
|
||
|
||
$closed = vv_ai_findings_list(['fixed', 'resolved']);
|
||
if ($closed) {
|
||
$lines = [];
|
||
foreach (array_slice($closed, 0, $maxFixes) as $f) {
|
||
$lines[] = ' ' . ($f['subject'] ?? '?') . ' / ' . ($f['ref'] ?? $f['conf_key'] ?? '?')
|
||
. ' — ' . ($f['state'] ?? '?')
|
||
. (!empty($f['note']) ? ': ' . mb_substr((string)$f['note'], 0, 140) : '');
|
||
}
|
||
$out[] = "Already dealt with on this host:\n" . implode("\n", $lines);
|
||
}
|
||
|
||
$spellings = vv_ai_spellings();
|
||
if ($spellings) {
|
||
$lines = [];
|
||
foreach (array_slice($spellings, 0, $maxAliases, true) as $typo => $meant) {
|
||
$lines[] = ' "' . $typo . '" = "' . $meant . '"';
|
||
}
|
||
$out[] = "The operator types quickly; known shorthand:\n" . implode("\n", $lines);
|
||
}
|
||
|
||
return implode("\n\n", $out);
|
||
}
|
||
|
||
// ── Resolving what the operator named ────────────────────────────────────────────────────────
|
||
// "turn off the zfs scrub", "change the emby api key" — a phrase in, an exact target out, or an
|
||
// honest refusal with the candidates that were considered.
|
||
//
|
||
// Four layers, most certain first, and each either answers exactly or declines:
|
||
//
|
||
// 1. A learned alias. The operator has used this term before and it settled on one target.
|
||
// 2. The literal key. "HOST1_EMBY_URL" or "host1 emby url" is not a phrase to interpret.
|
||
// 3. Every conf key containing all of the significant words. Exactly one is an answer; more
|
||
// than one is a question.
|
||
// 4. Script ids from the orchestrator arrays, matched the same way.
|
||
//
|
||
// What this deliberately does not do is score similarity. There is no closest match, no edit
|
||
// distance, no "did you mean". The same reasoning as resolve_tailscale_ip() refusing to guess at
|
||
// host identity: the failure mode of a near-match is silent and confident, and here it would
|
||
// write to a key the operator never named. Ambiguity is returned as ambiguity, and the assistant
|
||
// asks — which is also how the phrasebook learns, because the answer to that question is a
|
||
// correction worth recording.
|
||
//
|
||
// Stop words exist because "the", "for" and "in" appear in a conf key somewhere and would make
|
||
// every phrase match everything.
|
||
const VV_AI_RESOLVE_STOPWORDS = [
|
||
'the','a','an','to','for','in','on','of','and','or','is','it','this','that','my','our',
|
||
'please','can','you','set','change','turn','make','update','put','value','key','conf','config',
|
||
'setting','settings','off','on_','now','back','again','me','we','let','lets',
|
||
];
|
||
|
||
// A key or script id broken into its own words. HOST1_EMBY_API_KEY is four words, and
|
||
// Tools/zfs_pool_scrub.sh is five — the path separator and the extension are word boundaries too.
|
||
function vv_ai_resolve_segments(string $name): array {
|
||
$n = strtolower(preg_replace('/\.sh$/', '', $name));
|
||
return array_values(array_filter(preg_split('/[^a-z0-9]+/', $n, -1, PREG_SPLIT_NO_EMPTY) ?: []));
|
||
}
|
||
|
||
function vv_ai_resolve_tokens(string $phrase): array {
|
||
$p = strtolower(trim($phrase));
|
||
$p = preg_replace('/[^a-z0-9_\s-]+/', ' ', $p);
|
||
$words = preg_split('/[\s_-]+/', $p, -1, PREG_SPLIT_NO_EMPTY) ?: [];
|
||
return array_values(array_filter($words,
|
||
fn($w) => strlen($w) > 1 && !in_array($w, VV_AI_RESOLVE_STOPWORDS, true)));
|
||
}
|
||
|
||
// Script ids named in any *_SCRIPTS array, so "zfs scrub" can resolve to Tools/zfs_pool_scrub.sh
|
||
// — which is the shape of request that has no conf key at all.
|
||
function vv_ai_resolve_script_ids(): array {
|
||
$ids = [];
|
||
foreach (vv_get_conf_files() as $f) {
|
||
if (preg_match_all('/^\s*#?\s*"([A-Za-z0-9_\/.-]+\.sh)(?:\s[^"]*)?"/m',
|
||
vv_read_conf_raw($f), $m)) {
|
||
foreach ($m[1] as $id) $ids[$id] = true;
|
||
}
|
||
}
|
||
return array_keys($ids);
|
||
}
|
||
|
||
// Returns:
|
||
// ok=true with target, kind and via — one certain answer
|
||
// ok=false with candidates — several, and the caller must ask
|
||
// ok=false with candidates empty — nothing recognised
|
||
function vv_ai_resolve_target(string $phrase): array {
|
||
$none = ['ok' => false, 'target' => null, 'kind' => null, 'via' => 'none', 'candidates' => []];
|
||
$tokens = vv_ai_resolve_tokens($phrase);
|
||
if (!$tokens) return $none;
|
||
|
||
// 1 — learned
|
||
$alias = vv_ai_phrase_lookup(strtolower(trim($phrase)));
|
||
if ($alias === null) {
|
||
// Also try the significant words alone, since "turn off ai repair" and "ai repair" are
|
||
// the same instruction with different framing.
|
||
$alias = vv_ai_phrase_lookup(implode(' ', $tokens));
|
||
}
|
||
if ($alias !== null) {
|
||
return ['ok' => true, 'target' => $alias['target'], 'kind' => $alias['kind'],
|
||
'via' => 'alias', 'candidates' => []];
|
||
}
|
||
|
||
$vars = vv_conf_vars();
|
||
|
||
// 2 — the literal key, however it was spaced or cased
|
||
$literal = strtoupper(implode('_', $tokens));
|
||
if (array_key_exists($literal, $vars)) {
|
||
return ['ok' => true, 'target' => $literal, 'kind' => 'conf_key',
|
||
'via' => 'exact', 'candidates' => []];
|
||
}
|
||
|
||
// 3 — conf keys whose own words include every significant word
|
||
//
|
||
// Whole segments, not substrings. Substring matching made "mov" resolve to
|
||
// MOVER_STOP_TIMEOUT with full confidence, which is exactly the near-match this is supposed
|
||
// to refuse: a short fragment that happens to be unique is not the operator naming a key.
|
||
$keyHits = [];
|
||
foreach (array_keys($vars) as $key) {
|
||
$segs = vv_ai_resolve_segments($key);
|
||
foreach ($tokens as $t) {
|
||
if (!in_array($t, $segs, true)) continue 2;
|
||
}
|
||
$keyHits[] = $key;
|
||
}
|
||
if (count($keyHits) === 1) {
|
||
return ['ok' => true, 'target' => $keyHits[0], 'kind' => 'conf_key',
|
||
'via' => 'match', 'candidates' => []];
|
||
}
|
||
|
||
// 4 — script ids, same rule
|
||
$scriptHits = [];
|
||
foreach (vv_ai_resolve_script_ids() as $id) {
|
||
$segs = vv_ai_resolve_segments($id);
|
||
foreach ($tokens as $t) {
|
||
if (!in_array($t, $segs, true)) continue 2;
|
||
}
|
||
$scriptHits[] = $id;
|
||
}
|
||
if (!$keyHits && count($scriptHits) === 1) {
|
||
return ['ok' => true, 'target' => $scriptHits[0], 'kind' => 'script',
|
||
'via' => 'match', 'candidates' => []];
|
||
}
|
||
|
||
$all = array_merge($keyHits, $scriptHits);
|
||
if (!$all) return $none;
|
||
|
||
// Several. Returned rather than ranked — picking one here is the guess this avoids.
|
||
sort($all);
|
||
return ['ok' => false, 'target' => null, 'kind' => null, 'via' => 'ambiguous',
|
||
'candidates' => array_slice($all, 0, 12)];
|
||
}
|
||
|
||
// ── Spellings ────────────────────────────────────────────────────────────────────────────────
|
||
// The operator types quickly and knows it: "haversync" for "have rsync", "as it to the list" for
|
||
// "add it". Recorded when the meaning was obvious in context, so the next occurrence is read
|
||
// rather than puzzled over.
|
||
//
|
||
// Promoted on first sighting, unlike a term alias, and the difference is deliberate. A term alias
|
||
// decides what gets written to conf, so it has to be earned by repetition. A spelling decides how
|
||
// a sentence is read, costs a misreading at worst, and is rarely made identically three times —
|
||
// requiring repetition would mean never learning any of them.
|
||
//
|
||
// Recorded only when the intent was actually clear. A guess written down here is worse than
|
||
// leaving it out, because it will be applied silently every time afterwards.
|
||
function vv_ai_spelling_record(string $said, string $typo, string $meant): bool {
|
||
$typo = strtolower(trim($typo));
|
||
$meant = trim($meant);
|
||
if ($typo === '' || $meant === '' || strcasecmp($typo, $meant) === 0) return false;
|
||
return vv_ai_phrase_record($said, $typo, $meant, 'spelling');
|
||
}
|
||
|
||
// What the operator most likely meant by a word, or null. Last writing wins, so a correction to
|
||
// an earlier reading simply supersedes it.
|
||
function vv_ai_spelling_lookup(string $word): ?string {
|
||
$word = strtolower(trim($word));
|
||
$hit = null;
|
||
foreach (vv_ai_phrase_all() as $r) {
|
||
if (($r['kind'] ?? '') !== 'spelling' || ($r['outcome'] ?? '') === 'rejected') continue;
|
||
if (($r['term'] ?? '') === $word) $hit = (string)$r['target'];
|
||
}
|
||
return $hit;
|
||
}
|
||
|
||
function vv_ai_spellings(): array {
|
||
$out = [];
|
||
foreach (vv_ai_phrase_all() as $r) {
|
||
if (($r['kind'] ?? '') !== 'spelling' || ($r['outcome'] ?? '') === 'rejected') continue;
|
||
$out[(string)$r['term']] = (string)$r['target'];
|
||
}
|
||
ksort($out);
|
||
return $out;
|
||
}
|
||
|
||
// Terms that have been corrected and have not yet earned promotion — what the assistant is still
|
||
// getting wrong, and the thing worth reading when asking why it keeps mistaking something.
|
||
function vv_ai_phrase_unsettled(): array {
|
||
$out = [];
|
||
foreach (vv_ai_phrase_all() as $r) {
|
||
if (($r['outcome'] ?? '') !== 'corrected') continue;
|
||
$out[(string)$r['term']][] = ['said' => $r['said'], 'took_it_as' => $r['target'],
|
||
'meant' => $r['corrected_to'] ?? '', 'ts' => $r['ts'] ?? 0];
|
||
}
|
||
return $out;
|
||
}
|
||
|
||
// ── What the arrs say about themselves ───────────────────────────────────────────────────────
|
||
// Sonarr, Radarr and Lidarr each publish a health endpoint listing what they believe is wrong,
|
||
// already structured and already graded. No log parsing, no pattern that goes stale when a
|
||
// message is reworded, and no guessing at severity — the arr is the authority on whether its own
|
||
// condition is an error or a warning.
|
||
//
|
||
// This is the one source here that needs no triage at all. Everything else in this file exists
|
||
// because logs are prose; these arrive as records.
|
||
//
|
||
// Note the API version differs: Lidarr is v1 where Sonarr and Radarr are v3. vv_discover_arrs()
|
||
// already carries it per arr, which is why this reads it rather than assuming.
|
||
function vv_ai_arr_health_findings(): array {
|
||
if (!function_exists('vv_discover_arrs')) {
|
||
require_once __DIR__ . '/arrs.php';
|
||
}
|
||
|
||
$host = vv_detect_host();
|
||
$found = [];
|
||
|
||
foreach (vv_discover_arrs() as $node) {
|
||
if (($node['host'] ?? '') !== $host) continue;
|
||
|
||
foreach ($node['arrs'] ?? [] as $arr) {
|
||
$type = ucfirst((string)($arr['type'] ?? ''));
|
||
$url = (string)($arr['url'] ?? '');
|
||
$key = (string)($arr['key'] ?? '');
|
||
$api = (string)($arr['api'] ?? 'v3');
|
||
if ($type === '' || $url === '' || $key === '') continue;
|
||
|
||
$items = vv_arr_http($url, $key, "/api/$api/health", vv_ai_probe_timeout());
|
||
// null is unreachable, which is a different finding and one the log triage already
|
||
// raises. An empty array is the arr saying it is fine, and must not be confused with
|
||
// not having been able to ask.
|
||
if (!is_array($items)) continue;
|
||
|
||
foreach ($items as $item) {
|
||
$source = trim((string)($item['source'] ?? ''));
|
||
$message = trim((string)($item['message'] ?? ''));
|
||
if ($source === '' || $message === '') continue;
|
||
|
||
$found[] = [
|
||
'kind' => 'arr_health',
|
||
'subject' => $type,
|
||
'ref' => $source,
|
||
'conf_key' => '',
|
||
'conf_file' => '',
|
||
'arr_type' => strtolower((string)($item['type'] ?? 'warning')),
|
||
'observed' => $message,
|
||
'evidence' => $type . ' › ' . $source . ': ' . $message,
|
||
'source_log' => $type . ' /api/' . $api . '/health',
|
||
// The arr's own documentation for this check, which is the actual next step
|
||
// for most of them and costs nothing to carry.
|
||
'note' => trim((string)($item['wikiUrl'] ?? '')),
|
||
];
|
||
}
|
||
}
|
||
break;
|
||
}
|
||
return $found;
|
||
}
|
||
|
||
// ── Proving a candidate ──────────────────────────────────────────────────────────────────────
|
||
// The guard the whole unattended path rests on: nothing is written that has not answered.
|
||
//
|
||
// A model can be confident that a port should be 8686. A probe can report that 8686 answered.
|
||
// Only the second is a fact, and only facts get written to conf without being asked. Everything
|
||
// a probe cannot settle becomes a conversation instead — which is not a lesser outcome, it is
|
||
// the honest one for a value that cannot be derived from this machine.
|
||
//
|
||
// Deliberately narrow. These check reachability and identity, never correctness of behaviour:
|
||
// that Lidarr answers on 8686 does not prove 8686 is the port you meant, only that something is
|
||
// listening there and calling itself Lidarr. Proving intent is not a probe's job.
|
||
|
||
function vv_ai_probe_timeout(): int {
|
||
return max(1, (int)(vv_conf_vars()['AI_PROBE_TIMEOUT'] ?? 4));
|
||
}
|
||
|
||
// Does this URL answer at all? Any HTTP status counts, including 401 — a refusal is proof that
|
||
// something is listening and speaking HTTP, which is exactly what an address probe is asking.
|
||
// Distinguishing "wrong address" from "wrong credential" is the point of having both kinds.
|
||
function vv_ai_probe_url(string $url): array {
|
||
if (!preg_match('#^https?://[^\s/$.?\#][^\s]*$#i', $url)) {
|
||
return ['ok' => false, 'reason' => 'not a url'];
|
||
}
|
||
|
||
$ch = curl_init($url);
|
||
curl_setopt_array($ch, [
|
||
CURLOPT_RETURNTRANSFER => true,
|
||
CURLOPT_NOBODY => true,
|
||
CURLOPT_TIMEOUT => vv_ai_probe_timeout(),
|
||
CURLOPT_CONNECTTIMEOUT => vv_ai_probe_timeout(),
|
||
CURLOPT_SSL_VERIFYPEER => false, // these are LAN and Tailscale endpoints, often self-signed
|
||
CURLOPT_SSL_VERIFYHOST => false,
|
||
]);
|
||
curl_exec($ch);
|
||
$code = (int)curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
|
||
$err = curl_error($ch);
|
||
curl_close($ch);
|
||
|
||
return $code > 0
|
||
? ['ok' => true, 'code' => $code]
|
||
: ['ok' => false, 'reason' => $err !== '' ? $err : 'no response'];
|
||
}
|
||
|
||
// Every container on this host, by exact name. The membership test for unknown_target findings,
|
||
// and one docker call rather than one per candidate.
|
||
function vv_ai_container_names(): array {
|
||
static $names = null;
|
||
if ($names !== null) return $names;
|
||
|
||
$out = [];
|
||
@exec('timeout ' . vv_ai_probe_timeout() . " docker ps -a --format '{{.Names}}' 2>/dev/null", $out, $rc);
|
||
return $names = ($rc === 0) ? array_values(array_filter(array_map('trim', $out))) : [];
|
||
}
|
||
|
||
// Candidate corrections for a URL whose host or port stopped answering.
|
||
//
|
||
// Only two transformations, both conservative: the same host on a port that some other conf key
|
||
// already uses, and the same port on a host some other conf key already names. Both draw
|
||
// exclusively from values already present in this installation's conf — nothing is invented, and
|
||
// a scan of the port range is deliberately not attempted. Finding *a* listening port is not the
|
||
// same as finding the right service, and a probe that accepts any answer would happily point
|
||
// Lidarr at Sonarr.
|
||
function vv_ai_url_candidates(string $observed): array {
|
||
$parts = @parse_url($observed);
|
||
if (!is_array($parts) || empty($parts['host'])) return [];
|
||
|
||
$hosts = $ports = [];
|
||
foreach (vv_conf_vars() as $k => $v) {
|
||
$v = trim((string)$v);
|
||
if (!preg_match('#^https?://#i', $v)) continue;
|
||
$p = @parse_url($v);
|
||
if (!is_array($p) || empty($p['host'])) continue;
|
||
$hosts[$p['host']] = true;
|
||
if (!empty($p['port'])) $ports[(int)$p['port']] = true;
|
||
}
|
||
|
||
$scheme = $parts['scheme'] ?? 'http';
|
||
$path = $parts['path'] ?? '';
|
||
$out = [];
|
||
|
||
foreach (array_keys($ports) as $port) {
|
||
$c = $scheme . '://' . $parts['host'] . ':' . $port . $path;
|
||
if ($c !== $observed) $out[$c] = true;
|
||
}
|
||
foreach (array_keys($hosts) as $host) {
|
||
$port = !empty($parts['port']) ? ':' . $parts['port'] : '';
|
||
$c = $scheme . '://' . $host . $port . $path;
|
||
if ($c !== $observed) $out[$c] = true;
|
||
}
|
||
return array_keys($out);
|
||
}
|
||
|
||
// Try to prove a correction for one finding. Returns the finding with 'proposed' and 'proven'
|
||
// filled in, or unchanged when nothing could be proven — which is the common case and not a
|
||
// failure.
|
||
//
|
||
// auth_rejected and missing_value are never proven here on purpose. A credential cannot be
|
||
// derived from this host by definition: if it could be read from somewhere, it would not be a
|
||
// credential. Those go straight to the operator.
|
||
function vv_ai_probe_finding(array $f): array {
|
||
$kind = (string)($f['kind'] ?? '');
|
||
|
||
if ($kind === 'unknown_target') {
|
||
$observed = (string)($f['observed'] ?? '');
|
||
// Exact membership only. A container name is a literal, and "close to an existing name"
|
||
// is how a repair renames the wrong thing.
|
||
if (in_array($observed, vv_ai_container_names(), true)) {
|
||
return $f; // it exists after all — nothing to correct
|
||
}
|
||
return $f + ['state' => 'needs_operator'];
|
||
}
|
||
|
||
if ($kind !== 'unreachable') {
|
||
// Nothing on this machine can supply a credential, so there is nothing to prove.
|
||
$f['state'] = 'needs_operator';
|
||
return $f;
|
||
}
|
||
|
||
$observed = (string)($f['observed'] ?? '');
|
||
|
||
// If the observed address answers now, the fault has cleared on its own — a host that was
|
||
// rebooting, most often. Recording a proposal here would repair something already working.
|
||
if (vv_ai_probe_url($observed)['ok']) {
|
||
$f['state'] = 'resolved';
|
||
$f['note'] = 'Answered when probed — the address was reachable again by the time this ran.';
|
||
return $f;
|
||
}
|
||
|
||
$answered = [];
|
||
foreach (vv_ai_url_candidates($observed) as $candidate) {
|
||
if (vv_ai_probe_url($candidate)['ok']) $answered[] = $candidate;
|
||
}
|
||
|
||
// Exactly one, or none. Two addresses answering means the probe cannot say which is the
|
||
// right one, and picking either is the guess this exists to prevent.
|
||
if (count($answered) === 1) {
|
||
$f['proposed'] = $answered[0];
|
||
$f['proven'] = true;
|
||
$f['note'] = 'Probed ' . $answered[0] . ' and it answered; ' . $observed . ' did not.';
|
||
return $f;
|
||
}
|
||
|
||
$f['proven'] = false;
|
||
$f['state'] = 'needs_operator';
|
||
$f['note'] = $answered
|
||
? 'Several addresses answered (' . implode(', ', $answered) . '), so none was written.'
|
||
: 'Nothing answered at ' . $observed . ', and no address in the conf answered either.';
|
||
return $f;
|
||
}
|