'a configured address or port refused, timed out, or did not resolve', 'auth_rejected' => 'the endpoint answered, and rejected the credential', 'unknown_target' => 'a conf entry names a container or share that does not exist here', 'missing_value' => 'a conf key required by the job that ran is empty', 'arr_health' => 'an arr is reporting a problem about itself', 'system_fault' => 'the kernel reported a hardware or filesystem fault about this machine', 'container_fault' => 'a container is logging a fault about its own environment', 'media_misfiled' => 'a series is shelved somewhere its own metadata does not support', 'watchdog_strike' => 'a watchdog has counted something far enough to be worth a record', // The auth stack's two. Neither is a conf problem — one is a service behind a proxy host that // has stopped answering, the other is a hostname put behind Authelia that Authelia then lets // everyone past. They are recorded here rather than only drawn on the Auth tab because both // are conditions that persist for weeks without anyone opening that tab: one host on this // installation has served nothing but 5xx for months. 'proxy_down' => 'a proxy host has been failing long enough that it is not a blip', 'access_open' => 'a hostname behind an auth_request block is not actually protected by any rule', ]; // Which kinds are a statement about Varaverk's configuration, and which are a statement about // something else that is nonetheless worth recording. // // The rule was originally "a finding names a conf key or it is not a finding", to stop the store // filling with observations nobody could act on. What that rule was really protecting is that // every finding identifies something specific and actionable — naming a conf key was the proxy, // because at the time every source of findings was a conf problem. // // An arr reporting "all lists are unavailable" is specific and actionable, and there is no // Varaverk key to change: the action is in Radarr's own UI. So the identity widens to a general // reference, and the conf-key requirement narrows to the kinds it was written for. What has not // changed is that a finding with nothing to point at is still refused. const VV_AI_CONF_BOUND_KINDS = ['unreachable', 'auth_rejected', 'unknown_target', 'missing_value']; function vv_ai_kind_is_conf_bound(string $kind): bool { return in_array($kind, VV_AI_CONF_BOUND_KINDS, true); } // How a finding ended, when it ends. const VV_AI_FINDING_STATES = [ 'open' => 'seen, not yet acted on', 'needs_operator' => 'cannot be repaired here — the value is not derivable from this host', 'acknowledged' => 'the operator knows, and it stays quiet until the state it was acked at changes', 'fixed' => 'a proven value was written to conf', 'resolved' => 'the probe now passes; whatever was wrong is no longer wrong', 'dismissed' => 'the operator says this is not a problem, permanently', ]; // ── Why acknowledged is not dismissed ──────────────────────────────────────────────────────── // "I know critical rsync is off, stop telling me" and "this is never a problem" are different // instructions, and collapsing them loses the half that matters. An acknowledgement is scoped to // the state it was given in: CRITICAL_RSYNC_ENABLED being false is a deliberate choice today and // a stale note the moment it goes true again. // // So an ack records what the key read when it was given, and expires when that changes. The // finding comes back on its own, without the operator having to remember to look — which is the // difference between a note and a silence. // ── Toggles are the operator's, always ─────────────────────────────────────────────────────── // A repair may never enable or disable anything on its own. Not because it would get the value // wrong — a boolean has only two — but because the value is not a fact to be discovered. Whether // critical rsync should be on is a decision about intent, and a probe cannot prove intent the // way it can prove that a port answers. // // The Fix action still writes it when the operator asks for it. What is forbidden is the // unattended path choosing for them. function vv_ai_conf_is_toggle(string $key): bool { $v = strtolower(trim((string)(vv_conf_vars()[$key] ?? ''))); return $v === 'true' || $v === 'false'; } // May the sweep write this without being asked? Two conditions, both required: a probe actually // answered on the proposed value, and the key is not a toggle. function vv_ai_finding_may_autofix(array $f): bool { if (!vv_ai_repair_autofix_enabled()) return false; if (empty($f['proven'])) return false; if (($f['proposed'] ?? null) === null) return false; if (vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))) return false; // A third condition, for paths only. "Proven" means a probe answered, and every probe this // has is a network probe — nothing in it can answer a filesystem question, so a path // proposal reaches here carrying a proof that is about something else entirely. Requiring // the directory to exist is the equivalent evidence, and it is the difference between // pointing a cleanup at a real share and pointing it at a typo that will be created empty // by the first script to write there. // // vv_conf_path_write_ok() still runs inside the writer underneath this. That one refuses // what is dangerous; this one refuses what is merely unproven, which is a bar only the // unattended path has to clear. $proposed = (string)$f['proposed']; if ($proposed !== '' && $proposed[0] === '/' && !file_exists($proposed)) return false; return true; } // ── Severity is derived, never supplied ────────────────────────────────────────────────────── // Same ladder run_job.sh records runs against — ok / warn / error — so a finding and the run it // came from cannot describe the same event at two different volumes. // // The rule that matters: a finding whose key is a toggle can never be an error. Something not // happening because it was switched off is the switch working. That is true whether the switch // was flipped deliberately last month or by accident this morning, and the store cannot tell // those apart — so it reports the fact and lets the operator supply the intent. // // Everything else takes its level from what the fault costs. A credential the endpoint rejected // stops that integration dead; an address that does not answer might be a host still booting. function vv_ai_finding_severity(array $f): string { $key = (string)($f['conf_key'] ?? ''); // Deliberate-state findings never escalate, whatever their kind. if ($key !== '' && vv_ai_conf_is_toggle($key)) return 'warn'; // An arr grades its own health and is the authority on it — "error" from Radarr means Radarr // has stopped doing something, which is not a judgement to second-guess from out here. if (($f['kind'] ?? '') === 'arr_health') { return ($f['arr_type'] ?? '') === 'error' ? 'error' : 'warn'; } // Same shape as arr_health, for the same reason: the level belongs to the pattern that // matched, because "correctable PCIe error" and "I/O error on a disk" are the same kind of // finding and nothing like the same news. Carried on the candidate rather than inferred here. // Shared by both log-derived kinds — one field, because they mean exactly the same thing. if (in_array($f['kind'] ?? '', ['system_fault', 'container_fault'], true)) { return ($f['sys_level'] ?? '') === 'error' ? 'error' : 'warn'; } return match ($f['kind'] ?? '') { 'auth_rejected' => 'error', // answered and refused — nothing gets through until fixed 'missing_value' => 'error', // configured to use something that was never supplied 'unreachable' => 'warn', // may be transient; the strike system is what escalates it 'unknown_target' => 'warn', // A hostname deliberately put behind authentication that authenticates nobody is the one // finding here that is worse the longer it goes unnoticed, and it is never transient — // it is a state of the configuration, not a passing failure. 'access_open' => 'error', // The sweep only files this after the outage has already outlasted a restart, so it is // past the point where the strike system would still be deciding. 'proxy_down' => 'error', default => 'warn', }; } // ── Gates ──────────────────────────────────────────────────────────────────────────────────── // Two switches, because detecting and repairing are separate things to trust. // // AI_REPAIR_ENABLED alone gives a system that reads logs, files findings and offers fixes, and // writes nothing. That is the state this should live in first — long enough to read what it // found and disagree with some of it. A subsystem that starts by editing conf has to be believed // before there is any evidence for believing it. // // AI_REPAIR_AUTOFIX_ENABLED is what lets a proven value be written without being asked, and it // is meaningless on its own: nothing to write if nothing is looking. Both must be true, in the // same layered way AI_ENABLED is necessary but never sufficient. function vv_ai_repair_enabled(): bool { if (!vv_ai_config()['enabled']) return false; return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_ENABLED'] ?? 'false'))) === 'true'; } function vv_ai_repair_autofix_enabled(): bool { if (!vv_ai_repair_enabled()) return false; return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_AUTOFIX_ENABLED'] ?? 'false'))) === 'true'; } // The first consumer AI_ASSIST_WATCHDOG has ever had. It shipped with the rest of the AI_ASSIST_ // tier flags and nothing read it, which meant a switch that looked like it did something. // // Separate from AI_REPAIR_ENABLED rather than folded into it because the two answer different // questions. Repair asks "may this read job logs and offer to change conf". This asks "may this // have an opinion about what the watchdogs are counting" — a different subject, a different // appetite for noise, and one you may well want off while repair stays on. function vv_ai_assist_watchdog_enabled(): bool { if (!vv_ai_repair_enabled()) return false; return strtolower(trim((string)(vv_conf_vars()['AI_ASSIST_WATCHDOG'] ?? 'false'))) === 'true'; } function vv_ai_findings_dir(): string { $d = AI_DATA_DIR . '/ai_findings'; if (!is_dir($d)) @mkdir($d, 0755, true); return $d; } function vv_ai_finding_retain_days(): int { $n = (int)(vv_conf_vars()['AI_FINDING_RETAIN_DAYS'] ?? 90); return max(1, $n); } // kind + subject + reference. Deliberately not the message: the same wrong port produces slightly // different log text as the software around it changes, and that must not mint a second record. // // The reference is the conf key for a conf-bound kind, and whatever else identifies the thing // otherwise — for an arr health item, the check that raised it. "Indexers unavailable: NzbNoob" // becomes "Indexers unavailable: NzbNoob, Miatrix" as more fail, and that is the same finding // getting worse rather than a second one. // Identity includes the host, because "Bazarr is on the skip list" is a different fact on each // machine that says it. Without the host in the hash, the second node to report the same subject // would land on the first node's record and overwrite it — and the operator would see one finding // where two machines have the same problem, or worse, one machine's dismissal silencing another's // live fault. // // $host defaults to this node, so every existing caller keeps working and local findings are // unchanged in meaning. It is a parameter rather than always-local because a collected finding // from a partner has to hash as that partner's, not as ours. function vv_ai_finding_id(string $kind, string $subject, string $ref, string $host = ''): string { $host = $host !== '' ? $host : vv_detect_host(); return substr(sha1(strtolower($host . '|' . $kind . '|' . $subject . '|' . $ref)), 0, 12); } // What the id was before the host joined the hash. Kept solely so a finding already on disk can be // found once — see the migration in vv_ai_finding_write(). Nine dismissed findings existed when // this changed, and a dismissal that silently expires is the one outcome this store must never // produce: "stop telling me about this" has to outlast a refactor. function vv_ai_finding_id_legacy(string $kind, string $subject, string $ref): string { return substr(sha1(strtolower($kind . '|' . $subject . '|' . $ref)), 0, 12); } function vv_ai_finding_path(string $id): ?string { if (!preg_match('/^[0-9a-f]{12}$/', $id)) return null; return vv_ai_findings_dir() . '/' . $id . '.json'; } function vv_ai_finding_get(string $id): ?array { $p = vv_ai_finding_path($id); if ($p === null || !is_file($p)) return null; $r = json_decode((string)@file_get_contents($p), true); return is_array($r) ? $r : null; } // Files a finding, or increments the one already describing this fault. // // $f expects: kind, subject, conf_key, conf_file, observed, evidence, source_log // and optionally: proposed, proven, state, note function vv_ai_finding_write(array $f): array { $kind = (string)($f['kind'] ?? ''); $subject = trim((string)($f['subject'] ?? '')); $confKey = trim((string)($f['conf_key'] ?? '')); $evidence = trim((string)($f['evidence'] ?? '')); // What identifies this finding. Conf-bound kinds are identified by their key; everything else // supplies its own reference, and a finding with neither points at nothing and is refused. $ref = trim((string)($f['ref'] ?? $confKey)); if (!isset(VV_AI_FINDING_KINDS[$kind])) return ['ok' => false, 'error' => 'unknown kind']; if ($subject === '' || $ref === '') return ['ok' => false, 'error' => 'subject and a reference are required']; if ($evidence === '') return ['ok' => false, 'error' => 'evidence required']; if (vv_ai_kind_is_conf_bound($kind)) { if ($confKey === '') return ['ok' => false, 'error' => 'conf_key required for this kind']; // The key has to be a real shell identifier for the same reason the conf writer insists // on it: a finding is a proposal to edit that key, and a malformed one cannot be actioned. if (!vv_conf_key_valid($confKey)) return ['ok' => false, 'error' => 'malformed conf key']; } $state = (string)($f['state'] ?? 'open'); if (!isset(VV_AI_FINDING_STATES[$state])) $state = 'open'; $now = time(); $id = vv_ai_finding_id($kind, $subject, $ref); $rec = [ 'id' => $id, // Which machine this is about. Written even on a single-host install, because the store // outlives the topology — a finding filed today is still on disk when the second node // arrives, and one without a host is a record nobody can place. 'host' => (string)($f['host'] ?? vv_detect_host()), 'kind' => $kind, 'subject' => mb_substr($subject, 0, 120), 'conf_key' => $confKey, 'conf_file' => (string)($f['conf_file'] ?? 'master.conf'), // Secrets never enter this store. A finding about a rejected API key is about the key // being wrong, and the wrong value is of no use to anyone reading the record later. 'observed' => vv_conf_key_is_secret($confKey) ? '' : mb_substr((string)($f['observed'] ?? ''), 0, 300), 'proposed' => isset($f['proposed']) && !vv_conf_key_is_secret($confKey) ? mb_substr((string)$f['proposed'], 0, 300) : null, 'proven' => (bool)($f['proven'] ?? false), 'state' => $state, 'evidence' => mb_substr(vv_ai_redact($evidence), 0, 1200), 'source_log' => mb_substr((string)($f['source_log'] ?? ''), 0, 200), 'note' => mb_substr((string)($f['note'] ?? ''), 0, 1000), // Recomputed on every sighting rather than stored once: a key that becomes a toggle, or // a toggle that is replaced by a real value, changes what this finding means. 'ref' => mb_substr($ref, 0, 120), // Every field the grader consults has to be handed to it. It was given a hand-picked // three, so a kind added later that grades on a fourth silently came out as a warning — // which is how a disk throwing I/O errors would have been filed at the same level as a // switched-off toggle, and notified as a warning rather than an alert. 'severity' => vv_ai_finding_severity(['kind' => $kind, 'conf_key' => $confKey, 'arr_type' => (string)($f['arr_type'] ?? ''), 'sys_level' => (string)($f['sys_level'] ?? '')]), 'host' => vv_detect_host(), 'first' => $now, 'last' => $now, 'seen' => 1, 'closed_at' => null, // What the key read when the operator acknowledged it. Null unless acked; the ack // expires the moment the live value stops matching this. 'ack_value' => null, // A stable identity for the kinds whose evidence changes on every pass. Empty for the // rest, which pin to a conf value or to the evidence itself. Stored rather than derived // so an ack given in the browser pins to exactly what the sweep will compare against. 'pin' => mb_substr((string)($f['pin'] ?? ''), 0, 120), // What this finding looked like when it was last announced. Carried across sightings // below — a stamp that reset every fifteen minutes would be a notification every fifteen // minutes, which is how an operator learns to ignore the channel. 'notified' => null, ]; $p = vv_ai_finding_path($id); if ($p === null) return ['ok' => false, 'error' => 'bad id']; // One-time rename from the pre-host id, done here rather than as a startup sweep because this // is the only moment it matters: a finding is being written, and the question is whether this // node has said it before. A migration that ran anywhere else would have to walk the whole // store to answer a question only the write path asks. // // Idempotent by construction — after the rename the legacy path no longer exists, and a record // already carrying the new id never looks. if (!is_file($p)) { $legacy = vv_ai_finding_path(vv_ai_finding_id_legacy($kind, $subject, $ref)); if ($legacy !== null && $legacy !== $p && is_file($legacy)) @rename($legacy, $p); } if (is_file($p)) { $old = json_decode((string)@file_get_contents($p), true); if (is_array($old)) { $rec['first'] = $old['first'] ?? $now; $rec['seen'] = (int)($old['seen'] ?? 0) + 1; // A dismissed finding stays dismissed however many times the log repeats it — // otherwise "this is fine, stop telling me" lasts exactly one cycle. A fixed one // reopens, because seeing the fault again after a repair means the repair did not // hold, which is the single most important thing this store can tell anyone. if (($old['state'] ?? '') === 'dismissed') { $rec['state'] = 'dismissed'; $rec['closed_at'] = $old['closed_at'] ?? null; } // An acknowledgement holds only while the thing acknowledged is still true. Compare // what it is pinned to now against what it read when the ack was given: unchanged // means stay quiet, changed means the note is stale and the finding comes back by // itself. $rec carries this sighting's evidence, so a fault that has changed shape // fails this comparison even when no conf key is involved. if (($old['state'] ?? '') === 'acknowledged') { $ackedAt = (string)($old['ack_value'] ?? ''); if ($ackedAt === vv_ai_finding_ack_pin($rec)) { $rec['state'] = 'acknowledged'; $rec['ack_value'] = $ackedAt; $rec['closed_at'] = $old['closed_at'] ?? null; } // Otherwise $rec keeps the state this sighting computed — it has reopened. } // Preserve an operator's note over a generated one. if ($rec['note'] === '' && !empty($old['note'])) $rec['note'] = $old['note']; // Carried, never recomputed. This record is rebuilt from scratch on every sighting, // so anything not copied forward here is reset — and a reset announcement stamp // means this finding is announced again on the next pass, and the one after that. $rec['notified'] = $old['notified'] ?? null; } } if (@file_put_contents($p, json_encode($rec, JSON_PRETTY_PRINT)) === false) { return ['ok' => false, 'error' => 'write failed']; } return ['ok' => true, 'id' => $id, 'seen' => $rec['seen'], 'state' => $rec['state']]; } // $states filters; empty means everything. Newest activity first, because a finding seen in the // last cycle matters more than one that has been sitting fixed for a month. function vv_ai_findings_list(array $states = ['open', 'needs_operator']): array { $out = []; foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) { $r = json_decode((string)@file_get_contents($file), true); if (!is_array($r)) continue; if ($states && !in_array($r['state'] ?? 'open', $states, true)) continue; $out[] = $r; } usort($out, fn($a, $b) => ($b['last'] ?? 0) <=> ($a['last'] ?? 0)); return $out; } function vv_ai_finding_set_state(string $id, string $state, string $note = ''): bool { if (!isset(VV_AI_FINDING_STATES[$state])) return false; $r = vv_ai_finding_get($id); if ($r === null) return false; $r['state'] = $state; $r['closed_at'] = in_array($state, ['open', 'needs_operator'], true) ? null : time(); if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000); // No attribution here on purpose. This setter is called by the sweep as well as by a person — // it is how a candidate becomes needs_operator — and stamping every transition would credit a // machine for deciding something it only classified. Attribution belongs to the answer, not to // the bookkeeping, so vv_ai_finding_apply_action() records it and this does not. $p = vv_ai_finding_path($id); return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false; } function vv_ai_finding_close(string $id, string $note = ''): bool { return vv_ai_finding_set_state($id, 'resolved', $note); } function vv_ai_finding_dismiss(string $id, string $note = ''): bool { return vv_ai_finding_set_state($id, 'dismissed', $note); } // What an acknowledgement is pinned to — the thing that has to stay the same for the ack to keep // meaning what it meant. // // For a conf-bound finding that is the key's value, which is what the operator was looking at // when they said "I know". A finding with no key had nothing to pin to and so compared '' with // '' — every ack on an arr health item was silently permanent, which is dismiss wearing ack's // label. Those pin to the shape of the fault instead: "indexers unavailable: NzbNoob" and // "indexers unavailable: NzbNoob, Miatrix" are one finding getting worse, and an ack given for // the first has not been given for the second. // A finding may supply its own pin, and the log-derived kinds must. Their evidence carries a line // count and the timestamp of the first matching line, so it is different on every single pass — // pinning to it would expire an acknowledgement within fifteen minutes and re-announce a fault // the operator had just said they knew about, forever. A PCIe controller that has thrown // correctable errors since the machine was built is exactly that case. // // The pin those kinds supply is their identity: the fault class and the thing it is about. A // fault that gets genuinely worse — correctable becoming uncorrectable — has a different class, // so it is a different finding and announces on its own rather than hiding behind this one's ack. function vv_ai_finding_ack_pin(array $f): string { if (($f['pin'] ?? '') !== '') return (string)$f['pin']; $key = (string)($f['conf_key'] ?? ''); if ($key !== '') return (string)(vv_conf_vars()[$key] ?? ''); return 'ev:' . substr(sha1((string)($f['evidence'] ?? '')), 0, 16); } // "I know about this — leave it, and tell me if it changes." // // Stamps what it is pinned to onto the record. Every later sighting compares against that stamp, // so the acknowledgement covers this state and not the finding forever. Acking that critical // rsync is off says nothing about critical rsync being on. function vv_ai_finding_ack(string $id, string $note = ''): bool { $r = vv_ai_finding_get($id); if ($r === null) return false; $r['state'] = 'acknowledged'; $r['ack_value'] = vv_ai_finding_ack_pin($r); $r['closed_at'] = time(); if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000); $p = vv_ai_finding_path($id); return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false; } // What the operator can do about a finding, and what each choice means. Returned rather than // hardcoded in the UI so the chat and the page cannot offer different options for the same row, // and enforced in vv_ai_finding_apply_action() so neither can act on one it was not offered. // // Fix appears for anything with a proposed value, toggle or not — the prohibition is on the // sweep choosing, never on the operator choosing. Open rows also carry ack, dismiss and cancel, // because "I know", "this is never a problem" and "not now" are all valid answers to being told // something, and they are three different answers. function vv_ai_finding_actions(array $f): array { // A closed finding has one question left, and it is not the original one: was closing it // right? Offering fix or ack on a row that is already dismissed is offering to decide // something that has been decided. Reopen is here because dismiss is otherwise permanent — // the write path keeps a dismissed finding dismissed however many times the fault recurs, // so a mis-click would need someone editing JSON on disk to undo. if (!in_array((string)($f['state'] ?? 'open'), ['open', 'needs_operator'], true)) { return ['reopen' => 'Put it back on the list — either closing it was wrong, or it is back']; } $actions = []; if (($f['proposed'] ?? null) !== null) { $actions['fix'] = vv_ai_conf_is_toggle((string)($f['conf_key'] ?? '')) ? 'Set ' . $f['conf_key'] . ' — a toggle, so this only ever happens because you asked' : 'Write the proven value to ' . $f['conf_key']; } // Offered only when a destination was recorded, which is only for a series the triage called // misfiled. "Uncertain" carries no move_to, so it gets the same buttons every other finding // has and no way to act on a judgement nobody made. if (($f['move_to'] ?? '') !== '' && (int)($f['move_id'] ?? 0) > 0) { $actions['move'] = 'Move ' . ($f['subject'] ?? 'it') . ' to ' . $f['move_to'] . ' — relocates the files on disk'; } $actions['ack'] = ($f['conf_key'] ?? '') !== '' ? 'Known and intended. Stays quiet until ' . $f['conf_key'] . ' changes' : 'Known and intended. Stays quiet until the fault itself changes'; $actions['dismiss'] = 'Not a problem, ever. Stays closed even when it is seen again'; $actions['cancel'] = 'Leave it alone for now'; return $actions; } // Closed findings are kept for a while because "this happened before and here is what fixed it" // is worth more than the disk it costs. Open ones are never pruned — an unresolved problem does // not stop mattering because it is old. function vv_ai_findings_prune(): int { $cutoff = time() - (vv_ai_finding_retain_days() * 86400); $n = 0; foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) { $r = json_decode((string)@file_get_contents($file), true); if (!is_array($r)) continue; if (in_array($r['state'] ?? 'open', ['open', 'needs_operator'], true)) continue; if ((int)($r['closed_at'] ?? 0) > $cutoff) continue; if (@unlink($file)) $n++; } return $n; } // ── The system's own log ───────────────────────────────────────────────────────────────────── // Everything above this reads Varaverk's logs and asks the arrs about themselves. Neither can // see a disk throwing I/O errors, a filesystem going read-only, or a PCIe link retraining every // two minutes — and those are the faults that explain the ones Varaverk does notice. // // /var/log/syslog and not dmesg. Unraid's syslog carries the kernel ring buffer's lines already, // prefixed with `kernel:`, and it carries a real date on every line. dmesg's ring buffer wraps, // has no persistence across a reboot, and its -T timestamps are derived from uptime rather than // recorded, which makes "since the last pass" unanswerable from it. // // What is deliberately NOT here: // Container restarts, crash loops and OOMKilled — docker_watchdog owns those and notifies on // them, and a second opinion arriving by a second channel is not more information. // ZFS pool health — `zpool status` answers that exactly, and inferring it from log lines when // the authoritative command is one call away would be guessing on purpose. // // A host-level OOM kill IS here, and does overlap docker_watchdog when the process killed was in // a container. They are different halves: the watchdog reports that a container restarted, this // reports what the kernel killed and that it was memory. The store folds repeats into one row. const VV_AI_SYSLOG_PATTERNS = [ // Verified against this host's own syslog, which produced 109 of these in a day. Correctable // means the link recovered, which is why it is a warning and not an error — but a device // retraining continuously is failing slowly. ['re' => '/AER: (?:Multiple )?Corrected error.*?(?:from|received from) (?[0-9a-f:.]+)/i', 'level' => 'warn', 'what' => 'PCIe correctable errors'], ['re' => '/AER: (?:Multiple )?Correctable error message received from (?[0-9a-f:.]+)/i', 'level' => 'warn', 'what' => 'PCIe correctable errors'], // The severity word is followed by a parenthetical often enough that requiring "error" to // come straight after it misses the common form: "Uncorrected (Non-Fatal) error received // from". Found by a fixture, not by reasoning about it. ['re' => '/AER: (?:Multiple )?(?:Uncorrected|Fatal|Uncorrectable)[^:]{0,20}? error.*?from (?[0-9a-f:.]+)/i', 'level' => 'error', 'what' => 'PCIe uncorrectable errors'], // Block layer. blk_update_request is where a failed read or write surfaces with the device // named, which is the line worth keeping — the ATA exception above it names a port, not a // disk, and a port number is not something an operator can act on. ['re' => '/blk_update_request: (?:critical )?(?:I\/O|medium|target|nonexistent) error, dev (?[a-z0-9]+)/i', 'level' => 'error', 'what' => 'block I/O errors'], ['re' => '/Buffer I\/O error on dev (?[a-z0-9]+)/i', 'level' => 'error', 'what' => 'buffer I/O errors'], // The SCSI layer writes the device in brackets — "sd 1:0:3:0: [sdo] tag#28 FAILED" — so the // name is not followed by a colon the way it is everywhere else. Both forms accepted. ['re' => '/(?:\[(?sd[a-z]+|nvme\d+n\d+)\]|(?sd[a-z]+|nvme\d+n\d+):).*?(?:unrecovered read error|medium error|rejecting I\/O to|failed command)/i', 'level' => 'error', 'what' => 'device errors'], // Filesystems. Each names the device inside its own parentheses, which is the identity the // finding is keyed on — one finding per filesystem, however many lines it emits. ['re' => '/XFS \((?[^)]+)\): (?:Metadata|Corruption|corruption|log I\/O error|writeback error|Internal error)/', 'level' => 'error', 'what' => 'XFS errors'], ['re' => '/BTRFS (?:error|critical) \(device (?[^)]+)\)/', 'level' => 'error', 'what' => 'BTRFS errors'], ['re' => '/EXT4-fs error \(device (?[^)]+)\)/', 'level' => 'error', 'what' => 'ext4 errors'], ['re' => '/(?:Remounting|remounting) filesystem read-only/', 'level' => 'error', 'what' => 'a filesystem went read-only', 'subject' => 'filesystem'], // Memory. The process name is the subject, so "the kernel keeps killing shfs" is one finding // rather than one per occurrence. ['re' => '/Out of memory: Killed process \d+ \((?[^)]+)\)/', 'level' => 'error', 'what' => 'out-of-memory kills'], // The kernel saying it has broken. No subject to extract that means anything, so the finding // is about the machine. ['re' => '/(?:kernel BUG at|general protection fault|Oops: |Kernel panic)/', 'level' => 'error', 'what' => 'kernel faults', 'subject' => 'kernel'], ]; function vv_ai_syslog_path(): string { return '/var/log/syslog'; } function vv_ai_syslog_enabled(): bool { if (!vv_ai_repair_enabled()) return false; return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_SYSLOG_ENABLED'] ?? 'true'))) !== 'false'; } function vv_ai_syslog_max_lines(): int { $n = (int)(vv_conf_vars()['AI_REPAIR_SYSLOG_MAX_LINES'] ?? 4000); return max(100, min(50000, $n)); } // "Aug 9 21:46:07" — syslog's format carries no year, which is a real problem exactly once a // year. Parsed against the current one, and anything landing more than a day in the future is // read as last year's: on 1 January, December's lines would otherwise be stamped eleven months // ahead and every one of them would look newer than the last sweep, forever. // // Returns null for a line that does not start with a timestamp, which is a continuation line and // belongs to whatever preceded it. function vv_ai_syslog_ts(string $line, ?int $now = null): ?int { if (!preg_match('/^([A-Z][a-z]{2}\s+\d{1,2} \d{2}:\d{2}:\d{2})/', $line, $m)) return null; $now = $now ?? time(); $ts = strtotime($m[1] . ' ' . date('Y', $now)); if ($ts === false) return null; if ($ts > $now + 86400) { $ts = strtotime($m[1] . ' ' . (int)(date('Y', $now) - 1)); if ($ts === false) return null; } return $ts; } // The tail of syslog, bounded twice: by the line cap and by the timestamp. The cap is applied // first and is what stops a rotation, a boot, or a flood from turning one pass into a scan of // the whole file. function vv_ai_syslog_lines(int $since, ?int $maxLines = null, ?string $path = null): array { $path = $path ?? vv_ai_syslog_path(); if (!is_readable($path)) return []; $max = $maxLines ?? vv_ai_syslog_max_lines(); $out = []; $rc = 0; exec('tail -n ' . (int)$max . ' ' . escapeshellarg($path) . ' 2>/dev/null', $out, $rc); if ($rc !== 0) return []; $now = time(); $kept = []; foreach ($out as $line) { $ts = vv_ai_syslog_ts($line, $now); // A line with no timestamp cannot be placed in time. Kept only if the line before it was // kept, since that is what a continuation is. if ($ts === null) { if ($kept) $kept[] = $line; continue; } if ($ts <= $since) continue; $kept[] = $line; } return $kept; } // Log lines in, finding candidates out. One candidate per (pattern, subject) however many lines // matched, with the count carried in the evidence — a disk that threw four hundred I/O errors in // a quarter of an hour is one fault, and four hundred findings would be four hundred ways to // miss it. function vv_ai_syslog_findings(int $since, ?array $lines = null): array { if (!vv_ai_syslog_enabled()) return []; $lines = $lines ?? vv_ai_syslog_lines($since); if (!$lines) return []; $agg = []; foreach ($lines as $line) { foreach (VV_AI_SYSLOG_PATTERNS as $p) { if (!preg_match($p['re'], $line, $m)) continue; // subject2 is the second branch of an alternation. PHP refuses two groups with the // same name in one pattern, so a pattern that can find its subject in either of two // shapes has to name them apart and try both here. $subject = trim((string)($m['subject'] ?? '')); if ($subject === '') $subject = trim((string)($m['subject2'] ?? '')); if ($subject === '') $subject = trim((string)($p['subject'] ?? '')); if ($subject === '') $subject = 'system'; $key = $p['what'] . '|' . $subject; if (!isset($agg[$key])) { $agg[$key] = ['what' => $p['what'], 'subject' => $subject, 'level' => $p['level'], 'count' => 0, 'first_line' => trim($line)]; } $agg[$key]['count']++; // The worst level wins when two patterns describe the same subject. if ($p['level'] === 'error') $agg[$key]['level'] = 'error'; break; // one pattern per line; the list is ordered most specific first } } $found = []; foreach ($agg as $a) { $found[] = [ 'kind' => 'system_fault', 'subject' => $a['subject'], // Identity, not contents — the evidence below counts lines and quotes a timestamp, // so it differs every pass and would expire an acknowledgement immediately. 'pin' => 'sys:' . $a['what'] . '|' . $a['subject'], // What identifies it: the class of fault, not the message. The wording of a kernel // line changes between releases and the fault does not. 'ref' => $a['what'], 'conf_key' => '', 'conf_file' => '', 'sys_level' => $a['level'], 'observed' => $a['count'] . ' since the last pass', 'evidence' => sprintf('%s on %s — %d line%s since the last pass. First: %s', $a['what'], $a['subject'], $a['count'], $a['count'] === 1 ? '' : 's', mb_substr($a['first_line'], 0, 200)), 'source_log' => vv_ai_syslog_path(), // Nothing here is repairable from conf, so it goes straight to the operator rather // than sitting open waiting for a probe that will never run. 'state' => 'needs_operator', ]; } return $found; } // ── What the containers are saying ─────────────────────────────────────────────────────────── // docker_watchdog watches container *state* — is it up, does its port answer, is it restarting. // None of that reads a line the application wrote, so a container that is running perfectly and // has been unable to write to its database for a day looks completely healthy from out there. // // The patterns are deliberately about the container's environment rather than its behaviour. // Fifty containers run here and they are fifty different applications; there is no useful shared // vocabulary for "this app is malfunctioning". There is an exact shared vocabulary for "the disk // is full", "the filesystem is read-only" and "my database is corrupt", because those come from // libc, the kernel and SQLite rather than from the application — the same string in every one of // them. Anything app-specific belongs in that app's own health endpoint, which is where the arr // checks already come from. // // Noise is the whole risk here. Every pattern below was run against the real logs of all fifty // containers on this host before being kept; see Tools/ai_container_check.sh. const VV_AI_CONTAINER_PATTERNS = [ ['re' => '/no space left on device/i', 'level' => 'error', 'what' => 'disk full'], ['re' => '/read-only file ?system|Read-only file system/i', 'level' => 'error', 'what' => 'read-only filesystem'], ['re' => '/disk quota exceeded/i', 'level' => 'error', 'what' => 'disk quota exceeded'], // SQLite's own wording. "database is locked" is deliberately absent: it is contention, it is // transient, and the arrs emit it in normal operation. ['re' => '/database disk image is malformed|database or disk is full|file is (?:not a database|encrypted or is not a database)/i', 'level' => 'error', 'what' => 'database corruption'], ['re' => '/too many open files/i', 'level' => 'error', 'what' => 'file descriptor limit'], ['re' => '/certificate has expired|certificate is not yet valid|certificate verify failed/i', 'level' => 'error', 'what' => 'certificate problems'], ]; function vv_ai_container_logs_enabled(): bool { if (!vv_ai_repair_enabled()) return false; return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_CONTAINER_LOGS_ENABLED'] ?? 'true'))) !== 'false'; } function vv_ai_container_log_lines(): int { $n = (int)(vv_conf_vars()['AI_REPAIR_CONTAINER_LOG_LINES'] ?? 400); return max(50, min(5000, $n)); } // Running containers only. A stopped one has nothing new to say, and its last words before it // stopped are docker_watchdog's business. function vv_ai_running_containers(): array { $out = []; $rc = 0; exec('docker ps --format {{.Names}} 2>/dev/null', $out, $rc); if ($rc !== 0) return []; return array_values(array_filter(array_map('trim', $out), fn($n) => $n !== '')); } // One container's log since the marker, bounded by both a time and a line count. --since alone // is not a bound: a container that logged a million lines in the last quarter hour would hand // back all of them. function vv_ai_container_log(string $name, int $since, ?int $maxLines = null): array { if (!preg_match('/^[A-Za-z0-9][A-Za-z0-9_.-]*$/', $name)) return []; $max = $maxLines ?? vv_ai_container_log_lines(); // Both streams: applications disagree about which one an error belongs on, and several of // these write everything to stdout. $cmd = 'docker logs --since ' . escapeshellarg((string)max(1, $since)) . ' --tail ' . (int)$max . ' ' . escapeshellarg($name) . ' 2>&1'; $out = []; $rc = 0; exec($cmd, $out, $rc); return $rc === 0 ? $out : []; } // Same aggregation as the syslog pass: one candidate per (container, fault class), with the // count in the evidence rather than a finding per line. function vv_ai_container_findings(int $since, ?array $logsByContainer = null): array { if ($logsByContainer === null && !vv_ai_container_logs_enabled()) return []; $agg = []; $names = $logsByContainer !== null ? array_keys($logsByContainer) : vv_ai_running_containers(); foreach ($names as $name) { $lines = $logsByContainer !== null ? $logsByContainer[$name] : vv_ai_container_log($name, $since); foreach ($lines as $line) { foreach (VV_AI_CONTAINER_PATTERNS as $p) { if (!preg_match($p['re'], $line)) continue; $key = $name . '|' . $p['what']; if (!isset($agg[$key])) { $agg[$key] = ['name' => $name, 'what' => $p['what'], 'level' => $p['level'], 'count' => 0, 'first_line' => trim($line)]; } $agg[$key]['count']++; break; } } } $found = []; foreach ($agg as $a) { $found[] = [ 'kind' => 'container_fault', 'subject' => $a['name'], 'pin' => 'ctr:' . $a['what'] . '|' . $a['name'], 'ref' => $a['what'], 'conf_key' => '', 'conf_file' => '', 'sys_level' => $a['level'], 'observed' => $a['count'] . ' since the last pass', 'evidence' => sprintf('%s in %s — %d line%s since the last pass. First: %s', $a['what'], $a['name'], $a['count'], $a['count'] === 1 ? '' : 's', mb_substr($a['first_line'], 0, 200)), 'source_log' => 'docker logs ' . $a['name'], 'state' => 'needs_operator', ]; } return $found; } // ── What the watchdogs have counted ────────────────────────────────────────────────────────── // The fourth candidate source. The other three read logs; this reads the counters the watchdogs // keep, and turns the ones that have gone far enough into a record with a name on it. // // It reads vv_wd_all() rather than the state files, for the reason the Watchdog tab does: the // meaning of a strike lives in include/watchdog.php, and a second parser here would be a second // opinion about it. It also gets the thresholds in the same payload, so a count is always judged // against the limit that was in force when it was read rather than one fetched separately. // // Deliberately not conf-bound. 'watchdog_strike' is absent from VV_AI_CONF_BOUND_KINDS, so these // findings carry no conf key, so vv_ai_finding_may_autofix() can never return true for one. That // is a structural guarantee rather than a switch: there is no setting that makes this able to // change anything, and none can be added without also deciding which key it would write. // // What it does NOT do is explain. The text below is derived, not generated — no model is asked // anything on this path. The reading is a click away in the troubleshoot profile, already scoped // to the orchestrator's log by the Why? buttons on the Watchdog tab. Detection has to be worth // believing before it is worth narrating, and a wrong cause stated fluently is worse than none. const VV_AI_WD_PRESSURE_LEVELS = [2 => 'medium', 3 => 'hard']; function vv_ai_watchdog_findings(?array $payload = null): array { if ($payload === null && !vv_ai_assist_watchdog_enabled()) return []; $wd = $payload ?? vv_cache_read('watchdog', 900); if ($wd === null) { require_once __DIR__ . '/watchdog.php'; $wd = vv_wd_all(); } $cfg = $wd['cfg'] ?? []; $found = []; foreach ($wd['nodes'] ?? [] as $node) { // This host only. A partner's counters are its own to file findings about, and the // operator reading them here could not act on them from this side anyway. if (empty($node['is_me'])) continue; $st = $node['states'] ?? null; if (!$st) continue; $id = (string)($node['id'] ?? 'this host'); // The operator has already said they know about these. Suppressing here rather than at // write time keeps them out of the pin store entirely — 7DaysToDie is the worked example: // knowingly broken, deliberately ignored, and a finding about it every fifteen minutes // would be exactly the crying-wolf this is supposed to avoid. $ignore = array_flip((array)($node['config']['ignore'] ?? [])); $add = function (string $subject, string $ref, string $level, string $observed, string $evidence) use (&$found) { $found[] = [ 'kind' => 'watchdog_strike', 'subject' => $subject, 'pin' => 'wd:' . $ref . '|' . $subject, 'ref' => $ref, 'conf_key' => '', 'conf_file' => '', 'sys_level' => $level, 'observed' => $observed, 'evidence' => $evidence, // Every watchdog logs through the orchestrator; none keeps a log of its own. 'source_log' => 'Orchestrators/watchdog_orchestrator', 'state' => 'needs_operator', ]; }; // 1. Skip list — the terminal state. The watchdog tried its restart limit and stopped, // which means nothing further is coming automatically. Always worth a record. foreach ((array)($st['skiplist'] ?? []) as $name) { if (isset($ignore[$name])) continue; $add($name, 'skiplist', 'error', 'on the skip list', sprintf('%s is on the docker watchdog skip list on %s. It was restarted up to ' . 'the limit of %d and did not stay up, so the watchdog has stopped trying ' . 'and will not restart it again without intervention.', $name, $id, (int)($cfg['restart_limit'] ?? 3))); } // 2. Container check strikes at or past their limit. Below the limit the watchdog is // still working the problem and a finding would be premature. $lim = max(1, (int)($cfg['cpu_fail_lim'] ?? 2)); foreach ((array)($st['ctr_strikes'] ?? []) as $key => $count) { $name = preg_replace('/_(cpu|http|api|docker)$/', '', (string)$key); if (isset($ignore[$name]) || (int)$count < $lim) continue; $check = preg_match('/_(cpu|http|api|docker)$/', (string)$key, $m) ? $m[1] : 'health'; $add($name, 'strike:' . $check, 'warn', $count . ' / ' . $lim, sprintf('The docker watchdog has counted %d consecutive %s failures for %s on %s, ' . 'against a limit of %d.', (int)$count, $check, $name, $id, $lim)); } // 3. Repeated restarts inside the 24h window. Distinct from a strike: the container is // coming back each time, which is why nothing has escalated, and is also why this is // easy to never notice. $rc = []; foreach ((array)($st['restarts'] ?? []) as $r) { $n = (string)($r['name'] ?? ''); if ($n !== '') $rc[$n] = ($rc[$n] ?? 0) + 1; } $rlim = max(2, (int)($cfg['restart_limit'] ?? 3)); foreach ($rc as $name => $count) { if (isset($ignore[$name]) || $count < $rlim) continue; $add($name, 'restarts', 'warn', $count . ' in 24h', sprintf('%s has been restarted %d times in the last 24 hours on %s. Each one ' . 'succeeded, so no strike escalated — but a container that needs restarting ' . 'this often is failing at something between restarts.', $name, $count, $id)); } // 4. Any unattended reboot. Not thresholded: the machine restarting itself is worth // saying once every time it happens, however normal the count is. $reboots = (array)($st['reboots'] ?? []); if ($reboots) { $add($id, 'reboots', 'error', count($reboots) . ' / ' . (int)($cfg['reboot_limit'] ?? 3), sprintf('The stability watchdog has rebooted %s %d time%s in the last %d hours, ' . 'against a limit of %d before it stops trying.', $id, count($reboots), count($reboots) === 1 ? '' : 's', (int)($cfg['reboot_window'] ?? 12), (int)($cfg['reboot_limit'] ?? 3))); } // 5. Sustained memory pressure. Level 1 is soft and self-clears constantly; from level 2 // the watchdog has acted on containers, and that is a thing that happened. $lvl = (int)($st['rw_level'] ?? 0); if (isset(VV_AI_WD_PRESSURE_LEVELS[$lvl])) { $paused = array_values(array_filter((array)($st['rw_paused'] ?? []))); $stopped = array_values(array_filter((array)($st['rw_stopped'] ?? []))); $acted = array_merge( $paused ? ['paused ' . implode(', ', $paused)] : [], $stopped ? ['stopped ' . implode(', ', $stopped)] : [] ); $add($id, 'pressure', $lvl >= 3 ? 'error' : 'warn', VV_AI_WD_PRESSURE_LEVELS[$lvl] . ' pressure', sprintf('The resource watchdog is at %s pressure on %s%s.%s', VV_AI_WD_PRESSURE_LEVELS[$lvl], $id, $acted ? ' and has ' . implode(' and ', $acted) : '', !empty($st['mem_shutdown']) ? ' The memory shutdown flag is set, so ' . 'docker_watchdog is deferring container restarts.' : '')); } } return $found; } // ── What the AI adds to which script ────────────────────────────────────────────────────────── // Declared here rather than in each script's header, and that is a deliberate departure from // api/scriptinfo.php's usual rule that the header next to the code is authoritative. It is // authoritative about what the script does. An enhancement is not the script's behaviour — it is // something else reading the script's output afterwards, implemented in PHP, gated by a flag the // script has never heard of. Writing it into the bash header would put a description of PHP inside // a file that cannot enforce it, and the two would drift the first time either changed. // // A script may appear here without the enhancement being about its output. sonarr discovery is // listed because the triage is its safety net: discovery takes the first accessible root folder // (playback_aware_sonarr_discovery.sh:750) with no regard for whether the show is anime, kids or // general, and the classification scan is what notices the next night. That relationship is real, // it explains where misfiled series come from, and until now it was written down nowhere. const VV_AI_SCRIPT_ENHANCEMENTS = [ 'Arrs_Stack/sonarr_classification_scan.sh' => [ 'name' => 'Classification triage', 'flag' => 'AI_ASSIST_DISCOVERY', 'what' => 'This scan reports reverse-anime leaks and deliberately does not act on them — ' . 'its own header calls them genuine judgement calls, because no metadata field ' . 'separates a misfile from a deliberate choice. The triage sorts that bucket into ' . 'misfiled, donghua, anime-adjacent or uncertain.', 'acts' => 'Files a finding for misfiled and uncertain only. A misfiled series carries a ' . 'Move action, which relocates the files — pressed by a person, never automatic.', ], 'Arrs_Stack/playback_aware_sonarr_discovery.sh' => [ 'name' => 'Classification triage (safety net)', 'flag' => 'AI_ASSIST_DISCOVERY', 'what' => 'Discovery adds every show it finds to the first accessible root folder, without ' . 'classifying it. Anything it places wrongly is caught by the classification scan ' . 'that night, and the triage is what turns that scan\'s count into named titles.', 'acts' => 'Nothing here. The enhancement runs against the classification scan\'s output, ' . 'not against this script.', ], 'Orchestrators/watchdog_orchestrator.sh' => [ 'name' => 'Watchdog findings', 'flag' => 'AI_ASSIST_WATCHDOG', 'what' => 'Turns the counters the watchdogs keep — skip lists, strikes past their limit, ' . 'repeated restarts, unattended reboots, sustained pressure — into named findings ' . 'instead of numbers on a card.', 'acts' => 'Files findings only. The kind is not conf-bound, so it cannot autofix by ' . 'construction rather than by a setting.', ], ]; function vv_ai_script_enhancements(string $id = ''): array { if ($id === '') return VV_AI_SCRIPT_ENHANCEMENTS; $id = ltrim(trim($id), '/'); if (!str_ends_with($id, '.sh')) $id .= '.sh'; $e = VV_AI_SCRIPT_ENHANCEMENTS[$id] ?? null; if (!$e) return []; // The flag state travels with it. An enhancement listed against a script without saying // whether it is switched on is the same trap AI_ASSIST_WATCHDOG was for months: a name that // looks like a feature and does nothing. $e['enabled'] = strtolower(trim((string)(vv_conf_vars()[$e['flag']] ?? 'false'))) === 'true' && vv_ai_repair_enabled(); return $e; } // ── Classification triage ───────────────────────────────────────────────────────────────────── // The one place in the media stack where a model beats the rule it is helping. // // sonarr_classification_scan.sh decides anime/kids/regular from metadata — genre, certification, // network, original language — and that works: forward misses are zero. What it cannot decide is // the reverse direction, which its own header calls "genuine judgment calls". A series sitting in // the anime root with no anime signal is either misfiled or a deliberate choice, and no metadata // field distinguishes those. So it reports a count, every night, and a count is not actionable: // seventeen of them concealed two live-action crime dramas filed under anime for as long as // nobody read the list. // // This asks only about that bucket — eleven titles, not the eleven hundred the scan handles — and // only to sort them. It moves nothing. --move stays a flag a human types, because relocating media // is the one action here that is tedious to undo and impossible to notice going wrong. // // Reads the scan's own verdict from the review file rather than re-deriving it. A second opinion // computed from the same metadata would be the same answer with extra steps, and would drift from // the script the first time either changed. const VV_AI_TRIAGE_BUCKETS = ['misfiled', 'donghua', 'anime_adjacent', 'uncertain']; function vv_ai_assist_discovery_enabled(): bool { if (!vv_ai_repair_enabled()) return false; return strtolower(trim((string)(vv_conf_vars()['AI_ASSIST_DISCOVERY'] ?? 'false'))) === 'true'; } function vv_ai_classification_review(): array { $raw = @file_get_contents(STATE_DIR . '/arr_classification_review.json'); if ($raw === false) return []; $d = json_decode($raw, true); return is_array($d) ? $d : []; } // One call for the whole list, not one per title. Eleven separate requests would spend eleven // model loads on a question that fits in a paragraph, and the model reads the set better than the // items — "these six are all Chinese streaming platforms" is a judgement about the group. function vv_ai_triage_classification(array $items, ?callable $ask = null): array { if (!$items) return []; $lines = []; foreach ($items as $i => $x) { $lines[] = sprintf('%d. %s — network: %s, certification: %s', $i + 1, $x['title'] ?? '?', $x['network'] ?: 'unknown', $x['cert'] ?: 'unknown'); } $prompt = "These television series are filed in a library folder reserved for anime. A metadata " . "rule could not confirm any of them as anime, so each is either misfiled or a deliberate " . "choice by the library's owner.\n\n" . "Sort every one into exactly one bucket:\n" . " misfiled — not animation at all, or animation with no plausible claim to the " . "anime shelf. Live action belongs here.\n" . " donghua — Chinese or Korean animation.\n" . " anime_adjacent — Japanese-produced, or Western animation made in an anime style, where " . "shelving it as anime is defensible.\n" . " uncertain — you do not recognise it well enough to say.\n\n" . "Answer with one line per series, exactly: NUMBER|BUCKET|a short reason.\n" . "Use uncertain rather than guessing. Add nothing else.\n\n" . implode("\n", $lines); $reply = $ask ? $ask($prompt) : vv_ai_ask_model($prompt); if ($reply === null) return []; $out = []; foreach (explode("\n", $reply) as $line) { if (!preg_match('/^\s*(\d+)\s*\|\s*([a-z_]+)\s*\|\s*(.+)$/i', trim($line), $m)) continue; $idx = (int)$m[1] - 1; $b = strtolower(trim($m[2])); if (!isset($items[$idx]) || !in_array($b, VV_AI_TRIAGE_BUCKETS, true)) continue; $out[] = $items[$idx] + ['bucket' => $b, 'why' => mb_substr(trim($m[3]), 0, 160)]; } return $out; } // Deliberately not streaming and deliberately not the chat worker's path: this is one bounded // question with no conversation, no history and no capability grants, asked from a sweep that must // exit 0 whatever happens. A failure returns null and the caller files nothing. function vv_ai_ask_model(string $prompt, int $timeout = 120): ?string { $cfg = vv_ai_config(); if (empty($cfg['enabled']) || empty($cfg['url'])) return null; $ctx = stream_context_create(['http' => [ 'method' => 'POST', 'header' => "Content-Type: application/json\r\n", 'content' => json_encode([ 'model' => $cfg['model'], 'messages' => [['role' => 'user', 'content' => $prompt]], 'stream' => false, 'think' => false, 'options' => ['num_ctx' => 8192, 'temperature' => 0], ]), 'timeout' => $timeout, 'ignore_errors' => true, ]]); $raw = @file_get_contents(rtrim($cfg['url'], '/') . '/api/chat', false, $ctx); // Unpin on failure so the next resolution steps to another node that has a model. A refusal // or a bad reply is not a failure of the endpoint — only not reaching it is, which is why // this sits on the transport result and not on the parse below. if ($raw === false) { vv_ai_model_failed($cfg['model_host'] ?? ''); return null; } $d = json_decode($raw, true); return $d['message']['content'] ?? null; } // Findings, one per title that is not defensible where it sits. donghua and anime_adjacent are // answers, not problems — filing them would recreate the undifferentiated list this exists to // break up. uncertain is filed, because "the model could not tell either" is worth knowing and is // the honest outcome for an obscure title. function vv_ai_classification_findings(): array { if (!vv_ai_assist_discovery_enabled()) return []; $vars = vv_conf_vars(); $myId = strtoupper(vv_detect_host()); $rev = vv_ai_classification_review(); $items = $rev['reverse_anime'] ?? []; if (!$items) return []; $found = []; foreach (vv_ai_triage_classification($items) as $r) { if (!in_array($r['bucket'], ['misfiled', 'uncertain'], true)) continue; $title = (string)($r['title'] ?? '?'); // Only a misfiled series gets a destination. "Uncertain" means nobody knows where it // belongs, and offering a move for it would turn a shrug into a button. $target = $r['bucket'] === 'misfiled' ? (string)($vars[$myId . '_SONARR_GENERAL_ROOT'] ?? '') : ''; $found[] = [ 'kind' => 'media_misfiled', 'subject' => $title, 'pin' => 'cls:' . $r['bucket'] . '|' . $title, 'ref' => 'classification', 'conf_key' => '', 'conf_file' => '', 'sys_level' => $r['bucket'] === 'misfiled' ? 'warn' : 'info', 'observed' => $r['bucket'], 'evidence' => sprintf('%s sits in %s. The metadata rule found no anime signal, and ' . 'the triage calls it %s — %s. Network %s, certification %s.%s', $title, $r['root'] ?? 'the anime root', $r['bucket'], $r['why'] ?? 'no reason given', $r['network'] ?: 'unknown', $r['cert'] ?: 'unknown', $target !== '' ? ' Moving it would relocate the files to ' . $target . '.' : ' Nothing has been moved.'), // What Accept would do. Carried on the finding rather than recomputed at press time, // so the button acts on the judgement that was shown rather than on a fresh one. 'move_arr' => 'sonarr', 'move_id' => (int)($r['id'] ?? 0), 'move_from' => (string)($r['root'] ?? ''), 'move_to' => $target, 'source_log' => 'Arrs_Stack/sonarr_classification_scan', 'state' => 'needs_operator', ]; } return $found; } // Relocates a series to another root folder, files and all, through Sonarr's own API. // // The one genuinely destructive thing this file can do, so it is deliberately awkward to reach: // only from the 'move' action, which is only offered on a finding that recorded a destination, // which only happens for a series the triage called misfiled. Three narrowings, each of which has // to hold before a file is touched. // // moveFiles=true is the whole point — changing rootFolderPath without it leaves Sonarr pointing at // a path where nothing lives, which is worse than the misfiling it was correcting. // // Re-reads the series from Sonarr rather than trusting the finding's copy. The finding may be // hours old and the operator may have moved it by hand in the meantime; PUTting a stale record // back would undo that silently. function vv_ai_move_series(array $f): array { $id = (int)($f['move_id'] ?? 0); $to = trim((string)($f['move_to'] ?? '')); if ($id <= 0 || $to === '') return ['ok' => false, 'error' => 'no destination recorded']; require_once __DIR__ . '/arrs.php'; $arr = null; foreach (vv_discover_arrs() as $node) { foreach ($node['arrs'] as $a) if ($a['type'] === 'sonarr') { $arr = $a; break 2; } } if (!$arr) return ['ok' => false, 'error' => 'no sonarr configured']; $cur = vv_arr_http($arr['url'], $arr['key'], '/api/v3/series/' . $id, 10); if (!is_array($cur) || empty($cur['id'])) return ['ok' => false, 'error' => 'series not found']; // Already where it should be — a no-op is success, not a failure, and saying so stops the // finding being reopened forever by a fault that is already fixed. if (($cur['rootFolderPath'] ?? '') === $to) { return ['ok' => true, 'note' => 'already there']; } $cur['rootFolderPath'] = $to; $ctx = stream_context_create(['http' => [ 'method' => 'PUT', 'header' => "X-Api-Key: {$arr['key']}\r\nContent-Type: application/json\r\n", 'content' => json_encode($cur), 'timeout' => 30, 'ignore_errors' => true, ]]); $res = @file_get_contents(rtrim($arr['url'], '/') . '/api/v3/series/' . $id . '?moveFiles=true', false, $ctx); if ($res === false) return ['ok' => false, 'error' => 'sonarr did not answer']; $d = json_decode($res, true); if (!is_array($d) || ($d['rootFolderPath'] ?? '') !== $to) { return ['ok' => false, 'error' => 'sonarr did not accept the move']; } vv_ai_audit_move($f, $to); return ['ok' => true]; } // A move leaves no trace in conf_changes.log because it is not a conf change, and the arr's own // history records it as an edit without saying who asked. Written here so "why did this series // move" has an answer that names the finding. function vv_ai_audit_move(array $f, string $to): void { @file_put_contents(LOG_DIR . '/conf_changes.log', sprintf("%s finding=%s move subject=%s from=%s to=%s ip=%s\n", date('Y-m-d H:i:s'), $f['id'] ?? '?', $f['subject'] ?? '?', $f['move_from'] ?? '?', $to, $_SERVER['REMOTE_ADDR'] ?? 'cli'), FILE_APPEND | LOCK_EX); } // ── Reaching the operator ──────────────────────────────────────────────────────────────────── // A finding nobody is told about is a finding nobody has. The card on the AI tab shows them, but // only to someone who opens the tab, and the point of this subsystem is that it works while // nobody is looking. // // Only needs_operator is announced. An open finding may still be repaired by the next pass, and // announcing one would be telling the operator about a problem that has already been handled by // the time they read it. acknowledged and dismissed are the operator's own answers and are never // announced at all. // // AI_REPAIR_NOTIFY_ENABLED defaults to true when absent, unlike the two switches above it. Those // gate reading and writing, which are things to be trusted first; this gates telling someone, // which is the point of having found anything. It still cannot fire unless AI_REPAIR_ENABLED is // on, and notify() itself is subject to NOTIFY_UNRAID and the host's webhook. function vv_ai_notify_enabled(): bool { if (!vv_ai_repair_enabled()) return false; return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_NOTIFY_ENABLED'] ?? 'true'))) !== 'false'; } // What has to change before this finding is worth mentioning twice. // // The same pin as an acknowledgement, plus the severity. That means a finding is announced once // and then stays quiet — through every fifteen-minute pass, however many times it is seen — until // either the thing it is about changes or it gets worse. "Indexers unavailable: NzbNoob" becoming // "NzbNoob, Miatrix" is news; the same sentence for the ninth time is not. function vv_ai_finding_notify_pin(array $f): string { return vv_ai_finding_ack_pin($f) . '|' . (string)($f['severity'] ?? 'warn'); } // One notification for everything that needs saying, not one per finding. Returns what it did so // the sweep can log it and the tests can read it without a notification having to be sent. function vv_ai_findings_announce(bool $dryRun = false): array { $out = ['sent' => false, 'count' => 0, 'subject' => '', 'message' => '', 'ids' => []]; if (!vv_ai_notify_enabled()) return $out + ['skipped' => 'AI_REPAIR_NOTIFY_ENABLED is false']; // Nothing switched on to receive it. Checked before anything is composed, and reported as a // configuration state rather than a delivery failure: a failure is retried on the next pass // and logged loudly, and neither is the right response to "no channel has been set up". if (!vv_notify_available()) { return $out + ['skipped' => 'no notification channel — NOTIFY_UNRAID is false and no webhook is set']; } $due = []; foreach (vv_ai_findings_list(['needs_operator']) as $f) { if (($f['notified'] ?? null) === vv_ai_finding_notify_pin($f)) continue; $due[] = $f; } if (!$due) return $out; // Worst first: if the message is truncated, the part that survives is the part that matters. usort($due, fn($a, $b) => (($b['severity'] ?? '') === 'error' ? 1 : 0) <=> (($a['severity'] ?? '') === 'error' ? 1 : 0)); $lines = []; foreach (array_slice($due, 0, 3) as $f) { // Evidence is a log excerpt and routinely spans lines. Collapsed here rather than left // for vv_notify() to tidy on the way out: this string is also what gets logged and what // the sweep summary returns, and only one of those three readers strips newlines. // Collapsed before truncating, so 90 characters means 90 visible ones. $ev = trim(preg_replace('/\s+/u', ' ', (string)($f['evidence'] ?? ''))); $lines[] = sprintf('%s - %s: %s', $f['subject'] ?? '?', $f['ref'] ?? '?', mb_substr($ev, 0, 90)); } if (count($due) > 3) $lines[] = sprintf('and %d more', count($due) - 3); $n = count($due); $anyErr = false; foreach ($due as $f) if (($f['severity'] ?? '') === 'error') $anyErr = true; $out['count'] = $n; $out['subject'] = sprintf('Varaverk repair - %d finding%s need%s you', $n, $n === 1 ? '' : 's', $n === 1 ? 's' : ''); // ' · ' rather than newlines: notify() builds its Discord payload by printf-ing into a JSON // string literal, and a raw newline there produces a body the webhook rejects. $out['message'] = implode(' | ', $lines); $out['ids'] = array_column($due, 'id'); if ($dryRun) return $out; $out['sent'] = vv_notify($out['message'], $out['subject'], $anyErr ? 'alert' : 'warning'); // Stamped only on a delivery that worked. A channel that is down should retry on the next // pass rather than mark these as told and go quiet about them forever. if ($out['sent']) { foreach ($due as $f) { $rec = vv_ai_finding_get($f['id']); if ($rec === null) continue; $rec['notified'] = vv_ai_finding_notify_pin($rec); $p = vv_ai_finding_path($f['id']); if ($p !== null) @file_put_contents($p, json_encode($rec, JSON_PRETTY_PRINT)); } } return $out; } // What the assistant should raise when a page loads: things that need the operator, newest // first. Repaired findings are deliberately not here — a fix that worked is a log entry, not a // conversation, and opening every session with a list of things that already went right is how // an operator learns to close the panel without reading it. function vv_ai_findings_for_chat(int $limit = 3): array { return array_slice(vv_ai_findings_list(['needs_operator']), 0, max(1, $limit)); } // ── The sweep ──────────────────────────────────────────────────────────────────────────────── // There is no post-run hook in Varaverk — nothing fires when a job finishes. Rather than add a // call to forty scripts, this picks up run records that completed since the last pass. One entry // in an orchestrator's list instead of forty edits, and it batches naturally. // // Runs that reported ok are read too. A container failing its HTTP check warns and leaves the // watchdog exiting 0, so "only look at failures" would miss the whole class of fault this exists // for: the job worked, and told you something is wrong. function vv_ai_sweep_marker_path(): string { return STATE_DIR . '/ai_repair_sweep.db'; } function vv_ai_sweep_last(): int { return (int)trim((string)@file_get_contents(vv_ai_sweep_marker_path())); } function vv_ai_sweep_mark(int $ts): void { if (!is_dir(STATE_DIR)) @mkdir(STATE_DIR, 0755, true); @file_put_contents(vv_ai_sweep_marker_path(), (string)$ts, LOCK_EX); } // Run records that finished after $since. A record still marked running is skipped rather than // read half-written — it will be picked up on the pass after it finishes. function vv_ai_recent_runs(int $since): array { $out = []; $base = realpath(LOG_DIR); if ($base === false) return []; foreach ((array)@glob($base . '/{,*/,*/*/,*/*/*/}*.json', GLOB_BRACE) as $path) { $r = json_decode((string)@file_get_contents($path), true); if (!is_array($r) || empty($r['id']) || ($r['status'] ?? '') === 'running') continue; $end = (int)($r['end'] ?? 0); if ($end <= $since) continue; $log = preg_replace('/\.json$/', '.log', $path); if (!is_file($log)) continue; $out[] = ['id' => (string)$r['id'], 'status' => (string)($r['status'] ?? '?'), 'start' => (int)($r['start'] ?? 0), 'end' => $end, 'log' => $log]; } usort($out, fn($a, $b) => $a['end'] <=> $b['end']); return $out; } // The lines one run wrote, and only those. Logs are appended across runs, so a tail alone would // re-read the previous run's output and re-report faults that have already been dealt with. // Filtering on the leading timestamp scopes the evidence to the run being examined. function vv_ai_run_log_lines(string $logPath, int $startTs, int $maxLines = 2000): array { $out = []; $rc = 0; @exec('tail -n ' . (int)$maxLines . ' ' . escapeshellarg($logPath) . ' 2>/dev/null', $out, $rc); if ($rc !== 0) return []; $kept = []; foreach ($out as $line) { if (preg_match('/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/', $line, $m)) { // A line older than the run belongs to a previous one. Two seconds of slack because // the record's start is stamped by the runner, not by the first line the job writes. if (strtotime($m[1]) < $startTs - 2) continue; } $kept[] = $line; } return $kept; } // One pass. Returns a summary rather than logging it, so the caller decides what to record and // the whole thing stays testable without a log to read afterwards. // // $dryRun does everything except write conf and move the marker — including probing, which is // the point: it answers "what would this have done" with real evidence rather than a guess. function vv_ai_repair_sweep(bool $dryRun = false): array { if (!vv_ai_repair_enabled()) { return ['ok' => false, 'error' => 'AI_REPAIR_ENABLED is not true', 'runs' => 0]; } $started = time(); $since = vv_ai_sweep_last(); $runs = vv_ai_recent_runs($since); $sum = ['ok' => true, 'runs' => count($runs), 'findings' => 0, 'fixed' => 0, 'needs_operator' => 0, 'resolved' => 0, 'quiet' => 0, 'details' => []]; // Candidates from four sources. Log triage is bounded to runs that finished since the last // pass; the arrs are asked every time, because their health is a current state rather than // something that appeared in a log once. Asking costs three local HTTP calls. // // The system log is bounded the same way the run logs are — everything since the marker — // so a fault that has been shouting for an hour is counted once per pass rather than once // per line, and stops producing candidates the moment it stops being logged. // // The watchdog counters are asked every time for the same reason the arrs are: a strike count // is a current state, not something that appeared in a log once. It is also the cheapest of // the four — one cached file read — and gated separately, on AI_ASSIST_WATCHDOG. $candidates = vv_ai_arr_health_findings(); foreach (vv_ai_syslog_findings($since) as $c) $candidates[] = $c; foreach (vv_ai_container_findings($since) as $c) $candidates[] = $c; foreach (vv_ai_watchdog_findings() as $c) $candidates[] = $c; // The only source here that asks the model anything. Gated separately on AI_ASSIST_DISCOVERY, // and it does nothing at all unless the classification scan has left a review file — so on a // host that never runs that scan this costs one failed file read. foreach (vv_ai_classification_findings() as $c) $candidates[] = $c; foreach ($runs as $run) { $lines = vv_ai_run_log_lines($run['log'], $run['start']); if (!$lines) continue; $rel = ltrim(str_replace(realpath(LOG_DIR), '', $run['log']), '/'); foreach (vv_ai_triage_log($lines, $rel) as $c) $candidates[] = $c; } foreach ($candidates as $cand) { $cand = vv_ai_probe_finding($cand); $sum['findings']++; // Write first, so a finding exists even if the repair below fails. A repair that // errored without leaving a record is the one failure mode there is no way back from. $w = vv_ai_finding_write($cand); if (!($w['ok'] ?? false)) continue; $id = $w['id']; // Already acknowledged or dismissed — the operator has spoken, and re-fixing behind // them would be the opposite of what an acknowledgement means. if (in_array($w['state'] ?? '', ['acknowledged', 'dismissed'], true)) { $sum['quiet']++; continue; } if (($cand['state'] ?? '') === 'resolved') { vv_ai_finding_close($id, (string)($cand['note'] ?? '')); $sum['resolved']++; continue; } if (vv_ai_finding_may_autofix($cand)) { if ($dryRun) { $sum['details'][] = "would fix {$cand['conf_key']} → {$cand['proposed']}"; continue; } $r = vv_ai_finding_apply_action($id, 'fix', 'Probed and written by the repair sweep.'); if ($r['ok'] ?? false) { $sum['fixed']++; $sum['details'][] = "fixed {$cand['conf_key']}"; } else { $sum['needs_operator']++; vv_ai_finding_set_state($id, 'needs_operator', (string)($r['error'] ?? '')); } continue; } if (($cand['state'] ?? '') === 'needs_operator') $sum['needs_operator']++; } // Marked only on a completed pass, and to when the pass began — a job that finished while // this was running is then picked up next time instead of being skipped for having ended // before a marker written at the end. if (!$dryRun) vv_ai_sweep_mark($started); // Announced after the marker rather than before, and over the whole store rather than only // what this pass touched. A finding that reached needs_operator two passes ago and was never // successfully delivered is still owed to the operator, and the pin is what stops that from // meaning it is announced twice. $sum['announced'] = vv_ai_findings_announce($dryRun); return $sum; } // ── Answering a finding in words ───────────────────────────────────────────────────────────── // The buttons are unambiguous by construction. This is for the other path — replying "yeah go // ahead" in the chat that raised the finding — and it is matched here rather than asked of the // model, because the model's answer would be a conf write and a wrong reading of "no, leave it" // is not recoverable by apologising. // // Same shape as vv_ai_route_from_chat(): anchored patterns, most specific first, and anything // unrecognised returns null so the assistant asks again instead of guessing. Two actions both // matching is also null — "leave it, I know" and "leave it for now" differ by one clause and // mean different things, so a phrase that supports both is not an instruction yet. const VV_AI_ACTION_PATTERNS = [ // Acknowledge — "this is deliberate, stop telling me". 'ack' => [ '/\b(i|we) know\b/u', '/\b(that|this|it)(?:\'s| is) (fine|expected|intentional|deliberate|on purpose)\b/u', '/\bon purpose\b/u', '/\b(aware|acknowledge|ack)\b/u', '/\bmeant to be\b/u', ], // Apply the proposed value. 'fix' => [ '/\bfix (it|that|this|them)?\b/u', '/\b(go ahead|do it|apply|make the change|change it|update it|correct it)\b/u', // A bare affirmative, as the whole message — "yes" answering "shall I fix it" is an // instruction, "yes it looks wrong" is agreement about the diagnosis and nothing more. // A trailing please is still bare. '/\b(yes|yeah|yep|yup|sure|ok|okay)\b(\s*,?\s*please)?[\s,.!]*$/u', '/\bplease do\b/u', ], // Not now — no state written, it comes back next sweep. 'cancel' => [ // "leave it" is matched bare, not only as "leave it alone" / "leave it for now". The // ambiguity this function is built to refuse — "leave it, I know" reading as both cancel // and ack — did not actually arise with the longer forms, so that sentence resolved to // ack and silenced the finding until the fault changed. The looser pattern is what makes // the two readings collide and sends it back to be restated. '/\b(not now|later|leave it\b|skip( it)?|cancel|ignore for now)\b/u', '/\b(no|nope|nah)\b[\s,.!]*$/u', '/\b(don\'?t|do not) (fix|touch|change|write|apply)\b/u', ], // Never a problem. Deliberately narrow: this is the one answer that cannot expire on its // own, so it is only read from a sentence that says so outright. Anything vaguer than these // is meant to land on ack, which comes back by itself when the fault changes. 'dismiss' => [ '/\bdismiss\b/u', '/\b(this|that|it)(?:\'s| is) not (a |an )?(problem|bug|issue|real)\b/u', '/\bnever (a problem|an issue|report this)\b/u', ], // Undo a close. 'reopen' => [ '/\breopen\b/u', '/\bun-?dismiss\b/u', ], ]; // Returns one of the keys in VV_AI_ACTION_PATTERNS, or null when the reply does not clearly mean // exactly one of them. // // Only call this when a finding is actually pending. A bare "yes" means fix in answer to "shall // I fix it" and means nothing at all on its own, and the difference is context this function // cannot see. function vv_ai_finding_action_from_text(string $text): ?string { $t = strtolower(trim($text)); if ($t === '') return null; $t = preg_replace('/\s+/', ' ', $t); $matched = []; foreach (VV_AI_ACTION_PATTERNS as $action => $patterns) { foreach ($patterns as $re) { if (preg_match($re, $t)) { $matched[$action] = true; break; } } } // Exactly one reading, or none. "leave it, I know" hits both ack and cancel; that is a // sentence the operator should be asked to restate, not one to pick a winner from. return count($matched) === 1 ? array_key_first($matched) : null; } // Carry out an answered action against a stored finding. // // Fix goes through the same guarded write path as everything else, and is the one place a // toggle may be written — because reaching here means the operator asked for it by name. The // unattended sweep never calls this. // Records which node answered a finding and how. Separate from the action itself so every route // through apply_action() is stamped the same way — a switch arm that forgot would be a finding // with an outcome and no author, which is exactly the record the mesh needs and the hardest kind // of gap to notice afterwards. function vv_ai_finding_stamp_actor(string $id, string $action): bool { $r = vv_ai_finding_get($id); if ($r === null) return false; $r['acted_by'] = vv_detect_host(); $r['acted_at'] = time(); $r['acted'] = $action; $p = vv_ai_finding_path($id); return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false; } function vv_ai_finding_apply_action(string $id, string $action, string $note = ''): array { $f = vv_ai_finding_get($id); if ($f === null) return ['ok' => false, 'error' => 'no such finding']; // Only what this finding actually offers, in the state it is actually in. The page renders // its buttons from the same function, but a stale tab holds buttons the store has moved past // — a row acked in one window is still showing Fix in another — and the endpoint is reachable // without either. Checking here is what makes vv_ai_finding_actions() the authority rather // than a suggestion. if (!isset(vv_ai_finding_actions($f)[$action])) { return ['ok' => false, 'error' => 'not offered for this finding: ' . $action]; } // Stamped before the action runs, not after, so a move that half-succeeds still records who // asked for it. 'cancel' is excluded because it is the button for changing your mind, and // recording it would make "nobody did anything" look like a decision. // // Deliberately the node, not a person: this interface authenticates as root and has no user // to name. In a mesh the useful question is which machine answered — one node dismissing what // another raised is the case worth being able to reconstruct. if ($action !== 'cancel') { vv_ai_finding_stamp_actor($id, $action); $f = vv_ai_finding_get($id) ?? $f; } switch ($action) { case 'ack': return ['ok' => vv_ai_finding_ack($id, $note), 'action' => 'ack']; case 'dismiss': return ['ok' => vv_ai_finding_dismiss($id, $note), 'action' => 'dismiss']; // Back to open, never straight back to needs_operator: whether it still cannot be // repaired here is the next sweep's finding to make, not a state to restore. // The only action in this file that changes something outside conf. It relocates files on // disk through Sonarr, which is why it is a button and not something a sweep decides: // vv_ai_move_series() is reached from here and from nowhere else. case 'move': $r = vv_ai_move_series($f); if (!($r['ok'] ?? false)) return $r + ['action' => 'move']; // Closed rather than acked. Acked means "known and intended, stay quiet"; this one // was acted on, and the record should say the fault is gone rather than tolerated. vv_ai_finding_close($id, 'Moved to ' . ($f['move_to'] ?? '?') . '. ' . $note); return ['ok' => true, 'action' => 'move', 'moved_to' => $f['move_to'] ?? '']; case 'reopen': return ['ok' => vv_ai_finding_set_state($id, 'open', $note), 'action' => 'reopen']; case 'cancel': // Deliberately writes nothing at all. "Not now" is not a state, it is the absence of // one — recording it would make the finding look decided when it is still open. return ['ok' => true, 'action' => 'cancel']; case 'fix': $proposed = $f['proposed'] ?? null; if ($proposed === null) return ['ok' => false, 'error' => 'nothing proposed to write']; $key = (string)$f['conf_key']; $ok = vv_conf_write_changes([[ 'file' => (string)($f['conf_file'] ?? 'master.conf'), 'key' => $key, 'value' => (string)$proposed, 'type' => 'scalar', ]]); $wrote = !in_array(false, $ok, true); if ($wrote) { vv_ai_finding_set_state($id, 'fixed', $note !== '' ? $note : 'Wrote ' . $key . ' at the operator\'s request.'); } return ['ok' => $wrote, 'action' => 'fix', 'error' => $wrote ? null : 'conf write refused — see conf_changes.log']; } return ['ok' => false, 'error' => 'unknown action']; } // ── Resolving a log line back to a conf key ────────────────────────────────────────────────── // By value wherever possible, by name only as a fallback. // // A log line usually contains the thing that failed — the URL that did not answer. That value // came from a conf key, so searching the conf for which key holds it is an exact lookup with a // definite answer. Guessing the key from the container's name is inference, and the failure mode // is silent: HOST1_EMBY_URL and HOST1_EMBY_EXTERNAL_URL are both plausible for "Emby" and only // one of them is the value that just failed. // // Same principle as resolve_tailscale_ip() refusing similarity matching for host identity: an // exact match or an honest nothing. // Every conf key whose value the given text starts with, longest first. Prefix rather than // equality because a log reports the URL it actually called — the conf value plus an endpoint // path — and the longest match is the most specific key that could have produced it. function vv_ai_conf_keys_for_value(string $value): array { $value = trim($value); if ($value === '' || strlen($value) < 6) return []; $hits = []; foreach (vv_conf_vars() as $k => $v) { $v = trim((string)$v); if ($v === '' || strlen($v) < 6) continue; if ($v === $value || str_starts_with($value, $v)) $hits[$k] = strlen($v); } arsort($hits); return array_keys($hits); } // __, built literally and then checked for existence. Nothing is // inferred: either the conf holds a key by exactly that name or this returns null. function vv_ai_conf_key_for_subject(string $subject, string $suffix): ?string { $norm = strtoupper(preg_replace('/[^A-Za-z0-9]+/', '_', trim($subject))); if ($norm === '') return null; $key = strtoupper(vv_detect_host()) . '_' . $norm . '_' . strtoupper($suffix); return array_key_exists($key, vv_conf_vars()) ? $key : null; } // Which conf file a key lives in. A finding has to name the file it would be edited in, and // host keys are not in master.conf. function vv_ai_conf_file_for_key(string $key): string { foreach (vv_get_conf_files() as $f) { if (preg_match('/^\s*' . preg_quote($key, '/') . '\s*=/m', vv_read_conf_raw($f))) return $f; } return 'master.conf'; } // ── Deterministic triage ───────────────────────────────────────────────────────────────────── // Patterns are written against log formats that exist in this repo today, taken from the emit // sites rather than imagined. A pattern that stops matching because its log line was reworded // produces no finding, which is the safe direction — the alternative is a pattern loose enough // to match anything, which fills the store with rows nobody can act on. // // No model runs here. "Connection refused" is not a judgement call, and a 14B model invoked // after every cron job to notice it would be slower, costlier and less reliable than a regex. // The model's job starts where these stop: explaining a finding, and talking the operator // through the ones that cannot be repaired automatically. const VV_AI_TRIAGE_PATTERNS = [ // docker_watchdog HTTP check — the richest signal, carrying both subject and failing URL. [ 're' => '/^(?P\S+) — not responding at (?P\S+) \(strike/u', 'kind' => 'unreachable', 'suffix' => 'URL', ], // docker_watchdog API check, 401/403. The endpoint answered, so the address is right and // the credential is not. [ 're' => '/^(?P\S+) — API check skipped \(HTTP (?:401|403)/u', 'kind' => 'auth_rejected', 'suffix' => 'API_KEY', ], // Configured for an API check with nothing to authenticate with. [ 're' => '/^(?P\S+) — API check skipped \(no key configured\)/u', 'kind' => 'missing_value', 'suffix' => 'API_KEY', ], [ 're' => '/^Skipping (?P\S+) — placeholder API key/u', 'kind' => 'missing_value', 'suffix' => 'API_KEY', ], [ 're' => '/^(?P\S+) (?:—\s*)?API unreachable/u', 'kind' => 'unreachable', 'suffix' => 'URL', ], ]; // One log line in, at most one finding candidate out. Returns null for everything else, which is // almost every line. function vv_ai_triage_line(string $line): ?array { // Strip the timestamp and level decoration the logger adds, so patterns can anchor on ^. $body = preg_replace('/^\S+\s+\S+\s+(?:[^\[]*\[[A-Z]+\]\s*)?/u', '', rtrim($line)); $body = trim((string)$body); if ($body === '') return null; foreach (VV_AI_TRIAGE_PATTERNS as $p) { if (!preg_match($p['re'], $body, $m)) continue; $subject = trim($m['subject'] ?? ''); $observed = trim($m['observed'] ?? ''); if ($subject === '') continue; // Value first, name second. Only one candidate key is accepted — two keys holding the // same value means the log cannot say which one produced it, and picking either is the // guess this whole approach exists to avoid. $key = null; if ($observed !== '') { $byValue = vv_ai_conf_keys_for_value($observed); if (count($byValue) === 1) $key = $byValue[0]; } if ($key === null) $key = vv_ai_conf_key_for_subject($subject, $p['suffix']); if ($key === null) continue; // nothing actionable — no row return [ 'kind' => $p['kind'], 'subject' => $subject, 'conf_key' => $key, 'conf_file' => vv_ai_conf_file_for_key($key), 'observed' => $observed !== '' ? $observed : (string)(vv_conf_vars()[$key] ?? ''), 'evidence' => $body, ]; } return null; } // A whole log tail in, one candidate per distinct fault out. A job that retried twelve times // produces twelve identical lines, and the store's dedupe would collapse them anyway — doing it // here keeps the sweep from writing the same file twelve times in a row. function vv_ai_triage_log(array $lines, string $sourceLog = ''): array { $found = []; foreach ($lines as $line) { $c = vv_ai_triage_line((string)$line); if ($c === null) continue; $c['source_log'] = $sourceLog; $found[vv_ai_finding_id($c['kind'], $c['subject'], $c['conf_key'])] = $c; } return array_values($found); } // ── The phrasebook ─────────────────────────────────────────────────────────────────────────── // What the operator said, what it was taken to mean, and whether that was right. // // The point is not to fine-tune anything. It is that "daily", said three times and meaning // daily_sync_maintenance.sh all three, stops being an inference and becomes a lookup. This file // is how a term earns that promotion, and everything it promotes is exact — the model keeps the // language, the resolution stays deterministic. Same division as the rest of this subsystem. // // The corrections are the rows that matter. A resolution that was right confirms what was // already believed; a resolution that was wrong, with what it should have been, is the only // record of a mistake that would otherwise be repeated indefinitely. // // JSON Lines rather than the pipe-delimited shape the token ledger uses: that file holds numbers // and a hostname, this one holds whatever the operator typed, and a delimiter that occurs in the // data is not a delimiter. Append-only, one object per line, so a truncated write costs the last // row rather than the corpus. function vv_ai_phrasebook_path(): string { return AI_DATA_DIR . '/ai_phrasebook.jsonl'; } // $said what the operator actually wrote, verbatim but redacted // $term the fragment that carried the meaning — "daily", "critical rsync", "emby key" // $target what it resolved to: a conf key, a script id, an array name // $kind conf_key | script | array | section | unknown // $outcome accepted | corrected | rejected // $correctedTo what it should have been, when the resolution was wrong // // $target and $correctedTo must be a canonical identifier — CONF_BACKUP_DIR, not // "CONF_BACKUP_DIR=${DATA_DIR}/Backups/Confs" and not "the backups directory". Promotion works // by counting how often a term resolved to the same thing, so a target described three different // ways is three meanings, and the term never promotes. Learned the hard way while seeding this // with real corrections: the same fix, written up three ways, taught nothing. function vv_ai_phrase_record(string $said, string $term, string $target, string $kind, string $outcome = 'accepted', string $correctedTo = ''): bool { $said = trim($said); $term = strtolower(trim($term)); if ($said === '' || $term === '') return false; if (!in_array($outcome, ['accepted', 'corrected', 'rejected'], true)) return false; if (!is_dir(AI_DATA_DIR)) @mkdir(AI_DATA_DIR, 0755, true); // An operator asking to set a credential types the credential. This file is long-lived and // read back for years; it is the last place a key should be preserved verbatim. $row = [ 'ts' => time(), 'said' => mb_substr(vv_ai_redact($said), 0, 500), 'term' => mb_substr($term, 0, 80), 'target' => mb_substr(trim($target), 0, 160), 'kind' => $kind, 'outcome' => $outcome, ]; if ($correctedTo !== '') $row['corrected_to'] = mb_substr(vv_ai_redact($correctedTo), 0, 160); return @file_put_contents(vv_ai_phrasebook_path(), json_encode($row, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE) . "\n", FILE_APPEND | LOCK_EX) !== false; } function vv_ai_phrase_all(): array { $out = []; foreach ((array)@file(vv_ai_phrasebook_path(), FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES) as $line) { $r = json_decode($line, true); if (is_array($r) && isset($r['term'])) $out[] = $r; } return $out; } // Terms that have earned a deterministic mapping: seen at least $minSeen times, resolving to one // target every time, and never corrected away from it. // // A single contradiction disqualifies the term outright rather than going with the majority. The // whole value of promoting a term is that it stops being a guess — a term that meant two things // is still a guess, and a confident wrong alias is worse than no alias, because nothing downstream // will question it. function vv_ai_phrase_aliases(int $minSeen = 3): array { $seen = []; foreach (vv_ai_phrase_all() as $r) { $term = (string)$r['term']; // A correction records both the wrong reading and the right one. The right one is what // the term means; the wrong one is what it must never be promoted to again. $target = ($r['outcome'] === 'corrected' && !empty($r['corrected_to'])) ? (string)$r['corrected_to'] : (string)$r['target']; if ($r['outcome'] === 'rejected' || $target === '') continue; $seen[$term]['targets'][$target] = ($seen[$term]['targets'][$target] ?? 0) + 1; $seen[$term]['kind'] = (string)($r['kind'] ?? 'unknown'); if ($r['outcome'] === 'corrected') $seen[$term]['wrong'][(string)$r['target']] = true; } $aliases = []; foreach ($seen as $term => $d) { if (count($d['targets']) !== 1) continue; // meant two things — still a guess $target = array_key_first($d['targets']); if (isset($d['wrong'][$target])) continue; // was itself corrected away from if ($d['targets'][$target] < $minSeen) continue; $aliases[$term] = ['target' => $target, 'kind' => $d['kind'], 'seen' => $d['targets'][$target]]; } return $aliases; } // Exact alias hit for a term, or null. Deliberately not fuzzy: an alias exists precisely so that // this lookup is certain, and a near-match would reintroduce the guessing it replaced. function vv_ai_phrase_lookup(string $term, int $minSeen = 3): ?array { return vv_ai_phrase_aliases($minSeen)[strtolower(trim($term))] ?? null; } // ── What the repair profile is given before it answers ─────────────────────────────────────── // Two records of what has already happened, assembled as plain text for the prompt. // // The phrasebook half is the operator's vocabulary — and the corrections matter more than the // settled terms, because a term that has been corrected is one the assistant has already got // wrong once and would otherwise get wrong again. // // The closed-findings half is this installation's history: the same fault, and what actually // ended it. "Emby stopped answering last month and the address had changed" is worth more at the // start of a repair conversation than any amount of reasoning from first principles. // // Bounded hard. This goes into a 16k context that retrieval and a log tail are also competing // for, and an unbounded history would crowd out the evidence for the fault actually being // discussed. function vv_ai_repair_context(int $maxAliases = 20, int $maxFixes = 8): string { $out = []; $aliases = vv_ai_phrase_aliases(); if ($aliases) { $lines = []; foreach (array_slice($aliases, 0, $maxAliases, true) as $term => $a) { $lines[] = ' "' . $term . '" means ' . $a['target']; } $out[] = "What the operator calls things:\n" . implode("\n", $lines); } $unsettled = vv_ai_phrase_unsettled(); if ($unsettled) { $lines = []; foreach (array_slice($unsettled, 0, $maxAliases, true) as $term => $c) { $last = end($c); $lines[] = ' "' . $term . '" was read as ' . ($last['took_it_as'] ?: '?') . ' and meant ' . ($last['meant'] ?: '?'); } $out[] = "Corrected before — do not repeat these:\n" . implode("\n", $lines); } $closed = vv_ai_findings_list(['fixed', 'resolved']); if ($closed) { $lines = []; foreach (array_slice($closed, 0, $maxFixes) as $f) { $lines[] = ' ' . ($f['subject'] ?? '?') . ' / ' . ($f['ref'] ?? $f['conf_key'] ?? '?') . ' — ' . ($f['state'] ?? '?') . (!empty($f['note']) ? ': ' . mb_substr((string)$f['note'], 0, 140) : ''); } $out[] = "Already dealt with on this host:\n" . implode("\n", $lines); } $spellings = vv_ai_spellings(); if ($spellings) { $lines = []; foreach (array_slice($spellings, 0, $maxAliases, true) as $typo => $meant) { $lines[] = ' "' . $typo . '" = "' . $meant . '"'; } $out[] = "The operator types quickly; known shorthand:\n" . implode("\n", $lines); } return implode("\n\n", $out); } // ── Resolving what the operator named ──────────────────────────────────────────────────────── // "turn off the zfs scrub", "change the emby api key" — a phrase in, an exact target out, or an // honest refusal with the candidates that were considered. // // Four layers, most certain first, and each either answers exactly or declines: // // 1. A learned alias. The operator has used this term before and it settled on one target. // 2. The literal key. "HOST1_EMBY_URL" or "host1 emby url" is not a phrase to interpret. // 3. Every conf key containing all of the significant words. Exactly one is an answer; more // than one is a question. // 4. Script ids from the orchestrator arrays, matched the same way. // // What this deliberately does not do is score similarity. There is no closest match, no edit // distance, no "did you mean". The same reasoning as resolve_tailscale_ip() refusing to guess at // host identity: the failure mode of a near-match is silent and confident, and here it would // write to a key the operator never named. Ambiguity is returned as ambiguity, and the assistant // asks — which is also how the phrasebook learns, because the answer to that question is a // correction worth recording. // // Stop words exist because "the", "for" and "in" appear in a conf key somewhere and would make // every phrase match everything. const VV_AI_RESOLVE_STOPWORDS = [ 'the','a','an','to','for','in','on','of','and','or','is','it','this','that','my','our', 'please','can','you','set','change','turn','make','update','put','value','key','conf','config', 'setting','settings','off','on_','now','back','again','me','we','let','lets', ]; // A key or script id broken into its own words. HOST1_EMBY_API_KEY is four words, and // Tools/zfs_pool_scrub.sh is five — the path separator and the extension are word boundaries too. function vv_ai_resolve_segments(string $name): array { $n = strtolower(preg_replace('/\.sh$/', '', $name)); return array_values(array_filter(preg_split('/[^a-z0-9]+/', $n, -1, PREG_SPLIT_NO_EMPTY) ?: [])); } function vv_ai_resolve_tokens(string $phrase): array { $p = strtolower(trim($phrase)); $p = preg_replace('/[^a-z0-9_\s-]+/', ' ', $p); $words = preg_split('/[\s_-]+/', $p, -1, PREG_SPLIT_NO_EMPTY) ?: []; return array_values(array_filter($words, fn($w) => strlen($w) > 1 && !in_array($w, VV_AI_RESOLVE_STOPWORDS, true))); } // Script ids named in any *_SCRIPTS array, so "zfs scrub" can resolve to Tools/zfs_pool_scrub.sh // — which is the shape of request that has no conf key at all. function vv_ai_resolve_script_ids(): array { $ids = []; foreach (vv_get_conf_files() as $f) { if (preg_match_all('/^\s*#?\s*"([A-Za-z0-9_\/.-]+\.sh)(?:\s[^"]*)?"/m', vv_read_conf_raw($f), $m)) { foreach ($m[1] as $id) $ids[$id] = true; } } return array_keys($ids); } // Returns: // ok=true with target, kind and via — one certain answer // ok=false with candidates — several, and the caller must ask // ok=false with candidates empty — nothing recognised function vv_ai_resolve_target(string $phrase): array { $none = ['ok' => false, 'target' => null, 'kind' => null, 'via' => 'none', 'candidates' => []]; $tokens = vv_ai_resolve_tokens($phrase); if (!$tokens) return $none; // 1 — learned $alias = vv_ai_phrase_lookup(strtolower(trim($phrase))); if ($alias === null) { // Also try the significant words alone, since "turn off ai repair" and "ai repair" are // the same instruction with different framing. $alias = vv_ai_phrase_lookup(implode(' ', $tokens)); } if ($alias !== null) { return ['ok' => true, 'target' => $alias['target'], 'kind' => $alias['kind'], 'via' => 'alias', 'candidates' => []]; } $vars = vv_conf_vars(); // 2 — the literal key, however it was spaced or cased $literal = strtoupper(implode('_', $tokens)); if (array_key_exists($literal, $vars)) { return ['ok' => true, 'target' => $literal, 'kind' => 'conf_key', 'via' => 'exact', 'candidates' => []]; } // 3 — conf keys whose own words include every significant word // // Whole segments, not substrings. Substring matching made "mov" resolve to // MOVER_STOP_TIMEOUT with full confidence, which is exactly the near-match this is supposed // to refuse: a short fragment that happens to be unique is not the operator naming a key. $keyHits = []; foreach (array_keys($vars) as $key) { $segs = vv_ai_resolve_segments($key); foreach ($tokens as $t) { if (!in_array($t, $segs, true)) continue 2; } $keyHits[] = $key; } if (count($keyHits) === 1) { return ['ok' => true, 'target' => $keyHits[0], 'kind' => 'conf_key', 'via' => 'match', 'candidates' => []]; } // 4 — script ids, same rule $scriptHits = []; foreach (vv_ai_resolve_script_ids() as $id) { $segs = vv_ai_resolve_segments($id); foreach ($tokens as $t) { if (!in_array($t, $segs, true)) continue 2; } $scriptHits[] = $id; } if (!$keyHits && count($scriptHits) === 1) { return ['ok' => true, 'target' => $scriptHits[0], 'kind' => 'script', 'via' => 'match', 'candidates' => []]; } $all = array_merge($keyHits, $scriptHits); if (!$all) return $none; // Several. Returned rather than ranked — picking one here is the guess this avoids. sort($all); return ['ok' => false, 'target' => null, 'kind' => null, 'via' => 'ambiguous', 'candidates' => array_slice($all, 0, 12)]; } // ── Spellings ──────────────────────────────────────────────────────────────────────────────── // The operator types quickly and knows it: "haversync" for "have rsync", "as it to the list" for // "add it". Recorded when the meaning was obvious in context, so the next occurrence is read // rather than puzzled over. // // Promoted on first sighting, unlike a term alias, and the difference is deliberate. A term alias // decides what gets written to conf, so it has to be earned by repetition. A spelling decides how // a sentence is read, costs a misreading at worst, and is rarely made identically three times — // requiring repetition would mean never learning any of them. // // Recorded only when the intent was actually clear. A guess written down here is worse than // leaving it out, because it will be applied silently every time afterwards. function vv_ai_spelling_record(string $said, string $typo, string $meant): bool { $typo = strtolower(trim($typo)); $meant = trim($meant); if ($typo === '' || $meant === '' || strcasecmp($typo, $meant) === 0) return false; return vv_ai_phrase_record($said, $typo, $meant, 'spelling'); } // What the operator most likely meant by a word, or null. Last writing wins, so a correction to // an earlier reading simply supersedes it. function vv_ai_spelling_lookup(string $word): ?string { $word = strtolower(trim($word)); $hit = null; foreach (vv_ai_phrase_all() as $r) { if (($r['kind'] ?? '') !== 'spelling' || ($r['outcome'] ?? '') === 'rejected') continue; if (($r['term'] ?? '') === $word) $hit = (string)$r['target']; } return $hit; } function vv_ai_spellings(): array { $out = []; foreach (vv_ai_phrase_all() as $r) { if (($r['kind'] ?? '') !== 'spelling' || ($r['outcome'] ?? '') === 'rejected') continue; $out[(string)$r['term']] = (string)$r['target']; } ksort($out); return $out; } // Terms that have been corrected and have not yet earned promotion — what the assistant is still // getting wrong, and the thing worth reading when asking why it keeps mistaking something. function vv_ai_phrase_unsettled(): array { $out = []; foreach (vv_ai_phrase_all() as $r) { if (($r['outcome'] ?? '') !== 'corrected') continue; $out[(string)$r['term']][] = ['said' => $r['said'], 'took_it_as' => $r['target'], 'meant' => $r['corrected_to'] ?? '', 'ts' => $r['ts'] ?? 0]; } return $out; } // ── What the arrs say about themselves ─────────────────────────────────────────────────────── // Sonarr, Radarr and Lidarr each publish a health endpoint listing what they believe is wrong, // already structured and already graded. No log parsing, no pattern that goes stale when a // message is reworded, and no guessing at severity — the arr is the authority on whether its own // condition is an error or a warning. // // This is the one source here that needs no triage at all. Everything else in this file exists // because logs are prose; these arrive as records. // // Note the API version differs: Lidarr is v1 where Sonarr and Radarr are v3. vv_discover_arrs() // already carries it per arr, which is why this reads it rather than assuming. function vv_ai_arr_health_findings(): array { if (!function_exists('vv_discover_arrs')) { require_once __DIR__ . '/arrs.php'; } $host = vv_detect_host(); $found = []; foreach (vv_discover_arrs() as $node) { if (($node['host'] ?? '') !== $host) continue; foreach ($node['arrs'] ?? [] as $arr) { $type = ucfirst((string)($arr['type'] ?? '')); $url = (string)($arr['url'] ?? ''); $key = (string)($arr['key'] ?? ''); $api = (string)($arr['api'] ?? 'v3'); if ($type === '' || $url === '' || $key === '') continue; $items = vv_arr_http($url, $key, "/api/$api/health", vv_ai_probe_timeout()); // null is unreachable, which is a different finding and one the log triage already // raises. An empty array is the arr saying it is fine, and must not be confused with // not having been able to ask. if (!is_array($items)) continue; foreach ($items as $item) { $source = trim((string)($item['source'] ?? '')); $message = trim((string)($item['message'] ?? '')); if ($source === '' || $message === '') continue; $found[] = [ 'kind' => 'arr_health', 'subject' => $type, 'ref' => $source, 'conf_key' => '', 'conf_file' => '', 'arr_type' => strtolower((string)($item['type'] ?? 'warning')), 'observed' => $message, 'evidence' => $type . ' › ' . $source . ': ' . $message, 'source_log' => $type . ' /api/' . $api . '/health', // The arr's own documentation for this check, which is the actual next step // for most of them and costs nothing to carry. 'note' => trim((string)($item['wikiUrl'] ?? '')), ]; } } break; } return $found; } // ── Proving a candidate ────────────────────────────────────────────────────────────────────── // The guard the whole unattended path rests on: nothing is written that has not answered. // // A model can be confident that a port should be 8686. A probe can report that 8686 answered. // Only the second is a fact, and only facts get written to conf without being asked. Everything // a probe cannot settle becomes a conversation instead — which is not a lesser outcome, it is // the honest one for a value that cannot be derived from this machine. // // Deliberately narrow. These check reachability and identity, never correctness of behaviour: // that Lidarr answers on 8686 does not prove 8686 is the port you meant, only that something is // listening there and calling itself Lidarr. Proving intent is not a probe's job. function vv_ai_probe_timeout(): int { return max(1, (int)(vv_conf_vars()['AI_PROBE_TIMEOUT'] ?? 4)); } // Does this URL answer at all? Any HTTP status counts, including 401 — a refusal is proof that // something is listening and speaking HTTP, which is exactly what an address probe is asking. // Distinguishing "wrong address" from "wrong credential" is the point of having both kinds. function vv_ai_probe_url(string $url): array { if (!preg_match('#^https?://[^\s/$.?\#][^\s]*$#i', $url)) { return ['ok' => false, 'reason' => 'not a url']; } $ch = curl_init($url); curl_setopt_array($ch, [ CURLOPT_RETURNTRANSFER => true, CURLOPT_NOBODY => true, CURLOPT_TIMEOUT => vv_ai_probe_timeout(), CURLOPT_CONNECTTIMEOUT => vv_ai_probe_timeout(), CURLOPT_SSL_VERIFYPEER => false, // these are LAN and Tailscale endpoints, often self-signed CURLOPT_SSL_VERIFYHOST => false, ]); curl_exec($ch); $code = (int)curl_getinfo($ch, CURLINFO_RESPONSE_CODE); $err = curl_error($ch); curl_close($ch); return $code > 0 ? ['ok' => true, 'code' => $code] : ['ok' => false, 'reason' => $err !== '' ? $err : 'no response']; } // Every container on this host, by exact name. The membership test for unknown_target findings, // and one docker call rather than one per candidate. function vv_ai_container_names(): array { static $names = null; if ($names !== null) return $names; $out = []; @exec('timeout ' . vv_ai_probe_timeout() . " docker ps -a --format '{{.Names}}' 2>/dev/null", $out, $rc); return $names = ($rc === 0) ? array_values(array_filter(array_map('trim', $out))) : []; } // Candidate corrections for a URL whose host or port stopped answering. // // Only two transformations, both conservative: the same host on a port that some other conf key // already uses, and the same port on a host some other conf key already names. Both draw // exclusively from values already present in this installation's conf — nothing is invented, and // a scan of the port range is deliberately not attempted. Finding *a* listening port is not the // same as finding the right service, and a probe that accepts any answer would happily point // Lidarr at Sonarr. function vv_ai_url_candidates(string $observed): array { $parts = @parse_url($observed); if (!is_array($parts) || empty($parts['host'])) return []; $hosts = $ports = []; foreach (vv_conf_vars() as $k => $v) { $v = trim((string)$v); if (!preg_match('#^https?://#i', $v)) continue; $p = @parse_url($v); if (!is_array($p) || empty($p['host'])) continue; $hosts[$p['host']] = true; if (!empty($p['port'])) $ports[(int)$p['port']] = true; } $scheme = $parts['scheme'] ?? 'http'; $path = $parts['path'] ?? ''; $out = []; foreach (array_keys($ports) as $port) { $c = $scheme . '://' . $parts['host'] . ':' . $port . $path; if ($c !== $observed) $out[$c] = true; } foreach (array_keys($hosts) as $host) { $port = !empty($parts['port']) ? ':' . $parts['port'] : ''; $c = $scheme . '://' . $host . $port . $path; if ($c !== $observed) $out[$c] = true; } return array_keys($out); } // Try to prove a correction for one finding. Returns the finding with 'proposed' and 'proven' // filled in, or unchanged when nothing could be proven — which is the common case and not a // failure. // // auth_rejected and missing_value are never proven here on purpose. A credential cannot be // derived from this host by definition: if it could be read from somewhere, it would not be a // credential. Those go straight to the operator. function vv_ai_probe_finding(array $f): array { $kind = (string)($f['kind'] ?? ''); if ($kind === 'unknown_target') { $observed = (string)($f['observed'] ?? ''); // Exact membership only. A container name is a literal, and "close to an existing name" // is how a repair renames the wrong thing. if (in_array($observed, vv_ai_container_names(), true)) { return $f; // it exists after all — nothing to correct } return $f + ['state' => 'needs_operator']; } if ($kind !== 'unreachable') { // Nothing on this machine can supply a credential, so there is nothing to prove. $f['state'] = 'needs_operator'; return $f; } $observed = (string)($f['observed'] ?? ''); // If the observed address answers now, the fault has cleared on its own — a host that was // rebooting, most often. Recording a proposal here would repair something already working. if (vv_ai_probe_url($observed)['ok']) { $f['state'] = 'resolved'; $f['note'] = 'Answered when probed — the address was reachable again by the time this ran.'; return $f; } $answered = []; foreach (vv_ai_url_candidates($observed) as $candidate) { if (vv_ai_probe_url($candidate)['ok']) $answered[] = $candidate; } // Exactly one, or none. Two addresses answering means the probe cannot say which is the // right one, and picking either is the guess this exists to prevent. if (count($answered) === 1) { $f['proposed'] = $answered[0]; $f['proven'] = true; $f['note'] = 'Probed ' . $answered[0] . ' and it answered; ' . $observed . ' did not.'; return $f; } $f['proven'] = false; $f['state'] = 'needs_operator'; $f['note'] = $answered ? 'Several addresses answered (' . implode(', ', $answered) . '), so none was written.' : 'Nothing answered at ' . $observed . ', and no address in the conf answered either.'; return $f; }