Add config-vs-reality health checks, loaded models, and log evidence

The failures this subsystem actually has are configuration drift, so each
check names the setting to change rather than reporting that retrieval
failed. Notably it catches a conf model tag that is no longer installed, and
an index built by a different embedder than the one configured — vectors
from two models are not comparable, and that failure returns confident
nonsense rather than erroring. Diagnostic questions also get recent log
warnings, attached only then because they cost budget the passages need.
This commit is contained in:
Gmer4Lfe
2026-08-02 17:40:43 -04:00
parent 264ba57cbb
commit 237156c46f
3 changed files with 322 additions and 7 deletions
+50 -6
View File
@@ -50,6 +50,13 @@
// generated answer with no grounding is exactly the confident hallucination this whole
// subsystem exists to prevent.
//
// Live state is attached only to diagnostic questions.
// Failing health checks and recent log warnings are several thousand tokens. On a
// 16384 context that is budget taken directly from the retrieved passages, so it is
// spent only when the question is asking why something broke. Log lines are marked as
// evidence rather than citable sources, so the model cannot cite a log line as though
// it were documentation.
//
// Every failure path writes the job file.
// Including the ones that would otherwise be silent — unreachable Ollama, unparseable
// response, empty content — so the tab always converges on a state it can render.
@@ -116,13 +123,50 @@ foreach ($r['results'] as $i => $x) {
$context .= '[' . ($i + 1) . '] ' . $label . "\n" . trim($x['content'] ?? '') . "\n\n";
}
// Diagnostic questions get live state as well as documentation. The docs say what a script is
// supposed to do; only the logs and the current config say what it actually did. Attached only
// when the question is asking why something failed — otherwise it is a few thousand tokens of
// noise competing with the retrieved passages for a context budget that is already tight.
$diagnostic = (bool)preg_match(
'/\b(why|fail(ed|ing|ure)?|error|broken?|not work|isn.t work|wrong|stuck|hang|'
. 'never runs?|didn.t|won.t|debug|troubleshoot|diagnos)/i',
$question
);
$diagBlock = '';
if ($diagnostic) {
$bad = array_filter(vv_ai_health(), fn($c) => in_array($c['state'], ['bad', 'warn'], true));
if ($bad) {
$diagBlock .= "CURRENT CONFIGURATION PROBLEMS\n";
foreach ($bad as $c) {
$diagBlock .= '- [' . strtoupper($c['state']) . '] ' . $c['label'] . ': ' . $c['detail']
. ($c['fix'] !== '' ? ' — ' . $c['fix'] : '') . "\n";
}
$diagBlock .= "\n";
}
$logs = vv_ai_recent_logs(40);
if ($logs) {
$diagBlock .= "RECENT WARNINGS AND ERRORS (newest last)\n" . implode("\n", $logs) . "\n\n";
}
}
$system = "You are Varaverk's documentation assistant. Varaverk is this user's private "
. "two-server Unraid media ecosystem; it is not in your training data, so the passages "
. "below are the only thing you know about it.\n\n"
. "Answer only from these passages and cite them inline as [1], [2]. If they do not "
. "contain the answer, say so plainly and name what is missing — do not fill the gap "
. "from general knowledge. Prefer the user's own terminology.\n\n"
. "PASSAGES\n" . $context;
. "two-server Unraid media ecosystem; it is not in your training data, so the material "
. "below is the only thing you know about it.\n\n"
. "Answer only from this material and cite the passages inline as [1], [2]. If it does "
. "not contain the answer, say so plainly and name what is missing — do not fill the "
. "gap from general knowledge. Prefer the user's own terminology.\n\n";
if ($diagBlock !== '') {
$system .= "This is a diagnostic question, so live system state is included alongside the "
. "documentation. Use the passages to explain how the thing is supposed to work, "
. "and the live state to say what is actually wrong. When a configuration problem "
. "is listed, name the specific setting and the file it lives in. Log lines are "
. "evidence, not citations — cite only the numbered passages.\n\n"
. $diagBlock;
}
$system .= "PASSAGES\n" . $context;
$messages = [['role' => 'system', 'content' => $system]];
$hist = json_decode($historyJson ?: '[]', true);