FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Share one AI across the mesh instead of confining it to the owner

Browse Source 
Curated state copied to every node is state that can disagree, so the index,
the model and the shared memory stay on the owner and each node reaches them
over the SSH trust onboarding already builds. Chats stay on the node that had
them; memory and bug reports stay the owner's to write.
This commit is contained in:
Gmer4Lfe
2026-08-20 19:53:28 -04:00
parent 4accd6b67e
commit 0621f66889
11 changed files with 1013 additions and 541 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Plugin/unraid/include/ai_actions.php
+542
View File
@@ -0,0 +1,542 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Every AI action, in one dispatcher, independent of how the request arrived. api/ai.php calls
// it for browser requests; Tools/ai_rpc.php calls it for mesh requests forwarded from another
// node over SSH. Both get identical behaviour because there is only one implementation.
//
// DESIGN PRINCIPLES
// Transport-free. Nothing here reads a superglobal, sets a header, echoes, or exits. Handlers
// take a params array and return the response body as an array. That is what makes the
// same code serveable over HTTP and over SSH — the handlers cannot tell the difference,
// so the two paths cannot drift.
//
// The dispatcher owns behaviour; the caller owns access.
// Authentication, CSRF and the node/action gates live in the caller. This file assumes
// the request is already allowed. Two callers with different trust models share one set
// of handlers precisely because the handlers do not re-litigate trust.
//
// Extracted, not rewritten.
// This was the body of api/ai.php. The transformation is mechanical — `echo json_encode(X);
// exit` became `return X`, a 405 became $httpStatus plus a return, and superglobals became
// $p. Every validation, every whitelist and every comment is the original. A refactor of a
// file this security-sensitive earns nothing by also being a redesign.
//
// OPERATIONAL SAFEGUARDS
// Tokens are minted here and never accepted from elsewhere.
// random_bytes, hex, fixed length. A job's answer is readable by anyone who can guess its
// token, so the token is not guessable.
//
// Every token is validated as hex before it composes a path.
// vv_ai_job_path() returns null for anything else, and each caller checks. That pattern is
// what confines reads and deletes to the job directory.
//
// History is validated per message, not trusted as a blob.
// Role must be user or assistant, content must be a non-empty string, and each is
// truncated. A crafted history could otherwise inject a system role or push the context
// past the offload ceiling. chat_save applies the same validation, because a stored chat
// is replayed into a later prompt.
//
// Scope is whitelisted, not escaped.
// It reaches the model as text, so anything richer than a file name is an
// instruction-injection surface for no benefit.
//
// stop signals one verified pid, never a process group.
// The cmdline must name both the worker and this job's own token before anything is
// signalled, because pid reuse is real. Group signalling took the WebGUI down on
// 2026-08-07.
//
// EXPORTS
// vv_ai_dispatch() action → response body
// vv_ai_log() the shared request/action trace at /var/log/varaverk/ai.log
//
// DEPENDS ON
// include/ai.php config, stats, memory, chats, bugs, job files
// include/ai_memory_learn.php loaded per action — learned-memory proposals
// include/ai_repair.php loaded per action — the largest include in the plugin
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/ai.php';
// History depth is per profile and still decided server-side rather than by the page — the page
// simply no longer carries a second copy of the numbers. include/ai_profiles.php holds them.
// Reasoning is not stored in history, so it does not compound.
const VV_AI_MAX_QUESTION = 4000; // characters
const VV_AI_MAX_HIST_MSG = 4000; // characters per retained message
const VV_AI_JOB_TTL = 3600; // seconds before a job file is reaped
// Request trace. There is no nginx access log on this host and the CSRF prepend exits with an
// empty body, so without this there is no way to tell "the request never arrived" from "the
// request arrived and failed" — which is exactly the ambiguity that made the first hang
// undiagnosable. Callers exclude poll, which would otherwise write a line per second per open tab.
function vv_ai_log(string $msg): void {
if (!is_dir('/var/log/varaverk')) return;
@file_put_contents('/var/log/varaverk/ai.log',
date('Y-m-d H:i:s') . ' ' . $msg . "\n", FILE_APPEND | LOCK_EX);
}
// Run one action. $p carries the request parameters regardless of how they arrived; $isPost is
// still passed separately because the POST-only checks are a CSRF guarantee, not a parameter.
//
// $httpStatus is by reference and named for what it is — `$status` is taken, by stop's own local
// for a job's state, and shadowing that would be a silent bug in the one handler that kills a
// process.
function vv_ai_dispatch(string $action, array $p, bool $isPost, int &$httpStatus = 200): array {
$postOnly = function () use (&$httpStatus): array {
$httpStatus = 405;
return ['ok' => false, 'error' => 'POST only'];
};
// ── stats ─────────────────────────────────────────────────────────────────────
// Served from the shared 'ai' cache that Tools/api_cache_writer.sh refreshes every minute, on
// the same terms as the monitor and arrs payloads. This action is polled every 30 seconds by
// every open tab and used to pay a full collection each time — around a second, most of it
// spent waiting on Ollama and nvidia-smi — for numbers that only change when the writer runs.
//
// live=1 bypasses it, for the case where something was just changed and the point is to see
// the result. A missing cache always falls back to collecting, so the cache can never be the
// reason the banner fails to render.
if ($action === 'stats') {
return ['ok' => true, 'stats' => vv_ai_stats_cached(isset($p['live']))];
}
// ── tokens ────────────────────────────────────────────────────────────────────
// Separate from stats rather than folded into it. stats is polled every 30 seconds by every
// open tab; this reads a file that grows without bound between prunes. The totals only move
// when a turn completes, and the page knows exactly when that happened, so it asks then.
if ($action === 'tokens') {
return ['ok' => true, 'tokens' => vv_ai_token_stats()];
}
// ── poll ──────────────────────────────────────────────────────────────────────
if ($action === 'poll') {
$token = trim($p['token'] ?? '');
if (vv_ai_job_path($token) === null) {
return ['ok' => false, 'error' => 'Invalid token'];
}
$job = vv_ai_job_read($token);
if ($job === null) {
// The worker writes its first state after this request may already have arrived.
return ['ok' => true, 'job' => ['status' => 'pending']];
}
return ['ok' => true, 'job' => $job];
}
// ── memory ────────────────────────────────────────────────────────────────────
if ($action === 'memory_get') {
$m = vv_ai_memory_read();
return ['ok' => true, 'memory' => $m['text'], 'chars' => $m['chars'],
'max' => vv_ai_memory_max(), 'exists' => $m['exists'],
'path' => vv_ai_memory_path()];
}
if ($action === 'memory_set') {
if (!$isPost) return $postOnly();
$r = vv_ai_memory_write((string)($p['memory'] ?? ''));
vv_ai_log('memory_set ' . ($r['ok'] ? 'ok chars=' . $r['chars'] : 'FAILED: ' . $r['error']));
return $r + ['max' => vv_ai_memory_max()];
}
// ── learned-memory proposals ──────────────────────────────────────────────────
// The store the assistant files candidates into. Accepting is the only path by which
// model-written text reaches a prompt, and it is a POST so the CSRF prepend covers it.
if ($action === 'mem_proposals') {
require_once __DIR__ . '/ai_memory_learn.php';
$m = vv_ai_memory_read('learned');
return [
'ok' => true,
'enabled' => vv_ai_mem_learn_enabled(),
'auto' => vv_ai_mem_learn_auto(),
// The list states the gate as well as the rows: an empty list means "nothing proposed"
// when learning is on and "nothing is looking" when it is off, and those are different.
'open' => vv_ai_mem_list('open'),
'recent' => array_slice(vv_ai_mem_list(), 0, 25),
'learned' => ['chars' => $m['chars'], 'max' => vv_ai_memory_learned_max()],
];
}
if ($action === 'mem_proposal_action') {
if (!$isPost) return $postOnly();
require_once __DIR__ . '/ai_memory_learn.php';
$id = trim($p['id'] ?? '');
$act = trim($p['act'] ?? '');
$r = vv_ai_mem_action($id, $act);
vv_ai_log(sprintf('mem_proposal id=%s act=%s %s', $id, $act,
$r['ok'] ? 'ok' : ('FAILED: ' . ($r['error'] ?? '?'))));
return $r;
}
// ── stop ──────────────────────────────────────────────────────────────────────
// Cancels a generation in flight. Only ever signals ONE pid, verified to be the worker for
// this exact job — never a process group. Signalling a group is what took the WebGUI down on
// 2026-08-07, and no group kill is needed here: the worker is a single php process whose only
// child-like thing is an HTTP connection to Ollama, which dies with it.
//
// Whatever was already generated is kept. A turn stopped at 80% is usually stopped because the
// operator has seen enough, not because they want it discarded.
if ($action === 'stop') {
if (!$isPost) return $postOnly();
$token = trim($p['token'] ?? '');
if (vv_ai_job_path($token) === null) {
return ['ok' => false, 'error' => 'Invalid token'];
}
$job = vv_ai_job_read($token);
if ($job === null) return ['ok' => false, 'error' => 'No such job'];
$status = (string)($job['status'] ?? '');
if ($status === 'done' || $status === 'error' || $status === 'stopped') {
return ['ok' => true, 'already' => true, 'status' => $status];
}
$pid = (int)($job['pid'] ?? 0);
// Below 2 is init or nonsense. A pid we cannot verify is a pid we do not signal.
$killed = false;
if ($pid >= 2) {
// Pid reuse is the reason for this: the recorded worker may have exited seconds ago
// and the number been handed to something else entirely. The cmdline must name both
// this worker and this job's own file before anything is signalled.
$cmdline = @file_get_contents("/proc/$pid/cmdline");
$cmdline = $cmdline === false ? '' : str_replace("\0", ' ', $cmdline);
if (strpos($cmdline, 'ai_chat_worker.php') !== false && strpos($cmdline, $token) !== false) {
$killed = @posix_kill($pid, SIGTERM);
// No escalation ladder. The worker holds no lock and writes the job file
// atomically, so there is no cleanup that a delay would protect — and a SIGKILL
// race could land between the temp write and the rename.
}
}
// The job file is rewritten either way. If the pid could not be verified the worker is
// already gone, and the page still needs a terminal state instead of polling to its
// ceiling.
$job['status'] = 'stopped';
$job['stopped'] = true;
$job['answer'] = trim((string)($job['partial'] ?? $job['answer'] ?? ''));
unset($job['partial']);
@file_put_contents(vv_ai_job_path($token), json_encode($job));
vv_ai_log(sprintf('stop token=%s pid=%d signalled=%s kept=%d chars',
substr($token, 0, 12), $pid, $killed ? 'yes' : 'no', strlen($job['answer'])));
return ['ok' => true, 'signalled' => $killed, 'kept' => strlen($job['answer'])];
}
// ── clear ─────────────────────────────────────────────────────────────────────
if ($action === 'clear') {
if (!$isPost) return $postOnly();
$path = vv_ai_job_path(trim($p['token'] ?? ''));
if ($path === null) return ['ok' => false, 'error' => 'Invalid token'];
if (file_exists($path)) @unlink($path);
return ['ok' => true];
}
// ── chats ─────────────────────────────────────────────────────────────────────
// Stored conversations. Listing and reading are GET because they change nothing; saving and
// deleting are POST, so they ride Unraid's CSRF prepend like every other mutation here.
//
// Messages are validated per message on the way in, exactly as ask validates history and for
// the same reason: a stored chat is replayed into a later prompt when the operator reopens it,
// so a crafted role in the store would be an injection that survives a reload.
if ($action === 'chats') {
return ['ok' => true, 'chats' => vv_ai_chats_list(), 'max' => vv_ai_chats_max()];
}
if ($action === 'chat_get') {
$chat = vv_ai_chat_read(trim($p['id'] ?? ''));
if ($chat === null) return ['ok' => false, 'error' => 'No such chat'];
return ['ok' => true, 'chat' => $chat];
}
if ($action === 'chat_save') {
if (!$isPost) return $postOnly();
$profile = trim($p['profile'] ?? 'chat');
if (!vv_ai_profile_ok($profile)) {
return ['ok' => false, 'error' => 'Unknown profile: ' . $profile];
}
$clean = [];
$msgs = json_decode($p['messages'] ?? '[]', true);
if (is_array($msgs)) {
foreach ($msgs as $m) {
$role = $m['role'] ?? '';
$text = trim((string)($m['content'] ?? ''));
if (!in_array($role, ['user', 'assistant'], true) || $text === '') continue;
$clean[] = ['role' => $role, 'content' => mb_substr($text, 0, VV_AI_MAX_HIST_MSG)];
}
}
// Capped at the deepest profile's window rather than that of the profile in hand. A chat
// saved under one profile can be reopened under another, and the reopened turn is trimmed
// again on the way back out by ask — so storing a little more than any single profile will
// send costs nothing and keeps the transcript readable.
$cap = vv_ai_profiles_max_turns() * 2;
if (count($clean) > $cap) $clean = array_slice($clean, -$cap);
// Whitelisted exactly as ask's is, and for the same reason: a scope is only ever a name
// from a page's own view state, it is stored and later replayed into a prompt, and
// anything richer than a file name is an instruction-injection surface for no benefit.
$scope = trim($p['scope'] ?? '');
if ($scope !== '' && !vv_ai_scope_ok($scope)) $scope = '';
$r = vv_ai_chat_save(trim($p['id'] ?? ''), $profile, $clean, $scope);
vv_ai_log('chat_save ' . ($r['ok'] ? 'ok id=' . substr($r['id'], 0, 12)
: 'FAILED: ' . $r['error']));
return $r;
}
if ($action === 'chat_delete') {
if (!$isPost) return $postOnly();
$ok = vv_ai_chat_delete(trim($p['id'] ?? ''));
return ['ok' => $ok, 'error' => $ok ? null : 'No such chat'];
}
// ── bugs / bug_close ──────────────────────────────────────────────────────────
// Reports the troubleshooter filed. Listing is a GET because it changes nothing; dismissing is
// a POST, like every other mutation in this plugin.
if ($action === 'bugs') {
return ['ok' => true, 'bugs' => vv_ai_bugs_list(($p['all'] ?? '') !== '1')];
}
if ($action === 'bug_close') {
if (!$isPost) return $postOnly();
$ok = vv_ai_bug_set_open(trim($p['id'] ?? ''), ($p['open'] ?? '0') === '1');
return ['ok' => $ok];
}
// The report, rendered server-side. Read-only by design: what the operator reviews is byte for
// byte what gets sent, so approving one text and transmitting another is not possible. It is
// also the only renderer — the page used to build its own markdown, which is two formats to
// keep in step and one of them always losing.
if ($action === 'bug_report') {
$id = trim($p['id'] ?? '');
$bug = null;
foreach (vv_ai_bugs_list(false) as $b) if (($b['id'] ?? '') === $id) { $bug = $b; break; }
if (!$bug) return ['ok' => false, 'error' => 'no such report'];
$t = vv_ai_bug_targets();
$title = '[' . ($bug['component'] ?? '?') . '] ' . ($bug['summary'] ?? '');
return [
'ok' => true,
'title' => $title,
'markdown' => vv_ai_bug_report($bug),
'targets' => $t,
// Built here because the repo name lives here. Length is the caller's problem to
// notice: GitHub truncates a very long query rather than refusing it, which would
// silently send a half report — so the page checks and falls back to the copy box.
'github' => 'https://github.com/' . $t['github_repo'] . '/issues/new?title='
. rawurlencode($title) . '&body=' . rawurlencode(vv_ai_bug_report($bug)),
];
}
// Sends to the operator's own Gitea, and only there. Never falls back to GitHub on failure:
// the two destinations are different people, and a silent substitution is how a report meant
// for a private backlog ends up public.
if ($action === 'bug_send_local') {
if (!$isPost) return $postOnly();
$id = trim($p['id'] ?? '');
$bug = null;
foreach (vv_ai_bugs_list(false) as $b) if (($b['id'] ?? '') === $id) { $bug = $b; break; }
if (!$bug) return ['ok' => false, 'error' => 'no such report'];
// Re-rendered from the store rather than taken from the request. The browser showed this
// text read-only; accepting a body from the page would make that guarantee decorative.
$r = vv_ai_bug_send_local('[' . ($bug['component'] ?? '?') . '] ' . ($bug['summary'] ?? ''),
vv_ai_bug_report($bug));
vv_ai_log(sprintf('bug_send_local id=%s %s', $id,
$r['ok'] ? 'ok ' . ($r['url'] ?? '') : 'failed: ' . ($r['error'] ?? '?')));
return $r;
}
// ── findings / finding_action ─────────────────────────────────────────────────
// What the repair sweep found, and the operator's answer to it. include/ai_repair.php is
// pulled in here rather than at the top of the file: it is the largest include in the plugin
// and poll runs once a second per open tab, so it is loaded by the two actions that need it
// and by nothing else.
//
// Neither action is gated on AI_REPAIR_ENABLED. Findings filed while it was on do not stop
// being true when it goes off, and answering them — including saying "this was never a
// problem" — is exactly what an operator turning the feature off is likely to want to do
// first. The gate states are reported instead, so the card can say what is running rather
// than the endpoint pretending the store is empty.
if ($action === 'findings' || $action === 'finding_action') {
require_once __DIR__ . '/ai_repair.php';
if ($action === 'findings') {
// Closed findings are the history — what was dismissed, what a fix actually fixed —
// and they are asked for explicitly rather than shipped with every poll of the list.
$rows = [];
$open = 0; $needs = 0;
foreach (vv_ai_findings_list(($p['all'] ?? '') === '1' ? [] : ['open', 'needs_operator']) as $f) {
$state = (string)($f['state'] ?? 'open');
if ($state === 'open') $open++;
elseif ($state === 'needs_operator') $needs++;
// The three things the page must not decide for itself: which actions this row
// offers, and what its state and kind mean in words.
$f['actions'] = vv_ai_finding_actions($f);
$f['state_label'] = VV_AI_FINDING_STATES[$state] ?? '';
$f['kind_label'] = VV_AI_FINDING_KINDS[(string)($f['kind'] ?? '')] ?? '';
$rows[] = $f;
}
return ['ok' => true,
'repair' => ['enabled' => vv_ai_repair_enabled(),
'autofix' => vv_ai_repair_autofix_enabled(),
'last' => vv_ai_sweep_last()],
'findings' => $rows,
'counts' => ['open' => $open, 'needs_operator' => $needs, 'shown' => count($rows)]];
}
// POST, because fix writes conf through the guarded path and every other answer writes
// state. Which actions are legal for a given row is vv_ai_finding_apply_action()'s call,
// not this endpoint's — a tab left open overnight is holding buttons the store has moved
// past.
if (!$isPost) return $postOnly();
$fid = trim($p['id'] ?? '');
$act = trim($p['act'] ?? '');
$r = vv_ai_finding_apply_action($fid, $act, trim($p['note'] ?? ''));
vv_ai_log(sprintf('finding_action id=%s act=%s %s', $fid, $act,
$r['ok'] ? 'ok' : 'FAILED: ' . ($r['error'] ?? '?')));
return $r;
}
// ── finding_write ─────────────────────────────────────────────────────────────
// A sweep on another node filing what it found. Not reachable from a browser — vv_ai_route()
// never returns LOCAL for it off the owner and the page has no caller — it exists so that
// "sweep local, store central" needs no second store and no reconciliation.
//
// The host is taken from the transport's own view of who connected, never from the payload.
// The record decides which machine a fault is about and is what the finding id hashes on, so
// letting the body name it would let one node file findings as another.
if ($action === 'finding_write') {
if (!$isPost) return $postOnly();
require_once __DIR__ . '/ai_repair.php';
$f = json_decode((string)($p['finding'] ?? ''), true);
if (!is_array($f)) return ['ok' => false, 'error' => 'finding_write: unreadable finding'];
$node = trim((string)($p['_vv_node'] ?? ''));
if (!preg_match('/^host\d+$/', $node)) {
return ['ok' => false, 'error' => 'finding_write: caller did not identify a node'];
}
$f['host'] = $node;
$r = vv_ai_finding_write_local($f);
vv_ai_log(sprintf('finding_write from=%s kind=%s %s', $node, (string)($f['kind'] ?? '?'),
($r['ok'] ?? false) ? 'ok id=' . ($r['id'] ?? '?') : 'FAILED: ' . ($r['error'] ?? '?')));
return $r;
}
// ── incident_add ──────────────────────────────────────────────────────────────
// Appends one operator-written "this was the fix" note against a scope. POST only, and the
// scope is whitelisted the same way ask's is — it is written to a file that later rides in a
// prompt, so it gets the same treatment as anything else that reaches the model.
if ($action === 'incident_add') {
if (!$isPost) return $postOnly();
return vv_ai_incident_add(
trim($p['scope'] ?? ''), trim($p['symptom'] ?? ''), trim($p['fix'] ?? ''));
}
// ── ask ───────────────────────────────────────────────────────────────────────
if ($action === 'ask') {
if (!$isPost) return $postOnly();
$cfg = vv_ai_config();
if ($cfg['model'] === '') {
return ['ok' => false, 'error' => 'No generation model configured'];
}
$question = trim($p['question'] ?? '');
if ($question === '') return ['ok' => false, 'error' => 'question is required'];
if (mb_strlen($question) > VV_AI_MAX_QUESTION) {
return ['ok' => false, 'error' => 'question exceeds ' . VV_AI_MAX_QUESTION . ' characters'];
}
$profile = trim($p['profile'] ?? 'varaverk');
if (!vv_ai_profile_ok($profile)) {
return ['ok' => false, 'error' => 'Unknown profile: ' . $profile];
}
$maxTurns = vv_ai_profile_turns($profile);
// Where the caller is standing — "master.conf", "daily_sync_maintenance.sh", a log name.
// The scheduler page sends it so a question can say "this setting" and mean something;
// the AI tab sends nothing and the worker simply omits the location line.
//
// Whitelisted hard, not escaped and hoped for. It reaches the model as text, so anything
// richer than a file name is an instruction-injection surface for no benefit — a scope is
// only ever a name from this page's own view state.
$scope = trim($p['scope'] ?? '');
if ($scope !== '' && !vv_ai_scope_ok($scope)) $scope = '';
// The retrieval filter only means anything to the profile that retrieves.
$kind = vv_ai_profile_can($profile, 'kind_filter') ? trim($p['kind'] ?? '') : '';
if ($kind !== '' && !in_array($kind, VV_AI_KINDS, true)) {
return ['ok' => false, 'error' => 'Unknown kind: ' . $kind];
}
// Validate per message rather than trusting the blob: a crafted history could otherwise
// inject a system role, or push the context past the offload ceiling.
$clean = [];
$hist = json_decode($p['history'] ?? '[]', true);
if (is_array($hist)) {
foreach ($hist as $m) {
$role = $m['role'] ?? '';
$text = trim((string)($m['content'] ?? ''));
if (!in_array($role, ['user', 'assistant'], true) || $text === '') continue;
$clean[] = ['role' => $role, 'content' => mb_substr($text, 0, VV_AI_MAX_HIST_MSG)];
}
}
if (count($clean) > $maxTurns * 2) {
$clean = array_slice($clean, -($maxTurns * 2));
}
$dir = vv_ai_job_dir();
foreach (glob($dir . '/*.json') ?: [] as $old) {
if (time() - (int)@filemtime($old) > VV_AI_JOB_TTL) @unlink($old);
}
$token = bin2hex(random_bytes(16));
$jobFile = vv_ai_job_path($token);
$worker = dirname(__DIR__) . '/Tools/ai_chat_worker.php';
if (!file_exists($worker)) {
return ['ok' => false, 'error' => 'ai_chat_worker.php not found'];
}
// Not suppressed: if the job file cannot be written the worker has nowhere to report and
// the page polls a token that will never resolve — which looks exactly like a hang.
if (file_put_contents($jobFile, json_encode(['status' => 'pending'])) === false) {
vv_ai_log('ask FAILED — cannot write ' . $jobFile);
return ['ok' => false, 'error' => 'Cannot write job file to ' . VV_AI_JOB_DIR];
}
// setsid, not just nohup. nohup detaches from the terminal but leaves the child in the
// caller's process group — php-fpm's. That is the arrangement that took the WebGUI down
// on 2026-08-07 when a Stop signalled a group it did not own. Stop above signals one
// verified pid and never a group, so this is belt and braces, but it also means a php-fpm
// restart no longer takes a running generation with it.
$cmd = 'setsid nohup php ' . escapeshellarg($worker) . ' '
. escapeshellarg($jobFile) . ' '
. escapeshellarg($question) . ' '
. escapeshellarg(json_encode($clean)) . ' '
. escapeshellarg($kind) . ' '
. escapeshellarg(($p['think'] ?? '1') === '1' ? '1' : '0') . ' '
. escapeshellarg($profile) . ' '
. escapeshellarg($scope) . ' '
// Asked for per turn. Only meaningful on a profile holding web_search — the worker
// checks that, so a crafted web=1 against any other profile changes nothing.
. escapeshellarg(($p['web'] ?? '') === '1' ? '1' : '0')
. ' >/dev/null 2>&1 </dev/null &';
$out = []; $rc = 0;
exec($cmd, $out, $rc);
// Redacted before it is logged, for the same reason the stored transcript is: asking the
// assistant to set a credential means typing one, and ai.log is neither 0600 nor pruned.
vv_ai_log(sprintf('ask token=%s rc=%d profile=%s kind=%s q=%s',
substr($token, 0, 12), $rc, $profile, $kind ?: '-',
mb_substr(vv_ai_redact($question), 0, 80)));
return ['ok' => true, 'token' => $token];
}
return ['ok' => false, 'error' => 'Unknown action'];
}
Plugin/unraid/include/ai_repair.php
+24 -4
View File
@@ -310,9 +310,25 @@ function vv_ai_finding_get(string $id): ?array {
// Files a finding, or increments the one already describing this fault.
//
// Sweeps run on every node — each reads its own logs, which is the only place they exist — but the
// store is the AI owner's, so the operator answers one list instead of one per machine. A mirror
// therefore forwards; the owner writes. The split is here rather than in the sweep so that every
// caller files a finding the same way and none of them has to know where the store lives.
//
// $f expects: kind, subject, conf_key, conf_file, observed, evidence, source_log
// and optionally: proposed, proven, state, note
// and optionally: proposed, proven, state, note, host
function vv_ai_finding_write(array $f): array {
if (vv_ai_is_owner()) return vv_ai_finding_write_local($f);
require_once __DIR__ . '/ai_rpc.php';
$status = 200;
$r = vv_ai_rpc('finding_write', ['finding' => json_encode($f)], true, $status);
// Not swallowed. A sweep that cannot reach the owner has found something and failed to record
// it, and a caller told "ok" would move on and never retry.
return is_array($r) ? $r : ['ok' => false, 'error' => 'finding_write: no response from the AI owner'];
}
function vv_ai_finding_write_local(array $f): array {
$kind = (string)($f['kind'] ?? '');
$subject = trim((string)($f['subject'] ?? ''));
$confKey = trim((string)($f['conf_key'] ?? ''));
@@ -337,13 +353,18 @@ function vv_ai_finding_write(array $f): array {
if (!isset(VV_AI_FINDING_STATES[$state])) $state = 'open';
$now = time();
$id = vv_ai_finding_id($kind, $subject, $ref);
// Which machine this finding is about, resolved once and used for both the hash and the
// record. $f['host'] was accepted by the array below but the id was always hashed locally,
// so a collected partner finding hashed as ours — the exact collision the host-in-the-hash
// comment above exists to prevent.
$host = trim((string)($f['host'] ?? '')) ?: vv_detect_host();
$id = vv_ai_finding_id($kind, $subject, $ref, $host);
$rec = [
'id' => $id,
// Which machine this is about. Written even on a single-host install, because the store
// outlives the topology — a finding filed today is still on disk when the second node
// arrives, and one without a host is a record nobody can place.
'host' => (string)($f['host'] ?? vv_detect_host()),
'host' => $host,
'kind' => $kind,
'subject' => mb_substr($subject, 0, 120),
'conf_key' => $confKey,
@@ -369,7 +390,6 @@ function vv_ai_finding_write(array $f): array {
'severity' => vv_ai_finding_severity(['kind' => $kind, 'conf_key' => $confKey,
'arr_type' => (string)($f['arr_type'] ?? ''),
'sys_level' => (string)($f['sys_level'] ?? '')]),
'host' => vv_detect_host(),
'first' => $now,
'last' => $now,
'seen' => 1,
Plugin/unraid/include/ai_rpc.php
+188
View File
@@ -0,0 +1,188 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Where an AI action runs, and how it gets there. The mesh shares one AI: the owner node holds
// the model, the index and the shared memory, and every other node reaches them over SSH rather
// than keeping a second copy of any of it.
//
// Two exports. vv_ai_route() answers "local, remote, or refused" for one action on this node;
// vv_ai_rpc() carries a remote one to the owner and brings back its answer verbatim.
//
// OPERATIONAL MODEL
// Three routes, decided per action rather than per host:
//
// LOCAL this node answers. Everything on the owner. On a mirror, the chat store only —
// conversations are per-node by design, so they never leave the box they were had on.
// REMOTE forwarded to the owner: generation, retrieval, stats, the shared memory, findings.
// DENY refused with a 404. The curated writes — memory and bug reports — are the owner's.
//
// Transport is SSH over the trust partnership_onboard.sh already establishes, the same as
// node_chat and conf_sync: no listener, no new port, Tailscale-only for free. The request is
// JSON on stdin, the response is JSON on stdout, and Tools/ai_rpc.php on the far side hands both
// to the same vv_ai_dispatch() this node would have called locally.
//
// DESIGN PRINCIPLES
// Share, not copy.
// A mirror does not hold the index, the memory or the model and does not sync them. There
// is one of each, on the owner, and the mesh asks it. Copies of curated, hand-edited state
// are copies that can disagree, and reconciling them needs tombstones and an offline story
// — the same complexity node_chat's local-only delete deliberately refused.
//
// The job lives where the model lives.
// ask returns the owner's token and poll asks the owner about it, so the token-and-poll
// contract is unchanged; it simply resolves on another box. Nothing about the page changes.
//
// Chats stay home, memory is shared.
// A conversation is this operator's, on this node. What the assistant *knows* — the memory
// file, the learned notes, the phrasebook — is the owner's and is shared by everyone. The
// history for a turn travels in the request, so where chats are stored is independent of
// where generation happens.
//
// OPERATIONAL SAFEGUARDS
// The remote path is the WebGUI symlink, not a discovered one.
// /usr/local/emhttp/plugins/varaverk is what Unraid serves the plugin from on every node,
// whatever storage mode it uses. node_chat reads the partner's varaverk.cfg first because a
// delivery is occasional; poll runs once a second per open tab and cannot afford a second
// SSH round trip to find a path. If this symlink is wrong the whole plugin is already
// broken on that node, so it is not a weaker assumption than the one it replaces.
//
// One multiplexed connection, not one per call.
// ControlMaster with ControlPersist, socket in tmpfs. A fresh SSH handshake is 100-300ms;
// paying it per poll, per open tab, would make the assistant feel broken on a mirror.
//
// A transport failure is named, never rendered as an empty success.
// Unreachable owner, missing shim and unparseable output are three different errors and
// each says so. An empty banner that looks like "nothing to report" is the failure mode
// worth spending three messages on.
//
// Nothing here decides trust.
// Possession of the partnership SSH key is the authorization, established at onboard. This
// file routes; it does not authenticate.
//
// EXPORTS
// VV_AI_ROUTE_LOCAL / _REMOTE / _DENY
// vv_ai_route() action → route for this node
// vv_ai_rpc() forward one action to the owner, return its response body
//
// DEPENDS ON
// include/config.php vv_ai_owner_host(), vv_ai_is_owner(), vv_resolve_tailscale_ip()
// Tools/ai_rpc.php the far side — reached at the WebGUI symlink path
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/config.php';
const VV_AI_ROUTE_LOCAL = 0;
const VV_AI_ROUTE_REMOTE = 1;
const VV_AI_ROUTE_DENY = 2;
// Curated on one node on purpose. Memory is the text that rides in every prompt; a bug report is
// what leaves this mesh for a tracker. Both are the owner's to write, and a second node keeping a
// divergent copy of either is the failure this refuses.
const VV_AI_OWNER_ONLY = ['memory_set', 'mem_proposal_action',
'bugs', 'bug_close', 'bug_report', 'bug_send_local'];
// Answered on the node that asked, even on a mirror. A conversation belongs to the operator in
// front of it, and the transcript for a turn travels in the request anyway.
const VV_AI_NODE_LOCAL = ['chats', 'chat_get', 'chat_save', 'chat_delete'];
// Reached over the mesh by a sweep on another node, never by a browser. Denied from the web on
// every node — on a mirror because it is not a page action, and on the owner because the only
// legitimate caller is the RPC shim, which does not come through here.
const VV_AI_RPC_ONLY = ['finding_write'];
function vv_ai_route(string $action): int {
if (in_array($action, VV_AI_RPC_ONLY, true)) return VV_AI_ROUTE_DENY;
if (vv_ai_is_owner()) return VV_AI_ROUTE_LOCAL;
if (in_array($action, VV_AI_NODE_LOCAL, true)) return VV_AI_ROUTE_LOCAL;
if (in_array($action, VV_AI_OWNER_ONLY, true)) return VV_AI_ROUTE_DENY;
return VV_AI_ROUTE_REMOTE;
}
// Where the multiplexed control socket lives. tmpfs is the right lifetime — a reboot should not
// inherit a stale socket — and the path is kept short because a unix socket path is capped near
// 108 characters and ssh composes this one with the user and host appended.
function vv_ai_rpc_socket_dir(): string {
$dir = rtrim(VV_CACHE_ROOT, '/') . '/ssh';
if (!is_dir($dir)) @mkdir($dir, 0700, true);
return $dir;
}
// Forward one action to the AI owner and return its response body.
//
// $httpStatus is set from the owner's own status when it reports one, so a 405 raised over there
// arrives here as a 405 rather than as a 200 carrying an error string.
function vv_ai_rpc(string $action, array $params, bool $isPost, int &$httpStatus = 200): array {
$vars = vv_conf_vars();
$me = strtoupper(vv_detect_host());
$owner = vv_ai_owner_host();
$sshKey = $vars[$me . '_SSH_KEY'] ?? '';
if (!$sshKey || !is_file($sshKey)) {
return ['ok' => false, 'error' => "No SSH key for this node ({$me}_SSH_KEY) — cannot reach the AI owner"];
}
$hostname = trim((string)($vars[strtoupper($owner)] ?? ''));
if ($hostname === '') {
return ['ok' => false, 'error' => "No hostname recorded for the AI owner ($owner)"];
}
$ip = vv_resolve_tailscale_ip($hostname);
if (!$ip) {
return ['ok' => false, 'error' => "Cannot resolve $hostname on the tailnet — the AI owner is unreachable"];
}
// Which node is asking. Not an authorization claim — the SSH key already settled that — but a
// label, so findings and incidents filed from here are stored against the node they describe.
$params['_vv_node'] = strtolower(vv_detect_host());
$remote = '/usr/local/emhttp/plugins/varaverk/Tools/ai_rpc.php';
$sock = vv_ai_rpc_socket_dir() . '/ai-%h';
$cmd = 'ssh -i ' . escapeshellarg($sshKey)
. ' -o BatchMode=yes -o StrictHostKeyChecking=no'
. ' -o ConnectTimeout=8'
. ' -o ControlMaster=auto -o ControlPersist=60s'
. ' -o ControlPath=' . escapeshellarg($sock)
. ' root@' . escapeshellarg($ip)
. ' ' . escapeshellarg('[ -f ' . $remote . ' ] || exit 127; php ' . $remote);
$desc = [0 => ['pipe', 'r'], 1 => ['pipe', 'w'], 2 => ['pipe', 'w']];
$pr = @proc_open($cmd, $desc, $pipes);
if (!is_resource($pr)) {
return ['ok' => false, 'error' => 'Cannot start ssh to the AI owner'];
}
fwrite($pipes[0], json_encode([
'action' => $action,
'params' => $params,
'is_post' => $isPost,
], JSON_UNESCAPED_SLASHES));
fclose($pipes[0]);
$out = stream_get_contents($pipes[1]); fclose($pipes[1]);
$err = stream_get_contents($pipes[2]); fclose($pipes[2]);
$rc = proc_close($pr);
// 127 is the guard above finding no shim — the owner is reachable but has not pulled a build
// that has one. Distinguished from a transport failure because the fix is entirely different.
if ($rc === 127) {
return ['ok' => false, 'error' => 'The AI owner has no Tools/ai_rpc.php — it needs a git pull'];
}
if ($rc !== 0) {
$detail = trim($err) !== '' ? ': ' . mb_substr(trim($err), 0, 200) : '';
return ['ok' => false, 'error' => "Cannot reach the AI owner ($hostname)$detail"];
}
$decoded = json_decode(trim($out), true);
if (!is_array($decoded)) {
return ['ok' => false, 'error' => 'The AI owner returned an unreadable response'];
}
// The shim wraps the body so a status can travel with it. An older owner that answers with a
// bare body still works — it simply carries no status, which is the 200 default.
if (isset($decoded['_vv_rpc'])) {
$httpStatus = (int)($decoded['status'] ?? 200);
return is_array($decoded['body'] ?? null) ? $decoded['body'] : ['ok' => false, 'error' => 'Malformed response from the AI owner'];
}
return $decoded;
}
Plugin/unraid/include/config.php
+19 -7
View File
@@ -586,16 +586,28 @@ function vv_is_ai_host(): bool {
// its definition is missing is worse than no gate. Reads AI_ENABLED directly for the same
// reason. Fail-closed on anything but the literal "true", matching the conf's own contract.
// May this node show AI features — the assistant docks, the findings strips, the AI rows on the
// Monitor card. No longer "am I host1": a node without a GPU borrows the owner's model over the
// mesh, so every node in the mesh gets the assistant. What it does not get is the AI tab; see
// vv_ai_owner_ui_on().
// Monitor card. No longer "am I the owner": the mesh shares one AI, and every node reaches it
// through include/ai_rpc.php. What a mirror does not get is the AI tab; see vv_ai_owner_ui_on().
//
// Still fails closed. A node with no local URL and no owner URL resolves to nothing, and an
// assistant that cannot reach a model is worse than an absent one.
// The two halves are asymmetric on purpose. The owner still fails closed on its own URL, because
// a missing local URL there is a conf error nothing can work around. A mirror does not test
// reachability at all: it is one SSH round trip away from the answer, on a link that goes down
// and comes back, and hiding the entire assistant on a transient blip is worse than showing it
// and reporting the failure in place. vv_ai_rpc() names every transport failure precisely so this
// gate does not have to guess at one.
//
// It lives here rather than in include/ai.php because the pages that need it do not all load that
// file; the Scheduler loads only config.php, and a gate that silently answers false where its
// definition is missing is worse than no gate.
function vv_ai_ui_on(): bool {
if (strtolower(trim(vv_conf_vars()['AI_ENABLED'] ?? 'false')) !== 'true') return false;
$host = strtoupper(vv_ai_model_host());
return trim((string)(vv_conf_vars()[$host . '_OLLAMA_URL'] ?? '')) !== '';
if (vv_ai_is_owner()) {
return trim((string)(vv_conf_vars()[strtoupper(vv_ai_owner_host()) . '_OLLAMA_URL'] ?? '')) !== '';
}
// A mirror needs somewhere to send the request. Without a hostname for the owner there is no
// round trip to attempt, and that is a conf gap rather than a transient one.
return trim((string)(vv_conf_vars()[strtoupper(vv_ai_owner_host())] ?? '')) !== '';
}
// May this node show the AI tab. Owner only, and deliberately so: that page carries the bug