Compare commits

...
10 Commits
Author SHA1 Message Date
Gmer4Lfe f2d4d2fae1 Stop reparsing both confs, and forking, on every conf lookup
vv_conf_vars() read and regex-parsed ~1,900 lines per call and reached a shell fork through
vv_detect_host(); the repair sweep asks it for a key per log line. Keyed on a hash of the
contents rather than mtime and size, which missed a same-second rewrite to the same length.
2026-08-09 19:46:17 -04:00
Gmer4Lfe e91a74d31f Sweep finished runs for misconfigurations instead of waiting to be told
Nothing in Varaverk fires when a job ends, so this picks up run records that completed
since the last pass rather than adding a hook to forty scripts. Ahead of stability in the
cycle on purpose: a wrong port is not fixed by rebooting the machine.
2026-08-09 19:38:02 -04:00
Gmer4Lfe 955f50b94e Let a finding be answered in words, matched rather than judged
The reply to "shall I fix it" ends in a conf write, so the three actions are matched by
pattern with an ambiguity refusal instead of being inferred — and a phrase that supports
two readings asks again rather than picking one.
2026-08-09 19:30:44 -04:00
Gmer4Lfe a8654280a7 Prove a repair before writing it, and gate writing separately from looking
A model can be confident a port should be 8686; a probe can report that 8686 answered, and
only the second is a fact. Detecting and repairing are also separate things to trust, so
the feature runs read-only until the write switch is turned on under it.
2026-08-09 19:23:57 -04:00
Gmer4Lfe 083b5ec5ad Record what is misconfigured on this host, apart from what is broken in Varaverk
A bug stays open until the code changes and nothing here can close it; a finding closes
itself when the probe that proved the fault starts passing, so the two cannot share a
store. Acknowledging one is scoped to the value it was acknowledged at.
2026-08-09 19:07:45 -04:00
Gmer4Lfe 500f9d92c8 Redact credentials on the way into a stored chat
A stored transcript is replayed into a later prompt when reopened, so a key typed while
changing a setting would be handed back to the model on every subsequent turn.
2026-08-09 19:07:34 -04:00
Gmer4Lfe 67eabdc17c Route every conf writer through the guarded path
Eleven call sites wrote master.conf with tmp+rename and nothing else — no backup, no
parse check, no audit — including the two toggles the UI uses most and the raw editor
that installs a whole hand-edited file.
2026-08-09 19:07:28 -04:00
Gmer4Lfe d9f917ecef Give conf writes a backup, a lock and a read-back before they count
These files are gitignored, so a bad write had nothing to go back to; the escaping order
also stored any value containing a quote truncated but still parseable, which only a
read-back can catch.
2026-08-09 19:07:22 -04:00
Gmer4Lfe eb4512aee3 Report a child that exited 1 as a warning, not an error
run_job.sh already grades runs as ok/warn/error by exit code, so a job that exited 1
recorded a warn while its log showed a red failure line — and the log is the louder of
the two.
2026-08-09 19:06:56 -04:00
Gmer4Lfe 0785a46fe3 Judge watchdog strike keys by their container, not the key itself
The state file is keyed both ways — a bare container name for the container check,
container plus suffix for the HTTP, API, CPU and docker checks — so inspecting the key
asked docker about Emby_http and deleted a live counter every cycle, putting the
two-strike limits out of reach.
2026-08-09 18:33:08 -04:00
24 changed files with 1873 additions and 154 deletions
+4 -1
View File
@@ -3,7 +3,10 @@
# Templates (*.template) are safe and remain tracked.
Configurations/host*.conf
Configurations/master.conf
Configurations/*.bak
# *.bak alone missed conf_upgrade's real output — it writes host1.conf.bak-20260802, which does
# not end in .bak — so those sat untracked rather than ignored, one `git add -A` from being
# pushed. The glob has to cover the suffix.
Configurations/*.bak*
.vscode
# ── Personal scratch notes — dev-only, never pushed ───────────────────────────
+25
View File
@@ -299,6 +299,31 @@ Reopened chats render as plain turns: sources, reasoning and timings describe on
are not stored, because redrawing them beside a transcript that may be continued under a
different profile would be citing evidence for an answer no longer being made.
### Secrets are redacted on the way to disk
A conversation about settings is a conversation containing credentials — asking for an API key to
be changed means typing one. Message bodies are redacted in `vv_ai_chat_save()`, and the question
is redacted again before it reaches `ai.log`.
**On the way out, never in flight.** The live turn keeps the real value, because the model needs
it to carry out what was asked. What it does not need is that value still in the transcript a week
later — and a stored chat is replayed into a later prompt when reopened, so an unredacted one
would hand the credential back on every subsequent turn, indefinitely.
Two passes, because they catch different things:
| Pass | Catches | Method |
|---|---|---|
| Known values | a credential this host already holds | exact match against secret-shaped conf keys, longest first |
| Assignment shapes | a credential arriving that is not in the conf yet | `NAME=value`, `"api_key": value`, "set the token to …" |
The second pass is the one that matters for settings changes: *"change the Emby API key to X"* is
a secret arriving, and X matches nothing on disk until after the write it is requesting.
Ordinary prose is left alone — the patterns anchor on a secret-shaped *name*, so `CACHE_WARN_GB=100`
and "turn off the zfs scrub" pass through untouched. `vv_conf_key_is_secret()` is shared with the
conf audit log, so the two cannot disagree about what counts as a secret.
## ━━━ TOKEN ACCOUNTING ━━━
Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`):
+40
View File
@@ -144,6 +144,8 @@
AI_DATA_DIR="${DATA_DIR}/ai"
CACHE_BACKUP_DIR="${DATA_DIR}/cache"
LOG_ARCHIVE_DIR="${DATA_DIR}/logs"
BACKUP_DIR="${DATA_DIR}/Backups"
CONF_BACKUP_DIR="${BACKUP_DIR}/Confs" # 0700 — holds credentials
PERSISTENT_CONF_CACHE="${CACHE_BACKUP_DIR}/conf"
ARR_CACHE_BACKUP_DIR="${CACHE_BACKUP_DIR}/arr"
@@ -401,8 +403,13 @@
"Watchdogs/resource_watchdog.sh" # reduce system pressure before healing attempts
"Watchdogs/docker_watchdog.sh" # heal containers with freed resources
"Watchdogs/system_watchdog.sh" # system component health — storage + webgui
"Plugin/unraid/Tools/ai_repair_sweep.sh" # read what the last cycle logged; off unless AI_REPAIR_ENABLED
"Watchdogs/stability_watchdog.sh" # reboot if all else fails — last line of defense
)
# The repair sweep sits ahead of stability deliberately, which is the one exception to "stability
# last". It reads the previous cycle's logs and may correct the very misconfiguration stability
# would otherwise reboot for — a wrong port is not fixed by restarting the machine. It is bounded
# by AI_PROBE_TIMEOUT, exits 0 in every case, and does nothing at all unless AI_REPAIR_ENABLED.
# ━━━ System Watchdog ━━━
# system_watchdog.sh runs SYSTEM_WATCHDOG_SCRIPTS sequentially each cycle.
@@ -1659,6 +1666,13 @@
# ==============================================================================================
# ── AI / RAG ──────────────────────────────────────────────────────────────────────────────────
# ━━━ Conf Backups ━━━
# Every write through the plugin copies the conf aside first, to CONF_BACKUP_DIR with an
# ISO-8601 stamp. The confs are gitignored, so that directory is the whole recovery path — there
# is no history to revert to. A backup that cannot be taken cancels the write.
# Oldest are pruned past this count, per conf file.
CONF_BACKUP_RETAIN=30
# ==============================================================================================
# Varaverk works exactly as well with AI off as with it on. Nothing below is required for any
# script to function — every feature that can lean on AI has a complete non-AI path, and the
@@ -1755,6 +1769,32 @@
AI_ASSIST_CLEANUP=false # tier 2 — orphan and stuck-import triage
AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance
# ━━━ AI Repair ━━━
# Reads the log of a job that has finished, turns known error shapes into findings, and offers
# or applies a repair. Two switches, because detecting and repairing are separate things to
# trust.
#
# AI_REPAIR_ENABLED on its own reads, files findings and offers fixes, and writes nothing. Run
# it there first — long enough to read what it finds and disagree with some of it.
#
# AI_REPAIR_AUTOFIX_ENABLED is what allows a value to be written unattended, and only ever a
# value a probe has answered on. Never a toggle: whether something should be switched on is a
# decision about intent, and a probe cannot prove intent the way it can prove a port answers.
AI_REPAIR_ENABLED=false
AI_REPAIR_AUTOFIX_ENABLED=false
# ━━━ AI Repair Findings ━━━
# Misconfigurations found in this installation, as opposed to defects in Varaverk — those go to
# ai_bugs. A finding is open until the configuration is right, and closes itself when the probe
# that proved the fault starts passing. Closed ones are kept for a while, because "this happened
# before and here is what fixed it" is worth more than the disk. Open findings are never pruned:
# an unresolved problem does not stop mattering because it is old.
AI_FINDING_RETAIN_DAYS=90
# Seconds a single probe may take. Nothing is written to conf that has not answered a probe, so
# this is the budget for proving a candidate — kept short because a sweep may try several, and
# an address worth switching to answers quickly or is not worth switching to.
AI_PROBE_TIMEOUT=4
# ━━━ AI Conf Write Access ━━━
# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
# never credentials, never a container name. An empty whitelist means no writes regardless of
+1 -1
View File
@@ -162,7 +162,7 @@ if ($profile === 'chat') {
if ($to !== '' && $to !== $profile) {
$profile = $to;
$escalated = true;
wlog('handoff chat -> ' . $to . ': ' . mb_substr($question, 0, 80));
wlog('handoff chat -> ' . $to . ': ' . mb_substr(vv_ai_redact($question), 0, 80));
}
}
+103
View File
@@ -0,0 +1,103 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// One pass of the repair sweep. Reads the logs of jobs that finished since the last pass,
// turns known error shapes into findings, probes for a correction, and either writes a proven
// value or leaves the finding for the operator.
//
// OPERATIONAL MODEL
// Called from ai_repair_sweep.sh, which the watchdog orchestrator runs. Everything it does
// lives in include/ai_repair.php so the same code path is what the tests exercise; this file
// owns only the lock, the log line, and the exit code.
//
// DESIGN PRINCIPLES
// Non-fatal, always.
// Exits 0 on a disabled feature, a held lock, or a sweep that found nothing. Repair is an
// enhancement — the orchestrator that runs it has real work to do either side, and this
// must never be the reason a watchdog cycle reports failure.
//
// The summary is logged, not the reasoning.
// What was scanned, what was found, what was written. The findings themselves are the
// record; duplicating their contents into a log would be two copies to keep in step.
//
// OPERATIONAL SAFEGUARDS
// One sweep at a time.
// flock, non-blocking. A pass that overruns its fifteen-minute slot must not have a second
// copy start probing and writing conf underneath it.
//
// Nothing is written unless two switches say so.
// AI_REPAIR_ENABLED gates the sweep; AI_REPAIR_AUTOFIX_ENABLED gates writing. With only
// the first, this reads and files and changes no configuration at all.
//
// RUNTIME MODES
// ai_repair_sweep.php one pass
// ai_repair_sweep.php --dry-run probe and report, write nothing, leave the marker alone
// ai_repair_sweep.php --status what the last pass did, and what is open
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once dirname(__DIR__) . '/include/ai_repair.php';
$args = array_slice($argv ?? [], 1);
$dryRun = in_array('--dry-run', $args, true);
$status = in_array('--status', $args, true);
function rlog(string $msg): void {
if (!is_dir(LOG_DIR)) return;
@file_put_contents(LOG_DIR . '/ai_repair.log',
date('Y-m-d H:i:s') . ' ' . $msg . "\n", FILE_APPEND | LOCK_EX);
}
if ($status) {
$last = vv_ai_sweep_last();
$open = vv_ai_findings_list(['open', 'needs_operator']);
printf("repair: %s\n", vv_ai_repair_enabled() ? 'enabled' : 'disabled');
printf("autofix: %s\n", vv_ai_repair_autofix_enabled() ? 'enabled' : 'disabled (detect only)');
printf("last pass: %s\n", $last ? date('Y-m-d H:i:s', $last) : 'never');
printf("open findings: %d\n", count($open));
foreach ($open as $f) {
printf(" [%s] %-22s %-28s seen %d — %s\n",
$f['severity'] ?? '?', $f['subject'] ?? '?', $f['conf_key'] ?? '?',
(int)($f['seen'] ?? 0), $f['state'] ?? '?');
}
exit(0);
}
if (!vv_ai_repair_enabled()) exit(0); // silent: a disabled feature is not an event
$lockPath = VV_CACHE_ROOT . '/ai_repair.lock';
if (!is_dir(dirname($lockPath))) @mkdir(dirname($lockPath), 0755, true);
$lock = @fopen($lockPath, 'c');
if ($lock === false || !flock($lock, LOCK_EX | LOCK_NB)) {
if ($lock) fclose($lock);
rlog('skipped — a sweep is already running');
exit(0);
}
try {
$t0 = microtime(true);
$sum = vv_ai_repair_sweep($dryRun);
$ms = (int)round((microtime(true) - $t0) * 1000);
if (!($sum['ok'] ?? false)) {
rlog('sweep refused — ' . ($sum['error'] ?? 'unknown'));
exit(0);
}
// Nothing found and nothing to say. A line every fifteen minutes reporting no news is how a
// log stops being read.
if ($sum['findings'] === 0) {
if ($sum['runs'] > 0 && $dryRun) rlog(sprintf('dry-run: %d run(s), nothing found (%dms)', $sum['runs'], $ms));
exit(0);
}
rlog(sprintf('%s%d run(s): %d finding(s), %d fixed, %d for the operator, %d resolved, %d quiet (%dms)',
$dryRun ? 'dry-run: ' : '', $sum['runs'], $sum['findings'], $sum['fixed'],
$sum['needs_operator'], $sum['resolved'], $sum['quiet'], $ms));
foreach ($sum['details'] as $d) rlog(' ' . $d);
} finally {
flock($lock, LOCK_UN);
fclose($lock);
}
exit(0);
+78
View File
@@ -0,0 +1,78 @@
#!/bin/bash
# ==============================================================================================
# ================================== AI Repair Sweep ===========================================
# ==============================================================================================
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Reads the logs of jobs that finished since the last pass, turns known error shapes into
# findings, probes for a correction, and either writes a proven value or leaves the finding
# for the operator to answer.
# Runs from the watchdog orchestrator. Off unless AI_REPAIR_ENABLED is true.
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
# A one-line shim: exec php on ai_repair_sweep.php in the same directory.
# The logic is PHP because everything it needs already is — the conf writer with its backups
# and read-back verification, the findings store, and the probe layer are all functions the
# WebGUI shares. A bash reimplementation would be a second conf writer, which is precisely the
# drift the guarded write path exists to prevent.
#
# There is no post-run hook in Varaverk; nothing fires when a job finishes. The sweep picks up
# completed run records instead, so this is one entry in an orchestrator list rather than a
# call added to forty scripts.
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
# A Shim, Not a Program
# This file exists only because the scheduler runs shell scripts and the work is PHP.
# Anything added here would be logic the WebGUI cannot reach, and the operator answering a
# finding in the browser must take exactly the same path as the sweep that filed it.
#
# Detecting And Repairing Are Separate Trusts
# AI_REPAIR_ENABLED alone reads logs, files findings and proposes fixes, writing nothing.
# AI_REPAIR_AUTOFIX_ENABLED is what allows a value to be written, and only ever one a probe
# has answered on. Both live in master.conf; neither is set by this script.
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
# Never Fatal
# Always exits 0 — on a disabled feature, a held lock, or a failed pass. The watchdog
# orchestrator runs real work either side of this, and a repair sweep must never be the
# reason a cycle reports failure.
#
# One Sweep At A Time
# The PHP takes a non-blocking flock. A pass that overruns its slot cannot have a second
# copy start probing and writing conf underneath it.
#
# Nothing Is Written That Has Not Answered
# A value reaches conf only after a probe got a response from it. Toggles are never written
# unattended at all — whether something should be switched on is a decision about intent,
# and a probe cannot prove intent.
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# AI_ENABLED master switch; nothing here runs without it
# AI_REPAIR_ENABLED read logs and file findings
# AI_REPAIR_AUTOFIX_ENABLED allow a proven value to be written unattended
# AI_PROBE_TIMEOUT seconds a single probe may take
# AI_FINDING_RETAIN_DAYS how long closed findings are kept
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# ai_repair_sweep.sh
# One pass. Files findings, applies proven fixes if autofix is on.
#
# ai_repair_sweep.sh --dry-run
# Probes and reports what it would do. Writes no conf and does not move the marker, so the
# same runs are examined again next pass.
#
# ai_repair_sweep.sh --status
# Both switches, when the last pass ran, and every open finding.
#
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
php "$SCRIPT_DIR/ai_repair_sweep.php" "$@"
+4 -1
View File
@@ -386,8 +386,11 @@ if ($action === 'ask') {
. ' >/dev/null 2>&1 </dev/null &';
$out = []; $rc = 0;
exec($cmd, $out, $rc);
// Redacted before it is logged, for the same reason the stored transcript is: asking the
// assistant to set a credential means typing one, and ai.log is neither 0600 nor pruned.
vv_ai_log(sprintf('ask token=%s rc=%d profile=%s kind=%s q=%s',
substr($token, 0, 12), $rc, $profile, $kind ?: '-', mb_substr($question, 0, 80)));
substr($token, 0, 12), $rc, $profile, $kind ?: '-',
mb_substr(vv_ai_redact($question), 0, 80)));
echo json_encode(['ok' => true, 'token' => $token]);
exit;
+12 -5
View File
@@ -43,10 +43,17 @@
// outside one, so a matching string in a comment or an unrelated variable cannot be
// rewritten.
//
// The write is atomic.
// vv_conf_toggle_script() writes through vv_write_conf_raw() (tmp + rename). Every
// script sources master.conf; a truncated write here would be a system-wide outage
// rather than a lost toggle.
// The write is atomic, backed up, verified and logged.
// vv_conf_toggle_script() goes through vv_conf_edit(), the one guarded conf write path:
// an exclusive lock, a timestamped copy into CONF_BACKUP_DIR, bash -n on the candidate,
// tmp + rename to install it, then the installed file is sourced to prove it still loads.
// There is no single key to read back for a commented array member, so a clean source is
// the whole assertion. Every script sources master.conf; a truncated or unparseable write
// here would be a system-wide outage rather than a lost toggle.
//
// A script in no array writes nothing at all.
// The rewrite returns the contents unchanged, which reports success without taking a
// backup or touching the file. "Already in the requested state" is not a write.
//
// REQUEST
// POST id=<Category/script.sh> enabled=0|1
@@ -56,7 +63,7 @@
// {"ok":false,"error":"POST only"|"Invalid id"|"Failed to write master.conf"}
//
// DEPENDS ON
// include/scheduler.php vv_conf_toggle_script() → vv_write_conf_raw()
// include/scheduler.php vv_conf_toggle_script() → vv_conf_edit() → vv_write_conf_raw()
// Configurations/master.conf the *_SCRIPTS arrays
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
+21 -23
View File
@@ -38,13 +38,17 @@
// is the difference between a rejected save and a silent, total outage, so a file that
// does not parse is refused and the previous version is left untouched.
//
// The temp copy is created with tempnam() and always removed.
// The candidate is never written next to the real conf and never under a predictable
// name, so a failed validation cannot leave a stray file for a script to source.
// Checked here via vv_conf_syntax_error() only so the editor can show bash's own
// complaint with a line number. vv_conf_edit() checks again before installing; this one
// is for the message, not the decision.
//
// The real write is atomic.
// vv_write_conf_raw() writes .vv.tmp and rename()s, so a script sourcing the conf
// during the save reads either the old file or the new one, never a half-written one.
// The write goes through the one guarded conf path.
// vv_conf_edit() takes an exclusive lock, copies the previous file into CONF_BACKUP_DIR,
// re-checks the syntax, installs via .vv.tmp + rename() so a concurrent reader sees the
// old file or the new one but never a half-written one, then sources the installed file
// to prove it still loads and restores the backup if it does not. The whole-file nature
// of this endpoint is why that matters most here: there is no key to verify, so a clean
// source is the only assertion available.
//
// REQUEST
// POST file=<allowed conf name> content=<full file text>
@@ -54,10 +58,12 @@
// {"ok":false,"error":"POST only"|"File not permitted"|"Syntax error: …"|"Failed to write file"}
//
// DEPENDS ON
// include/config.php vv_get_conf_files(), vv_write_conf_raw(), CONF_DIR
// include/config.php vv_get_conf_files(), CONF_DIR
// include/confform.php vv_conf_syntax_error(), vv_conf_edit()
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
require_once dirname(__DIR__) . '/include/config.php';
require_once dirname(__DIR__) . '/include/confform.php';
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
echo json_encode(['ok' => false, 'error' => 'POST only']);
@@ -75,22 +81,14 @@ if (!$file || !in_array($file, $allowed)) {
}
// Every script sources these. A syntax error here takes the whole system down, so the
// candidate is parsed before it is allowed to replace a working file.
$check = tempnam(sys_get_temp_dir(), 'vvconf');
if ($check !== false) {
file_put_contents($check, $content);
$out = []; $rc = 0;
exec('bash -n ' . escapeshellarg($check) . ' 2>&1', $out, $rc);
@unlink($check);
if ($rc !== 0) {
$msg = implode(' ', array_filter(array_map('trim', $out)));
echo json_encode([
'ok' => false,
'error' => 'Syntax error: ' . str_replace($check, $file, $msg ?: 'conf does not parse'),
]);
exit;
}
// candidate is parsed before it is allowed to replace a working file. Checked here as well as
// inside vv_conf_edit() so the editor can show bash's own complaint; the write path only knows
// whether to proceed, not what to tell the person typing.
$syntax = vv_conf_syntax_error($content, $file);
if ($syntax !== null) {
echo json_encode(['ok' => false, 'error' => 'Syntax error: ' . $syntax]);
exit;
}
$ok = vv_write_conf_raw($file, $content);
$ok = vv_conf_edit($file, fn() => $content, [], ['whole-file']);
echo json_encode(['ok' => $ok, 'error' => $ok ? null : 'Failed to write file']);
+7 -4
View File
@@ -48,9 +48,12 @@
// missing, malformed, or unexpected parameter disables rather than enables. Failing
// toward off is the safe direction for a flag that starts data movement.
//
// The conf write is atomic.
// vv_conf_flag_set() writes through vv_write_conf_raw() (tmp + rename). Every script
// sources master.conf, so a truncated write would be a system-wide outage.
// The conf write is atomic, backed up, verified and logged.
// vv_conf_flag_set() goes through vv_conf_edit(), the one guarded conf write path: an
// exclusive lock, a timestamped copy into CONF_BACKUP_DIR, bash -n on the candidate,
// tmp + rename to install it, then the file is sourced and the flag read back — a value
// that does not come back as asked restores the backup. Every script sources master.conf,
// so a truncated or unparseable write would be a system-wide outage.
//
// The push only happens after a confirmed local write.
// Guarded on $ok, so a failed edit cannot distribute a stale or partly-written conf to
@@ -69,7 +72,7 @@
// "push":[]}
//
// DEPENDS ON
// include/scheduler.php vv_conf_flag_set() → vv_write_conf_raw()
// include/scheduler.php vv_conf_flag_set() → vv_conf_edit() → vv_write_conf_raw()
// include/config.php vv_push_master_conf(), vv_push_setup_state()
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
+12 -1
View File
@@ -81,6 +81,7 @@
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
require_once dirname(__DIR__) . '/include/config.php';
require_once dirname(__DIR__) . '/include/confform.php';
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
echo json_encode(['ok' => false, 'error' => 'POST only']);
@@ -106,6 +107,9 @@ if (!file_exists($confPath)) {
}
$lines = file($confPath, FILE_KEEP_BLANK_LINES);
// Captured before anything is rebuilt from it — this is what the write compares against to
// prove master.conf did not change while the move was being worked out.
$origConf = implode('', $lines ?: []);
if (!$lines) {
echo json_encode(['ok' => false, 'error' => 'Could not read master.conf']);
exit;
@@ -148,7 +152,14 @@ if ($toArray) {
$newLines = $resultLines;
}
if (!vv_write_conf_raw('master.conf', implode('', $newLines))) {
// The array was rebuilt from a copy read before the lock was taken. Handing vv_conf_edit() a
// closure that compares against the current contents turns that into an optimistic check: if
// anything changed master.conf in between, the move is abandoned rather than written over the
// top of it. Everything else — backup, bash -n, read-back, audit — comes with the shared path.
$ok = vv_conf_edit('master.conf', fn(string $cur): ?string =>
$cur === $origConf ? implode('', $newLines) : null, [], [$script]);
if (!$ok) {
echo json_encode(['ok' => false, 'error' => 'Write failed']);
exit;
}
+18 -21
View File
@@ -45,9 +45,13 @@
// The temp copy is created with tempnam() and always removed, so a rejected save cannot
// leave a stray file beside the real conf for a script to source.
//
// The real write is atomic.
// vv_write_conf_raw() writes .vv.tmp and rename()s, so a script sourcing the conf during
// the save reads either the old file or the new one, never a half-written one.
// The write goes through the one guarded conf path.
// vv_conf_edit() takes an exclusive lock, copies the previous file into CONF_BACKUP_DIR,
// re-checks the syntax, installs via .vv.tmp + rename() so a concurrent reader sees the
// old file or the new one but never a half-written one, then sources the installed file
// to prove it still loads and restores the backup if it does not. That backup matters
// more here than anywhere else: this endpoint replaces a whole hand-edited file, and the
// confs are gitignored, so before it existed a bad paste had nothing to go back to.
//
// The push only happens after a confirmed write.
// Guarded on $written, so a failed save cannot distribute a stale or partly-written
@@ -68,11 +72,13 @@
// {"ok":false,"error":"Not allowed"|"Syntax error: …"|"Method not allowed"}
//
// DEPENDS ON
// include/config.php vv_get_conf_files(), vv_read_conf_raw(), vv_write_conf_raw(),
// vv_push_master_conf(), vv_push_setup_state()
// include/config.php vv_get_conf_files(), vv_read_conf_raw(),
// vv_push_master_conf(), vv_push_setup_state()
// include/confform.php vv_conf_syntax_error(), vv_conf_edit()
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
require_once dirname(__DIR__) . '/include/config.php';
require_once dirname(__DIR__) . '/include/confform.php';
$allowed = vv_get_conf_files();
@@ -95,24 +101,15 @@ if ($_SERVER['REQUEST_METHOD'] === 'POST') {
}
// Every script sources these, and master.conf is pushed to every partner from here — a
// syntax error saved through this endpoint would propagate the outage across the mesh.
$check = tempnam(sys_get_temp_dir(), 'vvconf');
if ($check !== false) {
file_put_contents($check, $content);
$out = []; $rc = 0;
exec('bash -n ' . escapeshellarg($check) . ' 2>&1', $out, $rc);
@unlink($check);
if ($rc !== 0) {
$msg = implode(' ', array_filter(array_map('trim', $out)));
echo json_encode([
'ok' => false,
'error' => 'Syntax error: ' . str_replace($check, $file, $msg ?: 'conf does not parse'),
'push' => [],
]);
exit;
}
// Checked here as well as inside vv_conf_edit() so the editor can show bash's own complaint;
// the write path only knows whether to proceed, not what to tell the person typing.
$syntax = vv_conf_syntax_error($content, $file);
if ($syntax !== null) {
echo json_encode(['ok' => false, 'error' => 'Syntax error: ' . $syntax, 'push' => []]);
exit;
}
$written = vv_write_conf_raw($file, $content);
$written = vv_conf_edit($file, fn() => $content, [], ['whole-file']);
$push = [];
if ($written && $file === 'master.conf') {
$push = vv_push_master_conf();
+11 -1
View File
@@ -82,6 +82,7 @@
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
require_once dirname(__DIR__) . '/include/config.php';
require_once dirname(__DIR__) . '/include/confform.php';
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
echo json_encode(['ok' => false, 'error' => 'POST only']);
@@ -122,6 +123,9 @@ if (!file_exists($confPath)) {
}
$lines = file($confPath, FILE_KEEP_BLANK_LINES);
// Captured before array_splice rewrites the block in place — this is what the write compares
// against to prove master.conf did not change while the new order was being assembled.
$origConf = implode('', $lines ?: []);
if (!$lines) {
echo json_encode(['ok' => false, 'error' => 'Could not read master.conf']);
exit;
@@ -183,7 +187,13 @@ $newBlockLines[] = $lines[$blockEnd];
// Replace the original block in $lines
array_splice($lines, $blockStart, $blockEnd - $blockStart + 1, $newBlockLines);
if (!vv_write_conf_raw('master.conf', implode('', $lines))) {
// See movescript.php: the block was rebuilt from a copy read before the lock, so the closure
// compares against the current contents and abandons the reorder if anything changed in
// between. Backup, bash -n, read-back and audit come with the shared path.
$ok = vv_conf_edit('master.conf', fn(string $cur): ?string =>
$cur === $origConf ? implode('', $lines) : null, [], [$arrayName]);
if (!$ok) {
echo json_encode(['ok' => false, 'error' => 'Write failed']);
exit;
}
+10 -3
View File
@@ -167,6 +167,9 @@ if ($action === 'save' && $_SERVER['REQUEST_METHOD'] === 'POST') {
if (is_array($scripts)) {
$confPath = CONF_DIR . '/master.conf';
$lines = file($confPath, FILE_KEEP_BLANK_LINES) ?: [];
// Captured before array_splice rewrites the block in place — the write compares against
// it to prove master.conf did not change while the new block was being assembled.
$origConf = implode('', $lines);
$esc = preg_quote($scriptsVar, '/');
$blockStart = $blockEnd = null;
$depth = 0;
@@ -213,9 +216,13 @@ if ($action === 'save' && $_SERVER['REQUEST_METHOD'] === 'POST') {
}
$newBlock[] = $lines[$blockEnd];
array_splice($lines, $blockStart, $blockEnd - $blockStart + 1, $newBlock);
// tmp+rename — every script sources master.conf, so a truncated write here is a
// system-wide outage, not a lost edit.
if (!vv_write_conf_raw('master.conf', implode('', $lines))) {
// Shared guarded path: lock, backup, bash -n, atomic install, read-back, audit. The
// closure compares against the current contents first, so a master.conf that changed
// while this block was being assembled abandons the write instead of clobbering it.
$wrote = vv_conf_edit('master.conf', fn(string $cur): ?string =>
$cur === $origConf ? implode('', $lines) : null, [], [$scriptsVar]);
if (!$wrote) {
$errors[] = 'scripts write failed';
} else {
// master.conf is shared — mirrors reorderarray/movescript/rawconf.
+12 -3
View File
@@ -120,6 +120,7 @@
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
require_once dirname(__DIR__) . '/include/config.php';
require_once dirname(__DIR__) . '/include/confform.php';
$action = ($_SERVER['REQUEST_METHOD'] === 'GET')
? trim($_GET['action'] ?? '')
@@ -277,7 +278,10 @@ if ($action === 'pull') {
'${1}"' . $sshKey . '"', $conf);
$conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
'${1}' . $storageInternal2, $conf);
vv_write_conf_raw($confFile, $conf);
// allowCreate: this is first-run setup, so the host conf does not exist yet. There
// is no prior content to back up, and a candidate that fails bash -n is removed
// rather than restored.
vv_conf_edit($confFile, fn(): string => $conf, [], ["{$hostId}_SSH_KEY"], true);
}
}
@@ -313,6 +317,9 @@ if (!preg_match('/^host\d+$/', $mySlot)) {
// Write HOST1 / HOST2 into master.conf
$master = vv_read_conf_raw('master.conf');
// Captured before the substitutions below — the write compares against it so a master.conf that
// changed while setup was being filled in is not silently overwritten.
$origMaster = $master;
if ($master === '') {
echo json_encode(['ok' => false, 'error' => 'master.conf not found — check SCRIPTS_DIR in varaverk.cfg']);
exit;
@@ -333,7 +340,8 @@ if ($slotNum > 2 && !empty($myHostname)) {
}
}
if (!vv_write_conf_raw('master.conf', $master)) {
if (!vv_conf_edit('master.conf', fn(string $cur): ?string => $cur === $origMaster ? $master : null,
[], ['HOST1', 'HOST2'])) {
echo json_encode(['ok' => false, 'error' => 'Failed to write master.conf']);
exit;
}
@@ -368,7 +376,8 @@ if (!file_exists(CONF_DIR . '/' . $confFile)) {
'${1}"' . $sshKeyPath . '"', $conf);
$conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
'${1}' . $storageInternal, $conf);
if (!vv_write_conf_raw($confFile, $conf)) {
// allowCreate — see the sibling write above; this is the same first-run create.
if (!vv_conf_edit($confFile, fn(): string => $conf, [], ["{$hostId}_SSH_KEY"], true)) {
echo json_encode(['ok' => false, 'error' => "Failed to write $confFile"]);
exit;
}
+87
View File
@@ -78,6 +78,9 @@
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/config.php';
// vv_conf_key_is_secret() — the same definition of "this key holds a credential" that decides
// what the conf audit log redacts. One list, so the two cannot disagree about what a secret is.
require_once __DIR__ . '/confform.php';
// VV_AI_JOB_DIR and VV_AI_TOKEN_CACHE_DIR are defined in config.php with every other cache path,
// read from master.conf so this layer and load_config.sh cannot disagree about where they are.
@@ -1216,6 +1219,84 @@ function vv_ai_chats_dir(): string {
return $d;
}
// ── Secret redaction ─────────────────────────────────────────────────────────────────────────
// A conversation about settings is a conversation that contains credentials. Asking the
// assistant to change an API key means typing the key, and a stored transcript is replayed into
// a later prompt when the thread is reopened — so without this a secret would be written to
// disk in cleartext and then handed back to the model on every subsequent turn, indefinitely.
//
// Redaction happens on the way to disk and to the log, never to the live turn. The model needs
// the real value to carry out the change being asked for; what it does not need is that value
// still sitting in the transcript a week later.
const VV_AI_REDACTED = '[redacted]';
// The values this host actually holds, longest first so a key that contains another as a
// substring cannot be half-replaced. Only secret-shaped conf keys contribute, and only values
// long enough to be a credential — redacting every occurrence of a two-character setting would
// shred ordinary prose.
function vv_ai_known_secrets(): array {
static $cache = null;
if ($cache !== null) return $cache;
$vals = [];
foreach (vv_conf_vars() as $k => $v) {
$v = trim((string)$v);
if (strlen($v) < 8) continue;
if (!vv_conf_key_is_secret((string)$k)) continue;
// A value that is still a ${...} reference is a template, not a credential.
if (str_contains($v, '${')) continue;
$vals[] = $v;
}
$vals = array_values(array_unique($vals));
usort($vals, fn($a, $b) => strlen($b) <=> strlen($a));
return $cache = $vals;
}
// Two passes, because they catch different things.
//
// Known values catch a credential this host already holds, however it is worded — pasted bare,
// quoted, or buried mid-sentence. Exact string matching, so there are no false positives.
//
// Assignment shapes catch the credential that is not in the conf yet, which is precisely the
// case that matters here: "change the Emby API key to <new value>" is a secret arriving, and it
// will not match anything on disk until after the write it is asking for.
function vv_ai_redact(string $text): string {
if ($text === '') return $text;
foreach (vv_ai_known_secrets() as $secret) {
$text = str_replace($secret, VV_AI_REDACTED, $text);
}
// NAME=value / "api_key": value / api-key: value — anchored on a secret-shaped name so an
// ordinary "count=12" is untouched.
$text = preg_replace(
'/\b([A-Za-z0-9_\-]*(?:api[_\-]?key|password|passwd|secret|token|_pass|apikey)[A-Za-z0-9_\-]*)'
. '(\s*["\']?\s*[:=]\s*["\']?)([^\s"\',;]{6,})/i',
'$1$2' . VV_AI_REDACTED,
$text
) ?? $text;
// "set the api key to <value>", "password is <value>" — the same secret arriving as prose
// rather than as an assignment.
$text = preg_replace(
'/\b((?:api[ _\-]?key|password|passphrase|secret|token)\s+(?:to|is|as|=)\s+)["\']?([^\s"\',;]{6,})/i',
'$1' . VV_AI_REDACTED,
$text
) ?? $text;
return $text;
}
// Redacts every message body in place, leaving roles and any other metadata alone.
function vv_ai_redact_messages(array $messages): array {
foreach ($messages as &$m) {
if (isset($m['content']) && is_string($m['content'])) $m['content'] = vv_ai_redact($m['content']);
}
unset($m);
return $messages;
}
function vv_ai_chats_max(): int {
$n = (int)(vv_conf_vars()['AI_CHAT_HISTORY_MAX'] ?? 10);
return max(1, min(50, $n));
@@ -1308,6 +1389,12 @@ function vv_ai_chat_save(string $id, string $profile, array $messages, string $s
$prev = vv_ai_chat_read($id);
$created = (int)($prev['created'] ?? time());
// Redacted here rather than at the point the message was composed, because the live turn
// needs the real value to carry out what was asked. This is the boundary between "in flight"
// and "on disk", and it is the last place the cleartext exists. The title is derived after,
// so a credential cannot survive in the conversation list either.
$messages = vv_ai_redact_messages($messages);
// Scope travels with the conversation. A Scheduler thread is bound to what the operator had
// open — a script, a log, a conf key — and its turns carry log excerpts chosen for that
// thing. Storing the scope means reopening the thread anywhere restores the context it was
+907
View File
@@ -0,0 +1,907 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Findings about this installation being misconfigured, and the record of what was done about
// them. A finding is "Emby is not answering at the address the conf gives", not "Varaverk has
// a bug" — the second is what ai_bugs holds.
//
// OPERATIONAL MODEL
// A sweep reads a completed run's log, deterministic patterns turn error lines into typed
// findings, and each finding is either repaired or handed to the operator. Findings persist
// because the repair may need something only a human can supply, and that conversation has to
// survive the page being closed.
//
// WHY THIS IS NOT ai_bugs
// Same storage shape, different lifecycle, and the difference is the whole reason for a second
// store. A bug is open until Varaverk's code changes; nothing on this host can close it. A
// finding is open until this host's configuration is right, and the same probe that proved a
// fix can later prove the fault is gone — so findings close themselves and bugs cannot.
//
// Filing them together would mean a list where half the rows are actionable by the operator
// and half are actionable by whoever maintains the project, with no way to tell which is which
// except by reading them.
//
// DESIGN PRINCIPLES
// A finding names a conf key or it is not a finding.
// The point of the record is that something can be done about it. "The daily sync looked
// unhappy" is a feeling; "HOST1_EMBY_URL points at a host that refuses connections" is a
// finding. The triage patterns that cannot resolve a key produce nothing rather than a
// vague row.
//
// Evidence is the log line, quoted.
// Same rule as ai_bugs, for the same reason: a finding that cannot show the line it came
// from cannot be checked, and this store is meant to be checkable.
//
// Identity is kind + subject + key, not the message text.
// A port that has been wrong for a week is one finding seen 400 times, not 400 findings.
// Wording drifts as logs change; the thing being wrong does not.
//
// OPERATIONAL SAFEGUARDS
// A proposed value is recorded, never trusted.
// 'proposed' is what something might be changed to; 'proven' is whether a probe actually
// got an answer from it. Only proven values are ever written to conf, and the two fields
// are kept separate so a record cannot imply verification it did not have.
//
// Closing is evidence-driven, not time-driven.
// A finding closes when its probe passes or the operator dismisses it. It does not expire,
// because "we stopped seeing it in the log" is equally consistent with the job no longer
// running at all.
//
// Under data/ and therefore gitignored: these quote this installation's logs and name its
// hosts, ports and containers.
//
// EXPORTS
// vv_ai_findings_dir() the store
// vv_ai_finding_write() file or increment one finding
// vv_ai_findings_list() findings, newest activity first
// vv_ai_finding_get() one by id
// vv_ai_finding_close() mark resolved, with how
// vv_ai_finding_dismiss() operator says this is not a problem
// vv_ai_findings_for_chat() the open ones worth opening a conversation about
//
// CONFIGURATION
// AI_DATA_DIR findings live in ai_findings/ beneath it
// AI_FINDING_RETAIN_DAYS closed findings older than this are removed (default 90)
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/config.php';
require_once __DIR__ . '/ai.php';
// What a finding can be. The kind decides which repair is even conceivable, so an unknown kind
// is refused rather than stored as an untyped row nothing knows how to act on.
const VV_AI_FINDING_KINDS = [
'unreachable' => 'a configured address or port refused, timed out, or did not resolve',
'auth_rejected' => 'the endpoint answered, and rejected the credential',
'unknown_target' => 'a conf entry names a container or share that does not exist here',
'missing_value' => 'a conf key required by the job that ran is empty',
];
// How a finding ended, when it ends.
const VV_AI_FINDING_STATES = [
'open' => 'seen, not yet acted on',
'needs_operator' => 'cannot be repaired here — the value is not derivable from this host',
'acknowledged' => 'the operator knows, and it stays quiet until the state it was acked at changes',
'fixed' => 'a proven value was written to conf',
'resolved' => 'the probe now passes; whatever was wrong is no longer wrong',
'dismissed' => 'the operator says this is not a problem, permanently',
];
// ── Why acknowledged is not dismissed ────────────────────────────────────────────────────────
// "I know critical rsync is off, stop telling me" and "this is never a problem" are different
// instructions, and collapsing them loses the half that matters. An acknowledgement is scoped to
// the state it was given in: CRITICAL_RSYNC_ENABLED being false is a deliberate choice today and
// a stale note the moment it goes true again.
//
// So an ack records what the key read when it was given, and expires when that changes. The
// finding comes back on its own, without the operator having to remember to look — which is the
// difference between a note and a silence.
// ── Toggles are the operator's, always ───────────────────────────────────────────────────────
// A repair may never enable or disable anything on its own. Not because it would get the value
// wrong — a boolean has only two — but because the value is not a fact to be discovered. Whether
// critical rsync should be on is a decision about intent, and a probe cannot prove intent the
// way it can prove that a port answers.
//
// The Fix action still writes it when the operator asks for it. What is forbidden is the
// unattended path choosing for them.
function vv_ai_conf_is_toggle(string $key): bool {
$v = strtolower(trim((string)(vv_conf_vars()[$key] ?? '')));
return $v === 'true' || $v === 'false';
}
// May the sweep write this without being asked? Two conditions, both required: a probe actually
// answered on the proposed value, and the key is not a toggle.
function vv_ai_finding_may_autofix(array $f): bool {
if (!vv_ai_repair_autofix_enabled()) return false;
if (empty($f['proven'])) return false;
if (($f['proposed'] ?? null) === null) return false;
if (vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))) return false;
return true;
}
// ── Severity is derived, never supplied ──────────────────────────────────────────────────────
// Same ladder run_job.sh records runs against — ok / warn / error — so a finding and the run it
// came from cannot describe the same event at two different volumes.
//
// The rule that matters: a finding whose key is a toggle can never be an error. Something not
// happening because it was switched off is the switch working. That is true whether the switch
// was flipped deliberately last month or by accident this morning, and the store cannot tell
// those apart — so it reports the fact and lets the operator supply the intent.
//
// Everything else takes its level from what the fault costs. A credential the endpoint rejected
// stops that integration dead; an address that does not answer might be a host still booting.
function vv_ai_finding_severity(array $f): string {
$key = (string)($f['conf_key'] ?? '');
// Deliberate-state findings never escalate, whatever their kind.
if (vv_ai_conf_is_toggle($key)) return 'warn';
return match ($f['kind'] ?? '') {
'auth_rejected' => 'error', // answered and refused — nothing gets through until fixed
'missing_value' => 'error', // configured to use something that was never supplied
'unreachable' => 'warn', // may be transient; the strike system is what escalates it
'unknown_target' => 'warn',
default => 'warn',
};
}
// ── Gates ────────────────────────────────────────────────────────────────────────────────────
// Two switches, because detecting and repairing are separate things to trust.
//
// AI_REPAIR_ENABLED alone gives a system that reads logs, files findings and offers fixes, and
// writes nothing. That is the state this should live in first — long enough to read what it
// found and disagree with some of it. A subsystem that starts by editing conf has to be believed
// before there is any evidence for believing it.
//
// AI_REPAIR_AUTOFIX_ENABLED is what lets a proven value be written without being asked, and it
// is meaningless on its own: nothing to write if nothing is looking. Both must be true, in the
// same layered way AI_ENABLED is necessary but never sufficient.
function vv_ai_repair_enabled(): bool {
if (!vv_ai_config()['enabled']) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_ENABLED'] ?? 'false'))) === 'true';
}
function vv_ai_repair_autofix_enabled(): bool {
if (!vv_ai_repair_enabled()) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_AUTOFIX_ENABLED'] ?? 'false'))) === 'true';
}
function vv_ai_findings_dir(): string {
$d = AI_DATA_DIR . '/ai_findings';
if (!is_dir($d)) @mkdir($d, 0755, true);
return $d;
}
function vv_ai_finding_retain_days(): int {
$n = (int)(vv_conf_vars()['AI_FINDING_RETAIN_DAYS'] ?? 90);
return max(1, $n);
}
// kind + subject + conf key. Deliberately not the message: the same wrong port produces slightly
// different log text as the software around it changes, and that must not mint a second record.
function vv_ai_finding_id(string $kind, string $subject, string $confKey): string {
return substr(sha1(strtolower($kind . '|' . $subject . '|' . $confKey)), 0, 12);
}
function vv_ai_finding_path(string $id): ?string {
if (!preg_match('/^[0-9a-f]{12}$/', $id)) return null;
return vv_ai_findings_dir() . '/' . $id . '.json';
}
function vv_ai_finding_get(string $id): ?array {
$p = vv_ai_finding_path($id);
if ($p === null || !is_file($p)) return null;
$r = json_decode((string)@file_get_contents($p), true);
return is_array($r) ? $r : null;
}
// Files a finding, or increments the one already describing this fault.
//
// $f expects: kind, subject, conf_key, conf_file, observed, evidence, source_log
// and optionally: proposed, proven, state, note
function vv_ai_finding_write(array $f): array {
$kind = (string)($f['kind'] ?? '');
$subject = trim((string)($f['subject'] ?? ''));
$confKey = trim((string)($f['conf_key'] ?? ''));
$evidence = trim((string)($f['evidence'] ?? ''));
if (!isset(VV_AI_FINDING_KINDS[$kind])) return ['ok' => false, 'error' => 'unknown kind'];
if ($subject === '' || $confKey === '') return ['ok' => false, 'error' => 'subject and conf_key required'];
if ($evidence === '') return ['ok' => false, 'error' => 'evidence required'];
// The key has to be a real shell identifier for the same reason the conf writer insists on
// it: a finding is a proposal to edit that key, and a malformed one can never be actioned.
if (!vv_conf_key_valid($confKey)) return ['ok' => false, 'error' => 'malformed conf key'];
$state = (string)($f['state'] ?? 'open');
if (!isset(VV_AI_FINDING_STATES[$state])) $state = 'open';
$now = time();
$id = vv_ai_finding_id($kind, $subject, $confKey);
$rec = [
'id' => $id,
'kind' => $kind,
'subject' => mb_substr($subject, 0, 120),
'conf_key' => $confKey,
'conf_file' => (string)($f['conf_file'] ?? 'master.conf'),
// Secrets never enter this store. A finding about a rejected API key is about the key
// being wrong, and the wrong value is of no use to anyone reading the record later.
'observed' => vv_conf_key_is_secret($confKey) ? '<redacted>'
: mb_substr((string)($f['observed'] ?? ''), 0, 300),
'proposed' => isset($f['proposed']) && !vv_conf_key_is_secret($confKey)
? mb_substr((string)$f['proposed'], 0, 300) : null,
'proven' => (bool)($f['proven'] ?? false),
'state' => $state,
'evidence' => mb_substr(vv_ai_redact($evidence), 0, 1200),
'source_log' => mb_substr((string)($f['source_log'] ?? ''), 0, 200),
'note' => mb_substr((string)($f['note'] ?? ''), 0, 1000),
// Recomputed on every sighting rather than stored once: a key that becomes a toggle, or
// a toggle that is replaced by a real value, changes what this finding means.
'severity' => vv_ai_finding_severity(['kind' => $kind, 'conf_key' => $confKey]),
'host' => vv_detect_host(),
'first' => $now,
'last' => $now,
'seen' => 1,
'closed_at' => null,
// What the key read when the operator acknowledged it. Null unless acked; the ack
// expires the moment the live value stops matching this.
'ack_value' => null,
];
$p = vv_ai_finding_path($id);
if ($p === null) return ['ok' => false, 'error' => 'bad id'];
if (is_file($p)) {
$old = json_decode((string)@file_get_contents($p), true);
if (is_array($old)) {
$rec['first'] = $old['first'] ?? $now;
$rec['seen'] = (int)($old['seen'] ?? 0) + 1;
// A dismissed finding stays dismissed however many times the log repeats it —
// otherwise "this is fine, stop telling me" lasts exactly one cycle. A fixed one
// reopens, because seeing the fault again after a repair means the repair did not
// hold, which is the single most important thing this store can tell anyone.
if (($old['state'] ?? '') === 'dismissed') {
$rec['state'] = 'dismissed';
$rec['closed_at'] = $old['closed_at'] ?? null;
}
// An acknowledgement holds only while the thing acknowledged is still true. Compare
// the live value against what it read when the ack was given: unchanged means stay
// quiet, changed means the note is stale and the finding comes back by itself.
if (($old['state'] ?? '') === 'acknowledged') {
$ackedAt = (string)($old['ack_value'] ?? '');
if ($ackedAt === (string)(vv_conf_vars()[$confKey] ?? '')) {
$rec['state'] = 'acknowledged';
$rec['ack_value'] = $ackedAt;
$rec['closed_at'] = $old['closed_at'] ?? null;
}
// Otherwise $rec keeps the state this sighting computed — it has reopened.
}
// Preserve an operator's note over a generated one.
if ($rec['note'] === '' && !empty($old['note'])) $rec['note'] = $old['note'];
}
}
if (@file_put_contents($p, json_encode($rec, JSON_PRETTY_PRINT)) === false) {
return ['ok' => false, 'error' => 'write failed'];
}
return ['ok' => true, 'id' => $id, 'seen' => $rec['seen'], 'state' => $rec['state']];
}
// $states filters; empty means everything. Newest activity first, because a finding seen in the
// last cycle matters more than one that has been sitting fixed for a month.
function vv_ai_findings_list(array $states = ['open', 'needs_operator']): array {
$out = [];
foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) {
$r = json_decode((string)@file_get_contents($file), true);
if (!is_array($r)) continue;
if ($states && !in_array($r['state'] ?? 'open', $states, true)) continue;
$out[] = $r;
}
usort($out, fn($a, $b) => ($b['last'] ?? 0) <=> ($a['last'] ?? 0));
return $out;
}
function vv_ai_finding_set_state(string $id, string $state, string $note = ''): bool {
if (!isset(VV_AI_FINDING_STATES[$state])) return false;
$r = vv_ai_finding_get($id);
if ($r === null) return false;
$r['state'] = $state;
$r['closed_at'] = in_array($state, ['open', 'needs_operator'], true) ? null : time();
if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000);
$p = vv_ai_finding_path($id);
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
}
function vv_ai_finding_close(string $id, string $note = ''): bool {
return vv_ai_finding_set_state($id, 'resolved', $note);
}
function vv_ai_finding_dismiss(string $id, string $note = ''): bool {
return vv_ai_finding_set_state($id, 'dismissed', $note);
}
// "I know about this — leave it, and tell me if it changes."
//
// Stamps the key's current value onto the record. Every later sighting compares against that
// stamp, so the acknowledgement covers this state and not the key forever. Acking that critical
// rsync is off says nothing about critical rsync being on.
function vv_ai_finding_ack(string $id, string $note = ''): bool {
$r = vv_ai_finding_get($id);
if ($r === null) return false;
$r['state'] = 'acknowledged';
$r['ack_value'] = (string)(vv_conf_vars()[$r['conf_key'] ?? ''] ?? '');
$r['closed_at'] = time();
if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000);
$p = vv_ai_finding_path($id);
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
}
// What the operator can do about a finding, and what each choice means. Returned rather than
// hardcoded in the UI so the chat and the page cannot offer different options for the same row.
//
// Fix appears for anything with a proposed value, toggle or not — the prohibition is on the
// sweep choosing, never on the operator choosing. Everything carries ack and cancel, because
// "I know" and "not now" are always valid answers to being told something.
function vv_ai_finding_actions(array $f): array {
$actions = [];
if (($f['proposed'] ?? null) !== null) {
$actions['fix'] = vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))
? 'Set ' . $f['conf_key'] . ' — a toggle, so this only ever happens because you asked'
: 'Write the proven value to ' . $f['conf_key'];
}
$actions['ack'] = 'Known and intended. Stays quiet until ' . ($f['conf_key'] ?? 'it') . ' changes';
$actions['cancel'] = 'Leave it alone for now';
return $actions;
}
// Closed findings are kept for a while because "this happened before and here is what fixed it"
// is worth more than the disk it costs. Open ones are never pruned — an unresolved problem does
// not stop mattering because it is old.
function vv_ai_findings_prune(): int {
$cutoff = time() - (vv_ai_finding_retain_days() * 86400);
$n = 0;
foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) {
$r = json_decode((string)@file_get_contents($file), true);
if (!is_array($r)) continue;
if (in_array($r['state'] ?? 'open', ['open', 'needs_operator'], true)) continue;
if ((int)($r['closed_at'] ?? 0) > $cutoff) continue;
if (@unlink($file)) $n++;
}
return $n;
}
// What the assistant should raise when a page loads: things that need the operator, newest
// first. Repaired findings are deliberately not here — a fix that worked is a log entry, not a
// conversation, and opening every session with a list of things that already went right is how
// an operator learns to close the panel without reading it.
function vv_ai_findings_for_chat(int $limit = 3): array {
return array_slice(vv_ai_findings_list(['needs_operator']), 0, max(1, $limit));
}
// ── The sweep ────────────────────────────────────────────────────────────────────────────────
// There is no post-run hook in Varaverk — nothing fires when a job finishes. Rather than add a
// call to forty scripts, this picks up run records that completed since the last pass. One entry
// in an orchestrator's list instead of forty edits, and it batches naturally.
//
// Runs that reported ok are read too. A container failing its HTTP check warns and leaves the
// watchdog exiting 0, so "only look at failures" would miss the whole class of fault this exists
// for: the job worked, and told you something is wrong.
function vv_ai_sweep_marker_path(): string {
return STATE_DIR . '/ai_repair_sweep.db';
}
function vv_ai_sweep_last(): int {
return (int)trim((string)@file_get_contents(vv_ai_sweep_marker_path()));
}
function vv_ai_sweep_mark(int $ts): void {
if (!is_dir(STATE_DIR)) @mkdir(STATE_DIR, 0755, true);
@file_put_contents(vv_ai_sweep_marker_path(), (string)$ts, LOCK_EX);
}
// Run records that finished after $since. A record still marked running is skipped rather than
// read half-written — it will be picked up on the pass after it finishes.
function vv_ai_recent_runs(int $since): array {
$out = [];
$base = realpath(LOG_DIR);
if ($base === false) return [];
foreach ((array)@glob($base . '/{,*/,*/*/,*/*/*/}*.json', GLOB_BRACE) as $path) {
$r = json_decode((string)@file_get_contents($path), true);
if (!is_array($r) || empty($r['id']) || ($r['status'] ?? '') === 'running') continue;
$end = (int)($r['end'] ?? 0);
if ($end <= $since) continue;
$log = preg_replace('/\.json$/', '.log', $path);
if (!is_file($log)) continue;
$out[] = ['id' => (string)$r['id'], 'status' => (string)($r['status'] ?? '?'),
'start' => (int)($r['start'] ?? 0), 'end' => $end, 'log' => $log];
}
usort($out, fn($a, $b) => $a['end'] <=> $b['end']);
return $out;
}
// The lines one run wrote, and only those. Logs are appended across runs, so a tail alone would
// re-read the previous run's output and re-report faults that have already been dealt with.
// Filtering on the leading timestamp scopes the evidence to the run being examined.
function vv_ai_run_log_lines(string $logPath, int $startTs, int $maxLines = 2000): array {
$out = []; $rc = 0;
@exec('tail -n ' . (int)$maxLines . ' ' . escapeshellarg($logPath) . ' 2>/dev/null', $out, $rc);
if ($rc !== 0) return [];
$kept = [];
foreach ($out as $line) {
if (preg_match('/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/', $line, $m)) {
// A line older than the run belongs to a previous one. Two seconds of slack because
// the record's start is stamped by the runner, not by the first line the job writes.
if (strtotime($m[1]) < $startTs - 2) continue;
}
$kept[] = $line;
}
return $kept;
}
// One pass. Returns a summary rather than logging it, so the caller decides what to record and
// the whole thing stays testable without a log to read afterwards.
//
// $dryRun does everything except write conf and move the marker — including probing, which is
// the point: it answers "what would this have done" with real evidence rather than a guess.
function vv_ai_repair_sweep(bool $dryRun = false): array {
if (!vv_ai_repair_enabled()) {
return ['ok' => false, 'error' => 'AI_REPAIR_ENABLED is not true', 'runs' => 0];
}
$started = time();
$since = vv_ai_sweep_last();
$runs = vv_ai_recent_runs($since);
$sum = ['ok' => true, 'runs' => count($runs), 'findings' => 0, 'fixed' => 0,
'needs_operator' => 0, 'resolved' => 0, 'quiet' => 0, 'details' => []];
foreach ($runs as $run) {
$lines = vv_ai_run_log_lines($run['log'], $run['start']);
if (!$lines) continue;
$rel = ltrim(str_replace(realpath(LOG_DIR), '', $run['log']), '/');
foreach (vv_ai_triage_log($lines, $rel) as $cand) {
$cand = vv_ai_probe_finding($cand);
$sum['findings']++;
// Write first, so a finding exists even if the repair below fails. A repair that
// errored without leaving a record is the one failure mode there is no way back from.
$w = vv_ai_finding_write($cand);
if (!($w['ok'] ?? false)) continue;
$id = $w['id'];
// Already acknowledged or dismissed — the operator has spoken, and re-fixing behind
// them would be the opposite of what an acknowledgement means.
if (in_array($w['state'] ?? '', ['acknowledged', 'dismissed'], true)) {
$sum['quiet']++;
continue;
}
if (($cand['state'] ?? '') === 'resolved') {
vv_ai_finding_close($id, (string)($cand['note'] ?? ''));
$sum['resolved']++;
continue;
}
if (vv_ai_finding_may_autofix($cand)) {
if ($dryRun) { $sum['details'][] = "would fix {$cand['conf_key']}{$cand['proposed']}"; continue; }
$r = vv_ai_finding_apply_action($id, 'fix', 'Probed and written by the repair sweep.');
if ($r['ok'] ?? false) { $sum['fixed']++; $sum['details'][] = "fixed {$cand['conf_key']}"; }
else { $sum['needs_operator']++; vv_ai_finding_set_state($id, 'needs_operator', (string)($r['error'] ?? '')); }
continue;
}
if (($cand['state'] ?? '') === 'needs_operator') $sum['needs_operator']++;
}
}
// Marked only on a completed pass, and to when the pass began — a job that finished while
// this was running is then picked up next time instead of being skipped for having ended
// before a marker written at the end.
if (!$dryRun) vv_ai_sweep_mark($started);
return $sum;
}
// ── Answering a finding in words ─────────────────────────────────────────────────────────────
// The buttons are unambiguous by construction. This is for the other path — replying "yeah go
// ahead" in the chat that raised the finding — and it is matched here rather than asked of the
// model, because the model's answer would be a conf write and a wrong reading of "no, leave it"
// is not recoverable by apologising.
//
// Same shape as vv_ai_route_from_chat(): anchored patterns, most specific first, and anything
// unrecognised returns null so the assistant asks again instead of guessing. Two actions both
// matching is also null — "leave it, I know" and "leave it for now" differ by one clause and
// mean different things, so a phrase that supports both is not an instruction yet.
const VV_AI_ACTION_PATTERNS = [
// Acknowledge — "this is deliberate, stop telling me".
'ack' => [
'/\b(i|we) know\b/u',
'/\b(that|this|it)(?:\'s| is) (fine|expected|intentional|deliberate|on purpose)\b/u',
'/\bon purpose\b/u',
'/\b(aware|acknowledge|ack)\b/u',
'/\bmeant to be\b/u',
],
// Apply the proposed value.
'fix' => [
'/\bfix (it|that|this|them)?\b/u',
'/\b(go ahead|do it|apply|make the change|change it|update it|correct it)\b/u',
// A bare affirmative, as the whole message — "yes" answering "shall I fix it" is an
// instruction, "yes it looks wrong" is agreement about the diagnosis and nothing more.
// A trailing please is still bare.
'/\b(yes|yeah|yep|yup|sure|ok|okay)\b(\s*,?\s*please)?[\s,.!]*$/u',
'/\bplease do\b/u',
],
// Not now — no state written, it comes back next sweep.
'cancel' => [
'/\b(not now|later|leave it (alone|for now)|skip( it)?|cancel|ignore for now)\b/u',
'/\b(no|nope|nah)\b[\s,.!]*$/u',
'/\b(don\'?t|do not) (fix|touch|change|write|apply)\b/u',
],
];
// Returns 'fix' | 'ack' | 'cancel', or null when the reply does not clearly mean one of them.
//
// Only call this when a finding is actually pending. A bare "yes" means fix in answer to "shall
// I fix it" and means nothing at all on its own, and the difference is context this function
// cannot see.
function vv_ai_finding_action_from_text(string $text): ?string {
$t = strtolower(trim($text));
if ($t === '') return null;
$t = preg_replace('/\s+/', ' ', $t);
$matched = [];
foreach (VV_AI_ACTION_PATTERNS as $action => $patterns) {
foreach ($patterns as $re) {
if (preg_match($re, $t)) { $matched[$action] = true; break; }
}
}
// Exactly one reading, or none. "leave it, I know" hits both ack and cancel; that is a
// sentence the operator should be asked to restate, not one to pick a winner from.
return count($matched) === 1 ? array_key_first($matched) : null;
}
// Carry out an answered action against a stored finding.
//
// Fix goes through the same guarded write path as everything else, and is the one place a
// toggle may be written — because reaching here means the operator asked for it by name. The
// unattended sweep never calls this.
function vv_ai_finding_apply_action(string $id, string $action, string $note = ''): array {
$f = vv_ai_finding_get($id);
if ($f === null) return ['ok' => false, 'error' => 'no such finding'];
switch ($action) {
case 'ack':
return ['ok' => vv_ai_finding_ack($id, $note), 'action' => 'ack'];
case 'cancel':
// Deliberately writes nothing at all. "Not now" is not a state, it is the absence of
// one — recording it would make the finding look decided when it is still open.
return ['ok' => true, 'action' => 'cancel'];
case 'fix':
$proposed = $f['proposed'] ?? null;
if ($proposed === null) return ['ok' => false, 'error' => 'nothing proposed to write'];
$key = (string)$f['conf_key'];
$ok = vv_conf_write_changes([[
'file' => (string)($f['conf_file'] ?? 'master.conf'),
'key' => $key,
'value' => (string)$proposed,
'type' => 'scalar',
]]);
$wrote = !in_array(false, $ok, true);
if ($wrote) {
vv_ai_finding_set_state($id, 'fixed',
$note !== '' ? $note : 'Wrote ' . $key . ' at the operator\'s request.');
}
return ['ok' => $wrote, 'action' => 'fix',
'error' => $wrote ? null : 'conf write refused — see conf_changes.log'];
}
return ['ok' => false, 'error' => 'unknown action'];
}
// ── Resolving a log line back to a conf key ──────────────────────────────────────────────────
// By value wherever possible, by name only as a fallback.
//
// A log line usually contains the thing that failed — the URL that did not answer. That value
// came from a conf key, so searching the conf for which key holds it is an exact lookup with a
// definite answer. Guessing the key from the container's name is inference, and the failure mode
// is silent: HOST1_EMBY_URL and HOST1_EMBY_EXTERNAL_URL are both plausible for "Emby" and only
// one of them is the value that just failed.
//
// Same principle as resolve_tailscale_ip() refusing similarity matching for host identity: an
// exact match or an honest nothing.
// Every conf key whose value the given text starts with, longest first. Prefix rather than
// equality because a log reports the URL it actually called — the conf value plus an endpoint
// path — and the longest match is the most specific key that could have produced it.
function vv_ai_conf_keys_for_value(string $value): array {
$value = trim($value);
if ($value === '' || strlen($value) < 6) return [];
$hits = [];
foreach (vv_conf_vars() as $k => $v) {
$v = trim((string)$v);
if ($v === '' || strlen($v) < 6) continue;
if ($v === $value || str_starts_with($value, $v)) $hits[$k] = strlen($v);
}
arsort($hits);
return array_keys($hits);
}
// <HOSTID>_<SUBJECT>_<SUFFIX>, built literally and then checked for existence. Nothing is
// inferred: either the conf holds a key by exactly that name or this returns null.
function vv_ai_conf_key_for_subject(string $subject, string $suffix): ?string {
$norm = strtoupper(preg_replace('/[^A-Za-z0-9]+/', '_', trim($subject)));
if ($norm === '') return null;
$key = strtoupper(vv_detect_host()) . '_' . $norm . '_' . strtoupper($suffix);
return array_key_exists($key, vv_conf_vars()) ? $key : null;
}
// Which conf file a key lives in. A finding has to name the file it would be edited in, and
// host keys are not in master.conf.
function vv_ai_conf_file_for_key(string $key): string {
foreach (vv_get_conf_files() as $f) {
if (preg_match('/^\s*' . preg_quote($key, '/') . '\s*=/m', vv_read_conf_raw($f))) return $f;
}
return 'master.conf';
}
// ── Deterministic triage ─────────────────────────────────────────────────────────────────────
// Patterns are written against log formats that exist in this repo today, taken from the emit
// sites rather than imagined. A pattern that stops matching because its log line was reworded
// produces no finding, which is the safe direction — the alternative is a pattern loose enough
// to match anything, which fills the store with rows nobody can act on.
//
// No model runs here. "Connection refused" is not a judgement call, and a 14B model invoked
// after every cron job to notice it would be slower, costlier and less reliable than a regex.
// The model's job starts where these stop: explaining a finding, and talking the operator
// through the ones that cannot be repaired automatically.
const VV_AI_TRIAGE_PATTERNS = [
// docker_watchdog HTTP check — the richest signal, carrying both subject and failing URL.
[
're' => '/^(?P<subject>\S+) — not responding at (?P<observed>\S+) \(strike/u',
'kind' => 'unreachable',
'suffix' => 'URL',
],
// docker_watchdog API check, 401/403. The endpoint answered, so the address is right and
// the credential is not.
[
're' => '/^(?P<subject>\S+) — API check skipped \(HTTP (?:401|403)/u',
'kind' => 'auth_rejected',
'suffix' => 'API_KEY',
],
// Configured for an API check with nothing to authenticate with.
[
're' => '/^(?P<subject>\S+) — API check skipped \(no key configured\)/u',
'kind' => 'missing_value',
'suffix' => 'API_KEY',
],
[
're' => '/^Skipping (?P<subject>\S+) — placeholder API key/u',
'kind' => 'missing_value',
'suffix' => 'API_KEY',
],
[
're' => '/^(?P<subject>\S+) (?:—\s*)?API unreachable/u',
'kind' => 'unreachable',
'suffix' => 'URL',
],
];
// One log line in, at most one finding candidate out. Returns null for everything else, which is
// almost every line.
function vv_ai_triage_line(string $line): ?array {
// Strip the timestamp and level decoration the logger adds, so patterns can anchor on ^.
$body = preg_replace('/^\S+\s+\S+\s+(?:[^\[]*\[[A-Z]+\]\s*)?/u', '', rtrim($line));
$body = trim((string)$body);
if ($body === '') return null;
foreach (VV_AI_TRIAGE_PATTERNS as $p) {
if (!preg_match($p['re'], $body, $m)) continue;
$subject = trim($m['subject'] ?? '');
$observed = trim($m['observed'] ?? '');
if ($subject === '') continue;
// Value first, name second. Only one candidate key is accepted — two keys holding the
// same value means the log cannot say which one produced it, and picking either is the
// guess this whole approach exists to avoid.
$key = null;
if ($observed !== '') {
$byValue = vv_ai_conf_keys_for_value($observed);
if (count($byValue) === 1) $key = $byValue[0];
}
if ($key === null) $key = vv_ai_conf_key_for_subject($subject, $p['suffix']);
if ($key === null) continue; // nothing actionable — no row
return [
'kind' => $p['kind'],
'subject' => $subject,
'conf_key' => $key,
'conf_file' => vv_ai_conf_file_for_key($key),
'observed' => $observed !== '' ? $observed : (string)(vv_conf_vars()[$key] ?? ''),
'evidence' => $body,
];
}
return null;
}
// A whole log tail in, one candidate per distinct fault out. A job that retried twelve times
// produces twelve identical lines, and the store's dedupe would collapse them anyway — doing it
// here keeps the sweep from writing the same file twelve times in a row.
function vv_ai_triage_log(array $lines, string $sourceLog = ''): array {
$found = [];
foreach ($lines as $line) {
$c = vv_ai_triage_line((string)$line);
if ($c === null) continue;
$c['source_log'] = $sourceLog;
$found[vv_ai_finding_id($c['kind'], $c['subject'], $c['conf_key'])] = $c;
}
return array_values($found);
}
// ── Proving a candidate ──────────────────────────────────────────────────────────────────────
// The guard the whole unattended path rests on: nothing is written that has not answered.
//
// A model can be confident that a port should be 8686. A probe can report that 8686 answered.
// Only the second is a fact, and only facts get written to conf without being asked. Everything
// a probe cannot settle becomes a conversation instead — which is not a lesser outcome, it is
// the honest one for a value that cannot be derived from this machine.
//
// Deliberately narrow. These check reachability and identity, never correctness of behaviour:
// that Lidarr answers on 8686 does not prove 8686 is the port you meant, only that something is
// listening there and calling itself Lidarr. Proving intent is not a probe's job.
function vv_ai_probe_timeout(): int {
return max(1, (int)(vv_conf_vars()['AI_PROBE_TIMEOUT'] ?? 4));
}
// Does this URL answer at all? Any HTTP status counts, including 401 — a refusal is proof that
// something is listening and speaking HTTP, which is exactly what an address probe is asking.
// Distinguishing "wrong address" from "wrong credential" is the point of having both kinds.
function vv_ai_probe_url(string $url): array {
if (!preg_match('#^https?://[^\s/$.?\#][^\s]*$#i', $url)) {
return ['ok' => false, 'reason' => 'not a url'];
}
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_NOBODY => true,
CURLOPT_TIMEOUT => vv_ai_probe_timeout(),
CURLOPT_CONNECTTIMEOUT => vv_ai_probe_timeout(),
CURLOPT_SSL_VERIFYPEER => false, // these are LAN and Tailscale endpoints, often self-signed
CURLOPT_SSL_VERIFYHOST => false,
]);
curl_exec($ch);
$code = (int)curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
$err = curl_error($ch);
curl_close($ch);
return $code > 0
? ['ok' => true, 'code' => $code]
: ['ok' => false, 'reason' => $err !== '' ? $err : 'no response'];
}
// Every container on this host, by exact name. The membership test for unknown_target findings,
// and one docker call rather than one per candidate.
function vv_ai_container_names(): array {
static $names = null;
if ($names !== null) return $names;
$out = [];
@exec('timeout ' . vv_ai_probe_timeout() . " docker ps -a --format '{{.Names}}' 2>/dev/null", $out, $rc);
return $names = ($rc === 0) ? array_values(array_filter(array_map('trim', $out))) : [];
}
// Candidate corrections for a URL whose host or port stopped answering.
//
// Only two transformations, both conservative: the same host on a port that some other conf key
// already uses, and the same port on a host some other conf key already names. Both draw
// exclusively from values already present in this installation's conf — nothing is invented, and
// a scan of the port range is deliberately not attempted. Finding *a* listening port is not the
// same as finding the right service, and a probe that accepts any answer would happily point
// Lidarr at Sonarr.
function vv_ai_url_candidates(string $observed): array {
$parts = @parse_url($observed);
if (!is_array($parts) || empty($parts['host'])) return [];
$hosts = $ports = [];
foreach (vv_conf_vars() as $k => $v) {
$v = trim((string)$v);
if (!preg_match('#^https?://#i', $v)) continue;
$p = @parse_url($v);
if (!is_array($p) || empty($p['host'])) continue;
$hosts[$p['host']] = true;
if (!empty($p['port'])) $ports[(int)$p['port']] = true;
}
$scheme = $parts['scheme'] ?? 'http';
$path = $parts['path'] ?? '';
$out = [];
foreach (array_keys($ports) as $port) {
$c = $scheme . '://' . $parts['host'] . ':' . $port . $path;
if ($c !== $observed) $out[$c] = true;
}
foreach (array_keys($hosts) as $host) {
$port = !empty($parts['port']) ? ':' . $parts['port'] : '';
$c = $scheme . '://' . $host . $port . $path;
if ($c !== $observed) $out[$c] = true;
}
return array_keys($out);
}
// Try to prove a correction for one finding. Returns the finding with 'proposed' and 'proven'
// filled in, or unchanged when nothing could be proven — which is the common case and not a
// failure.
//
// auth_rejected and missing_value are never proven here on purpose. A credential cannot be
// derived from this host by definition: if it could be read from somewhere, it would not be a
// credential. Those go straight to the operator.
function vv_ai_probe_finding(array $f): array {
$kind = (string)($f['kind'] ?? '');
if ($kind === 'unknown_target') {
$observed = (string)($f['observed'] ?? '');
// Exact membership only. A container name is a literal, and "close to an existing name"
// is how a repair renames the wrong thing.
if (in_array($observed, vv_ai_container_names(), true)) {
return $f; // it exists after all — nothing to correct
}
return $f + ['state' => 'needs_operator'];
}
if ($kind !== 'unreachable') {
// Nothing on this machine can supply a credential, so there is nothing to prove.
$f['state'] = 'needs_operator';
return $f;
}
$observed = (string)($f['observed'] ?? '');
// If the observed address answers now, the fault has cleared on its own — a host that was
// rebooting, most often. Recording a proposal here would repair something already working.
if (vv_ai_probe_url($observed)['ok']) {
$f['state'] = 'resolved';
$f['note'] = 'Answered when probed — the address was reachable again by the time this ran.';
return $f;
}
$answered = [];
foreach (vv_ai_url_candidates($observed) as $candidate) {
if (vv_ai_probe_url($candidate)['ok']) $answered[] = $candidate;
}
// Exactly one, or none. Two addresses answering means the probe cannot say which is the
// right one, and picking either is the guess this exists to prevent.
if (count($answered) === 1) {
$f['proposed'] = $answered[0];
$f['proven'] = true;
$f['note'] = 'Probed ' . $answered[0] . ' and it answered; ' . $observed . ' did not.';
return $f;
}
$f['proven'] = false;
$f['state'] = 'needs_operator';
$f['note'] = $answered
? 'Several addresses answered (' . implode(', ', $answered) . '), so none was written.'
: 'Nothing answered at ' . $observed . ', and no address in the conf answered either.';
return $f;
}
+313 -14
View File
@@ -18,7 +18,27 @@
//
// Structure only; values are not validated.
// Consistent with conf_upgrade.sh, this reconciles shape and leaves correctness to the
// consuming script.
// consuming script. Shape is enforced — the file must parse, source, and read back the
// value that was asked for. Whether 300 is a sensible timeout is still not this file's
// question.
//
// One guarded write path, not one set of guards per caller.
// vv_conf_edit() owns the lock, the backup, the validation and the audit line, and every
// conf write in the plugin goes through it — the Settings form, both raw editors, the
// flag toggles, orchestrator membership, script moves and reorders, the rsync window
// arrays, the docker folder map and first-run setup. vv_write_conf_raw() has exactly one
// caller left, inside vv_conf_edit() itself.
//
// A caller that reaches past it gets tmp + rename and nothing else: no backup, no bash -n,
// no read-back, no audit line. That is how this started — seven files each with their own
// partial idea of what a safe conf write was, two of them carrying a copy-pasted bash -n
// block that failed open.
//
// The rewrite happens inside the lock, or it proves nothing moved.
// $mutate receives the current contents, so a caller that can rebuild from them has no
// window at all. Callers that must assemble the result first — the ones driven by a form
// payload — compare against what they read and return null if it no longer matches,
// which abandons the write rather than reverting a concurrent edit.
//
// OPERATIONAL SAFEGUARDS
// An unmatched section yields no fields rather than a wrong write.
@@ -30,6 +50,33 @@
// Each field carries the exact line it came from, so a write cannot land outside the
// subsection it was read from.
//
// A value is shell source code, so command substitution is refused outright.
// Scalars are written inside double quotes and array values are spliced in verbatim, so
// $(...) or a backtick in a value runs as root in every script that sources the conf. No
// conf in this repo uses either, so rejecting them costs nothing. $VAR and ${VAR} stay
// legal — AI_DATA_DIR="${DATA_DIR}/ai" is the established idiom, and a reference resolves
// to a value where a substitution runs a program.
//
// Nothing is written without a backup in hand.
// The previous contents are copied to CONF_BACKUP_DIR/<file>.<stamp> first, and a backup
// that cannot be taken cancels the write. The confs are gitignored, so that directory is
// the entire recovery story — there is no history to revert to.
//
// Validated, then verified, then rolled back on failure.
// bash -n proves the candidate parses; sourcing the installed file and reading the keys
// back proves the values survived quoting. A scalar that does not read back as the value
// requested restores the backup. A conf that parses cleanly and holds the wrong string is
// the failure a syntax check cannot see.
//
// Held under an exclusive lock for the whole read-modify-write.
// Two concurrent savers would otherwise read the same original, and the second rename
// would discard the first one's change without either reporting a failure.
//
// Every outcome is logged, and secrets are logged by name only.
// LOG_DIR/conf_changes.log records applied, rejected, failed and rolled-back alike. A
// credential-shaped key logs value=<redacted> — the log proves a change happened, it is
// not a second copy of the secret.
//
// EXPORTS
// vv_conf_has_sections() does this script have an editable conf section
// vv_conf_parse_subsection() fields within one named subsection
@@ -38,7 +85,10 @@
// vv_conf_write_changes() apply edits back to the conf file
//
// CONFIGURATION
// CONF_DIR master.conf and host*.conf are the read and write targets
// CONF_DIR master.conf and host*.conf are the read and write targets
// CONF_BACKUP_DIR DATA_DIR/Backups/Confs — pre-write copies, 0700
// LOG_DIR conf_changes.log is written here
// CONF_BACKUP_RETAIN backups kept per conf file (default 30)
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once __DIR__ . '/config.php';
@@ -110,6 +160,8 @@ const VV_SCRIPT_CONF_SECTIONS = [
'AI/ai_index.sh' => ['AI Retrieval Index', 'AI Master Switch', 'Ollama'],
'AI/ai_query.sh' => ['AI Retrieval Index', 'AI Master Switch', 'Ollama'],
'AI/ai_token_sync.sh' => ['AI Feature Toggles'],
'Plugin/unraid/Tools/ai_repair_sweep.sh' => ['AI Repair', 'AI Repair Findings',
'AI Master Switch'],
// ── Shared host sections ──────────────────────────────────────────────────────────────────
// A script's settings are not only the ones named after it. Anything talking to Lidarr reads
// the host's Lidarr block; anything reading playback reads Emby. Those blocks are where the
@@ -307,19 +359,132 @@ function vv_conf_fields_for_script(string $id): array {
return $groups;
}
// A conf key must be a plain shell identifier. Every downstream use — the replacement regex,
// the source-verification subshell, the audit line — treats the key as trusted text, so it is
// validated once here rather than escaped differently in three places.
function vv_conf_key_valid(string $key): bool {
return (bool) preg_match('/^[A-Za-z_][A-Za-z0-9_]*$/', $key);
}
// Credential-shaped keys are logged by name only. The audit log is the record that a change
// happened, not a second copy of the secret that changed.
function vv_conf_key_is_secret(string $key): bool {
return (bool) preg_match('/(PASS|PASSWORD|SECRET|TOKEN|API_KEY|APIKEY|_KEY)$|(PASS|PASSWORD|SECRET|TOKEN|APIKEY)/i', $key);
}
// Scalar values are written inside double quotes and array values are spliced in verbatim, so
// a value is shell source code the moment any script reads the conf. Command substitution in a
// value therefore executes on every load — in every script, as root. No conf in this repo uses
// it, so rejecting it costs nothing and closes the path.
//
// $VAR and ${VAR} stay legal on purpose: AI_DATA_DIR="${DATA_DIR}/ai" is the established idiom
// here, and a reference resolves to a value where a substitution runs a program.
function vv_conf_value_safe(string $value): bool {
return !preg_match('/\$\(|`|<\(|>\(/', $value);
}
// Under DATA_DIR, not beside the confs. data/ is the one on-disk root and is gitignored whole,
// so a backup here cannot become a tracked file the way Configurations/*.bak did. 0700 because
// these are verbatim copies of files holding every credential on the host.
function vv_conf_backup_dir(): string {
if (!is_dir(CONF_BACKUP_DIR)) @mkdir(CONF_BACKUP_DIR, 0700, true);
return CONF_BACKUP_DIR;
}
// Copy the current conf aside before it is touched. Returns the backup path, or null when no
// backup could be taken — which the caller treats as a reason not to write, because the whole
// recovery story for these files is this directory.
function vv_conf_backup(string $filename): ?string {
$src = CONF_DIR . '/' . $filename;
if (!is_file($src)) return null;
$dest = vv_conf_backup_dir() . '/' . $filename . '.' . date('Y-m-d\THis');
// Same second, second change: keep both rather than silently overwrite the older one.
if (file_exists($dest)) {
$n = 1;
while (file_exists($dest . '.' . $n)) $n++;
$dest .= '.' . $n;
}
if (!@copy($src, $dest)) return null;
@chmod($dest, 0600);
vv_conf_prune_backups($filename);
return $dest;
}
// Retain the newest CONF_BACKUP_RETAIN backups per conf file. Pruned by filename, which sorts
// chronologically because the stamp is ISO-8601 — no stat() per candidate.
function vv_conf_prune_backups(string $filename): void {
$keep = (int) (vv_conf_vars()['CONF_BACKUP_RETAIN'] ?? 30);
if ($keep < 1) $keep = 30;
$found = glob(vv_conf_backup_dir() . '/' . $filename . '.*') ?: [];
if (count($found) <= $keep) return;
sort($found);
foreach (array_slice($found, 0, count($found) - $keep) as $old) @unlink($old);
}
// Source a candidate conf in a subshell and read back the keys that were just written.
// bash -n proves the file parses; this proves the values survived quoting and arrived as
// intended. Returns null when the file could not be sourced at all.
function vv_conf_read_back(string $path, array $keys): ?array {
$script = 'source ' . escapeshellarg($path) . ' >/dev/null 2>&1 || exit 90; ';
foreach ($keys as $k) {
if (!vv_conf_key_valid($k)) continue;
$script .= 'printf "%s\t%s\n" ' . escapeshellarg($k) . ' "${' . $k . '-}"; ';
}
$out = []; $rc = 0;
exec('bash -c ' . escapeshellarg($script) . ' 2>/dev/null', $out, $rc);
if ($rc !== 0) return null;
$vals = [];
foreach ($out as $line) {
$parts = explode("\t", $line, 2);
if (count($parts) === 2) $vals[$parts[0]] = $parts[1];
}
return $vals;
}
// One line per conf change, best-effort and never able to block the write itself.
function vv_conf_audit(string $file, string $key, string $outcome, string $detail = ''): void {
$line = date('Y-m-d H:i:s')
. " file={$file} key={$key} outcome={$outcome}"
. ($detail !== '' ? " {$detail}" : '')
. ' ip=' . ($_SERVER['REMOTE_ADDR'] ?? 'cli')
. "\n";
@file_put_contents(LOG_DIR . '/conf_changes.log', $line, FILE_APPEND | LOCK_EX);
}
// Write a batch of field changes back to their respective conf files.
// Each change: {file, key, value, type}
function vv_conf_write_changes(array $changes): array {
$byFile = [];
foreach ($changes as $c) {
if (!empty($c['file']) && !empty($c['key'])) $byFile[$c['file']][] = $c;
if (empty($c['file']) || empty($c['key'])) continue;
if (!vv_conf_key_valid($c['key'])) {
vv_conf_audit((string) $c['file'], (string) $c['key'], 'rejected', 'reason=malformed-key');
continue;
}
if (!vv_conf_value_safe((string) ($c['value'] ?? ''))) {
vv_conf_audit((string) $c['file'], (string) $c['key'], 'rejected', 'reason=command-substitution');
continue;
}
$byFile[$c['file']][] = $c;
}
$results = [];
foreach ($byFile as $file => $fileChanges) {
$raw = vv_read_conf_raw($file);
if ($raw === '') { $results[$file] = false; continue; }
$results[$file] = vv_conf_write_file($file, $fileChanges);
}
return $results;
}
// Read-modify-write for one conf file. The surgical replacement lives here; the guards that
// make installing it safe live in vv_conf_install(), which every conf writer shares.
function vv_conf_write_file(string $file, array $fileChanges): bool {
return vv_conf_edit($file, function (string $raw) use ($fileChanges): ?string {
foreach ($fileChanges as $c) {
$qKey = preg_quote($c['key'], '/');
$value = $c['value'];
@@ -328,7 +493,10 @@ function vv_conf_write_changes(array $changes): array {
if ($type === 'scalar') {
$raw = preg_replace_callback(
'/^(\s*' . $qKey . '\s*=\s*)("(?:[^"\\\\]|\\\\.)*"|\'(?:[^\'\\\\]|\\\\.)*\'|[^#\n]*?)(\s*(?:#[^\n]*)?)$/m',
fn($m) => $m[1] . '"' . str_replace(['"', '\\'], ['\\"', '\\\\'], $value) . '"' . $m[3],
// Backslashes first. Escaping quotes first meant the backslash just inserted
// was itself doubled on the next pass — " became \\" — which closed the
// string early and stored a truncated value that still parsed cleanly.
fn($m) => $m[1] . '"' . str_replace(['\\', '"'], ['\\\\', '\\"'], $value) . '"' . $m[3],
$raw
) ?? $raw;
@@ -354,23 +522,154 @@ function vv_conf_write_changes(array $changes): array {
) ?? $raw;
}
}
// Only the scalar path escapes its value; the array, array_single and assoc_array
// paths splice the caller's text into the file verbatim, and the type comes from the
// request. Every script sources these files, so the result is parsed before it is
// allowed to replace a working conf.
$results[$file] = vv_conf_syntax_ok($raw) && vv_write_conf_raw($file, $raw);
return $raw;
// Scalars are verified because their intended value is known exactly. The array types splice
// caller-supplied text whose sourced form is legitimately not equal to what was written, so
// for those a clean source is the whole assertion.
}, vv_conf_expected_scalars($fileChanges), array_column($fileChanges, 'key'));
}
// The intended end state for the keys that can be checked against one.
function vv_conf_expected_scalars(array $fileChanges): array {
$expect = [];
foreach ($fileChanges as $c) {
if (($c['type'] ?? 'scalar') === 'scalar') $expect[$c['key']] = (string) $c['value'];
}
return $results;
return $expect;
}
// The one guarded way to install a changed conf. $mutate receives the current contents and
// returns the rewritten ones, or null to abort without touching anything. Every conf writer in
// the plugin goes through here, so the lock, the backup, the validation and the audit trail are
// written once and cannot be forgotten by a new caller.
//
// $expect key => intended value, verified by sourcing the installed file
// $subjects names for the audit line, when there is no key to assert (a toggled member)
// $allowCreate write a conf that does not exist yet. Off by default: for every caller except
// first-run setup, a missing target means the filename is wrong, and creating it
// would leave a stray conf that shadows nothing and is sourced by nobody.
function vv_conf_edit(string $file, callable $mutate, array $expect = [], array $subjects = [],
bool $allowCreate = false): bool {
$path = CONF_DIR . '/' . $file;
$subjects = $subjects ?: (array_keys($expect) ?: ['-']);
$audit = function (string $outcome, string $detail = '') use ($file, $subjects): void {
foreach ($subjects as $s) vv_conf_audit($file, (string) $s, $outcome, $detail);
};
// Held across the whole read-modify-write. Two concurrent savers would otherwise each read
// the same original, and the second rename would silently discard the first one's change.
$lockFh = @fopen(CONF_DIR . '/.conf-write.lock', 'c');
if ($lockFh === false || !flock($lockFh, LOCK_EX)) {
if ($lockFh) fclose($lockFh);
$audit('failed', 'reason=lock');
return false;
}
try {
// A conf that does not exist yet is a create, not an edit — first-run setup writes
// host*.conf before there is anything to read. There is no prior content to lose, so
// there is nothing to back up, and undoing a failed create means removing the file.
$exists = is_file($path);
if (!$exists && !$allowCreate) { $audit('failed', 'reason=unreadable'); return false; }
$raw = $exists ? vv_read_conf_raw($file) : '';
if ($exists && $raw === '') { $audit('failed', 'reason=unreadable'); return false; }
$before = $exists ? (vv_conf_read_back($path, $subjects) ?? []) : [];
$new = $mutate($raw);
if ($new === null) { $audit('failed', 'reason=no-match'); return false; }
if ($exists && $new === $raw) { $audit('no-change'); return true; }
$backup = null;
if ($exists) {
$backup = vv_conf_backup($file);
if ($backup === null) {
// No recovery path for this write means the write does not happen. These files
// are gitignored, so a backup not taken cannot be reconstructed afterwards.
$audit('failed', 'reason=backup');
return false;
}
}
$undo = function () use ($backup, $path): void {
if ($backup !== null) @copy($backup, $path); else @unlink($path);
};
if (!vv_conf_syntax_ok($new)) { $audit('rejected', 'reason=syntax'); return false; }
if (!vv_write_conf_raw($file, $new)) { $audit('failed', 'reason=write'); return false; }
// bash -n proved the candidate parses. This proves the installed file still sources and
// that each value arrived intact — a quoting bug produces a file that parses perfectly
// and holds the wrong string, which is the failure the syntax check cannot see.
$after = vv_conf_read_back($path, $subjects);
if ($after === null) {
$undo();
$audit('rolled-back', 'reason=source-failed');
return false;
}
foreach ($expect as $key => $want) {
if (($after[$key] ?? null) !== $want) {
$undo();
vv_conf_audit($file, $key, 'rolled-back', 'reason=value-mismatch');
return false;
}
}
foreach ($subjects as $s) {
$s = (string) $s;
// A subject that is not a conf key — a script id, or a marker for a whole-file save
// — has no value to read back, so there is no before and after to report.
if (!vv_conf_key_valid($s)) { vv_conf_audit($file, $s, 'applied'); continue; }
if (vv_conf_key_is_secret($s)) { vv_conf_audit($file, $s, 'applied', 'value=<redacted>'); continue; }
vv_conf_audit($file, $s, 'applied',
'from=' . vv_conf_audit_val($before[$s] ?? '') . ' to=' . vv_conf_audit_val($after[$s] ?? ''));
}
return true;
} finally {
flock($lockFh, LOCK_UN);
fclose($lockFh);
}
}
// Audit values are single-line and bounded. A conf value can be a 40-line array; the log is a
// record of what changed, and an unbounded splat of it makes the log unreadable at the moment
// it is actually needed.
function vv_conf_audit_val(string $v): string {
$v = preg_replace('/\s+/', ' ', trim($v));
if (strlen($v) > 120) $v = substr($v, 0, 117) . '...';
return '"' . $v . '"';
}
// bash -n against a private temp copy. Returns true when the content parses as a sourceable
// conf, false otherwise — never writes anything itself.
function vv_conf_syntax_ok(string $content): bool {
return vv_conf_syntax_error($content) === null;
}
// The same check, with bash's own complaint when it fails — the raw editors show it to whoever
// is typing, where "conf does not parse" alone would mean hunting the line by hand. $label
// replaces the temp path in the message so the reader sees their own filename.
//
// Fails closed. This used to pass when the temp file could not be created, which was defensible
// while every write came from a human clicking Save on a form. The assistant writes through here
// too, so an unverified conf is not installed — a refused write is recoverable, a conf that no
// script can source is a system-wide outage.
function vv_conf_syntax_error(string $content, string $label = 'conf'): ?string {
$tmp = tempnam(sys_get_temp_dir(), 'vvconf');
if ($tmp === false) return true; // cannot check — do not block the write
if ($tmp === false) return 'cannot verify: no writable temp directory';
file_put_contents($tmp, $content);
$out = []; $rc = 0;
exec('bash -n ' . escapeshellarg($tmp) . ' 2>&1', $out, $rc);
@unlink($tmp);
return $rc === 0;
if ($rc === 0) return null;
$msg = implode(' ', array_filter(array_map('trim', $out)));
return str_replace($tmp, $label, $msg ?: 'conf does not parse');
}
+80 -9
View File
@@ -117,6 +117,8 @@ define('STATE_DIR', DATA_DIR . '/state');
define('AI_DATA_DIR', DATA_DIR . '/ai');
define('CACHE_BACKUP_DIR', DATA_DIR . '/cache');
define('LOG_ARCHIVE_DIR', DATA_DIR . '/logs');
define('BACKUP_DIR', DATA_DIR . '/Backups');
define('CONF_BACKUP_DIR', BACKUP_DIR . '/Confs');
define('LOG_DIR', '/var/log/varaverk');
// User-authored custom scripts (scheduler page "+ Create Script") — kept outside the git
// repo entirely, alongside the User Scripts plugin's own storage. Any *.sh file placed
@@ -298,8 +300,13 @@ function vv_push_master_conf(): array {
return $results;
}
function vv_get_hostname(): string {
return trim(shell_exec('hostname -s') ?: '');
// Cached: this forks a shell, and it is reached from vv_conf_vars() by way of vv_detect_host(),
// which the repair sweep calls per log line. A machine does not rename itself mid-request.
function vv_get_hostname(bool $flush = false): string {
static $name = null;
if ($flush) { $name = null; return ''; }
if ($name === null) $name = trim(shell_exec('hostname -s') ?: '');
return $name;
}
// Mirror of common.sh resolve_tailscale_ip(): tries `tailscale ip -4` first (Tailscale manages
@@ -326,7 +333,19 @@ function vv_resolve_tailscale_ip(string $hostname): string {
return count($matches) === 1 ? $matches[0] : '';
}
function vv_detect_host(): string {
// Cached alongside the others: this reads master.conf in full and is called by vv_conf_vars() on
// every lookup, so leaving it uncached would mean the conf is still read once per key even with
// the parsed values cached.
function vv_detect_host(bool $flush = false): string {
static $host = null;
if ($flush) { $host = null; return ''; }
if ($host !== null) return $host;
$host = _vv_detect_host_uncached();
return $host;
}
function _vv_detect_host_uncached(): string {
// Reads master.conf for HOST1="name" (or HOST1_NAME="name") and matches running hostname.
// Returns 'host1', 'host2', 'host3', ... or 'unknown'. Works for any number of hosts.
$master = vv_read_conf_raw('master.conf');
@@ -395,7 +414,13 @@ function vv_write_conf_raw(string $filename, string $content): bool {
if ($content !== '' && !str_ends_with($content, "\n")) $content .= "\n";
if (file_put_contents($tmp, $content) === false) return false;
return rename($tmp, $path);
if (!rename($tmp, $path)) return false;
// Every conf write in the plugin lands here, so this is the one place the parsed-conf cache
// has to be dropped. Doing it in the callers instead would mean a new writer inheriting a
// stale cache and no obvious reason why.
vv_conf_vars_flush();
return true;
}
function vv_get_conf_files(): array {
@@ -418,15 +443,59 @@ function vv_get_conf_files(): array {
return $files;
}
// Parse conf into key=>value map for $VAR substitution in docs
function vv_conf_vars(): array {
$vars = [];
// Drops the parsed-conf cache. Called by vv_write_conf_raw() — the single point at which a conf
// changes on disk — so no writer has to remember to do it.
//
// Necessary on top of the mtime check below, not instead of it: filesystem mtimes have one-second
// resolution, and a write followed by a read inside the same second is exactly what the conf
// writer does when it verifies a value it just wrote. Without this, that read could be served the
// value from before the write and the verification would compare a value against itself.
function vv_conf_vars_flush(): void {
vv_conf_vars(true);
// Host identity is derived from master.conf too, so a write that changes HOST1 has to
// invalidate it as well — first-run setup does exactly that, then asks which host this is.
vv_detect_host(true);
vv_get_hostname(true);
}
// Parse conf into key=>value map for $VAR substitution in docs.
//
// Cached, because this is not the cheap function its callers assume. It reads and regex-parses
// both conf files on every call — around 1,900 lines — and the repair sweep asks it for a key per
// log line. The measured cost of exactly this pattern is on record: the 30 minutes play_state_sync
// spent in 2026-07 was parse and fork overhead, not the API it was blamed on.
//
// Keyed on a hash of the contents, so an edit made outside this process — the raw editor in
// another tab, conf_upgrade from bash, a hand edit over SSH — is still picked up.
//
// Content rather than mtime and size, which was the first attempt and was wrong. Both conf files
// were rewritten within the same second and to the same byte count — "true" and "false" trading
// places across two keys — and the stale values were served straight back. mtime has one-second
// resolution and a same-size edit is not a rare shape in a file of booleans.
//
// Reading both files every call is not what costs anything here; parsing them is. The read is
// tens of microseconds against a parse of ~1,900 lines followed by two resolution passes.
function vv_conf_vars(bool $flush = false): array {
static $cache = null;
static $stamp = null;
if ($flush) { $cache = null; $stamp = null; return []; }
$files = ['master.conf'];
$host = vv_detect_host();
if (preg_match('/^host\d+$/', $host)) $files[] = $host . '.conf';
$raws = [];
$sig = '';
foreach ($files as $f) {
$raw = vv_read_conf_raw($f);
$raws[$f] = vv_read_conf_raw($f);
$sig .= md5($raws[$f]);
}
if ($cache !== null && $stamp === $sig) return $cache;
$vars = [];
foreach ($files as $f) {
$raw = $raws[$f];
// Match: VAR_NAME="value" or VAR_NAME=value (no quotes)
preg_match_all('/^\s*([A-Z0-9_]+)\s*=\s*["\']?([^"\'#\n]*?)["\']?\s*(?:#.*)?$/m', $raw, $m);
foreach ($m[1] as $i => $key) {
@@ -447,7 +516,9 @@ function vv_conf_vars(): array {
}
}
unset($v);
return $vars;
$stamp = $sig;
return $cache = $vars;
}
// Query the Unraid GraphQL API for a given host.
+27 -22
View File
@@ -51,6 +51,7 @@
// Reads/writes HOST*_DOCKER_FOLDER_MAP in host*.conf (used by onboard scripts)
require_once __DIR__ . '/config.php';
require_once __DIR__ . '/confform.php';
define('VV_DOCKER_JSON', SCRIPTS_DIR . '/docker_folders.json');
define('VV_FV3_JSON', '/boot/config/plugins/folder.view3/docker.json');
@@ -344,17 +345,19 @@ function vv_dk_rename_folder(string $folderId, string $newName): array {
$data[$folderId]['name'] = $newName;
if (!vv_dk_write_json($data)) return ['ok' => false, 'error' => 'JSON write failed'];
// Update conf: replace old folder name with new name in the map
// Update conf: replace old folder name with new name in the map.
// The rebuild is a pure function of the current contents, so it runs inside vv_conf_edit()'s
// lock rather than against a copy read beforehand — there is no window to lose an edit in.
$currentHost = vv_detect_host();
$myId = strtoupper($currentHost);
$raw = vv_read_conf_raw($currentHost . '.conf');
$map = vv_dk_read_conf_map($raw, $myId);
foreach ($map as &$v) {
if ($v === $oldName) $v = $newName;
}
unset($v);
$updated = vv_dk_write_conf_map($raw, $myId, $map);
vv_write_conf_raw($currentHost . '.conf', $updated);
vv_conf_edit($currentHost . '.conf', function (string $raw) use ($myId, $oldName, $newName): string {
$map = vv_dk_read_conf_map($raw, $myId);
foreach ($map as &$v) {
if ($v === $oldName) $v = $newName;
}
unset($v);
return vv_dk_write_conf_map($raw, $myId, $map);
}, [], ["{$myId}_DOCKER_FOLDER_MAP"]);
return ['ok' => true];
}
@@ -367,14 +370,14 @@ function vv_dk_delete_folder(string $folderId): array {
unset($data[$folderId]);
if (!vv_dk_write_json($data)) return ['ok' => false, 'error' => 'JSON write failed'];
// Remove from conf map
// Remove from conf map — rebuilt inside the lock, see vv_dk_rename_folder() above.
$currentHost = vv_detect_host();
$myId = strtoupper($currentHost);
$raw = vv_read_conf_raw($currentHost . '.conf');
$map = vv_dk_read_conf_map($raw, $myId);
$map = array_filter($map, fn($v) => $v !== $folderName);
$updated = vv_dk_write_conf_map($raw, $myId, $map);
vv_write_conf_raw($currentHost . '.conf', $updated);
vv_conf_edit($currentHost . '.conf', function (string $raw) use ($myId, $folderName): string {
$map = vv_dk_read_conf_map($raw, $myId);
$map = array_filter($map, fn($v) => $v !== $folderName);
return vv_dk_write_conf_map($raw, $myId, $map);
}, [], ["{$myId}_DOCKER_FOLDER_MAP"]);
return ['ok' => true];
}
@@ -421,22 +424,24 @@ function vv_dk_sync_json_to_conf(): array {
$data = vv_dk_read_json();
$currentHost = vv_detect_host();
$myId = strtoupper($currentHost);
$raw = vv_read_conf_raw($currentHost . '.conf');
_vv_dk_sync_conf_from_json($data, $raw, $currentHost, $myId);
_vv_dk_sync_conf_from_json($data, $currentHost, $myId);
return ['ok' => true];
}
// Internal: rebuild conf map from current json state and write it
function _vv_dk_sync_conf_from_json(array $data, string $raw = '', string $host = '', string $id = ''): void {
// Internal: rebuild conf map from current json state and write it.
// The contents to splice into are read inside vv_conf_edit()'s lock. The caller used to be able
// to hand in a copy it had already read; that was only ever an optimisation, and passing a stale
// copy would have written the rest of the conf back as it looked before the lock was taken.
function _vv_dk_sync_conf_from_json(array $data, string $host = '', string $id = ''): void {
if (!$host) $host = vv_detect_host();
if (!$id) $id = strtoupper($host);
if (!$raw) $raw = vv_read_conf_raw($host . '.conf');
$map = [];
foreach ($data as $f) {
$name = $f['name'] ?? '';
foreach ($f['containers'] ?? [] as $c) $map[$c] = $name;
}
$updated = vv_dk_write_conf_map($raw, $id, $map);
vv_write_conf_raw($host . '.conf', $updated);
vv_conf_edit($host . '.conf', fn(string $raw): string => vv_dk_write_conf_map($raw, $id, $map),
[], ["{$id}_DOCKER_FOLDER_MAP"]);
}
+44 -38
View File
@@ -598,50 +598,56 @@ function vv_conf_flag_value(string $name): bool {
}
// Write a boolean flag value to master.conf.
// Goes through vv_conf_edit() for the lock, the pre-write backup, the syntax check and the audit
// line. This used to call vv_write_conf_raw() directly, which gave it tmp+rename atomicity and
// nothing else — no backup, and no check that the file still sourced afterwards.
function vv_conf_flag_set(string $name, bool $value): bool {
$confPath = CONF_DIR . '/master.conf';
$content = file_get_contents($confPath);
if ($content === false) return false;
$val = $value ? 'true' : 'false';
$new = preg_replace(
'/^(\s*' . preg_quote($name, '/') . '\s*=\s*)(true|false)(\s*(?:#.*)?)$/m',
'${1}' . $val . '${3}',
$content, -1, $count
);
if (!$count) return false;
// tmp+rename — every script sources master.conf, so a truncated write here is a
// system-wide outage, not a lost toggle.
return vv_write_conf_raw('master.conf', $new);
if (!vv_conf_key_valid($name)) return false;
$val = $value ? 'true' : 'false';
return vv_conf_edit('master.conf', function (string $content) use ($name, $val): ?string {
$new = preg_replace(
'/^(\s*' . preg_quote($name, '/') . '\s*=\s*)(true|false)(\s*(?:#.*)?)$/m',
'${1}' . $val . '${3}',
$content, -1, $count
);
// A name that matches no true/false line is a caller error, not an already-correct
// state — unlike the membership toggle below, where absence genuinely means nothing
// to do. Returning null keeps the write from happening and logs reason=no-match.
return $count ? $new : null;
}, [$name => $val]);
}
// Comment or uncomment a script's line in the first master.conf array that contains it.
// Goes through vv_conf_edit() for the lock, the pre-write backup, the syntax check and the audit
// line — see vv_conf_flag_set() above for what that replaced. There is no key to verify here,
// so the audit subject is the script id and a clean source is the whole assertion.
function vv_conf_toggle_script(string $rel, bool $enable): bool {
$confPath = CONF_DIR . '/master.conf';
$lines = file($confPath, FILE_KEEP_BLANK_LINES);
if (!$lines) return false;
$changed = false;
$inArray = false;
$relEsc = preg_quote($rel, '/');
foreach ($lines as &$line) {
if (preg_match('/^\s*[A-Z_]+_SCRIPTS\s*=\s*\(/', $line)) $inArray = true;
if ($inArray && preg_match('/^\s*\)\s*(?:#.*)?$/', $line) && !str_contains($line, '(')) $inArray = false;
if (!$inArray) continue;
if (!preg_match('/^\s*(?:#\s*)?"' . $relEsc . '(?:\s[^"]*)?"/', $line)) continue;
$isCommented = (bool)preg_match('/^\s*#/', $line);
if ($enable && $isCommented) {
$line = preg_replace('/^(\s*)#\s*("' . $relEsc . ')/', '$1$2', $line);
$changed = true;
} elseif (!$enable && !$isCommented) {
$line = preg_replace('/^(\s*)("' . $relEsc . ')/', '$1# $2', $line);
$changed = true;
return vv_conf_edit('master.conf', function (string $content) use ($rel, $enable): ?string {
$lines = preg_split('/(?<=\n)/', $content) ?: [];
$changed = false;
$inArray = false;
$relEsc = preg_quote($rel, '/');
foreach ($lines as &$line) {
if (preg_match('/^\s*[A-Z_]+_SCRIPTS\s*=\s*\(/', $line)) $inArray = true;
if ($inArray && preg_match('/^\s*\)\s*(?:#.*)?$/', $line) && !str_contains($line, '(')) $inArray = false;
if (!$inArray) continue;
if (!preg_match('/^\s*(?:#\s*)?"' . $relEsc . '(?:\s[^"]*)?"/', $line)) continue;
$isCommented = (bool)preg_match('/^\s*#/', $line);
if ($enable && $isCommented) {
$line = preg_replace('/^(\s*)#\s*("' . $relEsc . ')/', '$1$2', $line);
$changed = true;
} elseif (!$enable && !$isCommented) {
$line = preg_replace('/^(\s*)("' . $relEsc . ')/', '$1# $2', $line);
$changed = true;
}
break;
}
break;
}
unset($line);
if (!$changed) return true;
// tmp+rename — every script sources master.conf, so a truncated write here is a
// system-wide outage, not a lost toggle.
return vv_write_conf_raw('master.conf', implode('', $lines));
unset($line);
// A script in no array has nothing to toggle and the conf already reads the way the
// caller asked. Returning the content unchanged reports success without a write.
return $changed ? implode('', $lines) : $content;
}, [], [$rel]);
}
// Parse an orchestrator script to find which child scripts it calls.
+37 -6
View File
@@ -144,6 +144,14 @@
# clearing path requires seeing the container running again, which never happens once
# it is uninstalled. Without pruning, one removed container makes this host report
# unhealthy permanently. Reserved daemon_* keys in WATCHDOG_STATE_FILE are never pruned.
#
# WATCHDOG_STATE_FILE is keyed two ways and the prune has to know it. The container check
# stores a bare name; the HTTP, API, CPU and docker checks store container + suffix. The
# existence test is always against the base container — inspecting the composite key means
# asking docker about "Emby_http", which fails while Emby is running perfectly, and deleting
# the live counter every cycle. That made RESP_FAIL_LIMIT and CPU_FAIL_LIMIT unreachable,
# since a counter reset each cycle never reaches 2. Only the four known suffixes are
# stripped: a general split on the last underscore would break PostgreSQL_Immich.
# Pruning runs under --dry-run as well: the documented contract for that flag is that
# nothing gets restarted, and dropping a record of a container that no longer exists is
# housekeeping, not an action. It is idempotent — repeated runs converge.
@@ -732,15 +740,38 @@ CYCLE_START=$(date +%s)
if [[ -s "$WATCHDOG_STATE_FILE" ]]; then
mapfile -t _prune_snapshot < "$WATCHDOG_STATE_FILE"
for _prune_line in "${_prune_snapshot[@]}"; do
_prune_container="${_prune_line%%:*}"
[[ -z "$_prune_container" ]] && continue
[[ "$_prune_container" == daemon_* ]] && continue # reserved keys, not containers
_prune_key="${_prune_line%%:*}"
[[ -z "$_prune_key" ]] && continue
[[ "$_prune_key" == daemon_* ]] && continue # reserved keys, not containers
# Keys come in two shapes: the bare container name for the container check, and
# container + check suffix for the others. `docker inspect Emby_http` fails for a
# perfectly healthy Emby, so inspecting the key itself deleted every live per-check
# counter on every cycle — before the checks below read them. RESP_FAIL_LIMIT and
# CPU_FAIL_LIMIT of 2 were therefore unreachable: a counter wiped each cycle can
# only ever reach 1. Latent rather than harmful only because no container here had
# yet failed one of those checks.
#
# Strip only the four known suffixes, never on the last underscore: PostgreSQL_Immich
# is a real container, and splitting it would prune a name that does exist.
#
# Known limit: a container literally named something_http or something_docker would
# have its own counter judged by whether "something" exists. No container here ends
# in one of the four, and the alternative — inspecting the key, then the base — pays
# a second docker call for every composite key on every cycle to cover a name nobody
# uses. Worth revisiting only alongside replacing these per-key inspects with one
# `docker ps -a` membership test.
_prune_container="$_prune_key"
case "$_prune_key" in
*_http|*_api|*_cpu|*_docker) _prune_container="${_prune_key%_*}" ;;
esac
if ! timeout "$DOCKER_TIMEOUT" docker inspect "$_prune_container" &>/dev/null; then
sed -i "/^${_prune_container}:/d" "$WATCHDOG_STATE_FILE" 2>/dev/null
warn "$_prune_container no longer exists — removed from $(basename "$WATCHDOG_STATE_FILE")"
sed -i "/^${_prune_key}:/d" "$WATCHDOG_STATE_FILE" 2>/dev/null
warn "$_prune_container no longer exists — removed $_prune_key from $(basename "$WATCHDOG_STATE_FILE")"
fi
done
unset _prune_snapshot _prune_line _prune_container
unset _prune_snapshot _prune_line _prune_key _prune_container
fi
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
+11 -1
View File
@@ -2008,7 +2008,17 @@ run_orch_child() {
return 0
else
_ec=$?
error "$label — failed (exit $_ec, $(format_duration $(( $(date +%s) - _start ))))"
# Match the severity ladder run_job.sh already records against: exit 1 is a warning,
# exit 2+ is an error. The status in the run record was always derived that way, so a
# child exiting 1 produced a run marked "warn" whose log was full of ❌ ERROR lines —
# and the log is the louder of the two. A job that skipped work it was told to skip read
# exactly like a job that broke.
#
# The label still lands in JOB_FAIL either way. Whether the child succeeded is a separate
# question from how loudly to report it, and the orchestrator's own exit code depends on
# the first one.
local _msg="$label — failed (exit $_ec, $(format_duration $(( $(date +%s) - _start ))))"
if (( _ec == 1 )); then warn "$_msg"; else error "$_msg"; fi
JOB_FAIL+=("$label")
return 1
fi
+9
View File
@@ -19,6 +19,8 @@ data/
├── cache/ persistent backups of the tmpfs caches — and only those
│ ├── arr/ *_tracked_cache.json, restored into tmpfs on demand
│ └── conf/ partner host*.conf snapshot (0700 — holds credentials)
├── Backups/ point-in-time copies kept so a bad write can be undone
│ └── Confs/ pre-write copies of this host's own confs (0700 — holds credentials)
└── logs/ retained log output
```
@@ -31,6 +33,8 @@ variable is what a storage-mode migration rewrites.
| `state/` | `STATE_DIR` | ALL state files must use this. No `/tmp`, no repo root. |
| `ai/` | `AI_DATA_DIR` | |
| `cache/` | `CACHE_BACKUP_DIR` | `ARR_CACHE_BACKUP_DIR`, `PERSISTENT_CONF_CACHE` sit under it |
| `Backups/` | `BACKUP_DIR` | parent only; each kind of backup gets a subdirectory |
| `Backups/Confs/` | `CONF_BACKUP_DIR` | retained per `CONF_BACKUP_RETAIN`; **not** a cache — see below |
| `logs/` | `LOG_ARCHIVE_DIR` | live logging still goes to `LOG_DIR` (`/var/log/varaverk`) |
---
@@ -74,6 +78,11 @@ Ask what happens if it is deleted.
called — `lidarr_art_miss_cache.tsv` has "cache" in its name and lives in `db/` for exactly that
reason.
`Backups/` is the same trap from the other side: it has "backup" in its name but backs up nothing
that exists elsewhere. The confs are gitignored, so a pre-write copy under `Backups/Confs/` is the
only prior version of that file anywhere. Deleting it loses something permanently, which is why it
is a root of its own and not a subdirectory of `cache/`.
---
## History