Compare commits
10
Commits
78c9875d92
...
f2d4d2fae1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f2d4d2fae1 | ||
|
|
e91a74d31f | ||
|
|
955f50b94e | ||
|
|
a8654280a7 | ||
|
|
083b5ec5ad | ||
|
|
500f9d92c8 | ||
|
|
67eabdc17c | ||
|
|
d9f917ecef | ||
|
|
eb4512aee3 | ||
|
|
0785a46fe3 |
+4
-1
@@ -3,7 +3,10 @@
|
|||||||
# Templates (*.template) are safe and remain tracked.
|
# Templates (*.template) are safe and remain tracked.
|
||||||
Configurations/host*.conf
|
Configurations/host*.conf
|
||||||
Configurations/master.conf
|
Configurations/master.conf
|
||||||
Configurations/*.bak
|
# *.bak alone missed conf_upgrade's real output — it writes host1.conf.bak-20260802, which does
|
||||||
|
# not end in .bak — so those sat untracked rather than ignored, one `git add -A` from being
|
||||||
|
# pushed. The glob has to cover the suffix.
|
||||||
|
Configurations/*.bak*
|
||||||
.vscode
|
.vscode
|
||||||
|
|
||||||
# ── Personal scratch notes — dev-only, never pushed ───────────────────────────
|
# ── Personal scratch notes — dev-only, never pushed ───────────────────────────
|
||||||
|
|||||||
@@ -299,6 +299,31 @@ Reopened chats render as plain turns: sources, reasoning and timings describe on
|
|||||||
are not stored, because redrawing them beside a transcript that may be continued under a
|
are not stored, because redrawing them beside a transcript that may be continued under a
|
||||||
different profile would be citing evidence for an answer no longer being made.
|
different profile would be citing evidence for an answer no longer being made.
|
||||||
|
|
||||||
|
### Secrets are redacted on the way to disk
|
||||||
|
|
||||||
|
A conversation about settings is a conversation containing credentials — asking for an API key to
|
||||||
|
be changed means typing one. Message bodies are redacted in `vv_ai_chat_save()`, and the question
|
||||||
|
is redacted again before it reaches `ai.log`.
|
||||||
|
|
||||||
|
**On the way out, never in flight.** The live turn keeps the real value, because the model needs
|
||||||
|
it to carry out what was asked. What it does not need is that value still in the transcript a week
|
||||||
|
later — and a stored chat is replayed into a later prompt when reopened, so an unredacted one
|
||||||
|
would hand the credential back on every subsequent turn, indefinitely.
|
||||||
|
|
||||||
|
Two passes, because they catch different things:
|
||||||
|
|
||||||
|
| Pass | Catches | Method |
|
||||||
|
|---|---|---|
|
||||||
|
| Known values | a credential this host already holds | exact match against secret-shaped conf keys, longest first |
|
||||||
|
| Assignment shapes | a credential arriving that is not in the conf yet | `NAME=value`, `"api_key": value`, "set the token to …" |
|
||||||
|
|
||||||
|
The second pass is the one that matters for settings changes: *"change the Emby API key to X"* is
|
||||||
|
a secret arriving, and X matches nothing on disk until after the write it is requesting.
|
||||||
|
|
||||||
|
Ordinary prose is left alone — the patterns anchor on a secret-shaped *name*, so `CACHE_WARN_GB=100`
|
||||||
|
and "turn off the zfs scrub" pass through untouched. `vv_conf_key_is_secret()` is shared with the
|
||||||
|
conf audit log, so the two cannot disagree about what counts as a secret.
|
||||||
|
|
||||||
## ━━━ TOKEN ACCOUNTING ━━━
|
## ━━━ TOKEN ACCOUNTING ━━━
|
||||||
|
|
||||||
Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`):
|
Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`):
|
||||||
|
|||||||
@@ -144,6 +144,8 @@
|
|||||||
AI_DATA_DIR="${DATA_DIR}/ai"
|
AI_DATA_DIR="${DATA_DIR}/ai"
|
||||||
CACHE_BACKUP_DIR="${DATA_DIR}/cache"
|
CACHE_BACKUP_DIR="${DATA_DIR}/cache"
|
||||||
LOG_ARCHIVE_DIR="${DATA_DIR}/logs"
|
LOG_ARCHIVE_DIR="${DATA_DIR}/logs"
|
||||||
|
BACKUP_DIR="${DATA_DIR}/Backups"
|
||||||
|
CONF_BACKUP_DIR="${BACKUP_DIR}/Confs" # 0700 — holds credentials
|
||||||
PERSISTENT_CONF_CACHE="${CACHE_BACKUP_DIR}/conf"
|
PERSISTENT_CONF_CACHE="${CACHE_BACKUP_DIR}/conf"
|
||||||
ARR_CACHE_BACKUP_DIR="${CACHE_BACKUP_DIR}/arr"
|
ARR_CACHE_BACKUP_DIR="${CACHE_BACKUP_DIR}/arr"
|
||||||
|
|
||||||
@@ -401,8 +403,13 @@
|
|||||||
"Watchdogs/resource_watchdog.sh" # reduce system pressure before healing attempts
|
"Watchdogs/resource_watchdog.sh" # reduce system pressure before healing attempts
|
||||||
"Watchdogs/docker_watchdog.sh" # heal containers with freed resources
|
"Watchdogs/docker_watchdog.sh" # heal containers with freed resources
|
||||||
"Watchdogs/system_watchdog.sh" # system component health — storage + webgui
|
"Watchdogs/system_watchdog.sh" # system component health — storage + webgui
|
||||||
|
"Plugin/unraid/Tools/ai_repair_sweep.sh" # read what the last cycle logged; off unless AI_REPAIR_ENABLED
|
||||||
"Watchdogs/stability_watchdog.sh" # reboot if all else fails — last line of defense
|
"Watchdogs/stability_watchdog.sh" # reboot if all else fails — last line of defense
|
||||||
)
|
)
|
||||||
|
# The repair sweep sits ahead of stability deliberately, which is the one exception to "stability
|
||||||
|
# last". It reads the previous cycle's logs and may correct the very misconfiguration stability
|
||||||
|
# would otherwise reboot for — a wrong port is not fixed by restarting the machine. It is bounded
|
||||||
|
# by AI_PROBE_TIMEOUT, exits 0 in every case, and does nothing at all unless AI_REPAIR_ENABLED.
|
||||||
|
|
||||||
# ━━━ System Watchdog ━━━
|
# ━━━ System Watchdog ━━━
|
||||||
# system_watchdog.sh runs SYSTEM_WATCHDOG_SCRIPTS sequentially each cycle.
|
# system_watchdog.sh runs SYSTEM_WATCHDOG_SCRIPTS sequentially each cycle.
|
||||||
@@ -1659,6 +1666,13 @@
|
|||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ── AI / RAG ──────────────────────────────────────────────────────────────────────────────────
|
# ── AI / RAG ──────────────────────────────────────────────────────────────────────────────────
|
||||||
|
# ━━━ Conf Backups ━━━
|
||||||
|
# Every write through the plugin copies the conf aside first, to CONF_BACKUP_DIR with an
|
||||||
|
# ISO-8601 stamp. The confs are gitignored, so that directory is the whole recovery path — there
|
||||||
|
# is no history to revert to. A backup that cannot be taken cancels the write.
|
||||||
|
# Oldest are pruned past this count, per conf file.
|
||||||
|
CONF_BACKUP_RETAIN=30
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# Varaverk works exactly as well with AI off as with it on. Nothing below is required for any
|
# Varaverk works exactly as well with AI off as with it on. Nothing below is required for any
|
||||||
# script to function — every feature that can lean on AI has a complete non-AI path, and the
|
# script to function — every feature that can lean on AI has a complete non-AI path, and the
|
||||||
@@ -1755,6 +1769,32 @@
|
|||||||
AI_ASSIST_CLEANUP=false # tier 2 — orphan and stuck-import triage
|
AI_ASSIST_CLEANUP=false # tier 2 — orphan and stuck-import triage
|
||||||
AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance
|
AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance
|
||||||
|
|
||||||
|
# ━━━ AI Repair ━━━
|
||||||
|
# Reads the log of a job that has finished, turns known error shapes into findings, and offers
|
||||||
|
# or applies a repair. Two switches, because detecting and repairing are separate things to
|
||||||
|
# trust.
|
||||||
|
#
|
||||||
|
# AI_REPAIR_ENABLED on its own reads, files findings and offers fixes, and writes nothing. Run
|
||||||
|
# it there first — long enough to read what it finds and disagree with some of it.
|
||||||
|
#
|
||||||
|
# AI_REPAIR_AUTOFIX_ENABLED is what allows a value to be written unattended, and only ever a
|
||||||
|
# value a probe has answered on. Never a toggle: whether something should be switched on is a
|
||||||
|
# decision about intent, and a probe cannot prove intent the way it can prove a port answers.
|
||||||
|
AI_REPAIR_ENABLED=false
|
||||||
|
AI_REPAIR_AUTOFIX_ENABLED=false
|
||||||
|
|
||||||
|
# ━━━ AI Repair Findings ━━━
|
||||||
|
# Misconfigurations found in this installation, as opposed to defects in Varaverk — those go to
|
||||||
|
# ai_bugs. A finding is open until the configuration is right, and closes itself when the probe
|
||||||
|
# that proved the fault starts passing. Closed ones are kept for a while, because "this happened
|
||||||
|
# before and here is what fixed it" is worth more than the disk. Open findings are never pruned:
|
||||||
|
# an unresolved problem does not stop mattering because it is old.
|
||||||
|
AI_FINDING_RETAIN_DAYS=90
|
||||||
|
# Seconds a single probe may take. Nothing is written to conf that has not answered a probe, so
|
||||||
|
# this is the budget for proving a candidate — kept short because a sweep may try several, and
|
||||||
|
# an address worth switching to answers quickly or is not worth switching to.
|
||||||
|
AI_PROBE_TIMEOUT=4
|
||||||
|
|
||||||
# ━━━ AI Conf Write Access ━━━
|
# ━━━ AI Conf Write Access ━━━
|
||||||
# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
|
# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
|
||||||
# never credentials, never a container name. An empty whitelist means no writes regardless of
|
# never credentials, never a container name. An empty whitelist means no writes regardless of
|
||||||
|
|||||||
@@ -162,7 +162,7 @@ if ($profile === 'chat') {
|
|||||||
if ($to !== '' && $to !== $profile) {
|
if ($to !== '' && $to !== $profile) {
|
||||||
$profile = $to;
|
$profile = $to;
|
||||||
$escalated = true;
|
$escalated = true;
|
||||||
wlog('handoff chat -> ' . $to . ': ' . mb_substr($question, 0, 80));
|
wlog('handoff chat -> ' . $to . ': ' . mb_substr(vv_ai_redact($question), 0, 80));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,103 @@
|
|||||||
|
<?php
|
||||||
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
|
// PURPOSE
|
||||||
|
// One pass of the repair sweep. Reads the logs of jobs that finished since the last pass,
|
||||||
|
// turns known error shapes into findings, probes for a correction, and either writes a proven
|
||||||
|
// value or leaves the finding for the operator.
|
||||||
|
//
|
||||||
|
// OPERATIONAL MODEL
|
||||||
|
// Called from ai_repair_sweep.sh, which the watchdog orchestrator runs. Everything it does
|
||||||
|
// lives in include/ai_repair.php so the same code path is what the tests exercise; this file
|
||||||
|
// owns only the lock, the log line, and the exit code.
|
||||||
|
//
|
||||||
|
// DESIGN PRINCIPLES
|
||||||
|
// Non-fatal, always.
|
||||||
|
// Exits 0 on a disabled feature, a held lock, or a sweep that found nothing. Repair is an
|
||||||
|
// enhancement — the orchestrator that runs it has real work to do either side, and this
|
||||||
|
// must never be the reason a watchdog cycle reports failure.
|
||||||
|
//
|
||||||
|
// The summary is logged, not the reasoning.
|
||||||
|
// What was scanned, what was found, what was written. The findings themselves are the
|
||||||
|
// record; duplicating their contents into a log would be two copies to keep in step.
|
||||||
|
//
|
||||||
|
// OPERATIONAL SAFEGUARDS
|
||||||
|
// One sweep at a time.
|
||||||
|
// flock, non-blocking. A pass that overruns its fifteen-minute slot must not have a second
|
||||||
|
// copy start probing and writing conf underneath it.
|
||||||
|
//
|
||||||
|
// Nothing is written unless two switches say so.
|
||||||
|
// AI_REPAIR_ENABLED gates the sweep; AI_REPAIR_AUTOFIX_ENABLED gates writing. With only
|
||||||
|
// the first, this reads and files and changes no configuration at all.
|
||||||
|
//
|
||||||
|
// RUNTIME MODES
|
||||||
|
// ai_repair_sweep.php one pass
|
||||||
|
// ai_repair_sweep.php --dry-run probe and report, write nothing, leave the marker alone
|
||||||
|
// ai_repair_sweep.php --status what the last pass did, and what is open
|
||||||
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
|
require_once dirname(__DIR__) . '/include/ai_repair.php';
|
||||||
|
|
||||||
|
$args = array_slice($argv ?? [], 1);
|
||||||
|
$dryRun = in_array('--dry-run', $args, true);
|
||||||
|
$status = in_array('--status', $args, true);
|
||||||
|
|
||||||
|
function rlog(string $msg): void {
|
||||||
|
if (!is_dir(LOG_DIR)) return;
|
||||||
|
@file_put_contents(LOG_DIR . '/ai_repair.log',
|
||||||
|
date('Y-m-d H:i:s') . ' ' . $msg . "\n", FILE_APPEND | LOCK_EX);
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($status) {
|
||||||
|
$last = vv_ai_sweep_last();
|
||||||
|
$open = vv_ai_findings_list(['open', 'needs_operator']);
|
||||||
|
printf("repair: %s\n", vv_ai_repair_enabled() ? 'enabled' : 'disabled');
|
||||||
|
printf("autofix: %s\n", vv_ai_repair_autofix_enabled() ? 'enabled' : 'disabled (detect only)');
|
||||||
|
printf("last pass: %s\n", $last ? date('Y-m-d H:i:s', $last) : 'never');
|
||||||
|
printf("open findings: %d\n", count($open));
|
||||||
|
foreach ($open as $f) {
|
||||||
|
printf(" [%s] %-22s %-28s seen %d — %s\n",
|
||||||
|
$f['severity'] ?? '?', $f['subject'] ?? '?', $f['conf_key'] ?? '?',
|
||||||
|
(int)($f['seen'] ?? 0), $f['state'] ?? '?');
|
||||||
|
}
|
||||||
|
exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!vv_ai_repair_enabled()) exit(0); // silent: a disabled feature is not an event
|
||||||
|
|
||||||
|
$lockPath = VV_CACHE_ROOT . '/ai_repair.lock';
|
||||||
|
if (!is_dir(dirname($lockPath))) @mkdir(dirname($lockPath), 0755, true);
|
||||||
|
$lock = @fopen($lockPath, 'c');
|
||||||
|
if ($lock === false || !flock($lock, LOCK_EX | LOCK_NB)) {
|
||||||
|
if ($lock) fclose($lock);
|
||||||
|
rlog('skipped — a sweep is already running');
|
||||||
|
exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
$t0 = microtime(true);
|
||||||
|
$sum = vv_ai_repair_sweep($dryRun);
|
||||||
|
$ms = (int)round((microtime(true) - $t0) * 1000);
|
||||||
|
|
||||||
|
if (!($sum['ok'] ?? false)) {
|
||||||
|
rlog('sweep refused — ' . ($sum['error'] ?? 'unknown'));
|
||||||
|
exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Nothing found and nothing to say. A line every fifteen minutes reporting no news is how a
|
||||||
|
// log stops being read.
|
||||||
|
if ($sum['findings'] === 0) {
|
||||||
|
if ($sum['runs'] > 0 && $dryRun) rlog(sprintf('dry-run: %d run(s), nothing found (%dms)', $sum['runs'], $ms));
|
||||||
|
exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
rlog(sprintf('%s%d run(s): %d finding(s), %d fixed, %d for the operator, %d resolved, %d quiet (%dms)',
|
||||||
|
$dryRun ? 'dry-run: ' : '', $sum['runs'], $sum['findings'], $sum['fixed'],
|
||||||
|
$sum['needs_operator'], $sum['resolved'], $sum['quiet'], $ms));
|
||||||
|
|
||||||
|
foreach ($sum['details'] as $d) rlog(' ' . $d);
|
||||||
|
|
||||||
|
} finally {
|
||||||
|
flock($lock, LOCK_UN);
|
||||||
|
fclose($lock);
|
||||||
|
}
|
||||||
|
|
||||||
|
exit(0);
|
||||||
Executable
+78
@@ -0,0 +1,78 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# ==============================================================================================
|
||||||
|
# ================================== AI Repair Sweep ===========================================
|
||||||
|
# ==============================================================================================
|
||||||
|
# PURPOSE
|
||||||
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
|
# Reads the logs of jobs that finished since the last pass, turns known error shapes into
|
||||||
|
# findings, probes for a correction, and either writes a proven value or leaves the finding
|
||||||
|
# for the operator to answer.
|
||||||
|
# Runs from the watchdog orchestrator. Off unless AI_REPAIR_ENABLED is true.
|
||||||
|
# ==============================================================================================
|
||||||
|
# OPERATIONAL MODEL
|
||||||
|
# ==============================================================================================
|
||||||
|
# A one-line shim: exec php on ai_repair_sweep.php in the same directory.
|
||||||
|
# The logic is PHP because everything it needs already is — the conf writer with its backups
|
||||||
|
# and read-back verification, the findings store, and the probe layer are all functions the
|
||||||
|
# WebGUI shares. A bash reimplementation would be a second conf writer, which is precisely the
|
||||||
|
# drift the guarded write path exists to prevent.
|
||||||
|
#
|
||||||
|
# There is no post-run hook in Varaverk; nothing fires when a job finishes. The sweep picks up
|
||||||
|
# completed run records instead, so this is one entry in an orchestrator list rather than a
|
||||||
|
# call added to forty scripts.
|
||||||
|
# ==============================================================================================
|
||||||
|
# DESIGN PRINCIPLES
|
||||||
|
# ==============================================================================================
|
||||||
|
# A Shim, Not a Program
|
||||||
|
# This file exists only because the scheduler runs shell scripts and the work is PHP.
|
||||||
|
# Anything added here would be logic the WebGUI cannot reach, and the operator answering a
|
||||||
|
# finding in the browser must take exactly the same path as the sweep that filed it.
|
||||||
|
#
|
||||||
|
# Detecting And Repairing Are Separate Trusts
|
||||||
|
# AI_REPAIR_ENABLED alone reads logs, files findings and proposes fixes, writing nothing.
|
||||||
|
# AI_REPAIR_AUTOFIX_ENABLED is what allows a value to be written, and only ever one a probe
|
||||||
|
# has answered on. Both live in master.conf; neither is set by this script.
|
||||||
|
# ==============================================================================================
|
||||||
|
# OPERATIONAL SAFEGUARDS
|
||||||
|
# ==============================================================================================
|
||||||
|
# Never Fatal
|
||||||
|
# Always exits 0 — on a disabled feature, a held lock, or a failed pass. The watchdog
|
||||||
|
# orchestrator runs real work either side of this, and a repair sweep must never be the
|
||||||
|
# reason a cycle reports failure.
|
||||||
|
#
|
||||||
|
# One Sweep At A Time
|
||||||
|
# The PHP takes a non-blocking flock. A pass that overruns its slot cannot have a second
|
||||||
|
# copy start probing and writing conf underneath it.
|
||||||
|
#
|
||||||
|
# Nothing Is Written That Has Not Answered
|
||||||
|
# A value reaches conf only after a probe got a response from it. Toggles are never written
|
||||||
|
# unattended at all — whether something should be switched on is a decision about intent,
|
||||||
|
# and a probe cannot prove intent.
|
||||||
|
# ==============================================================================================
|
||||||
|
# CONFIGURATION
|
||||||
|
# ==============================================================================================
|
||||||
|
#
|
||||||
|
# AI_ENABLED master switch; nothing here runs without it
|
||||||
|
# AI_REPAIR_ENABLED read logs and file findings
|
||||||
|
# AI_REPAIR_AUTOFIX_ENABLED allow a proven value to be written unattended
|
||||||
|
# AI_PROBE_TIMEOUT seconds a single probe may take
|
||||||
|
# AI_FINDING_RETAIN_DAYS how long closed findings are kept
|
||||||
|
#
|
||||||
|
# ==============================================================================================
|
||||||
|
# RUNTIME MODES
|
||||||
|
# ==============================================================================================
|
||||||
|
#
|
||||||
|
# ai_repair_sweep.sh
|
||||||
|
# One pass. Files findings, applies proven fixes if autofix is on.
|
||||||
|
#
|
||||||
|
# ai_repair_sweep.sh --dry-run
|
||||||
|
# Probes and reports what it would do. Writes no conf and does not move the marker, so the
|
||||||
|
# same runs are examined again next pass.
|
||||||
|
#
|
||||||
|
# ai_repair_sweep.sh --status
|
||||||
|
# Both switches, when the last pass ran, and every open finding.
|
||||||
|
#
|
||||||
|
# ==============================================================================================
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
php "$SCRIPT_DIR/ai_repair_sweep.php" "$@"
|
||||||
@@ -386,8 +386,11 @@ if ($action === 'ask') {
|
|||||||
. ' >/dev/null 2>&1 </dev/null &';
|
. ' >/dev/null 2>&1 </dev/null &';
|
||||||
$out = []; $rc = 0;
|
$out = []; $rc = 0;
|
||||||
exec($cmd, $out, $rc);
|
exec($cmd, $out, $rc);
|
||||||
|
// Redacted before it is logged, for the same reason the stored transcript is: asking the
|
||||||
|
// assistant to set a credential means typing one, and ai.log is neither 0600 nor pruned.
|
||||||
vv_ai_log(sprintf('ask token=%s rc=%d profile=%s kind=%s q=%s',
|
vv_ai_log(sprintf('ask token=%s rc=%d profile=%s kind=%s q=%s',
|
||||||
substr($token, 0, 12), $rc, $profile, $kind ?: '-', mb_substr($question, 0, 80)));
|
substr($token, 0, 12), $rc, $profile, $kind ?: '-',
|
||||||
|
mb_substr(vv_ai_redact($question), 0, 80)));
|
||||||
|
|
||||||
echo json_encode(['ok' => true, 'token' => $token]);
|
echo json_encode(['ok' => true, 'token' => $token]);
|
||||||
exit;
|
exit;
|
||||||
|
|||||||
@@ -43,10 +43,17 @@
|
|||||||
// outside one, so a matching string in a comment or an unrelated variable cannot be
|
// outside one, so a matching string in a comment or an unrelated variable cannot be
|
||||||
// rewritten.
|
// rewritten.
|
||||||
//
|
//
|
||||||
// The write is atomic.
|
// The write is atomic, backed up, verified and logged.
|
||||||
// vv_conf_toggle_script() writes through vv_write_conf_raw() (tmp + rename). Every
|
// vv_conf_toggle_script() goes through vv_conf_edit(), the one guarded conf write path:
|
||||||
// script sources master.conf; a truncated write here would be a system-wide outage
|
// an exclusive lock, a timestamped copy into CONF_BACKUP_DIR, bash -n on the candidate,
|
||||||
// rather than a lost toggle.
|
// tmp + rename to install it, then the installed file is sourced to prove it still loads.
|
||||||
|
// There is no single key to read back for a commented array member, so a clean source is
|
||||||
|
// the whole assertion. Every script sources master.conf; a truncated or unparseable write
|
||||||
|
// here would be a system-wide outage rather than a lost toggle.
|
||||||
|
//
|
||||||
|
// A script in no array writes nothing at all.
|
||||||
|
// The rewrite returns the contents unchanged, which reports success without taking a
|
||||||
|
// backup or touching the file. "Already in the requested state" is not a write.
|
||||||
//
|
//
|
||||||
// REQUEST
|
// REQUEST
|
||||||
// POST id=<Category/script.sh> enabled=0|1
|
// POST id=<Category/script.sh> enabled=0|1
|
||||||
@@ -56,7 +63,7 @@
|
|||||||
// {"ok":false,"error":"POST only"|"Invalid id"|"Failed to write master.conf"}
|
// {"ok":false,"error":"POST only"|"Invalid id"|"Failed to write master.conf"}
|
||||||
//
|
//
|
||||||
// DEPENDS ON
|
// DEPENDS ON
|
||||||
// include/scheduler.php vv_conf_toggle_script() → vv_write_conf_raw()
|
// include/scheduler.php vv_conf_toggle_script() → vv_conf_edit() → vv_write_conf_raw()
|
||||||
// Configurations/master.conf the *_SCRIPTS arrays
|
// Configurations/master.conf the *_SCRIPTS arrays
|
||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
header('Content-Type: application/json');
|
header('Content-Type: application/json');
|
||||||
|
|||||||
@@ -38,13 +38,17 @@
|
|||||||
// is the difference between a rejected save and a silent, total outage, so a file that
|
// is the difference between a rejected save and a silent, total outage, so a file that
|
||||||
// does not parse is refused and the previous version is left untouched.
|
// does not parse is refused and the previous version is left untouched.
|
||||||
//
|
//
|
||||||
// The temp copy is created with tempnam() and always removed.
|
// Checked here via vv_conf_syntax_error() only so the editor can show bash's own
|
||||||
// The candidate is never written next to the real conf and never under a predictable
|
// complaint with a line number. vv_conf_edit() checks again before installing; this one
|
||||||
// name, so a failed validation cannot leave a stray file for a script to source.
|
// is for the message, not the decision.
|
||||||
//
|
//
|
||||||
// The real write is atomic.
|
// The write goes through the one guarded conf path.
|
||||||
// vv_write_conf_raw() writes .vv.tmp and rename()s, so a script sourcing the conf
|
// vv_conf_edit() takes an exclusive lock, copies the previous file into CONF_BACKUP_DIR,
|
||||||
// during the save reads either the old file or the new one, never a half-written one.
|
// re-checks the syntax, installs via .vv.tmp + rename() so a concurrent reader sees the
|
||||||
|
// old file or the new one but never a half-written one, then sources the installed file
|
||||||
|
// to prove it still loads and restores the backup if it does not. The whole-file nature
|
||||||
|
// of this endpoint is why that matters most here: there is no key to verify, so a clean
|
||||||
|
// source is the only assertion available.
|
||||||
//
|
//
|
||||||
// REQUEST
|
// REQUEST
|
||||||
// POST file=<allowed conf name> content=<full file text>
|
// POST file=<allowed conf name> content=<full file text>
|
||||||
@@ -54,10 +58,12 @@
|
|||||||
// {"ok":false,"error":"POST only"|"File not permitted"|"Syntax error: …"|"Failed to write file"}
|
// {"ok":false,"error":"POST only"|"File not permitted"|"Syntax error: …"|"Failed to write file"}
|
||||||
//
|
//
|
||||||
// DEPENDS ON
|
// DEPENDS ON
|
||||||
// include/config.php vv_get_conf_files(), vv_write_conf_raw(), CONF_DIR
|
// include/config.php vv_get_conf_files(), CONF_DIR
|
||||||
|
// include/confform.php vv_conf_syntax_error(), vv_conf_edit()
|
||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
header('Content-Type: application/json');
|
header('Content-Type: application/json');
|
||||||
require_once dirname(__DIR__) . '/include/config.php';
|
require_once dirname(__DIR__) . '/include/config.php';
|
||||||
|
require_once dirname(__DIR__) . '/include/confform.php';
|
||||||
|
|
||||||
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
|
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
|
||||||
echo json_encode(['ok' => false, 'error' => 'POST only']);
|
echo json_encode(['ok' => false, 'error' => 'POST only']);
|
||||||
@@ -75,22 +81,14 @@ if (!$file || !in_array($file, $allowed)) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Every script sources these. A syntax error here takes the whole system down, so the
|
// Every script sources these. A syntax error here takes the whole system down, so the
|
||||||
// candidate is parsed before it is allowed to replace a working file.
|
// candidate is parsed before it is allowed to replace a working file. Checked here as well as
|
||||||
$check = tempnam(sys_get_temp_dir(), 'vvconf');
|
// inside vv_conf_edit() so the editor can show bash's own complaint; the write path only knows
|
||||||
if ($check !== false) {
|
// whether to proceed, not what to tell the person typing.
|
||||||
file_put_contents($check, $content);
|
$syntax = vv_conf_syntax_error($content, $file);
|
||||||
$out = []; $rc = 0;
|
if ($syntax !== null) {
|
||||||
exec('bash -n ' . escapeshellarg($check) . ' 2>&1', $out, $rc);
|
echo json_encode(['ok' => false, 'error' => 'Syntax error: ' . $syntax]);
|
||||||
@unlink($check);
|
exit;
|
||||||
if ($rc !== 0) {
|
|
||||||
$msg = implode(' ', array_filter(array_map('trim', $out)));
|
|
||||||
echo json_encode([
|
|
||||||
'ok' => false,
|
|
||||||
'error' => 'Syntax error: ' . str_replace($check, $file, $msg ?: 'conf does not parse'),
|
|
||||||
]);
|
|
||||||
exit;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
$ok = vv_write_conf_raw($file, $content);
|
$ok = vv_conf_edit($file, fn() => $content, [], ['whole-file']);
|
||||||
echo json_encode(['ok' => $ok, 'error' => $ok ? null : 'Failed to write file']);
|
echo json_encode(['ok' => $ok, 'error' => $ok ? null : 'Failed to write file']);
|
||||||
|
|||||||
@@ -48,9 +48,12 @@
|
|||||||
// missing, malformed, or unexpected parameter disables rather than enables. Failing
|
// missing, malformed, or unexpected parameter disables rather than enables. Failing
|
||||||
// toward off is the safe direction for a flag that starts data movement.
|
// toward off is the safe direction for a flag that starts data movement.
|
||||||
//
|
//
|
||||||
// The conf write is atomic.
|
// The conf write is atomic, backed up, verified and logged.
|
||||||
// vv_conf_flag_set() writes through vv_write_conf_raw() (tmp + rename). Every script
|
// vv_conf_flag_set() goes through vv_conf_edit(), the one guarded conf write path: an
|
||||||
// sources master.conf, so a truncated write would be a system-wide outage.
|
// exclusive lock, a timestamped copy into CONF_BACKUP_DIR, bash -n on the candidate,
|
||||||
|
// tmp + rename to install it, then the file is sourced and the flag read back — a value
|
||||||
|
// that does not come back as asked restores the backup. Every script sources master.conf,
|
||||||
|
// so a truncated or unparseable write would be a system-wide outage.
|
||||||
//
|
//
|
||||||
// The push only happens after a confirmed local write.
|
// The push only happens after a confirmed local write.
|
||||||
// Guarded on $ok, so a failed edit cannot distribute a stale or partly-written conf to
|
// Guarded on $ok, so a failed edit cannot distribute a stale or partly-written conf to
|
||||||
@@ -69,7 +72,7 @@
|
|||||||
// "push":[]}
|
// "push":[]}
|
||||||
//
|
//
|
||||||
// DEPENDS ON
|
// DEPENDS ON
|
||||||
// include/scheduler.php vv_conf_flag_set() → vv_write_conf_raw()
|
// include/scheduler.php vv_conf_flag_set() → vv_conf_edit() → vv_write_conf_raw()
|
||||||
// include/config.php vv_push_master_conf(), vv_push_setup_state()
|
// include/config.php vv_push_master_conf(), vv_push_setup_state()
|
||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
header('Content-Type: application/json');
|
header('Content-Type: application/json');
|
||||||
|
|||||||
@@ -81,6 +81,7 @@
|
|||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
header('Content-Type: application/json');
|
header('Content-Type: application/json');
|
||||||
require_once dirname(__DIR__) . '/include/config.php';
|
require_once dirname(__DIR__) . '/include/config.php';
|
||||||
|
require_once dirname(__DIR__) . '/include/confform.php';
|
||||||
|
|
||||||
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
|
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
|
||||||
echo json_encode(['ok' => false, 'error' => 'POST only']);
|
echo json_encode(['ok' => false, 'error' => 'POST only']);
|
||||||
@@ -106,6 +107,9 @@ if (!file_exists($confPath)) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
$lines = file($confPath, FILE_KEEP_BLANK_LINES);
|
$lines = file($confPath, FILE_KEEP_BLANK_LINES);
|
||||||
|
// Captured before anything is rebuilt from it — this is what the write compares against to
|
||||||
|
// prove master.conf did not change while the move was being worked out.
|
||||||
|
$origConf = implode('', $lines ?: []);
|
||||||
if (!$lines) {
|
if (!$lines) {
|
||||||
echo json_encode(['ok' => false, 'error' => 'Could not read master.conf']);
|
echo json_encode(['ok' => false, 'error' => 'Could not read master.conf']);
|
||||||
exit;
|
exit;
|
||||||
@@ -148,7 +152,14 @@ if ($toArray) {
|
|||||||
$newLines = $resultLines;
|
$newLines = $resultLines;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!vv_write_conf_raw('master.conf', implode('', $newLines))) {
|
// The array was rebuilt from a copy read before the lock was taken. Handing vv_conf_edit() a
|
||||||
|
// closure that compares against the current contents turns that into an optimistic check: if
|
||||||
|
// anything changed master.conf in between, the move is abandoned rather than written over the
|
||||||
|
// top of it. Everything else — backup, bash -n, read-back, audit — comes with the shared path.
|
||||||
|
$ok = vv_conf_edit('master.conf', fn(string $cur): ?string =>
|
||||||
|
$cur === $origConf ? implode('', $newLines) : null, [], [$script]);
|
||||||
|
|
||||||
|
if (!$ok) {
|
||||||
echo json_encode(['ok' => false, 'error' => 'Write failed']);
|
echo json_encode(['ok' => false, 'error' => 'Write failed']);
|
||||||
exit;
|
exit;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -45,9 +45,13 @@
|
|||||||
// The temp copy is created with tempnam() and always removed, so a rejected save cannot
|
// The temp copy is created with tempnam() and always removed, so a rejected save cannot
|
||||||
// leave a stray file beside the real conf for a script to source.
|
// leave a stray file beside the real conf for a script to source.
|
||||||
//
|
//
|
||||||
// The real write is atomic.
|
// The write goes through the one guarded conf path.
|
||||||
// vv_write_conf_raw() writes .vv.tmp and rename()s, so a script sourcing the conf during
|
// vv_conf_edit() takes an exclusive lock, copies the previous file into CONF_BACKUP_DIR,
|
||||||
// the save reads either the old file or the new one, never a half-written one.
|
// re-checks the syntax, installs via .vv.tmp + rename() so a concurrent reader sees the
|
||||||
|
// old file or the new one but never a half-written one, then sources the installed file
|
||||||
|
// to prove it still loads and restores the backup if it does not. That backup matters
|
||||||
|
// more here than anywhere else: this endpoint replaces a whole hand-edited file, and the
|
||||||
|
// confs are gitignored, so before it existed a bad paste had nothing to go back to.
|
||||||
//
|
//
|
||||||
// The push only happens after a confirmed write.
|
// The push only happens after a confirmed write.
|
||||||
// Guarded on $written, so a failed save cannot distribute a stale or partly-written
|
// Guarded on $written, so a failed save cannot distribute a stale or partly-written
|
||||||
@@ -68,11 +72,13 @@
|
|||||||
// {"ok":false,"error":"Not allowed"|"Syntax error: …"|"Method not allowed"}
|
// {"ok":false,"error":"Not allowed"|"Syntax error: …"|"Method not allowed"}
|
||||||
//
|
//
|
||||||
// DEPENDS ON
|
// DEPENDS ON
|
||||||
// include/config.php vv_get_conf_files(), vv_read_conf_raw(), vv_write_conf_raw(),
|
// include/config.php vv_get_conf_files(), vv_read_conf_raw(),
|
||||||
// vv_push_master_conf(), vv_push_setup_state()
|
// vv_push_master_conf(), vv_push_setup_state()
|
||||||
|
// include/confform.php vv_conf_syntax_error(), vv_conf_edit()
|
||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
header('Content-Type: application/json');
|
header('Content-Type: application/json');
|
||||||
require_once dirname(__DIR__) . '/include/config.php';
|
require_once dirname(__DIR__) . '/include/config.php';
|
||||||
|
require_once dirname(__DIR__) . '/include/confform.php';
|
||||||
|
|
||||||
$allowed = vv_get_conf_files();
|
$allowed = vv_get_conf_files();
|
||||||
|
|
||||||
@@ -95,24 +101,15 @@ if ($_SERVER['REQUEST_METHOD'] === 'POST') {
|
|||||||
}
|
}
|
||||||
// Every script sources these, and master.conf is pushed to every partner from here — a
|
// Every script sources these, and master.conf is pushed to every partner from here — a
|
||||||
// syntax error saved through this endpoint would propagate the outage across the mesh.
|
// syntax error saved through this endpoint would propagate the outage across the mesh.
|
||||||
$check = tempnam(sys_get_temp_dir(), 'vvconf');
|
// Checked here as well as inside vv_conf_edit() so the editor can show bash's own complaint;
|
||||||
if ($check !== false) {
|
// the write path only knows whether to proceed, not what to tell the person typing.
|
||||||
file_put_contents($check, $content);
|
$syntax = vv_conf_syntax_error($content, $file);
|
||||||
$out = []; $rc = 0;
|
if ($syntax !== null) {
|
||||||
exec('bash -n ' . escapeshellarg($check) . ' 2>&1', $out, $rc);
|
echo json_encode(['ok' => false, 'error' => 'Syntax error: ' . $syntax, 'push' => []]);
|
||||||
@unlink($check);
|
exit;
|
||||||
if ($rc !== 0) {
|
|
||||||
$msg = implode(' ', array_filter(array_map('trim', $out)));
|
|
||||||
echo json_encode([
|
|
||||||
'ok' => false,
|
|
||||||
'error' => 'Syntax error: ' . str_replace($check, $file, $msg ?: 'conf does not parse'),
|
|
||||||
'push' => [],
|
|
||||||
]);
|
|
||||||
exit;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
$written = vv_write_conf_raw($file, $content);
|
$written = vv_conf_edit($file, fn() => $content, [], ['whole-file']);
|
||||||
$push = [];
|
$push = [];
|
||||||
if ($written && $file === 'master.conf') {
|
if ($written && $file === 'master.conf') {
|
||||||
$push = vv_push_master_conf();
|
$push = vv_push_master_conf();
|
||||||
|
|||||||
@@ -82,6 +82,7 @@
|
|||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
header('Content-Type: application/json');
|
header('Content-Type: application/json');
|
||||||
require_once dirname(__DIR__) . '/include/config.php';
|
require_once dirname(__DIR__) . '/include/config.php';
|
||||||
|
require_once dirname(__DIR__) . '/include/confform.php';
|
||||||
|
|
||||||
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
|
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
|
||||||
echo json_encode(['ok' => false, 'error' => 'POST only']);
|
echo json_encode(['ok' => false, 'error' => 'POST only']);
|
||||||
@@ -122,6 +123,9 @@ if (!file_exists($confPath)) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
$lines = file($confPath, FILE_KEEP_BLANK_LINES);
|
$lines = file($confPath, FILE_KEEP_BLANK_LINES);
|
||||||
|
// Captured before array_splice rewrites the block in place — this is what the write compares
|
||||||
|
// against to prove master.conf did not change while the new order was being assembled.
|
||||||
|
$origConf = implode('', $lines ?: []);
|
||||||
if (!$lines) {
|
if (!$lines) {
|
||||||
echo json_encode(['ok' => false, 'error' => 'Could not read master.conf']);
|
echo json_encode(['ok' => false, 'error' => 'Could not read master.conf']);
|
||||||
exit;
|
exit;
|
||||||
@@ -183,7 +187,13 @@ $newBlockLines[] = $lines[$blockEnd];
|
|||||||
// Replace the original block in $lines
|
// Replace the original block in $lines
|
||||||
array_splice($lines, $blockStart, $blockEnd - $blockStart + 1, $newBlockLines);
|
array_splice($lines, $blockStart, $blockEnd - $blockStart + 1, $newBlockLines);
|
||||||
|
|
||||||
if (!vv_write_conf_raw('master.conf', implode('', $lines))) {
|
// See movescript.php: the block was rebuilt from a copy read before the lock, so the closure
|
||||||
|
// compares against the current contents and abandons the reorder if anything changed in
|
||||||
|
// between. Backup, bash -n, read-back and audit come with the shared path.
|
||||||
|
$ok = vv_conf_edit('master.conf', fn(string $cur): ?string =>
|
||||||
|
$cur === $origConf ? implode('', $lines) : null, [], [$arrayName]);
|
||||||
|
|
||||||
|
if (!$ok) {
|
||||||
echo json_encode(['ok' => false, 'error' => 'Write failed']);
|
echo json_encode(['ok' => false, 'error' => 'Write failed']);
|
||||||
exit;
|
exit;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -167,6 +167,9 @@ if ($action === 'save' && $_SERVER['REQUEST_METHOD'] === 'POST') {
|
|||||||
if (is_array($scripts)) {
|
if (is_array($scripts)) {
|
||||||
$confPath = CONF_DIR . '/master.conf';
|
$confPath = CONF_DIR . '/master.conf';
|
||||||
$lines = file($confPath, FILE_KEEP_BLANK_LINES) ?: [];
|
$lines = file($confPath, FILE_KEEP_BLANK_LINES) ?: [];
|
||||||
|
// Captured before array_splice rewrites the block in place — the write compares against
|
||||||
|
// it to prove master.conf did not change while the new block was being assembled.
|
||||||
|
$origConf = implode('', $lines);
|
||||||
$esc = preg_quote($scriptsVar, '/');
|
$esc = preg_quote($scriptsVar, '/');
|
||||||
$blockStart = $blockEnd = null;
|
$blockStart = $blockEnd = null;
|
||||||
$depth = 0;
|
$depth = 0;
|
||||||
@@ -213,9 +216,13 @@ if ($action === 'save' && $_SERVER['REQUEST_METHOD'] === 'POST') {
|
|||||||
}
|
}
|
||||||
$newBlock[] = $lines[$blockEnd];
|
$newBlock[] = $lines[$blockEnd];
|
||||||
array_splice($lines, $blockStart, $blockEnd - $blockStart + 1, $newBlock);
|
array_splice($lines, $blockStart, $blockEnd - $blockStart + 1, $newBlock);
|
||||||
// tmp+rename — every script sources master.conf, so a truncated write here is a
|
// Shared guarded path: lock, backup, bash -n, atomic install, read-back, audit. The
|
||||||
// system-wide outage, not a lost edit.
|
// closure compares against the current contents first, so a master.conf that changed
|
||||||
if (!vv_write_conf_raw('master.conf', implode('', $lines))) {
|
// while this block was being assembled abandons the write instead of clobbering it.
|
||||||
|
$wrote = vv_conf_edit('master.conf', fn(string $cur): ?string =>
|
||||||
|
$cur === $origConf ? implode('', $lines) : null, [], [$scriptsVar]);
|
||||||
|
|
||||||
|
if (!$wrote) {
|
||||||
$errors[] = 'scripts write failed';
|
$errors[] = 'scripts write failed';
|
||||||
} else {
|
} else {
|
||||||
// master.conf is shared — mirrors reorderarray/movescript/rawconf.
|
// master.conf is shared — mirrors reorderarray/movescript/rawconf.
|
||||||
|
|||||||
@@ -120,6 +120,7 @@
|
|||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
header('Content-Type: application/json');
|
header('Content-Type: application/json');
|
||||||
require_once dirname(__DIR__) . '/include/config.php';
|
require_once dirname(__DIR__) . '/include/config.php';
|
||||||
|
require_once dirname(__DIR__) . '/include/confform.php';
|
||||||
|
|
||||||
$action = ($_SERVER['REQUEST_METHOD'] === 'GET')
|
$action = ($_SERVER['REQUEST_METHOD'] === 'GET')
|
||||||
? trim($_GET['action'] ?? '')
|
? trim($_GET['action'] ?? '')
|
||||||
@@ -277,7 +278,10 @@ if ($action === 'pull') {
|
|||||||
'${1}"' . $sshKey . '"', $conf);
|
'${1}"' . $sshKey . '"', $conf);
|
||||||
$conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
|
$conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
|
||||||
'${1}' . $storageInternal2, $conf);
|
'${1}' . $storageInternal2, $conf);
|
||||||
vv_write_conf_raw($confFile, $conf);
|
// allowCreate: this is first-run setup, so the host conf does not exist yet. There
|
||||||
|
// is no prior content to back up, and a candidate that fails bash -n is removed
|
||||||
|
// rather than restored.
|
||||||
|
vv_conf_edit($confFile, fn(): string => $conf, [], ["{$hostId}_SSH_KEY"], true);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -313,6 +317,9 @@ if (!preg_match('/^host\d+$/', $mySlot)) {
|
|||||||
|
|
||||||
// Write HOST1 / HOST2 into master.conf
|
// Write HOST1 / HOST2 into master.conf
|
||||||
$master = vv_read_conf_raw('master.conf');
|
$master = vv_read_conf_raw('master.conf');
|
||||||
|
// Captured before the substitutions below — the write compares against it so a master.conf that
|
||||||
|
// changed while setup was being filled in is not silently overwritten.
|
||||||
|
$origMaster = $master;
|
||||||
if ($master === '') {
|
if ($master === '') {
|
||||||
echo json_encode(['ok' => false, 'error' => 'master.conf not found — check SCRIPTS_DIR in varaverk.cfg']);
|
echo json_encode(['ok' => false, 'error' => 'master.conf not found — check SCRIPTS_DIR in varaverk.cfg']);
|
||||||
exit;
|
exit;
|
||||||
@@ -333,7 +340,8 @@ if ($slotNum > 2 && !empty($myHostname)) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!vv_write_conf_raw('master.conf', $master)) {
|
if (!vv_conf_edit('master.conf', fn(string $cur): ?string => $cur === $origMaster ? $master : null,
|
||||||
|
[], ['HOST1', 'HOST2'])) {
|
||||||
echo json_encode(['ok' => false, 'error' => 'Failed to write master.conf']);
|
echo json_encode(['ok' => false, 'error' => 'Failed to write master.conf']);
|
||||||
exit;
|
exit;
|
||||||
}
|
}
|
||||||
@@ -368,7 +376,8 @@ if (!file_exists(CONF_DIR . '/' . $confFile)) {
|
|||||||
'${1}"' . $sshKeyPath . '"', $conf);
|
'${1}"' . $sshKeyPath . '"', $conf);
|
||||||
$conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
|
$conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
|
||||||
'${1}' . $storageInternal, $conf);
|
'${1}' . $storageInternal, $conf);
|
||||||
if (!vv_write_conf_raw($confFile, $conf)) {
|
// allowCreate — see the sibling write above; this is the same first-run create.
|
||||||
|
if (!vv_conf_edit($confFile, fn(): string => $conf, [], ["{$hostId}_SSH_KEY"], true)) {
|
||||||
echo json_encode(['ok' => false, 'error' => "Failed to write $confFile"]);
|
echo json_encode(['ok' => false, 'error' => "Failed to write $confFile"]);
|
||||||
exit;
|
exit;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -78,6 +78,9 @@
|
|||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
require_once __DIR__ . '/config.php';
|
require_once __DIR__ . '/config.php';
|
||||||
|
// vv_conf_key_is_secret() — the same definition of "this key holds a credential" that decides
|
||||||
|
// what the conf audit log redacts. One list, so the two cannot disagree about what a secret is.
|
||||||
|
require_once __DIR__ . '/confform.php';
|
||||||
|
|
||||||
// VV_AI_JOB_DIR and VV_AI_TOKEN_CACHE_DIR are defined in config.php with every other cache path,
|
// VV_AI_JOB_DIR and VV_AI_TOKEN_CACHE_DIR are defined in config.php with every other cache path,
|
||||||
// read from master.conf so this layer and load_config.sh cannot disagree about where they are.
|
// read from master.conf so this layer and load_config.sh cannot disagree about where they are.
|
||||||
@@ -1216,6 +1219,84 @@ function vv_ai_chats_dir(): string {
|
|||||||
return $d;
|
return $d;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Secret redaction ─────────────────────────────────────────────────────────────────────────
|
||||||
|
// A conversation about settings is a conversation that contains credentials. Asking the
|
||||||
|
// assistant to change an API key means typing the key, and a stored transcript is replayed into
|
||||||
|
// a later prompt when the thread is reopened — so without this a secret would be written to
|
||||||
|
// disk in cleartext and then handed back to the model on every subsequent turn, indefinitely.
|
||||||
|
//
|
||||||
|
// Redaction happens on the way to disk and to the log, never to the live turn. The model needs
|
||||||
|
// the real value to carry out the change being asked for; what it does not need is that value
|
||||||
|
// still sitting in the transcript a week later.
|
||||||
|
const VV_AI_REDACTED = '[redacted]';
|
||||||
|
|
||||||
|
// The values this host actually holds, longest first so a key that contains another as a
|
||||||
|
// substring cannot be half-replaced. Only secret-shaped conf keys contribute, and only values
|
||||||
|
// long enough to be a credential — redacting every occurrence of a two-character setting would
|
||||||
|
// shred ordinary prose.
|
||||||
|
function vv_ai_known_secrets(): array {
|
||||||
|
static $cache = null;
|
||||||
|
if ($cache !== null) return $cache;
|
||||||
|
|
||||||
|
$vals = [];
|
||||||
|
foreach (vv_conf_vars() as $k => $v) {
|
||||||
|
$v = trim((string)$v);
|
||||||
|
if (strlen($v) < 8) continue;
|
||||||
|
if (!vv_conf_key_is_secret((string)$k)) continue;
|
||||||
|
// A value that is still a ${...} reference is a template, not a credential.
|
||||||
|
if (str_contains($v, '${')) continue;
|
||||||
|
$vals[] = $v;
|
||||||
|
}
|
||||||
|
$vals = array_values(array_unique($vals));
|
||||||
|
usort($vals, fn($a, $b) => strlen($b) <=> strlen($a));
|
||||||
|
|
||||||
|
return $cache = $vals;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Two passes, because they catch different things.
|
||||||
|
//
|
||||||
|
// Known values catch a credential this host already holds, however it is worded — pasted bare,
|
||||||
|
// quoted, or buried mid-sentence. Exact string matching, so there are no false positives.
|
||||||
|
//
|
||||||
|
// Assignment shapes catch the credential that is not in the conf yet, which is precisely the
|
||||||
|
// case that matters here: "change the Emby API key to <new value>" is a secret arriving, and it
|
||||||
|
// will not match anything on disk until after the write it is asking for.
|
||||||
|
function vv_ai_redact(string $text): string {
|
||||||
|
if ($text === '') return $text;
|
||||||
|
|
||||||
|
foreach (vv_ai_known_secrets() as $secret) {
|
||||||
|
$text = str_replace($secret, VV_AI_REDACTED, $text);
|
||||||
|
}
|
||||||
|
|
||||||
|
// NAME=value / "api_key": value / api-key: value — anchored on a secret-shaped name so an
|
||||||
|
// ordinary "count=12" is untouched.
|
||||||
|
$text = preg_replace(
|
||||||
|
'/\b([A-Za-z0-9_\-]*(?:api[_\-]?key|password|passwd|secret|token|_pass|apikey)[A-Za-z0-9_\-]*)'
|
||||||
|
. '(\s*["\']?\s*[:=]\s*["\']?)([^\s"\',;]{6,})/i',
|
||||||
|
'$1$2' . VV_AI_REDACTED,
|
||||||
|
$text
|
||||||
|
) ?? $text;
|
||||||
|
|
||||||
|
// "set the api key to <value>", "password is <value>" — the same secret arriving as prose
|
||||||
|
// rather than as an assignment.
|
||||||
|
$text = preg_replace(
|
||||||
|
'/\b((?:api[ _\-]?key|password|passphrase|secret|token)\s+(?:to|is|as|=)\s+)["\']?([^\s"\',;]{6,})/i',
|
||||||
|
'$1' . VV_AI_REDACTED,
|
||||||
|
$text
|
||||||
|
) ?? $text;
|
||||||
|
|
||||||
|
return $text;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Redacts every message body in place, leaving roles and any other metadata alone.
|
||||||
|
function vv_ai_redact_messages(array $messages): array {
|
||||||
|
foreach ($messages as &$m) {
|
||||||
|
if (isset($m['content']) && is_string($m['content'])) $m['content'] = vv_ai_redact($m['content']);
|
||||||
|
}
|
||||||
|
unset($m);
|
||||||
|
return $messages;
|
||||||
|
}
|
||||||
|
|
||||||
function vv_ai_chats_max(): int {
|
function vv_ai_chats_max(): int {
|
||||||
$n = (int)(vv_conf_vars()['AI_CHAT_HISTORY_MAX'] ?? 10);
|
$n = (int)(vv_conf_vars()['AI_CHAT_HISTORY_MAX'] ?? 10);
|
||||||
return max(1, min(50, $n));
|
return max(1, min(50, $n));
|
||||||
@@ -1308,6 +1389,12 @@ function vv_ai_chat_save(string $id, string $profile, array $messages, string $s
|
|||||||
$prev = vv_ai_chat_read($id);
|
$prev = vv_ai_chat_read($id);
|
||||||
$created = (int)($prev['created'] ?? time());
|
$created = (int)($prev['created'] ?? time());
|
||||||
|
|
||||||
|
// Redacted here rather than at the point the message was composed, because the live turn
|
||||||
|
// needs the real value to carry out what was asked. This is the boundary between "in flight"
|
||||||
|
// and "on disk", and it is the last place the cleartext exists. The title is derived after,
|
||||||
|
// so a credential cannot survive in the conversation list either.
|
||||||
|
$messages = vv_ai_redact_messages($messages);
|
||||||
|
|
||||||
// Scope travels with the conversation. A Scheduler thread is bound to what the operator had
|
// Scope travels with the conversation. A Scheduler thread is bound to what the operator had
|
||||||
// open — a script, a log, a conf key — and its turns carry log excerpts chosen for that
|
// open — a script, a log, a conf key — and its turns carry log excerpts chosen for that
|
||||||
// thing. Storing the scope means reopening the thread anywhere restores the context it was
|
// thing. Storing the scope means reopening the thread anywhere restores the context it was
|
||||||
|
|||||||
@@ -0,0 +1,907 @@
|
|||||||
|
<?php
|
||||||
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
|
// PURPOSE
|
||||||
|
// Findings about this installation being misconfigured, and the record of what was done about
|
||||||
|
// them. A finding is "Emby is not answering at the address the conf gives", not "Varaverk has
|
||||||
|
// a bug" — the second is what ai_bugs holds.
|
||||||
|
//
|
||||||
|
// OPERATIONAL MODEL
|
||||||
|
// A sweep reads a completed run's log, deterministic patterns turn error lines into typed
|
||||||
|
// findings, and each finding is either repaired or handed to the operator. Findings persist
|
||||||
|
// because the repair may need something only a human can supply, and that conversation has to
|
||||||
|
// survive the page being closed.
|
||||||
|
//
|
||||||
|
// WHY THIS IS NOT ai_bugs
|
||||||
|
// Same storage shape, different lifecycle, and the difference is the whole reason for a second
|
||||||
|
// store. A bug is open until Varaverk's code changes; nothing on this host can close it. A
|
||||||
|
// finding is open until this host's configuration is right, and the same probe that proved a
|
||||||
|
// fix can later prove the fault is gone — so findings close themselves and bugs cannot.
|
||||||
|
//
|
||||||
|
// Filing them together would mean a list where half the rows are actionable by the operator
|
||||||
|
// and half are actionable by whoever maintains the project, with no way to tell which is which
|
||||||
|
// except by reading them.
|
||||||
|
//
|
||||||
|
// DESIGN PRINCIPLES
|
||||||
|
// A finding names a conf key or it is not a finding.
|
||||||
|
// The point of the record is that something can be done about it. "The daily sync looked
|
||||||
|
// unhappy" is a feeling; "HOST1_EMBY_URL points at a host that refuses connections" is a
|
||||||
|
// finding. The triage patterns that cannot resolve a key produce nothing rather than a
|
||||||
|
// vague row.
|
||||||
|
//
|
||||||
|
// Evidence is the log line, quoted.
|
||||||
|
// Same rule as ai_bugs, for the same reason: a finding that cannot show the line it came
|
||||||
|
// from cannot be checked, and this store is meant to be checkable.
|
||||||
|
//
|
||||||
|
// Identity is kind + subject + key, not the message text.
|
||||||
|
// A port that has been wrong for a week is one finding seen 400 times, not 400 findings.
|
||||||
|
// Wording drifts as logs change; the thing being wrong does not.
|
||||||
|
//
|
||||||
|
// OPERATIONAL SAFEGUARDS
|
||||||
|
// A proposed value is recorded, never trusted.
|
||||||
|
// 'proposed' is what something might be changed to; 'proven' is whether a probe actually
|
||||||
|
// got an answer from it. Only proven values are ever written to conf, and the two fields
|
||||||
|
// are kept separate so a record cannot imply verification it did not have.
|
||||||
|
//
|
||||||
|
// Closing is evidence-driven, not time-driven.
|
||||||
|
// A finding closes when its probe passes or the operator dismisses it. It does not expire,
|
||||||
|
// because "we stopped seeing it in the log" is equally consistent with the job no longer
|
||||||
|
// running at all.
|
||||||
|
//
|
||||||
|
// Under data/ and therefore gitignored: these quote this installation's logs and name its
|
||||||
|
// hosts, ports and containers.
|
||||||
|
//
|
||||||
|
// EXPORTS
|
||||||
|
// vv_ai_findings_dir() the store
|
||||||
|
// vv_ai_finding_write() file or increment one finding
|
||||||
|
// vv_ai_findings_list() findings, newest activity first
|
||||||
|
// vv_ai_finding_get() one by id
|
||||||
|
// vv_ai_finding_close() mark resolved, with how
|
||||||
|
// vv_ai_finding_dismiss() operator says this is not a problem
|
||||||
|
// vv_ai_findings_for_chat() the open ones worth opening a conversation about
|
||||||
|
//
|
||||||
|
// CONFIGURATION
|
||||||
|
// AI_DATA_DIR findings live in ai_findings/ beneath it
|
||||||
|
// AI_FINDING_RETAIN_DAYS closed findings older than this are removed (default 90)
|
||||||
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
|
require_once __DIR__ . '/config.php';
|
||||||
|
require_once __DIR__ . '/ai.php';
|
||||||
|
|
||||||
|
// What a finding can be. The kind decides which repair is even conceivable, so an unknown kind
|
||||||
|
// is refused rather than stored as an untyped row nothing knows how to act on.
|
||||||
|
const VV_AI_FINDING_KINDS = [
|
||||||
|
'unreachable' => 'a configured address or port refused, timed out, or did not resolve',
|
||||||
|
'auth_rejected' => 'the endpoint answered, and rejected the credential',
|
||||||
|
'unknown_target' => 'a conf entry names a container or share that does not exist here',
|
||||||
|
'missing_value' => 'a conf key required by the job that ran is empty',
|
||||||
|
];
|
||||||
|
|
||||||
|
// How a finding ended, when it ends.
|
||||||
|
const VV_AI_FINDING_STATES = [
|
||||||
|
'open' => 'seen, not yet acted on',
|
||||||
|
'needs_operator' => 'cannot be repaired here — the value is not derivable from this host',
|
||||||
|
'acknowledged' => 'the operator knows, and it stays quiet until the state it was acked at changes',
|
||||||
|
'fixed' => 'a proven value was written to conf',
|
||||||
|
'resolved' => 'the probe now passes; whatever was wrong is no longer wrong',
|
||||||
|
'dismissed' => 'the operator says this is not a problem, permanently',
|
||||||
|
];
|
||||||
|
|
||||||
|
// ── Why acknowledged is not dismissed ────────────────────────────────────────────────────────
|
||||||
|
// "I know critical rsync is off, stop telling me" and "this is never a problem" are different
|
||||||
|
// instructions, and collapsing them loses the half that matters. An acknowledgement is scoped to
|
||||||
|
// the state it was given in: CRITICAL_RSYNC_ENABLED being false is a deliberate choice today and
|
||||||
|
// a stale note the moment it goes true again.
|
||||||
|
//
|
||||||
|
// So an ack records what the key read when it was given, and expires when that changes. The
|
||||||
|
// finding comes back on its own, without the operator having to remember to look — which is the
|
||||||
|
// difference between a note and a silence.
|
||||||
|
|
||||||
|
// ── Toggles are the operator's, always ───────────────────────────────────────────────────────
|
||||||
|
// A repair may never enable or disable anything on its own. Not because it would get the value
|
||||||
|
// wrong — a boolean has only two — but because the value is not a fact to be discovered. Whether
|
||||||
|
// critical rsync should be on is a decision about intent, and a probe cannot prove intent the
|
||||||
|
// way it can prove that a port answers.
|
||||||
|
//
|
||||||
|
// The Fix action still writes it when the operator asks for it. What is forbidden is the
|
||||||
|
// unattended path choosing for them.
|
||||||
|
function vv_ai_conf_is_toggle(string $key): bool {
|
||||||
|
$v = strtolower(trim((string)(vv_conf_vars()[$key] ?? '')));
|
||||||
|
return $v === 'true' || $v === 'false';
|
||||||
|
}
|
||||||
|
|
||||||
|
// May the sweep write this without being asked? Two conditions, both required: a probe actually
|
||||||
|
// answered on the proposed value, and the key is not a toggle.
|
||||||
|
function vv_ai_finding_may_autofix(array $f): bool {
|
||||||
|
if (!vv_ai_repair_autofix_enabled()) return false;
|
||||||
|
if (empty($f['proven'])) return false;
|
||||||
|
if (($f['proposed'] ?? null) === null) return false;
|
||||||
|
if (vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))) return false;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Severity is derived, never supplied ──────────────────────────────────────────────────────
|
||||||
|
// Same ladder run_job.sh records runs against — ok / warn / error — so a finding and the run it
|
||||||
|
// came from cannot describe the same event at two different volumes.
|
||||||
|
//
|
||||||
|
// The rule that matters: a finding whose key is a toggle can never be an error. Something not
|
||||||
|
// happening because it was switched off is the switch working. That is true whether the switch
|
||||||
|
// was flipped deliberately last month or by accident this morning, and the store cannot tell
|
||||||
|
// those apart — so it reports the fact and lets the operator supply the intent.
|
||||||
|
//
|
||||||
|
// Everything else takes its level from what the fault costs. A credential the endpoint rejected
|
||||||
|
// stops that integration dead; an address that does not answer might be a host still booting.
|
||||||
|
function vv_ai_finding_severity(array $f): string {
|
||||||
|
$key = (string)($f['conf_key'] ?? '');
|
||||||
|
|
||||||
|
// Deliberate-state findings never escalate, whatever their kind.
|
||||||
|
if (vv_ai_conf_is_toggle($key)) return 'warn';
|
||||||
|
|
||||||
|
return match ($f['kind'] ?? '') {
|
||||||
|
'auth_rejected' => 'error', // answered and refused — nothing gets through until fixed
|
||||||
|
'missing_value' => 'error', // configured to use something that was never supplied
|
||||||
|
'unreachable' => 'warn', // may be transient; the strike system is what escalates it
|
||||||
|
'unknown_target' => 'warn',
|
||||||
|
default => 'warn',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Gates ────────────────────────────────────────────────────────────────────────────────────
|
||||||
|
// Two switches, because detecting and repairing are separate things to trust.
|
||||||
|
//
|
||||||
|
// AI_REPAIR_ENABLED alone gives a system that reads logs, files findings and offers fixes, and
|
||||||
|
// writes nothing. That is the state this should live in first — long enough to read what it
|
||||||
|
// found and disagree with some of it. A subsystem that starts by editing conf has to be believed
|
||||||
|
// before there is any evidence for believing it.
|
||||||
|
//
|
||||||
|
// AI_REPAIR_AUTOFIX_ENABLED is what lets a proven value be written without being asked, and it
|
||||||
|
// is meaningless on its own: nothing to write if nothing is looking. Both must be true, in the
|
||||||
|
// same layered way AI_ENABLED is necessary but never sufficient.
|
||||||
|
function vv_ai_repair_enabled(): bool {
|
||||||
|
if (!vv_ai_config()['enabled']) return false;
|
||||||
|
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_ENABLED'] ?? 'false'))) === 'true';
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_repair_autofix_enabled(): bool {
|
||||||
|
if (!vv_ai_repair_enabled()) return false;
|
||||||
|
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_AUTOFIX_ENABLED'] ?? 'false'))) === 'true';
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_findings_dir(): string {
|
||||||
|
$d = AI_DATA_DIR . '/ai_findings';
|
||||||
|
if (!is_dir($d)) @mkdir($d, 0755, true);
|
||||||
|
return $d;
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_finding_retain_days(): int {
|
||||||
|
$n = (int)(vv_conf_vars()['AI_FINDING_RETAIN_DAYS'] ?? 90);
|
||||||
|
return max(1, $n);
|
||||||
|
}
|
||||||
|
|
||||||
|
// kind + subject + conf key. Deliberately not the message: the same wrong port produces slightly
|
||||||
|
// different log text as the software around it changes, and that must not mint a second record.
|
||||||
|
function vv_ai_finding_id(string $kind, string $subject, string $confKey): string {
|
||||||
|
return substr(sha1(strtolower($kind . '|' . $subject . '|' . $confKey)), 0, 12);
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_finding_path(string $id): ?string {
|
||||||
|
if (!preg_match('/^[0-9a-f]{12}$/', $id)) return null;
|
||||||
|
return vv_ai_findings_dir() . '/' . $id . '.json';
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_finding_get(string $id): ?array {
|
||||||
|
$p = vv_ai_finding_path($id);
|
||||||
|
if ($p === null || !is_file($p)) return null;
|
||||||
|
$r = json_decode((string)@file_get_contents($p), true);
|
||||||
|
return is_array($r) ? $r : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Files a finding, or increments the one already describing this fault.
|
||||||
|
//
|
||||||
|
// $f expects: kind, subject, conf_key, conf_file, observed, evidence, source_log
|
||||||
|
// and optionally: proposed, proven, state, note
|
||||||
|
function vv_ai_finding_write(array $f): array {
|
||||||
|
$kind = (string)($f['kind'] ?? '');
|
||||||
|
$subject = trim((string)($f['subject'] ?? ''));
|
||||||
|
$confKey = trim((string)($f['conf_key'] ?? ''));
|
||||||
|
$evidence = trim((string)($f['evidence'] ?? ''));
|
||||||
|
|
||||||
|
if (!isset(VV_AI_FINDING_KINDS[$kind])) return ['ok' => false, 'error' => 'unknown kind'];
|
||||||
|
if ($subject === '' || $confKey === '') return ['ok' => false, 'error' => 'subject and conf_key required'];
|
||||||
|
if ($evidence === '') return ['ok' => false, 'error' => 'evidence required'];
|
||||||
|
// The key has to be a real shell identifier for the same reason the conf writer insists on
|
||||||
|
// it: a finding is a proposal to edit that key, and a malformed one can never be actioned.
|
||||||
|
if (!vv_conf_key_valid($confKey)) return ['ok' => false, 'error' => 'malformed conf key'];
|
||||||
|
|
||||||
|
$state = (string)($f['state'] ?? 'open');
|
||||||
|
if (!isset(VV_AI_FINDING_STATES[$state])) $state = 'open';
|
||||||
|
|
||||||
|
$now = time();
|
||||||
|
$id = vv_ai_finding_id($kind, $subject, $confKey);
|
||||||
|
$rec = [
|
||||||
|
'id' => $id,
|
||||||
|
'kind' => $kind,
|
||||||
|
'subject' => mb_substr($subject, 0, 120),
|
||||||
|
'conf_key' => $confKey,
|
||||||
|
'conf_file' => (string)($f['conf_file'] ?? 'master.conf'),
|
||||||
|
// Secrets never enter this store. A finding about a rejected API key is about the key
|
||||||
|
// being wrong, and the wrong value is of no use to anyone reading the record later.
|
||||||
|
'observed' => vv_conf_key_is_secret($confKey) ? '<redacted>'
|
||||||
|
: mb_substr((string)($f['observed'] ?? ''), 0, 300),
|
||||||
|
'proposed' => isset($f['proposed']) && !vv_conf_key_is_secret($confKey)
|
||||||
|
? mb_substr((string)$f['proposed'], 0, 300) : null,
|
||||||
|
'proven' => (bool)($f['proven'] ?? false),
|
||||||
|
'state' => $state,
|
||||||
|
'evidence' => mb_substr(vv_ai_redact($evidence), 0, 1200),
|
||||||
|
'source_log' => mb_substr((string)($f['source_log'] ?? ''), 0, 200),
|
||||||
|
'note' => mb_substr((string)($f['note'] ?? ''), 0, 1000),
|
||||||
|
// Recomputed on every sighting rather than stored once: a key that becomes a toggle, or
|
||||||
|
// a toggle that is replaced by a real value, changes what this finding means.
|
||||||
|
'severity' => vv_ai_finding_severity(['kind' => $kind, 'conf_key' => $confKey]),
|
||||||
|
'host' => vv_detect_host(),
|
||||||
|
'first' => $now,
|
||||||
|
'last' => $now,
|
||||||
|
'seen' => 1,
|
||||||
|
'closed_at' => null,
|
||||||
|
// What the key read when the operator acknowledged it. Null unless acked; the ack
|
||||||
|
// expires the moment the live value stops matching this.
|
||||||
|
'ack_value' => null,
|
||||||
|
];
|
||||||
|
|
||||||
|
$p = vv_ai_finding_path($id);
|
||||||
|
if ($p === null) return ['ok' => false, 'error' => 'bad id'];
|
||||||
|
|
||||||
|
if (is_file($p)) {
|
||||||
|
$old = json_decode((string)@file_get_contents($p), true);
|
||||||
|
if (is_array($old)) {
|
||||||
|
$rec['first'] = $old['first'] ?? $now;
|
||||||
|
$rec['seen'] = (int)($old['seen'] ?? 0) + 1;
|
||||||
|
// A dismissed finding stays dismissed however many times the log repeats it —
|
||||||
|
// otherwise "this is fine, stop telling me" lasts exactly one cycle. A fixed one
|
||||||
|
// reopens, because seeing the fault again after a repair means the repair did not
|
||||||
|
// hold, which is the single most important thing this store can tell anyone.
|
||||||
|
if (($old['state'] ?? '') === 'dismissed') {
|
||||||
|
$rec['state'] = 'dismissed';
|
||||||
|
$rec['closed_at'] = $old['closed_at'] ?? null;
|
||||||
|
}
|
||||||
|
// An acknowledgement holds only while the thing acknowledged is still true. Compare
|
||||||
|
// the live value against what it read when the ack was given: unchanged means stay
|
||||||
|
// quiet, changed means the note is stale and the finding comes back by itself.
|
||||||
|
if (($old['state'] ?? '') === 'acknowledged') {
|
||||||
|
$ackedAt = (string)($old['ack_value'] ?? '');
|
||||||
|
if ($ackedAt === (string)(vv_conf_vars()[$confKey] ?? '')) {
|
||||||
|
$rec['state'] = 'acknowledged';
|
||||||
|
$rec['ack_value'] = $ackedAt;
|
||||||
|
$rec['closed_at'] = $old['closed_at'] ?? null;
|
||||||
|
}
|
||||||
|
// Otherwise $rec keeps the state this sighting computed — it has reopened.
|
||||||
|
}
|
||||||
|
// Preserve an operator's note over a generated one.
|
||||||
|
if ($rec['note'] === '' && !empty($old['note'])) $rec['note'] = $old['note'];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (@file_put_contents($p, json_encode($rec, JSON_PRETTY_PRINT)) === false) {
|
||||||
|
return ['ok' => false, 'error' => 'write failed'];
|
||||||
|
}
|
||||||
|
return ['ok' => true, 'id' => $id, 'seen' => $rec['seen'], 'state' => $rec['state']];
|
||||||
|
}
|
||||||
|
|
||||||
|
// $states filters; empty means everything. Newest activity first, because a finding seen in the
|
||||||
|
// last cycle matters more than one that has been sitting fixed for a month.
|
||||||
|
function vv_ai_findings_list(array $states = ['open', 'needs_operator']): array {
|
||||||
|
$out = [];
|
||||||
|
foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) {
|
||||||
|
$r = json_decode((string)@file_get_contents($file), true);
|
||||||
|
if (!is_array($r)) continue;
|
||||||
|
if ($states && !in_array($r['state'] ?? 'open', $states, true)) continue;
|
||||||
|
$out[] = $r;
|
||||||
|
}
|
||||||
|
usort($out, fn($a, $b) => ($b['last'] ?? 0) <=> ($a['last'] ?? 0));
|
||||||
|
return $out;
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_finding_set_state(string $id, string $state, string $note = ''): bool {
|
||||||
|
if (!isset(VV_AI_FINDING_STATES[$state])) return false;
|
||||||
|
$r = vv_ai_finding_get($id);
|
||||||
|
if ($r === null) return false;
|
||||||
|
|
||||||
|
$r['state'] = $state;
|
||||||
|
$r['closed_at'] = in_array($state, ['open', 'needs_operator'], true) ? null : time();
|
||||||
|
if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000);
|
||||||
|
|
||||||
|
$p = vv_ai_finding_path($id);
|
||||||
|
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_finding_close(string $id, string $note = ''): bool {
|
||||||
|
return vv_ai_finding_set_state($id, 'resolved', $note);
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_finding_dismiss(string $id, string $note = ''): bool {
|
||||||
|
return vv_ai_finding_set_state($id, 'dismissed', $note);
|
||||||
|
}
|
||||||
|
|
||||||
|
// "I know about this — leave it, and tell me if it changes."
|
||||||
|
//
|
||||||
|
// Stamps the key's current value onto the record. Every later sighting compares against that
|
||||||
|
// stamp, so the acknowledgement covers this state and not the key forever. Acking that critical
|
||||||
|
// rsync is off says nothing about critical rsync being on.
|
||||||
|
function vv_ai_finding_ack(string $id, string $note = ''): bool {
|
||||||
|
$r = vv_ai_finding_get($id);
|
||||||
|
if ($r === null) return false;
|
||||||
|
|
||||||
|
$r['state'] = 'acknowledged';
|
||||||
|
$r['ack_value'] = (string)(vv_conf_vars()[$r['conf_key'] ?? ''] ?? '');
|
||||||
|
$r['closed_at'] = time();
|
||||||
|
if ($note !== '') $r['note'] = mb_substr(vv_ai_redact($note), 0, 1000);
|
||||||
|
|
||||||
|
$p = vv_ai_finding_path($id);
|
||||||
|
return $p !== null && @file_put_contents($p, json_encode($r, JSON_PRETTY_PRINT)) !== false;
|
||||||
|
}
|
||||||
|
|
||||||
|
// What the operator can do about a finding, and what each choice means. Returned rather than
|
||||||
|
// hardcoded in the UI so the chat and the page cannot offer different options for the same row.
|
||||||
|
//
|
||||||
|
// Fix appears for anything with a proposed value, toggle or not — the prohibition is on the
|
||||||
|
// sweep choosing, never on the operator choosing. Everything carries ack and cancel, because
|
||||||
|
// "I know" and "not now" are always valid answers to being told something.
|
||||||
|
function vv_ai_finding_actions(array $f): array {
|
||||||
|
$actions = [];
|
||||||
|
|
||||||
|
if (($f['proposed'] ?? null) !== null) {
|
||||||
|
$actions['fix'] = vv_ai_conf_is_toggle((string)($f['conf_key'] ?? ''))
|
||||||
|
? 'Set ' . $f['conf_key'] . ' — a toggle, so this only ever happens because you asked'
|
||||||
|
: 'Write the proven value to ' . $f['conf_key'];
|
||||||
|
}
|
||||||
|
|
||||||
|
$actions['ack'] = 'Known and intended. Stays quiet until ' . ($f['conf_key'] ?? 'it') . ' changes';
|
||||||
|
$actions['cancel'] = 'Leave it alone for now';
|
||||||
|
|
||||||
|
return $actions;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Closed findings are kept for a while because "this happened before and here is what fixed it"
|
||||||
|
// is worth more than the disk it costs. Open ones are never pruned — an unresolved problem does
|
||||||
|
// not stop mattering because it is old.
|
||||||
|
function vv_ai_findings_prune(): int {
|
||||||
|
$cutoff = time() - (vv_ai_finding_retain_days() * 86400);
|
||||||
|
$n = 0;
|
||||||
|
foreach ((array)@glob(vv_ai_findings_dir() . '/*.json') as $file) {
|
||||||
|
$r = json_decode((string)@file_get_contents($file), true);
|
||||||
|
if (!is_array($r)) continue;
|
||||||
|
if (in_array($r['state'] ?? 'open', ['open', 'needs_operator'], true)) continue;
|
||||||
|
if ((int)($r['closed_at'] ?? 0) > $cutoff) continue;
|
||||||
|
if (@unlink($file)) $n++;
|
||||||
|
}
|
||||||
|
return $n;
|
||||||
|
}
|
||||||
|
|
||||||
|
// What the assistant should raise when a page loads: things that need the operator, newest
|
||||||
|
// first. Repaired findings are deliberately not here — a fix that worked is a log entry, not a
|
||||||
|
// conversation, and opening every session with a list of things that already went right is how
|
||||||
|
// an operator learns to close the panel without reading it.
|
||||||
|
function vv_ai_findings_for_chat(int $limit = 3): array {
|
||||||
|
return array_slice(vv_ai_findings_list(['needs_operator']), 0, max(1, $limit));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── The sweep ────────────────────────────────────────────────────────────────────────────────
|
||||||
|
// There is no post-run hook in Varaverk — nothing fires when a job finishes. Rather than add a
|
||||||
|
// call to forty scripts, this picks up run records that completed since the last pass. One entry
|
||||||
|
// in an orchestrator's list instead of forty edits, and it batches naturally.
|
||||||
|
//
|
||||||
|
// Runs that reported ok are read too. A container failing its HTTP check warns and leaves the
|
||||||
|
// watchdog exiting 0, so "only look at failures" would miss the whole class of fault this exists
|
||||||
|
// for: the job worked, and told you something is wrong.
|
||||||
|
|
||||||
|
function vv_ai_sweep_marker_path(): string {
|
||||||
|
return STATE_DIR . '/ai_repair_sweep.db';
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_sweep_last(): int {
|
||||||
|
return (int)trim((string)@file_get_contents(vv_ai_sweep_marker_path()));
|
||||||
|
}
|
||||||
|
|
||||||
|
function vv_ai_sweep_mark(int $ts): void {
|
||||||
|
if (!is_dir(STATE_DIR)) @mkdir(STATE_DIR, 0755, true);
|
||||||
|
@file_put_contents(vv_ai_sweep_marker_path(), (string)$ts, LOCK_EX);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run records that finished after $since. A record still marked running is skipped rather than
|
||||||
|
// read half-written — it will be picked up on the pass after it finishes.
|
||||||
|
function vv_ai_recent_runs(int $since): array {
|
||||||
|
$out = [];
|
||||||
|
$base = realpath(LOG_DIR);
|
||||||
|
if ($base === false) return [];
|
||||||
|
|
||||||
|
foreach ((array)@glob($base . '/{,*/,*/*/,*/*/*/}*.json', GLOB_BRACE) as $path) {
|
||||||
|
$r = json_decode((string)@file_get_contents($path), true);
|
||||||
|
if (!is_array($r) || empty($r['id']) || ($r['status'] ?? '') === 'running') continue;
|
||||||
|
|
||||||
|
$end = (int)($r['end'] ?? 0);
|
||||||
|
if ($end <= $since) continue;
|
||||||
|
|
||||||
|
$log = preg_replace('/\.json$/', '.log', $path);
|
||||||
|
if (!is_file($log)) continue;
|
||||||
|
|
||||||
|
$out[] = ['id' => (string)$r['id'], 'status' => (string)($r['status'] ?? '?'),
|
||||||
|
'start' => (int)($r['start'] ?? 0), 'end' => $end, 'log' => $log];
|
||||||
|
}
|
||||||
|
usort($out, fn($a, $b) => $a['end'] <=> $b['end']);
|
||||||
|
return $out;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The lines one run wrote, and only those. Logs are appended across runs, so a tail alone would
|
||||||
|
// re-read the previous run's output and re-report faults that have already been dealt with.
|
||||||
|
// Filtering on the leading timestamp scopes the evidence to the run being examined.
|
||||||
|
function vv_ai_run_log_lines(string $logPath, int $startTs, int $maxLines = 2000): array {
|
||||||
|
$out = []; $rc = 0;
|
||||||
|
@exec('tail -n ' . (int)$maxLines . ' ' . escapeshellarg($logPath) . ' 2>/dev/null', $out, $rc);
|
||||||
|
if ($rc !== 0) return [];
|
||||||
|
|
||||||
|
$kept = [];
|
||||||
|
foreach ($out as $line) {
|
||||||
|
if (preg_match('/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/', $line, $m)) {
|
||||||
|
// A line older than the run belongs to a previous one. Two seconds of slack because
|
||||||
|
// the record's start is stamped by the runner, not by the first line the job writes.
|
||||||
|
if (strtotime($m[1]) < $startTs - 2) continue;
|
||||||
|
}
|
||||||
|
$kept[] = $line;
|
||||||
|
}
|
||||||
|
return $kept;
|
||||||
|
}
|
||||||
|
|
||||||
|
// One pass. Returns a summary rather than logging it, so the caller decides what to record and
|
||||||
|
// the whole thing stays testable without a log to read afterwards.
|
||||||
|
//
|
||||||
|
// $dryRun does everything except write conf and move the marker — including probing, which is
|
||||||
|
// the point: it answers "what would this have done" with real evidence rather than a guess.
|
||||||
|
function vv_ai_repair_sweep(bool $dryRun = false): array {
|
||||||
|
if (!vv_ai_repair_enabled()) {
|
||||||
|
return ['ok' => false, 'error' => 'AI_REPAIR_ENABLED is not true', 'runs' => 0];
|
||||||
|
}
|
||||||
|
|
||||||
|
$started = time();
|
||||||
|
$since = vv_ai_sweep_last();
|
||||||
|
$runs = vv_ai_recent_runs($since);
|
||||||
|
|
||||||
|
$sum = ['ok' => true, 'runs' => count($runs), 'findings' => 0, 'fixed' => 0,
|
||||||
|
'needs_operator' => 0, 'resolved' => 0, 'quiet' => 0, 'details' => []];
|
||||||
|
|
||||||
|
foreach ($runs as $run) {
|
||||||
|
$lines = vv_ai_run_log_lines($run['log'], $run['start']);
|
||||||
|
if (!$lines) continue;
|
||||||
|
|
||||||
|
$rel = ltrim(str_replace(realpath(LOG_DIR), '', $run['log']), '/');
|
||||||
|
foreach (vv_ai_triage_log($lines, $rel) as $cand) {
|
||||||
|
$cand = vv_ai_probe_finding($cand);
|
||||||
|
$sum['findings']++;
|
||||||
|
|
||||||
|
// Write first, so a finding exists even if the repair below fails. A repair that
|
||||||
|
// errored without leaving a record is the one failure mode there is no way back from.
|
||||||
|
$w = vv_ai_finding_write($cand);
|
||||||
|
if (!($w['ok'] ?? false)) continue;
|
||||||
|
$id = $w['id'];
|
||||||
|
|
||||||
|
// Already acknowledged or dismissed — the operator has spoken, and re-fixing behind
|
||||||
|
// them would be the opposite of what an acknowledgement means.
|
||||||
|
if (in_array($w['state'] ?? '', ['acknowledged', 'dismissed'], true)) {
|
||||||
|
$sum['quiet']++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (($cand['state'] ?? '') === 'resolved') {
|
||||||
|
vv_ai_finding_close($id, (string)($cand['note'] ?? ''));
|
||||||
|
$sum['resolved']++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (vv_ai_finding_may_autofix($cand)) {
|
||||||
|
if ($dryRun) { $sum['details'][] = "would fix {$cand['conf_key']} → {$cand['proposed']}"; continue; }
|
||||||
|
$r = vv_ai_finding_apply_action($id, 'fix', 'Probed and written by the repair sweep.');
|
||||||
|
if ($r['ok'] ?? false) { $sum['fixed']++; $sum['details'][] = "fixed {$cand['conf_key']}"; }
|
||||||
|
else { $sum['needs_operator']++; vv_ai_finding_set_state($id, 'needs_operator', (string)($r['error'] ?? '')); }
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (($cand['state'] ?? '') === 'needs_operator') $sum['needs_operator']++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Marked only on a completed pass, and to when the pass began — a job that finished while
|
||||||
|
// this was running is then picked up next time instead of being skipped for having ended
|
||||||
|
// before a marker written at the end.
|
||||||
|
if (!$dryRun) vv_ai_sweep_mark($started);
|
||||||
|
|
||||||
|
return $sum;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Answering a finding in words ─────────────────────────────────────────────────────────────
|
||||||
|
// The buttons are unambiguous by construction. This is for the other path — replying "yeah go
|
||||||
|
// ahead" in the chat that raised the finding — and it is matched here rather than asked of the
|
||||||
|
// model, because the model's answer would be a conf write and a wrong reading of "no, leave it"
|
||||||
|
// is not recoverable by apologising.
|
||||||
|
//
|
||||||
|
// Same shape as vv_ai_route_from_chat(): anchored patterns, most specific first, and anything
|
||||||
|
// unrecognised returns null so the assistant asks again instead of guessing. Two actions both
|
||||||
|
// matching is also null — "leave it, I know" and "leave it for now" differ by one clause and
|
||||||
|
// mean different things, so a phrase that supports both is not an instruction yet.
|
||||||
|
const VV_AI_ACTION_PATTERNS = [
|
||||||
|
// Acknowledge — "this is deliberate, stop telling me".
|
||||||
|
'ack' => [
|
||||||
|
'/\b(i|we) know\b/u',
|
||||||
|
'/\b(that|this|it)(?:\'s| is) (fine|expected|intentional|deliberate|on purpose)\b/u',
|
||||||
|
'/\bon purpose\b/u',
|
||||||
|
'/\b(aware|acknowledge|ack)\b/u',
|
||||||
|
'/\bmeant to be\b/u',
|
||||||
|
],
|
||||||
|
// Apply the proposed value.
|
||||||
|
'fix' => [
|
||||||
|
'/\bfix (it|that|this|them)?\b/u',
|
||||||
|
'/\b(go ahead|do it|apply|make the change|change it|update it|correct it)\b/u',
|
||||||
|
// A bare affirmative, as the whole message — "yes" answering "shall I fix it" is an
|
||||||
|
// instruction, "yes it looks wrong" is agreement about the diagnosis and nothing more.
|
||||||
|
// A trailing please is still bare.
|
||||||
|
'/\b(yes|yeah|yep|yup|sure|ok|okay)\b(\s*,?\s*please)?[\s,.!]*$/u',
|
||||||
|
'/\bplease do\b/u',
|
||||||
|
],
|
||||||
|
// Not now — no state written, it comes back next sweep.
|
||||||
|
'cancel' => [
|
||||||
|
'/\b(not now|later|leave it (alone|for now)|skip( it)?|cancel|ignore for now)\b/u',
|
||||||
|
'/\b(no|nope|nah)\b[\s,.!]*$/u',
|
||||||
|
'/\b(don\'?t|do not) (fix|touch|change|write|apply)\b/u',
|
||||||
|
],
|
||||||
|
];
|
||||||
|
|
||||||
|
// Returns 'fix' | 'ack' | 'cancel', or null when the reply does not clearly mean one of them.
|
||||||
|
//
|
||||||
|
// Only call this when a finding is actually pending. A bare "yes" means fix in answer to "shall
|
||||||
|
// I fix it" and means nothing at all on its own, and the difference is context this function
|
||||||
|
// cannot see.
|
||||||
|
function vv_ai_finding_action_from_text(string $text): ?string {
|
||||||
|
$t = strtolower(trim($text));
|
||||||
|
if ($t === '') return null;
|
||||||
|
$t = preg_replace('/\s+/', ' ', $t);
|
||||||
|
|
||||||
|
$matched = [];
|
||||||
|
foreach (VV_AI_ACTION_PATTERNS as $action => $patterns) {
|
||||||
|
foreach ($patterns as $re) {
|
||||||
|
if (preg_match($re, $t)) { $matched[$action] = true; break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Exactly one reading, or none. "leave it, I know" hits both ack and cancel; that is a
|
||||||
|
// sentence the operator should be asked to restate, not one to pick a winner from.
|
||||||
|
return count($matched) === 1 ? array_key_first($matched) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Carry out an answered action against a stored finding.
|
||||||
|
//
|
||||||
|
// Fix goes through the same guarded write path as everything else, and is the one place a
|
||||||
|
// toggle may be written — because reaching here means the operator asked for it by name. The
|
||||||
|
// unattended sweep never calls this.
|
||||||
|
function vv_ai_finding_apply_action(string $id, string $action, string $note = ''): array {
|
||||||
|
$f = vv_ai_finding_get($id);
|
||||||
|
if ($f === null) return ['ok' => false, 'error' => 'no such finding'];
|
||||||
|
|
||||||
|
switch ($action) {
|
||||||
|
case 'ack':
|
||||||
|
return ['ok' => vv_ai_finding_ack($id, $note), 'action' => 'ack'];
|
||||||
|
|
||||||
|
case 'cancel':
|
||||||
|
// Deliberately writes nothing at all. "Not now" is not a state, it is the absence of
|
||||||
|
// one — recording it would make the finding look decided when it is still open.
|
||||||
|
return ['ok' => true, 'action' => 'cancel'];
|
||||||
|
|
||||||
|
case 'fix':
|
||||||
|
$proposed = $f['proposed'] ?? null;
|
||||||
|
if ($proposed === null) return ['ok' => false, 'error' => 'nothing proposed to write'];
|
||||||
|
|
||||||
|
$key = (string)$f['conf_key'];
|
||||||
|
$ok = vv_conf_write_changes([[
|
||||||
|
'file' => (string)($f['conf_file'] ?? 'master.conf'),
|
||||||
|
'key' => $key,
|
||||||
|
'value' => (string)$proposed,
|
||||||
|
'type' => 'scalar',
|
||||||
|
]]);
|
||||||
|
$wrote = !in_array(false, $ok, true);
|
||||||
|
|
||||||
|
if ($wrote) {
|
||||||
|
vv_ai_finding_set_state($id, 'fixed',
|
||||||
|
$note !== '' ? $note : 'Wrote ' . $key . ' at the operator\'s request.');
|
||||||
|
}
|
||||||
|
return ['ok' => $wrote, 'action' => 'fix',
|
||||||
|
'error' => $wrote ? null : 'conf write refused — see conf_changes.log'];
|
||||||
|
}
|
||||||
|
return ['ok' => false, 'error' => 'unknown action'];
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Resolving a log line back to a conf key ──────────────────────────────────────────────────
|
||||||
|
// By value wherever possible, by name only as a fallback.
|
||||||
|
//
|
||||||
|
// A log line usually contains the thing that failed — the URL that did not answer. That value
|
||||||
|
// came from a conf key, so searching the conf for which key holds it is an exact lookup with a
|
||||||
|
// definite answer. Guessing the key from the container's name is inference, and the failure mode
|
||||||
|
// is silent: HOST1_EMBY_URL and HOST1_EMBY_EXTERNAL_URL are both plausible for "Emby" and only
|
||||||
|
// one of them is the value that just failed.
|
||||||
|
//
|
||||||
|
// Same principle as resolve_tailscale_ip() refusing similarity matching for host identity: an
|
||||||
|
// exact match or an honest nothing.
|
||||||
|
|
||||||
|
// Every conf key whose value the given text starts with, longest first. Prefix rather than
|
||||||
|
// equality because a log reports the URL it actually called — the conf value plus an endpoint
|
||||||
|
// path — and the longest match is the most specific key that could have produced it.
|
||||||
|
function vv_ai_conf_keys_for_value(string $value): array {
|
||||||
|
$value = trim($value);
|
||||||
|
if ($value === '' || strlen($value) < 6) return [];
|
||||||
|
|
||||||
|
$hits = [];
|
||||||
|
foreach (vv_conf_vars() as $k => $v) {
|
||||||
|
$v = trim((string)$v);
|
||||||
|
if ($v === '' || strlen($v) < 6) continue;
|
||||||
|
if ($v === $value || str_starts_with($value, $v)) $hits[$k] = strlen($v);
|
||||||
|
}
|
||||||
|
arsort($hits);
|
||||||
|
return array_keys($hits);
|
||||||
|
}
|
||||||
|
|
||||||
|
// <HOSTID>_<SUBJECT>_<SUFFIX>, built literally and then checked for existence. Nothing is
|
||||||
|
// inferred: either the conf holds a key by exactly that name or this returns null.
|
||||||
|
function vv_ai_conf_key_for_subject(string $subject, string $suffix): ?string {
|
||||||
|
$norm = strtoupper(preg_replace('/[^A-Za-z0-9]+/', '_', trim($subject)));
|
||||||
|
if ($norm === '') return null;
|
||||||
|
|
||||||
|
$key = strtoupper(vv_detect_host()) . '_' . $norm . '_' . strtoupper($suffix);
|
||||||
|
return array_key_exists($key, vv_conf_vars()) ? $key : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Which conf file a key lives in. A finding has to name the file it would be edited in, and
|
||||||
|
// host keys are not in master.conf.
|
||||||
|
function vv_ai_conf_file_for_key(string $key): string {
|
||||||
|
foreach (vv_get_conf_files() as $f) {
|
||||||
|
if (preg_match('/^\s*' . preg_quote($key, '/') . '\s*=/m', vv_read_conf_raw($f))) return $f;
|
||||||
|
}
|
||||||
|
return 'master.conf';
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Deterministic triage ─────────────────────────────────────────────────────────────────────
|
||||||
|
// Patterns are written against log formats that exist in this repo today, taken from the emit
|
||||||
|
// sites rather than imagined. A pattern that stops matching because its log line was reworded
|
||||||
|
// produces no finding, which is the safe direction — the alternative is a pattern loose enough
|
||||||
|
// to match anything, which fills the store with rows nobody can act on.
|
||||||
|
//
|
||||||
|
// No model runs here. "Connection refused" is not a judgement call, and a 14B model invoked
|
||||||
|
// after every cron job to notice it would be slower, costlier and less reliable than a regex.
|
||||||
|
// The model's job starts where these stop: explaining a finding, and talking the operator
|
||||||
|
// through the ones that cannot be repaired automatically.
|
||||||
|
const VV_AI_TRIAGE_PATTERNS = [
|
||||||
|
// docker_watchdog HTTP check — the richest signal, carrying both subject and failing URL.
|
||||||
|
[
|
||||||
|
're' => '/^(?P<subject>\S+) — not responding at (?P<observed>\S+) \(strike/u',
|
||||||
|
'kind' => 'unreachable',
|
||||||
|
'suffix' => 'URL',
|
||||||
|
],
|
||||||
|
// docker_watchdog API check, 401/403. The endpoint answered, so the address is right and
|
||||||
|
// the credential is not.
|
||||||
|
[
|
||||||
|
're' => '/^(?P<subject>\S+) — API check skipped \(HTTP (?:401|403)/u',
|
||||||
|
'kind' => 'auth_rejected',
|
||||||
|
'suffix' => 'API_KEY',
|
||||||
|
],
|
||||||
|
// Configured for an API check with nothing to authenticate with.
|
||||||
|
[
|
||||||
|
're' => '/^(?P<subject>\S+) — API check skipped \(no key configured\)/u',
|
||||||
|
'kind' => 'missing_value',
|
||||||
|
'suffix' => 'API_KEY',
|
||||||
|
],
|
||||||
|
[
|
||||||
|
're' => '/^Skipping (?P<subject>\S+) — placeholder API key/u',
|
||||||
|
'kind' => 'missing_value',
|
||||||
|
'suffix' => 'API_KEY',
|
||||||
|
],
|
||||||
|
[
|
||||||
|
're' => '/^(?P<subject>\S+) (?:—\s*)?API unreachable/u',
|
||||||
|
'kind' => 'unreachable',
|
||||||
|
'suffix' => 'URL',
|
||||||
|
],
|
||||||
|
];
|
||||||
|
|
||||||
|
// One log line in, at most one finding candidate out. Returns null for everything else, which is
|
||||||
|
// almost every line.
|
||||||
|
function vv_ai_triage_line(string $line): ?array {
|
||||||
|
// Strip the timestamp and level decoration the logger adds, so patterns can anchor on ^.
|
||||||
|
$body = preg_replace('/^\S+\s+\S+\s+(?:[^\[]*\[[A-Z]+\]\s*)?/u', '', rtrim($line));
|
||||||
|
$body = trim((string)$body);
|
||||||
|
if ($body === '') return null;
|
||||||
|
|
||||||
|
foreach (VV_AI_TRIAGE_PATTERNS as $p) {
|
||||||
|
if (!preg_match($p['re'], $body, $m)) continue;
|
||||||
|
|
||||||
|
$subject = trim($m['subject'] ?? '');
|
||||||
|
$observed = trim($m['observed'] ?? '');
|
||||||
|
if ($subject === '') continue;
|
||||||
|
|
||||||
|
// Value first, name second. Only one candidate key is accepted — two keys holding the
|
||||||
|
// same value means the log cannot say which one produced it, and picking either is the
|
||||||
|
// guess this whole approach exists to avoid.
|
||||||
|
$key = null;
|
||||||
|
if ($observed !== '') {
|
||||||
|
$byValue = vv_ai_conf_keys_for_value($observed);
|
||||||
|
if (count($byValue) === 1) $key = $byValue[0];
|
||||||
|
}
|
||||||
|
if ($key === null) $key = vv_ai_conf_key_for_subject($subject, $p['suffix']);
|
||||||
|
if ($key === null) continue; // nothing actionable — no row
|
||||||
|
|
||||||
|
return [
|
||||||
|
'kind' => $p['kind'],
|
||||||
|
'subject' => $subject,
|
||||||
|
'conf_key' => $key,
|
||||||
|
'conf_file' => vv_ai_conf_file_for_key($key),
|
||||||
|
'observed' => $observed !== '' ? $observed : (string)(vv_conf_vars()[$key] ?? ''),
|
||||||
|
'evidence' => $body,
|
||||||
|
];
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A whole log tail in, one candidate per distinct fault out. A job that retried twelve times
|
||||||
|
// produces twelve identical lines, and the store's dedupe would collapse them anyway — doing it
|
||||||
|
// here keeps the sweep from writing the same file twelve times in a row.
|
||||||
|
function vv_ai_triage_log(array $lines, string $sourceLog = ''): array {
|
||||||
|
$found = [];
|
||||||
|
foreach ($lines as $line) {
|
||||||
|
$c = vv_ai_triage_line((string)$line);
|
||||||
|
if ($c === null) continue;
|
||||||
|
$c['source_log'] = $sourceLog;
|
||||||
|
$found[vv_ai_finding_id($c['kind'], $c['subject'], $c['conf_key'])] = $c;
|
||||||
|
}
|
||||||
|
return array_values($found);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Proving a candidate ──────────────────────────────────────────────────────────────────────
|
||||||
|
// The guard the whole unattended path rests on: nothing is written that has not answered.
|
||||||
|
//
|
||||||
|
// A model can be confident that a port should be 8686. A probe can report that 8686 answered.
|
||||||
|
// Only the second is a fact, and only facts get written to conf without being asked. Everything
|
||||||
|
// a probe cannot settle becomes a conversation instead — which is not a lesser outcome, it is
|
||||||
|
// the honest one for a value that cannot be derived from this machine.
|
||||||
|
//
|
||||||
|
// Deliberately narrow. These check reachability and identity, never correctness of behaviour:
|
||||||
|
// that Lidarr answers on 8686 does not prove 8686 is the port you meant, only that something is
|
||||||
|
// listening there and calling itself Lidarr. Proving intent is not a probe's job.
|
||||||
|
|
||||||
|
function vv_ai_probe_timeout(): int {
|
||||||
|
return max(1, (int)(vv_conf_vars()['AI_PROBE_TIMEOUT'] ?? 4));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Does this URL answer at all? Any HTTP status counts, including 401 — a refusal is proof that
|
||||||
|
// something is listening and speaking HTTP, which is exactly what an address probe is asking.
|
||||||
|
// Distinguishing "wrong address" from "wrong credential" is the point of having both kinds.
|
||||||
|
function vv_ai_probe_url(string $url): array {
|
||||||
|
if (!preg_match('#^https?://[^\s/$.?\#][^\s]*$#i', $url)) {
|
||||||
|
return ['ok' => false, 'reason' => 'not a url'];
|
||||||
|
}
|
||||||
|
|
||||||
|
$ch = curl_init($url);
|
||||||
|
curl_setopt_array($ch, [
|
||||||
|
CURLOPT_RETURNTRANSFER => true,
|
||||||
|
CURLOPT_NOBODY => true,
|
||||||
|
CURLOPT_TIMEOUT => vv_ai_probe_timeout(),
|
||||||
|
CURLOPT_CONNECTTIMEOUT => vv_ai_probe_timeout(),
|
||||||
|
CURLOPT_SSL_VERIFYPEER => false, // these are LAN and Tailscale endpoints, often self-signed
|
||||||
|
CURLOPT_SSL_VERIFYHOST => false,
|
||||||
|
]);
|
||||||
|
curl_exec($ch);
|
||||||
|
$code = (int)curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
|
||||||
|
$err = curl_error($ch);
|
||||||
|
curl_close($ch);
|
||||||
|
|
||||||
|
return $code > 0
|
||||||
|
? ['ok' => true, 'code' => $code]
|
||||||
|
: ['ok' => false, 'reason' => $err !== '' ? $err : 'no response'];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every container on this host, by exact name. The membership test for unknown_target findings,
|
||||||
|
// and one docker call rather than one per candidate.
|
||||||
|
function vv_ai_container_names(): array {
|
||||||
|
static $names = null;
|
||||||
|
if ($names !== null) return $names;
|
||||||
|
|
||||||
|
$out = [];
|
||||||
|
@exec('timeout ' . vv_ai_probe_timeout() . " docker ps -a --format '{{.Names}}' 2>/dev/null", $out, $rc);
|
||||||
|
return $names = ($rc === 0) ? array_values(array_filter(array_map('trim', $out))) : [];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Candidate corrections for a URL whose host or port stopped answering.
|
||||||
|
//
|
||||||
|
// Only two transformations, both conservative: the same host on a port that some other conf key
|
||||||
|
// already uses, and the same port on a host some other conf key already names. Both draw
|
||||||
|
// exclusively from values already present in this installation's conf — nothing is invented, and
|
||||||
|
// a scan of the port range is deliberately not attempted. Finding *a* listening port is not the
|
||||||
|
// same as finding the right service, and a probe that accepts any answer would happily point
|
||||||
|
// Lidarr at Sonarr.
|
||||||
|
function vv_ai_url_candidates(string $observed): array {
|
||||||
|
$parts = @parse_url($observed);
|
||||||
|
if (!is_array($parts) || empty($parts['host'])) return [];
|
||||||
|
|
||||||
|
$hosts = $ports = [];
|
||||||
|
foreach (vv_conf_vars() as $k => $v) {
|
||||||
|
$v = trim((string)$v);
|
||||||
|
if (!preg_match('#^https?://#i', $v)) continue;
|
||||||
|
$p = @parse_url($v);
|
||||||
|
if (!is_array($p) || empty($p['host'])) continue;
|
||||||
|
$hosts[$p['host']] = true;
|
||||||
|
if (!empty($p['port'])) $ports[(int)$p['port']] = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
$scheme = $parts['scheme'] ?? 'http';
|
||||||
|
$path = $parts['path'] ?? '';
|
||||||
|
$out = [];
|
||||||
|
|
||||||
|
foreach (array_keys($ports) as $port) {
|
||||||
|
$c = $scheme . '://' . $parts['host'] . ':' . $port . $path;
|
||||||
|
if ($c !== $observed) $out[$c] = true;
|
||||||
|
}
|
||||||
|
foreach (array_keys($hosts) as $host) {
|
||||||
|
$port = !empty($parts['port']) ? ':' . $parts['port'] : '';
|
||||||
|
$c = $scheme . '://' . $host . $port . $path;
|
||||||
|
if ($c !== $observed) $out[$c] = true;
|
||||||
|
}
|
||||||
|
return array_keys($out);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try to prove a correction for one finding. Returns the finding with 'proposed' and 'proven'
|
||||||
|
// filled in, or unchanged when nothing could be proven — which is the common case and not a
|
||||||
|
// failure.
|
||||||
|
//
|
||||||
|
// auth_rejected and missing_value are never proven here on purpose. A credential cannot be
|
||||||
|
// derived from this host by definition: if it could be read from somewhere, it would not be a
|
||||||
|
// credential. Those go straight to the operator.
|
||||||
|
function vv_ai_probe_finding(array $f): array {
|
||||||
|
$kind = (string)($f['kind'] ?? '');
|
||||||
|
|
||||||
|
if ($kind === 'unknown_target') {
|
||||||
|
$observed = (string)($f['observed'] ?? '');
|
||||||
|
// Exact membership only. A container name is a literal, and "close to an existing name"
|
||||||
|
// is how a repair renames the wrong thing.
|
||||||
|
if (in_array($observed, vv_ai_container_names(), true)) {
|
||||||
|
return $f; // it exists after all — nothing to correct
|
||||||
|
}
|
||||||
|
return $f + ['state' => 'needs_operator'];
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($kind !== 'unreachable') {
|
||||||
|
// Nothing on this machine can supply a credential, so there is nothing to prove.
|
||||||
|
$f['state'] = 'needs_operator';
|
||||||
|
return $f;
|
||||||
|
}
|
||||||
|
|
||||||
|
$observed = (string)($f['observed'] ?? '');
|
||||||
|
|
||||||
|
// If the observed address answers now, the fault has cleared on its own — a host that was
|
||||||
|
// rebooting, most often. Recording a proposal here would repair something already working.
|
||||||
|
if (vv_ai_probe_url($observed)['ok']) {
|
||||||
|
$f['state'] = 'resolved';
|
||||||
|
$f['note'] = 'Answered when probed — the address was reachable again by the time this ran.';
|
||||||
|
return $f;
|
||||||
|
}
|
||||||
|
|
||||||
|
$answered = [];
|
||||||
|
foreach (vv_ai_url_candidates($observed) as $candidate) {
|
||||||
|
if (vv_ai_probe_url($candidate)['ok']) $answered[] = $candidate;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Exactly one, or none. Two addresses answering means the probe cannot say which is the
|
||||||
|
// right one, and picking either is the guess this exists to prevent.
|
||||||
|
if (count($answered) === 1) {
|
||||||
|
$f['proposed'] = $answered[0];
|
||||||
|
$f['proven'] = true;
|
||||||
|
$f['note'] = 'Probed ' . $answered[0] . ' and it answered; ' . $observed . ' did not.';
|
||||||
|
return $f;
|
||||||
|
}
|
||||||
|
|
||||||
|
$f['proven'] = false;
|
||||||
|
$f['state'] = 'needs_operator';
|
||||||
|
$f['note'] = $answered
|
||||||
|
? 'Several addresses answered (' . implode(', ', $answered) . '), so none was written.'
|
||||||
|
: 'Nothing answered at ' . $observed . ', and no address in the conf answered either.';
|
||||||
|
return $f;
|
||||||
|
}
|
||||||
@@ -18,7 +18,27 @@
|
|||||||
//
|
//
|
||||||
// Structure only; values are not validated.
|
// Structure only; values are not validated.
|
||||||
// Consistent with conf_upgrade.sh, this reconciles shape and leaves correctness to the
|
// Consistent with conf_upgrade.sh, this reconciles shape and leaves correctness to the
|
||||||
// consuming script.
|
// consuming script. Shape is enforced — the file must parse, source, and read back the
|
||||||
|
// value that was asked for. Whether 300 is a sensible timeout is still not this file's
|
||||||
|
// question.
|
||||||
|
//
|
||||||
|
// One guarded write path, not one set of guards per caller.
|
||||||
|
// vv_conf_edit() owns the lock, the backup, the validation and the audit line, and every
|
||||||
|
// conf write in the plugin goes through it — the Settings form, both raw editors, the
|
||||||
|
// flag toggles, orchestrator membership, script moves and reorders, the rsync window
|
||||||
|
// arrays, the docker folder map and first-run setup. vv_write_conf_raw() has exactly one
|
||||||
|
// caller left, inside vv_conf_edit() itself.
|
||||||
|
//
|
||||||
|
// A caller that reaches past it gets tmp + rename and nothing else: no backup, no bash -n,
|
||||||
|
// no read-back, no audit line. That is how this started — seven files each with their own
|
||||||
|
// partial idea of what a safe conf write was, two of them carrying a copy-pasted bash -n
|
||||||
|
// block that failed open.
|
||||||
|
//
|
||||||
|
// The rewrite happens inside the lock, or it proves nothing moved.
|
||||||
|
// $mutate receives the current contents, so a caller that can rebuild from them has no
|
||||||
|
// window at all. Callers that must assemble the result first — the ones driven by a form
|
||||||
|
// payload — compare against what they read and return null if it no longer matches,
|
||||||
|
// which abandons the write rather than reverting a concurrent edit.
|
||||||
//
|
//
|
||||||
// OPERATIONAL SAFEGUARDS
|
// OPERATIONAL SAFEGUARDS
|
||||||
// An unmatched section yields no fields rather than a wrong write.
|
// An unmatched section yields no fields rather than a wrong write.
|
||||||
@@ -30,6 +50,33 @@
|
|||||||
// Each field carries the exact line it came from, so a write cannot land outside the
|
// Each field carries the exact line it came from, so a write cannot land outside the
|
||||||
// subsection it was read from.
|
// subsection it was read from.
|
||||||
//
|
//
|
||||||
|
// A value is shell source code, so command substitution is refused outright.
|
||||||
|
// Scalars are written inside double quotes and array values are spliced in verbatim, so
|
||||||
|
// $(...) or a backtick in a value runs as root in every script that sources the conf. No
|
||||||
|
// conf in this repo uses either, so rejecting them costs nothing. $VAR and ${VAR} stay
|
||||||
|
// legal — AI_DATA_DIR="${DATA_DIR}/ai" is the established idiom, and a reference resolves
|
||||||
|
// to a value where a substitution runs a program.
|
||||||
|
//
|
||||||
|
// Nothing is written without a backup in hand.
|
||||||
|
// The previous contents are copied to CONF_BACKUP_DIR/<file>.<stamp> first, and a backup
|
||||||
|
// that cannot be taken cancels the write. The confs are gitignored, so that directory is
|
||||||
|
// the entire recovery story — there is no history to revert to.
|
||||||
|
//
|
||||||
|
// Validated, then verified, then rolled back on failure.
|
||||||
|
// bash -n proves the candidate parses; sourcing the installed file and reading the keys
|
||||||
|
// back proves the values survived quoting. A scalar that does not read back as the value
|
||||||
|
// requested restores the backup. A conf that parses cleanly and holds the wrong string is
|
||||||
|
// the failure a syntax check cannot see.
|
||||||
|
//
|
||||||
|
// Held under an exclusive lock for the whole read-modify-write.
|
||||||
|
// Two concurrent savers would otherwise read the same original, and the second rename
|
||||||
|
// would discard the first one's change without either reporting a failure.
|
||||||
|
//
|
||||||
|
// Every outcome is logged, and secrets are logged by name only.
|
||||||
|
// LOG_DIR/conf_changes.log records applied, rejected, failed and rolled-back alike. A
|
||||||
|
// credential-shaped key logs value=<redacted> — the log proves a change happened, it is
|
||||||
|
// not a second copy of the secret.
|
||||||
|
//
|
||||||
// EXPORTS
|
// EXPORTS
|
||||||
// vv_conf_has_sections() does this script have an editable conf section
|
// vv_conf_has_sections() does this script have an editable conf section
|
||||||
// vv_conf_parse_subsection() fields within one named subsection
|
// vv_conf_parse_subsection() fields within one named subsection
|
||||||
@@ -38,7 +85,10 @@
|
|||||||
// vv_conf_write_changes() apply edits back to the conf file
|
// vv_conf_write_changes() apply edits back to the conf file
|
||||||
//
|
//
|
||||||
// CONFIGURATION
|
// CONFIGURATION
|
||||||
// CONF_DIR master.conf and host*.conf are the read and write targets
|
// CONF_DIR master.conf and host*.conf are the read and write targets
|
||||||
|
// CONF_BACKUP_DIR DATA_DIR/Backups/Confs — pre-write copies, 0700
|
||||||
|
// LOG_DIR conf_changes.log is written here
|
||||||
|
// CONF_BACKUP_RETAIN backups kept per conf file (default 30)
|
||||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||||
require_once __DIR__ . '/config.php';
|
require_once __DIR__ . '/config.php';
|
||||||
|
|
||||||
@@ -110,6 +160,8 @@ const VV_SCRIPT_CONF_SECTIONS = [
|
|||||||
'AI/ai_index.sh' => ['AI Retrieval Index', 'AI Master Switch', 'Ollama'],
|
'AI/ai_index.sh' => ['AI Retrieval Index', 'AI Master Switch', 'Ollama'],
|
||||||
'AI/ai_query.sh' => ['AI Retrieval Index', 'AI Master Switch', 'Ollama'],
|
'AI/ai_query.sh' => ['AI Retrieval Index', 'AI Master Switch', 'Ollama'],
|
||||||
'AI/ai_token_sync.sh' => ['AI Feature Toggles'],
|
'AI/ai_token_sync.sh' => ['AI Feature Toggles'],
|
||||||
|
'Plugin/unraid/Tools/ai_repair_sweep.sh' => ['AI Repair', 'AI Repair Findings',
|
||||||
|
'AI Master Switch'],
|
||||||
// ── Shared host sections ──────────────────────────────────────────────────────────────────
|
// ── Shared host sections ──────────────────────────────────────────────────────────────────
|
||||||
// A script's settings are not only the ones named after it. Anything talking to Lidarr reads
|
// A script's settings are not only the ones named after it. Anything talking to Lidarr reads
|
||||||
// the host's Lidarr block; anything reading playback reads Emby. Those blocks are where the
|
// the host's Lidarr block; anything reading playback reads Emby. Those blocks are where the
|
||||||
@@ -307,19 +359,132 @@ function vv_conf_fields_for_script(string $id): array {
|
|||||||
return $groups;
|
return $groups;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A conf key must be a plain shell identifier. Every downstream use — the replacement regex,
|
||||||
|
// the source-verification subshell, the audit line — treats the key as trusted text, so it is
|
||||||
|
// validated once here rather than escaped differently in three places.
|
||||||
|
function vv_conf_key_valid(string $key): bool {
|
||||||
|
return (bool) preg_match('/^[A-Za-z_][A-Za-z0-9_]*$/', $key);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Credential-shaped keys are logged by name only. The audit log is the record that a change
|
||||||
|
// happened, not a second copy of the secret that changed.
|
||||||
|
function vv_conf_key_is_secret(string $key): bool {
|
||||||
|
return (bool) preg_match('/(PASS|PASSWORD|SECRET|TOKEN|API_KEY|APIKEY|_KEY)$|(PASS|PASSWORD|SECRET|TOKEN|APIKEY)/i', $key);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scalar values are written inside double quotes and array values are spliced in verbatim, so
|
||||||
|
// a value is shell source code the moment any script reads the conf. Command substitution in a
|
||||||
|
// value therefore executes on every load — in every script, as root. No conf in this repo uses
|
||||||
|
// it, so rejecting it costs nothing and closes the path.
|
||||||
|
//
|
||||||
|
// $VAR and ${VAR} stay legal on purpose: AI_DATA_DIR="${DATA_DIR}/ai" is the established idiom
|
||||||
|
// here, and a reference resolves to a value where a substitution runs a program.
|
||||||
|
function vv_conf_value_safe(string $value): bool {
|
||||||
|
return !preg_match('/\$\(|`|<\(|>\(/', $value);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Under DATA_DIR, not beside the confs. data/ is the one on-disk root and is gitignored whole,
|
||||||
|
// so a backup here cannot become a tracked file the way Configurations/*.bak did. 0700 because
|
||||||
|
// these are verbatim copies of files holding every credential on the host.
|
||||||
|
function vv_conf_backup_dir(): string {
|
||||||
|
if (!is_dir(CONF_BACKUP_DIR)) @mkdir(CONF_BACKUP_DIR, 0700, true);
|
||||||
|
return CONF_BACKUP_DIR;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy the current conf aside before it is touched. Returns the backup path, or null when no
|
||||||
|
// backup could be taken — which the caller treats as a reason not to write, because the whole
|
||||||
|
// recovery story for these files is this directory.
|
||||||
|
function vv_conf_backup(string $filename): ?string {
|
||||||
|
$src = CONF_DIR . '/' . $filename;
|
||||||
|
if (!is_file($src)) return null;
|
||||||
|
|
||||||
|
$dest = vv_conf_backup_dir() . '/' . $filename . '.' . date('Y-m-d\THis');
|
||||||
|
// Same second, second change: keep both rather than silently overwrite the older one.
|
||||||
|
if (file_exists($dest)) {
|
||||||
|
$n = 1;
|
||||||
|
while (file_exists($dest . '.' . $n)) $n++;
|
||||||
|
$dest .= '.' . $n;
|
||||||
|
}
|
||||||
|
if (!@copy($src, $dest)) return null;
|
||||||
|
@chmod($dest, 0600);
|
||||||
|
|
||||||
|
vv_conf_prune_backups($filename);
|
||||||
|
return $dest;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Retain the newest CONF_BACKUP_RETAIN backups per conf file. Pruned by filename, which sorts
|
||||||
|
// chronologically because the stamp is ISO-8601 — no stat() per candidate.
|
||||||
|
function vv_conf_prune_backups(string $filename): void {
|
||||||
|
$keep = (int) (vv_conf_vars()['CONF_BACKUP_RETAIN'] ?? 30);
|
||||||
|
if ($keep < 1) $keep = 30;
|
||||||
|
|
||||||
|
$found = glob(vv_conf_backup_dir() . '/' . $filename . '.*') ?: [];
|
||||||
|
if (count($found) <= $keep) return;
|
||||||
|
|
||||||
|
sort($found);
|
||||||
|
foreach (array_slice($found, 0, count($found) - $keep) as $old) @unlink($old);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Source a candidate conf in a subshell and read back the keys that were just written.
|
||||||
|
// bash -n proves the file parses; this proves the values survived quoting and arrived as
|
||||||
|
// intended. Returns null when the file could not be sourced at all.
|
||||||
|
function vv_conf_read_back(string $path, array $keys): ?array {
|
||||||
|
$script = 'source ' . escapeshellarg($path) . ' >/dev/null 2>&1 || exit 90; ';
|
||||||
|
foreach ($keys as $k) {
|
||||||
|
if (!vv_conf_key_valid($k)) continue;
|
||||||
|
$script .= 'printf "%s\t%s\n" ' . escapeshellarg($k) . ' "${' . $k . '-}"; ';
|
||||||
|
}
|
||||||
|
|
||||||
|
$out = []; $rc = 0;
|
||||||
|
exec('bash -c ' . escapeshellarg($script) . ' 2>/dev/null', $out, $rc);
|
||||||
|
if ($rc !== 0) return null;
|
||||||
|
|
||||||
|
$vals = [];
|
||||||
|
foreach ($out as $line) {
|
||||||
|
$parts = explode("\t", $line, 2);
|
||||||
|
if (count($parts) === 2) $vals[$parts[0]] = $parts[1];
|
||||||
|
}
|
||||||
|
return $vals;
|
||||||
|
}
|
||||||
|
|
||||||
|
// One line per conf change, best-effort and never able to block the write itself.
|
||||||
|
function vv_conf_audit(string $file, string $key, string $outcome, string $detail = ''): void {
|
||||||
|
$line = date('Y-m-d H:i:s')
|
||||||
|
. " file={$file} key={$key} outcome={$outcome}"
|
||||||
|
. ($detail !== '' ? " {$detail}" : '')
|
||||||
|
. ' ip=' . ($_SERVER['REMOTE_ADDR'] ?? 'cli')
|
||||||
|
. "\n";
|
||||||
|
@file_put_contents(LOG_DIR . '/conf_changes.log', $line, FILE_APPEND | LOCK_EX);
|
||||||
|
}
|
||||||
|
|
||||||
// Write a batch of field changes back to their respective conf files.
|
// Write a batch of field changes back to their respective conf files.
|
||||||
// Each change: {file, key, value, type}
|
// Each change: {file, key, value, type}
|
||||||
function vv_conf_write_changes(array $changes): array {
|
function vv_conf_write_changes(array $changes): array {
|
||||||
$byFile = [];
|
$byFile = [];
|
||||||
foreach ($changes as $c) {
|
foreach ($changes as $c) {
|
||||||
if (!empty($c['file']) && !empty($c['key'])) $byFile[$c['file']][] = $c;
|
if (empty($c['file']) || empty($c['key'])) continue;
|
||||||
|
if (!vv_conf_key_valid($c['key'])) {
|
||||||
|
vv_conf_audit((string) $c['file'], (string) $c['key'], 'rejected', 'reason=malformed-key');
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!vv_conf_value_safe((string) ($c['value'] ?? ''))) {
|
||||||
|
vv_conf_audit((string) $c['file'], (string) $c['key'], 'rejected', 'reason=command-substitution');
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
$byFile[$c['file']][] = $c;
|
||||||
}
|
}
|
||||||
|
|
||||||
$results = [];
|
$results = [];
|
||||||
foreach ($byFile as $file => $fileChanges) {
|
foreach ($byFile as $file => $fileChanges) {
|
||||||
$raw = vv_read_conf_raw($file);
|
$results[$file] = vv_conf_write_file($file, $fileChanges);
|
||||||
if ($raw === '') { $results[$file] = false; continue; }
|
}
|
||||||
|
return $results;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read-modify-write for one conf file. The surgical replacement lives here; the guards that
|
||||||
|
// make installing it safe live in vv_conf_install(), which every conf writer shares.
|
||||||
|
function vv_conf_write_file(string $file, array $fileChanges): bool {
|
||||||
|
return vv_conf_edit($file, function (string $raw) use ($fileChanges): ?string {
|
||||||
foreach ($fileChanges as $c) {
|
foreach ($fileChanges as $c) {
|
||||||
$qKey = preg_quote($c['key'], '/');
|
$qKey = preg_quote($c['key'], '/');
|
||||||
$value = $c['value'];
|
$value = $c['value'];
|
||||||
@@ -328,7 +493,10 @@ function vv_conf_write_changes(array $changes): array {
|
|||||||
if ($type === 'scalar') {
|
if ($type === 'scalar') {
|
||||||
$raw = preg_replace_callback(
|
$raw = preg_replace_callback(
|
||||||
'/^(\s*' . $qKey . '\s*=\s*)("(?:[^"\\\\]|\\\\.)*"|\'(?:[^\'\\\\]|\\\\.)*\'|[^#\n]*?)(\s*(?:#[^\n]*)?)$/m',
|
'/^(\s*' . $qKey . '\s*=\s*)("(?:[^"\\\\]|\\\\.)*"|\'(?:[^\'\\\\]|\\\\.)*\'|[^#\n]*?)(\s*(?:#[^\n]*)?)$/m',
|
||||||
fn($m) => $m[1] . '"' . str_replace(['"', '\\'], ['\\"', '\\\\'], $value) . '"' . $m[3],
|
// Backslashes first. Escaping quotes first meant the backslash just inserted
|
||||||
|
// was itself doubled on the next pass — " became \\" — which closed the
|
||||||
|
// string early and stored a truncated value that still parsed cleanly.
|
||||||
|
fn($m) => $m[1] . '"' . str_replace(['\\', '"'], ['\\\\', '\\"'], $value) . '"' . $m[3],
|
||||||
$raw
|
$raw
|
||||||
) ?? $raw;
|
) ?? $raw;
|
||||||
|
|
||||||
@@ -354,23 +522,154 @@ function vv_conf_write_changes(array $changes): array {
|
|||||||
) ?? $raw;
|
) ?? $raw;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Only the scalar path escapes its value; the array, array_single and assoc_array
|
|
||||||
// paths splice the caller's text into the file verbatim, and the type comes from the
|
return $raw;
|
||||||
// request. Every script sources these files, so the result is parsed before it is
|
|
||||||
// allowed to replace a working conf.
|
// Scalars are verified because their intended value is known exactly. The array types splice
|
||||||
$results[$file] = vv_conf_syntax_ok($raw) && vv_write_conf_raw($file, $raw);
|
// caller-supplied text whose sourced form is legitimately not equal to what was written, so
|
||||||
|
// for those a clean source is the whole assertion.
|
||||||
|
}, vv_conf_expected_scalars($fileChanges), array_column($fileChanges, 'key'));
|
||||||
|
}
|
||||||
|
|
||||||
|
// The intended end state for the keys that can be checked against one.
|
||||||
|
function vv_conf_expected_scalars(array $fileChanges): array {
|
||||||
|
$expect = [];
|
||||||
|
foreach ($fileChanges as $c) {
|
||||||
|
if (($c['type'] ?? 'scalar') === 'scalar') $expect[$c['key']] = (string) $c['value'];
|
||||||
}
|
}
|
||||||
return $results;
|
return $expect;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The one guarded way to install a changed conf. $mutate receives the current contents and
|
||||||
|
// returns the rewritten ones, or null to abort without touching anything. Every conf writer in
|
||||||
|
// the plugin goes through here, so the lock, the backup, the validation and the audit trail are
|
||||||
|
// written once and cannot be forgotten by a new caller.
|
||||||
|
//
|
||||||
|
// $expect key => intended value, verified by sourcing the installed file
|
||||||
|
// $subjects names for the audit line, when there is no key to assert (a toggled member)
|
||||||
|
// $allowCreate write a conf that does not exist yet. Off by default: for every caller except
|
||||||
|
// first-run setup, a missing target means the filename is wrong, and creating it
|
||||||
|
// would leave a stray conf that shadows nothing and is sourced by nobody.
|
||||||
|
function vv_conf_edit(string $file, callable $mutate, array $expect = [], array $subjects = [],
|
||||||
|
bool $allowCreate = false): bool {
|
||||||
|
$path = CONF_DIR . '/' . $file;
|
||||||
|
$subjects = $subjects ?: (array_keys($expect) ?: ['-']);
|
||||||
|
|
||||||
|
$audit = function (string $outcome, string $detail = '') use ($file, $subjects): void {
|
||||||
|
foreach ($subjects as $s) vv_conf_audit($file, (string) $s, $outcome, $detail);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Held across the whole read-modify-write. Two concurrent savers would otherwise each read
|
||||||
|
// the same original, and the second rename would silently discard the first one's change.
|
||||||
|
$lockFh = @fopen(CONF_DIR . '/.conf-write.lock', 'c');
|
||||||
|
if ($lockFh === false || !flock($lockFh, LOCK_EX)) {
|
||||||
|
if ($lockFh) fclose($lockFh);
|
||||||
|
$audit('failed', 'reason=lock');
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
// A conf that does not exist yet is a create, not an edit — first-run setup writes
|
||||||
|
// host*.conf before there is anything to read. There is no prior content to lose, so
|
||||||
|
// there is nothing to back up, and undoing a failed create means removing the file.
|
||||||
|
$exists = is_file($path);
|
||||||
|
if (!$exists && !$allowCreate) { $audit('failed', 'reason=unreadable'); return false; }
|
||||||
|
$raw = $exists ? vv_read_conf_raw($file) : '';
|
||||||
|
if ($exists && $raw === '') { $audit('failed', 'reason=unreadable'); return false; }
|
||||||
|
|
||||||
|
$before = $exists ? (vv_conf_read_back($path, $subjects) ?? []) : [];
|
||||||
|
|
||||||
|
$new = $mutate($raw);
|
||||||
|
if ($new === null) { $audit('failed', 'reason=no-match'); return false; }
|
||||||
|
if ($exists && $new === $raw) { $audit('no-change'); return true; }
|
||||||
|
|
||||||
|
$backup = null;
|
||||||
|
if ($exists) {
|
||||||
|
$backup = vv_conf_backup($file);
|
||||||
|
if ($backup === null) {
|
||||||
|
// No recovery path for this write means the write does not happen. These files
|
||||||
|
// are gitignored, so a backup not taken cannot be reconstructed afterwards.
|
||||||
|
$audit('failed', 'reason=backup');
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
$undo = function () use ($backup, $path): void {
|
||||||
|
if ($backup !== null) @copy($backup, $path); else @unlink($path);
|
||||||
|
};
|
||||||
|
|
||||||
|
if (!vv_conf_syntax_ok($new)) { $audit('rejected', 'reason=syntax'); return false; }
|
||||||
|
if (!vv_write_conf_raw($file, $new)) { $audit('failed', 'reason=write'); return false; }
|
||||||
|
|
||||||
|
// bash -n proved the candidate parses. This proves the installed file still sources and
|
||||||
|
// that each value arrived intact — a quoting bug produces a file that parses perfectly
|
||||||
|
// and holds the wrong string, which is the failure the syntax check cannot see.
|
||||||
|
$after = vv_conf_read_back($path, $subjects);
|
||||||
|
if ($after === null) {
|
||||||
|
$undo();
|
||||||
|
$audit('rolled-back', 'reason=source-failed');
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($expect as $key => $want) {
|
||||||
|
if (($after[$key] ?? null) !== $want) {
|
||||||
|
$undo();
|
||||||
|
vv_conf_audit($file, $key, 'rolled-back', 'reason=value-mismatch');
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($subjects as $s) {
|
||||||
|
$s = (string) $s;
|
||||||
|
// A subject that is not a conf key — a script id, or a marker for a whole-file save
|
||||||
|
// — has no value to read back, so there is no before and after to report.
|
||||||
|
if (!vv_conf_key_valid($s)) { vv_conf_audit($file, $s, 'applied'); continue; }
|
||||||
|
if (vv_conf_key_is_secret($s)) { vv_conf_audit($file, $s, 'applied', 'value=<redacted>'); continue; }
|
||||||
|
vv_conf_audit($file, $s, 'applied',
|
||||||
|
'from=' . vv_conf_audit_val($before[$s] ?? '') . ' to=' . vv_conf_audit_val($after[$s] ?? ''));
|
||||||
|
}
|
||||||
|
|
||||||
|
return true;
|
||||||
|
|
||||||
|
} finally {
|
||||||
|
flock($lockFh, LOCK_UN);
|
||||||
|
fclose($lockFh);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Audit values are single-line and bounded. A conf value can be a 40-line array; the log is a
|
||||||
|
// record of what changed, and an unbounded splat of it makes the log unreadable at the moment
|
||||||
|
// it is actually needed.
|
||||||
|
function vv_conf_audit_val(string $v): string {
|
||||||
|
$v = preg_replace('/\s+/', ' ', trim($v));
|
||||||
|
if (strlen($v) > 120) $v = substr($v, 0, 117) . '...';
|
||||||
|
return '"' . $v . '"';
|
||||||
}
|
}
|
||||||
|
|
||||||
// bash -n against a private temp copy. Returns true when the content parses as a sourceable
|
// bash -n against a private temp copy. Returns true when the content parses as a sourceable
|
||||||
// conf, false otherwise — never writes anything itself.
|
// conf, false otherwise — never writes anything itself.
|
||||||
function vv_conf_syntax_ok(string $content): bool {
|
function vv_conf_syntax_ok(string $content): bool {
|
||||||
|
return vv_conf_syntax_error($content) === null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The same check, with bash's own complaint when it fails — the raw editors show it to whoever
|
||||||
|
// is typing, where "conf does not parse" alone would mean hunting the line by hand. $label
|
||||||
|
// replaces the temp path in the message so the reader sees their own filename.
|
||||||
|
//
|
||||||
|
// Fails closed. This used to pass when the temp file could not be created, which was defensible
|
||||||
|
// while every write came from a human clicking Save on a form. The assistant writes through here
|
||||||
|
// too, so an unverified conf is not installed — a refused write is recoverable, a conf that no
|
||||||
|
// script can source is a system-wide outage.
|
||||||
|
function vv_conf_syntax_error(string $content, string $label = 'conf'): ?string {
|
||||||
$tmp = tempnam(sys_get_temp_dir(), 'vvconf');
|
$tmp = tempnam(sys_get_temp_dir(), 'vvconf');
|
||||||
if ($tmp === false) return true; // cannot check — do not block the write
|
if ($tmp === false) return 'cannot verify: no writable temp directory';
|
||||||
|
|
||||||
file_put_contents($tmp, $content);
|
file_put_contents($tmp, $content);
|
||||||
$out = []; $rc = 0;
|
$out = []; $rc = 0;
|
||||||
exec('bash -n ' . escapeshellarg($tmp) . ' 2>&1', $out, $rc);
|
exec('bash -n ' . escapeshellarg($tmp) . ' 2>&1', $out, $rc);
|
||||||
@unlink($tmp);
|
@unlink($tmp);
|
||||||
return $rc === 0;
|
if ($rc === 0) return null;
|
||||||
|
|
||||||
|
$msg = implode(' ', array_filter(array_map('trim', $out)));
|
||||||
|
return str_replace($tmp, $label, $msg ?: 'conf does not parse');
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -117,6 +117,8 @@ define('STATE_DIR', DATA_DIR . '/state');
|
|||||||
define('AI_DATA_DIR', DATA_DIR . '/ai');
|
define('AI_DATA_DIR', DATA_DIR . '/ai');
|
||||||
define('CACHE_BACKUP_DIR', DATA_DIR . '/cache');
|
define('CACHE_BACKUP_DIR', DATA_DIR . '/cache');
|
||||||
define('LOG_ARCHIVE_DIR', DATA_DIR . '/logs');
|
define('LOG_ARCHIVE_DIR', DATA_DIR . '/logs');
|
||||||
|
define('BACKUP_DIR', DATA_DIR . '/Backups');
|
||||||
|
define('CONF_BACKUP_DIR', BACKUP_DIR . '/Confs');
|
||||||
define('LOG_DIR', '/var/log/varaverk');
|
define('LOG_DIR', '/var/log/varaverk');
|
||||||
// User-authored custom scripts (scheduler page "+ Create Script") — kept outside the git
|
// User-authored custom scripts (scheduler page "+ Create Script") — kept outside the git
|
||||||
// repo entirely, alongside the User Scripts plugin's own storage. Any *.sh file placed
|
// repo entirely, alongside the User Scripts plugin's own storage. Any *.sh file placed
|
||||||
@@ -298,8 +300,13 @@ function vv_push_master_conf(): array {
|
|||||||
return $results;
|
return $results;
|
||||||
}
|
}
|
||||||
|
|
||||||
function vv_get_hostname(): string {
|
// Cached: this forks a shell, and it is reached from vv_conf_vars() by way of vv_detect_host(),
|
||||||
return trim(shell_exec('hostname -s') ?: '');
|
// which the repair sweep calls per log line. A machine does not rename itself mid-request.
|
||||||
|
function vv_get_hostname(bool $flush = false): string {
|
||||||
|
static $name = null;
|
||||||
|
if ($flush) { $name = null; return ''; }
|
||||||
|
if ($name === null) $name = trim(shell_exec('hostname -s') ?: '');
|
||||||
|
return $name;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Mirror of common.sh resolve_tailscale_ip(): tries `tailscale ip -4` first (Tailscale manages
|
// Mirror of common.sh resolve_tailscale_ip(): tries `tailscale ip -4` first (Tailscale manages
|
||||||
@@ -326,7 +333,19 @@ function vv_resolve_tailscale_ip(string $hostname): string {
|
|||||||
return count($matches) === 1 ? $matches[0] : '';
|
return count($matches) === 1 ? $matches[0] : '';
|
||||||
}
|
}
|
||||||
|
|
||||||
function vv_detect_host(): string {
|
// Cached alongside the others: this reads master.conf in full and is called by vv_conf_vars() on
|
||||||
|
// every lookup, so leaving it uncached would mean the conf is still read once per key even with
|
||||||
|
// the parsed values cached.
|
||||||
|
function vv_detect_host(bool $flush = false): string {
|
||||||
|
static $host = null;
|
||||||
|
if ($flush) { $host = null; return ''; }
|
||||||
|
if ($host !== null) return $host;
|
||||||
|
|
||||||
|
$host = _vv_detect_host_uncached();
|
||||||
|
return $host;
|
||||||
|
}
|
||||||
|
|
||||||
|
function _vv_detect_host_uncached(): string {
|
||||||
// Reads master.conf for HOST1="name" (or HOST1_NAME="name") and matches running hostname.
|
// Reads master.conf for HOST1="name" (or HOST1_NAME="name") and matches running hostname.
|
||||||
// Returns 'host1', 'host2', 'host3', ... or 'unknown'. Works for any number of hosts.
|
// Returns 'host1', 'host2', 'host3', ... or 'unknown'. Works for any number of hosts.
|
||||||
$master = vv_read_conf_raw('master.conf');
|
$master = vv_read_conf_raw('master.conf');
|
||||||
@@ -395,7 +414,13 @@ function vv_write_conf_raw(string $filename, string $content): bool {
|
|||||||
if ($content !== '' && !str_ends_with($content, "\n")) $content .= "\n";
|
if ($content !== '' && !str_ends_with($content, "\n")) $content .= "\n";
|
||||||
|
|
||||||
if (file_put_contents($tmp, $content) === false) return false;
|
if (file_put_contents($tmp, $content) === false) return false;
|
||||||
return rename($tmp, $path);
|
if (!rename($tmp, $path)) return false;
|
||||||
|
|
||||||
|
// Every conf write in the plugin lands here, so this is the one place the parsed-conf cache
|
||||||
|
// has to be dropped. Doing it in the callers instead would mean a new writer inheriting a
|
||||||
|
// stale cache and no obvious reason why.
|
||||||
|
vv_conf_vars_flush();
|
||||||
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
function vv_get_conf_files(): array {
|
function vv_get_conf_files(): array {
|
||||||
@@ -418,15 +443,59 @@ function vv_get_conf_files(): array {
|
|||||||
return $files;
|
return $files;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Parse conf into key=>value map for $VAR substitution in docs
|
// Drops the parsed-conf cache. Called by vv_write_conf_raw() — the single point at which a conf
|
||||||
function vv_conf_vars(): array {
|
// changes on disk — so no writer has to remember to do it.
|
||||||
$vars = [];
|
//
|
||||||
|
// Necessary on top of the mtime check below, not instead of it: filesystem mtimes have one-second
|
||||||
|
// resolution, and a write followed by a read inside the same second is exactly what the conf
|
||||||
|
// writer does when it verifies a value it just wrote. Without this, that read could be served the
|
||||||
|
// value from before the write and the verification would compare a value against itself.
|
||||||
|
function vv_conf_vars_flush(): void {
|
||||||
|
vv_conf_vars(true);
|
||||||
|
// Host identity is derived from master.conf too, so a write that changes HOST1 has to
|
||||||
|
// invalidate it as well — first-run setup does exactly that, then asks which host this is.
|
||||||
|
vv_detect_host(true);
|
||||||
|
vv_get_hostname(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse conf into key=>value map for $VAR substitution in docs.
|
||||||
|
//
|
||||||
|
// Cached, because this is not the cheap function its callers assume. It reads and regex-parses
|
||||||
|
// both conf files on every call — around 1,900 lines — and the repair sweep asks it for a key per
|
||||||
|
// log line. The measured cost of exactly this pattern is on record: the 30 minutes play_state_sync
|
||||||
|
// spent in 2026-07 was parse and fork overhead, not the API it was blamed on.
|
||||||
|
//
|
||||||
|
// Keyed on a hash of the contents, so an edit made outside this process — the raw editor in
|
||||||
|
// another tab, conf_upgrade from bash, a hand edit over SSH — is still picked up.
|
||||||
|
//
|
||||||
|
// Content rather than mtime and size, which was the first attempt and was wrong. Both conf files
|
||||||
|
// were rewritten within the same second and to the same byte count — "true" and "false" trading
|
||||||
|
// places across two keys — and the stale values were served straight back. mtime has one-second
|
||||||
|
// resolution and a same-size edit is not a rare shape in a file of booleans.
|
||||||
|
//
|
||||||
|
// Reading both files every call is not what costs anything here; parsing them is. The read is
|
||||||
|
// tens of microseconds against a parse of ~1,900 lines followed by two resolution passes.
|
||||||
|
function vv_conf_vars(bool $flush = false): array {
|
||||||
|
static $cache = null;
|
||||||
|
static $stamp = null;
|
||||||
|
|
||||||
|
if ($flush) { $cache = null; $stamp = null; return []; }
|
||||||
|
|
||||||
$files = ['master.conf'];
|
$files = ['master.conf'];
|
||||||
$host = vv_detect_host();
|
$host = vv_detect_host();
|
||||||
if (preg_match('/^host\d+$/', $host)) $files[] = $host . '.conf';
|
if (preg_match('/^host\d+$/', $host)) $files[] = $host . '.conf';
|
||||||
|
|
||||||
|
$raws = [];
|
||||||
|
$sig = '';
|
||||||
foreach ($files as $f) {
|
foreach ($files as $f) {
|
||||||
$raw = vv_read_conf_raw($f);
|
$raws[$f] = vv_read_conf_raw($f);
|
||||||
|
$sig .= md5($raws[$f]);
|
||||||
|
}
|
||||||
|
if ($cache !== null && $stamp === $sig) return $cache;
|
||||||
|
|
||||||
|
$vars = [];
|
||||||
|
foreach ($files as $f) {
|
||||||
|
$raw = $raws[$f];
|
||||||
// Match: VAR_NAME="value" or VAR_NAME=value (no quotes)
|
// Match: VAR_NAME="value" or VAR_NAME=value (no quotes)
|
||||||
preg_match_all('/^\s*([A-Z0-9_]+)\s*=\s*["\']?([^"\'#\n]*?)["\']?\s*(?:#.*)?$/m', $raw, $m);
|
preg_match_all('/^\s*([A-Z0-9_]+)\s*=\s*["\']?([^"\'#\n]*?)["\']?\s*(?:#.*)?$/m', $raw, $m);
|
||||||
foreach ($m[1] as $i => $key) {
|
foreach ($m[1] as $i => $key) {
|
||||||
@@ -447,7 +516,9 @@ function vv_conf_vars(): array {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
unset($v);
|
unset($v);
|
||||||
return $vars;
|
|
||||||
|
$stamp = $sig;
|
||||||
|
return $cache = $vars;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Query the Unraid GraphQL API for a given host.
|
// Query the Unraid GraphQL API for a given host.
|
||||||
|
|||||||
@@ -51,6 +51,7 @@
|
|||||||
// Reads/writes HOST*_DOCKER_FOLDER_MAP in host*.conf (used by onboard scripts)
|
// Reads/writes HOST*_DOCKER_FOLDER_MAP in host*.conf (used by onboard scripts)
|
||||||
|
|
||||||
require_once __DIR__ . '/config.php';
|
require_once __DIR__ . '/config.php';
|
||||||
|
require_once __DIR__ . '/confform.php';
|
||||||
|
|
||||||
define('VV_DOCKER_JSON', SCRIPTS_DIR . '/docker_folders.json');
|
define('VV_DOCKER_JSON', SCRIPTS_DIR . '/docker_folders.json');
|
||||||
define('VV_FV3_JSON', '/boot/config/plugins/folder.view3/docker.json');
|
define('VV_FV3_JSON', '/boot/config/plugins/folder.view3/docker.json');
|
||||||
@@ -344,17 +345,19 @@ function vv_dk_rename_folder(string $folderId, string $newName): array {
|
|||||||
$data[$folderId]['name'] = $newName;
|
$data[$folderId]['name'] = $newName;
|
||||||
if (!vv_dk_write_json($data)) return ['ok' => false, 'error' => 'JSON write failed'];
|
if (!vv_dk_write_json($data)) return ['ok' => false, 'error' => 'JSON write failed'];
|
||||||
|
|
||||||
// Update conf: replace old folder name with new name in the map
|
// Update conf: replace old folder name with new name in the map.
|
||||||
|
// The rebuild is a pure function of the current contents, so it runs inside vv_conf_edit()'s
|
||||||
|
// lock rather than against a copy read beforehand — there is no window to lose an edit in.
|
||||||
$currentHost = vv_detect_host();
|
$currentHost = vv_detect_host();
|
||||||
$myId = strtoupper($currentHost);
|
$myId = strtoupper($currentHost);
|
||||||
$raw = vv_read_conf_raw($currentHost . '.conf');
|
vv_conf_edit($currentHost . '.conf', function (string $raw) use ($myId, $oldName, $newName): string {
|
||||||
$map = vv_dk_read_conf_map($raw, $myId);
|
$map = vv_dk_read_conf_map($raw, $myId);
|
||||||
foreach ($map as &$v) {
|
foreach ($map as &$v) {
|
||||||
if ($v === $oldName) $v = $newName;
|
if ($v === $oldName) $v = $newName;
|
||||||
}
|
}
|
||||||
unset($v);
|
unset($v);
|
||||||
$updated = vv_dk_write_conf_map($raw, $myId, $map);
|
return vv_dk_write_conf_map($raw, $myId, $map);
|
||||||
vv_write_conf_raw($currentHost . '.conf', $updated);
|
}, [], ["{$myId}_DOCKER_FOLDER_MAP"]);
|
||||||
return ['ok' => true];
|
return ['ok' => true];
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -367,14 +370,14 @@ function vv_dk_delete_folder(string $folderId): array {
|
|||||||
unset($data[$folderId]);
|
unset($data[$folderId]);
|
||||||
if (!vv_dk_write_json($data)) return ['ok' => false, 'error' => 'JSON write failed'];
|
if (!vv_dk_write_json($data)) return ['ok' => false, 'error' => 'JSON write failed'];
|
||||||
|
|
||||||
// Remove from conf map
|
// Remove from conf map — rebuilt inside the lock, see vv_dk_rename_folder() above.
|
||||||
$currentHost = vv_detect_host();
|
$currentHost = vv_detect_host();
|
||||||
$myId = strtoupper($currentHost);
|
$myId = strtoupper($currentHost);
|
||||||
$raw = vv_read_conf_raw($currentHost . '.conf');
|
vv_conf_edit($currentHost . '.conf', function (string $raw) use ($myId, $folderName): string {
|
||||||
$map = vv_dk_read_conf_map($raw, $myId);
|
$map = vv_dk_read_conf_map($raw, $myId);
|
||||||
$map = array_filter($map, fn($v) => $v !== $folderName);
|
$map = array_filter($map, fn($v) => $v !== $folderName);
|
||||||
$updated = vv_dk_write_conf_map($raw, $myId, $map);
|
return vv_dk_write_conf_map($raw, $myId, $map);
|
||||||
vv_write_conf_raw($currentHost . '.conf', $updated);
|
}, [], ["{$myId}_DOCKER_FOLDER_MAP"]);
|
||||||
return ['ok' => true];
|
return ['ok' => true];
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -421,22 +424,24 @@ function vv_dk_sync_json_to_conf(): array {
|
|||||||
$data = vv_dk_read_json();
|
$data = vv_dk_read_json();
|
||||||
$currentHost = vv_detect_host();
|
$currentHost = vv_detect_host();
|
||||||
$myId = strtoupper($currentHost);
|
$myId = strtoupper($currentHost);
|
||||||
$raw = vv_read_conf_raw($currentHost . '.conf');
|
_vv_dk_sync_conf_from_json($data, $currentHost, $myId);
|
||||||
_vv_dk_sync_conf_from_json($data, $raw, $currentHost, $myId);
|
|
||||||
return ['ok' => true];
|
return ['ok' => true];
|
||||||
}
|
}
|
||||||
|
|
||||||
// Internal: rebuild conf map from current json state and write it
|
// Internal: rebuild conf map from current json state and write it.
|
||||||
function _vv_dk_sync_conf_from_json(array $data, string $raw = '', string $host = '', string $id = ''): void {
|
// The contents to splice into are read inside vv_conf_edit()'s lock. The caller used to be able
|
||||||
|
// to hand in a copy it had already read; that was only ever an optimisation, and passing a stale
|
||||||
|
// copy would have written the rest of the conf back as it looked before the lock was taken.
|
||||||
|
function _vv_dk_sync_conf_from_json(array $data, string $host = '', string $id = ''): void {
|
||||||
if (!$host) $host = vv_detect_host();
|
if (!$host) $host = vv_detect_host();
|
||||||
if (!$id) $id = strtoupper($host);
|
if (!$id) $id = strtoupper($host);
|
||||||
if (!$raw) $raw = vv_read_conf_raw($host . '.conf');
|
|
||||||
|
|
||||||
$map = [];
|
$map = [];
|
||||||
foreach ($data as $f) {
|
foreach ($data as $f) {
|
||||||
$name = $f['name'] ?? '';
|
$name = $f['name'] ?? '';
|
||||||
foreach ($f['containers'] ?? [] as $c) $map[$c] = $name;
|
foreach ($f['containers'] ?? [] as $c) $map[$c] = $name;
|
||||||
}
|
}
|
||||||
$updated = vv_dk_write_conf_map($raw, $id, $map);
|
|
||||||
vv_write_conf_raw($host . '.conf', $updated);
|
vv_conf_edit($host . '.conf', fn(string $raw): string => vv_dk_write_conf_map($raw, $id, $map),
|
||||||
|
[], ["{$id}_DOCKER_FOLDER_MAP"]);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -598,50 +598,56 @@ function vv_conf_flag_value(string $name): bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Write a boolean flag value to master.conf.
|
// Write a boolean flag value to master.conf.
|
||||||
|
// Goes through vv_conf_edit() for the lock, the pre-write backup, the syntax check and the audit
|
||||||
|
// line. This used to call vv_write_conf_raw() directly, which gave it tmp+rename atomicity and
|
||||||
|
// nothing else — no backup, and no check that the file still sourced afterwards.
|
||||||
function vv_conf_flag_set(string $name, bool $value): bool {
|
function vv_conf_flag_set(string $name, bool $value): bool {
|
||||||
$confPath = CONF_DIR . '/master.conf';
|
if (!vv_conf_key_valid($name)) return false;
|
||||||
$content = file_get_contents($confPath);
|
$val = $value ? 'true' : 'false';
|
||||||
if ($content === false) return false;
|
|
||||||
$val = $value ? 'true' : 'false';
|
return vv_conf_edit('master.conf', function (string $content) use ($name, $val): ?string {
|
||||||
$new = preg_replace(
|
$new = preg_replace(
|
||||||
'/^(\s*' . preg_quote($name, '/') . '\s*=\s*)(true|false)(\s*(?:#.*)?)$/m',
|
'/^(\s*' . preg_quote($name, '/') . '\s*=\s*)(true|false)(\s*(?:#.*)?)$/m',
|
||||||
'${1}' . $val . '${3}',
|
'${1}' . $val . '${3}',
|
||||||
$content, -1, $count
|
$content, -1, $count
|
||||||
);
|
);
|
||||||
if (!$count) return false;
|
// A name that matches no true/false line is a caller error, not an already-correct
|
||||||
// tmp+rename — every script sources master.conf, so a truncated write here is a
|
// state — unlike the membership toggle below, where absence genuinely means nothing
|
||||||
// system-wide outage, not a lost toggle.
|
// to do. Returning null keeps the write from happening and logs reason=no-match.
|
||||||
return vv_write_conf_raw('master.conf', $new);
|
return $count ? $new : null;
|
||||||
|
}, [$name => $val]);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Comment or uncomment a script's line in the first master.conf array that contains it.
|
// Comment or uncomment a script's line in the first master.conf array that contains it.
|
||||||
|
// Goes through vv_conf_edit() for the lock, the pre-write backup, the syntax check and the audit
|
||||||
|
// line — see vv_conf_flag_set() above for what that replaced. There is no key to verify here,
|
||||||
|
// so the audit subject is the script id and a clean source is the whole assertion.
|
||||||
function vv_conf_toggle_script(string $rel, bool $enable): bool {
|
function vv_conf_toggle_script(string $rel, bool $enable): bool {
|
||||||
$confPath = CONF_DIR . '/master.conf';
|
return vv_conf_edit('master.conf', function (string $content) use ($rel, $enable): ?string {
|
||||||
$lines = file($confPath, FILE_KEEP_BLANK_LINES);
|
$lines = preg_split('/(?<=\n)/', $content) ?: [];
|
||||||
if (!$lines) return false;
|
$changed = false;
|
||||||
$changed = false;
|
$inArray = false;
|
||||||
$inArray = false;
|
$relEsc = preg_quote($rel, '/');
|
||||||
$relEsc = preg_quote($rel, '/');
|
foreach ($lines as &$line) {
|
||||||
foreach ($lines as &$line) {
|
if (preg_match('/^\s*[A-Z_]+_SCRIPTS\s*=\s*\(/', $line)) $inArray = true;
|
||||||
if (preg_match('/^\s*[A-Z_]+_SCRIPTS\s*=\s*\(/', $line)) $inArray = true;
|
if ($inArray && preg_match('/^\s*\)\s*(?:#.*)?$/', $line) && !str_contains($line, '(')) $inArray = false;
|
||||||
if ($inArray && preg_match('/^\s*\)\s*(?:#.*)?$/', $line) && !str_contains($line, '(')) $inArray = false;
|
if (!$inArray) continue;
|
||||||
if (!$inArray) continue;
|
if (!preg_match('/^\s*(?:#\s*)?"' . $relEsc . '(?:\s[^"]*)?"/', $line)) continue;
|
||||||
if (!preg_match('/^\s*(?:#\s*)?"' . $relEsc . '(?:\s[^"]*)?"/', $line)) continue;
|
$isCommented = (bool)preg_match('/^\s*#/', $line);
|
||||||
$isCommented = (bool)preg_match('/^\s*#/', $line);
|
if ($enable && $isCommented) {
|
||||||
if ($enable && $isCommented) {
|
$line = preg_replace('/^(\s*)#\s*("' . $relEsc . ')/', '$1$2', $line);
|
||||||
$line = preg_replace('/^(\s*)#\s*("' . $relEsc . ')/', '$1$2', $line);
|
$changed = true;
|
||||||
$changed = true;
|
} elseif (!$enable && !$isCommented) {
|
||||||
} elseif (!$enable && !$isCommented) {
|
$line = preg_replace('/^(\s*)("' . $relEsc . ')/', '$1# $2', $line);
|
||||||
$line = preg_replace('/^(\s*)("' . $relEsc . ')/', '$1# $2', $line);
|
$changed = true;
|
||||||
$changed = true;
|
}
|
||||||
|
break;
|
||||||
}
|
}
|
||||||
break;
|
unset($line);
|
||||||
}
|
// A script in no array has nothing to toggle and the conf already reads the way the
|
||||||
unset($line);
|
// caller asked. Returning the content unchanged reports success without a write.
|
||||||
if (!$changed) return true;
|
return $changed ? implode('', $lines) : $content;
|
||||||
// tmp+rename — every script sources master.conf, so a truncated write here is a
|
}, [], [$rel]);
|
||||||
// system-wide outage, not a lost toggle.
|
|
||||||
return vv_write_conf_raw('master.conf', implode('', $lines));
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Parse an orchestrator script to find which child scripts it calls.
|
// Parse an orchestrator script to find which child scripts it calls.
|
||||||
|
|||||||
@@ -144,6 +144,14 @@
|
|||||||
# clearing path requires seeing the container running again, which never happens once
|
# clearing path requires seeing the container running again, which never happens once
|
||||||
# it is uninstalled. Without pruning, one removed container makes this host report
|
# it is uninstalled. Without pruning, one removed container makes this host report
|
||||||
# unhealthy permanently. Reserved daemon_* keys in WATCHDOG_STATE_FILE are never pruned.
|
# unhealthy permanently. Reserved daemon_* keys in WATCHDOG_STATE_FILE are never pruned.
|
||||||
|
#
|
||||||
|
# WATCHDOG_STATE_FILE is keyed two ways and the prune has to know it. The container check
|
||||||
|
# stores a bare name; the HTTP, API, CPU and docker checks store container + suffix. The
|
||||||
|
# existence test is always against the base container — inspecting the composite key means
|
||||||
|
# asking docker about "Emby_http", which fails while Emby is running perfectly, and deleting
|
||||||
|
# the live counter every cycle. That made RESP_FAIL_LIMIT and CPU_FAIL_LIMIT unreachable,
|
||||||
|
# since a counter reset each cycle never reaches 2. Only the four known suffixes are
|
||||||
|
# stripped: a general split on the last underscore would break PostgreSQL_Immich.
|
||||||
# Pruning runs under --dry-run as well: the documented contract for that flag is that
|
# Pruning runs under --dry-run as well: the documented contract for that flag is that
|
||||||
# nothing gets restarted, and dropping a record of a container that no longer exists is
|
# nothing gets restarted, and dropping a record of a container that no longer exists is
|
||||||
# housekeeping, not an action. It is idempotent — repeated runs converge.
|
# housekeeping, not an action. It is idempotent — repeated runs converge.
|
||||||
@@ -732,15 +740,38 @@ CYCLE_START=$(date +%s)
|
|||||||
if [[ -s "$WATCHDOG_STATE_FILE" ]]; then
|
if [[ -s "$WATCHDOG_STATE_FILE" ]]; then
|
||||||
mapfile -t _prune_snapshot < "$WATCHDOG_STATE_FILE"
|
mapfile -t _prune_snapshot < "$WATCHDOG_STATE_FILE"
|
||||||
for _prune_line in "${_prune_snapshot[@]}"; do
|
for _prune_line in "${_prune_snapshot[@]}"; do
|
||||||
_prune_container="${_prune_line%%:*}"
|
_prune_key="${_prune_line%%:*}"
|
||||||
[[ -z "$_prune_container" ]] && continue
|
[[ -z "$_prune_key" ]] && continue
|
||||||
[[ "$_prune_container" == daemon_* ]] && continue # reserved keys, not containers
|
[[ "$_prune_key" == daemon_* ]] && continue # reserved keys, not containers
|
||||||
|
|
||||||
|
# Keys come in two shapes: the bare container name for the container check, and
|
||||||
|
# container + check suffix for the others. `docker inspect Emby_http` fails for a
|
||||||
|
# perfectly healthy Emby, so inspecting the key itself deleted every live per-check
|
||||||
|
# counter on every cycle — before the checks below read them. RESP_FAIL_LIMIT and
|
||||||
|
# CPU_FAIL_LIMIT of 2 were therefore unreachable: a counter wiped each cycle can
|
||||||
|
# only ever reach 1. Latent rather than harmful only because no container here had
|
||||||
|
# yet failed one of those checks.
|
||||||
|
#
|
||||||
|
# Strip only the four known suffixes, never on the last underscore: PostgreSQL_Immich
|
||||||
|
# is a real container, and splitting it would prune a name that does exist.
|
||||||
|
#
|
||||||
|
# Known limit: a container literally named something_http or something_docker would
|
||||||
|
# have its own counter judged by whether "something" exists. No container here ends
|
||||||
|
# in one of the four, and the alternative — inspecting the key, then the base — pays
|
||||||
|
# a second docker call for every composite key on every cycle to cover a name nobody
|
||||||
|
# uses. Worth revisiting only alongside replacing these per-key inspects with one
|
||||||
|
# `docker ps -a` membership test.
|
||||||
|
_prune_container="$_prune_key"
|
||||||
|
case "$_prune_key" in
|
||||||
|
*_http|*_api|*_cpu|*_docker) _prune_container="${_prune_key%_*}" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
if ! timeout "$DOCKER_TIMEOUT" docker inspect "$_prune_container" &>/dev/null; then
|
if ! timeout "$DOCKER_TIMEOUT" docker inspect "$_prune_container" &>/dev/null; then
|
||||||
sed -i "/^${_prune_container}:/d" "$WATCHDOG_STATE_FILE" 2>/dev/null
|
sed -i "/^${_prune_key}:/d" "$WATCHDOG_STATE_FILE" 2>/dev/null
|
||||||
warn "$_prune_container no longer exists — removed from $(basename "$WATCHDOG_STATE_FILE")"
|
warn "$_prune_container no longer exists — removed $_prune_key from $(basename "$WATCHDOG_STATE_FILE")"
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
unset _prune_snapshot _prune_line _prune_container
|
unset _prune_snapshot _prune_line _prune_key _prune_container
|
||||||
fi
|
fi
|
||||||
|
|
||||||
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
|
||||||
|
|||||||
@@ -2008,7 +2008,17 @@ run_orch_child() {
|
|||||||
return 0
|
return 0
|
||||||
else
|
else
|
||||||
_ec=$?
|
_ec=$?
|
||||||
error "$label — failed (exit $_ec, $(format_duration $(( $(date +%s) - _start ))))"
|
# Match the severity ladder run_job.sh already records against: exit 1 is a warning,
|
||||||
|
# exit 2+ is an error. The status in the run record was always derived that way, so a
|
||||||
|
# child exiting 1 produced a run marked "warn" whose log was full of ❌ ERROR lines —
|
||||||
|
# and the log is the louder of the two. A job that skipped work it was told to skip read
|
||||||
|
# exactly like a job that broke.
|
||||||
|
#
|
||||||
|
# The label still lands in JOB_FAIL either way. Whether the child succeeded is a separate
|
||||||
|
# question from how loudly to report it, and the orchestrator's own exit code depends on
|
||||||
|
# the first one.
|
||||||
|
local _msg="$label — failed (exit $_ec, $(format_duration $(( $(date +%s) - _start ))))"
|
||||||
|
if (( _ec == 1 )); then warn "$_msg"; else error "$_msg"; fi
|
||||||
JOB_FAIL+=("$label")
|
JOB_FAIL+=("$label")
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
|
|||||||
@@ -19,6 +19,8 @@ data/
|
|||||||
├── cache/ persistent backups of the tmpfs caches — and only those
|
├── cache/ persistent backups of the tmpfs caches — and only those
|
||||||
│ ├── arr/ *_tracked_cache.json, restored into tmpfs on demand
|
│ ├── arr/ *_tracked_cache.json, restored into tmpfs on demand
|
||||||
│ └── conf/ partner host*.conf snapshot (0700 — holds credentials)
|
│ └── conf/ partner host*.conf snapshot (0700 — holds credentials)
|
||||||
|
├── Backups/ point-in-time copies kept so a bad write can be undone
|
||||||
|
│ └── Confs/ pre-write copies of this host's own confs (0700 — holds credentials)
|
||||||
└── logs/ retained log output
|
└── logs/ retained log output
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -31,6 +33,8 @@ variable is what a storage-mode migration rewrites.
|
|||||||
| `state/` | `STATE_DIR` | ALL state files must use this. No `/tmp`, no repo root. |
|
| `state/` | `STATE_DIR` | ALL state files must use this. No `/tmp`, no repo root. |
|
||||||
| `ai/` | `AI_DATA_DIR` | |
|
| `ai/` | `AI_DATA_DIR` | |
|
||||||
| `cache/` | `CACHE_BACKUP_DIR` | `ARR_CACHE_BACKUP_DIR`, `PERSISTENT_CONF_CACHE` sit under it |
|
| `cache/` | `CACHE_BACKUP_DIR` | `ARR_CACHE_BACKUP_DIR`, `PERSISTENT_CONF_CACHE` sit under it |
|
||||||
|
| `Backups/` | `BACKUP_DIR` | parent only; each kind of backup gets a subdirectory |
|
||||||
|
| `Backups/Confs/` | `CONF_BACKUP_DIR` | retained per `CONF_BACKUP_RETAIN`; **not** a cache — see below |
|
||||||
| `logs/` | `LOG_ARCHIVE_DIR` | live logging still goes to `LOG_DIR` (`/var/log/varaverk`) |
|
| `logs/` | `LOG_ARCHIVE_DIR` | live logging still goes to `LOG_DIR` (`/var/log/varaverk`) |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -74,6 +78,11 @@ Ask what happens if it is deleted.
|
|||||||
called — `lidarr_art_miss_cache.tsv` has "cache" in its name and lives in `db/` for exactly that
|
called — `lidarr_art_miss_cache.tsv` has "cache" in its name and lives in `db/` for exactly that
|
||||||
reason.
|
reason.
|
||||||
|
|
||||||
|
`Backups/` is the same trap from the other side: it has "backup" in its name but backs up nothing
|
||||||
|
that exists elsewhere. The confs are gitignored, so a pre-write copy under `Backups/Confs/` is the
|
||||||
|
only prior version of that file anywhere. Deleting it loses something permanently, which is why it
|
||||||
|
is a root of its own and not a subdirectory of `cache/`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## History
|
## History
|
||||||
|
|||||||
Reference in New Issue
Block a user