FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make fallback coverage something that can actually happen, and say so on the page

Browse Source 
fallback.sh starts covered containers with docker start and never creates them, so a coverage list
the partner has never been sent is a promise nothing can keep — all twelve were missing. Adds the
push and remove paths, a readiness card that checks rather than infers, and the fallback state the
assistant needs to answer for it.
This commit is contained in:
Gmer4Lfe
2026-08-23 16:38:58 -04:00
parent d5cf3db2ec
commit 671e7ea5a4
7 changed files with 867 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Plugin/unraid/include/ai.php
+108
View File
@@ -1483,6 +1483,114 @@ function vv_ai_bug_report(array $b): string {
// Strictly read-only, and there is no counterpart that changes any of it. Knowing a container is
// down is what lets an explanation be about this machine instead of about Unraid in general;
// restarting it is a decision that belongs to a person looking at the screen.
// ══════════════════════════════════════════════════════════════════════════════════════════════
// Fallback readiness, for the assistant on the Fallback tab.
//
// Fallback differs from every other subsystem here in one way that shapes this whole function: it
// is DORMANT until it isn't. A watchdog leaves strikes and restarts to reason about; fallback
// leaves nothing at all until a real outage, so "looks fine" and "would work" are unrelated. On
// 2026-08-23 the coverage card showed 12 containers configured and every one of them was absent
// from the partner — a failover would have started nothing, and no surface said so.
//
// So this reports what would ACTUALLY happen, not what is configured to happen, and it is explicit
// about the difference between the two.
//
// Never blocks on the network. Partner presence costs an SSH round trip per container, which is far
// too slow for a question already waiting on a model, so it is read from the cache
// coverage_deploy.sh --status writes and reported WITH ITS AGE. A stale answer stated as stale is
// useful; a stale answer stated as current is the failure this whole feature exists to prevent.
// ══════════════════════════════════════════════════════════════════════════════════════════════
function vv_ai_fallback_state(): string {
$me = vv_detect_host();
if ($me === '') return '';
// vv_detect_host() returns the LOWERCASE slug (host1); the conf keys are uppercase
// (FALLBACK_HOST1_TIER1). Building the key from the slug as-is silently matched nothing and
// reported "NOTHING is covered" on a host with twelve covered containers — a confidently
// wrong answer, which is the one outcome this block must never produce.
$ME = strtoupper($me);
$conf = vv_read_conf_raw('master.conf');
if ($conf === '') return '';
$s = "FALLBACK READINESS (read-only — you cannot change any of it, and you must never tell the "
. "operator a failover will work unless the evidence below says so)\n";
// ── current state ───────────────────────────────────────────────────────────────────────
$stateFile = STATE_DIR . '/fallback_state.db';
$state = 'unknown'; $since = '';
if (is_readable($stateFile)) {
$raw = (string) @file_get_contents($stateFile);
if (preg_match('/^state=(\S+)/m', $raw, $m)) $state = $m[1];
if (preg_match('/^fallback_start=(\d+)/m', $raw, $m) && (int)$m[1] > 0) {
$since = ' since ' . date('Y-m-d H:i', (int) $m[1]);
}
$age = time() - (int) @filemtime($stateFile);
// The steady NORMAL path writes nothing, so an old mtime is not staleness — it is quiet.
$s .= "- state: $state$since (state file last written "
. ($age < 3600 ? round($age / 60) . ' minutes' : round($age / 86400) . ' days') . " ago; "
. "the NORMAL path writes nothing, so an old file means nothing has changed)\n";
} else {
// No file is not the same as not running: fallback.sh writes only on a transition.
$live = function_exists('vv_fb_proc') ? (vv_fb_proc('fallback')['running'] ?? false) : false;
$s .= $live
? "- state: NORMAL (inferred — the daemon is running and has never recorded a transition, "
. "so it has written no state file; this is health, not ignorance)\n"
: "- state: no state file AND no running daemon — fallback is not operating on this host\n";
}
foreach (['FALLBACK_ENABLED', 'FALLBACK_RSYNC_ENABLED'] as $k) {
if (preg_match('/^\s*' . $k . '\s*=\s*"?(\w+)"?/m', $conf, $m)) {
$s .= "- $k: {$m[1]}"
. ($k === 'FALLBACK_RSYNC_ENABLED' && $m[1] !== 'true'
? " <- handback writeback is OFF: anything the partner writes while covering "
. "for this host never comes home\n" : "\n");
}
}
// ── coverage, tier by tier, with the real delays ─────────────────────────────────────────
$hostConf = vv_read_conf_raw($me . '.conf');
$covered = [];
for ($t = 1; $t <= 4; $t++) {
$names = vv_parse_conf_list($hostConf, "FALLBACK_{$ME}_TIER{$t}");
if (!$names) continue;
$delay = '';
if ($t > 1 && preg_match('/^\s*' . $ME . '_TIER' . $t . '_DELAY\s*=\s*"?(\d+)/m', $hostConf, $m)) {
$delay = " after {$m[1]} minutes";
}
$s .= "- tier $t" . ($t === 1 ? ' (immediate)' : $delay) . ': ' . implode(', ', $names) . "\n";
foreach ($names as $n) $covered[] = $n;
}
if (!$covered) {
$s .= "- coverage: NOTHING is covered — a failover would start no containers at all\n";
return $s . "\n";
}
// ── does the partner actually have them ─────────────────────────────────────────────────
$cache = '/tmp/varaverk/api/fallback_presence.json';
if (is_readable($cache)) {
$j = json_decode((string) @file_get_contents($cache), true);
$age = time() - (int) @filemtime($cache);
$miss = (array) ($j['missing'] ?? []);
$have = array_keys((array) ($j['present'] ?? []));
$when = $age < 3600 ? round($age / 60) . ' minutes ago' : round($age / 3600) . ' hours ago';
if ($miss) {
$s .= "- ON THE PARTNER (checked $when): " . count($miss) . ' of ' . count($covered)
. " covered container(s) DO NOT EXIST there: " . implode(', ', $miss) . "\n"
. " fallback.sh starts a covered container with `docker start`; it never creates one, "
. "so each of those would fail during a real outage. Push them from the Fallback "
. "coverage card.\n";
} else {
$s .= "- ON THE PARTNER (checked $when): all " . count($have)
. " covered container(s) exist there\n";
}
} else {
$s .= "- ON THE PARTNER: not checked. Say so plainly — whether a failover would actually "
. "start anything is UNKNOWN until the coverage card's presence check runs.\n";
}
return $s . "\n";
}
function vv_ai_system_state(): string {
$p = '/tmp/varaverk/api/monitor.json';
if (!is_readable($p)) return '';
Plugin/unraid/include/ai_profiles.php
+3 -2
View File
@@ -116,7 +116,7 @@ const VV_AI_PROFILES_DEF = [
// system_state is read-only and shared with repair. Both need to know a container is down
// or a pool is full to explain anything about this machine rather than about Unraid in
// general; neither gets a way to act on it, and only repair can change a setting.
'caps' => ['retrieve', 'health', 'system_state', 'run_evidence', 'scoped_log',
'caps' => ['retrieve', 'health', 'system_state', 'fallback_state', 'run_evidence', 'scoped_log',
'incidents', 'conf_lookup', 'file_bugs'],
],
// The only profile that may change a setting, and the only one not offered as a button.
@@ -136,7 +136,7 @@ const VV_AI_PROFILES_DEF = [
'hint' => 'Works through a finding with you, and can apply a fix you approve.',
'turns' => 3,
'ui' => false,
'caps' => ['retrieve', 'health', 'system_state', 'run_evidence', 'scoped_log', 'incidents',
'caps' => ['retrieve', 'health', 'system_state', 'fallback_state', 'run_evidence', 'scoped_log', 'incidents',
'conf_lookup', 'conf_write', 'probe', 'file_findings', 'phrasebook', 'past_fixes'],
],
];
@@ -148,6 +148,7 @@ const VV_AI_CAP_MEANING = [
'kind_filter' => 'the retrieval kind filter the page exposes',
'health' => 'live health sweep measured at question time — the AI subsystem only',
'system_state' => 'read-only view of the machine: hardware, containers, pools, array, UPS',
'fallback_state' => 'whether a failover would actually work: state, tiers, and whether the partner really has the covered containers',
'run_evidence' => 'run record and log tail for a script named in the question',
'scoped_log' => 'log tail for whatever the operator currently has open',
'incidents' => 'operator-written history of what previously went wrong with this thing',
Plugin/unraid/include/fallback.php
+26 -1
View File
@@ -73,6 +73,8 @@ function vv_fb_parse_state(string $text): array {
'tier4_started' => false,
'partnership_suspended' => false,
'partner_lost_at' => 0,
// Set only when the verdict came from the daemon rather than the file — see vv_fb_all().
'inferred' => false,
];
foreach (explode("\n", $text) as $line) {
$line = trim($line);
@@ -313,6 +315,29 @@ function vv_fb_all(): array {
: vv_fb_remote_dryrun_state($ip, $mySshKey, (int)$proc['pid']);
}
// ── A live daemon that has simply never transitioned ────────────────────────────────
// fallback.sh writes its state file ONLY on a transition; the steady NORMAL path writes
// nothing at all. So a node that has run cleanly since it was built has no file, and
// reading the file alone reports it as UNKNOWN — the same verdict given to a node whose
// daemon is dead. Those are opposite conditions and they were rendered identically.
//
// Observed on HOST2 2026-08-23: daemon live (pid 2208657, valid lock), tailscale and ssh
// both fine, no state file, card said UNKNOWN.
//
// The rule this page is built on — never claim healthy for a host you cannot verify — is
// kept: this host CAN be verified, just not from the file that was being consulted. The
// daemon holding a live lock is the evidence. reach['state_file'] still reports false,
// because there genuinely is no file; 'inferred' says where the verdict came from instead.
// Captured before the inference below rewrites it: reach[state_file] must keep answering
// "was there a file", not "do we have a verdict". Deriving it after inference made the
// card claim a state file existed on a host that has none.
$hadStateFile = ($state['state'] ?? 'UNKNOWN') !== 'UNKNOWN';
if (($state['state'] ?? 'UNKNOWN') === 'UNKNOWN' && ($proc['running'] ?? null) === true) {
$state['state'] = 'NORMAL';
$state['inferred'] = true;
}
// How fresh the state actually is. The daemon rewrites its file every check interval, so
// an age far past that interval means it is wedged even while the process still exists.
$stateAge = null;
@@ -326,7 +351,7 @@ function vv_fb_all(): array {
'tailscale' => $isMe ? true : ($ts['online'] === true),
'ip' => $isMe ? null : $ip,
'ssh' => $isMe ? true : ($running !== [] || ($proc['running'] !== null)),
'state_file' => ($state['state'] ?? 'UNKNOWN') !== 'UNKNOWN',
'state_file' => $hadStateFile,
];
$nodes[] = [