Make fallback coverage something that can actually happen, and say so on the page
fallback.sh starts covered containers with docker start and never creates them, so a coverage list the partner has never been sent is a promise nothing can keep — all twelve were missing. Adds the push and remove paths, a readiness card that checks rather than infers, and the fallback state the assistant needs to answer for it.
This commit is contained in:
@@ -1483,6 +1483,114 @@ function vv_ai_bug_report(array $b): string {
|
||||
// Strictly read-only, and there is no counterpart that changes any of it. Knowing a container is
|
||||
// down is what lets an explanation be about this machine instead of about Unraid in general;
|
||||
// restarting it is a decision that belongs to a person looking at the screen.
|
||||
// ══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
// Fallback readiness, for the assistant on the Fallback tab.
|
||||
//
|
||||
// Fallback differs from every other subsystem here in one way that shapes this whole function: it
|
||||
// is DORMANT until it isn't. A watchdog leaves strikes and restarts to reason about; fallback
|
||||
// leaves nothing at all until a real outage, so "looks fine" and "would work" are unrelated. On
|
||||
// 2026-08-23 the coverage card showed 12 containers configured and every one of them was absent
|
||||
// from the partner — a failover would have started nothing, and no surface said so.
|
||||
//
|
||||
// So this reports what would ACTUALLY happen, not what is configured to happen, and it is explicit
|
||||
// about the difference between the two.
|
||||
//
|
||||
// Never blocks on the network. Partner presence costs an SSH round trip per container, which is far
|
||||
// too slow for a question already waiting on a model, so it is read from the cache
|
||||
// coverage_deploy.sh --status writes and reported WITH ITS AGE. A stale answer stated as stale is
|
||||
// useful; a stale answer stated as current is the failure this whole feature exists to prevent.
|
||||
// ══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
function vv_ai_fallback_state(): string {
|
||||
$me = vv_detect_host();
|
||||
if ($me === '') return '';
|
||||
// vv_detect_host() returns the LOWERCASE slug (host1); the conf keys are uppercase
|
||||
// (FALLBACK_HOST1_TIER1). Building the key from the slug as-is silently matched nothing and
|
||||
// reported "NOTHING is covered" on a host with twelve covered containers — a confidently
|
||||
// wrong answer, which is the one outcome this block must never produce.
|
||||
$ME = strtoupper($me);
|
||||
|
||||
$conf = vv_read_conf_raw('master.conf');
|
||||
if ($conf === '') return '';
|
||||
|
||||
$s = "FALLBACK READINESS (read-only — you cannot change any of it, and you must never tell the "
|
||||
. "operator a failover will work unless the evidence below says so)\n";
|
||||
|
||||
// ── current state ───────────────────────────────────────────────────────────────────────
|
||||
$stateFile = STATE_DIR . '/fallback_state.db';
|
||||
$state = 'unknown'; $since = '';
|
||||
if (is_readable($stateFile)) {
|
||||
$raw = (string) @file_get_contents($stateFile);
|
||||
if (preg_match('/^state=(\S+)/m', $raw, $m)) $state = $m[1];
|
||||
if (preg_match('/^fallback_start=(\d+)/m', $raw, $m) && (int)$m[1] > 0) {
|
||||
$since = ' since ' . date('Y-m-d H:i', (int) $m[1]);
|
||||
}
|
||||
$age = time() - (int) @filemtime($stateFile);
|
||||
// The steady NORMAL path writes nothing, so an old mtime is not staleness — it is quiet.
|
||||
$s .= "- state: $state$since (state file last written "
|
||||
. ($age < 3600 ? round($age / 60) . ' minutes' : round($age / 86400) . ' days') . " ago; "
|
||||
. "the NORMAL path writes nothing, so an old file means nothing has changed)\n";
|
||||
} else {
|
||||
// No file is not the same as not running: fallback.sh writes only on a transition.
|
||||
$live = function_exists('vv_fb_proc') ? (vv_fb_proc('fallback')['running'] ?? false) : false;
|
||||
$s .= $live
|
||||
? "- state: NORMAL (inferred — the daemon is running and has never recorded a transition, "
|
||||
. "so it has written no state file; this is health, not ignorance)\n"
|
||||
: "- state: no state file AND no running daemon — fallback is not operating on this host\n";
|
||||
}
|
||||
|
||||
foreach (['FALLBACK_ENABLED', 'FALLBACK_RSYNC_ENABLED'] as $k) {
|
||||
if (preg_match('/^\s*' . $k . '\s*=\s*"?(\w+)"?/m', $conf, $m)) {
|
||||
$s .= "- $k: {$m[1]}"
|
||||
. ($k === 'FALLBACK_RSYNC_ENABLED' && $m[1] !== 'true'
|
||||
? " <- handback writeback is OFF: anything the partner writes while covering "
|
||||
. "for this host never comes home\n" : "\n");
|
||||
}
|
||||
}
|
||||
|
||||
// ── coverage, tier by tier, with the real delays ─────────────────────────────────────────
|
||||
$hostConf = vv_read_conf_raw($me . '.conf');
|
||||
$covered = [];
|
||||
for ($t = 1; $t <= 4; $t++) {
|
||||
$names = vv_parse_conf_list($hostConf, "FALLBACK_{$ME}_TIER{$t}");
|
||||
if (!$names) continue;
|
||||
$delay = '';
|
||||
if ($t > 1 && preg_match('/^\s*' . $ME . '_TIER' . $t . '_DELAY\s*=\s*"?(\d+)/m', $hostConf, $m)) {
|
||||
$delay = " after {$m[1]} minutes";
|
||||
}
|
||||
$s .= "- tier $t" . ($t === 1 ? ' (immediate)' : $delay) . ': ' . implode(', ', $names) . "\n";
|
||||
foreach ($names as $n) $covered[] = $n;
|
||||
}
|
||||
if (!$covered) {
|
||||
$s .= "- coverage: NOTHING is covered — a failover would start no containers at all\n";
|
||||
return $s . "\n";
|
||||
}
|
||||
|
||||
// ── does the partner actually have them ─────────────────────────────────────────────────
|
||||
$cache = '/tmp/varaverk/api/fallback_presence.json';
|
||||
if (is_readable($cache)) {
|
||||
$j = json_decode((string) @file_get_contents($cache), true);
|
||||
$age = time() - (int) @filemtime($cache);
|
||||
$miss = (array) ($j['missing'] ?? []);
|
||||
$have = array_keys((array) ($j['present'] ?? []));
|
||||
$when = $age < 3600 ? round($age / 60) . ' minutes ago' : round($age / 3600) . ' hours ago';
|
||||
if ($miss) {
|
||||
$s .= "- ON THE PARTNER (checked $when): " . count($miss) . ' of ' . count($covered)
|
||||
. " covered container(s) DO NOT EXIST there: " . implode(', ', $miss) . "\n"
|
||||
. " fallback.sh starts a covered container with `docker start`; it never creates one, "
|
||||
. "so each of those would fail during a real outage. Push them from the Fallback "
|
||||
. "coverage card.\n";
|
||||
} else {
|
||||
$s .= "- ON THE PARTNER (checked $when): all " . count($have)
|
||||
. " covered container(s) exist there\n";
|
||||
}
|
||||
} else {
|
||||
$s .= "- ON THE PARTNER: not checked. Say so plainly — whether a failover would actually "
|
||||
. "start anything is UNKNOWN until the coverage card's presence check runs.\n";
|
||||
}
|
||||
|
||||
return $s . "\n";
|
||||
}
|
||||
|
||||
function vv_ai_system_state(): string {
|
||||
$p = '/tmp/varaverk/api/monitor.json';
|
||||
if (!is_readable($p)) return '';
|
||||
|
||||
Reference in New Issue
Block a user