FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Give the watchdog counters a voice in the repair sweep

Browse Source 
A fourth candidate source beside the arrs, the system log and container logs:
skip-listed containers, strikes past their limit, repeated restarts, unattended
reboots and sustained pressure become findings. The kind is deliberately not
conf-bound, so it cannot autofix by construction rather than by a switch — and
WATCHDOG_SCAN_IGNORE suppresses it, so a knowingly broken container stays quiet.
First consumer AI_ASSIST_WATCHDOG has ever had.
This commit is contained in:
Gmer4Lfe
2026-08-14 12:01:18 -04:00
parent cbcfbfaaad
commit 2f8b7276a4
2 changed files with 164 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Plugin/unraid/include/ai_repair.php
+163 -1
View File
@@ -11,6 +11,11 @@
// because the repair may need something only a human can supply, and that conversation has to
// survive the page being closed.
//
// Four candidate sources: the arrs' own health, the system log, container logs, and the
// watchdog counters. The first three read text; the fourth reads state the watchdogs already
// maintain, and is the only one whose findings can never be repaired from here — see
// vv_ai_watchdog_findings() for why that is structural rather than a setting.
//
// WHY THIS IS NOT ai_bugs
// Same storage shape, different lifecycle, and the difference is the whole reason for a second
// store. A bug is open until Varaverk's code changes; nothing on this host can close it. A
@@ -85,6 +90,7 @@ const VV_AI_FINDING_KINDS = [
'arr_health' => 'an arr is reporting a problem about itself',
'system_fault' => 'the kernel reported a hardware or filesystem fault about this machine',
'container_fault' => 'a container is logging a fault about its own environment',
'watchdog_strike' => 'a watchdog has counted something far enough to be worth a record',
];
// Which kinds are a statement about Varaverk's configuration, and which are a statement about
@@ -223,6 +229,18 @@ function vv_ai_repair_autofix_enabled(): bool {
return strtolower(trim((string)(vv_conf_vars()['AI_REPAIR_AUTOFIX_ENABLED'] ?? 'false'))) === 'true';
}
// The first consumer AI_ASSIST_WATCHDOG has ever had. It shipped with the rest of the AI_ASSIST_
// tier flags and nothing read it, which meant a switch that looked like it did something.
//
// Separate from AI_REPAIR_ENABLED rather than folded into it because the two answer different
// questions. Repair asks "may this read job logs and offer to change conf". This asks "may this
// have an opinion about what the watchdogs are counting" — a different subject, a different
// appetite for noise, and one you may well want off while repair stays on.
function vv_ai_assist_watchdog_enabled(): bool {
if (!vv_ai_repair_enabled()) return false;
return strtolower(trim((string)(vv_conf_vars()['AI_ASSIST_WATCHDOG'] ?? 'false'))) === 'true';
}
function vv_ai_findings_dir(): string {
$d = AI_DATA_DIR . '/ai_findings';
if (!is_dir($d)) @mkdir($d, 0755, true);
@@ -808,6 +826,145 @@ function vv_ai_container_findings(int $since, ?array $logsByContainer = null): a
return $found;
}
// ── What the watchdogs have counted ──────────────────────────────────────────────────────────
// The fourth candidate source. The other three read logs; this reads the counters the watchdogs
// keep, and turns the ones that have gone far enough into a record with a name on it.
//
// It reads vv_wd_all() rather than the state files, for the reason the Watchdog tab does: the
// meaning of a strike lives in include/watchdog.php, and a second parser here would be a second
// opinion about it. It also gets the thresholds in the same payload, so a count is always judged
// against the limit that was in force when it was read rather than one fetched separately.
//
// Deliberately not conf-bound. 'watchdog_strike' is absent from VV_AI_CONF_BOUND_KINDS, so these
// findings carry no conf key, so vv_ai_finding_may_autofix() can never return true for one. That
// is a structural guarantee rather than a switch: there is no setting that makes this able to
// change anything, and none can be added without also deciding which key it would write.
//
// What it does NOT do is explain. The text below is derived, not generated — no model is asked
// anything on this path. The reading is a click away in the troubleshoot profile, already scoped
// to the orchestrator's log by the Why? buttons on the Watchdog tab. Detection has to be worth
// believing before it is worth narrating, and a wrong cause stated fluently is worse than none.
const VV_AI_WD_PRESSURE_LEVELS = [2 => 'medium', 3 => 'hard'];
function vv_ai_watchdog_findings(?array $payload = null): array {
if ($payload === null && !vv_ai_assist_watchdog_enabled()) return [];
$wd = $payload ?? vv_cache_read('watchdog', 900);
if ($wd === null) {
require_once __DIR__ . '/watchdog.php';
$wd = vv_wd_all();
}
$cfg = $wd['cfg'] ?? [];
$found = [];
foreach ($wd['nodes'] ?? [] as $node) {
// This host only. A partner's counters are its own to file findings about, and the
// operator reading them here could not act on them from this side anyway.
if (empty($node['is_me'])) continue;
$st = $node['states'] ?? null;
if (!$st) continue;
$id = (string)($node['id'] ?? 'this host');
// The operator has already said they know about these. Suppressing here rather than at
// write time keeps them out of the pin store entirely — 7DaysToDie is the worked example:
// knowingly broken, deliberately ignored, and a finding about it every fifteen minutes
// would be exactly the crying-wolf this is supposed to avoid.
$ignore = array_flip((array)($node['config']['ignore'] ?? []));
$add = function (string $subject, string $ref, string $level, string $observed,
string $evidence) use (&$found) {
$found[] = [
'kind' => 'watchdog_strike',
'subject' => $subject,
'pin' => 'wd:' . $ref . '|' . $subject,
'ref' => $ref,
'conf_key' => '',
'conf_file' => '',
'sys_level' => $level,
'observed' => $observed,
'evidence' => $evidence,
// Every watchdog logs through the orchestrator; none keeps a log of its own.
'source_log' => 'Orchestrators/watchdog_orchestrator',
'state' => 'needs_operator',
];
};
// 1. Skip list — the terminal state. The watchdog tried its restart limit and stopped,
// which means nothing further is coming automatically. Always worth a record.
foreach ((array)($st['skiplist'] ?? []) as $name) {
if (isset($ignore[$name])) continue;
$add($name, 'skiplist', 'error', 'on the skip list',
sprintf('%s is on the docker watchdog skip list on %s. It was restarted up to '
. 'the limit of %d and did not stay up, so the watchdog has stopped trying '
. 'and will not restart it again without intervention.',
$name, $id, (int)($cfg['restart_limit'] ?? 3)));
}
// 2. Container check strikes at or past their limit. Below the limit the watchdog is
// still working the problem and a finding would be premature.
$lim = max(1, (int)($cfg['cpu_fail_lim'] ?? 2));
foreach ((array)($st['ctr_strikes'] ?? []) as $key => $count) {
$name = preg_replace('/_(cpu|http|api|docker)$/', '', (string)$key);
if (isset($ignore[$name]) || (int)$count < $lim) continue;
$check = preg_match('/_(cpu|http|api|docker)$/', (string)$key, $m) ? $m[1] : 'health';
$add($name, 'strike:' . $check, 'warn', $count . ' / ' . $lim,
sprintf('The docker watchdog has counted %d consecutive %s failures for %s on %s, '
. 'against a limit of %d.', (int)$count, $check, $name, $id, $lim));
}
// 3. Repeated restarts inside the 24h window. Distinct from a strike: the container is
// coming back each time, which is why nothing has escalated, and is also why this is
// easy to never notice.
$rc = [];
foreach ((array)($st['restarts'] ?? []) as $r) {
$n = (string)($r['name'] ?? '');
if ($n !== '') $rc[$n] = ($rc[$n] ?? 0) + 1;
}
$rlim = max(2, (int)($cfg['restart_limit'] ?? 3));
foreach ($rc as $name => $count) {
if (isset($ignore[$name]) || $count < $rlim) continue;
$add($name, 'restarts', 'warn', $count . ' in 24h',
sprintf('%s has been restarted %d times in the last 24 hours on %s. Each one '
. 'succeeded, so no strike escalated — but a container that needs restarting '
. 'this often is failing at something between restarts.', $name, $count, $id));
}
// 4. Any unattended reboot. Not thresholded: the machine restarting itself is worth
// saying once every time it happens, however normal the count is.
$reboots = (array)($st['reboots'] ?? []);
if ($reboots) {
$add($id, 'reboots', 'error', count($reboots) . ' / ' . (int)($cfg['reboot_limit'] ?? 3),
sprintf('The stability watchdog has rebooted %s %d time%s in the last %d hours, '
. 'against a limit of %d before it stops trying.',
$id, count($reboots), count($reboots) === 1 ? '' : 's',
(int)($cfg['reboot_window'] ?? 12), (int)($cfg['reboot_limit'] ?? 3)));
}
// 5. Sustained memory pressure. Level 1 is soft and self-clears constantly; from level 2
// the watchdog has acted on containers, and that is a thing that happened.
$lvl = (int)($st['rw_level'] ?? 0);
if (isset(VV_AI_WD_PRESSURE_LEVELS[$lvl])) {
$paused = array_values(array_filter((array)($st['rw_paused'] ?? [])));
$stopped = array_values(array_filter((array)($st['rw_stopped'] ?? [])));
$acted = array_merge(
$paused ? ['paused ' . implode(', ', $paused)] : [],
$stopped ? ['stopped ' . implode(', ', $stopped)] : []
);
$add($id, 'pressure', $lvl >= 3 ? 'error' : 'warn',
VV_AI_WD_PRESSURE_LEVELS[$lvl] . ' pressure',
sprintf('The resource watchdog is at %s pressure on %s%s.%s',
VV_AI_WD_PRESSURE_LEVELS[$lvl], $id,
$acted ? ' and has ' . implode(' and ', $acted) : '',
!empty($st['mem_shutdown']) ? ' The memory shutdown flag is set, so '
. 'docker_watchdog is deferring container restarts.' : ''));
}
}
return $found;
}
// ── Reaching the operator ────────────────────────────────────────────────────────────────────
// A finding nobody is told about is a finding nobody has. The card on the AI tab shows them, but
// only to someone who opens the tab, and the point of this subsystem is that it works while
@@ -994,16 +1151,21 @@ function vv_ai_repair_sweep(bool $dryRun = false): array {
$sum = ['ok' => true, 'runs' => count($runs), 'findings' => 0, 'fixed' => 0,
'needs_operator' => 0, 'resolved' => 0, 'quiet' => 0, 'details' => []];
// Candidates from three sources. Log triage is bounded to runs that finished since the last
// Candidates from four sources. Log triage is bounded to runs that finished since the last
// pass; the arrs are asked every time, because their health is a current state rather than
// something that appeared in a log once. Asking costs three local HTTP calls.
//
// The system log is bounded the same way the run logs are — everything since the marker —
// so a fault that has been shouting for an hour is counted once per pass rather than once
// per line, and stops producing candidates the moment it stops being logged.
//
// The watchdog counters are asked every time for the same reason the arrs are: a strike count
// is a current state, not something that appeared in a log once. It is also the cheapest of
// the four — one cached file read — and gated separately, on AI_ASSIST_WATCHDOG.
$candidates = vv_ai_arr_health_findings();
foreach (vv_ai_syslog_findings($since) as $c) $candidates[] = $c;
foreach (vv_ai_container_findings($since) as $c) $candidates[] = $c;
foreach (vv_ai_watchdog_findings() as $c) $candidates[] = $c;
foreach ($runs as $run) {
$lines = vv_ai_run_log_lines($run['log'], $run['start']);