Give the Watchdog tab an assistant scoped to it, and Why? on every strike

The page states what struck but never why, and the log that would say is the
orchestrator's, not one per watchdog. Why? opens troubleshoot against it —
read-only, because the page's contract is that it changes nothing.
This commit is contained in:
Gmer4Lfe
2026-08-14 10:09:04 -04:00
parent c6b56d580e
commit 8808cbfc04
+168 -20
View File
@@ -16,16 +16,25 @@
// Strictly read-only. There is no control on this page that clears a strike, restarts a // Strictly read-only. There is no control on this page that clears a strike, restarts a
// container, or resets a counter — those belong to the watchdog that owns the state. // container, or resets a counter — those belong to the watchdog that owns the state.
// //
// That holds for the assistant too, and is why the Why? buttons open troubleshoot rather than
// repair. troubleshoot may read the machine and the orchestrator's log; it holds no conf_write
// and no way to act. A page whose stated contract is "changes nothing" must not carry the one
// profile that can — see include/ai_profiles.php for why that grant is deliberately narrow.
//
// A node that fails to report renders as unavailable rather than healthy. Absence of data // A node that fails to report renders as unavailable rather than healthy. Absence of data
// is never drawn as an all-clear. // is never drawn as an all-clear.
// //
// RENDERS // RENDERS
// Per-node cards: resource / docker / system / storage / network watchdog state, // Per-node cards: resource / docker / system / storage / network watchdog state,
// strike pills, restart history, reboot and OOM counts // strike pills, restart history, reboot and OOM counts
// Assistant card, scoped to the Watchdog tab
// Watchdog conf sections, via the shared settings renderer
// //
// DEPENDS ON // DEPENDS ON
// api/watchdog.php polled every 30s → include/watchdog.php // api/watchdog.php polled every 30s → include/watchdog.php
// include/ai_chat.php → the shared chat component
require_once dirname(__DIR__) . '/include/confui.php'; require_once dirname(__DIR__) . '/include/confui.php';
require_once dirname(__DIR__) . '/include/ai_chat.php';
?> ?>
<style> <style>
.vv-wd-card { background:#161616;border:1px solid #2a2a2a;border-radius:6px;padding:10px;min-width:0; } .vv-wd-card { background:#161616;border:1px solid #2a2a2a;border-radius:6px;padding:10px;min-width:0; }
@@ -80,6 +89,43 @@ require_once dirname(__DIR__) . '/include/confui.php';
</div> </div>
<?php if (vv_ai_ui_on()): ?>
<div class="vv-card" id="vv-wd-ai-card" style="margin-top:12px;">
<?php
// All three, in this order, before the markup. vv_ai_chat_markup() emits only the boxes — the
// factory that brings them to life, the profile registry the picker draws from, and the store
// that keeps a thread across a reload are separate and none of them is implied by the others.
// Omitting them renders a chat that looks complete and dies on the first click with VvAiChat
// undefined, which is how the Scheduler's panel behaved before it was given the same three.
// Each is guarded internally, so calling them here costs nothing on a page that already has.
vv_ai_profiles_script();
vv_ai_chat_store_script();
vv_ai_chat_assets();
// Reads the page rather than sitting beside it. The scope is sent with every question, and the
// worker turns it into "they have Watchdog open — read bare references against it", so "why did
// it restart that" resolves here instead of asking which of forty containers is meant.
//
// Starts on the Varaverk assistant, not General Chat as the Monitor card does. The two pages ask
// different things: the dashboard is where an idle question gets typed, this is a page you only
// open because something struck. Answers about strikes and thresholds should come from this
// installation's own docs and state, with sources, rather than from the model's general
// impression of what a watchdog usually does.
//
// No 'height' triple — this card is in normal page flow, not a fixed panel, so the component's
// own defaults apply and the expand control still works against them. The Scheduler is the only
// placement whose size is a share of something and therefore has to be told.
vv_ai_chat_markup('vv-wd-ai', [
'profile' => 'varaverk',
'compact' => true,
'title' => 'Assistant',
'scopeLabel' => 'Watchdog',
'empty' => 'Ask about a strike, a restart, or a threshold on this page. '
. 'Why? on any strike asks against the orchestrator\'s log.',
'placeholder' => 'Ask about what is on this page…',
]); ?>
</div>
<?php endif; ?>
<?php <?php
// The conf sections this page is about, drawn by the shared renderer. They were reachable // The conf sections this page is about, drawn by the shared renderer. They were reachable
// only from the Settings tab's catch-all, which is a long way to go for a setting named // only from the Settings tab's catch-all, which is a long way to go for a setting named
@@ -91,6 +137,40 @@ vv_conf_ui_card('vv-cf-watchdog', 'watchdog', 'Watchdog settings');
const GB = 1073741824; const GB = 1073741824;
// The assistant instance, once the card is on the page; null whenever AI is off. Every Why?
// button checks it, and none of them render without it — a control that opens a panel that does
// not exist is worse than no control.
let vvWdChat = null;
// What the chat says the operator is looking at. Starts as the tab itself and moves to the
// orchestrator's log when a Why? is pressed, because that is the only place any of this is
// written down: the watchdogs log through watchdog_orchestrator.sh and have no log files of
// their own — checked against /var/log/varaverk, where Watchdogs/ does not exist.
let vvWdScope = 'Watchdog';
const WD_LOG = 'Orchestrators/watchdog_orchestrator';
// Rendered as data attributes and read back by one delegated listener rather than written into an
// onclick. Container names and log paths reach these buttons from conf and from the filesystem, and
// a name containing a quote closes the attribute early and takes the rest of the handler with it.
// vvEscAttr is the one that escapes quotes; vvEscHtml deliberately does not.
function _why(label, q) {
if (!vvWdChat) return '';
return `<button class="vv-btn-sm vv-why-btn" title="Ask the assistant about this"
data-label="${vvEscAttr(label)}" data-q="${vvEscAttr(q)}">Why?</button>`;
}
function vvWdWhy(label, q) {
if (!vvWdChat || vvWdChat.busy()) return;
// Troubleshoot, never repair. This page changes nothing, and repair is the one profile that can
// — see the safeguard note in the header.
vvWdScope = WD_LOG;
vvWdChat.retarget('troubleshoot', label, 'now looking at ' + label);
const input = document.getElementById('vv-wd-ai-input');
if (input) input.value = q;
vvWdChat.send();
const card = document.getElementById('vv-wd-ai-card');
if (card) card.scrollIntoView({ behavior: 'smooth', block: 'nearest' });
}
function _fmtBytes(b) { function _fmtBytes(b) {
if (!b) return '0'; if (!b) return '0';
if (b >= GB) return (b / GB).toFixed(1) + 'G'; if (b >= GB) return (b / GB).toFixed(1) + 'G';
@@ -122,8 +202,11 @@ function _bar(pct, col) {
return `<div class="vv-wd-bar"><div class="vv-wd-bar-fill" style="width:${Math.min(pct,100)}%;background:${col}"></div></div>`; return `<div class="vv-wd-bar"><div class="vv-wd-bar-fill" style="width:${Math.min(pct,100)}%;background:${col}"></div></div>`;
} }
// Escapes its own label, so every caller passes plain text and none of them has to remember. The
// labels here are container names, log paths and DDNS domains — all of them values this page reads
// rather than writes, which is exactly the category that must not be trusted into markup.
function _pill(label, cls) { function _pill(label, cls) {
return `<span class="vv-wd-pill ${cls}">${label}</span>`; return `<span class="vv-wd-pill ${cls}">${vvEscHtml(label)}</span>`;
} }
function _levelLabel(level) { function _levelLabel(level) {
@@ -149,7 +232,10 @@ function _pressureCard(node) {
return `<div class="vv-wd-card vv-wd-pressure" style="grid-column:1/-1"> return `<div class="vv-wd-card vv-wd-pressure" style="grid-column:1/-1">
<div style="display:flex;align-items:center;gap:10px;margin-bottom:8px;"> <div style="display:flex;align-items:center;gap:10px;margin-bottom:8px;">
<span class="vv-wd-badge ${cls}">PRESSURE ${label}</span> <span class="vv-wd-badge ${cls}">PRESSURE ${label}</span>
<span style="font-size:11px;color:#7a5020;">${node.id} (${node.hostname})</span> <span style="font-size:11px;color:#7a5020;">${vvEscHtml(node.id)} (${vvEscHtml(node.hostname)})</span>
${_why(node.id + ' memory pressure',
'The resource watchdog is at ' + label + ' pressure on ' + node.id
+ '. What drove it there, and what did it pause or stop?')}
${st.mem_shutdown ? `<span class="vv-wd-badge hard" style="margin-left:auto;">MEM SHUTDOWN ACTIVE</span>` : ''} ${st.mem_shutdown ? `<span class="vv-wd-badge hard" style="margin-left:auto;">MEM SHUTDOWN ACTIVE</span>` : ''}
</div> </div>
${paused.length ? `<div style="margin-bottom:4px;"><span class="vv-wd-lbl">Paused:</span> <span class="vv-wd-pill-row" style="display:inline-flex;">${pausedHtml}</span></div>` : ''} ${paused.length ? `<div style="margin-bottom:4px;"><span class="vv-wd-lbl">Paused:</span> <span class="vv-wd-pill-row" style="display:inline-flex;">${pausedHtml}</span></div>` : ''}
@@ -164,8 +250,8 @@ function _systemCard(node, cfg) {
return `<div class="vv-wd-card"> return `<div class="vv-wd-card">
<div class="vv-wd-node-h"> <div class="vv-wd-node-h">
<span class="vv-wd-dot" style="background:#444"></span> <span class="vv-wd-dot" style="background:#444"></span>
<span class="vv-wd-node-id">${node.id}</span> <span class="vv-wd-node-id">${vvEscHtml(node.id)}</span>
<span style="font-size:11px;color:#3a3a3a;">${node.hostname}</span> <span style="font-size:11px;color:#3a3a3a;">${vvEscHtml(node.hostname)}</span>
<span class="vv-wd-badge" style="margin-left:auto;background:#1a1a1a;color:#444">UNREACHABLE</span> <span class="vv-wd-badge" style="margin-left:auto;background:#1a1a1a;color:#444">UNREACHABLE</span>
</div> </div>
</div>`; </div>`;
@@ -197,8 +283,8 @@ function _systemCard(node, cfg) {
return `<div class="vv-wd-card"> return `<div class="vv-wd-card">
<div class="vv-wd-node-h"> <div class="vv-wd-node-h">
<span class="vv-wd-dot" style="background:${dotCol}"></span> <span class="vv-wd-dot" style="background:${dotCol}"></span>
<span class="vv-wd-node-id">${node.id}</span> <span class="vv-wd-node-id">${vvEscHtml(node.id)}</span>
<span style="font-size:11px;color:#3a3a3a;">${node.hostname}</span> <span style="font-size:11px;color:#3a3a3a;">${vvEscHtml(node.hostname)}</span>
<span class="vv-wd-badge ${apiOnly ? '' : _levelCls(level)}" <span class="vv-wd-badge ${apiOnly ? '' : _levelCls(level)}"
style="margin-left:auto;${apiOnly ? 'background:#0d1f2a;color:#4a9eff;' : ''}" style="margin-left:auto;${apiOnly ? 'background:#0d1f2a;color:#4a9eff;' : ''}"
>${apiOnly ? 'API ONLY' : _levelLabel(level)}</span> >${apiOnly ? 'API ONLY' : _levelLabel(level)}</span>
@@ -246,7 +332,12 @@ function _dockerCard(node, cfg) {
strikesHtml += `<div class="vv-wd-strike-row"><span class="vv-wd-strike-name">daemon</span><span class="vv-wd-strike-cnt">${st.daemon_strikes}</span></div>`; strikesHtml += `<div class="vv-wd-strike-row"><span class="vv-wd-strike-name">daemon</span><span class="vv-wd-strike-cnt">${st.daemon_strikes}</span></div>`;
} }
for (const [name, cnt] of strikes) { for (const [name, cnt] of strikes) {
strikesHtml += `<div class="vv-wd-strike-row"><span class="vv-wd-strike-name">${name}</span><span class="vv-wd-strike-cnt">${cnt} / ${cfg.cpu_fail_lim}</span></div>`; strikesHtml += `<div class="vv-wd-strike-row">
<span class="vv-wd-strike-name">${vvEscHtml(name)}</span>
<span class="vv-wd-strike-cnt">${cnt} / ${cfg.cpu_fail_lim}</span>
${_why(name + ' strikes', 'The docker watchdog has ' + cnt + ' strike(s) against '
+ name + '. What is failing its check?')}
</div>`;
} }
} }
@@ -263,7 +354,12 @@ function _dockerCard(node, cfg) {
restartHtml = '<div style="color:#333;font-size:11px;">none (24h)</div>'; restartHtml = '<div style="color:#333;font-size:11px;">none (24h)</div>';
} else { } else {
restartHtml = rcEntries.map(([n, c]) => restartHtml = rcEntries.map(([n, c]) =>
`<div class="vv-wd-row"><span class="vv-wd-lbl">${n}</span><span class="vv-wd-val" style="color:${c >= cfg.restart_limit ? '#ef5350' : '#ffb74d'}">${c}×</span></div>` `<div class="vv-wd-row">
<span class="vv-wd-lbl">${vvEscHtml(n)}</span>
<span class="vv-wd-val" style="color:${c >= cfg.restart_limit ? '#ef5350' : '#ffb74d'}">${c}×</span>
${_why(n + ' restarts', n + ' has been restarted ' + c
+ ' time(s) in the last 24 hours. Why does it keep needing a restart?')}
</div>`
).join(''); ).join('');
} }
@@ -298,13 +394,23 @@ function _stabilityCard(node, cfg) {
sysHtml = '<div style="color:#333;font-size:11px;">0 active strikes</div>'; sysHtml = '<div style="color:#333;font-size:11px;">0 active strikes</div>';
} else { } else {
sysHtml = sysStr.map(([k, v]) => sysHtml = sysStr.map(([k, v]) =>
`<div class="vv-wd-strike-row"><span class="vv-wd-strike-name" style="font-size:10px;">${k.replace(/_/g,' ')}</span><span class="vv-wd-strike-cnt">${v}</span></div>` `<div class="vv-wd-strike-row">
<span class="vv-wd-strike-name" style="font-size:10px;">${vvEscHtml(k.replace(/_/g,' '))}</span>
<span class="vv-wd-strike-cnt">${v}</span>
${_why(k.replace(/_/g,' ') + ' strikes', 'The stability watchdog has ' + v
+ ' strike(s) for ' + k.replace(/_/g,' ') + '. What is it detecting?')}
</div>`
).join(''); ).join('');
} }
// The one entry on this page that is always worth explaining, so the button is on the heading
// rather than each row — an unattended reboot is a single question, not one per occurrence.
const rebootHtml = reboots.length === 0 const rebootHtml = reboots.length === 0
? '<div style="color:#333;font-size:11px;">none (12h)</div>' ? '<div style="color:#333;font-size:11px;">none (12h)</div>'
: reboots.map(ts => `<div class="vv-wd-reboot-ts">${_relTime(ts)}</div>`).join(''); : reboots.map(ts => `<div class="vv-wd-reboot-ts">${_relTime(ts)}</div>`).join('')
+ `<div style="margin-top:5px;">${_why(node.id + ' reboots',
'The stability watchdog rebooted ' + node.id + ' ' + reboots.length
+ ' time(s) in the last ' + cfg.reboot_window + ' hours. What forced it?')}</div>`;
return `<div class="vv-wd-card"> return `<div class="vv-wd-card">
<div class="vv-wd-sec">Stability</div> <div class="vv-wd-sec">Stability</div>
@@ -334,7 +440,7 @@ function _configCard(node) {
? '<div style="color:#333;font-size:11px;">none</div>' ? '<div style="color:#333;font-size:11px;">none</div>'
: mon.map(([name, mb]) => { : mon.map(([name, mb]) => {
const gb = (mb / 1024).toFixed(0); const gb = (mb / 1024).toFixed(0);
return `<div class="vv-wd-ctr-row"><span class="vv-wd-ctr-name">${name}</span><span class="vv-wd-ctr-lim">${gb} GB</span></div>`; return `<div class="vv-wd-ctr-row"><span class="vv-wd-ctr-name">${vvEscHtml(name)}</span><span class="vv-wd-ctr-lim">${gb} GB</span></div>`;
}).join(''); }).join('');
const reqHtml = req.length === 0 const reqHtml = req.length === 0
@@ -355,8 +461,8 @@ function _configCard(node) {
return `<div class="vv-wd-card" style="grid-column:span 4;"> return `<div class="vv-wd-card" style="grid-column:span 4;">
<div style="display:flex;gap:6px;align-items:center;margin-bottom:8px;"> <div style="display:flex;gap:6px;align-items:center;margin-bottom:8px;">
<span class="vv-wd-node-id">${node.id}</span> <span class="vv-wd-node-id">${vvEscHtml(node.id)}</span>
<span style="font-size:11px;color:#3a3a3a;">${node.hostname}</span> <span style="font-size:11px;color:#3a3a3a;">${vvEscHtml(node.hostname)}</span>
</div> </div>
<div style="display:grid;grid-template-columns:1fr 1fr;gap:10px;"> <div style="display:grid;grid-template-columns:1fr 1fr;gap:10px;">
<div> <div>
@@ -406,7 +512,7 @@ function _storageCard(node, cfg) {
// Suppress ceilings configured for this host // Suppress ceilings configured for this host
const sizes = Object.entries(nodeCfg.appdata_sizes || {}); const sizes = Object.entries(nodeCfg.appdata_sizes || {});
const sizesHtml = sizes.length const sizesHtml = sizes.length
? sizes.map(([c, mb]) => _pill(`${c} <${Math.round(mb/1024)}GB`, '')).join('') ? sizes.map(([c, mb]) => _pill(c + ' <' + Math.round(mb/1024) + 'GB', '')).join('')
: ''; : '';
let growthHtml = ''; let growthHtml = '';
@@ -415,8 +521,11 @@ function _storageCard(node, cfg) {
} else { } else {
growthHtml = growthStr.map(([name, cnt]) => growthHtml = growthStr.map(([name, cnt]) =>
`<div class="vv-wd-strike-row"> `<div class="vv-wd-strike-row">
<span class="vv-wd-strike-name">${name}</span> <span class="vv-wd-strike-name">${vvEscHtml(name)}</span>
<span class="vv-wd-strike-cnt">${cnt} / ${cfg.stor_strike_lim}</span> <span class="vv-wd-strike-cnt">${cnt} / ${cfg.stor_strike_lim}</span>
${_why(name + ' appdata growth', name + ' has ' + cnt
+ ' growth strike(s) — its appdata is growing more than '
+ cfg.growth_gb + 'GB per cycle. What is filling up?')}
</div>` </div>`
).join(''); ).join('');
} }
@@ -428,8 +537,11 @@ function _storageCard(node, cfg) {
logHtml = logStr.map(([key, cnt]) => { logHtml = logStr.map(([key, cnt]) => {
const display = key.length > 36 ? '…' + key.slice(-36) : key; const display = key.length > 36 ? '…' + key.slice(-36) : key;
return `<div class="vv-wd-strike-row"> return `<div class="vv-wd-strike-row">
<span class="vv-wd-strike-name" style="font-size:10px;" title="${key}">${display}</span> <span class="vv-wd-strike-name" style="font-size:10px;" title="${vvEscAttr(key)}">${vvEscHtml(display)}</span>
<span class="vv-wd-strike-cnt">${cnt} / ${cfg.stor_strike_lim}</span> <span class="vv-wd-strike-cnt">${cnt} / ${cfg.stor_strike_lim}</span>
${_why('oversized log ' + key, key + ' has ' + cnt
+ ' log-size strike(s) against a ' + cfg.log_max_gb
+ 'GB ceiling. What is writing to it?')}
</div>`; </div>`;
}).join(''); }).join('');
} }
@@ -471,13 +583,16 @@ function _networkCard(node, cfg) {
const npmUrl = nodeCfg.npm_url || ''; const npmUrl = nodeCfg.npm_url || '';
const ddnsHtml = ddnsDomain const ddnsHtml = ddnsDomain
? `${_row('DDNS domain', `<span style="color:#888;">${ddnsDomain}</span>`)} ? `${_row('DDNS domain', `<span style="color:#888;">${vvEscHtml(ddnsDomain)}</span>`)}
${ddnsCtr ? _row('DDNS container', `<span style="color:#888;">${ddnsCtr}</span>`) : ''}` ${ddnsCtr ? _row('DDNS container', `<span style="color:#888;">${vvEscHtml(ddnsCtr)}</span>`) : ''}`
: _row('DDNS', '<span style="color:#444;">not configured for this host</span>'); : _row('DDNS', '<span style="color:#444;">not configured for this host</span>');
const npmHtml = npmUrl const npmHtml = npmUrl
? `${_row('NPM URL', `<span style="color:#888;font-size:10px;">${npmUrl}</span>`)} ? `${_row('NPM URL', `<span style="color:#888;font-size:10px;">${vvEscHtml(npmUrl)}</span>`)}
${_row('NPM strikes', `<span class="vv-wd-pill ${npmCls}" style="font-size:10px;">${npmStr} / ${cfg.npm_strike_lim}</span>`)}` ${_row('NPM strikes', `<span class="vv-wd-pill ${npmCls}" style="font-size:10px;">${npmStr} / ${cfg.npm_strike_lim}</span>`
+ (npmStr > 0 ? ' ' + _why('NPM strikes on ' + node.id,
'The network watchdog has ' + npmStr + ' strike(s) against NPM on '
+ node.id + '. What check is failing?') : ''))}`
: _row('NPM check', '<span style="color:#444;">not configured for this host</span>'); : _row('NPM check', '<span style="color:#444;">not configured for this host</span>');
return `<div class="vv-wd-card"> return `<div class="vv-wd-card">
@@ -536,6 +651,39 @@ function vvWdLoad() {
}); });
} }
// One listener on the grid rather than a handler per button. The grid is replaced wholesale every
// 30 seconds, so anything bound to a button inside it would be thrown away on the next poll — and
// re-binding after each render is the version of this that quietly stops working the day a card
// gains a button and nobody updates the binder.
document.getElementById('vv-wd-grid').addEventListener('click', ev => {
const b = ev.target.closest('.vv-why-btn');
if (!b) return;
vvWdWhy(b.dataset.label || 'this watchdog', b.dataset.q || 'What does this mean?');
});
// Before the first load, not after. _why() renders nothing while vvWdChat is null, so a render
// that beat the instance into existence would draw the whole page without a single Why? button
// and not correct itself until the next poll thirty seconds later.
if (document.getElementById('vv-wd-ai-chat')) {
vvWdChat = VvAiChat({
prefix: 'vv-wd-ai',
profile: 'varaverk',
scopeLabel: 'Watchdog',
// Read at send time, not captured: a Why? moves the scope to the orchestrator's log, and the
// question is sent from that same click.
scope: () => vvWdScope,
// Same reason the Monitor card pins its own: without it the card resumes whatever thread was
// last touched anywhere, so opening this tab could land it mid-conversation from the Scheduler
// under a profile this page never offers.
resumeProfile: 'varaverk',
// Reasoning is worth watching for a diagnosis and noise for a definition — same rule the
// Scheduler applies.
think: p => p === 'troubleshoot',
empty: 'Ask about a strike, a restart, or a threshold on this page. '
+ 'Why? on any strike asks against the orchestrator\'s log.',
});
}
vvWdLoad(); vvWdLoad();
setInterval(vvWdLoad, 30000); setInterval(vvWdLoad, 30000);