Read the system log and the containers, not only Varaverk's own logs

Triage could see what Varaverk wrote about itself and what the arrs said about themselves, and
nothing else — so a disk throwing I/O errors, a filesystem going read-only or a PCIe link
retraining every two minutes was invisible to the thing whose job is noticing. Container state
was already watched; no line any container actually wrote ever was.

Container patterns match the environment rather than the application. Fifty containers are fifty
programs with no shared vocabulary for malfunctioning, but an exact shared one for a full disk or
a corrupt SQLite file, because those strings come from libc and SQLite rather than from the app.

Both halves are checked by Tools/ai_log_check.sh, which is two tests because the failure modes
are opposite: fixtures for recall on faults this host has never had, and a replay of its real
logs for precision — 74,519 syslog lines and 79,193 container lines, matching only the PCIe
errors it genuinely has.

Severity was being graded from a hand-picked three fields, so every one of these would have been
filed as a warning however bad it was, and notified as one.
This commit is contained in:
Gmer4Lfe
2026-08-09 22:16:04 -04:00
parent 91357a2d03
commit 426ca2e5c7
5 changed files with 601 additions and 2 deletions
+22
View File
@@ -1805,6 +1805,28 @@
# to be trusted first. This gates telling someone, which is the reason for having looked.
AI_REPAIR_NOTIFY_ENABLED=true
# ━━━ AI Repair: what it reads ━━━
# Beyond Varaverk's own logs and the arrs' health endpoints.
#
# The system log catches what Varaverk cannot see about itself — a disk throwing I/O errors, a
# filesystem going read-only, a PCIe link retraining every two minutes. Container restarts and
# OOM kills are deliberately left to docker_watchdog, which already notifies on them.
#
# Container logs catch the opposite blind spot: docker_watchdog watches whether a container is
# up and answering, which a container that has been unable to write to its database all day
# passes perfectly. Only environment faults are matched — disk full, read-only filesystem,
# corrupt database, fd limits, expired certificates — because those strings come from libc, the
# kernel and SQLite and mean the same thing in all fifty containers. Anything app-specific
# belongs in that app's own health endpoint.
#
# Both are bounded by time (since the last pass) and by a line cap, so a flood costs one pass.
# Tools/ai_log_check.sh replays this host's real logs against the patterns — run it after
# changing any of them.
AI_REPAIR_SYSLOG_ENABLED=true
AI_REPAIR_SYSLOG_MAX_LINES=4000
AI_REPAIR_CONTAINER_LOGS_ENABLED=true
AI_REPAIR_CONTAINER_LOG_LINES=400
# ━━━ AI Conf Write Access ━━━
# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
# never credentials, never a container name. An empty whitelist means no writes regardless of