Read the system log and the containers, not only Varaverk's own logs
Triage could see what Varaverk wrote about itself and what the arrs said about themselves, and nothing else — so a disk throwing I/O errors, a filesystem going read-only or a PCIe link retraining every two minutes was invisible to the thing whose job is noticing. Container state was already watched; no line any container actually wrote ever was. Container patterns match the environment rather than the application. Fifty containers are fifty programs with no shared vocabulary for malfunctioning, but an exact shared one for a full disk or a corrupt SQLite file, because those strings come from libc and SQLite rather than from the app. Both halves are checked by Tools/ai_log_check.sh, which is two tests because the failure modes are opposite: fixtures for recall on faults this host has never had, and a replay of its real logs for precision — 74,519 syslog lines and 79,193 container lines, matching only the PCIe errors it genuinely has. Severity was being graded from a hand-picked three fields, so every one of these would have been filed as a warning however bad it was, and notified as one.
This commit is contained in:
@@ -1805,6 +1805,28 @@
|
||||
# to be trusted first. This gates telling someone, which is the reason for having looked.
|
||||
AI_REPAIR_NOTIFY_ENABLED=true
|
||||
|
||||
# ━━━ AI Repair: what it reads ━━━
|
||||
# Beyond Varaverk's own logs and the arrs' health endpoints.
|
||||
#
|
||||
# The system log catches what Varaverk cannot see about itself — a disk throwing I/O errors, a
|
||||
# filesystem going read-only, a PCIe link retraining every two minutes. Container restarts and
|
||||
# OOM kills are deliberately left to docker_watchdog, which already notifies on them.
|
||||
#
|
||||
# Container logs catch the opposite blind spot: docker_watchdog watches whether a container is
|
||||
# up and answering, which a container that has been unable to write to its database all day
|
||||
# passes perfectly. Only environment faults are matched — disk full, read-only filesystem,
|
||||
# corrupt database, fd limits, expired certificates — because those strings come from libc, the
|
||||
# kernel and SQLite and mean the same thing in all fifty containers. Anything app-specific
|
||||
# belongs in that app's own health endpoint.
|
||||
#
|
||||
# Both are bounded by time (since the last pass) and by a line cap, so a flood costs one pass.
|
||||
# Tools/ai_log_check.sh replays this host's real logs against the patterns — run it after
|
||||
# changing any of them.
|
||||
AI_REPAIR_SYSLOG_ENABLED=true
|
||||
AI_REPAIR_SYSLOG_MAX_LINES=4000
|
||||
AI_REPAIR_CONTAINER_LOGS_ENABLED=true
|
||||
AI_REPAIR_CONTAINER_LOG_LINES=400
|
||||
|
||||
# ━━━ AI Conf Write Access ━━━
|
||||
# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
|
||||
# never credentials, never a container name. An empty whitelist means no writes regardless of
|
||||
|
||||
Reference in New Issue
Block a user