Gmer4Lfe
426ca2e5c7
Triage could see what Varaverk wrote about itself and what the arrs said about themselves, and nothing else — so a disk throwing I/O errors, a filesystem going read-only or a PCIe link retraining every two minutes was invisible to the thing whose job is noticing. Container state was already watched; no line any container actually wrote ever was. Container patterns match the environment rather than the application. Fifty containers are fifty programs with no shared vocabulary for malfunctioning, but an exact shared one for a full disk or a corrupt SQLite file, because those strings come from libc and SQLite rather than from the app. Both halves are checked by Tools/ai_log_check.sh, which is two tests because the failure modes are opposite: fixtures for recall on faults this host has never had, and a replay of its real logs for precision — 74,519 syslog lines and 79,193 container lines, matching only the PCIe errors it genuinely has. Severity was being graded from a hand-picked three fields, so every one of these would have been filed as a warning however bad it was, and notified as one.
101 lines
8.7 KiB
Plaintext
101 lines
8.7 KiB
Plaintext
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
# Log lines the repair sweep must recognise, one case per line, checked by ai_log_check.sh.
|
|
#
|
|
# WHY THIS FILE EXISTS
|
|
# The triage patterns for Varaverk's own logs were written against its emit sites — the format
|
|
# strings are in this repo and can be read. The kernel's are not, and this host is healthy:
|
|
# 74,519 lines of its real syslog contain PCIe AER errors and nothing else. So the patterns for
|
|
# a failing disk or a corrupt filesystem cannot be verified against anything this machine has
|
|
# said, and a pattern nobody has ever seen match is a pattern that does not work.
|
|
#
|
|
# These lines are what those faults look like. Precision is checked separately and against
|
|
# reality — ai_syslog_check.sh replays the host's whole syslog history and fails if anything
|
|
# fires that should not. This file is the other half: recall, on faults the host has not had.
|
|
#
|
|
# THE RULE FOR ADDING ONE
|
|
# Paste the real line, from this machine or from a kernel that emitted it. Do not compose one
|
|
# from what the pattern already matches — that tests the regex against itself and passes
|
|
# forever while the real format drifts away underneath it.
|
|
#
|
|
# FORMAT
|
|
# source | subject | level | line
|
|
#
|
|
# source sys = /var/log/syslog | ctr = a container log line
|
|
#
|
|
# subject what the finding must be about, after extraction
|
|
# level warn | error
|
|
# line the syslog line, verbatim, including its timestamp and host prefix
|
|
#
|
|
# A line beginning with ! must match NOTHING. Those are the near misses — normal operation that
|
|
# reads like a fault — and they are the reason the pattern list is not simply /error/i.
|
|
# Their format is: ! source | line
|
|
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
|
|
# ── PCIe. Verified: this host emitted 427 of these across three syslogs. ──────────────────────
|
|
sys | 0000:03:00.0 | warn | Aug 9 21:46:07 unRAID-Gmer4Lfe kernel: pcieport 0000:00:01.1: AER: Multiple Correctable error message received from 0000:03:00.0
|
|
sys | 0000:02:02.0 | warn | Aug 7 11:02:41 unRAID-Gmer4Lfe kernel: pcieport 0000:00:03.1: AER: Correctable error message received from 0000:02:02.0
|
|
sys | 0000:0a:00.0 | error | Aug 9 03:14:02 tower kernel: pcieport 0000:00:1c.0: AER: Uncorrected (Non-Fatal) error received from 0000:0a:00.0
|
|
sys | 0000:0b:00.0 | error | Aug 9 03:14:03 tower kernel: pcieport 0000:00:1c.4: AER: Fatal error received from 0000:0b:00.0
|
|
|
|
# ── Block layer. The device is the subject; the ATA port above it is not actionable. ─────────
|
|
sys | sdo | error | Aug 9 04:21:09 tower kernel: blk_update_request: critical medium error, dev sdo, sector 1953525161 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
|
|
sys | sde | error | Aug 9 04:22:11 tower kernel: blk_update_request: I/O error, dev sde, sector 8 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
|
|
sys | sdc | error | Aug 9 04:23:00 tower kernel: Buffer I/O error on dev sdc, logical block 0, async page read
|
|
sys | sdo | error | Aug 9 04:24:55 tower kernel: sd 1:0:3:0: [sdo] tag#28 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE unrecovered read error
|
|
|
|
# ── Filesystems. Each names its own device inside the parentheses. ────────────────────────────
|
|
sys | md1 | error | Aug 9 05:00:01 tower kernel: XFS (md1): Metadata corruption detected at xfs_agf_verify+0x1f0/0x1f9 [xfs], xfs_agf block 0x1
|
|
sys | md2 | error | Aug 9 05:01:02 tower kernel: XFS (md2): Internal error xfs_trans_cancel at line 990 of file fs/xfs/xfs_trans.c
|
|
sys | sdb1 | error | Aug 9 05:02:03 tower kernel: BTRFS error (device sdb1): parent transid verify failed on 1298432 wanted 12 found 11
|
|
sys | sdd1 | error | Aug 9 05:03:04 tower kernel: EXT4-fs error (device sdd1): ext4_find_entry:1455: inode #2: comm ls: reading directory lblock 0
|
|
sys | filesystem | error | Aug 9 05:04:05 tower kernel: EXT4-fs (sdd1): Remounting filesystem read-only
|
|
|
|
# ── Memory and the kernel itself. ─────────────────────────────────────────────────────────────
|
|
sys | shfs | error | Aug 9 06:00:00 tower kernel: Out of memory: Killed process 9823 (shfs) total-vm:2451234kB, anon-rss:1923412kB
|
|
sys | mono | error | Aug 9 06:00:01 tower kernel: Out of memory: Killed process 1122 (mono) total-vm:900000kB, anon-rss:800000kB
|
|
sys | kernel | error | Aug 9 07:00:00 tower kernel: kernel BUG at fs/xfs/xfs_message.c:102!
|
|
sys | kernel | error | Aug 9 07:00:01 tower kernel: general protection fault: 0000 [#1] SMP NOPTI
|
|
|
|
# ── Near misses. Normal operation that reads like a fault; none of these may match. ───────────
|
|
! sys | Aug 9 04:40:01 unRAID-Gmer4Lfe kernel: md: recovery thread: exit status: 0
|
|
! sys | Aug 9 04:40:02 unRAID-Gmer4Lfe kernel: md: sync done. time=17452sec
|
|
! sys | Aug 9 04:40:03 unRAID-Gmer4Lfe kernel: md: import disk6: (sdi) TOSHIBA_MG07ACA12TEY_Z0N0ANF9SG size: 11718885324
|
|
! sys | Aug 9 12:00:00 unRAID-Gmer4Lfe shfs: /usr/sbin/zfs destroy 'cache/isos' 2>&1
|
|
! sys | Aug 9 12:00:01 unRAID-Gmer4Lfe emhttpd: action: Enable all features using 'zpool upgrade'.
|
|
! sys | Aug 9 12:00:02 unRAID-Gmer4Lfe root: Fix Common Problems: Error: Docker image file is getting full
|
|
! sys | Aug 9 12:00:03 unRAID-Gmer4Lfe kernel: XFS (md1): Mounting V5 Filesystem
|
|
! sys | Aug 9 12:00:04 unRAID-Gmer4Lfe kernel: XFS (md1): Ending clean mount
|
|
! sys | Aug 9 12:00:05 unRAID-Gmer4Lfe sshd[1234]: error: kex_exchange_identification: Connection closed by remote host
|
|
! sys | Aug 9 12:00:06 unRAID-Gmer4Lfe nginx: 2026/08/09 12:00:06 [error] 1#1: *1 open() failed
|
|
! sys | Aug 9 12:00:07 unRAID-Gmer4Lfe kernel: docker0: port 3(veth1a2b3c) entered disabled state
|
|
! sys | Aug 9 12:00:08 unRAID-Gmer4Lfe kernel: eth0: Link is Down
|
|
|
|
# ══ CONTAINER LOGS ═════════════════════════════════════════════════════════════════════════════
|
|
# Fifty containers run here and they are fifty applications with nothing in common. These patterns
|
|
# are about the environment underneath them — the strings come from libc, the kernel and SQLite,
|
|
# so they are identical in every one. Precision was checked against all fifty containers' real
|
|
# logs: 79,180 lines, zero matches, which is what a healthy machine should produce.
|
|
#
|
|
# The subject of a container finding is the container, so it is not asserted per line here — the
|
|
# name comes from which log the line was read from, not from the text. Subject is left as the
|
|
# container name the checker supplies.
|
|
|
|
ctr | fixture-container | error | 2026-08-09 12:00:00 ERROR sqlite3.DatabaseError: database disk image is malformed
|
|
ctr | fixture-container | error | [Errno 28] No space left on device: '/config/logs/sonarr.txt'
|
|
ctr | fixture-container | error | OSError: [Errno 30] Read-only file system: '/data/media'
|
|
ctr | fixture-container | error | sqlite3.OperationalError: database or disk is full
|
|
ctr | fixture-container | error | Error: EMFILE: too many open files, open '/config/db'
|
|
ctr | fixture-container | error | curl: (60) SSL certificate problem: certificate has expired
|
|
ctr | fixture-container | error | ssl.SSLCertVerificationError: certificate verify failed: unable to get local issuer
|
|
ctr | fixture-container | error | write /var/lib/data: disk quota exceeded
|
|
ctr | fixture-container | error | sqlite3.DatabaseError: file is not a database
|
|
|
|
# Near misses from real arr and media-server logs. None of these may match.
|
|
! ctr | 2026-08-09 12:00:00 WARN sqlite3.OperationalError: database is locked, retrying in 200ms
|
|
! ctr | 2026-08-09 12:00:01 INFO Import failed: file already exists in the destination
|
|
! ctr | 2026-08-09 12:00:02 ERROR Permission denied reading /downloads/incomplete/foo.part
|
|
! ctr | 2026-08-09 12:00:03 WARN Certificate expires in 21 days, renewal scheduled
|
|
! ctr | 2026-08-09 12:00:04 ERROR Connection refused connecting to 127.0.0.1:9117
|
|
! ctr | 2026-08-09 12:00:05 INFO Deleted 4 files to free space on device sda
|
|
! ctr | 2026-08-09 12:00:06 ERROR HTTP 500 while calling /api/v3/command
|