Gmer4Lfe
99b58c0c4f
A low uptime figure, a refused login and a certificate that stopped renewing all looked the same from the row: a number, with the reason split across NPM, an Authelia config and the directory. The why-check goes and looks — TCP to the forward target, HTTP through the proxy, a second handshake with verification off to tell a broken certificate from a broken service. Forward hosts are docker names that only resolve on NPM's network, so an unresolvable one is redirected to the container address and the substitution is reported; a check that could not be made must never read as a check that failed. The access simulator walks the rules the way Authelia does and shows the ones it stepped over, reading whichever instance the chosen host points at rather than the one conf names — there are two here. Cert triage counts runs rather than log lines and orders by rotation suffix rather than mtime, both of which change the answer.
79 lines
4.4 KiB
PHP
Executable File
79 lines
4.4 KiB
PHP
Executable File
<?php
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
// PURPOSE
|
|
// Reads certbot's own logs and names why renewals failed, in the handful of categories they
|
|
// actually fall into — rather than leaving 639 MB of Python tracebacks as the only record.
|
|
//
|
|
// WHY IT EXISTS
|
|
// Tools/cert_history.sh counts failures. It infers them from an expiry in the past, so it knows
|
|
// that a domain stopped renewing and nothing about why. The why is in certbot's log, which on
|
|
// this installation is 1001 rotated files, and the answer to "why did ten certificates stop
|
|
// renewing" was previously a person reading them by hand.
|
|
//
|
|
// The categories matter more than the count, because they are not independent. Missing DNS
|
|
// produces a failure; the failure is retried; the retries exhaust Let's Encrypt's rate limit;
|
|
// and the rate limit then fails every *other* domain too. A count says "2079 rate limit errors"
|
|
// and points at the symptom. The chain says "three hostnames have no DNS records, and that is
|
|
// what burned the rate limit for everything else".
|
|
//
|
|
// OPERATIONAL MODEL
|
|
// Reads the newest N log files and classifies each one. One file is one certbot run, and a run
|
|
// is what gets counted — a single failure writes its reason into the ACME response, the Python
|
|
// traceback and certbot's own summary, so counting lines reports one failure as three and makes
|
|
// the noisier categories look larger than the quiet ones. Bounded three ways, because this can
|
|
// be called from a page request:
|
|
//
|
|
// files CERT_TRIAGE_FILES, newest first
|
|
// bytes CERT_TRIAGE_MAX_BYTES per file, read from the end
|
|
// order by the numeric rotation suffix, never by mtime
|
|
//
|
|
// The suffix is load-bearing. Every one of these files carries the same mtime here — they are
|
|
// synced as a set, so the filesystem timestamps say they were all written at once. Sorting by
|
|
// mtime would pick an arbitrary thousand-file-old sample and report it as current.
|
|
//
|
|
// DESIGN PRINCIPLES
|
|
// Classify, never guess. A run that errored and matched no known pattern is counted as
|
|
// unclassified and said so, rather than being folded into the nearest category — an "unknown"
|
|
// that is honest is worth more than a tidy chart that is wrong.
|
|
//
|
|
// The root causes and the consequences are reported separately. Rate limiting is almost always
|
|
// downstream of something else here, and listing it alongside its own cause invites fixing the
|
|
// symptom.
|
|
//
|
|
// RUNTIME MODES
|
|
// cert_triage.php summary — categories, affected domains, and the causal reading
|
|
// cert_triage.php --json the same as JSON, for the Certs tab
|
|
// cert_triage.php --files=N override how many rotated logs to read
|
|
//
|
|
// CONFIGURATION
|
|
// CERT_TRIAGE_FILES rotated logs to read, newest first (default 40)
|
|
// CERT_TRIAGE_MAX_BYTES bytes read from the end of each (default 262144)
|
|
// CERT_TRIAGE_LOG_DIR override the log directory; normally found from the NPM container
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
|
|
require_once dirname(__DIR__) . '/include/auth.php';
|
|
|
|
$json = in_array('--json', $argv, true);
|
|
$filesOverride = 0;
|
|
foreach ($argv as $a) if (preg_match('/^--files=(\d+)$/', $a, $m)) $filesOverride = (int) $m[1];
|
|
|
|
$r = vv_cert_triage($filesOverride);
|
|
|
|
if ($json) { echo json_encode($r), "\n"; exit(0); }
|
|
|
|
if (!($r['ok'] ?? false)) { echo ($r['error'] ?? 'failed'), "\n"; exit(0); }
|
|
|
|
printf("%d certbot runs read, %d failed, %d of those matched nothing known\n\n",
|
|
$r['files_read'], $r['total'], $r['unclassified']);
|
|
|
|
if (!$r['total']) { echo "No renewal failures in the logs read.\n"; exit(0); }
|
|
|
|
foreach ($r['categories'] as $c) {
|
|
printf(" %-22s %5d %s\n", $c['id'], $c['count'], $c['what']);
|
|
foreach (array_slice($c['domains'], 0, 6) as $d) printf(" %s\n", $d);
|
|
if (count($c['domains']) > 6) printf(" … and %d more\n", count($c['domains']) - 6);
|
|
}
|
|
|
|
if ($r['reading']) { echo "\n"; foreach ($r['reading'] as $l) echo " $l\n"; }
|
|
exit(0);
|