79 lines
4.4 KiB
PHP
79 lines
4.4 KiB
PHP
<?php
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
// PURPOSE
|
|
// Reads certbot's own logs and names why renewals failed, in the handful of categories they
|
|
// actually fall into — rather than leaving 639 MB of Python tracebacks as the only record.
|
|
//
|
|
// WHY IT EXISTS
|
|
// Tools/cert_history.sh counts failures. It infers them from an expiry in the past, so it knows
|
|
// that a domain stopped renewing and nothing about why. The why is in certbot's log, which on
|
|
// this installation is 1001 rotated files, and the answer to "why did ten certificates stop
|
|
// renewing" was previously a person reading them by hand.
|
|
//
|
|
// The categories matter more than the count, because they are not independent. Missing DNS
|
|
// produces a failure; the failure is retried; the retries exhaust Let's Encrypt's rate limit;
|
|
// and the rate limit then fails every *other* domain too. A count says "2079 rate limit errors"
|
|
// and points at the symptom. The chain says "three hostnames have no DNS records, and that is
|
|
// what burned the rate limit for everything else".
|
|
//
|
|
// OPERATIONAL MODEL
|
|
// Reads the newest N log files and classifies each one. One file is one certbot run, and a run
|
|
// is what gets counted — a single failure writes its reason into the ACME response, the Python
|
|
// traceback and certbot's own summary, so counting lines reports one failure as three and makes
|
|
// the noisier categories look larger than the quiet ones. Bounded three ways, because this can
|
|
// be called from a page request:
|
|
//
|
|
// files CERT_TRIAGE_FILES, newest first
|
|
// bytes CERT_TRIAGE_MAX_BYTES per file, read from the end
|
|
// order by the numeric rotation suffix, never by mtime
|
|
//
|
|
// The suffix is load-bearing. Every one of these files carries the same mtime here — they are
|
|
// synced as a set, so the filesystem timestamps say they were all written at once. Sorting by
|
|
// mtime would pick an arbitrary thousand-file-old sample and report it as current.
|
|
//
|
|
// DESIGN PRINCIPLES
|
|
// Classify, never guess. A run that errored and matched no known pattern is counted as
|
|
// unclassified and said so, rather than being folded into the nearest category — an "unknown"
|
|
// that is honest is worth more than a tidy chart that is wrong.
|
|
//
|
|
// The root causes and the consequences are reported separately. Rate limiting is almost always
|
|
// downstream of something else here, and listing it alongside its own cause invites fixing the
|
|
// symptom.
|
|
//
|
|
// RUNTIME MODES
|
|
// cert_triage.php summary — categories, affected domains, and the causal reading
|
|
// cert_triage.php --json the same as JSON, for the Certs tab
|
|
// cert_triage.php --files=N override how many rotated logs to read
|
|
//
|
|
// CONFIGURATION
|
|
// CERT_TRIAGE_FILES rotated logs to read, newest first (default 40)
|
|
// CERT_TRIAGE_MAX_BYTES bytes read from the end of each (default 262144)
|
|
// CERT_TRIAGE_LOG_DIR override the log directory; normally found from the NPM container
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
|
|
require_once dirname(__DIR__) . '/include/auth.php';
|
|
|
|
$json = in_array('--json', $argv, true);
|
|
$filesOverride = 0;
|
|
foreach ($argv as $a) if (preg_match('/^--files=(\d+)$/', $a, $m)) $filesOverride = (int) $m[1];
|
|
|
|
$r = vv_cert_triage($filesOverride);
|
|
|
|
if ($json) { echo json_encode($r), "\n"; exit(0); }
|
|
|
|
if (!($r['ok'] ?? false)) { echo ($r['error'] ?? 'failed'), "\n"; exit(0); }
|
|
|
|
printf("%d certbot runs read, %d failed, %d of those matched nothing known\n\n",
|
|
$r['files_read'], $r['total'], $r['unclassified']);
|
|
|
|
if (!$r['total']) { echo "No renewal failures in the logs read.\n"; exit(0); }
|
|
|
|
foreach ($r['categories'] as $c) {
|
|
printf(" %-22s %5d %s\n", $c['id'], $c['count'], $c['what']);
|
|
foreach (array_slice($c['domains'], 0, 6) as $d) printf(" %s\n", $d);
|
|
if (count($c['domains']) > 6) printf(" … and %d more\n", count($c['domains']) - 6);
|
|
}
|
|
|
|
if ($r['reading']) { echo "\n"; foreach ($r['reading'] as $l) echo " $l\n"; }
|
|
exit(0);
|