Make the Auth tab explain a number instead of only showing it
A low uptime figure, a refused login and a certificate that stopped renewing all looked the same from the row: a number, with the reason split across NPM, an Authelia config and the directory. The why-check goes and looks — TCP to the forward target, HTTP through the proxy, a second handshake with verification off to tell a broken certificate from a broken service. Forward hosts are docker names that only resolve on NPM's network, so an unresolvable one is redirected to the container address and the substitution is reported; a check that could not be made must never read as a check that failed. The access simulator walks the rules the way Authelia does and shows the ones it stepped over, reading whichever instance the chosen host points at rather than the one conf names — there are two here. Cert triage counts runs rather than log lines and orders by rotation suffix rather than mtime, both of which change the answer.
This commit is contained in:
Gmer4Lfe
Executable
+78
@@ -0,0 +1,78 @@
|
||||
<?php
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
// PURPOSE
|
||||
// Reads certbot's own logs and names why renewals failed, in the handful of categories they
|
||||
// actually fall into — rather than leaving 639 MB of Python tracebacks as the only record.
|
||||
//
|
||||
// WHY IT EXISTS
|
||||
// Tools/cert_history.sh counts failures. It infers them from an expiry in the past, so it knows
|
||||
// that a domain stopped renewing and nothing about why. The why is in certbot's log, which on
|
||||
// this installation is 1001 rotated files, and the answer to "why did ten certificates stop
|
||||
// renewing" was previously a person reading them by hand.
|
||||
//
|
||||
// The categories matter more than the count, because they are not independent. Missing DNS
|
||||
// produces a failure; the failure is retried; the retries exhaust Let's Encrypt's rate limit;
|
||||
// and the rate limit then fails every *other* domain too. A count says "2079 rate limit errors"
|
||||
// and points at the symptom. The chain says "three hostnames have no DNS records, and that is
|
||||
// what burned the rate limit for everything else".
|
||||
//
|
||||
// OPERATIONAL MODEL
|
||||
// Reads the newest N log files and classifies each one. One file is one certbot run, and a run
|
||||
// is what gets counted — a single failure writes its reason into the ACME response, the Python
|
||||
// traceback and certbot's own summary, so counting lines reports one failure as three and makes
|
||||
// the noisier categories look larger than the quiet ones. Bounded three ways, because this can
|
||||
// be called from a page request:
|
||||
//
|
||||
// files CERT_TRIAGE_FILES, newest first
|
||||
// bytes CERT_TRIAGE_MAX_BYTES per file, read from the end
|
||||
// order by the numeric rotation suffix, never by mtime
|
||||
//
|
||||
// The suffix is load-bearing. Every one of these files carries the same mtime here — they are
|
||||
// synced as a set, so the filesystem timestamps say they were all written at once. Sorting by
|
||||
// mtime would pick an arbitrary thousand-file-old sample and report it as current.
|
||||
//
|
||||
// DESIGN PRINCIPLES
|
||||
// Classify, never guess. A run that errored and matched no known pattern is counted as
|
||||
// unclassified and said so, rather than being folded into the nearest category — an "unknown"
|
||||
// that is honest is worth more than a tidy chart that is wrong.
|
||||
//
|
||||
// The root causes and the consequences are reported separately. Rate limiting is almost always
|
||||
// downstream of something else here, and listing it alongside its own cause invites fixing the
|
||||
// symptom.
|
||||
//
|
||||
// RUNTIME MODES
|
||||
// cert_triage.php summary — categories, affected domains, and the causal reading
|
||||
// cert_triage.php --json the same as JSON, for the Certs tab
|
||||
// cert_triage.php --files=N override how many rotated logs to read
|
||||
//
|
||||
// CONFIGURATION
|
||||
// CERT_TRIAGE_FILES rotated logs to read, newest first (default 40)
|
||||
// CERT_TRIAGE_MAX_BYTES bytes read from the end of each (default 262144)
|
||||
// CERT_TRIAGE_LOG_DIR override the log directory; normally found from the NPM container
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
require_once dirname(__DIR__) . '/include/auth.php';
|
||||
|
||||
$json = in_array('--json', $argv, true);
|
||||
$filesOverride = 0;
|
||||
foreach ($argv as $a) if (preg_match('/^--files=(\d+)$/', $a, $m)) $filesOverride = (int) $m[1];
|
||||
|
||||
$r = vv_cert_triage($filesOverride);
|
||||
|
||||
if ($json) { echo json_encode($r), "\n"; exit(0); }
|
||||
|
||||
if (!($r['ok'] ?? false)) { echo ($r['error'] ?? 'failed'), "\n"; exit(0); }
|
||||
|
||||
printf("%d certbot runs read, %d failed, %d of those matched nothing known\n\n",
|
||||
$r['files_read'], $r['total'], $r['unclassified']);
|
||||
|
||||
if (!$r['total']) { echo "No renewal failures in the logs read.\n"; exit(0); }
|
||||
|
||||
foreach ($r['categories'] as $c) {
|
||||
printf(" %-22s %5d %s\n", $c['id'], $c['count'], $c['what']);
|
||||
foreach (array_slice($c['domains'], 0, 6) as $d) printf(" %s\n", $d);
|
||||
if (count($c['domains']) > 6) printf(" … and %d more\n", count($c['domains']) - 6);
|
||||
}
|
||||
|
||||
if ($r['reading']) { echo "\n"; foreach ($r['reading'] as $l) echo " $l\n"; }
|
||||
exit(0);
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ==================================== Cert Triage =============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Reads certbot's own logs and names why renewals failed, in the handful of categories they
|
||||
# actually fall into.
|
||||
#
|
||||
# cert_history.sh counts failures — it notices an expiry in the past. It cannot say why. The why
|
||||
# is in certbot's log, which here is 1001 rotated files and 639 MB, and the last time anyone
|
||||
# answered "why did ten certificates stop renewing" they read them by hand.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# A wrapper. The work is in cert_triage.php.
|
||||
#
|
||||
# Counts runs, not lines. One log file is one certbot invocation, and one failure writes its
|
||||
# reason three times — in the ACME response, the traceback, and certbot's own summary. Counting
|
||||
# lines reports a single failure as three and inflates whichever category is most verbose.
|
||||
#
|
||||
# Reads the newest logs by rotation suffix, never by mtime. Every file here carries the same
|
||||
# mtime because they are synced as a set, so mtime order is meaningless.
|
||||
#
|
||||
# The categories are separated into causes and consequences. Rate limiting is nearly always
|
||||
# downstream — retries against a hostname with no DNS record exhaust the allowance, which then
|
||||
# fails renewals for domains that have nothing wrong with them.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# cert_triage.sh summary — categories, affected domains, and the causal reading
|
||||
# cert_triage.sh --json the same as JSON, for the Certs tab
|
||||
# cert_triage.sh --files=N override how many rotated logs to read
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# CERT_TRIAGE_FILES rotated logs to read, newest first
|
||||
# CERT_TRIAGE_MAX_BYTES bytes read from the end of each
|
||||
# CERT_TRIAGE_LOG_DIR override the log directory; normally found from the NPM container
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
php "$SCRIPT_DIR/cert_triage.php" "$@"
|
||||
@@ -97,10 +97,6 @@ function vv_uptime_write(array $d): bool {
|
||||
return true;
|
||||
}
|
||||
|
||||
function vv_uptime_pct(int $up, int $total): ?float {
|
||||
return $total > 0 ? round($up / $total * 100, 2) : null;
|
||||
}
|
||||
|
||||
// ── Report ────────────────────────────────────────────────────────────────────
|
||||
// Anything that was not perfect over the last seven days, for the Sunday report. Prints nothing
|
||||
// and exits 0 when every domain was clean — the orchestrator's job is to be quiet on a good week,
|
||||
@@ -157,14 +153,12 @@ if ($status || $events) {
|
||||
exit(0);
|
||||
}
|
||||
|
||||
// Shared by --status, --report and the API. Buckets are keyed by time so "the last N" is a key sort, not an assumption about how many
|
||||
// samples a period should contain — a pass that did not run leaves no bucket rather than a zero.
|
||||
// Shared by --status, --report and the API, and defined once in include/auth.php so the three
|
||||
// callers cannot drift apart on what "the last N" means. Buckets are keyed by time, so it is a key
|
||||
// sort rather than an assumption about how many samples a period should contain — a pass that did
|
||||
// not run leaves no bucket rather than a zero.
|
||||
function vv_uptime_window(array $buckets, int $n): ?float {
|
||||
if (!$buckets) return null;
|
||||
krsort($buckets);
|
||||
$u = $t = 0;
|
||||
foreach (array_slice($buckets, 0, $n, true) as $b) { $u += $b['u'] ?? 0; $t += $b['t'] ?? 0; }
|
||||
return vv_uptime_pct($u, $t);
|
||||
return vv_auth_uptime_window($buckets, $n);
|
||||
}
|
||||
|
||||
// ── One pass ──────────────────────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user