Gmer4Lfe
99b58c0c4f
A low uptime figure, a refused login and a certificate that stopped renewing all looked the same from the row: a number, with the reason split across NPM, an Authelia config and the directory. The why-check goes and looks — TCP to the forward target, HTTP through the proxy, a second handshake with verification off to tell a broken certificate from a broken service. Forward hosts are docker names that only resolve on NPM's network, so an unresolvable one is redirected to the container address and the substitution is reported; a check that could not be made must never read as a check that failed. The access simulator walks the rules the way Authelia does and shows the ones it stepped over, reading whichever instance the chosen host points at rather than the one conf names — there are two here. Cert triage counts runs rather than log lines and orders by rotation suffix rather than mtime, both of which change the answer.
52 lines
2.9 KiB
Bash
52 lines
2.9 KiB
Bash
#!/bin/bash
|
|
# ==============================================================================================
|
|
# ==================================== Cert Triage =============================================
|
|
# ==============================================================================================
|
|
#
|
|
# PURPOSE
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Reads certbot's own logs and names why renewals failed, in the handful of categories they
|
|
# actually fall into.
|
|
#
|
|
# cert_history.sh counts failures — it notices an expiry in the past. It cannot say why. The why
|
|
# is in certbot's log, which here is 1001 rotated files and 639 MB, and the last time anyone
|
|
# answered "why did ten certificates stop renewing" they read them by hand.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL MODEL
|
|
# ==============================================================================================
|
|
#
|
|
# A wrapper. The work is in cert_triage.php.
|
|
#
|
|
# Counts runs, not lines. One log file is one certbot invocation, and one failure writes its
|
|
# reason three times — in the ACME response, the traceback, and certbot's own summary. Counting
|
|
# lines reports a single failure as three and inflates whichever category is most verbose.
|
|
#
|
|
# Reads the newest logs by rotation suffix, never by mtime. Every file here carries the same
|
|
# mtime because they are synced as a set, so mtime order is meaningless.
|
|
#
|
|
# The categories are separated into causes and consequences. Rate limiting is nearly always
|
|
# downstream — retries against a hostname with no DNS record exhaust the allowance, which then
|
|
# fails renewals for domains that have nothing wrong with them.
|
|
#
|
|
# ==============================================================================================
|
|
# RUNTIME MODES
|
|
# ==============================================================================================
|
|
#
|
|
# cert_triage.sh summary — categories, affected domains, and the causal reading
|
|
# cert_triage.sh --json the same as JSON, for the Certs tab
|
|
# cert_triage.sh --files=N override how many rotated logs to read
|
|
#
|
|
# ==============================================================================================
|
|
# CONFIGURATION
|
|
# ==============================================================================================
|
|
#
|
|
# CERT_TRIAGE_FILES rotated logs to read, newest first
|
|
# CERT_TRIAGE_MAX_BYTES bytes read from the end of each
|
|
# CERT_TRIAGE_LOG_DIR override the log directory; normally found from the NPM container
|
|
#
|
|
# ==============================================================================================
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
php "$SCRIPT_DIR/cert_triage.php" "$@"
|