88 lines
4.9 KiB
Bash
Executable File
88 lines
4.9 KiB
Bash
Executable File
#!/bin/bash
|
|
# ==============================================================================================
|
|
# ==================================== Cert Triage =============================================
|
|
# ==============================================================================================
|
|
#
|
|
# PURPOSE
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Reads certbot's own logs and names why renewals failed, in the handful of categories they
|
|
# actually fall into.
|
|
#
|
|
# cert_history.sh counts failures — it notices an expiry in the past. It cannot say why. The why
|
|
# is in certbot's log, which here is 1001 rotated files and 639 MB, and the last time anyone
|
|
# answered "why did ten certificates stop renewing" they read them by hand.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL MODEL
|
|
# ==============================================================================================
|
|
#
|
|
# A wrapper. The work is in cert_triage.php.
|
|
#
|
|
# Counts runs, not lines. One log file is one certbot invocation, and one failure writes its
|
|
# reason three times — in the ACME response, the traceback, and certbot's own summary. Counting
|
|
# lines reports a single failure as three and inflates whichever category is most verbose.
|
|
#
|
|
# Reads the newest logs by rotation suffix, never by mtime. Every file here carries the same
|
|
# mtime because they are synced as a set, so mtime order is meaningless.
|
|
#
|
|
# The categories are separated into causes and consequences. Rate limiting is nearly always
|
|
# downstream — retries against a hostname with no DNS record exhaust the allowance, which then
|
|
# fails renewals for domains that have nothing wrong with them.
|
|
#
|
|
# ==============================================================================================
|
|
# DESIGN PRINCIPLES
|
|
# ==============================================================================================
|
|
#
|
|
# One invocation is one data point.
|
|
# A single failure writes its reason three times — the ACME response, the traceback, and
|
|
# certbot's summary. Counting lines would report it as three and inflate whichever category
|
|
# happens to be the most verbose, which is the opposite of what triage is for.
|
|
#
|
|
# Causes are separated from consequences.
|
|
# Rate limiting is nearly always downstream: retries against a hostname with no DNS record
|
|
# exhaust the allowance, which then fails renewals for domains that have nothing wrong with
|
|
# them. Reporting the rate limit as the problem sends the operator to fix the wrong domains.
|
|
#
|
|
# Rotation suffix is the clock, not mtime.
|
|
# Every file in this directory carries the same mtime because they arrive as a synced set, so
|
|
# ordering by mtime returns an arbitrary answer that looks authoritative.
|
|
#
|
|
# Bounded by design, not by hope.
|
|
# 639 MB across 1001 files cannot be read in a page load. Only CERT_TRIAGE_FILES logs are
|
|
# opened, and only CERT_TRIAGE_MAX_BYTES from the end of each, because a failure explains
|
|
# itself at the end of the run rather than the start.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL SAFEGUARDS
|
|
# ==============================================================================================
|
|
#
|
|
# Read-only throughout. It opens certbot's logs and nothing else — no certificate is requested,
|
|
# renewed or deleted, and no log is rotated or truncated by anything here.
|
|
#
|
|
# Diagnosis only. This names why renewals failed; acting on that is the operator's, and
|
|
# cert_history.sh remains the thing that records what happened.
|
|
#
|
|
# The log directory is normally discovered from the NPM container rather than hardcoded, so a
|
|
# container path change surfaces as "no logs found" rather than as a silently empty triage.
|
|
#
|
|
# ==============================================================================================
|
|
# RUNTIME MODES
|
|
# ==============================================================================================
|
|
#
|
|
# cert_triage.sh summary — categories, affected domains, and the causal reading
|
|
# cert_triage.sh --json the same as JSON, for the Certs tab
|
|
# cert_triage.sh --files=N override how many rotated logs to read
|
|
#
|
|
# ==============================================================================================
|
|
# CONFIGURATION
|
|
# ==============================================================================================
|
|
#
|
|
# CERT_TRIAGE_FILES rotated logs to read, newest first
|
|
# CERT_TRIAGE_MAX_BYTES bytes read from the end of each
|
|
# CERT_TRIAGE_LOG_DIR override the log directory; normally found from the NPM container
|
|
#
|
|
# ==============================================================================================
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
php "$SCRIPT_DIR/cert_triage.php" "$@"
|