#!/bin/bash # ============================================================================================== # ==================================== Cert Triage ============================================= # ============================================================================================== # # PURPOSE # ───────────────────────────────────────────────────────────────────────────── # Reads certbot's own logs and names why renewals failed, in the handful of categories they # actually fall into. # # cert_history.sh counts failures — it notices an expiry in the past. It cannot say why. The why # is in certbot's log, which here is 1001 rotated files and 639 MB, and the last time anyone # answered "why did ten certificates stop renewing" they read them by hand. # # ============================================================================================== # OPERATIONAL MODEL # ============================================================================================== # # A wrapper. The work is in cert_triage.php. # # Counts runs, not lines. One log file is one certbot invocation, and one failure writes its # reason three times — in the ACME response, the traceback, and certbot's own summary. Counting # lines reports a single failure as three and inflates whichever category is most verbose. # # Reads the newest logs by rotation suffix, never by mtime. Every file here carries the same # mtime because they are synced as a set, so mtime order is meaningless. # # The categories are separated into causes and consequences. Rate limiting is nearly always # downstream — retries against a hostname with no DNS record exhaust the allowance, which then # fails renewals for domains that have nothing wrong with them. # # ============================================================================================== # DESIGN PRINCIPLES # ============================================================================================== # # One invocation is one data point. # A single failure writes its reason three times — the ACME response, the traceback, and # certbot's summary. Counting lines would report it as three and inflate whichever category # happens to be the most verbose, which is the opposite of what triage is for. # # Causes are separated from consequences. # Rate limiting is nearly always downstream: retries against a hostname with no DNS record # exhaust the allowance, which then fails renewals for domains that have nothing wrong with # them. Reporting the rate limit as the problem sends the operator to fix the wrong domains. # # Rotation suffix is the clock, not mtime. # Every file in this directory carries the same mtime because they arrive as a synced set, so # ordering by mtime returns an arbitrary answer that looks authoritative. # # Bounded by design, not by hope. # 639 MB across 1001 files cannot be read in a page load. Only CERT_TRIAGE_FILES logs are # opened, and only CERT_TRIAGE_MAX_BYTES from the end of each, because a failure explains # itself at the end of the run rather than the start. # # ============================================================================================== # OPERATIONAL SAFEGUARDS # ============================================================================================== # # Read-only throughout. It opens certbot's logs and nothing else — no certificate is requested, # renewed or deleted, and no log is rotated or truncated by anything here. # # Diagnosis only. This names why renewals failed; acting on that is the operator's, and # cert_history.sh remains the thing that records what happened. # # The log directory is normally discovered from the NPM container rather than hardcoded, so a # container path change surfaces as "no logs found" rather than as a silently empty triage. # # ============================================================================================== # RUNTIME MODES # ============================================================================================== # # cert_triage.sh summary — categories, affected domains, and the causal reading # cert_triage.sh --json the same as JSON, for the Certs tab # cert_triage.sh --files=N override how many rotated logs to read # # ============================================================================================== # CONFIGURATION # ============================================================================================== # # CERT_TRIAGE_FILES rotated logs to read, newest first # CERT_TRIAGE_MAX_BYTES bytes read from the end of each # CERT_TRIAGE_LOG_DIR override the log directory; normally found from the NPM container # # ============================================================================================== SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" php "$SCRIPT_DIR/cert_triage.php" "$@"