FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Track what happens to every certificate, and show it on the Certs tab

Browse Source 
NPM knows what a certificate is today and nothing about what it was, so ten of them could fail
renewal for months — 1001 certbot runs, zero successes — without anything on any page saying so.
Counts start at zero and are only ever observed; only first_seen is seeded, from NPM's own date.
This commit is contained in:
Gmer4Lfe
2026-08-15 18:51:38 -04:00
parent 13de1dab82
commit 71ba0a239f
6 changed files with 554 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Plugin/unraid/Tools/cert_history.php
+292
View File
@@ -0,0 +1,292 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Keeps a running record of every certificate NPM holds: when it was first seen, how many times
// it has renewed, how many times it has been found expired, and how long it has been tracked.
// The Certs tab reads what this writes; nothing here draws anything.
//
// WHY IT EXISTS
// NPM knows what a certificate is today and nothing about what it was. cert_monitor.sh checks
// the two domains in CERT_MONITOR_DOMAINS over TLS and writes a snapshot that the next run
// overwrites. So "has this domain been renewing cleanly for the last year" had no answer
// anywhere, and the ten certificates that had been failing renewal for months — 1001 failed
// certbot runs, zero successes — were visible only by reading certbot's logs by hand.
//
// OPERATIONAL MODEL
// One pass per invocation, over the certificate list NPM returns:
// 1. A domain not in the store is added, with first_seen taken from NPM's created_on rather
// than from now — that date is real and this file should not pretend tracking began the
// day it was installed.
// 2. An expiry that has moved later than the stored one is a renewal.
// 3. An expiry in the past is a failure, and a strike.
// 4. CERT_HISTORY_STRIKES strikes retires the domain from the active list. It stays in the
// store — the history is the point — but stops being counted as something that works.
// 5. A tracked domain that is no longer in NPM is marked removed, not struck. Deleting a
// certificate is a decision; failing to renew one is not.
//
// DESIGN PRINCIPLES
// Counts start at zero and are only ever real.
// There is no renewal history anywhere on this host to seed from — every Let's Encrypt
// archive directory holds exactly one generation, so no renewal has ever completed through
// this NPM. Back-filling a plausible number would make the card a guess wearing a
// statistic. first_seen is seeded because it is a fact NPM already holds.
//
// Keyed by domain, not by NPM id.
// A certificate deleted and re-issued gets a new id and is the same domain. Keying on the
// id would restart the history of anything ever recreated, which is exactly the moment the
// history is worth having.
//
// A renewal is an expiry that moved forward.
// Derived from the list NPM already returns rather than from a TLS handshake per domain:
// forty openssl connections to answer a question the API has already answered is a lot of
// runtime for the same fact. cert_monitor.sh still does the handshake for the domains that
// need the outside world's view.
//
// OPERATIONAL SAFEGUARDS
// Non-fatal, always. A missing NPM, bad credentials or an unreadable store exits 0 with a
// message. This runs inside the Sunday report and must never be the reason it fails.
//
// One pass at a time — flock, non-blocking.
//
// The store is written atomically, tmp + rename, and a write that cannot be verified leaves
// the previous file in place. This is append-mostly history; a truncated write loses all of it.
//
// RUNTIME MODES
// cert_history.php one pass, updates the store
// cert_history.php --dry-run reports what it would change, writes nothing
// cert_history.php --status prints the store as a table
//
// CONFIGURATION
// CERT_HISTORY_STRIKES consecutive failed passes before a domain is retired (default 5)
// DB_DIR cert_history.json lives here
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once dirname(__DIR__) . '/include/auth.php';
$dryRun = in_array('--dry-run', $argv, true);
$status = in_array('--status', $argv, true);
const VV_CERT_HISTORY_FILE = 'cert_history.json';
function vv_cert_history_path(): string {
return rtrim(defined('DB_DIR') ? DB_DIR : (DATA_DIR . '/db'), '/') . '/' . VV_CERT_HISTORY_FILE;
}
function vv_cert_history_read(): array {
$p = vv_cert_history_path();
if (!is_file($p)) return ['domains' => [], 'created' => time()];
$j = json_decode((string) @file_get_contents($p), true);
// A corrupt store is not overwritten from here — it is reported and left alone, because the
// alternative is a pass that silently restarts every counter from zero.
if (!is_array($j) || !isset($j['domains']) || !is_array($j['domains'])) return [];
return $j;
}
function vv_cert_history_write(array $data): bool {
$p = vv_cert_history_path();
$dir = dirname($p);
if (!is_dir($dir) && !@mkdir($dir, 0755, true)) return false;
$data['updated'] = time();
$json = json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES);
if ($json === false) return false;
$tmp = $p . '.vv.tmp';
if (@file_put_contents($tmp, $json) === false) return false;
// Verified before it replaces the real file. This is the only copy of the history.
if (json_decode((string) @file_get_contents($tmp), true) === null) { @unlink($tmp); return false; }
if (!@rename($tmp, $p)) { @unlink($tmp); return false; }
return true;
}
function vv_cert_strike_limit(): int {
$n = (int) (vv_conf_vars()['CERT_HISTORY_STRIKES'] ?? 5);
return $n > 0 ? $n : 5;
}
// ── Status ────────────────────────────────────────────────────────────────────
if ($status) {
$h = vv_cert_history_read();
if (!$h) { echo "cert_history.json is unreadable or malformed\n"; exit(0); }
$d = $h['domains'] ?? [];
printf("%-32s %7s %7s %7s %7s %-10s %s\n",
'domain', 'checks', 'renews', 'fails', 'strikes', 'expires', 'tracked');
foreach ($d as $dom => $r) {
printf("%-32s %7d %7d %7d %7d %-10s %s%s\n", substr($dom, 0, 32),
$r['checks'] ?? 0, $r['renewals'] ?? 0, $r['failures'] ?? 0, $r['strikes'] ?? 0,
$r['last_expiry'] ?? '-', vv_cert_span($r['first_seen'] ?? time()),
!empty($r['retired_at']) ? ' RETIRED' : (!empty($r['removed_at']) ? ' removed' : ''));
}
printf("\n%d tracked, strike limit %d\n", count($d), vv_cert_strike_limit());
exit(0);
}
// Years, months and days rather than a day count. "3 years 6 months and 22 days" is the shape the
// question is asked in; 1298 days is the same fact in a unit nobody thinks in.
function vv_cert_span(int $from, ?int $to = null): string {
$a = (new DateTime())->setTimestamp($from);
$b = (new DateTime())->setTimestamp($to ?? time());
if ($b < $a) return '0d';
$d = $a->diff($b);
$out = [];
if ($d->y) $out[] = $d->y . 'y';
if ($d->m) $out[] = $d->m . 'mo';
if ($d->d || !$out) $out[] = $d->d . 'd';
return implode(' ', $out);
}
// What one observation of one domain does to its record. Pure — takes the record and the facts,
// returns the new record and what happened — so the strike ladder can be tested without waiting
// for a certificate to expire. That mattered: nothing on this host is expired right now, so the
// failure branch would otherwise ship having never run.
//
// $exp the certificate's expiry, as a timestamp
// $now the moment of this pass
// $limit strikes before retirement
function vv_cert_apply(array $r, int $exp, int $now, int $limit): array {
$out = ['renewed' => false, 'failed' => false, 'retired' => false, 'from' => '', 'to' => ''];
$r['checks'] = ($r['checks'] ?? 0) + 1;
$r['last_seen'] = $now;
// Cleared on sight: a domain that is back in NPM is not removed any more, whatever it was
// last pass.
$r['removed_at'] = null;
// Both sides reduced to a date. last_expiry is stored as Y-m-d and NPM's expires_on carries a
// time, so comparing raw timestamps made every re-read of the same certificate look like a
// renewal to a few hours later.
$expDay = strtotime(date('Y-m-d', $exp));
$prev = !empty($r['last_expiry']) ? strtotime($r['last_expiry']) : null;
if ($prev !== null && $expDay > $prev) {
$r['renewals'] = ($r['renewals'] ?? 0) + 1;
$r['last_renewal'] = $now;
// A renewal clears the strikes and un-retires. The point of a strike count is "how long
// has this been broken", and it is no longer broken.
$r['strikes'] = 0;
$r['retired_at'] = null;
$out['renewed'] = true;
$out['from'] = date('Y-m-d', $prev);
$out['to'] = date('Y-m-d', $expDay);
}
if ($exp < $now) {
$r['failures'] = ($r['failures'] ?? 0) + 1;
$r['strikes'] = ($r['strikes'] ?? 0) + 1;
$out['failed'] = true;
if ($r['strikes'] >= $limit && empty($r['retired_at'])) {
$r['retired_at'] = $now;
$r['retired_reason'] = "expired for {$r['strikes']} consecutive passes";
$out['retired'] = true;
}
}
$r['last_expiry'] = date('Y-m-d', $expDay);
$out['record'] = $r;
return $out;
}
// ── One pass ──────────────────────────────────────────────────────────────────
$lockPath = sys_get_temp_dir() . '/vv_cert_history.lock';
$lock = @fopen($lockPath, 'c');
if ($lock === false || !flock($lock, LOCK_EX | LOCK_NB)) {
echo "another pass is running\n";
exit(0);
}
try {
if ($miss = vv_auth_creds_missing('npm')) { echo "$miss\n"; exit(0); }
$certs = vv_npm_list_certs();
if (!$certs) { echo "NPM returned no certificates — nothing to record\n"; exit(0); }
$hist = vv_cert_history_read();
if (!$hist) { echo "cert_history.json is malformed — refusing to overwrite it\n"; exit(1); }
$store = $hist['domains'] ?? [];
$limit = vv_cert_strike_limit();
$now = time();
$seen = [];
$added = $renewed = $failed = $retired = $removed = 0;
$notes = [];
// Resolved to one certificate per domain before anything is counted. A domain can appear on
// more than one certificate — three do here, left behind by re-issuing rather than replacing —
// and walking the list directly counted each of them as a separate check of the same domain,
// then read the second one's expiry as a renewal of the first.
//
// The winner is the latest expiry, because that is the one actually worth serving; the earliest
// created is kept as first_seen for the same reason NPM's created_on is used at all.
$byDomain = [];
foreach ($certs as $c) {
$exp = !empty($c['expires_on']) ? strtotime((string) $c['expires_on']) : false;
if ($exp === false) continue;
$created = !empty($c['created_on']) ? strtotime((string) $c['created_on']) : $now;
foreach (($c['domain_names'] ?? []) as $d) {
$d = strtolower(trim((string) $d));
if ($d === '') continue;
if (!isset($byDomain[$d]) || $exp > $byDomain[$d]['exp'])
$byDomain[$d] = ['exp' => $exp, 'cert' => $c, 'created' => $created];
else
$byDomain[$d]['created'] = min($byDomain[$d]['created'], $created ?: $now);
}
}
{
foreach ($byDomain as $domain => $info) {
$exp = $info['exp'];
$c = $info['cert'];
$seen[$domain] = true;
if (!isset($store[$domain])) {
$created = $info['created'];
$store[$domain] = [
'first_seen' => $created !== false ? $created : $now,
'seeded_from' => 'npm_created_on',
'checks' => 0, 'renewals' => 0, 'failures' => 0, 'strikes' => 0,
'last_expiry' => null, 'last_renewal' => null,
'retired_at' => null, 'removed_at' => null,
];
$added++;
$notes[] = "added $domain (first seen " . date('Y-m-d', $store[$domain]['first_seen']) . ')';
}
$r = &$store[$domain];
$r['npm_id'] = $c['id'] ?? null;
$r['provider'] = $c['provider'] ?? null;
$res = vv_cert_apply($r, $exp, $now, $limit);
$r = $res['record'];
if ($res['renewed']) { $renewed++; $notes[] = "renewed $domain ($res[from] → $res[to])"; }
if ($res['failed']) { $failed++; }
if ($res['retired']) { $retired++; $notes[] = "RETIRED $domain after {$r['strikes']} strikes"; }
unset($r);
}
}
// Tracked but no longer in NPM. Marked, never struck and never deleted from the store — the
// history of a domain that used to exist is the reason this file is kept.
foreach ($store as $domain => &$r) {
if (isset($seen[$domain])) continue;
if (empty($r['removed_at'])) {
$r['removed_at'] = $now;
$removed++;
$notes[] = "no longer in NPM: $domain";
}
}
unset($r);
ksort($store);
$hist['domains'] = $store;
$hist['last_pass'] = $now;
printf("%d certificates, %d domains tracked — added %d, renewed %d, failed %d, retired %d, removed %d\n",
count($certs), count($store), $added, $renewed, $failed, $retired, $removed);
foreach ($notes as $n) echo " $n\n";
if ($dryRun) { echo "dry run — nothing written\n"; exit(0); }
if (!vv_cert_history_write($hist)) { echo "could not write " . vv_cert_history_path() . "\n"; exit(1); }
echo 'wrote ' . vv_cert_history_path() . "\n";
exit(0);
} finally {
flock($lock, LOCK_UN);
fclose($lock);
}