Gmer4Lfe
01601d210b
Both checks already answered their question on demand and both needed somebody to press a button on the right row. One host here has returned nothing but 5xx for months. Filed as findings, which is the existing answer to a condition that persists while nobody is looking. Grouped by cause rather than by hostname: a default policy of bypass produced twenty-two findings that were one sentence repeated, and they have one fix between them.
295 lines
16 KiB
PHP
295 lines
16 KiB
PHP
<?php
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
// PURPOSE
|
|
// Looks at the whole auth stack the way the Auth tab's two checks look at one thing, and files
|
|
// what it finds. Two questions, asked of every host NPM serves:
|
|
//
|
|
// Is this host actually serving? — the why-check, for anything below threshold
|
|
// Is this host actually protected? — the access check, for anything behind Authelia
|
|
//
|
|
// WHY IT EXISTS
|
|
// Both answers already existed on demand, and both required somebody to open the Auth tab and
|
|
// press a button on the right row. Nobody does that on a working day. One host here has returned
|
|
// nothing but 5xx to every request for months, and the hostnames that sit behind an auth_request
|
|
// block which then waves everyone through are invisible from every page in the plugin, because
|
|
// the fact is split across NPM, an Authelia config and the directory.
|
|
//
|
|
// Findings are the existing answer to "a condition that persists and nobody is looking" — the AI
|
|
// tab already lists, grades, ages out and notifies on them. This adds the auth stack as a source
|
|
// rather than inventing a second place for the same idea.
|
|
//
|
|
// OPERATIONAL MODEL
|
|
// One pass = read NPM once, then per host: the recorded uptime, and — only where it is warranted
|
|
// — the live checks. The expensive part is deliberately gated:
|
|
//
|
|
// proxy_down filed when 24h uptime is below AUTH_SWEEP_UPTIME_MIN *and* the host has been in
|
|
// that state longer than AUTH_SWEEP_DOWN_MIN. A restart or a reboot produces a
|
|
// perfectly ordinary dip, and a finding for every one of those is a list nobody
|
|
// reads. Live checks run only for hosts that pass this gate.
|
|
//
|
|
// access_open filed when a host carries an auth_request block and the Authelia deciding it
|
|
// reaches its default policy of bypass for a member of no relevant group. No
|
|
// network calls at all — this is three config files compared.
|
|
//
|
|
// Nothing is repaired, restarted or rewritten. This is a reporting pass, and it stays one: the
|
|
// remedies here are "start a container", "edit a rule", "change a default policy", and every one
|
|
// of them is a decision rather than a correction.
|
|
//
|
|
// DESIGN PRINCIPLES
|
|
// The gate is time, not count.
|
|
// A host that has been down for four hours is news whether the probe caught eight samples
|
|
// or eight hundred. Basing it on samples would file findings at a different threshold on a
|
|
// node whose probe runs at a different cadence.
|
|
//
|
|
// Findings are refreshed, not duplicated.
|
|
// vv_ai_finding_write() keys on kind + subject + ref, so a host still down tomorrow updates
|
|
// the record it filed today rather than adding a second one. The store closes what stops
|
|
// recurring on its own.
|
|
//
|
|
// The access check answers for a real user, not a hypothetical one.
|
|
// "Does the default policy let somebody in" is only meaningful about a person who exists.
|
|
// The sweep asks about a directory member holding no privileged group, because that is the
|
|
// account that reveals a rule which protects nothing.
|
|
//
|
|
// OPERATIONAL SAFEGUARDS
|
|
// Non-fatal, always. No NPM, no credentials, no domains, AI repair switched off — exits 0.
|
|
// One pass at a time, flock non-blocking, so a slow pass cannot overlap the next.
|
|
// Live probing is bounded by the gate above, so a total outage cannot turn one pass into
|
|
// thirty-five sequential timeouts.
|
|
//
|
|
// RUNTIME MODES
|
|
// auth_sweep.php one pass, files findings
|
|
// auth_sweep.php --dry-run report what it would file, write nothing
|
|
// auth_sweep.php --report one-screen summary for the Sunday report; silent when clean
|
|
//
|
|
// CONFIGURATION
|
|
// AUTH_SWEEP_ENABLED master switch (default true)
|
|
// AUTH_SWEEP_UPTIME_MIN 24h percentage below which a host is a candidate (default 96)
|
|
// AUTH_SWEEP_DOWN_MIN minutes it must have been failing before filing (default 120)
|
|
// AUTH_SWEEP_ACCESS_CHECK whether to run the protection half at all (default true)
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
|
|
require_once dirname(__DIR__) . '/include/auth.php';
|
|
require_once dirname(__DIR__) . '/include/ai_repair.php';
|
|
|
|
$dryRun = in_array('--dry-run', $argv, true);
|
|
$report = in_array('--report', $argv, true);
|
|
|
|
$lock = @fopen(sys_get_temp_dir() . '/vv_auth_sweep.lock', 'c');
|
|
if ($lock === false || !flock($lock, LOCK_EX | LOCK_NB)) { echo "another pass is running\n"; exit(0); }
|
|
|
|
try {
|
|
$v = vv_conf_vars();
|
|
if (strtolower(trim($v['AUTH_SWEEP_ENABLED'] ?? 'true')) === 'false') {
|
|
echo "AUTH_SWEEP_ENABLED is false\n"; exit(0);
|
|
}
|
|
$minPct = (float) ($v['AUTH_SWEEP_UPTIME_MIN'] ?? 96);
|
|
$minDown = max(1, (int) ($v['AUTH_SWEEP_DOWN_MIN'] ?? 120)) * 60;
|
|
$doAccess = strtolower(trim($v['AUTH_SWEEP_ACCESS_CHECK'] ?? 'true')) !== 'false';
|
|
|
|
$p = vv_npm_list_proxies();
|
|
if (!($p['ok'] ?? false)) { echo 'NPM: ' . ($p['error'] ?? 'unreadable') . "\n"; exit(0); }
|
|
|
|
$uptime = is_file(vv_auth_db_file('uptime.json'))
|
|
? (json_decode((string) @file_get_contents(vv_auth_db_file('uptime.json')), true) ?: []) : [];
|
|
$doms = $uptime['domains'] ?? [];
|
|
|
|
$filed = $skipped = 0;
|
|
$lines = [];
|
|
|
|
// ── Half one: hosts that are not serving ──
|
|
foreach ($p['proxies'] as $h) {
|
|
// A host switched off is not a fault, it is a decision. Filing one would put a finding on
|
|
// the list for every host the operator has deliberately parked.
|
|
if (($h['enabled'] ?? true) === false) continue;
|
|
|
|
$names = [];
|
|
foreach ($h['domain_names'] ?? [] as $d) {
|
|
$d = strtolower(trim((string) $d));
|
|
if ($d !== '' && !str_contains($d, '*')) $names[] = $d;
|
|
}
|
|
if (!$names) continue;
|
|
|
|
// The worst name decides, for the same reason the row does: a host is only as reachable as
|
|
// its least reachable hostname, and averaging hides one dead name behind three healthy ones.
|
|
$worstPct = null; $worstDom = ''; $worstRec = null;
|
|
foreach ($names as $d) {
|
|
$r = $doms[$d] ?? null;
|
|
if (!$r) continue;
|
|
$pct = vv_auth_uptime_window($r['hours'] ?? [], 24);
|
|
if ($pct === null) continue;
|
|
if ($worstPct === null || $pct < $worstPct) { $worstPct = $pct; $worstDom = $d; $worstRec = $r; }
|
|
}
|
|
if ($worstPct === null || $worstPct >= $minPct) continue;
|
|
|
|
// How long it has actually been like this. last_change is when the state last flipped, so
|
|
// a host that went down two minutes ago is excluded here and caught on a later pass — which
|
|
// is the whole point of the gate.
|
|
$since = (int) ($worstRec['last_change'] ?? 0);
|
|
$for = $since > 0 ? time() - $since : 0;
|
|
if (($worstRec['state'] ?? '') === 'down' && $for < $minDown) { $skipped++; continue; }
|
|
|
|
$why = vv_npm_why((int) ($h['id'] ?? 0));
|
|
if (!($why['ok'] ?? false)) continue;
|
|
|
|
// The findings the check already writes, which is the whole reason this does not have its
|
|
// own opinion about what is wrong. Only the decisive ones are carried into the record.
|
|
$said = [];
|
|
foreach ($why['findings'] as $f) if (in_array($f['level'], ['bad', 'warn'], true)) $said[] = $f['text'];
|
|
if (!$said) { $skipped++; continue; }
|
|
|
|
$evidence = sprintf("%s is at %.2f%% over 24h%s.\n\n%s",
|
|
$worstDom, $worstPct,
|
|
$for > 0 ? ' and has been ' . ($worstRec['state'] ?? 'failing') . ' for ' . round($for / 3600, 1) . ' hours' : '',
|
|
implode("\n", $said));
|
|
|
|
$lines[] = sprintf(' %-34s %6.2f%% %s', $worstDom, $worstPct, $said[0]);
|
|
if ($dryRun) { $filed++; continue; }
|
|
|
|
$w = vv_ai_finding_write([
|
|
'kind' => 'proxy_down',
|
|
'subject' => implode(', ', $names),
|
|
'ref' => 'npm:proxy:' . ($h['id'] ?? 0),
|
|
'evidence' => $evidence,
|
|
'observed' => sprintf('%.2f%% over 24h', $worstPct),
|
|
// Proven, because these are measurements rather than an inference: a TCP connect either
|
|
// completed or it did not, and the access log either counted 5xx or it did not.
|
|
'proven' => true,
|
|
]);
|
|
$w['ok'] ? $filed++ : $skipped++;
|
|
}
|
|
|
|
// ── Half two: hosts that are guarded but not protected ──
|
|
//
|
|
// Grouped by Authelia instance, not filed per hostname. The first version of this produced
|
|
// twenty-two findings that were all the same sentence, because they all had the same cause: a
|
|
// default policy of bypass means every host whose rule does not name your group lets you
|
|
// through, so the number of findings was really the number of hostnames. One finding per
|
|
// instance, naming the hosts it affects, is the fact — and it has one fix rather than
|
|
// twenty-two.
|
|
$openLines = [];
|
|
if ($doAccess) {
|
|
// Somebody who exists and holds none of the groups the rules name. A rule that still lets
|
|
// this account through is a rule protecting nothing, and asking about an invented username
|
|
// would prove nothing about the directory.
|
|
$probe = vv_auth_sweep_ordinary_user();
|
|
$byInstance = [];
|
|
|
|
foreach ($p['proxies'] as $h) {
|
|
if (($h['enabled'] ?? true) === false) continue;
|
|
if (!str_contains((string) ($h['advanced_config'] ?? ''), 'auth_request')) continue;
|
|
|
|
foreach ($h['domain_names'] ?? [] as $d) {
|
|
$d = strtolower(trim((string) $d));
|
|
if ($d === '' || str_contains($d, '*')) continue;
|
|
|
|
$a = vv_auth_access_check($d, $probe);
|
|
if (!($a['ok'] ?? false)) continue;
|
|
if (($a['policy'] ?? '') !== 'bypass') continue;
|
|
|
|
$inst = (string) ($a['authelia']['container'] ?? '?');
|
|
$byInstance[$inst]['default'] = (string) ($a['default_policy'] ?? '?');
|
|
$byInstance[$inst]['config'] = (string) ($a['authelia']['config'] ?? '');
|
|
// Which of the two shapes this is, per host: a hostname no rule mentions, or one a
|
|
// rule covers and then steps over. They have different fixes — write a rule, or
|
|
// widen an existing one — so the record keeps them apart.
|
|
$stepped = false;
|
|
foreach ($a['trace'] ?? [] as $t) if (($t['skip'] ?? '') === 'subject') $stepped = true;
|
|
$byInstance[$inst][$stepped ? 'stepped' : 'unlisted'][] = $d;
|
|
}
|
|
}
|
|
|
|
foreach ($byInstance as $inst => $g) {
|
|
$unlisted = $g['unlisted'] ?? [];
|
|
$stepped = $g['stepped'] ?? [];
|
|
$all = array_merge($unlisted, $stepped);
|
|
if (!$all) continue;
|
|
|
|
$ev = [];
|
|
$ev[] = $inst . ' has default_policy: ' . ($g['default'] ?? '?') . ', so any request its rules do '
|
|
. 'not decide is allowed through. ' . count($all) . ' hostname'
|
|
. (count($all) === 1 ? '' : 's') . ' behind an auth_request block pointing at it reach the '
|
|
. 'application without being asked to authenticate.';
|
|
if ($unlisted) $ev[] = "\nNo rule mentions these at all:\n " . implode("\n ", $unlisted);
|
|
if ($stepped) $ev[] = "\nA rule covers these but does not apply to an ordinary account"
|
|
. ($probe ? ' (tested as ' . $probe . ')' : '') . ":\n " . implode("\n ", $stepped);
|
|
$ev[] = "\nRules are in " . ($g['config'] ?: 'a config that was not found') . '.';
|
|
// Said plainly because "bypass" reads as harmless and it is the single most
|
|
// consequential line in that file.
|
|
$ev[] = "\nThe fix is a default_policy of deny with an explicit rule for anything that is "
|
|
. "meant to be public — not a rule per hostname above.";
|
|
|
|
$openLines[] = sprintf(' %-22s default bypass — %d hostname%s unprotected',
|
|
$inst, count($all), count($all) === 1 ? '' : 's');
|
|
if ($dryRun) { $filed++; continue; }
|
|
|
|
$w = vv_ai_finding_write([
|
|
'kind' => 'access_open',
|
|
'subject' => $inst . ' — ' . count($all) . ' hostnames not protected',
|
|
'ref' => 'authelia:' . $inst . ':default_policy',
|
|
'evidence' => implode("\n", $ev),
|
|
'observed' => 'default_policy: ' . ($g['default'] ?? '?'),
|
|
'proven' => true,
|
|
]);
|
|
$w['ok'] ? $filed++ : $skipped++;
|
|
}
|
|
}
|
|
|
|
if ($report) {
|
|
// Silent on a clean week. The orchestrator's job is to say nothing when there is nothing
|
|
// to say, and a section that always prints is a section that stops being read.
|
|
if (!$lines && !$openLines) exit(0);
|
|
echo "Auth stack review\n";
|
|
if ($lines) { echo "\nProxy hosts not serving:\n"; foreach ($lines as $l) echo "$l\n"; }
|
|
if ($openLines) { echo "\nBehind Authelia but not protected:\n"; foreach ($openLines as $l) echo "$l\n"; }
|
|
exit(1);
|
|
}
|
|
|
|
printf("%s%d finding%s, %d skipped\n", $dryRun ? 'dry run — ' : '', $filed, $filed === 1 ? '' : 's', $skipped);
|
|
foreach (array_merge($lines, $openLines) as $l) echo "$l\n";
|
|
exit(0);
|
|
|
|
} finally {
|
|
flock($lock, LOCK_UN);
|
|
fclose($lock);
|
|
}
|
|
|
|
// A directory member holding none of the groups any rule names. Returns '' when every user is
|
|
// privileged or the directory cannot be read, and the caller then asks about no user at all —
|
|
// which still answers the "no rule mentions this host" case and simply cannot answer the
|
|
// "the rule stepped over this person" one.
|
|
function vv_auth_sweep_ordinary_user(): string {
|
|
$named = [];
|
|
// Groups named by the rules of every Authelia instance in play, not just the configured one —
|
|
// a .us hostname is decided by a config this conf file does not point at.
|
|
foreach (vv_auth_sweep_configs() as $cfg) {
|
|
$r = vv_authelia_read_rules($cfg);
|
|
foreach (($r['ok'] ?? false) ? $r['rules'] : [] as $rule) {
|
|
$s = $rule['subject'] ?? null;
|
|
foreach (is_array($s) ? $s : [$s] as $alt)
|
|
foreach (is_array($alt) ? $alt : [$alt] as $one)
|
|
if (is_string($one) && str_starts_with($one, 'group:')) $named[strtolower(substr($one, 6))] = true;
|
|
}
|
|
}
|
|
|
|
$u = vv_lldap_list_users();
|
|
foreach (($u['ok'] ?? false) ? $u['users'] : [] as $user) {
|
|
$mine = array_map('strtolower', array_filter(array_column($user['groups'] ?? [], 'displayName')));
|
|
if (array_intersect($mine, array_keys($named))) continue;
|
|
return (string) ($user['id'] ?? '');
|
|
}
|
|
return '';
|
|
}
|
|
|
|
// Every Authelia configuration this installation actually uses, discovered through the proxy hosts
|
|
// rather than listed anywhere. Two instances run here and conf names one.
|
|
function vv_auth_sweep_configs(): array {
|
|
$out = [];
|
|
$p = vv_npm_list_proxies();
|
|
foreach (($p['ok'] ?? false) ? $p['proxies'] : [] as $h) {
|
|
$i = vv_authelia_instance_for($h);
|
|
if (($i['config'] ?? '') !== '' && is_file($i['config'])) $out[$i['config']] = true;
|
|
}
|
|
return array_keys($out);
|
|
}
|