Watch the auth stack on a schedule, so nobody has to open the tab

Both checks already answered their question on demand and both needed
somebody to press a button on the right row. One host here has returned
nothing but 5xx for months.

Filed as findings, which is the existing answer to a condition that
persists while nobody is looking. Grouped by cause rather than by
hostname: a default policy of bypass produced twenty-two findings that
were one sentence repeated, and they have one fix between them.
This commit is contained in:
Gmer4Lfe
2026-08-16 01:59:11 -04:00
parent 99b58c0c4f
commit 01601d210b
4 changed files with 397 additions and 0 deletions
+39
View File
@@ -494,6 +494,11 @@
# per-domain renewal and failure counts.
"Plugin/unraid/Tools/cert_history.sh" # record cert renewals, failures and age per domain
"Plugin/unraid/Tools/npm_access_stats.sh" # aggregate NPM per-host access logs into request and byte totals
# After both of the above, because it reads what they write — the uptime history and the
# per-host request totals are two of the four things it reasons from. Daily rather than
# hourly: everything it files is a condition that has already lasted hours by the time
# AUTH_SWEEP_DOWN_MIN lets it through, so a faster cadence would find nothing new.
"Plugin/unraid/Tools/auth_sweep.sh" # file findings for hosts that are not serving, and hostnames Authelia is not protecting
)
# Pull latest images for DAILY_RESTART_CONTAINERS before the daily restart.
@@ -1498,6 +1503,40 @@
UPTIME_PROBE_TIMEOUT=8 # seconds per domain before it counts as down
UPTIME_PROBE_LIST_TTL=900 # seconds to reuse the domain list from NPM before re-reading it
# ── Auth Sweep ──
# Tools/auth_sweep.sh asks the two questions the Auth tab answers about one host, about every host,
# and files what it finds as findings on the AI tab. Reports only — it starts nothing and rewrites
# nothing, because every remedy here (start a container, edit a rule, change a default policy) is a
# decision rather than a correction.
#
# proxy_down a host below UPTIME_MIN that has been failing longer than DOWN_MIN. The time gate
# is what keeps a reboot from filing a finding for every hostname on the machine.
# access_open an Authelia instance whose default policy lets through every hostname its rules do
# not decide. Filed once per instance, not once per hostname — they all have the
# same single fix, and one finding per name is twenty-two copies of one sentence.
#
# The access half reads whichever Authelia each proxy host actually points at, which is not always
# the one HOST*_AUTHELIA_CONFIG names — this installation runs two.
AUTH_SWEEP_ENABLED=true # master switch
AUTH_SWEEP_UPTIME_MIN=96 # 24h percentage below which a host becomes a candidate
AUTH_SWEEP_DOWN_MIN=120 # minutes it must have been failing before anything is filed
AUTH_SWEEP_ACCESS_CHECK=true # run the "is it actually protected" half at all
# ── Cert Triage ──
# Tools/cert_triage.sh reads certbot's own logs and names why renewals failed. cert_history.sh
# counts failures by noticing an expiry in the past; this reads the reason.
#
# Counts runs, not lines: one log file is one certbot invocation, and one failure writes its
# reason into the ACME response, the traceback and certbot's summary, so line counting reports it
# three times and inflates whichever category is most verbose.
#
# The log directory is found from the NPM container's own mount. Set CERT_TRIAGE_LOG_DIR only if
# that lookup cannot work. Both bounds exist because this is reachable from a page request and the
# directory here is 639 MB across a thousand rotated files.
CERT_TRIAGE_FILES=40 # rotated logs to read, newest first by rotation suffix
CERT_TRIAGE_MAX_BYTES=262144 # bytes read from the end of each — a run's reason is always last
CERT_TRIAGE_LOG_DIR="" # empty = find it from the NPM container
# ━━━ Backup Verify ━━━
# Verifies rsync mirror health by comparing random file checksums between servers.
# Catches silent corruption or incomplete syncs that rsync itself wouldn't detect.