A low uptime figure, a refused login and a certificate that stopped
renewing all looked the same from the row: a number, with the reason
split across NPM, an Authelia config and the directory.
The why-check goes and looks — TCP to the forward target, HTTP through
the proxy, a second handshake with verification off to tell a broken
certificate from a broken service. Forward hosts are docker names that
only resolve on NPM's network, so an unresolvable one is redirected to
the container address and the substitution is reported; a check that
could not be made must never read as a check that failed.
The access simulator walks the rules the way Authelia does and shows the
ones it stepped over, reading whichever instance the chosen host points
at rather than the one conf names — there are two here.
Cert triage counts runs rather than log lines and orders by rotation
suffix rather than mtime, both of which change the answer.
NPM's access log only describes hosts somebody visited; the host most likely to be quietly broken
is the one nobody does. Probes carry a User-Agent npm_access_stats.sh drops — unmarked, this
monitor would be fifty thousand requests a day in the very logs it reports on.
NPM writes an access log per host and counts nothing, so 475 MB of logs held the only answer to
"is anything using this". Aggregated on a schedule and read from a few kB of JSON; the row also
now says whether an auth_request block is in front of the site, which nothing showed before.
Authentik is the likely destination and the page had Authelia and lldap wired in at every
level, so the seam goes in now: the panels and every endpoint action route off one conf value,
and a stack that cannot be driven yet says so rather than drawing controls with nothing behind.
lldap lets you edit five things about a user and one about a group; the page reached two of
them, so correcting a surname or a group's name still meant opening the container's own WebUI.
The block is rebuilt from the parsed model on every save and the parser discarded comments,
so one press of Save deleted the five ## lines that are the only thing in the file saying
what each rule is for — which inline editing was about to make far easier to press.
Unraid already enforces CSRF on every POST via auto_prepend, but its
injector is jQuery-only — the plugin's native fetch() calls carried no
token and were being terminated before the endpoint ran, silently,
because csrf_terminate exits with an empty body that r.json() swallows.
Writing down what each endpoint actually guarantees made the places it
didn't obvious — shell arguments reaching a crontab or a bash -c
unescaped, master.conf written without tmp+rename, and conf edits that
could be saved without ever being parsed.
Gmer4Lfe