Add Emby/Jellyfin deep API health checks to docker_watchdog.sh

Both had a basic HTTP check (Emby) or no coverage at all (Jellyfin), but
neither would have caught today's real incident: Jellyfin's SQLite
database locked up hard (repeated 'database table is locked' errors,
30s+ query timeouts) while its own /System/Info endpoint kept responding
200 the whole time — a basic HTTP check on that endpoint would never have
tripped. /Users forces an actual DB round-trip and was confirmed live to
hang during the exact incident.

Generalized the API check's success condition to also accept array-shaped
responses (/Users returns an array; the existing check only recognized
object fields like .ServerName/.Id/.Version, which would error when
applied to an array) — benefits any future array-returning endpoint, not
just this one. Also corrected the host.conf.template's API_CHECKS format
comment, which described a 3-field format the code never actually used.
This commit is contained in:
Gmer4Lfe
2026-07-11 18:23:00 -04:00
parent 2eb595552d
commit 958391e326
2 changed files with 11 additions and 2 deletions
+7 -1
View File
@@ -331,14 +331,20 @@
# HTTP health check URLs — checked every cycle.
declare -A HOSTN_WATCHDOG_CONTAINER_URLS=(
# ["Emby"]="http://localhost:8096"
# ["Jellyfin"]="http://localhost:8095"
# ["NginxProxyManager"]="http://localhost:7818"
# ["Authelia"]="http://localhost:9091/api/health"
# ["Authelia-Secondary"]="http://localhost:9092/api/health"
# ["Lldap"]="http://localhost:17170"
)
# API-level health checks. Format: ["ContainerName"]="url|expected_json_key|expected_value"
# API-level health checks — catches HTTP-200-but-internally-frozen containers (DB lock,
# deadlocked thread, etc.) that a basic HTTP check above would miss. Pick an endpoint that
# forces a real DB round-trip — a lightweight status endpoint may stay 200 even while the
# rest of the app is locked up. Format: ["ContainerName"]="url|APIKey"
declare -A HOSTN_WATCHDOG_CONTAINER_API_CHECKS=(
# ["Emby"]="${HOSTN_EMBY_URL}/Users|${HOSTN_EMBY_API_KEY}"
# ["Jellyfin"]="${HOSTN_JELLYFIN_URL}/Users|${HOSTN_JELLYFIN_API_KEY}"
)
# Required containers — must always be running.
+4 -1
View File
@@ -899,7 +899,10 @@ CYCLE_START=$(date +%s)
continue
fi
if echo "$_resp" | jq -e '.ServerName // .Id // .Version' >/dev/null 2>&1; then
# Object-shaped responses (e.g. /System/Info) pass via ServerName/Id/Version.
# Array-shaped responses (e.g. /Users) pass on any valid array — indexing an
# array with a string key would itself error in jq, so branch on type first.
if echo "$_resp" | jq -e 'if type == "array" then true else (.ServerName // .Id // .Version) != null end' >/dev/null 2>&1; then
set_strikes "${container}_api" 0 "$WATCHDOG_STATE_FILE"
else
API_STRIKES=$(get_strikes "${container}_api" "$WATCHDOG_STATE_FILE")