From 958391e326c74688e370e51c9170ff4dd8ab4fc5 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 11 Jul 2026 18:23:00 -0400 Subject: [PATCH] Add Emby/Jellyfin deep API health checks to docker_watchdog.sh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both had a basic HTTP check (Emby) or no coverage at all (Jellyfin), but neither would have caught today's real incident: Jellyfin's SQLite database locked up hard (repeated 'database table is locked' errors, 30s+ query timeouts) while its own /System/Info endpoint kept responding 200 the whole time — a basic HTTP check on that endpoint would never have tripped. /Users forces an actual DB round-trip and was confirmed live to hang during the exact incident. Generalized the API check's success condition to also accept array-shaped responses (/Users returns an array; the existing check only recognized object fields like .ServerName/.Id/.Version, which would error when applied to an array) — benefits any future array-returning endpoint, not just this one. Also corrected the host.conf.template's API_CHECKS format comment, which described a 3-field format the code never actually used. --- Deployment/host.conf.template | 8 +++++++- Watchdogs/docker_watchdog.sh | 5 ++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/Deployment/host.conf.template b/Deployment/host.conf.template index 69c3f20..de1c2de 100644 --- a/Deployment/host.conf.template +++ b/Deployment/host.conf.template @@ -331,14 +331,20 @@ # HTTP health check URLs — checked every cycle. declare -A HOSTN_WATCHDOG_CONTAINER_URLS=( # ["Emby"]="http://localhost:8096" + # ["Jellyfin"]="http://localhost:8095" # ["NginxProxyManager"]="http://localhost:7818" # ["Authelia"]="http://localhost:9091/api/health" # ["Authelia-Secondary"]="http://localhost:9092/api/health" # ["Lldap"]="http://localhost:17170" ) -# API-level health checks. Format: ["ContainerName"]="url|expected_json_key|expected_value" +# API-level health checks — catches HTTP-200-but-internally-frozen containers (DB lock, +# deadlocked thread, etc.) that a basic HTTP check above would miss. Pick an endpoint that +# forces a real DB round-trip — a lightweight status endpoint may stay 200 even while the +# rest of the app is locked up. Format: ["ContainerName"]="url|APIKey" declare -A HOSTN_WATCHDOG_CONTAINER_API_CHECKS=( + # ["Emby"]="${HOSTN_EMBY_URL}/Users|${HOSTN_EMBY_API_KEY}" + # ["Jellyfin"]="${HOSTN_JELLYFIN_URL}/Users|${HOSTN_JELLYFIN_API_KEY}" ) # Required containers — must always be running. diff --git a/Watchdogs/docker_watchdog.sh b/Watchdogs/docker_watchdog.sh index c70b0af..12cb930 100755 --- a/Watchdogs/docker_watchdog.sh +++ b/Watchdogs/docker_watchdog.sh @@ -899,7 +899,10 @@ CYCLE_START=$(date +%s) continue fi - if echo "$_resp" | jq -e '.ServerName // .Id // .Version' >/dev/null 2>&1; then + # Object-shaped responses (e.g. /System/Info) pass via ServerName/Id/Version. + # Array-shaped responses (e.g. /Users) pass on any valid array — indexing an + # array with a string key would itself error in jq, so branch on type first. + if echo "$_resp" | jq -e 'if type == "array" then true else (.ServerName // .Id // .Version) != null end' >/dev/null 2>&1; then set_strikes "${container}_api" 0 "$WATCHDOG_STATE_FILE" else API_STRIKES=$(get_strikes "${container}_api" "$WATCHDOG_STATE_FILE")