Varaverk/Docker_Essentials/README-Docker_Essentials.md

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🐳 DOCKER ESSENTIALS

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Self-healing container lifecycle management for a 50+ container unRAID stack. Health monitoring that catches problems as they happen. Scheduled restarts that prevent degradation before it becomes visible. Network configuration that survives reboots and unRAID updates. Container image updates woven into the maintenance windows. Recovery tooling for when something genuinely breaks.

Why this folder exists: Docker on unRAID does not heal itself. A container that crashes stays crashed. A memory leak accumulates silently for days until the system starts swapping. A container whose dependency restarted first fails in a loop while the dependency comes up fine five seconds later. None of this surfaces clearly — it just builds into a system that feels flaky without a clear reason why. These scripts are the answer to all of that.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Running a large Docker stack on unRAID is genuinely powerful — but Docker's own tooling gives you almost nothing between "container is running" and "container has been dead for three days and you just noticed."

The built-in restart policies help with outright crashes, but they have zero visibility into memory leaks, frozen application layers, dependency ordering, restart loops, or the difference between a container that crashed and one you intentionally stopped. They give you no way to know why something keeps restarting — just that it does.

These are the specific problems that led to building this, in roughly the order they were encountered:

---

🔴 Memory Leaks Accumulating Silently

Emby's transcode session handling occasionally leaks memory — a session ends but its memory allocation doesn't fully release. SABnzbd's Python process slowly expands over days as it processes downloads. Without hard memory limits and automatic enforcement, these don't fail dramatically — they just consume more and more RAM until the system starts swapping and everything slows to a crawl. By then, nothing in the Docker logs tells you why the system feels slow. docker stats shows a container at 22GB and climbing, but Docker itself does nothing about it.

The fix: hard per-container memory ceilings in WATCHDOG_CONTAINERS. When a container exceeds its limit the watchdog restarts it immediately — no strikes, no waiting. A memory leak is not a transient spike. Immediate action is correct.

---

🔴 Containers That Look Running But Aren't Responding

Docker reports a container as Up 14 days while its application layer has been silently frozen for hours. The reverse proxy happily forwards traffic to a service that returns nothing. Users see a broken page. Docker sees a healthy container. The container process is technically running — it just isn't doing anything.

Docker's built-in health checks require a HEALTHCHECK instruction in the image itself, which most self-hosted images don't have. Even those that do often check something too shallow — a process exists, not whether it's actually serving requests.

The fix: HTTP health checks on the actual service port. curl to the real endpoint on every watchdog cycle. If the service doesn't respond within CURL_TIMEOUT seconds, that's a strike. Two consecutive failures trigger a restart. The distinction between "process running" and "service responding" is the distinction that matters.

---

🔴 Dependency Failures on Restart

Authelia connects to MariaDB at startup. If both are down simultaneously — say, after a power cut — and the watchdog restarts Authelia first, Authelia fails to connect, exits immediately, and goes into a crash loop. Meanwhile MariaDB is coming up fine in the background. The watchdog sees Authelia crash three times, adds it to the skip list, and sends a critical notification. Authelia was never broken. It just came up in the wrong order, failed at startup, and got punished for it.

The fix: dependency ordering. If a container's dependency is also down, the dependent is skipped entirely this cycle. The dependency is restarted first. On the next cycle — once MariaDB is actually accepting connections — Authelia is restarted and comes up cleanly. The skip list is never involved. No false alarms. No manual recovery needed.

---

🔴 Restart Loops Corrupting State

Some containers corrupt their internal state if restarted repeatedly in rapid succession. SQLite databases that don't get a clean shutdown write incomplete transactions. Partially applied database migrations leave schema in an inconsistent state. A container that crashes on startup after a bad migration gets restarted immediately, crashes again, gets restarted again — each restart has a chance of making the database worse, not better.

A naive watchdog that just keeps hammering a crashed container is actively harmful in this scenario. More restarts mean more corruption risk. The right response when restarts aren't working is to stop restarting and alert the operator.

The fix: restart loop protection. After WATCHDOG_CONTAINER_RESTART_LIMIT restarts within a rolling WATCHDOG_CONTAINER_RESTART_WINDOW hour window, the container goes on the skip list. A critical notification goes out. The watchdog stops touching it. The operator investigates and clears the skip list once the underlying problem is fixed.

---

🔴 Slow Degradation That Never Becomes a Failure

NginxProxyManager accumulates stale entries in its connection table over weeks of uptime. Dispatcharr's Live TV scheduler builds up internal scheduling state that makes decisions progressively slower after months of continuous operation. These containers never crash. They never throw errors. They just get progressively worse in ways that are hard to attribute to anything specific — until someone notices that the proxy feels slower than it used to, or that Live TV channel changes take longer than they should.

The fix: scheduled restarts. Not because something is broken, but because some containers simply perform better with a clean start. Daily at 1am for connection-heavy services. Weekly for less-critical services that run fine for weeks but benefit from a clean slate. Zero user impact — happens while everyone is asleep.

---

🔴 Network Configuration Lost After Updates

unRAID occasionally wipes custom Docker networks after updates — particularly networks created by Docker Compose stacks or the NextCloud AIO container. Any container that depended on those networks for internal communication suddenly can't reach its peers. memcached can't talk to NextCloud. CrowdSec can't talk to NginxProxyManager. Services appear to be up but silently fail to communicate with each other.

The fix: network recreation at every array start. docker_network_connect.sh checks every configured network on startup, creates any that are missing, and connects all configured containers to them. Idempotent — if everything is already correct, it does nothing and produces no output. If a network had to be created, it notifies — that only happens after an update, and you want to know when it does.

---

🔴 No Visibility Into What the Watchdog Already Tried

A container keeps appearing in a broken state. You SSH in and see it's stopped. You don't know if the watchdog tried to restart it and failed, gave up and skip-listed it, is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip list file in $STATE_DIR, check the restart history file, check the state file — none of which have obvious formats.

The fix: watchdog_skip_list_manager.sh (in Tools/). One command to see exactly what's on the skip list, which containers are stopped vs running, how many restarts were attempted, and what the watchdog's current state is. One command to clear a specific container and its history after you've fixed the problem. No manual file editing required.

---

━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Docker_Essentials/   ← acts on containers          (this folder)
System_Essentials/   ← acts on the server itself
Watchdogs/           ← reactive monitoring + last-resort stability
Monitors/            ← observes, measures, reports
Rsync/               ← moves data between servers

docker_watchdog.sh has moved to Watchdogs/. Container healing, memory limits, HTTP health checks, and skip list management are documented in Watchdogs/README-Watchdogs.md and Watchdogs/Manual-Watchdogs.md.

Four distinct responsibilities in this folder, each handled by dedicated scripts:

---

♻️ Proactive Freshness — docker_daily_restart.sh + docker_weekly_restart.sh

Scheduled restarts that prevent slow degradation before it becomes visible. Not because something broke — because some containers simply work better after a clean start.

Called by the maintenance orchestrators (daily_sync_maintenance.sh and weekly_sync_maintenance.sh) — not run standalone. They run inside the maintenance windows so any downtime from restarts is absorbed by the window that's already happening.

---

🔄 Image Currency — docker_update.sh

One script, three modes — there is no separate remainder script.

Mode	Targets	Runs
(default)	DAILY_RESTART_CONTAINERS	Daily, before docker_daily_restart.sh
--weekly	WEEKLY_RESTART_CONTAINERS	Weekly, before docker_weekly_restart.sh
--remainder	Everything running that is in neither list	Monthly, via monthly_maintenance.sh

The update always runs before its matching restart, so the restart lands on the freshly pulled image. Reversing that order would restart onto the old image and leave the new one sitting unused until the next window.

Each mode reuses the restart list it pairs with rather than maintaining its own — add a container to DAILY_RESTART_CONTAINERS once and it gets both the restart and the image pull. Remainder mode needs no list at all: it derives its targets from docker ps minus the daily list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers.

Fallback containers are deliberately excluded from remainder mode. This server only runs them during a fallback; the remote owns their version. Updating them here would risk the remote's older image meeting data written by a newer one after a handback.

---

🌐 Network Integrity — docker_network_connect.sh

Ensures custom Docker networks exist and containers are connected to them at every array start. Silent when everything is correct. Notifies when it has to create something — which means something was wiped and you should know about it.

---

🧹 Downloader Hygiene — downloaders_reset.sh

Maintenance reset for all download clients (slskd, SABnzbd, qBittorrent) every 30 minutes. Clears stuck searches, dead transfers, failed imports, and stale queue entries that download clients accumulate but never clean up themselves. Never touches active or in-progress downloads.

---

━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

docker_watchdog.sh has moved to Watchdogs/ and is now one of four coordinated single-pass scripts called every 15 minutes by Orchestrators/watchdog_orchestrator.sh.

Watchdogs/resource_watchdog.sh  ← reduces pressure before healing attempts
Watchdogs/docker_watchdog.sh    ← heals containers (reads resource_watchdog state)
Watchdogs/System/storage_watchdog.sh   ← pool growth + runaway log detection
Watchdogs/stability_watchdog.sh  ← last resort — reboots when healing has failed

Scripts in this folder (daily restart, updates, network connect) are unaffected — they run on their own schedules via the maintenance orchestrators and are not part of the every-minute watchdog cycle. See Watchdogs/README-Watchdogs.md for the full coordination model between all four watchdogs.

---

━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Script	Role	When It Runs
docker_daily_restart.sh	Nightly proactive restart of degradation-prone containers	1am via daily_sync_maintenance.sh
docker_weekly_restart.sh	Weekly restart of less-critical services	2:30am Sunday via weekly_sync_maintenance.sh
docker_update.sh	Container image updates — three modes (default / --weekly / --remainder)	Daily and weekly before each restart; monthly for the remainder
docker_network_connect.sh	Network existence + container connection enforcement	Every array start
docker_container_stop.sh	Ordered container shutdown — graceful then forced	Called by array_stopping.sh
downloaders_reset.sh	Download client hygiene — slskd / SABnzbd / qBittorrent	Every 30min via critical_sync_maintenance.sh

watchdog_skip_list_manager.sh — manual recovery tool for the skip list. Lives in Tools/ because it's an operator utility, not a lifecycle script.

---

━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Array starts
    │
    └── docker_network_connect.sh ────── run once at start
            ensure networks + connections exist
            silent if correct, notify if creating

Every 15 minutes (Orchestrators/watchdog_orchestrator.sh):
    └── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md

Daily maintenance window (1am):
    daily_sync_maintenance.sh
        ├── docker_update.sh ──────────── pull latest images (DAILY_RESTART_CONTAINERS)
        └── docker_daily_restart.sh ───── restart connection-heavy services

Weekly maintenance window (2:30am Sunday):
    weekly_sync_maintenance.sh
        ├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS)
        └── docker_weekly_restart.sh ──── restart onto the fresh image

Monthly maintenance window:
    monthly_maintenance.sh
        ├── docker_update.sh --remainder  update everything not in the managed lists
        └── Tools/docker_prune_images.sh --all

Critical maintenance window (every 30min):
    critical_sync_maintenance.sh
        └── downloaders_reset.sh ──────── clear stuck downloads / stale queue entries

Array stopping:
    array_stopping.sh
        └── docker_container_stop.sh ──── ordered graceful shutdown, verified per container

---

━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Every script here talks to the Docker daemon, so they share the same protections. Each script's own header documents its full set; these are the ones worth knowing folder-wide.

Daemon health is checked, not assumed. A hung daemon returns an empty container list, which is indistinguishable from "no containers running". Without the check, docker_container_stop.sh would report a clean shutdown that never happened, and docker_update.sh --remainder would report "nothing to update" while doing nothing.

Every docker call is timeout-wrapped. A wedged daemon cannot stall a maintenance window or hold a lock open. The one deliberate exception is docker pull — a large image legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer.

State is respected. Running containers get restarted; stopped ones stay stopped. A stopped container was almost certainly stopped on purpose, and none of these scripts has the authority to overrule that.

Restarts are verified, not assumed. After each restart the container is re-checked once it has had time to settle. A container that starts and immediately crashes is recorded as a failure and notified — a restart that did not stick is never reported as success.

Dependency ordering is shared with the watchdog. Restarts follow HOST*_WATCHDOG_DEPENDENCIES, with CONTAINER_DELAY between a dependency and its dependents, so a dependent is never brought up while what it needs is still initialising.

Locks prevent overlap. Long windows can outlast their interval; downloaders_reset.sh uses wait-mode because it runs every 30 minutes and the previous pass may still be finishing.

---