# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 🐳 DOCKER ESSENTIALS # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ **Self-healing container lifecycle management for a 50+ container unRAID stack.** Health monitoring that catches problems as they happen. Scheduled restarts that prevent degradation before it becomes visible. Network configuration that survives reboots and unRAID updates. Container image updates woven into the maintenance windows. Recovery tooling for when something genuinely breaks. > **Why this folder exists:** Docker on unRAID does not heal itself. A container that > crashes stays crashed. A memory leak accumulates silently for days until the system > starts swapping. A container whose dependency restarted first fails in a loop while > the dependency comes up fine five seconds later. None of this surfaces clearly — it > just builds into a system that feels flaky without a clear reason why. These scripts > are the answer to all of that. --- ## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Running a large Docker stack on unRAID is genuinely powerful — but Docker's own tooling gives you almost nothing between "container is running" and "container has been dead for three days and you just noticed." The built-in restart policies help with outright crashes, but they have zero visibility into memory leaks, frozen application layers, dependency ordering, restart loops, or the difference between a container that crashed and one you intentionally stopped. They give you no way to know *why* something keeps restarting — just that it does. These are the specific problems that led to building this, in roughly the order they were encountered: --- ### 🔴 Memory Leaks Accumulating Silently Emby's transcode session handling occasionally leaks memory — a session ends but its memory allocation doesn't fully release. SABnzbd's Python process slowly expands over days as it processes downloads. Without hard memory limits and automatic enforcement, these don't fail dramatically — they just consume more and more RAM until the system starts swapping and everything slows to a crawl. By then, nothing in the Docker logs tells you why the system feels slow. `docker stats` shows a container at 22GB and climbing, but Docker itself does nothing about it. The fix: hard per-container memory ceilings in `WATCHDOG_CONTAINERS`. When a container exceeds its limit the watchdog restarts it immediately — no strikes, no waiting. A memory leak is not a transient spike. Immediate action is correct. --- ### 🔴 Containers That Look Running But Aren't Responding Docker reports a container as `Up 14 days` while its application layer has been silently frozen for hours. The reverse proxy happily forwards traffic to a service that returns nothing. Users see a broken page. Docker sees a healthy container. The container process is technically running — it just isn't doing anything. Docker's built-in health checks require a `HEALTHCHECK` instruction in the image itself, which most self-hosted images don't have. Even those that do often check something too shallow — a process exists, not whether it's actually serving requests. The fix: HTTP health checks on the actual service port. `curl` to the real endpoint on every watchdog cycle. If the service doesn't respond within `CURL_TIMEOUT` seconds, that's a strike. Two consecutive failures trigger a restart. The distinction between "process running" and "service responding" is the distinction that matters. --- ### 🔴 Dependency Failures on Restart Authelia connects to MariaDB at startup. If both are down simultaneously — say, after a power cut — and the watchdog restarts Authelia first, Authelia fails to connect, exits immediately, and goes into a crash loop. Meanwhile MariaDB is coming up fine in the background. The watchdog sees Authelia crash three times, adds it to the skip list, and sends a critical notification. Authelia was never broken. It just came up in the wrong order, failed at startup, and got punished for it. The fix: dependency ordering. If a container's dependency is also down, the dependent is skipped entirely this cycle. The dependency is restarted first. On the next cycle — once MariaDB is actually accepting connections — Authelia is restarted and comes up cleanly. The skip list is never involved. No false alarms. No manual recovery needed. --- ### 🔴 Restart Loops Corrupting State Some containers corrupt their internal state if restarted repeatedly in rapid succession. SQLite databases that don't get a clean shutdown write incomplete transactions. Partially applied database migrations leave schema in an inconsistent state. A container that crashes on startup after a bad migration gets restarted immediately, crashes again, gets restarted again — each restart has a chance of making the database worse, not better. A naive watchdog that just keeps hammering a crashed container is actively harmful in this scenario. More restarts mean more corruption risk. The right response when restarts aren't working is to stop restarting and alert the operator. The fix: restart loop protection. After `WATCHDOG_CONTAINER_RESTART_LIMIT` restarts within a rolling `WATCHDOG_CONTAINER_RESTART_WINDOW` hour window, the container goes on the skip list. A critical notification goes out. The watchdog stops touching it. The operator investigates and clears the skip list once the underlying problem is fixed. --- ### 🔴 Slow Degradation That Never Becomes a Failure NginxProxyManager accumulates stale entries in its connection table over weeks of uptime. Dispatcharr's Live TV scheduler builds up internal scheduling state that makes decisions progressively slower after months of continuous operation. These containers never crash. They never throw errors. They just get progressively worse in ways that are hard to attribute to anything specific — until someone notices that the proxy feels slower than it used to, or that Live TV channel changes take longer than they should. The fix: scheduled restarts. Not because something is broken, but because some containers simply perform better with a clean start. Daily at 1am for connection-heavy services. Weekly for less-critical services that run fine for weeks but benefit from a clean slate. Zero user impact — happens while everyone is asleep. --- ### 🔴 Network Configuration Lost After Updates unRAID occasionally wipes custom Docker networks after updates — particularly networks created by Docker Compose stacks or the NextCloud AIO container. Any container that depended on those networks for internal communication suddenly can't reach its peers. memcached can't talk to NextCloud. CrowdSec can't talk to NginxProxyManager. Services appear to be up but silently fail to communicate with each other. The fix: network recreation at every array start. `docker_network_connect.sh` checks every configured network on startup, creates any that are missing, and connects all configured containers to them. Idempotent — if everything is already correct, it does nothing and produces no output. If a network had to be created, it notifies — that only happens after an update, and you want to know when it does. --- ### 🔴 No Visibility Into What the Watchdog Already Tried A container keeps appearing in a broken state. You SSH in and see it's stopped. You don't know if the watchdog tried to restart it and failed, gave up and skip-listed it, is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip list file on `/boot/config/`, check the restart history file, check the state file — none of which have obvious formats. The fix: `watchdog_skip_list_manager.sh` (in `Tools/`). One command to see exactly what's on the skip list, which containers are stopped vs running, how many restarts were attempted, and what the watchdog's current state is. One command to clear a specific container and its history after you've fixed the problem. No manual file editing required. --- ## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ``` Docker_Essentials/ ← acts on containers (this folder) unRAID_Essentials/ ← acts on the server itself Monitors/ ← observes, measures, reports Rsync/ ← moves data between servers ``` Five distinct responsibilities, each handled by dedicated scripts: --- ### 🔁 Reactive Healing — `docker_watchdog.sh` Continuous two-tier monitoring that catches problems as they happen and acts on them immediately. Runs as a background process started at array start. Every 15 minutes it checks the full stack and fixes what it can — silently when everything is fine, visibly when something needs attention. Two tiers because different containers need different monitoring strategies: - **Tier 1** — explicit per-container configuration with specific thresholds - **Tier 2** — global scan of everything that's running with catch-all health checks --- ### ♻️ Proactive Freshness — `docker_daily_restart.sh` + `docker_weekly_restart.sh` Scheduled restarts that prevent slow degradation before it becomes visible. Not because something broke — because some containers simply work better after a clean start. Called by the maintenance orchestrators (`daily_sync_maintenance.sh` and `weekly_sync_maintenance.sh`) — not run standalone. They run inside the maintenance windows so any downtime from restarts is absorbed by the window that's already happening. --- ### 🔄 Image Currency — `docker_update.sh` + `docker_update_remaining.sh` Keeps all container images current without manual intervention. Daily updates for the auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly remainder pass for everything else — derives the target list automatically from `docker ps` minus what was already updated, so there is no second list to maintain. --- ### 🌐 Network Integrity — `docker_network_connect.sh` Ensures custom Docker networks exist and containers are connected to them at every array start. Silent when everything is correct. Notifies when it has to create something — which means something was wiped and you should know about it. --- ### 🧹 Downloader Hygiene — `downloaders_reset.sh` Maintenance reset for all download clients (slskd, SABnzbd, qBittorrent) every 15 minutes. Clears stuck searches, dead transfers, failed imports, and stale queue entries that download clients accumulate but never clean up themselves. Never touches active or in-progress downloads. --- ## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Two watchdogs run simultaneously. They are designed to work together, not compete: ``` system_watchdog.sh ← watches the server: RAM, CPU, disk, kernel, daemon health docker_watchdog.sh ← watches the containers: memory, CPU, HTTP response, crashes ``` **The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh` would see stopped containers on its next cycle and restart them — directly undoing the RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover. The system would eventually hit the reboot threshold anyway, having accomplished nothing. **The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When `system_watchdog.sh` triggers a RAM emergency shutdown it writes `mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every cycle and defers all container restart logic until it clears. Health URL checks for excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still run. Everything else stands down. **The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh` checks how long ago the state file was modified. If it's more than 2 hours old while `mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running. `docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced indefinitely by a stale flag from a process that's no longer running. --- ## ━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | Script | Role | When It Runs | |--------|------|-------------| | `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop — array start | | `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` | | `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` | | `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window | | `docker_update_remaining.sh` | Image update + prune for all containers not in managed lists | End of weekly maintenance window | | `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start | | `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` | | `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 15min via `critical_sync_maintenance.sh` | > `watchdog_skip_list_manager.sh` — manual recovery tool for the skip list. Lives in `Tools/` because it's an operator utility, not a lifecycle script. --- ## ━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ``` Array starts │ ├── docker_network_connect.sh ────── run once at start │ ensure networks + connections exist │ silent if correct, notify if creating │ └── docker_watchdog.sh ──────────── continuous background loop (every 15min) Tier 1: memory, CPU, HTTP, required containers Tier 2: global unhealthy / OOM / crash / dead scan reads system_watchdog state (RAM emergency deferral) │ │ (on skip list event → operator uses) └── Watchdogs/watchdog_skip_list_manager.sh inspect state, clear after fixing root cause Daily maintenance window (1am): daily_sync_maintenance.sh ├── docker_update.sh ──────────── pull latest images (DAILY_RESTART_CONTAINERS) └── docker_daily_restart.sh ───── restart connection-heavy services Weekly maintenance window (2:30am Sunday): weekly_sync_maintenance.sh ├── docker_weekly_restart.sh ──── restart less-critical services ├── docker_update.sh --remainder ─ update containers not in managed lists └── docker_update_remaining.sh ── prune dangling images Critical maintenance window (every 15min): critical_sync_maintenance.sh └── downloaders_reset.sh ──────── clear stuck downloads / stale queue entries Array stopping: array_stopping.sh └── docker_container_stop.sh ──── ordered graceful shutdown ```