Varaverk/Docker_Essentials/README-Docker_Essentials.md

# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🐳  DOCKER ESSENTIALS
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Self-healing container lifecycle management for a 50+ container unRAID stack.**
Health monitoring that catches problems as they happen. Scheduled restarts that prevent
degradation before it becomes visible. Network configuration that survives reboots and
unRAID updates. Container image updates woven into the maintenance windows. Recovery
tooling for when something genuinely breaks.

> **Why this folder exists:** Docker on unRAID does not heal itself. A container that
> crashes stays crashed. A memory leak accumulates silently for days until the system
> starts swapping. A container whose dependency restarted first fails in a loop while
> the dependency comes up fine five seconds later. None of this surfaces clearly — it
> just builds into a system that feels flaky without a clear reason why. These scripts
> are the answer to all of that.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Running a large Docker stack on unRAID is genuinely powerful — but Docker's own tooling
gives you almost nothing between "container is running" and "container has been dead for
three days and you just noticed."

The built-in restart policies help with outright crashes, but they have zero visibility
into memory leaks, frozen application layers, dependency ordering, restart loops, or the
difference between a container that crashed and one you intentionally stopped. They give
you no way to know *why* something keeps restarting — just that it does.

These are the specific problems that led to building this, in roughly the order they were
encountered:

---

### 🔴  Memory Leaks Accumulating Silently

Emby's transcode session handling occasionally leaks memory — a session ends but its
memory allocation doesn't fully release. SABnzbd's Python process slowly expands over
days as it processes downloads. Without hard memory limits and automatic enforcement,
these don't fail dramatically — they just consume more and more RAM until the system
starts swapping and everything slows to a crawl. By then, nothing in the Docker logs
tells you why the system feels slow. `docker stats` shows a container at 22GB and
climbing, but Docker itself does nothing about it.

The fix: hard per-container memory ceilings in `WATCHDOG_CONTAINERS`. When a container
exceeds its limit the watchdog restarts it immediately — no strikes, no waiting. A
memory leak is not a transient spike. Immediate action is correct.

---

### 🔴  Containers That Look Running But Aren't Responding

Docker reports a container as `Up 14 days` while its application layer has been silently
frozen for hours. The reverse proxy happily forwards traffic to a service that returns
nothing. Users see a broken page. Docker sees a healthy container. The container process
is technically running — it just isn't doing anything.

Docker's built-in health checks require a `HEALTHCHECK` instruction in the image itself,
which most self-hosted images don't have. Even those that do often check something too
shallow — a process exists, not whether it's actually serving requests.

The fix: HTTP health checks on the actual service port. `curl` to the real endpoint on
every watchdog cycle. If the service doesn't respond within `CURL_TIMEOUT` seconds, that's
a strike. Two consecutive failures trigger a restart. The distinction between "process
running" and "service responding" is the distinction that matters.

---

### 🔴  Dependency Failures on Restart

Authelia connects to MariaDB at startup. If both are down simultaneously — say, after a
power cut — and the watchdog restarts Authelia first, Authelia fails to connect, exits
immediately, and goes into a crash loop. Meanwhile MariaDB is coming up fine in the
background. The watchdog sees Authelia crash three times, adds it to the skip list,
and sends a critical notification. Authelia was never broken. It just came up in the
wrong order, failed at startup, and got punished for it.

The fix: dependency ordering. If a container's dependency is also down, the dependent is
skipped entirely this cycle. The dependency is restarted first. On the next cycle — once
MariaDB is actually accepting connections — Authelia is restarted and comes up cleanly.
The skip list is never involved. No false alarms. No manual recovery needed.

---

### 🔴  Restart Loops Corrupting State

Some containers corrupt their internal state if restarted repeatedly in rapid succession.
SQLite databases that don't get a clean shutdown write incomplete transactions. Partially
applied database migrations leave schema in an inconsistent state. A container that
crashes on startup after a bad migration gets restarted immediately, crashes again,
gets restarted again — each restart has a chance of making the database worse, not better.

A naive watchdog that just keeps hammering a crashed container is actively harmful in
this scenario. More restarts mean more corruption risk. The right response when restarts
aren't working is to stop restarting and alert the operator.

The fix: restart loop protection. After `WATCHDOG_CONTAINER_RESTART_LIMIT` restarts
within a rolling `WATCHDOG_CONTAINER_RESTART_WINDOW` hour window, the container goes
on the skip list. A critical notification goes out. The watchdog stops touching it.
The operator investigates and clears the skip list once the underlying problem is fixed.

---

### 🔴  Slow Degradation That Never Becomes a Failure

NginxProxyManager accumulates stale entries in its connection table over weeks of uptime.
Dispatcharr's Live TV scheduler builds up internal scheduling state that makes decisions
progressively slower after months of continuous operation. These containers never crash.
They never throw errors. They just get progressively worse in ways that are hard to
attribute to anything specific — until someone notices that the proxy feels slower than
it used to, or that Live TV channel changes take longer than they should.

The fix: scheduled restarts. Not because something is broken, but because some containers
simply perform better with a clean start. Daily at 1am for connection-heavy services.
Weekly for less-critical services that run fine for weeks but benefit from a clean slate.
Zero user impact — happens while everyone is asleep.

---

### 🔴  Network Configuration Lost After Updates

unRAID occasionally wipes custom Docker networks after updates — particularly networks
created by Docker Compose stacks or the NextCloud AIO container. Any container that
depended on those networks for internal communication suddenly can't reach its peers.
memcached can't talk to NextCloud. CrowdSec can't talk to NginxProxyManager. Services
appear to be up but silently fail to communicate with each other.

The fix: network recreation at every array start. `docker_network_connect.sh` checks
every configured network on startup, creates any that are missing, and connects all
configured containers to them. Idempotent — if everything is already correct, it does
nothing and produces no output. If a network had to be created, it notifies — that only
happens after an update, and you want to know when it does.

---

### 🔴  No Visibility Into What the Watchdog Already Tried

A container keeps appearing in a broken state. You SSH in and see it's stopped. You
don't know if the watchdog tried to restart it and failed, gave up and skip-listed it,
is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip
list file on `/boot/config/`, check the restart history file, check the state file —
none of which have obvious formats.

The fix: `watchdog_skip_list_manager.sh` (in `Tools/`). One command to see exactly what's
on the skip list, which containers are stopped vs running, how many restarts were attempted,
and what the watchdog's current state is. One command to clear a specific container and
its history after you've fixed the problem. No manual file editing required.

---

## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
Docker_Essentials/   ← acts on containers          (this folder)
System_Essentials/   ← acts on the server itself
Watchdogs/           ← reactive monitoring + last-resort stability
Monitors/            ← observes, measures, reports
Rsync/               ← moves data between servers
```

> `docker_watchdog.sh` has moved to `Watchdogs/`. Container healing, memory limits,
> HTTP health checks, and skip list management are documented in
> `Watchdogs/README-Watchdogs.md` and `Watchdogs/Manual-Watchdogs.md`.

Four distinct responsibilities in this folder, each handled by dedicated scripts:

---

### ♻️  Proactive Freshness — `docker_daily_restart.sh` + `docker_weekly_restart.sh`

Scheduled restarts that prevent slow degradation before it becomes visible. Not because
something broke — because some containers simply work better after a clean start.

Called by the maintenance orchestrators (`daily_sync_maintenance.sh` and
`weekly_sync_maintenance.sh`) — not run standalone. They run inside the maintenance
windows so any downtime from restarts is absorbed by the window that's already happening.

---

### 🔄  Image Currency — `docker_update.sh` + `docker_update_remaining.sh`

Keeps all container images current without manual intervention. Daily updates for the
auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
remainder pass for everything else — derives the target list automatically from `docker ps`
minus what was already updated, so there is no second list to maintain.

---

### 🌐  Network Integrity — `docker_network_connect.sh`

Ensures custom Docker networks exist and containers are connected to them at every
array start. Silent when everything is correct. Notifies when it has to create something
— which means something was wiped and you should know about it.

---

### 🧹  Downloader Hygiene — `downloaders_reset.sh`

Maintenance reset for all download clients (slskd, SABnzbd, qBittorrent) every 30
minutes. Clears stuck searches, dead transfers, failed imports, and stale queue entries
that download clients accumulate but never clean up themselves. Never touches active
or in-progress downloads.

---

## ━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

`docker_watchdog.sh` has moved to `Watchdogs/` and is now one of four coordinated
single-pass scripts called every 15 minutes by `Orchestrators/watchdog_orchestrator.sh`.

```
Watchdogs/resource_watchdog.sh  ← reduces pressure before healing attempts
Watchdogs/docker_watchdog.sh    ← heals containers (reads resource_watchdog state)
Watchdogs/System/storage_watchdog.sh   ← pool growth + runaway log detection
Watchdogs/stability_watchdog.sh  ← last resort — reboots when healing has failed
```

Scripts in this folder (daily restart, updates, network connect) are unaffected —
they run on their own schedules via the maintenance orchestrators and are not part
of the every-minute watchdog cycle. See `Watchdogs/README-Watchdogs.md` for the
full coordination model between all four watchdogs.

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

| Script | Role | When It Runs |
|--------|------|-------------|
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window |
| `docker_update_remaining.sh` | Image update + prune for all containers not in managed lists | End of weekly maintenance window |
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |

> `watchdog_skip_list_manager.sh` — manual recovery tool for the skip list. Lives in `Tools/` because it's an operator utility, not a lifecycle script.

---

## ━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
Array starts
    │
    └── docker_network_connect.sh ────── run once at start
            ensure networks + connections exist
            silent if correct, notify if creating

Every 15 minutes (Orchestrators/watchdog_orchestrator.sh):
    └── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md

Daily maintenance window (1am):
    daily_sync_maintenance.sh
        ├── docker_update.sh ──────────── pull latest images (DAILY_RESTART_CONTAINERS)
        └── docker_daily_restart.sh ───── restart connection-heavy services

Weekly maintenance window (2:30am Sunday):
    weekly_sync_maintenance.sh
        ├── docker_weekly_restart.sh ──── restart less-critical services
        ├── docker_update.sh --remainder ─ update containers not in managed lists
        └── docker_update_remaining.sh ── prune dangling images

Critical maintenance window (every 30min):
    critical_sync_maintenance.sh
        └── downloaders_reset.sh ──────── clear stuck downloads / stale queue entries

Array stopping:
    array_stopping.sh
        └── docker_container_stop.sh ──── ordered graceful shutdown
```