Varaverk/Docker_Essentials/README-Docker_Essentials.md

# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🐳  DOCKER ESSENTIALS
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Self-healing container lifecycle management for a 50+ container unRAID stack.**
Health monitoring that catches problems as they happen. Scheduled restarts that prevent
degradation before it becomes visible. Network configuration that survives reboots and
unRAID updates. Container image updates woven into the maintenance windows. Recovery
tooling for when something genuinely breaks.

> **Why this folder exists:** Docker on unRAID does not heal itself. A container that
> crashes stays crashed. A memory leak accumulates silently for days until the system
> starts swapping. A container whose dependency restarted first fails in a loop while
> the dependency comes up fine five seconds later. None of this surfaces clearly — it
> just builds into a system that feels flaky without a clear reason why. These scripts
> are the answer to all of that.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Running a large Docker stack on unRAID is genuinely powerful — but Docker's own tooling
gives you almost nothing between "container is running" and "container has been dead for
three days and you just noticed."

The built-in restart policies help with outright crashes, but they have zero visibility
into memory leaks, frozen application layers, dependency ordering, restart loops, or the
difference between a container that crashed and one you intentionally stopped. They give
you no way to know *why* something keeps restarting — just that it does.

These are the specific problems that led to building this, in roughly the order they were
encountered:

---

### 🔴  Memory Leaks Accumulating Silently

Emby's transcode session handling occasionally leaks memory — a session ends but its
memory allocation doesn't fully release. SABnzbd's Python process slowly expands over
days as it processes downloads. Without hard memory limits and automatic enforcement,
these don't fail dramatically — they just consume more and more RAM until the system
starts swapping and everything slows to a crawl. By then, nothing in the Docker logs
tells you why the system feels slow. `docker stats` shows a container at 22GB and
climbing, but Docker itself does nothing about it.

The fix: hard per-container memory ceilings in `WATCHDOG_CONTAINERS`. When a container
exceeds its limit the watchdog restarts it immediately — no strikes, no waiting. A
memory leak is not a transient spike. Immediate action is correct.

---

### 🔴  Containers That Look Running But Aren't Responding

Docker reports a container as `Up 14 days` while its application layer has been silently
frozen for hours. The reverse proxy happily forwards traffic to a service that returns
nothing. Users see a broken page. Docker sees a healthy container. The container process
is technically running — it just isn't doing anything.

Docker's built-in health checks require a `HEALTHCHECK` instruction in the image itself,
which most self-hosted images don't have. Even those that do often check something too
shallow — a process exists, not whether it's actually serving requests.

The fix: HTTP health checks on the actual service port. `curl` to the real endpoint on
every watchdog cycle. If the service doesn't respond within `CURL_TIMEOUT` seconds, that's
a strike. Two consecutive failures trigger a restart. The distinction between "process
running" and "service responding" is the distinction that matters.

---

### 🔴  Dependency Failures on Restart

Authelia connects to MariaDB at startup. If both are down simultaneously — say, after a
power cut — and the watchdog restarts Authelia first, Authelia fails to connect, exits
immediately, and goes into a crash loop. Meanwhile MariaDB is coming up fine in the
background. The watchdog sees Authelia crash three times, adds it to the skip list,
and sends a critical notification. Authelia was never broken. It just came up in the
wrong order, failed at startup, and got punished for it.

The fix: dependency ordering. If a container's dependency is also down, the dependent is
skipped entirely this cycle. The dependency is restarted first. On the next cycle — once
MariaDB is actually accepting connections — Authelia is restarted and comes up cleanly.
The skip list is never involved. No false alarms. No manual recovery needed.

---

### 🔴  Restart Loops Corrupting State

Some containers corrupt their internal state if restarted repeatedly in rapid succession.
SQLite databases that don't get a clean shutdown write incomplete transactions. Partially
applied database migrations leave schema in an inconsistent state. A container that
crashes on startup after a bad migration gets restarted immediately, crashes again,
gets restarted again — each restart has a chance of making the database worse, not better.

A naive watchdog that just keeps hammering a crashed container is actively harmful in
this scenario. More restarts mean more corruption risk. The right response when restarts
aren't working is to stop restarting and alert the operator.

The fix: restart loop protection. After `WATCHDOG_CONTAINER_RESTART_LIMIT` restarts
within a rolling `WATCHDOG_CONTAINER_RESTART_WINDOW` hour window, the container goes
on the skip list. A critical notification goes out. The watchdog stops touching it.
The operator investigates and clears the skip list once the underlying problem is fixed.

---

### 🔴  Slow Degradation That Never Becomes a Failure

NginxProxyManager accumulates stale entries in its connection table over weeks of uptime.
Dispatcharr's Live TV scheduler builds up internal scheduling state that makes decisions
progressively slower after months of continuous operation. These containers never crash.
They never throw errors. They just get progressively worse in ways that are hard to
attribute to anything specific — until someone notices that the proxy feels slower than
it used to, or that Live TV channel changes take longer than they should.

The fix: scheduled restarts. Not because something is broken, but because some containers
simply perform better with a clean start. Daily at 1am for connection-heavy services.
Weekly for less-critical services that run fine for weeks but benefit from a clean slate.
Zero user impact — happens while everyone is asleep.

---

### 🔴  Network Configuration Lost After Updates

unRAID occasionally wipes custom Docker networks after updates — particularly networks
created by Docker Compose stacks or the NextCloud AIO container. Any container that
depended on those networks for internal communication suddenly can't reach its peers.
memcached can't talk to NextCloud. CrowdSec can't talk to NginxProxyManager. Services
appear to be up but silently fail to communicate with each other.

The fix: network recreation at every array start. `docker_network_connect.sh` checks
every configured network on startup, creates any that are missing, and connects all
configured containers to them. Idempotent — if everything is already correct, it does
nothing and produces no output. If a network had to be created, it notifies — that only
happens after an update, and you want to know when it does.

---

### 🔴  No Visibility Into What the Watchdog Already Tried

A container keeps appearing in a broken state. You SSH in and see it's stopped. You
don't know if the watchdog tried to restart it and failed, gave up and skip-listed it,
is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip
list file on `/boot/config/`, check the restart history file, check the state file —
none of which have obvious formats.

The fix: `watchdog_skip_list_manager.sh` (in `Tools/`). One command to see exactly what's
on the skip list, which containers are stopped vs running, how many restarts were attempted,
and what the watchdog's current state is. One command to clear a specific container and
its history after you've fixed the problem. No manual file editing required.

---

## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
Docker_Essentials/   ← acts on containers          (this folder)
unRAID_Essentials/   ← acts on the server itself
Monitors/            ← observes, measures, reports
Rsync/               ← moves data between servers
```

Five distinct responsibilities, each handled by dedicated scripts:

---

### 🔁  Reactive Healing — `docker_watchdog.sh`

Continuous two-tier monitoring that catches problems as they happen and acts on them
immediately. Runs as a background process started at array start. Every 15 minutes it
checks the full stack and fixes what it can — silently when everything is fine, visibly
when something needs attention.

Two tiers because different containers need different monitoring strategies:
- **Tier 1** — explicit per-container configuration with specific thresholds
- **Tier 2** — global scan of everything that's running with catch-all health checks

---

### ♻️  Proactive Freshness — `docker_daily_restart.sh` + `docker_weekly_restart.sh`

Scheduled restarts that prevent slow degradation before it becomes visible. Not because
something broke — because some containers simply work better after a clean start.

Called by the maintenance orchestrators (`daily_sync_maintenance.sh` and
`weekly_sync_maintenance.sh`) — not run standalone. They run inside the maintenance
windows so any downtime from restarts is absorbed by the window that's already happening.

---

### 🔄  Image Currency — `docker_update.sh` + `docker_update_remaining.sh`

Keeps all container images current without manual intervention. Daily updates for the
auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
remainder pass for everything else — derives the target list automatically from `docker ps`
minus what was already updated, so there is no second list to maintain.

---

### 🌐  Network Integrity — `docker_network_connect.sh`

Ensures custom Docker networks exist and containers are connected to them at every
array start. Silent when everything is correct. Notifies when it has to create something
— which means something was wiped and you should know about it.

---

### 🧹  Downloader Hygiene — `downloaders_reset.sh`

Maintenance reset for all download clients (slskd, SABnzbd, qBittorrent) every 15
minutes. Clears stuck searches, dead transfers, failed imports, and stale queue entries
that download clients accumulate but never clean up themselves. Never touches active
or in-progress downloads.

---

## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Two watchdogs run simultaneously. They are designed to work together, not compete:

```
system_watchdog.sh   ← watches the server:     RAM, CPU, disk, kernel, daemon health
docker_watchdog.sh   ← watches the containers: memory, CPU, HTTP response, crashes
```

**The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops
non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh`
would see stopped containers on its next cycle and restart them — directly undoing the
RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover.
The system would eventually hit the reboot threshold anyway, having accomplished nothing.

**The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When
`system_watchdog.sh` triggers a RAM emergency shutdown it writes
`mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every
cycle and defers all container restart logic until it clears. Health URL checks for
excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still
run. Everything else stands down.

**The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state
file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh`
checks how long ago the state file was modified. If it's more than 2 hours old while
`mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running.
`docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced
indefinitely by a stale flag from a process that's no longer running.

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

| Script | Role | When It Runs |
|--------|------|-------------|
| `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop — array start |
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window |
| `docker_update_remaining.sh` | Image update + prune for all containers not in managed lists | End of weekly maintenance window |
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 15min via `critical_sync_maintenance.sh` |

> `watchdog_skip_list_manager.sh` — manual recovery tool for the skip list. Lives in `Tools/` because it's an operator utility, not a lifecycle script.

---

## ━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
Array starts
    │
    ├── docker_network_connect.sh ────── run once at start
    │       ensure networks + connections exist
    │       silent if correct, notify if creating
    │
    └── docker_watchdog.sh ──────────── continuous background loop (every 15min)
            Tier 1: memory, CPU, HTTP, required containers
            Tier 2: global unhealthy / OOM / crash / dead scan
            reads system_watchdog state (RAM emergency deferral)
            │
            │  (on skip list event → operator uses)
            └── Tools/watchdog_skip_list_manager.sh
                    inspect state, clear after fixing root cause

Daily maintenance window (1am):
    daily_sync_maintenance.sh
        ├── docker_update.sh ──────────── pull latest images (DAILY_RESTART_CONTAINERS)
        └── docker_daily_restart.sh ───── restart connection-heavy services

Weekly maintenance window (2:30am Sunday):
    weekly_sync_maintenance.sh
        ├── docker_weekly_restart.sh ──── restart less-critical services
        ├── docker_update.sh --remainder ─ update containers not in managed lists
        └── docker_update_remaining.sh ── prune dangling images

Critical maintenance window (every 15min):
    critical_sync_maintenance.sh
        └── downloaders_reset.sh ──────── clear stuck downloads / stale queue entries

Array stopping:
    array_stopping.sh
        └── docker_container_stop.sh ──── ordered graceful shutdown
```