Varaverk/Docker_Essentials/README-Docker_Essentials.md

# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🐳  DOCKER ESSENTIALS
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Self-healing container lifecycle management for a 50+ container unRAID stack.**
Health monitoring that catches problems as they happen. Scheduled restarts that prevent
degradation before it becomes visible. Network configuration that survives reboots and
unRAID updates. Container image updates woven into the maintenance windows. Recovery
tooling for when something genuinely breaks.

> **Why this folder exists:** Docker on unRAID does not heal itself. A container that
> crashes stays crashed. A memory leak accumulates silently for days until the system
> starts swapping. A container whose dependency restarted first fails in a loop while
> the dependency comes up fine five seconds later. None of this surfaces clearly — it
> just builds into a system that feels flaky without a clear reason why. These scripts
> are the answer to all of that.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Running a large Docker stack on unRAID is genuinely powerful — but Docker's own tooling
gives you almost nothing between "container is running" and "container has been dead for
three days and you just noticed."

The built-in restart policies help with outright crashes, but they have zero visibility
into memory leaks, frozen application layers, dependency ordering, restart loops, or the
difference between a container that crashed and one you intentionally stopped. They give
you no way to know *why* something keeps restarting — just that it does.

These are the specific problems that led to building this, in roughly the order they were
encountered:

---

### 🔴  Memory Leaks Accumulating Silently

Emby's transcode session handling occasionally leaks memory — a session ends but its
memory allocation doesn't fully release. SABnzbd's Python process slowly expands over
days as it processes downloads. Without hard memory limits and automatic enforcement,
these don't fail dramatically — they just consume more and more RAM until the system
starts swapping and everything slows to a crawl. By then, nothing in the Docker logs
tells you why the system feels slow. `docker stats` shows a container at 22GB and
climbing, but Docker itself does nothing about it.

The fix: hard per-container memory ceilings in `WATCHDOG_CONTAINERS`. When a container
exceeds its limit the watchdog restarts it immediately — no strikes, no waiting. A
memory leak is not a transient spike. Immediate action is correct.

---

### 🔴  Containers That Look Running But Aren't Responding

Docker reports a container as `Up 14 days` while its application layer has been silently
frozen for hours. The reverse proxy happily forwards traffic to a service that returns
nothing. Users see a broken page. Docker sees a healthy container. The container process
is technically running — it just isn't doing anything.

Docker's built-in health checks require a `HEALTHCHECK` instruction in the image itself,
which most self-hosted images don't have. Even those that do often check something too
shallow — a process exists, not whether it's actually serving requests.

The fix: HTTP health checks on the actual service port. `curl` to the real endpoint on
every watchdog cycle. If the service doesn't respond within `CURL_TIMEOUT` seconds, that's
a strike. Two consecutive failures trigger a restart. The distinction between "process
running" and "service responding" is the distinction that matters.

---

### 🔴  Dependency Failures on Restart

Authelia connects to MariaDB at startup. If both are down simultaneously — say, after a
power cut — and the watchdog restarts Authelia first, Authelia fails to connect, exits
immediately, and goes into a crash loop. Meanwhile MariaDB is coming up fine in the
background. The watchdog sees Authelia crash three times, adds it to the skip list,
and sends a critical notification. Authelia was never broken. It just came up in the
wrong order, failed at startup, and got punished for it.

The fix: dependency ordering. If a container's dependency is also down, the dependent is
skipped entirely this cycle. The dependency is restarted first. On the next cycle — once
MariaDB is actually accepting connections — Authelia is restarted and comes up cleanly.
The skip list is never involved. No false alarms. No manual recovery needed.

---

### 🔴  Restart Loops Corrupting State

Some containers corrupt their internal state if restarted repeatedly in rapid succession.
SQLite databases that don't get a clean shutdown write incomplete transactions. Partially
applied database migrations leave schema in an inconsistent state. A container that
crashes on startup after a bad migration gets restarted immediately, crashes again,
gets restarted again — each restart has a chance of making the database worse, not better.

A naive watchdog that just keeps hammering a crashed container is actively harmful in
this scenario. More restarts mean more corruption risk. The right response when restarts
aren't working is to stop restarting and alert the operator.

The fix: restart loop protection. After `WATCHDOG_CONTAINER_RESTART_LIMIT` restarts
within a rolling `WATCHDOG_CONTAINER_RESTART_WINDOW` hour window, the container goes
on the skip list. A critical notification goes out. The watchdog stops touching it.
The operator investigates and clears the skip list once the underlying problem is fixed.

---

### 🔴  Slow Degradation That Never Becomes a Failure

NginxProxyManager accumulates stale entries in its connection table over weeks of uptime.
Dispatcharr's Live TV scheduler builds up internal scheduling state that makes decisions
progressively slower after months of continuous operation. These containers never crash.
They never throw errors. They just get progressively worse in ways that are hard to
attribute to anything specific — until someone notices that the proxy feels slower than
it used to, or that Live TV channel changes take longer than they should.

The fix: scheduled restarts. Not because something is broken, but because some containers
simply perform better with a clean start. Daily at 1am for connection-heavy services.
Weekly for less-critical services that run fine for weeks but benefit from a clean slate.
Zero user impact — happens while everyone is asleep.

---

### 🔴  Network Configuration Lost After Updates

unRAID occasionally wipes custom Docker networks after updates — particularly networks
created by Docker Compose stacks or the NextCloud AIO container. Any container that
depended on those networks for internal communication suddenly can't reach its peers.
memcached can't talk to NextCloud. CrowdSec can't talk to NginxProxyManager. Services
appear to be up but silently fail to communicate with each other.

The fix: network recreation at every array start. `docker_network_connect.sh` checks
every configured network on startup, creates any that are missing, and connects all
configured containers to them. Idempotent — if everything is already correct, it does
nothing and produces no output. If a network had to be created, it notifies — that only
happens after an update, and you want to know when it does.

---

### 🔴  No Visibility Into What the Watchdog Already Tried

A container keeps appearing in a broken state. You SSH in and see it's stopped. You
don't know if the watchdog tried to restart it and failed, gave up and skip-listed it,
is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip
list file in `$STATE_DIR`, check the restart history file, check the state file —
none of which have obvious formats.

The fix: `watchdog_skip_list_manager.sh` (in `Tools/`). One command to see exactly what's
on the skip list, which containers are stopped vs running, how many restarts were attempted,
and what the watchdog's current state is. One command to clear a specific container and
its history after you've fixed the problem. No manual file editing required.

---

## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
Docker_Essentials/   ← acts on containers          (this folder)
System_Essentials/   ← acts on the server itself
Watchdogs/           ← reactive monitoring + last-resort stability
Monitors/            ← observes, measures, reports
Rsync/               ← moves data between servers
```

> `docker_watchdog.sh` has moved to `Watchdogs/`. Container healing, memory limits,
> HTTP health checks, and skip list management are documented in
> `Watchdogs/README-Watchdogs.md` and `Watchdogs/Manual-Watchdogs.md`.

Four distinct responsibilities in this folder, each handled by dedicated scripts:

---

### ♻️  Proactive Freshness — `docker_daily_restart.sh` + `docker_weekly_restart.sh`

Scheduled restarts that prevent slow degradation before it becomes visible. Not because
something broke — because some containers simply work better after a clean start.

Called by the maintenance orchestrators (`daily_sync_maintenance.sh` and
`weekly_sync_maintenance.sh`) — not run standalone. They run inside the maintenance
windows so any downtime from restarts is absorbed by the window that's already happening.

---

### 🔄  Image Currency — `docker_update.sh`

One script, three modes — there is no separate remainder script.

| Mode | Targets | Runs |
|------|---------|------|
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, **before** `docker_daily_restart.sh` |
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, **before** `docker_weekly_restart.sh` |
| `--remainder` | Everything running that is in neither list | Monthly, via `monthly_maintenance.sh` |

The update always runs *before* its matching restart, so the restart lands on the freshly
pulled image. Reversing that order would restart onto the old image and leave the new one
sitting unused until the next window.

Each mode reuses the restart list it pairs with rather than maintaining its own — add a
container to `DAILY_RESTART_CONTAINERS` once and it gets both the restart and the image pull.
Remainder mode needs no list at all: it derives its targets from `docker ps` minus the daily
list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers.

Fallback containers are deliberately excluded from remainder mode. This server only runs them
during a fallback; the remote owns their version. Updating them here would risk the remote's
older image meeting data written by a newer one after a handback.

---

### 🌐  Network Integrity — `docker_network_connect.sh`

Ensures custom Docker networks exist and containers are connected to them at every
array start. Silent when everything is correct. Notifies when it has to create something
— which means something was wiped and you should know about it.

---

### 🧹  Downloader Hygiene — `downloaders_reset.sh`

Maintenance reset for all download clients (slskd, SABnzbd, qBittorrent) every 30
minutes. Clears stuck searches, dead transfers, failed imports, and stale queue entries
that download clients accumulate but never clean up themselves. Never touches active
or in-progress downloads.

---

## ━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

`docker_watchdog.sh` has moved to `Watchdogs/` and is now one of four coordinated
single-pass scripts called every 15 minutes by `Orchestrators/watchdog_orchestrator.sh`.

```
Watchdogs/resource_watchdog.sh  ← reduces pressure before healing attempts
Watchdogs/docker_watchdog.sh    ← heals containers (reads resource_watchdog state)
Watchdogs/System/storage_watchdog.sh   ← pool growth + runaway log detection
Watchdogs/stability_watchdog.sh  ← last resort — reboots when healing has failed
```

Scripts in this folder (daily restart, updates, network connect) are unaffected —
they run on their own schedules via the maintenance orchestrators and are not part
of the every-minute watchdog cycle. See `Watchdogs/README-Watchdogs.md` for the
full coordination model between all four watchdogs.

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

| Script | Role | When It Runs |
|--------|------|-------------|
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
| `docker_update.sh` | Container image updates — three modes (default / `--weekly` / `--remainder`) | Daily and weekly before each restart; monthly for the remainder |
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |

> `watchdog_skip_list_manager.sh` — manual recovery tool for the skip list. Lives in `Tools/` because it's an operator utility, not a lifecycle script.

---

## ━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
Array starts
    │
    └── docker_network_connect.sh ────── run once at start
            ensure networks + connections exist
            silent if correct, notify if creating

Every 15 minutes (Orchestrators/watchdog_orchestrator.sh):
    └── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md

Daily maintenance window (1am):
    daily_sync_maintenance.sh
        ├── docker_update.sh ──────────── pull latest images (DAILY_RESTART_CONTAINERS)
        └── docker_daily_restart.sh ───── restart connection-heavy services

Weekly maintenance window (2:30am Sunday):
    weekly_sync_maintenance.sh
        ├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS)
        └── docker_weekly_restart.sh ──── restart onto the fresh image

Monthly maintenance window:
    monthly_maintenance.sh
        ├── docker_update.sh --remainder  update everything not in the managed lists
        └── Tools/docker_prune_images.sh --all

Critical maintenance window (every 30min):
    critical_sync_maintenance.sh
        └── downloaders_reset.sh ──────── clear stuck downloads / stale queue entries

Array stopping:
    array_stopping.sh
        └── docker_container_stop.sh ──── ordered graceful shutdown, verified per container
```

---

## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Every script here talks to the Docker daemon, so they share the same protections. Each
script's own header documents its full set; these are the ones worth knowing folder-wide.

**Daemon health is checked, not assumed.** A hung daemon returns an empty container list,
which is indistinguishable from "no containers running". Without the check,
`docker_container_stop.sh` would report a clean shutdown that never happened, and
`docker_update.sh --remainder` would report "nothing to update" while doing nothing.

**Every docker call is timeout-wrapped.** A wedged daemon cannot stall a maintenance window
or hold a lock open. The one deliberate exception is `docker pull` — a large image
legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer.

**State is respected.** Running containers get restarted; stopped ones stay stopped. A
stopped container was almost certainly stopped on purpose, and none of these scripts has the
authority to overrule that.

**Restarts are verified, not assumed.** After each restart the container is re-checked once
it has had time to settle. A container that starts and immediately crashes is recorded as a
failure and notified — a restart that did not stick is never reported as success.

**Dependency ordering is shared with the watchdog.** Restarts follow
`HOST*_WATCHDOG_DEPENDENCIES`, with `CONTAINER_DELAY` between a dependency and its dependents,
so a dependent is never brought up while what it needs is still initialising.

**Locks prevent overlap.** Long windows can outlast their interval;
`downloaders_reset.sh` uses wait-mode because it runs every 30 minutes and the previous pass
may still be finishing.

---