1211 lines
65 KiB
Markdown
1211 lines
65 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 🐳 DOCKER ESSENTIALS
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Self-healing container lifecycle management for a 50+ container unRAID stack.**
|
|
Health monitoring that catches problems as they happen. Scheduled restarts that prevent
|
|
degradation before it becomes visible. Network configuration that survives reboots and
|
|
unRAID updates. Recovery tooling for when something genuinely breaks.
|
|
|
|
> **Why this folder exists:** Docker on unRAID does not heal itself. A container that
|
|
> crashes stays crashed. A memory leak accumulates silently for days until the system
|
|
> starts swapping. A container whose dependency restarted first fails in a loop while
|
|
> the dependency comes up fine five seconds later. None of this surfaces clearly — it
|
|
> just builds into a system that feels flaky without a clear reason why. These scripts
|
|
> are the answer to all of that.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Running a large Docker stack on unRAID is genuinely powerful — but Docker's own tooling
|
|
gives you almost nothing between "container is running" and "container has been dead for
|
|
three days and you just noticed."
|
|
|
|
The built-in restart policies help with outright crashes, but they have zero visibility
|
|
into memory leaks, frozen application layers, dependency ordering, restart loops, or the
|
|
difference between a container that crashed and one you intentionally stopped. They give
|
|
you no way to know *why* something keeps restarting — just that it does.
|
|
|
|
These are the specific problems that led to building this, in roughly the order they were
|
|
encountered:
|
|
|
|
---
|
|
|
|
### 🔴 Memory Leaks Accumulating Silently
|
|
|
|
Emby's transcode session handling occasionally leaks memory — a session ends but its
|
|
memory allocation doesn't fully release. SABnzbd's Python process slowly expands over
|
|
days as it processes downloads. Without hard memory limits and automatic enforcement,
|
|
these don't fail dramatically — they just consume more and more RAM until the system
|
|
starts swapping and everything slows to a crawl. By then, nothing in the Docker logs
|
|
tells you why the system feels slow. `docker stats` shows a container at 22GB and
|
|
climbing, but Docker itself does nothing about it.
|
|
|
|
The fix: hard per-container memory ceilings in `WATCHDOG_CONTAINERS`. When a container
|
|
exceeds its limit the watchdog restarts it immediately — no strikes, no waiting. A
|
|
memory leak is not a transient spike. Immediate action is correct.
|
|
|
|
---
|
|
|
|
### 🔴 Containers That Look Running But Aren't Responding
|
|
|
|
Docker reports a container as `Up 14 days` while its application layer has been silently
|
|
frozen for hours. The reverse proxy happily forwards traffic to a service that returns
|
|
nothing. Users see a broken page. Docker sees a healthy container. The container process
|
|
is technically running — it just isn't doing anything.
|
|
|
|
Docker's built-in health checks require a `HEALTHCHECK` instruction in the image itself,
|
|
which most self-hosted images don't have. Even those that do often check something too
|
|
shallow — a process exists, not whether it's actually serving requests.
|
|
|
|
The fix: HTTP health checks on the actual service port. `curl` to the real endpoint on
|
|
every watchdog cycle. If the service doesn't respond within `CURL_TIMEOUT` seconds, that's
|
|
a strike. Two consecutive failures trigger a restart. The distinction between "process
|
|
running" and "service responding" is the distinction that matters.
|
|
|
|
---
|
|
|
|
### 🔴 Dependency Failures on Restart
|
|
|
|
Authelia connects to MariaDB at startup. If both are down simultaneously — say, after a
|
|
power cut — and the watchdog restarts Authelia first, Authelia fails to connect, exits
|
|
immediately, and goes into a crash loop. Meanwhile MariaDB is coming up fine in the
|
|
background. The watchdog sees Authelia crash three times, adds it to the skip list,
|
|
and sends a critical notification. Authelia was never broken. It just came up in the
|
|
wrong order, failed at startup, and got punished for it.
|
|
|
|
The fix: dependency ordering. If a container's dependency is also down, the dependent is
|
|
skipped entirely this cycle. The dependency is restarted first. On the next cycle — once
|
|
MariaDB is actually accepting connections — Authelia is restarted and comes up cleanly.
|
|
The skip list is never involved. No false alarms. No manual recovery needed.
|
|
|
|
---
|
|
|
|
### 🔴 Restart Loops Corrupting State
|
|
|
|
Some containers corrupt their internal state if restarted repeatedly in rapid succession.
|
|
SQLite databases that don't get a clean shutdown write incomplete transactions. Partially
|
|
applied database migrations leave schema in an inconsistent state. A container that
|
|
crashes on startup after a bad migration gets restarted immediately, crashes again,
|
|
gets restarted again — each restart has a chance of making the database worse, not better.
|
|
|
|
A naive watchdog that just keeps hammering a crashed container is actively harmful in
|
|
this scenario. More restarts mean more corruption risk. The right response when restarts
|
|
aren't working is to stop restarting and alert the operator.
|
|
|
|
The fix: restart loop protection. After `WATCHDOG_CONTAINER_RESTART_LIMIT` restarts
|
|
within a rolling `WATCHDOG_CONTAINER_RESTART_WINDOW` hour window, the container goes
|
|
on the skip list. A critical notification goes out. The watchdog stops touching it.
|
|
The operator investigates and clears the skip list once the underlying problem is fixed.
|
|
|
|
---
|
|
|
|
### 🔴 Slow Degradation That Never Becomes a Failure
|
|
|
|
NginxProxyManager accumulates stale entries in its connection table over weeks of uptime.
|
|
Dispatcharr's Live TV scheduler builds up internal scheduling state that makes decisions
|
|
progressively slower after months of continuous operation. These containers never crash.
|
|
They never throw errors. They just get progressively worse in ways that are hard to
|
|
attribute to anything specific — until someone notices that the proxy feels slower than
|
|
it used to, or that Live TV channel changes take longer than they should.
|
|
|
|
The fix: scheduled restarts. Not because something is broken, but because some containers
|
|
simply perform better with a clean start. Daily at 1am for connection-heavy services.
|
|
Weekly for less-critical services that run fine for weeks but benefit from a clean slate.
|
|
Zero user impact — happens while everyone is asleep.
|
|
|
|
---
|
|
|
|
### 🔴 Network Configuration Lost After Updates
|
|
|
|
unRAID occasionally wipes custom Docker networks after updates — particularly networks
|
|
created by Docker Compose stacks or the NextCloud AIO container. Any container that
|
|
depended on those networks for internal communication suddenly can't reach its peers.
|
|
memcached can't talk to NextCloud. CrowdSec can't talk to NginxProxyManager. Services
|
|
appear to be up but silently fail to communicate with each other.
|
|
|
|
The fix: network recreation at every array start. `docker_network_connect.sh` checks
|
|
every configured network on startup, creates any that are missing, and connects all
|
|
configured containers to them. Idempotent — if everything is already correct, it does
|
|
nothing and produces no output. If a network had to be created, it notifies — that only
|
|
happens after an update, and you want to know when it does.
|
|
|
|
---
|
|
|
|
### 🔴 No Visibility Into What the Watchdog Already Tried
|
|
|
|
A container keeps appearing in a broken state. You SSH in and see it's stopped. You
|
|
don't know if the watchdog tried to restart it and failed, gave up and skip-listed it,
|
|
is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip
|
|
list file on `/boot/config/`, check the restart history file, check the state file —
|
|
none of which have obvious formats.
|
|
|
|
The fix: `watchdog_skip_list_manager.sh`. One command to see exactly what's on the
|
|
skip list, which containers are stopped vs running, how many restarts were attempted,
|
|
and what the watchdog's current state is. One command to clear a specific container and
|
|
its history after you've fixed the problem. No manual file editing required.
|
|
|
|
---
|
|
|
|
## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```
|
|
Docker_Essentials/ ← acts on containers (this folder)
|
|
unRAID_Essentials/ ← acts on the server itself
|
|
Monitors/ ← observes, measures, reports
|
|
Rsync/ ← moves data between servers
|
|
```
|
|
|
|
Four distinct roles, each handled by dedicated scripts:
|
|
|
|
---
|
|
|
|
### 🔁 Reactive Healing — `docker_watchdog.sh`
|
|
|
|
Continuous two-tier monitoring that catches problems as they happen and acts on them
|
|
immediately. Runs as a background process started at array start. Every 15 minutes it
|
|
checks the full stack and fixes what it can — silently when everything is fine, visibly
|
|
when something needs attention.
|
|
|
|
Two tiers because different containers need different monitoring strategies:
|
|
- **Tier 1** — explicit per-container configuration with specific thresholds
|
|
- **Tier 2** — global scan of everything that's running with catch-all health checks
|
|
|
|
---
|
|
|
|
### ♻️ Proactive Freshness — `docker_daily_restart.sh` + `docker_weekly_restart.sh`
|
|
|
|
Scheduled restarts that prevent slow degradation before it becomes visible. Not because
|
|
something broke — because some containers simply work better after a clean start.
|
|
|
|
Called by the maintenance orchestrators (`daily_sync_maintenance.sh` and
|
|
`weekly_sync_maintenance.sh`) — not run standalone. They run inside the maintenance
|
|
windows so any downtime from restarts is absorbed by the window that's already happening.
|
|
|
|
---
|
|
|
|
### 🌐 Network Integrity — `docker_network_connect.sh`
|
|
|
|
Ensures custom Docker networks exist and containers are connected to them at every
|
|
array start. Silent when everything is correct. Notifies when it has to create something
|
|
— which means something was wiped and you should know about it.
|
|
|
|
---
|
|
|
|
### 🔧 Recovery Tooling — `watchdog_skip_list_manager.sh`
|
|
|
|
Manual tool for the moments when automatic recovery hasn't worked. Gives a clear picture
|
|
of what the watchdog has already tried, lets you clear the skip list cleanly after fixing
|
|
the root cause, and warns you if clearing might immediately re-add the container.
|
|
|
|
---
|
|
|
|
## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Two watchdogs run simultaneously. They are designed to work together, not compete:
|
|
|
|
```
|
|
system_watchdog.sh ← watches the server: RAM, CPU, disk, kernel, daemon health
|
|
docker_watchdog.sh ← watches the containers: memory, CPU, HTTP response, crashes
|
|
```
|
|
|
|
**The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops
|
|
non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh`
|
|
would see stopped containers on its next cycle and restart them — directly undoing the
|
|
RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover.
|
|
The system would eventually hit the reboot threshold anyway, having accomplished nothing.
|
|
|
|
**The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When
|
|
`system_watchdog.sh` triggers a RAM emergency shutdown it writes
|
|
`mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every
|
|
cycle and defers all container restart logic until it clears. Health URL checks for
|
|
excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still
|
|
run. Everything else stands down.
|
|
|
|
**The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state
|
|
file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh`
|
|
checks how long ago the state file was modified. If it's more than 2 hours old while
|
|
`mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running.
|
|
`docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced
|
|
indefinitely by a stale flag from a process that's no longer running.
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
| Script | Role | When It Runs |
|
|
|--------|------|-------------|
|
|
| `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop |
|
|
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
|
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
|
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
|
|
| `watchdog_skip_list_manager.sh` | Skip list inspection and manual recovery | On demand |
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🐳 docker_watchdog.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
The self-healing heart of the ecosystem. Two tiers of monitoring run every 15 minutes
|
|
as a background process. Tier 1 applies specific thresholds to explicitly configured
|
|
containers. Tier 2 scans everything else for generic health problems. Together they
|
|
catch the full range of container failures — from subtle memory leaks to outright crashes.
|
|
|
|
```bash
|
|
# Started automatically at array start via array_start.sh
|
|
# Runs continuously until array stops (SIGTERM → clean shutdown)
|
|
# Interval: DOCKER_WATCHDOG_INTERVAL=900 (15 minutes)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Tier 1 — Strict Per-Container Monitoring ────────────────────────────────
|
|
|
|
Applies **only** to containers you explicitly configure. These are the containers
|
|
you care most about — the ones that affect users when they fail. Configure them
|
|
once in `master_host*.conf` and they are monitored with specific, appropriate
|
|
thresholds forever.
|
|
|
|
---
|
|
|
|
#### 💾 Memory Hard Limits
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Memory hard limits in MB — immediate restart when exceeded, no strike system.
|
|
# Each entry: "ContainerName:LimitInMB"
|
|
#
|
|
# These are NOT soft targets — they are hard ceilings. A container that hits
|
|
# its limit gets restarted immediately on the current cycle. Memory leaks are
|
|
# not brief spikes. Waiting for a second confirmation just allows more leak.
|
|
#
|
|
# How to size limits:
|
|
# Check normal usage: docker stats ContainerName
|
|
# Set limit at ~150-200% of normal peak usage
|
|
# Emby peaks around 8-12GB during heavy transcoding — 18GB gives headroom
|
|
# without triggering on legitimate load spikes
|
|
#
|
|
HOST1_WATCHDOG_CONTAINERS=(
|
|
"Emby:18432" # 18GB — peaks ~12GB under heavy transcode load
|
|
"LidaTube:6144" # 6GB — YouTube downloader, can grow with large queues
|
|
"Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature
|
|
"Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous
|
|
)
|
|
```
|
|
|
|
A soft warning threshold fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit —
|
|
giving early visibility into a container approaching its ceiling before a restart is
|
|
triggered. Useful for catching gradual leaks before they become events.
|
|
|
|
---
|
|
|
|
#### 📊 CPU Thresholds
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# CPU thresholds are normalised against the server's total core count.
|
|
# This makes the threshold meaningful regardless of hardware.
|
|
#
|
|
# Why normalised:
|
|
# A container using 85% of one core on a 16-core machine = 5.3% normalised
|
|
# → a nothing, don't touch it
|
|
# A container using 85% normalised on a 16-core machine = 13.6 cores worth
|
|
# → a runaway process, restart it
|
|
#
|
|
# CPU uses a STRIKE SYSTEM — not immediate restart like memory.
|
|
# Brief CPU spikes are completely normal (Tdarr encoding, Emby transcoding,
|
|
# SABnzbd unpacking). The strike system ignores spikes and acts on sustained usage.
|
|
#
|
|
# Strike 1: CPU above HARD_CPU_THRESHOLD this cycle → warn, increment strike
|
|
# Strike 2: CPU above threshold next cycle → restart, reset strike counter
|
|
# Recovery: CPU drops below threshold any cycle → reset strike counter to 0
|
|
#
|
|
SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only
|
|
HARD_CPU_THRESHOLD=85 # strike at 85% normalised
|
|
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
|
|
```
|
|
|
|
---
|
|
|
|
#### 🌐 HTTP Responsiveness
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# HTTP health checks hit the actual service endpoint on every watchdog cycle.
|
|
# "Container running" and "service responding" are not the same thing.
|
|
#
|
|
# A frozen Emby will respond to nothing but show as Up in Docker.
|
|
# A curl timeout catches this where Docker's own checks never would.
|
|
#
|
|
# Also uses a STRIKE SYSTEM — network hiccups and brief restarts happen.
|
|
# Two consecutive non-responses before acting prevents false positives
|
|
# from momentary connectivity issues.
|
|
#
|
|
# Format: "ContainerName:http://host:port/optional-path"
|
|
# The path can be a lightweight health endpoint or just the root URL
|
|
#
|
|
HOST1_WATCHDOG_CONTAINER_URLS=(
|
|
"Emby:http://localhost:8096" # Emby WebUI root — fast to respond
|
|
"NginxProxyManager:http://localhost:81" # NPM admin interface
|
|
)
|
|
#
|
|
# CURL_TIMEOUT=5 # seconds before a non-response counts as a failure
|
|
# RESP_FAIL_LIMIT=2 # consecutive failures before restart
|
|
```
|
|
|
|
> **Note on Docker HEALTHCHECK:** Docker has its own `HEALTHCHECK` mechanism but it
|
|
> requires the image to define a health check command — most self-hosted images don't.
|
|
> These HTTP checks work regardless of what the image defines. They check what actually
|
|
> matters: does the service respond to a request?
|
|
|
|
---
|
|
|
|
#### ✅ Required Containers
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Containers that must always be running. If found stopped, the watchdog
|
|
# attempts to restart them every cycle until they are running or hit the
|
|
# restart loop limit.
|
|
#
|
|
# These are the containers whose absence breaks everything else:
|
|
# NginxProxyManager — all external traffic routes through this
|
|
# Authelia — authentication for every protected service
|
|
# Mariadb-Authelia — Authelia's database — Authelia cannot start without it
|
|
# Redis-Authelia — Authelia's session store — same dependency
|
|
#
|
|
# Required containers use the STRIKE SYSTEM — one miss might be mid-restart.
|
|
# Persistent failure → skip list → critical notification.
|
|
#
|
|
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
|
"NginxProxyManager" # reverse proxy — external access depends on this
|
|
"Authelia" # SSO authentication — all protected services need it
|
|
"Mariadb-Authelia" # Authelia database — must be up before Authelia starts
|
|
"Redis-Authelia" # Authelia session cache — same startup dependency
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Tier 2 — Global Health Scan ─────────────────────────────────────────────
|
|
|
|
Scans **every running container** when `WATCHDOG_SCAN_ALL=true`. No per-container
|
|
configuration required — this is the catch-all that protects everything not explicitly
|
|
configured in Tier 1.
|
|
|
|
Containers in `WATCHDOG_SCAN_IGNORE` are excluded from Tier 2 but still covered by
|
|
Tier 1 if configured there.
|
|
|
|
---
|
|
|
|
| Check | What Triggers It | What Happens | Why This Matters |
|
|
|-------|-----------------|--------------|-----------------|
|
|
| `WATCHDOG_RESTART_UNHEALTHY` | Docker HEALTHCHECK reports `unhealthy` | Restart | Catches containers with built-in health checks that are failing |
|
|
| `WATCHDOG_NOTIFY_OOM` | Kernel OOM-killed the container | Restart + notify | OOM kills are silent by default — you'd never know without this |
|
|
| `WATCHDOG_NOTIFY_CRASHLOOP` | Docker RestartCount climbing | Notify; above `WATCHDOG_CRASH_LIMIT` → restart → skip list | Distinguishes "just restarted once" from "has crashed 12 times" |
|
|
| `WATCHDOG_RESTART_DEAD` | Container in `dead` state | Remove + start | Dead containers can't be restarted — must be removed first |
|
|
| `WATCHDOG_RESTART_CRASHED` | Non-zero exit code, exited state | Restart | Catches clean-exit crashes that Docker's restart policy misses |
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Tier 2 toggles — disable any check that produces false positives in your
|
|
# environment. Each is independent — disabling one doesn't affect the others.
|
|
#
|
|
WATCHDOG_SCAN_ALL=true # enable Tier 2 global scan
|
|
WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures
|
|
WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills
|
|
WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts
|
|
WATCHDOG_RESTART_DEAD=true # recover containers in dead state
|
|
WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero
|
|
WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list
|
|
|
|
# Containers to exclude from Tier 2 global scan entirely.
|
|
# Use for containers you intentionally stop/start manually, or containers that
|
|
# have benign non-zero exits as part of their normal operation.
|
|
WATCHDOG_SCAN_IGNORE=(
|
|
"my-one-shot-container" # runs and exits normally — don't treat as crash
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Cross-Cutting Intelligence ──────────────────────────────────────────────
|
|
|
|
These mechanisms apply to **both tiers** on every watchdog cycle. They are what
|
|
separates intelligent monitoring from naive restart-on-failure.
|
|
|
|
---
|
|
|
|
#### ⏱️ Startup Grace Period
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# After array start, containers take time to initialise — databases run recovery,
|
|
# services wait for dependencies, Emby scans its library. During this window,
|
|
# a container that isn't responding yet is not broken — it's just starting.
|
|
#
|
|
# Without a grace period, the watchdog fires false-positive restarts in the
|
|
# first minutes after every array start. With it, checks run and log normally
|
|
# but restart actions are suppressed until the grace period expires.
|
|
#
|
|
# The grace period clock starts from when the watchdog process itself starts —
|
|
# not from when the array starts — so it's accurate even if array start
|
|
# takes a few minutes.
|
|
#
|
|
WATCHDOG_STARTUP_GRACE=600 # 10 minutes — restarts suppressed, checks still log
|
|
```
|
|
|
|
---
|
|
|
|
#### 🔗 Dependency Ordering
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# When a container and its dependency are both down, restart the dependency first.
|
|
# Skip the dependent container entirely on this cycle. On the next cycle the
|
|
# dependency should be healthy, and the dependent can restart cleanly.
|
|
#
|
|
# Without this: Authelia starts, can't connect to MariaDB (still starting),
|
|
# exits immediately, strike 1. Next cycle: same thing, strike 2. Next cycle:
|
|
# restart, skip list, critical notification. MariaDB was fine the whole time.
|
|
#
|
|
# With this: MariaDB restarted first. Authelia skipped this cycle.
|
|
# Next cycle: MariaDB healthy → Authelia restarts cleanly. No false alarms.
|
|
#
|
|
# Format: "DependentContainer:dependency1 dependency2"
|
|
# Multiple dependencies space-separated. All must be running before dependent restarts.
|
|
#
|
|
HOST1_WATCHDOG_DEPENDENCIES=(
|
|
"Authelia:Mariadb-Authelia Redis-Authelia" # both db and cache must be up first
|
|
"Authelia-Secondary:Mariadb-Authelia Redis-Authelia-Secondary"
|
|
"NextCloud:Postgres-NextCloud" # NextCloud needs its postgres first
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
#### 🔒 Restart Loop Protection + Skip List
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# If the watchdog restarts the same container N times within a rolling time
|
|
# window and it keeps crashing, something is genuinely broken that restarts
|
|
# are not fixing. Continued restart attempts risk making it worse (database
|
|
# corruption from incomplete shutdowns, etc.).
|
|
#
|
|
# When a container hits the limit:
|
|
# 1. Added to the persistent skip list on /boot/config/
|
|
# 2. Critical notification sent — this needs human attention
|
|
# 3. Watchdog stops touching it — completely hands off
|
|
#
|
|
# The skip list survives reboots — it lives on /boot/. This is intentional.
|
|
# If a container was in a bad enough state to be skip-listed, a reboot
|
|
# doesn't fix the underlying problem. It stays on the list until cleared.
|
|
#
|
|
# AUTO-CLEAR: The watchdog checks the skip list every cycle and removes any
|
|
# container it finds running. So if the container recovers on its own (e.g.
|
|
# Docker's own restart policy eventually succeeds after a longer backoff),
|
|
# the watchdog detects it running and resumes normal monitoring automatically.
|
|
# Manual clear only needed when the container is stuck stopped.
|
|
#
|
|
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list
|
|
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
|
|
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
|
|
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
|
```
|
|
|
|
---
|
|
|
|
#### ⚡ Docker Daemon Health Check
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Every cycle begins with a daemon health check before any container operations.
|
|
# A hung Docker daemon makes every subsequent docker command hang — which would
|
|
# stall the entire watchdog indefinitely, leaving containers unmonitored.
|
|
#
|
|
# If the daemon doesn't respond within DOCKER_TIMEOUT seconds:
|
|
# 1. Attempt daemon restart via /etc/rc.d/rc.docker restart
|
|
# 2. Wait 15 seconds for recovery
|
|
# 3. Verify daemon is responding again
|
|
# 4. If recovered: log and continue the cycle normally
|
|
# 5. If still hung: log critical error, skip the rest of this cycle
|
|
# system_watchdog.sh will escalate from here (it has its own daemon check
|
|
# in Tier 1 Critical — bypass strikes, reboot if daemon stays down)
|
|
#
|
|
# DOCKER_TIMEOUT=10 # seconds — tight enough to detect hangs, not false positives
|
|
```
|
|
|
|
---
|
|
|
|
#### 📬 Notification Batching
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# All events from a single watchdog cycle are collected and sent as ONE
|
|
# notification at the end of the cycle.
|
|
#
|
|
# Why batching matters:
|
|
# On a 50+ container system, a single problem (e.g. shared database goes down)
|
|
# can cascade into 10+ dependent containers failing simultaneously. Without
|
|
# batching: 10 individual "container X failed" notifications arrive in rapid
|
|
# succession. With batching: one summary notification lists all affected
|
|
# containers. The first format is overwhelming. The second is actionable.
|
|
#
|
|
WATCHDOG_BATCH_NOTIFY=true
|
|
```
|
|
|
|
---
|
|
|
|
#### 🔇 Silent When Healthy
|
|
|
|
```
|
|
Runs 96 times per day. If it produced output every run, the logs would be
|
|
useless noise. The watchdog produces no output on clean cycles — only when
|
|
something needs attention or a periodic heartbeat fires.
|
|
|
|
Heartbeat interval: SYSTEM_WATCHDOG_HEARTBEAT_HOURS=1
|
|
→ "♥ docker_watchdog alive — HOST1 — ~4hr uptime" once per hour
|
|
→ proof the watchdog is running without log spam
|
|
```
|
|
|
|
---
|
|
|
|
### ── State Files ─────────────────────────────────────────────────────────────
|
|
|
|
| File | Configured As | Location | Resets | Purpose |
|
|
|------|--------------|----------|--------|---------|
|
|
| Strike counts | `WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | Per-container CPU/HTTP strike counters |
|
|
| Skip list | `SYS_WATCHDOG_FAILED_FILE` | `/boot/config/` | Never (manual / auto-clear) | Containers that exhausted restart attempts |
|
|
| Restart history | `WATCHDOG_CONTAINER_RESTART_LOG` | `/boot/config/` | Auto-purge after window | Restart loop detection data |
|
|
| Shared state | `SYS_WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | RAM emergency flag + cycle heartbeat from system_watchdog.sh |
|
|
|
|
`/tmp/` files reset on every reboot — correct, because strike counts from before
|
|
a reboot are meaningless after it. `/boot/config/` files survive reboots — also
|
|
correct, because a container that was skip-listed before a reboot is still broken after it.
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Normal operation — started automatically by array_start.sh, runs until
|
|
# array stops. You do not need to run this manually under normal circumstances.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
docker_watchdog.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Dry run — walk through a full watchdog cycle without restarting anything.
|
|
# Shows what the watchdog would do based on current container states.
|
|
# Useful for verifying configuration before enabling automatic restarts.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
docker_watchdog.sh --dry-run
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Status — show current watchdog state at a glance:
|
|
# • Strike counts for all monitored containers
|
|
# • Current skip list contents + which are running vs stopped
|
|
# • Whether grace period is active and how long remains
|
|
# • Whether RAM emergency deferral is active
|
|
# • Last cycle timing and daemon health
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
docker_watchdog.sh --status
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Verbose — show full detail for every container checked, every decision made.
|
|
# Useful for debugging why a container is or isn't being restarted.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
docker_watchdog.sh --log
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## ♻️ docker_daily_restart.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Restarts configured containers every night at 1am. Called by `daily_sync_maintenance.sh`
|
|
via `DAILY_MAINTENANCE_SCRIPTS` — not scheduled directly. The maintenance window already
|
|
owns this timeslot, so any downtime from restarts is absorbed by a window that's already
|
|
happening.
|
|
|
|
---
|
|
|
|
### ── Why Daily Restarts ───────────────────────────────────────────────────────
|
|
|
|
Not all container degradation triggers a watchdog response. Memory leaks that grow
|
|
slowly over days stay well below the hard limit for weeks. Connection tables that fill
|
|
up over a month never cross a clear threshold. Live TV schedulers that accumulate
|
|
internal state don't fail — they just get progressively slower.
|
|
|
|
Daily restarts at 1am clear all of this with zero user impact. Proactive maintenance
|
|
at a time when no one is using the services. The containers in this list were specifically
|
|
chosen because they are known to degrade — not because they are unreliable.
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Containers restarted every night at 1am.
|
|
# Each entry is just the container name — no configuration needed.
|
|
#
|
|
# Good candidates for daily restarts:
|
|
# - Reverse proxies (connection table management)
|
|
# - Authentication services (session cache clearing)
|
|
# - Live TV schedulers (accumulated scheduling state)
|
|
# - Download managers (connection pool maintenance)
|
|
#
|
|
HOST1_DAILY_RESTART_CONTAINERS=(
|
|
"NginxProxyManager" # connection table fills slowly over weeks
|
|
"Authelia" # session cache benefits from periodic clearing
|
|
"Dispatcharr" # Live TV scheduler accumulates state
|
|
"Dispatcharr-Basic" # secondary Live TV scheduler — same reason
|
|
"ErsatzTV-Emby" # channel schedule builder, stale entries accumulate
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Behaviour Rules ──────────────────────────────────────────────────────────
|
|
|
|
```
|
|
Running containers → docker restart (graceful stop + start — the correct approach)
|
|
Stopped containers → left stopped (was intentionally stopped — state is respected)
|
|
Missing containers → logged + skip (not found on this server — not an error)
|
|
```
|
|
|
|
The "was running → restart, was stopped → leave stopped" rule is consistent across
|
|
the entire ecosystem. Container state is always respected. The restart scripts
|
|
never bring back a container that was intentionally stopped.
|
|
|
|
---
|
|
|
|
### ── Safeguards ──────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# DEPENDENCY ORDERING — restarts happen in dependency-safe order.
|
|
# The same WATCHDOG_DEPENDENCIES configuration used by the watchdog applies
|
|
# here. If Authelia depends on Mariadb and Redis, those restart first.
|
|
# CONTAINER_DELAY seconds wait between dependency restart and dependent restart
|
|
# — gives the dependency time to fully initialise before the dependent tries
|
|
# to connect.
|
|
#
|
|
# RESTART VERIFICATION — after each restart, the script waits a settle period
|
|
# then checks if the container is still running. A container that starts and
|
|
# immediately crashes is marked as failed with a notification sent. The script
|
|
# does not silently pass a restart that didn't stick.
|
|
#
|
|
# DOCKER_TIMEOUT=30 — every docker command is wrapped in a 30 second timeout.
|
|
# A hung Docker daemon cannot cause this script to hang indefinitely.
|
|
# Timed-out commands are retried per RETRY_COUNT before marking as failed.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Normal — called by daily_sync_maintenance.sh, rarely run directly.
|
|
# Safe to run manually for ad hoc restarts when needed.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
docker_daily_restart.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Dry run — show exactly which containers would be restarted and which would
|
|
# be skipped (with reason). Run this before scheduling to verify the list.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
docker_daily_restart.sh --dry-run
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Status — show configured restart list, current container states,
|
|
# and dependency ordering for this server.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
docker_daily_restart.sh --status
|
|
|
|
# Verbose per-container output
|
|
docker_daily_restart.sh --log
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## ♻️ docker_weekly_restart.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Restarts configured containers once per week. Called by `weekly_sync_maintenance.sh`
|
|
via `WEEKLY_MAINTENANCE_SCRIPTS` — runs at 2:30am Sunday after the sync window has
|
|
completed and already restarted the critical containers (Emby, auth stack).
|
|
|
|
---
|
|
|
|
### ── Context: After the Sync Window ─────────────────────────────────────────
|
|
|
|
`weekly_sync_maintenance.sh` stops the critical container set (Emby, auth stack) for
|
|
a clean sync, then restarts them. This script runs **after** that restart — targeting a
|
|
**different** set of less-critical services that benefit from a weekly restart but do
|
|
not need to be stopped for the sync itself.
|
|
|
|
These containers are already running when this script executes. The distinction between
|
|
daily and weekly is purely about how often each service needs a clean start to maintain
|
|
its best performance:
|
|
|
|
```
|
|
Daily: Connection-heavy infrastructure (proxy, auth, Live TV)
|
|
→ degrades faster, benefits from more frequent resets
|
|
|
|
Weekly: Productivity and media services (NextCloud, AdGuard, Immich)
|
|
→ degrades slowly, monthly restart is overkill, daily is unnecessary
|
|
```
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Containers restarted every Sunday at 2:30am (after weekly sync completes).
|
|
#
|
|
HOST1_WEEKLY_RESTART_CONTAINERS=(
|
|
"NextCloud" # file sync — benefits from clean weekly start
|
|
"AdGuard-Home" # DNS — cache and stat accumulation
|
|
"Immich" # photo library — index/cache maintenance
|
|
)
|
|
```
|
|
|
|
Behaviour, dependency ordering, restart verification, and DOCKER_TIMEOUT are identical
|
|
to `docker_daily_restart.sh`. Same rules apply: running → restart, stopped → leave,
|
|
missing → skip.
|
|
|
|
```bash
|
|
docker_weekly_restart.sh # normal run
|
|
docker_weekly_restart.sh --dry-run # preview
|
|
docker_weekly_restart.sh --status # show config and current states
|
|
docker_weekly_restart.sh --log # verbose
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🌐 docker_network_connect.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Ensures custom Docker networks exist and connects specified containers to them at every
|
|
array start. Idempotent — if everything is already correct it produces no output.
|
|
Run via `ARRAY_START_SCRIPTS` — starts early in the array start sequence, before the
|
|
watchdogs begin their first cycle.
|
|
|
|
---
|
|
|
|
### ── What It Does ────────────────────────────────────────────────────────────
|
|
|
|
For each configured network, in order:
|
|
|
|
```
|
|
1. Does the network exist?
|
|
NO → Create it (bridge driver, Docker assigns subnet automatically)
|
|
→ Send notification — this should not happen except after an update
|
|
→ Log the subnet that was assigned
|
|
YES → Skip creation silently — correct state, nothing to do
|
|
|
|
2. For each configured container:
|
|
Already connected? → Skip silently — correct state
|
|
Not connected? → Connect it
|
|
Container missing? → Warn and skip — container may not be running yet,
|
|
not treated as fatal, will succeed on next array start
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Network Recreation Matters ─────────────────────────────────────────
|
|
|
|
Docker containers get their networks assigned at creation time via the unRAID template.
|
|
Changing the network assignment means deleting and recreating the container — which loses
|
|
any state not stored in the appdata volume. For containers created by other containers
|
|
(NextCloud AIO spawns its own stack, for example), you can't even touch the network
|
|
assignment through the unRAID UI.
|
|
|
|
`high-availability` is the main shared network in this setup — most containers join it
|
|
so they can communicate internally without going through the reverse proxy. After a
|
|
unRAID update wipes custom networks, every container on `high-availability` suddenly
|
|
can't reach its peers. This script recreates the network and reconnects everything at
|
|
the next array start.
|
|
|
|
The notification on creation is intentional. Network creation should only happen after
|
|
an update — if it's happening regularly, something is wrong with the network configuration
|
|
and you need to know.
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Networks to ensure exist + containers to connect to each network.
|
|
# The mapping is many-to-many: every container connects to every network.
|
|
#
|
|
# Containers listed here do not need to be running — the script handles
|
|
# missing containers gracefully (warns + skips). They will be connected on
|
|
# the next array start if they come up later.
|
|
#
|
|
HOST1_NETWORK_CONNECT_NETWORKS=(
|
|
"high-availability" # main internal network — most containers should be on this
|
|
)
|
|
|
|
HOST1_NETWORK_CONNECT_CONTAINERS=(
|
|
"memcached" # NextCloud's cache — needs to reach NextCloud AIO network
|
|
"Npm-CrowdSec" # CrowdSec bouncer — needs to reach NPM's network
|
|
)
|
|
```
|
|
|
|
> **Timing dependency:** Networks created by Docker Compose stacks exist only after
|
|
> those stacks start. If NextCloud AIO creates the `nextcloud-aio` network at startup
|
|
> and this script runs before NextCloud AIO starts, the network won't exist yet and
|
|
> the connection fails this run. It will succeed on the next array start once the
|
|
> network exists. This is a known limitation — schedule the Compose stacks early in
|
|
> the ARRAY_START_SCRIPTS order to minimise the window.
|
|
|
|
```bash
|
|
docker_network_connect.sh # normal run (at array start)
|
|
docker_network_connect.sh --dry-run # show what would be created / connected
|
|
docker_network_connect.sh --status # show current network and connection state
|
|
docker_network_connect.sh --log # verbose per-network per-container output
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🔧 watchdog_skip_list_manager.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Manual recovery tool. Used when automatic healing hasn't worked and a container needs
|
|
human intervention. Gives a clear picture of what the watchdog has already tried and
|
|
provides a clean path to resume normal monitoring after the problem is fixed.
|
|
|
|
---
|
|
|
|
### ── What the Skip List Is ───────────────────────────────────────────────────
|
|
|
|
When `docker_watchdog.sh` restarts the same container `WATCHDOG_CONTAINER_RESTART_LIMIT`
|
|
times within `WATCHDOG_CONTAINER_RESTART_WINDOW` hours, it concludes that restarts are
|
|
not fixing whatever is wrong. The container is added to the persistent skip list at
|
|
`SYS_WATCHDOG_FAILED_FILE` on `/boot/config/`. The watchdog stops touching it entirely.
|
|
A critical notification goes out.
|
|
|
|
This is the correct response. A watchdog that keeps hammering a broken container is
|
|
not helpful — it's potentially destructive (database corruption, incomplete writes).
|
|
Stopping automated attempts and alerting the operator is the right escalation path.
|
|
|
|
**Auto-clear:** The watchdog checks the skip list every cycle and removes any container
|
|
it finds running. If the container recovers on its own — Docker's built-in restart
|
|
policy eventually succeeds after a longer backoff, or someone manually starts it — the
|
|
watchdog detects it running and resumes normal monitoring without any intervention needed.
|
|
Manual clear is only necessary when the container is stuck stopped and cannot self-recover.
|
|
|
|
---
|
|
|
|
### ── Standard Recovery Workflow ─────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Step 1 — Understand the situation
|
|
# Shows the skip list, which containers are running vs stopped, how many
|
|
# restarts were attempted, and the restart history for each.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
watchdog_skip_list_manager.sh --status
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Step 2 — Fix the underlying problem first
|
|
# Check container logs: docker logs ContainerName --tail 100
|
|
# Check for disk issues: df -h /mnt/user
|
|
# Check for database issues: docker exec ContainerName sqlite3 /path/to.db ".tables"
|
|
# Fix whatever caused the repeated crashes before clearing the skip list.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Step 3 — Clear the container from the skip list + its restart history
|
|
# Clearing restart history is important — otherwise the counter carries over
|
|
# and the container hits the limit again almost immediately if it has any
|
|
# startup trouble.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
watchdog_skip_list_manager.sh --clear ContainerName
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Step 4 — Start the container manually
|
|
# Starting it yourself confirms your fix worked before handing it back
|
|
# to the watchdog. If it crashes immediately, you know the fix didn't work.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
docker start ContainerName
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Step 5 — Normal monitoring resumes automatically
|
|
# On the watchdog's next cycle it will see the container running and remove
|
|
# it from the skip list (if you cleared it manually, it's already gone).
|
|
# Restart history is clean. Back to normal.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Safety Warning ──────────────────────────────────────────────────────────
|
|
|
|
```
|
|
⚠️ If docker_watchdog.sh is currently running when you clear the skip list,
|
|
it may re-add the container on its very next cycle if the container is
|
|
still in a bad state.
|
|
|
|
The script detects this and warns you:
|
|
"docker_watchdog.sh is currently RUNNING — cleared container may be
|
|
re-added on next cycle if still failing"
|
|
|
|
Fix the root cause BEFORE clearing. Clearing the skip list without fixing
|
|
the underlying problem just resets the counter — the container will exhaust
|
|
its restart attempts again and return to the skip list.
|
|
```
|
|
|
|
---
|
|
|
|
### ── All Actions ─────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Show status — skip list contents, container states, restart history
|
|
# This is the default action — running without arguments shows status
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
watchdog_skip_list_manager.sh
|
|
watchdog_skip_list_manager.sh --status # explicit
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Clear specific container — removes from skip list + clears restart history
|
|
# Requires confirmation (type YES) unless --force is passed
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
watchdog_skip_list_manager.sh --clear ContainerName
|
|
watchdog_skip_list_manager.sh --clear ContainerName --force # no prompt
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Clear everything — full reset of skip list and restart history
|
|
# Use when multiple containers are affected or after a systemic problem is fixed
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
watchdog_skip_list_manager.sh --clear-all
|
|
watchdog_skip_list_manager.sh --clear-all --force # non-interactive
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Dry run — show what would be cleared without actually clearing
|
|
# Works with --clear and --clear-all
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
watchdog_skip_list_manager.sh --clear-all --dry-run
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ HOW THE SCRIPTS FIT TOGETHER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```
|
|
Array starts
|
|
│
|
|
▼
|
|
docker_network_connect.sh ─────── run once
|
|
│ ensure networks + connections
|
|
│ silent if correct, notify if creating
|
|
│
|
|
▼ (continuous background)
|
|
docker_watchdog.sh ─────────────── every 15 minutes
|
|
│ Tier 1: memory, CPU, HTTP, required
|
|
│ Tier 2: global unhealthy/OOM/crash/dead scan
|
|
│ reads system_watchdog state (RAM emergency)
|
|
│
|
|
│ (on skip list event)
|
|
▼
|
|
watchdog_skip_list_manager.sh ──── manual
|
|
inspect state, clear after fixing
|
|
|
|
Scheduled maintenance windows:
|
|
│
|
|
▼ (1am daily)
|
|
daily_sync_maintenance.sh
|
|
└── docker_daily_restart.sh ── restart connection-heavy services
|
|
NPM, Authelia, Dispatcharr, ErsatzTV
|
|
│
|
|
▼ (2:30am Sunday)
|
|
weekly_sync_maintenance.sh
|
|
└── docker_weekly_restart.sh ─ restart less-critical services
|
|
NextCloud, AdGuard, Immich
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All configuration lives in two places. `detect_hosts()` in `common.sh` aliases all
|
|
`HOST1_` and `HOST2_` prefixed variables to their unprefixed names so the scripts
|
|
always use the right values for the server they're running on.
|
|
|
|
---
|
|
|
|
### 📋 master_host*.conf — Per-Host Configuration
|
|
|
|
These vary between HOST1 and HOST2 because each server runs different containers
|
|
at different resource limits with different network requirements.
|
|
|
|
```bash
|
|
# master_host1.conf (or master_host2.conf for HOST2)
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# Tier 1 — memory limits ("ContainerName:LimitInMB")
|
|
HOST1_WATCHDOG_CONTAINERS=(
|
|
"Emby:18432"
|
|
"LidaTube:6144"
|
|
)
|
|
|
|
# Tier 1 — HTTP health check endpoints ("ContainerName:http://host:port")
|
|
HOST1_WATCHDOG_CONTAINER_URLS=(
|
|
"Emby:http://localhost:8096"
|
|
)
|
|
|
|
# Tier 1 — must always be running
|
|
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
|
"NginxProxyManager"
|
|
"Authelia"
|
|
"Mariadb-Authelia"
|
|
"Redis-Authelia"
|
|
)
|
|
|
|
# Tier 1 + 2 — dependency ordering ("Dependent:dep1 dep2")
|
|
HOST1_WATCHDOG_DEPENDENCIES=(
|
|
"Authelia:Mariadb-Authelia Redis-Authelia"
|
|
)
|
|
|
|
# Daily restart list
|
|
HOST1_DAILY_RESTART_CONTAINERS=(
|
|
"NginxProxyManager"
|
|
"Authelia"
|
|
"Dispatcharr"
|
|
)
|
|
|
|
# Weekly restart list
|
|
HOST1_WEEKLY_RESTART_CONTAINERS=(
|
|
"NextCloud"
|
|
"AdGuard-Home"
|
|
)
|
|
|
|
# Networks to ensure exist
|
|
HOST1_NETWORK_CONNECT_NETWORKS=(
|
|
"high-availability"
|
|
)
|
|
|
|
# Containers to connect to every configured network
|
|
HOST1_NETWORK_CONNECT_CONTAINERS=(
|
|
"memcached"
|
|
"Npm-CrowdSec"
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### 📋 master.conf — Shared Configuration
|
|
|
|
These apply equally to both servers — thresholds, intervals, toggle switches.
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# ── Watchdog Intervals and Timing ──────────────────────────────────────────
|
|
DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes)
|
|
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
|
|
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
|
|
|
|
# ── Memory Thresholds ──────────────────────────────────────────────────────
|
|
SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart)
|
|
|
|
# ── CPU Thresholds ─────────────────────────────────────────────────────────
|
|
SOFT_CPU_THRESHOLD=50 # warn threshold — normalised % of total cores
|
|
HARD_CPU_THRESHOLD=85 # strike threshold — normalised % of total cores
|
|
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
|
|
|
|
# ── HTTP Health Check ──────────────────────────────────────────────────────
|
|
CURL_TIMEOUT=5 # seconds before curl times out
|
|
RESP_FAIL_LIMIT=2 # consecutive failures before restart
|
|
|
|
# ── Restart Loop Protection ────────────────────────────────────────────────
|
|
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts before skip list
|
|
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
|
|
|
|
# ── Tier 2 Global Scan ─────────────────────────────────────────────────────
|
|
WATCHDOG_SCAN_ALL=true # enable global scan
|
|
WATCHDOG_SCAN_IGNORE=() # containers excluded from Tier 2
|
|
WATCHDOG_RESTART_UNHEALTHY=true
|
|
WATCHDOG_NOTIFY_OOM=true
|
|
WATCHDOG_NOTIFY_CRASHLOOP=true
|
|
WATCHDOG_CRASH_LIMIT=5
|
|
WATCHDOG_RESTART_DEAD=true
|
|
WATCHDOG_RESTART_CRASHED=true
|
|
|
|
# ── Notifications ──────────────────────────────────────────────────────────
|
|
WATCHDOG_BATCH_NOTIFY=true # collect events, send one summary
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ ADDING A CONTAINER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Adding a new container to monitoring is additive — add the relevant lines to
|
|
`master_host1.conf` (and `master_host2.conf` if it runs there too). No changes to
|
|
any script needed. `detect_hosts()` picks up the new configuration on the next cycle.
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Example: adding "MyApp" to full Tier 1 monitoring + daily restarts
|
|
|
|
# 1. Memory hard limit — restart immediately if exceeded
|
|
HOST1_WATCHDOG_CONTAINERS=(
|
|
"Emby:18432"
|
|
"MyApp:2048" # 2GB — check with `docker stats MyApp` to size this correctly
|
|
)
|
|
|
|
# 2. HTTP health check — restart after 2 consecutive failures
|
|
HOST1_WATCHDOG_CONTAINER_URLS=(
|
|
"Emby:http://localhost:8096"
|
|
"MyApp:http://localhost:8080/health" # or just the root if no /health endpoint
|
|
)
|
|
|
|
# 3. Required — must always be running
|
|
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
|
"NginxProxyManager"
|
|
"Authelia"
|
|
"MyApp" # add here if it should always be running
|
|
)
|
|
|
|
# 4. Dependency — if MyApp needs its database up first
|
|
HOST1_WATCHDOG_DEPENDENCIES=(
|
|
"Authelia:Mariadb-Authelia Redis-Authelia"
|
|
"MyApp:MyApp-Database" # database restarts first, then MyApp
|
|
)
|
|
|
|
# 5. Daily restart — if MyApp degrades over time
|
|
HOST1_DAILY_RESTART_CONTAINERS=(
|
|
"NginxProxyManager"
|
|
"Authelia"
|
|
"MyApp"
|
|
)
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Tier 2 picks up MyApp automatically — no configuration needed.
|
|
# It will be included in the global unhealthy/OOM/crash/dead scan without
|
|
# any additional setup. Tier 2 is the catch-all for everything not in Tier 1.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# To EXCLUDE MyApp from Tier 2 (e.g. it intentionally exits between runs):
|
|
# master.conf
|
|
WATCHDOG_SCAN_IGNORE=(
|
|
"MyApp" # one-shot container — exits normally, not a crash
|
|
)
|
|
``` |