# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 🐳 DOCKER ESSENTIALS # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ **Self-healing container lifecycle management for a 50+ container unRAID stack.** Health monitoring that catches problems as they happen. Scheduled restarts that prevent degradation before it becomes visible. Network configuration that survives reboots and unRAID updates. Recovery tooling for when something genuinely breaks. > **Why this folder exists:** Docker on unRAID does not heal itself. A container that > crashes stays crashed. A memory leak accumulates silently for days until the system > starts swapping. A container whose dependency restarted first fails in a loop while > the dependency comes up fine five seconds later. None of this surfaces clearly — it > just builds into a system that feels flaky without a clear reason why. These scripts > are the answer to all of that. --- ## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Running a large Docker stack on unRAID is genuinely powerful — but Docker's own tooling gives you almost nothing between "container is running" and "container has been dead for three days and you just noticed." The built-in restart policies help with outright crashes, but they have zero visibility into memory leaks, frozen application layers, dependency ordering, restart loops, or the difference between a container that crashed and one you intentionally stopped. They give you no way to know *why* something keeps restarting — just that it does. These are the specific problems that led to building this, in roughly the order they were encountered: --- ### 🔴 Memory Leaks Accumulating Silently Emby's transcode session handling occasionally leaks memory — a session ends but its memory allocation doesn't fully release. SABnzbd's Python process slowly expands over days as it processes downloads. Without hard memory limits and automatic enforcement, these don't fail dramatically — they just consume more and more RAM until the system starts swapping and everything slows to a crawl. By then, nothing in the Docker logs tells you why the system feels slow. `docker stats` shows a container at 22GB and climbing, but Docker itself does nothing about it. The fix: hard per-container memory ceilings in `WATCHDOG_CONTAINERS`. When a container exceeds its limit the watchdog restarts it immediately — no strikes, no waiting. A memory leak is not a transient spike. Immediate action is correct. --- ### 🔴 Containers That Look Running But Aren't Responding Docker reports a container as `Up 14 days` while its application layer has been silently frozen for hours. The reverse proxy happily forwards traffic to a service that returns nothing. Users see a broken page. Docker sees a healthy container. The container process is technically running — it just isn't doing anything. Docker's built-in health checks require a `HEALTHCHECK` instruction in the image itself, which most self-hosted images don't have. Even those that do often check something too shallow — a process exists, not whether it's actually serving requests. The fix: HTTP health checks on the actual service port. `curl` to the real endpoint on every watchdog cycle. If the service doesn't respond within `CURL_TIMEOUT` seconds, that's a strike. Two consecutive failures trigger a restart. The distinction between "process running" and "service responding" is the distinction that matters. --- ### 🔴 Dependency Failures on Restart Authelia connects to MariaDB at startup. If both are down simultaneously — say, after a power cut — and the watchdog restarts Authelia first, Authelia fails to connect, exits immediately, and goes into a crash loop. Meanwhile MariaDB is coming up fine in the background. The watchdog sees Authelia crash three times, adds it to the skip list, and sends a critical notification. Authelia was never broken. It just came up in the wrong order, failed at startup, and got punished for it. The fix: dependency ordering. If a container's dependency is also down, the dependent is skipped entirely this cycle. The dependency is restarted first. On the next cycle — once MariaDB is actually accepting connections — Authelia is restarted and comes up cleanly. The skip list is never involved. No false alarms. No manual recovery needed. --- ### 🔴 Restart Loops Corrupting State Some containers corrupt their internal state if restarted repeatedly in rapid succession. SQLite databases that don't get a clean shutdown write incomplete transactions. Partially applied database migrations leave schema in an inconsistent state. A container that crashes on startup after a bad migration gets restarted immediately, crashes again, gets restarted again — each restart has a chance of making the database worse, not better. A naive watchdog that just keeps hammering a crashed container is actively harmful in this scenario. More restarts mean more corruption risk. The right response when restarts aren't working is to stop restarting and alert the operator. The fix: restart loop protection. After `WATCHDOG_CONTAINER_RESTART_LIMIT` restarts within a rolling `WATCHDOG_CONTAINER_RESTART_WINDOW` hour window, the container goes on the skip list. A critical notification goes out. The watchdog stops touching it. The operator investigates and clears the skip list once the underlying problem is fixed. --- ### 🔴 Slow Degradation That Never Becomes a Failure NginxProxyManager accumulates stale entries in its connection table over weeks of uptime. Dispatcharr's Live TV scheduler builds up internal scheduling state that makes decisions progressively slower after months of continuous operation. These containers never crash. They never throw errors. They just get progressively worse in ways that are hard to attribute to anything specific — until someone notices that the proxy feels slower than it used to, or that Live TV channel changes take longer than they should. The fix: scheduled restarts. Not because something is broken, but because some containers simply perform better with a clean start. Daily at 1am for connection-heavy services. Weekly for less-critical services that run fine for weeks but benefit from a clean slate. Zero user impact — happens while everyone is asleep. --- ### 🔴 Network Configuration Lost After Updates unRAID occasionally wipes custom Docker networks after updates — particularly networks created by Docker Compose stacks or the NextCloud AIO container. Any container that depended on those networks for internal communication suddenly can't reach its peers. memcached can't talk to NextCloud. CrowdSec can't talk to NginxProxyManager. Services appear to be up but silently fail to communicate with each other. The fix: network recreation at every array start. `docker_network_connect.sh` checks every configured network on startup, creates any that are missing, and connects all configured containers to them. Idempotent — if everything is already correct, it does nothing and produces no output. If a network had to be created, it notifies — that only happens after an update, and you want to know when it does. --- ### 🔴 No Visibility Into What the Watchdog Already Tried A container keeps appearing in a broken state. You SSH in and see it's stopped. You don't know if the watchdog tried to restart it and failed, gave up and skip-listed it, is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip list file on `/boot/config/`, check the restart history file, check the state file — none of which have obvious formats. The fix: `watchdog_skip_list_manager.sh`. One command to see exactly what's on the skip list, which containers are stopped vs running, how many restarts were attempted, and what the watchdog's current state is. One command to clear a specific container and its history after you've fixed the problem. No manual file editing required. --- ## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ``` Docker_Essentials/ ← acts on containers (this folder) unRAID_Essentials/ ← acts on the server itself Monitors/ ← observes, measures, reports Rsync/ ← moves data between servers ``` Four distinct roles, each handled by dedicated scripts: --- ### 🔁 Reactive Healing — `docker_watchdog.sh` Continuous two-tier monitoring that catches problems as they happen and acts on them immediately. Runs as a background process started at array start. Every 15 minutes it checks the full stack and fixes what it can — silently when everything is fine, visibly when something needs attention. Two tiers because different containers need different monitoring strategies: - **Tier 1** — explicit per-container configuration with specific thresholds - **Tier 2** — global scan of everything that's running with catch-all health checks --- ### ♻️ Proactive Freshness — `docker_daily_restart.sh` + `docker_weekly_restart.sh` Scheduled restarts that prevent slow degradation before it becomes visible. Not because something broke — because some containers simply work better after a clean start. Called by the maintenance orchestrators (`daily_sync_maintenance.sh` and `weekly_sync_maintenance.sh`) — not run standalone. They run inside the maintenance windows so any downtime from restarts is absorbed by the window that's already happening. --- ### 🌐 Network Integrity — `docker_network_connect.sh` Ensures custom Docker networks exist and containers are connected to them at every array start. Silent when everything is correct. Notifies when it has to create something — which means something was wiped and you should know about it. --- ### 🔧 Recovery Tooling — `watchdog_skip_list_manager.sh` Manual tool for the moments when automatic recovery hasn't worked. Gives a clear picture of what the watchdog has already tried, lets you clear the skip list cleanly after fixing the root cause, and warns you if clearing might immediately re-add the container. --- ## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Two watchdogs run simultaneously. They are designed to work together, not compete: ``` system_watchdog.sh ← watches the server: RAM, CPU, disk, kernel, daemon health docker_watchdog.sh ← watches the containers: memory, CPU, HTTP response, crashes ``` **The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh` would see stopped containers on its next cycle and restart them — directly undoing the RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover. The system would eventually hit the reboot threshold anyway, having accomplished nothing. **The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When `system_watchdog.sh` triggers a RAM emergency shutdown it writes `mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every cycle and defers all container restart logic until it clears. Health URL checks for excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still run. Everything else stands down. **The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh` checks how long ago the state file was modified. If it's more than 2 hours old while `mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running. `docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced indefinitely by a stale flag from a process that's no longer running. --- ## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | Script | Role | When It Runs | |--------|------|-------------| | `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop | | `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` | | `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` | | `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start | | `watchdog_skip_list_manager.sh` | Skip list inspection and manual recovery | On demand | --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🐳 docker_watchdog.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ The self-healing heart of the ecosystem. Two tiers of monitoring run every 15 minutes as a background process. Tier 1 applies specific thresholds to explicitly configured containers. Tier 2 scans everything else for generic health problems. Together they catch the full range of container failures — from subtle memory leaks to outright crashes. ```bash # Started automatically at array start via array_start.sh # Runs continuously until array stops (SIGTERM → clean shutdown) # Interval: DOCKER_WATCHDOG_INTERVAL=900 (15 minutes) ``` --- ### ── Tier 1 — Strict Per-Container Monitoring ──────────────────────────────── Applies **only** to containers you explicitly configure. These are the containers you care most about — the ones that affect users when they fail. Configure them once in `master_host*.conf` and they are monitored with specific, appropriate thresholds forever. --- #### 💾 Memory Hard Limits ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Memory hard limits in MB — immediate restart when exceeded, no strike system. # Each entry: "ContainerName:LimitInMB" # # These are NOT soft targets — they are hard ceilings. A container that hits # its limit gets restarted immediately on the current cycle. Memory leaks are # not brief spikes. Waiting for a second confirmation just allows more leak. # # How to size limits: # Check normal usage: docker stats ContainerName # Set limit at ~150-200% of normal peak usage # Emby peaks around 8-12GB during heavy transcoding — 18GB gives headroom # without triggering on legitimate load spikes # HOST1_WATCHDOG_CONTAINERS=( "Emby:18432" # 18GB — peaks ~12GB under heavy transcode load "LidaTube:6144" # 6GB — YouTube downloader, can grow with large queues "Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature "Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous ) ``` A soft warning threshold fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — giving early visibility into a container approaching its ceiling before a restart is triggered. Useful for catching gradual leaks before they become events. --- #### 📊 CPU Thresholds ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # CPU thresholds are normalised against the server's total core count. # This makes the threshold meaningful regardless of hardware. # # Why normalised: # A container using 85% of one core on a 16-core machine = 5.3% normalised # → a nothing, don't touch it # A container using 85% normalised on a 16-core machine = 13.6 cores worth # → a runaway process, restart it # # CPU uses a STRIKE SYSTEM — not immediate restart like memory. # Brief CPU spikes are completely normal (Tdarr encoding, Emby transcoding, # SABnzbd unpacking). The strike system ignores spikes and acts on sustained usage. # # Strike 1: CPU above HARD_CPU_THRESHOLD this cycle → warn, increment strike # Strike 2: CPU above threshold next cycle → restart, reset strike counter # Recovery: CPU drops below threshold any cycle → reset strike counter to 0 # SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only HARD_CPU_THRESHOLD=85 # strike at 85% normalised CPU_FAIL_LIMIT=2 # consecutive strikes before restart ``` --- #### 🌐 HTTP Responsiveness ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # HTTP health checks hit the actual service endpoint on every watchdog cycle. # "Container running" and "service responding" are not the same thing. # # A frozen Emby will respond to nothing but show as Up in Docker. # A curl timeout catches this where Docker's own checks never would. # # Also uses a STRIKE SYSTEM — network hiccups and brief restarts happen. # Two consecutive non-responses before acting prevents false positives # from momentary connectivity issues. # # Format: "ContainerName:http://host:port/optional-path" # The path can be a lightweight health endpoint or just the root URL # HOST1_WATCHDOG_CONTAINER_URLS=( "Emby:http://localhost:8096" # Emby WebUI root — fast to respond "NginxProxyManager:http://localhost:81" # NPM admin interface ) # # CURL_TIMEOUT=5 # seconds before a non-response counts as a failure # RESP_FAIL_LIMIT=2 # consecutive failures before restart ``` > **Note on Docker HEALTHCHECK:** Docker has its own `HEALTHCHECK` mechanism but it > requires the image to define a health check command — most self-hosted images don't. > These HTTP checks work regardless of what the image defines. They check what actually > matters: does the service respond to a request? --- #### ✅ Required Containers ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Containers that must always be running. If found stopped, the watchdog # attempts to restart them every cycle until they are running or hit the # restart loop limit. # # These are the containers whose absence breaks everything else: # NginxProxyManager — all external traffic routes through this # Authelia — authentication for every protected service # Mariadb-Authelia — Authelia's database — Authelia cannot start without it # Redis-Authelia — Authelia's session store — same dependency # # Required containers use the STRIKE SYSTEM — one miss might be mid-restart. # Persistent failure → skip list → critical notification. # HOST1_WATCHDOG_REQUIRED_CONTAINERS=( "NginxProxyManager" # reverse proxy — external access depends on this "Authelia" # SSO authentication — all protected services need it "Mariadb-Authelia" # Authelia database — must be up before Authelia starts "Redis-Authelia" # Authelia session cache — same startup dependency ) ``` --- ### ── Tier 2 — Global Health Scan ───────────────────────────────────────────── Scans **every running container** when `WATCHDOG_SCAN_ALL=true`. No per-container configuration required — this is the catch-all that protects everything not explicitly configured in Tier 1. Containers in `WATCHDOG_SCAN_IGNORE` are excluded from Tier 2 but still covered by Tier 1 if configured there. --- | Check | What Triggers It | What Happens | Why This Matters | |-------|-----------------|--------------|-----------------| | `WATCHDOG_RESTART_UNHEALTHY` | Docker HEALTHCHECK reports `unhealthy` | Restart | Catches containers with built-in health checks that are failing | | `WATCHDOG_NOTIFY_OOM` | Kernel OOM-killed the container | Restart + notify | OOM kills are silent by default — you'd never know without this | | `WATCHDOG_NOTIFY_CRASHLOOP` | Docker RestartCount climbing | Notify; above `WATCHDOG_CRASH_LIMIT` → restart → skip list | Distinguishes "just restarted once" from "has crashed 12 times" | | `WATCHDOG_RESTART_DEAD` | Container in `dead` state | Remove + start | Dead containers can't be restarted — must be removed first | | `WATCHDOG_RESTART_CRASHED` | Non-zero exit code, exited state | Restart | Catches clean-exit crashes that Docker's restart policy misses | ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # Tier 2 toggles — disable any check that produces false positives in your # environment. Each is independent — disabling one doesn't affect the others. # WATCHDOG_SCAN_ALL=true # enable Tier 2 global scan WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts WATCHDOG_RESTART_DEAD=true # recover containers in dead state WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list # Containers to exclude from Tier 2 global scan entirely. # Use for containers you intentionally stop/start manually, or containers that # have benign non-zero exits as part of their normal operation. WATCHDOG_SCAN_IGNORE=( "my-one-shot-container" # runs and exits normally — don't treat as crash ) ``` --- ### ── Cross-Cutting Intelligence ────────────────────────────────────────────── These mechanisms apply to **both tiers** on every watchdog cycle. They are what separates intelligent monitoring from naive restart-on-failure. --- #### ⏱️ Startup Grace Period ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # After array start, containers take time to initialise — databases run recovery, # services wait for dependencies, Emby scans its library. During this window, # a container that isn't responding yet is not broken — it's just starting. # # Without a grace period, the watchdog fires false-positive restarts in the # first minutes after every array start. With it, checks run and log normally # but restart actions are suppressed until the grace period expires. # # The grace period clock starts from when the watchdog process itself starts — # not from when the array starts — so it's accurate even if array start # takes a few minutes. # WATCHDOG_STARTUP_GRACE=600 # 10 minutes — restarts suppressed, checks still log ``` --- #### 🔗 Dependency Ordering ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # When a container and its dependency are both down, restart the dependency first. # Skip the dependent container entirely on this cycle. On the next cycle the # dependency should be healthy, and the dependent can restart cleanly. # # Without this: Authelia starts, can't connect to MariaDB (still starting), # exits immediately, strike 1. Next cycle: same thing, strike 2. Next cycle: # restart, skip list, critical notification. MariaDB was fine the whole time. # # With this: MariaDB restarted first. Authelia skipped this cycle. # Next cycle: MariaDB healthy → Authelia restarts cleanly. No false alarms. # # Format: "DependentContainer:dependency1 dependency2" # Multiple dependencies space-separated. All must be running before dependent restarts. # HOST1_WATCHDOG_DEPENDENCIES=( "Authelia:Mariadb-Authelia Redis-Authelia" # both db and cache must be up first "Authelia-Secondary:Mariadb-Authelia Redis-Authelia-Secondary" "NextCloud:Postgres-NextCloud" # NextCloud needs its postgres first ) ``` --- #### 🔒 Restart Loop Protection + Skip List ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # If the watchdog restarts the same container N times within a rolling time # window and it keeps crashing, something is genuinely broken that restarts # are not fixing. Continued restart attempts risk making it worse (database # corruption from incomplete shutdowns, etc.). # # When a container hits the limit: # 1. Added to the persistent skip list on /boot/config/ # 2. Critical notification sent — this needs human attention # 3. Watchdog stops touching it — completely hands off # # The skip list survives reboots — it lives on /boot/. This is intentional. # If a container was in a bad enough state to be skip-listed, a reboot # doesn't fix the underlying problem. It stays on the list until cleared. # # AUTO-CLEAR: The watchdog checks the skip list every cycle and removes any # container it finds running. So if the container recovers on its own (e.g. # Docker's own restart policy eventually succeeds after a longer backoff), # the watchdog detects it running and resumes normal monitoring automatically. # Manual clear only needed when the container is stuck stopped. # WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db" SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db" ``` --- #### ⚡ Docker Daemon Health Check ```bash # ───────────────────────────────────────────────────────────────────────────── # Every cycle begins with a daemon health check before any container operations. # A hung Docker daemon makes every subsequent docker command hang — which would # stall the entire watchdog indefinitely, leaving containers unmonitored. # # If the daemon doesn't respond within DOCKER_TIMEOUT seconds: # 1. Attempt daemon restart via /etc/rc.d/rc.docker restart # 2. Wait 15 seconds for recovery # 3. Verify daemon is responding again # 4. If recovered: log and continue the cycle normally # 5. If still hung: log critical error, skip the rest of this cycle # system_watchdog.sh will escalate from here (it has its own daemon check # in Tier 1 Critical — bypass strikes, reboot if daemon stays down) # # DOCKER_TIMEOUT=10 # seconds — tight enough to detect hangs, not false positives ``` --- #### 📬 Notification Batching ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # All events from a single watchdog cycle are collected and sent as ONE # notification at the end of the cycle. # # Why batching matters: # On a 50+ container system, a single problem (e.g. shared database goes down) # can cascade into 10+ dependent containers failing simultaneously. Without # batching: 10 individual "container X failed" notifications arrive in rapid # succession. With batching: one summary notification lists all affected # containers. The first format is overwhelming. The second is actionable. # WATCHDOG_BATCH_NOTIFY=true ``` --- #### 🔇 Silent When Healthy ``` Runs 96 times per day. If it produced output every run, the logs would be useless noise. The watchdog produces no output on clean cycles — only when something needs attention or a periodic heartbeat fires. Heartbeat interval: SYSTEM_WATCHDOG_HEARTBEAT_HOURS=1 → "♥ docker_watchdog alive — HOST1 — ~4hr uptime" once per hour → proof the watchdog is running without log spam ``` --- ### ── State Files ───────────────────────────────────────────────────────────── | File | Configured As | Location | Resets | Purpose | |------|--------------|----------|--------|---------| | Strike counts | `WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | Per-container CPU/HTTP strike counters | | Skip list | `SYS_WATCHDOG_FAILED_FILE` | `/boot/config/` | Never (manual / auto-clear) | Containers that exhausted restart attempts | | Restart history | `WATCHDOG_CONTAINER_RESTART_LOG` | `/boot/config/` | Auto-purge after window | Restart loop detection data | | Shared state | `SYS_WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | RAM emergency flag + cycle heartbeat from system_watchdog.sh | `/tmp/` files reset on every reboot — correct, because strike counts from before a reboot are meaningless after it. `/boot/config/` files survive reboots — also correct, because a container that was skip-listed before a reboot is still broken after it. --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # Normal operation — started automatically by array_start.sh, runs until # array stops. You do not need to run this manually under normal circumstances. # ───────────────────────────────────────────────────────────────────────────── docker_watchdog.sh # ───────────────────────────────────────────────────────────────────────────── # Dry run — walk through a full watchdog cycle without restarting anything. # Shows what the watchdog would do based on current container states. # Useful for verifying configuration before enabling automatic restarts. # ───────────────────────────────────────────────────────────────────────────── docker_watchdog.sh --dry-run # ───────────────────────────────────────────────────────────────────────────── # Status — show current watchdog state at a glance: # • Strike counts for all monitored containers # • Current skip list contents + which are running vs stopped # • Whether grace period is active and how long remains # • Whether RAM emergency deferral is active # • Last cycle timing and daemon health # ───────────────────────────────────────────────────────────────────────────── docker_watchdog.sh --status # ───────────────────────────────────────────────────────────────────────────── # Verbose — show full detail for every container checked, every decision made. # Useful for debugging why a container is or isn't being restarted. # ───────────────────────────────────────────────────────────────────────────── docker_watchdog.sh --log ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ♻️ docker_daily_restart.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Restarts configured containers every night at 1am. Called by `daily_sync_maintenance.sh` via `DAILY_MAINTENANCE_SCRIPTS` — not scheduled directly. The maintenance window already owns this timeslot, so any downtime from restarts is absorbed by a window that's already happening. --- ### ── Why Daily Restarts ─────────────────────────────────────────────────────── Not all container degradation triggers a watchdog response. Memory leaks that grow slowly over days stay well below the hard limit for weeks. Connection tables that fill up over a month never cross a clear threshold. Live TV schedulers that accumulate internal state don't fail — they just get progressively slower. Daily restarts at 1am clear all of this with zero user impact. Proactive maintenance at a time when no one is using the services. The containers in this list were specifically chosen because they are known to degrade — not because they are unreliable. ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Containers restarted every night at 1am. # Each entry is just the container name — no configuration needed. # # Good candidates for daily restarts: # - Reverse proxies (connection table management) # - Authentication services (session cache clearing) # - Live TV schedulers (accumulated scheduling state) # - Download managers (connection pool maintenance) # HOST1_DAILY_RESTART_CONTAINERS=( "NginxProxyManager" # connection table fills slowly over weeks "Authelia" # session cache benefits from periodic clearing "Dispatcharr" # Live TV scheduler accumulates state "Dispatcharr-Basic" # secondary Live TV scheduler — same reason "ErsatzTV-Emby" # channel schedule builder, stale entries accumulate ) ``` --- ### ── Behaviour Rules ────────────────────────────────────────────────────────── ``` Running containers → docker restart (graceful stop + start — the correct approach) Stopped containers → left stopped (was intentionally stopped — state is respected) Missing containers → logged + skip (not found on this server — not an error) ``` The "was running → restart, was stopped → leave stopped" rule is consistent across the entire ecosystem. Container state is always respected. The restart scripts never bring back a container that was intentionally stopped. --- ### ── Safeguards ────────────────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # DEPENDENCY ORDERING — restarts happen in dependency-safe order. # The same WATCHDOG_DEPENDENCIES configuration used by the watchdog applies # here. If Authelia depends on Mariadb and Redis, those restart first. # CONTAINER_DELAY seconds wait between dependency restart and dependent restart # — gives the dependency time to fully initialise before the dependent tries # to connect. # # RESTART VERIFICATION — after each restart, the script waits a settle period # then checks if the container is still running. A container that starts and # immediately crashes is marked as failed with a notification sent. The script # does not silently pass a restart that didn't stick. # # DOCKER_TIMEOUT=30 — every docker command is wrapped in a 30 second timeout. # A hung Docker daemon cannot cause this script to hang indefinitely. # Timed-out commands are retried per RETRY_COUNT before marking as failed. # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # Normal — called by daily_sync_maintenance.sh, rarely run directly. # Safe to run manually for ad hoc restarts when needed. # ───────────────────────────────────────────────────────────────────────────── docker_daily_restart.sh # ───────────────────────────────────────────────────────────────────────────── # Dry run — show exactly which containers would be restarted and which would # be skipped (with reason). Run this before scheduling to verify the list. # ───────────────────────────────────────────────────────────────────────────── docker_daily_restart.sh --dry-run # ───────────────────────────────────────────────────────────────────────────── # Status — show configured restart list, current container states, # and dependency ordering for this server. # ───────────────────────────────────────────────────────────────────────────── docker_daily_restart.sh --status # Verbose per-container output docker_daily_restart.sh --log ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ♻️ docker_weekly_restart.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Restarts configured containers once per week. Called by `weekly_sync_maintenance.sh` via `WEEKLY_MAINTENANCE_SCRIPTS` — runs at 2:30am Sunday after the sync window has completed and already restarted the critical containers (Emby, auth stack). --- ### ── Context: After the Sync Window ───────────────────────────────────────── `weekly_sync_maintenance.sh` stops the critical container set (Emby, auth stack) for a clean sync, then restarts them. This script runs **after** that restart — targeting a **different** set of less-critical services that benefit from a weekly restart but do not need to be stopped for the sync itself. These containers are already running when this script executes. The distinction between daily and weekly is purely about how often each service needs a clean start to maintain its best performance: ``` Daily: Connection-heavy infrastructure (proxy, auth, Live TV) → degrades faster, benefits from more frequent resets Weekly: Productivity and media services (NextCloud, AdGuard, Immich) → degrades slowly, monthly restart is overkill, daily is unnecessary ``` ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Containers restarted every Sunday at 2:30am (after weekly sync completes). # HOST1_WEEKLY_RESTART_CONTAINERS=( "NextCloud" # file sync — benefits from clean weekly start "AdGuard-Home" # DNS — cache and stat accumulation "Immich" # photo library — index/cache maintenance ) ``` Behaviour, dependency ordering, restart verification, and DOCKER_TIMEOUT are identical to `docker_daily_restart.sh`. Same rules apply: running → restart, stopped → leave, missing → skip. ```bash docker_weekly_restart.sh # normal run docker_weekly_restart.sh --dry-run # preview docker_weekly_restart.sh --status # show config and current states docker_weekly_restart.sh --log # verbose ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🌐 docker_network_connect.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Ensures custom Docker networks exist and connects specified containers to them at every array start. Idempotent — if everything is already correct it produces no output. Run via `ARRAY_START_SCRIPTS` — starts early in the array start sequence, before the watchdogs begin their first cycle. --- ### ── What It Does ──────────────────────────────────────────────────────────── For each configured network, in order: ``` 1. Does the network exist? NO → Create it (bridge driver, Docker assigns subnet automatically) → Send notification — this should not happen except after an update → Log the subnet that was assigned YES → Skip creation silently — correct state, nothing to do 2. For each configured container: Already connected? → Skip silently — correct state Not connected? → Connect it Container missing? → Warn and skip — container may not be running yet, not treated as fatal, will succeed on next array start ``` --- ### ── Why Network Recreation Matters ───────────────────────────────────────── Docker containers get their networks assigned at creation time via the unRAID template. Changing the network assignment means deleting and recreating the container — which loses any state not stored in the appdata volume. For containers created by other containers (NextCloud AIO spawns its own stack, for example), you can't even touch the network assignment through the unRAID UI. `high-availability` is the main shared network in this setup — most containers join it so they can communicate internally without going through the reverse proxy. After a unRAID update wipes custom networks, every container on `high-availability` suddenly can't reach its peers. This script recreates the network and reconnects everything at the next array start. The notification on creation is intentional. Network creation should only happen after an update — if it's happening regularly, something is wrong with the network configuration and you need to know. ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Networks to ensure exist + containers to connect to each network. # The mapping is many-to-many: every container connects to every network. # # Containers listed here do not need to be running — the script handles # missing containers gracefully (warns + skips). They will be connected on # the next array start if they come up later. # HOST1_NETWORK_CONNECT_NETWORKS=( "high-availability" # main internal network — most containers should be on this ) HOST1_NETWORK_CONNECT_CONTAINERS=( "memcached" # NextCloud's cache — needs to reach NextCloud AIO network "Npm-CrowdSec" # CrowdSec bouncer — needs to reach NPM's network ) ``` > **Timing dependency:** Networks created by Docker Compose stacks exist only after > those stacks start. If NextCloud AIO creates the `nextcloud-aio` network at startup > and this script runs before NextCloud AIO starts, the network won't exist yet and > the connection fails this run. It will succeed on the next array start once the > network exists. This is a known limitation — schedule the Compose stacks early in > the ARRAY_START_SCRIPTS order to minimise the window. ```bash docker_network_connect.sh # normal run (at array start) docker_network_connect.sh --dry-run # show what would be created / connected docker_network_connect.sh --status # show current network and connection state docker_network_connect.sh --log # verbose per-network per-container output ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🔧 watchdog_skip_list_manager.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Manual recovery tool. Used when automatic healing hasn't worked and a container needs human intervention. Gives a clear picture of what the watchdog has already tried and provides a clean path to resume normal monitoring after the problem is fixed. --- ### ── What the Skip List Is ─────────────────────────────────────────────────── When `docker_watchdog.sh` restarts the same container `WATCHDOG_CONTAINER_RESTART_LIMIT` times within `WATCHDOG_CONTAINER_RESTART_WINDOW` hours, it concludes that restarts are not fixing whatever is wrong. The container is added to the persistent skip list at `SYS_WATCHDOG_FAILED_FILE` on `/boot/config/`. The watchdog stops touching it entirely. A critical notification goes out. This is the correct response. A watchdog that keeps hammering a broken container is not helpful — it's potentially destructive (database corruption, incomplete writes). Stopping automated attempts and alerting the operator is the right escalation path. **Auto-clear:** The watchdog checks the skip list every cycle and removes any container it finds running. If the container recovers on its own — Docker's built-in restart policy eventually succeeds after a longer backoff, or someone manually starts it — the watchdog detects it running and resumes normal monitoring without any intervention needed. Manual clear is only necessary when the container is stuck stopped and cannot self-recover. --- ### ── Standard Recovery Workflow ───────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # Step 1 — Understand the situation # Shows the skip list, which containers are running vs stopped, how many # restarts were attempted, and the restart history for each. # ───────────────────────────────────────────────────────────────────────────── watchdog_skip_list_manager.sh --status # ───────────────────────────────────────────────────────────────────────────── # Step 2 — Fix the underlying problem first # Check container logs: docker logs ContainerName --tail 100 # Check for disk issues: df -h /mnt/user # Check for database issues: docker exec ContainerName sqlite3 /path/to.db ".tables" # Fix whatever caused the repeated crashes before clearing the skip list. # ───────────────────────────────────────────────────────────────────────────── # ───────────────────────────────────────────────────────────────────────────── # Step 3 — Clear the container from the skip list + its restart history # Clearing restart history is important — otherwise the counter carries over # and the container hits the limit again almost immediately if it has any # startup trouble. # ───────────────────────────────────────────────────────────────────────────── watchdog_skip_list_manager.sh --clear ContainerName # ───────────────────────────────────────────────────────────────────────────── # Step 4 — Start the container manually # Starting it yourself confirms your fix worked before handing it back # to the watchdog. If it crashes immediately, you know the fix didn't work. # ───────────────────────────────────────────────────────────────────────────── docker start ContainerName # ───────────────────────────────────────────────────────────────────────────── # Step 5 — Normal monitoring resumes automatically # On the watchdog's next cycle it will see the container running and remove # it from the skip list (if you cleared it manually, it's already gone). # Restart history is clean. Back to normal. # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Safety Warning ────────────────────────────────────────────────────────── ``` ⚠️ If docker_watchdog.sh is currently running when you clear the skip list, it may re-add the container on its very next cycle if the container is still in a bad state. The script detects this and warns you: "docker_watchdog.sh is currently RUNNING — cleared container may be re-added on next cycle if still failing" Fix the root cause BEFORE clearing. Clearing the skip list without fixing the underlying problem just resets the counter — the container will exhaust its restart attempts again and return to the skip list. ``` --- ### ── All Actions ───────────────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # Show status — skip list contents, container states, restart history # This is the default action — running without arguments shows status # ───────────────────────────────────────────────────────────────────────────── watchdog_skip_list_manager.sh watchdog_skip_list_manager.sh --status # explicit # ───────────────────────────────────────────────────────────────────────────── # Clear specific container — removes from skip list + clears restart history # Requires confirmation (type YES) unless --force is passed # ───────────────────────────────────────────────────────────────────────────── watchdog_skip_list_manager.sh --clear ContainerName watchdog_skip_list_manager.sh --clear ContainerName --force # no prompt # ───────────────────────────────────────────────────────────────────────────── # Clear everything — full reset of skip list and restart history # Use when multiple containers are affected or after a systemic problem is fixed # ───────────────────────────────────────────────────────────────────────────── watchdog_skip_list_manager.sh --clear-all watchdog_skip_list_manager.sh --clear-all --force # non-interactive # ───────────────────────────────────────────────────────────────────────────── # Dry run — show what would be cleared without actually clearing # Works with --clear and --clear-all # ───────────────────────────────────────────────────────────────────────────── watchdog_skip_list_manager.sh --clear-all --dry-run ``` --- ## ━━━ HOW THE SCRIPTS FIT TOGETHER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ``` Array starts │ ▼ docker_network_connect.sh ─────── run once │ ensure networks + connections │ silent if correct, notify if creating │ ▼ (continuous background) docker_watchdog.sh ─────────────── every 15 minutes │ Tier 1: memory, CPU, HTTP, required │ Tier 2: global unhealthy/OOM/crash/dead scan │ reads system_watchdog state (RAM emergency) │ │ (on skip list event) ▼ watchdog_skip_list_manager.sh ──── manual inspect state, clear after fixing Scheduled maintenance windows: │ ▼ (1am daily) daily_sync_maintenance.sh └── docker_daily_restart.sh ── restart connection-heavy services NPM, Authelia, Dispatcharr, ErsatzTV │ ▼ (2:30am Sunday) weekly_sync_maintenance.sh └── docker_weekly_restart.sh ─ restart less-critical services NextCloud, AdGuard, Immich ``` --- ## ━━━ CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ All configuration lives in two places. `detect_hosts()` in `common.sh` aliases all `HOST1_` and `HOST2_` prefixed variables to their unprefixed names so the scripts always use the right values for the server they're running on. --- ### 📋 master_host*.conf — Per-Host Configuration These vary between HOST1 and HOST2 because each server runs different containers at different resource limits with different network requirements. ```bash # master_host1.conf (or master_host2.conf for HOST2) # ───────────────────────────────────────────────────────────────────────────── # Tier 1 — memory limits ("ContainerName:LimitInMB") HOST1_WATCHDOG_CONTAINERS=( "Emby:18432" "LidaTube:6144" ) # Tier 1 — HTTP health check endpoints ("ContainerName:http://host:port") HOST1_WATCHDOG_CONTAINER_URLS=( "Emby:http://localhost:8096" ) # Tier 1 — must always be running HOST1_WATCHDOG_REQUIRED_CONTAINERS=( "NginxProxyManager" "Authelia" "Mariadb-Authelia" "Redis-Authelia" ) # Tier 1 + 2 — dependency ordering ("Dependent:dep1 dep2") HOST1_WATCHDOG_DEPENDENCIES=( "Authelia:Mariadb-Authelia Redis-Authelia" ) # Daily restart list HOST1_DAILY_RESTART_CONTAINERS=( "NginxProxyManager" "Authelia" "Dispatcharr" ) # Weekly restart list HOST1_WEEKLY_RESTART_CONTAINERS=( "NextCloud" "AdGuard-Home" ) # Networks to ensure exist HOST1_NETWORK_CONNECT_NETWORKS=( "high-availability" ) # Containers to connect to every configured network HOST1_NETWORK_CONNECT_CONTAINERS=( "memcached" "Npm-CrowdSec" ) ``` --- ### 📋 master.conf — Shared Configuration These apply equally to both servers — thresholds, intervals, toggle switches. ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # ── Watchdog Intervals and Timing ────────────────────────────────────────── DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes) WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot CONTAINER_DELAY=15 # seconds between dependency + dependent restart # ── Memory Thresholds ────────────────────────────────────────────────────── SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart) # ── CPU Thresholds ───────────────────────────────────────────────────────── SOFT_CPU_THRESHOLD=50 # warn threshold — normalised % of total cores HARD_CPU_THRESHOLD=85 # strike threshold — normalised % of total cores CPU_FAIL_LIMIT=2 # consecutive strikes before restart # ── HTTP Health Check ────────────────────────────────────────────────────── CURL_TIMEOUT=5 # seconds before curl times out RESP_FAIL_LIMIT=2 # consecutive failures before restart # ── Restart Loop Protection ──────────────────────────────────────────────── WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts before skip list WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours # ── Tier 2 Global Scan ───────────────────────────────────────────────────── WATCHDOG_SCAN_ALL=true # enable global scan WATCHDOG_SCAN_IGNORE=() # containers excluded from Tier 2 WATCHDOG_RESTART_UNHEALTHY=true WATCHDOG_NOTIFY_OOM=true WATCHDOG_NOTIFY_CRASHLOOP=true WATCHDOG_CRASH_LIMIT=5 WATCHDOG_RESTART_DEAD=true WATCHDOG_RESTART_CRASHED=true # ── Notifications ────────────────────────────────────────────────────────── WATCHDOG_BATCH_NOTIFY=true # collect events, send one summary ``` --- ## ━━━ ADDING A CONTAINER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Adding a new container to monitoring is additive — add the relevant lines to `master_host1.conf` (and `master_host2.conf` if it runs there too). No changes to any script needed. `detect_hosts()` picks up the new configuration on the next cycle. ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Example: adding "MyApp" to full Tier 1 monitoring + daily restarts # 1. Memory hard limit — restart immediately if exceeded HOST1_WATCHDOG_CONTAINERS=( "Emby:18432" "MyApp:2048" # 2GB — check with `docker stats MyApp` to size this correctly ) # 2. HTTP health check — restart after 2 consecutive failures HOST1_WATCHDOG_CONTAINER_URLS=( "Emby:http://localhost:8096" "MyApp:http://localhost:8080/health" # or just the root if no /health endpoint ) # 3. Required — must always be running HOST1_WATCHDOG_REQUIRED_CONTAINERS=( "NginxProxyManager" "Authelia" "MyApp" # add here if it should always be running ) # 4. Dependency — if MyApp needs its database up first HOST1_WATCHDOG_DEPENDENCIES=( "Authelia:Mariadb-Authelia Redis-Authelia" "MyApp:MyApp-Database" # database restarts first, then MyApp ) # 5. Daily restart — if MyApp degrades over time HOST1_DAILY_RESTART_CONTAINERS=( "NginxProxyManager" "Authelia" "MyApp" ) # ───────────────────────────────────────────────────────────────────────────── # Tier 2 picks up MyApp automatically — no configuration needed. # It will be included in the global unhealthy/OOM/crash/dead scan without # any additional setup. Tier 2 is the catch-all for everything not in Tier 1. # ───────────────────────────────────────────────────────────────────────────── # To EXCLUDE MyApp from Tier 2 (e.g. it intentionally exits between runs): # master.conf WATCHDOG_SCAN_IGNORE=( "MyApp" # one-shot container — exits normally, not a crash ) ```