Files

T

FailedProxy 8c5667b1df added all ai generated Readme files, added the last of the tools scripts

2026-04-18 15:40:41 -04:00

13 KiB

Raw Blame History

Docker Essentials

Container lifecycle management — health monitoring, scheduled restarts, and network configuration. These scripts keep your Docker stack healthy, fresh, and correctly connected without manual intervention.

Monitors/            — observes containers, reports issues
Docker_Essentials/   — acts on containers (this folder)
unRAID_Essentials/   — acts on the server itself

Scripts

`docker_watchdog.sh`

The self-healing container monitoring system. Two tiers of monitoring that work together to keep every container in the stack healthy — from strict per-container thresholds down to global health scanning of everything that's running.

# Scheduled as: */15 * * * *  (every 15 minutes)
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh

Tier 1 — Strict Monitoring

Explicitly configured containers with per-container thresholds. Every container you care about most lives here.

Memory hard limits:

declare -A WATCHDOG_CONTAINERS=(
    ["Emby"]=16384      # 16GB hard limit — immediate restart if exceeded
    ["LidaTube"]=6144   # 6GB
    ["Tdarr"]=6144
    ["Code-Server"]=1024
)

Memory is checked against the configured limit in MB. If a container exceeds its hard limit it is restarted immediately — no strike system, no waiting. Memory leaks are real and immediate action is right.

A soft threshold (SOFT_MEM_THRESHOLD=80) warns when a container reaches 80% of its hard limit — useful for spotting gradual leaks before they become problems.

CPU thresholds:

CPU is normalised against total core count automatically. A container using 85% of one core on a 16-core system is ~5.3% normalised — not a problem. 85% normalised on a 16-core system means 13.6 cores worth of CPU — that's a problem.

The strike system prevents restarts on brief spikes:

CPU above HARD_CPU_THRESHOLD → strike 1
CPU above HARD_CPU_THRESHOLD next cycle → strike 2 → restart
CPU recovers → strike count resets

HTTP responsiveness:

declare -A WATCHDOG_CONTAINER_URLS=(
    ["Emby"]="http://localhost:8096"
)

Containers with configured URLs are checked via curl. If the endpoint doesn't respond within CURL_TIMEOUT seconds that's a strike. Two consecutive failures trigger a restart. A container can be running and appear healthy to Docker while its application layer is frozen — HTTP checks catch this.

Required containers:

WATCHDOG_REQUIRED_CONTAINERS=(
    "NginxProxyManager"
    "Lldap-Gmer4Lfe"
    "Authelia"
    "Mariadb-Authelia"
    "Redis-Authelia"
    "Authelia-Secondary"
    "Redis-Authelia-Secondary"
)

These must always be running. If any are found stopped, the watchdog attempts to restart them. Strike system applies — persistent failures get added to the skip list.

Tier 2 — Global Health Scan

Scans every running container for health issues — catches anything not explicitly configured in Tier 1.

Check	Trigger	Action
`WATCHDOG_RESTART_UNHEALTHY`	Docker HEALTHCHECK reports `unhealthy`	Restart
`WATCHDOG_NOTIFY_OOM`	Kernel OOM-killed the container	Restart + notify
`WATCHDOG_NOTIFY_CRASHLOOP`	Docker RestartCount climbing	Notify (critical above `WATCHDOG_CRASH_LIMIT`)
`WATCHDOG_RESTART_DEAD`	Container in dead state	Remove + restart
`WATCHDOG_RESTART_CRASHED`	Non-zero exit code	Restart

Each check is independently toggleable — disable checks that cause false positives in your environment.

Note on HEALTHCHECK: Only containers with a HEALTHCHECK instruction defined in their Docker image report health status. Containers without one are invisible to the unhealthy check but still caught by crash, dead, and OOM checks. You can add custom HEALTHCHECKs via unRAID's Extra Parameters field — see the health check guide for your specific containers.

Cross-cutting Intelligence

These apply to both tiers on every watchdog run:

Startup grace period:

WATCHDOG_STARTUP_GRACE=600  # seconds after boot

For the first 10 minutes after array start, checks run but restarts are suppressed. Containers need time to come up — false positives during boot are common without this. Checks still run and report so you can see what's happening, but no restarts fire.

Dependency ordering:

declare -A WATCHDOG_DEPENDENCIES=(
    ["Authelia"]="Mariadb-Authelia Redis-Authelia"
    ["Authelia-Secondary"]="Mariadb-Authelia Redis-Authelia-Secondary"
    ["NextCloud"]="Postgres-NextCloud"
)

If a container's dependency is also down, the dependent is skipped this cycle. The dependency gets restarted first. On the next cycle — once the database is up and accepting connections — the dependent container gets restarted. This prevents the classic failure mode where Authelia is restarted before its database is ready and fails immediately, triggering another restart attempt.

Restart loop protection:

WATCHDOG_CONTAINER_RESTART_LIMIT=3
WATCHDOG_CONTAINER_RESTART_WINDOW=1   # hours
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"

If the watchdog restarts the same container 3 times within 1 hour, that container is added to the persistent skip list. Something is genuinely broken that restarts are not fixing — continued hammering wastes resources and masks the real problem. A critical notification is sent when a container hits the skip list.

The skip list lives on /boot/ — it survives reboots. The container stays on the skip list until it is found running again (manually fixed or recovered after a reboot), at which point it's automatically removed and the restart history is cleared.

Notification batching:

WATCHDOG_BATCH_NOTIFY=true

All events from a single watchdog run are collected and sent as one notification at the end. On a system with 50+ containers, individual per-event notifications during a problem cascade would be unmanageable. One clean summary tells you what happened without flooding your notification channel.

Skip List Management

The skip list (/boot/config/system_watchdog_failed.db) is the persistent memory of containers that have exhausted restart attempts.

# View current skip list
cat /boot/config/system_watchdog_failed.db

# A container auto-removes itself when found running again
# To manually clear a specific container:
sed -i '/ContainerName/d' /boot/config/system_watchdog_failed.db

# To clear the entire skip list:
> /boot/config/system_watchdog_failed.db

Also clear the restart history when manually fixing a container:

sed -i '/ContainerName|/d' /boot/config/container_restart_history.db

State Files

File	Location	Resets	Purpose
`container_watchdog_state.db`	`/tmp/`	On reboot	Strike counts for all containers
`system_watchdog_failed.db`	`/boot/config/`	Never (manual)	Persistent skip list
`container_restart_history.db`	`/boot/config/`	Auto-purge after window	Restart loop detection

`docker_daily_restart.sh`

Restarts configured containers every day.

# Scheduled as: 0 3 * * *  (3am daily)
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh

Why daily restarts:

Some containers accumulate memory over time — connection pools that don't shrink, caches that grow without bound, log buffers that don't rotate. A daily restart clears all of this. It's simpler and more reliable than trying to tune every container's internal memory management.

Containers that benefit from daily restarts are typically those handling lots of short-lived connections — reverse proxies, auth servers, and live TV schedulers.

DAILY_RESTART_CONTAINERS=(
    "NginxProxyManager"     # connection pool accumulation
    "Authelia"              # session and token cache
    "Dispatcharr"           # live TV connection management
    "Dispatcharr-Basic"
    "Dispatcharr-Iptv-Users"
    "ErsatzTV-Emby"         # channel scheduling state
)

Retry logic: Uses RETRY_COUNT and SLEEP from Master.conf. If a container fails to restart it retries before marking it as failed and notifying.

`docker_weekly_restart.sh`

Restarts configured containers once per week.

# Scheduled as: 0 3 * * 0  (Sunday 3am weekly)
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh

For less critical services that benefit from periodic restarts but don't need daily cycling. Typically productivity and self-hosted application containers that are stable but benefit from a clean weekly slate.

WEEKLY_RESTART_CONTAINERS=(
    "NextCloud"
    "Organizrv2-Gmer4Lfe"
    "AdGuard-Home"
    "Immich-Gmer4Lfe"
)

Sunday morning is the natural maintenance window — it runs alongside the weekly log clear, ZFS snapshot, SMART check and backup verify. Everything happens while load is lowest.

`docker_network_connect.sh`

Connects containers to extra Docker networks on array start.

# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_network_connect.sh

The problem it solves:

Docker containers are assigned networks at creation time via the unRAID template. Sometimes containers need to communicate with containers on a different network that wasn't configured in the original template — for example, memcached needing to talk to the nextcloud-aio network so NextCloud can use it for caching.

The correct solution is to add the network in the template. But some containers are created by other containers (like nextcloud-aio) and their network assignments can't easily be changed. This script handles those edge cases at array start.

Every container in NETWORK_CONNECT_CONTAINERS is connected to every network in NETWORK_CONNECT_NETWORKS — many-to-many. Already-connected containers are skipped cleanly — safe to run multiple times.

NETWORK_CONNECT_CONTAINERS=(
    "memcached"
    "Npm-CrowdSec"
)

NETWORK_CONNECT_NETWORKS=(
    "nextcloud-aio"     # Docker network name — must exist before array start
)

Note: The target network must exist before this script runs. Networks created by Docker Compose or the nextcloud-aio stack are created when their containers start — if those containers start after this script, the connection will fail. The unRAID User Scripts plugin "At Startup of Array" timing usually handles this correctly but be aware of the dependency.

Relationship Between Scripts

docker_network_connect.sh   — runs once at array start
        ↓
docker_watchdog.sh          — runs every 15 minutes
  ├── Tier 1: strict per-container monitoring
  └── Tier 2: global health scan of everything

docker_daily_restart.sh     — runs at 3am every day
docker_weekly_restart.sh    — runs at 3am every Sunday

The watchdog is the continuous monitor. The restart scripts are the scheduled maintenance. Together they cover both reactive healing (watchdog) and proactive freshness (restarts).

Adding a New Container to the Watchdog

Tier 1 — memory monitoring:

# Add to WATCHDOG_CONTAINERS in Master.conf
declare -A WATCHDOG_CONTAINERS=(
    ["Emby"]=16384
    ["MyNewContainer"]=2048   # 2GB hard limit
)

Tier 1 — HTTP check:

declare -A WATCHDOG_CONTAINER_URLS=(
    ["Emby"]="http://localhost:8096"
    ["MyNewContainer"]="http://localhost:9000/health"
)

Tier 1 — required container:

WATCHDOG_REQUIRED_CONTAINERS=(
    "NginxProxyManager"
    "MyNewContainer"    # must always be running
)

Tier 2 — dependency:

declare -A WATCHDOG_DEPENDENCIES=(
    ["Authelia"]="Mariadb-Authelia Redis-Authelia"
    ["MyNewContainer"]="its-database-container"   # restart db first
)

Tier 2 — ignore in global scan:

WATCHDOG_SCAN_IGNORE=(
    "intentionally-stopped-container"   # skip this in global scan
)

Scheduled Summary

# At Startup of Array
docker_network_connect.sh

# Every 15 minutes
*/15 * * * *    docker_watchdog.sh

# Daily — 3am
0 3 * * *       docker_daily_restart.sh

# Weekly — Sunday 3am
0 3 * * 0       docker_weekly_restart.sh

--dry-run Support

All scripts support --dry-run. Always test before scheduling:

/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --dry-run
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh --dry-run
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh --dry-run
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_network_connect.sh --dry-run

docker_watchdog.sh --status shows current strike counts, skip list contents, and grace period status without running any checks.

13 KiB Raw Blame History