Varaverk/Docker_Essentials/Manual-Docker_Essentials.md

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🐳 DOCKER ESSENTIALS — Manual

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Configuration reference, setup procedures, and operational workflows. For folder overview and design philosophy see README-Docker_Essentials.md. For per-script detail see the script headers directly.

---

━━━ WATCHDOG CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

All watchdog configuration lives in master_host*.conf (per-container lists) and master.conf (shared thresholds and toggles). detect_hosts() aliases all HOST1_ / HOST2_ prefixed vars to their unprefixed names at runtime — scripts always read the right values for the server they're running on.

---

── Memory Hard Limits ────────────────────────────────────────────────────────

# master_host1.conf
# Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
#
# Sizing guidance:
#   Check normal peak:  docker stats ContainerName
#   Set limit at ~150-200% of normal peak
#   Emby peaks ~12GB under heavy transcode load → 18GB gives headroom
#   without triggering on legitimate load spikes
#
HOST1_WATCHDOG_CONTAINERS=(
    "Emby:18432"        # 18GB — peaks ~12GB under heavy transcode load
    "LidaTube:6144"     # 6GB  — YouTube downloader, grows with large queues
    "Tdarr:6144"        # 6GB  — video transcoder, memory-intensive by nature
    "Code-Server:1024"  # 1GB  — IDE, should be light; 1GB is generous
)

A soft warning fires at SOFT_MEM_THRESHOLD=80 percent of the hard limit — early visibility into a container approaching its ceiling before a restart is triggered.

---

── CPU Thresholds ────────────────────────────────────────────────────────────

# master.conf
# CPU is normalised against total core count — meaningful regardless of hardware.
#
# Why normalised:
#   85% on one core of a 16-core machine = 5.3% normalised → ignore it
#   85% normalised on a 16-core machine = 13.6 cores worth → runaway process
#
# CPU uses a STRIKE SYSTEM — not immediate restart like memory.
# Brief spikes are normal (Tdarr encoding, Emby transcoding, SABnzbd unpacking).
# The strike system ignores spikes and acts on sustained high usage.
#
#   Strike 1: above HARD_CPU_THRESHOLD this cycle → warn, increment strike
#   Strike 2: above threshold next cycle → restart, reset counter
#   Recovery: drops below threshold any cycle → reset counter to 0
#
SOFT_CPU_THRESHOLD=50    # warn at 50% normalised — informational only
HARD_CPU_THRESHOLD=85    # strike at 85% normalised
CPU_FAIL_LIMIT=2         # consecutive strikes before restart

---

── HTTP Health Checks ────────────────────────────────────────────────────────

# master_host1.conf
# Format: "ContainerName:http://host:port/optional-path"
# Hits the actual service endpoint on every watchdog cycle.
# "Container running" and "service responding" are not the same thing.
#
# Uses a STRIKE SYSTEM — network hiccups and brief restarts happen.
# Two consecutive non-responses before acting prevents false positives.
#
# The path can be a lightweight health endpoint or the root URL.
# Docker's own HEALTHCHECK requires the image to define it — most don't.
# These checks work regardless of what the image defines.
#
HOST1_WATCHDOG_CONTAINER_URLS=(
    "Emby:http://localhost:8096"            # Emby WebUI root — fast to respond
    "NginxProxyManager:http://localhost:81" # NPM admin interface
)

# master.conf
CURL_TIMEOUT=5      # seconds before non-response counts as a failure
RESP_FAIL_LIMIT=2   # consecutive failures before restart

---

── Required Containers ───────────────────────────────────────────────────────

# master_host1.conf
# Containers that must always be running.
# Found stopped → watchdog attempts restart every cycle until running or skip-listed.
# Uses the STRIKE SYSTEM — one miss might be mid-restart.
# Persistent failure → skip list → critical notification.
#
# These are the containers whose absence breaks everything else:
#
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
    "NginxProxyManager"     # reverse proxy — all external traffic routes through this
    "Authelia"              # SSO authentication — all protected services need it
    "Mariadb-Authelia"      # Authelia database — must be up before Authelia starts
    "Redis-Authelia"        # Authelia session cache — same startup dependency
)

---

── Dependency Ordering ───────────────────────────────────────────────────────

# master_host1.conf
# Format: "DependentContainer:dependency1 dependency2"
# Multiple dependencies space-separated. All must be running before dependent restarts.
#
# When a container and its dependency are both down:
#   → restart the dependency first
#   → skip the dependent this cycle
#   → next cycle: dependency healthy → dependent restarts cleanly
#
# Without this: Authelia starts, can't connect to MariaDB (still starting),
#   exits immediately, strike 1. Next cycle: same, strike 2. Skip list.
#   MariaDB was fine the whole time.
#
HOST1_WATCHDOG_DEPENDENCIES=(
    "Authelia:Mariadb-Authelia Redis-Authelia"
    "Authelia-Secondary:Mariadb-Authelia Redis-Authelia-Secondary"
    "NextCloud:Postgres-NextCloud"
)

The same dependency configuration is used by docker_daily_restart.sh and docker_weekly_restart.sh — configure once, applies everywhere.

---

── Startup Grace Period ──────────────────────────────────────────────────────

# master.conf
# Suppress restart actions for N seconds after array start.
# Checks still run and log — only restart actions are suppressed.
# Clock starts from when the watchdog process itself starts.
#
# Without this: false-positive restarts fire in the first minutes after
# every array start while containers are still initialising.
#
WATCHDOG_STARTUP_GRACE=600   # 10 minutes

---

── Tier 2 Global Scan ────────────────────────────────────────────────────────

Tier 2 scans every running container when WATCHDOG_SCAN_ALL=true. No per-container configuration required — it's the catch-all for everything not explicitly in Tier 1.

# master.conf
WATCHDOG_SCAN_ALL=true               # enable global scan
WATCHDOG_RESTART_UNHEALTHY=true      # act on Docker HEALTHCHECK failures
WATCHDOG_NOTIFY_OOM=true             # detect and notify kernel OOM kills
WATCHDOG_NOTIFY_CRASHLOOP=true       # detect escalating restart counts
WATCHDOG_RESTART_DEAD=true           # recover containers in dead state
WATCHDOG_RESTART_CRASHED=true        # restart containers that exited non-zero
WATCHDOG_CRASH_LIMIT=5               # RestartCount above this → restart + skip list

# Containers excluded from Tier 2 entirely.
# Use for containers you intentionally stop/start manually, or containers that
# have benign non-zero exits as part of their normal operation.
WATCHDOG_SCAN_IGNORE=(
    "my-one-shot-container"    # runs and exits normally — not a crash
)

Each toggle is independent — disable any check that produces false positives in your environment without affecting the others.

---

── Restart Loop Protection ──────────────────────────────────────────────────

# master.conf
# N restarts within a rolling window → skip list + critical notification.
# The watchdog stops touching the container entirely.
# Skip list lives on /boot/config/ — survives reboots intentionally.
# A container bad enough to be skip-listed is still broken after a reboot.
#
WATCHDOG_CONTAINER_RESTART_LIMIT=3    # restarts in the window before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1   # rolling window in hours
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"

Auto-clear: The watchdog checks the skip list every cycle and removes any container it finds running. If the container recovers on its own, monitoring resumes automatically. Manual clear is only needed when the container is stuck stopped.

---

── Notification Batching ────────────────────────────────────────────────────

# master.conf
# All events from one cycle collected → one notification at end of cycle.
#
# Why: a shared database going down can cascade 10+ containers failing
# simultaneously. Without batching: 10 individual pings. With batching:
# one summary listing all affected containers. Actionable vs overwhelming.
#
WATCHDOG_BATCH_NOTIFY=true

---

── State Files Reference ────────────────────────────────────────────────────

File	Config Var	Location	Resets	Purpose
Strike counts	WATCHDOG_STATE_FILE	/tmp/	On reboot	Per-container CPU/HTTP strike counters
Skip list	SYS_WATCHDOG_FAILED_FILE	/boot/config/	Never (manual / auto-clear)	Containers that exhausted restart attempts
Restart history	WATCHDOG_CONTAINER_RESTART_LOG	/boot/config/	Auto-purge after window	Restart loop detection data
Shared state	SYS_WATCHDOG_STATE_FILE	/tmp/	On reboot	RAM emergency flag + cycle heartbeat from system_watchdog.sh

/tmp/ resets on reboot — correct, strike counts before a reboot are meaningless after it. /boot/config/ survives reboots — correct, a skip-listed container is still broken after a reboot.

---

━━━ RESTART SCHEDULE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

---

── Daily Restart List ───────────────────────────────────────────────────────

# master_host1.conf
# Restarted every night at 1am via daily_sync_maintenance.sh.
#
# Good candidates:
#   Reverse proxies         — connection table fills slowly over weeks
#   Authentication services — session cache benefits from periodic clearing
#   Live TV schedulers      — accumulated scheduling state slows decisions
#   Download managers       — connection pool maintenance
#
HOST1_DAILY_RESTART_CONTAINERS=(
    "NginxProxyManager"     # connection table fills slowly over weeks
    "Authelia"              # session cache benefits from periodic clearing
    "Dispatcharr"           # Live TV scheduler accumulates state
    "Dispatcharr-Basic"     # secondary Live TV scheduler — same reason
    "ErsatzTV-Emby"         # channel schedule builder, stale entries accumulate
)

This list also drives docker_update.sh in normal mode — containers added here get their images updated daily before the restart. Add a container once, it gets both.

---

── Weekly Restart List ──────────────────────────────────────────────────────

# master_host1.conf
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
#
# Daily vs Weekly decision:
#   Daily:  connection-heavy infrastructure — degrades faster (proxy, auth, Live TV)
#   Weekly: productivity and media services — degrades slowly (NextCloud, AdGuard, Immich)
#
HOST1_WEEKLY_RESTART_CONTAINERS=(
    "NextCloud"       # file sync — benefits from clean weekly start
    "AdGuard-Home"    # DNS — cache and stat accumulation
    "Immich"          # photo library — index/cache maintenance
)

---

━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

# master_host1.conf
# Networks to ensure exist + containers to connect to each network.
# Many-to-many: every container connects to every network listed.
#
# Containers do not need to be running — script handles missing containers
# gracefully (warns + skips). They connect on the next array start.
#
HOST1_NETWORK_CONNECT_NETWORKS=(
    "high-availability"     # main internal network — most containers should be on this
)

HOST1_NETWORK_CONNECT_CONTAINERS=(
    "memcached"             # NextCloud's cache — needs to reach NextCloud AIO network
    "Npm-CrowdSec"          # CrowdSec bouncer — needs to reach NPM's network
)

Timing dependency: Networks created by Docker Compose stacks (e.g. NextCloud AIO) only exist after those stacks start. If this script runs before the Compose stack, the network won't exist yet and the connection fails this run. It will succeed on the next array start. Schedule Compose stacks early in ARRAY_START_SCRIPTS order to minimise the window.

---

━━━ CONTAINER UPDATE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

# master.conf
DAILY_CONTAINER_UPDATES=true     # enable/disable daily image pull
                                  # docker_daily_restart.sh still runs regardless
                                  # update and restart are independent

WEEKLY_REMAINING_UPDATES=true    # enable/disable weekly remainder pull + prune
                                  # to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS

docker_update.sh in normal mode targets DAILY_RESTART_CONTAINERS — the same list used by docker_daily_restart.sh. No second list to maintain.

docker_update_remaining.sh derives its target list automatically: all running containers minus DAILY_RESTART_CONTAINERS minus WEEKLY_RESTART_CONTAINERS. Everything gets updated at least once per week with no explicit configuration.

---

━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

── master_host*.conf ────────────────────────────────────────────────────────

# Per-host — varies between HOST1 and HOST2

# Tier 1 — memory limits ("ContainerName:LimitInMB")
HOST1_WATCHDOG_CONTAINERS=()

# Tier 1 — HTTP health check endpoints ("ContainerName:http://host:port")
HOST1_WATCHDOG_CONTAINER_URLS=()

# Tier 1 — must always be running
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()

# Tier 1 + 2 — dependency ordering ("Dependent:dep1 dep2")
HOST1_WATCHDOG_DEPENDENCIES=()

# Daily restart list (also drives docker_update.sh normal mode)
HOST1_DAILY_RESTART_CONTAINERS=()

# Weekly restart list
HOST1_WEEKLY_RESTART_CONTAINERS=()

# Networks to ensure exist
HOST1_NETWORK_CONNECT_NETWORKS=()

# Containers to connect to every configured network
HOST1_NETWORK_CONNECT_CONTAINERS=()

---

── master.conf ──────────────────────────────────────────────────────────────

# Shared — applies to both servers

# ── Watchdog timing ────────────────────────────────────────────────────────
DOCKER_WATCHDOG_INTERVAL=900         # seconds between cycles (15 minutes)
WATCHDOG_STARTUP_GRACE=600           # seconds before restarts begin after boot
CONTAINER_DELAY=15                   # seconds between dependency + dependent restart

# ── Memory ─────────────────────────────────────────────────────────────────
SOFT_MEM_THRESHOLD=80                # warn at % of hard limit (no restart)

# ── CPU ────────────────────────────────────────────────────────────────────
SOFT_CPU_THRESHOLD=50                # warn threshold — normalised %
HARD_CPU_THRESHOLD=85                # strike threshold — normalised %
CPU_FAIL_LIMIT=2                     # consecutive strikes before restart

# ── HTTP health check ──────────────────────────────────────────────────────
CURL_TIMEOUT=5                       # seconds before curl times out
RESP_FAIL_LIMIT=2                    # consecutive failures before restart

# ── Restart loop protection ────────────────────────────────────────────────
WATCHDOG_CONTAINER_RESTART_LIMIT=3   # restarts before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1  # rolling window in hours

# ── Tier 2 global scan ─────────────────────────────────────────────────────
WATCHDOG_SCAN_ALL=true
WATCHDOG_SCAN_IGNORE=()
WATCHDOG_RESTART_UNHEALTHY=true
WATCHDOG_NOTIFY_OOM=true
WATCHDOG_NOTIFY_CRASHLOOP=true
WATCHDOG_CRASH_LIMIT=5
WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true

# ── Notifications ──────────────────────────────────────────────────────────
WATCHDOG_BATCH_NOTIFY=true

# ── Container updates ──────────────────────────────────────────────────────
DAILY_CONTAINER_UPDATES=true
WEEKLY_REMAINING_UPDATES=true

# ── Retry behaviour (shared by restart scripts) ────────────────────────────
RETRY_COUNT=3                        # retry attempts before marking failed
SLEEP=5                              # seconds between retry attempts

---

━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

── Adding a Container to Monitoring ────────────────────────────────────────

Adding a container is purely additive — add the relevant lines to master_host*.conf. No script changes. detect_hosts() picks up the new config on the next watchdog cycle.

# master_host1.conf — example: adding "MyApp" to full Tier 1 + daily restarts

# 1. Memory hard limit — size at ~150-200% of normal peak (check: docker stats MyApp)
HOST1_WATCHDOG_CONTAINERS=(
    ...existing...
    "MyApp:2048"    # 2GB ceiling
)

# 2. HTTP health check
HOST1_WATCHDOG_CONTAINER_URLS=(
    ...existing...
    "MyApp:http://localhost:8080/health"    # or root URL if no /health endpoint
)

# 3. Required — add only if absence breaks other services
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
    ...existing...
    "MyApp"
)

# 4. Dependency — add if MyApp needs another container up first
HOST1_WATCHDOG_DEPENDENCIES=(
    ...existing...
    "MyApp:MyApp-Database"
)

# 5. Daily restart — add if MyApp degrades over time
HOST1_DAILY_RESTART_CONTAINERS=(
    ...existing...
    "MyApp"    # also adds it to the daily image update
)

Tier 2 picks up MyApp automatically — no configuration needed. It will be included in the global health scan from the next cycle onward.

To exclude MyApp from Tier 2 (e.g. it's a one-shot container that exits normally):

# master.conf
WATCHDOG_SCAN_IGNORE=(
    "MyApp"    # one-shot — exits cleanly, don't treat as crash
)

---

── Skip List Recovery ────────────────────────────────────────────────────────

Used when docker_watchdog.sh has skip-listed a container after exhausting restart attempts. The watchdog stops touching it and sends a critical notification. Human intervention required.

# Step 1 — understand the situation
# Shows skip list contents, container states, restart history
watchdog_skip_list_manager.sh --status

# Step 2 — fix the underlying problem first
# Check logs:           docker logs ContainerName --tail 100
# Check disk:           df -h /mnt/user
# Check database:       docker exec ContainerName sqlite3 /path/to.db ".tables"
# Fix before clearing — clearing without fixing just resets the counter

# Step 3 — clear the container from the skip list + restart history
# Clearing history is important: the counter carries over otherwise and
# the container hits the limit again almost immediately on any startup trouble
watchdog_skip_list_manager.sh --clear ContainerName

# Step 4 — start the container manually
# Confirms your fix worked before handing back to the watchdog
docker start ContainerName

# Step 5 — monitoring resumes automatically
# Next watchdog cycle: container seen running → removed from skip list
# Restart history clean. Back to normal.

⚠️ If docker_watchdog.sh is currently running when you clear the skip list, it may re-add the container on its very next cycle if the container is still in a bad state. The script detects this and warns you. Fix the root cause before clearing.

Other skip list actions:

watchdog_skip_list_manager.sh                        # status (default)
watchdog_skip_list_manager.sh --clear-all            # clear everything
watchdog_skip_list_manager.sh --clear-all --force    # non-interactive
watchdog_skip_list_manager.sh --clear-all --dry-run  # preview what would clear

---

━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

All scripts support these standard flags:

Flag	What it does
--dry-run	Preview actions without making changes. Shows exactly what would happen.
--status	Show current config, container states, and relevant runtime info, then exit.
--log	Verbose mode — adds per-container banners, action lines, pull output, and list details.

Output tiers: Scripts have two output levels. Without --log, each script processes silently and always concludes with a summary block: identity, duration, counts, and a status line. Per-container detail — individual container names, pull Status lines, skip reasons — only appears with --log. Warnings and errors are always visible regardless of --log.

docker_watchdog.sh is the exception — it runs 96 cycles/day as a background daemon and is silent on clean cycles by design. Its summary block only fires when there are restarts or warnings. Use --log to see per-cycle detail on clean cycles.

docker_watchdog.sh --status shows:

Strike counts for all monitored containers, current skip list contents, whether grace period is active and how long remains, whether RAM emergency deferral is active, last cycle timing.

docker_watchdog.sh --dry-run shows:

A full watchdog cycle without restarting anything. Shows what the watchdog would do based on current container states. Useful for verifying configuration before enabling automatic restarts.

docker_update.sh --remainder

Switches to remainder mode — updates all running containers not in the managed daily/weekly lists. Called by weekly_sync_maintenance.sh. Can be run manually to sweep containers that haven't been updated recently.