Varaverk/Docker_Essentials/Manual-Docker_Essentials.md

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🐳 DOCKER ESSENTIALS — Manual

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Configuration reference, setup procedures, and operational workflows. For folder overview and design philosophy see README-Docker_Essentials.md. For per-script detail see the script headers directly.

Watchdog configuration has moved. docker_watchdog.sh is now in Watchdogs/. Memory limits, CPU thresholds, HTTP health checks, dependency ordering, skip list recovery, and all watchdog config vars are in Watchdogs/Manual-Watchdogs.md.

---

━━━ RESTART SCHEDULE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

---

── Daily Restart List ───────────────────────────────────────────────────────

# host1.conf
# Restarted every night at 1am via daily_sync_maintenance.sh.
#
# Good candidates:
#   Reverse proxies         — connection table fills slowly over weeks
#   Authentication services — session cache benefits from periodic clearing
#   Live TV schedulers      — accumulated scheduling state slows decisions
#   Download managers       — connection pool maintenance
#
HOST1_DAILY_RESTART_CONTAINERS=(
    "NginxProxyManager"     # connection table fills slowly over weeks
    "Authelia"              # session cache benefits from periodic clearing
    "Dispatcharr"           # Live TV scheduler accumulates state
    "Dispatcharr-Basic"     # secondary Live TV scheduler — same reason
    "ErsatzTV-Emby"         # channel schedule builder, stale entries accumulate
)

This list also drives docker_update.sh in normal mode — containers added here get their images updated daily before the restart. Add a container once, it gets both.

---

── Weekly Restart List ──────────────────────────────────────────────────────

# host1.conf
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
#
# Daily vs Weekly decision:
#   Daily:  connection-heavy infrastructure — degrades faster (proxy, auth, Live TV)
#   Weekly: productivity and media services — degrades slowly (NextCloud, AdGuard, Immich)
#
HOST1_WEEKLY_RESTART_CONTAINERS=(
    "NextCloud"       # file sync — benefits from clean weekly start
    "AdGuard-Home"    # DNS — cache and stat accumulation
    "Immich"          # photo library — index/cache maintenance
)

---

━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

# host1.conf
# Networks to ensure exist + containers to connect to each network.
# Many-to-many: every container connects to every network listed.
#
# Containers do not need to be running — script handles missing containers
# gracefully (warns + skips). They connect on the next array start.
#
HOST1_NETWORK_CONNECT_NETWORKS=(
    "high-availability"     # main internal network — most containers should be on this
)

HOST1_NETWORK_CONNECT_CONTAINERS=(
    "memcached"             # NextCloud's cache — needs to reach NextCloud AIO network
    "Npm-CrowdSec"          # CrowdSec bouncer — needs to reach NPM's network
)

Timing dependency: Networks created by Docker Compose stacks (e.g. NextCloud AIO) only exist after those stacks start. If this script runs before the Compose stack, the network won't exist yet and the connection fails this run. It will succeed on the next array start. Schedule Compose stacks early in ARRAY_START_SCRIPTS order to minimise the window.

---

━━━ CONTAINER UPDATE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

# master.conf
DAILY_CONTAINER_UPDATES=true      # enable/disable the daily image pull
                                  # docker_daily_restart.sh still runs regardless
                                  # update and restart are independent

WEEKLY_CONTAINER_UPDATES=true     # enable/disable the weekly image pull
                                  # docker_weekly_restart.sh still runs regardless

MONTHLY_REMAINING_UPDATES=true    # enable/disable the monthly remainder pull
                                  # to disable: set false, or remove
                                  # "docker_update.sh --remainder" from
                                  # MONTHLY_MAINTENANCE_SCRIPTS

There is one update script, docker_update.sh, with three modes. Each reuses the restart list it pairs with, so there is no second list to maintain:

Mode	Targets	Runs
(default)	DAILY_RESTART_CONTAINERS	Daily, before docker_daily_restart.sh
--weekly	WEEKLY_RESTART_CONTAINERS	Weekly, before docker_weekly_restart.sh
--remainder	derived, see below	Monthly, via MONTHLY_MAINTENANCE_SCRIPTS

Remainder mode needs no configuration at all. It takes every running container and subtracts:

  DAILY_RESTART_CONTAINERS                          already updated daily
  WEEKLY_RESTART_CONTAINERS                         already updated weekly
  PROFILE_CRITICAL_CONTAINER_NAMES[emby]            updated inline by the weekly sync window
  PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]   updated inline by the weekly sync window
  FALLBACK_<REMOTE>_TIER1..4                        owned by the remote's update cycle

Stopped containers are never targeted in any mode — pulling for a stopped container adds nothing, and it was most likely stopped deliberately.

The fallback exclusion is a correctness rule, not an optimisation. This server only runs those containers during a fallback; the remote owns their version. If remainder updated them independently and a handback then occurred, the remote's older image could meet data written by the newer one.

Ordering matters. The update always runs before its matching restart so the restart lands on the freshly pulled image. If a container's image actually changed, docker_update.sh rebuilds it from its template (a plain docker restart reuses the image ID baked in at creation time and would never pick up the new digest) and records it in DOCKER_UPDATE_REBUILT_*_FILE. The restart script reads that file and skips those containers rather than restarting them a second time — and discards the file as stale if it is older than DOCKER_UPDATE_REBUILT_STALE_HOURS.

---

━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

── host*.conf ────────────────────────────────────────────────────────

# Per-host — varies between HOST1 and HOST2

# Daily restart list (also drives docker_update.sh normal mode)
HOST1_DAILY_RESTART_CONTAINERS=()

# Weekly restart list
HOST1_WEEKLY_RESTART_CONTAINERS=()

# Networks to ensure exist
HOST1_NETWORK_CONNECT_NETWORKS=()

# Containers to connect to every configured network
HOST1_NETWORK_CONNECT_CONTAINERS=()

# Watchdog config (memory limits, HTTP checks, required, dependencies):
# → see Watchdogs/Manual-Watchdogs.md

---

── master.conf ──────────────────────────────────────────────────────────────

# Shared — applies to both servers

# ── Container updates ──────────────────────────────────────────────────
DAILY_CONTAINER_UPDATES=true         # daily pull (DAILY_RESTART_CONTAINERS)
WEEKLY_CONTAINER_UPDATES=true        # weekly pull (WEEKLY_RESTART_CONTAINERS)
MONTHLY_REMAINING_UPDATES=true       # monthly pull (everything else)

# Handoff between update and restart — written by docker_update.sh,
# read by the matching restart script so it skips containers already
# rebuilt onto a new image this run.
DOCKER_UPDATE_REBUILT_DAILY_FILE
DOCKER_UPDATE_REBUILT_WEEKLY_FILE
DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all

# ── Retry behaviour (shared by restart scripts) ────────────────────────
RETRY_COUNT=3                        # retry attempts before marking failed
SLEEP=5                              # seconds between retry attempts
CONTAINER_DELAY                      # seconds between a dependency and its dependents
RESTART_VERIFY_WAIT=3                # settle time before verifying a restart stuck

# Watchdog thresholds (CPU, memory, HTTP, restart loop):
# → see Watchdogs/Manual-Watchdogs.md

---

━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

── Adding a Container to Monitoring ────────────────────────────────────────

Adding a container to watchdog monitoring is purely additive — add lines to host*.conf. No script changes. detect_hosts() picks up the new config on the next watchdog cycle. See Watchdogs/Manual-Watchdogs.md for the full procedure.

To add a container to daily restarts (and daily image updates):

# host1.conf
HOST1_DAILY_RESTART_CONTAINERS=(
    ...existing...
    "MyApp"    # also adds it to the daily image update
)

To exclude a container from Tier 2 global scan (e.g. one-shot that exits normally):

# master.conf
WATCHDOG_SCAN_IGNORE=(
    "MyApp"    # one-shot — exits cleanly, don't treat as crash
)

---

── Skip List Recovery ────────────────────────────────────────────────────────

When docker_watchdog.sh skip-lists a container, use Tools/watchdog_skip_list_manager.sh. Full procedure in Watchdogs/Manual-Watchdogs.md.

Tools/watchdog_skip_list_manager.sh --status          # see skip list + container states
Tools/watchdog_skip_list_manager.sh --clear MyApp     # clear after fixing root cause
Tools/watchdog_skip_list_manager.sh --clear-all       # clear everything

---

━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

All scripts support these standard flags:

Flag	What it does
--dry-run	Preview actions without making changes. Shows exactly what would happen.
--status	Show current config, container states, and relevant runtime info, then exit.
--log	Verbose mode — adds per-container banners, action lines, pull output, and list details.

Output tiers: Without --log, each script processes silently and concludes with a summary block: identity, duration, counts, and a status line. Per-container detail only appears with --log. Warnings and errors are always visible regardless of --log.

Script-specific flags

Script	Flag	What it does
docker_update.sh	--weekly	Target WEEKLY_RESTART_CONTAINERS. Called by weekly_sync_maintenance.sh before the weekly restart.
docker_update.sh	--remainder	Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by monthly_maintenance.sh. Safe to run manually to sweep anything missed.
media_cleaner.sh (Media/)	<anime|media>	Required positional profile — there is no default.

docker_update.sh with no mode flag is normal mode: DAILY_RESTART_CONTAINERS, called by daily_sync_maintenance.sh before the daily restart.

Exit codes

Code	Meaning
0	Success, or nothing to do (empty list, disabled toggle, no running containers)
1	One or more containers failed — the summary names them

docker_update.sh deliberately exits 0 even when pulls fail: a failed pull is not a reason to abort the restart that follows, which simply proceeds on the existing image. The failure is reported in the summary.