Gmer4Lfe
8a2707ee37
The docs had drifted from the scripts — a script that no longer exists, three wrong variable names, a reversed run order, and seven scheduled scripts that were never documented at all.
279 lines
13 KiB
Markdown
279 lines
13 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 🐳 DOCKER ESSENTIALS — Manual
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Configuration reference, setup procedures, and operational workflows.
|
|
For folder overview and design philosophy see `README-Docker_Essentials.md`.
|
|
For per-script detail see the script headers directly.
|
|
|
|
> **Watchdog configuration has moved.** `docker_watchdog.sh` is now in `Watchdogs/`.
|
|
> Memory limits, CPU thresholds, HTTP health checks, dependency ordering, skip list
|
|
> recovery, and all watchdog config vars are in `Watchdogs/Manual-Watchdogs.md`.
|
|
|
|
---
|
|
|
|
## ━━━ RESTART SCHEDULE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### ── Daily Restart List ───────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# host1.conf
|
|
# Restarted every night at 1am via daily_sync_maintenance.sh.
|
|
#
|
|
# Good candidates:
|
|
# Reverse proxies — connection table fills slowly over weeks
|
|
# Authentication services — session cache benefits from periodic clearing
|
|
# Live TV schedulers — accumulated scheduling state slows decisions
|
|
# Download managers — connection pool maintenance
|
|
#
|
|
HOST1_DAILY_RESTART_CONTAINERS=(
|
|
"NginxProxyManager" # connection table fills slowly over weeks
|
|
"Authelia" # session cache benefits from periodic clearing
|
|
"Dispatcharr" # Live TV scheduler accumulates state
|
|
"Dispatcharr-Basic" # secondary Live TV scheduler — same reason
|
|
"ErsatzTV-Emby" # channel schedule builder, stale entries accumulate
|
|
)
|
|
```
|
|
|
|
This list also drives `docker_update.sh` in normal mode — containers added here get
|
|
their images updated daily before the restart. Add a container once, it gets both.
|
|
|
|
---
|
|
|
|
### ── Weekly Restart List ──────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# host1.conf
|
|
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
|
|
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
|
|
#
|
|
# Daily vs Weekly decision:
|
|
# Daily: connection-heavy infrastructure — degrades faster (proxy, auth, Live TV)
|
|
# Weekly: productivity and media services — degrades slowly (NextCloud, AdGuard, Immich)
|
|
#
|
|
HOST1_WEEKLY_RESTART_CONTAINERS=(
|
|
"NextCloud" # file sync — benefits from clean weekly start
|
|
"AdGuard-Home" # DNS — cache and stat accumulation
|
|
"Immich" # photo library — index/cache maintenance
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```bash
|
|
# host1.conf
|
|
# Networks to ensure exist + containers to connect to each network.
|
|
# Many-to-many: every container connects to every network listed.
|
|
#
|
|
# Containers do not need to be running — script handles missing containers
|
|
# gracefully (warns + skips). They connect on the next array start.
|
|
#
|
|
HOST1_NETWORK_CONNECT_NETWORKS=(
|
|
"high-availability" # main internal network — most containers should be on this
|
|
)
|
|
|
|
HOST1_NETWORK_CONNECT_CONTAINERS=(
|
|
"memcached" # NextCloud's cache — needs to reach NextCloud AIO network
|
|
"Npm-CrowdSec" # CrowdSec bouncer — needs to reach NPM's network
|
|
)
|
|
```
|
|
|
|
> **Timing dependency:** Networks created by Docker Compose stacks (e.g. NextCloud AIO)
|
|
> only exist after those stacks start. If this script runs before the Compose stack,
|
|
> the network won't exist yet and the connection fails this run. It will succeed on the
|
|
> next array start. Schedule Compose stacks early in `ARRAY_START_SCRIPTS` order to
|
|
> minimise the window.
|
|
|
|
---
|
|
|
|
## ━━━ CONTAINER UPDATE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```bash
|
|
# master.conf
|
|
DAILY_CONTAINER_UPDATES=true # enable/disable the daily image pull
|
|
# docker_daily_restart.sh still runs regardless
|
|
# update and restart are independent
|
|
|
|
WEEKLY_CONTAINER_UPDATES=true # enable/disable the weekly image pull
|
|
# docker_weekly_restart.sh still runs regardless
|
|
|
|
MONTHLY_REMAINING_UPDATES=true # enable/disable the monthly remainder pull
|
|
# to disable: set false, or remove
|
|
# "docker_update.sh --remainder" from
|
|
# MONTHLY_MAINTENANCE_SCRIPTS
|
|
```
|
|
|
|
There is one update script, `docker_update.sh`, with three modes. Each reuses the restart
|
|
list it pairs with, so there is no second list to maintain:
|
|
|
|
| Mode | Targets | Runs |
|
|
|------|---------|------|
|
|
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` |
|
|
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` |
|
|
| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` |
|
|
|
|
Remainder mode needs no configuration at all. It takes every **running** container and
|
|
subtracts:
|
|
|
|
```
|
|
DAILY_RESTART_CONTAINERS already updated daily
|
|
WEEKLY_RESTART_CONTAINERS already updated weekly
|
|
PROFILE_CRITICAL_CONTAINER_NAMES[emby] updated inline by the weekly sync window
|
|
PROFILE_CRITICAL_CONTAINER_NAMES[critical-data] updated inline by the weekly sync window
|
|
FALLBACK_<REMOTE>_TIER1..4 owned by the remote's update cycle
|
|
```
|
|
|
|
Stopped containers are never targeted in any mode — pulling for a stopped container adds
|
|
nothing, and it was most likely stopped deliberately.
|
|
|
|
The fallback exclusion is a correctness rule, not an optimisation. This server only runs
|
|
those containers during a fallback; the remote owns their version. If remainder updated them
|
|
independently and a handback then occurred, the remote's older image could meet data written
|
|
by the newer one.
|
|
|
|
**Ordering matters.** The update always runs *before* its matching restart so the restart
|
|
lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh`
|
|
rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at
|
|
creation time and would never pick up the new digest) and records it in
|
|
`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers
|
|
rather than restarting them a second time — and discards the file as stale if it is older
|
|
than `DOCKER_UPDATE_REBUILT_STALE_HOURS`.
|
|
|
|
---
|
|
|
|
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
### ── host*.conf ────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Per-host — varies between HOST1 and HOST2
|
|
|
|
# Daily restart list (also drives docker_update.sh normal mode)
|
|
HOST1_DAILY_RESTART_CONTAINERS=()
|
|
|
|
# Weekly restart list
|
|
HOST1_WEEKLY_RESTART_CONTAINERS=()
|
|
|
|
# Networks to ensure exist
|
|
HOST1_NETWORK_CONNECT_NETWORKS=()
|
|
|
|
# Containers to connect to every configured network
|
|
HOST1_NETWORK_CONNECT_CONTAINERS=()
|
|
|
|
# Watchdog config (memory limits, HTTP checks, required, dependencies):
|
|
# → see Watchdogs/Manual-Watchdogs.md
|
|
```
|
|
|
|
---
|
|
|
|
### ── master.conf ──────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Shared — applies to both servers
|
|
|
|
# ── Container updates ──────────────────────────────────────────────────
|
|
DAILY_CONTAINER_UPDATES=true # daily pull (DAILY_RESTART_CONTAINERS)
|
|
WEEKLY_CONTAINER_UPDATES=true # weekly pull (WEEKLY_RESTART_CONTAINERS)
|
|
MONTHLY_REMAINING_UPDATES=true # monthly pull (everything else)
|
|
|
|
# Handoff between update and restart — written by docker_update.sh,
|
|
# read by the matching restart script so it skips containers already
|
|
# rebuilt onto a new image this run.
|
|
DOCKER_UPDATE_REBUILT_DAILY_FILE
|
|
DOCKER_UPDATE_REBUILT_WEEKLY_FILE
|
|
DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all
|
|
|
|
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
|
RETRY_COUNT=3 # retry attempts before marking failed
|
|
SLEEP=5 # seconds between retry attempts
|
|
CONTAINER_DELAY # seconds between a dependency and its dependents
|
|
RESTART_VERIFY_WAIT=3 # settle time before verifying a restart stuck
|
|
|
|
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
|
# → see Watchdogs/Manual-Watchdogs.md
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
### ── Adding a Container to Monitoring ────────────────────────────────────────
|
|
|
|
Adding a container to watchdog monitoring is purely additive — add lines to `host*.conf`.
|
|
No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
|
|
See `Watchdogs/Manual-Watchdogs.md` for the full procedure.
|
|
|
|
To add a container to **daily restarts** (and daily image updates):
|
|
|
|
```bash
|
|
# host1.conf
|
|
HOST1_DAILY_RESTART_CONTAINERS=(
|
|
...existing...
|
|
"MyApp" # also adds it to the daily image update
|
|
)
|
|
```
|
|
|
|
To **exclude** a container from Tier 2 global scan (e.g. one-shot that exits normally):
|
|
|
|
```bash
|
|
# master.conf
|
|
WATCHDOG_SCAN_IGNORE=(
|
|
"MyApp" # one-shot — exits cleanly, don't treat as crash
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Skip List Recovery ────────────────────────────────────────────────────────
|
|
|
|
When `docker_watchdog.sh` skip-lists a container, use `Tools/watchdog_skip_list_manager.sh`.
|
|
Full procedure in `Watchdogs/Manual-Watchdogs.md`.
|
|
|
|
```bash
|
|
Tools/watchdog_skip_list_manager.sh --status # see skip list + container states
|
|
Tools/watchdog_skip_list_manager.sh --clear MyApp # clear after fixing root cause
|
|
Tools/watchdog_skip_list_manager.sh --clear-all # clear everything
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All scripts support these standard flags:
|
|
|
|
| Flag | What it does |
|
|
|------|-------------|
|
|
| `--dry-run` | Preview actions without making changes. Shows exactly what would happen. |
|
|
| `--status` | Show current config, container states, and relevant runtime info, then exit. |
|
|
| `--log` | Verbose mode — adds per-container banners, action lines, pull output, and list details. |
|
|
|
|
**Output tiers:** Without `--log`, each script processes silently and concludes with a
|
|
summary block: identity, duration, counts, and a status line. Per-container detail only
|
|
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
|
|
|
### Script-specific flags
|
|
|
|
| Script | Flag | What it does |
|
|
|--------|------|-------------|
|
|
| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. |
|
|
| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. |
|
|
| `media_cleaner.sh` *(Media/)* | `<anime\|media>` | Required positional profile — there is no default. |
|
|
|
|
`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by
|
|
`daily_sync_maintenance.sh` before the daily restart.
|
|
|
|
### Exit codes
|
|
|
|
| Code | Meaning |
|
|
|------|---------|
|
|
| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) |
|
|
| `1` | One or more containers failed — the summary names them |
|
|
|
|
`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason
|
|
to abort the restart that follows, which simply proceeds on the existing image. The failure
|
|
is reported in the summary.
|