Gmer4Lfe
8a3e22c9b8
New docs: Watchdogs/README-Watchdogs.md — design, relationships, script table, state file map Watchdogs/Manual-Watchdogs.md — full config reference for all 4 watchdogs Docker_Essentials/: README — remove docker_watchdog, update folder description and diagrams Manual — strip watchdog config sections, add pointer to Watchdogs/Manual unRAID_Essentials/: README — remove system/resource watchdog, update diagrams and script table Manual — strip system/resource watchdog sections, update TOC + config reference README.md: Add Watchdogs/ to folder structure Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start Fix daily cycle and monitoring diagrams
218 lines
9.5 KiB
Markdown
218 lines
9.5 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 🐳 DOCKER ESSENTIALS — Manual
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Configuration reference, setup procedures, and operational workflows.
|
|
For folder overview and design philosophy see `README-Docker_Essentials.md`.
|
|
For per-script detail see the script headers directly.
|
|
|
|
> **Watchdog configuration has moved.** `docker_watchdog.sh` is now in `Watchdogs/`.
|
|
> Memory limits, CPU thresholds, HTTP health checks, dependency ordering, skip list
|
|
> recovery, and all watchdog config vars are in `Watchdogs/Manual-Watchdogs.md`.
|
|
|
|
---
|
|
|
|
## ━━━ RESTART SCHEDULE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### ── Daily Restart List ───────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# host1.conf
|
|
# Restarted every night at 1am via daily_sync_maintenance.sh.
|
|
#
|
|
# Good candidates:
|
|
# Reverse proxies — connection table fills slowly over weeks
|
|
# Authentication services — session cache benefits from periodic clearing
|
|
# Live TV schedulers — accumulated scheduling state slows decisions
|
|
# Download managers — connection pool maintenance
|
|
#
|
|
HOST1_DAILY_RESTART_CONTAINERS=(
|
|
"NginxProxyManager" # connection table fills slowly over weeks
|
|
"Authelia" # session cache benefits from periodic clearing
|
|
"Dispatcharr" # Live TV scheduler accumulates state
|
|
"Dispatcharr-Basic" # secondary Live TV scheduler — same reason
|
|
"ErsatzTV-Emby" # channel schedule builder, stale entries accumulate
|
|
)
|
|
```
|
|
|
|
This list also drives `docker_update.sh` in normal mode — containers added here get
|
|
their images updated daily before the restart. Add a container once, it gets both.
|
|
|
|
---
|
|
|
|
### ── Weekly Restart List ──────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# host1.conf
|
|
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
|
|
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
|
|
#
|
|
# Daily vs Weekly decision:
|
|
# Daily: connection-heavy infrastructure — degrades faster (proxy, auth, Live TV)
|
|
# Weekly: productivity and media services — degrades slowly (NextCloud, AdGuard, Immich)
|
|
#
|
|
HOST1_WEEKLY_RESTART_CONTAINERS=(
|
|
"NextCloud" # file sync — benefits from clean weekly start
|
|
"AdGuard-Home" # DNS — cache and stat accumulation
|
|
"Immich" # photo library — index/cache maintenance
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```bash
|
|
# host1.conf
|
|
# Networks to ensure exist + containers to connect to each network.
|
|
# Many-to-many: every container connects to every network listed.
|
|
#
|
|
# Containers do not need to be running — script handles missing containers
|
|
# gracefully (warns + skips). They connect on the next array start.
|
|
#
|
|
HOST1_NETWORK_CONNECT_NETWORKS=(
|
|
"high-availability" # main internal network — most containers should be on this
|
|
)
|
|
|
|
HOST1_NETWORK_CONNECT_CONTAINERS=(
|
|
"memcached" # NextCloud's cache — needs to reach NextCloud AIO network
|
|
"Npm-CrowdSec" # CrowdSec bouncer — needs to reach NPM's network
|
|
)
|
|
```
|
|
|
|
> **Timing dependency:** Networks created by Docker Compose stacks (e.g. NextCloud AIO)
|
|
> only exist after those stacks start. If this script runs before the Compose stack,
|
|
> the network won't exist yet and the connection fails this run. It will succeed on the
|
|
> next array start. Schedule Compose stacks early in `ARRAY_START_SCRIPTS` order to
|
|
> minimise the window.
|
|
|
|
---
|
|
|
|
## ━━━ CONTAINER UPDATE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```bash
|
|
# master.conf
|
|
DAILY_CONTAINER_UPDATES=true # enable/disable daily image pull
|
|
# docker_daily_restart.sh still runs regardless
|
|
# update and restart are independent
|
|
|
|
WEEKLY_REMAINING_UPDATES=true # enable/disable weekly remainder pull + prune
|
|
# to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS
|
|
```
|
|
|
|
`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list
|
|
used by `docker_daily_restart.sh`. No second list to maintain.
|
|
|
|
`docker_update_remaining.sh` derives its target list automatically:
|
|
all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`.
|
|
Everything gets updated at least once per week with no explicit configuration.
|
|
|
|
---
|
|
|
|
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
### ── host*.conf ────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Per-host — varies between HOST1 and HOST2
|
|
|
|
# Daily restart list (also drives docker_update.sh normal mode)
|
|
HOST1_DAILY_RESTART_CONTAINERS=()
|
|
|
|
# Weekly restart list
|
|
HOST1_WEEKLY_RESTART_CONTAINERS=()
|
|
|
|
# Networks to ensure exist
|
|
HOST1_NETWORK_CONNECT_NETWORKS=()
|
|
|
|
# Containers to connect to every configured network
|
|
HOST1_NETWORK_CONNECT_CONTAINERS=()
|
|
|
|
# Watchdog config (memory limits, HTTP checks, required, dependencies):
|
|
# → see Watchdogs/Manual-Watchdogs.md
|
|
```
|
|
|
|
---
|
|
|
|
### ── master.conf ──────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Shared — applies to both servers
|
|
|
|
# ── Container updates ──────────────────────────────────────────────────
|
|
DAILY_CONTAINER_UPDATES=true
|
|
WEEKLY_REMAINING_UPDATES=true
|
|
|
|
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
|
RETRY_COUNT=3 # retry attempts before marking failed
|
|
SLEEP=5 # seconds between retry attempts
|
|
|
|
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
|
# → see Watchdogs/Manual-Watchdogs.md
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
### ── Adding a Container to Monitoring ────────────────────────────────────────
|
|
|
|
Adding a container to watchdog monitoring is purely additive — add lines to `host*.conf`.
|
|
No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
|
|
See `Watchdogs/Manual-Watchdogs.md` for the full procedure.
|
|
|
|
To add a container to **daily restarts** (and daily image updates):
|
|
|
|
```bash
|
|
# host1.conf
|
|
HOST1_DAILY_RESTART_CONTAINERS=(
|
|
...existing...
|
|
"MyApp" # also adds it to the daily image update
|
|
)
|
|
```
|
|
|
|
To **exclude** a container from Tier 2 global scan (e.g. one-shot that exits normally):
|
|
|
|
```bash
|
|
# master.conf
|
|
WATCHDOG_SCAN_IGNORE=(
|
|
"MyApp" # one-shot — exits cleanly, don't treat as crash
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Skip List Recovery ────────────────────────────────────────────────────────
|
|
|
|
When `docker_watchdog.sh` skip-lists a container, use `Tools/watchdog_skip_list_manager.sh`.
|
|
Full procedure in `Watchdogs/Manual-Watchdogs.md`.
|
|
|
|
```bash
|
|
Tools/watchdog_skip_list_manager.sh --status # see skip list + container states
|
|
Tools/watchdog_skip_list_manager.sh --clear MyApp # clear after fixing root cause
|
|
Tools/watchdog_skip_list_manager.sh --clear-all # clear everything
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All scripts support these standard flags:
|
|
|
|
| Flag | What it does |
|
|
|------|-------------|
|
|
| `--dry-run` | Preview actions without making changes. Shows exactly what would happen. |
|
|
| `--status` | Show current config, container states, and relevant runtime info, then exit. |
|
|
| `--log` | Verbose mode — adds per-container banners, action lines, pull output, and list details. |
|
|
|
|
**Output tiers:** Without `--log`, each script processes silently and concludes with a
|
|
summary block: identity, duration, counts, and a status line. Per-container detail only
|
|
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
|
|
|
### `docker_update.sh --remainder`
|
|
Switches to remainder mode — updates all running containers not in the managed daily/weekly
|
|
lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers
|
|
that haven't been updated recently.
|