# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🐳  DOCKER ESSENTIALS — Manual
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Configuration reference, setup procedures, and operational workflows.
For folder overview and design philosophy see `README-Docker_Essentials.md`.
For per-script detail see the script headers directly.

> **Watchdog configuration has moved.** `docker_watchdog.sh` is now in `Watchdogs/`.
> Memory limits, CPU thresholds, HTTP health checks, dependency ordering, skip list
> recovery, and all watchdog config vars are in `Watchdogs/Manual-Watchdogs.md`.

---

## ━━━ RESTART SCHEDULE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

---

### ── Daily Restart List ───────────────────────────────────────────────────────

```bash
# host1.conf
# Restarted every night at 1am via daily_sync_maintenance.sh.
#
# Good candidates:
#   Reverse proxies         — connection table fills slowly over weeks
#   Authentication services — session cache benefits from periodic clearing
#   Live TV schedulers      — accumulated scheduling state slows decisions
#   Download managers       — connection pool maintenance
#
HOST1_DAILY_RESTART_CONTAINERS=(
    "NginxProxyManager"     # connection table fills slowly over weeks
    "Authelia"              # session cache benefits from periodic clearing
    "Dispatcharr"           # Live TV scheduler accumulates state
    "Dispatcharr-Basic"     # secondary Live TV scheduler — same reason
    "ErsatzTV-Emby"         # channel schedule builder, stale entries accumulate
)
```

This list also drives `docker_update.sh` in normal mode — containers added here get
their images updated daily before the restart. Add a container once, it gets both.

---

### ── Weekly Restart List ──────────────────────────────────────────────────────

```bash
# host1.conf
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
#
# Daily vs Weekly decision:
#   Daily:  connection-heavy infrastructure — degrades faster (proxy, auth, Live TV)
#   Weekly: productivity and media services — degrades slowly (NextCloud, AdGuard, Immich)
#
HOST1_WEEKLY_RESTART_CONTAINERS=(
    "NextCloud"       # file sync — benefits from clean weekly start
    "AdGuard-Home"    # DNS — cache and stat accumulation
    "Immich"          # photo library — index/cache maintenance
)
```

---

## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```bash
# host1.conf
# Networks to ensure exist + containers to connect to each network.
# Many-to-many: every container connects to every network listed.
#
# Containers do not need to be running — script handles missing containers
# gracefully (warns + skips). They connect on the next array start.
#
HOST1_NETWORK_CONNECT_NETWORKS=(
    "high-availability"     # main internal network — most containers should be on this
)

HOST1_NETWORK_CONNECT_CONTAINERS=(
    "memcached"             # NextCloud's cache — needs to reach NextCloud AIO network
    "Npm-CrowdSec"          # CrowdSec bouncer — needs to reach NPM's network
)
```

> **Timing dependency:** Networks created by Docker Compose stacks (e.g. NextCloud AIO)
> only exist after those stacks start. If this script runs before the Compose stack,
> the network won't exist yet and the connection fails this run. It will succeed on the
> next array start. Schedule Compose stacks early in `ARRAY_START_SCRIPTS` order to
> minimise the window.

---

## ━━━ CONTAINER UPDATE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```bash
# master.conf
DAILY_CONTAINER_UPDATES=true      # enable/disable the daily image pull
                                  # docker_daily_restart.sh still runs regardless
                                  # update and restart are independent

WEEKLY_CONTAINER_UPDATES=true     # enable/disable the weekly image pull
                                  # docker_weekly_restart.sh still runs regardless

MONTHLY_REMAINING_UPDATES=true    # enable/disable the monthly remainder pull
                                  # to disable: set false, or remove
                                  # "docker_update.sh --remainder" from
                                  # MONTHLY_MAINTENANCE_SCRIPTS
```

There is one update script, `docker_update.sh`, with three modes. Each reuses the restart
list it pairs with, so there is no second list to maintain:

| Mode | Targets | Runs |
|------|---------|------|
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` |
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` |
| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` |

Remainder mode needs no configuration at all. It takes every **running** container and
subtracts:

```
  DAILY_RESTART_CONTAINERS                          already updated daily
  WEEKLY_RESTART_CONTAINERS                         already updated weekly
  PROFILE_CRITICAL_CONTAINER_NAMES[emby]            updated inline by the weekly sync window
  PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]   updated inline by the weekly sync window
  FALLBACK_<REMOTE>_TIER1..4                        owned by the remote's update cycle
```

Stopped containers are never targeted in any mode — pulling for a stopped container adds
nothing, and it was most likely stopped deliberately.

The fallback exclusion is a correctness rule, not an optimisation. This server only runs
those containers during a fallback; the remote owns their version. If remainder updated them
independently and a handback then occurred, the remote's older image could meet data written
by the newer one.

**Ordering matters.** The update always runs *before* its matching restart so the restart
lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh`
rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at
creation time and would never pick up the new digest) and records it in
`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers
rather than restarting them a second time — and discards the file as stale if it is older
than `DOCKER_UPDATE_REBUILT_STALE_HOURS`.

---

## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

### ── host*.conf ────────────────────────────────────────────────────────

```bash
# Per-host — varies between HOST1 and HOST2

# Daily restart list (also drives docker_update.sh normal mode)
HOST1_DAILY_RESTART_CONTAINERS=()

# Weekly restart list
HOST1_WEEKLY_RESTART_CONTAINERS=()

# Networks to ensure exist
HOST1_NETWORK_CONNECT_NETWORKS=()

# Containers to connect to every configured network
HOST1_NETWORK_CONNECT_CONTAINERS=()

# Watchdog config (memory limits, HTTP checks, required, dependencies):
# → see Watchdogs/Manual-Watchdogs.md
```

---

### ── master.conf ──────────────────────────────────────────────────────────────

```bash
# Shared — applies to both servers

# ── Container updates ──────────────────────────────────────────────────
DAILY_CONTAINER_UPDATES=true         # daily pull (DAILY_RESTART_CONTAINERS)
WEEKLY_CONTAINER_UPDATES=true        # weekly pull (WEEKLY_RESTART_CONTAINERS)
MONTHLY_REMAINING_UPDATES=true       # monthly pull (everything else)

# Handoff between update and restart — written by docker_update.sh,
# read by the matching restart script so it skips containers already
# rebuilt onto a new image this run.
DOCKER_UPDATE_REBUILT_DAILY_FILE
DOCKER_UPDATE_REBUILT_WEEKLY_FILE
DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all

# ── Retry behaviour (shared by restart scripts) ────────────────────────
RETRY_COUNT=3                        # retry attempts before marking failed
SLEEP=5                              # seconds between retry attempts
CONTAINER_DELAY                      # seconds between a dependency and its dependents
RESTART_VERIFY_WAIT=3                # settle time before verifying a restart stuck

# Watchdog thresholds (CPU, memory, HTTP, restart loop):
# → see Watchdogs/Manual-Watchdogs.md
```

---

## ━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

### ── Adding a Container to Monitoring ────────────────────────────────────────

Adding a container to watchdog monitoring is purely additive — add lines to `host*.conf`.
No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
See `Watchdogs/Manual-Watchdogs.md` for the full procedure.

To add a container to **daily restarts** (and daily image updates):

```bash
# host1.conf
HOST1_DAILY_RESTART_CONTAINERS=(
    ...existing...
    "MyApp"    # also adds it to the daily image update
)
```

To **exclude** a container from Tier 2 global scan (e.g. one-shot that exits normally):

```bash
# master.conf
WATCHDOG_SCAN_IGNORE=(
    "MyApp"    # one-shot — exits cleanly, don't treat as crash
)
```

---

### ── Skip List Recovery ────────────────────────────────────────────────────────

When `docker_watchdog.sh` skip-lists a container, use `Tools/watchdog_skip_list_manager.sh`.
Full procedure in `Watchdogs/Manual-Watchdogs.md`.

```bash
Tools/watchdog_skip_list_manager.sh --status          # see skip list + container states
Tools/watchdog_skip_list_manager.sh --clear MyApp     # clear after fixing root cause
Tools/watchdog_skip_list_manager.sh --clear-all       # clear everything
```

---

## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

All scripts support these standard flags:

| Flag | What it does |
|------|-------------|
| `--dry-run` | Preview actions without making changes. Shows exactly what would happen. |
| `--status` | Show current config, container states, and relevant runtime info, then exit. |
| `--log` | Verbose mode — adds per-container banners, action lines, pull output, and list details. |

**Output tiers:** Without `--log`, each script processes silently and concludes with a
summary block: identity, duration, counts, and a status line. Per-container detail only
appears with `--log`. Warnings and errors are always visible regardless of `--log`.

### Script-specific flags

| Script | Flag | What it does |
|--------|------|-------------|
| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. |
| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. |
| `media_cleaner.sh` *(Media/)* | `<anime\|media>` | Required positional profile — there is no default. |

`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by
`daily_sync_maintenance.sh` before the daily restart.

### Exit codes

| Code | Meaning |
|------|---------|
| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) |
| `1` | One or more containers failed — the summary names them |

`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason
to abort the restart that follows, which simply proceeds on the existing image. The failure
is reported in the summary.
