FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: slskd reconnect guard in downloaders_reset, mass v2 sync

Browse Source 
- downloaders_reset: connection check block before slskd API sections;
  triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
  gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
  Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
  common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
2026-05-19 20:00:10 -04:00
parent 5cb16d4b18
commit e13f2fa14f
81 changed files with 12164 additions and 10656 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Docker_Essentials/Manual-Docker_Essentials.md
+546
View File
@@ -0,0 +1,546 @@
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🐳 DOCKER ESSENTIALS — Manual
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Configuration reference, setup procedures, and operational workflows.
For folder overview and design philosophy see `README-Docker_Essentials.md`.
For per-script detail see the script headers directly.
---
## ━━━ WATCHDOG CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All watchdog configuration lives in `master_host*.conf` (per-container lists) and
`master.conf` (shared thresholds and toggles). `detect_hosts()` aliases all
`HOST1_` / `HOST2_` prefixed vars to their unprefixed names at runtime — scripts
always read the right values for the server they're running on.
---
### ── Memory Hard Limits ────────────────────────────────────────────────────────
```bash
# master_host1.conf
# Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
#
# Sizing guidance:
# Check normal peak: docker stats ContainerName
# Set limit at ~150-200% of normal peak
# Emby peaks ~12GB under heavy transcode load → 18GB gives headroom
# without triggering on legitimate load spikes
#
HOST1_WATCHDOG_CONTAINERS=(
"Emby:18432" # 18GB — peaks ~12GB under heavy transcode load
"LidaTube:6144" # 6GB — YouTube downloader, grows with large queues
"Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature
"Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous
)
```
A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
visibility into a container approaching its ceiling before a restart is triggered.
---
### ── CPU Thresholds ────────────────────────────────────────────────────────────
```bash
# master.conf
# CPU is normalised against total core count — meaningful regardless of hardware.
#
# Why normalised:
# 85% on one core of a 16-core machine = 5.3% normalised → ignore it
# 85% normalised on a 16-core machine = 13.6 cores worth → runaway process
#
# CPU uses a STRIKE SYSTEM — not immediate restart like memory.
# Brief spikes are normal (Tdarr encoding, Emby transcoding, SABnzbd unpacking).
# The strike system ignores spikes and acts on sustained high usage.
#
# Strike 1: above HARD_CPU_THRESHOLD this cycle → warn, increment strike
# Strike 2: above threshold next cycle → restart, reset counter
# Recovery: drops below threshold any cycle → reset counter to 0
#
SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only
HARD_CPU_THRESHOLD=85 # strike at 85% normalised
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
```
---
### ── HTTP Health Checks ────────────────────────────────────────────────────────
```bash
# master_host1.conf
# Format: "ContainerName:http://host:port/optional-path"
# Hits the actual service endpoint on every watchdog cycle.
# "Container running" and "service responding" are not the same thing.
#
# Uses a STRIKE SYSTEM — network hiccups and brief restarts happen.
# Two consecutive non-responses before acting prevents false positives.
#
# The path can be a lightweight health endpoint or the root URL.
# Docker's own HEALTHCHECK requires the image to define it — most don't.
# These checks work regardless of what the image defines.
#
HOST1_WATCHDOG_CONTAINER_URLS=(
"Emby:http://localhost:8096" # Emby WebUI root — fast to respond
"NginxProxyManager:http://localhost:81" # NPM admin interface
)
# master.conf
CURL_TIMEOUT=5 # seconds before non-response counts as a failure
RESP_FAIL_LIMIT=2 # consecutive failures before restart
```
---
### ── Required Containers ───────────────────────────────────────────────────────
```bash
# master_host1.conf
# Containers that must always be running.
# Found stopped → watchdog attempts restart every cycle until running or skip-listed.
# Uses the STRIKE SYSTEM — one miss might be mid-restart.
# Persistent failure → skip list → critical notification.
#
# These are the containers whose absence breaks everything else:
#
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
"NginxProxyManager" # reverse proxy — all external traffic routes through this
"Authelia" # SSO authentication — all protected services need it
"Mariadb-Authelia" # Authelia database — must be up before Authelia starts
"Redis-Authelia" # Authelia session cache — same startup dependency
)
```
---
### ── Dependency Ordering ───────────────────────────────────────────────────────
```bash
# master_host1.conf
# Format: "DependentContainer:dependency1 dependency2"
# Multiple dependencies space-separated. All must be running before dependent restarts.
#
# When a container and its dependency are both down:
# → restart the dependency first
# → skip the dependent this cycle
# → next cycle: dependency healthy → dependent restarts cleanly
#
# Without this: Authelia starts, can't connect to MariaDB (still starting),
# exits immediately, strike 1. Next cycle: same, strike 2. Skip list.
# MariaDB was fine the whole time.
#
HOST1_WATCHDOG_DEPENDENCIES=(
"Authelia:Mariadb-Authelia Redis-Authelia"
"Authelia-Secondary:Mariadb-Authelia Redis-Authelia-Secondary"
"NextCloud:Postgres-NextCloud"
)
```
The same dependency configuration is used by `docker_daily_restart.sh` and
`docker_weekly_restart.sh` — configure once, applies everywhere.
---
### ── Startup Grace Period ──────────────────────────────────────────────────────
```bash
# master.conf
# Suppress restart actions for N seconds after array start.
# Checks still run and log — only restart actions are suppressed.
# Clock starts from when the watchdog process itself starts.
#
# Without this: false-positive restarts fire in the first minutes after
# every array start while containers are still initialising.
#
WATCHDOG_STARTUP_GRACE=600 # 10 minutes
```
---
### ── Tier 2 Global Scan ────────────────────────────────────────────────────────
Tier 2 scans every running container when `WATCHDOG_SCAN_ALL=true`. No per-container
configuration required — it's the catch-all for everything not explicitly in Tier 1.
```bash
# master.conf
WATCHDOG_SCAN_ALL=true # enable global scan
WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures
WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills
WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts
WATCHDOG_RESTART_DEAD=true # recover containers in dead state
WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero
WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list
# Containers excluded from Tier 2 entirely.
# Use for containers you intentionally stop/start manually, or containers that
# have benign non-zero exits as part of their normal operation.
WATCHDOG_SCAN_IGNORE=(
"my-one-shot-container" # runs and exits normally — not a crash
)
```
Each toggle is independent — disable any check that produces false positives in your
environment without affecting the others.
---
### ── Restart Loop Protection ──────────────────────────────────────────────────
```bash
# master.conf
# N restarts within a rolling window → skip list + critical notification.
# The watchdog stops touching the container entirely.
# Skip list lives on /boot/config/ — survives reboots intentionally.
# A container bad enough to be skip-listed is still broken after a reboot.
#
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
```
**Auto-clear:** The watchdog checks the skip list every cycle and removes any container
it finds running. If the container recovers on its own, monitoring resumes automatically.
Manual clear is only needed when the container is stuck stopped.
---
### ── Notification Batching ────────────────────────────────────────────────────
```bash
# master.conf
# All events from one cycle collected → one notification at end of cycle.
#
# Why: a shared database going down can cascade 10+ containers failing
# simultaneously. Without batching: 10 individual pings. With batching:
# one summary listing all affected containers. Actionable vs overwhelming.
#
WATCHDOG_BATCH_NOTIFY=true
```
---
### ── State Files Reference ────────────────────────────────────────────────────
| File | Config Var | Location | Resets | Purpose |
|------|-----------|----------|--------|---------|
| Strike counts | `WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | Per-container CPU/HTTP strike counters |
| Skip list | `SYS_WATCHDOG_FAILED_FILE` | `/boot/config/` | Never (manual / auto-clear) | Containers that exhausted restart attempts |
| Restart history | `WATCHDOG_CONTAINER_RESTART_LOG` | `/boot/config/` | Auto-purge after window | Restart loop detection data |
| Shared state | `SYS_WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | RAM emergency flag + cycle heartbeat from `system_watchdog.sh` |
`/tmp/` resets on reboot — correct, strike counts before a reboot are meaningless after it.
`/boot/config/` survives reboots — correct, a skip-listed container is still broken after a reboot.
---
## ━━━ RESTART SCHEDULE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
---
### ── Daily Restart List ───────────────────────────────────────────────────────
```bash
# master_host1.conf
# Restarted every night at 1am via daily_sync_maintenance.sh.
#
# Good candidates:
# Reverse proxies — connection table fills slowly over weeks
# Authentication services — session cache benefits from periodic clearing
# Live TV schedulers — accumulated scheduling state slows decisions
# Download managers — connection pool maintenance
#
HOST1_DAILY_RESTART_CONTAINERS=(
"NginxProxyManager" # connection table fills slowly over weeks
"Authelia" # session cache benefits from periodic clearing
"Dispatcharr" # Live TV scheduler accumulates state
"Dispatcharr-Basic" # secondary Live TV scheduler — same reason
"ErsatzTV-Emby" # channel schedule builder, stale entries accumulate
)
```
This list also drives `docker_update.sh` in normal mode — containers added here get
their images updated daily before the restart. Add a container once, it gets both.
---
### ── Weekly Restart List ──────────────────────────────────────────────────────
```bash
# master_host1.conf
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
#
# Daily vs Weekly decision:
# Daily: connection-heavy infrastructure — degrades faster (proxy, auth, Live TV)
# Weekly: productivity and media services — degrades slowly (NextCloud, AdGuard, Immich)
#
HOST1_WEEKLY_RESTART_CONTAINERS=(
"NextCloud" # file sync — benefits from clean weekly start
"AdGuard-Home" # DNS — cache and stat accumulation
"Immich" # photo library — index/cache maintenance
)
```
---
## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```bash
# master_host1.conf
# Networks to ensure exist + containers to connect to each network.
# Many-to-many: every container connects to every network listed.
#
# Containers do not need to be running — script handles missing containers
# gracefully (warns + skips). They connect on the next array start.
#
HOST1_NETWORK_CONNECT_NETWORKS=(
"high-availability" # main internal network — most containers should be on this
)
HOST1_NETWORK_CONNECT_CONTAINERS=(
"memcached" # NextCloud's cache — needs to reach NextCloud AIO network
"Npm-CrowdSec" # CrowdSec bouncer — needs to reach NPM's network
)
```
> **Timing dependency:** Networks created by Docker Compose stacks (e.g. NextCloud AIO)
> only exist after those stacks start. If this script runs before the Compose stack,
> the network won't exist yet and the connection fails this run. It will succeed on the
> next array start. Schedule Compose stacks early in `ARRAY_START_SCRIPTS` order to
> minimise the window.
---
## ━━━ CONTAINER UPDATE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```bash
# master.conf
DAILY_CONTAINER_UPDATES=true # enable/disable daily image pull
# docker_daily_restart.sh still runs regardless
# update and restart are independent
WEEKLY_REMAINING_UPDATES=true # enable/disable weekly remainder pull + prune
# to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS
```
`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list
used by `docker_daily_restart.sh`. No second list to maintain.
`docker_update_remaining.sh` derives its target list automatically:
all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`.
Everything gets updated at least once per week with no explicit configuration.
---
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
### ── master_host*.conf ────────────────────────────────────────────────────────
```bash
# Per-host — varies between HOST1 and HOST2
# Tier 1 — memory limits ("ContainerName:LimitInMB")
HOST1_WATCHDOG_CONTAINERS=()
# Tier 1 — HTTP health check endpoints ("ContainerName:http://host:port")
HOST1_WATCHDOG_CONTAINER_URLS=()
# Tier 1 — must always be running
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
# Tier 1 + 2 — dependency ordering ("Dependent:dep1 dep2")
HOST1_WATCHDOG_DEPENDENCIES=()
# Daily restart list (also drives docker_update.sh normal mode)
HOST1_DAILY_RESTART_CONTAINERS=()
# Weekly restart list
HOST1_WEEKLY_RESTART_CONTAINERS=()
# Networks to ensure exist
HOST1_NETWORK_CONNECT_NETWORKS=()
# Containers to connect to every configured network
HOST1_NETWORK_CONNECT_CONTAINERS=()
```
---
### ── master.conf ──────────────────────────────────────────────────────────────
```bash
# Shared — applies to both servers
# ── Watchdog timing ────────────────────────────────────────────────────────
DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes)
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
# ── Memory ─────────────────────────────────────────────────────────────────
SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart)
# ── CPU ────────────────────────────────────────────────────────────────────
SOFT_CPU_THRESHOLD=50 # warn threshold — normalised %
HARD_CPU_THRESHOLD=85 # strike threshold — normalised %
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
# ── HTTP health check ──────────────────────────────────────────────────────
CURL_TIMEOUT=5 # seconds before curl times out
RESP_FAIL_LIMIT=2 # consecutive failures before restart
# ── Restart loop protection ────────────────────────────────────────────────
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
# ── Tier 2 global scan ─────────────────────────────────────────────────────
WATCHDOG_SCAN_ALL=true
WATCHDOG_SCAN_IGNORE=()
WATCHDOG_RESTART_UNHEALTHY=true
WATCHDOG_NOTIFY_OOM=true
WATCHDOG_NOTIFY_CRASHLOOP=true
WATCHDOG_CRASH_LIMIT=5
WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
# ── Notifications ──────────────────────────────────────────────────────────
WATCHDOG_BATCH_NOTIFY=true
# ── Container updates ──────────────────────────────────────────────────────
DAILY_CONTAINER_UPDATES=true
WEEKLY_REMAINING_UPDATES=true
# ── Retry behaviour (shared by restart scripts) ────────────────────────────
RETRY_COUNT=3 # retry attempts before marking failed
SLEEP=5 # seconds between retry attempts
```
---
## ━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
### ── Adding a Container to Monitoring ────────────────────────────────────────
Adding a container is purely additive — add the relevant lines to `master_host*.conf`.
No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
```bash
# master_host1.conf — example: adding "MyApp" to full Tier 1 + daily restarts
# 1. Memory hard limit — size at ~150-200% of normal peak (check: docker stats MyApp)
HOST1_WATCHDOG_CONTAINERS=(
...existing...
"MyApp:2048" # 2GB ceiling
)
# 2. HTTP health check
HOST1_WATCHDOG_CONTAINER_URLS=(
...existing...
"MyApp:http://localhost:8080/health" # or root URL if no /health endpoint
)
# 3. Required — add only if absence breaks other services
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
...existing...
"MyApp"
)
# 4. Dependency — add if MyApp needs another container up first
HOST1_WATCHDOG_DEPENDENCIES=(
...existing...
"MyApp:MyApp-Database"
)
# 5. Daily restart — add if MyApp degrades over time
HOST1_DAILY_RESTART_CONTAINERS=(
...existing...
"MyApp" # also adds it to the daily image update
)
```
Tier 2 picks up MyApp automatically — no configuration needed. It will be included in
the global health scan from the next cycle onward.
To **exclude** MyApp from Tier 2 (e.g. it's a one-shot container that exits normally):
```bash
# master.conf
WATCHDOG_SCAN_IGNORE=(
"MyApp" # one-shot — exits cleanly, don't treat as crash
)
```
---
### ── Skip List Recovery ────────────────────────────────────────────────────────
Used when `docker_watchdog.sh` has skip-listed a container after exhausting restart
attempts. The watchdog stops touching it and sends a critical notification. Human
intervention required.
```bash
# Step 1 — understand the situation
# Shows skip list contents, container states, restart history
watchdog_skip_list_manager.sh --status
# Step 2 — fix the underlying problem first
# Check logs: docker logs ContainerName --tail 100
# Check disk: df -h /mnt/user
# Check database: docker exec ContainerName sqlite3 /path/to.db ".tables"
# Fix before clearing — clearing without fixing just resets the counter
# Step 3 — clear the container from the skip list + restart history
# Clearing history is important: the counter carries over otherwise and
# the container hits the limit again almost immediately on any startup trouble
watchdog_skip_list_manager.sh --clear ContainerName
# Step 4 — start the container manually
# Confirms your fix worked before handing back to the watchdog
docker start ContainerName
# Step 5 — monitoring resumes automatically
# Next watchdog cycle: container seen running → removed from skip list
# Restart history clean. Back to normal.
```
> ⚠️ **If `docker_watchdog.sh` is currently running when you clear the skip list**, it
> may re-add the container on its very next cycle if the container is still in a bad
> state. The script detects this and warns you. Fix the root cause **before** clearing.
Other skip list actions:
```bash
watchdog_skip_list_manager.sh # status (default)
watchdog_skip_list_manager.sh --clear-all # clear everything
watchdog_skip_list_manager.sh --clear-all --force # non-interactive
watchdog_skip_list_manager.sh --clear-all --dry-run # preview what would clear
```
---
## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All scripts support these standard flags:
| Flag | What it does |
|------|-------------|
| `--dry-run` | Preview actions without making changes. Shows exactly what would happen. |
| `--status` | Show current config, container states, and relevant runtime info, then exit. |
| `--log` | Verbose output — full detail for every container checked and every decision made. |
### `docker_watchdog.sh --status` shows:
Strike counts for all monitored containers, current skip list contents, whether grace
period is active and how long remains, whether RAM emergency deferral is active, last
cycle timing.
### `docker_watchdog.sh --dry-run` shows:
A full watchdog cycle without restarting anything. Shows what the watchdog would do
based on current container states. Useful for verifying configuration before enabling
automatic restarts.
### `docker_update.sh --remainder`
Switches to remainder mode — updates all running containers not in the managed daily/weekly
lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers
that haven't been updated recently.
Docker_Essentials/README-Docker_Essentials.md
+57 -969
View File
File diff suppressed because it is too large Load Diff
Docker_Essentials/docker_container_stop.sh
+105 -26
View File
@@ -2,37 +2,116 @@
# ==============================================================================================
# ============================= Docker Container Stop ==========================================
# ==============================================================================================
# Stops all running Docker containers one at a time, verifying each is stopped before
# moving to the next. Called by array_stopping.sh as part of a planned shutdown sequence.
#
# ── STOP SEQUENCE PER CONTAINER ──────────────────────────────────────────────────────────────
# 1. docker stop -t 30 (SIGTERM + 30s grace period — docker sends SIGKILL if needed)
# 2. Verify stopped — if still running, retry up to RETRY_COUNT times
# 3. docker kill (SIGKILL) if all retries exhausted
# 4. Final verify — error if still running after force-kill
# Never moves to the next container until the current one is confirmed stopped.
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Gracefully stops all running Docker containers in a verified sequential order.
#
# ── WHY SEQUENTIAL ────────────────────────────────────────────────────────────────────────────
# Containers may have dependencies — stopping one at a time avoids abruptly severing a
# service while its dependents are still running and trying to use it.
# Called by array_stopping.sh during planned shutdowns, maintenance windows,
# and controlled reboot operations.
#
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
# Root check — docker requires root
# Per-container verify — confirmed stopped before proceeding to next
# Retry loop — RETRY_COUNT attempts before escalating to force-kill
# SIGTERM → SIGKILL — graceful then forced, never skips graceful
# DOCKER_TIMEOUT — all docker calls protected against hung daemon
# notify on failures — alert if any container cannot be stopped
# The script guarantees each container is fully stopped before moving to the
# next one — preventing dependency breakage, partial shutdown states, and
# abrupt service termination cascades.
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
# RETRY_COUNT — retry attempts before force-kill (default 3)
# SLEEP — seconds between retries
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Shutdown is processed one container at a time:
#
# 1. docker stop -t 30
# → sends SIGTERM with graceful shutdown window
#
# 2. Verify container state
# → confirm container fully stopped before continuing
#
# 3. Retry if still running
# → up to RETRY_COUNT attempts
#
# 4. Escalate to docker kill
# → SIGKILL only after graceful attempts exhausted
#
# 5. Final verification
# → failure notification if container survives SIGKILL
#
# The script NEVER advances to the next container until the current one
# is confirmed stopped or declared failed.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Sequential Shutdown
# Containers may depend on upstream services still being available during
# shutdown. Sequential processing reduces dependency severance during stop.
#
# Graceful First, Forced Last
# SIGTERM is always attempted before SIGKILL. The script never force-kills
# first unless Docker itself escalates internally after timeout expiration.
#
# Verification Over Assumption
# Docker command success alone is not trusted. Container state is verified
# after every stop attempt.
#
# Fail Loudly
# Containers that cannot be stopped generate notifications and non-zero exit
# status so orchestrators know shutdown integrity was compromised.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Root Enforcement
# Docker operations require root privileges.
#
# Docker Presence Check
# Verifies docker binary exists before execution.
#
# Timeout Protection
# All docker commands wrapped in timeout protection to prevent daemon hangs
# from stalling shutdown indefinitely.
#
# Retry Escalation
# Graceful retries occur before SIGKILL escalation.
#
# Per-Container Validation
# Every container state verified before progressing to the next.
#
# Deterministic Ordering
# Running container list sorted before processing for stable execution order.
#
# Failure Notification
# Containers surviving SIGKILL trigger operator notification.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master.conf
#
# RETRY_COUNT
# Graceful retry attempts before force-kill escalation
#
# SLEEP
# Delay in seconds between retry attempts
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# docker_container_stop.sh
# Stop all running containers in verified sequential order
#
# docker_container_stop.sh --dry-run
# Preview shutdown actions without stopping containers
#
# docker_container_stop.sh --status
# Show currently running containers and configuration state
#
# docker_container_stop.sh --log
# Verbose per-container execution logging
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_container_stop.sh — stop all running containers
# docker_container_stop.sh --dry-run — show which containers would be stopped
# docker_container_stop.sh --status — show running containers and exit
# docker_container_stop.sh --log — verbose output
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Docker_Essentials/docker_daily_restart.sh
+87 -37
View File
@@ -2,52 +2,102 @@
# ==============================================================================================
# ================================= Docker Daily Restart =======================================
# ==============================================================================================
# Restarts or starts all containers in HOST*_DAILY_RESTART_CONTAINERS.
# Called by daily_sync_maintenance.sh via DAILY_MAINTENANCE_SCRIPTS every night at 1am.
# Can also be run manually for ad hoc restarts.
#
# ── WHY DAILY RESTARTS ────────────────────────────────────────────────────────────────────────
# Some containers degrade over time without a restart:
# Dispatcharr — Live TV scheduler accumulates state and slows down
# NginxProxyManager — connection table grows, occasional stale proxy entries
# Authelia — session cache benefits from periodic clearing
# Daily restart is intentional maintenance, not just housekeeping.
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Restarts configured containers every night at 1am as proactive maintenance.
#
# ── BEHAVIOUR ─────────────────────────────────────────────────────────────────────────────────
# Running containers → docker restart (graceful stop + start)
# Stopped containers → left stopped — was down intentionally, do not bring back up
# Missing containers → logged and skipped — not treated as fatal
# Each action uses RETRY_COUNT + SLEEP from master.conf for retry logic.
# Called by daily_sync_maintenance.sh via DAILY_MAINTENANCE_SCRIPTS. Runs inside
# the daily maintenance window — any service downtime is absorbed by a window
# that is already happening. Also drives docker_update.sh in normal mode: the
# same DAILY_RESTART_CONTAINERS list is used for both restarts and image pulls,
# so there is no second list to maintain.
#
# The "was running → restart, was stopped → leave stopped" rule is consistent
# across the entire ecosystem — container state is always respected.
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
# Dependency ordering — containers restart in dependency-safe order using
# HOST*_WATCHDOG_DEPENDENCIES from master_host*.conf. If Authelia depends on
# Mariadb + Redis, those restart first with CONTAINER_DELAY before Authelia starts.
# Proactive Maintenance
# Daily restarts target containers known to degrade over time without
# crossing a clear failure threshold — connection table growth, scheduler
# state accumulation, session cache bloat. The watchdog cannot detect this
# class of degradation. Scheduled restarts clear it before it becomes visible.
#
# Restart verification — after each restart, container state is checked after a short
# settle period. If the container fails to stay running it is marked as failed and
# a notification is sent rather than silently passing.
# State Respect
# Running containers are restarted. Stopped containers are left stopped — they
# were intentionally halted and this script has no authority to override that
# decision. This rule is consistent across the entire ecosystem.
#
# Timeout protection — all docker commands are wrapped in a 30 second timeout.
# A hung Docker daemon cannot cause this script to hang indefinitely.
# Timed-out commands are retried per RETRY_COUNT before marking as failed.
# Dependency-Safe Ordering
# Restarts follow the same dependency ordering used by docker_watchdog.sh.
# Services that other containers depend on restart first. A dependent is never
# restarted while its dependency is still coming up.
#
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
# HOST*_DAILY_RESTART_CONTAINERS — list of containers to restart daily
# Set by detect_hosts() alias → DAILY_RESTART_CONTAINERS used by this script
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
# RETRY_COUNT — retry attempts before giving up on a container
# SLEEP — seconds between retry attempts
# Dependency Ordering
# Containers restart in dependency-safe order using HOST*_WATCHDOG_DEPENDENCIES.
# CONTAINER_DELAY seconds between dependency restart and dependent restart gives
# the dependency time to fully initialise before dependents try to connect.
#
# Restart Verification
# After each restart, container state is checked after a settle period. A
# container that starts and immediately crashes is marked failed and a
# notification is sent — the script does not silently pass a restart that
# did not stick.
#
# Timeout Protection
# All docker commands wrapped in a 30 second timeout. A hung Docker daemon
# cannot cause this script to hang indefinitely. Timed-out commands retry
# per RETRY_COUNT before marking as failed.
#
# Lock Acquisition
# acquire_lock() prevents concurrent execution if a previous run is still active.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
#
# HOST*_DAILY_RESTART_CONTAINERS
# Containers restarted nightly. Also used by docker_update.sh normal mode
# for image pulls — add a container once, it gets both. Aliased by
# detect_hosts() → DAILY_RESTART_CONTAINERS
#
# HOST*_WATCHDOG_DEPENDENCIES
# Dependency ordering shared with docker_watchdog.sh. Aliased by
# detect_hosts() → WATCHDOG_DEPENDENCIES
#
# master.conf
#
# RETRY_COUNT
# Retry attempts before giving up on a container
#
# SLEEP
# Seconds between retry attempts
#
# CONTAINER_DELAY
# Seconds to wait after restarting a dependency before starting its dependents
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# docker_daily_restart.sh
# Restart all containers in DAILY_RESTART_CONTAINERS
#
# docker_daily_restart.sh --dry-run
# Preview which containers would be restarted and which would be skipped
#
# docker_daily_restart.sh --status
# Show configured restart list, container states, and dependency ordering
#
# docker_daily_restart.sh --log
# Verbose per-container execution output
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_daily_restart.sh — normal restart
# docker_daily_restart.sh --dry-run — preview without restarting
# docker_daily_restart.sh --log — verbose output
# docker_daily_restart.sh --status — show config and exit
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Docker_Essentials/docker_network_connect.sh
+89 -35
View File
@@ -2,46 +2,100 @@
# ==============================================================================================
# ================================= Docker Network Connect =====================================
# ==============================================================================================
# Ensures custom Docker networks exist then connects specified containers to them.
# Run once at array start via ARRAY_START_SCRIPTS — idempotent, safe to re-run anytime.
#
# ── WHAT IT DOES ──────────────────────────────────────────────────────────────────────────────
# For each network in NETWORK_CONNECT_NETWORKS:
# 1. Check if network exists
# → missing → create it (bridge driver, Docker assigns subnet automatically)
# notifies on creation — unexpected, usually means unRAID wiped networks
# → exists → skip creation silently
# 2. Connect each container in NETWORK_CONNECT_CONTAINERS to the network
# → already connected → skip cleanly
# → not connected → connect it
# → container not found → warn and skip (not an error — may not be running yet)
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Ensures custom Docker networks exist and connects configured containers to
# them at every array start.
#
# ── USE CASE ──────────────────────────────────────────────────────────────────────────────────
# high-availability is the main custom network — shared by most containers.
# After a unRAID update wipes custom networks → recreated automatically at next array start.
# Containers on their own networks (NextCloud AIO etc.) can be added to
# NETWORK_CONNECT_CONTAINERS so they also join high-availability without touching their
# primary network configuration.
# Called via ARRAY_START_SCRIPTS — runs early in the array start sequence,
# before watchdogs begin their first cycle. Idempotent: safe to re-run at any
# time. Silent when everything is already correct. Notifies when a network had
# to be created — that only happens after a unRAID update wipes custom networks,
# and it is worth knowing when it does.
#
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
# Docker daemon check — verifies daemon is responsive before any network operations
# Command validation — validates unRAID notify script before use
# Timeout protection — all docker commands wrapped in timeout — daemon hangs cannot stall
# Empty array guards — warns and exits cleanly if arrays are unconfigured
# Silent by default — only warnings and errors produce output (v3.4 standard)
# network creation always warns — unexpected, means networks were wiped
# Idempotent — safe to run multiple times, skips what is already correct
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
# HOST*_NETWORK_CONNECT_NETWORKS — networks to ensure exist
# HOST*_NETWORK_CONNECT_CONTAINERS — containers to connect to each network
# Aliased by detect_hosts() — script uses unprefixed names
# For each configured network:
#
# 1. Does the network exist?
# NO → create it (bridge driver, Docker assigns subnet automatically)
# → send notification — creation is unexpected outside post-update recovery
# YES → skip silently
#
# 2. For each configured container:
# Already connected → skip silently
# Not connected → connect it
# Container missing → warn and skip — may not be running yet, not fatal
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Idempotent by Design
# Running this script multiple times produces the same result as running it
# once. Skips anything already in the correct state without error or noise.
#
# Silent When Correct
# Produces no output on a clean run. The absence of output is confirmation
# that everything is already correct.
#
# Notify on Creation
# Network creation is always notified because it should only happen after a
# unRAID update. If it happens regularly something is misconfigured and the
# operator needs to know.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Docker Daemon Check
# Verifies daemon is responsive before any network operations. Network
# commands against a hung daemon hang indefinitely.
#
# Timeout Protection
# All docker commands wrapped in timeout. Daemon hangs cannot stall the
# array start sequence.
#
# Empty Array Guards
# Warns and exits cleanly if NETWORK_CONNECT_NETWORKS or
# NETWORK_CONNECT_CONTAINERS are unconfigured.
#
# Command Validation
# Validates unRAID notify script before use.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
#
# HOST*_NETWORK_CONNECT_NETWORKS
# Networks to ensure exist at array start. Aliased by detect_hosts() →
# NETWORK_CONNECT_NETWORKS
#
# HOST*_NETWORK_CONNECT_CONTAINERS
# Containers to connect to every configured network. Aliased by
# detect_hosts() → NETWORK_CONNECT_CONTAINERS
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# docker_network_connect.sh
# Ensure all configured networks exist and containers are connected
#
# docker_network_connect.sh --dry-run
# Preview what would be created or connected without making changes
#
# docker_network_connect.sh --status
# Show current network state and container connection status
#
# docker_network_connect.sh --log
# Verbose per-network per-container output
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_network_connect.sh — normal run
# docker_network_connect.sh --dry-run — preview without making changes
# docker_network_connect.sh --log — verbose output
# docker_network_connect.sh --status — show current network state and exit
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Docker_Essentials/docker_update.sh
+103 -39
View File
@@ -2,53 +2,117 @@
# ==============================================================================================
# ================================= Docker Update ==============================================
# ==============================================================================================
# Two modes — normal (daily) and remainder (weekly).
#
# ── NORMAL MODE (daily) ───────────────────────────────────────────────────────────────────────
# Pulls the latest image for each container in HOST*_DAILY_RESTART_CONTAINERS.
# Called by daily_sync_maintenance.sh before docker_daily_restart.sh — containers stay
# running during the pull, so there is no extra downtime.
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Pulls the latest images for configured containers. Two modes: normal (daily)
# and remainder (weekly).
#
# ── REMAINDER MODE (weekly) ───────────────────────────────────────────────────────────────────
# Called by weekly_sync_maintenance.sh as the last step.
# Updates all currently running containers that are NOT in:
# DAILY_RESTART_CONTAINERS — already updated daily
# emby + critical-data profiles — already updated inline by the weekly sync window
# FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER* — owned by the remote server's update cycle
# Normal mode is called by daily_sync_maintenance.sh before docker_daily_restart.sh.
# Containers stay running during the pull — no extra downtime beyond what the
# nightly restart already causes.
#
# Fallback containers are excluded because this server only runs them during a failover.
# The remote server is the version owner — if remainder updates them independently and a
# handback writeback occurs, the remote's older version may not handle the newer data.
# This catches everything local-only (Organizr, AdGuard, etc.) once a week.
# Remainder mode is called by weekly_sync_maintenance.sh as the final update step.
# It catches everything that normal mode and the weekly sync window did not already
# update — derived automatically from docker ps, nothing to configure.
#
# ── WHY SAME LIST AS DAILY RESTART (normal mode) ─────────────────────────────────────────────
# Containers that restart daily (auth stack: Authelia, NPM, Mariadb, Redis, etc.) are
# exactly the containers that benefit from staying current. Reusing DAILY_RESTART_CONTAINERS
# means no second list to maintain — add/remove a container once and both update + restart
# reflect the change automatically.
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# ── WHAT THIS DOES ────────────────────────────────────────────────────────────────────────────
# docker pull <image> — fetches the latest digest from the registry
# Old vs new image ID comparison — distinguishes "updated" from "already current"
# Containers keep running — pull does not affect the live container
# docker_daily_restart.sh runs after (normal mode) — containers restart on the fresh image
# Normal mode (daily):
# Targets DAILY_RESTART_CONTAINERS — same list used by docker_daily_restart.sh.
# Pull → compare old vs new image ID → mark updated or already current.
# docker_daily_restart.sh runs after — containers restart onto the fresh image.
#
# ── TOGGLE ────────────────────────────────────────────────────────────────────────────────────
# DAILY_CONTAINER_UPDATES=false in master.conf — skips normal mode, exits cleanly
# docker_daily_restart.sh still runs regardless — update and restart are independent
# Remainder mode has no toggle — exclude it from WEEKLY_MAINTENANCE_SCRIPTS to disable
# Remainder mode (weekly):
# Targets all currently running containers NOT in:
# DAILY_RESTART_CONTAINERS — already updated daily
# emby + critical-data profiles — updated inline by the weekly sync window
# FALLBACK_*_TIER* — owned by the remote server's update cycle
# Pull → compare → restart if updated → prune dangling images.
#
# ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
# master.conf: DAILY_CONTAINER_UPDATES — enable/disable normal mode (default: true)
# master_host*.conf: HOST*_DAILY_RESTART_CONTAINERS — containers to update (normal mode)
# master.conf: PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data] — remainder exclusions
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Single List
# Normal mode reuses DAILY_RESTART_CONTAINERS rather than maintaining a
# separate update list. Adding or removing a container from the restart list
# automatically updates the image pull list — one change, both places.
#
# Version Ownership
# Fallback containers are excluded from remainder mode. This server only runs
# them during a failover. The remote server owns their version — if remainder
# updates them independently and a handback occurs, the remote's older image
# may not handle data written by the newer version.
#
# State Respect
# Stopped containers are never targeted. Pulling while stopped adds no value
# and a stopped container was likely halted intentionally.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Root Enforcement
# Docker operations require root privileges.
#
# DAILY_CONTAINER_UPDATES Toggle
# Normal mode exits cleanly when disabled. docker_daily_restart.sh still runs
# regardless — update and restart are independent operations.
#
# Fallback Exclusion
# Remainder mode excludes containers owned by the remote server's update cycle
# to prevent version divergence across the failover boundary.
#
# Running-Only Filter
# Stopped containers excluded from remainder mode — intentionally down.
#
# Image ID Comparison
# Containers not restarted unless their image actually changed. Pulls that
# result in "already up to date" produce no restart.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master.conf
#
# DAILY_CONTAINER_UPDATES
# Enable or disable normal mode. docker_daily_restart.sh runs regardless.
# (default: true)
#
# PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data]
# Container names for emby and critical-data profiles — excluded from
# remainder mode (already updated by the weekly sync window)
#
# master_host*.conf
#
# HOST*_DAILY_RESTART_CONTAINERS
# Containers updated in normal mode. Aliased by detect_hosts() →
# DAILY_RESTART_CONTAINERS
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# docker_update.sh
# Normal mode — pull latest images for DAILY_RESTART_CONTAINERS
#
# docker_update.sh --remainder
# Remainder mode — pull all running containers not in managed lists,
# restart those that received updates, prune dangling images
#
# docker_update.sh --dry-run
# Preview which containers would be pulled without making changes
#
# docker_update.sh --status
# Show configuration and container list for current mode
#
# docker_update.sh --log
# Verbose per-container pull and comparison output
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_update.sh — normal mode: update DAILY_RESTART_CONTAINERS
# docker_update.sh --remainder — remainder mode: update all except daily + weekly sync containers
# docker_update.sh --dry-run — show what would be pulled
# docker_update.sh --log — verbose output
# docker_update.sh --status — show config and exit
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Docker_Essentials/docker_update_remaining.sh
+60 -30
View File
@@ -2,43 +2,73 @@
# ==============================================================================================
# ============================= Docker Update — Remaining ======================================
# ==============================================================================================
# Pulls the latest image for every running container NOT already covered by the daily or
# weekly update/restart cycles. Restarts containers that received a new image, then prunes
# dangling images. Runs at the end of the weekly maintenance window.
#
# ── WHAT THIS COVERS ──────────────────────────────────────────────────────────────────────────
# Daily update: DAILY_RESTART_CONTAINERS — auth stack, NPM, Dispatcharr, etc.
# Weekly update: WEEKLY_RESTART_CONTAINERS — NextCloud, AdGuard, Immich, etc.
# This script: everything else running on the system (media stack, utilities, etc.)
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Weekly sweep that pulls the latest image for every running container not
# already covered by the daily or weekly managed update cycles. Restarts
# containers that received a new image, then prunes dangling images.
#
# Together the three scripts ensure every deployed container receives at least one image
# pull per week, with no container list to maintain here — it derives the remainder
# automatically from `docker ps` minus the two managed lists.
# Called by weekly_sync_maintenance.sh as the final step in the weekly window.
# Derives its target list automatically from docker ps minus the two managed
# lists — there is nothing to configure for this script.
#
# ── WHAT THIS DOES ────────────────────────────────────────────────────────────────────────────
# 1. Pull latest image for each remaining running container
# 2. Restart containers whose image ID changed (new update landed)
# 3. Prune dangling images left behind by the updates
# Containers already up to date are not restarted.
# Together with docker_update.sh (normal + remainder modes), every deployed
# container receives at least one image pull per week without any per-container
# configuration required here.
#
# ── EXCLUSION LOGIC ───────────────────────────────────────────────────────────────────────────
# Exclusion set = DAILY_RESTART_CONTAINERS + WEEKLY_RESTART_CONTAINERS (aliased by detect_hosts)
# Only running containers are targeted — stopped containers are intentionally excluded
# (stopped = likely paused intentionally; pulling while stopped adds no value).
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# ── TOGGLE ────────────────────────────────────────────────────────────────────────────────────
# WEEKLY_REMAINING_UPDATES=false in master.conf — skips all pulls, exits cleanly
# Root Enforcement
# Docker operations require root privileges.
#
# ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
# master.conf: WEEKLY_REMAINING_UPDATES — enable/disable (default: true)
# master_host*.conf: HOST*_DAILY_RESTART_CONTAINERS — excluded from this script
# HOST*_WEEKLY_RESTART_CONTAINERS — excluded from this script
# WEEKLY_REMAINING_UPDATES Toggle
# Exits cleanly when disabled via master.conf.
#
# Running-Only Filter
# Stopped containers excluded — intentionally down, pulling adds no value.
#
# Image ID Comparison
# Containers not restarted unless their image actually changed.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master.conf
#
# WEEKLY_REMAINING_UPDATES
# Enable or disable this script. (default: true)
# To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
#
# master_host*.conf
#
# HOST*_DAILY_RESTART_CONTAINERS
# Excluded from this script — already updated daily. Aliased by
# detect_hosts() → DAILY_RESTART_CONTAINERS
#
# HOST*_WEEKLY_RESTART_CONTAINERS
# Excluded from this script — already updated by weekly sync window.
# Aliased by detect_hosts() → WEEKLY_RESTART_CONTAINERS
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# docker_update_remaining.sh
# Pull all remaining running containers, restart those updated, prune images
#
# docker_update_remaining.sh --dry-run
# Preview which containers would be pulled and restarted
#
# docker_update_remaining.sh --status
# Show exclusion lists and current remaining container count
#
# docker_update_remaining.sh --log
# Verbose per-container pull and restart output
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_update_remaining.sh — normal run
# docker_update_remaining.sh --dry-run — show which containers would be pulled/restarted
# docker_update_remaining.sh --log — verbose output
# docker_update_remaining.sh --status — show config and exit
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Docker_Essentials/docker_watchdog.sh
+190 -54
View File
@@ -2,81 +2,217 @@
# ==============================================================================================
# ================================= Docker Watchdog ============================================
# ==============================================================================================
# Two-tier self-healing container monitoring system.
# Runs continuously as a background process — started by array_started.sh at array start.
# Shuts down cleanly on SIGTERM/SIGINT when array stops.
#
# ── TIER 1 — STRICT MONITORING ────────────────────────────────────────────────────────────────
# Applies only to explicitly configured containers (HOST*_WATCHDOG_CONTAINERS etc.)
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Two-tier self-healing container monitoring system. Runs as a continuous
# background daemon started by array_started.sh at array start. Shuts down
# cleanly on SIGTERM/SIGINT when the array stops.
#
# Memory hard limits — immediate restart if container exceeds configured MB ceiling
# Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of hard limit (no restart)
# Every DOCKER_WATCHDOG_INTERVAL seconds the watchdog runs a full cycle:
# Tier 1 applies specific thresholds to explicitly configured containers.
# Tier 2 scans everything else for generic health problems. Silent on clean
# cycles, loud when something needs attention.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Tier 1 — Strict Per-Container Monitoring
# Applies only to containers explicitly configured in master_host*.conf.
#
# Memory hard limits — immediate restart if container exceeds MB ceiling
# Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of limit (no restart)
# CPU thresholds — strike system: warn at SOFT_CPU_THRESHOLD, restart after
# CPU_FAIL_LIMIT consecutive strikes at HARD_CPU_THRESHOLD
# HTTP responsiveness — strike system: restart after RESP_FAIL_LIMIT consecutive failures
# HTTP responsiveness — strike system: restart after RESP_FAIL_LIMIT consecutive
# failures against the configured endpoint
# Required containers — must always be running; strike system before restart;
# skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in window;
# auto-clears when container recovers
# skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in window
#
# ── TIER 2 — GLOBAL HEALTH SCAN ───────────────────────────────────────────────────────────────
# Scans ALL running containers when WATCHDOG_SCAN_ALL=true.
# Containers in WATCHDOG_SCAN_IGNORE are excluded from Tier 2.
# Tier 2 — Global Health Scan
# Scans ALL running containers when WATCHDOG_SCAN_ALL=true.
# Containers in WATCHDOG_SCAN_IGNORE are excluded.
#
# Unhealthy status — Docker HEALTHCHECK unhealthy → safe_restart()
# OOM killed — kernel OOM killed → safe_restart() + notify
# OOM state tracked per-session to prevent restart loop
# Crash loop detection — RestartCount climbing → notify; above WATCHDOG_CRASH_LIMIT
# → safe_restart() → skip list if restart limit hit
# Dead containers — safe_restart() via remove + start
# Unexpected exits — non-zero exit code → safe_restart()
# Unhealthy status — Docker HEALTHCHECK unhealthy → restart
# OOM killed — kernel OOM kill detected → restart + notify
# Crash loop — RestartCount climbing → notify; above WATCHDOG_CRASH_LIMIT
# → restart → skip list if restart limit hit
# Dead containers — remove + start (dead state cannot be restarted directly)
# Unexpected exits — non-zero exit code → restart
#
# Cross-Cutting Intelligence
# Applies to both tiers on every cycle.
#
# ── CROSS-CUTTING INTELLIGENCE ────────────────────────────────────────────────────────────────
# Startup grace period — no restarts for WATCHDOG_STARTUP_GRACE seconds after boot
# Dependency ordering — waits for dependencies before restarting a dependent container
# Restart loop protect — skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in rolling window
# Skip list auto-clear — clears when container is seen running again
# Dependency ordering — dependency restarted first, dependent skipped this cycle
# Restart loop protect — skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in window
# Skip list auto-clear — removed when container seen running again
# Notification batching — one summary per cycle, not one ping per event
# Parity awareness — skips restart actions during parity check
# Timeout protection — all docker commands wrapped in timeout — daemon hangs cannot
# stall the watchdog and leave containers unmonitored
# Docker daemon check — first check every cycle; hung daemon → strike system →
# restart daemon via /etc/rc.d/rc.docker → verify recovery
# system_watchdog.sh handles escalation if restart fails
# Quiet when healthy — only logs when something needs attention (plus heartbeat)
# Timeout protection — all docker commands wrapped in timeout
# Docker daemon check — each cycle begins with daemon health check; hung daemon →
# restart via rc.docker → system_watchdog.sh escalates if needed
# RAM emergency defer — reads SYS_WATCHDOG_STATE_FILE; stands down while
# system_watchdog.sh is managing a RAM emergency
#
# ── STATE FILES ───────────────────────────────────────────────────────────────────────────────
# WATCHDOG_STATE_FILE — strike counts (/tmp — resets on reboot, correct)
# SYS_WATCHDOG_FAILED_FILE — skip list (/boot — survives reboots, intentional)
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Tiered Monitoring
# Not all containers need the same monitoring strategy. Tier 1 gives explicit
# control over the containers that matter most. Tier 2 is the catch-all that
# requires no configuration and protects everything else.
#
# Strike vs Immediate
# CPU spikes and HTTP failures are transient — brief spikes are normal during
# transcoding or library scans. Memory leaks are not transient. CPU and HTTP
# use a strike system to distinguish sustained problems from momentary ones.
# Memory triggers immediate restart because a container at its ceiling is
# actively leaking, not spiking.
#
# Loop Protection Over Persistence
# A watchdog that keeps restarting a broken container is not helpful — it risks
# making a database corruption worse. After WATCHDOG_CONTAINER_RESTART_LIMIT
# attempts the container is skip-listed and the operator is notified. Automated
# recovery stops. Human investigation begins.
#
# Dependency-Safe Ordering
# When a container and its dependency are both down, restart the dependency
# first and skip the dependent this cycle. Prevents false-alarm skip-listing
# of containers whose only failure was starting before their dependency was ready.
#
# Silent When Healthy
# Runs 96 times per day. Producing output on every clean cycle would make
# logs useless. Output only when something needs attention or a heartbeat fires.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Startup Grace Period
# Restart actions suppressed for WATCHDOG_STARTUP_GRACE seconds after the
# watchdog starts. Checks still run and log — only restarts are suppressed.
# Prevents false-positive restarts while containers are still initialising.
#
# Restart Loop Protection
# WATCHDOG_CONTAINER_RESTART_LIMIT restarts within WATCHDOG_CONTAINER_RESTART_WINDOW
# hours triggers skip-listing and a critical notification. Skip list persists on
# /boot/config/ — survives reboots intentionally. Auto-clears when container
# is seen running again.
#
# Docker Daemon Health Check
# First operation every cycle. Daemon not responding within DOCKER_TIMEOUT →
# restart via /etc/rc.d/rc.docker → verify recovery. If still hung: log
# critical, skip cycle. system_watchdog.sh handles further escalation.
#
# RAM Emergency Deferral
# Reads SYS_WATCHDOG_STATE_FILE each cycle. If system_watchdog.sh has set
# mem_shutdown_active=true, all restart logic defers until the flag clears.
# Stale state guard: if file is >2 hours old with flag still set,
# system_watchdog.sh has likely stopped — watchdog resumes normal operation.
#
# Timeout Protection
# All docker commands wrapped in timeout. Daemon hangs cannot stall the
# watchdog and leave containers unmonitored between cycles.
#
# Notification Batching
# Events collected across a full cycle and sent as a single summary.
# Prevents notification floods when a shared dependency failure cascades.
#
# ==============================================================================================
# STATE FILES
# ==============================================================================================
#
# WATCHDOG_STATE_FILE — strike counts (default: /tmp — resets on reboot)
# SYS_WATCHDOG_FAILED_FILE — skip list (default: /boot/config — survives reboots)
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection
# SYS_WATCHDOG_STATE_FILE — shared state with system_watchdog.sh (RAM emergency flag)
#
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
# HOST*_WATCHDOG_CONTAINERS — memory hard limits per container
# HOST*_WATCHDOG_CONTAINER_URLS — HTTP health check URLs
# HOST*_WATCHDOG_REQUIRED_CONTAINERS — must always be running
# HOST*_WATCHDOG_SCAN_IGNORE — skip in Tier 2 scan
# HOST*_WATCHDOG_DEPENDENCIES — dependency ordering for restart decisions
# All aliased by detect_hosts() — script uses unprefixed names
# /tmp files reset on reboot — correct, pre-reboot strike counts are meaningless after it.
# /boot/config files survive reboots — correct, a skip-listed container is still broken after one.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
#
# HOST*_WATCHDOG_CONTAINERS
# Memory hard limits per container. Format: "ContainerName:LimitInMB"
# Aliased by detect_hosts() → WATCHDOG_CONTAINERS
#
# HOST*_WATCHDOG_CONTAINER_URLS
# HTTP health check endpoints. Format: "ContainerName:http://host:port"
# Aliased by detect_hosts() → WATCHDOG_CONTAINER_URLS
#
# HOST*_WATCHDOG_REQUIRED_CONTAINERS
# Containers that must always be running. Aliased by detect_hosts() →
# WATCHDOG_REQUIRED_CONTAINERS
#
# HOST*_WATCHDOG_SCAN_IGNORE
# Containers excluded from Tier 2 global scan. Aliased by detect_hosts() →
# WATCHDOG_SCAN_IGNORE
#
# HOST*_WATCHDOG_DEPENDENCIES
# Dependency ordering. Format: "Dependent:dep1 dep2". Aliased by
# detect_hosts() → WATCHDOG_DEPENDENCIES
#
# master.conf
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
# SOFT_CPU_THRESHOLD / HARD_CPU_THRESHOLD / CPU_FAIL_LIMIT
# SOFT_MEM_THRESHOLD
# RESP_FAIL_LIMIT / CURL_TIMEOUT
# DOCKER_WATCHDOG_INTERVAL
# DOCKER_WATCHDOG_HEARTBEAT / DOCKER_WATCHDOG_HEARTBEAT_HOURS
# Seconds between full watchdog cycles (default: 900)
#
# WATCHDOG_STARTUP_GRACE
# Seconds before restart actions begin after watchdog starts (default: 600)
#
# SOFT_MEM_THRESHOLD
# Warn at this % of hard memory limit — no restart (default: 80)
#
# SOFT_CPU_THRESHOLD / HARD_CPU_THRESHOLD / CPU_FAIL_LIMIT
# CPU monitoring thresholds and strike limit
#
# CURL_TIMEOUT / RESP_FAIL_LIMIT
# HTTP health check timeout and consecutive failure limit
#
# WATCHDOG_SCAN_ALL
# Enable Tier 2 global health scan (default: true)
#
# WATCHDOG_RESTART_UNHEALTHY / WATCHDOG_RESTART_DEAD / WATCHDOG_RESTART_CRASHED
# WATCHDOG_NOTIFY_OOM / WATCHDOG_NOTIFY_CRASHLOOP
# WATCHDOG_CRASH_LIMIT
# WATCHDOG_STARTUP_GRACE
# Tier 2 action toggles
#
# WATCHDOG_NOTIFY_OOM / WATCHDOG_NOTIFY_CRASHLOOP / WATCHDOG_CRASH_LIMIT
# OOM and crash loop detection toggles and threshold
#
# WATCHDOG_CONTAINER_RESTART_LIMIT / WATCHDOG_CONTAINER_RESTART_WINDOW
# Restart loop protection: attempt limit and rolling window in hours
#
# WATCHDOG_BATCH_NOTIFY
# WATCHDOG_STATE_FILE / SYS_WATCHDOG_FAILED_FILE / WATCHDOG_CONTAINER_RESTART_LOG
# Collect cycle events and send as one notification (default: true)
#
# DOCKER_WATCHDOG_HEARTBEAT_HOURS
# Hours between alive heartbeat log entries
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# docker_watchdog.sh
# Start continuous monitoring loop — normally launched by array_started.sh
#
# docker_watchdog.sh --dry-run
# Run a full watchdog cycle without restarting anything. Shows what would
# happen based on current container states. Use to verify configuration.
#
# docker_watchdog.sh --status
# Show strike counts, skip list contents, grace period status, RAM emergency
# deferral state, and last cycle timing. Then exit.
#
# docker_watchdog.sh --log
# Verbose output — full detail for every container checked and every decision.
# Use to debug why a container is or is not being restarted.
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_watchdog.sh — normal start (continuous loop)
# docker_watchdog.sh --dry-run — preview without restarting
# docker_watchdog.sh --status — show config and exit
# docker_watchdog.sh --log — verbose output
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Docker_Essentials/docker_weekly_restart.sh
+64 -37
View File
@@ -2,52 +2,79 @@
# ==============================================================================================
# ================================= Docker Weekly Restart ======================================
# ==============================================================================================
# Restarts all running containers in HOST*_WEEKLY_RESTART_CONTAINERS.
# Called by weekly_sync_maintenance.sh via WEEKLY_MAINTENANCE_SCRIPTS every Sunday at 2:30am.
# Can also be run manually for ad hoc weekly restarts.
#
# ── CONTEXT ───────────────────────────────────────────────────────────────────────────────────
# weekly_sync_maintenance.sh stops containers before syncing and restarts them after.
# This script runs AFTER that restart — targeting a different set of less critical services
# that benefit from a weekly restart but don't need to be stopped for the sync itself.
# These containers are typically already running when this script executes.
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Restarts configured containers every Sunday at 2:30am as proactive maintenance.
#
# ── BEHAVIOUR ─────────────────────────────────────────────────────────────────────────────────
# Running containers → docker restart (graceful stop + start)
# Stopped containers → left stopped — was down intentionally, do not bring back up
# Missing containers → logged and skipped — not treated as fatal
# Each action uses RETRY_COUNT + SLEEP from master.conf for retry logic.
# Called by weekly_sync_maintenance.sh via WEEKLY_MAINTENANCE_SCRIPTS. Runs after
# the sync window has completed and already restarted its own critical containers
# (Emby, auth stack). Targets a separate set of less-critical services that benefit
# from a weekly clean start but do not need to be stopped for the sync itself.
#
# The "was running → restart, was stopped → leave stopped" rule is consistent
# across the entire ecosystem — container state is always respected.
# Same behavioural rules as docker_daily_restart.sh: running → restart,
# stopped → leave, missing → skip. Container state is always respected.
#
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
# Dependency ordering — containers restart in dependency-safe order using
# HOST*_WATCHDOG_DEPENDENCIES from master_host*.conf. Dependencies restart
# first with CONTAINER_DELAY before their dependents.
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Restart verification — after each restart, container state is checked after a short
# settle period. If the container fails to stay running it is marked as failed and
# a notification is sent rather than silently passing.
# Dependency Ordering
# Containers restart in dependency-safe order using HOST*_WATCHDOG_DEPENDENCIES.
# CONTAINER_DELAY seconds between dependency restart and dependent restart.
#
# Timeout protection — all docker commands are wrapped in a 30 second timeout.
# A hung Docker daemon cannot cause this script to hang indefinitely.
# Restart Verification
# Container state checked after a settle period. A container that crashes
# immediately after restart is marked failed with a notification sent.
#
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
# HOST*_WEEKLY_RESTART_CONTAINERS — list of containers to restart weekly
# HOST*_WATCHDOG_DEPENDENCIES — dependency ordering for restart sequence
# Set by detect_hosts() alias → WEEKLY_RESTART_CONTAINERS used by this script
# Timeout Protection
# All docker commands wrapped in a 30 second timeout. A hung Docker daemon
# cannot cause this script to hang indefinitely.
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
# RETRY_COUNT — retry attempts before giving up on a container
# SLEEP — seconds between retry attempts
# CONTAINER_DELAY — seconds to wait between dependency and dependent restart
# Lock Acquisition
# acquire_lock() prevents concurrent execution.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
#
# HOST*_WEEKLY_RESTART_CONTAINERS
# Containers restarted weekly. Aliased by detect_hosts() →
# WEEKLY_RESTART_CONTAINERS
#
# HOST*_WATCHDOG_DEPENDENCIES
# Dependency ordering shared with docker_watchdog.sh. Aliased by
# detect_hosts() → WATCHDOG_DEPENDENCIES
#
# master.conf
#
# RETRY_COUNT
# Retry attempts before giving up on a container
#
# SLEEP
# Seconds between retry attempts
#
# CONTAINER_DELAY
# Seconds to wait after restarting a dependency before starting its dependents
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# docker_weekly_restart.sh
# Restart all containers in WEEKLY_RESTART_CONTAINERS
#
# docker_weekly_restart.sh --dry-run
# Preview which containers would be restarted and which would be skipped
#
# docker_weekly_restart.sh --status
# Show configured restart list, container states, and dependency ordering
#
# docker_weekly_restart.sh --log
# Verbose per-container execution output
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_weekly_restart.sh — normal restart
# docker_weekly_restart.sh --dry-run — preview without restarting
# docker_weekly_restart.sh --log — verbose output
# docker_weekly_restart.sh --status — show config and exit
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Docker_Essentials/downloaders_reset.sh
+52 -2
View File
@@ -176,6 +176,56 @@ fi
[[ -n "$SABNZBD_URL" ]] && log "SABnzbd active on $MY_ID"
[[ -n "$QBIT_URL" ]] && log "qBittorrent active on $MY_ID"
# ==============================================================================================
# ━━━ slskd — Connection Check ━━━
# ==============================================================================================
# slskd's internal watchdog doesn't always recover from disconnection. Check before
# running API-dependent sections; attempt reconnect if down.
SLSKD_CONNECTED=false
if [[ -n "$SLSKD_URL" ]] && [[ -n "$SLSKD_API_KEY" ]]; then
echo ""
echo "━━━ $ICON_SYNC slskd — Connection Check ━━━"
_slskd_is_connected() {
local state
state=$(curl -sf --max-time 10 \
-H "X-Api-Key: $SLSKD_API_KEY" \
"$SLSKD_URL/api/v0/application" 2>/dev/null | \
jq -r '.server.isConnected // false' 2>/dev/null)
[[ "$state" == "true" ]]
}
if _slskd_is_connected; then
log "slskd connected to Soulseek ✅"
SLSKD_CONNECTED=true
else
warn "slskd disconnected — triggering reconnect"
curl -sf --max-time 10 -X PUT \
-H "X-Api-Key: $SLSKD_API_KEY" \
-H "Content-Type: application/json" \
"$SLSKD_URL/api/v0/server" \
-d '{"address":"server.slsknet.org","port":2242}' \
>/dev/null 2>&1
_ELAPSED=0
while [[ "$_ELAPSED" -lt 60 ]]; do
sleep 10
_ELAPSED=$(( _ELAPSED + 10 ))
if _slskd_is_connected; then
log "slskd reconnected after ${_ELAPSED}s ✅"
SLSKD_CONNECTED=true
break
fi
log " waiting... (${_ELAPSED}s / 60s)"
done
[[ "$SLSKD_CONNECTED" != true ]] && \
warn "slskd still disconnected after 60s — skipping API-dependent sections"
fi
fi
# ==============================================================================================
# ━━━ slskd — Stuck Searches ━━━
# ==============================================================================================
@@ -183,7 +233,7 @@ fi
# Prevents 409 Conflict error on next Soularr startup when it tries to
# create a search with the same ID that already exists in a terminal state.
if [[ -n "$SLSKD_URL" ]] && [[ -n "$SLSKD_API_KEY" ]]; then
if [[ -n "$SLSKD_URL" ]] && [[ -n "$SLSKD_API_KEY" ]] && [[ "$SLSKD_CONNECTED" == true ]]; then
echo ""
echo "━━━ 🔍 slskd — Stuck Searches ━━━"
@@ -236,7 +286,7 @@ fi
# Prevents Soularr 404 loop when polling a user whose transfer no longer exists.
# Safety: NEVER removes transfers that are InProgress or Queued — active downloads protected.
if [[ -n "$SLSKD_URL" ]] && [[ -n "$SLSKD_API_KEY" ]]; then
if [[ -n "$SLSKD_URL" ]] && [[ -n "$SLSKD_API_KEY" ]] && [[ "$SLSKD_CONNECTED" == true ]]; then
echo ""
echo "━━━ 🔍 slskd — Dead Transfer Records ━━━"