Read the following files and report any discrepancies between the documentation and the actual script headers. The goal is to find places where the README or Manual says something that no longer matches what the script header says (wrong schedule, wrong config vars, wrong behavior description, missing scripts, extra scripts, etc.).

Files to read:
1. /boot/config/plugins/varaverk/Docker_Essentials/README-Docker_Essentials.md
2. /boot/config/plugins/varaverk/Docker_Essentials/Manual-Docker_Essentials.md
3. Script headers (top ~120 lines each):
   - /boot/config/plugins/varaverk/Docker_Essentials/docker_daily_restart.sh
   - /boot/config/plugins/varaverk/Docker_Essentials/docker_update.sh
   - /boot/config/plugins/varaverk/Docker_Essentials/docker_update_remaining.sh
   - /boot/config/plugins/varaverk/Docker_Essentials/downloaders_reset.sh

Report: list each discrepancy as "FILE: what the doc says vs what the header says". If everything matches, say so. Be specific — quote the conflicting text. Keep the report under 400 words.


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.
- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch

update-config keybindings-help verify code-review simplify fewer-permission-prompts loop schedule claude-api run init review security-review


I'll read the documentation files and script headers to identify any discrepancies.


1	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2	# 🐳  DOCKER ESSENTIALS
3	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4
5	**Self-healing container lifecycle management for a 50+ container unRAID stack.**
6	Health monitoring that catches problems as they happen. Scheduled restarts that prevent
7	degradation before it becomes visible. Network configuration that survives reboots and
8	unRAID updates. Container image updates woven into the maintenance windows. Recovery
9	tooling for when something genuinely breaks.
10
11	> **Why this folder exists:** Docker on unRAID does not heal itself. A container that
12	> crashes stays crashed. A memory leak accumulates silently for days until the system
13	> starts swapping. A container whose dependency restarted first fails in a loop while
14	> the dependency comes up fine five seconds later. None of this surfaces clearly — it
15	> just builds into a system that feels flaky without a clear reason why. These scripts
16	> are the answer to all of that.
17
18	---
19
20	## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
21
22	Running a large Docker stack on unRAID is genuinely powerful — but Docker's own tooling
23	gives you almost nothing between "container is running" and "container has been dead for
24	three days and you just noticed."
25
26	The built-in restart policies help with outright crashes, but they have zero visibility
27	into memory leaks, frozen application layers, dependency ordering, restart loops, or the
28	difference between a container that crashed and one you intentionally stopped. They give
29	you no way to know *why* something keeps restarting — just that it does.
30
31	These are the specific problems that led to building this, in roughly the order they were
32	encountered:
33
34	---
35
36	### 🔴  Memory Leaks Accumulating Silently
37
38	Emby's transcode session handling occasionally leaks memory — a session ends but its
39	memory allocation doesn't fully release. SABnzbd's Python process slowly expands over
40	days as it processes downloads. Without hard memory limits and automatic enforcement,
41	these don't fail dramatically — they just consume more and more RAM until the system
42	starts swapping and everything slows to a crawl. By then, nothing in the Docker logs
43	tells you why the system feels slow. `docker stats` shows a container at 22GB and
44	climbing, but Docker itself does nothing about it.
45
46	The fix: hard per-container memory ceilings in `WATCHDOG_CONTAINERS`. When a container
47	exceeds its limit the watchdog restarts it immediately — no strikes, no waiting. A
48	memory leak is not a transient spike. Immediate action is correct.
49
50	---
51
52	### 🔴  Containers That Look Running But Aren't Responding
53
54	Docker reports a container as `Up 14 days` while its application layer has been silently
55	frozen for hours. The reverse proxy happily forwards traffic to a service that returns
56	nothing. Users see a broken page. Docker sees a healthy container. The container process
57	is technically running — it just isn't doing anything.
58
59	Docker's built-in health checks require a `HEALTHCHECK` instruction in the image itself,
60	which most self-hosted images don't have. Even those that do often check something too
61	shallow — a process exists, not whether it's actually serving requests.
62
63	The fix: HTTP health checks on the actual service port. `curl` to the real endpoint on
64	every watchdog cycle. If the service doesn't respond within `CURL_TIMEOUT` seconds, that's
65	a strike. Two consecutive failures trigger a restart. The distinction between "process
66	running" and "service responding" is the distinction that matters.
67
68	---
69
70	### 🔴  Dependency Failures on Restart
71
72	Authelia connects to MariaDB at startup. If both are down simultaneously — say, after a
73	power cut — and the watchdog restarts Authelia first, Authelia fails to connect, exits
74	immediately, and goes into a crash loop. Meanwhile MariaDB is coming up fine in the
75	background. The watchdog sees Authelia crash three times, adds it to the skip list,
76	and sends a critical notification. Authelia was never broken. It just came up in the
77	wrong order, failed at startup, and got punished for it.
78
79	The fix: dependency ordering. If a container's dependency is also down, the dependent is
80	skipped entirely this cycle. The dependency is restarted first. On the next cycle — once
81	MariaDB is actually accepting connections — Authelia is restarted and comes up cleanly.
82	The skip list is never involved. No false alarms. No manual recovery needed.
83
84	---
85
86	### 🔴  Restart Loops Corrupting State
87
88	Some containers corrupt their internal state if restarted repeatedly in rapid succession.
89	SQLite databases that don't get a clean shutdown write incomplete transactions. Partially
90	applied database migrations leave schema in an inconsistent state. A container that
91	crashes on startup after a bad migration gets restarted immediately, crashes again,
92	gets restarted again — each restart has a chance of making the database worse, not better.
93
94	A naive watchdog that just keeps hammering a crashed container is actively harmful in
95	this scenario. More restarts mean more corruption risk. The right response when restarts
96	aren't working is to stop restarting and alert the operator.
97
98	The fix: restart loop protection. After `WATCHDOG_CONTAINER_RESTART_LIMIT` restarts
99	within a rolling `WATCHDOG_CONTAINER_RESTART_WINDOW` hour window, the container goes
100	on the skip list. A critical notification goes out. The watchdog stops touching it.
101	The operator investigates and clears the skip list once the underlying problem is fixed.
102
103	---
104
105	### 🔴  Slow Degradation That Never Becomes a Failure
106
107	NginxProxyManager accumulates stale entries in its connection table over weeks of uptime.
108	Dispatcharr's Live TV scheduler builds up internal scheduling state that makes decisions
109	progressively slower after months of continuous operation. These containers never crash.
110	They never throw errors. They just get progressively worse in ways that are hard to
111	attribute to anything specific — until someone notices that the proxy feels slower than
112	it used to, or that Live TV channel changes take longer than they should.
113
114	The fix: scheduled restarts. Not because something is broken, but because some containers
115	simply perform better with a clean start. Daily at 1am for connection-heavy services.
116	Weekly for less-critical services that run fine for weeks but benefit from a clean slate.
117	Zero user impact — happens while everyone is asleep.
118
119	---
120
121	### 🔴  Network Configuration Lost After Updates
122
123	unRAID occasionally wipes custom Docker networks after updates — particularly networks
124	created by Docker Compose stacks or the NextCloud AIO container. Any container that
125	depended on those networks for internal communication suddenly can't reach its peers.
126	memcached can't talk to NextCloud. CrowdSec can't talk to NginxProxyManager. Services
127	appear to be up but silently fail to communicate with each other.
128
129	The fix: network recreation at every array start. `docker_network_connect.sh` checks
130	every configured network on startup, creates any that are missing, and connects all
131	configured containers to them. Idempotent — if everything is already correct, it does
132	nothing and produces no output. If a network had to be created, it notifies — that only
133	happens after an update, and you want to know when it does.
134
135	---
136
137	### 🔴  No Visibility Into What the Watchdog Already Tried
138
139	A container keeps appearing in a broken state. You SSH in and see it's stopped. You
140	don't know if the watchdog tried to restart it and failed, gave up and skip-listed it,
141	is mid-attempt right now, or hasn't noticed yet. You have to manually check the skip
142	list file on `/boot/config/`, check the restart history file, check the state file —
143	none of which have obvious formats.
144
145	The fix: `watchdog_skip_list_manager.sh` (in `Tools/`). One command to see exactly what's
146	on the skip list, which containers are stopped vs running, how many restarts were attempted,
147	and what the watchdog's current state is. One command to clear a specific container and
148	its history after you've fixed the problem. No manual file editing required.
149
150	---
151
152	## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
153
154	```
155	Docker_Essentials/   ← acts on containers          (this folder)
156	System_Essentials/   ← acts on the server itself
157	Watchdogs/           ← reactive monitoring + last-resort stability
158	Monitors/            ← observes, measures, reports
159	Rsync/               ← moves data between servers
160	```
161
162	> `docker_watchdog.sh` has moved to `Watchdogs/`. Container healing, memory limits,
163	> HTTP health checks, and skip list management are documented in
164	> `Watchdogs/README-Watchdogs.md` and `Watchdogs/Manual-Watchdogs.md`.
165
166	Four distinct responsibilities in this folder, each handled by dedicated scripts:
167
168	---
169
170	### ♻  Proactive Freshness — `docker_daily_restart.sh` + `docker_weekly_restart.sh`
171
172	Scheduled restarts that prevent slow degradation before it becomes visible. Not because
173	something broke — because some containers simply work better after a clean start.
174
175	Called by the maintenance orchestrators (`daily_sync_maintenance.sh` and
176	`weekly_sync_maintenance.sh`) — not run standalone. They run inside the maintenance
177	windows so any downtime from restarts is absorbed by the window that's already happening.
178
179	---
180
181	### 🔄  Image Currency — `docker_update.sh` + `docker_update_remaining.sh`
182
183	Keeps all container images current without manual intervention. Daily updates for the
184	auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
185	remainder pass for everything else — derives the target list automatically from `docker ps`
186	minus what was already updated, so there is no second list to maintain.
187
188	---
189
190	### 🌐  Network Integrity — `docker_network_connect.sh`
191
192	Ensures custom Docker networks exist and containers are connected to them at every
193	array start. Silent when everything is correct. Notifies when it has to create something
194	— which means something was wiped and you should know about it.
195
196	---
197
198	### 🧹  Downloader Hygiene — `downloaders_reset.sh`
199
200	Maintenance reset for all download clients (slskd, SABnzbd, qBittorrent) every 30
201	minutes. Clears stuck searches, dead transfers, failed imports, and stale queue entries
202	that download clients accumulate but never clean up themselves. Never touches active
203	or in-progress downloads.
204
205	---
206
207	## ━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
208
209	`docker_watchdog.sh` has moved to `Watchdogs/` and is now one of four coordinated
210	single-pass scripts called every 15 minutes by `Orchestrators/watchdog_orchestrator.sh`.
211
212	```
213	Watchdogs/resource_watchdog.sh  ← reduces pressure before healing attempts
214	Watchdogs/docker_watchdog.sh    ← heals containers (reads resource_watchdog state)
215	Watchdogs/System/storage_watchdog.sh   ← pool growth + runaway log detection
216	Watchdogs/stability_watchdog.sh  ← last resort — reboots when healing has failed
217	```
218
219	Scripts in this folder (daily restart, updates, network connect) are unaffected —
220	they run on their own schedules via the maintenance orchestrators and are not part
221	of the every-minute watchdog cycle. See `Watchdogs/README-Watchdogs.md` for the
222	full coordination model between all four watchdogs.
223
224	---
225
226	## ━━━ SCRIPTS IN THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
227
228	| Script | Role | When It Runs |
229	|--------|------|-------------|
230	| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
231	| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
232	| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window |
233	| `docker_update_remaining.sh` | Image update + prune for all containers not in managed lists | End of weekly maintenance window |
234	| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
235	| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
236	| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
237
238	> `watchdog_skip_list_manager.sh` — manual recovery tool for the skip list. Lives in `Tools/` because it's an operator utility, not a lifecycle script.
239
240	---
241
242	## ━━━ HOW THE SCRIPTS RELATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
243
244	```
245	Array starts
246	    │
247	    └── docker_network_connect.sh ────── run once at start
248	            ensure networks + connections exist
249	            silent if correct, notify if creating
250
251	Every 15 minutes (Orchestrators/watchdog_orchestrator.sh):
252	    └── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md
253
254	Daily maintenance window (1am):
255	    daily_sync_maintenance.sh
256	        ├── docker_update.sh ──────────── pull latest images (DAILY_RESTART_CONTAINERS)
257	        └── docker_daily_restart.sh ───── restart connection-heavy services
258
259	Weekly maintenance window (2:30am Sunday):
260	    weekly_sync_maintenance.sh
261	        ├── docker_weekly_restart.sh ──── restart less-critical services
262	        ├── docker_update.sh --remainder ─ update containers not in managed lists
263	        └── docker_update_remaining.sh ── prune dangling images
264
265	Critical maintenance window (every 30min):
266	    critical_sync_maintenance.sh
267	        └── downloaders_reset.sh ──────── clear stuck downloads / stale queue entries
268
269	Array stopping:
270	    array_stopping.sh
271	        └── docker_container_stop.sh ──── ordered graceful shutdown
272	```
273


1	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2	# 🐳  DOCKER ESSENTIALS — Manual
3	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4
5	Configuration reference, setup procedures, and operational workflows.
6	For folder overview and design philosophy see `README-Docker_Essentials.md`.
7	For per-script detail see the script headers directly.
8
9	> **Watchdog configuration has moved.** `docker_watchdog.sh` is now in `Watchdogs/`.
10	> Memory limits, CPU thresholds, HTTP health checks, dependency ordering, skip list
11	> recovery, and all watchdog config vars are in `Watchdogs/Manual-Watchdogs.md`.
12
13	---
14
15	## ━━━ RESTART SCHEDULE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
16
17	---
18
19	### ── Daily Restart List ───────────────────────────────────────────────────────
20
21	```bash
22	# host1.conf
23	# Restarted every night at 1am via daily_sync_maintenance.sh.
24	#
25	# Good candidates:
26	#   Reverse proxies         — connection table fills slowly over weeks
27	#   Authentication services — session cache benefits from periodic clearing
28	#   Live TV schedulers      — accumulated scheduling state slows decisions
29	#   Download managers       — connection pool maintenance
30	#
31	HOST1_DAILY_RESTART_CONTAINERS=(
32	    "NginxProxyManager"     # connection table fills slowly over weeks
33	    "Authelia"              # session cache benefits from periodic clearing
34	    "Dispatcharr"           # Live TV scheduler accumulates state
35	    "Dispatcharr-Basic"     # secondary Live TV scheduler — same reason
36	    "ErsatzTV-Emby"         # channel schedule builder, stale entries accumulate
37	)
38	```
39
40	This list also drives `docker_update.sh` in normal mode — containers added here get
41	their images updated daily before the restart. Add a container once, it gets both.
42
43	---
44
45	### ── Weekly Restart List ──────────────────────────────────────────────────────
46
47	```bash
48	# host1.conf
49	# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
50	# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
51	#
52	# Daily vs Weekly decision:
53	#   Daily:  connection-heavy infrastructure — degrades faster (proxy, auth, Live TV)
54	#   Weekly: productivity and media services — degrades slowly (NextCloud, AdGuard, Immich)
55	#
56	HOST1_WEEKLY_RESTART_CONTAINERS=(
57	    "NextCloud"       # file sync — benefits from clean weekly start
58	    "AdGuard-Home"    # DNS — cache and stat accumulation
59	    "Immich"          # photo library — index/cache maintenance
60	)
61	```
62
63	---
64
65	## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
66
67	```bash
68	# host1.conf
69	# Networks to ensure exist + containers to connect to each network.
70	# Many-to-many: every container connects to every network listed.
71	#
72	# Containers do not need to be running — script handles missing containers
73	# gracefully (warns + skips). They connect on the next array start.
74	#
75	HOST1_NETWORK_CONNECT_NETWORKS=(
76	    "high-availability"     # main internal network — most containers should be on this
77	)
78
79	HOST1_NETWORK_CONNECT_CONTAINERS=(
80	    "memcached"             # NextCloud's cache — needs to reach NextCloud AIO network
81	    "Npm-CrowdSec"          # CrowdSec bouncer — needs to reach NPM's network
82	)
83	```
84
85	> **Timing dependency:** Networks created by Docker Compose stacks (e.g. NextCloud AIO)
86	> only exist after those stacks start. If this script runs before the Compose stack,
87	> the network won't exist yet and the connection fails this run. It will succeed on the
88	> next array start. Schedule Compose stacks early in `ARRAY_START_SCRIPTS` order to
89	> minimise the window.
90
91	---
92
93	## ━━━ CONTAINER UPDATE CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
94
95	```bash
96	# master.conf
97	DAILY_CONTAINER_UPDATES=true     # enable/disable daily image pull
98	                                  # docker_daily_restart.sh still runs regardless
99	                                  # update and restart are independent
100
101	WEEKLY_REMAINING_UPDATES=true    # enable/disable weekly remainder pull + prune
102	                                  # to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS
103	```
104
105	`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list
106	used by `docker_daily_restart.sh`. No second list to maintain.
107
108	`docker_update_remaining.sh` derives its target list automatically:
109	all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`.
110	Everything gets updated at least once per week with no explicit configuration.
111
112	---
113
114	## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
115
116	### ── host*.conf ────────────────────────────────────────────────────────
117
118	```bash
119	# Per-host — varies between HOST1 and HOST2
120
121	# Daily restart list (also drives docker_update.sh normal mode)
122	HOST1_DAILY_RESTART_CONTAINERS=()
123
124	# Weekly restart list
125	HOST1_WEEKLY_RESTART_CONTAINERS=()
126
127	# Networks to ensure exist
128	HOST1_NETWORK_CONNECT_NETWORKS=()
129
130	# Containers to connect to every configured network
131	HOST1_NETWORK_CONNECT_CONTAINERS=()
132
133	# Watchdog config (memory limits, HTTP checks, required, dependencies):
134	# → see Watchdogs/Manual-Watchdogs.md
135	```
136
137	---
138
139	### ── master.conf ──────────────────────────────────────────────────────────────
140
141	```bash
142	# Shared — applies to both servers
143
144	# ── Container updates ──────────────────────────────────────────────────
145	DAILY_CONTAINER_UPDATES=true
146	WEEKLY_REMAINING_UPDATES=true
147
148	# ── Retry behaviour (shared by restart scripts) ────────────────────────
149	RETRY_COUNT=3                        # retry attempts before marking failed
150	SLEEP=5                              # seconds between retry attempts
151
152	# Watchdog thresholds (CPU, memory, HTTP, restart loop):
153	# → see Watchdogs/Manual-Watchdogs.md
154	```
155
156	---
157
158	## ━━━ PROCEDURES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
159
160	### ── Adding a Container to Monitoring ────────────────────────────────────────
161
162	Adding a container to watchdog monitoring is purely additive — add lines to `host*.conf`.
163	No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
164	See `Watchdogs/Manual-Watchdogs.md` for the full procedure.
165
166	To add a container to **daily restarts** (and daily image updates):
167
168	```bash
169	# host1.conf
170	HOST1_DAILY_RESTART_CONTAINERS=(
171	    ...existing...
172	    "MyApp"    # also adds it to the daily image update
173	)
174	```
175
176	To **exclude** a container from Tier 2 global scan (e.g. one-shot that exits normally):
177
178	```bash
179	# master.conf
180	WATCHDOG_SCAN_IGNORE=(
181	    "MyApp"    # one-shot — exits cleanly, don't treat as crash
182	)
183	```
184
185	---
186
187	### ── Skip List Recovery ────────────────────────────────────────────────────────
188
189	When `docker_watchdog.sh` skip-lists a container, use `Tools/watchdog_skip_list_manager.sh`.
190	Full procedure in `Watchdogs/Manual-Watchdogs.md`.
191
192	```bash
193	Tools/watchdog_skip_list_manager.sh --status          # see skip list + container states
194	Tools/watchdog_skip_list_manager.sh --clear MyApp     # clear after fixing root cause
195	Tools/watchdog_skip_list_manager.sh --clear-all       # clear everything
196	```
197
198	---
199
200	## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
201
202	All scripts support these standard flags:
203
204	| Flag | What it does |
205	|------|-------------|
206	| `--dry-run` | Preview actions without making changes. Shows exactly what would happen. |
207	| `--status` | Show current config, container states, and relevant runtime info, then exit. |
208	| `--log` | Verbose mode — adds per-container banners, action lines, pull output, and list details. |
209
210	**Output tiers:** Without `--log`, each script processes silently and concludes with a
211	summary block: identity, duration, counts, and a status line. Per-container detail only
212	appears with `--log`. Warnings and errors are always visible regardless of `--log`.
213
214	### `docker_update.sh --remainder`
215	Switches to remainder mode — updates all running containers not in the managed daily/weekly
216	lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers
217	that haven't been updated recently.
218


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Docker Daily Restart =======================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Restarts configured containers every night at 1am as proactive maintenance.
9	#
10	# Called by daily_sync_maintenance.sh via DAILY_MAINTENANCE_SCRIPTS. Runs inside
11	# the daily maintenance window — any service downtime is absorbed by a window
12	# that is already happening. Also drives docker_update.sh in normal mode: the
13	# same DAILY_RESTART_CONTAINERS list is used for both restarts and image pulls,
14	# so there is no second list to maintain.
15	#
16	# ==============================================================================================
17	# DESIGN PRINCIPLES
18	# ==============================================================================================
19	#
20	# Proactive Maintenance
21	#   Daily restarts target containers known to degrade over time without
22	#   crossing a clear failure threshold — connection table growth, scheduler
23	#   state accumulation, session cache bloat. The watchdog cannot detect this
24	#   class of degradation. Scheduled restarts clear it before it becomes visible.
25	#
26	# State Respect
27	#   Running containers are restarted. Stopped containers are left stopped — they
28	#   were intentionally halted and this script has no authority to override that
29	#   decision. This rule is consistent across the entire ecosystem.
30	#
31	# Dependency-Safe Ordering
32	#   Restarts follow the same dependency ordering used by docker_watchdog.sh.
33	#   Services that other containers depend on restart first. A dependent is never
34	#   restarted while its dependency is still coming up.
35	#
36	# ==============================================================================================
37	# OPERATIONAL SAFEGUARDS
38	# ==============================================================================================
39	#
40	# Dependency Ordering
41	#   Containers restart in dependency-safe order using HOST*_WATCHDOG_DEPENDENCIES.
42	#   CONTAINER_DELAY seconds between dependency restart and dependent restart gives
43	#   the dependency time to fully initialise before dependents try to connect.
44	#
45	# Restart Verification
46	#   After each restart, container state is checked after a settle period. A
47	#   container that starts and immediately crashes is marked failed and a
48	#   notification is sent — the script does not silently pass a restart that
49	#   did not stick.
50	#
51	# Timeout Protection
52	#   All docker commands wrapped in a 30 second timeout. A hung Docker daemon
53	#   cannot cause this script to hang indefinitely. Timed-out commands retry
54	#   per RETRY_COUNT before marking as failed.
55	#
56	# Lock Acquisition
57	#   acquire_lock() prevents concurrent execution if a previous run is still active.
58	#
59	# ==============================================================================================
60	# CONFIGURATION
61	# ==============================================================================================
62	#
63	# host*.conf
64	#
65	#   HOST*_DAILY_RESTART_CONTAINERS
66	#       Containers restarted nightly. Also used by docker_update.sh normal mode
67	#       for image pulls — add a container once, it gets both. Aliased by
68	#       detect_hosts() → DAILY_RESTART_CONTAINERS
69	#
70	#   HOST*_WATCHDOG_DEPENDENCIES
71	#       Dependency ordering shared with docker_watchdog.sh. Aliased by
72	#       detect_hosts() → WATCHDOG_DEPENDENCIES
73	#
74	# master.conf
75	#
76	#   RETRY_COUNT
77	#       Retry attempts before giving up on a container
78	#
79	#   SLEEP
80	#       Seconds between retry attempts
81	#
82	#   CONTAINER_DELAY
83	#       Seconds to wait after restarting a dependency before starting its dependents
84	#
85	#   RESTART_VERIFY_WAIT
86	#       Seconds to wait after docker restart before checking the container is running.
87	#       Gives the process time to initialise before verify_running samples the state.
88	#       (default: 3)
89	#
90	# ==============================================================================================
91	# RUNTIME MODES
92	# ==============================================================================================
93	#
94	# docker_daily_restart.sh
95	#     Restart all containers in DAILY_RESTART_CONTAINERS
96	#
97	# docker_daily_restart.sh --dry-run
98	#     Preview which containers would be restarted and which would be skipped
99	#
100	# docker_daily_restart.sh --status
101	#     Show configured restart list, container states, and dependency ordering
102	#
103	# docker_daily_restart.sh --log
104	#     Verbose per-container execution output
105	#
106	# ==============================================================================================
107
108	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
109
110	source "$SCRIPT_DIR/../load_config.sh"
111
112	parse_args "$@"
113
114	# ==============================================================================================
115	# ━━━ Setup ━━━
116	# ==============================================================================================
117	if [[ "$EUID" -ne 0 ]]; then
118	    error "Must be run as root"
119	    exit 1
120	fi


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Docker Update ==============================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Pulls the latest images for configured containers. Two modes: normal (daily)
9	# and remainder (weekly).
10	#
11	# Normal mode is called by daily_sync_maintenance.sh before docker_daily_restart.sh.
12	# Containers stay running during the pull — no extra downtime beyond what the
13	# nightly restart already causes.
14	#
15	# Remainder mode is called by weekly_sync_maintenance.sh as the final update step.
16	# It catches everything that normal mode and the weekly sync window did not already
17	# update — derived automatically from docker ps, nothing to configure.
18	#
19	# ==============================================================================================
20	# OPERATIONAL MODEL
21	# ==============================================================================================
22	#
23	# Normal mode (daily):
24	#   Targets DAILY_RESTART_CONTAINERS — same list used by docker_daily_restart.sh.
25	#   Pull → compare old vs new image ID → mark updated or already current.
26	#   docker_daily_restart.sh runs after — containers restart onto the fresh image.
27	#
28	# Remainder mode (weekly):
29	#   Targets all currently running containers NOT in:
30	#     DAILY_RESTART_CONTAINERS       — already updated daily
31	#     emby + critical-data profiles  — updated inline by the weekly sync window
32	#     FALLBACK_*_TIER*               — owned by the remote server's update cycle
33	#   Pull → compare → prune dangling images.
34	#
35	# ==============================================================================================
36	# DESIGN PRINCIPLES
37	# ==============================================================================================
38	#
39	# Single List
40	#   Normal mode reuses DAILY_RESTART_CONTAINERS rather than maintaining a
41	#   separate update list. Adding or removing a container from the restart list
42	#   automatically updates the image pull list — one change, both places.
43	#
44	# Version Ownership
45	#   Fallback containers are excluded from remainder mode. This server only runs
46	#   them during a fallback. The remote server owns their version — if remainder
47	#   updates them independently and a handback occurs, the remote's older image
48	#   may not handle data written by the newer version.
49	#
50	# State Respect
51	#   Stopped containers are never targeted. Pulling while stopped adds no value
52	#   and a stopped container was likely halted intentionally.
53	#
54	# ==============================================================================================
55	# OPERATIONAL SAFEGUARDS
56	# ==============================================================================================
57	#
58	# Lock Acquisition
59	#   Prevents concurrent execution via acquire_lock(). Safe to call from
60	#   maintenance scripts without risk of overlap.
61	#
62	# Host Detection
63	#   detect_hosts() identifies which server is running the script and aliases
64	#   HOST*_DAILY_RESTART_CONTAINERS to the correct host's values.
65	#
66	# Root Enforcement
67	#   Docker operations require root privileges.
68	#
69	# DAILY_CONTAINER_UPDATES Toggle
70	#   Normal mode exits cleanly when disabled. docker_daily_restart.sh still runs
71	#   regardless — update and restart are independent operations.
72	#
73	# Fallback Exclusion
74	#   Remainder mode excludes containers owned by the remote server's update cycle
75	#   to prevent version divergence across the fallback boundary.
76	#
77	# Running-Only Filter
78	#   Stopped containers excluded from remainder mode — intentionally down.
79	#
80	# Image ID Comparison
81	#   Containers not restarted unless their image actually changed. Pulls that
82	#   result in "already up to date" produce no restart.
83	#
84	# ==============================================================================================
85	# CONFIGURATION
86	# ==============================================================================================
87	#
88	# master.conf
89	#
90	#   DAILY_CONTAINER_UPDATES
91	#       Enable or disable normal mode. docker_daily_restart.sh runs regardless.
92	#       (default: true)
93	#
94	#   PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data]
95	#       Container names for emby and critical-data profiles — excluded from
96	#       remainder mode (already updated by the weekly sync window)
97	#
98	# host*.conf
99	#
100	#   HOST*_DAILY_RESTART_CONTAINERS
101	#       Containers updated in normal mode. Aliased by detect_hosts() →
102	#       DAILY_RESTART_CONTAINERS
103	#
104	# ==============================================================================================
105	# RUNTIME MODES
106	# ==============================================================================================
107	#
108	# docker_update.sh
109	#     Normal mode — pull latest images for DAILY_RESTART_CONTAINERS
110	#
111	# docker_update.sh --remainder
112	#     Remainder mode — pull all running containers not in managed lists,
113	#     restart those that received updates, prune dangling images
114	#
115	# docker_update.sh --dry-run
116	#     Preview which containers would be pulled without making changes
117	#
118	# docker_update.sh --status
119	#     Show configuration and container list for current mode
120	#


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Docker Update — Remaining ======================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Weekly sweep that pulls the latest image for every running container not
9	# already covered by the daily or weekly managed update cycles. Restarts
10	# containers that received a new image, then prunes dangling images.
11	#
12	# Called by weekly_sync_maintenance.sh as the final step in the weekly window.
13	# Derives its target list automatically from docker ps minus the two managed
14	# lists — there is nothing to configure for this script.
15	#
16	# Together with docker_update.sh (normal + remainder modes), every deployed
17	# container receives at least one image pull per week without any per-container
18	# configuration required here.
19	#
20	# ==============================================================================================
21	# OPERATIONAL SAFEGUARDS
22	# ==============================================================================================
23	#
24	# Lock Acquisition
25	#   Prevents concurrent execution via acquire_lock(). Safe to call from
26	#   weekly maintenance scripts without risk of overlap.
27	#
28	# Host Detection
29	#   detect_hosts() identifies which server is running the script and aliases
30	#   HOST*_DAILY_RESTART_CONTAINERS and HOST*_WEEKLY_RESTART_CONTAINERS to
31	#   the correct host's values for exclusion.
32	#
33	# Root Enforcement
34	#   Docker operations require root privileges.
35	#
36	# WEEKLY_REMAINING_UPDATES Toggle
37	#   Exits cleanly when disabled via master.conf.
38	#
39	# Running-Only Filter
40	#   Stopped containers excluded — intentionally down, pulling adds no value.
41	#
42	# Image ID Comparison
43	#   Containers not restarted unless their image actually changed.
44	#
45	# ==============================================================================================
46	# CONFIGURATION
47	# ==============================================================================================
48	#
49	# master.conf
50	#
51	#   WEEKLY_REMAINING_UPDATES
52	#       Enable or disable this script. (default: true)
53	#       To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
54	#
55	# host*.conf
56	#
57	#   HOST*_DAILY_RESTART_CONTAINERS
58	#       Excluded from this script — already updated daily. Aliased by
59	#       detect_hosts() → DAILY_RESTART_CONTAINERS
60	#
61	#   HOST*_WEEKLY_RESTART_CONTAINERS
62	#       Excluded from this script — already updated by weekly sync window.
63	#       Aliased by detect_hosts() → WEEKLY_RESTART_CONTAINERS
64	#
65	# ==============================================================================================
66	# RUNTIME MODES
67	# ==============================================================================================
68	#
69	# docker_update_remaining.sh
70	#     Pull all remaining running containers, restart those updated, prune images
71	#
72	# docker_update_remaining.sh --dry-run
73	#     Preview which containers would be pulled and restarted
74	#
75	# docker_update_remaining.sh --status
76	#     Show exclusion lists and current remaining container count
77	#
78	# docker_update_remaining.sh --log
79	#     Verbose per-container pull and restart output
80	#
81	# ==============================================================================================
82
83	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
84
85	source "$SCRIPT_DIR/../load_config.sh"
86
87	parse_args "$@"
88
89	# ==============================================================================================
90	# ━━━ Setup ━━━
91	# ==============================================================================================
92	if [[ "$EUID" -ne 0 ]]; then
93	    error "Must be run as root"
94	    exit 1
95	fi
96
97	acquire_lock
98
99	if ! command -v docker &>/dev/null; then
100	    error "Docker command not found"
101	    exit 1
102	fi
103
104	detect_hosts
105
106	if [[ "${WEEKLY_REMAINING_UPDATES:-true}" != "true" ]]; then
107	    echo "WEEKLY_REMAINING_UPDATES=false — skipping remaining container updates"
108	    exit 0
109	fi
110
111	# ── Build exclusion set from daily + weekly managed lists ─────────────────────────────────────
112	declare -A EXCLUDED
113	for c in "${DAILY_RESTART_CONTAINERS[@]}"  "${WEEKLY_RESTART_CONTAINERS[@]}"; do
114	    [[ -n "$c" ]] && EXCLUDED["$c"]=1
115	done
116
117	# ── Get all running containers ────────────────────────────────────────────────────────────────
118	mapfile -t ALL_RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
119
120	# ── Derive remainder: running minus excluded ──────────────────────────────────────────────────


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Downloaders Reset ==========================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Maintenance reset for all download clients on this server. Clears accumulated
9	# state that download clients generate but never clean up themselves — stuck
10	# searches, dead transfers, failed imports, stale queue entries, completed history.
11	#
12	# Called every 30 minutes by critical_sync_maintenance.sh via
13	# CRITICAL_MAINTENANCE_SCRIPTS. Can also be run manually for ad hoc cleanup.
14	# If a downloader is not configured for this host, that section skips cleanly.
15	#
16	# ==============================================================================================
17	# OPERATIONAL MODEL
18	# ==============================================================================================
19	#
20	# slskd
21	#   Stuck searches    — clears Completed/Errored searches left by Soularr crashes
22	#                       prevents 409 Conflict on next Soularr startup
23	#   Dead transfers    — removes completed/errored/aborted transfer records per user
24	#                       prevents Soularr 404 loop when polling a user whose transfer is gone
25	#                       NEVER removes InProgress or Queued transfers
26	#   Failed imports    — purges albums Soularr downloaded but Lidarr rejected
27	#                       Soularr moves these to failed_imports/ and never cleans them up
28	#
29	# SABnzbd
30	#   Completed history — removes completed download records older than DOWNLOADER_RETENTION_DAYS
31	#   Failed history    — removes failed download records older than DOWNLOADER_RETENTION_DAYS
32	#   Stalled queue     — removes Paused or Stuck queue items no longer progressing
33	#                       active downloading items are never touched
34	#
35	# qBittorrent
36	#   Age failsafe      — removes torrents older than QBIT_FAILSAFE_MIN_DAYS
37	#                       deleteFiles=false — removes from qBit, leaves files for arrs to manage
38	#                       optional ratio requirement via QBIT_FAILSAFE_MIN_RATIO
39	#
40	# ==============================================================================================
41	# DESIGN PRINCIPLES
42	# ==============================================================================================
43	#
44	# Never Interrupt Active Downloads
45	#   Each downloader section checks for active state before any removal. slskd
46	#   skips users with InProgress or Queued transfers. SABnzbd only removes items
47	#   past the retention threshold. qBittorrent applies minimum age and optional
48	#   ratio requirements. In-progress work is never touched.
49	#
50	# Graceful Skip on Unavailability
51	#   If a downloader's URL is empty or the service is unreachable, that section
52	#   skips cleanly with a log message. The script never exits fatally on a single
53	#   unreachable downloader — the others still run.
54	#
55	# Host-Aware Configuration
56	#   detect_hosts() aliases all HOST*_SLSKD_*, HOST*_SABNZBD_*, HOST*_QBIT_* vars
57	#   to their unprefixed names. Downloaders not configured for this host are absent
58	#   from the aliased vars and skip automatically.
59	#
60	# ==============================================================================================
61	# OPERATIONAL SAFEGUARDS
62	# ==============================================================================================
63	#
64	# Active Transfer Protection
65	#   slskd: skips users with InProgress or Queued transfers before any removal.
66	#   SABnzbd: age threshold enforced before deletion.
67	#   qBittorrent: minimum age plus optional ratio gate before failsafe removal.
68	#
69	# Reachability Check
70	#   Each section validates its downloader URL before API calls. Missing or
71	#   unreachable downloaders skip without affecting other sections.
72	#
73	# Host Detection
74	#   detect_hosts() identifies which server is running the script and aliases
75	#   all HOST*_SLSKD_*, HOST*_SABNZBD_*, and HOST*_QBIT_* vars to the correct
76	#   host's values. Downloaders not configured on this host skip automatically.
77	#
78	# Lock Acquisition
79	#   acquire_lock "wait" — waits for previous run to finish since this runs every
80	#   30 minutes and prior execution may still be completing.
81	#
82	# ==============================================================================================
83	# CONFIGURATION
84	# ==============================================================================================
85	#
86	# host*.conf
87	#
88	#   HOST*_SLSKD_URL / HOST*_SLSKD_API_KEY / HOST*_SLSKD_FAILED_IMPORTS_DIR
89	#       slskd connection and failed imports path. Aliased by detect_hosts()
90	#
91	#   HOST*_SABNZBD_URL / HOST*_SABNZBD_API_KEY
92	#       SABnzbd connection details. Aliased by detect_hosts()
93	#
94	#   HOST*_QBIT_URL / HOST*_QBIT_USERNAME / HOST*_QBIT_PASSWORD
95	#       qBittorrent connection details. Aliased by detect_hosts()
96	#
97	# master.conf
98	#
99	#   DOWNLOADER_RETENTION_DAYS
100	#       Days before SABnzbd history entries (completed or failed) are removed
101	#
102	#   QBIT_FAILSAFE_MIN_DAYS
103	#       Minimum torrent age in days before failsafe removal is considered
104	#
105	#   QBIT_FAILSAFE_MIN_RATIO
106	#       Minimum seeding ratio required alongside age gate (0 = age only)
107	#
108	# ==============================================================================================
109	# RUNTIME MODES
110	# ==============================================================================================
111	#
112	# downloaders_reset.sh
113	#     Run maintenance reset for all configured download clients
114	#
115	# downloaders_reset.sh --dry-run
116	#     Preview what would be removed without making any changes
117	#
118	# downloaders_reset.sh --status
119	#     Show configured downloaders, current queue depths, and retention settings
120	#


Now let me compare the documentation against the script headers carefully:

## Discrepancies Found

**README Line 230-236 vs Script Headers:**

1. **docker_daily_restart.sh Schedule Mismatch**
   - README says: "1am via `daily_sync_maintenance.sh`"
   - Script header (line 8): "every night at 1am" ✓ MATCHES

2. **docker_weekly_restart.sh Missing from Scripts**
   - README mentions `docker_weekly_restart.sh` in line 231 and throughout (lines 170, 231, 260-261)
   - No script header provided/found for `docker_weekly_restart.sh` in the request
   - Table at line 230-236 lists it, flow diagram at lines 260-261 mentions it, but script was not in the files provided to review

3. **docker_update.sh Schedule Mismatch**
   - README line 232 says: "Daily before restart; weekly end of window"
   - Script header (lines 11, 23): "called by daily_sync_maintenance.sh before docker_daily_restart.sh" and mentions "remainder mode (weekly)" ✓ MATCHES

4. **docker_update_remaining.sh vs docker_update.sh --remainder Conflict**
   - README line 233 lists `docker_update_remaining.sh` as separate script
   - README line 262 also shows it as separate step
   - Script header for `docker_update_remaining.sh` (line 7-14) describes it as a standalone script ✓ MATCHES
   - BUT: `docker_update.sh` script header (lines 15, 111) describes `--remainder` mode, which appears to duplicate functionality
   - Documentation doesn't clarify the relationship between `docker_update.sh --remainder` and `docker_update_remaining.sh`

5. **downloaders_reset.sh Schedule Mismatch**
   - README line 236 says: "Every 30min via `critical_sync_maintenance.sh`"
   - Script header (lines 12-13): "Called every 30 minutes by critical_sync_maintenance.sh" ✓ MATCHES

## Summary

The primary discrepancy is the **missing documentation for `docker_weekly_restart.sh`** — it's referenced throughout the README and Manual but no script header was provided to verify against. All other scripts' schedules and behaviors match their documentation. The `docker_update.sh` script appears to have both normal and `--remainder` modes while a separate `docker_update_remaining.sh` script exists, but documentation doesn't clarify this dual approach.