Read the following files and report any discrepancies between the documentation and the actual script headers. The goal is to find places where the README or Manual says something that no longer matches what the script header says (wrong schedule, wrong config vars, wrong behavior description, missing scripts, extra scripts, etc.).

Files to read:
1. /boot/config/plugins/varaverk/System_Essentials/README-System_Essentials.md
2. /boot/config/plugins/varaverk/System_Essentials/Manual-System_Essentials.md
3. /boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md
4. Script headers (top ~120 lines each):
   - /boot/config/plugins/varaverk/System_Essentials/conf_sync.sh
   - /boot/config/plugins/varaverk/System_Essentials/arrs_failed_stalled_recovery.sh
   - /boot/config/plugins/varaverk/Orchestrators/array_started.sh
   - /boot/config/plugins/varaverk/Orchestrators/watchdog_orchestrator.sh
   - /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh
   - /boot/config/plugins/varaverk/Orchestrators/weekly_sync_maintenance.sh
   - /boot/config/plugins/varaverk/Orchestrators/sunday_morning_coffee_report.sh
   - /boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh
   - /boot/config/plugins/varaverk/Orchestrators/intermediate_sync_maintenance.sh
   - /boot/config/plugins/varaverk/Orchestrators/monthly_maintenance.sh

Report: list each discrepancy as "FILE: what the doc says vs what the header says". If everything matches, say so. Be specific — quote the conflicting text. Keep the report under 600 words.


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.
- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch

update-config keybindings-help verify code-review simplify fewer-permission-prompts loop schedule claude-api run init review security-review


I'll read the documentation files and script headers to identify any discrepancies.


1	# ━━━━━ SYSTEM ESSENTIALS ━━━━━
2
3	**System-level scripts that act on the server itself — not containers,
4	not media, not monitoring.** Keeping the server stable under load, recovering a
5	frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
6	shutdowns with proper warning sequences.
7
8	---
9
10	## ━━━ THE PROBLEM THAT BUILT THIS ━━━
11
12	**Server Getting Into Unstable States With No Recovery Path**
13	A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
14	watchdog tries to restart the container — but Docker itself is barely responding.
15	The restart hangs. The server needs a reboot, but nothing in the ecosystem is
16	authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
17	write log files. The server is functionally dead but still technically running.
18
19	Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
20	failures, OOM-confirmed bypass for RAM crises, strike system for sustained
21	threshold breaches. Last line of defense before a hard crash.
22
23	**WebGUI Freezing and Nobody Noticing**
24	The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
25	are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
26	load, settings don't save, containers can't be started or stopped via the UI. No
27	container-level alert fires because this isn't a container problem — it's a web
28	server problem. By the time someone notices it may have been broken for hours.
29
30	Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
31	php-fpm → emhttp. Lightest fix first. Silent when healthy.
32
33	**50+ Containers Starting and Filling Syslog With Veth Noise**
34	Array starts. 50+ containers come up simultaneously. Docker creates a virtual
35	network interface for each one. Each interface generates multiple syslog entries.
36	In the first minute after array start, syslog is buried under 200–400 lines of
37	`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
38	— a failed mount, a permission error, a service that didn't start — are invisible.
39
40	Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
41	starts. Applied at array start. Idempotent — silent when already correct.
42
43	**WebGUI Queuing Requests Under Load Without Explanation**
44	The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
45	explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
46	multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
47	those workers saturate immediately. New requests queue behind active ones.
48
49	Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
50	250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
51
52	**inotify Exhaustion Producing Unexplained Failures**
53	When inotify limits are exhausted, containers silently stop receiving filesystem
54	events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
55	file changes." Code-Server with node_modules alone can consume 100K–200K watches,
56	and all containers share the same pool.
57
58	Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
59	run FIRST in ARRAY_START_SCRIPTS before any containers start.
60
61	**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
62	The mover is running — moving a large batch of files from cache to array. A reboot
63	is triggered. The mover stops mid-file. The file exists partially on both cache and
64	array simultaneously. unRAID's deduplication layer is confused.
65
66	Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
67	SIGTERM (graceful — finishes current file), SIGKILL only if needed.
68
69	---
70
71	## ━━━ WHAT THIS FOLDER DOES ━━━
72
73	```
74	WebGUI availability     webgui_watchdog.sh        — nginx → php-fpm → emhttp escalation
75	Kernel tuning           inotify_tuning.sh        — file watch limits
76	                        php_fpm_max_children.sh  — PHP worker count
77	Log hygiene             docker_syslog_filter.sh  — suppress veth noise at start
78	                        clear_logs.sh            — weekly log trimming
79	Graceful operations     mover_stop.sh            — clean mover stop
80	                        rsync_stop.sh            — smart rsync stop (orchestrator-aware)
81	                        user_scripts_stop.sh     — stop running User Scripts
82	                        server_reboot.sh         — clean reboot with pre-flight warnings
83	```
84
85	> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
86	> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
87
88	---
89
90	## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
91
92	```
93	Orchestrators/
94	  array_started.sh ─────────────────────────► inotify_tuning.sh       (first in sequence)
95	                 ─────────────────────────► docker_syslog_filter.sh  (second)
96	                 ─────────────────────────► php_fpm_max_children.sh
97	  weekly_maintenance.sh ──────────────────► clear_logs.sh
98
99	  server_reboot.sh ────────────────────────► user_scripts_stop.sh   (called internally)
100
101	Watchdogs/
102	  stability_watchdog.sh and resource_watchdog.sh now live here.
103	  See Watchdogs/README-Watchdogs.md for how they relate to each other
104	  and to docker_watchdog.sh and System/storage_watchdog.sh.
105	```
106
107	---
108
109	## ━━━ SCRIPTS IN THIS FOLDER ━━━
110
111	| Script | Role | When It Runs |
112	|--------|------|-------------|
113	| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
114	| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
115	| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
116	| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
117	| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
118	| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
119	| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
120	| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
121	| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
122
123	---
124
125	## ━━━ HOW THE SCRIPTS RELATE ━━━
126
127	```
128	Array starts
129	    │
130	    ├─ inotify_tuning.sh       ← FIRST — kernel limits inherited at container launch
131	    ├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
132	    └─ php_fpm_max_children.sh ← before WebGUI is under load
133
134	Every minute (watchdog_orchestrator.sh in Orchestrators/):
135	    → Watchdogs/resource_watchdog.sh
136	    → Watchdogs/docker_watchdog.sh
137	    → Watchdogs/system_watchdog.sh (thin orchestrator)
138	        └─ Watchdogs/System/storage_watchdog.sh
139	        └─ Watchdogs/System/webgui_watchdog.sh
140	               WebGUI OK → silent exit
141	               Not responding:
142	                 Step 1: restart nginx       → recheck
143	                 Step 2: restart php-fpm     → recheck
144	                 Step 3: restart emhttp      → recheck
145	                 All failed → notify, exit 1
146	        └─ Watchdogs/System/network_watchdog.sh
147	    → Watchdogs/stability_watchdog.sh
148	    (see Watchdogs/README-Watchdogs.md for full flow)
149
150	Weekly (weekly_maintenance.sh):
151	    └─ clear_logs.sh
152	         System logs: clear if > LOG_MIN_SIZE_MB
153	         Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
154
155	Manual operations:
156	    mover_stop.sh        → wall → SIGTERM → SIGKILL → verify stopped
157	    rsync_stop.sh        → detect orchestrator → kill rsync (or orchestrator+rsync)
158	    user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
159	    server_reboot.sh     → pre-flight → wall → wait → VMs → Docker → sync → reboot
160	```
161


1	# ━━━━━ SYSTEM ESSENTIALS — Manual ━━━━━
2
3	Configuration reference, operational procedures, and troubleshooting for
4	system-level scripts. Read the ARRAY_START_SCRIPTS order section before
5	adding or reordering scripts at array start.
6
7	> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
8	> now live in `Watchdogs/`. Their configuration reference and troubleshooting
9	> procedures are in `Watchdogs/Manual-Watchdogs.md`.
10
11	---
12
13	## ━━━ CONTENTS ━━━
14
15	- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
16	- [webgui_watchdog.sh](#webgui_watchdogsh)
17	- [inotify_tuning.sh](#inotify_tuningsh)
18	- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
19	- [docker_syslog_filter.sh](#docker_syslog_filtersh)
20	- [clear_logs.sh](#clear_logssh)
21	- [mover_stop.sh](#mover_stopsh)
22	- [rsync_stop.sh](#rsync_stopsh)
23	- [user_scripts_stop.sh](#user_scripts_stopsh)
24	- [server_reboot.sh](#server_rebootsh)
25	- [Full Configuration Reference](#full-configuration-reference)
26	- [Troubleshooting](#troubleshooting)
27
28	---
29
30	## Output Tiers
31
32	All scripts use a two-tier output model: `echo` lines are always visible; `log`
33	lines only appear when `--log` is passed.
34
35	**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
36	state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
37	Per-check detail suppressed.
38
39	**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
40	`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
41	`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
42	final summary are visible. Per-item detail suppressed.
43
44	---
45
46	## ARRAY_START_SCRIPTS Order
47
48	> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
49	> scripts.** Getting it wrong causes subtle failures that don't show up
50	> immediately.
51
52	```bash
53	# master.conf
54	ARRAY_START_SCRIPTS=(
55	    "inotify_tuning.sh"           # 1 — FIRST: kernel limits must be set before
56	                                   #    any container starts. Containers inherit
57	                                   #    inotify limits at launch, not dynamically.
58	    "docker_syslog_filter.sh"     # 2 — SECOND: before any veth interfaces are
59	                                   #    created. If a container starts first, its
60	                                   #    veth creation is already in syslog.
61	    "php_fpm_max_children.sh"     # 3 — before WebGUI is under load
62	    "ramdisk_setup.sh"            # (from Transcodes/) before Emby starts
63	    ...
64	)
65	# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
66	# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
67	```
68
69	Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
70	the old low limits. The limits are kernel-wide — a restart of Code-Server picks
71	up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
72
73	Why docker_syslog_filter SECOND: The filter must be in place before any container
74	starts creating veth interfaces. The first container start after array start
75	generates veth messages — these will appear in syslog if the filter isn't active.
76
77	---
78
79	## webgui_watchdog.sh
80
81	### Escalation Logic
82
83	```
84	curl $WEBGUI_URL → 200 OK → exit 0 (silent)
85
86	Not responding:
87	  1. platform_restart_service nginx
88	     wait WEBGUI_NGINX_WAIT (15s) → recheck
89	     → recovered: notify, exit 0
90
91	  2. platform_restart_service php-fpm
92	     wait WEBGUI_PHP_WAIT (10s) → recheck
93	     → recovered: notify, exit 0
94
95	  3. platform_restart_service emhttp
96	     wait WEBGUI_EMHTTP_WAIT (30s) → recheck
97	     → recovered: notify, exit 0
98
99	  All three failed → notify warning, exit 1
100	```
101
102	### Configuration
103
104	```bash
105	WEBGUI_URL="http://localhost"   # URL to check
106	WEBGUI_TIMEOUT=5                # curl timeout in seconds
107	WEBGUI_NGINX_WAIT=15            # seconds after nginx restart before recheck
108	WEBGUI_PHP_WAIT=10              # seconds after php-fpm restart before recheck
109	WEBGUI_EMHTTP_WAIT=30           # seconds after emhttp restart before recheck
110	```
111
112	### WebGUI Frozen — Manual Recovery
113
114	```bash
115	# Check which services are running:
116	webgui_watchdog.sh --status
117
118	# Try manual restart sequence (mirrors what the script calls):
119	# Source the ecosystem first to get platform functions:
120	source /boot/config/plugins/varaverk/load_config.sh
121	platform_restart_service nginx
122	# wait 15s, then:
123	curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
124
125	# If nginx did not fix it, php-fpm:
126	platform_restart_service php-fpm
127
128	# If still down, emhttp:
129	platform_restart_service emhttp
130
131	# Raw equivalents (no source needed — paste directly into terminal):
132	# /etc/rc.d/rc.nginx restart
133	# /etc/rc.d/rc.php-fpm restart
134	# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
135
136	# If all three failed:
137	server_reboot.sh --status   # check for active sessions first
138	```
139
140	---
141
142	## inotify_tuning.sh
143
144	### What It Sets
145
146	```bash
147	INOTIFY_MAX_INSTANCES=1024       # max inotify fd objects per user (default: 128)
148	INOTIFY_MAX_WATCHES=1048576      # max watches shared across all users (default: 8192)
149	INOTIFY_MAX_QUEUED_EVENTS=32768  # max buffered events (default: 16384)
150	```
151
152	Verify current values:
153
154	```bash
155	inotify_tuning.sh --status
156	# Shows current vs target for each limit, active instance count, top consumers
157	```
158
159	### If Code-Server Shows "Unable to Watch for File Changes"
160
161	```bash
162	# 1. Verify limits are set:
163	sysctl fs.inotify.max_user_watches
164	# Expected: 1048576
165
166	# 2. Check total usage across all containers:
167	find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l
168
169	# 3. If limits are set but Code-Server still shows the error:
170	docker restart Code-Server
171	# Running containers inherit limits at launch. Restart picks up the new values.
172
173	# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
174	inotify_tuning.sh --log
175	```
176
177	---
178
179	## php_fpm_max_children.sh
180
181	### What It Sets
182
183	```bash
184	PHP_MAX_CHILDREN=250             # target pm.max_children (default: 4-8 on unRAID)
185	PHP_CONF="/etc/php83/php-fpm.d/www.conf"
186	```
187
188	250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
189	The default of 4–8 saturates immediately under load on a busy server.
190
191	### Verify
192
193	```bash
194	php_fpm_max_children.sh --status
195	# Shows current value vs target, PHP-FPM worker count
196
197	# Manual verify:
198	grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
199	# Expected: pm.max_children = 250
200	```
201
202	### If WebGUI Is Slow Despite the Setting
203
204	```bash
205	# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
206	# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
207	# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
208	# pm.max_children value may still be too low, or there's a PHP worker leak.
209
210	# Check running worker count:
211	pgrep -fc php-fpm
212	# Compare to pm.max_children — if equal, workers are saturated
213
214	# Increase if needed:
215	# master.conf: PHP_MAX_CHILDREN=350
216	# Then: php_fpm_max_children.sh --log  (will update and restart php-fpm)
217	```
218
219	---
220
221	## docker_syslog_filter.sh
222
223	### What It Creates
224
225	```
226	/etc/rsyslog.d/ignore-docker-veth.conf:
227	  if ($msg contains "veth" or $msg contains "docker0") then {
228	      stop
229	  }
230	```
231
232	This drops any syslog message containing "veth" or "docker0" before it reaches
233	any output target, including the log file. The drop rule is applied at rsyslog
234	level — not at the log viewer level.
235
236	### Verify
237
238	```bash
239	docker_syslog_filter.sh --status
240	# Shows filter file content and rsyslog process state
241
242	# Manual verify:
243	cat /etc/rsyslog.d/ignore-docker-veth.conf
244	pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"
245
246	# Test the filter is active (should produce no syslog output):
247	logger "test veth message"
248	grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"
249	```
250
251	### If Syslog Still Has Veth Noise
252
253	```bash
254	# 1. Verify filter file exists with correct content:
255	docker_syslog_filter.sh --status
256
257	# 2. If content differs — re-apply:
258	docker_syslog_filter.sh --log
259
260	# 3. Verify rsyslog is using the conf.d directory:
261	grep -r "IncludeConfig" /etc/rsyslog.conf
262	# Expected: IncludeConfig /etc/rsyslog.d/*.conf  (or similar)
263	```
264
265	---
266
267	## clear_logs.sh
268
269	### Thresholds
270
271	```bash
272	LOG_MIN_SIZE_MB=10       # skip system log if under this — keep recent history
273	LOG_DOCKER_MAX_MB=100    # clear Docker container log only if over this
274
275	LOG_FILES=(
276	    "/var/log/syslog"
277	    "/var/log/messages"
278	    "/var/log/dmesg"
279	)
280	```
281
282	### Why Truncation Not Deletion
283
284	unRAID writes logs to tmpfs (`/var/log`). Truncation (`: > file`) keeps the file
285	descriptor open and valid while emptying content — syslogd continues writing to
286	the same fd without interruption. Deleting the file would orphan the file
287	descriptor and syslog would stop writing until restarted.
288
289	### Identifying Large Docker Logs
290
291	```bash
292	clear_logs.sh --status
293	# Shows top 10 Docker logs by size, current size vs threshold
294
295	# Find the biggest log manually:
296	du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
297	```
298
299	---
300
301	## mover_stop.sh
302
303	### Stop Sequence
304
305	```
306	1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
307	2. Wall message to all logged-in terminal users
308	3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
309	4. SIGTERM — mover finishes its current file operation, then stops
310	5. Wait 5 seconds → verify stopped
311	6. SIGKILL if still running — forced stop, partial files possible
312	7. Final verify — error if still running after SIGKILL
313	```
314
315	SIGTERM first because the mover can finish the file it is currently moving,
316	leaving no partial copies split across cache and array. SIGKILL is a last resort.
317
318	### Configuration
319
320	```bash
321	MOVER_STOP_TIMEOUT=30   # seconds between wall warning and SIGTERM
322	```
323
324	### Usage
325
326	```bash
327	mover_stop.sh              # check and stop if running
328	mover_stop.sh --status     # show current mover state and PID
329	mover_stop.sh --dry-run    # show what would happen without stopping
330	```
331
332	---
333
334	## rsync_stop.sh
335
336	### Auto-Detection Logic
337
338	rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync)
339	is the parent of the running rsync process by scanning lock files in `$LOCK_DIR`.
340
341	**Default behavior (orchestrator detected):**
342	Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the
343	next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.
344
345	**Default behavior (no orchestrator):**
346	Kills rsync directly (standalone rsync.sh run).
347
348	**--full-stop:**
349	Kills the orchestrator first, then kills rsync. Nothing continues after this.
350	Use when everything needs to stop immediately.
351
352	### Container Recovery
353
354	After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTAINER_NAMES`
355	for any that were stopped by the interrupted rsync session and restarts them.
356	Remote containers are left for docker_watchdog.sh to recover.
357
358	Skip container recovery with `--rsync-only` — used when called by other scripts
359	that handle recovery themselves.
360
361	### Usage
362
363	```bash
364	rsync_stop.sh                        # smart stop (auto-detect orchestrator)
365	rsync_stop.sh --full-stop            # kill orchestrator + rsync
366	rsync_stop.sh --rsync-only           # kill rsync, skip container recovery
367	rsync_stop.sh --status               # show local and remote rsync state
368	rsync_stop.sh --dry-run              # preview without changes
369	rsync_stop.sh --full-stop --dry-run  # preview full stop
370	```
371
372	---
373
374	## user_scripts_stop.sh
375
376	### Process Identification
377
378	Scans `/proc/*/cmdline` for any process whose command line contains
379	`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
380	`/tmp/user.scripts/` before execution — this signature is reliable regardless of
381	what the script is named or how it was launched.
382
383	Script names are extracted from the path for display: you see which scripts are
384	being stopped, not just PIDs.
385
386	### Self-Exclusion
387
388	If this script is run via the User Scripts plugin, it would find its own PID in
389	the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
390	from the kill list.
391
392	### Usage
393
394	```bash
395	user_scripts_stop.sh              # stop all User Script processes
396	user_scripts_stop.sh --status     # show running scripts with names and elapsed time
397	user_scripts_stop.sh --dry-run    # show what would be stopped
398	```
399
400	---
401
402	## server_reboot.sh
403
404	### Full Shutdown Sequence
405
406	```
407	1. Pre-flight checks (warn only — do not block):
408	   - rsync running?     → warn, suggest rsync_stop.sh first
409	   - mover running?     → warn, suggest mover_stop.sh first
410	   - Emby sessions?     → warn (active streams will be interrupted)
411
412	2. Wall message to all logged-in terminal users
413
414	3. unRAID dashboard notification
415
416	4. Wait REBOOT_SLEEP seconds (default: 30)
417
418	5. Graceful VM shutdown:
419	   virsh shutdown <each VM>  (ACPI signal — clean shutdown)
420	   Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond
421
422	6. /etc/rc.d/rc.libvirt stop   (VM Manager)
423
424	7. Orchestrators/array_stopping.sh — safe ordered array stop:
425	   - user_scripts_stop.sh        stop background User Scripts
426	   - fallback.sh --stop          graceful fallback teardown
427	   - rsync_stop.sh --rsync-only  kill active rsync transfers
428	   - mover_stop.sh               stop mover
429	   - docker_container_stop.sh    stop all containers gracefully
430
431	8. sync   (flush filesystem buffers to disk)
432
433	9. /sbin/reboot
434	```
435
436	### Configuration
437
438	```bash
439	REBOOT_SLEEP=30      # seconds between warning and shutdown sequence
440	REBOOT_VM_WAIT=30    # seconds to wait for VMs to shut down gracefully
441	```
442
443	### Recommended Pre-Reboot Sequence
444
445	For a clean reboot when services are active:
446
447	```bash
448	rsync_stop.sh          # stop any active rsync (smart mode)
449	mover_stop.sh          # stop mover gracefully
450	server_reboot.sh --status  # check what's still running
451	server_reboot.sh --reason="planned maintenance"
452	```
453
454	### Usage
455
456	```bash
457	server_reboot.sh                         # reboot with 30s warning
458	server_reboot.sh --dry-run               # walk through without rebooting
459	server_reboot.sh --status                # show what would be affected
460	server_reboot.sh --reason="disk work"    # include reason in notification
461	```
462
463	---
464
465	## Full Configuration Reference
466
467	> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
468	> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
469
470	```bash
471	# master.conf
472
473	# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
474	WEBGUI_URL="http://localhost"
475	WEBGUI_TIMEOUT=5
476	WEBGUI_NGINX_WAIT=15
477	WEBGUI_PHP_WAIT=10
478	WEBGUI_EMHTTP_WAIT=30
479
480	# ── inotify Tuning ─────────────────────────────────────────────────────────────
481	INOTIFY_MAX_INSTANCES=1024
482	INOTIFY_MAX_WATCHES=1048576
483	INOTIFY_MAX_QUEUED_EVENTS=32768
484
485	# ── PHP-FPM ────────────────────────────────────────────────────────────────────
486	PHP_MAX_CHILDREN=250
487	PHP_CONF="/etc/php83/php-fpm.d/www.conf"
488
489	# ── Syslog Filter ──────────────────────────────────────────────────────────────
490	FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
491
492	# ── Log Cleaner ────────────────────────────────────────────────────────────────
493	LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
494	LOG_MIN_SIZE_MB=10
495	LOG_DOCKER_MAX_MB=100
496
497	# ── Mover Stop ─────────────────────────────────────────────────────────────────
498	MOVER_STOP_TIMEOUT=30
499
500	# ── Server Reboot ──────────────────────────────────────────────────────────────
501	REBOOT_SLEEP=30
502	REBOOT_VM_WAIT=30
503	```
504
505	---
506
507	## Troubleshooting
508
509	> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
510	> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
511
512	### rsync_stop Killed the Wrong Thing
513
514	If --full-stop killed an orchestrator you didn't intend to kill:
515
516	```bash
517	# Next time use default mode (no --full-stop) to kill only rsync subprocess.
518	# To verify what would be killed before running:
519	rsync_stop.sh --status       # shows running rsync and detected orchestrators
520	rsync_stop.sh --dry-run      # shows smart mode decision
521	rsync_stop.sh --full-stop --dry-run  # shows full-stop decision
522	```
523
524	### WebGUI Recovery After All Three Steps Failed
525
526	```bash
527	# Check if processes are running:
528	pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
529	pgrep emhttpd   && echo "emhttp: yes" || echo "emhttp: no"
530	pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
531
532	# Check recent nginx errors:
533	cat /var/log/nginx/error.log | tail -20
534
535	# Check emhttp log:
536	tail -20 /var/log/syslog | grep emhttp
537
538	# Last resort — reboot:
539	server_reboot.sh --reason="WebGUI unrecoverable"
540	```
541
542	### PHP-FPM Config Not Found After unRAID Update
543
544	unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
545	errors with "config file not found":
546
547	```bash
548	# Find the new config path:
549	find /etc -name "www.conf" 2>/dev/null
550
551	# Update PHP_CONF in master.conf:
552	PHP_CONF="/etc/php84/php-fpm.d/www.conf"   # example for php84
553
554	# Verify with:
555	php_fpm_max_children.sh --status
556	```
557


1	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2	# 🎯  ORCHESTRATORS
3	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4
5	**Sequential job runners that coordinate multiple scripts into single scheduled units.**
6	Orchestrators contain no business logic — they call other scripts in order, track
7	pass/fail per job, and produce one clean summary. Configuration lives in `master.conf`.
8	Adding or removing a job never requires touching the orchestrator script itself.
9
10	> **The Varaverk scheduler runs only orchestrators.** Every cron entry, every array
11	> start/stop event, every scheduled operation runs through an orchestrator. The individual
12	> scripts it calls are never scheduled directly — they run in a defined order inside a
13	> coordinated window, with a unified summary at the end.
14
15	---
16
17	## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
18
19	---
20
21	### 🔴  Race Conditions From Independent Scheduling
22
23	Media permissions ran at 01:05. Arr cleanup ran at 01:00. Arr cleanup started
24	five minutes before permissions were applied — running against files that were still
25	owned by root, silently failing to delete the ones it should have cleaned up. Both
26	scripts reported success. Neither knew about the other. The result was a library
27	that looked cleaned but wasn't.
28
29	The fix: orchestrators enforce order. `daily_sync_maintenance.sh` runs permissions
30	first, then arr cleanup. The arr scripts see correct ownership every time because the
31	orchestrator guarantees it. No race, no silent failure, no coordinating cron entries.
32
33	---
34
35	### 🔴  Six Separate Notifications Instead of One
36
37	Before orchestrators, each script sent its own notification on completion. A single
38	daily run produced six separate notification pings — one for permissions, one for
39	each arr cleanup, one for the cleaner, one for docker restart. Six bells for one
40	maintenance window. Worse, if something failed in the middle, you'd get some
41	notifications and not others, and figuring out which step failed meant correlating
42	timestamps across multiple notification messages.
43
44	The fix: orchestrators collect all results and send one notification at the end.
45	One summary. One bell. Clear pass/fail count. If something failed, the summary
46	tells you which job and what happened — no correlation needed.
47
48	---
49
50	### 🔴  Transcode Manager Triggering Unnecessary SSD Flips
51
52	`transcode_manager.sh` ran every 7 minutes on its own. It checked ramdisk usage —
53	saw 6.8GB used, threshold is 6.5GB, flipped sessions to SSD. One minute later
54	`transcode_cleanup.sh` ran and removed 4GB of stale segment files from ended sessions.
55	Actual usage was 2.8GB. Sessions were now on SSD for no reason. Users experiencing
56	slightly worse performance. The flip counter incremented for nothing.
57
58	The fix: `transcode_management.sh` runs cleanup first, manager second, every cycle.
59	The manager always sees post-cleanup usage. Stale files can't trigger a flip because
60	they're gone before the manager looks. The correct order requires exactly one
61	orchestrator to enforce it.
62
63	---
64
65	### 🔴  Failed Imports Sitting Stalled for Days
66
67	A release downloads successfully but Lidarr can't import it — wrong format, incorrect
68	tags, file already exists. Lidarr marks it `importFailed` and stops. Nobody notices.
69	The download client has the file, Lidarr has given up, and nothing is going to happen
70	until someone manually opens Lidarr, identifies the problem, blocklists the release,
71	and triggers a new search. This takes minutes to do — but nobody does it at 3am
72	when it usually happens.
73
74	The fix: `arrs_failed_stalled_recovery.sh` runs every 6 hours. It finds all
75	`importFailed`, `importPending`, `error`, and `stalled` items, blocklists them, removes
76	them from the queue, and triggers a new search — automatically. By morning the failed
77	import has already been replaced by a working one. No manual intervention required.
78
79	---
80
81	### 🔴  Array Start Scripts Running in Wrong Order or Not at All
82
83	Startup scripts configured individually ran in an unpredictable order. The ramdisk
84	setup might run after Emby starts. The syslog filter might run after containers have
85	already created veth interfaces. PHP-FPM tuning might run after the WebGUI has already
86	served its first requests. Each script competed for the same startup slot with no
87	guaranteed order.
88
89	The fix: `array_started.sh` is the only array-start entry in the Varaverk scheduler.
90	It launches every startup script in a defined order, with one-second settle between
91	each, and reports which succeeded and which failed. Order is guaranteed. Nothing starts
92	before its dependency. Everything is visible in a single summary.
93
94	---
95
96	## ━━━ THE ORCHESTRATOR MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
97
98	The Varaverk scheduler contains exactly these entries:
99
100	```bash
101	# Array start event (Varaverk disks_mounted hook → cron: "array_start"):
102	array_started.sh
103
104	# Cron — managed via Varaverk Scheduler:
105	*/7   * * * *    transcode_management.sh
106	*/15  * * * *    watchdog_orchestrator.sh          ← resource → docker → system → stability
107	*/30  * * * *    critical_sync_maintenance.sh      ← auth + Emby dirty sync + partnership
108	0  */4 * * *     intermediate_sync_maintenance.sh  ← arr sync + failed recovery + optional rsync
109	0   1 * * *      daily_sync_maintenance.sh
110	0   7 * * 0      sunday_morning_coffee_report.sh
111	30  2 * * 0      weekly_sync_maintenance.sh
112	0   0 15 * *     monthly_maintenance.sh            ← uptime-gated: ZFS scrub, SMART tests
113
114	# Manual only (not scheduled):
115	fallback_test.sh, emby_database_repair.sh, repair tools
116	```
117
118	Every job list is configured in the `ORCHESTRATORS` section of `master.conf`.
119	No orchestrator script ever changes when jobs are added or removed — only `master.conf` changes.
120
121	---
122
123	## ━━━ THE ORCHESTRATOR PATTERN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
124
125	All orchestrators follow the same structure:
126
127	```
128	1. Setup          → validate config, detect_hosts(), acquire_lock
129	2. Pre-flight     → fail fast checks before any work begins
130	3. Job loop       → run each job, track pass/fail, continue on failure
131	4. Summary        → one clean report of all job results
132	5. Notification   → one notify per run on failure (never per job)
133	```
134
135	Properties that apply to every orchestrator:
136
137	```
138	Consistent output    → every orchestrator looks the same in logs
139	No silent failures   → pass/fail tracked per job, all in summary
140	Resilient            → one job failing does not stop the rest
141	Single notification  → one bell per run, not one per job
142	--dry-run cascade    → passes --dry-run through to every child script
143	--status support     → show configured jobs and exit
144	```
145
146	---
147
148	## ━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
149
150	All scripts use a two-tier output model: `echo` lines are always visible; `log`
151	lines only appear when `--log` is passed.
152
153	**One-shot orchestrators** (`array_started.sh`, `array_stopping.sh`, `sunday_morning_coffee_report.sh`):
154	without `--log`, section headers, per-phase results, and the final summary are
155	visible. Per-item detail suppressed.
156
157	**Periodic orchestrators** (`critical_sync_maintenance.sh`, `daily_sync_maintenance.sh`,
158	`intermediate_sync_maintenance.sh`, `weekly_sync_maintenance.sh`): without `--log`,
159	phase headers, per-phase completion status, and the final summary are visible. Per-share
160	and per-job detail suppressed.
161
162	**High-frequency orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent
163	during clean cycles. Only state transitions, errors, and startup-grace expiry shown
164	without `--log`.
165
166	---
167
168	## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
169
170	| Script | What It Orchestrates | Schedule |
171	|--------|---------------------|----------|
172	| `array_started.sh` | All array startup scripts in order | `array_start` event (Varaverk plugin hook) |
173	| `watchdog_orchestrator.sh` | resource → docker → system → api_renew → stability watchdogs | Every 15 minutes |
174	| `transcode_management.sh` | Cleanup then manager — order critical | Every 7 minutes |
175	| `arrs_failed_stalled_recovery.sh` | Failed import + stalled download recovery | Every 6 hours |
176	| `daily_sync_maintenance.sh` | git pull → sync → media maintenance → restarts | 1am daily |
177	| `weekly_sync_maintenance.sh` | Stop → update → clean sync → start → weekly restarts | 2:30am Sunday |
178	| `monthly_maintenance.sh` | Uptime-triggered heavy tasks — ZFS scrub, SMART tests | Daily check, fires when uptime ≥ 30d |
179	| `intermediate_sync_maintenance.sh` | arr library sync, artwork, failed recovery | Every 4 hours |
180
181	---
182
183	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
184	## 🚀  array_started.sh
185	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
186
187	Single array-start entry for the entire ecosystem. Fired by the Varaverk plugin's
188	`disks_mounted` event hook. Launches every startup script in order — each as a
189	background process — and reports which succeeded and which failed.
190
191	```bash
192	# Triggered by: Plugin/unraid/event/disks_mounted/array_start_jobs
193	# schedule.json entry: "Orchestrators/array_started.sh" → cron: "array_start"
194	```
195
196	---
197
198	### ── Execution Order ──────────────────────────────────────────────────────────
199
200	```bash
201	# master.conf
202	# ─────────────────────────────────────────────────────────────────────────────
203	# Order matters — each entry depends on the previous ones having run.
204	# See comments for why each position is correct.
205	#
206	ARRAY_START_SCRIPTS=(
207	    # ── One-shot scripts — run and exit naturally ─────────────────────────────
208	    "System_Essentials/unraid_api_key_renew.sh"   # re-register API key FIRST — unraid-api
209	                                                   # registry is ephemeral, lost on service restart
210	    "System_Essentials/inotify_tuning.sh"         # raise inotify BEFORE containers start
211	                                                   # containers inherit limits at startup —
212	                                                   # if Code-Server starts with low limits
213	                                                   # it keeps them until restart
214	    "System_Essentials/docker_syslog_filter.sh"   # suppress veth noise BEFORE containers create
215	                                                   # veth interfaces — otherwise the first boot
216	                                                   # always has unfiltered veth spam
217	    "System_Essentials/php_fpm_max_children.sh"   # WebGUI tuning — before any WebGUI requests
218	    "Transcodes/ramdisk_setup.sh"                 # create tmpfs + symlink BEFORE Emby starts —
219	                                                   # Emby needs the transcode path to exist
220	    "Docker_Essentials/docker_network_connect.sh"  # ensure networks + connections BEFORE
221	                                                   # watchdogs check container states
222
223	    # ── Continuous scripts — run until array stops ─────────────────────────────
224	    "Fallback/fallback.sh"                         # fallback LAST — needs everything else stable
225	)
226
227	# NOTE: watchdogs (docker_watchdog, system_watchdog, stability_watchdog) are NOT here.
228	# They run via watchdog_orchestrator.sh on cron every 15 minutes — not as daemons.
229	```
230
231	---
232
233	### ── One-Shot vs Continuous Detection ───────────────────────────────────────
234
235	```bash
236	# ─────────────────────────────────────────────────────────────────────────────
237	# Each script is launched with `bash script.sh &` — background process.
238	# After 1 second:
239	#   PID still alive → continuous script (running in background)
240	#                     logged as: "fallback.sh — running (PID 12345)"
241	#   PID dead + exit 0 → one-shot completed successfully
242	#                        logged as: "inotify_tuning.sh — completed (one-shot)"
243	#   PID dead + exit N → failure
244	#                        logged as: "ramdisk_setup.sh — exited with code 1"
245	#                        full path printed — debugging is immediate
246	#
247	# This means the orchestrator correctly identifies and reports all startup
248	# scripts without needing to know in advance which ones are continuous.
249	```
250
251	---
252
253	### ── Auto-Fix Permissions ────────────────────────────────────────────────────
254
255	```bash
256	# ─────────────────────────────────────────────────────────────────────────────
257	# Scripts that are not executable are chmod +x'd automatically before launch.
258	# A permissions problem on a startup script does not cause a silent skip —
259	# the orchestrator fixes it and proceeds, then logs that it did so.
260	# This prevents "why didn't X run on startup" questions.
261	```
262
263	---
264
265	### ── Usage ───────────────────────────────────────────────────────────────────
266
267	```bash
268	# ─────────────────────────────────────────────────────────────────────────────
269	# Normal — fired by Varaverk disks_mounted event hook. Never run manually in production.
270	# array_started.sh runs once and exits — the continuous scripts it launched
271	# keep running as background processes.
272	# ─────────────────────────────────────────────────────────────────────────────
273	array_started.sh
274
275	# ─────────────────────────────────────────────────────────────────────────────
276	# Dry run — show what would be launched, in order, without launching anything.
277	# Use to verify the ARRAY_START_SCRIPTS list before an array restart.
278	# ─────────────────────────────────────────────────────────────────────────────
279	array_started.sh --dry-run
280
281	# ─────────────────────────────────────────────────────────────────────────────
282	# Status — show each configured script with its current running state.
283	#   RUNNING (PID XXXXX) — continuous script currently active
284	#   not running          — one-shot that has completed, or continuous not yet started
285	#   FILE NOT FOUND       — script path wrong or missing
286	# ─────────────────────────────────────────────────────────────────────────────
287	array_started.sh --status
288	```
289
290	---
291
292	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
293	## 🎬  transcode_management.sh
294	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
295
296	Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every
297	7 minutes. Exists because the order is not optional — the manager must always see
298	post-cleanup usage to make accurate flip decisions.
299
300	```bash
301	# Scheduled: */7 * * * *   (every 7 minutes)
302	```
303
304	---
305
306	### ── Why Order Is Non-Negotiable ─────────────────────────────────────────────
307
308	```bash
309	# ─────────────────────────────────────────────────────────────────────────────
310	# Emby writes segment files to the transcode directory as it buffers streams.
311	# When a stream ends, Emby deletes its own active files — but may leave behind
312	# stale segment files from sessions that ended uncleanly. These consume real
313	# ramdisk space. The manager has no way to know if they're active or stale.
314	#
315	# Without correct order:
316	#   Manager runs → sees 6.8GB used (stale files inflating) → exceeds threshold
317	#   → flips sessions to SSD → flip counter incremented
318	#   Cleanup runs → removes 4GB of stale files → actual usage was 2.8GB
319	#   → flip was unnecessary — sessions now on SSD for no reason
320	#
321	# With correct order (this orchestrator):
322	#   Cleanup runs → removes stale files → actual usage 2.8GB
323	#   Manager runs → sees 2.8GB → below threshold → stays on ramdisk ✅
324	#   → no flip, no wasted counter, correct decision every time
325	# ─────────────────────────────────────────────────────────────────────────────
326	```
327
328	---
329
330	### ── What Each Child Script Does ────────────────────────────────────────────
331
332	```bash
333	# ─────────────────────────────────────────────────────────────────────────────
334	# transcode_cleanup.sh:
335	#   Identifies segment files not currently open by any process (via lsof)
336	#   Removes them from the transcode directory
337	#   If usage drops enough after cleanup → triggers flip-back to ramdisk
338	#   (handles the recovery direction so manager doesn't have to)
339	#
340	# transcode_manager.sh:
341	#   Reads current ramdisk usage after cleanup has run
342	#   Compares against RAMDISK_WARN_GB threshold
343	#   Flips the symlink if needed (ramdisk → SSD or SSD → ramdisk)
344	#   Writes one entry to TRANSCODE_DAILY_LOG for the weekly coffee report
345	#   Shows active Emby sessions with their play method
346	# ─────────────────────────────────────────────────────────────────────────────
347	```
348
349	---
350
351	### ── Daily Log ────────────────────────────────────────────────────────────────
352
353	```bash
354	# master.conf
355	# ─────────────────────────────────────────────────────────────────────────────
356	# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each cycle.
357	# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
358	#
359	# The sunday_morning_coffee_report.sh reads this log for weekly stats:
360	#   Peak ramdisk usage across the week
361	#   Total flip count (unnecessary flips visible here)
362	#   Session split: how often ramdisk vs SSD was used
363	#
364	# Log trimmed to TRANSCODE_LOG_RETENTION days on every write — bounded, never grows.
365	#
366	    TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
367	    TRANSCODE_LOG_RETENTION=90    # days
368	    TRANSCODE_STATE_FILE="/tmp/transcode_state.db"    # /tmp — resets on reboot
369	```
370
371	---
372
373	### ── Usage ───────────────────────────────────────────────────────────────────
374
375	```bash
376	transcode_management.sh              # normal run (every 7 min via cron)
377	transcode_management.sh --dry-run    # passes --dry-run to both child scripts
378	transcode_management.sh --status     # show config, current state, daily log stats
379	transcode_management.sh --log        # verbose output from both child scripts
380	```
381
382	---
383
384	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
385	## 🔄  arrs_failed_stalled_recovery.sh
386	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
387
388	Automatic recovery from failed imports and stalled downloads across Sonarr, Radarr,
389	and Lidarr. Blocklists the bad release, removes it from the queue, and triggers a
390	new search — hands-free recovery while you sleep.
391
392	```bash
393	# Scheduled: 0 */6 * * *   (every 6 hours)
394	```
395
396	---
397
398	### ── Four Problem Types ───────────────────────────────────────────────────────
399
400	```bash
401	# ─────────────────────────────────────────────────────────────────────────────
402	# The arr queue API reports these status values for problem items:
403	#
404	# importFailed   — downloaded successfully but arr couldn't import the file
405	#                  Common causes: wrong format for quality profile, corrupted file,
406	#                  duplicate already in library, permission issue on import path
407	#                  Self-resolution: never — arr stops trying after first failure
408	#
409	# importPending  — downloaded, stuck waiting for import to begin
410	#                  Common causes: import queue backed up, arr paused, API error
411	#                  Self-resolution: sometimes — but stuck for hours is always wrong
412	#
413	# error          — serious failure state not covered by the above
414	#                  Common causes: indexer issues, download client unreachable, disk full
415	#
416	# stalled        — download stuck with no connections or no progress
417	#                  Common causes: no seeders, VPN routing issue, tracker ban
418	#                  Self-resolution: never without a source change
419	#
420	# NOT touched — items with status "downloading" or "imported" are never touched.
421	# Safe to run at any time — only processes items that are already broken.
422	# ─────────────────────────────────────────────────────────────────────────────
423	```
424
425	---
426
427	### ── What It Does Per Problem Item ──────────────────────────────────────────
428
429	```bash
430	# ─────────────────────────────────────────────────────────────────────────────
431	# For each problem item found, in order:
432	#
433	# 1. Blocklist the release
434	#    Prevents the arr from grabbing the exact same release again immediately.
435	#    The bad indexer result goes into the blocklist — future searches skip it.
436	#
437	# 2. Remove from queue
438	#    Tells the download client to stop and remove the failed download.
439	#    Frees up the slot for the replacement.
440	#
441	# 3. Trigger new search
442	#    Arr searches for a different release meeting the quality profile.
443	#    If a suitable alternative exists, it starts downloading automatically.
444	#    If not, the item is marked as "awaiting upgrade" — arr will retry when
445	#    a new indexer result appears.
446	#
447	# The entire cycle from "failed import" to "replacement downloading" happens
448	# without any human involvement.
449	# ─────────────────────────────────────────────────────────────────────────────
450	```
451
452	---
453
454	### ── Age Threshold ────────────────────────────────────────────────────────────
455
456	```bash
457	# master.conf
458	# ─────────────────────────────────────────────────────────────────────────────
459	# Items newer than ARR_IMPORT_RECOVERY_AGE hours are skipped.
460	# Arrs retry on their own schedule after initial failures — a 30-minute-old
461	# importFailed may still resolve itself. Waiting 6 hours before intervening
462	# gives the arr a full retry cycle before this script steps in.
463	#
464	    ARR_IMPORT_RECOVERY_AGE=6    # hours — skip items newer than this
465	```
466
467	---
468
469	### ── Host Awareness ───────────────────────────────────────────────────────────
470
471	```bash
472	# master.conf + host*.conf
473	# ─────────────────────────────────────────────────────────────────────────────
474	# Each arr is independently toggled per host.
475	# Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action.
476	# HOST2 has its own Sonarr and Radarr for its anime shares.
477	#
478	    HOST1_SONARR_RECOVERY=true    # HOST1 Sonarr — Tv_Shows
479	    HOST1_RADARR_RECOVERY=true    # HOST1 Radarr — Movies
480	    HOST1_LIDARR_RECOVERY=true    # HOST1 Lidarr — Music (HOST1 only)
481	    HOST2_SONARR_RECOVERY=true    # HOST2 Sonarr — Anime_Shows
482	    HOST2_RADARR_RECOVERY=true    # HOST2 Radarr — Anime_Movies
483	#
484	# API versions (current — update if arr major version changes):
485	#   Sonarr v4  → /api/v3/
486	#   Radarr v6  → /api/v3/
487	#   Lidarr v3  → /api/v1/
488	```
489
490	---
491
492	### ── Usage ───────────────────────────────────────────────────────────────────
493
494	```bash
495	arrs_failed_stalled_recovery.sh              # normal run
496	arrs_failed_stalled_recovery.sh --dry-run    # show what would be actioned, no changes
497	arrs_failed_stalled_recovery.sh --log        # verbose — show each queue item evaluated
498	arrs_failed_stalled_recovery.sh --status     # show arr config and API connectivity
499	```
500
501	---
502
503	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
504	## 📅  daily_sync_maintenance.sh
505	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
506
507	Full daily maintenance window orchestrator. The entire 1am window — git pull, media
508	share sync, media maintenance, and docker daily restarts — in one scheduled entry.
509	Runs on both servers; `detect_hosts()` determines which direction each sync goes.
510
511	```bash
512	# Scheduled: 0 1 * * *   (1am daily — on BOTH servers)
513	```
514
515	---
516
517	### ── Execution Order ──────────────────────────────────────────────────────────
518
519	```bash
520	# ─────────────────────────────────────────────────────────────────────────────
521	# The split into pre-sync and post-sync is based on a simple rule:
522	#   git pull runs before anything — maintenance uses the latest scripts
523	#   rsync runs in the middle — media management uses the post-sync state
524	#   docker restart runs last — after everything else has completed
525	#
526	# 1. Pre-sync: git_pull_execute.sh
527	#    Pull latest scripts first. Everything that follows runs on current code.
528	#
529	# 2. Rsync window: HOST*_DAILY_SYNC_SHARES + HOST*_PERSONAL_SHARES
530	#    Each server pushes its own truth shares to the other.
531	#    HOST1 → pushes Movies, Tv_Shows, Music → HOST2
532	#    HOST2 → pushes Anime_Shows, Anime_Movies → HOST1
533	#    Personal encrypted shares appended after standard shares.
534	#    Drive temperature exit codes respected — skip share or abort all on CRIT.
535	#
536	# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
537	#    media_management.sh → permissions + cleaners + arr cleanup
538	#    docker_daily_restart.sh → nightly container restarts
539	# ─────────────────────────────────────────────────────────────────────────────
540	```
541
542	---
543
544	### ── Bidirectional — Same Script, Correct Direction Automatic ────────────────
545
546	```bash
547	# ─────────────────────────────────────────────────────────────────────────────
548	# detect_hosts() sets MY_ID at runtime and aliases HOST*_DAILY_SYNC_SHARES
549	# to DAILY_SYNC_SHARES. The script uses DAILY_SYNC_SHARES — always the right
550	# list for whichever server is running.
551	#
552	# No HOST1/HOST2 comparisons in the script. Configuration drives direction.
553	#
554	# HOST1 runs this script at 1am:
555	#   → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
556	#   → media_management.sh on HOST1's shares
557	#   → docker_daily_restart.sh on HOST1's containers
558	#
559	# HOST2 runs this script at 1am:
560	#   → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
561	#   → media_management.sh on HOST2's shares
562	#   → docker_daily_restart.sh on HOST2's containers
563	# ─────────────────────────────────────────────────────────────────────────────
564	```
565
566	---
567
568	### ── Configuration ────────────────────────────────────────────────────────────
569
570	```bash
571	# master.conf
572	# ─────────────────────────────────────────────────────────────────────────────
573	# Script lists — git_pull is split out as pre-sync; everything else is post-sync.
574	# The orchestrator recognises git_pull_execute.sh by name and routes it correctly.
575	#
576	DAILY_MAINTENANCE_SCRIPTS=(
577	    "git_pull_execute.sh"                          # PRE-SYNC — always first
578	    "Media/media_shares_permissions.sh"            # POST-SYNC — permissions before arr
579	    "Media/media_cleaner.sh anime"                 # POST-SYNC — junk before orphan scan
580	    "Media/media_cleaner.sh media"                 # POST-SYNC
581	    "Media/lidarr_cleanup.sh"                      # POST-SYNC — orphan cleanup last
582	    "Media/sonarr_cleanup.sh"                      # POST-SYNC
583	    "Media/radarr_cleanup.sh"                      # POST-SYNC
584	    "Docker_Essentials/docker_daily_restart.sh"    # POST-SYNC — restarts after everything
585	)
586
587	# host1.conf
588	HOST1_DAILY_SYNC_SHARES=(
589	    "/mnt/user/Movies"          # both servers — arr_sync union, rsync spreads files
590	    "/mnt/user/Tv_Shows"        # both servers
591	    "/mnt/user/Music"           # both servers
592	    "/mnt/user/Kids_Movies"
593	    "/mnt/user/Kids_Tv_Shows"
594	    "/mnt/user/Sports"
595	    "/mnt/user/stand-up_comedy"
596	)
597
598	HOST1_PERSONAL_SHARES=(
599	    "/mnt/user/Personal"        # encrypted personal share — appended after standard
600	)
601
602	# host2.conf
603	HOST2_DAILY_SYNC_SHARES=(
604	    "/mnt/user/Anime_Shows"     # both servers — arr_sync union, rsync spreads files
605	    "/mnt/user/Anime_Movies"    # both servers
606	)
607	```
608
609	---
610
611	### ── Adding or Removing a Job ────────────────────────────────────────────────
612
613	```bash
614	# master.conf
615	# ─────────────────────────────────────────────────────────────────────────────
616	# Add a new media script — just insert it in the right position:
617	DAILY_MAINTENANCE_SCRIPTS=(
618	    "git_pull_execute.sh"
619	    "Media/media_shares_permissions.sh"
620	    "Media/media_cleaner.sh anime"
621	    "Media/media_cleaner.sh media"
622	    "Media/my_new_script.sh"               # ← add here in the correct order
623	    "Media/lidarr_cleanup.sh"
624	    "Media/sonarr_cleanup.sh"
625	    "Media/radarr_cleanup.sh"
626	    "Docker_Essentials/docker_daily_restart.sh"
627	)
628
629	# Disable a job temporarily — comment it out, do not delete:
630	DAILY_MAINTENANCE_SCRIPTS=(
631	    "git_pull_execute.sh"
632	    "Media/media_shares_permissions.sh"
633	#   "Media/media_cleaner.sh anime"         # ← temporarily disabled
634	    "Media/media_cleaner.sh media"
635	    "Media/lidarr_cleanup.sh"
636	    ...
637	)
638	# ─────────────────────────────────────────────────────────────────────────────
639	# No changes to daily_sync_maintenance.sh needed in either case.
640	```
641
642	---
643
644	### ── Drive Temperature Exit Codes ───────────────────────────────────────────
645
646	```bash
647	# ─────────────────────────────────────────────────────────────────────────────
648	# rsync.sh returns specific exit codes for temperature issues.
649	# The orchestrator handles these correctly — one hot drive does not abort all others.
650	#
651	#   exit 0 → success — continue to next share
652	#   exit 1 → temp WARN — skip this share, continue to next share
653	#   exit 2 → temp CRITICAL — abort ALL remaining shares in this window
654	#             notify immediately with which share triggered the abort
655	#   exit N → other failure — skip this share, continue to next share
656	# ─────────────────────────────────────────────────────────────────────────────
657	```
658
659	---
660
661	### ── Relationship to Fallback Writeback ──────────────────────────────────────
662
663	```bash
664	# ─────────────────────────────────────────────────────────────────────────────
665	# The same HOST*_DAILY_SYNC_SHARES lists are used by fallback.sh for Tier 4
666	# writeback — but in the opposite direction.
667	#
668	# Normal (daily_sync_maintenance.sh):
669	#   HOST1 → pushes Movies, Tv_Shows → HOST2
670	#
671	# Tier 4 fallback writeback (HOST1 returns after 24hr+ outage):
672	#   HOST2 → pushes Movies, Tv_Shows → HOST1
673	#   (HOST2 was running HOST1's arrs and accumulated content)
674	#
675	# Same list, correct direction for the situation, zero duplication.
676	# No separate writeback list to maintain.
677	# ─────────────────────────────────────────────────────────────────────────────
678	```
679
680	---
681
682	### ── Usage ───────────────────────────────────────────────────────────────────
683
684	```bash
685	daily_sync_maintenance.sh              # normal run
686	daily_sync_maintenance.sh --dry-run    # preview all jobs without syncing or changing
687	daily_sync_maintenance.sh --log        # verbose per-share and per-job output
688	daily_sync_maintenance.sh --status     # show configured shares and jobs for this host
689	```
690
691	---
692
693	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
694	## 📅  weekly_sync_maintenance.sh
695	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
696
697	Weekly maintenance window orchestrator — clean sync of Emby and auth stack, container
698	image updates, and weekly docker restarts. Runs Sunday 2:30am; fits inside the Sunday
699	maintenance block before the 7am coffee report.
700
701	```bash
702	# Scheduled: 30 2 * * 0   (Sunday 2:30am)
703	```
704
705	---
706
707	### ── Execution Order ──────────────────────────────────────────────────────────
708
709	```bash
710	# ─────────────────────────────────────────────────────────────────────────────
711	# Containers stop BEFORE sync — clean static source, full bandwidth.
712	# Containers start AFTER sync — on fresh data, in dependency order.
713	# DDNS and fallback continue running throughout — only managed containers stop.
714	#
715	# 1. Pre-flight checks      — connectivity, remote Docker daemon, remote rootfs
716	# 2. Stop local containers  — Emby + auth stack stopped on this server
717	# 3. Stop remote containers — Emby + auth stack stopped on remote via SSH
718	# 4. Pull updates locally   — if WEEKLY_SYNC_UPDATES=true (containers already stopped)
719	# 5. Pull updates remotely  — if WEEKLY_SYNC_UPDATES_REMOTE=true
720	# 6. rsync WEEKLY_SYNC_SHARES — full clean mirror at full bandwidth
721	# 7. Start remote containers — new image, correct dependency order
722	# 8. Start local containers  — new image, correct dependency order
723	# Post-sync jobs: docker_weekly_restart.sh
724	# ─────────────────────────────────────────────────────────────────────────────
725	```
726
727	---
728
729	### ── Why Weekly Not Nightly for Emby ─────────────────────────────────────────
730
731	```bash
732	# ─────────────────────────────────────────────────────────────────────────────
733	# Two Emby syncs run in parallel — dirty and clean:
734	#
735	# emby-fallback dirty sync (every 30 minutes via critical_sync_maintenance.sh, Emby running):
736	#   watch states, library deltas, user activity — continuous coverage
737	#   WAL files excluded — safe to copy while Emby writes
738	#   HOST2 always within 30 minutes of HOST1 on playback state
739	#
740	# weekly clean sync (Sunday 2:30am, Emby stopped):
741	#   Full clean mirror — all databases checkpointed and flushed
742	#   Metadata, plugins, config all included
743	#   ~30 seconds of Emby downtime — both instances stopped during rsync only
744	#
745	# Why not nightly:
746	#   Emby builds a warm image thumbnail cache on HOST2 throughout the week.
747	#   Nightly sync resets this cache — cold loads every morning for users.
748	#   Weekly sync: cache stays warm for 6 days. Resets Sunday night while users sleep.
749	#   One weekly reset at an acceptable time is better than six unnecessary resets.
750	# ─────────────────────────────────────────────────────────────────────────────
751	```
752
753	---
754
755	### ── Container Update Window ─────────────────────────────────────────────────
756
757	```bash
758	# master.conf
759	# ─────────────────────────────────────────────────────────────────────────────
760	# Containers are already stopped for the sync — pulling updated images costs
761	# nothing extra in downtime. Both servers start on the same image version
762	# after the window completes.
763	#
764	    WEEKLY_SYNC_UPDATES=true          # pull updated images locally
765	    WEEKLY_SYNC_UPDATES_REMOTE=true   # pull updated images on remote via SSH
766	#
767	# Toggle false to skip updates without changing the schedule:
768	#   WEEKLY_SYNC_UPDATES=false         # skips pulls, containers still restart on current image
769	```
770
771	---
772
773	### ── Why Auth Stack Weekly Sync Matters ──────────────────────────────────────
774
775	```bash
776	# ─────────────────────────────────────────────────────────────────────────────
777	# NPM, Authelia, LLDAP, Mariadb run warm on both servers continuously.
778	# HOST1 is source of truth — changes propagate to HOST2 via weekly clean sync.
779	#
780	# What propagates automatically every Sunday:
781	#   New users added in LLDAP on HOST1       → appear on HOST2 by Monday
782	#   Proxy rules changed in NPM on HOST1     → live on HOST2 by Monday
783	#   Authelia policies updated on HOST1      → enforced on HOST2 by Monday
784	#   TLS certificates renewed on HOST1       → valid on HOST2 by Monday
785	#
786	# No manual sync needed for routine auth administration.
787	# Anything done on HOST1 is on HOST2 within a week.
788	# ─────────────────────────────────────────────────────────────────────────────
789	```
790
791	---
792
793	### ── Sunday Maintenance Window ───────────────────────────────────────────────
794
795	```bash
796	# ─────────────────────────────────────────────────────────────────────────────
797	# This script is part of a coordinated Sunday maintenance block:
798	#
799	# 2:30am  weekly_sync_maintenance.sh   ← clean sync + image updates (~5-10min)
800	# 2:50am  CA Auto Update plugin        ← unRAID plugin updates
801	# 2:55am  CA container updates         ← additional container updates
802	# 3:00am  Network reboot               ← router/switch restart
803	#
804	# Everything comes back clean:
805	#   Network fresh, Emby + auth updated, containers on latest images.
806	#   All in one window while users sleep.
807	#   Sunday morning coffee report at 7am shows the post-maintenance state.
808	# ─────────────────────────────────────────────────────────────────────────────
809	```
810
811	---
812
813	### ── Configuration ────────────────────────────────────────────────────────────
814
815	```bash
816	# master.conf
817	    WEEKLY_SYNC_SHARES=(
818	        "/mnt/user/Media_Server/Emby"                    # emby profile — full clean mirror
819	        "/mnt/user/appdata-Fallback/Critical-Data"       # critical-data — auth stack clean state
820	    )
821
822	    WEEKLY_MAINTENANCE_SCRIPTS=(
823	        "Docker_Essentials/docker_weekly_restart.sh"     # weekly restart of less-critical services
824	    )
825
826	    WEEKLY_SYNC_UPDATES=true
827	    WEEKLY_SYNC_UPDATES_REMOTE=true
828	```
829
830	---
831
832	### ── Usage ───────────────────────────────────────────────────────────────────
833
834	```bash
835	weekly_sync_maintenance.sh              # normal run
836	weekly_sync_maintenance.sh --dry-run    # preview — no stops, no syncs, no starts
837	weekly_sync_maintenance.sh --log        # verbose per-share per-job output
838	weekly_sync_maintenance.sh --status     # show configured shares, jobs, update toggles
839	```
840
841	---
842
843	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
844	## 📆  monthly_maintenance.sh
845	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
846
847	Uptime-triggered orchestrator for long-running system tasks — ZFS scrub, SMART
848	long tests — that should only run on stable systems that have been up for at least
849	30 days. Called daily by cron; most invocations are silent no-ops.
850
851	Two gates must both pass before any job runs:
852	1. Server uptime ≥ `MONTHLY_UPTIME_THRESHOLD_DAYS`
853	2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file on `/boot/config/` — survives reboots)
854
855	If either gate fails, the script exits 0 with no output. This is expected — it runs
856	daily and most days are no-ops.
857
858	`--force` bypasses both gates and runs the job list immediately. Use for testing or
859	when a scrub was missed and the server hasn't reached the uptime threshold yet.
860
861	### Configuration (master.conf)
862
863	```bash
864	MONTHLY_MAINTENANCE_SCRIPTS=(
865	    #"Tools/zfs_pool_scrub.sh"
866	    #"Tools/smart_long_test.sh"
867	)
868	MONTHLY_UPTIME_THRESHOLD_DAYS=30
869	MONTHLY_RUN_INTERVAL_DAYS=30
870	MONTHLY_LAST_RUN_FILE="/boot/config/monthly_maintenance_last_run.db"
871	```
872
873	Scripts are commented out by default — uncomment what applies to your hardware.
874
875	### Usage
876
877	```bash
878	monthly_maintenance.sh              # normal run (daily cron — silent no-op when gates not met)
879	monthly_maintenance.sh --force      # bypass both gates — run immediately
880	monthly_maintenance.sh --dry-run    # show what would run without running it
881	monthly_maintenance.sh --status     # show gate state: uptime, last run, next eligible
882	monthly_maintenance.sh --log        # verbose output from each child script
883	```
884
885	---
886
887	## ━━━ COMPLETE SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
888
889	```bash
890	# ─────────────────────────────────────────────────────────────────────────────
891	# Array start event (Varaverk disks_mounted hook):
892	# ─────────────────────────────────────────────────────────────────────────────
893	array_started.sh
894
895	# ─────────────────────────────────────────────────────────────────────────────
896	# Every 7 minutes:
897	# ─────────────────────────────────────────────────────────────────────────────
898	*/7 * * * *     transcode_management.sh
899
900	# ─────────────────────────────────────────────────────────────────────────────
901	# Every 15 minutes — watchdog cycle:
902	# ─────────────────────────────────────────────────────────────────────────────
903	*/15 * * * *    watchdog_orchestrator.sh
904
905	# ─────────────────────────────────────────────────────────────────────────────
906	# Every 30 minutes — auth stack + Emby dirty sync + partnership check:
907	# ─────────────────────────────────────────────────────────────────────────────
908	*/30 * * * *    critical_sync_maintenance.sh
909
910	# ─────────────────────────────────────────────────────────────────────────────
911	# Every 4 hours — arr library sync + failed import recovery + optional rsync:
912	# ─────────────────────────────────────────────────────────────────────────────
913	0 */4 * * *     intermediate_sync_maintenance.sh
914
915	# ─────────────────────────────────────────────────────────────────────────────
916	# Daily — 1am:
917	#   git pull → rsync all truth shares → permissions + cleaners + arr cleanup → docker restart
918	# ─────────────────────────────────────────────────────────────────────────────
919	0 1 * * *       daily_sync_maintenance.sh
920
921	# ─────────────────────────────────────────────────────────────────────────────
922	# Weekly — Sunday maintenance block:
923	# ─────────────────────────────────────────────────────────────────────────────
924	30 2 * * 0      weekly_sync_maintenance.sh    # clean sync + updates (~5-10min)
925	50 2 * * 0      CA Auto Update plugin         # plugin updates
926	55 2 * * 0      CA container updates          # container image updates
927	0  3 * * 0      Network reboot                # router/switch restart
928	0  7 * * 0      sunday_morning_coffee_report.sh
929
930	# ─────────────────────────────────────────────────────────────────────────────
931	# 15th of each month (uptime-gated — silent no-op if uptime < 30 days):
932	# ─────────────────────────────────────────────────────────────────────────────
933	0 0 15 * *      monthly_maintenance.sh
934	```
935
936	---
937
938	## ━━━ ADDING A NEW ORCHESTRATOR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
939
940	If you find yourself running 3+ related scripts on the same schedule, wrap them
941	in a new orchestrator. Model directly on `media_management.sh` which has the
942	complete pattern — dry-run passthrough, status display, pass/fail tracking, summary.
943
944	```bash
945	# Minimal skeleton — the full pattern in its simplest form:
946	#!/bin/bash
947	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
948	source "$SCRIPT_DIR/../load_config.sh"
949	parse_args "$@"
950
951	SCRIPTS_ROOT="$SCRIPT_DIR/.."
952	PASS=()
953	FAIL=()
954
955	# Read job list from master.conf — never hardcode jobs in the orchestrator
956	for script_entry in "${MY_MAINTENANCE_JOBS[@]:-}"; do
957	    [[ -z "$script_entry" ]] && continue
958
959	    read -r -a parts <<< "$script_entry"
960	    script_path="$SCRIPTS_ROOT/${parts[0]}"
961	    script_name=$(basename "${parts[0]}")
962	    extra_args=("${parts[@]:1}")
963
964	    [[ "$DRY_RUN" == true ]] && extra_args+=("--dry-run")
965
966	    if bash "$script_path" "${extra_args[@]}"; then
967	        PASS+=("$script_name")
968	    else
969	        FAIL+=("$script_name")
970	    fi
971	done
972
973	# One summary — one notification
974	echo "Passed: ${#PASS[@]}  Failed: ${#FAIL[@]}"
975	[[ ${#FAIL[@]} -gt 0 ]] && \
976	    notify "My maintenance failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
977	        "My Orchestrator" "warning"
978	```


Now I'll read the script headers to check for discrepancies.


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Conf Cache Sync ================================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Maintains a RAM-resident conf cache at /tmp/.vv/config/cached/.confs/.
9	# Credentials and partner keys live in RAM only — never on disk across hosts.
10	#
11	# On array start (default / --array-start):
12	#   1. Copy own conf to local cache
13	#   2. Pull each available partner's conf from their disk → local cache
14	#   3. Push own conf to each available partner's /tmp/.vv/ cache
15	#
16	# On conf save (--push-only):
17	#   Fast path — push updated own conf to all partners' /tmp/.vv/ cache only.
18	#   No pulls, no local cache rebuild.
19	#
20	# Cache is /tmp (tmpfs) — cleared every reboot, repopulated by this script
21	# on next array start. Scripts source from cache for partner vars; own vars
22	# always come from disk (load_config.sh skips cached copy of own conf).
23	#
24	# ==============================================================================================
25	# RUNTIME MODES
26	# ==============================================================================================
27	#
28	#   conf_sync.sh                   Full sync: pull from all partners + push to all partners
29	#   conf_sync.sh --push-only       Push own conf to all partners (fast, for conf-save hook)
30	#   conf_sync.sh --pull-only       Pull partner confs into local cache only (for intermediate orch)
31	#   conf_sync.sh --dry-run         Show what would happen, no changes
32	#   conf_sync.sh --log             Verbose output
33	#
34	# ==============================================================================================
35
36	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
37	source "$SCRIPT_DIR/../load_config.sh"
38
39	PUSH_ONLY=false
40	PULL_ONLY=false
41	FILTERED_ARGS=()
42	for arg in "$@"; do
43	    case "$arg" in
44	        --push-only) PUSH_ONLY=true ;;
45	        --pull-only) PULL_ONLY=true ;;
46	        *)           FILTERED_ARGS+=("$arg") ;;
47	    esac
48	done
49	parse_args "${FILTERED_ARGS[@]}"
50	detect_hosts
51
52	if [[ "${CONF_SYNC_ENABLED:-true}" == false ]]; then
53	    log "CONF_SYNC_ENABLED=false — skipping"
54	    exit 0
55	fi
56
57	CACHE_DIR="/tmp/.vv/config/cached/.confs"
58	MY_CONF="$SCRIPTS_ROOT/Configurations/${MY_ID,,}.conf"
59	SSH_TIMEOUT=10
60
61	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
62
63	# ── Ensure cache dir exists ───────────────────────────────────────────────────
64	if [[ "$DRY_RUN" == false ]]; then
65	    mkdir -p "$CACHE_DIR"
66	fi
67
68	# ── Copy own conf into local cache ───────────────────────────────────────────
69	if [[ "$PUSH_ONLY" == false ]] && [[ "$PULL_ONLY" == false ]]; then
70	    if [[ -f "$MY_CONF" ]]; then
71	        if [[ "$DRY_RUN" == true ]]; then
72	            warn "DRY RUN — would copy $(basename "$MY_CONF") → $CACHE_DIR/"
73	        else
74	            cp "$MY_CONF" "$CACHE_DIR/${MY_ID,,}.conf" && \
75	                log "Own conf cached ✅" || warn "Failed to cache own conf"
76	        fi
77	    else
78	        warn "Own conf not found: $MY_CONF"
79	    fi
80	fi
81
82	# ── Per-partner sync ──────────────────────────────────────────────────────────
83	PUSHED=0
84	PULLED=0
85	FAILED=0
86
87	for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
88	    partner_host="${!host_var}"
89	    [[ -z "$partner_host" ]] && continue
90	    [[ "${host_var,,}" == "${MY_ID,,}" ]] && continue
91
92	    partner_slot="${host_var,,}"   # e.g. host2
93	    partner_ip=$(resolve_tailscale_ip "$partner_host" 2>/dev/null || true)
94
95	    if [[ -z "$partner_ip" ]]; then
96	        warn "$partner_host — cannot resolve Tailscale IP, skipping"
97	        (( FAILED++ ))
98	        continue
99	    fi
100
101	    # ── Pull: grab partner's conf from their disk → our local cache ──────────
102	    if [[ "$PUSH_ONLY" == false ]]; then
103	        remote_conf="${SCRIPTS_DIR}/Configurations/${partner_slot}.conf"
104	        if [[ "$DRY_RUN" == true ]]; then
105	            warn "DRY RUN — would pull $partner_host:$remote_conf → $CACHE_DIR/${partner_slot}.conf"
106	        elif timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
107	            -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
108	            "root@${partner_ip}:${remote_conf}" \
109	            "$CACHE_DIR/${partner_slot}.conf" 2>/dev/null; then
110	            log "Pulled ${partner_slot}.conf from $partner_host ✅"
111	            (( PULLED++ ))
112	        else
113	            warn "Could not pull ${partner_slot}.conf from $partner_host"
114	            (( FAILED++ ))
115	        fi
116	    fi
117
118	    # ── Push: send own conf to partner's /tmp/.vv/ cache ────────────────────
119	    if [[ "$PULL_ONLY" == true ]]; then
120	        continue


Error: File does not exist. Note: your current working directory is /boot/config/plugins/varaverk.


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Array Start Orchestrator ===================================
4	# ==============================================================================================
5	# Single entry point for array start — fired by the Varaverk plugin's
6	# disks_mounted event hook (Plugin/unraid/event/disks_mounted/array_start_jobs).
7	# Launches everything configured in ARRAY_START_SCRIPTS in master.conf.
8	# This script exits after launching all scripts — the event hook sees it complete normally.
9	#
10	# ── WHAT IT LAUNCHES ──────────────────────────────────────────────────────────────────────────
11	# Configured in master.conf ARRAY_START_SCRIPTS — no changes to this script ever needed.
12	# Current order (order matters — see below):
13	#
14	#   ONE-SHOT (run and exit naturally):
15	#     System_Essentials/unraid_api_key_renew.sh — re-register Varaverk API key at boot
16	#     System_Essentials/inotify_tuning.sh       — raise inotify limits before containers start
17	#     System_Essentials/docker_syslog_filter.sh — suppress veth log noise before logs fill
18	#     System_Essentials/php_fpm_max_children.sh — WebGUI performance tuning
19	#     Transcodes/ramdisk_setup.sh               — create tmpfs + symlink before Emby starts
20	#     Docker_Essentials/docker_network_connect.sh — ensure networks + container connections
21	#
22	#   CONTINUOUS (run until array stops):
23	#     Fallback/fallback.sh                      — mutual fallback monitor
24	#
25	#   NOTE: watchdogs (docker, system, stability) are NOT launched here.
26	#   They run via watchdog_orchestrator.sh every 15 min (cron), not as daemons.
27	#
28	# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
29	#   unraid_api_key_renew.sh — before anything else — self-heals API registry at boot
30	#   inotify_tuning.sh     — must run BEFORE Code-Server and other containers start
31	#                           containers that start with low inotify limits keep them ✅
32	#   docker_syslog_filter  — must run BEFORE any container starts creating veth interfaces
33	#   ramdisk_setup.sh      — must run BEFORE Emby starts transcoding
34	#   docker_network_connect — must run BEFORE watchdogs check container states
35	#   fallback.sh           — last — needs everything else stable to make decisions
36	#
37	# ── ONE-SHOT vs CONTINUOUS DETECTION ─────────────────────────────────────────────────────────
38	# Script is launched in background with bash script.sh &
39	# After 1 second: if PID still alive → continuous (running in background)
40	#                  if PID dead + exit 0 → one-shot completed successfully
41	#                  if PID dead + exit N → failure
42	#
43	# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
44	#   Root check           — all launched scripts require root
45	#   acquire_lock         — prevents duplicate array start launches
46	#   detect_hosts()       — MY_ID in notifications
47	#   platform_require_cmd  — notify validated before use
48	#   chmod +x auto-fix    — non-executable scripts fixed before launch
49	#   Full path on failure — shows exact path for debugging
50	#   notify on failures   — alert if any script fails to launch
51	#
52	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
53	#   ARRAY_START_SCRIPTS — ordered list of scripts to launch at array start
54	#
55	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
56	#   array_started.sh              — normal launch (called by Varaverk disks_mounted event hook)
57	#   array_started.sh --dry-run    — show what would be launched without launching
58	#   array_started.sh --status     — show configured scripts and their current state
59	#   array_started.sh --log        — verbose output per script
60	# ==============================================================================================
61
62	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
63	ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
64
65	source "$ECOSYSTEM_ROOT/load_config.sh"
66
67	parse_args "$@"
68
69	# ==============================================================================================
70	# ━━━ Setup ━━━
71	# ==============================================================================================
72	if [[ "$EUID" -ne 0 ]]; then
73	    error "Must be run as root"
74	    exit 1
75	fi
76
77
78	acquire_lock
79
80	detect_hosts
81
82	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — scripts will not be launched"
83
84	# ==============================================================================================
85	# ━━━ Status ━━━
86	# ==============================================================================================
87	if [[ "$SHOW_STATUS" == true ]]; then
88	    echo ""
89	    echo "━━━━━ $ICON_SUMMARY ARRAY START STATUS ━━━━━"
90	    echo "$ICON_HOST  Identity: $MY_ID ($LOCAL_SERVER_NAME)"
91	    echo "$ICON_GEAR  Scripts:  ${#ARRAY_START_SCRIPTS[@]} configured"
92	    echo ""
93
94	    for relative_path in "${ARRAY_START_SCRIPTS[@]}"; do
95	        [[ -z "$relative_path" ]] && continue
96	        script_path="$ECOSYSTEM_ROOT/$relative_path"
97	        script_name=$(basename "$script_path")
98
99	        if [[ ! -f "$script_path" ]]; then
100	            echo "  $ICON_ERROR $script_name — FILE NOT FOUND"
101	            echo "    $script_path"
102	            continue
103	        fi
104
105	        [[ ! -x "$script_path" ]] && flag=" (not executable — will auto-fix)" || flag=""
106
107	        # Check if currently running
108	        if pgrep -f "$script_path" >/dev/null 2>&1; then
109	            RUN_PID=$(pgrep -f "$script_path" | head -1)
110	            echo "  $ICON_RUNNING $script_name — RUNNING (PID $RUN_PID)${flag}"
111	        else
112	            echo "  $ICON_NOT_RUNNING $script_name — not running${flag}"
113	        fi
114	    done
115	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
116	    exit 0
117	fi
118
119	# ==============================================================================================
120	# ━━━ Launch Scripts ━━━


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================ Watchdog Orchestrator ===========================================
4	# ==============================================================================================
5	# Runs WATCHDOG_ORCHESTRATOR_SCRIPTS in order each cron cycle.
6	# Schedule: */15 * * * * (every 15 minutes)
7	#
8	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
9	#   Driven by WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf — add, remove, or reorder there.
10	#   Default: resource_watchdog → docker_watchdog → system_watchdog → unraid_api_key_renew → stability_watchdog
11	#
12	# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
13	#   Resource Watchdog first — frees RAM and CPU before healing attempts container restarts.
14	#                            Containers restarted into a resource-pressured system just fail again.
15	#   Docker Watchdog second — restarts with pressure already reduced, more likely to stabilise.
16	#   System Watchdog third  — system component health after containers are healed.
17	#   API key renew fourth   — self-heals unraid-api registry loss; check-first, silent when valid.
18	#   Stability Watchdog last — only reboots when all prior layers could not resolve the issue.
19	#
20	# ── ARRAY CHECK ───────────────────────────────────────────────────────────────────────────────
21	#   Exits immediately if /mnt/user is not mounted as shfs (array not started).
22	#   Watchdogs check Docker containers and storage — meaningless without the array.
23	#   Prevents false positives and unnecessary reboots when array is stopped or stopping.
24	#
25	# ── STARTUP GRACE ─────────────────────────────────────────────────────────────────────────────
26	#   No action until system uptime >= WATCHDOG_STARTUP_GRACE seconds.
27	#   Prevents false positives from containers still starting at array launch.
28	#   Each sub-script enforces this independently — orchestrator exits early to avoid log noise.
29	#
30	# ── OVERLAP PROTECTION ────────────────────────────────────────────────────────────────────────
31	#   acquire_lock() — exits immediately if a prior cycle is still in progress.
32	#   Prevents pile-up when a cycle runs long (daemon restart attempt = 30s, etc.).
33	#
34	# ── REPLACES ──────────────────────────────────────────────────────────────────────────────────
35	#   Continuous loops previously in system_watchdog.sh and docker_watchdog.sh.
36	#   Those scripts are now single-pass — this orchestrator provides the cadence.
37	#
38	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
39	#   WATCHDOG_ORCHESTRATOR_SCRIPTS         — watchdogs to run, in order
40	#   WATCHDOG_STARTUP_GRACE                — seconds after boot before checks activate
41	#   WATCHDOG_ORCHESTRATOR_HEARTBEAT       — periodic heartbeat log toggle
42	#   WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS — heartbeat interval in hours
43	#
44	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
45	#   watchdog_orchestrator.sh           — normal run (called by cron every 15 minutes)
46	#   watchdog_orchestrator.sh --dry-run — pass --dry-run to all sub-scripts
47	#   watchdog_orchestrator.sh --status  — show script paths and current grace state
48	#   watchdog_orchestrator.sh --log     — verbose output from all sub-scripts
49	# ==============================================================================================
50
51	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
52	ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
53
54	source "$ECOSYSTEM_ROOT/load_config.sh"
55
56	parse_args "$@"
57
58	# ==============================================================================================
59	# ━━━ Setup ━━━
60	# ==============================================================================================
61	if [[ "$EUID" -ne 0 ]]; then
62	    error "Must be run as root"
63	    exit 1
64	fi
65
66	# Skip immediately if another cycle is still running — no pile-up
67	acquire_lock
68
69	detect_hosts
70
71	log "$ICON_GEAR Config: grace=${WATCHDOG_STARTUP_GRACE}s heartbeat=${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true}/${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}hr scripts=${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"
72	log "$ICON_WATCHDOG Order: $(for s in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do printf '%s ' "${s##*/}"; done)"
73
74	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing --dry-run to all sub-scripts"
75
76	# Derive a display name from a script path: "resource_watchdog.sh" → "Resource Watchdog"
77	_watchdog_display_name() {
78	    local path="$1"
79	    local base="${path##*/}"
80	    base="${base%.sh}"
81	    base="${base//_/ }"
82	    echo "$base" | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) substr($i,2); print}'
83	}
84
85	# ==============================================================================================
86	# ━━━ Status ━━━
87	# ==============================================================================================
88	if [[ "$SHOW_STATUS" == true ]]; then
89	    echo ""
90	    echo "━━━━━ $ICON_SUMMARY WATCHDOG ORCHESTRATOR STATUS ━━━━━"
91	    echo "$ICON_HOST  Identity: $MY_ID ($LOCAL_SERVER_NAME)"
92	    echo ""
93
94	    UPTIME_S=$(awk '{print int($1)}' /proc/uptime)
95	    if [[ "$UPTIME_S" -lt "$WATCHDOG_STARTUP_GRACE" ]]; then
96	        warn "Within startup grace — $(format_duration $UPTIME_S) / $(format_duration $WATCHDOG_STARTUP_GRACE)"
97	    else
98	        echo "Past startup grace — $(format_duration $UPTIME_S) uptime"
99	    fi
100
101	    echo ""
102	    echo "── Sub-scripts ──"
103	    for entry in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do
104	        local_path="$ECOSYSTEM_ROOT/$entry"
105	        label="$(_watchdog_display_name "$entry")"
106	        if [[ -f "$local_path" ]]; then
107	            [[ -x "$local_path" ]] && icon="$ICON_DONE" || icon="$ICON_WARN"
108	            echo "  $icon $label — ${local_path##*/}"
109	        else
110	            echo "  $ICON_ERROR $label — NOT FOUND: $local_path"
111	        fi
112	    done
113
114	    echo ""
115	    echo "  Schedule:   */15 * * * * (every 15 minutes)"
116	    echo "  Heartbeat:  ${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true} / every ${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}hr"
117	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
118	    exit 0
119	fi
120


Let me continue reading the remaining script headers: