1	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2	# 📋  USER SCRIPTS PLUGIN — COMPLETE SETUP REFERENCE
3	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4
5	> **Fully superseded.** The Varaverk plugin handles all scheduling natively — no User
6	> Scripts entries are needed. Array start and stop are triggered by the plugin's own
7	> `disks_mounted` / `disks_unmounting` event hooks. Cron is managed via `schedule.json`
8	> and rebuilt on every boot. User Scripts is not required.
9	> See `Orchestrators/README-Orchestrators.md` for the current complete schedule.
10
11	**Historical reference** — documents what each script does and how it was previously
12	configured via User Scripts. Still useful as a script catalogue. Paths updated to
13	reflect current boot-drive installation location.
14
15	---
16
17	## ━━━ REQUIRED PLUGINS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
18
19	Install both from Community Applications before creating any entries:
20
21	```
22	User Scripts          — Andrew Zawadzki
23	                        Core plugin. Runs scripts via unRAID UI without terminal.
24	                        Required — nothing works without this.
25
26	User Scripts Enhanced — Kurotaku
27	                        Adds categories, drag-to-reorder, UI improvements.
28	                        Required for the category organisation used here.
29	```
30
31	---
32
33	## ━━━ GLOBAL RULES — READ BEFORE CREATING ENTRIES ━━━━━━━━━━━━━━━━━━━━━━━━━━━
34
35	```
36	1. Every script entry → Run in Background: YES
37	   Without background mode the plugin UI freezes waiting for the script to complete.
38	   Long-running scripts and continuous loops (fallback, watchdogs) require background.
39	   There are no exceptions — set background on everything.
40
41	2. Run as root: YES
42	   All scripts require root. Docker operations, rsync, kernel tuning, reboots.
43
44	3. Category names match repo folder names exactly
45	   When the plugin category and the git folder are identical, you never wonder
46	   which folder a script lives in. No mental translation.
47
48	4. Script name matches the script filename (without .sh)
49	   daily_sync_maintenance.sh → name: daily_sync_maintenance
50	   When logs reference a script name, you find the plugin entry immediately.
51
52	5. --dry-run exists on every script
53	   Test anything unfamiliar with --dry-run first. Pre-flight checks still run.
54	   It shows what would happen without making changes.
55
56	6. Most scripts never need to be run directly
57	   The orchestrators handle timing and order. Individual script entries exist
58	   for manual testing and one-off maintenance — not as replacement cron jobs.
59	```
60
61	---
62
63	## ━━━ CATEGORY STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
64
65	Create these categories in User Scripts Enhanced before adding any entries:
66
67	```
68	Orchestrators      — scheduled maintenance windows and startup launcher
69	Fallback           — mutual fallback state machine and test tools
70	Monitors           — weekly reports, health checks, trend tracking
71	Rsync              — manual per-profile sync triggers
72	Docker_Essentials  — container management tools (manual use)
73	Media              — library maintenance tools (manual use)
74	Transcodes         — ramdisk and transcode tools (manual use)
75	System_Essentials  — server management tools (manual use)
76	Partnership        — two-server relationship tools (manual use)
77	Tools              — repair, recovery, and one-time utilities
78	Git                — script deployment
79	```
80
81	---
82
83	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
84	## 🎯  ORCHESTRATORS
85	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
86
87	Orchestrators are the only scripts that run on a schedule. They coordinate multiple
88	child scripts in the correct order. Adding or removing a job means editing master.conf
89	— never the orchestrator script itself.
90
91	---
92
93	### ── array_start ─────────────────────────────────────────────────────────────
94
95	```
96	Category:   Orchestrators
97	Name:       array_start
98	Schedule:   At Startup of Array
99	Background: YES — required (starts continuous background processes)
100	```
101
102	```bash
103	#!/bin/bash
104	/boot/config/plugins/varaverk/Orchestrators/array_started.sh
105	```
106
107	**What it does:**
108	The single "At Startup of Array" entry for the entire ecosystem. Launches every
109	startup script in the order defined by ARRAY_START_SCRIPTS in master.conf. You
110	never add other startup scripts directly to User Scripts — add them to master.conf.
111
112	**What it starts (in order):**
113	```
114	inotify_tuning.sh        — raises kernel inotify limits before containers start
115	                           containers inherit limits at startup — must be first
116	docker_syslog_filter.sh  — suppresses Docker veth noise before containers create interfaces
117	php_fpm_max_children.sh  — sets PHP-FPM workers to 250 before WebGUI serves requests
118	ramdisk_setup.sh         — creates /mnt/ramdisk_transcodes and symlink before Emby starts
119	docker_network_connect.sh— ensures custom networks exist + containers connected
120	stability_watchdog.sh   ← last-resort — three-tier server stability watchdog
121	docker_watchdog.sh       ← continuous — two-tier container healing watchdog
122	fallback.sh              ← continuous — mutual fallback state machine
123	```
124
125	**After running:**
126	One-shot scripts exit cleanly. Three continuous processes run in the background until
127	the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|fallback"`
128
129	---
130
131	### ── transcode_management ─────────────────────────────────────────────────────
132
133	```
134	Category:   Orchestrators
135	Name:       transcode_management
136	Schedule:   */7 * * * *   (every 7 minutes)
137	Background: YES
138	```
139
140	```bash
141	#!/bin/bash
142	/boot/config/plugins/varaverk/Orchestrators/transcode_management.sh
143	```
144
145	**What it does:**
146	Runs transcode_cleanup.sh then transcode_manager.sh — in that exact order every
147	7 minutes. Order is non-negotiable: cleanup removes stale files first so the manager
148	sees real active session usage, not inflated numbers from ended sessions.
149
150	**transcode_cleanup.sh:**
151	Removes HLS segment files older than TRANSCODE_MAX_AGE minutes that are not currently
152	open by any process. Uses one lsof call per location — not one per file. Covers both
153	ramdisk and SSD fallback. Never touches the transcoding-temp directory.
154
155	**transcode_manager.sh:**
156	Checks ramdisk usage against RAMDISK_WARN_GB. Flips the /mnt/ram-transcode symlink
157	between ramdisk and SSD fallback based on current usage. Shows active Emby sessions
158	with their play method. Writes one daily stats entry to TRANSCODE_DAILY_LOG.
159	Runs safety checks every cycle: missing symlink → recreate, ramdisk disappeared →
160	auto-flip to SSD, missing transcoding-temp → recreate immediately.
161
162	**Output when healthy:** session list only — no flip messages = ramdisk managing fine
163
164	> ⚠ The Emby/Jellyfin Docker container must have
165	> `--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode`
166	> in Extra Parameters. Do **not** add `bind-propagation=shared` — runc v1.3.5+
167	> (Docker 29.x, Unraid 7.3+) rejects it with a mountinfo error. The ramdisk tmpfs
168	> is already MS\_SHARED at the kernel level — propagation is inherited automatically.
169
170	---
171
172	### ── critical_sync_maintenance ───────────────────────────────────────────────
173
174	```
175	Category:   Orchestrators
176	Name:       critical_sync_maintenance
177	Schedule:   */30 * * * *  (every 30 minutes)
178	Background: YES
179	```
180
181	```bash
182	#!/bin/bash
183	/boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh
184	```
185
186	**What it does:**
187	Keeps the auth stack and Emby watch state current on HOST2 — 30 minutes is the
188	maximum staleness for fallback-critical data.
189
190	**Jobs in order:**
191	```
192	rsync Critical-Data     — pushes auth stack to HOST2 (NPM rules, certs, LLDAP users,
193	                           Authelia config). HOST2's auth is always within 30min of HOST1.
194	rsync Emby dirty sync   — pushes watch states, user activity, library delta to HOST2
195	                           with Emby running. WAL files excluded — safe mid-write.
196	                           HOST2 Emby restarts after sync to pick up config changes.
197	partnership --check     — reads remote state file, increments offline counter if HOST2
198	                           unreachable, triggers auto-offboard at PARTNERSHIP_OFFLINE_THRESHOLD
199	```
200
201	**Silent when healthy:** no output on clean cycles — runs 48 times per day.
202
203	---
204
205	### ── arrs_failed_stalled_recovery ───────────────────────────────────────────
206
207	```
208	Category:   Orchestrators
209	Name:       arrs_failed_stalled_recovery
210	Schedule:   0 */6 * * *   (every 6 hours)
211	Background: YES
212	```
213
214	```bash
215	#!/bin/bash
216	/boot/config/plugins/varaverk/Orchestrators/arrs_failed_stalled_recovery.sh
217	```
218
219	**What it does:**
220	Automatically recovers failed imports and stalled downloads across Sonarr, Radarr,
221	and Lidarr. A release that downloaded but failed to import sits in the queue
222	permanently without this — no arr retries importFailed items on its own.
223
224	**What it finds and acts on:**
225	```
226	importFailed   — downloaded but couldn't import → blocklist + remove + new search
227	importPending  — stuck in import queue → blocklist + remove + new search
228	error          — serious failure state → blocklist + remove + new search
229	stalled        — no seeders or no progress → blocklist + remove + new search
230	```
231
232	**What it never touches:** items with status "downloading" or "imported".
233	Age gate: items newer than ARR_IMPORT_RECOVERY_AGE (default: 6 hours) are skipped —
234	the arr may still retry on its own.
235
236	**Result:** failed imports are automatically replaced while you sleep.
237
238	---
239
240	### ── daily_sync_maintenance ──────────────────────────────────────────────────
241
242	```
243	Category:   Orchestrators
244	Name:       daily_sync_maintenance
245	Schedule:   0 1 * * *    (1am daily)
246	Background: YES
247	```
248
249	```bash
250	#!/bin/bash
251	/boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh
252	```
253
254	**What it does:**
255	The full 1am daily maintenance window — everything that needs to happen overnight
256	in one orchestrated sequence.
257
258	**Execution order:**
259	```
260	1. git_pull_execute.sh        — pull latest scripts FIRST, everything below runs on
261	                                 current code
262
263	2. rsync DAILY_SYNC_SHARES    — push this server's truth shares to the other server
264	   HOST1 pushes:                Movies, Tv_Shows, Music, Kids content, Sports...
265	   HOST2 pushes:                Anime_Shows, Anime_Movies
266	   Personal encrypted shares    appended after — encrypted blocks, remote can't read
267
268	   Drive temp exit codes:
269	     exit 1 (WARN)  → skip this share, continue to next
270	     exit 2 (CRIT)  → abort ALL remaining syncs, notify immediately
271
272	3. media_shares_permissions.sh — apply nobody:users 755/664 to all media shares
273	                                   arr cleanup depends on correct ownership
274
275	4. media_cleaner.sh anime      — remove .sfv .nfo .rar .sample and other scene junk
276	   media_cleaner.sh media        from anime and media share folders
277	                                   junk removal BEFORE orphan detection — cleaner results
278
279	5. lidarr_cleanup.sh           — identify and delete orphaned music files not tracked
280	   sonarr_cleanup.sh             by the arr API. 7 safety checks must all pass before
281	   radarr_cleanup.sh             a single file is deleted. notify_emby_scan() called
282	                                   after any deletion — Emby library cleaned immediately
283
284	6. docker_daily_restart.sh     — restart connection-heavy containers that degrade over time
285	                                   NginxProxyManager, Authelia, Dispatcharr, ErsatzTV-Emby
286	                                   Running → docker restart. Stopped → left stopped.
287	```
288
289	**Notify:** silent on success. One notification on failure with count of failed jobs.
290	**Both servers run this independently at 1am** — same script, correct direction via detect_hosts().
291
292	---
293
294	### ── rsync_emby_fallback ─────────────────────────────────────────────────────
295
296	> **Superseded by `critical_sync_maintenance`.** Delete this User Script from the
297	> plugin. The emby-fallback dirty sync now runs as step 2 inside
298	> `critical_sync_maintenance.sh` every 30 minutes alongside the auth stack sync
299	> and partnership check. Running both produces duplicate syncs.
300
301	The Emby dirty sync (`/mnt/user/Media_Server/Emby --profile=emby-fallback`) is
302	configured in `HOST1_CRITICAL_SYNC_SHARES` in host1.conf. It runs automatically
303	when critical_sync_maintenance.sh runs.
304
305	---
306
307	### ── weekly_sync_maintenance ─────────────────────────────────────────────────
308
309	```
310	Category:   Orchestrators
311	Name:       weekly_sync_maintenance
312	Schedule:   30 2 * * 0   (Sunday 2:30am)
313	Background: YES
314	```
315
316	```bash
317	#!/bin/bash
318	/boot/config/plugins/varaverk/Orchestrators/weekly_sync_maintenance.sh
319	```
320
321	**What it does:**
322	The full Sunday maintenance window — clean sync with containers stopped, image
323	updates at zero extra downtime, weekly container restarts.
324
325	**Execution order:**
326	```
327	1. Pre-flight            — connectivity, remote Docker daemon, remote rootfs
328	2. Stop local containers — Emby + auth stack stopped on this server
329	3. Stop remote containers— Emby + auth stack stopped on remote via SSH
330	4. Pull images locally   — if WEEKLY_SYNC_UPDATES=true (containers already stopped,
331	                            free downtime, both servers start on identical image versions)
332	5. Pull images remotely  — if WEEKLY_SYNC_UPDATES_REMOTE=true (SSH to remote)
333	6. rsync WEEKLY_SYNC_SHARES:
334	   /mnt/user/Media_Server/Emby           — full clean mirror, databases checkpointed
335	   /mnt/user/appdata-Fallback/Critical-Data — auth stack clean state
336	7. Start remote containers— dependency order, new image, settle + verify each
337	8. Start local containers — same
338	9. docker_weekly_restart.sh — restarts less-critical services that benefit from
339	                               weekly clean start: NextCloud, AdGuard, Immich
340	```
341
342	**Why containers stop for Emby sync:**
343	Weekly sync is a full clean mirror including databases. Emby must be stopped to
344	checkpoint WAL — ensures a consistent database copy, not a snapshot mid-write.
345	~30 seconds of downtime. The 30-minute dirty sync covers users until this clean sync.
346
347	**Why weekly and not nightly:**
348	Emby builds a warm image thumbnail cache on HOST2 throughout the week. Nightly sync
349	resets this cache — cold loads every morning. Weekly sync: cache stays warm 6 days,
350	resets Sunday night while users sleep.
351
352	---
353
354	### ── sunday_morning_coffee_report ────────────────────────────────────────────
355
356	```
357	Category:   Orchestrators
358	Name:       sunday_morning_coffee_report
359	Schedule:   0 7 * * 0    (Sunday 7am)
360	Background: YES
361	```
362
363	```bash
364	#!/bin/bash
365	/boot/config/plugins/varaverk/Orchestrators/sunday_morning_coffee_report.sh
366	```
367
368	**What it does:**
369	Full weekly system overview — everything that happened this week in one clean read.
370	Designed to be read over coffee after the 2:30am maintenance window has completed
371	and run all the Sunday morning monitor scripts.
372
373	**Sections:**
374	```
375	🖥  System         — uptime, memory, boot drive, cache usage, reboots this week
376	📀  Array          — disk count, parity status, ZFS health, drive temperatures
377	🎬  Transcodes     — weekly ramdisk peak, flip count, session split (ramdisk vs SSD)
378	🎵  Media Activity — arr cleanup stats (orphans removed, space freed), queue depth
379	🌐  Rsync          — weekly transfer totals, per-share breakdown, failures
380	🛡  Watchdog       — system + docker watchdog state, active strikes, skip list
381	🔐  Security       — SSL cert expiry days remaining per domain (live TLS check)
382	📊  Emby           — version, active streams now, weekly play count, top users
383	⚙  System Health  — inotify peak/avg/warnings, php-fpm peak/avg, SMART, git sync
384	⚠  Issues         — anything requiring attention from any section above
385	```
386
387	**Delivery:** unRAID notification + Discord webhook (if configured).
388	**Silent when healthy:** footer shows ✅ if no issues, ⚠ with count if something needs attention.
389
390	---
391
392	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
393	## 📊  MONITORS
394	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
395
396	Monitor scripts observe and report — they never act. Read-only except for two bounded
397	log writers. All are safe to run manually at any time.
398
399	---
400
401	### ── weekly_health_digest ────────────────────────────────────────────────────
402
403	```
404	Category:   Monitors
405	Name:       weekly_health_digest
406	Schedule:   0 8 * * *    (8am daily — DIGEST_PROFILE in master.conf controls notify behaviour)
407	Background: YES
408	```
409
410	```bash
411	#!/bin/bash
412	/boot/config/plugins/varaverk/Monitors/weekly_health_digest.sh
413	```
414
415	**What it does:**
416	Reads state files from across the ecosystem — watchdog strikes, fallback state, skip
417	list, bandwidth history, transcode stats, cert expiry — and compiles them into a
418	digest notification. Reads only, writes nothing.
419
420	**DIGEST_PROFILE controls when it notifies (set in master.conf):**
421	```
422	always  — sends every run regardless of content (daily digest, even when healthy)
423	smart   — sends only when something worth reporting is found
424	          triggers: watchdog strikes, non-NORMAL fallback, cert warnings, bandwidth spikes
425	weekly  — sends once per week on DIGEST_DAY (Sunday), silent all other days
426	```
427
428	Schedule never changes (8am daily). Switching between profiles requires only one
429	line change in master.conf — no cron edit needed.
430
431	---
432
433	### ── system_tuning_monitor ───────────────────────────────────────────────────
434
435	```
436	Category:   Monitors
437	Name:       system_tuning_monitor
438	Schedule:   0 */6 * * *  (every 6 hours)
439	Background: YES
440	```
441
442	```bash
443	#!/bin/bash
444	/boot/config/plugins/varaverk/Monitors/system_tuning_monitor.sh
445	```
446
447	**What it does:**
448	Snapshots inotify and PHP-FPM utilisation every 6 hours. Writes one bounded log
449	entry. The sunday_morning_coffee_report reads this log for peak, average, and
450	warning counts over the week.
451
452	**inotify monitoring:**
453	Current instances in use vs kernel limit. Warns above INOTIFY_WARN_PCT (80%).
454	Shows top 5 consumers. inotify exhaustion = downloads complete but arrs don't detect
455	them, Live TV stutters, library updates stop — no obvious error anywhere.
456
457	**PHP-FPM monitoring:**
458	Active workers vs PHP_MAX_CHILDREN. Warns above PHP_FPM_WARN_PCT (80%).
459	Saturation = WebGUI slows, settings saves hang, container starts from UI timeout.
460
461	**Log:** TUNING_MONITOR_LOG — bounded, trimmed to TUNING_LOG_RETENTION days on write.
462
463	---
464
465	### ── zfs_memory_snapshot ─────────────────────────────────────────────────────
466
467	```
468	Category:   Monitors
469	Name:       zfs_memory_snapshot
470	Schedule:   0 6 * * 0    (Sunday 6am — first Sunday script, before everything else)
471	Background: YES
472	```
473
474	```bash
475	#!/bin/bash
476	/boot/config/plugins/varaverk/Monitors/zfs_memory_snapshot.sh
477	```
478
479	**What it does:**
480	Weekly ZFS pool health and memory diagnostic. Runs before other Sunday monitors so
481	pool state is captured while the system is post-maintenance and settled.
482
483	**Reports:**
484	ZFS pool status (ONLINE/DEGRADED/FAULTED) per pool. ARC size vs max, hit rate, metadata
485	ratio. Total and available RAM. Top 10 Docker containers by memory. Kernel pressure
486	via vmstat. Warns if ARC > ZFS_REPORT_ARC_WARN_PCT or RAM < ZFS_REPORT_FREE_WARN_GB.
487	Output goes to both console and /var/log/zfs-weekly-health.log for week-over-week comparison.
488
489	---
490
491	### ── smart_health ────────────────────────────────────────────────────────────
492
493	```
494	Category:   Monitors
495	Name:       smart_health
496	Schedule:   0 7 * * 0    (Sunday 7am)
497	Background: YES
498	```
499
500	```bash
501	#!/bin/bash
502	/boot/config/plugins/varaverk/Monitors/smart_health.sh
503	```
504
505	**What it does:**
506	Queries SMART attributes for every /dev/sd* and /dev/nvme* drive weekly. Drives
507	in HOST*_SMART_IGNORE_DRIVES (typically sda — the boot USB) are skipped.
508
509	**What triggers a notification:**
510	```
511	Reallocated_Sector_Ct > 0     — drive found and swapped a bad sector — warning sign
512	Current_Pending_Sector > 0    — drive suspects sectors bad, not yet confirmed
513	Offline_Uncorrectable > 0     — sectors that couldn't be corrected — data loss risk
514	Temperature > thresholds       — reads from dynamix.cfg (same thresholds as unRAID UI)
515	SMART overall-health FAILED    — get data off this drive immediately
516	```
517
518	**Silent when healthy:** no notification if all drives pass all checks.
519
520	---
521
522	### ── cert_monitor ─────────────────────────────────────────────────────────────
523
524	```
525	Category:   Monitors
526	Name:       cert_monitor
527	Schedule:   0 9 * * 0    (Sunday 9am)
528	Background: YES
529	```
530
531	```bash
532	#!/bin/bash
533	/boot/config/plugins/varaverk/Monitors/cert_monitor.sh
534	```
535
536	**What it does:**
537	Connects to each domain via live `openssl s_client` TLS connection and reads the
538	certificate actually being served — not NPM's API, not the certificate file.
539
540	**Why live TLS and not an API check:**
541	NPM can report a cert as renewed while the web server is still serving the old
542	expired cert (nginx not reloaded after renewal). A live TLS check catches this.
543	If a user would see a certificate error, this script catches it first.
544
545	**Thresholds (master.conf):**
546	```
547	> CERT_WARN_DAYS (30)   → silent ✅
548	≤ CERT_WARN_DAYS (30)   → notification — renew soon
549	≤ CERT_CRIT_DAYS (7)    → urgent notification — renew now
550	connection failed        → notification — cert unreachable
551	```
552
553	---
554
555	### ── backup_verify ───────────────────────────────────────────────────────────
556
557	```
558	Category:   Monitors
559	Name:       backup_verify
560	Schedule:   0 10 * * 0   (Sunday 10am)
561	Background: YES
562	```
563
564	```bash
565	#!/bin/bash
566	/boot/config/plugins/varaverk/Monitors/backup_verify.sh
567	```
568
569	**What it does:**
570	Randomly samples BACKUP_VERIFY_SAMPLE (default: 10) files per share, computes MD5
571	checksums on both sides via SSH, and compares. Does not trust rsync's exit code —
572	verifies actual content matches.
573
574	**Why verify independently:**
575	rsync exit 0 does not mean the copy is correct. Silent data corruption from faulty
576	hardware produces a file that exists on both sides with matching size and mtime but
577	different content. rsync never detects this. MD5 does.
578
579	**Pre-flight guards:**
580	```
581	check_connectivity()   — aborts if remote Tailscale IP unreachable
582	                          prevents all files showing as MISSING from network failure
583	check_remote_array()   — aborts if /mnt/user not mounted on remote
584	                          prevents all files showing as MISSING from array not started
585	```
586
587	---
588
589	### ── emby_session_report ─────────────────────────────────────────────────────
590
591	```
592	Category:   Monitors
593	Name:       emby_session_report
594	Schedule:   0 11 * * 0   (Sunday 11am)
595	Background: YES
596	```
597
598	```bash
599	#!/bin/bash
600	/boot/config/plugins/varaverk/Monitors/emby_session_report.sh
601	```
602
603	**What it does:**
604	Queries the Emby API for the past EMBY_REPORT_DAYS (7) days and reports streaming
605	activity. Each server reports on its own Emby instance via HOST*_EMBY_URL.
606
607	**Reports:**
608	Server info + version. Active streams now with direct play vs transcode breakdown.
609	Library counts (movies, episodes, songs). Weekly play count. Top EMBY_REPORT_TOP_N
610	most-played items. Most active users. Transcode ratio — notifies if > 80% transcoding
611	(usually means a client/format compatibility issue worth fixing).
612
613	---
614
615	### ── bandwidth_monitor_report ────────────────────────────────────────────────
616
617	```
618	Category:   Monitors
619	Name:       bandwidth_monitor_report
620	Schedule:   0 11 * * 0   (Sunday 11am — runs alongside emby_session_report)
621	Background: YES
622	```
623
624	```bash
625	#!/bin/bash
626	/boot/config/plugins/varaverk/Monitors/bandwidth_monitor.sh --report
627	```
628
629	**What it does:**
630	Reads the accumulated BANDWIDTH_LOG (written by rsync.sh after each sync) and
631	generates a weekly transfer summary.
632
633	**The log is written automatically by rsync.sh** — you never call
634	`--log-transfer` manually. This `--report` entry reads what rsync already logged.
635
636	**Reports:**
637	Per-profile run count, total transferred, average duration, failures.
638	Last 7 days timeline. Large transfer flags (> BANDWIDTH_WARN_GB).
639	Log format: DATE|TIME|profile|duration|status|bytes — version-stable field names
640	that survive rsync updates without parser changes.
641
642	---
643
644	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
645	## 🔄  FALLBACK
646	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
647
648	> **fallback.sh is started by array_started.sh — not scheduled separately.**
649	> These entries exist for manual control and testing only.
650
651	---
652
653	### ── fallback_status ─────────────────────────────────────────────────────────
654
655	```
656	Category:   Fallback
657	Name:       fallback_status
658	Schedule:   None — run manually
659	Background: NO  (runs and exits — status check)
660	```
661
662	```bash
663	#!/bin/bash
664	/boot/config/plugins/varaverk/Fallback/fallback.sh --status
665	```
666
667	**What it does:**
668	Shows current fallback state at a glance without affecting the running loop.
669	State (NORMAL/FALLBACK/NO_INTERNET/DARK). Which tier containers are active.
670	Outage duration if in FALLBACK. Handback strike count if remote has returned.
671	DDNS containers currently running. Remote server Tailscale visibility.
672
673	**Use before any manual fallback intervention.** Understanding the current state
674	before acting prevents making things worse.
675
676	---
677
678	### ── fallback_test ────────────────────────────────────────────────────────────
679
680	```
681	Category:   Fallback
682	Name:       fallback_test
683	Schedule:   None — run manually (during a planned maintenance window)
684	Background: YES — test takes several minutes
685	```
686
687	```bash
688	#!/bin/bash
689	/boot/config/plugins/varaverk/Fallback/fallback_test.sh --dry-run
690	```
691
692	**What it does:**
693	Validates the full fallback lifecycle — detection, Tier 1 container start, handback,
694	container stop, DNS return — using an iptables rule to simulate remote unreachability.
695	No real outage. Safety trap removes the iptables rule on any exit condition including
696	ctrl-c and crashes — remote is always reachable after the test.
697
698	> ⚠ **Always run with --dry-run first.** Dry run walks all 8 phases with full
699	> output but no iptables changes and no container starts/stops. Review timing and
700	> configuration before committing to a live test.
701
702	**For a live test** (real container lifecycle, brief service interruption):
703	Change `--dry-run` to nothing and run during a maintenance window.
704
705	**8 phases:** pre-flight → block remote → detect FALLBACK → verify Tier 1 started
706	→ restore remote → wait for handback → verify NORMAL restored → report pass/fail.
707
708	---
709
710	### ── fallback_state_reset ────────────────────────────────────────────────────
711
712	> **Not yet built.** `fallback_state_reset.sh` is a planned utility — not in `Fallback/` yet.
713	>
714	> **Manual workaround** when state file is stuck in FALLBACK after a failed test or
715	> forced kill of fallback.sh:
716	> ```bash
717	> # Verify first — right containers on right server, DDNS correct, fallback.sh not running
718	> cat /boot/config/fallback_state.db
719	> echo "state=NORMAL" > /boot/config/fallback_state.db
720	> ```
721	> Resets state only — does NOT start or stop any containers.
722
723	**When you need this:**
724	State file stuck in FALLBACK after a test that didn't complete cleanly. After killing
725	fallback.sh directly (not via plugin Abort button). After a failed handback.
726
727	**Verify before resetting:**
728	Right containers on right server. DDNS correct (`nslookup Gmer4Lfe.com`).
729	Both servers Tailscale connected. fallback.sh not currently running.
730
731	---
732
733	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
734	## 🔄  RSYNC
735	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
736
737	Manual trigger entries for individual rsync profiles. Used when you need to sync
738	one specific share outside the normal nightly window — after a bulk import, before a
739	planned migration, or to verify connectivity.
740
741	> **These are NOT scheduled.** Daily and weekly orchestrators handle all scheduled syncs.
742	> These exist purely for on-demand manual triggers.
743
744	**Every rsync run includes:**
745	Pre-flight checks (connectivity, version parity, remote daemon, remote rootfs, share exists).
746	Profile-matched container stop/start (if profile requires it). Bandwidth logging.
747	Drive temperature check — exit 1 on WARN, exit 2 on CRITICAL.
748
749	---
750
751	### ── rsync_critical_data ─────────────────────────────────────────────────────
752
753	```
754	Category:   Rsync
755	Name:       rsync_critical_data
756	Schedule:   None — run manually
757	Background: YES
758	```
759
760	```bash
761	#!/bin/bash
762	/boot/config/plugins/varaverk/Rsync/rsync.sh \
763	    /mnt/user/appdata-Fallback/Critical-Data
764	```
765
766	**What syncs:** auth stack — NPM proxy rules, TLS certificates, LLDAP user data,
767	Authelia policies and config, MariaDB database. Profile: critical-data. Stops auth
768	containers on both sides, syncs, restarts with dependency ordering (databases before auth).
769
770	---
771
772	### ── rsync_emby_clean ─────────────────────────────────────────────────────────
773
774	```
775	Category:   Rsync
776	Name:       rsync_emby_clean
777	Schedule:   None — run manually (also runs via weekly_sync_maintenance.sh)
778	Background: YES
779	```
780
781	```bash
782	#!/bin/bash
783	/boot/config/plugins/varaverk/Rsync/rsync.sh \
784	    /mnt/user/Media_Server/Emby --profile=emby
785	```
786
787	**What syncs:** full clean Emby mirror — all databases checkpointed, metadata, plugins,
788	config. Both Emby instances stopped during sync. Excludes transcodes, logs, cache, crash
789	files. This is the full-fidelity sync (vs emby-fallback which is the dirty 30-minute sync).
790
791	---
792
793	### ── rsync_arrs_stack ─────────────────────────────────────────────────────────
794
795	```
796	Category:   Rsync
797	Name:       rsync_arrs_stack
798	Schedule:   None — run manually
799	Background: YES
800	```
801
802	```bash
803	#!/bin/bash
804	/boot/config/plugins/varaverk/Rsync/rsync.sh \
805	    /mnt/user/appdata-Fallback/Arrs_Stack
806	```
807
808	**What syncs:** all arr appdata — Sonarr, Radarr, Lidarr, Prowlarr, Bazarr, Pinchflat
809	databases and config. Profile: arrs_stack. Arr containers stopped on both sides during
810	sync to ensure clean SQLite snapshot (no mid-write database corruption).
811
812	---
813
814	### ── rsync_important_data ────────────────────────────────────────────────────
815
816	```
817	Category:   Rsync
818	Name:       rsync_important_data
819	Schedule:   None — run manually
820	Background: YES
821	```
822
823	```bash
824	#!/bin/bash
825	/boot/config/plugins/varaverk/Rsync/rsync.sh \
826	    /mnt/user/appdata-Fallback/Important-Data
827	```
828
829	**What syncs:** NextCloud, Postgres-NextCloud, Immich, PostgreSQL-Immich appdata.
830	Profile: important-data. Postgres containers stop first, then NextCloud after delay.
831
832	---
833
834	### ── rsync_gmer4lfe ───────────────────────────────────────────────────────────
835
836	```
837	Category:   Rsync
838	Name:       rsync_gmer4lfe
839	Schedule:   None — run manually
840	Background: YES
841	```
842
843	```bash
844	#!/bin/bash
845	/boot/config/plugins/varaverk/Rsync/rsync.sh \
846	    /mnt/user/appdata-Fallback/Gmer4Lfe
847	```
848
849	**What syncs:** HOST1 server appdata backup — everything in the Gmer4Lfe appdata share.
850	No profile container stop required. Incremental — safe to run at any time.
851
852	---
853
854	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
855	## 🐳  DOCKER_ESSENTIALS
856	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
857
858	> **docker_watchdog.sh is called every 15 minutes by watchdog_orchestrator.sh — not scheduled separately.**
859	> docker_daily_restart.sh is called by daily_sync_maintenance.sh.
860	> docker_weekly_restart.sh is called by weekly_sync_maintenance.sh.
861	> docker_network_connect.sh is called by array_started.sh.
862	> These entries exist for manual control only.
863
864	---
865
866	### ── docker_watchdog_status ──────────────────────────────────────────────────
867
868	```
869	Category:   Docker_Essentials
870	Name:       docker_watchdog_status
871	Schedule:   None — run manually
872	Background: NO
873	```
874
875	```bash
876	#!/bin/bash
877	/boot/config/plugins/varaverk/Watchdogs/docker_watchdog.sh --status
878	```
879
880	**What it shows:**
881	Strike counts for all monitored containers. Current skip list with running/stopped state.
882	Whether startup grace period is active and how long remains. Whether RAM emergency
883	deferral is active (mem_shutdown_active from resource_watchdog). Last cycle timing.
884
885	---
886
887	### ── watchdog_skip_list_manager ──────────────────────────────────────────────
888
889	```
890	Category:   Docker_Essentials
891	Name:       watchdog_skip_list_manager
892	Schedule:   None — run manually (after fixing a skip-listed container)
893	Background: NO
894	```
895
896	```bash
897	#!/bin/bash
898	/boot/config/plugins/varaverk/Docker_Essentials/watchdog_skip_list_manager.sh --status
899	```
900
901	**What it does:**
902	Views and manages the persistent container skip list. When docker_watchdog.sh restarts
903	the same container 3 times in 1 hour it adds it to the skip list and stops trying.
904	A critical notification is sent. This tool clears the list after you've fixed the cause.
905
906	**Recovery workflow:**
907	```
908	1. Run --status to see what's skip-listed and its restart history
909	2. Fix the underlying problem (check docker logs, disk space, database)
910	3. --clear ContainerName to remove from skip list + clear restart history
911	4. docker start ContainerName to confirm your fix works
912	5. Watchdog resumes monitoring on next cycle automatically
913	```
914
915	**Actions:**
916	```
917	--status                       — show skip list, running/stopped state, restart counts
918	--clear ContainerName          — clear specific container (prompts YES)
919	--clear ContainerName --force  — no prompt
920	--clear-all                    — clear everything
921	--dry-run                      — preview any clear action
922	```
923
924	---
925
926	### ── docker_daily_restart_manual ─────────────────────────────────────────────
927
928	```
929	Category:   Docker_Essentials
930	Name:       docker_daily_restart_manual
931	Schedule:   None — run manually (called automatically by daily_sync_maintenance.sh)
932	Background: YES
933	```
934
935	```bash
936	#!/bin/bash
937	/boot/config/plugins/varaverk/Docker_Essentials/docker_daily_restart.sh --dry-run
938	```
939
940	**What it does:**
941	Restarts containers in HOST*_DAILY_RESTART_CONTAINERS. Running → docker restart
942	(graceful stop + start). Stopped → left stopped (state respected). Missing → skip.
943	Dependency ordering: databases restart before applications. CONTAINER_DELAY wait
944	between dependency and dependent restarts.
945
946	**Default configured containers:** NginxProxyManager, Authelia, Dispatcharr,
947	Dispatcharr-Basic, ErsatzTV-Emby — services that degrade with uptime.
948
949	> Remove `--dry-run` to run live. Default is dry-run for safety when run manually.
950
951	---
952
953	### ── docker_network_connect_manual ───────────────────────────────────────────
954
955	```
956	Category:   Docker_Essentials
957	Name:       docker_network_connect_manual
958	Schedule:   None — run manually (runs automatically via array_started.sh)
959	Background: NO
960	```
961
962	```bash
963	#!/bin/bash
964	/boot/config/plugins/varaverk/Docker_Essentials/docker_network_connect.sh --status
965	```
966
967	**What it does:**
968	Ensures custom Docker networks exist and containers are connected to them. Idempotent
969	— safe to run any time. If a network is missing (typically wiped by a unRAID update),
970	it creates it and notifies.
971
972	**Use after unRAID updates** if containers can't reach each other — the update may have
973	wiped custom networks. Run this to recreate them without restarting the array.
974
975	---
976
977	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
978	## 🎬  MEDIA
979	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
980
981	> **All media scripts are called by daily_sync_maintenance.sh via media_management.sh.**
982	> These entries exist for manual testing and targeted one-off runs.
983	> Run --dry-run --log for any new system or after configuration changes.
984
985	---
986
987	### ── media_shares_permissions_manual ────────────────────────────────────────
988
989	```
990	Category:   Media
991	Name:       media_shares_permissions_manual
992	Schedule:   None — run manually (runs automatically via daily_sync_maintenance.sh)
993	Background: YES
994	```
995
996	```bash
997	#!/bin/bash
998	/boot/config/plugins/varaverk/Media/media_shares_permissions.sh --dry-run
999	```
1000
1001	**What it does:**
1002	Applies `nobody:users` ownership and 755 (dirs) / 664 (files) permissions to every
1003	file in HOST*_MEDIA_PERMISSION_SHARES. Two separate passes — directories and files
1004	get different modes. Idempotent — only changes what's wrong.
1005
1006	**When ownership drifts:** scp/cp from another machine leaves root:root. New container
1007	before PUID/PGID is set. rsync without --chown. This fixes it all in one pass.
1008
1009	> Remove `--dry-run` to run live. Dry run shows count of files that would be corrected.
1010
1011	---
1012
1013	### ── media_cleaner_anime_manual ──────────────────────────────────────────────
1014
1015	```
1016	Category:   Media
1017	Name:       media_cleaner_anime_manual
1018	Schedule:   None — run manually (runs automatically via daily_sync_maintenance.sh)
1019	Background: YES
1020	```
1021
1022	```bash
1023	#!/bin/bash
1024	/boot/config/plugins/varaverk/Media/media_cleaner.sh anime --dry-run
1025	```
1026
1027	**What it does:**
1028	Removes junk files from anime share folders using ANIME_FILE_PATTERNS. Patterns
1029	include: *.sfv *.nfo *.rar *.sample* *.torrent *.url *.log and other scene release
1030	debris. Runs before arr cleanup so orphan detection only sees actual media files.
1031
1032	---
1033
1034	### ── media_cleaner_media_manual ──────────────────────────────────────────────
1035
1036	```
1037	Category:   Media
1038	Name:       media_cleaner_media_manual
1039	Schedule:   None — run manually
1040	Background: YES
1041	```
1042
1043	```bash
1044	#!/bin/bash
1045	/boot/config/plugins/varaverk/Media/media_cleaner.sh media --dry-run
1046	```
1047
1048	**What it does:**
1049	Same as above but for media share folders using MEDIA_FILE_PATTERNS — all anime
1050	patterns plus *.iso and *.lrc.
1051
1052	---
1053
1054	### ── lidarr_cleanup_manual ───────────────────────────────────────────────────
1055
1056	```
1057	Category:   Media
1058	Name:       lidarr_cleanup_manual
1059	Schedule:   None — run manually (runs automatically via daily_sync_maintenance.sh)
1060	Background: YES
1061	```
1062
1063	```bash
1064	#!/bin/bash
1065	/boot/config/plugins/varaverk/Media/lidarr_cleanup.sh --dry-run --log
1066	```
1067
1068	**What it does:**
1069	Queries Lidarr API for all tracked file paths. Walks /mnt/user/Music. Files on disk
1070	not in the API response that are older than LIDARR_ORPHAN_AGE (7 days) and not
1071	currently open by any process = orphans = deleted.
1072
1073	**7 safety checks must ALL pass before any deletion:**
1074	Lidarr running + healthy. API reachable. API version matches. Artist count > 0.
1075	Tracked file count > 0. Count >= 80% of last known (detects database issues).
1076	Total deletion size < LIDARR_MAX_DELETE_GB (50GB) — stops catastrophic path mismatches.
1077
1078	**Protected always:** *.jpg *.png (cover art), *.nfo (metadata), *.lrc (lyrics)
1079	**After deletion:** notify_emby_scan() triggers "Clean Missing Files" in Emby immediately.
1080
1081	> Remove `--dry-run` only after reviewing the full log output. See README-Media.md
1082	> for the complete safe testing procedure.
1083
1084	---
1085
1086	### ── sonarr_cleanup_manual ───────────────────────────────────────────────────
1087
1088	```
1089	Category:   Media
1090	Name:       sonarr_cleanup_manual
1091	Schedule:   None — run manually
1092	Background: YES
1093	```
1094
1095	```bash
1096	#!/bin/bash
1097	/boot/config/plugins/varaverk/Media/sonarr_cleanup.sh --dry-run --log
1098	```
1099
1100	**What it does:**
1101	Same logic as lidarr_cleanup but for TV files. Walks HOST*_SONARR_TV_ROOT.
1102	Protected: *.jpg *.png (artwork), *.nfo, *.srt *.sub *.ass *.ssa (Bazarr subtitles).
1103	Host-aware: HOST1 manages Tv_Shows, HOST2 manages Anime_Shows — correct root via detect_hosts().
1104
1105	---
1106
1107	### ── radarr_cleanup_manual ───────────────────────────────────────────────────
1108
1109	```
1110	Category:   Media
1111	Name:       radarr_cleanup_manual
1112	Schedule:   None — run manually
1113	Background: YES
1114	```
1115
1116	```bash
1117	#!/bin/bash
1118	/boot/config/plugins/varaverk/Media/radarr_cleanup.sh --dry-run --log
1119	```
1120
1121	**What it does:**
1122	Same logic for movie files. Walks HOST*_RADARR_MOVIES_ROOT.
1123	Protected: *.jpg *.png (movie artwork), *.nfo, subtitles.
1124	Host-aware: HOST1 manages Movies, HOST2 manages Anime_Movies.
1125
1126	---
1127
1128	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1129	## 🎬  TRANSCODES
1130	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1131
1132	> **ramdisk_setup.sh is called by array_started.sh — not scheduled separately.**
1133	> **transcode_manager.sh and transcode_cleanup.sh are called by transcode_management.sh.**
1134	> These entries exist for manual status checks and emergency use.
1135
1136	---
1137
1138	### ── ramdisk_status ────────────────────────────────────────────────────────────
1139
1140	```
1141	Category:   Transcodes
1142	Name:       ramdisk_status
1143	Schedule:   None — run manually
1144	Background: NO
1145	```
1146
1147	```bash
1148	#!/bin/bash
1149	/boot/config/plugins/varaverk/Transcodes/ramdisk_setup.sh --status
1150	```
1151
1152	**What it shows:**
1153	Whether ramdisk is mounted at RAMDISK_PATH. Current symlink target (ramdisk or SSD).
1154	Whether transcoding-temp exists on ramdisk. SSD fallback directory state.
1155	Current usage on both ramdisk and SSD.
1156
1157	---
1158
1159	### ── transcode_manager_manual ────────────────────────────────────────────────
1160
1161	```
1162	Category:   Transcodes
1163	Name:       transcode_manager_manual
1164	Schedule:   None — run manually (runs via transcode_management.sh every 7 min)
1165	Background: NO
1166	```
1167
1168	```bash
1169	#!/bin/bash
1170	/boot/config/plugins/varaverk/Transcodes/transcode_manager.sh --status
1171	```
1172
1173	**What it shows:**
1174	Current mode (smart/ramdisk/ssd). Ramdisk usage vs thresholds. Current symlink target.
1175	Active sessions per configured TRANSCODE_SERVERS. Daily log stats (peak this week,
1176	flip count, session split ramdisk vs SSD).
1177
1178	---
1179
1180	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1181	## 🖥  UNRAID_ESSENTIALS
1182	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1183
1184	> **stability_watchdog.sh is called by watchdog_orchestrator.sh — not scheduled separately.**
1185	> **webgui_watchdog.sh, system_tuning_monitor.sh, clear_logs.sh are scheduled below.**
1186	> **inotify_tuning.sh, php_fpm_max_children.sh, docker_syslog_filter.sh run via array_started.sh.**
1187
1188	---
1189
1190	### ── webgui_watchdog ───────────────────────────────────────────────────────────
1191
1192	```
1193	Category:   System_Essentials
1194	Name:       webgui_watchdog
1195	Schedule:   */10 * * * *  (every 10 minutes)
1196	Background: YES
1197	```
1198
1199	```bash
1200	#!/bin/bash
1201	/boot/config/plugins/varaverk/Watchdogs/System/webgui_watchdog.sh
1202	```
1203
1204	**What it does:**
1205	Checks WebGUI responsiveness and recovers it automatically. Silent when healthy —
1206	runs 144 times per day with no output when the WebGUI is responding normally.
1207
1208	**Escalation path (each step only runs if previous failed):**
1209	```
1210	1. curl http://localhost → unresponsive for WEBGUI_TIMEOUT (5s)?
1211	2. Restart nginx                    → wait 15s → recheck
1212	3. Restart php-fpm                  → wait 10s → recheck  (handles worker saturation)
1213	4. Restart emhttp (core daemon)     → wait 30s → recheck  (most disruptive, last resort)
1214	5. All failed → notify warning — manual investigation needed
1215	```
1216
1217	A notification fires on any restart so you know what recovered and when.
1218
1219	---
1220
1221	### ── clear_logs ───────────────────────────────────────────────────────────────
1222
1223	```
1224	Category:   System_Essentials
1225	Name:       clear_logs
1226	Schedule:   0 5 * * 0    (Sunday 5am — before Sunday maintenance window)
1227	Background: YES
1228	```
1229
1230	```bash
1231	#!/bin/bash
1232	/boot/config/plugins/varaverk/System_Essentials/clear_logs.sh
1233	```
1234
1235	**What it does:**
1236	Clears system and Docker container logs using size thresholds — only logs large
1237	enough to be worth clearing.
1238
1239	**System logs** (/var/log/syslog, messages, dmesg): cleared only if > LOG_MIN_SIZE_MB (10MB).
1240	Smaller logs contain useful recent history — not cleared.
1241
1242	**Docker container logs**: cleared per-container only if > LOG_DOCKER_MAX_MB (100MB).
1243	Active containers like Emby and SABnzbd grow fastest. Truncation not rotation —
1244	`> file` keeps the file descriptor valid while emptying. Safe for running services.
1245
1246	---
1247
1248	### ── stability_watchdog_status ──────────────────────────────────────────────────
1249
1250	```
1251	Category:   System_Essentials
1252	Name:       stability_watchdog_status
1253	Schedule:   None — run manually
1254	Background: NO
1255	```
1256
1257	```bash
1258	#!/bin/bash
1259	/boot/config/plugins/varaverk/Watchdogs/stability_watchdog.sh --status
1260	```
1261
1262	**What it shows:**
1263	All three tiers with current thresholds. Active strike counts per check. RAM tier
1264	status. Whether mem_shutdown_active is set (RAM emergency in progress). Recent
1265	watchdog-triggered reboots. All 18 per-host toggle states.
1266
1267	---
1268
1269	### ── mover_stop ───────────────────────────────────────────────────────────────
1270
1271	```
1272	Category:   System_Essentials
1273	Name:       mover_stop
1274	Schedule:   None — run manually (before array operations needing mover stopped)
1275	Background: NO
1276	```
1277
1278	```bash
1279	#!/bin/bash
1280	/boot/config/plugins/varaverk/System_Essentials/mover_stop.sh
1281	```
1282
1283	**What it does:**
1284	Stops the unRAID mover cleanly. Wall message warning to all logged-in users. Waits
1285	MOVER_STOP_TIMEOUT seconds. SIGTERM (allows mover to finish current file — no partial
1286	files). Waits + verifies. SIGKILL if still running (warns that partial files are possible).
1287
1288	---
1289
1290	### ── rsync_stop ────────────────────────────────────────────────────────────────
1291
1292	```
1293	Category:   System_Essentials
1294	Name:       rsync_stop
1295	Schedule:   None — run manually (when rsync needs stopping)
1296	Background: NO
1297	```
1298
1299	```bash
1300	#!/bin/bash
1301	/boot/config/plugins/varaverk/System_Essentials/rsync_stop.sh --status
1302	```
1303
1304	**What it does:**
1305	Stops rsync intelligently. Auto-detects if an orchestrator is running:
1306	Smart mode (default): kills rsync subprocess only — orchestrator sees rsync died and
1307	exits cleanly. No orphaned lock files. Containers recover via docker_watchdog.sh.
1308	`--full-stop`: kills orchestrator first then rsync — use for emergency stop.
1309
1310	---
1311
1312	### ── user_scripts_stop ────────────────────────────────────────────────────────
1313
1314	```
1315	Category:   System_Essentials
1316	Name:       user_scripts_stop
1317	Schedule:   None — run manually (before reboots, or when a script hangs)
1318	Background: NO
1319	```
1320
1321	```bash
1322	#!/bin/bash
1323	/boot/config/plugins/varaverk/System_Essentials/user_scripts_stop.sh --status
1324	```
1325
1326	**What it does:**
1327	Finds all running User Script processes by /tmp/user.scripts path signature. Shows
1328	script names (not just PIDs — you know what you're stopping). SIGTERM → verify →
1329	SIGKILL. Self-exclusion prevents the script from killing itself. Called automatically
1330	by server_reboot.sh before reboot.
1331
1332	---
1333
1334	### ── server_reboot ────────────────────────────────────────────────────────────
1335
1336	```
1337	Category:   System_Essentials
1338	Name:       server_reboot
1339	Schedule:   None — run manually (planned maintenance reboots)
1340	Background: YES — sequence takes 1-2 minutes
1341	```
1342
1343	```bash
1344	#!/bin/bash
1345	/boot/config/plugins/varaverk/System_Essentials/server_reboot.sh --dry-run
1346	```
1347
1348	**What it does:**
1349	Graceful reboot with full pre-flight and clean shutdown. Wall message + unRAID
1350	notification with MY_ID. Waits REBOOT_SLEEP (30s). virsh shutdown each running VM
1351	then waits REBOOT_VM_WAIT for graceful exit. Stops Docker. Stops libvirt. sync.
1352	/sbin/reboot.
1353
1354	Pre-flight warnings (not blocks): rsync running, mover running, active Emby sessions.
1355	These show in the summary — you chose to reboot, these are context.
1356
1357	> Remove `--dry-run` to execute. Dry run walks the full sequence without rebooting —
1358	> use it to verify what processes are active before committing.
1359
1360	---
1361
1362	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1363	## 🤝  PARTNERSHIP
1364	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1365
1366	Manages the two-server auth-sharing relationship lifecycle. Automated check runs
1367	inside critical_sync_maintenance.sh. These entries are for manual operations only.
1368
1369	---
1370
1371	### ── partnership_status ────────────────────────────────────────────────────────
1372
1373	```
1374	Category:   Partnership
1375	Name:       partnership_status
1376	Schedule:   None — run manually
1377	Background: NO
1378	```
1379
1380	```bash
1381	#!/bin/bash
1382	/boot/config/plugins/varaverk/Partnership/partnership_manage.sh --status
1383	```
1384
1385	**What it shows:**
1386	Both server state files (ACTIVE/INACTIVE). Offline counter (days since last successful
1387	sync). Which server is the auth stack owner. Current WebUI redirect state on HOST2.
1388	Whether CRITICAL_RSYNC_ENABLED is true.
1389
1390	---
1391
1392	### ── partnership_onboard ──────────────────────────────────────────────────────
1393
1394	```
1395	Category:   Partnership
1396	Name:       partnership_onboard
1397	Schedule:   None — run manually (HOST1/owner only, once at relationship start)
1398	Background: YES
1399	```
1400
1401	```bash
1402	#!/bin/bash
1403	/boot/config/plugins/varaverk/Partnership/partnership_manage.sh --onboard --dry-run
1404	```
1405
1406	**What it does:**
1407	Establishes the auth-sharing partnership. Reconfigures HOST2 Docker template WebUI
1408	URLs to redirect to HOST1 via Tailscale. Verifies routing works. Writes ACTIVE state
1409	on both servers. Enables Critical-Data sync.
1410
1411	> Remove `--dry-run` to run live. Dry run shows exactly which templates would be
1412	> changed and what URLs they'd become — review before committing.
1413
1414	---
1415
1416	### ── partnership_offboard ─────────────────────────────────────────────────────
1417
1418	```
1419	Category:   Partnership
1420	Name:       partnership_offboard
1421	Schedule:   None — run manually (either server, when ending the partnership)
1422	Background: YES
1423	```
1424
1425	```bash
1426	#!/bin/bash
1427	/boot/config/plugins/varaverk/Partnership/partnership_manage.sh --offboard --dry-run
1428	```
1429
1430	**What it does:**
1431	Clean partnership separation. Stops rsync. Final sync to HOST2. Reconfigures HOST2
1432	WebUI URLs back to localhost (HOST2 manages its own auth). Disables critical sync.
1433	Grace period (PARTNERSHIP_GRACE_HOURS). Removes HOST2 from Tailscale tailnet.
1434
1435	Both servers can initiate. If HOST1 is unreachable when HOST2 offboards, deferred
1436	finalisation completes automatically when HOST1 can next reach HOST2.
1437
1438	> Remove `--dry-run` to run live. 10-second countdown shown before any changes.
1439
1440	---
1441
1442	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1443	## 🔧  TOOLS
1444	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1445
1446	Situational utilities — run when something needs fixing, not on a schedule.
1447
1448	---
1449
1450	### ── bulk_permissions_repair ─────────────────────────────────────────────────
1451
1452	```
1453	Category:   Tools
1454	Name:       bulk_permissions_repair
1455	Schedule:   None — run manually
1456	Background: YES
1457	```
1458
1459	```bash
1460	#!/bin/bash
1461	/boot/config/plugins/varaverk/Tools/bulk_permissions_repair.sh /mnt/user/Movies --dry-run
1462	```
1463
1464	**What it does:**
1465	Applies nobody:users 755/664 to a specific path immediately — without waiting for
1466	the nightly media_shares_permissions.sh run. Use when an admin copy, scp, or bad
1467	container left root-owned files that are blocking arr operations right now.
1468	Replace /mnt/user/Movies with the actual path that needs fixing.
1469
1470	---
1471
1472	### ── container_data_export ────────────────────────────────────────────────────
1473
1474	```
1475	Category:   Tools
1476	Name:       container_data_export
1477	Schedule:   None — run manually (before risky updates or migrations)
1478	Background: YES
1479	```
1480
1481	```bash
1482	#!/bin/bash
1483	/boot/config/plugins/varaverk/Tools/container_data_export.sh \
1484	    Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/
1485	```
1486
1487	**What it does:**
1488	Stops the container cleanly. Archives the entire appdata directory to a timestamped
1489	.tar.gz. Verifies archive integrity. Restarts the container (always, even if archiving
1490	failed). Space check before stopping — won't start if output drive is too small.
1491
1492	Replace ContainerName, /path/to/appdata, and /output/dir for each use.
1493	Output filename: ContainerName_YYYY-MM-DD_HH-MM.tar.gz
1494
1495	---
1496
1497	### ── emby_database_repair ─────────────────────────────────────────────────────
1498
1499	```
1500	Category:   Tools
1501	Name:       emby_database_repair
1502	Schedule:   None — run manually (after power loss or repeated Emby crashes)
1503	Background: YES
1504	```
1505
1506	```bash
1507	#!/bin/bash
1508	/boot/config/plugins/varaverk/Tools/emby_database_repair.sh
1509	```
1510
1511	**What it does:**
1512	Stops Emby. Runs SQLite `PRAGMA integrity_check` on every Emby database. Restarts
1513	Emby. Reports per-database with specific recovery guidance.
1514
1515	**Per-database recovery guidance:**
1516	```
1517	library.db          — CORRUPT → safe to delete, Emby fully rebuilds from media files
1518	users.db            — CORRUPT → deleting resets ALL user accounts + watch history
1519	                                check for recent backup (weekly_sync_maintenance.sh)
1520	authentication.db   — CORRUPT → safe to delete, API keys regenerated on restart
1521	activity.db         — CORRUPT → safe to delete, it's a log only
1522	```
1523
1524	**This script checks and reports only** — it does not automatically delete anything.
1525	Recovery is a human decision based on whether you have backups.
1526
1527	---
1528
1529	### ── zfs_pool_scrub ────────────────────────────────────────────────────────────
1530
1531	```
1532	Category:   Tools
1533	Name:       zfs_pool_scrub
1534	Schedule:   None — run manually (monthly or after any disk/power event)
1535	Background: YES
1536	```
1537
1538	```bash
1539	#!/bin/bash
1540	/boot/config/plugins/varaverk/Tools/zfs_pool_scrub.sh
1541	```
1542
1543	**What it does:**
1544	Triggers ZFS scrub on all pools (except ZFS_REPORT_IGNORE_POOLS), waits for completion,
1545	notifies with results. ZFS checksums every block and verifies — catches silent data
1546	corruption before you discover it on reads. Safe to run while system is active.
1547	For a specific pool: add the pool name as argument (bypasses ignore list).
1548
1549	---
1550
1551	### ── recreate_shares ──────────────────────────────────────────────────────────
1552
1553	```
1554	Category:   Tools
1555	Name:       recreate_shares
1556	Schedule:   None — run manually (after fresh install or disk replacement)
1557	Background: YES
1558	```
1559
1560	```bash
1561	#!/bin/bash
1562	/boot/config/plugins/varaverk/Tools/recreate_shares.sh --dry-run
1563	```
1564
1565	**What it does:**
1566	Reads every .cfg in /boot/config/shares/, parses the shareInclude disk list, and
1567	creates /mnt/diskN/sharename on each included disk. Run after restoring share .cfg
1568	files to a fresh install — shares appear in the UI but directories are missing until
1569	this creates them. chown nobody:users on creation.
1570
1571	---
1572
1573	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1574	## 🔀  GIT
1575	## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1576
1577	---
1578
1579	### ── git_pull ─────────────────────────────────────────────────────────────────
1580
1581	```
1582	Category:   Git
1583	Name:       git_pull
1584	Schedule:   None — run manually (called automatically by daily_sync_maintenance.sh)
1585	Background: NO
1586	```
1587
1588	```bash
1589	#!/bin/bash
1590	/boot/config/plugins/varaverk/System_Essentials/git_pull_execute.sh
1591	```
1592
1593	**What it does:**
1594	Pulls latest commits from Gitea. Sets execute permission on all .sh files. This is
1595	the deployment mechanism for the ecosystem — push changes from VS Code to Gitea,
1596	then run this on both servers to deploy. daily_sync_maintenance.sh runs it
1597	automatically at 1am as the first job in the window.
1598
1599	---
1600
1601	## ━━━ AT STARTUP OF ARRAY — ORDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1602
1603	**Only one entry needed at startup:** `array_start`. It handles everything.
1604
1605	`array_started.sh` is the single "At Startup of Array" entry. It reads ARRAY_START_SCRIPTS
1606	from master.conf and launches each in order. No other startup entries. No dragging to
1607	reorder individual scripts. Change the order by editing master.conf.
1608
1609	```
1610	Correct startup setup:
1611	  1 entry: array_start   — At Startup of Array, Background: YES
1612
1613	Wrong startup setup (old approach):
1614	  ramdisk_setup        — At Startup of Array
1615	  docker_syslog_filter — At Startup of Array
1616	  stability_watchdog     — Every 15 Minutes via watchdog_orchestrator
1617	  docker_watchdog      — At Startup of Array
1618	  fallback             — At Startup of Array
1619	  (unpredictable order, no coordination)
1620	```
1621
1622	---
1623
1624	## ━━━ COMPLETE SCHEDULE REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1625
1626	```
1627	At Startup of Array:
1628	  array_start                          BG: YES — single entry, handles everything
1629
1630	*/7 * * * *   (every 7 min):
1631	  transcode_management                 BG: YES — cleanup then manager
1632
1633	*/30 * * * *  (every 30 min):
1634	  critical_sync_maintenance            BG: YES — auth stack + Emby dirty sync + partnership
1635
1636	0 */6 * * *   (every 6 hours):
1637	  arrs_failed_stalled_recovery         BG: YES — blocklist + re-search failed imports
1638	  system_tuning_monitor                BG: YES — inotify + php-fpm snapshot
1639
1640	0 1 * * *     (1am daily):
1641	  daily_sync_maintenance               BG: YES — full nightly window
1642
1643	0 5 * * 0     (Sunday 5am):
1644	  clear_logs                           BG: YES — size-threshold log cleanup
1645
1646	0 6 * * 0     (Sunday 6am):
1647	  zfs_memory_snapshot                  BG: YES — ZFS + memory report
1648
1649	0 7 * * 0     (Sunday 7am):
1650	  sunday_morning_coffee_report         BG: YES — full weekly digest
1651
1652	0 8 * * *     (8am daily):
1653	  weekly_health_digest                 BG: YES — profile-controlled notification
1654
1655	0 7 * * 0     (Sunday 7am):
1656	  smart_health                         BG: YES — SMART attribute check
1657
1658	0 9 * * 0     (Sunday 9am):
1659	  cert_monitor                         BG: YES — live TLS cert expiry check
1660
1661	0 10 * * 0    (Sunday 10am):
1662	  backup_verify                        BG: YES — MD5 checksum mirror verification
1663
1664	0 11 * * 0    (Sunday 11am):
1665	  emby_session_report                  BG: YES — Emby weekly usage stats
1666	  bandwidth_monitor_report             BG: YES — rsync transfer summary
1667
1668	30 2 * * 0    (Sunday 2:30am):
1669	  weekly_sync_maintenance              BG: YES — clean sync + image updates
1670	```
1671
1672	---
1673
1674	## ━━━ AFTER CREATING ALL ENTRIES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1675
1676	Run each of these once manually before relying on scheduled runs:
1677
1678	```
1679	array_start --dry-run          verify all startup scripts found and sequenced correctly
1680	transcode_management --status  confirm ramdisk mounted and symlink pointing at ramdisk
1681	critical_sync_maintenance      verify Critical-Data sync completes to HOST2
1682	daily_sync_maintenance --dry-run  verify remote connectivity and share list
1683	fallback_status                confirm NORMAL state on both servers
1684	docker_watchdog_status         confirm required containers all running
1685	stability_watchdog_status         confirm no threshold alerts on first check
1686	```


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= CONFIGURATION LOADER =======================================
4	# ==============================================================================================
5	# Single entry point for all configuration sourcing across the ecosystem.
6	# Every script sources this file instead of sourcing master.conf files directly.
7	#
8	# ── HOW IT WORKS ──────────────────────────────────────────────────────────────────────────────
9	# 1. Detects OS platform → PLATFORM=unraid|truenas|unknown; exports SCRIPTS_DIR
10	# 2. Sources master.conf (shared config — hostnames, thresholds, toggles, profiles, job lists)
11	# 3. Auto-discovers and sources all host*.conf files in the same directory
12	#    Each host conf extends the shared profile arrays and adds host-specific credentials
13	# 4. Sources common.sh (shared functions — detect_hosts, logging, notifications etc.)
14	# 5. Sources Plugin/<platform>/adapter.sh (platform_*() functions for OS-specific ops)
15	#
16	# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────
17	# Without this loader every script had to explicitly source each conf file:
18	#   source master.conf
19	#   source host1.conf
20	#   source host2.conf
21	#   source common.sh
22	#
23	# Adding a new server meant updating every script.
24	# With this loader — add host3.conf to the git repo and every server
25	# auto-discovers it on next git pull. Zero script changes required. Ever.
26	#
27	# ── ADDING A NEW SERVER ───────────────────────────────────────────────────────────────────────
28	# 1. Create Configurations/host3.conf following the same structure as HOST1/HOST2
29	# 2. Commit and push to git repo
30	# 3. All servers pull it automatically — no other changes needed
31	#
32	# ── USAGE IN SCRIPTS ──────────────────────────────────────────────────────────────────────────
33	# Replace the three source lines at the top of every script with:
34	#
35	#   SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
36	#   source "$SCRIPT_DIR/../load_config.sh"
37	#
38	# Scripts in subdirectories (Rsync/, Docker_Essentials/ etc.) use ../ to reach root.
39	# Scripts in root directory use ./ instead:
40	#
41	#   SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
42	#   source "$SCRIPT_DIR/load_config.sh"
43	#
44	# ── SPARSE CHECKOUT NOTE ──────────────────────────────────────────────────────────────────────
45	# Sparse checkout controls which host*.conf files each server receives.
46	# HOST1 only pulls Configurations/host1.conf — never HOST2's credentials.
47	# HOST2 only pulls Configurations/host2.conf — never HOST1's credentials.
48	# This loader sources whatever conf files ARE present — sparse checkout handles the rest.
49	# Both servers pull all non-credential conf files (Configurations/master.conf, common.sh, load_config.sh).
50	#


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= COMMON LIBRARY =============================================
4	# ==============================================================================================
5	# Shared functions used by every script in the ecosystem.
6	# Sourced automatically by load_config.sh — do not source directly.
7	#
8	# ── WHAT THIS FILE PROVIDES ───────────────────────────────────────────────────────────────────
9	#   Icons            — consistent visual language across all script output
10	#   Output helpers   — info, warn, error, success, log, notify
11	#   Arg parser       — unified --dry-run, --log, --status, KEY=VALUE handling
12	#   Validation       — require_var, validate_int
13	#   Host detection   — detect_hosts() identifies local/remote server, sets all aliases
14	#   Connectivity     — resolve_remote_ip, check_connectivity, ping_remote, ping_internet
15	#   Health checks    — local/remote array, docker daemon, rootfs, share, disk temps
16	#   Disk verification — check_remote_disks() — XFS, ZFS, cache pools, nested paths
17	#   Container mgmt   — stop/start local and remote containers with state tracking
18	#   Rsync helpers    — get_rsync_opts, check_rsync_enabled
19	#   Locking          — acquire_lock, acquire_rsync_lock — concurrent execution protection
20	#   Path translation — translate_path() — container → host path for arr cleanup scripts
21	#   Arr utilities    — check_arr_version() — API version safety gate
22	#   Status display   — show_status() — runtime config dump on --status flag
23	#
24	# ── CHANGELOG ─────────────────────────────────────────────────────────────────────────────────
25	#
26	#   Version  What changed
27	#   ───────────────────────────────────────────────────────────────────────────────────────────
28	#   v1.0     Initial stable framework — output helpers, arg parser, host detection
29	#
30	#   v1.1     format_duration() moved here from daily_sync_maintenance.sh for shared use
31	#            SSH_KEY naming collision resolved — gitea key renamed GITEA_SSH_KEY in master.conf
32	#
33	#   v1.2     check_connectivity() added — fatal ping with Tailscale hint on failure
34	#            Function header comment blocks standardised across all functions
35	#
36	#   v1.3     check_remote_rootfs() — aborts rsync if remote rootfs exceeds ROOTFS_WARN
37	#            check_remote_share() — aborts if target directory missing or empty on remote
38	#            Both protect against rsync running when remote array is down
39	#
40	#   v1.4     check_remote_disks() — verifies all physical disks backing a share are mounted
41	#            Auto-discovers disk layout at runtime from disks.ini — no config required
42	#            Aborts if any disk backing the share is offline or unmounted
43	#
44	#   v1.5-2.9 Icon set expanded progressively — each operation has its own distinct icon
45	#            All function output updated to use correct icon per context
46	#            notify() added — unRAID native + Discord webhook notifications
47	#            Script locking system added — acquire_lock(), acquire_rsync_lock()
48	#            translate_path() added — container → host path for arr cleanup scripts
49	#            check_arr_version() added — API version safety gate before arr operations
50	#            check_api() added — pre-flight API reachability check
51	#            ping_remote(), ping_internet() — non-fatal ping for fallback use
52	#            check_local_array(), check_remote_array() — array health checks
53	#            check_remote_docker() — Docker daemon health check
54	#            get_unraid_temp_thresholds() — reads thresholds from dynamix.cfg
55	#            check_local_disk_temps() — pre-rsync temp check with exit codes 0/1/2
56	#            stop/start local containers added alongside existing remote variants
57	#
58	#   v3.1     Three-file config split — master.conf + host1.conf + host2.conf
59	#            load_config.sh introduced — auto-discovers all host*.conf files
60	#            detect_hosts() rewritten — sets MY_ID/REMOTE_ID and aliases all HOST* vars


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Git Pull & Execute =========================================
4	# ==============================================================================================
5	# Pulls the latest scripts from the Gitea repository via SSH.
6	# Lives at the repo root — sources load_config.sh from the same directory.
7	#
8	# ── WHAT THIS SCRIPT DOES ─────────────────────────────────────────────────────────────────────
9	# 1. Detects which server it's running on via detect_hosts() (MY_ID)
10	# 2. Configures sparse checkout to exclude other servers' credential files
11	#    Each server only pulls its own host*.conf — never sees peer credentials
12	# 3. Pulls or clones latest scripts from Gitea
13	# 4. Sets executable permissions on all .sh files
14	#
15	# ── SPARSE CHECKOUT ───────────────────────────────────────────────────────────────────────────
16	# Sparse checkout ensures each server only receives its own host conf:
17	#   HOST1 pulls:  master.conf + host1.conf + all scripts
18	#   HOST1 skips:  host2.conf, host3.conf etc.
19	#   HOST2 pulls:  master.conf + host2.conf + all scripts
20	#   HOST2 skips:  host1.conf, host3.conf etc.
21	#
22	# Adding a new server:
23	#   Create host3.conf in the repo
24	#   All existing servers automatically exclude it on next pull
25	#   New server gets only its own conf ✅
26	#
27	# ── GITEA LOCATION DETECTION ──────────────────────────────────────────────────────────────────
28	# Detects where Gitea is running at runtime — works through fallback:
29	#   Gitea local  → connects via local IP
30	#   Gitea remote → connects via Tailscale IP
31	#   Both fail    → falls back to GITEA_DOMAIN if configured
32	#
33	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
34	#   GITEA_CONTAINER  — Docker container name for Gitea
35	#   GITEA_REPO_PATH  — repo path on Gitea (e.g. Varaverk/varaverk.git)
36	#   GITEA_DOMAIN     — public domain fallback (optional)
37	#   TARGET_DIR       — local path to clone/pull into
38	#   GITEA_SSH_KEY    — SSH key for Gitea authentication
39	#   SSH_PORT         — Gitea SSH port (often 221 or 222)
40	#
41	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
42	#   git_pull_execute.sh              — normal pull
43	#   git_pull_execute.sh --dry-run    — preview without making changes
44	#   git_pull_execute.sh --log        — verbose output
45	#   git_pull_execute.sh --status     — show config and exit
46	# ==============================================================================================
47
48	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
49
50	# Root level script — load_config.sh is in the same directory
51	source "$SCRIPT_DIR/load_config.sh"
52
53	parse_args "$@"
54
55	# ==============================================================================================
56	# ━━━ Setup ━━━
57	# ==============================================================================================
58	if [[ "$EUID" -ne 0 ]]; then
59	    error "Must be run as root"
60	    exit 1


Now let me search for any additional documentation that might reference schedules or scripts to cross-check:


Bash

find /boot/config/plugins/varaverk -name "*.md" -type f | head -20


Let me check the Orchestrators README for schedule claims:


1	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2	# 🎯  ORCHESTRATORS
3	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4
5	**Sequential job runners that coordinate multiple scripts into single scheduled units.**
6	Orchestrators contain no business logic — they call other scripts in order, track
7	pass/fail per job, and produce one clean summary. Configuration lives in `master.conf`.
8	Adding or removing a job never requires touching the orchestrator script itself.
9
10	> **The Varaverk scheduler runs only orchestrators.** Every cron entry, every array
11	> start/stop event, every scheduled operation runs through an orchestrator. The individual
12	> scripts it calls are never scheduled directly — they run in a defined order inside a
13	> coordinated window, with a unified summary at the end.
14
15	---
16
17	## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
18
19	---
20
21	### 🔴  Race Conditions From Independent Scheduling
22
23	Media permissions ran at 01:05. Arr cleanup ran at 01:00. Arr cleanup started
24	five minutes before permissions were applied — running against files that were still
25	owned by root, silently failing to delete the ones it should have cleaned up. Both
26	scripts reported success. Neither knew about the other. The result was a library
27	that looked cleaned but wasn't.
28
29	The fix: orchestrators enforce order. `daily_sync_maintenance.sh` runs permissions
30	first, then arr cleanup. The arr scripts see correct ownership every time because the
31	orchestrator guarantees it. No race, no silent failure, no coordinating cron entries.
32
33	---
34
35	### 🔴  Six Separate Notifications Instead of One
36
37	Before orchestrators, each script sent its own notification on completion. A single
38	daily run produced six separate notification pings — one for permissions, one for
39	each arr cleanup, one for the cleaner, one for docker restart. Six bells for one
40	maintenance window. Worse, if something failed in the middle, you'd get some
41	notifications and not others, and figuring out which step failed meant correlating
42	timestamps across multiple notification messages.
43
44	The fix: orchestrators collect all results and send one notification at the end.
45	One summary. One bell. Clear pass/fail count. If something failed, the summary
46	tells you which job and what happened — no correlation needed.
47
48	---
49
50	### 🔴  Transcode Manager Triggering Unnecessary SSD Flips
51
52	`transcode_manager.sh` ran every 7 minutes on its own. It checked ramdisk usage —
53	saw 6.8GB used, threshold is 6.5GB, flipped sessions to SSD. One minute later
54	`transcode_cleanup.sh` ran and removed 4GB of stale segment files from ended sessions.
55	Actual usage was 2.8GB. Sessions were now on SSD for no reason. Users experiencing
56	slightly worse performance. The flip counter incremented for nothing.
57
58	The fix: `transcode_management.sh` runs cleanup first, manager second, every cycle.
59	The manager always sees post-cleanup usage. Stale files can't trigger a flip because
60	they're gone before the manager looks. The correct order requires exactly one
61	orchestrator to enforce it.
62
63	---
64
65	### 🔴  Failed Imports Sitting Stalled for Days
66
67	A release downloads successfully but Lidarr can't import it — wrong format, incorrect
68	tags, file already exists. Lidarr marks it `importFailed` and stops. Nobody notices.
69	The download client has the file, Lidarr has given up, and nothing is going to happen
70	until someone manually opens Lidarr, identifies the problem, blocklists the release,
71	and triggers a new search. This takes minutes to do — but nobody does it at 3am
72	when it usually happens.
73
74	The fix: `arrs_failed_stalled_recovery.sh` runs every 6 hours. It finds all
75	`importFailed`, `importPending`, `error`, and `stalled` items, blocklists them, removes
76	them from the queue, and triggers a new search — automatically. By morning the failed
77	import has already been replaced by a working one. No manual intervention required.
78
79	---
80
81	### 🔴  Array Start Scripts Running in Wrong Order or Not at All
82
83	Startup scripts configured individually ran in an unpredictable order. The ramdisk
84	setup might run after Emby starts. The syslog filter might run after containers have
85	already created veth interfaces. PHP-FPM tuning might run after the WebGUI has already
86	served its first requests. Each script competed for the same startup slot with no
87	guaranteed order.
88
89	The fix: `array_started.sh` is the only array-start entry in the Varaverk scheduler.
90	It launches every startup script in a defined order, with one-second settle between
91	each, and reports which succeeded and which failed. Order is guaranteed. Nothing starts
92	before its dependency. Everything is visible in a single summary.
93
94	---
95
96	## ━━━ THE ORCHESTRATOR MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
97
98	The Varaverk scheduler contains exactly these entries:
99
100	```bash


Now let me check for any claims about monthly maintenance in README.md that might reference a schedule:


grep -n "monthly\|0 0 15" /boot/config/plugins/varaverk/README.md | head -20


(Bash completed with no output)


Let me check the Manual.md for any schedule claims:


grep -n "0 0 15\|monthly_maintenance" /boot/config/plugins/varaverk/Manual.md


324:| `monthly_maintenance` | `0 0 15 * *` | ZFS scrub, SMART tests (uptime-gated) |


Let me verify if this script actually exists:


find /boot/config/plugins/varaverk -name "*monthly*" -type f


Good, the file exists. Let me check its actual schedule claims and purpose:


1	#!/bin/bash
2	# ==============================================================================================
3	# ========================= Monthly Maintenance Orchestrator ===================================
4	# ==============================================================================================
5	# Uptime-triggered monthly maintenance — runs heavy tasks that need a stable, settled system.
6	# Fires only when BOTH gates pass:
7	#   1. Server uptime >= MONTHLY_UPTIME_THRESHOLD_DAYS days
8	#   2. Last run was >= MONTHLY_RUN_INTERVAL_DAYS days ago (or never run)
9	#
10	# ── WHY UPTIME-GATED ─────────────────────────────────────────────────────────────────────────
11	#   A scheduled reboot resets uptime. Monthly tasks (ZFS scrub, SMART long test) need a
12	#   stable, settled system — not one that just rebooted. Uptime-gating ensures maintenance
13	#   only runs after the server has been healthy for a full month, never immediately post-boot.
14	#   If uptime or interval gate is not met on the 15th, the run is skipped until next month.
15	#
16	# ── HOW TO CALL ──────────────────────────────────────────────────────────────────────────────
17	# Schedule: 0 0 15 * *  (15th of each month at midnight)
18	#   Silent exit 0 when either gate is not met. Only outputs when maintenance actually fires.
19	#
20	# ── STATE FILE ────────────────────────────────────────────────────────────────────────────────
21	#   MONTHLY_LAST_RUN_FILE — /boot/config — survives reboots, available before array starts.
22	#   Written after each run (pass or partial fail). Format: Unix timestamp.
23	#   A reboot does NOT reset the last-run state — the interval gate survives independently
24	#   of the uptime gate. Both must pass before maintenance fires again.
25	#
26	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
27	#   MONTHLY_MAINTENANCE_SCRIPTS    — ordered list of scripts to run
28	#   MONTHLY_UPTIME_THRESHOLD_DAYS  — minimum uptime in days before maintenance fires
29	#   MONTHLY_RUN_INTERVAL_DAYS      — minimum days since last run before running again
30	#   MONTHLY_LAST_RUN_FILE          — state file path — /boot/config, survives reboots
31	#
32	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
33	#   monthly_maintenance.sh           — normal run (uptime + interval gates enforced)
34	#   monthly_maintenance.sh --dry-run — preview gate state and scripts without running
35	#   monthly_maintenance.sh --status  — show gate state, last run, and configured scripts
36	#   monthly_maintenance.sh --force   — bypass uptime + interval gates (manual override)
37	#   monthly_maintenance.sh --log     — verbose output
38	# ==============================================================================================
39
40	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
41	ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
42
43	source "$ECOSYSTEM_ROOT/load_config.sh"
44
45	# ── Parse --force before common parse_args ────────────────────────────────────────────────────
46	FORCE_RUN=false
47	FILTERED_ARGS=()
48	for _arg in "$@"; do
49	    if [[ "$_arg" == "--force" ]]; then
50	        FORCE_RUN=true
51	    else
52	        FILTERED_ARGS+=("$_arg")
53	    fi
54	done
55	parse_args "${FILTERED_ARGS[@]}"
56
57	# ==============================================================================================
58	# ━━━ Setup ━━━
59	# ==============================================================================================
60	if [[ "$EUID" -ne 0 ]]; then


Good, that matches. Now let me check the README-User_Script_Plug-in.md for the claim about `rsync_emby_fallback` being superseded. Let me search for what it actually says about this:


grep -n -A 5 "rsync_emby_fallback\|Superseded" /boot/config/plugins/varaverk/README-User_Script_Plug-in.md | head -30


294:### ── rsync_emby_fallback ─────────────────────────────────────────────────────
295-
296:> **Superseded by `critical_sync_maintenance`.** Delete this User Script from the
297-> plugin. The emby-fallback dirty sync now runs as step 2 inside
298-> `critical_sync_maintenance.sh` every 30 minutes alongside the auth stack sync
299-> and partnership check. Running both produces duplicate syncs.
300-
301-The Emby dirty sync (`/mnt/user/Media_Server/Emby --profile=emby-fallback`) is


Perfect. Now let me cross-check what `critical_sync_maintenance.sh` actually does according to its header and the actual README listing:


grep -n "intermediate_sync_maintenance\|arrs_failed_stalled_recovery" /boot/config/plugins/varaverk/README-User_Script_Plug-in.md | head -20


205:### ── arrs_failed_stalled_recovery ───────────────────────────────────────────
209:Name:       arrs_failed_stalled_recovery
216:/boot/config/plugins/varaverk/Orchestrators/arrs_failed_stalled_recovery.sh
1637:  arrs_failed_stalled_recovery         BG: YES — blocklist + re-search failed imports


Let me check if there's an `intermediate_sync_maintenance.sh`:


find /boot/config/plugins/varaverk -name "intermediate_sync_maintenance.sh" -o -name "*intermediate*"


Good, it exists. Let me check its header:


1	#!/bin/bash
2	# ==============================================================================================
3	# =========================== Intermediate Sync Maintenance ====================================
4	# ==============================================================================================
5	# 4-hour orchestrator — arr library reconciliation, artwork fetching, and optional rsync.
6	# Schedule: 0 */4 * * * (every 4 hours)
7	#
8	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
9	#   1. conf_sync.sh --pull-only    — refresh partner conf cache in RAM (/tmp/.vv/)
10	#   2. arr_sync.sh                 — sync Lidarr/Sonarr/Radarr libraries across all nodes
11	#   3. Rsync window (optional)     — INTERMEDIATE_SYNC_SHARES, if any configured
12	#   4. INTERMEDIATE_MAINTENANCE_SCRIPTS — artwork fetch and any future 4-hour jobs
13	#
14	# ── WHY A SEPARATE ORCHESTRATOR ───────────────────────────────────────────────────────────────
15	# arr libraries need to converge more frequently than once a day. If a remote node adds
16	# something at 2am, the next daily window is 23 hours away — remote arrs search for content
17	# they don't know is already owned. Running every 4 hours closes that gap.
18	#
19	# lidarr_missing_art.sh is idempotent — skips existing files, runs fast after initial fill.
20	# Pairing it here means artwork catches up within 4 hours of a new album landing.
21	#
22	# Rsync is optional — INTERMEDIATE_SYNC_SHARES empty by default. Add shares to the config
23	# if a subset of data needs mid-day propagation (e.g. watch state, metadata). Full media
24	# share sync stays in the daily window.
25	#
26	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
27	# detect_hosts() sets MY_ID and aliases HOST*_INTERMEDIATE_SYNC_SHARES → INTERMEDIATE_SYNC_SHARES.
28	# Each server can have a different set of mid-day shares — configure in host*.conf.
29	# Each script in INTERMEDIATE_MAINTENANCE_SCRIPTS handles its own host logic.
30	#
31	# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
32	# Same as daily_sync_maintenance.sh:
33	#   exit 1 = temp WARN — skip this share, continue to next
34	#   exit 2 = temp CRITICAL — abort ALL remaining syncs in this window
35	#
36	# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
37	#   Root check           — scripts called here require root
38	#   acquire_lock         — prevents concurrent intermediate windows
39	#   check_connectivity   — verified before any rsync (skipped if no shares)
40	#   check_remote_rootfs  — aborts rsync if remote rootfs nearly full


Now let me check the README-User_Script_Plug-in.md to see if it lists `intermediate_sync_maintenance`:


grep -n "intermediate_sync_maintenance" /boot/config/plugins/varaverk/README-User_Script_Plug-in.md


So `intermediate_sync_maintenance` is NOT mentioned in README-User_Script_Plug-in.md at all! Let me verify if it's listed in the Manual.md:


grep -n "intermediate_sync_maintenance" /boot/config/plugins/varaverk/Manual.md


321:| `intermediate_sync_maintenance` | `0 */4 * * *` | arr sync + failed recovery |


Good. Now let me check what the Manual says about what `arrs_failed_stalled_recovery` is and compare to the README-User_Script_Plug-in.md claim:


205	### ── arrs_failed_stalled_recovery ───────────────────────────────────────────
206
207	```
208	Category:   Orchestrators
209	Name:       arrs_failed_stalled_recovery
210	Schedule:   0 */6 * * *   (every 6 hours)
211	Background: YES
212	```
213
214	```bash
215	#!/bin/bash
216	/boot/config/plugins/varaverk/Orchestrators/arrs_failed_stalled_recovery.sh
217	```
218
219	**What it does:**
220	Automatically recovers failed imports and stalled downloads across Sonarr, Radarr,
221	and Lidarr. A release that downloaded but failed to import sits in the queue
222	permanently without this — no arr retries importFailed items on its own.
223
224	**What it finds and acts on:**
225	```
226	importFailed   — downloaded but couldn't import → blocklist + remove + new search
227	importPending  — stuck in import queue → blocklist + remove + new search
228	error          — serious failure state → blocklist + remove + new search
229	stalled        — no seeders or no progress → blocklist + remove + new search
230	```
231
232	**What it never touches:** items with status "downloading" or "imported".
233	Age gate: items newer than ARR_IMPORT_RECOVERY_AGE (default: 6 hours) are skipped —
234	the arr may still retry on its own.
235
236	**Result:** failed imports are automatically replaced while you sleep.
237
238	---
239


Now I need to check the actual description in the Manual.md for the intermediate_sync table entry:


312	### ── Schedule (Varaverk Scheduler) ───────────────────────────────────────────
313
314	| Script | Event/Cron | Purpose |
315	|--------|-----------|---------|
316	| `array_started` | `array_start` | All array startup scripts in order |
317	| `array_stopping` | `array_stop` | Ordered graceful shutdown |
318	| `transcode_management` | `*/7 * * * *` | Cleanup then manager — order critical |
319	| `watchdog_orchestrator` | `*/15 * * * *` | resource → docker → system → stability |
320	| `critical_sync_maintenance` | `*/30 * * * *` | auth + Emby dirty sync + partnership |
321	| `intermediate_sync_maintenance` | `0 */4 * * *` | arr sync + failed recovery |
322	| `daily_sync_maintenance` | `0 1 * * *` | full daily maintenance window |
323	| `weekly_sync_maintenance` | `30 2 * * 0` | clean sync + image updates |
324	| `monthly_maintenance` | `0 0 15 * *` | ZFS scrub, SMART tests (uptime-gated) |
325
326	Concurrent runs are prevented by `acquire_lock`. For the complete schedule see
327	[README-Orchestrators.md](Orchestrators/README-Orchestrators.md).
328
329	---
330
331	## ━━━ STEP 6: INITIAL SYNC ━━━


Now let me verify if the intermediate_sync_maintenance actually contains arrs_failed_stalled_recovery or arr_sync or both. Let me read the full file:


head -100 /boot/config/plugins/varaverk/Orchestrators/intermediate_sync_maintenance.sh | tail -50


#   intermediate_sync_maintenance.sh              — normal run
#   intermediate_sync_maintenance.sh --dry-run    — preview without changes
#   intermediate_sync_maintenance.sh --log        — verbose per-job output
#   intermediate_sync_maintenance.sh --status     — show configured shares/jobs and exit
# ==============================================================================================

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

source "$SCRIPT_DIR/../load_config.sh"

RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
SCRIPTS_ROOT="$SCRIPT_DIR/.."

parse_args "$@"

# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
    error "Must be run as root"
    exit 1
fi

acquire_lock

detect_hosts
resolve_remote_ip

[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"

# ── Helper — run a maintenance job, track pass/fail ───────────────────────────────────────────
run_job() {
    local script_entry="$1"
    local extra_dry=""
    [[ "$DRY_RUN" == true ]] && extra_dry="--dry-run"

    read -r -a script_args <<< "$script_entry"
    local script_path="$SCRIPTS_ROOT/${script_args[0]}"
    local script_name
    script_name=$(basename "${script_args[0]}")
    local extra_args=("${script_args[@]:1}")

    if [[ ! -f "$script_path" ]]; then
        error "$script_name — not found at $script_path"
        JOB_FAIL+=("$script_name")
        return 1
    fi

    log "Running: $script_name ${extra_args[*]}"


Now let me see the actual job execution part:


grep -n "INTERMEDIATE_MAINTENANCE_SCRIPTS\|arr_sync\|arrs_failed" /boot/config/plugins/varaverk/Orchestrators/intermediate_sync_maintenance.sh | head -20


10:#   2. arr_sync.sh                 — sync Lidarr/Sonarr/Radarr libraries across all nodes
12:#   4. INTERMEDIATE_MAINTENANCE_SCRIPTS — artwork fetch and any future 4-hour jobs
29:# Each script in INTERMEDIATE_MAINTENANCE_SCRIPTS handles its own host logic.
41:#   Non-fatal jobs       — a failed arr_sync warns but does not block rsync or artwork fetch
47:#   master.conf:         INTERMEDIATE_MAINTENANCE_SCRIPTS — jobs run after rsync
48:#   master.conf:         ARR_SYNC_ENABLED               — toggle inside arr_sync.sh
133:    if [[ ${#INTERMEDIATE_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
136:        for entry in "${INTERMEDIATE_MAINTENANCE_SCRIPTS[@]}"; do
182:ARR_SYNC_SCRIPT="$SCRIPTS_ROOT/Media/arr_sync.sh"
186:    warn "arr_sync.sh not found at $ARR_SYNC_SCRIPT — skipping"
187:    JOB_FAIL+=("arr_sync.sh")
189:    _arr_sync_args=()
190:    [[ "$DRY_RUN" == true ]] && _arr_sync_args+=("--dry-run")
191:    if bash "$ARR_SYNC_SCRIPT" "${_arr_sync_args[@]}"; then
193:        JOB_PASS+=("arr_sync.sh")
196:        JOB_FAIL+=("arr_sync.sh")
198:    unset _arr_sync_args
273:if [[ ${#INTERMEDIATE_MAINTENANCE_SCRIPTS[@]} -gt 0 ]]; then
276:    for script_entry in "${INTERMEDIATE_MAINTENANCE_SCRIPTS[@]}"; do


So `intermediate_sync_maintenance` does NOT call `arrs_failed_stalled_recovery.sh` directly - it only calls `arr_sync.sh` and scripts from the `INTERMEDIATE_MAINTENANCE_SCRIPTS` array! Let me check what the description in the Manual says:

The Manual.md line 321 says:
```
| `intermediate_sync_maintenance` | `0 */4 * * *` | arr sync + failed recovery |
```

But the actual script only calls `arr_sync.sh`, and `arrs_failed_stalled_recovery.sh` is a SEPARATE scheduled orchestrator that runs every 6 hours. Let me verify the schedule of arrs_failed_stalled_recovery:


Error: Exit code 1
head: cannot open '/boot/config/plugins/varaverk/Orchestrators/arrs_failed_stalled_recovery.sh' for reading: No such file or directory


Interesting! Let me find the correct name:


find /boot/config/plugins/varaverk -name "*failed*" -o -name "*stalled*" -o -name "*recovery*" 2>/dev/null


/boot/config/plugins/varaverk/data/arr_recovery_stats.db
/boot/config/plugins/varaverk/Media/arrs_failed_stalled_recovery.sh
/boot/config/plugins/varaverk/State_Files/docker_watchdog_failed.db


Ah! The script is in `Media/` not `Orchestrators/`. Let me check its header:


1	#!/bin/bash
2	# ==============================================================================================
3	# ========================= Arrs Failed / Stalled Recovery =====================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Detect and recover failed imports and stalled downloads across Sonarr, Radarr,
9	# and Lidarr. Blocklists the bad release and triggers a re-search — hands-free
10	# overnight recovery.
11	#
12	# ==============================================================================================
13	# OPERATIONAL MODEL
14	# ==============================================================================================
15	#
16	# Four problem types detected from the arr queue API:
17	#   importFailed   — downloaded but arr couldn't import the file
18	#   importPending  — downloaded, stuck waiting to import (will not self-resolve)
19	#   error status   — serious failure not covered by the above two states
20	#   stalled        — download stuck with no connections or no progress
21	#
22	# Never touches items with state "downloading" or "imported" — safe to run anytime.
23	# Items newer than ARR_IMPORT_RECOVERY_AGE are skipped — gives arr time to retry first.
24	#
25	# Per problem item (3-step response):
26	#   1. Blocklist the release  — prevents re-grabbing the same bad release
27	#   2. Remove from queue      — cleans up the failed item
28	#   3. Trigger new search     — finds a different release automatically
29	#
30	# ==============================================================================================


Now let me check where this script is actually meant to be called from:


grep -r "arrs_failed_stalled_recovery" /boot/config/plugins/varaverk/Orchestrators/ 2>/dev/null


/boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md:The fix: `arrs_failed_stalled_recovery.sh` runs every 6 hours. It finds all
/boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md:| `arrs_failed_stalled_recovery.sh` | Failed import + stalled download recovery | Every 6 hours |
/boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md:## 🔄  arrs_failed_stalled_recovery.sh
/boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md:arrs_failed_stalled_recovery.sh              # normal run
/boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md:arrs_failed_stalled_recovery.sh --dry-run    # show what would be actioned, no changes
/boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md:arrs_failed_stalled_recovery.sh --log        # verbose — show each queue item evaluated
/boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md:arrs_failed_stalled_recovery.sh --status     # show arr config and API connectivity


So README-Orchestrators says it runs every 6 hours as a standalone. Let me check if there's an orchestrator wrapper for it:


ls -la /boot/config/plugins/varaverk/Orchestrators/*.sh | grep -E "arrs|failed|recovery"


(Bash completed with no output)


So there is NO orchestrator for `arrs_failed_stalled_recovery.sh`! But README-User_Script_Plug-in.md claims it's scheduled as an Orchestrator at 6 hours. Let me verify what the README-Orchestrators actually says about the full schedule:


grep -A 50 "THE ORCHESTRATOR MODEL" /boot/config/plugins/varaverk/Orchestrators/README-Orchestrators.md | head -70