Read the following files and report any discrepancies between the documentation and the actual project structure/script headers. The goal is to find places where the root README or Manual says something that no longer matches what the scripts actually do (wrong schedules, wrong config vars, missing/extra scripts referenced, wrong directory names, etc.).

Files to read:
1. /boot/config/plugins/varaverk/README.md
2. /boot/config/plugins/varaverk/Manual.md
3. /boot/config/plugins/varaverk/README-User_Script_Plug-in.md
4. For spot-checking claims, read headers of:
   - /boot/config/plugins/varaverk/load_config.sh (first 50 lines)
   - /boot/config/plugins/varaverk/common.sh (first 60 lines)
   - /boot/config/plugins/varaverk/git_pull_execute.sh (first 60 lines)

Report: list each discrepancy as "FILE: what the doc says vs what is actually true". If everything matches, say so. Be specific — quote the conflicting text. Keep the report under 500 words.


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.
- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch

update-config keybindings-help verify code-review simplify fewer-permission-prompts loop schedule claude-api run init review security-review


I'll read these files to identify any discrepancies between documentation and actual implementation.


1	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2	# 🏠  VARAVERK
3	# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
4
5	**A complete self-healing, self-maintaining, mutually-redundant two-server home server
6	ecosystem.** One codebase runs on both servers. Both servers run their own lives
7	independently. When one goes down the other covers it — not because of conventional
8	fallback infrastructure, but because both servers already have copies of everything
9	that matters and both know how to run it.
10
11	> **This is not conventional failover.** There is no primary and standby. There is no
12	> shared storage. There is no cluster manager. There are two fully independent servers,
13	> living in different locations on different power utilities, each running their own
14	> household's media stack — and a layer of scripts that keeps them synchronised,
15	> coordinated, and mutually covering. The goal is minimal disruption to a media stack,
16	> not enterprise high availability.
17
18	**Started on Unraid, built for portability.** The ecosystem runs on Unraid today. The
19	platform adapter layer (`Plugin/unraid/adapter.sh`) isolates every OS-specific call so
20	bash scripts stay clean. A `Plugin/truenas/adapter.sh` or `Plugin/ubuntu/adapter.sh`
21	provides the same function names — `load_config.sh` detects the OS and sources the
22	correct adapter. The scripts themselves never branch on the OS.
23
24	---
25
26	## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
27
28	```
29	HOST1 — unRAID-Gmer4Lfe
30	  Hardware:  Threadripper 1950X, 128GB RAM, ZFS cache pools
31	  Location:  Primary site
32	  Domain:    Gmer4Lfe.com
33	  Arrs:      Movies, Tv_Shows, Music — union with HOST2 via arr_sync
34	             Auth stack — source of truth (HOST1 owned, syncs to HOST2)
35	             Emby — primary media server
36
37	HOST2 — unRAID-Jayred365
38	  Hardware:  Intel i5 10th gen, 64GB RAM
39	  Location:  Remote — different building, different power utility
40	  Domain:    Gmer4Lfe.us
41	  Arrs:      Anime_Shows, Anime_Movies — union with HOST1 via arr_sync
42	             His own Emby — for his household
43	```
44
45	**Hardware does not need to match.** Everything is accessed through `/mnt/user/` —
46	unRAID's unified share layer. HOST1 has a Threadripper with ZFS. HOST2 has a completely
47	different CPU and fewer drives. A fallback container on HOST2 mounts `/mnt/user/Movies`
48	and sees mirrored data because the share names match. The hardware underneath is
49	irrelevant.
50
51	---
52
53	## ━━━ CONTAINER NAMING — YOU NEED TO KNOW THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
54
55	This is the most important convention in the ecosystem. Get it wrong and the scripts
56	break silently.
57
58	---
59
60	### ── Shared Containers — No Suffix ───────────────────────────────────────────
61
62	```bash
63	# ─────────────────────────────────────────────────────────────────────────────
64	# Shared containers — meaning both servers run this container and scripts treat
65	# them as the same logical service — use the plain name with no suffix.
66	#
67	# The convention: "no suffix = this container participates in the shared stack"
68	#
69	    Emby                    # the shared Emby — runs on HOST1, picked up by HOST2 on fallback
70	    NginxProxyManager       # shared NPM — runs on both, same config via sync
71	    Lldap-Gmer4Lfe          # shared LLDAP — same user directory, both servers
72	    Authelia                # shared auth — both servers, same config
73	    Mariadb-Authelia        # Authelia's database — same on both
74	    Redis-Authelia          # Authelia's cache — same on both
75	    VaultWarden             # shared password manager
76	    Dispatcharr             # shared Live TV scheduler
77	```
78
79	When `fallback.sh` on HOST2 reads `FALLBACK_HOST2_COVERS_HOST1_TIER1=(..."Emby"...)`,
80	it looks for a container called exactly `Emby` and starts it. HOST2 must have a
81	container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
82	its volume mounts pointing at the mirrored share paths. This is what makes one
83	codebase work on both servers without modification.
84
85	---
86
87	### ── User-Specific Containers — Suffixed ────────────────────────────────────
88
89	```bash
90	# ─────────────────────────────────────────────────────────────────────────────
91	# User-specific containers — things that belong to one person and only ever
92	# run on their server — use a suffix identifying the user/owner.
93	#
94	    Emby-Gmer4Lfe           # Gmer4Lfe's personal Emby instance (HOST1, never covered)
95	    Emby-Jayred365          # Jayred365's personal Emby instance (HOST2, never covered)
96	    Organizer-Gmer4Lfe      # Gmer4Lfe's dashboard — personal, stays on HOST1
97	    Sonarr-Gmer4Lfe         # Gmer4Lfe's Sonarr — managing his Tv_Shows share
98	    Radarr-Gmer4Lfe         # Gmer4Lfe's Radarr — managing his Movies share
99	    Immich-Gmer4Lfe         # Gmer4Lfe's photo library — personal
100	```
101
102	User-specific containers are covered during fallback (see Tiered Fallback below) but
103	are distinct from the shared stack. They are never confused with each other because
104	the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts targeting
105	`Emby-Gmer4Lfe` are targeting different containers for different purposes.
106
107	---
108
109	### ── Why This Matters ─────────────────────────────────────────────────────────
110
111	```bash
112	# ─────────────────────────────────────────────────────────────────────────────
113	# The entire ecosystem uses container names as identifiers.
114	# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
115	#   → looks for exactly "NginxProxyManager" on the running server
116	# fallback.sh: FALLBACK_HOST2_COVERS_HOST1_TIER1=("Emby" "VaultWarden")
117	#   → looks for exactly "Emby" and "VaultWarden" on HOST2
118	# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
119	#   → looks for exactly those names in docker ps
120	#
121	# A container named differently than expected is not found — no error, no restart,
122	# no fallback coverage. It silently does not happen.
123	# Keep names consistent on both servers for shared containers.
124	# ─────────────────────────────────────────────────────────────────────────────
125	```
126
127	---
128
129	## ━━━ SHARE NAMING — ALSO YOU NEED TO KNOW THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
130
131	Shares must have **identical names on both servers**. The ecosystem uses share paths
132	as sync targets and mount points — a mismatch is a broken path.
133
134	```bash
135	# ─────────────────────────────────────────────────────────────────────────────
136	# The rule: if a share exists on HOST1 and HOST2 needs a mirror of it,
137	# the share on HOST2 must have the exact same name.
138	#
139	# Same name on both servers:
140	    /mnt/user/Movies          # both servers — arr_sync.sh keeps Radarr in union
141	    /mnt/user/Tv_Shows        # both servers — arr_sync.sh keeps Sonarr in union
142	    /mnt/user/Music           # both servers — arr_sync.sh keeps Lidarr in union
143	    /mnt/user/Anime_Shows     # both servers — arr_sync.sh keeps Sonarr in union
144	    /mnt/user/Anime_Movies    # both servers — arr_sync.sh keeps Radarr in union
145	#
146	# rsync.sh syncs /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2.
147	# If HOST2 has it at /mnt/user/Movies-Mirror → rsync aborts: path not found.
148	# ─────────────────────────────────────────────────────────────────────────────
149	```
150
151	---
152
153	## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
154
155	### ── Common Libraries — Shared Pool ─────────────────────────────────────────
156
157	Common media like Movies, Music, and TV Shows are a **shared pool**. Every container
158	that needs music — Emby, LidaTube, Navidrome, whatever — mounts the same `/mnt/user/Music`
159	share. There is one share, one location, one source of truth. No duplication of content
160	across separate shares per-user or per-application.
161
162	```bash
163	# ─────────────────────────────────────────────────────────────────────────────
164	# One share for the library — all containers that need it mount the same path.
165	# Emby container: /mnt/user/Music → /media/Music (read)
166	# Lidarr:         /mnt/user/Music → /music       (read+write)
167	# LidaTube:       /mnt/user/Music → /downloads   (write)
168	# Navidrome:      /mnt/user/Music → /music       (read)
169	#
170	# All pointing at the same /mnt/user/Music. No per-user music copies.
171	# When Lidarr downloads and Emby scans — they're looking at the same files.
172	# ─────────────────────────────────────────────────────────────────────────────
173	```
174
175	---
176
177	### ── Both Arrs Manage All Shares — arr_sync.sh Is the Source of Truth ───────
178
179	```bash
180	# ─────────────────────────────────────────────────────────────────────────────
181	# Both servers run Sonarr, Radarr, and Lidarr — all pointing at the same
182	# share names. Either server can download anything to any share at any time.
183	#
184	# How it stays coherent:
185	#   arr_sync.sh runs first (before rsync) — bidirectional library union.
186	#   Both Sonarr instances know about every TV episode on either server.
187	#   Both Radarr instances know about every movie on either server.
188	#   Both Lidarr instances know about every album on either server.
189	#
190	# Then rsync spreads the files additively (no --delete).
191	# Any file downloaded on HOST1 propagates to HOST2 on the next cycle.
192	# Any file downloaded on HOST2 propagates to HOST1 on the next cycle.
193	#
194	# HOST1 downloads Tv_Shows episode:
195	#   → HOST1 Sonarr tracks it immediately
196	#   → arr_sync.sh runs → HOST2 Sonarr now tracks it
197	#   → daily rsync → file lands on HOST2
198	#
199	# HOST2 downloads the same show's next episode that same night:
200	#   → HOST2 Sonarr tracks it immediately
201	#   → arr_sync.sh runs → HOST1 Sonarr now tracks it
202	#   → daily rsync → file lands on HOST1
203	#
204	# Both servers converge. Neither owns the share. The union is the truth.
205	# ─────────────────────────────────────────────────────────────────────────────
206	```
207
208	---
209
210	### ── How arr_sync.sh Keeps the Union Stable ────────────────────────────────
211
212	```bash
213	# ─────────────────────────────────────────────────────────────────────────────
214	# arr_sync.sh runs as the fixed first step of intermediate_sync_maintenance.sh
215	# (every 4 hours). It syncs all arr libraries bidirectionally before any rsync.
216	#
217	# What it does:
218	#   Reads each arr's full library via the local API
219	#   SSHes to the remote, reads the remote arr library via its API
220	#   Adds anything missing on either side — union, not overwrite
221	#   Blocked items (blocklist) are excluded from sync
222	#
223	# What this means in practice:
224	#   No share has an owner. No server is read-only.
225	#   Either server can download content to any share at any time.
226	#   arr_sync.sh guarantees both arrs agree on what exists before files move.
227	#   rsync then spreads the files to make storage match what arrs already know.
228	#
229	# arr_cleanup.sh (daily) removes true orphans — files no arr tracks.
230	# arr_cleanup uses the union model too: a file is only an orphan if
231	# neither arr on either server has it indexed.
232	# ─────────────────────────────────────────────────────────────────────────────
233	```
234
235	---
236
237	## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
238
239	Not everything needs to sync. Not everything should sync. Understanding what goes
240	where determines whether you need encryption and whether sync failure matters.
241
242	---
243
244	### ── Shared Fallback Shares — Sync Required ─────────────────────────────────
245
246	```bash
247	# ─────────────────────────────────────────────────────────────────────────────
248	# These shares enable fallback — HOST2 must have current copies for the covered
249	# services to work correctly when they start on HOST2.
250	#
251	    /mnt/user/Media_Server/Emby              # Emby userdata — watch history, playstates
252	    /mnt/user/appdata-Fallback/Critical-Data # auth stack — NPM rules, certs, user accounts
253	    /mnt/user/Movies                         # media library content
254	    /mnt/user/Tv_Shows                       # media library content
255	    /mnt/user/Music                          # media library content
256	#
257	# Sync failure here = fallback runs but with stale data (acceptable) or missing
258	# data (problematic). These sync every night via daily_sync_maintenance.sh.
259	# Critical-Data and Emby also sync every 30 minutes via critical_sync_maintenance.sh.
260	# ─────────────────────────────────────────────────────────────────────────────
261	```
262
263	---
264
265	### ── Non-Fallback Shares — Sync as Backup, No Encryption Required ───────────
266
267	```bash
268	# ─────────────────────────────────────────────────────────────────────────────
269	# Some shares don't participate in fallback — their container only runs on one
270	# server and is covered during fallback from scratch (or not at all).
271	# These shares can sync as offsite backup without encryption.
272	# Content is not sensitive — anyone with access to HOST2 could read it.
273	#
274	# Examples:
275	    /mnt/user/appdata-Fallback/Gmer4Lfe      # Gmer4Lfe server appdata backup
276	    /mnt/user/appdata-Fallback/Important-Data # NextCloud, Immich appdata
277	    /mnt/user/Kids_Movies                    # kids content — not sensitive
278	    /mnt/user/Sports                         # sports content — not sensitive
279	#
280	# These sync in DAILY_SYNC_SHARES — straight rsync, no special handling.
281	# If HOST2 is compromised, the content is readable. That is acceptable.
282	# ─────────────────────────────────────────────────────────────────────────────
283	```
284
285	---
286
287	### ── Personal Shares — Always Encrypted ─────────────────────────────────────
288
289	```bash
290	# ─────────────────────────────────────────────────────────────────────────────
291	# Personal shares contain content only the owner should see — documents,
292	# photos, financial records, private files.
293	# ZFS encrypts at the dataset level — HOST2 receives encrypted blocks.
294	# Without your passphrase or keyfile, HOST2 cannot read the content.
295	#
296	    HOST1_PERSONAL_SHARES=(
297	        "/mnt/user/Gmer4Lfe-Personal"    # encrypted ZFS dataset
298	    )
299	    HOST2_PERSONAL_SHARES=(
300	        "/mnt/user/Jayred365-Personal"   # encrypted ZFS dataset
301	    )
302	#
303	# Append to HOST*_DAILY_SYNC_SHARES — synced nightly with everything else.
304	# Same rsync, same timing — but the content on HOST2 is ciphertext only you can unlock.
305	#
306	# Setup: README-Rsync_Setup.md → Step 8 — Personal Encrypted Shares
307	# ─────────────────────────────────────────────────────────────────────────────
308	```
309
310	---
311
312	## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
313
314	> **Understanding this distinction prevents a lot of confusion.**
315
316	Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a
317	primary server and a standby server with shared storage or synchronous replication.
318	The standby exists only to take over when the primary fails. It runs nothing
319	independently. Its purpose is 100% failover coverage with no service interruption.
320
321	**This is not that.**
322
323	---
324
325	### ── What This Actually Is ─────────────────────────────────────────────────────
326
327	```
328	Two fully independent servers, each running their own household's services.
329	Each server has its own domain, its own users, its own media library management.
330	Each server is the primary for its own services every single day.
331
332	The "fallback" component is additive:
333	  When HOST1 goes down → HOST2 starts HOST1's containers on top of its own
334	  When HOST2 comes back → HOST1's containers hand back cleanly
335	  HOST2 never stopped running its own services during this entire time
336
337	This is not standby. This is mutual coverage between peers.
338	```
339
340	---
341
342	### ── The Actual Goal: Minimal Disruption to a Media Stack ────────────────────
343
344	```bash
345	# ─────────────────────────────────────────────────────────────────────────────
346	# "Minimal disruption" means:
347	#   Live TV keeps playing         → Dispatcharr is Tier 1 (starts immediately)
348	#   Passwords are accessible      → VaultWarden is Tier 1 (people lock themselves out)
349	#   Emby keeps serving            → Emby is Tier 1
350	#   Auth works                    → NPM, Authelia, LLDAP already warm on both servers
351	#
352	# "Minimal disruption" does NOT mean:
353	#   Zero downtime (DDNS TTL = 1 minute, brief gap expected)
354	#   Perfect consistency (Emby userdata is 30min stale at most)
355	#   All services covered immediately (tiered start over 24 hours)
356	#
357	# The system is designed around the real experience of a household:
358	#   "The TV stopped working" → unacceptable, fixed in Tier 1 (< 2 minutes)
359	#   "NextCloud is down"      → annoying, fixed in Tier 2 (4 hours)
360	#   "Gitea is down"          → unacceptable, fixed in Tier 1 (< 2 minutes)
361	#   "Sonarr isn't downloading" → fine, fixed in Tier 4 (24 hours)
362	# ─────────────────────────────────────────────────────────────────────────────
363	```
364
365	---
366
367	### ── What You Give Up With This Approach ─────────────────────────────────────
368
369	```bash
370	# ─────────────────────────────────────────────────────────────────────────────
371	# Things conventional failover provides that this does NOT:
372	#   Zero RPO (recovery point objective) — there is ~30min of potential staleness
373	#   Synchronous state replication — changes don't reach HOST2 instantly
374	#   Automatic failure detection in seconds — fallback.sh checks every 2 minutes
375	#   Transparent IP migration — DNS TTL means up to 1 minute of routing to old IP
376	#
377	# What you get instead:
378	#   Both servers run fully independently — no single point of failure in software
379	#   Geographic redundancy — different buildings, different power, different ISP
380	#   Both households stay operational during most disruption scenarios
381	#   Simple, auditable scripts — no cluster manager black box
382	# ─────────────────────────────────────────────────────────────────────────────
383	```
384
385	---
386
387	## ━━━ HOW FALLBACK ACTUALLY WORKS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
388
389	`fallback.sh` runs on both servers as a continuous background process. Every 2 minutes
390	it pings two things: the other server's Tailscale IP, and `8.8.8.8`. The combination
391	of those two answers determines the current state.
392
393	```
394	Remote ping PASS + Internet PASS → NORMAL — both servers running independently
395
396	Remote ping FAIL + Internet PASS → FALLBACK — remote is down, start covering it
397	  Tier 1 immediately: DDNS + Emby + Auth + VaultWarden + Live TV + Gitea
398	  Tier 2 at 4 hours: NextCloud + Immich + Jellyseerr
399	  Tier 3 at 12 hours: AdGuard + Uptime monitoring
400	  Tier 4 at 24 hours: Arrs + download clients
401
402	Internet FAIL (either) → NO_INTERNET — stop own DDNS, wait
403	  Can't serve anyone without internet — stop claiming DNS ownership
404
405	Remote FAIL + Internet FAIL → DARK — conservative, same as NO_INTERNET
406	```
407
408	The DDNS sequencing is critical — one domain, one DDNS container, at all times. The
409	handback sequence stops remote DDNS first, rsyncs, starts containers, starts local
410	DDNS last. Split-brain DNS is prevented by design.
411
412	**→ Full detail: [README-Fallback.md](Fallback/README-Fallback.md)**
413
414	---
415
416	## ━━━ THE AUTH STACK — WARM ON BOTH SERVERS ALWAYS ━━━━━━━━━━━━━━━━━━━━━━━━━━
417
418	NPM, LLDAP, Authelia, MariaDB, Redis — all running actively on both servers at all
419	times. Not cold standby. Not started at fallback. Running right now, serving each
420	server's own domain.
421
422	```bash
423	# ─────────────────────────────────────────────────────────────────────────────
424	# Why this eliminates the fallback dead zone:
425	#
426	# Cold standby approach: HOST1 goes down → DNS switches to HOST2 → HOST2 starts
427	# auth containers → 30-60 seconds before auth is ready → users hit the auth wall
428	# before the service behind it responds → failed logins, broken sessions
429	#
430	# Warm approach: HOST1 goes down → DNS switches to HOST2 → auth is ALREADY
431	# running → users are served immediately with no dead zone
432	#
433	# HOST1 is source of truth for the auth configuration. All changes to NPM proxy
434	# rules, LLDAP users, Authelia policies, and certs happen on HOST1 and propagate
435	# to HOST2 via Critical-Data sync every 30 minutes.
436	#
437	# HOST2 operator's experience: click NginxProxyManager in Docker UI →
438	# browser opens HOST1's NPM via Tailscale redirect → make changes there →
439	# 30 minutes later live on both servers. Zero auth management overhead on HOST2.
440	# ─────────────────────────────────────────────────────────────────────────────
441	```
442
443	**→ Full detail: [README-Partnership.md](Partnership/README-Partnership.md)**
444
445	---
446
447	## ━━━ THE COMPLETE PICTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
448
449	```
450	                    HOST1 (Gmer4Lfe.com)        HOST2 (Gmer4Lfe.us)
451	                    ──────────────────          ──────────────────────
452
453	Always running:
454	  Own stack       Full service stack            Full service stack (anime focus)
455	  Auth            NPM, Authelia, LLDAP          NPM, Authelia, LLDAP (mirrored)
456
457	Source of truth:
458	  Media           Movies, Tv_Shows, Music  ←──  Anime_Movies, Anime_Shows
459	                  ───────────────────────→       (bidirectional daily push)
460	  Config          auth stack config        ──→   mirrored every 30 min
461	  Emby state      watch history, playstates──→   dirty sync every 30 min
462
463	Fallback coverage:
464	  If HOST1 down:  —                             starts Emby, VaultWarden,
465	                                                Dispatcharr, all arr stack
466	                                                serves Gmer4Lfe.com via DDNS
467
468	  If HOST2 down:  starts his Emby, his arrs,   —
469	                  serves Gmer4Lfe.us via DDNS
470
471	Monitoring:
472	  Both servers    resource/docker/system/        resource/docker/system/
473	                  stability_watchdog.sh         stability_watchdog.sh
474	                  fallback.sh                   fallback.sh
475	                  Sunday morning coffee report  Sunday morning coffee report
476	```
477
478	---
479
480	## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
481
482	```
483	varaverk/
484	│
485	├── Configurations/          ← All configuration files — the only files you edit regularly
486	│   ├── master.conf          ← Shared: thresholds, toggles, job lists, watchdog settings
487	│   ├── host1.conf           ← HOST1-specific: share lists, container names, API keys
488	│   └── host2.conf           ← HOST2-specific: same structure, different values
489	├── common.sh                ← Shared library — all functions used by every script
490	├── load_config.sh           ← Detects OS platform → sources Configurations/, common.sh, adapter
491	│
492	├── Plugin/                  ← Platform adapters + Unraid web UI
493	│   └── unraid/
494	│       ├── adapter.sh       ← Unraid adapter: platform_*() functions (rc.d, emhttp, dynamix)
495	│       ├── api/             ← PHP API endpoints
496	│       ├── include/         ← PHP business logic
497	│       ├── pages/           ← Per-tab page includes
498	│       └── ...              ← Full web UI (scheduler, monitor, docker, arrs, fallback)
499	│
500	├── Orchestrators/           ← Scheduled job runners — what Varaverk scheduler executes
501	│   README: README-Orchestrators.md
502	│
503	├── Rsync/                   ← Core rsync script — called per share with profile
504	│   README: README-Rsync_Setup.md
505	│
506	├── Fallback/                ← Mutual automatic fallback — continuous background process
507	│   README: README-Fallback.md
508	│
509	├── Watchdogs/               ← All watchdog scripts: resource, docker, system, stability
510	│   README: README-Watchdogs.md
511	│
512	├── Docker_Essentials/       ← Container lifecycle: restarts, updates, networks
513	│   README: README-Docker_Essentials.md
514	│
515	├── System_Essentials/       ← Server-level: WebGUI recovery, log hygiene, kernel tuning, reboot
516	│   README: README-System_Essentials.md
517	│
518	├── Media/                   ← Library health + behavior-driven discovery: permissions, junk cleanup, orphan removal, weekly arr adds
519	│   README: README-Media.md
520	│
521	├── Transcodes/              ← Ramdisk transcode management with SSD fallback
522	│   README: README-Transcoding.md
523	│
524	├── Monitors/                ← Read-only observation: certs, SMART, bandwidth, digests
525	│   README: README-Monitors.md
526	│
527	├── Partnership/             ← Two-server relationship lifecycle: onboard, offboard, transfer
528	│   README: README-Partnership.md
529	│
530	├── Kernel/                  ← Kernel module config — loaded at boot by go file
531	│   README: README-Kernel.md
532	│
533	└── Tools/                   ← Situational utilities: repair, export, emergency tools
534	    README: README-Tools.md
535	```
536
537	---
538
539	## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━
540
541	```bash
542	# ─────────────────────────────────────────────────────────────────────────────
543	# All three conf files live in Configurations/. load_config.sh sources them.
544	# Sparse checkout ensures each server only receives its own host*.conf.
545	#
546	# load_config.sh does five things in order:
547	#   1. Detects OS platform (/etc/unraid-version → PLATFORM=unraid)
548	#   2. Sources master.conf
549	#   3. Auto-discovers and sources all host*.conf present
550	#   4. Sources common.sh
551	#   5. Sources Plugin/$PLATFORM/adapter.sh (platform_*() functions)
552	#
553	# master.conf — shared across both servers
554	#   Everything that applies equally to both: thresholds, schedules, profile
555	#   definitions, watchdog settings, arr cleanup config, DDNS timing, etc.
556	#   Pushed to both servers via git. Never contains server-specific values.
557	#
558	# host1.conf — sourced only on HOST1
559	#   HOST1_* prefixed variables: share lists, container names, API keys,
560	#   ramdisk size, specific paths, per-server toggle overrides.
561	#
562	# host2.conf — sourced only on HOST2
563	#   HOST2_* prefixed variables: same structure, different values.
564	#
565	# detect_hosts() in common.sh:
566	#   Reads the current server's hostname. Matches it against HOST1 and HOST2.
567	#   Aliases HOST1_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST1)
568	#   Aliases HOST2_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST2)
569	#   Every script uses the unprefixed name. The same script on either server
570	#   automatically uses the correct values. No hostname comparisons in scripts.
571	#
572	# Plugin/$PLATFORM/adapter.sh — OS-specific function implementations
573	#   Scripts call platform_restart_service(), platform_storage_healthy(), etc.
574	#   The adapter maps those to rc.d scripts, emhttp commands, and dynamix files
575	#   on Unraid — or to systemctl, mount checks, etc. on a future platform.
576	# ─────────────────────────────────────────────────────────────────────────────
577	```
578
579	---
580
581	## ━━━ WHAT RUNS WHEN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
582
583	The Varaverk scheduler runs only orchestrators — never individual scripts directly.
584
585	```
586	Array start (Varaverk event hook):
587	  array_started.sh              launches all startup scripts in order
588	    → inotify_tuning.sh       raise kernel limits before containers start
589	    → docker_syslog_filter.sh suppress veth noise before containers create interfaces
590	    → php_fpm_max_children.sh WebGUI tuning before first request
591	    → ramdisk_setup.sh        create ramdisk before Emby starts
592	    → docker_network_connect.sh connect containers to extra networks
593	    → fallback.sh             continuous — mutual fallback state machine
594
595	Every 15 minutes:
596	  watchdog_orchestrator.sh    fires each watchdog in sequence
597	    → resource_watchdog.sh    reduce pressure before healing attempts
598	    → docker_watchdog.sh      two-tier container healing
599	    → system_watchdog.sh      system components: storage + WebGUI health
600	    → stability_watchdog.sh   last resort — reboots when all else fails
601
602	Every 7 minutes:
603	  transcode_management.sh     cleanup → manager (order non-negotiable)
604
605	Every 30 minutes:
606	  critical_sync_maintenance.sh  Critical-Data + Emby dirty sync + partnership check
607
608	Every 30 minutes:
609	  rsync.sh --profile=emby-fallback   Emby watch state sync (separate cron entry)
610
611	Every 6 hours:
612	  arrs_failed_stalled_recovery.sh   blocklist + re-search failed imports
613
614	1am daily:
615	  daily_sync_maintenance.sh   git pull → sync shares → media maintenance → restarts
616
617	2:30am Sunday:
618	  weekly_sync_maintenance.sh  stop → update images → clean sync → start → weekly restarts
619	                              + playback_aware_lidarr/radarr/sonarr_discovery.sh  (behavior-driven arr discovery)
620
621	8am daily:
622	  weekly_health_digest.sh     DIGEST_PROFILE controls when it actually notifies
623
624	Every 6 hours:
625	  system_tuning_monitor.sh    inotify + php-fpm utilisation snapshot
626
627	Sunday morning block (6–11am):
628	  zfs_memory_snapshot.sh, smart_health.sh, cert_monitor.sh,
629	  backup_verify.sh, emby_session_report.sh, bandwidth_monitor.sh --report
630	```
631
632	**→ Complete schedule: [README-Orchestrators.md](Orchestrators/README-Orchestrators.md)**
633
634	---
635
636	## ━━━ THE DAILY CYCLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
637
638	What actually happens on a typical day, from the ecosystem's perspective:
639
640	```
641	Throughout the day (every 15 minutes via watchdog_orchestrator.sh):
642	  resource_watchdog.sh        managing: system pressure (throttle/pause/stop)
643	  docker_watchdog.sh          healing: memory leaks, HTTP failures, required containers
644	  system_watchdog.sh          watching: system components (storage growth, WebGUI)
645	  stability_watchdog.sh       watching: RAM, CPU temp, rootfs, kernel, daemon
646
647	Throughout the day:
648	  fallback.sh                 watching: remote server, internet connectivity
649	  transcode_management.sh     managing: ramdisk ↔ SSD, session cleanup (every 7min)
650	  critical_sync_maintenance.sh keeping: auth stack + Emby current (every 30min)
651
652	1am:
653	  git pull (latest scripts first)
654	  rsync Movies, Tv_Shows, Music → HOST2        (HOST1 pushes its truth shares)
655	  rsync Anime_Shows, Anime_Movies → HOST1      (HOST2 pushes its truth shares)
656	  media_shares_permissions.sh                  (fix ownership before arr cleanup)
657	  media_cleaner.sh anime + media               (remove scene junk)
658	  lidarr/sonarr/radarr_cleanup.sh              (remove orphaned files)
659	  docker_daily_restart.sh                      (NPM, Authelia, Dispatcharr fresh start)
660
661	2:30am Sunday:
662	  Stop Emby + auth stack both sides
663	  Pull updated Docker images (zero extra downtime — already stopped)
664	  rsync Emby + Critical-Data clean mirror
665	  Start containers on new images
666	  docker_weekly_restart.sh                     (NextCloud, AdGuard, Immich)
667
668	7am Sunday:
669	  sunday_morning_coffee_report.sh              (everything that happened this week)
670	```
671
672	---
673
674	## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
675
676	Problems are handled at the most appropriate level. Scripts escalate only when the
677	level below them has failed:
678
679	```
680	Problem: Emby container using 22GB RAM (limit: 18GB)
681	  → docker_watchdog.sh Tier 1: restart Emby immediately
682	  → System continues, no reboot
683
684	Problem: Authelia crashed and won't restart
685	  → docker_watchdog.sh Tier 1: attempt restart (dependency check: DB + Redis first)
686	  → Fails 3 times in 1 hour → skip list → critical notification
687	  → Tools: watchdog_skip_list_manager.sh to clear after fixing
688
689	Problem: RAM drops to 3GB
690	  → stability_watchdog.sh: stop non-essential containers
691	  → Wait for RAM to recover to 30GB
692	  → docker_watchdog.sh defers (reads mem_shutdown_active=true)
693	  → If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot
694
695	Problem: Docker daemon hung
696	  → docker_watchdog.sh: attempt platform_restart_service docker (rc.d on Unraid)
697	  → If still hung: sets daemon_confirmed_down flag → stability_watchdog escalates → reboot
698
699	Problem: HOST1 loses power
700	  → fallback.sh on HOST2 detects at next 2-minute check
701	  → Tier 1 starts immediately: DDNS, Emby, VaultWarden, Live TV
702	  → Household barely notices. Live TV may buffer for < 2 minutes.
703	```
704
705	**→ Container healing: [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md)**
706	**→ System stability: [README-System_Essentials.md](System_Essentials/README-System_Essentials.md)**
707
708	---
709
710	## ━━━ GPU CONTAINERS — IGNORE GOOGLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
711
712	> **Every tutorial, forum post, and Reddit thread shows the wrong way to add GPU
713	> access to a Docker container on unRAID.** The old method breaks silently in ways
714	> that are very hard to diagnose. Read this before touching any GPU container.
715
716	### The Wrong Way (What Google Shows)
717
718	```bash
719	# DO NOT DO THIS
720	Extra Parameters:  --runtime=nvidia
721	Variables:         NVIDIA_VISIBLE_DEVICES=GPU-xxxxxxxx
722	                   NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
723	```
724
725	This is `--runtime=nvidia` — the old NVIDIA Container Toolkit approach. It is what
726	every guide written before Docker 19.03 recommends. It still shows up in official
727	unRAID forum posts, community applications templates, and most search results.
728
729	**Why it breaks here:**
730
731	1. `--runtime=nvidia` combined with `NVIDIA_VISIBLE_DEVICES` has caused conflicts
732	   historically and breaks on container rebuilds. With Docker 29.x (Unraid 7.3+),
733	   `bind-propagation=shared` crashes runc v1.3.5 entirely — with any container,
734	   GPU or not. The container fails to start with:
735	   ```
736	   OCI runtime create failed: unable to start container process:
737	   error jailing process inside rootfs:
738	   open /proc/self/mountinfo: no such file or directory
739	   ```
740	   The ramdisk tmpfs is already MS\_SHARED at the kernel level. Do not use
741	   `bind-propagation=shared` in Extra Parameters on Unraid 7.3+.
742
743	2. When unRAID rebuilds a container from template (update, reinstall from Community
744	   Applications), the `--runtime=nvidia` field and the NVIDIA env vars can be dropped
745	   or corrupted. Rebuilding a container is a normal maintenance operation — it should
746	   not break GPU access. With the old method, it sometimes does.
747
748	### The Right Way
749
750	```bash
751	# DO THIS — everything in Extra Parameters, nothing in Variables
752	--gpus "device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1" --pids-limit=0 --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
753	```
754
755	`--gpus "device=UUID"` is Docker-native GPU support (Docker 19.03+, released 2019).
756	One field in Extra Parameters. The UUID pins a specific device — no ambiguity on
757	single-GPU systems. No env vars needed. No `--runtime=nvidia`. Survives container
758	rebuilds. This is the correct method.
759
760	Find your GPU UUID: `nvidia-smi -L`
761
762	**HOST1 GPU UUID (Quadro P2000):** `GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1`
763
764	### GPU Memory and Media Servers
765
766	A failure mode that is extremely non-obvious and not documented anywhere:
767
768	Any container that uses VRAM and does not release it will starve Emby and Jellyfin
769	of the GPU memory they need to start transcoding sessions. The two media servers
770	respond differently:
771
772	- **Jellyfin:** hard fails — no session is created, playback stops entirely
773	- **Emby:** falls back to CPU transcoding silently and keeps going
774
775	The non-obvious source is OCR sidecars — credit detection plugins, subtitle
776	extraction tools — that talk to a GPU-accelerated OCR container. PaddleOCR in
777	particular loads a neural network into VRAM at first use and does not release it
778	between runs. One credit scan, and VRAM stays consumed until the container restarts.
779
780	**Emby startup probe:** Emby runs a one-shot NVIDIA hardware detection at container
781	start. If VRAM is exhausted when Emby starts, NVIDIA is marked unavailable for the
782	entire session — no retry. Restarting Emby while VRAM is still full just fails the
783	probe again. Correct recovery: free VRAM first, confirm with `nvidia-smi`, then
784	restart Emby.
785
786	For the EmbyCredits plugin: use the `yock1/embycreditocr` Tesseract image (port 8884,
787	CPU-based) instead of PaddleOCR. Same plugin, no VRAM.
788
789	**→ Full detail: [Manual-Transcoding.md](Transcodes/Manual-Transcoding.md)**
790
791	---
792
793	## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
794
795	If you're setting this up from scratch on two servers:
796
797	```
798	1. Follow Manual.md — end-to-end setup guide
799	   Tailscale → SSH keys → git clone → master.conf → Varaverk scheduler schedule
800
801	2. Set your container names correctly before anything else
802	   Shared services: no suffix (Emby, NginxProxyManager, etc.)
803	   Personal services: suffixed (Emby-Gmer4Lfe, Organizer-Gmer4Lfe)
804
805	3. Set your share names identically on both servers
806	   /mnt/user/Movies on both, /mnt/user/Tv_Shows on both, etc.
807
808	4. Run initial sync to populate HOST2
809	   bash daily_sync_maintenance.sh --log
810
811	5. Verify fallback works before relying on it
812	   bash fallback_test.sh --dry-run first
813	   bash fallback_test.sh when ready — uses iptables simulation, no real downtime
814	```
815
816	---
817
818	## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
819
820	| README | What It Covers |
821	|--------|---------------|
822	| [README-Fallback.md](Fallback/README-Fallback.md) | DDNS sequencing, tiered fallback, handback, split brain prevention, fallback_test.sh |
823	| [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md) | Two-tier container watchdog, memory limits, dependency ordering, skip list, daily/weekly restarts |
824	| [README-System_Essentials.md](System_Essentials/README-System_Essentials.md) | WebGUI recovery, inotify tuning, PHP-FPM, log hygiene, mover/reboot/rsync stop |
825	| [README-Orchestrators.md](Orchestrators/README-Orchestrators.md) | What runs when, execution order, daily/weekly windows, adding jobs |
826	| [README-Media.md](Media/README-Media.md) | Permissions model, junk cleanup, arr orphan removal, safety layers, behavior-driven discovery, testing procedure |
827	| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
828	| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
829	| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
830	| [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
831	| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
832	| [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
833
834	---
835
836	## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
837
838	Things that are different from what you might expect:
839
840	```bash
841	# ─────────────────────────────────────────────────────────────────────────────
842	# 1. The Varaverk scheduler has very few entries.
843	#    Everything runs through orchestrators. Individual scripts are never
844	#    scheduled directly. array_started.sh is the only array_start event entry.
845
846	# 2. Scripts are silent when healthy.
847	#    stability_watchdog.sh runs 288 times per day. It produces zero output on clean
848	#    cycles. If you see output, something needs attention. If you see nothing,
849	#    the system is healthy. This is by design.
850
851	# 3. master.conf is the only file you edit.
852	#    Master_host*.conf for per-server values. Everything else is code — don't
853	#    add configuration into scripts directly. Everything configurable has a var.
854
855	# 4. --dry-run works on everything.
856	#    Every single script supports --dry-run. Pre-flight checks still run.
857	#    Review what would happen before anything changes. Always available.
858
859	# 5. detect_hosts() does the heavy lifting.
860	#    Scripts never compare hostnames directly. detect_hosts() aliases the correct
861	#    HOST*_ vars based on the running server. The same script runs correctly on
862	#    HOST1 and HOST2 — no branching on server identity inside scripts.
863
864	# 6. GPU containers: ignore every Google result. Use --gpus, not --runtime=nvidia.
865	#    --runtime=nvidia breaks on container rebuilds. See GPU CONTAINERS section above.
866
867	# 7. The Emby/Jellyfin Docker mount is unusual and critical.
868	#    --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
869	#    in Extra Parameters — not a standard path mapping. Do NOT add bind-propagation=shared:
870	#    runc v1.3.5+ (Docker 29.x, Unraid 7.3+) rejects it. The ramdisk is already
871	#    MS_SHARED at the kernel level. README-Transcoding.md explains in detail.
872
873	# 8. arr cleanup scripts can delete files.
874	#    lidarr_cleanup.sh, sonarr_cleanup.sh, radarr_cleanup.sh permanently delete
875	#    orphaned files. Safety layers prevent catastrophic runs but always test with
876	#    --dry-run --log first on a new system. README-Media.md has the full procedure.
877
878	# 9. Scripts never call OS commands directly.
879	#    All platform-specific operations go through plugin_*() functions defined in
880	#    Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all
881	#    isolated in the adapter. This is what makes the codebase portable.
882	#    Adding a new OS = writing one adapter file, not patching dozens of scripts.
883
884	# 10. System_Essentials/ is platform-agnostic at the script level.
885	#    Scripts like server_reboot.sh, mover_stop.sh, and webgui_watchdog.sh call
886	#    adapter functions (platform_is_mover_running, platform_restart_service, etc.)
887	#    The adapter handles what those mean on the actual OS. The scripts are clean.
888	# ─────────────────────────────────────────────────────────────────────────────
889	```
890
891	---
892
893	## ━━━ PROJECT NAME ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
894
895	```
896	Name:         Varaverk
897	Repository:   git@192.168.50.2:Varaverk/varaverk.git
898	Install path: /boot/config/plugins/varaverk
899	```


1	# ━━━━━ VARAVERK — Setup Manual ━━━━━
2
3	Getting a fresh two-server ecosystem running from scratch.
4	For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.
5
6	---
7
8	## ━━━ BEFORE YOU START ━━━
9
10	Things that must be in place before you touch any scripts.
11
12	---
13
14	### ── Two unRAID Servers ───────────────────────────────────────────────────────
15
16	Both servers need to be up and have their arrays started. Hardware does not need to
17	match. The scripts work via `/mnt/user/` — the same share name on both servers resolves
18	correctly regardless of the hardware underneath.
19
20	Decide which server is **HOST1 (owner)** and which is **HOST2 (mirror)**. This is just a
21	label — both servers run their own full stack. The owner manages the shared auth
22	configuration. Pick the more capable or more reliably-online server as owner.
23
24	---
25
26	### ── Plugins — install on both servers ────────────────────────────────────────
27
28	Install these via **Apps** (Community Applications) on both servers:
29
30	| Plugin | Why |
31	|--------|-----|
32	| **Tailscale** | Encrypted VPN between servers — all script traffic travels over it |
33
34	---
35
36	## ━━━ STEP 1: TAILSCALE ━━━
37
38	Both servers must be visible to each other on the same Tailscale network.
39
40	---
41
42	### ── Connect both servers ────────────────────────────────────────────────────
43
44	1. Open the Tailscale plugin on HOST1. Log in with your Tailscale account.
45	2. Open the Tailscale plugin on HOST2. Log in with the **same account** — or have the
46	   HOST2 owner share their machine into your tailnet (Tailscale → Share → invite by email).
47	3. Confirm both devices appear in your Tailscale admin panel.
48
49	---
50
51	### ── Verify reachability ──────────────────────────────────────────────────────
52
53	On HOST1:
54	```bash
55	# Get your Tailscale IP
56	tailscale ip -4
57
58	# Ping HOST2 by its Tailscale hostname (the unRAID server name, e.g. unRAID-Jayred365)
59	ping -c 3 unRAID-Jayred365
60	```
61
62	On HOST2 — same in reverse. Both pings must succeed before continuing.
63
64	---
65
66	### ── Note the Tailscale hostnames ──────────────────────────────────────────────
67
68	The scripts use Tailscale DNS to resolve IPs — no hardcoded addresses anywhere.
69	For this to work, the Tailscale device name must match the unRAID server hostname exactly.
70
71	Check the unRAID hostname: **Settings → System Identification → Server Name**
72	Check the Tailscale device name: Tailscale admin panel → Machines
73
74	If they differ, rename the Tailscale device to match the unRAID hostname. Case matters.
75
76	```bash
77	# On each server — confirm the hostname the scripts will use
78	hostname
79	# Should return e.g. "unRAID-Gmer4Lfe"
80	```
81
82	---
83
84	## ━━━ STEP 2: CLONE THE REPOSITORY ━━━
85
86	---
87
88	### ── Clone on HOST1 ───────────────────────────────────────────────────────────
89
90	```bash
91	git clone <your_repo_url> /boot/config/plugins/varaverk
92	cd /boot/config/plugins/varaverk
93	```
94
95	---
96
97	### ── Clone on HOST2 ───────────────────────────────────────────────────────────
98
99	```bash
100	git clone <your_repo_url> /boot/config/plugins/varaverk
101	cd /boot/config/plugins/varaverk
102	```
103
104	---
105
106	### ── Sparse checkout — keep credentials separate ────────────────────────────
107
108	`host1.conf` contains HOST1's SSH keys, API keys, and passwords.
109	`host2.conf` contains HOST2's equivalents. Neither server should hold the other's
110	credentials. Set up sparse checkout so each server only receives its own conf file.
111
112	**On HOST1** — exclude host2.conf:
113	```bash
114	cd /boot/config/plugins/varaverk
115	git sparse-checkout init --no-cone
116	git sparse-checkout set '/*' '!/Configurations/host2.conf'
117	git checkout
118	```
119
120	**On HOST2** — exclude host1.conf:
121	```bash
122	cd /boot/config/plugins/varaverk
123	git sparse-checkout init --no-cone
124	git sparse-checkout set '/*' '!/Configurations/host1.conf'
125	git checkout
126	```
127
128	Verify: `ls Configurations/` — each server should see `master.conf` and only its own `host*.conf`.
129
130	> If you are not using git (deploying manually) skip this and ensure you only copy
131	> each server's own host*.conf when deploying.
132
133	---
134
135	## ━━━ STEP 3: CONFIGURE ━━━
136
137	Configuration lives in `Configurations/`. Three files:
138	- `master.conf` — shared, both servers see it — thresholds, toggles, schedules
139	- `host1.conf` — HOST1 only — credentials, share lists, container names
140	- `host2.conf` — HOST2 only — same structure, different values
141
142	Edit these with any text editor. Everything in the scripts is controlled through these
143	files — never edit scripts directly to change behaviour.
144
145	---
146
147	### ── 3a. master.conf — server hostnames (do this first) ─────────────────────
148
149	Open `Configurations/master.conf`. Find the HOST IDENTITIES section near the top:
150
151	```bash
152	HOST1="unRAID-Gmer4Lfe"
153	HOST2="unRAID-Jayred365"
154	```
155
156	Replace these with your actual unRAID server hostnames. These must match exactly —
157	`detect_hosts()` compares the running server's hostname against these two values to
158	know which server it is on. Everything else in the ecosystem flows from this.
159
160	---
161
162	### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────
163
164	Open `Configurations/host1.conf`. Fill in the IDENTITY section:
165
166	```bash
167	HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"   # path — ssh_setup.sh will create this
168	HOST1_OWNER="gmer4lfe"                                  # short name, lowercase, no spaces
169	HOST1_OWNER_EMAIL="you@example.com"                     # for partnership notifications
170	```
171
172	The SSH key does not exist yet — `ssh_setup.sh` in Step 4 will create it and fill in
173	the path automatically. Leave the path as-is for now; it will be overwritten.
174
175	---
176
177	### ── 3c. host2.conf — HOST2 identity ────────────────────────────────────────
178
179	Open `Configurations/host2.conf`. Fill in the same fields with HOST2's values:
180
181	```bash
182	HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"
183	HOST2_OWNER="jayred365"
184	HOST2_OWNER_EMAIL="them@example.com"
185	```
186
187	---
188
189	### ── 3d. Remaining configuration ────────────────────────────────────────────
190
191	The scripts will not break if the remaining fields are empty — they self-guard
192	when things are not configured. Fill in sections as you enable each feature. The
193	minimum set needed to get a working sync + fallback:
194
195	**In host1.conf (and host2.conf mirror):**
196	```bash
197	# Shares HOST1 pushes to HOST2 nightly
198	HOST1_DAILY_SYNC_SHARES=(
199	    "/mnt/user/Movies"
200	    "/mnt/user/Tv_Shows"
201	    "/mnt/user/Music"
202	)
203
204	# Shares synced every 30 minutes (auth config + Emby watch state)
205	HOST1_CRITICAL_SYNC_SHARES=(
206	    "/mnt/user/appdata-Fallback/Critical-Data|critical-data"
207	)
208
209	# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate)
210	HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=()   # HOST2's own covers for HOST1, and vice versa
211	HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=(
212	    "Emby"
213	    "VaultWarden"
214	    "NginxProxyManager"
215	)
216	```
217
218	For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
219	and `Manual-Watchdogs.md`.
220
221	---
222
223	## ━━━ STEP 4: SSH KEYS ━━━
224
225	The servers need passwordless SSH access to each other. `ssh_setup.sh` handles
226	key generation and remote installation in one step.
227
228	---
229
230	### ── Run on HOST2 first ────────────────────────────────────────────────────
231
232	The mirror runs first — it generates its own key and copies it to HOST1. You will be
233	prompted for HOST1's root password once (for the initial key copy). After that,
234	all SSH is keyless.
235
236	```bash
237	cd /boot/config/plugins/varaverk
238	bash Partnership/ssh_setup.sh
239	```
240
241	When prompted, enter HOST1's root password. The script:
242	- Generates `/root/.ssh/jayred365_rsync_automation` (ed25519)
243	- Copies the public key to HOST1's `~/.ssh/authorized_keys`
244	- Updates `host2.conf` with the key path
245
246	---
247
248	### ── Run on HOST1 ──────────────────────────────────────────────────────────
249
250	```bash
251	cd /boot/config/plugins/varaverk
252	bash Partnership/ssh_setup.sh
253	```
254
255	Enter HOST2's root password when prompted. Same steps, opposite direction.
256
257	---
258
259	### ── Verify both directions ────────────────────────────────────────────────
260
261	```bash
262	# On HOST1 — SSH to HOST2 should succeed without a password
263	ssh -i /root/.ssh/gmer4lfe_rsync_automation root@unRAID-Jayred365 "hostname"
264	# Expected: unRAID-Jayred365
265
266	# On HOST2 — SSH to HOST1 should succeed without a password
267	ssh -i /root/.ssh/jayred365_rsync_automation root@unRAID-Gmer4Lfe "hostname"
268	# Expected: unRAID-Gmer4Lfe
269	```
270
271	If either fails:
272	```bash
273	# Check status and fingerprints
274	bash Partnership/ssh_setup.sh --status
275
276	# Re-run with --force to regenerate and re-copy
277	bash Partnership/ssh_setup.sh --force
278	```
279
280	---
281
282	### ── Tailscale SSH note ──────────────────────────────────────────────────────
283
284	The scripts SSH via Tailscale hostnames, not local IPs. The first connection to a
285	new host requires accepting the host key. Accept it now so scripts never block on an
286	interactive prompt:
287
288	```bash
289	# On HOST1
290	ssh -o StrictHostKeyChecking=accept-new root@unRAID-Jayred365 "echo ok"
291
292	# On HOST2
293	ssh -o StrictHostKeyChecking=accept-new root@unRAID-Gmer4Lfe "echo ok"
294	```
295
296	---
297
298	## ━━━ STEP 5: SCHEDULER SETUP ━━━
299
300	**Everything runs through the Varaverk plugin's built-in scheduler** — no User Scripts
301	entries needed. The plugin handles all triggers natively:
302
303	- **Array start** → `Plugin/unraid/event/disks_mounted/array_start_jobs` fires `array_started.sh`
304	- **Array stop** → `Plugin/unraid/event/disks_unmounting/array_stop_jobs` fires `array_stopping.sh`
305	- **Cron** → `Plugin/unraid/event/disks_mounted/rebuild_cron` rebuilds the cron file from `schedule.json` on every boot
306
307	Configure via the Varaverk plugin Scheduler tab (or edit `schedule.json` directly).
308	Individual scripts are never scheduled — only orchestrators.
309
310	---
311
312	### ── Schedule (Varaverk Scheduler) ───────────────────────────────────────────
313
314	| Script | Event/Cron | Purpose |
315	|--------|-----------|---------|
316	| `array_started` | `array_start` | All array startup scripts in order |
317	| `array_stopping` | `array_stop` | Ordered graceful shutdown |
318	| `transcode_management` | `*/7 * * * *` | Cleanup then manager — order critical |
319	| `watchdog_orchestrator` | `*/15 * * * *` | resource → docker → system → stability |
320	| `critical_sync_maintenance` | `*/30 * * * *` | auth + Emby dirty sync + partnership |
321	| `intermediate_sync_maintenance` | `0 */4 * * *` | arr sync + failed recovery |
322	| `daily_sync_maintenance` | `0 1 * * *` | full daily maintenance window |
323	| `weekly_sync_maintenance` | `30 2 * * 0` | clean sync + image updates |
324	| `monthly_maintenance` | `0 0 15 * *` | ZFS scrub, SMART tests (uptime-gated) |
325
326	Concurrent runs are prevented by `acquire_lock`. For the complete schedule see
327	[README-Orchestrators.md](Orchestrators/README-Orchestrators.md).
328
329	---
330
331	## ━━━ STEP 6: INITIAL SYNC ━━━
332
333	Before enabling fallback, HOST2 needs a copy of HOST1's data. This first sync
334	will take a while depending on library size.
335
336	---
337
338	### ── Dry run first ────────────────────────────────────────────────────────────
339
340	```bash
341	# On HOST1 — preview what would sync, check paths and profiles
342	bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run --log
343	```
344
345	Review the output. Check that:
346	- Share paths resolve correctly on both sides
347	- No pre-flight errors (disk space, rootfs, connectivity)
348	- Profiles look correct for each share
349
350	---
351
352	### ── Run the initial sync ─────────────────────────────────────────────────────
353
354	```bash
355	# On HOST1 — run the full daily sync (this includes rsync for all DAILY_SYNC_SHARES)
356	bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --log
357	```
358
359	This will take longer than future nightly runs — it is transferring everything for
360	the first time. Monitor progress via the Varaverk log viewer or:
361
362	```bash
363	# Watch transfer in real time
364	watch -n5 'ls -lh /mnt/user/Movies/ | tail -5'
365	```
366
367	After it finishes, also run the critical sync to populate auth and Emby state:
368	```bash
369	bash /boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh --log
370	```
371
372	---
373
374	## ━━━ STEP 7: ENABLE FALLBACK ━━━
375
376	`fallback.sh` is disabled by default — it should only be enabled when both servers
377	are correctly synced and the data on HOST2 is current.
378
379	---
380
381	### ── Enable in master.conf ────────────────────────────────────────────────────
382
383	```bash
384	FALLBACK_ENABLED=true
385	```
386
387	Commit and pull on both servers so both pick up the change.
388
389	---
390
391	### ── Fallback starts automatically on array start ────────────────────────────
392
393	`array_started.sh` launches `fallback.sh` as a background process. Once
394	`FALLBACK_ENABLED=true` is in master.conf, restarting the array starts it.
395
396	To start it now without a reboot:
397	```bash
398	bash /boot/config/plugins/varaverk/Fallback/fallback.sh &
399	```
400
401	---
402
403	### ── Verify it is running ──────────────────────────────────────────────────────
404
405	```bash
406	pgrep -a -f fallback.sh
407	# Should show the running process
408	```
409
410	Check current state:
411	```bash
412	cat /boot/config/fallback_state.db
413	# state=NORMAL — both servers up
414	```
415
416	---
417
418	## ━━━ STEP 8: TEST FALLBACK ━━━
419
420	Before relying on the system, confirm it actually triggers. `fallback_test.sh`
421	simulates an outage using `iptables` — no real downtime, no real data changes.
422
423	```bash
424	# On HOST2 — dry run first (sees the sequence, no container starts)
425	bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --dry-run --log
426
427	# When ready — real test (uses iptables to simulate HOST1 unreachable)
428	bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log
429	```
430
431	The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
432	detect it and start Tier 1 containers, then unblocks and waits for handback.
433
434	See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
435	how to interpret the output.
436
437	---
438
439	## ━━━ STEP 9: PARTNERSHIP ONBOARD ━━━
440
441	The partnership is the optional layer that shares HOST1's auth stack with HOST2 —
442	NPM, Authelia, LLDAP, MariaDB, Redis all warm on HOST2, managed centrally from HOST1.
443
444	Skip this step if HOST2 does not need HOST1's auth stack (it runs fully independent auth).
445
446	---
447
448	### ── Prerequisites ────────────────────────────────────────────────────────────
449
450	Before running partnership onboard:
451
452	- Steps 1–8 complete on both servers
453	- HOST1's auth containers are running and healthy
454	- `PARTNERSHIP_AUTH_STACK` in `host1.conf` lists the auth container XML templates
455	- `PARTNERSHIP_ARR_STACK` in `host1.conf` lists arr container XML templates (if sharing arrs)
456	- FolderView3 plugin installed on both servers (if using `PARTNERSHIP_FOLDERVIEW3=true`)
457
458	---
459
460	### ── Run on HOST2 first (mirror) ────────────────────────────────────────────
461
462	HOST2 runs first — it generates its SSH key and waits. The owner completes setup remotely.
463
464	```bash
465	# On HOST2
466	bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
467	# Review output, then:
468	bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log
469	```
470
471	HOST2's work is done after Step 1 (SSH key setup). The rest happens on HOST1.
472
473	---
474
475	### ── Run on HOST1 (owner) ────────────────────────────────────────────────────
476
477	```bash
478	# On HOST1
479	bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
480	# Review output — this will deploy auth + arr stacks to HOST2 remotely
481	bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log
482	```
483
484	The owner path deploys containers to HOST2 via SSH, configures WebUI targets, runs
485	an initial arr library sync, and writes the partnership state files on both servers.
486
487	For full detail on what each step does see
488	[Manual-Partnership.md](Partnership/Manual-Partnership.md).
489
490	---
491
492	## ━━━ STEP 10: VERIFY THE ECOSYSTEM ━━━
493
494	---
495
496	### ── Check script output ────────────────────────────────────────────────────
497
498	Run the main orchestrators manually with `--log` to see verbose output:
499
500	```bash
501	# Watchdogs — should find everything healthy
502	bash Orchestrators/watchdog_orchestrator.sh --log
503
504	# Daily sync — dry run to confirm share paths and profiles
505	bash Orchestrators/daily_sync_maintenance.sh --dry-run --log
506	```
507
508	---
509
510	### ── Check fallback state ────────────────────────────────────────────────────
511
512	```bash
513	cat /boot/config/fallback_state.db
514	# Expected:
515	# state=NORMAL
516	# fallback_start=0
517	# handback_strikes=0
518	# tier2_started=false
519	# tier3_started=false
520	# tier4_started=false
521	```
522
523	---
524
525	### ── Check sync logs ─────────────────────────────────────────────────────────
526
527	Bandwidth monitor records every sync:
528	```bash
529	# Latest transfer log
530	ls -lt /boot/config/plugins/varaverk/data/bandwidth_monitor/
531	cat /boot/config/plugins/varaverk/data/bandwidth_monitor/<latest_log>
532	```
533
534	---
535
536	### ── Run status on key scripts ─────────────────────────────────────────────
537
538	```bash
539	bash Fallback/fallback.sh --status
540	bash Partnership/ssh_setup.sh --status
541	bash Monitors/cert_monitor.sh --status
542	bash Watchdogs/resource_watchdog.sh --status
543	```
544
545	---
546
547	## ━━━ COMMON FIRST-RUN ISSUES ━━━
548
549	---
550
551	### ── Script exits silently / does nothing ────────────────────────────────────
552
553	```bash
554	# Run with --log to see verbose output
555	bash <script> --log
556
557	# Check if the script is skipping due to a toggle being false
558	grep "ENABLED\|_TOGGLE\|ENABLED=" Configurations/master.conf | grep "false"
559	```
560
561	Most scripts have an `ENABLED=true/false` toggle in `master.conf`. Check that the
562	feature you are testing is enabled.
563
564	---
565
566	### ── SSH connection refused or times out ────────────────────────────────────
567
568	```bash
569	# Verify Tailscale is running and the remote is reachable
570	tailscale status
571	ping -c 3 <remote_hostname>
572
573	# Verify the key is authorised on the remote
574	ssh -v -i /root/.ssh/<your_key> root@<remote_hostname> "echo ok"
575
576	# Re-run ssh_setup.sh if the key is not installed
577	bash Partnership/ssh_setup.sh --force
578	```
579
580	---
581
582	### ── detect_hosts() exits — MY_ID unknown ────────────────────────────────────
583
584	```bash
585	# The running server's hostname doesn't match HOST1 or HOST2 in master.conf
586	hostname
587	# Compare to master.conf
588	grep "^    HOST[12]=" Configurations/master.conf
589	# They must match exactly — case sensitive
590	```
591
592	---
593
594	### ── Fallback triggers immediately on startup ────────────────────────────────
595
596	The remote server is unreachable on the first check. Common causes:
597	- Tailscale not running on the remote
598	- Remote server is actually offline
599	- `HOST1`/`HOST2` hostnames in `master.conf` don't match Tailscale device names
600
601	```bash
602	# Confirm which state fallback is in
603	cat /boot/config/fallback_state.db
604	# If stuck in FALLBACK after remote comes back: reset state
605	bash Tools/fallback_state_reset.sh
606	```
607
608	---
609
610	### ── Rsync fails — remote share not found ─────────────────────────────────────
611
612	```bash
613	# Check the share exists on the remote
614	ssh -i /root/.ssh/<your_key> root@<remote_hostname> "ls /mnt/user/"
615
616	# Share names must be identical on both servers
617	# /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2
618	# A mismatch = rsync writes to a non-existent path or aborts pre-flight
619	```
620
621	---
622
623	## ━━━ WHAT'S NEXT ━━━
624
625	Once the core is running, enable and configure the remaining features at your own pace.
626	Each subsystem has a README with the design decisions and a Manual with the config
627	reference and troubleshooting. The `--status` flag on any script shows the current
628	configuration and state.
629
630	```
631	Watchdogs/        → README-Watchdogs.md        configure memory limits, container lists
632	Media/            → README-Media.md             enable arr cleanup, discovery scripts
633	Monitors/         → README-Monitors.md          cert monitoring, SMART, health digest
634	Transcodes/       → README-Transcoding.md       ramdisk setup for Emby transcode
635	Partnership/      → README-Partnership.md       auth stack sharing, full lifecycle
636	```
637