Varaverk/Manual.md

━━━━━ VARAVERK — Setup Manual ━━━━━

Getting a fresh two-server ecosystem running from scratch.
For system overview see README.md. For individual subsystem detail see folder READMEs and script headers.

Ten steps, and they're in this order for a reason. Each one assumes the previous one actually worked — not that you ran it, that it worked. Every step below ends with a way to check, and skipping those checks is how you end up three steps later debugging the wrong thing entirely.

The steps that hurt most when rushed are 3 (naming) and 8 (testing failover). Step 3 because renaming anything afterwards means chasing it through every conf, and step 8 because untested failover isn't redundancy — it's a guess you haven't checked yet.

Budget an evening. It is not hard, but it is not five minutes either.

---

━━━ BEFORE YOU START ━━━

None of this is optional and none of it is Varaverk's job to install. Get these in place first — every step after here assumes they're already true.

---

── Two unRAID Servers ───────────────────────────────────────────────────────

Both servers need to be up and have their arrays started. Hardware does not need to match. The scripts work via /mnt/user/ — the same share name on both servers resolves correctly regardless of the hardware underneath.

Decide which server is HOST1 (owner) and which is HOST2 (mirror). This is just a label — both servers run their own full stack. The owner manages the shared auth configuration. Pick the more capable or more reliably-online server as owner.

---

── Plugins — install on both servers ────────────────────────────────────────

Install these via Apps (Community Applications) on both servers:

Plugin	Why
Tailscale	Encrypted VPN between servers — all script traffic travels over it

---

━━━ STEP 1: TAILSCALE ━━━

Both servers must be visible to each other on the same Tailscale network.

---

── Connect both servers ────────────────────────────────────────────────────

1. Open the Tailscale plugin on HOST1. Log in with your Tailscale account.
2. Open the Tailscale plugin on HOST2. Log in with the same account — or have the HOST2 owner share their machine into your tailnet (Tailscale → Share → invite by email).
3. Confirm both devices appear in your Tailscale admin panel.

---

── Verify reachability ──────────────────────────────────────────────────────

On HOST1:

# Get your Tailscale IP
tailscale ip -4

# Ping HOST2 by its Tailscale hostname (the unRAID server name, e.g. unRAID-Jayred365)
ping -c 3 unRAID-Jayred365

On HOST2 — same in reverse. Both pings must succeed before continuing.

---

── Note the Tailscale hostnames ──────────────────────────────────────────────

The scripts use Tailscale DNS to resolve IPs — no hardcoded addresses anywhere. For this to work, the Tailscale device name must match the unRAID server hostname exactly.

Check the unRAID hostname: Settings → System Identification → Server Name
Check the Tailscale device name: Tailscale admin panel → Machines

If they differ, rename the Tailscale device to match the unRAID hostname. Case matters.

# On each server — confirm the hostname the scripts will use
hostname
# Should return e.g. "unRAID-Gmer4Lfe"

---

━━━ STEP 2: CLONE THE REPOSITORY ━━━

---

── Clone on HOST1 ───────────────────────────────────────────────────────────

git clone <your_repo_url> /boot/config/plugins/varaverk
cd /boot/config/plugins/varaverk

---

── Clone on HOST2 ───────────────────────────────────────────────────────────

git clone <your_repo_url> /boot/config/plugins/varaverk
cd /boot/config/plugins/varaverk

---

── Sparse checkout — keep credentials separate ────────────────────────────

host1.conf contains HOST1's SSH keys, API keys, and passwords. host2.conf contains HOST2's equivalents. Neither server should hold the other's credentials. Set up sparse checkout so each server only receives its own conf file.

On HOST1 — exclude host2.conf:

cd /boot/config/plugins/varaverk
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/Configurations/host2.conf'
git checkout

On HOST2 — exclude host1.conf:

cd /boot/config/plugins/varaverk
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/Configurations/host1.conf'
git checkout

Verify: ls Configurations/ — each server should see master.conf and only its own host*.conf.

If you are not using git (deploying manually) skip this and ensure you only copy each server's own host*.conf when deploying.

---

━━━ STEP 3: CONFIGURE ━━━

Configuration lives in Configurations/. Three files:

• master.conf — shared, both servers see it — thresholds, toggles, schedules
• host1.conf — HOST1 only — credentials, share lists, container names
• host2.conf — HOST2 only — same structure, different values

Edit these with any text editor. Everything in the scripts is controlled through these files — never edit scripts directly to change behaviour.

---

── 3a. master.conf — server hostnames (do this first) ─────────────────────

Open Configurations/master.conf. Find the HOST IDENTITIES section near the top:

HOST1="unRAID-Gmer4Lfe"
HOST2="unRAID-Jayred365"

Replace these with your actual unRAID server hostnames. These must match exactly — detect_hosts() compares the running server's hostname against these two values to know which server it is on. Everything else in the ecosystem flows from this.

Get this wrong and nothing works, but nothing errors either. If neither value matches, MY_ID is never set, and every script that depends on it either exits early or resolves ${MY_ID}_SOMETHING to an empty variable and quietly takes the wrong branch. Copy the value straight out of hostname on each box rather than typing what you think it is:

hostname     # run this on each server, paste the exact output

The 15-character trap. unRAID truncates the Server Name to 15 characters for NetBIOS. If your name is longer, what you set in the WebGUI and what hostname returns are two different strings. detect_hosts() has a fallback that matches a truncated 15-char hostname against a longer configured value — but only when the live hostname is exactly 15 characters. Don't rely on it. Use the real hostname output.

---

── 3b. host1.conf — HOST1 identity ────────────────────────────────────────

Open Configurations/host1.conf. Fill in the IDENTITY section:

HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"   # path — ssh_setup.sh will create this
HOST1_OWNER="gmer4lfe"                                  # short name, lowercase, no spaces
HOST1_OWNER_EMAIL="you@example.com"                     # for partnership notifications

The SSH key does not exist yet — ssh_setup.sh in Step 4 will create it and fill in the path automatically. Leave the path as-is for now; it will be overwritten.

---

── 3c. host2.conf — HOST2 identity ────────────────────────────────────────

Open Configurations/host2.conf. Fill in the same fields with HOST2's values:

HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"
HOST2_OWNER="jayred365"
HOST2_OWNER_EMAIL="them@example.com"

---

── 3d. Remaining configuration ────────────────────────────────────────────

The scripts will not break if the remaining fields are empty — they self-guard when things are not configured. Fill in sections as you enable each feature. The minimum set needed to get a working sync + fallback:

In host1.conf (and host2.conf mirror):

# Shares HOST1 pushes to HOST2 nightly
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"
    "/mnt/user/Tv_Shows"
    "/mnt/user/Music"
)

# Shares synced every 30 minutes (auth config + Emby watch state)
HOST1_CRITICAL_SYNC_SHARES=(
    "/mnt/user/appdata-Fallback/Critical-Data|critical-data"
)

# Containers to start when HOST1 goes down (Tier 1 = immediate)
# Lives in host1.conf. Named for the host being COVERED, not the host doing the covering.
FALLBACK_HOST1_TIER1=(
    "Emby"
    "VaultWarden"
    "NginxProxyManager"
)

Read that naming carefully — it catches people. The variable is FALLBACK_${REMOTE_ID}_TIER${N}, and REMOTE_ID is the host that went down. So FALLBACK_HOST1_TIER1 is the list HOST2 reads when HOST1 is unreachable. It defines "what HOST1 needs covering", not "what HOST2 runs".

It lives in host1.conf for the same reason — HOST1 owns the description of its own service stack. HOST2 receives it through the conf cache rather than keeping its own opinion about what HOST1 runs. One list, one owner, no drift.

Tiers 2–4 activate progressively after HOST1_TIER2_DELAY etc., so a brief outage never drags the entire remote stack across.

For full configuration reference see Manual-Fallback.md, Manual-Rsync.md, and Manual-Watchdogs.md.

---

━━━ STEP 4: SSH KEYS ━━━

The servers need passwordless SSH access to each other. ssh_setup.sh handles key generation and remote installation in one step.

---

── Run on HOST2 first ────────────────────────────────────────────────────

The mirror runs first — it generates its own key and copies it to HOST1. You will be prompted for HOST1's root password once (for the initial key copy). After that, all SSH is keyless.

cd /boot/config/plugins/varaverk
bash Partnership/ssh_setup.sh

When prompted, enter HOST1's root password. The script:

• Generates /root/.ssh/jayred365_rsync_automation (ed25519)
• Copies the public key to HOST1's ~/.ssh/authorized_keys
• Updates host2.conf with the key path

---

── Run on HOST1 ──────────────────────────────────────────────────────────

cd /boot/config/plugins/varaverk
bash Partnership/ssh_setup.sh

Enter HOST2's root password when prompted. Same steps, opposite direction.

---

── Verify both directions ────────────────────────────────────────────────

# On HOST1 — SSH to HOST2 should succeed without a password
ssh -i /root/.ssh/gmer4lfe_rsync_automation root@unRAID-Jayred365 "hostname"
# Expected: unRAID-Jayred365

# On HOST2 — SSH to HOST1 should succeed without a password
ssh -i /root/.ssh/jayred365_rsync_automation root@unRAID-Gmer4Lfe "hostname"
# Expected: unRAID-Gmer4Lfe

If either fails:

# Check status and fingerprints
bash Partnership/ssh_setup.sh --status

# Re-run with --force to regenerate and re-copy
bash Partnership/ssh_setup.sh --force

---

── Tailscale SSH note ──────────────────────────────────────────────────────

The scripts SSH via Tailscale hostnames, not local IPs. The first connection to a new host requires accepting the host key. Accept it now so scripts never block on an interactive prompt:

# On HOST1
ssh -o StrictHostKeyChecking=accept-new root@unRAID-Jayred365 "echo ok"

# On HOST2
ssh -o StrictHostKeyChecking=accept-new root@unRAID-Gmer4Lfe "echo ok"

---

━━━ STEP 5: SCHEDULER SETUP ━━━

Everything runs through the Varaverk plugin's built-in scheduler — no User Scripts entries needed. The plugin handles all triggers natively:

• Array start → Plugin/unraid/event/disks_mounted/array_start_jobs fires array_started.sh
• Array stop → Plugin/unraid/event/disks_unmounting/array_stop_jobs fires array_stopping.sh
• Cron → Plugin/unraid/event/disks_mounted/rebuild_cron rebuilds the cron file from schedule.json on every boot

Configure via the Varaverk plugin Scheduler tab (or edit schedule.json directly). The built-in job list schedules orchestrators, never individual repo scripts directly — but the Scheduler tab's Custom Scripts card is the one place individual scripts are scheduled directly (see below).

---

── Custom Scripts ────────────────────────────────────────────────────────────

The Scheduler tab has a Custom Scripts card for one-off scripts that aren't part of the repo's orchestrator pipeline — personal tooling, quick fixes, anything you don't want to wire into master.conf.

Scripts live in /boot/config/plugins/user.scripts/Varaverk/Scripts/ — deliberately outside the Varaverk git repo (that folder is never pushed to GitHub), in the same place the Unraid User Scripts plugin keeps its own scripts, so it's a folder location admins are already used to.

Two ways to get a script there:

• Click + Create Script on the Scheduler tab — opens an inline editor, writes the file to that folder, and adds a schedule.json entry automatically.
• Drop any .sh file into the folder yourself (e.g. via terminal, or Unraid's own Custom Scripts / User Scripts plugin pointed at the same path). The Scheduler tab auto-detects it — discovery is a folder scan, not a registry, so it doesn't matter how the file got there. It shows up disabled with no cron until you configure one.

---

── Schedule (Varaverk Scheduler) ───────────────────────────────────────────

Script	Event/Cron	Purpose
array_started	array_start	All array startup scripts in order
array_stopping	array_stop	Ordered graceful shutdown
transcode_management	*/7 * * * *	Cleanup then manager — order critical
watchdog_orchestrator	*/15 * * * *	resource → docker → system → stability
critical_sync_maintenance	*/30 * * * *	auth + Emby dirty sync + partnership
intermediate_sync_maintenance	0 */4 * * *	arr sync + failed recovery
daily_sync_maintenance	0 1 * * *	full daily maintenance window
weekly_sync_maintenance	30 2 * * 0	clean sync + image updates
monthly_maintenance	0 0 15 * *	ZFS scrub, SMART tests (uptime-gated)

Concurrent runs are prevented by acquire_lock. For the complete schedule see README-Orchestrators.md.

---

━━━ STEP 6: INITIAL SYNC ━━━

Before enabling fallback, HOST2 needs a copy of HOST1's data. This first sync will take a while depending on library size.

---

── Dry run first ────────────────────────────────────────────────────────────

# On HOST1 — preview what would sync, check paths and profiles
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run --log

Review the output. Check that:

• Share paths resolve correctly on both sides
• No pre-flight errors (disk space, rootfs, connectivity)
• Profiles look correct for each share

---

── Run the initial sync ─────────────────────────────────────────────────────

# On HOST1 — run the full daily sync (this includes rsync for all DAILY_SYNC_SHARES)
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --log

This will take longer than future nightly runs — it is transferring everything for the first time. Monitor progress via the Varaverk log viewer or:

# Watch transfer in real time
watch -n5 'ls -lh /mnt/user/Movies/ | tail -5'

After it finishes, also run the critical sync to populate auth and Emby state:

bash /boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh --log

---

━━━ STEP 7: ENABLE FALLBACK ━━━

fallback.sh is disabled by default — it should only be enabled when both servers are correctly synced and the data on HOST2 is current.

---

── Enable in master.conf ────────────────────────────────────────────────────

FALLBACK_ENABLED=true

Commit and pull on both servers so both pick up the change.

---

── Fallback starts automatically on array start ────────────────────────────

array_started.sh launches fallback.sh as a background process. Once FALLBACK_ENABLED=true is in master.conf, restarting the array starts it.

To start it now without a reboot:

bash /boot/config/plugins/varaverk/Fallback/fallback.sh &

---

── Verify it is running ──────────────────────────────────────────────────────

pgrep -a -f fallback.sh
# Should show the running process

Check current state:

cat "$STATE_DIR/fallback_state.db"
# state=NORMAL — both servers up

---

━━━ STEP 8: TEST FALLBACK ━━━

Before relying on the system, confirm it actually triggers. fallback_test.sh simulates an outage using iptables — no real downtime, no real data changes.

# On HOST2 — dry run first (sees the sequence, no container starts)
bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --dry-run --log

# When ready — real test (uses iptables to simulate HOST1 unreachable)
bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log

The test runs in phases — blocks HOST1's Tailscale IP, waits for fallback.sh to detect it and start Tier 1 containers, then unblocks and waits for handback.

Do not skip this step. Everything up to here you can verify by reading output. Failover is the one part you cannot confirm by looking at it — a typo in a tier list, a container name that doesn't exist on the other side, a DDNS container that was renamed six months ago: all of it sits there looking completely fine until the moment it's needed.

The test costs you twenty minutes and no downtime. The alternative is finding out at 2am, during the outage, when you have exactly one chance to get it right.

Watch the handback as closely as the failover. Coming back is the harder half — it has to stop the remote's DDNS, sync the data back, start the local containers, and only then bring local DDNS up. Failover starting correctly tells you nothing about whether handback does.

See Manual-Fallback.md for what each phase does and how to interpret the output.

---

━━━ STEP 9: PARTNERSHIP ONBOARD ━━━

The partnership is the optional layer that shares HOST1's auth stack with HOST2 — NPM, Authelia, LLDAP, MariaDB, Redis all warm on HOST2, managed centrally from HOST1.

Skip this step if HOST2 does not need HOST1's auth stack (it runs fully independent auth).

---

── Prerequisites ────────────────────────────────────────────────────────────

Before running partnership onboard:

• Steps 1–8 complete on both servers
• HOST1's auth containers are running and healthy
• PARTNERSHIP_AUTH_STACK in host1.conf lists the auth container XML templates
• PARTNERSHIP_ARR_STACK in host1.conf lists arr container XML templates (if sharing arrs)
• FolderView3 plugin installed on both servers (if using PARTNERSHIP_FOLDERVIEW3=true)

---

── Run on HOST2 first (mirror) ────────────────────────────────────────────

HOST2 runs first — it generates its SSH key and waits. The owner completes setup remotely.

# On HOST2
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
# Review output, then:
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log

HOST2's work is done after Step 1 (SSH key setup). The rest happens on HOST1.

---

── Run on HOST1 (owner) ────────────────────────────────────────────────────

# On HOST1
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
# Review output — this will deploy auth + arr stacks to HOST2 remotely
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log

The owner path deploys containers to HOST2 via SSH, configures WebUI targets, runs an initial arr library sync, and writes the partnership state files on both servers.

For full detail on what each step does see Manual-Partnership.md.

---

━━━ STEP 10: VERIFY THE ECOSYSTEM ━━━

---

── Check script output ────────────────────────────────────────────────────

Run the main orchestrators manually with --log to see verbose output:

# Watchdogs — should find everything healthy
bash Orchestrators/watchdog_orchestrator.sh --log

# Daily sync — dry run to confirm share paths and profiles
bash Orchestrators/daily_sync_maintenance.sh --dry-run --log

---

── Check fallback state ────────────────────────────────────────────────────

cat "$STATE_DIR/fallback_state.db"
# Expected:
# state=NORMAL
# fallback_start=0
# handback_strikes=0
# tier2_started=false
# tier3_started=false
# tier4_started=false

---

── Check sync logs ─────────────────────────────────────────────────────────

Bandwidth monitor records every sync:

# Latest transfer log
ls -lt /boot/config/plugins/varaverk/data/bandwidth_monitor/
cat /boot/config/plugins/varaverk/data/bandwidth_monitor/<latest_log>

---

── Run status on key scripts ─────────────────────────────────────────────

bash Fallback/fallback.sh --status
bash Partnership/ssh_setup.sh --status
bash Monitors/cert_monitor.sh --status
bash Watchdogs/resource_watchdog.sh --status

---

━━━ COMMON FIRST-RUN ISSUES ━━━

---

── Script exits silently / does nothing ────────────────────────────────────

# Run with --log to see verbose output
bash <script> --log

# Check if the script is skipping due to a toggle being false
grep "ENABLED\|_TOGGLE\|ENABLED=" Configurations/master.conf | grep "false"

Most scripts have an ENABLED=true/false toggle in master.conf. Check that the feature you are testing is enabled.

---

── SSH connection refused or times out ────────────────────────────────────

# Verify Tailscale is running and the remote is reachable
tailscale status
ping -c 3 <remote_hostname>

# Verify the key is authorised on the remote
ssh -v -i /root/.ssh/<your_key> root@<remote_hostname> "echo ok"

# Re-run ssh_setup.sh if the key is not installed
bash Partnership/ssh_setup.sh --force

---

── detect_hosts() exits — MY_ID unknown ────────────────────────────────────

# The running server's hostname doesn't match HOST1 or HOST2 in master.conf
hostname
# Compare to master.conf
grep "^    HOST[12]=" Configurations/master.conf
# They must match exactly — case sensitive

---

── Fallback triggers immediately on startup ────────────────────────────────

The remote server is unreachable on the first check. Common causes:

• Tailscale not running on the remote
• Remote server is actually offline
• HOST1/HOST2 hostnames in master.conf don't match Tailscale device names

# Confirm which state fallback is in
cat "$STATE_DIR/fallback_state.db"
# If stuck in FALLBACK after remote comes back: reset state
bash Tools/fallback_state_reset.sh

---

── Rsync fails — remote share not found ─────────────────────────────────────

# Check the share exists on the remote
ssh -i /root/.ssh/<your_key> root@<remote_hostname> "ls /mnt/user/"

# Share names must be identical on both servers
# /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2
# A mismatch = rsync writes to a non-existent path or aborts pre-flight

---

━━━ WHAT'S NEXT ━━━

Once the core is running, enable and configure the remaining features at your own pace. Each subsystem has a README with the design decisions and a Manual with the config reference and troubleshooting. The --status flag on any script shows the current configuration and state.

Turn things on one at a time and give each one a few days. Everything below is off by default on a fresh install, and that's deliberate — a stack where six new subsystems went live the same night is a stack where you have no idea which one to blame. Enable, watch it through a full daily cycle, then enable the next.

Anything that deletes files — the arr cleanup scripts especially — gets a --dry-run --log first. Read the list. Every time, not just the first time.

Watchdogs/        → README-Watchdogs.md        configure memory limits, container lists
Media/            → README-Media.md             enable arr cleanup, discovery scripts
Monitors/         → README-Monitors.md          cert monitoring, SMART, health digest
Transcodes/       → README-Transcoding.md       ramdisk setup for Emby transcode
Partnership/      → README-Partnership.md       auth stack sharing, full lifecycle