# ━━━━━ VARAVERK — Setup Manual ━━━━━ Getting a fresh two-server ecosystem running from scratch. For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers. --- ## ━━━ BEFORE YOU START ━━━ Things that must be in place before you touch any scripts. --- ### ── Two unRAID Servers ─────────────────────────────────────────────────────── Both servers need to be up and have their arrays started. Hardware does not need to match. The scripts work via `/mnt/user/` — the same share name on both servers resolves correctly regardless of the hardware underneath. Decide which server is **HOST1 (owner)** and which is **HOST2 (mirror)**. This is just a label — both servers run their own full stack. The owner manages the shared auth configuration. Pick the more capable or more reliably-online server as owner. --- ### ── Plugins — install on both servers ──────────────────────────────────────── Install these via **Apps** (Community Applications) on both servers: | Plugin | Why | |--------|-----| | **Tailscale** | Encrypted VPN between servers — all script traffic travels over it | | **User Scripts** | Required for "At Startup of Array" hook — array_started.sh entry only | --- ## ━━━ STEP 1: TAILSCALE ━━━ Both servers must be visible to each other on the same Tailscale network. --- ### ── Connect both servers ──────────────────────────────────────────────────── 1. Open the Tailscale plugin on HOST1. Log in with your Tailscale account. 2. Open the Tailscale plugin on HOST2. Log in with the **same account** — or have the HOST2 owner share their machine into your tailnet (Tailscale → Share → invite by email). 3. Confirm both devices appear in your Tailscale admin panel. --- ### ── Verify reachability ────────────────────────────────────────────────────── On HOST1: ```bash # Get your Tailscale IP tailscale ip -4 # Ping HOST2 by its Tailscale hostname (the unRAID server name, e.g. unRAID-Jayred365) ping -c 3 unRAID-Jayred365 ``` On HOST2 — same in reverse. Both pings must succeed before continuing. --- ### ── Note the Tailscale hostnames ────────────────────────────────────────────── The scripts use Tailscale DNS to resolve IPs — no hardcoded addresses anywhere. For this to work, the Tailscale device name must match the unRAID server hostname exactly. Check the unRAID hostname: **Settings → System Identification → Server Name** Check the Tailscale device name: Tailscale admin panel → Machines If they differ, rename the Tailscale device to match the unRAID hostname. Case matters. ```bash # On each server — confirm the hostname the scripts will use hostname # Should return e.g. "unRAID-Gmer4Lfe" ``` --- ## ━━━ STEP 2: CLONE THE REPOSITORY ━━━ --- ### ── Clone on HOST1 ─────────────────────────────────────────────────────────── ```bash git clone /boot/config/plugins/varaverk cd /boot/config/plugins/varaverk ``` --- ### ── Clone on HOST2 ─────────────────────────────────────────────────────────── ```bash git clone /boot/config/plugins/varaverk cd /boot/config/plugins/varaverk ``` --- ### ── Sparse checkout — keep credentials separate ──────────────────────────── `host1.conf` contains HOST1's SSH keys, API keys, and passwords. `host2.conf` contains HOST2's equivalents. Neither server should hold the other's credentials. Set up sparse checkout so each server only receives its own conf file. **On HOST1** — exclude host2.conf: ```bash cd /boot/config/plugins/varaverk git sparse-checkout init --no-cone git sparse-checkout set '/*' '!/Configurations/host2.conf' git checkout ``` **On HOST2** — exclude host1.conf: ```bash cd /boot/config/plugins/varaverk git sparse-checkout init --no-cone git sparse-checkout set '/*' '!/Configurations/host1.conf' git checkout ``` Verify: `ls Configurations/` — each server should see `master.conf` and only its own `host*.conf`. > If you are not using git (deploying manually) skip this and ensure you only copy > each server's own host*.conf when deploying. --- ## ━━━ STEP 3: CONFIGURE ━━━ Configuration lives in `Configurations/`. Three files: - `master.conf` — shared, both servers see it — thresholds, toggles, schedules - `host1.conf` — HOST1 only — credentials, share lists, container names - `host2.conf` — HOST2 only — same structure, different values Edit these with any text editor. Everything in the scripts is controlled through these files — never edit scripts directly to change behaviour. --- ### ── 3a. master.conf — server hostnames (do this first) ───────────────────── Open `Configurations/master.conf`. Find the HOST IDENTITIES section near the top: ```bash HOST1="unRAID-Gmer4Lfe" HOST2="unRAID-Jayred365" ``` Replace these with your actual unRAID server hostnames. These must match exactly — `detect_hosts()` compares the running server's hostname against these two values to know which server it is on. Everything else in the ecosystem flows from this. --- ### ── 3b. host1.conf — HOST1 identity ──────────────────────────────────────── Open `Configurations/host1.conf`. Fill in the IDENTITY section: ```bash HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation" # path — ssh_setup.sh will create this HOST1_OWNER="gmer4lfe" # short name, lowercase, no spaces HOST1_OWNER_EMAIL="you@example.com" # for partnership notifications ``` The SSH key does not exist yet — `ssh_setup.sh` in Step 4 will create it and fill in the path automatically. Leave the path as-is for now; it will be overwritten. --- ### ── 3c. host2.conf — HOST2 identity ──────────────────────────────────────── Open `Configurations/host2.conf`. Fill in the same fields with HOST2's values: ```bash HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation" HOST2_OWNER="jayred365" HOST2_OWNER_EMAIL="them@example.com" ``` --- ### ── 3d. Remaining configuration ──────────────────────────────────────────── The scripts will not break if the remaining fields are empty — they self-guard when things are not configured. Fill in sections as you enable each feature. The minimum set needed to get a working sync + fallback: **In host1.conf (and host2.conf mirror):** ```bash # Shares HOST1 pushes to HOST2 nightly HOST1_DAILY_SYNC_SHARES=( "/mnt/user/Movies" "/mnt/user/Tv_Shows" "/mnt/user/Music" ) # Shares synced every 30 minutes (auth config + Emby watch state) HOST1_CRITICAL_SYNC_SHARES=( "/mnt/user/appdata-Fallback/Critical-Data|critical-data" ) # Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate) HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=() # HOST2's own covers for HOST1, and vice versa HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=( "Emby" "VaultWarden" "NginxProxyManager" ) ``` For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`, and `Manual-Watchdogs.md`. --- ## ━━━ STEP 4: SSH KEYS ━━━ The servers need passwordless SSH access to each other. `ssh_setup.sh` handles key generation and remote installation in one step. --- ### ── Run on HOST2 first ──────────────────────────────────────────────────── The mirror runs first — it generates its own key and copies it to HOST1. You will be prompted for HOST1's root password once (for the initial key copy). After that, all SSH is keyless. ```bash cd /boot/config/plugins/varaverk bash Partnership/ssh_setup.sh ``` When prompted, enter HOST1's root password. The script: - Generates `/root/.ssh/jayred365_rsync_automation` (ed25519) - Copies the public key to HOST1's `~/.ssh/authorized_keys` - Updates `host2.conf` with the key path --- ### ── Run on HOST1 ────────────────────────────────────────────────────────── ```bash cd /boot/config/plugins/varaverk bash Partnership/ssh_setup.sh ``` Enter HOST2's root password when prompted. Same steps, opposite direction. --- ### ── Verify both directions ──────────────────────────────────────────────── ```bash # On HOST1 — SSH to HOST2 should succeed without a password ssh -i /root/.ssh/gmer4lfe_rsync_automation root@unRAID-Jayred365 "hostname" # Expected: unRAID-Jayred365 # On HOST2 — SSH to HOST1 should succeed without a password ssh -i /root/.ssh/jayred365_rsync_automation root@unRAID-Gmer4Lfe "hostname" # Expected: unRAID-Gmer4Lfe ``` If either fails: ```bash # Check status and fingerprints bash Partnership/ssh_setup.sh --status # Re-run with --force to regenerate and re-copy bash Partnership/ssh_setup.sh --force ``` --- ### ── Tailscale SSH note ────────────────────────────────────────────────────── The scripts SSH via Tailscale hostnames, not local IPs. The first connection to a new host requires accepting the host key. Accept it now so scripts never block on an interactive prompt: ```bash # On HOST1 ssh -o StrictHostKeyChecking=accept-new root@unRAID-Jayred365 "echo ok" # On HOST2 ssh -o StrictHostKeyChecking=accept-new root@unRAID-Gmer4Lfe "echo ok" ``` --- ## ━━━ STEP 5: SCHEDULER SETUP ━━━ **Array start** uses the User Scripts plugin (one entry only). **Everything else** runs through the Varaverk plugin's built-in scheduler — configure via the Scheduler tab or `schedule.json`. Individual scripts are never scheduled directly. --- ### ── User Scripts — array start only ───────────────────────────────────────── Open **Settings → User Scripts**. Create one entry: | Script | Schedule | Command | |--------|----------|---------| | `array_started` | At Array Start | `bash /boot/config/plugins/varaverk/Orchestrators/array_started.sh` | Set to **Background Task**. This launches everything in ARRAY_START_SCRIPTS. --- ### ── Varaverk Scheduler — all cron entries ──────────────────────────────────── Configure via the Varaverk plugin Scheduler tab (or edit `schedule.json` directly): | Script | Cron | Purpose | |--------|------|---------| | `transcode_management` | `*/7 * * * *` | Cleanup then manager — order critical | | `watchdog_orchestrator` | `*/15 * * * *` | resource → docker → system → stability | | `critical_sync_maintenance` | `*/30 * * * *` | auth + Emby dirty sync + partnership | | `intermediate_sync_maintenance` | `0 */4 * * *` | arr sync + failed recovery | | `daily_sync_maintenance` | `0 1 * * *` | full daily maintenance window | | `weekly_sync_maintenance` | `30 2 * * 0` | clean sync + image updates | | `monthly_maintenance` | `0 0 15 * *` | ZFS scrub, SMART tests (uptime-gated) | Concurrent runs are prevented by `acquire_lock` — if the previous run is still active the new one exits immediately. No need to stagger entries manually. For the complete schedule and what each orchestrator runs, see [README-Orchestrators.md](Orchestrators/README-Orchestrators.md). --- ## ━━━ STEP 6: INITIAL SYNC ━━━ Before enabling fallback, HOST2 needs a copy of HOST1's data. This first sync will take a while depending on library size. --- ### ── Dry run first ──────────────────────────────────────────────────────────── ```bash # On HOST1 — preview what would sync, check paths and profiles bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run --log ``` Review the output. Check that: - Share paths resolve correctly on both sides - No pre-flight errors (disk space, rootfs, connectivity) - Profiles look correct for each share --- ### ── Run the initial sync ───────────────────────────────────────────────────── ```bash # On HOST1 — run the full daily sync (this includes rsync for all DAILY_SYNC_SHARES) bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --log ``` This will take longer than future nightly runs — it is transferring everything for the first time. Monitor progress via the Varaverk log viewer or: ```bash # Watch transfer in real time watch -n5 'ls -lh /mnt/user/Movies/ | tail -5' ``` After it finishes, also run the critical sync to populate auth and Emby state: ```bash bash /boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh --log ``` --- ## ━━━ STEP 7: ENABLE FALLBACK ━━━ `fallback.sh` is disabled by default — it should only be enabled when both servers are correctly synced and the data on HOST2 is current. --- ### ── Enable in master.conf ──────────────────────────────────────────────────── ```bash FALLBACK_ENABLED=true ``` Commit and pull on both servers so both pick up the change. --- ### ── Fallback starts automatically on array start ──────────────────────────── `array_started.sh` launches `fallback.sh` as a background process. Once `FALLBACK_ENABLED=true` is in master.conf, restarting the array starts it. To start it now without a reboot: ```bash bash /boot/config/plugins/varaverk/Fallback/fallback.sh & ``` --- ### ── Verify it is running ────────────────────────────────────────────────────── ```bash pgrep -a -f fallback.sh # Should show the running process ``` Check current state: ```bash cat /boot/config/fallback_state.db # state=NORMAL — both servers up ``` --- ## ━━━ STEP 8: TEST FAILOVER ━━━ Before relying on the system, confirm it actually triggers. `fallback_test.sh` simulates an outage using `iptables` — no real downtime, no real data changes. ```bash # On HOST2 — dry run first (sees the sequence, no container starts) bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --dry-run --log # When ready — real test (uses iptables to simulate HOST1 unreachable) bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log ``` The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to detect it and start Tier 1 containers, then unblocks and waits for handback. See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and how to interpret the output. --- ## ━━━ STEP 9: PARTNERSHIP ONBOARD ━━━ The partnership is the optional layer that shares HOST1's auth stack with HOST2 — NPM, Authelia, LLDAP, MariaDB, Redis all warm on HOST2, managed centrally from HOST1. Skip this step if HOST2 does not need HOST1's auth stack (it runs fully independent auth). --- ### ── Prerequisites ──────────────────────────────────────────────────────────── Before running partnership onboard: - Steps 1–8 complete on both servers - HOST1's auth containers are running and healthy - `PARTNERSHIP_AUTH_STACK` in `host1.conf` lists the auth container XML templates - `PARTNERSHIP_ARR_STACK` in `host1.conf` lists arr container XML templates (if sharing arrs) - FolderView3 plugin installed on both servers (if using `PARTNERSHIP_FOLDERVIEW3=true`) --- ### ── Run on HOST2 first (mirror) ──────────────────────────────────────────── HOST2 runs first — it generates its SSH key and waits. The owner completes setup remotely. ```bash # On HOST2 bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log # Review output, then: bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log ``` HOST2's work is done after Step 1 (SSH key setup). The rest happens on HOST1. --- ### ── Run on HOST1 (owner) ──────────────────────────────────────────────────── ```bash # On HOST1 bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log # Review output — this will deploy auth + arr stacks to HOST2 remotely bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log ``` The owner path deploys containers to HOST2 via SSH, configures WebUI targets, runs an initial arr library sync, and writes the partnership state files on both servers. For full detail on what each step does see [Manual-Partnership.md](Partnership/Manual-Partnership.md). --- ## ━━━ STEP 10: VERIFY THE ECOSYSTEM ━━━ --- ### ── Check script output ──────────────────────────────────────────────────── Run the main orchestrators manually with `--log` to see verbose output: ```bash # Watchdogs — should find everything healthy bash Orchestrators/watchdog_orchestrator.sh --log # Daily sync — dry run to confirm share paths and profiles bash Orchestrators/daily_sync_maintenance.sh --dry-run --log ``` --- ### ── Check fallback state ──────────────────────────────────────────────────── ```bash cat /boot/config/fallback_state.db # Expected: # state=NORMAL # fallback_start=0 # handback_strikes=0 # tier2_started=false # tier3_started=false # tier4_started=false ``` --- ### ── Check sync logs ───────────────────────────────────────────────────────── Bandwidth monitor records every sync: ```bash # Latest transfer log ls -lt /boot/config/plugins/varaverk/data/bandwidth_monitor/ cat /boot/config/plugins/varaverk/data/bandwidth_monitor/ ``` --- ### ── Run status on key scripts ───────────────────────────────────────────── ```bash bash Fallback/fallback.sh --status bash Partnership/ssh_setup.sh --status bash Monitors/cert_monitor.sh --status bash Watchdogs/resource_watchdog.sh --status ``` --- ## ━━━ COMMON FIRST-RUN ISSUES ━━━ --- ### ── Script exits silently / does nothing ──────────────────────────────────── ```bash # Run with --log to see verbose output bash