From 2e04821b75364f1d6bee1700eb9c4761d7db074c Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 23 May 2026 10:23:27 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20add=20Manual.md=20=E2=80=94=20end-to-en?= =?UTF-8?q?d=20setup=20guide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step-by-step deployment guide: prerequisites, Tailscale, git clone, sparse checkout, master.conf/host*.conf config, SSH keys via ssh_setup.sh, Arbiter cron table, initial sync, fallback enable, partnership onboard, verification, and common first-run troubleshooting. Fixes the broken README-Rsync_Setup.md reference in README.md. --- Manual.md | 670 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 4 +- 2 files changed, 672 insertions(+), 2 deletions(-) create mode 100644 Manual.md diff --git a/Manual.md b/Manual.md new file mode 100644 index 0000000..ef10845 --- /dev/null +++ b/Manual.md @@ -0,0 +1,670 @@ +# ━━━━━ VARAVERK — Setup Manual ━━━━━ + +Getting a fresh two-server ecosystem running from scratch. +For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers. + +--- + +## ━━━ BEFORE YOU START ━━━ + +Things that must be in place before you touch any scripts. + +--- + +### ── Two unRAID Servers ─────────────────────────────────────────────────────── + +Both servers need to be up and have their arrays started. Hardware does not need to +match. The scripts work via `/mnt/user/` — the same share name on both servers resolves +correctly regardless of the hardware underneath. + +Decide which server is **HOST1 (owner)** and which is **HOST2 (mirror)**. This is just a +label — both servers run their own full stack. The owner manages the shared auth +configuration. Pick the more capable or more reliably-online server as owner. + +--- + +### ── Plugins — install on both servers ──────────────────────────────────────── + +Install these via **Apps** (Community Applications) on both servers: + +| Plugin | Why | +|--------|-----| +| **Tailscale** | Encrypted VPN between servers — all script traffic travels over it | +| **User Scripts** | Schedules the orchestrators (replaces per-script cron entries) | + +--- + +### ── CLI Tools — install on both servers ──────────────────────────────────── + +Required by multiple scripts. Install via **Tools → NerdPack** (install NerdPack first +from Apps if not present): + +| Tool | Why | +|------|-----| +| `git` | Pulling script updates, version tracking | +| `jq` | JSON parsing — arr APIs, Tailscale API, Docker inspect | +| `curl` | API calls — arrs, Emby, Tailscale, Discord webhooks | +| `openssl` | Certificate monitoring (`cert_monitor.sh`) | + +Verify after install: +```bash +git --version && jq --version && curl --version | head -1 && openssl version +``` + +--- + +## ━━━ STEP 1: TAILSCALE ━━━ + +Both servers must be visible to each other on the same Tailscale network. + +--- + +### ── Connect both servers ──────────────────────────────────────────────────── + +1. Open the Tailscale plugin on HOST1. Log in with your Tailscale account. +2. Open the Tailscale plugin on HOST2. Log in with the **same account** — or have the + HOST2 owner share their machine into your tailnet (Tailscale → Share → invite by email). +3. Confirm both devices appear in your Tailscale admin panel. + +--- + +### ── Verify reachability ────────────────────────────────────────────────────── + +On HOST1: +```bash +# Get your Tailscale IP +tailscale ip -4 + +# Ping HOST2 by its Tailscale hostname (the unRAID server name, e.g. unRAID-Jayred365) +ping -c 3 unRAID-Jayred365 +``` + +On HOST2 — same in reverse. Both pings must succeed before continuing. + +--- + +### ── Note the Tailscale hostnames ────────────────────────────────────────────── + +The scripts use Tailscale DNS to resolve IPs — no hardcoded addresses anywhere. +For this to work, the Tailscale device name must match the unRAID server hostname exactly. + +Check the unRAID hostname: **Settings → System Identification → Server Name** +Check the Tailscale device name: Tailscale admin panel → Machines + +If they differ, rename the Tailscale device to match the unRAID hostname. Case matters. + +```bash +# On each server — confirm the hostname the scripts will use +hostname +# Should return e.g. "unRAID-Gmer4Lfe" +``` + +--- + +## ━━━ STEP 2: CLONE THE REPOSITORY ━━━ + +--- + +### ── Clone on HOST1 ─────────────────────────────────────────────────────────── + +```bash +git clone /mnt/user/appdata/unraid_scripts +cd /mnt/user/appdata/unraid_scripts +``` + +--- + +### ── Clone on HOST2 ─────────────────────────────────────────────────────────── + +```bash +git clone /mnt/user/appdata/unraid_scripts +cd /mnt/user/appdata/unraid_scripts +``` + +--- + +### ── Sparse checkout — keep credentials separate ──────────────────────────── + +`host1.conf` contains HOST1's SSH keys, API keys, and passwords. +`host2.conf` contains HOST2's equivalents. Neither server should hold the other's +credentials. Set up sparse checkout so each server only receives its own conf file. + +**On HOST1** — exclude host2.conf: +```bash +cd /mnt/user/appdata/unraid_scripts +git sparse-checkout init --no-cone +git sparse-checkout set '/*' '!/Configurations/host2.conf' +git checkout +``` + +**On HOST2** — exclude host1.conf: +```bash +cd /mnt/user/appdata/unraid_scripts +git sparse-checkout init --no-cone +git sparse-checkout set '/*' '!/Configurations/host1.conf' +git checkout +``` + +Verify: `ls Configurations/` — each server should see `master.conf` and only its own `host*.conf`. + +> If you are not using git (deploying manually) skip this and ensure you only copy +> each server's own host*.conf when deploying. + +--- + +## ━━━ STEP 3: CONFIGURE ━━━ + +Configuration lives in `Configurations/`. Three files: +- `master.conf` — shared, both servers see it — thresholds, toggles, schedules +- `host1.conf` — HOST1 only — credentials, share lists, container names +- `host2.conf` — HOST2 only — same structure, different values + +Edit these with any text editor. Everything in the scripts is controlled through these +files — never edit scripts directly to change behaviour. + +--- + +### ── 3a. master.conf — server hostnames (do this first) ───────────────────── + +Open `Configurations/master.conf`. Find the HOST IDENTITIES section near the top: + +```bash +HOST1="unRAID-Gmer4Lfe" +HOST2="unRAID-Jayred365" +``` + +Replace these with your actual unRAID server hostnames. These must match exactly — +`detect_hosts()` compares the running server's hostname against these two values to +know which server it is on. Everything else in the ecosystem flows from this. + +--- + +### ── 3b. host1.conf — HOST1 identity ──────────────────────────────────────── + +Open `Configurations/host1.conf`. Fill in the IDENTITY section: + +```bash +HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation" # path — ssh_setup.sh will create this +HOST1_OWNER="gmer4lfe" # short name, lowercase, no spaces +HOST1_OWNER_EMAIL="you@example.com" # for partnership notifications +``` + +The SSH key does not exist yet — `ssh_setup.sh` in Step 4 will create it and fill in +the path automatically. Leave the path as-is for now; it will be overwritten. + +--- + +### ── 3c. host2.conf — HOST2 identity ──────────────────────────────────────── + +Open `Configurations/host2.conf`. Fill in the same fields with HOST2's values: + +```bash +HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation" +HOST2_OWNER="jayred365" +HOST2_OWNER_EMAIL="them@example.com" +``` + +--- + +### ── 3d. Remaining configuration ──────────────────────────────────────────── + +The scripts will not break if the remaining fields are empty — they self-guard +when things are not configured. Fill in sections as you enable each feature. The +minimum set needed to get a working sync + fallback: + +**In host1.conf (and host2.conf mirror):** +```bash +# Shares HOST1 pushes to HOST2 nightly +HOST1_DAILY_SYNC_SHARES=( + "/mnt/user/Movies" + "/mnt/user/Tv_Shows" + "/mnt/user/Music" +) + +# Shares synced every 30 minutes (auth config + Emby watch state) +HOST1_CRITICAL_SYNC_SHARES=( + "/mnt/user/appdata-Fallback/Critical-Data|critical-data" +) + +# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate) +HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=() # HOST2's own covers for HOST1, and vice versa +HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=( + "Emby" + "VaultWarden" + "NginxProxyManager" +) +``` + +For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`, +and `Manual-Watchdogs.md`. + +--- + +## ━━━ STEP 4: SSH KEYS ━━━ + +The servers need passwordless SSH access to each other. `ssh_setup.sh` handles +key generation and remote installation in one step. + +--- + +### ── Run on HOST2 first ──────────────────────────────────────────────────── + +The mirror runs first — it generates its own key and copies it to HOST1. You will be +prompted for HOST1's root password once (for the initial key copy). After that, +all SSH is keyless. + +```bash +cd /mnt/user/appdata/unraid_scripts +bash Partnership/ssh_setup.sh +``` + +When prompted, enter HOST1's root password. The script: +- Generates `/root/.ssh/jayred365_rsync_automation` (ed25519) +- Copies the public key to HOST1's `~/.ssh/authorized_keys` +- Updates `host2.conf` with the key path + +--- + +### ── Run on HOST1 ────────────────────────────────────────────────────────── + +```bash +cd /mnt/user/appdata/unraid_scripts +bash Partnership/ssh_setup.sh +``` + +Enter HOST2's root password when prompted. Same steps, opposite direction. + +--- + +### ── Verify both directions ──────────────────────────────────────────────── + +```bash +# On HOST1 — SSH to HOST2 should succeed without a password +ssh -i /root/.ssh/gmer4lfe_rsync_automation root@unRAID-Jayred365 "hostname" +# Expected: unRAID-Jayred365 + +# On HOST2 — SSH to HOST1 should succeed without a password +ssh -i /root/.ssh/jayred365_rsync_automation root@unRAID-Gmer4Lfe "hostname" +# Expected: unRAID-Gmer4Lfe +``` + +If either fails: +```bash +# Check status and fingerprints +bash Partnership/ssh_setup.sh --status + +# Re-run with --force to regenerate and re-copy +bash Partnership/ssh_setup.sh --force +``` + +--- + +### ── Tailscale SSH note ────────────────────────────────────────────────────── + +The scripts SSH via Tailscale hostnames, not local IPs. The first connection to a +new host requires accepting the host key. Accept it now so scripts never block on an +interactive prompt: + +```bash +# On HOST1 +ssh -o StrictHostKeyChecking=accept-new root@unRAID-Jayred365 "echo ok" + +# On HOST2 +ssh -o StrictHostKeyChecking=accept-new root@unRAID-Gmer4Lfe "echo ok" +``` + +--- + +## ━━━ STEP 5: ARBITER — USER SCRIPTS SETUP ━━━ + +Arbiter (the User Scripts plugin) is the only thing Arbiter runs — never individual +scripts directly. Each entry is an orchestrator that calls everything in its window. + +Open **Settings → User Scripts** in the unRAID WebGUI. + +--- + +### ── Required entries (both servers) ──────────────────────────────────────── + +Create one User Script entry for each row. Set the schedule in the cron field and +paste the command. Name the entry to match the script name. + +| Script | Cron schedule | Command | +|--------|--------------|---------| +| `array_started` | At Array Start | `bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_started.sh` | +| `watchdog_orchestrator` | `* * * * *` | `bash /mnt/user/appdata/unraid_scripts/Orchestrators/watchdog_orchestrator.sh` | +| `transcode_management` | `*/7 * * * *` | `bash /mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh` | +| `critical_sync_maintenance` | `*/30 * * * *` | `bash /mnt/user/appdata/unraid_scripts/Orchestrators/critical_sync_maintenance.sh` | +| `emby_fallback_sync` | `*/30 * * * *` | `bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback` | +| `intermediate_sync_maintenance` | `0 */4 * * *` | `bash /mnt/user/appdata/unraid_scripts/Orchestrators/intermediate_sync_maintenance.sh` | +| `daily_sync_maintenance` | `0 1 * * *` | `bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh` | +| `weekly_sync_maintenance` | `30 2 * * 0` | `bash /mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh` | +| `weekly_health_digest` | `0 8 * * *` | `bash /mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh` | + +> `array_started` uses the built-in "At Array Start" schedule option in User Scripts — +> it is not a cron expression. + +> `emby_fallback_sync` is the only script with a direct Arbiter entry. It is separate +> from `critical_sync_maintenance` because it needs its own 30-minute slot and Emby +> must stay running during this sync (dirty sync — no container stop). + +--- + +### ── Schedule note ──────────────────────────────────────────────────────────── + +`transcode_management` and `critical_sync_maintenance` both run every 7 and 30 minutes +respectively. In User Scripts, concurrent runs of the same script are prevented by the +lock system (`acquire_lock`) — if the previous run is still active the new one exits +immediately. No need to stagger them manually. + +For the complete schedule and what each orchestrator runs, see +[README-Orchestrators.md](Orchestrators/README-Orchestrators.md). + +--- + +## ━━━ STEP 6: INITIAL SYNC ━━━ + +Before enabling fallback, HOST2 needs a copy of HOST1's data. This first sync +will take a while depending on library size. + +--- + +### ── Dry run first ──────────────────────────────────────────────────────────── + +```bash +# On HOST1 — preview what would sync, check paths and profiles +bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --dry-run --log +``` + +Review the output. Check that: +- Share paths resolve correctly on both sides +- No pre-flight errors (disk space, rootfs, connectivity) +- Profiles look correct for each share + +--- + +### ── Run the initial sync ───────────────────────────────────────────────────── + +```bash +# On HOST1 — run the full daily sync (this includes rsync for all DAILY_SYNC_SHARES) +bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --log +``` + +This will take longer than future nightly runs — it is transferring everything for +the first time. Monitor progress via the User Scripts output or: + +```bash +# Watch transfer in real time +watch -n5 'ls -lh /mnt/user/Movies/ | tail -5' +``` + +After it finishes, also run the critical sync to populate auth and Emby state: +```bash +bash /mnt/user/appdata/unraid_scripts/Orchestrators/critical_sync_maintenance.sh --log +``` + +--- + +## ━━━ STEP 7: ENABLE FALLBACK ━━━ + +`fallback.sh` is disabled by default — it should only be enabled when both servers +are correctly synced and the data on HOST2 is current. + +--- + +### ── Enable in master.conf ──────────────────────────────────────────────────── + +```bash +FALLBACK_ENABLED=true +``` + +Commit and pull on both servers so both pick up the change. + +--- + +### ── Fallback starts automatically on array start ──────────────────────────── + +`array_started.sh` launches `fallback.sh` as a background process. Once +`FALLBACK_ENABLED=true` is in master.conf, restarting the array starts it. + +To start it now without a reboot: +```bash +bash /mnt/user/appdata/unraid_scripts/Fallback/fallback.sh & +``` + +--- + +### ── Verify it is running ────────────────────────────────────────────────────── + +```bash +pgrep -a -f fallback.sh +# Should show the running process +``` + +Check current state: +```bash +cat /boot/config/fallback_state.db +# state=NORMAL — both servers up +``` + +--- + +## ━━━ STEP 8: TEST FAILOVER ━━━ + +Before relying on the system, confirm it actually triggers. `fallback_test.sh` +simulates an outage using `iptables` — no real downtime, no real data changes. + +```bash +# On HOST2 — dry run first (sees the sequence, no container starts) +bash /mnt/user/appdata/unraid_scripts/Fallback/fallback_test.sh --dry-run --log + +# When ready — real test (uses iptables to simulate HOST1 unreachable) +bash /mnt/user/appdata/unraid_scripts/Fallback/fallback_test.sh --log +``` + +The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to +detect it and start Tier 1 containers, then unblocks and waits for handback. + +See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and +how to interpret the output. + +--- + +## ━━━ STEP 9: PARTNERSHIP ONBOARD ━━━ + +The partnership is the optional layer that shares HOST1's auth stack with HOST2 — +NPM, Authelia, LLDAP, MariaDB, Redis all warm on HOST2, managed centrally from HOST1. + +Skip this step if HOST2 does not need HOST1's auth stack (it runs fully independent auth). + +--- + +### ── Prerequisites ──────────────────────────────────────────────────────────── + +Before running partnership onboard: + +- Steps 1–8 complete on both servers +- HOST1's auth containers are running and healthy +- `PARTNERSHIP_AUTH_STACK` in `host1.conf` lists the auth container XML templates +- `PARTNERSHIP_ARR_STACK` in `host1.conf` lists arr container XML templates (if sharing arrs) +- FolderView3 plugin installed on both servers (if using `PARTNERSHIP_FOLDERVIEW3=true`) + +--- + +### ── Run on HOST2 first (mirror) ──────────────────────────────────────────── + +HOST2 runs first — it generates its SSH key and waits. The owner completes setup remotely. + +```bash +# On HOST2 +bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --dry-run --log +# Review output, then: +bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --log +``` + +HOST2's work is done after Step 1 (SSH key setup). The rest happens on HOST1. + +--- + +### ── Run on HOST1 (owner) ──────────────────────────────────────────────────── + +```bash +# On HOST1 +bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --dry-run --log +# Review output — this will deploy auth + arr stacks to HOST2 remotely +bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --log +``` + +The owner path deploys containers to HOST2 via SSH, configures WebUI targets, runs +an initial arr library sync, and writes the partnership state files on both servers. + +For full detail on what each step does see +[Manual-Partnership.md](Partnership/Manual-Partnership.md). + +--- + +## ━━━ STEP 10: VERIFY THE ECOSYSTEM ━━━ + +--- + +### ── Check script output ──────────────────────────────────────────────────── + +Run the main orchestrators manually with `--log` to see verbose output: + +```bash +# Watchdogs — should find everything healthy +bash Orchestrators/watchdog_orchestrator.sh --log + +# Daily sync — dry run to confirm share paths and profiles +bash Orchestrators/daily_sync_maintenance.sh --dry-run --log +``` + +--- + +### ── Check fallback state ──────────────────────────────────────────────────── + +```bash +cat /boot/config/fallback_state.db +# Expected: +# state=NORMAL +# fallback_start=0 +# handback_strikes=0 +# tier2_started=false +# tier3_started=false +# tier4_started=false +``` + +--- + +### ── Check sync logs ───────────────────────────────────────────────────────── + +Bandwidth monitor records every sync: +```bash +# Latest transfer log +ls -lt /mnt/user/appdata/unraid_scripts/data/bandwidth_monitor/ +cat /mnt/user/appdata/unraid_scripts/data/bandwidth_monitor/ +``` + +--- + +### ── Run status on key scripts ───────────────────────────────────────────── + +```bash +bash Fallback/fallback.sh --status +bash Partnership/ssh_setup.sh --status +bash Monitors/cert_monitor.sh --status +bash Watchdogs/resource_watchdog.sh --status +``` + +--- + +## ━━━ COMMON FIRST-RUN ISSUES ━━━ + +--- + +### ── Script exits silently / does nothing ──────────────────────────────────── + +```bash +# Run with --log to see verbose output +bash