# ━━━━━ VARAVERK — Setup Manual ━━━━━

Getting a fresh two-server ecosystem running from scratch.  
For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.

**Ten steps, and they're in this order for a reason.** Each one assumes the previous one
actually worked — not that you ran it, that it *worked*. Every step below ends with a way to
check, and skipping those checks is how you end up three steps later debugging the wrong
thing entirely.

The steps that hurt most when rushed are **3** (naming) and **8** (testing failover). Step 3
because renaming anything afterwards means chasing it through every conf, and step 8 because
untested failover isn't redundancy — it's a guess you haven't checked yet.

Budget an evening. It is not hard, but it is not five minutes either.

---

## ━━━ BEFORE YOU START ━━━

None of this is optional and none of it is Varaverk's job to install. Get these in place
first — every step after here assumes they're already true.

---

### ── Two unRAID Servers ───────────────────────────────────────────────────────

Both servers need to be up and have their arrays started. Hardware does not need to
match. The scripts work via `/mnt/user/` — the same share name on both servers resolves
correctly regardless of the hardware underneath.

Decide which server is **HOST1 (owner)** and which is **HOST2 (mirror)**. This is just a
label — both servers run their own full stack. The owner manages the shared auth
configuration. Pick the more capable or more reliably-online server as owner.

---

### ── Plugins — install on both servers ────────────────────────────────────────

Install these via **Apps** (Community Applications) on both servers:

| Plugin | Why |
|--------|-----|
| **Tailscale** | Encrypted VPN between servers — all script traffic travels over it |

---

## ━━━ STEP 1: TAILSCALE ━━━

Both servers must be visible to each other on the same Tailscale network.

---

### ── Connect both servers ────────────────────────────────────────────────────

1. Open the Tailscale plugin on HOST1. Log in with your Tailscale account.
2. Open the Tailscale plugin on HOST2. Log in with the **same account** — or have the
   HOST2 owner share their machine into your tailnet (Tailscale → Share → invite by email).
3. Confirm both devices appear in your Tailscale admin panel.

---

### ── Verify reachability ──────────────────────────────────────────────────────

On HOST1:
```bash
# Get your Tailscale IP
tailscale ip -4

# Ping HOST2 by its Tailscale hostname (the unRAID server name, e.g. unRAID-Jayred365)
ping -c 3 unRAID-Jayred365
```

On HOST2 — same in reverse. Both pings must succeed before continuing.

---

### ── Note the Tailscale hostnames ──────────────────────────────────────────────

The scripts use Tailscale DNS to resolve IPs — no hardcoded addresses anywhere.
For this to work, the Tailscale device name must match the unRAID server hostname exactly.

Check the unRAID hostname: **Settings → System Identification → Server Name**  
Check the Tailscale device name: Tailscale admin panel → Machines

If they differ, rename the Tailscale device to match the unRAID hostname. Case matters.

```bash
# On each server — confirm the hostname the scripts will use
hostname
# Should return e.g. "unRAID-Gmer4Lfe"
```

---

## ━━━ STEP 2: CLONE THE REPOSITORY ━━━

---

### ── Clone on HOST1 ───────────────────────────────────────────────────────────

```bash
git clone <your_repo_url> /boot/config/plugins/varaverk
cd /boot/config/plugins/varaverk
```

---

### ── Clone on HOST2 ───────────────────────────────────────────────────────────

```bash
git clone <your_repo_url> /boot/config/plugins/varaverk
cd /boot/config/plugins/varaverk
```

---

### ── Sparse checkout — keep credentials separate ────────────────────────────

`host1.conf` contains HOST1's SSH keys, API keys, and passwords.
`host2.conf` contains HOST2's equivalents. Neither server should hold the other's
credentials. Set up sparse checkout so each server only receives its own conf file.

**On HOST1** — exclude host2.conf:
```bash
cd /boot/config/plugins/varaverk
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/Configurations/host2.conf'
git checkout
```

**On HOST2** — exclude host1.conf:
```bash
cd /boot/config/plugins/varaverk
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/Configurations/host1.conf'
git checkout
```

Verify: `ls Configurations/` — each server should see `master.conf` and only its own `host*.conf`.

> If you are not using git (deploying manually) skip this and ensure you only copy
> each server's own host*.conf when deploying.

---

## ━━━ STEP 3: CONFIGURE ━━━

Configuration lives in `Configurations/`. Three files:
- `master.conf` — shared, both servers see it — thresholds, toggles, schedules
- `host1.conf` — HOST1 only — credentials, share lists, container names
- `host2.conf` — HOST2 only — same structure, different values

Edit these with any text editor. Everything in the scripts is controlled through these
files — never edit scripts directly to change behaviour.

---

### ── 3a. master.conf — server hostnames (do this first) ─────────────────────

Open `Configurations/master.conf`. Find the HOST IDENTITIES section near the top:

```bash
HOST1="unRAID-Gmer4Lfe"
HOST2="unRAID-Jayred365"
```

Replace these with your actual unRAID server hostnames. These must match exactly —
`detect_hosts()` compares the running server's hostname against these two values to
know which server it is on. Everything else in the ecosystem flows from this.

**Get this wrong and nothing works, but nothing errors either.** If neither value matches,
`MY_ID` is never set, and every script that depends on it either exits early or resolves
`${MY_ID}_SOMETHING` to an empty variable and quietly takes the wrong branch. Copy the value
straight out of `hostname` on each box rather than typing what you think it is:

```bash
hostname     # run this on each server, paste the exact output
```

> **The 15-character trap.** unRAID truncates the Server Name to 15 characters for NetBIOS.
> If your name is longer, what you set in the WebGUI and what `hostname` returns are two
> different strings. `detect_hosts()` has a fallback that matches a truncated 15-char
> hostname against a longer configured value — but only when the live hostname is
> *exactly* 15 characters. Don't rely on it. Use the real `hostname` output.

---

### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────

Open `Configurations/host1.conf`. Fill in the IDENTITY section:

```bash
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"   # path — ssh_setup.sh will create this
HOST1_OWNER="gmer4lfe"                                  # short name, lowercase, no spaces
HOST1_OWNER_EMAIL="you@example.com"                     # for partnership notifications
```

The SSH key does not exist yet — `ssh_setup.sh` in Step 4 will create it and fill in
the path automatically. Leave the path as-is for now; it will be overwritten.

---

### ── 3c. host2.conf — HOST2 identity ────────────────────────────────────────

Open `Configurations/host2.conf`. Fill in the same fields with HOST2's values:

```bash
HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"
HOST2_OWNER="jayred365"
HOST2_OWNER_EMAIL="them@example.com"
```

---

### ── 3d. Remaining configuration ────────────────────────────────────────────

The scripts will not break if the remaining fields are empty — they self-guard
when things are not configured. Fill in sections as you enable each feature. The
minimum set needed to get a working sync + fallback:

**In host1.conf (and host2.conf mirror):**
```bash
# Shares HOST1 pushes to HOST2 nightly
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"
    "/mnt/user/Tv_Shows"
    "/mnt/user/Music"
)

# Shares synced every 30 minutes (auth config + Emby watch state)
HOST1_CRITICAL_SYNC_SHARES=(
    "/mnt/user/appdata-Fallback/Critical-Data|critical-data"
)

# Containers to start when HOST1 goes down (Tier 1 = immediate)
# Lives in host1.conf. Named for the host being COVERED, not the host doing the covering.
FALLBACK_HOST1_TIER1=(
    "Emby"
    "VaultWarden"
    "NginxProxyManager"
)
```

**Read that naming carefully — it catches people.** The variable is
`FALLBACK_${REMOTE_ID}_TIER${N}`, and `REMOTE_ID` is *the host that went down*. So
`FALLBACK_HOST1_TIER1` is the list HOST2 reads when HOST1 is unreachable. It defines
"what HOST1 needs covering", not "what HOST2 runs".

It lives in `host1.conf` for the same reason — HOST1 owns the description of its own
service stack. HOST2 receives it through the conf cache rather than keeping its own
opinion about what HOST1 runs. One list, one owner, no drift.

Tiers 2–4 activate progressively after `HOST1_TIER2_DELAY` etc., so a brief outage never
drags the entire remote stack across.

For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
and `Manual-Watchdogs.md`.

---

## ━━━ STEP 4: SSH KEYS ━━━

The servers need passwordless SSH access to each other. `ssh_setup.sh` handles
key generation and remote installation in one step.

---

### ── Run on HOST2 first ────────────────────────────────────────────────────

The mirror runs first — it generates its own key and copies it to HOST1. You will be
prompted for HOST1's root password once (for the initial key copy). After that,
all SSH is keyless.

```bash
cd /boot/config/plugins/varaverk
bash Partnership/ssh_setup.sh
```

When prompted, enter HOST1's root password. The script:
- Generates `/root/.ssh/jayred365_rsync_automation` (ed25519)
- Copies the public key to HOST1's `~/.ssh/authorized_keys`
- Updates `host2.conf` with the key path

---

### ── Run on HOST1 ──────────────────────────────────────────────────────────

```bash
cd /boot/config/plugins/varaverk
bash Partnership/ssh_setup.sh
```

Enter HOST2's root password when prompted. Same steps, opposite direction.

---

### ── Verify both directions ────────────────────────────────────────────────

```bash
# On HOST1 — SSH to HOST2 should succeed without a password
ssh -i /root/.ssh/gmer4lfe_rsync_automation root@unRAID-Jayred365 "hostname"
# Expected: unRAID-Jayred365

# On HOST2 — SSH to HOST1 should succeed without a password
ssh -i /root/.ssh/jayred365_rsync_automation root@unRAID-Gmer4Lfe "hostname"
# Expected: unRAID-Gmer4Lfe
```

If either fails:
```bash
# Check status and fingerprints
bash Partnership/ssh_setup.sh --status

# Re-run with --force to regenerate and re-copy
bash Partnership/ssh_setup.sh --force
```

---

### ── Tailscale SSH note ──────────────────────────────────────────────────────

The scripts SSH via Tailscale hostnames, not local IPs. The first connection to a
new host requires accepting the host key. Accept it now so scripts never block on an
interactive prompt:

```bash
# On HOST1
ssh -o StrictHostKeyChecking=accept-new root@unRAID-Jayred365 "echo ok"

# On HOST2
ssh -o StrictHostKeyChecking=accept-new root@unRAID-Gmer4Lfe "echo ok"
```

---

## ━━━ STEP 5: SCHEDULER SETUP ━━━

**Everything runs through the Varaverk plugin's built-in scheduler** — no User Scripts
entries needed. The plugin handles all triggers natively:

- **Array start** → `Plugin/unraid/event/disks_mounted/array_start_jobs` fires `array_started.sh`
- **Array stop** → `Plugin/unraid/event/disks_unmounting/array_stop_jobs` fires `array_stopping.sh`
- **Cron** → `Plugin/unraid/event/disks_mounted/rebuild_cron` rebuilds the cron file from `schedule.json` on every boot

Configure via the Varaverk plugin Scheduler tab (or edit `schedule.json` directly).
The built-in job list schedules orchestrators, never individual repo scripts directly —
but the Scheduler tab's **Custom Scripts** card is the one place individual scripts
*are* scheduled directly (see below).

---

### ── Custom Scripts ────────────────────────────────────────────────────────────

The Scheduler tab has a **Custom Scripts** card for one-off scripts that aren't part of
the repo's orchestrator pipeline — personal tooling, quick fixes, anything you don't
want to wire into `master.conf`.

Scripts live in `/boot/config/plugins/user.scripts/Varaverk/Scripts/` — deliberately
**outside** the Varaverk git repo (that folder is never pushed to GitHub), in the same
place the Unraid User Scripts plugin keeps its own scripts, so it's a folder location
admins are already used to.

Two ways to get a script there:

- Click **+ Create Script** on the Scheduler tab — opens an inline editor, writes the
  file to that folder, and adds a `schedule.json` entry automatically.
- Drop any `.sh` file into the folder yourself (e.g. via terminal, or Unraid's own
  Custom Scripts / User Scripts plugin pointed at the same path). The Scheduler tab
  **auto-detects** it — discovery is a folder scan, not a registry, so it doesn't matter
  how the file got there. It shows up disabled with no cron until you configure one.

---

### ── Schedule (Varaverk Scheduler) ───────────────────────────────────────────

| Script | Event/Cron | Purpose |
|--------|-----------|---------|
| `array_started` | `array_start` | All array startup scripts in order |
| `array_stopping` | `array_stop` | Ordered graceful shutdown |
| `transcode_management` | `*/7 * * * *` | Cleanup then manager — order critical |
| `watchdog_orchestrator` | `*/15 * * * *` | resource → docker → system → stability |
| `critical_sync_maintenance` | `*/30 * * * *` | auth + Emby dirty sync + partnership |
| `intermediate_sync_maintenance` | `0 */4 * * *` | arr sync + failed recovery |
| `daily_sync_maintenance` | `0 1 * * *` | full daily maintenance window |
| `weekly_sync_maintenance` | `30 2 * * 0` | clean sync + image updates |
| `monthly_maintenance` | `0 0 15 * *` | ZFS scrub, SMART tests (uptime-gated) |

Concurrent runs are prevented by `acquire_lock`. For the complete schedule see
[README-Orchestrators.md](Orchestrators/README-Orchestrators.md).

---

## ━━━ STEP 6: INITIAL SYNC ━━━

Before enabling fallback, HOST2 needs a copy of HOST1's data. This first sync
will take a while depending on library size.

---

### ── Dry run first ────────────────────────────────────────────────────────────

```bash
# On HOST1 — preview what would sync, check paths and profiles
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run --log
```

Review the output. Check that:
- Share paths resolve correctly on both sides
- No pre-flight errors (disk space, rootfs, connectivity)
- Profiles look correct for each share

---

### ── Run the initial sync ─────────────────────────────────────────────────────

```bash
# On HOST1 — run the full daily sync (this includes rsync for all DAILY_SYNC_SHARES)
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --log
```

This will take longer than future nightly runs — it is transferring everything for
the first time. Monitor progress via the Varaverk log viewer or:

```bash
# Watch transfer in real time
watch -n5 'ls -lh /mnt/user/Movies/ | tail -5'
```

After it finishes, also run the critical sync to populate auth and Emby state:
```bash
bash /boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh --log
```

---

## ━━━ STEP 7: ENABLE FALLBACK ━━━

`fallback.sh` is disabled by default — it should only be enabled when both servers
are correctly synced and the data on HOST2 is current.

---

### ── Enable in master.conf ────────────────────────────────────────────────────

```bash
FALLBACK_ENABLED=true
```

Commit and pull on both servers so both pick up the change.

---

### ── Fallback starts automatically on array start ────────────────────────────

`array_started.sh` launches `fallback.sh` as a background process. Once
`FALLBACK_ENABLED=true` is in master.conf, restarting the array starts it.

To start it now without a reboot:
```bash
bash /boot/config/plugins/varaverk/Fallback/fallback.sh &
```

---

### ── Verify it is running ──────────────────────────────────────────────────────

```bash
pgrep -a -f fallback.sh
# Should show the running process
```

Check current state:
```bash
cat "$STATE_DIR/fallback_state.db"
# state=NORMAL — both servers up
```

---

## ━━━ STEP 8: TEST FALLBACK ━━━

Before relying on the system, confirm it actually triggers. `fallback_test.sh`
simulates an outage using `iptables` — no real downtime, no real data changes.

```bash
# On HOST2 — dry run first (sees the sequence, no container starts)
bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --dry-run --log

# When ready — real test (uses iptables to simulate HOST1 unreachable)
bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log
```

The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
detect it and start Tier 1 containers, then unblocks and waits for handback.

**Do not skip this step.** Everything up to here you can verify by reading output. Failover
is the one part you cannot confirm by looking at it — a typo in a tier list, a container name
that doesn't exist on the other side, a DDNS container that was renamed six months ago: all
of it sits there looking completely fine until the moment it's needed.

The test costs you twenty minutes and no downtime. The alternative is finding out at 2am,
during the outage, when you have exactly one chance to get it right.

**Watch the handback as closely as the failover.** Coming back is the harder half — it has to
stop the remote's DDNS, sync the data back, start the local containers, and only then bring
local DDNS up. Failover starting correctly tells you nothing about whether handback does.

See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
how to interpret the output.

---

## ━━━ STEP 9: PARTNERSHIP ONBOARD ━━━

The partnership is the optional layer that shares HOST1's auth stack with HOST2 —
NPM, Authelia, LLDAP, MariaDB, Redis all warm on HOST2, managed centrally from HOST1.

Skip this step if HOST2 does not need HOST1's auth stack (it runs fully independent auth).

---

### ── Prerequisites ────────────────────────────────────────────────────────────

Before running partnership onboard:

- Steps 1–8 complete on both servers
- HOST1's auth containers are running and healthy
- `PARTNERSHIP_AUTH_STACK` in `host1.conf` lists the auth container XML templates
- `PARTNERSHIP_ARR_STACK` in `host1.conf` lists arr container XML templates (if sharing arrs)
- FolderView3 plugin installed on both servers (if using `PARTNERSHIP_FOLDERVIEW3=true`)

---

### ── Run on HOST2 first (mirror) ────────────────────────────────────────────

HOST2 runs first — it generates its SSH key and waits. The owner completes setup remotely.

```bash
# On HOST2
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
# Review output, then:
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log
```

HOST2's work is done after Step 1 (SSH key setup). The rest happens on HOST1.

---

### ── Run on HOST1 (owner) ────────────────────────────────────────────────────

```bash
# On HOST1
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
# Review output — this will deploy auth + arr stacks to HOST2 remotely
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log
```

The owner path deploys containers to HOST2 via SSH, configures WebUI targets, runs
an initial arr library sync, and writes the partnership state files on both servers.

For full detail on what each step does see
[Manual-Partnership.md](Partnership/Manual-Partnership.md).

---

## ━━━ STEP 10: VERIFY THE ECOSYSTEM ━━━

---

### ── Check script output ────────────────────────────────────────────────────

Run the main orchestrators manually with `--log` to see verbose output:

```bash
# Watchdogs — should find everything healthy
bash Orchestrators/watchdog_orchestrator.sh --log

# Daily sync — dry run to confirm share paths and profiles
bash Orchestrators/daily_sync_maintenance.sh --dry-run --log
```

---

### ── Check fallback state ────────────────────────────────────────────────────

```bash
cat "$STATE_DIR/fallback_state.db"
# Expected:
# state=NORMAL
# fallback_start=0
# handback_strikes=0
# tier2_started=false
# tier3_started=false
# tier4_started=false
```

---

### ── Check sync logs ─────────────────────────────────────────────────────────

Bandwidth monitor records every sync:
```bash
# Latest transfer log
ls -lt /boot/config/plugins/varaverk/data/bandwidth_monitor/
cat /boot/config/plugins/varaverk/data/bandwidth_monitor/<latest_log>
```

---

### ── Run status on key scripts ─────────────────────────────────────────────

```bash
bash Fallback/fallback.sh --status
bash Partnership/ssh_setup.sh --status
bash Monitors/cert_monitor.sh --status
bash Watchdogs/resource_watchdog.sh --status
```

---

## ━━━ COMMON FIRST-RUN ISSUES ━━━

---

### ── Script exits silently / does nothing ────────────────────────────────────

```bash
# Run with --log to see verbose output
bash <script> --log

# Check if the script is skipping due to a toggle being false
grep "ENABLED\|_TOGGLE\|ENABLED=" Configurations/master.conf | grep "false"
```

Most scripts have an `ENABLED=true/false` toggle in `master.conf`. Check that the
feature you are testing is enabled.

---

### ── SSH connection refused or times out ────────────────────────────────────

```bash
# Verify Tailscale is running and the remote is reachable
tailscale status
ping -c 3 <remote_hostname>

# Verify the key is authorised on the remote
ssh -v -i /root/.ssh/<your_key> root@<remote_hostname> "echo ok"

# Re-run ssh_setup.sh if the key is not installed
bash Partnership/ssh_setup.sh --force
```

---

### ── detect_hosts() exits — MY_ID unknown ────────────────────────────────────

```bash
# The running server's hostname doesn't match HOST1 or HOST2 in master.conf
hostname
# Compare to master.conf
grep "^    HOST[12]=" Configurations/master.conf
# They must match exactly — case sensitive
```

---

### ── Fallback triggers immediately on startup ────────────────────────────────

The remote server is unreachable on the first check. Common causes:
- Tailscale not running on the remote
- Remote server is actually offline
- `HOST1`/`HOST2` hostnames in `master.conf` don't match Tailscale device names

```bash
# Confirm which state fallback is in
cat "$STATE_DIR/fallback_state.db"
# If stuck in FALLBACK after remote comes back: reset state
bash Tools/fallback_state_reset.sh
```

---

### ── Rsync fails — remote share not found ─────────────────────────────────────

```bash
# Check the share exists on the remote
ssh -i /root/.ssh/<your_key> root@<remote_hostname> "ls /mnt/user/"

# Share names must be identical on both servers
# /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2
# A mismatch = rsync writes to a non-existent path or aborts pre-flight
```

---

## ━━━ WHAT'S NEXT ━━━

Once the core is running, enable and configure the remaining features at your own pace.
Each subsystem has a README with the design decisions and a Manual with the config
reference and troubleshooting. The `--status` flag on any script shows the current
configuration and state.

**Turn things on one at a time and give each one a few days.** Everything below is off by
default on a fresh install, and that's deliberate — a stack where six new subsystems went
live the same night is a stack where you have no idea which one to blame. Enable, watch it
through a full daily cycle, then enable the next.

Anything that deletes files — the arr cleanup scripts especially — gets a `--dry-run --log`
first. Read the list. Every time, not just the first time.

```
Watchdogs/        → README-Watchdogs.md        configure memory limits, container lists
Media/            → README-Media.md             enable arr cleanup, discovery scripts
Monitors/         → README-Monitors.md          cert monitoring, SMART, health digest
Transcodes/       → README-Transcoding.md       ramdisk setup for Emby transcode
Partnership/      → README-Partnership.md       auth stack sharing, full lifecycle
```
