Varaverk/Manual.md

# ━━━━━ VARAVERK — Setup Manual ━━━━━

Getting a fresh two-server ecosystem running from scratch.  
For system overview see [README.md](README.md). For individual subsystem detail see folder READMEs and script headers.

---

## ━━━ BEFORE YOU START ━━━

Things that must be in place before you touch any scripts.

---

### ── Two unRAID Servers ───────────────────────────────────────────────────────

Both servers need to be up and have their arrays started. Hardware does not need to
match. The scripts work via `/mnt/user/` — the same share name on both servers resolves
correctly regardless of the hardware underneath.

Decide which server is **HOST1 (owner)** and which is **HOST2 (mirror)**. This is just a
label — both servers run their own full stack. The owner manages the shared auth
configuration. Pick the more capable or more reliably-online server as owner.

---

### ── Plugins — install on both servers ────────────────────────────────────────

Install these via **Apps** (Community Applications) on both servers:

| Plugin | Why |
|--------|-----|
| **Tailscale** | Encrypted VPN between servers — all script traffic travels over it |
| **User Scripts** | Required for "At Startup of Array" hook — array_started.sh entry only |

---

## ━━━ STEP 1: TAILSCALE ━━━

Both servers must be visible to each other on the same Tailscale network.

---

### ── Connect both servers ────────────────────────────────────────────────────

1. Open the Tailscale plugin on HOST1. Log in with your Tailscale account.
2. Open the Tailscale plugin on HOST2. Log in with the **same account** — or have the
   HOST2 owner share their machine into your tailnet (Tailscale → Share → invite by email).
3. Confirm both devices appear in your Tailscale admin panel.

---

### ── Verify reachability ──────────────────────────────────────────────────────

On HOST1:
```bash
# Get your Tailscale IP
tailscale ip -4

# Ping HOST2 by its Tailscale hostname (the unRAID server name, e.g. unRAID-Jayred365)
ping -c 3 unRAID-Jayred365
```

On HOST2 — same in reverse. Both pings must succeed before continuing.

---

### ── Note the Tailscale hostnames ──────────────────────────────────────────────

The scripts use Tailscale DNS to resolve IPs — no hardcoded addresses anywhere.
For this to work, the Tailscale device name must match the unRAID server hostname exactly.

Check the unRAID hostname: **Settings → System Identification → Server Name**  
Check the Tailscale device name: Tailscale admin panel → Machines

If they differ, rename the Tailscale device to match the unRAID hostname. Case matters.

```bash
# On each server — confirm the hostname the scripts will use
hostname
# Should return e.g. "unRAID-Gmer4Lfe"
```

---

## ━━━ STEP 2: CLONE THE REPOSITORY ━━━

---

### ── Clone on HOST1 ───────────────────────────────────────────────────────────

```bash
git clone <your_repo_url> /boot/config/plugins/varaverk
cd /boot/config/plugins/varaverk
```

---

### ── Clone on HOST2 ───────────────────────────────────────────────────────────

```bash
git clone <your_repo_url> /boot/config/plugins/varaverk
cd /boot/config/plugins/varaverk
```

---

### ── Sparse checkout — keep credentials separate ────────────────────────────

`host1.conf` contains HOST1's SSH keys, API keys, and passwords.
`host2.conf` contains HOST2's equivalents. Neither server should hold the other's
credentials. Set up sparse checkout so each server only receives its own conf file.

**On HOST1** — exclude host2.conf:
```bash
cd /boot/config/plugins/varaverk
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/Configurations/host2.conf'
git checkout
```

**On HOST2** — exclude host1.conf:
```bash
cd /boot/config/plugins/varaverk
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/Configurations/host1.conf'
git checkout
```

Verify: `ls Configurations/` — each server should see `master.conf` and only its own `host*.conf`.

> If you are not using git (deploying manually) skip this and ensure you only copy
> each server's own host*.conf when deploying.

---

## ━━━ STEP 3: CONFIGURE ━━━

Configuration lives in `Configurations/`. Three files:
- `master.conf` — shared, both servers see it — thresholds, toggles, schedules
- `host1.conf` — HOST1 only — credentials, share lists, container names
- `host2.conf` — HOST2 only — same structure, different values

Edit these with any text editor. Everything in the scripts is controlled through these
files — never edit scripts directly to change behaviour.

---

### ── 3a. master.conf — server hostnames (do this first) ─────────────────────

Open `Configurations/master.conf`. Find the HOST IDENTITIES section near the top:

```bash
HOST1="unRAID-Gmer4Lfe"
HOST2="unRAID-Jayred365"
```

Replace these with your actual unRAID server hostnames. These must match exactly —
`detect_hosts()` compares the running server's hostname against these two values to
know which server it is on. Everything else in the ecosystem flows from this.

---

### ── 3b. host1.conf — HOST1 identity ────────────────────────────────────────

Open `Configurations/host1.conf`. Fill in the IDENTITY section:

```bash
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"   # path — ssh_setup.sh will create this
HOST1_OWNER="gmer4lfe"                                  # short name, lowercase, no spaces
HOST1_OWNER_EMAIL="you@example.com"                     # for partnership notifications
```

The SSH key does not exist yet — `ssh_setup.sh` in Step 4 will create it and fill in
the path automatically. Leave the path as-is for now; it will be overwritten.

---

### ── 3c. host2.conf — HOST2 identity ────────────────────────────────────────

Open `Configurations/host2.conf`. Fill in the same fields with HOST2's values:

```bash
HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"
HOST2_OWNER="jayred365"
HOST2_OWNER_EMAIL="them@example.com"
```

---

### ── 3d. Remaining configuration ────────────────────────────────────────────

The scripts will not break if the remaining fields are empty — they self-guard
when things are not configured. Fill in sections as you enable each feature. The
minimum set needed to get a working sync + fallback:

**In host1.conf (and host2.conf mirror):**
```bash
# Shares HOST1 pushes to HOST2 nightly
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"
    "/mnt/user/Tv_Shows"
    "/mnt/user/Music"
)

# Shares synced every 30 minutes (auth config + Emby watch state)
HOST1_CRITICAL_SYNC_SHARES=(
    "/mnt/user/appdata-Fallback/Critical-Data|critical-data"
)

# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate)
HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=()   # HOST2's own covers for HOST1, and vice versa
HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=(
    "Emby"
    "VaultWarden"
    "NginxProxyManager"
)
```

For full configuration reference see `Manual-Fallback.md`, `Manual-Rsync.md`,
and `Manual-Watchdogs.md`.

---

## ━━━ STEP 4: SSH KEYS ━━━

The servers need passwordless SSH access to each other. `ssh_setup.sh` handles
key generation and remote installation in one step.

---

### ── Run on HOST2 first ────────────────────────────────────────────────────

The mirror runs first — it generates its own key and copies it to HOST1. You will be
prompted for HOST1's root password once (for the initial key copy). After that,
all SSH is keyless.

```bash
cd /boot/config/plugins/varaverk
bash Partnership/ssh_setup.sh
```

When prompted, enter HOST1's root password. The script:
- Generates `/root/.ssh/jayred365_rsync_automation` (ed25519)
- Copies the public key to HOST1's `~/.ssh/authorized_keys`
- Updates `host2.conf` with the key path

---

### ── Run on HOST1 ──────────────────────────────────────────────────────────

```bash
cd /boot/config/plugins/varaverk
bash Partnership/ssh_setup.sh
```

Enter HOST2's root password when prompted. Same steps, opposite direction.

---

### ── Verify both directions ────────────────────────────────────────────────

```bash
# On HOST1 — SSH to HOST2 should succeed without a password
ssh -i /root/.ssh/gmer4lfe_rsync_automation root@unRAID-Jayred365 "hostname"
# Expected: unRAID-Jayred365

# On HOST2 — SSH to HOST1 should succeed without a password
ssh -i /root/.ssh/jayred365_rsync_automation root@unRAID-Gmer4Lfe "hostname"
# Expected: unRAID-Gmer4Lfe
```

If either fails:
```bash
# Check status and fingerprints
bash Partnership/ssh_setup.sh --status

# Re-run with --force to regenerate and re-copy
bash Partnership/ssh_setup.sh --force
```

---

### ── Tailscale SSH note ──────────────────────────────────────────────────────

The scripts SSH via Tailscale hostnames, not local IPs. The first connection to a
new host requires accepting the host key. Accept it now so scripts never block on an
interactive prompt:

```bash
# On HOST1
ssh -o StrictHostKeyChecking=accept-new root@unRAID-Jayred365 "echo ok"

# On HOST2
ssh -o StrictHostKeyChecking=accept-new root@unRAID-Gmer4Lfe "echo ok"
```

---

## ━━━ STEP 5: SCHEDULER SETUP ━━━

**Array start** uses the User Scripts plugin (one entry only). **Everything else** runs
through the Varaverk plugin's built-in scheduler — configure via the Scheduler tab or
`schedule.json`. Individual scripts are never scheduled directly.

---

### ── User Scripts — array start only ─────────────────────────────────────────

Open **Settings → User Scripts**. Create one entry:

| Script | Schedule | Command |
|--------|----------|---------|
| `array_started` | At Array Start | `bash /boot/config/plugins/varaverk/Orchestrators/array_started.sh` |

Set to **Background Task**. This launches everything in ARRAY_START_SCRIPTS.

---

### ── Varaverk Scheduler — all cron entries ────────────────────────────────────

Configure via the Varaverk plugin Scheduler tab (or edit `schedule.json` directly):

| Script | Cron | Purpose |
|--------|------|---------|
| `transcode_management` | `*/7 * * * *` | Cleanup then manager — order critical |
| `watchdog_orchestrator` | `*/15 * * * *` | resource → docker → system → stability |
| `critical_sync_maintenance` | `*/30 * * * *` | auth + Emby dirty sync + partnership |
| `intermediate_sync_maintenance` | `0 */4 * * *` | arr sync + failed recovery |
| `daily_sync_maintenance` | `0 1 * * *` | full daily maintenance window |
| `weekly_sync_maintenance` | `30 2 * * 0` | clean sync + image updates |
| `monthly_maintenance` | `0 0 15 * *` | ZFS scrub, SMART tests (uptime-gated) |

Concurrent runs are prevented by `acquire_lock` — if the previous run is still active
the new one exits immediately. No need to stagger entries manually.

For the complete schedule and what each orchestrator runs, see
[README-Orchestrators.md](Orchestrators/README-Orchestrators.md).

---

## ━━━ STEP 6: INITIAL SYNC ━━━

Before enabling fallback, HOST2 needs a copy of HOST1's data. This first sync
will take a while depending on library size.

---

### ── Dry run first ────────────────────────────────────────────────────────────

```bash
# On HOST1 — preview what would sync, check paths and profiles
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run --log
```

Review the output. Check that:
- Share paths resolve correctly on both sides
- No pre-flight errors (disk space, rootfs, connectivity)
- Profiles look correct for each share

---

### ── Run the initial sync ─────────────────────────────────────────────────────

```bash
# On HOST1 — run the full daily sync (this includes rsync for all DAILY_SYNC_SHARES)
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --log
```

This will take longer than future nightly runs — it is transferring everything for
the first time. Monitor progress via the Varaverk log viewer or:

```bash
# Watch transfer in real time
watch -n5 'ls -lh /mnt/user/Movies/ | tail -5'
```

After it finishes, also run the critical sync to populate auth and Emby state:
```bash
bash /boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh --log
```

---

## ━━━ STEP 7: ENABLE FALLBACK ━━━

`fallback.sh` is disabled by default — it should only be enabled when both servers
are correctly synced and the data on HOST2 is current.

---

### ── Enable in master.conf ────────────────────────────────────────────────────

```bash
FALLBACK_ENABLED=true
```

Commit and pull on both servers so both pick up the change.

---

### ── Fallback starts automatically on array start ────────────────────────────

`array_started.sh` launches `fallback.sh` as a background process. Once
`FALLBACK_ENABLED=true` is in master.conf, restarting the array starts it.

To start it now without a reboot:
```bash
bash /boot/config/plugins/varaverk/Fallback/fallback.sh &
```

---

### ── Verify it is running ──────────────────────────────────────────────────────

```bash
pgrep -a -f fallback.sh
# Should show the running process
```

Check current state:
```bash
cat /boot/config/fallback_state.db
# state=NORMAL — both servers up
```

---

## ━━━ STEP 8: TEST FAILOVER ━━━

Before relying on the system, confirm it actually triggers. `fallback_test.sh`
simulates an outage using `iptables` — no real downtime, no real data changes.

```bash
# On HOST2 — dry run first (sees the sequence, no container starts)
bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --dry-run --log

# When ready — real test (uses iptables to simulate HOST1 unreachable)
bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log
```

The test runs in phases — blocks HOST1's Tailscale IP, waits for `fallback.sh` to
detect it and start Tier 1 containers, then unblocks and waits for handback.

See [Manual-Fallback.md](Fallback/Manual-Fallback.md) for what each phase does and
how to interpret the output.

---

## ━━━ STEP 9: PARTNERSHIP ONBOARD ━━━

The partnership is the optional layer that shares HOST1's auth stack with HOST2 —
NPM, Authelia, LLDAP, MariaDB, Redis all warm on HOST2, managed centrally from HOST1.

Skip this step if HOST2 does not need HOST1's auth stack (it runs fully independent auth).

---

### ── Prerequisites ────────────────────────────────────────────────────────────

Before running partnership onboard:

- Steps 1–8 complete on both servers
- HOST1's auth containers are running and healthy
- `PARTNERSHIP_AUTH_STACK` in `host1.conf` lists the auth container XML templates
- `PARTNERSHIP_ARR_STACK` in `host1.conf` lists arr container XML templates (if sharing arrs)
- FolderView3 plugin installed on both servers (if using `PARTNERSHIP_FOLDERVIEW3=true`)

---

### ── Run on HOST2 first (mirror) ────────────────────────────────────────────

HOST2 runs first — it generates its SSH key and waits. The owner completes setup remotely.

```bash
# On HOST2
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
# Review output, then:
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log
```

HOST2's work is done after Step 1 (SSH key setup). The rest happens on HOST1.

---

### ── Run on HOST1 (owner) ────────────────────────────────────────────────────

```bash
# On HOST1
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
# Review output — this will deploy auth + arr stacks to HOST2 remotely
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log
```

The owner path deploys containers to HOST2 via SSH, configures WebUI targets, runs
an initial arr library sync, and writes the partnership state files on both servers.

For full detail on what each step does see
[Manual-Partnership.md](Partnership/Manual-Partnership.md).

---

## ━━━ STEP 10: VERIFY THE ECOSYSTEM ━━━

---

### ── Check script output ────────────────────────────────────────────────────

Run the main orchestrators manually with `--log` to see verbose output:

```bash
# Watchdogs — should find everything healthy
bash Orchestrators/watchdog_orchestrator.sh --log

# Daily sync — dry run to confirm share paths and profiles
bash Orchestrators/daily_sync_maintenance.sh --dry-run --log
```

---

### ── Check fallback state ────────────────────────────────────────────────────

```bash
cat /boot/config/fallback_state.db
# Expected:
# state=NORMAL
# fallback_start=0
# handback_strikes=0
# tier2_started=false
# tier3_started=false
# tier4_started=false
```

---

### ── Check sync logs ─────────────────────────────────────────────────────────

Bandwidth monitor records every sync:
```bash
# Latest transfer log
ls -lt /boot/config/plugins/varaverk/data/bandwidth_monitor/
cat /boot/config/plugins/varaverk/data/bandwidth_monitor/<latest_log>
```

---

### ── Run status on key scripts ─────────────────────────────────────────────

```bash
bash Fallback/fallback.sh --status
bash Partnership/ssh_setup.sh --status
bash Monitors/cert_monitor.sh --status
bash Watchdogs/resource_watchdog.sh --status
```

---

## ━━━ COMMON FIRST-RUN ISSUES ━━━

---

### ── Script exits silently / does nothing ────────────────────────────────────

```bash
# Run with --log to see verbose output
bash <script> --log

# Check if the script is skipping due to a toggle being false
grep "ENABLED\|_TOGGLE\|ENABLED=" Configurations/master.conf | grep "false"
```

Most scripts have an `ENABLED=true/false` toggle in `master.conf`. Check that the
feature you are testing is enabled.

---

### ── SSH connection refused or times out ────────────────────────────────────

```bash
# Verify Tailscale is running and the remote is reachable
tailscale status
ping -c 3 <remote_hostname>

# Verify the key is authorised on the remote
ssh -v -i /root/.ssh/<your_key> root@<remote_hostname> "echo ok"

# Re-run ssh_setup.sh if the key is not installed
bash Partnership/ssh_setup.sh --force
```

---

### ── detect_hosts() exits — MY_ID unknown ────────────────────────────────────

```bash
# The running server's hostname doesn't match HOST1 or HOST2 in master.conf
hostname
# Compare to master.conf
grep "^    HOST[12]=" Configurations/master.conf
# They must match exactly — case sensitive
```

---

### ── Fallback triggers immediately on startup ────────────────────────────────

The remote server is unreachable on the first check. Common causes:
- Tailscale not running on the remote
- Remote server is actually offline
- `HOST1`/`HOST2` hostnames in `master.conf` don't match Tailscale device names

```bash
# Confirm which state fallback is in
cat /boot/config/fallback_state.db
# If stuck in FAILOVER after remote comes back: reset state
bash Tools/fallback_state_reset.sh
```

---

### ── Rsync fails — remote share not found ─────────────────────────────────────

```bash
# Check the share exists on the remote
ssh -i /root/.ssh/<your_key> root@<remote_hostname> "ls /mnt/user/"

# Share names must be identical on both servers
# /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2
# A mismatch = rsync writes to a non-existent path or aborts pre-flight
```

---

## ━━━ WHAT'S NEXT ━━━

Once the core is running, enable and configure the remaining features at your own pace.
Each subsystem has a README with the design decisions and a Manual with the config
reference and troubleshooting. The `--status` flag on any script shows the current
configuration and state.

```
Watchdogs/        → README-Watchdogs.md        configure memory limits, container lists
Media/            → README-Media.md             enable arr cleanup, discovery scripts
Monitors/         → README-Monitors.md          cert monitoring, SMART, health digest
Transcodes/       → README-Transcoding.md       ramdisk setup for Emby transcode
Partnership/      → README-Partnership.md       auth stack sharing, full lifecycle
```