Files

T

Gmer4Lfe 4effb55756 docs: fix root Manual, README, User_Script_Plug-in — boot drive paths, 15min watchdog, scheduler setup replaces User Scripts section

2026-06-01 19:50:31 -04:00

23 KiB

Raw Blame History

━━━━━ VARAVERK — Setup Manual ━━━━━

Getting a fresh two-server ecosystem running from scratch.
For system overview see README.md. For individual subsystem detail see folder READMEs and script headers.

━━━ BEFORE YOU START ━━━

Things that must be in place before you touch any scripts.

── Two unRAID Servers ───────────────────────────────────────────────────────

Both servers need to be up and have their arrays started. Hardware does not need to match. The scripts work via /mnt/user/ — the same share name on both servers resolves correctly regardless of the hardware underneath.

Decide which server is HOST1 (owner) and which is HOST2 (mirror). This is just a label — both servers run their own full stack. The owner manages the shared auth configuration. Pick the more capable or more reliably-online server as owner.

── Plugins — install on both servers ────────────────────────────────────────

Install these via Apps (Community Applications) on both servers:

Plugin	Why
Tailscale	Encrypted VPN between servers — all script traffic travels over it
User Scripts	Required for "At Startup of Array" hook — array_started.sh entry only

━━━ STEP 1: TAILSCALE ━━━

Both servers must be visible to each other on the same Tailscale network.

── Connect both servers ────────────────────────────────────────────────────

Open the Tailscale plugin on HOST1. Log in with your Tailscale account.
Open the Tailscale plugin on HOST2. Log in with the same account — or have the HOST2 owner share their machine into your tailnet (Tailscale → Share → invite by email).
Confirm both devices appear in your Tailscale admin panel.

── Verify reachability ──────────────────────────────────────────────────────

On HOST1:

# Get your Tailscale IP
tailscale ip -4

# Ping HOST2 by its Tailscale hostname (the unRAID server name, e.g. unRAID-Jayred365)
ping -c 3 unRAID-Jayred365

On HOST2 — same in reverse. Both pings must succeed before continuing.

── Note the Tailscale hostnames ──────────────────────────────────────────────

The scripts use Tailscale DNS to resolve IPs — no hardcoded addresses anywhere. For this to work, the Tailscale device name must match the unRAID server hostname exactly.

Check the unRAID hostname: Settings → System Identification → Server Name
Check the Tailscale device name: Tailscale admin panel → Machines

If they differ, rename the Tailscale device to match the unRAID hostname. Case matters.

# On each server — confirm the hostname the scripts will use
hostname
# Should return e.g. "unRAID-Gmer4Lfe"

━━━ STEP 2: CLONE THE REPOSITORY ━━━

── Clone on HOST1 ───────────────────────────────────────────────────────────

git clone <your_repo_url> /boot/config/plugins/varaverk
cd /boot/config/plugins/varaverk

── Clone on HOST2 ───────────────────────────────────────────────────────────

git clone <your_repo_url> /boot/config/plugins/varaverk
cd /boot/config/plugins/varaverk

── Sparse checkout — keep credentials separate ────────────────────────────

host1.conf contains HOST1's SSH keys, API keys, and passwords. host2.conf contains HOST2's equivalents. Neither server should hold the other's credentials. Set up sparse checkout so each server only receives its own conf file.

On HOST1 — exclude host2.conf:

cd /boot/config/plugins/varaverk
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/Configurations/host2.conf'
git checkout

On HOST2 — exclude host1.conf:

cd /boot/config/plugins/varaverk
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/Configurations/host1.conf'
git checkout

Verify: ls Configurations/ — each server should see master.conf and only its own host*.conf.

If you are not using git (deploying manually) skip this and ensure you only copy each server's own host*.conf when deploying.

━━━ STEP 3: CONFIGURE ━━━

Configuration lives in Configurations/. Three files:

master.conf — shared, both servers see it — thresholds, toggles, schedules
host1.conf — HOST1 only — credentials, share lists, container names
host2.conf — HOST2 only — same structure, different values

Edit these with any text editor. Everything in the scripts is controlled through these files — never edit scripts directly to change behaviour.

── 3a. master.conf — server hostnames (do this first) ─────────────────────

Open Configurations/master.conf. Find the HOST IDENTITIES section near the top:

HOST1="unRAID-Gmer4Lfe"
HOST2="unRAID-Jayred365"

Replace these with your actual unRAID server hostnames. These must match exactly — detect_hosts() compares the running server's hostname against these two values to know which server it is on. Everything else in the ecosystem flows from this.

── 3b. host1.conf — HOST1 identity ────────────────────────────────────────

Open Configurations/host1.conf. Fill in the IDENTITY section:

HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"   # path — ssh_setup.sh will create this
HOST1_OWNER="gmer4lfe"                                  # short name, lowercase, no spaces
HOST1_OWNER_EMAIL="you@example.com"                     # for partnership notifications

The SSH key does not exist yet — ssh_setup.sh in Step 4 will create it and fill in the path automatically. Leave the path as-is for now; it will be overwritten.

── 3c. host2.conf — HOST2 identity ────────────────────────────────────────

Open Configurations/host2.conf. Fill in the same fields with HOST2's values:

HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"
HOST2_OWNER="jayred365"
HOST2_OWNER_EMAIL="them@example.com"

── 3d. Remaining configuration ────────────────────────────────────────────

The scripts will not break if the remaining fields are empty — they self-guard when things are not configured. Fill in sections as you enable each feature. The minimum set needed to get a working sync + fallback:

In host1.conf (and host2.conf mirror):

# Shares HOST1 pushes to HOST2 nightly
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"
    "/mnt/user/Tv_Shows"
    "/mnt/user/Music"
)

# Shares synced every 30 minutes (auth config + Emby watch state)
HOST1_CRITICAL_SYNC_SHARES=(
    "/mnt/user/appdata-Fallback/Critical-Data|critical-data"
)

# Containers fallback.sh starts on HOST2 when HOST1 goes down (Tier 1 = immediate)
HOST2_FALLBACK_HOST1_COVERS_HOST2_TIER1=()   # HOST2's own covers for HOST1, and vice versa
HOST1_FALLBACK_HOST2_COVERS_HOST1_TIER1=(
    "Emby"
    "VaultWarden"
    "NginxProxyManager"
)

For full configuration reference see Manual-Fallback.md, Manual-Rsync.md, and Manual-Watchdogs.md.

━━━ STEP 4: SSH KEYS ━━━

The servers need passwordless SSH access to each other. ssh_setup.sh handles key generation and remote installation in one step.

── Run on HOST2 first ────────────────────────────────────────────────────

The mirror runs first — it generates its own key and copies it to HOST1. You will be prompted for HOST1's root password once (for the initial key copy). After that, all SSH is keyless.

cd /boot/config/plugins/varaverk
bash Partnership/ssh_setup.sh

When prompted, enter HOST1's root password. The script:

Generates /root/.ssh/jayred365_rsync_automation (ed25519)
Copies the public key to HOST1's ~/.ssh/authorized_keys
Updates host2.conf with the key path

── Run on HOST1 ──────────────────────────────────────────────────────────

cd /boot/config/plugins/varaverk
bash Partnership/ssh_setup.sh

Enter HOST2's root password when prompted. Same steps, opposite direction.

── Verify both directions ────────────────────────────────────────────────

# On HOST1 — SSH to HOST2 should succeed without a password
ssh -i /root/.ssh/gmer4lfe_rsync_automation root@unRAID-Jayred365 "hostname"
# Expected: unRAID-Jayred365

# On HOST2 — SSH to HOST1 should succeed without a password
ssh -i /root/.ssh/jayred365_rsync_automation root@unRAID-Gmer4Lfe "hostname"
# Expected: unRAID-Gmer4Lfe

If either fails:

# Check status and fingerprints
bash Partnership/ssh_setup.sh --status

# Re-run with --force to regenerate and re-copy
bash Partnership/ssh_setup.sh --force

── Tailscale SSH note ──────────────────────────────────────────────────────

The scripts SSH via Tailscale hostnames, not local IPs. The first connection to a new host requires accepting the host key. Accept it now so scripts never block on an interactive prompt:

# On HOST1
ssh -o StrictHostKeyChecking=accept-new root@unRAID-Jayred365 "echo ok"

# On HOST2
ssh -o StrictHostKeyChecking=accept-new root@unRAID-Gmer4Lfe "echo ok"

━━━ STEP 5: SCHEDULER SETUP ━━━

Array start uses the User Scripts plugin (one entry only). Everything else runs through the Varaverk plugin's built-in scheduler — configure via the Scheduler tab or schedule.json. Individual scripts are never scheduled directly.

── User Scripts — array start only ─────────────────────────────────────────

Open Settings → User Scripts. Create one entry:

Script	Schedule	Command
`array_started`	At Array Start	`bash /boot/config/plugins/varaverk/Orchestrators/array_started.sh`

Set to Background Task. This launches everything in ARRAY_START_SCRIPTS.

── Varaverk Scheduler — all cron entries ────────────────────────────────────

Configure via the Varaverk plugin Scheduler tab (or edit schedule.json directly):

Script	Cron	Purpose
`transcode_management`	`/7 * * *`	Cleanup then manager — order critical
`watchdog_orchestrator`	`/15 * * *`	resource → docker → system → stability
`critical_sync_maintenance`	`/30 * * *`	auth + Emby dirty sync + partnership
`intermediate_sync_maintenance`	`0 /4 * *`	arr sync + failed recovery
`daily_sync_maintenance`	`0 1 * * *`	full daily maintenance window
`weekly_sync_maintenance`	`30 2 * * 0`	clean sync + image updates
`monthly_maintenance`	`0 0 15 * *`	ZFS scrub, SMART tests (uptime-gated)

Concurrent runs are prevented by acquire_lock — if the previous run is still active the new one exits immediately. No need to stagger entries manually.

For the complete schedule and what each orchestrator runs, see README-Orchestrators.md.

━━━ STEP 6: INITIAL SYNC ━━━

Before enabling fallback, HOST2 needs a copy of HOST1's data. This first sync will take a while depending on library size.

── Dry run first ────────────────────────────────────────────────────────────

# On HOST1 — preview what would sync, check paths and profiles
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run --log

Review the output. Check that:

Share paths resolve correctly on both sides
No pre-flight errors (disk space, rootfs, connectivity)
Profiles look correct for each share

── Run the initial sync ─────────────────────────────────────────────────────

# On HOST1 — run the full daily sync (this includes rsync for all DAILY_SYNC_SHARES)
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --log

This will take longer than future nightly runs — it is transferring everything for the first time. Monitor progress via the Varaverk log viewer or:

# Watch transfer in real time
watch -n5 'ls -lh /mnt/user/Movies/ | tail -5'

After it finishes, also run the critical sync to populate auth and Emby state:

bash /boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh --log

━━━ STEP 7: ENABLE FALLBACK ━━━

fallback.sh is disabled by default — it should only be enabled when both servers are correctly synced and the data on HOST2 is current.

── Enable in master.conf ────────────────────────────────────────────────────

FALLBACK_ENABLED=true

Commit and pull on both servers so both pick up the change.

── Fallback starts automatically on array start ────────────────────────────

array_started.sh launches fallback.sh as a background process. Once FALLBACK_ENABLED=true is in master.conf, restarting the array starts it.

To start it now without a reboot:

bash /boot/config/plugins/varaverk/Fallback/fallback.sh &

── Verify it is running ──────────────────────────────────────────────────────

pgrep -a -f fallback.sh
# Should show the running process

Check current state:

cat /boot/config/fallback_state.db
# state=NORMAL — both servers up

━━━ STEP 8: TEST FAILOVER ━━━

Before relying on the system, confirm it actually triggers. fallback_test.sh simulates an outage using iptables — no real downtime, no real data changes.

# On HOST2 — dry run first (sees the sequence, no container starts)
bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --dry-run --log

# When ready — real test (uses iptables to simulate HOST1 unreachable)
bash /boot/config/plugins/varaverk/Fallback/fallback_test.sh --log

The test runs in phases — blocks HOST1's Tailscale IP, waits for fallback.sh to detect it and start Tier 1 containers, then unblocks and waits for handback.

See Manual-Fallback.md for what each phase does and how to interpret the output.

━━━ STEP 9: PARTNERSHIP ONBOARD ━━━

The partnership is the optional layer that shares HOST1's auth stack with HOST2 — NPM, Authelia, LLDAP, MariaDB, Redis all warm on HOST2, managed centrally from HOST1.

Skip this step if HOST2 does not need HOST1's auth stack (it runs fully independent auth).

── Prerequisites ────────────────────────────────────────────────────────────

Before running partnership onboard:

Steps 1–8 complete on both servers
HOST1's auth containers are running and healthy
PARTNERSHIP_AUTH_STACK in host1.conf lists the auth container XML templates
PARTNERSHIP_ARR_STACK in host1.conf lists arr container XML templates (if sharing arrs)
FolderView3 plugin installed on both servers (if using PARTNERSHIP_FOLDERVIEW3=true)

── Run on HOST2 first (mirror) ────────────────────────────────────────────

HOST2 runs first — it generates its SSH key and waits. The owner completes setup remotely.

# On HOST2
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
# Review output, then:
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log

HOST2's work is done after Step 1 (SSH key setup). The rest happens on HOST1.

── Run on HOST1 (owner) ────────────────────────────────────────────────────

# On HOST1
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --dry-run --log
# Review output — this will deploy auth + arr stacks to HOST2 remotely
bash /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh --log

The owner path deploys containers to HOST2 via SSH, configures WebUI targets, runs an initial arr library sync, and writes the partnership state files on both servers.

For full detail on what each step does see Manual-Partnership.md.

━━━ STEP 10: VERIFY THE ECOSYSTEM ━━━

── Check script output ────────────────────────────────────────────────────

Run the main orchestrators manually with --log to see verbose output:

# Watchdogs — should find everything healthy
bash Orchestrators/watchdog_orchestrator.sh --log

# Daily sync — dry run to confirm share paths and profiles
bash Orchestrators/daily_sync_maintenance.sh --dry-run --log

── Check fallback state ────────────────────────────────────────────────────

cat /boot/config/fallback_state.db
# Expected:
# state=NORMAL
# fallback_start=0
# handback_strikes=0
# tier2_started=false
# tier3_started=false
# tier4_started=false

── Check sync logs ─────────────────────────────────────────────────────────

Bandwidth monitor records every sync:

# Latest transfer log
ls -lt /boot/config/plugins/varaverk/data/bandwidth_monitor/
cat /boot/config/plugins/varaverk/data/bandwidth_monitor/<latest_log>

── Run status on key scripts ─────────────────────────────────────────────

bash Fallback/fallback.sh --status
bash Partnership/ssh_setup.sh --status
bash Monitors/cert_monitor.sh --status
bash Watchdogs/resource_watchdog.sh --status

━━━ COMMON FIRST-RUN ISSUES ━━━

── Script exits silently / does nothing ────────────────────────────────────

# Run with --log to see verbose output
bash <script> --log

# Check if the script is skipping due to a toggle being false
grep "ENABLED\|_TOGGLE\|ENABLED=" Configurations/master.conf | grep "false"

Most scripts have an ENABLED=true/false toggle in master.conf. Check that the feature you are testing is enabled.

── SSH connection refused or times out ────────────────────────────────────

# Verify Tailscale is running and the remote is reachable
tailscale status
ping -c 3 <remote_hostname>

# Verify the key is authorised on the remote
ssh -v -i /root/.ssh/<your_key> root@<remote_hostname> "echo ok"

# Re-run ssh_setup.sh if the key is not installed
bash Partnership/ssh_setup.sh --force

── detect_hosts() exits — MY_ID unknown ────────────────────────────────────

# The running server's hostname doesn't match HOST1 or HOST2 in master.conf
hostname
# Compare to master.conf
grep "^    HOST[12]=" Configurations/master.conf
# They must match exactly — case sensitive

── Fallback triggers immediately on startup ────────────────────────────────

The remote server is unreachable on the first check. Common causes:

Tailscale not running on the remote
Remote server is actually offline
HOST1/HOST2 hostnames in master.conf don't match Tailscale device names

# Confirm which state fallback is in
cat /boot/config/fallback_state.db
# If stuck in FAILOVER after remote comes back: reset state
bash Tools/fallback_state_reset.sh

# Check the share exists on the remote
ssh -i /root/.ssh/<your_key> root@<remote_hostname> "ls /mnt/user/"

# Share names must be identical on both servers
# /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2
# A mismatch = rsync writes to a non-existent path or aborts pre-flight

━━━ WHAT'S NEXT ━━━

Once the core is running, enable and configure the remaining features at your own pace. Each subsystem has a README with the design decisions and a Manual with the config reference and troubleshooting. The --status flag on any script shows the current configuration and state.

Watchdogs/        → README-Watchdogs.md        configure memory limits, container lists
Media/            → README-Media.md             enable arr cleanup, discovery scripts
Monitors/         → README-Monitors.md          cert monitoring, SMART, health digest
Transcodes/       → README-Transcoding.md       ramdisk setup for Emby transcode
Partnership/      → README-Partnership.md       auth stack sharing, full lifecycle

23 KiB Raw Blame History Unescape Escape