993 lines
47 KiB
Markdown
993 lines
47 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 🔄 RSYNC SETUP GUIDE
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Complete setup guide for the two-server rsync ecosystem.** By the end of this guide
|
|
both servers will have SSH keys configured, Tailscale connected, the git repository
|
|
cloned, and all scheduled operations running automatically.
|
|
|
|
> **This is a setup guide, not a script reference.** For how rsync.sh works internally,
|
|
> profiles, safety checks, and operational details — those belong in the orchestrator
|
|
> and rsync script documentation. This guide is about standing the ecosystem up from
|
|
> scratch and verifying it works.
|
|
|
|
---
|
|
|
|
## ━━━ WHAT YOU'RE BUILDING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```
|
|
HOST1 (unRAID-Gmer4Lfe) HOST2 (unRAID-Jayred365)
|
|
────────────────────── ──────────────────────
|
|
Source of truth: Source of truth:
|
|
Movies, Tv_Shows, Music Anime_Shows, Anime_Movies
|
|
Critical-Data (auth stack)
|
|
Emby userdata
|
|
|
|
Pushes to HOST2 daily: ──────→ Receives:
|
|
All HOST1 shares Mirror of HOST1 shares
|
|
Personal encrypted shares Personal (encrypted blocks)
|
|
|
|
Receives from HOST2 daily: ←────── Pushes:
|
|
Anime_Shows, Anime_Movies All HOST2 shares
|
|
|
|
Weekly clean sync (both sides stopped):
|
|
Emby — full clean mirror ←──────→ Emby
|
|
Critical-Data ──────→ Auth stack (HOST1 → HOST2)
|
|
|
|
Every 30 minutes — dirty sync:
|
|
Emby watch states ──────→ HOST2 stays current on playback
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ PREREQUISITES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Both servers need these before starting:
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Required on both servers:
|
|
unRAID 7.x
|
|
Community Applications plugin — search "Community Applications" in unRAID plugins
|
|
User Scripts plugin — install via Community Applications
|
|
Tailscale plugin — install via Community Applications
|
|
Terminal access — unRAID UI → Tools → Terminal, or SSH
|
|
|
|
# Optional but recommended:
|
|
Gitea (Docker container on HOST1) — self-hosted git for the script repository
|
|
Working Emby installation — for transcode management and failover
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ STEP 1 — TAILSCALE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Tailscale provides the encrypted mesh network between servers. Scripts resolve the
|
|
remote server's IP via Tailscale at runtime — no hardcoded IPs, no VPN configuration,
|
|
no open ports. All server-to-server communication goes through Tailscale.
|
|
|
|
---
|
|
|
|
### ── Install on Both Servers ────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# On each server:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# 1. Open Apps in the unRAID UI
|
|
# 2. Search for "Tailscale" — install the plugin
|
|
# 3. Settings → Tailscale → Connect
|
|
# 4. Authenticate with your Tailscale account (browser opens on your machine)
|
|
# 5. Verify both servers appear: https://login.tailscale.com/admin/machines
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Verify Connectivity ─────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# From HOST1 — should return HOST2's 100.x.x.x Tailscale IP:
|
|
tailscale ip -4 unRAID-Jayred365
|
|
|
|
# From HOST2 — should return HOST1's 100.x.x.x Tailscale IP:
|
|
tailscale ip -4 unRAID-Gmer4Lfe
|
|
|
|
# Test actual connectivity:
|
|
tailscale ping unRAID-Jayred365 # run from HOST1
|
|
```
|
|
|
|
> **Critical:** The hostnames in `master.conf` (`HOST1` and `HOST2`) must match the
|
|
> Tailscale machine names **exactly** — case sensitive. The ecosystem resolves all
|
|
> remote IPs via `tailscale ip -4 HOSTNAME` at runtime. A name mismatch means every
|
|
> script that touches the remote will fail at the IP resolution step.
|
|
|
|
---
|
|
|
|
## ━━━ STEP 2 — ENABLE SSH ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
unRAID 7.x has SSH disabled by default. Enable it on both servers.
|
|
|
|
```bash
|
|
# On each server:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Settings → Management Access → Secure Shell
|
|
# SSH: Enabled
|
|
# SSH port: 22
|
|
# Apply
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
> SSH is only exposed on your local network and Tailscale interface. Scripts connect
|
|
> via Tailscale IP — all traffic is encrypted end-to-end. No ports are opened to
|
|
> the public internet.
|
|
|
|
---
|
|
|
|
## ━━━ STEP 3 — SSH KEYS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Two sets of keys needed: server-to-server for rsync and failover, and Gitea access
|
|
for script repository pull. Generate all keys before configuring anything else.
|
|
|
|
---
|
|
|
|
### ── 3a — Server-to-Server Keys ─────────────────────────────────────────────
|
|
|
|
```bash
|
|
# On HOST1 — generate HOST1's key pair:
|
|
ssh-keygen -t ed25519 -f /root/.ssh/Gmer4Lfe-rsync-key -C "gmer4lfe-rsync" -N ""
|
|
|
|
# On HOST2 — generate HOST2's key pair:
|
|
ssh-keygen -t ed25519 -f /root/.ssh/Jayred365-rsync-key -C "jayred365-rsync" -N ""
|
|
```
|
|
|
|
---
|
|
|
|
### ── 3b — Authorise Keys Bidirectionally ─────────────────────────────────────
|
|
|
|
```bash
|
|
# HOST1's public key must be authorised on HOST2 (so HOST1 can SSH into HOST2):
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# On HOST1 — print the public key:
|
|
cat /root/.ssh/Gmer4Lfe-rsync-key.pub
|
|
|
|
# On HOST2 — create authorized_keys and paste HOST1's public key:
|
|
mkdir -p /root/.ssh
|
|
echo "PASTE_HOST1_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
|
|
chmod 600 /root/.ssh/authorized_keys
|
|
|
|
# HOST2's public key must be authorised on HOST1 (so HOST2 can SSH into HOST1):
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# On HOST2 — print the public key:
|
|
cat /root/.ssh/Jayred365-rsync-key.pub
|
|
|
|
# On HOST1 — append HOST2's public key:
|
|
echo "PASTE_HOST2_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
|
|
```
|
|
|
|
---
|
|
|
|
### ── 3c — Test Both Directions ────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# From HOST1 — should print "connected" without a password prompt:
|
|
ssh -i /root/.ssh/Gmer4Lfe-rsync-key \
|
|
root@$(tailscale ip -4 unRAID-Jayred365) \
|
|
"echo connected"
|
|
|
|
# From HOST2 — should print "connected" without a password prompt:
|
|
ssh -i /root/.ssh/Jayred365-rsync-key \
|
|
root@$(tailscale ip -4 unRAID-Gmer4Lfe) \
|
|
"echo connected"
|
|
```
|
|
|
|
```
|
|
If prompted for a password: the key was not authorised correctly.
|
|
→ Recheck Step 3b — the public key content must be on one line
|
|
→ Check permissions: chmod 600 /root/.ssh/authorized_keys
|
|
→ Check the key file referenced in the SSH command matches what was generated
|
|
```
|
|
|
|
---
|
|
|
|
### ── 3d — Gitea SSH Key ───────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# On BOTH servers — generate a key for Gitea access:
|
|
ssh-keygen -t ed25519 -f /root/.ssh/unraid_gitea -C "unraid-gitea" -N ""
|
|
|
|
# Print the public key to add to Gitea:
|
|
cat /root/.ssh/unraid_gitea.pub
|
|
|
|
# In Gitea: Settings → SSH / GPG Keys → Add Key → paste the output above
|
|
# Do this for both servers if they have separate Gitea accounts, or once if shared
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ STEP 4 — CLONE THE REPOSITORY ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Both servers clone from the same Gitea repository. Updates pushed to the repo
|
|
propagate to both servers automatically via `git_pull_execute.sh` at the start of
|
|
each daily maintenance window.
|
|
|
|
---
|
|
|
|
### ── On Both Servers ─────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Create the target directory:
|
|
mkdir -p /mnt/user/appdata/unraid_scripts
|
|
|
|
# Clone the repository:
|
|
GIT_SSH_COMMAND="ssh -i /root/.ssh/unraid_gitea" \
|
|
git clone git@YOUR_GITEA_HOST:FailedProxy/Unraid_Scripts.git \
|
|
/mnt/user/appdata/unraid_scripts
|
|
|
|
# Replace YOUR_GITEA_HOST with your Gitea server address and port
|
|
# Example: git@192.168.50.2:221
|
|
```
|
|
|
|
---
|
|
|
|
### ── Verify the Structure ─────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
ls /mnt/user/appdata/unraid_scripts/
|
|
```
|
|
|
|
```
|
|
Expected output:
|
|
master.conf ← all user configuration — the only file you edit
|
|
master_host1.conf ← HOST1-specific configuration
|
|
master_host2.conf ← HOST2-specific configuration
|
|
common.sh ← shared library — functions used by all scripts
|
|
load_config.sh ← config loader
|
|
Orchestrators/
|
|
Rsync/
|
|
Failover/
|
|
Docker_Essentials/
|
|
unRAID_Essentials/
|
|
Media/
|
|
Transcodes/
|
|
Monitors/
|
|
Tools/
|
|
Partnership/
|
|
```
|
|
|
|
---
|
|
|
|
### ── Make Scripts Executable ─────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Execute permission on all scripts — required once after clone:
|
|
find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \;
|
|
```
|
|
|
|
> `array_start.sh` auto-fixes permissions on scripts that lost the execute bit —
|
|
> but this initial chmod ensures the first run works before that safeguard is active.
|
|
|
|
---
|
|
|
|
## ━━━ STEP 5 — CONFIGURE MASTER.CONF ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All user configuration lives in `master.conf`. Every value with a comment
|
|
`# REQUIRED` must be set before the first run. Everything else has working defaults.
|
|
|
|
```bash
|
|
nano /mnt/user/appdata/unraid_scripts/master.conf
|
|
```
|
|
|
|
---
|
|
|
|
### ── Host Identity ─────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# These must match Tailscale machine names exactly — case sensitive.
|
|
# The ecosystem uses these to resolve remote IPs at runtime.
|
|
#
|
|
HOST1="unRAID-Gmer4Lfe" # REQUIRED — must match tailscale machine name
|
|
HOST2="unRAID-Jayred365" # REQUIRED — same
|
|
|
|
# SSH key paths — each server's key for authenticating to the other:
|
|
HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key" # HOST1 uses this to SSH to HOST2
|
|
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key" # HOST2 uses this to SSH to HOST1
|
|
```
|
|
|
|
---
|
|
|
|
### ── Emby API Keys ─────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Used by: emby_session_report.sh, sunday_morning_coffee_report.sh,
|
|
# sonarr/radarr cleanup (notify_emby_scan after deletion)
|
|
#
|
|
# Get from: Emby Dashboard → Settings → API Keys → + New API Key
|
|
#
|
|
HOST1_EMBY_URL="http://192.168.50.2:8096"
|
|
HOST1_EMBY_API_KEY="your-host1-emby-api-key" # REQUIRED for Emby features
|
|
HOST1_EMBY_CONTAINER="Emby"
|
|
|
|
# master_host2.conf
|
|
HOST2_EMBY_URL="http://localhost:8096"
|
|
HOST2_EMBY_API_KEY="your-host2-emby-api-key"
|
|
HOST2_EMBY_CONTAINER="Emby"
|
|
```
|
|
|
|
---
|
|
|
|
### ── Git Repository ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Used by git_pull_execute.sh — pulls latest scripts at start of each daily window.
|
|
#
|
|
GITEA_CONTAINER="Gitea" # exact Docker container name
|
|
GITEA_REPO_PATH="FailedProxy/Unraid_Scripts.git"
|
|
TARGET_DIR="/mnt/user/appdata/unraid_scripts"
|
|
GITEA_SSH_KEY="/root/.ssh/unraid_gitea"
|
|
SSH_PORT=221 # your Gitea SSH port
|
|
```
|
|
|
|
---
|
|
|
|
### ── Daily Sync Shares ────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Shares HOST1 is source of truth for — pushed to HOST2 every night at 1am.
|
|
# HOST2 treats these as read-only mirrors. Never put the same share in both lists.
|
|
#
|
|
HOST1_DAILY_SYNC_SHARES=(
|
|
"/mnt/user/Movies" # HOST1 manages this — Radarr runs here
|
|
"/mnt/user/Tv_Shows" # HOST1 manages this — Sonarr runs here
|
|
"/mnt/user/Music" # HOST1 manages this — Lidarr runs here
|
|
"/mnt/user/Kids_Movies"
|
|
"/mnt/user/Kids_Tv_Shows"
|
|
"/mnt/user/Sports"
|
|
"/mnt/user/stand-up_comedy"
|
|
)
|
|
|
|
# master_host2.conf
|
|
HOST2_DAILY_SYNC_SHARES=(
|
|
"/mnt/user/Anime_Shows" # HOST2 manages this — his Sonarr runs here
|
|
"/mnt/user/Anime_Movies" # HOST2 manages this — his Radarr runs here
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Weekly Sync Shares ───────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Synced during the Sunday 2:30am window — containers stopped both sides.
|
|
# Do NOT add these to a separate cron schedule — they run via weekly_sync_maintenance.sh.
|
|
#
|
|
WEEKLY_SYNC_SHARES=(
|
|
"/mnt/user/Media_Server/Emby" # full clean Emby mirror
|
|
"/mnt/user/appdata-Failover/Critical-Data" # auth stack clean state
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ STEP 6 — MASTER_HOST*.CONF ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Per-host configuration lives in `master_host1.conf` and `master_host2.conf`.
|
|
`detect_hosts()` in `common.sh` reads which server is running and aliases the correct
|
|
`HOST*_` prefixed variables to their unprefixed names. Scripts only ever reference the
|
|
unprefixed name — they work identically on both servers.
|
|
|
|
```bash
|
|
# master_host1.conf is only sourced on HOST1
|
|
# master_host2.conf is only sourced on HOST2
|
|
# Changes go in the right file for the right server
|
|
|
|
nano /mnt/user/appdata/unraid_scripts/master_host1.conf # on HOST1
|
|
nano /mnt/user/appdata/unraid_scripts/master_host2.conf # on HOST2
|
|
```
|
|
|
|
See each conf file's comments — every variable is documented with its purpose
|
|
and the reasoning behind the value.
|
|
|
|
---
|
|
|
|
## ━━━ STEP 7 — RSYNC PROFILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Profiles control per-share behaviour — which containers to stop, rsync flags,
|
|
bandwidth limits, what to exclude. Profile is matched by directory basename
|
|
(lowercased). Override with `--profile=name`.
|
|
|
|
---
|
|
|
|
### ── How Profile Matching Works ──────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
|
|
# basename: Arrs_Stack
|
|
# lowercased: arrs_stack
|
|
# matched profile: [arrs_stack]
|
|
#
|
|
# rsync.sh /mnt/user/Movies
|
|
# basename: Movies
|
|
# lowercased: movies
|
|
# no matching profile → global defaults apply (no containers stopped)
|
|
#
|
|
# rsync.sh /mnt/user/appdata-Failover/Critical-Data --profile=critical-failover
|
|
# explicit override → uses [critical-failover] profile regardless of path
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Current Profiles ─────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf — profile definitions
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Each profile defines which containers to stop, rsync flags, excludes, etc.
|
|
# Containers in PROFILE_CRITICAL_CONTAINER_NAMES are stopped on BOTH servers.
|
|
# PROFILE_DELAYED_CONTAINERS restart after PROFILE_CONTAINER_DELAY seconds.
|
|
|
|
# ── arrs_stack ─────────────────────────────────────────────────────────────
|
|
# Arr databases — stopped for clean SQLite snapshot
|
|
PROFILES["arrs_stack_CRITICAL_CONTAINER_NAMES"]=(
|
|
"Sonarr" "Radarr" "Lidarr" "Prowlarr" "Bazarr" "Pinchflat"
|
|
)
|
|
|
|
# ── critical-data ──────────────────────────────────────────────────────────
|
|
# Auth stack — stopped for clean database snapshot, delayed restart
|
|
PROFILES["critical-data_CRITICAL_CONTAINER_NAMES"]=(
|
|
"Mariadb-Authelia" "Redis-Authelia"
|
|
"NginxProxyManager" "Lldap-Gmer4Lfe"
|
|
)
|
|
PROFILES["critical-data_DELAYED_CONTAINERS"]=(
|
|
"Authelia" "Authelia-Secondary" # auth services restart after delay
|
|
)
|
|
PROFILES["critical-data_CONTAINER_DELAY"]=30 # seconds before delayed containers start
|
|
|
|
# ── important-data ─────────────────────────────────────────────────────────
|
|
# NextCloud + Postgres — stopped for clean snapshot
|
|
PROFILES["important-data_CRITICAL_CONTAINER_NAMES"]=(
|
|
"Postgres-NextCloud"
|
|
)
|
|
PROFILES["important-data_DELAYED_CONTAINERS"]=("NextCloud")
|
|
|
|
# ── emby ───────────────────────────────────────────────────────────────────
|
|
# Weekly full clean sync — both Emby instances stopped
|
|
PROFILES["emby_CRITICAL_CONTAINER_NAMES"]=("Emby")
|
|
PROFILES["emby_EXCLUDE_DIRS"]=(
|
|
"transcodes/" "logs/" "crash*" "cache/"
|
|
)
|
|
|
|
# ── emby-failover ──────────────────────────────────────────────────────────
|
|
# Every 30 minutes, Emby STAYS RUNNING — dirty sync of critical state only
|
|
# WAL and SHM excluded — safe to copy while Emby is writing
|
|
PROFILES["emby-failover_CRITICAL_CONTAINER_NAMES"]=() # empty — nothing stops
|
|
PROFILES["emby-failover_EXCLUDE_DIRS"]=(
|
|
"*.wal" "*.shm" # WAL files — unsafe mid-write
|
|
"transcodes/" "logs/" "crash*" "cache/" # volatile data — skip
|
|
)
|
|
PROFILES["emby-failover_REMOTE_RESTART_CONTAINERS"]=("Emby")
|
|
# Emby on HOST2 restarts after sync to pick up config changes
|
|
```
|
|
|
|
---
|
|
|
|
### ── Two Emby Profiles — Why Both Exist ──────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# emby-failover — every 30 minutes, Emby stays running:
|
|
# What syncs: users.db, library.db, authentication.db, config/
|
|
# What skips: *.wal *.shm transcodes/ logs/ cache/
|
|
# Why: WAL files are being written while Emby runs — copying them
|
|
# would produce a corrupt database on HOST2
|
|
# Result: HOST2 is always within 30 minutes of HOST1 on watch state
|
|
# and user activity. Failover is seamless — nobody notices.
|
|
#
|
|
# emby — Sunday 2:30am, both Emby instances stopped:
|
|
# What syncs: everything except transcodes, logs, cache, crash files
|
|
# What includes: metadata, plugins, full database state, all config
|
|
# Why: WAL is checkpointed on clean shutdown — safe to copy everything
|
|
# Full consistent mirror including metadata and plugin state
|
|
# Result: HOST2 has a gold-standard Emby state once per week
|
|
# Image cache warm for 6 days — only reset Sunday when users sleep
|
|
#
|
|
# The two profiles work together:
|
|
# emby-failover: keeps HOST2 current on what matters for immediate failover
|
|
# emby: gives HOST2 full fidelity once per week
|
|
# Neither alone is sufficient — both are needed.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ STEP 8 — PERSONAL ENCRYPTED SHARES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Personal shares are synced to the remote server for offsite backup. ZFS encrypts at
|
|
the dataset level — the remote server receives encrypted blocks and cannot read the
|
|
content without your passphrase or keyfile.
|
|
|
|
---
|
|
|
|
### ── Create an Encrypted ZFS Dataset ────────────────────────────────────────
|
|
|
|
```bash
|
|
# In the unRAID UI:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Main → click your ZFS pool name → + Dataset
|
|
# Name: Gmer4Lfe-Personal
|
|
# Encryption: Enabled
|
|
# Passphrase: [your passphrase]
|
|
# ⚠️ Write your passphrase down — if lost, data is completely unrecoverable
|
|
#
|
|
# Settings → Shares → Add Share
|
|
# Share path: point to the new encrypted dataset
|
|
# Use cache: Only — keeps data on ZFS pool, not array
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# Verify encryption is active before syncing:
|
|
zfs get encryption poolname/Gmer4Lfe-Personal
|
|
# Should show: encryption aes-256-gcm
|
|
```
|
|
|
|
---
|
|
|
|
### ── Auto-Unlock on Boot (Optional) ─────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Keyfile approach — passphrase stored in a file, loaded at boot.
|
|
# More convenient but the keyfile is a secret that must be protected.
|
|
# Never sync the keyfile to the remote server.
|
|
#
|
|
# Create keyfile — on HOST1 only:
|
|
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
|
|
chmod 600 /root/.zfs-keys/personal.key
|
|
|
|
# Set dataset to use keyfile instead of passphrase:
|
|
zfs change-key \
|
|
-o keylocation=file:///root/.zfs-keys/personal.key \
|
|
-o keyformat=raw \
|
|
poolname/Gmer4Lfe-Personal
|
|
|
|
# Add to ramdisk_setup.sh or array_start.sh custom scripts:
|
|
zfs load-key poolname/Gmer4Lfe-Personal
|
|
zfs mount poolname/Gmer4Lfe-Personal
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Manual unlock alternative (most secure — passphrase only in your head):
|
|
zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase
|
|
zfs mount poolname/Gmer4Lfe-Personal
|
|
```
|
|
|
|
---
|
|
|
|
### ── Add to master_host1.conf ────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Personal shares append to the daily sync after DAILY_SYNC_SHARES.
|
|
# Remote server receives encrypted blocks — cannot read content without your key.
|
|
#
|
|
HOST1_PERSONAL_SHARES=(
|
|
"/mnt/user/Gmer4Lfe-Personal"
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ STEP 9 — USER SCRIPTS SETUP ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
The ecosystem is designed so the User Scripts plugin has only a small number of entries —
|
|
each one an orchestrator. Individual scripts are never scheduled directly.
|
|
|
|
---
|
|
|
|
### ── At Startup of Array ─────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Create one script entry named "array start":
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
#!/bin/bash
|
|
bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Schedule: At Startup of Array
|
|
# Run as: Background Task
|
|
#
|
|
# This is the ONLY "At Startup of Array" entry needed.
|
|
# It launches everything in ARRAY_START_SCRIPTS from master.conf:
|
|
# inotify_tuning.sh — raise inotify limits before containers start
|
|
# docker_syslog_filter.sh — suppress veth log noise
|
|
# php_fpm_max_children.sh — WebGUI tuning
|
|
# ramdisk_setup.sh — create ramdisk before Emby starts
|
|
# docker_network_connect.sh — connect containers to extra networks
|
|
# system_watchdog.sh — continuous system health monitor
|
|
# docker_watchdog.sh — continuous container health monitor
|
|
# failover.sh — continuous mutual failover
|
|
```
|
|
|
|
---
|
|
|
|
### ── Cron Schedule ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Create one script entry per cron schedule below.
|
|
# All entries: Run as Background Task
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# Every 3 minutes — transcode cleanup + manager:
|
|
*/3 * * * *
|
|
bash /mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh
|
|
|
|
# Every 30 minutes — Emby dirty sync (watch states, library delta):
|
|
*/30 * * * *
|
|
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
|
/mnt/user/Media_Server/Emby --profile=emby-failover
|
|
|
|
# Every 6 hours — failed import + stalled download recovery:
|
|
0 */6 * * *
|
|
bash /mnt/user/appdata/unraid_scripts/Orchestrators/arrs_failed_stalled_recovery.sh
|
|
|
|
# 1am daily — full maintenance window:
|
|
# git pull → rsync all shares → permissions → cleaners → arr cleanup → docker restart
|
|
0 1 * * *
|
|
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh
|
|
|
|
# 2:30am Sunday — weekly maintenance window:
|
|
# stop containers → pull updates → clean sync → start containers → weekly restarts
|
|
30 2 * * 0
|
|
bash /mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh
|
|
|
|
# 8am daily — health digest (DIGEST_PROFILE in master.conf controls when it notifies):
|
|
0 8 * * *
|
|
bash /mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh
|
|
|
|
# Every 6 hours — inotify + php-fpm utilisation snapshot:
|
|
0 */6 * * *
|
|
bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh
|
|
|
|
# Sunday morning — weekly reports:
|
|
0 6 * * 0 bash .../Monitors/zfs_memory_snapshot.sh
|
|
0 7 * * 0 bash .../Monitors/smart_health.sh
|
|
0 9 * * 0 bash .../Monitors/cert_monitor.sh
|
|
0 10 * * 0 bash .../Monitors/backup_verify.sh
|
|
0 11 * * 0 bash .../Monitors/emby_session_report.sh
|
|
0 11 * * 0 bash .../Monitors/bandwidth_monitor.sh --report
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
> **Set all entries to "Background Task"** — output streams correctly to the User
|
|
> Scripts log rather than buffering in the browser tab. Non-background tasks can
|
|
> appear to hang on long-running scripts.
|
|
|
|
---
|
|
|
|
## ━━━ STEP 10 — VERIFY THE SETUP ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Before relying on scheduled jobs, test manually from the terminal on HOST1.
|
|
Test with `--dry-run` first — no changes made, but the full pre-flight and
|
|
configuration resolution runs.
|
|
|
|
---
|
|
|
|
### ── Test a Single Profile Sync ─────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Dry run with verbose output — shows every decision the script makes:
|
|
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
|
/mnt/user/appdata-Failover/Arrs_Stack --dry-run --log
|
|
```
|
|
|
|
```
|
|
Expected output (healthy):
|
|
━━━ ⚙️ Setup ━━━
|
|
Host: HOST1 (unRAID-Gmer4Lfe) → HOST2 (unRAID-Jayred365)
|
|
Remote IP: 100.x.x.x
|
|
Profile: arrs_stack
|
|
|
|
━━━ 🛡️ Pre-flight Checks ━━━
|
|
✅ Remote reachable
|
|
✅ version parity — both on unRAID X.Y.Z
|
|
✅ Remote Docker daemon responding
|
|
✅ Remote rootfs: 12% (threshold: 75%)
|
|
✅ Remote share exists and not empty
|
|
✅ All pre-flight checks passed
|
|
```
|
|
|
|
---
|
|
|
|
### ── Test the Daily Orchestrator ─────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Dry run of the full daily window — shows every job that would run:
|
|
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --dry-run
|
|
```
|
|
|
|
```
|
|
If any pre-flight check fails, the script aborts with a clear error message
|
|
before touching anything. Fix the reported issue and re-run --dry-run.
|
|
|
|
Common pre-flight failures and their causes:
|
|
"Remote not reachable" → Tailscale not connected on HOST2
|
|
"Version mismatch" → different unRAID versions — update before syncing
|
|
"Remote rootfs above X%" → HOST2's root filesystem nearly full
|
|
"Remote share missing" → share doesn't exist on HOST2 yet (see Step 11)
|
|
"Docker daemon not responding" → HOST2's Docker service not started
|
|
```
|
|
|
|
---
|
|
|
|
### ── Check the Configuration Resolved Correctly ──────────────────────────────
|
|
|
|
```bash
|
|
# --status shows how master.conf resolved for this server and profile:
|
|
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
|
/mnt/user/appdata-Failover/Arrs_Stack --status
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ STEP 11 — INITIAL HOST2 SYNC ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
If HOST2 is being set up from scratch with empty shares:
|
|
|
|
---
|
|
|
|
### ── Create Share Structure on HOST2 ────────────────────────────────────────
|
|
|
|
```bash
|
|
# On HOST2 — start the array and create shares via the unRAID UI.
|
|
# Or use the share recreation tool to create disk directories from HOST1's cfg files:
|
|
bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh
|
|
```
|
|
|
|
---
|
|
|
|
### ── Initial Push From HOST1 ─────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# On HOST1 — push all shares to HOST2 for the first time:
|
|
# Use --log for verbose output on first run
|
|
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --log
|
|
```
|
|
|
|
```
|
|
First run may take several hours for large libraries — this is normal.
|
|
The scheduled nightly sync will be incremental after the initial push.
|
|
Progress shows per-share throughout.
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ NAMING CONSISTENCY — THIS IS REQUIRED ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
The ecosystem uses one codebase on both servers. This only works if containers and
|
|
shares have identical names on both servers. This is not configurable — it is a
|
|
design requirement.
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Container names must match exactly on both servers:
|
|
"Emby" ← both HOST1 and HOST2
|
|
"NginxProxyManager" ← both HOST1 and HOST2
|
|
"Mariadb-Authelia" ← both HOST1 and HOST2
|
|
|
|
# Share paths must match exactly on both servers:
|
|
/mnt/user/Movies ← both HOST1 and HOST2 (HOST2 has a mirror)
|
|
/mnt/user/Tv_Shows ← both HOST1 and HOST2
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# If a container has a different name on one server: the script skips it
|
|
# without error. It silently does the wrong thing. You only notice when
|
|
# the container is not stopped during a sync that requires it to stop.
|
|
#
|
|
# If a share has a different path: rsync.sh aborts with "remote share missing".
|
|
# Easier to catch — but still requires renaming the share to fix.
|
|
#
|
|
# Keep names consistent and one codebase covers both servers automatically.
|
|
# Diverge and every script that touches containers or shares needs custom logic.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ REPOSITORY STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```
|
|
Unraid_Scripts/
|
|
├── master.conf ← All shared configuration — edit this file
|
|
├── master_host1.conf ← HOST1-specific configuration
|
|
├── master_host2.conf ← HOST2-specific configuration
|
|
├── common.sh ← Shared library — functions used by all scripts
|
|
├── load_config.sh ← Config loader — sources all conf files
|
|
│
|
|
├── Orchestrators/
|
|
│ ├── array_start.sh ← Single "At Startup of Array" entry point
|
|
│ ├── daily_sync_maintenance.sh ← 1am daily window orchestrator
|
|
│ ├── weekly_sync_maintenance.sh ← Sunday 2:30am window orchestrator
|
|
│ ├── critical_sync_maintenance.sh ← Every 15 minutes — critical sync + partnership
|
|
│ ├── media_management.sh ← Permissions + cleaners + arr cleanup
|
|
│ ├── transcode_management.sh ← Transcode cleanup then manager
|
|
│ └── arrs_failed_stalled_recovery.sh ← Failed import + stalled download recovery
|
|
│
|
|
├── Rsync/
|
|
│ └── rsync.sh ← Core rsync script — called per share
|
|
│
|
|
├── Failover/
|
|
│ ├── failover.sh ← Mutual container failover — continuous loop
|
|
│ ├── failover_test.sh ← Controlled iptables failover simulation
|
|
│ └── failover_state_reset.sh ← Reset failover state manually
|
|
│
|
|
├── Docker_Essentials/
|
|
│ ├── docker_watchdog.sh ← Two-tier container monitor — continuous loop
|
|
│ ├── docker_daily_restart.sh ← Nightly container restarts
|
|
│ ├── docker_weekly_restart.sh ← Weekly container restarts
|
|
│ ├── docker_network_connect.sh ← Ensure networks + connections at array start
|
|
│ └── watchdog_skip_list_manager.sh ← Skip list inspection and recovery
|
|
│
|
|
├── unRAID_Essentials/
|
|
│ ├── system_watchdog.sh ← Three-tier system health monitor — continuous
|
|
│ ├── ramdisk_setup.sh ← Creates ramdisk + symlink at array start
|
|
│ ├── inotify_tuning.sh ← Raise inotify limits at array start
|
|
│ ├── docker_syslog_filter.sh ← Suppress veth log noise
|
|
│ ├── php_fpm_max_children.sh ← WebGUI performance tuning
|
|
│ ├── server_reboot.sh ← Graceful reboot with pre-flight warnings
|
|
│ ├── mover_stop.sh ← Stop mover cleanly with wall warning
|
|
│ ├── clear_logs.sh ← Size-threshold log cleanup
|
|
│ ├── webgui_restart.sh ← nginx → php-fpm → emhttp escalation
|
|
│ └── git_pull_execute.sh ← Pull latest scripts from Gitea
|
|
│
|
|
├── Media/
|
|
│ ├── media_shares_permissions.sh ← Apply permissions to media shares
|
|
│ ├── media_cleaner.sh ← Remove junk files from media shares
|
|
│ ├── lidarr_cleanup.sh ← Remove orphaned music files (HOST1 only)
|
|
│ ├── sonarr_cleanup.sh ← Remove orphaned TV files (host-aware)
|
|
│ └── radarr_cleanup.sh ← Remove orphaned movie files (host-aware)
|
|
│
|
|
├── Transcodes/
|
|
│ ├── transcode_manager.sh ← Ramdisk/SSD symlink management
|
|
│ └── transcode_cleanup.sh ← Remove stale segment files
|
|
│
|
|
├── Monitors/
|
|
│ ├── cert_monitor.sh ← SSL cert expiry via live TLS connection
|
|
│ ├── backup_verify.sh ← rsync mirror MD5 checksum verification
|
|
│ ├── smart_health.sh ← Drive SMART attribute monitoring
|
|
│ ├── zfs_memory_snapshot.sh ← ZFS health + ARC + memory report
|
|
│ ├── bandwidth_monitor.sh ← rsync transfer logging + weekly report
|
|
│ ├── weekly_health_digest.sh ← Full ecosystem health aggregation
|
|
│ ├── emby_session_report.sh ← Emby streaming usage statistics
|
|
│ ├── system_tuning_monitor.sh ← inotify + php-fpm utilisation tracking
|
|
│ └── continuous_scripts_status.sh ← Live dashboard for background processes
|
|
│
|
|
├── Partnership/
|
|
│ └── partnership_manage.sh ← Two-server relationship lifecycle manager
|
|
│
|
|
└── Tools/
|
|
├── recreate_shares.sh ← Create share directories from cfg files
|
|
├── bulk_permissions_repair.sh ← One-shot permission repair
|
|
├── rsync_stop.sh ← Stop active rsync jobs cleanly
|
|
├── user_scripts_stop.sh ← Stop running user script processes
|
|
├── server_reboot.sh ← Graceful scheduled reboot
|
|
├── zfs_pool_scrub.sh ← Trigger ZFS pool scrub
|
|
└── container_data_export.sh ← Export container configuration
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### 🔴 SSH Connection Refused / Timeout
|
|
|
|
```bash
|
|
# Verify SSH is enabled on the remote:
|
|
# Settings → Management Access → Secure Shell → Enabled
|
|
|
|
# Verify Tailscale is connected:
|
|
tailscale ip -4 unRAID-Jayred365 # should return 100.x.x.x
|
|
|
|
# Test SSH manually with the key:
|
|
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@$(tailscale ip -4 unRAID-Jayred365) "hostname"
|
|
# Expected: unRAID-Jayred365
|
|
# If password prompted: key not authorised — recheck Step 3b
|
|
```
|
|
|
|
---
|
|
|
|
### 🔴 Pre-flight Aborts on Remote Rootfs
|
|
|
|
```bash
|
|
# Remote rootfs above ROOTFS_WARN threshold
|
|
# Check current usage on remote:
|
|
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "df /"
|
|
|
|
# Common cause: array not started, drives not mounted
|
|
# Verify array is started on HOST2 before running syncs
|
|
```
|
|
|
|
---
|
|
|
|
### 🔴 Remote Share Missing
|
|
|
|
```bash
|
|
# Share exists locally but not on remote
|
|
# Verify the share exists on HOST2:
|
|
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "ls /mnt/user/"
|
|
|
|
# If missing — create the share on HOST2 first, then run initial sync (Step 11)
|
|
# Or run recreate_shares.sh on HOST2 to create directories from HOST1's cfg files
|
|
```
|
|
|
|
---
|
|
|
|
### 🔴 Containers Not Stopping / Starting
|
|
|
|
```bash
|
|
# Verify container names in master_host*.conf match Docker exactly — case sensitive
|
|
# Check what Docker actually calls the container:
|
|
docker ps --format "{{.Names}}"
|
|
|
|
# Test Docker commands to remote manually:
|
|
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Names}}'"
|
|
```
|
|
|
|
---
|
|
|
|
### 🔴 Profile Not Matching
|
|
|
|
```bash
|
|
# Profile key = directory basename lowercased
|
|
# /mnt/user/appdata-Failover/Arrs_Stack → basename: Arrs_Stack → key: arrs_stack
|
|
|
|
# Override explicitly if basename doesn't match a profile name:
|
|
bash rsync.sh /mnt/user/appdata-Failover/My_Stuff --profile=arrs_stack
|
|
|
|
# Verify what profile resolved for a path:
|
|
bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status
|
|
```
|
|
|
|
---
|
|
|
|
### 🔴 Script Not Found
|
|
|
|
```bash
|
|
# Verify repo was cloned to the correct location:
|
|
ls /mnt/user/appdata/unraid_scripts/master.conf
|
|
|
|
# Make scripts executable:
|
|
find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \;
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ AVAILABLE FLAGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All scripts support these flags. Use `--dry-run` before any live operation.
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
--dry-run run without making any changes — pre-flight still runs ✅
|
|
--log verbose output — show every decision made
|
|
--status show resolved configuration and exit — no rsync, no sync
|
|
--no-log suppress verbose output (some scripts)
|
|
|
|
# Examples:
|
|
bash rsync.sh /mnt/user/Movies --dry-run --log # preview a media sync
|
|
bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status # check profile resolution
|
|
bash daily_sync_maintenance.sh --dry-run # preview full daily window
|
|
bash docker_watchdog.sh --status # check watchdog state
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
``` |