449 lines
14 KiB
Markdown
449 lines
14 KiB
Markdown
# Rsync Setup Guide
|
||
> **Status:** Work in Progress
|
||
> For the unRAID Rsync Ecosystem — `common.sh` · `Master.conf` · `rsync.sh` · `daily_sync.sh`
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
This guide walks through setting up the rsync ecosystem on both your primary and secondary unRAID 7.x servers. By the end you will have:
|
||
|
||
- A Gitea repository cloned to both servers
|
||
- SSH keys configured for server-to-server communication
|
||
- Tailscale running on both servers for secure networking
|
||
- Scripts scheduled and running via the User Scripts plugin
|
||
- Automated daily sync of media shares and appdata profiles
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
Both servers need the following before starting:
|
||
|
||
- unRAID 7.x
|
||
- [User Scripts plugin](https://forums.unraid.net/topic/48286-plugin-user-scripts/) installed via Community Applications
|
||
- [Tailscale plugin](https://forums.unraid.net/topic/136889-tailscale-plugin/) installed via Community Applications
|
||
- Access to a Gitea instance (self-hosted or remote)
|
||
- Terminal access to both servers (via unRAID UI → Tools → Terminal, or SSH)
|
||
|
||
---
|
||
|
||
## Step 1 — Tailscale Setup
|
||
|
||
Tailscale provides the secure network tunnel between your two servers. The scripts resolve the remote server's IP via Tailscale at runtime.
|
||
|
||
### On Both Servers
|
||
|
||
1. Open **Apps** in the unRAID UI
|
||
2. Search for **Tailscale** and install the plugin
|
||
3. Once installed go to **Settings → Tailscale**
|
||
4. Click **Connect** and authenticate with your Tailscale account
|
||
5. Verify both servers appear in your [Tailscale admin console](https://login.tailscale.com/admin/machines)
|
||
|
||
### Verify Connectivity
|
||
|
||
Run this on the primary to confirm it can see the secondary:
|
||
|
||
```bash
|
||
tailscale ip -4 unRAID-Jayred365
|
||
```
|
||
|
||
You should get back a `100.x.x.x` IP. If not, check that both machines are authenticated and connected in the Tailscale admin console.
|
||
|
||
> **Note:** The hostnames used in `Master.conf` (`HOST1` and `HOST2`) must match the Tailscale machine names exactly — these are case sensitive.
|
||
|
||
---
|
||
|
||
## Step 2 — Generate SSH Keys
|
||
|
||
The scripts use SSH keys for two purposes:
|
||
- **Server-to-server rsync** — primary authenticates to secondary (and vice versa)
|
||
- **Gitea access** — both servers pull from the git repository
|
||
|
||
### 2a — Server-to-Server Keys
|
||
|
||
Run the following on **each server** to generate its rsync key. Replace the filename with the appropriate server name.
|
||
|
||
**On Primary (unRAID-Gmer4Lfe):**
|
||
|
||
```bash
|
||
ssh-keygen -t ed25519 -f /root/.ssh/Gmer4Lfe-rsync-key -C "gmer4lfe-rsync" -N ""
|
||
```
|
||
|
||
**On Secondary (unRAID-Jayred365):**
|
||
|
||
```bash
|
||
ssh-keygen -t ed25519 -f /root/.ssh/Jayred365-rsync-key -C "jayred365-rsync" -N ""
|
||
```
|
||
|
||
### 2b — Copy Public Keys to Each Server
|
||
|
||
The primary's public key must be authorised on the secondary, and vice versa.
|
||
|
||
**Copy primary key → secondary:**
|
||
|
||
```bash
|
||
# Run on primary
|
||
cat /root/.ssh/Gmer4Lfe-rsync-key.pub
|
||
```
|
||
|
||
Copy the output. Then on the secondary:
|
||
|
||
```bash
|
||
# Run on secondary
|
||
mkdir -p /root/.ssh
|
||
echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
|
||
chmod 600 /root/.ssh/authorized_keys
|
||
```
|
||
|
||
**Copy secondary key → primary:**
|
||
|
||
```bash
|
||
# Run on secondary
|
||
cat /root/.ssh/Jayred365-rsync-key.pub
|
||
```
|
||
|
||
Copy the output. Then on the primary:
|
||
|
||
```bash
|
||
# Run on primary
|
||
mkdir -p /root/.ssh
|
||
echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
|
||
chmod 600 /root/.ssh/authorized_keys
|
||
```
|
||
|
||
### 2c — Test the Connection
|
||
|
||
From the primary, test that it can SSH to the secondary without a password prompt:
|
||
|
||
```bash
|
||
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@$(tailscale ip -4 unRAID-Jayred365) "echo connected"
|
||
```
|
||
|
||
You should see `connected`. If prompted for a password the key was not authorised correctly — recheck Step 2b.
|
||
|
||
### 2d — Gitea SSH Key
|
||
|
||
Generate a separate key for Gitea access on each server:
|
||
|
||
```bash
|
||
ssh-keygen -t ed25519 -f /root/.ssh/id_gitea_rsync -C "unraid-gitea" -N ""
|
||
```
|
||
|
||
Add the public key to your Gitea account:
|
||
|
||
```bash
|
||
cat /root/.ssh/id_gitea_rsync.pub
|
||
```
|
||
|
||
Copy the output and add it in Gitea under **Settings → SSH / GPG Keys → Add Key**.
|
||
|
||
---
|
||
|
||
## Step 3 — Enable SSH on unRAID
|
||
|
||
unRAID 7.x has SSH disabled by default. Enable it on both servers so the scripts can connect between them.
|
||
|
||
1. Go to **Settings → Management Access**
|
||
2. Under **Secure Shell** set **SSH** to `Enabled`
|
||
3. Set **SSH port** to `22` (default)
|
||
4. Click **Apply**
|
||
|
||
> **Security note:** SSH is only exposed on your local network and Tailscale interface. The rsync scripts connect via the Tailscale IP so traffic is encrypted end-to-end.
|
||
|
||
---
|
||
|
||
## Step 4 — Clone the Git Repository
|
||
|
||
The scripts live in a Gitea repository. Both servers clone from the same repo so updates propagate everywhere via a single git pull.
|
||
|
||
### On Both Servers
|
||
|
||
```bash
|
||
# Create the target directory
|
||
mkdir -p /mnt/user/appdata/unraid_scripts
|
||
|
||
# Clone the repository
|
||
GIT_SSH_COMMAND="ssh -i /root/.ssh/id_gitea_rsync" \
|
||
git clone git@YOUR_GITEA_HOST:YOUR_USER/Unraid_Scripts.git \
|
||
/mnt/user/appdata/unraid_scripts
|
||
```
|
||
|
||
Replace `YOUR_GITEA_HOST` and `YOUR_USER` with your Gitea server address and username.
|
||
|
||
### Verify the Structure
|
||
|
||
```bash
|
||
ls /mnt/user/appdata/unraid_scripts
|
||
```
|
||
|
||
You should see:
|
||
|
||
```
|
||
Master.conf
|
||
common.sh
|
||
Rsync/
|
||
rsync.sh
|
||
Orchestrators/
|
||
daily_sync.sh
|
||
Tools/
|
||
recreate_shares.sh
|
||
```
|
||
|
||
### Make Scripts Executable
|
||
|
||
```bash
|
||
chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh
|
||
chmod +x /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
|
||
chmod +x /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh
|
||
```
|
||
|
||
---
|
||
|
||
## Step 5 — Configure Master.conf
|
||
|
||
All user configuration lives in `Master.conf`. Open it and adjust the following to match your setup:
|
||
|
||
```bash
|
||
nano /mnt/user/appdata/unraid_scripts/Master.conf
|
||
```
|
||
|
||
### Required Changes
|
||
|
||
| Variable | Description | Example |
|
||
|---|---|---|
|
||
| `HOST1` | Hostname of your primary server | `unRAID-Gmer4Lfe` |
|
||
| `HOST2` | Hostname of your secondary server | `unRAID-Jayred365` |
|
||
| `HOST1_SSH_KEY` | Path to primary's rsync private key | `/root/.ssh/Gmer4Lfe-rsync-key` |
|
||
| `HOST2_SSH_KEY` | Path to secondary's rsync private key | `/root/.ssh/Jayred365-rsync-key` |
|
||
| `REPO_SSH` | SSH URL of your Gitea repository | `git@192.168.50.2:User/Unraid_Scripts.git` |
|
||
| `GITEA_SSH_KEY` | Path to Gitea private key | `/root/.ssh/id_gitea_rsync` |
|
||
| `BW_LIMIT` | Global bandwidth limit in KB/s | `12500` |
|
||
| `ROOTFS_WARN` | Remote rootfs % threshold before aborting | `75` |
|
||
|
||
### Daily Sync Shares
|
||
|
||
Add the full paths of all media shares you want synced nightly:
|
||
|
||
```bash
|
||
DAILY_SYNC_SHARES=(
|
||
/mnt/user/Movies
|
||
/mnt/user/Tv_Shows
|
||
/mnt/user/Music
|
||
# add more here
|
||
)
|
||
```
|
||
|
||
### Profiles
|
||
|
||
Profiles control per-share rsync behaviour for your frequently synced appdata shares. Each profile is matched by the directory basename (lowercased) — or overridden with `--profile=name`.
|
||
|
||
```bash
|
||
# One array drives both local and remote container stops
|
||
# Local stops first (flush databases) then remote stops (clean receive)
|
||
# Same container names on both HOST1 and HOST2 — no duplication needed
|
||
# Containers not found on a server are skipped gracefully
|
||
declare -A PROFILE_CRITICAL_CONTAINER_NAMES=(
|
||
[critical-data]="Mariadb-Authelia Mariadb-Authelia-Secondary Redis-Authelia Redis-Authelia-Secondary Lldap-Gmer4Lfe NginxProxyManager Authelia Authelia-Secondary"
|
||
[important-data]="Postgres-NextCloud NextCloud"
|
||
[emby]="Emby" # nightly clean sync — Emby stopped both sides
|
||
[emby-failover]="" # dirty sync — Emby stays running
|
||
)
|
||
```
|
||
|
||
Any share with no matching profile falls through to the global `DEFAULT_RSYNC_OPTS`. Containers not found on a server are skipped gracefully — only containers that were actually running get restarted.
|
||
|
||
**Two Emby profiles:**
|
||
|
||
```
|
||
emby-failover — frequent dirty sync (every 30-60min):
|
||
Emby stays running on both sides
|
||
WAL and SHM excluded — safe while Emby is active
|
||
Only critical failover data: users.db, library.db, authentication.db, config/
|
||
Fast, small dataset, high bandwidth
|
||
This is also what gets written back during failover handback
|
||
|
||
emby — nightly clean sync (via nightly_sync.sh at 2:30am):
|
||
Emby stopped on both sides — WAL checkpointed on shutdown
|
||
Full mirror: metadata, plugins, config all included
|
||
Minimal excludes: logs, transcodes, cache, crash files only
|
||
Complete faithful state pushed once per night
|
||
```
|
||
|
||
Usage with profile override:
|
||
```bash
|
||
# Dirty sync — Emby stays running
|
||
bash rsync.sh /mnt/user/appdata-Failover/Emby --profile=emby-failover
|
||
|
||
# Clean sync — called by nightly_sync.sh, Emby stopped via profile
|
||
bash rsync.sh /mnt/user/appdata-Failover/Emby
|
||
```
|
||
|
||
---
|
||
|
||
## Step 6 — Set Up User Scripts
|
||
|
||
The User Scripts plugin is how unRAID schedules and runs the scripts. Each sync job is its own script entry in the plugin.
|
||
|
||
### Frequent Sync Jobs (Scheduled Individually)
|
||
|
||
Create one script entry per appdata profile. Go to **Plugins → User Scripts → Add New Script**.
|
||
|
||
Name it descriptively — e.g. `rsync appdata arrs_stack`.
|
||
|
||
In the script body paste:
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
|
||
```
|
||
|
||
Set the schedule to match your desired frequency:
|
||
|
||
| Profile | Schedule | Notes |
|
||
|---|---|---|
|
||
| `emby-failover` | Every 30-60 min | Dirty sync — Emby running, critical data only |
|
||
| `emby` | via `nightly_sync.sh` | Clean sync — Emby stopped, full mirror |
|
||
| `Critical-Data` | via `nightly_sync.sh` | Auth stack — clean nightly sync |
|
||
| `Important-Data` | Every 6-12 hours | NextCloud file changes |
|
||
| `Arrs_Stack` | Every 12-24 hours | Arr databases |
|
||
| `Gmer4Lfe` | Daily or weekly | Personal appdata, rarely changes |
|
||
|
||
> **Important:** Set each script to run as a **Background Task** — this ensures output streams correctly to the log rather than buffering in the browser.
|
||
|
||
### Daily Sync Orchestrator
|
||
|
||
Create one more script entry for the daily media sync:
|
||
|
||
Name it `daily media sync`.
|
||
|
||
In the script body paste:
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
|
||
```
|
||
|
||
Set the schedule to **Daily at 01:00**.
|
||
|
||
---
|
||
|
||
## Step 7 — Verify the Setup
|
||
|
||
Before letting the scheduled jobs run, do a manual test from the terminal on the primary:
|
||
|
||
```bash
|
||
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --log
|
||
```
|
||
|
||
A healthy run will show:
|
||
|
||
```
|
||
━━━ ⚙️ Setup ━━━
|
||
ℹ️ [INFO] 🖥️ Host: unRAID-Gmer4Lfe → unRAID-Jayred365
|
||
ℹ️ [INFO] 🌐 Remote IP: 100.x.x.x
|
||
|
||
━━━ 🛡️ Pre-flight Checks ━━━
|
||
ℹ️ [INFO] 📡 unRAID-Jayred365 is reachable
|
||
ℹ️ [INFO] 🩺 Remote rootfs: 12% used (threshold: 75%)
|
||
ℹ️ [INFO] 🩺 Remote share verified: ...
|
||
ℹ️ [INFO] 💾 disk1 🟢 — share present
|
||
✅ [OK] All disks backing share are online
|
||
```
|
||
|
||
If any pre-flight check fails the script will abort with a clear error and hint before touching anything.
|
||
|
||
---
|
||
|
||
## Step 8 — Secondary Server Initial Setup
|
||
|
||
If setting up the secondary from scratch (no existing data):
|
||
|
||
1. Complete Steps 1–5 on the secondary
|
||
2. Start the array and create your shares in the unRAID UI
|
||
3. Run the share recreation tool to create disk directories from your cfg files:
|
||
|
||
```bash
|
||
bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh
|
||
```
|
||
|
||
4. Temporarily remove `--delete` from `DEFAULT_RSYNC_OPTS` in `Master.conf`
|
||
5. Run the initial push from the primary — the `.recovery` marker files allow rsync to populate empty shares without aborting
|
||
6. Once complete, restore `--delete` to `Master.conf`
|
||
7. The next nightly run will clean up the `.recovery` marker files automatically
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### SSH connection refused
|
||
- Verify SSH is enabled on the target server (Step 3)
|
||
- Check the correct key is referenced in `Master.conf`
|
||
- Confirm the Tailscale IP resolves: `tailscale ip -4 HOSTNAME`
|
||
|
||
### Pre-flight aborts on rootfs
|
||
- Remote rootfs is above `ROOTFS_WARN` threshold
|
||
- Check if the remote array is started and drives are mounted
|
||
- `df /` on the remote to see current usage
|
||
|
||
### Pre-flight aborts on empty share
|
||
- Share exists but has no content — drives may not be mounted
|
||
- Run `recreate_shares.sh` if setting up fresh
|
||
- Check array status on the remote server
|
||
|
||
### Pre-flight aborts on disk check
|
||
- One or more disks backing the share are not mounted
|
||
- Check **Main → Array Devices** on the remote for offline disks
|
||
- Verify disk assignments are correct after any hardware changes
|
||
|
||
### Script not found
|
||
- Verify the repo was cloned to `/mnt/user/appdata/unraid_scripts/`
|
||
- Check scripts are executable: `chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh`
|
||
|
||
### Containers not stopping/starting
|
||
- Verify container names in `Master.conf` match exactly what Docker shows
|
||
- Check SSH key has access to run docker commands on the remote
|
||
- Test manually: `ssh -i /root/.ssh/KEY root@REMOTE_IP "docker ps"`
|
||
|
||
---
|
||
|
||
## Available Flags
|
||
|
||
All scripts support the following flags:
|
||
|
||
| Flag | Description |
|
||
|---|---|
|
||
| `--dry-run` or `-n` | Run without making any changes |
|
||
| `--log` | Enable verbose logging output |
|
||
| `--no-log` | Disable logging (overrides Master.conf) |
|
||
| `--status` | Print resolved configuration and exit |
|
||
|
||
Example:
|
||
|
||
```bash
|
||
# Preview what would be synced without transferring anything
|
||
bash rsync.sh /mnt/user/Movies --dry-run --log
|
||
|
||
# Check what profile and settings resolved for a share
|
||
bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status
|
||
```
|
||
|
||
---
|
||
|
||
## Repository Structure
|
||
|
||
```
|
||
Unraid_Scripts/
|
||
├── Master.conf # All user configuration — edit this file only
|
||
├── common.sh # Shared library — functions used by all scripts
|
||
├── Rsync/
|
||
│ └── rsync.sh # Core rsync script — called per share
|
||
├── Orchestrators/
|
||
│ └── daily_sync.sh # Daily media sync orchestrator
|
||
└── Tools/
|
||
└── recreate_shares.sh # Share directory recreation from cfg files
|
||
```
|
||
|
||
---
|
||
|
||
*Guide version aligned with common.sh v1.6* |