Varaverk/Rsync/README-Rsync_Setup.md

# Rsync Setup Guide
> **Status:** Work in Progress  
> For the unRAID Rsync Ecosystem — `common.sh` · `Master.conf` · `rsync.sh` · `daily_sync.sh`

---

## Overview

This guide walks through setting up the rsync ecosystem on both your primary and secondary unRAID 7.x servers. By the end you will have:

- A Gitea repository cloned to both servers
- SSH keys configured for server-to-server communication
- Tailscale running on both servers for secure networking
- Scripts scheduled and running via the User Scripts plugin
- Automated daily sync of media shares and appdata profiles

---

## Prerequisites

Both servers need the following before starting:

- unRAID 7.x
- [User Scripts plugin](https://forums.unraid.net/topic/48286-plugin-user-scripts/) installed via Community Applications
- [Tailscale plugin](https://forums.unraid.net/topic/136889-tailscale-plugin/) installed via Community Applications
- Access to a Gitea instance (self-hosted or remote)
- Terminal access to both servers (via unRAID UI → Tools → Terminal, or SSH)

---

## Step 1 — Tailscale Setup

Tailscale provides the secure network tunnel between your two servers. The scripts resolve the remote server's IP via Tailscale at runtime.

### On Both Servers

1. Open **Apps** in the unRAID UI
2. Search for **Tailscale** and install the plugin
3. Once installed go to **Settings → Tailscale**
4. Click **Connect** and authenticate with your Tailscale account
5. Verify both servers appear in your [Tailscale admin console](https://login.tailscale.com/admin/machines)

### Verify Connectivity

Run this on the primary to confirm it can see the secondary:

```bash
tailscale ip -4 unRAID-Jayred365
```

You should get back a `100.x.x.x` IP. If not, check that both machines are authenticated and connected in the Tailscale admin console.

> **Note:** The hostnames used in `Master.conf` (`HOST1` and `HOST2`) must match the Tailscale machine names exactly — these are case sensitive.

---

## Step 2 — Generate SSH Keys

The scripts use SSH keys for two purposes:
- **Server-to-server rsync** — primary authenticates to secondary (and vice versa)
- **Gitea access** — both servers pull from the git repository

### 2a — Server-to-Server Keys

Run the following on **each server** to generate its rsync key. Replace the filename with the appropriate server name.

**On Primary (unRAID-Gmer4Lfe):**

```bash
ssh-keygen -t ed25519 -f /root/.ssh/Gmer4Lfe-rsync-key -C "gmer4lfe-rsync" -N ""
```

**On Secondary (unRAID-Jayred365):**

```bash
ssh-keygen -t ed25519 -f /root/.ssh/Jayred365-rsync-key -C "jayred365-rsync" -N ""
```

### 2b — Copy Public Keys to Each Server

The primary's public key must be authorised on the secondary, and vice versa.

**Copy primary key → secondary:**

```bash
# Run on primary
cat /root/.ssh/Gmer4Lfe-rsync-key.pub
```

Copy the output. Then on the secondary:

```bash
# Run on secondary
mkdir -p /root/.ssh
echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
```

**Copy secondary key → primary:**

```bash
# Run on secondary
cat /root/.ssh/Jayred365-rsync-key.pub
```

Copy the output. Then on the primary:

```bash
# Run on primary
mkdir -p /root/.ssh
echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
```

### 2c — Test the Connection

From the primary, test that it can SSH to the secondary without a password prompt:

```bash
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@$(tailscale ip -4 unRAID-Jayred365) "echo connected"
```

You should see `connected`. If prompted for a password the key was not authorised correctly — recheck Step 2b.

### 2d — Gitea SSH Key

Generate a separate key for Gitea access on each server:

```bash
ssh-keygen -t ed25519 -f /root/.ssh/id_gitea_rsync -C "unraid-gitea" -N ""
```

Add the public key to your Gitea account:

```bash
cat /root/.ssh/id_gitea_rsync.pub
```

Copy the output and add it in Gitea under **Settings → SSH / GPG Keys → Add Key**.

---

## Step 3 — Enable SSH on unRAID

unRAID 7.x has SSH disabled by default. Enable it on both servers so the scripts can connect between them.

1. Go to **Settings → Management Access**
2. Under **Secure Shell** set **SSH** to `Enabled`
3. Set **SSH port** to `22` (default)
4. Click **Apply**

> **Security note:** SSH is only exposed on your local network and Tailscale interface. The rsync scripts connect via the Tailscale IP so traffic is encrypted end-to-end.

---

## Step 4 — Clone the Git Repository

The scripts live in a Gitea repository. Both servers clone from the same repo so updates propagate everywhere via a single git pull.

### On Both Servers

```bash
# Create the target directory
mkdir -p /mnt/user/appdata/unraid_scripts

# Clone the repository
GIT_SSH_COMMAND="ssh -i /root/.ssh/id_gitea_rsync" \
    git clone git@YOUR_GITEA_HOST:YOUR_USER/Unraid_Scripts.git \
    /mnt/user/appdata/unraid_scripts
```

Replace `YOUR_GITEA_HOST` and `YOUR_USER` with your Gitea server address and username.

### Verify the Structure

```bash
ls /mnt/user/appdata/unraid_scripts
```

You should see:

```
Master.conf
common.sh
Rsync/
    rsync.sh
Orchestrators/
    daily_sync.sh
Tools/
    recreate_shares.sh
```

### Make Scripts Executable

```bash
chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh
chmod +x /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
chmod +x /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh
```

---

## Step 5 — Configure Master.conf

All user configuration lives in `Master.conf`. Open it and adjust the following to match your setup:

```bash
nano /mnt/user/appdata/unraid_scripts/Master.conf
```

### Required Changes

| Variable | Description | Example |
|---|---|---|
| `HOST1` | Hostname of your primary server | `unRAID-Gmer4Lfe` |
| `HOST2` | Hostname of your secondary server | `unRAID-Jayred365` |
| `HOST1_SSH_KEY` | Path to primary's rsync private key | `/root/.ssh/Gmer4Lfe-rsync-key` |
| `HOST2_SSH_KEY` | Path to secondary's rsync private key | `/root/.ssh/Jayred365-rsync-key` |
| `REPO_SSH` | SSH URL of your Gitea repository | `git@192.168.50.2:User/Unraid_Scripts.git` |
| `GITEA_SSH_KEY` | Path to Gitea private key | `/root/.ssh/id_gitea_rsync` |
| `BW_LIMIT` | Global bandwidth limit in KB/s | `12500` |
| `ROOTFS_WARN` | Remote rootfs % threshold before aborting | `75` |

### Daily Sync Shares

Add the full paths of all media shares you want synced nightly:

```bash
DAILY_SYNC_SHARES=(
    /mnt/user/Movies
    /mnt/user/Tv_Shows
    /mnt/user/Music
    # add more here
)
```

### Profiles

Profiles control per-share rsync behaviour for your frequently synced appdata shares. Each profile is matched by the directory basename (lowercased) — or overridden with `--profile=name`.

```bash
# One array drives both local and remote container stops
# Local stops first (flush databases) then remote stops (clean receive)
# Same container names on both HOST1 and HOST2 — no duplication needed
# Containers not found on a server are skipped gracefully
declare -A PROFILE_CRITICAL_CONTAINER_NAMES=(
    [critical-data]="Mariadb-Authelia Mariadb-Authelia-Secondary Redis-Authelia Redis-Authelia-Secondary Lldap-Gmer4Lfe NginxProxyManager Authelia Authelia-Secondary"
    [important-data]="Postgres-NextCloud NextCloud"
    [emby]="Emby"           # nightly clean sync — Emby stopped both sides
    [emby-failover]=""      # dirty sync — Emby stays running
)
```

Any share with no matching profile falls through to the global `DEFAULT_RSYNC_OPTS`. Containers not found on a server are skipped gracefully — only containers that were actually running get restarted.

**Two Emby profiles:**

```
emby-failover — frequent dirty sync (every 30-60min):
  Emby stays running on both sides
  WAL and SHM excluded — safe while Emby is active
  Only critical failover data: users.db, library.db, authentication.db, config/
  Fast, small dataset, high bandwidth
  This is also what gets written back during failover handback

emby — weekly clean sync (via nightly_critical_full_sync.sh Sunday 2:30am):
  Emby stopped on both sides — WAL checkpointed on shutdown
  Full mirror: metadata, plugins, config all included
  Minimal excludes: logs, transcodes, cache, crash files only
  Complete faithful state pushed once per week
  Cache stays warm on HOST2 all week — only reset on Sunday
  emby-failover handles the critical state between weekly syncs
```

Usage with profile override:
```bash
# Dirty sync — Emby stays running
bash rsync.sh /mnt/user/Media_Server/Emby --profile=emby-failover

# Clean sync — called by nightly_critical_full_sync.sh, Emby stopped via profile
bash rsync.sh /mnt/user/Media_Server/Emby
```

---

## Step 6 — Set Up User Scripts

The User Scripts plugin is how unRAID schedules and runs the scripts. Each sync job is its own script entry in the plugin.

### Frequent Sync Jobs (Scheduled Individually)

Create one script entry per appdata profile. Go to **Plugins → User Scripts → Add New Script**.

Name it descriptively — e.g. `rsync appdata arrs_stack`.

In the script body paste:

```bash
#!/bin/bash
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
```

Set the schedule to match your desired frequency:

| Profile | Schedule | Notes |
|---|---|---|
| `emby-failover` | Every 30-60 min | Dirty sync — Emby running, critical data only |
| `emby` | Weekly Sunday via `nightly_critical_full_sync.sh` | Clean sync — Emby stopped, full mirror, cache stays warm all week |
| `Critical-Data` | Weekly Sunday via `nightly_critical_full_sync.sh` | Auth stack — clean weekly sync |
| `Important-Data` | Every 6-12 hours | NextCloud file changes |
| `Arrs_Stack` | Every 12-24 hours | Arr databases |
| `Gmer4Lfe` | Daily or weekly | Personal appdata, rarely changes |

> **Important:** Set each script to run as a **Background Task** — this ensures output streams correctly to the log rather than buffering in the browser.

### Daily Sync Orchestrator

Create one more script entry for the daily media sync:

Name it `daily media sync`.

In the script body paste:

```bash
#!/bin/bash
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
```

Set the schedule to **Daily at 01:00**.

---

## Step 7 — Verify the Setup

Before letting the scheduled jobs run, do a manual test from the terminal on the primary:

```bash
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --log
```

A healthy run will show:

```
━━━ ⚙️ Setup ━━━
ℹ️  [INFO]  🖥️ Host: unRAID-Gmer4Lfe → unRAID-Jayred365
ℹ️  [INFO]  🌐 Remote IP: 100.x.x.x

━━━ 🛡️ Pre-flight Checks ━━━
ℹ️  [INFO]  📡 unRAID-Jayred365 is reachable
ℹ️  [INFO]  🩺 Remote rootfs: 12% used (threshold: 75%)
ℹ️  [INFO]  🩺 Remote share verified: ...
ℹ️  [INFO]  💾 disk1 🟢 — share present
✅ [OK]  All disks backing share are online
```

If any pre-flight check fails the script will abort with a clear error and hint before touching anything.

---

## Step 8 — Secondary Server Initial Setup

If setting up the secondary from scratch (no existing data):

1. Complete Steps 1–5 on the secondary
2. Start the array and create your shares in the unRAID UI
3. Run the share recreation tool to create disk directories from your cfg files:

```bash
bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh
```

4. Temporarily remove `--delete` from `DEFAULT_RSYNC_OPTS` in `Master.conf`
5. Run the initial push from the primary — the `.recovery` marker files allow rsync to populate empty shares without aborting
6. Once complete, restore `--delete` to `Master.conf`
7. The next nightly run will clean up the `.recovery` marker files automatically

---

## Troubleshooting

### SSH connection refused
- Verify SSH is enabled on the target server (Step 3)
- Check the correct key is referenced in `Master.conf`
- Confirm the Tailscale IP resolves: `tailscale ip -4 HOSTNAME`

### Pre-flight aborts on rootfs
- Remote rootfs is above `ROOTFS_WARN` threshold
- Check if the remote array is started and drives are mounted
- `df /` on the remote to see current usage

### Pre-flight aborts on empty share
- Share exists but has no content — drives may not be mounted
- Run `recreate_shares.sh` if setting up fresh
- Check array status on the remote server

### Pre-flight aborts on disk check
- One or more disks backing the share are not mounted
- Check **Main → Array Devices** on the remote for offline disks
- Verify disk assignments are correct after any hardware changes

### Script not found
- Verify the repo was cloned to `/mnt/user/appdata/unraid_scripts/`
- Check scripts are executable: `chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh`

### Containers not stopping/starting
- Verify container names in `Master.conf` match exactly what Docker shows
- Check SSH key has access to run docker commands on the remote
- Test manually: `ssh -i /root/.ssh/KEY root@REMOTE_IP "docker ps"`

---

## Available Flags

All scripts support the following flags:

| Flag | Description |
|---|---|
| `--dry-run` or `-n` | Run without making any changes |
| `--log` | Enable verbose logging output |
| `--no-log` | Disable logging (overrides Master.conf) |
| `--status` | Print resolved configuration and exit |

Example:

```bash
# Preview what would be synced without transferring anything
bash rsync.sh /mnt/user/Movies --dry-run --log

# Check what profile and settings resolved for a share
bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status
```

---

## Repository Structure

```
Unraid_Scripts/
├── Master.conf               # All user configuration — edit this file only
├── common.sh                 # Shared library — functions used by all scripts
├── Rsync/
│   └── rsync.sh              # Core rsync script — called per share
├── Orchestrators/
│   └── daily_sync.sh         # Daily media sync orchestrator
└── Tools/
    └── recreate_shares.sh    # Share directory recreation from cfg files
```

---

*Guide version aligned with common.sh v1.6*