true bidirectional with split truth modle set up. all scripts are fully bidirectional

2026-04-21 18:20:47 -04:00
parent 6b36b8c07a
commit d4bf31a5f4
9 changed files with 846 additions and 161 deletions
@@ -1,215 +1,438 @@
-This guide is a work in progress and generated by ai, dont have time for guides. so this works
+# Rsync Setup Guide
+> **Status:** Work in Progress  
+> For the unRAID Rsync Ecosystem — `common.sh` · `Master.conf` · `rsync.sh` · `daily_sync.sh`

-Rsync Setup Guide
+---

-Status: Work in Progress
-For the unRAID Rsync Ecosystem — common.sh · Master.conf · rsync.sh · daily_sync.sh
-
-Overview
+## Overview

 This guide walks through setting up the rsync ecosystem on both your primary and secondary unRAID 7.x servers. By the end you will have:

-SSH keys configured for server-to-server communication
-Tailscale running on both servers for secure networking
-Scripts scheduled and running via the User Scripts plugin
-Automated daily sync of media shares and appdata profiles
-Prerequisites
+- A Gitea repository cloned to both servers
+- SSH keys configured for server-to-server communication
+- Tailscale running on both servers for secure networking
+- Scripts scheduled and running via the User Scripts plugin
+- Automated daily sync of media shares and appdata profiles
+
+---
+
+## Prerequisites

 Both servers need the following before starting:

-unRAID 7.x
-User Scripts plugin installed via Community Applications
-Tailscale plugin installed via Community Applications
-Terminal access to both servers (via unRAID UI → Tools → Terminal, or SSH)
-Step 1 — Tailscale Setup
+- unRAID 7.x
+- [User Scripts plugin](https://forums.unraid.net/topic/48286-plugin-user-scripts/) installed via Community Applications
+- [Tailscale plugin](https://forums.unraid.net/topic/136889-tailscale-plugin/) installed via Community Applications
+- Access to a Gitea instance (self-hosted or remote)
+- Terminal access to both servers (via unRAID UI → Tools → Terminal, or SSH)
+
+---
+
+## Step 1 — Tailscale Setup

 Tailscale provides the secure network tunnel between your two servers. The scripts resolve the remote server's IP via Tailscale at runtime.

-On Both Servers
+### On Both Servers

-Open Apps in the unRAID UI
-Search for Tailscale and install the plugin
-Once installed, go to Settings → Tailscale
-Click Connect and authenticate with your Tailscale account
-Verify both servers appear in your Tailscale admin console
+1. Open **Apps** in the unRAID UI
+2. Search for **Tailscale** and install the plugin
+3. Once installed go to **Settings → Tailscale**
+4. Click **Connect** and authenticate with your Tailscale account
+5. Verify both servers appear in your [Tailscale admin console](https://login.tailscale.com/admin/machines)

-Verify Connectivity
+### Verify Connectivity

 Run this on the primary to confirm it can see the secondary:

+```bash
 tailscale ip -4 unRAID-Jayred365
+```

-You should get back a 100.x.x.x IP. If not, check both servers in the Tailscale admin console.
+You should get back a `100.x.x.x` IP. If not, check that both machines are authenticated and connected in the Tailscale admin console.

-Note: The hostnames used in Master.conf (HOST1 and HOST2) must match the Tailscale machine names exactly — these are case sensitive.
+> **Note:** The hostnames used in `Master.conf` (`HOST1` and `HOST2`) must match the Tailscale machine names exactly — these are case sensitive.

-Step 2 — Generate SSH Keys (Server-to-Server)
+---

-The scripts use SSH keys for server-to-server rsync.
+## Step 2 — Generate SSH Keys

-On Primary (unRAID-Gmer4Lfe):
+The scripts use SSH keys for two purposes:
+- **Server-to-server rsync** — primary authenticates to secondary (and vice versa)
+- **Gitea access** — both servers pull from the git repository

+### 2a — Server-to-Server Keys
+
+Run the following on **each server** to generate its rsync key. Replace the filename with the appropriate server name.
+
+**On Primary (unRAID-Gmer4Lfe):**
+
+```bash
 ssh-keygen -t ed25519 -f /root/.ssh/Gmer4Lfe-rsync-key -C "gmer4lfe-rsync" -N ""
+```

-On Secondary (unRAID-Jayred365):
+**On Secondary (unRAID-Jayred365):**

+```bash
 ssh-keygen -t ed25519 -f /root/.ssh/Jayred365-rsync-key -C "jayred365-rsync" -N ""
-Copy Public Keys to Each Server
+```

-Primary → Secondary
+### 2b — Copy Public Keys to Each Server

-# On primary
+The primary's public key must be authorised on the secondary, and vice versa.
+
+**Copy primary key → secondary:**
+
+```bash
+# Run on primary
 cat /root/.ssh/Gmer4Lfe-rsync-key.pub
+```

-Copy the output. Then on secondary:
+Copy the output. Then on the secondary:

+```bash
+# Run on secondary
 mkdir -p /root/.ssh
 echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
 chmod 600 /root/.ssh/authorized_keys
+```

-Secondary → Primary
+**Copy secondary key → primary:**

-# On secondary
+```bash
+# Run on secondary
 cat /root/.ssh/Jayred365-rsync-key.pub
+```

-Copy the output. Then on primary:
+Copy the output. Then on the primary:

+```bash
+# Run on primary
 mkdir -p /root/.ssh
 echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
 chmod 600 /root/.ssh/authorized_keys
-Test the Connection
+```

-From the primary, test SSH access to the secondary:
+### 2c — Test the Connection

+From the primary, test that it can SSH to the secondary without a password prompt:
+
+```bash
 ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@$(tailscale ip -4 unRAID-Jayred365) "echo connected"
+```

-You should see connected. If prompted for a password, recheck the key authorization.
+You should see `connected`. If prompted for a password the key was not authorised correctly — recheck Step 2b.

-Step 3 — Enable SSH on unRAID
+### 2d — Gitea SSH Key

-unRAID 7.x has SSH disabled by default. Enable it on both servers so the scripts can connect:
+Generate a separate key for Gitea access on each server:

-Go to Settings → Management Access
-Under Secure Shell, set SSH to Enabled
-Set SSH port to 22 (default)
-Click Apply
+```bash
+ssh-keygen -t ed25519 -f /root/.ssh/id_gitea_rsync -C "unraid-gitea" -N ""
+```

-Security note: SSH is only exposed on your local network and Tailscale interface. The rsync scripts connect via Tailscale IP, so traffic is encrypted end-to-end.
+Add the public key to your Gitea account:

-Step 4 — Configure Master.conf
+```bash
+cat /root/.ssh/id_gitea_rsync.pub
+```

-All user configuration lives in Master.conf. Open it and adjust the following to match your setup:
+Copy the output and add it in Gitea under **Settings → SSH / GPG Keys → Add Key**.

+---
+
+## Step 3 — Enable SSH on unRAID
+
+unRAID 7.x has SSH disabled by default. Enable it on both servers so the scripts can connect between them.
+
+1. Go to **Settings → Management Access**
+2. Under **Secure Shell** set **SSH** to `Enabled`
+3. Set **SSH port** to `22` (default)
+4. Click **Apply**
+
+> **Security note:** SSH is only exposed on your local network and Tailscale interface. The rsync scripts connect via the Tailscale IP so traffic is encrypted end-to-end.
+
+---
+
+## Step 4 — Clone the Git Repository
+
+The scripts live in a Gitea repository. Both servers clone from the same repo so updates propagate everywhere via a single git pull.
+
+### On Both Servers
+
+```bash
+# Create the target directory
+mkdir -p /mnt/user/appdata/unraid_scripts
+
+# Clone the repository
+GIT_SSH_COMMAND="ssh -i /root/.ssh/id_gitea_rsync" \
+    git clone git@YOUR_GITEA_HOST:YOUR_USER/Unraid_Scripts.git \
+    /mnt/user/appdata/unraid_scripts
+```
+
+Replace `YOUR_GITEA_HOST` and `YOUR_USER` with your Gitea server address and username.
+
+### Verify the Structure
+
+```bash
+ls /mnt/user/appdata/unraid_scripts
+```
+
+You should see:
+
+```
+Master.conf
+common.sh
+Rsync/
+    rsync.sh
+Orchestrators/
+    daily_sync.sh
+Tools/
+    recreate_shares.sh
+```
+
+### Make Scripts Executable
+
+```bash
+chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh
+chmod +x /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
+chmod +x /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh
+```
+
+---
+
+## Step 5 — Configure Master.conf
+
+All user configuration lives in `Master.conf`. Open it and adjust the following to match your setup:
+
+```bash
 nano /mnt/user/appdata/unraid_scripts/Master.conf
+```

-Required Changes:
+### Required Changes

-Variable	Description	Example
-HOST1	Hostname of your primary server	unRAID-Gmer4Lfe
-HOST2	Hostname of your secondary server	unRAID-Jayred365
-HOST1_SSH_KEY	Path to primary's rsync private key	/root/.ssh/Gmer4Lfe-rsync-key
-HOST2_SSH_KEY	Path to secondary's rsync private key	/root/.ssh/Jayred365-rsync-key
-BW_LIMIT	Global bandwidth limit in KB/s	12500
-ROOTFS_WARN	Remote rootfs % threshold before aborting	75
+| Variable | Description | Example |
+|---|---|---|
+| `HOST1` | Hostname of your primary server | `unRAID-Gmer4Lfe` |
+| `HOST2` | Hostname of your secondary server | `unRAID-Jayred365` |
+| `HOST1_SSH_KEY` | Path to primary's rsync private key | `/root/.ssh/Gmer4Lfe-rsync-key` |
+| `HOST2_SSH_KEY` | Path to secondary's rsync private key | `/root/.ssh/Jayred365-rsync-key` |
+| `REPO_SSH` | SSH URL of your Gitea repository | `git@192.168.50.2:User/Unraid_Scripts.git` |
+| `GITEA_SSH_KEY` | Path to Gitea private key | `/root/.ssh/id_gitea_rsync` |
+| `BW_LIMIT` | Global bandwidth limit in KB/s | `12500` |
+| `ROOTFS_WARN` | Remote rootfs % threshold before aborting | `75` |

-Daily Sync Shares
+### Daily Sync Shares

 Add the full paths of all media shares you want synced nightly:

+```bash
 DAILY_SYNC_SHARES=(
    /mnt/user/Movies
    /mnt/user/Tv_Shows
    /mnt/user/Music
    # add more here
 )
+```

-Profiles
+### Profiles

-Profiles control per-share rsync behaviour for your frequently synced appdata shares.
+Profiles control per-share rsync behaviour for your frequently synced appdata shares. Each profile is matched by the directory basename (lowercased) — or overridden with `--profile=name`.

+```bash
+# One array drives both local and remote container stops
+# Local stops first (flush databases) then remote stops (clean receive)
+# Same container names on both HOST1 and HOST2 — no duplication needed
+# Containers not found on a server are skipped gracefully
 declare -A PROFILE_CRITICAL_CONTAINER_NAMES=(
-    [arrs_stack]="Sonarr Radarr Lidarr Prowlarr"
+    [critical-data]="Mariadb-Authelia Mariadb-Authelia-Secondary Redis-Authelia Redis-Authelia-Secondary Lldap-Gmer4Lfe NginxProxyManager Authelia Authelia-Secondary"
    [important-data]="Postgres-NextCloud NextCloud"
+    [emby]="Emby"           # nightly clean sync — Emby stopped both sides
+    [emby-failover]=""      # dirty sync — Emby stays running
 )
+```

-Any share with no matching profile falls through to the global DEFAULT_RSYNC_OPTS.
+Any share with no matching profile falls through to the global `DEFAULT_RSYNC_OPTS`. Containers not found on a server are skipped gracefully — only containers that were actually running get restarted.

-Step 5 — Set Up User Scripts
+**Two Emby profiles:**

-The User Scripts plugin schedules and runs the scripts. Each sync job is its own script entry.
+```
+emby-failover — frequent dirty sync (every 30-60min):
+  Emby stays running on both sides
+  WAL and SHM excluded — safe while Emby is active
+  Only critical failover data: users.db, library.db, authentication.db, config/
+  Fast, small dataset, high bandwidth
+  This is also what gets written back during failover handback

-Frequent Sync Jobs
+emby — nightly clean sync (via nightly_sync.sh at 2:30am):
+  Emby stopped on both sides — WAL checkpointed on shutdown
+  Full mirror: metadata, plugins, config all included
+  Minimal excludes: logs, transcodes, cache, crash files only
+  Complete faithful state pushed once per night
+```

-Create one script entry per appdata profile (Plugins → User Scripts → Add New Script)
-Example for ARR stack:
+Usage with profile override:
+```bash
+# Dirty sync — Emby stays running
+bash rsync.sh /mnt/user/appdata-Failover/Emby --profile=emby-failover
+
+# Clean sync — called by nightly_sync.sh, Emby stopped via profile
+bash rsync.sh /mnt/user/appdata-Failover/Emby
+```
+
+---
+
+## Step 6 — Set Up User Scripts
+
+The User Scripts plugin is how unRAID schedules and runs the scripts. Each sync job is its own script entry in the plugin.
+
+### Frequent Sync Jobs (Scheduled Individually)
+
+Create one script entry per appdata profile. Go to **Plugins → User Scripts → Add New Script**.
+
+Name it descriptively — e.g. `rsync appdata arrs_stack`.
+
+In the script body paste:
+
+```bash
 #!/bin/bash
 bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
+```

-Daily Sync Orchestrator
+Set the schedule to match your desired frequency:

+| Profile | Schedule | Notes |
+|---|---|---|
+| `emby-failover` | Every 30-60 min | Dirty sync — Emby running, critical data only |
+| `emby` | via `nightly_sync.sh` | Clean sync — Emby stopped, full mirror |
+| `Critical-Data` | via `nightly_sync.sh` | Auth stack — clean nightly sync |
+| `Important-Data` | Every 6-12 hours | NextCloud file changes |
+| `Arrs_Stack` | Every 12-24 hours | Arr databases |
+| `Gmer4Lfe` | Daily or weekly | Personal appdata, rarely changes |
+
+> **Important:** Set each script to run as a **Background Task** — this ensures output streams correctly to the log rather than buffering in the browser.
+
+### Daily Sync Orchestrator
+
+Create one more script entry for the daily media sync:
+
+Name it `daily media sync`.
+
+In the script body paste:
+
+```bash
 #!/bin/bash
 bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
-Schedule daily at 01:00
-Set scripts to run as Background Tasks
-Step 6 — Verify the Setup
+```

-Manually test from the primary:
+Set the schedule to **Daily at 01:00**.

+---
+
+## Step 7 — Verify the Setup
+
+Before letting the scheduled jobs run, do a manual test from the terminal on the primary:
+
+```bash
 bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --log
+```

-You should see a healthy run with:
+A healthy run will show:

+```
 ━━━ ⚙️ Setup ━━━
 ℹ️  [INFO]  🖥️ Host: unRAID-Gmer4Lfe → unRAID-Jayred365
 ℹ️  [INFO]  🌐 Remote IP: 100.x.x.x
-...
+
+━━━ 🛡️ Pre-flight Checks ━━━
+ℹ️  [INFO]  📡 unRAID-Jayred365 is reachable
+ℹ️  [INFO]  🩺 Remote rootfs: 12% used (threshold: 75%)
+ℹ️  [INFO]  🩺 Remote share verified: ...
+ℹ️  [INFO]  💾 disk1 🟢 — share present
 ✅ [OK]  All disks backing share are online
-Step 7 — Secondary Server Initial Setup
+```

-If setting up secondary from scratch:
+If any pre-flight check fails the script will abort with a clear error and hint before touching anything.

-Complete Steps 1–5 on the secondary
-Start the array and create your shares in the unRAID UI
-Run the share recreation tool:
+---
+
+## Step 8 — Secondary Server Initial Setup
+
+If setting up the secondary from scratch (no existing data):
+
+1. Complete Steps 1–5 on the secondary
+2. Start the array and create your shares in the unRAID UI
+3. Run the share recreation tool to create disk directories from your cfg files:
+
+```bash
 bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh
-Temporarily remove --delete from DEFAULT_RSYNC_OPTS for initial push
-Restore --delete after the first run; next nightly run will clean .recovery files automatically
-Troubleshooting
+```

-SSH connection refused
+4. Temporarily remove `--delete` from `DEFAULT_RSYNC_OPTS` in `Master.conf`
+5. Run the initial push from the primary — the `.recovery` marker files allow rsync to populate empty shares without aborting
+6. Once complete, restore `--delete` to `Master.conf`
+7. The next nightly run will clean up the `.recovery` marker files automatically

-Verify SSH is enabled on target server (Step 3)
-Check correct key referenced in Master.conf
-Confirm Tailscale IP resolves
+---

-Pre-flight aborts
+## Troubleshooting

-Rootfs above ROOTFS_WARN
-Share exists but empty
-Disk offline
+### SSH connection refused
+- Verify SSH is enabled on the target server (Step 3)
+- Check the correct key is referenced in `Master.conf`
+- Confirm the Tailscale IP resolves: `tailscale ip -4 HOSTNAME`

-Script not found
+### Pre-flight aborts on rootfs
+- Remote rootfs is above `ROOTFS_WARN` threshold
+- Check if the remote array is started and drives are mounted
+- `df /` on the remote to see current usage

-Verify scripts exist in /mnt/user/appdata/unraid_scripts/
-Make scripts executable:
-chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh
-chmod +x /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
-chmod +x /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh
-Available Flags
-Flag	Description
--dry-run or -n	Run without making changes
--log	Enable verbose logging
--no-log	Disable logging
--status	Print resolved configuration and exit
+### Pre-flight aborts on empty share
+- Share exists but has no content — drives may not be mounted
+- Run `recreate_shares.sh` if setting up fresh
+- Check array status on the remote server

-Examples
+### Pre-flight aborts on disk check
+- One or more disks backing the share are not mounted
+- Check **Main → Array Devices** on the remote for offline disks
+- Verify disk assignments are correct after any hardware changes

-# Preview what would be synced without transferring
+### Script not found
+- Verify the repo was cloned to `/mnt/user/appdata/unraid_scripts/`
+- Check scripts are executable: `chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh`
+
+### Containers not stopping/starting
+- Verify container names in `Master.conf` match exactly what Docker shows
+- Check SSH key has access to run docker commands on the remote
+- Test manually: `ssh -i /root/.ssh/KEY root@REMOTE_IP "docker ps"`
+
+---
+
+## Available Flags
+
+All scripts support the following flags:
+
+| Flag | Description |
+|---|---|
+| `--dry-run` or `-n` | Run without making any changes |
+| `--log` | Enable verbose logging output |
+| `--no-log` | Disable logging (overrides Master.conf) |
+| `--status` | Print resolved configuration and exit |
+
+Example:
+
+```bash
+# Preview what would be synced without transferring anything
 bash rsync.sh /mnt/user/Movies --dry-run --log

-# Check profile and settings resolved for a share
+# Check what profile and settings resolved for a share
 bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status
-Repository Structure
+```
+
+---
+
+## Repository Structure
+
+```
 Unraid_Scripts/
 ├── Master.conf               # All user configuration — edit this file only
 ├── common.sh                 # Shared library — functions used by all scripts
@@ -218,4 +441,9 @@ Unraid_Scripts/
 ├── Orchestrators/
 │   └── daily_sync.sh         # Daily media sync orchestrator
 └── Tools/
-    └── recreate_shares.sh    # Share directory recreation from cfg fil
+    └── recreate_shares.sh    # Share directory recreation from cfg files
+```
+
+---
+
+*Guide version aligned with common.sh v1.6*