diff --git a/Rsync/README_Rsync_Setup.md b/Rsync/README_Rsync_Setup.md index 383d89e..75da868 100644 --- a/Rsync/README_Rsync_Setup.md +++ b/Rsync/README_Rsync_Setup.md @@ -2,197 +2,119 @@ This guide is a work in progress and generated by ai, dont have time for guides. Rsync Setup Guide - Status: Work in Progress - For the unRAID Rsync Ecosystem — common.sh · Master.conf · rsync.sh · daily_sync.sh - - +Status: Work in Progress +For the unRAID Rsync Ecosystem — common.sh · Master.conf · rsync.sh · daily_sync.sh Overview This guide walks through setting up the rsync ecosystem on both your primary and secondary unRAID 7.x servers. By the end you will have: - A Gitea repository cloned to both servers - SSH keys configured for server-to-server communication - Tailscale running on both servers for secure networking - Scripts scheduled and running via the User Scripts plugin - Automated daily sync of media shares and appdata profiles - +SSH keys configured for server-to-server communication +Tailscale running on both servers for secure networking +Scripts scheduled and running via the User Scripts plugin +Automated daily sync of media shares and appdata profiles Prerequisites Both servers need the following before starting: - unRAID 7.x - User Scripts plugin installed via Community Applications - Tailscale plugin installed via Community Applications - Access to a Gitea instance (self-hosted or remote) - Terminal access to both servers (via unRAID UI → Tools → Terminal, or SSH) - +unRAID 7.x +User Scripts plugin installed via Community Applications +Tailscale plugin installed via Community Applications +Terminal access to both servers (via unRAID UI → Tools → Terminal, or SSH) Step 1 — Tailscale Setup Tailscale provides the secure network tunnel between your two servers. The scripts resolve the remote server's IP via Tailscale at runtime. + On Both Servers - Open Apps in the unRAID UI - Search for Tailscale and install the plugin - Once installed go to Settings → Tailscale - Click Connect and authenticate with your Tailscale account - Verify both servers appear in your Tailscale admin console +Open Apps in the unRAID UI +Search for Tailscale and install the plugin +Once installed, go to Settings → Tailscale +Click Connect and authenticate with your Tailscale account +Verify both servers appear in your Tailscale admin console Verify Connectivity Run this on the primary to confirm it can see the secondary: -bash tailscale ip -4 unRAID-Jayred365 -You should get back a 100.x.x.x IP. If not, check that both machines are authenticated and connected in the Tailscale admin console. +You should get back a 100.x.x.x IP. If not, check both servers in the Tailscale admin console. - Note: The hostnames used in Master.conf (HOST1 and HOST2) must match the Tailscale machine names exactly — these are case sensitive. +Note: The hostnames used in Master.conf (HOST1 and HOST2) must match the Tailscale machine names exactly — these are case sensitive. -Step 2 — Generate SSH Keys +Step 2 — Generate SSH Keys (Server-to-Server) -The scripts use SSH keys for two purposes: - - Server-to-server rsync — primary authenticates to secondary (and vice versa) - Gitea access — both servers pull from the git repository - -2a — Server-to-Server Keys - -Run the following on each server to generate its rsync key. Replace the filename with the appropriate server name. +The scripts use SSH keys for server-to-server rsync. On Primary (unRAID-Gmer4Lfe): -bash ssh-keygen -t ed25519 -f /root/.ssh/Gmer4Lfe-rsync-key -C "gmer4lfe-rsync" -N "" On Secondary (unRAID-Jayred365): -bash ssh-keygen -t ed25519 -f /root/.ssh/Jayred365-rsync-key -C "jayred365-rsync" -N "" +Copy Public Keys to Each Server -2b — Copy Public Keys to Each Server +Primary → Secondary -The primary's public key must be authorised on the secondary, and vice versa. - -Copy primary key → secondary: -bash - -# Run on primary +# On primary cat /root/.ssh/Gmer4Lfe-rsync-key.pub -Copy the output. Then on the secondary: -bash +Copy the output. Then on secondary: -# Run on secondary mkdir -p /root/.ssh echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys chmod 600 /root/.ssh/authorized_keys -Copy secondary key → primary: -bash +Secondary → Primary -# Run on secondary +# On secondary cat /root/.ssh/Jayred365-rsync-key.pub -Copy the output. Then on the primary: -bash +Copy the output. Then on primary: -# Run on primary mkdir -p /root/.ssh echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys chmod 600 /root/.ssh/authorized_keys +Test the Connection -2c — Test the Connection - -From the primary, test that it can SSH to the secondary without a password prompt: -bash +From the primary, test SSH access to the secondary: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@$(tailscale ip -4 unRAID-Jayred365) "echo connected" -You should see connected. If prompted for a password the key was not authorised correctly — recheck Step 2b. -2d — Gitea SSH Key +You should see connected. If prompted for a password, recheck the key authorization. -Generate a separate key for Gitea access on each server: -bash - -ssh-keygen -t ed25519 -f /root/.ssh/id_gitea_rsync -C "unraid-gitea" -N "" - -Add the public key to your Gitea account: -bash - -cat /root/.ssh/id_gitea_rsync.pub - -Copy the output and add it in Gitea under Settings → SSH / GPG Keys → Add Key. Step 3 — Enable SSH on unRAID -unRAID 7.x has SSH disabled by default. Enable it on both servers so the scripts can connect between them. +unRAID 7.x has SSH disabled by default. Enable it on both servers so the scripts can connect: - Go to Settings → Management Access - Under Secure Shell set SSH to Enabled - Set SSH port to 22 (default) - Click Apply +Go to Settings → Management Access +Under Secure Shell, set SSH to Enabled +Set SSH port to 22 (default) +Click Apply - Security note: SSH is only exposed on your local network and Tailscale interface. The rsync scripts connect via the Tailscale IP so traffic is encrypted end-to-end. +Security note: SSH is only exposed on your local network and Tailscale interface. The rsync scripts connect via Tailscale IP, so traffic is encrypted end-to-end. -Step 4 — Clone the Git Repository - -The scripts live in a Gitea repository. Both servers clone from the same repo so updates propagate everywhere via a single git pull. -On Both Servers -bash - -# Create the target directory -mkdir -p /mnt/user/appdata/unraid_scripts - -# Clone the repository -GIT_SSH_COMMAND="ssh -i /root/.ssh/id_gitea_rsync" \ - git clone git@YOUR_GITEA_HOST:YOUR_USER/Unraid_Scripts.git \ - /mnt/user/appdata/unraid_scripts - -Replace YOUR_GITEA_HOST and YOUR_USER with your Gitea server address and username. -Verify the Structure -bash - -ls /mnt/user/appdata/unraid_scripts - -You should see: - -Master.conf -common.sh -Rsync/ - rsync.sh -Orchestrators/ - daily_sync.sh -Tools/ - recreate_shares.sh - -Make Scripts Executable -bash - -chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh -chmod +x /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh -chmod +x /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh - -Step 5 — Configure Master.conf +Step 4 — Configure Master.conf All user configuration lives in Master.conf. Open it and adjust the following to match your setup: -bash nano /mnt/user/appdata/unraid_scripts/Master.conf -Required Changes +Required Changes: + Variable Description Example HOST1 Hostname of your primary server unRAID-Gmer4Lfe HOST2 Hostname of your secondary server unRAID-Jayred365 HOST1_SSH_KEY Path to primary's rsync private key /root/.ssh/Gmer4Lfe-rsync-key HOST2_SSH_KEY Path to secondary's rsync private key /root/.ssh/Jayred365-rsync-key -REPO_SSH SSH URL of your Gitea repository git@192.168.50.2:User/Unraid_Scripts.git -GITEA_SSH_KEY Path to Gitea private key /root/.ssh/id_gitea_rsync BW_LIMIT Global bandwidth limit in KB/s 12500 ROOTFS_WARN Remote rootfs % threshold before aborting 75 + Daily Sync Shares Add the full paths of all media shares you want synced nightly: -bash DAILY_SYNC_SHARES=( /mnt/user/Movies @@ -203,8 +125,7 @@ DAILY_SYNC_SHARES=( Profiles -Profiles control per-share rsync behaviour for your frequently synced appdata shares. Each profile is matched by the directory basename (lowercased). Add a key to each profile array for any share that needs custom settings: -bash +Profiles control per-share rsync behaviour for your frequently synced appdata shares. declare -A PROFILE_CRITICAL_CONTAINER_NAMES=( [arrs_stack]="Sonarr Radarr Lidarr Prowlarr" @@ -212,137 +133,83 @@ declare -A PROFILE_CRITICAL_CONTAINER_NAMES=( ) Any share with no matching profile falls through to the global DEFAULT_RSYNC_OPTS. -Step 6 — Set Up User Scripts -The User Scripts plugin is how unRAID schedules and runs the scripts. Each sync job is its own script entry in the plugin. -Frequent Sync Jobs (Scheduled Individually) +Step 5 — Set Up User Scripts -Create one script entry per appdata profile. Go to Plugins → User Scripts → Add New Script. +The User Scripts plugin schedules and runs the scripts. Each sync job is its own script entry. -Name it descriptively — e.g. rsync appdata arrs_stack. - -In the script body paste: -bash +Frequent Sync Jobs +Create one script entry per appdata profile (Plugins → User Scripts → Add New Script) +Example for ARR stack: #!/bin/bash bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack -Set the schedule to match your desired frequency: -Profile Recommended Schedule -Emby Every 1 hour -Critical-Data Every 2 hours -Important-Data Every 6 hours -Arrs_Stack Every 12 hours - - Important: Set each script to run as a Background Task — this ensures output streams correctly to the log rather than buffering in the browser. - Daily Sync Orchestrator -Create one more script entry for the daily media sync: - -Name it daily media sync. - -In the script body paste: -bash - #!/bin/bash bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh +Schedule daily at 01:00 +Set scripts to run as Background Tasks +Step 6 — Verify the Setup -Set the schedule to Daily at 01:00. -Step 7 — Verify the Setup - -Before letting the scheduled jobs run, do a manual test from the terminal on the primary: -bash +Manually test from the primary: bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --log -A healthy run will show: +You should see a healthy run with: ━━━ ⚙️ Setup ━━━ ℹ️ [INFO] 🖥️ Host: unRAID-Gmer4Lfe → unRAID-Jayred365 ℹ️ [INFO] 🌐 Remote IP: 100.x.x.x - -━━━ 🛡️ Pre-flight Checks ━━━ -ℹ️ [INFO] 📡 unRAID-Jayred365 is reachable -ℹ️ [INFO] 🩺 Remote rootfs: 12% used (threshold: 75%) -ℹ️ [INFO] 🩺 Remote share verified: ... -ℹ️ [INFO] 💾 disk1 🟢 — share present +... ✅ [OK] All disks backing share are online +Step 7 — Secondary Server Initial Setup -If any pre-flight check fails the script will abort with a clear error and hint before touching anything. -Step 8 — Secondary Server Initial Setup - -If setting up the secondary from scratch (no existing data): - - Complete Steps 1–5 on the secondary - Start the array and create your shares in the unRAID UI - Run the share recreation tool to create disk directories from your cfg files: - -bash +If setting up secondary from scratch: +Complete Steps 1–5 on the secondary +Start the array and create your shares in the unRAID UI +Run the share recreation tool: bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh - - Temporarily remove --delete from DEFAULT_RSYNC_OPTS in Master.conf - Run the initial push from the primary — the .recovery marker files allow rsync to populate empty shares without aborting - Once complete, restore --delete to Master.conf - The next nightly run will clean up the .recovery marker files automatically - +Temporarily remove --delete from DEFAULT_RSYNC_OPTS for initial push +Restore --delete after the first run; next nightly run will clean .recovery files automatically Troubleshooting + SSH connection refused - Verify SSH is enabled on the target server (Step 3) - Check the correct key is referenced in Master.conf - Confirm the Tailscale IP resolves: tailscale ip -4 HOSTNAME +Verify SSH is enabled on target server (Step 3) +Check correct key referenced in Master.conf +Confirm Tailscale IP resolves -Pre-flight aborts on rootfs +Pre-flight aborts - Remote rootfs is above ROOTFS_WARN threshold - Check if the remote array is started and drives are mounted - df / on the remote to see current usage - -Pre-flight aborts on empty share - - Share exists but has no content — drives may not be mounted - Run recreate_shares.sh if setting up fresh - Check array status on the remote server - -Pre-flight aborts on disk check - - One or more disks backing the share are not mounted - Check Main → Array Devices on the remote for offline disks - Verify disk assignments are correct after any hardware changes +Rootfs above ROOTFS_WARN +Share exists but empty +Disk offline Script not found - Verify the repo was cloned to /mnt/user/appdata/unraid_scripts/ - Check scripts are executable: chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh - -Containers not stopping/starting - - Verify container names in Master.conf match exactly what Docker shows - Check SSH key has access to run docker commands on the remote - Test manually: ssh -i /root/.ssh/KEY root@REMOTE_IP "docker ps" - +Verify scripts exist in /mnt/user/appdata/unraid_scripts/ +Make scripts executable: +chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh +chmod +x /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh +chmod +x /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh Available Flags - -All scripts support the following flags: Flag Description ---dry-run or -n Run without making any changes ---log Enable verbose logging output ---no-log Disable logging (overrides Master.conf) +--dry-run or -n Run without making changes +--log Enable verbose logging +--no-log Disable logging --status Print resolved configuration and exit -Example: -bash +Examples -# Preview what would be synced without transferring anything +# Preview what would be synced without transferring bash rsync.sh /mnt/user/Movies --dry-run --log -# Check what profile and settings resolved for a share +# Check profile and settings resolved for a share bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status - Repository Structure - Unraid_Scripts/ ├── Master.conf # All user configuration — edit this file only ├── common.sh # Shared library — functions used by all scripts @@ -351,4 +218,4 @@ Unraid_Scripts/ ├── Orchestrators/ │ └── daily_sync.sh # Daily media sync orchestrator └── Tools/ - └── recreate_shares.sh # Share directory recreation from cfg files \ No newline at end of file + └── recreate_shares.sh # Share directory recreation from cfg fil \ No newline at end of file