diff --git a/Rsync/README_Rsync_Setup.md b/Rsync/README_Rsync_Setup.md index 28ca1b8..383d89e 100644 --- a/Rsync/README_Rsync_Setup.md +++ b/Rsync/README_Rsync_Setup.md @@ -1,23 +1,354 @@ -This is the Rsync Setup Guide - This guide is a work in progress +This guide is a work in progress and generated by ai, dont have time for guides. so this works + +Rsync Setup Guide + + Status: Work in Progress + For the unRAID Rsync Ecosystem — common.sh · Master.conf · rsync.sh · daily_sync.sh -First git Repository need installed at /mnt/user/appdata/unraid_scripts/ - Then made executable, copy and paste contents of git_pull_execute.sh script to a new script in unRAID User Scripts -in unRAID User Scripts, to use scripts in git - Create a new Script and name it whatever - You can copy contents of user_script_pluin.sh and paste then change acordingly - or - add a line pointing to script plus local folder to be synced - Exampl = /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-test/Arrs/ +Overview -This opens the script and passes the folder over as an argument - This allows for multiple rsync ops running off a single core script +This guide walks through setting up the rsync ecosystem on both your primary and secondary unRAID 7.x servers. By the end you will have: -All changes made to 1 single core script then works for all user scripts that point to rsync.sh + A Gitea repository cloned to both servers + SSH keys configured for server-to-server communication + Tailscale running on both servers for secure networking + Scripts scheduled and running via the User Scripts plugin + Automated daily sync of media shares and appdata profiles -The Master.conf file is where all user changeable variables live, This is for all scripts +Prerequisites -A common.sh file also is used to share all common codes that multiple scripts use - This allows for easier managment of scripts, editing, and what not \ No newline at end of file +Both servers need the following before starting: + + unRAID 7.x + User Scripts plugin installed via Community Applications + Tailscale plugin installed via Community Applications + Access to a Gitea instance (self-hosted or remote) + Terminal access to both servers (via unRAID UI → Tools → Terminal, or SSH) + +Step 1 — Tailscale Setup + +Tailscale provides the secure network tunnel between your two servers. The scripts resolve the remote server's IP via Tailscale at runtime. +On Both Servers + + Open Apps in the unRAID UI + Search for Tailscale and install the plugin + Once installed go to Settings → Tailscale + Click Connect and authenticate with your Tailscale account + Verify both servers appear in your Tailscale admin console + +Verify Connectivity + +Run this on the primary to confirm it can see the secondary: +bash + +tailscale ip -4 unRAID-Jayred365 + +You should get back a 100.x.x.x IP. If not, check that both machines are authenticated and connected in the Tailscale admin console. + + Note: The hostnames used in Master.conf (HOST1 and HOST2) must match the Tailscale machine names exactly — these are case sensitive. + +Step 2 — Generate SSH Keys + +The scripts use SSH keys for two purposes: + + Server-to-server rsync — primary authenticates to secondary (and vice versa) + Gitea access — both servers pull from the git repository + +2a — Server-to-Server Keys + +Run the following on each server to generate its rsync key. Replace the filename with the appropriate server name. + +On Primary (unRAID-Gmer4Lfe): +bash + +ssh-keygen -t ed25519 -f /root/.ssh/Gmer4Lfe-rsync-key -C "gmer4lfe-rsync" -N "" + +On Secondary (unRAID-Jayred365): +bash + +ssh-keygen -t ed25519 -f /root/.ssh/Jayred365-rsync-key -C "jayred365-rsync" -N "" + +2b — Copy Public Keys to Each Server + +The primary's public key must be authorised on the secondary, and vice versa. + +Copy primary key → secondary: +bash + +# Run on primary +cat /root/.ssh/Gmer4Lfe-rsync-key.pub + +Copy the output. Then on the secondary: +bash + +# Run on secondary +mkdir -p /root/.ssh +echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys +chmod 600 /root/.ssh/authorized_keys + +Copy secondary key → primary: +bash + +# Run on secondary +cat /root/.ssh/Jayred365-rsync-key.pub + +Copy the output. Then on the primary: +bash + +# Run on primary +mkdir -p /root/.ssh +echo "PASTE_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys +chmod 600 /root/.ssh/authorized_keys + +2c — Test the Connection + +From the primary, test that it can SSH to the secondary without a password prompt: +bash + +ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@$(tailscale ip -4 unRAID-Jayred365) "echo connected" + +You should see connected. If prompted for a password the key was not authorised correctly — recheck Step 2b. +2d — Gitea SSH Key + +Generate a separate key for Gitea access on each server: +bash + +ssh-keygen -t ed25519 -f /root/.ssh/id_gitea_rsync -C "unraid-gitea" -N "" + +Add the public key to your Gitea account: +bash + +cat /root/.ssh/id_gitea_rsync.pub + +Copy the output and add it in Gitea under Settings → SSH / GPG Keys → Add Key. +Step 3 — Enable SSH on unRAID + +unRAID 7.x has SSH disabled by default. Enable it on both servers so the scripts can connect between them. + + Go to Settings → Management Access + Under Secure Shell set SSH to Enabled + Set SSH port to 22 (default) + Click Apply + + Security note: SSH is only exposed on your local network and Tailscale interface. The rsync scripts connect via the Tailscale IP so traffic is encrypted end-to-end. + +Step 4 — Clone the Git Repository + +The scripts live in a Gitea repository. Both servers clone from the same repo so updates propagate everywhere via a single git pull. +On Both Servers +bash + +# Create the target directory +mkdir -p /mnt/user/appdata/unraid_scripts + +# Clone the repository +GIT_SSH_COMMAND="ssh -i /root/.ssh/id_gitea_rsync" \ + git clone git@YOUR_GITEA_HOST:YOUR_USER/Unraid_Scripts.git \ + /mnt/user/appdata/unraid_scripts + +Replace YOUR_GITEA_HOST and YOUR_USER with your Gitea server address and username. +Verify the Structure +bash + +ls /mnt/user/appdata/unraid_scripts + +You should see: + +Master.conf +common.sh +Rsync/ + rsync.sh +Orchestrators/ + daily_sync.sh +Tools/ + recreate_shares.sh + +Make Scripts Executable +bash + +chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh +chmod +x /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh +chmod +x /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh + +Step 5 — Configure Master.conf + +All user configuration lives in Master.conf. Open it and adjust the following to match your setup: +bash + +nano /mnt/user/appdata/unraid_scripts/Master.conf + +Required Changes +Variable Description Example +HOST1 Hostname of your primary server unRAID-Gmer4Lfe +HOST2 Hostname of your secondary server unRAID-Jayred365 +HOST1_SSH_KEY Path to primary's rsync private key /root/.ssh/Gmer4Lfe-rsync-key +HOST2_SSH_KEY Path to secondary's rsync private key /root/.ssh/Jayred365-rsync-key +REPO_SSH SSH URL of your Gitea repository git@192.168.50.2:User/Unraid_Scripts.git +GITEA_SSH_KEY Path to Gitea private key /root/.ssh/id_gitea_rsync +BW_LIMIT Global bandwidth limit in KB/s 12500 +ROOTFS_WARN Remote rootfs % threshold before aborting 75 +Daily Sync Shares + +Add the full paths of all media shares you want synced nightly: +bash + +DAILY_SYNC_SHARES=( + /mnt/user/Movies + /mnt/user/Tv_Shows + /mnt/user/Music + # add more here +) + +Profiles + +Profiles control per-share rsync behaviour for your frequently synced appdata shares. Each profile is matched by the directory basename (lowercased). Add a key to each profile array for any share that needs custom settings: +bash + +declare -A PROFILE_CRITICAL_CONTAINER_NAMES=( + [arrs_stack]="Sonarr Radarr Lidarr Prowlarr" + [important-data]="Postgres-NextCloud NextCloud" +) + +Any share with no matching profile falls through to the global DEFAULT_RSYNC_OPTS. +Step 6 — Set Up User Scripts + +The User Scripts plugin is how unRAID schedules and runs the scripts. Each sync job is its own script entry in the plugin. +Frequent Sync Jobs (Scheduled Individually) + +Create one script entry per appdata profile. Go to Plugins → User Scripts → Add New Script. + +Name it descriptively — e.g. rsync appdata arrs_stack. + +In the script body paste: +bash + +#!/bin/bash +bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack + +Set the schedule to match your desired frequency: +Profile Recommended Schedule +Emby Every 1 hour +Critical-Data Every 2 hours +Important-Data Every 6 hours +Arrs_Stack Every 12 hours + + Important: Set each script to run as a Background Task — this ensures output streams correctly to the log rather than buffering in the browser. + +Daily Sync Orchestrator + +Create one more script entry for the daily media sync: + +Name it daily media sync. + +In the script body paste: +bash + +#!/bin/bash +bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh + +Set the schedule to Daily at 01:00. +Step 7 — Verify the Setup + +Before letting the scheduled jobs run, do a manual test from the terminal on the primary: +bash + +bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --log + +A healthy run will show: + +━━━ ⚙️ Setup ━━━ +ℹ️ [INFO] 🖥️ Host: unRAID-Gmer4Lfe → unRAID-Jayred365 +ℹ️ [INFO] 🌐 Remote IP: 100.x.x.x + +━━━ 🛡️ Pre-flight Checks ━━━ +ℹ️ [INFO] 📡 unRAID-Jayred365 is reachable +ℹ️ [INFO] 🩺 Remote rootfs: 12% used (threshold: 75%) +ℹ️ [INFO] 🩺 Remote share verified: ... +ℹ️ [INFO] 💾 disk1 🟢 — share present +✅ [OK] All disks backing share are online + +If any pre-flight check fails the script will abort with a clear error and hint before touching anything. +Step 8 — Secondary Server Initial Setup + +If setting up the secondary from scratch (no existing data): + + Complete Steps 1–5 on the secondary + Start the array and create your shares in the unRAID UI + Run the share recreation tool to create disk directories from your cfg files: + +bash + +bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh + + Temporarily remove --delete from DEFAULT_RSYNC_OPTS in Master.conf + Run the initial push from the primary — the .recovery marker files allow rsync to populate empty shares without aborting + Once complete, restore --delete to Master.conf + The next nightly run will clean up the .recovery marker files automatically + +Troubleshooting +SSH connection refused + + Verify SSH is enabled on the target server (Step 3) + Check the correct key is referenced in Master.conf + Confirm the Tailscale IP resolves: tailscale ip -4 HOSTNAME + +Pre-flight aborts on rootfs + + Remote rootfs is above ROOTFS_WARN threshold + Check if the remote array is started and drives are mounted + df / on the remote to see current usage + +Pre-flight aborts on empty share + + Share exists but has no content — drives may not be mounted + Run recreate_shares.sh if setting up fresh + Check array status on the remote server + +Pre-flight aborts on disk check + + One or more disks backing the share are not mounted + Check Main → Array Devices on the remote for offline disks + Verify disk assignments are correct after any hardware changes + +Script not found + + Verify the repo was cloned to /mnt/user/appdata/unraid_scripts/ + Check scripts are executable: chmod +x /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh + +Containers not stopping/starting + + Verify container names in Master.conf match exactly what Docker shows + Check SSH key has access to run docker commands on the remote + Test manually: ssh -i /root/.ssh/KEY root@REMOTE_IP "docker ps" + +Available Flags + +All scripts support the following flags: +Flag Description +--dry-run or -n Run without making any changes +--log Enable verbose logging output +--no-log Disable logging (overrides Master.conf) +--status Print resolved configuration and exit + +Example: +bash + +# Preview what would be synced without transferring anything +bash rsync.sh /mnt/user/Movies --dry-run --log + +# Check what profile and settings resolved for a share +bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status + +Repository Structure + +Unraid_Scripts/ +├── Master.conf # All user configuration — edit this file only +├── common.sh # Shared library — functions used by all scripts +├── Rsync/ +│ └── rsync.sh # Core rsync script — called per share +├── Orchestrators/ +│ └── daily_sync.sh # Daily media sync orchestrator +└── Tools/ + └── recreate_shares.sh # Share directory recreation from cfg files \ No newline at end of file