# ━━━━━ RSYNC — Manual ━━━━━ Config reference, procedures, operational workflows. For overview see README-Rsync.md. For per-script detail see the rsync.sh header. This document also serves as the **initial setup guide** for standing up the two-server ecosystem from scratch. --- ## ━━━ WHAT YOU'RE BUILDING ━━━ ``` HOST1 (unRAID-Gmer4Lfe) HOST2 (unRAID-Jayred365) ────────────────────── ────────────────────── Downloads to: Downloads to: /mnt/user/Movies /mnt/user/Anime_Movies /mnt/user/Tv_Shows /mnt/user/Anime_Shows /mnt/user/Music Both servers' arrs track ALL content across both servers. It does not matter who downloaded what or when. Daily 3-phase window (1am): Phase 1 — arr_sync.sh (runs first): All arrs on all nodes reconcile libraries using external IDs (TMDB, TVDB, MusicBrainz). Union model — any node that tracks an item, all nodes get it. After this phase both servers' arrs know about all content regardless of who downloaded it. Phase 2 — rsync (bidirectional, no --delete on media shares): HOST1 pushes its shares ──────► HOST2 receives files HOST1 receives files ◄────── HOST2 pushes its shares Files arrive already tracked by the remote arr (arr_sync ran first). No --delete — media files only spread. Arr cleanup handles deletions. Phase 3 — arr cleanup: sonarr_cleanup / radarr_cleanup / lidarr_cleanup query the live arr API and remove any files no longer tracked. Emby notified after. Note: profiled syncs (arrs_stack, emby, critical-data…) use their own PROFILE_RSYNC_OPTS which include --delete. Only the no-profile media shares use DEFAULT_RSYNC_OPTS (spread only). Weekly clean sync (2:30am Sunday, containers stopped): Emby — full clean mirror ◄──────► Emby Critical-Data ──────► Auth stack (HOST1 → HOST2) Every 30 minutes — dirty sync: Emby watch states ──────► HOST2 stays current on playback ``` --- ## ━━━ PREREQUISITES ━━━ Required on both servers before starting: ``` unRAID 7.x Community Applications plugin — search "Community Applications" in unRAID plugins User Scripts plugin — install via Community Applications Tailscale plugin — install via Community Applications Terminal access — unRAID UI → Tools → Terminal, or SSH ``` Optional but recommended: ``` Gitea (Docker container on HOST1) — self-hosted git for the script repository Working Emby installation — for transcode management and failover ``` --- ## ━━━ STEP 1 — TAILSCALE ━━━ Tailscale provides the encrypted mesh network between servers. Scripts resolve the remote server's IP via Tailscale at runtime — no hardcoded IPs, no VPN configuration, no open ports. All server-to-server communication goes through Tailscale. ### Install on Both Servers ``` 1. Open Apps in the unRAID UI 2. Search "Tailscale" — install the plugin 3. Settings → Tailscale → Connect 4. Authenticate with your Tailscale account (browser opens on your machine) 5. Verify both servers appear: https://login.tailscale.com/admin/machines ``` ### Verify Connectivity ```bash # From HOST1 — should return HOST2's 100.x.x.x Tailscale IP: tailscale ip -4 unRAID-Jayred365 # Test actual connectivity: tailscale ping unRAID-Jayred365 ``` > **Critical:** The hostnames in `master.conf` (`HOST1` and `HOST2`) must match the > Tailscale machine names **exactly** — case sensitive. All remote IP resolution goes > through `tailscale ip -4 HOSTNAME`. A name mismatch causes every remote operation to > fail at the IP resolution step. --- ## ━━━ STEP 2 — ENABLE SSH ━━━ unRAID 7.x has SSH disabled by default. Enable it on both servers. ``` Settings → Management Access → Secure Shell SSH: Enabled SSH port: 22 Apply ``` SSH is only exposed on your local network and Tailscale interface. No ports are opened to the public internet. --- ## ━━━ STEP 3 — SSH KEYS ━━━ Two sets of keys needed: rsync automation keys (server-to-server) and a Gitea access key (for script repository pulls). ### 3a — Rsync Automation Keys (ssh_setup.sh) Run on **each server**. `ssh_setup.sh` generates the key, copies it to the remote, and updates `host*.conf` with the key path automatically. ```bash # On HOST1 — after the repo is cloned: bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh # On HOST2: bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh ``` Key naming convention: hostname lowercased, `unraid-` prefix stripped. ``` unRAID-Gmer4Lfe → /root/.ssh/gmer4lfe_rsync_automation unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation ``` `host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the generated key. Run `--status` to verify: ```bash bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --status ``` > **`ssh_setup.sh` is idempotent** — safe to re-run. Use `--force` to regenerate > a key (e.g., after a security incident) and re-copy it to the remote. ### 3b — Gitea SSH Key (manual) ```bash # On BOTH servers: ssh-keygen -t ed25519 -f /root/.ssh/unraid_gitea -C "unraid-gitea" -N "" # Print the public key to add to Gitea: cat /root/.ssh/unraid_gitea.pub # In Gitea: Settings → SSH / GPG Keys → Add Key → paste above ``` --- ## ━━━ STEP 4 — CLONE THE REPOSITORY ━━━ ```bash # Create target directory: mkdir -p /mnt/user/appdata/unraid_scripts # Clone: GIT_SSH_COMMAND="ssh -i /root/.ssh/unraid_gitea" \ git clone git@YOUR_GITEA_HOST:FailedProxy/Unraid_Scripts.git \ /mnt/user/appdata/unraid_scripts # Make scripts executable: find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \; ``` Expected structure after clone: ``` master.conf ← all shared configuration host1.conf ← HOST1-specific configuration host2.conf ← HOST2-specific configuration common.sh ← shared library load_config.sh ← config loader Orchestrators/ Rsync/ Fallback/ Docker_Essentials/ ... ``` --- ## ━━━ STEP 5 — CONFIGURE MASTER.CONF ━━━ ```bash nano /mnt/user/appdata/unraid_scripts/master.conf ``` ### Host Identity ```bash # These must match Tailscale machine names exactly — case sensitive. HOST1="unRAID-Gmer4Lfe" # REQUIRED HOST2="unRAID-Jayred365" # REQUIRED # SSH keys — each server uses its own key to authenticate to the other: HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key" HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key" ``` ### Git Repository ```bash GITEA_CONTAINER="Gitea" GITEA_REPO_PATH="FailedProxy/Unraid_Scripts.git" TARGET_DIR="/mnt/user/appdata/unraid_scripts" GITEA_SSH_KEY="/root/.ssh/unraid_gitea" SSH_PORT=221 ``` ### Daily Sync Shares ```bash # host1.conf # Shares this server downloads to — pushed to the other server nightly. # arr_sync.sh runs before rsync in the daily window, so the receiving # server's arrs already track incoming files when they arrive. # Never put the same path in both lists. HOST1_DAILY_SYNC_SHARES=( "/mnt/user/Movies" "/mnt/user/Tv_Shows" "/mnt/user/Music" "/mnt/user/Kids_Movies" "/mnt/user/Kids_Tv_Shows" "/mnt/user/Sports" "/mnt/user/stand-up_comedy" ) # host2.conf HOST2_DAILY_SYNC_SHARES=( "/mnt/user/Anime_Shows" "/mnt/user/Anime_Movies" ) ``` ### Weekly Sync Shares ```bash # master.conf # Synced during the Sunday 2:30am window — containers stopped both sides. WEEKLY_SYNC_SHARES=( "/mnt/user/Media_Server/Emby" "/mnt/user/appdata-Failover/Critical-Data" ) ``` --- ## ━━━ STEP 6 — MASTER_HOST*.CONF ━━━ `detect_hosts()` reads which server is running and aliases `HOST*_` prefixed vars to their unprefixed names. Scripts only ever reference the unprefixed name — they work identically on both servers. ```bash nano /mnt/user/appdata/unraid_scripts/host1.conf # on HOST1 nano /mnt/user/appdata/unraid_scripts/host2.conf # on HOST2 ``` Every variable is documented in the conf files. Key values to set: - SSH key paths - DAILY_SYNC_SHARES - EMBY_URL / EMBY_API_KEY - CERT_MONITOR_DOMAINS - SMART_IGNORE_DRIVES - ZFS_REPORT_IGNORE_POOLS --- ## ━━━ RSYNC PROFILES ━━━ Profiles control per-share behavior. Profile key = directory basename lowercased. Override with `--profile=name`. ### Profile Matching ```bash # rsync.sh /mnt/user/appdata-Failover/Arrs_Stack # basename: Arrs_Stack → lowercased: arrs_stack → matches [arrs_stack] profile # # rsync.sh /mnt/user/Movies # basename: Movies → no profile match → global defaults (no containers stopped) # # rsync.sh /mnt/user/appdata-Failover/Critical-Data --profile=critical-data # explicit override ``` ### Current Profile Definitions ```bash # master.conf # ── arrs_stack ─────────────────────────────────────────────────────────────── # Arr databases — stopped for clean SQLite snapshot PROFILES["arrs_stack_CRITICAL_CONTAINER_NAMES"]=( "Sonarr" "Radarr" "Lidarr" "Prowlarr" "Bazarr" "Pinchflat" ) # ── critical-data ───────────────────────────────────────────────────────────── # Auth stack — stopped for clean database snapshot, Authelia delayed restart PROFILES["critical-data_CRITICAL_CONTAINER_NAMES"]=( "Mariadb-Authelia" "Redis-Authelia" "NginxProxyManager" "Lldap-Gmer4Lfe" ) PROFILES["critical-data_DELAYED_CONTAINERS"]=( "Authelia" "Authelia-Secondary" ) PROFILES["critical-data_CONTAINER_DELAY"]=30 # ── important-data ──────────────────────────────────────────────────────────── # NextCloud + Postgres — stopped for clean snapshot PROFILES["important-data_CRITICAL_CONTAINER_NAMES"]=( "Postgres-NextCloud" ) PROFILES["important-data_DELAYED_CONTAINERS"]=("NextCloud") # ── emby ────────────────────────────────────────────────────────────────────── # Weekly full clean sync — both Emby instances stopped PROFILES["emby_CRITICAL_CONTAINER_NAMES"]=("Emby") PROFILES["emby_EXCLUDE_DIRS"]=( "transcodes/" "logs/" "crash*" "cache/" ) # ── emby-failover ───────────────────────────────────────────────────────────── # Every 30 minutes, Emby STAYS RUNNING — dirty sync of critical state only PROFILES["emby-failover_CRITICAL_CONTAINER_NAMES"]=() # nothing stops PROFILES["emby-failover_EXCLUDE_DIRS"]=( "*.wal" "*.shm" # unsafe mid-write "transcodes/" "logs/" "crash*" "cache/" ) PROFILES["emby-failover_REMOTE_RESTART_CONTAINERS"]=("Emby") ``` ### Two Emby Profiles — Why Both Exist **emby-failover** (every 30 minutes, Emby stays running): - Syncs: users.db, library.db, authentication.db, config/ - Skips: \*.wal, \*.shm, transcodes/, logs/, cache/ - Why: WAL files are written while Emby runs — copying them would produce a corrupt database on HOST2 - Result: HOST2 is always within 30 minutes of HOST1 on watch state and user activity **emby** (Sunday 2:30am, both Emby instances stopped): - Syncs: everything except transcodes, logs, cache, crash files - Includes: metadata, plugins, full database state, all config - Why: WAL is checkpointed on clean shutdown — safe to copy everything - Result: HOST2 gets a gold-standard Emby state once per week The two profiles work together. emby-failover keeps HOST2 current for immediate failover. emby gives HOST2 full fidelity once per week. Neither alone is sufficient. --- ## ━━━ PERSONAL ENCRYPTED SHARES ━━━ Personal shares sync to the remote for offsite backup. ZFS encrypts at the dataset level — the remote receives encrypted blocks and cannot read the content without your passphrase or keyfile. ### Create an Encrypted ZFS Dataset ```bash # In the unRAID UI: # Main → click your ZFS pool name → + Dataset # Name: Gmer4Lfe-Personal # Encryption: Enabled # Passphrase: [your passphrase] # Write your passphrase down — if lost, data is completely unrecoverable # Verify encryption is active before syncing: zfs get encryption poolname/Gmer4Lfe-Personal # Should show: encryption aes-256-gcm ``` ### Auto-Unlock on Boot (Optional) ```bash # Keyfile approach — more convenient, but the keyfile itself is a secret dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key chmod 600 /root/.zfs-keys/personal.key zfs change-key \ -o keylocation=file:///root/.zfs-keys/personal.key \ -o keyformat=raw \ poolname/Gmer4Lfe-Personal # Add to array_started.sh or ramdisk_setup.sh: zfs load-key poolname/Gmer4Lfe-Personal zfs mount poolname/Gmer4Lfe-Personal # Manual unlock alternative (most secure): zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase zfs mount poolname/Gmer4Lfe-Personal ``` ### Add to host1.conf ```bash HOST1_PERSONAL_SHARES=( "/mnt/user/Gmer4Lfe-Personal" ) ``` --- ## ━━━ STEP 7 — USER SCRIPTS SETUP ━━━ Only a small number of User Scripts entries are needed — each one an orchestrator. Individual scripts are never scheduled directly except the 30-minute Emby sync. ### At Startup of Array ```bash #!/bin/bash bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_started.sh # Schedule: At Startup of Array # Run as: Background Task ``` This single entry launches everything defined in ARRAY_START_SCRIPTS from master.conf: inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup, docker_network_connect, fallback. Watchdogs (resource, docker, system, stability) run separately via watchdog_orchestrator. ### Cron Schedule ```bash # Every 30 minutes — Emby dirty sync: */30 * * * * bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \ /mnt/user/Media_Server/Emby --profile=emby-failover # Every 6 hours — failed import + stalled download recovery: 0 */6 * * * bash /mnt/user/appdata/unraid_scripts/Orchestrators/arrs_failed_stalled_recovery.sh # 1am daily — full maintenance window: 0 1 * * * bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh # 2:30am Sunday — weekly maintenance window: 30 2 * * 0 bash /mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh # 8am daily — health digest: 0 8 * * * bash /mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh # Every 6 hours — inotify + php-fpm snapshot: 0 */6 * * * bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh # Sunday morning — weekly reports: 0 6 * * 0 bash .../Monitors/zfs_memory_snapshot.sh 0 7 * * 0 bash .../Monitors/smart_health.sh 0 9 * * 0 bash .../Monitors/cert_monitor.sh 0 10 * * 0 bash .../Monitors/backup_verify.sh 0 11 * * 0 bash .../Monitors/emby_session_report.sh 0 11 * * 0 bash .../Monitors/bandwidth_monitor.sh --report ``` Set all entries to **Background Task** — non-background tasks can appear to hang on long-running scripts. --- ## ━━━ VERIFICATION ━━━ Test with `--dry-run` first — all pre-flight checks run, no changes made. ### Test a Single Profile Sync ```bash bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \ /mnt/user/appdata-Failover/Arrs_Stack --dry-run --log ``` Expected output (healthy): ``` ━━━ Setup ━━━ Host: HOST1 (unRAID-Gmer4Lfe) → HOST2 (unRAID-Jayred365) Remote IP: 100.x.x.x Profile: arrs_stack ━━━ Pre-flight ━━━ Remote reachable version parity — both on unRAID X.Y.Z Remote Docker daemon responding Remote rootfs: 12% (threshold: 75%) Remote share exists and not empty All pre-flight checks passed ``` ### Test the Daily Orchestrator ```bash bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --dry-run ``` ### Check Configuration Resolution ```bash bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \ /mnt/user/appdata-Failover/Arrs_Stack --status ``` --- ## ━━━ INITIAL HOST2 SYNC ━━━ If HOST2 is being set up from scratch with empty shares: ```bash # Create share directories on HOST2: bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh # Initial push from HOST1 — first run may take several hours for large libraries: bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --log ``` The scheduled nightly sync will be incremental after the initial push. --- ## ━━━ NAMING CONSISTENCY REQUIREMENT ━━━ The ecosystem uses one codebase on both servers. This only works if containers and shares have identical names on both servers. **This is not configurable — it is a design requirement.** ``` Container names must match exactly on both servers: "Emby" ← both HOST1 and HOST2 "NginxProxyManager" ← both HOST1 and HOST2 "Mariadb-Authelia" ← both HOST1 and HOST2 Share paths must match exactly on both servers: /mnt/user/Movies ← both HOST1 and HOST2 /mnt/user/Tv_Shows ← both HOST1 and HOST2 ``` If a container has a different name on one server: the script skips it silently. You only notice when the container is not stopped during a sync that requires it. If a share has a different path: rsync.sh aborts with "remote share missing." Easier to catch — but still requires renaming the share to fix. --- ## ━━━ TROUBLESHOOTING ━━━ ### SSH Connection Refused / Timeout ```bash # Verify SSH is enabled on the remote: # Settings → Management Access → Secure Shell → Enabled # Verify Tailscale is connected: tailscale ip -4 unRAID-Jayred365 # Test SSH manually (key path from --status output): ssh -i /root/.ssh/gmer4lfe_rsync_automation \ root@$(tailscale ip -4 unRAID-Jayred365) "hostname" # If password prompted: key not authorised — re-run ssh_setup.sh # Re-run setup (idempotent, re-copies key to remote): bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh ``` ### Pre-flight Aborts on Remote Rootfs ```bash # Check current usage on remote: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "df /" # Common cause: array not started, drives not mounted ``` ### Remote Share Missing ```bash # Verify the share exists on HOST2: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "ls /mnt/user/" # If missing: create the share on HOST2, then run initial sync ``` ### Containers Not Stopping / Starting ```bash # Verify container names match Docker exactly — case sensitive: docker ps --format "{{.Names}}" # Test remote: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Names}}'" ``` ### Profile Not Matching ```bash # Profile key = directory basename lowercased # /mnt/user/appdata-Failover/Arrs_Stack → key: arrs_stack # Override explicitly: bash rsync.sh /mnt/user/appdata-Failover/My_Stuff --profile=arrs_stack # Verify what profile resolved: bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status ``` --- ## ━━━ FULL CONFIGURATION REFERENCE ━━━ ### master.conf ```bash # Rsync engine RSYNC_ENABLED=true DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace" # No --delete in DEFAULT_RSYNC_OPTS — media share rsync is additive only. # arr_sync.sh keeps all arr databases in union — either server can download anything. # arr cleanup handles deletions (orphans only). Profiles use --delete explicitly. BW_LIMIT=0 # KB/s, 0 = unlimited RETRY_COUNT=3 SLEEP=60 # seconds between retries ROOTFS_WARN_PCT=75 # abort if remote rootfs above this % # Shared with Monitors/ BANDWIDTH_LOG="/boot/config/bandwidth_history.db" BANDWIDTH_LOG_RETENTION=90 BANDWIDTH_WARN_GB=50 # Profile definitions (see RSYNC PROFILES section above) declare -A PROFILES declare -A PROFILE_BW_LIMIT declare -A PROFILE_RETRY_COUNT declare -A PROFILE_SLEEP declare -A PROFILE_CONTAINER_DELAY declare -A PROFILE_CRITICAL_CONTAINER_NAMES declare -A PROFILE_DELAYED_CONTAINERS declare -A PROFILE_EXCLUDE_DIRS declare -A PROFILE_REMOTE_RESTART_CONTAINERS ``` ### host*.conf ```bash # Daily sync shares — one list per server (mutually exclusive) HOST1_DAILY_SYNC_SHARES=(...) HOST2_DAILY_SYNC_SHARES=(...) # Personal encrypted shares HOST1_PERSONAL_SHARES=(...) # SSH key for this server to authenticate to the remote # Set automatically by ssh_setup.sh — do not edit manually HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation" HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation" ``` --- ## ━━━ OUTPUT TIERS ━━━ Without `--log`, each run shows: section headers (Pre-flight, Transfer, Stop/Start Containers), the transfer identity block (source, remote, profile, identity), a progress indicator while rsync is running, and a summary block with duration, bytes transferred, and status. Warnings and errors are always visible. With `--log`, every decision is shown — pre-flight check results, profile resolution, per-container stop/start state, retry details, and bandwidth log confirmation. Use for first-run validation or when investigating unexpected behavior. --- ## ━━━ FLAG REFERENCE ━━━ All flags work on `rsync.sh` and all orchestrators. ### --dry-run Runs all pre-flight checks. Shows what rsync would transfer. No transfer, no container stops, no bandwidth log entry. Safe to run at any time. ```bash rsync.sh /mnt/user/Movies --dry-run rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --dry-run --log daily_sync_maintenance.sh --dry-run ``` ### --status Shows resolved configuration — profile, remote identity, all vars that would be used — then exits. No pre-flight checks, no rsync. Use to verify configuration loaded correctly. ```bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status ``` ### --log Verbose output throughout. Every decision, every container operation, every rsync progress line. Use for first-time runs or when investigating issues. ### --profile=name Override profile selection. Bypasses basename inference. Use when the directory name doesn't match any profile key, or when testing a specific profile. ```bash rsync.sh /mnt/user/appdata-Failover/Critical-Data --profile=critical-data rsync.sh /mnt/user/Media_Server/Emby --profile=emby-failover ```