# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 🔄 RSYNC SETUP GUIDE # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ **Complete setup guide for the two-server rsync ecosystem.** By the end of this guide both servers will have SSH keys configured, Tailscale connected, the git repository cloned, and all scheduled operations running automatically. > **This is a setup guide, not a script reference.** For how rsync.sh works internally, > profiles, safety checks, and operational details — those belong in the orchestrator > and rsync script documentation. This guide is about standing the ecosystem up from > scratch and verifying it works. --- ## ━━━ WHAT YOU'RE BUILDING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ``` HOST1 (unRAID-Gmer4Lfe) HOST2 (unRAID-Jayred365) ────────────────────── ────────────────────── Source of truth: Source of truth: Movies, Tv_Shows, Music Anime_Shows, Anime_Movies Critical-Data (auth stack) Emby userdata Pushes to HOST2 daily: ──────→ Receives: All HOST1 shares Mirror of HOST1 shares Personal encrypted shares Personal (encrypted blocks) Receives from HOST2 daily: ←────── Pushes: Anime_Shows, Anime_Movies All HOST2 shares Weekly clean sync (both sides stopped): Emby — full clean mirror ←──────→ Emby Critical-Data ──────→ Auth stack (HOST1 → HOST2) Every 30 minutes — dirty sync: Emby watch states ──────→ HOST2 stays current on playback ``` --- ## ━━━ PREREQUISITES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Both servers need these before starting: ```bash # ───────────────────────────────────────────────────────────────────────────── # Required on both servers: unRAID 7.x Community Applications plugin — search "Community Applications" in unRAID plugins User Scripts plugin — install via Community Applications Tailscale plugin — install via Community Applications Terminal access — unRAID UI → Tools → Terminal, or SSH # Optional but recommended: Gitea (Docker container on HOST1) — self-hosted git for the script repository Working Emby installation — for transcode management and failover # ───────────────────────────────────────────────────────────────────────────── ``` --- ## ━━━ STEP 1 — TAILSCALE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Tailscale provides the encrypted mesh network between servers. Scripts resolve the remote server's IP via Tailscale at runtime — no hardcoded IPs, no VPN configuration, no open ports. All server-to-server communication goes through Tailscale. --- ### ── Install on Both Servers ──────────────────────────────────────────────── ```bash # On each server: # ───────────────────────────────────────────────────────────────────────────── # 1. Open Apps in the unRAID UI # 2. Search for "Tailscale" — install the plugin # 3. Settings → Tailscale → Connect # 4. Authenticate with your Tailscale account (browser opens on your machine) # 5. Verify both servers appear: https://login.tailscale.com/admin/machines # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Verify Connectivity ───────────────────────────────────────────────────── ```bash # From HOST1 — should return HOST2's 100.x.x.x Tailscale IP: tailscale ip -4 unRAID-Jayred365 # From HOST2 — should return HOST1's 100.x.x.x Tailscale IP: tailscale ip -4 unRAID-Gmer4Lfe # Test actual connectivity: tailscale ping unRAID-Jayred365 # run from HOST1 ``` > **Critical:** The hostnames in `master.conf` (`HOST1` and `HOST2`) must match the > Tailscale machine names **exactly** — case sensitive. The ecosystem resolves all > remote IPs via `tailscale ip -4 HOSTNAME` at runtime. A name mismatch means every > script that touches the remote will fail at the IP resolution step. --- ## ━━━ STEP 2 — ENABLE SSH ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unRAID 7.x has SSH disabled by default. Enable it on both servers. ```bash # On each server: # ───────────────────────────────────────────────────────────────────────────── # Settings → Management Access → Secure Shell # SSH: Enabled # SSH port: 22 # Apply # ───────────────────────────────────────────────────────────────────────────── ``` > SSH is only exposed on your local network and Tailscale interface. Scripts connect > via Tailscale IP — all traffic is encrypted end-to-end. No ports are opened to > the public internet. --- ## ━━━ STEP 3 — SSH KEYS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Two sets of keys needed: server-to-server for rsync and failover, and Gitea access for script repository pull. Generate all keys before configuring anything else. --- ### ── 3a — Server-to-Server Keys ───────────────────────────────────────────── ```bash # On HOST1 — generate HOST1's key pair: ssh-keygen -t ed25519 -f /root/.ssh/Gmer4Lfe-rsync-key -C "gmer4lfe-rsync" -N "" # On HOST2 — generate HOST2's key pair: ssh-keygen -t ed25519 -f /root/.ssh/Jayred365-rsync-key -C "jayred365-rsync" -N "" ``` --- ### ── 3b — Authorise Keys Bidirectionally ───────────────────────────────────── ```bash # HOST1's public key must be authorised on HOST2 (so HOST1 can SSH into HOST2): # ───────────────────────────────────────────────────────────────────────────── # On HOST1 — print the public key: cat /root/.ssh/Gmer4Lfe-rsync-key.pub # On HOST2 — create authorized_keys and paste HOST1's public key: mkdir -p /root/.ssh echo "PASTE_HOST1_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys chmod 600 /root/.ssh/authorized_keys # HOST2's public key must be authorised on HOST1 (so HOST2 can SSH into HOST1): # ───────────────────────────────────────────────────────────────────────────── # On HOST2 — print the public key: cat /root/.ssh/Jayred365-rsync-key.pub # On HOST1 — append HOST2's public key: echo "PASTE_HOST2_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys ``` --- ### ── 3c — Test Both Directions ──────────────────────────────────────────────── ```bash # From HOST1 — should print "connected" without a password prompt: ssh -i /root/.ssh/Gmer4Lfe-rsync-key \ root@$(tailscale ip -4 unRAID-Jayred365) \ "echo connected" # From HOST2 — should print "connected" without a password prompt: ssh -i /root/.ssh/Jayred365-rsync-key \ root@$(tailscale ip -4 unRAID-Gmer4Lfe) \ "echo connected" ``` ``` If prompted for a password: the key was not authorised correctly. → Recheck Step 3b — the public key content must be on one line → Check permissions: chmod 600 /root/.ssh/authorized_keys → Check the key file referenced in the SSH command matches what was generated ``` --- ### ── 3d — Gitea SSH Key ─────────────────────────────────────────────────────── ```bash # On BOTH servers — generate a key for Gitea access: ssh-keygen -t ed25519 -f /root/.ssh/unraid_gitea -C "unraid-gitea" -N "" # Print the public key to add to Gitea: cat /root/.ssh/unraid_gitea.pub # In Gitea: Settings → SSH / GPG Keys → Add Key → paste the output above # Do this for both servers if they have separate Gitea accounts, or once if shared ``` --- ## ━━━ STEP 4 — CLONE THE REPOSITORY ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Both servers clone from the same Gitea repository. Updates pushed to the repo propagate to both servers automatically via `git_pull_execute.sh` at the start of each daily maintenance window. --- ### ── On Both Servers ───────────────────────────────────────────────────────── ```bash # Create the target directory: mkdir -p /mnt/user/appdata/unraid_scripts # Clone the repository: GIT_SSH_COMMAND="ssh -i /root/.ssh/unraid_gitea" \ git clone git@YOUR_GITEA_HOST:FailedProxy/Unraid_Scripts.git \ /mnt/user/appdata/unraid_scripts # Replace YOUR_GITEA_HOST with your Gitea server address and port # Example: git@192.168.50.2:221 ``` --- ### ── Verify the Structure ───────────────────────────────────────────────────── ```bash ls /mnt/user/appdata/unraid_scripts/ ``` ``` Expected output: master.conf ← all user configuration — the only file you edit master_host1.conf ← HOST1-specific configuration master_host2.conf ← HOST2-specific configuration common.sh ← shared library — functions used by all scripts load_config.sh ← config loader Orchestrators/ Rsync/ Failover/ Docker_Essentials/ unRAID_Essentials/ Media/ Transcodes/ Monitors/ Tools/ Partnership/ ``` --- ### ── Make Scripts Executable ───────────────────────────────────────────────── ```bash # Execute permission on all scripts — required once after clone: find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \; ``` > `array_start.sh` auto-fixes permissions on scripts that lost the execute bit — > but this initial chmod ensures the first run works before that safeguard is active. --- ## ━━━ STEP 5 — CONFIGURE MASTER.CONF ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ All user configuration lives in `master.conf`. Every value with a comment `# REQUIRED` must be set before the first run. Everything else has working defaults. ```bash nano /mnt/user/appdata/unraid_scripts/master.conf ``` --- ### ── Host Identity ───────────────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # These must match Tailscale machine names exactly — case sensitive. # The ecosystem uses these to resolve remote IPs at runtime. # HOST1="unRAID-Gmer4Lfe" # REQUIRED — must match tailscale machine name HOST2="unRAID-Jayred365" # REQUIRED — same # SSH key paths — each server's key for authenticating to the other: HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key" # HOST1 uses this to SSH to HOST2 HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key" # HOST2 uses this to SSH to HOST1 ``` --- ### ── Emby API Keys ───────────────────────────────────────────────────────────── ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Used by: emby_session_report.sh, sunday_morning_coffee_report.sh, # sonarr/radarr cleanup (notify_emby_scan after deletion) # # Get from: Emby Dashboard → Settings → API Keys → + New API Key # HOST1_EMBY_URL="http://192.168.50.2:8096" HOST1_EMBY_API_KEY="your-host1-emby-api-key" # REQUIRED for Emby features HOST1_EMBY_CONTAINER="Emby" # master_host2.conf HOST2_EMBY_URL="http://localhost:8096" HOST2_EMBY_API_KEY="your-host2-emby-api-key" HOST2_EMBY_CONTAINER="Emby" ``` --- ### ── Git Repository ──────────────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # Used by git_pull_execute.sh — pulls latest scripts at start of each daily window. # GITEA_CONTAINER="Gitea" # exact Docker container name GITEA_REPO_PATH="FailedProxy/Unraid_Scripts.git" TARGET_DIR="/mnt/user/appdata/unraid_scripts" GITEA_SSH_KEY="/root/.ssh/unraid_gitea" SSH_PORT=221 # your Gitea SSH port ``` --- ### ── Daily Sync Shares ──────────────────────────────────────────────────────── ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Shares HOST1 is source of truth for — pushed to HOST2 every night at 1am. # HOST2 treats these as read-only mirrors. Never put the same share in both lists. # HOST1_DAILY_SYNC_SHARES=( "/mnt/user/Movies" # HOST1 manages this — Radarr runs here "/mnt/user/Tv_Shows" # HOST1 manages this — Sonarr runs here "/mnt/user/Music" # HOST1 manages this — Lidarr runs here "/mnt/user/Kids_Movies" "/mnt/user/Kids_Tv_Shows" "/mnt/user/Sports" "/mnt/user/stand-up_comedy" ) # master_host2.conf HOST2_DAILY_SYNC_SHARES=( "/mnt/user/Anime_Shows" # HOST2 manages this — his Sonarr runs here "/mnt/user/Anime_Movies" # HOST2 manages this — his Radarr runs here ) ``` --- ### ── Weekly Sync Shares ─────────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # Synced during the Sunday 2:30am window — containers stopped both sides. # Do NOT add these to a separate cron schedule — they run via weekly_sync_maintenance.sh. # WEEKLY_SYNC_SHARES=( "/mnt/user/Media_Server/Emby" # full clean Emby mirror "/mnt/user/appdata-Failover/Critical-Data" # auth stack clean state ) ``` --- ## ━━━ STEP 6 — MASTER_HOST*.CONF ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Per-host configuration lives in `master_host1.conf` and `master_host2.conf`. `detect_hosts()` in `common.sh` reads which server is running and aliases the correct `HOST*_` prefixed variables to their unprefixed names. Scripts only ever reference the unprefixed name — they work identically on both servers. ```bash # master_host1.conf is only sourced on HOST1 # master_host2.conf is only sourced on HOST2 # Changes go in the right file for the right server nano /mnt/user/appdata/unraid_scripts/master_host1.conf # on HOST1 nano /mnt/user/appdata/unraid_scripts/master_host2.conf # on HOST2 ``` See each conf file's comments — every variable is documented with its purpose and the reasoning behind the value. --- ## ━━━ STEP 7 — RSYNC PROFILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Profiles control per-share behaviour — which containers to stop, rsync flags, bandwidth limits, what to exclude. Profile is matched by directory basename (lowercased). Override with `--profile=name`. --- ### ── How Profile Matching Works ────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # rsync.sh /mnt/user/appdata-Failover/Arrs_Stack # basename: Arrs_Stack # lowercased: arrs_stack # matched profile: [arrs_stack] # # rsync.sh /mnt/user/Movies # basename: Movies # lowercased: movies # no matching profile → global defaults apply (no containers stopped) # # rsync.sh /mnt/user/appdata-Failover/Critical-Data --profile=critical-failover # explicit override → uses [critical-failover] profile regardless of path # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Current Profiles ───────────────────────────────────────────────────────── ```bash # master.conf — profile definitions # ───────────────────────────────────────────────────────────────────────────── # Each profile defines which containers to stop, rsync flags, excludes, etc. # Containers in PROFILE_CRITICAL_CONTAINER_NAMES are stopped on BOTH servers. # PROFILE_DELAYED_CONTAINERS restart after PROFILE_CONTAINER_DELAY seconds. # ── arrs_stack ───────────────────────────────────────────────────────────── # Arr databases — stopped for clean SQLite snapshot PROFILES["arrs_stack_CRITICAL_CONTAINER_NAMES"]=( "Sonarr" "Radarr" "Lidarr" "Prowlarr" "Bazarr" "Pinchflat" ) # ── critical-data ────────────────────────────────────────────────────────── # Auth stack — stopped for clean database snapshot, delayed restart PROFILES["critical-data_CRITICAL_CONTAINER_NAMES"]=( "Mariadb-Authelia" "Redis-Authelia" "NginxProxyManager" "Lldap-Gmer4Lfe" ) PROFILES["critical-data_DELAYED_CONTAINERS"]=( "Authelia" "Authelia-Secondary" # auth services restart after delay ) PROFILES["critical-data_CONTAINER_DELAY"]=30 # seconds before delayed containers start # ── important-data ───────────────────────────────────────────────────────── # NextCloud + Postgres — stopped for clean snapshot PROFILES["important-data_CRITICAL_CONTAINER_NAMES"]=( "Postgres-NextCloud" ) PROFILES["important-data_DELAYED_CONTAINERS"]=("NextCloud") # ── emby ─────────────────────────────────────────────────────────────────── # Weekly full clean sync — both Emby instances stopped PROFILES["emby_CRITICAL_CONTAINER_NAMES"]=("Emby") PROFILES["emby_EXCLUDE_DIRS"]=( "transcodes/" "logs/" "crash*" "cache/" ) # ── emby-failover ────────────────────────────────────────────────────────── # Every 30 minutes, Emby STAYS RUNNING — dirty sync of critical state only # WAL and SHM excluded — safe to copy while Emby is writing PROFILES["emby-failover_CRITICAL_CONTAINER_NAMES"]=() # empty — nothing stops PROFILES["emby-failover_EXCLUDE_DIRS"]=( "*.wal" "*.shm" # WAL files — unsafe mid-write "transcodes/" "logs/" "crash*" "cache/" # volatile data — skip ) PROFILES["emby-failover_REMOTE_RESTART_CONTAINERS"]=("Emby") # Emby on HOST2 restarts after sync to pick up config changes ``` --- ### ── Two Emby Profiles — Why Both Exist ────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # emby-failover — every 30 minutes, Emby stays running: # What syncs: users.db, library.db, authentication.db, config/ # What skips: *.wal *.shm transcodes/ logs/ cache/ # Why: WAL files are being written while Emby runs — copying them # would produce a corrupt database on HOST2 # Result: HOST2 is always within 30 minutes of HOST1 on watch state # and user activity. Failover is seamless — nobody notices. # # emby — Sunday 2:30am, both Emby instances stopped: # What syncs: everything except transcodes, logs, cache, crash files # What includes: metadata, plugins, full database state, all config # Why: WAL is checkpointed on clean shutdown — safe to copy everything # Full consistent mirror including metadata and plugin state # Result: HOST2 has a gold-standard Emby state once per week # Image cache warm for 6 days — only reset Sunday when users sleep # # The two profiles work together: # emby-failover: keeps HOST2 current on what matters for immediate failover # emby: gives HOST2 full fidelity once per week # Neither alone is sufficient — both are needed. # ───────────────────────────────────────────────────────────────────────────── ``` --- ## ━━━ STEP 8 — PERSONAL ENCRYPTED SHARES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Personal shares are synced to the remote server for offsite backup. ZFS encrypts at the dataset level — the remote server receives encrypted blocks and cannot read the content without your passphrase or keyfile. --- ### ── Create an Encrypted ZFS Dataset ──────────────────────────────────────── ```bash # In the unRAID UI: # ───────────────────────────────────────────────────────────────────────────── # Main → click your ZFS pool name → + Dataset # Name: Gmer4Lfe-Personal # Encryption: Enabled # Passphrase: [your passphrase] # ⚠️ Write your passphrase down — if lost, data is completely unrecoverable # # Settings → Shares → Add Share # Share path: point to the new encrypted dataset # Use cache: Only — keeps data on ZFS pool, not array # ───────────────────────────────────────────────────────────────────────────── # Verify encryption is active before syncing: zfs get encryption poolname/Gmer4Lfe-Personal # Should show: encryption aes-256-gcm ``` --- ### ── Auto-Unlock on Boot (Optional) ───────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # Keyfile approach — passphrase stored in a file, loaded at boot. # More convenient but the keyfile is a secret that must be protected. # Never sync the keyfile to the remote server. # # Create keyfile — on HOST1 only: dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key chmod 600 /root/.zfs-keys/personal.key # Set dataset to use keyfile instead of passphrase: zfs change-key \ -o keylocation=file:///root/.zfs-keys/personal.key \ -o keyformat=raw \ poolname/Gmer4Lfe-Personal # Add to ramdisk_setup.sh or array_start.sh custom scripts: zfs load-key poolname/Gmer4Lfe-Personal zfs mount poolname/Gmer4Lfe-Personal # ───────────────────────────────────────────────────────────────────────────── # Manual unlock alternative (most secure — passphrase only in your head): zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase zfs mount poolname/Gmer4Lfe-Personal ``` --- ### ── Add to master_host1.conf ──────────────────────────────────────────────── ```bash # master_host1.conf # ───────────────────────────────────────────────────────────────────────────── # Personal shares append to the daily sync after DAILY_SYNC_SHARES. # Remote server receives encrypted blocks — cannot read content without your key. # HOST1_PERSONAL_SHARES=( "/mnt/user/Gmer4Lfe-Personal" ) ``` --- ## ━━━ STEP 9 — USER SCRIPTS SETUP ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ The ecosystem is designed so the User Scripts plugin has only a small number of entries — each one an orchestrator. Individual scripts are never scheduled directly. --- ### ── At Startup of Array ───────────────────────────────────────────────────── ```bash # Create one script entry named "array start": # ───────────────────────────────────────────────────────────────────────────── #!/bin/bash bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh # ───────────────────────────────────────────────────────────────────────────── # Schedule: At Startup of Array # Run as: Background Task # # This is the ONLY "At Startup of Array" entry needed. # It launches everything in ARRAY_START_SCRIPTS from master.conf: # inotify_tuning.sh — raise inotify limits before containers start # docker_syslog_filter.sh — suppress veth log noise # php_fpm_max_children.sh — WebGUI tuning # ramdisk_setup.sh — create ramdisk before Emby starts # docker_network_connect.sh — connect containers to extra networks # system_watchdog.sh — continuous system health monitor # docker_watchdog.sh — continuous container health monitor # failover.sh — continuous mutual failover ``` --- ### ── Cron Schedule ──────────────────────────────────────────────────────────── ```bash # Create one script entry per cron schedule below. # All entries: Run as Background Task # ───────────────────────────────────────────────────────────────────────────── # Every 3 minutes — transcode cleanup + manager: */3 * * * * bash /mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh # Every 30 minutes — Emby dirty sync (watch states, library delta): */30 * * * * bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \ /mnt/user/Media_Server/Emby --profile=emby-failover # Every 6 hours — failed import + stalled download recovery: 0 */6 * * * bash /mnt/user/appdata/unraid_scripts/Orchestrators/arrs_failed_stalled_recovery.sh # 1am daily — full maintenance window: # git pull → rsync all shares → permissions → cleaners → arr cleanup → docker restart 0 1 * * * bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh # 2:30am Sunday — weekly maintenance window: # stop containers → pull updates → clean sync → start containers → weekly restarts 30 2 * * 0 bash /mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh # 8am daily — health digest (DIGEST_PROFILE in master.conf controls when it notifies): 0 8 * * * bash /mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh # Every 6 hours — inotify + php-fpm utilisation snapshot: 0 */6 * * * bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh # Sunday morning — weekly reports: 0 6 * * 0 bash .../Monitors/zfs_memory_snapshot.sh 0 7 * * 0 bash .../Monitors/smart_health.sh 0 9 * * 0 bash .../Monitors/cert_monitor.sh 0 10 * * 0 bash .../Monitors/backup_verify.sh 0 11 * * 0 bash .../Monitors/emby_session_report.sh 0 11 * * 0 bash .../Monitors/bandwidth_monitor.sh --report # ───────────────────────────────────────────────────────────────────────────── ``` > **Set all entries to "Background Task"** — output streams correctly to the User > Scripts log rather than buffering in the browser tab. Non-background tasks can > appear to hang on long-running scripts. --- ## ━━━ STEP 10 — VERIFY THE SETUP ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Before relying on scheduled jobs, test manually from the terminal on HOST1. Test with `--dry-run` first — no changes made, but the full pre-flight and configuration resolution runs. --- ### ── Test a Single Profile Sync ───────────────────────────────────────────── ```bash # Dry run with verbose output — shows every decision the script makes: bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \ /mnt/user/appdata-Failover/Arrs_Stack --dry-run --log ``` ``` Expected output (healthy): ━━━ ⚙️ Setup ━━━ Host: HOST1 (unRAID-Gmer4Lfe) → HOST2 (unRAID-Jayred365) Remote IP: 100.x.x.x Profile: arrs_stack ━━━ 🛡️ Pre-flight Checks ━━━ ✅ Remote reachable ✅ version parity — both on unRAID X.Y.Z ✅ Remote Docker daemon responding ✅ Remote rootfs: 12% (threshold: 75%) ✅ Remote share exists and not empty ✅ All pre-flight checks passed ``` --- ### ── Test the Daily Orchestrator ───────────────────────────────────────────── ```bash # Dry run of the full daily window — shows every job that would run: bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --dry-run ``` ``` If any pre-flight check fails, the script aborts with a clear error message before touching anything. Fix the reported issue and re-run --dry-run. Common pre-flight failures and their causes: "Remote not reachable" → Tailscale not connected on HOST2 "Version mismatch" → different unRAID versions — update before syncing "Remote rootfs above X%" → HOST2's root filesystem nearly full "Remote share missing" → share doesn't exist on HOST2 yet (see Step 11) "Docker daemon not responding" → HOST2's Docker service not started ``` --- ### ── Check the Configuration Resolved Correctly ────────────────────────────── ```bash # --status shows how master.conf resolved for this server and profile: bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \ /mnt/user/appdata-Failover/Arrs_Stack --status ``` --- ## ━━━ STEP 11 — INITIAL HOST2 SYNC ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ If HOST2 is being set up from scratch with empty shares: --- ### ── Create Share Structure on HOST2 ──────────────────────────────────────── ```bash # On HOST2 — start the array and create shares via the unRAID UI. # Or use the share recreation tool to create disk directories from HOST1's cfg files: bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh ``` --- ### ── Initial Push From HOST1 ───────────────────────────────────────────────── ```bash # On HOST1 — push all shares to HOST2 for the first time: # Use --log for verbose output on first run bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --log ``` ``` First run may take several hours for large libraries — this is normal. The scheduled nightly sync will be incremental after the initial push. Progress shows per-share throughout. ``` --- ## ━━━ NAMING CONSISTENCY — THIS IS REQUIRED ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ The ecosystem uses one codebase on both servers. This only works if containers and shares have identical names on both servers. This is not configurable — it is a design requirement. ```bash # ───────────────────────────────────────────────────────────────────────────── # Container names must match exactly on both servers: "Emby" ← both HOST1 and HOST2 "NginxProxyManager" ← both HOST1 and HOST2 "Mariadb-Authelia" ← both HOST1 and HOST2 # Share paths must match exactly on both servers: /mnt/user/Movies ← both HOST1 and HOST2 (HOST2 has a mirror) /mnt/user/Tv_Shows ← both HOST1 and HOST2 # ───────────────────────────────────────────────────────────────────────────── # If a container has a different name on one server: the script skips it # without error. It silently does the wrong thing. You only notice when # the container is not stopped during a sync that requires it to stop. # # If a share has a different path: rsync.sh aborts with "remote share missing". # Easier to catch — but still requires renaming the share to fix. # # Keep names consistent and one codebase covers both servers automatically. # Diverge and every script that touches containers or shares needs custom logic. # ───────────────────────────────────────────────────────────────────────────── ``` --- ## ━━━ REPOSITORY STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ``` Unraid_Scripts/ ├── master.conf ← All shared configuration — edit this file ├── master_host1.conf ← HOST1-specific configuration ├── master_host2.conf ← HOST2-specific configuration ├── common.sh ← Shared library — functions used by all scripts ├── load_config.sh ← Config loader — sources all conf files │ ├── Orchestrators/ │ ├── array_start.sh ← Single "At Startup of Array" entry point │ ├── daily_sync_maintenance.sh ← 1am daily window orchestrator │ ├── weekly_sync_maintenance.sh ← Sunday 2:30am window orchestrator │ ├── critical_sync_maintenance.sh ← Every 15 minutes — critical sync + partnership │ ├── media_management.sh ← Permissions + cleaners + arr cleanup │ ├── transcode_management.sh ← Transcode cleanup then manager │ └── arrs_failed_stalled_recovery.sh ← Failed import + stalled download recovery │ ├── Rsync/ │ └── rsync.sh ← Core rsync script — called per share │ ├── Failover/ │ ├── failover.sh ← Mutual container failover — continuous loop │ ├── failover_test.sh ← Controlled iptables failover simulation │ └── failover_state_reset.sh ← Reset failover state manually │ ├── Docker_Essentials/ │ ├── docker_watchdog.sh ← Two-tier container monitor — continuous loop │ ├── docker_daily_restart.sh ← Nightly container restarts │ ├── docker_weekly_restart.sh ← Weekly container restarts │ ├── docker_network_connect.sh ← Ensure networks + connections at array start │ └── watchdog_skip_list_manager.sh ← Skip list inspection and recovery │ ├── unRAID_Essentials/ │ ├── system_watchdog.sh ← Three-tier system health monitor — continuous │ ├── ramdisk_setup.sh ← Creates ramdisk + symlink at array start │ ├── inotify_tuning.sh ← Raise inotify limits at array start │ ├── docker_syslog_filter.sh ← Suppress veth log noise │ ├── php_fpm_max_children.sh ← WebGUI performance tuning │ ├── server_reboot.sh ← Graceful reboot with pre-flight warnings │ ├── mover_stop.sh ← Stop mover cleanly with wall warning │ ├── clear_logs.sh ← Size-threshold log cleanup │ ├── webgui_restart.sh ← nginx → php-fpm → emhttp escalation │ └── git_pull_execute.sh ← Pull latest scripts from Gitea │ ├── Media/ │ ├── media_shares_permissions.sh ← Apply permissions to media shares │ ├── media_cleaner.sh ← Remove junk files from media shares │ ├── lidarr_cleanup.sh ← Remove orphaned music files (HOST1 only) │ ├── sonarr_cleanup.sh ← Remove orphaned TV files (host-aware) │ └── radarr_cleanup.sh ← Remove orphaned movie files (host-aware) │ ├── Transcodes/ │ ├── transcode_manager.sh ← Ramdisk/SSD symlink management │ └── transcode_cleanup.sh ← Remove stale segment files │ ├── Monitors/ │ ├── cert_monitor.sh ← SSL cert expiry via live TLS connection │ ├── backup_verify.sh ← rsync mirror MD5 checksum verification │ ├── smart_health.sh ← Drive SMART attribute monitoring │ ├── zfs_memory_snapshot.sh ← ZFS health + ARC + memory report │ ├── bandwidth_monitor.sh ← rsync transfer logging + weekly report │ ├── weekly_health_digest.sh ← Full ecosystem health aggregation │ ├── emby_session_report.sh ← Emby streaming usage statistics │ ├── system_tuning_monitor.sh ← inotify + php-fpm utilisation tracking │ └── continuous_scripts_status.sh ← Live dashboard for background processes │ ├── Partnership/ │ └── partnership_manage.sh ← Two-server relationship lifecycle manager │ └── Tools/ ├── recreate_shares.sh ← Create share directories from cfg files ├── bulk_permissions_repair.sh ← One-shot permission repair ├── rsync_stop.sh ← Stop active rsync jobs cleanly ├── user_scripts_stop.sh ← Stop running user script processes ├── server_reboot.sh ← Graceful scheduled reboot ├── zfs_pool_scrub.sh ← Trigger ZFS pool scrub └── container_data_export.sh ← Export container configuration ``` --- ## ━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ --- ### 🔴 SSH Connection Refused / Timeout ```bash # Verify SSH is enabled on the remote: # Settings → Management Access → Secure Shell → Enabled # Verify Tailscale is connected: tailscale ip -4 unRAID-Jayred365 # should return 100.x.x.x # Test SSH manually with the key: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@$(tailscale ip -4 unRAID-Jayred365) "hostname" # Expected: unRAID-Jayred365 # If password prompted: key not authorised — recheck Step 3b ``` --- ### 🔴 Pre-flight Aborts on Remote Rootfs ```bash # Remote rootfs above ROOTFS_WARN threshold # Check current usage on remote: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "df /" # Common cause: array not started, drives not mounted # Verify array is started on HOST2 before running syncs ``` --- ### 🔴 Remote Share Missing ```bash # Share exists locally but not on remote # Verify the share exists on HOST2: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "ls /mnt/user/" # If missing — create the share on HOST2 first, then run initial sync (Step 11) # Or run recreate_shares.sh on HOST2 to create directories from HOST1's cfg files ``` --- ### 🔴 Containers Not Stopping / Starting ```bash # Verify container names in master_host*.conf match Docker exactly — case sensitive # Check what Docker actually calls the container: docker ps --format "{{.Names}}" # Test Docker commands to remote manually: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Names}}'" ``` --- ### 🔴 Profile Not Matching ```bash # Profile key = directory basename lowercased # /mnt/user/appdata-Failover/Arrs_Stack → basename: Arrs_Stack → key: arrs_stack # Override explicitly if basename doesn't match a profile name: bash rsync.sh /mnt/user/appdata-Failover/My_Stuff --profile=arrs_stack # Verify what profile resolved for a path: bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status ``` --- ### 🔴 Script Not Found ```bash # Verify repo was cloned to the correct location: ls /mnt/user/appdata/unraid_scripts/master.conf # Make scripts executable: find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \; ``` --- ## ━━━ AVAILABLE FLAGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ All scripts support these flags. Use `--dry-run` before any live operation. ```bash # ───────────────────────────────────────────────────────────────────────────── --dry-run run without making any changes — pre-flight still runs ✅ --log verbose output — show every decision made --status show resolved configuration and exit — no rsync, no sync --no-log suppress verbose output (some scripts) # Examples: bash rsync.sh /mnt/user/Movies --dry-run --log # preview a media sync bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status # check profile resolution bash daily_sync_maintenance.sh --dry-run # preview full daily window bash docker_watchdog.sh --status # check watchdog state # ───────────────────────────────────────────────────────────────────────────── ```