# ━━━━━ RSYNC — Manual ━━━━━ Config reference, procedures, operational workflows. For overview see README-Rsync.md. For per-script detail see the rsync.sh header. This document also serves as the **initial setup guide** for standing up the two-server ecosystem from scratch. --- ## ━━━ WHAT YOU'RE BUILDING ━━━ ``` HOST1 (unRAID-Gmer4Lfe) HOST2 (unRAID-Jayred365) ────────────────────── ────────────────────── Downloads to: Downloads to: /mnt/user/Movies /mnt/user/Anime_Movies /mnt/user/Tv_Shows /mnt/user/Anime_Shows /mnt/user/Music Both servers' arrs track ALL content across both servers. It does not matter who downloaded what or when. Daily 3-phase window (1am): Phase 1 — arr_sync.sh (runs first): All arrs on all nodes reconcile libraries using external IDs (TMDB, TVDB, MusicBrainz). Union model — any node that tracks an item, all nodes get it. After this phase both servers' arrs know about all content regardless of who downloaded it. Phase 2 — rsync (bidirectional, no --delete on media shares): HOST1 pushes its shares ──────► HOST2 receives files HOST1 receives files ◄────── HOST2 pushes its shares Files arrive already tracked by the remote arr (arr_sync ran first). No --delete — media files only spread. Arr cleanup handles deletions. Phase 3 — arr cleanup: sonarr_cleanup / radarr_cleanup / lidarr_cleanup query the live arr API and remove any files no longer tracked. Emby notified after. Note: profiled syncs (arrs_stack, emby, critical-data…) use their own PROFILE_RSYNC_OPTS which include --delete. Only the no-profile media shares use DEFAULT_RSYNC_OPTS (spread only). Weekly clean sync (2:30am Sunday, containers stopped): Emby — full clean mirror ◄──────► Emby Critical-Data ──────► Auth stack (HOST1 → HOST2) Every 30 minutes — dirty sync: Emby watch states ──────► HOST2 stays current on playback ``` --- ## ━━━ PREREQUISITES ━━━ Required on both servers before starting: ``` unRAID 7.x Community Applications plugin — search "Community Applications" in unRAID plugins User Scripts plugin — install via Community Applications Tailscale plugin — install via Community Applications Terminal access — unRAID UI → Tools → Terminal, or SSH ``` Optional but recommended: ``` Gitea (Docker container on HOST1) — self-hosted git for the script repository Working Emby installation — for transcode management and fallback ``` --- ## ━━━ STEP 1 — TAILSCALE ━━━ Tailscale provides the encrypted mesh network between servers. Scripts resolve the remote server's IP via Tailscale at runtime — no hardcoded IPs, no VPN configuration, no open ports. All server-to-server communication goes through Tailscale. ### Install on Both Servers ``` 1. Open Apps in the unRAID UI 2. Search "Tailscale" — install the plugin 3. Settings → Tailscale → Connect 4. Authenticate with your Tailscale account (browser opens on your machine) 5. Verify both servers appear: https://login.tailscale.com/admin/machines ``` ### Verify Connectivity ```bash # From HOST1 — should return HOST2's 100.x.x.x Tailscale IP: tailscale ip -4 unRAID-Jayred365 # Test actual connectivity: tailscale ping unRAID-Jayred365 ``` > **Critical:** The hostnames in `master.conf` (`HOST1` and `HOST2`) must match the > Tailscale machine names **exactly** — case sensitive. All remote IP resolution goes > through `tailscale ip -4 HOSTNAME`. A name mismatch causes every remote operation to > fail at the IP resolution step. --- ## ━━━ STEP 2 — ENABLE SSH ━━━ unRAID 7.x has SSH disabled by default. Enable it on both servers. ``` Settings → Management Access → Secure Shell SSH: Enabled SSH port: 22 Apply ``` SSH is only exposed on your local network and Tailscale interface. No ports are opened to the public internet. --- ## ━━━ STEP 3 — SSH KEYS ━━━ Two sets of keys needed: rsync automation keys (server-to-server) and a Gitea access key (for script repository pulls). ### 3a — Rsync Automation Keys (ssh_setup.sh) Run on **each server**. `ssh_setup.sh` generates the key, copies it to the remote, and updates `host*.conf` with the key path automatically. ```bash # On HOST1 — after the repo is cloned: bash /boot/config/plugins/varaverk/Partnership/ssh_setup.sh # On HOST2: bash /boot/config/plugins/varaverk/Partnership/ssh_setup.sh ``` Key naming convention: hostname lowercased, `unraid-` prefix stripped. ``` unRAID-Gmer4Lfe → /root/.ssh/gmer4lfe_rsync_automation unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation ``` `host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the generated key. Run `--status` to verify: ```bash bash /boot/config/plugins/varaverk/Partnership/ssh_setup.sh --status ``` > **`ssh_setup.sh` is idempotent** — safe to re-run. Use `--force` to regenerate > a key (e.g., after a security incident) and re-copy it to the remote. ### 3b — Gitea SSH Key (manual) ```bash # On BOTH servers: ssh-keygen -t ed25519 -f /root/.ssh/unraid_gitea -C "unraid-gitea" -N "" # Print the public key to add to Gitea: cat /root/.ssh/unraid_gitea.pub # In Gitea: Settings → SSH / GPG Keys → Add Key → paste above ``` --- ## ━━━ STEP 4 — CLONE THE REPOSITORY ━━━ ```bash # Create target directory: mkdir -p /boot/config/plugins/varaverk # Clone: GIT_SSH_COMMAND="ssh -i /root/.ssh/unraid_gitea" \ git clone git@YOUR_GITEA_HOST:Varaverk/varaverk.git \ /boot/config/plugins/varaverk # Make scripts executable: find /boot/config/plugins/varaverk -name "*.sh" -exec chmod +x {} \; ``` Expected structure after clone: ``` master.conf ← all shared configuration host1.conf ← HOST1-specific configuration host2.conf ← HOST2-specific configuration common.sh ← shared library load_config.sh ← config loader Orchestrators/ Rsync/ Fallback/ Docker_Essentials/ ... ``` --- ## ━━━ STEP 5 — CONFIGURE MASTER.CONF ━━━ ```bash nano /boot/config/plugins/varaverk/master.conf ``` ### Host Identity ```bash # These must match Tailscale machine names exactly — case sensitive. HOST1="unRAID-Gmer4Lfe" # REQUIRED HOST2="unRAID-Jayred365" # REQUIRED # SSH keys — each server uses its own key to authenticate to the other: HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key" HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key" ``` ### Git Repository ```bash GITEA_CONTAINER="Gitea" GITEA_REPO_PATH="Varaverk/varaverk.git" TARGET_DIR="/boot/config/plugins/varaverk" GITEA_SSH_KEY="/root/.ssh/unraid_gitea" SSH_PORT=221 ``` ### Daily Sync Shares ```bash # host1.conf # Shares this server downloads to — pushed to the other server nightly. # arr_sync.sh runs before rsync in the daily window, so the receiving # server's arrs already track incoming files when they arrive. # Never put the same path in both lists. HOST1_DAILY_SYNC_SHARES=( "/mnt/user/Movies" "/mnt/user/Tv_Shows" "/mnt/user/Music" "/mnt/user/Kids_Movies" "/mnt/user/Kids_Tv_Shows" "/mnt/user/Sports" "/mnt/user/stand-up_comedy" ) # host2.conf HOST2_DAILY_SYNC_SHARES=( "/mnt/user/Anime_Shows" "/mnt/user/Anime_Movies" ) ``` ### Weekly Sync Shares ```bash # master.conf # Synced during the Sunday 2:30am window — containers stopped both sides. WEEKLY_SYNC_SHARES=( "/mnt/user/Media_Server/Emby" "/mnt/user/appdata-Fallback/Critical-Data" ) ``` --- ## ━━━ STEP 6 — MASTER_HOST*.CONF ━━━ `detect_hosts()` reads which server is running and aliases `HOST*_` prefixed vars to their unprefixed names. Scripts only ever reference the unprefixed name — they work identically on both servers. ```bash nano /boot/config/plugins/varaverk/host1.conf # on HOST1 nano /boot/config/plugins/varaverk/host2.conf # on HOST2 ``` Every variable is documented in the conf files. Key values to set: - SSH key paths - DAILY_SYNC_SHARES - EMBY_URL / EMBY_API_KEY - CERT_MONITOR_DOMAINS - SMART_IGNORE_DRIVES - ZFS_REPORT_IGNORE_POOLS --- ## ━━━ RSYNC PROFILES ━━━ Profiles control per-share behavior. Profile key = directory basename lowercased. Override with `--profile=name`. ### Profile Matching ```bash # rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack # basename: Arrs_Stack → lowercased: arrs_stack → matches [arrs_stack] profile # # rsync.sh /mnt/user/Movies # basename: Movies → no profile match → global defaults (no containers stopped) # # rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data # explicit override ``` ### Current Profile Definitions ```bash # master.conf # ── arrs_stack ─────────────────────────────────────────────────────────────── # Arr databases — stopped for clean SQLite snapshot PROFILES["arrs_stack_CRITICAL_CONTAINER_NAMES"]=( "Sonarr" "Radarr" "Lidarr" "Prowlarr" "Bazarr" "Pinchflat" ) # ── critical-data ───────────────────────────────────────────────────────────── # Auth stack — stopped for clean database snapshot, Authelia delayed restart PROFILES["critical-data_CRITICAL_CONTAINER_NAMES"]=( "Mariadb-Authelia" "Redis-Authelia" "NginxProxyManager" "Lldap" ) PROFILES["critical-data_DELAYED_CONTAINERS"]=( "Authelia" "Authelia-Secondary" ) PROFILES["critical-data_CONTAINER_DELAY"]=30 # ── important-data ──────────────────────────────────────────────────────────── # NextCloud + Postgres — stopped for clean snapshot PROFILES["important-data_CRITICAL_CONTAINER_NAMES"]=( "Postgres-NextCloud" ) PROFILES["important-data_DELAYED_CONTAINERS"]=("NextCloud") # ── emby ────────────────────────────────────────────────────────────────────── # Weekly full clean sync — both Emby instances stopped PROFILES["emby_CRITICAL_CONTAINER_NAMES"]=("Emby") PROFILES["emby_EXCLUDE_DIRS"]=( "transcodes/" "logs/" "crash*" "cache/" ) ``` ### The `emby` Profile and Watch-State Sync **emby** (Sunday 2:30am, both Emby instances stopped): - Syncs: everything except transcodes, logs, cache, crash files - Includes: metadata, plugins, full database state, all config - Why: WAL is checkpointed on clean shutdown — safe to copy everything - Result: HOST2 gets a gold-standard Emby state once per week Between weekly mirrors, Emby **watch/resume state** is kept current by `Media/play_state_sync.sh`, which reads and writes play state over the Emby API every 30 minutes (CRITICAL_MAINTENANCE_SCRIPTS) — so HOST2 never falls more than one cycle behind on what's been watched. This replaced the old `emby-fallback` dirty-rsync profile: copying Emby's live SQLite databases mid-write risked landing a half-written WAL file on HOST2, and an API-level sync sidesteps the database-consistency problem entirely. --- ## ━━━ PERSONAL ENCRYPTED SHARES ━━━ Personal shares sync to the remote for offsite backup. ZFS encrypts at the dataset level — the remote receives encrypted blocks and cannot read the content without your passphrase or keyfile. ### Create an Encrypted ZFS Dataset ```bash # In the unRAID UI: # Main → click your ZFS pool name → + Dataset # Name: Gmer4Lfe-Personal # Encryption: Enabled # Passphrase: [your passphrase] # Write your passphrase down — if lost, data is completely unrecoverable # Verify encryption is active before syncing: zfs get encryption poolname/Gmer4Lfe-Personal # Should show: encryption aes-256-gcm ``` ### Auto-Unlock on Boot (Optional) ```bash # Keyfile approach — more convenient, but the keyfile itself is a secret dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key chmod 600 /root/.zfs-keys/personal.key zfs change-key \ -o keylocation=file:///root/.zfs-keys/personal.key \ -o keyformat=raw \ poolname/Gmer4Lfe-Personal # Add to array_started.sh or ramdisk_setup.sh: zfs load-key poolname/Gmer4Lfe-Personal zfs mount poolname/Gmer4Lfe-Personal # Manual unlock alternative (most secure): zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase zfs mount poolname/Gmer4Lfe-Personal ``` ### Add to host1.conf ```bash HOST1_PERSONAL_SHARES=( "/mnt/user/Gmer4Lfe-Personal" ) ``` --- ## ━━━ STEP 7 — SCHEDULER SETUP ━━━ All scheduling is managed by the Varaverk plugin — no User Scripts entries needed. ### Array start (Varaverk disks_mounted event hook) The Varaverk plugin fires `array_started.sh` automatically via its `disks_mounted` event hook. It's registered in `schedule.json`: ```json "Orchestrators/array_started.sh": { "cron": "array_start", "enabled": true } ``` Launches everything in ARRAY_START_SCRIPTS: api_key_renew, inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup, docker_network_connect, fallback. Watchdogs run separately via watchdog_orchestrator.sh (cron — every 15 minutes). ### Cron Schedule (Varaverk Scheduler) ```bash */7 * * * * transcode_management.sh */15 * * * * watchdog_orchestrator.sh */30 * * * * critical_sync_maintenance.sh 0 */4 * * * intermediate_sync_maintenance.sh 0 1 * * * daily_sync_maintenance.sh 0 7 * * 0 sunday_morning_coffee_report.sh 30 2 * * 0 weekly_sync_maintenance.sh 0 0 15 * * monthly_maintenance.sh ``` All entries are configured via the Varaverk scheduler UI or `schedule.json`. Scripts live at `/boot/config/plugins/varaverk/`. --- ## ━━━ VERIFICATION ━━━ Test with `--dry-run` first — all pre-flight checks run, no changes made. ### Test a Single Profile Sync ```bash bash /boot/config/plugins/varaverk/Rsync/rsync.sh \ /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log ``` Expected output (healthy): ``` ━━━ Setup ━━━ Host: HOST1 (unRAID-Gmer4Lfe) → HOST2 (unRAID-Jayred365) Remote IP: 100.x.x.x Profile: arrs_stack ━━━ Pre-flight ━━━ Remote reachable version parity — both on unRAID X.Y.Z Remote Docker daemon responding Remote rootfs: 12% (threshold: 75%) Remote share exists and not empty All pre-flight checks passed ``` ### Test the Daily Orchestrator ```bash bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run ``` ### Check Configuration Resolution ```bash bash /boot/config/plugins/varaverk/Rsync/rsync.sh \ /mnt/user/appdata-Fallback/Arrs_Stack --status ``` --- ## ━━━ INITIAL HOST2 SYNC ━━━ If HOST2 is being set up from scratch with empty shares: ```bash # Create share directories on HOST2: bash /boot/config/plugins/varaverk/Tools/recreate_shares.sh # Initial push from HOST1 — first run may take several hours for large libraries: bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --log ``` The scheduled nightly sync will be incremental after the initial push. --- ## ━━━ NAMING CONSISTENCY REQUIREMENT ━━━ The ecosystem uses one codebase on both servers. This only works if containers and shares have identical names on both servers. **This is not configurable — it is a design requirement.** ``` Container names must match exactly on both servers: "Emby" ← both HOST1 and HOST2 "NginxProxyManager" ← both HOST1 and HOST2 "Mariadb-Authelia" ← both HOST1 and HOST2 Share paths must match exactly on both servers: /mnt/user/Movies ← both HOST1 and HOST2 /mnt/user/Tv_Shows ← both HOST1 and HOST2 ``` If a container has a different name on one server: the script skips it silently. You only notice when the container is not stopped during a sync that requires it. If a share has a different path: rsync.sh aborts with "remote share missing." Easier to catch — but still requires renaming the share to fix. --- ## ━━━ TROUBLESHOOTING ━━━ ### SSH Connection Refused / Timeout ```bash # Verify SSH is enabled on the remote: # Settings → Management Access → Secure Shell → Enabled # Verify Tailscale is connected: tailscale ip -4 unRAID-Jayred365 # Test SSH manually (key path from --status output): ssh -i /root/.ssh/gmer4lfe_rsync_automation \ root@$(tailscale ip -4 unRAID-Jayred365) "hostname" # If password prompted: key not authorised — re-run ssh_setup.sh # Re-run setup (idempotent, re-copies key to remote): bash /boot/config/plugins/varaverk/Partnership/ssh_setup.sh ``` ### Pre-flight Aborts on Remote Rootfs ```bash # Check current usage on remote: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "df /" # Common cause: array not started, drives not mounted ``` ### Remote Share Missing ```bash # Verify the share exists on HOST2: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "ls /mnt/user/" # If missing: create the share on HOST2, then run initial sync ``` ### Containers Not Stopping / Starting ```bash # Verify container names match Docker exactly — case sensitive: docker ps --format "{{.Names}}" # Test remote: ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Names}}'" ``` ### Profile Not Matching ```bash # Profile key = directory basename lowercased # /mnt/user/appdata-Fallback/Arrs_Stack → key: arrs_stack # Override explicitly: bash rsync.sh /mnt/user/appdata-Fallback/My_Stuff --profile=arrs_stack # Verify what profile resolved: bash rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status ``` --- ## ━━━ FULL CONFIGURATION REFERENCE ━━━ ### master.conf ```bash # Rsync engine RSYNC_ENABLED=true DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace" # No --delete in DEFAULT_RSYNC_OPTS — media share rsync is additive only. # arr_sync.sh keeps all arr databases in union — either server can download anything. # arr cleanup handles deletions (orphans only). Profiles use --delete explicitly. BW_LIMIT=0 # KB/s, 0 = unlimited RETRY_COUNT=3 SLEEP=60 # seconds between retries ROOTFS_WARN=75 # abort if remote rootfs above this % # Shared with Monitors/ BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db" BANDWIDTH_LOG_RETENTION=90 BANDWIDTH_WARN_GB=50 # Profile definitions (see RSYNC PROFILES section above) declare -A PROFILES declare -A PROFILE_BW_LIMIT declare -A PROFILE_RETRY_COUNT declare -A PROFILE_SLEEP declare -A PROFILE_CONTAINER_DELAY declare -A PROFILE_CRITICAL_CONTAINER_NAMES declare -A PROFILE_DELAYED_CONTAINERS declare -A PROFILE_EXCLUDE_DIRS declare -A PROFILE_REMOTE_RESTART_CONTAINERS ``` ### host*.conf ```bash # Daily sync shares — one list per server (mutually exclusive) HOST1_DAILY_SYNC_SHARES=(...) HOST2_DAILY_SYNC_SHARES=(...) # Personal encrypted shares HOST1_PERSONAL_SHARES=(...) # SSH key for this server to authenticate to the remote # Set automatically by ssh_setup.sh — do not edit manually HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation" HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation" ``` --- ## ━━━ OUTPUT TIERS ━━━ Without `--log`, each run shows: section headers (Pre-flight, Transfer, Stop/Start Containers), the transfer identity block (source, remote, profile, identity), a progress indicator while rsync is running, and a summary block with duration, bytes transferred, and status. Warnings and errors are always visible. With `--log`, every decision is shown — pre-flight check results, profile resolution, per-container stop/start state, retry details, and bandwidth log confirmation. Use for first-run validation or when investigating unexpected behavior. --- ## ━━━ FLAG REFERENCE ━━━ All flags work on `rsync.sh` and all orchestrators. ### --dry-run Runs all pre-flight checks. Shows what rsync would transfer. No transfer, no container stops, no bandwidth log entry. Safe to run at any time. ```bash rsync.sh /mnt/user/Movies --dry-run rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log daily_sync_maintenance.sh --dry-run ``` ### --status Shows resolved configuration — profile, remote identity, all vars that would be used — then exits. No pre-flight checks, no rsync. Use to verify configuration loaded correctly. ```bash rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status ``` ### --log Verbose output throughout. Every decision, every container operation, every rsync progress line. Use for first-time runs or when investigating issues. ### --profile=name Override profile selection. Bypasses basename inference. Use when the directory name doesn't match any profile key, or when testing a specific profile. ```bash rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data rsync.sh /mnt/user/Media_Server/Emby --profile=emby ``` ### --seed Skip the empty-remote-share guard. Normally `check_remote_share()` aborts when the target directory on the remote is missing or empty, because that usually means the array is not mounted and syncing into it would write onto the underlying root filesystem. `--seed` is the deliberate exception, for the first push of a genuinely new share. ```bash rsync.sh /mnt/user/New_Share --seed ``` ### --merge-run Bidirectional merge. Two passes: ``` Pass 1 pull remote → local, with --ignore-existing Pass 2 push local → remote, with --delete ``` Local ends up authoritative: the remote may contribute files the local lacks, but never a competing version of a file the local already has. Also triggered automatically when the pre-scan detects ≥75% directory overlap with the remote. **`--delete` is applied only if pass 1 completed.** Its entire justification is "local is now the authoritative superset", and a pull that was capped by `RSYNC_MAX_RUNTIME_HOURS` or failed outright means it is not — the remote still holds content the local never received. On an incomplete pull the push runs **without** `--delete`, warns, and notifies. Local content still propagates; nothing remote-unique is destroyed; the delete happens on a later run whose pull actually succeeded. > This was a real defect, fixed 2026-08-01. `--delete` used to be unconditional, so a capped > or failed pull would permanently delete remote content the pull had not yet retrieved — and > the timeout branch's own comment promised it would "resume the pull next run", which was > impossible once the data was gone. If you are changing this function, that interlock is the > part that matters. --- ## ━━━ SOURCE PATH GUARDS ━━━ `rsync.sh` pushes to `root@remote` at the **same absolute path**, and `--merge-run` adds `--delete`. The source argument is therefore a remote-side hazard, not just a local one, so two guards run before anything else: | Guard | Rejects | |-------|---------| | Absolute path, ≥3 components | `/`, `/mnt`, `/mnt/user`, empty | | Directory must exist locally | typo'd paths | Every real job path clears both — see `HOST*_DAILY_SYNC_SHARES` and friends, all of which are `/mnt/user/` or deeper. Without the existence check, a mistyped path under `--merge-run` would be *created* locally by the pull pass, filled from the remote, then pushed back with `--delete`. --- ## ━━━ RUNTIME CEILING ━━━ No single transfer attempt may exceed `RSYNC_MAX_RUNTIME_HOURS` (default 23). On timeout the transfer is terminated and resumes on the next scheduled run. This is only safe because `--partial` is in `DEFAULT_RSYNC_OPTS` — a terminated transfer picks up where it left off rather than restarting. The cap exists so one huge or stuck transfer cannot hold its per-profile lock indefinitely and starve every other profile of a turn. Both `--merge-run` passes are capped independently.