Files

T

Gmer4Lfe 53aa72d38c Correct Fallback tier variable naming and document rsync merge-run

The fallback docs used an older COVERS naming in the wrong conf file, and rsync's merge-run
mode — the only path that carries --delete — was undocumented entirely.

2026-08-01 23:05:30 -04:00

24 KiB

Raw Permalink Blame History

━━━━━ RSYNC — Manual ━━━━━

Config reference, procedures, operational workflows. For overview see README-Rsync.md. For per-script detail see the rsync.sh header.

This document also serves as the initial setup guide for standing up the two-server ecosystem from scratch.

━━━ WHAT YOU'RE BUILDING ━━━

HOST1 (unRAID-Gmer4Lfe)                    HOST2 (unRAID-Jayred365)
──────────────────────                     ──────────────────────
Downloads to:                              Downloads to:
  /mnt/user/Movies                           /mnt/user/Anime_Movies
  /mnt/user/Tv_Shows                         /mnt/user/Anime_Shows
  /mnt/user/Music

Both servers' arrs track ALL content across both servers.
It does not matter who downloaded what or when.

Daily 3-phase window (1am):

  Phase 1 — arr_sync.sh (runs first):
    All arrs on all nodes reconcile libraries using external IDs
    (TMDB, TVDB, MusicBrainz). Union model — any node that tracks
    an item, all nodes get it. After this phase both servers' arrs
    know about all content regardless of who downloaded it.

  Phase 2 — rsync (bidirectional, no --delete on media shares):
    HOST1 pushes its shares  ──────►  HOST2 receives files
    HOST1 receives files     ◄──────  HOST2 pushes its shares
    Files arrive already tracked by the remote arr (arr_sync ran first).
    No --delete — media files only spread. Arr cleanup handles deletions.

  Phase 3 — arr cleanup:
    sonarr_cleanup / radarr_cleanup / lidarr_cleanup query the live arr
    API and remove any files no longer tracked. Emby notified after.

Note: profiled syncs (arrs_stack, emby, critical-data…) use their own
PROFILE_RSYNC_OPTS which include --delete. Only the no-profile media
shares use DEFAULT_RSYNC_OPTS (spread only).

Weekly clean sync (2:30am Sunday, containers stopped):
  Emby — full clean mirror       ◄──────►  Emby
  Critical-Data                  ──────►   Auth stack (HOST1 → HOST2)

Every 30 minutes — dirty sync:
  Emby watch states              ──────►   HOST2 stays current on playback

━━━ PREREQUISITES ━━━

Required on both servers before starting:

unRAID 7.x
Community Applications plugin     — search "Community Applications" in unRAID plugins
User Scripts plugin               — install via Community Applications
Tailscale plugin                  — install via Community Applications
Terminal access                   — unRAID UI → Tools → Terminal, or SSH

Optional but recommended:

Gitea (Docker container on HOST1)  — self-hosted git for the script repository
Working Emby installation          — for transcode management and fallback

━━━ STEP 1 — TAILSCALE ━━━

Tailscale provides the encrypted mesh network between servers. Scripts resolve the remote server's IP via Tailscale at runtime — no hardcoded IPs, no VPN configuration, no open ports. All server-to-server communication goes through Tailscale.

Install on Both Servers

1. Open Apps in the unRAID UI
2. Search "Tailscale" — install the plugin
3. Settings → Tailscale → Connect
4. Authenticate with your Tailscale account (browser opens on your machine)
5. Verify both servers appear: https://login.tailscale.com/admin/machines

Verify Connectivity

# From HOST1 — should return HOST2's 100.x.x.x Tailscale IP:
tailscale ip -4 unRAID-Jayred365

# Test actual connectivity:
tailscale ping unRAID-Jayred365

Critical: The hostnames in master.conf (HOST1 and HOST2) must match the Tailscale machine names exactly — case sensitive. All remote IP resolution goes through tailscale ip -4 HOSTNAME. A name mismatch causes every remote operation to fail at the IP resolution step.

━━━ STEP 2 — ENABLE SSH ━━━

unRAID 7.x has SSH disabled by default. Enable it on both servers.

Settings → Management Access → Secure Shell
  SSH: Enabled
  SSH port: 22
Apply

SSH is only exposed on your local network and Tailscale interface. No ports are opened to the public internet.

━━━ STEP 3 — SSH KEYS ━━━

Two sets of keys needed: rsync automation keys (server-to-server) and a Gitea access key (for script repository pulls).

3a — Rsync Automation Keys (ssh_setup.sh)

Run on each server. ssh_setup.sh generates the key, copies it to the remote, and updates host*.conf with the key path automatically.

# On HOST1 — after the repo is cloned:
bash /boot/config/plugins/varaverk/Partnership/ssh_setup.sh

# On HOST2:
bash /boot/config/plugins/varaverk/Partnership/ssh_setup.sh

Key naming convention: hostname lowercased, unraid- prefix stripped.

unRAID-Gmer4Lfe  → /root/.ssh/gmer4lfe_rsync_automation
unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation

host*.conf is updated automatically with HOST*_SSH_KEY pointing to the generated key. Run --status to verify:

bash /boot/config/plugins/varaverk/Partnership/ssh_setup.sh --status

ssh_setup.sh is idempotent — safe to re-run. Use --force to regenerate a key (e.g., after a security incident) and re-copy it to the remote.

3b — Gitea SSH Key (manual)

# On BOTH servers:
ssh-keygen -t ed25519 -f /root/.ssh/unraid_gitea -C "unraid-gitea" -N ""

# Print the public key to add to Gitea:
cat /root/.ssh/unraid_gitea.pub

# In Gitea: Settings → SSH / GPG Keys → Add Key → paste above

━━━ STEP 4 — CLONE THE REPOSITORY ━━━

# Create target directory:
mkdir -p /boot/config/plugins/varaverk

# Clone:
GIT_SSH_COMMAND="ssh -i /root/.ssh/unraid_gitea" \
    git clone git@YOUR_GITEA_HOST:Varaverk/varaverk.git \
    /boot/config/plugins/varaverk

# Make scripts executable:
find /boot/config/plugins/varaverk -name "*.sh" -exec chmod +x {} \;

Expected structure after clone:

master.conf          ← all shared configuration
host1.conf    ← HOST1-specific configuration
host2.conf    ← HOST2-specific configuration
common.sh            ← shared library
load_config.sh       ← config loader
Orchestrators/
Rsync/
Fallback/
Docker_Essentials/
...

━━━ STEP 5 — CONFIGURE MASTER.CONF ━━━

nano /boot/config/plugins/varaverk/master.conf

Host Identity

# These must match Tailscale machine names exactly — case sensitive.
HOST1="unRAID-Gmer4Lfe"       # REQUIRED
HOST2="unRAID-Jayred365"       # REQUIRED

# SSH keys — each server uses its own key to authenticate to the other:
HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key"
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"

Git Repository

GITEA_CONTAINER="Gitea"
GITEA_REPO_PATH="Varaverk/varaverk.git"
TARGET_DIR="/boot/config/plugins/varaverk"
GITEA_SSH_KEY="/root/.ssh/unraid_gitea"
SSH_PORT=221

Daily Sync Shares

# host1.conf
# Shares this server downloads to — pushed to the other server nightly.
# arr_sync.sh runs before rsync in the daily window, so the receiving
# server's arrs already track incoming files when they arrive.
# Never put the same path in both lists.
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"
    "/mnt/user/Tv_Shows"
    "/mnt/user/Music"
    "/mnt/user/Kids_Movies"
    "/mnt/user/Kids_Tv_Shows"
    "/mnt/user/Sports"
    "/mnt/user/stand-up_comedy"
)

# host2.conf
HOST2_DAILY_SYNC_SHARES=(
    "/mnt/user/Anime_Shows"
    "/mnt/user/Anime_Movies"
)

Weekly Sync Shares

# master.conf
# Synced during the Sunday 2:30am window — containers stopped both sides.
WEEKLY_SYNC_SHARES=(
    "/mnt/user/Media_Server/Emby"
    "/mnt/user/appdata-Fallback/Critical-Data"
)

━━━ STEP 6 — MASTER_HOST*.CONF ━━━

detect_hosts() reads which server is running and aliases HOST*_ prefixed vars to their unprefixed names. Scripts only ever reference the unprefixed name — they work identically on both servers.

nano /boot/config/plugins/varaverk/host1.conf    # on HOST1
nano /boot/config/plugins/varaverk/host2.conf    # on HOST2

Every variable is documented in the conf files. Key values to set:

SSH key paths
DAILY_SYNC_SHARES
EMBY_URL / EMBY_API_KEY
CERT_MONITOR_DOMAINS
SMART_IGNORE_DRIVES
ZFS_REPORT_IGNORE_POOLS

━━━ RSYNC PROFILES ━━━

Profiles control per-share behavior. Profile key = directory basename lowercased. Override with --profile=name.

Profile Matching

# rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack
#   basename: Arrs_Stack → lowercased: arrs_stack → matches [arrs_stack] profile
#
# rsync.sh /mnt/user/Movies
#   basename: Movies → no profile match → global defaults (no containers stopped)
#
# rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
#   explicit override

Current Profile Definitions

# master.conf

# ── arrs_stack ───────────────────────────────────────────────────────────────
# Arr databases — stopped for clean SQLite snapshot
PROFILES["arrs_stack_CRITICAL_CONTAINER_NAMES"]=(
    "Sonarr" "Radarr" "Lidarr" "Prowlarr" "Bazarr" "Pinchflat"
)

# ── critical-data ─────────────────────────────────────────────────────────────
# Auth stack — stopped for clean database snapshot, Authelia delayed restart
PROFILES["critical-data_CRITICAL_CONTAINER_NAMES"]=(
    "Mariadb-Authelia" "Redis-Authelia"
    "NginxProxyManager" "Lldap"
)
PROFILES["critical-data_DELAYED_CONTAINERS"]=(
    "Authelia" "Authelia-Secondary"
)
PROFILES["critical-data_CONTAINER_DELAY"]=30

# ── important-data ────────────────────────────────────────────────────────────
# NextCloud + Postgres — stopped for clean snapshot
PROFILES["important-data_CRITICAL_CONTAINER_NAMES"]=(
    "Postgres-NextCloud"
)
PROFILES["important-data_DELAYED_CONTAINERS"]=("NextCloud")

# ── emby ──────────────────────────────────────────────────────────────────────
# Weekly full clean sync — both Emby instances stopped
PROFILES["emby_CRITICAL_CONTAINER_NAMES"]=("Emby")
PROFILES["emby_EXCLUDE_DIRS"]=(
    "transcodes/" "logs/" "crash*" "cache/"
)

The `emby` Profile and Watch-State Sync

emby (Sunday 2:30am, both Emby instances stopped):

Syncs: everything except transcodes, logs, cache, crash files
Includes: metadata, plugins, full database state, all config
Why: WAL is checkpointed on clean shutdown — safe to copy everything
Result: HOST2 gets a gold-standard Emby state once per week

Between weekly mirrors, Emby watch/resume state is kept current by Media/play_state_sync.sh, which reads and writes play state over the Emby API every 30 minutes (CRITICAL_MAINTENANCE_SCRIPTS) — so HOST2 never falls more than one cycle behind on what's been watched. This replaced the old emby-fallback dirty-rsync profile: copying Emby's live SQLite databases mid-write risked landing a half-written WAL file on HOST2, and an API-level sync sidesteps the database-consistency problem entirely.

━━━ PERSONAL ENCRYPTED SHARES ━━━

Personal shares sync to the remote for offsite backup. ZFS encrypts at the dataset level — the remote receives encrypted blocks and cannot read the content without your passphrase or keyfile.

Create an Encrypted ZFS Dataset

# In the unRAID UI:
# Main → click your ZFS pool name → + Dataset
#   Name: Gmer4Lfe-Personal
#   Encryption: Enabled
#   Passphrase: [your passphrase]
#   Write your passphrase down — if lost, data is completely unrecoverable

# Verify encryption is active before syncing:
zfs get encryption poolname/Gmer4Lfe-Personal
# Should show: encryption  aes-256-gcm

Auto-Unlock on Boot (Optional)

# Keyfile approach — more convenient, but the keyfile itself is a secret
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
chmod 600 /root/.zfs-keys/personal.key

zfs change-key \
    -o keylocation=file:///root/.zfs-keys/personal.key \
    -o keyformat=raw \
    poolname/Gmer4Lfe-Personal

# Add to array_started.sh or ramdisk_setup.sh:
zfs load-key poolname/Gmer4Lfe-Personal
zfs mount poolname/Gmer4Lfe-Personal

# Manual unlock alternative (most secure):
zfs load-key poolname/Gmer4Lfe-Personal    # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal

Add to host1.conf

HOST1_PERSONAL_SHARES=(
    "/mnt/user/Gmer4Lfe-Personal"
)

━━━ STEP 7 — SCHEDULER SETUP ━━━

All scheduling is managed by the Varaverk plugin — no User Scripts entries needed.

Array start (Varaverk disks_mounted event hook)

The Varaverk plugin fires array_started.sh automatically via its disks_mounted event hook. It's registered in schedule.json:

"Orchestrators/array_started.sh": { "cron": "array_start", "enabled": true }

Launches everything in ARRAY_START_SCRIPTS: api_key_renew, inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup, docker_network_connect, fallback. Watchdogs run separately via watchdog_orchestrator.sh (cron — every 15 minutes).

Cron Schedule (Varaverk Scheduler)

*/7   * * * *   transcode_management.sh
*/15  * * * *   watchdog_orchestrator.sh
*/30  * * * *   critical_sync_maintenance.sh
0 */4  * * *    intermediate_sync_maintenance.sh
0  1   * * *    daily_sync_maintenance.sh
0  7   * * 0    sunday_morning_coffee_report.sh
30 2   * * 0    weekly_sync_maintenance.sh
0  0  15 * *    monthly_maintenance.sh

All entries are configured via the Varaverk scheduler UI or schedule.json. Scripts live at /boot/config/plugins/varaverk/.

━━━ VERIFICATION ━━━

Test with --dry-run first — all pre-flight checks run, no changes made.

Test a Single Profile Sync

bash /boot/config/plugins/varaverk/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log

Expected output (healthy):

━━━ Setup ━━━
  Host: HOST1 (unRAID-Gmer4Lfe) → HOST2 (unRAID-Jayred365)
  Remote IP: 100.x.x.x
  Profile: arrs_stack

━━━ Pre-flight ━━━
  Remote reachable
  version parity — both on unRAID X.Y.Z
  Remote Docker daemon responding
  Remote rootfs: 12% (threshold: 75%)
  Remote share exists and not empty
  All pre-flight checks passed

Test the Daily Orchestrator

bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run

Check Configuration Resolution

bash /boot/config/plugins/varaverk/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Arrs_Stack --status

━━━ INITIAL HOST2 SYNC ━━━

If HOST2 is being set up from scratch with empty shares:

# Create share directories on HOST2:
bash /boot/config/plugins/varaverk/Tools/recreate_shares.sh

# Initial push from HOST1 — first run may take several hours for large libraries:
bash /boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh --log

The scheduled nightly sync will be incremental after the initial push.

━━━ NAMING CONSISTENCY REQUIREMENT ━━━

The ecosystem uses one codebase on both servers. This only works if containers and shares have identical names on both servers. This is not configurable — it is a design requirement.

Container names must match exactly on both servers:
  "Emby"                ← both HOST1 and HOST2
  "NginxProxyManager"   ← both HOST1 and HOST2
  "Mariadb-Authelia"    ← both HOST1 and HOST2

Share paths must match exactly on both servers:
  /mnt/user/Movies      ← both HOST1 and HOST2
  /mnt/user/Tv_Shows    ← both HOST1 and HOST2

If a container has a different name on one server: the script skips it silently. You only notice when the container is not stopped during a sync that requires it.

If a share has a different path: rsync.sh aborts with "remote share missing." Easier to catch — but still requires renaming the share to fix.

━━━ TROUBLESHOOTING ━━━

SSH Connection Refused / Timeout

# Verify SSH is enabled on the remote:
#   Settings → Management Access → Secure Shell → Enabled

# Verify Tailscale is connected:
tailscale ip -4 unRAID-Jayred365

# Test SSH manually (key path from --status output):
ssh -i /root/.ssh/gmer4lfe_rsync_automation \
    root@$(tailscale ip -4 unRAID-Jayred365) "hostname"
# If password prompted: key not authorised — re-run ssh_setup.sh

# Re-run setup (idempotent, re-copies key to remote):
bash /boot/config/plugins/varaverk/Partnership/ssh_setup.sh

Pre-flight Aborts on Remote Rootfs

# Check current usage on remote:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "df /"

# Common cause: array not started, drives not mounted

# Verify the share exists on HOST2:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "ls /mnt/user/"

# If missing: create the share on HOST2, then run initial sync

Containers Not Stopping / Starting

# Verify container names match Docker exactly — case sensitive:
docker ps --format "{{.Names}}"

# Test remote:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Names}}'"

Profile Not Matching

# Profile key = directory basename lowercased
# /mnt/user/appdata-Fallback/Arrs_Stack → key: arrs_stack

# Override explicitly:
bash rsync.sh /mnt/user/appdata-Fallback/My_Stuff --profile=arrs_stack

# Verify what profile resolved:
bash rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status

━━━ FULL CONFIGURATION REFERENCE ━━━

master.conf

# Rsync engine
RSYNC_ENABLED=true
DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace"
# No --delete in DEFAULT_RSYNC_OPTS — media share rsync is additive only.
# arr_sync.sh keeps all arr databases in union — either server can download anything.
# arr cleanup handles deletions (orphans only). Profiles use --delete explicitly.
BW_LIMIT=0                  # KB/s, 0 = unlimited
RETRY_COUNT=3
SLEEP=60                    # seconds between retries
ROOTFS_WARN=75              # abort if remote rootfs above this %

# Shared with Monitors/
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90
BANDWIDTH_WARN_GB=50

# Profile definitions (see RSYNC PROFILES section above)
declare -A PROFILES
declare -A PROFILE_BW_LIMIT
declare -A PROFILE_RETRY_COUNT
declare -A PROFILE_SLEEP
declare -A PROFILE_CONTAINER_DELAY
declare -A PROFILE_CRITICAL_CONTAINER_NAMES
declare -A PROFILE_DELAYED_CONTAINERS
declare -A PROFILE_EXCLUDE_DIRS
declare -A PROFILE_REMOTE_RESTART_CONTAINERS

host*.conf

# Daily sync shares — one list per server (mutually exclusive)
HOST1_DAILY_SYNC_SHARES=(...)
HOST2_DAILY_SYNC_SHARES=(...)

# Personal encrypted shares
HOST1_PERSONAL_SHARES=(...)

# SSH key for this server to authenticate to the remote
# Set automatically by ssh_setup.sh — do not edit manually
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"

━━━ OUTPUT TIERS ━━━

Without --log, each run shows: section headers (Pre-flight, Transfer, Stop/Start Containers), the transfer identity block (source, remote, profile, identity), a progress indicator while rsync is running, and a summary block with duration, bytes transferred, and status. Warnings and errors are always visible.

With --log, every decision is shown — pre-flight check results, profile resolution, per-container stop/start state, retry details, and bandwidth log confirmation. Use for first-run validation or when investigating unexpected behavior.

━━━ FLAG REFERENCE ━━━

All flags work on rsync.sh and all orchestrators.

--dry-run

Runs all pre-flight checks. Shows what rsync would transfer. No transfer, no container stops, no bandwidth log entry. Safe to run at any time.

rsync.sh /mnt/user/Movies --dry-run
rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log
daily_sync_maintenance.sh --dry-run

--status

Shows resolved configuration — profile, remote identity, all vars that would be used — then exits. No pre-flight checks, no rsync. Use to verify configuration loaded correctly.

rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status

--log

Verbose output throughout. Every decision, every container operation, every rsync progress line. Use for first-time runs or when investigating issues.

--profile=name

Override profile selection. Bypasses basename inference. Use when the directory name doesn't match any profile key, or when testing a specific profile.

rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
rsync.sh /mnt/user/Media_Server/Emby --profile=emby

--seed

Skip the empty-remote-share guard. Normally check_remote_share() aborts when the target directory on the remote is missing or empty, because that usually means the array is not mounted and syncing into it would write onto the underlying root filesystem. --seed is the deliberate exception, for the first push of a genuinely new share.

rsync.sh /mnt/user/New_Share --seed

--merge-run

Bidirectional merge. Two passes:

  Pass 1   pull   remote → local, with --ignore-existing
  Pass 2   push   local → remote, with --delete

Local ends up authoritative: the remote may contribute files the local lacks, but never a competing version of a file the local already has. Also triggered automatically when the pre-scan detects ≥75% directory overlap with the remote.

--delete is applied only if pass 1 completed. Its entire justification is "local is now the authoritative superset", and a pull that was capped by RSYNC_MAX_RUNTIME_HOURS or failed outright means it is not — the remote still holds content the local never received. On an incomplete pull the push runs without --delete, warns, and notifies. Local content still propagates; nothing remote-unique is destroyed; the delete happens on a later run whose pull actually succeeded.

This was a real defect, fixed 2026-08-01. --delete used to be unconditional, so a capped or failed pull would permanently delete remote content the pull had not yet retrieved — and the timeout branch's own comment promised it would "resume the pull next run", which was impossible once the data was gone. If you are changing this function, that interlock is the part that matters.

━━━ SOURCE PATH GUARDS ━━━

rsync.sh pushes to root@remote at the same absolute path, and --merge-run adds --delete. The source argument is therefore a remote-side hazard, not just a local one, so two guards run before anything else:

Guard	Rejects
Absolute path, ≥3 components	`/`, `/mnt`, `/mnt/user`, empty
Directory must exist locally	typo'd paths

Every real job path clears both — see HOST*_DAILY_SYNC_SHARES and friends, all of which are /mnt/user/<Share> or deeper.

Without the existence check, a mistyped path under --merge-run would be created locally by the pull pass, filled from the remote, then pushed back with --delete.

━━━ RUNTIME CEILING ━━━

No single transfer attempt may exceed RSYNC_MAX_RUNTIME_HOURS (default 23). On timeout the transfer is terminated and resumes on the next scheduled run.

This is only safe because --partial is in DEFAULT_RSYNC_OPTS — a terminated transfer picks up where it left off rather than restarting. The cap exists so one huge or stuck transfer cannot hold its per-profile lock indefinitely and starve every other profile of a turn.

Both --merge-run passes are capped independently.

24 KiB Raw Permalink Blame History