Files

T

Gmer4Lfe 0fe99cf2a9 rename: appdata/varaverk → appdata/Varaverk (capital V throughout)

2026-05-30 10:22:51 -04:00

22 KiB

Raw Blame History

━━━━━ RSYNC — Manual ━━━━━

Config reference, procedures, operational workflows. For overview see README-Rsync.md. For per-script detail see the rsync.sh header.

This document also serves as the initial setup guide for standing up the two-server ecosystem from scratch.

━━━ WHAT YOU'RE BUILDING ━━━

HOST1 (unRAID-Gmer4Lfe)                    HOST2 (unRAID-Jayred365)
──────────────────────                     ──────────────────────
Downloads to:                              Downloads to:
  /mnt/user/Movies                           /mnt/user/Anime_Movies
  /mnt/user/Tv_Shows                         /mnt/user/Anime_Shows
  /mnt/user/Music

Both servers' arrs track ALL content across both servers.
It does not matter who downloaded what or when.

Daily 3-phase window (1am):

  Phase 1 — arr_sync.sh (runs first):
    All arrs on all nodes reconcile libraries using external IDs
    (TMDB, TVDB, MusicBrainz). Union model — any node that tracks
    an item, all nodes get it. After this phase both servers' arrs
    know about all content regardless of who downloaded it.

  Phase 2 — rsync (bidirectional, no --delete on media shares):
    HOST1 pushes its shares  ──────►  HOST2 receives files
    HOST1 receives files     ◄──────  HOST2 pushes its shares
    Files arrive already tracked by the remote arr (arr_sync ran first).
    No --delete — media files only spread. Arr cleanup handles deletions.

  Phase 3 — arr cleanup:
    sonarr_cleanup / radarr_cleanup / lidarr_cleanup query the live arr
    API and remove any files no longer tracked. Emby notified after.

Note: profiled syncs (arrs_stack, emby, critical-data…) use their own
PROFILE_RSYNC_OPTS which include --delete. Only the no-profile media
shares use DEFAULT_RSYNC_OPTS (spread only).

Weekly clean sync (2:30am Sunday, containers stopped):
  Emby — full clean mirror       ◄──────►  Emby
  Critical-Data                  ──────►   Auth stack (HOST1 → HOST2)

Every 30 minutes — dirty sync:
  Emby watch states              ──────►   HOST2 stays current on playback

━━━ PREREQUISITES ━━━

Required on both servers before starting:

unRAID 7.x
Community Applications plugin     — search "Community Applications" in unRAID plugins
User Scripts plugin               — install via Community Applications
Tailscale plugin                  — install via Community Applications
Terminal access                   — unRAID UI → Tools → Terminal, or SSH

Optional but recommended:

Gitea (Docker container on HOST1)  — self-hosted git for the script repository
Working Emby installation          — for transcode management and failover

━━━ STEP 1 — TAILSCALE ━━━

Tailscale provides the encrypted mesh network between servers. Scripts resolve the remote server's IP via Tailscale at runtime — no hardcoded IPs, no VPN configuration, no open ports. All server-to-server communication goes through Tailscale.

Install on Both Servers

1. Open Apps in the unRAID UI
2. Search "Tailscale" — install the plugin
3. Settings → Tailscale → Connect
4. Authenticate with your Tailscale account (browser opens on your machine)
5. Verify both servers appear: https://login.tailscale.com/admin/machines

Verify Connectivity

# From HOST1 — should return HOST2's 100.x.x.x Tailscale IP:
tailscale ip -4 unRAID-Jayred365

# Test actual connectivity:
tailscale ping unRAID-Jayred365

Critical: The hostnames in master.conf (HOST1 and HOST2) must match the Tailscale machine names exactly — case sensitive. All remote IP resolution goes through tailscale ip -4 HOSTNAME. A name mismatch causes every remote operation to fail at the IP resolution step.

━━━ STEP 2 — ENABLE SSH ━━━

unRAID 7.x has SSH disabled by default. Enable it on both servers.

Settings → Management Access → Secure Shell
  SSH: Enabled
  SSH port: 22
Apply

SSH is only exposed on your local network and Tailscale interface. No ports are opened to the public internet.

━━━ STEP 3 — SSH KEYS ━━━

Two sets of keys needed: rsync automation keys (server-to-server) and a Gitea access key (for script repository pulls).

3a — Rsync Automation Keys (ssh_setup.sh)

Run on each server. ssh_setup.sh generates the key, copies it to the remote, and updates host*.conf with the key path automatically.

# On HOST1 — after the repo is cloned:
bash /mnt/user/appdata/Varaverk/Partnership/ssh_setup.sh

# On HOST2:
bash /mnt/user/appdata/Varaverk/Partnership/ssh_setup.sh

Key naming convention: hostname lowercased, unraid- prefix stripped.

unRAID-Gmer4Lfe  → /root/.ssh/gmer4lfe_rsync_automation
unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation

host*.conf is updated automatically with HOST*_SSH_KEY pointing to the generated key. Run --status to verify:

bash /mnt/user/appdata/Varaverk/Partnership/ssh_setup.sh --status

ssh_setup.sh is idempotent — safe to re-run. Use --force to regenerate a key (e.g., after a security incident) and re-copy it to the remote.

3b — Gitea SSH Key (manual)

# On BOTH servers:
ssh-keygen -t ed25519 -f /root/.ssh/unraid_gitea -C "unraid-gitea" -N ""

# Print the public key to add to Gitea:
cat /root/.ssh/unraid_gitea.pub

# In Gitea: Settings → SSH / GPG Keys → Add Key → paste above

━━━ STEP 4 — CLONE THE REPOSITORY ━━━

# Create target directory:
mkdir -p /mnt/user/appdata/Varaverk

# Clone:
GIT_SSH_COMMAND="ssh -i /root/.ssh/unraid_gitea" \
    git clone git@YOUR_GITEA_HOST:Varaverk/varaverk.git \
    /mnt/user/appdata/Varaverk

# Make scripts executable:
find /mnt/user/appdata/Varaverk -name "*.sh" -exec chmod +x {} \;

Expected structure after clone:

master.conf          ← all shared configuration
host1.conf    ← HOST1-specific configuration
host2.conf    ← HOST2-specific configuration
common.sh            ← shared library
load_config.sh       ← config loader
Orchestrators/
Rsync/
Fallback/
Docker_Essentials/
...

━━━ STEP 5 — CONFIGURE MASTER.CONF ━━━

nano /mnt/user/appdata/Varaverk/master.conf

Host Identity

# These must match Tailscale machine names exactly — case sensitive.
HOST1="unRAID-Gmer4Lfe"       # REQUIRED
HOST2="unRAID-Jayred365"       # REQUIRED

# SSH keys — each server uses its own key to authenticate to the other:
HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key"
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"

Git Repository

GITEA_CONTAINER="Gitea"
GITEA_REPO_PATH="Varaverk/varaverk.git"
TARGET_DIR="/mnt/user/appdata/Varaverk"
GITEA_SSH_KEY="/root/.ssh/unraid_gitea"
SSH_PORT=221

Daily Sync Shares

# host1.conf
# Shares this server downloads to — pushed to the other server nightly.
# arr_sync.sh runs before rsync in the daily window, so the receiving
# server's arrs already track incoming files when they arrive.
# Never put the same path in both lists.
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"
    "/mnt/user/Tv_Shows"
    "/mnt/user/Music"
    "/mnt/user/Kids_Movies"
    "/mnt/user/Kids_Tv_Shows"
    "/mnt/user/Sports"
    "/mnt/user/stand-up_comedy"
)

# host2.conf
HOST2_DAILY_SYNC_SHARES=(
    "/mnt/user/Anime_Shows"
    "/mnt/user/Anime_Movies"
)

Weekly Sync Shares

# master.conf
# Synced during the Sunday 2:30am window — containers stopped both sides.
WEEKLY_SYNC_SHARES=(
    "/mnt/user/Media_Server/Emby"
    "/mnt/user/appdata-Fallback/Critical-Data"
)

━━━ STEP 6 — MASTER_HOST*.CONF ━━━

detect_hosts() reads which server is running and aliases HOST*_ prefixed vars to their unprefixed names. Scripts only ever reference the unprefixed name — they work identically on both servers.

nano /mnt/user/appdata/Varaverk/host1.conf    # on HOST1
nano /mnt/user/appdata/Varaverk/host2.conf    # on HOST2

Every variable is documented in the conf files. Key values to set:

SSH key paths
DAILY_SYNC_SHARES
EMBY_URL / EMBY_API_KEY
CERT_MONITOR_DOMAINS
SMART_IGNORE_DRIVES
ZFS_REPORT_IGNORE_POOLS

━━━ RSYNC PROFILES ━━━

Profiles control per-share behavior. Profile key = directory basename lowercased. Override with --profile=name.

Profile Matching

# rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack
#   basename: Arrs_Stack → lowercased: arrs_stack → matches [arrs_stack] profile
#
# rsync.sh /mnt/user/Movies
#   basename: Movies → no profile match → global defaults (no containers stopped)
#
# rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
#   explicit override

Current Profile Definitions

# master.conf

# ── arrs_stack ───────────────────────────────────────────────────────────────
# Arr databases — stopped for clean SQLite snapshot
PROFILES["arrs_stack_CRITICAL_CONTAINER_NAMES"]=(
    "Sonarr" "Radarr" "Lidarr" "Prowlarr" "Bazarr" "Pinchflat"
)

# ── critical-data ─────────────────────────────────────────────────────────────
# Auth stack — stopped for clean database snapshot, Authelia delayed restart
PROFILES["critical-data_CRITICAL_CONTAINER_NAMES"]=(
    "Mariadb-Authelia" "Redis-Authelia"
    "NginxProxyManager" "Lldap-Gmer4Lfe"
)
PROFILES["critical-data_DELAYED_CONTAINERS"]=(
    "Authelia" "Authelia-Secondary"
)
PROFILES["critical-data_CONTAINER_DELAY"]=30

# ── important-data ────────────────────────────────────────────────────────────
# NextCloud + Postgres — stopped for clean snapshot
PROFILES["important-data_CRITICAL_CONTAINER_NAMES"]=(
    "Postgres-NextCloud"
)
PROFILES["important-data_DELAYED_CONTAINERS"]=("NextCloud")

# ── emby ──────────────────────────────────────────────────────────────────────
# Weekly full clean sync — both Emby instances stopped
PROFILES["emby_CRITICAL_CONTAINER_NAMES"]=("Emby")
PROFILES["emby_EXCLUDE_DIRS"]=(
    "transcodes/" "logs/" "crash*" "cache/"
)

# ── emby-fallback ─────────────────────────────────────────────────────────────
# Every 30 minutes, Emby STAYS RUNNING — dirty sync of critical state only
PROFILES["emby-fallback_CRITICAL_CONTAINER_NAMES"]=()   # nothing stops
PROFILES["emby-fallback_EXCLUDE_DIRS"]=(
    "*.wal" "*.shm"                            # unsafe mid-write
    "transcodes/" "logs/" "crash*" "cache/"
)
PROFILES["emby-fallback_REMOTE_RESTART_CONTAINERS"]=("Emby")

Two Emby Profiles — Why Both Exist

emby-fallback (every 30 minutes, Emby stays running):

Syncs: users.db, library.db, authentication.db, config/
Skips: *.wal, *.shm, transcodes/, logs/, cache/
Why: WAL files are written while Emby runs — copying them would produce a corrupt database on HOST2
Result: HOST2 is always within 30 minutes of HOST1 on watch state and user activity

emby (Sunday 2:30am, both Emby instances stopped):

Syncs: everything except transcodes, logs, cache, crash files
Includes: metadata, plugins, full database state, all config
Why: WAL is checkpointed on clean shutdown — safe to copy everything
Result: HOST2 gets a gold-standard Emby state once per week

The two profiles work together. emby-fallback keeps HOST2 current for immediate failover. emby gives HOST2 full fidelity once per week. Neither alone is sufficient.

━━━ PERSONAL ENCRYPTED SHARES ━━━

Personal shares sync to the remote for offsite backup. ZFS encrypts at the dataset level — the remote receives encrypted blocks and cannot read the content without your passphrase or keyfile.

Create an Encrypted ZFS Dataset

# In the unRAID UI:
# Main → click your ZFS pool name → + Dataset
#   Name: Gmer4Lfe-Personal
#   Encryption: Enabled
#   Passphrase: [your passphrase]
#   Write your passphrase down — if lost, data is completely unrecoverable

# Verify encryption is active before syncing:
zfs get encryption poolname/Gmer4Lfe-Personal
# Should show: encryption  aes-256-gcm

Auto-Unlock on Boot (Optional)

# Keyfile approach — more convenient, but the keyfile itself is a secret
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
chmod 600 /root/.zfs-keys/personal.key

zfs change-key \
    -o keylocation=file:///root/.zfs-keys/personal.key \
    -o keyformat=raw \
    poolname/Gmer4Lfe-Personal

# Add to array_started.sh or ramdisk_setup.sh:
zfs load-key poolname/Gmer4Lfe-Personal
zfs mount poolname/Gmer4Lfe-Personal

# Manual unlock alternative (most secure):
zfs load-key poolname/Gmer4Lfe-Personal    # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal

Add to host1.conf

HOST1_PERSONAL_SHARES=(
    "/mnt/user/Gmer4Lfe-Personal"
)

━━━ STEP 7 — USER SCRIPTS SETUP ━━━

Only a small number of User Scripts entries are needed — each one an orchestrator. Individual scripts are never scheduled directly except the 30-minute Emby sync.

At Startup of Array

#!/bin/bash
bash /mnt/user/appdata/Varaverk/Orchestrators/array_started.sh
# Schedule: At Startup of Array
# Run as: Background Task

This single entry launches everything defined in ARRAY_START_SCRIPTS from master.conf: inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup, docker_network_connect, fallback. Watchdogs (resource, docker, system, stability) run separately via watchdog_orchestrator.

Cron Schedule

# Every 30 minutes — Emby dirty sync:
*/30 * * * *
bash /mnt/user/appdata/Varaverk/Rsync/rsync.sh \
    /mnt/user/Media_Server/Emby --profile=emby-fallback

# Every 6 hours — failed import + stalled download recovery:
0 */6 * * *
bash /mnt/user/appdata/Varaverk/Orchestrators/arrs_failed_stalled_recovery.sh

# 1am daily — full maintenance window:
0 1 * * *
bash /mnt/user/appdata/Varaverk/Orchestrators/daily_sync_maintenance.sh

# 2:30am Sunday — weekly maintenance window:
30 2 * * 0
bash /mnt/user/appdata/Varaverk/Orchestrators/weekly_sync_maintenance.sh

# 8am daily — health digest:
0 8 * * *
bash /mnt/user/appdata/Varaverk/Monitors/weekly_health_digest.sh

# Every 6 hours — inotify + php-fpm snapshot:
0 */6 * * *
bash /mnt/user/appdata/Varaverk/Monitors/system_tuning_monitor.sh

# Sunday morning — weekly reports:
0  6 * * 0   bash .../Monitors/zfs_memory_snapshot.sh
0  7 * * 0   bash .../Monitors/smart_health.sh
0  9 * * 0   bash .../Monitors/cert_monitor.sh
0 10 * * 0   bash .../Monitors/backup_verify.sh
0 11 * * 0   bash .../Monitors/emby_session_report.sh
0 11 * * 0   bash .../Monitors/bandwidth_monitor.sh --report

Set all entries to Background Task — non-background tasks can appear to hang on long-running scripts.

━━━ VERIFICATION ━━━

Test with --dry-run first — all pre-flight checks run, no changes made.

Test a Single Profile Sync

bash /mnt/user/appdata/Varaverk/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log

Expected output (healthy):

━━━ Setup ━━━
  Host: HOST1 (unRAID-Gmer4Lfe) → HOST2 (unRAID-Jayred365)
  Remote IP: 100.x.x.x
  Profile: arrs_stack

━━━ Pre-flight ━━━
  Remote reachable
  version parity — both on unRAID X.Y.Z
  Remote Docker daemon responding
  Remote rootfs: 12% (threshold: 75%)
  Remote share exists and not empty
  All pre-flight checks passed

Test the Daily Orchestrator

bash /mnt/user/appdata/Varaverk/Orchestrators/daily_sync_maintenance.sh --dry-run

Check Configuration Resolution

bash /mnt/user/appdata/Varaverk/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Arrs_Stack --status

━━━ INITIAL HOST2 SYNC ━━━

If HOST2 is being set up from scratch with empty shares:

# Create share directories on HOST2:
bash /mnt/user/appdata/Varaverk/Tools/recreate_shares.sh

# Initial push from HOST1 — first run may take several hours for large libraries:
bash /mnt/user/appdata/Varaverk/Orchestrators/daily_sync_maintenance.sh --log

The scheduled nightly sync will be incremental after the initial push.

━━━ NAMING CONSISTENCY REQUIREMENT ━━━

The ecosystem uses one codebase on both servers. This only works if containers and shares have identical names on both servers. This is not configurable — it is a design requirement.

Container names must match exactly on both servers:
  "Emby"                ← both HOST1 and HOST2
  "NginxProxyManager"   ← both HOST1 and HOST2
  "Mariadb-Authelia"    ← both HOST1 and HOST2

Share paths must match exactly on both servers:
  /mnt/user/Movies      ← both HOST1 and HOST2
  /mnt/user/Tv_Shows    ← both HOST1 and HOST2

If a container has a different name on one server: the script skips it silently. You only notice when the container is not stopped during a sync that requires it.

If a share has a different path: rsync.sh aborts with "remote share missing." Easier to catch — but still requires renaming the share to fix.

━━━ TROUBLESHOOTING ━━━

SSH Connection Refused / Timeout

# Verify SSH is enabled on the remote:
#   Settings → Management Access → Secure Shell → Enabled

# Verify Tailscale is connected:
tailscale ip -4 unRAID-Jayred365

# Test SSH manually (key path from --status output):
ssh -i /root/.ssh/gmer4lfe_rsync_automation \
    root@$(tailscale ip -4 unRAID-Jayred365) "hostname"
# If password prompted: key not authorised — re-run ssh_setup.sh

# Re-run setup (idempotent, re-copies key to remote):
bash /mnt/user/appdata/Varaverk/Partnership/ssh_setup.sh

Pre-flight Aborts on Remote Rootfs

# Check current usage on remote:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "df /"

# Common cause: array not started, drives not mounted

# Verify the share exists on HOST2:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "ls /mnt/user/"

# If missing: create the share on HOST2, then run initial sync

Containers Not Stopping / Starting

# Verify container names match Docker exactly — case sensitive:
docker ps --format "{{.Names}}"

# Test remote:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Names}}'"

Profile Not Matching

# Profile key = directory basename lowercased
# /mnt/user/appdata-Fallback/Arrs_Stack → key: arrs_stack

# Override explicitly:
bash rsync.sh /mnt/user/appdata-Fallback/My_Stuff --profile=arrs_stack

# Verify what profile resolved:
bash rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status

━━━ FULL CONFIGURATION REFERENCE ━━━

master.conf

# Rsync engine
RSYNC_ENABLED=true
DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace"
# No --delete in DEFAULT_RSYNC_OPTS — media share rsync is additive only.
# arr_sync.sh keeps all arr databases in union — either server can download anything.
# arr cleanup handles deletions (orphans only). Profiles use --delete explicitly.
BW_LIMIT=0                  # KB/s, 0 = unlimited
RETRY_COUNT=3
SLEEP=60                    # seconds between retries
ROOTFS_WARN_PCT=75          # abort if remote rootfs above this %

# Shared with Monitors/
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90
BANDWIDTH_WARN_GB=50

# Profile definitions (see RSYNC PROFILES section above)
declare -A PROFILES
declare -A PROFILE_BW_LIMIT
declare -A PROFILE_RETRY_COUNT
declare -A PROFILE_SLEEP
declare -A PROFILE_CONTAINER_DELAY
declare -A PROFILE_CRITICAL_CONTAINER_NAMES
declare -A PROFILE_DELAYED_CONTAINERS
declare -A PROFILE_EXCLUDE_DIRS
declare -A PROFILE_REMOTE_RESTART_CONTAINERS

host*.conf

# Daily sync shares — one list per server (mutually exclusive)
HOST1_DAILY_SYNC_SHARES=(...)
HOST2_DAILY_SYNC_SHARES=(...)

# Personal encrypted shares
HOST1_PERSONAL_SHARES=(...)

# SSH key for this server to authenticate to the remote
# Set automatically by ssh_setup.sh — do not edit manually
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"

━━━ OUTPUT TIERS ━━━

Without --log, each run shows: section headers (Pre-flight, Transfer, Stop/Start Containers), the transfer identity block (source, remote, profile, identity), a progress indicator while rsync is running, and a summary block with duration, bytes transferred, and status. Warnings and errors are always visible.

With --log, every decision is shown — pre-flight check results, profile resolution, per-container stop/start state, retry details, and bandwidth log confirmation. Use for first-run validation or when investigating unexpected behavior.

━━━ FLAG REFERENCE ━━━

All flags work on rsync.sh and all orchestrators.

--dry-run

Runs all pre-flight checks. Shows what rsync would transfer. No transfer, no container stops, no bandwidth log entry. Safe to run at any time.

rsync.sh /mnt/user/Movies --dry-run
rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log
daily_sync_maintenance.sh --dry-run

--status

Shows resolved configuration — profile, remote identity, all vars that would be used — then exits. No pre-flight checks, no rsync. Use to verify configuration loaded correctly.

rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status

--log

Verbose output throughout. Every decision, every container operation, every rsync progress line. Use for first-time runs or when investigating issues.

--profile=name

Override profile selection. Bypasses basename inference. Use when the directory name doesn't match any profile key, or when testing a specific profile.

rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback

22 KiB Raw Blame History