FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
09e9cec0c3395779f44427a86643225915171e14
Varaverk/Rsync/README-Rsync_Setup.md
T

47 KiB
Raw Blame History

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔄 RSYNC SETUP GUIDE

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Complete setup guide for the two-server rsync ecosystem. By the end of this guide both servers will have SSH keys configured, Tailscale connected, the git repository cloned, and all scheduled operations running automatically.

This is a setup guide, not a script reference. For how rsync.sh works internally, profiles, safety checks, and operational details — those belong in the orchestrator and rsync script documentation. This guide is about standing the ecosystem up from scratch and verifying it works.

━━━ WHAT YOU'RE BUILDING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HOST1 (unRAID-Gmer4Lfe)                    HOST2 (unRAID-Jayred365)
──────────────────────                     ──────────────────────
Source of truth:                           Source of truth:
  Movies, Tv_Shows, Music                    Anime_Shows, Anime_Movies
  Critical-Data (auth stack)
  Emby userdata

Pushes to HOST2 daily:           ──────→   Receives:
  All HOST1 shares                            Mirror of HOST1 shares
  Personal encrypted shares                   Personal (encrypted blocks)

Receives from HOST2 daily:       ←──────   Pushes:
  Anime_Shows, Anime_Movies                   All HOST2 shares

Weekly clean sync (both sides stopped):
  Emby — full clean mirror       ←──────→  Emby
  Critical-Data                  ──────→   Auth stack (HOST1 → HOST2)

Every 30 minutes — dirty sync:
  Emby watch states              ──────→   HOST2 stays current on playback

━━━ PREREQUISITES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Both servers need these before starting:

# ─────────────────────────────────────────────────────────────────────────────
# Required on both servers:
  unRAID 7.x
  Community Applications plugin     — search "Community Applications" in unRAID plugins
  User Scripts plugin               — install via Community Applications
  Tailscale plugin                  — install via Community Applications
  Terminal access                   — unRAID UI → Tools → Terminal, or SSH

# Optional but recommended:
  Gitea (Docker container on HOST1)  — self-hosted git for the script repository
  Working Emby installation          — for transcode management and failover
# ─────────────────────────────────────────────────────────────────────────────

━━━ STEP 1 — TAILSCALE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Tailscale provides the encrypted mesh network between servers. Scripts resolve the remote server's IP via Tailscale at runtime — no hardcoded IPs, no VPN configuration, no open ports. All server-to-server communication goes through Tailscale.

── Install on Both Servers ────────────────────────────────────────────────

# On each server:
# ─────────────────────────────────────────────────────────────────────────────
# 1. Open Apps in the unRAID UI
# 2. Search for "Tailscale" — install the plugin
# 3. Settings → Tailscale → Connect
# 4. Authenticate with your Tailscale account (browser opens on your machine)
# 5. Verify both servers appear: https://login.tailscale.com/admin/machines
# ─────────────────────────────────────────────────────────────────────────────

── Verify Connectivity ─────────────────────────────────────────────────────

# From HOST1 — should return HOST2's 100.x.x.x Tailscale IP:
tailscale ip -4 unRAID-Jayred365

# From HOST2 — should return HOST1's 100.x.x.x Tailscale IP:
tailscale ip -4 unRAID-Gmer4Lfe

# Test actual connectivity:
tailscale ping unRAID-Jayred365    # run from HOST1

Critical: The hostnames in master.conf (HOST1 and HOST2) must match the Tailscale machine names exactly — case sensitive. The ecosystem resolves all remote IPs via tailscale ip -4 HOSTNAME at runtime. A name mismatch means every script that touches the remote will fail at the IP resolution step.

━━━ STEP 2 — ENABLE SSH ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

unRAID 7.x has SSH disabled by default. Enable it on both servers.

# On each server:
# ─────────────────────────────────────────────────────────────────────────────
# Settings → Management Access → Secure Shell
#   SSH: Enabled
#   SSH port: 22
# Apply
# ─────────────────────────────────────────────────────────────────────────────

SSH is only exposed on your local network and Tailscale interface. Scripts connect via Tailscale IP — all traffic is encrypted end-to-end. No ports are opened to the public internet.

━━━ STEP 3 — SSH KEYS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Two sets of keys needed: server-to-server for rsync and failover, and Gitea access for script repository pull. Generate all keys before configuring anything else.

── 3a — Server-to-Server Keys ─────────────────────────────────────────────

# On HOST1 — generate HOST1's key pair:
ssh-keygen -t ed25519 -f /root/.ssh/Gmer4Lfe-rsync-key -C "gmer4lfe-rsync" -N ""

# On HOST2 — generate HOST2's key pair:
ssh-keygen -t ed25519 -f /root/.ssh/Jayred365-rsync-key -C "jayred365-rsync" -N ""

── 3b — Authorise Keys Bidirectionally ─────────────────────────────────────

# HOST1's public key must be authorised on HOST2 (so HOST1 can SSH into HOST2):
# ─────────────────────────────────────────────────────────────────────────────

# On HOST1 — print the public key:
cat /root/.ssh/Gmer4Lfe-rsync-key.pub

# On HOST2 — create authorized_keys and paste HOST1's public key:
mkdir -p /root/.ssh
echo "PASTE_HOST1_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

# HOST2's public key must be authorised on HOST1 (so HOST2 can SSH into HOST1):
# ─────────────────────────────────────────────────────────────────────────────

# On HOST2 — print the public key:
cat /root/.ssh/Jayred365-rsync-key.pub

# On HOST1 — append HOST2's public key:
echo "PASTE_HOST2_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys

── 3c — Test Both Directions ────────────────────────────────────────────────

# From HOST1 — should print "connected" without a password prompt:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key \
    root@$(tailscale ip -4 unRAID-Jayred365) \
    "echo connected"

# From HOST2 — should print "connected" without a password prompt:
ssh -i /root/.ssh/Jayred365-rsync-key \
    root@$(tailscale ip -4 unRAID-Gmer4Lfe) \
    "echo connected"

If prompted for a password: the key was not authorised correctly.
  → Recheck Step 3b — the public key content must be on one line
  → Check permissions: chmod 600 /root/.ssh/authorized_keys
  → Check the key file referenced in the SSH command matches what was generated

── 3d — Gitea SSH Key ───────────────────────────────────────────────────────

# On BOTH servers — generate a key for Gitea access:
ssh-keygen -t ed25519 -f /root/.ssh/unraid_gitea -C "unraid-gitea" -N ""

# Print the public key to add to Gitea:
cat /root/.ssh/unraid_gitea.pub

# In Gitea: Settings → SSH / GPG Keys → Add Key → paste the output above
# Do this for both servers if they have separate Gitea accounts, or once if shared

━━━ STEP 4 — CLONE THE REPOSITORY ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Both servers clone from the same Gitea repository. Updates pushed to the repo propagate to both servers automatically via git_pull_execute.sh at the start of each daily maintenance window.

── On Both Servers ─────────────────────────────────────────────────────────

# Create the target directory:
mkdir -p /mnt/user/appdata/unraid_scripts

# Clone the repository:
GIT_SSH_COMMAND="ssh -i /root/.ssh/unraid_gitea" \
    git clone git@YOUR_GITEA_HOST:FailedProxy/Unraid_Scripts.git \
    /mnt/user/appdata/unraid_scripts

# Replace YOUR_GITEA_HOST with your Gitea server address and port
# Example: git@192.168.50.2:221

── Verify the Structure ─────────────────────────────────────────────────────

ls /mnt/user/appdata/unraid_scripts/

Expected output:
  master.conf          ← all user configuration — the only file you edit
  master_host1.conf    ← HOST1-specific configuration
  master_host2.conf    ← HOST2-specific configuration
  common.sh            ← shared library — functions used by all scripts
  load_config.sh       ← config loader
  Orchestrators/
  Rsync/
  Failover/
  Docker_Essentials/
  unRAID_Essentials/
  Media/
  Transcodes/
  Monitors/
  Tools/
  Partnership/

── Make Scripts Executable ─────────────────────────────────────────────────

# Execute permission on all scripts — required once after clone:
find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \;

array_start.sh auto-fixes permissions on scripts that lost the execute bit — but this initial chmod ensures the first run works before that safeguard is active.

━━━ STEP 5 — CONFIGURE MASTER.CONF ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

All user configuration lives in master.conf. Every value with a comment # REQUIRED must be set before the first run. Everything else has working defaults.

nano /mnt/user/appdata/unraid_scripts/master.conf

── Host Identity ─────────────────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# These must match Tailscale machine names exactly — case sensitive.
# The ecosystem uses these to resolve remote IPs at runtime.
#
    HOST1="unRAID-Gmer4Lfe"       # REQUIRED — must match tailscale machine name
    HOST2="unRAID-Jayred365"       # REQUIRED — same

# SSH key paths — each server's key for authenticating to the other:
    HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key"   # HOST1 uses this to SSH to HOST2
    HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"   # HOST2 uses this to SSH to HOST1

── Emby API Keys ─────────────────────────────────────────────────────────────

# master_host1.conf
# ─────────────────────────────────────────────────────────────────────────────
# Used by: emby_session_report.sh, sunday_morning_coffee_report.sh,
#          sonarr/radarr cleanup (notify_emby_scan after deletion)
#
# Get from: Emby Dashboard → Settings → API Keys → + New API Key
#
    HOST1_EMBY_URL="http://192.168.50.2:8096"
    HOST1_EMBY_API_KEY="your-host1-emby-api-key"    # REQUIRED for Emby features
    HOST1_EMBY_CONTAINER="Emby"

# master_host2.conf
    HOST2_EMBY_URL="http://localhost:8096"
    HOST2_EMBY_API_KEY="your-host2-emby-api-key"
    HOST2_EMBY_CONTAINER="Emby"

── Git Repository ────────────────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Used by git_pull_execute.sh — pulls latest scripts at start of each daily window.
#
    GITEA_CONTAINER="Gitea"                              # exact Docker container name
    GITEA_REPO_PATH="FailedProxy/Unraid_Scripts.git"
    TARGET_DIR="/mnt/user/appdata/unraid_scripts"
    GITEA_SSH_KEY="/root/.ssh/unraid_gitea"
    SSH_PORT=221                                          # your Gitea SSH port

── Daily Sync Shares ────────────────────────────────────────────────────────

# master_host1.conf
# ─────────────────────────────────────────────────────────────────────────────
# Shares HOST1 is source of truth for — pushed to HOST2 every night at 1am.
# HOST2 treats these as read-only mirrors. Never put the same share in both lists.
#
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"           # HOST1 manages this — Radarr runs here
    "/mnt/user/Tv_Shows"         # HOST1 manages this — Sonarr runs here
    "/mnt/user/Music"            # HOST1 manages this — Lidarr runs here
    "/mnt/user/Kids_Movies"
    "/mnt/user/Kids_Tv_Shows"
    "/mnt/user/Sports"
    "/mnt/user/stand-up_comedy"
)

# master_host2.conf
HOST2_DAILY_SYNC_SHARES=(
    "/mnt/user/Anime_Shows"      # HOST2 manages this — his Sonarr runs here
    "/mnt/user/Anime_Movies"     # HOST2 manages this — his Radarr runs here
)

── Weekly Sync Shares ───────────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Synced during the Sunday 2:30am window — containers stopped both sides.
# Do NOT add these to a separate cron schedule — they run via weekly_sync_maintenance.sh.
#
WEEKLY_SYNC_SHARES=(
    "/mnt/user/Media_Server/Emby"                   # full clean Emby mirror
    "/mnt/user/appdata-Failover/Critical-Data"      # auth stack clean state
)

━━━ STEP 6 — MASTER_HOST*.CONF ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Per-host configuration lives in master_host1.conf and master_host2.conf. detect_hosts() in common.sh reads which server is running and aliases the correct HOST*_ prefixed variables to their unprefixed names. Scripts only ever reference the unprefixed name — they work identically on both servers.

# master_host1.conf is only sourced on HOST1
# master_host2.conf is only sourced on HOST2
# Changes go in the right file for the right server

nano /mnt/user/appdata/unraid_scripts/master_host1.conf    # on HOST1
nano /mnt/user/appdata/unraid_scripts/master_host2.conf    # on HOST2

See each conf file's comments — every variable is documented with its purpose and the reasoning behind the value.

━━━ STEP 7 — RSYNC PROFILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Profiles control per-share behaviour — which containers to stop, rsync flags, bandwidth limits, what to exclude. Profile is matched by directory basename (lowercased). Override with --profile=name.

── How Profile Matching Works ──────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
#   basename: Arrs_Stack
#   lowercased: arrs_stack
#   matched profile: [arrs_stack]
#
# rsync.sh /mnt/user/Movies
#   basename: Movies
#   lowercased: movies
#   no matching profile → global defaults apply (no containers stopped)
#
# rsync.sh /mnt/user/appdata-Failover/Critical-Data --profile=critical-failover
#   explicit override → uses [critical-failover] profile regardless of path
# ─────────────────────────────────────────────────────────────────────────────

── Current Profiles ─────────────────────────────────────────────────────────

# master.conf — profile definitions
# ─────────────────────────────────────────────────────────────────────────────
# Each profile defines which containers to stop, rsync flags, excludes, etc.
# Containers in PROFILE_CRITICAL_CONTAINER_NAMES are stopped on BOTH servers.
# PROFILE_DELAYED_CONTAINERS restart after PROFILE_CONTAINER_DELAY seconds.

# ── arrs_stack ─────────────────────────────────────────────────────────────
# Arr databases — stopped for clean SQLite snapshot
    PROFILES["arrs_stack_CRITICAL_CONTAINER_NAMES"]=(
        "Sonarr" "Radarr" "Lidarr" "Prowlarr" "Bazarr" "Pinchflat"
    )

# ── critical-data ──────────────────────────────────────────────────────────
# Auth stack — stopped for clean database snapshot, delayed restart
    PROFILES["critical-data_CRITICAL_CONTAINER_NAMES"]=(
        "Mariadb-Authelia" "Redis-Authelia"
        "NginxProxyManager" "Lldap-Gmer4Lfe"
    )
    PROFILES["critical-data_DELAYED_CONTAINERS"]=(
        "Authelia" "Authelia-Secondary"    # auth services restart after delay
    )
    PROFILES["critical-data_CONTAINER_DELAY"]=30    # seconds before delayed containers start

# ── important-data ─────────────────────────────────────────────────────────
# NextCloud + Postgres — stopped for clean snapshot
    PROFILES["important-data_CRITICAL_CONTAINER_NAMES"]=(
        "Postgres-NextCloud"
    )
    PROFILES["important-data_DELAYED_CONTAINERS"]=("NextCloud")

# ── emby ───────────────────────────────────────────────────────────────────
# Weekly full clean sync — both Emby instances stopped
    PROFILES["emby_CRITICAL_CONTAINER_NAMES"]=("Emby")
    PROFILES["emby_EXCLUDE_DIRS"]=(
        "transcodes/" "logs/" "crash*" "cache/"
    )

# ── emby-failover ──────────────────────────────────────────────────────────
# Every 30 minutes, Emby STAYS RUNNING — dirty sync of critical state only
# WAL and SHM excluded — safe to copy while Emby is writing
    PROFILES["emby-failover_CRITICAL_CONTAINER_NAMES"]=()    # empty — nothing stops
    PROFILES["emby-failover_EXCLUDE_DIRS"]=(
        "*.wal" "*.shm"                            # WAL files — unsafe mid-write
        "transcodes/" "logs/" "crash*" "cache/"    # volatile data — skip
    )
    PROFILES["emby-failover_REMOTE_RESTART_CONTAINERS"]=("Emby")
    # Emby on HOST2 restarts after sync to pick up config changes

── Two Emby Profiles — Why Both Exist ──────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# emby-failover — every 30 minutes, Emby stays running:
#   What syncs:   users.db, library.db, authentication.db, config/
#   What skips:   *.wal *.shm transcodes/ logs/ cache/
#   Why:          WAL files are being written while Emby runs — copying them
#                 would produce a corrupt database on HOST2
#   Result:       HOST2 is always within 30 minutes of HOST1 on watch state
#                 and user activity. Failover is seamless — nobody notices.
#
# emby — Sunday 2:30am, both Emby instances stopped:
#   What syncs:   everything except transcodes, logs, cache, crash files
#   What includes: metadata, plugins, full database state, all config
#   Why:          WAL is checkpointed on clean shutdown — safe to copy everything
#                 Full consistent mirror including metadata and plugin state
#   Result:       HOST2 has a gold-standard Emby state once per week
#                 Image cache warm for 6 days — only reset Sunday when users sleep
#
# The two profiles work together:
#   emby-failover: keeps HOST2 current on what matters for immediate failover
#   emby:         gives HOST2 full fidelity once per week
#   Neither alone is sufficient — both are needed.
# ─────────────────────────────────────────────────────────────────────────────

━━━ STEP 8 — PERSONAL ENCRYPTED SHARES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Personal shares are synced to the remote server for offsite backup. ZFS encrypts at the dataset level — the remote server receives encrypted blocks and cannot read the content without your passphrase or keyfile.

── Create an Encrypted ZFS Dataset ────────────────────────────────────────

# In the unRAID UI:
# ─────────────────────────────────────────────────────────────────────────────
# Main → click your ZFS pool name → + Dataset
#   Name: Gmer4Lfe-Personal
#   Encryption: Enabled
#   Passphrase: [your passphrase]
#   ⚠️  Write your passphrase down — if lost, data is completely unrecoverable
#
# Settings → Shares → Add Share
#   Share path: point to the new encrypted dataset
#   Use cache: Only — keeps data on ZFS pool, not array
# ─────────────────────────────────────────────────────────────────────────────

# Verify encryption is active before syncing:
zfs get encryption poolname/Gmer4Lfe-Personal
# Should show: encryption  aes-256-gcm

── Auto-Unlock on Boot (Optional) ─────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Keyfile approach — passphrase stored in a file, loaded at boot.
# More convenient but the keyfile is a secret that must be protected.
# Never sync the keyfile to the remote server.
#
# Create keyfile — on HOST1 only:
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
chmod 600 /root/.zfs-keys/personal.key

# Set dataset to use keyfile instead of passphrase:
zfs change-key \
    -o keylocation=file:///root/.zfs-keys/personal.key \
    -o keyformat=raw \
    poolname/Gmer4Lfe-Personal

# Add to ramdisk_setup.sh or array_start.sh custom scripts:
zfs load-key poolname/Gmer4Lfe-Personal
zfs mount poolname/Gmer4Lfe-Personal
# ─────────────────────────────────────────────────────────────────────────────
# Manual unlock alternative (most secure — passphrase only in your head):
zfs load-key poolname/Gmer4Lfe-Personal    # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal

── Add to master_host1.conf ────────────────────────────────────────────────

# master_host1.conf
# ─────────────────────────────────────────────────────────────────────────────
# Personal shares append to the daily sync after DAILY_SYNC_SHARES.
# Remote server receives encrypted blocks — cannot read content without your key.
#
HOST1_PERSONAL_SHARES=(
    "/mnt/user/Gmer4Lfe-Personal"
)

━━━ STEP 9 — USER SCRIPTS SETUP ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The ecosystem is designed so the User Scripts plugin has only a small number of entries — each one an orchestrator. Individual scripts are never scheduled directly.

── At Startup of Array ─────────────────────────────────────────────────────

# Create one script entry named "array start":
# ─────────────────────────────────────────────────────────────────────────────
#!/bin/bash
bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh
# ─────────────────────────────────────────────────────────────────────────────
# Schedule: At Startup of Array
# Run as: Background Task
#
# This is the ONLY "At Startup of Array" entry needed.
# It launches everything in ARRAY_START_SCRIPTS from master.conf:
#   inotify_tuning.sh       — raise inotify limits before containers start
#   docker_syslog_filter.sh — suppress veth log noise
#   php_fpm_max_children.sh — WebGUI tuning
#   ramdisk_setup.sh        — create ramdisk before Emby starts
#   docker_network_connect.sh — connect containers to extra networks
#   system_watchdog.sh      — continuous system health monitor
#   docker_watchdog.sh      — continuous container health monitor
#   failover.sh             — continuous mutual failover

── Cron Schedule ────────────────────────────────────────────────────────────

# Create one script entry per cron schedule below.
# All entries: Run as Background Task
# ─────────────────────────────────────────────────────────────────────────────

# Every 3 minutes — transcode cleanup + manager:
*/3 * * * *
bash /mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh

# Every 30 minutes — Emby dirty sync (watch states, library delta):
*/30 * * * *
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
    /mnt/user/Media_Server/Emby --profile=emby-failover

# Every 6 hours — failed import + stalled download recovery:
0 */6 * * *
bash /mnt/user/appdata/unraid_scripts/Orchestrators/arrs_failed_stalled_recovery.sh

# 1am daily — full maintenance window:
#   git pull → rsync all shares → permissions → cleaners → arr cleanup → docker restart
0 1 * * *
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh

# 2:30am Sunday — weekly maintenance window:
#   stop containers → pull updates → clean sync → start containers → weekly restarts
30 2 * * 0
bash /mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh

# 8am daily — health digest (DIGEST_PROFILE in master.conf controls when it notifies):
0 8 * * *
bash /mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh

# Every 6 hours — inotify + php-fpm utilisation snapshot:
0 */6 * * *
bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh

# Sunday morning — weekly reports:
0 6 * * 0   bash .../Monitors/zfs_memory_snapshot.sh
0 7 * * 0   bash .../Monitors/smart_health.sh
0 9 * * 0   bash .../Monitors/cert_monitor.sh
0 10 * * 0  bash .../Monitors/backup_verify.sh
0 11 * * 0  bash .../Monitors/emby_session_report.sh
0 11 * * 0  bash .../Monitors/bandwidth_monitor.sh --report
# ─────────────────────────────────────────────────────────────────────────────

Set all entries to "Background Task" — output streams correctly to the User Scripts log rather than buffering in the browser tab. Non-background tasks can appear to hang on long-running scripts.

━━━ STEP 10 — VERIFY THE SETUP ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Before relying on scheduled jobs, test manually from the terminal on HOST1. Test with --dry-run first — no changes made, but the full pre-flight and configuration resolution runs.

── Test a Single Profile Sync ─────────────────────────────────────────────

# Dry run with verbose output — shows every decision the script makes:
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
    /mnt/user/appdata-Failover/Arrs_Stack --dry-run --log

Expected output (healthy):
  ━━━ ⚙️ Setup ━━━
    Host: HOST1 (unRAID-Gmer4Lfe) → HOST2 (unRAID-Jayred365)
    Remote IP: 100.x.x.x
    Profile: arrs_stack

  ━━━ 🛡️ Pre-flight Checks ━━━
    ✅ Remote reachable
    ✅ version parity — both on unRAID X.Y.Z
    ✅ Remote Docker daemon responding
    ✅ Remote rootfs: 12% (threshold: 75%)
    ✅ Remote share exists and not empty
    ✅ All pre-flight checks passed

── Test the Daily Orchestrator ─────────────────────────────────────────────

# Dry run of the full daily window — shows every job that would run:
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --dry-run

If any pre-flight check fails, the script aborts with a clear error message
before touching anything. Fix the reported issue and re-run --dry-run.

Common pre-flight failures and their causes:
  "Remote not reachable"     → Tailscale not connected on HOST2
  "Version mismatch"         → different unRAID versions — update before syncing
  "Remote rootfs above X%"   → HOST2's root filesystem nearly full
  "Remote share missing"     → share doesn't exist on HOST2 yet (see Step 11)
  "Docker daemon not responding" → HOST2's Docker service not started

── Check the Configuration Resolved Correctly ──────────────────────────────

# --status shows how master.conf resolved for this server and profile:
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
    /mnt/user/appdata-Failover/Arrs_Stack --status

━━━ STEP 11 — INITIAL HOST2 SYNC ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

If HOST2 is being set up from scratch with empty shares:

── Create Share Structure on HOST2 ────────────────────────────────────────

# On HOST2 — start the array and create shares via the unRAID UI.
# Or use the share recreation tool to create disk directories from HOST1's cfg files:
bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh

── Initial Push From HOST1 ─────────────────────────────────────────────────

# On HOST1 — push all shares to HOST2 for the first time:
# Use --log for verbose output on first run
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --log

First run may take several hours for large libraries — this is normal.
The scheduled nightly sync will be incremental after the initial push.
Progress shows per-share throughout.

━━━ NAMING CONSISTENCY — THIS IS REQUIRED ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The ecosystem uses one codebase on both servers. This only works if containers and shares have identical names on both servers. This is not configurable — it is a design requirement.

# ─────────────────────────────────────────────────────────────────────────────
# Container names must match exactly on both servers:
    "Emby"                ← both HOST1 and HOST2
    "NginxProxyManager"   ← both HOST1 and HOST2
    "Mariadb-Authelia"    ← both HOST1 and HOST2

# Share paths must match exactly on both servers:
    /mnt/user/Movies      ← both HOST1 and HOST2 (HOST2 has a mirror)
    /mnt/user/Tv_Shows    ← both HOST1 and HOST2
# ─────────────────────────────────────────────────────────────────────────────
# If a container has a different name on one server: the script skips it
# without error. It silently does the wrong thing. You only notice when
# the container is not stopped during a sync that requires it to stop.
#
# If a share has a different path: rsync.sh aborts with "remote share missing".
# Easier to catch — but still requires renaming the share to fix.
#
# Keep names consistent and one codebase covers both servers automatically.
# Diverge and every script that touches containers or shares needs custom logic.
# ─────────────────────────────────────────────────────────────────────────────

━━━ REPOSITORY STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Unraid_Scripts/
├── master.conf                          ← All shared configuration — edit this file
├── master_host1.conf                    ← HOST1-specific configuration
├── master_host2.conf                    ← HOST2-specific configuration
├── common.sh                            ← Shared library — functions used by all scripts
├── load_config.sh                       ← Config loader — sources all conf files
│
├── Orchestrators/
│   ├── array_start.sh                   ← Single "At Startup of Array" entry point
│   ├── daily_sync_maintenance.sh        ← 1am daily window orchestrator
│   ├── weekly_sync_maintenance.sh       ← Sunday 2:30am window orchestrator
│   ├── critical_sync_maintenance.sh     ← Every 15 minutes — critical sync + partnership
│   ├── media_management.sh              ← Permissions + cleaners + arr cleanup
│   ├── transcode_management.sh          ← Transcode cleanup then manager
│   └── arrs_failed_stalled_recovery.sh  ← Failed import + stalled download recovery
│
├── Rsync/
│   └── rsync.sh                         ← Core rsync script — called per share
│
├── Failover/
│   ├── failover.sh                      ← Mutual container failover — continuous loop
│   ├── failover_test.sh                 ← Controlled iptables failover simulation
│   └── failover_state_reset.sh          ← Reset failover state manually
│
├── Docker_Essentials/
│   ├── docker_watchdog.sh               ← Two-tier container monitor — continuous loop
│   ├── docker_daily_restart.sh          ← Nightly container restarts
│   ├── docker_weekly_restart.sh         ← Weekly container restarts
│   ├── docker_network_connect.sh        ← Ensure networks + connections at array start
│   └── watchdog_skip_list_manager.sh    ← Skip list inspection and recovery
│
├── unRAID_Essentials/
│   ├── system_watchdog.sh               ← Three-tier system health monitor — continuous
│   ├── ramdisk_setup.sh                 ← Creates ramdisk + symlink at array start
│   ├── inotify_tuning.sh                ← Raise inotify limits at array start
│   ├── docker_syslog_filter.sh          ← Suppress veth log noise
│   ├── php_fpm_max_children.sh          ← WebGUI performance tuning
│   ├── server_reboot.sh                 ← Graceful reboot with pre-flight warnings
│   ├── mover_stop.sh                    ← Stop mover cleanly with wall warning
│   ├── clear_logs.sh                    ← Size-threshold log cleanup
│   ├── webgui_restart.sh                ← nginx → php-fpm → emhttp escalation
│   └── git_pull_execute.sh              ← Pull latest scripts from Gitea
│
├── Media/
│   ├── media_shares_permissions.sh      ← Apply permissions to media shares
│   ├── media_cleaner.sh                 ← Remove junk files from media shares
│   ├── lidarr_cleanup.sh                ← Remove orphaned music files (HOST1 only)
│   ├── sonarr_cleanup.sh                ← Remove orphaned TV files (host-aware)
│   └── radarr_cleanup.sh                ← Remove orphaned movie files (host-aware)
│
├── Transcodes/
│   ├── transcode_manager.sh             ← Ramdisk/SSD symlink management
│   └── transcode_cleanup.sh             ← Remove stale segment files
│
├── Monitors/
│   ├── cert_monitor.sh                  ← SSL cert expiry via live TLS connection
│   ├── backup_verify.sh                 ← rsync mirror MD5 checksum verification
│   ├── smart_health.sh                  ← Drive SMART attribute monitoring
│   ├── zfs_memory_snapshot.sh           ← ZFS health + ARC + memory report
│   ├── bandwidth_monitor.sh             ← rsync transfer logging + weekly report
│   ├── weekly_health_digest.sh          ← Full ecosystem health aggregation
│   ├── emby_session_report.sh           ← Emby streaming usage statistics
│   ├── system_tuning_monitor.sh         ← inotify + php-fpm utilisation tracking
│   └── continuous_scripts_status.sh     ← Live dashboard for background processes
│
├── Partnership/
│   └── partnership_manage.sh            ← Two-server relationship lifecycle manager
│
└── Tools/
    ├── recreate_shares.sh               ← Create share directories from cfg files
    ├── bulk_permissions_repair.sh       ← One-shot permission repair
    ├── rsync_stop.sh                    ← Stop active rsync jobs cleanly
    ├── user_scripts_stop.sh             ← Stop running user script processes
    ├── server_reboot.sh                 ← Graceful scheduled reboot
    ├── zfs_pool_scrub.sh                ← Trigger ZFS pool scrub
    └── container_data_export.sh         ← Export container configuration

━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔴 SSH Connection Refused / Timeout

# Verify SSH is enabled on the remote:
#   Settings → Management Access → Secure Shell → Enabled

# Verify Tailscale is connected:
tailscale ip -4 unRAID-Jayred365    # should return 100.x.x.x

# Test SSH manually with the key:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@$(tailscale ip -4 unRAID-Jayred365) "hostname"
# Expected: unRAID-Jayred365
# If password prompted: key not authorised — recheck Step 3b

🔴 Pre-flight Aborts on Remote Rootfs

# Remote rootfs above ROOTFS_WARN threshold
# Check current usage on remote:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "df /"

# Common cause: array not started, drives not mounted
# Verify array is started on HOST2 before running syncs

🔴 Remote Share Missing

# Share exists locally but not on remote
# Verify the share exists on HOST2:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "ls /mnt/user/"

# If missing — create the share on HOST2 first, then run initial sync (Step 11)
# Or run recreate_shares.sh on HOST2 to create directories from HOST1's cfg files

🔴 Containers Not Stopping / Starting

# Verify container names in master_host*.conf match Docker exactly — case sensitive
# Check what Docker actually calls the container:
docker ps --format "{{.Names}}"

# Test Docker commands to remote manually:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Names}}'"

🔴 Profile Not Matching

# Profile key = directory basename lowercased
# /mnt/user/appdata-Failover/Arrs_Stack → basename: Arrs_Stack → key: arrs_stack

# Override explicitly if basename doesn't match a profile name:
bash rsync.sh /mnt/user/appdata-Failover/My_Stuff --profile=arrs_stack

# Verify what profile resolved for a path:
bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status

🔴 Script Not Found

# Verify repo was cloned to the correct location:
ls /mnt/user/appdata/unraid_scripts/master.conf

# Make scripts executable:
find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \;

━━━ AVAILABLE FLAGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

All scripts support these flags. Use --dry-run before any live operation.

# ─────────────────────────────────────────────────────────────────────────────
    --dry-run    run without making any changes — pre-flight still runs ✅
    --log        verbose output — show every decision made
    --status     show resolved configuration and exit — no rsync, no sync
    --no-log     suppress verbose output (some scripts)

# Examples:
bash rsync.sh /mnt/user/Movies --dry-run --log        # preview a media sync
bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status  # check profile resolution
bash daily_sync_maintenance.sh --dry-run              # preview full daily window
bash docker_watchdog.sh --status                      # check watchdog state
# ─────────────────────────────────────────────────────────────────────────────
Reference in New Issue View Git Blame Copy Permalink