Varaverk/Rsync/Manual-Rsync.md

# ━━━━━ RSYNC — Manual ━━━━━

Config reference, procedures, operational workflows.
For overview see README-Rsync.md. For per-script detail see the rsync.sh header.

This document also serves as the **initial setup guide** for standing up the
two-server ecosystem from scratch.

---

## ━━━ WHAT YOU'RE BUILDING ━━━

```
HOST1 (unRAID-Gmer4Lfe)                    HOST2 (unRAID-Jayred365)
──────────────────────                     ──────────────────────
Downloads to:                              Downloads to:
  /mnt/user/Movies                           /mnt/user/Anime_Movies
  /mnt/user/Tv_Shows                         /mnt/user/Anime_Shows
  /mnt/user/Music

Both servers' arrs track ALL content across both servers.
It does not matter who downloaded what or when.

Daily 3-phase window (1am):

  Phase 1 — arr_sync.sh (runs first):
    All arrs on all nodes reconcile libraries using external IDs
    (TMDB, TVDB, MusicBrainz). Union model — any node that tracks
    an item, all nodes get it. After this phase both servers' arrs
    know about all content regardless of who downloaded it.

  Phase 2 — rsync (bidirectional, no --delete on media shares):
    HOST1 pushes its shares  ──────►  HOST2 receives files
    HOST1 receives files     ◄──────  HOST2 pushes its shares
    Files arrive already tracked by the remote arr (arr_sync ran first).
    No --delete — media files only spread. Arr cleanup handles deletions.

  Phase 3 — arr cleanup:
    sonarr_cleanup / radarr_cleanup / lidarr_cleanup query the live arr
    API and remove any files no longer tracked. Emby notified after.

Note: profiled syncs (arrs_stack, emby, critical-data…) use their own
PROFILE_RSYNC_OPTS which include --delete. Only the no-profile media
shares use DEFAULT_RSYNC_OPTS (spread only).

Weekly clean sync (2:30am Sunday, containers stopped):
  Emby — full clean mirror       ◄──────►  Emby
  Critical-Data                  ──────►   Auth stack (HOST1 → HOST2)

Every 30 minutes — dirty sync:
  Emby watch states              ──────►   HOST2 stays current on playback
```

---

## ━━━ PREREQUISITES ━━━

Required on both servers before starting:

```
unRAID 7.x
Community Applications plugin     — search "Community Applications" in unRAID plugins
User Scripts plugin               — install via Community Applications
Tailscale plugin                  — install via Community Applications
Terminal access                   — unRAID UI → Tools → Terminal, or SSH
```

Optional but recommended:
```
Gitea (Docker container on HOST1)  — self-hosted git for the script repository
Working Emby installation          — for transcode management and failover
```

---

## ━━━ STEP 1 — TAILSCALE ━━━

Tailscale provides the encrypted mesh network between servers. Scripts resolve the
remote server's IP via Tailscale at runtime — no hardcoded IPs, no VPN configuration,
no open ports. All server-to-server communication goes through Tailscale.

### Install on Both Servers

```
1. Open Apps in the unRAID UI
2. Search "Tailscale" — install the plugin
3. Settings → Tailscale → Connect
4. Authenticate with your Tailscale account (browser opens on your machine)
5. Verify both servers appear: https://login.tailscale.com/admin/machines
```

### Verify Connectivity

```bash
# From HOST1 — should return HOST2's 100.x.x.x Tailscale IP:
tailscale ip -4 unRAID-Jayred365

# Test actual connectivity:
tailscale ping unRAID-Jayred365
```

> **Critical:** The hostnames in `master.conf` (`HOST1` and `HOST2`) must match the
> Tailscale machine names **exactly** — case sensitive. All remote IP resolution goes
> through `tailscale ip -4 HOSTNAME`. A name mismatch causes every remote operation to
> fail at the IP resolution step.

---

## ━━━ STEP 2 — ENABLE SSH ━━━

unRAID 7.x has SSH disabled by default. Enable it on both servers.

```
Settings → Management Access → Secure Shell
  SSH: Enabled
  SSH port: 22
Apply
```

SSH is only exposed on your local network and Tailscale interface. No ports are
opened to the public internet.

---

## ━━━ STEP 3 — SSH KEYS ━━━

Two sets of keys needed: rsync automation keys (server-to-server) and a Gitea
access key (for script repository pulls).

### 3a — Rsync Automation Keys (ssh_setup.sh)

Run on **each server**. `ssh_setup.sh` generates the key, copies it to the remote,
and updates `host*.conf` with the key path automatically.

```bash
# On HOST1 — after the repo is cloned:
bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh

# On HOST2:
bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh
```

Key naming convention: hostname lowercased, `unraid-` prefix stripped.
```
unRAID-Gmer4Lfe  → /root/.ssh/gmer4lfe_rsync_automation
unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation
```

`host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the
generated key. Run `--status` to verify:

```bash
bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --status
```

> **`ssh_setup.sh` is idempotent** — safe to re-run. Use `--force` to regenerate
> a key (e.g., after a security incident) and re-copy it to the remote.

### 3b — Gitea SSH Key (manual)

```bash
# On BOTH servers:
ssh-keygen -t ed25519 -f /root/.ssh/unraid_gitea -C "unraid-gitea" -N ""

# Print the public key to add to Gitea:
cat /root/.ssh/unraid_gitea.pub

# In Gitea: Settings → SSH / GPG Keys → Add Key → paste above
```

---

## ━━━ STEP 4 — CLONE THE REPOSITORY ━━━

```bash
# Create target directory:
mkdir -p /mnt/user/appdata/unraid_scripts

# Clone:
GIT_SSH_COMMAND="ssh -i /root/.ssh/unraid_gitea" \
    git clone git@YOUR_GITEA_HOST:FailedProxy/Unraid_Scripts.git \
    /mnt/user/appdata/unraid_scripts

# Make scripts executable:
find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \;
```

Expected structure after clone:
```
master.conf          ← all shared configuration
host1.conf    ← HOST1-specific configuration
host2.conf    ← HOST2-specific configuration
common.sh            ← shared library
load_config.sh       ← config loader
Orchestrators/
Rsync/
Fallback/
Docker_Essentials/
...
```

---

## ━━━ STEP 5 — CONFIGURE MASTER.CONF ━━━

```bash
nano /mnt/user/appdata/unraid_scripts/master.conf
```

### Host Identity

```bash
# These must match Tailscale machine names exactly — case sensitive.
HOST1="unRAID-Gmer4Lfe"       # REQUIRED
HOST2="unRAID-Jayred365"       # REQUIRED

# SSH keys — each server uses its own key to authenticate to the other:
HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key"
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"
```

### Git Repository

```bash
GITEA_CONTAINER="Gitea"
GITEA_REPO_PATH="FailedProxy/Unraid_Scripts.git"
TARGET_DIR="/mnt/user/appdata/unraid_scripts"
GITEA_SSH_KEY="/root/.ssh/unraid_gitea"
SSH_PORT=221
```

### Daily Sync Shares

```bash
# host1.conf
# Shares this server downloads to — pushed to the other server nightly.
# arr_sync.sh runs before rsync in the daily window, so the receiving
# server's arrs already track incoming files when they arrive.
# Never put the same path in both lists.
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"
    "/mnt/user/Tv_Shows"
    "/mnt/user/Music"
    "/mnt/user/Kids_Movies"
    "/mnt/user/Kids_Tv_Shows"
    "/mnt/user/Sports"
    "/mnt/user/stand-up_comedy"
)

# host2.conf
HOST2_DAILY_SYNC_SHARES=(
    "/mnt/user/Anime_Shows"
    "/mnt/user/Anime_Movies"
)
```

### Weekly Sync Shares

```bash
# master.conf
# Synced during the Sunday 2:30am window — containers stopped both sides.
WEEKLY_SYNC_SHARES=(
    "/mnt/user/Media_Server/Emby"
    "/mnt/user/appdata-Fallback/Critical-Data"
)
```

---

## ━━━ STEP 6 — MASTER_HOST*.CONF ━━━

`detect_hosts()` reads which server is running and aliases `HOST*_` prefixed vars
to their unprefixed names. Scripts only ever reference the unprefixed name — they
work identically on both servers.

```bash
nano /mnt/user/appdata/unraid_scripts/host1.conf    # on HOST1
nano /mnt/user/appdata/unraid_scripts/host2.conf    # on HOST2
```

Every variable is documented in the conf files. Key values to set:
- SSH key paths
- DAILY_SYNC_SHARES
- EMBY_URL / EMBY_API_KEY
- CERT_MONITOR_DOMAINS
- SMART_IGNORE_DRIVES
- ZFS_REPORT_IGNORE_POOLS

---

## ━━━ RSYNC PROFILES ━━━

Profiles control per-share behavior. Profile key = directory basename lowercased.
Override with `--profile=name`.

### Profile Matching

```bash
# rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack
#   basename: Arrs_Stack → lowercased: arrs_stack → matches [arrs_stack] profile
#
# rsync.sh /mnt/user/Movies
#   basename: Movies → no profile match → global defaults (no containers stopped)
#
# rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
#   explicit override
```

### Current Profile Definitions

```bash
# master.conf

# ── arrs_stack ───────────────────────────────────────────────────────────────
# Arr databases — stopped for clean SQLite snapshot
PROFILES["arrs_stack_CRITICAL_CONTAINER_NAMES"]=(
    "Sonarr" "Radarr" "Lidarr" "Prowlarr" "Bazarr" "Pinchflat"
)

# ── critical-data ─────────────────────────────────────────────────────────────
# Auth stack — stopped for clean database snapshot, Authelia delayed restart
PROFILES["critical-data_CRITICAL_CONTAINER_NAMES"]=(
    "Mariadb-Authelia" "Redis-Authelia"
    "NginxProxyManager" "Lldap-Gmer4Lfe"
)
PROFILES["critical-data_DELAYED_CONTAINERS"]=(
    "Authelia" "Authelia-Secondary"
)
PROFILES["critical-data_CONTAINER_DELAY"]=30

# ── important-data ────────────────────────────────────────────────────────────
# NextCloud + Postgres — stopped for clean snapshot
PROFILES["important-data_CRITICAL_CONTAINER_NAMES"]=(
    "Postgres-NextCloud"
)
PROFILES["important-data_DELAYED_CONTAINERS"]=("NextCloud")

# ── emby ──────────────────────────────────────────────────────────────────────
# Weekly full clean sync — both Emby instances stopped
PROFILES["emby_CRITICAL_CONTAINER_NAMES"]=("Emby")
PROFILES["emby_EXCLUDE_DIRS"]=(
    "transcodes/" "logs/" "crash*" "cache/"
)

# ── emby-fallback ─────────────────────────────────────────────────────────────
# Every 30 minutes, Emby STAYS RUNNING — dirty sync of critical state only
PROFILES["emby-fallback_CRITICAL_CONTAINER_NAMES"]=()   # nothing stops
PROFILES["emby-fallback_EXCLUDE_DIRS"]=(
    "*.wal" "*.shm"                            # unsafe mid-write
    "transcodes/" "logs/" "crash*" "cache/"
)
PROFILES["emby-fallback_REMOTE_RESTART_CONTAINERS"]=("Emby")
```

### Two Emby Profiles — Why Both Exist

**emby-fallback** (every 30 minutes, Emby stays running):
- Syncs: users.db, library.db, authentication.db, config/
- Skips: \*.wal, \*.shm, transcodes/, logs/, cache/
- Why: WAL files are written while Emby runs — copying them would produce a corrupt database on HOST2
- Result: HOST2 is always within 30 minutes of HOST1 on watch state and user activity

**emby** (Sunday 2:30am, both Emby instances stopped):
- Syncs: everything except transcodes, logs, cache, crash files
- Includes: metadata, plugins, full database state, all config
- Why: WAL is checkpointed on clean shutdown — safe to copy everything
- Result: HOST2 gets a gold-standard Emby state once per week

The two profiles work together. emby-fallback keeps HOST2 current for immediate failover.
emby gives HOST2 full fidelity once per week. Neither alone is sufficient.

---

## ━━━ PERSONAL ENCRYPTED SHARES ━━━

Personal shares sync to the remote for offsite backup. ZFS encrypts at the dataset
level — the remote receives encrypted blocks and cannot read the content without your
passphrase or keyfile.

### Create an Encrypted ZFS Dataset

```bash
# In the unRAID UI:
# Main → click your ZFS pool name → + Dataset
#   Name: Gmer4Lfe-Personal
#   Encryption: Enabled
#   Passphrase: [your passphrase]
#   Write your passphrase down — if lost, data is completely unrecoverable

# Verify encryption is active before syncing:
zfs get encryption poolname/Gmer4Lfe-Personal
# Should show: encryption  aes-256-gcm
```

### Auto-Unlock on Boot (Optional)

```bash
# Keyfile approach — more convenient, but the keyfile itself is a secret
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
chmod 600 /root/.zfs-keys/personal.key

zfs change-key \
    -o keylocation=file:///root/.zfs-keys/personal.key \
    -o keyformat=raw \
    poolname/Gmer4Lfe-Personal

# Add to array_started.sh or ramdisk_setup.sh:
zfs load-key poolname/Gmer4Lfe-Personal
zfs mount poolname/Gmer4Lfe-Personal

# Manual unlock alternative (most secure):
zfs load-key poolname/Gmer4Lfe-Personal    # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal
```

### Add to host1.conf

```bash
HOST1_PERSONAL_SHARES=(
    "/mnt/user/Gmer4Lfe-Personal"
)
```

---

## ━━━ STEP 7 — USER SCRIPTS SETUP ━━━

Only a small number of User Scripts entries are needed — each one an orchestrator.
Individual scripts are never scheduled directly except the 30-minute Emby sync.

### At Startup of Array

```bash
#!/bin/bash
bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_started.sh
# Schedule: At Startup of Array
# Run as: Background Task
```

This single entry launches everything defined in ARRAY_START_SCRIPTS from master.conf:
inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup,
docker_network_connect, fallback.
Watchdogs (resource, docker, system, stability) run separately via watchdog_orchestrator.

### Cron Schedule

```bash
# Every 30 minutes — Emby dirty sync:
*/30 * * * *
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
    /mnt/user/Media_Server/Emby --profile=emby-fallback

# Every 6 hours — failed import + stalled download recovery:
0 */6 * * *
bash /mnt/user/appdata/unraid_scripts/Orchestrators/arrs_failed_stalled_recovery.sh

# 1am daily — full maintenance window:
0 1 * * *
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh

# 2:30am Sunday — weekly maintenance window:
30 2 * * 0
bash /mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh

# 8am daily — health digest:
0 8 * * *
bash /mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh

# Every 6 hours — inotify + php-fpm snapshot:
0 */6 * * *
bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh

# Sunday morning — weekly reports:
0  6 * * 0   bash .../Monitors/zfs_memory_snapshot.sh
0  7 * * 0   bash .../Monitors/smart_health.sh
0  9 * * 0   bash .../Monitors/cert_monitor.sh
0 10 * * 0   bash .../Monitors/backup_verify.sh
0 11 * * 0   bash .../Monitors/emby_session_report.sh
0 11 * * 0   bash .../Monitors/bandwidth_monitor.sh --report
```

Set all entries to **Background Task** — non-background tasks can appear to hang
on long-running scripts.

---

## ━━━ VERIFICATION ━━━

Test with `--dry-run` first — all pre-flight checks run, no changes made.

### Test a Single Profile Sync

```bash
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log
```

Expected output (healthy):
```
━━━ Setup ━━━
  Host: HOST1 (unRAID-Gmer4Lfe) → HOST2 (unRAID-Jayred365)
  Remote IP: 100.x.x.x
  Profile: arrs_stack

━━━ Pre-flight ━━━
  Remote reachable
  version parity — both on unRAID X.Y.Z
  Remote Docker daemon responding
  Remote rootfs: 12% (threshold: 75%)
  Remote share exists and not empty
  All pre-flight checks passed
```

### Test the Daily Orchestrator

```bash
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --dry-run
```

### Check Configuration Resolution

```bash
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Arrs_Stack --status
```

---

## ━━━ INITIAL HOST2 SYNC ━━━

If HOST2 is being set up from scratch with empty shares:

```bash
# Create share directories on HOST2:
bash /mnt/user/appdata/unraid_scripts/Tools/recreate_shares.sh

# Initial push from HOST1 — first run may take several hours for large libraries:
bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --log
```

The scheduled nightly sync will be incremental after the initial push.

---

## ━━━ NAMING CONSISTENCY REQUIREMENT ━━━

The ecosystem uses one codebase on both servers. This only works if containers and
shares have identical names on both servers. **This is not configurable — it is a
design requirement.**

```
Container names must match exactly on both servers:
  "Emby"                ← both HOST1 and HOST2
  "NginxProxyManager"   ← both HOST1 and HOST2
  "Mariadb-Authelia"    ← both HOST1 and HOST2

Share paths must match exactly on both servers:
  /mnt/user/Movies      ← both HOST1 and HOST2
  /mnt/user/Tv_Shows    ← both HOST1 and HOST2
```

If a container has a different name on one server: the script skips it silently.
You only notice when the container is not stopped during a sync that requires it.

If a share has a different path: rsync.sh aborts with "remote share missing."
Easier to catch — but still requires renaming the share to fix.

---

## ━━━ TROUBLESHOOTING ━━━

### SSH Connection Refused / Timeout

```bash
# Verify SSH is enabled on the remote:
#   Settings → Management Access → Secure Shell → Enabled

# Verify Tailscale is connected:
tailscale ip -4 unRAID-Jayred365

# Test SSH manually (key path from --status output):
ssh -i /root/.ssh/gmer4lfe_rsync_automation \
    root@$(tailscale ip -4 unRAID-Jayred365) "hostname"
# If password prompted: key not authorised — re-run ssh_setup.sh

# Re-run setup (idempotent, re-copies key to remote):
bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh
```

### Pre-flight Aborts on Remote Rootfs

```bash
# Check current usage on remote:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "df /"

# Common cause: array not started, drives not mounted
```

### Remote Share Missing

```bash
# Verify the share exists on HOST2:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "ls /mnt/user/"

# If missing: create the share on HOST2, then run initial sync
```

### Containers Not Stopping / Starting

```bash
# Verify container names match Docker exactly — case sensitive:
docker ps --format "{{.Names}}"

# Test remote:
ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Names}}'"
```

### Profile Not Matching

```bash
# Profile key = directory basename lowercased
# /mnt/user/appdata-Fallback/Arrs_Stack → key: arrs_stack

# Override explicitly:
bash rsync.sh /mnt/user/appdata-Fallback/My_Stuff --profile=arrs_stack

# Verify what profile resolved:
bash rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status
```

---

## ━━━ FULL CONFIGURATION REFERENCE ━━━

### master.conf

```bash
# Rsync engine
RSYNC_ENABLED=true
DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace"
# No --delete in DEFAULT_RSYNC_OPTS — media share rsync is additive only.
# arr_sync.sh keeps all arr databases in union — either server can download anything.
# arr cleanup handles deletions (orphans only). Profiles use --delete explicitly.
BW_LIMIT=0                  # KB/s, 0 = unlimited
RETRY_COUNT=3
SLEEP=60                    # seconds between retries
ROOTFS_WARN_PCT=75          # abort if remote rootfs above this %

# Shared with Monitors/
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90
BANDWIDTH_WARN_GB=50

# Profile definitions (see RSYNC PROFILES section above)
declare -A PROFILES
declare -A PROFILE_BW_LIMIT
declare -A PROFILE_RETRY_COUNT
declare -A PROFILE_SLEEP
declare -A PROFILE_CONTAINER_DELAY
declare -A PROFILE_CRITICAL_CONTAINER_NAMES
declare -A PROFILE_DELAYED_CONTAINERS
declare -A PROFILE_EXCLUDE_DIRS
declare -A PROFILE_REMOTE_RESTART_CONTAINERS
```

### host*.conf

```bash
# Daily sync shares — one list per server (mutually exclusive)
HOST1_DAILY_SYNC_SHARES=(...)
HOST2_DAILY_SYNC_SHARES=(...)

# Personal encrypted shares
HOST1_PERSONAL_SHARES=(...)

# SSH key for this server to authenticate to the remote
# Set automatically by ssh_setup.sh — do not edit manually
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"
```

---

## ━━━ OUTPUT TIERS ━━━

Without `--log`, each run shows: section headers (Pre-flight, Transfer, Stop/Start
Containers), the transfer identity block (source, remote, profile, identity), a
progress indicator while rsync is running, and a summary block with duration,
bytes transferred, and status. Warnings and errors are always visible.

With `--log`, every decision is shown — pre-flight check results, profile resolution,
per-container stop/start state, retry details, and bandwidth log confirmation.
Use for first-run validation or when investigating unexpected behavior.

---

## ━━━ FLAG REFERENCE ━━━

All flags work on `rsync.sh` and all orchestrators.

### --dry-run

Runs all pre-flight checks. Shows what rsync would transfer. No transfer, no container
stops, no bandwidth log entry. Safe to run at any time.

```bash
rsync.sh /mnt/user/Movies --dry-run
rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log
daily_sync_maintenance.sh --dry-run
```

### --status

Shows resolved configuration — profile, remote identity, all vars that would be
used — then exits. No pre-flight checks, no rsync. Use to verify configuration
loaded correctly.

```bash
rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status
```

### --log

Verbose output throughout. Every decision, every container operation, every rsync
progress line. Use for first-time runs or when investigating issues.

### --profile=name

Override profile selection. Bypasses basename inference. Use when the directory
name doesn't match any profile key, or when testing a specific profile.

```bash
rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback
```