7.3 KiB
━━━━━ RSYNC ━━━━━
The transfer engine for the two-server ecosystem. rsync.sh is the single script
called by every orchestrator that moves data between servers — it handles profiles,
pre-flight checks, container stops, the actual transfer, and bandwidth logging.
Orchestrators decide what to sync and when. rsync.sh decides how to do it safely.
Never schedule rsync.sh directly for daily/weekly syncs. Use the orchestrators in
Orchestrators/. rsync.sh is called directly only for manual runs and the 30-minute Emby dirty sync, which needs its own cron entry.
━━━ THE PROBLEM THAT BUILT THIS ━━━
rsync Alone Isn't Safe Enough for Live Databases
Running rsync against a share while SQLite databases are being written produces
corrupt snapshots on the remote. The arr databases, Emby library database, and
Authelia session store all write continuously. A plain rsync copies them mid-write.
The remote gets a file that opens cleanly but has internal inconsistencies.
Fix: profiles stop specific containers before syncing and restart them after.
The database is quiesced, rsync runs against a static snapshot, containers come back up.
Each Share Needs Different Behavior
Media shares (Movies, TV) just spread files — nothing stops, no --delete
(arr cleanup scripts own deletions, and arr_sync ensures both arrs already
track incoming files before they arrive). Arr databases need containers stopped,
clean SQLite snapshot, restart. Emby has two modes: weekly full-stop clean mirror
and 30-minute dirty sync while Emby stays running (WAL files excluded). Critical-Data
stops the auth stack, waits for Authelia to come back after restart delay.
Fix: the profile system — one script, behavior defined entirely by the profile key.
A Failed Remote Shouldn't Corrupt a Live Sync
If the remote's rootfs is nearly full, an rsync that starts will write partial files
then fail mid-transfer, leaving the remote in a worse state than before. If the remote's
backing disks are offline, rsync writes to an empty mount point and "succeeds."
Fix: pre-flight checks abort before touching anything if remote conditions are wrong.
━━━ WHAT THIS FOLDER DOES ━━━
One script. One job: move data from this server to the remote safely.
rsync.sh handles the full transfer lifecycle:
- Infer or accept a profile for the given directory
- Run pre-flight checks (connectivity, rootfs, disk temps, remote share exists)
- Stop containers specified by the profile (both sides)
- Run rsync with profile flags, excludes, and bandwidth limit
- Restart containers (with delay if configured)
- Restart remote containers if dirty-sync profile specifies it
- Log the transfer to bandwidth_monitor.sh
Everything else — deciding which shares to sync, in what order, on what schedule — lives in the orchestrators.
━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
Media/
arr_sync.sh ── runs before rsync in daily window ──────────► all arrs agree on library
↓
Orchestrators/ ← decides what to sync, when, and in what order
daily_sync_maintenance.sh ──────────────────────────────────► rsync.sh (per share)
weekly_sync_maintenance.sh ──────────────────────────────────► rsync.sh (emby, critical-data)
critical_sync_maintenance.sh ────────────────────────────────► rsync.sh (partnership shares)
Note: Emby watch/resume state is NOT synced by rsync — Media/play_state_sync.sh
does it over the Emby API every 30 min (CRITICAL_MAINTENANCE_SCRIPTS).
Monitors/
bandwidth_monitor.sh ◄─── called by rsync.sh after each sync (--log-transfer)
Fallback/
fallback.sh ──── rsync writeback during handback ──► rsync.sh
rsync.sh never calls other scripts except bandwidth_monitor.sh at the end of a sync.
All orchestration logic lives in the callers. arr_sync.sh (Media/) is a peer that runs
before rsync in the daily window — it is not called by rsync.sh directly.
━━━ THE PROFILE SYSTEM ━━━
Profile key = directory basename lowercased. --profile=name overrides.
| Profile | What It Syncs | Containers Stopped | Notes |
|---|---|---|---|
| (none) | Media shares (Movies, TV, Music…) | None | Bidirectional spread — no --delete in DEFAULT_RSYNC_OPTS. Arr cleanup scripts own deletions. |
arrs_stack |
Arr databases | Sonarr, Radarr, Lidarr, Prowlarr, Bazarr, Pinchflat | Clean SQLite snapshot |
critical-data |
Auth stack | Mariadb-Authelia, Redis-Authelia, NPM, Lldap | Authelia has restart delay |
important-data |
NextCloud + Postgres | Postgres-NextCloud | NextCloud has restart delay |
emby |
Full Emby mirror | Emby (both sides) | Weekly — Sunday 2:30am |
For full profile definitions see Manual-Rsync.md.
━━━ SCRIPTS IN THIS FOLDER ━━━
| Script | Role | When It Runs |
|---|---|---|
rsync.sh |
Core transfer engine — profile resolution, pre-flights, container management, transfer, bandwidth logging | Called by orchestrators; directly for manual syncs |
━━━ HOW THE SCRIPTS RELATE ━━━
Callers (Orchestrators/) ──────────────────────────────────────────────────────
daily_sync_maintenance.sh │
weekly_sync_maintenance.sh rsync.sh /path/to/share [--profile=name] │
critical_sync_maintenance.sh ─────────────────────────────────────────────► │
fallback.sh (writeback) │
▼
┌─────────────────────────────────────┐
│ 1. Infer/accept profile │
│ 2. Pre-flight checks │
│ - connectivity │
│ - rootfs / disk temp / disks │
│ - remote share exists │
│ 3. Stop containers (profile) │
│ 4. rsync transfer │
│ 5. Restart containers │
│ 6. Remote restart (dirty sync) │
│ 7. Log to bandwidth_monitor.sh │
└─────────────────────────────────────┘