Gmer4Lfe
1a47f864f9
Tier 2 beside the per-orchestrator gates. Unset reads as on — the toggle postdates the seed.
143 lines
8.2 KiB
Markdown
143 lines
8.2 KiB
Markdown
# ━━━━━ RSYNC ━━━━━
|
|
|
|
The transfer engine for the two-server ecosystem. `rsync.sh` is the single script
|
|
called by every orchestrator that moves data between servers — it handles profiles,
|
|
pre-flight checks, container stops, the actual transfer, and bandwidth logging.
|
|
Orchestrators decide what to sync and when. `rsync.sh` decides how to do it safely.
|
|
|
|
> **Never schedule rsync.sh directly for daily/weekly syncs.** Use the orchestrators
|
|
> in `Orchestrators/`. rsync.sh is called directly only for manual runs and the
|
|
> 30-minute Emby dirty sync, which needs its own cron entry.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
|
|
|
**rsync Alone Isn't Safe Enough for Live Databases**
|
|
Running rsync against a share while SQLite databases are being written produces
|
|
corrupt snapshots on the remote. The arr databases, Emby library database, and
|
|
Authelia session store all write continuously. A plain rsync copies them mid-write.
|
|
The remote gets a file that opens cleanly but has internal inconsistencies.
|
|
Fix: profiles stop specific containers before syncing and restart them after.
|
|
The database is quiesced, rsync runs against a static snapshot, containers come back up.
|
|
|
|
**Each Share Needs Different Behavior**
|
|
Media shares (Movies, TV) just spread files — nothing stops, no `--delete`
|
|
(arr cleanup scripts own deletions, and arr_sync ensures both arrs already
|
|
track incoming files before they arrive). Arr databases need containers stopped,
|
|
clean SQLite snapshot, restart. Emby has two modes: weekly full-stop clean mirror
|
|
and 30-minute dirty sync while Emby stays running (WAL files excluded). Critical-Data
|
|
stops the auth stack, waits for Authelia to come back after restart delay.
|
|
Fix: the profile system — one script, behavior defined entirely by the profile key.
|
|
|
|
**A Failed Remote Shouldn't Corrupt a Live Sync**
|
|
If the remote's rootfs is nearly full, an rsync that starts will write partial files
|
|
then fail mid-transfer, leaving the remote in a worse state than before. If the remote's
|
|
backing disks are offline, rsync writes to an empty mount point and "succeeds."
|
|
Fix: pre-flight checks abort before touching anything if remote conditions are wrong.
|
|
|
|
---
|
|
|
|
## ━━━ WHAT THIS FOLDER DOES ━━━
|
|
|
|
One script. One job: move data from this server to the remote safely.
|
|
|
|
`rsync.sh` handles the full transfer lifecycle:
|
|
1. Infer or accept a profile for the given directory
|
|
2. Run pre-flight checks (connectivity, rootfs, disk temps, remote share exists)
|
|
3. Stop containers specified by the profile (both sides)
|
|
4. Run rsync with profile flags, excludes, and bandwidth limit
|
|
5. Restart containers (with delay if configured)
|
|
6. Restart remote containers if dirty-sync profile specifies it
|
|
7. Log the transfer to bandwidth_monitor.sh
|
|
|
|
Everything else — deciding which shares to sync, in what order, on what schedule —
|
|
lives in the orchestrators.
|
|
|
|
---
|
|
|
|
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
|
|
|
```
|
|
Media/
|
|
arr_sync.sh ── runs before rsync in daily window ──────────► all arrs agree on library
|
|
↓
|
|
Orchestrators/ ← decides what to sync, when, and in what order
|
|
daily_sync_maintenance.sh ──────────────────────────────────► rsync.sh (per share)
|
|
weekly_sync_maintenance.sh ──────────────────────────────────► rsync.sh (emby, critical-data)
|
|
critical_sync_maintenance.sh ────────────────────────────────► rsync.sh (partnership shares)
|
|
|
|
Note: Emby watch/resume state is NOT synced by rsync — Media/play_state_sync.sh
|
|
does it over the Emby API every 30 min (CRITICAL_MAINTENANCE_SCRIPTS).
|
|
|
|
Monitors/
|
|
bandwidth_monitor.sh ◄─── called by rsync.sh after each sync (--log-transfer)
|
|
|
|
Fallback/
|
|
fallback.sh ──── rsync writeback during handback ──► rsync.sh
|
|
```
|
|
|
|
rsync.sh never calls other scripts except `bandwidth_monitor.sh` at the end of a sync.
|
|
All orchestration logic lives in the callers. arr_sync.sh (Media/) is a peer that runs
|
|
before rsync in the daily window — it is not called by rsync.sh directly.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROFILE SYSTEM ━━━
|
|
|
|
Profile key = directory basename lowercased. `--profile=name` overrides.
|
|
|
|
| Profile | What It Syncs | Containers Stopped | Notes |
|
|
|---------|--------------|-------------------|-------|
|
|
| *(none)* | Media shares (Movies, TV, Music…) | None | Bidirectional spread — no `--delete` in DEFAULT_RSYNC_OPTS. Arr cleanup scripts own deletions. |
|
|
| `arrs_stack` | Arr databases | Sonarr, Radarr, Lidarr, Prowlarr, Bazarr, Pinchflat | Clean SQLite snapshot |
|
|
| `critical-data` | Auth stack | Mariadb-Authelia, Redis-Authelia, NPM, Lldap | Authelia has restart delay |
|
|
| `important-data` | NextCloud + Postgres | Postgres-NextCloud | NextCloud has restart delay |
|
|
| `emby` | Full Emby mirror | Emby (both sides) | Weekly — Sunday 2:30am |
|
|
|
|
For full profile definitions see `Manual-Rsync.md`.
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
|
|
|
| Script | Role | When It Runs |
|
|
|--------|------|-------------|
|
|
| `rsync.sh` | Core transfer engine — profile resolution, pre-flights, container management, transfer, bandwidth logging | Called by orchestrators; directly for manual syncs |
|
|
| `media_seed.sh` | First-fill of a new partner — loops every `DAILY_SYNC_SHARES` entry through `rsync.sh --seed` | Dispatched detached by partnership onboard Step 13; restartable from the Partnership tab |
|
|
|
|
`media_seed.sh` is the one script here that is expected to run for weeks. A first seed of
|
|
HOST1's library is ~28 TB against a 12.5 MB/s `--bwlimit`, which is why onboard dispatches it
|
|
rather than waiting on it, and why the Partnership tab gives it a Stop button — `rsync.sh` runs
|
|
`--inplace --partial`, so stopping costs the file in flight, not the share.
|
|
|
|
It answers to `MEDIA_SEED_ENABLED` in `master.conf` — a Tier 2 toggle beside the per-orchestrator
|
|
ones, still under Tier 1 `RSYNC_ENABLED`. Off means onboard finishes without ever dispatching it,
|
|
for a partner being filled from a moved disk or one that already holds the library. An unset
|
|
`MEDIA_SEED_ENABLED` reads as on, so a conf that predates the toggle keeps its behaviour.
|
|
|
|
---
|
|
|
|
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
|
|
|
```
|
|
Callers (Orchestrators/) ──────────────────────────────────────────────────────
|
|
daily_sync_maintenance.sh │
|
|
weekly_sync_maintenance.sh rsync.sh /path/to/share [--profile=name] │
|
|
critical_sync_maintenance.sh ─────────────────────────────────────────────► │
|
|
fallback.sh (writeback) │
|
|
▼
|
|
┌─────────────────────────────────────┐
|
|
│ 1. Infer/accept profile │
|
|
│ 2. Pre-flight checks │
|
|
│ - connectivity │
|
|
│ - rootfs / disk temp / disks │
|
|
│ - remote share exists │
|
|
│ 3. Stop containers (profile) │
|
|
│ 4. rsync transfer │
|
|
│ 5. Restart containers │
|
|
│ 6. Remote restart (dirty sync) │
|
|
│ 7. Log to bandwidth_monitor.sh │
|
|
└─────────────────────────────────────┘
|
|
```
|