feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,133 @@
|
||||
# ━━━━━ RSYNC ━━━━━
|
||||
|
||||
The transfer engine for the two-server ecosystem. `rsync.sh` is the single script
|
||||
called by every orchestrator that moves data between servers — it handles profiles,
|
||||
pre-flight checks, container stops, the actual transfer, and bandwidth logging.
|
||||
Orchestrators decide what to sync and when. `rsync.sh` decides how to do it safely.
|
||||
|
||||
> **Never schedule rsync.sh directly for daily/weekly syncs.** Use the orchestrators
|
||||
> in `Orchestrators/`. rsync.sh is called directly only for manual runs and the
|
||||
> 30-minute Emby dirty sync, which needs its own cron entry.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**rsync Alone Isn't Safe Enough for Live Databases**
|
||||
Running rsync against a share while SQLite databases are being written produces
|
||||
corrupt snapshots on the remote. The arr databases, Emby library database, and
|
||||
Authelia session store all write continuously. A plain rsync copies them mid-write.
|
||||
The remote gets a file that opens cleanly but has internal inconsistencies.
|
||||
Fix: profiles stop specific containers before syncing and restart them after.
|
||||
The database is quiesced, rsync runs against a static snapshot, containers come back up.
|
||||
|
||||
**Each Share Needs Different Behavior**
|
||||
Media shares (Movies, TV) just spread files — nothing stops, no `--delete`
|
||||
(arr cleanup scripts own deletions, and arr_sync ensures both arrs already
|
||||
track incoming files before they arrive). Arr databases need containers stopped,
|
||||
clean SQLite snapshot, restart. Emby has two modes: weekly full-stop clean mirror
|
||||
and 30-minute dirty sync while Emby stays running (WAL files excluded). Critical-Data
|
||||
stops the auth stack, waits for Authelia to come back after restart delay.
|
||||
Fix: the profile system — one script, behavior defined entirely by the profile key.
|
||||
|
||||
**A Failed Remote Shouldn't Corrupt a Live Sync**
|
||||
If the remote's rootfs is nearly full, an rsync that starts will write partial files
|
||||
then fail mid-transfer, leaving the remote in a worse state than before. If the remote's
|
||||
backing disks are offline, rsync writes to an empty mount point and "succeeds."
|
||||
Fix: pre-flight checks abort before touching anything if remote conditions are wrong.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
One script. One job: move data from this server to the remote safely.
|
||||
|
||||
`rsync.sh` handles the full transfer lifecycle:
|
||||
1. Infer or accept a profile for the given directory
|
||||
2. Run pre-flight checks (connectivity, rootfs, disk temps, remote share exists)
|
||||
3. Stop containers specified by the profile (both sides)
|
||||
4. Run rsync with profile flags, excludes, and bandwidth limit
|
||||
5. Restart containers (with delay if configured)
|
||||
6. Restart remote containers if dirty-sync profile specifies it
|
||||
7. Log the transfer to bandwidth_monitor.sh
|
||||
|
||||
Everything else — deciding which shares to sync, in what order, on what schedule —
|
||||
lives in the orchestrators.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
```
|
||||
Media/
|
||||
arr_sync.sh ── runs before rsync in daily window ──────────► all arrs agree on library
|
||||
↓
|
||||
Orchestrators/ ← decides what to sync, when, and in what order
|
||||
daily_sync_maintenance.sh ──────────────────────────────────► rsync.sh (per share)
|
||||
weekly_sync_maintenance.sh ──────────────────────────────────► rsync.sh (emby, critical-data)
|
||||
critical_sync_maintenance.sh ────────────────────────────────► rsync.sh (partnership shares)
|
||||
|
||||
Cron (direct):
|
||||
*/30 * * * * ──────────────────────────────────► rsync.sh --profile=emby-failover
|
||||
|
||||
Monitors/
|
||||
bandwidth_monitor.sh ◄─── called by rsync.sh after each sync (--log-transfer)
|
||||
|
||||
Fallback/
|
||||
fallback.sh ──── rsync writeback during handback ──► rsync.sh
|
||||
```
|
||||
|
||||
rsync.sh never calls other scripts except `bandwidth_monitor.sh` at the end of a sync.
|
||||
All orchestration logic lives in the callers. arr_sync.sh (Media/) is a peer that runs
|
||||
before rsync in the daily window — it is not called by rsync.sh directly.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROFILE SYSTEM ━━━
|
||||
|
||||
Profile key = directory basename lowercased. `--profile=name` overrides.
|
||||
|
||||
| Profile | What It Syncs | Containers Stopped | Notes |
|
||||
|---------|--------------|-------------------|-------|
|
||||
| *(none)* | Media shares (Movies, TV, Music…) | None | Bidirectional spread — no `--delete` in DEFAULT_RSYNC_OPTS. Arr cleanup scripts own deletions. |
|
||||
| `arrs_stack` | Arr databases | Sonarr, Radarr, Lidarr, Prowlarr, Bazarr, Pinchflat | Clean SQLite snapshot |
|
||||
| `critical-data` | Auth stack | Mariadb-Authelia, Redis-Authelia, NPM, Lldap | Authelia has restart delay |
|
||||
| `important-data` | NextCloud + Postgres | Postgres-NextCloud | NextCloud has restart delay |
|
||||
| `emby` | Full Emby mirror | Emby (both sides) | Weekly — Sunday 2:30am |
|
||||
| `emby-failover` | Emby watch state delta | None | Dirty sync — Emby stays running |
|
||||
|
||||
For full profile definitions see `Manual-Rsync.md`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `rsync.sh` | Core transfer engine — profile resolution, pre-flights, container management, transfer, bandwidth logging | Called by orchestrators; directly for manual and 30-min Emby dirty sync |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
Callers (Orchestrators/) ──────────────────────────────────────────────────────
|
||||
daily_sync_maintenance.sh │
|
||||
weekly_sync_maintenance.sh rsync.sh /path/to/share [--profile=name] │
|
||||
critical_sync_maintenance.sh ─────────────────────────────────────────────► │
|
||||
fallback.sh (writeback) │
|
||||
Direct cron (emby-failover) │
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ 1. Infer/accept profile │
|
||||
│ 2. Pre-flight checks │
|
||||
│ - connectivity │
|
||||
│ - rootfs / disk temp / disks │
|
||||
│ - remote share exists │
|
||||
│ 3. Stop containers (profile) │
|
||||
│ 4. rsync transfer │
|
||||
│ 5. Restart containers │
|
||||
│ 6. Remote restart (dirty sync) │
|
||||
│ 7. Log to bandwidth_monitor.sh │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
Reference in New Issue
Block a user