Gmer4Lfe
2fa78a21b6
- Tools/Manual-Tools.md: fallback_state_reset.sh marked not yet built; manual workaround added; TOC entry updated; failover→fallback rename throughout - Watchdogs/Manual-Watchdogs.md: System Watchdog config block updated — remove pre-split vars (MEM_WARN_GB, MEM_SHUTDOWN_GB, MEM_GB, MEM_RECOVER_GB, SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED, SYS_WATCHDOG_REQUIRED_CONTAINERS); add SYS_WATCHDOG_MEM_GB (reboot trigger only); fix state file path names (watchdog_state.db → container_watchdog_state.db, sys_watchdog_state.db → system_watchdog_state.db) - Orchestrators/README-Orchestrators.md: add monthly_maintenance.sh section with two-gate model, configuration reference, and usage examples - Monitors/README-Monitors.md: add mesh_monitor.sh to both tables and diagram - Monitors/Manual-Monitors.md: add mesh_monitor.sh section - README-User_Script_Plug-in.md, Rsync/README-Rsync.md, Rsync/Manual-Rsync.md, README.md: emby-failover → emby-fallback (missed from prior batch)
134 lines
7.5 KiB
Markdown
134 lines
7.5 KiB
Markdown
# ━━━━━ RSYNC ━━━━━
|
|
|
|
The transfer engine for the two-server ecosystem. `rsync.sh` is the single script
|
|
called by every orchestrator that moves data between servers — it handles profiles,
|
|
pre-flight checks, container stops, the actual transfer, and bandwidth logging.
|
|
Orchestrators decide what to sync and when. `rsync.sh` decides how to do it safely.
|
|
|
|
> **Never schedule rsync.sh directly for daily/weekly syncs.** Use the orchestrators
|
|
> in `Orchestrators/`. rsync.sh is called directly only for manual runs and the
|
|
> 30-minute Emby dirty sync, which needs its own cron entry.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
|
|
|
**rsync Alone Isn't Safe Enough for Live Databases**
|
|
Running rsync against a share while SQLite databases are being written produces
|
|
corrupt snapshots on the remote. The arr databases, Emby library database, and
|
|
Authelia session store all write continuously. A plain rsync copies them mid-write.
|
|
The remote gets a file that opens cleanly but has internal inconsistencies.
|
|
Fix: profiles stop specific containers before syncing and restart them after.
|
|
The database is quiesced, rsync runs against a static snapshot, containers come back up.
|
|
|
|
**Each Share Needs Different Behavior**
|
|
Media shares (Movies, TV) just spread files — nothing stops, no `--delete`
|
|
(arr cleanup scripts own deletions, and arr_sync ensures both arrs already
|
|
track incoming files before they arrive). Arr databases need containers stopped,
|
|
clean SQLite snapshot, restart. Emby has two modes: weekly full-stop clean mirror
|
|
and 30-minute dirty sync while Emby stays running (WAL files excluded). Critical-Data
|
|
stops the auth stack, waits for Authelia to come back after restart delay.
|
|
Fix: the profile system — one script, behavior defined entirely by the profile key.
|
|
|
|
**A Failed Remote Shouldn't Corrupt a Live Sync**
|
|
If the remote's rootfs is nearly full, an rsync that starts will write partial files
|
|
then fail mid-transfer, leaving the remote in a worse state than before. If the remote's
|
|
backing disks are offline, rsync writes to an empty mount point and "succeeds."
|
|
Fix: pre-flight checks abort before touching anything if remote conditions are wrong.
|
|
|
|
---
|
|
|
|
## ━━━ WHAT THIS FOLDER DOES ━━━
|
|
|
|
One script. One job: move data from this server to the remote safely.
|
|
|
|
`rsync.sh` handles the full transfer lifecycle:
|
|
1. Infer or accept a profile for the given directory
|
|
2. Run pre-flight checks (connectivity, rootfs, disk temps, remote share exists)
|
|
3. Stop containers specified by the profile (both sides)
|
|
4. Run rsync with profile flags, excludes, and bandwidth limit
|
|
5. Restart containers (with delay if configured)
|
|
6. Restart remote containers if dirty-sync profile specifies it
|
|
7. Log the transfer to bandwidth_monitor.sh
|
|
|
|
Everything else — deciding which shares to sync, in what order, on what schedule —
|
|
lives in the orchestrators.
|
|
|
|
---
|
|
|
|
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
|
|
|
```
|
|
Media/
|
|
arr_sync.sh ── runs before rsync in daily window ──────────► all arrs agree on library
|
|
↓
|
|
Orchestrators/ ← decides what to sync, when, and in what order
|
|
daily_sync_maintenance.sh ──────────────────────────────────► rsync.sh (per share)
|
|
weekly_sync_maintenance.sh ──────────────────────────────────► rsync.sh (emby, critical-data)
|
|
critical_sync_maintenance.sh ────────────────────────────────► rsync.sh (partnership shares)
|
|
|
|
Cron (direct):
|
|
*/30 * * * * ──────────────────────────────────► rsync.sh --profile=emby-fallback
|
|
|
|
Monitors/
|
|
bandwidth_monitor.sh ◄─── called by rsync.sh after each sync (--log-transfer)
|
|
|
|
Fallback/
|
|
fallback.sh ──── rsync writeback during handback ──► rsync.sh
|
|
```
|
|
|
|
rsync.sh never calls other scripts except `bandwidth_monitor.sh` at the end of a sync.
|
|
All orchestration logic lives in the callers. arr_sync.sh (Media/) is a peer that runs
|
|
before rsync in the daily window — it is not called by rsync.sh directly.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROFILE SYSTEM ━━━
|
|
|
|
Profile key = directory basename lowercased. `--profile=name` overrides.
|
|
|
|
| Profile | What It Syncs | Containers Stopped | Notes |
|
|
|---------|--------------|-------------------|-------|
|
|
| *(none)* | Media shares (Movies, TV, Music…) | None | Bidirectional spread — no `--delete` in DEFAULT_RSYNC_OPTS. Arr cleanup scripts own deletions. |
|
|
| `arrs_stack` | Arr databases | Sonarr, Radarr, Lidarr, Prowlarr, Bazarr, Pinchflat | Clean SQLite snapshot |
|
|
| `critical-data` | Auth stack | Mariadb-Authelia, Redis-Authelia, NPM, Lldap | Authelia has restart delay |
|
|
| `important-data` | NextCloud + Postgres | Postgres-NextCloud | NextCloud has restart delay |
|
|
| `emby` | Full Emby mirror | Emby (both sides) | Weekly — Sunday 2:30am |
|
|
| `emby-fallback` | Emby watch state delta | None | Dirty sync — Emby stays running |
|
|
|
|
For full profile definitions see `Manual-Rsync.md`.
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
|
|
|
| Script | Role | When It Runs |
|
|
|--------|------|-------------|
|
|
| `rsync.sh` | Core transfer engine — profile resolution, pre-flights, container management, transfer, bandwidth logging | Called by orchestrators; directly for manual and 30-min Emby dirty sync |
|
|
|
|
---
|
|
|
|
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
|
|
|
```
|
|
Callers (Orchestrators/) ──────────────────────────────────────────────────────
|
|
daily_sync_maintenance.sh │
|
|
weekly_sync_maintenance.sh rsync.sh /path/to/share [--profile=name] │
|
|
critical_sync_maintenance.sh ─────────────────────────────────────────────► │
|
|
fallback.sh (writeback) │
|
|
Direct cron (emby-fallback) │
|
|
▼
|
|
┌─────────────────────────────────────┐
|
|
│ 1. Infer/accept profile │
|
|
│ 2. Pre-flight checks │
|
|
│ - connectivity │
|
|
│ - rootfs / disk temp / disks │
|
|
│ - remote share exists │
|
|
│ 3. Stop containers (profile) │
|
|
│ 4. rsync transfer │
|
|
│ 5. Restart containers │
|
|
│ 6. Remote restart (dirty sync) │
|
|
│ 7. Log to bandwidth_monitor.sh │
|
|
└─────────────────────────────────────┘
|
|
```
|