true bidirectional with split truth modle set up. all scripts are fully bidirectional
This commit is contained in:
@@ -135,6 +135,68 @@ The same share lists are used by `failover.sh` for Tier 4 writeback — but in t
|
||||
|
||||
---
|
||||
|
||||
### `nightly_critical_full_sync.sh`
|
||||
|
||||
Runs a clean nightly sync for Emby and the auth stack (Critical-Data) with containers stopped. This is the companion to the hourly dirty sync — it provides a fully consistent state on HOST2 once per night.
|
||||
|
||||
```bash
|
||||
# Scheduled as: 30 2 * * * (2:30am daily — after daily_sync.sh finishes)
|
||||
/mnt/user/appdata/unraid_scripts/Orchestrators/nightly_critical_full_sync.sh
|
||||
```
|
||||
|
||||
**Why two Emby syncs:**
|
||||
|
||||
The hourly dirty sync runs with Emby up — WAL files excluded, watch states pushed continuously. This means HOST2 is never more than an hour behind on watch state. But it's not a clean database snapshot.
|
||||
|
||||
The nightly clean sync stops Emby, syncs the full clean database state, then restarts. HOST2 gets a fully consistent Emby state every night. The two syncs work together:
|
||||
|
||||
```
|
||||
Hourly dirty sync (Emby running):
|
||||
users.db, library.db, authentication.db, config/
|
||||
WAL excluded — safe mid-write
|
||||
HOST2 always within 1hr of HOST1 on watch state
|
||||
|
||||
Nightly clean sync (Emby stopped):
|
||||
Full clean snapshot — all databases flushed
|
||||
No WAL files in flight
|
||||
HOST2 gets gold-standard state once per night
|
||||
```
|
||||
|
||||
**Why clean auth sync matters:**
|
||||
|
||||
The auth stack runs warm on both servers continuously. During normal operation HOST2's auth stack serves its own domain — it doesn't receive dirty updates from HOST1. The nightly clean sync is the only time auth state propagates.
|
||||
|
||||
This means:
|
||||
- New user added on HOST1 → propagates to HOST2 overnight automatically
|
||||
- Proxy rule changes → propagated overnight
|
||||
- No manual intervention needed for most auth changes
|
||||
|
||||
For users who just want failover to work — this script handles it. No thinking required about dirty writes, WAL files, or when to sync.
|
||||
|
||||
**What it syncs:**
|
||||
|
||||
```
|
||||
Emby appdata:
|
||||
users.db, library.db, authentication.db, config/
|
||||
Containers stopped → clean flush → safe copy
|
||||
|
||||
Critical-Data (auth stack):
|
||||
NPM proxy rules + SSL certs
|
||||
Authelia config + database
|
||||
Mariadb-Authelia data
|
||||
Redis-Authelia session store
|
||||
LLDAP users and groups database
|
||||
All auth containers stopped → clean databases → safe copy
|
||||
Authelia delayed start on restart — Mariadb + Redis must be ready first
|
||||
```
|
||||
|
||||
**What it excludes (per rsync profile):**
|
||||
|
||||
```
|
||||
Emby: logs, transcodes, cache, metadata, *.db-wal, *.db-shm
|
||||
Auth: logs, *.tmp, nginx/temp, nginx/cache, notification.txt
|
||||
```
|
||||
|
||||
### `media_management.sh`
|
||||
|
||||
Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Scheduled once daily, typically after the nightly sync.
|
||||
@@ -240,12 +302,11 @@ This pattern means:
|
||||
# Recommended schedule
|
||||
*/3 * * * * transcode_management.sh # cleanup then manager — every 3 minutes
|
||||
0 1 * * * daily_sync.sh # 1am — media shares to remote
|
||||
0 2 * * * media_management.sh # 2am — after sync completes
|
||||
0 2 * * * media_management.sh # 2am — permissions, cleaners, arr cleanup
|
||||
30 2 * * * nightly_critical_full_sync.sh # 2:30am — clean Emby + auth stack sync
|
||||
```
|
||||
|
||||
The 1 hour gap between them is intentional. `daily_sync.sh` can take 30-60 minutes on a large library. Starting `media_management.sh` before it finishes risks permission and cleanup operations running on files that are mid-transfer.
|
||||
|
||||
If your sync consistently finishes well under an hour, reduce the gap. If it regularly runs long, increase it.
|
||||
The gaps are intentional. `daily_sync.sh` can take 30-60 minutes on a large library. `media_management.sh` starts at 2am giving daily_sync an hour to finish. `nightly_critical_full_sync.sh` starts at 2:30am giving media_management time to complete its permission and cleanup pass before Emby is stopped for the clean sync.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/bin/bash
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# --------------------------------- Nightly Critical Full Sync Orchestrator ----------------------------------
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Runs clean nightly sync for Emby and the auth stack (Critical-Data).
|
||||
# Both require containers stopped for a consistent, safe state sync.
|
||||
#
|
||||
# Why this exists as a separate orchestrator from daily_sync.sh:
|
||||
# daily_sync.sh handles media shares — large, runs at 1am, no container stops needed
|
||||
# This script handles appdata that needs containers stopped for clean state:
|
||||
#
|
||||
# Emby:
|
||||
# Hourly dirty sync runs continuously (containers up, WAL excluded)
|
||||
# Nightly clean sync stops Emby → syncs full clean state → restarts Emby
|
||||
# Ensures HOST2 has a fully consistent Emby database nightly
|
||||
#
|
||||
# Critical-Data (auth stack):
|
||||
# NPM, Authelia, Mariadb-Authelia, Redis-Authelia, LLDAP
|
||||
# Containers stopped during sync — databases flush cleanly
|
||||
# HOST2 gets a clean auth state nightly
|
||||
# Users, groups, proxy rules, SSL certs all consistent
|
||||
# Adding a user on HOST1 → propagates to HOST2 overnight automatically
|
||||
#
|
||||
# Schedule: 2:30am daily — after daily_sync.sh (1am) finishes
|
||||
# Container stop time is brief — Emby ~30s, auth stack ~15s
|
||||
# Users experience a short Emby interruption at 2:30am — acceptable tradeoff
|
||||
#
|
||||
# For most users: this script ensures failover always has a clean working auth state
|
||||
# without needing to think about dirty writes or WAL files.
|
||||
#
|
||||
# All configuration in Master.conf — rsync profiles handle container stops automatically.
|
||||
# Supports --dry-run to walk through without stopping containers or syncing.
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
|
||||
RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_GEAR Setup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Running as root"
|
||||
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
resolve_remote_ip
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SHIELD Pre-flight Checks ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_SHIELD Pre-flight Checks ━━━"
|
||||
|
||||
check_connectivity
|
||||
check_remote_rootfs
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Tracking
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
PASS=()
|
||||
FAIL=()
|
||||
SHARE_TIMES=()
|
||||
TOTAL_START=$(date +%s)
|
||||
|
||||
SYNC_JOBS=(
|
||||
"/mnt/user/appdata-Failover/Emby"
|
||||
"/mnt/user/appdata-Failover/Critical-Data"
|
||||
)
|
||||
|
||||
SHARE_COUNT=${#SYNC_JOBS[@]}
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SYNC Nightly Clean Sync ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Nightly Clean Sync Starting — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_SUMMARY Jobs: $SHARE_COUNT"
|
||||
echo ""
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — containers will not be stopped"
|
||||
|
||||
JOB_NUM=0
|
||||
for JOB in "${SYNC_JOBS[@]}"; do
|
||||
((JOB_NUM++))
|
||||
JOB_NAME=$(basename "$JOB")
|
||||
echo "━━━ [$JOB_NUM/$SHARE_COUNT] $JOB_NAME ━━━"
|
||||
|
||||
JOB_START=$(date +%s)
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
bash "$RSYNC_SCRIPT" "$JOB" --dry-run
|
||||
else
|
||||
bash "$RSYNC_SCRIPT" "$JOB"
|
||||
fi
|
||||
|
||||
EXIT_CODE=$?
|
||||
JOB_END=$(date +%s)
|
||||
JOB_DURATION=$(format_duration $(( JOB_END - JOB_START )))
|
||||
|
||||
if [[ "$EXIT_CODE" -eq 0 ]]; then
|
||||
PASS+=("$JOB_NAME")
|
||||
success "$JOB_NAME — $ICON_SUCCESS done in $JOB_DURATION"
|
||||
else
|
||||
FAIL+=("$JOB_NAME")
|
||||
error "$JOB_NAME — $ICON_ERROR failed after $JOB_DURATION"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
done
|
||||
|
||||
TOTAL_END=$(date +%s)
|
||||
TOTAL_DURATION=$(format_duration $(( TOTAL_END - TOTAL_START )))
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo "━━━━━ $ICON_SUMMARY NIGHTLY SYNC SUMMARY ━━━━━"
|
||||
echo "$ICON_TIME Duration: $TOTAL_DURATION"
|
||||
echo "$ICON_SUCCESS Passed: ${#PASS[@]} $ICON_ERROR Failed: ${#FAIL[@]}"
|
||||
echo ""
|
||||
|
||||
if [[ ${#PASS[@]} -gt 0 ]]; then
|
||||
for job in "${PASS[@]}"; do echo " $ICON_SUCCESS $job"; done
|
||||
fi
|
||||
if [[ ${#FAIL[@]} -gt 0 ]]; then
|
||||
for job in "${FAIL[@]}"; do echo " $ICON_ERROR $job"; done
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
echo "$ICON_WARN Status: DRY RUN — no changes made"
|
||||
elif [[ ${#FAIL[@]} -eq 0 ]]; then
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS ALL JOBS COMPLETE"
|
||||
notify "Nightly sync complete on $(hostname) — Emby + auth stack synced cleanly" "Nightly Sync" "normal"
|
||||
else
|
||||
echo "$ICON_ERROR Status: $ICON_ERROR ${#FAIL[@]} JOB(S) FAILED"
|
||||
notify "Nightly sync failed on $(hostname) — failed: ${FAIL[*]}" "Nightly Sync" "warning"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
Reference in New Issue
Block a user