Gmer4Lfe
d5cf3db2ec
A gated-off section left nothing failed, so the weekly could run for hours and report "all complete" beside "0 shares synced"; skipped is now derived from what was expected rather than self-reported, and the verdict degrades to PARTIAL instead of flattering.
286 lines
13 KiB
Bash
Executable File
286 lines
13 KiB
Bash
Executable File
#!/bin/bash
|
|
# ==============================================================================================
|
|
# ============================= Critical Sync Maintenance ======================================
|
|
# ==============================================================================================
|
|
#
|
|
# PURPOSE
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Orchestrator for time-sensitive syncs running every 30 minutes. Keeps the
|
|
# mirror current between the less frequent daily and weekly windows.
|
|
# Schedule: */30 * * * * (every 30 minutes via User Scripts plugin)
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL MODEL
|
|
# ==============================================================================================
|
|
#
|
|
# 1. Critical-Data rsync — auth stack, NPM config, certs (containers stopped both sides)
|
|
# 2. CRITICAL_MAINTENANCE_SCRIPTS — play_state_sync + any other per-window scripts
|
|
# 3. partnership --check — read both state files, detect changes, act accordingly
|
|
#
|
|
# RSYNC GATE
|
|
# RSYNC_ENABLED=false → skips all syncs (global gate)
|
|
# CRITICAL_RSYNC_ENABLED=false → skips critical syncs only (per-orchestrator gate)
|
|
# partnership --check always runs regardless — state check doesn't need rsync.
|
|
#
|
|
# ==============================================================================================
|
|
# DESIGN PRINCIPLES
|
|
# ==============================================================================================
|
|
#
|
|
# Silent When Healthy
|
|
# Runs 48 times per day — clean runs must produce zero output. Only failures
|
|
# and notable events produce visible output.
|
|
#
|
|
# Auth-First Window
|
|
# Auth stack changes (new users, proxy rules, certs) propagate within 30min.
|
|
# Emby watch states stay in sync — mirror users see correct playback position.
|
|
# Partnership state changes detected and acted on quickly.
|
|
#
|
|
# Strict Lock, Never Queue
|
|
# acquire_lock "strict" — if the previous 30-min run is still going, skip
|
|
# this cycle entirely. Critical-Data taking > 30min is a problem worth
|
|
# knowing about. Strict mode prevents pile-up without waiting.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL SAFEGUARDS
|
|
# ==============================================================================================
|
|
#
|
|
# Root Enforcement
|
|
# rsync over SSH and container stop/start both require root.
|
|
#
|
|
# Lock Acquisition
|
|
# acquire_lock in strict mode — a cycle is skipped rather than queued if the previous
|
|
# one is still running. At a 30-minute cadence, queuing would let a slow sync stack
|
|
# windows behind it.
|
|
#
|
|
# Host Detection
|
|
# detect_hosts() sets MY_ID and REMOTE_ID for routing and logs.
|
|
#
|
|
# Empty Job List Guard
|
|
# Exits with an error and a notification if CRITICAL_MAINTENANCE_SCRIPTS is empty —
|
|
# a silently empty critical tier would stop downloader resets and play-state sync
|
|
# while still reporting success every 30 minutes.
|
|
#
|
|
# Remote IP Resolution
|
|
# resolve_remote_ip confirms the partner is reachable before any transfer is attempted.
|
|
#
|
|
# RSYNC_ENABLED Gate
|
|
# The global kill switch is respected before any rsync call, so disabling rsync
|
|
# ecosystem-wide genuinely stops it here too.
|
|
#
|
|
# Non-Fatal Steps
|
|
# A failing job is recorded and the rest of the tier still runs.
|
|
#
|
|
# ==============================================================================================
|
|
# CONFIGURATION
|
|
# ==============================================================================================
|
|
#
|
|
# master.conf
|
|
#
|
|
# CRITICAL_RSYNC_ENABLED — enable/disable rsync section
|
|
# CRITICAL_SYNC_SHARES — shares synced every 30min (HOST*_CRITICAL_SYNC_SHARES)
|
|
# CRITICAL_MAINTENANCE_SCRIPTS — scripts run in critical window (optional)
|
|
# PARTNERSHIP_ENABLED — enable/disable partnership check
|
|
#
|
|
# ==============================================================================================
|
|
# RUNTIME MODES
|
|
# ==============================================================================================
|
|
#
|
|
# critical_sync_maintenance.sh
|
|
# Normal run.
|
|
#
|
|
# critical_sync_maintenance.sh --dry-run
|
|
# Preview syncs without transferring.
|
|
#
|
|
# critical_sync_maintenance.sh --log
|
|
# Verbose per-share output.
|
|
#
|
|
# critical_sync_maintenance.sh --status
|
|
# Show configuration and exit.
|
|
#
|
|
# ==============================================================================================
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
|
|
source "$SCRIPT_DIR/../load_config.sh"
|
|
|
|
parse_args "$@"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Setup ━━━
|
|
# ==============================================================================================
|
|
if [[ "$EUID" -ne 0 ]]; then
|
|
error "Must be run as root"
|
|
exit 1
|
|
fi
|
|
|
|
|
|
acquire_lock "strict"
|
|
|
|
detect_hosts
|
|
|
|
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
|
# from a healthy run. Fail loudly instead of silently doing no work.
|
|
if [[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
|
|
error "CRITICAL_MAINTENANCE_SCRIPTS is empty — no critical maintenance scripts will run"
|
|
error "Check CRITICAL_MAINTENANCE_SCRIPTS in master.conf"
|
|
notify "critical maintenance scripts skipped on $(hostname) ($MY_ID) — CRITICAL_MAINTENANCE_SCRIPTS is empty" \
|
|
"$(basename "$0" .sh)" "warning"
|
|
exit 1
|
|
fi
|
|
resolve_remote_ip
|
|
|
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Status ━━━
|
|
# ==============================================================================================
|
|
if [[ "$SHOW_STATUS" == true ]]; then
|
|
echo ""
|
|
echo "━━━━━ $ICON_SUMMARY CRITICAL SYNC STATUS ━━━━━"
|
|
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
|
echo "$ICON_NET Remote: $REMOTE_ID ($REMOTE_SERVER_NAME)"
|
|
echo "$ICON_SYNC Rsync enabled: ${RSYNC_ENABLED:-true}"
|
|
echo "$ICON_SYNC Critical enabled: ${CRITICAL_RSYNC_ENABLED:-false}"
|
|
echo "$ICON_SHIELD Partnership: ${PARTNERSHIP_ENABLED:-false}"
|
|
echo ""
|
|
echo "━━━ Critical Sync Shares ━━━"
|
|
if [[ ${#CRITICAL_SYNC_SHARES[@]} -eq 0 ]]; then
|
|
warn " No CRITICAL_SYNC_SHARES configured"
|
|
else
|
|
for share in "${CRITICAL_SYNC_SHARES[@]}"; do
|
|
[[ -z "$share" ]] && continue
|
|
SHARE_PATH="${share%%|*}"
|
|
SHARE_PROFILE="${share##*|}"
|
|
SHARE_NAME=$(basename "$SHARE_PATH")
|
|
[[ "$SHARE_PATH" == "$SHARE_PROFILE" ]] && \
|
|
echo " $ICON_SYNC $SHARE_NAME — no profile" || \
|
|
echo " $ICON_SYNC $SHARE_NAME — profile: $SHARE_PROFILE"
|
|
done
|
|
fi
|
|
echo ""
|
|
echo "━━━ Critical Maintenance Scripts ━━━"
|
|
if [[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
|
|
echo " None configured"
|
|
else
|
|
for entry in "${CRITICAL_MAINTENANCE_SCRIPTS[@]}"; do
|
|
[[ -z "$entry" || "$entry" == \#* ]] && continue
|
|
echo " $ICON_GEAR $(basename "${entry%% *}")"
|
|
done
|
|
fi
|
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
|
exit 0
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Critical Shares Sync ━━━
|
|
# ==============================================================================================
|
|
START=$(date +%s)
|
|
RSYNC_OK=false
|
|
PASS=()
|
|
FAIL=()
|
|
|
|
log "$ICON_SYNC Critical shares (${#CRITICAL_SYNC_SHARES[@]}): $(for s in "${CRITICAL_SYNC_SHARES[@]}"; do printf '%s ' "$(basename "${s%%|*}")"; done)"
|
|
[[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -gt 0 ]] && \
|
|
log "$ICON_GEAR Maintenance scripts: $(for s in "${CRITICAL_MAINTENANCE_SCRIPTS[@]}"; do printf '%s ' "$(basename "${s%% *}")"; done)"
|
|
|
|
if ! check_rsync_enabled "CRITICAL"; then
|
|
echo "Critical rsync disabled — skipping sync, running partnership check only"
|
|
elif [[ ${#CRITICAL_SYNC_SHARES[@]} -eq 0 ]]; then
|
|
warn "CRITICAL_RSYNC_ENABLED=true but CRITICAL_SYNC_SHARES is empty for $MY_ID"
|
|
warn "Check HOST*_CRITICAL_SYNC_SHARES in host*.conf"
|
|
else
|
|
echo "Critical sync — $MY_ID → $REMOTE_ID — $(date '+%H:%M:%S')"
|
|
|
|
# Build dry-run flag to pass through
|
|
RSYNC_DRY=""
|
|
[[ "$DRY_RUN" == true ]] && RSYNC_DRY="--dry-run"
|
|
|
|
for share in "${CRITICAL_SYNC_SHARES[@]}"; do
|
|
[[ -z "$share" ]] && continue
|
|
|
|
# Parse optional profile flag: "/path/to/share|profile-name"
|
|
SHARE_PATH="${share%%|*}"
|
|
SHARE_PROFILE="${share##*|}"
|
|
SHARE_NAME=$(basename "$SHARE_PATH")
|
|
|
|
SHARE_START=$(date +%s)
|
|
|
|
if [[ "$SHARE_PATH" == "$SHARE_PROFILE" ]]; then
|
|
bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$SHARE_PATH" $RSYNC_DRY
|
|
else
|
|
bash "$SCRIPT_DIR/../Rsync/rsync.sh" "$SHARE_PATH" \
|
|
--profile="$SHARE_PROFILE" $RSYNC_DRY
|
|
fi
|
|
|
|
RSYNC_EXIT=$?
|
|
SHARE_DUR=$(format_duration $(( $(date +%s) - SHARE_START )))
|
|
|
|
if [[ "$RSYNC_EXIT" -eq 0 ]]; then
|
|
PASS+=("$SHARE_NAME")
|
|
echo "$SHARE_NAME — done in $SHARE_DUR ✅"
|
|
RSYNC_OK=true
|
|
else
|
|
FAIL+=("$SHARE_NAME")
|
|
error "$SHARE_NAME — failed after $SHARE_DUR (exit $RSYNC_EXIT)"
|
|
fi
|
|
done
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Critical Maintenance Scripts ━━━
|
|
# ==============================================================================================
|
|
JOB_PASS=()
|
|
JOB_FAIL=()
|
|
for script_entry in "${CRITICAL_MAINTENANCE_SCRIPTS[@]}"; do
|
|
[[ -z "$script_entry" || "$script_entry" == \#* ]] && continue
|
|
run_orch_child "$script_entry"
|
|
done
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Partnership Check ━━━
|
|
# ==============================================================================================
|
|
if [[ "${PARTNERSHIP_ENABLED:-false}" == true ]]; then
|
|
PARTNER_DRY=""
|
|
[[ "$DRY_RUN" == true ]] && PARTNER_DRY="--dry-run"
|
|
|
|
# A successful rsync proves the partner answered; a failed one is evidence it did not. But
|
|
# rsync being switched off is neither — and it used to be read as "unseen", so the offline
|
|
# counter climbed every 30 minutes toward the 30-day auto-offboard on a partnership whose
|
|
# only fault was that RSYNC_ENABLED=false. That is how a deliberately paused sync ends up
|
|
# dismantling the partnership it was paused for. With no rsync attempt there is nothing to
|
|
# report, so the check runs without touching the counter either way.
|
|
# Tier 2 counts as "switched off" here exactly as much as Tier 1 does. The guard below used
|
|
# to test RSYNC_ENABLED alone, but it is CRITICAL_RSYNC_ENABLED that governs whether this
|
|
# orchestrator attempts an rsync at all — so with Tier 1 open and Tier 2 closed, no transfer
|
|
# was attempted, RSYNC_OK stayed false, and the run fell through to --remote-unseen and
|
|
# incremented the counter every 30 minutes against a partner that was answering fine.
|
|
#
|
|
# Onboard Step 1d now leaves precisely that posture on purpose — Tier 1 open so provisioning
|
|
# can run, every Tier 2 gate closed so nothing is scheduled. A freshly onboarded, perfectly
|
|
# healthy partnership would have auto-offboarded itself 30 days later.
|
|
if [[ "$RSYNC_OK" == true ]]; then
|
|
bash "$SCRIPT_DIR/../Partnership/partnership_manager.sh" \
|
|
--check --remote-seen $PARTNER_DRY
|
|
elif [[ "${RSYNC_ENABLED:-false}" != true || "${CRITICAL_RSYNC_ENABLED:-false}" != true ]]; then
|
|
echo "Critical rsync gated off — partnership check runs, offline counter untouched"
|
|
bash "$SCRIPT_DIR/../Partnership/partnership_manager.sh" \
|
|
--check $PARTNER_DRY
|
|
else
|
|
bash "$SCRIPT_DIR/../Partnership/partnership_manager.sh" \
|
|
--check --remote-unseen $PARTNER_DRY
|
|
fi
|
|
else
|
|
echo "Partnership disabled — skipping check"
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Summary ━━━
|
|
# ==============================================================================================
|
|
END=$(date +%s)
|
|
DURATION=$(format_duration $(( END - START )))
|
|
|
|
# Standard ending, quiet mode — 30-min cadence, so a healthy cycle stays one line.
|
|
[[ ${#PASS[@]} -gt 0 ]] && echo "Synced: ${PASS[*]}"
|
|
orchestrator_summary "CRITICAL SYNC" "$START" "Critical Sync" quiet
|
|
exit $? |