added write back timing checks correlastes with tiers timing
This commit is contained in:
+102
-28
@@ -115,7 +115,7 @@
|
||||
|
||||
# ━━━ Rsync Defaults ━━━
|
||||
# Global fallback values used when no profile match is found for a directory.
|
||||
# Shares in DAILY_SYNC_SHARES always use these globals — no profile is defined for them.
|
||||
# Shares in HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES always use these globals — no profile is defined for them.
|
||||
# Appdata shares (Arrs_Stack, Critical-Data etc.) match profiles by directory basename.
|
||||
# If a profile key exists in a PROFILE_* array that value overrides the global.
|
||||
# If a profile key is missing the global below is used as the fallback.
|
||||
@@ -143,24 +143,40 @@
|
||||
|
||||
# ━━━ Daily Sync Shares ━━━
|
||||
# Media shares synced once daily by Orchestrators/daily_sync.sh.
|
||||
# These shares have no profile entry — all use DEFAULT_RSYNC_OPTS above.
|
||||
# Add or remove paths here to control what syncs each night.
|
||||
# Each server only syncs the shares it is source of truth for — direction is automatic.
|
||||
# detect_hosts() determines which server is running and picks the correct list.
|
||||
#
|
||||
# HOST1 pushes its truth shares TO HOST2.
|
||||
# HOST2 pushes its truth shares TO HOST1.
|
||||
# Never both pushing the same share — one server is always the truth holder.
|
||||
#
|
||||
# These shares use DEFAULT_RSYNC_OPTS — no profile entry needed.
|
||||
# For shares needing custom bandwidth or container stops — create a profile below instead.
|
||||
DAILY_SYNC_SHARES=(
|
||||
/mnt/user/Anime_Movies-Old
|
||||
/mnt/user/Anime_Shows-Old
|
||||
/mnt/user/Anime_Shows
|
||||
|
||||
# HOST1 truth shares — pushed from HOST1 to HOST2 nightly
|
||||
HOST1_DAILY_SYNC_SHARES=(
|
||||
/mnt/user/Books
|
||||
/mnt/user/Intros
|
||||
/mnt/user/Kids_Movies
|
||||
/mnt/user/Kids_Tv_Shows
|
||||
/mnt/user/Movies
|
||||
/mnt/user/Music
|
||||
/mnt/user/Music_Videos
|
||||
/mnt/user/Nextcloud
|
||||
/mnt/user/stand-up_comedy
|
||||
/mnt/user/Sports
|
||||
/mnt/user/Tv_Shows
|
||||
)
|
||||
|
||||
# HOST2 truth shares — pushed from HOST2 to HOST1 nightly
|
||||
# HOST2 is source of truth for anime — his arrs manage these shares
|
||||
HOST2_DAILY_SYNC_SHARES=(
|
||||
/mnt/user/Anime_Movies
|
||||
/mnt/user/Anime_Movies-Old
|
||||
/mnt/user/Anime_Shows
|
||||
/mnt/user/Anime_Shows-Old
|
||||
)
|
||||
|
||||
# ━━━ Personal Encrypted Shares ━━━
|
||||
# Personal shares synced to the remote server for offsite backup.
|
||||
# These are independent of the failover container stack — data backup only.
|
||||
@@ -535,27 +551,84 @@ HOST2_TIER4_DELAY=1080
|
||||
# Containers are stopped before this runs — clean source, no competing writes.
|
||||
# Full bandwidth available — DDNS stopped, containers stopped, nothing competing.
|
||||
#
|
||||
# Priority:
|
||||
# Critical — Emby userdata/playstates (small, fast, important)
|
||||
# Critical — Auth stack data
|
||||
# Skip — Media files (already on primary, never moved)
|
||||
# Skip — Downloads (start fresh — cleaner than syncing partial state)
|
||||
# ── WRITEBACK SKIP WINDOW ─────────────────────────────────────────────────────────────────────
|
||||
# Short outages do not benefit from writeback — the covering server accumulated dirty
|
||||
# or minimal data not worth writing over the primary's cleaner state.
|
||||
#
|
||||
# Format: "/path/to/source" — matched to rsync profile by directory basename
|
||||
# Emby syncs every 30min from a live running container (dirty sync).
|
||||
# A clean full sync runs nightly at 2:30am with Emby stopped.
|
||||
# After a short outage HOST1's nightly clean state is more reliable
|
||||
# than HOST2's dirty 30min sync data — skip writeback entirely.
|
||||
#
|
||||
# Real world outage profile:
|
||||
# 2-10 minutes — power blip, most common → skip writeback
|
||||
# 10-60 minutes — ISP issue, router restart → skip writeback
|
||||
# 1hr+ — actual problem → writeback worthwhile
|
||||
# 18hr+ — Tier 4 activated → always writeback
|
||||
#
|
||||
# Tier 1 writeback delay — separate from Tier 1 start delay (Tier 1 always starts immediately)
|
||||
# Tier 2+ reuse their existing TIER_DELAY vars — if containers started, time passed = writeback warranted
|
||||
# Tier 4 writeback — skips if under HOST*_TIER4_DELAY (same threshold as container start)
|
||||
# Only runs if outage was long enough to activate Tier 4 containers
|
||||
# Automatically uses the opposing host's daily sync share list (HOST*_DAILY_SYNC_SHARES)
|
||||
# FAILOVER_HOST*_WRITEBACK_TIER4 is for edge cases only — empty by default
|
||||
|
||||
# HOST1 writeback — run by HOST2 during HOST1 handback
|
||||
FAILOVER_HOST1_WRITEBACK=(
|
||||
"/mnt/user/appdata-Failover/Critical-Data" # auth stack — Authelia, Mariadb, Redis, LLDAP, NPM
|
||||
"/mnt/user/appdata-Failover/Important-Data" # NextCloud + Postgres
|
||||
HOST1_TIER1_WRITEBACK_DELAY=60 # minutes — skip Tier 1 writeback if outage under this
|
||||
HOST2_TIER1_WRITEBACK_DELAY=60 # minutes — skip Tier 1 writeback if outage under this
|
||||
# Tier 2 writeback threshold = HOST1_TIER2_DELAY (reused)
|
||||
# Tier 3 writeback threshold = HOST1_TIER3_DELAY (reused)
|
||||
# Tier 4 always writebacks — no threshold
|
||||
|
||||
# ── WRITEBACK JOB LISTS ───────────────────────────────────────────────────────────────────────
|
||||
# Organised by tier — writeback runs per tier based on outage duration
|
||||
# Priority:
|
||||
# Tier 1 — Emby userdata, auth stack — small, fast, most important
|
||||
# Tier 2 — NextCloud, Immich — user files that may have changed
|
||||
# Tier 3 — secondary services
|
||||
# Tier 4 — always runs if Tier 4 activated — arrs accumulated meaningful state
|
||||
# Skip — media files (already on primary, never moved)
|
||||
# Skip — downloads (start fresh — cleaner than syncing partial state)
|
||||
|
||||
# HOST1 writeback tiers — run by HOST2 during HOST1 handback
|
||||
FAILOVER_HOST1_WRITEBACK_TIER1=(
|
||||
"/mnt/user/appdata-Failover/Emby" # Emby userdata, playstates, metadata
|
||||
"/mnt/user/appdata-Failover/Gmer4Lfe" # server specific appdata — Organizr, UptimeKuma
|
||||
# "/mnt/user/appdata-Failover/Arrs_Stack" # skip — arrs start fresh on handback
|
||||
"/mnt/user/appdata-Failover/Critical-Data" # auth stack — Authelia, Mariadb, Redis, LLDAP, NPM
|
||||
)
|
||||
|
||||
# HOST2 writeback — run by HOST1 during HOST2 handback
|
||||
FAILOVER_HOST2_WRITEBACK=(
|
||||
# "/mnt/user/appdata-Failover/Jayred365" # HOST2 specific appdata
|
||||
# "container-placeholder"
|
||||
FAILOVER_HOST1_WRITEBACK_TIER2=(
|
||||
"/mnt/user/appdata-Failover/Important-Data" # NextCloud + Postgres
|
||||
)
|
||||
|
||||
FAILOVER_HOST1_WRITEBACK_TIER3=(
|
||||
"/mnt/user/appdata-Failover/Gmer4Lfe" # server specific appdata — Organizr, UptimeKuma
|
||||
)
|
||||
|
||||
FAILOVER_HOST1_WRITEBACK_TIER4=(
|
||||
# Edge case paths outside of normal HOST1_DAILY_SYNC_SHARES
|
||||
# Tier 4 automatically syncs HOST1_DAILY_SYNC_SHARES back — add extras here only
|
||||
# Examples: a share that exists but isn't in the daily sync list,
|
||||
# a special appdata path only relevant after extended outage
|
||||
# "/mnt/user/SomeSpecialShare"
|
||||
)
|
||||
|
||||
# HOST2 writeback tiers — run by HOST1 during HOST2 handback
|
||||
FAILOVER_HOST2_WRITEBACK_TIER1=(
|
||||
# "/mnt/user/appdata-Failover/Jayred365-Emby"
|
||||
# "/mnt/user/appdata-Failover/Jayred365-Critical"
|
||||
)
|
||||
|
||||
FAILOVER_HOST2_WRITEBACK_TIER2=(
|
||||
# "/mnt/user/appdata-Failover/Jayred365-Important"
|
||||
)
|
||||
|
||||
FAILOVER_HOST2_WRITEBACK_TIER3=(
|
||||
# "/mnt/user/appdata-Failover/Jayred365"
|
||||
)
|
||||
|
||||
FAILOVER_HOST2_WRITEBACK_TIER4=(
|
||||
# Edge case paths outside of normal HOST2_DAILY_SYNC_SHARES
|
||||
# Tier 4 automatically syncs HOST2_DAILY_SYNC_SHARES back — add extras here only
|
||||
# "/mnt/user/SomeSpecialShare"
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -893,13 +966,13 @@ MEDIA_MAINTENANCE_JOBS=(
|
||||
# and new sessions land on SSD permanently for that container run.
|
||||
|
||||
RAMDISK_PATH="/mnt/ramdisk_transcodes" # tmpfs mount point created at array start
|
||||
RAMDISK_SIZE="8G" # ceiling — tmpfs only uses RAM actually needed
|
||||
RAMDISK_SIZE="10G" # ceiling — tmpfs only uses RAM actually needed
|
||||
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at — location never changes
|
||||
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback location
|
||||
|
||||
# Usage thresholds in GB — hysteresis gap between WARN and LOW prevents flip-flop
|
||||
RAMDISK_WARN_GB=6.8 # flip symlink to SSD at or above this usage
|
||||
RAMDISK_LOW_GB=5.5 # flip symlink back to ramdisk when usage drops here
|
||||
RAMDISK_WARN_GB=8.8 # flip symlink to SSD at or above this usage
|
||||
RAMDISK_LOW_GB=6.5 # flip symlink back to ramdisk when usage drops here
|
||||
RAMDISK_SSD_MIN_GB=20 # minimum free GB on SSD required before allowing flip to SSD
|
||||
|
||||
# Cleanup age thresholds — files must be older than these AND not open by any process
|
||||
@@ -953,9 +1026,10 @@ CERT_MONITOR_DOMAINS=(
|
||||
# ━━━ Backup Verify ━━━
|
||||
# Verifies the rsync mirror is healthy by comparing random file checksums between servers.
|
||||
# Uses existing SSH keys — no additional configuration needed beyond the share list.
|
||||
# Leave BACKUP_VERIFY_SHARES empty to automatically use DAILY_SYNC_SHARES as the target list.
|
||||
# Leave BACKUP_VERIFY_SHARES empty to automatically use the local host's daily sync shares
|
||||
# (HOST1_DAILY_SYNC_SHARES or HOST2_DAILY_SYNC_SHARES based on detect_hosts()).
|
||||
BACKUP_VERIFY_SHARES=(
|
||||
# leave empty to use DAILY_SYNC_SHARES automatically
|
||||
# leave empty to use host-specific daily sync shares automatically
|
||||
)
|
||||
BACKUP_VERIFY_SAMPLE=10 # number of files to randomly sample per share per run
|
||||
BACKUP_VERIFY_MIN_SIZE=1M # skip files smaller than this — avoids tiny junk files
|
||||
|
||||
Reference in New Issue
Block a user