rename appdata-Failover→appdata-Fallback in docs; critical sync 15→30min; transcode 3→7min
appdata-Failover → appdata-Fallback: All .md files updated — scripts and confs already used appdata-Fallback. Share was renamed; docs were the only stale references. critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day: critical_sync_maintenance.sh header, master.conf, host1/2.conf comments, Orchestrators README, User Script Plug-in README, Partnership README/manual/script, Docker_Essentials README/downloaders_reset.sh, Fallback README, weekly_sync_maintenance.sh, user_script_plug-in.sh all updated. rsync_emby_failover User Script superseded: Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside critical_sync_maintenance.sh every 30min alongside auth sync and partnership check. Section in User Script Plug-in README marked for deletion from plugin. transcode_management.sh: */3 → */7: transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers, Transcodes README/manual, Orchestrators README, User Script Plug-in README, user_script_plug-in.sh all updated.
This commit is contained in:
Gmer4Lfe
@@ -23,7 +23,7 @@
|
||||
# ── RSYNC ──────────────────────────────────────────────────────────────────────────────────
|
||||
# DAILY SYNC SHARES media shares HOST1 owns and pushes to HOST2
|
||||
# WEEKLY SYNC SHARES appdata shares synced weekly (Sunday 2:30am)
|
||||
# CRITICAL SYNC SHARES appdata shares synced every 15 minutes
|
||||
# CRITICAL SYNC SHARES appdata shares synced every 30 minutes
|
||||
# BACKUP VERIFY shares for checksum verification against remote
|
||||
# HOST1 RSYNC PROFILE host1-appdata profile for HOST1-specific appdata syncs
|
||||
#
|
||||
@@ -221,7 +221,7 @@
|
||||
)
|
||||
|
||||
# ━━━ Critical Sync Shares ━━━
|
||||
# Appdata shares synced every 15 minutes by critical_sync_maintenance.sh.
|
||||
# Appdata shares synced every 30 minutes by critical_sync_maintenance.sh.
|
||||
# Format: "/path/to/share" or "/path/to/share|profile-name"
|
||||
# Order matters — Critical-Data first (auth stack), then Emby dirty sync.
|
||||
HOST1_CRITICAL_SYNC_SHARES=(
|
||||
@@ -558,7 +558,7 @@
|
||||
# Add one entry per root folder in arr Settings → Media Management → Root Folders.
|
||||
|
||||
# ━━━ Downloaders ━━━
|
||||
# Used by downloaders_reset.sh — runs every 15min via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Used by downloaders_reset.sh — runs every 30min via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Clears stuck states, purges old history, prepares each client for a clean cycle.
|
||||
|
||||
# slskd — clears stuck searches, dead transfers, purges expired failed imports.
|
||||
|
||||
@@ -27,7 +27,7 @@
|
||||
# ── RSYNC ──────────────────────────────────────────────────────────────────────────────────
|
||||
# DAILY SYNC SHARES media shares HOST2 owns and pushes to HOST1
|
||||
# WEEKLY SYNC SHARES appdata shares synced weekly (Sunday 2:30am)
|
||||
# CRITICAL SYNC SHARES appdata shares synced every 15 minutes
|
||||
# CRITICAL SYNC SHARES appdata shares synced every 30 minutes
|
||||
# BACKUP VERIFY shares for checksum verification against remote
|
||||
# HOST2 RSYNC PROFILE host2-appdata profile for HOST2-specific appdata syncs
|
||||
#
|
||||
@@ -204,7 +204,7 @@
|
||||
)
|
||||
|
||||
# ━━━ Critical Sync Shares ━━━
|
||||
# Appdata shares synced every 15 minutes by critical_sync_maintenance.sh.
|
||||
# Appdata shares synced every 30 minutes by critical_sync_maintenance.sh.
|
||||
# Format: "/path/to/share" or "/path/to/share|profile-name"
|
||||
HOST2_CRITICAL_SYNC_SHARES=(
|
||||
# fill in when HOST2 is back online
|
||||
|
||||
@@ -40,7 +40,7 @@
|
||||
# MONTHLY MAINTENANCE Uptime-triggered heavy tasks — ZFS scrub, SMART tests (monthly_maintenance.sh)
|
||||
# DAILY SYNC MAINTENANCE Job list + media shares (daily_sync_maintenance.sh)
|
||||
# WEEKLY SYNC MAINTENANCE Job list + sync shares + update toggles (weekly_sync_maintenance.sh)
|
||||
# CRITICAL SYNC MAINTENANCE 15-minute jobs + sync shares + partnership check (critical_sync_maintenance.sh)
|
||||
# CRITICAL SYNC MAINTENANCE 30-minute jobs + sync shares + partnership check (critical_sync_maintenance.sh)
|
||||
#
|
||||
# ── RSYNC ──────────────────────────────────────────────────────────────────────────────────
|
||||
# RSYNC ENABLE/DISABLE Two-tier toggle system — global gate + per-orchestrator
|
||||
@@ -154,7 +154,7 @@
|
||||
# /boot/config/partnership_HOST2.db ← HOST2 writes only
|
||||
# Propagated via SSH — no rsync needed
|
||||
#
|
||||
# critical_sync_maintenance.sh runs --check every 15min:
|
||||
# critical_sync_maintenance.sh runs --check every 30min:
|
||||
# Reads both state files via SSH
|
||||
# Detects offboard requests → finalises from owner side
|
||||
# Increments offline counter → auto-offboards after threshold
|
||||
@@ -330,15 +330,15 @@
|
||||
MONTHLY_LAST_RUN_FILE="/boot/config/monthly_maintenance_last_run.db"
|
||||
|
||||
# ━━━ Critical Sync Maintenance ━━━
|
||||
# critical_sync_maintenance.sh runs every 15 minutes.
|
||||
# critical_sync_maintenance.sh runs every 30 minutes.
|
||||
# Order: CRITICAL_MAINTENANCE_SCRIPTS (jobs) → CRITICAL_SYNC_SHARES (rsync) → partnership --check
|
||||
# partnership --check always runs last regardless of rsync gate.
|
||||
# Comment out entries to disable without removing.
|
||||
CRITICAL_MAINTENANCE_SCRIPTS=(
|
||||
"Docker_Essentials/downloaders_reset.sh" # clear stuck download states every 15min
|
||||
"Docker_Essentials/downloaders_reset.sh" # clear stuck download states every 30min
|
||||
)
|
||||
|
||||
# Shares synced every 15 minutes — defined per host in host*.conf.
|
||||
# Shares synced every 30 minutes — defined per host in host*.conf.
|
||||
# HOST1_CRITICAL_SYNC_SHARES / HOST2_CRITICAL_SYNC_SHARES
|
||||
# Format: "/path/to/share" or "/path/to/share|profile-name"
|
||||
# Order matters — Critical-Data first (auth stack), then Emby dirty sync.
|
||||
@@ -442,7 +442,7 @@
|
||||
# RSYNC_ENABLED=true ← rsync works, individual scripts run fine
|
||||
# DAILY_RSYNC_ENABLED=false ← skip daily HDD syncs during rebuild
|
||||
# WEEKLY_RSYNC_ENABLED=true ← Emby + Critical-Data still sync (NVMe)
|
||||
# CRITICAL_RSYNC_ENABLED=true ← 15min auth stack sync still runs
|
||||
# CRITICAL_RSYNC_ENABLED=true ← 30min auth stack sync still runs
|
||||
# FALLBACK_RSYNC_ENABLED=true ← handback writeback still works
|
||||
# → Run individual: bash Rsync/rsync.sh /mnt/user/Movies
|
||||
# → When ready: DAILY_RSYNC_ENABLED=true
|
||||
@@ -493,14 +493,14 @@
|
||||
# critical-data — auth stack — full stop both sides, Authelia delayed start
|
||||
# called by weekly_sync_maintenance.sh — full clean sync weekly
|
||||
# critical-fallback — dirty sync — auth stays running both sides, WAL excluded
|
||||
# called by critical_sync_maintenance.sh every 15min
|
||||
# called by critical_sync_maintenance.sh every 30min
|
||||
# host1-appdata — HOST1 server-specific appdata — defined in host1.conf
|
||||
# host2-appdata — HOST2 server-specific appdata — defined in host2.conf
|
||||
# important-data — NextCloud + Postgres — NextCloud delayed start after Postgres
|
||||
# emby — weekly clean sync — both Emby stopped, full mirror
|
||||
# called by weekly_sync_maintenance.sh only — do NOT schedule separately
|
||||
# emby-fallback — dirty sync — Emby stays running, WAL excluded
|
||||
# called by critical_sync_maintenance.sh every 15min
|
||||
# called by critical_sync_maintenance.sh every 30min
|
||||
|
||||
declare -A PROFILE_RSYNC_OPTS=(
|
||||
[arrs_stack]="-av --info=progress2 --human-readable --bwlimit=$BW_LIMIT --delete --inplace"
|
||||
@@ -632,7 +632,7 @@
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Downloaders Reset ━━━
|
||||
# Runs every 15 minutes via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Runs every 30 minutes via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Clears stuck states, purges old history, prepares each download client for a clean cycle.
|
||||
# Per-host URLs and API keys live in host*.conf.
|
||||
DOWNLOADER_RETENTION_DAYS=7 # days — purge history older than this
|
||||
|
||||
@@ -197,7 +197,7 @@ array start. Silent when everything is correct. Notifies when it has to create s
|
||||
|
||||
### 🧹 Downloader Hygiene — `downloaders_reset.sh`
|
||||
|
||||
Maintenance reset for all download clients (slskd, SABnzbd, qBittorrent) every 15
|
||||
Maintenance reset for all download clients (slskd, SABnzbd, qBittorrent) every 30
|
||||
minutes. Clears stuck searches, dead transfers, failed imports, and stale queue entries
|
||||
that download clients accumulate but never clean up themselves. Never touches active
|
||||
or in-progress downloads.
|
||||
@@ -233,7 +233,7 @@ full coordination model between all four watchdogs.
|
||||
| `docker_update_remaining.sh` | Image update + prune for all containers not in managed lists | End of weekly maintenance window |
|
||||
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
|
||||
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
|
||||
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 15min via `critical_sync_maintenance.sh` |
|
||||
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
|
||||
|
||||
> `watchdog_skip_list_manager.sh` — manual recovery tool for the skip list. Lives in `Tools/` because it's an operator utility, not a lifecycle script.
|
||||
|
||||
@@ -262,7 +262,7 @@ Weekly maintenance window (2:30am Sunday):
|
||||
├── docker_update.sh --remainder ─ update containers not in managed lists
|
||||
└── docker_update_remaining.sh ── prune dangling images
|
||||
|
||||
Critical maintenance window (every 15min):
|
||||
Critical maintenance window (every 30min):
|
||||
critical_sync_maintenance.sh
|
||||
└── downloaders_reset.sh ──────── clear stuck downloads / stale queue entries
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
# state that download clients generate but never clean up themselves — stuck
|
||||
# searches, dead transfers, failed imports, stale queue entries, completed history.
|
||||
#
|
||||
# Called every 15 minutes by critical_sync_maintenance.sh via
|
||||
# Called every 30 minutes by critical_sync_maintenance.sh via
|
||||
# CRITICAL_MAINTENANCE_SCRIPTS. Can also be run manually for ad hoc cleanup.
|
||||
# If a downloader is not configured for this host, that section skips cleanly.
|
||||
#
|
||||
@@ -137,7 +137,7 @@ if [[ "$EUID" -ne 0 ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Lock first — wait mode since this runs every 15min and previous may still be finishing
|
||||
# Lock first — wait mode since this runs every 30min and previous may still be finishing
|
||||
acquire_lock "wait"
|
||||
|
||||
# detect_hosts() sets MY_ID and aliases all HOST*_SLSKD_*, HOST*_SABNZBD_*, HOST*_QBIT_* vars
|
||||
|
||||
@@ -242,16 +242,16 @@ HOST1_TIER1_WRITEBACK_DELAY=60
|
||||
|
||||
# Writeback paths — synced back to HOST1 on handback
|
||||
FALLBACK_HOST1_WRITEBACK_TIER1=(
|
||||
"/mnt/user/appdata-Failover/Critical-Data" # auth stack — Authelia + NPM + certs
|
||||
"/mnt/user/appdata-Fallback/Critical-Data" # auth stack — Authelia + NPM + certs
|
||||
"/mnt/user/Media_Server/Emby" # Emby userdata — watch history, playstates
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER2=(
|
||||
"/mnt/user/appdata-Failover/Important-Data" # NextCloud + Postgres + Immich
|
||||
"/mnt/user/appdata-Fallback/Important-Data" # NextCloud + Postgres + Immich
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER3=(
|
||||
"/mnt/user/appdata-Failover/Gmer4Lfe" # secondary appdata accumulated changes
|
||||
"/mnt/user/appdata-Fallback/Gmer4Lfe" # secondary appdata accumulated changes
|
||||
)
|
||||
|
||||
# Tier 4 writeback uses HOST1_DAILY_SYNC_SHARES automatically — no list needed here.
|
||||
@@ -355,15 +355,15 @@ ssh root@[HOST2-tailscale-ip] "docker inspect Emby --format '{{.State.Status}}'"
|
||||
These shares must exist on HOST2 with current data from HOST1 before failover is needed:
|
||||
|
||||
```
|
||||
/mnt/user/appdata-Failover/Critical-Data # auth stack — NPM, LLDAP, Authelia, certs
|
||||
/mnt/user/appdata-Failover/Important-Data # NextCloud + Postgres + Immich
|
||||
/mnt/user/appdata-Fallback/Critical-Data # auth stack — NPM, LLDAP, Authelia, certs
|
||||
/mnt/user/appdata-Fallback/Important-Data # NextCloud + Postgres + Immich
|
||||
/mnt/user/Media_Server/Emby # Emby userdata — watch history, playstates
|
||||
/mnt/user/appdata-Failover/Gmer4Lfe # server appdata
|
||||
/mnt/user/appdata-Fallback/Gmer4Lfe # server appdata
|
||||
```
|
||||
|
||||
```bash
|
||||
# Verify data is current — check modification times
|
||||
ssh root@[HOST2-tailscale-ip] "ls -la /mnt/user/appdata-Failover/Critical-Data/"
|
||||
ssh root@[HOST2-tailscale-ip] "ls -la /mnt/user/appdata-Fallback/Critical-Data/"
|
||||
```
|
||||
|
||||
Sync is maintained continuously by `daily_sync_maintenance.sh` critical-data profile.
|
||||
|
||||
@@ -102,7 +102,7 @@ know the full library — arr_sync kept them in union while both servers were on
|
||||
|
||||
NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running
|
||||
to serve his own users daily — this is not a fallback-only configuration. HOST1 is source of
|
||||
truth: all changes mirror to HOST2 every 15 minutes via critical sync.
|
||||
truth: all changes mirror to HOST2 every 30 minutes via critical sync.
|
||||
|
||||
When DNS cuts over, auth is already running on the covering server. The 30-60 second dead
|
||||
zone where auth is coming up after DNS has already switched does not exist.
|
||||
|
||||
@@ -49,7 +49,7 @@ tells you which job and what happened — no correlation needed.
|
||||
|
||||
### 🔴 Transcode Manager Triggering Unnecessary SSD Flips
|
||||
|
||||
`transcode_manager.sh` ran every 3 minutes on its own. It checked ramdisk usage —
|
||||
`transcode_manager.sh` ran every 7 minutes on its own. It checked ramdisk usage —
|
||||
saw 6.8GB used, threshold is 6.5GB, flipped sessions to SSD. One minute later
|
||||
`transcode_cleanup.sh` ran and removed 4GB of stale segment files from ended sessions.
|
||||
Actual usage was 2.8GB. Sessions were now on SSD for no reason. Users experiencing
|
||||
@@ -102,14 +102,14 @@ The User Scripts plugin contains exactly these entries:
|
||||
array_started.sh
|
||||
|
||||
# Cron — one entry per maintenance window:
|
||||
*/3 * * * * transcode_management.sh
|
||||
*/7 * * * * transcode_management.sh
|
||||
0 */6 * * * arrs_failed_stalled_recovery.sh
|
||||
*/30 * * * * rsync.sh emby-fallback ← not an orchestrator, direct call
|
||||
*/30 * * * * critical_sync_maintenance.sh ← auth + Emby dirty sync + partnership
|
||||
0 1 * * * daily_sync_maintenance.sh
|
||||
30 2 * * 0 weekly_sync_maintenance.sh
|
||||
|
||||
# Manual only (not scheduled):
|
||||
failover_test.sh, emby_database_repair.sh, repair tools
|
||||
fallback_test.sh, emby_database_repair.sh, repair tools
|
||||
```
|
||||
|
||||
Every job list is configured in the `ORCHESTRATORS` section of `master.conf`.
|
||||
@@ -168,7 +168,7 @@ without `--log`.
|
||||
|--------|---------------------|----------|
|
||||
| `array_started.sh` | All array startup scripts in order | At Startup of Array |
|
||||
| `watchdog_orchestrator.sh` | resource → docker → system → stability watchdogs | Every minute |
|
||||
| `transcode_management.sh` | Cleanup then manager — order critical | Every 3 minutes |
|
||||
| `transcode_management.sh` | Cleanup then manager — order critical | Every 7 minutes |
|
||||
| `arrs_failed_stalled_recovery.sh` | Failed import + stalled download recovery | Every 6 hours |
|
||||
| `daily_sync_maintenance.sh` | git pull → sync → media maintenance → restarts | 1am daily |
|
||||
| `weekly_sync_maintenance.sh` | Stop → update → clean sync → start → weekly restarts | 2:30am Sunday |
|
||||
@@ -292,11 +292,11 @@ array_started.sh --status
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every
|
||||
3 minutes. Exists because the order is not optional — the manager must always see
|
||||
7 minutes. Exists because the order is not optional — the manager must always see
|
||||
post-cleanup usage to make accurate flip decisions.
|
||||
|
||||
```bash
|
||||
# Scheduled: */3 * * * * (every 3 minutes)
|
||||
# Scheduled: */7 * * * * (every 7 minutes)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -371,7 +371,7 @@ post-cleanup usage to make accurate flip decisions.
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
transcode_management.sh # normal run (every 3 min via cron)
|
||||
transcode_management.sh # normal run (every 7 min via cron)
|
||||
transcode_management.sh --dry-run # passes --dry-run to both child scripts
|
||||
transcode_management.sh --status # show config, current state, daily log stats
|
||||
transcode_management.sh --log # verbose output from both child scripts
|
||||
@@ -730,7 +730,7 @@ maintenance block before the 7am coffee report.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Two Emby syncs run in parallel — dirty and clean:
|
||||
#
|
||||
# emby-fallback dirty sync (every 30 minutes, Emby running):
|
||||
# emby-fallback dirty sync (every 30 minutes via critical_sync_maintenance.sh, Emby running):
|
||||
# watch states, library deltas, user activity — continuous coverage
|
||||
# WAL files excluded — safe to copy while Emby writes
|
||||
# HOST2 always within 30 minutes of HOST1 on playback state
|
||||
@@ -814,7 +814,7 @@ maintenance block before the 7am coffee report.
|
||||
# master.conf
|
||||
WEEKLY_SYNC_SHARES=(
|
||||
"/mnt/user/Media_Server/Emby" # emby profile — full clean mirror
|
||||
"/mnt/user/appdata-Failover/Critical-Data" # critical-data — auth stack clean state
|
||||
"/mnt/user/appdata-Fallback/Critical-Data" # critical-data — auth stack clean state
|
||||
)
|
||||
|
||||
WEEKLY_MAINTENANCE_SCRIPTS=(
|
||||
@@ -943,14 +943,14 @@ media_management.sh --status # show configured job list and exit
|
||||
array_started.sh
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Frequent — every 3 minutes:
|
||||
# Frequent — every 7 minutes:
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
*/3 * * * * transcode_management.sh
|
||||
*/7 * * * * transcode_management.sh
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Periodic — not an orchestrator, direct rsync call:
|
||||
# Every 30 minutes — auth stack + Emby dirty sync + partnership check:
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
*/30 * * * * rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback
|
||||
*/30 * * * * critical_sync_maintenance.sh
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Every 6 hours:
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Critical Sync Maintenance ======================================
|
||||
# ==============================================================================================
|
||||
# Orchestrator for time-sensitive syncs that run every 15 minutes.
|
||||
# Orchestrator for time-sensitive syncs that run every 30 minutes.
|
||||
# Keeps the mirror current between the less frequent daily and weekly windows.
|
||||
# Schedule: */15 * * * * (every 15 minutes via User Scripts plugin)
|
||||
# Schedule: */30 * * * * (every 30 minutes via User Scripts plugin)
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# 1. Critical-Data rsync — auth stack, NPM config, certs (containers stopped both sides)
|
||||
@@ -12,8 +12,8 @@
|
||||
# 3. CRITICAL_MAINTENANCE_SCRIPTS — any scripts configured for critical window
|
||||
# 4. partnership --check — read both state files, detect changes, act accordingly
|
||||
#
|
||||
# ── WHY EVERY 15 MINUTES ──────────────────────────────────────────────────────────────────────
|
||||
# Auth stack changes (new users, proxy rules, certs) propagate within 15min ✅
|
||||
# ── WHY EVERY 30 MINUTES ──────────────────────────────────────────────────────────────────────
|
||||
# Auth stack changes (new users, proxy rules, certs) propagate within 30min ✅
|
||||
# Emby watch states stay in sync — mirror users see correct playback position ✅
|
||||
# Partnership state changes detected and acted on quickly ✅
|
||||
# Lock prevents: daily rsync doing Critical-Data mid-critical window ✅
|
||||
@@ -24,17 +24,17 @@
|
||||
# partnership --check always runs regardless — state check doesn't need rsync
|
||||
#
|
||||
# ── LOCK BEHAVIOUR ────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock "strict" — if previous 15min run still going, skip this cycle entirely
|
||||
# Critical-Data taking > 15min is a problem worth knowing about
|
||||
# acquire_lock "strict" — if previous 30min run still going, skip this cycle entirely
|
||||
# Critical-Data taking > 30min is a problem worth knowing about
|
||||
# Strict mode prevents pile-up without waiting — log and move on ✅
|
||||
#
|
||||
# ── SILENT WHEN HEALTHY ───────────────────────────────────────────────────────────────────────
|
||||
# Runs 96 times per day — clean runs must produce zero output ✅
|
||||
# Runs 48 times per day — clean runs must produce zero output ✅
|
||||
# Only failures and notable events produce visible output
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# CRITICAL_RSYNC_ENABLED — enable/disable rsync section
|
||||
# CRITICAL_SYNC_SHARES — shares synced every 15min (HOST*_CRITICAL_SYNC_SHARES)
|
||||
# CRITICAL_SYNC_SHARES — shares synced every 30min (HOST*_CRITICAL_SYNC_SHARES)
|
||||
# CRITICAL_MAINTENANCE_SCRIPTS — scripts run in critical window (optional)
|
||||
# PARTNERSHIP_ENABLED — enable/disable partnership check
|
||||
#
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# ==============================================================================================
|
||||
# Orchestrator — runs transcode_cleanup.sh then transcode_manager.sh in the correct order.
|
||||
# Replace individual transcode_manager and transcode_cleanup cron entries with this.
|
||||
# Schedule: */3 * * * * (every 3 minutes via User Scripts plugin)
|
||||
# Schedule: */7 * * * * (every 7 minutes via User Scripts plugin)
|
||||
#
|
||||
# ── WHY CLEANUP BEFORE MANAGER ────────────────────────────────────────────────────────────────
|
||||
# Cleanup runs first — removes stale segment files from ended sessions.
|
||||
@@ -47,7 +47,7 @@
|
||||
# All TRANSCODE_* threshold vars — see master.conf Transcode Manager section
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# transcode_management.sh — normal run (every 3 minutes via cron)
|
||||
# transcode_management.sh — normal run (every 7 minutes via cron)
|
||||
# transcode_management.sh --dry-run — preview without changes (passed to children)
|
||||
# transcode_management.sh --status — show configuration and current state
|
||||
# transcode_management.sh --log — verbose output from both child scripts
|
||||
@@ -92,7 +92,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo "$ICON_RAM Ramdisk: $RAMDISK_PATH ($RAMDISK_SIZE)"
|
||||
echo "$ICON_DISK SSD fallback: $TRANSCODE_SSD"
|
||||
echo "$ICON_LINK Symlink: $TRANSCODE_LINK"
|
||||
echo "$ICON_TIME Schedule: every 3 minutes"
|
||||
echo "$ICON_TIME Schedule: every 7 minutes"
|
||||
echo ""
|
||||
echo "━━━ Child Scripts ━━━"
|
||||
[[ -f "$CLEANUP_SCRIPT" ]] && \
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
# Emby builds a warm image cache on HOST2 throughout the week.
|
||||
# Syncing nightly resets cache — cold loads every morning for users.
|
||||
# Weekly sync: cache stays warm 6 days, resets Sunday night while users sleep.
|
||||
# emby-fallback dirty sync covers watch states + library every 15min between weekly syncs.
|
||||
# emby-fallback dirty sync covers watch states + library every 30min between weekly syncs.
|
||||
#
|
||||
# ── CONTAINER UPDATES ─────────────────────────────────────────────────────────────────────────
|
||||
# Containers already stopped for sync — updates pull at zero extra downtime.
|
||||
|
||||
@@ -18,20 +18,20 @@ NPM proxy rules NPM serving mirror's domain
|
||||
LLDAP users LLDAP — same users
|
||||
Authelia policies Authelia — same policies
|
||||
Certs Certs — mirrored, valid
|
||||
Emby Emby — dirty-synced every 15min
|
||||
Emby Emby — dirty-synced every 30min
|
||||
|
||||
Changes made here ──→ Propagated every 15 minutes
|
||||
Changes made here ──→ Propagated every 30 minutes
|
||||
WebUI management ──→ Redirected to HOST1 via Tailscale
|
||||
Config → git push ──→ Received via git pull on next cycle
|
||||
```
|
||||
|
||||
**Mirror's daily experience:** open unRAID Docker UI, click NginxProxyManager,
|
||||
browser opens HOST1's NPM WebUI automatically via Tailscale. Make a proxy rule
|
||||
change. 15 minutes later it's live on HOST2. The mirror operator never manages
|
||||
change. 30 minutes later it's live on HOST2. The mirror operator never manages
|
||||
auth directly — every WebUI redirect is transparent and automatic.
|
||||
|
||||
**Owner's daily experience:** manage auth as normal. Opens NPM → adds a proxy
|
||||
rule → 15min later live on both servers. Never needs to SSH to HOST2 or think
|
||||
rule → 30min later live on both servers. Never needs to SSH to HOST2 or think
|
||||
about HOST2 during normal operation.
|
||||
|
||||
---
|
||||
|
||||
@@ -34,7 +34,7 @@ in one place and forgotten in the other.
|
||||
**The fix:** one auth stack with a managed mirror. HOST1 owns the configuration.
|
||||
HOST2 runs a warm copy that stays current via 15-minute sync. HOST2's operator makes
|
||||
zero auth management decisions — clicking an auth container opens HOST1's WebUI via
|
||||
Tailscale. Changes happen there, propagate to HOST2 in 15 minutes. One place to
|
||||
Tailscale. Changes happen there, propagate to HOST2 in 30 minutes. One place to
|
||||
manage everything for both households.
|
||||
|
||||
---
|
||||
@@ -107,7 +107,7 @@ Called by `partnership_manager.sh --offboard` but runnable directly.
|
||||
|
||||
**Lifecycle management** (`partnership_manager.sh`) — dispatcher and monitor.
|
||||
- Manually: `--onboard`, `--offboard` (delegates to offboard script), `--transfer`, `--status`, `--unblock`
|
||||
- Automatically: `--check` called every 15 minutes by `critical_sync_maintenance.sh`
|
||||
- Automatically: `--check` called every 30 minutes by `critical_sync_maintenance.sh`
|
||||
|
||||
**SSH management** (`ssh_setup.sh`) — generates the keypair for rsync automation,
|
||||
installs it on the remote, and tracks auth failures with a configurable strike system.
|
||||
@@ -119,7 +119,7 @@ re-keying.
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
**Orchestrators/** — `critical_sync_maintenance.sh` calls `partnership_manager.sh --check`
|
||||
every 15 minutes, passing `--remote-seen` or `--remote-unseen` based on whether the
|
||||
every 30 minutes, passing `--remote-seen` or `--remote-unseen` based on whether the
|
||||
rsync to the partner succeeded. The rsync outcome is the connectivity signal — no
|
||||
separate ping needed.
|
||||
|
||||
@@ -140,7 +140,7 @@ independently at 4-hour cadence.
|
||||
|--------|------|--------------|
|
||||
| `partnership_onboard.sh` | One-time setup — SSH keys, stack deploy, arr bootstrap | Manually, once per server per partnership |
|
||||
| `partnership_offboard.sh` | Clean separation — both paths, both roles | Via `partnership_manager.sh --offboard`; or directly |
|
||||
| `partnership_manager.sh` | Dispatcher + monitor — onboard WebUIs, health check, transfer, status | `--check` every 15min; all other modes manually |
|
||||
| `partnership_manager.sh` | Dispatcher + monitor — onboard WebUIs, health check, transfer, status | `--check` every 30min; all other modes manually |
|
||||
| `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
|
||||
|
||||
---
|
||||
@@ -166,7 +166,7 @@ HOST1 (owner) runs:
|
||||
├─ partnership_manager.sh --onboard reconfigures WebUIs, writes ACTIVE state
|
||||
└─ arr_sync.sh bootstraps full library on both servers
|
||||
|
||||
ONGOING OPERATION (every 15min)
|
||||
ONGOING OPERATION (every 30min)
|
||||
─────────────────────────────────────────────────────────────────────────────
|
||||
critical_sync_maintenance.sh
|
||||
├─ Critical-Data rsync keeps auth appdata current on mirror
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Manages the full lifecycle of a two-server partnership — onboard, daily health
|
||||
# monitoring, offboard, and ownership transfer. Called by partnership_onboard.sh
|
||||
# during initial setup, and by critical_sync_maintenance.sh every 15 minutes for
|
||||
# during initial setup, and by critical_sync_maintenance.sh every 30 minutes for
|
||||
# the --check mode. All other modes are run manually.
|
||||
#
|
||||
# PARTNERSHIP_OWNER_HOST flips to "HOST2" after a successful --transfer.
|
||||
@@ -34,7 +34,7 @@
|
||||
# Reconfigures both servers, flips PARTNERSHIP_OWNER_HOST in master.conf
|
||||
# Requires confirmation string + consecutive health check passes
|
||||
#
|
||||
# --check (called every 15min by critical_sync_maintenance.sh)
|
||||
# --check (called every 30min by critical_sync_maintenance.sh)
|
||||
# --remote-seen: rsync succeeded → reset offline counter, read remote state
|
||||
# --remote-unseen: rsync failed → increment offline counter → auto-offboard at threshold
|
||||
# Silent when healthy — never noisy on clean runs
|
||||
@@ -172,7 +172,7 @@
|
||||
# Flip ownership — owner only. Requires exact confirmation string.
|
||||
#
|
||||
# partnership_manager.sh --check --remote-seen|--remote-unseen
|
||||
# Called by critical_sync_maintenance.sh every 15min — do not run manually
|
||||
# Called by critical_sync_maintenance.sh every 30min — do not run manually
|
||||
#
|
||||
# partnership_manager.sh --status
|
||||
# Show state files, blocklist, SSH key status from both servers
|
||||
@@ -1300,7 +1300,7 @@ fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Check ━━━
|
||||
# Called every 15min by critical_sync_maintenance.sh — must be silent when healthy
|
||||
# Called every 30min by critical_sync_maintenance.sh — must be silent when healthy
|
||||
# ==============================================================================================
|
||||
if [[ "$MODE" == "check" ]]; then
|
||||
|
||||
@@ -1316,7 +1316,7 @@ if [[ "$MODE" == "check" ]]; then
|
||||
OFFLINE_COUNT=$(( OFFLINE_COUNT + 1 ))
|
||||
echo "$OFFLINE_COUNT" > "$OFFLINE_COUNTER"
|
||||
|
||||
# Auto-offboard threshold: threshold_days × 96 intervals/day (every 15min)
|
||||
# Auto-offboard threshold: threshold_days × 48 intervals/day (every 30min)
|
||||
THRESHOLD_INTERVALS=$(( ${PARTNERSHIP_OFFLINE_THRESHOLD:-30} * 96 ))
|
||||
if [[ "$OFFLINE_COUNT" -ge "$THRESHOLD_INTERVALS" ]]; then
|
||||
warn "Remote offline for ${PARTNERSHIP_OFFLINE_THRESHOLD} days — triggering auto-offboard"
|
||||
|
||||
@@ -127,7 +127,7 @@ the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|fai
|
||||
```
|
||||
Category: Orchestrators
|
||||
Name: transcode_management
|
||||
Schedule: */3 * * * * (every 3 minutes)
|
||||
Schedule: */7 * * * * (every 7 minutes)
|
||||
Background: YES
|
||||
```
|
||||
|
||||
@@ -138,7 +138,7 @@ Background: YES
|
||||
|
||||
**What it does:**
|
||||
Runs transcode_cleanup.sh then transcode_manager.sh — in that exact order every
|
||||
3 minutes. Order is non-negotiable: cleanup removes stale files first so the manager
|
||||
7 minutes. Order is non-negotiable: cleanup removes stale files first so the manager
|
||||
sees real active session usage, not inflated numbers from ended sessions.
|
||||
|
||||
**transcode_cleanup.sh:**
|
||||
@@ -167,7 +167,7 @@ auto-flip to SSD, missing transcoding-temp → recreate immediately.
|
||||
```
|
||||
Category: Orchestrators
|
||||
Name: critical_sync_maintenance
|
||||
Schedule: */15 * * * * (every 15 minutes)
|
||||
Schedule: */30 * * * * (every 30 minutes)
|
||||
Background: YES
|
||||
```
|
||||
|
||||
@@ -177,13 +177,13 @@ Background: YES
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
Keeps the auth stack and Emby watch state current on HOST2 — 15 minutes is the
|
||||
maximum staleness for failover-critical data.
|
||||
Keeps the auth stack and Emby watch state current on HOST2 — 30 minutes is the
|
||||
maximum staleness for fallback-critical data.
|
||||
|
||||
**Jobs in order:**
|
||||
```
|
||||
rsync Critical-Data — pushes auth stack to HOST2 (NPM rules, certs, LLDAP users,
|
||||
Authelia config). HOST2's auth is always within 15min of HOST1.
|
||||
Authelia config). HOST2's auth is always within 30min of HOST1.
|
||||
rsync Emby dirty sync — pushes watch states, user activity, library delta to HOST2
|
||||
with Emby running. WAL files excluded — safe mid-write.
|
||||
HOST2 Emby restarts after sync to pick up config changes.
|
||||
@@ -191,7 +191,7 @@ partnership --check — reads remote state file, increments offline counter
|
||||
unreachable, triggers auto-offboard at PARTNERSHIP_OFFLINE_THRESHOLD
|
||||
```
|
||||
|
||||
**Silent when healthy:** no output on clean cycles — runs 96 times per day.
|
||||
**Silent when healthy:** no output on clean cycles — runs 48 times per day.
|
||||
|
||||
---
|
||||
|
||||
@@ -286,35 +286,14 @@ in one orchestrated sequence.
|
||||
|
||||
### ── rsync_emby_failover ─────────────────────────────────────────────────────
|
||||
|
||||
```
|
||||
Category: Orchestrators
|
||||
Name: rsync_emby_failover
|
||||
Schedule: */30 * * * * (every 30 minutes)
|
||||
Background: YES
|
||||
```
|
||||
> **Superseded by `critical_sync_maintenance`.** Delete this User Script from the
|
||||
> plugin. The emby-fallback dirty sync now runs as step 2 inside
|
||||
> `critical_sync_maintenance.sh` every 30 minutes alongside the auth stack sync
|
||||
> and partnership check. Running both produces duplicate syncs.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
||||
/mnt/user/Media_Server/Emby --profile=emby-fallback
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
Keeps HOST2's Emby within 30 minutes of HOST1 on watch states, play positions, and
|
||||
library changes — with Emby running on both sides. Not a full sync (that's weekly) —
|
||||
this is continuous coverage of what matters for failover.
|
||||
|
||||
**What syncs (emby-fallback profile):**
|
||||
```
|
||||
Syncs: users.db, library.db, authentication.db, config/
|
||||
Skips: *.wal *.shm — WAL files being written by Emby, unsafe to copy mid-write
|
||||
transcodes/ — volatile, never needed on HOST2
|
||||
logs/ — local diagnostics only
|
||||
cache/ — rebuilt automatically
|
||||
```
|
||||
|
||||
**HOST2 Emby restarts after each sync** to pick up config changes.
|
||||
**Result:** if HOST1 goes down, users resume from a position at most 30min stale.
|
||||
The Emby dirty sync (`/mnt/user/Media_Server/Emby --profile=emby-fallback`) is
|
||||
configured in `HOST1_CRITICAL_SYNC_SHARES` in host1.conf. It runs automatically
|
||||
when critical_sync_maintenance.sh runs.
|
||||
|
||||
---
|
||||
|
||||
@@ -346,7 +325,7 @@ updates at zero extra downtime, weekly container restarts.
|
||||
5. Pull images remotely — if WEEKLY_SYNC_UPDATES_REMOTE=true (SSH to remote)
|
||||
6. rsync WEEKLY_SYNC_SHARES:
|
||||
/mnt/user/Media_Server/Emby — full clean mirror, databases checkpointed
|
||||
/mnt/user/appdata-Failover/Critical-Data — auth stack clean state
|
||||
/mnt/user/appdata-Fallback/Critical-Data — auth stack clean state
|
||||
7. Start remote containers— dependency order, new image, settle + verify each
|
||||
8. Start local containers — same
|
||||
9. docker_weekly_restart.sh — restarts less-critical services that benefit from
|
||||
@@ -774,7 +753,7 @@ Background: YES
|
||||
```bash
|
||||
#!/bin/bash
|
||||
/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
||||
/mnt/user/appdata-Failover/Critical-Data
|
||||
/mnt/user/appdata-Fallback/Critical-Data
|
||||
```
|
||||
|
||||
**What syncs:** auth stack — NPM proxy rules, TLS certificates, LLDAP user data,
|
||||
@@ -816,7 +795,7 @@ Background: YES
|
||||
```bash
|
||||
#!/bin/bash
|
||||
/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
||||
/mnt/user/appdata-Failover/Arrs_Stack
|
||||
/mnt/user/appdata-Fallback/Arrs_Stack
|
||||
```
|
||||
|
||||
**What syncs:** all arr appdata — Sonarr, Radarr, Lidarr, Prowlarr, Bazarr, Pinchflat
|
||||
@@ -837,7 +816,7 @@ Background: YES
|
||||
```bash
|
||||
#!/bin/bash
|
||||
/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
||||
/mnt/user/appdata-Failover/Important-Data
|
||||
/mnt/user/appdata-Fallback/Important-Data
|
||||
```
|
||||
|
||||
**What syncs:** NextCloud, Postgres-NextCloud, Immich, PostgreSQL-Immich appdata.
|
||||
@@ -857,7 +836,7 @@ Background: YES
|
||||
```bash
|
||||
#!/bin/bash
|
||||
/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
||||
/mnt/user/appdata-Failover/Gmer4Lfe
|
||||
/mnt/user/appdata-Fallback/Gmer4Lfe
|
||||
```
|
||||
|
||||
**What syncs:** HOST1 server appdata backup — everything in the Gmer4Lfe appdata share.
|
||||
@@ -1175,7 +1154,7 @@ Current usage on both ramdisk and SSD.
|
||||
```
|
||||
Category: Transcodes
|
||||
Name: transcode_manager_manual
|
||||
Schedule: None — run manually (runs via transcode_management.sh every 3 min)
|
||||
Schedule: None — run manually (runs via transcode_management.sh every 7 min)
|
||||
Background: NO
|
||||
```
|
||||
|
||||
@@ -1641,17 +1620,11 @@ Wrong startup setup (old approach):
|
||||
At Startup of Array:
|
||||
array_start BG: YES — single entry, handles everything
|
||||
|
||||
*/3 * * * * (every 3 min):
|
||||
*/7 * * * * (every 7 min):
|
||||
transcode_management BG: YES — cleanup then manager
|
||||
|
||||
*/10 * * * * (every 10 min):
|
||||
webgui_watchdog BG: YES — silent when healthy
|
||||
|
||||
*/15 * * * * (every 15 min):
|
||||
critical_sync_maintenance BG: YES — auth stack + Emby dirty + partnership check
|
||||
|
||||
*/30 * * * * (every 30 min):
|
||||
rsync_emby_failover BG: YES — Emby watch state sync
|
||||
critical_sync_maintenance BG: YES — auth stack + Emby dirty sync + partnership
|
||||
|
||||
0 */6 * * * (every 6 hours):
|
||||
arrs_failed_stalled_recovery BG: YES — blocklist + re-search failed imports
|
||||
|
||||
@@ -243,14 +243,14 @@ where determines whether you need encryption and whether sync failure matters.
|
||||
# services to work correctly when they start on HOST2.
|
||||
#
|
||||
/mnt/user/Media_Server/Emby # Emby userdata — watch history, playstates
|
||||
/mnt/user/appdata-Failover/Critical-Data # auth stack — NPM rules, certs, user accounts
|
||||
/mnt/user/appdata-Fallback/Critical-Data # auth stack — NPM rules, certs, user accounts
|
||||
/mnt/user/Movies # media library content
|
||||
/mnt/user/Tv_Shows # media library content
|
||||
/mnt/user/Music # media library content
|
||||
#
|
||||
# Sync failure here = failover runs but with stale data (acceptable) or missing
|
||||
# data (problematic). These sync every night via daily_sync_maintenance.sh.
|
||||
# Critical-Data and Emby also sync every 15 minutes via critical_sync_maintenance.sh.
|
||||
# Critical-Data and Emby also sync every 30 minutes via critical_sync_maintenance.sh.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
@@ -266,8 +266,8 @@ where determines whether you need encryption and whether sync failure matters.
|
||||
# Content is not sensitive — anyone with access to HOST2 could read it.
|
||||
#
|
||||
# Examples:
|
||||
/mnt/user/appdata-Failover/Gmer4Lfe # Gmer4Lfe server appdata backup
|
||||
/mnt/user/appdata-Failover/Important-Data # NextCloud, Immich appdata
|
||||
/mnt/user/appdata-Fallback/Gmer4Lfe # Gmer4Lfe server appdata backup
|
||||
/mnt/user/appdata-Fallback/Important-Data # NextCloud, Immich appdata
|
||||
/mnt/user/Kids_Movies # kids content — not sensitive
|
||||
/mnt/user/Sports # sports content — not sensitive
|
||||
#
|
||||
@@ -345,7 +345,7 @@ This is not standby. This is mutual coverage between peers.
|
||||
#
|
||||
# "Minimal disruption" does NOT mean:
|
||||
# Zero downtime (DDNS TTL = 1 minute, brief gap expected)
|
||||
# Perfect consistency (Emby userdata is 15min stale at most)
|
||||
# Perfect consistency (Emby userdata is 30min stale at most)
|
||||
# All services covered immediately (tiered start over 24 hours)
|
||||
#
|
||||
# The system is designed around the real experience of a household:
|
||||
@@ -363,7 +363,7 @@ This is not standby. This is mutual coverage between peers.
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Things conventional failover provides that this does NOT:
|
||||
# Zero RPO (recovery point objective) — there is ~15min of potential staleness
|
||||
# Zero RPO (recovery point objective) — there is ~30min of potential staleness
|
||||
# Synchronous state replication — changes don't reach HOST2 instantly
|
||||
# Automatic failure detection in seconds — fallback.sh checks every 2 minutes
|
||||
# Transparent IP migration — DNS TTL means up to 1 minute of routing to old IP
|
||||
@@ -426,7 +426,7 @@ server's own domain.
|
||||
#
|
||||
# HOST1 is source of truth for the auth configuration. All changes to NPM proxy
|
||||
# rules, LLDAP users, Authelia policies, and certs happen on HOST1 and propagate
|
||||
# to HOST2 via Critical-Data sync every 15 minutes.
|
||||
# to HOST2 via Critical-Data sync every 30 minutes.
|
||||
#
|
||||
# HOST2 operator's experience: click NginxProxyManager in Docker UI →
|
||||
# browser opens HOST1's NPM via Tailscale redirect → make changes there →
|
||||
@@ -451,7 +451,7 @@ Always running:
|
||||
Source of truth:
|
||||
Media Movies, Tv_Shows, Music ←── Anime_Movies, Anime_Shows
|
||||
───────────────────────→ (bidirectional daily push)
|
||||
Config auth stack config ──→ mirrored every 15 min
|
||||
Config auth stack config ──→ mirrored every 30 min
|
||||
Emby state watch history, playstates──→ dirty sync every 30 min
|
||||
|
||||
Failover coverage:
|
||||
@@ -621,7 +621,7 @@ Throughout the day (every minute via watchdog_orchestrator.sh):
|
||||
Throughout the day:
|
||||
fallback.sh watching: remote server, internet connectivity
|
||||
transcode_management.sh managing: ramdisk ↔ SSD, session cleanup (every 3min)
|
||||
critical_sync_maintenance.sh keeping: auth stack + Emby current (every 15min)
|
||||
critical_sync_maintenance.sh keeping: auth stack + Emby current (every 30min)
|
||||
|
||||
1am:
|
||||
git pull (latest scripts first)
|
||||
|
||||
+11
-11
@@ -261,7 +261,7 @@ HOST2_DAILY_SYNC_SHARES=(
|
||||
# Synced during the Sunday 2:30am window — containers stopped both sides.
|
||||
WEEKLY_SYNC_SHARES=(
|
||||
"/mnt/user/Media_Server/Emby"
|
||||
"/mnt/user/appdata-Failover/Critical-Data"
|
||||
"/mnt/user/appdata-Fallback/Critical-Data"
|
||||
)
|
||||
```
|
||||
|
||||
@@ -296,13 +296,13 @@ Override with `--profile=name`.
|
||||
### Profile Matching
|
||||
|
||||
```bash
|
||||
# rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
|
||||
# rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack
|
||||
# basename: Arrs_Stack → lowercased: arrs_stack → matches [arrs_stack] profile
|
||||
#
|
||||
# rsync.sh /mnt/user/Movies
|
||||
# basename: Movies → no profile match → global defaults (no containers stopped)
|
||||
#
|
||||
# rsync.sh /mnt/user/appdata-Failover/Critical-Data --profile=critical-data
|
||||
# rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
|
||||
# explicit override
|
||||
```
|
||||
|
||||
@@ -492,7 +492,7 @@ Test with `--dry-run` first — all pre-flight checks run, no changes made.
|
||||
|
||||
```bash
|
||||
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
||||
/mnt/user/appdata-Failover/Arrs_Stack --dry-run --log
|
||||
/mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log
|
||||
```
|
||||
|
||||
Expected output (healthy):
|
||||
@@ -521,7 +521,7 @@ bash /mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh --
|
||||
|
||||
```bash
|
||||
bash /mnt/user/appdata/unraid_scripts/Rsync/rsync.sh \
|
||||
/mnt/user/appdata-Failover/Arrs_Stack --status
|
||||
/mnt/user/appdata-Fallback/Arrs_Stack --status
|
||||
```
|
||||
|
||||
---
|
||||
@@ -619,13 +619,13 @@ ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-ip] "docker ps --format '{{.Nam
|
||||
|
||||
```bash
|
||||
# Profile key = directory basename lowercased
|
||||
# /mnt/user/appdata-Failover/Arrs_Stack → key: arrs_stack
|
||||
# /mnt/user/appdata-Fallback/Arrs_Stack → key: arrs_stack
|
||||
|
||||
# Override explicitly:
|
||||
bash rsync.sh /mnt/user/appdata-Failover/My_Stuff --profile=arrs_stack
|
||||
bash rsync.sh /mnt/user/appdata-Fallback/My_Stuff --profile=arrs_stack
|
||||
|
||||
# Verify what profile resolved:
|
||||
bash rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status
|
||||
bash rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status
|
||||
```
|
||||
|
||||
---
|
||||
@@ -705,7 +705,7 @@ stops, no bandwidth log entry. Safe to run at any time.
|
||||
|
||||
```bash
|
||||
rsync.sh /mnt/user/Movies --dry-run
|
||||
rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --dry-run --log
|
||||
rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --dry-run --log
|
||||
daily_sync_maintenance.sh --dry-run
|
||||
```
|
||||
|
||||
@@ -716,7 +716,7 @@ used — then exits. No pre-flight checks, no rsync. Use to verify configuration
|
||||
loaded correctly.
|
||||
|
||||
```bash
|
||||
rsync.sh /mnt/user/appdata-Failover/Arrs_Stack --status
|
||||
rsync.sh /mnt/user/appdata-Fallback/Arrs_Stack --status
|
||||
```
|
||||
|
||||
### --log
|
||||
@@ -730,6 +730,6 @@ Override profile selection. Bypasses basename inference. Use when the directory
|
||||
name doesn't match any profile key, or when testing a specific profile.
|
||||
|
||||
```bash
|
||||
rsync.sh /mnt/user/appdata-Failover/Critical-Data --profile=critical-data
|
||||
rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
|
||||
rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback
|
||||
```
|
||||
|
||||
@@ -26,11 +26,11 @@ All three scripts follow a two-tier output model: `echo` lines are always visibl
|
||||
**ramdisk_setup.sh** — one-shot at array start. Without `--log`, section headers
|
||||
and the final summary are visible. Per-step creation detail suppressed.
|
||||
|
||||
**transcode_manager.sh** — runs every 3 minutes. Without `--log`, only state
|
||||
**transcode_manager.sh** — runs every 7 minutes. Without `--log`, only state
|
||||
transitions (flips, warnings, errors) and active session display are shown. When
|
||||
nothing changes, a single one-line confirmation is printed. Per-check detail suppressed.
|
||||
|
||||
**transcode_cleanup.sh** — runs every 3 minutes. Without `--log`, the cleanup
|
||||
**transcode_cleanup.sh** — runs every 7 minutes. Without `--log`, the cleanup
|
||||
summary (files removed, space freed) is always visible. Per-file deletion detail
|
||||
suppressed.
|
||||
|
||||
@@ -376,7 +376,7 @@ At Startup of Array (via array_started.sh in unRAID_Essentials/):
|
||||
ramdisk_setup.sh — creates ramdisk, symlink, transcoding-temp
|
||||
Must run BEFORE Emby starts
|
||||
|
||||
Every 3 minutes (via transcode_management.sh in Orchestrators/):
|
||||
Every 7 minutes (via transcode_management.sh in Orchestrators/):
|
||||
1. transcode_cleanup.sh — remove stale files, check flip-back
|
||||
2. transcode_manager.sh — check usage, flip if needed, display sessions
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ it at mount time. Both protections together prevent this permanently.
|
||||
**lsof Per File on a Live TV System**
|
||||
Early cleanup called `lsof filename` per file to check if anything had it open. On a busy
|
||||
Live TV night with 5 simultaneous streams, the ramdisk contains thousands of HLS segment
|
||||
files — thousands of subprocess calls every 3 minutes.
|
||||
files — thousands of subprocess calls every 7 minutes.
|
||||
Fix: lsof called once per location to build a complete open-file map. All subsequent
|
||||
checks are O(1) lookups against that map.
|
||||
|
||||
@@ -82,7 +82,7 @@ unRAID_Essentials/
|
||||
Orchestrators/
|
||||
transcode_management.sh ──── cleanup first ──► transcode_cleanup.sh
|
||||
──── then manager ──► transcode_manager.sh
|
||||
(every 3 minutes — order non-negotiable)
|
||||
(every 7 minutes — order non-negotiable)
|
||||
|
||||
Monitors/
|
||||
weekly_health_digest.sh ◄─── reads ──────────── TRANSCODE_DAILY_LOG
|
||||
@@ -121,7 +121,7 @@ Emby starts, reads transcode path from config
|
||||
All new sessions write to: /mnt/ram-transcode → /mnt/ramdisk_transcodes/
|
||||
|
||||
|
||||
Every 3 minutes (transcode_management.sh):
|
||||
Every 7 minutes (transcode_management.sh):
|
||||
│
|
||||
├─ transcode_cleanup.sh
|
||||
│ Remove files older than TRANSCODE_MAX_AGE, not open by any process
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
#
|
||||
# lsof Called Once, Not Per File
|
||||
# On a busy Live TV system the ramdisk contains thousands of HLS segment files.
|
||||
# Calling lsof once per file creates thousands of subprocess calls every 3 minutes.
|
||||
# Calling lsof once per file creates thousands of subprocess calls every 7 minutes.
|
||||
# lsof is called once per location to build a complete open-file map. All subsequent
|
||||
# checks are O(1) lookups against that map — thousands of files, one lsof call.
|
||||
#
|
||||
@@ -59,7 +59,7 @@
|
||||
# `! -name "transcoding-temp"` in the find command — protected unconditionally.
|
||||
#
|
||||
# Silent by Default
|
||||
# Runs every 3 minutes — must not produce noise when healthy.
|
||||
# Runs every 7 minutes — must not produce noise when healthy.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Monitors ramdisk usage and manages the transcode symlink direction. Called by
|
||||
# transcode_management.sh (Orchestrators/) every 3 minutes — always after
|
||||
# transcode_management.sh (Orchestrators/) every 7 minutes — always after
|
||||
# transcode_cleanup.sh runs first. Must be fast, non-blocking, and silent when
|
||||
# nothing has changed.
|
||||
#
|
||||
@@ -59,7 +59,7 @@
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# Silent by Default
|
||||
# Runs every 3 minutes — only speaks when something changes or needs attention.
|
||||
# Runs every 7 minutes — only speaks when something changes or needs attention.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
|
||||
+10
-10
@@ -17,8 +17,8 @@
|
||||
# array_started.sh at array start — launches ALL startup scripts in order
|
||||
# array_stopping.sh at array stop — graceful shutdown sequence
|
||||
# watchdog_orchestrator.sh every minute — resource + docker + system watchdog
|
||||
# transcode_management.sh every 3 min — cleanup then manager (order critical)
|
||||
# critical_sync_maintenance.sh every 15 min — auth + Emby dirty sync + partnership
|
||||
# transcode_management.sh every 7 min — cleanup then manager (order critical)
|
||||
# critical_sync_maintenance.sh every 30 min — auth + Emby dirty sync + partnership
|
||||
# intermediate_sync_maintenance.sh every 4 hours — arr library sync + artwork fetch
|
||||
# daily_sync_maintenance.sh 1am daily — git + rsync + media + restart
|
||||
# rsync.sh --profile=emby-fallback every 30 min — Emby watch state dirty sync
|
||||
@@ -174,7 +174,7 @@
|
||||
|
||||
|
||||
# ── TRANSCODE MANAGEMENT ──────────────────────────────────────────────────────────────────────
|
||||
# Schedule: */3 * * * * (every 3 minutes)
|
||||
# Schedule: */7 * * * * (every 7 minutes)
|
||||
# Background: YES
|
||||
#
|
||||
# Cleanup first, then manager — order is non-negotiable. Stale segment files are removed
|
||||
@@ -202,15 +202,15 @@
|
||||
|
||||
|
||||
# ── CRITICAL SYNC MAINTENANCE ─────────────────────────────────────────────────────────────────
|
||||
# Schedule: */15 * * * * (every 15 minutes)
|
||||
# Schedule: */30 * * * * (every 30 minutes)
|
||||
# Background: YES
|
||||
#
|
||||
# Keeps failover-critical data current on HOST2. 15 minutes is the maximum staleness
|
||||
# Keeps fallback-critical data current on HOST2. 30 minutes is the maximum staleness
|
||||
# for the auth stack and Emby watch states. Run on HOST1 — bidirectional via detect_hosts().
|
||||
#
|
||||
# rsync Critical-Data:
|
||||
# Auth stack to HOST2: NPM proxy rules, TLS certs, LLDAP user accounts,
|
||||
# Authelia config and policies. HOST2 auth always within 15min of HOST1.
|
||||
# Authelia config and policies. HOST2 auth always within 30min of HOST1.
|
||||
#
|
||||
# rsync Emby dirty sync:
|
||||
# Watch states, user activity, library delta — Emby stays running on both sides.
|
||||
@@ -221,7 +221,7 @@
|
||||
# Reads remote state file. Increments offline counter on failed sync.
|
||||
# Triggers auto-offboard at PARTNERSHIP_OFFLINE_THRESHOLD (30 days) unreachable.
|
||||
#
|
||||
# Silent on clean cycles: runs 96 times per day, no output when healthy.
|
||||
# Silent on clean cycles: runs 48 times per day, no output when healthy.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/critical_sync_maintenance.sh
|
||||
|
||||
@@ -684,7 +684,7 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_update_remaining.sh
|
||||
|
||||
# downloaders_reset.sh — maintenance reset for all download clients on this server
|
||||
# Called every 15 minutes by critical_sync_maintenance.sh via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Called every 30 minutes by critical_sync_maintenance.sh via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Host-aware: sections skip gracefully if a downloader URL is empty for this server.
|
||||
#
|
||||
# slskd:
|
||||
@@ -1297,13 +1297,13 @@
|
||||
# * * * * * every minute:
|
||||
# Orchestrators/watchdog_orchestrator.sh
|
||||
#
|
||||
# */3 * * * * every 3 minutes:
|
||||
# */7 * * * * every 7 minutes:
|
||||
# Orchestrators/transcode_management.sh
|
||||
#
|
||||
# */10 * * * * every 10 minutes:
|
||||
# Watchdogs/System/webgui_watchdog.sh
|
||||
#
|
||||
# */15 * * * * every 15 minutes:
|
||||
# */30 * * * * every 30 minutes:
|
||||
# Orchestrators/critical_sync_maintenance.sh
|
||||
#
|
||||
# */30 * * * * every 30 minutes:
|
||||
|
||||
Reference in New Issue
Block a user