# Orchestrators Sequential job runners that coordinate multiple scripts into a single scheduled operation. Orchestrators do not contain business logic — they call other scripts in order, track pass/fail per job, and report a clean summary. All configuration lives in `Master.conf`. Adding or removing a job never requires touching the orchestrator script itself. --- ## Why Orchestrators Without orchestrators, each script runs independently on its own schedule. This works but creates problems: - **Race conditions** — two scripts running simultaneously on the same data - **Order dependency failures** — media cleaner runs before permissions, finds wrong ownership - **No combined summary** — 6 separate notifications instead of one clean report - **Scheduling complexity** — 6+ cron entries instead of one Orchestrators solve this by making a set of related scripts into a single scheduled unit with a defined execution order and a unified summary. --- ## Scripts ### `transcode_management.sh` Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every 3 minutes. Replaces two separate cron entries with one. ```bash # Scheduled as: */3 * * * * /mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh ``` **Why cleanup must run before manager:** If the manager runs first it may see inflated ramdisk usage from stale segment files left by ended sessions — and trigger an unnecessary flip to SSD. Cleanup runs first to clear those files, then the manager makes its threshold decision based on real active session usage. ``` Without correct order: Manager checks usage → 6.8GB (includes stale files) → flips to SSD Cleanup runs → removes stale files → actual usage 2.1GB Manager was wrong — unnecessary flip With correct order: Cleanup runs → removes stale files → actual usage 2.1GB Manager checks usage → 2.1GB → stays on ramdisk ✅ ``` **Daily statistics tracking:** Every cycle `transcode_management.sh` records stats to `/boot/config/transcode_daily.db`: - Peak ramdisk usage for the day - Total flip count for the day - Ramdisk vs SSD session counts - Files cleaned `weekly_health_digest.sh` reads this log for the weekly transcode summary. The log is bounded to `TRANSCODE_LOG_RETENTION` days — auto-purges on every write. --- ### `media_shares_sync.sh` Syncs each server's source-of-truth media shares to the remote server sequentially. Each server only pushes the shares it owns — direction and share list are automatic based on which server is running the script. ```bash # Scheduled as: 0 1 * * * (1am daily — on both servers) /mnt/user/appdata/unraid_scripts/Orchestrators/media_shares_sync.sh ``` **Bidirectional — same script, correct direction automatically:** ``` HOST1 runs media_shares_sync.sh → pushes HOST1_DAILY_SYNC_SHARES → TO HOST2 Movies, Tv_Shows, Music, Books etc. — HOST1 is source of truth HOST2 runs media_shares_sync.sh → pushes HOST2_DAILY_SYNC_SHARES → TO HOST1 Anime_Shows, Anime_Movies — HOST2 is source of truth ``` `detect_hosts()` determines which server is local at runtime and selects the correct share list. No script changes needed to reconfigure who syncs what — only `Master.conf` changes required. **What it does:** 1. Detects local server via `detect_hosts()` — determines HOST1 or HOST2 2. Resolves remote Tailscale IP 3. Runs a single pre-flight check — connectivity + remote rootfs 4. Builds share list from `HOST1_DAILY_SYNC_SHARES` or `HOST2_DAILY_SYNC_SHARES` 5. Appends personal shares (`HOST1_PERSONAL_SHARES` or `HOST2_PERSONAL_SHARES`) 6. Calls `Rsync/rsync.sh` for each share 7. Tracks pass/fail and duration per share 8. Reports a combined summary **Why one pre-flight check upfront:** Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast. Individual share existence and disk checks still run per-share inside `rsync.sh`. **Configuration:** ```bash # Master.conf — per-host share lists # HOST1 truth shares — pushed from HOST1 to HOST2 nightly HOST1_DAILY_SYNC_SHARES=( /mnt/user/Movies /mnt/user/Tv_Shows /mnt/user/Music # ... ) # HOST2 truth shares — pushed from HOST2 to HOST1 nightly HOST2_DAILY_SYNC_SHARES=( /mnt/user/Anime_Shows /mnt/user/Anime_Movies # ... ) ``` These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call `rsync.sh` directly on a separate schedule instead. **Relationship to failover writeback:** The same share lists are used by `failover.sh` for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 18hr+ and HOST2's arrs downloaded new content, Tier 4 writeback pushes `HOST1_DAILY_SYNC_SHARES` FROM HOST2 back TO HOST1. No duplicate configuration needed. **Example output:** ``` ━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━ 📋 Shares: 11 ━━━ [1/11] Movies ━━━ ...rsync output... ✅ Movies — 4m32s ━━━ [2/11] Tv_Shows ━━━ ... ━━━━━ 📋 DAILY SYNC SUMMARY ━━━━━ ✅ Pass: 10 ❌ Fail: 1 ⏱️ Duration: 47m12s ❌ Failed: Anime_Shows-Old ``` --- ### `critical_shares_full_sync.sh` Runs a clean nightly sync for Emby and the auth stack (Critical-Data) with containers stopped. This is the companion to the hourly dirty sync — it provides a fully consistent state on HOST2 once per night. ```bash # Scheduled as: 30 2 * * 0 (2:30am Sunday — weekly clean sync) /mnt/user/appdata/unraid_scripts/Orchestrators/critical_shares_full_sync.sh ``` **Why two Emby syncs:** The hourly dirty sync runs with Emby up — WAL files excluded, watch states pushed continuously. This means HOST2 is never more than an hour behind on watch state. But it's not a clean database snapshot. The nightly clean sync stops Emby, syncs the full clean database state, then restarts. HOST2 gets a fully consistent Emby state every night. The two syncs work together: ``` Hourly dirty sync (Emby running): users.db, library.db, authentication.db, config/ WAL excluded — safe mid-write HOST2 always within 1hr of HOST1 on watch state Nightly clean sync (Emby stopped): Full clean snapshot — all databases flushed No WAL files in flight HOST2 gets gold-standard state once per night ``` **Why clean auth sync matters:** The auth stack runs warm on both servers continuously. During normal operation HOST2's auth stack serves its own domain — it doesn't receive dirty updates from HOST1. The nightly clean sync is the only time auth state propagates. This means: - New user added on HOST1 → propagates to HOST2 overnight automatically - Proxy rule changes → propagated overnight - No manual intervention needed for most auth changes For users who just want failover to work — this script handles it. No thinking required about dirty writes, WAL files, or when to sync. **What it syncs:** ``` Emby appdata: users.db, library.db, authentication.db, config/ Containers stopped → clean flush → safe copy Critical-Data (auth stack): NPM proxy rules + SSL certs Authelia config + database Mariadb-Authelia data Redis-Authelia session store LLDAP users and groups database All auth containers stopped → clean databases → safe copy Authelia delayed start on restart — Mariadb + Redis must be ready first ``` **What it excludes (per rsync profile):** ``` Emby: logs, transcodes, cache, metadata, *.db-wal, *.db-shm Auth: logs, *.tmp, nginx/temp, nginx/cache, notification.txt ``` ### `media_management.sh` Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Scheduled once daily, typically after the nightly sync. ```bash # Scheduled as: 0 2 * * * (2am daily — after media_shares_sync.sh) /mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh ``` **What it does:** 1. Reads `MEDIA_MAINTENANCE_JOBS` from `Master.conf` 2. Runs each job in order — script path + optional argument 3. Tracks pass/fail per job 4. Reports a combined summary 5. A failure in one job does not stop the others **Why order matters:** ``` 1. media_shares_permissions.sh ← permissions first — everything else depends on correct ownership 2. media_cleaner.sh anime ← clean junk before arr scripts scan 3. media_cleaner.sh media ← same 4. lidarr_cleanup.sh ← arr cleanup last — depends on clean folders 5. sonarr_cleanup.sh 6. radarr_cleanup.sh ``` If arr cleanup runs before permissions, it may fail to delete files it doesn't have access to. If it runs before the cleaner, it finds junk files mixed in with real content. The order is intentional. **Configuration:** ```bash # Master.conf — add, remove, or reorder jobs here # Format: "folder/script.sh optional_argument" MEDIA_MAINTENANCE_JOBS=( "Media/media_shares_permissions.sh" "Media/media_cleaner.sh anime" "Media/media_cleaner.sh media" "Media/lidarr_cleanup.sh" "Media/sonarr_cleanup.sh" "Media/radarr_cleanup.sh" ) ``` **Adding a new job:** ```bash # Add a line to MEDIA_MAINTENANCE_JOBS — no script changes needed MEDIA_MAINTENANCE_JOBS=( "Media/media_shares_permissions.sh" "Media/media_cleaner.sh anime" "Media/media_cleaner.sh media" "Media/my_new_script.sh" # ← just add it here "Media/lidarr_cleanup.sh" "Media/sonarr_cleanup.sh" "Media/radarr_cleanup.sh" ) ``` **Disabling a job temporarily:** ```bash # Comment it out — easy to re-enable MEDIA_MAINTENANCE_JOBS=( "Media/media_shares_permissions.sh" # "Media/media_cleaner.sh anime" # ← disabled, not deleted "Media/media_cleaner.sh media" "Media/lidarr_cleanup.sh" "Media/sonarr_cleanup.sh" "Media/radarr_cleanup.sh" ) ``` **--dry-run support:** `media_management.sh --dry-run` passes `--dry-run` through to every child script. All scripts report what they would do without making changes. Useful for testing a new job before adding it to the live schedule. ```bash /mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run ``` --- ## The Orchestrator Pattern Both orchestrators follow the same pattern. This is by design — any script that needs to coordinate multiple operations should follow it: ``` 1. Setup — validate config, detect hosts if needed 2. Pre-flight — fail fast checks before doing any work 3. Job loop — run each job, track pass/fail, continue on failure 4. Summary — one clean report of all results 5. Notification — one notification per run, not one per job ``` This pattern means: - **Consistent output** — every orchestrator looks the same in the logs - **No silent failures** — pass/fail tracked per job, reported in summary - **Single notification** — one bell ring, not six - **Resilient** — one job failing doesn't stop the rest --- ## Scheduling ```bash # Recommended schedule */3 * * * * transcode_management.sh # cleanup then manager — every 3 minutes 0 1 * * * media_shares_sync.sh # 1am — media shares to remote 0 2 * * * media_management.sh # 2am — permissions, cleaners, arr cleanup 30 2 * * 0 critical_shares_full_sync.sh # 2:30am Sunday — clean Emby + auth stack ``` `media_shares_sync.sh` and `media_management.sh` run nightly — media shares and maintenance. `critical_shares_full_sync.sh` runs weekly on Sunday — it stops Emby and the auth stack for a clean consistent sync. Running it weekly instead of nightly lets Emby's image cache stay warm on HOST2 throughout the week. The emby-failover dirty sync handles watch states, library structure, and auth every 30-60 minutes — the weekly clean sync covers metadata, plugins, and a full database flush. --- ## Adding a New Orchestrator If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. The pattern is simple: ```bash # Minimal orchestrator skeleton JOBS=( "Folder/script1.sh" "Folder/script2.sh arg" ) PASS=() FAIL=() for JOB in "${JOBS[@]}"; do SCRIPT=$(echo "$JOB" | cut -d' ' -f1) ARG=$(echo "$JOB" | cut -d' ' -f2-) if bash "$ECOSYSTEM_ROOT/$SCRIPT" $ARG; then PASS+=("$SCRIPT") else FAIL+=("$SCRIPT") fi done ``` Better yet — model it directly on `media_management.sh` which already handles dry-run passthrough, status display, pass/fail tracking and summary reporting.