278 lines
9.8 KiB
Markdown
278 lines
9.8 KiB
Markdown
# Orchestrators
|
|
|
|
Sequential job runners that coordinate multiple scripts into a single scheduled operation.
|
|
|
|
Orchestrators do not contain business logic — they call other scripts in order, track pass/fail per job, and report a clean summary. All configuration lives in `Master.conf`. Adding or removing a job never requires touching the orchestrator script itself.
|
|
|
|
---
|
|
|
|
## Why Orchestrators
|
|
|
|
Without orchestrators, each script runs independently on its own schedule. This works but creates problems:
|
|
|
|
- **Race conditions** — two scripts running simultaneously on the same data
|
|
- **Order dependency failures** — media cleaner runs before permissions, finds wrong ownership
|
|
- **No combined summary** — 6 separate notifications instead of one clean report
|
|
- **Scheduling complexity** — 6+ cron entries instead of one
|
|
|
|
Orchestrators solve this by making a set of related scripts into a single scheduled unit with a defined execution order and a unified summary.
|
|
|
|
---
|
|
|
|
## Scripts
|
|
|
|
### `transcode_management.sh`
|
|
|
|
Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every 3 minutes. Replaces two separate cron entries with one.
|
|
|
|
```bash
|
|
# Scheduled as: */3 * * * *
|
|
/mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh
|
|
```
|
|
|
|
**Why cleanup must run before manager:**
|
|
|
|
If the manager runs first it may see inflated ramdisk usage from stale segment files left by ended sessions — and trigger an unnecessary flip to SSD. Cleanup runs first to clear those files, then the manager makes its threshold decision based on real active session usage.
|
|
|
|
```
|
|
Without correct order:
|
|
Manager checks usage → 6.8GB (includes stale files) → flips to SSD
|
|
Cleanup runs → removes stale files → actual usage 2.1GB
|
|
Manager was wrong — unnecessary flip
|
|
|
|
With correct order:
|
|
Cleanup runs → removes stale files → actual usage 2.1GB
|
|
Manager checks usage → 2.1GB → stays on ramdisk ✅
|
|
```
|
|
|
|
**Daily statistics tracking:**
|
|
|
|
Every cycle `transcode_management.sh` records stats to `/boot/config/transcode_daily.db`:
|
|
- Peak ramdisk usage for the day
|
|
- Total flip count for the day
|
|
- Ramdisk vs SSD session counts
|
|
- Files cleaned
|
|
|
|
`weekly_health_digest.sh` reads this log for the weekly transcode summary. The log is bounded to `TRANSCODE_LOG_RETENTION` days — auto-purges on every write.
|
|
|
|
---
|
|
|
|
### `daily_sync.sh`
|
|
|
|
Syncs each server's source-of-truth media shares to the remote server sequentially. Each server only pushes the shares it owns — direction and share list are automatic based on which server is running the script.
|
|
|
|
```bash
|
|
# Scheduled as: 0 1 * * * (1am daily — on both servers)
|
|
/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
|
|
```
|
|
|
|
**Bidirectional — same script, correct direction automatically:**
|
|
|
|
```
|
|
HOST1 runs daily_sync.sh → pushes HOST1_DAILY_SYNC_SHARES → TO HOST2
|
|
Movies, Tv_Shows, Music, Books etc. — HOST1 is source of truth
|
|
|
|
HOST2 runs daily_sync.sh → pushes HOST2_DAILY_SYNC_SHARES → TO HOST1
|
|
Anime_Shows, Anime_Movies — HOST2 is source of truth
|
|
```
|
|
|
|
`detect_hosts()` determines which server is local at runtime and selects the correct share list. No script changes needed to reconfigure who syncs what — only `Master.conf` changes required.
|
|
|
|
**What it does:**
|
|
1. Detects local server via `detect_hosts()` — determines HOST1 or HOST2
|
|
2. Resolves remote Tailscale IP
|
|
3. Runs a single pre-flight check — connectivity + remote rootfs
|
|
4. Builds share list from `HOST1_DAILY_SYNC_SHARES` or `HOST2_DAILY_SYNC_SHARES`
|
|
5. Appends personal shares (`HOST1_PERSONAL_SHARES` or `HOST2_PERSONAL_SHARES`)
|
|
6. Calls `Rsync/rsync.sh` for each share
|
|
7. Tracks pass/fail and duration per share
|
|
8. Reports a combined summary
|
|
|
|
**Why one pre-flight check upfront:**
|
|
Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast. Individual share existence and disk checks still run per-share inside `rsync.sh`.
|
|
|
|
**Configuration:**
|
|
```bash
|
|
# Master.conf — per-host share lists
|
|
# HOST1 truth shares — pushed from HOST1 to HOST2 nightly
|
|
HOST1_DAILY_SYNC_SHARES=(
|
|
/mnt/user/Movies
|
|
/mnt/user/Tv_Shows
|
|
/mnt/user/Music
|
|
# ...
|
|
)
|
|
|
|
# HOST2 truth shares — pushed from HOST2 to HOST1 nightly
|
|
HOST2_DAILY_SYNC_SHARES=(
|
|
/mnt/user/Anime_Shows
|
|
/mnt/user/Anime_Movies
|
|
# ...
|
|
)
|
|
```
|
|
|
|
These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call `rsync.sh` directly on a separate schedule instead.
|
|
|
|
**Relationship to failover writeback:**
|
|
|
|
The same share lists are used by `failover.sh` for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 18hr+ and HOST2's arrs downloaded new content, Tier 4 writeback pushes `HOST1_DAILY_SYNC_SHARES` FROM HOST2 back TO HOST1. No duplicate configuration needed.
|
|
|
|
**Example output:**
|
|
```
|
|
━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━
|
|
📋 Shares: 11
|
|
|
|
━━━ [1/11] Movies ━━━
|
|
...rsync output...
|
|
✅ Movies — 4m32s
|
|
|
|
━━━ [2/11] Tv_Shows ━━━
|
|
...
|
|
━━━━━ 📋 DAILY SYNC SUMMARY ━━━━━
|
|
✅ Pass: 10 ❌ Fail: 1
|
|
⏱️ Duration: 47m12s
|
|
❌ Failed: Anime_Shows-Old
|
|
```
|
|
|
|
---
|
|
|
|
### `media_management.sh`
|
|
|
|
Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Scheduled once daily, typically after the nightly sync.
|
|
|
|
```bash
|
|
# Scheduled as: 0 2 * * * (2am daily — after daily_sync.sh)
|
|
/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh
|
|
```
|
|
|
|
**What it does:**
|
|
1. Reads `MEDIA_MAINTENANCE_JOBS` from `Master.conf`
|
|
2. Runs each job in order — script path + optional argument
|
|
3. Tracks pass/fail per job
|
|
4. Reports a combined summary
|
|
5. A failure in one job does not stop the others
|
|
|
|
**Why order matters:**
|
|
|
|
```
|
|
1. media_shares_permissions.sh ← permissions first — everything else depends on correct ownership
|
|
2. media_cleaner.sh anime ← clean junk before arr scripts scan
|
|
3. media_cleaner.sh media ← same
|
|
4. lidarr_cleanup.sh ← arr cleanup last — depends on clean folders
|
|
5. sonarr_cleanup.sh
|
|
6. radarr_cleanup.sh
|
|
```
|
|
|
|
If arr cleanup runs before permissions, it may fail to delete files it doesn't have access to. If it runs before the cleaner, it finds junk files mixed in with real content. The order is intentional.
|
|
|
|
**Configuration:**
|
|
```bash
|
|
# Master.conf — add, remove, or reorder jobs here
|
|
# Format: "folder/script.sh optional_argument"
|
|
MEDIA_MAINTENANCE_JOBS=(
|
|
"Media/media_shares_permissions.sh"
|
|
"Media/media_cleaner.sh anime"
|
|
"Media/media_cleaner.sh media"
|
|
"Media/lidarr_cleanup.sh"
|
|
"Media/sonarr_cleanup.sh"
|
|
"Media/radarr_cleanup.sh"
|
|
)
|
|
```
|
|
|
|
**Adding a new job:**
|
|
```bash
|
|
# Add a line to MEDIA_MAINTENANCE_JOBS — no script changes needed
|
|
MEDIA_MAINTENANCE_JOBS=(
|
|
"Media/media_shares_permissions.sh"
|
|
"Media/media_cleaner.sh anime"
|
|
"Media/media_cleaner.sh media"
|
|
"Media/my_new_script.sh" # ← just add it here
|
|
"Media/lidarr_cleanup.sh"
|
|
"Media/sonarr_cleanup.sh"
|
|
"Media/radarr_cleanup.sh"
|
|
)
|
|
```
|
|
|
|
**Disabling a job temporarily:**
|
|
```bash
|
|
# Comment it out — easy to re-enable
|
|
MEDIA_MAINTENANCE_JOBS=(
|
|
"Media/media_shares_permissions.sh"
|
|
# "Media/media_cleaner.sh anime" # ← disabled, not deleted
|
|
"Media/media_cleaner.sh media"
|
|
"Media/lidarr_cleanup.sh"
|
|
"Media/sonarr_cleanup.sh"
|
|
"Media/radarr_cleanup.sh"
|
|
)
|
|
```
|
|
|
|
**--dry-run support:**
|
|
`media_management.sh --dry-run` passes `--dry-run` through to every child script. All scripts report what they would do without making changes. Useful for testing a new job before adding it to the live schedule.
|
|
|
|
```bash
|
|
/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run
|
|
```
|
|
|
|
---
|
|
|
|
## The Orchestrator Pattern
|
|
|
|
Both orchestrators follow the same pattern. This is by design — any script that needs to coordinate multiple operations should follow it:
|
|
|
|
```
|
|
1. Setup — validate config, detect hosts if needed
|
|
2. Pre-flight — fail fast checks before doing any work
|
|
3. Job loop — run each job, track pass/fail, continue on failure
|
|
4. Summary — one clean report of all results
|
|
5. Notification — one notification per run, not one per job
|
|
```
|
|
|
|
This pattern means:
|
|
- **Consistent output** — every orchestrator looks the same in the logs
|
|
- **No silent failures** — pass/fail tracked per job, reported in summary
|
|
- **Single notification** — one bell ring, not six
|
|
- **Resilient** — one job failing doesn't stop the rest
|
|
|
|
---
|
|
|
|
## Scheduling
|
|
|
|
```bash
|
|
# Recommended schedule
|
|
*/3 * * * * transcode_management.sh # cleanup then manager — every 3 minutes
|
|
0 1 * * * daily_sync.sh # 1am — media shares to remote
|
|
0 2 * * * media_management.sh # 2am — after sync completes
|
|
```
|
|
|
|
The 1 hour gap between them is intentional. `daily_sync.sh` can take 30-60 minutes on a large library. Starting `media_management.sh` before it finishes risks permission and cleanup operations running on files that are mid-transfer.
|
|
|
|
If your sync consistently finishes well under an hour, reduce the gap. If it regularly runs long, increase it.
|
|
|
|
---
|
|
|
|
## Adding a New Orchestrator
|
|
|
|
If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. The pattern is simple:
|
|
|
|
```bash
|
|
# Minimal orchestrator skeleton
|
|
JOBS=(
|
|
"Folder/script1.sh"
|
|
"Folder/script2.sh arg"
|
|
)
|
|
|
|
PASS=()
|
|
FAIL=()
|
|
|
|
for JOB in "${JOBS[@]}"; do
|
|
SCRIPT=$(echo "$JOB" | cut -d' ' -f1)
|
|
ARG=$(echo "$JOB" | cut -d' ' -f2-)
|
|
|
|
if bash "$ECOSYSTEM_ROOT/$SCRIPT" $ARG; then
|
|
PASS+=("$SCRIPT")
|
|
else
|
|
FAIL+=("$SCRIPT")
|
|
fi
|
|
done
|
|
```
|
|
|
|
Better yet — model it directly on `media_management.sh` which already handles dry-run passthrough, status display, pass/fail tracking and summary reporting. |