Varaverk/Orchestrators/README-Orchestrators.md

Orchestrators

Sequential job runners that coordinate multiple scripts into a single scheduled operation.

Orchestrators do not contain business logic — they call other scripts in order, track pass/fail per job, and report a clean summary. All configuration lives in Master.conf. Adding or removing a job never requires touching the orchestrator script itself.

---

Why Orchestrators

Without orchestrators, each script runs independently on its own schedule. This works but creates problems:

• Race conditions — two scripts running simultaneously on the same data
• Order dependency failures — media cleaner runs before permissions, finds wrong ownership
• No combined summary — 6 separate notifications instead of one clean report
• Scheduling complexity — 6+ cron entries instead of one

Orchestrators solve this by making a set of related scripts into a single scheduled unit with a defined execution order and a unified summary.

---

Scripts

transcode_management.sh

Runs transcode_cleanup.sh then transcode_manager.sh in the correct order every 3 minutes. Replaces two separate cron entries with one.

# Scheduled as: */3 * * * *
/mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh

Why cleanup must run before manager:

If the manager runs first it may see inflated ramdisk usage from stale segment files left by ended sessions — and trigger an unnecessary flip to SSD. Cleanup runs first to clear those files, then the manager makes its threshold decision based on real active session usage.

Without correct order:
  Manager checks usage → 6.8GB (includes stale files) → flips to SSD
  Cleanup runs → removes stale files → actual usage 2.1GB
  Manager was wrong — unnecessary flip

With correct order:
  Cleanup runs → removes stale files → actual usage 2.1GB
  Manager checks usage → 2.1GB → stays on ramdisk ✅

Daily statistics tracking:

Every cycle transcode_management.sh records stats to /boot/config/transcode_daily.db:

• Peak ramdisk usage for the day
• Total flip count for the day
• Ramdisk vs SSD session counts
• Files cleaned

weekly_health_digest.sh reads this log for the weekly transcode summary. The log is bounded to TRANSCODE_LOG_RETENTION days — auto-purges on every write.

---

daily_sync.sh

Syncs each server's source-of-truth media shares to the remote server sequentially. Each server only pushes the shares it owns — direction and share list are automatic based on which server is running the script.

# Scheduled as: 0 1 * * *  (1am daily — on both servers)
/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh

Bidirectional — same script, correct direction automatically:

HOST1 runs daily_sync.sh → pushes HOST1_DAILY_SYNC_SHARES → TO HOST2
  Movies, Tv_Shows, Music, Books etc. — HOST1 is source of truth

HOST2 runs daily_sync.sh → pushes HOST2_DAILY_SYNC_SHARES → TO HOST1
  Anime_Shows, Anime_Movies — HOST2 is source of truth

detect_hosts() determines which server is local at runtime and selects the correct share list. No script changes needed to reconfigure who syncs what — only Master.conf changes required.

What it does:

1. Detects local server via detect_hosts() — determines HOST1 or HOST2
2. Resolves remote Tailscale IP
3. Runs a single pre-flight check — connectivity + remote rootfs
4. Builds share list from HOST1_DAILY_SYNC_SHARES or HOST2_DAILY_SYNC_SHARES
5. Appends personal shares (HOST1_PERSONAL_SHARES or HOST2_PERSONAL_SHARES)
6. Calls Rsync/rsync.sh for each share
7. Tracks pass/fail and duration per share
8. Reports a combined summary

Why one pre-flight check upfront: Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast. Individual share existence and disk checks still run per-share inside rsync.sh.

Configuration:

# Master.conf — per-host share lists
# HOST1 truth shares — pushed from HOST1 to HOST2 nightly
HOST1_DAILY_SYNC_SHARES=(
    /mnt/user/Movies
    /mnt/user/Tv_Shows
    /mnt/user/Music
    # ...
)

# HOST2 truth shares — pushed from HOST2 to HOST1 nightly
HOST2_DAILY_SYNC_SHARES=(
    /mnt/user/Anime_Shows
    /mnt/user/Anime_Movies
    # ...
)

These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call rsync.sh directly on a separate schedule instead.

Relationship to failover writeback:

The same share lists are used by failover.sh for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 18hr+ and HOST2's arrs downloaded new content, Tier 4 writeback pushes HOST1_DAILY_SYNC_SHARES FROM HOST2 back TO HOST1. No duplicate configuration needed.

Example output:

━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━
📋 Shares: 11

━━━ [1/11] Movies ━━━
...rsync output...
✅ Movies — 4m32s

━━━ [2/11] Tv_Shows ━━━
...
━━━━━ 📋 DAILY SYNC SUMMARY ━━━━━
✅ Pass: 10   ❌ Fail: 1
⏱️  Duration: 47m12s
❌ Failed: Anime_Shows-Old

---

media_management.sh

Runs all media maintenance scripts sequentially in the order defined in Master.conf. Scheduled once daily, typically after the nightly sync.

# Scheduled as: 0 2 * * *  (2am daily — after daily_sync.sh)
/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh

What it does:

1. Reads MEDIA_MAINTENANCE_JOBS from Master.conf
2. Runs each job in order — script path + optional argument
3. Tracks pass/fail per job
4. Reports a combined summary
5. A failure in one job does not stop the others

Why order matters:

1. media_shares_permissions.sh  ← permissions first — everything else depends on correct ownership
2. media_cleaner.sh anime       ← clean junk before arr scripts scan
3. media_cleaner.sh media       ← same
4. lidarr_cleanup.sh            ← arr cleanup last — depends on clean folders
5. sonarr_cleanup.sh
6. radarr_cleanup.sh

If arr cleanup runs before permissions, it may fail to delete files it doesn't have access to. If it runs before the cleaner, it finds junk files mixed in with real content. The order is intentional.

Configuration:

# Master.conf — add, remove, or reorder jobs here
# Format: "folder/script.sh optional_argument"
MEDIA_MAINTENANCE_JOBS=(
    "Media/media_shares_permissions.sh"
    "Media/media_cleaner.sh anime"
    "Media/media_cleaner.sh media"
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)

Adding a new job:

# Add a line to MEDIA_MAINTENANCE_JOBS — no script changes needed
MEDIA_MAINTENANCE_JOBS=(
    "Media/media_shares_permissions.sh"
    "Media/media_cleaner.sh anime"
    "Media/media_cleaner.sh media"
    "Media/my_new_script.sh"            # ← just add it here
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)

Disabling a job temporarily:

# Comment it out — easy to re-enable
MEDIA_MAINTENANCE_JOBS=(
    "Media/media_shares_permissions.sh"
#   "Media/media_cleaner.sh anime"      # ← disabled, not deleted
    "Media/media_cleaner.sh media"
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)

--dry-run support: media_management.sh --dry-run passes --dry-run through to every child script. All scripts report what they would do without making changes. Useful for testing a new job before adding it to the live schedule.

/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run

---

The Orchestrator Pattern

Both orchestrators follow the same pattern. This is by design — any script that needs to coordinate multiple operations should follow it:

1. Setup          — validate config, detect hosts if needed
2. Pre-flight     — fail fast checks before doing any work
3. Job loop       — run each job, track pass/fail, continue on failure
4. Summary        — one clean report of all results
5. Notification   — one notification per run, not one per job

This pattern means:

• Consistent output — every orchestrator looks the same in the logs
• No silent failures — pass/fail tracked per job, reported in summary
• Single notification — one bell ring, not six
• Resilient — one job failing doesn't stop the rest

---

Scheduling

# Recommended schedule
*/3 * * * *   transcode_management.sh    # cleanup then manager — every 3 minutes
0 1 * * *     daily_sync.sh             # 1am — media shares to remote
0 2 * * *     media_management.sh       # 2am — after sync completes

The 1 hour gap between them is intentional. daily_sync.sh can take 30-60 minutes on a large library. Starting media_management.sh before it finishes risks permission and cleanup operations running on files that are mid-transfer.

If your sync consistently finishes well under an hour, reduce the gap. If it regularly runs long, increase it.

---

Adding a New Orchestrator

If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. The pattern is simple:

# Minimal orchestrator skeleton
JOBS=(
    "Folder/script1.sh"
    "Folder/script2.sh arg"
)

PASS=()
FAIL=()

for JOB in "${JOBS[@]}"; do
    SCRIPT=$(echo "$JOB" | cut -d' ' -f1)
    ARG=$(echo "$JOB" | cut -d' ' -f2-)
    
    if bash "$ECOSYSTEM_ROOT/$SCRIPT" $ARG; then
        PASS+=("$SCRIPT")
    else
        FAIL+=("$SCRIPT")
    fi
done

Better yet — model it directly on media_management.sh which already handles dry-run passthrough, status display, pass/fail tracking and summary reporting.