Varaverk/Orchestrators/README-orchestrators.md

Orchestrators

Sequential job runners that coordinate multiple scripts into a single scheduled operation.

Orchestrators do not contain business logic — they call other scripts in order, track pass/fail per job, and report a clean summary. All configuration lives in Master.conf. Adding or removing a job never requires touching the orchestrator script itself.

---

Why Orchestrators

Without orchestrators, each script runs independently on its own schedule. This works but creates problems:

• Race conditions — two scripts running simultaneously on the same data
• Order dependency failures — media cleaner runs before permissions, finds wrong ownership
• No combined summary — 6 separate notifications instead of one clean report
• Scheduling complexity — 6+ cron entries instead of one

Orchestrators solve this by making a set of related scripts into a single scheduled unit with a defined execution order and a unified summary.

---

Scripts

daily_sync.sh

Syncs all bulk media shares to the remote server sequentially. Scheduled once daily.

# Scheduled as: 0 1 * * *  (1am daily)
/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh

What it does:

1. Detects local and remote server via detect_hosts()
2. Resolves remote Tailscale IP
3. Runs a single pre-flight check — connectivity + remote rootfs
4. Iterates through every share in DAILY_SYNC_SHARES
5. Calls Rsync/rsync.sh for each share
6. Tracks pass/fail and duration per share
7. Reports a combined summary

Why one pre-flight check upfront: Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast before attempting 12 shares. Individual share existence and disk checks still run per-share inside rsync.sh.

Configuration:

# Master.conf — add or remove paths to control what syncs nightly
DAILY_SYNC_SHARES=(
    /mnt/user/Movies
    /mnt/user/Tv_Shows
    /mnt/user/Anime_Shows
    # ...
)

These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call rsync.sh directly on a separate schedule instead.

Example output:

━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━
📋 Shares: 11

━━━ [1/11] Movies ━━━
...rsync output...
✅ Movies — 4m32s

━━━ [2/11] Tv_Shows ━━━
...
━━━━━ 📋 DAILY SYNC SUMMARY ━━━━━
✅ Pass: 10   ❌ Fail: 1
⏱️  Duration: 47m12s
❌ Failed: Anime_Shows-Old

---

media_management.sh

Runs all media maintenance scripts sequentially in the order defined in Master.conf. Scheduled once daily, typically after the nightly sync.

# Scheduled as: 0 2 * * *  (2am daily — after daily_sync.sh)
/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh

What it does:

1. Reads MEDIA_MAINTENANCE_JOBS from Master.conf
2. Runs each job in order — script path + optional argument
3. Tracks pass/fail per job
4. Reports a combined summary
5. A failure in one job does not stop the others

Why order matters:

1. media_shares_permissions.sh  ← permissions first — everything else depends on correct ownership
2. media_cleaner.sh anime       ← clean junk before arr scripts scan
3. media_cleaner.sh media       ← same
4. lidarr_cleanup.sh            ← arr cleanup last — depends on clean folders
5. sonarr_cleanup.sh
6. radarr_cleanup.sh

If arr cleanup runs before permissions, it may fail to delete files it doesn't have access to. If it runs before the cleaner, it finds junk files mixed in with real content. The order is intentional.

Configuration:

# Master.conf — add, remove, or reorder jobs here
# Format: "folder/script.sh optional_argument"
MEDIA_MAINTENANCE_JOBS=(
    "Media/media_shares_permissions.sh"
    "Media/media_cleaner.sh anime"
    "Media/media_cleaner.sh media"
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)

Adding a new job:

# Add a line to MEDIA_MAINTENANCE_JOBS — no script changes needed
MEDIA_MAINTENANCE_JOBS=(
    "Media/media_shares_permissions.sh"
    "Media/media_cleaner.sh anime"
    "Media/media_cleaner.sh media"
    "Media/my_new_script.sh"            # ← just add it here
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)

Disabling a job temporarily:

# Comment it out — easy to re-enable
MEDIA_MAINTENANCE_JOBS=(
    "Media/media_shares_permissions.sh"
#   "Media/media_cleaner.sh anime"      # ← disabled, not deleted
    "Media/media_cleaner.sh media"
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)

--dry-run support: media_management.sh --dry-run passes --dry-run through to every child script. All scripts report what they would do without making changes. Useful for testing a new job before adding it to the live schedule.

/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run

---

The Orchestrator Pattern

Both orchestrators follow the same pattern. This is by design — any script that needs to coordinate multiple operations should follow it:

1. Setup          — validate config, detect hosts if needed
2. Pre-flight     — fail fast checks before doing any work
3. Job loop       — run each job, track pass/fail, continue on failure
4. Summary        — one clean report of all results
5. Notification   — one notification per run, not one per job

This pattern means:

• Consistent output — every orchestrator looks the same in the logs
• No silent failures — pass/fail tracked per job, reported in summary
• Single notification — one bell ring, not six
• Resilient — one job failing doesn't stop the rest

---

Scheduling

# Recommended schedule
0 1 * * *   daily_sync.sh         # 1am — media shares to remote
0 2 * * *   media_management.sh   # 2am — after sync completes

The 1 hour gap between them is intentional. daily_sync.sh can take 30-60 minutes on a large library. Starting media_management.sh before it finishes risks permission and cleanup operations running on files that are mid-transfer.

If your sync consistently finishes well under an hour, reduce the gap. If it regularly runs long, increase it.

---

Adding a New Orchestrator

If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. The pattern is simple:

# Minimal orchestrator skeleton
JOBS=(
    "Folder/script1.sh"
    "Folder/script2.sh arg"
)

PASS=()
FAIL=()

for JOB in "${JOBS[@]}"; do
    SCRIPT=$(echo "$JOB" | cut -d' ' -f1)
    ARG=$(echo "$JOB" | cut -d' ' -f2-)
    
    if bash "$ECOSYSTEM_ROOT/$SCRIPT" $ARG; then
        PASS+=("$SCRIPT")
    else
        FAIL+=("$SCRIPT")
    fi
done

Better yet — model it directly on media_management.sh which already handles dry-run passthrough, status display, pass/fail tracking and summary reporting.