Varaverk/Orchestrators/README-Orchestrators.md

# Orchestrators

Sequential job runners that coordinate multiple scripts into a single scheduled operation.

Orchestrators do not contain business logic — they call other scripts in order, track pass/fail per job, and report a clean summary. All configuration lives in `Master.conf`. Adding or removing a job never requires touching the orchestrator script itself.

---

## Why Orchestrators

Without orchestrators, each script runs independently on its own schedule. This works but creates problems:

- **Race conditions** — two scripts running simultaneously on the same data
- **Order dependency failures** — media cleaner runs before permissions, finds wrong ownership
- **No combined summary** — 6 separate notifications instead of one clean report
- **Scheduling complexity** — many cron entries instead of a few clean ones

Orchestrators solve this by making a set of related scripts into a single scheduled unit with a defined execution order and a unified summary.

---

## The Orchestrator Model

The ecosystem is designed so the User Scripts plugin contains only a small number of entries — each one an orchestrator that owns a domain:

```
At Startup of Array:
  array_start.sh                 ← single entry, launches everything

Cron:
  transcode_management.sh        ← */3 * * * *
  arrs_failed_stalled_recovery.sh ← 0 */6 * * *
  rsync.sh ... emby-failover     ← */30 * * * *
  daily_sync_maintenance.sh      ← 0 1 * * *
  weekly_sync_maintenance.sh     ← 30 2 * * 0
  weekly_health_digest.sh        ← Saturday morning

Manual only:
  failover_test.sh, emby_database_repair.sh, repair tools
```

All job lists are configured in the `ORCHESTRATORS` section of `Master.conf`. No changes to orchestrator scripts needed when adding or removing jobs.

---

## Scripts

### `array_start.sh`

Single entry point for the User Scripts "At Startup of Array" schedule. Launches all array-start scripts in order — each as a background process.

```bash
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh
```

One-shot scripts (ramdisk, syslog filter, php-fpm, network connect) run and exit naturally. Continuous scripts (system watchdog, docker watchdog, failover) run until the array stops.

**Configuration:**

```bash
# Master.conf — ORCHESTRATORS section
ARRAY_START_SCRIPTS=(
    "unRAID_Essentials/ramdisk_setup.sh"          # creates ramdisk before Emby starts
    "unRAID_Essentials/docker_syslog_filter.sh"   # suppress veth log noise
    "unRAID_Essentials/php_fpm_max_children.sh"   # WebGUI tuning
    "Docker_Essentials/docker_network_connect.sh" # connect containers to extra networks
    "unRAID_Essentials/system_watchdog.sh"        # continuous system health monitor
    "Docker_Essentials/docker_watchdog.sh"        # continuous container health monitor
    "Failover/failover.sh"                        # continuous mutual failover
)
```

Add or remove scripts from `ARRAY_START_SCRIPTS` — no changes to `array_start.sh` needed. Order matters — ramdisk before Emby, network before watchdogs, watchdogs before failover.

---

### `transcode_management.sh`

Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every 3 minutes. Replaces two separate cron entries with one.

```bash
# Scheduled as: */3 * * * *
/mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh
```

**Why cleanup must run before manager:**

If the manager runs first it sees inflated ramdisk usage from stale segment files left by ended sessions — and triggers an unnecessary flip to SSD. Cleanup runs first to clear those files, then the manager decides based on real active session usage.

```
Without correct order:
  Manager checks usage → 6.8GB (includes stale files) → flips to SSD
  Cleanup runs → removes stale files → actual usage 2.1GB
  Unnecessary flip — sessions now on SSD

With correct order:
  Cleanup runs → removes stale files → actual usage 2.1GB
  Manager checks usage → 2.1GB → stays on ramdisk ✅
```

**Daily statistics tracking:**

Every cycle `transcode_management.sh` records stats to `TRANSCODE_DAILY_LOG`:
- Peak ramdisk usage for the day
- Total flip count for the day
- Ramdisk vs SSD session counts
- Files cleaned

`weekly_health_digest.sh` reads this log for the weekly transcode summary. The log is bounded to `TRANSCODE_LOG_RETENTION` days — auto-purges on every write.

---

### `arrs_failed_stalled_recovery.sh`

Automatically detects and recovers from failed imports and stalled downloads across Sonarr, Radarr, and Lidarr. Blocklists the bad release and triggers a new search — hands-free recovery while you sleep.

```bash
# Scheduled as: 0 */6 * * *  (every 6 hours)
/mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh
```

Targets four problem types: `importFailed`, `importPending`, `error` status, and `stalled` downloads. Items newer than `ARR_IMPORT_RECOVERY_AGE` (6 hours) are skipped — gives the arr time to retry on its own first.

**API versions:** Sonarr v4 → `/api/v3/` — Radarr v6 → `/api/v3/` — Lidarr v3 → `/api/v1/`

Lidarr runs on HOST1 only — exits cleanly on HOST2.

**Configuration:**

```bash
# Master.conf — MEDIA section
ARR_IMPORT_RECOVERY_AGE=6      # hours — skip items newer than this

HOST1_SONARR_RECOVERY=true
HOST1_RADARR_RECOVERY=true
HOST1_LIDARR_RECOVERY=true
HOST2_SONARR_RECOVERY=true
HOST2_RADARR_RECOVERY=true
```

---

### `daily_sync_maintenance.sh`

Full daily maintenance window orchestrator — git pull, media share sync, media management, and docker daily restarts. All driven by `Master.conf` arrays.

```bash
# Scheduled as: 0 1 * * *  (1am daily — on both servers)
/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh
```

**Execution order:**

```
1. Pre-sync jobs (DAILY_MAINTENANCE_SCRIPTS — git pull first):
   git_pull_execute.sh          ← always runs first — pulls latest scripts

2. Media share sync:
   HOST*_DAILY_SYNC_SHARES      ← each server pushes its own truth shares
   HOST*_PERSONAL_SHARES        ← personal encrypted shares appended after

3. Post-sync jobs (DAILY_MAINTENANCE_SCRIPTS — remaining):
   media_management.sh          ← permissions + cleaners + arr cleanup
   docker_daily_restart.sh      ← daily container restarts
```

**Bidirectional — same script, correct direction automatically:**

```
HOST1 runs daily_sync_maintenance.sh:
  git pull → sync HOST1_DAILY_SYNC_SHARES → TO HOST2 → media_management → docker restart

HOST2 runs daily_sync_maintenance.sh:
  git pull → sync HOST2_DAILY_SYNC_SHARES → TO HOST1 → media_management → docker restart
```

`detect_hosts()` determines which server is local at runtime and selects the correct share list. No script changes needed — only `Master.conf` changes required.

**Why one pre-flight check upfront:**

Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or rootfs is nearly full, the whole run fails fast. Individual share checks still run per-share inside `rsync.sh`.

**Configuration:**

```bash
# Master.conf — ORCHESTRATORS section

DAILY_MAINTENANCE_SCRIPTS=(
    "git_pull_execute.sh"                         # always first
    "Docker_Essentials/docker_daily_restart.sh"   # after sync and media jobs
)

# Media jobs run between sync and docker restart
# Permissions first, cleaners second, arr cleanup last
MEDIA_MANAGEMENT_JOBS=(
    "Media/media_shares_permissions.sh"           # permissions — everything depends on this
    "Media/media_cleaner.sh anime"                # clean junk before arr scripts scan
    "Media/media_cleaner.sh media"
    "Media/lidarr_cleanup.sh"                     # arr cleanup last — depends on clean folders
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)

HOST1_DAILY_SYNC_SHARES=(
    /mnt/user/Movies
    /mnt/user/Tv_Shows
    /mnt/user/Music
    # all HOST1-owned shares
)

HOST2_DAILY_SYNC_SHARES=(
    /mnt/user/Anime_Shows
    /mnt/user/Anime_Movies
)
```

**Adding a media job:**

```bash
MEDIA_MANAGEMENT_JOBS=(
    "Media/media_shares_permissions.sh"
    "Media/media_cleaner.sh anime"
    "Media/media_cleaner.sh media"
    "Media/my_new_script.sh"            # ← just add it here
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)
```

**Disabling a media job temporarily:**

```bash
MEDIA_MANAGEMENT_JOBS=(
    "Media/media_shares_permissions.sh"
#   "Media/media_cleaner.sh anime"      # ← disabled, not deleted
    "Media/media_cleaner.sh media"
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
)
```

**Relationship to failover writeback:**

The same share lists are used by `failover.sh` for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 24hr+ and HOST2's arrs accumulated content, writeback pushes `HOST1_DAILY_SYNC_SHARES` FROM HOST2 BACK TO HOST1. No duplicate configuration needed.

---

### `weekly_sync_maintenance.sh`

Weekly maintenance window orchestrator — critical appdata clean sync, container updates, and weekly docker restarts. Runs Sunday 2:30am, fits before the 3am network reboot.

```bash
# Scheduled as: 30 2 * * 0  (Sunday 2:30am)
/mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh
```

**Execution order:**

```
1. Stop local containers   — auth stack + Emby stopped locally
2. Stop remote containers  — auth stack + Emby stopped remotely via SSH
3. Pull updates locally    — if CRITICAL_SYNC_UPDATES=true
4. Pull updates remotely   — if CRITICAL_SYNC_UPDATES_REMOTE=true
5. rsync WEEKLY_SYNC_JOBS  — Emby + Critical-Data clean sync
6. Start remote containers — starts on new images, correct order
7. Start local containers  — starts on new images, correct order

Post-sync jobs (WEEKLY_MAINTENANCE_SCRIPTS):
8. docker_weekly_restart.sh
```

**Why two Emby syncs:**

The emby-failover dirty sync runs every 30-60 minutes with Emby running — WAL files excluded, watch states and library pushed continuously. HOST2 stays current on what users are watching. But it is not a clean database snapshot.

The weekly clean sync stops Emby on both sides, checkpoints the WAL, and pushes a full consistent mirror. HOST2 gets a gold-standard Emby state once per week.

```
emby-failover every 30-60min (Emby running):
  users.db, library.db, authentication.db, config/
  WAL excluded — safe mid-write
  HOST2 always within 30-60min of HOST1 on watch state

weekly clean sync Sunday 2:30am (Emby stopped):
  Full clean mirror — all databases flushed
  metadata, plugins, config all included
  ~30s downtime — both Emby instances down during sync only
  Cache stays warm on HOST2 all week — only reset Sunday
```

**Why weekly instead of nightly:**

Emby builds a warm image cache on HOST2 naturally throughout the week. Syncing nightly resets this cache — users experience slow image loads every morning. Weekly sync lets the cache stay warm for 6 days and only resets on Sunday night when most users are asleep.

**What it syncs:**

```
WEEKLY_SYNC_JOBS (configurable in Master.conf):
  /mnt/user/Media_Server/Emby         ← emby profile — full clean mirror
  /mnt/user/appdata-Failover/Critical-Data ← critical-data profile — auth stack

Emby excludes:    logs, transcodes, cache, crash*
Auth excludes:    logs, *.tmp, nginx/temp, nginx/cache, notification.txt
```

**Container update window:**

Containers are already stopped for the sync — container image updates pull at zero extra downtime. Both servers start on the same new image version after the sync.

```bash
# Master.conf toggles
CRITICAL_SYNC_UPDATES=true          # pull updates locally
CRITICAL_SYNC_UPDATES_REMOTE=true   # pull updates on remote via SSH

# Toggle false to skip updates without changing the schedule
CRITICAL_SYNC_UPDATES=false
```

**Why auth stack matters:**

The auth stack (Authelia, NPM, Mariadb, Redis, LLDAP) runs warm on both servers. During normal operation HOST2 serves its own domain independently. The weekly clean sync is the only time auth state propagates from HOST1 to HOST2.

- New user added on HOST1 → propagates to HOST2 on Sunday automatically
- Proxy rule changes → propagated Sunday
- No manual intervention needed for routine auth changes

**Sunday maintenance window:**

```
2:30am  weekly_sync_maintenance.sh   ← clean sync + updates (~3-5min)
2:50am  CA Auto Update plugin        ← plugin updates
2:55am  CA container updates         ← docker container updates
3:00am  Network reboot               ← router/switch restart

Everything comes back clean:
  Network fresh, Emby updated, auth stack updated
  All in one maintenance window while users sleep
```

**Configuration:**

```bash
# Master.conf — ORCHESTRATORS section

WEEKLY_SYNC_JOBS=(
    "/mnt/user/Media_Server/Emby"
    "/mnt/user/appdata-Failover/Critical-Data"
)

WEEKLY_MAINTENANCE_SCRIPTS=(
    "Docker_Essentials/docker_weekly_restart.sh"
)

CRITICAL_SYNC_UPDATES=true
CRITICAL_SYNC_UPDATES_REMOTE=true
```

---

### `media_management.sh`

Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Absorbed into `daily_sync_maintenance.sh` via `MEDIA_MANAGEMENT_JOBS` — not scheduled separately. Available for manual runs.

```bash
# Manual use only — called automatically by daily_sync_maintenance.sh
bash /mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run
bash /mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh
```

---

## The Orchestrator Pattern

All orchestrators follow the same pattern:

```
1. Setup          — validate config, detect hosts if needed, acquire lock
2. Pre-flight     — fail fast checks before doing any work
3. Job loop       — run each job, track pass/fail, continue on failure
4. Summary        — one clean report of all results
5. Notification   — one notification per run, not one per job
```

This means:
- **Consistent output** — every orchestrator looks the same in logs
- **No silent failures** — pass/fail tracked per job, reported in summary
- **Single notification** — one bell ring per run
- **Resilient** — one job failing doesn't stop the rest

---

## Scheduling

```bash
# At Startup of Array
array_start.sh                          # single entry — launches all startup scripts

# Every 3 minutes
*/3 * * * *    transcode_management.sh

# Every 6 hours
0 */6 * * *    arrs_failed_stalled_recovery.sh

# Every 30-60 minutes
*/30 * * * *   rsync.sh /mnt/user/Media_Server/Emby --profile=emby-failover

# Daily 1am — full daily maintenance window:
#   git pull → media sync → permissions → cleaners → arr cleanup → docker restart
0 1 * * *      daily_sync_maintenance.sh

# Weekly — Sunday morning
30 2 * * 0     weekly_sync_maintenance.sh   # clean sync + updates + docker weekly restart
50 2 * * 0     CA plugin update
55 2 * * 0     CA container updates
```

`daily_sync_maintenance.sh` owns the entire daily window — git pull, media sync, permissions, cleaners, arr cleanup, and docker restarts in one scheduled run. Everything configured in `Master.conf` via `DAILY_MAINTENANCE_SCRIPTS` and `MEDIA_MANAGEMENT_JOBS`.

---

## Adding a New Orchestrator

If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. Model it directly on `media_management.sh` which handles dry-run passthrough, status display, pass/fail tracking and summary reporting.

Minimal skeleton:

```bash
JOBS=(
    "Folder/script1.sh"
    "Folder/script2.sh arg"
)

PASS=()
FAIL=()

for JOB in "${JOBS[@]}"; do
    SCRIPT=$(echo "$JOB" | cut -d' ' -f1)
    ARG=$(echo "$JOB" | cut -d' ' -f2-)

    if bash "$ECOSYSTEM_ROOT/$SCRIPT" $ARG; then
        PASS+=("$SCRIPT")
    else
        FAIL+=("$SCRIPT")
    fi
done
```