Varaverk/Orchestrators/README-Orchestrators.md

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🎯 ORCHESTRATORS

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Sequential job runners that coordinate multiple scripts into single scheduled units. Orchestrators contain no business logic — they call other scripts in order, track pass/fail per job, and produce one clean summary. Configuration lives in master.conf. Adding or removing a job never requires touching the orchestrator script itself.

The User Scripts plugin contains only orchestrators. Every cron entry, every "At Startup of Array" entry, every scheduled operation runs through an orchestrator. The individual scripts it calls are never scheduled directly — they run in a defined order inside a coordinated window, with a unified summary at the end.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

---

🔴 Race Conditions From Independent Scheduling

Media permissions ran at 01:05. Arr cleanup ran at 01:00. Arr cleanup started five minutes before permissions were applied — running against files that were still owned by root, silently failing to delete the ones it should have cleaned up. Both scripts reported success. Neither knew about the other. The result was a library that looked cleaned but wasn't.

The fix: orchestrators enforce order. daily_sync_maintenance.sh runs permissions first, then arr cleanup. The arr scripts see correct ownership every time because the orchestrator guarantees it. No race, no silent failure, no coordinating cron entries.

---

🔴 Six Separate Notifications Instead of One

Before orchestrators, each script sent its own notification on completion. A single daily run produced six separate notification pings — one for permissions, one for each arr cleanup, one for the cleaner, one for docker restart. Six bells for one maintenance window. Worse, if something failed in the middle, you'd get some notifications and not others, and figuring out which step failed meant correlating timestamps across multiple notification messages.

The fix: orchestrators collect all results and send one notification at the end. One summary. One bell. Clear pass/fail count. If something failed, the summary tells you which job and what happened — no correlation needed.

---

🔴 Transcode Manager Triggering Unnecessary SSD Flips

transcode_manager.sh ran every 3 minutes on its own. It checked ramdisk usage — saw 6.8GB used, threshold is 6.5GB, flipped sessions to SSD. One minute later transcode_cleanup.sh ran and removed 4GB of stale segment files from ended sessions. Actual usage was 2.8GB. Sessions were now on SSD for no reason. Users experiencing slightly worse performance. The flip counter incremented for nothing.

The fix: transcode_management.sh runs cleanup first, manager second, every cycle. The manager always sees post-cleanup usage. Stale files can't trigger a flip because they're gone before the manager looks. The correct order requires exactly one orchestrator to enforce it.

---

🔴 Failed Imports Sitting Stalled for Days

A release downloads successfully but Lidarr can't import it — wrong format, incorrect tags, file already exists. Lidarr marks it importFailed and stops. Nobody notices. The download client has the file, Lidarr has given up, and nothing is going to happen until someone manually opens Lidarr, identifies the problem, blocklists the release, and triggers a new search. This takes minutes to do — but nobody does it at 3am when it usually happens.

The fix: arrs_failed_stalled_recovery.sh runs every 6 hours. It finds all importFailed, importPending, error, and stalled items, blocklists them, removes them from the queue, and triggers a new search — automatically. By morning the failed import has already been replaced by a working one. No manual intervention required.

---

🔴 Array Start Scripts Running in Wrong Order or Not at All

Scripts configured in the User Scripts plugin as "At Startup of Array" run in an unpredictable order. The ramdisk setup might run after Emby starts. The syslog filter might run after containers have already created veth interfaces. PHP-FPM tuning might run after the WebGUI has already served its first requests. Each script competes for the same startup slot with no guaranteed order.

The fix: array_started.sh is the only "At Startup of Array" entry. It launches every startup script in a defined order, with one-second settle between each, and reports which succeeded and which failed. Order is guaranteed. Nothing starts before its dependency. Everything is visible in a single summary.

---

━━━ THE ORCHESTRATOR MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The User Scripts plugin contains exactly these entries:

# At Startup of Array — single entry for all startup scripts:
array_started.sh

# Cron — one entry per maintenance window:
*/3   * * * *    transcode_management.sh
0  */6 * * *     arrs_failed_stalled_recovery.sh
*/30  * * * *    rsync.sh emby-failover          ← not an orchestrator, direct call
0   1 * * *      daily_sync_maintenance.sh
30  2 * * 0      weekly_sync_maintenance.sh

# Manual only (not scheduled):
failover_test.sh, emby_database_repair.sh, repair tools

Every job list is configured in the ORCHESTRATORS section of master.conf. No orchestrator script ever changes when jobs are added or removed — only master.conf changes.

---

━━━ THE ORCHESTRATOR PATTERN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

All orchestrators follow the same structure:

1. Setup          → validate config, detect_hosts(), acquire_lock
2. Pre-flight     → fail fast checks before any work begins
3. Job loop       → run each job, track pass/fail, continue on failure
4. Summary        → one clean report of all job results
5. Notification   → one notify per run on failure (never per job)

Properties that apply to every orchestrator:

Consistent output    → every orchestrator looks the same in logs
No silent failures   → pass/fail tracked per job, all in summary
Resilient            → one job failing does not stop the rest
Single notification  → one bell per run, not one per job
--dry-run cascade    → passes --dry-run through to every child script
--status support     → show configured jobs and exit

---

━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

All scripts use a two-tier output model: echo lines are always visible; log lines only appear when --log is passed.

One-shot orchestrators (array_started.sh, array_stopping.sh, sunday_morning_coffee_report.sh): without --log, section headers, per-phase results, and the final summary are visible. Per-item detail suppressed.

Periodic orchestrators (critical_sync_maintenance.sh, daily_sync_maintenance.sh, intermediate_sync_maintenance.sh, weekly_sync_maintenance.sh): without --log, phase headers, per-phase completion status, and the final summary are visible. Per-share and per-job detail suppressed.

Daemon orchestrators (watchdog_orchestrator.sh, transcode_management.sh): silent during clean cycles. Only state transitions, errors, and startup-grace expiry shown without --log.

---

━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Script	What It Orchestrates	Schedule
array_started.sh	All array startup scripts in order	At Startup of Array
watchdog_orchestrator.sh	resource → docker → system → stability watchdogs	Every minute
transcode_management.sh	Cleanup then manager — order critical	Every 3 minutes
arrs_failed_stalled_recovery.sh	Failed import + stalled download recovery	Every 6 hours
daily_sync_maintenance.sh	git pull → sync → media maintenance → restarts	1am daily
weekly_sync_maintenance.sh	Stop → update → clean sync → start → weekly restarts	2:30am Sunday
monthly_maintenance.sh	Uptime-triggered heavy tasks — ZFS scrub, SMART tests	Daily check, fires when uptime ≥ 30d
media_management.sh	Permissions → cleaners → arr cleanup	Via daily_sync (or manual)

---

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🚀 array_started.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Single "At Startup of Array" entry for the entire ecosystem. Launches every startup script in order — each as a background process — and reports which succeeded and which failed. You never need to add individual scripts to the User Scripts startup list.

# Scheduled: At Startup of Array (User Scripts plugin)
# This is the ONLY "At Startup of Array" entry in User Scripts

---

── Execution Order ──────────────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Order matters — each entry depends on the previous ones having run.
# See comments for why each position is correct.
#
ARRAY_START_SCRIPTS=(
    # ── One-shot scripts — run and exit naturally ─────────────────────────────
    "unRAID_Essentials/inotify_tuning.sh"         # raise inotify BEFORE containers start
                                                   # containers inherit limits at startup —
                                                   # if Code-Server starts with low limits
                                                   # it keeps them until restart
    "unRAID_Essentials/docker_syslog_filter.sh"   # suppress veth noise BEFORE containers create
                                                   # veth interfaces — otherwise the first boot
                                                   # always has unfiltered veth spam
    "unRAID_Essentials/php_fpm_max_children.sh"   # WebGUI tuning — before any WebGUI requests
    "unRAID_Essentials/ramdisk_setup.sh"           # create tmpfs + symlink BEFORE Emby starts —
                                                   # Emby needs the transcode path to exist
    "Docker_Essentials/docker_network_connect.sh"  # ensure networks + connections BEFORE
                                                   # watchdogs check container states

    # ── Continuous scripts — run until array stops ─────────────────────────────
    "Watchdogs/stability_watchdog.sh"         # last line of defense — reboots when all else fails —
                                                   # system watchdog writes state file that
                                                   # docker watchdog reads every cycle
    "Watchdogs/docker_watchdog.sh"         # container health BEFORE failover —
                                                   # containers must be healthy for failover
                                                   # to make reliable decisions
    "Fallback/fallback.sh"                          # fallback LAST — needs everything else stable
)

---

── One-Shot vs Continuous Detection ───────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Each script is launched with `bash script.sh &` — background process.
# After 1 second:
#   PID still alive → continuous script (running in background)
#                     logged as: "fallback.sh — running (PID 12345)"
#   PID dead + exit 0 → one-shot completed successfully
#                        logged as: "inotify_tuning.sh — completed (one-shot)"
#   PID dead + exit N → failure
#                        logged as: "ramdisk_setup.sh — exited with code 1"
#                        full path printed — debugging is immediate
#
# This means the orchestrator correctly identifies and reports all startup
# scripts without needing to know in advance which ones are continuous.

---

── Auto-Fix Permissions ────────────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Scripts that are not executable are chmod +x'd automatically before launch.
# A permissions problem on a startup script does not cause a silent skip —
# the orchestrator fixes it and proceeds, then logs that it did so.
# This prevents "why didn't X run on startup" questions.

---

── Usage ───────────────────────────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Normal — called by User Scripts at array start. Never run manually in production.
# array_started.sh runs once and exits — the continuous scripts it launched
# keep running as background processes.
# ─────────────────────────────────────────────────────────────────────────────
array_started.sh

# ─────────────────────────────────────────────────────────────────────────────
# Dry run — show what would be launched, in order, without launching anything.
# Use to verify the ARRAY_START_SCRIPTS list before an array restart.
# ─────────────────────────────────────────────────────────────────────────────
array_started.sh --dry-run

# ─────────────────────────────────────────────────────────────────────────────
# Status — show each configured script with its current running state.
#   RUNNING (PID XXXXX) — continuous script currently active
#   not running          — one-shot that has completed, or continuous not yet started
#   FILE NOT FOUND       — script path wrong or missing
# ─────────────────────────────────────────────────────────────────────────────
array_started.sh --status

---

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🎬 transcode_management.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Runs transcode_cleanup.sh then transcode_manager.sh in the correct order every 3 minutes. Exists because the order is not optional — the manager must always see post-cleanup usage to make accurate flip decisions.

# Scheduled: */3 * * * *   (every 3 minutes)

---

── Why Order Is Non-Negotiable ─────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Emby writes segment files to the transcode directory as it buffers streams.
# When a stream ends, Emby deletes its own active files — but may leave behind
# stale segment files from sessions that ended uncleanly. These consume real
# ramdisk space. The manager has no way to know if they're active or stale.
#
# Without correct order:
#   Manager runs → sees 6.8GB used (stale files inflating) → exceeds threshold
#   → flips sessions to SSD → flip counter incremented
#   Cleanup runs → removes 4GB of stale files → actual usage was 2.8GB
#   → flip was unnecessary — sessions now on SSD for no reason
#
# With correct order (this orchestrator):
#   Cleanup runs → removes stale files → actual usage 2.8GB
#   Manager runs → sees 2.8GB → below threshold → stays on ramdisk ✅
#   → no flip, no wasted counter, correct decision every time
# ─────────────────────────────────────────────────────────────────────────────

---

── What Each Child Script Does ────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# transcode_cleanup.sh:
#   Identifies segment files not currently open by any process (via lsof)
#   Removes them from the transcode directory
#   If usage drops enough after cleanup → triggers flip-back to ramdisk
#   (handles the recovery direction so manager doesn't have to)
#
# transcode_manager.sh:
#   Reads current ramdisk usage after cleanup has run
#   Compares against RAMDISK_WARN_GB threshold
#   Flips the symlink if needed (ramdisk → SSD or SSD → ramdisk)
#   Writes one entry to TRANSCODE_DAILY_LOG for the weekly coffee report
#   Shows active Emby sessions with their play method
# ─────────────────────────────────────────────────────────────────────────────

---

── Daily Log ────────────────────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each cycle.
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
#
# The sunday_morning_coffee_report.sh reads this log for weekly stats:
#   Peak ramdisk usage across the week
#   Total flip count (unnecessary flips visible here)
#   Session split: how often ramdisk vs SSD was used
#
# Log trimmed to TRANSCODE_LOG_RETENTION days on every write — bounded, never grows.
#
    TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
    TRANSCODE_LOG_RETENTION=90    # days
    TRANSCODE_STATE_FILE="/tmp/transcode_state.db"    # /tmp — resets on reboot

---

── Usage ───────────────────────────────────────────────────────────────────

transcode_management.sh              # normal run (every 3 min via cron)
transcode_management.sh --dry-run    # passes --dry-run to both child scripts
transcode_management.sh --status     # show config, current state, daily log stats
transcode_management.sh --log        # verbose output from both child scripts

---

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔄 arrs_failed_stalled_recovery.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Automatic recovery from failed imports and stalled downloads across Sonarr, Radarr, and Lidarr. Blocklists the bad release, removes it from the queue, and triggers a new search — hands-free recovery while you sleep.

# Scheduled: 0 */6 * * *   (every 6 hours)

---

── Four Problem Types ───────────────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# The arr queue API reports these status values for problem items:
#
# importFailed   — downloaded successfully but arr couldn't import the file
#                  Common causes: wrong format for quality profile, corrupted file,
#                  duplicate already in library, permission issue on import path
#                  Self-resolution: never — arr stops trying after first failure
#
# importPending  — downloaded, stuck waiting for import to begin
#                  Common causes: import queue backed up, arr paused, API error
#                  Self-resolution: sometimes — but stuck for hours is always wrong
#
# error          — serious failure state not covered by the above
#                  Common causes: indexer issues, download client unreachable, disk full
#
# stalled        — download stuck with no connections or no progress
#                  Common causes: no seeders, VPN routing issue, tracker ban
#                  Self-resolution: never without a source change
#
# NOT touched — items with status "downloading" or "imported" are never touched.
# Safe to run at any time — only processes items that are already broken.
# ─────────────────────────────────────────────────────────────────────────────

---

── What It Does Per Problem Item ──────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# For each problem item found, in order:
#
# 1. Blocklist the release
#    Prevents the arr from grabbing the exact same release again immediately.
#    The bad indexer result goes into the blocklist — future searches skip it.
#
# 2. Remove from queue
#    Tells the download client to stop and remove the failed download.
#    Frees up the slot for the replacement.
#
# 3. Trigger new search
#    Arr searches for a different release meeting the quality profile.
#    If a suitable alternative exists, it starts downloading automatically.
#    If not, the item is marked as "awaiting upgrade" — arr will retry when
#    a new indexer result appears.
#
# The entire cycle from "failed import" to "replacement downloading" happens
# without any human involvement.
# ─────────────────────────────────────────────────────────────────────────────

---

── Age Threshold ────────────────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Items newer than ARR_IMPORT_RECOVERY_AGE hours are skipped.
# Arrs retry on their own schedule after initial failures — a 30-minute-old
# importFailed may still resolve itself. Waiting 6 hours before intervening
# gives the arr a full retry cycle before this script steps in.
#
    ARR_IMPORT_RECOVERY_AGE=6    # hours — skip items newer than this

---

── Host Awareness ───────────────────────────────────────────────────────────

# master.conf + host*.conf
# ─────────────────────────────────────────────────────────────────────────────
# Each arr is independently toggled per host.
# Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action.
# HOST2 has its own Sonarr and Radarr for its anime shares.
#
    HOST1_SONARR_RECOVERY=true    # HOST1 Sonarr — Tv_Shows
    HOST1_RADARR_RECOVERY=true    # HOST1 Radarr — Movies
    HOST1_LIDARR_RECOVERY=true    # HOST1 Lidarr — Music (HOST1 only)
    HOST2_SONARR_RECOVERY=true    # HOST2 Sonarr — Anime_Shows
    HOST2_RADARR_RECOVERY=true    # HOST2 Radarr — Anime_Movies
#
# API versions (current — update if arr major version changes):
#   Sonarr v4  → /api/v3/
#   Radarr v6  → /api/v3/
#   Lidarr v3  → /api/v1/

---

── Usage ───────────────────────────────────────────────────────────────────

arrs_failed_stalled_recovery.sh              # normal run
arrs_failed_stalled_recovery.sh --dry-run    # show what would be actioned, no changes
arrs_failed_stalled_recovery.sh --log        # verbose — show each queue item evaluated
arrs_failed_stalled_recovery.sh --status     # show arr config and API connectivity

---

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📅 daily_sync_maintenance.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Full daily maintenance window orchestrator. The entire 1am window — git pull, media share sync, media maintenance, and docker daily restarts — in one scheduled entry. Runs on both servers; detect_hosts() determines which direction each sync goes.

# Scheduled: 0 1 * * *   (1am daily — on BOTH servers)

---

── Execution Order ──────────────────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# The split into pre-sync and post-sync is based on a simple rule:
#   git pull runs before anything — maintenance uses the latest scripts
#   rsync runs in the middle — media management uses the post-sync state
#   docker restart runs last — after everything else has completed
#
# 1. Pre-sync: git_pull_execute.sh
#    Pull latest scripts first. Everything that follows runs on current code.
#
# 2. Rsync window: HOST*_DAILY_SYNC_SHARES + HOST*_PERSONAL_SHARES
#    Each server pushes its own truth shares to the other.
#    HOST1 → pushes Movies, Tv_Shows, Music → HOST2
#    HOST2 → pushes Anime_Shows, Anime_Movies → HOST1
#    Personal encrypted shares appended after standard shares.
#    Drive temperature exit codes respected — skip share or abort all on CRIT.
#
# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
#    media_management.sh → permissions + cleaners + arr cleanup
#    docker_daily_restart.sh → nightly container restarts
# ─────────────────────────────────────────────────────────────────────────────

---

── Bidirectional — Same Script, Correct Direction Automatic ────────────────

# ─────────────────────────────────────────────────────────────────────────────
# detect_hosts() sets MY_ID at runtime and aliases HOST*_DAILY_SYNC_SHARES
# to DAILY_SYNC_SHARES. The script uses DAILY_SYNC_SHARES — always the right
# list for whichever server is running.
#
# No HOST1/HOST2 comparisons in the script. Configuration drives direction.
#
# HOST1 runs this script at 1am:
#   → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
#   → media_management.sh on HOST1's shares
#   → docker_daily_restart.sh on HOST1's containers
#
# HOST2 runs this script at 1am:
#   → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
#   → media_management.sh on HOST2's shares
#   → docker_daily_restart.sh on HOST2's containers
# ─────────────────────────────────────────────────────────────────────────────

---

── Configuration ────────────────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Script lists — git_pull is split out as pre-sync; everything else is post-sync.
# The orchestrator recognises git_pull_execute.sh by name and routes it correctly.
#
DAILY_MAINTENANCE_SCRIPTS=(
    "git_pull_execute.sh"                          # PRE-SYNC — always first
    "Media/media_shares_permissions.sh"            # POST-SYNC — permissions before arr
    "Media/media_cleaner.sh anime"                 # POST-SYNC — junk before orphan scan
    "Media/media_cleaner.sh media"                 # POST-SYNC
    "Media/lidarr_cleanup.sh"                      # POST-SYNC — orphan cleanup last
    "Media/sonarr_cleanup.sh"                      # POST-SYNC
    "Media/radarr_cleanup.sh"                      # POST-SYNC
    "Docker_Essentials/docker_daily_restart.sh"    # POST-SYNC — restarts after everything
)

# host1.conf
HOST1_DAILY_SYNC_SHARES=(
    "/mnt/user/Movies"          # both servers — arr_sync union, rsync spreads files
    "/mnt/user/Tv_Shows"        # both servers
    "/mnt/user/Music"           # both servers
    "/mnt/user/Kids_Movies"
    "/mnt/user/Kids_Tv_Shows"
    "/mnt/user/Sports"
    "/mnt/user/stand-up_comedy"
)

HOST1_PERSONAL_SHARES=(
    "/mnt/user/Personal"        # encrypted personal share — appended after standard
)

# host2.conf
HOST2_DAILY_SYNC_SHARES=(
    "/mnt/user/Anime_Shows"     # both servers — arr_sync union, rsync spreads files
    "/mnt/user/Anime_Movies"    # both servers
)

---

── Adding or Removing a Job ────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Add a new media script — just insert it in the right position:
DAILY_MAINTENANCE_SCRIPTS=(
    "git_pull_execute.sh"
    "Media/media_shares_permissions.sh"
    "Media/media_cleaner.sh anime"
    "Media/media_cleaner.sh media"
    "Media/my_new_script.sh"               # ← add here in the correct order
    "Media/lidarr_cleanup.sh"
    "Media/sonarr_cleanup.sh"
    "Media/radarr_cleanup.sh"
    "Docker_Essentials/docker_daily_restart.sh"
)

# Disable a job temporarily — comment it out, do not delete:
DAILY_MAINTENANCE_SCRIPTS=(
    "git_pull_execute.sh"
    "Media/media_shares_permissions.sh"
#   "Media/media_cleaner.sh anime"         # ← temporarily disabled
    "Media/media_cleaner.sh media"
    "Media/lidarr_cleanup.sh"
    ...
)
# ─────────────────────────────────────────────────────────────────────────────
# No changes to daily_sync_maintenance.sh needed in either case.

---

── Drive Temperature Exit Codes ───────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# rsync.sh returns specific exit codes for temperature issues.
# The orchestrator handles these correctly — one hot drive does not abort all others.
#
#   exit 0 → success — continue to next share
#   exit 1 → temp WARN — skip this share, continue to next share
#   exit 2 → temp CRITICAL — abort ALL remaining shares in this window
#             notify immediately with which share triggered the abort
#   exit N → other failure — skip this share, continue to next share
# ─────────────────────────────────────────────────────────────────────────────

---

── Relationship to Failover Writeback ──────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# The same HOST*_DAILY_SYNC_SHARES lists are used by fallback.sh for Tier 4
# writeback — but in the opposite direction.
#
# Normal (daily_sync_maintenance.sh):
#   HOST1 → pushes Movies, Tv_Shows → HOST2
#
# Tier 4 failover writeback (HOST1 returns after 24hr+ outage):
#   HOST2 → pushes Movies, Tv_Shows → HOST1
#   (HOST2 was running HOST1's arrs and accumulated content)
#
# Same list, correct direction for the situation, zero duplication.
# No separate writeback list to maintain.
# ─────────────────────────────────────────────────────────────────────────────

---

── Usage ───────────────────────────────────────────────────────────────────

daily_sync_maintenance.sh              # normal run
daily_sync_maintenance.sh --dry-run    # preview all jobs without syncing or changing
daily_sync_maintenance.sh --log        # verbose per-share and per-job output
daily_sync_maintenance.sh --status     # show configured shares and jobs for this host

---

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📅 weekly_sync_maintenance.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Weekly maintenance window orchestrator — clean sync of Emby and auth stack, container image updates, and weekly docker restarts. Runs Sunday 2:30am; fits inside the Sunday maintenance block before the 7am coffee report.

# Scheduled: 30 2 * * 0   (Sunday 2:30am)

---

── Execution Order ──────────────────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Containers stop BEFORE sync — clean static source, full bandwidth.
# Containers start AFTER sync — on fresh data, in dependency order.
# DDNS and failover continue running throughout — only managed containers stop.
#
# 1. Pre-flight checks      — connectivity, remote Docker daemon, remote rootfs
# 2. Stop local containers  — Emby + auth stack stopped on this server
# 3. Stop remote containers — Emby + auth stack stopped on remote via SSH
# 4. Pull updates locally   — if WEEKLY_SYNC_UPDATES=true (containers already stopped)
# 5. Pull updates remotely  — if WEEKLY_SYNC_UPDATES_REMOTE=true
# 6. rsync WEEKLY_SYNC_SHARES — full clean mirror at full bandwidth
# 7. Start remote containers — new image, correct dependency order
# 8. Start local containers  — new image, correct dependency order
# Post-sync jobs: docker_weekly_restart.sh
# ─────────────────────────────────────────────────────────────────────────────

---

── Why Weekly Not Nightly for Emby ─────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Two Emby syncs run in parallel — dirty and clean:
#
# emby-failover dirty sync (every 30 minutes, Emby running):
#   watch states, library deltas, user activity — continuous coverage
#   WAL files excluded — safe to copy while Emby writes
#   HOST2 always within 30 minutes of HOST1 on playback state
#
# weekly clean sync (Sunday 2:30am, Emby stopped):
#   Full clean mirror — all databases checkpointed and flushed
#   Metadata, plugins, config all included
#   ~30 seconds of Emby downtime — both instances stopped during rsync only
#
# Why not nightly:
#   Emby builds a warm image thumbnail cache on HOST2 throughout the week.
#   Nightly sync resets this cache — cold loads every morning for users.
#   Weekly sync: cache stays warm for 6 days. Resets Sunday night while users sleep.
#   One weekly reset at an acceptable time is better than six unnecessary resets.
# ─────────────────────────────────────────────────────────────────────────────

---

── Container Update Window ─────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Containers are already stopped for the sync — pulling updated images costs
# nothing extra in downtime. Both servers start on the same image version
# after the window completes.
#
    WEEKLY_SYNC_UPDATES=true          # pull updated images locally
    WEEKLY_SYNC_UPDATES_REMOTE=true   # pull updated images on remote via SSH
#
# Toggle false to skip updates without changing the schedule:
#   WEEKLY_SYNC_UPDATES=false         # skips pulls, containers still restart on current image

---

── Why Auth Stack Weekly Sync Matters ──────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# NPM, Authelia, LLDAP, Mariadb run warm on both servers continuously.
# HOST1 is source of truth — changes propagate to HOST2 via weekly clean sync.
#
# What propagates automatically every Sunday:
#   New users added in LLDAP on HOST1       → appear on HOST2 by Monday
#   Proxy rules changed in NPM on HOST1     → live on HOST2 by Monday
#   Authelia policies updated on HOST1      → enforced on HOST2 by Monday
#   TLS certificates renewed on HOST1       → valid on HOST2 by Monday
#
# No manual sync needed for routine auth administration.
# Anything done on HOST1 is on HOST2 within a week.
# ─────────────────────────────────────────────────────────────────────────────

---

── Sunday Maintenance Window ───────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# This script is part of a coordinated Sunday maintenance block:
#
# 2:30am  weekly_sync_maintenance.sh   ← clean sync + image updates (~5-10min)
# 2:50am  CA Auto Update plugin        ← unRAID plugin updates
# 2:55am  CA container updates         ← additional container updates
# 3:00am  Network reboot               ← router/switch restart
#
# Everything comes back clean:
#   Network fresh, Emby + auth updated, containers on latest images.
#   All in one window while users sleep.
#   Sunday morning coffee report at 7am shows the post-maintenance state.
# ─────────────────────────────────────────────────────────────────────────────

---

── Configuration ────────────────────────────────────────────────────────────

# master.conf
    WEEKLY_SYNC_SHARES=(
        "/mnt/user/Media_Server/Emby"                    # emby profile — full clean mirror
        "/mnt/user/appdata-Failover/Critical-Data"       # critical-data — auth stack clean state
    )

    WEEKLY_MAINTENANCE_SCRIPTS=(
        "Docker_Essentials/docker_weekly_restart.sh"     # weekly restart of less-critical services
    )

    WEEKLY_SYNC_UPDATES=true
    WEEKLY_SYNC_UPDATES_REMOTE=true

---

── Usage ───────────────────────────────────────────────────────────────────

weekly_sync_maintenance.sh              # normal run
weekly_sync_maintenance.sh --dry-run    # preview — no stops, no syncs, no starts
weekly_sync_maintenance.sh --log        # verbose per-share per-job output
weekly_sync_maintenance.sh --status     # show configured shares, jobs, update toggles

---

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🧹 media_management.sh

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Runs all media maintenance scripts sequentially in the order defined by MEDIA_MAINTENANCE_JOBS in master.conf. Called by daily_sync_maintenance.sh as a post-sync job — not scheduled separately. Available for manual runs when media maintenance is needed outside the normal window.

# Called by: daily_sync_maintenance.sh (post-sync)
# Manual use: run directly for ad hoc media maintenance

---

── Execution Order ──────────────────────────────────────────────────────────

# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Order is critical — see README-Media.md for detailed explanation.
# Short version: permissions first, cleaners second, arr cleanup last.
# Each step depends on the previous having completed correctly.
#
MEDIA_MAINTENANCE_JOBS=(
    "Media/media_shares_permissions.sh"    # 1. permissions — arr cleanup depends on this
    "Media/media_cleaner.sh anime"         # 2. junk removal — orphan scan depends on this
    "Media/media_cleaner.sh media"         # 3. same for media shares
    "Media/lidarr_cleanup.sh"             # 4. orphan cleanup — last, after permissions + clean
    "Media/sonarr_cleanup.sh"             # 5.
    "Media/radarr_cleanup.sh"             # 6.
)
# ─────────────────────────────────────────────────────────────────────────────
# Add a script: insert in the correct position for your use case.
# Remove a script: comment it out with #
# No changes to media_management.sh needed in either case.

---

── Usage ───────────────────────────────────────────────────────────────────

media_management.sh              # normal run — all jobs in order
media_management.sh --dry-run    # preview without any deletions or changes
media_management.sh --log        # verbose output from all jobs
media_management.sh --status     # show configured job list and exit

---

━━━ COMPLETE SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

# ─────────────────────────────────────────────────────────────────────────────
# At Startup of Array — single entry:
# ─────────────────────────────────────────────────────────────────────────────
array_started.sh

# ─────────────────────────────────────────────────────────────────────────────
# Frequent — every 3 minutes:
# ─────────────────────────────────────────────────────────────────────────────
*/3 * * * *     transcode_management.sh

# ─────────────────────────────────────────────────────────────────────────────
# Periodic — not an orchestrator, direct rsync call:
# ─────────────────────────────────────────────────────────────────────────────
*/30 * * * *    rsync.sh /mnt/user/Media_Server/Emby --profile=emby-failover

# ─────────────────────────────────────────────────────────────────────────────
# Every 6 hours:
# ─────────────────────────────────────────────────────────────────────────────
0 */6 * * *     arrs_failed_stalled_recovery.sh

# ─────────────────────────────────────────────────────────────────────────────
# Daily — 1am:
#   git pull → rsync all truth shares → permissions + cleaners + arr cleanup → docker restart
# ─────────────────────────────────────────────────────────────────────────────
0 1 * * *       daily_sync_maintenance.sh

# ─────────────────────────────────────────────────────────────────────────────
# Weekly — Sunday maintenance block:
# ─────────────────────────────────────────────────────────────────────────────
30 2 * * 0      weekly_sync_maintenance.sh    # clean sync + updates (~5-10min)
50 2 * * 0      CA Auto Update plugin         # plugin updates
55 2 * * 0      CA container updates          # container image updates
0  3 * * 0      Network reboot                # router/switch restart

---

━━━ ADDING A NEW ORCHESTRATOR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

If you find yourself running 3+ related scripts on the same schedule, wrap them in a new orchestrator. Model directly on media_management.sh which has the complete pattern — dry-run passthrough, status display, pass/fail tracking, summary.

# Minimal skeleton — the full pattern in its simplest form:
#!/bin/bash
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"

SCRIPTS_ROOT="$SCRIPT_DIR/.."
PASS=()
FAIL=()

# Read job list from master.conf — never hardcode jobs in the orchestrator
for script_entry in "${MY_MAINTENANCE_JOBS[@]:-}"; do
    [[ -z "$script_entry" ]] && continue

    read -r -a parts <<< "$script_entry"
    script_path="$SCRIPTS_ROOT/${parts[0]}"
    script_name=$(basename "${parts[0]}")
    extra_args=("${parts[@]:1}")

    [[ "$DRY_RUN" == true ]] && extra_args+=("--dry-run")

    if bash "$script_path" "${extra_args[@]}"; then
        PASS+=("$script_name")
    else
        FAIL+=("$script_name")
    fi
done

# One summary — one notification
echo "Passed: ${#PASS[@]}  Failed: ${#FAIL[@]}"
[[ ${#FAIL[@]} -gt 0 ]] && \
    notify "My maintenance failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
        "My Orchestrator" "warning"