55 KiB
55 KiB
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🎯 ORCHESTRATORS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sequential job runners that coordinate multiple scripts into single scheduled units.
Orchestrators contain no business logic — they call other scripts in order, track
pass/fail per job, and produce one clean summary. Configuration lives in master.conf.
Adding or removing a job never requires touching the orchestrator script itself.
The Varaverk scheduler runs only orchestrators. Every cron entry, every array start/stop event, every scheduled operation runs through an orchestrator. The individual scripts it calls are never scheduled directly — they run in a defined order inside a coordinated window, with a unified summary at the end.
━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔴 Race Conditions From Independent Scheduling
Media permissions ran at 01:05. Arr cleanup ran at 01:00. Arr cleanup started five minutes before permissions were applied — running against files that were still owned by root, silently failing to delete the ones it should have cleaned up. Both scripts reported success. Neither knew about the other. The result was a library that looked cleaned but wasn't.
The fix: orchestrators enforce order. daily_sync_maintenance.sh runs permissions
first, then arr cleanup. The arr scripts see correct ownership every time because the
orchestrator guarantees it. No race, no silent failure, no coordinating cron entries.
🔴 Six Separate Notifications Instead of One
Before orchestrators, each script sent its own notification on completion. A single daily run produced six separate notification pings — one for permissions, one for each arr cleanup, one for the cleaner, one for docker restart. Six bells for one maintenance window. Worse, if something failed in the middle, you'd get some notifications and not others, and figuring out which step failed meant correlating timestamps across multiple notification messages.
The fix: orchestrators collect all results and send one notification at the end. One summary. One bell. Clear pass/fail count. If something failed, the summary tells you which job and what happened — no correlation needed.
🔴 Transcode Manager Triggering Unnecessary SSD Flips
transcode_manager.sh ran every 7 minutes on its own. It checked ramdisk usage —
saw 6.8GB used, threshold is 6.5GB, flipped sessions to SSD. One minute later
transcode_cleanup.sh ran and removed 4GB of stale segment files from ended sessions.
Actual usage was 2.8GB. Sessions were now on SSD for no reason. Users experiencing
slightly worse performance. The flip counter incremented for nothing.
The fix: transcode_management.sh runs cleanup first, manager second, every cycle.
The manager always sees post-cleanup usage. Stale files can't trigger a flip because
they're gone before the manager looks. The correct order requires exactly one
orchestrator to enforce it.
🔴 Failed Imports Sitting Stalled for Days
A release downloads successfully but Lidarr can't import it — wrong format, incorrect
tags, file already exists. Lidarr marks it importFailed and stops. Nobody notices.
The download client has the file, Lidarr has given up, and nothing is going to happen
until someone manually opens Lidarr, identifies the problem, blocklists the release,
and triggers a new search. This takes minutes to do — but nobody does it at 3am
when it usually happens.
The fix: arrs_failed_stalled_recovery.sh runs every 6 hours. It finds all
importFailed, importPending, error, and stalled items, blocklists them, removes
them from the queue, and triggers a new search — automatically. By morning the failed
import has already been replaced by a working one. No manual intervention required.
🔴 Array Start Scripts Running in Wrong Order or Not at All
Startup scripts configured individually ran in an unpredictable order. The ramdisk setup might run after Emby starts. The syslog filter might run after containers have already created veth interfaces. PHP-FPM tuning might run after the WebGUI has already served its first requests. Each script competed for the same startup slot with no guaranteed order.
The fix: array_started.sh is the only array-start entry in the Varaverk scheduler.
It launches every startup script in a defined order, with one-second settle between
each, and reports which succeeded and which failed. Order is guaranteed. Nothing starts
before its dependency. Everything is visible in a single summary.
━━━ THE ORCHESTRATOR MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The Varaverk scheduler contains exactly these entries:
# Array start event (Varaverk disks_mounted hook → cron: "array_start"):
array_started.sh
# Cron — managed via Varaverk Scheduler:
*/7 * * * * transcode_management.sh
*/15 * * * * watchdog_orchestrator.sh ← resource → docker → system → stability
*/30 * * * * critical_sync_maintenance.sh ← auth + Emby dirty sync + partnership
0 */4 * * * intermediate_sync_maintenance.sh ← arr sync + failed recovery + optional rsync
0 1 * * * daily_sync_maintenance.sh
0 7 * * 0 sunday_morning_coffee_report.sh
30 2 * * 0 weekly_sync_maintenance.sh
0 0 15 * * monthly_maintenance.sh ← uptime-gated: ZFS scrub, SMART tests
# Manual only (not scheduled):
fallback_test.sh, emby_database_repair.sh, repair tools
Every job list is configured in the ORCHESTRATORS section of master.conf.
No orchestrator script ever changes when jobs are added or removed — only master.conf changes.
━━━ THE ORCHESTRATOR PATTERN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All orchestrators follow the same structure:
1. Setup → validate config, detect_hosts(), acquire_lock
2. Pre-flight → fail fast checks before any work begins
3. Job loop → run each job, track pass/fail, continue on failure
4. Summary → one clean report of all job results
5. Notification → one notify per run on failure (never per job)
Properties that apply to every orchestrator:
Consistent output → every orchestrator looks the same in logs
No silent failures → pass/fail tracked per job, all in summary
Resilient → one job failing does not stop the rest
Single notification → one bell per run, not one per job
--dry-run cascade → passes --dry-run through to every child script
--status support → show configured jobs and exit
━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All scripts use a two-tier output model: echo lines are always visible; log
lines only appear when --log is passed.
One-shot orchestrators (array_started.sh, array_stopping.sh, sunday_morning_coffee_report.sh):
without --log, section headers, per-phase results, and the final summary are
visible. Per-item detail suppressed.
Periodic orchestrators (critical_sync_maintenance.sh, daily_sync_maintenance.sh,
intermediate_sync_maintenance.sh, weekly_sync_maintenance.sh): without --log,
phase headers, per-phase completion status, and the final summary are visible. Per-share
and per-job detail suppressed.
High-frequency orchestrators (watchdog_orchestrator.sh, transcode_management.sh): silent
during clean cycles. Only state transitions, errors, and startup-grace expiry shown
without --log.
━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
| Script | What It Orchestrates | Schedule |
|---|---|---|
array_started.sh |
All array startup scripts in order | array_start event (Varaverk plugin hook) |
watchdog_orchestrator.sh |
resource → docker → system → api_renew → stability watchdogs | Every 15 minutes |
transcode_management.sh |
Cleanup then manager — order critical | Every 7 minutes |
arrs_failed_stalled_recovery.sh |
Failed import + stalled download recovery | Every 6 hours |
daily_sync_maintenance.sh |
git pull → sync → media maintenance → restarts | 1am daily |
weekly_sync_maintenance.sh |
Stop → update → clean sync → start → weekly restarts | 2:30am Sunday |
monthly_maintenance.sh |
Uptime-triggered heavy tasks — ZFS scrub, SMART tests | Daily check, fires when uptime ≥ 30d |
intermediate_sync_maintenance.sh |
arr library sync, artwork, failed recovery | Every 4 hours |
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🚀 array_started.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Single array-start entry for the entire ecosystem. Fired by the Varaverk plugin's
disks_mounted event hook. Launches every startup script in order — each as a
background process — and reports which succeeded and which failed.
# Triggered by: Plugin/unraid/event/disks_mounted/array_start_jobs
# schedule.json entry: "Orchestrators/array_started.sh" → cron: "array_start"
── Execution Order ──────────────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Order matters — each entry depends on the previous ones having run.
# See comments for why each position is correct.
#
ARRAY_START_SCRIPTS=(
# ── One-shot scripts — run and exit naturally ─────────────────────────────
"unRAID_Essentials/unraid_api_key_renew.sh" # re-register API key FIRST — unraid-api
# registry is ephemeral, lost on service restart
"unRAID_Essentials/inotify_tuning.sh" # raise inotify BEFORE containers start
# containers inherit limits at startup —
# if Code-Server starts with low limits
# it keeps them until restart
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise BEFORE containers create
# veth interfaces — otherwise the first boot
# always has unfiltered veth spam
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI tuning — before any WebGUI requests
"Transcodes/ramdisk_setup.sh" # create tmpfs + symlink BEFORE Emby starts —
# Emby needs the transcode path to exist
"Docker_Essentials/docker_network_connect.sh" # ensure networks + connections BEFORE
# watchdogs check container states
# ── Continuous scripts — run until array stops ─────────────────────────────
"Fallback/fallback.sh" # fallback LAST — needs everything else stable
)
# NOTE: watchdogs (docker_watchdog, system_watchdog, stability_watchdog) are NOT here.
# They run via watchdog_orchestrator.sh on cron every 15 minutes — not as daemons.
── One-Shot vs Continuous Detection ───────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Each script is launched with `bash script.sh &` — background process.
# After 1 second:
# PID still alive → continuous script (running in background)
# logged as: "fallback.sh — running (PID 12345)"
# PID dead + exit 0 → one-shot completed successfully
# logged as: "inotify_tuning.sh — completed (one-shot)"
# PID dead + exit N → failure
# logged as: "ramdisk_setup.sh — exited with code 1"
# full path printed — debugging is immediate
#
# This means the orchestrator correctly identifies and reports all startup
# scripts without needing to know in advance which ones are continuous.
── Auto-Fix Permissions ────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Scripts that are not executable are chmod +x'd automatically before launch.
# A permissions problem on a startup script does not cause a silent skip —
# the orchestrator fixes it and proceeds, then logs that it did so.
# This prevents "why didn't X run on startup" questions.
── Usage ───────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Normal — fired by Varaverk disks_mounted event hook. Never run manually in production.
# array_started.sh runs once and exits — the continuous scripts it launched
# keep running as background processes.
# ─────────────────────────────────────────────────────────────────────────────
array_started.sh
# ─────────────────────────────────────────────────────────────────────────────
# Dry run — show what would be launched, in order, without launching anything.
# Use to verify the ARRAY_START_SCRIPTS list before an array restart.
# ─────────────────────────────────────────────────────────────────────────────
array_started.sh --dry-run
# ─────────────────────────────────────────────────────────────────────────────
# Status — show each configured script with its current running state.
# RUNNING (PID XXXXX) — continuous script currently active
# not running — one-shot that has completed, or continuous not yet started
# FILE NOT FOUND — script path wrong or missing
# ─────────────────────────────────────────────────────────────────────────────
array_started.sh --status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🎬 transcode_management.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Runs transcode_cleanup.sh then transcode_manager.sh in the correct order every
7 minutes. Exists because the order is not optional — the manager must always see
post-cleanup usage to make accurate flip decisions.
# Scheduled: */7 * * * * (every 7 minutes)
── Why Order Is Non-Negotiable ─────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Emby writes segment files to the transcode directory as it buffers streams.
# When a stream ends, Emby deletes its own active files — but may leave behind
# stale segment files from sessions that ended uncleanly. These consume real
# ramdisk space. The manager has no way to know if they're active or stale.
#
# Without correct order:
# Manager runs → sees 6.8GB used (stale files inflating) → exceeds threshold
# → flips sessions to SSD → flip counter incremented
# Cleanup runs → removes 4GB of stale files → actual usage was 2.8GB
# → flip was unnecessary — sessions now on SSD for no reason
#
# With correct order (this orchestrator):
# Cleanup runs → removes stale files → actual usage 2.8GB
# Manager runs → sees 2.8GB → below threshold → stays on ramdisk ✅
# → no flip, no wasted counter, correct decision every time
# ─────────────────────────────────────────────────────────────────────────────
── What Each Child Script Does ────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# transcode_cleanup.sh:
# Identifies segment files not currently open by any process (via lsof)
# Removes them from the transcode directory
# If usage drops enough after cleanup → triggers flip-back to ramdisk
# (handles the recovery direction so manager doesn't have to)
#
# transcode_manager.sh:
# Reads current ramdisk usage after cleanup has run
# Compares against RAMDISK_WARN_GB threshold
# Flips the symlink if needed (ramdisk → SSD or SSD → ramdisk)
# Writes one entry to TRANSCODE_DAILY_LOG for the weekly coffee report
# Shows active Emby sessions with their play method
# ─────────────────────────────────────────────────────────────────────────────
── Daily Log ────────────────────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each cycle.
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
#
# The sunday_morning_coffee_report.sh reads this log for weekly stats:
# Peak ramdisk usage across the week
# Total flip count (unnecessary flips visible here)
# Session split: how often ramdisk vs SSD was used
#
# Log trimmed to TRANSCODE_LOG_RETENTION days on every write — bounded, never grows.
#
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
TRANSCODE_LOG_RETENTION=90 # days
TRANSCODE_STATE_FILE="/tmp/transcode_state.db" # /tmp — resets on reboot
── Usage ───────────────────────────────────────────────────────────────────
transcode_management.sh # normal run (every 7 min via cron)
transcode_management.sh --dry-run # passes --dry-run to both child scripts
transcode_management.sh --status # show config, current state, daily log stats
transcode_management.sh --log # verbose output from both child scripts
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔄 arrs_failed_stalled_recovery.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Automatic recovery from failed imports and stalled downloads across Sonarr, Radarr, and Lidarr. Blocklists the bad release, removes it from the queue, and triggers a new search — hands-free recovery while you sleep.
# Scheduled: 0 */6 * * * (every 6 hours)
── Four Problem Types ───────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# The arr queue API reports these status values for problem items:
#
# importFailed — downloaded successfully but arr couldn't import the file
# Common causes: wrong format for quality profile, corrupted file,
# duplicate already in library, permission issue on import path
# Self-resolution: never — arr stops trying after first failure
#
# importPending — downloaded, stuck waiting for import to begin
# Common causes: import queue backed up, arr paused, API error
# Self-resolution: sometimes — but stuck for hours is always wrong
#
# error — serious failure state not covered by the above
# Common causes: indexer issues, download client unreachable, disk full
#
# stalled — download stuck with no connections or no progress
# Common causes: no seeders, VPN routing issue, tracker ban
# Self-resolution: never without a source change
#
# NOT touched — items with status "downloading" or "imported" are never touched.
# Safe to run at any time — only processes items that are already broken.
# ─────────────────────────────────────────────────────────────────────────────
── What It Does Per Problem Item ──────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# For each problem item found, in order:
#
# 1. Blocklist the release
# Prevents the arr from grabbing the exact same release again immediately.
# The bad indexer result goes into the blocklist — future searches skip it.
#
# 2. Remove from queue
# Tells the download client to stop and remove the failed download.
# Frees up the slot for the replacement.
#
# 3. Trigger new search
# Arr searches for a different release meeting the quality profile.
# If a suitable alternative exists, it starts downloading automatically.
# If not, the item is marked as "awaiting upgrade" — arr will retry when
# a new indexer result appears.
#
# The entire cycle from "failed import" to "replacement downloading" happens
# without any human involvement.
# ─────────────────────────────────────────────────────────────────────────────
── Age Threshold ────────────────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Items newer than ARR_IMPORT_RECOVERY_AGE hours are skipped.
# Arrs retry on their own schedule after initial failures — a 30-minute-old
# importFailed may still resolve itself. Waiting 6 hours before intervening
# gives the arr a full retry cycle before this script steps in.
#
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
── Host Awareness ───────────────────────────────────────────────────────────
# master.conf + host*.conf
# ─────────────────────────────────────────────────────────────────────────────
# Each arr is independently toggled per host.
# Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action.
# HOST2 has its own Sonarr and Radarr for its anime shares.
#
HOST1_SONARR_RECOVERY=true # HOST1 Sonarr — Tv_Shows
HOST1_RADARR_RECOVERY=true # HOST1 Radarr — Movies
HOST1_LIDARR_RECOVERY=true # HOST1 Lidarr — Music (HOST1 only)
HOST2_SONARR_RECOVERY=true # HOST2 Sonarr — Anime_Shows
HOST2_RADARR_RECOVERY=true # HOST2 Radarr — Anime_Movies
#
# API versions (current — update if arr major version changes):
# Sonarr v4 → /api/v3/
# Radarr v6 → /api/v3/
# Lidarr v3 → /api/v1/
── Usage ───────────────────────────────────────────────────────────────────
arrs_failed_stalled_recovery.sh # normal run
arrs_failed_stalled_recovery.sh --dry-run # show what would be actioned, no changes
arrs_failed_stalled_recovery.sh --log # verbose — show each queue item evaluated
arrs_failed_stalled_recovery.sh --status # show arr config and API connectivity
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📅 daily_sync_maintenance.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Full daily maintenance window orchestrator. The entire 1am window — git pull, media
share sync, media maintenance, and docker daily restarts — in one scheduled entry.
Runs on both servers; detect_hosts() determines which direction each sync goes.
# Scheduled: 0 1 * * * (1am daily — on BOTH servers)
── Execution Order ──────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# The split into pre-sync and post-sync is based on a simple rule:
# git pull runs before anything — maintenance uses the latest scripts
# rsync runs in the middle — media management uses the post-sync state
# docker restart runs last — after everything else has completed
#
# 1. Pre-sync: git_pull_execute.sh
# Pull latest scripts first. Everything that follows runs on current code.
#
# 2. Rsync window: HOST*_DAILY_SYNC_SHARES + HOST*_PERSONAL_SHARES
# Each server pushes its own truth shares to the other.
# HOST1 → pushes Movies, Tv_Shows, Music → HOST2
# HOST2 → pushes Anime_Shows, Anime_Movies → HOST1
# Personal encrypted shares appended after standard shares.
# Drive temperature exit codes respected — skip share or abort all on CRIT.
#
# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
# media_management.sh → permissions + cleaners + arr cleanup
# docker_daily_restart.sh → nightly container restarts
# ─────────────────────────────────────────────────────────────────────────────
── Bidirectional — Same Script, Correct Direction Automatic ────────────────
# ─────────────────────────────────────────────────────────────────────────────
# detect_hosts() sets MY_ID at runtime and aliases HOST*_DAILY_SYNC_SHARES
# to DAILY_SYNC_SHARES. The script uses DAILY_SYNC_SHARES — always the right
# list for whichever server is running.
#
# No HOST1/HOST2 comparisons in the script. Configuration drives direction.
#
# HOST1 runs this script at 1am:
# → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
# → media_management.sh on HOST1's shares
# → docker_daily_restart.sh on HOST1's containers
#
# HOST2 runs this script at 1am:
# → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
# → media_management.sh on HOST2's shares
# → docker_daily_restart.sh on HOST2's containers
# ─────────────────────────────────────────────────────────────────────────────
── Configuration ────────────────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Script lists — git_pull is split out as pre-sync; everything else is post-sync.
# The orchestrator recognises git_pull_execute.sh by name and routes it correctly.
#
DAILY_MAINTENANCE_SCRIPTS=(
"git_pull_execute.sh" # PRE-SYNC — always first
"Media/media_shares_permissions.sh" # POST-SYNC — permissions before arr
"Media/media_cleaner.sh anime" # POST-SYNC — junk before orphan scan
"Media/media_cleaner.sh media" # POST-SYNC
"Media/lidarr_cleanup.sh" # POST-SYNC — orphan cleanup last
"Media/sonarr_cleanup.sh" # POST-SYNC
"Media/radarr_cleanup.sh" # POST-SYNC
"Docker_Essentials/docker_daily_restart.sh" # POST-SYNC — restarts after everything
)
# host1.conf
HOST1_DAILY_SYNC_SHARES=(
"/mnt/user/Movies" # both servers — arr_sync union, rsync spreads files
"/mnt/user/Tv_Shows" # both servers
"/mnt/user/Music" # both servers
"/mnt/user/Kids_Movies"
"/mnt/user/Kids_Tv_Shows"
"/mnt/user/Sports"
"/mnt/user/stand-up_comedy"
)
HOST1_PERSONAL_SHARES=(
"/mnt/user/Personal" # encrypted personal share — appended after standard
)
# host2.conf
HOST2_DAILY_SYNC_SHARES=(
"/mnt/user/Anime_Shows" # both servers — arr_sync union, rsync spreads files
"/mnt/user/Anime_Movies" # both servers
)
── Adding or Removing a Job ────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Add a new media script — just insert it in the right position:
DAILY_MAINTENANCE_SCRIPTS=(
"git_pull_execute.sh"
"Media/media_shares_permissions.sh"
"Media/media_cleaner.sh anime"
"Media/media_cleaner.sh media"
"Media/my_new_script.sh" # ← add here in the correct order
"Media/lidarr_cleanup.sh"
"Media/sonarr_cleanup.sh"
"Media/radarr_cleanup.sh"
"Docker_Essentials/docker_daily_restart.sh"
)
# Disable a job temporarily — comment it out, do not delete:
DAILY_MAINTENANCE_SCRIPTS=(
"git_pull_execute.sh"
"Media/media_shares_permissions.sh"
# "Media/media_cleaner.sh anime" # ← temporarily disabled
"Media/media_cleaner.sh media"
"Media/lidarr_cleanup.sh"
...
)
# ─────────────────────────────────────────────────────────────────────────────
# No changes to daily_sync_maintenance.sh needed in either case.
── Drive Temperature Exit Codes ───────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# rsync.sh returns specific exit codes for temperature issues.
# The orchestrator handles these correctly — one hot drive does not abort all others.
#
# exit 0 → success — continue to next share
# exit 1 → temp WARN — skip this share, continue to next share
# exit 2 → temp CRITICAL — abort ALL remaining shares in this window
# notify immediately with which share triggered the abort
# exit N → other failure — skip this share, continue to next share
# ─────────────────────────────────────────────────────────────────────────────
── Relationship to Failover Writeback ──────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# The same HOST*_DAILY_SYNC_SHARES lists are used by fallback.sh for Tier 4
# writeback — but in the opposite direction.
#
# Normal (daily_sync_maintenance.sh):
# HOST1 → pushes Movies, Tv_Shows → HOST2
#
# Tier 4 failover writeback (HOST1 returns after 24hr+ outage):
# HOST2 → pushes Movies, Tv_Shows → HOST1
# (HOST2 was running HOST1's arrs and accumulated content)
#
# Same list, correct direction for the situation, zero duplication.
# No separate writeback list to maintain.
# ─────────────────────────────────────────────────────────────────────────────
── Usage ───────────────────────────────────────────────────────────────────
daily_sync_maintenance.sh # normal run
daily_sync_maintenance.sh --dry-run # preview all jobs without syncing or changing
daily_sync_maintenance.sh --log # verbose per-share and per-job output
daily_sync_maintenance.sh --status # show configured shares and jobs for this host
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📅 weekly_sync_maintenance.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Weekly maintenance window orchestrator — clean sync of Emby and auth stack, container image updates, and weekly docker restarts. Runs Sunday 2:30am; fits inside the Sunday maintenance block before the 7am coffee report.
# Scheduled: 30 2 * * 0 (Sunday 2:30am)
── Execution Order ──────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Containers stop BEFORE sync — clean static source, full bandwidth.
# Containers start AFTER sync — on fresh data, in dependency order.
# DDNS and failover continue running throughout — only managed containers stop.
#
# 1. Pre-flight checks — connectivity, remote Docker daemon, remote rootfs
# 2. Stop local containers — Emby + auth stack stopped on this server
# 3. Stop remote containers — Emby + auth stack stopped on remote via SSH
# 4. Pull updates locally — if WEEKLY_SYNC_UPDATES=true (containers already stopped)
# 5. Pull updates remotely — if WEEKLY_SYNC_UPDATES_REMOTE=true
# 6. rsync WEEKLY_SYNC_SHARES — full clean mirror at full bandwidth
# 7. Start remote containers — new image, correct dependency order
# 8. Start local containers — new image, correct dependency order
# Post-sync jobs: docker_weekly_restart.sh
# ─────────────────────────────────────────────────────────────────────────────
── Why Weekly Not Nightly for Emby ─────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Two Emby syncs run in parallel — dirty and clean:
#
# emby-fallback dirty sync (every 30 minutes via critical_sync_maintenance.sh, Emby running):
# watch states, library deltas, user activity — continuous coverage
# WAL files excluded — safe to copy while Emby writes
# HOST2 always within 30 minutes of HOST1 on playback state
#
# weekly clean sync (Sunday 2:30am, Emby stopped):
# Full clean mirror — all databases checkpointed and flushed
# Metadata, plugins, config all included
# ~30 seconds of Emby downtime — both instances stopped during rsync only
#
# Why not nightly:
# Emby builds a warm image thumbnail cache on HOST2 throughout the week.
# Nightly sync resets this cache — cold loads every morning for users.
# Weekly sync: cache stays warm for 6 days. Resets Sunday night while users sleep.
# One weekly reset at an acceptable time is better than six unnecessary resets.
# ─────────────────────────────────────────────────────────────────────────────
── Container Update Window ─────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Containers are already stopped for the sync — pulling updated images costs
# nothing extra in downtime. Both servers start on the same image version
# after the window completes.
#
WEEKLY_SYNC_UPDATES=true # pull updated images locally
WEEKLY_SYNC_UPDATES_REMOTE=true # pull updated images on remote via SSH
#
# Toggle false to skip updates without changing the schedule:
# WEEKLY_SYNC_UPDATES=false # skips pulls, containers still restart on current image
── Why Auth Stack Weekly Sync Matters ──────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# NPM, Authelia, LLDAP, Mariadb run warm on both servers continuously.
# HOST1 is source of truth — changes propagate to HOST2 via weekly clean sync.
#
# What propagates automatically every Sunday:
# New users added in LLDAP on HOST1 → appear on HOST2 by Monday
# Proxy rules changed in NPM on HOST1 → live on HOST2 by Monday
# Authelia policies updated on HOST1 → enforced on HOST2 by Monday
# TLS certificates renewed on HOST1 → valid on HOST2 by Monday
#
# No manual sync needed for routine auth administration.
# Anything done on HOST1 is on HOST2 within a week.
# ─────────────────────────────────────────────────────────────────────────────
── Sunday Maintenance Window ───────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# This script is part of a coordinated Sunday maintenance block:
#
# 2:30am weekly_sync_maintenance.sh ← clean sync + image updates (~5-10min)
# 2:50am CA Auto Update plugin ← unRAID plugin updates
# 2:55am CA container updates ← additional container updates
# 3:00am Network reboot ← router/switch restart
#
# Everything comes back clean:
# Network fresh, Emby + auth updated, containers on latest images.
# All in one window while users sleep.
# Sunday morning coffee report at 7am shows the post-maintenance state.
# ─────────────────────────────────────────────────────────────────────────────
── Configuration ────────────────────────────────────────────────────────────
# master.conf
WEEKLY_SYNC_SHARES=(
"/mnt/user/Media_Server/Emby" # emby profile — full clean mirror
"/mnt/user/appdata-Fallback/Critical-Data" # critical-data — auth stack clean state
)
WEEKLY_MAINTENANCE_SCRIPTS=(
"Docker_Essentials/docker_weekly_restart.sh" # weekly restart of less-critical services
)
WEEKLY_SYNC_UPDATES=true
WEEKLY_SYNC_UPDATES_REMOTE=true
── Usage ───────────────────────────────────────────────────────────────────
weekly_sync_maintenance.sh # normal run
weekly_sync_maintenance.sh --dry-run # preview — no stops, no syncs, no starts
weekly_sync_maintenance.sh --log # verbose per-share per-job output
weekly_sync_maintenance.sh --status # show configured shares, jobs, update toggles
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📆 monthly_maintenance.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Uptime-triggered orchestrator for long-running system tasks — ZFS scrub, SMART long tests — that should only run on stable systems that have been up for at least 30 days. Called daily by cron; most invocations are silent no-ops.
Two gates must both pass before any job runs:
- Server uptime ≥
MONTHLY_UPTIME_THRESHOLD_DAYS - Last run ≥
MONTHLY_RUN_INTERVAL_DAYSago (state file on/boot/config/— survives reboots)
If either gate fails, the script exits 0 with no output. This is expected — it runs daily and most days are no-ops.
--force bypasses both gates and runs the job list immediately. Use for testing or
when a scrub was missed and the server hasn't reached the uptime threshold yet.
Configuration (master.conf)
MONTHLY_MAINTENANCE_SCRIPTS=(
#"Tools/zfs_pool_scrub.sh"
#"Tools/smart_long_test.sh"
)
MONTHLY_UPTIME_THRESHOLD_DAYS=30
MONTHLY_RUN_INTERVAL_DAYS=30
MONTHLY_LAST_RUN_FILE="/boot/config/monthly_maintenance_last_run.db"
Scripts are commented out by default — uncomment what applies to your hardware.
Usage
monthly_maintenance.sh # normal run (daily cron — silent no-op when gates not met)
monthly_maintenance.sh --force # bypass both gates — run immediately
monthly_maintenance.sh --dry-run # show what would run without running it
monthly_maintenance.sh --status # show gate state: uptime, last run, next eligible
monthly_maintenance.sh --log # verbose output from each child script
━━━ COMPLETE SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# ─────────────────────────────────────────────────────────────────────────────
# Array start event (Varaverk disks_mounted hook):
# ─────────────────────────────────────────────────────────────────────────────
array_started.sh
# ─────────────────────────────────────────────────────────────────────────────
# Every 7 minutes:
# ─────────────────────────────────────────────────────────────────────────────
*/7 * * * * transcode_management.sh
# ─────────────────────────────────────────────────────────────────────────────
# Every 15 minutes — watchdog cycle:
# ─────────────────────────────────────────────────────────────────────────────
*/15 * * * * watchdog_orchestrator.sh
# ─────────────────────────────────────────────────────────────────────────────
# Every 30 minutes — auth stack + Emby dirty sync + partnership check:
# ─────────────────────────────────────────────────────────────────────────────
*/30 * * * * critical_sync_maintenance.sh
# ─────────────────────────────────────────────────────────────────────────────
# Every 4 hours — arr library sync + failed import recovery + optional rsync:
# ─────────────────────────────────────────────────────────────────────────────
0 */4 * * * intermediate_sync_maintenance.sh
# ─────────────────────────────────────────────────────────────────────────────
# Daily — 1am:
# git pull → rsync all truth shares → permissions + cleaners + arr cleanup → docker restart
# ─────────────────────────────────────────────────────────────────────────────
0 1 * * * daily_sync_maintenance.sh
# ─────────────────────────────────────────────────────────────────────────────
# Weekly — Sunday maintenance block:
# ─────────────────────────────────────────────────────────────────────────────
30 2 * * 0 weekly_sync_maintenance.sh # clean sync + updates (~5-10min)
50 2 * * 0 CA Auto Update plugin # plugin updates
55 2 * * 0 CA container updates # container image updates
0 3 * * 0 Network reboot # router/switch restart
0 7 * * 0 sunday_morning_coffee_report.sh
# ─────────────────────────────────────────────────────────────────────────────
# 15th of each month (uptime-gated — silent no-op if uptime < 30 days):
# ─────────────────────────────────────────────────────────────────────────────
0 0 15 * * monthly_maintenance.sh
━━━ ADDING A NEW ORCHESTRATOR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
If you find yourself running 3+ related scripts on the same schedule, wrap them
in a new orchestrator. Model directly on media_management.sh which has the
complete pattern — dry-run passthrough, status display, pass/fail tracking, summary.
# Minimal skeleton — the full pattern in its simplest form:
#!/bin/bash
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
SCRIPTS_ROOT="$SCRIPT_DIR/.."
PASS=()
FAIL=()
# Read job list from master.conf — never hardcode jobs in the orchestrator
for script_entry in "${MY_MAINTENANCE_JOBS[@]:-}"; do
[[ -z "$script_entry" ]] && continue
read -r -a parts <<< "$script_entry"
script_path="$SCRIPTS_ROOT/${parts[0]}"
script_name=$(basename "${parts[0]}")
extra_args=("${parts[@]:1}")
[[ "$DRY_RUN" == true ]] && extra_args+=("--dry-run")
if bash "$script_path" "${extra_args[@]}"; then
PASS+=("$script_name")
else
FAIL+=("$script_name")
fi
done
# One summary — one notification
echo "Passed: ${#PASS[@]} Failed: ${#FAIL[@]}"
[[ ${#FAIL[@]} -gt 0 ]] && \
notify "My maintenance failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
"My Orchestrator" "warning"