Gmer4Lfe
8d72fbf0e2
Failover/ was renamed to Fallback/ — all script paths and README links updated across README.md, Orchestrators README, and User Script Plug-in README. array_start.sh → array_started.sh in Orchestrators scripts table (wrong name). fallback_state_reset.sh: script is referenced in User Script Plug-in README but does not exist in Fallback/. Entry updated to document it as not yet built and provides the manual workaround (direct state file write). Note: /mnt/user/appdata-Failover/ share path references left unchanged — that is the actual unRAID share name on the server, not the script folder.
973 lines
54 KiB
Markdown
973 lines
54 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 🎯 ORCHESTRATORS
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Sequential job runners that coordinate multiple scripts into single scheduled units.**
|
|
Orchestrators contain no business logic — they call other scripts in order, track
|
|
pass/fail per job, and produce one clean summary. Configuration lives in `master.conf`.
|
|
Adding or removing a job never requires touching the orchestrator script itself.
|
|
|
|
> **The User Scripts plugin contains only orchestrators.** Every cron entry, every
|
|
> "At Startup of Array" entry, every scheduled operation runs through an orchestrator.
|
|
> The individual scripts it calls are never scheduled directly — they run in a defined
|
|
> order inside a coordinated window, with a unified summary at the end.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### 🔴 Race Conditions From Independent Scheduling
|
|
|
|
Media permissions ran at 01:05. Arr cleanup ran at 01:00. Arr cleanup started
|
|
five minutes before permissions were applied — running against files that were still
|
|
owned by root, silently failing to delete the ones it should have cleaned up. Both
|
|
scripts reported success. Neither knew about the other. The result was a library
|
|
that looked cleaned but wasn't.
|
|
|
|
The fix: orchestrators enforce order. `daily_sync_maintenance.sh` runs permissions
|
|
first, then arr cleanup. The arr scripts see correct ownership every time because the
|
|
orchestrator guarantees it. No race, no silent failure, no coordinating cron entries.
|
|
|
|
---
|
|
|
|
### 🔴 Six Separate Notifications Instead of One
|
|
|
|
Before orchestrators, each script sent its own notification on completion. A single
|
|
daily run produced six separate notification pings — one for permissions, one for
|
|
each arr cleanup, one for the cleaner, one for docker restart. Six bells for one
|
|
maintenance window. Worse, if something failed in the middle, you'd get some
|
|
notifications and not others, and figuring out which step failed meant correlating
|
|
timestamps across multiple notification messages.
|
|
|
|
The fix: orchestrators collect all results and send one notification at the end.
|
|
One summary. One bell. Clear pass/fail count. If something failed, the summary
|
|
tells you which job and what happened — no correlation needed.
|
|
|
|
---
|
|
|
|
### 🔴 Transcode Manager Triggering Unnecessary SSD Flips
|
|
|
|
`transcode_manager.sh` ran every 3 minutes on its own. It checked ramdisk usage —
|
|
saw 6.8GB used, threshold is 6.5GB, flipped sessions to SSD. One minute later
|
|
`transcode_cleanup.sh` ran and removed 4GB of stale segment files from ended sessions.
|
|
Actual usage was 2.8GB. Sessions were now on SSD for no reason. Users experiencing
|
|
slightly worse performance. The flip counter incremented for nothing.
|
|
|
|
The fix: `transcode_management.sh` runs cleanup first, manager second, every cycle.
|
|
The manager always sees post-cleanup usage. Stale files can't trigger a flip because
|
|
they're gone before the manager looks. The correct order requires exactly one
|
|
orchestrator to enforce it.
|
|
|
|
---
|
|
|
|
### 🔴 Failed Imports Sitting Stalled for Days
|
|
|
|
A release downloads successfully but Lidarr can't import it — wrong format, incorrect
|
|
tags, file already exists. Lidarr marks it `importFailed` and stops. Nobody notices.
|
|
The download client has the file, Lidarr has given up, and nothing is going to happen
|
|
until someone manually opens Lidarr, identifies the problem, blocklists the release,
|
|
and triggers a new search. This takes minutes to do — but nobody does it at 3am
|
|
when it usually happens.
|
|
|
|
The fix: `arrs_failed_stalled_recovery.sh` runs every 6 hours. It finds all
|
|
`importFailed`, `importPending`, `error`, and `stalled` items, blocklists them, removes
|
|
them from the queue, and triggers a new search — automatically. By morning the failed
|
|
import has already been replaced by a working one. No manual intervention required.
|
|
|
|
---
|
|
|
|
### 🔴 Array Start Scripts Running in Wrong Order or Not at All
|
|
|
|
Scripts configured in the User Scripts plugin as "At Startup of Array" run in an
|
|
unpredictable order. The ramdisk setup might run after Emby starts. The syslog
|
|
filter might run after containers have already created veth interfaces. PHP-FPM
|
|
tuning might run after the WebGUI has already served its first requests. Each
|
|
script competes for the same startup slot with no guaranteed order.
|
|
|
|
The fix: `array_start.sh` is the only "At Startup of Array" entry. It launches
|
|
every startup script in a defined order, with one-second settle between each, and
|
|
reports which succeeded and which failed. Order is guaranteed. Nothing starts before
|
|
its dependency. Everything is visible in a single summary.
|
|
|
|
---
|
|
|
|
## ━━━ THE ORCHESTRATOR MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
The User Scripts plugin contains exactly these entries:
|
|
|
|
```bash
|
|
# At Startup of Array — single entry for all startup scripts:
|
|
array_start.sh
|
|
|
|
# Cron — one entry per maintenance window:
|
|
*/3 * * * * transcode_management.sh
|
|
0 */6 * * * arrs_failed_stalled_recovery.sh
|
|
*/30 * * * * rsync.sh emby-failover ← not an orchestrator, direct call
|
|
0 1 * * * daily_sync_maintenance.sh
|
|
30 2 * * 0 weekly_sync_maintenance.sh
|
|
|
|
# Manual only (not scheduled):
|
|
failover_test.sh, emby_database_repair.sh, repair tools
|
|
```
|
|
|
|
Every job list is configured in the `ORCHESTRATORS` section of `master.conf`.
|
|
No orchestrator script ever changes when jobs are added or removed — only `master.conf` changes.
|
|
|
|
---
|
|
|
|
## ━━━ THE ORCHESTRATOR PATTERN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All orchestrators follow the same structure:
|
|
|
|
```
|
|
1. Setup → validate config, detect_hosts(), acquire_lock
|
|
2. Pre-flight → fail fast checks before any work begins
|
|
3. Job loop → run each job, track pass/fail, continue on failure
|
|
4. Summary → one clean report of all job results
|
|
5. Notification → one notify per run on failure (never per job)
|
|
```
|
|
|
|
Properties that apply to every orchestrator:
|
|
|
|
```
|
|
Consistent output → every orchestrator looks the same in logs
|
|
No silent failures → pass/fail tracked per job, all in summary
|
|
Resilient → one job failing does not stop the rest
|
|
Single notification → one bell per run, not one per job
|
|
--dry-run cascade → passes --dry-run through to every child script
|
|
--status support → show configured jobs and exit
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
|
lines only appear when `--log` is passed.
|
|
|
|
**One-shot orchestrators** (`array_started.sh`, `array_stopping.sh`, `sunday_morning_coffee_report.sh`):
|
|
without `--log`, section headers, per-phase results, and the final summary are
|
|
visible. Per-item detail suppressed.
|
|
|
|
**Periodic orchestrators** (`critical_sync_maintenance.sh`, `daily_sync_maintenance.sh`,
|
|
`intermediate_sync_maintenance.sh`, `weekly_sync_maintenance.sh`): without `--log`,
|
|
phase headers, per-phase completion status, and the final summary are visible. Per-share
|
|
and per-job detail suppressed.
|
|
|
|
**Daemon orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent
|
|
during clean cycles. Only state transitions, errors, and startup-grace expiry shown
|
|
without `--log`.
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
| Script | What It Orchestrates | Schedule |
|
|
|--------|---------------------|----------|
|
|
| `array_started.sh` | All array startup scripts in order | At Startup of Array |
|
|
| `watchdog_orchestrator.sh` | resource → docker → system → stability watchdogs | Every minute |
|
|
| `transcode_management.sh` | Cleanup then manager — order critical | Every 3 minutes |
|
|
| `arrs_failed_stalled_recovery.sh` | Failed import + stalled download recovery | Every 6 hours |
|
|
| `daily_sync_maintenance.sh` | git pull → sync → media maintenance → restarts | 1am daily |
|
|
| `weekly_sync_maintenance.sh` | Stop → update → clean sync → start → weekly restarts | 2:30am Sunday |
|
|
| `monthly_maintenance.sh` | Uptime-triggered heavy tasks — ZFS scrub, SMART tests | Daily check, fires when uptime ≥ 30d |
|
|
| `media_management.sh` | Permissions → cleaners → arr cleanup | Via daily_sync (or manual) |
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🚀 array_start.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Single "At Startup of Array" entry for the entire ecosystem. Launches every startup
|
|
script in order — each as a background process — and reports which succeeded and which
|
|
failed. You never need to add individual scripts to the User Scripts startup list.
|
|
|
|
```bash
|
|
# Scheduled: At Startup of Array (User Scripts plugin)
|
|
# This is the ONLY "At Startup of Array" entry in User Scripts
|
|
```
|
|
|
|
---
|
|
|
|
### ── Execution Order ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Order matters — each entry depends on the previous ones having run.
|
|
# See comments for why each position is correct.
|
|
#
|
|
ARRAY_START_SCRIPTS=(
|
|
# ── One-shot scripts — run and exit naturally ─────────────────────────────
|
|
"unRAID_Essentials/inotify_tuning.sh" # raise inotify BEFORE containers start
|
|
# containers inherit limits at startup —
|
|
# if Code-Server starts with low limits
|
|
# it keeps them until restart
|
|
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise BEFORE containers create
|
|
# veth interfaces — otherwise the first boot
|
|
# always has unfiltered veth spam
|
|
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI tuning — before any WebGUI requests
|
|
"unRAID_Essentials/ramdisk_setup.sh" # create tmpfs + symlink BEFORE Emby starts —
|
|
# Emby needs the transcode path to exist
|
|
"Docker_Essentials/docker_network_connect.sh" # ensure networks + connections BEFORE
|
|
# watchdogs check container states
|
|
|
|
# ── Continuous scripts — run until array stops ─────────────────────────────
|
|
"Watchdogs/stability_watchdog.sh" # last line of defense — reboots when all else fails —
|
|
# system watchdog writes state file that
|
|
# docker watchdog reads every cycle
|
|
"Watchdogs/docker_watchdog.sh" # container health BEFORE failover —
|
|
# containers must be healthy for failover
|
|
# to make reliable decisions
|
|
"Fallback/fallback.sh" # fallback LAST — needs everything else stable
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── One-Shot vs Continuous Detection ───────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Each script is launched with `bash script.sh &` — background process.
|
|
# After 1 second:
|
|
# PID still alive → continuous script (running in background)
|
|
# logged as: "fallback.sh — running (PID 12345)"
|
|
# PID dead + exit 0 → one-shot completed successfully
|
|
# logged as: "inotify_tuning.sh — completed (one-shot)"
|
|
# PID dead + exit N → failure
|
|
# logged as: "ramdisk_setup.sh — exited with code 1"
|
|
# full path printed — debugging is immediate
|
|
#
|
|
# This means the orchestrator correctly identifies and reports all startup
|
|
# scripts without needing to know in advance which ones are continuous.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Auto-Fix Permissions ────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Scripts that are not executable are chmod +x'd automatically before launch.
|
|
# A permissions problem on a startup script does not cause a silent skip —
|
|
# the orchestrator fixes it and proceeds, then logs that it did so.
|
|
# This prevents "why didn't X run on startup" questions.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Normal — called by User Scripts at array start. Never run manually in production.
|
|
# array_start.sh runs once and exits — the continuous scripts it launched
|
|
# keep running as background processes.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
array_start.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Dry run — show what would be launched, in order, without launching anything.
|
|
# Use to verify the ARRAY_START_SCRIPTS list before an array restart.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
array_start.sh --dry-run
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Status — show each configured script with its current running state.
|
|
# RUNNING (PID XXXXX) — continuous script currently active
|
|
# not running — one-shot that has completed, or continuous not yet started
|
|
# FILE NOT FOUND — script path wrong or missing
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
array_start.sh --status
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🎬 transcode_management.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every
|
|
3 minutes. Exists because the order is not optional — the manager must always see
|
|
post-cleanup usage to make accurate flip decisions.
|
|
|
|
```bash
|
|
# Scheduled: */3 * * * * (every 3 minutes)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Order Is Non-Negotiable ─────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Emby writes segment files to the transcode directory as it buffers streams.
|
|
# When a stream ends, Emby deletes its own active files — but may leave behind
|
|
# stale segment files from sessions that ended uncleanly. These consume real
|
|
# ramdisk space. The manager has no way to know if they're active or stale.
|
|
#
|
|
# Without correct order:
|
|
# Manager runs → sees 6.8GB used (stale files inflating) → exceeds threshold
|
|
# → flips sessions to SSD → flip counter incremented
|
|
# Cleanup runs → removes 4GB of stale files → actual usage was 2.8GB
|
|
# → flip was unnecessary — sessions now on SSD for no reason
|
|
#
|
|
# With correct order (this orchestrator):
|
|
# Cleanup runs → removes stale files → actual usage 2.8GB
|
|
# Manager runs → sees 2.8GB → below threshold → stays on ramdisk ✅
|
|
# → no flip, no wasted counter, correct decision every time
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── What Each Child Script Does ────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# transcode_cleanup.sh:
|
|
# Identifies segment files not currently open by any process (via lsof)
|
|
# Removes them from the transcode directory
|
|
# If usage drops enough after cleanup → triggers flip-back to ramdisk
|
|
# (handles the recovery direction so manager doesn't have to)
|
|
#
|
|
# transcode_manager.sh:
|
|
# Reads current ramdisk usage after cleanup has run
|
|
# Compares against RAMDISK_WARN_GB threshold
|
|
# Flips the symlink if needed (ramdisk → SSD or SSD → ramdisk)
|
|
# Writes one entry to TRANSCODE_DAILY_LOG for the weekly coffee report
|
|
# Shows active Emby sessions with their play method
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Daily Log ────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each cycle.
|
|
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
|
|
#
|
|
# The sunday_morning_coffee_report.sh reads this log for weekly stats:
|
|
# Peak ramdisk usage across the week
|
|
# Total flip count (unnecessary flips visible here)
|
|
# Session split: how often ramdisk vs SSD was used
|
|
#
|
|
# Log trimmed to TRANSCODE_LOG_RETENTION days on every write — bounded, never grows.
|
|
#
|
|
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
|
|
TRANSCODE_LOG_RETENTION=90 # days
|
|
TRANSCODE_STATE_FILE="/tmp/transcode_state.db" # /tmp — resets on reboot
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
transcode_management.sh # normal run (every 3 min via cron)
|
|
transcode_management.sh --dry-run # passes --dry-run to both child scripts
|
|
transcode_management.sh --status # show config, current state, daily log stats
|
|
transcode_management.sh --log # verbose output from both child scripts
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🔄 arrs_failed_stalled_recovery.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Automatic recovery from failed imports and stalled downloads across Sonarr, Radarr,
|
|
and Lidarr. Blocklists the bad release, removes it from the queue, and triggers a
|
|
new search — hands-free recovery while you sleep.
|
|
|
|
```bash
|
|
# Scheduled: 0 */6 * * * (every 6 hours)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Four Problem Types ───────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# The arr queue API reports these status values for problem items:
|
|
#
|
|
# importFailed — downloaded successfully but arr couldn't import the file
|
|
# Common causes: wrong format for quality profile, corrupted file,
|
|
# duplicate already in library, permission issue on import path
|
|
# Self-resolution: never — arr stops trying after first failure
|
|
#
|
|
# importPending — downloaded, stuck waiting for import to begin
|
|
# Common causes: import queue backed up, arr paused, API error
|
|
# Self-resolution: sometimes — but stuck for hours is always wrong
|
|
#
|
|
# error — serious failure state not covered by the above
|
|
# Common causes: indexer issues, download client unreachable, disk full
|
|
#
|
|
# stalled — download stuck with no connections or no progress
|
|
# Common causes: no seeders, VPN routing issue, tracker ban
|
|
# Self-resolution: never without a source change
|
|
#
|
|
# NOT touched — items with status "downloading" or "imported" are never touched.
|
|
# Safe to run at any time — only processes items that are already broken.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── What It Does Per Problem Item ──────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# For each problem item found, in order:
|
|
#
|
|
# 1. Blocklist the release
|
|
# Prevents the arr from grabbing the exact same release again immediately.
|
|
# The bad indexer result goes into the blocklist — future searches skip it.
|
|
#
|
|
# 2. Remove from queue
|
|
# Tells the download client to stop and remove the failed download.
|
|
# Frees up the slot for the replacement.
|
|
#
|
|
# 3. Trigger new search
|
|
# Arr searches for a different release meeting the quality profile.
|
|
# If a suitable alternative exists, it starts downloading automatically.
|
|
# If not, the item is marked as "awaiting upgrade" — arr will retry when
|
|
# a new indexer result appears.
|
|
#
|
|
# The entire cycle from "failed import" to "replacement downloading" happens
|
|
# without any human involvement.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Age Threshold ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Items newer than ARR_IMPORT_RECOVERY_AGE hours are skipped.
|
|
# Arrs retry on their own schedule after initial failures — a 30-minute-old
|
|
# importFailed may still resolve itself. Waiting 6 hours before intervening
|
|
# gives the arr a full retry cycle before this script steps in.
|
|
#
|
|
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
|
|
```
|
|
|
|
---
|
|
|
|
### ── Host Awareness ───────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf + host*.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Each arr is independently toggled per host.
|
|
# Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action.
|
|
# HOST2 has its own Sonarr and Radarr for its anime shares.
|
|
#
|
|
HOST1_SONARR_RECOVERY=true # HOST1 Sonarr — Tv_Shows
|
|
HOST1_RADARR_RECOVERY=true # HOST1 Radarr — Movies
|
|
HOST1_LIDARR_RECOVERY=true # HOST1 Lidarr — Music (HOST1 only)
|
|
HOST2_SONARR_RECOVERY=true # HOST2 Sonarr — Anime_Shows
|
|
HOST2_RADARR_RECOVERY=true # HOST2 Radarr — Anime_Movies
|
|
#
|
|
# API versions (current — update if arr major version changes):
|
|
# Sonarr v4 → /api/v3/
|
|
# Radarr v6 → /api/v3/
|
|
# Lidarr v3 → /api/v1/
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
arrs_failed_stalled_recovery.sh # normal run
|
|
arrs_failed_stalled_recovery.sh --dry-run # show what would be actioned, no changes
|
|
arrs_failed_stalled_recovery.sh --log # verbose — show each queue item evaluated
|
|
arrs_failed_stalled_recovery.sh --status # show arr config and API connectivity
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 📅 daily_sync_maintenance.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Full daily maintenance window orchestrator. The entire 1am window — git pull, media
|
|
share sync, media maintenance, and docker daily restarts — in one scheduled entry.
|
|
Runs on both servers; `detect_hosts()` determines which direction each sync goes.
|
|
|
|
```bash
|
|
# Scheduled: 0 1 * * * (1am daily — on BOTH servers)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Execution Order ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# The split into pre-sync and post-sync is based on a simple rule:
|
|
# git pull runs before anything — maintenance uses the latest scripts
|
|
# rsync runs in the middle — media management uses the post-sync state
|
|
# docker restart runs last — after everything else has completed
|
|
#
|
|
# 1. Pre-sync: git_pull_execute.sh
|
|
# Pull latest scripts first. Everything that follows runs on current code.
|
|
#
|
|
# 2. Rsync window: HOST*_DAILY_SYNC_SHARES + HOST*_PERSONAL_SHARES
|
|
# Each server pushes its own truth shares to the other.
|
|
# HOST1 → pushes Movies, Tv_Shows, Music → HOST2
|
|
# HOST2 → pushes Anime_Shows, Anime_Movies → HOST1
|
|
# Personal encrypted shares appended after standard shares.
|
|
# Drive temperature exit codes respected — skip share or abort all on CRIT.
|
|
#
|
|
# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
|
|
# media_management.sh → permissions + cleaners + arr cleanup
|
|
# docker_daily_restart.sh → nightly container restarts
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Bidirectional — Same Script, Correct Direction Automatic ────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# detect_hosts() sets MY_ID at runtime and aliases HOST*_DAILY_SYNC_SHARES
|
|
# to DAILY_SYNC_SHARES. The script uses DAILY_SYNC_SHARES — always the right
|
|
# list for whichever server is running.
|
|
#
|
|
# No HOST1/HOST2 comparisons in the script. Configuration drives direction.
|
|
#
|
|
# HOST1 runs this script at 1am:
|
|
# → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
|
|
# → media_management.sh on HOST1's shares
|
|
# → docker_daily_restart.sh on HOST1's containers
|
|
#
|
|
# HOST2 runs this script at 1am:
|
|
# → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
|
|
# → media_management.sh on HOST2's shares
|
|
# → docker_daily_restart.sh on HOST2's containers
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Script lists — git_pull is split out as pre-sync; everything else is post-sync.
|
|
# The orchestrator recognises git_pull_execute.sh by name and routes it correctly.
|
|
#
|
|
DAILY_MAINTENANCE_SCRIPTS=(
|
|
"git_pull_execute.sh" # PRE-SYNC — always first
|
|
"Media/media_shares_permissions.sh" # POST-SYNC — permissions before arr
|
|
"Media/media_cleaner.sh anime" # POST-SYNC — junk before orphan scan
|
|
"Media/media_cleaner.sh media" # POST-SYNC
|
|
"Media/lidarr_cleanup.sh" # POST-SYNC — orphan cleanup last
|
|
"Media/sonarr_cleanup.sh" # POST-SYNC
|
|
"Media/radarr_cleanup.sh" # POST-SYNC
|
|
"Docker_Essentials/docker_daily_restart.sh" # POST-SYNC — restarts after everything
|
|
)
|
|
|
|
# host1.conf
|
|
HOST1_DAILY_SYNC_SHARES=(
|
|
"/mnt/user/Movies" # both servers — arr_sync union, rsync spreads files
|
|
"/mnt/user/Tv_Shows" # both servers
|
|
"/mnt/user/Music" # both servers
|
|
"/mnt/user/Kids_Movies"
|
|
"/mnt/user/Kids_Tv_Shows"
|
|
"/mnt/user/Sports"
|
|
"/mnt/user/stand-up_comedy"
|
|
)
|
|
|
|
HOST1_PERSONAL_SHARES=(
|
|
"/mnt/user/Personal" # encrypted personal share — appended after standard
|
|
)
|
|
|
|
# host2.conf
|
|
HOST2_DAILY_SYNC_SHARES=(
|
|
"/mnt/user/Anime_Shows" # both servers — arr_sync union, rsync spreads files
|
|
"/mnt/user/Anime_Movies" # both servers
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Adding or Removing a Job ────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Add a new media script — just insert it in the right position:
|
|
DAILY_MAINTENANCE_SCRIPTS=(
|
|
"git_pull_execute.sh"
|
|
"Media/media_shares_permissions.sh"
|
|
"Media/media_cleaner.sh anime"
|
|
"Media/media_cleaner.sh media"
|
|
"Media/my_new_script.sh" # ← add here in the correct order
|
|
"Media/lidarr_cleanup.sh"
|
|
"Media/sonarr_cleanup.sh"
|
|
"Media/radarr_cleanup.sh"
|
|
"Docker_Essentials/docker_daily_restart.sh"
|
|
)
|
|
|
|
# Disable a job temporarily — comment it out, do not delete:
|
|
DAILY_MAINTENANCE_SCRIPTS=(
|
|
"git_pull_execute.sh"
|
|
"Media/media_shares_permissions.sh"
|
|
# "Media/media_cleaner.sh anime" # ← temporarily disabled
|
|
"Media/media_cleaner.sh media"
|
|
"Media/lidarr_cleanup.sh"
|
|
...
|
|
)
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# No changes to daily_sync_maintenance.sh needed in either case.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Drive Temperature Exit Codes ───────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# rsync.sh returns specific exit codes for temperature issues.
|
|
# The orchestrator handles these correctly — one hot drive does not abort all others.
|
|
#
|
|
# exit 0 → success — continue to next share
|
|
# exit 1 → temp WARN — skip this share, continue to next share
|
|
# exit 2 → temp CRITICAL — abort ALL remaining shares in this window
|
|
# notify immediately with which share triggered the abort
|
|
# exit N → other failure — skip this share, continue to next share
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Relationship to Failover Writeback ──────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# The same HOST*_DAILY_SYNC_SHARES lists are used by failover.sh for Tier 4
|
|
# writeback — but in the opposite direction.
|
|
#
|
|
# Normal (daily_sync_maintenance.sh):
|
|
# HOST1 → pushes Movies, Tv_Shows → HOST2
|
|
#
|
|
# Tier 4 failover writeback (HOST1 returns after 24hr+ outage):
|
|
# HOST2 → pushes Movies, Tv_Shows → HOST1
|
|
# (HOST2 was running HOST1's arrs and accumulated content)
|
|
#
|
|
# Same list, correct direction for the situation, zero duplication.
|
|
# No separate writeback list to maintain.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
daily_sync_maintenance.sh # normal run
|
|
daily_sync_maintenance.sh --dry-run # preview all jobs without syncing or changing
|
|
daily_sync_maintenance.sh --log # verbose per-share and per-job output
|
|
daily_sync_maintenance.sh --status # show configured shares and jobs for this host
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 📅 weekly_sync_maintenance.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Weekly maintenance window orchestrator — clean sync of Emby and auth stack, container
|
|
image updates, and weekly docker restarts. Runs Sunday 2:30am; fits inside the Sunday
|
|
maintenance block before the 7am coffee report.
|
|
|
|
```bash
|
|
# Scheduled: 30 2 * * 0 (Sunday 2:30am)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Execution Order ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Containers stop BEFORE sync — clean static source, full bandwidth.
|
|
# Containers start AFTER sync — on fresh data, in dependency order.
|
|
# DDNS and failover continue running throughout — only managed containers stop.
|
|
#
|
|
# 1. Pre-flight checks — connectivity, remote Docker daemon, remote rootfs
|
|
# 2. Stop local containers — Emby + auth stack stopped on this server
|
|
# 3. Stop remote containers — Emby + auth stack stopped on remote via SSH
|
|
# 4. Pull updates locally — if WEEKLY_SYNC_UPDATES=true (containers already stopped)
|
|
# 5. Pull updates remotely — if WEEKLY_SYNC_UPDATES_REMOTE=true
|
|
# 6. rsync WEEKLY_SYNC_SHARES — full clean mirror at full bandwidth
|
|
# 7. Start remote containers — new image, correct dependency order
|
|
# 8. Start local containers — new image, correct dependency order
|
|
# Post-sync jobs: docker_weekly_restart.sh
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Weekly Not Nightly for Emby ─────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Two Emby syncs run in parallel — dirty and clean:
|
|
#
|
|
# emby-failover dirty sync (every 30 minutes, Emby running):
|
|
# watch states, library deltas, user activity — continuous coverage
|
|
# WAL files excluded — safe to copy while Emby writes
|
|
# HOST2 always within 30 minutes of HOST1 on playback state
|
|
#
|
|
# weekly clean sync (Sunday 2:30am, Emby stopped):
|
|
# Full clean mirror — all databases checkpointed and flushed
|
|
# Metadata, plugins, config all included
|
|
# ~30 seconds of Emby downtime — both instances stopped during rsync only
|
|
#
|
|
# Why not nightly:
|
|
# Emby builds a warm image thumbnail cache on HOST2 throughout the week.
|
|
# Nightly sync resets this cache — cold loads every morning for users.
|
|
# Weekly sync: cache stays warm for 6 days. Resets Sunday night while users sleep.
|
|
# One weekly reset at an acceptable time is better than six unnecessary resets.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Container Update Window ─────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Containers are already stopped for the sync — pulling updated images costs
|
|
# nothing extra in downtime. Both servers start on the same image version
|
|
# after the window completes.
|
|
#
|
|
WEEKLY_SYNC_UPDATES=true # pull updated images locally
|
|
WEEKLY_SYNC_UPDATES_REMOTE=true # pull updated images on remote via SSH
|
|
#
|
|
# Toggle false to skip updates without changing the schedule:
|
|
# WEEKLY_SYNC_UPDATES=false # skips pulls, containers still restart on current image
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Auth Stack Weekly Sync Matters ──────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# NPM, Authelia, LLDAP, Mariadb run warm on both servers continuously.
|
|
# HOST1 is source of truth — changes propagate to HOST2 via weekly clean sync.
|
|
#
|
|
# What propagates automatically every Sunday:
|
|
# New users added in LLDAP on HOST1 → appear on HOST2 by Monday
|
|
# Proxy rules changed in NPM on HOST1 → live on HOST2 by Monday
|
|
# Authelia policies updated on HOST1 → enforced on HOST2 by Monday
|
|
# TLS certificates renewed on HOST1 → valid on HOST2 by Monday
|
|
#
|
|
# No manual sync needed for routine auth administration.
|
|
# Anything done on HOST1 is on HOST2 within a week.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Sunday Maintenance Window ───────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# This script is part of a coordinated Sunday maintenance block:
|
|
#
|
|
# 2:30am weekly_sync_maintenance.sh ← clean sync + image updates (~5-10min)
|
|
# 2:50am CA Auto Update plugin ← unRAID plugin updates
|
|
# 2:55am CA container updates ← additional container updates
|
|
# 3:00am Network reboot ← router/switch restart
|
|
#
|
|
# Everything comes back clean:
|
|
# Network fresh, Emby + auth updated, containers on latest images.
|
|
# All in one window while users sleep.
|
|
# Sunday morning coffee report at 7am shows the post-maintenance state.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
WEEKLY_SYNC_SHARES=(
|
|
"/mnt/user/Media_Server/Emby" # emby profile — full clean mirror
|
|
"/mnt/user/appdata-Failover/Critical-Data" # critical-data — auth stack clean state
|
|
)
|
|
|
|
WEEKLY_MAINTENANCE_SCRIPTS=(
|
|
"Docker_Essentials/docker_weekly_restart.sh" # weekly restart of less-critical services
|
|
)
|
|
|
|
WEEKLY_SYNC_UPDATES=true
|
|
WEEKLY_SYNC_UPDATES_REMOTE=true
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
weekly_sync_maintenance.sh # normal run
|
|
weekly_sync_maintenance.sh --dry-run # preview — no stops, no syncs, no starts
|
|
weekly_sync_maintenance.sh --log # verbose per-share per-job output
|
|
weekly_sync_maintenance.sh --status # show configured shares, jobs, update toggles
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🧹 media_management.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Runs all media maintenance scripts sequentially in the order defined by
|
|
`MEDIA_MAINTENANCE_JOBS` in `master.conf`. Called by `daily_sync_maintenance.sh`
|
|
as a post-sync job — not scheduled separately. Available for manual runs when
|
|
media maintenance is needed outside the normal window.
|
|
|
|
```bash
|
|
# Called by: daily_sync_maintenance.sh (post-sync)
|
|
# Manual use: run directly for ad hoc media maintenance
|
|
```
|
|
|
|
---
|
|
|
|
### ── Execution Order ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Order is critical — see README-Media.md for detailed explanation.
|
|
# Short version: permissions first, cleaners second, arr cleanup last.
|
|
# Each step depends on the previous having completed correctly.
|
|
#
|
|
MEDIA_MAINTENANCE_JOBS=(
|
|
"Media/media_shares_permissions.sh" # 1. permissions — arr cleanup depends on this
|
|
"Media/media_cleaner.sh anime" # 2. junk removal — orphan scan depends on this
|
|
"Media/media_cleaner.sh media" # 3. same for media shares
|
|
"Media/lidarr_cleanup.sh" # 4. orphan cleanup — last, after permissions + clean
|
|
"Media/sonarr_cleanup.sh" # 5.
|
|
"Media/radarr_cleanup.sh" # 6.
|
|
)
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Add a script: insert in the correct position for your use case.
|
|
# Remove a script: comment it out with #
|
|
# No changes to media_management.sh needed in either case.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
media_management.sh # normal run — all jobs in order
|
|
media_management.sh --dry-run # preview without any deletions or changes
|
|
media_management.sh --log # verbose output from all jobs
|
|
media_management.sh --status # show configured job list and exit
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ COMPLETE SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# At Startup of Array — single entry:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
array_start.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Frequent — every 3 minutes:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
*/3 * * * * transcode_management.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Periodic — not an orchestrator, direct rsync call:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
*/30 * * * * rsync.sh /mnt/user/Media_Server/Emby --profile=emby-failover
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Every 6 hours:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
0 */6 * * * arrs_failed_stalled_recovery.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Daily — 1am:
|
|
# git pull → rsync all truth shares → permissions + cleaners + arr cleanup → docker restart
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
0 1 * * * daily_sync_maintenance.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Weekly — Sunday maintenance block:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
30 2 * * 0 weekly_sync_maintenance.sh # clean sync + updates (~5-10min)
|
|
50 2 * * 0 CA Auto Update plugin # plugin updates
|
|
55 2 * * 0 CA container updates # container image updates
|
|
0 3 * * 0 Network reboot # router/switch restart
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ ADDING A NEW ORCHESTRATOR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
If you find yourself running 3+ related scripts on the same schedule, wrap them
|
|
in a new orchestrator. Model directly on `media_management.sh` which has the
|
|
complete pattern — dry-run passthrough, status display, pass/fail tracking, summary.
|
|
|
|
```bash
|
|
# Minimal skeleton — the full pattern in its simplest form:
|
|
#!/bin/bash
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
source "$SCRIPT_DIR/../load_config.sh"
|
|
parse_args "$@"
|
|
|
|
SCRIPTS_ROOT="$SCRIPT_DIR/.."
|
|
PASS=()
|
|
FAIL=()
|
|
|
|
# Read job list from master.conf — never hardcode jobs in the orchestrator
|
|
for script_entry in "${MY_MAINTENANCE_JOBS[@]:-}"; do
|
|
[[ -z "$script_entry" ]] && continue
|
|
|
|
read -r -a parts <<< "$script_entry"
|
|
script_path="$SCRIPTS_ROOT/${parts[0]}"
|
|
script_name=$(basename "${parts[0]}")
|
|
extra_args=("${parts[@]:1}")
|
|
|
|
[[ "$DRY_RUN" == true ]] && extra_args+=("--dry-run")
|
|
|
|
if bash "$script_path" "${extra_args[@]}"; then
|
|
PASS+=("$script_name")
|
|
else
|
|
FAIL+=("$script_name")
|
|
fi
|
|
done
|
|
|
|
# One summary — one notification
|
|
echo "Passed: ${#PASS[@]} Failed: ${#FAIL[@]}"
|
|
[[ ${#FAIL[@]} -gt 0 ]] && \
|
|
notify "My maintenance failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
|
|
"My Orchestrator" "warning"
|
|
``` |