982 lines
55 KiB
Markdown
982 lines
55 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 🎯 ORCHESTRATORS
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Sequential job runners that coordinate multiple scripts into single scheduled units.**
|
|
Orchestrators contain no business logic — they call other scripts in order, track
|
|
pass/fail per job, and produce one clean summary. Configuration lives in `master.conf`.
|
|
Adding or removing a job never requires touching the orchestrator script itself.
|
|
|
|
> **The Varaverk scheduler runs only orchestrators.** Every cron entry, every array
|
|
> start/stop event, every scheduled operation runs through an orchestrator. The individual
|
|
> scripts it calls are never scheduled directly — they run in a defined order inside a
|
|
> coordinated window, with a unified summary at the end.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### 🔴 Race Conditions From Independent Scheduling
|
|
|
|
Media permissions ran at 01:05. Arr cleanup ran at 01:00. Arr cleanup started
|
|
five minutes before permissions were applied — running against files that were still
|
|
owned by root, silently failing to delete the ones it should have cleaned up. Both
|
|
scripts reported success. Neither knew about the other. The result was a library
|
|
that looked cleaned but wasn't.
|
|
|
|
The fix: orchestrators enforce order. `daily_sync_maintenance.sh` runs permissions
|
|
first, then arr cleanup. The arr scripts see correct ownership every time because the
|
|
orchestrator guarantees it. No race, no silent failure, no coordinating cron entries.
|
|
|
|
---
|
|
|
|
### 🔴 Six Separate Notifications Instead of One
|
|
|
|
Before orchestrators, each script sent its own notification on completion. A single
|
|
daily run produced six separate notification pings — one for permissions, one for
|
|
each arr cleanup, one for the cleaner, one for docker restart. Six bells for one
|
|
maintenance window. Worse, if something failed in the middle, you'd get some
|
|
notifications and not others, and figuring out which step failed meant correlating
|
|
timestamps across multiple notification messages.
|
|
|
|
The fix: orchestrators collect all results and send one notification at the end.
|
|
One summary. One bell. Clear pass/fail count. If something failed, the summary
|
|
tells you which job and what happened — no correlation needed.
|
|
|
|
---
|
|
|
|
### 🔴 Transcode Manager Triggering Unnecessary SSD Flips
|
|
|
|
`transcode_manager.sh` ran every 7 minutes on its own. It checked ramdisk usage —
|
|
saw 6.8GB used, threshold is 6.5GB, flipped sessions to SSD. One minute later
|
|
`transcode_cleanup.sh` ran and removed 4GB of stale segment files from ended sessions.
|
|
Actual usage was 2.8GB. Sessions were now on SSD for no reason. Users experiencing
|
|
slightly worse performance. The flip counter incremented for nothing.
|
|
|
|
The fix: `transcode_management.sh` runs cleanup first, manager second, every cycle.
|
|
The manager always sees post-cleanup usage. Stale files can't trigger a flip because
|
|
they're gone before the manager looks. The correct order requires exactly one
|
|
orchestrator to enforce it.
|
|
|
|
---
|
|
|
|
### 🔴 Failed Imports Sitting Stalled for Days
|
|
|
|
A release downloads successfully but Lidarr can't import it — wrong format, incorrect
|
|
tags, file already exists. Lidarr marks it `importFailed` and stops. Nobody notices.
|
|
The download client has the file, Lidarr has given up, and nothing is going to happen
|
|
until someone manually opens Lidarr, identifies the problem, blocklists the release,
|
|
and triggers a new search. This takes minutes to do — but nobody does it at 3am
|
|
when it usually happens.
|
|
|
|
The fix: `arrs_failed_stalled_recovery.sh` runs every 4 hours (via `intermediate_sync_maintenance.sh`). It finds all
|
|
`importFailed`, `importPending`, `error`, and `stalled` items, blocklists them, removes
|
|
them from the queue, and triggers a new search — automatically. By morning the failed
|
|
import has already been replaced by a working one. No manual intervention required.
|
|
|
|
---
|
|
|
|
### 🔴 Array Start Scripts Running in Wrong Order or Not at All
|
|
|
|
Startup scripts configured individually ran in an unpredictable order. The ramdisk
|
|
setup might run after Emby starts. The syslog filter might run after containers have
|
|
already created veth interfaces. PHP-FPM tuning might run after the WebGUI has already
|
|
served its first requests. Each script competed for the same startup slot with no
|
|
guaranteed order.
|
|
|
|
The fix: `array_started.sh` is the only array-start entry in the Varaverk scheduler.
|
|
It launches every startup script in a defined order, with one-second settle between
|
|
each, and reports which succeeded and which failed. Order is guaranteed. Nothing starts
|
|
before its dependency. Everything is visible in a single summary.
|
|
|
|
---
|
|
|
|
## ━━━ THE ORCHESTRATOR MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
The Varaverk scheduler contains exactly these entries:
|
|
|
|
```bash
|
|
# Array start event (Varaverk disks_mounted hook → cron: "array_start"):
|
|
array_started.sh
|
|
|
|
# Cron — managed via Varaverk Scheduler:
|
|
*/7 * * * * transcode_management.sh
|
|
*/15 * * * * watchdog_orchestrator.sh ← resource → docker → system → stability
|
|
*/30 * * * * critical_sync_maintenance.sh ← auth + play_state_sync + partnership
|
|
0 */4 * * * intermediate_sync_maintenance.sh ← arr sync + failed recovery + optional rsync
|
|
0 1 * * * daily_sync_maintenance.sh
|
|
0 7 * * 0 sunday_morning_coffee_report.sh
|
|
30 2 * * 0 weekly_sync_maintenance.sh
|
|
0 0 15 * * monthly_maintenance.sh ← uptime-gated: ZFS scrub, SMART tests
|
|
|
|
# Manual only (not scheduled):
|
|
fallback_test.sh, emby_database_repair.sh, repair tools
|
|
```
|
|
|
|
Every job list is configured in the `ORCHESTRATORS` section of `master.conf`.
|
|
No orchestrator script ever changes when jobs are added or removed — only `master.conf` changes.
|
|
|
|
---
|
|
|
|
## ━━━ THE ORCHESTRATOR PATTERN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All orchestrators follow the same structure:
|
|
|
|
```
|
|
1. Setup → validate config, detect_hosts(), acquire_lock
|
|
2. Pre-flight → fail fast checks before any work begins
|
|
3. Job loop → run each job, track pass/fail, continue on failure
|
|
4. Summary → one clean report of all job results
|
|
5. Notification → one notify per run on failure (never per job)
|
|
```
|
|
|
|
Properties that apply to every orchestrator:
|
|
|
|
```
|
|
Consistent output → every orchestrator looks the same in logs
|
|
No silent failures → pass/fail tracked per job, all in summary
|
|
Resilient → one job failing does not stop the rest
|
|
Single notification → one bell per run, not one per job
|
|
--dry-run cascade → passes --dry-run through to every child script
|
|
--status support → show configured jobs and exit
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
|
lines only appear when `--log` is passed.
|
|
|
|
**One-shot orchestrators** (`array_started.sh`, `array_stopping.sh`, `sunday_morning_coffee_report.sh`):
|
|
without `--log`, section headers, per-phase results, and the final summary are
|
|
visible. Per-item detail suppressed.
|
|
|
|
**Periodic orchestrators** (`critical_sync_maintenance.sh`, `daily_sync_maintenance.sh`,
|
|
`intermediate_sync_maintenance.sh`, `weekly_sync_maintenance.sh`): without `--log`,
|
|
phase headers, per-phase completion status, and the final summary are visible. Per-share
|
|
and per-job detail suppressed.
|
|
|
|
**High-frequency orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent
|
|
during clean cycles. Only state transitions, errors, and startup-grace expiry shown
|
|
without `--log`.
|
|
|
|
---
|
|
|
|
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
| Script | What It Orchestrates | Schedule |
|
|
|--------|---------------------|----------|
|
|
| `array_started.sh` | All array startup scripts in order | `array_start` event (Varaverk plugin hook) |
|
|
| `watchdog_orchestrator.sh` | resource → docker → system → api_renew → stability watchdogs | Every 15 minutes |
|
|
| `transcode_management.sh` | Cleanup then manager — order critical | Every 7 minutes |
|
|
| `daily_sync_maintenance.sh` | git pull → sync → media maintenance → restarts | 1am daily |
|
|
| `weekly_sync_maintenance.sh` | Stop → update → clean sync → start → weekly restarts | 2:30am Sunday |
|
|
| `monthly_maintenance.sh` | Uptime-triggered heavy tasks — ZFS scrub, SMART tests | Daily check, fires when uptime ≥ 30d |
|
|
| `intermediate_sync_maintenance.sh` | arr sync + arrs_failed_stalled_recovery + optional rsync | Every 4 hours |
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🚀 array_started.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Single array-start entry for the entire ecosystem. Fired by the Varaverk plugin's
|
|
`disks_mounted` event hook. Launches every startup script in order — each as a
|
|
background process — and reports which succeeded and which failed.
|
|
|
|
```bash
|
|
# Triggered by: Plugin/unraid/event/disks_mounted/array_start_jobs
|
|
# schedule.json entry: "Orchestrators/array_started.sh" → cron: "array_start"
|
|
```
|
|
|
|
---
|
|
|
|
### ── Execution Order ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Order matters — each entry depends on the previous ones having run.
|
|
# See comments for why each position is correct.
|
|
#
|
|
ARRAY_START_SCRIPTS=(
|
|
# ── One-shot scripts — run and exit naturally ─────────────────────────────
|
|
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # re-register API key FIRST —
|
|
# registry is ephemeral,
|
|
# lost on service restart
|
|
"System_Essentials/conf_sync.sh" # pull conf from remote BEFORE anything
|
|
# needs current config
|
|
"System_Essentials/conf_cache_restore.sh" # restore cached conf if pull failed —
|
|
# ensures config is always available
|
|
"Transcodes/ramdisk_setup.sh" # create tmpfs + symlink BEFORE Emby starts —
|
|
# Emby needs the transcode path to exist
|
|
"System_Essentials/docker_syslog_filter.sh" # suppress veth noise BEFORE containers create
|
|
# veth interfaces — otherwise first boot has
|
|
# unfiltered veth spam
|
|
"Plugin/unraid/System_Essentials/php_fpm_max_children.sh" # WebGUI tuning —
|
|
# before any WebGUI requests
|
|
"System_Essentials/inotify_tuning.sh" # raise inotify BEFORE containers start —
|
|
# containers inherit limits at startup
|
|
"Docker_Essentials/docker_network_connect.sh" # ensure networks + connections BEFORE
|
|
# watchdogs check container states
|
|
|
|
# ── Continuous scripts — run until array stops ─────────────────────────────
|
|
"Arrs_Stack/start_webhook_listener.sh" # start webhook listener before arrs POST events
|
|
"Fallback/fallback.sh" # fallback LAST — needs everything else stable
|
|
)
|
|
|
|
# NOTE: watchdogs (docker_watchdog, system_watchdog, stability_watchdog) are NOT here.
|
|
# They run via watchdog_orchestrator.sh on cron every 15 minutes — not as daemons.
|
|
```
|
|
|
|
---
|
|
|
|
### ── One-Shot vs Continuous Detection ───────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Each script is launched with `bash script.sh &` — background process.
|
|
# After 1 second:
|
|
# PID still alive → continuous script (running in background)
|
|
# logged as: "fallback.sh — running (PID 12345)"
|
|
# PID dead + exit 0 → one-shot completed successfully
|
|
# logged as: "inotify_tuning.sh — completed (one-shot)"
|
|
# PID dead + exit N → failure
|
|
# logged as: "ramdisk_setup.sh — exited with code 1"
|
|
# full path printed — debugging is immediate
|
|
#
|
|
# This means the orchestrator correctly identifies and reports all startup
|
|
# scripts without needing to know in advance which ones are continuous.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Auto-Fix Permissions ────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Scripts that are not executable are chmod +x'd automatically before launch.
|
|
# A permissions problem on a startup script does not cause a silent skip —
|
|
# the orchestrator fixes it and proceeds, then logs that it did so.
|
|
# This prevents "why didn't X run on startup" questions.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Normal — fired by Varaverk disks_mounted event hook. Never run manually in production.
|
|
# array_started.sh runs once and exits — the continuous scripts it launched
|
|
# keep running as background processes.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
array_started.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Dry run — show what would be launched, in order, without launching anything.
|
|
# Use to verify the ARRAY_START_SCRIPTS list before an array restart.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
array_started.sh --dry-run
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Status — show each configured script with its current running state.
|
|
# RUNNING (PID XXXXX) — continuous script currently active
|
|
# not running — one-shot that has completed, or continuous not yet started
|
|
# FILE NOT FOUND — script path wrong or missing
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
array_started.sh --status
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🎬 transcode_management.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every
|
|
7 minutes. Exists because the order is not optional — the manager must always see
|
|
post-cleanup usage to make accurate flip decisions.
|
|
|
|
```bash
|
|
# Scheduled: */7 * * * * (every 7 minutes)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Order Is Non-Negotiable ─────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Emby writes segment files to the transcode directory as it buffers streams.
|
|
# When a stream ends, Emby deletes its own active files — but may leave behind
|
|
# stale segment files from sessions that ended uncleanly. These consume real
|
|
# ramdisk space. The manager has no way to know if they're active or stale.
|
|
#
|
|
# Without correct order:
|
|
# Manager runs → sees 6.8GB used (stale files inflating) → exceeds threshold
|
|
# → flips sessions to SSD → flip counter incremented
|
|
# Cleanup runs → removes 4GB of stale files → actual usage was 2.8GB
|
|
# → flip was unnecessary — sessions now on SSD for no reason
|
|
#
|
|
# With correct order (this orchestrator):
|
|
# Cleanup runs → removes stale files → actual usage 2.8GB
|
|
# Manager runs → sees 2.8GB → below threshold → stays on ramdisk ✅
|
|
# → no flip, no wasted counter, correct decision every time
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── What Each Child Script Does ────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# transcode_cleanup.sh:
|
|
# Identifies segment files not currently open by any process (via lsof)
|
|
# Removes them from the transcode directory
|
|
# If usage drops enough after cleanup → triggers flip-back to ramdisk
|
|
# (handles the recovery direction so manager doesn't have to)
|
|
#
|
|
# transcode_manager.sh:
|
|
# Reads current ramdisk usage after cleanup has run
|
|
# Compares against RAMDISK_WARN_GB threshold
|
|
# Flips the symlink if needed (ramdisk → SSD or SSD → ramdisk)
|
|
# Writes one entry to TRANSCODE_DAILY_LOG for the weekly coffee report
|
|
# Shows active Emby sessions with their play method
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Daily Log ────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each cycle.
|
|
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
|
|
#
|
|
# The sunday_morning_coffee_report.sh reads this log for weekly stats:
|
|
# Peak ramdisk usage across the week
|
|
# Total flip count (unnecessary flips visible here)
|
|
# Session split: how often ramdisk vs SSD was used
|
|
#
|
|
# Log trimmed to TRANSCODE_LOG_RETENTION days on every write — bounded, never grows.
|
|
#
|
|
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
|
|
TRANSCODE_LOG_RETENTION=90 # days
|
|
TRANSCODE_STATE_FILE="/tmp/transcode_state.db" # /tmp — resets on reboot
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
transcode_management.sh # normal run (every 7 min via cron)
|
|
transcode_management.sh --dry-run # passes --dry-run to both child scripts
|
|
transcode_management.sh --status # show config, current state, daily log stats
|
|
transcode_management.sh --log # verbose output from both child scripts
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🔄 arrs_failed_stalled_recovery.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Automatic recovery from failed imports and stalled downloads across Sonarr, Radarr,
|
|
and Lidarr. Blocklists the bad release, removes it from the queue, and triggers a
|
|
new search — hands-free recovery while you sleep.
|
|
|
|
```bash
|
|
# Called by: intermediate_sync_maintenance.sh (every 4 hours)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Four Problem Types ───────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# The arr queue API reports these status values for problem items:
|
|
#
|
|
# importFailed — downloaded successfully but arr couldn't import the file
|
|
# Common causes: wrong format for quality profile, corrupted file,
|
|
# duplicate already in library, permission issue on import path
|
|
# Self-resolution: never — arr stops trying after first failure
|
|
#
|
|
# importPending — downloaded, stuck waiting for import to begin
|
|
# Common causes: import queue backed up, arr paused, API error
|
|
# Self-resolution: sometimes — but stuck for hours is always wrong
|
|
#
|
|
# error — serious failure state not covered by the above
|
|
# Common causes: indexer issues, download client unreachable, disk full
|
|
#
|
|
# stalled — download stuck with no connections or no progress
|
|
# Common causes: no seeders, VPN routing issue, tracker ban
|
|
# Self-resolution: never without a source change
|
|
#
|
|
# NOT touched — items with status "downloading" or "imported" are never touched.
|
|
# Safe to run at any time — only processes items that are already broken.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── What It Does Per Problem Item ──────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# For each problem item found, in order:
|
|
#
|
|
# 1. Blocklist the release
|
|
# Prevents the arr from grabbing the exact same release again immediately.
|
|
# The bad indexer result goes into the blocklist — future searches skip it.
|
|
#
|
|
# 2. Remove from queue
|
|
# Tells the download client to stop and remove the failed download.
|
|
# Frees up the slot for the replacement.
|
|
#
|
|
# 3. Trigger new search
|
|
# Arr searches for a different release meeting the quality profile.
|
|
# If a suitable alternative exists, it starts downloading automatically.
|
|
# If not, the item is marked as "awaiting upgrade" — arr will retry when
|
|
# a new indexer result appears.
|
|
#
|
|
# The entire cycle from "failed import" to "replacement downloading" happens
|
|
# without any human involvement.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Age Threshold ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Items newer than ARR_IMPORT_RECOVERY_AGE hours are skipped.
|
|
# Arrs retry on their own schedule after initial failures — a 30-minute-old
|
|
# importFailed may still resolve itself. Waiting 6 hours before intervening
|
|
# gives the arr a full retry cycle before this script steps in.
|
|
#
|
|
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
|
|
```
|
|
|
|
---
|
|
|
|
### ── Host Awareness ───────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf + host*.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Each arr is independently toggled per host.
|
|
# Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action.
|
|
# HOST2 has its own Sonarr and Radarr for its anime shares.
|
|
#
|
|
HOST1_SONARR_RECOVERY=true # HOST1 Sonarr — Tv_Shows
|
|
HOST1_RADARR_RECOVERY=true # HOST1 Radarr — Movies
|
|
HOST1_LIDARR_RECOVERY=true # HOST1 Lidarr — Music (HOST1 only)
|
|
HOST2_SONARR_RECOVERY=true # HOST2 Sonarr — Anime_Shows
|
|
HOST2_RADARR_RECOVERY=true # HOST2 Radarr — Anime_Movies
|
|
#
|
|
# API versions (current — update if arr major version changes):
|
|
# Sonarr v4 → /api/v3/
|
|
# Radarr v6 → /api/v3/
|
|
# Lidarr v3 → /api/v1/
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
arrs_failed_stalled_recovery.sh # normal run
|
|
arrs_failed_stalled_recovery.sh --dry-run # show what would be actioned, no changes
|
|
arrs_failed_stalled_recovery.sh --log # verbose — show each queue item evaluated
|
|
arrs_failed_stalled_recovery.sh --status # show arr config and API connectivity
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 📅 daily_sync_maintenance.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Full daily maintenance window orchestrator. The entire 1am window — git pull, media
|
|
share sync, media maintenance, and docker daily restarts — in one scheduled entry.
|
|
Runs on both servers; `detect_hosts()` determines which direction each sync goes.
|
|
|
|
```bash
|
|
# Scheduled: 0 1 * * * (1am daily — on BOTH servers)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Execution Order ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# The split into pre-sync and post-sync is based on a simple rule:
|
|
# git pull runs before anything — maintenance uses the latest scripts
|
|
# rsync runs in the middle — media management uses the post-sync state
|
|
# docker restart runs last — after everything else has completed
|
|
#
|
|
# 1. Pre-sync: git_pull_execute.sh
|
|
# Pull latest scripts first. Everything that follows runs on current code.
|
|
#
|
|
# 2. Rsync window: HOST*_DAILY_SYNC_SHARES + HOST*_PERSONAL_SHARES
|
|
# Each server pushes its own truth shares to the other.
|
|
# HOST1 → pushes Movies, Tv_Shows, Music → HOST2
|
|
# HOST2 → pushes Anime_Shows, Anime_Movies → HOST1
|
|
# Personal encrypted shares appended after standard shares.
|
|
# Drive temperature exit codes respected — skip share or abort all on CRIT.
|
|
#
|
|
# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
|
|
# media_management.sh → permissions + cleaners + arr cleanup
|
|
# docker_daily_restart.sh → nightly container restarts
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Bidirectional — Same Script, Correct Direction Automatic ────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# detect_hosts() sets MY_ID at runtime and aliases HOST*_DAILY_SYNC_SHARES
|
|
# to DAILY_SYNC_SHARES. The script uses DAILY_SYNC_SHARES — always the right
|
|
# list for whichever server is running.
|
|
#
|
|
# No HOST1/HOST2 comparisons in the script. Configuration drives direction.
|
|
#
|
|
# HOST1 runs this script at 1am:
|
|
# → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
|
|
# → media_management.sh on HOST1's shares
|
|
# → docker_daily_restart.sh on HOST1's containers
|
|
#
|
|
# HOST2 runs this script at 1am:
|
|
# → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
|
|
# → media_management.sh on HOST2's shares
|
|
# → docker_daily_restart.sh on HOST2's containers
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Script lists — git_pull is split out as pre-sync; everything else is post-sync.
|
|
# The orchestrator recognises git_pull_execute.sh by name and routes it correctly.
|
|
#
|
|
DAILY_MAINTENANCE_SCRIPTS=(
|
|
"git_pull_execute.sh" # PRE-SYNC — always first
|
|
"Media/media_shares_permissions.sh" # POST-SYNC — permissions before arr
|
|
"Media/media_cleaner.sh anime" # POST-SYNC — junk before orphan scan
|
|
"Media/media_cleaner.sh media" # POST-SYNC
|
|
"Arrs_Stack/lidarr_cleanup.sh" # POST-SYNC — orphan cleanup last
|
|
"Arrs_Stack/sonarr_cleanup.sh" # POST-SYNC
|
|
"Arrs_Stack/radarr_cleanup.sh" # POST-SYNC
|
|
"Docker_Essentials/docker_daily_restart.sh" # POST-SYNC — restarts after everything
|
|
)
|
|
|
|
# host1.conf
|
|
HOST1_DAILY_SYNC_SHARES=(
|
|
"/mnt/user/Movies" # both servers — arr_sync union, rsync spreads files
|
|
"/mnt/user/Tv_Shows" # both servers
|
|
"/mnt/user/Music" # both servers
|
|
"/mnt/user/Kids_Movies"
|
|
"/mnt/user/Kids_Tv_Shows"
|
|
"/mnt/user/Sports"
|
|
"/mnt/user/stand-up_comedy"
|
|
)
|
|
|
|
HOST1_PERSONAL_SHARES=(
|
|
"/mnt/user/Personal" # encrypted personal share — appended after standard
|
|
)
|
|
|
|
# host2.conf
|
|
HOST2_DAILY_SYNC_SHARES=(
|
|
"/mnt/user/Anime_Shows" # both servers — arr_sync union, rsync spreads files
|
|
"/mnt/user/Anime_Movies" # both servers
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Adding or Removing a Job ────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Add a new media script — just insert it in the right position:
|
|
DAILY_MAINTENANCE_SCRIPTS=(
|
|
"git_pull_execute.sh"
|
|
"Media/media_shares_permissions.sh"
|
|
"Media/media_cleaner.sh anime"
|
|
"Media/media_cleaner.sh media"
|
|
"Media/my_new_script.sh" # ← add here in the correct order
|
|
"Arrs_Stack/lidarr_cleanup.sh"
|
|
"Arrs_Stack/sonarr_cleanup.sh"
|
|
"Arrs_Stack/radarr_cleanup.sh"
|
|
"Docker_Essentials/docker_daily_restart.sh"
|
|
)
|
|
|
|
# Disable a job temporarily — comment it out, do not delete:
|
|
DAILY_MAINTENANCE_SCRIPTS=(
|
|
"git_pull_execute.sh"
|
|
"Media/media_shares_permissions.sh"
|
|
# "Media/media_cleaner.sh anime" # ← temporarily disabled
|
|
"Media/media_cleaner.sh media"
|
|
"Arrs_Stack/lidarr_cleanup.sh"
|
|
...
|
|
)
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# No changes to daily_sync_maintenance.sh needed in either case.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Drive Temperature Exit Codes ───────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# rsync.sh returns specific exit codes for temperature issues.
|
|
# The orchestrator handles these correctly — one hot drive does not abort all others.
|
|
#
|
|
# exit 0 → success — continue to next share
|
|
# exit 1 → temp WARN — skip this share, continue to next share
|
|
# exit 2 → temp CRITICAL — abort ALL remaining shares in this window
|
|
# notify immediately with which share triggered the abort
|
|
# exit N → other failure — skip this share, continue to next share
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Relationship to Fallback Writeback ──────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# The same HOST*_DAILY_SYNC_SHARES lists are used by fallback.sh for Tier 4
|
|
# writeback — but in the opposite direction.
|
|
#
|
|
# Normal (daily_sync_maintenance.sh):
|
|
# HOST1 → pushes Movies, Tv_Shows → HOST2
|
|
#
|
|
# Tier 4 fallback writeback (HOST1 returns after 24hr+ outage):
|
|
# HOST2 → pushes Movies, Tv_Shows → HOST1
|
|
# (HOST2 was running HOST1's arrs and accumulated content)
|
|
#
|
|
# Same list, correct direction for the situation, zero duplication.
|
|
# No separate writeback list to maintain.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
daily_sync_maintenance.sh # normal run
|
|
daily_sync_maintenance.sh --dry-run # preview all jobs without syncing or changing
|
|
daily_sync_maintenance.sh --log # verbose per-share and per-job output
|
|
daily_sync_maintenance.sh --status # show configured shares and jobs for this host
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 📅 weekly_sync_maintenance.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Weekly maintenance window orchestrator — clean sync of Emby and auth stack, container
|
|
image updates, and weekly docker restarts. Runs Sunday 2:30am; fits inside the Sunday
|
|
maintenance block before the 7am coffee report.
|
|
|
|
```bash
|
|
# Scheduled: 30 2 * * 0 (Sunday 2:30am)
|
|
```
|
|
|
|
---
|
|
|
|
### ── Execution Order ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Containers stop BEFORE sync — clean static source, full bandwidth.
|
|
# Containers start AFTER sync — on fresh data, in dependency order.
|
|
# DDNS and fallback continue running throughout — only managed containers stop.
|
|
#
|
|
# 1. Pre-flight checks — connectivity, remote Docker daemon, remote rootfs
|
|
# 2. Stop local containers — Emby + auth stack stopped on this server
|
|
# 3. Stop remote containers — Emby + auth stack stopped on remote via SSH
|
|
# 4. Pull updates locally — if WEEKLY_SYNC_UPDATES=true (containers already stopped)
|
|
# 5. Pull updates remotely — if WEEKLY_SYNC_UPDATES_REMOTE=true
|
|
# 6. rsync WEEKLY_SYNC_SHARES — full clean mirror at full bandwidth
|
|
# 7. Start remote containers — new image, correct dependency order
|
|
# 8. Start local containers — new image, correct dependency order
|
|
# Post-sync jobs: docker_weekly_restart.sh
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Weekly Not Nightly for Emby ─────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Two Emby sync mechanisms keep the mirror current:
|
|
#
|
|
# play_state_sync (every 30 minutes via critical_sync_maintenance.sh):
|
|
# API-based — syncs watched state and resume points directly via Emby API
|
|
# No rsync of live files — no WAL risk, no partial-write corruption
|
|
# HOST2 always within 30 minutes of HOST1 on playback state
|
|
#
|
|
# weekly clean sync (Sunday 2:30am, Emby stopped):
|
|
# Full clean mirror — all databases checkpointed and flushed
|
|
# Metadata, plugins, config all included
|
|
# ~30 seconds of Emby downtime — both instances stopped during rsync only
|
|
#
|
|
# Why not nightly:
|
|
# Emby builds a warm image thumbnail cache on HOST2 throughout the week.
|
|
# Nightly sync resets this cache — cold loads every morning for users.
|
|
# Weekly sync: cache stays warm for 6 days. Resets Sunday night while users sleep.
|
|
# One weekly reset at an acceptable time is better than six unnecessary resets.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Container Update Window ─────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Containers are already stopped for the sync — pulling updated images costs
|
|
# nothing extra in downtime. Both servers start on the same image version
|
|
# after the window completes.
|
|
#
|
|
WEEKLY_SYNC_UPDATES=true # pull updated images locally
|
|
WEEKLY_SYNC_UPDATES_REMOTE=true # pull updated images on remote via SSH
|
|
#
|
|
# Toggle false to skip updates without changing the schedule:
|
|
# WEEKLY_SYNC_UPDATES=false # skips pulls, containers still restart on current image
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Auth Stack Weekly Sync Matters ──────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# NPM, Authelia, LLDAP, Mariadb run warm on both servers continuously.
|
|
# HOST1 is source of truth — changes propagate to HOST2 via weekly clean sync.
|
|
#
|
|
# What propagates automatically every Sunday:
|
|
# New users added in LLDAP on HOST1 → appear on HOST2 by Monday
|
|
# Proxy rules changed in NPM on HOST1 → live on HOST2 by Monday
|
|
# Authelia policies updated on HOST1 → enforced on HOST2 by Monday
|
|
# TLS certificates renewed on HOST1 → valid on HOST2 by Monday
|
|
#
|
|
# No manual sync needed for routine auth administration.
|
|
# Anything done on HOST1 is on HOST2 within a week.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Sunday Maintenance Window ───────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# This script is part of a coordinated Sunday maintenance block:
|
|
#
|
|
# 2:30am weekly_sync_maintenance.sh ← clean sync + image updates (~5-10min)
|
|
# 2:50am CA Auto Update plugin ← unRAID plugin updates
|
|
# 2:55am CA container updates ← additional container updates
|
|
# 3:00am Network reboot ← router/switch restart
|
|
#
|
|
# Everything comes back clean:
|
|
# Network fresh, Emby + auth updated, containers on latest images.
|
|
# All in one window while users sleep.
|
|
# Sunday morning coffee report at 7am shows the post-maintenance state.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Configuration ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
WEEKLY_SYNC_SHARES=(
|
|
"/mnt/user/Media_Server/Emby" # emby profile — full clean mirror
|
|
"/mnt/user/appdata-Fallback/Critical-Data" # critical-data — auth stack clean state
|
|
)
|
|
|
|
WEEKLY_MAINTENANCE_SCRIPTS=(
|
|
"Docker_Essentials/docker_weekly_restart.sh" # weekly restart of less-critical services
|
|
)
|
|
|
|
WEEKLY_SYNC_UPDATES=true
|
|
WEEKLY_SYNC_UPDATES_REMOTE=true
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
weekly_sync_maintenance.sh # normal run
|
|
weekly_sync_maintenance.sh --dry-run # preview — no stops, no syncs, no starts
|
|
weekly_sync_maintenance.sh --log # verbose per-share per-job output
|
|
weekly_sync_maintenance.sh --status # show configured shares, jobs, update toggles
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 📆 monthly_maintenance.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Uptime-triggered orchestrator for long-running system tasks — ZFS scrub, SMART
|
|
long tests — that should only run on stable systems that have been up for at least
|
|
30 days. Called daily by cron; most invocations are silent no-ops.
|
|
|
|
Two gates must both pass before any job runs:
|
|
1. Server uptime ≥ `MONTHLY_UPTIME_THRESHOLD_DAYS`
|
|
2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file in `$STATE_DIR` — survives reboots)
|
|
|
|
If either gate fails, the script exits 0 with no output. This is expected — it runs
|
|
daily and most days are no-ops.
|
|
|
|
`--force` bypasses both gates and runs the job list immediately. Use for testing or
|
|
when a scrub was missed and the server hasn't reached the uptime threshold yet.
|
|
|
|
### Configuration (master.conf)
|
|
|
|
```bash
|
|
MONTHLY_MAINTENANCE_SCRIPTS=(
|
|
#"Tools/zfs_pool_scrub.sh"
|
|
#"Tools/smart_long_test.sh"
|
|
)
|
|
MONTHLY_UPTIME_THRESHOLD_DAYS=30
|
|
MONTHLY_RUN_INTERVAL_DAYS=30
|
|
MONTHLY_LAST_RUN_FILE="$STATE_DIR/monthly_maintenance_last_run.db"
|
|
```
|
|
|
|
Scripts are commented out by default — uncomment what applies to your hardware.
|
|
|
|
### Usage
|
|
|
|
```bash
|
|
monthly_maintenance.sh # normal run (daily cron — silent no-op when gates not met)
|
|
monthly_maintenance.sh --force # bypass both gates — run immediately
|
|
monthly_maintenance.sh --dry-run # show what would run without running it
|
|
monthly_maintenance.sh --status # show gate state: uptime, last run, next eligible
|
|
monthly_maintenance.sh --log # verbose output from each child script
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ COMPLETE SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Array start event (Varaverk disks_mounted hook):
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
array_started.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Every 7 minutes:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
*/7 * * * * transcode_management.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Every 15 minutes — watchdog cycle:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
*/15 * * * * watchdog_orchestrator.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Every 30 minutes — auth stack + play_state_sync + partnership check:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
*/30 * * * * critical_sync_maintenance.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Every 4 hours — arr library sync + failed import recovery + optional rsync:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
0 */4 * * * intermediate_sync_maintenance.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Daily — 1am:
|
|
# git pull → rsync all truth shares → permissions + cleaners + arr cleanup → docker restart
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
0 1 * * * daily_sync_maintenance.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Weekly — Sunday maintenance block:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
30 2 * * 0 weekly_sync_maintenance.sh # clean sync + updates (~5-10min)
|
|
50 2 * * 0 CA Auto Update plugin # plugin updates
|
|
55 2 * * 0 CA container updates # container image updates
|
|
0 3 * * 0 Network reboot # router/switch restart
|
|
0 7 * * 0 sunday_morning_coffee_report.sh
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# 15th of each month (uptime-gated — silent no-op if uptime < 30 days):
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
0 0 15 * * monthly_maintenance.sh
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ ADDING A NEW ORCHESTRATOR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
If you find yourself running 3+ related scripts on the same schedule, wrap them
|
|
in a new orchestrator. Model directly on `media_management.sh` which has the
|
|
complete pattern — dry-run passthrough, status display, pass/fail tracking, summary.
|
|
|
|
```bash
|
|
# Minimal skeleton — the full pattern in its simplest form:
|
|
#!/bin/bash
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
source "$SCRIPT_DIR/../load_config.sh"
|
|
parse_args "$@"
|
|
|
|
SCRIPTS_ROOT="$SCRIPT_DIR/.."
|
|
PASS=()
|
|
FAIL=()
|
|
|
|
# Read job list from master.conf — never hardcode jobs in the orchestrator
|
|
for script_entry in "${MY_MAINTENANCE_JOBS[@]:-}"; do
|
|
[[ -z "$script_entry" ]] && continue
|
|
|
|
read -r -a parts <<< "$script_entry"
|
|
script_path="$SCRIPTS_ROOT/${parts[0]}"
|
|
script_name=$(basename "${parts[0]}")
|
|
extra_args=("${parts[@]:1}")
|
|
|
|
[[ "$DRY_RUN" == true ]] && extra_args+=("--dry-run")
|
|
|
|
if bash "$script_path" "${extra_args[@]}"; then
|
|
PASS+=("$script_name")
|
|
else
|
|
FAIL+=("$script_name")
|
|
fi
|
|
done
|
|
|
|
# One summary — one notification
|
|
echo "Passed: ${#PASS[@]} Failed: ${#FAIL[@]}"
|
|
[[ ${#FAIL[@]} -gt 0 ]] && \
|
|
notify "My maintenance failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
|
|
"My Orchestrator" "warning"
|
|
``` |