created loop for both watchdogs and consolidated orch lists
This commit is contained in:
@@ -13,14 +13,68 @@ Without orchestrators, each script runs independently on its own schedule. This
|
||||
- **Race conditions** — two scripts running simultaneously on the same data
|
||||
- **Order dependency failures** — media cleaner runs before permissions, finds wrong ownership
|
||||
- **No combined summary** — 6 separate notifications instead of one clean report
|
||||
- **Scheduling complexity** — 6+ cron entries instead of one
|
||||
- **Scheduling complexity** — many cron entries instead of a few clean ones
|
||||
|
||||
Orchestrators solve this by making a set of related scripts into a single scheduled unit with a defined execution order and a unified summary.
|
||||
|
||||
---
|
||||
|
||||
## The Orchestrator Model
|
||||
|
||||
The ecosystem is designed so the User Scripts plugin contains only a small number of entries — each one an orchestrator that owns a domain:
|
||||
|
||||
```
|
||||
At Startup of Array:
|
||||
array_start.sh ← single entry, launches everything
|
||||
|
||||
Cron:
|
||||
transcode_management.sh ← */3 * * * *
|
||||
arrs_failed_stalled_recovery.sh ← 0 */6 * * *
|
||||
rsync.sh ... emby-failover ← */30 * * * *
|
||||
daily_sync_maintenance.sh ← 0 1 * * *
|
||||
weekly_sync_maintenance.sh ← 30 2 * * 0
|
||||
weekly_health_digest.sh ← Saturday morning
|
||||
|
||||
Manual only:
|
||||
failover_test.sh, emby_database_repair.sh, repair tools
|
||||
```
|
||||
|
||||
All job lists are configured in the `ORCHESTRATORS` section of `Master.conf`. No changes to orchestrator scripts needed when adding or removing jobs.
|
||||
|
||||
---
|
||||
|
||||
## Scripts
|
||||
|
||||
### `array_start.sh`
|
||||
|
||||
Single entry point for the User Scripts "At Startup of Array" schedule. Launches all array-start scripts in order — each as a background process.
|
||||
|
||||
```bash
|
||||
# Scheduled as: At Startup of Array
|
||||
/mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh
|
||||
```
|
||||
|
||||
One-shot scripts (ramdisk, syslog filter, php-fpm, network connect) run and exit naturally. Continuous scripts (system watchdog, docker watchdog, failover) run until the array stops.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```bash
|
||||
# Master.conf — ORCHESTRATORS section
|
||||
ARRAY_START_SCRIPTS=(
|
||||
"unRAID_Essentials/ramdisk_setup.sh" # creates ramdisk before Emby starts
|
||||
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth log noise
|
||||
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI tuning
|
||||
"Docker_Essentials/docker_network_connect.sh" # connect containers to extra networks
|
||||
"unRAID_Essentials/system_watchdog.sh" # continuous system health monitor
|
||||
"Docker_Essentials/docker_watchdog.sh" # continuous container health monitor
|
||||
"Failover/failover.sh" # continuous mutual failover
|
||||
)
|
||||
```
|
||||
|
||||
Add or remove scripts from `ARRAY_START_SCRIPTS` — no changes to `array_start.sh` needed. Order matters — ramdisk before Emby, network before watchdogs, watchdogs before failover.
|
||||
|
||||
---
|
||||
|
||||
### `transcode_management.sh`
|
||||
|
||||
Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every 3 minutes. Replaces two separate cron entries with one.
|
||||
@@ -32,13 +86,13 @@ Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order eve
|
||||
|
||||
**Why cleanup must run before manager:**
|
||||
|
||||
If the manager runs first it may see inflated ramdisk usage from stale segment files left by ended sessions — and trigger an unnecessary flip to SSD. Cleanup runs first to clear those files, then the manager makes its threshold decision based on real active session usage.
|
||||
If the manager runs first it sees inflated ramdisk usage from stale segment files left by ended sessions — and triggers an unnecessary flip to SSD. Cleanup runs first to clear those files, then the manager decides based on real active session usage.
|
||||
|
||||
```
|
||||
Without correct order:
|
||||
Manager checks usage → 6.8GB (includes stale files) → flips to SSD
|
||||
Cleanup runs → removes stale files → actual usage 2.1GB
|
||||
Manager was wrong — unnecessary flip
|
||||
Unnecessary flip — sessions now on SSD
|
||||
|
||||
With correct order:
|
||||
Cleanup runs → removes stale files → actual usage 2.1GB
|
||||
@@ -47,7 +101,7 @@ With correct order:
|
||||
|
||||
**Daily statistics tracking:**
|
||||
|
||||
Every cycle `transcode_management.sh` records stats to `/boot/config/transcode_daily.db`:
|
||||
Every cycle `transcode_management.sh` records stats to `TRANSCODE_DAILY_LOG`:
|
||||
- Peak ramdisk usage for the day
|
||||
- Total flip count for the day
|
||||
- Ramdisk vs SSD session counts
|
||||
@@ -57,193 +111,114 @@ Every cycle `transcode_management.sh` records stats to `/boot/config/transcode_d
|
||||
|
||||
---
|
||||
|
||||
### `media_shares_sync.sh`
|
||||
### `arrs_failed_stalled_recovery.sh`
|
||||
|
||||
Syncs each server's source-of-truth media shares to the remote server sequentially. Each server only pushes the shares it owns — direction and share list are automatic based on which server is running the script.
|
||||
Automatically detects and recovers from failed imports and stalled downloads across Sonarr, Radarr, and Lidarr. Blocklists the bad release and triggers a new search — hands-free recovery while you sleep.
|
||||
|
||||
```bash
|
||||
# Scheduled as: 0 */6 * * * (every 6 hours)
|
||||
/mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh
|
||||
```
|
||||
|
||||
Targets four problem types: `importFailed`, `importPending`, `error` status, and `stalled` downloads. Items newer than `ARR_IMPORT_RECOVERY_AGE` (6 hours) are skipped — gives the arr time to retry on its own first.
|
||||
|
||||
**API versions:** Sonarr v4 → `/api/v3/` — Radarr v6 → `/api/v3/` — Lidarr v3 → `/api/v1/`
|
||||
|
||||
Lidarr runs on HOST1 only — exits cleanly on HOST2.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```bash
|
||||
# Master.conf — MEDIA section
|
||||
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
|
||||
|
||||
HOST1_SONARR_RECOVERY=true
|
||||
HOST1_RADARR_RECOVERY=true
|
||||
HOST1_LIDARR_RECOVERY=true
|
||||
HOST2_SONARR_RECOVERY=true
|
||||
HOST2_RADARR_RECOVERY=true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `daily_sync_maintenance.sh`
|
||||
|
||||
Full daily maintenance window orchestrator — git pull, media share sync, media management, and docker daily restarts. All driven by `Master.conf` arrays.
|
||||
|
||||
```bash
|
||||
# Scheduled as: 0 1 * * * (1am daily — on both servers)
|
||||
/mnt/user/appdata/unraid_scripts/Orchestrators/media_shares_sync.sh
|
||||
/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh
|
||||
```
|
||||
|
||||
**Execution order:**
|
||||
|
||||
```
|
||||
1. Pre-sync jobs (DAILY_MAINTENANCE_SCRIPTS — git pull first):
|
||||
git_pull_execute.sh ← always runs first — pulls latest scripts
|
||||
|
||||
2. Media share sync:
|
||||
HOST*_DAILY_SYNC_SHARES ← each server pushes its own truth shares
|
||||
HOST*_PERSONAL_SHARES ← personal encrypted shares appended after
|
||||
|
||||
3. Post-sync jobs (DAILY_MAINTENANCE_SCRIPTS — remaining):
|
||||
media_management.sh ← permissions + cleaners + arr cleanup
|
||||
docker_daily_restart.sh ← daily container restarts
|
||||
```
|
||||
|
||||
**Bidirectional — same script, correct direction automatically:**
|
||||
|
||||
```
|
||||
HOST1 runs media_shares_sync.sh → pushes HOST1_DAILY_SYNC_SHARES → TO HOST2
|
||||
Movies, Tv_Shows, Music, Books etc. — HOST1 is source of truth
|
||||
HOST1 runs daily_sync_maintenance.sh:
|
||||
git pull → sync HOST1_DAILY_SYNC_SHARES → TO HOST2 → media_management → docker restart
|
||||
|
||||
HOST2 runs media_shares_sync.sh → pushes HOST2_DAILY_SYNC_SHARES → TO HOST1
|
||||
Anime_Shows, Anime_Movies — HOST2 is source of truth
|
||||
HOST2 runs daily_sync_maintenance.sh:
|
||||
git pull → sync HOST2_DAILY_SYNC_SHARES → TO HOST1 → media_management → docker restart
|
||||
```
|
||||
|
||||
`detect_hosts()` determines which server is local at runtime and selects the correct share list. No script changes needed to reconfigure who syncs what — only `Master.conf` changes required.
|
||||
|
||||
**What it does:**
|
||||
1. Detects local server via `detect_hosts()` — determines HOST1 or HOST2
|
||||
2. Resolves remote Tailscale IP
|
||||
3. Runs a single pre-flight check — connectivity + remote rootfs
|
||||
4. Builds share list from `HOST1_DAILY_SYNC_SHARES` or `HOST2_DAILY_SYNC_SHARES`
|
||||
5. Appends personal shares (`HOST1_PERSONAL_SHARES` or `HOST2_PERSONAL_SHARES`)
|
||||
6. Calls `Rsync/rsync.sh` for each share
|
||||
7. Tracks pass/fail and duration per share
|
||||
8. Reports a combined summary
|
||||
`detect_hosts()` determines which server is local at runtime and selects the correct share list. No script changes needed — only `Master.conf` changes required.
|
||||
|
||||
**Why one pre-flight check upfront:**
|
||||
Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast. Individual share existence and disk checks still run per-share inside `rsync.sh`.
|
||||
|
||||
Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or rootfs is nearly full, the whole run fails fast. Individual share checks still run per-share inside `rsync.sh`.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```bash
|
||||
# Master.conf — per-host share lists
|
||||
# HOST1 truth shares — pushed from HOST1 to HOST2 nightly
|
||||
# Master.conf — ORCHESTRATORS section
|
||||
|
||||
DAILY_MAINTENANCE_SCRIPTS=(
|
||||
"git_pull_execute.sh" # always first
|
||||
"Docker_Essentials/docker_daily_restart.sh" # after sync and media jobs
|
||||
)
|
||||
|
||||
# Media jobs run between sync and docker restart
|
||||
# Permissions first, cleaners second, arr cleanup last
|
||||
MEDIA_MANAGEMENT_JOBS=(
|
||||
"Media/media_shares_permissions.sh" # permissions — everything depends on this
|
||||
"Media/media_cleaner.sh anime" # clean junk before arr scripts scan
|
||||
"Media/media_cleaner.sh media"
|
||||
"Media/lidarr_cleanup.sh" # arr cleanup last — depends on clean folders
|
||||
"Media/sonarr_cleanup.sh"
|
||||
"Media/radarr_cleanup.sh"
|
||||
)
|
||||
|
||||
HOST1_DAILY_SYNC_SHARES=(
|
||||
/mnt/user/Movies
|
||||
/mnt/user/Tv_Shows
|
||||
/mnt/user/Music
|
||||
# ...
|
||||
# all HOST1-owned shares
|
||||
)
|
||||
|
||||
# HOST2 truth shares — pushed from HOST2 to HOST1 nightly
|
||||
HOST2_DAILY_SYNC_SHARES=(
|
||||
/mnt/user/Anime_Shows
|
||||
/mnt/user/Anime_Movies
|
||||
# ...
|
||||
)
|
||||
```
|
||||
|
||||
These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call `rsync.sh` directly on a separate schedule instead.
|
||||
|
||||
**Relationship to failover writeback:**
|
||||
|
||||
The same share lists are used by `failover.sh` for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 18hr+ and HOST2's arrs downloaded new content, Tier 4 writeback pushes `HOST1_DAILY_SYNC_SHARES` FROM HOST2 back TO HOST1. No duplicate configuration needed.
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━
|
||||
📋 Shares: 11
|
||||
|
||||
━━━ [1/11] Movies ━━━
|
||||
...rsync output...
|
||||
✅ Movies — 4m32s
|
||||
|
||||
━━━ [2/11] Tv_Shows ━━━
|
||||
...
|
||||
━━━━━ 📋 DAILY SYNC SUMMARY ━━━━━
|
||||
✅ Pass: 10 ❌ Fail: 1
|
||||
⏱️ Duration: 47m12s
|
||||
❌ Failed: Anime_Shows-Old
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `critical_shares_full_sync.sh`
|
||||
|
||||
Runs a clean nightly sync for Emby and the auth stack (Critical-Data) with containers stopped. This is the companion to the hourly dirty sync — it provides a fully consistent state on HOST2 once per night.
|
||||
**Adding a media job:**
|
||||
|
||||
```bash
|
||||
# Scheduled as: 30 2 * * 0 (2:30am Sunday — weekly clean sync)
|
||||
/mnt/user/appdata/unraid_scripts/Orchestrators/critical_shares_full_sync.sh
|
||||
```
|
||||
|
||||
**Why two Emby syncs:**
|
||||
|
||||
The hourly dirty sync runs with Emby up — WAL files excluded, watch states pushed continuously. This means HOST2 is never more than an hour behind on watch state. But it's not a clean database snapshot.
|
||||
|
||||
The nightly clean sync stops Emby, syncs the full clean database state, then restarts. HOST2 gets a fully consistent Emby state every night. The two syncs work together:
|
||||
|
||||
```
|
||||
Hourly dirty sync (Emby running):
|
||||
users.db, library.db, authentication.db, config/
|
||||
WAL excluded — safe mid-write
|
||||
HOST2 always within 1hr of HOST1 on watch state
|
||||
|
||||
Nightly clean sync (Emby stopped):
|
||||
Full clean snapshot — all databases flushed
|
||||
No WAL files in flight
|
||||
HOST2 gets gold-standard state once per night
|
||||
```
|
||||
|
||||
**Why clean auth sync matters:**
|
||||
|
||||
The auth stack runs warm on both servers continuously. During normal operation HOST2's auth stack serves its own domain — it doesn't receive dirty updates from HOST1. The nightly clean sync is the only time auth state propagates.
|
||||
|
||||
This means:
|
||||
- New user added on HOST1 → propagates to HOST2 overnight automatically
|
||||
- Proxy rule changes → propagated overnight
|
||||
- No manual intervention needed for most auth changes
|
||||
|
||||
For users who just want failover to work — this script handles it. No thinking required about dirty writes, WAL files, or when to sync.
|
||||
|
||||
**What it syncs:**
|
||||
|
||||
```
|
||||
Emby appdata:
|
||||
users.db, library.db, authentication.db, config/
|
||||
Containers stopped → clean flush → safe copy
|
||||
|
||||
Critical-Data (auth stack):
|
||||
NPM proxy rules + SSL certs
|
||||
Authelia config + database
|
||||
Mariadb-Authelia data
|
||||
Redis-Authelia session store
|
||||
LLDAP users and groups database
|
||||
All auth containers stopped → clean databases → safe copy
|
||||
Authelia delayed start on restart — Mariadb + Redis must be ready first
|
||||
```
|
||||
|
||||
**What it excludes (per rsync profile):**
|
||||
|
||||
```
|
||||
Emby: logs, transcodes, cache, metadata, *.db-wal, *.db-shm
|
||||
Auth: logs, *.tmp, nginx/temp, nginx/cache, notification.txt
|
||||
```
|
||||
|
||||
### `media_management.sh`
|
||||
|
||||
Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Scheduled once daily, typically after the nightly sync.
|
||||
|
||||
```bash
|
||||
# Scheduled as: 0 2 * * * (2am daily — after media_shares_sync.sh)
|
||||
/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Reads `MEDIA_MAINTENANCE_JOBS` from `Master.conf`
|
||||
2. Runs each job in order — script path + optional argument
|
||||
3. Tracks pass/fail per job
|
||||
4. Reports a combined summary
|
||||
5. A failure in one job does not stop the others
|
||||
|
||||
**Why order matters:**
|
||||
|
||||
```
|
||||
1. media_shares_permissions.sh ← permissions first — everything else depends on correct ownership
|
||||
2. media_cleaner.sh anime ← clean junk before arr scripts scan
|
||||
3. media_cleaner.sh media ← same
|
||||
4. lidarr_cleanup.sh ← arr cleanup last — depends on clean folders
|
||||
5. sonarr_cleanup.sh
|
||||
6. radarr_cleanup.sh
|
||||
```
|
||||
|
||||
If arr cleanup runs before permissions, it may fail to delete files it doesn't have access to. If it runs before the cleaner, it finds junk files mixed in with real content. The order is intentional.
|
||||
|
||||
**Configuration:**
|
||||
```bash
|
||||
# Master.conf — add, remove, or reorder jobs here
|
||||
# Format: "folder/script.sh optional_argument"
|
||||
MEDIA_MAINTENANCE_JOBS=(
|
||||
"Media/media_shares_permissions.sh"
|
||||
"Media/media_cleaner.sh anime"
|
||||
"Media/media_cleaner.sh media"
|
||||
"Media/lidarr_cleanup.sh"
|
||||
"Media/sonarr_cleanup.sh"
|
||||
"Media/radarr_cleanup.sh"
|
||||
)
|
||||
```
|
||||
|
||||
**Adding a new job:**
|
||||
```bash
|
||||
# Add a line to MEDIA_MAINTENANCE_JOBS — no script changes needed
|
||||
MEDIA_MAINTENANCE_JOBS=(
|
||||
MEDIA_MANAGEMENT_JOBS=(
|
||||
"Media/media_shares_permissions.sh"
|
||||
"Media/media_cleaner.sh anime"
|
||||
"Media/media_cleaner.sh media"
|
||||
@@ -254,10 +229,10 @@ MEDIA_MAINTENANCE_JOBS=(
|
||||
)
|
||||
```
|
||||
|
||||
**Disabling a job temporarily:**
|
||||
**Disabling a media job temporarily:**
|
||||
|
||||
```bash
|
||||
# Comment it out — easy to re-enable
|
||||
MEDIA_MAINTENANCE_JOBS=(
|
||||
MEDIA_MANAGEMENT_JOBS=(
|
||||
"Media/media_shares_permissions.sh"
|
||||
# "Media/media_cleaner.sh anime" # ← disabled, not deleted
|
||||
"Media/media_cleaner.sh media"
|
||||
@@ -267,31 +242,152 @@ MEDIA_MAINTENANCE_JOBS=(
|
||||
)
|
||||
```
|
||||
|
||||
**--dry-run support:**
|
||||
`media_management.sh --dry-run` passes `--dry-run` through to every child script. All scripts report what they would do without making changes. Useful for testing a new job before adding it to the live schedule.
|
||||
**Relationship to failover writeback:**
|
||||
|
||||
The same share lists are used by `failover.sh` for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 24hr+ and HOST2's arrs accumulated content, writeback pushes `HOST1_DAILY_SYNC_SHARES` FROM HOST2 BACK TO HOST1. No duplicate configuration needed.
|
||||
|
||||
---
|
||||
|
||||
### `weekly_sync_maintenance.sh`
|
||||
|
||||
Weekly maintenance window orchestrator — critical appdata clean sync, container updates, and weekly docker restarts. Runs Sunday 2:30am, fits before the 3am network reboot.
|
||||
|
||||
```bash
|
||||
/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run
|
||||
# Scheduled as: 30 2 * * 0 (Sunday 2:30am)
|
||||
/mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh
|
||||
```
|
||||
|
||||
**Execution order:**
|
||||
|
||||
```
|
||||
1. Stop local containers — auth stack + Emby stopped locally
|
||||
2. Stop remote containers — auth stack + Emby stopped remotely via SSH
|
||||
3. Pull updates locally — if CRITICAL_SYNC_UPDATES=true
|
||||
4. Pull updates remotely — if CRITICAL_SYNC_UPDATES_REMOTE=true
|
||||
5. rsync WEEKLY_SYNC_JOBS — Emby + Critical-Data clean sync
|
||||
6. Start remote containers — starts on new images, correct order
|
||||
7. Start local containers — starts on new images, correct order
|
||||
|
||||
Post-sync jobs (WEEKLY_MAINTENANCE_SCRIPTS):
|
||||
8. docker_weekly_restart.sh
|
||||
```
|
||||
|
||||
**Why two Emby syncs:**
|
||||
|
||||
The emby-failover dirty sync runs every 30-60 minutes with Emby running — WAL files excluded, watch states and library pushed continuously. HOST2 stays current on what users are watching. But it is not a clean database snapshot.
|
||||
|
||||
The weekly clean sync stops Emby on both sides, checkpoints the WAL, and pushes a full consistent mirror. HOST2 gets a gold-standard Emby state once per week.
|
||||
|
||||
```
|
||||
emby-failover every 30-60min (Emby running):
|
||||
users.db, library.db, authentication.db, config/
|
||||
WAL excluded — safe mid-write
|
||||
HOST2 always within 30-60min of HOST1 on watch state
|
||||
|
||||
weekly clean sync Sunday 2:30am (Emby stopped):
|
||||
Full clean mirror — all databases flushed
|
||||
metadata, plugins, config all included
|
||||
~30s downtime — both Emby instances down during sync only
|
||||
Cache stays warm on HOST2 all week — only reset Sunday
|
||||
```
|
||||
|
||||
**Why weekly instead of nightly:**
|
||||
|
||||
Emby builds a warm image cache on HOST2 naturally throughout the week. Syncing nightly resets this cache — users experience slow image loads every morning. Weekly sync lets the cache stay warm for 6 days and only resets on Sunday night when most users are asleep.
|
||||
|
||||
**What it syncs:**
|
||||
|
||||
```
|
||||
WEEKLY_SYNC_JOBS (configurable in Master.conf):
|
||||
/mnt/user/Media_Server/Emby ← emby profile — full clean mirror
|
||||
/mnt/user/appdata-Failover/Critical-Data ← critical-data profile — auth stack
|
||||
|
||||
Emby excludes: logs, transcodes, cache, crash*
|
||||
Auth excludes: logs, *.tmp, nginx/temp, nginx/cache, notification.txt
|
||||
```
|
||||
|
||||
**Container update window:**
|
||||
|
||||
Containers are already stopped for the sync — container image updates pull at zero extra downtime. Both servers start on the same new image version after the sync.
|
||||
|
||||
```bash
|
||||
# Master.conf toggles
|
||||
CRITICAL_SYNC_UPDATES=true # pull updates locally
|
||||
CRITICAL_SYNC_UPDATES_REMOTE=true # pull updates on remote via SSH
|
||||
|
||||
# Toggle false to skip updates without changing the schedule
|
||||
CRITICAL_SYNC_UPDATES=false
|
||||
```
|
||||
|
||||
**Why auth stack matters:**
|
||||
|
||||
The auth stack (Authelia, NPM, Mariadb, Redis, LLDAP) runs warm on both servers. During normal operation HOST2 serves its own domain independently. The weekly clean sync is the only time auth state propagates from HOST1 to HOST2.
|
||||
|
||||
- New user added on HOST1 → propagates to HOST2 on Sunday automatically
|
||||
- Proxy rule changes → propagated Sunday
|
||||
- No manual intervention needed for routine auth changes
|
||||
|
||||
**Sunday maintenance window:**
|
||||
|
||||
```
|
||||
2:30am weekly_sync_maintenance.sh ← clean sync + updates (~3-5min)
|
||||
2:50am CA Auto Update plugin ← plugin updates
|
||||
2:55am CA container updates ← docker container updates
|
||||
3:00am Network reboot ← router/switch restart
|
||||
|
||||
Everything comes back clean:
|
||||
Network fresh, Emby updated, auth stack updated
|
||||
All in one maintenance window while users sleep
|
||||
```
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```bash
|
||||
# Master.conf — ORCHESTRATORS section
|
||||
|
||||
WEEKLY_SYNC_JOBS=(
|
||||
"/mnt/user/Media_Server/Emby"
|
||||
"/mnt/user/appdata-Failover/Critical-Data"
|
||||
)
|
||||
|
||||
WEEKLY_MAINTENANCE_SCRIPTS=(
|
||||
"Docker_Essentials/docker_weekly_restart.sh"
|
||||
)
|
||||
|
||||
CRITICAL_SYNC_UPDATES=true
|
||||
CRITICAL_SYNC_UPDATES_REMOTE=true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `media_management.sh`
|
||||
|
||||
Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Absorbed into `daily_sync_maintenance.sh` via `MEDIA_MANAGEMENT_JOBS` — not scheduled separately. Available for manual runs.
|
||||
|
||||
```bash
|
||||
# Manual use only — called automatically by daily_sync_maintenance.sh
|
||||
bash /mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run
|
||||
bash /mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The Orchestrator Pattern
|
||||
|
||||
Both orchestrators follow the same pattern. This is by design — any script that needs to coordinate multiple operations should follow it:
|
||||
All orchestrators follow the same pattern:
|
||||
|
||||
```
|
||||
1. Setup — validate config, detect hosts if needed
|
||||
1. Setup — validate config, detect hosts if needed, acquire lock
|
||||
2. Pre-flight — fail fast checks before doing any work
|
||||
3. Job loop — run each job, track pass/fail, continue on failure
|
||||
4. Summary — one clean report of all results
|
||||
5. Notification — one notification per run, not one per job
|
||||
```
|
||||
|
||||
This pattern means:
|
||||
- **Consistent output** — every orchestrator looks the same in the logs
|
||||
This means:
|
||||
- **Consistent output** — every orchestrator looks the same in logs
|
||||
- **No silent failures** — pass/fail tracked per job, reported in summary
|
||||
- **Single notification** — one bell ring, not six
|
||||
- **Single notification** — one bell ring per run
|
||||
- **Resilient** — one job failing doesn't stop the rest
|
||||
|
||||
---
|
||||
@@ -299,23 +395,39 @@ This pattern means:
|
||||
## Scheduling
|
||||
|
||||
```bash
|
||||
# Recommended schedule
|
||||
*/3 * * * * transcode_management.sh # cleanup then manager — every 3 minutes
|
||||
0 1 * * * media_shares_sync.sh # 1am — media shares to remote
|
||||
0 2 * * * media_management.sh # 2am — permissions, cleaners, arr cleanup
|
||||
30 2 * * 0 critical_shares_full_sync.sh # 2:30am Sunday — clean Emby + auth stack
|
||||
# At Startup of Array
|
||||
array_start.sh # single entry — launches all startup scripts
|
||||
|
||||
# Every 3 minutes
|
||||
*/3 * * * * transcode_management.sh
|
||||
|
||||
# Every 6 hours
|
||||
0 */6 * * * arrs_failed_stalled_recovery.sh
|
||||
|
||||
# Every 30-60 minutes
|
||||
*/30 * * * * rsync.sh /mnt/user/Media_Server/Emby --profile=emby-failover
|
||||
|
||||
# Daily 1am — full daily maintenance window:
|
||||
# git pull → media sync → permissions → cleaners → arr cleanup → docker restart
|
||||
0 1 * * * daily_sync_maintenance.sh
|
||||
|
||||
# Weekly — Sunday morning
|
||||
30 2 * * 0 weekly_sync_maintenance.sh # clean sync + updates + docker weekly restart
|
||||
50 2 * * 0 CA plugin update
|
||||
55 2 * * 0 CA container updates
|
||||
```
|
||||
|
||||
`media_shares_sync.sh` and `media_management.sh` run nightly — media shares and maintenance. `critical_shares_full_sync.sh` runs weekly on Sunday — it stops Emby and the auth stack for a clean consistent sync. Running it weekly instead of nightly lets Emby's image cache stay warm on HOST2 throughout the week. The emby-failover dirty sync handles watch states, library structure, and auth every 30-60 minutes — the weekly clean sync covers metadata, plugins, and a full database flush.
|
||||
`daily_sync_maintenance.sh` owns the entire daily window — git pull, media sync, permissions, cleaners, arr cleanup, and docker restarts in one scheduled run. Everything configured in `Master.conf` via `DAILY_MAINTENANCE_SCRIPTS` and `MEDIA_MANAGEMENT_JOBS`.
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Orchestrator
|
||||
|
||||
If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. The pattern is simple:
|
||||
If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. Model it directly on `media_management.sh` which handles dry-run passthrough, status display, pass/fail tracking and summary reporting.
|
||||
|
||||
Minimal skeleton:
|
||||
|
||||
```bash
|
||||
# Minimal orchestrator skeleton
|
||||
JOBS=(
|
||||
"Folder/script1.sh"
|
||||
"Folder/script2.sh arg"
|
||||
@@ -327,13 +439,11 @@ FAIL=()
|
||||
for JOB in "${JOBS[@]}"; do
|
||||
SCRIPT=$(echo "$JOB" | cut -d' ' -f1)
|
||||
ARG=$(echo "$JOB" | cut -d' ' -f2-)
|
||||
|
||||
|
||||
if bash "$ECOSYSTEM_ROOT/$SCRIPT" $ARG; then
|
||||
PASS+=("$SCRIPT")
|
||||
else
|
||||
FAIL+=("$SCRIPT")
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
Better yet — model it directly on `media_management.sh` which already handles dry-run passthrough, status display, pass/fail tracking and summary reporting.
|
||||
```
|
||||
Reference in New Issue
Block a user