FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

created loop for both watchdogs and consolidated orch lists

Browse Source
This commit is contained in:
FailedProxy
2026-04-23 22:44:56 -04:00
parent 8f60e5dde4
commit a3f90ed175
11 changed files with 2530 additions and 1467 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Orchestrators/README-Orchestrators.md
+293 -183
View File
@@ -13,14 +13,68 @@ Without orchestrators, each script runs independently on its own schedule. This
- **Race conditions** — two scripts running simultaneously on the same data
- **Order dependency failures** — media cleaner runs before permissions, finds wrong ownership
- **No combined summary** — 6 separate notifications instead of one clean report
- **Scheduling complexity** — 6+ cron entries instead of one
- **Scheduling complexity** — many cron entries instead of a few clean ones
Orchestrators solve this by making a set of related scripts into a single scheduled unit with a defined execution order and a unified summary.
---
## The Orchestrator Model
The ecosystem is designed so the User Scripts plugin contains only a small number of entries — each one an orchestrator that owns a domain:
```
At Startup of Array:
array_start.sh ← single entry, launches everything
Cron:
transcode_management.sh ← */3 * * * *
arrs_failed_stalled_recovery.sh ← 0 */6 * * *
rsync.sh ... emby-failover ← */30 * * * *
daily_sync_maintenance.sh ← 0 1 * * *
weekly_sync_maintenance.sh ← 30 2 * * 0
weekly_health_digest.sh ← Saturday morning
Manual only:
failover_test.sh, emby_database_repair.sh, repair tools
```
All job lists are configured in the `ORCHESTRATORS` section of `Master.conf`. No changes to orchestrator scripts needed when adding or removing jobs.
---
## Scripts
### `array_start.sh`
Single entry point for the User Scripts "At Startup of Array" schedule. Launches all array-start scripts in order — each as a background process.
```bash
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh
```
One-shot scripts (ramdisk, syslog filter, php-fpm, network connect) run and exit naturally. Continuous scripts (system watchdog, docker watchdog, failover) run until the array stops.
**Configuration:**
```bash
# Master.conf — ORCHESTRATORS section
ARRAY_START_SCRIPTS=(
"unRAID_Essentials/ramdisk_setup.sh" # creates ramdisk before Emby starts
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth log noise
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI tuning
"Docker_Essentials/docker_network_connect.sh" # connect containers to extra networks
"unRAID_Essentials/system_watchdog.sh" # continuous system health monitor
"Docker_Essentials/docker_watchdog.sh" # continuous container health monitor
"Failover/failover.sh" # continuous mutual failover
)
```
Add or remove scripts from `ARRAY_START_SCRIPTS` — no changes to `array_start.sh` needed. Order matters — ramdisk before Emby, network before watchdogs, watchdogs before failover.
---
### `transcode_management.sh`
Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order every 3 minutes. Replaces two separate cron entries with one.
@@ -32,13 +86,13 @@ Runs `transcode_cleanup.sh` then `transcode_manager.sh` in the correct order eve
**Why cleanup must run before manager:**
If the manager runs first it may see inflated ramdisk usage from stale segment files left by ended sessions — and trigger an unnecessary flip to SSD. Cleanup runs first to clear those files, then the manager makes its threshold decision based on real active session usage.
If the manager runs first it sees inflated ramdisk usage from stale segment files left by ended sessions — and triggers an unnecessary flip to SSD. Cleanup runs first to clear those files, then the manager decides based on real active session usage.
```
Without correct order:
Manager checks usage → 6.8GB (includes stale files) → flips to SSD
Cleanup runs → removes stale files → actual usage 2.1GB
Manager was wrong — unnecessary flip
Unnecessary flip — sessions now on SSD
With correct order:
Cleanup runs → removes stale files → actual usage 2.1GB
@@ -47,7 +101,7 @@ With correct order:
**Daily statistics tracking:**
Every cycle `transcode_management.sh` records stats to `/boot/config/transcode_daily.db`:
Every cycle `transcode_management.sh` records stats to `TRANSCODE_DAILY_LOG`:
- Peak ramdisk usage for the day
- Total flip count for the day
- Ramdisk vs SSD session counts
@@ -57,193 +111,114 @@ Every cycle `transcode_management.sh` records stats to `/boot/config/transcode_d
---
### `media_shares_sync.sh`
### `arrs_failed_stalled_recovery.sh`
Syncs each server's source-of-truth media shares to the remote server sequentially. Each server only pushes the shares it owns — direction and share list are automatic based on which server is running the script.
Automatically detects and recovers from failed imports and stalled downloads across Sonarr, Radarr, and Lidarr. Blocklists the bad release and triggers a new search — hands-free recovery while you sleep.
```bash
# Scheduled as: 0 */6 * * * (every 6 hours)
/mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh
```
Targets four problem types: `importFailed`, `importPending`, `error` status, and `stalled` downloads. Items newer than `ARR_IMPORT_RECOVERY_AGE` (6 hours) are skipped — gives the arr time to retry on its own first.
**API versions:** Sonarr v4 → `/api/v3/` — Radarr v6 → `/api/v3/` — Lidarr v3 → `/api/v1/`
Lidarr runs on HOST1 only — exits cleanly on HOST2.
**Configuration:**
```bash
# Master.conf — MEDIA section
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
HOST1_SONARR_RECOVERY=true
HOST1_RADARR_RECOVERY=true
HOST1_LIDARR_RECOVERY=true
HOST2_SONARR_RECOVERY=true
HOST2_RADARR_RECOVERY=true
```
---
### `daily_sync_maintenance.sh`
Full daily maintenance window orchestrator — git pull, media share sync, media management, and docker daily restarts. All driven by `Master.conf` arrays.
```bash
# Scheduled as: 0 1 * * * (1am daily — on both servers)
/mnt/user/appdata/unraid_scripts/Orchestrators/media_shares_sync.sh
/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync_maintenance.sh
```
**Execution order:**
```
1. Pre-sync jobs (DAILY_MAINTENANCE_SCRIPTS — git pull first):
git_pull_execute.sh ← always runs first — pulls latest scripts
2. Media share sync:
HOST*_DAILY_SYNC_SHARES ← each server pushes its own truth shares
HOST*_PERSONAL_SHARES ← personal encrypted shares appended after
3. Post-sync jobs (DAILY_MAINTENANCE_SCRIPTS — remaining):
media_management.sh ← permissions + cleaners + arr cleanup
docker_daily_restart.sh ← daily container restarts
```
**Bidirectional — same script, correct direction automatically:**
```
HOST1 runs media_shares_sync.sh → pushes HOST1_DAILY_SYNC_SHARES → TO HOST2
Movies, Tv_Shows, Music, Books etc. — HOST1 is source of truth
HOST1 runs daily_sync_maintenance.sh:
git pull → sync HOST1_DAILY_SYNC_SHARES → TO HOST2 → media_management → docker restart
HOST2 runs media_shares_sync.sh → pushes HOST2_DAILY_SYNC_SHARES → TO HOST1
Anime_Shows, Anime_Movies — HOST2 is source of truth
HOST2 runs daily_sync_maintenance.sh:
git pull → sync HOST2_DAILY_SYNC_SHARES → TO HOST1 → media_management → docker restart
```
`detect_hosts()` determines which server is local at runtime and selects the correct share list. No script changes needed to reconfigure who syncs what — only `Master.conf` changes required.
**What it does:**
1. Detects local server via `detect_hosts()` — determines HOST1 or HOST2
2. Resolves remote Tailscale IP
3. Runs a single pre-flight check — connectivity + remote rootfs
4. Builds share list from `HOST1_DAILY_SYNC_SHARES` or `HOST2_DAILY_SYNC_SHARES`
5. Appends personal shares (`HOST1_PERSONAL_SHARES` or `HOST2_PERSONAL_SHARES`)
6. Calls `Rsync/rsync.sh` for each share
7. Tracks pass/fail and duration per share
8. Reports a combined summary
`detect_hosts()` determines which server is local at runtime and selects the correct share list. No script changes needed — only `Master.conf` changes required.
**Why one pre-flight check upfront:**
Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast. Individual share existence and disk checks still run per-share inside `rsync.sh`.
Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or rootfs is nearly full, the whole run fails fast. Individual share checks still run per-share inside `rsync.sh`.
**Configuration:**
```bash
# Master.conf — per-host share lists
# HOST1 truth shares — pushed from HOST1 to HOST2 nightly
# Master.conf — ORCHESTRATORS section
DAILY_MAINTENANCE_SCRIPTS=(
"git_pull_execute.sh" # always first
"Docker_Essentials/docker_daily_restart.sh" # after sync and media jobs
)
# Media jobs run between sync and docker restart
# Permissions first, cleaners second, arr cleanup last
MEDIA_MANAGEMENT_JOBS=(
"Media/media_shares_permissions.sh" # permissions — everything depends on this
"Media/media_cleaner.sh anime" # clean junk before arr scripts scan
"Media/media_cleaner.sh media"
"Media/lidarr_cleanup.sh" # arr cleanup last — depends on clean folders
"Media/sonarr_cleanup.sh"
"Media/radarr_cleanup.sh"
)
HOST1_DAILY_SYNC_SHARES=(
/mnt/user/Movies
/mnt/user/Tv_Shows
/mnt/user/Music
# ...
# all HOST1-owned shares
)
# HOST2 truth shares — pushed from HOST2 to HOST1 nightly
HOST2_DAILY_SYNC_SHARES=(
/mnt/user/Anime_Shows
/mnt/user/Anime_Movies
# ...
)
```
These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call `rsync.sh` directly on a separate schedule instead.
**Relationship to failover writeback:**
The same share lists are used by `failover.sh` for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 18hr+ and HOST2's arrs downloaded new content, Tier 4 writeback pushes `HOST1_DAILY_SYNC_SHARES` FROM HOST2 back TO HOST1. No duplicate configuration needed.
**Example output:**
```
━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━
📋 Shares: 11
━━━ [1/11] Movies ━━━
...rsync output...
✅ Movies — 4m32s
━━━ [2/11] Tv_Shows ━━━
...
━━━━━ 📋 DAILY SYNC SUMMARY ━━━━━
✅ Pass: 10 ❌ Fail: 1
⏱️ Duration: 47m12s
❌ Failed: Anime_Shows-Old
```
---
### `critical_shares_full_sync.sh`
Runs a clean nightly sync for Emby and the auth stack (Critical-Data) with containers stopped. This is the companion to the hourly dirty sync — it provides a fully consistent state on HOST2 once per night.
**Adding a media job:**
```bash
# Scheduled as: 30 2 * * 0 (2:30am Sunday — weekly clean sync)
/mnt/user/appdata/unraid_scripts/Orchestrators/critical_shares_full_sync.sh
```
**Why two Emby syncs:**
The hourly dirty sync runs with Emby up — WAL files excluded, watch states pushed continuously. This means HOST2 is never more than an hour behind on watch state. But it's not a clean database snapshot.
The nightly clean sync stops Emby, syncs the full clean database state, then restarts. HOST2 gets a fully consistent Emby state every night. The two syncs work together:
```
Hourly dirty sync (Emby running):
users.db, library.db, authentication.db, config/
WAL excluded — safe mid-write
HOST2 always within 1hr of HOST1 on watch state
Nightly clean sync (Emby stopped):
Full clean snapshot — all databases flushed
No WAL files in flight
HOST2 gets gold-standard state once per night
```
**Why clean auth sync matters:**
The auth stack runs warm on both servers continuously. During normal operation HOST2's auth stack serves its own domain — it doesn't receive dirty updates from HOST1. The nightly clean sync is the only time auth state propagates.
This means:
- New user added on HOST1 → propagates to HOST2 overnight automatically
- Proxy rule changes → propagated overnight
- No manual intervention needed for most auth changes
For users who just want failover to work — this script handles it. No thinking required about dirty writes, WAL files, or when to sync.
**What it syncs:**
```
Emby appdata:
users.db, library.db, authentication.db, config/
Containers stopped → clean flush → safe copy
Critical-Data (auth stack):
NPM proxy rules + SSL certs
Authelia config + database
Mariadb-Authelia data
Redis-Authelia session store
LLDAP users and groups database
All auth containers stopped → clean databases → safe copy
Authelia delayed start on restart — Mariadb + Redis must be ready first
```
**What it excludes (per rsync profile):**
```
Emby: logs, transcodes, cache, metadata, *.db-wal, *.db-shm
Auth: logs, *.tmp, nginx/temp, nginx/cache, notification.txt
```
### `media_management.sh`
Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Scheduled once daily, typically after the nightly sync.
```bash
# Scheduled as: 0 2 * * * (2am daily — after media_shares_sync.sh)
/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh
```
**What it does:**
1. Reads `MEDIA_MAINTENANCE_JOBS` from `Master.conf`
2. Runs each job in order — script path + optional argument
3. Tracks pass/fail per job
4. Reports a combined summary
5. A failure in one job does not stop the others
**Why order matters:**
```
1. media_shares_permissions.sh ← permissions first — everything else depends on correct ownership
2. media_cleaner.sh anime ← clean junk before arr scripts scan
3. media_cleaner.sh media ← same
4. lidarr_cleanup.sh ← arr cleanup last — depends on clean folders
5. sonarr_cleanup.sh
6. radarr_cleanup.sh
```
If arr cleanup runs before permissions, it may fail to delete files it doesn't have access to. If it runs before the cleaner, it finds junk files mixed in with real content. The order is intentional.
**Configuration:**
```bash
# Master.conf — add, remove, or reorder jobs here
# Format: "folder/script.sh optional_argument"
MEDIA_MAINTENANCE_JOBS=(
"Media/media_shares_permissions.sh"
"Media/media_cleaner.sh anime"
"Media/media_cleaner.sh media"
"Media/lidarr_cleanup.sh"
"Media/sonarr_cleanup.sh"
"Media/radarr_cleanup.sh"
)
```
**Adding a new job:**
```bash
# Add a line to MEDIA_MAINTENANCE_JOBS — no script changes needed
MEDIA_MAINTENANCE_JOBS=(
MEDIA_MANAGEMENT_JOBS=(
"Media/media_shares_permissions.sh"
"Media/media_cleaner.sh anime"
"Media/media_cleaner.sh media"
@@ -254,10 +229,10 @@ MEDIA_MAINTENANCE_JOBS=(
)
```
**Disabling a job temporarily:**
**Disabling a media job temporarily:**
```bash
# Comment it out — easy to re-enable
MEDIA_MAINTENANCE_JOBS=(
MEDIA_MANAGEMENT_JOBS=(
"Media/media_shares_permissions.sh"
# "Media/media_cleaner.sh anime" # ← disabled, not deleted
"Media/media_cleaner.sh media"
@@ -267,31 +242,152 @@ MEDIA_MAINTENANCE_JOBS=(
)
```
**--dry-run support:**
`media_management.sh --dry-run` passes `--dry-run` through to every child script. All scripts report what they would do without making changes. Useful for testing a new job before adding it to the live schedule.
**Relationship to failover writeback:**
The same share lists are used by `failover.sh` for Tier 4 writeback — but in the opposite direction. If HOST1 was down for 24hr+ and HOST2's arrs accumulated content, writeback pushes `HOST1_DAILY_SYNC_SHARES` FROM HOST2 BACK TO HOST1. No duplicate configuration needed.
---
### `weekly_sync_maintenance.sh`
Weekly maintenance window orchestrator — critical appdata clean sync, container updates, and weekly docker restarts. Runs Sunday 2:30am, fits before the 3am network reboot.
```bash
/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run
# Scheduled as: 30 2 * * 0 (Sunday 2:30am)
/mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh
```
**Execution order:**
```
1. Stop local containers — auth stack + Emby stopped locally
2. Stop remote containers — auth stack + Emby stopped remotely via SSH
3. Pull updates locally — if CRITICAL_SYNC_UPDATES=true
4. Pull updates remotely — if CRITICAL_SYNC_UPDATES_REMOTE=true
5. rsync WEEKLY_SYNC_JOBS — Emby + Critical-Data clean sync
6. Start remote containers — starts on new images, correct order
7. Start local containers — starts on new images, correct order
Post-sync jobs (WEEKLY_MAINTENANCE_SCRIPTS):
8. docker_weekly_restart.sh
```
**Why two Emby syncs:**
The emby-failover dirty sync runs every 30-60 minutes with Emby running — WAL files excluded, watch states and library pushed continuously. HOST2 stays current on what users are watching. But it is not a clean database snapshot.
The weekly clean sync stops Emby on both sides, checkpoints the WAL, and pushes a full consistent mirror. HOST2 gets a gold-standard Emby state once per week.
```
emby-failover every 30-60min (Emby running):
users.db, library.db, authentication.db, config/
WAL excluded — safe mid-write
HOST2 always within 30-60min of HOST1 on watch state
weekly clean sync Sunday 2:30am (Emby stopped):
Full clean mirror — all databases flushed
metadata, plugins, config all included
~30s downtime — both Emby instances down during sync only
Cache stays warm on HOST2 all week — only reset Sunday
```
**Why weekly instead of nightly:**
Emby builds a warm image cache on HOST2 naturally throughout the week. Syncing nightly resets this cache — users experience slow image loads every morning. Weekly sync lets the cache stay warm for 6 days and only resets on Sunday night when most users are asleep.
**What it syncs:**
```
WEEKLY_SYNC_JOBS (configurable in Master.conf):
/mnt/user/Media_Server/Emby ← emby profile — full clean mirror
/mnt/user/appdata-Failover/Critical-Data ← critical-data profile — auth stack
Emby excludes: logs, transcodes, cache, crash*
Auth excludes: logs, *.tmp, nginx/temp, nginx/cache, notification.txt
```
**Container update window:**
Containers are already stopped for the sync — container image updates pull at zero extra downtime. Both servers start on the same new image version after the sync.
```bash
# Master.conf toggles
CRITICAL_SYNC_UPDATES=true # pull updates locally
CRITICAL_SYNC_UPDATES_REMOTE=true # pull updates on remote via SSH
# Toggle false to skip updates without changing the schedule
CRITICAL_SYNC_UPDATES=false
```
**Why auth stack matters:**
The auth stack (Authelia, NPM, Mariadb, Redis, LLDAP) runs warm on both servers. During normal operation HOST2 serves its own domain independently. The weekly clean sync is the only time auth state propagates from HOST1 to HOST2.
- New user added on HOST1 → propagates to HOST2 on Sunday automatically
- Proxy rule changes → propagated Sunday
- No manual intervention needed for routine auth changes
**Sunday maintenance window:**
```
2:30am weekly_sync_maintenance.sh ← clean sync + updates (~3-5min)
2:50am CA Auto Update plugin ← plugin updates
2:55am CA container updates ← docker container updates
3:00am Network reboot ← router/switch restart
Everything comes back clean:
Network fresh, Emby updated, auth stack updated
All in one maintenance window while users sleep
```
**Configuration:**
```bash
# Master.conf — ORCHESTRATORS section
WEEKLY_SYNC_JOBS=(
"/mnt/user/Media_Server/Emby"
"/mnt/user/appdata-Failover/Critical-Data"
)
WEEKLY_MAINTENANCE_SCRIPTS=(
"Docker_Essentials/docker_weekly_restart.sh"
)
CRITICAL_SYNC_UPDATES=true
CRITICAL_SYNC_UPDATES_REMOTE=true
```
---
### `media_management.sh`
Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Absorbed into `daily_sync_maintenance.sh` via `MEDIA_MANAGEMENT_JOBS` — not scheduled separately. Available for manual runs.
```bash
# Manual use only — called automatically by daily_sync_maintenance.sh
bash /mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run
bash /mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh
```
---
## The Orchestrator Pattern
Both orchestrators follow the same pattern. This is by design — any script that needs to coordinate multiple operations should follow it:
All orchestrators follow the same pattern:
```
1. Setup — validate config, detect hosts if needed
1. Setup — validate config, detect hosts if needed, acquire lock
2. Pre-flight — fail fast checks before doing any work
3. Job loop — run each job, track pass/fail, continue on failure
4. Summary — one clean report of all results
5. Notification — one notification per run, not one per job
```
This pattern means:
- **Consistent output** — every orchestrator looks the same in the logs
This means:
- **Consistent output** — every orchestrator looks the same in logs
- **No silent failures** — pass/fail tracked per job, reported in summary
- **Single notification** — one bell ring, not six
- **Single notification** — one bell ring per run
- **Resilient** — one job failing doesn't stop the rest
---
@@ -299,23 +395,39 @@ This pattern means:
## Scheduling
```bash
# Recommended schedule
*/3 * * * * transcode_management.sh # cleanup then manager — every 3 minutes
0 1 * * * media_shares_sync.sh # 1am — media shares to remote
0 2 * * * media_management.sh # 2am — permissions, cleaners, arr cleanup
30 2 * * 0 critical_shares_full_sync.sh # 2:30am Sunday — clean Emby + auth stack
# At Startup of Array
array_start.sh # single entry — launches all startup scripts
# Every 3 minutes
*/3 * * * * transcode_management.sh
# Every 6 hours
0 */6 * * * arrs_failed_stalled_recovery.sh
# Every 30-60 minutes
*/30 * * * * rsync.sh /mnt/user/Media_Server/Emby --profile=emby-failover
# Daily 1am — full daily maintenance window:
# git pull → media sync → permissions → cleaners → arr cleanup → docker restart
0 1 * * * daily_sync_maintenance.sh
# Weekly — Sunday morning
30 2 * * 0 weekly_sync_maintenance.sh # clean sync + updates + docker weekly restart
50 2 * * 0 CA plugin update
55 2 * * 0 CA container updates
```
`media_shares_sync.sh` and `media_management.sh` run nightly — media shares and maintenance. `critical_shares_full_sync.sh` runs weekly on Sunday — it stops Emby and the auth stack for a clean consistent sync. Running it weekly instead of nightly lets Emby's image cache stay warm on HOST2 throughout the week. The emby-failover dirty sync handles watch states, library structure, and auth every 30-60 minutes — the weekly clean sync covers metadata, plugins, and a full database flush.
`daily_sync_maintenance.sh` owns the entire daily window — git pull, media sync, permissions, cleaners, arr cleanup, and docker restarts in one scheduled run. Everything configured in `Master.conf` via `DAILY_MAINTENANCE_SCRIPTS` and `MEDIA_MANAGEMENT_JOBS`.
---
## Adding a New Orchestrator
If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. The pattern is simple:
If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. Model it directly on `media_management.sh` which handles dry-run passthrough, status display, pass/fail tracking and summary reporting.
Minimal skeleton:
```bash
# Minimal orchestrator skeleton
JOBS=(
"Folder/script1.sh"
"Folder/script2.sh arg"
@@ -327,13 +439,11 @@ FAIL=()
for JOB in "${JOBS[@]}"; do
SCRIPT=$(echo "$JOB" | cut -d' ' -f1)
ARG=$(echo "$JOB" | cut -d' ' -f2-)
if bash "$ECOSYSTEM_ROOT/$SCRIPT" $ARG; then
PASS+=("$SCRIPT")
else
FAIL+=("$SCRIPT")
fi
done
```
Better yet — model it directly on `media_management.sh` which already handles dry-run passthrough, status display, pass/fail tracking and summary reporting.
```
Orchestrators/array_start.sh
+106
View File
@@ -0,0 +1,106 @@
#!/bin/bash
# -----------------------------------------------------------------------------------------------
# --------------------------------- Array Start Orchestrator -----------------------------------
# -----------------------------------------------------------------------------------------------
# Launches all scripts configured in ARRAY_START_SCRIPTS when the unRAID array comes online.
# Set this script to run at "Startup of Array" in the User Scripts plugin.
#
# Each script is launched as a background process:
# One-shot scripts (ramdisk_setup, syslog_filter etc.) run and exit naturally
# Continuous scripts (system_watchdog, docker_watchdog, failover) run until array stops
#
# Scripts are launched in the order defined in ARRAY_START_SCRIPTS in Master.conf.
# Order matters — ramdisk before Emby, network before watchdogs, watchdogs before failover.
#
# To add or remove a script: edit ARRAY_START_SCRIPTS in Master.conf.
# No changes to this script needed.
#
# Logs: each script logs its own output independently.
# This orchestrator exits after launching all scripts — unRAID sees it complete normally.
# -----------------------------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/Master.conf"
source "$SCRIPT_DIR/common.sh"
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_GEAR Setup ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_GEAR Array Start — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
success "Running as root"
SCRIPT_COUNT=${#ARRAY_START_SCRIPTS[@]}
info "Launching $SCRIPT_COUNT script(s)..."
echo ""
LAUNCHED=0
FAILED=0
for relative_path in "${ARRAY_START_SCRIPTS[@]}"; do
[[ -z "$relative_path" ]] && continue
SCRIPT_PATH="$SCRIPT_DIR/$relative_path"
SCRIPT_NAME=$(basename "$SCRIPT_PATH")
if [[ ! -f "$SCRIPT_PATH" ]]; then
error "$SCRIPT_NAME — not found at $SCRIPT_PATH"
((FAILED++))
continue
fi
if [[ ! -x "$SCRIPT_PATH" ]]; then
error "$SCRIPT_NAME — not executable"
((FAILED++))
continue
fi
info "$ICON_START Launching $SCRIPT_NAME..."
bash "$SCRIPT_PATH" &
PID=$!
# Brief pause to let script initialize and catch immediate failures
sleep 1
if kill -0 "$PID" 2>/dev/null; then
success "$SCRIPT_NAME — running (PID $PID)"
((LAUNCHED++))
else
# Script exited — check if it was a one-shot (exit 0) or a failure
wait "$PID"
EXIT_CODE=$?
if [[ "$EXIT_CODE" -eq 0 ]]; then
success "$SCRIPT_NAME — completed (one-shot)"
((LAUNCHED++))
else
error "$SCRIPT_NAME — exited with code $EXIT_CODE"
((FAILED++))
fi
fi
done
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SUMMARY Summary ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━━━ $ICON_SUMMARY ARRAY START SUMMARY ━━━━━"
echo "$ICON_SUCCESS Launched: $LAUNCHED"
echo "$ICON_ERROR Failed: $FAILED"
echo "$ICON_TIME Time: $(date '+%H:%M:%S')"
echo ""
if [[ "$FAILED" -gt 0 ]]; then
echo "$ICON_WARN Status: $FAILED script(s) failed to launch — check logs"
notify "Array start on $(hostname) — $FAILED script(s) failed to launch" "Array Start" "warning"
else
echo "$ICON_DONE Status: $ICON_SUCCESS All scripts launched"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
Orchestrators/daily_sync_maintenance.sh
+284
View File
@@ -0,0 +1,284 @@
#!/bin/bash
# -----------------------------------------------------------------------------------------------
# --------------------------------- Daily Sync Maintenance ------------------------------------
# -----------------------------------------------------------------------------------------------
# Daily orchestrator — runs the full daily maintenance window in the correct order.
#
# What it does:
# 1. Iterates DAILY_MAINTENANCE_SCRIPTS — runs git pull first, then additional jobs
# 2. Syncs all media shares in the correct direction for the local server
# 3. Additional scripts in DAILY_MAINTENANCE_SCRIPTS run after the sync completes
#
# Media share sync:
# Each server pushes only the shares it owns (source of truth) — direction is automatic.
# HOST1 pushes: Movies, Tv_Shows, Music, Books etc. → HOST2
# HOST2 pushes: Anime_Shows, Anime_Movies → HOST1
# Personal encrypted shares synced after media shares.
# detect_hosts() determines which server is running — no script changes needed.
# Share lists configured in Master.conf ORCHESTRATORS section.
#
# Per-share rsync handled by rsync.sh — this script tracks pass/fail and total time.
# All job lists configured in Master.conf — no script changes needed to add or remove jobs.
# Schedule: 0 1 * * * (1am daily — configured in User Scripts plugin)
# -----------------------------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../Master.conf"
source "$SCRIPT_DIR/../common.sh"
RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
SCRIPTS_ROOT="$SCRIPT_DIR/.."
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_GEAR Setup ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
detect_hosts
resolve_remote_ip
acquire_lock
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SHIELD Pre-flight Checks ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_SHIELD Pre-flight Checks ━━━"
check_connectivity
check_remote_rootfs
WINDOW_START=$(date +%s)
JOB_PASS=()
JOB_FAIL=()
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_GIT Pre-sync Jobs ━━━
# git_pull_execute.sh runs first — pulls latest scripts before anything else runs
# Identified by script name — all other DAILY_MAINTENANCE_SCRIPTS run after sync
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_SYNC Pre-sync Jobs ━━━"
PRE_SYNC_SCRIPTS=()
POST_SYNC_SCRIPTS=()
for script_entry in "${DAILY_MAINTENANCE_SCRIPTS[@]}"; do
[[ -z "$script_entry" ]] && continue
script_name=$(basename "${script_entry%% *}")
if [[ "$script_name" == "git_pull_execute.sh" ]]; then
PRE_SYNC_SCRIPTS+=("$script_entry")
else
POST_SYNC_SCRIPTS+=("$script_entry")
fi
done
for script_entry in "${PRE_SYNC_SCRIPTS[@]}"; do
[[ -z "$script_entry" ]] && continue
script_args=($script_entry)
script_path="$SCRIPTS_ROOT/${script_args[0]}"
script_name=$(basename "${script_args[0]}")
extra_args=("${script_args[@]:1}")
echo ""
info "$ICON_START Running: $script_name"
if [[ ! -f "$script_path" ]]; then
error "$script_name — not found at $script_path"
JOB_FAIL+=("$script_name")
continue
fi
if bash "$script_path" "${extra_args[@]}"; then
success "$script_name — done"
JOB_PASS+=("$script_name")
else
error "$script_name — failed"
JOB_FAIL+=("$script_name")
fi
done
# -----------------------------------------------------------------------------------------------
# Build share list — host-specific truth shares + personal shares
# -----------------------------------------------------------------------------------------------
PASS=()
FAIL=()
SHARE_TIMES=()
TOTAL_START=$(date +%s)
ALL_SHARES=()
if [[ "$LOCAL_SERVER_NAME" == "$HOST1" ]]; then
for share in "${HOST1_DAILY_SYNC_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
for share in "${HOST1_PERSONAL_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
elif [[ "$LOCAL_SERVER_NAME" == "$HOST2" ]]; then
for share in "${HOST2_DAILY_SYNC_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
for share in "${HOST2_PERSONAL_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
fi
SHARE_COUNT=${#ALL_SHARES[@]}
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SYNC Media Share Sync ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_SYNC Media Share Sync — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo "$ICON_SUMMARY Shares: $SHARE_COUNT"
echo ""
SHARE_INDEX=0
for SHARE in "${ALL_SHARES[@]}"; do
SHARE_INDEX=$((SHARE_INDEX + 1))
SHARE_NAME=$(basename "$SHARE")
SHARE_START=$(date +%s)
echo "━━━ $ICON_SYNC Share $SHARE_INDEX of $SHARE_COUNT: $SHARE_NAME ━━━"
if bash "$RSYNC_SCRIPT" "$SHARE"; then
SHARE_END=$(date +%s)
SHARE_TIMES+=("$SHARE_NAME:$((SHARE_END - SHARE_START))")
PASS+=("$SHARE_NAME")
echo "$ICON_DONE $SHARE_NAME complete"
else
SHARE_END=$(date +%s)
SHARE_TIMES+=("$SHARE_NAME:$((SHARE_END - SHARE_START))")
FAIL+=("$SHARE_NAME")
error "$SHARE_NAME failed — continuing to next share"
fi
echo ""
done
TOTAL_END=$(date +%s)
TOTAL_DURATION=$((TOTAL_END - TOTAL_START))
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_CLEAN Post-sync Media Jobs ━━━
# Reads MEDIA_MANAGEMENT_JOBS from Master.conf — permissions, cleaners, arr cleanup
# Runs after sync completes — correct ownership available, clean folders guaranteed
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_CLEAN Post-sync Media Jobs ━━━"
if [[ ${#MEDIA_MANAGEMENT_JOBS[@]} -gt 0 ]]; then
for script_entry in "${MEDIA_MANAGEMENT_JOBS[@]}"; do
[[ -z "$script_entry" ]] && continue
script_args=($script_entry)
script_path="$SCRIPTS_ROOT/${script_args[0]}"
script_name=$(basename "${script_args[0]}")
extra_args=("${script_args[@]:1}")
echo ""
info "$ICON_START Running: $script_name ${extra_args[*]}"
if [[ ! -f "$script_path" ]]; then
error "$script_name — not found at $script_path"
JOB_FAIL+=("$script_name ${extra_args[*]}")
continue
fi
if [[ "$DRY_RUN" == true ]]; then
if bash "$script_path" "${extra_args[@]}" --dry-run; then
success "$script_name — done (dry run)"
JOB_PASS+=("$script_name ${extra_args[*]}")
else
error "$script_name — failed"
JOB_FAIL+=("$script_name ${extra_args[*]}")
fi
else
if bash "$script_path" "${extra_args[@]}"; then
success "$script_name — done"
JOB_PASS+=("$script_name ${extra_args[*]}")
else
error "$script_name — failed"
JOB_FAIL+=("$script_name ${extra_args[*]}")
fi
fi
done
fi
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_GEAR Post-sync System Jobs ━━━
# Reads remaining DAILY_MAINTENANCE_SCRIPTS — docker restart etc.
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_GEAR Post-sync Jobs ━━━"
for script_entry in "${POST_SYNC_SCRIPTS[@]}"; do
[[ -z "$script_entry" ]] && continue
script_args=($script_entry)
script_path="$SCRIPTS_ROOT/${script_args[0]}"
script_name=$(basename "${script_args[0]}")
extra_args=("${script_args[@]:1}")
echo ""
info "$ICON_START Running: $script_name"
if [[ ! -f "$script_path" ]]; then
error "$script_name — not found at $script_path"
JOB_FAIL+=("$script_name")
continue
fi
if bash "$script_path" "${extra_args[@]}"; then
success "$script_name — done"
JOB_PASS+=("$script_name")
else
error "$script_name — failed"
JOB_FAIL+=("$script_name")
fi
done
WINDOW_END=$(date +%s)
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SUMMARY Summary ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━━━ $ICON_SUMMARY DAILY SYNC MAINTENANCE SUMMARY ━━━━━"
echo "$ICON_TIME Window: $(date -d @$WINDOW_START '+%Y-%m-%d %H:%M:%S') → $(date -d @$WINDOW_END '+%H:%M:%S')"
echo "$ICON_TIME Duration: $(format_duration $(( WINDOW_END - WINDOW_START )))"
echo ""
echo "$ICON_SYNC Media shares:"
for entry in "${SHARE_TIMES[@]}"; do
SHARE_NAME="${entry%%:*}"
DURATION="${entry##*:}"
if printf '%s\n' "${FAIL[@]}" | grep -qx "$SHARE_NAME"; then
echo " $ICON_ERROR $SHARE_NAME — $(format_duration $DURATION)"
else
echo " $ICON_DONE $SHARE_NAME — $(format_duration $DURATION)"
fi
done
echo " Passed: ${#PASS[@]}/$SHARE_COUNT Failed: ${#FAIL[@]}/$SHARE_COUNT"
echo ""
if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
echo "$ICON_GEAR Jobs (media + system):"
for job in "${JOB_PASS[@]}"; do echo " $ICON_DONE $job"; done
for job in "${JOB_FAIL[@]}"; do echo " $ICON_ERROR $job"; done
echo ""
fi
TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
if [[ "$TOTAL_FAIL" -gt 0 ]]; then
echo "$ICON_WARN Status: $TOTAL_FAIL failure(s) — check logs"
notify "Daily sync maintenance completed with failures on $(hostname) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" "Daily Maintenance" "warning"
exit 1
else
echo "$ICON_DONE Status: $ICON_SUCCESS ALL COMPLETE"
notify "Daily sync maintenance complete on $(hostname) — ${#PASS[@]} shares synced, ${#JOB_PASS[@]} jobs run in $(format_duration $(( WINDOW_END - WINDOW_START )))" "Daily Maintenance" "normal"
exit 0
fi
Orchestrators/media_management.sh
-177
View File
@@ -1,177 +0,0 @@
#!/bin/bash
# -----------------------------------------------------------------------------------------------
# --------------------------------- Media Management Orchestrator ------------------------------
# -----------------------------------------------------------------------------------------------
# Runs all media maintenance scripts sequentially in the order defined in Master.conf.
# Each job in MEDIA_MAINTENANCE_JOBS is a script path with an optional argument.
# Scripts are resolved relative to the ecosystem root directory.
#
# To add a new job — edit MEDIA_MAINTENANCE_JOBS in Master.conf:
# "Media/my_new_script.sh" — script with no argument
# "Media/media_cleaner.sh profile" — script with argument
#
# Order matters — permissions runs before cleaners so files are correctly owned first.
# Each job runs independently — a failure in one does not stop the others.
# Supports --dry-run — passes through to all child scripts.
# -----------------------------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$ECOSYSTEM_ROOT/Master.conf"
source "$ECOSYSTEM_ROOT/common.sh"
parse_args "$@"
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_GEAR Setup ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
success "Running as root"
acquire_lock
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing --dry-run to all child scripts"
# Validate job list
if [[ ${#MEDIA_MAINTENANCE_JOBS[@]} -eq 0 ]]; then
warn "MEDIA_MAINTENANCE_JOBS is empty in Master.conf — nothing to run"
exit 0
fi
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SUMMARY Status ━━━
# -----------------------------------------------------------------------------------------------
if [[ "$SHOW_STATUS" == true ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
echo "$ICON_CLEAN Jobs to run: ${#MEDIA_MAINTENANCE_JOBS[@]}"
local_idx=1
for job in "${MEDIA_MAINTENANCE_JOBS[@]}"; do
echo " $local_idx. $job"
((local_idx++))
done
echo "$ICON_GEAR Dry Run: $DRY_RUN"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
# -----------------------------------------------------------------------------------------------
# Tracking
# -----------------------------------------------------------------------------------------------
PASS=()
FAIL=()
JOB_TIMES=()
TOTAL_START=$(date +%s)
# -----------------------------------------------------------------------------------------------
# JOB RUNNER
# Splits each MEDIA_MAINTENANCE_JOBS entry into script path + optional argument.
# Resolves script relative to ecosystem root. Passes --dry-run if active.
# Records pass/fail and duration for summary.
# -----------------------------------------------------------------------------------------------
run_job() {
local entry="$1"
# Split entry into script path and optional argument
local script_rel arg=""
read -r script_rel arg <<< "$entry"
local script="$ECOSYSTEM_ROOT/$script_rel"
local label
label="$(basename "$script_rel" .sh)${arg:+ $arg}"
local job_start
job_start=$(date +%s)
echo ""
echo "━━━ $ICON_CLEAN $label ━━━"
if [[ ! -f "$script" ]]; then
error "$script not found — skipping"
FAIL+=("$label")
JOB_TIMES+=("$label:0")
return
fi
if [[ ! -x "$script" ]]; then
warn "$script is not executable — attempting to fix"
chmod +x "$script"
fi
local dry_flag=""
[[ "$DRY_RUN" == true ]] && dry_flag="--dry-run"
# Run script with optional argument and optional dry-run flag
if bash "$script" $arg $dry_flag; then
local job_end
job_end=$(date +%s)
PASS+=("$label")
JOB_TIMES+=("$label:$((job_end - job_start))")
success "$label complete"
else
local job_end
job_end=$(date +%s)
FAIL+=("$label")
JOB_TIMES+=("$label:$((job_end - job_start))")
error "$label failed — continuing to next job"
fi
}
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_CLEAN Media Management ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_CLEAN Media Management — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo "$ICON_SUMMARY Jobs: ${#MEDIA_MAINTENANCE_JOBS[@]}"
echo ""
JOB_INDEX=1
for job in "${MEDIA_MAINTENANCE_JOBS[@]}"; do
info "Job $JOB_INDEX of ${#MEDIA_MAINTENANCE_JOBS[@]}: $job"
run_job "$job"
((JOB_INDEX++))
done
TOTAL_END=$(date +%s)
TOTAL_DURATION=$((TOTAL_END - TOTAL_START))
JOB_COUNT=$(( ${#PASS[@]} + ${#FAIL[@]} ))
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SUMMARY Summary ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━━━ $ICON_SUMMARY MEDIA MANAGEMENT SUMMARY ━━━━━"
echo "$ICON_TIME Started: $(date -d @$TOTAL_START '+%Y-%m-%d %H:%M:%S')"
echo "$ICON_TIME Finished: $(date -d @$TOTAL_END '+%Y-%m-%d %H:%M:%S')"
echo ""
for entry in "${JOB_TIMES[@]}"; do
label="${entry%%:*}"
duration="${entry##*:}"
if printf '%s\n' "${FAIL[@]}" | grep -qx "$label"; then
echo " $ICON_ERROR $label — $(format_duration $duration)"
else
echo " $ICON_DONE $label — $(format_duration $duration)"
fi
done
echo ""
echo " $ICON_SUCCESS Passed: ${#PASS[@]}/$JOB_COUNT"
echo " $ICON_ERROR Failed: ${#FAIL[@]}/$JOB_COUNT"
echo " $ICON_TIME Duration: $(format_duration $TOTAL_DURATION)"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
if [[ ${#FAIL[@]} -gt 0 ]]; then
notify "Media management completed with failures on $(hostname) — failed: ${FAIL[*]}" "Media Management" "warning"
exit 1
else
notify "Media management complete on $(hostname) — ${#PASS[@]}/$JOB_COUNT jobs in $(format_duration $TOTAL_DURATION)" "Media Management" "normal"
exit 0
fi
Orchestrators/media_shares_sync.sh
-141
View File
@@ -1,141 +0,0 @@
#!/bin/bash
# -----------------------------------------------------------------------------------------------
# --------------------------------- Media Shares Sync ------------------------------------
# -----------------------------------------------------------------------------------------------
# Runs all media shares sequentially in the correct direction for the local server.
# Each server pushes its own source-of-truth shares to the remote — direction is automatic.
#
# HOST1 pushes: its truth shares (Movies, Tv_Shows, Music etc.) + personal → HOST2
# HOST2 pushes: its truth shares (Anime_Shows, Anime_Movies etc.) + personal → HOST1
#
# detect_hosts() determines which server is running the script at runtime.
# Share lists are configured per host in Master.conf — no script changes needed
# to add, remove, or reconfigure shares.
#
# Per-share rsync handled by rsync.sh — this script tracks pass/fail and total time only.
# Scheduled via unRAID User Scripts plugin at 1am on both servers.
# -----------------------------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../Master.conf"
source "$SCRIPT_DIR/../common.sh"
RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_GEAR Setup ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
detect_hosts
resolve_remote_ip
acquire_lock
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SHIELD Pre-flight Checks ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_SHIELD Pre-flight Checks ━━━"
# Single connectivity and rootfs check upfront — fail fast before attempting all shares
# Individual share and disk checks run per-share inside rsync.sh
check_connectivity
check_remote_rootfs
# -----------------------------------------------------------------------------------------------
# Build share list — host-specific truth shares + personal shares
# Each server only syncs the shares it is source of truth for
# Personal shares appended after media shares
# -----------------------------------------------------------------------------------------------
PASS=()
FAIL=()
SHARE_TIMES=()
TOTAL_START=$(date +%s)
ALL_SHARES=()
if [[ "$LOCAL_SERVER_NAME" == "$HOST1" ]]; then
for share in "${HOST1_DAILY_SYNC_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
for share in "${HOST1_PERSONAL_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
elif [[ "$LOCAL_SERVER_NAME" == "$HOST2" ]]; then
for share in "${HOST2_DAILY_SYNC_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
for share in "${HOST2_PERSONAL_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
fi
SHARE_COUNT=${#ALL_SHARES[@]}
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SYNC Transfer ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━ $ICON_SYNC Daily Sync Starting — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo "$ICON_SUMMARY Shares: $SHARE_COUNT"
echo ""
SHARE_INDEX=0
for SHARE in "${ALL_SHARES[@]}"; do
SHARE_INDEX=$((SHARE_INDEX + 1))
SHARE_NAME=$(basename "$SHARE")
SHARE_START=$(date +%s)
echo "━━━ $ICON_SYNC Share $SHARE_INDEX of $SHARE_COUNT: $SHARE_NAME ━━━"
if bash "$RSYNC_SCRIPT" "$SHARE"; then
SHARE_END=$(date +%s)
SHARE_TIMES+=("$SHARE_NAME:$((SHARE_END - SHARE_START))")
PASS+=("$SHARE_NAME")
echo "$ICON_DONE $SHARE_NAME complete"
else
SHARE_END=$(date +%s)
SHARE_TIMES+=("$SHARE_NAME:$((SHARE_END - SHARE_START))")
FAIL+=("$SHARE_NAME")
error "$SHARE_NAME failed — continuing to next share"
fi
echo ""
done
TOTAL_END=$(date +%s)
TOTAL_DURATION=$((TOTAL_END - TOTAL_START))
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SUMMARY Summary ━━━
# -----------------------------------------------------------------------------------------------
echo "━━━━━ $ICON_SUMMARY DAILY SYNC SUMMARY ━━━━━"
echo "$ICON_TIME Started: $(date -d @$TOTAL_START '+%Y-%m-%d %H:%M:%S')"
echo "$ICON_TIME Finished: $(date -d @$TOTAL_END '+%Y-%m-%d %H:%M:%S')"
echo ""
for entry in "${SHARE_TIMES[@]}"; do
SHARE_NAME="${entry%%:*}"
DURATION="${entry##*:}"
if printf '%s\n' "${FAIL[@]}" | grep -qx "$SHARE_NAME"; then
echo " $ICON_ERROR $SHARE_NAME — $(format_duration $DURATION)"
else
echo " $ICON_DONE $SHARE_NAME — $(format_duration $DURATION)"
fi
done
echo ""
echo " $ICON_SUCCESS Passed: ${#PASS[@]}/$SHARE_COUNT"
echo " $ICON_ERROR Failed: ${#FAIL[@]}/$SHARE_COUNT"
echo " $ICON_TIME Duration: $(format_duration $TOTAL_DURATION)"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
if [[ ${#FAIL[@]} -gt 0 ]]; then
notify "Daily sync completed with failures — ${#FAIL[@]}/$SHARE_COUNT failed: ${FAIL[*]}" "Daily Sync" "warning"
exit 1
else
notify "Daily sync complete — ${#PASS[@]}/$SHARE_COUNT shares in $(format_duration $TOTAL_DURATION)" "Daily Sync" "normal"
exit 0
fi
Orchestrators/critical_shares_maintenance.sh → Orchestrators/weekly _sync_maintenance.sh
+64 -15
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# -----------------------------------------------------------------------------------------------
# ----------------------------- Critical Shares Maintenance ------------------------------------
# ----------------------------- Weekly Sync Maintenance ------------------------------------
# -----------------------------------------------------------------------------------------------
# Maintenance window orchestrator for Emby and the auth stack (Critical-Data).
# Both local and remote containers are stopped for the entire window — clean state
@@ -161,11 +161,7 @@ PASS=()
FAIL=()
TOTAL_START=$(date +%s)
SYNC_JOBS=(
"/mnt/user/Media_Server/Emby"
"/mnt/user/appdata-Failover/Critical-Data"
)
SYNC_JOBS=("${WEEKLY_SYNC_JOBS[@]}")
SHARE_COUNT=${#SYNC_JOBS[@]}
echo ""
@@ -221,32 +217,85 @@ fi
TOTAL_END=$(date +%s)
TOTAL_DURATION=$(format_duration $(( TOTAL_END - TOTAL_START )))
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_GEAR Post-sync Jobs ━━━
# docker_weekly_restart.sh and any other WEEKLY_MAINTENANCE_SCRIPTS run after sync
# -----------------------------------------------------------------------------------------------
JOB_PASS=()
JOB_FAIL=()
SCRIPTS_ROOT="$SCRIPT_DIR/.."
if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -gt 0 ]]; then
echo ""
echo "━━━ $ICON_GEAR Post-sync Jobs ━━━"
for script_entry in "${WEEKLY_MAINTENANCE_SCRIPTS[@]}"; do
[[ -z "$script_entry" ]] && continue
script_args=($script_entry)
script_path="$SCRIPTS_ROOT/${script_args[0]}"
script_name=$(basename "${script_args[0]}")
extra_args=("${script_args[@]:1}")
echo ""
info "$ICON_START Running: $script_name"
if [[ ! -f "$script_path" ]]; then
error "$script_name — not found at $script_path"
JOB_FAIL+=("$script_name")
continue
fi
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would run: $script_name"
JOB_PASS+=("$script_name (dry run)")
elif bash "$script_path" "${extra_args[@]}"; then
success "$script_name — done"
JOB_PASS+=("$script_name")
else
error "$script_name — failed"
JOB_FAIL+=("$script_name")
fi
done
fi
WINDOW_END=$(date +%s)
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SUMMARY Summary ━━━
# -----------------------------------------------------------------------------------------------
echo ""
echo "━━━━━ $ICON_SUMMARY CRITICAL SHARES MAINTENANCE SUMMARY ━━━━━"
echo "━━━━━ $ICON_SUMMARY WEEKLY SYNC MAINTENANCE SUMMARY ━━━━━"
echo "$ICON_TIME Duration: $TOTAL_DURATION"
echo "$ICON_GEAR Updates: local=$CRITICAL_SYNC_UPDATES remote=$CRITICAL_SYNC_UPDATES_REMOTE"
echo "$ICON_SUCCESS Passed: ${#PASS[@]} $ICON_ERROR Failed: ${#FAIL[@]}"
echo ""
echo "$ICON_SYNC Sync jobs:"
if [[ ${#PASS[@]} -gt 0 ]]; then
for job in "${PASS[@]}"; do echo " $ICON_SUCCESS $job"; done
for job in "${PASS[@]}"; do echo " $ICON_SUCCESS $job"; done
fi
if [[ ${#FAIL[@]} -gt 0 ]]; then
for job in "${FAIL[@]}"; do echo " $ICON_ERROR $job"; done
for job in "${FAIL[@]}"; do echo " $ICON_ERROR $job"; done
fi
echo " Passed: ${#PASS[@]} Failed: ${#FAIL[@]}"
if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
echo ""
echo "$ICON_GEAR Post-sync jobs:"
for job in "${JOB_PASS[@]}"; do echo " $ICON_SUCCESS $job"; done
for job in "${JOB_FAIL[@]}"; do echo " $ICON_ERROR $job"; done
fi
echo ""
TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
if [[ "$DRY_RUN" == true ]]; then
echo "$ICON_WARN Status: DRY RUN — no changes made"
elif [[ ${#FAIL[@]} -eq 0 ]]; then
echo "$ICON_DONE Status: $ICON_SUCCESS ALL JOBS COMPLETE"
notify "Critical maintenance complete on $(hostname) — synced + updated (local=$CRITICAL_SYNC_UPDATES remote=$CRITICAL_SYNC_UPDATES_REMOTE)" "Critical Maintenance" "normal"
elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
echo "$ICON_DONE Status: $ICON_SUCCESS ALL COMPLETE"
notify "Weekly sync maintenance complete on $(hostname) — synced + updated (local=$CRITICAL_SYNC_UPDATES remote=$CRITICAL_SYNC_UPDATES_REMOTE)" "Weekly Maintenance" "normal"
else
echo "$ICON_ERROR Status: $ICON_ERROR ${#FAIL[@]} JOB(S) FAILED"
notify "Critical maintenance failed on $(hostname) — failed: ${FAIL[*]}" "Critical Maintenance" "warning"
echo "$ICON_ERROR Status: $TOTAL_FAIL failure(s) — check logs"
notify "Weekly sync maintenance failed on $(hostname) — sync: ${#FAIL[@]} failed, jobs: ${#JOB_FAIL[@]} failed" "Weekly Maintenance" "warning"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"