1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Daily Sync Maintenance =========================================
4	# ==============================================================================================
5	# Daily orchestrator — runs the full daily maintenance window in the correct order.
6	# Schedule: 0 1 * * * (1am daily via User Scripts plugin)
7	#
8	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
9	#   Pre-sync:
10	#     git_pull_execute.sh              — pull latest scripts first, always
11	#
12	#   Arr Sync (arr_sync.sh):
13	#     Syncs Lidarr/Sonarr/Radarr libraries across all nodes bidirectionally.
14	#     All nodes agree on tracked library before any files are transferred.
15	#     Remote nodes that don't have an arr running are skipped gracefully.
16	#
17	#   Rsync window (DAILY_SYNC_SHARES per host):
18	#     HOST*_DAILY_SYNC_SHARES          — media shares spread to all nodes (no --delete)
19	#     HOST*_PERSONAL_SHARES            — encrypted personal shares
20	#
21	#   Post-sync maintenance (DAILY_MAINTENANCE_SCRIPTS):
22	#     media_shares_permissions.sh      — fix ownership before arr cleanup
23	#     media_cleaner.sh anime           — remove junk from anime shares
24	#     media_cleaner.sh media           — remove junk from media shares
25	#     lidarr_cleanup.sh                — remove orphaned music files (local arr = truth)
26	#     sonarr_cleanup.sh                — remove orphaned TV files (local arr = truth)
27	#     radarr_cleanup.sh                — remove orphaned movie files (local arr = truth)
28	#     docker_daily_restart.sh          — restart containers needing daily restart
29	#
30	# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
31	#   git pull first        — maintenance runs on latest code, not yesterday's
32	#   arr sync before rsync — all nodes track the same library before files are spread;
33	#                           prevents remote arrs from searching for content already owned
34	#   rsync before cleanup  — cleanup sees fully spread state, rsync has no --delete
35	#   permissions before arr cleanup — arrs need correct ownership to delete/rename
36	#   arr cleanup after permissions  — clean ownership = successful orphan deletion
37	#   docker restart last   — containers already processed by cleanup
38	#
39	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
40	# Bidirectional — same script runs on both servers, correct direction automatic.
41	# detect_hosts() aliases DAILY_SYNC_SHARES and PERSONAL_SHARES from HOST*_ vars.
42	# No manual HOST1/HOST2 comparisons — MY_ID routes correctly on any server.
43	#
44	# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
45	# rsync.sh returns exit codes for temperature issues:
46	#   exit 1 = temp WARN — skip this share, continue to next
47	#   exit 2 = temp CRITICAL — abort ALL remaining syncs in this window
48	#   All other failures — skip share, continue to next
49	#
50	# ── SILENT WHEN HEALTHY ───────────────────────────────────────────────────────────────────────
51	# Runs daily at 1am — clean run should produce minimal output.
52	# Each job reports log() on success (silent), warn()/error() on failure (visible).
53	# Summary always shown — gives window timing and share/job counts.
54	# Notify only on failure — successful daily maintenance doesn't need notification.
55	#
56	# ── CONFIGURATION (master.conf + host*.conf) ───────────────────────────────────────────
57	#   HOST*_DAILY_SYNC_SHARES      — shares pushed to mirror each day
58	#   HOST*_PERSONAL_SHARES        — encrypted personal shares
59	#   DAILY_MAINTENANCE_SCRIPTS    — maintenance jobs (permissions, cleanup, restart)
60	#   DAILY_RSYNC_ENABLED          — enable/disable rsync section
61	#
62	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
63	#   daily_sync_maintenance.sh              — normal run
64	#   daily_sync_maintenance.sh --dry-run    — preview without syncing or changing
65	#   daily_sync_maintenance.sh --log        — verbose per-share/per-job output
66	#   daily_sync_maintenance.sh --status     — show configured shares and jobs
67	# ==============================================================================================
68
69	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
70
71	source "$SCRIPT_DIR/../load_config.sh"
72
73	RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
74	SCRIPTS_ROOT="$SCRIPT_DIR/.."
75
76	parse_args "$@"
77
78	# ==============================================================================================
79	# ━━━ Setup ━━━
80	# ==============================================================================================
81	if [[ "$EUID" -ne 0 ]]; then
82	    error "Must be run as root"
83	    exit 1
84	fi
85
86
87	if ! command -v docker &>/dev/null; then
88	    error "Docker command not found"
89	    exit 1
90	fi
91
92	detect_hosts
93	resolve_remote_ip
94
95	acquire_lock
96
97	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
98
99	# ==============================================================================================
100	# ━━━ Status ━━━
101	# ==============================================================================================
102	if [[ "$SHOW_STATUS" == true ]]; then
103	    echo ""
104	    echo "━━━━━ $ICON_SUMMARY DAILY SYNC STATUS ━━━━━"
105	    echo "$ICON_HOST   Identity:        $MY_ID ($LOCAL_SERVER_NAME)"
106	    echo "$ICON_NET    Remote:          $REMOTE_ID ($REMOTE_SERVER_NAME)"
107	    echo "$ICON_SYNC   Rsync enabled:   ${RSYNC_ENABLED:-true}"
108	    echo "$ICON_SYNC   Daily enabled:   ${DAILY_RSYNC_ENABLED:-false}"
109	    echo ""
110	    echo "━━━ Daily Sync Shares ━━━"
111	    for share in "${DAILY_SYNC_SHARES[@]}"; do
112	        [[ -n "$share" ]] && echo "  $ICON_SYNC $share"
113	    done
114	    for share in "${PERSONAL_SHARES[@]}"; do
115	        [[ -n "$share" ]] && echo "  $ICON_SYNC $share (personal)"
116	    done
117	    echo ""
118	    echo "━━━ Daily Maintenance Scripts ━━━"
119	    for entry in "${DAILY_MAINTENANCE_SCRIPTS[@]}"; do
120	        [[ -z "$entry" ]] && continue


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Weekly Sync Maintenance ========================================
4	# ==============================================================================================
5	# Weekly maintenance window orchestrator — clean sync, container updates, weekly restarts.
6	# Schedule: 30 2 * * 0 (Sunday 2:30am — before Sunday 7am coffee report)
7	#
8	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
9	#   1. Stop local containers         — Emby + auth stack stopped locally
10	#   2. Stop remote containers        — Emby + auth stack stopped remotely via SSH
11	#   3. Pull updates locally          — if WEEKLY_SYNC_UPDATES=true (zero extra downtime)
12	#   4. Pull updates remotely         — if WEEKLY_SYNC_UPDATES_REMOTE=true
13	#   5. rsync WEEKLY_SYNC_SHARES      — full clean mirror, containers stopped both sides
14	#   6. Start remote containers       — correct order, delayed start respected
15	#   7. Start local containers        — correct order, delayed start respected
16	#   8. WEEKLY_MAINTENANCE_SCRIPTS    — weekly restarts etc. (docker_weekly_restart.sh)
17	#   9. docker_update.sh --remainder  — update all containers not in daily or weekly sync window
18	#
19	# ── WHY WEEKLY NOT NIGHTLY FOR EMBY ──────────────────────────────────────────────────────────
20	# Emby builds a warm image cache on HOST2 throughout the week.
21	# Syncing nightly resets cache — cold loads every morning for users.
22	# Weekly sync: cache stays warm 6 days, resets Sunday night while users sleep.
23	# emby-fallback dirty sync covers watch states + library every 30min between weekly syncs.
24	#
25	# ── CONTAINER UPDATES ─────────────────────────────────────────────────────────────────────────
26	# Containers already stopped for sync — updates pull at zero extra downtime.
27	# Both servers start on identical image versions after the window completes.
28	# Toggle: WEEKLY_SYNC_UPDATES / WEEKLY_SYNC_UPDATES_REMOTE in master.conf
29	#
30	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
31	# detect_hosts() sets MY_ID — used in banner, summary, and notifications.
32	# WEEKLY_SYNC_SHARES and WEEKLY_MAINTENANCE_SCRIPTS configured in master.conf.
33	# Same script runs correctly on both servers.
34	#
35	# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
36	#   Root check           — stop/start containers, rsync require root
37	#   acquire_lock         — prevents concurrent weekly windows
38	#   check_connectivity   — verifies remote before any remote operations
39	#   check_remote_rootfs  — aborts if remote rootfs nearly full
40	#   DOCKER_TIMEOUT       — all docker calls protected
41	#   SSH_TIMEOUT          — all SSH calls protected
42	#   platform_require_cmd  — notify validated before use
43	#   Silent on success    — runs weekly, only failures warrant notification
44	#
45	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
46	#   WEEKLY_SYNC_SHARES             — shares synced during window
47	#   WEEKLY_MAINTENANCE_SCRIPTS     — scripts run after sync
48	#   WEEKLY_SYNC_UPDATES            — toggle local container updates
49	#   WEEKLY_SYNC_UPDATES_REMOTE     — toggle remote container updates
50	#   WEEKLY_RSYNC_ENABLED           — enable/disable rsync section
51	#
52	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
53	#   weekly_sync_maintenance.sh              — normal run
54	#   weekly_sync_maintenance.sh --dry-run    — preview without stopping containers or syncing
55	#   weekly_sync_maintenance.sh --log        — verbose per-share/per-job output
56	#   weekly_sync_maintenance.sh --status     — show configuration and exit
57	# ==============================================================================================
58
59	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
60
61	source "$SCRIPT_DIR/../load_config.sh"
62
63	RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
64	SCRIPTS_ROOT="$SCRIPT_DIR/.."
65
66	parse_args "$@"
67
68	DOCKER_TIMEOUT=30   # container stop/start needs longer than normal
69	SSH_TIMEOUT=30      # remote pulls can be slow
70
71	# ==============================================================================================
72	# ━━━ Setup ━━━
73	# ==============================================================================================
74	if [[ "$EUID" -ne 0 ]]; then
75	    error "Must be run as root"
76	    exit 1
77	fi
78
79
80	acquire_lock
81
82	if ! command -v docker &>/dev/null; then
83	    error "Docker command not found"
84	    exit 1
85	fi
86
87	detect_hosts
88	resolve_remote_ip
89
90	WINDOW_START=$(date +%s)
91	PASS=()
92	FAIL=()
93	JOB_PASS=()
94	JOB_FAIL=()
95
96	# Load container lists from profile config
97	read -r -a MAINTENANCE_CONTAINERS <<< \
98	    "${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
99
100	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no containers will be stopped, no sync, no updates"
101
102	# ── Helper — run a post-sync maintenance script ────────────────────────────────────────────────
103	run_job() {
104	    local script_entry="$1"
105	    local extra_dry=""
106	    [[ "$DRY_RUN" == true ]] && extra_dry="--dry-run"
107
108	    read -r -a script_args <<< "$script_entry"
109	    local script_path="$SCRIPTS_ROOT/${script_args[0]}"
110	    local script_name
111	    script_name=$(basename "${script_args[0]}")
112	    local extra_args=("${script_args[@]:1}")
113
114	    if [[ ! -f "$script_path" ]]; then
115	        error "$script_name — not found at $script_path"
116	        JOB_FAIL+=("$script_name")
117	        return 1
118	    fi
119
120	    log "Running: $script_name ${extra_args[*]}"


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Sunday Morning Coffee Report ===================================
4	# ==============================================================================================
5	# Weekly monitoring orchestrator — runs all Sunday monitor scripts in sequence.
6	# Designed to be read over coffee Sunday morning while the system is fully caught up
7	# from the 2:30am maintenance window.
8	# Schedule: 0 7 * * 0 (7am Sunday — after weekly_sync_maintenance.sh finishes at ~3am)
9	#
10	# ── SCRIPTS (master.conf COFFEE_REPORT_SCRIPTS) ───────────────────────────────────────────────
11	#   Each script runs independently, logs to its own output, and notifies on findings.
12	#   All scripts receive --dry-run and --log flags from this orchestrator when set.
13	#
14	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
15	#   detect_hosts() sets MY_ID — used in banner and summary.
16	#   HOST1 primary: runs all scripts. HOST2: limited to host-aware scripts only.
17	#
18	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
19	#   sunday_morning_coffee_report.sh              — normal run
20	#   sunday_morning_coffee_report.sh --dry-run    — preview without any writes or notifications
21	#   sunday_morning_coffee_report.sh --log        — verbose per-script output
22	#   sunday_morning_coffee_report.sh --status     — show configured scripts and exit
23	# ==============================================================================================
24
25	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
26
27	source "$SCRIPT_DIR/../load_config.sh"
28
29	SCRIPTS_ROOT="$SCRIPT_DIR/.."
30
31	parse_args "$@"
32	detect_hosts
33
34	# ==============================================================================================
35	# ━━━ Helpers ━━━
36	# ==============================================================================================
37	JOB_PASS=()
38	JOB_FAIL=()
39
40	run_job() {
41	    local script_entry="$1"
42	    local extra_dry=""
43	    local extra_log=""
44	    [[ "$DRY_RUN"        == true ]] && extra_dry="--dry-run"
45	    [[ "$ENABLE_LOGGING" == true ]] && extra_log="--log"
46
47	    read -r -a script_args <<< "$script_entry"
48	    local script_path="$SCRIPTS_ROOT/${script_args[0]}"
49	    local script_name
50	    script_name=$(basename "${script_args[0]}")
51	    local extra_args=("${script_args[@]:1}")
52
53	    if [[ ! -f "$script_path" ]]; then
54	        error "$script_name — not found at $script_path"
55	        JOB_FAIL+=("$script_name")
56	        return 1
57	    fi
58
59	    log "Running: $script_name ${extra_args[*]}"
60	    if bash "$script_path" "${extra_args[@]}" $extra_dry $extra_log; then
61	        log "$script_name — done ✅"
62	        JOB_PASS+=("$script_name")
63	    else
64	        error "$script_name — failed (exit $?)"
65	        JOB_FAIL+=("$script_name")
66	    fi
67	}
68
69	# ==============================================================================================
70	# ━━━ Status ━━━
71	# ==============================================================================================
72	if [[ "$SHOW_STATUS" == true ]]; then
73	    echo "━━━ Sunday Morning Coffee Report — Status ━━━"
74	    echo "  Host:     $MY_ID"
75	    echo "  Scripts:"
76	    if [[ ${#COFFEE_REPORT_SCRIPTS[@]} -eq 0 ]]; then
77	        echo "    None configured (COFFEE_REPORT_SCRIPTS in master.conf)"
78	    else
79	        for entry in "${COFFEE_REPORT_SCRIPTS[@]}"; do
80	            [[ -n "$entry" ]] && echo "    ☕ ${entry##*/}"
81	        done
82	    fi
83	    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
84	    exit 0
85	fi
86
87	# ==============================================================================================
88	# ━━━ Main ━━━
89	# ==============================================================================================
90	_unraid_ver=$(platform_get_os_version 2>/dev/null || echo "unknown")
91	_uptime_s=$(awk '{print int($1)}' /proc/uptime 2>/dev/null || echo 0)
92	_container_count=$(docker ps -q 2>/dev/null | wc -l || echo 0)
93	_rootfs_pct=$(df / --output=pcent 2>/dev/null | tail -1 | tr -d ' %')
94
95	echo ""
96	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
97	echo "☕  Sunday Morning Coffee Report — $MY_ID"
98	[[ "$DRY_RUN" == true ]] && echo "    [DRY RUN]"
99	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
100	log "$ICON_HOST Server:     $MY_ID ($LOCAL_SERVER_NAME) — unRAID $_unraid_ver"
101	log "$ICON_TIME Uptime:     $(format_duration $_uptime_s)"
102	log "$ICON_CONTAINERS Docker: $_container_count container(s) running"
103	log "$ICON_HEALTH Rootfs:   ${_rootfs_pct:-?}% used"
104	log "$ICON_GEAR Scripts:    ${#COFFEE_REPORT_SCRIPTS[@]} configured"
105
106	if [[ ${#COFFEE_REPORT_SCRIPTS[@]} -eq 0 ]]; then
107	    warn "No scripts configured — add entries to COFFEE_REPORT_SCRIPTS in master.conf"
108	    exit 0
109	fi
110
111	for script_entry in "${COFFEE_REPORT_SCRIPTS[@]}"; do
112	    [[ -z "$script_entry" ]] && continue
113	    echo ""
114	    run_job "$script_entry"
115	done
116
117	# ==============================================================================================
118	# ━━━ Summary ━━━
119	# ==============================================================================================
120	echo ""


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Critical Sync Maintenance ======================================
4	# ==============================================================================================
5	# Orchestrator for time-sensitive syncs that run every 30 minutes.
6	# Keeps the mirror current between the less frequent daily and weekly windows.
7	# Schedule: */30 * * * * (every 30 minutes via User Scripts plugin)
8	#
9	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
10	#   1. Critical-Data rsync    — auth stack, NPM config, certs (containers stopped both sides)
11	#   2. emby-fallback rsync    — dirty Emby sync (watch states, library — Emby stays running)
12	#   3. CRITICAL_MAINTENANCE_SCRIPTS — any scripts configured for critical window
13	#   4. partnership --check    — read both state files, detect changes, act accordingly
14	#
15	# ── WHY EVERY 30 MINUTES ──────────────────────────────────────────────────────────────────────
16	#   Auth stack changes (new users, proxy rules, certs) propagate within 30min ✅
17	#   Emby watch states stay in sync — mirror users see correct playback position ✅
18	#   Partnership state changes detected and acted on quickly ✅
19	#   Lock prevents: daily rsync doing Critical-Data mid-critical window ✅
20	#
21	# ── RSYNC GATE ────────────────────────────────────────────────────────────────────────────────
22	#   RSYNC_ENABLED=false           → skips all syncs (global gate)
23	#   CRITICAL_RSYNC_ENABLED=false  → skips critical syncs only (per-orchestrator gate)
24	#   partnership --check always runs regardless — state check doesn't need rsync
25	#
26	# ── LOCK BEHAVIOUR ────────────────────────────────────────────────────────────────────────────
27	#   acquire_lock "strict" — if previous 30min run still going, skip this cycle entirely
28	#   Critical-Data taking > 30min is a problem worth knowing about
29	#   Strict mode prevents pile-up without waiting — log and move on ✅
30	#
31	# ── SILENT WHEN HEALTHY ───────────────────────────────────────────────────────────────────────
32	#   Runs 48 times per day — clean runs must produce zero output ✅
33	#   Only failures and notable events produce visible output
34	#
35	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
36	#   CRITICAL_RSYNC_ENABLED       — enable/disable rsync section
37	#   CRITICAL_SYNC_SHARES         — shares synced every 30min (HOST*_CRITICAL_SYNC_SHARES)
38	#   CRITICAL_MAINTENANCE_SCRIPTS — scripts run in critical window (optional)
39	#   PARTNERSHIP_ENABLED          — enable/disable partnership check
40	#
41	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
42	#   critical_sync_maintenance.sh              — normal run
43	#   critical_sync_maintenance.sh --dry-run    — preview syncs without transferring
44	#   critical_sync_maintenance.sh --log        — verbose per-share output
45	#   critical_sync_maintenance.sh --status     — show configuration and exit
46	# ==============================================================================================
47
48	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
49
50	source "$SCRIPT_DIR/../load_config.sh"
51
52	parse_args "$@"
53
54	# ==============================================================================================
55	# ━━━ Setup ━━━
56	# ==============================================================================================
57	if [[ "$EUID" -ne 0 ]]; then
58	    error "Must be run as root"
59	    exit 1
60	fi
61
62
63	acquire_lock "strict"
64
65	detect_hosts
66	resolve_remote_ip
67
68	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
69
70	# ==============================================================================================
71	# ━━━ Status ━━━
72	# ==============================================================================================
73	if [[ "$SHOW_STATUS" == true ]]; then
74	    echo ""
75	    echo "━━━━━ $ICON_SUMMARY CRITICAL SYNC STATUS ━━━━━"
76	    echo "$ICON_HOST    Identity:           $MY_ID ($LOCAL_SERVER_NAME)"
77	    echo "$ICON_NET     Remote:             $REMOTE_ID ($REMOTE_SERVER_NAME)"
78	    echo "$ICON_SYNC    Rsync enabled:      ${RSYNC_ENABLED:-true}"
79	    echo "$ICON_SYNC    Critical enabled:   ${CRITICAL_RSYNC_ENABLED:-false}"
80	    echo "$ICON_SHIELD  Partnership:        ${PARTNERSHIP_ENABLED:-false}"
81	    echo ""
82	    echo "━━━ Critical Sync Shares ━━━"
83	    if [[ ${#CRITICAL_SYNC_SHARES[@]} -eq 0 ]]; then
84	        warn "  No CRITICAL_SYNC_SHARES configured"
85	    else
86	        for share in "${CRITICAL_SYNC_SHARES[@]}"; do
87	            [[ -z "$share" ]] && continue
88	            SHARE_PATH="${share%%|*}"
89	            SHARE_PROFILE="${share##*|}"
90	            SHARE_NAME=$(basename "$SHARE_PATH")
91	            [[ "$SHARE_PATH" == "$SHARE_PROFILE" ]] && \
92	                echo "  $ICON_SYNC $SHARE_NAME — no profile" || \
93	                echo "  $ICON_SYNC $SHARE_NAME — profile: $SHARE_PROFILE"
94	        done
95	    fi
96	    echo ""
97	    echo "━━━ Critical Maintenance Scripts ━━━"
98	    if [[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
99	        echo "  None configured"
100	    else
101	        for entry in "${CRITICAL_MAINTENANCE_SCRIPTS[@]}"; do
102	            [[ -z "$entry" || "$entry" == \#* ]] && continue
103	            echo "  $ICON_GEAR $(basename "${entry%% *}")"
104	        done
105	    fi
106	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
107	    exit 0
108	fi
109
110	# ==============================================================================================
111	# ━━━ Critical Shares Sync ━━━
112	# ==============================================================================================
113	START=$(date +%s)
114	RSYNC_OK=false
115	PASS=()
116	FAIL=()
117
118	log "$ICON_SYNC Critical shares (${#CRITICAL_SYNC_SHARES[@]}): $(for s in "${CRITICAL_SYNC_SHARES[@]}"; do printf '%s ' "$(basename "${s%%|*}")"; done)"
119	[[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -gt 0 ]] && \
120	    log "$ICON_GEAR Maintenance scripts: $(for s in "${CRITICAL_MAINTENANCE_SCRIPTS[@]}"; do printf '%s ' "$(basename "${s%% *}")"; done)"


Now let me read the final two script headers:


1	#!/bin/bash
2	# ==============================================================================================
3	# =========================== Intermediate Sync Maintenance ====================================
4	# ==============================================================================================
5	# 4-hour orchestrator — arr library reconciliation, artwork fetching, and optional rsync.
6	# Schedule: 0 */4 * * * (every 4 hours)
7	#
8	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
9	#   1. conf_sync.sh --pull-only    — refresh partner conf cache in RAM (/tmp/.vv/)
10	#   2. arr_sync.sh                 — sync Lidarr/Sonarr/Radarr libraries across all nodes
11	#   3. Rsync window (optional)     — INTERMEDIATE_SYNC_SHARES, if any configured
12	#   4. INTERMEDIATE_MAINTENANCE_SCRIPTS — artwork fetch and any future 4-hour jobs
13	#
14	# ── WHY A SEPARATE ORCHESTRATOR ───────────────────────────────────────────────────────────────
15	# arr libraries need to converge more frequently than once a day. If a remote node adds
16	# something at 2am, the next daily window is 23 hours away — remote arrs search for content
17	# they don't know is already owned. Running every 4 hours closes that gap.
18	#
19	# lidarr_missing_art.sh is idempotent — skips existing files, runs fast after initial fill.
20	# Pairing it here means artwork catches up within 4 hours of a new album landing.
21	#
22	# Rsync is optional — INTERMEDIATE_SYNC_SHARES empty by default. Add shares to the config
23	# if a subset of data needs mid-day propagation (e.g. watch state, metadata). Full media
24	# share sync stays in the daily window.
25	#
26	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
27	# detect_hosts() sets MY_ID and aliases HOST*_INTERMEDIATE_SYNC_SHARES → INTERMEDIATE_SYNC_SHARES.
28	# Each server can have a different set of mid-day shares — configure in host*.conf.
29	# Each script in INTERMEDIATE_MAINTENANCE_SCRIPTS handles its own host logic.
30	#
31	# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
32	# Same as daily_sync_maintenance.sh:
33	#   exit 1 = temp WARN — skip this share, continue to next
34	#   exit 2 = temp CRITICAL — abort ALL remaining syncs in this window
35	#
36	# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
37	#   Root check           — scripts called here require root
38	#   acquire_lock         — prevents concurrent intermediate windows
39	#   check_connectivity   — verified before any rsync (skipped if no shares)
40	#   check_remote_rootfs  — aborts rsync if remote rootfs nearly full
41	#   Non-fatal jobs       — a failed arr_sync warns but does not block rsync or artwork fetch
42	#   Silent on success    — runs 4x/day, only failures warrant notification
43	#
44	# ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
45	#   host*.conf:   HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped)
46	#   master.conf:         INTERMEDIATE_RSYNC_ENABLED     — enable/disable rsync section (default: true)
47	#   master.conf:         INTERMEDIATE_MAINTENANCE_SCRIPTS — jobs run after rsync
48	#   master.conf:         ARR_SYNC_ENABLED               — toggle inside arr_sync.sh
49	#
50	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
51	#   intermediate_sync_maintenance.sh              — normal run
52	#   intermediate_sync_maintenance.sh --dry-run    — preview without changes
53	#   intermediate_sync_maintenance.sh --log        — verbose per-job output
54	#   intermediate_sync_maintenance.sh --status     — show configured shares/jobs and exit
55	# ==============================================================================================
56
57	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
58
59	source "$SCRIPT_DIR/../load_config.sh"
60
61	RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
62	SCRIPTS_ROOT="$SCRIPT_DIR/.."
63
64	parse_args "$@"
65
66	# ==============================================================================================
67	# ━━━ Setup ━━━
68	# ==============================================================================================
69	if [[ "$EUID" -ne 0 ]]; then
70	    error "Must be run as root"
71	    exit 1
72	fi
73
74
75	acquire_lock
76
77	detect_hosts
78	resolve_remote_ip
79
80	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
81
82	# ── Helper — run a maintenance job, track pass/fail ───────────────────────────────────────────
83	run_job() {
84	    local script_entry="$1"
85	    local extra_dry=""
86	    [[ "$DRY_RUN" == true ]] && extra_dry="--dry-run"
87
88	    read -r -a script_args <<< "$script_entry"
89	    local script_path="$SCRIPTS_ROOT/${script_args[0]}"
90	    local script_name
91	    script_name=$(basename "${script_args[0]}")
92	    local extra_args=("${script_args[@]:1}")
93
94	    if [[ ! -f "$script_path" ]]; then
95	        error "$script_name — not found at $script_path"
96	        JOB_FAIL+=("$script_name")
97	        return 1
98	    fi
99
100	    log "Running: $script_name ${extra_args[*]}"
101	    # shellcheck disable=SC2086
102	    if bash "$script_path" "${extra_args[@]}" $extra_dry; then
103	        log "$script_name — done ✅"
104	        JOB_PASS+=("$script_name ${extra_args[*]}")
105	    else
106	        warn "$script_name — failed (exit $?)"
107	        JOB_FAIL+=("$script_name ${extra_args[*]}")
108	    fi
109	}
110
111	# ==============================================================================================
112	# ━━━ Status ━━━
113	# ==============================================================================================
114	if [[ "$SHOW_STATUS" == true ]]; then
115	    echo ""
116	    echo "━━━━━ $ICON_SUMMARY INTERMEDIATE SYNC STATUS ━━━━━"
117	    echo "$ICON_HOST   Identity:        $MY_ID ($LOCAL_SERVER_NAME)"
118	    echo "$ICON_NET    Remote:          $REMOTE_ID ($REMOTE_SERVER_NAME)"
119	    echo "$ICON_SYNC   Rsync enabled:   ${RSYNC_ENABLED:-true}"
120	    echo "$ICON_SYNC   Interm. enabled: ${INTERMEDIATE_RSYNC_ENABLED:-true}"


1	#!/bin/bash
2	# ==============================================================================================
3	# ========================= Monthly Maintenance Orchestrator ===================================
4	# ==============================================================================================
5	# Uptime-triggered monthly maintenance — runs heavy tasks that need a stable, settled system.
6	# Fires only when BOTH gates pass:
7	#   1. Server uptime >= MONTHLY_UPTIME_THRESHOLD_DAYS days
8	#   2. Last run was >= MONTHLY_RUN_INTERVAL_DAYS days ago (or never run)
9	#
10	# ── WHY UPTIME-GATED ─────────────────────────────────────────────────────────────────────────
11	#   A scheduled reboot resets uptime. Monthly tasks (ZFS scrub, SMART long test) need a
12	#   stable, settled system — not one that just rebooted. Uptime-gating ensures maintenance
13	#   only runs after the server has been healthy for a full month, never immediately post-boot.
14	#   If uptime or interval gate is not met on the 15th, the run is skipped until next month.
15	#
16	# ── HOW TO CALL ──────────────────────────────────────────────────────────────────────────────
17	# Schedule: 0 0 15 * *  (15th of each month at midnight)
18	#   Silent exit 0 when either gate is not met. Only outputs when maintenance actually fires.
19	#
20	# ── STATE FILE ────────────────────────────────────────────────────────────────────────────────
21	#   MONTHLY_LAST_RUN_FILE — /boot/config — survives reboots, available before array starts.
22	#   Written after each run (pass or partial fail). Format: Unix timestamp.
23	#   A reboot does NOT reset the last-run state — the interval gate survives independently
24	#   of the uptime gate. Both must pass before maintenance fires again.
25	#
26	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
27	#   MONTHLY_MAINTENANCE_SCRIPTS    — ordered list of scripts to run
28	#   MONTHLY_UPTIME_THRESHOLD_DAYS  — minimum uptime in days before maintenance fires
29	#   MONTHLY_RUN_INTERVAL_DAYS      — minimum days since last run before running again
30	#   MONTHLY_LAST_RUN_FILE          — state file path — /boot/config, survives reboots
31	#
32	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
33	#   monthly_maintenance.sh           — normal run (uptime + interval gates enforced)
34	#   monthly_maintenance.sh --dry-run — preview gate state and scripts without running
35	#   monthly_maintenance.sh --status  — show gate state, last run, and configured scripts
36	#   monthly_maintenance.sh --force   — bypass uptime + interval gates (manual override)
37	#   monthly_maintenance.sh --log     — verbose output
38	# ==============================================================================================
39
40	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
41	ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
42
43	source "$ECOSYSTEM_ROOT/load_config.sh"
44
45	# ── Parse --force before common parse_args ────────────────────────────────────────────────────
46	FORCE_RUN=false
47	FILTERED_ARGS=()
48	for _arg in "$@"; do
49	    if [[ "$_arg" == "--force" ]]; then
50	        FORCE_RUN=true
51	    else
52	        FILTERED_ARGS+=("$_arg")
53	    fi
54	done
55	parse_args "${FILTERED_ARGS[@]}"
56
57	# ==============================================================================================
58	# ━━━ Setup ━━━
59	# ==============================================================================================
60	if [[ "$EUID" -ne 0 ]]; then
61	    error "Must be run as root"
62	    exit 1
63	fi
64
65	acquire_lock
66
67	detect_hosts
68
69	[[ "$DRY_RUN" == true ]]   && warn "DRY RUN — no scripts will be executed"
70	[[ "$FORCE_RUN" == true ]] && warn "FORCE — uptime and interval gates bypassed"
71
72	# Defaults — overridden by master.conf values
73	MONTHLY_UPTIME_THRESHOLD_DAYS="${MONTHLY_UPTIME_THRESHOLD_DAYS:-30}"
74	MONTHLY_RUN_INTERVAL_DAYS="${MONTHLY_RUN_INTERVAL_DAYS:-30}"
75	MONTHLY_LAST_RUN_FILE="${MONTHLY_LAST_RUN_FILE:-${STATE_DIR:-/tmp}/monthly_maintenance_last_run.db}"
76
77	UPTIME_THRESHOLD_SECS=$(( MONTHLY_UPTIME_THRESHOLD_DAYS * 86400 ))
78	INTERVAL_SECS=$(( MONTHLY_RUN_INTERVAL_DAYS * 86400 ))
79	UPTIME_SECS=$(awk '{print int($1)}' /proc/uptime)
80	NOW=$(date +%s)
81
82	# ==============================================================================================
83	# ━━━ Gate Evaluation ━━━
84	# ==============================================================================================
85	_uptime_days=$(( UPTIME_SECS / 86400 ))
86	_uptime_hrs=$(( (UPTIME_SECS % 86400) / 3600 ))
87
88	UPTIME_GATE_PASS=false
89	if [[ "$UPTIME_SECS" -ge "$UPTIME_THRESHOLD_SECS" ]]; then
90	    UPTIME_GATE_PASS=true
91	fi
92
93	INTERVAL_GATE_PASS=false
94	LAST_RUN=0
95	DAYS_SINCE_LAST_RUN="never"
96	if [[ -f "$MONTHLY_LAST_RUN_FILE" ]]; then
97	    LAST_RUN=$(cat "$MONTHLY_LAST_RUN_FILE" 2>/dev/null || echo 0)
98	    ELAPSED=$(( NOW - LAST_RUN ))
99	    DAYS_SINCE=$(( ELAPSED / 86400 ))
100	    DAYS_SINCE_LAST_RUN="${DAYS_SINCE}d"
101	    if [[ "$ELAPSED" -ge "$INTERVAL_SECS" ]]; then
102	        INTERVAL_GATE_PASS=true
103	    fi
104	else
105	    INTERVAL_GATE_PASS=true  # never run
106	fi
107
108	# ==============================================================================================
109	# ━━━ Status ━━━
110	# ==============================================================================================
111	if [[ "$SHOW_STATUS" == true ]]; then
112	    echo ""
113	    echo "━━━━━ $ICON_SUMMARY MONTHLY MAINTENANCE STATUS ━━━━━"
114	    echo "$ICON_HOST  Identity: $MY_ID ($LOCAL_SERVER_NAME)"
115	    echo ""
116	    echo "── Gates ──"
117
118	    if [[ "$UPTIME_GATE_PASS" == true ]]; then
119	        echo "  $ICON_DONE Uptime: ${_uptime_days}d ${_uptime_hrs}h (threshold: ${MONTHLY_UPTIME_THRESHOLD_DAYS}d) ✅"
120	    else


Now let me check if arrs_failed_stalled_recovery.sh exists and read its header:


Bash

find /boot/config/plugins/varaverk -name "arrs_failed_stalled_recovery.sh" -type f 2>/dev/null


The script is in Media, not System_Essentials. Let me read it:


1	#!/bin/bash
2	# ==============================================================================================
3	# ========================= Arrs Failed / Stalled Recovery =====================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Detect and recover failed imports and stalled downloads across Sonarr, Radarr,
9	# and Lidarr. Blocklists the bad release and triggers a re-search — hands-free
10	# overnight recovery.
11	#
12	# ==============================================================================================
13	# OPERATIONAL MODEL
14	# ==============================================================================================
15	#
16	# Four problem types detected from the arr queue API:
17	#   importFailed   — downloaded but arr couldn't import the file
18	#   importPending  — downloaded, stuck waiting to import (will not self-resolve)
19	#   error status   — serious failure not covered by the above two states
20	#   stalled        — download stuck with no connections or no progress
21	#
22	# Never touches items with state "downloading" or "imported" — safe to run anytime.
23	# Items newer than ARR_IMPORT_RECOVERY_AGE are skipped — gives arr time to retry first.
24	#
25	# Per problem item (3-step response):
26	#   1. Blocklist the release  — prevents re-grabbing the same bad release
27	#   2. Remove from queue      — cleans up the failed item
28	#   3. Trigger new search     — finds a different release automatically
29	#
30	# ==============================================================================================
31	# OPERATIONAL SAFEGUARDS
32	# ==============================================================================================
33	#
34	#   acquire_lock      — prevents concurrent runs overlapping
35	#   jq validation     — exits if jq not installed (required for JSON parsing)
36	#   API pre-flight    — checks each arr is reachable before querying queue
37	#   Version check     — check_arr_version() verifies running arr matches master.conf major
38	#                       version; exits rather than silently misoperating after upgrade
39	#   Age threshold     — skips items newer than ARR_IMPORT_RECOVERY_AGE (default 6hr)
40	#   Silent by default — only problems produce output, clean arrs stay silent
41	#
42	# API version mapping (endpoint paths differ from major version labels):
43	#   Sonarr  v4 → /api/v3/ (v3 endpoint retained in v4)
44	#   Radarr  v6 → /api/v3/ (v3 endpoint retained in v6)
45	#   Lidarr  v3 → /api/v1/ (different from Sonarr/Radarr)
46	#
47	# ==============================================================================================
48	# STATE FILES
49	# ==============================================================================================
50	#
51	#   ARR_RECOVERY_STATS — stats file written after each run (read by coffee report)
52	#
53	# ==============================================================================================
54	# CONFIGURATION
55	# ==============================================================================================
56	#
57	# host*.conf
58	#
59	#   HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_RECOVERY
60	#   HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_RECOVERY
61	#   HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY / HOST1_LIDARR_RECOVERY
62	#   All aliased by detect_hosts() — script uses unprefixed names
63	#
64	# master.conf
65	#
66	#   ARR_IMPORT_RECOVERY_AGE — hours before item is eligible for recovery (default: 6)
67	#   SONARR_VERSION_MAJOR    — expected Sonarr major version (e.g. 4)
68	#   RADARR_VERSION_MAJOR    — expected Radarr major version (e.g. 6)
69	#   LIDARR_VERSION_MAJOR    — expected Lidarr major version (e.g. 3)
70	#   ARR_RECOVERY_STATS      — stats file path (read by coffee report)
71	#
72	# ==============================================================================================
73	# RUNTIME MODES
74	# ==============================================================================================
75	#
76	#   arrs_failed_stalled_recovery.sh              — normal run
77	#   arrs_failed_stalled_recovery.sh --dry-run    — show what would be actioned, no changes
78	#   arrs_failed_stalled_recovery.sh --log        — verbose output
79	#   arrs_failed_stalled_recovery.sh --status     — show config and exit
80	#
81	# Recommended schedule: 0 5 * * * (5am daily)
82	# Or every 6hr: 0 */6 * * * (matches ARR_IMPORT_RECOVERY_AGE default)
83	#
84	# ==============================================================================================
85
86	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
87
88	source "$SCRIPT_DIR/../load_config.sh"
89
90	parse_args "$@"
91
92	# ==============================================================================================
93	# ━━━ Setup ━━━
94	# ==============================================================================================
95	if [[ "$EUID" -ne 0 ]]; then
96	    error "Must be run as root"
97	    exit 1
98	fi
99
100	acquire_lock
101
102	# detect_hosts() sets MY_ID and aliases SONARR_*, RADARR_*, LIDARR_* vars
103	detect_hosts
104
105	# jq is required — not optional — for JSON parsing
106	if ! command -v jq >/dev/null 2>&1; then
107	    error "jq is not installed — required for arr API JSON parsing"
108	    error "Install: apt-get install jq  or  brew install jq"
109	    notify "arrs_failed_stalled_recovery failed on $(hostname) — jq not installed" \
110	        "Arr Recovery" "warning"
111	    exit 1
112	fi
113	log "jq found"
114
115
116	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no items will be blocklisted or searched"
117
118	# Age threshold in seconds
119	AGE_THRESHOLD_SECONDS=$(( ARR_IMPORT_RECOVERY_AGE * 3600 ))
120


Perfect! Now I have all the information I need. Let me analyze the discrepancies:

## REPORT: Documentation vs Script Header Discrepancies

Based on a thorough review of the README, Manual, and script headers, I found the following discrepancies:

**CRITICAL LOCATION MISMATCH:**

1. **README-Orchestrators.md (line 175):** Documents `arrs_failed_stalled_recovery.sh` as running "Every 6 hours" under the Orchestrators section. However, the actual script header shows "Recommended schedule: 0 5 * * * (5am daily) Or every 6hr: 0 */6 * * *" - it's optional per-schedule, and more importantly, **the script lives in `/boot/config/plugins/varaverk/Media/` not Orchestrators/**. The README tables it as an orchestrator when it's actually a standalone script typically called by an orchestrator.

**SCHEDULE DISCREPANCIES:**

2. **README-Orchestrators.md (line 109):** States intermediate_sync_maintenance.sh runs "0 */4 * * *" (every 4 hours). However, the README header at line 6 says "Schedule: 0 */4 * * * (every 4 hours)" — this matches and is correct.

3. **README-Orchestrators.md (lines 105-112):** Lists "arrs_failed_stalled_recovery.sh" in the cron schedule section as "Every 6 hours" but this script is NOT an orchestrator and should NOT be in this list. It's orchestrated by being called FROM orchestrators as needed, not scheduled directly.

**CONTENT ACCURACY ISSUES:**

4. **README-Orchestrators.md (line 178):** States arrs_failed_stalled_recovery.sh "runs every 6 hours" in the table. The actual script header (Media/arrs_failed_stalled_recovery.sh line 81-82) shows TWO possible schedules: "5am daily" OR "every 6hr" — it's flexible and depends on configuration, not fixed at 6 hours.

5. **README-Orchestrators.md (line 109-112):** The "COMPLETE SCHEDULE" section includes arrs_failed_stalled_recovery.sh as "0 */6 * * *" but this contradicts the fact that it's shown as callable from orchestrators (line 175 references it under the arrs_failed_stalled_recovery section in its own dedicated area, not as part of the main orchestrator list).

**SUMMARY OF ISSUES:**
- arrs_failed_stalled_recovery.sh is documented as an orchestrator when it's actually a Media script
- Schedule flexibility (5am vs every 6hr) is documented as fixed "every 6 hours"
- Script location mismatch between documentation layout and actual filesystem

All other scripts match their documentation correctly regarding schedules, execution order, and purposes.