1	#!/bin/bash
2	# ==============================================================================================
3	# ========================= Monthly Maintenance Orchestrator ===================================
4	# ==============================================================================================
5	# Uptime-triggered monthly maintenance — runs heavy tasks that need a stable, settled system.
6	# Fires only when BOTH gates pass:
7	#   1. Server uptime >= MONTHLY_UPTIME_THRESHOLD_DAYS days
8	#   2. Last run was >= MONTHLY_RUN_INTERVAL_DAYS days ago (or never run)
9	#
10	# ── WHY UPTIME-GATED ─────────────────────────────────────────────────────────────────────────
11	#   A scheduled reboot resets uptime. Monthly tasks (ZFS scrub, SMART long test) need a
12	#   stable, settled system — not one that just rebooted. Uptime-gating ensures maintenance
13	#   only runs after the server has been healthy for a full month, never immediately post-boot.
14	#   If uptime or interval gate is not met on the 15th, the run is skipped until next month.
15	#
16	# ── HOW TO CALL ──────────────────────────────────────────────────────────────────────────────
17	# Schedule: 0 0 15 * *  (15th of each month at midnight)
18	#   Silent exit 0 when either gate is not met. Only outputs when maintenance actually fires.
19	#
20	# ── STATE FILE ────────────────────────────────────────────────────────────────────────────────
21	#   MONTHLY_LAST_RUN_FILE — /boot/config — survives reboots, available before array starts.
22	#   Written after each run (pass or partial fail). Format: Unix timestamp.
23	#   A reboot does NOT reset the last-run state — the interval gate survives independently
24	#   of the uptime gate. Both must pass before maintenance fires again.
25	#
26	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
27	#   MONTHLY_MAINTENANCE_SCRIPTS    — ordered list of scripts to run
28	#   MONTHLY_UPTIME_THRESHOLD_DAYS  — minimum uptime in days before maintenance fires
29	#   MONTHLY_RUN_INTERVAL_DAYS      — minimum days since last run before running again
30	#   MONTHLY_LAST_RUN_FILE          — state file path — /boot/config, survives reboots
31	#
32	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
33	#   monthly_maintenance.sh           — normal run (uptime + interval gates enforced)
34	#   monthly_maintenance.sh --dry-run — preview gate state and scripts without running
35	#   monthly_maintenance.sh --status  — show gate state, last run, and configured scripts
36	#   monthly_maintenance.sh --force   — bypass uptime + interval gates (manual override)
37	#   monthly_maintenance.sh --log     — verbose output
38	# ==============================================================================================
39
40	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
41	ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
42
43	source "$ECOSYSTEM_ROOT/load_config.sh"
44
45	# ── Parse --force before common parse_args ────────────────────────────────────────────────────
46	FORCE_RUN=false
47	FILTERED_ARGS=()
48	for _arg in "$@"; do
49	    if [[ "$_arg" == "--force" ]]; then
50	        FORCE_RUN=true
51	    else
52	        FILTERED_ARGS+=("$_arg")
53	    fi
54	done
55	parse_args "${FILTERED_ARGS[@]}"
56
57	# ==============================================================================================
58	# ━━━ Setup ━━━
59	# ==============================================================================================
60	if [[ "$EUID" -ne 0 ]]; then
61	    error "Must be run as root"
62	    exit 1
63	fi
64
65	acquire_lock
66
67	detect_hosts
68
69	[[ "$DRY_RUN" == true ]]   && warn "DRY RUN — no scripts will be executed"
70	[[ "$FORCE_RUN" == true ]] && warn "FORCE — uptime and interval gates bypassed"
71
72	# Defaults — overridden by master.conf values
73	MONTHLY_UPTIME_THRESHOLD_DAYS="${MONTHLY_UPTIME_THRESHOLD_DAYS:-30}"
74	MONTHLY_RUN_INTERVAL_DAYS="${MONTHLY_RUN_INTERVAL_DAYS:-30}"
75	MONTHLY_LAST_RUN_FILE="${MONTHLY_LAST_RUN_FILE:-${STATE_DIR:-/tmp}/monthly_maintenance_last_run.db}"
76
77	UPTIME_THRESHOLD_SECS=$(( MONTHLY_UPTIME_THRESHOLD_DAYS * 86400 ))
78	INTERVAL_SECS=$(( MONTHLY_RUN_INTERVAL_DAYS * 86400 ))
79	UPTIME_SECS=$(awk '{print int($1)}' /proc/uptime)
80	NOW=$(date +%s)
81
82	# ==============================================================================================
83	# ━━━ Gate Evaluation ━━━
84	# ==============================================================================================
85	_uptime_days=$(( UPTIME_SECS / 86400 ))
86	_uptime_hrs=$(( (UPTIME_SECS % 86400) / 3600 ))
87
88	UPTIME_GATE_PASS=false
89	if [[ "$UPTIME_SECS" -ge "$UPTIME_THRESHOLD_SECS" ]]; then
90	    UPTIME_GATE_PASS=true
91	fi
92
93	INTERVAL_GATE_PASS=false
94	LAST_RUN=0
95	DAYS_SINCE_LAST_RUN="never"
96	if [[ -f "$MONTHLY_LAST_RUN_FILE" ]]; then
97	    LAST_RUN=$(cat "$MONTHLY_LAST_RUN_FILE" 2>/dev/null || echo 0)
98	    ELAPSED=$(( NOW - LAST_RUN ))
99	    DAYS_SINCE=$(( ELAPSED / 86400 ))
100	    DAYS_SINCE_LAST_RUN="${DAYS_SINCE}d"
101	    if [[ "$ELAPSED" -ge "$INTERVAL_SECS" ]]; then
102	        INTERVAL_GATE_PASS=true
103	    fi
104	else
105	    INTERVAL_GATE_PASS=true  # never run
106	fi
107
108	# ==============================================================================================
109	# ━━━ Status ━━━
110	# ==============================================================================================
111	if [[ "$SHOW_STATUS" == true ]]; then
112	    echo ""
113	    echo "━━━━━ $ICON_SUMMARY MONTHLY MAINTENANCE STATUS ━━━━━"
114	    echo "$ICON_HOST  Identity: $MY_ID ($LOCAL_SERVER_NAME)"
115	    echo ""
116	    echo "── Gates ──"
117
118	    if [[ "$UPTIME_GATE_PASS" == true ]]; then
119	        echo "  $ICON_DONE Uptime: ${_uptime_days}d ${_uptime_hrs}h (threshold: ${MONTHLY_UPTIME_THRESHOLD_DAYS}d) ✅"
120	    else
121	        echo "  $ICON_WARN Uptime: ${_uptime_days}d ${_uptime_hrs}h / ${MONTHLY_UPTIME_THRESHOLD_DAYS}d needed — NOT met"
122	    fi
123
124	    if [[ "$INTERVAL_GATE_PASS" == true ]]; then
125	        echo "  $ICON_DONE Interval: last run ${DAYS_SINCE_LAST_RUN} ago (threshold: ${MONTHLY_RUN_INTERVAL_DAYS}d) ✅"
126	    else
127	        echo "  $ICON_WARN Interval: last run ${DAYS_SINCE_LAST_RUN} ago / ${MONTHLY_RUN_INTERVAL_DAYS}d needed — NOT met"
128	    fi
129
130	    if [[ "$LAST_RUN" -gt 0 ]]; then
131	        echo "  Last run: $(date -d "@$LAST_RUN" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || date -r "$LAST_RUN" '+%Y-%m-%d %H:%M:%S')"
132	    else
133	        echo "  Last run: never"
134	    fi
135
136	    echo ""
137	    echo "── Scripts ──"
138	    if [[ "${#MONTHLY_MAINTENANCE_SCRIPTS[@]}" -eq 0 ]]; then
139	        echo "  (none configured — add to MONTHLY_MAINTENANCE_SCRIPTS in master.conf)"
140	    else
141	        for entry in "${MONTHLY_MAINTENANCE_SCRIPTS[@]}"; do
142	            [[ -z "$entry" ]] && continue
143	            read -r -a parts <<< "$entry"
144	            script_path="$ECOSYSTEM_ROOT/${parts[0]}"
145	            script_name=$(basename "${parts[0]}")
146	            if [[ -f "$script_path" ]]; then
147	                [[ -x "$script_path" ]] && echo "  $ICON_DONE $script_name" || echo "  $ICON_WARN $script_name (not executable)"
148	            else
149	                echo "  $ICON_ERROR $script_name — NOT FOUND: $script_path"
150	            fi
151	        done
152	    fi
153
154	    echo ""
155	    echo "  Schedule:   0 0 15 * * (15th of each month at midnight)"
156	    echo "  Force flag: --force bypasses both gates"
157	    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
158	    exit 0
159	fi
160
161	# ==============================================================================================
162	# ━━━ Gate Check — Silent Exit When Not Due ━━━
163	# ==============================================================================================
164	if [[ "$FORCE_RUN" != true ]]; then
165	    if [[ "$UPTIME_GATE_PASS" != true ]]; then
166	        echo "Uptime gate not met — ${_uptime_days}d ${_uptime_hrs}h / ${MONTHLY_UPTIME_THRESHOLD_DAYS}d — no-op"
167	        exit 0
168	    fi
169	    if [[ "$INTERVAL_GATE_PASS" != true ]]; then
170	        echo "Interval gate not met — last run ${DAYS_SINCE_LAST_RUN} ago / ${MONTHLY_RUN_INTERVAL_DAYS}d — no-op"
171	        exit 0
172	    fi
173	fi
174
175	# ==============================================================================================
176	# ━━━ Pre-flight ━━━
177	# ==============================================================================================
178	if [[ "${#MONTHLY_MAINTENANCE_SCRIPTS[@]}" -eq 0 ]]; then
179	    warn "No scripts in MONTHLY_MAINTENANCE_SCRIPTS — nothing to run"
180	    exit 0
181	fi
182
183	echo ""
184	echo "━━━ $ICON_START Monthly Maintenance — $MY_ID — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
185	echo "$ICON_HOST  Uptime: ${_uptime_days}d ${_uptime_hrs}h | Last run: ${DAYS_SINCE_LAST_RUN} ago${FORCE_RUN:+ | FORCED}"
186	echo "$ICON_GEAR  Scripts: ${#MONTHLY_MAINTENANCE_SCRIPTS[@]}"
187	echo ""
188
189	START=$(date +%s)
190	PASSED=()
191	FAILED=()
192	STEP=0
193
194	# ==============================================================================================
195	# ━━━ Run Sequence ━━━
196	# ==============================================================================================
197	for entry in "${MONTHLY_MAINTENANCE_SCRIPTS[@]}"; do
198	    [[ -z "$entry" ]] && continue
199	    (( STEP++ ))
200
201	    read -r -a parts <<< "$entry"
202	    script_path="$ECOSYSTEM_ROOT/${parts[0]}"
203	    script_name=$(basename "${parts[0]}")
204	    extra_args=("${parts[@]:1}")
205
206	    echo "━━━ $ICON_GEAR Step $STEP: $script_name${extra_args:+ ${extra_args[*]}} ━━━"
207
208	    if [[ ! -f "$script_path" ]]; then
209	        error "$script_name — not found at $script_path"
210	        FAILED+=("$script_name")
211	        echo ""
212	        continue
213	    fi
214
215	    if [[ ! -x "$script_path" ]]; then
216	        warn "$script_name — not executable, fixing..."
217	        chmod +x "$script_path" || {
218	            error "$script_name — chmod +x failed"
219	            FAILED+=("$script_name")
220	            echo ""
221	            continue
222	        }
223	    fi
224
225	    if [[ "$DRY_RUN" == true ]]; then
226	        warn "DRY RUN — would run: $script_name${extra_args:+ ${extra_args[*]}}"
227	        PASSED+=("$script_name")
228	        echo ""
229	        continue
230	    fi
231
232	    local_args=()
233	    [[ "$VERBOSE" == true ]] && local_args+=("--log")
234
235	    if bash "$script_path" "${extra_args[@]}" "${local_args[@]}"; then
236	        log "$script_name — done ✅"
237	        PASSED+=("$script_name")
238	    else
239	        warn "$script_name — failed (exit $?) — continuing to next step"
240	        FAILED+=("$script_name")
241	    fi
242
243	    echo ""
244	done
245
246	END=$(date +%s)
247
248	# ==============================================================================================
249	# ━━━ Record Last Run ━━━
250	# ==============================================================================================
251	# Write timestamp whether we passed or partially failed — prevents hammering broken scripts.
252	if [[ "$DRY_RUN" != true ]]; then
253	    echo "$NOW" > "$MONTHLY_LAST_RUN_FILE"
254	    log "Last run recorded: $(date -d "@$NOW" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || date -r "$NOW" '+%Y-%m-%d %H:%M:%S')"
255	fi
256
257	# ==============================================================================================
258	# ━━━ Summary ━━━
259	# ==============================================================================================
260	echo "━━━━━ $ICON_SUMMARY MONTHLY MAINTENANCE SUMMARY ━━━━━"
261	echo "$ICON_HOST    Identity: $MY_ID ($LOCAL_SERVER_NAME)"
262	echo "$ICON_TIME    Duration: $(format_duration $(( END - START )))"
263	[[ ${#PASSED[@]} -gt 0 ]] && echo "$ICON_DONE    Passed:   ${PASSED[*]}"
264	[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR   Failed:   ${FAILED[*]}"
265	echo ""
266
267	if [[ "$DRY_RUN" == true ]]; then
268	    warn "DRY RUN — no changes made"
269	elif [[ ${#FAILED[@]} -eq 0 ]]; then
270	    echo "$ICON_DONE Status: all $STEP step(s) complete ✅"
271	    notify "Monthly maintenance complete on $(hostname) ($MY_ID) — $STEP step(s) done" \
272	        "Monthly Maintenance" "normal"
273	else
274	    warn "Status: ${#FAILED[@]} step(s) failed — ${FAILED[*]}"
275	    notify "Monthly maintenance on $(hostname) ($MY_ID) — ${#FAILED[@]} step(s) failed: ${FAILED[*]}" \
276	        "Monthly Maintenance" "warning"
277	fi
278	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
279
280	[[ ${#FAILED[@]} -gt 0 ]] && exit 1
281	exit 0
282


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Sunday Morning Coffee Report ===================================
4	# ==============================================================================================
5	# Weekly monitoring orchestrator — runs all Sunday monitor scripts in sequence.
6	# Designed to be read over coffee Sunday morning while the system is fully caught up
7	# from the 2:30am maintenance window.
8	# Schedule: 0 7 * * 0 (7am Sunday — after weekly_sync_maintenance.sh finishes at ~3am)
9	#
10	# ── SCRIPTS (master.conf COFFEE_REPORT_SCRIPTS) ───────────────────────────────────────────────
11	#   Each script runs independently, logs to its own output, and notifies on findings.
12	#   All scripts receive --dry-run and --log flags from this orchestrator when set.
13	#
14	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
15	#   detect_hosts() sets MY_ID — used in banner and summary.
16	#   HOST1 primary: runs all scripts. HOST2: limited to host-aware scripts only.
17	#
18	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
19	#   sunday_morning_coffee_report.sh              — normal run
20	#   sunday_morning_coffee_report.sh --dry-run    — preview without any writes or notifications
21	#   sunday_morning_coffee_report.sh --log        — verbose per-script output
22	#   sunday_morning_coffee_report.sh --status     — show configured scripts and exit
23	# ==============================================================================================
24
25	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
26
27	source "$SCRIPT_DIR/../load_config.sh"
28
29	SCRIPTS_ROOT="$SCRIPT_DIR/.."
30
31	parse_args "$@"
32	detect_hosts
33
34	# ==============================================================================================
35	# ━━━ Helpers ━━━
36	# ==============================================================================================
37	JOB_PASS=()
38	JOB_FAIL=()
39
40	run_job() {
41	    local script_entry="$1"
42	    local extra_dry=""
43	    local extra_log=""
44	    [[ "$DRY_RUN"        == true ]] && extra_dry="--dry-run"
45	    [[ "$ENABLE_LOGGING" == true ]] && extra_log="--log"
46
47	    read -r -a script_args <<< "$script_entry"
48	    local script_path="$SCRIPTS_ROOT/${script_args[0]}"
49	    local script_name
50	    script_name=$(basename "${script_args[0]}")
51	    local extra_args=("${script_args[@]:1}")
52
53	    if [[ ! -f "$script_path" ]]; then
54	        error "$script_name — not found at $script_path"
55	        JOB_FAIL+=("$script_name")
56	        return 1
57	    fi
58
59	    log "Running: $script_name ${extra_args[*]}"
60	    if bash "$script_path" "${extra_args[@]}" $extra_dry $extra_log; then
61	        log "$script_name — done ✅"
62	        JOB_PASS+=("$script_name")
63	    else
64	        error "$script_name — failed (exit $?)"
65	        JOB_FAIL+=("$script_name")
66	    fi
67	}
68
69	# ==============================================================================================
70	# ━━━ Status ━━━
71	# ==============================================================================================
72	if [[ "$SHOW_STATUS" == true ]]; then
73	    echo "━━━ Sunday Morning Coffee Report — Status ━━━"
74	    echo "  Host:     $MY_ID"
75	    echo "  Scripts:"
76	    if [[ ${#COFFEE_REPORT_SCRIPTS[@]} -eq 0 ]]; then
77	        echo "    None configured (COFFEE_REPORT_SCRIPTS in master.conf)"
78	    else
79	        for entry in "${COFFEE_REPORT_SCRIPTS[@]}"; do
80	            [[ -n "$entry" ]] && echo "    ☕ ${entry##*/}"
81	        done
82	    fi
83	    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
84	    exit 0
85	fi
86
87	# ==============================================================================================
88	# ━━━ Main ━━━
89	# ==============================================================================================
90	_unraid_ver=$(platform_get_os_version 2>/dev/null || echo "unknown")
91	_uptime_s=$(awk '{print int($1)}' /proc/uptime 2>/dev/null || echo 0)
92	_container_count=$(docker ps -q 2>/dev/null | wc -l || echo 0)
93	_rootfs_pct=$(df / --output=pcent 2>/dev/null | tail -1 | tr -d ' %')
94
95	echo ""
96	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
97	echo "☕  Sunday Morning Coffee Report — $MY_ID"
98	[[ "$DRY_RUN" == true ]] && echo "    [DRY RUN]"
99	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
100	log "$ICON_HOST Server:     $MY_ID ($LOCAL_SERVER_NAME) — unRAID $_unraid_ver"
101	log "$ICON_TIME Uptime:     $(format_duration $_uptime_s)"
102	log "$ICON_CONTAINERS Docker: $_container_count container(s) running"
103	log "$ICON_HEALTH Rootfs:   ${_rootfs_pct:-?}% used"
104	log "$ICON_GEAR Scripts:    ${#COFFEE_REPORT_SCRIPTS[@]} configured"
105
106	if [[ ${#COFFEE_REPORT_SCRIPTS[@]} -eq 0 ]]; then
107	    warn "No scripts configured — add entries to COFFEE_REPORT_SCRIPTS in master.conf"
108	    exit 0
109	fi
110
111	for script_entry in "${COFFEE_REPORT_SCRIPTS[@]}"; do
112	    [[ -z "$script_entry" ]] && continue
113	    echo ""
114	    run_job "$script_entry"
115	done
116
117	# ==============================================================================================
118	# ━━━ Summary ━━━
119	# ==============================================================================================
120	echo ""
121	echo "━━━━━ ☕ COFFEE REPORT SUMMARY ━━━━━"
122	echo "🖥  Host:    $MY_ID"
123	echo "⏱  Done:    $(date '+%Y-%m-%d %H:%M:%S')"
124
125	if [[ ${#JOB_PASS[@]} -gt 0 ]]; then
126	    echo "✅  Passed:  ${JOB_PASS[*]}"
127	fi
128	if [[ ${#JOB_FAIL[@]} -gt 0 ]]; then
129	    echo "❌  Failed:  ${JOB_FAIL[*]}"
130	fi
131
132	[[ ${#JOB_FAIL[@]} -gt 0 ]] && exit 1
133	exit 0
134


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Transcode Management ===========================================
4	# ==============================================================================================
5	# Orchestrator — runs transcode_cleanup.sh then transcode_manager.sh in the correct order.
6	# Replace individual transcode_manager and transcode_cleanup cron entries with this.
7	# Schedule: */7 * * * * (every 7 minutes via User Scripts plugin)
8	#
9	# ── WHY CLEANUP BEFORE MANAGER ────────────────────────────────────────────────────────────────
10	# Cleanup runs first — removes stale segment files from ended sessions.
11	# Manager runs after — threshold decisions based on real current usage post-cleanup.
12	# Without this order, stale files inflate the ramdisk usage reading and trigger
13	# unnecessary SSD flips even when active sessions would fit on the ramdisk.
14	#
15	# ── WHAT EACH SCRIPT DOES ─────────────────────────────────────────────────────────────────────
16	#   transcode_cleanup.sh  — removes aged segment files not open by any process
17	#                           uses lsof for O(1) per-file active check (never per-file lsof)
18	#                           also triggers flip-back to ramdisk after cleanup if recovered ✅
19	#
20	#   transcode_manager.sh  — checks ramdisk usage against thresholds
21	#                           flips symlink between ramdisk and SSD as needed
22	#                           writes one entry to TRANSCODE_DAILY_LOG after each run
23	#                           shows active Emby sessions with play method
24	#
25	# ── DAILY LOG ─────────────────────────────────────────────────────────────────────────────────
26	# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each run:
27	#   Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
28	# This orchestrator does NOT write its own log — manager handles it ✅
29	# Log trimmed to TRANSCODE_LOG_RETENTION days by manager on each write.
30	# Read by sunday_morning_coffee_report.sh and weekly_health_digest.sh.
31	#
32	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
33	# detect_hosts() sets MY_ID and aliases RAMDISK_PATH, TRANSCODE_SSD, RAMDISK_WARN_GB etc.
34	# Each server manages its own transcode location independently.
35	#
36	# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
37	#   Root check      — mount and docker operations require root
38	#   acquire_lock    — prevents concurrent 3-minute cycles overlapping
39	#   detect_hosts()  — correct paths per host
40	#   --dry-run       — passed through to both child scripts
41	#   Exit code       — worst exit code of both scripts returned
42	#
43	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
44	#   TRANSCODE_DAILY_LOG    — daily stats log (written by transcode_manager.sh)
45	#   TRANSCODE_LOG_RETENTION — days to keep (trimmed by manager)
46	#   TRANSCODE_STATE_FILE   — current state (ramdisk_used, flip_count etc.)
47	#   All TRANSCODE_* threshold vars — see master.conf Transcode Manager section
48	#
49	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
50	#   transcode_management.sh              — normal run (every 7 minutes via cron)
51	#   transcode_management.sh --dry-run    — preview without changes (passed to children)
52	#   transcode_management.sh --status     — show configuration and current state
53	#   transcode_management.sh --log        — verbose output from both child scripts
54	# ==============================================================================================
55
56	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
57
58	source "$SCRIPT_DIR/../load_config.sh"
59
60	parse_args "$@"
61
62	CLEANUP_SCRIPT="$SCRIPT_DIR/../Transcodes/transcode_cleanup.sh"
63	MANAGER_SCRIPT="$SCRIPT_DIR/../Transcodes/transcode_manager.sh"
64
65	# ==============================================================================================
66	# ━━━ Setup ━━━
67	# ==============================================================================================
68	if [[ "$EUID" -ne 0 ]]; then
69	    error "Must be run as root"
70	    exit 1
71	fi
72
73	acquire_lock "wait"
74
75	if ! command -v docker &>/dev/null; then
76	    error "Docker command not found"
77	    exit 1
78	fi
79
80	# detect_hosts() sets MY_ID and aliases all HOST*_TRANSCODE_* vars
81	detect_hosts
82
83	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing through to child scripts"
84
85	# ==============================================================================================
86	# ━━━ Status ━━━
87	# ==============================================================================================
88	if [[ "$SHOW_STATUS" == true ]]; then
89	    echo ""
90	    echo "━━━━━ $ICON_SUMMARY TRANSCODE MANAGEMENT STATUS ━━━━━"
91	    echo "$ICON_HOST   Identity:       $MY_ID ($LOCAL_SERVER_NAME)"
92	    echo "$ICON_RAM    Ramdisk:        $RAMDISK_PATH ($RAMDISK_SIZE)"
93	    echo "$ICON_DISK   SSD fallback:   $TRANSCODE_SSD"
94	    echo "$ICON_LINK   Symlink:        $TRANSCODE_LINK"
95	    echo "$ICON_TIME   Schedule:       every 7 minutes"
96	    echo ""
97	    echo "━━━ Child Scripts ━━━"
98	    [[ -f "$CLEANUP_SCRIPT" ]] && \
99	        echo "  $ICON_SUCCESS transcode_cleanup.sh — found" || \
100	        echo "  $ICON_ERROR  transcode_cleanup.sh — NOT FOUND at $CLEANUP_SCRIPT"
101	    [[ -f "$MANAGER_SCRIPT" ]] && \
102	        echo "  $ICON_SUCCESS transcode_manager.sh — found" || \
103	        echo "  $ICON_ERROR  transcode_manager.sh — NOT FOUND at $MANAGER_SCRIPT"
104	    echo ""
105	    echo "━━━ Daily Log ━━━"
106	    if [[ -f "${TRANSCODE_DAILY_LOG:-}" ]] && [[ -s "$TRANSCODE_DAILY_LOG" ]]; then
107	        ENTRY_COUNT=$(wc -l < "$TRANSCODE_DAILY_LOG")
108	        OLDEST=$(awk -F'|' 'NR==1{print $1}' "$TRANSCODE_DAILY_LOG")
109	        NEWEST=$(awk -F'|' 'END{print $1}' "$TRANSCODE_DAILY_LOG")
110	        echo "  $ICON_SUCCESS $TRANSCODE_DAILY_LOG ($ENTRY_COUNT entries, $OLDEST → $NEWEST)"
111	    else
112	        echo "  $ICON_SKIP $TRANSCODE_DAILY_LOG — no data yet"
113	    fi
114	    echo ""
115	    echo "━━━ Current State ━━━"
116	    if [[ -f "${TRANSCODE_STATE_FILE:-/tmp/transcode_state.db}" ]]; then
117	        while IFS='=' read -r key val; do
118	            [[ -n "$key" ]] && echo "  $key = $val"
119	        done < "${TRANSCODE_STATE_FILE:-/tmp/transcode_state.db}"
120	    else
121	        echo "  State file not found (ramdisk_setup.sh creates it at array start)"
122	    fi
123	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
124	    exit 0
125	fi
126
127	# ==============================================================================================
128	# ━━━ Validate Child Scripts ━━━
129	# ==============================================================================================
130	if [[ ! -f "$CLEANUP_SCRIPT" ]]; then
131	    error "transcode_cleanup.sh not found: $CLEANUP_SCRIPT"
132	    exit 1
133	fi
134
135	if [[ ! -f "$MANAGER_SCRIPT" ]]; then
136	    error "transcode_manager.sh not found: $MANAGER_SCRIPT"
137	    exit 1
138	fi
139
140	# ==============================================================================================
141	# ━━━ Pre-run State Snapshot ━━━
142	# ==============================================================================================
143	if mountpoint -q "${RAMDISK_PATH:-}" 2>/dev/null; then
144	    _rd_used=$(df "$RAMDISK_PATH" --output=used  2>/dev/null | tail -1 | tr -d ' ')
145	    _rd_avail=$(df "$RAMDISK_PATH" --output=avail 2>/dev/null | tail -1 | tr -d ' ')
146	    _rd_used_gb=$(awk "BEGIN {printf \"%.2f\", ${_rd_used:-0}/1048576}")
147	    _rd_avail_gb=$(awk "BEGIN {printf \"%.2f\", ${_rd_avail:-0}/1048576}")
148	    _rd_target=$(readlink "${TRANSCODE_LINK:-}" 2>/dev/null || echo "unknown")
149	    log "$ICON_RAM Ramdisk: ${_rd_used_gb}GB used / ${_rd_avail_gb}GB avail — symlink → ${_rd_target##*/}"
150	else
151	    log "$ICON_RAM Ramdisk: not mounted"
152	fi
153
154	# ==============================================================================================
155	# ━━━ Run Cleanup ━━━
156	# ==============================================================================================
157	DRY_FLAG=""
158	[[ "$DRY_RUN" == true ]] && DRY_FLAG="--dry-run"
159
160	_cleanup_start=$(date +%s)
161	bash "$CLEANUP_SCRIPT" $DRY_FLAG
162	CLEANUP_EXIT=$?
163	log "cleanup: $(format_duration $(( $(date +%s) - _cleanup_start ))) (exit $CLEANUP_EXIT)"
164
165	# ==============================================================================================
166	# ━━━ Run Manager ━━━
167	# ==============================================================================================
168	# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each run
169	# No --no-log flag here — manager owns the log write for this cycle ✅
170	_manager_start=$(date +%s)
171	bash "$MANAGER_SCRIPT" $DRY_FLAG
172	MANAGER_EXIT=$?
173	log "manager: $(format_duration $(( $(date +%s) - _manager_start ))) (exit $MANAGER_EXIT)"
174
175	# ==============================================================================================
176	# ━━━ Exit ━━━
177	# ==============================================================================================
178	# Return worst exit code — caller knows if either script failed
179	[[ "$CLEANUP_EXIT" -ne 0 || "$MANAGER_EXIT" -ne 0 ]] && exit 1
180	exit 0


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================ Watchdog Orchestrator ===========================================
4	# ==============================================================================================
5	# Runs WATCHDOG_ORCHESTRATOR_SCRIPTS in order each cron cycle.
6	# Schedule: */15 * * * * (every 15 minutes)
7	#
8	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
9	#   Driven by WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf — add, remove, or reorder there.
10	#   Default: resource_watchdog → docker_watchdog → system_watchdog → unraid_api_key_renew → stability_watchdog
11	#
12	# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
13	#   Resource Watchdog first — frees RAM and CPU before healing attempts container restarts.
14	#                            Containers restarted into a resource-pressured system just fail again.
15	#   Docker Watchdog second — restarts with pressure already reduced, more likely to stabilise.
16	#   System Watchdog third  — system component health after containers are healed.
17	#   API key renew fourth   — self-heals unraid-api registry loss; check-first, silent when valid.
18	#   Stability Watchdog last — only reboots when all prior layers could not resolve the issue.
19	#
20	# ── ARRAY CHECK ───────────────────────────────────────────────────────────────────────────────
21	#   Exits immediately if /mnt/user is not mounted as shfs (array not started).
22	#   Watchdogs check Docker containers and storage — meaningless without the array.
23	#   Prevents false positives and unnecessary reboots when array is stopped or stopping.
24	#
25	# ── STARTUP GRACE ─────────────────────────────────────────────────────────────────────────────
26	#   No action until system uptime >= WATCHDOG_STARTUP_GRACE seconds.
27	#   Prevents false positives from containers still starting at array launch.
28	#   Each sub-script enforces this independently — orchestrator exits early to avoid log noise.
29	#
30	# ── OVERLAP PROTECTION ────────────────────────────────────────────────────────────────────────
31	#   acquire_lock() — exits immediately if a prior cycle is still in progress.
32	#   Prevents pile-up when a cycle runs long (daemon restart attempt = 30s, etc.).
33	#
34	# ── REPLACES ──────────────────────────────────────────────────────────────────────────────────
35	#   Continuous loops previously in system_watchdog.sh and docker_watchdog.sh.
36	#   Those scripts are now single-pass — this orchestrator provides the cadence.
37	#
38	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
39	#   WATCHDOG_ORCHESTRATOR_SCRIPTS         — watchdogs to run, in order
40	#   WATCHDOG_STARTUP_GRACE                — seconds after boot before checks activate
41	#   WATCHDOG_ORCHESTRATOR_HEARTBEAT       — periodic heartbeat log toggle
42	#   WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS — heartbeat interval in hours
43	#
44	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
45	#   watchdog_orchestrator.sh           — normal run (called by cron every 15 minutes)
46	#   watchdog_orchestrator.sh --dry-run — pass --dry-run to all sub-scripts
47	#   watchdog_orchestrator.sh --status  — show script paths and current grace state
48	#   watchdog_orchestrator.sh --log     — verbose output from all sub-scripts
49	# ==============================================================================================
50
51	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
52	ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
53
54	source "$ECOSYSTEM_ROOT/load_config.sh"
55
56	parse_args "$@"
57
58	# ==============================================================================================
59	# ━━━ Setup ━━━
60	# ==============================================================================================
61	if [[ "$EUID" -ne 0 ]]; then
62	    error "Must be run as root"
63	    exit 1
64	fi
65
66	# Skip immediately if another cycle is still running — no pile-up
67	acquire_lock
68
69	detect_hosts
70
71	log "$ICON_GEAR Config: grace=${WATCHDOG_STARTUP_GRACE}s heartbeat=${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true}/${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}hr scripts=${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"
72	log "$ICON_WATCHDOG Order: $(for s in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do printf '%s ' "${s##*/}"; done)"
73
74	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing --dry-run to all sub-scripts"
75
76	# Derive a display name from a script path: "resource_watchdog.sh" → "Resource Watchdog"
77	_watchdog_display_name() {
78	    local path="$1"
79	    local base="${path##*/}"
80	    base="${base%.sh}"
81	    base="${base//_/ }"
82	    echo "$base" | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) substr($i,2); print}'
83	}
84
85	# ==============================================================================================
86	# ━━━ Status ━━━
87	# ==============================================================================================
88	if [[ "$SHOW_STATUS" == true ]]; then
89	    echo ""
90	    echo "━━━━━ $ICON_SUMMARY WATCHDOG ORCHESTRATOR STATUS ━━━━━"
91	    echo "$ICON_HOST  Identity: $MY_ID ($LOCAL_SERVER_NAME)"
92	    echo ""
93
94	    UPTIME_S=$(awk '{print int($1)}' /proc/uptime)
95	    if [[ "$UPTIME_S" -lt "$WATCHDOG_STARTUP_GRACE" ]]; then
96	        warn "Within startup grace — $(format_duration $UPTIME_S) / $(format_duration $WATCHDOG_STARTUP_GRACE)"
97	    else
98	        echo "Past startup grace — $(format_duration $UPTIME_S) uptime"
99	    fi
100
101	    echo ""
102	    echo "── Sub-scripts ──"
103	    for entry in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do
104	        local_path="$ECOSYSTEM_ROOT/$entry"
105	        label="$(_watchdog_display_name "$entry")"
106	        if [[ -f "$local_path" ]]; then
107	            [[ -x "$local_path" ]] && icon="$ICON_DONE" || icon="$ICON_WARN"
108	            echo "  $icon $label — ${local_path##*/}"
109	        else
110	            echo "  $ICON_ERROR $label — NOT FOUND: $local_path"
111	        fi
112	    done
113
114	    echo ""
115	    echo "  Schedule:   */15 * * * * (every 15 minutes)"
116	    echo "  Heartbeat:  ${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true} / every ${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}hr"
117	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
118	    exit 0
119	fi
120
121	# ==============================================================================================
122	# ━━━ Array Check ━━━
123	# ==============================================================================================
124	if ! platform_storage_healthy; then
125	    echo "Array not started — skipping watchdog cycle"
126	    exit 0
127	fi
128	log "$ICON_DISK Array: $(platform_storage_path) mounted ✅"
129
130	# ==============================================================================================
131	# ━━━ Startup Grace ━━━
132	# ==============================================================================================
133	UPTIME_SECONDS=$(awk '{print int($1)}' /proc/uptime)
134	if [[ "$UPTIME_SECONDS" -lt "$WATCHDOG_STARTUP_GRACE" ]]; then
135	    echo "Startup grace — $(format_duration $UPTIME_SECONDS) / $(format_duration $WATCHDOG_STARTUP_GRACE) — skipping cycle"
136	    exit 0
137	fi
138	log "Startup grace: past — uptime $(format_duration $UPTIME_SECONDS)"
139
140	# ==============================================================================================
141	# ━━━ Run Watchdog Cycle ━━━
142	# ==============================================================================================
143	CYCLE_START=$(date +%s)
144	PASS=()
145	FAIL=()
146
147	run_watchdog() {
148	    local name="$1" script="$2"
149
150	    if [[ ! -f "$script" ]]; then
151	        error "$name — not found: $script"
152	        FAIL+=("$name:missing")
153	        return 1
154	    fi
155
156	    [[ ! -x "$script" ]] && chmod +x "$script"
157
158	    local extra_args=()
159	    [[ "$DRY_RUN" == true ]] && extra_args+=("--dry-run")
160	    [[ "$VERBOSE" == true ]] && extra_args+=("--log")
161
162	    local _ws
163	    _ws=$(date +%s)
164	    log "$ICON_START $name"
165	    if bash "$script" "${extra_args[@]}"; then
166	        log "$ICON_DONE $name — done in $(format_duration $(( $(date +%s) - _ws )))"
167	        PASS+=("$name")
168	        return 0
169	    else
170	        error "$name — non-zero exit ($(format_duration $(( $(date +%s) - _ws ))))"
171	        FAIL+=("$name")
172	        return 1
173	    fi
174	}
175
176	for _entry in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do
177	    run_watchdog "$(_watchdog_display_name "$_entry")" "$ECOSYSTEM_ROOT/$_entry"
178	done
179
180	CYCLE_END=$(date +%s)
181	DURATION=$(( CYCLE_END - CYCLE_START ))
182
183	# ==============================================================================================
184	# ━━━ Heartbeat ━━━
185	# ==============================================================================================
186	if [[ "${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true}" == true ]]; then
187	    HB_SECONDS=$(( ${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1} * 3600 ))
188	    HB_COUNT_FILE="${STATE_DIR:-/tmp}/watchdog_orch_hb.count"
189	    HB_COUNT=$(cat "$HB_COUNT_FILE" 2>/dev/null || echo 0)
190	    HB_COUNT=$(( HB_COUNT + 1 ))
191	    echo "$HB_COUNT" > "$HB_COUNT_FILE"
192	    # Each cron run = ~60s — use count × 60 as uptime approximation
193	    HB_ELAPSED=$(( HB_COUNT * 60 ))
194	    if [[ "$HB_SECONDS" -gt 0 ]] && (( HB_ELAPSED % HB_SECONDS < 60 )) && [[ "$HB_COUNT" -gt 1 ]]; then
195	        HB_HR=$(( HB_ELAPSED / 3600 ))
196	        warn "♥ watchdog_orchestrator alive — $MY_ID — ~${HB_HR}hr ($(date '+%H:%M:%S'))"
197	    fi
198	fi
199
200	# ==============================================================================================
201	# ━━━ Summary — only shown on failures or --log ━━━
202	# ==============================================================================================
203	if [[ "${#FAIL[@]}" -gt 0 || "$VERBOSE" == true ]]; then
204	    echo ""
205	    echo "━━━━━ $ICON_SUMMARY WATCHDOG CYCLE — $MY_ID — $(date '+%H:%M:%S') ━━━━━"
206	    for p in "${PASS[@]}"; do log "  $ICON_DONE  $p"; done
207	    for f in "${FAIL[@]}"; do error "  $ICON_ERROR $f"; done
208	    echo "$ICON_TIME  Duration: $(format_duration $DURATION)"
209	    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
210
211	    if [[ "${#FAIL[@]}" -gt 0 ]]; then
212	        notify "Watchdog cycle failure on $(hostname) ($MY_ID) — ${FAIL[*]}" \
213	            "Watchdog Orchestrator" "warning"
214	    fi
215	fi
216


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Weekly Sync Maintenance ========================================
4	# ==============================================================================================
5	# Weekly maintenance window orchestrator — clean sync, container updates, weekly restarts.
6	# Schedule: 30 2 * * 0 (Sunday 2:30am — before Sunday 7am coffee report)
7	#
8	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
9	#   1. Stop local containers         — Emby + auth stack stopped locally
10	#   2. Stop remote containers        — Emby + auth stack stopped remotely via SSH
11	#   3. Pull updates locally          — if WEEKLY_SYNC_UPDATES=true (zero extra downtime)
12	#   4. Pull updates remotely         — if WEEKLY_SYNC_UPDATES_REMOTE=true
13	#   5. rsync WEEKLY_SYNC_SHARES      — full clean mirror, containers stopped both sides
14	#   6. Start remote containers       — correct order, delayed start respected
15	#   7. Start local containers        — correct order, delayed start respected
16	#   8. WEEKLY_MAINTENANCE_SCRIPTS    — weekly restarts etc. (docker_weekly_restart.sh)
17	#   9. docker_update.sh --remainder  — update all containers not in daily or weekly sync window
18	#
19	# ── WHY WEEKLY NOT NIGHTLY FOR EMBY ──────────────────────────────────────────────────────────
20	# Emby builds a warm image cache on HOST2 throughout the week.
21	# Syncing nightly resets cache — cold loads every morning for users.
22	# Weekly sync: cache stays warm 6 days, resets Sunday night while users sleep.
23	# emby-fallback dirty sync covers watch states + library every 30min between weekly syncs.
24	#
25	# ── CONTAINER UPDATES ─────────────────────────────────────────────────────────────────────────
26	# Containers already stopped for sync — updates pull at zero extra downtime.
27	# Both servers start on identical image versions after the window completes.
28	# Toggle: WEEKLY_SYNC_UPDATES / WEEKLY_SYNC_UPDATES_REMOTE in master.conf
29	#
30	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
31	# detect_hosts() sets MY_ID — used in banner, summary, and notifications.
32	# WEEKLY_SYNC_SHARES and WEEKLY_MAINTENANCE_SCRIPTS configured in master.conf.
33	# Same script runs correctly on both servers.
34	#
35	# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
36	#   Root check           — stop/start containers, rsync require root
37	#   acquire_lock         — prevents concurrent weekly windows
38	#   check_connectivity   — verifies remote before any remote operations
39	#   check_remote_rootfs  — aborts if remote rootfs nearly full
40	#   DOCKER_TIMEOUT       — all docker calls protected
41	#   SSH_TIMEOUT          — all SSH calls protected
42	#   platform_require_cmd  — notify validated before use
43	#   Silent on success    — runs weekly, only failures warrant notification
44	#
45	# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
46	#   WEEKLY_SYNC_SHARES             — shares synced during window
47	#   WEEKLY_MAINTENANCE_SCRIPTS     — scripts run after sync
48	#   WEEKLY_SYNC_UPDATES            — toggle local container updates
49	#   WEEKLY_SYNC_UPDATES_REMOTE     — toggle remote container updates
50	#   WEEKLY_RSYNC_ENABLED           — enable/disable rsync section
51	#
52	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
53	#   weekly_sync_maintenance.sh              — normal run
54	#   weekly_sync_maintenance.sh --dry-run    — preview without stopping containers or syncing
55	#   weekly_sync_maintenance.sh --log        — verbose per-share/per-job output
56	#   weekly_sync_maintenance.sh --status     — show configuration and exit
57	# ==============================================================================================
58
59	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
60
61	source "$SCRIPT_DIR/../load_config.sh"
62
63	RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
64	SCRIPTS_ROOT="$SCRIPT_DIR/.."
65
66	parse_args "$@"
67
68	DOCKER_TIMEOUT=30   # container stop/start needs longer than normal
69	SSH_TIMEOUT=30      # remote pulls can be slow
70
71	# ==============================================================================================
72	# ━━━ Setup ━━━
73	# ==============================================================================================
74	if [[ "$EUID" -ne 0 ]]; then
75	    error "Must be run as root"
76	    exit 1
77	fi
78
79
80	acquire_lock
81
82	if ! command -v docker &>/dev/null; then
83	    error "Docker command not found"
84	    exit 1
85	fi
86
87	detect_hosts
88	resolve_remote_ip
89
90	WINDOW_START=$(date +%s)
91	PASS=()
92	FAIL=()
93	JOB_PASS=()
94	JOB_FAIL=()
95
96	# Load container lists from profile config
97	read -r -a MAINTENANCE_CONTAINERS <<< \
98	    "${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
99
100	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no containers will be stopped, no sync, no updates"
101
102	# ── Helper — run a post-sync maintenance script ────────────────────────────────────────────────
103	run_job() {
104	    local script_entry="$1"
105	    local extra_dry=""
106	    [[ "$DRY_RUN" == true ]] && extra_dry="--dry-run"
107
108	    read -r -a script_args <<< "$script_entry"
109	    local script_path="$SCRIPTS_ROOT/${script_args[0]}"
110	    local script_name
111	    script_name=$(basename "${script_args[0]}")
112	    local extra_args=("${script_args[@]:1}")
113
114	    if [[ ! -f "$script_path" ]]; then
115	        error "$script_name — not found at $script_path"
116	        JOB_FAIL+=("$script_name")
117	        return 1
118	    fi
119
120	    log "Running: $script_name ${extra_args[*]}"
121	    # shellcheck disable=SC2086
122	    if bash "$script_path" "${extra_args[@]}" $extra_dry; then
123	        log "$script_name — done ✅"
124	        JOB_PASS+=("$script_name ${extra_args[*]}")
125	    else
126	        error "$script_name — failed (exit $?)"
127	        JOB_FAIL+=("$script_name ${extra_args[*]}")
128	    fi
129	}
130
131	# ==============================================================================================
132	# ━━━ Status ━━━
133	# ==============================================================================================
134	if [[ "$SHOW_STATUS" == true ]]; then
135	    echo ""
136	    echo "━━━━━ $ICON_SUMMARY WEEKLY SYNC STATUS ━━━━━"
137	    echo "$ICON_HOST   Identity:        $MY_ID ($LOCAL_SERVER_NAME)"
138	    echo "$ICON_NET    Remote:          $REMOTE_ID ($REMOTE_SERVER_NAME)"
139	    echo "$ICON_SYNC   Rsync enabled:   ${RSYNC_ENABLED:-true}"
140	    echo "$ICON_SYNC   Weekly enabled:  ${WEEKLY_RSYNC_ENABLED:-false}"
141	    echo "$ICON_GEAR   Local updates:   ${WEEKLY_SYNC_UPDATES:-false}"
142	    echo "$ICON_GEAR   Remote updates:  ${WEEKLY_SYNC_UPDATES_REMOTE:-false}"
143	    echo ""
144	    echo "━━━ Weekly Sync Shares ━━━"
145	    if [[ ${#WEEKLY_SYNC_SHARES[@]} -eq 0 ]]; then
146	        warn "  No WEEKLY_SYNC_SHARES configured"
147	    else
148	        for share in "${WEEKLY_SYNC_SHARES[@]}"; do
149	            [[ -n "$share" ]] && echo "  $ICON_SYNC $(basename "$share") ($share)"
150	        done
151	    fi
152	    echo ""
153	    echo "━━━ Weekly Maintenance Scripts ━━━"
154	    if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
155	        echo "  None configured"
156	    else
157	        for entry in "${WEEKLY_MAINTENANCE_SCRIPTS[@]}"; do
158	            [[ -n "$entry" ]] && echo "  $ICON_GEAR ${entry##*/}"
159	        done
160	    fi
161	    echo ""
162	    echo "━━━ Containers (from profile config) ━━━"
163	    for c in "${MAINTENANCE_CONTAINERS[@]}"; do
164	        [[ -n "$c" ]] && echo "  $ICON_CONTAINERS $c"
165	    done
166	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
167	    exit 0
168	fi
169
170	# ==============================================================================================
171	# ━━━ Pre-flight Checks ━━━
172	# ==============================================================================================
173	echo ""
174	echo "━━━ $ICON_GEAR Weekly Sync Maintenance — $MY_ID — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
175	echo ""
176	echo "━━━ $ICON_SHIELD Pre-flight ━━━"
177
178	check_connectivity
179	check_remote_rootfs
180
181	# ==============================================================================================
182	# ━━━ Stop Containers ━━━
183	# ==============================================================================================
184	echo ""
185	echo "━━━ $ICON_STOP $ICON_CONTAINERS Stop Containers ━━━"
186
187	if [[ "$DRY_RUN" == true ]]; then
188	    warn "DRY RUN — containers will not be stopped"
189	else
190	    # Load container lists for stop functions
191	    read -r -a CRITICAL_CONTAINER_NAMES <<< \
192	        "${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-}"
193	    read -r -a DELAYED_CONTAINERS       <<< "${PROFILE_DELAYED_CONTAINERS[critical-data]:-}"
194	    CONTAINER_DELAY="${PROFILE_CONTAINER_DELAY[critical-data]:-15}"
195
196	    stop_local_containers
197	    stop_containers
198	fi
199
200	# ==============================================================================================
201	# ━━━ Container Updates ━━━
202	# ==============================================================================================
203	echo ""
204	echo "━━━ $ICON_GEAR Container Updates ━━━"
205
206	if [[ "$WEEKLY_SYNC_UPDATES" == true ]]; then
207	    if [[ "$DRY_RUN" == true ]]; then
208	        for c in "${MAINTENANCE_CONTAINERS[@]}"; do
209	            [[ -n "$c" ]] && warn "DRY RUN — would pull: $c"
210	        done
211	    else
212	        log "Pulling local container updates..."
213	        for c in "${MAINTENANCE_CONTAINERS[@]}"; do
214	            [[ -z "$c" ]] && continue
215	            IMAGE=$(timeout "$DOCKER_TIMEOUT" docker inspect \
216	                "$c" --format '{{.Config.Image}}' 2>/dev/null)
217	            if [[ -z "$IMAGE" ]]; then
218	                log "$c — not found locally, skipping update"
219	                continue
220	            fi
221	            log "Pulling $IMAGE for $c..."
222	            if docker pull "$IMAGE" >/dev/null 2>&1; then
223	                log "$c — image updated ✅"
224	            else
225	                warn "$c — pull failed, will start on existing image"
226	            fi
227	        done
228	    fi
229	else
230	    echo "WEEKLY_SYNC_UPDATES=false — skipping local updates"
231	fi
232
233	if [[ "$WEEKLY_SYNC_UPDATES_REMOTE" == true ]]; then
234	    if [[ "$DRY_RUN" == true ]]; then
235	        warn "DRY RUN — would pull updates on $REMOTE_SERVER_NAME"
236	    else
237	        log "Pulling remote container updates on $REMOTE_SERVER_NAME..."
238	        for c in "${MAINTENANCE_CONTAINERS[@]}"; do
239	            [[ -z "$c" ]] && continue
240	            IMAGE=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
241	                -o ConnectTimeout="$SSH_TIMEOUT" \
242	                root@"$REMOTE_SERVER" \
243	                "docker inspect $c --format '{{.Config.Image}}' 2>/dev/null" 2>/dev/null)
244	            if [[ -z "$IMAGE" ]]; then
245	                log "$c — not found on remote, skipping update"
246	                continue
247	            fi
248	            log "Pulling $IMAGE for $c on $REMOTE_SERVER_NAME..."
249	            if timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
250	                -o ConnectTimeout="$SSH_TIMEOUT" \
251	                root@"$REMOTE_SERVER" \
252	                "docker pull $IMAGE" >/dev/null 2>&1; then
253	                log "$c — remote image updated ✅"
254	            else
255	                warn "$c — remote pull failed, will start on existing image"
256	            fi
257	        done
258	    fi
259	else
260	    echo "WEEKLY_SYNC_UPDATES_REMOTE=false — skipping remote updates"
261	fi
262
263	# ==============================================================================================
264	# ━━━ Critical Shares Sync ━━━
265	# ==============================================================================================
266	SHARE_COUNT=${#WEEKLY_SYNC_SHARES[@]}
267
268	echo ""
269	echo "━━━ $ICON_SYNC Critical Shares Sync — $SHARE_COUNT share(s) ━━━"
270
271	SYNC_START=$(date +%s)
272	JOB_NUM=0
273
274	if ! check_rsync_enabled "WEEKLY"; then
275	    warn "Weekly rsync disabled — skipping all $SHARE_COUNT sync job(s)"
276	    warn "Proceeding to container start and maintenance scripts..."
277	elif [[ "$SHARE_COUNT" -eq 0 ]]; then
278	    warn "No WEEKLY_SYNC_SHARES configured — skipping sync"
279	    warn "Check WEEKLY_SYNC_SHARES in master.conf"
280	else
281	    RSYNC_DRY=""
282	    [[ "$DRY_RUN" == true ]] && RSYNC_DRY="--dry-run"
283
284	    for JOB in "${WEEKLY_SYNC_SHARES[@]}"; do
285	        (( JOB_NUM++ ))
286	        JOB_NAME=$(basename "$JOB")
287
288	        echo ""
289	        echo "━━━ [$JOB_NUM/$SHARE_COUNT] $JOB_NAME ━━━"
290
291	        JOB_START=$(date +%s)
292	        bash "$RSYNC_SCRIPT" "$JOB" $RSYNC_DRY
293	        EXIT_CODE=$?
294	        JOB_DUR=$(format_duration $(( $(date +%s) - JOB_START )))
295
296	        if [[ "$EXIT_CODE" -eq 0 ]]; then
297	            PASS+=("$JOB_NAME")
298	            log "$JOB_NAME — done in $JOB_DUR ✅"
299	        else
300	            FAIL+=("$JOB_NAME")
301	            error "$JOB_NAME — failed after $JOB_DUR (exit $EXIT_CODE)"
302	        fi
303	        echo ""
304	    done
305	fi
306
307	SYNC_END=$(date +%s)
308
309	# ==============================================================================================
310	# ━━━ Start Containers ━━━
311	# ==============================================================================================
312	echo ""
313	echo "━━━ $ICON_START $ICON_CONTAINERS Start Containers ━━━"
314
315	if [[ "$DRY_RUN" == true ]]; then
316	    warn "DRY RUN — containers will not be started"
317	else
318	    start_containers
319	    start_local_containers
320	fi
321
322	# ==============================================================================================
323	# ━━━ Post-sync Jobs ━━━
324	# ==============================================================================================
325	if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -gt 0 ]]; then
326	    echo ""
327	    echo "━━━ $ICON_GEAR Post-sync Jobs ━━━"
328	    for script_entry in "${WEEKLY_MAINTENANCE_SCRIPTS[@]}"; do
329	        [[ -z "$script_entry" ]] && continue
330	        echo ""
331	        run_job "$script_entry"
332	    done
333	fi
334
335	# ==============================================================================================
336	# ━━━ Remainder Container Updates ━━━
337	# ==============================================================================================
338	# Updates all running containers not already covered by daily or the weekly sync window.
339	# Runs last — weekly sync-window containers are back up before this pulls their peers.
340	echo ""
341	echo "━━━ $ICON_CONTAINERS Remainder Container Updates ━━━"
342
343	DOCKER_UPDATE_SCRIPT="$SCRIPTS_ROOT/Docker_Essentials/docker_update.sh"
344	if [[ ! -f "$DOCKER_UPDATE_SCRIPT" ]]; then
345	    warn "docker_update.sh not found — skipping remainder updates"
346	    JOB_FAIL+=("docker_update.sh --remainder")
347	else
348	    _remainder_args=("--remainder")
349	    [[ "$DRY_RUN" == true ]] && _remainder_args+=("--dry-run")
350	    if bash "$DOCKER_UPDATE_SCRIPT" "${_remainder_args[@]}"; then
351	        echo "Remainder updates complete ✅"
352	        JOB_PASS+=("docker_update.sh --remainder")
353	    else
354	        warn "Remainder updates completed with errors"
355	        JOB_FAIL+=("docker_update.sh --remainder")
356	    fi
357	    unset _remainder_args
358	fi
359
360	WINDOW_END=$(date +%s)
361
362	# ==============================================================================================
363	# ━━━ Summary ━━━
364	# ==============================================================================================
365	echo ""
366	echo "━━━━━ $ICON_SUMMARY WEEKLY SYNC MAINTENANCE SUMMARY ━━━━━"
367	echo "$ICON_HOST   Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
368	echo "$ICON_TIME   Window:    $(date -d @"$WINDOW_START" '+%Y-%m-%d %H:%M:%S') → $(date -d @"$WINDOW_END" '+%H:%M:%S')"
369	echo "$ICON_TIME   Duration:  $(format_duration $(( WINDOW_END - WINDOW_START )))"
370	echo "$ICON_GEAR   Updates:   local=${WEEKLY_SYNC_UPDATES:-false} remote=${WEEKLY_SYNC_UPDATES_REMOTE:-false}"
371	echo ""
372
373	echo "$ICON_SYNC  Sync jobs ($SHARE_COUNT):"
374	for job in "${PASS[@]}"; do echo "  $ICON_DONE  $job"; done
375	for job in "${FAIL[@]}"; do echo "  $ICON_ERROR $job"; done
376	echo "  Passed: ${#PASS[@]}  Failed: ${#FAIL[@]}"
377
378	if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
379	    echo ""
380	    echo "$ICON_GEAR  Post-sync jobs:"
381	    for job in "${JOB_PASS[@]}"; do echo "  $ICON_DONE  $job"; done
382	    for job in "${JOB_FAIL[@]}"; do echo "  $ICON_ERROR $job"; done
383	fi
384
385	TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
386
387	echo ""
388	if [[ "$DRY_RUN" == true ]]; then
389	    warn "DRY RUN — no changes made"
390	elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
391	    echo "$ICON_DONE Status: all complete ✅ — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
392	else
393	    warn "Status: $TOTAL_FAIL failure(s)"
394	    notify "Weekly maintenance failed on $(hostname) ($MY_ID) — sync: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
395	        "Weekly Maintenance" "warning"
396	fi
397	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
398
399	[[ "$TOTAL_FAIL" -gt 0 ]] && exit 1
400	exit 0


Now let me analyze all the scripts systematically for discrepancies.

## Audit Report

Based on my review of all 19 bash scripts, here are the real discrepancies between header documentation and actual code:

**Monitors:**

✅ backup_verify.sh

✅ bandwidth_monitor.sh

cert_monitor.sh
- Header states "RUNTIME MODES: cert_monitor.sh --log — Verbose per-domain output during the run" but no actual logging of per-domain details occurs—the script only outputs when --log is passed but the code doesn't change behavior based on VERBOSE/ENABLE_LOGGING flags from parse_args.

✅ emby_session_report.sh

✅ mesh_monitor.sh

✅ smart_health.sh

✅ system_tuning_monitor.sh

weekly_health_digest.sh
- Header lists "DIGEST_SMART_ON_BANDWIDTH" as config but code never uses this variable (line 300-305 only checks if FAILED_COUNT > 0, not DIGEST_SMART_ON_BANDWIDTH). Line 289 references an undefined variable when filtering BANDWIDTH_LOG: column $7 is tested for "LARGE" but the new log format documented at line 289 is "date|time|profile|duration|status|bytes|warn_flag" — column $7 exists and is correct, but the logic doesn't match the documented DIGEST_SMART_ON_BANDWIDTH gate.

zfs_memory_snapshot.sh
- Header states "Output goes to both console (for User Scripts output log) and ZFS_REPORT_LOG for week-over-week comparison" but when DRY_RUN=true (line 150), the tee is never executed, so normal --dry-run output still goes to console. The header claim about "both console and log" implies log write during dry-run, which doesn't happen.

**Orchestrators:**

✅ array_started.sh

✅ array_stopping.sh

✅ critical_sync_maintenance.sh

✅ daily_sync_maintenance.sh

✅ intermediate_sync_maintenance.sh

✅ monthly_maintenance.sh

sunday_morning_coffee_report.sh
- Header at line 45 states flag is "--log" but code at line 45 checks ENABLE_LOGGING (which comes from parse_args), and line 44 uses variable name ENABLE_LOGGING while the rest of the system uses VERBOSE. The actual flag is parsed as --log, so the documentation is correct, but variable naming is inconsistent.

✅ transcode_management.sh

watchdog_orchestrator.sh
- Header at line 41 lists config "WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS" but code only defines WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS with a fallback to 1 (line 187). However, the code never validates this is set, and line 187 uses `${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}` correctly. The header is accurate.

✅ weekly_sync_maintenance.sh

---

**Summary of Real Discrepancies:**

1. **cert_monitor.sh** — Header documents `--log` flag produces "verbose per-domain output" but the code doesn't actually change output behavior based on VERBOSE flag.

2. **weekly_health_digest.sh** — Header documents DIGEST_SMART_ON_BANDWIDTH config variable but it's never used in the code (bandwidth section always notifies on large transfers regardless of this gate).

3. **zfs_memory_snapshot.sh** — Header states output goes to "both console and log file" but during --dry-run, output only goes to console (line 147 tee is skipped).