Gmer4Lfe
d5cf3db2ec
A gated-off section left nothing failed, so the weekly could run for hours and report "all complete" beside "0 shares synced"; skipped is now derived from what was expected rather than self-reported, and the verdict degrades to PARTIAL instead of flattering.
273 lines
11 KiB
Bash
Executable File
273 lines
11 KiB
Bash
Executable File
#!/bin/bash
|
|
# ==============================================================================================
|
|
# ================================= Array Start Orchestrator ===================================
|
|
# ==============================================================================================
|
|
#
|
|
# PURPOSE
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Single entry point for array start — fired by the Varaverk plugin's
|
|
# disks_mounted event hook (Plugin/unraid/event/disks_mounted/array_start_jobs).
|
|
# Launches everything configured in ARRAY_START_SCRIPTS in master.conf.
|
|
# This script exits after launching all scripts — the event hook sees it complete normally.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL MODEL
|
|
# ==============================================================================================
|
|
#
|
|
# Configured in master.conf ARRAY_START_SCRIPTS — no changes to this script ever needed.
|
|
# Current order (order matters — see below):
|
|
#
|
|
# ONE-SHOT (run and exit naturally):
|
|
# System_Essentials/unraid_api_key_renew.sh — re-register Varaverk API key at boot
|
|
# System_Essentials/inotify_tuning.sh — raise inotify limits before containers start
|
|
# System_Essentials/docker_syslog_filter.sh — suppress veth log noise before logs fill
|
|
# System_Essentials/php_fpm_max_children.sh — WebGUI performance tuning
|
|
# Transcodes/ramdisk_setup.sh — create tmpfs + symlink before Emby starts
|
|
# Docker_Essentials/docker_network_connect.sh — ensure networks + container connections
|
|
#
|
|
# CONTINUOUS (run until array stops):
|
|
# Fallback/fallback.sh — mutual fallback monitor
|
|
#
|
|
# NOTE: watchdogs (docker, system, stability) are NOT launched here.
|
|
# They run via watchdog_orchestrator.sh every 15 min (cron), not as daemons.
|
|
#
|
|
# Script is launched in background with bash script.sh &
|
|
# After 1 second: if PID still alive → continuous (running in background)
|
|
# if PID dead + exit 0 → one-shot completed successfully
|
|
# if PID dead + exit N → failure
|
|
#
|
|
# ==============================================================================================
|
|
# DESIGN PRINCIPLES
|
|
# ==============================================================================================
|
|
#
|
|
# Order Is Load-Bearing
|
|
# inotify limits must be raised before containers start — containers that
|
|
# start with low limits keep them. Ramdisk must exist before Emby starts.
|
|
# Docker networks must be connected before watchdogs check container states.
|
|
# fallback.sh goes last — it needs everything else stable to make decisions.
|
|
#
|
|
# Configuration Owns the List
|
|
# ARRAY_START_SCRIPTS in master.conf is the only place scripts are added or
|
|
# removed. This orchestrator never needs to be edited to change what runs —
|
|
# one-shot vs continuous behaviour is auto-detected from the PID after launch.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL SAFEGUARDS
|
|
# ==============================================================================================
|
|
#
|
|
# Root Enforcement
|
|
# Every script launched here requires root.
|
|
#
|
|
# Lock Acquisition
|
|
# acquire_lock prevents duplicate array start launches. Unraid can fire the array
|
|
# start hook more than once, and a second pass would re-launch continuous scripts
|
|
# that are already running.
|
|
#
|
|
# Host Detection
|
|
# detect_hosts() sets MY_ID for notifications.
|
|
#
|
|
# Empty Job List Guard
|
|
# Exits with an error and a notification if ARRAY_START_SCRIPTS is empty. An empty
|
|
# list would silently bring the array up with no ramdisk, no network setup, no
|
|
# watchdogs and no fallback — while reporting a clean start.
|
|
#
|
|
# Executable Auto-Fix
|
|
# Non-executable scripts are chmod +x'd before launch. A permission bit lost to a
|
|
# git checkout or a file copy should not silently disable a boot-time component.
|
|
#
|
|
# Full Path on Failure
|
|
# Failures report the exact resolved path, so a missing script is immediately
|
|
# distinguishable from a script that ran and failed.
|
|
#
|
|
# Failure Notification
|
|
# Any script that fails to launch raises a notification — array start is unattended,
|
|
# so a silent failure here would only surface much later as a missing service.
|
|
#
|
|
# ==============================================================================================
|
|
# CONFIGURATION
|
|
# ==============================================================================================
|
|
#
|
|
# master.conf
|
|
#
|
|
# ARRAY_START_SCRIPTS — ordered list of scripts to launch at array start
|
|
#
|
|
# ==============================================================================================
|
|
# RUNTIME MODES
|
|
# ==============================================================================================
|
|
#
|
|
# array_started.sh
|
|
# Normal launch — called by Varaverk disks_mounted event hook.
|
|
#
|
|
# array_started.sh --dry-run
|
|
# Show what would be launched without launching.
|
|
#
|
|
# array_started.sh --status
|
|
# Show configured scripts and their current state.
|
|
#
|
|
# array_started.sh --log
|
|
# Verbose output per script.
|
|
#
|
|
# ==============================================================================================
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
|
|
source "$ECOSYSTEM_ROOT/load_config.sh"
|
|
|
|
parse_args "$@"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Setup ━━━
|
|
# ==============================================================================================
|
|
if [[ "$EUID" -ne 0 ]]; then
|
|
error "Must be run as root"
|
|
exit 1
|
|
fi
|
|
|
|
|
|
acquire_lock
|
|
|
|
detect_hosts
|
|
|
|
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
|
# from a healthy run. Fail loudly instead of silently doing no work.
|
|
if [[ ${#ARRAY_START_SCRIPTS[@]} -eq 0 ]]; then
|
|
error "ARRAY_START_SCRIPTS is empty — no array start scripts will run"
|
|
error "Check ARRAY_START_SCRIPTS in master.conf"
|
|
notify "array start scripts skipped on $(hostname) ($MY_ID) — ARRAY_START_SCRIPTS is empty" \
|
|
"$(basename "$0" .sh)" "warning"
|
|
exit 1
|
|
fi
|
|
|
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — scripts will not be launched"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Status ━━━
|
|
# ==============================================================================================
|
|
if [[ "$SHOW_STATUS" == true ]]; then
|
|
echo ""
|
|
echo "━━━━━ $ICON_SUMMARY ARRAY START STATUS ━━━━━"
|
|
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
|
echo "$ICON_GEAR Scripts: ${#ARRAY_START_SCRIPTS[@]} configured"
|
|
echo ""
|
|
|
|
for relative_path in "${ARRAY_START_SCRIPTS[@]}"; do
|
|
[[ -z "$relative_path" ]] && continue
|
|
script_path="$ECOSYSTEM_ROOT/$relative_path"
|
|
script_name=$(basename "$script_path")
|
|
|
|
if [[ ! -f "$script_path" ]]; then
|
|
echo " $ICON_ERROR $script_name — FILE NOT FOUND"
|
|
echo " $script_path"
|
|
continue
|
|
fi
|
|
|
|
[[ ! -x "$script_path" ]] && flag=" (not executable — will auto-fix)" || flag=""
|
|
|
|
# Check if currently running
|
|
if pgrep -f "$script_path" >/dev/null 2>&1; then
|
|
RUN_PID=$(pgrep -f "$script_path" | head -1)
|
|
echo " $ICON_RUNNING $script_name — RUNNING (PID $RUN_PID)${flag}"
|
|
else
|
|
echo " $ICON_NOT_RUNNING $script_name — not running${flag}"
|
|
fi
|
|
done
|
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
|
exit 0
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Launch Scripts ━━━
|
|
# ==============================================================================================
|
|
if [[ ${#ARRAY_START_SCRIPTS[@]} -eq 0 ]]; then
|
|
notify "Array started on $LOCAL_SERVER_NAME ($MY_ID) but ARRAY_START_SCRIPTS is empty — boot sequence skipped. Check master.conf." \
|
|
"Array Start" "alert"
|
|
error "ARRAY_START_SCRIPTS is empty — check master.conf"
|
|
exit 1
|
|
fi
|
|
|
|
echo ""
|
|
echo "━━━ $ICON_GEAR Array Start — $MY_ID — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
|
log "Ecosystem root: $ECOSYSTEM_ROOT"
|
|
log "Launching ${#ARRAY_START_SCRIPTS[@]} script(s)..."
|
|
echo ""
|
|
|
|
START=$(date +%s)
|
|
JOB_PASS=()
|
|
JOB_FAIL=()
|
|
|
|
for relative_path in "${ARRAY_START_SCRIPTS[@]}"; do
|
|
[[ -z "$relative_path" ]] && continue
|
|
|
|
SCRIPT_PATH="$ECOSYSTEM_ROOT/$relative_path"
|
|
SCRIPT_NAME=$(basename "$SCRIPT_PATH")
|
|
|
|
# File existence check
|
|
if [[ ! -f "$SCRIPT_PATH" ]]; then
|
|
error "$SCRIPT_NAME — not found"
|
|
error " Expected: $SCRIPT_PATH"
|
|
JOB_FAIL+=("$SCRIPT_NAME")
|
|
continue
|
|
fi
|
|
|
|
# Auto-fix permissions — chmod +x if needed
|
|
if [[ ! -x "$SCRIPT_PATH" ]]; then
|
|
warn "$SCRIPT_NAME — not executable, fixing..."
|
|
chmod +x "$SCRIPT_PATH" || {
|
|
error "$SCRIPT_NAME — chmod +x failed"
|
|
JOB_FAIL+=("$SCRIPT_NAME")
|
|
continue
|
|
}
|
|
fi
|
|
|
|
if [[ "$DRY_RUN" == true ]]; then
|
|
warn "DRY RUN — would launch: $SCRIPT_NAME"
|
|
JOB_PASS+=("$SCRIPT_NAME")
|
|
continue
|
|
fi
|
|
|
|
log "$ICON_START Launching $SCRIPT_NAME..."
|
|
bash "$SCRIPT_PATH" &
|
|
PID=$!
|
|
|
|
# Brief settle — 1s enough to detect immediate failures
|
|
sleep 1
|
|
|
|
if kill -0 "$PID" 2>/dev/null; then
|
|
# Still running → continuous script
|
|
log "$SCRIPT_NAME — running (PID $PID) ✅"
|
|
JOB_PASS+=("$SCRIPT_NAME")
|
|
else
|
|
# Exited — check if one-shot success or failure
|
|
wait "$PID"
|
|
EXIT_CODE=$?
|
|
if [[ "$EXIT_CODE" -eq 0 ]]; then
|
|
echo "$SCRIPT_NAME — completed (one-shot) ✅"
|
|
JOB_PASS+=("$SCRIPT_NAME")
|
|
else
|
|
error "$SCRIPT_NAME — exited with code $EXIT_CODE"
|
|
error " Path: $SCRIPT_PATH"
|
|
JOB_FAIL+=("$SCRIPT_NAME")
|
|
fi
|
|
fi
|
|
|
|
done
|
|
|
|
END=$(date +%s)
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Summary ━━━
|
|
# ==============================================================================================
|
|
echo ""
|
|
echo "━━━━━ $ICON_SUMMARY ARRAY START SUMMARY ━━━━━"
|
|
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
|
echo "$ICON_SUCCESS Launched: ${#JOB_PASS[@]}"
|
|
[[ ${#JOB_FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR Failed: ${#JOB_FAIL[@]} — ${JOB_FAIL[*]}"
|
|
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
|
echo ""
|
|
|
|
# The configured list is the denominator — a script the conf names but that never launched is
|
|
# skipped, not absent, and only shows up if something counts it.
|
|
JOB_COUNT="${#ARRAY_START_SCRIPTS[@]}"
|
|
orchestrator_summary "ARRAY START" "$START" "Array Start"
|
|
exit $? |