Gmer4Lfe
69189bbf18
Same audit as the orchestrator standardization pass (2a062e5), extended to
every remaining script. Found the same class of bug independently recurring:
ramdisk_stop.sh checked $LOG (nothing assigns it, should be $ENABLE_LOGGING),
partnership_onboard.sh checked $LOG_MODE (same issue), emby_session_report.sh
checked $TRANSCODE_PCT which was never computed so the high-transcode alert
could never fire, and storage_migrate.sh never called detect_hosts() so
$MY_ID was empty, silently breaking the post-migration host*.conf update.
partnership_manager.sh used `local` at top-level script scope (invalid outside
a function) and had two master.conf path references missing "Configurations/".
Along the way: several scripts (share_setup.sh, conf_sync.sh,
downloaders_reset.sh, transcode_cleanup.sh, transcode_manager.sh,
remote_arr_cache_writer.sh, upgrade_webhook_handler.sh) had no explicit
trailing exit code, so they always reported success regardless of real
failures. play_state_sync.sh was missing the partnership gate its own header
documented, so remote play-state sync ran even with PARTNERSHIP_ENABLED=false;
it also always exited 0 on sync errors. arr_profile_enforcer.sh and
webhook_setup.sh hand-rolled their own flag parsing instead of common.sh's
parse_args, so --log silently did nothing on either.
system_watchdog.sh was itself an un-standardized mini-orchestrator — converted
to the shared run_orch_child()/JOB_PASS/JOB_FAIL pattern, added the missing
failure notification, and fixed dry-run to pass --dry-run down to children
instead of skipping them outright. Also fixed a stale webgui_watchdog.sh path
in master.conf.template that would break system_watchdog.sh on any fresh
install.
Closed a sibling-drift gap: radarr_cleanup.sh and sonarr_cleanup.sh were
missing lidarr_cleanup.sh's tracked-count percentage-drop safety gate and its
"not configured on this host, skip cleanly" guard — both now match Lidarr's
7-gate model.
144 lines
5.8 KiB
Bash
Executable File
144 lines
5.8 KiB
Bash
Executable File
#!/bin/bash
|
|
# ==============================================================================================
|
|
# ================================= System Watchdog ============================================
|
|
# ==============================================================================================
|
|
#
|
|
# PURPOSE
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Thin orchestrator — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially.
|
|
# Called by watchdog_orchestrator.sh each cycle. Covers system component health:
|
|
# storage pool growth, runaway logs, WebGUI availability, and network connectivity.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL MODEL
|
|
# ==============================================================================================
|
|
#
|
|
# Driven by SYSTEM_WATCHDOG_SCRIPTS in master.conf — add, remove, or reorder there.
|
|
# Default: storage_watchdog → webgui_watchdog → network_watchdog
|
|
#
|
|
# All scripts run in the foreground. Each must complete before the next starts.
|
|
# A failed script is logged but does not prevent remaining scripts from running.
|
|
#
|
|
# ==============================================================================================
|
|
# DESIGN PRINCIPLES
|
|
# ==============================================================================================
|
|
#
|
|
# Configuration Owns the List
|
|
# SYSTEM_WATCHDOG_SCRIPTS in master.conf is the only place scripts are added
|
|
# or removed. This orchestrator never needs to be edited to change what runs.
|
|
#
|
|
# Non-Fatal Steps
|
|
# A failed watchdog step is logged and noted in the summary, but the remaining
|
|
# steps still execute. Partial coverage is better than a halted watchdog chain.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL SAFEGUARDS
|
|
# ==============================================================================================
|
|
#
|
|
# Root check — child scripts require root
|
|
# acquire_lock — prevents concurrent system watchdog runs
|
|
# detect_hosts() — MY_ID in notifications and logs
|
|
# Non-fatal steps — a failed step is logged; remaining steps still run
|
|
#
|
|
# ==============================================================================================
|
|
# CONFIGURATION
|
|
# ==============================================================================================
|
|
#
|
|
# master.conf
|
|
#
|
|
# SYSTEM_WATCHDOG_SCRIPTS — ordered list of system component watchdog scripts to run
|
|
#
|
|
# ==============================================================================================
|
|
# RUNTIME MODES
|
|
# ==============================================================================================
|
|
#
|
|
# system_watchdog.sh
|
|
# Run all system component watchdogs.
|
|
#
|
|
# system_watchdog.sh --dry-run
|
|
# Passes --dry-run to each sub-script — no changes made.
|
|
#
|
|
# system_watchdog.sh --status
|
|
# Show configured scripts and exit.
|
|
#
|
|
# system_watchdog.sh --log
|
|
# Passes --log to each sub-script for verbose output.
|
|
#
|
|
# ==============================================================================================
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
|
|
source "$ECOSYSTEM_ROOT/load_config.sh"
|
|
|
|
parse_args "$@"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Setup ━━━
|
|
# ==============================================================================================
|
|
if [[ "$EUID" -ne 0 ]]; then
|
|
error "Must be run as root"
|
|
exit 1
|
|
fi
|
|
|
|
|
|
acquire_lock
|
|
|
|
detect_hosts
|
|
|
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing --dry-run to all sub-scripts"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Status ━━━
|
|
# ==============================================================================================
|
|
if [[ "$SHOW_STATUS" == true ]]; then
|
|
echo ""
|
|
echo "━━━━━ $ICON_SUMMARY SYSTEM WATCHDOG STATUS ━━━━━"
|
|
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
|
echo "$ICON_GEAR Scripts: ${#SYSTEM_WATCHDOG_SCRIPTS[@]} configured"
|
|
echo ""
|
|
|
|
for entry in "${SYSTEM_WATCHDOG_SCRIPTS[@]}"; do
|
|
[[ -z "$entry" ]] && continue
|
|
read -r -a parts <<< "$entry"
|
|
script_path="$ECOSYSTEM_ROOT/${parts[0]}"
|
|
script_name=$(basename "${parts[0]}")
|
|
if [[ ! -f "$script_path" ]]; then
|
|
echo " $ICON_ERROR $script_name — FILE NOT FOUND"
|
|
else
|
|
echo " $ICON_GEAR $script_name"
|
|
fi
|
|
done
|
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
|
exit 0
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Run Sequence ━━━
|
|
# ==============================================================================================
|
|
echo "━━━ $ICON_SHIELD System Watchdog — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
|
|
|
START=$(date +%s)
|
|
JOB_PASS=()
|
|
JOB_FAIL=()
|
|
|
|
for entry in "${SYSTEM_WATCHDOG_SCRIPTS[@]}"; do
|
|
[[ -z "$entry" ]] && continue
|
|
run_orch_child "$entry"
|
|
done
|
|
|
|
END=$(date +%s)
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Summary ━━━
|
|
# ==============================================================================================
|
|
log "System watchdog — ${#JOB_PASS[@]}/${#SYSTEM_WATCHDOG_SCRIPTS[@]} passed — $(format_duration $(( END - START )))"
|
|
|
|
if [[ ${#JOB_FAIL[@]} -gt 0 ]]; then
|
|
notify "System watchdog failed on $(hostname) ($MY_ID) — ${JOB_FAIL[*]}" \
|
|
"System Watchdog" "warning"
|
|
exit 1
|
|
fi
|
|
|
|
exit 0
|