Gmer4Lfe
2a062e5140
Every orchestrator invoked its children differently — four near-duplicate run_job() copies, a differently-shaped run_watchdog(), or plain inline bash calls, each with its own take on path resolution, pass/fail naming, and dry-run threading. Extracted one shared run_orch_child() into common.sh so there's a single place to fix or extend this behavior going forward. Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log silently did nothing beyond basic logging on those two. Fixed to $ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no trailing exit, so their exit codes reflected whatever the last command happened to return rather than actual success/failure. transcode_management.sh had no failure notification and no summary at all. Also made transcode_management.sh's two-script pipeline config-driven (TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room to extend it later without editing the orchestrator itself.
229 lines
10 KiB
Bash
Executable File
229 lines
10 KiB
Bash
Executable File
#!/bin/bash
|
|
# ==============================================================================================
|
|
# ============================= Transcode Management ===========================================
|
|
# ==============================================================================================
|
|
#
|
|
# PURPOSE
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Runs TRANSCODE_MANAGEMENT_SCRIPTS in order each cron cycle — this is the
|
|
# single cron entry replacing individual entries for each child script.
|
|
# Schedule: */7 * * * * (every 7 minutes)
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL MODEL
|
|
# ==============================================================================================
|
|
#
|
|
# Order driven by TRANSCODE_MANAGEMENT_SCRIPTS in master.conf.
|
|
# Default: transcode_cleanup.sh → transcode_manager.sh
|
|
#
|
|
# transcode_cleanup.sh
|
|
# Removes aged segment files not open by any process. Uses lsof for O(1)
|
|
# per-file active check. Triggers flip-back to ramdisk after cleanup if
|
|
# the ramdisk usage has recovered.
|
|
#
|
|
# transcode_manager.sh
|
|
# Checks ramdisk usage against thresholds. Flips the symlink between ramdisk
|
|
# and SSD as needed. Writes one entry to TRANSCODE_DAILY_LOG after each run.
|
|
# Shows active Emby sessions with play method.
|
|
#
|
|
# DAILY LOG (written by transcode_manager.sh, not this orchestrator):
|
|
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
|
|
# Trimmed to TRANSCODE_LOG_RETENTION days on each write.
|
|
# Read by sunday_morning_coffee_report.sh and weekly_health_digest.sh.
|
|
#
|
|
# ==============================================================================================
|
|
# DESIGN PRINCIPLES
|
|
# ==============================================================================================
|
|
#
|
|
# Cleanup First, Decide Later
|
|
# Stale segment files from ended sessions inflate the ramdisk usage reading
|
|
# and trigger unnecessary SSD flips even when active sessions would fit on
|
|
# the ramdisk. Cleanup runs first so the manager measures real current usage.
|
|
# Order is config-driven (TRANSCODE_MANAGEMENT_SCRIPTS) but this dependency
|
|
# is real — reordering the array changes what the manager measures.
|
|
#
|
|
# Delegated Logging
|
|
# This orchestrator does not write its own log — transcode_manager.sh owns
|
|
# the TRANSCODE_DAILY_LOG write. One log writer, one format, no duplication.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL SAFEGUARDS
|
|
# ==============================================================================================
|
|
#
|
|
# Root check — mount and docker operations require root
|
|
# acquire_lock — prevents concurrent 7-minute cycles overlapping
|
|
# detect_hosts() — aliases RAMDISK_PATH, TRANSCODE_SSD, RAMDISK_WARN_GB per host
|
|
# --dry-run — passed through to every script in TRANSCODE_MANAGEMENT_SCRIPTS
|
|
# Exit code — worst exit code across all scripts returned to cron
|
|
# notify() — pushed on failure, skipped in --dry-run
|
|
#
|
|
# ==============================================================================================
|
|
# CONFIGURATION
|
|
# ==============================================================================================
|
|
#
|
|
# master.conf
|
|
#
|
|
# TRANSCODE_MANAGEMENT_SCRIPTS — scripts to run, in order
|
|
# TRANSCODE_DAILY_LOG — daily stats log (written by transcode_manager.sh)
|
|
# TRANSCODE_LOG_RETENTION — days to keep (trimmed by manager)
|
|
# TRANSCODE_STATE_FILE — current state (ramdisk_used, flip_count, etc.)
|
|
# TRANSCODE_* threshold vars — see master.conf Transcode Manager section
|
|
#
|
|
# ==============================================================================================
|
|
# RUNTIME MODES
|
|
# ==============================================================================================
|
|
#
|
|
# transcode_management.sh
|
|
# Normal run (every 7 minutes via cron).
|
|
#
|
|
# transcode_management.sh --dry-run
|
|
# Preview without changes (passed to every script in TRANSCODE_MANAGEMENT_SCRIPTS).
|
|
#
|
|
# transcode_management.sh --status
|
|
# Show configuration and current state.
|
|
#
|
|
# transcode_management.sh --log
|
|
# Verbose output from every child script.
|
|
#
|
|
# ==============================================================================================
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
|
|
source "$SCRIPT_DIR/../load_config.sh"
|
|
|
|
parse_args "$@"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Setup ━━━
|
|
# ==============================================================================================
|
|
if [[ "$EUID" -ne 0 ]]; then
|
|
error "Must be run as root"
|
|
exit 1
|
|
fi
|
|
|
|
acquire_lock "wait"
|
|
|
|
if ! command -v docker &>/dev/null; then
|
|
error "Docker command not found"
|
|
exit 1
|
|
fi
|
|
|
|
# detect_hosts() sets MY_ID and aliases all HOST*_TRANSCODE_* vars
|
|
detect_hosts
|
|
|
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing through to child scripts"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Status ━━━
|
|
# ==============================================================================================
|
|
if [[ "$SHOW_STATUS" == true ]]; then
|
|
echo ""
|
|
echo "━━━━━ $ICON_SUMMARY TRANSCODE MANAGEMENT STATUS ━━━━━"
|
|
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
|
echo "$ICON_RAM Ramdisk: $RAMDISK_PATH ($RAMDISK_SIZE)"
|
|
echo "$ICON_DISK SSD fallback: $TRANSCODE_SSD"
|
|
echo "$ICON_LINK Symlink: $TRANSCODE_LINK"
|
|
echo "$ICON_TIME Schedule: every 7 minutes"
|
|
echo ""
|
|
echo "━━━ Child Scripts ━━━"
|
|
for _entry in "${TRANSCODE_MANAGEMENT_SCRIPTS[@]}"; do
|
|
_script_path="$SCRIPT_DIR/../$_entry"
|
|
_script_name=$(basename "$_entry")
|
|
if [[ -f "$_script_path" ]]; then
|
|
echo " $ICON_SUCCESS $_script_name — found"
|
|
else
|
|
echo " $ICON_ERROR $_script_name — NOT FOUND at $_script_path"
|
|
fi
|
|
done
|
|
echo ""
|
|
echo "━━━ Daily Log ━━━"
|
|
if [[ -f "${TRANSCODE_DAILY_LOG:-}" ]] && [[ -s "$TRANSCODE_DAILY_LOG" ]]; then
|
|
ENTRY_COUNT=$(wc -l < "$TRANSCODE_DAILY_LOG")
|
|
OLDEST=$(awk -F'|' 'NR==1{print $1}' "$TRANSCODE_DAILY_LOG")
|
|
NEWEST=$(awk -F'|' 'END{print $1}' "$TRANSCODE_DAILY_LOG")
|
|
echo " $ICON_SUCCESS $TRANSCODE_DAILY_LOG ($ENTRY_COUNT entries, $OLDEST → $NEWEST)"
|
|
else
|
|
echo " $ICON_SKIP $TRANSCODE_DAILY_LOG — no data yet"
|
|
fi
|
|
echo ""
|
|
echo "━━━ Current State ━━━"
|
|
if [[ -f "${TRANSCODE_STATE_FILE:-/tmp/transcode_state.db}" ]]; then
|
|
while IFS='=' read -r key val; do
|
|
[[ -n "$key" ]] && echo " $key = $val"
|
|
done < "${TRANSCODE_STATE_FILE:-/tmp/transcode_state.db}"
|
|
else
|
|
echo " State file not found (ramdisk_setup.sh creates it at array start)"
|
|
fi
|
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
|
exit 0
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Validate Child Scripts ━━━
|
|
# ==============================================================================================
|
|
for _entry in "${TRANSCODE_MANAGEMENT_SCRIPTS[@]}"; do
|
|
_script_path="$SCRIPT_DIR/../$_entry"
|
|
if [[ ! -f "$_script_path" ]]; then
|
|
error "$(basename "$_entry") not found: $_script_path"
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Pre-run State Snapshot ━━━
|
|
# ==============================================================================================
|
|
if mountpoint -q "${RAMDISK_PATH:-}" 2>/dev/null; then
|
|
_rd_used=$(df "$RAMDISK_PATH" --output=used 2>/dev/null | tail -1 | tr -d ' ')
|
|
_rd_avail=$(df "$RAMDISK_PATH" --output=avail 2>/dev/null | tail -1 | tr -d ' ')
|
|
_rd_used_gb=$(awk "BEGIN {printf \"%.2f\", ${_rd_used:-0}/1048576}")
|
|
_rd_avail_gb=$(awk "BEGIN {printf \"%.2f\", ${_rd_avail:-0}/1048576}")
|
|
_rd_target=$(readlink "${TRANSCODE_LINK:-}" 2>/dev/null || echo "unknown")
|
|
log "$ICON_RAM Ramdisk: ${_rd_used_gb}GB used / ${_rd_avail_gb}GB avail — symlink → ${_rd_target##*/}"
|
|
else
|
|
log "$ICON_RAM Ramdisk: not mounted"
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Run Scripts ━━━
|
|
# ==============================================================================================
|
|
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each run — no flag here
|
|
# suppresses that; it owns the log write for this cycle regardless of position ✅
|
|
DRY_FLAG=""
|
|
[[ "$DRY_RUN" == true ]] && DRY_FLAG="--dry-run"
|
|
|
|
WORST_EXIT=0
|
|
PASS_COUNT=0
|
|
FAIL_NAMES=()
|
|
for _entry in "${TRANSCODE_MANAGEMENT_SCRIPTS[@]}"; do
|
|
_script_path="$SCRIPT_DIR/../$_entry"
|
|
_script_name=$(basename "$_entry" .sh)
|
|
_start=$(date +%s)
|
|
bash "$_script_path" $DRY_FLAG
|
|
_exit=$?
|
|
log "$_script_name: $(format_duration $(( $(date +%s) - _start ))) (exit $_exit)"
|
|
if [[ "$_exit" -ne 0 ]]; then
|
|
WORST_EXIT=1
|
|
FAIL_NAMES+=("$_script_name")
|
|
else
|
|
PASS_COUNT=$(( PASS_COUNT + 1 ))
|
|
fi
|
|
done
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Summary — minimal one-liner by default (7-min cadence — keep it quiet when healthy) ━━━
|
|
# ==============================================================================================
|
|
if [[ "$WORST_EXIT" -eq 0 ]]; then
|
|
echo "$ICON_SUCCESS Transcode cycle — $PASS_COUNT/${#TRANSCODE_MANAGEMENT_SCRIPTS[@]} passed"
|
|
else
|
|
error "Transcode cycle — failed: ${FAIL_NAMES[*]}"
|
|
if [[ "$DRY_RUN" != true ]]; then
|
|
notify "Transcode management failure on $(hostname) ($MY_ID) — ${FAIL_NAMES[*]}" \
|
|
"Transcode Management" "warning"
|
|
fi
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Exit ━━━
|
|
# ==============================================================================================
|
|
# Return worst exit code — caller knows if any script failed
|
|
exit "$WORST_EXIT" |