Gmer4Lfe
d5cf3db2ec
A gated-off section left nothing failed, so the weekly could run for hours and report "all complete" beside "0 shares synced"; skipped is now derived from what was expected rather than self-reported, and the verdict degrades to PARTIAL instead of flattering.
238 lines
11 KiB
Bash
Executable File
238 lines
11 KiB
Bash
Executable File
#!/bin/bash
|
|
# ==============================================================================================
|
|
# ============================= Transcode Management ===========================================
|
|
# ==============================================================================================
|
|
#
|
|
# PURPOSE
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Runs TRANSCODE_MANAGEMENT_SCRIPTS in order each cron cycle — this is the
|
|
# single cron entry replacing individual entries for each child script.
|
|
# Schedule: */7 * * * * (every 7 minutes)
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL MODEL
|
|
# ==============================================================================================
|
|
#
|
|
# Order driven by TRANSCODE_MANAGEMENT_SCRIPTS in master.conf.
|
|
# Default: transcode_cleanup.sh → transcode_manager.sh
|
|
#
|
|
# transcode_cleanup.sh
|
|
# Removes aged segment files not open by any process. Uses lsof for O(1)
|
|
# per-file active check. Triggers flip-back to ramdisk after cleanup if
|
|
# the ramdisk usage has recovered.
|
|
#
|
|
# transcode_manager.sh
|
|
# Checks ramdisk usage against thresholds. Flips the symlink between ramdisk
|
|
# and SSD as needed. Writes one entry to TRANSCODE_DAILY_LOG after each run.
|
|
# Shows active Emby sessions with play method.
|
|
#
|
|
# DAILY LOG (written by transcode_manager.sh, not this orchestrator):
|
|
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
|
|
# Trimmed to TRANSCODE_LOG_RETENTION days on each write.
|
|
# Read by sunday_morning_coffee_report.sh and weekly_health_digest.sh.
|
|
#
|
|
# ==============================================================================================
|
|
# DESIGN PRINCIPLES
|
|
# ==============================================================================================
|
|
#
|
|
# Cleanup First, Decide Later
|
|
# Stale segment files from ended sessions inflate the ramdisk usage reading
|
|
# and trigger unnecessary SSD flips even when active sessions would fit on
|
|
# the ramdisk. Cleanup runs first so the manager measures real current usage.
|
|
# Order is config-driven (TRANSCODE_MANAGEMENT_SCRIPTS) but this dependency
|
|
# is real — reordering the array changes what the manager measures.
|
|
#
|
|
# Delegated Logging
|
|
# This orchestrator does not write its own log — transcode_manager.sh owns
|
|
# the TRANSCODE_DAILY_LOG write. One log writer, one format, no duplication.
|
|
#
|
|
# ==============================================================================================
|
|
# OPERATIONAL SAFEGUARDS
|
|
# ==============================================================================================
|
|
#
|
|
# Root Enforcement
|
|
# Mount and docker operations require root.
|
|
#
|
|
# Lock Acquisition
|
|
# acquire_lock prevents concurrent 7-minute cycles overlapping. Cleanup and the manager
|
|
# both touch the same ramdisk, and two cycles at once could have one deleting files
|
|
# while the other is measuring usage to decide whether to flip.
|
|
#
|
|
# Host Detection
|
|
# detect_hosts() aliases RAMDISK_PATH, TRANSCODE_SSD and RAMDISK_WARN_GB per host.
|
|
#
|
|
# Empty Job List Guard
|
|
# Exits with an error and a notification if TRANSCODE_MANAGEMENT_SCRIPTS is empty —
|
|
# without it the ramdisk would silently stop being cleaned or flipped, and the first
|
|
# symptom would be a full ramdisk stalling playback.
|
|
#
|
|
# Ordering Is Load-Bearing
|
|
# Cleanup runs before the manager so the manager measures real active-session usage
|
|
# rather than usage inflated by stale files. Reversing them would trigger flips that
|
|
# a cleanup two seconds later would have made unnecessary.
|
|
#
|
|
# Dry Run Propagation
|
|
# --dry-run is passed through to every script in TRANSCODE_MANAGEMENT_SCRIPTS.
|
|
#
|
|
# Any-Failure Exit Code
|
|
# Exits 1 if any child failed, 0 otherwise — the individual exit codes are not
|
|
# propagated, only whether anything failed. A failure in an early child is therefore
|
|
# never masked by a later success.
|
|
#
|
|
# Notification Contract
|
|
# notify() fires on failure and is skipped in --dry-run.
|
|
#
|
|
# ==============================================================================================
|
|
# CONFIGURATION
|
|
# ==============================================================================================
|
|
#
|
|
# master.conf
|
|
#
|
|
# TRANSCODE_MANAGEMENT_SCRIPTS — scripts to run, in order
|
|
# TRANSCODE_DAILY_LOG — daily stats log (written by transcode_manager.sh)
|
|
# TRANSCODE_LOG_RETENTION — days to keep (trimmed by manager)
|
|
# TRANSCODE_STATE_FILE — current state (ramdisk_used, flip_count, etc.)
|
|
# TRANSCODE_* threshold vars — see master.conf Transcode Manager section
|
|
#
|
|
# ==============================================================================================
|
|
# RUNTIME MODES
|
|
# ==============================================================================================
|
|
#
|
|
# transcode_management.sh
|
|
# Normal run (every 7 minutes via cron).
|
|
#
|
|
# transcode_management.sh --dry-run
|
|
# Preview without changes (passed to every script in TRANSCODE_MANAGEMENT_SCRIPTS).
|
|
#
|
|
# transcode_management.sh --status
|
|
# Show configuration and current state.
|
|
#
|
|
# transcode_management.sh --log
|
|
# Verbose output from every child script.
|
|
#
|
|
# ==============================================================================================
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
|
|
source "$ECOSYSTEM_ROOT/load_config.sh"
|
|
|
|
parse_args "$@"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Setup ━━━
|
|
# ==============================================================================================
|
|
if [[ "$EUID" -ne 0 ]]; then
|
|
error "Must be run as root"
|
|
exit 1
|
|
fi
|
|
|
|
acquire_lock "wait"
|
|
|
|
if ! command -v docker &>/dev/null; then
|
|
error "Docker command not found"
|
|
exit 1
|
|
fi
|
|
|
|
# detect_hosts() sets MY_ID and aliases all HOST*_TRANSCODE_* vars
|
|
detect_hosts
|
|
|
|
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
|
# from a healthy run. Fail loudly instead of silently doing no work.
|
|
# This orchestrator never timed itself, so its summary could not report a duration. Set before
|
|
# any work so the figure means the cycle, not the tail of it.
|
|
CYCLE_START=$(date +%s)
|
|
|
|
if [[ ${#TRANSCODE_MANAGEMENT_SCRIPTS[@]} -eq 0 ]]; then
|
|
error "TRANSCODE_MANAGEMENT_SCRIPTS is empty — no transcode management scripts will run"
|
|
error "Check TRANSCODE_MANAGEMENT_SCRIPTS in master.conf"
|
|
notify "transcode management scripts skipped on $(hostname) ($MY_ID) — TRANSCODE_MANAGEMENT_SCRIPTS is empty" \
|
|
"$(basename "$0" .sh)" "warning"
|
|
exit 1
|
|
fi
|
|
|
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing through to child scripts"
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Status ━━━
|
|
# ==============================================================================================
|
|
if [[ "$SHOW_STATUS" == true ]]; then
|
|
echo ""
|
|
echo "━━━━━ $ICON_SUMMARY TRANSCODE MANAGEMENT STATUS ━━━━━"
|
|
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
|
echo "$ICON_RAM Ramdisk: $RAMDISK_PATH ($RAMDISK_SIZE)"
|
|
echo "$ICON_DISK SSD fallback: $TRANSCODE_SSD"
|
|
echo "$ICON_LINK Symlink: $TRANSCODE_LINK"
|
|
echo "$ICON_TIME Schedule: every 7 minutes"
|
|
echo ""
|
|
echo "━━━ Child Scripts ━━━"
|
|
for _entry in "${TRANSCODE_MANAGEMENT_SCRIPTS[@]}"; do
|
|
_script_path="$SCRIPT_DIR/../$_entry"
|
|
_script_name=$(basename "$_entry")
|
|
if [[ -f "$_script_path" ]]; then
|
|
echo " $ICON_SUCCESS $_script_name — found"
|
|
else
|
|
echo " $ICON_ERROR $_script_name — NOT FOUND at $_script_path"
|
|
fi
|
|
done
|
|
echo ""
|
|
echo "━━━ Daily Log ━━━"
|
|
if [[ -f "${TRANSCODE_DAILY_LOG:-}" ]] && [[ -s "$TRANSCODE_DAILY_LOG" ]]; then
|
|
ENTRY_COUNT=$(wc -l < "$TRANSCODE_DAILY_LOG")
|
|
OLDEST=$(awk -F'|' 'NR==1{print $1}' "$TRANSCODE_DAILY_LOG")
|
|
NEWEST=$(awk -F'|' 'END{print $1}' "$TRANSCODE_DAILY_LOG")
|
|
echo " $ICON_SUCCESS $TRANSCODE_DAILY_LOG ($ENTRY_COUNT entries, $OLDEST → $NEWEST)"
|
|
else
|
|
echo " $ICON_SKIP $TRANSCODE_DAILY_LOG — no data yet"
|
|
fi
|
|
echo ""
|
|
echo "━━━ Current State ━━━"
|
|
if [[ -f "${TRANSCODE_STATE_FILE:-/tmp/transcode_state.db}" ]]; then
|
|
while IFS='=' read -r key val; do
|
|
[[ -n "$key" ]] && echo " $key = $val"
|
|
done < "${TRANSCODE_STATE_FILE:-/tmp/transcode_state.db}"
|
|
else
|
|
echo " State file not found (ramdisk_setup.sh creates it at array start)"
|
|
fi
|
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
|
exit 0
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Pre-run State Snapshot ━━━
|
|
# ==============================================================================================
|
|
if mountpoint -q "${RAMDISK_PATH:-}" 2>/dev/null; then
|
|
_rd_used=$(df "$RAMDISK_PATH" --output=used 2>/dev/null | tail -1 | tr -d ' ')
|
|
_rd_avail=$(df "$RAMDISK_PATH" --output=avail 2>/dev/null | tail -1 | tr -d ' ')
|
|
_rd_used_gb=$(awk "BEGIN {printf \"%.2f\", ${_rd_used:-0}/1048576}")
|
|
_rd_avail_gb=$(awk "BEGIN {printf \"%.2f\", ${_rd_avail:-0}/1048576}")
|
|
_rd_target=$(readlink "${TRANSCODE_LINK:-}" 2>/dev/null || echo "unknown")
|
|
log "$ICON_RAM Ramdisk: ${_rd_used_gb}GB used / ${_rd_avail_gb}GB avail — symlink → ${_rd_target##*/}"
|
|
else
|
|
log "$ICON_RAM Ramdisk: not mounted"
|
|
fi
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Run Scripts ━━━
|
|
# ==============================================================================================
|
|
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each run — no flag here
|
|
# suppresses that; it owns the log write for this cycle regardless of position ✅
|
|
JOB_PASS=()
|
|
JOB_FAIL=()
|
|
for _entry in "${TRANSCODE_MANAGEMENT_SCRIPTS[@]}"; do
|
|
[[ -z "$_entry" ]] && continue
|
|
run_orch_child "$_entry"
|
|
done
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Summary — minimal one-liner by default (7-min cadence — keep it quiet when healthy) ━━━
|
|
# ==============================================================================================
|
|
# Quiet by default — 7-min cadence. Anything failed or skipped expands on its own.
|
|
JOB_COUNT="${#TRANSCODE_MANAGEMENT_SCRIPTS[@]}"
|
|
orchestrator_summary "TRANSCODE CYCLE" "${CYCLE_START:-$(date +%s)}" "Transcode Management" quiet
|
|
|
|
# ==============================================================================================
|
|
# ━━━ Exit ━━━
|
|
# ==============================================================================================
|
|
[[ "${#JOB_FAIL[@]}" -gt 0 ]] && exit 1
|
|
exit 0 |