massive update. Master conf split, now modular with a load sceriprt to drive all configs to scripts. with unraid scpecific safeguard tests , and improved standardized ux. including dynamic host detect, who am i who else it there. EVERY SINGLE SCRIPT UPDATED. DEBATING THAT THIS IS ACUALLY V2
This commit is contained in:
@@ -1,132 +1,127 @@
|
||||
#!/bin/bash
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# --------------------------------- Transcode Management ---------------------------------------
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Runs transcode cleanup then transcode manager in the correct order every cycle.
|
||||
# Cleanup runs first — clears stale files so manager sees accurate usage.
|
||||
# Manager runs after — threshold decisions based on real current usage post-cleanup.
|
||||
#
|
||||
# Running cleanup before manager prevents unnecessary SSD flips caused by stale
|
||||
# segment files from ended sessions inflating the ramdisk usage reading.
|
||||
#
|
||||
# Also tracks daily transcode statistics to a bounded log for weekly_health_digest.sh:
|
||||
# Peak ramdisk usage per day
|
||||
# Total flip count per day
|
||||
# Ramdisk vs SSD session ratio
|
||||
# Files cleaned per day
|
||||
#
|
||||
# Scheduled as: */3 * * * * (every 3 minutes)
|
||||
# ==============================================================================================
|
||||
# ============================= Transcode Management ===========================================
|
||||
# ==============================================================================================
|
||||
# Orchestrator — runs transcode_cleanup.sh then transcode_manager.sh in the correct order.
|
||||
# Replace individual transcode_manager and transcode_cleanup cron entries with this.
|
||||
# Schedule: */3 * * * * (every 3 minutes via User Scripts plugin)
|
||||
#
|
||||
# All configuration in Master.conf under Transcode Manager section.
|
||||
# Supports --dry-run — passes through to both child scripts.
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ── WHY CLEANUP BEFORE MANAGER ────────────────────────────────────────────────────────────────
|
||||
# Cleanup runs first — removes stale segment files from ended sessions.
|
||||
# Manager runs after — threshold decisions based on real current usage post-cleanup.
|
||||
# Without this order, stale files inflate the ramdisk usage reading and trigger
|
||||
# unnecessary SSD flips even when active sessions would fit on the ramdisk.
|
||||
#
|
||||
# ── WHAT EACH SCRIPT DOES ─────────────────────────────────────────────────────────────────────
|
||||
# transcode_cleanup.sh — removes aged segment files not open by any process
|
||||
# uses lsof for O(1) per-file active check (never per-file lsof)
|
||||
# also triggers flip-back to ramdisk after cleanup if recovered ✅
|
||||
#
|
||||
# transcode_manager.sh — checks ramdisk usage against thresholds
|
||||
# flips symlink between ramdisk and SSD as needed
|
||||
# writes one entry to TRANSCODE_DAILY_LOG after each run
|
||||
# shows active Emby sessions with play method
|
||||
#
|
||||
# ── DAILY LOG ─────────────────────────────────────────────────────────────────────────────────
|
||||
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each run:
|
||||
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
|
||||
# This orchestrator does NOT write its own log — manager handles it ✅
|
||||
# Log trimmed to TRANSCODE_LOG_RETENTION days by manager on each write.
|
||||
# Read by sunday_morning_coffee_report.sh and weekly_health_digest.sh.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases RAMDISK_PATH, TRANSCODE_SSD, RAMDISK_WARN_GB etc.
|
||||
# Each server manages its own transcode location independently.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — mount and docker operations require root
|
||||
# acquire_lock — prevents concurrent 3-minute cycles overlapping
|
||||
# detect_hosts() — correct paths per host
|
||||
# --dry-run — passed through to both child scripts
|
||||
# Exit code — worst exit code of both scripts returned
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# TRANSCODE_DAILY_LOG — daily stats log (written by transcode_manager.sh)
|
||||
# TRANSCODE_LOG_RETENTION — days to keep (trimmed by manager)
|
||||
# TRANSCODE_STATE_FILE — current state (ramdisk_used, flip_count etc.)
|
||||
# All TRANSCODE_* threshold vars — see master.conf Transcode Manager section
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# transcode_management.sh — normal run (every 3 minutes via cron)
|
||||
# transcode_management.sh --dry-run — preview without changes (passed to children)
|
||||
# transcode_management.sh --status — show configuration and current state
|
||||
# transcode_management.sh --log — verbose output from both child scripts
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# State and log files
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db"
|
||||
TRANSCODE_STATE_FILE="/tmp/transcode_state.db"
|
||||
CLEANUP_SCRIPT="$SCRIPT_DIR/../Transcodes/transcode_cleanup.sh"
|
||||
MANAGER_SCRIPT="$SCRIPT_DIR/../Transcodes/transcode_manager.sh"
|
||||
|
||||
# Bounded log — keeps last 90 days
|
||||
TRANSCODE_LOG_RETENTION=90
|
||||
|
||||
touch "$TRANSCODE_DAILY_LOG" 2>/dev/null
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_GEAR Setup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing to child scripts"
|
||||
acquire_lock "wait"
|
||||
|
||||
acquire_lock
|
||||
# detect_hosts() sets MY_ID and aliases all HOST*_TRANSCODE_* vars
|
||||
detect_hosts
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# HELPERS
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing through to child scripts"
|
||||
|
||||
# Read a value from transcode state file
|
||||
state_get() {
|
||||
grep "^${1}=" "$TRANSCODE_STATE_FILE" 2>/dev/null | cut -d= -f2
|
||||
}
|
||||
|
||||
# Get today's date key
|
||||
today() {
|
||||
date '+%Y-%m-%d'
|
||||
}
|
||||
|
||||
# Update daily log entry for today
|
||||
# Format: YYYY-MM-DD|peak_gb|flip_count|ram_sessions|ssd_sessions|files_cleaned
|
||||
update_daily_log() {
|
||||
local peak_gb="$1"
|
||||
local flips="$2"
|
||||
local ram_sessions="$3"
|
||||
local ssd_sessions="$4"
|
||||
local files_cleaned="$5"
|
||||
local today_key
|
||||
today_key=$(today)
|
||||
|
||||
local existing
|
||||
existing=$(grep "^${today_key}|" "$TRANSCODE_DAILY_LOG" 2>/dev/null)
|
||||
|
||||
if [[ -z "$existing" ]]; then
|
||||
# New entry for today
|
||||
echo "${today_key}|${peak_gb}|${flips}|${ram_sessions}|${ssd_sessions}|${files_cleaned}" \
|
||||
>> "$TRANSCODE_DAILY_LOG"
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY TRANSCODE MANAGEMENT STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_RAM Ramdisk: $RAMDISK_PATH ($RAMDISK_SIZE)"
|
||||
echo "$ICON_DISK SSD fallback: $TRANSCODE_SSD"
|
||||
echo "$ICON_LINK Symlink: $TRANSCODE_LINK"
|
||||
echo "$ICON_TIME Schedule: every 3 minutes"
|
||||
echo ""
|
||||
echo "━━━ Child Scripts ━━━"
|
||||
[[ -f "$CLEANUP_SCRIPT" ]] && \
|
||||
echo " $ICON_SUCCESS transcode_cleanup.sh — found" || \
|
||||
echo " $ICON_ERROR transcode_cleanup.sh — NOT FOUND at $CLEANUP_SCRIPT"
|
||||
[[ -f "$MANAGER_SCRIPT" ]] && \
|
||||
echo " $ICON_SUCCESS transcode_manager.sh — found" || \
|
||||
echo " $ICON_ERROR transcode_manager.sh — NOT FOUND at $MANAGER_SCRIPT"
|
||||
echo ""
|
||||
echo "━━━ Daily Log ━━━"
|
||||
if [[ -f "${TRANSCODE_DAILY_LOG:-}" ]] && [[ -s "$TRANSCODE_DAILY_LOG" ]]; then
|
||||
ENTRY_COUNT=$(wc -l < "$TRANSCODE_DAILY_LOG")
|
||||
OLDEST=$(awk -F'|' 'NR==1{print $1}' "$TRANSCODE_DAILY_LOG")
|
||||
NEWEST=$(awk -F'|' 'END{print $1}' "$TRANSCODE_DAILY_LOG")
|
||||
echo " $ICON_SUCCESS $TRANSCODE_DAILY_LOG ($ENTRY_COUNT entries, $OLDEST → $NEWEST)"
|
||||
else
|
||||
# Update existing — keep highest peak, accumulate flips, sessions, files
|
||||
local old_peak old_flips old_ram old_ssd old_files
|
||||
old_peak=$(echo "$existing" | cut -d'|' -f2)
|
||||
old_flips=$(echo "$existing" | cut -d'|' -f3)
|
||||
old_ram=$(echo "$existing" | cut -d'|' -f4)
|
||||
old_ssd=$(echo "$existing" | cut -d'|' -f5)
|
||||
old_files=$(echo "$existing" | cut -d'|' -f6)
|
||||
|
||||
# Peak — keep highest
|
||||
local new_peak
|
||||
new_peak=$(awk "BEGIN {print ($peak_gb > $old_peak) ? $peak_gb : $old_peak}")
|
||||
|
||||
# Accumulate
|
||||
local new_flips=$(( old_flips + flips ))
|
||||
local new_ram=$(( old_ram + ram_sessions ))
|
||||
local new_ssd=$(( old_ssd + ssd_sessions ))
|
||||
local new_files=$(( old_files + files_cleaned ))
|
||||
|
||||
# Replace line
|
||||
sed -i "s|^${today_key}|.*|${today_key}|${new_peak}|${new_flips}|${new_ram}|${new_ssd}|${new_files}|" \
|
||||
"$TRANSCODE_DAILY_LOG" 2>/dev/null || {
|
||||
# sed replacement failed — remove and re-add
|
||||
sed -i "/^${today_key}|/d" "$TRANSCODE_DAILY_LOG"
|
||||
echo "${today_key}|${new_peak}|${new_flips}|${new_ram}|${new_ssd}|${new_files}" \
|
||||
>> "$TRANSCODE_DAILY_LOG"
|
||||
}
|
||||
echo " $ICON_SKIP $TRANSCODE_DAILY_LOG — no data yet"
|
||||
fi
|
||||
echo ""
|
||||
echo "━━━ Current State ━━━"
|
||||
if [[ -f "${TRANSCODE_STATE_FILE:-/tmp/transcode_state.db}" ]]; then
|
||||
while IFS='=' read -r key val; do
|
||||
[[ -n "$key" ]] && echo " $key = $val"
|
||||
done < "${TRANSCODE_STATE_FILE:-/tmp/transcode_state.db}"
|
||||
else
|
||||
echo " State file not found (ramdisk_setup.sh creates it at array start)"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Purge old entries beyond retention
|
||||
local cutoff
|
||||
cutoff=$(date -d "${TRANSCODE_LOG_RETENTION} days ago" '+%Y-%m-%d')
|
||||
awk -F'|' -v cutoff="$cutoff" '$1 >= cutoff' \
|
||||
"$TRANSCODE_DAILY_LOG" > "${TRANSCODE_DAILY_LOG}.tmp" && \
|
||||
mv "${TRANSCODE_DAILY_LOG}.tmp" "$TRANSCODE_DAILY_LOG"
|
||||
}
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SYNC Run Cleanup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
CLEANUP_SCRIPT="$SCRIPT_DIR/../Transcodes/transcode_cleanup.sh"
|
||||
MANAGER_SCRIPT="$SCRIPT_DIR/../Transcodes/transcode_manager.sh"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Validate Child Scripts ━━━
|
||||
# ==============================================================================================
|
||||
if [[ ! -f "$CLEANUP_SCRIPT" ]]; then
|
||||
error "transcode_cleanup.sh not found: $CLEANUP_SCRIPT"
|
||||
exit 1
|
||||
@@ -137,50 +132,26 @@ if [[ ! -f "$MANAGER_SCRIPT" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Capture cleanup output for file count
|
||||
CLEANUP_OUTPUT=$(bash "$CLEANUP_SCRIPT" $([[ "$DRY_RUN" == true ]] && echo "--dry-run") 2>&1)
|
||||
# ==============================================================================================
|
||||
# ━━━ Run Cleanup ━━━
|
||||
# ==============================================================================================
|
||||
DRY_FLAG=""
|
||||
[[ "$DRY_RUN" == true ]] && DRY_FLAG="--dry-run"
|
||||
|
||||
bash "$CLEANUP_SCRIPT" $DRY_FLAG
|
||||
CLEANUP_EXIT=$?
|
||||
echo "$CLEANUP_OUTPUT"
|
||||
|
||||
# Extract files cleaned from cleanup output
|
||||
FILES_CLEANED=$(echo "$CLEANUP_OUTPUT" | grep -oE "Removed [0-9]+ file" | grep -oE "[0-9]+" | head -1)
|
||||
FILES_CLEANED="${FILES_CLEANED:-0}"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_WATCHDOG Run Manager ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
MANAGER_OUTPUT=$(bash "$MANAGER_SCRIPT" $([[ "$DRY_RUN" == true ]] && echo "--dry-run") 2>&1)
|
||||
# ==============================================================================================
|
||||
# ━━━ Run Manager ━━━
|
||||
# ==============================================================================================
|
||||
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each run
|
||||
# No --no-log flag here — manager owns the log write for this cycle ✅
|
||||
bash "$MANAGER_SCRIPT" $DRY_FLAG
|
||||
MANAGER_EXIT=$?
|
||||
echo "$MANAGER_OUTPUT"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Collect stats for daily log
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Ramdisk usage from state file
|
||||
RAMDISK_USED_GB=$(state_get "ramdisk_used_gb" 2>/dev/null || echo "0")
|
||||
[[ -z "$RAMDISK_USED_GB" || "$RAMDISK_USED_GB" == "0" ]] && \
|
||||
RAMDISK_USED_GB=$(df "$RAMDISK_PATH" --output=used 2>/dev/null | tail -1 | \
|
||||
awk '{printf "%.2f", $1/1048576}' || echo "0")
|
||||
|
||||
# Flip count from state file
|
||||
FLIP_COUNT=$(state_get "flip_count_hour" 2>/dev/null || echo "0")
|
||||
FLIP_COUNT="${FLIP_COUNT:-0}"
|
||||
|
||||
# Session counts from manager output
|
||||
RAM_SESSIONS=$(echo "$MANAGER_OUTPUT" | grep -oE "ramdisk \([0-9]+\)" | grep -oE "[0-9]+" | head -1)
|
||||
SSD_SESSIONS=$(echo "$MANAGER_OUTPUT" | grep -oE "SSD \([0-9]+\)" | grep -oE "[0-9]+" | head -1)
|
||||
RAM_SESSIONS="${RAM_SESSIONS:-0}"
|
||||
SSD_SESSIONS="${SSD_SESSIONS:-0}"
|
||||
|
||||
# Update daily log
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
update_daily_log "$RAMDISK_USED_GB" "$FLIP_COUNT" "$RAM_SESSIONS" "$SSD_SESSIONS" "$FILES_CLEANED"
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Exit with worst exit code
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
if [[ "$CLEANUP_EXIT" -ne 0 || "$MANAGER_EXIT" -ne 0 ]]; then
|
||||
exit 1
|
||||
fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Exit ━━━
|
||||
# ==============================================================================================
|
||||
# Return worst exit code — caller knows if either script failed
|
||||
[[ "$CLEANUP_EXIT" -ne 0 || "$MANAGER_EXIT" -ne 0 ]] && exit 1
|
||||
exit 0
|
||||
Reference in New Issue
Block a user