diff --git a/Master.conf b/Master.conf index 4086e5a..75001e9 100644 --- a/Master.conf +++ b/Master.conf @@ -161,6 +161,59 @@ DAILY_SYNC_SHARES=( /mnt/user/Tv_Shows ) +# ━━━ Personal Encrypted Shares ━━━ +# Personal shares synced to the remote server for offsite backup. +# These are independent of the failover container stack — data backup only. +# Each user syncs their own personal share to the other server. +# +# ── ZFS ENCRYPTION SETUP (unRAID 7) ───────────────────────────────────────────────────────── +# Encrypting your personal share means the remote admin can see the share exists +# and its file sizes but cannot read any content without your passphrase or keyfile. +# ZFS encrypts at the dataset level — rsync copies encrypted blocks as-is. +# The remote server never needs your key. +# +# Setup steps on HOST1: +# 1. In unRAID UI → go to your ZFS pool (Main tab → pool name) +# 2. Click the pool to expand it +# 3. Click "+ Dataset" to create a new dataset +# 4. Name it: e.g. Gmer4Lfe-Personal +# 5. Enable Encryption → set your passphrase (or keyfile path) +# ⚠️ Write your passphrase down — if lost, data is unrecoverable +# 6. Go to Settings → Shares → Add Share +# 7. Set the share path to your new encrypted dataset +# 8. Set Use cache: Only (keeps data on ZFS pool, not array) +# +# Auto-unlock on boot (optional — keyfile approach): +# 1. Create a keyfile: dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key +# 2. Store keyfile on HOST1 only — never sync it to HOST2 +# 3. Set the dataset to use keyfile instead of passphrase +# 4. Add to /etc/rc.local or a startup script: +# zfs load-key -L file:///root/.zfs-keys/personal.key poolname/Gmer4Lfe-Personal +# zfs mount poolname/Gmer4Lfe-Personal +# Manual unlock alternative (most secure): +# zfs load-key poolname/Gmer4Lfe-Personal (prompts for passphrase) +# zfs mount poolname/Gmer4Lfe-Personal +# +# Verify encryption is active before syncing: +# zfs get encryption poolname/Gmer4Lfe-Personal +# Should show: encryption aes-256-gcm (or similar) +# +# Once set up — add the share to PERSONAL_SYNC_SHARES below. +# rsync copies encrypted blocks to remote — remote admin cannot decrypt without your key. +# ───────────────────────────────────────────────────────────────────────────────────────────── + +# HOST1 personal shares synced to HOST2 for offsite backup +# These sync via daily_sync.sh or on their own schedule +# Encrypted datasets sync as encrypted — remote cannot read content +HOST1_PERSONAL_SHARES=( + # /mnt/user/Gmer4Lfe-Personal # uncomment after creating encrypted dataset +) + +# HOST2 personal shares synced to HOST1 for offsite backup +HOST2_PERSONAL_SHARES=( + # /mnt/user/Jayred365-Personal # uncomment after creating encrypted dataset +) + # ━━━ Rsync Profile System ━━━ # Profiles allow per-share rsync behaviour without touching script logic. # The profile key is matched automatically by the basename of the directory @@ -822,19 +875,31 @@ MEDIA_MAINTENANCE_JOBS=( # ============================================================================================== # ── TRANSCODES ──────────────────────────────────────────────────────────────────────────────── # ============================================================================================== -# ⚠️ One mount only in Emby container: /mnt/ram-transcode → /ext-ram-transcode -# Do NOT add a static SSD path — Emby will use it independently of the symlink. -# The symlink IS your emergency lever — flip it manually if needed: -# ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode +# Session-based storage allocator using filesystem symlink indirection. +# ffmpeg resolves the symlink ONCE at session start — existing sessions are never affected. +# Only new sessions care about where the symlink currently points. +# +# How it works: +# ramdisk_setup.sh — run once at array start, creates tmpfs and sets symlink +# transcode_management.sh — every 3 min, runs cleanup then manager in correct order +# transcode_cleanup.sh — called by transcode_management.sh — removes old inactive files +# transcode_manager.sh — called by transcode_management.sh — manages symlink direction +# +# ⚠️ Docker mount warning: +# Mount must use shared propagation so symlink flips are visible inside the container. +# In unRAID Extra Parameters — do NOT use standard path mapping for this mount: +# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared +# Standard bind mounts use rprivate — Docker locks the inode on first symlink flip +# and new sessions land on SSD permanently for that container run. RAMDISK_PATH="/mnt/ramdisk_transcodes" # tmpfs mount point created at array start - RAMDISK_SIZE="8G" # ceiling — tmpfs only uses RAM actually needed + RAMDISK_SIZE="10G" # ceiling — tmpfs only uses RAM actually needed TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at — location never changes TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback location # Usage thresholds in GB — hysteresis gap between WARN and LOW prevents flip-flop - RAMDISK_WARN_GB=6.8 # flip symlink to SSD at or above this usage - RAMDISK_LOW_GB=5.5 # flip symlink back to ramdisk when usage drops here + RAMDISK_WARN_GB=8.8 # flip symlink to SSD at or above this usage + RAMDISK_LOW_GB=6.5 # flip symlink back to ramdisk when usage drops here RAMDISK_SSD_MIN_GB=20 # minimum free GB on SSD required before allowing flip to SSD # Cleanup age thresholds — files must be older than these AND not open by any process @@ -863,6 +928,12 @@ MEDIA_MAINTENANCE_JOBS=( TRANSCODE_CHECK_EMBY=true TRANSCODE_EMBY_CONTAINER="Emby" # exact Docker container name — case sensitive +# Daily transcode statistics log — read by weekly_health_digest.sh +# Tracks peak usage, flip count, session ratio, files cleaned per day +# Bounded to TRANSCODE_LOG_RETENTION days — auto-purges old entries + TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db" + TRANSCODE_LOG_RETENTION=90 + # ============================================================================================== # ── MONITORS ────────────────────────────────────────────────────────────────────────────────── # ============================================================================================== diff --git a/Monitors/weekly_health_digest.sh b/Monitors/weekly_health_digest.sh index 1256d5c..a70377c 100644 --- a/Monitors/weekly_health_digest.sh +++ b/Monitors/weekly_health_digest.sh @@ -105,6 +105,45 @@ if mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then RAMDISK_AVAIL_GB=$(awk "BEGIN {printf \"%.2f\", $RAMDISK_AVAIL_KB / 1048576}") SYMLINK_TARGET=$(readlink "$TRANSCODE_LINK" 2>/dev/null || echo "unknown") DIGEST_LINES+=("$ICON_RAM Transcodes: ${RAMDISK_USED_GB}GB used / ${RAMDISK_AVAIL_GB}GB free → $SYMLINK_TARGET") + + # Read weekly transcode stats from daily log if available + TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db" + if [[ -f "$TRANSCODE_DAILY_LOG" ]]; then + WEEK_CUTOFF=$(date -d "7 days ago" '+%Y-%m-%d') + + # Peak ramdisk usage this week + WEEK_PEAK=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \ + '$1 >= cutoff {if ($2 > max) max=$2} END {printf "%.2f", max+0}' \ + "$TRANSCODE_DAILY_LOG") + + # Total flips this week + WEEK_FLIPS=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \ + '$1 >= cutoff {sum+=$3} END {print sum+0}' \ + "$TRANSCODE_DAILY_LOG") + + # Total files cleaned this week + WEEK_FILES=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \ + '$1 >= cutoff {sum+=$6} END {print sum+0}' \ + "$TRANSCODE_DAILY_LOG") + + # Ram vs SSD session ratio + WEEK_RAM=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \ + '$1 >= cutoff {sum+=$4} END {print sum+0}' \ + "$TRANSCODE_DAILY_LOG") + WEEK_SSD=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \ + '$1 >= cutoff {sum+=$5} END {print sum+0}' \ + "$TRANSCODE_DAILY_LOG") + + DIGEST_LINES+=("$ICON_RAM Transcode week: peak ${WEEK_PEAK}GB | flips: ${WEEK_FLIPS} | cleaned: ${WEEK_FILES} files") + DIGEST_LINES+=("$ICON_RAM Session storage: ${WEEK_RAM} ramdisk / ${WEEK_SSD} SSD") + + # Warn if peak is getting close to threshold + PEAK_INT=$(printf "%.0f" "$WEEK_PEAK") + WARN_INT=$(printf "%.0f" "$RAMDISK_WARN_GB") + if [[ "$PEAK_INT" -ge "$WARN_INT" ]]; then + DIGEST_LINES+=("$ICON_WARN Transcode peak ${WEEK_PEAK}GB near threshold ${RAMDISK_WARN_GB}GB — consider increasing RAMDISK_SIZE") + fi + fi else DIGEST_LINES+=("$ICON_RAM Transcodes: ramdisk not mounted") ISSUES+=("Ramdisk not mounted") diff --git a/Orchestrators/README-orchestrators.md b/Orchestrators/README-Orchestrators.md similarity index 100% rename from Orchestrators/README-orchestrators.md rename to Orchestrators/README-Orchestrators.md diff --git a/Orchestrators/daily_sync.sh b/Orchestrators/daily_sync.sh index 6d34bbd..d337124 100644 --- a/Orchestrators/daily_sync.sh +++ b/Orchestrators/daily_sync.sh @@ -41,7 +41,21 @@ PASS=() FAIL=() SHARE_TIMES=() TOTAL_START=$(date +%s) -SHARE_COUNT=${#DAILY_SYNC_SHARES[@]} + +# Build combined share list — media shares + personal shares for this host +ALL_SHARES=("${DAILY_SYNC_SHARES[@]}") + +if [[ "$LOCAL_SERVER_NAME" == "$HOST1" ]]; then + for share in "${HOST1_PERSONAL_SHARES[@]}"; do + [[ -n "$share" ]] && ALL_SHARES+=("$share") + done +elif [[ "$LOCAL_SERVER_NAME" == "$HOST2" ]]; then + for share in "${HOST2_PERSONAL_SHARES[@]}"; do + [[ -n "$share" ]] && ALL_SHARES+=("$share") + done +fi + +SHARE_COUNT=${#ALL_SHARES[@]} # ----------------------------------------------------------------------------------------------- # ━━━ $ICON_SYNC Transfer ━━━ @@ -53,7 +67,7 @@ echo "" SHARE_INDEX=0 -for SHARE in "${DAILY_SYNC_SHARES[@]}"; do +for SHARE in "${ALL_SHARES[@]}"; do SHARE_INDEX=$((SHARE_INDEX + 1)) SHARE_NAME=$(basename "$SHARE") SHARE_START=$(date +%s) diff --git a/Orchestrators/transcode_management.sh b/Orchestrators/transcode_management.sh new file mode 100644 index 0000000..c95b9ba --- /dev/null +++ b/Orchestrators/transcode_management.sh @@ -0,0 +1,184 @@ +#!/bin/bash +# ----------------------------------------------------------------------------------------------- +# --------------------------------- Transcode Management --------------------------------------- +# ----------------------------------------------------------------------------------------------- +# Runs transcode cleanup then transcode manager in the correct order every cycle. +# Cleanup runs first — clears stale files so manager sees accurate usage. +# Manager runs after — threshold decisions based on real current usage post-cleanup. +# +# Running cleanup before manager prevents unnecessary SSD flips caused by stale +# segment files from ended sessions inflating the ramdisk usage reading. +# +# Also tracks daily transcode statistics to a bounded log for weekly_health_digest.sh: +# Peak ramdisk usage per day +# Total flip count per day +# Ramdisk vs SSD session ratio +# Files cleaned per day +# +# Scheduled as: */3 * * * * (every 3 minutes) +# Replace individual transcode_manager and transcode_cleanup cron entries with this. +# +# All configuration in Master.conf under Transcode Manager section. +# Supports --dry-run — passes through to both child scripts. +# ----------------------------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +source "$SCRIPT_DIR/../Master.conf" +source "$SCRIPT_DIR/../common.sh" + +parse_args "$@" + +# ----------------------------------------------------------------------------------------------- +# State and log files +# ----------------------------------------------------------------------------------------------- +TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db" +TRANSCODE_STATE_FILE="/tmp/transcode_state.db" + +# Bounded log — keeps last 90 days +TRANSCODE_LOG_RETENTION=90 + +touch "$TRANSCODE_DAILY_LOG" 2>/dev/null + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_GEAR Setup ━━━ +# ----------------------------------------------------------------------------------------------- +if [[ "$EUID" -ne 0 ]]; then + error "Must be run as root" + exit 1 +fi + +[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing to child scripts" + +# ----------------------------------------------------------------------------------------------- +# HELPERS +# ----------------------------------------------------------------------------------------------- + +# Read a value from transcode state file +state_get() { + grep "^${1}=" "$TRANSCODE_STATE_FILE" 2>/dev/null | cut -d= -f2 +} + +# Get today's date key +today() { + date '+%Y-%m-%d' +} + +# Update daily log entry for today +# Format: YYYY-MM-DD|peak_gb|flip_count|ram_sessions|ssd_sessions|files_cleaned +update_daily_log() { + local peak_gb="$1" + local flips="$2" + local ram_sessions="$3" + local ssd_sessions="$4" + local files_cleaned="$5" + local today_key + today_key=$(today) + + local existing + existing=$(grep "^${today_key}|" "$TRANSCODE_DAILY_LOG" 2>/dev/null) + + if [[ -z "$existing" ]]; then + # New entry for today + echo "${today_key}|${peak_gb}|${flips}|${ram_sessions}|${ssd_sessions}|${files_cleaned}" \ + >> "$TRANSCODE_DAILY_LOG" + else + # Update existing — keep highest peak, accumulate flips, sessions, files + local old_peak old_flips old_ram old_ssd old_files + old_peak=$(echo "$existing" | cut -d'|' -f2) + old_flips=$(echo "$existing" | cut -d'|' -f3) + old_ram=$(echo "$existing" | cut -d'|' -f4) + old_ssd=$(echo "$existing" | cut -d'|' -f5) + old_files=$(echo "$existing" | cut -d'|' -f6) + + # Peak — keep highest + local new_peak + new_peak=$(awk "BEGIN {print ($peak_gb > $old_peak) ? $peak_gb : $old_peak}") + + # Accumulate + local new_flips=$(( old_flips + flips )) + local new_ram=$(( old_ram + ram_sessions )) + local new_ssd=$(( old_ssd + ssd_sessions )) + local new_files=$(( old_files + files_cleaned )) + + # Replace line + sed -i "s|^${today_key}|.*|${today_key}|${new_peak}|${new_flips}|${new_ram}|${new_ssd}|${new_files}|" \ + "$TRANSCODE_DAILY_LOG" 2>/dev/null || { + # sed replacement failed — remove and re-add + sed -i "/^${today_key}|/d" "$TRANSCODE_DAILY_LOG" + echo "${today_key}|${new_peak}|${new_flips}|${new_ram}|${new_ssd}|${new_files}" \ + >> "$TRANSCODE_DAILY_LOG" + } + fi + + # Purge old entries beyond retention + local cutoff + cutoff=$(date -d "${TRANSCODE_LOG_RETENTION} days ago" '+%Y-%m-%d') + awk -F'|' -v cutoff="$cutoff" '$1 >= cutoff' \ + "$TRANSCODE_DAILY_LOG" > "${TRANSCODE_DAILY_LOG}.tmp" && \ + mv "${TRANSCODE_DAILY_LOG}.tmp" "$TRANSCODE_DAILY_LOG" +} + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SYNC Run Cleanup ━━━ +# ----------------------------------------------------------------------------------------------- +CLEANUP_SCRIPT="$SCRIPT_DIR/transcode_cleanup.sh" +MANAGER_SCRIPT="$SCRIPT_DIR/transcode_manager.sh" + +if [[ ! -f "$CLEANUP_SCRIPT" ]]; then + error "transcode_cleanup.sh not found: $CLEANUP_SCRIPT" + exit 1 +fi + +if [[ ! -f "$MANAGER_SCRIPT" ]]; then + error "transcode_manager.sh not found: $MANAGER_SCRIPT" + exit 1 +fi + +# Capture cleanup output for file count +CLEANUP_OUTPUT=$(bash "$CLEANUP_SCRIPT" ${DRY_RUN:+--dry-run} 2>&1) +CLEANUP_EXIT=$? +echo "$CLEANUP_OUTPUT" + +# Extract files cleaned from cleanup output +FILES_CLEANED=$(echo "$CLEANUP_OUTPUT" | grep -oE "Removed [0-9]+ file" | grep -oE "[0-9]+" | head -1) +FILES_CLEANED="${FILES_CLEANED:-0}" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_WATCHDOG Run Manager ━━━ +# ----------------------------------------------------------------------------------------------- +MANAGER_OUTPUT=$(bash "$MANAGER_SCRIPT" ${DRY_RUN:+--dry-run} 2>&1) +MANAGER_EXIT=$? +echo "$MANAGER_OUTPUT" + +# ----------------------------------------------------------------------------------------------- +# Collect stats for daily log +# ----------------------------------------------------------------------------------------------- +# Ramdisk usage from state file +RAMDISK_USED_GB=$(state_get "ramdisk_used_gb" 2>/dev/null || echo "0") +[[ -z "$RAMDISK_USED_GB" || "$RAMDISK_USED_GB" == "0" ]] && \ + RAMDISK_USED_GB=$(df "$RAMDISK_PATH" --output=used 2>/dev/null | tail -1 | \ + awk '{printf "%.2f", $1/1048576}' || echo "0") + +# Flip count from state file +FLIP_COUNT=$(state_get "flip_count_hour" 2>/dev/null || echo "0") +FLIP_COUNT="${FLIP_COUNT:-0}" + +# Session counts from manager output +RAM_SESSIONS=$(echo "$MANAGER_OUTPUT" | grep -oE "ramdisk \([0-9]+\)" | grep -oE "[0-9]+" | head -1) +SSD_SESSIONS=$(echo "$MANAGER_OUTPUT" | grep -oE "SSD \([0-9]+\)" | grep -oE "[0-9]+" | head -1) +RAM_SESSIONS="${RAM_SESSIONS:-0}" +SSD_SESSIONS="${SSD_SESSIONS:-0}" + +# Update daily log +if [[ "$DRY_RUN" == false ]]; then + update_daily_log "$RAMDISK_USED_GB" "$FLIP_COUNT" "$RAM_SESSIONS" "$SSD_SESSIONS" "$FILES_CLEANED" +fi + +# ----------------------------------------------------------------------------------------------- +# Exit with worst exit code +# ----------------------------------------------------------------------------------------------- +if [[ "$CLEANUP_EXIT" -ne 0 || "$MANAGER_EXIT" -ne 0 ]]; then + exit 1 +fi +exit 0 \ No newline at end of file diff --git a/README.md b/README.md index eca63ee..095878a 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,95 @@ Custom network: high-availability NginxProxyManager → NextCloud (by name) ``` -This flexibility means any two unRAID servers can participate — regardless of hardware generation, CPU, drive count, or storage layout. The only requirements are matching share names, matching container names for shared services, and the same custom Docker network names. +**Personal shares — backup without failover:** + +Beyond the shared media library, each user can sync personal shares to the other server purely for offsite backup — no failover container involvement, just data protection. + +``` +HOST1: /mnt/user/Gmer4Lfe-Personal → rsync nightly → HOST2 (encrypted backup) +HOST2: /mnt/user/Jayred365-Personal → rsync nightly → HOST1 (encrypted backup) +``` + +Configure in `Master.conf`: +```bash +HOST1_PERSONAL_SHARES=( + "/mnt/user/Gmer4Lfe-Personal" +) +HOST2_PERSONAL_SHARES=( + "/mnt/user/Jayred365-Personal" +) +``` + +`daily_sync.sh` automatically picks up the personal shares for the local host and syncs them alongside the media shares. + +**Encrypting personal shares — unRAID 7 ZFS:** + +ZFS native encryption in unRAID 7 means both admins can see the share exists and file sizes but neither can read content without your passphrase or keyfile. rsync copies encrypted blocks as-is — the remote server never needs your key. + +**Setup on HOST1 (your personal share):** + +**Step 1 — Create an encrypted ZFS dataset:** +``` +unRAID UI → Main tab → click your ZFS pool +→ Click "+ Dataset" +→ Name: Gmer4Lfe-Personal +→ Enable Encryption: Yes +→ Encryption type: passphrase (simplest) or keyfile (auto-unlock capable) +→ Enter your passphrase — write it down, if lost data is unrecoverable +→ Create +``` + +**Step 2 — Create the share:** +``` +Settings → Shares → Add Share +→ Name: Gmer4Lfe-Personal +→ Primary storage: your ZFS pool +→ Use cache: Only (keeps data on ZFS pool, not array) +→ Add +``` + +**Step 3 — Verify encryption is active:** +```bash +zfs get encryption poolname/Gmer4Lfe-Personal +# Should show: encryption aes-256-gcm +``` + +**Step 4 — Add to Master.conf and sync:** +```bash +HOST1_PERSONAL_SHARES=( + "/mnt/user/Gmer4Lfe-Personal" +) +``` + +**Auto-unlock on boot (keyfile approach):** + +If you want the share to mount automatically after reboot without entering a passphrase: +```bash +# Create keyfile — on HOST1 only, never sync this file +mkdir -p /root/.zfs-keys +dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key +chmod 400 /root/.zfs-keys/personal.key + +# Set dataset to use keyfile +zfs change-key -o keylocation=file:///root/.zfs-keys/personal.key \ + -o keyformat=raw poolname/Gmer4Lfe-Personal + +# Add to array start script (unRAID_Essentials or User Scripts) +zfs load-key poolname/Gmer4Lfe-Personal +zfs mount poolname/Gmer4Lfe-Personal +``` + +**Manual unlock (most secure — you control when it's readable):** +```bash +zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase +zfs mount poolname/Gmer4Lfe-Personal +``` + +**What the remote admin sees:** + +The share directory exists on HOST2. File names and sizes are visible (ZFS encrypts content, not metadata by default). File contents are unreadable without your key. To hide filenames too, enable `zfs set encryption=aes-256-gcm` with `dnodesize=auto` — this is a more advanced setup. + +**Current status:** The infrastructure supports encrypted personal share syncing. The ZFS dataset setup is a one-time manual step per server. Once set up it syncs automatically like any other share. --- diff --git a/Tools/Recreate_shares.sh b/Tools/recreate_shares.sh similarity index 100% rename from Tools/Recreate_shares.sh rename to Tools/recreate_shares.sh diff --git a/user_script_pluin.sh b/user_script_plug-in.sh similarity index 100% rename from user_script_pluin.sh rename to user_script_plug-in.sh