fixed ttypos

This commit is contained in:
2026-04-19 11:44:58 -04:00
parent f48752afb9
commit 804a435b31
8 changed files with 406 additions and 10 deletions
+78 -7
View File
@@ -161,6 +161,59 @@ DAILY_SYNC_SHARES=(
/mnt/user/Tv_Shows
)
# ━━━ Personal Encrypted Shares ━━━
# Personal shares synced to the remote server for offsite backup.
# These are independent of the failover container stack — data backup only.
# Each user syncs their own personal share to the other server.
#
# ── ZFS ENCRYPTION SETUP (unRAID 7) ─────────────────────────────────────────────────────────
# Encrypting your personal share means the remote admin can see the share exists
# and its file sizes but cannot read any content without your passphrase or keyfile.
# ZFS encrypts at the dataset level — rsync copies encrypted blocks as-is.
# The remote server never needs your key.
#
# Setup steps on HOST1:
# 1. In unRAID UI → go to your ZFS pool (Main tab → pool name)
# 2. Click the pool to expand it
# 3. Click "+ Dataset" to create a new dataset
# 4. Name it: e.g. Gmer4Lfe-Personal
# 5. Enable Encryption → set your passphrase (or keyfile path)
# ⚠️ Write your passphrase down — if lost, data is unrecoverable
# 6. Go to Settings → Shares → Add Share
# 7. Set the share path to your new encrypted dataset
# 8. Set Use cache: Only (keeps data on ZFS pool, not array)
#
# Auto-unlock on boot (optional — keyfile approach):
# 1. Create a keyfile: dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
# 2. Store keyfile on HOST1 only — never sync it to HOST2
# 3. Set the dataset to use keyfile instead of passphrase
# 4. Add to /etc/rc.local or a startup script:
# zfs load-key -L file:///root/.zfs-keys/personal.key poolname/Gmer4Lfe-Personal
# zfs mount poolname/Gmer4Lfe-Personal
# Manual unlock alternative (most secure):
# zfs load-key poolname/Gmer4Lfe-Personal (prompts for passphrase)
# zfs mount poolname/Gmer4Lfe-Personal
#
# Verify encryption is active before syncing:
# zfs get encryption poolname/Gmer4Lfe-Personal
# Should show: encryption aes-256-gcm (or similar)
#
# Once set up — add the share to PERSONAL_SYNC_SHARES below.
# rsync copies encrypted blocks to remote — remote admin cannot decrypt without your key.
# ─────────────────────────────────────────────────────────────────────────────────────────────
# HOST1 personal shares synced to HOST2 for offsite backup
# These sync via daily_sync.sh or on their own schedule
# Encrypted datasets sync as encrypted — remote cannot read content
HOST1_PERSONAL_SHARES=(
# /mnt/user/Gmer4Lfe-Personal # uncomment after creating encrypted dataset
)
# HOST2 personal shares synced to HOST1 for offsite backup
HOST2_PERSONAL_SHARES=(
# /mnt/user/Jayred365-Personal # uncomment after creating encrypted dataset
)
# ━━━ Rsync Profile System ━━━
# Profiles allow per-share rsync behaviour without touching script logic.
# The profile key is matched automatically by the basename of the directory
@@ -822,19 +875,31 @@ MEDIA_MAINTENANCE_JOBS=(
# ==============================================================================================
# ── TRANSCODES ────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# ⚠️ One mount only in Emby container: /mnt/ram-transcode → /ext-ram-transcode
# Do NOT add a static SSD path — Emby will use it independently of the symlink.
# The symlink IS your emergency lever — flip it manually if needed:
# ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
# Session-based storage allocator using filesystem symlink indirection.
# ffmpeg resolves the symlink ONCE at session start — existing sessions are never affected.
# Only new sessions care about where the symlink currently points.
#
# How it works:
# ramdisk_setup.sh — run once at array start, creates tmpfs and sets symlink
# transcode_management.sh — every 3 min, runs cleanup then manager in correct order
# transcode_cleanup.sh — called by transcode_management.sh — removes old inactive files
# transcode_manager.sh — called by transcode_management.sh — manages symlink direction
#
# ⚠️ Docker mount warning:
# Mount must use shared propagation so symlink flips are visible inside the container.
# In unRAID Extra Parameters — do NOT use standard path mapping for this mount:
# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
# Standard bind mounts use rprivate — Docker locks the inode on first symlink flip
# and new sessions land on SSD permanently for that container run.
RAMDISK_PATH="/mnt/ramdisk_transcodes" # tmpfs mount point created at array start
RAMDISK_SIZE="8G" # ceiling — tmpfs only uses RAM actually needed
RAMDISK_SIZE="10G" # ceiling — tmpfs only uses RAM actually needed
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at — location never changes
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback location
# Usage thresholds in GB — hysteresis gap between WARN and LOW prevents flip-flop
RAMDISK_WARN_GB=6.8 # flip symlink to SSD at or above this usage
RAMDISK_LOW_GB=5.5 # flip symlink back to ramdisk when usage drops here
RAMDISK_WARN_GB=8.8 # flip symlink to SSD at or above this usage
RAMDISK_LOW_GB=6.5 # flip symlink back to ramdisk when usage drops here
RAMDISK_SSD_MIN_GB=20 # minimum free GB on SSD required before allowing flip to SSD
# Cleanup age thresholds — files must be older than these AND not open by any process
@@ -863,6 +928,12 @@ MEDIA_MAINTENANCE_JOBS=(
TRANSCODE_CHECK_EMBY=true
TRANSCODE_EMBY_CONTAINER="Emby" # exact Docker container name — case sensitive
# Daily transcode statistics log — read by weekly_health_digest.sh
# Tracks peak usage, flip count, session ratio, files cleaned per day
# Bounded to TRANSCODE_LOG_RETENTION days — auto-purges old entries
TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db"
TRANSCODE_LOG_RETENTION=90
# ==============================================================================================
# ── MONITORS ──────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
+39
View File
@@ -105,6 +105,45 @@ if mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
RAMDISK_AVAIL_GB=$(awk "BEGIN {printf \"%.2f\", $RAMDISK_AVAIL_KB / 1048576}")
SYMLINK_TARGET=$(readlink "$TRANSCODE_LINK" 2>/dev/null || echo "unknown")
DIGEST_LINES+=("$ICON_RAM Transcodes: ${RAMDISK_USED_GB}GB used / ${RAMDISK_AVAIL_GB}GB free → $SYMLINK_TARGET")
# Read weekly transcode stats from daily log if available
TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db"
if [[ -f "$TRANSCODE_DAILY_LOG" ]]; then
WEEK_CUTOFF=$(date -d "7 days ago" '+%Y-%m-%d')
# Peak ramdisk usage this week
WEEK_PEAK=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \
'$1 >= cutoff {if ($2 > max) max=$2} END {printf "%.2f", max+0}' \
"$TRANSCODE_DAILY_LOG")
# Total flips this week
WEEK_FLIPS=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \
'$1 >= cutoff {sum+=$3} END {print sum+0}' \
"$TRANSCODE_DAILY_LOG")
# Total files cleaned this week
WEEK_FILES=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \
'$1 >= cutoff {sum+=$6} END {print sum+0}' \
"$TRANSCODE_DAILY_LOG")
# Ram vs SSD session ratio
WEEK_RAM=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \
'$1 >= cutoff {sum+=$4} END {print sum+0}' \
"$TRANSCODE_DAILY_LOG")
WEEK_SSD=$(awk -F'|' -v cutoff="$WEEK_CUTOFF" \
'$1 >= cutoff {sum+=$5} END {print sum+0}' \
"$TRANSCODE_DAILY_LOG")
DIGEST_LINES+=("$ICON_RAM Transcode week: peak ${WEEK_PEAK}GB | flips: ${WEEK_FLIPS} | cleaned: ${WEEK_FILES} files")
DIGEST_LINES+=("$ICON_RAM Session storage: ${WEEK_RAM} ramdisk / ${WEEK_SSD} SSD")
# Warn if peak is getting close to threshold
PEAK_INT=$(printf "%.0f" "$WEEK_PEAK")
WARN_INT=$(printf "%.0f" "$RAMDISK_WARN_GB")
if [[ "$PEAK_INT" -ge "$WARN_INT" ]]; then
DIGEST_LINES+=("$ICON_WARN Transcode peak ${WEEK_PEAK}GB near threshold ${RAMDISK_WARN_GB}GB — consider increasing RAMDISK_SIZE")
fi
fi
else
DIGEST_LINES+=("$ICON_RAM Transcodes: ramdisk not mounted")
ISSUES+=("Ramdisk not mounted")
+16 -2
View File
@@ -41,7 +41,21 @@ PASS=()
FAIL=()
SHARE_TIMES=()
TOTAL_START=$(date +%s)
SHARE_COUNT=${#DAILY_SYNC_SHARES[@]}
# Build combined share list — media shares + personal shares for this host
ALL_SHARES=("${DAILY_SYNC_SHARES[@]}")
if [[ "$LOCAL_SERVER_NAME" == "$HOST1" ]]; then
for share in "${HOST1_PERSONAL_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
elif [[ "$LOCAL_SERVER_NAME" == "$HOST2" ]]; then
for share in "${HOST2_PERSONAL_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
fi
SHARE_COUNT=${#ALL_SHARES[@]}
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SYNC Transfer ━━━
@@ -53,7 +67,7 @@ echo ""
SHARE_INDEX=0
for SHARE in "${DAILY_SYNC_SHARES[@]}"; do
for SHARE in "${ALL_SHARES[@]}"; do
SHARE_INDEX=$((SHARE_INDEX + 1))
SHARE_NAME=$(basename "$SHARE")
SHARE_START=$(date +%s)
+184
View File
@@ -0,0 +1,184 @@
#!/bin/bash
# -----------------------------------------------------------------------------------------------
# --------------------------------- Transcode Management ---------------------------------------
# -----------------------------------------------------------------------------------------------
# Runs transcode cleanup then transcode manager in the correct order every cycle.
# Cleanup runs first — clears stale files so manager sees accurate usage.
# Manager runs after — threshold decisions based on real current usage post-cleanup.
#
# Running cleanup before manager prevents unnecessary SSD flips caused by stale
# segment files from ended sessions inflating the ramdisk usage reading.
#
# Also tracks daily transcode statistics to a bounded log for weekly_health_digest.sh:
# Peak ramdisk usage per day
# Total flip count per day
# Ramdisk vs SSD session ratio
# Files cleaned per day
#
# Scheduled as: */3 * * * * (every 3 minutes)
# Replace individual transcode_manager and transcode_cleanup cron entries with this.
#
# All configuration in Master.conf under Transcode Manager section.
# Supports --dry-run — passes through to both child scripts.
# -----------------------------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../Master.conf"
source "$SCRIPT_DIR/../common.sh"
parse_args "$@"
# -----------------------------------------------------------------------------------------------
# State and log files
# -----------------------------------------------------------------------------------------------
TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db"
TRANSCODE_STATE_FILE="/tmp/transcode_state.db"
# Bounded log — keeps last 90 days
TRANSCODE_LOG_RETENTION=90
touch "$TRANSCODE_DAILY_LOG" 2>/dev/null
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_GEAR Setup ━━━
# -----------------------------------------------------------------------------------------------
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing to child scripts"
# -----------------------------------------------------------------------------------------------
# HELPERS
# -----------------------------------------------------------------------------------------------
# Read a value from transcode state file
state_get() {
grep "^${1}=" "$TRANSCODE_STATE_FILE" 2>/dev/null | cut -d= -f2
}
# Get today's date key
today() {
date '+%Y-%m-%d'
}
# Update daily log entry for today
# Format: YYYY-MM-DD|peak_gb|flip_count|ram_sessions|ssd_sessions|files_cleaned
update_daily_log() {
local peak_gb="$1"
local flips="$2"
local ram_sessions="$3"
local ssd_sessions="$4"
local files_cleaned="$5"
local today_key
today_key=$(today)
local existing
existing=$(grep "^${today_key}|" "$TRANSCODE_DAILY_LOG" 2>/dev/null)
if [[ -z "$existing" ]]; then
# New entry for today
echo "${today_key}|${peak_gb}|${flips}|${ram_sessions}|${ssd_sessions}|${files_cleaned}" \
>> "$TRANSCODE_DAILY_LOG"
else
# Update existing — keep highest peak, accumulate flips, sessions, files
local old_peak old_flips old_ram old_ssd old_files
old_peak=$(echo "$existing" | cut -d'|' -f2)
old_flips=$(echo "$existing" | cut -d'|' -f3)
old_ram=$(echo "$existing" | cut -d'|' -f4)
old_ssd=$(echo "$existing" | cut -d'|' -f5)
old_files=$(echo "$existing" | cut -d'|' -f6)
# Peak — keep highest
local new_peak
new_peak=$(awk "BEGIN {print ($peak_gb > $old_peak) ? $peak_gb : $old_peak}")
# Accumulate
local new_flips=$(( old_flips + flips ))
local new_ram=$(( old_ram + ram_sessions ))
local new_ssd=$(( old_ssd + ssd_sessions ))
local new_files=$(( old_files + files_cleaned ))
# Replace line
sed -i "s|^${today_key}|.*|${today_key}|${new_peak}|${new_flips}|${new_ram}|${new_ssd}|${new_files}|" \
"$TRANSCODE_DAILY_LOG" 2>/dev/null || {
# sed replacement failed — remove and re-add
sed -i "/^${today_key}|/d" "$TRANSCODE_DAILY_LOG"
echo "${today_key}|${new_peak}|${new_flips}|${new_ram}|${new_ssd}|${new_files}" \
>> "$TRANSCODE_DAILY_LOG"
}
fi
# Purge old entries beyond retention
local cutoff
cutoff=$(date -d "${TRANSCODE_LOG_RETENTION} days ago" '+%Y-%m-%d')
awk -F'|' -v cutoff="$cutoff" '$1 >= cutoff' \
"$TRANSCODE_DAILY_LOG" > "${TRANSCODE_DAILY_LOG}.tmp" && \
mv "${TRANSCODE_DAILY_LOG}.tmp" "$TRANSCODE_DAILY_LOG"
}
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_SYNC Run Cleanup ━━━
# -----------------------------------------------------------------------------------------------
CLEANUP_SCRIPT="$SCRIPT_DIR/transcode_cleanup.sh"
MANAGER_SCRIPT="$SCRIPT_DIR/transcode_manager.sh"
if [[ ! -f "$CLEANUP_SCRIPT" ]]; then
error "transcode_cleanup.sh not found: $CLEANUP_SCRIPT"
exit 1
fi
if [[ ! -f "$MANAGER_SCRIPT" ]]; then
error "transcode_manager.sh not found: $MANAGER_SCRIPT"
exit 1
fi
# Capture cleanup output for file count
CLEANUP_OUTPUT=$(bash "$CLEANUP_SCRIPT" ${DRY_RUN:+--dry-run} 2>&1)
CLEANUP_EXIT=$?
echo "$CLEANUP_OUTPUT"
# Extract files cleaned from cleanup output
FILES_CLEANED=$(echo "$CLEANUP_OUTPUT" | grep -oE "Removed [0-9]+ file" | grep -oE "[0-9]+" | head -1)
FILES_CLEANED="${FILES_CLEANED:-0}"
# -----------------------------------------------------------------------------------------------
# ━━━ $ICON_WATCHDOG Run Manager ━━━
# -----------------------------------------------------------------------------------------------
MANAGER_OUTPUT=$(bash "$MANAGER_SCRIPT" ${DRY_RUN:+--dry-run} 2>&1)
MANAGER_EXIT=$?
echo "$MANAGER_OUTPUT"
# -----------------------------------------------------------------------------------------------
# Collect stats for daily log
# -----------------------------------------------------------------------------------------------
# Ramdisk usage from state file
RAMDISK_USED_GB=$(state_get "ramdisk_used_gb" 2>/dev/null || echo "0")
[[ -z "$RAMDISK_USED_GB" || "$RAMDISK_USED_GB" == "0" ]] && \
RAMDISK_USED_GB=$(df "$RAMDISK_PATH" --output=used 2>/dev/null | tail -1 | \
awk '{printf "%.2f", $1/1048576}' || echo "0")
# Flip count from state file
FLIP_COUNT=$(state_get "flip_count_hour" 2>/dev/null || echo "0")
FLIP_COUNT="${FLIP_COUNT:-0}"
# Session counts from manager output
RAM_SESSIONS=$(echo "$MANAGER_OUTPUT" | grep -oE "ramdisk \([0-9]+\)" | grep -oE "[0-9]+" | head -1)
SSD_SESSIONS=$(echo "$MANAGER_OUTPUT" | grep -oE "SSD \([0-9]+\)" | grep -oE "[0-9]+" | head -1)
RAM_SESSIONS="${RAM_SESSIONS:-0}"
SSD_SESSIONS="${SSD_SESSIONS:-0}"
# Update daily log
if [[ "$DRY_RUN" == false ]]; then
update_daily_log "$RAMDISK_USED_GB" "$FLIP_COUNT" "$RAM_SESSIONS" "$SSD_SESSIONS" "$FILES_CLEANED"
fi
# -----------------------------------------------------------------------------------------------
# Exit with worst exit code
# -----------------------------------------------------------------------------------------------
if [[ "$CLEANUP_EXIT" -ne 0 || "$MANAGER_EXIT" -ne 0 ]]; then
exit 1
fi
exit 0
+89 -1
View File
@@ -99,7 +99,95 @@ Custom network: high-availability
NginxProxyManager → NextCloud (by name)
```
This flexibility means any two unRAID servers can participate — regardless of hardware generation, CPU, drive count, or storage layout. The only requirements are matching share names, matching container names for shared services, and the same custom Docker network names.
**Personal shares — backup without failover:**
Beyond the shared media library, each user can sync personal shares to the other server purely for offsite backup — no failover container involvement, just data protection.
```
HOST1: /mnt/user/Gmer4Lfe-Personal → rsync nightly → HOST2 (encrypted backup)
HOST2: /mnt/user/Jayred365-Personal → rsync nightly → HOST1 (encrypted backup)
```
Configure in `Master.conf`:
```bash
HOST1_PERSONAL_SHARES=(
"/mnt/user/Gmer4Lfe-Personal"
)
HOST2_PERSONAL_SHARES=(
"/mnt/user/Jayred365-Personal"
)
```
`daily_sync.sh` automatically picks up the personal shares for the local host and syncs them alongside the media shares.
**Encrypting personal shares — unRAID 7 ZFS:**
ZFS native encryption in unRAID 7 means both admins can see the share exists and file sizes but neither can read content without your passphrase or keyfile. rsync copies encrypted blocks as-is — the remote server never needs your key.
**Setup on HOST1 (your personal share):**
**Step 1 — Create an encrypted ZFS dataset:**
```
unRAID UI → Main tab → click your ZFS pool
→ Click "+ Dataset"
→ Name: Gmer4Lfe-Personal
→ Enable Encryption: Yes
→ Encryption type: passphrase (simplest) or keyfile (auto-unlock capable)
→ Enter your passphrase — write it down, if lost data is unrecoverable
→ Create
```
**Step 2 — Create the share:**
```
Settings → Shares → Add Share
→ Name: Gmer4Lfe-Personal
→ Primary storage: your ZFS pool
→ Use cache: Only (keeps data on ZFS pool, not array)
→ Add
```
**Step 3 — Verify encryption is active:**
```bash
zfs get encryption poolname/Gmer4Lfe-Personal
# Should show: encryption aes-256-gcm
```
**Step 4 — Add to Master.conf and sync:**
```bash
HOST1_PERSONAL_SHARES=(
"/mnt/user/Gmer4Lfe-Personal"
)
```
**Auto-unlock on boot (keyfile approach):**
If you want the share to mount automatically after reboot without entering a passphrase:
```bash
# Create keyfile — on HOST1 only, never sync this file
mkdir -p /root/.zfs-keys
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
chmod 400 /root/.zfs-keys/personal.key
# Set dataset to use keyfile
zfs change-key -o keylocation=file:///root/.zfs-keys/personal.key \
-o keyformat=raw poolname/Gmer4Lfe-Personal
# Add to array start script (unRAID_Essentials or User Scripts)
zfs load-key poolname/Gmer4Lfe-Personal
zfs mount poolname/Gmer4Lfe-Personal
```
**Manual unlock (most secure — you control when it's readable):**
```bash
zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal
```
**What the remote admin sees:**
The share directory exists on HOST2. File names and sizes are visible (ZFS encrypts content, not metadata by default). File contents are unreadable without your key. To hide filenames too, enable `zfs set encryption=aes-256-gcm` with `dnodesize=auto` — this is a more advanced setup.
**Current status:** The infrastructure supports encrypted personal share syncing. The ZFS dataset setup is a one-time manual step per server. Once set up it syncs automatically like any other share.
---