724 lines
39 KiB
Markdown
724 lines
39 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
# 🎬 TRANSCODING
|
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
**Ramdisk-based transcode storage with automatic SSD fallback, session-safe symlink
|
|
flipping, and stale file cleanup.** Emby transcodes to RAM at full speed. If the ramdisk
|
|
fills up, new sessions automatically shift to SSD — without interrupting anything
|
|
already playing. When pressure drops, new sessions shift back.
|
|
|
|
> **This system has two subtle configuration requirements that are not obvious and
|
|
> both were discovered the hard way in production.** The Docker mount must use
|
|
> `bind-propagation=shared` or symlink flips are silently ignored after the first
|
|
> flip. The `transcoding-temp` directory must be pre-created on the ramdisk or Emby
|
|
> finds the SSD version and routes all sessions there until restarted. Both are
|
|
> documented and both will bite you if missed.
|
|
|
|
---
|
|
|
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### 🔴 Three Storage Options, None Perfect on Their Own
|
|
|
|
Emby transcodes generate hundreds of small HLS segment files written and read
|
|
continuously at high throughput. Where those files live matters a lot:
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Hard drives:
|
|
# Seek times cause buffering on simultaneous multi-stream transcoding.
|
|
# 5 streams trying to seek on spinning disks = constant buffering for everyone.
|
|
#
|
|
# SSD (cache pool):
|
|
# Fast enough for any realistic load.
|
|
# But: constant small file writes at Emby volume accelerate SSD wear.
|
|
# A busy Live TV night writes and deletes thousands of segment files.
|
|
# Over months, this adds up.
|
|
#
|
|
# RAM (tmpfs):
|
|
# Fastest possible — no disk I/O at all.
|
|
# No wear — RAM doesn't have write cycles.
|
|
# Files disappear instantly on session end — no cleanup needed for normal exits.
|
|
# One risk: running out of RAM during heavy load.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
The correct answer is RAM by default, SSD as a safety net. The ramdisk handles normal
|
|
operation. The SSD absorbs unexpected load spikes. The system manages the transition
|
|
automatically.
|
|
|
|
---
|
|
|
|
### 🔴 Changing Transcode Location Requires Restarting Emby
|
|
|
|
The obvious approach — configure Emby to use the ramdisk, configure SSD as fallback in
|
|
Emby's settings — requires restarting Emby to switch between them. Restarting Emby
|
|
during active streams drops everyone. A 7-person household with 5 Live TV streams
|
|
running at 9pm is not a good moment to restart Emby.
|
|
|
|
The fix: symlink indirection. Emby points at a fixed path that never changes.
|
|
The symlink target changes. ffmpeg resolves the symlink once at session start and
|
|
holds the resolved path — existing sessions are completely unaffected by symlink
|
|
changes. Only new sessions care about where the symlink currently points.
|
|
|
|
The transition is seamless. Sessions in progress when the flip happens continue
|
|
writing to wherever they started. Only new sessions after the flip go to the new
|
|
location. No restart. No interruption.
|
|
|
|
---
|
|
|
|
### 🔴 Docker Bind Mount Silently Ignored After First Flip
|
|
|
|
Got the symlink system working. First flip from ramdisk to SSD: works. Flip back to
|
|
ramdisk: nothing. All new sessions still land on SSD. The symlink on the host clearly
|
|
points at the ramdisk — `readlink /mnt/ram-transcode` shows the correct path — but
|
|
Emby keeps writing to SSD.
|
|
|
|
The cause: Docker's default bind mount uses `rprivate` propagation. With `rprivate`,
|
|
Docker resolves the symlink target at first mount and locks that inode. When the
|
|
symlink flips, Docker ignores it — it already has a private SSD binding locked in.
|
|
The container sees the path but the mount behind it doesn't update.
|
|
|
|
The fix: `bind-propagation=shared` in Extra Parameters. With shared propagation, host
|
|
mount changes propagate into the container in real time. Symlink flips on the host are
|
|
immediately visible inside the container. This requires using `--mount` syntax instead
|
|
of a standard template path mapping — that syntax supports propagation, the path
|
|
mapping UI does not.
|
|
|
|
---
|
|
|
|
### 🔴 Sessions Drifting to SSD After a Day of Operation
|
|
|
|
System working correctly for hours. Then gradually sessions start landing on SSD even
|
|
though the ramdisk has plenty of space and the symlink points at the ramdisk. Next day
|
|
all sessions are on SSD.
|
|
|
|
The cause: cleanup was removing the empty `transcoding-temp` directory from the
|
|
ramdisk. When `transcoding-temp` doesn't exist on the ramdisk, Emby searches its
|
|
accessible paths for an existing one. It finds the SSD fallback version. All subsequent
|
|
sessions route there until Emby is restarted.
|
|
|
|
The fix: `transcoding-temp` is protected from cleanup — excluded by name from the
|
|
`find` command. And `ramdisk_setup.sh` pre-creates it at mount time so Emby always
|
|
finds it on the ramdisk first. Both fixes together prevent this permanently.
|
|
|
|
---
|
|
|
|
### 🔴 lsof Per File on a Live TV System
|
|
|
|
Early cleanup implementation called `lsof filename` per file to check if anything had
|
|
it open. On a busy Live TV night with 5 simultaneous streams, the ramdisk contains
|
|
thousands of HLS segment files. Calling `lsof` once per file was creating thousands
|
|
of subprocess calls every 3 minutes. The cleanup script was spending more time on
|
|
lsof calls than on actual cleanup.
|
|
|
|
The fix: `lsof` is called once per location to build a complete open-file map. All
|
|
subsequent file checks are O(1) lookups against that pre-built map. Thousands of files,
|
|
one `lsof` call, no performance penalty.
|
|
|
|
---
|
|
|
|
## ━━━ THE DESIGN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
### ── The Symlink Architecture ─────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Emby is configured to write transcodes to TRANSCODE_LINK.
|
|
# TRANSCODE_LINK is a symlink — its target is managed at runtime.
|
|
#
|
|
# Normal operation (ramdisk has headroom):
|
|
# /mnt/ram-transcode → /mnt/ramdisk_transcodes/ (ramdisk, fast, no wear)
|
|
#
|
|
# Heavy load (ramdisk filling up):
|
|
# /mnt/ram-transcode → /mnt/cache/Temp_Storage/Emby/Transcodes/ (SSD)
|
|
#
|
|
# Emby doesn't know this symlink exists. It writes to /mnt/ram-transcode.
|
|
# ffmpeg resolves the symlink ONCE when a session starts.
|
|
# After that it holds a direct reference to the actual directory.
|
|
# Flipping the symlink has ZERO effect on sessions already in progress.
|
|
# Only NEW sessions care about where the symlink currently points.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── The Threshold Logic ──────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Two thresholds with a hysteresis gap to prevent flip-flopping.
|
|
#
|
|
RAMDISK_WARN_GB=8.8 # flip to SSD when ramdisk usage exceeds this
|
|
RAMDISK_LOW_GB=6.5 # flip back to ramdisk when usage drops below this
|
|
#
|
|
# The 2.3GB hysteresis gap:
|
|
# Without this gap: usage hovers at 8.7GB → flip to SSD → sessions drain
|
|
# → usage drops to 8.5GB → flip back → new sessions fill → flip again
|
|
# The symlink would oscillate every few minutes under moderate load.
|
|
#
|
|
# With the gap: usage must drop all the way to 6.5GB before flipping back.
|
|
# That requires multiple sessions to end completely — a genuine recovery,
|
|
# not a brief fluctuation. Stable, predictable behaviour.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── What Lives Where ──────────────────────────────────────────────────────────
|
|
|
|
```
|
|
/mnt/ramdisk_transcodes/ ← tmpfs (HOST*_RAMDISK_SIZE ceiling)
|
|
transcoding-temp/ ← pre-created by ramdisk_setup.sh — always here
|
|
E0D8DC/ ← Emby session (Live TV HLS segments)
|
|
F1A9BB/ ← another session
|
|
|
|
/mnt/ram-transcode ← symlink — managed at runtime by transcode_manager.sh
|
|
currently points at: /mnt/ramdisk_transcodes/
|
|
|
|
/mnt/cache/Temp_Storage/Emby/Transcodes/ ← SSD fallback
|
|
transcoding-temp/ ← also pre-created — Emby finds ramdisk version first
|
|
xyz789/ ← sessions that started when ramdisk was full
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## ⚙️ DOCKER MOUNT — READ THIS BEFORE ANYTHING ELSE
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
> **This is the most important configuration requirement in the entire folder.**
|
|
> Get this wrong and symlink flips silently stop working after the first flip.
|
|
> The system appears to work initially and fails subtly.
|
|
|
|
---
|
|
|
|
### ── Required Mount Configuration ───────────────────────────────────────────
|
|
|
|
```bash
|
|
# In Emby's Extra Parameters in the unRAID Docker template:
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# This REPLACES the transcode path in the standard template path mapping UI.
|
|
# Do NOT add this via the path mapping UI — that UI does not support propagation.
|
|
# Use Extra Parameters only.
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why `shared` Is Required ────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# rprivate (Docker's default):
|
|
# Docker resolves the symlink target at first mount and locks that inode.
|
|
# Flip: ramdisk → SSD → flip works.
|
|
# Flip: SSD → ramdisk → Docker ignores it. Container still sees SSD binding.
|
|
# All sessions continue to land on SSD forever until Emby restarts.
|
|
# Symptom: symlink on host is correct, Emby still uses SSD. Confusing.
|
|
#
|
|
# shared:
|
|
# Host mount changes propagate into the container in real time.
|
|
# Flip: ramdisk → SSD → visible in container immediately ✅
|
|
# Flip: SSD → ramdisk → visible in container immediately ✅
|
|
# System works as designed. Every flip is immediate and correct.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Verify the Mount ─────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Check the propagation — must show "shared" not "rprivate":
|
|
docker inspect Emby | grep -A4 "ext-ram"
|
|
# Expected output includes: "Propagation": "shared"
|
|
# Wrong output: "Propagation": "rprivate"
|
|
|
|
# Check Emby's transcode path setting (inside container):
|
|
docker exec Emby cat /config/config/encoding.xml | grep TranscodingTempPath
|
|
# Expected: /ext-ram-transcode
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── What NOT to Do ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# DO NOT add a static SSD transcode path as a second path mapping in the template:
|
|
# /mnt/cache/Temp_Storage/Emby/Transcodes → /ssd-transcode
|
|
#
|
|
# If the SSD path is mounted inside the container, Emby can see it as an
|
|
# accessible transcode location. It will route sessions there independently of
|
|
# the symlink — completely bypassing the management system. Sessions land on SSD
|
|
# regardless of symlink state. The whole system stops working.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🚀 ramdisk_setup.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Creates the ramdisk, SSD fallback directory, symlink, and `transcoding-temp` on the
|
|
ramdisk. Run once at array start. Idempotent — if the ramdisk is already mounted it
|
|
reports status and exits cleanly.
|
|
|
|
```bash
|
|
# Scheduled: At Startup of Array (via array_start.sh)
|
|
# This runs BEFORE Emby starts — order matters in ARRAY_START_SCRIPTS
|
|
```
|
|
|
|
---
|
|
|
|
### ── What It Creates ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# 1. RAMDISK_PATH (/mnt/ramdisk_transcodes)
|
|
# mount -t tmpfs -o size=HOST*_RAMDISK_SIZE tmpfs /mnt/ramdisk_transcodes
|
|
# In-memory tmpfs — uses only as much RAM as actually needed.
|
|
# RAMDISK_SIZE is a ceiling, not a reservation — an empty ramdisk uses ~0 RAM.
|
|
#
|
|
# 2. transcoding-temp inside the ramdisk
|
|
# mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
|
|
# Pre-created so Emby always finds it here first.
|
|
# Without this: Emby creates transcoding-temp at its first writable location,
|
|
# which may be the SSD fallback even when the symlink points at the ramdisk.
|
|
# chown nobody:users — correct ownership for Emby to write as PUID=99
|
|
#
|
|
# 3. TRANSCODE_SSD (/mnt/cache/Temp_Storage/Emby/Transcodes/)
|
|
# mkdir -p — creates if missing, silent if exists
|
|
# Also pre-creates transcoding-temp/ inside SSD fallback for consistency
|
|
#
|
|
# 4. TRANSCODE_LINK (/mnt/ram-transcode)
|
|
# ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
|
|
# Always reset to ramdisk at array start — clean state every boot
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Verify After Setup ───────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# Run these after first setup — all three must pass:
|
|
|
|
mountpoint /mnt/ramdisk_transcodes
|
|
# Expected: /mnt/ramdisk_transcodes is a mountpoint
|
|
|
|
readlink /mnt/ram-transcode
|
|
# Expected: /mnt/ramdisk_transcodes
|
|
|
|
ls /mnt/ramdisk_transcodes/
|
|
# Expected: transcoding-temp/
|
|
```
|
|
|
|
---
|
|
|
|
### ── Usage ───────────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
ramdisk_setup.sh # normal run (at array start via array_start.sh)
|
|
ramdisk_setup.sh --dry-run # show what would be created without creating
|
|
ramdisk_setup.sh --status # show current ramdisk, symlink, and SSD state
|
|
ramdisk_setup.sh --log # verbose — show each creation step
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🔄 transcode_manager.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Monitors ramdisk usage and manages the symlink direction. Called by
|
|
`transcode_management.sh` — not scheduled directly. Every 3 minutes it checks usage,
|
|
makes a flip decision if needed, runs safety checks, and shows active sessions.
|
|
|
|
```bash
|
|
# Called by: transcode_management.sh (every 3 minutes)
|
|
# Not scheduled directly — use transcode_management.sh
|
|
```
|
|
|
|
---
|
|
|
|
### ── Three Modes ──────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
|
#
|
|
# smart:
|
|
# Auto-flips between ramdisk and SSD based on usage thresholds.
|
|
# Normal operation — use this in production.
|
|
# Ramdisk above RAMDISK_WARN_GB → flip to SSD.
|
|
# Ramdisk below RAMDISK_LOW_GB → flip back to ramdisk.
|
|
#
|
|
# ramdisk:
|
|
# Always uses ramdisk. Never flips to SSD.
|
|
# Use: light load server, guaranteed RAM performance, testing ramdisk behaviour.
|
|
# Warning logged if usage exceeds threshold — no automatic action.
|
|
#
|
|
# ssd:
|
|
# Always uses SSD. Never uses ramdisk.
|
|
# Use: post-flip drain (waiting for ramdisk sessions to end naturally),
|
|
# maintenance windows, ramdisk capacity testing.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Safety Checks — Every Run Regardless of Mode ───────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# These run on EVERY cycle — they protect against state drift:
|
|
#
|
|
# Symlink missing or broken
|
|
# → Recreate pointing at ramdisk, notify
|
|
# → Can happen after manual intervention or filesystem issue
|
|
#
|
|
# Ramdisk disappeared (unmounted)
|
|
# → Auto-flip to SSD immediately, notify warning
|
|
# → Can happen if tmpfs was manually unmounted or system ran out of memory
|
|
#
|
|
# SSD path missing
|
|
# → Disable SSD fallback (mode=ssd: error)
|
|
# → Can happen if SSD pool is not mounted
|
|
#
|
|
# transcoding-temp missing from ramdisk
|
|
# → Recreate immediately, no notification
|
|
# → Prevents sessions silently routing to SSD version
|
|
#
|
|
# Permissions drift
|
|
# → Fix silently every run
|
|
# → nobody:users ownership, TRANSCODE_CHMOD mode
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Session Display ──────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Each run shows active sessions from all configured TRANSCODE_SERVERS:
|
|
#
|
|
━━━ 🎬 Active Emby Sessions ━━━
|
|
🎬 Total: 7 | 💨 Live TV: 5 | 🔄 Transcoding: 5 | 🏁 Direct: 2
|
|
🔗 Storage: 💨 ramdisk
|
|
|
|
🎬 Sunny — ABC (WTAE) — Live TV — Transcode
|
|
🎬 Mama Bear — Cinemax — Live TV — Transcode
|
|
🎬 Gmer4Lfe — WAN Show — TV Show — Transcode
|
|
|
|
# Split state — sessions on both ramdisk and SSD simultaneously:
|
|
# Normal during a flip — ramdisk sessions draining, new sessions on SSD
|
|
#
|
|
⚠️ Split state — 4 folder(s) on ramdisk / 2 on SSD
|
|
🔗 Storage: 💨 ramdisk (4) + 💾 SSD (2)
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Multi-Server Configuration ─────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Any number of media servers can share the same ramdisk scratch space.
|
|
# They never touch each other's files — each writes to its own session subfolder.
|
|
# Format: "ContainerName|URL|APIKey|Type"
|
|
#
|
|
TRANSCODE_SERVERS=(
|
|
"Emby|http://localhost:8096|0c27448d93a7431f9ac63569f7655829|emby"
|
|
|
|
# Temporarily cover HOST2's Emby during maintenance:
|
|
# "Emby-Jayred365|http://100.x.x.x:8096|HOST2-api-key|emby"
|
|
|
|
# Jellyfin instance (separate port):
|
|
# "Jellyfin|http://localhost:8097|jellyfin-api-key|jellyfin"
|
|
)
|
|
#
|
|
# Type field controls which API endpoint format is used:
|
|
# emby → /Sessions endpoint
|
|
# jellyfin → /Sessions endpoint (same format, same code path)
|
|
# plex → /status/sessions (different format)
|
|
#
|
|
# ⚠️ Tdarr does NOT belong here.
|
|
# Tdarr encodes full video files — large working files would fill the ramdisk
|
|
# rapidly and cause constant flips. Tdarr belongs on SSD permanently.
|
|
# dedicated tdarr_cleanup.sh handles Tdarr orphan management separately.
|
|
#
|
|
# Entries with placeholder API keys are skipped automatically.
|
|
# Comment out unused entries rather than deleting — placeholders show what's available.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Sizing the Ramdisk ───────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# master_host1.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# tmpfs uses only as much RAM as actually needed — RAMDISK_SIZE is a ceiling.
|
|
# An empty ramdisk uses essentially zero RAM.
|
|
#
|
|
HOST1_RAMDISK_SIZE="10G" # 10GB ceiling — verified against production usage below
|
|
|
|
# Production data from this setup (7-household Live TV system):
|
|
# Normal (2-3 streams) → ~1.5-2.0GB
|
|
# Busy evening (5-6 streams) → ~3.5-4.5GB
|
|
# Peak (8 streams, Live TV) → ~5.2GB
|
|
# Current ramdisk → 10G with 8.8GB threshold → 1.2GB safety headroom
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# Threshold sizing — keep ~1.5-2.5GB hysteresis gap:
|
|
#
|
|
# ┌──────────────┬─────────────────┬────────────────┐
|
|
# │ RAMDISK_SIZE │ RAMDISK_WARN_GB │ RAMDISK_LOW_GB │
|
|
# ├──────────────┼─────────────────┼────────────────┤
|
|
# │ 6G │ 4.8 │ 3.5 │
|
|
# │ 8G │ 6.8 │ 5.5 │
|
|
# │ 10G │ 8.8 │ 6.5 │
|
|
# │ 12G │ 10.5 │ 8.5 │
|
|
# └──────────────┴─────────────────┴────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## 🧹 transcode_cleanup.sh
|
|
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
Removes stale transcode files from both ramdisk and SSD fallback locations. Called by
|
|
`transcode_management.sh` before `transcode_manager.sh` — order is critical.
|
|
|
|
```bash
|
|
# Called by: transcode_management.sh (cleanup runs BEFORE manager)
|
|
# Not scheduled directly — use transcode_management.sh
|
|
```
|
|
|
|
---
|
|
|
|
### ── Deletion Rules ────────────────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# A file is eligible for deletion only when ALL conditions are true:
|
|
#
|
|
# 1. Older than TRANSCODE_MAX_AGE minutes (mtime — last write time)
|
|
# Active segments are being written every few seconds.
|
|
# A file not touched in 20 minutes is from a session that ended.
|
|
#
|
|
# 2. Not currently open by any process
|
|
# lsof map built once per location — O(1) lookup per file.
|
|
# If ffmpeg has a file open, it is not deleted regardless of age.
|
|
# "Session ended in API but ffmpeg still writing" → safe, not deleted.
|
|
#
|
|
# transcoding-temp directory:
|
|
# NEVER deleted, even when empty.
|
|
# Protected by name exclusion in find command.
|
|
# Deleting it causes Emby to find the SSD version and route there permanently.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Why Not Session-Aware Cleanup ────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# ffmpeg generates its own folder names inside transcoding-temp, independently
|
|
# of the media server API session IDs. There is no reliable mapping between
|
|
# API session IDs and the actual folder names on disk.
|
|
#
|
|
# Attempting to correlate them: session E0D8DC → folder E0D8DC → safe assumption?
|
|
# No. The folder name is an internal ffmpeg identifier. It may match, may not.
|
|
# Using this correlation would falsely treat active sessions as ended.
|
|
#
|
|
# lsof is the correct check:
|
|
# If ffmpeg has a file open, the file is active regardless of session state.
|
|
# If no process has the file open, the file is safely deletable.
|
|
# No correlation needed. No race condition. Always correct.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
### ── Flip-Back After Cleanup ──────────────────────────────────────────────────
|
|
|
|
```bash
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# transcode_cleanup.sh checks ramdisk usage AFTER removing stale files.
|
|
# If usage dropped below RAMDISK_LOW_GB → triggers flip-back to ramdisk.
|
|
# This handles the recovery direction so transcode_manager.sh doesn't have to.
|
|
#
|
|
# Without cleanup running first, the manager would see inflated usage from stale
|
|
# files and potentially flip to SSD unnecessarily.
|
|
# With cleanup running first, the manager always sees real active session usage.
|
|
# This is why the order in transcode_management.sh is non-negotiable.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```bash
|
|
# master.conf
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
# ── Paths ──────────────────────────────────────────────────────────────────
|
|
TRANSCODE_LINK="/mnt/ram-transcode" # the symlink Emby points at
|
|
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback
|
|
|
|
# ── Per-Host (master_host*.conf) ──────────────────────────────────────────
|
|
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes" # ramdisk mount point
|
|
HOST1_RAMDISK_SIZE="10G" # tmpfs ceiling (not a reservation)
|
|
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
|
|
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
|
|
|
|
# ── Thresholds ─────────────────────────────────────────────────────────────
|
|
RAMDISK_SSD_MIN_GB=20 # minimum SSD free space before allowing SSD flip
|
|
# prevents filling the SSD cache pool accidentally
|
|
TRANSCODE_FLIP_WARN=3 # notify if symlink flips this many times in an hour
|
|
# high flip count = ramdisk undersized
|
|
|
|
# ── Cleanup ────────────────────────────────────────────────────────────────
|
|
TRANSCODE_MAX_AGE=20 # minutes — files older than this are stale
|
|
TRANSCODE_ORPHAN_AGE=30 # minutes — orphaned session folders removed after this
|
|
|
|
# ── Manager Mode ───────────────────────────────────────────────────────────
|
|
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
|
TRANSCODE_CHECK_EMBY=true # skip threshold checks when Emby not running
|
|
# prevents unnecessary flips at night
|
|
|
|
# ── Permissions ────────────────────────────────────────────────────────────
|
|
TRANSCODE_OWNER="nobody:users" # matches PUID=99 PGID=100 container env
|
|
TRANSCODE_CHMOD="755"
|
|
|
|
# ── Daily Log ──────────────────────────────────────────────────────────────
|
|
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
|
|
TRANSCODE_LOG_RETENTION=90 # days — bounded, trimmed on every write
|
|
TRANSCODE_STATE_FILE="/tmp/transcode_state.db" # /tmp — resets on reboot
|
|
|
|
# ── Multi-Server ───────────────────────────────────────────────────────────
|
|
TRANSCODE_SERVERS=(
|
|
"ContainerName|http://host:port|api-key|type" # emby|jellyfin|plex
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
```
|
|
At Startup of Array:
|
|
ramdisk_setup.sh ← via array_start.sh — creates ramdisk + symlink + transcoding-temp
|
|
|
|
Every 3 minutes:
|
|
transcode_management.sh ← cleanup first, then manager — order non-negotiable
|
|
1. transcode_cleanup.sh ← remove stale files, check if flip-back possible
|
|
2. transcode_manager.sh ← check usage, flip if needed, show sessions
|
|
|
|
Do NOT schedule transcode_manager.sh or transcode_cleanup.sh directly.
|
|
```
|
|
|
|
---
|
|
|
|
## ━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
---
|
|
|
|
### 🔴 Sessions Landing on SSD Despite Symlink Pointing at Ramdisk
|
|
|
|
```bash
|
|
# Check 1 — Docker mount propagation (most common cause):
|
|
docker inspect Emby | grep Propagation
|
|
# Expected: "Propagation": "shared"
|
|
# Wrong: "Propagation": "rprivate"
|
|
#
|
|
# Fix: Add to Emby Extra Parameters and restart Emby:
|
|
# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
|
|
# Remove any standard path mapping for the transcode directory.
|
|
|
|
# Check 2 — transcoding-temp exists on ramdisk:
|
|
ls /mnt/ramdisk_transcodes/
|
|
# Expected: transcoding-temp/
|
|
#
|
|
# Fix if missing:
|
|
mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
|
|
chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp
|
|
|
|
# Check 3 — no duplicate SSD mount in Emby template:
|
|
docker inspect Emby | grep -A3 "Mounts"
|
|
# Should show: only /mnt/ram-transcode → /ext-ram-transcode
|
|
# Should NOT show: /mnt/cache/Temp_Storage/... as a second mount
|
|
```
|
|
|
|
---
|
|
|
|
### 🔴 Flip Count High — 3+ Per Hour
|
|
|
|
```bash
|
|
# Ramdisk filling up regularly — sessions draining before more arrive.
|
|
# Check peak usage from the weekly coffee report:
|
|
# Transcodes section → "Week peak: X.XGB"
|
|
#
|
|
# If peak is close to RAMDISK_WARN_GB → increase ramdisk:
|
|
# master_host1.conf
|
|
HOST1_RAMDISK_SIZE="12G" # increase by 2G
|
|
HOST1_RAMDISK_WARN_GB=10.5 # adjust thresholds accordingly
|
|
HOST1_RAMDISK_LOW_GB=8.5
|
|
#
|
|
# Then re-run ramdisk_setup.sh to remount at new size:
|
|
bash /mnt/user/appdata/unraid_scripts/Transcodes/ramdisk_setup.sh
|
|
```
|
|
|
|
---
|
|
|
|
### 🔴 Ramdisk Not Mounting at Array Start
|
|
|
|
```bash
|
|
# Check if tmpfs mounted:
|
|
mountpoint /mnt/ramdisk_transcodes
|
|
# "not a mountpoint" → setup failed or not run yet
|
|
|
|
# Run manually to see the error:
|
|
bash /mnt/user/appdata/unraid_scripts/Transcodes/ramdisk_setup.sh --log
|
|
|
|
# Common causes:
|
|
# /mnt/ramdisk_transcodes directory missing → mkdir -p /mnt/ramdisk_transcodes
|
|
# Insufficient RAM → check free RAM: free -h
|
|
# RAMDISK_SIZE too large for available RAM → reduce HOST*_RAMDISK_SIZE
|
|
```
|
|
|
|
---
|
|
|
|
### 🔴 Emergency Manual Flip
|
|
|
|
```bash
|
|
# Flip to SSD immediately — all new sessions go to SSD:
|
|
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
|
|
|
|
# Flip back to ramdisk — all new sessions go to ramdisk:
|
|
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
|
|
|
|
# Check current symlink target:
|
|
readlink /mnt/ram-transcode
|
|
|
|
# Existing sessions in progress are NEVER affected by these changes.
|
|
# Only new sessions follow the new symlink target.
|
|
``` |