Varaverk/Transcodes/Manual-Transcoding.md

━━━━━ TRANSCODES — Manual ━━━━━

Configuration reference, Docker mount setup, threshold sizing, and troubleshooting for the ramdisk transcode system. Read the Docker mount section before anything else.

---

━━━ CONTENTS ━━━

• Docker Mount — Required Configuration
• The Symlink Architecture
• ramdisk_setup.sh
• transcode_manager.sh
• transcode_cleanup.sh
• Full Configuration Reference
• Schedule
• Troubleshooting

---

Output Tiers

All three scripts follow a two-tier output model: echo lines are always visible; log lines only appear when --log is passed.

ramdisk_setup.sh — one-shot at array start. Without --log, section headers and the final summary are visible. Per-step creation detail suppressed.

transcode_manager.sh — runs every 7 minutes. Without --log, only state transitions (flips, warnings, errors) and active session display are shown. When nothing changes, a single one-line confirmation is printed. Per-check detail suppressed.

transcode_cleanup.sh — runs every 7 minutes. Without --log, the cleanup summary (files removed, space freed) is always visible. Per-file deletion detail suppressed.

---

Docker Mount — Required Configuration

This is the most important configuration requirement in this folder. Get this wrong and symlink flips silently stop working after the first flip. The system appears to work initially and fails subtly.

Required Mount

Everything goes in the unRAID Docker template Extra Parameters field. Do NOT use the path mapping UI for the transcode directory — it does not support bind-propagation.

Non-GPU containers:

--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared

GPU-accelerated containers (Emby, Jellyfin with NVENC/NVDEC):

--gpus "device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1" --pids-limit=0 --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared

Replace the UUID with your GPU's UUID (nvidia-smi -L to find it). In Emby and Jellyfin's transcoding settings, set the transcode path to /ext-ram-transcode.

---

--gpus vs --runtime=nvidia

There are two ways to give a container GPU access. Always use --gpus.

--runtime=nvidia is the old approach. It requires the NVIDIA Container Toolkit configured at the Docker daemon level and spreads across multiple XML fields (runtime flag + NVIDIA_VISIBLE_DEVICES env var + NVIDIA_DRIVER_CAPABILITIES env var). When unRAID rebuilds a container from template (update, reinstall) these fields can break or get dropped — requiring manual XML repair to recover. --runtime=nvidia combined with NVIDIA_VISIBLE_DEVICES also conflicts with bind-propagation=shared on unRAID 7.2.5+ due to a kernel change in mount namespace initialization (see Troubleshooting for the full error and fix).

--gpus "device=UUID" is Docker-native GPU support (Docker 19.03+). One field in Extra Parameters. Pins a specific GPU by UUID — no ambiguity on a single-GPU system. Survives container rebuilds cleanly. Does not conflict with bind-propagation=shared.

HOST1 GPU UUID (Quadro P2000): GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1

Why shared Is Required

rprivate (Docker's default):
  Docker resolves the symlink target at first mount and locks that inode.
  Flip: ramdisk → SSD → works (new target locked in)
  Flip: SSD → ramdisk → Docker ignores it. Container still sees SSD binding.
  All sessions continue to land on SSD forever until Emby restarts.
  Symptom: symlink on host is correct, Emby still uses SSD. Confusing.

shared:
  Host mount changes propagate into the container in real time.
  Every symlink flip is immediately visible inside the container. ✅

Verify the Mount

# Check propagation — must show "shared":
docker inspect Emby | grep -A4 "ext-ram"
# Expected: "Propagation": "shared"

# Check Emby's transcode path setting:
docker exec Emby cat /config/config/encoding.xml | grep TranscodingTempPath
# Expected: /ext-ram-transcode

What NOT to Do

Do NOT add a static SSD transcode path as a second path mapping in the template:

/mnt/cache/Temp_Storage/Emby/Transcodes  →  /ssd-transcode   ← do not do this

If the SSD path is mounted inside the container, Emby can see it as an accessible transcode location. It will route sessions there independently of the symlink — bypassing the management system entirely. Sessions land on SSD regardless of symlink state. The whole system stops working.

---

The Symlink Architecture

Emby is configured to write transcodes to TRANSCODE_LINK (/mnt/ram-transcode).
TRANSCODE_LINK is a symlink — its target is managed at runtime.

Normal operation (ramdisk has headroom):
  /mnt/ram-transcode  →  /mnt/ramdisk_transcodes/      (ramdisk — fast, no wear)

Heavy load (ramdisk filling up):
  /mnt/ram-transcode  →  /mnt/cache/Temp_Storage/Emby/Transcodes/  (SSD)

Emby doesn't know this symlink exists. It writes to /mnt/ram-transcode.
ffmpeg resolves the symlink ONCE when a session starts.
After that it holds a direct reference to the actual directory.

Flipping the symlink has ZERO effect on sessions already in progress.
Only NEW sessions care about where the symlink currently points.

What Lives Where

/mnt/ramdisk_transcodes/              ← tmpfs (HOST*_RAMDISK_SIZE ceiling)
  transcoding-temp/                   ← pre-created by ramdisk_setup.sh — always here
    E0D8DC/                           ← Emby session (Live TV HLS segments)
    F1A9BB/                           ← another session

/mnt/ram-transcode                    ← symlink — managed by transcode_manager.sh
  currently points at: /mnt/ramdisk_transcodes/

/mnt/cache/Temp_Storage/Emby/Transcodes/    ← SSD fallback
  transcoding-temp/                   ← also pre-created — Emby finds ramdisk version first
    xyz789/                           ← sessions that started when ramdisk was full

---

ramdisk_setup.sh

Creates the ramdisk, SSD fallback directory, symlink, and transcoding-temp on the ramdisk. Run once at array start. Idempotent — already-mounted ramdisk exits cleanly.

What It Creates

1. RAMDISK_PATH (/mnt/ramdisk_transcodes)
   mount -t tmpfs -o size=HOST*_RAMDISK_SIZE tmpfs /mnt/ramdisk_transcodes
   tmpfs uses only as much RAM as actually needed — RAMDISK_SIZE is a ceiling.
   An empty ramdisk uses essentially zero RAM.

2. transcoding-temp/ inside the ramdisk
   Pre-created so Emby always finds it here first.
   Without this: Emby creates transcoding-temp at its first writable location,
   which may be the SSD fallback even when the symlink points at the ramdisk.
   chown nobody:users — correct ownership for Emby (PUID=99)

3. TRANSCODE_SSD (/mnt/cache/Temp_Storage/Emby/Transcodes/)
   mkdir -p — created if missing, silent if exists
   Also pre-creates transcoding-temp/ inside SSD fallback for consistency

4. TRANSCODE_LINK (/mnt/ram-transcode)
   ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
   Always reset to ramdisk at array start — clean state every boot

Verify After Setup

# All three must pass:
mountpoint /mnt/ramdisk_transcodes
# Expected: /mnt/ramdisk_transcodes is a mountpoint

readlink /mnt/ram-transcode
# Expected: /mnt/ramdisk_transcodes

ls /mnt/ramdisk_transcodes/
# Expected: transcoding-temp/

Usage

ramdisk_setup.sh              # normal run (called by array_started.sh)
ramdisk_setup.sh --dry-run    # show what would be created without creating
ramdisk_setup.sh --status     # show current ramdisk, symlink, and SSD state
ramdisk_setup.sh --log        # verbose — show each creation step

---

transcode_manager.sh

Monitors ramdisk usage and manages the symlink direction. Called second in every 3-minute cycle by transcode_management.sh.

Three Modes

# master.conf
TRANSCODE_MANAGER_MODE="smart"    # smart | ramdisk | ssd

# smart (default — use in production):
#   Ramdisk above RAMDISK_WARN_GB → flip symlink to SSD
#   Ramdisk below RAMDISK_LOW_GB  → flip symlink back to ramdisk
#   Hysteresis gap prevents flip-flopping under moderate load

# ramdisk:
#   Always uses ramdisk. Warns if usage exceeds threshold. Never flips.
#   Use for: light load server, guaranteed RAM performance, testing ramdisk behaviour.

# ssd:
#   Always uses SSD. Never uses ramdisk.
#   Use for: post-flip drain (waiting for ramdisk sessions to end naturally),
#            maintenance windows, ramdisk capacity testing.

Threshold Sizing

# host*.conf
HOST1_RAMDISK_SIZE="10G"
HOST1_RAMDISK_WARN_GB=8.8    # flip to SSD above this
HOST1_RAMDISK_LOW_GB=6.5     # flip back below this

Keep ~1.5–2.5GB hysteresis gap between WARN and LOW. Without the gap, usage hovering near WARN causes constant flip-flopping. The gap requires multiple sessions to end completely before flipping back — a genuine recovery, not a brief fluctuation.

Production data from a 7-household Live TV system:

Normal (2–3 streams)       → ~1.5–2.0 GB
Busy evening (5–6 streams) → ~3.5–4.5 GB
Peak (8 streams, Live TV)  → ~5.2 GB
Current setup: 10G ramdisk, 8.8 GB threshold → 1.2 GB safety headroom

Recommended thresholds by ramdisk size:

RAMDISK_SIZE	RAMDISK_WARN_GB	RAMDISK_LOW_GB
6G	4.8	3.5
8G	6.8	5.5
10G	8.8	6.5
12G	10.5	8.5

Multi-Server Configuration

# master.conf
TRANSCODE_SERVERS=(
    "Emby|http://localhost:8096|your-api-key|emby"

    # Temporarily cover HOST2's Emby during maintenance:
    # "Emby-Jayred365|http://100.x.x.x:8096|HOST2-api-key|emby"

    # Jellyfin instance (separate port):
    # "Jellyfin|http://localhost:8097|jellyfin-api-key|jellyfin"
)

Type field: emby | jellyfin | plex — controls which API endpoint format is used. Entries with placeholder API keys are skipped automatically. Comment out unused entries rather than deleting — placeholders show what's available.

Tdarr does NOT belong here. Tdarr encodes full video files — large working files fill the ramdisk rapidly and cause constant flips. Tdarr belongs on SSD permanently.

Session Display

━━━ Active Emby Sessions ━━━
Total: 7  |  Live TV: 5  |  Transcoding: 5  |  Direct: 2
Storage: ramdisk

Sunny     — ABC (WTAE)   — Live TV  — Transcode
Mama Bear — Cinemax      — Live TV  — Transcode
Gmer4Lfe  — WAN Show     — TV Show  — Transcode

Split state during a flip (ramdisk sessions draining, new sessions on SSD):

⚠️  Split state — 4 folder(s) on ramdisk / 2 on SSD
Storage: ramdisk (4) + SSD (2)

Usage

transcode_manager.sh              # normal run
transcode_manager.sh --dry-run    # preview flip decision without flipping
transcode_manager.sh --status     # show current state, usage, sessions, flip history
transcode_manager.sh --log        # verbose output per safety check
transcode_manager.sh --no-log     # suppress daily log write

---

transcode_cleanup.sh

Removes stale transcode files from ramdisk and SSD fallback. Called first in every 3-minute cycle — cleanup before usage measurement is non-negotiable.

Deletion Rules

A file is eligible for deletion only when ALL conditions are true:

1. Older than TRANSCODE_MAX_AGE minutes (mtime — last write time). Active segments are written every few seconds. Not touched in 20 minutes = session ended.
2. Not currently open by any process (lsof pre-built map, O(1) lookup per file). If ffmpeg has a file open, it is not deleted regardless of age.

transcoding-temp/ is never deleted, even when empty. Protected by name exclusion in the find command — deleting it causes Emby to route all sessions to the SSD fallback.

Usage

transcode_cleanup.sh              # normal cleanup run
transcode_cleanup.sh --dry-run    # show what would be deleted
transcode_cleanup.sh --status     # show file counts, ages, open-file status per location
transcode_cleanup.sh --log        # verbose per-file output

---

Full Configuration Reference

# master.conf

# ── Paths ──────────────────────────────────────────────────────────────────────
TRANSCODE_LINK="/mnt/ram-transcode"                        # symlink Emby points at
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/"   # SSD fallback

# ── Per-Host (host*.conf) ───────────────────────────────────────────────
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes"   # ramdisk mount point
HOST1_RAMDISK_SIZE="10G"                        # tmpfs ceiling (not a reservation)
HOST1_RAMDISK_WARN_GB=8.8                       # flip to SSD above this
HOST1_RAMDISK_LOW_GB=6.5                        # flip back below this

# ── Thresholds ─────────────────────────────────────────────────────────────────
RAMDISK_SSD_MIN_GB=20    # minimum SSD free space before allowing SSD flip
                          # prevents accidentally filling the SSD cache pool
TRANSCODE_FLIP_WARN=3    # notify if symlink flips this many times in one hour
                          # high flip count = ramdisk undersized for the load

# ── Cleanup ────────────────────────────────────────────────────────────────────
TRANSCODE_MAX_AGE=20     # minutes — files older than this are stale
TRANSCODE_ORPHAN_AGE=30  # minutes — orphaned session folders removed after this

# ── Manager Mode ───────────────────────────────────────────────────────────────
TRANSCODE_MANAGER_MODE="smart"    # smart | ramdisk | ssd
TRANSCODE_CHECK_EMBY=true         # skip threshold checks when Emby not running
                                   # prevents unnecessary flips overnight

# ── Permissions ────────────────────────────────────────────────────────────────
TRANSCODE_OWNER="nobody:users"    # matches PUID=99 PGID=100 container env
TRANSCODE_CHMOD="755"

# ── Daily Log ──────────────────────────────────────────────────────────────────
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
TRANSCODE_LOG_RETENTION=90        # days — trimmed on every write
TRANSCODE_STATE_FILE="/tmp/transcode_state.db"   # /tmp — resets on reboot correctly

# ── Multi-Server ───────────────────────────────────────────────────────────────
TRANSCODE_SERVERS=(
    "ContainerName|http://host:port|api-key|emby"   # type: emby | jellyfin | plex
)

---

Schedule

At Startup of Array (via array_started.sh in unRAID_Essentials/):
  ramdisk_setup.sh     — creates ramdisk, symlink, transcoding-temp
                         Must run BEFORE Emby starts

Every 7 minutes (via transcode_management.sh in Orchestrators/):
  1. transcode_cleanup.sh  — remove stale files, check flip-back
  2. transcode_manager.sh  — check usage, flip if needed, display sessions

Do NOT schedule transcode_manager.sh or transcode_cleanup.sh directly.

---

Troubleshooting

Sessions Landing on SSD Despite Symlink Pointing at Ramdisk

# Check 1 — Docker mount propagation (most common cause):
docker inspect Emby | grep Propagation
# Expected: "Propagation": "shared"
# Wrong:    "Propagation": "rprivate"
#
# Fix: Update Emby Extra Parameters, restart Emby:
# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
# Remove any standard path mapping for the transcode directory.

# Check 2 — transcoding-temp exists on ramdisk:
ls /mnt/ramdisk_transcodes/
# Expected: transcoding-temp/
#
# Fix if missing:
mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp

# Check 3 — no duplicate SSD mount in Emby template:
docker inspect Emby | grep -A3 "Mounts"
# Should show only /mnt/ram-transcode → /ext-ram-transcode
# Should NOT show /mnt/cache/Temp_Storage/... as a second mount

Flip Count High — 3+ Per Hour

# Ramdisk filling up regularly — load exceeds the current ceiling.
# Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB"
#
# If peak is close to RAMDISK_WARN_GB → increase ramdisk size:
# host1.conf
HOST1_RAMDISK_SIZE="12G"       # increase by 2G
HOST1_RAMDISK_WARN_GB=10.5    # adjust thresholds accordingly
HOST1_RAMDISK_LOW_GB=8.5

# Remount at new size — run ramdisk_setup.sh manually:
ramdisk_setup.sh --log
# Ramdisk must be unmounted first if already mounted:
# umount /mnt/ramdisk_transcodes && ramdisk_setup.sh --log

Ramdisk Not Mounting at Array Start

# Check if tmpfs is mounted:
mountpoint /mnt/ramdisk_transcodes
# "not a mountpoint" → setup failed or not run yet

# Run manually to see the error:
ramdisk_setup.sh --log

# Common causes:
#   /mnt/ramdisk_transcodes missing → mkdir -p /mnt/ramdisk_transcodes
#   Insufficient RAM → check free RAM: free -h
#   RAMDISK_SIZE too large → reduce HOST*_RAMDISK_SIZE

Emergency Manual Flip

# Flip to SSD immediately — all new sessions go to SSD:
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode

# Flip back to ramdisk — all new sessions go to ramdisk:
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode

# Check current symlink target:
readlink /mnt/ram-transcode

# Existing sessions in progress are NEVER affected — only new sessions follow the flip.

Emby Won't Start After Dirty Shutdown

After an unclean shutdown, the --mount bind-propagation entry in Extra Parameters can leave the container in a broken state where Emby refuses to start at all.

# Symptom: Emby fails to start with the --mount extra parameter present.
# Cause: dirty shutdown left the bind mount in a state Docker can't recover.
#
# Recovery:
# 1. Remove the --mount line from Emby Extra Parameters in unRAID Docker UI
# 2. Start Emby and wait for it to fully load (check the WebUI is responsive)
# 3. Add the --mount line back to Extra Parameters:
#      --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
# 4. Save changes and restart Emby
#
# The container starts cleanly without the mount, which clears the broken state.
# Re-adding the mount after a clean start works reliably.

GPU Memory Exhausted — Jellyfin Fails, Emby Falls Back to CPU

Symptoms:

• Jellyfin refuses to play anything — session fails immediately, no transcode starts
• Emby plays but transcodes on CPU (logs show libx265 or libx264 instead of h264_nvenc/hevc_nvenc)

Both symptoms can appear at the same time and have the same root cause: another process has consumed all available VRAM and not released it. The media servers respond differently to a failed GPU session init:

• Jellyfin: hard fails — no session is created, playback stops entirely
• Emby: falls back to CPU transcoding silently and keeps going

# Confirm VRAM is exhausted:
nvidia-smi

# Find what is holding it:
nvidia-smi --query-compute-apps=pid,used_memory,name --format=csv,noheader

Common source — OCR plugin sidecars: Credit detection and subtitle extraction plugins often talk to a GPU-accelerated OCR container running alongside the media server. The EmbyCredits plugin (yocksers/EmbyCredits) can be configured to use a PaddleOCR backend. PaddleOCR loads a neural network into VRAM at first use and does not release it between runs. After a single credit scan the GPU memory stays consumed, starving Emby and Jellyfin of VRAM for transcoding.

This is not obvious because PaddleOCR is a separate container — nvidia-smi shows the process, but the connection to failing playback is not immediate.

Emby startup probe — the restart trap:

Emby runs a one-shot NVIDIA hardware detection when the container starts. If VRAM is exhausted at startup, NVIDIA is marked unavailable for the entire container session — there is no retry. Emby will use CPU for all transcoding until the container is restarted, and only after VRAM has been freed. Restarting Emby while VRAM is still exhausted causes the probe to fail again and NVIDIA is disabled again.

Correct recovery sequence:

1. Free VRAM (stop the offending container/process)
2. Confirm VRAM is free: nvidia-smi — GPU memory used should drop to near zero
3. Restart Emby — startup probe now succeeds, NVENC available

Fix — switch OCR to CPU:

For EmbyCredits: use the yock1/embycreditocr Tesseract image instead of PaddleOCR. Tesseract is CPU-based, never touches VRAM, and is the plugin's own documented backend.

docker run -d \
  --name EmbyCredit-OCR \
  -p 8884:8884 \
  --restart unless-stopped \
  yock1/embycreditocr

In EmbyCredits plugin settings, set the OCR endpoint to http://localhost:8884.

The accuracy tradeoff is real (PaddleOCR is stronger on non-Latin scripts) but GPU starvation is not an acceptable failure mode for a live media server.

---

Emby With NVIDIA GPU — bind-propagation=shared Fails to Start

Regression: This worked before unRAID 7.2.5. The kernel update changed how mount namespaces are initialized, exposing a conflict in the NVIDIA container runtime path.

--runtime=nvidia combined with a NVIDIA_VISIBLE_DEVICES environment variable conflicts with bind-propagation=shared. Docker fails during container init with:

OCI runtime create failed: runc create failed: unable to start container process:
error during container init: error jailing process inside rootfs:
open /proc/self/mountinfo: no such file or directory

--runtime=nvidia alone (no NVIDIA_VISIBLE_DEVICES env var) does NOT cause this. The conflict requires both.

Root cause: NVIDIA_VISIBLE_DEVICES triggers the NVIDIA container runtime to bind mount CUDA libraries and device nodes into the container namespace. This GPU device setup interferes with the mount namespace initialization that bind-propagation=shared requires.

Fix — replace the NVIDIA env var approach with the --gpus flag:

In the unRAID Docker template for Emby:

1. Extra Parameters — remove --runtime=nvidia, add --gpus with the GPU UUID:

   --gpus device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1 --pids-limit=0 \
   --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
   

2. Variables — remove the NVIDIA_VISIBLE_DEVICES variable entirely. The --gpus flag handles GPU assignment without the env var.

--gpus device=UUID uses Docker's native GPU device flag (default runtime) rather than the full NVIDIA container runtime setup. The container still gets GPU access; it just skips the device mount phase that conflicts with bind-propagation=shared.

Increasing Ramdisk Size After Initial Setup

# 1. Set new size and thresholds in host*.conf
# 2. Unmount the existing ramdisk (no sessions should be active):
umount /mnt/ramdisk_transcodes

# 3. Re-run setup to mount at new size:
ramdisk_setup.sh --log

# 4. Verify:
df -h /mnt/ramdisk_transcodes
# Should show new size as total