Varaverk/Transcodes/Manual-Transcoding.md

━━━━━ TRANSCODES — Manual ━━━━━

Configuration reference, Docker mount setup, threshold sizing, and troubleshooting for the ramdisk transcode system. Read the Docker mount section before anything else.

---

━━━ CONTENTS ━━━

• Docker Mount — Required Configuration
• The Symlink Architecture
• ramdisk_setup.sh
• transcode_manager.sh
• transcode_cleanup.sh
• Full Configuration Reference
• Schedule
• Troubleshooting

---

Output Tiers

All three scripts follow a two-tier output model: echo lines are always visible; log lines only appear when --log is passed.

ramdisk_setup.sh — one-shot at array start. Without --log, section headers and the final summary are visible. Per-step creation detail suppressed.

transcode_manager.sh — runs every 7 minutes. Without --log, only state transitions (flips, warnings, errors) and active session display are shown. When nothing changes, a single one-line confirmation is printed. Per-check detail suppressed.

transcode_cleanup.sh — runs every 7 minutes. Without --log, the cleanup summary (files removed, space freed) is always visible. Per-file deletion detail suppressed.

---

Docker Mount — Required Configuration

This is the most important configuration requirement in this folder. Get this wrong and symlink flips silently stop working after the first flip. The system appears to work initially and fails subtly.

Required Mount

Everything goes in the unRAID Docker template Extra Parameters field. Do NOT use the path mapping UI for the transcode directory — it does not support --mount syntax.

Non-GPU containers:

--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode

GPU-accelerated containers (Emby, Jellyfin with NVENC/NVDEC):

--gpus "device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1" --pids-limit=0 --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode

⚠️ Unraid 7.3+ / Docker 29.x / runc v1.3.5+: Do NOT add bind-propagation=shared. runc v1.3.5 crashes on it with open /proc/self/mountinfo: no such file or directory. The ramdisk tmpfs is already MS_SHARED at the kernel level — propagation is automatic.

Replace the UUID with your GPU's UUID (nvidia-smi -L to find it). In Emby and Jellyfin's transcoding settings, set the transcode path to /ext-ram-transcode.

---

--gpus vs --runtime=nvidia

There are two ways to give a container GPU access. Always use --gpus.

--runtime=nvidia is the old approach. It requires the NVIDIA Container Toolkit configured at the Docker daemon level and spreads across multiple XML fields (runtime flag + NVIDIA_VISIBLE_DEVICES env var + NVIDIA_DRIVER_CAPABILITIES env var). When unRAID rebuilds a container from template (update, reinstall) these fields can break or get dropped — requiring manual XML repair to recover. --runtime=nvidia combined with NVIDIA_VISIBLE_DEVICES breaks on container rebuilds and has historically conflicted with bind mount propagation settings.

--gpus "device=UUID" is Docker-native GPU support (Docker 19.03+). One field in Extra Parameters. Pins a specific GPU by UUID — no ambiguity on a single-GPU system. Survives container rebuilds cleanly.

HOST1 GPU UUID (Quadro P2000): GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1

Mount Propagation — Unraid 7.3+ / Docker 29.x

Do NOT use bind-propagation=shared on Unraid 7.3+ (Docker 29.x / runc v1.3.5+). runc v1.3.5 rejects it with open /proc/self/mountinfo: no such file or directory and the container fails to start. This applies to all containers — GPU and non-GPU alike.

The ramdisk tmpfs is mounted MS_SHARED at the kernel level (shared:N in /proc/self/mountinfo). Bind mounts from it inherit shared propagation automatically — specifying it in Docker is redundant and breaks on runc v1.3.5+.

Historical context (Unraid ≤ 7.2.5): bind-propagation=shared was required because Docker's default rprivate resolves the symlink target at mount time and locks that inode, making subsequent flips invisible to the container. On Unraid 7.3+ this option crashes container start entirely. The ramdisk's kernel-level MS_SHARED makes it unnecessary.

Verify the Mount

# Check the mount is present:
docker inspect Emby | grep -A4 "ext-ram"
# Expected: "Destination": "/ext-ram-transcode"

# Check Emby's transcode path setting:
docker exec Emby cat /config/config/encoding.xml | grep TranscodingTempPath
# Expected: /ext-ram-transcode

What NOT to Do

Do NOT add a static SSD transcode path as a second path mapping in the template:

/mnt/cache/Temp_Storage/Emby/Transcodes  →  /ssd-transcode   ← do not do this

If the SSD path is mounted inside the container, Emby can see it as an accessible transcode location. It will route sessions there independently of the symlink — bypassing the management system entirely. Sessions land on SSD regardless of symlink state. The whole system stops working.

---

The Symlink Architecture

Emby is configured to write transcodes to TRANSCODE_LINK (/mnt/ram-transcode).
TRANSCODE_LINK is a symlink — its target is managed at runtime.

Normal operation (ramdisk has headroom):
  /mnt/ram-transcode  →  /mnt/ramdisk_transcodes/      (ramdisk — fast, no wear)

Heavy load (ramdisk filling up):
  /mnt/ram-transcode  →  /mnt/cache/Temp_Storage/Emby/Transcodes/  (SSD)

Emby doesn't know this symlink exists. It writes to /mnt/ram-transcode.
ffmpeg resolves the symlink ONCE when a session starts.
After that it holds a direct reference to the actual directory.

Flipping the symlink has ZERO effect on sessions already in progress.
Only NEW sessions care about where the symlink currently points.

What Lives Where

/mnt/ramdisk_transcodes/              ← tmpfs (HOST*_RAMDISK_SIZE ceiling)
  transcoding-temp/                   ← pre-created by ramdisk_setup.sh — always here
    E0D8DC/                           ← Emby session (Live TV HLS segments)
    F1A9BB/                           ← another session

/mnt/ram-transcode                    ← symlink — managed by transcode_manager.sh
  currently points at: /mnt/ramdisk_transcodes/

/mnt/cache/Temp_Storage/Emby/Transcodes/    ← SSD fallback
  transcoding-temp/                   ← also pre-created — Emby finds ramdisk version first
    xyz789/                           ← sessions that started when ramdisk was full

---

ramdisk_setup.sh

Creates the ramdisk, SSD fallback directory, symlink, and transcoding-temp on the ramdisk. Run once at array start. Idempotent — already-mounted ramdisk exits cleanly.

What It Creates

1. RAMDISK_PATH (/mnt/ramdisk_transcodes)
   mount -t tmpfs -o size=HOST*_RAMDISK_SIZE tmpfs /mnt/ramdisk_transcodes
   tmpfs uses only as much RAM as actually needed — RAMDISK_SIZE is a ceiling.
   An empty ramdisk uses essentially zero RAM.

2. transcoding-temp/ inside the ramdisk
   Pre-created so Emby always finds it here first.
   Without this: Emby creates transcoding-temp at its first writable location,
   which may be the SSD fallback even when the symlink points at the ramdisk.
   chown nobody:users — correct ownership for Emby (PUID=99)

3. TRANSCODE_SSD (/mnt/cache/Temp_Storage/Emby/Transcodes/)
   mkdir -p — created if missing, silent if exists
   Also pre-creates transcoding-temp/ inside SSD fallback for consistency

4. TRANSCODE_LINK (/mnt/ram-transcode)
   ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
   Always reset to ramdisk at array start — clean state every boot

Verify After Setup

# All three must pass:
mountpoint /mnt/ramdisk_transcodes
# Expected: /mnt/ramdisk_transcodes is a mountpoint

readlink /mnt/ram-transcode
# Expected: /mnt/ramdisk_transcodes

ls /mnt/ramdisk_transcodes/
# Expected: transcoding-temp/

Usage

ramdisk_setup.sh              # normal run (called by array_started.sh)
ramdisk_setup.sh --dry-run    # show what would be created without creating
ramdisk_setup.sh --status     # show current ramdisk, symlink, and SSD state
ramdisk_setup.sh --log        # verbose — show each creation step

---

transcode_manager.sh

Monitors ramdisk usage and manages the symlink direction. Called second in every 7-minute cycle by transcode_management.sh.

Three Modes

# master.conf
TRANSCODE_MANAGER_MODE="smart"    # smart | ramdisk | ssd

# smart (default — use in production):
#   Ramdisk above RAMDISK_WARN_GB → flip symlink to SSD
#   Ramdisk below RAMDISK_LOW_GB  → flip symlink back to ramdisk
#   Hysteresis gap prevents flip-flopping under moderate load

# ramdisk:
#   Always uses ramdisk. Warns if usage exceeds threshold. Never flips.
#   Use for: light load server, guaranteed RAM performance, testing ramdisk behaviour.

# ssd:
#   Always uses SSD. Never uses ramdisk.
#   Use for: post-flip drain (waiting for ramdisk sessions to end naturally),
#            maintenance windows, ramdisk capacity testing.

Threshold Sizing

# host*.conf
HOST1_RAMDISK_SIZE="10G"
HOST1_RAMDISK_WARN_GB=8.8    # flip to SSD above this
HOST1_RAMDISK_LOW_GB=6.5     # flip back below this

Keep ~1.5–2.5GB hysteresis gap between WARN and LOW. Without the gap, usage hovering near WARN causes constant flip-flopping. The gap requires multiple sessions to end completely before flipping back — a genuine recovery, not a brief fluctuation.

Production data from a 7-household Live TV system:

Normal (2–3 streams)       → ~1.5–2.0 GB
Busy evening (5–6 streams) → ~3.5–4.5 GB
Peak (8 streams, Live TV)  → ~5.2 GB
Current setup: 10G ramdisk, 8.8 GB threshold → 1.2 GB safety headroom

Recommended thresholds by ramdisk size:

RAMDISK_SIZE	RAMDISK_WARN_GB	RAMDISK_LOW_GB
6G	4.8	3.5
8G	6.8	5.5
10G	8.8	6.5
12G	10.5	8.5

Multi-Server Configuration

# master.conf
TRANSCODE_SERVERS=(
    "Emby|http://localhost:8096|your-api-key|emby"

    # Temporarily cover HOST2's Emby during maintenance:
    # "Emby-Jayred365|http://100.x.x.x:8096|HOST2-api-key|emby"

    # Jellyfin instance (separate port):
    # "Jellyfin|http://localhost:8097|jellyfin-api-key|jellyfin"
)

Type field: emby | jellyfin | plex — controls which API endpoint format is used. Entries with placeholder API keys are skipped automatically. Comment out unused entries rather than deleting — placeholders show what's available.

Tdarr does NOT belong here. Tdarr encodes full video files — large working files fill the ramdisk rapidly and cause constant flips. Tdarr belongs on SSD permanently.

Session Display

━━━ Active Emby Sessions ━━━
Total: 7  |  Live TV: 5  |  Transcoding: 5  |  Direct: 2
Storage: ramdisk

Sunny     — ABC (WTAE)   — Live TV  — Transcode
Mama Bear — Cinemax      — Live TV  — Transcode
Gmer4Lfe  — WAN Show     — TV Show  — Transcode

Split state during a flip (ramdisk sessions draining, new sessions on SSD):

⚠️  Split state — 4 folder(s) on ramdisk / 2 on SSD
Storage: ramdisk (4) + SSD (2)

Usage

transcode_manager.sh              # normal run
transcode_manager.sh --dry-run    # preview flip decision without flipping
transcode_manager.sh --status     # show current state, usage, sessions, flip history
transcode_manager.sh --log        # verbose output per safety check
transcode_manager.sh --no-log     # suppress daily log write

---

transcode_cleanup.sh

Removes stale transcode files from ramdisk and SSD fallback. Called first in every 7-minute cycle — cleanup before usage measurement is non-negotiable.

Deletion Rules

A file is eligible for deletion only when ALL conditions are true:

1. Older than TRANSCODE_MAX_AGE minutes (mtime — last write time). Active segments are written every few seconds. Not touched in 20 minutes = session ended.
2. Not currently open by any process (lsof pre-built map, O(1) lookup per file). If ffmpeg has a file open, it is not deleted regardless of age.

transcoding-temp/ is never deleted, even when empty. Protected by name exclusion in the find command — deleting it causes Emby to route all sessions to the SSD fallback.

Usage

transcode_cleanup.sh              # normal cleanup run
transcode_cleanup.sh --dry-run    # show what would be deleted
transcode_cleanup.sh --status     # show file counts, ages, open-file status per location
transcode_cleanup.sh --log        # verbose per-file output

---

Full Configuration Reference

# master.conf

# ── Paths ──────────────────────────────────────────────────────────────────────
TRANSCODE_LINK="/mnt/ram-transcode"                        # symlink Emby points at
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/"   # SSD fallback

# ── Per-Host (host*.conf) ───────────────────────────────────────────────
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes"   # ramdisk mount point
HOST1_RAMDISK_SIZE="10G"                        # tmpfs ceiling (not a reservation)
HOST1_RAMDISK_WARN_GB=8.8                       # flip to SSD above this
HOST1_RAMDISK_LOW_GB=6.5                        # flip back below this

# ── Thresholds ─────────────────────────────────────────────────────────────────
RAMDISK_SSD_MIN_GB=20    # minimum SSD free space before allowing SSD flip
                          # prevents accidentally filling the SSD cache pool
TRANSCODE_FLIP_WARN=3    # notify if symlink flips this many times in one hour
                          # high flip count = ramdisk undersized for the load

# ── Cleanup ────────────────────────────────────────────────────────────────────
TRANSCODE_MAX_AGE=20     # minutes — files older than this are stale
TRANSCODE_ORPHAN_AGE=30  # minutes — orphaned session folders removed after this

# ── Manager Mode ───────────────────────────────────────────────────────────────
TRANSCODE_MANAGER_MODE="smart"    # smart | ramdisk | ssd
TRANSCODE_CHECK_EMBY=true         # skip threshold checks when Emby not running
                                   # prevents unnecessary flips overnight

# ── Permissions ────────────────────────────────────────────────────────────────
TRANSCODE_OWNER="nobody:users"    # matches PUID=99 PGID=100 container env
TRANSCODE_CHMOD="755"

# ── Daily Log ──────────────────────────────────────────────────────────────────
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
TRANSCODE_LOG_RETENTION=90        # days — trimmed on every write
TRANSCODE_STATE_FILE="/tmp/transcode_state.db"   # /tmp — resets on reboot correctly

# ── Multi-Server ───────────────────────────────────────────────────────────────
TRANSCODE_SERVERS=(
    "ContainerName|http://host:port|api-key|emby"   # type: emby | jellyfin | plex
)

---

Schedule

At array start (via array_started.sh in Orchestrators/):
  ramdisk_setup.sh     — creates ramdisk, symlink, transcoding-temp
                         Must run BEFORE Emby starts

Every 7 minutes (via transcode_management.sh in Orchestrators/):
  1. transcode_cleanup.sh  — remove stale files, check flip-back
  2. transcode_manager.sh  — check usage, flip if needed, display sessions

Do NOT schedule transcode_manager.sh or transcode_cleanup.sh directly.

---

Troubleshooting

Sessions Landing on SSD Despite Symlink Pointing at Ramdisk

# Check 1 — Docker mount is present:
docker inspect Emby | grep -A4 "ext-ram"
# Expected: "Destination": "/ext-ram-transcode"
# Wrong:    no ext-ram-transcode entry
#
# Fix: Update Emby Extra Parameters, restart Emby.
# GPU containers (full string):
# --gpus "device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1" --pids-limit=0 --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
# Non-GPU (mount only):
# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
# Remove any standard path mapping for the transcode directory.
# Do NOT add bind-propagation=shared — runc v1.3.5+ (Docker 29.x, Unraid 7.3+) rejects it.

# Check 2 — transcoding-temp exists on ramdisk:
ls /mnt/ramdisk_transcodes/
# Expected: transcoding-temp/
#
# Fix if missing:
mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp

# Check 3 — no duplicate SSD mount in Emby template:
docker inspect Emby | grep -A3 "Mounts"
# Should show only /mnt/ram-transcode → /ext-ram-transcode
# Should NOT show /mnt/cache/Temp_Storage/... as a second mount

Flip Count High — 3+ Per Hour

# Ramdisk filling up regularly — load exceeds the current ceiling.
# Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB"
#
# If peak is close to RAMDISK_WARN_GB → increase ramdisk size:
# host1.conf
HOST1_RAMDISK_SIZE="12G"       # increase by 2G
HOST1_RAMDISK_WARN_GB=10.5    # adjust thresholds accordingly
HOST1_RAMDISK_LOW_GB=8.5

# Remount at new size — run ramdisk_setup.sh manually:
ramdisk_setup.sh --log
# Ramdisk must be unmounted first if already mounted:
# umount /mnt/ramdisk_transcodes && ramdisk_setup.sh --log

Ramdisk Not Mounting at Array Start

# Check if tmpfs is mounted:
mountpoint /mnt/ramdisk_transcodes
# "not a mountpoint" → setup failed or not run yet

# Run manually to see the error:
ramdisk_setup.sh --log

# Common causes:
#   /mnt/ramdisk_transcodes missing → mkdir -p /mnt/ramdisk_transcodes
#   Insufficient RAM → check free RAM: free -h
#   RAMDISK_SIZE too large → reduce HOST*_RAMDISK_SIZE

Emergency Manual Flip

# Flip to SSD immediately — all new sessions go to SSD:
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode

# Flip back to ramdisk — all new sessions go to ramdisk:
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode

# Check current symlink target:
readlink /mnt/ram-transcode

# Existing sessions in progress are NEVER affected — only new sessions follow the flip.

Emby Won't Start After Dirty Shutdown

After an unclean shutdown, the --mount bind-propagation entry in Extra Parameters can leave the container in a broken state where Emby refuses to start at all.

# Symptom: Emby fails to start with the --mount extra parameter present.
# Cause: dirty shutdown left the bind mount in a state Docker can't recover.
#
# Recovery:
# 1. Remove the entire Extra Parameters line from Emby in unRAID Docker UI
# 2. Start Emby and wait for it to fully load (check the WebUI is responsive)
# 3. Add the full Extra Parameters line back:
#    GPU:     --gpus "device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1" --pids-limit=0 --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
#    Non-GPU: --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
# 4. Save changes and restart Emby
#
# The container starts cleanly without the mount, which clears the broken state.
# Re-adding the mount after a clean start works reliably.

GPU Memory Exhausted — Jellyfin Fails, Emby Falls Back to CPU

Symptoms:

• Jellyfin refuses to play anything — session fails immediately, no transcode starts
• Emby plays but transcodes on CPU (logs show libx265 or libx264 instead of h264_nvenc/hevc_nvenc)

Both symptoms can appear at the same time and have the same root cause: another process has consumed all available VRAM and not released it. The media servers respond differently to a failed GPU session init:

• Jellyfin: hard fails — no session is created, playback stops entirely
• Emby: falls back to CPU transcoding silently and keeps going

# Confirm VRAM is exhausted:
nvidia-smi

# Find what is holding it:
nvidia-smi --query-compute-apps=pid,used_memory,name --format=csv,noheader

Common source — OCR plugin sidecars: Credit detection and subtitle extraction plugins often talk to a GPU-accelerated OCR container running alongside the media server. The EmbyCredits plugin (yocksers/EmbyCredits) can be configured to use a PaddleOCR backend. PaddleOCR loads a neural network into VRAM at first use and does not release it between runs. After a single credit scan the GPU memory stays consumed, starving Emby and Jellyfin of VRAM for transcoding.

This is not obvious because PaddleOCR is a separate container — nvidia-smi shows the process, but the connection to failing playback is not immediate.

Emby startup probe — the restart trap:

Emby runs a one-shot NVIDIA hardware detection when the container starts. If VRAM is exhausted at startup, NVIDIA is marked unavailable for the entire container session — there is no retry. Emby will use CPU for all transcoding until the container is restarted, and only after VRAM has been freed. Restarting Emby while VRAM is still exhausted causes the probe to fail again and NVIDIA is disabled again.

Correct recovery sequence:

1. Free VRAM (stop the offending container/process)
2. Confirm VRAM is free: nvidia-smi — GPU memory used should drop to near zero
3. Restart Emby — startup probe now succeeds, NVENC available

Fix — switch OCR to CPU:

For EmbyCredits: use the yock1/embycreditocr Tesseract image instead of PaddleOCR. Tesseract is CPU-based, never touches VRAM, and is the plugin's own documented backend.

docker run -d \
  --name EmbyCredit-OCR \
  -p 8884:8884 \
  --restart unless-stopped \
  yock1/embycreditocr

In EmbyCredits plugin settings, set the OCR endpoint to http://localhost:8884.

The accuracy tradeoff is real (PaddleOCR is stronger on non-Latin scripts) but GPU starvation is not an acceptable failure mode for a live media server.

If you just switched from --runtime=nvidia to --gpus and it still fails:

Check nvidia-smi before concluding --gpus is wrong. The flag may be correct and something else may be holding all the VRAM. This is exactly what happens when the EmbyCredits PaddleOCR container is running — PaddleOCR steals all available VRAM while a credit scan runs and never releases it when the scan completes. Emby and Jellyfin then have no GPU access regardless of how the GPU is passed to their containers. The GPU assignment method is irrelevant when there is no VRAM left to assign. Always confirm VRAM is actually free before troubleshooting container GPU flags.

---

Container Fails to Start — open /proc/self/mountinfo: no such file or directory

Unraid 7.3+ / Docker 29.x / runc v1.3.5+ — any container with bind-propagation=shared in Extra Parameters crashes with this error. Not GPU-specific — affects all containers.

OCI runtime create failed: runc create failed: unable to start container process:
error during container init: error jailing process inside rootfs:
open /proc/self/mountinfo: no such file or directory

Root cause: runc v1.3.5 tries to read /proc/self/mountinfo inside the new mount namespace before /proc is mounted, crashing when bind-propagation=shared is present.

Fix: Remove bind-propagation=shared from Extra Parameters entirely.

In the unRAID Docker template:

1. Extra Parameters — remove bind-propagation=shared from the --mount option:

   --gpus "device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1" --pids-limit=0 --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
   

2. The ramdisk tmpfs is already MS_SHARED at the kernel level (shared:N in /proc/self/mountinfo). Propagation is inherited automatically — no Docker flag needed.

Historical note: On Unraid ≤ 7.2.5 (Docker <29.x), bind-propagation=shared was required for symlink flips to propagate into running containers. On Unraid 7.3+ it crashes container start. The kernel-level MS_SHARED on the tmpfs makes it unnecessary.

Increasing Ramdisk Size After Initial Setup

# 1. Set new size and thresholds in host*.conf
# 2. Unmount the existing ramdisk (no sessions should be active):
umount /mnt/ramdisk_transcodes

# 3. Re-run setup to mount at new size:
ramdisk_setup.sh --log

# 4. Verify:
df -h /mnt/ramdisk_transcodes
# Should show new size as total