Varaverk/Transcodes/Manual-Transcoding.md

━━━━━ TRANSCODES — Manual ━━━━━

Configuration reference, Docker mount setup, threshold sizing, and troubleshooting for the ramdisk transcode system. Read the Docker mount section before anything else.

---

━━━ CONTENTS ━━━

• Docker Mount — Required Configuration
• The Symlink Architecture
• ramdisk_setup.sh
• transcode_manager.sh
• transcode_cleanup.sh
• Full Configuration Reference
• Schedule
• Troubleshooting

---

Output Tiers

All three scripts follow a two-tier output model: echo lines are always visible; log lines only appear when --log is passed.

ramdisk_setup.sh — one-shot at array start. Without --log, section headers and the final summary are visible. Per-step creation detail suppressed.

transcode_manager.sh — runs every 7 minutes. Without --log, only state transitions (flips, warnings, errors) and active session display are shown. When nothing changes, a single one-line confirmation is printed. Per-check detail suppressed.

transcode_cleanup.sh — runs every 7 minutes. Without --log, the cleanup summary (files removed, space freed) is always visible. Per-file deletion detail suppressed.

---

Docker Mount — Required Configuration

This is the most important configuration requirement in this folder. Get this wrong and symlink flips silently stop working after the first flip. The system appears to work initially and fails subtly.

Required Mount

In Emby's Extra Parameters in the unRAID Docker template:

--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared

This replaces the transcode path in the standard template path mapping UI. Do NOT add this via the path mapping UI — that UI does not support propagation. Extra Parameters only.

In Emby's transcoding settings, set the transcode path to /ext-ram-transcode.

Why shared Is Required

rprivate (Docker's default):
  Docker resolves the symlink target at first mount and locks that inode.
  Flip: ramdisk → SSD → works (new target locked in)
  Flip: SSD → ramdisk → Docker ignores it. Container still sees SSD binding.
  All sessions continue to land on SSD forever until Emby restarts.
  Symptom: symlink on host is correct, Emby still uses SSD. Confusing.

shared:
  Host mount changes propagate into the container in real time.
  Every symlink flip is immediately visible inside the container. ✅

Verify the Mount

# Check propagation — must show "shared":
docker inspect Emby | grep -A4 "ext-ram"
# Expected: "Propagation": "shared"

# Check Emby's transcode path setting:
docker exec Emby cat /config/config/encoding.xml | grep TranscodingTempPath
# Expected: /ext-ram-transcode

What NOT to Do

Do NOT add a static SSD transcode path as a second path mapping in the template:

/mnt/cache/Temp_Storage/Emby/Transcodes  →  /ssd-transcode   ← do not do this

If the SSD path is mounted inside the container, Emby can see it as an accessible transcode location. It will route sessions there independently of the symlink — bypassing the management system entirely. Sessions land on SSD regardless of symlink state. The whole system stops working.

---

The Symlink Architecture

Emby is configured to write transcodes to TRANSCODE_LINK (/mnt/ram-transcode).
TRANSCODE_LINK is a symlink — its target is managed at runtime.

Normal operation (ramdisk has headroom):
  /mnt/ram-transcode  →  /mnt/ramdisk_transcodes/      (ramdisk — fast, no wear)

Heavy load (ramdisk filling up):
  /mnt/ram-transcode  →  /mnt/cache/Temp_Storage/Emby/Transcodes/  (SSD)

Emby doesn't know this symlink exists. It writes to /mnt/ram-transcode.
ffmpeg resolves the symlink ONCE when a session starts.
After that it holds a direct reference to the actual directory.

Flipping the symlink has ZERO effect on sessions already in progress.
Only NEW sessions care about where the symlink currently points.

What Lives Where

/mnt/ramdisk_transcodes/              ← tmpfs (HOST*_RAMDISK_SIZE ceiling)
  transcoding-temp/                   ← pre-created by ramdisk_setup.sh — always here
    E0D8DC/                           ← Emby session (Live TV HLS segments)
    F1A9BB/                           ← another session

/mnt/ram-transcode                    ← symlink — managed by transcode_manager.sh
  currently points at: /mnt/ramdisk_transcodes/

/mnt/cache/Temp_Storage/Emby/Transcodes/    ← SSD fallback
  transcoding-temp/                   ← also pre-created — Emby finds ramdisk version first
    xyz789/                           ← sessions that started when ramdisk was full

---

ramdisk_setup.sh

Creates the ramdisk, SSD fallback directory, symlink, and transcoding-temp on the ramdisk. Run once at array start. Idempotent — already-mounted ramdisk exits cleanly.

What It Creates

1. RAMDISK_PATH (/mnt/ramdisk_transcodes)
   mount -t tmpfs -o size=HOST*_RAMDISK_SIZE tmpfs /mnt/ramdisk_transcodes
   tmpfs uses only as much RAM as actually needed — RAMDISK_SIZE is a ceiling.
   An empty ramdisk uses essentially zero RAM.

2. transcoding-temp/ inside the ramdisk
   Pre-created so Emby always finds it here first.
   Without this: Emby creates transcoding-temp at its first writable location,
   which may be the SSD fallback even when the symlink points at the ramdisk.
   chown nobody:users — correct ownership for Emby (PUID=99)

3. TRANSCODE_SSD (/mnt/cache/Temp_Storage/Emby/Transcodes/)
   mkdir -p — created if missing, silent if exists
   Also pre-creates transcoding-temp/ inside SSD fallback for consistency

4. TRANSCODE_LINK (/mnt/ram-transcode)
   ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
   Always reset to ramdisk at array start — clean state every boot

Verify After Setup

# All three must pass:
mountpoint /mnt/ramdisk_transcodes
# Expected: /mnt/ramdisk_transcodes is a mountpoint

readlink /mnt/ram-transcode
# Expected: /mnt/ramdisk_transcodes

ls /mnt/ramdisk_transcodes/
# Expected: transcoding-temp/

Usage

ramdisk_setup.sh              # normal run (called by array_started.sh)
ramdisk_setup.sh --dry-run    # show what would be created without creating
ramdisk_setup.sh --status     # show current ramdisk, symlink, and SSD state
ramdisk_setup.sh --log        # verbose — show each creation step

---

transcode_manager.sh

Monitors ramdisk usage and manages the symlink direction. Called second in every 3-minute cycle by transcode_management.sh.

Three Modes

# master.conf
TRANSCODE_MANAGER_MODE="smart"    # smart | ramdisk | ssd

# smart (default — use in production):
#   Ramdisk above RAMDISK_WARN_GB → flip symlink to SSD
#   Ramdisk below RAMDISK_LOW_GB  → flip symlink back to ramdisk
#   Hysteresis gap prevents flip-flopping under moderate load

# ramdisk:
#   Always uses ramdisk. Warns if usage exceeds threshold. Never flips.
#   Use for: light load server, guaranteed RAM performance, testing ramdisk behaviour.

# ssd:
#   Always uses SSD. Never uses ramdisk.
#   Use for: post-flip drain (waiting for ramdisk sessions to end naturally),
#            maintenance windows, ramdisk capacity testing.

Threshold Sizing

# host*.conf
HOST1_RAMDISK_SIZE="10G"
HOST1_RAMDISK_WARN_GB=8.8    # flip to SSD above this
HOST1_RAMDISK_LOW_GB=6.5     # flip back below this

Keep ~1.5–2.5GB hysteresis gap between WARN and LOW. Without the gap, usage hovering near WARN causes constant flip-flopping. The gap requires multiple sessions to end completely before flipping back — a genuine recovery, not a brief fluctuation.

Production data from a 7-household Live TV system:

Normal (2–3 streams)       → ~1.5–2.0 GB
Busy evening (5–6 streams) → ~3.5–4.5 GB
Peak (8 streams, Live TV)  → ~5.2 GB
Current setup: 10G ramdisk, 8.8 GB threshold → 1.2 GB safety headroom

Recommended thresholds by ramdisk size:

RAMDISK_SIZE	RAMDISK_WARN_GB	RAMDISK_LOW_GB
6G	4.8	3.5
8G	6.8	5.5
10G	8.8	6.5
12G	10.5	8.5

Multi-Server Configuration

# master.conf
TRANSCODE_SERVERS=(
    "Emby|http://localhost:8096|your-api-key|emby"

    # Temporarily cover HOST2's Emby during maintenance:
    # "Emby-Jayred365|http://100.x.x.x:8096|HOST2-api-key|emby"

    # Jellyfin instance (separate port):
    # "Jellyfin|http://localhost:8097|jellyfin-api-key|jellyfin"
)

Type field: emby | jellyfin | plex — controls which API endpoint format is used. Entries with placeholder API keys are skipped automatically. Comment out unused entries rather than deleting — placeholders show what's available.

Tdarr does NOT belong here. Tdarr encodes full video files — large working files fill the ramdisk rapidly and cause constant flips. Tdarr belongs on SSD permanently.

Session Display

━━━ Active Emby Sessions ━━━
Total: 7  |  Live TV: 5  |  Transcoding: 5  |  Direct: 2
Storage: ramdisk

Sunny     — ABC (WTAE)   — Live TV  — Transcode
Mama Bear — Cinemax      — Live TV  — Transcode
Gmer4Lfe  — WAN Show     — TV Show  — Transcode

Split state during a flip (ramdisk sessions draining, new sessions on SSD):

⚠️  Split state — 4 folder(s) on ramdisk / 2 on SSD
Storage: ramdisk (4) + SSD (2)

Usage

transcode_manager.sh              # normal run
transcode_manager.sh --dry-run    # preview flip decision without flipping
transcode_manager.sh --status     # show current state, usage, sessions, flip history
transcode_manager.sh --log        # verbose output per safety check
transcode_manager.sh --no-log     # suppress daily log write

---

transcode_cleanup.sh

Removes stale transcode files from ramdisk and SSD fallback. Called first in every 3-minute cycle — cleanup before usage measurement is non-negotiable.

Deletion Rules

A file is eligible for deletion only when ALL conditions are true:

1. Older than TRANSCODE_MAX_AGE minutes (mtime — last write time). Active segments are written every few seconds. Not touched in 20 minutes = session ended.
2. Not currently open by any process (lsof pre-built map, O(1) lookup per file). If ffmpeg has a file open, it is not deleted regardless of age.

transcoding-temp/ is never deleted, even when empty. Protected by name exclusion in the find command — deleting it causes Emby to route all sessions to the SSD fallback.

Usage

transcode_cleanup.sh              # normal cleanup run
transcode_cleanup.sh --dry-run    # show what would be deleted
transcode_cleanup.sh --status     # show file counts, ages, open-file status per location
transcode_cleanup.sh --log        # verbose per-file output

---

Full Configuration Reference

# master.conf

# ── Paths ──────────────────────────────────────────────────────────────────────
TRANSCODE_LINK="/mnt/ram-transcode"                        # symlink Emby points at
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/"   # SSD fallback

# ── Per-Host (host*.conf) ───────────────────────────────────────────────
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes"   # ramdisk mount point
HOST1_RAMDISK_SIZE="10G"                        # tmpfs ceiling (not a reservation)
HOST1_RAMDISK_WARN_GB=8.8                       # flip to SSD above this
HOST1_RAMDISK_LOW_GB=6.5                        # flip back below this

# ── Thresholds ─────────────────────────────────────────────────────────────────
RAMDISK_SSD_MIN_GB=20    # minimum SSD free space before allowing SSD flip
                          # prevents accidentally filling the SSD cache pool
TRANSCODE_FLIP_WARN=3    # notify if symlink flips this many times in one hour
                          # high flip count = ramdisk undersized for the load

# ── Cleanup ────────────────────────────────────────────────────────────────────
TRANSCODE_MAX_AGE=20     # minutes — files older than this are stale
TRANSCODE_ORPHAN_AGE=30  # minutes — orphaned session folders removed after this

# ── Manager Mode ───────────────────────────────────────────────────────────────
TRANSCODE_MANAGER_MODE="smart"    # smart | ramdisk | ssd
TRANSCODE_CHECK_EMBY=true         # skip threshold checks when Emby not running
                                   # prevents unnecessary flips overnight

# ── Permissions ────────────────────────────────────────────────────────────────
TRANSCODE_OWNER="nobody:users"    # matches PUID=99 PGID=100 container env
TRANSCODE_CHMOD="755"

# ── Daily Log ──────────────────────────────────────────────────────────────────
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
TRANSCODE_LOG_RETENTION=90        # days — trimmed on every write
TRANSCODE_STATE_FILE="/tmp/transcode_state.db"   # /tmp — resets on reboot correctly

# ── Multi-Server ───────────────────────────────────────────────────────────────
TRANSCODE_SERVERS=(
    "ContainerName|http://host:port|api-key|emby"   # type: emby | jellyfin | plex
)

---

Schedule

At Startup of Array (via array_started.sh in unRAID_Essentials/):
  ramdisk_setup.sh     — creates ramdisk, symlink, transcoding-temp
                         Must run BEFORE Emby starts

Every 7 minutes (via transcode_management.sh in Orchestrators/):
  1. transcode_cleanup.sh  — remove stale files, check flip-back
  2. transcode_manager.sh  — check usage, flip if needed, display sessions

Do NOT schedule transcode_manager.sh or transcode_cleanup.sh directly.

---

Troubleshooting

Sessions Landing on SSD Despite Symlink Pointing at Ramdisk

# Check 1 — Docker mount propagation (most common cause):
docker inspect Emby | grep Propagation
# Expected: "Propagation": "shared"
# Wrong:    "Propagation": "rprivate"
#
# Fix: Update Emby Extra Parameters, restart Emby:
# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
# Remove any standard path mapping for the transcode directory.

# Check 2 — transcoding-temp exists on ramdisk:
ls /mnt/ramdisk_transcodes/
# Expected: transcoding-temp/
#
# Fix if missing:
mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp

# Check 3 — no duplicate SSD mount in Emby template:
docker inspect Emby | grep -A3 "Mounts"
# Should show only /mnt/ram-transcode → /ext-ram-transcode
# Should NOT show /mnt/cache/Temp_Storage/... as a second mount

Flip Count High — 3+ Per Hour

# Ramdisk filling up regularly — load exceeds the current ceiling.
# Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB"
#
# If peak is close to RAMDISK_WARN_GB → increase ramdisk size:
# host1.conf
HOST1_RAMDISK_SIZE="12G"       # increase by 2G
HOST1_RAMDISK_WARN_GB=10.5    # adjust thresholds accordingly
HOST1_RAMDISK_LOW_GB=8.5

# Remount at new size — run ramdisk_setup.sh manually:
ramdisk_setup.sh --log
# Ramdisk must be unmounted first if already mounted:
# umount /mnt/ramdisk_transcodes && ramdisk_setup.sh --log

Ramdisk Not Mounting at Array Start

# Check if tmpfs is mounted:
mountpoint /mnt/ramdisk_transcodes
# "not a mountpoint" → setup failed or not run yet

# Run manually to see the error:
ramdisk_setup.sh --log

# Common causes:
#   /mnt/ramdisk_transcodes missing → mkdir -p /mnt/ramdisk_transcodes
#   Insufficient RAM → check free RAM: free -h
#   RAMDISK_SIZE too large → reduce HOST*_RAMDISK_SIZE

Emergency Manual Flip

# Flip to SSD immediately — all new sessions go to SSD:
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode

# Flip back to ramdisk — all new sessions go to ramdisk:
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode

# Check current symlink target:
readlink /mnt/ram-transcode

# Existing sessions in progress are NEVER affected — only new sessions follow the flip.

Emby Won't Start After Dirty Shutdown

After an unclean shutdown, the --mount bind-propagation entry in Extra Parameters can leave the container in a broken state where Emby refuses to start at all.

# Symptom: Emby fails to start with the --mount extra parameter present.
# Cause: dirty shutdown left the bind mount in a state Docker can't recover.
#
# Recovery:
# 1. Remove the --mount line from Emby Extra Parameters in unRAID Docker UI
# 2. Start Emby and wait for it to fully load (check the WebUI is responsive)
# 3. Add the --mount line back to Extra Parameters:
#      --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
# 4. Save changes and restart Emby
#
# The container starts cleanly without the mount, which clears the broken state.
# Re-adding the mount after a clean start works reliably.

Increasing Ramdisk Size After Initial Setup

# 1. Set new size and thresholds in host*.conf
# 2. Unmount the existing ramdisk (no sessions should be active):
umount /mnt/ramdisk_transcodes

# 3. Re-run setup to mount at new size:
ramdisk_setup.sh --log

# 4. Verify:
df -h /mnt/ramdisk_transcodes
# Should show new size as total