Varaverk/Transcodes/README-Transcoding.md

# Transcodes

Ramdisk-based transcode storage management for Emby using filesystem symlink indirection.

---

## The Problem

Emby transcodes video on the fly for clients that can't play the source format directly. Each transcode session generates hundreds of small HLS segment files that are written and read continuously. Where those files live has a significant impact on performance:

- **Hard drives** — too slow for simultaneous multi-stream transcoding. Seek times cause buffering.
- **SSD (cache pool)** — fast enough, but constant small file writes accelerate wear over time.
- **RAM (tmpfs)** — fastest possible, no wear, disappears cleanly when sessions end.

A ramdisk is the ideal transcode location. The only risk is running out of RAM during heavy load — which is where this system comes in.

---

## The Design

### Symlink Indirection

Emby is pointed at a fixed path that never changes:

```
/mnt/ram-transcode  →  [currently: /mnt/ramdisk_transcodes]
```

This is a symlink. Emby doesn't know or care what's on the other end — it just writes to `/mnt/ram-transcode`. The transcode manager controls where that path actually points by updating the symlink target.

**The critical insight:** ffmpeg resolves the symlink path **once** at session start. After that, it has a direct reference to the actual directory. This means:

- **Existing sessions are never affected by symlink changes**
- When the symlink flips from ramdisk to SSD, sessions already in progress keep writing to ramdisk until they end naturally
- Only **new** sessions care about where the symlink currently points

This is what makes the fallback seamless. Users never experience a glitch.

---

## Docker Mount — Critical

**This must be configured correctly or the symlink system will not work.**

Emby must be configured using `--mount` in Extra Parameters — **not** as a standard path mapping in the unRAID template.

**In Emby Extra Parameters:**
```
--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
```

**Why `shared` propagation is required:**

Standard bind mounts in unRAID use `rprivate` propagation by default. With `rprivate`, Docker resolves the symlink target once — at the moment of the first mount change — and locks that inode for the lifetime of the container. When the symlink flips from ramdisk to SSD, Docker takes a private copy of that SSD binding. When the symlink later flips back to ramdisk, the container ignores it — it already has a private SSD binding locked in. All new sessions land on SSD permanently until Emby restarts.

With `shared` propagation, host mount changes propagate into the container in real time. Symlink flips on the host are immediately visible inside the container. The system works as designed.

**Verify the mount is configured correctly:**
```bash
docker inspect Emby | grep -A4 "ext-ram"
# Should show: "Propagation": "shared"
# NOT:         "Propagation": "rprivate"
```

**Do NOT add a static SSD transcode path as a second mount.** If the SSD path is mounted inside the container, Emby can see it as an accessible transcode location and will route sessions there independently of the symlink — completely bypassing the management system.

> **Both of these issues were discovered in production.** The static SSD mount caused sessions to bypass the symlink. The rprivate propagation caused sessions to lock onto SSD after the first flip. Both are now fixed in the correct configuration.

---

## The Scripts

### `ramdisk_setup.sh`
**Run at array start. Run once.**

Creates the tmpfs ramdisk, the SSD fallback directory, the symlink, and — critically — the `transcoding-temp` subdirectory on the ramdisk.

```bash
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/Transcodes/ramdisk_setup.sh
```

**Why `transcoding-temp` must be pre-created:**

Emby creates a `transcoding-temp` subdirectory inside its configured transcode path when it first needs to write. If `transcoding-temp` doesn't exist on the ramdisk, Emby may find and use an existing one on the SSD fallback path instead — locking all sessions onto SSD until Emby restarts.

`ramdisk_setup.sh` creates `transcoding-temp` on the ramdisk at mount time so Emby always finds it there first.

What it creates:
```
/mnt/ramdisk_transcodes/                 ← tmpfs mount (RAMDISK_SIZE ceiling)
/mnt/ramdisk_transcodes/transcoding-temp ← pre-created so Emby uses ramdisk
/mnt/ram-transcode                        ← symlink pointing at ramdisk
/mnt/cache/Temp_Storage/Emby/Transcodes/ ← SSD fallback directory
```

After running, verify:
```bash
mountpoint /mnt/ramdisk_transcodes        # should say "is a mountpoint"
readlink /mnt/ram-transcode               # should show /mnt/ramdisk_transcodes
ls /mnt/ramdisk_transcodes/               # should show transcoding-temp/
```

---

### `transcode_management.sh` (Orchestrators/)
**Run every 3 minutes via cron. Replaces separate manager and cleanup cron entries.**

Runs `transcode_cleanup.sh` first then `transcode_manager.sh` in the correct order. Cleanup runs first so the manager sees accurate post-cleanup usage before making threshold decisions.

```bash
# Scheduled as: */3 * * * *
/mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh
```

Also tracks daily transcode statistics to `/boot/config/transcode_daily.db` — read by `weekly_health_digest.sh` for the weekly report.

---

### `transcode_manager.sh`
**Called by `transcode_management.sh` — not scheduled directly.**

Monitors ramdisk usage and manages the symlink direction.

#### Operating Modes

Set `TRANSCODE_MANAGER_MODE` in `Master.conf`:

| Mode | Behavior | Use When |
|------|----------|----------|
| `smart` | Auto-flips between ramdisk and SSD based on thresholds | Normal operation — default |
| `ramdisk` | Always uses ramdisk, never flips to SSD | Light load, guaranteed RAM performance |
| `ssd` | Always uses SSD, never uses ramdisk | Maintenance, post-flip drain |

#### Smart Mode — How the Flip Works

```
Ramdisk usage rises above RAMDISK_WARN_GB (8.8GB)
  → Symlink flips to SSD
  → New sessions land on SSD
  → Existing sessions continue on ramdisk until they end

Ramdisk usage drops below RAMDISK_LOW_GB (6.5GB)
  → Symlink flips back to ramdisk
  → New sessions land on ramdisk again
```

The 2.3GB gap between `RAMDISK_WARN_GB` and `RAMDISK_LOW_GB` is the **hysteresis gap**. It prevents the symlink from flip-flopping when usage hovers near the threshold.

#### Safety Checks

Every run, regardless of mode:

| Condition | Action |
|-----------|--------|
| Ramdisk not mounted | Flip to SSD immediately, notify warning |
| SSD path missing | Disable fallback, notify warning |
| Symlink missing | Recreate pointing at ramdisk, notify |
| Symlink target gone | Reset to ramdisk, notify |
| `transcoding-temp` missing from ramdisk | Create it — prevents Emby falling back to SSD |
| Permissions drift | Fix silently every run |
| Emby not running | Skip threshold checks, verify symlink only |

#### Session Display

Each run queries the Emby API and shows active streams:

```
━━━ 🎬 Active Emby Sessions ━━━
🎬 Total: 7  |  💨 Live TV: 5  |  🔄 Transcoding: 5  |  🏁 Direct: 2
🔗 Storage: 💨 ramdisk

🎬 Sunny     — ABC (WTAE)   — Live TV  — Transcode
🎬 Mama Bear — Cinemax      — Live TV  — Transcode
🎬 Gmer4Lfe  — WAN Show     — TV Show  — Transcode
```

**Split state** is detected and displayed when sessions exist on both ramdisk and SSD simultaneously — normal during a symlink flip:

```
⚠️  Split state — 4 folder(s) on ramdisk / 2 on SSD
🔗 Storage: 💨 ramdisk (4) + 💾 SSD (2)
```

---

### `transcode_cleanup.sh`
**Called by `transcode_management.sh` — not scheduled directly.**

Removes old inactive transcode files from both ramdisk and SSD. Never deletes files that are currently open by any process.

**Deletion rules — a file is eligible only when ALL are true:**
1. Older than `TRANSCODE_MAX_AGE` minutes
2. Not currently open by any process

**`transcoding-temp` directory is protected from deletion.** Even when empty, `transcoding-temp` is never removed by cleanup. Deleting it causes Emby to fall back to the SSD version on next session start — this was the root cause of sessions drifting to SSD after a day of operation.

**Performance design:** `lsof` is called once per location to build a complete open file list — not once per file. On a busy Live TV system with thousands of HLS segments this is critical for performance.

---

## Configuration

All configuration in `Master.conf` under `── TRANSCODES ──`:

```bash
RAMDISK_PATH="/mnt/ramdisk_transcodes"
RAMDISK_SIZE="10G"                       # bumped from 8G — peak usage ~5.2GB on busy nights
TRANSCODE_LINK="/mnt/ram-transcode"
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/"

RAMDISK_WARN_GB=8.8     # flip to SSD above this — 1.2GB headroom from ceiling
RAMDISK_LOW_GB=6.5      # flip back to ramdisk below this — 2.3GB hysteresis gap
RAMDISK_SSD_MIN_GB=20   # minimum SSD free space before allowing flip

TRANSCODE_MAX_AGE=20    # minutes before file eligible for cleanup
TRANSCODE_ORPHAN_AGE=30 # minutes for orphaned files

TRANSCODE_FLIP_WARN=3   # notify if symlink flips this many times per hour

TRANSCODE_OWNER="nobody:users"
TRANSCODE_CHMOD="755"

TRANSCODE_MANAGER_MODE="smart"   # smart | ramdisk | ssd

TRANSCODE_CHECK_EMBY=true
TRANSCODE_EMBY_CONTAINER="Emby"

TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db"
TRANSCODE_LOG_RETENTION=90
```

### Sizing the Ramdisk

`tmpfs` only uses RAM actually needed — `RAMDISK_SIZE` is a ceiling, not a reservation.

**Production data from this setup:**
```
Normal load (2-3 streams)    → ~1.5-2.0GB
Busy evening (5-6 streams)   → ~3.5-4.5GB
Peak (8 streams, live TV)    → ~5.2GB
Current ramdisk              → 10G with 8.8GB threshold
```

### Sizing Thresholds

When adjusting `RAMDISK_SIZE`, adjust thresholds to match. Keep a 1.5-2.5GB hysteresis gap between WARN and LOW:

| Ramdisk Size | RAMDISK_WARN_GB | RAMDISK_LOW_GB |
|-------------|-----------------|----------------|
| 6G | 4.8 | 3.5 |
| 8G | 6.8 | 5.5 |
| 10G | 8.8 | 6.5 |
| 12G | 10.5 | 8.5 |

---

## Scheduled Run Summary

| Script | Schedule | Purpose |
|--------|----------|---------|
| `ramdisk_setup.sh` | At Startup of Array | Create ramdisk, symlink, transcoding-temp |
| `transcode_management.sh` | `*/3 * * * *` | Cleanup then manager — correct order, daily stats |

`transcode_manager.sh` and `transcode_cleanup.sh` are called by `transcode_management.sh` — do not schedule them separately.

---

## Troubleshooting

**Sessions landing on SSD despite symlink pointing at ramdisk:**

1. Check Docker mount propagation:
   ```bash
   docker inspect Emby | grep Propagation
   # Must show: "shared" not "rprivate"
   ```
   Fix: Add `--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared` to Extra Parameters and restart Emby.

2. Check `transcoding-temp` exists on ramdisk:
   ```bash
   ls /mnt/ramdisk_transcodes/
   # Must show: transcoding-temp/
   ```
   Fix: `mkdir -p /mnt/ramdisk_transcodes/transcoding-temp && chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp`

3. Check for duplicate SSD mount in Emby template — remove any static SSD transcode path mapping.

**`[LOG] Permissions fixed` on every run:**

Permissions are applied every run regardless — this is by design. If it logs every cycle it means Emby is resetting permissions on write. Not harmful — just informational.

**Flip count high — 3+ per hour:**

Ramdisk filling up regularly. Consider increasing `RAMDISK_SIZE` by 2GB and adjusting thresholds accordingly.

---

## Emergency Manual Flip

If you need to manually redirect all new transcodes to SSD:
```bash
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
```

To flip back to ramdisk:
```bash
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
```

Existing sessions are unaffected. Only new sessions follow the new target.