Varaverk/Transcodes/README-Transcoding.md

# Transcodes

Ramdisk-based transcode storage management for Emby using filesystem symlink indirection.

---

## The Problem

Emby transcodes video on the fly for clients that can't play the source format directly. Each transcode session generates hundreds of small HLS segment files that are written and read continuously. Where those files live has a significant impact on performance:

- **Hard drives** — too slow for simultaneous multi-stream transcoding. Seek times cause buffering.
- **SSD (cache pool)** — fast enough, but constant small file writes accelerate wear over time.
- **RAM (tmpfs)** — fastest possible, no wear, disappears cleanly when sessions end.

A ramdisk is the ideal transcode location. The only risk is running out of RAM during heavy load — which is where this system comes in.

---

## The Design

### Symlink Indirection

Emby is pointed at a fixed path that never changes:

```
/mnt/ram-transcode  →  [currently: /mnt/ramdisk_transcodes]
```

This is a symlink. Emby doesn't know or care what's on the other end — it just writes to `/mnt/ram-transcode`. The transcode manager controls where that path actually points by updating the symlink target.

**The critical insight:** ffmpeg resolves the symlink path **once** at session start. After that, it has a direct reference to the actual directory. This means:

- **Existing sessions are never affected by symlink changes**
- When the symlink flips from ramdisk to SSD, sessions already in progress keep writing to ramdisk until they end naturally
- Only **new** sessions care about where the symlink currently points

This is what makes the fallback seamless. Users never experience a glitch.

### Why Not Just Use the SSD Directly?

You could point Emby directly at the SSD and skip the ramdisk entirely. Many setups do this. The ramdisk approach gives you:

1. **Faster performance** — RAM is orders of magnitude faster than SSD for small random writes
2. **Zero SSD wear** — transcode segments are written and deleted constantly. On a busy server this adds up to significant SSD wear over months and years
3. **Automatic cleanup** — tmpfs is released back to the system when files are deleted. No fragmentation, no stale files surviving a crash
4. **Session isolation** — each session's files disappear completely when the session ends

---

## The Scripts

### `ramdisk_setup.sh`
**Run at array start. Run once.**

Creates the tmpfs ramdisk, the SSD fallback directory, and the symlink. If the ramdisk is already mounted it reports status and exits cleanly — safe to run multiple times.

```bash
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/Transcodes/ramdisk_setup.sh
```

What it creates:
```
/mnt/ramdisk_transcodes/    ← tmpfs mount (RAMDISK_SIZE ceiling)
/mnt/ram-transcode           ← symlink pointing at ramdisk
/mnt/cache/Temp_Storage/Emby/Transcodes/  ← SSD fallback directory
```

After running, verify:
```bash
mountpoint /mnt/ramdisk_transcodes   # should say "is a mountpoint"
readlink /mnt/ram-transcode          # should show /mnt/ramdisk_transcodes
```

---

### `transcode_manager.sh`
**Run every 3 minutes via cron.**

Monitors ramdisk usage and manages the symlink direction. The main brain of the system.

```bash
# Scheduled as: */3 * * * *
/mnt/user/appdata/unraid_scripts/Transcodes/transcode_manager.sh
```

#### Operating Modes

Set `TRANSCODE_MANAGER_MODE` in `Master.conf`:

| Mode | Behavior | Use When |
|------|----------|----------|
| `smart` | Auto-flips between ramdisk and SSD based on thresholds | Normal operation — default |
| `ramdisk` | Always uses ramdisk, never flips to SSD | Light load, guaranteed RAM performance |
| `ssd` | Always uses SSD, never uses ramdisk | Ramdisk maintenance, post-flip drain |

#### Smart Mode — How the Flip Works

```
Ramdisk usage rises above RAMDISK_WARN_GB (6.8GB)
  → Symlink flips to SSD
  → New sessions land on SSD
  → Existing sessions continue on ramdisk until they end

Ramdisk usage drops below RAMDISK_LOW_GB (5.5GB)
  → Symlink flips back to ramdisk
  → New sessions land on ramdisk again
```

The gap between `RAMDISK_WARN_GB` and `RAMDISK_LOW_GB` (1.3GB) is the **hysteresis gap**. It prevents the symlink from flip-flopping when usage hovers near the threshold. Without this gap you'd get constant flipping on a busy system.

#### Safety Checks

Every run, regardless of mode:

| Condition | Action |
|-----------|--------|
| Ramdisk not mounted | Flip symlink to SSD immediately, notify warning |
| SSD path missing | Disable fallback, notify warning. If mode is `ssd` — exit |
| Symlink missing | Recreate pointing at ramdisk, notify |
| Symlink target gone | Reset to ramdisk, notify |
| Permissions drift | Fix silently — `chmod` and `chown` applied every run |
| Emby not running | Skip threshold checks, verify symlink only |

#### Session Display

Each run queries the Emby API and shows active streams:

```
━━━ 🎬 Active Emby Sessions ━━━
  🎬 Total: 7  |  💨 Live TV: 5  |  🔄 Transcoding: 5  |  🏁 Direct: 2

  🔗 Storage: 💨 ramdisk

  🎬 Gmer4Lfe  — MLB: Pirates vs Nationals  — Live TV   — Transcode
  🎬 Rebecca   — MLB: Pirates vs Nationals  — Live TV   — Transcode
  🎬 Sunny     — AT&T Sportsnet Pittsburgh  — Live TV   — Transcode
  🎬 jaden     — TNT                        — Live TV   — Transcode
  🎬 Mama Bear — Con-Text                   — TV Show   — Direct Stream
```

**Split state** is detected and displayed when sessions exist on both ramdisk and SSD simultaneously — this happens naturally when the symlink flips while sessions are in progress:

```
  ⚠️  Split state — 4 folder(s) on ramdisk / 2 on SSD
  ⚠️  Older sessions remain on original location until they end naturally
  🔗 Storage: 💨 ramdisk (4) + 💾 SSD (2)
```

> **Why per-session location isn't shown:** Emby's internal transcode folder names don't match the session IDs returned by the API — there is no reliable way to map a specific user to a specific folder. The folder count on each location gives you the picture you need at a glance without false precision.

#### Flip Frequency Warning

If the symlink flips `TRANSCODE_FLIP_WARN` or more times in one hour, a notification is sent. This is a signal that `RAMDISK_SIZE` may need to be increased. Real production data from this setup:

```
Normal load (2-3 streams)   → ~1.5-2.0GB
Busy evening (5-6 streams)  → ~3.5-4.5GB
Peak (8 streams, live TV)   → ~5.2GB
Threshold trigger            → 6.8GB
```

---

### `transcode_cleanup.sh`
**Run every 5 minutes via cron.**

Removes old inactive transcode files from both ramdisk and SSD. Never deletes files that are currently open by any process.

```bash
# Scheduled as: */5 * * * *
/mnt/user/appdata/unraid_scripts/Transcodes/transcode_cleanup.sh
```

#### Deletion Rules

A file is eligible for deletion only when **all** of these are true:

1. Older than `TRANSCODE_MAX_AGE` minutes (default: 20 min)
2. Not currently open by any process

#### Performance Design

`lsof` is called **once per location** to build a complete list of open files — not once per file. This is critical on a busy Live TV system where a single location can have thousands of HLS segment files. A per-file `lsof` approach stalls the system under load.

---

## Docker Mount — Critical

Emby must be configured with **one transcode mount only:**

```
Host path:       /mnt/ram-transcode/
Container path:  /ext-ram-transcode
```

**Do NOT add a static SSD transcode path as a second volume mount.**

If the SSD path is mounted inside the container, Emby can see it as an accessible transcode location and will route sessions there independently of the symlink — completely bypassing the management system. This is not obvious and causes confusing split behavior that is hard to diagnose.

The symlink handles all routing. One mount is all that's needed.

> **This was learned in production.** The system worked correctly once the static SSD mount was removed. The symptom was new sessions landing on SSD even when the symlink pointed at ramdisk.

#### Emergency Manual Flip

If you need to manually redirect all new transcodes to SSD:

```bash
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
```

To flip back to ramdisk:

```bash
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
```

Existing sessions are unaffected — only new sessions follow the new target.

---

## Configuration

All configuration in `Master.conf` under the `── TRANSCODES ──` section.

```bash
# Paths
RAMDISK_PATH="/mnt/ramdisk_transcodes"   # tmpfs mount point
RAMDISK_SIZE="8G"                         # ceiling — only uses RAM actually needed
TRANSCODE_LINK="/mnt/ram-transcode"       # symlink — location never changes
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/"

# Smart mode thresholds
RAMDISK_WARN_GB=6.8     # flip to SSD above this
RAMDISK_LOW_GB=5.5      # flip back to ramdisk below this
RAMDISK_SSD_MIN_GB=20   # minimum SSD free space before allowing flip

# Cleanup
TRANSCODE_MAX_AGE=20    # minutes before file eligible for cleanup
TRANSCODE_ORPHAN_AGE=30 # minutes for orphaned files

# Alerts
TRANSCODE_FLIP_WARN=3   # notify if symlink flips this many times per hour

# Permissions
TRANSCODE_OWNER="nobody:users"
TRANSCODE_CHMOD="755"

# Mode
TRANSCODE_MANAGER_MODE="smart"   # smart | ramdisk | ssd

# Emby check
TRANSCODE_CHECK_EMBY=true
TRANSCODE_EMBY_CONTAINER="Emby"
```

### Sizing the Ramdisk

The ramdisk is a `tmpfs` — it only uses RAM that is actually needed. `RAMDISK_SIZE` is a ceiling, not a reservation. An 8GB ramdisk that holds 2GB of files only uses 2GB of RAM.

**Rule of thumb for sizing:**
- Count your maximum expected concurrent transcoding streams
- Multiply by ~0.5-1GB per stream (Live TV HLS streams use more than standard transcodes)
- Add 20-30% headroom above your threshold

**From production data on this setup:**
- 5 Live TV streams + 2 standard = ~4.5GB
- 8 streams peak = ~5.2GB
- Current ramdisk = 8GB with 6.8GB threshold — comfortable headroom

If you regularly hit `TRANSCODE_FLIP_WARN` or see 3+ flips per hour, increase `RAMDISK_SIZE` by 2GB and adjust thresholds accordingly.

---

## Sizing Thresholds

When adjusting `RAMDISK_SIZE`, adjust thresholds to match:

| Ramdisk Size | RAMDISK_WARN_GB | RAMDISK_LOW_GB |
|-------------|-----------------|----------------|
| 6G | 4.8 | 3.5 |
| 8G | 6.8 | 5.5 |
| 10G | 8.5 | 7.0 |
| 12G | 10.0 | 8.5 |

Keep a 1.0-1.5GB hysteresis gap between WARN and LOW. A gap smaller than this causes flip-flop behavior near the threshold.

---

## Scheduled Run Summary

| Script | Schedule | Purpose |
|--------|----------|---------|
| `ramdisk_setup.sh` | At Startup of Array | Create ramdisk and symlink |
| `transcode_manager.sh` | `*/3 * * * *` | Monitor usage, manage symlink, display sessions |
| `transcode_cleanup.sh` | `*/5 * * * *` | Remove old inactive files |

---

## Version 2 Roadmap

A future `advanced` mode is planned that allows per-media-type storage routing:

```bash
TRANSCODE_MANAGER_MODE="advanced"

TRANSCODE_FORCE_RAMDISK=(
    "LiveTv"    # always ramdisk — buffering is latency sensitive
)
TRANSCODE_FORCE_SSD=(
    "Audio"     # music downloads — no benefit from ramdisk
)
# Everything else follows smart threshold behavior
```

This requires the Emby API to expose media type at session start — the groundwork (session display and media type parsing) is already in place. Target: this fall.