audit echo vs log across all scripts — outcomes always visible, verbose for per-item loops
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,171 @@
|
||||
# ━━━━━ TRANSCODES ━━━━━
|
||||
|
||||
**Ramdisk-based transcode storage with automatic SSD fallback.** Emby transcodes to RAM
|
||||
at full speed. When the ramdisk fills, new sessions shift to SSD automatically —
|
||||
without interrupting anything already playing. When pressure drops, new sessions shift
|
||||
back to RAM.
|
||||
|
||||
> **Three configuration requirements that are not obvious and were all discovered the
|
||||
> hard way in production.** The `transcoding-temp` directory must be pre-created on
|
||||
> the ramdisk or Emby finds the SSD version and routes all sessions there until
|
||||
> restarted. GPU containers require `--gpus "device=UUID"` in Extra Parameters —
|
||||
> not `--runtime=nvidia`. Do **not** use `bind-propagation=shared` on Unraid 7.3+
|
||||
> (Docker 29.x / runc v1.3.5+) — it crashes container start; the ramdisk is already
|
||||
> MS\_SHARED at the kernel level. Additionally: any GPU-accelerated sidecar (OCR plugins,
|
||||
> credit detection) that holds VRAM and never releases it will starve Emby and Jellyfin
|
||||
> of VRAM for transcoding — Jellyfin hard-fails, Emby silently falls back to CPU.
|
||||
> All three are documented in Manual-Transcoding.md.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ REQUIRED EXTRA PARAMETERS ━━━
|
||||
|
||||
> **Stop. Set this before starting Emby or Jellyfin. If you Google how to add GPU
|
||||
> access to a Docker container on unRAID you will find the wrong answer.** Every
|
||||
> forum post and guide shows `--runtime=nvidia` + `NVIDIA_VISIBLE_DEVICES`. That
|
||||
> method breaks on container rebuilds. Use `--gpus` instead.
|
||||
|
||||
In the unRAID Docker template, open **Advanced View** and paste the following into
|
||||
the **Extra Parameters** field. Do not use the path mapping UI for the transcode
|
||||
directory — it does not support the `--mount` syntax.
|
||||
|
||||
**GPU-accelerated (Emby, Jellyfin with NVENC/NVDEC) — use this:**
|
||||
```
|
||||
--gpus "device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1" --pids-limit=0 --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
|
||||
```
|
||||
|
||||
**Non-GPU — use this:**
|
||||
```
|
||||
--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
|
||||
```
|
||||
|
||||
In Emby and Jellyfin's transcoding settings, set the transcode temp path to `/ext-ram-transcode`.
|
||||
|
||||
Find your GPU UUID: `nvidia-smi -L`
|
||||
HOST1 GPU UUID (Quadro P2000): `GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1`
|
||||
|
||||
**→ Full explanation: [Manual-Transcoding.md](Manual-Transcoding.md)**
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Three Storage Options, None Perfect on Their Own**
|
||||
Hard drives: seek times cause buffering on multi-stream transcoding. SSD: fast enough,
|
||||
but constant small file writes at Emby volume accelerate wear over months. RAM: fastest,
|
||||
no wear, files vanish instantly on session end — but limited by available memory.
|
||||
Fix: RAM by default, SSD as a safety net. The system manages the transition automatically.
|
||||
|
||||
**Changing Transcode Location Requires Restarting Emby**
|
||||
Configuring Emby to switch between ramdisk and SSD requires a restart. Restarting
|
||||
during active streams drops everyone. A 7-person household with 5 Live TV streams at
|
||||
9pm is not a good moment to restart Emby.
|
||||
Fix: symlink indirection. Emby points at a fixed path. The symlink target changes.
|
||||
ffmpeg resolves the symlink once at session start — existing sessions are completely
|
||||
unaffected by flips. Only new sessions follow the new target.
|
||||
|
||||
**Docker Bind Mount Silently Ignored After First Flip** *(historical — resolved differently)*
|
||||
Symlink flip from ramdisk → SSD worked. Flip back: nothing. All new sessions still land
|
||||
on SSD. The symlink on the host is correct. Emby doesn't see it.
|
||||
Original fix was `bind-propagation=shared` — but this crashes runc v1.3.5+ (Docker 29.x,
|
||||
Unraid 7.3+) on any container start. The ramdisk tmpfs is `MS_SHARED` at the kernel level,
|
||||
so propagation is inherited automatically without specifying it in Docker. Do not add
|
||||
`bind-propagation=shared` to Extra Parameters.
|
||||
|
||||
**Sessions Drifting to SSD After a Day of Operation**
|
||||
System working correctly for hours, then sessions gradually drift to SSD despite the
|
||||
ramdisk having plenty of space.
|
||||
Cause: cleanup was removing the empty `transcoding-temp` directory from the ramdisk.
|
||||
Emby then found the SSD fallback version and routed all sessions there.
|
||||
Fix: `transcoding-temp` is excluded from cleanup by name. `ramdisk_setup.sh` pre-creates
|
||||
it at mount time. Both protections together prevent this permanently.
|
||||
|
||||
**lsof Per File on a Live TV System**
|
||||
Early cleanup called `lsof filename` per file to check if anything had it open. On a busy
|
||||
Live TV night with 5 simultaneous streams, the ramdisk contains thousands of HLS segment
|
||||
files — thousands of subprocess calls every 7 minutes.
|
||||
Fix: lsof called once per location to build a complete open-file map. All subsequent
|
||||
checks are O(1) lookups against that map.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
Three scripts, one goal: keep transcodes on RAM, fall back to SSD when needed.
|
||||
|
||||
`ramdisk_setup.sh` runs at array start — creates the tmpfs, SSD fallback directory,
|
||||
symlink, and pre-creates `transcoding-temp`. Everything that must exist before Emby starts.
|
||||
|
||||
`transcode_cleanup.sh` runs first in every 7-minute cycle — removes stale files from both
|
||||
ramdisk and SSD. Cleans up before usage is measured, so the manager sees real load.
|
||||
|
||||
`transcode_manager.sh` runs second — measures ramdisk usage, flips the symlink if
|
||||
thresholds are crossed, runs safety checks, displays active sessions, writes the daily log.
|
||||
|
||||
The symlink is the mechanism that makes this seamless. Emby writes to a fixed path. That
|
||||
path is a symlink whose target is managed at runtime. Sessions in progress never notice.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
```
|
||||
System_Essentials/
|
||||
array_started.sh ──────────────────────────────► ramdisk_setup.sh (at array start)
|
||||
|
||||
Orchestrators/
|
||||
transcode_management.sh ──── cleanup first ──► transcode_cleanup.sh
|
||||
──── then manager ──► transcode_manager.sh
|
||||
(every 7 minutes — order non-negotiable)
|
||||
|
||||
Monitors/
|
||||
weekly_health_digest.sh ◄─── reads ──────────── TRANSCODE_DAILY_LOG
|
||||
```
|
||||
|
||||
Do not schedule `transcode_cleanup.sh` or `transcode_manager.sh` directly.
|
||||
Both are called by `transcode_management.sh` in the correct order.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `ramdisk_setup.sh` | Create tmpfs, SSD fallback dir, symlink, transcoding-temp | At array start (via array_started.sh) |
|
||||
| `transcode_cleanup.sh` | Remove stale files, check for flip-back opportunity | Every 3 min via transcode_management.sh — runs first |
|
||||
| `transcode_manager.sh` | Check usage, flip symlink, safety checks, session display, daily log | Every 3 min via transcode_management.sh — runs second |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
Array starts
|
||||
│
|
||||
▼
|
||||
ramdisk_setup.sh
|
||||
Creates: /mnt/ramdisk_transcodes (tmpfs)
|
||||
/mnt/ramdisk_transcodes/transcoding-temp/
|
||||
/mnt/cache/Temp_Storage/Emby/Transcodes/ (SSD fallback)
|
||||
/mnt/ram-transcode → /mnt/ramdisk_transcodes (symlink)
|
||||
│
|
||||
▼
|
||||
Emby starts, reads transcode path from config
|
||||
Sees: /ext-ram-transcode (bind-mounted from /mnt/ram-transcode)
|
||||
All new sessions write to: /mnt/ram-transcode → /mnt/ramdisk_transcodes/
|
||||
|
||||
|
||||
Every 7 minutes (transcode_management.sh):
|
||||
│
|
||||
├─ transcode_cleanup.sh
|
||||
│ Remove files older than TRANSCODE_MAX_AGE, not open by any process
|
||||
│ transcoding-temp: never deleted
|
||||
│ If ramdisk recovered below RAMDISK_LOW_GB → trigger flip-back
|
||||
│
|
||||
└─ transcode_manager.sh
|
||||
Safety checks (symlink, ramdisk mount, transcoding-temp, permissions)
|
||||
smart mode: ramdisk > RAMDISK_WARN_GB → flip symlink to SSD
|
||||
ramdisk < RAMDISK_LOW_GB → flip symlink back to ramdisk
|
||||
Session display (all TRANSCODE_SERVERS)
|
||||
Append to TRANSCODE_DAILY_LOG
|
||||
```
|
||||
Reference in New Issue
Block a user