feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,460 @@
|
||||
# ━━━━━ TRANSCODES — Manual ━━━━━
|
||||
|
||||
Configuration reference, Docker mount setup, threshold sizing, and troubleshooting
|
||||
for the ramdisk transcode system. Read the Docker mount section before anything else.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [Docker Mount — Required Configuration](#docker-mount--required-configuration)
|
||||
- [The Symlink Architecture](#the-symlink-architecture)
|
||||
- [ramdisk_setup.sh](#ramdisk_setupsh)
|
||||
- [transcode_manager.sh](#transcode_managersh)
|
||||
- [transcode_cleanup.sh](#transcode_cleanupsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Schedule](#schedule)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Docker Mount — Required Configuration
|
||||
|
||||
> **This is the most important configuration requirement in this folder.**
|
||||
> Get this wrong and symlink flips silently stop working after the first flip.
|
||||
> The system appears to work initially and fails subtly.
|
||||
|
||||
### Required Mount
|
||||
|
||||
In Emby's **Extra Parameters** in the unRAID Docker template:
|
||||
|
||||
```
|
||||
--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
|
||||
```
|
||||
|
||||
This **replaces** the transcode path in the standard template path mapping UI. Do NOT
|
||||
add this via the path mapping UI — that UI does not support propagation. Extra Parameters only.
|
||||
|
||||
In Emby's transcoding settings, set the transcode path to `/ext-ram-transcode`.
|
||||
|
||||
### Why `shared` Is Required
|
||||
|
||||
```
|
||||
rprivate (Docker's default):
|
||||
Docker resolves the symlink target at first mount and locks that inode.
|
||||
Flip: ramdisk → SSD → works (new target locked in)
|
||||
Flip: SSD → ramdisk → Docker ignores it. Container still sees SSD binding.
|
||||
All sessions continue to land on SSD forever until Emby restarts.
|
||||
Symptom: symlink on host is correct, Emby still uses SSD. Confusing.
|
||||
|
||||
shared:
|
||||
Host mount changes propagate into the container in real time.
|
||||
Every symlink flip is immediately visible inside the container. ✅
|
||||
```
|
||||
|
||||
### Verify the Mount
|
||||
|
||||
```bash
|
||||
# Check propagation — must show "shared":
|
||||
docker inspect Emby | grep -A4 "ext-ram"
|
||||
# Expected: "Propagation": "shared"
|
||||
|
||||
# Check Emby's transcode path setting:
|
||||
docker exec Emby cat /config/config/encoding.xml | grep TranscodingTempPath
|
||||
# Expected: /ext-ram-transcode
|
||||
```
|
||||
|
||||
### What NOT to Do
|
||||
|
||||
Do NOT add a static SSD transcode path as a second path mapping in the template:
|
||||
```
|
||||
/mnt/cache/Temp_Storage/Emby/Transcodes → /ssd-transcode ← do not do this
|
||||
```
|
||||
|
||||
If the SSD path is mounted inside the container, Emby can see it as an accessible
|
||||
transcode location. It will route sessions there independently of the symlink —
|
||||
bypassing the management system entirely. Sessions land on SSD regardless of symlink
|
||||
state. The whole system stops working.
|
||||
|
||||
---
|
||||
|
||||
## The Symlink Architecture
|
||||
|
||||
```
|
||||
Emby is configured to write transcodes to TRANSCODE_LINK (/mnt/ram-transcode).
|
||||
TRANSCODE_LINK is a symlink — its target is managed at runtime.
|
||||
|
||||
Normal operation (ramdisk has headroom):
|
||||
/mnt/ram-transcode → /mnt/ramdisk_transcodes/ (ramdisk — fast, no wear)
|
||||
|
||||
Heavy load (ramdisk filling up):
|
||||
/mnt/ram-transcode → /mnt/cache/Temp_Storage/Emby/Transcodes/ (SSD)
|
||||
|
||||
Emby doesn't know this symlink exists. It writes to /mnt/ram-transcode.
|
||||
ffmpeg resolves the symlink ONCE when a session starts.
|
||||
After that it holds a direct reference to the actual directory.
|
||||
|
||||
Flipping the symlink has ZERO effect on sessions already in progress.
|
||||
Only NEW sessions care about where the symlink currently points.
|
||||
```
|
||||
|
||||
### What Lives Where
|
||||
|
||||
```
|
||||
/mnt/ramdisk_transcodes/ ← tmpfs (HOST*_RAMDISK_SIZE ceiling)
|
||||
transcoding-temp/ ← pre-created by ramdisk_setup.sh — always here
|
||||
E0D8DC/ ← Emby session (Live TV HLS segments)
|
||||
F1A9BB/ ← another session
|
||||
|
||||
/mnt/ram-transcode ← symlink — managed by transcode_manager.sh
|
||||
currently points at: /mnt/ramdisk_transcodes/
|
||||
|
||||
/mnt/cache/Temp_Storage/Emby/Transcodes/ ← SSD fallback
|
||||
transcoding-temp/ ← also pre-created — Emby finds ramdisk version first
|
||||
xyz789/ ← sessions that started when ramdisk was full
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ramdisk_setup.sh
|
||||
|
||||
Creates the ramdisk, SSD fallback directory, symlink, and `transcoding-temp` on the
|
||||
ramdisk. Run once at array start. Idempotent — already-mounted ramdisk exits cleanly.
|
||||
|
||||
### What It Creates
|
||||
|
||||
```
|
||||
1. RAMDISK_PATH (/mnt/ramdisk_transcodes)
|
||||
mount -t tmpfs -o size=HOST*_RAMDISK_SIZE tmpfs /mnt/ramdisk_transcodes
|
||||
tmpfs uses only as much RAM as actually needed — RAMDISK_SIZE is a ceiling.
|
||||
An empty ramdisk uses essentially zero RAM.
|
||||
|
||||
2. transcoding-temp/ inside the ramdisk
|
||||
Pre-created so Emby always finds it here first.
|
||||
Without this: Emby creates transcoding-temp at its first writable location,
|
||||
which may be the SSD fallback even when the symlink points at the ramdisk.
|
||||
chown nobody:users — correct ownership for Emby (PUID=99)
|
||||
|
||||
3. TRANSCODE_SSD (/mnt/cache/Temp_Storage/Emby/Transcodes/)
|
||||
mkdir -p — created if missing, silent if exists
|
||||
Also pre-creates transcoding-temp/ inside SSD fallback for consistency
|
||||
|
||||
4. TRANSCODE_LINK (/mnt/ram-transcode)
|
||||
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
|
||||
Always reset to ramdisk at array start — clean state every boot
|
||||
```
|
||||
|
||||
### Verify After Setup
|
||||
|
||||
```bash
|
||||
# All three must pass:
|
||||
mountpoint /mnt/ramdisk_transcodes
|
||||
# Expected: /mnt/ramdisk_transcodes is a mountpoint
|
||||
|
||||
readlink /mnt/ram-transcode
|
||||
# Expected: /mnt/ramdisk_transcodes
|
||||
|
||||
ls /mnt/ramdisk_transcodes/
|
||||
# Expected: transcoding-temp/
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
ramdisk_setup.sh # normal run (called by array_start.sh)
|
||||
ramdisk_setup.sh --dry-run # show what would be created without creating
|
||||
ramdisk_setup.sh --status # show current ramdisk, symlink, and SSD state
|
||||
ramdisk_setup.sh --log # verbose — show each creation step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## transcode_manager.sh
|
||||
|
||||
Monitors ramdisk usage and manages the symlink direction. Called second in every 3-minute
|
||||
cycle by `transcode_management.sh`.
|
||||
|
||||
### Three Modes
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
||||
|
||||
# smart (default — use in production):
|
||||
# Ramdisk above RAMDISK_WARN_GB → flip symlink to SSD
|
||||
# Ramdisk below RAMDISK_LOW_GB → flip symlink back to ramdisk
|
||||
# Hysteresis gap prevents flip-flopping under moderate load
|
||||
|
||||
# ramdisk:
|
||||
# Always uses ramdisk. Warns if usage exceeds threshold. Never flips.
|
||||
# Use for: light load server, guaranteed RAM performance, testing ramdisk behaviour.
|
||||
|
||||
# ssd:
|
||||
# Always uses SSD. Never uses ramdisk.
|
||||
# Use for: post-flip drain (waiting for ramdisk sessions to end naturally),
|
||||
# maintenance windows, ramdisk capacity testing.
|
||||
```
|
||||
|
||||
### Threshold Sizing
|
||||
|
||||
```bash
|
||||
# master_host*.conf
|
||||
HOST1_RAMDISK_SIZE="10G"
|
||||
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
|
||||
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
|
||||
```
|
||||
|
||||
Keep ~1.5–2.5GB hysteresis gap between WARN and LOW. Without the gap, usage hovering
|
||||
near WARN causes constant flip-flopping. The gap requires multiple sessions to end
|
||||
completely before flipping back — a genuine recovery, not a brief fluctuation.
|
||||
|
||||
Production data from a 7-household Live TV system:
|
||||
```
|
||||
Normal (2–3 streams) → ~1.5–2.0 GB
|
||||
Busy evening (5–6 streams) → ~3.5–4.5 GB
|
||||
Peak (8 streams, Live TV) → ~5.2 GB
|
||||
Current setup: 10G ramdisk, 8.8 GB threshold → 1.2 GB safety headroom
|
||||
```
|
||||
|
||||
Recommended thresholds by ramdisk size:
|
||||
|
||||
| RAMDISK_SIZE | RAMDISK_WARN_GB | RAMDISK_LOW_GB |
|
||||
|-------------|----------------|----------------|
|
||||
| 6G | 4.8 | 3.5 |
|
||||
| 8G | 6.8 | 5.5 |
|
||||
| 10G | 8.8 | 6.5 |
|
||||
| 12G | 10.5 | 8.5 |
|
||||
|
||||
### Multi-Server Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
TRANSCODE_SERVERS=(
|
||||
"Emby|http://localhost:8096|your-api-key|emby"
|
||||
|
||||
# Temporarily cover HOST2's Emby during maintenance:
|
||||
# "Emby-Jayred365|http://100.x.x.x:8096|HOST2-api-key|emby"
|
||||
|
||||
# Jellyfin instance (separate port):
|
||||
# "Jellyfin|http://localhost:8097|jellyfin-api-key|jellyfin"
|
||||
)
|
||||
```
|
||||
|
||||
Type field: `emby` | `jellyfin` | `plex` — controls which API endpoint format is used.
|
||||
Entries with placeholder API keys are skipped automatically.
|
||||
Comment out unused entries rather than deleting — placeholders show what's available.
|
||||
|
||||
**Tdarr does NOT belong here.** Tdarr encodes full video files — large working files
|
||||
fill the ramdisk rapidly and cause constant flips. Tdarr belongs on SSD permanently.
|
||||
|
||||
### Session Display
|
||||
|
||||
```
|
||||
━━━ Active Emby Sessions ━━━
|
||||
Total: 7 | Live TV: 5 | Transcoding: 5 | Direct: 2
|
||||
Storage: ramdisk
|
||||
|
||||
Sunny — ABC (WTAE) — Live TV — Transcode
|
||||
Mama Bear — Cinemax — Live TV — Transcode
|
||||
Gmer4Lfe — WAN Show — TV Show — Transcode
|
||||
```
|
||||
|
||||
Split state during a flip (ramdisk sessions draining, new sessions on SSD):
|
||||
```
|
||||
⚠️ Split state — 4 folder(s) on ramdisk / 2 on SSD
|
||||
Storage: ramdisk (4) + SSD (2)
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
transcode_manager.sh # normal run
|
||||
transcode_manager.sh --dry-run # preview flip decision without flipping
|
||||
transcode_manager.sh --status # show current state, usage, sessions, flip history
|
||||
transcode_manager.sh --log # verbose output per safety check
|
||||
transcode_manager.sh --no-log # suppress daily log write
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## transcode_cleanup.sh
|
||||
|
||||
Removes stale transcode files from ramdisk and SSD fallback. Called first in every
|
||||
3-minute cycle — cleanup before usage measurement is non-negotiable.
|
||||
|
||||
### Deletion Rules
|
||||
|
||||
A file is eligible for deletion only when ALL conditions are true:
|
||||
1. **Older than TRANSCODE_MAX_AGE minutes** (mtime — last write time). Active segments
|
||||
are written every few seconds. Not touched in 20 minutes = session ended.
|
||||
2. **Not currently open by any process** (lsof pre-built map, O(1) lookup per file).
|
||||
If ffmpeg has a file open, it is not deleted regardless of age.
|
||||
|
||||
`transcoding-temp/` is **never deleted**, even when empty. Protected by name exclusion
|
||||
in the find command — deleting it causes Emby to route all sessions to the SSD fallback.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
transcode_cleanup.sh # normal cleanup run
|
||||
transcode_cleanup.sh --dry-run # show what would be deleted
|
||||
transcode_cleanup.sh --status # show file counts, ages, open-file status per location
|
||||
transcode_cleanup.sh --log # verbose per-file output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
|
||||
# ── Paths ──────────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at
|
||||
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback
|
||||
|
||||
# ── Per-Host (master_host*.conf) ───────────────────────────────────────────────
|
||||
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes" # ramdisk mount point
|
||||
HOST1_RAMDISK_SIZE="10G" # tmpfs ceiling (not a reservation)
|
||||
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
|
||||
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
|
||||
|
||||
# ── Thresholds ─────────────────────────────────────────────────────────────────
|
||||
RAMDISK_SSD_MIN_GB=20 # minimum SSD free space before allowing SSD flip
|
||||
# prevents accidentally filling the SSD cache pool
|
||||
TRANSCODE_FLIP_WARN=3 # notify if symlink flips this many times in one hour
|
||||
# high flip count = ramdisk undersized for the load
|
||||
|
||||
# ── Cleanup ────────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_MAX_AGE=20 # minutes — files older than this are stale
|
||||
TRANSCODE_ORPHAN_AGE=30 # minutes — orphaned session folders removed after this
|
||||
|
||||
# ── Manager Mode ───────────────────────────────────────────────────────────────
|
||||
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
||||
TRANSCODE_CHECK_EMBY=true # skip threshold checks when Emby not running
|
||||
# prevents unnecessary flips overnight
|
||||
|
||||
# ── Permissions ────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_OWNER="nobody:users" # matches PUID=99 PGID=100 container env
|
||||
TRANSCODE_CHMOD="755"
|
||||
|
||||
# ── Daily Log ──────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
|
||||
TRANSCODE_LOG_RETENTION=90 # days — trimmed on every write
|
||||
TRANSCODE_STATE_FILE="/tmp/transcode_state.db" # /tmp — resets on reboot correctly
|
||||
|
||||
# ── Multi-Server ───────────────────────────────────────────────────────────────
|
||||
TRANSCODE_SERVERS=(
|
||||
"ContainerName|http://host:port|api-key|emby" # type: emby | jellyfin | plex
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Schedule
|
||||
|
||||
```
|
||||
At Startup of Array (via array_start.sh in unRAID_Essentials/):
|
||||
ramdisk_setup.sh — creates ramdisk, symlink, transcoding-temp
|
||||
Must run BEFORE Emby starts
|
||||
|
||||
Every 3 minutes (via transcode_management.sh in Orchestrators/):
|
||||
1. transcode_cleanup.sh — remove stale files, check flip-back
|
||||
2. transcode_manager.sh — check usage, flip if needed, display sessions
|
||||
|
||||
Do NOT schedule transcode_manager.sh or transcode_cleanup.sh directly.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Sessions Landing on SSD Despite Symlink Pointing at Ramdisk
|
||||
|
||||
```bash
|
||||
# Check 1 — Docker mount propagation (most common cause):
|
||||
docker inspect Emby | grep Propagation
|
||||
# Expected: "Propagation": "shared"
|
||||
# Wrong: "Propagation": "rprivate"
|
||||
#
|
||||
# Fix: Update Emby Extra Parameters, restart Emby:
|
||||
# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
|
||||
# Remove any standard path mapping for the transcode directory.
|
||||
|
||||
# Check 2 — transcoding-temp exists on ramdisk:
|
||||
ls /mnt/ramdisk_transcodes/
|
||||
# Expected: transcoding-temp/
|
||||
#
|
||||
# Fix if missing:
|
||||
mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
|
||||
chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp
|
||||
|
||||
# Check 3 — no duplicate SSD mount in Emby template:
|
||||
docker inspect Emby | grep -A3 "Mounts"
|
||||
# Should show only /mnt/ram-transcode → /ext-ram-transcode
|
||||
# Should NOT show /mnt/cache/Temp_Storage/... as a second mount
|
||||
```
|
||||
|
||||
### Flip Count High — 3+ Per Hour
|
||||
|
||||
```bash
|
||||
# Ramdisk filling up regularly — load exceeds the current ceiling.
|
||||
# Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB"
|
||||
#
|
||||
# If peak is close to RAMDISK_WARN_GB → increase ramdisk size:
|
||||
# master_host1.conf
|
||||
HOST1_RAMDISK_SIZE="12G" # increase by 2G
|
||||
HOST1_RAMDISK_WARN_GB=10.5 # adjust thresholds accordingly
|
||||
HOST1_RAMDISK_LOW_GB=8.5
|
||||
|
||||
# Remount at new size — run ramdisk_setup.sh manually:
|
||||
ramdisk_setup.sh --log
|
||||
# Ramdisk must be unmounted first if already mounted:
|
||||
# umount /mnt/ramdisk_transcodes && ramdisk_setup.sh --log
|
||||
```
|
||||
|
||||
### Ramdisk Not Mounting at Array Start
|
||||
|
||||
```bash
|
||||
# Check if tmpfs is mounted:
|
||||
mountpoint /mnt/ramdisk_transcodes
|
||||
# "not a mountpoint" → setup failed or not run yet
|
||||
|
||||
# Run manually to see the error:
|
||||
ramdisk_setup.sh --log
|
||||
|
||||
# Common causes:
|
||||
# /mnt/ramdisk_transcodes missing → mkdir -p /mnt/ramdisk_transcodes
|
||||
# Insufficient RAM → check free RAM: free -h
|
||||
# RAMDISK_SIZE too large → reduce HOST*_RAMDISK_SIZE
|
||||
```
|
||||
|
||||
### Emergency Manual Flip
|
||||
|
||||
```bash
|
||||
# Flip to SSD immediately — all new sessions go to SSD:
|
||||
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
|
||||
|
||||
# Flip back to ramdisk — all new sessions go to ramdisk:
|
||||
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
|
||||
|
||||
# Check current symlink target:
|
||||
readlink /mnt/ram-transcode
|
||||
|
||||
# Existing sessions in progress are NEVER affected — only new sessions follow the flip.
|
||||
```
|
||||
|
||||
### Increasing Ramdisk Size After Initial Setup
|
||||
|
||||
```bash
|
||||
# 1. Set new size and thresholds in master_host*.conf
|
||||
# 2. Unmount the existing ramdisk (no sessions should be active):
|
||||
umount /mnt/ramdisk_transcodes
|
||||
|
||||
# 3. Re-run setup to mount at new size:
|
||||
ramdisk_setup.sh --log
|
||||
|
||||
# 4. Verify:
|
||||
df -h /mnt/ramdisk_transcodes
|
||||
# Should show new size as total
|
||||
```
|
||||
+105
-692
@@ -1,724 +1,137 @@
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# 🎬 TRANSCODING
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# ━━━━━ TRANSCODES ━━━━━
|
||||
|
||||
**Ramdisk-based transcode storage with automatic SSD fallback, session-safe symlink
|
||||
flipping, and stale file cleanup.** Emby transcodes to RAM at full speed. If the ramdisk
|
||||
fills up, new sessions automatically shift to SSD — without interrupting anything
|
||||
already playing. When pressure drops, new sessions shift back.
|
||||
**Ramdisk-based transcode storage with automatic SSD fallback.** Emby transcodes to RAM
|
||||
at full speed. When the ramdisk fills, new sessions shift to SSD automatically —
|
||||
without interrupting anything already playing. When pressure drops, new sessions shift
|
||||
back to RAM.
|
||||
|
||||
> **This system has two subtle configuration requirements that are not obvious and
|
||||
> both were discovered the hard way in production.** The Docker mount must use
|
||||
> `bind-propagation=shared` or symlink flips are silently ignored after the first
|
||||
> flip. The `transcoding-temp` directory must be pre-created on the ramdisk or Emby
|
||||
> finds the SSD version and routes all sessions there until restarted. Both are
|
||||
> documented and both will bite you if missed.
|
||||
> **Two configuration requirements that are not obvious and were both discovered the
|
||||
> hard way in production.** The Docker mount must use `bind-propagation=shared` or
|
||||
> symlink flips are silently ignored after the first flip. The `transcoding-temp`
|
||||
> directory must be pre-created on the ramdisk or Emby finds the SSD version and
|
||||
> routes all sessions there until restarted. Both are documented in Manual-Transcoding.md.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Three Storage Options, None Perfect on Their Own**
|
||||
Hard drives: seek times cause buffering on multi-stream transcoding. SSD: fast enough,
|
||||
but constant small file writes at Emby volume accelerate wear over months. RAM: fastest,
|
||||
no wear, files vanish instantly on session end — but limited by available memory.
|
||||
Fix: RAM by default, SSD as a safety net. The system manages the transition automatically.
|
||||
|
||||
**Changing Transcode Location Requires Restarting Emby**
|
||||
Configuring Emby to switch between ramdisk and SSD requires a restart. Restarting
|
||||
during active streams drops everyone. A 7-person household with 5 Live TV streams at
|
||||
9pm is not a good moment to restart Emby.
|
||||
Fix: symlink indirection. Emby points at a fixed path. The symlink target changes.
|
||||
ffmpeg resolves the symlink once at session start — existing sessions are completely
|
||||
unaffected by flips. Only new sessions follow the new target.
|
||||
|
||||
**Docker Bind Mount Silently Ignored After First Flip**
|
||||
Symlink flip from ramdisk → SSD worked. Flip back: nothing. All new sessions still land
|
||||
on SSD. The symlink on the host is correct. Emby doesn't see it.
|
||||
Cause: Docker's default `rprivate` propagation resolves the symlink target at mount time
|
||||
and locks that inode. Subsequent flips are invisible to the container.
|
||||
Fix: `bind-propagation=shared` in Extra Parameters. Host mount changes propagate into
|
||||
the container in real time. Requires `--mount` syntax — the path mapping UI doesn't
|
||||
support propagation.
|
||||
|
||||
**Sessions Drifting to SSD After a Day of Operation**
|
||||
System working correctly for hours, then sessions gradually drift to SSD despite the
|
||||
ramdisk having plenty of space.
|
||||
Cause: cleanup was removing the empty `transcoding-temp` directory from the ramdisk.
|
||||
Emby then found the SSD fallback version and routed all sessions there.
|
||||
Fix: `transcoding-temp` is excluded from cleanup by name. `ramdisk_setup.sh` pre-creates
|
||||
it at mount time. Both protections together prevent this permanently.
|
||||
|
||||
**lsof Per File on a Live TV System**
|
||||
Early cleanup called `lsof filename` per file to check if anything had it open. On a busy
|
||||
Live TV night with 5 simultaneous streams, the ramdisk contains thousands of HLS segment
|
||||
files — thousands of subprocess calls every 3 minutes.
|
||||
Fix: lsof called once per location to build a complete open-file map. All subsequent
|
||||
checks are O(1) lookups against that map.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Three Storage Options, None Perfect on Their Own
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
Emby transcodes generate hundreds of small HLS segment files written and read
|
||||
continuously at high throughput. Where those files live matters a lot:
|
||||
Three scripts, one goal: keep transcodes on RAM, fall back to SSD when needed.
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Hard drives:
|
||||
# Seek times cause buffering on simultaneous multi-stream transcoding.
|
||||
# 5 streams trying to seek on spinning disks = constant buffering for everyone.
|
||||
#
|
||||
# SSD (cache pool):
|
||||
# Fast enough for any realistic load.
|
||||
# But: constant small file writes at Emby volume accelerate SSD wear.
|
||||
# A busy Live TV night writes and deletes thousands of segment files.
|
||||
# Over months, this adds up.
|
||||
#
|
||||
# RAM (tmpfs):
|
||||
# Fastest possible — no disk I/O at all.
|
||||
# No wear — RAM doesn't have write cycles.
|
||||
# Files disappear instantly on session end — no cleanup needed for normal exits.
|
||||
# One risk: running out of RAM during heavy load.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
`ramdisk_setup.sh` runs at array start — creates the tmpfs, SSD fallback directory,
|
||||
symlink, and pre-creates `transcoding-temp`. Everything that must exist before Emby starts.
|
||||
|
||||
The correct answer is RAM by default, SSD as a safety net. The ramdisk handles normal
|
||||
operation. The SSD absorbs unexpected load spikes. The system manages the transition
|
||||
automatically.
|
||||
`transcode_cleanup.sh` runs first in every 3-minute cycle — removes stale files from both
|
||||
ramdisk and SSD. Cleans up before usage is measured, so the manager sees real load.
|
||||
|
||||
`transcode_manager.sh` runs second — measures ramdisk usage, flips the symlink if
|
||||
thresholds are crossed, runs safety checks, displays active sessions, writes the daily log.
|
||||
|
||||
The symlink is the mechanism that makes this seamless. Emby writes to a fixed path. That
|
||||
path is a symlink whose target is managed at runtime. Sessions in progress never notice.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Changing Transcode Location Requires Restarting Emby
|
||||
|
||||
The obvious approach — configure Emby to use the ramdisk, configure SSD as fallback in
|
||||
Emby's settings — requires restarting Emby to switch between them. Restarting Emby
|
||||
during active streams drops everyone. A 7-person household with 5 Live TV streams
|
||||
running at 9pm is not a good moment to restart Emby.
|
||||
|
||||
The fix: symlink indirection. Emby points at a fixed path that never changes.
|
||||
The symlink target changes. ffmpeg resolves the symlink once at session start and
|
||||
holds the resolved path — existing sessions are completely unaffected by symlink
|
||||
changes. Only new sessions care about where the symlink currently points.
|
||||
|
||||
The transition is seamless. Sessions in progress when the flip happens continue
|
||||
writing to wherever they started. Only new sessions after the flip go to the new
|
||||
location. No restart. No interruption.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Docker Bind Mount Silently Ignored After First Flip
|
||||
|
||||
Got the symlink system working. First flip from ramdisk to SSD: works. Flip back to
|
||||
ramdisk: nothing. All new sessions still land on SSD. The symlink on the host clearly
|
||||
points at the ramdisk — `readlink /mnt/ram-transcode` shows the correct path — but
|
||||
Emby keeps writing to SSD.
|
||||
|
||||
The cause: Docker's default bind mount uses `rprivate` propagation. With `rprivate`,
|
||||
Docker resolves the symlink target at first mount and locks that inode. When the
|
||||
symlink flips, Docker ignores it — it already has a private SSD binding locked in.
|
||||
The container sees the path but the mount behind it doesn't update.
|
||||
|
||||
The fix: `bind-propagation=shared` in Extra Parameters. With shared propagation, host
|
||||
mount changes propagate into the container in real time. Symlink flips on the host are
|
||||
immediately visible inside the container. This requires using `--mount` syntax instead
|
||||
of a standard template path mapping — that syntax supports propagation, the path
|
||||
mapping UI does not.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Sessions Drifting to SSD After a Day of Operation
|
||||
|
||||
System working correctly for hours. Then gradually sessions start landing on SSD even
|
||||
though the ramdisk has plenty of space and the symlink points at the ramdisk. Next day
|
||||
all sessions are on SSD.
|
||||
|
||||
The cause: cleanup was removing the empty `transcoding-temp` directory from the
|
||||
ramdisk. When `transcoding-temp` doesn't exist on the ramdisk, Emby searches its
|
||||
accessible paths for an existing one. It finds the SSD fallback version. All subsequent
|
||||
sessions route there until Emby is restarted.
|
||||
|
||||
The fix: `transcoding-temp` is protected from cleanup — excluded by name from the
|
||||
`find` command. And `ramdisk_setup.sh` pre-creates it at mount time so Emby always
|
||||
finds it on the ramdisk first. Both fixes together prevent this permanently.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 lsof Per File on a Live TV System
|
||||
|
||||
Early cleanup implementation called `lsof filename` per file to check if anything had
|
||||
it open. On a busy Live TV night with 5 simultaneous streams, the ramdisk contains
|
||||
thousands of HLS segment files. Calling `lsof` once per file was creating thousands
|
||||
of subprocess calls every 3 minutes. The cleanup script was spending more time on
|
||||
lsof calls than on actual cleanup.
|
||||
|
||||
The fix: `lsof` is called once per location to build a complete open-file map. All
|
||||
subsequent file checks are O(1) lookups against that pre-built map. Thousands of files,
|
||||
one `lsof` call, no performance penalty.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE DESIGN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
### ── The Symlink Architecture ─────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Emby is configured to write transcodes to TRANSCODE_LINK.
|
||||
# TRANSCODE_LINK is a symlink — its target is managed at runtime.
|
||||
#
|
||||
# Normal operation (ramdisk has headroom):
|
||||
# /mnt/ram-transcode → /mnt/ramdisk_transcodes/ (ramdisk, fast, no wear)
|
||||
#
|
||||
# Heavy load (ramdisk filling up):
|
||||
# /mnt/ram-transcode → /mnt/cache/Temp_Storage/Emby/Transcodes/ (SSD)
|
||||
#
|
||||
# Emby doesn't know this symlink exists. It writes to /mnt/ram-transcode.
|
||||
# ffmpeg resolves the symlink ONCE when a session starts.
|
||||
# After that it holds a direct reference to the actual directory.
|
||||
# Flipping the symlink has ZERO effect on sessions already in progress.
|
||||
# Only NEW sessions care about where the symlink currently points.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── The Threshold Logic ──────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Two thresholds with a hysteresis gap to prevent flip-flopping.
|
||||
#
|
||||
RAMDISK_WARN_GB=8.8 # flip to SSD when ramdisk usage exceeds this
|
||||
RAMDISK_LOW_GB=6.5 # flip back to ramdisk when usage drops below this
|
||||
#
|
||||
# The 2.3GB hysteresis gap:
|
||||
# Without this gap: usage hovers at 8.7GB → flip to SSD → sessions drain
|
||||
# → usage drops to 8.5GB → flip back → new sessions fill → flip again
|
||||
# The symlink would oscillate every few minutes under moderate load.
|
||||
#
|
||||
# With the gap: usage must drop all the way to 6.5GB before flipping back.
|
||||
# That requires multiple sessions to end completely — a genuine recovery,
|
||||
# not a brief fluctuation. Stable, predictable behaviour.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── What Lives Where ──────────────────────────────────────────────────────────
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
```
|
||||
/mnt/ramdisk_transcodes/ ← tmpfs (HOST*_RAMDISK_SIZE ceiling)
|
||||
transcoding-temp/ ← pre-created by ramdisk_setup.sh — always here
|
||||
E0D8DC/ ← Emby session (Live TV HLS segments)
|
||||
F1A9BB/ ← another session
|
||||
unRAID_Essentials/
|
||||
array_start.sh ──────────────────────────────► ramdisk_setup.sh (at array start)
|
||||
|
||||
/mnt/ram-transcode ← symlink — managed at runtime by transcode_manager.sh
|
||||
currently points at: /mnt/ramdisk_transcodes/
|
||||
Orchestrators/
|
||||
transcode_management.sh ──── cleanup first ──► transcode_cleanup.sh
|
||||
──── then manager ──► transcode_manager.sh
|
||||
(every 3 minutes — order non-negotiable)
|
||||
|
||||
/mnt/cache/Temp_Storage/Emby/Transcodes/ ← SSD fallback
|
||||
transcoding-temp/ ← also pre-created — Emby finds ramdisk version first
|
||||
xyz789/ ← sessions that started when ramdisk was full
|
||||
Monitors/
|
||||
weekly_health_digest.sh ◄─── reads ──────────── TRANSCODE_DAILY_LOG
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ⚙️ DOCKER MOUNT — READ THIS BEFORE ANYTHING ELSE
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
> **This is the most important configuration requirement in the entire folder.**
|
||||
> Get this wrong and symlink flips silently stop working after the first flip.
|
||||
> The system appears to work initially and fails subtly.
|
||||
Do not schedule `transcode_cleanup.sh` or `transcode_manager.sh` directly.
|
||||
Both are called by `transcode_management.sh` in the correct order.
|
||||
|
||||
---
|
||||
|
||||
### ── Required Mount Configuration ───────────────────────────────────────────
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
```bash
|
||||
# In Emby's Extra Parameters in the unRAID Docker template:
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# This REPLACES the transcode path in the standard template path mapping UI.
|
||||
# Do NOT add this via the path mapping UI — that UI does not support propagation.
|
||||
# Use Extra Parameters only.
|
||||
```
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `ramdisk_setup.sh` | Create tmpfs, SSD fallback dir, symlink, transcoding-temp | At array start (via array_start.sh) |
|
||||
| `transcode_cleanup.sh` | Remove stale files, check for flip-back opportunity | Every 3 min via transcode_management.sh — runs first |
|
||||
| `transcode_manager.sh` | Check usage, flip symlink, safety checks, session display, daily log | Every 3 min via transcode_management.sh — runs second |
|
||||
|
||||
---
|
||||
|
||||
### ── Why `shared` Is Required ────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# rprivate (Docker's default):
|
||||
# Docker resolves the symlink target at first mount and locks that inode.
|
||||
# Flip: ramdisk → SSD → flip works.
|
||||
# Flip: SSD → ramdisk → Docker ignores it. Container still sees SSD binding.
|
||||
# All sessions continue to land on SSD forever until Emby restarts.
|
||||
# Symptom: symlink on host is correct, Emby still uses SSD. Confusing.
|
||||
#
|
||||
# shared:
|
||||
# Host mount changes propagate into the container in real time.
|
||||
# Flip: ramdisk → SSD → visible in container immediately ✅
|
||||
# Flip: SSD → ramdisk → visible in container immediately ✅
|
||||
# System works as designed. Every flip is immediate and correct.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Verify the Mount ─────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Check the propagation — must show "shared" not "rprivate":
|
||||
docker inspect Emby | grep -A4 "ext-ram"
|
||||
# Expected output includes: "Propagation": "shared"
|
||||
# Wrong output: "Propagation": "rprivate"
|
||||
|
||||
# Check Emby's transcode path setting (inside container):
|
||||
docker exec Emby cat /config/config/encoding.xml | grep TranscodingTempPath
|
||||
# Expected: /ext-ram-transcode
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── What NOT to Do ──────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# DO NOT add a static SSD transcode path as a second path mapping in the template:
|
||||
# /mnt/cache/Temp_Storage/Emby/Transcodes → /ssd-transcode
|
||||
#
|
||||
# If the SSD path is mounted inside the container, Emby can see it as an
|
||||
# accessible transcode location. It will route sessions there independently of
|
||||
# the symlink — completely bypassing the management system. Sessions land on SSD
|
||||
# regardless of symlink state. The whole system stops working.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🚀 ramdisk_setup.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Creates the ramdisk, SSD fallback directory, symlink, and `transcoding-temp` on the
|
||||
ramdisk. Run once at array start. Idempotent — if the ramdisk is already mounted it
|
||||
reports status and exits cleanly.
|
||||
|
||||
```bash
|
||||
# Scheduled: At Startup of Array (via array_start.sh)
|
||||
# This runs BEFORE Emby starts — order matters in ARRAY_START_SCRIPTS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── What It Creates ──────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 1. RAMDISK_PATH (/mnt/ramdisk_transcodes)
|
||||
# mount -t tmpfs -o size=HOST*_RAMDISK_SIZE tmpfs /mnt/ramdisk_transcodes
|
||||
# In-memory tmpfs — uses only as much RAM as actually needed.
|
||||
# RAMDISK_SIZE is a ceiling, not a reservation — an empty ramdisk uses ~0 RAM.
|
||||
#
|
||||
# 2. transcoding-temp inside the ramdisk
|
||||
# mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
|
||||
# Pre-created so Emby always finds it here first.
|
||||
# Without this: Emby creates transcoding-temp at its first writable location,
|
||||
# which may be the SSD fallback even when the symlink points at the ramdisk.
|
||||
# chown nobody:users — correct ownership for Emby to write as PUID=99
|
||||
#
|
||||
# 3. TRANSCODE_SSD (/mnt/cache/Temp_Storage/Emby/Transcodes/)
|
||||
# mkdir -p — creates if missing, silent if exists
|
||||
# Also pre-creates transcoding-temp/ inside SSD fallback for consistency
|
||||
#
|
||||
# 4. TRANSCODE_LINK (/mnt/ram-transcode)
|
||||
# ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
|
||||
# Always reset to ramdisk at array start — clean state every boot
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Verify After Setup ───────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# Run these after first setup — all three must pass:
|
||||
|
||||
mountpoint /mnt/ramdisk_transcodes
|
||||
# Expected: /mnt/ramdisk_transcodes is a mountpoint
|
||||
|
||||
readlink /mnt/ram-transcode
|
||||
# Expected: /mnt/ramdisk_transcodes
|
||||
|
||||
ls /mnt/ramdisk_transcodes/
|
||||
# Expected: transcoding-temp/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
ramdisk_setup.sh # normal run (at array start via array_start.sh)
|
||||
ramdisk_setup.sh --dry-run # show what would be created without creating
|
||||
ramdisk_setup.sh --status # show current ramdisk, symlink, and SSD state
|
||||
ramdisk_setup.sh --log # verbose — show each creation step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔄 transcode_manager.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Monitors ramdisk usage and manages the symlink direction. Called by
|
||||
`transcode_management.sh` — not scheduled directly. Every 3 minutes it checks usage,
|
||||
makes a flip decision if needed, runs safety checks, and shows active sessions.
|
||||
|
||||
```bash
|
||||
# Called by: transcode_management.sh (every 3 minutes)
|
||||
# Not scheduled directly — use transcode_management.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Three Modes ──────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
||||
#
|
||||
# smart:
|
||||
# Auto-flips between ramdisk and SSD based on usage thresholds.
|
||||
# Normal operation — use this in production.
|
||||
# Ramdisk above RAMDISK_WARN_GB → flip to SSD.
|
||||
# Ramdisk below RAMDISK_LOW_GB → flip back to ramdisk.
|
||||
#
|
||||
# ramdisk:
|
||||
# Always uses ramdisk. Never flips to SSD.
|
||||
# Use: light load server, guaranteed RAM performance, testing ramdisk behaviour.
|
||||
# Warning logged if usage exceeds threshold — no automatic action.
|
||||
#
|
||||
# ssd:
|
||||
# Always uses SSD. Never uses ramdisk.
|
||||
# Use: post-flip drain (waiting for ramdisk sessions to end naturally),
|
||||
# maintenance windows, ramdisk capacity testing.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Safety Checks — Every Run Regardless of Mode ───────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# These run on EVERY cycle — they protect against state drift:
|
||||
#
|
||||
# Symlink missing or broken
|
||||
# → Recreate pointing at ramdisk, notify
|
||||
# → Can happen after manual intervention or filesystem issue
|
||||
#
|
||||
# Ramdisk disappeared (unmounted)
|
||||
# → Auto-flip to SSD immediately, notify warning
|
||||
# → Can happen if tmpfs was manually unmounted or system ran out of memory
|
||||
#
|
||||
# SSD path missing
|
||||
# → Disable SSD fallback (mode=ssd: error)
|
||||
# → Can happen if SSD pool is not mounted
|
||||
#
|
||||
# transcoding-temp missing from ramdisk
|
||||
# → Recreate immediately, no notification
|
||||
# → Prevents sessions silently routing to SSD version
|
||||
#
|
||||
# Permissions drift
|
||||
# → Fix silently every run
|
||||
# → nobody:users ownership, TRANSCODE_CHMOD mode
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Session Display ──────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Each run shows active sessions from all configured TRANSCODE_SERVERS:
|
||||
#
|
||||
━━━ 🎬 Active Emby Sessions ━━━
|
||||
🎬 Total: 7 | 💨 Live TV: 5 | 🔄 Transcoding: 5 | 🏁 Direct: 2
|
||||
🔗 Storage: 💨 ramdisk
|
||||
|
||||
🎬 Sunny — ABC (WTAE) — Live TV — Transcode
|
||||
🎬 Mama Bear — Cinemax — Live TV — Transcode
|
||||
🎬 Gmer4Lfe — WAN Show — TV Show — Transcode
|
||||
|
||||
# Split state — sessions on both ramdisk and SSD simultaneously:
|
||||
# Normal during a flip — ramdisk sessions draining, new sessions on SSD
|
||||
#
|
||||
⚠️ Split state — 4 folder(s) on ramdisk / 2 on SSD
|
||||
🔗 Storage: 💨 ramdisk (4) + 💾 SSD (2)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Multi-Server Configuration ─────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Any number of media servers can share the same ramdisk scratch space.
|
||||
# They never touch each other's files — each writes to its own session subfolder.
|
||||
# Format: "ContainerName|URL|APIKey|Type"
|
||||
#
|
||||
TRANSCODE_SERVERS=(
|
||||
"Emby|http://localhost:8096|0c27448d93a7431f9ac63569f7655829|emby"
|
||||
|
||||
# Temporarily cover HOST2's Emby during maintenance:
|
||||
# "Emby-Jayred365|http://100.x.x.x:8096|HOST2-api-key|emby"
|
||||
|
||||
# Jellyfin instance (separate port):
|
||||
# "Jellyfin|http://localhost:8097|jellyfin-api-key|jellyfin"
|
||||
)
|
||||
#
|
||||
# Type field controls which API endpoint format is used:
|
||||
# emby → /Sessions endpoint
|
||||
# jellyfin → /Sessions endpoint (same format, same code path)
|
||||
# plex → /status/sessions (different format)
|
||||
#
|
||||
# ⚠️ Tdarr does NOT belong here.
|
||||
# Tdarr encodes full video files — large working files would fill the ramdisk
|
||||
# rapidly and cause constant flips. Tdarr belongs on SSD permanently.
|
||||
# dedicated tdarr_cleanup.sh handles Tdarr orphan management separately.
|
||||
#
|
||||
# Entries with placeholder API keys are skipped automatically.
|
||||
# Comment out unused entries rather than deleting — placeholders show what's available.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Sizing the Ramdisk ───────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# tmpfs uses only as much RAM as actually needed — RAMDISK_SIZE is a ceiling.
|
||||
# An empty ramdisk uses essentially zero RAM.
|
||||
#
|
||||
HOST1_RAMDISK_SIZE="10G" # 10GB ceiling — verified against production usage below
|
||||
|
||||
# Production data from this setup (7-household Live TV system):
|
||||
# Normal (2-3 streams) → ~1.5-2.0GB
|
||||
# Busy evening (5-6 streams) → ~3.5-4.5GB
|
||||
# Peak (8 streams, Live TV) → ~5.2GB
|
||||
# Current ramdisk → 10G with 8.8GB threshold → 1.2GB safety headroom
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# Threshold sizing — keep ~1.5-2.5GB hysteresis gap:
|
||||
#
|
||||
# ┌──────────────┬─────────────────┬────────────────┐
|
||||
# │ RAMDISK_SIZE │ RAMDISK_WARN_GB │ RAMDISK_LOW_GB │
|
||||
# ├──────────────┼─────────────────┼────────────────┤
|
||||
# │ 6G │ 4.8 │ 3.5 │
|
||||
# │ 8G │ 6.8 │ 5.5 │
|
||||
# │ 10G │ 8.8 │ 6.5 │
|
||||
# │ 12G │ 10.5 │ 8.5 │
|
||||
# └──────────────┴─────────────────┴────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🧹 transcode_cleanup.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Removes stale transcode files from both ramdisk and SSD fallback locations. Called by
|
||||
`transcode_management.sh` before `transcode_manager.sh` — order is critical.
|
||||
|
||||
```bash
|
||||
# Called by: transcode_management.sh (cleanup runs BEFORE manager)
|
||||
# Not scheduled directly — use transcode_management.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Deletion Rules ────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# A file is eligible for deletion only when ALL conditions are true:
|
||||
#
|
||||
# 1. Older than TRANSCODE_MAX_AGE minutes (mtime — last write time)
|
||||
# Active segments are being written every few seconds.
|
||||
# A file not touched in 20 minutes is from a session that ended.
|
||||
#
|
||||
# 2. Not currently open by any process
|
||||
# lsof map built once per location — O(1) lookup per file.
|
||||
# If ffmpeg has a file open, it is not deleted regardless of age.
|
||||
# "Session ended in API but ffmpeg still writing" → safe, not deleted.
|
||||
#
|
||||
# transcoding-temp directory:
|
||||
# NEVER deleted, even when empty.
|
||||
# Protected by name exclusion in find command.
|
||||
# Deleting it causes Emby to find the SSD version and route there permanently.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Why Not Session-Aware Cleanup ────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# ffmpeg generates its own folder names inside transcoding-temp, independently
|
||||
# of the media server API session IDs. There is no reliable mapping between
|
||||
# API session IDs and the actual folder names on disk.
|
||||
#
|
||||
# Attempting to correlate them: session E0D8DC → folder E0D8DC → safe assumption?
|
||||
# No. The folder name is an internal ffmpeg identifier. It may match, may not.
|
||||
# Using this correlation would falsely treat active sessions as ended.
|
||||
#
|
||||
# lsof is the correct check:
|
||||
# If ffmpeg has a file open, the file is active regardless of session state.
|
||||
# If no process has the file open, the file is safely deletable.
|
||||
# No correlation needed. No race condition. Always correct.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Flip-Back After Cleanup ──────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# transcode_cleanup.sh checks ramdisk usage AFTER removing stale files.
|
||||
# If usage dropped below RAMDISK_LOW_GB → triggers flip-back to ramdisk.
|
||||
# This handles the recovery direction so transcode_manager.sh doesn't have to.
|
||||
#
|
||||
# Without cleanup running first, the manager would see inflated usage from stale
|
||||
# files and potentially flip to SSD unnecessarily.
|
||||
# With cleanup running first, the manager always sees real active session usage.
|
||||
# This is why the order in transcode_management.sh is non-negotiable.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# ── Paths ──────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_LINK="/mnt/ram-transcode" # the symlink Emby points at
|
||||
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback
|
||||
|
||||
# ── Per-Host (master_host*.conf) ──────────────────────────────────────────
|
||||
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes" # ramdisk mount point
|
||||
HOST1_RAMDISK_SIZE="10G" # tmpfs ceiling (not a reservation)
|
||||
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
|
||||
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
|
||||
|
||||
# ── Thresholds ─────────────────────────────────────────────────────────────
|
||||
RAMDISK_SSD_MIN_GB=20 # minimum SSD free space before allowing SSD flip
|
||||
# prevents filling the SSD cache pool accidentally
|
||||
TRANSCODE_FLIP_WARN=3 # notify if symlink flips this many times in an hour
|
||||
# high flip count = ramdisk undersized
|
||||
|
||||
# ── Cleanup ────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_MAX_AGE=20 # minutes — files older than this are stale
|
||||
TRANSCODE_ORPHAN_AGE=30 # minutes — orphaned session folders removed after this
|
||||
|
||||
# ── Manager Mode ───────────────────────────────────────────────────────────
|
||||
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
||||
TRANSCODE_CHECK_EMBY=true # skip threshold checks when Emby not running
|
||||
# prevents unnecessary flips at night
|
||||
|
||||
# ── Permissions ────────────────────────────────────────────────────────────
|
||||
TRANSCODE_OWNER="nobody:users" # matches PUID=99 PGID=100 container env
|
||||
TRANSCODE_CHMOD="755"
|
||||
|
||||
# ── Daily Log ──────────────────────────────────────────────────────────────
|
||||
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
|
||||
TRANSCODE_LOG_RETENTION=90 # days — bounded, trimmed on every write
|
||||
TRANSCODE_STATE_FILE="/tmp/transcode_state.db" # /tmp — resets on reboot
|
||||
|
||||
# ── Multi-Server ───────────────────────────────────────────────────────────
|
||||
TRANSCODE_SERVERS=(
|
||||
"ContainerName|http://host:port|api-key|type" # emby|jellyfin|plex
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
At Startup of Array:
|
||||
ramdisk_setup.sh ← via array_start.sh — creates ramdisk + symlink + transcoding-temp
|
||||
Array starts
|
||||
│
|
||||
▼
|
||||
ramdisk_setup.sh
|
||||
Creates: /mnt/ramdisk_transcodes (tmpfs)
|
||||
/mnt/ramdisk_transcodes/transcoding-temp/
|
||||
/mnt/cache/Temp_Storage/Emby/Transcodes/ (SSD fallback)
|
||||
/mnt/ram-transcode → /mnt/ramdisk_transcodes (symlink)
|
||||
│
|
||||
▼
|
||||
Emby starts, reads transcode path from config
|
||||
Sees: /ext-ram-transcode (bind-mounted from /mnt/ram-transcode)
|
||||
All new sessions write to: /mnt/ram-transcode → /mnt/ramdisk_transcodes/
|
||||
|
||||
Every 3 minutes:
|
||||
transcode_management.sh ← cleanup first, then manager — order non-negotiable
|
||||
1. transcode_cleanup.sh ← remove stale files, check if flip-back possible
|
||||
2. transcode_manager.sh ← check usage, flip if needed, show sessions
|
||||
|
||||
Do NOT schedule transcode_manager.sh or transcode_cleanup.sh directly.
|
||||
Every 3 minutes (transcode_management.sh):
|
||||
│
|
||||
├─ transcode_cleanup.sh
|
||||
│ Remove files older than TRANSCODE_MAX_AGE, not open by any process
|
||||
│ transcoding-temp: never deleted
|
||||
│ If ramdisk recovered below RAMDISK_LOW_GB → trigger flip-back
|
||||
│
|
||||
└─ transcode_manager.sh
|
||||
Safety checks (symlink, ramdisk mount, transcoding-temp, permissions)
|
||||
smart mode: ramdisk > RAMDISK_WARN_GB → flip symlink to SSD
|
||||
ramdisk < RAMDISK_LOW_GB → flip symlink back to ramdisk
|
||||
Session display (all TRANSCODE_SERVERS)
|
||||
Append to TRANSCODE_DAILY_LOG
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ TROUBLESHOOTING ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Sessions Landing on SSD Despite Symlink Pointing at Ramdisk
|
||||
|
||||
```bash
|
||||
# Check 1 — Docker mount propagation (most common cause):
|
||||
docker inspect Emby | grep Propagation
|
||||
# Expected: "Propagation": "shared"
|
||||
# Wrong: "Propagation": "rprivate"
|
||||
#
|
||||
# Fix: Add to Emby Extra Parameters and restart Emby:
|
||||
# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
|
||||
# Remove any standard path mapping for the transcode directory.
|
||||
|
||||
# Check 2 — transcoding-temp exists on ramdisk:
|
||||
ls /mnt/ramdisk_transcodes/
|
||||
# Expected: transcoding-temp/
|
||||
#
|
||||
# Fix if missing:
|
||||
mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
|
||||
chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp
|
||||
|
||||
# Check 3 — no duplicate SSD mount in Emby template:
|
||||
docker inspect Emby | grep -A3 "Mounts"
|
||||
# Should show: only /mnt/ram-transcode → /ext-ram-transcode
|
||||
# Should NOT show: /mnt/cache/Temp_Storage/... as a second mount
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Flip Count High — 3+ Per Hour
|
||||
|
||||
```bash
|
||||
# Ramdisk filling up regularly — sessions draining before more arrive.
|
||||
# Check peak usage from the weekly coffee report:
|
||||
# Transcodes section → "Week peak: X.XGB"
|
||||
#
|
||||
# If peak is close to RAMDISK_WARN_GB → increase ramdisk:
|
||||
# master_host1.conf
|
||||
HOST1_RAMDISK_SIZE="12G" # increase by 2G
|
||||
HOST1_RAMDISK_WARN_GB=10.5 # adjust thresholds accordingly
|
||||
HOST1_RAMDISK_LOW_GB=8.5
|
||||
#
|
||||
# Then re-run ramdisk_setup.sh to remount at new size:
|
||||
bash /mnt/user/appdata/unraid_scripts/Transcodes/ramdisk_setup.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Ramdisk Not Mounting at Array Start
|
||||
|
||||
```bash
|
||||
# Check if tmpfs mounted:
|
||||
mountpoint /mnt/ramdisk_transcodes
|
||||
# "not a mountpoint" → setup failed or not run yet
|
||||
|
||||
# Run manually to see the error:
|
||||
bash /mnt/user/appdata/unraid_scripts/Transcodes/ramdisk_setup.sh --log
|
||||
|
||||
# Common causes:
|
||||
# /mnt/ramdisk_transcodes directory missing → mkdir -p /mnt/ramdisk_transcodes
|
||||
# Insufficient RAM → check free RAM: free -h
|
||||
# RAMDISK_SIZE too large for available RAM → reduce HOST*_RAMDISK_SIZE
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Emergency Manual Flip
|
||||
|
||||
```bash
|
||||
# Flip to SSD immediately — all new sessions go to SSD:
|
||||
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
|
||||
|
||||
# Flip back to ramdisk — all new sessions go to ramdisk:
|
||||
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
|
||||
|
||||
# Check current symlink target:
|
||||
readlink /mnt/ram-transcode
|
||||
|
||||
# Existing sessions in progress are NEVER affected by these changes.
|
||||
# Only new sessions follow the new symlink target.
|
||||
```
|
||||
+91
-47
@@ -2,61 +2,105 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Ramdisk Setup ==============================================
|
||||
# ==============================================================================================
|
||||
# Creates a tmpfs ramdisk for Emby transcodes and points the transcode symlink at it.
|
||||
# Run once at array start via User Scripts plugin — scheduled as "At Startup of Array".
|
||||
# If ramdisk is already mounted reports status and exits cleanly without remounting.
|
||||
#
|
||||
# ── WHAT IT CREATES ───────────────────────────────────────────────────────────────────────────
|
||||
# RAMDISK_PATH — tmpfs mount point (in-memory transcode location)
|
||||
# Size: HOST*_RAMDISK_SIZE (e.g. 8G) — must fit in available RAM
|
||||
# TRANSCODE_SSD — SSD fallback directory (created if missing)
|
||||
# transcode_manager.sh flips symlink here if ramdisk fills up
|
||||
# TRANSCODE_LINK — symlink pointing at RAMDISK_PATH by default
|
||||
# transcoding-temp/ — pre-created inside ramdisk so Emby always finds it there
|
||||
# Without this Emby creates it at its own first-writable location
|
||||
# which may be SSD even when symlink points at ramdisk
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Creates the tmpfs ramdisk, SSD fallback directory, transcode symlink, and
|
||||
# pre-creates transcoding-temp on the ramdisk. Run once at array start via
|
||||
# array_start.sh (unRAID_Essentials/). Idempotent — already-mounted ramdisk
|
||||
# reports status and exits cleanly. Always resets the symlink to the ramdisk
|
||||
# on boot, ensuring a clean state regardless of what state it was in before
|
||||
# shutdown.
|
||||
#
|
||||
# ── TRANSCODE_LINK SYMLINK ────────────────────────────────────────────────────────────────────
|
||||
# Emby's transcode path is set to TRANSCODE_LINK in Emby config.
|
||||
# transcode_manager.sh flips the symlink between RAMDISK_PATH and TRANSCODE_SSD at runtime
|
||||
# based on ramdisk usage — Emby sessions automatically follow without restart.
|
||||
# This script always sets the link to RAMDISK_PATH at array start (clean state).
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── STATE FILE ────────────────────────────────────────────────────────────────────────────────
|
||||
# Initialises /tmp/transcode_state.db with current target and flip tracking counters.
|
||||
# /tmp resets on reboot — correct, transcode state is ephemeral.
|
||||
# Creates four things in order:
|
||||
# 1. RAMDISK_PATH — tmpfs mount (size: HOST*_RAMDISK_SIZE ceiling, not a reservation)
|
||||
# 2. TRANSCODE_SSD — SSD fallback directory and transcoding-temp inside it
|
||||
# 3. TRANSCODE_LINK — symlink reset to RAMDISK_PATH (clean state at every boot)
|
||||
# 4. transcoding-temp/ inside RAMDISK_PATH — pre-created before Emby starts
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_RAMDISK_SIZE → RAMDISK_SIZE.
|
||||
# HOST*_RAMDISK_SIZE, HOST*_RAMDISK_WARN_GB, HOST*_RAMDISK_LOW_GB must all be
|
||||
# configured per host — different servers have different amounts of RAM available.
|
||||
# The transcoding-temp pre-creation is critical: if it doesn't exist on the ramdisk
|
||||
# when Emby starts, Emby searches all accessible paths for an existing one and finds
|
||||
# the SSD fallback version — routing all sessions there until Emby restarts.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — mount and symlink require root
|
||||
# acquire_lock — prevents duplicate runs at array start
|
||||
# detect_hosts() — correct RAMDISK_SIZE per host
|
||||
# Already mounted — exits cleanly without remounting (idempotent)
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Silent on success — startup script runs every boot — no noise when healthy
|
||||
# Initialises /tmp/transcode_state.db with current target and flip counters.
|
||||
# /tmp resets on reboot — correct, transcode state should not persist across boots.
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_RAMDISK_SIZE — tmpfs size (e.g. 8G) — must change together with WARN_GB/LOW_GB
|
||||
# HOST*_RAMDISK_WARN_GB — warn threshold in GB
|
||||
# HOST*_RAMDISK_LOW_GB — flip to SSD threshold in GB
|
||||
# HOST*_TRANSCODE_SSD — SSD fallback path
|
||||
# HOST*_TRANSCODE_SERVERS — which servers run transcoding
|
||||
# Aliased by detect_hosts()
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# TRANSCODE_LINK — symlink path Emby uses as transcode directory
|
||||
# TRANSCODE_CHMOD — permissions applied to ramdisk and fallback
|
||||
# TRANSCODE_OWNER — owner applied (default nobody:users)
|
||||
# Root Required
|
||||
# mount and symlink creation require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate runs at array start.
|
||||
#
|
||||
# Idempotent Mount Check
|
||||
# If RAMDISK_PATH is already a mountpoint, reports status and exits cleanly
|
||||
# without attempting to remount or changing anything.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# Silent on Success
|
||||
# Startup script runs on every boot — no output when healthy.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_RAMDISK_SIZE
|
||||
# tmpfs ceiling (e.g. 10G). Must change together with WARN_GB and LOW_GB.
|
||||
# Aliased by detect_hosts() → RAMDISK_SIZE.
|
||||
#
|
||||
# HOST*_RAMDISK_WARN_GB
|
||||
# Usage level at which transcode_manager.sh flips symlink to SSD.
|
||||
#
|
||||
# HOST*_RAMDISK_LOW_GB
|
||||
# Usage level at which transcode_manager.sh flips back to ramdisk.
|
||||
#
|
||||
# HOST*_TRANSCODE_SSD
|
||||
# SSD fallback directory path.
|
||||
# Aliased by detect_hosts() → TRANSCODE_SSD.
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# TRANSCODE_LINK
|
||||
# Symlink path Emby uses as its transcode directory. Must match the path
|
||||
# configured in Emby's transcoding settings.
|
||||
#
|
||||
# TRANSCODE_CHMOD / TRANSCODE_OWNER
|
||||
# Permissions applied to both ramdisk and SSD directories. (default: 755 / nobody:users)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# /tmp/transcode_state.db — current symlink target + flip count tracking
|
||||
# Lives in /tmp (ephemeral — resets on reboot correctly)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ramdisk_setup.sh
|
||||
# Normal setup run. Called by array_start.sh at boot.
|
||||
#
|
||||
# ramdisk_setup.sh --dry-run
|
||||
# Show what would be created without creating anything.
|
||||
#
|
||||
# ramdisk_setup.sh --status
|
||||
# Show current ramdisk mount state, symlink target, and SSD directory state.
|
||||
#
|
||||
# ramdisk_setup.sh --log
|
||||
# Verbose output showing each creation step.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# ramdisk_setup.sh — normal setup (runs at array start)
|
||||
# ramdisk_setup.sh --dry-run — preview without making changes
|
||||
# ramdisk_setup.sh --status — show current ramdisk and symlink state
|
||||
# ramdisk_setup.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,66 +2,98 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Transcode Cleanup ==========================================
|
||||
# ==============================================================================================
|
||||
# Removes old inactive transcode files from both ramdisk and SSD fallback locations.
|
||||
# Called every 5 minutes by transcode_manager.sh — must be fast and non-blocking.
|
||||
# Never deletes files that are currently open by any process.
|
||||
#
|
||||
# ── SAFETY RULES ──────────────────────────────────────────────────────────────────────────────
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Removes stale transcode files from both ramdisk and SSD fallback locations.
|
||||
# Called by transcode_management.sh (Orchestrators/) before transcode_manager.sh —
|
||||
# cleanup must run first so the manager sees real active-session usage, not
|
||||
# inflated usage from stale files. Must be fast and non-blocking.
|
||||
#
|
||||
# A file is eligible for deletion only if ALL conditions are true:
|
||||
# 1. Older than TRANSCODE_MAX_AGE minutes (mtime — last modified time)
|
||||
# 1. Older than TRANSCODE_MAX_AGE minutes (mtime — last write time)
|
||||
# 2. Not currently open by any process (checked via lsof pre-built map)
|
||||
#
|
||||
# ── WHY NOT SESSION-AWARE CLEANUP ─────────────────────────────────────────────────────────────
|
||||
# ffmpeg generates folder names independently of the media server API session IDs.
|
||||
# There is no reliable correlation between API session IDs and transcoding-temp subfolder
|
||||
# names — matching them would falsely treat active sessions as ended.
|
||||
# lsof is the correct and reliable active file check — if ffmpeg has a file open,
|
||||
# lsof sees it regardless of folder naming or session state.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── TRANSCODING-TEMP PROTECTION ───────────────────────────────────────────────────────────────
|
||||
# The transcoding-temp directory is excluded from deletion even when empty.
|
||||
# If cleanup removes the empty transcoding-temp folder from the ramdisk, Emby finds
|
||||
# the SSD version instead and all new sessions land on SSD until Emby restarts.
|
||||
# ! -name "transcoding-temp" exclusion in find prevents this permanently.
|
||||
# lsof Called Once, Not Per File
|
||||
# On a busy Live TV system the ramdisk contains thousands of HLS segment files.
|
||||
# Calling lsof once per file creates thousands of subprocess calls every 3 minutes.
|
||||
# lsof is called once per location to build a complete open-file map. All subsequent
|
||||
# checks are O(1) lookups against that map — thousands of files, one lsof call.
|
||||
#
|
||||
# ── PERFORMANCE ───────────────────────────────────────────────────────────────────────────────
|
||||
# lsof is called ONCE per location — never once per file.
|
||||
# Per-file lsof stalls on busy systems with live TV buffering hundreds of segments.
|
||||
# No Session-Aware Cleanup
|
||||
# ffmpeg generates folder names independently of the media server API session IDs.
|
||||
# There is no reliable correlation between API session IDs and transcoding-temp
|
||||
# subfolder names. Attempting to correlate them would falsely treat active sessions
|
||||
# as ended. lsof is the correct check — if ffmpeg has a file open, it is active
|
||||
# regardless of folder naming or session state.
|
||||
#
|
||||
# Open file check uses in-memory associative array (OPEN_FILES_MAP):
|
||||
# Was: echo "$OPEN_FILES" | grep -qF "$file" — O(n) per file → O(n²) total
|
||||
# Now: [[ -n "${OPEN_FILES_MAP[$file]:-}" ]] — O(1) per file → O(n) total
|
||||
# Same lesson as TRACKED_MAP in arr cleanup scripts.
|
||||
# transcoding-temp Is Never Deleted
|
||||
# If cleanup removes the empty transcoding-temp folder from the ramdisk, Emby
|
||||
# searches all accessible paths for an existing one, finds the SSD fallback version,
|
||||
# and routes all new sessions there until Emby restarts. The directory is excluded
|
||||
# from find by name — protected even when completely empty.
|
||||
#
|
||||
# ── POST-CLEANUP SYMLINK FLIP ─────────────────────────────────────────────────────────────────
|
||||
# After cleanup, if ramdisk has recovered below RAMDISK_LOW_GB and symlink currently
|
||||
# points at SSD → triggers transcode_manager.sh to flip back to ramdisk.
|
||||
# Post-Cleanup Flip-Back
|
||||
# After removing stale files, checks whether ramdisk usage dropped below
|
||||
# RAMDISK_LOW_GB. If so — and symlink currently points at SSD — triggers a
|
||||
# flip back to ramdisk. This is the recovery path; the manager handles
|
||||
# the fill-up path.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases RAMDISK_PATH, TRANSCODE_SSD, RAMDISK_LOW_GB.
|
||||
# Each server cleans its own transcode locations at the correct thresholds.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock "wait" — wait if previous cleanup still running
|
||||
# detect_hosts() — correct paths and thresholds per host
|
||||
# lsof timeout — lsof call capped at 15 seconds per location
|
||||
# OPEN_FILES_MAP — in-memory O(1) active file lookup
|
||||
# transcoding-temp guard — never deletes this directory
|
||||
# Silent by default — runs every 5 minutes, must not produce noise when healthy
|
||||
# Wait Lock
|
||||
# acquire_lock "wait" — waits if a previous cleanup run is still active rather
|
||||
# than exiting. The caller's 3-minute interval can overlap on a slow system.
|
||||
#
|
||||
# lsof Timeout
|
||||
# lsof call capped at 15 seconds per location — prevents blocking indefinitely
|
||||
# on a system with many open files.
|
||||
#
|
||||
# transcoding-temp Guard
|
||||
# `! -name "transcoding-temp"` in the find command — protected unconditionally.
|
||||
#
|
||||
# Silent by Default
|
||||
# Runs every 3 minutes — must not produce noise when healthy.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD / HOST*_RAMDISK_LOW_GB
|
||||
# Aliased by detect_hosts()
|
||||
# Aliased by detect_hosts() → RAMDISK_PATH / TRANSCODE_SSD / RAMDISK_LOW_GB.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# TRANSCODE_MAX_AGE — minutes before an inactive transcode file is eligible
|
||||
# TRANSCODE_ORPHAN_AGE — minutes for orphan detection (informational — future use)
|
||||
# master.conf
|
||||
#
|
||||
# TRANSCODE_MAX_AGE
|
||||
# Minutes before an inactive transcode file is eligible for deletion. (default: 20)
|
||||
#
|
||||
# TRANSCODE_ORPHAN_AGE
|
||||
# Minutes for orphan folder detection. (default: 30)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# transcode_cleanup.sh
|
||||
# Remove stale files from ramdisk and SSD. Check for flip-back opportunity.
|
||||
#
|
||||
# transcode_cleanup.sh --dry-run
|
||||
# Show which files would be deleted. No deletions, no flip.
|
||||
#
|
||||
# transcode_cleanup.sh --status
|
||||
# Show current file counts, ages, and open-file status per location.
|
||||
#
|
||||
# transcode_cleanup.sh --log
|
||||
# Verbose per-file output including age, open status, and deletion result.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# transcode_cleanup.sh — normal cleanup run
|
||||
# transcode_cleanup.sh --dry-run — show what would be deleted
|
||||
# transcode_cleanup.sh --status — show current state
|
||||
# transcode_cleanup.sh --log — verbose per-file output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
+106
-37
@@ -2,56 +2,125 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Transcode Manager ==========================================
|
||||
# ==============================================================================================
|
||||
# Manages Emby transcode storage using filesystem symlink indirection.
|
||||
# Called every 5 minutes by User Scripts — must be fast, non-blocking, and silent when healthy.
|
||||
#
|
||||
# ── HOW IT WORKS ──────────────────────────────────────────────────────────────────────────────
|
||||
# Emby's transcode path is set to TRANSCODE_LINK (a symlink).
|
||||
# ffmpeg resolves the symlink ONCE at session start — existing sessions are never affected.
|
||||
# Only NEW sessions care about where the symlink currently points.
|
||||
# Flipping the symlink mid-stream is safe — in-progress transcodes continue uninterrupted.
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Monitors ramdisk usage and manages the transcode symlink direction. Called by
|
||||
# transcode_management.sh (Orchestrators/) every 3 minutes — always after
|
||||
# transcode_cleanup.sh runs first. Must be fast, non-blocking, and silent when
|
||||
# nothing has changed.
|
||||
#
|
||||
# ── THREE MODES ───────────────────────────────────────────────────────────────────────────────
|
||||
# Emby's transcode path points at TRANSCODE_LINK (a symlink). ffmpeg resolves
|
||||
# the symlink once at session start and holds a direct reference — existing
|
||||
# sessions are completely unaffected by symlink flips. Only new sessions care
|
||||
# where the symlink currently points.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three Modes (TRANSCODE_MANAGER_MODE)
|
||||
# smart — auto-flips between ramdisk and SSD based on usage thresholds (default)
|
||||
# ramdisk above RAMDISK_WARN_GB → flip to SSD
|
||||
# ramdisk below RAMDISK_LOW_GB → flip back to ramdisk
|
||||
# ramdisk — always uses ramdisk, warns if above threshold, never flips
|
||||
# ssd — always uses SSD, never uses ramdisk
|
||||
# ssd — always uses SSD, never uses ramdisk (use during drain or maintenance)
|
||||
#
|
||||
# ── SAFETY CHECKS — EVERY RUN ─────────────────────────────────────────────────────────────────
|
||||
# Symlink missing/broken → auto-recreate pointing at ramdisk, notify
|
||||
# Ramdisk disappeared → auto-flip to SSD, notify warning
|
||||
# SSD path missing → disable SSD fallback / error if mode=ssd
|
||||
# transcoding-temp missing → recreate on ramdisk immediately
|
||||
# Permissions drift → fix silently
|
||||
# Safety Checks — Every Run Regardless of Mode
|
||||
# Symlink missing/broken → recreate pointing at ramdisk, notify
|
||||
# Ramdisk disappeared → flip to SSD immediately, notify warning
|
||||
# SSD path missing → disable SSD fallback (error if mode=ssd)
|
||||
# transcoding-temp missing → recreate on ramdisk silently
|
||||
# Permissions drift → fix silently every run
|
||||
# Emby not running → skip threshold checks, verify symlink only
|
||||
#
|
||||
# ── SESSION DISPLAY ───────────────────────────────────────────────────────────────────────────
|
||||
# Shows active Emby/Jellyfin/Plex streams with user, title, type, and play method.
|
||||
# Split state shown when sessions exist on both ramdisk and SSD simultaneously —
|
||||
# this happens naturally when symlink flips mid-session.
|
||||
# Session Display
|
||||
# Shows active streams from all configured TRANSCODE_SERVERS with user, title,
|
||||
# type (Live TV / TV Show / Movie), and play method (Transcode / Direct).
|
||||
# Split state shown when sessions exist on both ramdisk and SSD — normal during
|
||||
# a flip while ramdisk sessions drain.
|
||||
#
|
||||
# ── DAILY LOG ─────────────────────────────────────────────────────────────────────────────────
|
||||
# Appends to TRANSCODE_DAILY_LOG after each run — read by weekly_health_digest.sh.
|
||||
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSION_COUNT|SSD_SESSION_COUNT|FILES_CLEANED
|
||||
# Daily Log
|
||||
# Appends one entry per run to TRANSCODE_DAILY_LOG, read by weekly_health_digest.sh.
|
||||
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSION_COUNT|SSD_SESSION_COUNT|FILES_CLEANED
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_TRANSCODE_SERVERS, HOST*_RAMDISK_PATH,
|
||||
# HOST*_TRANSCODE_SSD, HOST*_RAMDISK_WARN_GB, HOST*_RAMDISK_LOW_GB, HOST*_RAMDISK_SIZE.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock "wait" — wait if previous run still active
|
||||
# detect_hosts() — correct paths and thresholds per host
|
||||
# DOCKER_TIMEOUT — all docker calls protected against daemon hangs
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Silent by default — runs every 5 minutes, only speaks when something changes
|
||||
# Wait Lock
|
||||
# acquire_lock "wait" — waits if the previous run is still active. The 3-minute
|
||||
# interval can overlap on a system under heavy load.
|
||||
#
|
||||
# Docker Timeout
|
||||
# DOCKER_TIMEOUT caps all docker calls against a hung daemon.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# Silent by Default
|
||||
# Runs every 3 minutes — only speaks when something changes or needs attention.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD
|
||||
# Ramdisk mount point and SSD fallback path.
|
||||
# Aliased by detect_hosts().
|
||||
#
|
||||
# HOST*_RAMDISK_WARN_GB / HOST*_RAMDISK_LOW_GB / HOST*_RAMDISK_SIZE
|
||||
# Thresholds and ceiling. Change all three together.
|
||||
# Aliased by detect_hosts().
|
||||
#
|
||||
# HOST*_TRANSCODE_SERVERS
|
||||
# Array of media server definitions: "ContainerName|URL|APIKey|Type"
|
||||
# Type: emby | jellyfin | plex
|
||||
# Aliased by detect_hosts() → TRANSCODE_SERVERS.
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# TRANSCODE_MANAGER_MODE
|
||||
# smart | ramdisk | ssd. (default: smart)
|
||||
#
|
||||
# TRANSCODE_CHECK_EMBY
|
||||
# Skip threshold checks when Emby not running — prevents unnecessary flips
|
||||
# overnight when no sessions are active. (default: true)
|
||||
#
|
||||
# TRANSCODE_FLIP_WARN
|
||||
# Notify if symlink flips this many times in one hour — indicates ramdisk
|
||||
# is undersized for the load. (default: 3)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# /tmp/transcode_state.db — current symlink target, flip count, last flip time
|
||||
# Lives in /tmp (ephemeral — resets correctly on reboot)
|
||||
# TRANSCODE_DAILY_LOG — per-run append, read by weekly_health_digest.sh
|
||||
# Trimmed to TRANSCODE_LOG_RETENTION days on each write
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# transcode_manager.sh
|
||||
# Check usage, flip if needed, run safety checks, display active sessions.
|
||||
#
|
||||
# transcode_manager.sh --dry-run
|
||||
# Show current usage and what flip decision would be made. No changes.
|
||||
#
|
||||
# transcode_manager.sh --status
|
||||
# Show current symlink target, ramdisk usage, session counts, and flip history.
|
||||
#
|
||||
# transcode_manager.sh --log
|
||||
# Verbose output including per-check results and session detail.
|
||||
#
|
||||
# transcode_manager.sh --no-log
|
||||
# Suppress daily log write. Used internally when called by transcode_cleanup.sh.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# transcode_manager.sh — normal run
|
||||
# transcode_manager.sh --dry-run — preview without making changes
|
||||
# transcode_manager.sh --status — show current state and exit
|
||||
# transcode_manager.sh --log — verbose output
|
||||
# transcode_manager.sh --no-log — suppress daily log write (called by cleanup)
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user