FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fixed reADME AND USER SCRIPT PLUGIN

Browse Source
This commit is contained in:
FailedProxy
2026-04-19 12:12:49 -04:00
parent 14cc192a1f
commit 2ee3d607fb
4 changed files with 344 additions and 254 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Transcodes/README-Transcoding.md
+138 -144
View File
@@ -36,14 +36,35 @@ This is a symlink. Emby doesn't know or care what's on the other end — it just
This is what makes the fallback seamless. Users never experience a glitch.
### Why Not Just Use the SSD Directly?
---
You could point Emby directly at the SSD and skip the ramdisk entirely. Many setups do this. The ramdisk approach gives you:
## Docker Mount — Critical
1. **Faster performance** — RAM is orders of magnitude faster than SSD for small random writes
2. **Zero SSD wear** — transcode segments are written and deleted constantly. On a busy server this adds up to significant SSD wear over months and years
3. **Automatic cleanup** — tmpfs is released back to the system when files are deleted. No fragmentation, no stale files surviving a crash
4. **Session isolation** — each session's files disappear completely when the session ends
**This must be configured correctly or the symlink system will not work.**
Emby must be configured using `--mount` in Extra Parameters — **not** as a standard path mapping in the unRAID template.
**In Emby Extra Parameters:**
```
--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
```
**Why `shared` propagation is required:**
Standard bind mounts in unRAID use `rprivate` propagation by default. With `rprivate`, Docker resolves the symlink target once — at the moment of the first mount change — and locks that inode for the lifetime of the container. When the symlink flips from ramdisk to SSD, Docker takes a private copy of that SSD binding. When the symlink later flips back to ramdisk, the container ignores it — it already has a private SSD binding locked in. All new sessions land on SSD permanently until Emby restarts.
With `shared` propagation, host mount changes propagate into the container in real time. Symlink flips on the host are immediately visible inside the container. The system works as designed.
**Verify the mount is configured correctly:**
```bash
docker inspect Emby | grep -A4 "ext-ram"
# Should show: "Propagation": "shared"
# NOT: "Propagation": "rprivate"
```
**Do NOT add a static SSD transcode path as a second mount.** If the SSD path is mounted inside the container, Emby can see it as an accessible transcode location and will route sessions there independently of the symlink — completely bypassing the management system.
> **Both of these issues were discovered in production.** The static SSD mount caused sessions to bypass the symlink. The rprivate propagation caused sessions to lock onto SSD after the first flip. Both are now fixed in the correct configuration.
---
@@ -52,38 +73,55 @@ You could point Emby directly at the SSD and skip the ramdisk entirely. Many set
### `ramdisk_setup.sh`
**Run at array start. Run once.**
Creates the tmpfs ramdisk, the SSD fallback directory, and the symlink. If the ramdisk is already mounted it reports status and exits cleanly — safe to run multiple times.
Creates the tmpfs ramdisk, the SSD fallback directory, the symlink, and — critically — the `transcoding-temp` subdirectory on the ramdisk.
```bash
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/Transcodes/ramdisk_setup.sh
```
**Why `transcoding-temp` must be pre-created:**
Emby creates a `transcoding-temp` subdirectory inside its configured transcode path when it first needs to write. If `transcoding-temp` doesn't exist on the ramdisk, Emby may find and use an existing one on the SSD fallback path instead — locking all sessions onto SSD until Emby restarts.
`ramdisk_setup.sh` creates `transcoding-temp` on the ramdisk at mount time so Emby always finds it there first.
What it creates:
```
/mnt/ramdisk_transcodes/ ← tmpfs mount (RAMDISK_SIZE ceiling)
/mnt/ram-transcode ← symlink pointing at ramdisk
/mnt/cache/Temp_Storage/Emby/Transcodes/ ← SSD fallback directory
/mnt/ramdisk_transcodes/ ← tmpfs mount (RAMDISK_SIZE ceiling)
/mnt/ramdisk_transcodes/transcoding-temp ← pre-created so Emby uses ramdisk
/mnt/ram-transcode ← symlink pointing at ramdisk
/mnt/cache/Temp_Storage/Emby/Transcodes/ ← SSD fallback directory
```
After running, verify:
```bash
mountpoint /mnt/ramdisk_transcodes # should say "is a mountpoint"
readlink /mnt/ram-transcode # should show /mnt/ramdisk_transcodes
mountpoint /mnt/ramdisk_transcodes # should say "is a mountpoint"
readlink /mnt/ram-transcode # should show /mnt/ramdisk_transcodes
ls /mnt/ramdisk_transcodes/ # should show transcoding-temp/
```
---
### `transcode_manager.sh`
**Run every 3 minutes via cron.**
### `transcode_management.sh` (Orchestrators/)
**Run every 3 minutes via cron. Replaces separate manager and cleanup cron entries.**
Monitors ramdisk usage and manages the symlink direction. The main brain of the system.
Runs `transcode_cleanup.sh` first then `transcode_manager.sh` in the correct order. Cleanup runs first so the manager sees accurate post-cleanup usage before making threshold decisions.
```bash
# Scheduled as: */3 * * * *
/mnt/user/appdata/unraid_scripts/Transcodes/transcode_manager.sh
/mnt/user/appdata/unraid_scripts/Orchestrators/transcode_management.sh
```
Also tracks daily transcode statistics to `/boot/config/transcode_daily.db` — read by `weekly_health_digest.sh` for the weekly report.
---
### `transcode_manager.sh`
**Called by `transcode_management.sh` — not scheduled directly.**
Monitors ramdisk usage and manages the symlink direction.
#### Operating Modes
Set `TRANSCODE_MANAGER_MODE` in `Master.conf`:
@@ -92,22 +130,22 @@ Set `TRANSCODE_MANAGER_MODE` in `Master.conf`:
|------|----------|----------|
| `smart` | Auto-flips between ramdisk and SSD based on thresholds | Normal operation — default |
| `ramdisk` | Always uses ramdisk, never flips to SSD | Light load, guaranteed RAM performance |
| `ssd` | Always uses SSD, never uses ramdisk | Ramdisk maintenance, post-flip drain |
| `ssd` | Always uses SSD, never uses ramdisk | Maintenance, post-flip drain |
#### Smart Mode — How the Flip Works
```
Ramdisk usage rises above RAMDISK_WARN_GB (6.8GB)
Ramdisk usage rises above RAMDISK_WARN_GB (8.8GB)
→ Symlink flips to SSD
→ New sessions land on SSD
→ Existing sessions continue on ramdisk until they end
Ramdisk usage drops below RAMDISK_LOW_GB (5.5GB)
Ramdisk usage drops below RAMDISK_LOW_GB (6.5GB)
→ Symlink flips back to ramdisk
→ New sessions land on ramdisk again
```
The gap between `RAMDISK_WARN_GB` and `RAMDISK_LOW_GB` (1.3GB) is the **hysteresis gap**. It prevents the symlink from flip-flopping when usage hovers near the threshold. Without this gap you'd get constant flipping on a busy system.
The 2.3GB gap between `RAMDISK_WARN_GB` and `RAMDISK_LOW_GB` is the **hysteresis gap**. It prevents the symlink from flip-flopping when usage hovers near the threshold.
#### Safety Checks
@@ -115,11 +153,12 @@ Every run, regardless of mode:
| Condition | Action |
|-----------|--------|
| Ramdisk not mounted | Flip symlink to SSD immediately, notify warning |
| SSD path missing | Disable fallback, notify warning. If mode is `ssd` — exit |
| Ramdisk not mounted | Flip to SSD immediately, notify warning |
| SSD path missing | Disable fallback, notify warning |
| Symlink missing | Recreate pointing at ramdisk, notify |
| Symlink target gone | Reset to ramdisk, notify |
| Permissions drift | Fix silently — `chmod` and `chown` applied every run |
| `transcoding-temp` missing from ramdisk | Create it — prevents Emby falling back to SSD |
| Permissions drift | Fix silently every run |
| Emby not running | Skip threshold checks, verify symlink only |
#### Session Display
@@ -128,163 +167,91 @@ Each run queries the Emby API and shows active streams:
```
━━━ 🎬 Active Emby Sessions ━━━
🎬 Total: 7 | 💨 Live TV: 5 | 🔄 Transcoding: 5 | 🏁 Direct: 2
🎬 Total: 7 | 💨 Live TV: 5 | 🔄 Transcoding: 5 | 🏁 Direct: 2
🔗 Storage: 💨 ramdisk
🔗 Storage: 💨 ramdisk
🎬 Gmer4Lfe — MLB: Pirates vs Nationals — Live TV — Transcode
🎬 Rebecca — MLB: Pirates vs Nationals — Live TV — Transcode
🎬 Sunny — AT&T Sportsnet Pittsburgh — Live TV — Transcode
🎬 jaden — TNT — Live TV — Transcode
🎬 Mama Bear — Con-Text — TV Show — Direct Stream
🎬 Sunny — ABC (WTAE) — Live TV — Transcode
🎬 Mama Bear — Cinemax — Live TV — Transcode
🎬 Gmer4Lfe — WAN Show — TV Show — Transcode
```
**Split state** is detected and displayed when sessions exist on both ramdisk and SSD simultaneously — this happens naturally when the symlink flips while sessions are in progress:
**Split state** is detected and displayed when sessions exist on both ramdisk and SSD simultaneously — normal during a symlink flip:
```
⚠️ Split state — 4 folder(s) on ramdisk / 2 on SSD
⚠️ Older sessions remain on original location until they end naturally
🔗 Storage: 💨 ramdisk (4) + 💾 SSD (2)
```
> **Why per-session location isn't shown:** Emby's internal transcode folder names don't match the session IDs returned by the API — there is no reliable way to map a specific user to a specific folder. The folder count on each location gives you the picture you need at a glance without false precision.
#### Flip Frequency Warning
If the symlink flips `TRANSCODE_FLIP_WARN` or more times in one hour, a notification is sent. This is a signal that `RAMDISK_SIZE` may need to be increased. Real production data from this setup:
```
Normal load (2-3 streams) → ~1.5-2.0GB
Busy evening (5-6 streams) → ~3.5-4.5GB
Peak (8 streams, live TV) → ~5.2GB
Threshold trigger → 6.8GB
⚠️ Split state — 4 folder(s) on ramdisk / 2 on SSD
🔗 Storage: 💨 ramdisk (4) + 💾 SSD (2)
```
---
### `transcode_cleanup.sh`
**Run every 5 minutes via cron.**
**Called by `transcode_management.sh` — not scheduled directly.**
Removes old inactive transcode files from both ramdisk and SSD. Never deletes files that are currently open by any process.
```bash
# Scheduled as: */5 * * * *
/mnt/user/appdata/unraid_scripts/Transcodes/transcode_cleanup.sh
```
#### Deletion Rules
A file is eligible for deletion only when **all** of these are true:
1. Older than `TRANSCODE_MAX_AGE` minutes (default: 20 min)
**Deletion rules — a file is eligible only when ALL are true:**
1. Older than `TRANSCODE_MAX_AGE` minutes
2. Not currently open by any process
#### Performance Design
**`transcoding-temp` directory is protected from deletion.** Even when empty, `transcoding-temp` is never removed by cleanup. Deleting it causes Emby to fall back to the SSD version on next session start — this was the root cause of sessions drifting to SSD after a day of operation.
`lsof` is called **once per location** to build a complete list of open files — not once per file. This is critical on a busy Live TV system where a single location can have thousands of HLS segment files. A per-file `lsof` approach stalls the system under load.
---
## Docker Mount — Critical
Emby must be configured with **one transcode mount only:**
```
Host path: /mnt/ram-transcode/
Container path: /ext-ram-transcode
```
**Do NOT add a static SSD transcode path as a second volume mount.**
If the SSD path is mounted inside the container, Emby can see it as an accessible transcode location and will route sessions there independently of the symlink — completely bypassing the management system. This is not obvious and causes confusing split behavior that is hard to diagnose.
The symlink handles all routing. One mount is all that's needed.
> **This was learned in production.** The system worked correctly once the static SSD mount was removed. The symptom was new sessions landing on SSD even when the symlink pointed at ramdisk.
#### Emergency Manual Flip
If you need to manually redirect all new transcodes to SSD:
```bash
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
```
To flip back to ramdisk:
```bash
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
```
Existing sessions are unaffected — only new sessions follow the new target.
**Performance design:** `lsof` is called once per location to build a complete open file list — not once per file. On a busy Live TV system with thousands of HLS segments this is critical for performance.
---
## Configuration
All configuration in `Master.conf` under the `── TRANSCODES ──` section.
All configuration in `Master.conf` under `── TRANSCODES ──`:
```bash
# Paths
RAMDISK_PATH="/mnt/ramdisk_transcodes" # tmpfs mount point
RAMDISK_SIZE="8G" # ceiling — only uses RAM actually needed
TRANSCODE_LINK="/mnt/ram-transcode" # symlink — location never changes
RAMDISK_PATH="/mnt/ramdisk_transcodes"
RAMDISK_SIZE="10G" # bumped from 8G — peak usage ~5.2GB on busy nights
TRANSCODE_LINK="/mnt/ram-transcode"
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/"
# Smart mode thresholds
RAMDISK_WARN_GB=6.8 # flip to SSD above this
RAMDISK_LOW_GB=5.5 # flip back to ramdisk below this
RAMDISK_WARN_GB=8.8 # flip to SSD above this — 1.2GB headroom from ceiling
RAMDISK_LOW_GB=6.5 # flip back to ramdisk below this — 2.3GB hysteresis gap
RAMDISK_SSD_MIN_GB=20 # minimum SSD free space before allowing flip
# Cleanup
TRANSCODE_MAX_AGE=20 # minutes before file eligible for cleanup
TRANSCODE_ORPHAN_AGE=30 # minutes for orphaned files
# Alerts
TRANSCODE_FLIP_WARN=3 # notify if symlink flips this many times per hour
# Permissions
TRANSCODE_OWNER="nobody:users"
TRANSCODE_CHMOD="755"
# Mode
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
# Emby check
TRANSCODE_CHECK_EMBY=true
TRANSCODE_EMBY_CONTAINER="Emby"
TRANSCODE_DAILY_LOG="/boot/config/transcode_daily.db"
TRANSCODE_LOG_RETENTION=90
```
### Sizing the Ramdisk
The ramdisk is a `tmpfs` — it only uses RAM that is actually needed. `RAMDISK_SIZE` is a ceiling, not a reservation. An 8GB ramdisk that holds 2GB of files only uses 2GB of RAM.
`tmpfs` only uses RAM actually needed — `RAMDISK_SIZE` is a ceiling, not a reservation.
**Rule of thumb for sizing:**
- Count your maximum expected concurrent transcoding streams
- Multiply by ~0.5-1GB per stream (Live TV HLS streams use more than standard transcodes)
- Add 20-30% headroom above your threshold
**Production data from this setup:**
```
Normal load (2-3 streams) → ~1.5-2.0GB
Busy evening (5-6 streams) → ~3.5-4.5GB
Peak (8 streams, live TV) → ~5.2GB
Current ramdisk → 10G with 8.8GB threshold
```
**From production data on this setup:**
- 5 Live TV streams + 2 standard = ~4.5GB
- 8 streams peak = ~5.2GB
- Current ramdisk = 8GB with 6.8GB threshold — comfortable headroom
### Sizing Thresholds
If you regularly hit `TRANSCODE_FLIP_WARN` or see 3+ flips per hour, increase `RAMDISK_SIZE` by 2GB and adjust thresholds accordingly.
---
## Sizing Thresholds
When adjusting `RAMDISK_SIZE`, adjust thresholds to match:
When adjusting `RAMDISK_SIZE`, adjust thresholds to match. Keep a 1.5-2.5GB hysteresis gap between WARN and LOW:
| Ramdisk Size | RAMDISK_WARN_GB | RAMDISK_LOW_GB |
|-------------|-----------------|----------------|
| 6G | 4.8 | 3.5 |
| 8G | 6.8 | 5.5 |
| 10G | 8.5 | 7.0 |
| 12G | 10.0 | 8.5 |
Keep a 1.0-1.5GB hysteresis gap between WARN and LOW. A gap smaller than this causes flip-flop behavior near the threshold.
| 10G | 8.8 | 6.5 |
| 12G | 10.5 | 8.5 |
---
@@ -292,26 +259,53 @@ Keep a 1.0-1.5GB hysteresis gap between WARN and LOW. A gap smaller than this ca
| Script | Schedule | Purpose |
|--------|----------|---------|
| `ramdisk_setup.sh` | At Startup of Array | Create ramdisk and symlink |
| `transcode_manager.sh` | `*/3 * * * *` | Monitor usage, manage symlink, display sessions |
| `transcode_cleanup.sh` | `*/5 * * * *` | Remove old inactive files |
| `ramdisk_setup.sh` | At Startup of Array | Create ramdisk, symlink, transcoding-temp |
| `transcode_management.sh` | `*/3 * * * *` | Cleanup then manager — correct order, daily stats |
`transcode_manager.sh` and `transcode_cleanup.sh` are called by `transcode_management.sh` — do not schedule them separately.
---
## Version 2 Roadmap
## Troubleshooting
A future `advanced` mode is planned that allows per-media-type storage routing:
**Sessions landing on SSD despite symlink pointing at ramdisk:**
1. Check Docker mount propagation:
```bash
docker inspect Emby | grep Propagation
# Must show: "shared" not "rprivate"
```
Fix: Add `--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared` to Extra Parameters and restart Emby.
2. Check `transcoding-temp` exists on ramdisk:
```bash
ls /mnt/ramdisk_transcodes/
# Must show: transcoding-temp/
```
Fix: `mkdir -p /mnt/ramdisk_transcodes/transcoding-temp && chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp`
3. Check for duplicate SSD mount in Emby template — remove any static SSD transcode path mapping.
**`[LOG] Permissions fixed` on every run:**
Permissions are applied every run regardless — this is by design. If it logs every cycle it means Emby is resetting permissions on write. Not harmful — just informational.
**Flip count high — 3+ per hour:**
Ramdisk filling up regularly. Consider increasing `RAMDISK_SIZE` by 2GB and adjusting thresholds accordingly.
---
## Emergency Manual Flip
If you need to manually redirect all new transcodes to SSD:
```bash
TRANSCODE_MANAGER_MODE="advanced"
TRANSCODE_FORCE_RAMDISK=(
"LiveTv" # always ramdisk — buffering is latency sensitive
)
TRANSCODE_FORCE_SSD=(
"Audio" # music downloads — no benefit from ramdisk
)
# Everything else follows smart threshold behavior
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
```
This requires the Emby API to expose media type at session start — the groundwork (session display and media type parsing) is already in place. Target: this fall.
To flip back to ramdisk:
```bash
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
```
Existing sessions are unaffected. Only new sessions follow the new target.