feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,460 @@
|
||||
# ━━━━━ TRANSCODES — Manual ━━━━━
|
||||
|
||||
Configuration reference, Docker mount setup, threshold sizing, and troubleshooting
|
||||
for the ramdisk transcode system. Read the Docker mount section before anything else.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [Docker Mount — Required Configuration](#docker-mount--required-configuration)
|
||||
- [The Symlink Architecture](#the-symlink-architecture)
|
||||
- [ramdisk_setup.sh](#ramdisk_setupsh)
|
||||
- [transcode_manager.sh](#transcode_managersh)
|
||||
- [transcode_cleanup.sh](#transcode_cleanupsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Schedule](#schedule)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Docker Mount — Required Configuration
|
||||
|
||||
> **This is the most important configuration requirement in this folder.**
|
||||
> Get this wrong and symlink flips silently stop working after the first flip.
|
||||
> The system appears to work initially and fails subtly.
|
||||
|
||||
### Required Mount
|
||||
|
||||
In Emby's **Extra Parameters** in the unRAID Docker template:
|
||||
|
||||
```
|
||||
--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
|
||||
```
|
||||
|
||||
This **replaces** the transcode path in the standard template path mapping UI. Do NOT
|
||||
add this via the path mapping UI — that UI does not support propagation. Extra Parameters only.
|
||||
|
||||
In Emby's transcoding settings, set the transcode path to `/ext-ram-transcode`.
|
||||
|
||||
### Why `shared` Is Required
|
||||
|
||||
```
|
||||
rprivate (Docker's default):
|
||||
Docker resolves the symlink target at first mount and locks that inode.
|
||||
Flip: ramdisk → SSD → works (new target locked in)
|
||||
Flip: SSD → ramdisk → Docker ignores it. Container still sees SSD binding.
|
||||
All sessions continue to land on SSD forever until Emby restarts.
|
||||
Symptom: symlink on host is correct, Emby still uses SSD. Confusing.
|
||||
|
||||
shared:
|
||||
Host mount changes propagate into the container in real time.
|
||||
Every symlink flip is immediately visible inside the container. ✅
|
||||
```
|
||||
|
||||
### Verify the Mount
|
||||
|
||||
```bash
|
||||
# Check propagation — must show "shared":
|
||||
docker inspect Emby | grep -A4 "ext-ram"
|
||||
# Expected: "Propagation": "shared"
|
||||
|
||||
# Check Emby's transcode path setting:
|
||||
docker exec Emby cat /config/config/encoding.xml | grep TranscodingTempPath
|
||||
# Expected: /ext-ram-transcode
|
||||
```
|
||||
|
||||
### What NOT to Do
|
||||
|
||||
Do NOT add a static SSD transcode path as a second path mapping in the template:
|
||||
```
|
||||
/mnt/cache/Temp_Storage/Emby/Transcodes → /ssd-transcode ← do not do this
|
||||
```
|
||||
|
||||
If the SSD path is mounted inside the container, Emby can see it as an accessible
|
||||
transcode location. It will route sessions there independently of the symlink —
|
||||
bypassing the management system entirely. Sessions land on SSD regardless of symlink
|
||||
state. The whole system stops working.
|
||||
|
||||
---
|
||||
|
||||
## The Symlink Architecture
|
||||
|
||||
```
|
||||
Emby is configured to write transcodes to TRANSCODE_LINK (/mnt/ram-transcode).
|
||||
TRANSCODE_LINK is a symlink — its target is managed at runtime.
|
||||
|
||||
Normal operation (ramdisk has headroom):
|
||||
/mnt/ram-transcode → /mnt/ramdisk_transcodes/ (ramdisk — fast, no wear)
|
||||
|
||||
Heavy load (ramdisk filling up):
|
||||
/mnt/ram-transcode → /mnt/cache/Temp_Storage/Emby/Transcodes/ (SSD)
|
||||
|
||||
Emby doesn't know this symlink exists. It writes to /mnt/ram-transcode.
|
||||
ffmpeg resolves the symlink ONCE when a session starts.
|
||||
After that it holds a direct reference to the actual directory.
|
||||
|
||||
Flipping the symlink has ZERO effect on sessions already in progress.
|
||||
Only NEW sessions care about where the symlink currently points.
|
||||
```
|
||||
|
||||
### What Lives Where
|
||||
|
||||
```
|
||||
/mnt/ramdisk_transcodes/ ← tmpfs (HOST*_RAMDISK_SIZE ceiling)
|
||||
transcoding-temp/ ← pre-created by ramdisk_setup.sh — always here
|
||||
E0D8DC/ ← Emby session (Live TV HLS segments)
|
||||
F1A9BB/ ← another session
|
||||
|
||||
/mnt/ram-transcode ← symlink — managed by transcode_manager.sh
|
||||
currently points at: /mnt/ramdisk_transcodes/
|
||||
|
||||
/mnt/cache/Temp_Storage/Emby/Transcodes/ ← SSD fallback
|
||||
transcoding-temp/ ← also pre-created — Emby finds ramdisk version first
|
||||
xyz789/ ← sessions that started when ramdisk was full
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ramdisk_setup.sh
|
||||
|
||||
Creates the ramdisk, SSD fallback directory, symlink, and `transcoding-temp` on the
|
||||
ramdisk. Run once at array start. Idempotent — already-mounted ramdisk exits cleanly.
|
||||
|
||||
### What It Creates
|
||||
|
||||
```
|
||||
1. RAMDISK_PATH (/mnt/ramdisk_transcodes)
|
||||
mount -t tmpfs -o size=HOST*_RAMDISK_SIZE tmpfs /mnt/ramdisk_transcodes
|
||||
tmpfs uses only as much RAM as actually needed — RAMDISK_SIZE is a ceiling.
|
||||
An empty ramdisk uses essentially zero RAM.
|
||||
|
||||
2. transcoding-temp/ inside the ramdisk
|
||||
Pre-created so Emby always finds it here first.
|
||||
Without this: Emby creates transcoding-temp at its first writable location,
|
||||
which may be the SSD fallback even when the symlink points at the ramdisk.
|
||||
chown nobody:users — correct ownership for Emby (PUID=99)
|
||||
|
||||
3. TRANSCODE_SSD (/mnt/cache/Temp_Storage/Emby/Transcodes/)
|
||||
mkdir -p — created if missing, silent if exists
|
||||
Also pre-creates transcoding-temp/ inside SSD fallback for consistency
|
||||
|
||||
4. TRANSCODE_LINK (/mnt/ram-transcode)
|
||||
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
|
||||
Always reset to ramdisk at array start — clean state every boot
|
||||
```
|
||||
|
||||
### Verify After Setup
|
||||
|
||||
```bash
|
||||
# All three must pass:
|
||||
mountpoint /mnt/ramdisk_transcodes
|
||||
# Expected: /mnt/ramdisk_transcodes is a mountpoint
|
||||
|
||||
readlink /mnt/ram-transcode
|
||||
# Expected: /mnt/ramdisk_transcodes
|
||||
|
||||
ls /mnt/ramdisk_transcodes/
|
||||
# Expected: transcoding-temp/
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
ramdisk_setup.sh # normal run (called by array_start.sh)
|
||||
ramdisk_setup.sh --dry-run # show what would be created without creating
|
||||
ramdisk_setup.sh --status # show current ramdisk, symlink, and SSD state
|
||||
ramdisk_setup.sh --log # verbose — show each creation step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## transcode_manager.sh
|
||||
|
||||
Monitors ramdisk usage and manages the symlink direction. Called second in every 3-minute
|
||||
cycle by `transcode_management.sh`.
|
||||
|
||||
### Three Modes
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
||||
|
||||
# smart (default — use in production):
|
||||
# Ramdisk above RAMDISK_WARN_GB → flip symlink to SSD
|
||||
# Ramdisk below RAMDISK_LOW_GB → flip symlink back to ramdisk
|
||||
# Hysteresis gap prevents flip-flopping under moderate load
|
||||
|
||||
# ramdisk:
|
||||
# Always uses ramdisk. Warns if usage exceeds threshold. Never flips.
|
||||
# Use for: light load server, guaranteed RAM performance, testing ramdisk behaviour.
|
||||
|
||||
# ssd:
|
||||
# Always uses SSD. Never uses ramdisk.
|
||||
# Use for: post-flip drain (waiting for ramdisk sessions to end naturally),
|
||||
# maintenance windows, ramdisk capacity testing.
|
||||
```
|
||||
|
||||
### Threshold Sizing
|
||||
|
||||
```bash
|
||||
# master_host*.conf
|
||||
HOST1_RAMDISK_SIZE="10G"
|
||||
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
|
||||
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
|
||||
```
|
||||
|
||||
Keep ~1.5–2.5GB hysteresis gap between WARN and LOW. Without the gap, usage hovering
|
||||
near WARN causes constant flip-flopping. The gap requires multiple sessions to end
|
||||
completely before flipping back — a genuine recovery, not a brief fluctuation.
|
||||
|
||||
Production data from a 7-household Live TV system:
|
||||
```
|
||||
Normal (2–3 streams) → ~1.5–2.0 GB
|
||||
Busy evening (5–6 streams) → ~3.5–4.5 GB
|
||||
Peak (8 streams, Live TV) → ~5.2 GB
|
||||
Current setup: 10G ramdisk, 8.8 GB threshold → 1.2 GB safety headroom
|
||||
```
|
||||
|
||||
Recommended thresholds by ramdisk size:
|
||||
|
||||
| RAMDISK_SIZE | RAMDISK_WARN_GB | RAMDISK_LOW_GB |
|
||||
|-------------|----------------|----------------|
|
||||
| 6G | 4.8 | 3.5 |
|
||||
| 8G | 6.8 | 5.5 |
|
||||
| 10G | 8.8 | 6.5 |
|
||||
| 12G | 10.5 | 8.5 |
|
||||
|
||||
### Multi-Server Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
TRANSCODE_SERVERS=(
|
||||
"Emby|http://localhost:8096|your-api-key|emby"
|
||||
|
||||
# Temporarily cover HOST2's Emby during maintenance:
|
||||
# "Emby-Jayred365|http://100.x.x.x:8096|HOST2-api-key|emby"
|
||||
|
||||
# Jellyfin instance (separate port):
|
||||
# "Jellyfin|http://localhost:8097|jellyfin-api-key|jellyfin"
|
||||
)
|
||||
```
|
||||
|
||||
Type field: `emby` | `jellyfin` | `plex` — controls which API endpoint format is used.
|
||||
Entries with placeholder API keys are skipped automatically.
|
||||
Comment out unused entries rather than deleting — placeholders show what's available.
|
||||
|
||||
**Tdarr does NOT belong here.** Tdarr encodes full video files — large working files
|
||||
fill the ramdisk rapidly and cause constant flips. Tdarr belongs on SSD permanently.
|
||||
|
||||
### Session Display
|
||||
|
||||
```
|
||||
━━━ Active Emby Sessions ━━━
|
||||
Total: 7 | Live TV: 5 | Transcoding: 5 | Direct: 2
|
||||
Storage: ramdisk
|
||||
|
||||
Sunny — ABC (WTAE) — Live TV — Transcode
|
||||
Mama Bear — Cinemax — Live TV — Transcode
|
||||
Gmer4Lfe — WAN Show — TV Show — Transcode
|
||||
```
|
||||
|
||||
Split state during a flip (ramdisk sessions draining, new sessions on SSD):
|
||||
```
|
||||
⚠️ Split state — 4 folder(s) on ramdisk / 2 on SSD
|
||||
Storage: ramdisk (4) + SSD (2)
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
transcode_manager.sh # normal run
|
||||
transcode_manager.sh --dry-run # preview flip decision without flipping
|
||||
transcode_manager.sh --status # show current state, usage, sessions, flip history
|
||||
transcode_manager.sh --log # verbose output per safety check
|
||||
transcode_manager.sh --no-log # suppress daily log write
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## transcode_cleanup.sh
|
||||
|
||||
Removes stale transcode files from ramdisk and SSD fallback. Called first in every
|
||||
3-minute cycle — cleanup before usage measurement is non-negotiable.
|
||||
|
||||
### Deletion Rules
|
||||
|
||||
A file is eligible for deletion only when ALL conditions are true:
|
||||
1. **Older than TRANSCODE_MAX_AGE minutes** (mtime — last write time). Active segments
|
||||
are written every few seconds. Not touched in 20 minutes = session ended.
|
||||
2. **Not currently open by any process** (lsof pre-built map, O(1) lookup per file).
|
||||
If ffmpeg has a file open, it is not deleted regardless of age.
|
||||
|
||||
`transcoding-temp/` is **never deleted**, even when empty. Protected by name exclusion
|
||||
in the find command — deleting it causes Emby to route all sessions to the SSD fallback.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
transcode_cleanup.sh # normal cleanup run
|
||||
transcode_cleanup.sh --dry-run # show what would be deleted
|
||||
transcode_cleanup.sh --status # show file counts, ages, open-file status per location
|
||||
transcode_cleanup.sh --log # verbose per-file output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
|
||||
# ── Paths ──────────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at
|
||||
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback
|
||||
|
||||
# ── Per-Host (master_host*.conf) ───────────────────────────────────────────────
|
||||
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes" # ramdisk mount point
|
||||
HOST1_RAMDISK_SIZE="10G" # tmpfs ceiling (not a reservation)
|
||||
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
|
||||
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
|
||||
|
||||
# ── Thresholds ─────────────────────────────────────────────────────────────────
|
||||
RAMDISK_SSD_MIN_GB=20 # minimum SSD free space before allowing SSD flip
|
||||
# prevents accidentally filling the SSD cache pool
|
||||
TRANSCODE_FLIP_WARN=3 # notify if symlink flips this many times in one hour
|
||||
# high flip count = ramdisk undersized for the load
|
||||
|
||||
# ── Cleanup ────────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_MAX_AGE=20 # minutes — files older than this are stale
|
||||
TRANSCODE_ORPHAN_AGE=30 # minutes — orphaned session folders removed after this
|
||||
|
||||
# ── Manager Mode ───────────────────────────────────────────────────────────────
|
||||
TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
||||
TRANSCODE_CHECK_EMBY=true # skip threshold checks when Emby not running
|
||||
# prevents unnecessary flips overnight
|
||||
|
||||
# ── Permissions ────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_OWNER="nobody:users" # matches PUID=99 PGID=100 container env
|
||||
TRANSCODE_CHMOD="755"
|
||||
|
||||
# ── Daily Log ──────────────────────────────────────────────────────────────────
|
||||
TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
|
||||
TRANSCODE_LOG_RETENTION=90 # days — trimmed on every write
|
||||
TRANSCODE_STATE_FILE="/tmp/transcode_state.db" # /tmp — resets on reboot correctly
|
||||
|
||||
# ── Multi-Server ───────────────────────────────────────────────────────────────
|
||||
TRANSCODE_SERVERS=(
|
||||
"ContainerName|http://host:port|api-key|emby" # type: emby | jellyfin | plex
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Schedule
|
||||
|
||||
```
|
||||
At Startup of Array (via array_start.sh in unRAID_Essentials/):
|
||||
ramdisk_setup.sh — creates ramdisk, symlink, transcoding-temp
|
||||
Must run BEFORE Emby starts
|
||||
|
||||
Every 3 minutes (via transcode_management.sh in Orchestrators/):
|
||||
1. transcode_cleanup.sh — remove stale files, check flip-back
|
||||
2. transcode_manager.sh — check usage, flip if needed, display sessions
|
||||
|
||||
Do NOT schedule transcode_manager.sh or transcode_cleanup.sh directly.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Sessions Landing on SSD Despite Symlink Pointing at Ramdisk
|
||||
|
||||
```bash
|
||||
# Check 1 — Docker mount propagation (most common cause):
|
||||
docker inspect Emby | grep Propagation
|
||||
# Expected: "Propagation": "shared"
|
||||
# Wrong: "Propagation": "rprivate"
|
||||
#
|
||||
# Fix: Update Emby Extra Parameters, restart Emby:
|
||||
# --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
|
||||
# Remove any standard path mapping for the transcode directory.
|
||||
|
||||
# Check 2 — transcoding-temp exists on ramdisk:
|
||||
ls /mnt/ramdisk_transcodes/
|
||||
# Expected: transcoding-temp/
|
||||
#
|
||||
# Fix if missing:
|
||||
mkdir -p /mnt/ramdisk_transcodes/transcoding-temp
|
||||
chown nobody:users /mnt/ramdisk_transcodes/transcoding-temp
|
||||
|
||||
# Check 3 — no duplicate SSD mount in Emby template:
|
||||
docker inspect Emby | grep -A3 "Mounts"
|
||||
# Should show only /mnt/ram-transcode → /ext-ram-transcode
|
||||
# Should NOT show /mnt/cache/Temp_Storage/... as a second mount
|
||||
```
|
||||
|
||||
### Flip Count High — 3+ Per Hour
|
||||
|
||||
```bash
|
||||
# Ramdisk filling up regularly — load exceeds the current ceiling.
|
||||
# Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB"
|
||||
#
|
||||
# If peak is close to RAMDISK_WARN_GB → increase ramdisk size:
|
||||
# master_host1.conf
|
||||
HOST1_RAMDISK_SIZE="12G" # increase by 2G
|
||||
HOST1_RAMDISK_WARN_GB=10.5 # adjust thresholds accordingly
|
||||
HOST1_RAMDISK_LOW_GB=8.5
|
||||
|
||||
# Remount at new size — run ramdisk_setup.sh manually:
|
||||
ramdisk_setup.sh --log
|
||||
# Ramdisk must be unmounted first if already mounted:
|
||||
# umount /mnt/ramdisk_transcodes && ramdisk_setup.sh --log
|
||||
```
|
||||
|
||||
### Ramdisk Not Mounting at Array Start
|
||||
|
||||
```bash
|
||||
# Check if tmpfs is mounted:
|
||||
mountpoint /mnt/ramdisk_transcodes
|
||||
# "not a mountpoint" → setup failed or not run yet
|
||||
|
||||
# Run manually to see the error:
|
||||
ramdisk_setup.sh --log
|
||||
|
||||
# Common causes:
|
||||
# /mnt/ramdisk_transcodes missing → mkdir -p /mnt/ramdisk_transcodes
|
||||
# Insufficient RAM → check free RAM: free -h
|
||||
# RAMDISK_SIZE too large → reduce HOST*_RAMDISK_SIZE
|
||||
```
|
||||
|
||||
### Emergency Manual Flip
|
||||
|
||||
```bash
|
||||
# Flip to SSD immediately — all new sessions go to SSD:
|
||||
ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode
|
||||
|
||||
# Flip back to ramdisk — all new sessions go to ramdisk:
|
||||
ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode
|
||||
|
||||
# Check current symlink target:
|
||||
readlink /mnt/ram-transcode
|
||||
|
||||
# Existing sessions in progress are NEVER affected — only new sessions follow the flip.
|
||||
```
|
||||
|
||||
### Increasing Ramdisk Size After Initial Setup
|
||||
|
||||
```bash
|
||||
# 1. Set new size and thresholds in master_host*.conf
|
||||
# 2. Unmount the existing ramdisk (no sessions should be active):
|
||||
umount /mnt/ramdisk_transcodes
|
||||
|
||||
# 3. Re-run setup to mount at new size:
|
||||
ramdisk_setup.sh --log
|
||||
|
||||
# 4. Verify:
|
||||
df -h /mnt/ramdisk_transcodes
|
||||
# Should show new size as total
|
||||
```
|
||||
Reference in New Issue
Block a user