unRAID Script Ecosystem
A modular, git-managed automation ecosystem for two unRAID servers. One configuration file. Both servers stay in sync with a single git pull. Everything from daily media syncing to mutual container failover runs automatically — and when something goes wrong, the system tries to fix itself before waking you up.
The Goal
A self-hosted infrastructure that runs itself.
Step away → come back to a healthy system
Something breaks → system self-heals
Something can't be fixed → you get notified
Servers stay in sync → one edit propagates everywhere
Before this ecosystem existed, the same problems were solved by 60+ standalone scripts across two servers — different coding styles, no shared standards, every change applied twice. This ecosystem standardises everything into one codebase with one config file and one deployment mechanism.
The Servers
HOST1 — unRAID-Gmer4Lfe (Primary)
Hardware: Threadripper 1950X, 128GB RAM
Storage: Multiple ZFS pools + cache
Role: Primary services, full media stack, all Docker containers
HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
Hardware: Intel i5 10th gen — completely different hardware
Storage: Different disk count, different pool layout
Location: Remote — 50 miles from HOST1
Role: Mirror, failover coverage, independent services
Network: Tailscale — encrypted tunnel between both servers
Repo: Self-hosted Gitea on HOST1
Deployment: git pull on either server → both stay current
The servers do not need to be identical hardware. This is one of the most important design decisions in the ecosystem. Because everything is accessed through /mnt/user/ — unRAID's fused share layer — the underlying disk count, pool layout, and hardware generation don't matter. A share called Movies on HOST1 is /mnt/user/Movies. The same share name on HOST2 is also /mnt/user/Movies. rsync syncs between them. Failover containers on HOST2 mount /mnt/user/Movies and see the same data. The hardware underneath is completely irrelevant.
HOST1: Threadripper, 10 drives across 5 ZFS pools
HOST2: i5, 4 drives, completely different layout
Both: /mnt/user/Movies — identical path, same data, different everything else
What is actually mirrored:
Share names — /mnt/user/Movies is /mnt/user/Movies on both servers. rsync syncs the content nightly. Container volume mounts use the same path. No path translation anywhere.
Media shares — all media is mirrored to HOST2. Since both servers mount the same share names, every container on HOST2 — Emby, the arr stack, everything — has access to the same library HOST1 serves. HOST1 is the source of truth. HOST2 is the mirror. They share the same data pool when failover kicks in — no separate libraries, no separate metadata, no separate databases for media.
HOST1 /mnt/user/Movies → rsync nightly → HOST2 /mnt/user/Movies
HOST1 /mnt/user/Tv_Shows → rsync nightly → HOST2 /mnt/user/Tv_Shows
HOST1 /mnt/user/Music → rsync nightly → HOST2 /mnt/user/Music
...all media shares mirrored...
Normal operation — HOST2 piggybacks on HOST1:
In day-to-day operation HOST2 is essentially passive from HOST1's perspective. HOST1 handles everything — all arrs, all downloads, all transcoding, all active services. HOST2 keeps its mirror current and waits.
Normal operation:
HOST1 — source of truth, all active services running
HOST2 — mirror current, containers ready but stopped
no arrs running (would create conflicts with HOST1)
no downloaders running
just keeping data fresh and ready to take over
Don't run arrs on both servers simultaneously. The arr stack manages your library — two instances writing to the same share causes conflicts, duplicate downloads, and database corruption. Either designate one server as the arr master (HOST1 in this setup) or split arrs between servers by type. HOST2's arrs only start during Tier 4 failover — when HOST1 has been down 18+ hours and workflow continuity is genuinely needed.
Container names — failover containers on HOST2 must use the same Docker container name as on HOST1. The failover script starts them by name via SSH. Server-specific containers get server-specific names:
Shared containers — same name on both servers:
Emby ← starts on HOST2 during HOST1 outage
uses HOST2's mirrored media, same library
NginxProxyManager ← same name, same config, same network
Authelia ← same name
Server-specific containers — unique names:
Emby-Gmer4Lfe ← HOST1's own Emby instance, always on HOST1
Emby-Jayred365 ← HOST2's own Emby instance, always on HOST2
VaultWarden-Gmer4Lfe ← HOST1's password manager
VaultWarden-Jayred365← HOST2's password manager
Docker custom networks — containers communicate by container name within a custom network, not by IP address. When a container restarts and gets a new IP, NginxProxyManager still reaches it by name. Failover containers on HOST2 join the same named custom networks and are immediately reachable by NPM without any IP reconfiguration required.
Custom network: high-availability
NginxProxyManager → Authelia (by name — IP changes are invisible)
NginxProxyManager → Emby (by name)
NginxProxyManager → NextCloud (by name)
Personal shares — backup without failover:
Beyond the shared media library, each user can sync personal shares to the other server purely for offsite backup — no failover container involvement, just data protection.
HOST1: /mnt/user/Gmer4Lfe-Personal → rsync nightly → HOST2 (encrypted backup)
HOST2: /mnt/user/Jayred365-Personal → rsync nightly → HOST1 (encrypted backup)
Configure in Master.conf:
HOST1_PERSONAL_SHARES=(
"/mnt/user/Gmer4Lfe-Personal"
)
HOST2_PERSONAL_SHARES=(
"/mnt/user/Jayred365-Personal"
)
daily_sync.sh automatically picks up the personal shares for the local host and syncs them alongside the media shares.
Encrypting personal shares — unRAID 7 ZFS:
ZFS native encryption in unRAID 7 means both admins can see the share exists and file sizes but neither can read content without your passphrase or keyfile. rsync copies encrypted blocks as-is — the remote server never needs your key.
Setup on HOST1 (your personal share):
Step 1 — Create an encrypted ZFS dataset:
unRAID UI → Main tab → click your ZFS pool
→ Click "+ Dataset"
→ Name: Gmer4Lfe-Personal
→ Enable Encryption: Yes
→ Encryption type: passphrase (simplest) or keyfile (auto-unlock capable)
→ Enter your passphrase — write it down, if lost data is unrecoverable
→ Create
Step 2 — Create the share:
Settings → Shares → Add Share
→ Name: Gmer4Lfe-Personal
→ Primary storage: your ZFS pool
→ Use cache: Only (keeps data on ZFS pool, not array)
→ Add
Step 3 — Verify encryption is active:
zfs get encryption poolname/Gmer4Lfe-Personal
# Should show: encryption aes-256-gcm
Step 4 — Add to Master.conf and sync:
HOST1_PERSONAL_SHARES=(
"/mnt/user/Gmer4Lfe-Personal"
)
Auto-unlock on boot (keyfile approach):
If you want the share to mount automatically after reboot without entering a passphrase:
# Create keyfile — on HOST1 only, never sync this file
mkdir -p /root/.zfs-keys
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
chmod 400 /root/.zfs-keys/personal.key
# Set dataset to use keyfile
zfs change-key -o keylocation=file:///root/.zfs-keys/personal.key \
-o keyformat=raw poolname/Gmer4Lfe-Personal
# Add to array start script (unRAID_Essentials or User Scripts)
zfs load-key poolname/Gmer4Lfe-Personal
zfs mount poolname/Gmer4Lfe-Personal
Manual unlock (most secure — you control when it's readable):
zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal
What the remote admin sees:
The share directory exists on HOST2. File names and sizes are visible (ZFS encrypts content, not metadata by default). File contents are unreadable without your key. To hide filenames too, enable zfs set encryption=aes-256-gcm with dnodesize=auto — this is a more advanced setup.
Current status: The infrastructure supports encrypted personal share syncing. The ZFS dataset setup is a one-time manual step per server. Once set up it syncs automatically like any other share.
How It Works
Every script sources two files at startup:
source "$SCRIPT_DIR/../Master.conf" # all user configuration
source "$SCRIPT_DIR/../common.sh" # shared library
Master.conf is the single source of truth. Container names, thresholds, paths, API keys, rsync profiles, failover tiers — everything configurable lives here. Change a value, push to git, both servers pull — done.
common.sh provides shared functions used by every script — host detection, notifications, rsync helpers, output formatting, icon set. Scripts never duplicate this logic.
git_pull_execute.sh pulls the latest scripts from Gitea and sets executable permissions. Schedule it or run it manually on either server.
Repository Structure
Unraid_Scripts/
├── Master.conf # All user configuration — edit here only
├── common.sh # Shared library — used by all scripts
├── git_pull_execute.sh # Pull latest scripts from Gitea
├── User_Script_Template.sh # Paste into unRAID User Scripts plugin
│
├── Failover/ # Mutual container failover
│ ├── README-Failover.md
│ ├── failover.sh # State machine — runs continuously
│ └── failover_test.sh # Controlled simulation harness
│
├── Monitors/ # Health reporting — watch and report only
│ ├── README-Monitors.md
│ ├── cert_monitor.sh # SSL certificate expiry
│ ├── smart_health.sh # Drive SMART attributes
│ ├── zfs_memory_snapshot.sh # ZFS pool health + memory report
│ ├── backup_verify.sh # Random checksum verification vs remote
│ ├── bandwidth_monitor.sh # Rsync transfer history
│ ├── weekly_health_digest.sh # Aggregated system health summary
│ └── emby_session_report.sh # Emby usage statistics
│
├── Orchestrators/ # Sequential job runners
│ ├── README-Orchestrators.md
│ ├── daily_sync.sh # All media shares synced nightly
│ └── media_management.sh # Permissions + cleaners + arr cleanup
│
├── Rsync/ # Core sync engine
│ ├── README-Rsync.md
│ ├── rsync.sh # Per-share or per-profile sync
│ └── README_Rsync_Setup.md # Initial setup guide
│
├── Docker_Essentials/ # Container lifecycle management
│ ├── README-Docker_Essentials.md
│ ├── docker_watchdog.sh # Two-tier self-healing monitoring
│ ├── docker_daily_restart.sh # Daily container restarts
│ ├── docker_weekly_restart.sh # Weekly container restarts
│ └── docker_network_connect.sh # Extra network connections at boot
│
├── Media/ # Media library maintenance
│ ├── README-Media.md
│ ├── media_shares_permissions.sh # Recursive permission application
│ ├── media_cleaner.sh # Junk file removal (anime + media profiles)
│ ├── lidarr_cleanup.sh # Orphaned music file cleanup
│ ├── sonarr_cleanup.sh # Orphaned TV file cleanup
│ └── radarr_cleanup.sh # Orphaned movie file cleanup
│
├── Transcodes/ # Emby ramdisk transcode management
│ ├── README-Transcoding.md
│ ├── ramdisk_setup.sh # Create ramdisk + symlink at array start
│ ├── transcode_manager.sh # Monitor usage, manage symlink, display sessions
│ └── transcode_cleanup.sh # Remove old inactive transcode files
│
├── Tools/ # Situational utilities — run when needed
│ ├── README-Tools.md
│ ├── recreate_shares.sh # Recreate share dirs after incident
│ ├── failover_state_reset.sh # Reset failover state file to NORMAL
│ ├── watchdog_skip_list_manager.sh # Manage container watchdog skip lists
│ ├── bulk_permissions_repair.sh # Targeted permission repair for one share
│ ├── container_data_export.sh # Export container appdata to tar archive
│ ├── emby_database_repair.sh # SQLite integrity check on Emby databases
│ └── zfs_pool_scrub.sh # Trigger ZFS scrub with completion report
│
└── unRAID_Essentials/ # Server-level system management
├── README-unRAID_Essentials.md
├── system_watchdog.sh # Last line of defense — controlled reboot
├── webgui_restart.sh # WebGUI nginx + emhttp auto-restart
├── docker_syslog_filter.sh # Suppress Docker veth noise from syslog
├── php_fpm_max_children.sh # WebGUI PHP-FPM concurrency tuning
├── clear_logs.sh # Weekly system log clearing
├── mover_stop.sh # Graceful mover termination
├── rsync_stop.sh # Stop rsync + recover containers
├── server_reboot.sh # Graceful reboot with user warning
└── user_script_stop.sh # Stop all running User Scripts jobs
Folders at a Glance
| Folder | What it does | Key scripts |
|---|---|---|
| Failover | Mutual container failover — autonomous, tiered, DDNS-safe | failover.sh |
| Monitors | Watch and report — never act, minimal flash writes | weekly_health_digest.sh |
| Orchestrators | Sequential job runners with unified reporting | daily_sync.sh, media_management.sh |
| Rsync | Core sync engine with profile system | rsync.sh |
| Docker_Essentials | Two-tier self-healing container management | docker_watchdog.sh |
| Media | Permissions, junk cleanup, arr orphan cleanup | media_cleaner.sh, arr scripts |
| Transcodes | Ramdisk symlink routing for Emby | transcode_manager.sh |
| Tools | Recovery and situational utilities | failover_state_reset.sh, zfs_pool_scrub.sh |
| unRAID_Essentials | Server-level maintenance and last-resort recovery | system_watchdog.sh |
Key Design Principles
One config file. Master.conf is the only file you edit. No hunting through scripts to change a container name or a threshold.
Bidirectional. Both servers run identical scripts. detect_hosts() determines local vs remote at runtime. One codebase covers both directions.
Self-healing layers. Problems are addressed at the most targeted level first:
docker_watchdog.sh — container level, minimal disruption
system_watchdog.sh — system level, last resort
failover.sh — infrastructure level, other server covers
Strike systems, not hair triggers. Single spikes don't cause restarts or reboots. Sustained problems do. The strike system filters noise from genuine issues.
Notifications when action is needed, silence otherwise. The ecosystem is designed to run without daily attention. You hear from it when something needs human intervention — not as a regular occurrence.
Flash drive friendly. Scripts that write to /boot/ use bounded files with auto-purge. /tmp/ is used for ephemeral state that resets on reboot. Most scripts write nothing at all.
--dry-run everywhere. Every script supports --dry-run. Test before you schedule.
Scheduling Overview
# ━━━ At Startup of Array ━━━
failover.sh # background task — continuous loop
ramdisk_setup.sh
docker_syslog_filter.sh
php_fpm_max_children.sh
docker_network_connect.sh
# ━━━ Frequent ━━━
*/3 * * * * transcode_manager.sh
*/5 * * * * transcode_cleanup.sh
*/10 * * * * webgui_restart.sh
*/15 * * * * docker_watchdog.sh
*/15 * * * * system_watchdog.sh
# ━━━ Daily ━━━
0 1 * * * daily_sync.sh
0 2 * * * media_management.sh
0 3 * * * docker_daily_restart.sh
0 8 * * * weekly_health_digest.sh
# ━━━ Weekly — Sunday ━━━
0 3 * * 0 docker_weekly_restart.sh
0 5 * * 0 clear_logs.sh
0 6 * * 0 zfs_memory_snapshot.sh
0 7 * * 0 smart_health.sh
0 9 * * 0 cert_monitor.sh
0 10 * * 0 backup_verify.sh
0 11 * * 0 emby_session_report.sh
0 11 * * 0 bandwidth_monitor.sh --report
Folder READMEs
Each folder has a detailed README covering setup, configuration, usage, and the reasoning behind design decisions.
| README | Contents |
|---|---|
| README-Failover.md | DDNS split brain prevention, tiered failover, handback sequence, initial setup, troubleshooting |
| README-Monitors.md | All monitor scripts, flash drive write policy, scheduling |
| README-Orchestrators.md | Why orchestrators exist, job ordering, how to add jobs |
| README-Rsync.md | Profile system, SSH key setup, Tailscale requirements |
| README-Docker_Essentials.md | Two-tier watchdog, startup grace, dependency ordering, skip list management |
| README-Media.md | Execution order, cleaner profiles, arr cleanup safety procedure |
| README-Transcoding.md | Symlink indirection design, Docker mount warning, mode switching, sizing |
| README-Tools.md | All utility scripts, when to use each, how to add new tools |
| README-unRAID_Essentials.md | system_watchdog tiers, startup sequence, scheduled maintenance |
v2 Roadmap
The v1 ecosystem is production-proven and stable. These are the features planned for v2 — some require additional infrastructure, some are extensions of existing systems.
Transcode Manager — Advanced Mode
The current smart / ramdisk / ssd modes route all sessions to one location. v2 adds an advanced mode that routes by media type:
TRANSCODE_MANAGER_MODE="advanced"
TRANSCODE_FORCE_RAMDISK=(
"LiveTv" # always ramdisk — buffering is latency sensitive
)
TRANSCODE_FORCE_SSD=(
"Audio" # music/downloads — no benefit from ramdisk
)
# Everything else follows smart threshold behavior
The groundwork is already in place — session display with media type parsing is working. Advanced mode requires Emby to expose media type at the point ffmpeg resolves the symlink, which means the routing decision needs to happen before the session starts. This is the problem to solve.
Plugin Dashboard
The Monitors folder is already the data backend for a future unRAID plugin dashboard. Every monitor script writes structured state that a plugin could read and display:
Failover state → live status indicator
Container watchdog → strike counts, skip list
SMART health → per-drive status
ZFS pools → health + ARC utilization
Transcode sessions → live session display
Bandwidth history → transfer trend charts
Certificate expiry → days remaining per domain
The scripts exist. The data exists. The plugin is the frontend.
Failover — Tiered by Content Type
Currently failover starts containers based on time elapsed. A future enhancement would start containers based on what the primary server was doing before it went down:
Primary running Live TV sessions → start Live TV stack on secondary immediately
Primary idle → standard tier delays apply
Primary in heavy transcoding → start Emby immediately, defer others
This requires the failover state file to track active session types, which requires integration with the Emby API at failover trigger time.
Health Digest — Plugin Integration
Currently the digest sends a notification. In v2 it populates a persistent dashboard that shows a rolling week of system health at a glance — without requiring a notification for every event.
Container Health Checks — Standardised Library
The docker_watchdog already has HTTP check support per container. A v2 enhancement is a standardised health check library — pre-built check commands for every container in the stack that can be applied via Extra Parameters in the unRAID template without manual research per container.
Origin
This ecosystem grew from a single failover script — written as a first real bash project, refined through months of production use, redesigned multiple times as the stack grew more complex. Each script in the collection started as a one-off solution to a specific problem. Over time the patterns that worked were extracted into common.sh, the configuration was centralised into Master.conf, and the whole collection was standardised into what it is now.
The failover script that started it all is still the soul of the ecosystem — the DDNS sequencing, the two-ping state machine, the handback order. Everything else built on top of that foundation.
Two servers, one codebase, self-healing infrastructure. Step away. Come back to a happy system.