450 lines
22 KiB
Markdown
450 lines
22 KiB
Markdown
# unRAID Script Ecosystem
|
|
|
|
A modular, git-managed automation ecosystem for two unRAID servers. One configuration file. Both servers stay in sync with a single git pull. Everything from daily media syncing to mutual container failover runs automatically — and when something goes wrong, the system tries to fix itself before waking you up.
|
|
|
|
---
|
|
|
|
## The Goal
|
|
|
|
A self-hosted infrastructure that runs itself.
|
|
|
|
```
|
|
Step away → come back to a healthy system
|
|
Something breaks → system self-heals
|
|
Something can't be fixed → you get notified
|
|
Servers stay in sync → one edit propagates everywhere
|
|
```
|
|
|
|
Before this ecosystem existed, the same problems were solved by 60+ standalone scripts across two servers — different coding styles, no shared standards, every change applied twice. This ecosystem standardises everything into one codebase with one config file and one deployment mechanism.
|
|
|
|
---
|
|
|
|
## The Servers
|
|
|
|
```
|
|
HOST1 — unRAID-Gmer4Lfe (Primary)
|
|
Hardware: Threadripper 1950X, 128GB RAM
|
|
Storage: Multiple ZFS pools + cache
|
|
Role: Primary services, full media stack, all Docker containers
|
|
|
|
HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
|
|
Hardware: Intel i5 10th gen — completely different hardware
|
|
Storage: Different disk count, different pool layout
|
|
Location: Remote — 50 miles from HOST1
|
|
Role: Mirror, failover coverage, independent services
|
|
|
|
Network: Tailscale — encrypted tunnel between both servers
|
|
Repo: Self-hosted Gitea on HOST1
|
|
Deployment: git pull on either server → both stay current
|
|
```
|
|
|
|
**The servers do not need to be identical hardware.** This is one of the most important design decisions in the ecosystem. Because everything is accessed through `/mnt/user/` — unRAID's fused share layer — the underlying disk count, pool layout, and hardware generation don't matter. A share called `Movies` on HOST1 is `/mnt/user/Movies`. The same share name on HOST2 is also `/mnt/user/Movies`. rsync syncs between them. Failover containers on HOST2 mount `/mnt/user/Movies` and see the same data. The hardware underneath is completely irrelevant.
|
|
|
|
```
|
|
HOST1: Threadripper, 10 drives across 5 ZFS pools
|
|
HOST2: i5, 4 drives, completely different layout
|
|
Both: /mnt/user/Movies — identical path, same data, different everything else
|
|
```
|
|
|
|
**What is actually mirrored:**
|
|
|
|
**Share names** — `/mnt/user/Movies` is `/mnt/user/Movies` on both servers. rsync syncs the content nightly. Container volume mounts use the same path. No path translation anywhere.
|
|
|
|
**Media shares** — all media is mirrored to HOST2. Since both servers mount the same share names, every container on HOST2 — Emby, the arr stack, everything — has access to the same library HOST1 serves. HOST1 is the source of truth. HOST2 is the mirror. They share the same data pool when failover kicks in — no separate libraries, no separate metadata, no separate databases for media.
|
|
|
|
```
|
|
HOST1 /mnt/user/Movies → rsync nightly → HOST2 /mnt/user/Movies
|
|
HOST1 /mnt/user/Tv_Shows → rsync nightly → HOST2 /mnt/user/Tv_Shows
|
|
HOST1 /mnt/user/Music → rsync nightly → HOST2 /mnt/user/Music
|
|
...all media shares mirrored...
|
|
```
|
|
|
|
**Normal operation — HOST2 piggybacks on HOST1:**
|
|
|
|
In day-to-day operation HOST2 is essentially passive from HOST1's perspective. HOST1 handles everything — all arrs, all downloads, all transcoding, all active services. HOST2 keeps its mirror current and waits.
|
|
|
|
```
|
|
Normal operation:
|
|
HOST1 — source of truth, all active services running
|
|
HOST2 — mirror current, containers ready but stopped
|
|
no arrs running (would create conflicts with HOST1)
|
|
no downloaders running
|
|
just keeping data fresh and ready to take over
|
|
```
|
|
|
|
**Don't run arrs on both servers simultaneously.** The arr stack manages your library — two instances writing to the same share causes conflicts, duplicate downloads, and database corruption. Either designate one server as the arr master (HOST1 in this setup) or split arrs between servers by type. HOST2's arrs only start during Tier 4 failover — when HOST1 has been down 18+ hours and workflow continuity is genuinely needed.
|
|
|
|
**Container names** — failover containers on HOST2 must use the same Docker container name as on HOST1. The failover script starts them by name via SSH. Server-specific containers get server-specific names:
|
|
|
|
```
|
|
Shared containers — same name on both servers:
|
|
Emby ← starts on HOST2 during HOST1 outage
|
|
uses HOST2's mirrored media, same library
|
|
NginxProxyManager ← same name, same config, same network
|
|
Authelia ← same name
|
|
|
|
Server-specific containers — unique names:
|
|
Emby-Gmer4Lfe ← HOST1's own Emby instance, always on HOST1
|
|
Emby-Jayred365 ← HOST2's own Emby instance, always on HOST2
|
|
VaultWarden-Gmer4Lfe ← HOST1's password manager
|
|
VaultWarden-Jayred365← HOST2's password manager
|
|
```
|
|
|
|
**Docker custom networks** — containers communicate by container name within a custom network, not by IP address. When a container restarts and gets a new IP, NginxProxyManager still reaches it by name. Failover containers on HOST2 join the same named custom networks and are immediately reachable by NPM without any IP reconfiguration required.
|
|
|
|
```
|
|
Custom network: high-availability
|
|
NginxProxyManager → Authelia (by name — IP changes are invisible)
|
|
NginxProxyManager → Emby (by name)
|
|
NginxProxyManager → NextCloud (by name)
|
|
```
|
|
|
|
**Personal shares — backup without failover:**
|
|
|
|
Beyond the shared media library, each user can sync personal shares to the other server purely for offsite backup — no failover container involvement, just data protection.
|
|
|
|
```
|
|
HOST1: /mnt/user/Gmer4Lfe-Personal → rsync nightly → HOST2 (encrypted backup)
|
|
HOST2: /mnt/user/Jayred365-Personal → rsync nightly → HOST1 (encrypted backup)
|
|
```
|
|
|
|
Configure in `Master.conf`:
|
|
```bash
|
|
HOST1_PERSONAL_SHARES=(
|
|
"/mnt/user/Gmer4Lfe-Personal"
|
|
)
|
|
HOST2_PERSONAL_SHARES=(
|
|
"/mnt/user/Jayred365-Personal"
|
|
)
|
|
```
|
|
|
|
`daily_sync.sh` automatically picks up the personal shares for the local host and syncs them alongside the media shares.
|
|
|
|
**Encrypting personal shares — unRAID 7 ZFS:**
|
|
|
|
ZFS native encryption in unRAID 7 means both admins can see the share exists and file sizes but neither can read content without your passphrase or keyfile. rsync copies encrypted blocks as-is — the remote server never needs your key.
|
|
|
|
**Setup on HOST1 (your personal share):**
|
|
|
|
**Step 1 — Create an encrypted ZFS dataset:**
|
|
```
|
|
unRAID UI → Main tab → click your ZFS pool
|
|
→ Click "+ Dataset"
|
|
→ Name: Gmer4Lfe-Personal
|
|
→ Enable Encryption: Yes
|
|
→ Encryption type: passphrase (simplest) or keyfile (auto-unlock capable)
|
|
→ Enter your passphrase — write it down, if lost data is unrecoverable
|
|
→ Create
|
|
```
|
|
|
|
**Step 2 — Create the share:**
|
|
```
|
|
Settings → Shares → Add Share
|
|
→ Name: Gmer4Lfe-Personal
|
|
→ Primary storage: your ZFS pool
|
|
→ Use cache: Only (keeps data on ZFS pool, not array)
|
|
→ Add
|
|
```
|
|
|
|
**Step 3 — Verify encryption is active:**
|
|
```bash
|
|
zfs get encryption poolname/Gmer4Lfe-Personal
|
|
# Should show: encryption aes-256-gcm
|
|
```
|
|
|
|
**Step 4 — Add to Master.conf and sync:**
|
|
```bash
|
|
HOST1_PERSONAL_SHARES=(
|
|
"/mnt/user/Gmer4Lfe-Personal"
|
|
)
|
|
```
|
|
|
|
**Auto-unlock on boot (keyfile approach):**
|
|
|
|
If you want the share to mount automatically after reboot without entering a passphrase:
|
|
```bash
|
|
# Create keyfile — on HOST1 only, never sync this file
|
|
mkdir -p /root/.zfs-keys
|
|
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
|
|
chmod 400 /root/.zfs-keys/personal.key
|
|
|
|
# Set dataset to use keyfile
|
|
zfs change-key -o keylocation=file:///root/.zfs-keys/personal.key \
|
|
-o keyformat=raw poolname/Gmer4Lfe-Personal
|
|
|
|
# Add to array start script (unRAID_Essentials or User Scripts)
|
|
zfs load-key poolname/Gmer4Lfe-Personal
|
|
zfs mount poolname/Gmer4Lfe-Personal
|
|
```
|
|
|
|
**Manual unlock (most secure — you control when it's readable):**
|
|
```bash
|
|
zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase
|
|
zfs mount poolname/Gmer4Lfe-Personal
|
|
```
|
|
|
|
**What the remote admin sees:**
|
|
|
|
The share directory exists on HOST2. File names and sizes are visible (ZFS encrypts content, not metadata by default). File contents are unreadable without your key. To hide filenames too, enable `zfs set encryption=aes-256-gcm` with `dnodesize=auto` — this is a more advanced setup.
|
|
|
|
**Current status:** The infrastructure supports encrypted personal share syncing. The ZFS dataset setup is a one-time manual step per server. Once set up it syncs automatically like any other share.
|
|
|
|
---
|
|
|
|
## How It Works
|
|
|
|
Every script sources two files at startup:
|
|
|
|
```bash
|
|
source "$SCRIPT_DIR/../Master.conf" # all user configuration
|
|
source "$SCRIPT_DIR/../common.sh" # shared library
|
|
```
|
|
|
|
**`Master.conf`** is the single source of truth. Container names, thresholds, paths, API keys, rsync profiles, failover tiers — everything configurable lives here. Change a value, push to git, both servers pull — done.
|
|
|
|
**`common.sh`** provides shared functions used by every script — host detection, notifications, rsync helpers, output formatting, icon set. Scripts never duplicate this logic.
|
|
|
|
**`git_pull_execute.sh`** pulls the latest scripts from Gitea and sets executable permissions. Schedule it or run it manually on either server.
|
|
|
|
---
|
|
|
|
## Repository Structure
|
|
|
|
```
|
|
Unraid_Scripts/
|
|
├── Master.conf # All user configuration — edit here only
|
|
├── common.sh # Shared library — used by all scripts
|
|
├── git_pull_execute.sh # Pull latest scripts from Gitea
|
|
├── User_Script_Template.sh # Paste into unRAID User Scripts plugin
|
|
│
|
|
├── Failover/ # Mutual container failover
|
|
│ ├── README-Failover.md
|
|
│ ├── failover.sh # State machine — runs continuously
|
|
│ └── failover_test.sh # Controlled simulation harness
|
|
│
|
|
├── Monitors/ # Health reporting — watch and report only
|
|
│ ├── README-Monitors.md
|
|
│ ├── cert_monitor.sh # SSL certificate expiry
|
|
│ ├── smart_health.sh # Drive SMART attributes
|
|
│ ├── zfs_memory_snapshot.sh # ZFS pool health + memory report
|
|
│ ├── backup_verify.sh # Random checksum verification vs remote
|
|
│ ├── bandwidth_monitor.sh # Rsync transfer history
|
|
│ ├── weekly_health_digest.sh # Aggregated system health summary
|
|
│ └── emby_session_report.sh # Emby usage statistics
|
|
│
|
|
├── Orchestrators/ # Sequential job runners
|
|
│ ├── README-Orchestrators.md
|
|
│ ├── daily_sync.sh # All media shares synced nightly
|
|
│ └── media_management.sh # Permissions + cleaners + arr cleanup
|
|
│
|
|
├── Rsync/ # Core sync engine
|
|
│ ├── README-Rsync.md
|
|
│ ├── rsync.sh # Per-share or per-profile sync
|
|
│ └── README_Rsync_Setup.md # Initial setup guide
|
|
│
|
|
├── Docker_Essentials/ # Container lifecycle management
|
|
│ ├── README-Docker_Essentials.md
|
|
│ ├── docker_watchdog.sh # Two-tier self-healing monitoring
|
|
│ ├── docker_daily_restart.sh # Daily container restarts
|
|
│ ├── docker_weekly_restart.sh # Weekly container restarts
|
|
│ └── docker_network_connect.sh # Extra network connections at boot
|
|
│
|
|
├── Media/ # Media library maintenance
|
|
│ ├── README-Media.md
|
|
│ ├── media_shares_permissions.sh # Recursive permission application
|
|
│ ├── media_cleaner.sh # Junk file removal (anime + media profiles)
|
|
│ ├── lidarr_cleanup.sh # Orphaned music file cleanup
|
|
│ ├── sonarr_cleanup.sh # Orphaned TV file cleanup
|
|
│ └── radarr_cleanup.sh # Orphaned movie file cleanup
|
|
│
|
|
├── Transcodes/ # Emby ramdisk transcode management
|
|
│ ├── README-Transcoding.md
|
|
│ ├── ramdisk_setup.sh # Create ramdisk + symlink at array start
|
|
│ ├── transcode_manager.sh # Monitor usage, manage symlink, display sessions
|
|
│ └── transcode_cleanup.sh # Remove old inactive transcode files
|
|
│
|
|
├── Tools/ # Situational utilities — run when needed
|
|
│ ├── README-Tools.md
|
|
│ ├── recreate_shares.sh # Recreate share dirs after incident
|
|
│ ├── failover_state_reset.sh # Reset failover state file to NORMAL
|
|
│ ├── watchdog_skip_list_manager.sh # Manage container watchdog skip lists
|
|
│ ├── bulk_permissions_repair.sh # Targeted permission repair for one share
|
|
│ ├── container_data_export.sh # Export container appdata to tar archive
|
|
│ ├── emby_database_repair.sh # SQLite integrity check on Emby databases
|
|
│ └── zfs_pool_scrub.sh # Trigger ZFS scrub with completion report
|
|
│
|
|
└── unRAID_Essentials/ # Server-level system management
|
|
├── README-unRAID_Essentials.md
|
|
├── system_watchdog.sh # Last line of defense — controlled reboot
|
|
├── webgui_restart.sh # WebGUI nginx + emhttp auto-restart
|
|
├── docker_syslog_filter.sh # Suppress Docker veth noise from syslog
|
|
├── php_fpm_max_children.sh # WebGUI PHP-FPM concurrency tuning
|
|
├── clear_logs.sh # Weekly system log clearing
|
|
├── mover_stop.sh # Graceful mover termination
|
|
├── rsync_stop.sh # Stop rsync + recover containers
|
|
├── server_reboot.sh # Graceful reboot with user warning
|
|
└── user_script_stop.sh # Stop all running User Scripts jobs
|
|
```
|
|
|
|
---
|
|
|
|
## Folders at a Glance
|
|
|
|
| Folder | What it does | Key scripts |
|
|
|--------|-------------|-------------|
|
|
| **Failover** | Mutual container failover — autonomous, tiered, DDNS-safe | `failover.sh` |
|
|
| **Monitors** | Watch and report — never act, minimal flash writes | `weekly_health_digest.sh` |
|
|
| **Orchestrators** | Sequential job runners with unified reporting | `daily_sync.sh`, `media_management.sh` |
|
|
| **Rsync** | Core sync engine with profile system | `rsync.sh` |
|
|
| **Docker_Essentials** | Two-tier self-healing container management | `docker_watchdog.sh` |
|
|
| **Media** | Permissions, junk cleanup, arr orphan cleanup | `media_cleaner.sh`, arr scripts |
|
|
| **Transcodes** | Ramdisk symlink routing for Emby | `transcode_manager.sh` |
|
|
| **Tools** | Recovery and situational utilities | `failover_state_reset.sh`, `zfs_pool_scrub.sh` |
|
|
| **unRAID_Essentials** | Server-level maintenance and last-resort recovery | `system_watchdog.sh` |
|
|
|
|
---
|
|
|
|
## Key Design Principles
|
|
|
|
**One config file.** `Master.conf` is the only file you edit. No hunting through scripts to change a container name or a threshold.
|
|
|
|
**Bidirectional.** Both servers run identical scripts. `detect_hosts()` determines local vs remote at runtime. One codebase covers both directions.
|
|
|
|
**Self-healing layers.** Problems are addressed at the most targeted level first:
|
|
```
|
|
docker_watchdog.sh — container level, minimal disruption
|
|
system_watchdog.sh — system level, last resort
|
|
failover.sh — infrastructure level, other server covers
|
|
```
|
|
|
|
**Strike systems, not hair triggers.** Single spikes don't cause restarts or reboots. Sustained problems do. The strike system filters noise from genuine issues.
|
|
|
|
**Notifications when action is needed, silence otherwise.** The ecosystem is designed to run without daily attention. You hear from it when something needs human intervention — not as a regular occurrence.
|
|
|
|
**Flash drive friendly.** Scripts that write to `/boot/` use bounded files with auto-purge. `/tmp/` is used for ephemeral state that resets on reboot. Most scripts write nothing at all.
|
|
|
|
**--dry-run everywhere.** Every script supports `--dry-run`. Test before you schedule.
|
|
|
|
---
|
|
|
|
## Scheduling Overview
|
|
|
|
```bash
|
|
# ━━━ At Startup of Array ━━━
|
|
failover.sh # background task — continuous loop
|
|
ramdisk_setup.sh
|
|
docker_syslog_filter.sh
|
|
php_fpm_max_children.sh
|
|
docker_network_connect.sh
|
|
|
|
# ━━━ Frequent ━━━
|
|
*/3 * * * * transcode_manager.sh
|
|
*/5 * * * * transcode_cleanup.sh
|
|
*/10 * * * * webgui_restart.sh
|
|
*/15 * * * * docker_watchdog.sh
|
|
*/15 * * * * system_watchdog.sh
|
|
|
|
# ━━━ Daily ━━━
|
|
0 1 * * * daily_sync.sh
|
|
0 2 * * * media_management.sh
|
|
0 3 * * * docker_daily_restart.sh
|
|
0 8 * * * weekly_health_digest.sh
|
|
|
|
# ━━━ Weekly — Sunday ━━━
|
|
0 3 * * 0 docker_weekly_restart.sh
|
|
0 5 * * 0 clear_logs.sh
|
|
0 6 * * 0 zfs_memory_snapshot.sh
|
|
0 7 * * 0 smart_health.sh
|
|
0 9 * * 0 cert_monitor.sh
|
|
0 10 * * 0 backup_verify.sh
|
|
0 11 * * 0 emby_session_report.sh
|
|
0 11 * * 0 bandwidth_monitor.sh --report
|
|
```
|
|
|
|
---
|
|
|
|
## Folder READMEs
|
|
|
|
Each folder has a detailed README covering setup, configuration, usage, and the reasoning behind design decisions.
|
|
|
|
| README | Contents |
|
|
|--------|----------|
|
|
| [README-Failover.md](Failover/README-Failover.md) | DDNS split brain prevention, tiered failover, handback sequence, initial setup, troubleshooting |
|
|
| [README-Monitors.md](Monitors/README-Monitors.md) | All monitor scripts, flash drive write policy, scheduling |
|
|
| [README-Orchestrators.md](Orchestrators/README-Orchestrators.md) | Why orchestrators exist, job ordering, how to add jobs |
|
|
| [README-Rsync.md](Rsync/README-Rsync.md) | Profile system, SSH key setup, Tailscale requirements |
|
|
| [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md) | Two-tier watchdog, startup grace, dependency ordering, skip list management |
|
|
| [README-Media.md](Media/README-Media.md) | Execution order, cleaner profiles, arr cleanup safety procedure |
|
|
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Symlink indirection design, Docker mount warning, mode switching, sizing |
|
|
| [README-Tools.md](Tools/README-Tools.md) | All utility scripts, when to use each, how to add new tools |
|
|
| [README-unRAID_Essentials.md](unRAID_Essentials/README-unRAID_Essentials.md) | system_watchdog tiers, startup sequence, scheduled maintenance |
|
|
|
|
---
|
|
|
|
## v2 Roadmap
|
|
|
|
The v1 ecosystem is production-proven and stable. These are the features planned for v2 — some require additional infrastructure, some are extensions of existing systems.
|
|
|
|
### Transcode Manager — Advanced Mode
|
|
|
|
The current `smart` / `ramdisk` / `ssd` modes route all sessions to one location. v2 adds an `advanced` mode that routes by media type:
|
|
|
|
```bash
|
|
TRANSCODE_MANAGER_MODE="advanced"
|
|
|
|
TRANSCODE_FORCE_RAMDISK=(
|
|
"LiveTv" # always ramdisk — buffering is latency sensitive
|
|
)
|
|
TRANSCODE_FORCE_SSD=(
|
|
"Audio" # music/downloads — no benefit from ramdisk
|
|
)
|
|
# Everything else follows smart threshold behavior
|
|
```
|
|
|
|
The groundwork is already in place — session display with media type parsing is working. Advanced mode requires Emby to expose media type at the point ffmpeg resolves the symlink, which means the routing decision needs to happen before the session starts. This is the problem to solve.
|
|
|
|
### Plugin Dashboard
|
|
|
|
The Monitors folder is already the data backend for a future unRAID plugin dashboard. Every monitor script writes structured state that a plugin could read and display:
|
|
|
|
```
|
|
Failover state → live status indicator
|
|
Container watchdog → strike counts, skip list
|
|
SMART health → per-drive status
|
|
ZFS pools → health + ARC utilization
|
|
Transcode sessions → live session display
|
|
Bandwidth history → transfer trend charts
|
|
Certificate expiry → days remaining per domain
|
|
```
|
|
|
|
The scripts exist. The data exists. The plugin is the frontend.
|
|
|
|
### Failover — Tiered by Content Type
|
|
|
|
Currently failover starts containers based on time elapsed. A future enhancement would start containers based on what the primary server was doing before it went down:
|
|
|
|
```
|
|
Primary running Live TV sessions → start Live TV stack on secondary immediately
|
|
Primary idle → standard tier delays apply
|
|
Primary in heavy transcoding → start Emby immediately, defer others
|
|
```
|
|
|
|
This requires the failover state file to track active session types, which requires integration with the Emby API at failover trigger time.
|
|
|
|
### Health Digest — Plugin Integration
|
|
|
|
Currently the digest sends a notification. In v2 it populates a persistent dashboard that shows a rolling week of system health at a glance — without requiring a notification for every event.
|
|
|
|
### Container Health Checks — Standardised Library
|
|
|
|
The docker_watchdog already has HTTP check support per container. A v2 enhancement is a standardised health check library — pre-built check commands for every container in the stack that can be applied via Extra Parameters in the unRAID template without manual research per container.
|
|
|
|
---
|
|
|
|
## Origin
|
|
|
|
This ecosystem grew from a single failover script — written as a first real bash project, refined through months of production use, redesigned multiple times as the stack grew more complex. Each script in the collection started as a one-off solution to a specific problem. Over time the patterns that worked were extracted into `common.sh`, the configuration was centralised into `Master.conf`, and the whole collection was standardised into what it is now.
|
|
|
|
The failover script that started it all is still the soul of the ecosystem — the DDNS sequencing, the two-ping state machine, the handback order. Everything else built on top of that foundation.
|
|
|
|
Two servers, one codebase, self-healing infrastructure. Step away. Come back to a happy system. |