A modular, git-managed automation ecosystem for two unRAID servers. One configuration file. Both servers stay in sync with a single git pull. Everything from daily media syncing to mutual container failover runs automatically — and when something goes wrong, the system tries to fix itself before waking you up.

The Goal

A self-hosted infrastructure that runs itself.

Step away → come back to a healthy system
Something breaks → system self-heals
Something can't be fixed → you get notified
Servers stay in sync → one edit propagates everywhere

Before this ecosystem existed, the same problems were solved by 60+ standalone scripts across two servers — different coding styles, no shared standards, every change applied twice. This ecosystem standardises everything into one codebase with one config file and one deployment mechanism.

The Servers

HOST1 — unRAID-Gmer4Lfe (Primary)
  Hardware:   Threadripper 1950X, 128GB RAM
  Storage:    Multiple ZFS pools + cache
  Role:       Primary services, full media stack, all Docker containers

HOST2 — unRAID-Jayred365 (Secondary / Buddy server)
  Hardware:   Intel i5 10th gen — completely different hardware
  Storage:    Different disk count, different pool layout
  Location:   Remote — 50 miles from HOST1
  Role:       Mirror, failover coverage, independent services

Network:    Tailscale — encrypted tunnel between both servers
Repo:       Self-hosted Gitea on HOST1
Deployment: git pull on either server → both stay current

The servers do not need to be identical hardware. This is one of the most important design decisions in the ecosystem. Because everything is accessed through /mnt/user/ — unRAID's fused share layer — the underlying disk count, pool layout, and hardware generation don't matter. A share called Movies on HOST1 is /mnt/user/Movies. The same share name on HOST2 is also /mnt/user/Movies. rsync syncs between them. Failover containers on HOST2 mount /mnt/user/Movies and see the same data. The hardware underneath is completely irrelevant.

HOST1: Threadripper, 10 drives across 5 ZFS pools
HOST2: i5, 4 drives, completely different layout
Both:  /mnt/user/Movies — identical path, same data, different everything else

What is actually mirrored:

Share names — /mnt/user/Movies is /mnt/user/Movies on both servers. rsync syncs the content nightly. Container volume mounts use the same path. No path translation anywhere.

Media shares — all media is mirrored to HOST2. Since both servers mount the same share names, every container on HOST2 — Emby, the arr stack, everything — has access to the same library HOST1 serves. HOST1 is the source of truth. HOST2 is the mirror. They share the same data pool when failover kicks in — no separate libraries, no separate metadata, no separate databases for media.

HOST1 /mnt/user/Movies  →  rsync nightly  →  HOST2 /mnt/user/Movies
HOST1 /mnt/user/Tv_Shows →  rsync nightly  →  HOST2 /mnt/user/Tv_Shows
HOST1 /mnt/user/Music    →  rsync nightly  →  HOST2 /mnt/user/Music
...all media shares mirrored...

Normal operation — HOST2 piggybacks on HOST1:

In day-to-day operation HOST2 is essentially passive from HOST1's perspective. HOST1 handles everything — all arrs, all downloads, all transcoding, all active services. HOST2 keeps its mirror current and waits.

Normal operation:
  HOST1 — source of truth, all active services running
  HOST2 — mirror current, containers ready but stopped
          no arrs running (would create conflicts with HOST1)
          no downloaders running
          just keeping data fresh and ready to take over

Don't run arrs on both servers simultaneously. The arr stack manages your library — two instances writing to the same share causes conflicts, duplicate downloads, and database corruption. Either designate one server as the arr master (HOST1 in this setup) or split arrs between servers by type. HOST2's arrs only start during Tier 4 failover — when HOST1 has been down 18+ hours and workflow continuity is genuinely needed.

Container names — failover containers on HOST2 must use the same Docker container name as on HOST1. The failover script starts them by name via SSH. Server-specific containers get server-specific names:

Shared containers — same name on both servers:
  Emby                 ← starts on HOST2 during HOST1 outage
                         uses HOST2's mirrored media, same library
  NginxProxyManager    ← same name, same config, same network
  Authelia             ← same name

Server-specific containers — unique names:
  Emby-Gmer4Lfe        ← HOST1's own Emby instance, always on HOST1
  Emby-Jayred365       ← HOST2's own Emby instance, always on HOST2
  VaultWarden-Gmer4Lfe ← HOST1's password manager
  VaultWarden-Jayred365← HOST2's password manager

Docker custom networks — containers communicate by container name within a custom network, not by IP address. When a container restarts and gets a new IP, NginxProxyManager still reaches it by name. Failover containers on HOST2 join the same named custom networks and are immediately reachable by NPM without any IP reconfiguration required.

Custom network: high-availability
  NginxProxyManager → Authelia   (by name — IP changes are invisible)
  NginxProxyManager → Emby       (by name)
  NginxProxyManager → NextCloud  (by name)

This flexibility means any two unRAID servers can participate — regardless of hardware generation, CPU, drive count, or storage layout. The only requirements are matching share names, matching container names for shared services, and the same custom Docker network names.

How It Works

Every script sources two files at startup:

source "$SCRIPT_DIR/../Master.conf"   # all user configuration
source "$SCRIPT_DIR/../common.sh"     # shared library

Master.conf is the single source of truth. Container names, thresholds, paths, API keys, rsync profiles, failover tiers — everything configurable lives here. Change a value, push to git, both servers pull — done.

common.sh provides shared functions used by every script — host detection, notifications, rsync helpers, output formatting, icon set. Scripts never duplicate this logic.

git_pull_execute.sh pulls the latest scripts from Gitea and sets executable permissions. Schedule it or run it manually on either server.

Repository Structure

Unraid_Scripts/
├── Master.conf                        # All user configuration — edit here only
├── common.sh                          # Shared library — used by all scripts
├── git_pull_execute.sh                # Pull latest scripts from Gitea
├── User_Script_Template.sh            # Paste into unRAID User Scripts plugin
│
├── Failover/                          # Mutual container failover
│   ├── README-Failover.md
│   ├── failover.sh                    # State machine — runs continuously
│   └── failover_test.sh               # Controlled simulation harness
│
├── Monitors/                          # Health reporting — watch and report only
│   ├── README-Monitors.md
│   ├── cert_monitor.sh                # SSL certificate expiry
│   ├── smart_health.sh                # Drive SMART attributes
│   ├── zfs_memory_snapshot.sh         # ZFS pool health + memory report
│   ├── backup_verify.sh               # Random checksum verification vs remote
│   ├── bandwidth_monitor.sh           # Rsync transfer history
│   ├── weekly_health_digest.sh        # Aggregated system health summary
│   └── emby_session_report.sh         # Emby usage statistics
│
├── Orchestrators/                     # Sequential job runners
│   ├── README-Orchestrators.md
│   ├── daily_sync.sh                  # All media shares synced nightly
│   └── media_management.sh            # Permissions + cleaners + arr cleanup
│
├── Rsync/                             # Core sync engine
│   ├── README-Rsync.md
│   ├── rsync.sh                       # Per-share or per-profile sync
│   └── README_Rsync_Setup.md          # Initial setup guide
│
├── Docker_Essentials/                 # Container lifecycle management
│   ├── README-Docker_Essentials.md
│   ├── docker_watchdog.sh             # Two-tier self-healing monitoring
│   ├── docker_daily_restart.sh        # Daily container restarts
│   ├── docker_weekly_restart.sh       # Weekly container restarts
│   └── docker_network_connect.sh      # Extra network connections at boot
│
├── Media/                             # Media library maintenance
│   ├── README-Media.md
│   ├── media_shares_permissions.sh    # Recursive permission application
│   ├── media_cleaner.sh               # Junk file removal (anime + media profiles)
│   ├── lidarr_cleanup.sh              # Orphaned music file cleanup
│   ├── sonarr_cleanup.sh              # Orphaned TV file cleanup
│   └── radarr_cleanup.sh              # Orphaned movie file cleanup
│
├── Transcodes/                        # Emby ramdisk transcode management
│   ├── README-Transcoding.md
│   ├── ramdisk_setup.sh               # Create ramdisk + symlink at array start
│   ├── transcode_manager.sh           # Monitor usage, manage symlink, display sessions
│   └── transcode_cleanup.sh           # Remove old inactive transcode files
│
├── Tools/                             # Situational utilities — run when needed
│   ├── README-Tools.md
│   ├── recreate_shares.sh             # Recreate share dirs after incident
│   ├── failover_state_reset.sh        # Reset failover state file to NORMAL
│   ├── watchdog_skip_list_manager.sh  # Manage container watchdog skip lists
│   ├── bulk_permissions_repair.sh     # Targeted permission repair for one share
│   ├── container_data_export.sh       # Export container appdata to tar archive
│   ├── emby_database_repair.sh        # SQLite integrity check on Emby databases
│   └── zfs_pool_scrub.sh              # Trigger ZFS scrub with completion report
│
└── unRAID_Essentials/                 # Server-level system management
    ├── README-unRAID_Essentials.md
    ├── system_watchdog.sh             # Last line of defense — controlled reboot
    ├── webgui_restart.sh              # WebGUI nginx + emhttp auto-restart
    ├── docker_syslog_filter.sh        # Suppress Docker veth noise from syslog
    ├── php_fpm_max_children.sh        # WebGUI PHP-FPM concurrency tuning
    ├── clear_logs.sh                  # Weekly system log clearing
    ├── mover_stop.sh                  # Graceful mover termination
    ├── rsync_stop.sh                  # Stop rsync + recover containers
    ├── server_reboot.sh               # Graceful reboot with user warning
    └── user_script_stop.sh            # Stop all running User Scripts jobs

Folders at a Glance

Folder	What it does	Key scripts
Failover	Mutual container failover — autonomous, tiered, DDNS-safe	`failover.sh`
Monitors	Watch and report — never act, minimal flash writes	`weekly_health_digest.sh`
Orchestrators	Sequential job runners with unified reporting	`daily_sync.sh`, `media_management.sh`
Rsync	Core sync engine with profile system	`rsync.sh`
Docker_Essentials	Two-tier self-healing container management	`docker_watchdog.sh`
Media	Permissions, junk cleanup, arr orphan cleanup	`media_cleaner.sh`, arr scripts
Transcodes	Ramdisk symlink routing for Emby	`transcode_manager.sh`
Tools	Recovery and situational utilities	`failover_state_reset.sh`, `zfs_pool_scrub.sh`
unRAID_Essentials	Server-level maintenance and last-resort recovery	`system_watchdog.sh`

Key Design Principles

One config file. Master.conf is the only file you edit. No hunting through scripts to change a container name or a threshold.

Bidirectional. Both servers run identical scripts. detect_hosts() determines local vs remote at runtime. One codebase covers both directions.

Self-healing layers. Problems are addressed at the most targeted level first:

docker_watchdog.sh    — container level, minimal disruption
system_watchdog.sh    — system level, last resort
failover.sh           — infrastructure level, other server covers

Strike systems, not hair triggers. Single spikes don't cause restarts or reboots. Sustained problems do. The strike system filters noise from genuine issues.

Notifications when action is needed, silence otherwise. The ecosystem is designed to run without daily attention. You hear from it when something needs human intervention — not as a regular occurrence.

Flash drive friendly. Scripts that write to /boot/ use bounded files with auto-purge. /tmp/ is used for ephemeral state that resets on reboot. Most scripts write nothing at all.

--dry-run everywhere. Every script supports --dry-run. Test before you schedule.

Scheduling Overview

# ━━━ At Startup of Array ━━━
failover.sh                     # background task — continuous loop
ramdisk_setup.sh
docker_syslog_filter.sh
php_fpm_max_children.sh
docker_network_connect.sh

# ━━━ Frequent ━━━
*/3  * * * *    transcode_manager.sh
*/5  * * * *    transcode_cleanup.sh
*/10 * * * *    webgui_restart.sh
*/15 * * * *    docker_watchdog.sh
*/15 * * * *    system_watchdog.sh

# ━━━ Daily ━━━
0 1 * * *       daily_sync.sh
0 2 * * *       media_management.sh
0 3 * * *       docker_daily_restart.sh
0 8 * * *       weekly_health_digest.sh

# ━━━ Weekly — Sunday ━━━
0 3 * * 0       docker_weekly_restart.sh
0 5 * * 0       clear_logs.sh
0 6 * * 0       zfs_memory_snapshot.sh
0 7 * * 0       smart_health.sh
0 9 * * 0       cert_monitor.sh
0 10 * * 0      backup_verify.sh
0 11 * * 0      emby_session_report.sh
0 11 * * 0      bandwidth_monitor.sh --report

Folder READMEs

Each folder has a detailed README covering setup, configuration, usage, and the reasoning behind design decisions.

README	Contents
README-Failover.md	DDNS split brain prevention, tiered failover, handback sequence, initial setup, troubleshooting
README-Monitors.md	All monitor scripts, flash drive write policy, scheduling
README-Orchestrators.md	Why orchestrators exist, job ordering, how to add jobs
README-Rsync.md	Profile system, SSH key setup, Tailscale requirements
README-Docker_Essentials.md	Two-tier watchdog, startup grace, dependency ordering, skip list management
README-Media.md	Execution order, cleaner profiles, arr cleanup safety procedure
README-Transcoding.md	Symlink indirection design, Docker mount warning, mode switching, sizing
README-Tools.md	All utility scripts, when to use each, how to add new tools
README-unRAID_Essentials.md	system_watchdog tiers, startup sequence, scheduled maintenance

v2 Roadmap

The v1 ecosystem is production-proven and stable. These are the features planned for v2 — some require additional infrastructure, some are extensions of existing systems.

Transcode Manager — Advanced Mode

The current smart / ramdisk / ssd modes route all sessions to one location. v2 adds an advanced mode that routes by media type:

TRANSCODE_MANAGER_MODE="advanced"

TRANSCODE_FORCE_RAMDISK=(
    "LiveTv"    # always ramdisk — buffering is latency sensitive
)
TRANSCODE_FORCE_SSD=(
    "Audio"     # music/downloads — no benefit from ramdisk
)
# Everything else follows smart threshold behavior

The groundwork is already in place — session display with media type parsing is working. Advanced mode requires Emby to expose media type at the point ffmpeg resolves the symlink, which means the routing decision needs to happen before the session starts. This is the problem to solve.

Plugin Dashboard

The Monitors folder is already the data backend for a future unRAID plugin dashboard. Every monitor script writes structured state that a plugin could read and display:

Failover state          → live status indicator
Container watchdog      → strike counts, skip list
SMART health            → per-drive status
ZFS pools              → health + ARC utilization
Transcode sessions      → live session display
Bandwidth history       → transfer trend charts
Certificate expiry      → days remaining per domain

The scripts exist. The data exists. The plugin is the frontend.

Failover — Tiered by Content Type

Currently failover starts containers based on time elapsed. A future enhancement would start containers based on what the primary server was doing before it went down:

Primary running Live TV sessions → start Live TV stack on secondary immediately
Primary idle                     → standard tier delays apply
Primary in heavy transcoding     → start Emby immediately, defer others

This requires the failover state file to track active session types, which requires integration with the Emby API at failover trigger time.

Health Digest — Plugin Integration

Currently the digest sends a notification. In v2 it populates a persistent dashboard that shows a rolling week of system health at a glance — without requiring a notification for every event.

Container Health Checks — Standardised Library

The docker_watchdog already has HTTP check support per container. A v2 enhancement is a standardised health check library — pre-built check commands for every container in the stack that can be applied via Extra Parameters in the unRAID template without manual research per container.

Origin

This ecosystem grew from a single failover script — written as a first real bash project, refined through months of production use, redesigned multiple times as the stack grew more complex. Each script in the collection started as a one-off solution to a specific problem. Over time the patterns that worked were extracted into common.sh, the configuration was centralised into Master.conf, and the whole collection was standardised into what it is now.

The failover script that started it all is still the soul of the ecosystem — the DDNS sequencing, the two-ping state machine, the handback order. Everything else built on top of that foundation.

Two servers, one codebase, self-healing infrastructure. Step away. Come back to a happy system.