A modular, git-managed automation ecosystem for two unRAID servers. One configuration file. Both servers stay in sync with a single git pull. Everything from daily media syncing to mutual container failover runs automatically — and when something goes wrong, the system tries to fix itself before waking you up.

What This Is

Two friends. Hardware they already own. A mutual agreement to help each other.

This ecosystem delivers practical high availability and offsite backup between two unRAID servers at zero additional hardware cost. It is not enterprise-grade clustering. It does not promise zero disruption. What it delivers is minimal disruption — and for a homelab, that's genuinely good enough.

The alternative:

Proxmox cluster       — minimum 3 nodes
Ceph storage          — dedicated hardware
Enterprise networking — specialised equipment
Offsite infrastructure — at least 3 servers minimum
Cost                  — thousands in hardware, ongoing licensing

This ecosystem:

Two friends who already run unRAID
Hardware they already own
Mutual agreement — each covers the other
Cost: $0 beyond what they were already running

The honest user experience during failover:

Library transcode users  — buffer absorbs the cutover, usually seamless
Library direct play      — may notice, needs to resume
Live TV direct play      — no buffer, notices every time, has to resume
Watch history            — worst case 15-30min window behind
                           reality: a show or two, movies likely unaffected

Could force transcode on Live TV to get the buffer benefit —
not worth the permanent CPU/RAM overhead for an event that rarely happens.
Live TV disruption is the known, accepted tradeoff.

The real world math:

How often does a homelab server actually go down? How many of those times are during peak viewing? How many viewers notice vs just resume? Is the occasional minor annoyance worth thousands in enterprise hardware?

For most people — absolutely not. This ecosystem exists for those people.

Minimal disruption, not no disruption. Free, mutual, and genuinely useful.

The Goal

A self-hosted infrastructure that runs itself.

Step away → come back to a healthy system
Something breaks → system self-heals
Something can't be fixed → you get notified
Servers stay in sync → one edit propagates everywhere

Before this ecosystem existed, the same problems were solved by 60+ standalone scripts across two servers — different coding styles, no shared standards, every change applied twice. This ecosystem standardises everything into one codebase with one config file and one deployment mechanism.

The Servers

HOST1 — unRAID-Gmer4Lfe
  Hardware:   Threadripper 1950X, 128GB RAM
  Storage:    Multiple ZFS pools + cache
  Domain:     Gmer4Lfe.com
  Runs:       Full media stack, auth, live TV, arrs, downloaders
              Source of truth: Movies, Tv_Shows, Music
              Tdarr — runs anime transcoding (stronger hardware)

HOST2 — unRAID-Jayred365 (Buddy server)
  Hardware:   Intel i5 10th gen — completely different hardware
  Storage:    Different disk count, different pool layout
  RAM:        64GB
  Location:   Remote — 50 miles from HOST1
  Domain:     Gmer4Lfe.us (his own domain, his own services)
  Runs:       His own independent Emby, his own containers, his own users
              Source of truth: Anime_Shows, Anime_Movies
              His arrs manage anime — syncs to HOST1 as mirror

Network:    Tailscale — encrypted tunnel between both servers
Repo:       Self-hosted Gitea on HOST1
Deployment: git pull on either server → both stay current

This is a genuine mutual failover agreement between two friends, each running a fully independent server. Both servers are fully active in normal operation — each with their own service stack, their own domain, their own users. HOST2 is not a standby. He runs his own Emby, his own containers, and his own domain while sharing access to the mirrored media library.

Normal operation:
  HOST1  — Gmer4Lfe.com — full stack, live TV, arrs, everything
             Source of truth for all config, DNS, auth, certs
  HOST2  — Gmer4Lfe.us  — his Emby, his containers, his domain, his users
             Mirror of HOST1's auth stack — changes on HOST1 propagate here

The shared production Emby:

All users — from both servers — connect to one Emby instance running on HOST1. HOST1 has the stronger hardware and better bandwidth. Both domains route through NPM to the same Emby. This is intentional — one library, one watch history, one set of users.

Gmer4Lfe.com → NPM → Emby   (my users)
Gmer4Lfe.us  → NPM → Emby   (his users)
Both point at the same container on HOST1

Auth stack — runs on both servers simultaneously:

NPM, LLDAP, and Authelia run actively on both servers at all times. HOST2 needs them running to serve his users through his domain in normal operation. HOST1 is source of truth — all changes made there, mirrored to HOST2 every 15 minutes. One username and password works across all services on both domains. Group-based access controls what each user can see.

Change made on HOST1 LLDAP → mirrored to HOST2 → works on both domains
Cert renewed on HOST1       → mirrored to HOST2 → valid on both domains
NPM rule added on HOST1     → mirrored to HOST2 → routes on both servers

Why the auth stack runs on both — a year of testing:

Early versions started the auth stack from cold on failover. Testing showed 30-60 seconds of broken authentication after DNS cut over — Emby clients hit HOST2 before LLDAP, Authelia, and NPM were ready. For library transcode users with buffer this was invisible. For Live TV and direct play users this meant the stream died and reconnect failed, forcing a manual exit and resume.

Running the auth stack warm on both servers at all times eliminates this window entirely. When failover triggers, the auth stack is already running and ready. What actually starts from stopped:

Tier 1 — immediate:
  Emby          ← starts from stopped on covering server
  DDNS updater  ← remote domain updater starts on covering server

  Everything else verified healthy — NOT cold started:
  NPM, LLDAP, Authelia, certs — already running, already warm on both servers

Tier 4 — 18hr+ outage:
  Arrs_Stack    ← starts from stopped on covering server
                   has mirrored Arrs_Stack appdata — starts in known good state

When HOST1 goes down:

HOST2 detects HOST1 unreachable + internet up
→ Starts HOST1's DDNS updater — Gmer4Lfe.com now points at HOST2
→ Starts Emby — only real Tier 1 container starting from stopped
→ Auth stack already running on HOST2 — users authenticate immediately
→ DNS TTL 1 minute — cutover fast
→ Library transcode users — buffer absorbs, usually seamless
→ Live TV / direct play users — will notice, need to resume
→ Watch history — worst case 15-30min behind, reality a show or two
                  movies may be unaffected given their length
→ Tiers 2/3/4 escalate if outage extends
→ HOST2's own services keep running unaffected throughout

When HOST2 goes down:

HOST1 detects HOST2 unreachable + internet up
→ Starts HOST2's DDNS updater — Gmer4Lfe.us now points at HOST1
→ Starts his Emby — his users served from HOST1
→ His auth stack already running on HOST1 — serving his domain continuously
→ Tiers 2/3 escalate if outage extends
→ Tier 4 (18hr+): his Arrs_Stack starts from stopped on HOST1
                   mirrored appdata — starts in known good state
→ HOST1's own services keep running unaffected throughout

Handback — waits for stability:

Remote returns → FAILOVER_HANDBACK_STRIKES consecutive stable checks
→ Prevents handing back during a brief network blip
→ Stops remote DDNS first — no split brain during transition
→ Rsync writeback — userdata, watch states, auth changes during outage
→ Starts containers on correct server
→ Starts DDNS last — only after containers confirmed healthy
→ Returns to NORMAL

The hardware doesn't need to match. /mnt/user/ abstracts everything. A share called Movies is /mnt/user/Movies on both servers regardless of what drives or pools back it. rsync syncs the content. Container mounts use the same path. The hardware underneath is irrelevant.

Media library configurations — multiple valid setups:

The ecosystem supports several library arrangements. The right choice depends on how the two servers and their users relate to each other. All configurations use the same scripts — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES in Master.conf define who owns what.

Configuration A — Single source, one owner (simplest):

One server owns all media. The other mirrors everything and piggybacks.

HOST1 owns everything:
  Movies, Tv_Shows, Music, Anime — all managed by HOST1's arrs
  HOST2 mirrors all shares — read only in normal operation
  HOST2 arrs disabled or not running

HOST1_DAILY_SYNC_SHARES=(Movies Tv_Shows Music Anime_Shows Anime_Movies ...)
HOST2_DAILY_SYNC_SHARES=()   ← nothing — HOST2 owns nothing

Best for:
  One person running both servers
  Secondary server is pure failover + piggybacking
  Simpler arr management — one set of arrs, one library

Configuration B — Split source of truth (this setup):

Each server owns specific shares — managed by their own arrs. The other mirrors those shares read only.

HOST1 source of truth:            HOST2 source of truth:
  Movies    ← Radarr               Anime_Shows   ← his Sonarr
  Tv_Shows  ← Sonarr               Anime_Movies  ← his Radarr
  Music     ← Lidarr

HOST1 mirrors ← HOST2 anime        HOST2 mirrors ← HOST1 movies/shows/music

Both servers run arrs simultaneously — no conflict because they manage completely different shares. Scheduling keeps them further separated:

HOST2 arrs:   midnight → noon      downloading and managing anime
HOST1 Tdarr:  12:30   → 23:00     transcoding, syncs as source of truth

The rule is not "don't run arrs on both servers" — it's: never run two instances of the same arr against the same share simultaneously.

Each Emby instance on both servers can access all media — movies, shows, anime — because everything is mirrored in both directions. HOST1 mirrors anime from HOST2. HOST2 mirrors movies and shows from HOST1.

HOST1 /mnt/user/Movies      →  rsync  →  HOST2 /mnt/user/Movies
HOST2 /mnt/user/Anime_Shows →  rsync  →  HOST1 /mnt/user/Anime_Shows

At Tier 4 failover (18hr+ outage) each server's arr copies spin up to cover the other's shares — but only when the truth holder has genuinely been down long enough to need it.

Configuration C — Weighted split:

One server owns most shares, the other owns a smaller subset. Same mechanics as B — just an unequal split based on what each person actually manages.

HOST1 owns: Movies, Tv_Shows, Music, Kids_Movies, Stand-up, Sports
HOST2 owns: Anime_Shows, Anime_Movies

HOST1_DAILY_SYNC_SHARES=(Movies Tv_Shows Music Kids_Movies ...)
HOST2_DAILY_SYNC_SHARES=(Anime_Shows Anime_Movies)

Most real-world split setups will land here — one person has a larger collection, the other a specific niche.

Configuration D — Fully shared, no split:

Both servers have access to all media but neither "owns" it in a managed sense. No arrs running on either for the shared shares. Content added manually or by one designated manager.

HOST1_DAILY_SYNC_SHARES=(Movies Tv_Shows Music ...)
HOST2_DAILY_SYNC_SHARES=()

One-way push — HOST1 is the single truth holder
HOST2 is a read-only mirror for all media
Simpler than split — no arr coordination needed

What all configurations have in common:

Same share names on both servers        ← /mnt/user/Movies everywhere
Same container names on both servers    ← Emby, NginxProxyManager everywhere
HOST*_DAILY_SYNC_SHARES defines truth   ← who pushes what, automatically
daily_sync.sh reads the correct list    ← direction determined at runtime
Failover Tier 4 uses opposing list      ← writeback always correct direction

The scripts don't need to know which configuration you're running. The arrays in Master.conf define the truth holders — everything else follows automatically.

Container naming convention:

Server-specific containers — always on their own server, unique names:
  Emby-Gmer4Lfe         ← HOST1's personal Emby, always on HOST1
  Emby-Jayred365        ← HOST2's personal Emby, always on HOST2
  VaultWarden-Gmer4Lfe  ← HOST1's password manager
  VaultWarden-Jayred365 ← HOST2's password manager

Shared failover containers — same name on both servers:
  Emby                  ← HOST1's main Emby, starts on HOST2 during HOST1 outage
  NginxProxyManager     ← same name, same proxy rules, same networks
  Authelia              ← same name

Docker custom networks — containers communicate by name within custom networks, not by IP. When a container restarts and gets a new IP, NPM still reaches it by name. Failover containers join the same named networks on either server — no proxy rule changes, no IP reconfiguration, no downtime from IP drift.

Personal shares — backup without failover:

Beyond the shared media library, each user can sync personal shares to the other server purely for offsite backup — no failover container involvement, just data protection.

HOST1: /mnt/user/Gmer4Lfe-Personal  →  rsync nightly  →  HOST2 (encrypted backup)
HOST2: /mnt/user/Jayred365-Personal  →  rsync nightly  →  HOST1 (encrypted backup)

Configure in Master.conf:

HOST1_PERSONAL_SHARES=(
    "/mnt/user/Gmer4Lfe-Personal"
)
HOST2_PERSONAL_SHARES=(
    "/mnt/user/Jayred365-Personal"
)

daily_sync.sh automatically picks up the personal shares for the local host and syncs them alongside that host's truth shares.

Encrypting personal shares — unRAID 7 ZFS:

ZFS native encryption in unRAID 7 means both admins can see the share exists and file sizes but neither can read content without your passphrase or keyfile. rsync copies encrypted blocks as-is — the remote server never needs your key.

Setup on HOST1 (your personal share):

Step 1 — Create an encrypted ZFS dataset:

unRAID UI → Main tab → click your ZFS pool
→ Click "+ Dataset"
→ Name: Gmer4Lfe-Personal
→ Enable Encryption: Yes
→ Encryption type: passphrase (simplest) or keyfile (auto-unlock capable)
→ Enter your passphrase — write it down, if lost data is unrecoverable
→ Create

Step 2 — Create the share:

Settings → Shares → Add Share
→ Name: Gmer4Lfe-Personal
→ Primary storage: your ZFS pool
→ Use cache: Only (keeps data on ZFS pool, not array)
→ Add

Step 3 — Verify encryption is active:

zfs get encryption poolname/Gmer4Lfe-Personal
# Should show: encryption  aes-256-gcm

Step 4 — Add to Master.conf and sync:

HOST1_PERSONAL_SHARES=(
    "/mnt/user/Gmer4Lfe-Personal"
)

Auto-unlock on boot (keyfile approach):

If you want the share to mount automatically after reboot without entering a passphrase:

# Create keyfile — on HOST1 only, never sync this file
mkdir -p /root/.zfs-keys
dd if=/dev/urandom bs=32 count=1 | base64 > /root/.zfs-keys/personal.key
chmod 400 /root/.zfs-keys/personal.key

# Set dataset to use keyfile
zfs change-key -o keylocation=file:///root/.zfs-keys/personal.key \
    -o keyformat=raw poolname/Gmer4Lfe-Personal

# Add to array start script (unRAID_Essentials or User Scripts)
zfs load-key poolname/Gmer4Lfe-Personal
zfs mount poolname/Gmer4Lfe-Personal

Manual unlock (most secure — you control when it's readable):

zfs load-key poolname/Gmer4Lfe-Personal   # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal

What the remote admin sees:

The share directory exists on HOST2. File names and sizes are visible (ZFS encrypts content, not metadata by default). File contents are unreadable without your key. To hide filenames too, enable zfs set encryption=aes-256-gcm with dnodesize=auto — this is a more advanced setup.

Current status: The infrastructure supports encrypted personal share syncing. The ZFS dataset setup is a one-time manual step per server. Once set up it syncs automatically like any other share.

How It Works

Every script sources two files at startup:

source "$SCRIPT_DIR/../Master.conf"   # all user configuration
source "$SCRIPT_DIR/../common.sh"     # shared library

Master.conf is the single source of truth. Container names, thresholds, paths, API keys, rsync profiles, failover tiers — everything configurable lives here. Change a value, push to git, both servers pull — done.

common.sh provides shared functions used by every script — host detection, notifications, rsync helpers, output formatting, icon set. Scripts never duplicate this logic.

git_pull_execute.sh pulls the latest scripts from Gitea and sets executable permissions. Schedule it or run it manually on either server.

Repository Structure

Unraid_Scripts/
├── Master.conf                        # All user configuration — edit here only
├── common.sh                          # Shared library — used by all scripts
├── git_pull_execute.sh                # Pull latest scripts from Gitea
├── User_Script_Template.sh            # Paste into unRAID User Scripts plugin
│
├── Failover/                          # Mutual container failover
│   ├── README-Failover.md
│   ├── failover.sh                    # State machine — runs continuously
│   └── failover_test.sh               # Controlled simulation harness
│
├── Monitors/                          # Health reporting — watch and report only
│   ├── README-Monitors.md
│   ├── cert_monitor.sh                # SSL certificate expiry
│   ├── smart_health.sh                # Drive SMART attributes
│   ├── zfs_memory_snapshot.sh         # ZFS pool health + memory report
│   ├── backup_verify.sh               # Random checksum verification vs remote
│   ├── bandwidth_monitor.sh           # Rsync transfer history
│   ├── weekly_health_digest.sh        # Aggregated system health summary
│   └── emby_session_report.sh         # Emby usage statistics
│
├── Orchestrators/                     # Sequential job runners
│   ├── README-Orchestrators.md
│   ├── daily_sync.sh                  # All media shares synced nightly
│   ├── media_management.sh            # Permissions + cleaners + arr cleanup
│   └── transcode_management.sh        # Cleanup then manager every 3min + daily stats
│
├── Rsync/                             # Core sync engine
│   ├── README-Rsync.md
│   ├── rsync.sh                       # Per-share or per-profile sync
│   └── README_Rsync_Setup.md          # Initial setup guide
│
├── Docker_Essentials/                 # Container lifecycle management
│   ├── README-Docker_Essentials.md
│   ├── docker_watchdog.sh             # Two-tier self-healing monitoring
│   ├── docker_daily_restart.sh        # Daily container restarts
│   ├── docker_weekly_restart.sh       # Weekly container restarts
│   └── docker_network_connect.sh      # Extra network connections at boot
│
├── Media/                             # Media library maintenance
│   ├── README-Media.md
│   ├── media_shares_permissions.sh    # Recursive permission application
│   ├── media_cleaner.sh               # Junk file removal (anime + media profiles)
│   ├── lidarr_cleanup.sh              # Orphaned music file cleanup
│   ├── sonarr_cleanup.sh              # Orphaned TV file cleanup
│   └── radarr_cleanup.sh              # Orphaned movie file cleanup
│
├── Transcodes/                        # Emby ramdisk transcode management
│   ├── README-Transcoding.md
│   ├── ramdisk_setup.sh               # Create ramdisk + symlink at array start
│   ├── transcode_manager.sh           # Monitor usage, manage symlink, display sessions
│   └── transcode_cleanup.sh           # Remove old inactive transcode files
│
├── Tools/                             # Situational utilities — run when needed
│   ├── README-Tools.md
│   ├── recreate_shares.sh             # Recreate share dirs after incident
│   ├── failover_state_reset.sh        # Reset failover state file to NORMAL
│   ├── watchdog_skip_list_manager.sh  # Manage container watchdog skip lists
│   ├── bulk_permissions_repair.sh     # Targeted permission repair for one share
│   ├── container_data_export.sh       # Export container appdata to tar archive
│   ├── emby_database_repair.sh        # SQLite integrity check on Emby databases
│   └── zfs_pool_scrub.sh              # Trigger ZFS scrub with completion report
│
└── unRAID_Essentials/                 # Server-level system management
    ├── README-Unraid_Essentials.md
    ├── system_watchdog.sh             # Last line of defense — controlled reboot
    ├── webgui_restart.sh              # WebGUI nginx + emhttp auto-restart
    ├── docker_syslog_filter.sh        # Suppress Docker veth noise from syslog
    ├── php_fpm_max_children.sh        # WebGUI PHP-FPM concurrency tuning
    ├── clear_logs.sh                  # Weekly system log clearing
    ├── mover_stop.sh                  # Graceful mover termination
    ├── rsync_stop.sh                  # Stop rsync + recover containers
    ├── server_reboot.sh               # Graceful reboot with user warning
    └── user_scripts_stop.sh            # Stop all running User Scripts jobs

Folders at a Glance

Folder	What it does	Key scripts
Failover	Mutual container failover — autonomous, tiered, DDNS-safe	`failover.sh`
Monitors	Watch and report — never act, minimal flash writes	`weekly_health_digest.sh`
Orchestrators	Sequential job runners with unified reporting	`daily_sync.sh`, `media_management.sh`
Rsync	Core sync engine with profile system	`rsync.sh`
Docker_Essentials	Two-tier self-healing container management	`docker_watchdog.sh`
Media	Permissions, junk cleanup, arr orphan cleanup	`media_cleaner.sh`, arr scripts
Transcodes	Ramdisk symlink routing for Emby	`transcode_manager.sh`
Tools	Recovery and situational utilities	`failover_state_reset.sh`, `zfs_pool_scrub.sh`
unRAID_Essentials	Server-level maintenance and last-resort recovery	`system_watchdog.sh`

Key Design Principles

One config file. Master.conf is the only file you edit. No hunting through scripts to change a container name or a threshold.

Bidirectional. Both servers run identical scripts. detect_hosts() determines local vs remote at runtime. One codebase covers both directions.

Exact naming consistency across both servers — non-negotiable.

This is the most important operational principle and the one most likely to cause problems if ignored. The ecosystem is built on the assumption that containers and shares have identical names on both servers.

Container names:
  Emby                    ← HOST1 and HOST2
  Mariadb-Authelia        ← HOST1 and HOST2
  Lldap-Gmer4Lfe          ← HOST1 and HOST2
  NginxProxyManager       ← HOST1 and HOST2

Share names:
  /mnt/user/Movies        ← HOST1 and HOST2
  /mnt/user/Tv_Shows      ← HOST1 and HOST2
  /mnt/user/Anime_Shows   ← HOST1 and HOST2

Why this matters:

rsync.sh:    pushes /mnt/user/Movies → remote /mnt/user/Movies
             same path, both sides — no mapping, no translation

Container stops:
             PROFILE_CRITICAL_CONTAINER_NAMES stops "Emby" locally
             then stops "Emby" on remote
             one list, both servers, no duplication

failover.sh: starts "Emby" on covering server
             same name it always has everywhere

detect_hosts(): compares hostname against HOST1/HOST2 in Master.conf
             exact string match — hostnames must match exactly

Years of running with custom per-user share paths (/mnt/user/me/movies, /mnt/user/jayred/movies) taught the hard lesson — every difference becomes a maintenance burden. Every script needs to know about it. Every rsync needs a custom path. Every container stop list diverges.

Shared media shares should be shared — identical paths, identical names, no ownership prefix. User-specific data belongs in personal encrypted shares that are explicitly separate from the shared ecosystem. That boundary keeps the common infrastructure simple and the private data private.

If a container or share has a different name on one server — the script either skips it gracefully or fails to find it. It will not error catastrophically, but it also will not do what you expect. Consistent naming is what makes one codebase work on both servers without modification.

Self-healing layers. Problems are addressed at the most targeted level first:

docker_watchdog.sh    — container level, minimal disruption
system_watchdog.sh    — system level, last resort
failover.sh           — infrastructure level, other server covers

Strike systems, not hair triggers. Single spikes don't cause restarts or reboots. Sustained problems do. The strike system filters noise from genuine issues.

Notifications when action is needed, silence otherwise. The ecosystem is designed to run without daily attention. You hear from it when something needs human intervention — not as a regular occurrence.

Flash drive friendly. Scripts that write to /boot/ use bounded files with auto-purge. /tmp/ is used for ephemeral state that resets on reboot. Most scripts write nothing at all.

--dry-run everywhere. Every script supports --dry-run. Test before you schedule.

Hardware utilization without script changes. Any unRAID system can participate regardless of hardware. The stronger server runs the heavier jobs — not because the scripts force it, but because app scheduling and rsync timing make it happen naturally. No script changes needed to reconfigure who does what:

HOST1 — Threadripper, 128GB
  Tdarr transcoding: 12:30 → 23:00    ← heavy job on stronger hardware
  
HOST2 — i5, 64GB
  His arrs: midnight → noon            ← lighter load, scheduled around Tdarr
  Benefits from HOST1's transcoding
  without the CPU/RAM cost

Change a schedule in an app. Adjust an rsync timing. The ecosystem adapts. The scripts never need to know which server is doing what — they just do their jobs on whatever server they're running on. This makes the ecosystem applicable to any two unRAID servers in any configuration — not just this specific hardware pairing.

Mutually beneficial, not one-way. Both servers contribute what their hardware does best. Both benefit from what the other provides. The arrangement works because it's genuinely useful to both sides — not because one server is just a passive backup for the other.

Scheduling Overview

# ━━━ At Startup of Array ━━━
failover.sh                     # background task — continuous loop
ramdisk_setup.sh
docker_syslog_filter.sh
php_fpm_max_children.sh
docker_network_connect.sh

# ━━━ Frequent ━━━
*/3  * * * *    transcode_management.sh  # replaces separate manager + cleanup entries
*/10 * * * *    webgui_restart.sh
*/15 * * * *    docker_watchdog.sh
*/15 * * * *    system_watchdog.sh

# ━━━ Daily ━━━
0 1 * * *       daily_sync.sh
0 2 * * *       media_management.sh
0 3 * * *       docker_daily_restart.sh
0 8 * * *       weekly_health_digest.sh

# ━━━ Weekly — Sunday ━━━
0 3 * * 0       docker_weekly_restart.sh
0 5 * * 0       clear_logs.sh
0 6 * * 0       zfs_memory_snapshot.sh
0 7 * * 0       smart_health.sh
0 9 * * 0       cert_monitor.sh
0 10 * * 0      backup_verify.sh
0 11 * * 0      emby_session_report.sh
0 11 * * 0      bandwidth_monitor.sh --report

Folder READMEs

Each folder has a detailed README covering setup, configuration, usage, and the reasoning behind design decisions.

README	Contents
README-Failover.md	DDNS split brain prevention, tiered failover, handback sequence, initial setup, troubleshooting
README-Monitors.md	All monitor scripts, flash drive write policy, scheduling
README-Orchestrators.md	Why orchestrators exist, job ordering, how to add jobs
README-Rsync.md	Profile system, SSH key setup, Tailscale requirements
README-Docker_Essentials.md	Two-tier watchdog, startup grace, dependency ordering, skip list management
README-Media.md	Execution order, cleaner profiles, arr cleanup safety procedure
README-Transcoding.md	Symlink indirection design, Docker mount warning, mode switching, sizing
README-Tools.md	All utility scripts, when to use each, how to add new tools
README-Unraid_Essentials.md	system_watchdog tiers, startup sequence, scheduled maintenance

v2 Roadmap

The v1 ecosystem is production-proven and stable. These are the features planned for v2 — some require additional infrastructure, some are extensions of existing systems.

Transcode Manager — Advanced Mode

The current smart / ramdisk / ssd modes route all sessions to one location. v2 adds an advanced mode that routes by media type:

TRANSCODE_MANAGER_MODE="advanced"

TRANSCODE_FORCE_RAMDISK=(
    "LiveTv"    # always ramdisk — buffering is latency sensitive
)
TRANSCODE_FORCE_SSD=(
    "Audio"     # music/downloads — no benefit from ramdisk
)
# Everything else follows smart threshold behavior

The groundwork is already in place — session display with media type parsing is working. Advanced mode requires Emby to expose media type at the point ffmpeg resolves the symlink, which means the routing decision needs to happen before the session starts. This is the problem to solve.

Plugin Dashboard

The Monitors folder is already the data backend for a future unRAID plugin dashboard. Every monitor script writes structured state that a plugin could read and display:

Failover state          → live status indicator
Container watchdog      → strike counts, skip list
SMART health            → per-drive status
ZFS pools              → health + ARC utilization
Transcode sessions      → live session display
Bandwidth history       → transfer trend charts
Certificate expiry      → days remaining per domain

The scripts exist. The data exists. The plugin is the frontend.

Failover — Tiered by Content Type

Currently failover starts containers based on time elapsed. A future enhancement would start containers based on what the primary server was doing before it went down:

Primary running Live TV sessions → start Live TV stack on secondary immediately
Primary idle                     → standard tier delays apply
Primary in heavy transcoding     → start Emby immediately, defer others

This requires the failover state file to track active session types, which requires integration with the Emby API at failover trigger time.

Health Digest — Plugin Integration

Currently the digest sends a notification. In v2 it populates a persistent dashboard that shows a rolling week of system health at a glance — without requiring a notification for every event.

Container Health Checks — Standardised Library

The docker_watchdog already has HTTP check support per container. A v2 enhancement is a standardised health check library — pre-built check commands for every container in the stack that can be applied via Extra Parameters in the unRAID template without manual research per container.

Origin

This ecosystem grew from a single failover script — written as a first real bash project, refined through months of production use, redesigned multiple times as the stack grew more complex. Each script in the collection started as a one-off solution to a specific problem. Over time the patterns that worked were extracted into common.sh, the configuration was centralised into Master.conf, and the whole collection was standardised into what it is now.

The failover script that started it all is still the soul of the ecosystem — the DDNS sequencing, the two-ping state machine, the handback order. Everything else built on top of that foundation.

Two servers, one codebase, self-healing infrastructure. Step away. Come back to a happy system.