━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🏠 UNRAID SCRIPT ECOSYSTEM
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
A complete self-healing, self-maintaining, mutually-redundant two-server unRAID ecosystem. One codebase runs on both servers. Both servers run their own lives independently. When one goes down the other covers it — not because of conventional failover infrastructure, but because both servers already have copies of everything that matters and both know how to run it.
This is not conventional failover. There is no primary and standby. There is no shared storage. There is no cluster manager. There are two fully independent unRAID servers, living in different locations on different power utilities, each running their own household's media stack — and a layer of scripts that keeps them synchronised, coordinated, and mutually covering. The goal is minimal disruption to a media stack, not enterprise high availability.
━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HOST1 — unRAID-Gmer4Lfe
Hardware: Threadripper 1950X, 128GB RAM, ZFS cache pools
Location: Primary site
Domain: Gmer4Lfe.com
Owns: Movies, Tv_Shows, Music — source of truth
Auth stack — source of truth
Emby — primary media server
HOST2 — unRAID-Jayred365
Hardware: Intel i5 10th gen, 64GB RAM
Location: Remote — different building, different power utility
Domain: Gmer4Lfe.us
Owns: Anime_Shows, Anime_Movies — source of truth
His own Emby — for his household
Hardware does not need to match. Everything is accessed through /mnt/user/ —
unRAID's unified share layer. HOST1 has a Threadripper with ZFS. HOST2 has a completely
different CPU and fewer drives. A failover container on HOST2 mounts /mnt/user/Movies
and sees mirrored data because the share names match. The hardware underneath is
irrelevant.
━━━ CONTAINER NAMING — YOU NEED TO KNOW THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
This is the most important convention in the ecosystem. Get it wrong and the scripts break silently.
── Shared Containers — No Suffix ───────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Shared containers — meaning both servers run this container and scripts treat
# them as the same logical service — use the plain name with no suffix.
#
# The convention: "no suffix = this container participates in the shared stack"
#
Emby # the shared Emby — runs on HOST1, picked up by HOST2 on failover
NginxProxyManager # shared NPM — runs on both, same config via sync
Lldap-Gmer4Lfe # shared LLDAP — same user directory, both servers
Authelia # shared auth — both servers, same config
Mariadb-Authelia # Authelia's database — same on both
Redis-Authelia # Authelia's cache — same on both
VaultWarden # shared password manager
Dispatcharr # shared Live TV scheduler
When failover.sh on HOST2 reads FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1=(..."Emby"...),
it looks for a container called exactly Emby and starts it. HOST2 must have a
container called Emby — not Emby-Gmer4Lfe, not Emby-HOST1, just Emby — with
its volume mounts pointing at the mirrored share paths. This is what makes one
codebase work on both servers without modification.
── User-Specific Containers — Suffixed ────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# User-specific containers — things that belong to one person and only ever
# run on their server — use a suffix identifying the user/owner.
#
Emby-Gmer4Lfe # Gmer4Lfe's personal Emby instance (HOST1, never covered)
Emby-Jayred365 # Jayred365's personal Emby instance (HOST2, never covered)
Organizer-Gmer4Lfe # Gmer4Lfe's dashboard — personal, stays on HOST1
Sonarr-Gmer4Lfe # Gmer4Lfe's Sonarr — managing his Tv_Shows share
Radarr-Gmer4Lfe # Gmer4Lfe's Radarr — managing his Movies share
Immich-Gmer4Lfe # Gmer4Lfe's photo library — personal
User-specific containers are covered during failover (see Tiered Failover below) but
are distinct from the shared stack. They are never confused with each other because
the suffix makes the owner unambiguous. Scripts targeting Emby and scripts targeting
Emby-Gmer4Lfe are targeting different containers for different purposes.
── Why This Matters ─────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# The entire ecosystem uses container names as identifiers.
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
# → looks for exactly "NginxProxyManager" on the running server
# failover.sh: FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1=("Emby" "VaultWarden")
# → looks for exactly "Emby" and "VaultWarden" on HOST2
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
# → looks for exactly those names in docker ps
#
# A container named differently than expected is not found — no error, no restart,
# no failover coverage. It silently does not happen.
# Keep names consistent on both servers for shared containers.
# ─────────────────────────────────────────────────────────────────────────────
━━━ SHARE NAMING — ALSO YOU NEED TO KNOW THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Shares must have identical names on both servers. The ecosystem uses share paths as sync targets and mount points — a mismatch is a broken path.
# ─────────────────────────────────────────────────────────────────────────────
# The rule: if a share exists on HOST1 and HOST2 needs a mirror of it,
# the share on HOST2 must have the exact same name.
#
# Same name on both servers:
/mnt/user/Movies # HOST1 source of truth, HOST2 mirror
/mnt/user/Tv_Shows # HOST1 source of truth, HOST2 mirror
/mnt/user/Music # HOST1 source of truth, HOST2 mirror
/mnt/user/Anime_Shows # HOST2 source of truth, HOST1 mirror
/mnt/user/Anime_Movies # HOST2 source of truth, HOST1 mirror
#
# rsync.sh syncs /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2.
# If HOST2 has it at /mnt/user/Movies-Mirror → rsync aborts: path not found.
# ─────────────────────────────────────────────────────────────────────────────
━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
── Common Libraries — Shared Pool ─────────────────────────────────────────
Common media like Movies, Music, and TV Shows are a shared pool. Every container
that needs music — Emby, LidaTube, Navidrome, whatever — mounts the same /mnt/user/Music
share. There is one share, one location, one source of truth. No duplication of content
across separate shares per-user or per-application.
# ─────────────────────────────────────────────────────────────────────────────
# One share for the library — all containers that need it mount the same path.
# Emby container: /mnt/user/Music → /media/Music (read)
# Lidarr: /mnt/user/Music → /music (read+write)
# LidaTube: /mnt/user/Music → /downloads (write)
# Navidrome: /mnt/user/Music → /music (read)
#
# All pointing at the same /mnt/user/Music. No per-user music copies.
# When Lidarr downloads and Emby scans — they're looking at the same files.
# ─────────────────────────────────────────────────────────────────────────────
── Never Two Arrs Against the Same Share Simultaneously ───────────────────
# ─────────────────────────────────────────────────────────────────────────────
# This rule is absolute. No exceptions.
#
# Correct: HOST1 Sonarr manages /mnt/user/Tv_Shows
# HOST2 Sonarr manages /mnt/user/Anime_Shows
# Different shares — no conflict
#
# Correct: HOST1 Tdarr transcodes from /mnt/user/Anime_Shows
# HOST2 Sonarr downloads into /mnt/user/Anime_Shows
# But: they run in non-overlapping time windows
#
# WRONG: HOST1 Sonarr AND HOST2 Sonarr both managing /mnt/user/Tv_Shows
# Two arrs managing the same library = rename conflicts, double imports,
# deletion battles, corrupted databases.
#
# This is why source of truth is split:
HOST1 owns: Movies (Radarr), Tv_Shows (Sonarr), Music (Lidarr)
HOST2 owns: Anime_Movies (Radarr), Anime_Shows (Sonarr)
#
# When HOST2 enters failover and starts HOST1's arr containers — those arrs
# point at /mnt/user/Movies etc. HOST2's own arrs point at /mnt/user/Anime_*.
# Different shares, no conflict. This is by design.
# ─────────────────────────────────────────────────────────────────────────────
── Source of Truth ──────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Every share has exactly one server that owns it.
# That server's arr manages it. That server pushes it to the other via daily sync.
# The receiving server treats it as read-only.
#
# Rule: never add content to a mirrored share on the non-owning server.
# It will be overwritten on the next sync.
# ─────────────────────────────────────────────────────────────────────────────
━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Not everything needs to sync. Not everything should sync. Understanding what goes where determines whether you need encryption and whether sync failure matters.
── Shared Failover Shares — Sync Required ─────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# These shares enable failover — HOST2 must have current copies for the covered
# services to work correctly when they start on HOST2.
#
/mnt/user/Media_Server/Emby # Emby userdata — watch history, playstates
/mnt/user/appdata-Failover/Critical-Data # auth stack — NPM rules, certs, user accounts
/mnt/user/Movies # media library content
/mnt/user/Tv_Shows # media library content
/mnt/user/Music # media library content
#
# Sync failure here = failover runs but with stale data (acceptable) or missing
# data (problematic). These sync every night via daily_sync_maintenance.sh.
# Critical-Data and Emby also sync every 15 minutes via critical_sync_maintenance.sh.
# ─────────────────────────────────────────────────────────────────────────────
── Non-Failover Shares — Sync as Backup, No Encryption Required ───────────
# ─────────────────────────────────────────────────────────────────────────────
# Some shares don't participate in failover — their container only runs on one
# server and is covered during failover from scratch (or not at all).
# These shares can sync as offsite backup without encryption.
# Content is not sensitive — anyone with access to HOST2 could read it.
#
# Examples:
/mnt/user/appdata-Failover/Gmer4Lfe # Gmer4Lfe server appdata backup
/mnt/user/appdata-Failover/Important-Data # NextCloud, Immich appdata
/mnt/user/Kids_Movies # kids content — not sensitive
/mnt/user/Sports # sports content — not sensitive
#
# These sync in DAILY_SYNC_SHARES — straight rsync, no special handling.
# If HOST2 is compromised, the content is readable. That is acceptable.
# ─────────────────────────────────────────────────────────────────────────────
── Personal Shares — Always Encrypted ─────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Personal shares contain content only the owner should see — documents,
# photos, financial records, private files.
# ZFS encrypts at the dataset level — HOST2 receives encrypted blocks.
# Without your passphrase or keyfile, HOST2 cannot read the content.
#
HOST1_PERSONAL_SHARES=(
"/mnt/user/Gmer4Lfe-Personal" # encrypted ZFS dataset
)
HOST2_PERSONAL_SHARES=(
"/mnt/user/Jayred365-Personal" # encrypted ZFS dataset
)
#
# Append to HOST*_DAILY_SYNC_SHARES — synced nightly with everything else.
# Same rsync, same timing — but the content on HOST2 is ciphertext only you can unlock.
#
# Setup: README-Rsync_Setup.md → Step 8 — Personal Encrypted Shares
# ─────────────────────────────────────────────────────────────────────────────
━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Understanding this distinction prevents a lot of confusion.
Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a primary server and a standby server with shared storage or synchronous replication. The standby exists only to take over when the primary fails. It runs nothing independently. Its purpose is 100% failover coverage with no service interruption.
This is not that.
── What This Actually Is ─────────────────────────────────────────────────────
Two fully independent servers, each running their own household's services.
Each server has its own domain, its own users, its own media library management.
Each server is the primary for its own services every single day.
The "failover" component is additive:
When HOST1 goes down → HOST2 starts HOST1's containers on top of its own
When HOST2 comes back → HOST1's containers hand back cleanly
HOST2 never stopped running its own services during this entire time
This is not standby. This is mutual coverage between peers.
── The Actual Goal: Minimal Disruption to a Media Stack ────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# "Minimal disruption" means:
# Live TV keeps playing → Dispatcharr is Tier 1 (starts immediately)
# Passwords are accessible → VaultWarden is Tier 1 (people lock themselves out)
# Emby keeps serving → Emby is Tier 1
# Auth works → NPM, Authelia, LLDAP already warm on both servers
#
# "Minimal disruption" does NOT mean:
# Zero downtime (DDNS TTL = 1 minute, brief gap expected)
# Perfect consistency (Emby userdata is 15min stale at most)
# All services covered immediately (tiered start over 24 hours)
#
# The system is designed around the real experience of a household:
# "The TV stopped working" → unacceptable, fixed in Tier 1 (< 2 minutes)
# "NextCloud is down" → annoying, fixed in Tier 2 (4 hours)
# "Gitea is down" → inconvenient, fixed in Tier 3 (12 hours)
# "Sonarr isn't downloading" → fine, fixed in Tier 4 (24 hours)
# ─────────────────────────────────────────────────────────────────────────────
── What You Give Up With This Approach ─────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Things conventional failover provides that this does NOT:
# Zero RPO (recovery point objective) — there is ~15min of potential staleness
# Synchronous state replication — changes don't reach HOST2 instantly
# Automatic failure detection in seconds — failover.sh checks every 2 minutes
# Transparent IP migration — DNS TTL means up to 1 minute of routing to old IP
#
# What you get instead:
# Both servers run fully independently — no single point of failure in software
# Geographic redundancy — different buildings, different power, different ISP
# Both households stay operational during most disruption scenarios
# Simple, auditable scripts — no cluster manager black box
# ─────────────────────────────────────────────────────────────────────────────
━━━ HOW FAILOVER ACTUALLY WORKS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
failover.sh runs on both servers as a continuous background process. Every 2 minutes
it pings two things: the other server's Tailscale IP, and 8.8.8.8. The combination
of those two answers determines the current state.
Remote ping PASS + Internet PASS → NORMAL — both servers running independently
Remote ping FAIL + Internet PASS → FAILOVER — remote is down, start covering it
Tier 1 immediately: DDNS + Emby + Auth + VaultWarden + Live TV
Tier 2 at 4 hours: NextCloud + Immich + Jellyseerr
Tier 3 at 12 hours: AdGuard + Gitea + Uptime monitoring
Tier 4 at 24 hours: Arrs + download clients
Internet FAIL (either) → NO_INTERNET — stop own DDNS, wait
Can't serve anyone without internet — stop claiming DNS ownership
Remote FAIL + Internet FAIL → DARK — conservative, same as NO_INTERNET
The DDNS sequencing is critical — one domain, one DDNS container, at all times. The handback sequence stops remote DDNS first, rsyncs, starts containers, starts local DDNS last. Split-brain DNS is prevented by design.
→ Full detail: README-Failover.md
━━━ THE AUTH STACK — WARM ON BOTH SERVERS ALWAYS ━━━━━━━━━━━━━━━━━━━━━━━━━━
NPM, LLDAP, Authelia, MariaDB, Redis — all running actively on both servers at all times. Not cold standby. Not started at failover. Running right now, serving each server's own domain.
# ─────────────────────────────────────────────────────────────────────────────
# Why this eliminates the failover dead zone:
#
# Cold standby approach: HOST1 goes down → DNS switches to HOST2 → HOST2 starts
# auth containers → 30-60 seconds before auth is ready → users hit the auth wall
# before the service behind it responds → failed logins, broken sessions
#
# Warm approach: HOST1 goes down → DNS switches to HOST2 → auth is ALREADY
# running → users are served immediately with no dead zone
#
# HOST1 is source of truth for the auth configuration. All changes to NPM proxy
# rules, LLDAP users, Authelia policies, and certs happen on HOST1 and propagate
# to HOST2 via Critical-Data sync every 15 minutes.
#
# HOST2 operator's experience: click NginxProxyManager in Docker UI →
# browser opens HOST1's NPM via Tailscale redirect → make changes there →
# 15 minutes later live on both servers. Zero auth management overhead on HOST2.
# ─────────────────────────────────────────────────────────────────────────────
→ Full detail: README-Partnership.md
━━━ THE COMPLETE PICTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HOST1 (Gmer4Lfe.com) HOST2 (Gmer4Lfe.us)
────────────────── ──────────────────────
Always running:
Own stack Full service stack Full service stack (anime focus)
Auth NPM, Authelia, LLDAP NPM, Authelia, LLDAP (mirrored)
Source of truth:
Media Movies, Tv_Shows, Music ←── Anime_Movies, Anime_Shows
───────────────────────→ (bidirectional daily push)
Config auth stack config ──→ mirrored every 15 min
Emby state watch history, playstates──→ dirty sync every 30 min
Failover coverage:
If HOST1 down: — starts Emby, VaultWarden,
Dispatcharr, all arr stack
serves Gmer4Lfe.com via DDNS
If HOST2 down: starts his Emby, his arrs, —
serves Gmer4Lfe.us via DDNS
Monitoring:
Both servers system_watchdog.sh system_watchdog.sh
docker_watchdog.sh docker_watchdog.sh
failover.sh failover.sh
Sunday morning coffee report Sunday morning coffee report
━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Unraid_Scripts/
│
├── master.conf ← All shared configuration — the only file you edit regularly
├── master_host1.conf ← HOST1-specific: share lists, container names, API keys
├── master_host2.conf ← HOST2-specific: same structure, different values
├── common.sh ← Shared library — all functions used by every script
├── load_config.sh ← Sources all conf files and common.sh
│
├── Orchestrators/ ← Scheduled job runners — what User Scripts plugin executes
│ README: README-Orchestrators.md
│
├── Rsync/ ← Core rsync script — called per share with profile
│ README: README-Rsync_Setup.md
│
├── Failover/ ← Mutual automatic failover — continuous background process
│ README: README-Failover.md
│
├── Docker_Essentials/ ← Container lifecycle: watchdog, restarts, networks
│ README: README-Docker_Essentials.md
│
├── unRAID_Essentials/ ← Server-level: system watchdog, WebGUI, log hygiene, tuning
│ README: README-Unraid_Essentials.md
│
├── Media/ ← Library health: permissions, junk cleanup, orphan removal
│ README: README-Media.md
│
├── Transcodes/ ← Ramdisk transcode management with SSD fallback
│ README: README-Transcoding.md
│
├── Monitors/ ← Read-only observation: certs, SMART, bandwidth, digests
│ README: README-Monitors.md
│
├── Partnership/ ← Two-server relationship lifecycle: onboard, offboard, transfer
│ README: README-Partnership.md
│
└── Tools/ ← Situational utilities: repair, export, emergency tools
README: README-Tools.md
━━━ CONFIGURATION — THREE FILES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# ─────────────────────────────────────────────────────────────────────────────
# master.conf — shared across both servers
# Everything that applies equally to both: thresholds, schedules, profile
# definitions, watchdog settings, arr cleanup config, DDNS timing, etc.
# Pushed to both servers via git. Never contains server-specific values.
#
# master_host1.conf — sourced only on HOST1
# HOST1_* prefixed variables: share lists, container names, API keys,
# ramdisk size, specific paths, per-server toggle overrides.
#
# master_host2.conf — sourced only on HOST2
# HOST2_* prefixed variables: same structure, different values.
#
# detect_hosts() in common.sh:
# Reads the current server's hostname. Matches it against HOST1 and HOST2.
# Aliases HOST1_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST1)
# Aliases HOST2_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST2)
# Every script uses the unprefixed name. The same script on either server
# automatically uses the correct values. No hostname comparisons in scripts.
# ─────────────────────────────────────────────────────────────────────────────
━━━ WHAT RUNS WHEN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The User Scripts plugin runs only orchestrators — never individual scripts directly.
At Startup of Array:
array_start.sh launches all startup scripts in order
→ inotify_tuning.sh raise kernel limits before containers start
→ docker_syslog_filter.sh suppress veth noise before containers create interfaces
→ php_fpm_max_children.sh WebGUI tuning before first request
→ ramdisk_setup.sh create ramdisk before Emby starts
→ docker_network_connect.sh connect containers to extra networks
→ system_watchdog.sh continuous — last resort server stability
→ docker_watchdog.sh continuous — two-tier container healing
→ failover.sh continuous — mutual failover state machine
Every 3 minutes:
transcode_management.sh cleanup → manager (order non-negotiable)
Every 15 minutes:
critical_sync_maintenance.sh Critical-Data + Emby dirty sync + partnership check
Every 30 minutes:
rsync.sh --profile=emby-failover Emby watch state sync (separate cron entry)
Every 6 hours:
arrs_failed_stalled_recovery.sh blocklist + re-search failed imports
1am daily:
daily_sync_maintenance.sh git pull → sync shares → media maintenance → restarts
2:30am Sunday:
weekly_sync_maintenance.sh stop → update images → clean sync → start → weekly restarts
8am daily:
weekly_health_digest.sh DIGEST_PROFILE controls when it actually notifies
Every 6 hours:
system_tuning_monitor.sh inotify + php-fpm utilisation snapshot
Sunday morning block (6–11am):
zfs_memory_snapshot.sh, smart_health.sh, cert_monitor.sh,
backup_verify.sh, emby_session_report.sh, bandwidth_monitor.sh --report
→ Complete schedule: README-Orchestrators.md
━━━ THE DAILY CYCLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
What actually happens on a typical day, from the ecosystem's perspective:
Throughout the day:
system_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
docker_watchdog.sh watching: memory, CPU, HTTP health, required containers
failover.sh watching: remote server, internet connectivity
transcode_management.sh managing: ramdisk ↔ SSD, session cleanup (every 3min)
critical_sync_maintenance.sh keeping: auth stack + Emby current (every 15min)
1am:
git pull (latest scripts first)
rsync Movies, Tv_Shows, Music → HOST2 (HOST1 pushes its truth shares)
rsync Anime_Shows, Anime_Movies → HOST1 (HOST2 pushes its truth shares)
media_shares_permissions.sh (fix ownership before arr cleanup)
media_cleaner.sh anime + media (remove scene junk)
lidarr/sonarr/radarr_cleanup.sh (remove orphaned files)
docker_daily_restart.sh (NPM, Authelia, Dispatcharr fresh start)
2:30am Sunday:
Stop Emby + auth stack both sides
Pull updated Docker images (zero extra downtime — already stopped)
rsync Emby + Critical-Data clean mirror
Start containers on new images
docker_weekly_restart.sh (NextCloud, AdGuard, Immich)
7am Sunday:
sunday_morning_coffee_report.sh (everything that happened this week)
━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Problems are handled at the most appropriate level. Scripts escalate only when the level below them has failed:
Problem: Emby container using 22GB RAM (limit: 18GB)
→ docker_watchdog.sh Tier 1: restart Emby immediately
→ System continues, no reboot
Problem: Authelia crashed and won't restart
→ docker_watchdog.sh Tier 1: attempt restart (dependency check: DB + Redis first)
→ Fails 3 times in 1 hour → skip list → critical notification
→ Tools: watchdog_skip_list_manager.sh to clear after fixing
Problem: RAM drops to 3GB
→ system_watchdog.sh: stop non-essential containers
→ Wait for RAM to recover to 30GB
→ docker_watchdog.sh defers (reads mem_shutdown_active=true)
→ If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot
Problem: Docker daemon hung
→ system_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
→ If still hung: immediate reboot (no strikes needed — daemon can't be managed)
Problem: HOST1 loses power
→ failover.sh on HOST2 detects at next 2-minute check
→ Tier 1 starts immediately: DDNS, Emby, VaultWarden, Live TV
→ Household barely notices. Live TV may buffer for < 2 minutes.
→ Container healing: README-Docker_Essentials.md → System stability: README-Unraid_Essentials.md
━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
If you're setting this up from scratch on two servers:
1. Follow README-Rsync_Setup.md completely — end-to-end setup guide
Tailscale → SSH keys → git clone → master.conf → User Scripts schedule
2. Set your container names correctly before anything else
Shared services: no suffix (Emby, NginxProxyManager, etc.)
Personal services: suffixed (Emby-Gmer4Lfe, Organizer-Gmer4Lfe)
3. Set your share names identically on both servers
/mnt/user/Movies on both, /mnt/user/Tv_Shows on both, etc.
4. Run initial sync to populate HOST2
bash daily_sync_maintenance.sh --log
5. Verify failover works before relying on it
bash failover_test.sh --dry-run first
bash failover_test.sh when ready — uses iptables simulation, no real downtime
━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
| README | What It Covers |
|---|---|
| README-Failover.md | DDNS sequencing, tiered failover, handback, split brain prevention, failover_test.sh |
| README-Docker_Essentials.md | Two-tier container watchdog, memory limits, dependency ordering, skip list, daily/weekly restarts |
| README-Unraid_Essentials.md | Three-tier system watchdog, WebGUI recovery, inotify tuning, PHP-FPM, log hygiene |
| README-Orchestrators.md | What runs when, execution order, daily/weekly windows, adding jobs |
| README-Media.md | Permissions model, junk cleanup, arr orphan removal, safety layers, testing procedure |
| README-Transcoding.md | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
| README-Monitors.md | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
| README-Partnership.md | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
| README-Rsync_Setup.md | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, User Scripts |
| README-Tools.md | failover_state_reset, skip list manager, emby_database_repair, container export |
| README-Unraid_Essentials.md | git_pull_execute.sh, server_reboot.sh, mover_stop.sh, user_scripts_stop.sh |
━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Things that are different from what you might expect:
# ─────────────────────────────────────────────────────────────────────────────
# 1. The User Scripts plugin has very few entries.
# Everything runs through orchestrators. Individual scripts are never
# scheduled directly. array_start.sh is the only "At Startup of Array" entry.
# 2. Scripts are silent when healthy.
# system_watchdog.sh runs 288 times per day. It produces zero output on clean
# cycles. If you see output, something needs attention. If you see nothing,
# the system is healthy. This is by design.
# 3. master.conf is the only file you edit.
# Master_host*.conf for per-server values. Everything else is code — don't
# add configuration into scripts directly. Everything configurable has a var.
# 4. --dry-run works on everything.
# Every single script supports --dry-run. Pre-flight checks still run.
# Review what would happen before anything changes. Always available.
# 5. detect_hosts() does the heavy lifting.
# Scripts never compare hostnames directly. detect_hosts() aliases the correct
# HOST*_ vars based on the running server. The same script runs correctly on
# HOST1 and HOST2 — no branching on server identity inside scripts.
# 6. The Emby Docker mount is unusual and critical.
# --mount type=bind,...,bind-propagation=shared in Extra Parameters — not a
# standard path mapping. Without shared propagation symlink flips are silently
# ignored. README-Transcoding.md explains why in detail.
# 7. arr cleanup scripts can delete files.
# lidarr_cleanup.sh, sonarr_cleanup.sh, radarr_cleanup.sh permanently delete
# orphaned files. Safety layers prevent catastrophic runs but always test with
# --dry-run --log first on a new system. README-Media.md has the full procedure.
# ─────────────────────────────────────────────────────────────────────────────
━━━ PROJECT NAME ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Working name: Arbiter
Repository: git@192.168.50.2:FailedProxy/Unraid_Scripts.git
Install path: /mnt/user/appdata/unraid_scripts