297 lines
15 KiB
Markdown
297 lines
15 KiB
Markdown
# unRAID Script Ecosystem
|
|
|
|
A modular, git-managed automation ecosystem for two unRAID servers. One configuration file. Both servers stay in sync with a single git pull. Everything from daily media syncing to mutual container failover runs automatically — and when something goes wrong, the system tries to fix itself before waking you up.
|
|
|
|
---
|
|
|
|
## The Goal
|
|
|
|
A self-hosted infrastructure that runs itself.
|
|
|
|
```
|
|
Step away → come back to a healthy system
|
|
Something breaks → system self-heals
|
|
Something can't be fixed → you get notified
|
|
Servers stay in sync → one edit propagates everywhere
|
|
```
|
|
|
|
Before this ecosystem existed, the same problems were solved by 60+ standalone scripts across two servers — different coding styles, no shared standards, every change applied twice. This ecosystem standardises everything into one codebase with one config file and one deployment mechanism.
|
|
|
|
---
|
|
|
|
## The Servers
|
|
|
|
```
|
|
HOST1 — unRAID-Gmer4Lfe (Primary)
|
|
Hardware: Threadripper 1950X, 128GB RAM
|
|
Storage: Multiple ZFS pools + cache
|
|
Role: Primary services, full media stack, all Docker containers
|
|
|
|
HOST2 — unRAID-Jayred365 (Secondary)
|
|
Location: Remote — 50 miles from HOST1
|
|
Role: Mirror, failover coverage, independent services
|
|
|
|
Network: Tailscale — encrypted tunnel between both servers
|
|
Repo: Self-hosted Gitea on HOST1
|
|
Deployment: git pull on either server → both stay current
|
|
```
|
|
|
|
---
|
|
|
|
## How It Works
|
|
|
|
Every script sources two files at startup:
|
|
|
|
```bash
|
|
source "$SCRIPT_DIR/../Master.conf" # all user configuration
|
|
source "$SCRIPT_DIR/../common.sh" # shared library
|
|
```
|
|
|
|
**`Master.conf`** is the single source of truth. Container names, thresholds, paths, API keys, rsync profiles, failover tiers — everything configurable lives here. Change a value, push to git, both servers pull — done.
|
|
|
|
**`common.sh`** provides shared functions used by every script — host detection, notifications, rsync helpers, output formatting, icon set. Scripts never duplicate this logic.
|
|
|
|
**`git_pull_execute.sh`** pulls the latest scripts from Gitea and sets executable permissions. Schedule it or run it manually on either server.
|
|
|
|
---
|
|
|
|
## Repository Structure
|
|
|
|
```
|
|
Unraid_Scripts/
|
|
├── Master.conf # All user configuration — edit here only
|
|
├── common.sh # Shared library — used by all scripts
|
|
├── git_pull_execute.sh # Pull latest scripts from Gitea
|
|
├── User_Script_Template.sh # Paste into unRAID User Scripts plugin
|
|
│
|
|
├── Failover/ # Mutual container failover
|
|
│ ├── README-Failover.md
|
|
│ ├── failover.sh # State machine — runs continuously
|
|
│ └── failover_test.sh # Controlled simulation harness
|
|
│
|
|
├── Monitors/ # Health reporting — watch and report only
|
|
│ ├── README-Monitors.md
|
|
│ ├── cert_monitor.sh # SSL certificate expiry
|
|
│ ├── smart_health.sh # Drive SMART attributes
|
|
│ ├── zfs_memory_snapshot.sh # ZFS pool health + memory report
|
|
│ ├── backup_verify.sh # Random checksum verification vs remote
|
|
│ ├── bandwidth_monitor.sh # Rsync transfer history
|
|
│ ├── weekly_health_digest.sh # Aggregated system health summary
|
|
│ └── emby_session_report.sh # Emby usage statistics
|
|
│
|
|
├── Orchestrators/ # Sequential job runners
|
|
│ ├── README-Orchestrators.md
|
|
│ ├── daily_sync.sh # All media shares synced nightly
|
|
│ └── media_management.sh # Permissions + cleaners + arr cleanup
|
|
│
|
|
├── Rsync/ # Core sync engine
|
|
│ ├── README-Rsync.md
|
|
│ ├── rsync.sh # Per-share or per-profile sync
|
|
│ └── README_Rsync_Setup.md # Initial setup guide
|
|
│
|
|
├── Docker_Essentials/ # Container lifecycle management
|
|
│ ├── README-Docker_Essentials.md
|
|
│ ├── docker_watchdog.sh # Two-tier self-healing monitoring
|
|
│ ├── docker_daily_restart.sh # Daily container restarts
|
|
│ ├── docker_weekly_restart.sh # Weekly container restarts
|
|
│ └── docker_network_connect.sh # Extra network connections at boot
|
|
│
|
|
├── Media/ # Media library maintenance
|
|
│ ├── README-Media.md
|
|
│ ├── media_shares_permissions.sh # Recursive permission application
|
|
│ ├── media_cleaner.sh # Junk file removal (anime + media profiles)
|
|
│ ├── lidarr_cleanup.sh # Orphaned music file cleanup
|
|
│ ├── sonarr_cleanup.sh # Orphaned TV file cleanup
|
|
│ └── radarr_cleanup.sh # Orphaned movie file cleanup
|
|
│
|
|
├── Transcodes/ # Emby ramdisk transcode management
|
|
│ ├── README-Transcoding.md
|
|
│ ├── ramdisk_setup.sh # Create ramdisk + symlink at array start
|
|
│ ├── transcode_manager.sh # Monitor usage, manage symlink, display sessions
|
|
│ └── transcode_cleanup.sh # Remove old inactive transcode files
|
|
│
|
|
├── Tools/ # Situational utilities — run when needed
|
|
│ ├── README-Tools.md
|
|
│ ├── recreate_shares.sh # Recreate share dirs after incident
|
|
│ ├── failover_state_reset.sh # Reset failover state file to NORMAL
|
|
│ ├── watchdog_skip_list_manager.sh # Manage container watchdog skip lists
|
|
│ ├── bulk_permissions_repair.sh # Targeted permission repair for one share
|
|
│ ├── container_data_export.sh # Export container appdata to tar archive
|
|
│ ├── emby_database_repair.sh # SQLite integrity check on Emby databases
|
|
│ └── zfs_pool_scrub.sh # Trigger ZFS scrub with completion report
|
|
│
|
|
└── unRAID_Essentials/ # Server-level system management
|
|
├── README-unRAID_Essentials.md
|
|
├── system_watchdog.sh # Last line of defense — controlled reboot
|
|
├── webgui_restart.sh # WebGUI nginx + emhttp auto-restart
|
|
├── docker_syslog_filter.sh # Suppress Docker veth noise from syslog
|
|
├── php_fpm_max_children.sh # WebGUI PHP-FPM concurrency tuning
|
|
├── clear_logs.sh # Weekly system log clearing
|
|
├── mover_stop.sh # Graceful mover termination
|
|
├── rsync_stop.sh # Stop rsync + recover containers
|
|
├── server_reboot.sh # Graceful reboot with user warning
|
|
└── user_script_stop.sh # Stop all running User Scripts jobs
|
|
```
|
|
|
|
---
|
|
|
|
## Folders at a Glance
|
|
|
|
| Folder | What it does | Key scripts |
|
|
|--------|-------------|-------------|
|
|
| **Failover** | Mutual container failover — autonomous, tiered, DDNS-safe | `failover.sh` |
|
|
| **Monitors** | Watch and report — never act, minimal flash writes | `weekly_health_digest.sh` |
|
|
| **Orchestrators** | Sequential job runners with unified reporting | `daily_sync.sh`, `media_management.sh` |
|
|
| **Rsync** | Core sync engine with profile system | `rsync.sh` |
|
|
| **Docker_Essentials** | Two-tier self-healing container management | `docker_watchdog.sh` |
|
|
| **Media** | Permissions, junk cleanup, arr orphan cleanup | `media_cleaner.sh`, arr scripts |
|
|
| **Transcodes** | Ramdisk symlink routing for Emby | `transcode_manager.sh` |
|
|
| **Tools** | Recovery and situational utilities | `failover_state_reset.sh`, `zfs_pool_scrub.sh` |
|
|
| **unRAID_Essentials** | Server-level maintenance and last-resort recovery | `system_watchdog.sh` |
|
|
|
|
---
|
|
|
|
## Key Design Principles
|
|
|
|
**One config file.** `Master.conf` is the only file you edit. No hunting through scripts to change a container name or a threshold.
|
|
|
|
**Bidirectional.** Both servers run identical scripts. `detect_hosts()` determines local vs remote at runtime. One codebase covers both directions.
|
|
|
|
**Self-healing layers.** Problems are addressed at the most targeted level first:
|
|
```
|
|
docker_watchdog.sh — container level, minimal disruption
|
|
system_watchdog.sh — system level, last resort
|
|
failover.sh — infrastructure level, other server covers
|
|
```
|
|
|
|
**Strike systems, not hair triggers.** Single spikes don't cause restarts or reboots. Sustained problems do. The strike system filters noise from genuine issues.
|
|
|
|
**Notifications when action is needed, silence otherwise.** The ecosystem is designed to run without daily attention. You hear from it when something needs human intervention — not as a regular occurrence.
|
|
|
|
**Flash drive friendly.** Scripts that write to `/boot/` use bounded files with auto-purge. `/tmp/` is used for ephemeral state that resets on reboot. Most scripts write nothing at all.
|
|
|
|
**--dry-run everywhere.** Every script supports `--dry-run`. Test before you schedule.
|
|
|
|
---
|
|
|
|
## Scheduling Overview
|
|
|
|
```bash
|
|
# ━━━ At Startup of Array ━━━
|
|
failover.sh # background task — continuous loop
|
|
ramdisk_setup.sh
|
|
docker_syslog_filter.sh
|
|
php_fpm_max_children.sh
|
|
docker_network_connect.sh
|
|
|
|
# ━━━ Frequent ━━━
|
|
*/3 * * * * transcode_manager.sh
|
|
*/5 * * * * transcode_cleanup.sh
|
|
*/10 * * * * webgui_restart.sh
|
|
*/15 * * * * docker_watchdog.sh
|
|
*/15 * * * * system_watchdog.sh
|
|
|
|
# ━━━ Daily ━━━
|
|
0 1 * * * daily_sync.sh
|
|
0 2 * * * media_management.sh
|
|
0 3 * * * docker_daily_restart.sh
|
|
0 8 * * * weekly_health_digest.sh
|
|
|
|
# ━━━ Weekly — Sunday ━━━
|
|
0 3 * * 0 docker_weekly_restart.sh
|
|
0 5 * * 0 clear_logs.sh
|
|
0 6 * * 0 zfs_memory_snapshot.sh
|
|
0 7 * * 0 smart_health.sh
|
|
0 9 * * 0 cert_monitor.sh
|
|
0 10 * * 0 backup_verify.sh
|
|
0 11 * * 0 emby_session_report.sh
|
|
0 11 * * 0 bandwidth_monitor.sh --report
|
|
```
|
|
|
|
---
|
|
|
|
## Folder READMEs
|
|
|
|
Each folder has a detailed README covering setup, configuration, usage, and the reasoning behind design decisions.
|
|
|
|
| README | Contents |
|
|
|--------|----------|
|
|
| [README-Failover.md](Failover/README-Failover.md) | DDNS split brain prevention, tiered failover, handback sequence, initial setup, troubleshooting |
|
|
| [README-Monitors.md](Monitors/README-Monitors.md) | All monitor scripts, flash drive write policy, scheduling |
|
|
| [README-Orchestrators.md](Orchestrators/README-Orchestrators.md) | Why orchestrators exist, job ordering, how to add jobs |
|
|
| [README-Rsync.md](Rsync/README-Rsync.md) | Profile system, SSH key setup, Tailscale requirements |
|
|
| [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md) | Two-tier watchdog, startup grace, dependency ordering, skip list management |
|
|
| [README-Media.md](Media/README-Media.md) | Execution order, cleaner profiles, arr cleanup safety procedure |
|
|
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Symlink indirection design, Docker mount warning, mode switching, sizing |
|
|
| [README-Tools.md](Tools/README-Tools.md) | All utility scripts, when to use each, how to add new tools |
|
|
| [README-unRAID_Essentials.md](unRAID_Essentials/README-unRAID_Essentials.md) | system_watchdog tiers, startup sequence, scheduled maintenance |
|
|
|
|
---
|
|
|
|
## v2 Roadmap
|
|
|
|
The v1 ecosystem is production-proven and stable. These are the features planned for v2 — some require additional infrastructure, some are extensions of existing systems.
|
|
|
|
### Transcode Manager — Advanced Mode
|
|
|
|
The current `smart` / `ramdisk` / `ssd` modes route all sessions to one location. v2 adds an `advanced` mode that routes by media type:
|
|
|
|
```bash
|
|
TRANSCODE_MANAGER_MODE="advanced"
|
|
|
|
TRANSCODE_FORCE_RAMDISK=(
|
|
"LiveTv" # always ramdisk — buffering is latency sensitive
|
|
)
|
|
TRANSCODE_FORCE_SSD=(
|
|
"Audio" # music/downloads — no benefit from ramdisk
|
|
)
|
|
# Everything else follows smart threshold behavior
|
|
```
|
|
|
|
The groundwork is already in place — session display with media type parsing is working. Advanced mode requires Emby to expose media type at the point ffmpeg resolves the symlink, which means the routing decision needs to happen before the session starts. This is the problem to solve.
|
|
|
|
### Plugin Dashboard
|
|
|
|
The Monitors folder is already the data backend for a future unRAID plugin dashboard. Every monitor script writes structured state that a plugin could read and display:
|
|
|
|
```
|
|
Failover state → live status indicator
|
|
Container watchdog → strike counts, skip list
|
|
SMART health → per-drive status
|
|
ZFS pools → health + ARC utilization
|
|
Transcode sessions → live session display
|
|
Bandwidth history → transfer trend charts
|
|
Certificate expiry → days remaining per domain
|
|
```
|
|
|
|
The scripts exist. The data exists. The plugin is the frontend.
|
|
|
|
### Failover — Tiered by Content Type
|
|
|
|
Currently failover starts containers based on time elapsed. A future enhancement would start containers based on what the primary server was doing before it went down:
|
|
|
|
```
|
|
Primary running Live TV sessions → start Live TV stack on secondary immediately
|
|
Primary idle → standard tier delays apply
|
|
Primary in heavy transcoding → start Emby immediately, defer others
|
|
```
|
|
|
|
This requires the failover state file to track active session types, which requires integration with the Emby API at failover trigger time.
|
|
|
|
### Health Digest — Plugin Integration
|
|
|
|
Currently the digest sends a notification. In v2 it populates a persistent dashboard that shows a rolling week of system health at a glance — without requiring a notification for every event.
|
|
|
|
### Container Health Checks — Standardised Library
|
|
|
|
The docker_watchdog already has HTTP check support per container. A v2 enhancement is a standardised health check library — pre-built check commands for every container in the stack that can be applied via Extra Parameters in the unRAID template without manual research per container.
|
|
|
|
---
|
|
|
|
## Origin
|
|
|
|
This ecosystem grew from a single failover script — written as a first real bash project, refined through months of production use, redesigned multiple times as the stack grew more complex. Each script in the collection started as a one-off solution to a specific problem. Over time the patterns that worked were extracted into `common.sh`, the configuration was centralised into `Master.conf`, and the whole collection was standardised into what it is now.
|
|
|
|
The failover script that started it all is still the soul of the ecosystem — the DDNS sequencing, the two-ping state machine, the handback order. Everything else built on top of that foundation.
|
|
|
|
Two servers, one codebase, self-healing infrastructure. Step away. Come back to a happy system. |