From 8c5667b1dfec3a9caeaab53da2bbfaf201206933 Mon Sep 17 00:00:00 2001 From: FailedProxy Date: Sat, 18 Apr 2026 15:40:41 -0400 Subject: [PATCH] added all ai generated Readme files, added the last of the tools scripts --- Docker_Essentials/README-Docker_Essentials.md | 349 ++++++++++++++ Failover/README-Failover.md | 455 ++++++++++++++++++ Media/README-Media.md | 316 ++++++++++++ Monitors/README-Monitors.md | 376 +++++++++++++++ Orchestrators/README-orchestrators.md | 216 +++++++++ README.md | 299 +++++++++++- ...E_Rsync_Setup.md => README-Rsync_Setup.md} | 0 Tools/README-Tools.md | 262 ++++++++++ Tools/bulk_permissions_repair.sh | 133 +++++ Tools/container_data_export.sh | 173 +++++++ Tools/emby_database_repair.sh | 214 ++++++++ Tools/failover_state_reset.sh | 127 +++++ Tools/watchdog_skip_list_manager.sh | 175 +++++++ Tools/zfs_pool_scrub.sh | 193 ++++++++ Transcodes/README-Transcoding.md | 317 ++++++++++++ unRAID_Essentials/README-Unraid_Essentials.md | 319 ++++++++++++ 16 files changed, 3921 insertions(+), 3 deletions(-) create mode 100644 Docker_Essentials/README-Docker_Essentials.md create mode 100644 Failover/README-Failover.md create mode 100644 Media/README-Media.md create mode 100644 Monitors/README-Monitors.md create mode 100644 Orchestrators/README-orchestrators.md rename Rsync/{README_Rsync_Setup.md => README-Rsync_Setup.md} (100%) create mode 100644 Tools/README-Tools.md create mode 100644 Tools/bulk_permissions_repair.sh create mode 100644 Tools/container_data_export.sh create mode 100644 Tools/emby_database_repair.sh create mode 100644 Tools/failover_state_reset.sh create mode 100644 Tools/watchdog_skip_list_manager.sh create mode 100644 Tools/zfs_pool_scrub.sh create mode 100644 Transcodes/README-Transcoding.md create mode 100644 unRAID_Essentials/README-Unraid_Essentials.md diff --git a/Docker_Essentials/README-Docker_Essentials.md b/Docker_Essentials/README-Docker_Essentials.md new file mode 100644 index 0000000..602b01d --- /dev/null +++ b/Docker_Essentials/README-Docker_Essentials.md @@ -0,0 +1,349 @@ +# Docker Essentials + +Container lifecycle management — health monitoring, scheduled restarts, and network configuration. These scripts keep your Docker stack healthy, fresh, and correctly connected without manual intervention. + +``` +Monitors/ — observes containers, reports issues +Docker_Essentials/ — acts on containers (this folder) +unRAID_Essentials/ — acts on the server itself +``` + +--- + +## Scripts + +### `docker_watchdog.sh` + +**The self-healing container monitoring system.** Two tiers of monitoring that work together to keep every container in the stack healthy — from strict per-container thresholds down to global health scanning of everything that's running. + +```bash +# Scheduled as: */15 * * * * (every 15 minutes) +/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh +``` + +--- + +#### Tier 1 — Strict Monitoring + +Explicitly configured containers with per-container thresholds. Every container you care about most lives here. + +**Memory hard limits:** + +```bash +declare -A WATCHDOG_CONTAINERS=( + ["Emby"]=16384 # 16GB hard limit — immediate restart if exceeded + ["LidaTube"]=6144 # 6GB + ["Tdarr"]=6144 + ["Code-Server"]=1024 +) +``` + +Memory is checked against the configured limit in MB. If a container exceeds its hard limit it is restarted immediately — no strike system, no waiting. Memory leaks are real and immediate action is right. + +A soft threshold (`SOFT_MEM_THRESHOLD=80`) warns when a container reaches 80% of its hard limit — useful for spotting gradual leaks before they become problems. + +**CPU thresholds:** + +CPU is normalised against total core count automatically. A container using 85% of one core on a 16-core system is ~5.3% normalised — not a problem. 85% normalised on a 16-core system means 13.6 cores worth of CPU — that's a problem. + +The strike system prevents restarts on brief spikes: +``` +CPU above HARD_CPU_THRESHOLD → strike 1 +CPU above HARD_CPU_THRESHOLD next cycle → strike 2 → restart +CPU recovers → strike count resets +``` + +**HTTP responsiveness:** + +```bash +declare -A WATCHDOG_CONTAINER_URLS=( + ["Emby"]="http://localhost:8096" +) +``` + +Containers with configured URLs are checked via `curl`. If the endpoint doesn't respond within `CURL_TIMEOUT` seconds that's a strike. Two consecutive failures trigger a restart. A container can be running and appear healthy to Docker while its application layer is frozen — HTTP checks catch this. + +**Required containers:** + +```bash +WATCHDOG_REQUIRED_CONTAINERS=( + "NginxProxyManager" + "Lldap-Gmer4Lfe" + "Authelia" + "Mariadb-Authelia" + "Redis-Authelia" + "Authelia-Secondary" + "Redis-Authelia-Secondary" +) +``` + +These must always be running. If any are found stopped, the watchdog attempts to restart them. Strike system applies — persistent failures get added to the skip list. + +--- + +#### Tier 2 — Global Health Scan + +Scans every running container for health issues — catches anything not explicitly configured in Tier 1. + +| Check | Trigger | Action | +|-------|---------|--------| +| `WATCHDOG_RESTART_UNHEALTHY` | Docker HEALTHCHECK reports `unhealthy` | Restart | +| `WATCHDOG_NOTIFY_OOM` | Kernel OOM-killed the container | Restart + notify | +| `WATCHDOG_NOTIFY_CRASHLOOP` | Docker RestartCount climbing | Notify (critical above `WATCHDOG_CRASH_LIMIT`) | +| `WATCHDOG_RESTART_DEAD` | Container in dead state | Remove + restart | +| `WATCHDOG_RESTART_CRASHED` | Non-zero exit code | Restart | + +Each check is independently toggleable — disable checks that cause false positives in your environment. + +> **Note on HEALTHCHECK:** Only containers with a `HEALTHCHECK` instruction defined in their Docker image report health status. Containers without one are invisible to the unhealthy check but still caught by crash, dead, and OOM checks. You can add custom HEALTHCHECKs via unRAID's Extra Parameters field — see the health check guide for your specific containers. + +--- + +#### Cross-cutting Intelligence + +These apply to **both tiers** on every watchdog run: + +**Startup grace period:** +```bash +WATCHDOG_STARTUP_GRACE=600 # seconds after boot +``` +For the first 10 minutes after array start, checks run but restarts are suppressed. Containers need time to come up — false positives during boot are common without this. Checks still run and report so you can see what's happening, but no restarts fire. + +**Dependency ordering:** +```bash +declare -A WATCHDOG_DEPENDENCIES=( + ["Authelia"]="Mariadb-Authelia Redis-Authelia" + ["Authelia-Secondary"]="Mariadb-Authelia Redis-Authelia-Secondary" + ["NextCloud"]="Postgres-NextCloud" +) +``` +If a container's dependency is also down, the dependent is skipped this cycle. The dependency gets restarted first. On the next cycle — once the database is up and accepting connections — the dependent container gets restarted. This prevents the classic failure mode where Authelia is restarted before its database is ready and fails immediately, triggering another restart attempt. + +**Restart loop protection:** +```bash +WATCHDOG_CONTAINER_RESTART_LIMIT=3 +WATCHDOG_CONTAINER_RESTART_WINDOW=1 # hours +WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db" +``` +If the watchdog restarts the same container 3 times within 1 hour, that container is added to the persistent skip list. Something is genuinely broken that restarts are not fixing — continued hammering wastes resources and masks the real problem. A critical notification is sent when a container hits the skip list. + +The skip list lives on `/boot/` — it survives reboots. The container stays on the skip list until it is found running again (manually fixed or recovered after a reboot), at which point it's automatically removed and the restart history is cleared. + +**Notification batching:** +```bash +WATCHDOG_BATCH_NOTIFY=true +``` +All events from a single watchdog run are collected and sent as one notification at the end. On a system with 50+ containers, individual per-event notifications during a problem cascade would be unmanageable. One clean summary tells you what happened without flooding your notification channel. + +--- + +#### Skip List Management + +The skip list (`/boot/config/system_watchdog_failed.db`) is the persistent memory of containers that have exhausted restart attempts. + +```bash +# View current skip list +cat /boot/config/system_watchdog_failed.db + +# A container auto-removes itself when found running again +# To manually clear a specific container: +sed -i '/ContainerName/d' /boot/config/system_watchdog_failed.db + +# To clear the entire skip list: +> /boot/config/system_watchdog_failed.db +``` + +Also clear the restart history when manually fixing a container: +```bash +sed -i '/ContainerName|/d' /boot/config/container_restart_history.db +``` + +--- + +#### State Files + +| File | Location | Resets | Purpose | +|------|----------|--------|---------| +| `container_watchdog_state.db` | `/tmp/` | On reboot | Strike counts for all containers | +| `system_watchdog_failed.db` | `/boot/config/` | Never (manual) | Persistent skip list | +| `container_restart_history.db` | `/boot/config/` | Auto-purge after window | Restart loop detection | + +--- + +### `docker_daily_restart.sh` + +Restarts configured containers every day. + +```bash +# Scheduled as: 0 3 * * * (3am daily) +/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh +``` + +**Why daily restarts:** + +Some containers accumulate memory over time — connection pools that don't shrink, caches that grow without bound, log buffers that don't rotate. A daily restart clears all of this. It's simpler and more reliable than trying to tune every container's internal memory management. + +Containers that benefit from daily restarts are typically those handling lots of short-lived connections — reverse proxies, auth servers, and live TV schedulers. + +```bash +DAILY_RESTART_CONTAINERS=( + "NginxProxyManager" # connection pool accumulation + "Authelia" # session and token cache + "Dispatcharr" # live TV connection management + "Dispatcharr-Basic" + "Dispatcharr-Iptv-Users" + "ErsatzTV-Emby" # channel scheduling state +) +``` + +**Retry logic:** Uses `RETRY_COUNT` and `SLEEP` from `Master.conf`. If a container fails to restart it retries before marking it as failed and notifying. + +--- + +### `docker_weekly_restart.sh` + +Restarts configured containers once per week. + +```bash +# Scheduled as: 0 3 * * 0 (Sunday 3am weekly) +/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh +``` + +For less critical services that benefit from periodic restarts but don't need daily cycling. Typically productivity and self-hosted application containers that are stable but benefit from a clean weekly slate. + +```bash +WEEKLY_RESTART_CONTAINERS=( + "NextCloud" + "Organizrv2-Gmer4Lfe" + "AdGuard-Home" + "Immich-Gmer4Lfe" +) +``` + +Sunday morning is the natural maintenance window — it runs alongside the weekly log clear, ZFS snapshot, SMART check and backup verify. Everything happens while load is lowest. + +--- + +### `docker_network_connect.sh` + +Connects containers to extra Docker networks on array start. + +```bash +# Scheduled as: At Startup of Array +/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_network_connect.sh +``` + +**The problem it solves:** + +Docker containers are assigned networks at creation time via the unRAID template. Sometimes containers need to communicate with containers on a different network that wasn't configured in the original template — for example, `memcached` needing to talk to the `nextcloud-aio` network so NextCloud can use it for caching. + +The correct solution is to add the network in the template. But some containers are created by other containers (like `nextcloud-aio`) and their network assignments can't easily be changed. This script handles those edge cases at array start. + +Every container in `NETWORK_CONNECT_CONTAINERS` is connected to every network in `NETWORK_CONNECT_NETWORKS` — many-to-many. Already-connected containers are skipped cleanly — safe to run multiple times. + +```bash +NETWORK_CONNECT_CONTAINERS=( + "memcached" + "Npm-CrowdSec" +) + +NETWORK_CONNECT_NETWORKS=( + "nextcloud-aio" # Docker network name — must exist before array start +) +``` + +**Note:** The target network must exist before this script runs. Networks created by Docker Compose or the nextcloud-aio stack are created when their containers start — if those containers start after this script, the connection will fail. The unRAID User Scripts plugin "At Startup of Array" timing usually handles this correctly but be aware of the dependency. + +--- + +## Relationship Between Scripts + +``` +docker_network_connect.sh — runs once at array start + ↓ +docker_watchdog.sh — runs every 15 minutes + ├── Tier 1: strict per-container monitoring + └── Tier 2: global health scan of everything + +docker_daily_restart.sh — runs at 3am every day +docker_weekly_restart.sh — runs at 3am every Sunday +``` + +The watchdog is the continuous monitor. The restart scripts are the scheduled maintenance. Together they cover both reactive healing (watchdog) and proactive freshness (restarts). + +--- + +## Adding a New Container to the Watchdog + +**Tier 1 — memory monitoring:** +```bash +# Add to WATCHDOG_CONTAINERS in Master.conf +declare -A WATCHDOG_CONTAINERS=( + ["Emby"]=16384 + ["MyNewContainer"]=2048 # 2GB hard limit +) +``` + +**Tier 1 — HTTP check:** +```bash +declare -A WATCHDOG_CONTAINER_URLS=( + ["Emby"]="http://localhost:8096" + ["MyNewContainer"]="http://localhost:9000/health" +) +``` + +**Tier 1 — required container:** +```bash +WATCHDOG_REQUIRED_CONTAINERS=( + "NginxProxyManager" + "MyNewContainer" # must always be running +) +``` + +**Tier 2 — dependency:** +```bash +declare -A WATCHDOG_DEPENDENCIES=( + ["Authelia"]="Mariadb-Authelia Redis-Authelia" + ["MyNewContainer"]="its-database-container" # restart db first +) +``` + +**Tier 2 — ignore in global scan:** +```bash +WATCHDOG_SCAN_IGNORE=( + "intentionally-stopped-container" # skip this in global scan +) +``` + +--- + +## Scheduled Summary + +```bash +# At Startup of Array +docker_network_connect.sh + +# Every 15 minutes +*/15 * * * * docker_watchdog.sh + +# Daily — 3am +0 3 * * * docker_daily_restart.sh + +# Weekly — Sunday 3am +0 3 * * 0 docker_weekly_restart.sh +``` + +--- + +## --dry-run Support + +All scripts support `--dry-run`. Always test before scheduling: + +```bash +/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --dry-run +/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh --dry-run +/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh --dry-run +/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_network_connect.sh --dry-run +``` + +`docker_watchdog.sh --status` shows current strike counts, skip list contents, and grace period status without running any checks. \ No newline at end of file diff --git a/Failover/README-Failover.md b/Failover/README-Failover.md new file mode 100644 index 0000000..f824438 --- /dev/null +++ b/Failover/README-Failover.md @@ -0,0 +1,455 @@ +# Failover + +Mutual container failover between two unRAID servers. When one server goes down, the other automatically starts its containers. When it comes back, everything hands back cleanly with data synced and DNS cutting over at exactly the right moment. + +> This system was built from scratch and refined through a year of production testing before being standardised into this ecosystem. The DDNS sequencing and handback order were the hardest parts to get right — the logic is documented here so it's never lost. + +--- + +## The Setup + +Two fully independent unRAID servers connected via Tailscale: + +``` +HOST1 — unRAID-Gmer4Lfe (Primary) + Location: Local + Owns: Gmer4Lfe.com DDNS + Runs: Full service stack + +HOST2 — unRAID-Jayred365 (Secondary / Buddy server) + Location: Remote — 50 miles away + Owns: Gmer4Lfe.us DDNS + Runs: Its own service stack + mirrors HOST1 data +``` + +Both servers run `failover.sh` as a background task continuously. Neither server knows what the other is doing — they only know what they can ping from their own network perspective. + +--- + +## How It Works + +Every `FAILOVER_CHECK_INTERVAL` seconds (default: 120s) each server makes two pings: + +``` +1. Ping remote server (Tailscale IP) +2. Ping internet (8.8.8.8) +``` + +The combination of those two results determines the current state and what action to take. That's it. No SSH signaling between servers, no shared state files, no coordination — pure autonomous decision making based on observable facts. + +--- + +## States + +### NORMAL + +``` +Remote: reachable Internet: reachable +``` + +Both servers running normally. Each server runs its own containers. DDNS on — pointing DNS at this server's IP. Silent operation. + +### FAILOVER + +``` +Remote: unreachable Internet: reachable +``` + +The remote server is down but this server has internet. Start the remote server's containers locally. The remote's DDNS container is started first — DNS starts pointing at this server immediately. Failover is **additive** — your own containers keep running, remote containers are added on top. + +### NO_INTERNET + +``` +Internet: unreachable (remote state unknown) +``` + +This server has lost internet. Stop own DDNS immediately — no point updating DNS records when you can't reach the outside world and it would send conflicting updates. Do not start remote containers — there's no internet to serve them on. Wait for recovery. + +### DARK + +``` +Remote: unreachable Internet: unreachable +``` + +Both pings fail. Same actions as NO_INTERNET — can't determine if remote is truly down or just unreachable through the same outage affecting your internet. Conservative approach: stop DDNS, wait. + +--- + +## DDNS — The Critical Part + +**This took a year to get right. Do not change the sequencing.** + +Each server owns one DDNS container. The script controls when each DDNS runs — the network state never auto-starts DDNS. + +``` +NORMAL: + HOST1 DDNS (Gmer4Lfe.com) → ON — always on while HOST1 has internet + HOST2 DDNS (Gmer4Lfe.us) → ON — always on while HOST2 has internet + +HOST1 loses internet: + HOST1 DDNS → OFF — immediately stopped + HOST2 DDNS → stays ON (unaffected) + +HOST2 detects HOST1 is down: + HOST1 DDNS (on HOST2) → ON — HOST2 starts it as Tier 1 action + DNS now points at HOST2's IP + +HOST1 returns — handback: + HOST1 DDNS (on HOST2) → OFF — stopped FIRST before anything else + ... rsync runs ... + HOST1 containers start on HOST1 + HOST1 DDNS (on HOST1) → ON — started LAST after containers confirmed up +``` + +**Why DDNS never auto-starts on internet return:** + +If HOST1 lost internet and its DDNS auto-started when internet returned, you'd have both HOST1 and HOST2 running the same DDNS simultaneously — pointing DNS at two different IPs at the same time. DNS TTL is 1 minute — users would get routed randomly between servers. This is called split brain and it causes exactly the kind of intermittent failures that are hard to diagnose. + +The script is the sole authority over when DDNS starts. Network state coming back is not permission to start DDNS. Only the completion of the full handback sequence is. + +**One domain per server, one DDNS active per domain, always:** + +``` +Gmer4Lfe.com → runs on HOST1 normally, moves to HOST2 during HOST1 outage +Gmer4Lfe.us → runs on HOST2 normally, moves to HOST1 during HOST2 outage +``` + +--- + +## Tiered Failover + +Not every service needs to start immediately when the other server goes down. Starting the full stack on a secondary server wastes resources for short outages — most are resolved in minutes. + +``` +Tier 1 — Immediate (0 min) + Remote DDNS ← DNS coverage first + Emby ← media server, people are watching + NginxProxyManager ← reverse proxy, everything routes through this + Lldap, Mariadb, Redis ← auth stack, required by everything proxied + Authelia x2 ← authentication + VaultWarden ← passwords, needed immediately + Dispatcharr x3 ← Live TV, people are watching right now + ErsatzTV ← Live TV scheduling + +Tier 2 — After HOST1_TIER2_DELAY minutes (default: 120min) + NextCloud + Postgres ← file access + Immich + PostgreSQL ← photos + Jellyseerr ← media requests + +Tier 3 — After HOST1_TIER3_DELAY minutes (default: 360min) + Organizrv2 ← dashboard + AdGuard-Home ← DNS filtering + UptimeKuma ← monitoring + Gitea ← git server + Collabora-CODE ← document editing + +Tier 4 — After HOST1_TIER4_DELAY minutes (default: 1080min / 18hr) + Full arr stack ← Sonarr, Radarr, Lidarr, Prowlarr etc. + Downloaders ← SABnzbd, Qbittorrent, LidaTube, Pinchflat + 24hr+ outage = full workflow continuity +``` + +**Live TV is Tier 1 because people are watching.** You cannot tell a household mid-game that their live TV will be back in 2 hours. + +**Arrs and downloaders are Tier 4** because they generate significant I/O and have minimal writeback on handback — if HOST1 comes back before 18 hours, the arrs never started on HOST2 and there's nothing to sync back. + +--- + +## Handback Sequence + +When HOST1 returns after being down, the handback must happen in exactly this order: + +``` +1. Strike confirmation + — FAILOVER_HANDBACK_STRIKES consecutive remote-up checks + — prevents handing back during a brief network blip + — 2 strikes × 120s = 4 minute confirmation window + +2. Pre-flight checks + — Remote array is started + — Remote Docker daemon is responding + — Remote rootfs is not nearly full + — Abort if any check fails — retry next cycle + +3. Stop remote DDNS FIRST + — DNS stops updating before anything moves + — Prevents split brain during the transition window + — This is the most critical ordering step + +4. Stop remote containers + — Clean state before rsync + — No competing writes during transfer + — Containers are only down during the rsync window + — This minimises user disruption + +5. Rsync writeback + — Full bandwidth available — DDNS stopped, containers stopped + — Only critical data synced back: + appdata-Failover/Critical-Data (auth stack) + appdata-Failover/Important-Data (NextCloud + Postgres) + appdata-Failover/Emby (userdata, playstates) + appdata-Failover/Gmer4Lfe (server appdata) + — Media files skipped — already on HOST1, never moved + — Downloads skipped — start fresh is cleaner + +6. Start local containers + — Dependencies respected — databases before apps + — Brief pause to let databases initialise before dependents start + +7. Start local DDNS LAST + — DNS only cuts back after containers are confirmed up + — Users hit HOST1 only after it's actually ready to serve them + +8. Return to NORMAL + — State file reset + — Tier flags cleared + — Next cycle confirms everything is healthy +``` + +**Why containers stop before rsync:** + +Earlier versions synced while containers were still running on the remote. This caused: +- Rsync competing with active container I/O — slower transfers +- Files changing mid-transfer — potential inconsistency +- Database writes during sync — dirty state on handback + +Stopping containers first means rsync gets a clean static source at full bandwidth. The window where containers are down is the rsync duration only — typically minutes. + +--- + +## Mutual Failover — Both Directions + +The same script handles both directions. `detect_hosts()` in `common.sh` determines which server is local and which is remote at runtime, then selects the correct arrays from `Master.conf`. + +``` +HOST2 covers HOST1 (HOST1 goes down): + Uses: FAILOVER_HOST2_RUNS_FOR_HOST1_* arrays + Tiers configured by: HOST1_TIER*_DELAY variables + +HOST1 covers HOST2 (HOST2 goes down): + Uses: FAILOVER_HOST1_RUNS_FOR_HOST2_* arrays + Tiers configured by: HOST2_TIER*_DELAY variables +``` + +Both servers run identical scripts. The configuration in `Master.conf` controls what each server does for the other. + +--- + +## Configuration + +All configuration in `Master.conf` under the `── FAILOVER ──` section. + +```bash +# Core timing +EXTERNAL_IP="8.8.8.8" # internet ping target +FAILOVER_CHECK_INTERVAL=120 # seconds between checks +FAILOVER_HANDBACK_STRIKES=2 # confirmations before handback +FAILOVER_STATE_FILE="/boot/config/failover_state.db" + +# DDNS ownership — one per server, script controlled exclusively +HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com") +HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us") + +# What HOST2 runs for HOST1 (tiered) +FAILOVER_HOST2_RUNS_FOR_HOST1_IMMEDIATE=(...) +FAILOVER_HOST2_RUNS_FOR_HOST1_2HR=(...) +FAILOVER_HOST2_RUNS_FOR_HOST1_6HR=(...) +FAILOVER_HOST2_RUNS_FOR_HOST1_18HR=(...) + +# What HOST1 runs for HOST2 (tiered) +FAILOVER_HOST1_RUNS_FOR_HOST2_IMMEDIATE=(...) +# ... + +# Tier delays — configurable per host, in minutes +HOST1_TIER2_DELAY=120 +HOST1_TIER3_DELAY=360 +HOST1_TIER4_DELAY=1080 +HOST2_TIER2_DELAY=120 +HOST2_TIER3_DELAY=360 +HOST2_TIER4_DELAY=1080 + +# Writeback jobs on handback +FAILOVER_HOST1_WRITEBACK=( + "/mnt/user/appdata-Failover/Critical-Data" + "/mnt/user/appdata-Failover/Important-Data" + "/mnt/user/appdata-Failover/Emby" + "/mnt/user/appdata-Failover/Gmer4Lfe" +) +``` + +--- + +## Initial Setup Requirements + +Before `failover.sh` can run on both servers: + +**1. Tailscale connected on both servers** +```bash +# Verify on HOST1 +tailscale ip -4 unRAID-Jayred365 # should return HOST2's Tailscale IP + +# Verify on HOST2 +tailscale ip -4 unRAID-Gmer4Lfe # should return HOST1's Tailscale IP +``` + +**2. SSH keys configured** + +HOST1 must be able to SSH to HOST2 without a password, and vice versa: +```bash +# From HOST1 +ssh -i /root/.ssh/Gmer4Lfe-rsync-key root@[HOST2-tailscale-ip] "hostname" + +# From HOST2 +ssh -i /root/.ssh/Jayred365-rsync-key root@[HOST1-tailscale-ip] "hostname" +``` + +**3. Container names match** + +Failover containers must exist on the server that will start them. If HOST2 starts `Emby` for HOST1, the `Emby` container must be created (but stopped) on HOST2 with its volume mounts pointing to the mirrored data. + +**4. Data mirrored** + +Critical appdata synced to the remote server before failover is needed — not after. The daily rsync profiles keep this current: +``` +appdata-Failover/Critical-Data → auth stack +appdata-Failover/Important-Data → NextCloud + Postgres +appdata-Failover/Emby → Emby userdata +appdata-Failover/Gmer4Lfe → server appdata +``` + +**5. DDNS TTL set to 1 minute** + +In your DDNS provider settings. Higher TTL means users continue hitting the old IP for longer after failover. 1 minute is the minimum most providers allow — it means worst-case 1 minute of disruption. + +**6. Both servers running failover.sh** + +Both servers must be running the script simultaneously. Failover only works in one direction if only one server is running it. + +--- + +## Scripts + +### `failover.sh` + +The main state machine. Run as a background task at array start on both servers. + +```bash +# Scheduled as: At Startup of Array (Background Script) +/mnt/user/appdata/unraid_scripts/Failover/failover.sh + +# Check current state without restarting the loop +/mnt/user/appdata/unraid_scripts/Failover/failover.sh --status + +# Test logic without touching containers +/mnt/user/appdata/unraid_scripts/Failover/failover.sh --dry-run --log +``` + +**To stop:** Click Abort in the User Scripts plugin. Do NOT kill the process directly — the state file may be left inconsistent. Use `Tools/failover_state_reset.sh` to recover from a stuck state. + +--- + +### `failover_test.sh` + +Controlled simulation of the full failover lifecycle. Validates everything works before you need it. + +```bash +# Always dry run first +/mnt/user/appdata/unraid_scripts/Failover/failover_test.sh --dry-run + +# Live test — run during maintenance window +/mnt/user/appdata/unraid_scripts/Failover/failover_test.sh +``` + +**What it does:** + +1. Verifies both servers reachable and state is NORMAL +2. Adds iptables rule blocking all traffic to remote IP +3. Waits `FAILOVER_TEST_BLOCK_WAIT` seconds for `failover.sh` to detect outage +4. Verifies FAILOVER state and Tier 1 containers started +5. Removes iptables rule — remote becomes reachable again +6. Waits `FAILOVER_TEST_HANDBACK_WAIT` seconds for handback +7. Verifies containers returned and state is NORMAL +8. Full pass/fail report per phase + +**Safety trap:** The iptables rule is removed via `trap` on ANY exit — crash, error, ctrl-c, or normal completion. Remote connectivity is always restored regardless of test outcome. + +**⚠️ Run during a maintenance window.** Real containers start and stop during the test — users will experience a brief interruption. Schedule it for 3am or a quiet period. + +**Timing configuration:** +```bash +FAILOVER_TEST_BLOCK_WAIT=150 # must be > FAILOVER_CHECK_INTERVAL + buffer +FAILOVER_TEST_HANDBACK_WAIT=360 # covers strikes × interval + rsync time +``` + +--- + +## State File + +The state file at `/boot/config/failover_state.db` persists across reboots — it's on `/boot/` not `/tmp/`. This means the script remembers what state it was in before a reboot and can resume correctly. + +``` +state=NORMAL +failover_start=0 +handback_strikes=0 +tier2_started=false +tier3_started=false +tier4_started=false +last_reset=2026-04-14 03:00:00 +``` + +If the state file gets stuck in a non-NORMAL state after testing or a failed handback, use `Tools/failover_state_reset.sh` to reset it manually after verifying both servers are in their correct states. + +--- + +## Monitoring + +`Monitors/weekly_health_digest.sh` reads the failover state file and includes it in the weekly digest. If `DIGEST_SMART_ON_FAILOVER=true` and the state is not NORMAL, the digest sends a notification even in smart mode. + +`failover.sh --status` gives an instant snapshot of current state, active tier flags, outage duration, and handback strike count. + +--- + +## What Gets Written Back on Handback + +| Data | Written back | Reason | +|------|-------------|--------| +| Emby userdata / playstates | ✅ Yes | Small, important — watch history | +| Auth stack data | ✅ Yes | Authelia sessions, LLDAP data | +| NextCloud data | ✅ Yes | File changes during outage | +| Emby metadata | ✅ Yes | Any metadata scraped during outage | +| Media files | ❌ No | Already on HOST1, never moved | +| Downloads | ❌ No | Start fresh — cleaner than partial state | +| Arr databases | ❌ No (if under 18hr) | Arrs never started, nothing to sync | + +The minimal writeback is intentional and by design. The goal is to get HOST1 back to the state it was in before the outage plus the delta of what changed during it — not to sync everything. + +--- + +## Troubleshooting + +**Failover not triggering:** +- Is `failover.sh` running on HOST2? Check User Scripts plugin +- Is HOST2's Tailscale connected and can it ping HOST1? +- Check the state file — what state is HOST2 in? + +**Handback not completing:** +- Is HOST1's array fully started? +- Is Docker responding on HOST1? `docker ps` should work +- Is HOST1's rootfs below `ROOTFS_WARN`? +- Check rsync writeback jobs — a stalled rsync blocks handback + +**DDNS not cutting over:** +- Check the DDNS container is actually running on the covering server +- Check DNS TTL — if set high users won't see the cutover for a while +- Check your DDNS provider — are updates being accepted? + +**State file stuck:** +- Use `Tools/failover_state_reset.sh --status` to see what's in it +- Verify both servers manually — right containers on right server, DDNS correct +- Run `Tools/failover_state_reset.sh` to reset + +**Split brain (both DDNS running):** +- This should not happen if the handback sequence is followed +- Check both servers — one should have DDNS stopped +- Manually stop the duplicate DDNS container +- Reset the state file and restart `failover.sh` \ No newline at end of file diff --git a/Media/README-Media.md b/Media/README-Media.md new file mode 100644 index 0000000..ff6e745 --- /dev/null +++ b/Media/README-Media.md @@ -0,0 +1,316 @@ +# Media + +Scripts that maintain the health, cleanliness, and consistency of your media library. Permissions, junk file removal, and orphaned file cleanup across Lidarr, Sonarr, and Radarr. + +These scripts are run sequentially by `Orchestrators/media_management.sh` — not individually scheduled. The orchestrator handles ordering, pass/fail tracking, and the combined summary. + +--- + +## Why Order Matters + +``` +1. media_shares_permissions.sh ← permissions first +2. media_cleaner.sh anime ← clean junk before arr scripts scan +3. media_cleaner.sh media ← same +4. lidarr_cleanup.sh ← arr cleanup last +5. sonarr_cleanup.sh +6. radarr_cleanup.sh +``` + +**Permissions before everything else** — arr cleanup scripts need correct ownership to delete files. If a file is owned by root and the script runs as nobody, the delete fails silently. + +**Cleaner before arr cleanup** — junk files (.sfv, .rar, .txt etc.) mixed in with media files create noise in the orphan detection logic. Clean the junk first so arr cleanup only deals with real media files. + +**Arr cleanup last** — depends on clean folders and correct permissions to work reliably. + +--- + +## Scripts + +### `media_shares_permissions.sh` + +Applies correct ownership and permissions recursively to all configured media shares. + +```bash +# Called by media_management.sh — not scheduled directly +# Run manually when needed: +/mnt/user/appdata/unraid_scripts/Media/media_shares_permissions.sh +/mnt/user/appdata/unraid_scripts/Media/media_shares_permissions.sh --dry-run +``` + +**What it applies:** + +```bash +PERMISSIONS_MODE="777" # chmod applied recursively +PERMISSIONS_OWNER="nobody:users" # chown applied recursively +``` + +`777` and `nobody:users` is the standard for unRAID media shares accessible by Docker containers. All media server containers (Emby, Tdarr, arr stack) run as `nobody:users` — this ensures they can read, write, and delete files without permission errors. + +**Why recursive takes time:** + +On a large library with millions of files this can run for 20-30 minutes. This is expected and normal. Run it overnight via the orchestrator — not during peak usage hours. + +**Add or remove shares** in `Master.conf` under `MEDIA_PERMISSION_SHARES`. The script reads this list at runtime — no script changes needed. + +--- + +### `media_cleaner.sh` + +Removes junk files from media shares using configurable file pattern lists. Two profiles with independent folder lists and patterns. + +```bash +# Called by media_management.sh with profile argument +# Run manually with profile required: +/mnt/user/appdata/unraid_scripts/Media/media_cleaner.sh anime +/mnt/user/appdata/unraid_scripts/Media/media_cleaner.sh media +/mnt/user/appdata/unraid_scripts/Media/media_cleaner.sh anime --dry-run +/mnt/user/appdata/unraid_scripts/Media/media_cleaner.sh media --dry-run +``` + +**Always --dry-run first** — especially on first use or after adding new patterns. + +#### Anime Profile + +```bash +ANIME_CLEAN_FOLDERS=( + /mnt/user/Anime_Movies + /mnt/user/Anime_Movies-Old + /mnt/user/Anime_Shows + /mnt/user/Anime_Shows-Old +) + +ANIME_FILE_PATTERNS=( + '*.sfv' '*.md5' '*.sha1' '*.txt' '*.url' '*.lnk' + '*.rar' '*.zip' '*.info' '*.torrent' '*.sample*' '*.proof*' + '*sync-conflict*' '*.scr' '*.srr' '*.exe' '*.webp' + '*.log' '*.json' +) +``` + +Anime downloads from groups commonly include verification files (`.sfv`, `.md5`), RAR archives after extraction, proof files, and samples. These are safe to delete after the video files have been imported. + +#### Media Profile + +```bash +MEDIA_CLEAN_FOLDERS=( + /mnt/user/Kids_Movies + /mnt/user/Kids_Tv_Shows + /mnt/user/Movies + /mnt/user/Music + /mnt/user/Sports + /mnt/user/stand-up_comedy + /mnt/user/Tv_Shows +) +``` + +The media profile includes `*.iso` and `*.lrc` in addition to the anime patterns — ISO disc images after ripping and lyric files that aren't needed in the media folders. + +#### Adding New Patterns + +Add to the appropriate array in `Master.conf` — no script changes needed: + +```bash +ANIME_FILE_PATTERNS=( + '*.sfv' '*.md5' ... + '*.new-pattern' # ← just add here +) +``` + +#### Safety Note + +The cleaner deletes by pattern — it does not check what arr thinks about the files. It runs before arr cleanup specifically so arr cleanup sees clean folders. Do not add patterns that match media files you want to keep (`.mkv`, `.mp4` etc.). + +--- + +### `lidarr_cleanup.sh` + +Removes orphaned music files from the library that Lidarr no longer tracks. + +```bash +# Called by media_management.sh +# Always test first: +/mnt/user/appdata/unraid_scripts/Media/lidarr_cleanup.sh --dry-run --log +/mnt/user/appdata/unraid_scripts/Media/lidarr_cleanup.sh +``` + +**How it works:** + +1. Queries the Lidarr API for all tracked file paths +2. Walks `LIDARR_MUSIC_ROOT` on disk +3. Classifies every file found: + +| Classification | Condition | Action | +|---------------|-----------|--------| +| TRACKED | Lidarr API knows this exact path | Leave alone | +| PROTECTED | Matches `LIDARR_PROTECTED_PATTERNS` | Never delete | +| ORPHAN | Music extension, not tracked, older than `LIDARR_ORPHAN_AGE` days | Delete | +| JUNK | Not a music extension, not protected | Delete regardless of age | +| RECENT | Not tracked, under `LIDARR_ORPHAN_AGE` days | Skip — may be mid-import | + +**Why protected patterns are critical:** + +Lidarr generates cover art (`*.jpg`), metadata (`*.nfo`), and lyrics (`*.lrc`) alongside your music files. These do not appear in Lidarr's tracked file API response — they would be classified as orphans and deleted without the protected patterns list. This would break artwork display in Emby and Lidarr itself. + +```bash +LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc") +``` + +Never remove patterns from this list without understanding what Lidarr generates in your specific setup. + +**Why the age threshold matters:** + +When Lidarr downloads a file it exists on disk before it's fully processed and imported. The `LIDARR_ORPHAN_AGE=7` day window ensures files that are mid-import are never touched. 7 days is conservative — adjust if your import workflow takes longer than expected. + +**Configuration:** +```bash +LIDARR_URL="http://192.168.50.2:8686" +LIDARR_API_KEY="your-api-key" +LIDARR_MUSIC_ROOT="/mnt/user/Music-New" # must match Lidarr root path exactly +LIDARR_ORPHAN_AGE=7 +LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma") +LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc") +``` + +`LIDARR_MUSIC_ROOT` must match the root path configured in Lidarr Settings → Media Management → Root Folders exactly. A mismatch means all files appear untracked and everything gets deleted. + +--- + +### `sonarr_cleanup.sh` + +Removes orphaned TV episode files from the library that Sonarr no longer tracks. + +```bash +# Always test first: +/mnt/user/appdata/unraid_scripts/Media/sonarr_cleanup.sh --dry-run --log +/mnt/user/appdata/unraid_scripts/Media/sonarr_cleanup.sh +``` + +Same classification logic as `lidarr_cleanup.sh` applied to TV files. + +**Protected patterns cover:** +- Show artwork (`*.jpg`, `*.png`) — Sonarr generates per-series and per-episode artwork +- Metadata (`*.nfo`) — Sonarr generates NFO files for media center compatibility +- Subtitles (`*.srt`, `*.sub`, `*.ass`, `*.ssa`) — managed by Bazarr via Sonarr + +```bash +SONARR_URL="http://192.168.50.2:8989" +SONARR_API_KEY="your-api-key" +SONARR_TV_ROOT="/mnt/user/Tv_Shows" +SONARR_ORPHAN_AGE=7 +SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov") +SONARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa") +``` + +**`.ts` in extensions:** Transport stream files from Live TV recordings. These are tracked by Sonarr for recorded episodes — include this extension to allow cleanup of orphaned recordings. + +--- + +### `radarr_cleanup.sh` + +Removes orphaned movie files from the library that Radarr no longer tracks. + +```bash +# Always test first: +/mnt/user/appdata/unraid_scripts/Media/radarr_cleanup.sh --dry-run --log +/mnt/user/appdata/unraid_scripts/Media/radarr_cleanup.sh +``` + +Same classification logic applied to movie files. + +**Protected patterns cover:** +- Movie artwork (`*.jpg`, `*.png`) +- Metadata (`*.nfo`) +- Subtitles (`*.srt`, `*.sub`, `*.ass`, `*.ssa`) — managed by Bazarr + +```bash +RADARR_URL="http://192.168.50.2:7878" +RADARR_API_KEY="your-api-key" +RADARR_MOVIES_ROOT="/mnt/user/Movies" +RADARR_ORPHAN_AGE=7 +RADARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "wmv" "mov") +RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa") +``` + +--- + +## Safe Testing Procedure + +The arr cleanup scripts permanently delete files. Always test before running live — especially on first use, after API key changes, or after root path changes. + +**Step 1 — Dry run with logging:** +```bash +/mnt/user/appdata/unraid_scripts/Media/lidarr_cleanup.sh --dry-run --log +/mnt/user/appdata/unraid_scripts/Media/sonarr_cleanup.sh --dry-run --log +/mnt/user/appdata/unraid_scripts/Media/radarr_cleanup.sh --dry-run --log +``` + +`--log` enables verbose output showing every file classification decision. Review carefully: +- Are TRACKED files the ones you expect? +- Are ORPHAN files actually orphans or recently downloaded files? +- Are PROTECTED files being correctly identified? +- Is the root path correct — no files showing as orphans that shouldn't be? + +**Step 2 — Check the numbers make sense:** + +If dry run shows 50,000 files as orphans on a library you know is healthy — something is wrong. Common causes: +- Root path mismatch between Master.conf and arr settings +- API key incorrect — returns empty tracked list +- Arr library scan not complete — recently added files not yet indexed + +**Step 3 — Run live:** +```bash +/mnt/user/appdata/unraid_scripts/Media/lidarr_cleanup.sh +``` + +**Step 4 — Verify in arr UI:** + +After running, check the arr's library count hasn't dropped unexpectedly. A healthy cleanup removes a small number of genuinely orphaned files — not a significant percentage of your library. + +--- + +## Configuration Quick Reference + +All configuration in `Master.conf` under `── MEDIA ──` section. + +```bash +# Permissions +PERMISSIONS_MODE="777" +PERMISSIONS_OWNER="nobody:users" +MEDIA_PERMISSION_SHARES=(...) + +# Cleaner +ANIME_CLEAN_FOLDERS=(...) +MEDIA_CLEAN_FOLDERS=(...) +ANIME_FILE_PATTERNS=(...) +MEDIA_FILE_PATTERNS=(...) + +# Orchestrator job order +MEDIA_MAINTENANCE_JOBS=( + "Media/media_shares_permissions.sh" + "Media/media_cleaner.sh anime" + "Media/media_cleaner.sh media" + "Media/lidarr_cleanup.sh" + "Media/sonarr_cleanup.sh" + "Media/radarr_cleanup.sh" +) + +# Arr cleanup — per arr +LIDARR_URL / LIDARR_API_KEY / LIDARR_MUSIC_ROOT +LIDARR_ORPHAN_AGE / LIDARR_EXTENSIONS / LIDARR_PROTECTED_PATTERNS +# (same pattern for SONARR_ and RADARR_) +``` + +--- + +## Adding a New Arr + +To add Readarr or any other arr cleanup to the ecosystem: + +1. Copy `radarr_cleanup.sh` as the template — same classification logic applies +2. Update the API endpoint, variable names, and root path +3. Add configuration variables to `Master.conf` +4. Add the script to `MEDIA_MAINTENANCE_JOBS` in `Master.conf` +5. Test with `--dry-run --log` before running live + +The orchestrator picks it up automatically — no changes to `media_management.sh` needed. \ No newline at end of file diff --git a/Monitors/README-Monitors.md b/Monitors/README-Monitors.md new file mode 100644 index 0000000..6b58f77 --- /dev/null +++ b/Monitors/README-Monitors.md @@ -0,0 +1,376 @@ +# Monitors + +Watch and report only. Scripts in this folder never take action — they observe, measure, and notify. Intervention is handled by other parts of the ecosystem. + +``` +unRAID_Essentials/ — acts on the system (restarts, reboots, stops) +Docker_Essentials/ — acts on containers (watchdog, restarts) +Monitors/ — observes and reports (this folder) +``` + +All monitor scripts are safe to run at any time. None of them write to flash drives except `bandwidth_monitor.sh` which makes one bounded append per rsync run. All other monitors are read-only operations. + +--- + +## Scripts + +### `cert_monitor.sh` + +Monitors SSL certificate expiry for all configured domains. + +```bash +# Scheduled as: 0 9 * * 0 (Sunday 9am weekly) +/mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh +``` + +**How it works:** + +Connects directly to each domain via `openssl s_client` and reads the certificate the server is actually presenting. This is different from checking your certificate files directly — it catches real-world issues that file-based checks miss: + +- Certificate renewed but web server not reloaded — old cert still being served +- Wrong certificate being served for a domain +- Certificate chain issues invisible to the cert file itself + +Each domain and subdomain is a separate entry. They have independent certificates — `gmer4lfe.com` and `auth.gmer4lfe.com` may expire on different dates. + +**Notification behavior:** +- Silent when all certs are healthy +- One warning notification for all domains approaching `CERT_WARN_DAYS` +- One critical notification for all domains within `CERT_CRIT_DAYS` +- Notifications batched by severity — not one per domain + +**Configuration:** +```bash +CERT_MONITOR_DOMAINS=( + "Gmer4Lfe.com" + "Gmer4Lfe.us" + # "auth.Gmer4Lfe.com" # add subdomains as separate entries +) +CERT_WARN_DAYS=30 # warn when this many days remaining +CERT_CRIT_DAYS=7 # critical when this many days remaining +CERT_TIMEOUT=10 # seconds before giving up per domain +``` + +--- + +### `smart_health.sh` + +Checks SMART health attributes for all drives in the system. + +```bash +# Scheduled as: 0 7 * * 0 (Sunday 7am weekly) +/mnt/user/appdata/unraid_scripts/Monitors/smart_health.sh +``` + +**What it checks per drive:** + +| Attribute | Threshold | Meaning | +|-----------|-----------|---------| +| Reallocated_Sector_Ct | > 0 = warning | Bad sectors remapped — drive showing wear | +| Current_Pending_Sector | > 0 = warning | Sectors awaiting reallocation | +| Offline_Uncorrectable | > 0 = critical | Sectors that could not be corrected | +| Temperature_Celsius | SMART_TEMP_WARN/CRIT | Drive running hot | +| Power_On_Hours | informational | Drive age estimate | +| Overall health status | PASSED/FAILED | Drive's own self-assessment | + +Drive discovery is automatic — `/dev/sd*` and `/dev/nvme*` are scanned on every run. No drive list to maintain. + +**Why ignore the boot USB:** + +unRAID boots from a USB flash drive that typically appears as `sda`. Flash drives either don't support SMART or report meaningless values. Add it to `SMART_IGNORE_DRIVES` to keep it out of the report. + +**Notification behavior:** +- Silent when all drives are healthy +- One warning notification listing all drives with concerning attributes +- One critical notification if any drive has uncorrectable sectors + +**Configuration:** +```bash +SMART_TEMP_WARN=45 # degrees C +SMART_TEMP_CRIT=55 # degrees C +SMART_IGNORE_DRIVES=( + "sda" # boot USB — not meaningful to check +) +``` + +--- + +### `zfs_memory_snapshot.sh` + +Weekly ZFS pool health and memory diagnostic report. + +```bash +# Scheduled as: 0 6 * * 0 (Sunday 6am weekly) +/mnt/user/appdata/unraid_scripts/Monitors/zfs_memory_snapshot.sh +``` + +**What it reports:** + +- Pool status — ONLINE/DEGRADED/FAULTED per pool +- Pool overview — size, allocated, free, capacity, health +- ARC statistics — max, current, metadata usage, utilization % +- Memory status — total, free, available RAM vs thresholds +- Top N Docker containers by memory usage +- Kernel pressure snapshot via vmstat + +**Informational only.** This script reports what it finds. `system_watchdog.sh` handles threshold-based intervention — ARC reclaim, reboot decisions, memory pressure response. The snapshot gives you the weekly picture; the watchdog handles emergencies. + +**Output is written to both console and `ZFS_REPORT_LOG`** — the log file lets you compare pool health week over week without having to remember what last week's numbers were. + +**Pool ignore list:** + +Pools expected to run at high capacity can be excluded from health reporting. They remain fully monitored by unRAID — this only affects what appears in the weekly report. + +```bash +ZFS_REPORT_IGNORE_POOLS=( + "disk10" # high usage expected + "disk9" + "disk8" + "disk6" + "disk5" +) +``` + +**Configuration:** +```bash +ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log" +ZFS_REPORT_ARC_WARN_PCT=90 # warn if ARC utilization above this % +ZFS_REPORT_FREE_WARN_GB=10 # warn if free RAM below this GB +ZFS_REPORT_AVAIL_WARN_GB=20 # warn if available RAM below this GB +ZFS_REPORT_DOCKER_TOP=10 # top N Docker memory users to show +``` + +--- + +### `backup_verify.sh` + +Verifies the rsync mirror is healthy by comparing random file checksums between local and remote servers. + +```bash +# Scheduled as: 0 10 * * 0 (Sunday 10am weekly) +/mnt/user/appdata/unraid_scripts/Monitors/backup_verify.sh +``` + +**Why this matters:** + +`rsync.sh` copies files successfully. But does the copy match the original? `backup_verify.sh` answers that question by independently computing MD5 checksums on both sides and comparing them. It catches: + +- Silent data corruption during transfer +- Files that transferred but were corrupted at rest +- Partial transfers that rsync reported as success +- Storage hardware issues on either server + +**How it works:** + +1. Randomly samples `BACKUP_VERIFY_SAMPLE` files per share (files larger than `BACKUP_VERIFY_MIN_SIZE`) +2. Computes MD5 checksum locally +3. SSHes to remote and computes MD5 checksum there +4. Compares results + +**Result per file:** + +| Result | Meaning | +|--------|---------| +| MATCH | Checksums identical — file correctly mirrored | +| MISMATCH | File exists on both but checksums differ — sync may have failed | +| MISSING | File exists locally but not on remote — not yet synced or deleted | + +**Configuration:** +```bash +# Leave empty to use DAILY_SYNC_SHARES automatically +BACKUP_VERIFY_SHARES=( + # /mnt/user/Movies + # /mnt/user/Tv_Shows +) +BACKUP_VERIFY_SAMPLE=10 # files sampled per share per run +BACKUP_VERIFY_MIN_SIZE=1M # skip files smaller than this +``` + +If `BACKUP_VERIFY_SHARES` is empty the script automatically uses `DAILY_SYNC_SHARES` — no additional configuration needed for the standard setup. + +Uses the existing SSH keys already configured for rsync — no additional setup required. + +--- + +### `bandwidth_monitor.sh` + +Logs rsync transfer history and generates weekly summary reports. Called automatically by `rsync.sh` — you do not need to schedule the logging mode manually. + +```bash +# Log mode — called automatically by rsync.sh after each successful sync +# No manual scheduling needed + +# Report mode — run manually or schedule weekly +# Scheduled as: 0 11 * * 0 (Sunday 11am weekly) +/mnt/user/appdata/unraid_scripts/Monitors/bandwidth_monitor.sh --report +``` + +**Two modes:** + +**`--log-transfer profile duration status`** — called by `rsync.sh` after each sync. Appends one line to the log file and trims entries older than `BANDWIDTH_LOG_RETENTION` days. You never call this manually. + +**`--report`** — reads the log file and generates a weekly summary showing per-profile run counts, average durations, last 7 days activity, and failure counts. + +**Log format:** +``` +YYYY-MM-DD|HH:MM|profile|duration_seconds|status +2026-04-14|01:23|arrs_stack|287|success +2026-04-14|01:31|critical-data|143|success +2026-04-14|02:15|movies|1847|failed +``` + +**Why this format:** + +The log never parses rsync output. Earlier designs tried to extract bytes transferred from rsync's human-readable output — that approach breaks silently when rsync updates and changes its output format. The current format captures what's reliably available: profile, duration, and success/failure. This is version-proof and survives any rsync update. + +**Flash drive design:** + +The log lives on `/boot/` so it survives reboots. Each rsync run makes exactly one append and one trim — the file never grows beyond `BANDWIDTH_LOG_RETENTION` lines. Minimal flash wear. + +**Configuration:** +```bash +BANDWIDTH_LOG="/boot/config/bandwidth_history.db" +BANDWIDTH_LOG_RETENTION=90 # days — file stays bounded +BANDWIDTH_WARN_GB=50 # flag days exceeding this in reports + # (note: current log tracks duration not bytes) +``` + +--- + +### `weekly_health_digest.sh` + +Aggregates system health data from across the entire ecosystem into a single digest report. + +```bash +# Scheduled as: 0 8 * * * (8am daily — profile controls when it notifies) +/mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh +``` + +**The key design:** schedule it daily regardless of profile. The `DIGEST_PROFILE` setting in `Master.conf` controls when a notification is actually sent — not the cron schedule. + +**Three profiles:** + +| Profile | Behavior | Use When | +|---------|----------|----------| +| `always` | Sends every run | You want a daily health summary | +| `smart` | Sends only if something worth reporting | Quiet operation, alerts on issues | +| `weekly` | Sends once per week on `DIGEST_DAY` | Weekly digest, silent other days | + +Switch profiles by changing `DIGEST_PROFILE` in `Master.conf` — no cron changes needed. + +**Data sources — reads only, no writes:** + +| Source | What it reads | +|--------|---------------| +| `/tmp/transcode_state.db` | Ramdisk symlink and usage | +| `/tmp/container_watchdog_state.db` | Active container watchdog strikes | +| `/tmp/system_watchdog_state.db` | Active system watchdog strikes | +| `/boot/config/failover_state.db` | Current failover state | +| `/boot/config/system_watchdog_failed.db` | Container skip list | +| `/boot/config/bandwidth_history.db` | Recent transfer totals | +| Live `openssl` connection | SSL cert days remaining per domain | + +**Smart profile triggers:** +```bash +# Set true to include this check in smart mode's "worth reporting" decision +DIGEST_SMART_ON_WATCHDOG=true # any active watchdog strikes +DIGEST_SMART_ON_FAILOVER=true # failover state is not NORMAL +DIGEST_SMART_ON_CERT_WARN=true # any cert under CERT_WARN_DAYS +DIGEST_SMART_ON_BANDWIDTH=true # any transfer exceeded BANDWIDTH_WARN_GB +``` + +**Configuration:** +```bash +DIGEST_PROFILE="weekly" # always | smart | weekly +DIGEST_DAY="Sunday" # for weekly profile — must match date +%A output +``` + +--- + +### `emby_session_report.sh` + +Weekly Emby usage report via the Emby API. + +```bash +# Scheduled as: 0 11 * * 0 (Sunday 11am weekly) +/mnt/user/appdata/unraid_scripts/Monitors/emby_session_report.sh +``` + +**What it reports:** +- Active streams at time of run +- Stream breakdown — total, Live TV, transcoding, direct play +- Library counts — movies, episodes, songs +- Current ramdisk transcode usage and symlink state + +**No persistent writes** — queries the Emby API fresh on every run. No log files, no state. Run it any time for a current snapshot. + +**Requires an Emby API key:** +1. Open Emby Settings → API Keys +2. Generate a new key +3. Paste it into `Master.conf` as `EMBY_API_KEY` + +**Configuration:** +```bash +EMBY_URL="http://localhost:8096" +EMBY_API_KEY="" # get from Emby Settings → API Keys +EMBY_REPORT_DAYS=7 # report period in days +EMBY_REPORT_TOP_N=10 # top N content items to show +``` + +--- + +## Flash Drive Write Policy + +unRAID boots from a USB flash drive. Flash drives have limited write cycles. The Monitors folder is designed with this in mind: + +| Script | Writes to flash | Notes | +|--------|----------------|-------| +| `cert_monitor.sh` | Never | Read-only openssl checks | +| `smart_health.sh` | Never | Read-only smartctl checks | +| `zfs_memory_snapshot.sh` | Never | Writes to `/var/log/` (RAM disk) | +| `backup_verify.sh` | Never | SSH + MD5 comparison only | +| `bandwidth_monitor.sh` | One append + one trim per rsync run | Bounded — never exceeds retention days | +| `weekly_health_digest.sh` | Never | Reads existing state files only | +| `emby_session_report.sh` | Never | API queries only | + +The only flash write in the entire Monitors folder is `bandwidth_monitor.sh` — and it's designed to be minimal and bounded. + +--- + +## Recommended Schedule + +```bash +# Daily +0 8 * * * weekly_health_digest.sh # profile controls when it notifies + +# Weekly — Sunday morning block +0 6 * * 0 zfs_memory_snapshot.sh +0 7 * * 0 smart_health.sh +0 9 * * 0 cert_monitor.sh +0 10 * * 0 backup_verify.sh +0 11 * * 0 emby_session_report.sh +0 11 * * 0 bandwidth_monitor.sh --report + +# Automatic — no scheduling needed +# bandwidth_monitor.sh --log-transfer is called by rsync.sh after each sync +``` + +The Sunday morning block runs after the nightly maintenance window — by the time the monitors run, the weekly restarts, log clears, and media management jobs have completed. The health snapshot reflects a freshly maintained system. + +--- + +## --dry-run Support + +All monitor scripts support `--dry-run`. In dry-run mode: + +- Checks run and results are shown +- No notifications are sent +- No files are written + +Useful for testing configuration changes before scheduling: + +```bash +/mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh --dry-run +/mnt/user/appdata/unraid_scripts/Monitors/backup_verify.sh --dry-run +/mnt/user/appdata/unraid_scripts/Monitors/smart_health.sh --dry-run +``` \ No newline at end of file diff --git a/Orchestrators/README-orchestrators.md b/Orchestrators/README-orchestrators.md new file mode 100644 index 0000000..60b6be8 --- /dev/null +++ b/Orchestrators/README-orchestrators.md @@ -0,0 +1,216 @@ +# Orchestrators + +Sequential job runners that coordinate multiple scripts into a single scheduled operation. + +Orchestrators do not contain business logic — they call other scripts in order, track pass/fail per job, and report a clean summary. All configuration lives in `Master.conf`. Adding or removing a job never requires touching the orchestrator script itself. + +--- + +## Why Orchestrators + +Without orchestrators, each script runs independently on its own schedule. This works but creates problems: + +- **Race conditions** — two scripts running simultaneously on the same data +- **Order dependency failures** — media cleaner runs before permissions, finds wrong ownership +- **No combined summary** — 6 separate notifications instead of one clean report +- **Scheduling complexity** — 6+ cron entries instead of one + +Orchestrators solve this by making a set of related scripts into a single scheduled unit with a defined execution order and a unified summary. + +--- + +## Scripts + +### `daily_sync.sh` + +Syncs all bulk media shares to the remote server sequentially. Scheduled once daily. + +```bash +# Scheduled as: 0 1 * * * (1am daily) +/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh +``` + +**What it does:** +1. Detects local and remote server via `detect_hosts()` +2. Resolves remote Tailscale IP +3. Runs a single pre-flight check — connectivity + remote rootfs +4. Iterates through every share in `DAILY_SYNC_SHARES` +5. Calls `Rsync/rsync.sh` for each share +6. Tracks pass/fail and duration per share +7. Reports a combined summary + +**Why one pre-flight check upfront:** +Connectivity and rootfs are checked once before the loop starts — not once per share. If the remote is unreachable or the rootfs is nearly full, the whole run fails fast before attempting 12 shares. Individual share existence and disk checks still run per-share inside `rsync.sh`. + +**Configuration:** +```bash +# Master.conf — add or remove paths to control what syncs nightly +DAILY_SYNC_SHARES=( + /mnt/user/Movies + /mnt/user/Tv_Shows + /mnt/user/Anime_Shows + # ... +) +``` + +These shares use global rsync defaults — no profile needed. For shares requiring custom bandwidth limits, container stops, or different rsync options, create a named profile in the Rsync profile system and call `rsync.sh` directly on a separate schedule instead. + +**Example output:** +``` +━━━ 🔄 Daily Sync Starting — 2026-04-14 01:00:00 ━━━ +📋 Shares: 11 + +━━━ [1/11] Movies ━━━ +...rsync output... +✅ Movies — 4m32s + +━━━ [2/11] Tv_Shows ━━━ +... +━━━━━ 📋 DAILY SYNC SUMMARY ━━━━━ +✅ Pass: 10 ❌ Fail: 1 +⏱️ Duration: 47m12s +❌ Failed: Anime_Shows-Old +``` + +--- + +### `media_management.sh` + +Runs all media maintenance scripts sequentially in the order defined in `Master.conf`. Scheduled once daily, typically after the nightly sync. + +```bash +# Scheduled as: 0 2 * * * (2am daily — after daily_sync.sh) +/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh +``` + +**What it does:** +1. Reads `MEDIA_MAINTENANCE_JOBS` from `Master.conf` +2. Runs each job in order — script path + optional argument +3. Tracks pass/fail per job +4. Reports a combined summary +5. A failure in one job does not stop the others + +**Why order matters:** + +``` +1. media_shares_permissions.sh ← permissions first — everything else depends on correct ownership +2. media_cleaner.sh anime ← clean junk before arr scripts scan +3. media_cleaner.sh media ← same +4. lidarr_cleanup.sh ← arr cleanup last — depends on clean folders +5. sonarr_cleanup.sh +6. radarr_cleanup.sh +``` + +If arr cleanup runs before permissions, it may fail to delete files it doesn't have access to. If it runs before the cleaner, it finds junk files mixed in with real content. The order is intentional. + +**Configuration:** +```bash +# Master.conf — add, remove, or reorder jobs here +# Format: "folder/script.sh optional_argument" +MEDIA_MAINTENANCE_JOBS=( + "Media/media_shares_permissions.sh" + "Media/media_cleaner.sh anime" + "Media/media_cleaner.sh media" + "Media/lidarr_cleanup.sh" + "Media/sonarr_cleanup.sh" + "Media/radarr_cleanup.sh" +) +``` + +**Adding a new job:** +```bash +# Add a line to MEDIA_MAINTENANCE_JOBS — no script changes needed +MEDIA_MAINTENANCE_JOBS=( + "Media/media_shares_permissions.sh" + "Media/media_cleaner.sh anime" + "Media/media_cleaner.sh media" + "Media/my_new_script.sh" # ← just add it here + "Media/lidarr_cleanup.sh" + "Media/sonarr_cleanup.sh" + "Media/radarr_cleanup.sh" +) +``` + +**Disabling a job temporarily:** +```bash +# Comment it out — easy to re-enable +MEDIA_MAINTENANCE_JOBS=( + "Media/media_shares_permissions.sh" +# "Media/media_cleaner.sh anime" # ← disabled, not deleted + "Media/media_cleaner.sh media" + "Media/lidarr_cleanup.sh" + "Media/sonarr_cleanup.sh" + "Media/radarr_cleanup.sh" +) +``` + +**--dry-run support:** +`media_management.sh --dry-run` passes `--dry-run` through to every child script. All scripts report what they would do without making changes. Useful for testing a new job before adding it to the live schedule. + +```bash +/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh --dry-run +``` + +--- + +## The Orchestrator Pattern + +Both orchestrators follow the same pattern. This is by design — any script that needs to coordinate multiple operations should follow it: + +``` +1. Setup — validate config, detect hosts if needed +2. Pre-flight — fail fast checks before doing any work +3. Job loop — run each job, track pass/fail, continue on failure +4. Summary — one clean report of all results +5. Notification — one notification per run, not one per job +``` + +This pattern means: +- **Consistent output** — every orchestrator looks the same in the logs +- **No silent failures** — pass/fail tracked per job, reported in summary +- **Single notification** — one bell ring, not six +- **Resilient** — one job failing doesn't stop the rest + +--- + +## Scheduling + +```bash +# Recommended schedule +0 1 * * * daily_sync.sh # 1am — media shares to remote +0 2 * * * media_management.sh # 2am — after sync completes +``` + +The 1 hour gap between them is intentional. `daily_sync.sh` can take 30-60 minutes on a large library. Starting `media_management.sh` before it finishes risks permission and cleanup operations running on files that are mid-transfer. + +If your sync consistently finishes well under an hour, reduce the gap. If it regularly runs long, increase it. + +--- + +## Adding a New Orchestrator + +If you find yourself running 3 or more related scripts on the same schedule, consider wrapping them in a new orchestrator. The pattern is simple: + +```bash +# Minimal orchestrator skeleton +JOBS=( + "Folder/script1.sh" + "Folder/script2.sh arg" +) + +PASS=() +FAIL=() + +for JOB in "${JOBS[@]}"; do + SCRIPT=$(echo "$JOB" | cut -d' ' -f1) + ARG=$(echo "$JOB" | cut -d' ' -f2-) + + if bash "$ECOSYSTEM_ROOT/$SCRIPT" $ARG; then + PASS+=("$SCRIPT") + else + FAIL+=("$SCRIPT") + fi +done +``` + +Better yet — model it directly on `media_management.sh` which already handles dry-run passthrough, status display, pass/fail tracking and summary reporting. \ No newline at end of file diff --git a/README.md b/README.md index 10d0759..e843f81 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,297 @@ -# Unraid_Scripts +# unRAID Script Ecosystem -test -2ndtest \ No newline at end of file +A modular, git-managed automation ecosystem for two unRAID servers. One configuration file. Both servers stay in sync with a single git pull. Everything from daily media syncing to mutual container failover runs automatically — and when something goes wrong, the system tries to fix itself before waking you up. + +--- + +## The Goal + +A self-hosted infrastructure that runs itself. + +``` +Step away → come back to a healthy system +Something breaks → system self-heals +Something can't be fixed → you get notified +Servers stay in sync → one edit propagates everywhere +``` + +Before this ecosystem existed, the same problems were solved by 60+ standalone scripts across two servers — different coding styles, no shared standards, every change applied twice. This ecosystem standardises everything into one codebase with one config file and one deployment mechanism. + +--- + +## The Servers + +``` +HOST1 — unRAID-Gmer4Lfe (Primary) + Hardware: Threadripper 1950X, 128GB RAM + Storage: Multiple ZFS pools + cache + Role: Primary services, full media stack, all Docker containers + +HOST2 — unRAID-Jayred365 (Secondary) + Location: Remote — 50 miles from HOST1 + Role: Mirror, failover coverage, independent services + +Network: Tailscale — encrypted tunnel between both servers +Repo: Self-hosted Gitea on HOST1 +Deployment: git pull on either server → both stay current +``` + +--- + +## How It Works + +Every script sources two files at startup: + +```bash +source "$SCRIPT_DIR/../Master.conf" # all user configuration +source "$SCRIPT_DIR/../common.sh" # shared library +``` + +**`Master.conf`** is the single source of truth. Container names, thresholds, paths, API keys, rsync profiles, failover tiers — everything configurable lives here. Change a value, push to git, both servers pull — done. + +**`common.sh`** provides shared functions used by every script — host detection, notifications, rsync helpers, output formatting, icon set. Scripts never duplicate this logic. + +**`git_pull_execute.sh`** pulls the latest scripts from Gitea and sets executable permissions. Schedule it or run it manually on either server. + +--- + +## Repository Structure + +``` +Unraid_Scripts/ +├── Master.conf # All user configuration — edit here only +├── common.sh # Shared library — used by all scripts +├── git_pull_execute.sh # Pull latest scripts from Gitea +├── User_Script_Template.sh # Paste into unRAID User Scripts plugin +│ +├── Failover/ # Mutual container failover +│ ├── README-Failover.md +│ ├── failover.sh # State machine — runs continuously +│ └── failover_test.sh # Controlled simulation harness +│ +├── Monitors/ # Health reporting — watch and report only +│ ├── README-Monitors.md +│ ├── cert_monitor.sh # SSL certificate expiry +│ ├── smart_health.sh # Drive SMART attributes +│ ├── zfs_memory_snapshot.sh # ZFS pool health + memory report +│ ├── backup_verify.sh # Random checksum verification vs remote +│ ├── bandwidth_monitor.sh # Rsync transfer history +│ ├── weekly_health_digest.sh # Aggregated system health summary +│ └── emby_session_report.sh # Emby usage statistics +│ +├── Orchestrators/ # Sequential job runners +│ ├── README-Orchestrators.md +│ ├── daily_sync.sh # All media shares synced nightly +│ └── media_management.sh # Permissions + cleaners + arr cleanup +│ +├── Rsync/ # Core sync engine +│ ├── README-Rsync.md +│ ├── rsync.sh # Per-share or per-profile sync +│ └── README_Rsync_Setup.md # Initial setup guide +│ +├── Docker_Essentials/ # Container lifecycle management +│ ├── README-Docker_Essentials.md +│ ├── docker_watchdog.sh # Two-tier self-healing monitoring +│ ├── docker_daily_restart.sh # Daily container restarts +│ ├── docker_weekly_restart.sh # Weekly container restarts +│ └── docker_network_connect.sh # Extra network connections at boot +│ +├── Media/ # Media library maintenance +│ ├── README-Media.md +│ ├── media_shares_permissions.sh # Recursive permission application +│ ├── media_cleaner.sh # Junk file removal (anime + media profiles) +│ ├── lidarr_cleanup.sh # Orphaned music file cleanup +│ ├── sonarr_cleanup.sh # Orphaned TV file cleanup +│ └── radarr_cleanup.sh # Orphaned movie file cleanup +│ +├── Transcodes/ # Emby ramdisk transcode management +│ ├── README-Transcoding.md +│ ├── ramdisk_setup.sh # Create ramdisk + symlink at array start +│ ├── transcode_manager.sh # Monitor usage, manage symlink, display sessions +│ └── transcode_cleanup.sh # Remove old inactive transcode files +│ +├── Tools/ # Situational utilities — run when needed +│ ├── README-Tools.md +│ ├── recreate_shares.sh # Recreate share dirs after incident +│ ├── failover_state_reset.sh # Reset failover state file to NORMAL +│ ├── watchdog_skip_list_manager.sh # Manage container watchdog skip lists +│ ├── bulk_permissions_repair.sh # Targeted permission repair for one share +│ ├── container_data_export.sh # Export container appdata to tar archive +│ ├── emby_database_repair.sh # SQLite integrity check on Emby databases +│ └── zfs_pool_scrub.sh # Trigger ZFS scrub with completion report +│ +└── unRAID_Essentials/ # Server-level system management + ├── README-unRAID_Essentials.md + ├── system_watchdog.sh # Last line of defense — controlled reboot + ├── webgui_restart.sh # WebGUI nginx + emhttp auto-restart + ├── docker_syslog_filter.sh # Suppress Docker veth noise from syslog + ├── php_fpm_max_children.sh # WebGUI PHP-FPM concurrency tuning + ├── clear_logs.sh # Weekly system log clearing + ├── mover_stop.sh # Graceful mover termination + ├── rsync_stop.sh # Stop rsync + recover containers + ├── server_reboot.sh # Graceful reboot with user warning + └── user_script_stop.sh # Stop all running User Scripts jobs +``` + +--- + +## Folders at a Glance + +| Folder | What it does | Key scripts | +|--------|-------------|-------------| +| **Failover** | Mutual container failover — autonomous, tiered, DDNS-safe | `failover.sh` | +| **Monitors** | Watch and report — never act, minimal flash writes | `weekly_health_digest.sh` | +| **Orchestrators** | Sequential job runners with unified reporting | `daily_sync.sh`, `media_management.sh` | +| **Rsync** | Core sync engine with profile system | `rsync.sh` | +| **Docker_Essentials** | Two-tier self-healing container management | `docker_watchdog.sh` | +| **Media** | Permissions, junk cleanup, arr orphan cleanup | `media_cleaner.sh`, arr scripts | +| **Transcodes** | Ramdisk symlink routing for Emby | `transcode_manager.sh` | +| **Tools** | Recovery and situational utilities | `failover_state_reset.sh`, `zfs_pool_scrub.sh` | +| **unRAID_Essentials** | Server-level maintenance and last-resort recovery | `system_watchdog.sh` | + +--- + +## Key Design Principles + +**One config file.** `Master.conf` is the only file you edit. No hunting through scripts to change a container name or a threshold. + +**Bidirectional.** Both servers run identical scripts. `detect_hosts()` determines local vs remote at runtime. One codebase covers both directions. + +**Self-healing layers.** Problems are addressed at the most targeted level first: +``` +docker_watchdog.sh — container level, minimal disruption +system_watchdog.sh — system level, last resort +failover.sh — infrastructure level, other server covers +``` + +**Strike systems, not hair triggers.** Single spikes don't cause restarts or reboots. Sustained problems do. The strike system filters noise from genuine issues. + +**Notifications when action is needed, silence otherwise.** The ecosystem is designed to run without daily attention. You hear from it when something needs human intervention — not as a regular occurrence. + +**Flash drive friendly.** Scripts that write to `/boot/` use bounded files with auto-purge. `/tmp/` is used for ephemeral state that resets on reboot. Most scripts write nothing at all. + +**--dry-run everywhere.** Every script supports `--dry-run`. Test before you schedule. + +--- + +## Scheduling Overview + +```bash +# ━━━ At Startup of Array ━━━ +failover.sh # background task — continuous loop +ramdisk_setup.sh +docker_syslog_filter.sh +php_fpm_max_children.sh +docker_network_connect.sh + +# ━━━ Frequent ━━━ +*/3 * * * * transcode_manager.sh +*/5 * * * * transcode_cleanup.sh +*/10 * * * * webgui_restart.sh +*/15 * * * * docker_watchdog.sh +*/15 * * * * system_watchdog.sh + +# ━━━ Daily ━━━ +0 1 * * * daily_sync.sh +0 2 * * * media_management.sh +0 3 * * * docker_daily_restart.sh +0 8 * * * weekly_health_digest.sh + +# ━━━ Weekly — Sunday ━━━ +0 3 * * 0 docker_weekly_restart.sh +0 5 * * 0 clear_logs.sh +0 6 * * 0 zfs_memory_snapshot.sh +0 7 * * 0 smart_health.sh +0 9 * * 0 cert_monitor.sh +0 10 * * 0 backup_verify.sh +0 11 * * 0 emby_session_report.sh +0 11 * * 0 bandwidth_monitor.sh --report +``` + +--- + +## Folder READMEs + +Each folder has a detailed README covering setup, configuration, usage, and the reasoning behind design decisions. + +| README | Contents | +|--------|----------| +| [README-Failover.md](Failover/README-Failover.md) | DDNS split brain prevention, tiered failover, handback sequence, initial setup, troubleshooting | +| [README-Monitors.md](Monitors/README-Monitors.md) | All monitor scripts, flash drive write policy, scheduling | +| [README-Orchestrators.md](Orchestrators/README-Orchestrators.md) | Why orchestrators exist, job ordering, how to add jobs | +| [README-Rsync.md](Rsync/README-Rsync.md) | Profile system, SSH key setup, Tailscale requirements | +| [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md) | Two-tier watchdog, startup grace, dependency ordering, skip list management | +| [README-Media.md](Media/README-Media.md) | Execution order, cleaner profiles, arr cleanup safety procedure | +| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Symlink indirection design, Docker mount warning, mode switching, sizing | +| [README-Tools.md](Tools/README-Tools.md) | All utility scripts, when to use each, how to add new tools | +| [README-unRAID_Essentials.md](unRAID_Essentials/README-unRAID_Essentials.md) | system_watchdog tiers, startup sequence, scheduled maintenance | + +--- + +## v2 Roadmap + +The v1 ecosystem is production-proven and stable. These are the features planned for v2 — some require additional infrastructure, some are extensions of existing systems. + +### Transcode Manager — Advanced Mode + +The current `smart` / `ramdisk` / `ssd` modes route all sessions to one location. v2 adds an `advanced` mode that routes by media type: + +```bash +TRANSCODE_MANAGER_MODE="advanced" + +TRANSCODE_FORCE_RAMDISK=( + "LiveTv" # always ramdisk — buffering is latency sensitive +) +TRANSCODE_FORCE_SSD=( + "Audio" # music/downloads — no benefit from ramdisk +) +# Everything else follows smart threshold behavior +``` + +The groundwork is already in place — session display with media type parsing is working. Advanced mode requires Emby to expose media type at the point ffmpeg resolves the symlink, which means the routing decision needs to happen before the session starts. This is the problem to solve. + +### Plugin Dashboard + +The Monitors folder is already the data backend for a future unRAID plugin dashboard. Every monitor script writes structured state that a plugin could read and display: + +``` +Failover state → live status indicator +Container watchdog → strike counts, skip list +SMART health → per-drive status +ZFS pools → health + ARC utilization +Transcode sessions → live session display +Bandwidth history → transfer trend charts +Certificate expiry → days remaining per domain +``` + +The scripts exist. The data exists. The plugin is the frontend. + +### Failover — Tiered by Content Type + +Currently failover starts containers based on time elapsed. A future enhancement would start containers based on what the primary server was doing before it went down: + +``` +Primary running Live TV sessions → start Live TV stack on secondary immediately +Primary idle → standard tier delays apply +Primary in heavy transcoding → start Emby immediately, defer others +``` + +This requires the failover state file to track active session types, which requires integration with the Emby API at failover trigger time. + +### Health Digest — Plugin Integration + +Currently the digest sends a notification. In v2 it populates a persistent dashboard that shows a rolling week of system health at a glance — without requiring a notification for every event. + +### Container Health Checks — Standardised Library + +The docker_watchdog already has HTTP check support per container. A v2 enhancement is a standardised health check library — pre-built check commands for every container in the stack that can be applied via Extra Parameters in the unRAID template without manual research per container. + +--- + +## Origin + +This ecosystem grew from a single failover script — written as a first real bash project, refined through months of production use, redesigned multiple times as the stack grew more complex. Each script in the collection started as a one-off solution to a specific problem. Over time the patterns that worked were extracted into `common.sh`, the configuration was centralised into `Master.conf`, and the whole collection was standardised into what it is now. + +The failover script that started it all is still the soul of the ecosystem — the DDNS sequencing, the two-ping state machine, the handback order. Everything else built on top of that foundation. + +Two servers, one codebase, self-healing infrastructure. Step away. Come back to a happy system. \ No newline at end of file diff --git a/Rsync/README_Rsync_Setup.md b/Rsync/README-Rsync_Setup.md similarity index 100% rename from Rsync/README_Rsync_Setup.md rename to Rsync/README-Rsync_Setup.md diff --git a/Tools/README-Tools.md b/Tools/README-Tools.md new file mode 100644 index 0000000..5e01367 --- /dev/null +++ b/Tools/README-Tools.md @@ -0,0 +1,262 @@ +# Tools + +Utility scripts for specific operational situations — recovery, repair, migration, and one-time tasks that don't fit the scheduled maintenance model of the other folders. + +``` +unRAID_Essentials/ — regular system maintenance, scheduled +Docker_Essentials/ — regular container management, scheduled +Monitors/ — regular health reporting, scheduled +Tools/ — situational utilities, run when needed +``` + +--- + +## What Belongs Here + +A script belongs in Tools when it: + +- Solves a specific operational problem rather than ongoing maintenance +- Is run manually in response to a situation rather than on a schedule +- Is used rarely — recovery scenarios, repairs, migrations, initial setup +- Would be dangerous or meaningless to run routinely +- Doesn't fit cleanly into any of the other folders + +Tools scripts are not scheduled. They sit here ready for when you need them. + +--- + +## Scripts + +### `failover_state_reset.sh` + +Resets the failover state file to NORMAL manually. + +```bash +/mnt/user/appdata/unraid_scripts/Tools/failover_state_reset.sh --status +/mnt/user/appdata/unraid_scripts/Tools/failover_state_reset.sh --dry-run +/mnt/user/appdata/unraid_scripts/Tools/failover_state_reset.sh +``` + +**When you need this:** + +After failover testing, a failed handback, or manual intervention that left the state file inconsistent. The `failover.sh` state machine reads this file on every cycle — if it shows `FAILOVER` when the system is actually in `NORMAL` operation, the script will make incorrect decisions. + +**What it does:** Rewrites the state file with `state=NORMAL` and clears all tier flags. Does NOT start or stop any containers — state file only. + +**⚠️ Verify first:** Only run after manually confirming both servers are in their correct states — right containers running on the right server, DDNS pointing correctly. The reset doesn't check any of this — it just trusts you. + +**Confirmation required:** Type `YES` to proceed — prevents accidental runs. + +--- + +### `watchdog_skip_list_manager.sh` + +View and manage the persistent container skip list used by `docker_watchdog.sh`. + +```bash +# View current skip list and restart history +/mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --status + +# Clear a specific container +/mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear Authelia + +# Clear everything +/mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear-all +``` + +**When you need this:** + +When a container hits the restart loop limit and gets added to the skip list — it stops being monitored until manually cleared. The `--status` view shows which containers are on the list and whether they're currently running, so you can see at a glance what needs attention. + +**After clearing a container:** +1. Fix whatever caused the failure +2. Start it manually: `docker start ContainerName` +3. The watchdog monitors it normally on the next cycle + +**Files managed:** +``` +/boot/config/system_watchdog_failed.db — skip list +/boot/config/container_restart_history.db — restart loop tracking +``` + +Both are cleared per-container or together. The restart history is also cleared when clearing a specific container — gives it a fresh slate for the loop protection window. + +--- + +### `bulk_permissions_repair.sh` + +Applies correct permissions to a single share or specific path. Faster than running the full `media_shares_permissions.sh` which processes every share. + +```bash +# Single share +/mnt/user/appdata/unraid_scripts/Tools/bulk_permissions_repair.sh /mnt/user/Movies + +# Multiple shares +/mnt/user/appdata/unraid_scripts/Tools/bulk_permissions_repair.sh \ + /mnt/user/Movies /mnt/user/Tv_Shows + +# Dry run first +/mnt/user/appdata/unraid_scripts/Tools/bulk_permissions_repair.sh \ + /mnt/user/Movies --dry-run +``` + +**When you need this:** + +- A failed transfer left files owned by wrong user +- A container wrote files as root instead of `nobody:users` +- Manual file operations bypassed normal permission handling +- A new share needs permissions applied before the next nightly run + +Uses `PERMISSIONS_MODE` and `PERMISSIONS_OWNER` from `Master.conf` — same values as the full permissions script. Applies `chown` before `chmod` to ensure correct ownership before mode change. + +--- + +### `container_data_export.sh` + +Exports a container's appdata directory to a compressed tar archive. Stops the container before archiving for a clean consistent backup, restarts after. + +```bash +/mnt/user/appdata/unraid_scripts/Tools/container_data_export.sh \ + Emby \ + /mnt/media-servers/Media_Server/Emby \ + /mnt/user/Backups/ + +# Dry run — verify space and paths without stopping anything +/mnt/user/appdata/unraid_scripts/Tools/container_data_export.sh \ + Emby \ + /mnt/media-servers/Media_Server/Emby \ + /mnt/user/Backups/ \ + --dry-run +``` + +**Output filename:** `ContainerName_YYYY-MM-DD_HH-MM.tar.gz` + +**When you need this:** + +- Before a major container update you're not sure about +- Before migrating appdata to a different pool +- Before removing a container from the stack — archive its data first +- As a manual point-in-time backup before making significant config changes + +**Space check:** Script estimates required space from appdata size × 1.1 and aborts if the output directory doesn't have enough free space. The container is not stopped until the space check passes. + +**Recovery:** If archiving fails, the container is restarted before the script exits — it tries to leave things clean regardless of outcome. + +--- + +### `emby_database_repair.sh` + +Stops Emby, runs SQLite integrity checks on all Emby databases, and restarts. + +```bash +# Check and report (restarts Emby after) +/mnt/user/appdata/unraid_scripts/Tools/emby_database_repair.sh + +# Dry run — detect config path and show what would be checked +/mnt/user/appdata/unraid_scripts/Tools/emby_database_repair.sh --dry-run +``` + +**When you need this:** + +- Emby reports database errors in logs +- Unexpected Emby crashes with no clear cause +- Playback history or user data behaving strangely +- After a hard shutdown or power loss with Emby running + +**Databases checked:** + +| Database | Contains | If corrupted | +|----------|----------|--------------| +| `library.db` | Media library metadata | Delete — Emby rebuilds from media files | +| `users.db` | User accounts and settings | Delete resets all user accounts | +| `authentication.db` | API keys and sessions | Delete — keys regenerated on restart | +| `activity.db` | Activity log | Delete safely — log only | + +**Important:** This script checks and reports. It does NOT automatically delete or repair corrupted databases — that requires judgment about which database is corrupted and whether you have a backup. The summary provides specific guidance per database type. + +**Config path detection:** Automatically detects the Emby config path from Docker volume mounts — no configuration needed beyond `TRANSCODE_EMBY_CONTAINER` in `Master.conf`. + +--- + +### `zfs_pool_scrub.sh` + +Triggers ZFS scrub on all pools (or a specific pool), waits for completion, and reports results. + +```bash +# Scrub all pools (skips ZFS_REPORT_IGNORE_POOLS) +/mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh + +# Scrub a specific pool +/mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh gaming + +# Check current scrub status without starting a new one +/mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh --status + +# Dry run — show which pools would be scrubbed +/mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh --dry-run +``` + +**When you need this:** + +ZFS scrub reads every block on every pool and verifies checksums — it catches silent data corruption that would otherwise only surface when you try to read the corrupted data. Running monthly is recommended. + +**Safe to run while in use.** Scrub does not interrupt normal I/O — it runs in the background at low priority. The script polls every 60 seconds until all scrubs complete, then reports errors found. + +**Pool filtering:** Pools in `ZFS_REPORT_IGNORE_POOLS` are skipped during all-pool scrubs. To scrub an ignored pool explicitly, specify it by name. + +**Notifications:** +- Clean completion — normal notification with pool count and duration +- Errors found — warning notification listing affected pools + +--- + +## Scheduled Summary + +None. Tools are not scheduled — they run when needed. + +--- + +## Adding a New Tool + +When you encounter a situation that required manual bash commands to resolve — write a tool. You'll face it again. + +The pattern for a Tools script: + +```bash +#!/bin/bash +# Short description of what situation this solves. +# When to run it. +# Any warnings about destructive operations. + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/../Master.conf" +source "$SCRIPT_DIR/../common.sh" + +parse_args "$@" + +# Always: +# 1. Check for root +# 2. Support --dry-run +# 3. Confirm before destructive operations (read -p "Type YES:") +# 4. Notify on completion +``` + +Good candidates for future tools: +``` +array_migration.sh — move appdata from one pool to another + with container stop/start and path updates + +emby_metadata_refresh.sh — trigger full library refresh via API + useful after storage changes + +tailscale_rekey.sh — rotate Tailscale keys on both servers + with SSH key update on both ends +``` + +--- + +## Philosophy + +Tools exist because not every problem has a scheduled solution. Some things only need to happen once. Some things only happen after something goes wrong. Having a dedicated folder keeps the other folders clean and makes it obvious what runs routinely vs what runs situationally. + +Write the tool when you solve the problem. Store it here. Find it at 2am when you need it again. \ No newline at end of file diff --git a/Tools/bulk_permissions_repair.sh b/Tools/bulk_permissions_repair.sh new file mode 100644 index 0000000..c7efbb1 --- /dev/null +++ b/Tools/bulk_permissions_repair.sh @@ -0,0 +1,133 @@ +#!/bin/bash +# ----------------------------------------------------------------------------------------------- +# --------------------------------- Bulk Permissions Repair ------------------------------------ +# ----------------------------------------------------------------------------------------------- +# Applies correct permissions to a single share or specific path. +# Faster than running media_shares_permissions.sh which processes all shares. +# Use when a specific share has wrong ownership or permissions after: +# - A failed transfer that left files owned by wrong user +# - A container writing files as root instead of nobody:users +# - Manual file operations that bypassed normal permission handling +# - A new share that needs permissions applied before the next nightly run +# +# Usage: +# bulk_permissions_repair.sh /mnt/user/Movies +# bulk_permissions_repair.sh /mnt/user/Movies --dry-run +# bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows +# +# Uses PERMISSIONS_MODE and PERMISSIONS_OWNER from Master.conf. +# Supports --dry-run to show what would be changed without applying. +# ----------------------------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +source "$SCRIPT_DIR/../Master.conf" +source "$SCRIPT_DIR/../common.sh" + +parse_args "$@" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_GEAR Setup ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_GEAR Setup ━━━" + +if [[ "$EUID" -ne 0 ]]; then + error "Must be run as root" + exit 1 +fi + +success "Running as root" + +if [[ ${#PARSED_ARGS[@]} -eq 0 ]]; then + error "No paths specified" + error "Usage: bulk_permissions_repair.sh /path/to/share [/another/path]" + error " bulk_permissions_repair.sh /mnt/user/Movies --dry-run" + exit 1 +fi + +info "Mode: $PERMISSIONS_MODE" +info "Owner: $PERMISSIONS_OWNER" +[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permissions will be changed" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_PERMS Apply Permissions ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_PERMS Permissions Repair ━━━" + +START=$(date +%s) +PASS=() +FAIL=() + +for share_path in "${PARSED_ARGS[@]}"; do + [[ -z "$share_path" ]] && continue + + echo "" + echo "━━━ $ICON_PERMS $(basename "$share_path") ━━━" + + if [[ ! -d "$share_path" ]]; then + error "$share_path — not found" + FAIL+=("$share_path") + continue + fi + + # Count files for progress context + FILE_COUNT=$(find "$share_path" -type f 2>/dev/null | wc -l) + DIR_COUNT=$(find "$share_path" -type d 2>/dev/null | wc -l) + SIZE=$(du -sh "$share_path" 2>/dev/null | cut -f1) + + info "$share_path — $FILE_COUNT files, $DIR_COUNT dirs ($SIZE)" + + if [[ "$DRY_RUN" == true ]]; then + warn "DRY RUN — would apply: chmod -R $PERMISSIONS_MODE $share_path" + warn "DRY RUN — would apply: chown -R $PERMISSIONS_OWNER $share_path" + PASS+=("$(basename "$share_path")") + continue + fi + + # Apply ownership first — chmod after so files are owned correctly before mode change + info "Applying ownership: $PERMISSIONS_OWNER..." + chown -R "$PERMISSIONS_OWNER" "$share_path" 2>/dev/null + CHOWN_EXIT=$? + + info "Applying permissions: $PERMISSIONS_MODE..." + chmod -R "$PERMISSIONS_MODE" "$share_path" 2>/dev/null + CHMOD_EXIT=$? + + if [[ "$CHOWN_EXIT" -eq 0 && "$CHMOD_EXIT" -eq 0 ]]; then + success "$ICON_UNLOCKED $(basename "$share_path") — permissions applied" + PASS+=("$(basename "$share_path")") + else + error "$(basename "$share_path") — permission repair failed" + FAIL+=("$(basename "$share_path")") + fi +done + +END=$(date +%s) + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SUMMARY Summary ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━━━ $ICON_SUMMARY PERMISSIONS REPAIR SUMMARY ━━━━━" +echo "$ICON_PERMS Mode: $PERMISSIONS_MODE" +echo "$ICON_PERMS Owner: $PERMISSIONS_OWNER" +echo "$ICON_TIME Duration: $(format_duration $((END - START)))" +echo "" +echo " $ICON_SUCCESS Pass: ${#PASS[@]} $ICON_ERROR Fail: ${#FAIL[@]}" +echo "" +[[ ${#PASS[@]} -gt 0 ]] && for p in "${PASS[@]}"; do echo " $ICON_UNLOCKED $p"; done +[[ ${#FAIL[@]} -gt 0 ]] && for f in "${FAIL[@]}"; do echo " $ICON_ERROR $f"; done +echo "" + +if [[ "$DRY_RUN" == true ]]; then + echo "$ICON_WARN Status: DRY RUN — no changes made" +elif [[ ${#FAIL[@]} -gt 0 ]]; then + echo "$ICON_ERROR Status: SOME REPAIRS FAILED" + notify "Permissions repair failed on $(hostname) — failed shares: ${FAIL[*]}" "Permissions Repair" "warning" +else + echo "$ICON_DONE Status: $ICON_SUCCESS DONE" + notify "Permissions repair complete on $(hostname) — ${#PASS[@]} share(s) repaired" "Permissions Repair" "normal" +fi +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" \ No newline at end of file diff --git a/Tools/container_data_export.sh b/Tools/container_data_export.sh new file mode 100644 index 0000000..7acfcc8 --- /dev/null +++ b/Tools/container_data_export.sh @@ -0,0 +1,173 @@ +#!/bin/bash +# ----------------------------------------------------------------------------------------------- +# --------------------------------- Container Data Export -------------------------------------- +# ----------------------------------------------------------------------------------------------- +# Exports a container's appdata directory to a compressed tar archive. +# Stops the container before archiving and restarts it after — ensures clean consistent backup. +# +# Usage: +# container_data_export.sh ContainerName /path/to/appdata /path/to/output/dir +# container_data_export.sh Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/ +# +# Output file naming: +# ContainerName_YYYY-MM-DD_HH-MM.tar.gz +# +# Use before major container updates, pool migrations, or when archiving +# a container you are removing from the stack. +# +# Supports --dry-run to show what would be archived without making changes. +# ----------------------------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +source "$SCRIPT_DIR/../Master.conf" +source "$SCRIPT_DIR/../common.sh" + +parse_args "$@" + +# ----------------------------------------------------------------------------------------------- +# Args +# ----------------------------------------------------------------------------------------------- +CONTAINER_NAME="${PARSED_ARGS[0]:-}" +APPDATA_PATH="${PARSED_ARGS[1]:-}" +OUTPUT_DIR="${PARSED_ARGS[2]:-}" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_GEAR Setup ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_GEAR Setup ━━━" + +if [[ "$EUID" -ne 0 ]]; then + error "Must be run as root" + exit 1 +fi + +success "Running as root" + +if [[ -z "$CONTAINER_NAME" || -z "$APPDATA_PATH" || -z "$OUTPUT_DIR" ]]; then + error "Usage: container_data_export.sh " + error "Example: container_data_export.sh Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/" + exit 1 +fi + +if [[ ! -d "$APPDATA_PATH" ]]; then + error "Appdata path not found: $APPDATA_PATH" + exit 1 +fi + +if [[ ! -d "$OUTPUT_DIR" ]]; then + error "Output directory not found: $OUTPUT_DIR" + exit 1 +fi + +# Check free space — rough estimate: appdata size × 1.1 +APPDATA_SIZE_KB=$(du -sk "$APPDATA_PATH" 2>/dev/null | cut -f1) +OUTPUT_FREE_KB=$(df "$OUTPUT_DIR" --output=avail | tail -1 | tr -d ' ') +REQUIRED_KB=$(( APPDATA_SIZE_KB * 11 / 10 )) + +APPDATA_SIZE_H=$(du -sh "$APPDATA_PATH" 2>/dev/null | cut -f1) +OUTPUT_FREE_H=$(df -h "$OUTPUT_DIR" --output=avail | tail -1 | tr -d ' ') + +info "Container: $CONTAINER_NAME" +info "Appdata: $APPDATA_PATH ($APPDATA_SIZE_H)" +info "Output dir: $OUTPUT_DIR ($OUTPUT_FREE_H free)" + +if [[ "$OUTPUT_FREE_KB" -lt "$REQUIRED_KB" ]]; then + error "Insufficient space in $OUTPUT_DIR — need ~${APPDATA_SIZE_H}, have ${OUTPUT_FREE_H}" + exit 1 +fi + +success "Space check passed" +[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_STOP Stop Container ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_STOP Stop Container ━━━" + +CONTAINER_WAS_RUNNING=false +STATUS=$(docker inspect -f '{{.State.Running}}' "$CONTAINER_NAME" 2>/dev/null) + +if [[ "$STATUS" == "true" ]]; then + CONTAINER_WAS_RUNNING=true + info "$ICON_STOP Stopping $CONTAINER_NAME for clean export..." + if [[ "$DRY_RUN" == false ]]; then + docker stop "$CONTAINER_NAME" >/dev/null 2>&1 && \ + success "$ICON_STOPPED $CONTAINER_NAME stopped" || \ + { error "Failed to stop $CONTAINER_NAME"; exit 1; } + else + warn "DRY RUN — would stop $CONTAINER_NAME" + fi +else + info "$CONTAINER_NAME is not running — archiving as-is" +fi + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SYNC Archive ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_SYNC Archive ━━━" + +TIMESTAMP=$(date '+%Y-%m-%d_%H-%M') +ARCHIVE_NAME="${CONTAINER_NAME}_${TIMESTAMP}.tar.gz" +ARCHIVE_PATH="${OUTPUT_DIR}/${ARCHIVE_NAME}" + +info "Creating: $ARCHIVE_PATH" +START=$(date +%s) + +if [[ "$DRY_RUN" == false ]]; then + tar -czf "$ARCHIVE_PATH" -C "$(dirname "$APPDATA_PATH")" "$(basename "$APPDATA_PATH")" 2>/dev/null + TAR_EXIT=$? + + if [[ "$TAR_EXIT" -ne 0 ]]; then + error "Archive failed (exit code $TAR_EXIT)" + # Restart container before exiting + [[ "$CONTAINER_WAS_RUNNING" == true ]] && docker start "$CONTAINER_NAME" >/dev/null 2>&1 + exit 1 + fi + + ARCHIVE_SIZE=$(du -sh "$ARCHIVE_PATH" 2>/dev/null | cut -f1) + success "Archive created: $ARCHIVE_NAME ($ARCHIVE_SIZE)" +else + warn "DRY RUN — would create: $ARCHIVE_PATH" +fi + +END=$(date +%s) + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_START Restart Container ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_START Restart Container ━━━" + +if [[ "$CONTAINER_WAS_RUNNING" == true ]]; then + info "$ICON_START Restarting $CONTAINER_NAME..." + if [[ "$DRY_RUN" == false ]]; then + docker start "$CONTAINER_NAME" >/dev/null 2>&1 && \ + success "$ICON_STARTED $CONTAINER_NAME restarted" || \ + error "Failed to restart $CONTAINER_NAME — start it manually" + else + warn "DRY RUN — would restart $CONTAINER_NAME" + fi +else + info "$CONTAINER_NAME was not running — not restarting" +fi + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SUMMARY Summary ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━━━ $ICON_SUMMARY CONTAINER EXPORT SUMMARY ━━━━━" +echo "$ICON_CONTAINERS Container: $CONTAINER_NAME" +echo "$ICON_DISK Appdata: $APPDATA_PATH ($APPDATA_SIZE_H)" +echo "$ICON_SYNC Archive: ${ARCHIVE_NAME:-DRY RUN} ${ARCHIVE_SIZE:+($ARCHIVE_SIZE)}" +echo "$ICON_TIME Duration: $(format_duration $((END - START)))" +if [[ "$DRY_RUN" == true ]]; then + echo "$ICON_WARN Status: DRY RUN — no changes made" +else + echo "$ICON_DONE Status: $ICON_SUCCESS DONE" + notify "Container export complete — $CONTAINER_NAME archived to $ARCHIVE_NAME" "Container Export" "normal" +fi +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" \ No newline at end of file diff --git a/Tools/emby_database_repair.sh b/Tools/emby_database_repair.sh new file mode 100644 index 0000000..104edeb --- /dev/null +++ b/Tools/emby_database_repair.sh @@ -0,0 +1,214 @@ +#!/bin/bash +# ----------------------------------------------------------------------------------------------- +# --------------------------------- Emby Database Repair --------------------------------------- +# ----------------------------------------------------------------------------------------------- +# Stops Emby, runs SQLite integrity checks on all Emby databases, and restarts. +# Use when Emby reports database corruption, unexpected crashes, or playback state issues. +# +# Checks performed: +# integrity_check — full SQLite integrity verification per database file +# quick_check — faster check for common corruption patterns +# +# If corruption is found: +# Reports which database files are corrupted +# Does NOT automatically repair — corruption repair requires manual steps +# Provides guidance on next steps per database type +# +# Emby database files checked: +# library.db — media library metadata +# library.db-wal — write-ahead log (if exists) +# librarydb.db — legacy library database +# users.db — user accounts and settings +# authentication.db — API keys and sessions +# activity.db — activity log +# +# All configuration in Master.conf — uses TRANSCODE_EMBY_CONTAINER and EMBY_URL. +# Supports --dry-run to show what would be checked without stopping Emby. +# ----------------------------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +source "$SCRIPT_DIR/../Master.conf" +source "$SCRIPT_DIR/../common.sh" + +parse_args "$@" + +# Emby config path — inside the container it's /config, map to host path +# Detected from Docker inspect at runtime +EMBY_CONTAINER="$TRANSCODE_EMBY_CONTAINER" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_GEAR Setup ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_GEAR Setup ━━━" + +if [[ "$EUID" -ne 0 ]]; then + error "Must be run as root" + exit 1 +fi + +success "Running as root" + +if ! command -v sqlite3 >/dev/null 2>&1; then + error "sqlite3 not found — install sqlite package" + exit 1 +fi + +success "sqlite3 available" +[[ "$DRY_RUN" == true ]] && warn "DRY RUN — Emby will not be stopped" + +# Detect Emby config path from Docker mount +EMBY_CONFIG_HOST=$(docker inspect "$EMBY_CONTAINER" 2>/dev/null | \ + jq -r '.[] | .Mounts[] | select(.Destination == "/config") | .Source' 2>/dev/null) + +if [[ -z "$EMBY_CONFIG_HOST" ]]; then + error "Could not detect Emby config path from Docker mounts" + error "Make sure $EMBY_CONTAINER is the correct container name in Master.conf" + exit 1 +fi + +success "Emby config path: $EMBY_CONFIG_HOST" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_STOP Stop Emby ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_STOP Stop Emby ━━━" + +EMBY_WAS_RUNNING=false +STATUS=$(docker inspect -f '{{.State.Running}}' "$EMBY_CONTAINER" 2>/dev/null) + +if [[ "$STATUS" == "true" ]]; then + EMBY_WAS_RUNNING=true + warn "Stopping $EMBY_CONTAINER — active sessions will be interrupted" + if [[ "$DRY_RUN" == false ]]; then + docker stop "$EMBY_CONTAINER" >/dev/null 2>&1 && \ + success "$ICON_STOPPED $EMBY_CONTAINER stopped" || \ + { error "Failed to stop $EMBY_CONTAINER"; exit 1; } + sleep 3 # brief wait for file handles to release + else + warn "DRY RUN — would stop $EMBY_CONTAINER" + fi +else + info "$EMBY_CONTAINER is not running — proceeding with checks" +fi + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_HEALTH Database Integrity Check ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_HEALTH Database Integrity Check ━━━" + +START=$(date +%s) + +# Emby database files to check +DB_FILES=( + "data/library.db" + "data/librarydb.db" + "data/users.db" + "data/authentication.db" + "data/activity.db" +) + +PASS_DBS=() +FAIL_DBS=() +MISSING_DBS=() + +for db_rel in "${DB_FILES[@]}"; do + db_path="${EMBY_CONFIG_HOST}/${db_rel}" + db_name=$(basename "$db_rel") + + if [[ ! -f "$db_path" ]]; then + log "$db_name — not found, skipping" + MISSING_DBS+=("$db_name") + continue + fi + + DB_SIZE=$(du -sh "$db_path" 2>/dev/null | cut -f1) + info "$ICON_HEALTH Checking $db_name ($DB_SIZE)..." + + if [[ "$DRY_RUN" == true ]]; then + warn "DRY RUN — would check: $db_path" + continue + fi + + # Run integrity check + RESULT=$(sqlite3 "$db_path" "PRAGMA integrity_check;" 2>/dev/null) + EXIT_CODE=$? + + if [[ "$EXIT_CODE" -ne 0 ]]; then + error "$db_name — sqlite3 could not open database (may be locked or corrupt)" + FAIL_DBS+=("$db_name") + elif [[ "$RESULT" == "ok" ]]; then + success "$db_name — integrity check passed" + PASS_DBS+=("$db_name") + else + error "$db_name — integrity check FAILED" + echo "$RESULT" | head -10 | while IFS= read -r line; do + error " $line" + done + FAIL_DBS+=("$db_name") + fi +done + +END=$(date +%s) + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_START Restart Emby ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_START Restart Emby ━━━" + +if [[ "$EMBY_WAS_RUNNING" == true && "$DRY_RUN" == false ]]; then + docker start "$EMBY_CONTAINER" >/dev/null 2>&1 && \ + success "$ICON_STARTED $EMBY_CONTAINER restarted" || \ + error "Failed to restart $EMBY_CONTAINER — start it manually" +elif [[ "$DRY_RUN" == true && "$EMBY_WAS_RUNNING" == true ]]; then + warn "DRY RUN — would restart $EMBY_CONTAINER" +else + info "$EMBY_CONTAINER was not running — not restarting" +fi + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SUMMARY Summary ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━━━ $ICON_SUMMARY EMBY DATABASE REPAIR SUMMARY ━━━━━" +echo "$ICON_HEALTH Container: $EMBY_CONTAINER" +echo "$ICON_HEALTH Config path: $EMBY_CONFIG_HOST" +echo "$ICON_TIME Duration: $(format_duration $((END - START)))" +echo "" +echo " $ICON_SUCCESS Passed: ${#PASS_DBS[@]} $ICON_ERROR Failed: ${#FAIL_DBS[@]} $ICON_INFO Missing: ${#MISSING_DBS[@]}" +echo "" + +if [[ ${#PASS_DBS[@]} -gt 0 ]]; then + for db in "${PASS_DBS[@]}"; do echo " $ICON_SUCCESS $db"; done +fi +if [[ ${#FAIL_DBS[@]} -gt 0 ]]; then + for db in "${FAIL_DBS[@]}"; do echo " $ICON_ERROR $db"; done +fi + +echo "" + +if [[ "$DRY_RUN" == true ]]; then + echo "$ICON_WARN Status: DRY RUN — no checks performed" +elif [[ ${#FAIL_DBS[@]} -gt 0 ]]; then + echo "$ICON_ERROR Status: CORRUPTION FOUND" + echo "" + echo "$ICON_INFO Next steps for corrupted databases:" + echo " library.db — Stop Emby, delete library.db, restart" + echo " Emby will rebuild from media files (slow first start)" + echo " users.db — Stop Emby, restore from backup or delete" + echo " Deleting resets all user accounts" + echo " authentication.db — Stop Emby, delete, restart" + echo " API keys and sessions will be regenerated" + echo " activity.db — Stop Emby, delete, restart — activity log only" + echo "" + echo "$ICON_WARN Always take a backup before deleting any database file" + notify "Emby database corruption found on $(hostname) — failed: ${FAIL_DBS[*]} — manual intervention needed" "Emby DB Repair" "warning" +else + echo "$ICON_DONE Status: $ICON_SUCCESS ALL DATABASES HEALTHY" + notify "Emby database integrity check passed on $(hostname) — ${#PASS_DBS[@]} databases healthy" "Emby DB Repair" "normal" +fi +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" \ No newline at end of file diff --git a/Tools/failover_state_reset.sh b/Tools/failover_state_reset.sh new file mode 100644 index 0000000..37989c9 --- /dev/null +++ b/Tools/failover_state_reset.sh @@ -0,0 +1,127 @@ +#!/bin/bash +# ----------------------------------------------------------------------------------------------- +# --------------------------------- Failover State Reset --------------------------------------- +# ----------------------------------------------------------------------------------------------- +# Resets the failover state file to NORMAL and clears all tier flags. +# Use when the failover state file is stuck in a non-NORMAL state after testing, +# a failed handback, or manual intervention that left state inconsistent. +# +# Does NOT start or stop any containers — state file only. +# After reset, failover.sh will resume from NORMAL on its next cycle. +# +# ⚠️ Only run this when you have manually verified both servers are in their +# correct states — right containers running on the right server, DDNS correct. +# Resetting state without verifying the actual state can cause failover.sh +# to make incorrect decisions on its next cycle. +# +# Supports --dry-run to show what would be reset without changing anything. +# Supports --status to show the current state file contents. +# ----------------------------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +source "$SCRIPT_DIR/../Master.conf" +source "$SCRIPT_DIR/../common.sh" + +parse_args "$@" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_GEAR Setup ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_GEAR Setup ━━━" + +if [[ "$EUID" -ne 0 ]]; then + error "Must be run as root" + exit 1 +fi + +success "Running as root" +[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SUMMARY Current State ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_SUMMARY Current State ━━━" + +if [[ ! -f "$FAILOVER_STATE_FILE" ]]; then + warn "State file not found: $FAILOVER_STATE_FILE" + warn "Will be created fresh on reset" +else + info "State file: $FAILOVER_STATE_FILE" + echo "" + while IFS='=' read -r key value; do + [[ -z "$key" ]] && continue + echo " $ICON_INFO $key = $value" + done < "$FAILOVER_STATE_FILE" +fi + +if [[ "$SHOW_STATUS" == true ]]; then + exit 0 +fi + +# ----------------------------------------------------------------------------------------------- +# ━━━ Confirmation ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +warn "$ICON_WARN This will reset the failover state to NORMAL" +warn "Only proceed if you have verified both servers are in their correct states" +warn " — Right containers running on the right server" +warn " — DDNS pointing at the correct server" +warn " — No active failover in progress" +echo "" + +if [[ "$DRY_RUN" == false ]]; then + read -r -p "Type YES to confirm reset: " CONFIRM + if [[ "$CONFIRM" != "YES" ]]; then + info "Reset cancelled" + exit 0 + fi +fi + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_FAILOVER Reset State File ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_FAILOVER Resetting State File ━━━" + +if [[ "$DRY_RUN" == true ]]; then + warn "DRY RUN — would write:" + echo " state=NORMAL" + echo " failover_start=0" + echo " handback_strikes=0" + echo " tier2_started=false" + echo " tier3_started=false" + echo " tier4_started=false" + echo " last_reset=$(date '+%Y-%m-%d %H:%M:%S')" +else + mkdir -p "$(dirname "$FAILOVER_STATE_FILE")" + cat > "$FAILOVER_STATE_FILE" << EOF +state=NORMAL +failover_start=0 +handback_strikes=0 +tier2_started=false +tier3_started=false +tier4_started=false +last_reset=$(date '+%Y-%m-%d %H:%M:%S') +EOF + success "State file reset to NORMAL" +fi + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SUMMARY Summary ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━━━ $ICON_SUMMARY FAILOVER STATE RESET SUMMARY ━━━━━" +if [[ "$DRY_RUN" == true ]]; then + echo "$ICON_WARN Status: DRY RUN — no changes made" +else + echo "$ICON_DONE Status: $ICON_SUCCESS State reset to NORMAL" + echo "$ICON_TIME Reset at: $(date '+%Y-%m-%d %H:%M:%S')" + echo "" + echo "$ICON_INFO failover.sh will resume from NORMAL on next cycle" + echo "$ICON_INFO No containers were started or stopped" + notify "Failover state manually reset to NORMAL on $(hostname)" "Failover State Reset" "normal" +fi +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" \ No newline at end of file diff --git a/Tools/watchdog_skip_list_manager.sh b/Tools/watchdog_skip_list_manager.sh new file mode 100644 index 0000000..9b8c6b8 --- /dev/null +++ b/Tools/watchdog_skip_list_manager.sh @@ -0,0 +1,175 @@ +#!/bin/bash +# ----------------------------------------------------------------------------------------------- +# --------------------------------- Watchdog Skip List Manager --------------------------------- +# ----------------------------------------------------------------------------------------------- +# View and manage the persistent container skip list used by docker_watchdog.sh. +# Containers are added to the skip list when they exceed the restart loop limit. +# They stay there until manually cleared or until found running again automatically. +# +# Usage: +# watchdog_skip_list_manager.sh --status — show current skip list and restart history +# watchdog_skip_list_manager.sh --clear-all — clear all skip lists and restart history +# watchdog_skip_list_manager.sh --clear ContainerName — clear specific container +# +# After clearing a container from the skip list: +# 1. Fix whatever was causing the container to fail +# 2. Start the container manually: docker start ContainerName +# 3. The watchdog will monitor it normally on the next cycle +# +# Files managed: +# SYS_WATCHDOG_FAILED_FILE — persistent container skip list +# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection +# ----------------------------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +source "$SCRIPT_DIR/../Master.conf" +source "$SCRIPT_DIR/../common.sh" + +parse_args "$@" + +# Parse action from args +ACTION="" +TARGET_CONTAINER="" + +for arg in "${PARSED_ARGS[@]}"; do + case "$arg" in + --clear-all) ACTION="clear-all" ;; + --clear) ACTION="clear" ;; + --status) ACTION="status" ;; + *) + [[ "$ACTION" == "clear" && -z "$TARGET_CONTAINER" ]] && TARGET_CONTAINER="$arg" + ;; + esac +done + +[[ -z "$ACTION" ]] && ACTION="status" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_GEAR Setup ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_GEAR Setup ━━━" + +if [[ "$EUID" -ne 0 ]]; then + error "Must be run as root" + exit 1 +fi + +success "Running as root" +[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made" + +touch "$SYS_WATCHDOG_FAILED_FILE" "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null + +# ----------------------------------------------------------------------------------------------- +# ━━━ STATUS ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_WATCHDOG Skip List Status ━━━" + +SKIP_COUNT=$(grep -c "." "$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null || echo 0) +RESTART_COUNT=$(wc -l < "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null || echo 0) + +if [[ "$SKIP_COUNT" -eq 0 ]]; then + success "Skip list is empty — all containers healthy" +else + warn "$SKIP_COUNT container(s) on skip list:" + while IFS= read -r container; do + [[ -z "$container" ]] && continue + # Check if container is currently running + STATUS=$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null || echo "unknown") + if [[ "$STATUS" == "true" ]]; then + echo " $ICON_RUNNING $container — currently RUNNING (will auto-clear on next watchdog cycle)" + elif [[ "$STATUS" == "false" ]]; then + echo " $ICON_STOPPED $container — currently STOPPED — fix and start manually" + else + echo " $ICON_INFO $container — container not found" + fi + done < "$SYS_WATCHDOG_FAILED_FILE" +fi + +echo "" +echo "━━━ $ICON_WATCHDOG Restart History ━━━" +if [[ "$RESTART_COUNT" -eq 0 ]]; then + success "No restart history" +else + info "$RESTART_COUNT restart entries (window: ${WATCHDOG_CONTAINER_RESTART_WINDOW}h)" + echo "" + # Show per-container restart counts + awk -F'|' '{counts[$1]++} END {for (c in counts) printf " %-30s %d restart(s)\n", c, counts[c]}' \ + "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null | sort +fi + +[[ "$ACTION" == "status" ]] && exit 0 + +# ----------------------------------------------------------------------------------------------- +# ━━━ CLEAR ALL ━━━ +# ----------------------------------------------------------------------------------------------- +if [[ "$ACTION" == "clear-all" ]]; then + echo "" + echo "━━━ $ICON_TRASH Clear All Skip Lists ━━━" + warn "This will clear the skip list and restart history for ALL containers" + + if [[ "$DRY_RUN" == false ]]; then + read -r -p "Type YES to confirm: " CONFIRM + if [[ "$CONFIRM" != "YES" ]]; then + info "Cancelled" + exit 0 + fi + + > "$SYS_WATCHDOG_FAILED_FILE" + > "$WATCHDOG_CONTAINER_RESTART_LOG" + success "Skip list cleared" + success "Restart history cleared" + notify "Watchdog skip list manually cleared on $(hostname) — all containers will be monitored normally" "Watchdog Manager" "normal" + else + warn "DRY RUN — would clear: $SYS_WATCHDOG_FAILED_FILE" + warn "DRY RUN — would clear: $WATCHDOG_CONTAINER_RESTART_LOG" + fi +fi + +# ----------------------------------------------------------------------------------------------- +# ━━━ CLEAR SPECIFIC CONTAINER ━━━ +# ----------------------------------------------------------------------------------------------- +if [[ "$ACTION" == "clear" ]]; then + echo "" + echo "━━━ $ICON_TRASH Clear Container: $TARGET_CONTAINER ━━━" + + if [[ -z "$TARGET_CONTAINER" ]]; then + error "No container specified. Usage: --clear ContainerName" + exit 1 + fi + + if ! grep -q "^${TARGET_CONTAINER}$" "$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null; then + warn "$TARGET_CONTAINER is not on the skip list" + else + if [[ "$DRY_RUN" == false ]]; then + sed -i "/^${TARGET_CONTAINER}$/d" "$SYS_WATCHDOG_FAILED_FILE" + success "$TARGET_CONTAINER removed from skip list" + else + warn "DRY RUN — would remove $TARGET_CONTAINER from skip list" + fi + fi + + # Clear restart history for this container + HIST_COUNT=$(grep -c "^${TARGET_CONTAINER}|" "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null || echo 0) + if [[ "$HIST_COUNT" -gt 0 ]]; then + if [[ "$DRY_RUN" == false ]]; then + sed -i "/^${TARGET_CONTAINER}|/d" "$WATCHDOG_CONTAINER_RESTART_LOG" + success "Cleared $HIST_COUNT restart history entries for $TARGET_CONTAINER" + else + warn "DRY RUN — would clear $HIST_COUNT restart history entries" + fi + else + info "No restart history for $TARGET_CONTAINER" + fi + + echo "" + echo "$ICON_INFO Next steps:" + echo " 1. Fix whatever was causing $TARGET_CONTAINER to fail" + echo " 2. Start it manually: docker start $TARGET_CONTAINER" + echo " 3. Watchdog will monitor it normally on the next cycle" +fi + +echo "" +echo "━━━━━ $ICON_SUMMARY DONE ━━━━━" \ No newline at end of file diff --git a/Tools/zfs_pool_scrub.sh b/Tools/zfs_pool_scrub.sh new file mode 100644 index 0000000..75c0f4d --- /dev/null +++ b/Tools/zfs_pool_scrub.sh @@ -0,0 +1,193 @@ +#!/bin/bash +# ----------------------------------------------------------------------------------------------- +# --------------------------------- ZFS Pool Scrub --------------------------------------------- +# ----------------------------------------------------------------------------------------------- +# Triggers a ZFS scrub on all pools (or a specific pool) and waits for completion. +# Sends a notification when scrub completes with a summary of any errors found. +# +# ZFS scrub reads every block on every pool and verifies checksums — it catches +# silent data corruption that would otherwise only surface when you try to read +# the corrupted data. Running monthly is recommended for all ZFS pools. +# +# Usage: +# zfs_pool_scrub.sh — scrub all pools +# zfs_pool_scrub.sh poolname — scrub specific pool only +# zfs_pool_scrub.sh --status — show scrub status for all pools +# zfs_pool_scrub.sh --dry-run — show what would be scrubbed +# +# Pools in ZFS_REPORT_IGNORE_POOLS are skipped unless specified explicitly. +# Scrub runs in background — script polls until complete then reports. +# Safe to run while the pool is in use — scrub does not interrupt normal I/O. +# ----------------------------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +source "$SCRIPT_DIR/../Master.conf" +source "$SCRIPT_DIR/../common.sh" + +parse_args "$@" + +TARGET_POOL="${PARSED_ARGS[0]:-}" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_GEAR Setup ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_GEAR Setup ━━━" + +if [[ "$EUID" -ne 0 ]]; then + error "Must be run as root" + exit 1 +fi + +success "Running as root" + +if ! command -v zpool >/dev/null 2>&1; then + error "ZFS not available on this system" + exit 1 +fi + +success "ZFS available" +[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no scrubs will be started" + +# Build ignore map +declare -A IGNORE_MAP +for pool in "${ZFS_REPORT_IGNORE_POOLS[@]}"; do + [[ -n "$pool" ]] && IGNORE_MAP["$pool"]=1 +done + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SUMMARY Status ━━━ +# ----------------------------------------------------------------------------------------------- +if [[ "$SHOW_STATUS" == true ]]; then + echo "" + echo "━━━━━ $ICON_SUMMARY SCRUB STATUS ━━━━━" + zpool list -H -o name 2>/dev/null | while read -r pool; do + SCAN=$(zpool status "$pool" 2>/dev/null | grep "scan:") + echo " $ICON_ZFS $pool — $SCAN" + done + echo "━━━━━━━━━━━━━━━━━━━━━━━" + exit 0 +fi + +# ----------------------------------------------------------------------------------------------- +# Build pool list to scrub +# ----------------------------------------------------------------------------------------------- +POOLS_TO_SCRUB=() + +if [[ -n "$TARGET_POOL" ]]; then + # Specific pool requested — validate it exists + if ! zpool list "$TARGET_POOL" >/dev/null 2>&1; then + error "Pool not found: $TARGET_POOL" + exit 1 + fi + POOLS_TO_SCRUB=("$TARGET_POOL") +else + # All pools — skip ignored ones + while IFS= read -r pool; do + [[ -z "$pool" ]] && continue + if [[ -n "${IGNORE_MAP[$pool]:-}" ]]; then + info "Skipping $pool (in ZFS_REPORT_IGNORE_POOLS)" + continue + fi + POOLS_TO_SCRUB+=("$pool") + done < <(zpool list -H -o name 2>/dev/null) +fi + +if [[ ${#POOLS_TO_SCRUB[@]} -eq 0 ]]; then + warn "No pools to scrub" + exit 0 +fi + +info "Pools to scrub: ${POOLS_TO_SCRUB[*]}" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_ZFS Start Scrubs ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_ZFS Starting ZFS Scrubs ━━━" +START=$(date +%s) + +for pool in "${POOLS_TO_SCRUB[@]}"; do + info "$ICON_ZFS Starting scrub on $pool..." + if [[ "$DRY_RUN" == false ]]; then + zpool scrub "$pool" 2>/dev/null && \ + success "$pool scrub started" || \ + error "Failed to start scrub on $pool" + else + warn "DRY RUN — would scrub: $pool" + fi +done + +[[ "$DRY_RUN" == true ]] && { + echo "" + echo "━━━━━ $ICON_SUMMARY ZFS SCRUB SUMMARY ━━━━━" + echo "$ICON_WARN Status: DRY RUN — no scrubs started" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + exit 0 +} + +# ----------------------------------------------------------------------------------------------- +# ━━━ Poll until complete ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_TIME Waiting for scrubs to complete ━━━" +info "Polling every 60 seconds — this may take a while on large pools" +info "Safe to leave running — scrub continues even if this script is stopped" + +STILL_RUNNING=true +while [[ "$STILL_RUNNING" == true ]]; do + sleep 60 + STILL_RUNNING=false + for pool in "${POOLS_TO_SCRUB[@]}"; do + STATUS=$(zpool status "$pool" 2>/dev/null | grep "scan:" | grep -c "in progress" || true) + if [[ "$STATUS" -gt 0 ]]; then + STILL_RUNNING=true + REPAIRED=$(zpool status "$pool" 2>/dev/null | grep "scan:" | grep -oE "[0-9]+ repaired") + log "$pool — scrub in progress ${REPAIRED:+($REPAIRED)}" + fi + done +done + +END=$(date +%s) +success "All scrubs complete" + +# ----------------------------------------------------------------------------------------------- +# ━━━ $ICON_SUMMARY Results ━━━ +# ----------------------------------------------------------------------------------------------- +echo "" +echo "━━━ $ICON_ZFS Scrub Results ━━━" + +POOLS_OK=() +POOLS_ERRORS=() + +for pool in "${POOLS_TO_SCRUB[@]}"; do + SCAN_LINE=$(zpool status "$pool" 2>/dev/null | grep "scan:") + ERRORS=$(zpool status "$pool" 2>/dev/null | grep "errors:" | grep -v "No known data errors") + + if [[ -n "$ERRORS" ]]; then + error "$pool — $SCAN_LINE" + error "$pool — $ERRORS" + POOLS_ERRORS+=("$pool") + else + success "$pool — $SCAN_LINE" + POOLS_OK+=("$pool") + fi +done + +echo "" +echo "━━━━━ $ICON_SUMMARY ZFS SCRUB SUMMARY ━━━━━" +echo "$ICON_ZFS Pools scrubbed: ${#POOLS_TO_SCRUB[@]}" +echo "$ICON_SUCCESS Clean: ${#POOLS_OK[@]}" +echo "$ICON_ERROR Errors: ${#POOLS_ERRORS[@]}" +echo "$ICON_TIME Duration: $(format_duration $((END - START)))" +echo "" + +if [[ ${#POOLS_ERRORS[@]} -gt 0 ]]; then + echo "$ICON_ERROR Status: ERRORS FOUND — ${POOLS_ERRORS[*]}" + notify "ZFS scrub complete on $(hostname) — ERRORS found in pools: ${POOLS_ERRORS[*]}" "ZFS Scrub" "warning" +else + echo "$ICON_DONE Status: $ICON_SUCCESS ALL POOLS CLEAN" + notify "ZFS scrub complete on $(hostname) — ${#POOLS_OK[@]} pools clean in $(format_duration $((END - START)))" "ZFS Scrub" "normal" +fi +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" \ No newline at end of file diff --git a/Transcodes/README-Transcoding.md b/Transcodes/README-Transcoding.md new file mode 100644 index 0000000..043fd39 --- /dev/null +++ b/Transcodes/README-Transcoding.md @@ -0,0 +1,317 @@ +# Transcodes + +Ramdisk-based transcode storage management for Emby using filesystem symlink indirection. + +--- + +## The Problem + +Emby transcodes video on the fly for clients that can't play the source format directly. Each transcode session generates hundreds of small HLS segment files that are written and read continuously. Where those files live has a significant impact on performance: + +- **Hard drives** — too slow for simultaneous multi-stream transcoding. Seek times cause buffering. +- **SSD (cache pool)** — fast enough, but constant small file writes accelerate wear over time. +- **RAM (tmpfs)** — fastest possible, no wear, disappears cleanly when sessions end. + +A ramdisk is the ideal transcode location. The only risk is running out of RAM during heavy load — which is where this system comes in. + +--- + +## The Design + +### Symlink Indirection + +Emby is pointed at a fixed path that never changes: + +``` +/mnt/ram-transcode → [currently: /mnt/ramdisk_transcodes] +``` + +This is a symlink. Emby doesn't know or care what's on the other end — it just writes to `/mnt/ram-transcode`. The transcode manager controls where that path actually points by updating the symlink target. + +**The critical insight:** ffmpeg resolves the symlink path **once** at session start. After that, it has a direct reference to the actual directory. This means: + +- **Existing sessions are never affected by symlink changes** +- When the symlink flips from ramdisk to SSD, sessions already in progress keep writing to ramdisk until they end naturally +- Only **new** sessions care about where the symlink currently points + +This is what makes the fallback seamless. Users never experience a glitch. + +### Why Not Just Use the SSD Directly? + +You could point Emby directly at the SSD and skip the ramdisk entirely. Many setups do this. The ramdisk approach gives you: + +1. **Faster performance** — RAM is orders of magnitude faster than SSD for small random writes +2. **Zero SSD wear** — transcode segments are written and deleted constantly. On a busy server this adds up to significant SSD wear over months and years +3. **Automatic cleanup** — tmpfs is released back to the system when files are deleted. No fragmentation, no stale files surviving a crash +4. **Session isolation** — each session's files disappear completely when the session ends + +--- + +## The Scripts + +### `ramdisk_setup.sh` +**Run at array start. Run once.** + +Creates the tmpfs ramdisk, the SSD fallback directory, and the symlink. If the ramdisk is already mounted it reports status and exits cleanly — safe to run multiple times. + +```bash +# Scheduled as: At Startup of Array +/mnt/user/appdata/unraid_scripts/Transcodes/ramdisk_setup.sh +``` + +What it creates: +``` +/mnt/ramdisk_transcodes/ ← tmpfs mount (RAMDISK_SIZE ceiling) +/mnt/ram-transcode ← symlink pointing at ramdisk +/mnt/cache/Temp_Storage/Emby/Transcodes/ ← SSD fallback directory +``` + +After running, verify: +```bash +mountpoint /mnt/ramdisk_transcodes # should say "is a mountpoint" +readlink /mnt/ram-transcode # should show /mnt/ramdisk_transcodes +``` + +--- + +### `transcode_manager.sh` +**Run every 3 minutes via cron.** + +Monitors ramdisk usage and manages the symlink direction. The main brain of the system. + +```bash +# Scheduled as: */3 * * * * +/mnt/user/appdata/unraid_scripts/Transcodes/transcode_manager.sh +``` + +#### Operating Modes + +Set `TRANSCODE_MANAGER_MODE` in `Master.conf`: + +| Mode | Behavior | Use When | +|------|----------|----------| +| `smart` | Auto-flips between ramdisk and SSD based on thresholds | Normal operation — default | +| `ramdisk` | Always uses ramdisk, never flips to SSD | Light load, guaranteed RAM performance | +| `ssd` | Always uses SSD, never uses ramdisk | Ramdisk maintenance, post-flip drain | + +#### Smart Mode — How the Flip Works + +``` +Ramdisk usage rises above RAMDISK_WARN_GB (6.8GB) + → Symlink flips to SSD + → New sessions land on SSD + → Existing sessions continue on ramdisk until they end + +Ramdisk usage drops below RAMDISK_LOW_GB (5.5GB) + → Symlink flips back to ramdisk + → New sessions land on ramdisk again +``` + +The gap between `RAMDISK_WARN_GB` and `RAMDISK_LOW_GB` (1.3GB) is the **hysteresis gap**. It prevents the symlink from flip-flopping when usage hovers near the threshold. Without this gap you'd get constant flipping on a busy system. + +#### Safety Checks + +Every run, regardless of mode: + +| Condition | Action | +|-----------|--------| +| Ramdisk not mounted | Flip symlink to SSD immediately, notify warning | +| SSD path missing | Disable fallback, notify warning. If mode is `ssd` — exit | +| Symlink missing | Recreate pointing at ramdisk, notify | +| Symlink target gone | Reset to ramdisk, notify | +| Permissions drift | Fix silently — `chmod` and `chown` applied every run | +| Emby not running | Skip threshold checks, verify symlink only | + +#### Session Display + +Each run queries the Emby API and shows active streams: + +``` +━━━ 🎬 Active Emby Sessions ━━━ + 🎬 Total: 7 | 💨 Live TV: 5 | 🔄 Transcoding: 5 | 🏁 Direct: 2 + + 🔗 Storage: 💨 ramdisk + + 🎬 Gmer4Lfe — MLB: Pirates vs Nationals — Live TV — Transcode + 🎬 Rebecca — MLB: Pirates vs Nationals — Live TV — Transcode + 🎬 Sunny — AT&T Sportsnet Pittsburgh — Live TV — Transcode + 🎬 jaden — TNT — Live TV — Transcode + 🎬 Mama Bear — Con-Text — TV Show — Direct Stream +``` + +**Split state** is detected and displayed when sessions exist on both ramdisk and SSD simultaneously — this happens naturally when the symlink flips while sessions are in progress: + +``` + ⚠️ Split state — 4 folder(s) on ramdisk / 2 on SSD + ⚠️ Older sessions remain on original location until they end naturally + 🔗 Storage: 💨 ramdisk (4) + 💾 SSD (2) +``` + +> **Why per-session location isn't shown:** Emby's internal transcode folder names don't match the session IDs returned by the API — there is no reliable way to map a specific user to a specific folder. The folder count on each location gives you the picture you need at a glance without false precision. + +#### Flip Frequency Warning + +If the symlink flips `TRANSCODE_FLIP_WARN` or more times in one hour, a notification is sent. This is a signal that `RAMDISK_SIZE` may need to be increased. Real production data from this setup: + +``` +Normal load (2-3 streams) → ~1.5-2.0GB +Busy evening (5-6 streams) → ~3.5-4.5GB +Peak (8 streams, live TV) → ~5.2GB +Threshold trigger → 6.8GB +``` + +--- + +### `transcode_cleanup.sh` +**Run every 5 minutes via cron.** + +Removes old inactive transcode files from both ramdisk and SSD. Never deletes files that are currently open by any process. + +```bash +# Scheduled as: */5 * * * * +/mnt/user/appdata/unraid_scripts/Transcodes/transcode_cleanup.sh +``` + +#### Deletion Rules + +A file is eligible for deletion only when **all** of these are true: + +1. Older than `TRANSCODE_MAX_AGE` minutes (default: 20 min) +2. Not currently open by any process + +#### Performance Design + +`lsof` is called **once per location** to build a complete list of open files — not once per file. This is critical on a busy Live TV system where a single location can have thousands of HLS segment files. A per-file `lsof` approach stalls the system under load. + +--- + +## Docker Mount — Critical + +Emby must be configured with **one transcode mount only:** + +``` +Host path: /mnt/ram-transcode/ +Container path: /ext-ram-transcode +``` + +**Do NOT add a static SSD transcode path as a second volume mount.** + +If the SSD path is mounted inside the container, Emby can see it as an accessible transcode location and will route sessions there independently of the symlink — completely bypassing the management system. This is not obvious and causes confusing split behavior that is hard to diagnose. + +The symlink handles all routing. One mount is all that's needed. + +> **This was learned in production.** The system worked correctly once the static SSD mount was removed. The symptom was new sessions landing on SSD even when the symlink pointed at ramdisk. + +#### Emergency Manual Flip + +If you need to manually redirect all new transcodes to SSD: + +```bash +ln -sfn /mnt/cache/Temp_Storage/Emby/Transcodes /mnt/ram-transcode +``` + +To flip back to ramdisk: + +```bash +ln -sfn /mnt/ramdisk_transcodes /mnt/ram-transcode +``` + +Existing sessions are unaffected — only new sessions follow the new target. + +--- + +## Configuration + +All configuration in `Master.conf` under the `── TRANSCODES ──` section. + +```bash +# Paths +RAMDISK_PATH="/mnt/ramdisk_transcodes" # tmpfs mount point +RAMDISK_SIZE="8G" # ceiling — only uses RAM actually needed +TRANSCODE_LINK="/mnt/ram-transcode" # symlink — location never changes +TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" + +# Smart mode thresholds +RAMDISK_WARN_GB=6.8 # flip to SSD above this +RAMDISK_LOW_GB=5.5 # flip back to ramdisk below this +RAMDISK_SSD_MIN_GB=20 # minimum SSD free space before allowing flip + +# Cleanup +TRANSCODE_MAX_AGE=20 # minutes before file eligible for cleanup +TRANSCODE_ORPHAN_AGE=30 # minutes for orphaned files + +# Alerts +TRANSCODE_FLIP_WARN=3 # notify if symlink flips this many times per hour + +# Permissions +TRANSCODE_OWNER="nobody:users" +TRANSCODE_CHMOD="755" + +# Mode +TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd + +# Emby check +TRANSCODE_CHECK_EMBY=true +TRANSCODE_EMBY_CONTAINER="Emby" +``` + +### Sizing the Ramdisk + +The ramdisk is a `tmpfs` — it only uses RAM that is actually needed. `RAMDISK_SIZE` is a ceiling, not a reservation. An 8GB ramdisk that holds 2GB of files only uses 2GB of RAM. + +**Rule of thumb for sizing:** +- Count your maximum expected concurrent transcoding streams +- Multiply by ~0.5-1GB per stream (Live TV HLS streams use more than standard transcodes) +- Add 20-30% headroom above your threshold + +**From production data on this setup:** +- 5 Live TV streams + 2 standard = ~4.5GB +- 8 streams peak = ~5.2GB +- Current ramdisk = 8GB with 6.8GB threshold — comfortable headroom + +If you regularly hit `TRANSCODE_FLIP_WARN` or see 3+ flips per hour, increase `RAMDISK_SIZE` by 2GB and adjust thresholds accordingly. + +--- + +## Sizing Thresholds + +When adjusting `RAMDISK_SIZE`, adjust thresholds to match: + +| Ramdisk Size | RAMDISK_WARN_GB | RAMDISK_LOW_GB | +|-------------|-----------------|----------------| +| 6G | 4.8 | 3.5 | +| 8G | 6.8 | 5.5 | +| 10G | 8.5 | 7.0 | +| 12G | 10.0 | 8.5 | + +Keep a 1.0-1.5GB hysteresis gap between WARN and LOW. A gap smaller than this causes flip-flop behavior near the threshold. + +--- + +## Scheduled Run Summary + +| Script | Schedule | Purpose | +|--------|----------|---------| +| `ramdisk_setup.sh` | At Startup of Array | Create ramdisk and symlink | +| `transcode_manager.sh` | `*/3 * * * *` | Monitor usage, manage symlink, display sessions | +| `transcode_cleanup.sh` | `*/5 * * * *` | Remove old inactive files | + +--- + +## Version 2 Roadmap + +A future `advanced` mode is planned that allows per-media-type storage routing: + +```bash +TRANSCODE_MANAGER_MODE="advanced" + +TRANSCODE_FORCE_RAMDISK=( + "LiveTv" # always ramdisk — buffering is latency sensitive +) +TRANSCODE_FORCE_SSD=( + "Audio" # music downloads — no benefit from ramdisk +) +# Everything else follows smart threshold behavior +``` + +This requires the Emby API to expose media type at session start — the groundwork (session display and media type parsing) is already in place. Target: this fall. \ No newline at end of file diff --git a/unRAID_Essentials/README-Unraid_Essentials.md b/unRAID_Essentials/README-Unraid_Essentials.md new file mode 100644 index 0000000..d8b830b --- /dev/null +++ b/unRAID_Essentials/README-Unraid_Essentials.md @@ -0,0 +1,319 @@ +# unRAID Essentials + +System-level scripts that act on the unRAID server itself — not containers, not media, not monitoring. These scripts keep the server healthy, respond to problems, and handle graceful shutdown and restart sequences. + +``` +Monitors/ — observes and reports +Docker_Essentials/ — acts on containers +unRAID_Essentials/ — acts on the server itself (this folder) +``` + +Most scripts in this folder are either scheduled at array start or run on a weekly maintenance schedule. A few are utilities called manually or by other scripts. All support `--dry-run`. + +--- + +## Scripts + +### `system_watchdog.sh` + +**The last line of defense.** Reboots the system cleanly when it is about to become unstable. Works alongside `docker_watchdog.sh` — containers are the first line of healing, system reboot is the last resort. + +```bash +# Scheduled as: */15 * * * * (every 15 minutes) +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh +``` + +**Relationship with docker_watchdog.sh:** +``` +docker_watchdog.sh — container level, tries to self-heal first + memory limits, CPU, HTTP checks, restarts + +system_watchdog.sh — system level, acts when healing has failed + reboots when the server itself is unstable +``` + +**What it checks** (all individually toggleable in `Master.conf`): + +| Check | Threshold | Why It Matters | +|-------|-----------|----------------| +| rootfs usage | 95% | Fills rapidly when array is down — crash imminent | +| /var/log usage | 95% | Log spam filling rootfs — indicates something broken | +| Free RAM | 4GB | Critically low RAM means OOM kills or swap imminent | +| ZFS ARC pinned | 98% | ARC not releasing after reclaim — memory stuck | +| CPU temperature | 95°C | Sustained tjmax causes throttling or kernel panic | +| Load average | cores × 3 | Sustained high load — something stuck or runaway | +| Zombie processes | 50 | Large zombie count — serious process management failure | +| Docker daemon | responsive | Unresponsive daemon means containers cannot be managed | +| Required containers | running | Stopped containers the watchdog couldn't recover | + +**Strike system:** + +Checks use a strike system — a single spike doesn't trigger a reboot. The threshold must be hit on consecutive cycles. `SYS_WATCHDOG_STRIKE_LIMIT=2` means two consecutive 15-minute cycles above the threshold before acting. Single spikes are ignored. + +**Abort conditions:** + +Some conditions prevent a reboot even if thresholds are hit: + +| Condition | Default | Reason | +|-----------|---------|--------| +| ZFS pool unhealthy | abort | Rebooting with bad pool risks data loss | +| Parity running | reboot anyway | Aborting parity beats crashing mid-check | +| Mover running | reboot anyway | Aborting move beats crashing mid-move | + +Set `true` to abort reboot if condition is active. Set `false` to reboot regardless. Philosophy: a graceful reboot before a crash is always better than a hard crash mid-operation. + +**Reboot loop protection:** + +Tracks reboot timestamps in a persistent log on `/boot/` — survives reboots. If the server reboots `SYS_WATCHDOG_REBOOT_LIMIT` times within `SYS_WATCHDOG_REBOOT_WINDOW_HRS` hours it shuts down instead. A reboot loop means something is fundamentally broken that rebooting is not fixing — shutting down prevents hardware damage and gives you time to investigate. + +**State files:** +``` +/tmp/system_watchdog_state.db — strike counts (resets on reboot) +/boot/config/system_watchdog_failed.db — container skip list (persistent) +/boot/config/system_watchdog_reboots.db — reboot timestamps (persistent) +``` + +--- + +### `webgui_restart.sh` + +Monitors the unRAID WebGUI and restarts it automatically if unresponsive. + +```bash +# Scheduled as: */10 * * * * (every 10 minutes) +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/webgui_restart.sh +``` + +**Escalation path:** +``` +1. curl WebGUI → unresponsive +2. Restart nginx → wait WEBGUI_NGINX_WAIT seconds → recheck +3. Still unresponsive → restart emhttp → wait WEBGUI_EMHTTP_WAIT seconds → recheck +4. Still unresponsive → notify warning — manual intervention needed +``` + +**Why emhttp is more disruptive:** + +`nginx` is the web server layer — restarting it is fast and clean. `emhttp` is the core unRAID management daemon — it manages the array, Docker, VMs, and everything else. Restarting it takes longer but recovers cleanly. The escalation path tries the less disruptive option first. + +A notification is sent on any restart — nginx or emhttp — so you know what happened and when. + +**Configuration:** +```bash +WEBGUI_URL="http://localhost" # adjust if running non-standard port +WEBGUI_TIMEOUT=5 # seconds before curl gives up +WEBGUI_NGINX_WAIT=15 # wait after nginx restart before recheck +WEBGUI_EMHTTP_WAIT=30 # wait after emhttp restart — takes longer +``` + +--- + +### `docker_syslog_filter.sh` + +Suppresses noisy Docker network interface messages from the unRAID syslog. + +```bash +# Scheduled as: At Startup of Array +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/docker_syslog_filter.sh +``` + +**The problem it solves:** + +Every time Docker starts a container it creates virtual network interfaces (`veth` devices). Every time a container stops, they're removed. Each creation and removal generates syslog entries. On a server with 50+ containers starting at array start this creates hundreds of lines of noise that buries real log messages. + +The filter creates an rsyslog configuration file that suppresses these specific messages and restarts rsyslog to apply it. The filter file is recreated on every array start — rsyslog configuration doesn't survive reboots on unRAID. + +**Configuration:** +```bash +FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf" +``` + +--- + +### `php_fpm_max_children.sh` + +Sets the PHP-FPM `pm.max_children` value to allow more concurrent requests to the unRAID WebGUI. + +```bash +# Scheduled as: At Startup of Array +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/php_fpm_max_children.sh +``` + +**Why this is needed:** + +unRAID's WebGUI is served via PHP-FPM. The default `pm.max_children` value is conservative. On a server with many users, plugins, or automated tools hitting the API simultaneously, the default value can cause requests to queue or time out. Increasing it allows more concurrent PHP processes. + +The setting does not survive reboots — PHP-FPM configuration is reset on each boot. Running this at array start ensures it's always applied. + +**Configuration:** +```bash +PHP_CONF="/etc/php-fpm.d/www.conf" +PHP_MAX_CHILDREN=250 # set based on available RAM + # each PHP worker uses ~30-50MB + # 250 workers × 40MB = ~10GB worst case +``` + +Set `PHP_MAX_CHILDREN` based on your available RAM. Higher values allow more concurrency but use more memory if all workers are active simultaneously. + +--- + +### `clear_logs.sh` + +Clears unRAID system log files and Docker container logs. + +```bash +# Scheduled as: 0 5 * * 0 (Sunday 5am weekly) +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh +``` + +**Why weekly log clearing:** + +unRAID system logs live in RAM (`/var/log/`) — they don't persist across reboots. However on a stable server that runs for weeks without rebooting, these logs grow continuously. Docker container logs can grow particularly large if a container is verbose. Both can fill rootfs if left unchecked. + +Weekly clearing keeps rootfs clean without being too aggressive. If you need to investigate a problem the week's logs are still available. + +**Configuration:** +```bash +LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg) +# Docker container logs are cleared automatically — no configuration needed +``` + +--- + +### `mover_stop.sh` + +Safely stops the unRAID mover with a warning delay before terminating. + +```bash +# Run manually when needed — not scheduled +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/mover_stop.sh +``` + +**When you need this:** + +The mover moves files from the cache pool to the array. If you need to stop it mid-run — before an array stop, before maintenance, or because it's been running too long — killing it directly can leave files in an inconsistent state. This script warns first and gives the mover time to finish its current file operation cleanly. + +**Configuration:** +```bash +MOVER_STOP_TIMEOUT=300 # seconds to wait before sending SIGTERM + # gives mover time to finish current file +``` + +--- + +### `rsync_stop.sh` + +Stops all running rsync processes on both local and remote servers. + +```bash +# Run manually when needed — not scheduled +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/rsync_stop.sh +``` + +**When you need this:** + +An rsync job may need to be stopped — before an array stop, because it's consuming too much bandwidth, or because it started at the wrong time. Simply killing rsync can leave containers stopped on the remote server (since rsync.sh stops containers before syncing and restarts them after). + +This script: +1. Kills rsync processes locally +2. Kills rsync processes on the remote via SSH +3. Checks all configured profile containers locally +4. Restarts any containers that were left stopped by the interrupted rsync + +The remote is left in whatever container state it was in — the remote server's watchdog handles recovery on its own. + +--- + +### `server_reboot.sh` + +Gracefully reboots the unRAID server with a warning delay. + +```bash +# Run manually when needed — not typically scheduled +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/server_reboot.sh +``` + +**Sequence:** +``` +1. Broadcast warning to all logged-in users +2. Wait REBOOT_SLEEP seconds +3. Stop Docker gracefully +4. Stop VM Manager gracefully +5. Issue reboot +``` + +The warning gives users time to finish what they are doing — saving files in VS Code Server, finishing a download, wrapping up a session. The Docker and VM stop ensures containers and VMs shut down cleanly rather than being hard-killed by the reboot. + +**Configuration:** +```bash +REBOOT_SLEEP=300 # seconds of warning before reboot (default: 5 minutes) +``` + +**Note:** `system_watchdog.sh` calls this script automatically when thresholds are exceeded. You can also call it manually for planned maintenance. + +--- + +### `user_script_stop.sh` + +Stops all running User Scripts plugin jobs. + +```bash +# Run manually when needed — not typically scheduled +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/user_script_stop.sh +``` + +**When you need this:** + +Before stopping the array, before a reboot, or when a script has hung and needs to be cleared. The User Scripts plugin runs scripts in `/tmp/user.scripts/` — this script identifies all running processes with that path signature and terminates them cleanly. + +Useful before `server_reboot.sh` to ensure no scripts are mid-execution when the reboot happens. + +--- + +## Startup Sequence + +The recommended array start sequence for scripts in this folder: + +```bash +# At Startup of Array — in this order +ramdisk_setup.sh # Transcodes/ — creates ramdisk before anything uses it +docker_syslog_filter.sh # suppress veth noise before containers start +php_fpm_max_children.sh # WebGUI performance before anyone accesses it +docker_network_connect.sh # Docker_Essentials/ — connect containers to networks +``` + +These run once at array start. The order matters — ramdisk before containers, filter before logs fill with noise, PHP config before WebGUI requests arrive. + +--- + +## Scheduled Maintenance Summary + +```bash +# At Startup of Array +docker_syslog_filter.sh +php_fpm_max_children.sh + +# Every 10 minutes +*/10 * * * * webgui_restart.sh + +# Every 15 minutes +*/15 * * * * system_watchdog.sh + +# Weekly — Sunday morning +0 5 * * 0 clear_logs.sh +``` + +--- + +## --dry-run Support + +Every script in this folder supports `--dry-run`. Always test before scheduling: + +```bash +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/webgui_restart.sh --dry-run +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh --dry-run +/mnt/user/appdata/unraid_scripts/unRAID_Essentials/server_reboot.sh --dry-run +``` + +`server_reboot.sh --dry-run` is particularly useful — it walks through the entire shutdown sequence, shows what would be stopped, and exits without rebooting. \ No newline at end of file