━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🏠 VARAVERK

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

A complete self-healing, self-maintaining, mutually-redundant two-server home server ecosystem. One codebase runs on both servers. Both servers run their own lives independently. When one goes down the other covers it — not because of conventional fallback infrastructure, but because both servers already have copies of everything that matters and both know how to run it.

This is not conventional failover. There is no primary and standby. There is no shared storage. There is no cluster manager. There are two fully independent servers, living in different locations on different power utilities and isps, each running their own household's media stack — and a layer of scripts that keeps them synchronised, coordinated, and mutually covering. The goal is minimal disruption to a media stack, not enterprise high availability.

Started on Unraid, built for portability. The ecosystem runs on Unraid today. The platform adapter layer (Plugin/unraid/adapter.sh) isolates every OS-specific call so bash scripts stay clean. A Plugin/truenas/adapter.sh or Plugin/ubuntu/adapter.sh provides the same function names — load_config.sh detects the OS and sources the correct adapter. The scripts themselves never branch on the OS.

---

━━━ HOW THIS GOT HERE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

One server. One script. Three lines.

That's genuinely where this started. Something needed doing on a schedule, so I wrote three lines and moved on.

Then it was two servers, 50 miles apart, and about 60 scripts with nothing holding them together. Every one standalone. Every one with its own hardcoded paths, its own container names, its own idea of how to log something or send a notification.

And it worked. That's the part worth saying. The stack stayed up, the media kept playing, the backups kept running. It was not broken.

It was just miserable to change.

Every adjustment meant finding every script that touched the thing, on both machines, and editing each one the same way — then hoping I'd found them all. Miss one and it didn't error. It just quietly kept doing the old thing until I noticed weeks later. Two servers meant two copies of every mistake, and the two copies drifted apart a little more every time I touched them.

So it got rebuilt, in roughly this order:

git                 stop editing live scripts on two machines and calling it version control
common.sh           one place for log, notify, locking, host detection, retries
master.conf         thresholds and toggles — shared, one copy
host*.conf          credentials, paths, container names — per server
load_config.sh      sources all of it, in the right order, at the top of every script

Then the long part: refactoring 60 standalone scripts into that shape, one at a time, until every value lived in a conf file and every shared behaviour lived in common.sh. Change the retry count once, every script gets it. Rename a container in one place, everything follows.

That is the whole idea, and it's why the structure looks the way it does. The three-file config model and the shared library aren't design patterns picked out of a book — they're the specific answer to "I have 60 scripts on two machines and I'm scared to touch any of them."

That became Varaverk.

---

━━━ WHERE THIS ACTUALLY IS RIGHT NOW ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

This is a live system, not a finished product. HOST1 has been running the whole stack unattended for months. HOST2 is mid-rebuild, so some of what's described below is written and tested but currently switched off:

FALLBACK_ENABLED=true        ← running
PARTNERSHIP_ENABLED=true     ← running
RSYNC_ENABLED=false          ← off until HOST2 is back
CONF_SYNC_ENABLED=false      ← off until HOST2 is back

Everything here is built and audited. The two-server half is waiting on the second server. If a section reads like it's describing something that already happens every night — on HOST1, it is. Across both hosts, it will be.

I'd rather tell you that than have you find out by reading the toggles.

---

━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HOST1 — unRAID-Gmer4Lfe
  Hardware:  Threadripper 1950X, 128GB RAM, ZFS cache pools
  Location:  Primary site
  Domain:    Gmer4Lfe.com
  Arrs:      Movies, Tv_Shows, Music — union with HOST2 via arr_sync
             Auth stack — source of truth (HOST1 owned, syncs to HOST2)
             Emby — primary media server

HOST2 — unRAID-Jayred365
  Hardware:  Intel i5 10th gen, 64GB RAM
  Location:  Remote — 50 miles away, different power utility and ISP
  Domain:    Gmer4Lfe.us
  Arrs:      Anime_Shows, Anime_Movies — union with HOST1 via arr_sync
             His own Emby — for his household

Hardware does not need to match. Everything is accessed through /mnt/user/ — unRAID's unified share layer. HOST1 has a Threadripper with ZFS. HOST2 has a completely different CPU and fewer drives. A fallback container on HOST2 mounts /mnt/user/Movies and sees mirrored data because the share names match. The hardware underneath is irrelevant.

---

━━━ CONTAINER NAMING — YOU NEED TO KNOW THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

This is the most important convention in the ecosystem. Get it wrong and the scripts break silently.

---

── Shared Containers — No Suffix ───────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Shared containers — meaning both servers run this container and scripts treat
# them as the same logical service — use the plain name with no suffix.
#
# The convention: "no suffix = this container participates in the shared stack"
#
    Emby                    # the shared Emby — runs on HOST1, picked up by HOST2 on fallback
    NginxProxyManager       # shared NPM — runs on both, same config via sync
    Lldap                   # shared LLDAP — same user directory, both servers
    Authelia                # shared auth — both servers, same config
    Mariadb-Authelia        # Authelia's database — same on both
    Redis-Authelia          # Authelia's cache — same on both
    VaultWarden             # shared password manager
    Dispatcharr             # shared Live TV scheduler

When fallback.sh on HOST2 reads FALLBACK_HOST1_TIER1=(..."Emby"...), it looks for a container called exactly Emby and starts it. HOST2 must have a container called Emby — not Emby-Gmer4Lfe, not Emby-HOST1, just Emby — with its volume mounts pointing at the mirrored share paths. This is what makes one codebase work on both servers without modification.

---

── User-Specific Containers — Suffixed ────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# User-specific containers — things that belong to one person and only ever
# run on their server — use a suffix identifying the user/owner.
#
    Emby-Gmer4Lfe           # Gmer4Lfe's personal Emby instance (HOST1, never covered)
    Emby-Jayred365          # Jayred365's personal Emby instance (HOST2, never covered)
    Organizer-Gmer4Lfe      # Gmer4Lfe's dashboard — personal, stays on HOST1
    Sonarr-Gmer4Lfe         # Gmer4Lfe's Sonarr — managing his Tv_Shows share
    Radarr-Gmer4Lfe         # Gmer4Lfe's Radarr — managing his Movies share
    Immich-Gmer4Lfe         # Gmer4Lfe's photo library — personal

User-specific containers are covered during fallback (see Tiered Fallback below) but are distinct from the shared stack. They are never confused with each other because the suffix makes the owner unambiguous. Scripts targeting Emby and scripts targeting Emby-Gmer4Lfe are targeting different containers for different purposes.

---

── Why This Matters ─────────────────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# The entire ecosystem uses container names as identifiers.
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
#   → looks for exactly "NginxProxyManager" on the running server
# fallback.sh: FALLBACK_HOST1_TIER1=("Emby" "VaultWarden")
#   → looks for exactly "Emby" and "VaultWarden" on HOST2
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
#   → looks for exactly those names in docker ps
#
# A container named differently than expected is not found — no error, no restart,
# no fallback coverage. It silently does not happen.
# Keep names consistent on both servers for shared containers.
# ─────────────────────────────────────────────────────────────────────────────

---

━━━ SHARE NAMING — ALSO YOU NEED TO KNOW THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Shares must have identical names on both servers. The ecosystem uses share paths as sync targets and mount points — a mismatch is a broken path.

# ─────────────────────────────────────────────────────────────────────────────
# The rule: if a share exists on HOST1 and HOST2 needs a mirror of it,
# the share on HOST2 must have the exact same name.
#
# Same name on both servers:
    /mnt/user/Movies          # both servers — arr_sync.sh keeps Radarr in union
    /mnt/user/Tv_Shows        # both servers — arr_sync.sh keeps Sonarr in union
    /mnt/user/Music           # both servers — arr_sync.sh keeps Lidarr in union
    /mnt/user/Anime_Shows     # both servers — arr_sync.sh keeps Sonarr in union
    /mnt/user/Anime_Movies    # both servers — arr_sync.sh keeps Radarr in union
#
# rsync.sh syncs /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2.
# If HOST2 has it at /mnt/user/Movies-Mirror → rsync aborts: path not found.
# ─────────────────────────────────────────────────────────────────────────────

---

━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Two households, one library, and a hard line between "our media" and "your stuff." Everything in this section comes down to that split — media is pooled and shared freely, personal data never is, and no script gets to blur the two.

── Common Libraries — Shared Pool ─────────────────────────────────────────

Common media like Movies, Music, and TV Shows are a shared pool. Every container that needs music — Emby, LidaTube, Navidrome, whatever — mounts the same /mnt/user/Music share. There is one share, one location, one source of truth. No duplication of content across separate shares per-user or per-application.

# ─────────────────────────────────────────────────────────────────────────────
# One share for the library — all containers that need it mount the same path.
# Emby container: /mnt/user/Music → /media/Music (read)
# Lidarr:         /mnt/user/Music → /music       (read+write)
# LidaTube:       /mnt/user/Music → /downloads   (write)
# Navidrome:      /mnt/user/Music → /music       (read)
#
# All pointing at the same /mnt/user/Music. No per-user music copies.
# When Lidarr downloads and Emby scans — they're looking at the same files.
# ─────────────────────────────────────────────────────────────────────────────

---

── Both Arrs Manage All Shares — arr_sync.sh Is the Source of Truth ───────

# ─────────────────────────────────────────────────────────────────────────────
# Both servers run Sonarr, Radarr, and Lidarr — all pointing at the same
# share names. Either server can download anything to any share at any time.
#
# How it stays coherent:
#   arr_sync.sh runs first (before rsync) — bidirectional library union.
#   Both Sonarr instances know about every TV episode on either server.
#   Both Radarr instances know about every movie on either server.
#   Both Lidarr instances know about every album on either server.
#
# Then rsync spreads the files additively (no --delete).
# Any file downloaded on HOST1 propagates to HOST2 on the next cycle.
# Any file downloaded on HOST2 propagates to HOST1 on the next cycle.
#
# HOST1 downloads Tv_Shows episode:
#   → HOST1 Sonarr tracks it immediately
#   → arr_sync.sh runs → HOST2 Sonarr now tracks it
#   → daily rsync → file lands on HOST2
#
# HOST2 downloads the same show's next episode that same night:
#   → HOST2 Sonarr tracks it immediately
#   → arr_sync.sh runs → HOST1 Sonarr now tracks it
#   → daily rsync → file lands on HOST1
#
# Both servers converge. Neither owns the share. The union is the truth.
# ─────────────────────────────────────────────────────────────────────────────

---

── How arr_sync.sh Keeps the Union Stable ────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# arr_sync.sh runs as the fixed first step of intermediate_sync_maintenance.sh
# (every 4 hours). It syncs all arr libraries bidirectionally before any rsync.
#
# What it does:
#   Reads each arr's full library via the local API
#   SSHes to the remote, reads the remote arr library via its API
#   Adds anything missing on either side — union, not overwrite
#   Blocked items (blocklist) are excluded from sync
#
# What this means in practice:
#   No share has an owner. No server is read-only.
#   Either server can download content to any share at any time.
#   arr_sync.sh guarantees both arrs agree on what exists before files move.
#   rsync then spreads the files to make storage match what arrs already know.
#
# The per-arr cleanups (daily) remove true orphans — files no arr tracks.
# They use the union model too: a file is only an orphan if neither arr
# on either server has it indexed.
# ─────────────────────────────────────────────────────────────────────────────

---

━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Not everything needs to sync. Not everything should. Two households means some of this data is mine, some of it is his, and some of it belongs to both of us — and treating all three the same way is how you end up either leaking someone's photos or losing the thing you actually needed a copy of.

Where a piece of data lands in this table decides two things: whether it needs encryption, and whether a failed sync is something to fix tonight or something to shrug at.

---

── Shared Fallback Shares — Sync Required ─────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# These shares enable fallback — HOST2 must have current copies for the covered
# services to work correctly when they start on HOST2.
#
    /mnt/user/Media_Server/Emby              # Emby userdata — watch history, playstates
    /mnt/user/appdata-Fallback/Critical-Data # auth stack — NPM rules, certs, user accounts
    /mnt/user/Movies                         # media library content
    /mnt/user/Tv_Shows                       # media library content
    /mnt/user/Music                          # media library content
#
# Sync failure here = fallback runs but with stale data (acceptable) or missing
# data (problematic). These sync every night via daily_sync_maintenance.sh.
# Critical-Data and Emby also sync every 30 minutes via critical_sync_maintenance.sh.
# ─────────────────────────────────────────────────────────────────────────────

---

── Non-Fallback Shares — Sync as Backup, No Encryption Required ───────────

# ─────────────────────────────────────────────────────────────────────────────
# Some shares don't participate in fallback — their container only runs on one
# server and is covered during fallback from scratch (or not at all).
# These shares can sync as offsite backup without encryption.
# Content is not sensitive — anyone with access to HOST2 could read it.
#
# Examples:
    /mnt/user/appdata-Fallback/Gmer4Lfe      # Gmer4Lfe server appdata backup
    /mnt/user/appdata-Fallback/Important-Data # NextCloud, Immich appdata
    /mnt/user/Kids_Movies                    # kids content — not sensitive
    /mnt/user/Sports                         # sports content — not sensitive
#
# These sync in DAILY_SYNC_SHARES — straight rsync, no special handling.
# If HOST2 is compromised, the content is readable. That is acceptable.
# ─────────────────────────────────────────────────────────────────────────────

---

── Personal Shares — Always Encrypted ─────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Personal shares contain content only the owner should see — documents,
# photos, financial records, private files.
# ZFS encrypts at the dataset level — HOST2 receives encrypted blocks.
# Without your passphrase or keyfile, HOST2 cannot read the content.
#
    HOST1_PERSONAL_SHARES=(
        "/mnt/user/Gmer4Lfe-Personal"    # encrypted ZFS dataset
    )
    HOST2_PERSONAL_SHARES=(
        "/mnt/user/Jayred365-Personal"   # encrypted ZFS dataset
    )
#
# Append to HOST*_DAILY_SYNC_SHARES — synced nightly with everything else.
# Same rsync, same timing — but the content on HOST2 is ciphertext only you can unlock.
#
# Setup: README-Rsync_Setup.md → Step 8 — Personal Encrypted Shares
# ─────────────────────────────────────────────────────────────────────────────

---

━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Get this wrong and nothing else in here will make sense.

Every guide you'll find describes the same thing: a primary, a standby, shared storage or synchronous replication. The standby sits there doing nothing, existing only for the day the primary dies. Pacemaker, DRBD, Proxmox HA — all built for a datacenter where downtime is measured in dollars per second.

This is not that. Nobody is on call. There is no shared storage, because there is no datacenter — there are two houses. The standby isn't idle, because it isn't a standby: it's somebody's actual media server, and they're watching it right now.

The bar is not five nines. The bar is nobody notices their show stopped working.

---

── What This Actually Is ─────────────────────────────────────────────────────

Two fully independent servers, each running their own household's services.
Each server has its own domain, its own users, its own media library management.
Each server is the primary for its own services every single day.

The "fallback" component is additive:
  When HOST1 goes down → HOST2 starts HOST1's containers on top of its own
  When HOST2 comes back → HOST1's containers hand back cleanly
  HOST2 never stopped running its own services during this entire time

This is not standby. This is mutual coverage between peers.

---

── The Actual Goal: Minimal Disruption to a Media Stack ────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# "Minimal disruption" means:
#   Live TV keeps playing         → Dispatcharr is Tier 1 (starts immediately)
#   Passwords are accessible      → VaultWarden is Tier 1 (people lock themselves out)
#   Emby keeps serving            → Emby is Tier 1
#   Auth works                    → NPM, Authelia, LLDAP already warm on both servers
#
# "Minimal disruption" does NOT mean:
#   Zero downtime (DDNS TTL = 1 minute, brief gap expected)
#   Perfect consistency (Emby userdata is 30min stale at most)
#   All services covered immediately (tiered start over 24 hours)
#
# The system is designed around the real experience of a household:
#   "The TV stopped working" → unacceptable, fixed in Tier 1 (< 2 minutes)
#   "NextCloud is down"      → annoying, fixed in Tier 2 (4 hours)
#   "Gitea is down"          → unacceptable, fixed in Tier 1 (< 2 minutes)
#   "Sonarr isn't downloading" → fine, fixed in Tier 4 (24 hours)
# ─────────────────────────────────────────────────────────────────────────────

---

── What You Give Up With This Approach ─────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Things conventional failover provides that this does NOT:
#   Zero RPO (recovery point objective) — there is ~30min of potential staleness
#   Synchronous state replication — changes don't reach HOST2 instantly
#   Automatic failure detection in seconds — fallback.sh checks every 2 minutes
#   Transparent IP migration — DNS TTL means up to 1 minute of routing to old IP
#
# What you get instead:
#   Both servers run fully independently — no single point of failure in software
#   Geographic redundancy — different buildings, different power, different ISP
#   Both households stay operational during most disruption scenarios
#   Simple, auditable scripts — no cluster manager black box
# ─────────────────────────────────────────────────────────────────────────────

---

━━━ HOW FALLBACK ACTUALLY WORKS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

fallback.sh runs on both servers as a continuous background process. Every 2 minutes it pings two things: the other server's Tailscale IP, and 8.8.8.8. The combination of those two answers determines the current state.

Remote ping PASS + Internet PASS → NORMAL — both servers running independently

Remote ping FAIL + Internet PASS → FALLBACK — remote is down, start covering it
  Tier 1 immediately: DDNS + Emby + Auth + VaultWarden + Live TV + Gitea
  Tier 2 at 4 hours: NextCloud + Immich + Jellyseerr
  Tier 3 at 12 hours: AdGuard + Uptime monitoring
  Tier 4 at 24 hours: Arrs + download clients

Internet FAIL (either) → NO_INTERNET — stop own DDNS, wait
  Can't serve anyone without internet — stop claiming DNS ownership

Remote FAIL + Internet FAIL → DARK — conservative, same as NO_INTERNET

The DDNS sequencing is critical — one domain, one DDNS container, at all times. The handback sequence stops remote DDNS first, rsyncs, starts containers, starts local DDNS last. Split-brain DNS is prevented by design.

→ Full detail: README-Fallback.md

---

━━━ THE AUTH STACK — WARM ON BOTH SERVERS ALWAYS ━━━━━━━━━━━━━━━━━━━━━━━━━━

NPM, LLDAP, Authelia, MariaDB, Redis — all running actively on both servers at all times. Not cold standby. Not started at fallback. Running right now, serving each server's own domain.

# ─────────────────────────────────────────────────────────────────────────────
# Why this eliminates the fallback dead zone:
#
# Cold standby approach: HOST1 goes down → DNS switches to HOST2 → HOST2 starts
# auth containers → 30-60 seconds before auth is ready → users hit the auth wall
# before the service behind it responds → failed logins, broken sessions
#
# Warm approach: HOST1 goes down → DNS switches to HOST2 → auth is ALREADY
# running → users are served immediately with no dead zone
#
# HOST1 is source of truth for the auth configuration. All changes to NPM proxy
# rules, LLDAP users, Authelia policies, and certs happen on HOST1 and propagate
# to HOST2 via Critical-Data sync every 30 minutes.
#
# HOST2 operator's experience: click NginxProxyManager in Docker UI →
# browser opens HOST1's NPM via Tailscale redirect → make changes there →
# 30 minutes later live on both servers. Zero auth management overhead on HOST2.
# ─────────────────────────────────────────────────────────────────────────────

→ Full detail: README-Partnership.md

---

━━━ THE COMPLETE PICTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

                    HOST1 (Gmer4Lfe.com)        HOST2 (Gmer4Lfe.us)
                    ──────────────────          ──────────────────────

Always running:
  Own stack       Full service stack            Full service stack (anime focus)
  Auth            NPM, Authelia, LLDAP          NPM, Authelia, LLDAP (mirrored)

Source of truth:
  Media           Movies, Tv_Shows, Music  ←──  Anime_Movies, Anime_Shows
                  ───────────────────────→       (bidirectional daily push)
  Config          auth stack config        ──→   mirrored every 30 min
  Emby state      watch history, playstates──→   dirty sync every 30 min

Fallback coverage:
  If HOST1 down:  —                             starts Emby, VaultWarden,
                                                Dispatcharr, all arr stack
                                                serves Gmer4Lfe.com via DDNS

  If HOST2 down:  starts his Emby, his arrs,   —
                  serves Gmer4Lfe.us via DDNS

Monitoring:
  Both servers    resource/docker/system/        resource/docker/system/
                  stability_watchdog.sh         stability_watchdog.sh
                  fallback.sh                   fallback.sh
                  Sunday morning coffee report  Sunday morning coffee report

---

━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Folders are grouped by what a script does to the system, not by which service it happens to talk to. There's no Sonarr/ folder — cleaning up after Sonarr is media work, watching whether its container is alive is watchdog work, and those two things belong next to their own kind.

Every folder has its own README explaining the group, and every script has a header explaining itself. If you're looking for one specific thing, the README reference near the bottom of this page is the faster route.

varaverk/
│
├── Configurations/          ← All configuration files — the only files you edit regularly
│   ├── master.conf          ← Shared: thresholds, toggles, job lists, watchdog settings
│   ├── host1.conf           ← HOST1-specific: share lists, container names, API keys
│   └── host2.conf           ← HOST2-specific: same structure, different values
├── common.sh                ← Shared library — all functions used by every script
├── load_config.sh           ← Detects OS platform → sources Configurations/, common.sh, adapter
│
├── Plugin/                  ← Platform adapters + Unraid web UI
│   └── unraid/
│       ├── adapter.sh       ← Unraid adapter: platform_*() functions (rc.d, emhttp, dynamix)
│       ├── api/             ← PHP API endpoints
│       ├── include/         ← PHP business logic
│       ├── pages/           ← Per-tab page includes
│       └── ...              ← Full web UI (scheduler, monitor, docker, arrs, fallback)
│
├── Orchestrators/           ← Scheduled job runners — what Varaverk scheduler executes
│   README: README-Orchestrators.md
│
├── Rsync/                   ← Core rsync script — called per share with profile
│   README: README-Rsync_Setup.md
│
├── Fallback/                ← Mutual automatic fallback — continuous background process
│   README: README-Fallback.md
│
├── Watchdogs/               ← All watchdog scripts: resource, docker, system, stability
│   README: README-Watchdogs.md
│
├── Docker_Essentials/       ← Container lifecycle: restarts, updates, networks
│   README: README-Docker_Essentials.md
│
├── System_Essentials/       ← Server-level: WebGUI recovery, log hygiene, kernel tuning, reboot
│   README: README-System_Essentials.md
│
├── Media/                   ← Library health + behavior-driven discovery: permissions, junk cleanup, orphan removal, weekly arr adds
│   README: README-Media.md
│
├── Transcodes/              ← Ramdisk transcode management with SSD fallback
│   README: README-Transcoding.md
│
├── Monitors/                ← Read-only observation: certs, SMART, bandwidth, digests
│   README: README-Monitors.md
│
├── Partnership/             ← Two-server relationship lifecycle: onboard, offboard, transfer
│   README: README-Partnership.md
│
├── Kernel/                  ← Kernel module config — loaded at boot by go file
│   README: README-Kernel.md
│
├── Tools/                   ← Situational utilities: repair, export, emergency tools
│   README: README-Tools.md
│
└── data/                    ← Everything persisted on disk — the one on-disk root
    README: data/README.md      gitignored except that README; per-host, never synced
    ├── db/                  ← statistics, histories, counters, blocklists
    ├── state/               ← runtime state: watchdogs, fallback, transcode, setup
    ├── ai/                  ← retrieval index, memory, token ledger, bugs, saved chats
    ├── cache/               ← persistent backups of the tmpfs caches, and only those
    └── logs/                ← retained log output

State is data. It used to live in a sibling State_Files/, with the conf-cache backup off in a third place and the arr backups loose in data/'s root — no single decision wrong, but nothing answered "what does Varaverk keep on disk". STATE_DIR still exists and still means the same thing; only its value moved, which is why that restructure touched none of the scripts built on it.

The RAM half is separate on purpose. VV_CACHE_ROOT (/tmp/varaverk/) holds the WebGUI payload cache, the arr caches, the partner conf cache, partner token ledgers, and in-flight job files. It is read every second and rewritten by the hundred megabytes; it does not belong on flash. Both roots are defined once in master.conf and read by the shell and PHP layers alike.

---

━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━

This is the part that fixed the 60-script problem. Three files, one rule:

Thresholds and toggles go in master.conf. Credentials, paths and container names go in host*.conf. If you find yourself about to hardcode a value in a script, it belongs in one of these instead — that's the whole discipline, and everything else follows from it.

Sparse checkout means each server only ever receives its own host*.conf. HOST1 never has HOST2's credentials sitting on disk, and vice versa. The partner's conf arrives at runtime through the RAM cache, which is why conf_sync.sh exists.

# ─────────────────────────────────────────────────────────────────────────────
# All three conf files live in Configurations/. load_config.sh sources them.
# Sparse checkout ensures each server only receives its own host*.conf.
#
# load_config.sh does five things in order:
#   1. Detects OS platform (/etc/unraid-version → PLATFORM=unraid)
#   2. Sources master.conf
#   3. Auto-discovers and sources all host*.conf present
#   4. Sources common.sh
#   5. Sources Plugin/$PLATFORM/adapter.sh (platform_*() functions)
#
# master.conf — shared across both servers
#   Everything that applies equally to both: thresholds, schedules, profile
#   definitions, watchdog settings, arr cleanup config, DDNS timing, etc.
#   Pushed to both servers via git. Never contains server-specific values.
#
# host1.conf — sourced only on HOST1
#   HOST1_* prefixed variables: share lists, container names, API keys,
#   ramdisk size, specific paths, per-server toggle overrides.
#
# host2.conf — sourced only on HOST2
#   HOST2_* prefixed variables: same structure, different values.
#
# detect_hosts() in common.sh:
#   Reads the current server's hostname. Matches it against HOST1 and HOST2.
#   Aliases HOST1_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST1)
#   Aliases HOST2_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST2)
#   Every script uses the unprefixed name. The same script on either server
#   automatically uses the correct values. No hostname comparisons in scripts.
#
# Plugin/$PLATFORM/adapter.sh — OS-specific function implementations
#   Scripts call platform_restart_service(), platform_storage_healthy(), etc.
#   The adapter maps those to rc.d scripts, emhttp commands, and dynamix files
#   on Unraid — or to systemctl, mount checks, etc. on a future platform.
# ─────────────────────────────────────────────────────────────────────────────

---

━━━ WHAT RUNS WHEN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The Varaverk scheduler runs only orchestrators — never individual scripts directly.

Array start (Varaverk event hook):
  array_started.sh              launches all startup scripts in order
    → unraid_api_key_renew.sh re-register API key before anything needs it
    → conf_sync.sh            pull conf from remote before scripts need current config
    → conf_cache_restore.sh   restore cached conf if remote pull failed
    → ramdisk_setup.sh        create ramdisk before Emby starts
    → docker_syslog_filter.sh suppress veth noise before containers create interfaces
    → php_fpm_max_children.sh WebGUI tuning before first request
    → inotify_tuning.sh       raise kernel limits before containers start
    → docker_network_connect.sh connect containers to extra networks
    → start_webhook_listener.sh continuous — Node.js webhook server for arr upgrades
    → fallback.sh             continuous — mutual fallback state machine

Every 15 minutes:
  watchdog_orchestrator.sh    fires each watchdog in sequence
    → resource_watchdog.sh    reduce pressure before healing attempts
    → docker_watchdog.sh      two-tier container healing
    → system_watchdog.sh      system components: storage + WebGUI health
    → stability_watchdog.sh   last resort — reboots when all else fails

Every 7 minutes:
  transcode_management.sh     cleanup → manager (order non-negotiable)

Every 30 minutes:
  critical_sync_maintenance.sh  Critical-Data rsync + play_state_sync (Emby watch state via API) + partnership check

Every 4 hours:
  arrs_failed_stalled_recovery.sh   blocklist + re-search failed imports

1am daily:
  daily_sync_maintenance.sh   git pull → sync shares → media maintenance → restarts

2:30am Sunday:
  weekly_sync_maintenance.sh  stop → update images → clean sync → start → weekly restarts
                              + playback_aware_{lidarr,radarr,sonarr}_discovery.sh
                                (behaviour-driven discovery from real playback history)

8am daily:
  weekly_health_digest.sh     DIGEST_PROFILE controls when it actually notifies

Every 6 hours:
  system_tuning_monitor.sh    inotify + php-fpm utilisation snapshot

Sunday morning block (6–11am):
  zfs_memory_snapshot.sh, smart_health.sh, cert_monitor.sh,
  backup_verify.sh, emby_session_report.sh, bandwidth_monitor.sh --report

→ Complete schedule: README-Orchestrators.md

---

━━━ THE DAILY CYCLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

What actually happens on a typical day, from the ecosystem's perspective:

Throughout the day (every 15 minutes via watchdog_orchestrator.sh):
  resource_watchdog.sh        managing: system pressure (throttle/pause/stop)
  docker_watchdog.sh          healing: memory leaks, HTTP failures, required containers
  system_watchdog.sh          watching: system components (storage growth, WebGUI)
  stability_watchdog.sh       watching: RAM, CPU temp, rootfs, kernel, daemon

Throughout the day:
  fallback.sh                 watching: remote server, internet connectivity
  transcode_management.sh     managing: ramdisk ↔ SSD, session cleanup (every 7min)
  critical_sync_maintenance.sh keeping: auth stack + Emby current (every 30min)

1am:
  git pull (latest scripts first)
  rsync Movies, Tv_Shows, Music → HOST2        (HOST1 pushes its truth shares)
  rsync Anime_Shows, Anime_Movies → HOST1      (HOST2 pushes its truth shares)
  media_shares_permissions.sh                  (fix ownership before arr cleanup)
  media_cleaner.sh anime + media               (remove scene junk)
  lidarr_release_fixer.sh                      (fix wrong MusicBrainz editions first)
  lidarr_duplicate_artist_cleanup.sh           (drop phantom zero-file duplicates)
  lidarr/sonarr/radarr_cleanup.sh              (remove orphaned files — library side)
  arr_download_orphan_cleaner.sh               (remove orphans — download side)
  radarr/sonarr_classification_scan.sh --move  (anime/kids sitting in the wrong root)
  docker_update.sh → docker_daily_restart.sh   (pull first, then restart onto it)

2:30am Sunday:
  Stop Emby + auth stack both sides
  Pull updated Docker images (zero extra downtime — already stopped)
  rsync Emby + Critical-Data clean mirror
  Start containers on new images
  docker_update.sh --weekly → docker_weekly_restart.sh
  arr_corruption_scan.sh                       (ffprobe every tracked video)
  arr_full_rescan.sh                           (force a real disk↔database reconcile)

7am Sunday:
  sunday_morning_coffee_report.sh              (everything that happened this week)

Monthly (self-gated on 30-day uptime):
  ZFS scrub, SMART long tests
  docker_update.sh --remainder                 (everything not in the managed lists)

The ordering in the 1am block is not arbitrary. Permissions run before the cleanups because the cleanups need correct ownership to delete. The release fixer runs before the Lidarr cleanup so it isn't cleaning up files it's about to fix. Updates run before restarts so the restart lands on the new image. Change the order and things break quietly rather than loudly.

---

━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Nothing here jumps straight to the big hammer. Each layer only gets involved because the one below it already tried and failed — and the layer holding the biggest hammer, the one that can reboot the box, is the most reluctant of all:

Problem: Emby container using 22GB RAM (limit: 18GB)
  → docker_watchdog.sh Tier 1: restart Emby immediately
  → System continues, no reboot

Problem: Authelia crashed and won't restart
  → docker_watchdog.sh Tier 1: attempt restart (dependency check: DB + Redis first)
  → Fails 3 times in 1 hour → skip list → critical notification
  → Tools: watchdog_skip_list_manager.sh to clear after fixing

Problem: RAM drops to 3GB
  → stability_watchdog.sh: stop non-essential containers
  → Wait for RAM to recover to 30GB
  → docker_watchdog.sh defers (reads mem_shutdown_active=true)
  → If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot

Problem: Docker daemon hung
  → docker_watchdog.sh: attempt platform_restart_service docker (rc.d on Unraid)
  → If still hung: sets daemon_confirmed_down flag → stability_watchdog escalates → reboot

Problem: HOST1 loses power
  → fallback.sh on HOST2 detects at next 2-minute check
  → Tier 1 starts immediately: DDNS, Emby, VaultWarden, Live TV
  → Household barely notices. Live TV may buffer for < 2 minutes.

→ Container healing: README-Docker_Essentials.md → System stability: README-System_Essentials.md

---

━━━ GPU CONTAINERS — IGNORE GOOGLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Every tutorial, forum post, and Reddit thread shows the wrong way to add GPU access to a Docker container on unRAID. The old method breaks silently in ways that are very hard to diagnose. Read this before touching any GPU container.

The Wrong Way (What Google Shows)

# DO NOT DO THIS
Extra Parameters:  --runtime=nvidia
Variables:         NVIDIA_VISIBLE_DEVICES=GPU-xxxxxxxx
                   NVIDIA_DRIVER_CAPABILITIES=compute,video,utility

This is --runtime=nvidia — the old NVIDIA Container Toolkit approach. It is what every guide written before Docker 19.03 recommends. It still shows up in official unRAID forum posts, community applications templates, and most search results.

Why it breaks here:

1. --runtime=nvidia combined with NVIDIA_VISIBLE_DEVICES has caused conflicts historically and breaks on container rebuilds. With Docker 29.x (Unraid 7.3+), bind-propagation=shared crashes runc v1.3.5 entirely — with any container, GPU or not. The container fails to start with:

   OCI runtime create failed: unable to start container process:
   error jailing process inside rootfs:
   open /proc/self/mountinfo: no such file or directory
   

   The ramdisk tmpfs is already MS_SHARED at the kernel level. Do not use bind-propagation=shared in Extra Parameters on Unraid 7.3+.

2. When unRAID rebuilds a container from template (update, reinstall from Community Applications), the --runtime=nvidia field and the NVIDIA env vars can be dropped or corrupted. Rebuilding a container is a normal maintenance operation — it should not break GPU access. With the old method, it sometimes does.

The Right Way

# DO THIS — everything in Extra Parameters, nothing in Variables
--gpus "device=GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1" --pids-limit=0 --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode

--gpus "device=UUID" is Docker-native GPU support (Docker 19.03+, released 2019). One field in Extra Parameters. The UUID pins a specific device — no ambiguity on single-GPU systems. No env vars needed. No --runtime=nvidia. Survives container rebuilds. This is the correct method.

Find your GPU UUID: nvidia-smi -L

HOST1 GPU UUID (Quadro P2000): GPU-62e1659d-1ed4-935f-3df3-4bb4339438f1

GPU Memory and Media Servers

A failure mode that is extremely non-obvious and not documented anywhere:

Any container that uses VRAM and does not release it will starve Emby and Jellyfin of the GPU memory they need to start transcoding sessions. The two media servers respond differently:

• Jellyfin: hard fails — no session is created, playback stops entirely
• Emby: falls back to CPU transcoding silently and keeps going

The non-obvious source is OCR sidecars — credit detection plugins, subtitle extraction tools — that talk to a GPU-accelerated OCR container. PaddleOCR in particular loads a neural network into VRAM at first use and does not release it between runs. One credit scan, and VRAM stays consumed until the container restarts.

Emby startup probe: Emby runs a one-shot NVIDIA hardware detection at container start. If VRAM is exhausted when Emby starts, NVIDIA is marked unavailable for the entire session — no retry. Restarting Emby while VRAM is still full just fails the probe again. Correct recovery: free VRAM first, confirm with nvidia-smi, then restart Emby.

For the EmbyCredits plugin: use the yock1/embycreditocr Tesseract image (port 8884, CPU-based) instead of PaddleOCR. Same plugin, no VRAM.

→ Full detail: Manual-Transcoding.md

---

━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

If you're setting this up from scratch on two servers, the order below is not a suggestion. Steps 2 and 3 are cheap now and expensive later — renaming a container or a share after the confs, the fallback tiers and the sync profiles all reference it means touching every one of them, and missing one doesn't error.

Do the naming first. Everything after it assumes the names are already right.

1. Follow Manual.md — end-to-end setup guide
   Tailscale → SSH keys → git clone → master.conf → Varaverk scheduler schedule

2. Set your container names correctly before anything else
   Shared services: no suffix (Emby, NginxProxyManager, etc.)
   Personal services: suffixed (Emby-Gmer4Lfe, Organizer-Gmer4Lfe)

3. Set your share names identically on both servers
   /mnt/user/Movies on both, /mnt/user/Tv_Shows on both, etc.

4. Run initial sync to populate HOST2
   bash daily_sync_maintenance.sh --log

5. Verify fallback works before relying on it
   bash fallback_test.sh --dry-run first
   bash fallback_test.sh when ready — uses iptables simulation, no real downtime

Do step 5. Untested failover isn't redundancy, it's a belief. fallback_test.sh simulates the outage with iptables rather than actually pulling the other server down, so there's no excuse not to run it — and finding out your tier lists have a typo is a lot nicer on a Tuesday afternoon than at 2am when the power's actually out.

---

━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

README	What It Covers
README-Fallback.md	DDNS sequencing, tiered fallback, handback, split brain prevention, fallback_test.sh
README-Docker_Essentials.md	Two-tier container watchdog, memory limits, dependency ordering, skip list, daily/weekly restarts
README-System_Essentials.md	WebGUI recovery, inotify tuning, PHP-FPM, log hygiene, mover/reboot/rsync stop
README-Orchestrators.md	What runs when, execution order, daily/weekly windows, adding jobs
README-Media.md	Permissions model, junk cleanup, arr orphan removal, safety layers, behavior-driven discovery, testing procedure
README-Transcoding.md	Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback
README-Monitors.md	Cert monitoring, SMART health, bandwidth tracking, health digest profiles
README-Partnership.md	Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard
README-Arrs_Stack.md	Union model, orphan cleanup, classification scans, corruption scan, download-orphan cleaner, discovery
README-Watchdogs.md	The four tiers, strike systems, reboot rate limiting, cross-watchdog coordination
README-Rsync.md	Profiles, pre-flight checks, merge-run, source path guards, runtime ceiling
README-Deployment.md	The conf templates — the only versioned record of what configuration exists
README-Kernel.md	The scoring kernel consumers source for discovery decisions
README-Plugin.md	Platform adapter API, Unraid web UI, scheduler/monitor/docker pages
README-unraid.md	The Unraid platform layer — adapter contract, three-layer web UI, array lifecycle hooks
Manual.md	Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler
README-Tools.md	fallback_state_reset, skip list manager, emby_database_repair, container export

Every folder has a README-*.md for why it exists and most have a Manual-*.md for how to operate it. Every script has a header block covering purpose, operational model, design principles, safeguards, configuration and runtime modes — if you want to know what a script does, the top of the file is the authoritative answer, not this page.

---

━━━ THINGS THAT ACTUALLY WENT WRONG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Most of the safeguards in this repo exist because something bit me first. If you're wondering why a script checks something that looks paranoid, it's usually because it didn't, once.

The 755 GB nobody was looking at. Every cleanup script walked the library side. Nothing walked the download side. Completed downloads the arrs had quietly stopped tracking piled up in SABnzbd's Completed folders — oldest from 2022 — until the cache pool hit 89%. That's arr_download_orphan_cleaner.sh.

Lidarr said it had 23% of my music. 1,004 of 1,357 artists reporting a fraction of their real track count, with every file sitting right there on disk and no scan running. Organic scans only touch files involved in an import, so an untouched library just drifts. That's arr_full_rescan.sh.

"Full House" moved. Except it didn't. Sonarr's API reported episodeFileCount:192 at the new path while the actual 75 GB was still at the old one — moveFiles=true flips the database instantly, but the physical move is a separate async command draining one at a time behind ~20 others. Every classification move now polls its own command to completion before believing the database.

mtime lies about when a file arrived. Measured on one week's imports: 400 of 400 files had mtimes over 7 days old. One was 9,613 days old. An import preserves the release's original timestamp, so orphan detection has to gate on ctime — which means the nightly permissions job must only touch files that are actually wrong, because chown restamps ctime even when it changes nothing. Make that pass unconditional and orphan collection silently stops. Nothing errors. The pool just fills.

A rsync merge pass ran for 9.7 days. Now every transfer attempt is capped, and resumes rather than restarting — which only works because --partial is in the defaults.

R.E.M. was already in the library. Apparently. The duplicate check interpolated the artist name straight into a grep pattern, so R.E.M. happily matched a stored RxExMy. Music is full of regex metacharacters. It's a literal match now.

"Craig of the Creek" is not anime. TheTVDB tags it with the Anime genre anyway, which is why the anime signal needs corroboration from language or network before it counts. In the other direction, "The Pussycat Dolls Present" is a real MTV show with zero votes and no IMDb ID — which is why the junk-detection heuristic that works on movies is deliberately not ported to TV.

php -S silently drops POST bodies on Unraid's PHP 8.4. The webhook listener is Node.js for exactly that reason, and no other.

---

━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Every one of these cost me something to learn. Reading them here is cheaper.

# ─────────────────────────────────────────────────────────────────────────────
# 1. The Varaverk scheduler has very few entries.
#    Everything runs through orchestrators. Individual scripts are never
#    scheduled directly. array_started.sh is the only array_start event entry.

# 2. Scripts are silent when healthy.
#    stability_watchdog.sh runs 288 times per day. It produces zero output on clean
#    cycles. If you see output, something needs attention. If you see nothing,
#    the system is healthy. This is by design.

# 3. master.conf is the only file you edit.
#    Master_host*.conf for per-server values. Everything else is code — don't
#    add configuration into scripts directly. Everything configurable has a var.

# 4. --dry-run works on everything.
#    Every single script supports --dry-run. Pre-flight checks still run.
#    Review what would happen before anything changes. Always available.

# 5. detect_hosts() does the heavy lifting.
#    Scripts never compare hostnames directly. detect_hosts() aliases the correct
#    HOST*_ vars based on the running server. The same script runs correctly on
#    HOST1 and HOST2 — no branching on server identity inside scripts.

# 6. GPU containers: ignore every Google result. Use --gpus, not --runtime=nvidia.
#    --runtime=nvidia breaks on container rebuilds. See GPU CONTAINERS section above.

# 7. The Emby/Jellyfin Docker mount is unusual and critical.
#    --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode
#    in Extra Parameters — not a standard path mapping. Do NOT add bind-propagation=shared:
#    runc v1.3.5+ (Docker 29.x, Unraid 7.3+) rejects it. The ramdisk is already
#    MS_SHARED at the kernel level. README-Transcoding.md explains in detail.

# 8. arr cleanup scripts can delete files.
#    lidarr_cleanup.sh, sonarr_cleanup.sh, radarr_cleanup.sh permanently delete
#    orphaned files. Safety layers prevent catastrophic runs but always test with
#    --dry-run --log first on a new system. README-Media.md has the full procedure.

# 9. Scripts never call OS commands directly.
#    All platform-specific operations go through platform_*() functions defined in
#    Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all
#    isolated in the adapter. This is what makes the codebase portable.
#    Adding a new OS = writing one adapter file, not patching dozens of scripts.

# 10. Top-level System_Essentials/ is platform-agnostic. Plugin/unraid/ is not.
#    server_reboot.sh lives at the top level and calls adapter functions only —
#    platform_is_mover_running, platform_restart_service. It never names an OS.
#    Scripts that manipulate an Unraid subsystem directly — mover_stop.sh,
#    webgui_watchdog.sh, php_fpm_max_children.sh — live under Plugin/unraid/
#    instead, because there is no generic version of them to write.
#    Which folder a script is in tells you whether it survives a platform change.
# ─────────────────────────────────────────────────────────────────────────────

---

━━━ PROJECT NAME ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Name:         Varaverk
Repository:   git@192.168.50.2:FailedProxy/Varaverk.git
Install path: /boot/config/plugins/varaverk

---

This is a home media stack for two households. It is not an enterprise product, it doesn't want to be one, and it makes trade-offs no enterprise would accept — because the thing it's actually optimising for is that nobody in either house has to know any of this exists.

The measure of success here isn't uptime percentages. It's that the power goes out 50 miles away, and the show keeps playing.