Varaverk/README.md

# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🏠  VARAVERK
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**A complete self-healing, self-maintaining, mutually-redundant two-server unRAID
ecosystem.** One codebase runs on both servers. Both servers run their own lives
independently. When one goes down the other covers it — not because of conventional
failover infrastructure, but because both servers already have copies of everything
that matters and both know how to run it.

> **This is not conventional failover.** There is no primary and standby. There is no
> shared storage. There is no cluster manager. There are two fully independent unRAID
> servers, living in different locations on different power utilities, each running
> their own household's media stack — and a layer of scripts that keeps them
> synchronised, coordinated, and mutually covering. The goal is minimal disruption to
> a media stack, not enterprise high availability.

---

## ━━━ THE TWO SERVERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
HOST1 — unRAID-Gmer4Lfe
  Hardware:  Threadripper 1950X, 128GB RAM, ZFS cache pools
  Location:  Primary site
  Domain:    Gmer4Lfe.com
  Arrs:      Movies, Tv_Shows, Music — union with HOST2 via arr_sync
             Auth stack — source of truth (HOST1 owned, syncs to HOST2)
             Emby — primary media server

HOST2 — unRAID-Jayred365
  Hardware:  Intel i5 10th gen, 64GB RAM
  Location:  Remote — different building, different power utility
  Domain:    Gmer4Lfe.us
  Arrs:      Anime_Shows, Anime_Movies — union with HOST1 via arr_sync
             His own Emby — for his household
```

**Hardware does not need to match.** Everything is accessed through `/mnt/user/` —
unRAID's unified share layer. HOST1 has a Threadripper with ZFS. HOST2 has a completely
different CPU and fewer drives. A failover container on HOST2 mounts `/mnt/user/Movies`
and sees mirrored data because the share names match. The hardware underneath is
irrelevant.

---

## ━━━ CONTAINER NAMING — YOU NEED TO KNOW THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

This is the most important convention in the ecosystem. Get it wrong and the scripts
break silently.

---

### ── Shared Containers — No Suffix ───────────────────────────────────────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# Shared containers — meaning both servers run this container and scripts treat
# them as the same logical service — use the plain name with no suffix.
#
# The convention: "no suffix = this container participates in the shared stack"
#
    Emby                    # the shared Emby — runs on HOST1, picked up by HOST2 on failover
    NginxProxyManager       # shared NPM — runs on both, same config via sync
    Lldap-Gmer4Lfe          # shared LLDAP — same user directory, both servers
    Authelia                # shared auth — both servers, same config
    Mariadb-Authelia        # Authelia's database — same on both
    Redis-Authelia          # Authelia's cache — same on both
    VaultWarden             # shared password manager
    Dispatcharr             # shared Live TV scheduler
```

When `fallback.sh` on HOST2 reads `FALLBACK_HOST2_COVERS_HOST1_TIER1=(..."Emby"...)`,
it looks for a container called exactly `Emby` and starts it. HOST2 must have a
container called `Emby` — not `Emby-Gmer4Lfe`, not `Emby-HOST1`, just `Emby` — with
its volume mounts pointing at the mirrored share paths. This is what makes one
codebase work on both servers without modification.

---

### ── User-Specific Containers — Suffixed ────────────────────────────────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# User-specific containers — things that belong to one person and only ever
# run on their server — use a suffix identifying the user/owner.
#
    Emby-Gmer4Lfe           # Gmer4Lfe's personal Emby instance (HOST1, never covered)
    Emby-Jayred365          # Jayred365's personal Emby instance (HOST2, never covered)
    Organizer-Gmer4Lfe      # Gmer4Lfe's dashboard — personal, stays on HOST1
    Sonarr-Gmer4Lfe         # Gmer4Lfe's Sonarr — managing his Tv_Shows share
    Radarr-Gmer4Lfe         # Gmer4Lfe's Radarr — managing his Movies share
    Immich-Gmer4Lfe         # Gmer4Lfe's photo library — personal
```

User-specific containers are covered during failover (see Tiered Failover below) but
are distinct from the shared stack. They are never confused with each other because
the suffix makes the owner unambiguous. Scripts targeting `Emby` and scripts targeting
`Emby-Gmer4Lfe` are targeting different containers for different purposes.

---

### ── Why This Matters ─────────────────────────────────────────────────────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# The entire ecosystem uses container names as identifiers.
# docker_watchdog.sh: WATCHDOG_REQUIRED_CONTAINERS=("NginxProxyManager" "Authelia")
#   → looks for exactly "NginxProxyManager" on the running server
# fallback.sh: FALLBACK_HOST2_COVERS_HOST1_TIER1=("Emby" "VaultWarden")
#   → looks for exactly "Emby" and "VaultWarden" on HOST2
# docker_daily_restart.sh: DAILY_RESTART_CONTAINERS=("Dispatcharr" "Authelia")
#   → looks for exactly those names in docker ps
#
# A container named differently than expected is not found — no error, no restart,
# no failover coverage. It silently does not happen.
# Keep names consistent on both servers for shared containers.
# ─────────────────────────────────────────────────────────────────────────────
```

---

## ━━━ SHARE NAMING — ALSO YOU NEED TO KNOW THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Shares must have **identical names on both servers**. The ecosystem uses share paths
as sync targets and mount points — a mismatch is a broken path.

```bash
# ─────────────────────────────────────────────────────────────────────────────
# The rule: if a share exists on HOST1 and HOST2 needs a mirror of it,
# the share on HOST2 must have the exact same name.
#
# Same name on both servers:
    /mnt/user/Movies          # both servers — arr_sync.sh keeps Radarr in union
    /mnt/user/Tv_Shows        # both servers — arr_sync.sh keeps Sonarr in union
    /mnt/user/Music           # both servers — arr_sync.sh keeps Lidarr in union
    /mnt/user/Anime_Shows     # both servers — arr_sync.sh keeps Sonarr in union
    /mnt/user/Anime_Movies    # both servers — arr_sync.sh keeps Radarr in union
#
# rsync.sh syncs /mnt/user/Movies on HOST1 → /mnt/user/Movies on HOST2.
# If HOST2 has it at /mnt/user/Movies-Mirror → rsync aborts: path not found.
# ─────────────────────────────────────────────────────────────────────────────
```

---

## ━━━ THE MEDIA LIBRARY MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

### ── Common Libraries — Shared Pool ─────────────────────────────────────────

Common media like Movies, Music, and TV Shows are a **shared pool**. Every container
that needs music — Emby, LidaTube, Navidrome, whatever — mounts the same `/mnt/user/Music`
share. There is one share, one location, one source of truth. No duplication of content
across separate shares per-user or per-application.

```bash
# ─────────────────────────────────────────────────────────────────────────────
# One share for the library — all containers that need it mount the same path.
# Emby container: /mnt/user/Music → /media/Music (read)
# Lidarr:         /mnt/user/Music → /music       (read+write)
# LidaTube:       /mnt/user/Music → /downloads   (write)
# Navidrome:      /mnt/user/Music → /music       (read)
#
# All pointing at the same /mnt/user/Music. No per-user music copies.
# When Lidarr downloads and Emby scans — they're looking at the same files.
# ─────────────────────────────────────────────────────────────────────────────
```

---

### ── Both Arrs Manage All Shares — arr_sync.sh Is the Source of Truth ───────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# Both servers run Sonarr, Radarr, and Lidarr — all pointing at the same
# share names. Either server can download anything to any share at any time.
#
# How it stays coherent:
#   arr_sync.sh runs first (before rsync) — bidirectional library union.
#   Both Sonarr instances know about every TV episode on either server.
#   Both Radarr instances know about every movie on either server.
#   Both Lidarr instances know about every album on either server.
#
# Then rsync spreads the files additively (no --delete).
# Any file downloaded on HOST1 propagates to HOST2 on the next cycle.
# Any file downloaded on HOST2 propagates to HOST1 on the next cycle.
#
# HOST1 downloads Tv_Shows episode:
#   → HOST1 Sonarr tracks it immediately
#   → arr_sync.sh runs → HOST2 Sonarr now tracks it
#   → daily rsync → file lands on HOST2
#
# HOST2 downloads the same show's next episode that same night:
#   → HOST2 Sonarr tracks it immediately
#   → arr_sync.sh runs → HOST1 Sonarr now tracks it
#   → daily rsync → file lands on HOST1
#
# Both servers converge. Neither owns the share. The union is the truth.
# ─────────────────────────────────────────────────────────────────────────────
```

---

### ── How arr_sync.sh Keeps the Union Stable ────────────────────────────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# arr_sync.sh runs as the fixed first step of intermediate_sync_maintenance.sh
# (every 4 hours). It syncs all arr libraries bidirectionally before any rsync.
#
# What it does:
#   Reads each arr's full library via the local API
#   SSHes to the remote, reads the remote arr library via its API
#   Adds anything missing on either side — union, not overwrite
#   Blocked items (blocklist) are excluded from sync
#
# What this means in practice:
#   No share has an owner. No server is read-only.
#   Either server can download content to any share at any time.
#   arr_sync.sh guarantees both arrs agree on what exists before files move.
#   rsync then spreads the files to make storage match what arrs already know.
#
# arr_cleanup.sh (daily) removes true orphans — files no arr tracks.
# arr_cleanup uses the union model too: a file is only an orphan if
# neither arr on either server has it indexed.
# ─────────────────────────────────────────────────────────────────────────────
```

---

## ━━━ WHAT SYNCS AND WHAT DOESN'T ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Not everything needs to sync. Not everything should sync. Understanding what goes
where determines whether you need encryption and whether sync failure matters.

---

### ── Shared Failover Shares — Sync Required ─────────────────────────────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# These shares enable failover — HOST2 must have current copies for the covered
# services to work correctly when they start on HOST2.
#
    /mnt/user/Media_Server/Emby              # Emby userdata — watch history, playstates
    /mnt/user/appdata-Failover/Critical-Data # auth stack — NPM rules, certs, user accounts
    /mnt/user/Movies                         # media library content
    /mnt/user/Tv_Shows                       # media library content
    /mnt/user/Music                          # media library content
#
# Sync failure here = failover runs but with stale data (acceptable) or missing
# data (problematic). These sync every night via daily_sync_maintenance.sh.
# Critical-Data and Emby also sync every 15 minutes via critical_sync_maintenance.sh.
# ─────────────────────────────────────────────────────────────────────────────
```

---

### ── Non-Failover Shares — Sync as Backup, No Encryption Required ───────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# Some shares don't participate in failover — their container only runs on one
# server and is covered during failover from scratch (or not at all).
# These shares can sync as offsite backup without encryption.
# Content is not sensitive — anyone with access to HOST2 could read it.
#
# Examples:
    /mnt/user/appdata-Failover/Gmer4Lfe      # Gmer4Lfe server appdata backup
    /mnt/user/appdata-Failover/Important-Data # NextCloud, Immich appdata
    /mnt/user/Kids_Movies                    # kids content — not sensitive
    /mnt/user/Sports                         # sports content — not sensitive
#
# These sync in DAILY_SYNC_SHARES — straight rsync, no special handling.
# If HOST2 is compromised, the content is readable. That is acceptable.
# ─────────────────────────────────────────────────────────────────────────────
```

---

### ── Personal Shares — Always Encrypted ─────────────────────────────────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# Personal shares contain content only the owner should see — documents,
# photos, financial records, private files.
# ZFS encrypts at the dataset level — HOST2 receives encrypted blocks.
# Without your passphrase or keyfile, HOST2 cannot read the content.
#
    HOST1_PERSONAL_SHARES=(
        "/mnt/user/Gmer4Lfe-Personal"    # encrypted ZFS dataset
    )
    HOST2_PERSONAL_SHARES=(
        "/mnt/user/Jayred365-Personal"   # encrypted ZFS dataset
    )
#
# Append to HOST*_DAILY_SYNC_SHARES — synced nightly with everything else.
# Same rsync, same timing — but the content on HOST2 is ciphertext only you can unlock.
#
# Setup: README-Rsync_Setup.md → Step 8 — Personal Encrypted Shares
# ─────────────────────────────────────────────────────────────────────────────
```

---

## ━━━ THIS IS NOT CONVENTIONAL FAILOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

> **Understanding this distinction prevents a lot of confusion.**

Conventional failover (Pacemaker, DRBD, Proxmox HA) operates on the premise of a
primary server and a standby server with shared storage or synchronous replication.
The standby exists only to take over when the primary fails. It runs nothing
independently. Its purpose is 100% failover coverage with no service interruption.

**This is not that.**

---

### ── What This Actually Is ─────────────────────────────────────────────────────

```
Two fully independent servers, each running their own household's services.
Each server has its own domain, its own users, its own media library management.
Each server is the primary for its own services every single day.

The "failover" component is additive:
  When HOST1 goes down → HOST2 starts HOST1's containers on top of its own
  When HOST2 comes back → HOST1's containers hand back cleanly
  HOST2 never stopped running its own services during this entire time

This is not standby. This is mutual coverage between peers.
```

---

### ── The Actual Goal: Minimal Disruption to a Media Stack ────────────────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# "Minimal disruption" means:
#   Live TV keeps playing         → Dispatcharr is Tier 1 (starts immediately)
#   Passwords are accessible      → VaultWarden is Tier 1 (people lock themselves out)
#   Emby keeps serving            → Emby is Tier 1
#   Auth works                    → NPM, Authelia, LLDAP already warm on both servers
#
# "Minimal disruption" does NOT mean:
#   Zero downtime (DDNS TTL = 1 minute, brief gap expected)
#   Perfect consistency (Emby userdata is 15min stale at most)
#   All services covered immediately (tiered start over 24 hours)
#
# The system is designed around the real experience of a household:
#   "The TV stopped working" → unacceptable, fixed in Tier 1 (< 2 minutes)
#   "NextCloud is down"      → annoying, fixed in Tier 2 (4 hours)
#   "Gitea is down"          → unacceptable, fixed in Tier 1 (< 2 minutes)
#   "Sonarr isn't downloading" → fine, fixed in Tier 4 (24 hours)
# ─────────────────────────────────────────────────────────────────────────────
```

---

### ── What You Give Up With This Approach ─────────────────────────────────────

```bash
# ─────────────────────────────────────────────────────────────────────────────
# Things conventional failover provides that this does NOT:
#   Zero RPO (recovery point objective) — there is ~15min of potential staleness
#   Synchronous state replication — changes don't reach HOST2 instantly
#   Automatic failure detection in seconds — fallback.sh checks every 2 minutes
#   Transparent IP migration — DNS TTL means up to 1 minute of routing to old IP
#
# What you get instead:
#   Both servers run fully independently — no single point of failure in software
#   Geographic redundancy — different buildings, different power, different ISP
#   Both households stay operational during most disruption scenarios
#   Simple, auditable scripts — no cluster manager black box
# ─────────────────────────────────────────────────────────────────────────────
```

---

## ━━━ HOW FAILOVER ACTUALLY WORKS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

`fallback.sh` runs on both servers as a continuous background process. Every 2 minutes
it pings two things: the other server's Tailscale IP, and `8.8.8.8`. The combination
of those two answers determines the current state.

```
Remote ping PASS + Internet PASS → NORMAL — both servers running independently

Remote ping FAIL + Internet PASS → FAILOVER — remote is down, start covering it
  Tier 1 immediately: DDNS + Emby + Auth + VaultWarden + Live TV + Gitea
  Tier 2 at 4 hours: NextCloud + Immich + Jellyseerr
  Tier 3 at 12 hours: AdGuard + Uptime monitoring
  Tier 4 at 24 hours: Arrs + download clients

Internet FAIL (either) → NO_INTERNET — stop own DDNS, wait
  Can't serve anyone without internet — stop claiming DNS ownership

Remote FAIL + Internet FAIL → DARK — conservative, same as NO_INTERNET
```

The DDNS sequencing is critical — one domain, one DDNS container, at all times. The
handback sequence stops remote DDNS first, rsyncs, starts containers, starts local
DDNS last. Split-brain DNS is prevented by design.

**→ Full detail: [README-Fallback.md](Fallback/README-Fallback.md)**

---

## ━━━ THE AUTH STACK — WARM ON BOTH SERVERS ALWAYS ━━━━━━━━━━━━━━━━━━━━━━━━━━

NPM, LLDAP, Authelia, MariaDB, Redis — all running actively on both servers at all
times. Not cold standby. Not started at failover. Running right now, serving each
server's own domain.

```bash
# ─────────────────────────────────────────────────────────────────────────────
# Why this eliminates the failover dead zone:
#
# Cold standby approach: HOST1 goes down → DNS switches to HOST2 → HOST2 starts
# auth containers → 30-60 seconds before auth is ready → users hit the auth wall
# before the service behind it responds → failed logins, broken sessions
#
# Warm approach: HOST1 goes down → DNS switches to HOST2 → auth is ALREADY
# running → users are served immediately with no dead zone
#
# HOST1 is source of truth for the auth configuration. All changes to NPM proxy
# rules, LLDAP users, Authelia policies, and certs happen on HOST1 and propagate
# to HOST2 via Critical-Data sync every 15 minutes.
#
# HOST2 operator's experience: click NginxProxyManager in Docker UI →
# browser opens HOST1's NPM via Tailscale redirect → make changes there →
# 15 minutes later live on both servers. Zero auth management overhead on HOST2.
# ─────────────────────────────────────────────────────────────────────────────
```

**→ Full detail: [README-Partnership.md](Partnership/README-Partnership.md)**

---

## ━━━ THE COMPLETE PICTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
                    HOST1 (Gmer4Lfe.com)        HOST2 (Gmer4Lfe.us)
                    ──────────────────          ──────────────────────

Always running:
  Own stack       Full service stack            Full service stack (anime focus)
  Auth            NPM, Authelia, LLDAP          NPM, Authelia, LLDAP (mirrored)

Source of truth:
  Media           Movies, Tv_Shows, Music  ←──  Anime_Movies, Anime_Shows
                  ───────────────────────→       (bidirectional daily push)
  Config          auth stack config        ──→   mirrored every 15 min
  Emby state      watch history, playstates──→   dirty sync every 30 min

Failover coverage:
  If HOST1 down:  —                             starts Emby, VaultWarden,
                                                Dispatcharr, all arr stack
                                                serves Gmer4Lfe.com via DDNS

  If HOST2 down:  starts his Emby, his arrs,   —
                  serves Gmer4Lfe.us via DDNS

Monitoring:
  Both servers    resource/docker/system/        resource/docker/system/
                  stability_watchdog.sh         stability_watchdog.sh
                  fallback.sh                   fallback.sh
                  Sunday morning coffee report  Sunday morning coffee report
```

---

## ━━━ FOLDER STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
Unraid_Scripts/
│
├── Configurations/          ← All configuration files — the only files you edit regularly
│   ├── master.conf          ← Shared: thresholds, toggles, job lists, watchdog settings
│   ├── host1.conf           ← HOST1-specific: share lists, container names, API keys
│   └── host2.conf           ← HOST2-specific: same structure, different values
├── common.sh                ← Shared library — all functions used by every script
├── load_config.sh           ← Sources Configurations/ and common.sh at startup
│
├── Orchestrators/           ← Scheduled job runners — what Arbiter executes
│   README: README-Orchestrators.md
│
├── Rsync/                   ← Core rsync script — called per share with profile
│   README: README-Rsync_Setup.md
│
├── Fallback/                ← Mutual automatic failover — continuous background process
│   README: README-Fallback.md
│
├── Watchdogs/               ← All watchdog scripts: resource, docker, system, stability
│   README: README-Watchdogs.md
│
├── Docker_Essentials/       ← Container lifecycle: restarts, updates, networks
│   README: README-Docker_Essentials.md
│
├── unRAID_Essentials/       ← Server-level: WebGUI, log hygiene, kernel tuning
│   README: README-Unraid_Essentials.md
│
├── Media/                   ← Library health + behavior-driven discovery: permissions, junk cleanup, orphan removal, weekly arr adds
│   README: README-Media.md
│
├── Transcodes/              ← Ramdisk transcode management with SSD fallback
│   README: README-Transcoding.md
│
├── Monitors/                ← Read-only observation: certs, SMART, bandwidth, digests
│   README: README-Monitors.md
│
├── Partnership/             ← Two-server relationship lifecycle: onboard, offboard, transfer
│   README: README-Partnership.md
│
├── Kernel/                  ← Kernel module config — loaded at boot by go file
│   README: README-Kernel.md
│
└── Tools/                   ← Situational utilities: repair, export, emergency tools
    README: README-Tools.md
```

---

## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━

```bash
# ─────────────────────────────────────────────────────────────────────────────
# All three conf files live in Configurations/. load_config.sh sources them.
# Sparse checkout ensures each server only receives its own host*.conf.
#
# master.conf — shared across both servers
#   Everything that applies equally to both: thresholds, schedules, profile
#   definitions, watchdog settings, arr cleanup config, DDNS timing, etc.
#   Pushed to both servers via git. Never contains server-specific values.
#
# host1.conf — sourced only on HOST1
#   HOST1_* prefixed variables: share lists, container names, API keys,
#   ramdisk size, specific paths, per-server toggle overrides.
#
# host2.conf — sourced only on HOST2
#   HOST2_* prefixed variables: same structure, different values.
#
# detect_hosts() in common.sh:
#   Reads the current server's hostname. Matches it against HOST1 and HOST2.
#   Aliases HOST1_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST1)
#   Aliases HOST2_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST2)
#   Every script uses the unprefixed name. The same script on either server
#   automatically uses the correct values. No hostname comparisons in scripts.
# ─────────────────────────────────────────────────────────────────────────────
```

---

## ━━━ WHAT RUNS WHEN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Arbiter runs only orchestrators — never individual scripts directly.

```
At Startup of Array:
  array_started.sh              launches all startup scripts in order
    → inotify_tuning.sh       raise kernel limits before containers start
    → docker_syslog_filter.sh suppress veth noise before containers create interfaces
    → php_fpm_max_children.sh WebGUI tuning before first request
    → ramdisk_setup.sh        create ramdisk before Emby starts
    → docker_network_connect.sh connect containers to extra networks
    → fallback.sh             continuous — mutual failover state machine

Every minute:
  watchdog_orchestrator.sh    fires each watchdog in sequence
    → resource_watchdog.sh    reduce pressure before healing attempts
    → docker_watchdog.sh      two-tier container healing
    → system_watchdog.sh      system components: storage + WebGUI health
    → stability_watchdog.sh   last resort — reboots when all else fails

Every 3 minutes:
  transcode_management.sh     cleanup → manager (order non-negotiable)

Every 15 minutes:
  critical_sync_maintenance.sh  Critical-Data + Emby dirty sync + partnership check

Every 30 minutes:
  rsync.sh --profile=emby-failover   Emby watch state sync (separate cron entry)

Every 6 hours:
  arrs_failed_stalled_recovery.sh   blocklist + re-search failed imports

1am daily:
  daily_sync_maintenance.sh   git pull → sync shares → media maintenance → restarts

2:30am Sunday:
  weekly_sync_maintenance.sh  stop → update images → clean sync → start → weekly restarts
                              + playback_aware_lidarr/radarr/sonarr_discovery.sh  (behavior-driven arr discovery)

8am daily:
  weekly_health_digest.sh     DIGEST_PROFILE controls when it actually notifies

Every 6 hours:
  system_tuning_monitor.sh    inotify + php-fpm utilisation snapshot

Sunday morning block (6–11am):
  zfs_memory_snapshot.sh, smart_health.sh, cert_monitor.sh,
  backup_verify.sh, emby_session_report.sh, bandwidth_monitor.sh --report
```

**→ Complete schedule: [README-Orchestrators.md](Orchestrators/README-Orchestrators.md)**

---

## ━━━ THE DAILY CYCLE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

What actually happens on a typical day, from the ecosystem's perspective:

```
Throughout the day (every minute via watchdog_orchestrator.sh):
  resource_watchdog.sh        managing: system pressure (throttle/pause/stop)
  docker_watchdog.sh          healing: memory leaks, HTTP failures, required containers
  system_watchdog.sh          watching: system components (storage growth, WebGUI)
  stability_watchdog.sh       watching: RAM, CPU temp, rootfs, kernel, daemon

Throughout the day:
  fallback.sh                 watching: remote server, internet connectivity
  transcode_management.sh     managing: ramdisk ↔ SSD, session cleanup (every 3min)
  critical_sync_maintenance.sh keeping: auth stack + Emby current (every 15min)

1am:
  git pull (latest scripts first)
  rsync Movies, Tv_Shows, Music → HOST2        (HOST1 pushes its truth shares)
  rsync Anime_Shows, Anime_Movies → HOST1      (HOST2 pushes its truth shares)
  media_shares_permissions.sh                  (fix ownership before arr cleanup)
  media_cleaner.sh anime + media               (remove scene junk)
  lidarr/sonarr/radarr_cleanup.sh              (remove orphaned files)
  docker_daily_restart.sh                      (NPM, Authelia, Dispatcharr fresh start)

2:30am Sunday:
  Stop Emby + auth stack both sides
  Pull updated Docker images (zero extra downtime — already stopped)
  rsync Emby + Critical-Data clean mirror
  Start containers on new images
  docker_weekly_restart.sh                     (NextCloud, AdGuard, Immich)

7am Sunday:
  sunday_morning_coffee_report.sh              (everything that happened this week)
```

---

## ━━━ THE SELF-HEALING STACK ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Problems are handled at the most appropriate level. Scripts escalate only when the
level below them has failed:

```
Problem: Emby container using 22GB RAM (limit: 18GB)
  → docker_watchdog.sh Tier 1: restart Emby immediately
  → System continues, no reboot

Problem: Authelia crashed and won't restart
  → docker_watchdog.sh Tier 1: attempt restart (dependency check: DB + Redis first)
  → Fails 3 times in 1 hour → skip list → critical notification
  → Tools: watchdog_skip_list_manager.sh to clear after fixing

Problem: RAM drops to 3GB
  → stability_watchdog.sh: stop non-essential containers
  → Wait for RAM to recover to 30GB
  → docker_watchdog.sh defers (reads mem_shutdown_active=true)
  → If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot

Problem: Docker daemon hung
  → stability_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
  → If still hung: immediate reboot (no strikes needed — daemon can't be managed)

Problem: HOST1 loses power
  → fallback.sh on HOST2 detects at next 2-minute check
  → Tier 1 starts immediately: DDNS, Emby, VaultWarden, Live TV
  → Household barely notices. Live TV may buffer for < 2 minutes.
```

**→ Container healing: [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md)**
**→ System stability: [README-Unraid_Essentials.md](unRAID_Essentials/README-Unraid_Essentials.md)**

---

## ━━━ SETUP — WHERE TO START ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

If you're setting this up from scratch on two servers:

```
1. Follow README-Rsync_Setup.md completely — end-to-end setup guide
   Tailscale → SSH keys → git clone → master.conf → Arbiter schedule

2. Set your container names correctly before anything else
   Shared services: no suffix (Emby, NginxProxyManager, etc.)
   Personal services: suffixed (Emby-Gmer4Lfe, Organizer-Gmer4Lfe)

3. Set your share names identically on both servers
   /mnt/user/Movies on both, /mnt/user/Tv_Shows on both, etc.

4. Run initial sync to populate HOST2
   bash daily_sync_maintenance.sh --log

5. Verify failover works before relying on it
   bash failover_test.sh --dry-run first
   bash failover_test.sh when ready — uses iptables simulation, no real downtime
```

---

## ━━━ FOLDER README REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

| README | What It Covers |
|--------|---------------|
| [README-Fallback.md](Fallback/README-Fallback.md) | DDNS sequencing, tiered fallback, handback, split brain prevention, fallback_test.sh |
| [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md) | Two-tier container watchdog, memory limits, dependency ordering, skip list, daily/weekly restarts |
| [README-Unraid_Essentials.md](unRAID_Essentials/README-Unraid_Essentials.md) | Three-tier system watchdog, WebGUI recovery, inotify tuning, PHP-FPM, log hygiene |
| [README-Orchestrators.md](Orchestrators/README-Orchestrators.md) | What runs when, execution order, daily/weekly windows, adding jobs |
| [README-Media.md](Media/README-Media.md) | Permissions model, junk cleanup, arr orphan removal, safety layers, behavior-driven discovery, testing procedure |
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
| [README-Rsync_Setup.md](Rsync/README-Rsync_Setup.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Arbiter |
| [README-Tools.md](Tools/README-Tools.md) | failover_state_reset, skip list manager, emby_database_repair, container export |
| [README-Unraid_Essentials.md](unRAID_Essentials/README-Unraid_Essentials.md) | `git_pull_execute.sh`, `server_reboot.sh`, `mover_stop.sh`, `user_scripts_stop.sh` |

---

## ━━━ A FEW THINGS THAT WILL SURPRISE YOU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Things that are different from what you might expect:

```bash
# ─────────────────────────────────────────────────────────────────────────────
# 1. Arbiter has very few entries.
#    Everything runs through orchestrators. Individual scripts are never
#    scheduled directly. array_started.sh is the only "At Startup of Array" entry.

# 2. Scripts are silent when healthy.
#    stability_watchdog.sh runs 288 times per day. It produces zero output on clean
#    cycles. If you see output, something needs attention. If you see nothing,
#    the system is healthy. This is by design.

# 3. master.conf is the only file you edit.
#    Master_host*.conf for per-server values. Everything else is code — don't
#    add configuration into scripts directly. Everything configurable has a var.

# 4. --dry-run works on everything.
#    Every single script supports --dry-run. Pre-flight checks still run.
#    Review what would happen before anything changes. Always available.

# 5. detect_hosts() does the heavy lifting.
#    Scripts never compare hostnames directly. detect_hosts() aliases the correct
#    HOST*_ vars based on the running server. The same script runs correctly on
#    HOST1 and HOST2 — no branching on server identity inside scripts.

# 6. The Emby Docker mount is unusual and critical.
#    --mount type=bind,...,bind-propagation=shared in Extra Parameters — not a
#    standard path mapping. Without shared propagation symlink flips are silently
#    ignored. README-Transcoding.md explains why in detail.

# 7. arr cleanup scripts can delete files.
#    lidarr_cleanup.sh, sonarr_cleanup.sh, radarr_cleanup.sh permanently delete
#    orphaned files. Safety layers prevent catastrophic runs but always test with
#    --dry-run --log first on a new system. README-Media.md has the full procedure.
# ─────────────────────────────────────────────────────────────────────────────
```

---

## ━━━ PROJECT NAME ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
Name:         Varaverk
Repository:   git@192.168.50.2:Varaverk/Unraid_Scripts.git
Install path: /mnt/user/appdata/unraid_scripts
```