Varaverk/README-User_Script_Plug-in.md

# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 📋  USER SCRIPTS PLUGIN — COMPLETE SETUP REFERENCE
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Every script entry for the User Scripts plugin — what it does, when it runs,
how to set it up, and what to watch for.** No cross-referencing other READMEs to
understand the basics. The deeper documentation exists — this is your daily reference.

---

## ━━━ REQUIRED PLUGINS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Install both from Community Applications before creating any entries:

```
User Scripts          — Andrew Zawadzki
                        Core plugin. Runs scripts via unRAID UI without terminal.
                        Required — nothing works without this.

User Scripts Enhanced — Kurotaku
                        Adds categories, drag-to-reorder, UI improvements.
                        Required for the category organisation used here.
```

---

## ━━━ GLOBAL RULES — READ BEFORE CREATING ENTRIES ━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
1. Every script entry → Run in Background: YES
   Without background mode the plugin UI freezes waiting for the script to complete.
   Long-running scripts and continuous loops (failover, watchdogs) require background.
   There are no exceptions — set background on everything.

2. Run as root: YES
   All scripts require root. Docker operations, rsync, kernel tuning, reboots.

3. Category names match repo folder names exactly
   When the plugin category and the git folder are identical, you never wonder
   which folder a script lives in. No mental translation.

4. Script name matches the script filename (without .sh)
   daily_sync_maintenance.sh → name: daily_sync_maintenance
   When logs reference a script name, you find the plugin entry immediately.

5. --dry-run exists on every script
   Test anything unfamiliar with --dry-run first. Pre-flight checks still run.
   It shows what would happen without making changes.

6. Most scripts never need to be run directly
   The orchestrators handle timing and order. Individual script entries exist
   for manual testing and one-off maintenance — not as replacement cron jobs.
```

---

## ━━━ CATEGORY STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Create these categories in User Scripts Enhanced before adding any entries:

```
Orchestrators      — scheduled maintenance windows and startup launcher
Failover           — mutual failover state machine and test tools
Monitors           — weekly reports, health checks, trend tracking
Rsync              — manual per-profile sync triggers
Docker_Essentials  — container management tools (manual use)
Media              — library maintenance tools (manual use)
Transcodes         — ramdisk and transcode tools (manual use)
unRAID_Essentials  — server management tools (manual use)
Partnership        — two-server relationship tools (manual use)
Tools              — repair, recovery, and one-time utilities
Git                — script deployment
```

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🎯  ORCHESTRATORS
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Orchestrators are the only scripts that run on a schedule. They coordinate multiple
child scripts in the correct order. Adding or removing a job means editing master.conf
— never the orchestrator script itself.

---

### ── array_start ─────────────────────────────────────────────────────────────

```
Category:   Orchestrators
Name:       array_start
Schedule:   At Startup of Array
Background: YES — required (starts continuous background processes)
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Orchestrators/array_started.sh
```

**What it does:**
The single "At Startup of Array" entry for the entire ecosystem. Launches every
startup script in the order defined by ARRAY_START_SCRIPTS in master.conf. You
never add other startup scripts directly to User Scripts — add them to master.conf.

**What it starts (in order):**
```
inotify_tuning.sh        — raises kernel inotify limits before containers start
                           containers inherit limits at startup — must be first
docker_syslog_filter.sh  — suppresses Docker veth noise before containers create interfaces
php_fpm_max_children.sh  — sets PHP-FPM workers to 250 before WebGUI serves requests
ramdisk_setup.sh         — creates /mnt/ramdisk_transcodes and symlink before Emby starts
docker_network_connect.sh— ensures custom networks exist + containers connected
stability_watchdog.sh   ← last-resort — three-tier server stability watchdog
docker_watchdog.sh       ← continuous — two-tier container healing watchdog
fallback.sh              ← continuous — mutual failover state machine
```

**After running:**
One-shot scripts exit cleanly. Three continuous processes run in the background until
the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|failover"`

---

### ── transcode_management ─────────────────────────────────────────────────────

```
Category:   Orchestrators
Name:       transcode_management
Schedule:   */7 * * * *   (every 7 minutes)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Orchestrators/transcode_management.sh
```

**What it does:**
Runs transcode_cleanup.sh then transcode_manager.sh — in that exact order every
7 minutes. Order is non-negotiable: cleanup removes stale files first so the manager
sees real active session usage, not inflated numbers from ended sessions.

**transcode_cleanup.sh:**
Removes HLS segment files older than TRANSCODE_MAX_AGE minutes that are not currently
open by any process. Uses one lsof call per location — not one per file. Covers both
ramdisk and SSD fallback. Never touches the transcoding-temp directory.

**transcode_manager.sh:**
Checks ramdisk usage against RAMDISK_WARN_GB. Flips the /mnt/ram-transcode symlink
between ramdisk and SSD fallback based on current usage. Shows active Emby sessions
with their play method. Writes one daily stats entry to TRANSCODE_DAILY_LOG.
Runs safety checks every cycle: missing symlink → recreate, ramdisk disappeared →
auto-flip to SSD, missing transcoding-temp → recreate immediately.

**Output when healthy:** session list only — no flip messages = ramdisk managing fine

> ⚠️ The Emby/Jellyfin Docker container must have
> `--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode`
> in Extra Parameters. Do **not** add `bind-propagation=shared` — runc v1.3.5+
> (Docker 29.x, Unraid 7.3+) rejects it with a mountinfo error. The ramdisk tmpfs
> is already MS\_SHARED at the kernel level — propagation is inherited automatically.

---

### ── critical_sync_maintenance ───────────────────────────────────────────────

```
Category:   Orchestrators
Name:       critical_sync_maintenance
Schedule:   */30 * * * *  (every 30 minutes)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Orchestrators/critical_sync_maintenance.sh
```

**What it does:**
Keeps the auth stack and Emby watch state current on HOST2 — 30 minutes is the
maximum staleness for fallback-critical data.

**Jobs in order:**
```
rsync Critical-Data     — pushes auth stack to HOST2 (NPM rules, certs, LLDAP users,
                           Authelia config). HOST2's auth is always within 30min of HOST1.
rsync Emby dirty sync   — pushes watch states, user activity, library delta to HOST2
                           with Emby running. WAL files excluded — safe mid-write.
                           HOST2 Emby restarts after sync to pick up config changes.
partnership --check     — reads remote state file, increments offline counter if HOST2
                           unreachable, triggers auto-offboard at PARTNERSHIP_OFFLINE_THRESHOLD
```

**Silent when healthy:** no output on clean cycles — runs 48 times per day.

---

### ── arrs_failed_stalled_recovery ───────────────────────────────────────────

```
Category:   Orchestrators
Name:       arrs_failed_stalled_recovery
Schedule:   0 */6 * * *   (every 6 hours)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Orchestrators/arrs_failed_stalled_recovery.sh
```

**What it does:**
Automatically recovers failed imports and stalled downloads across Sonarr, Radarr,
and Lidarr. A release that downloaded but failed to import sits in the queue
permanently without this — no arr retries importFailed items on its own.

**What it finds and acts on:**
```
importFailed   — downloaded but couldn't import → blocklist + remove + new search
importPending  — stuck in import queue → blocklist + remove + new search
error          — serious failure state → blocklist + remove + new search
stalled        — no seeders or no progress → blocklist + remove + new search
```

**What it never touches:** items with status "downloading" or "imported".
Age gate: items newer than ARR_IMPORT_RECOVERY_AGE (default: 6 hours) are skipped —
the arr may still retry on its own.

**Result:** failed imports are automatically replaced while you sleep.

---

### ── daily_sync_maintenance ──────────────────────────────────────────────────

```
Category:   Orchestrators
Name:       daily_sync_maintenance
Schedule:   0 1 * * *    (1am daily)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Orchestrators/daily_sync_maintenance.sh
```

**What it does:**
The full 1am daily maintenance window — everything that needs to happen overnight
in one orchestrated sequence.

**Execution order:**
```
1. git_pull_execute.sh        — pull latest scripts FIRST, everything below runs on
                                 current code

2. rsync DAILY_SYNC_SHARES    — push this server's truth shares to the other server
   HOST1 pushes:                Movies, Tv_Shows, Music, Kids content, Sports...
   HOST2 pushes:                Anime_Shows, Anime_Movies
   Personal encrypted shares    appended after — encrypted blocks, remote can't read

   Drive temp exit codes:
     exit 1 (WARN)  → skip this share, continue to next
     exit 2 (CRIT)  → abort ALL remaining syncs, notify immediately

3. media_shares_permissions.sh — apply nobody:users 755/664 to all media shares
                                   arr cleanup depends on correct ownership

4. media_cleaner.sh anime      — remove .sfv .nfo .rar .sample and other scene junk
   media_cleaner.sh media        from anime and media share folders
                                   junk removal BEFORE orphan detection — cleaner results

5. lidarr_cleanup.sh           — identify and delete orphaned music files not tracked
   sonarr_cleanup.sh             by the arr API. 7 safety checks must all pass before
   radarr_cleanup.sh             a single file is deleted. notify_emby_scan() called
                                   after any deletion — Emby library cleaned immediately

6. docker_daily_restart.sh     — restart connection-heavy containers that degrade over time
                                   NginxProxyManager, Authelia, Dispatcharr, ErsatzTV-Emby
                                   Running → docker restart. Stopped → left stopped.
```

**Notify:** silent on success. One notification on failure with count of failed jobs.
**Both servers run this independently at 1am** — same script, correct direction via detect_hosts().

---

### ── rsync_emby_failover ─────────────────────────────────────────────────────

> **Superseded by `critical_sync_maintenance`.** Delete this User Script from the
> plugin. The emby-fallback dirty sync now runs as step 2 inside
> `critical_sync_maintenance.sh` every 30 minutes alongside the auth stack sync
> and partnership check. Running both produces duplicate syncs.

The Emby dirty sync (`/mnt/user/Media_Server/Emby --profile=emby-fallback`) is
configured in `HOST1_CRITICAL_SYNC_SHARES` in host1.conf. It runs automatically
when critical_sync_maintenance.sh runs.

---

### ── weekly_sync_maintenance ─────────────────────────────────────────────────

```
Category:   Orchestrators
Name:       weekly_sync_maintenance
Schedule:   30 2 * * 0   (Sunday 2:30am)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Orchestrators/weekly_sync_maintenance.sh
```

**What it does:**
The full Sunday maintenance window — clean sync with containers stopped, image
updates at zero extra downtime, weekly container restarts.

**Execution order:**
```
1. Pre-flight            — connectivity, remote Docker daemon, remote rootfs
2. Stop local containers — Emby + auth stack stopped on this server
3. Stop remote containers— Emby + auth stack stopped on remote via SSH
4. Pull images locally   — if WEEKLY_SYNC_UPDATES=true (containers already stopped,
                            free downtime, both servers start on identical image versions)
5. Pull images remotely  — if WEEKLY_SYNC_UPDATES_REMOTE=true (SSH to remote)
6. rsync WEEKLY_SYNC_SHARES:
   /mnt/user/Media_Server/Emby           — full clean mirror, databases checkpointed
   /mnt/user/appdata-Fallback/Critical-Data — auth stack clean state
7. Start remote containers— dependency order, new image, settle + verify each
8. Start local containers — same
9. docker_weekly_restart.sh — restarts less-critical services that benefit from
                               weekly clean start: NextCloud, AdGuard, Immich
```

**Why containers stop for Emby sync:**
Weekly sync is a full clean mirror including databases. Emby must be stopped to
checkpoint WAL — ensures a consistent database copy, not a snapshot mid-write.
~30 seconds of downtime. The 30-minute dirty sync covers users until this clean sync.

**Why weekly and not nightly:**
Emby builds a warm image thumbnail cache on HOST2 throughout the week. Nightly sync
resets this cache — cold loads every morning. Weekly sync: cache stays warm 6 days,
resets Sunday night while users sleep.

---

### ── sunday_morning_coffee_report ────────────────────────────────────────────

```
Category:   Orchestrators
Name:       sunday_morning_coffee_report
Schedule:   0 7 * * 0    (Sunday 7am)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Orchestrators/sunday_morning_coffee_report.sh
```

**What it does:**
Full weekly system overview — everything that happened this week in one clean read.
Designed to be read over coffee after the 2:30am maintenance window has completed
and run all the Sunday morning monitor scripts.

**Sections:**
```
🖥️  System         — uptime, memory, boot drive, cache usage, reboots this week
📀  Array          — disk count, parity status, ZFS health, drive temperatures
🎬  Transcodes     — weekly ramdisk peak, flip count, session split (ramdisk vs SSD)
🎵  Media Activity — arr cleanup stats (orphans removed, space freed), queue depth
🌐  Rsync          — weekly transfer totals, per-share breakdown, failures
🛡️  Watchdog       — system + docker watchdog state, active strikes, skip list
🔐  Security       — SSL cert expiry days remaining per domain (live TLS check)
📊  Emby           — version, active streams now, weekly play count, top users
⚙️  System Health  — inotify peak/avg/warnings, php-fpm peak/avg, SMART, git sync
⚠️  Issues         — anything requiring attention from any section above
```

**Delivery:** unRAID notification + Discord webhook (if configured).
**Silent when healthy:** footer shows ✅ if no issues, ⚠️ with count if something needs attention.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 📊  MONITORS
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Monitor scripts observe and report — they never act. Read-only except for two bounded
log writers. All are safe to run manually at any time.

---

### ── weekly_health_digest ────────────────────────────────────────────────────

```
Category:   Monitors
Name:       weekly_health_digest
Schedule:   0 8 * * *    (8am daily — DIGEST_PROFILE in master.conf controls notify behaviour)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Monitors/weekly_health_digest.sh
```

**What it does:**
Reads state files from across the ecosystem — watchdog strikes, failover state, skip
list, bandwidth history, transcode stats, cert expiry — and compiles them into a
digest notification. Reads only, writes nothing.

**DIGEST_PROFILE controls when it notifies (set in master.conf):**
```
always  — sends every run regardless of content (daily digest, even when healthy)
smart   — sends only when something worth reporting is found
          triggers: watchdog strikes, non-NORMAL failover, cert warnings, bandwidth spikes
weekly  — sends once per week on DIGEST_DAY (Sunday), silent all other days
```

Schedule never changes (8am daily). Switching between profiles requires only one
line change in master.conf — no cron edit needed.

---

### ── system_tuning_monitor ───────────────────────────────────────────────────

```
Category:   Monitors
Name:       system_tuning_monitor
Schedule:   0 */6 * * *  (every 6 hours)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Monitors/system_tuning_monitor.sh
```

**What it does:**
Snapshots inotify and PHP-FPM utilisation every 6 hours. Writes one bounded log
entry. The sunday_morning_coffee_report reads this log for peak, average, and
warning counts over the week.

**inotify monitoring:**
Current instances in use vs kernel limit. Warns above INOTIFY_WARN_PCT (80%).
Shows top 5 consumers. inotify exhaustion = downloads complete but arrs don't detect
them, Live TV stutters, library updates stop — no obvious error anywhere.

**PHP-FPM monitoring:**
Active workers vs PHP_MAX_CHILDREN. Warns above PHP_FPM_WARN_PCT (80%).
Saturation = WebGUI slows, settings saves hang, container starts from UI timeout.

**Log:** TUNING_MONITOR_LOG — bounded, trimmed to TUNING_LOG_RETENTION days on write.

---

### ── zfs_memory_snapshot ─────────────────────────────────────────────────────

```
Category:   Monitors
Name:       zfs_memory_snapshot
Schedule:   0 6 * * 0    (Sunday 6am — first Sunday script, before everything else)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Monitors/zfs_memory_snapshot.sh
```

**What it does:**
Weekly ZFS pool health and memory diagnostic. Runs before other Sunday monitors so
pool state is captured while the system is post-maintenance and settled.

**Reports:**
ZFS pool status (ONLINE/DEGRADED/FAULTED) per pool. ARC size vs max, hit rate, metadata
ratio. Total and available RAM. Top 10 Docker containers by memory. Kernel pressure
via vmstat. Warns if ARC > ZFS_REPORT_ARC_WARN_PCT or RAM < ZFS_REPORT_FREE_WARN_GB.
Output goes to both console and /var/log/zfs-weekly-health.log for week-over-week comparison.

---

### ── smart_health ────────────────────────────────────────────────────────────

```
Category:   Monitors
Name:       smart_health
Schedule:   0 7 * * 0    (Sunday 7am)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Monitors/smart_health.sh
```

**What it does:**
Queries SMART attributes for every /dev/sd* and /dev/nvme* drive weekly. Drives
in HOST*_SMART_IGNORE_DRIVES (typically sda — the boot USB) are skipped.

**What triggers a notification:**
```
Reallocated_Sector_Ct > 0     — drive found and swapped a bad sector — warning sign
Current_Pending_Sector > 0    — drive suspects sectors bad, not yet confirmed
Offline_Uncorrectable > 0     — sectors that couldn't be corrected — data loss risk
Temperature > thresholds       — reads from dynamix.cfg (same thresholds as unRAID UI)
SMART overall-health FAILED    — get data off this drive immediately
```

**Silent when healthy:** no notification if all drives pass all checks.

---

### ── cert_monitor ─────────────────────────────────────────────────────────────

```
Category:   Monitors
Name:       cert_monitor
Schedule:   0 9 * * 0    (Sunday 9am)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Monitors/cert_monitor.sh
```

**What it does:**
Connects to each domain via live `openssl s_client` TLS connection and reads the
certificate actually being served — not NPM's API, not the certificate file.

**Why live TLS and not an API check:**
NPM can report a cert as renewed while the web server is still serving the old
expired cert (nginx not reloaded after renewal). A live TLS check catches this.
If a user would see a certificate error, this script catches it first.

**Thresholds (master.conf):**
```
> CERT_WARN_DAYS (30)   → silent ✅
≤ CERT_WARN_DAYS (30)   → notification — renew soon
≤ CERT_CRIT_DAYS (7)    → urgent notification — renew now
connection failed        → notification — cert unreachable
```

---

### ── backup_verify ───────────────────────────────────────────────────────────

```
Category:   Monitors
Name:       backup_verify
Schedule:   0 10 * * 0   (Sunday 10am)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Monitors/backup_verify.sh
```

**What it does:**
Randomly samples BACKUP_VERIFY_SAMPLE (default: 10) files per share, computes MD5
checksums on both sides via SSH, and compares. Does not trust rsync's exit code —
verifies actual content matches.

**Why verify independently:**
rsync exit 0 does not mean the copy is correct. Silent data corruption from faulty
hardware produces a file that exists on both sides with matching size and mtime but
different content. rsync never detects this. MD5 does.

**Pre-flight guards:**
```
check_connectivity()   — aborts if remote Tailscale IP unreachable
                          prevents all files showing as MISSING from network failure
check_remote_array()   — aborts if /mnt/user not mounted on remote
                          prevents all files showing as MISSING from array not started
```

---

### ── emby_session_report ─────────────────────────────────────────────────────

```
Category:   Monitors
Name:       emby_session_report
Schedule:   0 11 * * 0   (Sunday 11am)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Monitors/emby_session_report.sh
```

**What it does:**
Queries the Emby API for the past EMBY_REPORT_DAYS (7) days and reports streaming
activity. Each server reports on its own Emby instance via HOST*_EMBY_URL.

**Reports:**
Server info + version. Active streams now with direct play vs transcode breakdown.
Library counts (movies, episodes, songs). Weekly play count. Top EMBY_REPORT_TOP_N
most-played items. Most active users. Transcode ratio — notifies if > 80% transcoding
(usually means a client/format compatibility issue worth fixing).

---

### ── bandwidth_monitor_report ────────────────────────────────────────────────

```
Category:   Monitors
Name:       bandwidth_monitor_report
Schedule:   0 11 * * 0   (Sunday 11am — runs alongside emby_session_report)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Monitors/bandwidth_monitor.sh --report
```

**What it does:**
Reads the accumulated BANDWIDTH_LOG (written by rsync.sh after each sync) and
generates a weekly transfer summary.

**The log is written automatically by rsync.sh** — you never call
`--log-transfer` manually. This `--report` entry reads what rsync already logged.

**Reports:**
Per-profile run count, total transferred, average duration, failures.
Last 7 days timeline. Large transfer flags (> BANDWIDTH_WARN_GB).
Log format: DATE|TIME|profile|duration|status|bytes — version-stable field names
that survive rsync updates without parser changes.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🔄  FAILOVER
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

> **fallback.sh is started by array_started.sh — not scheduled separately.**
> These entries exist for manual control and testing only.

---

### ── failover_status ─────────────────────────────────────────────────────────

```
Category:   Failover
Name:       failover_status
Schedule:   None — run manually
Background: NO  (runs and exits — status check)
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Fallback/fallback.sh --status
```

**What it does:**
Shows current failover state at a glance without affecting the running loop.
State (NORMAL/FAILOVER/NO_INTERNET/DARK). Which tier containers are active.
Outage duration if in FAILOVER. Handback strike count if remote has returned.
DDNS containers currently running. Remote server Tailscale visibility.

**Use before any manual failover intervention.** Understanding the current state
before acting prevents making things worse.

---

### ── failover_test ────────────────────────────────────────────────────────────

```
Category:   Failover
Name:       failover_test
Schedule:   None — run manually (during a planned maintenance window)
Background: YES — test takes several minutes
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Fallback/fallback_test.sh --dry-run
```

**What it does:**
Validates the full failover lifecycle — detection, Tier 1 container start, handback,
container stop, DNS return — using an iptables rule to simulate remote unreachability.
No real outage. Safety trap removes the iptables rule on any exit condition including
ctrl-c and crashes — remote is always reachable after the test.

> ⚠️ **Always run with --dry-run first.** Dry run walks all 8 phases with full
> output but no iptables changes and no container starts/stops. Review timing and
> configuration before committing to a live test.

**For a live test** (real container lifecycle, brief service interruption):
Change `--dry-run` to nothing and run during a maintenance window.

**8 phases:** pre-flight → block remote → detect FAILOVER → verify Tier 1 started
→ restore remote → wait for handback → verify NORMAL restored → report pass/fail.

---

### ── fallback_state_reset ────────────────────────────────────────────────────

> **Not yet built.** `fallback_state_reset.sh` is a planned utility — not in `Fallback/` yet.
>
> **Manual workaround** when state file is stuck in FAILOVER after a failed test or
> forced kill of fallback.sh:
> ```bash
> # Verify first — right containers on right server, DDNS correct, fallback.sh not running
> cat /boot/config/fallback_state.db
> echo "state=NORMAL" > /boot/config/fallback_state.db
> ```
> Resets state only — does NOT start or stop any containers.

**When you need this:**
State file stuck in FAILOVER after a test that didn't complete cleanly. After killing
fallback.sh directly (not via plugin Abort button). After a failed handback.

**Verify before resetting:**
Right containers on right server. DDNS correct (`nslookup Gmer4Lfe.com`).
Both servers Tailscale connected. fallback.sh not currently running.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🔄  RSYNC
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Manual trigger entries for individual rsync profiles. Used when you need to sync
one specific share outside the normal nightly window — after a bulk import, before a
planned migration, or to verify connectivity.

> **These are NOT scheduled.** Daily and weekly orchestrators handle all scheduled syncs.
> These exist purely for on-demand manual triggers.

**Every rsync run includes:**
Pre-flight checks (connectivity, version parity, remote daemon, remote rootfs, share exists).
Profile-matched container stop/start (if profile requires it). Bandwidth logging.
Drive temperature check — exit 1 on WARN, exit 2 on CRITICAL.

---

### ── rsync_critical_data ─────────────────────────────────────────────────────

```
Category:   Rsync
Name:       rsync_critical_data
Schedule:   None — run manually
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Critical-Data
```

**What syncs:** auth stack — NPM proxy rules, TLS certificates, LLDAP user data,
Authelia policies and config, MariaDB database. Profile: critical-data. Stops auth
containers on both sides, syncs, restarts with dependency ordering (databases before auth).

---

### ── rsync_emby_clean ─────────────────────────────────────────────────────────

```
Category:   Rsync
Name:       rsync_emby_clean
Schedule:   None — run manually (also runs via weekly_sync_maintenance.sh)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Rsync/rsync.sh \
    /mnt/user/Media_Server/Emby --profile=emby
```

**What syncs:** full clean Emby mirror — all databases checkpointed, metadata, plugins,
config. Both Emby instances stopped during sync. Excludes transcodes, logs, cache, crash
files. This is the full-fidelity sync (vs emby-fallback which is the dirty 30-minute sync).

---

### ── rsync_arrs_stack ─────────────────────────────────────────────────────────

```
Category:   Rsync
Name:       rsync_arrs_stack
Schedule:   None — run manually
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Arrs_Stack
```

**What syncs:** all arr appdata — Sonarr, Radarr, Lidarr, Prowlarr, Bazarr, Pinchflat
databases and config. Profile: arrs_stack. Arr containers stopped on both sides during
sync to ensure clean SQLite snapshot (no mid-write database corruption).

---

### ── rsync_important_data ────────────────────────────────────────────────────

```
Category:   Rsync
Name:       rsync_important_data
Schedule:   None — run manually
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Important-Data
```

**What syncs:** NextCloud, Postgres-NextCloud, Immich, PostgreSQL-Immich appdata.
Profile: important-data. Postgres containers stop first, then NextCloud after delay.

---

### ── rsync_gmer4lfe ───────────────────────────────────────────────────────────

```
Category:   Rsync
Name:       rsync_gmer4lfe
Schedule:   None — run manually
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Rsync/rsync.sh \
    /mnt/user/appdata-Fallback/Gmer4Lfe
```

**What syncs:** HOST1 server appdata backup — everything in the Gmer4Lfe appdata share.
No profile container stop required. Incremental — safe to run at any time.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🐳  DOCKER_ESSENTIALS
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

> **docker_watchdog.sh is called every minute by watchdog_orchestrator.sh — not scheduled separately.**
> docker_daily_restart.sh is called by daily_sync_maintenance.sh.
> docker_weekly_restart.sh is called by weekly_sync_maintenance.sh.
> docker_network_connect.sh is called by array_started.sh.
> These entries exist for manual control only.

---

### ── docker_watchdog_status ──────────────────────────────────────────────────

```
Category:   Docker_Essentials
Name:       docker_watchdog_status
Schedule:   None — run manually
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Watchdogs/docker_watchdog.sh --status
```

**What it shows:**
Strike counts for all monitored containers. Current skip list with running/stopped state.
Whether startup grace period is active and how long remains. Whether RAM emergency
deferral is active (mem_shutdown_active from resource_watchdog). Last cycle timing.

---

### ── watchdog_skip_list_manager ──────────────────────────────────────────────

```
Category:   Docker_Essentials
Name:       watchdog_skip_list_manager
Schedule:   None — run manually (after fixing a skip-listed container)
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Docker_Essentials/watchdog_skip_list_manager.sh --status
```

**What it does:**
Views and manages the persistent container skip list. When docker_watchdog.sh restarts
the same container 3 times in 1 hour it adds it to the skip list and stops trying.
A critical notification is sent. This tool clears the list after you've fixed the cause.

**Recovery workflow:**
```
1. Run --status to see what's skip-listed and its restart history
2. Fix the underlying problem (check docker logs, disk space, database)
3. --clear ContainerName to remove from skip list + clear restart history
4. docker start ContainerName to confirm your fix works
5. Watchdog resumes monitoring on next cycle automatically
```

**Actions:**
```
--status                       — show skip list, running/stopped state, restart counts
--clear ContainerName          — clear specific container (prompts YES)
--clear ContainerName --force  — no prompt
--clear-all                    — clear everything
--dry-run                      — preview any clear action
```

---

### ── docker_daily_restart_manual ─────────────────────────────────────────────

```
Category:   Docker_Essentials
Name:       docker_daily_restart_manual
Schedule:   None — run manually (called automatically by daily_sync_maintenance.sh)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Docker_Essentials/docker_daily_restart.sh --dry-run
```

**What it does:**
Restarts containers in HOST*_DAILY_RESTART_CONTAINERS. Running → docker restart
(graceful stop + start). Stopped → left stopped (state respected). Missing → skip.
Dependency ordering: databases restart before applications. CONTAINER_DELAY wait
between dependency and dependent restarts.

**Default configured containers:** NginxProxyManager, Authelia, Dispatcharr,
Dispatcharr-Basic, ErsatzTV-Emby — services that degrade with uptime.

> Remove `--dry-run` to run live. Default is dry-run for safety when run manually.

---

### ── docker_network_connect_manual ───────────────────────────────────────────

```
Category:   Docker_Essentials
Name:       docker_network_connect_manual
Schedule:   None — run manually (runs automatically via array_started.sh)
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Docker_Essentials/docker_network_connect.sh --status
```

**What it does:**
Ensures custom Docker networks exist and containers are connected to them. Idempotent
— safe to run any time. If a network is missing (typically wiped by a unRAID update),
it creates it and notifies.

**Use after unRAID updates** if containers can't reach each other — the update may have
wiped custom networks. Run this to recreate them without restarting the array.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🎬  MEDIA
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

> **All media scripts are called by daily_sync_maintenance.sh via media_management.sh.**
> These entries exist for manual testing and targeted one-off runs.
> Run --dry-run --log for any new system or after configuration changes.

---

### ── media_shares_permissions_manual ────────────────────────────────────────

```
Category:   Media
Name:       media_shares_permissions_manual
Schedule:   None — run manually (runs automatically via daily_sync_maintenance.sh)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Media/media_shares_permissions.sh --dry-run
```

**What it does:**
Applies `nobody:users` ownership and 755 (dirs) / 664 (files) permissions to every
file in HOST*_MEDIA_PERMISSION_SHARES. Two separate passes — directories and files
get different modes. Idempotent — only changes what's wrong.

**When ownership drifts:** scp/cp from another machine leaves root:root. New container
before PUID/PGID is set. rsync without --chown. This fixes it all in one pass.

> Remove `--dry-run` to run live. Dry run shows count of files that would be corrected.

---

### ── media_cleaner_anime_manual ──────────────────────────────────────────────

```
Category:   Media
Name:       media_cleaner_anime_manual
Schedule:   None — run manually (runs automatically via daily_sync_maintenance.sh)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Media/media_cleaner.sh anime --dry-run
```

**What it does:**
Removes junk files from anime share folders using ANIME_FILE_PATTERNS. Patterns
include: *.sfv *.nfo *.rar *.sample* *.torrent *.url *.log and other scene release
debris. Runs before arr cleanup so orphan detection only sees actual media files.

---

### ── media_cleaner_media_manual ──────────────────────────────────────────────

```
Category:   Media
Name:       media_cleaner_media_manual
Schedule:   None — run manually
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Media/media_cleaner.sh media --dry-run
```

**What it does:**
Same as above but for media share folders using MEDIA_FILE_PATTERNS — all anime
patterns plus *.iso and *.lrc.

---

### ── lidarr_cleanup_manual ───────────────────────────────────────────────────

```
Category:   Media
Name:       lidarr_cleanup_manual
Schedule:   None — run manually (runs automatically via daily_sync_maintenance.sh)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Media/lidarr_cleanup.sh --dry-run --log
```

**What it does:**
Queries Lidarr API for all tracked file paths. Walks /mnt/user/Music. Files on disk
not in the API response that are older than LIDARR_ORPHAN_AGE (7 days) and not
currently open by any process = orphans = deleted.

**7 safety checks must ALL pass before any deletion:**
Lidarr running + healthy. API reachable. API version matches. Artist count > 0.
Tracked file count > 0. Count >= 80% of last known (detects database issues).
Total deletion size < LIDARR_MAX_DELETE_GB (50GB) — stops catastrophic path mismatches.

**Protected always:** *.jpg *.png (cover art), *.nfo (metadata), *.lrc (lyrics)
**After deletion:** notify_emby_scan() triggers "Clean Missing Files" in Emby immediately.

> Remove `--dry-run` only after reviewing the full log output. See README-Media.md
> for the complete safe testing procedure.

---

### ── sonarr_cleanup_manual ───────────────────────────────────────────────────

```
Category:   Media
Name:       sonarr_cleanup_manual
Schedule:   None — run manually
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Media/sonarr_cleanup.sh --dry-run --log
```

**What it does:**
Same logic as lidarr_cleanup but for TV files. Walks HOST*_SONARR_TV_ROOT.
Protected: *.jpg *.png (artwork), *.nfo, *.srt *.sub *.ass *.ssa (Bazarr subtitles).
Host-aware: HOST1 manages Tv_Shows, HOST2 manages Anime_Shows — correct root via detect_hosts().

---

### ── radarr_cleanup_manual ───────────────────────────────────────────────────

```
Category:   Media
Name:       radarr_cleanup_manual
Schedule:   None — run manually
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Media/radarr_cleanup.sh --dry-run --log
```

**What it does:**
Same logic for movie files. Walks HOST*_RADARR_MOVIES_ROOT.
Protected: *.jpg *.png (movie artwork), *.nfo, subtitles.
Host-aware: HOST1 manages Movies, HOST2 manages Anime_Movies.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🎬  TRANSCODES
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

> **ramdisk_setup.sh is called by array_started.sh — not scheduled separately.**
> **transcode_manager.sh and transcode_cleanup.sh are called by transcode_management.sh.**
> These entries exist for manual status checks and emergency use.

---

### ── ramdisk_status ────────────────────────────────────────────────────────────

```
Category:   Transcodes
Name:       ramdisk_status
Schedule:   None — run manually
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Transcodes/ramdisk_setup.sh --status
```

**What it shows:**
Whether ramdisk is mounted at RAMDISK_PATH. Current symlink target (ramdisk or SSD).
Whether transcoding-temp exists on ramdisk. SSD fallback directory state.
Current usage on both ramdisk and SSD.

---

### ── transcode_manager_manual ────────────────────────────────────────────────

```
Category:   Transcodes
Name:       transcode_manager_manual
Schedule:   None — run manually (runs via transcode_management.sh every 7 min)
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Transcodes/transcode_manager.sh --status
```

**What it shows:**
Current mode (smart/ramdisk/ssd). Ramdisk usage vs thresholds. Current symlink target.
Active sessions per configured TRANSCODE_SERVERS. Daily log stats (peak this week,
flip count, session split ramdisk vs SSD).

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🖥️  UNRAID_ESSENTIALS
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

> **stability_watchdog.sh is called by watchdog_orchestrator.sh — not scheduled separately.**
> **webgui_watchdog.sh, system_tuning_monitor.sh, clear_logs.sh are scheduled below.**
> **inotify_tuning.sh, php_fpm_max_children.sh, docker_syslog_filter.sh run via array_started.sh.**

---

### ── webgui_watchdog ───────────────────────────────────────────────────────────

```
Category:   unRAID_Essentials
Name:       webgui_watchdog
Schedule:   */10 * * * *  (every 10 minutes)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Watchdogs/System/webgui_watchdog.sh
```

**What it does:**
Checks WebGUI responsiveness and recovers it automatically. Silent when healthy —
runs 144 times per day with no output when the WebGUI is responding normally.

**Escalation path (each step only runs if previous failed):**
```
1. curl http://localhost → unresponsive for WEBGUI_TIMEOUT (5s)?
2. Restart nginx                    → wait 15s → recheck
3. Restart php-fpm                  → wait 10s → recheck  (handles worker saturation)
4. Restart emhttp (core daemon)     → wait 30s → recheck  (most disruptive, last resort)
5. All failed → notify warning — manual investigation needed
```

A notification fires on any restart so you know what recovered and when.

---

### ── clear_logs ───────────────────────────────────────────────────────────────

```
Category:   unRAID_Essentials
Name:       clear_logs
Schedule:   0 5 * * 0    (Sunday 5am — before Sunday maintenance window)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/unRAID_Essentials/clear_logs.sh
```

**What it does:**
Clears system and Docker container logs using size thresholds — only logs large
enough to be worth clearing.

**System logs** (/var/log/syslog, messages, dmesg): cleared only if > LOG_MIN_SIZE_MB (10MB).
Smaller logs contain useful recent history — not cleared.

**Docker container logs**: cleared per-container only if > LOG_DOCKER_MAX_MB (100MB).
Active containers like Emby and SABnzbd grow fastest. Truncation not rotation —
`> file` keeps the file descriptor valid while emptying. Safe for running services.

---

### ── stability_watchdog_status ──────────────────────────────────────────────────

```
Category:   unRAID_Essentials
Name:       stability_watchdog_status
Schedule:   None — run manually
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Watchdogs/stability_watchdog.sh --status
```

**What it shows:**
All three tiers with current thresholds. Active strike counts per check. RAM tier
status. Whether mem_shutdown_active is set (RAM emergency in progress). Recent
watchdog-triggered reboots. All 18 per-host toggle states.

---

### ── mover_stop ───────────────────────────────────────────────────────────────

```
Category:   unRAID_Essentials
Name:       mover_stop
Schedule:   None — run manually (before array operations needing mover stopped)
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/unRAID_Essentials/mover_stop.sh
```

**What it does:**
Stops the unRAID mover cleanly. Wall message warning to all logged-in users. Waits
MOVER_STOP_TIMEOUT seconds. SIGTERM (allows mover to finish current file — no partial
files). Waits + verifies. SIGKILL if still running (warns that partial files are possible).

---

### ── rsync_stop ────────────────────────────────────────────────────────────────

```
Category:   unRAID_Essentials
Name:       rsync_stop
Schedule:   None — run manually (when rsync needs stopping)
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/unRAID_Essentials/rsync_stop.sh --status
```

**What it does:**
Stops rsync intelligently. Auto-detects if an orchestrator is running:
Smart mode (default): kills rsync subprocess only — orchestrator sees rsync died and
exits cleanly. No orphaned lock files. Containers recover via docker_watchdog.sh.
`--full-stop`: kills orchestrator first then rsync — use for emergency stop.

---

### ── user_scripts_stop ────────────────────────────────────────────────────────

```
Category:   unRAID_Essentials
Name:       user_scripts_stop
Schedule:   None — run manually (before reboots, or when a script hangs)
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/unRAID_Essentials/user_scripts_stop.sh --status
```

**What it does:**
Finds all running User Script processes by /tmp/user.scripts path signature. Shows
script names (not just PIDs — you know what you're stopping). SIGTERM → verify →
SIGKILL. Self-exclusion prevents the script from killing itself. Called automatically
by server_reboot.sh before reboot.

---

### ── server_reboot ────────────────────────────────────────────────────────────

```
Category:   unRAID_Essentials
Name:       server_reboot
Schedule:   None — run manually (planned maintenance reboots)
Background: YES — sequence takes 1-2 minutes
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/unRAID_Essentials/server_reboot.sh --dry-run
```

**What it does:**
Graceful reboot with full pre-flight and clean shutdown. Wall message + unRAID
notification with MY_ID. Waits REBOOT_SLEEP (30s). virsh shutdown each running VM
then waits REBOOT_VM_WAIT for graceful exit. Stops Docker. Stops libvirt. sync.
/sbin/reboot.

Pre-flight warnings (not blocks): rsync running, mover running, active Emby sessions.
These show in the summary — you chose to reboot, these are context.

> Remove `--dry-run` to execute. Dry run walks the full sequence without rebooting —
> use it to verify what processes are active before committing.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🤝  PARTNERSHIP
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Manages the two-server auth-sharing relationship lifecycle. Automated check runs
inside critical_sync_maintenance.sh. These entries are for manual operations only.

---

### ── partnership_status ────────────────────────────────────────────────────────

```
Category:   Partnership
Name:       partnership_status
Schedule:   None — run manually
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Partnership/partnership_manage.sh --status
```

**What it shows:**
Both server state files (ACTIVE/INACTIVE). Offline counter (days since last successful
sync). Which server is the auth stack owner. Current WebUI redirect state on HOST2.
Whether CRITICAL_RSYNC_ENABLED is true.

---

### ── partnership_onboard ──────────────────────────────────────────────────────

```
Category:   Partnership
Name:       partnership_onboard
Schedule:   None — run manually (HOST1/owner only, once at relationship start)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Partnership/partnership_manage.sh --onboard --dry-run
```

**What it does:**
Establishes the auth-sharing partnership. Reconfigures HOST2 Docker template WebUI
URLs to redirect to HOST1 via Tailscale. Verifies routing works. Writes ACTIVE state
on both servers. Enables Critical-Data sync.

> Remove `--dry-run` to run live. Dry run shows exactly which templates would be
> changed and what URLs they'd become — review before committing.

---

### ── partnership_offboard ─────────────────────────────────────────────────────

```
Category:   Partnership
Name:       partnership_offboard
Schedule:   None — run manually (either server, when ending the partnership)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Partnership/partnership_manage.sh --offboard --dry-run
```

**What it does:**
Clean partnership separation. Stops rsync. Final sync to HOST2. Reconfigures HOST2
WebUI URLs back to localhost (HOST2 manages its own auth). Disables critical sync.
Grace period (PARTNERSHIP_GRACE_HOURS). Removes HOST2 from Tailscale tailnet.

Both servers can initiate. If HOST1 is unreachable when HOST2 offboards, deferred
finalisation completes automatically when HOST1 can next reach HOST2.

> Remove `--dry-run` to run live. 10-second countdown shown before any changes.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🔧  TOOLS
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Situational utilities — run when something needs fixing, not on a schedule.

---

### ── bulk_permissions_repair ─────────────────────────────────────────────────

```
Category:   Tools
Name:       bulk_permissions_repair
Schedule:   None — run manually
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Tools/bulk_permissions_repair.sh /mnt/user/Movies --dry-run
```

**What it does:**
Applies nobody:users 755/664 to a specific path immediately — without waiting for
the nightly media_shares_permissions.sh run. Use when an admin copy, scp, or bad
container left root-owned files that are blocking arr operations right now.
Replace /mnt/user/Movies with the actual path that needs fixing.

---

### ── container_data_export ────────────────────────────────────────────────────

```
Category:   Tools
Name:       container_data_export
Schedule:   None — run manually (before risky updates or migrations)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Tools/container_data_export.sh \
    Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/
```

**What it does:**
Stops the container cleanly. Archives the entire appdata directory to a timestamped
.tar.gz. Verifies archive integrity. Restarts the container (always, even if archiving
failed). Space check before stopping — won't start if output drive is too small.

Replace ContainerName, /path/to/appdata, and /output/dir for each use.
Output filename: ContainerName_YYYY-MM-DD_HH-MM.tar.gz

---

### ── emby_database_repair ─────────────────────────────────────────────────────

```
Category:   Tools
Name:       emby_database_repair
Schedule:   None — run manually (after power loss or repeated Emby crashes)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Tools/emby_database_repair.sh
```

**What it does:**
Stops Emby. Runs SQLite `PRAGMA integrity_check` on every Emby database. Restarts
Emby. Reports per-database with specific recovery guidance.

**Per-database recovery guidance:**
```
library.db          — CORRUPT → safe to delete, Emby fully rebuilds from media files
users.db            — CORRUPT → deleting resets ALL user accounts + watch history
                                check for recent backup (weekly_sync_maintenance.sh)
authentication.db   — CORRUPT → safe to delete, API keys regenerated on restart
activity.db         — CORRUPT → safe to delete, it's a log only
```

**This script checks and reports only** — it does not automatically delete anything.
Recovery is a human decision based on whether you have backups.

---

### ── zfs_pool_scrub ────────────────────────────────────────────────────────────

```
Category:   Tools
Name:       zfs_pool_scrub
Schedule:   None — run manually (monthly or after any disk/power event)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Tools/zfs_pool_scrub.sh
```

**What it does:**
Triggers ZFS scrub on all pools (except ZFS_REPORT_IGNORE_POOLS), waits for completion,
notifies with results. ZFS checksums every block and verifies — catches silent data
corruption before you discover it on reads. Safe to run while system is active.
For a specific pool: add the pool name as argument (bypasses ignore list).

---

### ── recreate_shares ──────────────────────────────────────────────────────────

```
Category:   Tools
Name:       recreate_shares
Schedule:   None — run manually (after fresh install or disk replacement)
Background: YES
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/Tools/recreate_shares.sh --dry-run
```

**What it does:**
Reads every .cfg in /boot/config/shares/, parses the shareInclude disk list, and
creates /mnt/diskN/sharename on each included disk. Run after restoring share .cfg
files to a fresh install — shares appear in the UI but directories are missing until
this creates them. chown nobody:users on creation.

---

## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🔀  GIT
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

---

### ── git_pull ─────────────────────────────────────────────────────────────────

```
Category:   Git
Name:       git_pull
Schedule:   None — run manually (called automatically by daily_sync_maintenance.sh)
Background: NO
```

```bash
#!/bin/bash
/mnt/user/appdata/Varaverk/unRAID_Essentials/git_pull_execute.sh
```

**What it does:**
Pulls latest commits from Gitea. Sets execute permission on all .sh files. This is
the deployment mechanism for the ecosystem — push changes from VS Code to Gitea,
then run this on both servers to deploy. daily_sync_maintenance.sh runs it
automatically at 1am as the first job in the window.

---

## ━━━ AT STARTUP OF ARRAY — ORDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

**Only one entry needed at startup:** `array_start`. It handles everything.

`array_started.sh` is the single "At Startup of Array" entry. It reads ARRAY_START_SCRIPTS
from master.conf and launches each in order. No other startup entries. No dragging to
reorder individual scripts. Change the order by editing master.conf.

```
Correct startup setup:
  1 entry: array_start   — At Startup of Array, Background: YES

Wrong startup setup (old approach):
  ramdisk_setup        — At Startup of Array
  docker_syslog_filter — At Startup of Array
  stability_watchdog     — Every Minute via watchdog_orchestrator
  docker_watchdog      — At Startup of Array
  failover             — At Startup of Array
  (unpredictable order, no coordination)
```

---

## ━━━ COMPLETE SCHEDULE REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

```
At Startup of Array:
  array_start                          BG: YES — single entry, handles everything

*/7 * * * *   (every 7 min):
  transcode_management                 BG: YES — cleanup then manager

*/30 * * * *  (every 30 min):
  critical_sync_maintenance            BG: YES — auth stack + Emby dirty sync + partnership

0 */6 * * *   (every 6 hours):
  arrs_failed_stalled_recovery         BG: YES — blocklist + re-search failed imports
  system_tuning_monitor                BG: YES — inotify + php-fpm snapshot

0 1 * * *     (1am daily):
  daily_sync_maintenance               BG: YES — full nightly window

0 5 * * 0     (Sunday 5am):
  clear_logs                           BG: YES — size-threshold log cleanup

0 6 * * 0     (Sunday 6am):
  zfs_memory_snapshot                  BG: YES — ZFS + memory report

0 7 * * 0     (Sunday 7am):
  sunday_morning_coffee_report         BG: YES — full weekly digest

0 8 * * *     (8am daily):
  weekly_health_digest                 BG: YES — profile-controlled notification

0 7 * * 0     (Sunday 7am):
  smart_health                         BG: YES — SMART attribute check

0 9 * * 0     (Sunday 9am):
  cert_monitor                         BG: YES — live TLS cert expiry check

0 10 * * 0    (Sunday 10am):
  backup_verify                        BG: YES — MD5 checksum mirror verification

0 11 * * 0    (Sunday 11am):
  emby_session_report                  BG: YES — Emby weekly usage stats
  bandwidth_monitor_report             BG: YES — rsync transfer summary

30 2 * * 0    (Sunday 2:30am):
  weekly_sync_maintenance              BG: YES — clean sync + image updates
```

---

## ━━━ AFTER CREATING ALL ENTRIES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Run each of these once manually before relying on scheduled runs:

```
array_start --dry-run          verify all startup scripts found and sequenced correctly
transcode_management --status  confirm ramdisk mounted and symlink pointing at ramdisk
critical_sync_maintenance      verify Critical-Data sync completes to HOST2
daily_sync_maintenance --dry-run  verify remote connectivity and share list
failover_status                confirm NORMAL state on both servers
docker_watchdog_status         confirm required containers all running
stability_watchdog_status         confirm no threshold alerts on first check
```