diff --git a/Notes_To-Do.md b/Notes_To-Do.md index db87e8e..0df3b89 100755 --- a/Notes_To-Do.md +++ b/Notes_To-Do.md @@ -18,13 +18,13 @@ need a script that check file locations. like if a kids movie is in movies. it n +array started go through logs.... array stopped didnt trigger when stopping the array, and this was before standardization +check all scripts for the dead verbose bug, like we did in orchs.... maybe even look for standardizing all scripts more like we did with orchs - - - +pull to dev, dev is out of date diff --git a/README-User_Script_Plug-in.md b/README-User_Script_Plug-in.md deleted file mode 100644 index 5949964..0000000 --- a/README-User_Script_Plug-in.md +++ /dev/null @@ -1,1686 +0,0 @@ -# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -# 📋 USER SCRIPTS PLUGIN — COMPLETE SETUP REFERENCE -# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -> **Fully superseded.** The Varaverk plugin handles all scheduling natively — no User -> Scripts entries are needed. Array start and stop are triggered by the plugin's own -> `disks_mounted` / `disks_unmounting` event hooks. Cron is managed via `schedule.json` -> and rebuilt on every boot. User Scripts is not required. -> See `Orchestrators/README-Orchestrators.md` for the current complete schedule. - -**Historical reference** — documents what each script does and how it was previously -configured via User Scripts. Still useful as a script catalogue. Paths updated to -reflect current boot-drive installation location. - ---- - -## ━━━ REQUIRED PLUGINS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Install both from Community Applications before creating any entries: - -``` -User Scripts — Andrew Zawadzki - Core plugin. Runs scripts via unRAID UI without terminal. - Required — nothing works without this. - -User Scripts Enhanced — Kurotaku - Adds categories, drag-to-reorder, UI improvements. - Required for the category organisation used here. -``` - ---- - -## ━━━ GLOBAL RULES — READ BEFORE CREATING ENTRIES ━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -``` -1. Every script entry → Run in Background: YES - Without background mode the plugin UI freezes waiting for the script to complete. - Long-running scripts and continuous loops (fallback, watchdogs) require background. - There are no exceptions — set background on everything. - -2. Run as root: YES - All scripts require root. Docker operations, rsync, kernel tuning, reboots. - -3. Category names match repo folder names exactly - When the plugin category and the git folder are identical, you never wonder - which folder a script lives in. No mental translation. - -4. Script name matches the script filename (without .sh) - daily_sync_maintenance.sh → name: daily_sync_maintenance - When logs reference a script name, you find the plugin entry immediately. - -5. --dry-run exists on every script - Test anything unfamiliar with --dry-run first. Pre-flight checks still run. - It shows what would happen without making changes. - -6. Most scripts never need to be run directly - The orchestrators handle timing and order. Individual script entries exist - for manual testing and one-off maintenance — not as replacement cron jobs. -``` - ---- - -## ━━━ CATEGORY STRUCTURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Create these categories in User Scripts Enhanced before adding any entries: - -``` -Orchestrators — scheduled maintenance windows and startup launcher -Fallback — mutual fallback state machine and test tools -Monitors — weekly reports, health checks, trend tracking -Rsync — manual per-profile sync triggers -Docker_Essentials — container management tools (manual use) -Media — library maintenance tools (manual use) -Transcodes — ramdisk and transcode tools (manual use) -System_Essentials — server management tools (manual use) -Partnership — two-server relationship tools (manual use) -Tools — repair, recovery, and one-time utilities -Git — script deployment -``` - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🎯 ORCHESTRATORS -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Orchestrators are the only scripts that run on a schedule. They coordinate multiple -child scripts in the correct order. Adding or removing a job means editing master.conf -— never the orchestrator script itself. - ---- - -### ── array_start ───────────────────────────────────────────────────────────── - -``` -Category: Orchestrators -Name: array_start -Schedule: At Startup of Array -Background: YES — required (starts continuous background processes) -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Orchestrators/array_started.sh -``` - -**What it does:** -The single "At Startup of Array" entry for the entire ecosystem. Launches every -startup script in the order defined by ARRAY_START_SCRIPTS in master.conf. You -never add other startup scripts directly to User Scripts — add them to master.conf. - -**What it starts (in order):** -``` -inotify_tuning.sh — raises kernel inotify limits before containers start - containers inherit limits at startup — must be first -docker_syslog_filter.sh — suppresses Docker veth noise before containers create interfaces -php_fpm_max_children.sh — sets PHP-FPM workers to 250 before WebGUI serves requests -ramdisk_setup.sh — creates /mnt/ramdisk_transcodes and symlink before Emby starts -docker_network_connect.sh— ensures custom networks exist + containers connected -stability_watchdog.sh ← last-resort — three-tier server stability watchdog -docker_watchdog.sh ← continuous — two-tier container healing watchdog -fallback.sh ← continuous — mutual fallback state machine -``` - -**After running:** -One-shot scripts exit cleanly. Three continuous processes run in the background until -the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|fallback"` - ---- - -### ── transcode_management ───────────────────────────────────────────────────── - -``` -Category: Orchestrators -Name: transcode_management -Schedule: */7 * * * * (every 7 minutes) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Orchestrators/transcode_management.sh -``` - -**What it does:** -Runs transcode_cleanup.sh then transcode_manager.sh — in that exact order every -7 minutes. Order is non-negotiable: cleanup removes stale files first so the manager -sees real active session usage, not inflated numbers from ended sessions. - -**transcode_cleanup.sh:** -Removes HLS segment files older than TRANSCODE_MAX_AGE minutes that are not currently -open by any process. Uses one lsof call per location — not one per file. Covers both -ramdisk and SSD fallback. Never touches the transcoding-temp directory. - -**transcode_manager.sh:** -Checks ramdisk usage against RAMDISK_WARN_GB. Flips the /mnt/ram-transcode symlink -between ramdisk and SSD fallback based on current usage. Shows active Emby sessions -with their play method. Writes one daily stats entry to TRANSCODE_DAILY_LOG. -Runs safety checks every cycle: missing symlink → recreate, ramdisk disappeared → -auto-flip to SSD, missing transcoding-temp → recreate immediately. - -**Output when healthy:** session list only — no flip messages = ramdisk managing fine - -> ⚠️ The Emby/Jellyfin Docker container must have -> `--mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode` -> in Extra Parameters. Do **not** add `bind-propagation=shared` — runc v1.3.5+ -> (Docker 29.x, Unraid 7.3+) rejects it with a mountinfo error. The ramdisk tmpfs -> is already MS\_SHARED at the kernel level — propagation is inherited automatically. - ---- - -### ── critical_sync_maintenance ─────────────────────────────────────────────── - -``` -Category: Orchestrators -Name: critical_sync_maintenance -Schedule: */30 * * * * (every 30 minutes) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Orchestrators/critical_sync_maintenance.sh -``` - -**What it does:** -Keeps the auth stack and Emby watch state current on HOST2 — 30 minutes is the -maximum staleness for fallback-critical data. - -**Jobs in order:** -``` -rsync Critical-Data — pushes auth stack to HOST2 (NPM rules, certs, LLDAP users, - Authelia config). HOST2's auth is always within 30min of HOST1. -rsync Emby dirty sync — pushes watch states, user activity, library delta to HOST2 - with Emby running. WAL files excluded — safe mid-write. - HOST2 Emby restarts after sync to pick up config changes. -partnership --check — reads remote state file, increments offline counter if HOST2 - unreachable, triggers auto-offboard at PARTNERSHIP_OFFLINE_THRESHOLD -``` - -**Silent when healthy:** no output on clean cycles — runs 48 times per day. - ---- - -### ── arrs_failed_stalled_recovery ─────────────────────────────────────────── - -``` -Category: Orchestrators -Name: arrs_failed_stalled_recovery -Schedule: 0 */6 * * * (every 6 hours) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Orchestrators/arrs_failed_stalled_recovery.sh -``` - -**What it does:** -Automatically recovers failed imports and stalled downloads across Sonarr, Radarr, -and Lidarr. A release that downloaded but failed to import sits in the queue -permanently without this — no arr retries importFailed items on its own. - -**What it finds and acts on:** -``` -importFailed — downloaded but couldn't import → blocklist + remove + new search -importPending — stuck in import queue → blocklist + remove + new search -error — serious failure state → blocklist + remove + new search -stalled — no seeders or no progress → blocklist + remove + new search -``` - -**What it never touches:** items with status "downloading" or "imported". -Age gate: items newer than ARR_IMPORT_RECOVERY_AGE (default: 6 hours) are skipped — -the arr may still retry on its own. - -**Result:** failed imports are automatically replaced while you sleep. - ---- - -### ── daily_sync_maintenance ────────────────────────────────────────────────── - -``` -Category: Orchestrators -Name: daily_sync_maintenance -Schedule: 0 1 * * * (1am daily) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh -``` - -**What it does:** -The full 1am daily maintenance window — everything that needs to happen overnight -in one orchestrated sequence. - -**Execution order:** -``` -1. git_pull_execute.sh — pull latest scripts FIRST, everything below runs on - current code - -2. rsync DAILY_SYNC_SHARES — push this server's truth shares to the other server - HOST1 pushes: Movies, Tv_Shows, Music, Kids content, Sports... - HOST2 pushes: Anime_Shows, Anime_Movies - Personal encrypted shares appended after — encrypted blocks, remote can't read - - Drive temp exit codes: - exit 1 (WARN) → skip this share, continue to next - exit 2 (CRIT) → abort ALL remaining syncs, notify immediately - -3. media_shares_permissions.sh — apply nobody:users 755/664 to all media shares - arr cleanup depends on correct ownership - -4. media_cleaner.sh anime — remove .sfv .nfo .rar .sample and other scene junk - media_cleaner.sh media from anime and media share folders - junk removal BEFORE orphan detection — cleaner results - -5. lidarr_cleanup.sh — identify and delete orphaned music files not tracked - sonarr_cleanup.sh by the arr API. 7 safety checks must all pass before - radarr_cleanup.sh a single file is deleted. notify_emby_scan() called - after any deletion — Emby library cleaned immediately - -6. docker_daily_restart.sh — restart connection-heavy containers that degrade over time - NginxProxyManager, Authelia, Dispatcharr, ErsatzTV-Emby - Running → docker restart. Stopped → left stopped. -``` - -**Notify:** silent on success. One notification on failure with count of failed jobs. -**Both servers run this independently at 1am** — same script, correct direction via detect_hosts(). - ---- - -### ── rsync_emby_fallback ───────────────────────────────────────────────────── - -> **Superseded by `critical_sync_maintenance`.** Delete this User Script from the -> plugin. The emby-fallback dirty sync now runs as step 2 inside -> `critical_sync_maintenance.sh` every 30 minutes alongside the auth stack sync -> and partnership check. Running both produces duplicate syncs. - -The Emby dirty sync (`/mnt/user/Media_Server/Emby --profile=emby-fallback`) is -configured in `HOST1_CRITICAL_SYNC_SHARES` in host1.conf. It runs automatically -when critical_sync_maintenance.sh runs. - ---- - -### ── weekly_sync_maintenance ───────────────────────────────────────────────── - -``` -Category: Orchestrators -Name: weekly_sync_maintenance -Schedule: 30 2 * * 0 (Sunday 2:30am) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Orchestrators/weekly_sync_maintenance.sh -``` - -**What it does:** -The full Sunday maintenance window — clean sync with containers stopped, image -updates at zero extra downtime, weekly container restarts. - -**Execution order:** -``` -1. Pre-flight — connectivity, remote Docker daemon, remote rootfs -2. Stop local containers — Emby + auth stack stopped on this server -3. Stop remote containers— Emby + auth stack stopped on remote via SSH -4. Pull images locally — if WEEKLY_SYNC_UPDATES=true (containers already stopped, - free downtime, both servers start on identical image versions) -5. Pull images remotely — if WEEKLY_SYNC_UPDATES_REMOTE=true (SSH to remote) -6. rsync WEEKLY_SYNC_SHARES: - /mnt/user/Media_Server/Emby — full clean mirror, databases checkpointed - /mnt/user/appdata-Fallback/Critical-Data — auth stack clean state -7. Start remote containers— dependency order, new image, settle + verify each -8. Start local containers — same -9. docker_weekly_restart.sh — restarts less-critical services that benefit from - weekly clean start: NextCloud, AdGuard, Immich -``` - -**Why containers stop for Emby sync:** -Weekly sync is a full clean mirror including databases. Emby must be stopped to -checkpoint WAL — ensures a consistent database copy, not a snapshot mid-write. -~30 seconds of downtime. The 30-minute dirty sync covers users until this clean sync. - -**Why weekly and not nightly:** -Emby builds a warm image thumbnail cache on HOST2 throughout the week. Nightly sync -resets this cache — cold loads every morning. Weekly sync: cache stays warm 6 days, -resets Sunday night while users sleep. - ---- - -### ── sunday_morning_coffee_report ──────────────────────────────────────────── - -``` -Category: Orchestrators -Name: sunday_morning_coffee_report -Schedule: 0 7 * * 0 (Sunday 7am) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Orchestrators/sunday_morning_coffee_report.sh -``` - -**What it does:** -Full weekly system overview — everything that happened this week in one clean read. -Designed to be read over coffee after the 2:30am maintenance window has completed -and run all the Sunday morning monitor scripts. - -**Sections:** -``` -🖥️ System — uptime, memory, boot drive, cache usage, reboots this week -📀 Array — disk count, parity status, ZFS health, drive temperatures -🎬 Transcodes — weekly ramdisk peak, flip count, session split (ramdisk vs SSD) -🎵 Media Activity — arr cleanup stats (orphans removed, space freed), queue depth -🌐 Rsync — weekly transfer totals, per-share breakdown, failures -🛡️ Watchdog — system + docker watchdog state, active strikes, skip list -🔐 Security — SSL cert expiry days remaining per domain (live TLS check) -📊 Emby — version, active streams now, weekly play count, top users -⚙️ System Health — inotify peak/avg/warnings, php-fpm peak/avg, SMART, git sync -⚠️ Issues — anything requiring attention from any section above -``` - -**Delivery:** unRAID notification + Discord webhook (if configured). -**Silent when healthy:** footer shows ✅ if no issues, ⚠️ with count if something needs attention. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 📊 MONITORS -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Monitor scripts observe and report — they never act. Read-only except for two bounded -log writers. All are safe to run manually at any time. - ---- - -### ── weekly_health_digest ──────────────────────────────────────────────────── - -``` -Category: Monitors -Name: weekly_health_digest -Schedule: 0 8 * * * (8am daily — DIGEST_PROFILE in master.conf controls notify behaviour) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Monitors/weekly_health_digest.sh -``` - -**What it does:** -Reads state files from across the ecosystem — watchdog strikes, fallback state, skip -list, bandwidth history, transcode stats, cert expiry — and compiles them into a -digest notification. Reads only, writes nothing. - -**DIGEST_PROFILE controls when it notifies (set in master.conf):** -``` -always — sends every run regardless of content (daily digest, even when healthy) -smart — sends only when something worth reporting is found - triggers: watchdog strikes, non-NORMAL fallback, cert warnings, bandwidth spikes -weekly — sends once per week on DIGEST_DAY (Sunday), silent all other days -``` - -Schedule never changes (8am daily). Switching between profiles requires only one -line change in master.conf — no cron edit needed. - ---- - -### ── system_tuning_monitor ─────────────────────────────────────────────────── - -``` -Category: Monitors -Name: system_tuning_monitor -Schedule: 0 */6 * * * (every 6 hours) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Monitors/system_tuning_monitor.sh -``` - -**What it does:** -Snapshots inotify and PHP-FPM utilisation every 6 hours. Writes one bounded log -entry. The sunday_morning_coffee_report reads this log for peak, average, and -warning counts over the week. - -**inotify monitoring:** -Current instances in use vs kernel limit. Warns above INOTIFY_WARN_PCT (80%). -Shows top 5 consumers. inotify exhaustion = downloads complete but arrs don't detect -them, Live TV stutters, library updates stop — no obvious error anywhere. - -**PHP-FPM monitoring:** -Active workers vs PHP_MAX_CHILDREN. Warns above PHP_FPM_WARN_PCT (80%). -Saturation = WebGUI slows, settings saves hang, container starts from UI timeout. - -**Log:** TUNING_MONITOR_LOG — bounded, trimmed to TUNING_LOG_RETENTION days on write. - ---- - -### ── zfs_memory_snapshot ───────────────────────────────────────────────────── - -``` -Category: Monitors -Name: zfs_memory_snapshot -Schedule: 0 6 * * 0 (Sunday 6am — first Sunday script, before everything else) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Monitors/zfs_memory_snapshot.sh -``` - -**What it does:** -Weekly ZFS pool health and memory diagnostic. Runs before other Sunday monitors so -pool state is captured while the system is post-maintenance and settled. - -**Reports:** -ZFS pool status (ONLINE/DEGRADED/FAULTED) per pool. ARC size vs max, hit rate, metadata -ratio. Total and available RAM. Top 10 Docker containers by memory. Kernel pressure -via vmstat. Warns if ARC > ZFS_REPORT_ARC_WARN_PCT or RAM < ZFS_REPORT_FREE_WARN_GB. -Output goes to both console and /var/log/zfs-weekly-health.log for week-over-week comparison. - ---- - -### ── smart_health ──────────────────────────────────────────────────────────── - -``` -Category: Monitors -Name: smart_health -Schedule: 0 7 * * 0 (Sunday 7am) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Monitors/smart_health.sh -``` - -**What it does:** -Queries SMART attributes for every /dev/sd* and /dev/nvme* drive weekly. Drives -in HOST*_SMART_IGNORE_DRIVES (typically sda — the boot USB) are skipped. - -**What triggers a notification:** -``` -Reallocated_Sector_Ct > 0 — drive found and swapped a bad sector — warning sign -Current_Pending_Sector > 0 — drive suspects sectors bad, not yet confirmed -Offline_Uncorrectable > 0 — sectors that couldn't be corrected — data loss risk -Temperature > thresholds — reads from dynamix.cfg (same thresholds as unRAID UI) -SMART overall-health FAILED — get data off this drive immediately -``` - -**Silent when healthy:** no notification if all drives pass all checks. - ---- - -### ── cert_monitor ───────────────────────────────────────────────────────────── - -``` -Category: Monitors -Name: cert_monitor -Schedule: 0 9 * * 0 (Sunday 9am) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Monitors/cert_monitor.sh -``` - -**What it does:** -Connects to each domain via live `openssl s_client` TLS connection and reads the -certificate actually being served — not NPM's API, not the certificate file. - -**Why live TLS and not an API check:** -NPM can report a cert as renewed while the web server is still serving the old -expired cert (nginx not reloaded after renewal). A live TLS check catches this. -If a user would see a certificate error, this script catches it first. - -**Thresholds (master.conf):** -``` -> CERT_WARN_DAYS (30) → silent ✅ -≤ CERT_WARN_DAYS (30) → notification — renew soon -≤ CERT_CRIT_DAYS (7) → urgent notification — renew now -connection failed → notification — cert unreachable -``` - ---- - -### ── backup_verify ─────────────────────────────────────────────────────────── - -``` -Category: Monitors -Name: backup_verify -Schedule: 0 10 * * 0 (Sunday 10am) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Monitors/backup_verify.sh -``` - -**What it does:** -Randomly samples BACKUP_VERIFY_SAMPLE (default: 10) files per share, computes MD5 -checksums on both sides via SSH, and compares. Does not trust rsync's exit code — -verifies actual content matches. - -**Why verify independently:** -rsync exit 0 does not mean the copy is correct. Silent data corruption from faulty -hardware produces a file that exists on both sides with matching size and mtime but -different content. rsync never detects this. MD5 does. - -**Pre-flight guards:** -``` -check_connectivity() — aborts if remote Tailscale IP unreachable - prevents all files showing as MISSING from network failure -check_remote_array() — aborts if /mnt/user not mounted on remote - prevents all files showing as MISSING from array not started -``` - ---- - -### ── emby_session_report ───────────────────────────────────────────────────── - -``` -Category: Monitors -Name: emby_session_report -Schedule: 0 11 * * 0 (Sunday 11am) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Monitors/emby_session_report.sh -``` - -**What it does:** -Queries the Emby API for the past EMBY_REPORT_DAYS (7) days and reports streaming -activity. Each server reports on its own Emby instance via HOST*_EMBY_URL. - -**Reports:** -Server info + version. Active streams now with direct play vs transcode breakdown. -Library counts (movies, episodes, songs). Weekly play count. Top EMBY_REPORT_TOP_N -most-played items. Most active users. Transcode ratio — notifies if > 80% transcoding -(usually means a client/format compatibility issue worth fixing). - ---- - -### ── bandwidth_monitor_report ──────────────────────────────────────────────── - -``` -Category: Monitors -Name: bandwidth_monitor_report -Schedule: 0 11 * * 0 (Sunday 11am — runs alongside emby_session_report) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Monitors/bandwidth_monitor.sh --report -``` - -**What it does:** -Reads the accumulated BANDWIDTH_LOG (written by rsync.sh after each sync) and -generates a weekly transfer summary. - -**The log is written automatically by rsync.sh** — you never call -`--log-transfer` manually. This `--report` entry reads what rsync already logged. - -**Reports:** -Per-profile run count, total transferred, average duration, failures. -Last 7 days timeline. Large transfer flags (> BANDWIDTH_WARN_GB). -Log format: DATE|TIME|profile|duration|status|bytes — version-stable field names -that survive rsync updates without parser changes. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🔄 FALLBACK -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -> **fallback.sh is started by array_started.sh — not scheduled separately.** -> These entries exist for manual control and testing only. - ---- - -### ── fallback_status ───────────────────────────────────────────────────────── - -``` -Category: Fallback -Name: fallback_status -Schedule: None — run manually -Background: NO (runs and exits — status check) -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Fallback/fallback.sh --status -``` - -**What it does:** -Shows current fallback state at a glance without affecting the running loop. -State (NORMAL/FALLBACK/NO_INTERNET/DARK). Which tier containers are active. -Outage duration if in FALLBACK. Handback strike count if remote has returned. -DDNS containers currently running. Remote server Tailscale visibility. - -**Use before any manual fallback intervention.** Understanding the current state -before acting prevents making things worse. - ---- - -### ── fallback_test ──────────────────────────────────────────────────────────── - -``` -Category: Fallback -Name: fallback_test -Schedule: None — run manually (during a planned maintenance window) -Background: YES — test takes several minutes -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Fallback/fallback_test.sh --dry-run -``` - -**What it does:** -Validates the full fallback lifecycle — detection, Tier 1 container start, handback, -container stop, DNS return — using an iptables rule to simulate remote unreachability. -No real outage. Safety trap removes the iptables rule on any exit condition including -ctrl-c and crashes — remote is always reachable after the test. - -> ⚠️ **Always run with --dry-run first.** Dry run walks all 8 phases with full -> output but no iptables changes and no container starts/stops. Review timing and -> configuration before committing to a live test. - -**For a live test** (real container lifecycle, brief service interruption): -Change `--dry-run` to nothing and run during a maintenance window. - -**8 phases:** pre-flight → block remote → detect FALLBACK → verify Tier 1 started -→ restore remote → wait for handback → verify NORMAL restored → report pass/fail. - ---- - -### ── fallback_state_reset ──────────────────────────────────────────────────── - -> **Not yet built.** `fallback_state_reset.sh` is a planned utility — not in `Fallback/` yet. -> -> **Manual workaround** when state file is stuck in FALLBACK after a failed test or -> forced kill of fallback.sh: -> ```bash -> # Verify first — right containers on right server, DDNS correct, fallback.sh not running -> cat "$STATE_DIR/fallback_state.db" -> echo "state=NORMAL" > "$STATE_DIR/fallback_state.db" -> ``` -> Resets state only — does NOT start or stop any containers. - -**When you need this:** -State file stuck in FALLBACK after a test that didn't complete cleanly. After killing -fallback.sh directly (not via plugin Abort button). After a failed handback. - -**Verify before resetting:** -Right containers on right server. DDNS correct (`nslookup Gmer4Lfe.com`). -Both servers Tailscale connected. fallback.sh not currently running. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🔄 RSYNC -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Manual trigger entries for individual rsync profiles. Used when you need to sync -one specific share outside the normal nightly window — after a bulk import, before a -planned migration, or to verify connectivity. - -> **These are NOT scheduled.** Daily and weekly orchestrators handle all scheduled syncs. -> These exist purely for on-demand manual triggers. - -**Every rsync run includes:** -Pre-flight checks (connectivity, version parity, remote daemon, remote rootfs, share exists). -Profile-matched container stop/start (if profile requires it). Bandwidth logging. -Drive temperature check — exit 1 on WARN, exit 2 on CRITICAL. - ---- - -### ── rsync_critical_data ───────────────────────────────────────────────────── - -``` -Category: Rsync -Name: rsync_critical_data -Schedule: None — run manually -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Rsync/rsync.sh \ - /mnt/user/appdata-Fallback/Critical-Data -``` - -**What syncs:** auth stack — NPM proxy rules, TLS certificates, LLDAP user data, -Authelia policies and config, MariaDB database. Profile: critical-data. Stops auth -containers on both sides, syncs, restarts with dependency ordering (databases before auth). - ---- - -### ── rsync_emby_clean ───────────────────────────────────────────────────────── - -``` -Category: Rsync -Name: rsync_emby_clean -Schedule: None — run manually (also runs via weekly_sync_maintenance.sh) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Rsync/rsync.sh \ - /mnt/user/Media_Server/Emby --profile=emby -``` - -**What syncs:** full clean Emby mirror — all databases checkpointed, metadata, plugins, -config. Both Emby instances stopped during sync. Excludes transcodes, logs, cache, crash -files. This is the full-fidelity sync (vs emby-fallback which is the dirty 30-minute sync). - ---- - -### ── rsync_arrs_stack ───────────────────────────────────────────────────────── - -``` -Category: Rsync -Name: rsync_arrs_stack -Schedule: None — run manually -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Rsync/rsync.sh \ - /mnt/user/appdata-Fallback/Arrs_Stack -``` - -**What syncs:** all arr appdata — Sonarr, Radarr, Lidarr, Prowlarr, Bazarr, Pinchflat -databases and config. Profile: arrs_stack. Arr containers stopped on both sides during -sync to ensure clean SQLite snapshot (no mid-write database corruption). - ---- - -### ── rsync_important_data ──────────────────────────────────────────────────── - -``` -Category: Rsync -Name: rsync_important_data -Schedule: None — run manually -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Rsync/rsync.sh \ - /mnt/user/appdata-Fallback/Important-Data -``` - -**What syncs:** NextCloud, Postgres-NextCloud, Immich, PostgreSQL-Immich appdata. -Profile: important-data. Postgres containers stop first, then NextCloud after delay. - ---- - -### ── rsync_gmer4lfe ─────────────────────────────────────────────────────────── - -``` -Category: Rsync -Name: rsync_gmer4lfe -Schedule: None — run manually -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Rsync/rsync.sh \ - /mnt/user/appdata-Fallback/Gmer4Lfe -``` - -**What syncs:** HOST1 server appdata backup — everything in the Gmer4Lfe appdata share. -No profile container stop required. Incremental — safe to run at any time. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🐳 DOCKER_ESSENTIALS -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -> **docker_watchdog.sh is called every 15 minutes by watchdog_orchestrator.sh — not scheduled separately.** -> docker_daily_restart.sh is called by daily_sync_maintenance.sh. -> docker_weekly_restart.sh is called by weekly_sync_maintenance.sh. -> docker_network_connect.sh is called by array_started.sh. -> These entries exist for manual control only. - ---- - -### ── docker_watchdog_status ────────────────────────────────────────────────── - -``` -Category: Docker_Essentials -Name: docker_watchdog_status -Schedule: None — run manually -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Watchdogs/docker_watchdog.sh --status -``` - -**What it shows:** -Strike counts for all monitored containers. Current skip list with running/stopped state. -Whether startup grace period is active and how long remains. Whether RAM emergency -deferral is active (mem_shutdown_active from resource_watchdog). Last cycle timing. - ---- - -### ── watchdog_skip_list_manager ────────────────────────────────────────────── - -``` -Category: Docker_Essentials -Name: watchdog_skip_list_manager -Schedule: None — run manually (after fixing a skip-listed container) -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Docker_Essentials/watchdog_skip_list_manager.sh --status -``` - -**What it does:** -Views and manages the persistent container skip list. When docker_watchdog.sh restarts -the same container 3 times in 1 hour it adds it to the skip list and stops trying. -A critical notification is sent. This tool clears the list after you've fixed the cause. - -**Recovery workflow:** -``` -1. Run --status to see what's skip-listed and its restart history -2. Fix the underlying problem (check docker logs, disk space, database) -3. --clear ContainerName to remove from skip list + clear restart history -4. docker start ContainerName to confirm your fix works -5. Watchdog resumes monitoring on next cycle automatically -``` - -**Actions:** -``` ---status — show skip list, running/stopped state, restart counts ---clear ContainerName — clear specific container (prompts YES) ---clear ContainerName --force — no prompt ---clear-all — clear everything ---dry-run — preview any clear action -``` - ---- - -### ── docker_daily_restart_manual ───────────────────────────────────────────── - -``` -Category: Docker_Essentials -Name: docker_daily_restart_manual -Schedule: None — run manually (called automatically by daily_sync_maintenance.sh) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Docker_Essentials/docker_daily_restart.sh --dry-run -``` - -**What it does:** -Restarts containers in HOST*_DAILY_RESTART_CONTAINERS. Running → docker restart -(graceful stop + start). Stopped → left stopped (state respected). Missing → skip. -Dependency ordering: databases restart before applications. CONTAINER_DELAY wait -between dependency and dependent restarts. - -**Default configured containers:** NginxProxyManager, Authelia, Dispatcharr, -Dispatcharr-Basic, ErsatzTV-Emby — services that degrade with uptime. - -> Remove `--dry-run` to run live. Default is dry-run for safety when run manually. - ---- - -### ── docker_network_connect_manual ─────────────────────────────────────────── - -``` -Category: Docker_Essentials -Name: docker_network_connect_manual -Schedule: None — run manually (runs automatically via array_started.sh) -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Docker_Essentials/docker_network_connect.sh --status -``` - -**What it does:** -Ensures custom Docker networks exist and containers are connected to them. Idempotent -— safe to run any time. If a network is missing (typically wiped by a unRAID update), -it creates it and notifies. - -**Use after unRAID updates** if containers can't reach each other — the update may have -wiped custom networks. Run this to recreate them without restarting the array. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🎬 MEDIA -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -> **All media scripts are called by daily_sync_maintenance.sh via media_management.sh.** -> These entries exist for manual testing and targeted one-off runs. -> Run --dry-run --log for any new system or after configuration changes. - ---- - -### ── media_shares_permissions_manual ──────────────────────────────────────── - -``` -Category: Media -Name: media_shares_permissions_manual -Schedule: None — run manually (runs automatically via daily_sync_maintenance.sh) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Media/media_shares_permissions.sh --dry-run -``` - -**What it does:** -Applies `nobody:users` ownership and 755 (dirs) / 664 (files) permissions to every -file in HOST*_MEDIA_PERMISSION_SHARES. Two separate passes — directories and files -get different modes. Idempotent — only changes what's wrong. - -**When ownership drifts:** scp/cp from another machine leaves root:root. New container -before PUID/PGID is set. rsync without --chown. This fixes it all in one pass. - -> Remove `--dry-run` to run live. Dry run shows count of files that would be corrected. - ---- - -### ── media_cleaner_anime_manual ────────────────────────────────────────────── - -``` -Category: Media -Name: media_cleaner_anime_manual -Schedule: None — run manually (runs automatically via daily_sync_maintenance.sh) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Media/media_cleaner.sh anime --dry-run -``` - -**What it does:** -Removes junk files from anime share folders using ANIME_FILE_PATTERNS. Patterns -include: *.sfv *.nfo *.rar *.sample* *.torrent *.url *.log and other scene release -debris. Runs before arr cleanup so orphan detection only sees actual media files. - ---- - -### ── media_cleaner_media_manual ────────────────────────────────────────────── - -``` -Category: Media -Name: media_cleaner_media_manual -Schedule: None — run manually -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Media/media_cleaner.sh media --dry-run -``` - -**What it does:** -Same as above but for media share folders using MEDIA_FILE_PATTERNS — all anime -patterns plus *.iso and *.lrc. - ---- - -### ── lidarr_cleanup_manual ─────────────────────────────────────────────────── - -``` -Category: Media -Name: lidarr_cleanup_manual -Schedule: None — run manually (runs automatically via daily_sync_maintenance.sh) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Arrs_Stack/lidarr_cleanup.sh --dry-run --log -``` - -**What it does:** -Queries Lidarr API for all tracked file paths. Walks /mnt/user/Music. Files on disk -not in the API response that are older than LIDARR_ORPHAN_AGE (7 days) and not -currently open by any process = orphans = deleted. - -**7 safety checks must ALL pass before any deletion:** -Lidarr running + healthy. API reachable. API version matches. Artist count > 0. -Tracked file count > 0. Count >= 80% of last known (detects database issues). -Total deletion size < LIDARR_MAX_DELETE_GB (50GB) — stops catastrophic path mismatches. - -**Protected always:** *.jpg *.png (cover art), *.nfo (metadata), *.lrc (lyrics) -**After deletion:** notify_emby_scan() triggers "Clean Missing Files" in Emby immediately. - -> Remove `--dry-run` only after reviewing the full log output. See README-Media.md -> for the complete safe testing procedure. - ---- - -### ── sonarr_cleanup_manual ─────────────────────────────────────────────────── - -``` -Category: Media -Name: sonarr_cleanup_manual -Schedule: None — run manually -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Arrs_Stack/sonarr_cleanup.sh --dry-run --log -``` - -**What it does:** -Same logic as lidarr_cleanup but for TV files. Walks HOST*_SONARR_TV_ROOT. -Protected: *.jpg *.png (artwork), *.nfo, *.srt *.sub *.ass *.ssa (Bazarr subtitles). -Host-aware: HOST1 manages Tv_Shows, HOST2 manages Anime_Shows — correct root via detect_hosts(). - ---- - -### ── radarr_cleanup_manual ─────────────────────────────────────────────────── - -``` -Category: Media -Name: radarr_cleanup_manual -Schedule: None — run manually -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Arrs_Stack/radarr_cleanup.sh --dry-run --log -``` - -**What it does:** -Same logic for movie files. Walks HOST*_RADARR_MOVIES_ROOT. -Protected: *.jpg *.png (movie artwork), *.nfo, subtitles. -Host-aware: HOST1 manages Movies, HOST2 manages Anime_Movies. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🎬 TRANSCODES -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -> **ramdisk_setup.sh is called by array_started.sh — not scheduled separately.** -> **transcode_manager.sh and transcode_cleanup.sh are called by transcode_management.sh.** -> These entries exist for manual status checks and emergency use. - ---- - -### ── ramdisk_status ──────────────────────────────────────────────────────────── - -``` -Category: Transcodes -Name: ramdisk_status -Schedule: None — run manually -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Transcodes/ramdisk_setup.sh --status -``` - -**What it shows:** -Whether ramdisk is mounted at RAMDISK_PATH. Current symlink target (ramdisk or SSD). -Whether transcoding-temp exists on ramdisk. SSD fallback directory state. -Current usage on both ramdisk and SSD. - ---- - -### ── transcode_manager_manual ──────────────────────────────────────────────── - -``` -Category: Transcodes -Name: transcode_manager_manual -Schedule: None — run manually (runs via transcode_management.sh every 7 min) -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Transcodes/transcode_manager.sh --status -``` - -**What it shows:** -Current mode (smart/ramdisk/ssd). Ramdisk usage vs thresholds. Current symlink target. -Active sessions per configured TRANSCODE_SERVERS. Daily log stats (peak this week, -flip count, session split ramdisk vs SSD). - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🖥️ UNRAID_ESSENTIALS -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -> **stability_watchdog.sh is called by watchdog_orchestrator.sh — not scheduled separately.** -> **webgui_watchdog.sh, system_tuning_monitor.sh, clear_logs.sh are scheduled below.** -> **inotify_tuning.sh, php_fpm_max_children.sh, docker_syslog_filter.sh run via array_started.sh.** - ---- - -### ── webgui_watchdog ─────────────────────────────────────────────────────────── - -``` -Category: System_Essentials -Name: webgui_watchdog -Schedule: */10 * * * * (every 10 minutes) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Watchdogs/System/webgui_watchdog.sh -``` - -**What it does:** -Checks WebGUI responsiveness and recovers it automatically. Silent when healthy — -runs 144 times per day with no output when the WebGUI is responding normally. - -**Escalation path (each step only runs if previous failed):** -``` -1. curl http://localhost → unresponsive for WEBGUI_TIMEOUT (5s)? -2. Restart nginx → wait 15s → recheck -3. Restart php-fpm → wait 10s → recheck (handles worker saturation) -4. Restart emhttp (core daemon) → wait 30s → recheck (most disruptive, last resort) -5. All failed → notify warning — manual investigation needed -``` - -A notification fires on any restart so you know what recovered and when. - ---- - -### ── clear_logs ─────────────────────────────────────────────────────────────── - -``` -Category: System_Essentials -Name: clear_logs -Schedule: 0 5 * * 0 (Sunday 5am — before Sunday maintenance window) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/System_Essentials/clear_logs.sh -``` - -**What it does:** -Clears system and Docker container logs using size thresholds — only logs large -enough to be worth clearing. - -**System logs** (/var/log/syslog, messages, dmesg): cleared only if > LOG_MIN_SIZE_MB (10MB). -Smaller logs contain useful recent history — not cleared. - -**Docker container logs**: cleared per-container only if > LOG_DOCKER_MAX_MB (100MB). -Active containers like Emby and SABnzbd grow fastest. Truncation not rotation — -`> file` keeps the file descriptor valid while emptying. Safe for running services. - ---- - -### ── stability_watchdog_status ────────────────────────────────────────────────── - -``` -Category: System_Essentials -Name: stability_watchdog_status -Schedule: None — run manually -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Watchdogs/stability_watchdog.sh --status -``` - -**What it shows:** -All three tiers with current thresholds. Active strike counts per check. RAM tier -status. Whether mem_shutdown_active is set (RAM emergency in progress). Recent -watchdog-triggered reboots. All 18 per-host toggle states. - ---- - -### ── mover_stop ─────────────────────────────────────────────────────────────── - -``` -Category: System_Essentials -Name: mover_stop -Schedule: None — run manually (before array operations needing mover stopped) -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/System_Essentials/mover_stop.sh -``` - -**What it does:** -Stops the unRAID mover cleanly. Wall message warning to all logged-in users. Waits -MOVER_STOP_TIMEOUT seconds. SIGTERM (allows mover to finish current file — no partial -files). Waits + verifies. SIGKILL if still running (warns that partial files are possible). - ---- - -### ── rsync_stop ──────────────────────────────────────────────────────────────── - -``` -Category: System_Essentials -Name: rsync_stop -Schedule: None — run manually (when rsync needs stopping) -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/System_Essentials/rsync_stop.sh --status -``` - -**What it does:** -Stops rsync intelligently. Auto-detects if an orchestrator is running: -Smart mode (default): kills rsync subprocess only — orchestrator sees rsync died and -exits cleanly. No orphaned lock files. Containers recover via docker_watchdog.sh. -`--full-stop`: kills orchestrator first then rsync — use for emergency stop. - ---- - -### ── user_scripts_stop ──────────────────────────────────────────────────────── - -``` -Category: System_Essentials -Name: user_scripts_stop -Schedule: None — run manually (before reboots, or when a script hangs) -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/System_Essentials/user_scripts_stop.sh --status -``` - -**What it does:** -Finds all running User Script processes by /tmp/user.scripts path signature. Shows -script names (not just PIDs — you know what you're stopping). SIGTERM → verify → -SIGKILL. Self-exclusion prevents the script from killing itself. Called automatically -by server_reboot.sh before reboot. - ---- - -### ── server_reboot ──────────────────────────────────────────────────────────── - -``` -Category: System_Essentials -Name: server_reboot -Schedule: None — run manually (planned maintenance reboots) -Background: YES — sequence takes 1-2 minutes -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/System_Essentials/server_reboot.sh --dry-run -``` - -**What it does:** -Graceful reboot with full pre-flight and clean shutdown. Wall message + unRAID -notification with MY_ID. Waits REBOOT_SLEEP (30s). virsh shutdown each running VM -then waits REBOOT_VM_WAIT for graceful exit. Stops Docker. Stops libvirt. sync. -/sbin/reboot. - -Pre-flight warnings (not blocks): rsync running, mover running, active Emby sessions. -These show in the summary — you chose to reboot, these are context. - -> Remove `--dry-run` to execute. Dry run walks the full sequence without rebooting — -> use it to verify what processes are active before committing. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🤝 PARTNERSHIP -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Manages the two-server auth-sharing relationship lifecycle. Automated check runs -inside critical_sync_maintenance.sh. These entries are for manual operations only. - ---- - -### ── partnership_status ──────────────────────────────────────────────────────── - -``` -Category: Partnership -Name: partnership_status -Schedule: None — run manually -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Partnership/partnership_manage.sh --status -``` - -**What it shows:** -Both server state files (ACTIVE/INACTIVE). Offline counter (days since last successful -sync). Which server is the auth stack owner. Current WebUI redirect state on HOST2. -Whether CRITICAL_RSYNC_ENABLED is true. - ---- - -### ── partnership_onboard ────────────────────────────────────────────────────── - -``` -Category: Partnership -Name: partnership_onboard -Schedule: None — run manually (HOST1/owner only, once at relationship start) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Partnership/partnership_manage.sh --onboard --dry-run -``` - -**What it does:** -Establishes the auth-sharing partnership. Reconfigures HOST2 Docker template WebUI -URLs to redirect to HOST1 via Tailscale. Verifies routing works. Writes ACTIVE state -on both servers. Enables Critical-Data sync. - -> Remove `--dry-run` to run live. Dry run shows exactly which templates would be -> changed and what URLs they'd become — review before committing. - ---- - -### ── partnership_offboard ───────────────────────────────────────────────────── - -``` -Category: Partnership -Name: partnership_offboard -Schedule: None — run manually (either server, when ending the partnership) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Partnership/partnership_manage.sh --offboard --dry-run -``` - -**What it does:** -Clean partnership separation. Stops rsync. Final sync to HOST2. Reconfigures HOST2 -WebUI URLs back to localhost (HOST2 manages its own auth). Disables critical sync. -Grace period (PARTNERSHIP_GRACE_HOURS). Removes HOST2 from Tailscale tailnet. - -Both servers can initiate. If HOST1 is unreachable when HOST2 offboards, deferred -finalisation completes automatically when HOST1 can next reach HOST2. - -> Remove `--dry-run` to run live. 10-second countdown shown before any changes. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🔧 TOOLS -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Situational utilities — run when something needs fixing, not on a schedule. - ---- - -### ── bulk_permissions_repair ───────────────────────────────────────────────── - -``` -Category: Tools -Name: bulk_permissions_repair -Schedule: None — run manually -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Tools/bulk_permissions_repair.sh /mnt/user/Movies --dry-run -``` - -**What it does:** -Applies nobody:users 755/664 to a specific path immediately — without waiting for -the nightly media_shares_permissions.sh run. Use when an admin copy, scp, or bad -container left root-owned files that are blocking arr operations right now. -Replace /mnt/user/Movies with the actual path that needs fixing. - ---- - -### ── container_data_export ──────────────────────────────────────────────────── - -``` -Category: Tools -Name: container_data_export -Schedule: None — run manually (before risky updates or migrations) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Tools/container_data_export.sh \ - Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/ -``` - -**What it does:** -Stops the container cleanly. Archives the entire appdata directory to a timestamped -.tar.gz. Verifies archive integrity. Restarts the container (always, even if archiving -failed). Space check before stopping — won't start if output drive is too small. - -Replace ContainerName, /path/to/appdata, and /output/dir for each use. -Output filename: ContainerName_YYYY-MM-DD_HH-MM.tar.gz - ---- - -### ── emby_database_repair ───────────────────────────────────────────────────── - -``` -Category: Tools -Name: emby_database_repair -Schedule: None — run manually (after power loss or repeated Emby crashes) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Tools/emby_database_repair.sh -``` - -**What it does:** -Stops Emby. Runs SQLite `PRAGMA integrity_check` on every Emby database. Restarts -Emby. Reports per-database with specific recovery guidance. - -**Per-database recovery guidance:** -``` -library.db — CORRUPT → safe to delete, Emby fully rebuilds from media files -users.db — CORRUPT → deleting resets ALL user accounts + watch history - check for recent backup (weekly_sync_maintenance.sh) -authentication.db — CORRUPT → safe to delete, API keys regenerated on restart -activity.db — CORRUPT → safe to delete, it's a log only -``` - -**This script checks and reports only** — it does not automatically delete anything. -Recovery is a human decision based on whether you have backups. - ---- - -### ── zfs_pool_scrub ──────────────────────────────────────────────────────────── - -``` -Category: Tools -Name: zfs_pool_scrub -Schedule: None — run manually (monthly or after any disk/power event) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Tools/zfs_pool_scrub.sh -``` - -**What it does:** -Triggers ZFS scrub on all pools (except ZFS_REPORT_IGNORE_POOLS), waits for completion, -notifies with results. ZFS checksums every block and verifies — catches silent data -corruption before you discover it on reads. Safe to run while system is active. -For a specific pool: add the pool name as argument (bypasses ignore list). - ---- - -### ── recreate_shares ────────────────────────────────────────────────────────── - -``` -Category: Tools -Name: recreate_shares -Schedule: None — run manually (after fresh install or disk replacement) -Background: YES -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/Tools/recreate_shares.sh --dry-run -``` - -**What it does:** -Reads every .cfg in /boot/config/shares/, parses the shareInclude disk list, and -creates /mnt/diskN/sharename on each included disk. Run after restoring share .cfg -files to a fresh install — shares appear in the UI but directories are missing until -this creates them. chown nobody:users on creation. - ---- - -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🔀 GIT -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - ---- - -### ── git_pull ───────────────────────────────────────────────────────────────── - -``` -Category: Git -Name: git_pull -Schedule: None — run manually (called automatically by daily_sync_maintenance.sh) -Background: NO -``` - -```bash -#!/bin/bash -/boot/config/plugins/varaverk/System_Essentials/git_pull_execute.sh -``` - -**What it does:** -Pulls latest commits from Gitea. Sets execute permission on all .sh files. This is -the deployment mechanism for the ecosystem — push changes from VS Code to Gitea, -then run this on both servers to deploy. daily_sync_maintenance.sh runs it -automatically at 1am as the first job in the window. - ---- - -## ━━━ AT STARTUP OF ARRAY — ORDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -**Only one entry needed at startup:** `array_start`. It handles everything. - -`array_started.sh` is the single "At Startup of Array" entry. It reads ARRAY_START_SCRIPTS -from master.conf and launches each in order. No other startup entries. No dragging to -reorder individual scripts. Change the order by editing master.conf. - -``` -Correct startup setup: - 1 entry: array_start — At Startup of Array, Background: YES - -Wrong startup setup (old approach): - ramdisk_setup — At Startup of Array - docker_syslog_filter — At Startup of Array - stability_watchdog — Every 15 Minutes via watchdog_orchestrator - docker_watchdog — At Startup of Array - fallback — At Startup of Array - (unpredictable order, no coordination) -``` - ---- - -## ━━━ COMPLETE SCHEDULE REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -``` -At Startup of Array: - array_start BG: YES — single entry, handles everything - -*/7 * * * * (every 7 min): - transcode_management BG: YES — cleanup then manager - -*/30 * * * * (every 30 min): - critical_sync_maintenance BG: YES — auth stack + Emby dirty sync + partnership - -0 */6 * * * (every 6 hours): - arrs_failed_stalled_recovery BG: YES — blocklist + re-search failed imports - system_tuning_monitor BG: YES — inotify + php-fpm snapshot - -0 1 * * * (1am daily): - daily_sync_maintenance BG: YES — full nightly window - -0 5 * * 0 (Sunday 5am): - clear_logs BG: YES — size-threshold log cleanup - -0 6 * * 0 (Sunday 6am): - zfs_memory_snapshot BG: YES — ZFS + memory report - -0 7 * * 0 (Sunday 7am): - sunday_morning_coffee_report BG: YES — full weekly digest - -0 8 * * * (8am daily): - weekly_health_digest BG: YES — profile-controlled notification - -0 7 * * 0 (Sunday 7am): - smart_health BG: YES — SMART attribute check - -0 9 * * 0 (Sunday 9am): - cert_monitor BG: YES — live TLS cert expiry check - -0 10 * * 0 (Sunday 10am): - backup_verify BG: YES — MD5 checksum mirror verification - -0 11 * * 0 (Sunday 11am): - emby_session_report BG: YES — Emby weekly usage stats - bandwidth_monitor_report BG: YES — rsync transfer summary - -30 2 * * 0 (Sunday 2:30am): - weekly_sync_maintenance BG: YES — clean sync + image updates -``` - ---- - -## ━━━ AFTER CREATING ALL ENTRIES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Run each of these once manually before relying on scheduled runs: - -``` -array_start --dry-run verify all startup scripts found and sequenced correctly -transcode_management --status confirm ramdisk mounted and symlink pointing at ramdisk -critical_sync_maintenance verify Critical-Data sync completes to HOST2 -daily_sync_maintenance --dry-run verify remote connectivity and share list -fallback_status confirm NORMAL state on both servers -docker_watchdog_status confirm required containers all running -stability_watchdog_status confirm no threshold alerts on first check -``` \ No newline at end of file diff --git a/README.md b/README.md index 06468e8..c985e51 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ that matters and both know how to run it. > **This is not conventional failover.** There is no primary and standby. There is no > shared storage. There is no cluster manager. There are two fully independent servers, -> living in different locations on different power utilities, each running their own +> living in different locations on different power utilities and isps, each running their own > household's media stack — and a layer of scripts that keeps them synchronised, > coordinated, and mutually covering. The goal is minimal disruption to a media stack, > not enterprise high availability.