Audit the Varaverk plugin conf files against all script references. Goal: find (A) variables referenced in scripts but missing from confs, and (B) variables defined in confs but never referenced anywhere (stale/dead).

## Conf files to read (read all three completely):
- /boot/config/plugins/varaverk/Configurations/master.conf
- /boot/config/plugins/varaverk/Configurations/host1.conf
- /boot/config/plugins/varaverk/Configurations/host2.conf

Build a complete list of every variable NAME defined in each conf (e.g. `HOST1_NPM_URL`, `RSYNC_ENABLED`, etc). Ignore comment lines and blank lines.

## Scripts to scan:
All .sh files under /boot/config/plugins/varaverk/ (recursively), excluding anything under .git/.

Grep for every reference to a conf variable. Conf variables follow these patterns:
- `$VAR_NAME` or `${VAR_NAME}` where VAR_NAME is ALL_CAPS with underscores
- Specifically: anything matching `\$\{?[A-Z][A-Z0-9_]{3,}\}?` that isn't a bash builtin (skip: PATH, HOME, PWD, IFS, BASH, SECONDS, LINENO, PIPESTATUS, FUNCNAME, BASH_SOURCE, EUID, UID, HOSTNAME, RANDOM, OLDPWD, REPLY, OPTIND, OPTARG, PS1, PS2, TERM, SHELL, USER, LOGNAME, LANG, LC_ALL, LC_CTYPE, DISPLAY, EDITOR, VISUAL, PAGER, MANPATH, TMPDIR, TMP, TEMP, SHLVL, COLUMNS, LINES, HISTSIZE, HISTFILE, HISTCONTROL, CDPATH, GLOBIGNORE, TIMEFORMAT, TMOUT, MAILCHECK, MAIL, MAILPATH, PROMPT_COMMAND, IGNOREEOF, POSIXLY_CORRECT, BASHOPTS, SHELLOPTS, COMP_WORDS, COMP_CWORD, COMPREPLY, COMP_LINE, COMP_POINT, COMP_TYPE, COMP_KEY, COMP_WORDBREAKS, DIRSTACK, GROUPS, PIPESTATUS)
- Also skip local script variables that are clearly defined within the script itself (e.g. `local VAR=`, `VAR=` at script scope that aren't conf-sourced)

Also read /boot/config/plugins/varaverk/Configurations/host.conf.template to understand what's expected for new hosts.

## What to produce:

**Section 1: Variables referenced in scripts but NOT in any conf**
List each missing variable, which script(s) reference it, and a short note on what it likely is. Group by conf file where it probably belongs (master / host1 / host2 / both hosts).

**Section 2: Variables defined in confs but NOT referenced in any script**
List each stale variable and which conf it's in. Note if it might be used indirectly (e.g. passed via SSH, used by PHP, etc).

**Section 3: Variables present in host1.conf but missing from host2.conf (or vice versa)**
These should be symmetric — flag any asymmetry.

Be thorough. Read the actual conf files fully, not just headers. This is a correctness audit.


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.
- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch

update-config keybindings-help verify code-review simplify fewer-permission-prompts loop schedule claude-api run init review security-review


I'll conduct a thorough audit of the Varaverk plugin configuration files against script references. Let me start by reading all the configuration files completely, then scanning the scripts.


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= MASTER CONFIGURATION =======================================
4	# ==============================================================================================
5	# Shared configuration for the unRAID script ecosystem.
6	# Contains all settings that apply to every server — thresholds, toggles, profiles, job lists.
7	#
8	# ── HOW THE THREE-FILE SYSTEM WORKS ──────────────────────────────────────────────────────────
9	# Scripts source all three files at startup:
10	#   source master.conf          ← shared config (this file)
11	#   source host1.conf    ← HOST1 credentials, shares, container lists
12	#   source host2.conf    ← HOST2 credentials, shares, container lists
13	#
14	# Sparse checkout (git) ensures each server only pulls its own host*.conf.
15	# HOST2 never sees HOST1 credentials. HOST1 never sees HOST2 credentials.
16	#
17	# What belongs here:      thresholds, toggles, intervals, profiles, job lists
18	# What belongs in HOST*:  hostnames, SSH keys, API keys, passwords, share paths,
19	#                         container names, failover lists, watchdog containers
20	#
21	# ── HOW THIS FILE WORKS ───────────────────────────────────────────────────────────────────────
22	# Every script sources all three conf files and common.sh at startup.
23	# Change a value here and it affects all scripts that use it — no hunting through files.
24	# To disable something: comment it out with # rather than deleting it.
25	# To add a new rsync profile: add a key to each PROFILE_* array in the RSYNC section.
26	# To add or remove orchestrator jobs: edit the arrays in the ORCHESTRATORS section.
27	#
28	# ── INDEX ─────────────────────────────────────────────────────────────────────────────────────
29	#
30	#   Section                         Description
31	#   ───────────────────────────────────────────────────────────────────────────────────────────
32	#   SHARED HOST CONFIGURATION       DATA_DIR and shared connection settings
33	#   PARTNERSHIP                     Mirror relationship lifecycle — onboard/offboard/transfer
34	#   LOGGING                         Enable or disable verbose logging
35	#   NOTIFICATIONS                   unRAID native and Discord webhook settings
36	#   GIT / REPO                      Gitea repository and SSH settings
37	#
38	#   ── ORCHESTRATORS ──────────────────────────────────────────────────────────────────────────
39	#   ARRAY START                     Scripts launched at array start (array_started.sh)
40	#   ARRAY STOP                      Scripts run at planned shutdown (array_stopping.sh)
41	#   WATCHDOG ORCHESTRATOR           Per-minute watchdog runner (watchdog_orchestrator.sh)
42	#   SYSTEM WATCHDOG                 Sub-scripts called by watchdog_orchestrator.sh
43	#   CRITICAL SYNC MAINTENANCE       30-minute jobs + sync shares + partnership check (critical_sync_maintenance.sh)
44	#   INTERMEDIATE SYNC MAINTENANCE   Arr sync + optional mid-day rsync (intermediate_sync_maintenance.sh)
45	#   DAILY SYNC MAINTENANCE          Job list + media shares (daily_sync_maintenance.sh)
46	#   WEEKLY SYNC MAINTENANCE         Job list + sync shares + update toggles (weekly_sync_maintenance.sh)
47	#   MONTHLY MAINTENANCE             Uptime-triggered heavy tasks — ZFS scrub, SMART tests (monthly_maintenance.sh)
48	#
49	#   ── RSYNC ──────────────────────────────────────────────────────────────────────────────────
50	#   RSYNC ENABLE/DISABLE            Two-tier toggle system — global gate + per-orchestrator
51	#   RSYNC DEFAULTS                  Global fallback rsync options and limits
52	#   REMOTE HEALTH CHECKS            Rootfs threshold for pre-flight abort
53	#   RSYNC PROFILE SYSTEM            Per-profile overrides for appdata syncs
54	#
55	#   ── FALLBACK ───────────────────────────────────────────────────────────────────────────────
56	#   FALLBACK                        Mutual container failover shared settings
57	#   FALLBACK TEST                   Simulated outage settings for fallback_test.sh
58	#
59	#   ── DOCKER ESSENTIALS ──────────────────────────────────────────────────────────────────────
60	#   DOWNLOADERS RESET               Retention and thresholds for slskd, SABnzbd, qBittorrent
61	#   DOCKER DAILY RESTART            Containers restarted daily
62	#   DOCKER WEEKLY RESTART           Containers restarted weekly
63	#   DOCKER WATCHDOG                 Thresholds and toggles for container monitoring
64	#   DOCKER NETWORK CONNECT          Networks to ensure + containers to connect
65	#
66	#   ── UNRAID ESSENTIALS ──────────────────────────────────────────────────────────────────────
67	#   INOTIFY TUNING                  inotify limits — raised at array start by inotify_tuning.sh
68	#   SYSTEM TUNING MONITOR           Tracks inotify + php-fpm usage over time
69	#   REBOOT                          User warning delay before scheduled reboot
70	#   MOVER                           Mover stop timeout
71	#   SYSLOG FILTER                   Docker veth noise filter file path
72	#   PHP-FPM                         PHP-FPM max children config
73	#   CLEAR LOGS                      System log file paths
74	#   WEBGUI WATCHDOG                 WebGUI nginx + emhttp monitoring and restart
75	#
76	#   ── MEDIA ──────────────────────────────────────────────────────────────────────────────────
77	#   MEDIA PERMISSIONS               Share list, mode and owner for permissions script
78	#   MEDIA CLEANER                   Anime and media folder lists and file patterns
79	#   ARR CLEANUP                     Lidarr, Sonarr, Radarr orphan file cleanup settings
80	#   ARR FAILED/STALLED RECOVERY     Auto blocklist + re-search settings
81	#
82	#   ── TRANSCODES ─────────────────────────────────────────────────────────────────────────────
83	#   TRANSCODE MANAGER               Ramdisk and SSD fallback transcode management
84	#   TRANSCODE SERVER ARRAY          Multi-server session monitoring (Emby, Jellyfin, Plex)
85	#
86	#   ── MONITORS ───────────────────────────────────────────────────────────────────────────────
87	#   CERTIFICATE MONITOR             SSL certificate expiry monitoring
88	#   BACKUP VERIFY                   Random sample checksum verification against remote
89	#   SMART HEALTH                    Drive SMART attribute monitoring
90	#   ZFS MEMORY SNAPSHOT             Weekly ZFS health and memory diagnostic report
91	#   BANDWIDTH MONITOR               Daily rsync transfer logging and weekly summary
92	#   HEALTH DIGEST                   Aggregated system health digest — always/smart/weekly
93	#   EMBY SESSION REPORT             Weekly Emby usage statistics via API
94	#
95	#   ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
96	#   RESOURCE MANAGER                Containers paused/stopped under RAM pressure
97	#
98	#   ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
99	#   SYSTEM WATCHDOG                 Continuous system health monitoring — last line of defense
100	#
101	# ==============================================================================================
102
103	# ==============================================================================================
104	# ── HOST IDENTITIES ───────────────────────────────────────────────────────────────────────────
105	# ==============================================================================================
106	# Hostnames for every server in the ecosystem — not credentials, safe for all machines.
107	# Must match the exact unRAID hostname AND Tailscale device name (case sensitive).
108	# detect_hosts() in common.sh matches the local hostname against these to set MY_ID / REMOTE_ID.
109	# Tailscale IP resolution uses these names — no hardcoded IPs needed.
110	# To add a new server: add HOST3="unRAID-NewServer" here + create host3.conf.
111	    HOST1="unRAID-Gmer4Lfe"
112	    HOST2="unRAID-Jayred365"
113
114	# ==============================================================================================
115	# ── SHARED HOST CONFIGURATION ─────────────────────────────────────────────────────────────────
116	# ==============================================================================================
117	# Requirement: internal NVMe/SSD boot drive. Everything lives in the plugin folder on flash —
118	# scripts, state, and data are all available before the array mounts.
119	# Both directories are created automatically if they don't exist.
120	#
121	#   DATA_DIR   — historical logs, statistics, discovery histories, blocklists
122	#   STATE_DIR  — runtime state files for all scripts (watchdogs, fallback, transcode, etc.)
123	#                Requirement: ALL state files MUST use $STATE_DIR. No /tmp, no /boot/config root.
124	    DATA_DIR="/boot/config/plugins/varaverk/data"
125	    STATE_DIR="/boot/config/plugins/varaverk/State_Files"
126
127	# ── Version Parity ──
128	# Controls behaviour when local and remote unRAID versions differ.
129	# Major version mismatch always aborts regardless of this setting.
130	# "warn"  — log warning and continue (safe for minor/patch differences)
131	# "abort" — refuse to continue (strict — ensures both sides always match)
132	    UNRAID_VERSION_MISMATCH_ACTION="warn"
133
134	# ── Arr Sync ──
135	# arr_sync.sh syncs Lidarr/Sonarr/Radarr libraries across all nodes bidirectionally.
136	# Runs before rsync — all nodes agree on tracked library before files are transferred.
137	# Remote API keys are read live from each node's config.xml via SSH — never stored here.
138	    CONF_SYNC_ENABLED=false             # HOST2 rebuild in progress — re-enable when HOST2 is back online
139
140	    ARR_SYNC_ENABLED=true
141	    ARR_SYNC_BLOCKLIST="${DATA_DIR}/arr_sync_blocklist.tsv"
142	    ARR_SYNC_CONNECT_TIMEOUT=10     # seconds — SSH connect timeout per node
143	    ARR_SYNC_API_TIMEOUT=60         # seconds — curl timeout for library fetches
144	    DOCKER_APPDATA_BASE="/mnt/user/appdata"
145	    ARR_SYNC_LIDARR_PORT=8686
146	    ARR_SYNC_SONARR_PORT=8989
147	    ARR_SYNC_RADARR_PORT=7878
148
149	# ── Remote Docker Daemon ──
150	# Strike system for remote Docker daemon health checks.
151	# Scripts that issue remote container commands check the remote daemon first.
152	# Below limit → skip operation this run | At limit → notify critical + exit
153	    REMOTE_DOCKER_STRIKE_LIMIT=3       # consecutive failures before critical notify
154
155	# ==============================================================================================
156	# ── PARTNERSHIP ───────────────────────────────────────────────────────────────────────────────
157	# ==============================================================================================
158	# Manages the relationship lifecycle between two unRAID servers.
159	# HOST1 is always the owner (source of truth) — HOST2 is always the mirror.
160	# PARTNERSHIP_OWNER_HOST flips to "HOST2" after a --transfer operation.
161	# All identity vars (hostnames, SSH keys) live in host*.conf.
162	# Hostnames already match Tailscale device names — IP resolution is automatic.
163	#
164	# State files in STATE_DIR — survives reboots, array must be up (scripts run after array start):
165	#   STATE_DIR/partnership_HOST1.db ← HOST1 writes only
166	#   STATE_DIR/partnership_HOST2.db ← HOST2 writes only
167	#   STATE_DIR/partnership_blocklist.db ← shared blocklist
168	#   STATE_DIR/varaverk_setup.db ← onboarding progress state
169	#   Propagated via SSH — no rsync needed
170	#
171	# critical_sync_maintenance.sh runs --check every 30min:
172	#   Reads both state files via SSH
173	#   Detects offboard requests → finalises from owner side
174	#   Increments offline counter → auto-offboards after threshold
175	#   Silent when healthy ✅
176	#
177	# See README-Partnership.md for full lifecycle documentation.
178
179	    PARTNERSHIP_ENABLED=false
180	    PARTNERSHIP_OWNER_HOST="HOST1"          # "HOST1" or "HOST2" — flips on --transfer
181
182	# Auth containers reconfigured on onboard/offboard — defined per host in host*.conf.
183	# Format: "ContainerName|WebUIPort"
184	# HOST1_PARTNERSHIP_AUTH_WEBUIS / HOST2_PARTNERSHIP_AUTH_WEBUIS
185	# On onboard → WebUI pointed at owner's Tailscale IP
186	# On offboard → WebUI pointed back at localhost
187
188	# Paths to collect during the grace window after offboard — defined per host in host*.conf.
189	# HOST1_PARTNERSHIP_MIRROR_BACKUPS / HOST2_PARTNERSHIP_MIRROR_BACKUPS
190	# Notified on offboard — no auto-deletion, manual collection.
191
192	# Timing — single var controls both Tailscale removal and backup access expiry.
193	# Both expire at the same time — keeping backups accessible beyond Tailscale removal is pointless.
194	    PARTNERSHIP_GRACE_HOURS=6               # hours after offboard before Tailscale removal
195	                                            # backup access expires at the same time
196	    PARTNERSHIP_OFFLINE_THRESHOLD=30        # days either server unreachable before auto-offboard
197	                                            # works both directions independently
198
199	# Partnership and setup state files — all in STATE_DIR per the project requirement.
200	# Scripts use these variables; do not hardcode /boot/config paths in scripts.
201	    PARTNERSHIP_BLOCKLIST_FILE="$STATE_DIR/partnership_blocklist.db"
202	    VARAVERK_SETUP_FILE="$STATE_DIR/varaverk_setup.db"
203
204	# Tailscale removal on offboard.
205	    PARTNERSHIP_REMOVE_TAILSCALE=true       # remove mirror from Tailscale tailnet on offboard
206	                                            # false = skip removal (manual or testing)
207
208	# Tailscale API — required when PARTNERSHIP_REMOVE_TAILSCALE=true.
209	# Stays in shared conf — only owner uses it, and owner is always running this script.
210	# API key: https://login.tailscale.com/admin/settings/keys → Devices write scope
211	    TAILSCALE_API_KEY=""                    # tskey-api-...
212	    TAILSCALE_TAILNET=""                    # your tailnet name (e.g. yourname.github)
213
214	# Transfer safety.
215	    PARTNERSHIP_TRANSFER_CONFIRM="i-understand-this-transfers-ownership"
216	    PARTNERSHIP_TRANSFER_STRIKES=3          # consecutive health checks required
217	    PARTNERSHIP_TRANSFER_MAX_ATTEMPTS=20    # max health check attempts before giving up
218
219	# Onboard settings.
220	    PARTNERSHIP_ONBOARD_VERIFY=true         # verify WebUI reachable after reconfiguration
221	    PARTNERSHIP_ONBOARD_NOTIFY=true         # notify both servers on completion
222	    PARTNERSHIP_SYNC_INTERVAL=15            # minutes — informational, actual schedule in cron
223
224	# SSH key lifecycle — managed by Partnership/ssh_setup.sh.
225	# Key named after this server: hostname lowercased, unraid- prefix stripped.
226	#   unRAID-Gmer4Lfe → /root/.ssh/gmer4lfe_rsync_automation
227	# Run Partnership/partnership_onboard.sh to generate, copy, and update conf automatically.
228	    SSH_MAX_STRIKES=5                       # consecutive SSH auth failures before critical notify
229	    SSH_STRIKE_RESET_HRS=24                 # hours since last failure before strike counter resets
230
231	# FolderView3 plugin integration — chodeus/VladoPortos/scolcipitato.
232	# Creates a "{Partner}-Failover" folder on onboard, removes + cleans containers on offboard.
233	# Folder name derived from remote hostname (strip unraid- prefix case-insensitively).
234	# Plugin config: /boot/config/plugins/folder.view3/docker.json
235	    PARTNERSHIP_FOLDERVIEW3=true            # create/remove FolderView3 folder on onboard/offboard
236	    PARTNERSHIP_FOLDERVIEW3_URL=""          # CA plugin URL — leave empty to skip auto-install
237
238	# ==============================================================================================
239	# ── LOGGING ───────────────────────────────────────────────────────────────────────────────────
240	# ==============================================================================================
241
242	# Controls verbose [LOG] output across all scripts.
243	# true  = show detailed [LOG] lines — useful for debugging or first-time setup
244	# false = show only user-facing output — cleaner for scheduled runs
245	    ENABLE_LOGGING=false
246
247	# ==============================================================================================
248	# ── NOTIFICATIONS ─────────────────────────────────────────────────────────────────────────────
249	# ==============================================================================================
250
251	# unRAID native notification system — integrates with the bell icon in the WebGUI.
252	# normal = job completed successfully / warning = something failed or needs attention
253	    NOTIFY_UNRAID=true
254
255	# Discord webhook URL — defined per host in host*.conf.
256	# HOST1_DISCORD_WEBHOOK / HOST2_DISCORD_WEBHOOK
257	# Allows different webhooks per server, or only one server notifying.
258
259	# ==============================================================================================
260	# ── GIT / REPO ────────────────────────────────────────────────────────────────────────────────
261	# ==============================================================================================
262
263	# Gitea self-hosted repository — used by git_pull_execute.sh.
264	# Detects Gitea container location at runtime — works through failover automatically.
265	# Falls back to GITEA_DOMAIN if local and Tailscale both fail.
266	    GITEA_CONTAINER="Gitea"
267	    GITEA_REPO_PATH="FailedProxy/Varaverk.git"
268	    GITEA_DOMAIN=""                         # e.g. git.yourdomain.com — requires NPM + DNS
269	    TARGET_DIR="/boot/config/plugins/varaverk"
270	    GITEA_SSH_KEY="/root/.ssh/unraid_gitea"
271	    SSH_PORT=221                            # Gitea SSH port (default 22, Gitea often uses 221/222)
272	    GITEA_HTTP_PORT=3000                    # Gitea web/API port — used by gitea_ssh_setup.sh
273
274	# ==============================================================================================
275	# ── ORCHESTRATORS ─────────────────────────────────────────────────────────────────────────────
276	# ==============================================================================================
277	# All orchestrator job lists live here — edit arrays to add/remove scripts.
278	# No changes to orchestrator scripts needed when adding or removing jobs.
279	# Sections ordered by run frequency: array events first, then shortest interval to longest.
280
281	# ━━━ Array Start ━━━
282	# Scripts launched by array_started.sh when the array comes online.
283	# Launched in order — each as a background process.
284	# One-shot scripts (ramdisk, syslog, fpm, inotify, network) run and exit naturally.
285	# Continuous scripts (failover) run until array stops.
286	# Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via
287	# watchdog_orchestrator.sh — NOT launched here.
288	    ARRAY_START_SCRIPTS=(
289	        "Plugin/unraid/System_Essentials/unraid_api_key_renew.sh"   # re-register Varaverk API key — registry is ephemeral
290	        "System_Essentials/conf_sync.sh"              # pull partner confs + push own conf into /tmp/.vv/ RAM cache
291	        "Transcodes/ramdisk_setup.sh"                 # creates ramdisk + symlink before Emby starts
292	        "System_Essentials/docker_syslog_filter.sh"   # suppress veth noise before logs fill
293	        "Plugin/unraid/System_Essentials/php_fpm_max_children.sh"   # WebGUI performance tuning
294	        "System_Essentials/inotify_tuning.sh"         # bump inotify limits — containers miss events if exhausted
295	        "Docker_Essentials/docker_network_connect.sh" # ensure networks exist + connect containers
296	        "Tools/claude_startup.sh"                     # persist Claude data + binary to appdata; re-symlink on boot
297	        "Fallback/fallback.sh"                        # mutual failover — continuous
298	    )
299
300	# ━━━ Array Stop ━━━
301	# Scripts run by array_stopping.sh for a planned shutdown — stops everything cleanly in order.
302	# Run sequentially (foreground) — each must complete before the next starts.
303	# Order matters: user scripts first (prevents new ops), then data movement, then containers.
304	    ARRAY_STOP_SCRIPTS=(
305	        "Plugin/unraid/System_Essentials/user_scripts_stop.sh"        # stop background scripts before they start new ops
306	        "Fallback/fallback.sh --stop"                   # gracefully stop fallback (not caught by user_scripts_stop)
307	        "System_Essentials/rsync_stop.sh --rsync-only"  # kill rsync; skip container recovery (handled below)
308	        "Plugin/unraid/System_Essentials/mover_stop.sh"               # stop mover after rsync (they conflict on same files)
309	        "Docker_Essentials/docker_container_stop.sh"    # stop all containers last
310	    )
311
312	# ━━━ Watchdog Orchestrator ━━━
313	# watchdog_orchestrator.sh runs WATCHDOG_ORCHESTRATOR_SCRIPTS in order each cron cycle.
314	# Schedule: * * * * * (every minute)
315	# NOT in ARRAY_START_SCRIPTS — has its own cron entry.
316	# Order matters — resource first (frees pressure), docker second (heals with freed resources),
317	# system third (storage + webgui component health), stability last (last line of defense).
318	    WATCHDOG_ORCHESTRATOR_SCRIPTS=(
319	        "Watchdogs/resource_watchdog.sh"              # reduce system pressure before healing attempts
320	        "Watchdogs/docker_watchdog.sh"                # heal containers with freed resources
321	        "Watchdogs/system_watchdog.sh"                # system component health — storage + webgui
322	        "Plugin/unraid/System_Essentials/unraid_api_key_renew.sh"   # re-register API key if registry lost it
323	        "Watchdogs/stability_watchdog.sh"             # reboot if all else fails — last line of defense
324	    )
325
326	# ━━━ System Watchdog ━━━
327	# system_watchdog.sh runs SYSTEM_WATCHDOG_SCRIPTS sequentially each cycle.
328	# Called by watchdog_orchestrator.sh — not scheduled directly.
329	    SYSTEM_WATCHDOG_SCRIPTS=(
330	        "Watchdogs/System/storage_watchdog.sh"   # pool growth + runaway log detection
331	        "Plugin/unraid/Watchdogs/System/webgui_watchdog.sh"    # WebGUI availability — nginx → php-fpm → emhttp
332	        "Watchdogs/System/network_watchdog.sh"   # internet, DDNS, Tailscale, NPM proxy
333	    )
334
335	# ━━━ Critical Sync Maintenance ━━━
336	# critical_sync_maintenance.sh runs every 30 minutes.
337	# Order: CRITICAL_MAINTENANCE_SCRIPTS (jobs) → CRITICAL_SYNC_SHARES (rsync) → partnership --check
338	# partnership --check always runs last regardless of rsync gate.
339	# Comment out entries to disable without removing.
340	    CRITICAL_MAINTENANCE_SCRIPTS=(
341	        "Docker_Essentials/downloaders_reset.sh"        # clear stuck download states every 30min
342	        "Media/play_state_sync.sh"                      # sync watched/resume state across Emby + Jellyfin
343	    )
344
345	# Shares synced every 30 minutes — defined per host in host*.conf.
346	# HOST1_CRITICAL_SYNC_SHARES / HOST2_CRITICAL_SYNC_SHARES
347	# Format: "/path/to/share" or "/path/to/share|profile-name"
348	# Order matters — Critical-Data first (auth stack), then Emby dirty sync.
349
350	# ━━━ Intermediate Sync Maintenance ━━━
351	# intermediate_sync_maintenance.sh runs every 4 hours — arr library sync, artwork fetch,
352	# and optional mid-day rsync for any shares that need sub-daily propagation.
353	# Schedule: 0 */4 * * *
354	    # INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in host*.conf.
355
356	    INTERMEDIATE_MAINTENANCE_SCRIPTS=(
357	        "Media/arrs_failed_stalled_recovery.sh"  # blocklist + re-search failed/stalled arr queue items
358	    )
359	    # arr_sync.sh runs as a fixed first step in intermediate_sync_maintenance.sh — not listed here.
360	    # It is controlled by ARR_SYNC_ENABLED (see Arr Sync section above).
361
362	# ━━━ Daily Sync Maintenance ━━━
363	# daily_sync_maintenance.sh runs media share sync first, then iterates
364	# DAILY_MAINTENANCE_SCRIPTS for all jobs.
365	# Schedule: 0 1 * * * (1am daily)
366	    DAILY_MAINTENANCE_SCRIPTS=(
367	        "git_pull_execute.sh"                         # pull latest scripts — always runs first
368	        "Media/media_shares_permissions.sh"           # apply permissions — runs before cleaners
369	        "Media/media_cleaner.sh anime"                # remove junk from anime shares
370	        "Media/media_cleaner.sh media"                # remove junk from media shares
371	        #"Media/lidarr_cleanup.sh"                    # remove orphaned music files — enable when ready
372	        #"Media/sonarr_cleanup.sh"                    # remove orphaned TV files — enable when ready
373	        "Media/radarr_cleanup.sh"                     # remove orphaned movie files
374	        "Media/lidarr_missing_art.sh"                 # fetch missing album/artist artwork (HOST1 only — self-guards)
375	        "Media/radarr_tmdb_removed.sh"                # remove movies dropped from TMDb
376	        "Media/sonarr_tvdb_removed.sh"                # remove series dropped from TVDB
377	        "Docker_Essentials/docker_update.sh"          # pull container image updates before restart
378	        "Docker_Essentials/docker_daily_restart.sh"   # daily container restarts — runs last
379	    )
380
381	# Pull latest images for DAILY_RESTART_CONTAINERS before the daily restart.
382	# Containers keep running during pull — no extra downtime.
383	# Set false to skip updates while still running the daily restart.
384	    DAILY_CONTAINER_UPDATES=true
385
386	# Media shares synced daily by daily_sync_maintenance.sh.
387	# Defined per-host in host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
388	# Mesh model: every node pushes every media share. rsync has no --delete so pushes are additive.
389	# arr_sync (union) ensures all arr libraries converge first. arr_cleanup removes true orphans.
390	# Any node can download content to any share — it propagates to all nodes on the next cycle.
391	# Adding HOST3: list every media share in HOST3_DAILY_SYNC_SHARES. No ownership to track.
392	# These shares use DEFAULT_RSYNC_OPTS — no profile entry needed.
393	# For shares needing custom options or container stops — create a profile in RSYNC section.
394
395	# Personal encrypted shares defined per-host in host*.conf.
396	# ZFS encrypted at dataset level — remote receives encrypted blocks, cannot read content.
397	# See README-Rsync_Setup.md for ZFS encryption setup before uncommenting.
398
399	# ━━━ Weekly Sync Maintenance ━━━
400	# weekly_sync_maintenance.sh stops containers both sides → pulls updates →
401	# syncs WEEKLY_SYNC_SHARES → restarts → then iterates WEEKLY_MAINTENANCE_SCRIPTS.
402	# Schedule: 30 2 * * 0 (Sunday 2:30am)
403	    WEEKLY_MAINTENANCE_SCRIPTS=(
404	        "Docker_Essentials/docker_weekly_restart.sh"    # weekly container restarts after sync
405	        "Docker_Essentials/docker_update_remaining.sh"  # pull updates for all other containers
406	        "System_Essentials/clear_logs.sh"               # purge aged logs — Sunday only, low priority
407	        "Media/playback_aware_lidarr_discovery.sh"      # behavior-driven music discovery using weekly Emby playback history
408	        "Media/playback_aware_radarr_discovery.sh"      # behavior-driven movie discovery using TMDB recommendations
409	        "Media/playback_aware_sonarr_discovery.sh"      # behavior-driven TV discovery using TMDB recommendations
410	    )
411
412	# Pull updates for all running containers NOT in daily/weekly restart lists.
413	# Ensures every deployed container receives at least one image pull per week.
414	# Set false to skip — docker_weekly_restart.sh still runs regardless.
415	    WEEKLY_REMAINING_UPDATES=true
416
417	# Shares synced during the weekly maintenance window — defined per host in host*.conf.
418	# HOST1_WEEKLY_SYNC_SHARES / HOST2_WEEKLY_SYNC_SHARES
419	# Containers stopped both sides before sync — full clean state guaranteed.
420	# Profiles drive container stops, excludes, and options — configure in RSYNC section.
421	# Order matters — Emby first (larger), then Critical-Data (auth stack).
422
423	# Container update toggles for the weekly sync window.
424	# Containers already stopped for sync — updates pull at no extra downtime.
425	# Both false → sync only, no updates.
426	    WEEKLY_SYNC_UPDATES=true          # pull container updates locally during weekly window
427	    WEEKLY_SYNC_UPDATES_REMOTE=true   # pull container updates on remote via SSH
428
429	# ━━━ Monthly Maintenance ━━━
430	# monthly_maintenance.sh fires only when BOTH gates pass:
431	#   1. Server uptime >= MONTHLY_UPTIME_THRESHOLD_DAYS days
432	#   2. Last run was >= MONTHLY_RUN_INTERVAL_DAYS days ago (or never run)
433	# Cron: 0 3 * * * (daily 3am check — script self-gates, calling daily is safe)
434	# NOT in WATCHDOG_ORCHESTRATOR_SCRIPTS — has its own cron entry.
435	# Add scripts that require a long-stable settled system — scrubs, extended drive tests.
436	# State file on /boot/config — survives reboots (interval gate independent of uptime gate).
437	    MONTHLY_MAINTENANCE_SCRIPTS=(
438	        "Tools/zfs_pool_scrub.sh"           # ZFS pool integrity scrub — all pools, skip ZFS_REPORT_IGNORE_POOLS
439	        "Tools/smart_long_test.sh"          # SMART extended self-test — all drives, sequential
440	    )
441
442	    MONTHLY_UPTIME_THRESHOLD_DAYS=30    # minimum uptime in days before maintenance fires
443	    MONTHLY_RUN_INTERVAL_DAYS=30        # minimum days since last run before running again
444	    MONTHLY_LAST_RUN_FILE="$STATE_DIR/monthly_maintenance_last_run.db"
445
446	# ━━━ Sunday Morning Coffee Report ━━━
447	# Orchestrator that runs all Sunday monitor scripts in sequence.
448	# Schedule: 0 7 * * 0 (Sunday 7am — after weekly_sync_maintenance.sh finishes at ~3am)
449	# Each script runs independently and notifies on its own findings.
450	    COFFEE_REPORT_SCRIPTS=(
451	        "Monitors/zfs_memory_snapshot.sh"       # ZFS pool health + ARC + Docker memory snapshot
452	        "Monitors/smart_health.sh"              # drive SMART attributes — reallocated, pending, temp
453	        "Monitors/cert_monitor.sh"              # SSL certificate expiry for all configured domains
454	        "Monitors/backup_verify.sh"             # rsync mirror integrity via independent MD5 checksums
455	        "Monitors/bandwidth_monitor.sh"         # weekly rsync transfer totals and per-share breakdown
456	        "Monitors/emby_session_report.sh"       # Emby usage — streams, users, library, transcode ratio
457	        "Monitors/weekly_health_digest.sh"      # aggregated digest — watchdogs, fallback, skip list
458	    )
459
460	# ==============================================================================================
461	# ── RSYNC ─────────────────────────────────────────────────────────────────────────────────────
462	# ==============================================================================================
463
464	# ━━━ Rsync Enable/Disable ━━━
465	# Two-tier toggle system — Tier 1 overrides Tier 2.
466	#
467	# Tier 1 — Global gate:
468	#   RSYNC_ENABLED=false → ALL rsync stops everywhere, no exceptions
469	#   Use when: remote completely offline, major maintenance, disaster recovery
470	#
471	# Tier 2 — Per-orchestrator (only applies when Tier 1 is true):
472	#   Fine grained control — disable specific orchestrators while keeping others
473	#   Use when: rebuilding secondary, testing, per-window bandwidth management
474	#
475	# Example — HOST2 data rebuild:
476	#   RSYNC_ENABLED=true                ← rsync works, individual scripts run fine
477	#   CRITICAL_RSYNC_ENABLED=true       ← 30min auth stack sync still runs
478	#   INTERMEDIATE_RSYNC_ENABLED=false  ← skip 4h arr/mid-day rsync during rebuild
479	#   DAILY_RSYNC_ENABLED=false         ← skip daily HDD syncs during rebuild
480	#   WEEKLY_RSYNC_ENABLED=true         ← Emby + Critical-Data still sync (NVMe)
481	#   FALLBACK_RSYNC_ENABLED=true       ← handback writeback still works
482	#   → Run individual: bash Rsync/rsync.sh /mnt/user/Movies
483	#   → When ready: INTERMEDIATE_RSYNC_ENABLED=true  DAILY_RSYNC_ENABLED=true
484	    RSYNC_ENABLED=true                  # Tier 1 — global gate, overrides everything below
485	    CRITICAL_RSYNC_ENABLED=true         # Tier 2 — critical_sync_maintenance.sh rsync section
486	    INTERMEDIATE_RSYNC_ENABLED=true     # Tier 2 — intermediate_sync_maintenance.sh rsync section
487	    DAILY_RSYNC_ENABLED=true            # Tier 2 — daily_sync_maintenance.sh rsync section
488	    WEEKLY_RSYNC_ENABLED=true           # Tier 2 — weekly_sync_maintenance.sh rsync section
489	    FALLBACK_RSYNC_ENABLED=true         # Tier 2 — fallback.sh writeback jobs on handback
490
491	# ━━━ Rsync Defaults ━━━
492	# Global fallback values used when no profile match is found.
493	# Media shares in HOST*_DAILY_SYNC_SHARES always use these globals — no profile needed.
494	# Appdata shares match profiles by directory basename (lowercased).
495	    BW_LIMIT=12500                  # KB/s — 12500 ≈ 100Mbit
496	    RETRY_COUNT=3                   # retry attempts before giving up
497	    SLEEP=300                       # seconds between retry attempts
498	    CRITICAL_CONTAINER_NAMES=()     # containers stopped on REMOTE before rsync — profiles override
499	    DELAYED_CONTAINERS=()           # containers needing delay before starting — profiles override
500	    CONTAINER_DELAY=5               # seconds before starting delayed containers
501	    EXCLUDE_DIRS=()                 # directories excluded from transfer — profiles override
502
503	# --inplace      writes directly to destination — delta against existing file, better for large media
504	# --partial      keep partial file on interrupted transfer so next run resumes, not re-transfers
505	# --timeout      kill stalled transfers instead of hanging indefinitely
506	# --numeric-ids  use UIDs/GIDs numerically — prevents ownership mismatches between servers
507	# --delete intentionally omitted — arr_cleanup.sh enforces media truth post-rsync.
508	# Media shares use this default; rsync spreads files only, never removes them.
509	# Note: --no-whole-file removed — redundant over SSH (delta transfer is already the default).
510	# Note: arr_cleanup only catches true orphans under the union model — intentional removals
511	#       require arr_sync.sh --blocklist-add first, then manual file deletion.
512	    DEFAULT_RSYNC_OPTS=(-av --info=progress2 --human-readable --bwlimit="$BW_LIMIT" --inplace --partial --timeout=60 --numeric-ids)
513
514	# ━━━ Remote Health Checks ━━━
515	# Pre-flight — aborts if remote rootfs (/) usage is at or above this percentage.
516	# When remote array is down, rsync writes land on rootfs and fill it rapidly.
517	    ROOTFS_WARN=75
518
519	# ━━━ Rsync Profile System ━━━
520	# Profiles allow per-share rsync behaviour without touching script logic.
521	# Profile key matched by basename of directory passed to rsync.sh (lowercased).
522	# Override with --profile=name flag.
523	#
524	# IMPORTANT: PROFILE_RSYNC_OPTS does NOT inherit DEFAULT_RSYNC_OPTS.
525	#            List ALL desired options explicitly when defining a profile.
526	#
527	# Current profiles:
528	#   arrs_stack        — arr databases — lower bandwidth, containers stopped for consistency
529	#   critical-data     — auth stack — full stop both sides, Authelia delayed start
530	#                       called by weekly_sync_maintenance.sh — full clean sync weekly
531	#   critical-fallback — dirty sync — auth stays running both sides, WAL excluded
532	#                       called by critical_sync_maintenance.sh every 30min
533	#   host1-appdata     — HOST1 server-specific appdata — defined in host1.conf
534	#   host2-appdata     — HOST2 server-specific appdata — defined in host2.conf
535	#   important-data    — NextCloud + Postgres — NextCloud delayed start after Postgres
536	#   emby              — weekly clean sync — both Emby stopped, full mirror
537	#                       called by weekly_sync_maintenance.sh only — do NOT schedule separately
538	#   emby-fallback     — dirty sync — Emby stays running, WAL excluded
539	#                       called by critical_sync_maintenance.sh every 30min
540
541	    declare -A PROFILE_RSYNC_OPTS=(
542	        [arrs_stack]="-av --info=progress2 --human-readable --bwlimit=$BW_LIMIT --delete --inplace"
543	        [critical-data]="-av --human-readable --bwlimit=$BW_LIMIT --delete"
544	        [critical-fallback]="-av --human-readable --bwlimit=$BW_LIMIT --delete --inplace --no-whole-file"
545	        [important-data]="-av --human-readable --bwlimit=$BW_LIMIT"
546	        [emby]="-av --human-readable --bwlimit=$BW_LIMIT --delete --inplace --no-whole-file"
547	        [emby-fallback]="-av --human-readable --bwlimit=$BW_LIMIT --delete --inplace --no-whole-file"
548	    )
549
550	# Per-profile bandwidth limits in KB/s
551	    declare -A PROFILE_BW_LIMIT=(
552	        [arrs_stack]=5000           # lower — runs alongside other syncs
553	        [critical-data]=9500        # high — small dataset, sync fast
554	        [critical-fallback]=9500    # high — small dataset, sync fast
555	        [important-data]=9500       # high — database sync
556	        [emby]=8000                 # medium — large full mirror
557	        [emby-fallback]=9500        # high — small critical dataset
558	    )
559
560	# Retry attempts per profile
561	    declare -A PROFILE_RETRY_COUNT=(
562	        [arrs_stack]=3
563	        [critical-data]=3
564	        [critical-fallback]=3
565	        [important-data]=3
566	        [emby]=3
567	        [emby-fallback]=3
568	    )
569
570	# Seconds between retry attempts
571	    declare -A PROFILE_SLEEP=(
572	        [arrs_stack]=300
573	        [critical-data]=300
574	        [critical-fallback]=120     # shorter — frequent dirty sync, retry faster
575	        [important-data]=300
576	        [emby]=300
577	        [emby-fallback]=120         # shorter — frequent dirty sync, retry faster
578	    )
579
580	# Containers stopped on BOTH LOCAL and REMOTE before rsync.
581	# Local stops first — flushes databases cleanly. Remote stops next — prevents writes.
582	# Only running containers get restarted — stopped containers stay stopped.
583	# SPACE-SEPARATED STRINGS — converted to array at runtime
584	    declare -A PROFILE_CRITICAL_CONTAINER_NAMES=(
585	        [arrs_stack]="Sonarr Lidarr Readarr Radarr Prowlarr Bazarr Pinchflat"
586	        [critical-data]="Mariadb-Authelia Mariadb-Authelia-Secondary Redis-Authelia Redis-Authelia-Secondary Lldap-Gmer4Lfe NginxProxyManager Authelia Authelia-Secondary"
587	        [critical-fallback]=""      # dirty sync — auth stays running both sides
588	        [important-data]="Postgres-NextCloud NextCloud"
589	        [emby]="Emby"
590	        [emby-fallback]=""          # dirty sync — Emby stays running both sides
591	    )
592
593	# Containers needing a delay after rsync before starting.
594	# SPACE-SEPARATED STRINGS — converted to array at runtime
595	    declare -A PROFILE_DELAYED_CONTAINERS=(
596	        [arrs_stack]=""
597	        [critical-data]="Authelia Authelia-Secondary"   # wait for Mariadb + Redis
598	        [critical-fallback]=""
599	        [important-data]="NextCloud"                    # wait for Postgres
600	        [emby]=""
601	        [emby-fallback]=""
602	    )
603
604	# Seconds before starting delayed containers
605	    declare -A PROFILE_CONTAINER_DELAY=(
606	        [arrs_stack]=5
607	        [critical-data]=15          # Mariadb + Redis need time to accept connections
608	        [critical-fallback]=5
609	        [important-data]=10         # Postgres needs time before NextCloud
610	        [emby]=5
611	        [emby-fallback]=5
612	    )
613
614	# Directories excluded from rsync per profile.
615	# SPACE-SEPARATED STRINGS — converted to array at runtime
616	    declare -A PROFILE_EXCLUDE_DIRS=(
617	        [arrs_stack]="logs *.tmp"
618	        [critical-data]="logs *.tmp *.log nginx/temp nginx/cache __pycache__ notification.txt"
619	        [critical-fallback]="logs *.tmp *.log nginx/temp nginx/cache __pycache__ notification.txt *.db-wal *.db-shm"
620	        [important-data]="logs *.tmp"
621	        [emby]="logs transcodes cache crash*"
622	        [emby-fallback]="logs transcodes cache metadata *.db-wal *.db-shm crash* plugins root"
623	    )
624
625	# Remote restart after dirty sync — restart these on remote IF they were running before sync.
626	# Same logic as stop/start — was stopped = stays stopped, was running = gets restarted.
627	# Used by dirty sync profiles (critical-fallback, emby-fallback) so remote picks up changes.
628	# SPACE-SEPARATED STRINGS — converted to array at runtime
629	    declare -A PROFILE_REMOTE_RESTART_CONTAINERS=(
630	        [critical-fallback]="NginxProxyManager Authelia Authelia-Secondary Lldap-Gmer4Lfe Mariadb-Authelia Mariadb-Authelia-Secondary Redis-Authelia Redis-Authelia-Secondary"
631	        [emby-fallback]="Emby"
632	    )
633
634	# Note: disk check is auto-detected from disks.ini — no PROFILE_SKIP_DISK_CHECK needed.
635	# check_remote_disks() reads fsType per disk and handles XFS, ZFS, and cache pools automatically.
636
637	# ==============================================================================================
638	# ── FALLBACK ──────────────────────────────────────────────────────────────────────────────────
639	# ==============================================================================================
640	# Mutual container failover between two unRAID servers.
641	# Each server runs Fallback/fallback.sh independently via array_started.sh.
642	# All decisions based on two pings: remote reachable + internet reachable.
643	#
644	# States: NORMAL | FALLBACK | NO_INTERNET | DARK
645	#
646	# DDNS rules — absolute:
647	#   Internet loss → stop own DDNS immediately
648	#   Failover → start remote DDNS first (Tier 1)
649	#   Handback → stop remote DDNS → rsync → start containers → start local DDNS last
650	#
651	# Per-host container lists and tier delays live in host*.conf.
652	# Shared settings (intervals, state file, thresholds) live here.
653
654	    EXTERNAL_IP="8.8.8.8"
655	    FALLBACK_CHECK_INTERVAL=30      # seconds between fallback state checks
656	    FALLBACK_HANDBACK_STRIKES=3     # consecutive healthy checks before initiating handback (3×30s = 90s)
657	    FALLBACK_STATE_FILE="$STATE_DIR/fallback_state.db"
658	    FALLBACK_ENABLED=true           # HOST2 back online
659	                                    # false = suppresses "not running" warnings in status scripts
660	    FALLBACK_PARTNERSHIP_REQUIRED=true          # gate fallback on an active partnership
661	                                                # false = standalone mode, no partnership dependency
662	    FALLBACK_PARTNERSHIP_SUSPEND_AFTER=120      # minutes without active partnership before suspending
663	                                                # 0 = suspend immediately when partnership goes inactive
664
665	# ━━━ Failover Test ━━━
666	# Controlled simulation of a failover event — run manually via fallback_test.sh.
667	    FALLBACK_TEST_BLOCK_WAIT=150    # seconds to wait after blocking connectivity
668	    FALLBACK_TEST_HANDBACK_WAIT=360 # seconds to wait before initiating handback
669
670	# ==============================================================================================
671	# ── DOCKER ESSENTIALS ─────────────────────────────────────────────────────────────────────────
672	# ==============================================================================================
673
674	# ━━━ Downloaders Reset ━━━
675	# Runs every 30 minutes via CRITICAL_MAINTENANCE_SCRIPTS.
676	# Clears stuck states, purges old history, prepares each download client for a clean cycle.
677	# Per-host URLs and API keys live in host*.conf.
678	    DOWNLOADER_RETENTION_DAYS=7     # days — purge history older than this
679
680	# qBittorrent failsafe — removes torrents older than threshold regardless of ratio
681	# deleteFiles=false — removes from qBit but leaves files on disk (arr manages files)
682	    QBIT_FAILSAFE_MIN_DAYS=180      # days — minimum age before failsafe deletion
683	    QBIT_FAILSAFE_MIN_RATIO=0       # 0 = age only, no ratio requirement
684
685	# ━━━ Docker Daily Restart ━━━
686	# Containers restarted every day via DAILY_MAINTENANCE_SCRIPTS.
687	# Per-host lists live in host*.conf:
688	#   HOST1_DAILY_RESTART_CONTAINERS
689	#   HOST2_DAILY_RESTART_CONTAINERS
690	# detect_hosts() sets DAILY_RESTART_CONTAINERS to the correct host array at runtime.
691
692	# ━━━ Docker Weekly Restart ━━━
693	# Less critical services restarted weekly via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am).
694	# Containers already stopped for weekly sync — restart adds zero extra downtime.
695	# Per-host lists live in host*.conf:
696	#   HOST1_WEEKLY_RESTART_CONTAINERS
697	#   HOST2_WEEKLY_RESTART_CONTAINERS
698
699	# ━━━ Docker Watchdog ━━━
700	# Continuous two-tier self-healing container monitoring.
701	# Started by array_started.sh — runs until array stops.
702	# Re-sources all three conf files each cycle — add/remove containers without restarting watchdog.
703	#
704	# Tier 1 — strict monitoring of explicitly configured containers:
705	#   Memory hard limits    — immediate restart if exceeded
706	#   CPU thresholds        — strike system, restart after CPU_FAIL_LIMIT sustained strikes
707	#   HTTP responsiveness   — strike system, restart after RESP_FAIL_LIMIT failed checks
708	#   Required containers   — must always be running, strike + skip list with auto-clear
709	#
710	# Tier 2 — global health scan of ALL running containers:
711	#   Unhealthy status      — Docker HEALTHCHECK unhealthy → restart
712	#   OOM killed            — kernel killed → restart + notify
713	#   Crash loop detection  — RestartCount climbing → notify, critical above limit
714	#   Dead containers       — remove and restart
715	#   Unexpected exits      — non-zero exit code → restart
716	#
717	# All per-host container lists live in host*.conf:
718	#   HOST*_WATCHDOG_CONTAINERS         — memory hard limits per container
719	#   HOST*_WATCHDOG_CONTAINER_URLS     — HTTP health check URLs
720	#   HOST*_WATCHDOG_REQUIRED_CONTAINERS — must always be running
721	#   HOST*_WATCHDOG_SCAN_IGNORE        — skip in Tier 2 scan
722	#   HOST*_WATCHDOG_DEPENDENCIES       — dependency ordering for restart decisions
723
724	# Strike state file — persistent in STATE_DIR
725	    WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
726
727	# CPU thresholds — normalised against total core count at runtime
728	    SOFT_CPU_THRESHOLD=80           # warn at this % of total system CPU
729	    HARD_CPU_THRESHOLD=85           # strike at this % of total system CPU
730	    CPU_FAIL_LIMIT=2                # consecutive hard CPU strikes before container restart
731
732	# Memory soft threshold — warn when container reaches this % of its hard limit
733	    SOFT_MEM_THRESHOLD=80
734
735	# HTTP responsiveness
736	    RESP_FAIL_LIMIT=2               # consecutive failed checks before restart
737	    CURL_TIMEOUT=5                  # seconds per check before timeout
738
739	# Tier 2 master toggle
740	    WATCHDOG_SCAN_ALL=true          # false = only WATCHDOG_CONTAINERS + required containers
741
742	# Individual Tier 2 check toggles
743	    WATCHDOG_RESTART_UNHEALTHY=true
744	    WATCHDOG_RESTART_DEAD=true
745	    WATCHDOG_RESTART_CRASHED=true
746	    WATCHDOG_NOTIFY_OOM=true
747	    WATCHDOG_NOTIFY_CRASHLOOP=true
748
749	# Crash loop threshold
750	    WATCHDOG_CRASH_LIMIT=5          # RestartCount above this = critical crash loop
751
752	# Startup grace period — skip restarts while system is still booting
753	    WATCHDOG_STARTUP_GRACE=600      # seconds after boot before watchdog acts on failures
754
755	# Restart loop protection — prevents watchdog from endlessly restarting a broken container
756	    WATCHDOG_CONTAINER_RESTART_LIMIT=3
757	    WATCHDOG_CONTAINER_RESTART_WINDOW=1     # rolling window in hours
758	    WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
759
760	# Notification batching — one summary per cycle instead of one ping per event
761	    WATCHDOG_BATCH_NOTIFY=true
762
763	# Appdata size monitoring — two-part catch-all for runaway growth and oversized log files.
764	#
765	# Part 1 — Growth rate (zero-config):
766	#   Reads per-container dir totals each cycle via du, compares to previous cycle.
767	#   Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused *.log scan
768	#   inside that container. No per-container config required — new containers are covered
769	#   automatically. Baseline built on first run after boot; growth detection starts cycle 2.
770	#
771	# Part 2 — Absolute log size:
772	#   Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB across all appdata paths.
773	#   Catches logs that have already stabilised at a large size and are no longer actively growing.
774	#
775	# Strike system (reuses existing watchdog infrastructure):
776	#   Strike 1 — warn + notify: condition first detected
777	#   Strike 2 — warn + escalated notify: still present next cycle
778	#   Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
779	#     If WATCHDOG_APPDATA_TRUNCATE_LOGS=true: truncate *.log files in-place, clear strikes
780	#     If false: critical notify only, strikes held until condition resolves
781	#   Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
782	#
783	# HOST*_WATCHDOG_APPDATA_SIZES (in host*.conf) suppresses growth warnings for a
784	#   container until its dir exceeds the configured ceiling. Only needed when a container
785	#   legitimately has large stable data and you want to guarantee it never triggers a false alarm.
786	    WATCHDOG_CHECK_APPDATA=true
787	    WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
788	    WATCHDOG_APPDATA_GROWTH_GB=2            # flag containers growing more than this per cycle
789	    WATCHDOG_APPDATA_LOG_MAX_GB=2           # flag *.log files exceeding this size (absolute)
790	    WATCHDOG_APPDATA_TRUNCATE_LOGS=false    # set true to auto-truncate oversized *.log files on action cycle
791	    WATCHDOG_APPDATA_STRIKE_LIMIT=3         # cycles before action fires (matches existing watchdog pattern)
792	    WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
793	    STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"
794
795	# ━━━ Docker Network Connect ━━━
796	# Ensures custom networks exist and connects containers at array start.
797	# Runs once via ARRAY_START_SCRIPTS — idempotent, safe to re-run.
798	# Container and network lists are host-specific — defined in host*.conf:
799	#   HOST1_NETWORK_CONNECT_CONTAINERS / HOST2_NETWORK_CONNECT_CONTAINERS
800	#   HOST1_NETWORK_CONNECT_NETWORKS   / HOST2_NETWORK_CONNECT_NETWORKS
801
802	# ==============================================================================================
803	# ── UNRAID ESSENTIALS ─────────────────────────────────────────────────────────────────────────
804	# ==============================================================================================
805
806	# ━━━ inotify Tuning ━━━
807	# Linux inotify limits — applied at every array start by inotify_tuning.sh.
808	# Default unRAID values are very low — with many Docker containers watching files
809	# you can silently exhaust the limit causing containers to miss file events.
810	# Symptoms: containers miss file events, downloads not detected, library not updated.
811	# Live TV stutter is a known symptom of inotify exhaustion.
812	# These settings are lost on reboot — reapplied automatically at array start.
813	    INOTIFY_MAX_INSTANCES=1024      # default: 128   — max inotify instances per user
814	                                    # 1024 handles ~20-30 containers watching files simultaneously
815	    INOTIFY_MAX_WATCHES=1048576     # default: 8192  — SHARED budget across ALL users/containers
816	                                    # 524288 (512K) was previous value — raised to 1M (1048576)
817	                                    # VSCode/Code-Server alone needs ~50K-200K for large workspaces
818	                                    # with node_modules. All arr containers + Emby + VSCode share
819	                                    # this budget. 1M is safe on 128GB RAM (~128MB kernel memory)
820	                                    # If VSCode shows "unable to watch for file changes" → too low
821	    INOTIFY_MAX_QUEUED_EVENTS=32768 # default: 16384 — max events queued before dropping
822
823	# ━━━ System Tuning Monitor ━━━
824	# Tracks inotify and php-fpm usage over time.
825	# Snapshot written every 6 hours by system_tuning_monitor.sh.
826	# Read by sunday_morning_coffee_report.sh for weekly peak/avg/warning summary.
827	    INOTIFY_WARN_PCT=80             # warn if inotify instances exceed this % of limit
828	    PHP_FPM_WARN_PCT=80             # warn if php-fpm workers exceed this % of max_children
829	    TUNING_MONITOR_LOG="$DATA_DIR/system_tuning_history.db"
830	    TUNING_LOG_RETENTION=30         # days before old entries are purged
831
832	# ━━━ Reboot ━━━
833	# Seconds of warning broadcast to logged-in users before server_reboot.sh reboots.
834	# Gives users time to save work — 300s = 5 minutes.
835	    REBOOT_SLEEP=300
836
837	# ━━━ Mover ━━━
838	# Seconds to wait before mover_stop.sh sends SIGTERM to the mover process.
839	# Gives mover time to finish current file transfer before being interrupted.
840	    MOVER_STOP_TIMEOUT=300
841
842	# ━━━ Syslog Filter ━━━
843	# Path for the rsyslog filter file that suppresses Docker veth interface noise.
844	# Docker creates a new veth interface for each container — generates hundreds of
845	# log lines per hour that have no diagnostic value. Filter removes them at source.
846	    FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
847
848	# ━━━ PHP-FPM ━━━
849	# Higher max_children allows more concurrent PHP requests to the unRAID WebGUI.
850	# Default is very low — increasing it prevents WebGUI slowdowns under load.
851	# 250 is safe for servers with 32GB+ RAM.
852	    PHP_CONF="/etc/php-fpm.d/www.conf"
853	    PHP_MAX_CHILDREN=250
854
855	# ━━━ Clear Logs ━━━
856	# System log files cleared weekly to prevent rootfs fill over time.
857	# These grow continuously — without clearing they eventually consume all rootfs space.
858	# clear_logs.sh runs in WEEKLY_MAINTENANCE_SCRIPTS — Sunday 2:30am.
859	#
860	# Size threshold approach — only clear if log exceeds threshold.
861	# Avoids destroying useful recent diagnostic context when logs are small.
862	# LOG_MIN_SIZE_MB: skip clearing if log is under this size (not worth clearing)
863	# LOG_DOCKER_MAX_MB: clear a container log only if it exceeds this size
864	#   Docker logs grow fastest on active containers (Emby, SABnzbd, Sonarr)
865	#   100MB per container × 30 containers = 3GB before clearing kicks in
866	    LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg)
867	    LOG_MIN_SIZE_MB=10          # skip system log if under this size (already small)
868	    LOG_DOCKER_MAX_MB=100       # clear Docker container log only if over this size (MB)
869
870	# ━━━ Network Watchdog ━━━
871	# Services-layer connectivity — internet reachability, DDNS sync, Tailscale, NPM proxy.
872	# Host-specific values (domain, container, NPM URL) live in host*.conf.
873	    NETWORK_WATCHDOG_ENABLED=true
874	    NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
875	    NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
876	    NETWORK_WATCHDOG_CHECK_TAILSCALE=true
877	    NETWORK_WATCHDOG_NPM_TIMEOUT=10
878	    NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
879	    NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
880
881	# ━━━ WebGUI Watchdog ━━━
882	# Monitors unRAID WebGUI responsiveness — escalates through nginx restart → emhttp restart.
883	# Separate from docker_watchdog — this monitors the unRAID UI itself, not containers.
884	    WEBGUI_URL="http://localhost"
885	    WEBGUI_TIMEOUT=5        # seconds before curl gives up on WebGUI check
886	    WEBGUI_NGINX_WAIT=15    # seconds after nginx restart before rechecking
887	    WEBGUI_EMHTTP_WAIT=30   # seconds after emhttp restart before rechecking
888
889	# ==============================================================================================
890	# ── MEDIA ─────────────────────────────────────────────────────────────────────────────────────
891	# ==============================================================================================
892
893	# ━━━ Media Permissions ━━━
894	# Applied recursively by media_shares_permissions.sh.
895	# Runs first in DAILY_MAINTENANCE_SCRIPTS — arr cleanup depends on correct ownership.
896	#
897	# Why split directory vs file permissions:
898	#   Directories need execute bit to enter — 755 allows owner+group+others to traverse
899	#   Files should NOT be executable — 664 allows owner+group read/write, others read
900	#   This is POSIX best practice for media servers, not a blanket 777 band-aid
901	#
902	# Why nobody:users ownership:
903	#   All ecosystem containers run as PUID=99 (nobody) PGID=100 (users) on unRAID
904	#   Files owned by nobody:users are accessible to all containers without world-write
905	#   If this script fixes many files each run → a container has wrong PUID/PGID set
906	#   Fix: add PUID=99 PGID=100 to the container's environment variables
907	#
908	# This script runs daily as a failsafe — even with correct container config:
909	#   rsync may bring files with source ownership if not run with --chown
910	#   Manual admin copies create root:root files
911	#   New containers may not have PUID/PGID set yet
912	    PERMISSIONS_DIR_MODE="755"    # directories — traverse + list, no world-write
913	    PERMISSIONS_FILE_MODE="664"   # files — group read/write, no execute
914	    PERMISSIONS_OWNER="nobody:users"
915
916	# Share list defined per host in host*.conf:
917	#   HOST1_MEDIA_PERMISSION_SHARES / HOST2_MEDIA_PERMISSION_SHARES
918
919	# ━━━ Media Cleaner ━━━
920	# Removes junk files from media shares — two profiles: anime and media.
921	# Called via DAILY_MAINTENANCE_SCRIPTS. Run manually: Media/media_cleaner.sh anime|media
922	# Folder lists defined per host in host*.conf:
923	#   HOST1_ANIME_CLEAN_FOLDERS / HOST2_ANIME_CLEAN_FOLDERS
924	#   HOST1_MEDIA_CLEAN_FOLDERS / HOST2_MEDIA_CLEAN_FOLDERS
925
926	    ANIME_FILE_PATTERNS=(
927	        # ── Checksums ────────────────────────────────────────────────────────────────────────
928	        '*.sfv' '*.md5' '*.sha1'            # verification files — useless post-download
929
930	        # ── Scene / download metadata ─────────────────────────────────────────────────────────
931	        '*.url' '*.lnk'                     # scene links
932	        '*.info' '*.diz'                    # scene description files
933	        '*.nzb'                             # usenet download files
934	        '*.torrent'                         # torrent files left by download clients
935
936	        # ── Archives and segments ─────────────────────────────────────────────────────────────
937	        '*.rar' '*.zip' '*.7z' '*.ace'      # archives — source files not needed after extract
938	        '*.r00' '*.r01' '*.r02' '*.r03'     # multi-part rar segments
939	        '*.r04' '*.r05' '*.r06' '*.r07'
940	        '*.r08' '*.r09' '*.srr'             # scene repair files
941	        '*.001' '*.002' '*.003'             # split archive parts
942	        '*.gz' '*.tar' '*.bz2'             # linux archives
943
944	        # ── Scene samples and proofs ──────────────────────────────────────────────────────────
945	        '*.sample*' '*.proof*'              # scene samples — never needed in library
946
947	        # ── Executables and scripts ───────────────────────────────────────────────────────────
948	        '*.exe' '*.scr' '*.com'             # Windows executables
949	        '*.bat' '*.cmd' '*.vbs' '*.ps1'     # Windows scripts
950	        '*.msi' '*.dll' '*.sys'             # Windows installers and system files
951	        '*.sh'                              # shell scripts in media folders = suspicious
952
953	        # ── Incomplete downloads ──────────────────────────────────────────────────────────────
954	        '*.!ut' '*.!qB'                     # uTorrent / qBittorrent incomplete markers
955	        '*.crdownload' '*.opdownload'       # Chrome / Opera incomplete downloads
956	        '*.part'                            # partial download files
957
958	        # ── Sync conflicts ────────────────────────────────────────────────────────────────────
959	        '*sync-conflict*'                   # Syncthing conflict copies
960	    )
961
962	    MEDIA_FILE_PATTERNS=(
963	        # ── Checksums ────────────────────────────────────────────────────────────────────────
964	        '*.sfv' '*.md5' '*.sha1'            # verification files — useless post-download
965
966	        # ── Scene / download metadata ─────────────────────────────────────────────────────────
967	        '*.url' '*.lnk'                     # scene links
968	        '*.info' '*.diz'                    # scene description files
969	        '*.nzb'                             # usenet download files
970	        '*.torrent'                         # torrent files left by download clients
971
972	        # ── Archives and segments ─────────────────────────────────────────────────────────────
973	        '*.rar' '*.zip' '*.7z' '*.ace'      # archives — source files not needed after extract
974	        '*.r00' '*.r01' '*.r02' '*.r03'     # multi-part rar segments
975	        '*.r04' '*.r05' '*.r06' '*.r07'
976	        '*.r08' '*.r09' '*.srr'             # scene repair files
977	        '*.001' '*.002' '*.003'             # split archive parts
978	        '*.gz' '*.tar' '*.bz2'             # linux archives
979
980	        # ── Scene samples and proofs ──────────────────────────────────────────────────────────
981	        '*.sample*' '*.proof*'              # scene samples — never needed in library
982
983	        # ── Executables and scripts ───────────────────────────────────────────────────────────
984	        '*.exe' '*.scr' '*.com'             # Windows executables
985	        '*.bat' '*.cmd' '*.vbs' '*.ps1'     # Windows scripts
986	        '*.msi' '*.dll' '*.sys'             # Windows installers and system files
987	        '*.sh'                              # shell scripts in media folders = suspicious
988
989	        # ── Incomplete downloads ──────────────────────────────────────────────────────────────
990	        '*.!ut' '*.!qB'                     # uTorrent / qBittorrent incomplete markers
991	        '*.crdownload' '*.opdownload'       # Chrome / Opera incomplete downloads
992	        '*.part'                            # partial download files
993
994	        # ── Sync conflicts ────────────────────────────────────────────────────────────────────
995	        '*sync-conflict*'                   # Syncthing conflict copies
996
997	        # ── Media-specific extras ─────────────────────────────────────────────────────────────
998	        '*.iso'                             # disc images — extracted content is in library
999	        '*.lrc'                             # lyrics files — handled by music apps not Emby
1000	    )
1001
1002	# ━━━ Arr Cleanup ━━━
1003	# Orphan file cleanup via Lidarr, Sonarr, and Radarr APIs.
1004	# Per-host URLs, API keys, and path maps live in host*.conf.
1005	# detect_hosts() selects correct host vars at runtime.
1006	#
1007	# API versions — update MAJOR version here when script is updated for a new arr version:
1008	#   Sonarr v4  → /api/v3/series  → /api/v3/episodefile?seriesId=X
1009	#   Radarr v6  → /api/v3/movie   → /api/v3/moviefile?movieId=X
1010	#   Lidarr v3  → /api/v1/artist  → /api/v1/trackFile?artistId=X
1011	    SONARR_VERSION_MAJOR=4
1012	    RADARR_VERSION_MAJOR=6
1013	    LIDARR_VERSION_MAJOR=3
1014
1015	# Lidarr shared settings
1016	    LIDARR_LOCK_WARN_AGE=3600       # 1hr — large libraries take time, not stuck
1017	    LIDARR_ORPHAN_AGE=7             # days — files must be older than this before eligible for deletion
1018	    LIDARR_MAX_DELETE_GB=5          # require --i-know-what-im-doing if deletion exceeds this
1019	    LIDARR_MIN_TRACKED_PCT=80       # abort if tracked count drops below this % of last run
1020	                                    # protects against API returning partial data on a bad day
1021	    LIDARR_TRACKED_COUNT_FILE="$DATA_DIR/lidarr_tracked.count"
1022	    LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
1023	    LIDARR_PROTECTED_PATTERNS=(
1024	        # Metadata
1025	        "*.nfo" "*.tbn"
1026	        # Images — album art, artist images, Emby artwork
1027	        "*.jpg" "*.jpeg" "*.png" "*.webp" "*.svg"
1028	        "poster.*" "fanart.*" "backdrop.*" "clearlogo.*"
1029	        "banner.*" "thumb.*" "landscape.*"
1030	        "folder.*" "cover.*" "album.*" "artist.*" "disc.*"
1031	        # Lyrics
1032	        "*.lrc"
1033	        # Cue sheets and rip logs
1034	        "*.cue" "*.log"
1035	        # Theme music — Emby downloads artist themes into artist folders
1036	        "theme.mp3" "theme.flac" "theme.m4a" "theme.wav"
1037	    )
1038
1039	# Lidarr artwork fetcher settings
1040	    LIDARR_ART_MIN_SIZE=10000           # bytes — reject downloads smaller than this
1041	    LIDARR_ART_MAX_PARALLEL=4           # concurrent background download jobs
1042	    LIDARR_ART_RETRIES=2                # download retry attempts per image
1043	    LIDARR_ART_SLEEP_BETWEEN=0.2        # seconds between fanart.tv API calls
1044	#   HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in host*.conf
1045
1046	# Lidarr discovery settings (playback_aware_lidarr_discovery.sh)
1047	    LIDARR_DISCOVERY_THRESHOLD=70         # score to accept candidate (0-100)
1048	    LIDARR_DISCOVERY_LOOKBACK_DAYS=7      # Emby play history window in days
1049	    LIDARR_DISCOVERY_MIN_PLAYS=3          # min plays in window before evaluating an artist
1050	    LIDARR_DISCOVERY_USER_CAP_PCT=35      # max % any single user can contribute to play score (prevents one listener dominating)
1051	    LIDARR_DISCOVERY_MAX_ADDS=5           # max artists to add per run — quality over bulk
1052	    LIDARR_DISCOVERY_REJECT_COOLDOWN=30   # days before re-evaluating a rejected artist
1053	    LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db"
1054
1055	# Sonarr discovery settings (playback_aware_sonarr_discovery.sh)
1056	    SONARR_DISCOVERY_THRESHOLD=52         # score to accept candidate (0-100)
1057	    SONARR_DISCOVERY_LOOKBACK_DAYS=14     # Emby episode play history window in days (shorter than movies — TV watched more frequently)
1058	    SONARR_DISCOVERY_MAX_SEEDS=5          # max seed series from Stage 1
1059	    SONARR_DISCOVERY_MAX_ADDS=3           # max shows to add per run — TV is a larger commitment than movies
1060	    SONARR_DISCOVERY_MIN_VOTE_COUNT=50    # min TMDB votes (TV has fewer votes than movies at same popularity)
1061	    SONARR_DISCOVERY_MIN_RATING=65        # min TMDB vote_average × 10 (65 = 6.5/10)
1062	    SONARR_DISCOVERY_REJECT_COOLDOWN=60   # days before re-evaluating a rejected show
1063	    SONARR_DISCOVERY_USER_EPISODE_CAP=8   # max episodes any one user contributes to seed volume score
1064	    SONARR_DISCOVERY_MONITOR_MODE="all"    # Sonarr monitor mode on add: all | future | first | latest | none
1065	    SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db"
1066
1067	# Radarr discovery shared settings
1068	    RADARR_DISCOVERY_THRESHOLD=52         # score to accept candidate (0-100) — lower than Lidarr since diverse seeds rarely overlap
1069	    RADARR_DISCOVERY_LOOKBACK_DAYS=30     # Emby watch history window in days (movies rewatched less often)
1070	    RADARR_DISCOVERY_MAX_SEEDS=5          # max seed movies from Stage 1
1071	    RADARR_DISCOVERY_MAX_ADDS=5           # max movies to add per run
1072	    RADARR_DISCOVERY_MIN_VOTE_COUNT=100   # min TMDB votes to be considered a candidate
1073	    RADARR_DISCOVERY_MIN_RATING=60        # min TMDB vote_average × 10 (60 = 6.0/10)
1074	    RADARR_DISCOVERY_REJECT_COOLDOWN=60   # days before re-evaluating a rejected movie
1075	    RADARR_DISCOVERY_SEED_LIBRARIES=("Movies")  # Emby libraries to draw seed movies from
1076	    RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db"
1077
1078	# Emby → arr sync library allowlists
1079	# Only these Emby library names will be considered by the sync tools.
1080	# Names must match exactly as shown in Emby > Dashboard > Libraries.
1081	    SONARR_EMBY_LIBRARIES=("Anime" "Kids Shows" "Stand-Up Comedy" "TV shows")
1082	    RADARR_EMBY_LIBRARIES=("Movies" "Kid's Movies")
1083
1084	# Sonarr shared settings
1085	    SONARR_ORPHAN_AGE=7             # days — files must be older than this before eligible for deletion
1086	    SONARR_MAX_DELETE_GB=10         # require --i-know-what-im-doing if deletion exceeds this
1087	    SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov")
1088	    SONARR_PROTECTED_PATTERNS=(
1089	        # Subtitles
1090	        "*.srt" "*.sub" "*.ass" "*.ssa" "*.idx" "*.vtt"
1091	        # Metadata
1092	        "*.nfo" "*.tbn"
1093	        # Images
1094	        "*.jpg" "*.jpeg" "*.png" "*.webp" "*.svg"
1095	        "poster.*" "fanart.*" "backdrop.*" "clearlogo.*"
1096	        "banner.*" "thumb.*" "landscape.*"
1097	        # Kodi/Emby extras — not tracked by Sonarr API
1098	        "*-trailer.*" "*-featurette.*" "*-behindthescenes.*"
1099	        "*-interview.*" "*-scene.*" "*-short.*" "*-deleted.*"
1100	        "*-clip.*" "*-other.*"
1101	        # Theme songs — stored in show folder, not tracked by arr
1102	        "theme.mp3" "theme.flac" "theme.wav" "theme.m4a" "theme.mka"
1103	        "theme.mkv" "theme.mp4"
1104	        "trailer.*" "trailer *.*"
1105	    )
1106
1107	# Radarr shared settings
1108	    RADARR_ORPHAN_AGE=7             # days — files must be older than this before eligible for deletion
1109	    RADARR_MAX_DELETE_GB=15         # require --i-know-what-im-doing if deletion exceeds this
1110	    RADARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "wmv" "mov")
1111	    RADARR_PROTECTED_PATTERNS=(
1112	        # Subtitles
1113	        "*.srt" "*.sub" "*.ass" "*.ssa" "*.idx" "*.vtt"
1114	        # Metadata
1115	        "*.nfo" "*.tbn"
1116	        # Images
1117	        "*.jpg" "*.jpeg" "*.png" "*.webp" "*.svg"
1118	        "poster.*" "fanart.*" "backdrop.*" "clearlogo.*"
1119	        "banner.*" "thumb.*" "landscape.*"
1120	        # Kodi/Emby extras — not tracked by Radarr API
1121	        "*-trailer.*" "*-featurette.*" "*-behindthescenes.*"
1122	        "*-interview.*" "*-scene.*" "*-short.*" "*-deleted.*"
1123	        "*-clip.*" "*-other.*"
1124	        # Theme songs — stored in movie folder, not tracked by arr
1125	        "theme.mp3" "theme.flac" "theme.wav" "theme.m4a" "theme.mka"
1126	        "theme.mkv" "theme.mp4"
1127	        "trailer.*" "trailer *.*"
1128	    )
1129
1130	# Radarr/Sonarr TMDb/TVDB removed entry cleanup
1131	    RADARR_DROPPED_ADD_EXCLUSION=true   # add removed movies to import exclusion list
1132	    SONARR_DROPPED_ADD_EXCLUSION=true   # add removed series to import exclusion list
1133
1134	# ━━━ Arr Failed/Stalled Recovery ━━━
1135	# Auto blocklist + re-search failed imports and stalled downloads.
1136	# Runs every 6 hours — schedule: 0 */6 * * *
1137	#
1138	# Targets four problem types:
1139	#   importFailed   — downloaded but arr couldn't import
1140	#   importPending  — downloaded, stuck waiting to import (won't self-resolve)
1141	#   error status   — serious failure not covered above
1142	#   stalled        — download stuck with no connections or progress
1143	#
1144	# Items newer than ARR_IMPORT_RECOVERY_AGE are skipped — gives arr time to retry first.
1145	# Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in host*.conf.
1146	    ARR_IMPORT_RECOVERY_AGE=6           # hours — skip items newer than this
1147	                                        # matches cron interval — items eligible after one missed cycle
1148
1149	# ==============================================================================================
1150	# ── TRANSCODES ────────────────────────────────────────────────────────────────────────────────
1151	# ==============================================================================================
1152	# Session-based storage allocator using filesystem symlink indirection.
1153	# ffmpeg resolves the symlink ONCE at session start — existing sessions never affected.
1154	# Symlink flips between ramdisk and SSD are transparent to active streams.
1155	#
1156	# ⚠ Docker mount — must use shared propagation so symlink flips work inside container:
1157	#   --mount type=bind,source=/mnt/ram-transcode,target=/ext-ram-transcode,bind-propagation=shared
1158	#   Standard path mappings use rprivate — symlink changes are NOT visible inside container.
1159
1160	# ━━━ Transcode Manager ━━━
1161	# tmpfs mount point — created at array start by ramdisk_setup.sh.
1162	# Must exist before Emby starts so the symlink resolves correctly.
1163	    RAMDISK_PATH="/mnt/ramdisk_transcodes"
1164
1165	# Ramdisk size and flip thresholds — defined per host in host*.conf.
1166	# All three are coupled — if size changes, thresholds must change with it.
1167	#   HOST1_RAMDISK_SIZE / HOST2_RAMDISK_SIZE
1168	#   HOST1_RAMDISK_WARN_GB / HOST2_RAMDISK_WARN_GB  ← flip to SSD at this usage
1169	#   HOST1_RAMDISK_LOW_GB  / HOST2_RAMDISK_LOW_GB   ← flip back to ramdisk at this usage
1170
1171	# Symlink that Emby points at — this path NEVER changes regardless of ramdisk/SSD state.
1172	# Emby resolves the symlink once per session at start — symlink flips are transparent.
1173	# Must match the container path configured in Emby's Extra Parameters.
1174	    TRANSCODE_LINK="/mnt/ram-transcode"
1175
1176	# SSD fallback location — defined per host in host*.conf (cache path differs per server):
1177	#   HOST1_TRANSCODE_SSD / HOST2_TRANSCODE_SSD
1178
1179	# Minimum free GB on SSD before allowing flip from ramdisk to SSD.
1180	    RAMDISK_SSD_MIN_GB=20
1181
1182	    TRANSCODE_MAX_AGE=20            # minutes — HLS segment age before cleanup eligibility
1183	    TRANSCODE_ORPHAN_AGE=30         # minutes — files with no matching active session
1184	    TRANSCODE_FLIP_WARN=3           # notify if symlink flips this many times in one hour
1185
1186	    TRANSCODE_OWNER="nobody:users"
1187	    TRANSCODE_CHMOD="755"
1188
1189	# Operating mode — controls symlink direction behaviour.
1190	#   smart   — auto-flips between ramdisk and SSD based on thresholds (default)
1191	#   ramdisk — always uses ramdisk, warns if RAMDISK_WARN_GB exceeded but holds
1192	#   ssd     — always uses SSD, never flips to ramdisk
1193	    TRANSCODE_MANAGER_MODE="smart"
1194
1195	# Daily statistics log — read by weekly_health_digest.sh for transcode summary.
1196	    TRANSCODE_STATE_FILE="$STATE_DIR/transcode_state.db"
1197	    TRANSCODE_DAILY_LOG="$DATA_DIR/transcode_daily.db"
1198	    TRANSCODE_LOG_RETENTION=90      # days before old entries purged
1199
1200	    TRANSCODE_CHECK_EMBY=true
1201
1202	# ━━━ Transcode Server Array ━━━
1203	# All media servers sharing the ramdisk transcode space.
1204	# Format: "ContainerName|URL|APIKey|Type" — Type: emby | jellyfin | plex
1205	# Entries with placeholder API keys are skipped automatically.
1206	# ⚠ Tdarr does NOT belong here — keep Tdarr on SSD, not ramdisk.
1207	# Defined per host in host*.conf — Emby container names and keys differ per server:
1208	#   HOST1_TRANSCODE_SERVERS / HOST2_TRANSCODE_SERVERS
1209
1210	# ==============================================================================================
1211	# ── MONITORS ──────────────────────────────────────────────────────────────────────────────────
1212	# ==============================================================================================
1213
1214	# ━━━ Certificate Monitor ━━━
1215	# Checks SSL certificate expiry via direct openssl connection — no NPM dependency.
1216	# Checks the actual certificate served by each domain, not what NPM thinks it has.
1217	# Domains defined per host in host*.conf — each server monitors its own domains:
1218	#   HOST1_CERT_MONITOR_DOMAINS / HOST2_CERT_MONITOR_DOMAINS
1219	    CERT_WARN_DAYS=30           # warn when cert expires within this many days
1220	    CERT_CRIT_DAYS=7            # critical alert within this many days
1221	    CERT_TIMEOUT=10             # seconds per domain before giving up
1222
1223	# ━━━ Backup Verify ━━━
1224	# Verifies rsync mirror health by comparing random file checksums between servers.
1225	# Catches silent corruption or incomplete syncs that rsync itself wouldn't detect.
1226	# Defined per host in host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically:
1227	#   HOST1_BACKUP_VERIFY_SHARES / HOST2_BACKUP_VERIFY_SHARES
1228	    BACKUP_VERIFY_SAMPLE=10         # random files to check per share
1229	    BACKUP_VERIFY_MIN_SIZE=1M       # minimum file size to include in sample
1230
1231	# ━━━ SMART Health ━━━
1232	# Monitors drive SMART attributes — discovers all drives via /dev/sd* and /dev/nvme*.
1233	# Thresholds read from /boot/config/plugins/dynamix/dynamix.cfg at runtime
1234	# (hot/max/hotssd/maxssd) — these vars are fallback only if dynamix.cfg not found.
1235	    SMART_TEMP_WARN=45              # fallback — Celsius warn threshold
1236	    SMART_TEMP_CRIT=55              # fallback — Celsius critical threshold
1237	# Drives to ignore defined per host in host*.conf — hardware is server-specific:
1238	#   HOST1_SMART_IGNORE_DRIVES / HOST2_SMART_IGNORE_DRIVES
1239
1240	# ━━━ ZFS Memory Snapshot ━━━
1241	# Weekly ZFS pool health and memory diagnostic report — informational only.
1242	    ZFS_REPORT_LOG="$DATA_DIR/zfs-weekly-health.log"
1243	    ZFS_REPORT_ARC_WARN_PCT=90      # warn if ARC using more than this % of its max
1244	    ZFS_REPORT_FREE_WARN_GB=10      # warn if less than this GB free RAM
1245	    ZFS_REPORT_AVAIL_WARN_GB=20     # warn if less than this GB available on ZFS pool
1246	    ZFS_REPORT_DOCKER_TOP=10        # how many top Docker containers to show by memory
1247	# Pool ignore list defined per host in host*.conf — pool names are server-specific:
1248	#   HOST1_ZFS_REPORT_IGNORE_POOLS / HOST2_ZFS_REPORT_IGNORE_POOLS
1249
1250	# ━━━ Bandwidth Monitor ━━━
1251	# Called automatically by rsync.sh after each sync — one bounded write per run.
1252	# Tracks transfer size, duration and profile per sync for weekly summary reporting.
1253	    BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
1254	    BANDWIDTH_LOG_RETENTION=90      # days before old entries purged
1255	    BANDWIDTH_WARN_GB=50            # flag syncs larger than this in weekly report
1256
1257	# Stats files — written by cleanup and recovery scripts, read by coffee report.
1258	# All in DATA_DIR — array always running when these are written.
1259	    ARR_CLEANUP_STATS="$DATA_DIR/arr_cleanup_stats.db"     # lidarr/sonarr/radarr orphan stats
1260	    ARR_RECOVERY_STATS="$DATA_DIR/arr_recovery_stats.db"   # blocklist + re-search stats
1261
1262	# ━━━ Health Digest ━━━
1263	# Aggregated system health summary — reads existing state files, no new writes.
1264	# Three profiles control when the digest is sent:
1265	#   always  — sends every run regardless of findings
1266	#   smart   — sends only when DIGEST_SMART_ON_* conditions are found
1267	#   weekly  — sends once per week on DIGEST_DAY only
1268	    DIGEST_PROFILE="weekly"         # always | smart | weekly
1269	    DIGEST_DAY="Sunday"
1270	    DIGEST_SMART_ON_WATCHDOG=true   # send if any watchdog strikes are active
1271	    DIGEST_SMART_ON_FALLBACK=true   # send if fallback state is not NORMAL
1272	    DIGEST_SMART_ON_CERT_WARN=true  # send if any cert is under CERT_WARN_DAYS
1273	    DIGEST_SMART_ON_BANDWIDTH=true  # send if any transfer exceeded BANDWIDTH_WARN_GB
1274
1275	# ━━━ Emby Session Report ━━━
1276	# Weekly Emby usage statistics via API — no persistent writes, queries fresh each run.
1277	# URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in host*.conf.
1278	    EMBY_REPORT_DAYS=7              # days to include in the report period
1279	    EMBY_REPORT_TOP_N=10            # number of top content items to show
1280
1281	# ==============================================================================================
1282	# ── RESOURCE MANAGER ──────────────────────────────────────────────────────────────────────────
1283	# ==============================================================================================
1284	# Pressure reduction layer — keeps the system comfortable before things break.
1285	# Called by watchdog_orchestrator.sh. Single-pass, not a continuous loop.
1286	#
1287	# ── PRESSURE LEVELS ───────────────────────────────────────────────────────────────────────────
1288	#   Level 1 (soft)   — throttle SABnzbd + qBit download speeds
1289	#   Level 2 (medium) — further throttle + docker pause background containers
1290	#   Level 3 (hard)   — docker stop optional containers, signal docker_watchdog to defer
1291	#
1292	# ── PER-HOST CONTAINER LISTS ──────────────────────────────────────────────────────────────────
1293	#   HOST*_RW_PAUSE_CONTAINERS  — docker pause at medium pressure (in host*.conf)
1294	#   HOST*_RW_STOP_CONTAINERS   — docker stop at hard pressure (in host*.conf)
1295
1296	    RW_ENABLED=true
1297	    RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"
1298
1299	# ━━━ Pressure Thresholds ━━━
1300	# Graduated RAM response — resource_watchdog acts before system_watchdog reboots.
1301	# RW_RAM_SOFT_GB > RW_RAM_MEDIUM_GB > RW_RAM_HARD_GB > SYS_WATCHDOG_MEM_GB always
1302	    RW_RAM_SOFT_GB=12           # throttle start — reduce background load
1303	    RW_RAM_MEDIUM_GB=8          # pause background containers
1304	    RW_RAM_HARD_GB=6            # stop optional containers (was SYS_WATCHDOG_MEM_SHUTDOWN_GB)
1305	    RW_RAM_RECOVER_GB=20        # RAM must reach this before restoring hard-stopped containers
1306
1307	# Load average thresholds — multiplier × core count
1308	    RW_LOAD_SOFT_MULTIPLIER=2.0     # soft pressure: 2× cores sustained
1309	    RW_LOAD_MEDIUM_MULTIPLIER=3.0   # medium pressure: 3× cores sustained
1310
1311	# Consecutive runs at lower pressure before de-escalating
1312	    RW_RECOVER_CYCLES=3
1313
1314	# ━━━ SABnzbd Throttle ━━━
1315	# Speed values: "50M" = 50 MB/s, "0" = unlimited
1316	    RW_SABNZBD_ENABLED=true
1317	    RW_SABNZBD_SPEED_SOFT="50M"
1318	    RW_SABNZBD_SPEED_MEDIUM="10M"
1319
1320	# ━━━ qBittorrent Throttle ━━━
1321	# KB/s — 0 = unlimited
1322	    RW_QBIT_ENABLED=true
1323	    RW_QBIT_DL_SOFT=51200       # 50 MB/s
1324	    RW_QBIT_DL_MEDIUM=10240     # 10 MB/s
1325
1326	# ━━━ Critical Containers ━━━
1327	# Never paused or stopped regardless of pressure level.
1328	# Keep DNS, auth, media serving, and live TV always running.
1329	    RW_CRITICAL_CONTAINERS=(
1330	        "NginxProxyManager"             # reverse proxy — internet access
1331	        "Authelia"                      # auth — nothing accessible without it
1332	        "Authelia-Secondary"
1333	        "Mariadb-Authelia"              # Authelia dependency
1334	        "Mariadb-Authelia-Secondary"
1335	        "Redis-Authelia"                # Authelia dependency
1336	        "Redis-Authelia-Secondary"
1337	        "AdGuard-Home"                  # DNS — all LAN resolution
1338	        "Emby"                          # media server — Live TV buffering
1339	        "Dispatcharr"                   # Live TV scheduler — loses state if stopped
1340	        "Dispatcharr-Basic"
1341	        "Dispatcharr-Iptv-Users"
1342	    )
1343
1344	# ==============================================================================================
1345	# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
1346	# ==============================================================================================
1347	# Single-pass system health check — last line of defense before a crash.
1348	# Called by watchdog_orchestrator.sh every minute. NOT started by array_started.sh.
1349	# Re-sources config at each orchestrator run — config changes take effect immediately.
1350	#
1351	# ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
1352	#   CRITICAL  — bypass ALL strikes, reboot immediately
1353	#               Docker daemon down, rootfs 100%, kernel oops, FD exhaustion, /boot read-only
1354	#
1355	#   URGENT    — bypass strikes only when OOM confirms active crisis
1356	#               RAM < MEM_GB AND OOM kills >= OOM_LIMIT in this cycle → reboot NOW
1357	#               Without OOM confirmation → normal strike system
1358	#
1359	#   STANDARD  — strike system (N consecutive failures → reboot)
1360	#               RAM tiers, high load, CPU temp, zombies, /var/log, /tmp, containers
1361	#
1362	# ── RAM ───────────────────────────────────────────────────────────────────────────────────────
1363	#   SYS_WATCHDOG_MEM_GB — strike system → reboot (or OOM bypass)
1364	#   Warn/shutdown/recover RAM tiers are handled by resource_watchdog.sh
1365
1366	# ━━━ State Files ━━━
1367	    SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
1368	    DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
1369	    SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
1370	    SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
1371
1372	# ━━━ Strike and Reboot Loop Settings ━━━
1373	# Strike system: a check must fail this many consecutive cycles before action is taken.
1374	# Single spikes (one bad reading) are ignored — sustained problems trigger reboot.
1375	    SYS_WATCHDOG_STRIKE_LIMIT=2         # consecutive failures before reboot trigger
1376
1377	# Reboot loop protection — if system keeps rebooting something is seriously wrong.
1378	# After REBOOT_LIMIT reboots in REBOOT_WINDOW_HRS → shutdown instead of reboot.
1379	    SYS_WATCHDOG_REBOOT_LIMIT=3
1380	    SYS_WATCHDOG_REBOOT_WINDOW_HRS=12
1381
1382	# ━━━ RAM Reboot Threshold ━━━
1383	# Reboot trigger only — warn/shutdown/recover handled by resource_watchdog.sh
1384	# RW_RAM_HARD_GB > SYS_WATCHDOG_MEM_GB always (RM acts before watchdog reboots)
1385	    SYS_WATCHDOG_MEM_GB=4              # strike system → reboot
1386
1387	# ━━━ OOM Bypass Settings ━━━
1388	# OOM bypass: if RAM is critically low AND kernel OOM kills exceed this threshold
1389	# in a single cycle → bypass strike system and reboot immediately.
1390	# Rate-based: kills per 5-minute cycle, not absolute count.
1391	# Rationale: 1-2 kills = docker_watchdog handles it ✅
1392	#            3+ kills while RAM critical = system dying faster than watchdogs can heal ✅
1393	    SYS_WATCHDOG_OOM_LIMIT=3           # OOM kills in one cycle to trigger bypass
1394
1395	# ━━━ Thresholds ━━━
1396	# Set at "about to become unstable" — not "things are a bit high".
1397
1398	# rootfs (/) usage — two levels: strike at 95%, critical bypass at 99%
1399	    SYS_WATCHDOG_ROOTFS_PCT=95         # standard strike threshold
1400	    SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99 # bypass strikes — truly full, writes failing
1401
1402	# /var/log usage — log spam indicates something broken
1403	    SYS_WATCHDOG_LOG_PCT=95
1404
1405	# /tmp usage — tmpfs fills from downloads, lock files fail at 100%
1406	    SYS_WATCHDOG_TMP_PCT=90            # warn + attempt clear
1407	    SYS_WATCHDOG_TMP_CRITICAL_PCT=98   # bypass strikes if clear failed
1408
1409	# ZFS ARC pinned percentage — ARC not releasing after reclaim = memory stuck
1410	    SYS_WATCHDOG_ARC_PINNED_PCT=98
1411	    SYS_WATCHDOG_ARC_RELEASE_PCT=95
1412
1413	# Load average multiplier — threshold = MULTIPLIER × CPU core count
1414	# MULTIPLIER=3 on 16-core = load of 48 before triggering
1415	    SYS_WATCHDOG_LOAD_MULTIPLIER=3
1416
1417	# Zombie process count — large numbers = serious process management failure
1418	    SYS_WATCHDOG_ZOMBIE_LIMIT=50
1419
1420	# CPU temperature — sustained high temp causes throttling or kernel panic
1421	    SYS_WATCHDOG_CPU_TEMP_MAX=95
1422
1423	# File descriptor exhaustion — system-wide FD limit near exhaustion
1424	# New connections fail silently, Docker can't spawn processes, SSH fails
1425	    SYS_WATCHDOG_FD_CRITICAL_PCT=95    # bypass strikes — critical tier
1426
1427	# Runaway process — single non-container process consuming excessive CPU
1428	# Multiple strikes before action — single spikes are normal
1429	    SYS_WATCHDOG_RUNAWAY_CPU_PCT=90    # % single process must sustain
1430	    SYS_WATCHDOG_RUNAWAY_STRIKES=3     # consecutive cycles before warning
1431
1432	# Array disk errors — accumulating mdstat errors = disk failing NOW
1433	    SYS_WATCHDOG_MDSTAT_ERROR_LIMIT=5  # new errors in one cycle before acting
1434
1435	# ━━━ Check Toggles ━━━
1436	# Per-host — moved to host*.conf
1437	# Different servers may have different hardware, NICs, and check requirements
1438	# See HOST*_SYS_WATCHDOG_CHECK_* in host*.conf
1439
1440	# ━━━ Abort Toggles ━━━
1441	# Conditions that prevent reboot even when a threshold is hit.
1442	# CRITICAL tier bypasses these — a truly critical condition reboots regardless.
1443	# true  = abort standard reboot if this condition is active
1444	# false = reboot anyway
1445	    SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true    # rebooting with bad pool risks data loss
1446	    SYS_WATCHDOG_ABORT_ON_PARITY=false           # aborting parity better than crashing mid-check
1447	    SYS_WATCHDOG_ABORT_ON_MOVER=false            # aborting move better than crashing mid-move
1448
1449	# ==============================================================================================
1450	# ──────────────────────── End Of User Variables ───────────────────────────────────────────────
1451	# ==============================================================================================
1452