Watchdogs/ folder + host conf rename

Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
This commit is contained in:
Gmer4Lfe
2026-05-22 17:08:36 -04:00
parent 9ee8af1a71
commit 95151c2278
73 changed files with 904 additions and 328 deletions
+76 -51
View File
@@ -8,10 +8,10 @@
# ── HOW THE THREE-FILE SYSTEM WORKS ──────────────────────────────────────────────────────────
# Scripts source all three files at startup:
# source master.conf ← shared config (this file)
# source master_host1.conf ← HOST1 credentials, shares, container lists
# source master_host2.conf ← HOST2 credentials, shares, container lists
# source host1.conf ← HOST1 credentials, shares, container lists
# source host2.conf ← HOST2 credentials, shares, container lists
#
# Sparse checkout (git) ensures each server only pulls its own master_host*.conf.
# Sparse checkout (git) ensures each server only pulls its own host*.conf.
# HOST2 never sees HOST1 credentials. HOST1 never sees HOST2 credentials.
#
# What belongs here: thresholds, toggles, intervals, profiles, job lists
@@ -99,7 +99,7 @@
# Must match the exact unRAID hostname AND Tailscale device name (case sensitive).
# detect_hosts() in common.sh matches the local hostname against these to set MY_ID / REMOTE_ID.
# Tailscale IP resolution uses these names — no hardcoded IPs needed.
# To add a new server: add HOST3="unRAID-NewServer" here + create master_host3.conf.
# To add a new server: add HOST3="unRAID-NewServer" here + create host3.conf.
HOST1="unRAID-Gmer4Lfe"
HOST2="unRAID-Jayred365"
@@ -145,7 +145,7 @@
# Manages the relationship lifecycle between two unRAID servers.
# HOST1 is always the owner (source of truth) — HOST2 is always the mirror.
# PARTNERSHIP_OWNER_HOST flips to "HOST2" after a --transfer operation.
# All identity vars (hostnames, SSH keys) live in master_host*.conf.
# All identity vars (hostnames, SSH keys) live in host*.conf.
# Hostnames already match Tailscale device names — IP resolution is automatic.
#
# State files on /boot/config — survives reboots, available before array starts:
@@ -164,13 +164,13 @@
PARTNERSHIP_ENABLED=false
PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer
# Auth containers reconfigured on onboard/offboard — defined per host in master_host*.conf.
# Auth containers reconfigured on onboard/offboard — defined per host in host*.conf.
# Format: "ContainerName|WebUIPort"
# HOST1_PARTNERSHIP_AUTH_WEBUIS / HOST2_PARTNERSHIP_AUTH_WEBUIS
# On onboard → WebUI pointed at owner's Tailscale IP
# On offboard → WebUI pointed back at localhost
# Paths to collect during the grace window after offboard — defined per host in master_host*.conf.
# Paths to collect during the grace window after offboard — defined per host in host*.conf.
# HOST1_PARTNERSHIP_MIRROR_BACKUPS / HOST2_PARTNERSHIP_MIRROR_BACKUPS
# Notified on offboard — no auto-deletion, manual collection.
@@ -219,13 +219,6 @@
# ── LOGGING ───────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Silent-by-default output model — ecosystem only speaks when something is wrong.
# true = only warn() and error() produce output (default — reduces notification spam)
# false = all output visible — use for monitor scripts or debugging
# Override per-run: script --log sets ENABLE_LOGGING=true for [LOG] detail
# Monitor scripts (coffee_report, health_digest etc.) set SILENT_MODE=false themselves
SILENT_MODE=true
# Controls verbose [LOG] output across all scripts.
# true = show detailed [LOG] lines — useful for debugging or first-time setup
# false = show only user-facing output — cleaner for scheduled runs
@@ -239,7 +232,7 @@
# normal = job completed successfully / warning = something failed or needs attention
NOTIFY_UNRAID=true
# Discord webhook URL — defined per host in master_host*.conf.
# Discord webhook URL — defined per host in host*.conf.
# HOST1_DISCORD_WEBHOOK / HOST2_DISCORD_WEBHOOK
# Allows different webhooks per server, or only one server notifying.
@@ -284,7 +277,6 @@
# Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via
# watchdog_orchestrator.sh — NOT launched here.
ARRAY_START_SCRIPTS=(
"git_pull_execute.sh" # pull latest scripts before anything starts
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
@@ -298,11 +290,12 @@
# Schedule: * * * * * (every minute)
# NOT in ARRAY_START_SCRIPTS — has its own cron entry.
# Order matters — resource first (frees pressure), docker second (heals with freed resources),
# system last (reboots only if prior layers failed).
# storage third (pool/data health — never reboots), system last (last line of defense).
WATCHDOG_ORCHESTRATOR_SCRIPTS=(
"unRAID_Essentials/resource_watchdog.sh" # reduce system pressure before healing attempts
"Docker_Essentials/docker_watchdog.sh" # heal containers with freed resources
"unRAID_Essentials/system_watchdog.sh" # reboot if all else fails — last line of defense
"Watchdogs/resource_watchdog.sh" # reduce system pressure before healing attempts
"Watchdogs/docker_watchdog.sh" # heal containers with freed resources
"Watchdogs/storage_watchdog.sh" # pool and appdata health — alert and remediate
"Watchdogs/system_watchdog.sh" # reboot if all else fails — last line of defense
)
WATCHDOG_ORCHESTRATOR_HEARTBEAT=true
@@ -317,7 +310,7 @@
"Docker_Essentials/downloaders_reset.sh" # clear stuck download states every 15min
)
# Shares synced every 15 minutes — defined per host in master_host*.conf.
# Shares synced every 15 minutes — defined per host in host*.conf.
# HOST1_CRITICAL_SYNC_SHARES / HOST2_CRITICAL_SYNC_SHARES
# Format: "/path/to/share" or "/path/to/share|profile-name"
# Order matters — Critical-Data first (auth stack), then Emby dirty sync.
@@ -326,7 +319,7 @@
# intermediate_sync_maintenance.sh runs every 4 hours — arr library sync, artwork fetch,
# and optional mid-day rsync for any shares that need sub-daily propagation.
# Schedule: 0 */4 * * *
# INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in master_host*.conf.
# INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in host*.conf.
INTERMEDIATE_RSYNC_ENABLED=true # set false to disable mid-day rsync without removing shares
INTERMEDIATE_MAINTENANCE_SCRIPTS=(
@@ -346,7 +339,7 @@
"Media/media_cleaner.sh media" # remove junk from media shares
#"Media/lidarr_cleanup.sh" # remove orphaned music files — enable when ready
#"Media/sonarr_cleanup.sh" # remove orphaned TV files — enable when ready
#"Media/radarr_cleanup.sh" # remove orphaned movie files — enable when ready
"Media/radarr_cleanup.sh" # remove orphaned movie files
"Media/lidarr_missing_art.sh" # fetch missing album/artist artwork (HOST1 only — self-guards)
"Media/radarr_tmdb_removed.sh" # remove movies dropped from TMDb
"Media/sonarr_tvdb_removed.sh" # remove series dropped from TVDB
@@ -360,7 +353,7 @@
DAILY_CONTAINER_UPDATES=true
# Media shares synced daily by daily_sync_maintenance.sh.
# Defined per-host in master_host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
# Defined per-host in host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
# Mesh model: every node pushes every media share. rsync has no --delete so pushes are additive.
# arr_sync (union) ensures all arr libraries converge first. arr_cleanup removes true orphans.
# Any node can download content to any share — it propagates to all nodes on the next cycle.
@@ -368,7 +361,7 @@
# These shares use DEFAULT_RSYNC_OPTS — no profile entry needed.
# For shares needing custom options or container stops — create a profile in RSYNC section.
# Personal encrypted shares defined per-host in master_host*.conf.
# Personal encrypted shares defined per-host in host*.conf.
# ZFS encrypted at dataset level — remote receives encrypted blocks, cannot read content.
# See README-Rsync_Setup.md for ZFS encryption setup before uncommenting.
@@ -390,7 +383,7 @@
# Set false to skip — docker_weekly_restart.sh still runs regardless.
WEEKLY_REMAINING_UPDATES=true
# Shares synced during the weekly maintenance window — defined per host in master_host*.conf.
# Shares synced during the weekly maintenance window — defined per host in host*.conf.
# HOST1_WEEKLY_SYNC_SHARES / HOST2_WEEKLY_SYNC_SHARES
# Containers stopped both sides before sync — full clean state guaranteed.
# Profiles drive container stops, excludes, and options — configure in RSYNC section.
@@ -473,8 +466,8 @@
# called by weekly_sync_maintenance.sh — full clean sync weekly
# critical-fallback — dirty sync — auth stays running both sides, WAL excluded
# called by critical_sync_maintenance.sh every 15min
# host1-appdata — HOST1 server-specific appdata — defined in master_host1.conf
# host2-appdata — HOST2 server-specific appdata — defined in master_host2.conf
# host1-appdata — HOST1 server-specific appdata — defined in host1.conf
# host2-appdata — HOST2 server-specific appdata — defined in host2.conf
# important-data — NextCloud + Postgres — NextCloud delayed start after Postgres
# emby — weekly clean sync — both Emby stopped, full mirror
# called by weekly_sync_maintenance.sh only — do NOT schedule separately
@@ -591,7 +584,7 @@
# Failover → start remote DDNS first (Tier 1)
# Handback → stop remote DDNS → rsync → start containers → start local DDNS last
#
# Per-host container lists and tier delays live in master_host*.conf.
# Per-host container lists and tier delays live in host*.conf.
# Shared settings (intervals, state file, thresholds) live here.
EXTERNAL_IP="8.8.8.8"
@@ -613,7 +606,7 @@
# ━━━ Downloaders Reset ━━━
# Runs every 15 minutes via CRITICAL_MAINTENANCE_SCRIPTS.
# Clears stuck states, purges old history, prepares each download client for a clean cycle.
# Per-host URLs and API keys live in master_host*.conf.
# Per-host URLs and API keys live in host*.conf.
DOWNLOADER_RETENTION_DAYS=7 # days — purge history older than this
# qBittorrent failsafe — removes torrents older than threshold regardless of ratio
@@ -623,7 +616,7 @@
# ━━━ Docker Daily Restart ━━━
# Containers restarted every day via DAILY_MAINTENANCE_SCRIPTS.
# Per-host lists live in master_host*.conf:
# Per-host lists live in host*.conf:
# HOST1_DAILY_RESTART_CONTAINERS
# HOST2_DAILY_RESTART_CONTAINERS
# detect_hosts() sets DAILY_RESTART_CONTAINERS to the correct host array at runtime.
@@ -631,7 +624,7 @@
# ━━━ Docker Weekly Restart ━━━
# Less critical services restarted weekly via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am).
# Containers already stopped for weekly sync — restart adds zero extra downtime.
# Per-host lists live in master_host*.conf:
# Per-host lists live in host*.conf:
# HOST1_WEEKLY_RESTART_CONTAINERS
# HOST2_WEEKLY_RESTART_CONTAINERS
@@ -653,7 +646,7 @@
# Dead containers — remove and restart
# Unexpected exits — non-zero exit code → restart
#
# All per-host container lists live in master_host*.conf:
# All per-host container lists live in host*.conf:
# HOST*_WATCHDOG_CONTAINERS — memory hard limits per container
# HOST*_WATCHDOG_CONTAINER_URLS — HTTP health check URLs
# HOST*_WATCHDOG_REQUIRED_CONTAINERS — must always be running
@@ -706,10 +699,42 @@
# Notification batching — one summary per cycle instead of one ping per event
WATCHDOG_BATCH_NOTIFY=true
# Appdata size monitoring — two-part catch-all for runaway growth and oversized log files.
#
# Part 1 — Growth rate (zero-config):
# Reads per-container dir totals each cycle via du, compares to previous cycle.
# Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused *.log scan
# inside that container. No per-container config required — new containers are covered
# automatically. Baseline built on first run after boot; growth detection starts cycle 2.
#
# Part 2 — Absolute log size:
# Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB across all appdata paths.
# Catches logs that have already stabilised at a large size and are no longer actively growing.
#
# Strike system (reuses existing watchdog infrastructure):
# Strike 1 — warn + notify: condition first detected
# Strike 2 — warn + escalated notify: still present next cycle
# Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
# If WATCHDOG_APPDATA_TRUNCATE_LOGS=true: truncate *.log files in-place, clear strikes
# If false: critical notify only, strikes held until condition resolves
# Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
#
# HOST*_WATCHDOG_APPDATA_SIZES (in host*.conf) suppresses growth warnings for a
# container until its dir exceeds the configured ceiling. Only needed when a container
# legitimately has large stable data and you want to guarantee it never triggers a false alarm.
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2 # flag containers growing more than this per cycle
WATCHDOG_APPDATA_LOG_MAX_GB=2 # flag *.log files exceeding this size (absolute)
WATCHDOG_APPDATA_TRUNCATE_LOGS=false # set true to auto-truncate oversized *.log files on action cycle
WATCHDOG_APPDATA_STRIKE_LIMIT=3 # cycles before action fires (matches existing watchdog pattern)
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db" # /tmp — resets on reboot ✅
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db" # /tmp — resets on reboot ✅
# ━━━ Docker Network Connect ━━━
# Ensures custom networks exist and connects containers at array start.
# Runs once via ARRAY_START_SCRIPTS — idempotent, safe to re-run.
# Container and network lists are host-specific — defined in master_host*.conf:
# Container and network lists are host-specific — defined in host*.conf:
# HOST1_NETWORK_CONNECT_CONTAINERS / HOST2_NETWORK_CONNECT_CONTAINERS
# HOST1_NETWORK_CONNECT_NETWORKS / HOST2_NETWORK_CONNECT_NETWORKS
@@ -816,13 +841,13 @@
PERMISSIONS_FILE_MODE="664" # files — group read/write, no execute
PERMISSIONS_OWNER="nobody:users"
# Share list defined per host in master_host*.conf:
# Share list defined per host in host*.conf:
# HOST1_MEDIA_PERMISSION_SHARES / HOST2_MEDIA_PERMISSION_SHARES
# ━━━ Media Cleaner ━━━
# Removes junk files from media shares — two profiles: anime and media.
# Called via DAILY_MAINTENANCE_SCRIPTS. Run manually: Media/media_cleaner.sh anime|media
# Folder lists defined per host in master_host*.conf:
# Folder lists defined per host in host*.conf:
# HOST1_ANIME_CLEAN_FOLDERS / HOST2_ANIME_CLEAN_FOLDERS
# HOST1_MEDIA_CLEAN_FOLDERS / HOST2_MEDIA_CLEAN_FOLDERS
@@ -904,7 +929,7 @@
# ━━━ Arr Cleanup ━━━
# Orphan file cleanup via Lidarr, Sonarr, and Radarr APIs.
# Per-host URLs, API keys, and path maps live in master_host*.conf.
# Per-host URLs, API keys, and path maps live in host*.conf.
# detect_hosts() selects correct host vars at runtime.
#
# API versions — update MAJOR version here when script is updated for a new arr version:
@@ -940,7 +965,7 @@
LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs
LIDARR_ART_RETRIES=2 # download retry attempts per image
LIDARR_ART_SLEEP_BETWEEN=0.2 # seconds between fanart.tv API calls
# HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in master_host*.conf
# HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in host*.conf
# Lidarr discovery settings (playback_aware_lidarr_discovery.sh)
LIDARR_DISCOVERY_THRESHOLD=70 # score to accept candidate (0-100)
@@ -1037,7 +1062,7 @@
# stalled — download stuck with no connections or progress
#
# Items newer than ARR_IMPORT_RECOVERY_AGE are skipped — gives arr time to retry first.
# Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in master_host*.conf.
# Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in host*.conf.
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
# matches cron interval — items eligible after one missed cycle
@@ -1057,7 +1082,7 @@
# Must exist before Emby starts so the symlink resolves correctly.
RAMDISK_PATH="/mnt/ramdisk_transcodes"
# Ramdisk size and flip thresholds — defined per host in master_host*.conf.
# Ramdisk size and flip thresholds — defined per host in host*.conf.
# All three are coupled — if size changes, thresholds must change with it.
# HOST1_RAMDISK_SIZE / HOST2_RAMDISK_SIZE
# HOST1_RAMDISK_WARN_GB / HOST2_RAMDISK_WARN_GB ← flip to SSD at this usage
@@ -1068,7 +1093,7 @@
# Must match the container path configured in Emby's Extra Parameters.
TRANSCODE_LINK="/mnt/ram-transcode"
# SSD fallback location — defined per host in master_host*.conf (cache path differs per server):
# SSD fallback location — defined per host in host*.conf (cache path differs per server):
# HOST1_TRANSCODE_SSD / HOST2_TRANSCODE_SSD
# Minimum free GB on SSD before allowing flip from ramdisk to SSD.
@@ -1098,7 +1123,7 @@
# Format: "ContainerName|URL|APIKey|Type" — Type: emby | jellyfin | plex
# Entries with placeholder API keys are skipped automatically.
# ⚠️ Tdarr does NOT belong here — keep Tdarr on SSD, not ramdisk.
# Defined per host in master_host*.conf — Emby container names and keys differ per server:
# Defined per host in host*.conf — Emby container names and keys differ per server:
# HOST1_TRANSCODE_SERVERS / HOST2_TRANSCODE_SERVERS
# ==============================================================================================
@@ -1108,7 +1133,7 @@
# ━━━ Certificate Monitor ━━━
# Checks SSL certificate expiry via direct openssl connection — no NPM dependency.
# Checks the actual certificate served by each domain, not what NPM thinks it has.
# Domains defined per host in master_host*.conf — each server monitors its own domains:
# Domains defined per host in host*.conf — each server monitors its own domains:
# HOST1_CERT_MONITOR_DOMAINS / HOST2_CERT_MONITOR_DOMAINS
CERT_WARN_DAYS=30 # warn when cert expires within this many days
CERT_CRIT_DAYS=7 # critical alert within this many days
@@ -1117,7 +1142,7 @@
# ━━━ Backup Verify ━━━
# Verifies rsync mirror health by comparing random file checksums between servers.
# Catches silent corruption or incomplete syncs that rsync itself wouldn't detect.
# Defined per host in master_host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically:
# Defined per host in host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically:
# HOST1_BACKUP_VERIFY_SHARES / HOST2_BACKUP_VERIFY_SHARES
BACKUP_VERIFY_SAMPLE=10 # random files to check per share
BACKUP_VERIFY_MIN_SIZE=1M # minimum file size to include in sample
@@ -1128,7 +1153,7 @@
# (hot/max/hotssd/maxssd) — these vars are fallback only if dynamix.cfg not found.
SMART_TEMP_WARN=45 # fallback — Celsius warn threshold
SMART_TEMP_CRIT=55 # fallback — Celsius critical threshold
# Drives to ignore defined per host in master_host*.conf — hardware is server-specific:
# Drives to ignore defined per host in host*.conf — hardware is server-specific:
# HOST1_SMART_IGNORE_DRIVES / HOST2_SMART_IGNORE_DRIVES
# ━━━ ZFS Memory Snapshot ━━━
@@ -1138,7 +1163,7 @@
ZFS_REPORT_FREE_WARN_GB=10 # warn if less than this GB free RAM
ZFS_REPORT_AVAIL_WARN_GB=20 # warn if less than this GB available on ZFS pool
ZFS_REPORT_DOCKER_TOP=10 # how many top Docker containers to show by memory
# Pool ignore list defined per host in master_host*.conf — pool names are server-specific:
# Pool ignore list defined per host in host*.conf — pool names are server-specific:
# HOST1_ZFS_REPORT_IGNORE_POOLS / HOST2_ZFS_REPORT_IGNORE_POOLS
# ━━━ Bandwidth Monitor ━━━
@@ -1168,7 +1193,7 @@
# ━━━ Emby Session Report ━━━
# Weekly Emby usage statistics via API — no persistent writes, queries fresh each run.
# URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in master_host*.conf.
# URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in host*.conf.
EMBY_REPORT_DAYS=7 # days to include in the report period
EMBY_REPORT_TOP_N=10 # number of top content items to show
@@ -1184,8 +1209,8 @@
# Level 3 (hard) — docker stop optional containers, signal docker_watchdog to defer
#
# ── PER-HOST CONTAINER LISTS ──────────────────────────────────────────────────────────────────
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (in master_host*.conf)
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (in master_host*.conf)
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (in host*.conf)
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (in host*.conf)
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
@@ -1335,9 +1360,9 @@
SYS_WATCHDOG_MDSTAT_ERROR_LIMIT=5 # new errors in one cycle before acting
# ━━━ Check Toggles ━━━
# Per-host — moved to master_host*.conf
# Per-host — moved to host*.conf
# Different servers may have different hardware, NICs, and check requirements
# See HOST*_SYS_WATCHDOG_CHECK_* in master_host*.conf
# See HOST*_SYS_WATCHDOG_CHECK_* in host*.conf
# ━━━ Abort Toggles ━━━
# Conditions that prevent reboot even when a threshold is hit.