FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Watchdogs/ folder + host conf rename

Browse Source 
Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
This commit is contained in:
Gmer4Lfe
2026-05-22 17:08:36 -04:00
parent 9ee8af1a71
commit 95151c2278
73 changed files with 904 additions and 328 deletions
Download Patch File Download Diff File Expand all files Collapse all files
user_script_plug-in.sh
+24 -24
View File
@@ -44,7 +44,7 @@
# ── PATHS ─────────────────────────────────────────────────────────────────────────────────────
# Scripts: /mnt/user/appdata/unraid_scripts/
# Configuration: /mnt/user/appdata/unraid_scripts/master.conf
# Per-host: /mnt/user/appdata/unraid_scripts/master_host1.conf (or master_host2.conf)
# Per-host: /mnt/user/appdata/unraid_scripts/host1.conf (or host2.conf)
# Library: /mnt/user/appdata/unraid_scripts/common.sh
#
# ── PLUGIN SETTINGS (apply to every entry) ────────────────────────────────────────────────────
@@ -168,9 +168,9 @@
# Overlap protection: acquire_lock() exits immediately if a prior cycle is still running —
# prevents pile-up when a restart attempt or daemon check takes longer than 60 seconds.
#
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/watchdog_orchestrator.sh
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/watchdog_orchestrator.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/watchdog_orchestrator.sh --status
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_orchestrator.sh
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_orchestrator.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_orchestrator.sh --status
# ── TRANSCODE MANAGEMENT ──────────────────────────────────────────────────────────────────────
@@ -570,7 +570,7 @@
# docker_watchdog.sh — two-tier container self-healing monitor (single-pass)
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
#
# Tier 1 — explicit per-container (configured in master_host*.conf):
# Tier 1 — explicit per-container (configured in host*.conf):
# Memory hard limits: immediate restart when exceeded — no strikes, no waiting
# CPU strike system: 2 consecutive cycles above HARD_CPU_THRESHOLD → restart
# HTTP health checks: curl to configured URL — 2 consecutive failures → restart
@@ -589,8 +589,8 @@
# RAM emergency: reads mem_shutdown_active from system_watchdog state file, defers all restarts
# Silent on clean cycles — only outputs events and hourly heartbeat
#
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --dry-run
# watchdog_skip_list_manager.sh — view and manage the container skip list
# When docker_watchdog restarts the same container 3 times in 1hr → skip-listed.
@@ -605,11 +605,11 @@
# docker start ContainerName → confirm fix works before handing back to watchdog
# watchdog resumes normal monitoring on next cycle automatically
#
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --status
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear ContainerName
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear ContainerName --force
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear-all
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --status
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --clear ContainerName
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --clear ContainerName --force
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --clear-all
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --dry-run
# docker_container_stop.sh — stop all running containers sequentially with verification
# Called by array_stopping.sh as the final step in the planned shutdown sequence.
@@ -631,7 +631,7 @@
# Running → docker restart (graceful). Stopped → left stopped (state respected). Missing → skip.
# Dependency ordering via WATCHDOG_DEPENDENCIES — databases before applications.
# Restart verification: checks container still up after settle period, notifies if not.
# Configured via HOST*_DAILY_RESTART_CONTAINERS (master_host*.conf):
# Configured via HOST*_DAILY_RESTART_CONTAINERS (host*.conf):
# NginxProxyManager, Authelia, Dispatcharr, Dispatcharr-Basic, ErsatzTV-Emby
#
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh --dry-run
@@ -642,7 +642,7 @@
# Called by weekly_sync_maintenance.sh after sync completes and containers are back up.
# Targets services that benefit from weekly clean start but don't stop for the sync itself.
# Same rules as daily: running→restart, stopped→leave, missing→skip.
# Configured via HOST*_WEEKLY_RESTART_CONTAINERS (master_host*.conf):
# Configured via HOST*_WEEKLY_RESTART_CONTAINERS (host*.conf):
# NextCloud, AdGuard-Home, Immich
#
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh --dry-run
@@ -711,7 +711,7 @@
# system_watchdog.sh — three-tier server stability last-resort watchdog (single-pass)
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
# All 18 checks independently toggleable per host in master_host*.conf.
# All 18 checks independently toggleable per host in host*.conf.
#
# Tier 1 CRITICAL — bypass ALL strikes, reboot immediately:
# Docker daemon hung → attempt rc.docker restart → still hung → reboot
@@ -733,8 +733,8 @@
# Reboot loop protection: N reboots in X hours → shutdown instead.
# State file heartbeat: writes watchdog_cycle=N every cycle (docker_watchdog stale guard).
#
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --dry-run
# resource_watchdog.sh — three-level pressure reduction layer (single-pass, called by watchdog_orchestrator)
# Reduces system load intelligently BEFORE docker_watchdog attempts container restarts.
@@ -748,11 +748,11 @@
# Recovery: pressure must stay below current threshold for RW_RECOVER_CYCLES consecutive runs
# before de-escalating. One level at a time — prevents flip-flopping.
# Coordination: at Level 3 writes mem_shutdown_active=true → docker_watchdog defers all restarts.
# HOST*_RW_PAUSE_CONTAINERS and HOST*_RW_STOP_CONTAINERS configured per host in master_host*.conf.
# HOST*_RW_PAUSE_CONTAINERS and HOST*_RW_STOP_CONTAINERS configured per host in host*.conf.
#
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh
# inotify_tuning.sh — raise Linux inotify kernel limits at array start
# Called by array_started.sh FIRST — must run before containers start (they inherit limits).
@@ -858,7 +858,7 @@
# Run BEFORE arr cleanup — removes non-media files that would otherwise appear as orphans.
# Patterns: *.sfv *.md5 *.sha1 *.nfo *.url *.lnk *.rar *.zip *.info *.torrent
# *.sample* *.proof* *sync-conflict* *.scr *.exe *.srr *.log *.json
# Two profiles with separate folder lists (configured in master_host*.conf):
# Two profiles with separate folder lists (configured in host*.conf):
# anime HOST*_ANIME_CLEAN_FOLDERS — anime share folders
# media HOST*_MEDIA_CLEAN_FOLDERS — Movies, Tv_Shows, Music, Sports etc.
# ALWAYS --dry-run when adding new patterns or folders — verify before committing.
@@ -1080,7 +1080,7 @@
# Catches: renewed-but-not-reloaded (nginx not reloaded after certbot renewal),
# wrong cert served, chain issues visible externally but not internally.
# If a user would see a certificate error in their browser, this catches it first.
# Configured via HOST*_CERT_MONITOR_DOMAINS in master_host*.conf.
# Configured via HOST*_CERT_MONITOR_DOMAINS in host*.conf.
# Thresholds: > 30 days = silent, <= 30 = warning, <= CERT_CRIT_DAYS (7) = urgent.
#
# bash /mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh --status
@@ -1295,7 +1295,7 @@
# Orchestrators/array_stopping.sh (single entry — graceful shutdown)
#
# * * * * * every minute:
# Orchestrators/watchdog_orchestrator.sh
# Watchdogs/watchdog_orchestrator.sh
#
# */3 * * * * every 3 minutes:
# Orchestrators/transcode_management.sh