Watchdogs/ folder + host conf rename

Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
This commit is contained in:
Gmer4Lfe
2026-05-22 17:08:36 -04:00
parent 9ee8af1a71
commit 95151c2278
73 changed files with 904 additions and 328 deletions
+11 -11
View File
@@ -10,7 +10,7 @@ For per-script detail see the script headers directly.
## ━━━ WATCHDOG CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ WATCHDOG CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All watchdog configuration lives in `master_host*.conf` (per-container lists) and All watchdog configuration lives in `host*.conf` (per-container lists) and
`master.conf` (shared thresholds and toggles). `detect_hosts()` aliases all `master.conf` (shared thresholds and toggles). `detect_hosts()` aliases all
`HOST1_` / `HOST2_` prefixed vars to their unprefixed names at runtime — scripts `HOST1_` / `HOST2_` prefixed vars to their unprefixed names at runtime — scripts
always read the right values for the server they're running on. always read the right values for the server they're running on.
@@ -20,7 +20,7 @@ always read the right values for the server they're running on.
### ── Memory Hard Limits ──────────────────────────────────────────────────────── ### ── Memory Hard Limits ────────────────────────────────────────────────────────
```bash ```bash
# master_host1.conf # host1.conf
# Format: "ContainerName:LimitInMB" # Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes. # Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
# #
@@ -71,7 +71,7 @@ CPU_FAIL_LIMIT=2 # consecutive strikes before restart
### ── HTTP Health Checks ──────────────────────────────────────────────────────── ### ── HTTP Health Checks ────────────────────────────────────────────────────────
```bash ```bash
# master_host1.conf # host1.conf
# Format: "ContainerName:http://host:port/optional-path" # Format: "ContainerName:http://host:port/optional-path"
# Hits the actual service endpoint on every watchdog cycle. # Hits the actual service endpoint on every watchdog cycle.
# "Container running" and "service responding" are not the same thing. # "Container running" and "service responding" are not the same thing.
@@ -98,7 +98,7 @@ RESP_FAIL_LIMIT=2 # consecutive failures before restart
### ── Required Containers ─────────────────────────────────────────────────────── ### ── Required Containers ───────────────────────────────────────────────────────
```bash ```bash
# master_host1.conf # host1.conf
# Containers that must always be running. # Containers that must always be running.
# Found stopped → watchdog attempts restart every cycle until running or skip-listed. # Found stopped → watchdog attempts restart every cycle until running or skip-listed.
# Uses the STRIKE SYSTEM — one miss might be mid-restart. # Uses the STRIKE SYSTEM — one miss might be mid-restart.
@@ -119,7 +119,7 @@ HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
### ── Dependency Ordering ─────────────────────────────────────────────────────── ### ── Dependency Ordering ───────────────────────────────────────────────────────
```bash ```bash
# master_host1.conf # host1.conf
# Format: "DependentContainer:dependency1 dependency2" # Format: "DependentContainer:dependency1 dependency2"
# Multiple dependencies space-separated. All must be running before dependent restarts. # Multiple dependencies space-separated. All must be running before dependent restarts.
# #
@@ -245,7 +245,7 @@ WATCHDOG_BATCH_NOTIFY=true
### ── Daily Restart List ─────────────────────────────────────────────────────── ### ── Daily Restart List ───────────────────────────────────────────────────────
```bash ```bash
# master_host1.conf # host1.conf
# Restarted every night at 1am via daily_sync_maintenance.sh. # Restarted every night at 1am via daily_sync_maintenance.sh.
# #
# Good candidates: # Good candidates:
@@ -271,7 +271,7 @@ their images updated daily before the restart. Add a container once, it gets bot
### ── Weekly Restart List ────────────────────────────────────────────────────── ### ── Weekly Restart List ──────────────────────────────────────────────────────
```bash ```bash
# master_host1.conf # host1.conf
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh. # Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack). # Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
# #
@@ -291,7 +291,7 @@ HOST1_WEEKLY_RESTART_CONTAINERS=(
## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```bash ```bash
# master_host1.conf # host1.conf
# Networks to ensure exist + containers to connect to each network. # Networks to ensure exist + containers to connect to each network.
# Many-to-many: every container connects to every network listed. # Many-to-many: every container connects to every network listed.
# #
@@ -339,7 +339,7 @@ Everything gets updated at least once per week with no explicit configuration.
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
### ── master_host*.conf ──────────────────────────────────────────────────────── ### ── host*.conf ────────────────────────────────────────────────────────
```bash ```bash
# Per-host — varies between HOST1 and HOST2 # Per-host — varies between HOST1 and HOST2
@@ -425,11 +425,11 @@ SLEEP=5 # seconds between retry attempts
### ── Adding a Container to Monitoring ──────────────────────────────────────── ### ── Adding a Container to Monitoring ────────────────────────────────────────
Adding a container is purely additive — add the relevant lines to `master_host*.conf`. Adding a container is purely additive — add the relevant lines to `host*.conf`.
No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle. No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
```bash ```bash
# master_host1.conf — example: adding "MyApp" to full Tier 1 + daily restarts # host1.conf — example: adding "MyApp" to full Tier 1 + daily restarts
# 1. Memory hard limit — size at ~150-200% of normal peak (check: docker stats MyApp) # 1. Memory hard limit — size at ~150-200% of normal peak (check: docker stats MyApp)
HOST1_WATCHDOG_CONTAINERS=( HOST1_WATCHDOG_CONTAINERS=(
@@ -275,7 +275,7 @@ Array starts
reads system_watchdog state (RAM emergency deferral) reads system_watchdog state (RAM emergency deferral)
│ (on skip list event → operator uses) │ (on skip list event → operator uses)
└── Tools/watchdog_skip_list_manager.sh └── Watchdogs/watchdog_skip_list_manager.sh
inspect state, clear after fixing root cause inspect state, clear after fixing root cause
Daily maintenance window (1am): Daily maintenance window (1am):
+2 -2
View File
@@ -60,7 +60,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_DAILY_RESTART_CONTAINERS # HOST*_DAILY_RESTART_CONTAINERS
# Containers restarted nightly. Also used by docker_update.sh normal mode # Containers restarted nightly. Also used by docker_update.sh normal mode
@@ -126,7 +126,7 @@ detect_hosts
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart" warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf" warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
exit 0 exit 0
fi fi
+3 -3
View File
@@ -78,7 +78,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_NETWORK_CONNECT_NETWORKS # HOST*_NETWORK_CONNECT_NETWORKS
# Networks to ensure exist at array start. Aliased by detect_hosts() → # Networks to ensure exist at array start. Aliased by detect_hosts() →
@@ -144,13 +144,13 @@ fi
# Empty array guards # Empty array guards
if [[ ${#NETWORK_CONNECT_NETWORKS[@]} -eq 0 ]]; then if [[ ${#NETWORK_CONNECT_NETWORKS[@]} -eq 0 ]]; then
warn "NETWORK_CONNECT_NETWORKS is empty for $MY_ID — nothing to do" warn "NETWORK_CONNECT_NETWORKS is empty for $MY_ID — nothing to do"
warn "Check HOST*_NETWORK_CONNECT_NETWORKS in master_host*.conf" warn "Check HOST*_NETWORK_CONNECT_NETWORKS in host*.conf"
exit 0 exit 0
fi fi
if [[ ${#NETWORK_CONNECT_CONTAINERS[@]} -eq 0 ]]; then if [[ ${#NETWORK_CONNECT_CONTAINERS[@]} -eq 0 ]]; then
warn "NETWORK_CONNECT_CONTAINERS is empty for $MY_ID — no containers to connect" warn "NETWORK_CONNECT_CONTAINERS is empty for $MY_ID — no containers to connect"
warn "Check HOST*_NETWORK_CONNECT_CONTAINERS in master_host*.conf" warn "Check HOST*_NETWORK_CONNECT_CONTAINERS in host*.conf"
exit 0 exit 0
fi fi
+2 -2
View File
@@ -95,7 +95,7 @@
# Container names for emby and critical-data profiles — excluded from # Container names for emby and critical-data profiles — excluded from
# remainder mode (already updated by the weekly sync window) # remainder mode (already updated by the weekly sync window)
# #
# master_host*.conf # host*.conf
# #
# HOST*_DAILY_RESTART_CONTAINERS # HOST*_DAILY_RESTART_CONTAINERS
# Containers updated in normal mode. Aliased by detect_hosts() → # Containers updated in normal mode. Aliased by detect_hosts() →
@@ -205,7 +205,7 @@ else
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update" warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf" warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
exit 0 exit 0
fi fi
+1 -1
View File
@@ -52,7 +52,7 @@
# Enable or disable this script. (default: true) # Enable or disable this script. (default: true)
# To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS. # To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
# #
# master_host*.conf # host*.conf
# #
# HOST*_DAILY_RESTART_CONTAINERS # HOST*_DAILY_RESTART_CONTAINERS
# Excluded from this script — already updated daily. Aliased by # Excluded from this script — already updated daily. Aliased by
+2 -2
View File
@@ -43,7 +43,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_WEEKLY_RESTART_CONTAINERS # HOST*_WEEKLY_RESTART_CONTAINERS
# Containers restarted weekly. Aliased by detect_hosts() → # Containers restarted weekly. Aliased by detect_hosts() →
@@ -108,7 +108,7 @@ detect_hosts
if [[ ${#WEEKLY_RESTART_CONTAINERS[@]} -eq 0 ]]; then if [[ ${#WEEKLY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
warn "WEEKLY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart" warn "WEEKLY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
warn "Check HOST*_WEEKLY_RESTART_CONTAINERS in master_host*.conf" warn "Check HOST*_WEEKLY_RESTART_CONTAINERS in host*.conf"
exit 0 exit 0
fi fi
+2 -2
View File
@@ -83,7 +83,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_SLSKD_URL / HOST*_SLSKD_API_KEY / HOST*_SLSKD_FAILED_IMPORTS_DIR # HOST*_SLSKD_URL / HOST*_SLSKD_API_KEY / HOST*_SLSKD_FAILED_IMPORTS_DIR
# slskd connection and failed imports path. Aliased by detect_hosts() # slskd connection and failed imports path. Aliased by detect_hosts()
@@ -567,7 +567,7 @@ if [[ -n "$QBIT_URL" ]] && [[ -n "$QBIT_USERNAME" ]]; then
if [[ -z "$QBIT_COOKIE" ]]; then if [[ -z "$QBIT_COOKIE" ]]; then
error "Failed to authenticate with qBittorrent — check QBIT_USERNAME/PASSWORD" error "Failed to authenticate with qBittorrent — check QBIT_USERNAME/PASSWORD"
notify "qBittorrent auth failed on $(hostname) — check credentials in master_host*.conf" "Downloaders Reset" "warning" notify "qBittorrent auth failed on $(hostname) — check credentials in host*.conf" "Downloaders Reset" "warning"
((TOTAL_FAIL++)) ((TOTAL_FAIL++))
else else
TORRENTS=$(curl -sf --max-time 15 \ TORRENTS=$(curl -sf --max-time 15 \
+4 -4
View File
@@ -80,7 +80,7 @@ container start time. At 3 strikes × 30s + ~2min rsync + ~1min container start:
--- ---
## ━━━ CONFIGURATION — master_host*.conf ━━━ ## ━━━ CONFIGURATION — host*.conf ━━━
``` ```
HOST*_DDNS_CONTAINERS=("...") HOST*_DDNS_CONTAINERS=("...")
@@ -181,7 +181,7 @@ FALLBACK_TEST_HANDBACK_WAIT=300
--- ---
### master_host2.conf — HOST2 covering HOST1 ### host2.conf — HOST2 covering HOST1
```bash ```bash
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS") HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
@@ -259,7 +259,7 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
--- ---
### master_host1.conf — HOST1 covering HOST2 ### host1.conf — HOST1 covering HOST2
```bash ```bash
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS") HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
@@ -486,7 +486,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
1. Create the container on the covering server (stopped), with volume mounts pointing at 1. Create the container on the covering server (stopped), with volume mounts pointing at
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server) the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in master_host*.conf 2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf
in the appropriate tier position (dependency ordering — databases before apps) in the appropriate tier position (dependency ordering — databases before apps)
3. Verify: `fallback.sh --status` shows the container in the expected tier list 3. Verify: `fallback.sh --status` shows the container in the expected tier list
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid 4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
+2 -2
View File
@@ -219,8 +219,8 @@ determines which server is local and which is remote at runtime, then selects th
container arrays and tier delays from config via MY_ID. container arrays and tier delays from config via MY_ID.
``` ```
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in master_host2.conf) HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf)
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in master_host1.conf) HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf)
``` ```
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
+1 -1
View File
@@ -132,7 +132,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_DDNS_CONTAINERS # HOST*_DDNS_CONTAINERS
# DDNS containers this host manages — stopped on internet loss, started # DDNS containers this host manages — stopped on internet loss, started
+1 -1
View File
@@ -289,7 +289,7 @@ fi
# Tier 1 containers configured # Tier 1 containers configured
if [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then if [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then
error "No Tier 1 containers configured for $MY_ID$REMOTE_ID" error "No Tier 1 containers configured for $MY_ID$REMOTE_ID"
error "Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in master_host*.conf" error "Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in host*.conf"
phase_fail "Pre-flight" phase_fail "Pre-flight"
exit 1 exit 1
fi fi
+7 -7
View File
@@ -228,7 +228,7 @@ LIDARR_DISCOVERY_REJECT_COOLDOWN=30 # days before re-evaluating a Stage 2 reje
LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db" LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db"
``` ```
Requires `HOST*_LASTFM_API_KEY` in `master_host*.conf`. Requires `HOST*_LASTFM_API_KEY` in `host*.conf`.
--- ---
@@ -246,7 +246,7 @@ RADARR_DISCOVERY_SEED_LIBRARIES=("Movies") # Emby libraries to draw seeds from
RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db" RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db"
``` ```
Requires `HOST*_TMDB_API_KEY` in `master_host*.conf`. Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
--- ---
@@ -266,7 +266,7 @@ SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db"
# SONARR_EMBY_LIBRARIES is shared with emby_to_sonarr_sync — see Orchestrator Job Order # SONARR_EMBY_LIBRARIES is shared with emby_to_sonarr_sync — see Orchestrator Job Order
``` ```
Requires `HOST*_TMDB_API_KEY` in `master_host*.conf`. Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
> **MONITOR_MODE note:** Use `"all"` (default) to have Sonarr search all existing seasons > **MONITOR_MODE note:** Use `"all"` (default) to have Sonarr search all existing seasons
> after adding a show. `"future"` only marks upcoming seasons as monitored — shows where > after adding a show. `"future"` only marks upcoming seasons as monitored — shows where
@@ -289,9 +289,9 @@ MEDIA_MAINTENANCE_JOBS=(
--- ---
## ━━━ CONFIGURATION — master_host*.conf ━━━ ## ━━━ CONFIGURATION — host*.conf ━━━
### master_host1.conf ### host1.conf
```bash ```bash
# Shares this server applies permissions to # Shares this server applies permissions to
@@ -429,7 +429,7 @@ cp radarr_cleanup.sh readarr_cleanup.sh
# 2. Replace RADARR_ prefix with READARR_ throughout # 2. Replace RADARR_ prefix with READARR_ throughout
# Update API endpoint, tracked file API path, extension list, protected patterns # Update API endpoint, tracked file API path, extension list, protected patterns
# 3. Add to master_host*.conf # 3. Add to host*.conf
HOST1_READARR_URL="http://192.168.50.2:8787" HOST1_READARR_URL="http://192.168.50.2:8787"
HOST1_READARR_API_KEY="your-api-key" HOST1_READARR_API_KEY="your-api-key"
HOST1_READARR_BOOKS_ROOT="/mnt/user/Books" HOST1_READARR_BOOKS_ROOT="/mnt/user/Books"
@@ -534,7 +534,7 @@ If ghosts persist:
1. Is Emby's API responding? 1. Is Emby's API responding?
curl -s "http://[emby-ip]:8096/System/Info/Public" curl -s "http://[emby-ip]:8096/System/Info/Public"
2. Is EMBY_URL / EMBY_API_KEY correct in master_host*.conf? 2. Is EMBY_URL / EMBY_API_KEY correct in host*.conf?
Run: sonarr_cleanup.sh --status (shows Emby config) Run: sonarr_cleanup.sh --status (shows Emby config)
3. Trigger manually in Emby: 3. Trigger manually in Emby:
+1 -1
View File
@@ -78,7 +78,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_LIDARR_URL / HOST*_LIDARR_API_KEY — local Lidarr (aliased by detect_hosts) # HOST*_LIDARR_URL / HOST*_LIDARR_API_KEY — local Lidarr (aliased by detect_hosts)
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY — local Sonarr # HOST*_SONARR_URL / HOST*_SONARR_API_KEY — local Sonarr
+1 -1
View File
@@ -54,7 +54,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_RECOVERY # HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_RECOVERY
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_RECOVERY # HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_RECOVERY
+1 -1
View File
@@ -58,7 +58,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY / HOST1_LIDARR_MUSIC_ROOT # HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY / HOST1_LIDARR_MUSIC_ROOT
# HOST1_LIDARR_PATH_MAP — container path → host path translation # HOST1_LIDARR_PATH_MAP — container path → host path translation
+1 -1
View File
@@ -42,7 +42,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY # HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY
# Aliased by detect_hosts() — script uses LIDARR_URL / LIDARR_API_KEY # Aliased by detect_hosts() — script uses LIDARR_URL / LIDARR_API_KEY
+29 -16
View File
@@ -5,10 +5,10 @@
# #
# PURPOSE # PURPOSE
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# Remove junk files from media shares using configurable file patterns. Two # Remove scene debris, tool artifacts, and unsafe files from media shares
# profiles — anime and media — each with their own folder list and patterns. # before the orphan scan — so arr cleanup only encounters actual media files.
# Runs second in the daily maintenance window, after permissions and before # Runs second in the daily window, after permissions and before arr cleanup.
# arr cleanup, so orphan detection only encounters actual media files. # Two profiles — anime and media — each with their own folder list and patterns.
# #
# ============================================================================================== # ==============================================================================================
# OPERATIONAL MODEL # OPERATIONAL MODEL
@@ -18,15 +18,29 @@
# anime — cleans ANIME_CLEAN_FOLDERS using ANIME_FILE_PATTERNS # anime — cleans ANIME_CLEAN_FOLDERS using ANIME_FILE_PATTERNS
# media — cleans MEDIA_CLEAN_FOLDERS using MEDIA_FILE_PATTERNS (adds *.iso *.lrc) # media — cleans MEDIA_CLEAN_FOLDERS using MEDIA_FILE_PATTERNS (adds *.iso *.lrc)
# #
# Removes junk left by download clients, scene releases, and tools: # Scene debris — left by scene releases and download clients:
# *.sfv *.md5 *.sha1 — checksum files — useless post-download # *.sfv *.md5 *.sha1 — checksums — useless post-download
# *.nfo *.url *.lnk — scene info files — not needed in media library # *.nzb *.torrent — download files left by clients
# *.rar *.zip — archives — source files not needed after extraction # *.url *.lnk *.info *.diz — scene metadata
# *.sample* *.proof* — scene samples — never needed # *.sample* *.proof* — scene samples — never needed in library
# *sync-conflict* — Syncthing conflict files # *sync-conflict* — Syncthing conflict copies
# *.scr *.exe — executables — should never be in a media folder # *.rar *.zip *.7z *.ace — archives — source not needed after extraction
# *.torrent torrent files left by download clients # *.r00-*.r09 *.srrmulti-part rar segments and repair files
# *.log *.json — tool output files # *.001 *.002 *.003 — split archive parts
# *.gz *.tar *.bz2 — linux archives
#
# Tool artifacts — incomplete or stale files from download clients:
# *.!ut *.!qB — uTorrent / qBittorrent incomplete markers
# *.crdownload *.opdownload — Chrome / Opera incomplete downloads
# *.part — partial download files
#
# Unsafe files — executables that should never appear in a media folder:
# *.exe *.scr *.com — Windows executables
# *.bat *.cmd *.vbs *.ps1 — Windows scripts
# *.msi *.dll *.sys — Windows system files
# *.sh — shell scripts in media folders = suspicious
#
# Media profile also removes: *.iso *.lrc
# #
# ============================================================================================== # ==============================================================================================
# OPERATIONAL SAFEGUARDS # OPERATIONAL SAFEGUARDS
@@ -37,13 +51,12 @@
# Empty array guards — warns and exits cleanly if no folders or patterns configured # Empty array guards — warns and exits cleanly if no folders or patterns configured
# Folder existence — skips missing folders with warning, continues others # Folder existence — skips missing folders with warning, continues others
# validate_unraid_cmd — notify script validated before use # validate_unraid_cmd — notify script validated before use
# Silent by default — only problems and removals produce output
# #
# ============================================================================================== # ==============================================================================================
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_ANIME_CLEAN_FOLDERS — folders cleaned by the anime profile on this host # HOST*_ANIME_CLEAN_FOLDERS — folders cleaned by the anime profile on this host
# HOST*_MEDIA_CLEAN_FOLDERS — folders cleaned by the media profile on this host # HOST*_MEDIA_CLEAN_FOLDERS — folders cleaned by the media profile on this host
@@ -128,7 +141,7 @@ esac
# Empty array guards # Empty array guards
if [[ ${#CLEAN_FOLDERS[@]} -eq 0 ]]; then if [[ ${#CLEAN_FOLDERS[@]} -eq 0 ]]; then
warn "No folders configured for profile '$PROFILE' on $MY_ID" warn "No folders configured for profile '$PROFILE' on $MY_ID"
warn "Check HOST*_${PROFILE^^}_CLEAN_FOLDERS in master_host*.conf" warn "Check HOST*_${PROFILE^^}_CLEAN_FOLDERS in host*.conf"
exit 0 exit 0
fi fi
+2 -2
View File
@@ -34,7 +34,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_MEDIA_PERMISSION_SHARES — shares this host applies permissions to # HOST*_MEDIA_PERMISSION_SHARES — shares this host applies permissions to
# Aliased by detect_hosts() — script uses MEDIA_PERMISSION_SHARES # Aliased by detect_hosts() — script uses MEDIA_PERMISSION_SHARES
@@ -83,7 +83,7 @@ detect_hosts
# Empty array guard # Empty array guard
if [[ ${#MEDIA_PERMISSION_SHARES[@]} -eq 0 ]]; then if [[ ${#MEDIA_PERMISSION_SHARES[@]} -eq 0 ]]; then
warn "MEDIA_PERMISSION_SHARES is empty for $MY_ID — nothing to do" warn "MEDIA_PERMISSION_SHARES is empty for $MY_ID — nothing to do"
warn "Check HOST*_MEDIA_PERMISSION_SHARES in master_host*.conf" warn "Check HOST*_MEDIA_PERMISSION_SHARES in host*.conf"
exit 0 exit 0
fi fi
+4 -4
View File
@@ -57,7 +57,7 @@
# ============================================================================================== # ==============================================================================================
# #
# Last.fm API key — required for both stages # Last.fm API key — required for both stages
# Configure HOST*_LASTFM_API_KEY in master_host*.conf # Configure HOST*_LASTFM_API_KEY in host*.conf
# #
# ============================================================================================== # ==============================================================================================
# CONFIGURATION (master.conf) # CONFIGURATION (master.conf)
@@ -135,18 +135,18 @@ if ! command -v jq >/dev/null 2>&1; then
fi fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf" error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then
error "LIDARR_URL / LIDARR_API_KEY not configured — check master_host*.conf" error "LIDARR_URL / LIDARR_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${LASTFM_API_KEY:-}" ]]; then if [[ -z "${LASTFM_API_KEY:-}" ]]; then
error "LASTFM_API_KEY not configured — required for discovery" error "LASTFM_API_KEY not configured — required for discovery"
error "Configure HOST*_LASTFM_API_KEY in master_host*.conf" error "Configure HOST*_LASTFM_API_KEY in host*.conf"
exit 1 exit 1
fi fi
+4 -4
View File
@@ -55,7 +55,7 @@
# ============================================================================================== # ==============================================================================================
# #
# TMDB API key — required for Stage 2 recommendations # TMDB API key — required for Stage 2 recommendations
# Configure HOST*_TMDB_API_KEY in master_host*.conf # Configure HOST*_TMDB_API_KEY in host*.conf
# Free key at: https://www.themoviedb.org/settings/api # Free key at: https://www.themoviedb.org/settings/api
# #
# ============================================================================================== # ==============================================================================================
@@ -136,19 +136,19 @@ if ! command -v jq >/dev/null 2>&1; then
fi fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf" error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${RADARR_URL:-}" || -z "${RADARR_API_KEY:-}" ]]; then if [[ -z "${RADARR_URL:-}" || -z "${RADARR_API_KEY:-}" ]]; then
error "RADARR_URL / RADARR_API_KEY not configured — check master_host*.conf" error "RADARR_URL / RADARR_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${TMDB_API_KEY:-}" ]]; then if [[ -z "${TMDB_API_KEY:-}" ]]; then
error "TMDB_API_KEY not configured — required for discovery" error "TMDB_API_KEY not configured — required for discovery"
error "Get a free key at https://www.themoviedb.org/settings/api" error "Get a free key at https://www.themoviedb.org/settings/api"
error "Configure HOST*_TMDB_API_KEY in master_host*.conf" error "Configure HOST*_TMDB_API_KEY in host*.conf"
exit 1 exit 1
fi fi
+4 -4
View File
@@ -62,7 +62,7 @@
# ============================================================================================== # ==============================================================================================
# #
# TMDB API key — required for Stage 2 recommendations and external_ids lookup # TMDB API key — required for Stage 2 recommendations and external_ids lookup
# Configure HOST*_TMDB_API_KEY in master_host*.conf # Configure HOST*_TMDB_API_KEY in host*.conf
# Free key at: https://www.themoviedb.org/settings/api # Free key at: https://www.themoviedb.org/settings/api
# #
# ============================================================================================== # ==============================================================================================
@@ -150,19 +150,19 @@ if ! command -v jq >/dev/null 2>&1; then
fi fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf" error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${SONARR_URL:-}" || -z "${SONARR_API_KEY:-}" ]]; then if [[ -z "${SONARR_URL:-}" || -z "${SONARR_API_KEY:-}" ]]; then
error "SONARR_URL / SONARR_API_KEY not configured — check master_host*.conf" error "SONARR_URL / SONARR_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${TMDB_API_KEY:-}" ]]; then if [[ -z "${TMDB_API_KEY:-}" ]]; then
error "TMDB_API_KEY not configured — required for discovery" error "TMDB_API_KEY not configured — required for discovery"
error "Get a free key at https://www.themoviedb.org/settings/api" error "Get a free key at https://www.themoviedb.org/settings/api"
error "Configure HOST*_TMDB_API_KEY in master_host*.conf" error "Configure HOST*_TMDB_API_KEY in host*.conf"
exit 1 exit 1
fi fi
+1 -1
View File
@@ -53,7 +53,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_MOVIES_ROOT # HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_MOVIES_ROOT
# HOST*_RADARR_PATH_MAP — container path → host path translation # HOST*_RADARR_PATH_MAP — container path → host path translation
+1 -1
View File
@@ -39,7 +39,7 @@
# #
# RADARR_DROPPED_ADD_EXCLUSION — add removed movies to import exclusion (default: true) # RADARR_DROPPED_ADD_EXCLUSION — add removed movies to import exclusion (default: true)
# #
# master_host*.conf # host*.conf
# #
# HOST1_RADARR_URL / HOST1_RADARR_API_KEY # HOST1_RADARR_URL / HOST1_RADARR_API_KEY
# Aliased by detect_hosts() — script uses RADARR_URL / RADARR_API_KEY # Aliased by detect_hosts() — script uses RADARR_URL / RADARR_API_KEY
+1 -1
View File
@@ -53,7 +53,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_TV_ROOT # HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_TV_ROOT
# HOST*_SONARR_PATH_MAP — container path → host path translation # HOST*_SONARR_PATH_MAP — container path → host path translation
+1 -1
View File
@@ -38,7 +38,7 @@
# #
# SONARR_DROPPED_ADD_EXCLUSION — add removed series to import exclusion (default: true) # SONARR_DROPPED_ADD_EXCLUSION — add removed series to import exclusion (default: true)
# #
# master_host*.conf # host*.conf
# #
# HOST1_SONARR_URL / HOST1_SONARR_API_KEY # HOST1_SONARR_URL / HOST1_SONARR_API_KEY
# Aliased by detect_hosts() — script uses SONARR_URL / SONARR_API_KEY # Aliased by detect_hosts() — script uses SONARR_URL / SONARR_API_KEY
+14 -14
View File
@@ -13,7 +13,7 @@ Each domain and subdomain has an independent TLS certificate and must be listed
separately. `cert_monitor.sh` makes one openssl connection per entry. separately. `cert_monitor.sh` makes one openssl connection per entry.
```bash ```bash
# master_host1.conf # host1.conf
HOST1_CERT_MONITOR_DOMAINS=( HOST1_CERT_MONITOR_DOMAINS=(
"Gmer4Lfe.com" "Gmer4Lfe.com"
"Gmer4Lfe.us" "Gmer4Lfe.us"
@@ -21,7 +21,7 @@ HOST1_CERT_MONITOR_DOMAINS=(
# "cloud.Gmer4Lfe.com" # "cloud.Gmer4Lfe.com"
) )
# master_host2.conf # host2.conf
HOST2_CERT_MONITOR_DOMAINS=( HOST2_CERT_MONITOR_DOMAINS=(
"Jayred365.com" "Jayred365.com"
# "auth.Jayred365.com" # "auth.Jayred365.com"
@@ -63,12 +63,12 @@ CERT_TIMEOUT=10 # seconds per domain before declaring FAILED
### Drive Ignore List ### Drive Ignore List
```bash ```bash
# master_host1.conf # host1.conf
HOST1_SMART_IGNORE_DRIVES=( HOST1_SMART_IGNORE_DRIVES=(
"sda" # boot USB flash drive — no meaningful SMART data "sda" # boot USB flash drive — no meaningful SMART data
) )
# master_host2.conf # host2.conf
HOST2_SMART_IGNORE_DRIVES=( HOST2_SMART_IGNORE_DRIVES=(
"sda" # boot USB flash drive "sda" # boot USB flash drive
) )
@@ -109,7 +109,7 @@ if `dynamix.cfg` is not found (e.g., running outside of unRAID).
### Share Configuration ### Share Configuration
```bash ```bash
# master_host1.conf # host1.conf
HOST1_BACKUP_VERIFY_SHARES=( HOST1_BACKUP_VERIFY_SHARES=(
# empty — uses HOST1_DAILY_SYNC_SHARES automatically # empty — uses HOST1_DAILY_SYNC_SHARES automatically
# "/mnt/user/Movies" # override to check specific shares only # "/mnt/user/Movies" # override to check specific shares only
@@ -230,9 +230,9 @@ data — it just doesn't trigger a notification for that condition.
| State File | Source | What It Shows | | State File | Source | What It Shows |
|-----------|--------|---------------| |-----------|--------|---------------|
| `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) | | `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
| `SYS_WATCHDOG_FAILED_FILE` | `Docker_Essentials/docker_watchdog.sh` | Container skip list — needs human attention | | `SYS_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
| `WATCHDOG_STATE_FILE` | `Docker_Essentials/docker_watchdog.sh` | Active container strike counts | | `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts |
| `SYS_WATCHDOG_STATE_FILE` | `unRAID_Essentials/system_watchdog.sh` | Active system watchdog strikes | | `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/system_watchdog.sh` | Active system watchdog strikes |
| `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history | | `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |
| `TRANSCODE_DAILY_LOG` | `Transcodes/` | Weekly transcode statistics | | `TRANSCODE_DAILY_LOG` | `Transcodes/` | Weekly transcode statistics |
| `CERT_MONITOR_DOMAINS` | live openssl check | Current cert status per domain | | `CERT_MONITOR_DOMAINS` | live openssl check | Current cert status per domain |
@@ -244,11 +244,11 @@ data — it just doesn't trigger a notification for that condition.
### Emby Credentials ### Emby Credentials
```bash ```bash
# master_host1.conf # host1.conf
HOST1_EMBY_URL="http://192.168.50.2:8096" HOST1_EMBY_URL="http://192.168.50.2:8096"
HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829" HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829"
# master_host2.conf # host2.conf
HOST2_EMBY_URL="http://192.168.50.3:8096" HOST2_EMBY_URL="http://192.168.50.3:8096"
HOST2_EMBY_API_KEY="<host2_api_key>" HOST2_EMBY_API_KEY="<host2_api_key>"
``` ```
@@ -306,7 +306,7 @@ in `unRAID_Essentials/`.
### Pool Ignore List ### Pool Ignore List
```bash ```bash
# master_host1.conf # host1.conf
HOST1_ZFS_REPORT_IGNORE_POOLS=( HOST1_ZFS_REPORT_IGNORE_POOLS=(
"disk10" # JBOD member — high usage expected, exclude from report noise "disk10" # JBOD member — high usage expected, exclude from report noise
"disk9" "disk9"
@@ -315,7 +315,7 @@ HOST1_ZFS_REPORT_IGNORE_POOLS=(
"disk5" "disk5"
) )
# master_host2.conf # host2.conf
HOST2_ZFS_REPORT_IGNORE_POOLS=( HOST2_ZFS_REPORT_IGNORE_POOLS=(
# list host2's JBOD members here # list host2's JBOD members here
) )
@@ -392,7 +392,7 @@ ZFS_REPORT_AVAIL_WARN_GB=20
ZFS_REPORT_DOCKER_TOP=10 ZFS_REPORT_DOCKER_TOP=10
``` ```
### master_host*.conf ### host*.conf
```bash ```bash
# cert_monitor.sh # cert_monitor.sh
@@ -485,7 +485,7 @@ emby_session_report.sh --dry-run # test connectivity, generate report, no noti
### --status ### --status
Shows current configuration and exits without running checks. Use to verify Shows current configuration and exits without running checks. Use to verify
configuration is loaded correctly after editing `master.conf` or `master_host*.conf`. configuration is loaded correctly after editing `master.conf` or `host*.conf`.
```bash ```bash
cert_monitor.sh --status # domain list, CERT_WARN_DAYS, CERT_CRIT_DAYS, timeout cert_monitor.sh --status # domain list, CERT_WARN_DAYS, CERT_CRIT_DAYS, timeout
+2 -2
View File
@@ -63,7 +63,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_BACKUP_VERIFY_SHARES # HOST*_BACKUP_VERIFY_SHARES
# Shares to verify. Leave empty to use HOST*_DAILY_SYNC_SHARES automatically. # Shares to verify. Leave empty to use HOST*_DAILY_SYNC_SHARES automatically.
@@ -140,7 +140,7 @@ fi
if [[ ${#VERIFY_SHARES[@]} -eq 0 ]]; then if [[ ${#VERIFY_SHARES[@]} -eq 0 ]]; then
warn "No shares configured for $MY_ID — nothing to verify" warn "No shares configured for $MY_ID — nothing to verify"
warn "Check HOST*_BACKUP_VERIFY_SHARES or HOST*_DAILY_SYNC_SHARES in master_host*.conf" warn "Check HOST*_BACKUP_VERIFY_SHARES or HOST*_DAILY_SYNC_SHARES in host*.conf"
exit 0 exit 0
fi fi
-3
View File
@@ -90,9 +90,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh" source "$SCRIPT_DIR/../load_config.sh"
# Monitor script — output is the point
SILENT_MODE=false
parse_args "$@" parse_args "$@"
# ── Parse mode from PARSED_ARGS ─────────────────────────────────────────────────────────────── # ── Parse mode from PARSED_ARGS ───────────────────────────────────────────────────────────────
+2 -5
View File
@@ -51,7 +51,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_CERT_MONITOR_DOMAINS # HOST*_CERT_MONITOR_DOMAINS
# Domains this host monitors. Each domain and subdomain is a separate entry — # Domains this host monitors. Each domain and subdomain is a separate entry —
@@ -91,9 +91,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh" source "$SCRIPT_DIR/../load_config.sh"
# Monitor script — output is the point
SILENT_MODE=false
parse_args "$@" parse_args "$@"
# ============================================================================================== # ==============================================================================================
@@ -126,7 +123,7 @@ detect_hosts
# Empty array guard # Empty array guard
if [[ ${#CERT_MONITOR_DOMAINS[@]} -eq 0 ]]; then if [[ ${#CERT_MONITOR_DOMAINS[@]} -eq 0 ]]; then
warn "CERT_MONITOR_DOMAINS is empty for $MY_ID" warn "CERT_MONITOR_DOMAINS is empty for $MY_ID"
warn "Check HOST*_CERT_MONITOR_DOMAINS in master_host*.conf" warn "Check HOST*_CERT_MONITOR_DOMAINS in host*.conf"
exit 0 exit 0
fi fi
+1 -4
View File
@@ -42,7 +42,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_EMBY_URL # HOST*_EMBY_URL
# Emby server URL for this host. Aliased by detect_hosts() → EMBY_URL. # Emby server URL for this host. Aliased by detect_hosts() → EMBY_URL.
@@ -87,9 +87,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh" source "$SCRIPT_DIR/../load_config.sh"
# Monitor/report script — output is the point
SILENT_MODE=false
parse_args "$@" parse_args "$@"
# ============================================================================================== # ==============================================================================================
+1 -4
View File
@@ -43,7 +43,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_SMART_IGNORE_DRIVES # HOST*_SMART_IGNORE_DRIVES
# Drives skipped in SMART monitoring. Aliased by detect_hosts() → # Drives skipped in SMART monitoring. Aliased by detect_hosts() →
@@ -80,9 +80,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh" source "$SCRIPT_DIR/../load_config.sh"
# Monitor script — output is the point
SILENT_MODE=false
parse_args "$@" parse_args "$@"
# ============================================================================================== # ==============================================================================================
-3
View File
@@ -116,9 +116,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh" source "$SCRIPT_DIR/../load_config.sh"
# Report/monitor script — output is the point when sending
SILENT_MODE=false
parse_args "$@" parse_args "$@"
# ============================================================================================== # ==============================================================================================
+1 -4
View File
@@ -67,7 +67,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_ZFS_REPORT_IGNORE_POOLS # HOST*_ZFS_REPORT_IGNORE_POOLS
# Pools excluded from health reporting. Single-disk JBOD members generate # Pools excluded from health reporting. Single-disk JBOD members generate
@@ -113,9 +113,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh" source "$SCRIPT_DIR/../load_config.sh"
# Monitor/report script — output is the point
SILENT_MODE=false
parse_args "$@" parse_args "$@"
DOCKER_TIMEOUT=15 DOCKER_TIMEOUT=15
+1 -1
View File
@@ -17,7 +17,7 @@
# #
# ADDING A NEW ARR # ADDING A NEW ARR
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# 1. Add HOST*_<ARR>_URL, API_KEY, MEDIA_ROOT, PATH_MAP to master_host*.conf # 1. Add HOST*_<ARR>_URL, API_KEY, MEDIA_ROOT, PATH_MAP to host*.conf
# 2. Add <ARR>_ORPHAN_AGE, MAX_DELETE_GB, EXTENSIONS, etc. to master.conf # 2. Add <ARR>_ORPHAN_AGE, MAX_DELETE_GB, EXTENSIONS, etc. to master.conf
# 3. Add a profile entry to ARR_PROFILES in arr_cleanup.py (6 values) # 3. Add a profile entry to ARR_PROFILES in arr_cleanup.py (6 values)
# 4. Add an export block for the new arr below (copy Sonarr block, change prefix) # 4. Add an export block for the new arr below (copy Sonarr block, change prefix)
+5 -5
View File
@@ -214,10 +214,10 @@ ARRAY_START_SCRIPTS=(
# watchdogs check container states # watchdogs check container states
# ── Continuous scripts — run until array stops ───────────────────────────── # ── Continuous scripts — run until array stops ─────────────────────────────
"unRAID_Essentials/system_watchdog.sh" # system health BEFORE docker watchdog — "Watchdogs/system_watchdog.sh" # system health BEFORE docker watchdog —
# system watchdog writes state file that # system watchdog writes state file that
# docker watchdog reads every cycle # docker watchdog reads every cycle
"Docker_Essentials/docker_watchdog.sh" # container health BEFORE failover — "Watchdogs/docker_watchdog.sh" # container health BEFORE failover —
# containers must be healthy for failover # containers must be healthy for failover
# to make reliable decisions # to make reliable decisions
"Failover/failover.sh" # failover LAST — needs everything else stable "Failover/failover.sh" # failover LAST — needs everything else stable
@@ -465,7 +465,7 @@ new search — hands-free recovery while you sleep.
### ── Host Awareness ─────────────────────────────────────────────────────────── ### ── Host Awareness ───────────────────────────────────────────────────────────
```bash ```bash
# master.conf + master_host*.conf # master.conf + host*.conf
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# Each arr is independently toggled per host. # Each arr is independently toggled per host.
# Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action. # Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action.
@@ -580,7 +580,7 @@ DAILY_MAINTENANCE_SCRIPTS=(
"Docker_Essentials/docker_daily_restart.sh" # POST-SYNC — restarts after everything "Docker_Essentials/docker_daily_restart.sh" # POST-SYNC — restarts after everything
) )
# master_host1.conf # host1.conf
HOST1_DAILY_SYNC_SHARES=( HOST1_DAILY_SYNC_SHARES=(
"/mnt/user/Movies" # HOST1 source of truth — push to HOST2 "/mnt/user/Movies" # HOST1 source of truth — push to HOST2
"/mnt/user/Tv_Shows" # HOST1 source of truth "/mnt/user/Tv_Shows" # HOST1 source of truth
@@ -595,7 +595,7 @@ HOST1_PERSONAL_SHARES=(
"/mnt/user/Personal" # encrypted personal share — appended after standard "/mnt/user/Personal" # encrypted personal share — appended after standard
) )
# master_host2.conf # host2.conf
HOST2_DAILY_SYNC_SHARES=( HOST2_DAILY_SYNC_SHARES=(
"/mnt/user/Anime_Shows" # HOST2 source of truth — push to HOST1 "/mnt/user/Anime_Shows" # HOST2 source of truth — push to HOST1
"/mnt/user/Anime_Movies" # HOST2 source of truth "/mnt/user/Anime_Movies" # HOST2 source of truth
+2 -2
View File
@@ -18,8 +18,8 @@
# Docker_Essentials/docker_network_connect.sh — ensure networks + container connections # Docker_Essentials/docker_network_connect.sh — ensure networks + container connections
# #
# CONTINUOUS (run until array stops): # CONTINUOUS (run until array stops):
# unRAID_Essentials/system_watchdog.sh — system health monitor (last line of defense) # Watchdogs/system_watchdog.sh — system health monitor (last line of defense)
# Docker_Essentials/docker_watchdog.sh — container health monitor # Watchdogs/docker_watchdog.sh — container health monitor
# Fallback/fallback.sh — mutual failover monitor # Fallback/fallback.sh — mutual failover monitor
# #
# ── WHY ORDER MATTERS ───────────────────────────────────────────────────────────────────────── # ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
+1 -1
View File
@@ -123,7 +123,7 @@ if ! check_rsync_enabled "CRITICAL"; then
echo "Critical rsync disabled — skipping sync, running partnership check only" echo "Critical rsync disabled — skipping sync, running partnership check only"
elif [[ ${#CRITICAL_SYNC_SHARES[@]} -eq 0 ]]; then elif [[ ${#CRITICAL_SYNC_SHARES[@]} -eq 0 ]]; then
warn "CRITICAL_RSYNC_ENABLED=true but CRITICAL_SYNC_SHARES is empty for $MY_ID" warn "CRITICAL_RSYNC_ENABLED=true but CRITICAL_SYNC_SHARES is empty for $MY_ID"
warn "Check HOST*_CRITICAL_SYNC_SHARES in master_host*.conf" warn "Check HOST*_CRITICAL_SYNC_SHARES in host*.conf"
else else
echo "Critical sync — $MY_ID$REMOTE_ID$(date '+%H:%M:%S')" echo "Critical sync — $MY_ID$REMOTE_ID$(date '+%H:%M:%S')"
+2 -2
View File
@@ -53,7 +53,7 @@
# Summary always shown — gives window timing and share/job counts. # Summary always shown — gives window timing and share/job counts.
# Notify only on failure — successful daily maintenance doesn't need notification. # Notify only on failure — successful daily maintenance doesn't need notification.
# #
# ── CONFIGURATION (master.conf + master_host*.conf) ─────────────────────────────────────────── # ── CONFIGURATION (master.conf + host*.conf) ───────────────────────────────────────────
# HOST*_DAILY_SYNC_SHARES — shares pushed to mirror each day # HOST*_DAILY_SYNC_SHARES — shares pushed to mirror each day
# HOST*_PERSONAL_SHARES — encrypted personal shares # HOST*_PERSONAL_SHARES — encrypted personal shares
# DAILY_MAINTENANCE_SCRIPTS — maintenance jobs (permissions, cleanup, restart) # DAILY_MAINTENANCE_SCRIPTS — maintenance jobs (permissions, cleanup, restart)
@@ -244,7 +244,7 @@ if ! check_rsync_enabled "DAILY"; then
warn "Daily rsync disabled — skipping all $SHARE_COUNT share syncs" warn "Daily rsync disabled — skipping all $SHARE_COUNT share syncs"
warn "Proceeding to maintenance jobs..." warn "Proceeding to maintenance jobs..."
elif [[ "$SHARE_COUNT" -eq 0 ]]; then elif [[ "$SHARE_COUNT" -eq 0 ]]; then
warn "No shares configured for $MY_ID — check HOST*_DAILY_SYNC_SHARES in master_host*.conf" warn "No shares configured for $MY_ID — check HOST*_DAILY_SYNC_SHARES in host*.conf"
else else
# Pre-flight — connectivity then remote rootfs # Pre-flight — connectivity then remote rootfs
check_connectivity check_connectivity
@@ -24,7 +24,7 @@
# #
# ── HOST AWARENESS ──────────────────────────────────────────────────────────────────────────── # ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
# detect_hosts() sets MY_ID and aliases HOST*_INTERMEDIATE_SYNC_SHARES → INTERMEDIATE_SYNC_SHARES. # detect_hosts() sets MY_ID and aliases HOST*_INTERMEDIATE_SYNC_SHARES → INTERMEDIATE_SYNC_SHARES.
# Each server can have a different set of mid-day shares — configure in master_host*.conf. # Each server can have a different set of mid-day shares — configure in host*.conf.
# Each script in INTERMEDIATE_MAINTENANCE_SCRIPTS handles its own host logic. # Each script in INTERMEDIATE_MAINTENANCE_SCRIPTS handles its own host logic.
# #
# ── DRIVE TEMP HANDLING ─────────────────────────────────────────────────────────────────────── # ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
@@ -41,7 +41,7 @@
# Silent on success — runs 4x/day, only failures warrant notification # Silent on success — runs 4x/day, only failures warrant notification
# #
# ── CONFIGURATION ───────────────────────────────────────────────────────────────────────────── # ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
# master_host*.conf: HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped) # host*.conf: HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped)
# master.conf: INTERMEDIATE_RSYNC_ENABLED — enable/disable rsync section (default: true) # master.conf: INTERMEDIATE_RSYNC_ENABLED — enable/disable rsync section (default: true)
# master.conf: INTERMEDIATE_MAINTENANCE_SCRIPTS — jobs run after rsync # master.conf: INTERMEDIATE_MAINTENANCE_SCRIPTS — jobs run after rsync
# master.conf: ARR_SYNC_ENABLED — toggle inside arr_sync.sh # master.conf: ARR_SYNC_ENABLED — toggle inside arr_sync.sh
@@ -154,6 +154,8 @@ line() { REPORT+=(" $1"); }
issue() { ISSUES+=("$1"); REPORT+=(" ⚠️ $1"); } issue() { ISSUES+=("$1"); REPORT+=(" ⚠️ $1"); }
finding() { FINDINGS+=("$1"); REPORT+=(" $1"); } finding() { FINDINGS+=("$1"); REPORT+=(" $1"); }
get_array() { eval "echo \"\${${1}[*]}\""; }
format_bytes() { format_bytes() {
local bytes=$1 local bytes=$1
if (( bytes > 1073741824 )); then if (( bytes > 1073741824 )); then
@@ -656,6 +658,90 @@ if command -v tailscale >/dev/null 2>&1; then
fi fi
fi fi
# ==============================================================================================
# ━━━ 🔗 MESH ━━━
# ==============================================================================================
section "🔗 MESH"
_ALL_HOST_IDS=()
for _h in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do
[[ -n "${!_h:-}" ]] && _ALL_HOST_IDS+=("$_h")
done
# ── Members ──
if [[ ${#_ALL_HOST_IDS[@]} -eq 0 ]]; then
line "No hosts defined"
else
line "Members:"
for _h in "${_ALL_HOST_IDS[@]}"; do
_server="${!_h}"
_owner_var="${_h}_OWNER"; _owner="${!_owner_var:-unknown}"
_email_var="${_h}_OWNER_EMAIL"; _email="${!_email_var:-(not set)}"
line " $_h $_server / $_owner / $_email"
done
fi
# ── Protected Services ──
_coverage_found=false
for _covered in "${_ALL_HOST_IDS[@]}"; do
_covered_owner_var="${_covered}_OWNER"; _covered_owner="${!_covered_owner_var:-$_covered}"
_covered_email_var="${_covered}_OWNER_EMAIL"; _covered_email="${!_covered_email_var:-}"
declare -A _tier_containers=()
declare -A _tier_delays=()
_covered_by=""
_any_tiers=false
for _covering in "${_ALL_HOST_IDS[@]}"; do
[[ "$_covering" == "$_covered" ]] && continue
for _tier in 1 2 3 4; do
_containers=$(get_array "FALLBACK_${_covering}_COVERS_${_covered}_TIER${_tier}")
[[ -z "$_containers" ]] && continue
_any_tiers=true
_tier_containers[$_tier]="$_containers"
_delay_var="${_covered}_TIER${_tier}_DELAY"
_tier_delays[$_tier]="${!_delay_var:-0}"
done
if [[ "$_any_tiers" == true ]]; then
_cov_owner_var="${_covering}_OWNER"; _cov_owner="${!_cov_owner_var:-$_covering}"
_covered_by="$_covering ($_cov_owner)"
fi
done
[[ "$_any_tiers" == false ]] && { unset _tier_containers _tier_delays; declare -A _tier_containers=() _tier_delays=(); continue; }
_coverage_found=true
_hdr="$_covered_owner"
[[ -n "$_covered_email" ]] && _hdr+="$_covered_email"
line "Protected: $_hdr"
for _tier in 1 2 3 4; do
[[ -z "${_tier_containers[$_tier]:-}" ]] && continue
_d="${_tier_delays[$_tier]:-0}"
if (( _d == 0 )); then _dlabel="immediate"
elif (( _d >= 1440 )); then _dlabel="$(( _d / 1440 ))d"
elif (( _d >= 60 )); then _dlabel="$(( _d / 60 ))hr"
else _dlabel="${_d}min"
fi
line " Tier $_tier (${_dlabel}): ${_tier_containers[$_tier]// /, }"
done
[[ -n "$_covered_by" ]] && line " Covered by: $_covered_by"
unset _tier_containers _tier_delays
declare -A _tier_containers=() _tier_delays=()
done
[[ "$_coverage_found" == false ]] && line "No fallback coverage configured"
# ── Partnership ──
if [[ "${PARTNERSHIP_ENABLED:-false}" == true ]]; then
_po_host="${PARTNERSHIP_OWNER_HOST:-HOST1}"
_po_server="${!_po_host:-unknown}"
_po_name_var="${_po_host}_OWNER"; _po_name="${!_po_name_var:-unknown}"
line "Partnership: enabled — owner $_po_host ($_po_server / $_po_name), sync every ${PARTNERSHIP_SYNC_INTERVAL:-15}min"
else
line "Partnership: disabled"
fi
# ============================================================================================== # ==============================================================================================
# ━━━ 🔐 SECURITY ━━━ # ━━━ 🔐 SECURITY ━━━
# ============================================================================================== # ==============================================================================================
+5 -5
View File
@@ -130,7 +130,7 @@ PARTNERSHIP_ONBOARD_VERIFY=true # curl-verify each WebUI after onboard
PARTNERSHIP_ONBOARD_NOTIFY=true # notify both servers on successful onboard PARTNERSHIP_ONBOARD_NOTIFY=true # notify both servers on successful onboard
``` ```
### master_host1.conf (owner side) ### host1.conf (owner side)
```bash ```bash
# Containers whose WebUI URLs are redirected to owner's Tailscale IP on onboard. # Containers whose WebUI URLs are redirected to owner's Tailscale IP on onboard.
@@ -179,7 +179,7 @@ HOST1_PARTNERSHIP_OWN_CONTAINERS=(
) )
``` ```
### master_host2.conf (mirror side) ### host2.conf (mirror side)
```bash ```bash
# Containers whose WebUI URLs are redirected on onboard. # Containers whose WebUI URLs are redirected on onboard.
@@ -342,9 +342,9 @@ Partnership/partnership_onboard.sh
### Adding a New Container to the Auth Stack ### Adding a New Container to the Auth Stack
1. Create the XML template on HOST1 (`/boot/config/plugins/dockerMan/templates-user/my-NewContainer.xml`) 1. Create the XML template on HOST1 (`/boot/config/plugins/dockerMan/templates-user/my-NewContainer.xml`)
2. Add the XML filename to `HOST1_PARTNERSHIP_AUTH_STACK` in `master_host1.conf` 2. Add the XML filename to `HOST1_PARTNERSHIP_AUTH_STACK` in `host1.conf`
- If it has a database dependency, put the dep earlier in the array - If it has a database dependency, put the dep earlier in the array
3. Add the container name to `HOST2_PARTNERSHIP_REPLACE_CONTAINERS` in `master_host2.conf` 3. Add the container name to `HOST2_PARTNERSHIP_REPLACE_CONTAINERS` in `host2.conf`
4. Re-run the auth stack portion: 4. Re-run the auth stack portion:
```bash ```bash
Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync
@@ -514,7 +514,7 @@ automatically, but if HOST2 had no pre-existing auth stack of its own, it needs
docker ps docker ps
# Verify own parked containers came back up: # Verify own parked containers came back up:
# (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in master_host2.conf) # (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in host2.conf)
# If you need a fresh auth stack, deploy from HOST2's own XML templates: # If you need a fresh auth stack, deploy from HOST2's own XML templates:
docker create ... && docker start NginxProxyManager # etc. docker create ... && docker start NginxProxyManager # etc.
+2 -2
View File
@@ -141,7 +141,7 @@
# TAILSCALE_API_KEY / TAILSCALE_TAILNET # TAILSCALE_API_KEY / TAILSCALE_TAILNET
# Required when PARTNERSHIP_REMOVE_TAILSCALE=true # Required when PARTNERSHIP_REMOVE_TAILSCALE=true
# #
# master_host*.conf # host*.conf
# #
# HOST*_PARTNERSHIP_AUTH_WEBUIS # HOST*_PARTNERSHIP_AUTH_WEBUIS
# Containers reconfigured on onboard/offboard. Format: "ContainerName|WebUIPort" # Containers reconfigured on onboard/offboard. Format: "ContainerName|WebUIPort"
@@ -253,7 +253,7 @@ OWNER="${!OWNER_ID}" # hostname string
MIRROR="${!MIRROR_ID}" MIRROR="${!MIRROR_ID}"
# SSH_KEY (set by detect_hosts) is this server's own private key. # SSH_KEY (set by detect_hosts) is this server's own private key.
# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh. # The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
# With sparse checkout, each server only has its own master_host{N}.conf — the other server's # With sparse checkout, each server only has its own host{N}.conf — the other server's
# key path is never available here. Use SSH_KEY for all outbound SSH regardless of mode. # key path is never available here. Use SSH_KEY for all outbound SSH regardless of mode.
MIRROR_SSH_KEY="$SSH_KEY" MIRROR_SSH_KEY="$SSH_KEY"
OWNER_SSH_KEY="$SSH_KEY" OWNER_SSH_KEY="$SSH_KEY"
+1 -1
View File
@@ -39,7 +39,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_PARTNERSHIP_AUTH_STACK # HOST*_PARTNERSHIP_AUTH_STACK
# Auth container XMLs to push during onboard — used on offboard to identify what # Auth container XMLs to push during onboard — used on offboard to identify what
+6 -6
View File
@@ -45,7 +45,7 @@
# #
# Dependency ordering in the auth stack is owner-enforced # Dependency ordering in the auth stack is owner-enforced
# PARTNERSHIP_AUTH_STACK order matters: Mariadb and Redis must come before Authelia. # PARTNERSHIP_AUTH_STACK order matters: Mariadb and Redis must come before Authelia.
# The array is ordered correctly in master_host1.conf. After each Mariadb/Redis deploy, # The array is ordered correctly in host1.conf. After each Mariadb/Redis deploy,
# the script waits for the container to be healthy before continuing. This is a remote # the script waits for the container to be healthy before continuing. This is a remote
# health check — the container must be running (or report healthy) before the next # health check — the container must be running (or report healthy) before the next
# dependent is deployed. # dependent is deployed.
@@ -73,7 +73,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_PARTNERSHIP_AUTH_STACK # HOST*_PARTNERSHIP_AUTH_STACK
# XML filenames (from this server's templates-user/) to push and deploy on the # XML filenames (from this server's templates-user/) to push and deploy on the
@@ -82,7 +82,7 @@
# #
# HOST*_PARTNERSHIP_REPLACE_CONTAINERS # HOST*_PARTNERSHIP_REPLACE_CONTAINERS
# Containers to stop on the mirror before deploying the auth stack. # Containers to stop on the mirror before deploying the auth stack.
# Defined in the MIRROR's own conf (master_host*.conf on HOST2) — never in HOST1's conf. # Defined in the MIRROR's own conf (host*.conf on HOST2) — never in HOST1's conf.
# Read live from the mirror via SSH during Step 3 (sources mirror's load_config.sh at # Read live from the mirror via SSH during Step 3 (sources mirror's load_config.sh at
# the same $SCRIPTS_ROOT path — convention: both servers use the same repo location). # the same $SCRIPTS_ROOT path — convention: both servers use the same repo location).
# Leave empty on HOST2 if no conflicting containers exist (fresh mirror: nothing to stop). # Leave empty on HOST2 if no conflicting containers exist (fresh mirror: nothing to stop).
@@ -171,7 +171,7 @@ OWNER="${!OWNER_ID}"
MIRROR="${!MIRROR_ID}" MIRROR="${!MIRROR_ID}"
# SSH_KEY (set by detect_hosts) is this server's own private key. # SSH_KEY (set by detect_hosts) is this server's own private key.
# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh. # The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
# HOST{N}_SSH_KEY lives in master_host{N}.conf — with sparse checkout, the other server's # HOST{N}_SSH_KEY lives in host{N}.conf — with sparse checkout, the other server's
# conf is never present here. Always use SSH_KEY (local private key) for outbound SSH. # conf is never present here. Always use SSH_KEY (local private key) for outbound SSH.
MIRROR_SSH_KEY="$SSH_KEY" MIRROR_SSH_KEY="$SSH_KEY"
@@ -359,7 +359,7 @@ wait_for_container_healthy() {
# #
# SSHes to the mirror, sources its load_config.sh at the same $SCRIPTS_ROOT path (both servers # SSHes to the mirror, sources its load_config.sh at the same $SCRIPTS_ROOT path (both servers
# use the same convention), and reads the named config array from the mirror's own conf. # use the same convention), and reads the named config array from the mirror's own conf.
# HOST2's container list stays in HOST2's master_host2.conf — not duplicated in HOST1's conf. # HOST2's container list stays in HOST2's host2.conf — not duplicated in HOST1's conf.
# Fails gracefully if scripts aren't present yet or the array is empty (nothing to stop). # Fails gracefully if scripts aren't present yet or the array is empty (nothing to stop).
# #
# deploy_container_from_xml() already stops/removes containers with the same name as what's # deploy_container_from_xml() already stops/removes containers with the same name as what's
@@ -547,7 +547,7 @@ if [[ "$SKIP_AUTH_STACK" == true ]]; then
warn "Skipping (--skip-auth-stack)" warn "Skipping (--skip-auth-stack)"
elif [[ ${#PARTNERSHIP_AUTH_STACK[@]} -eq 0 ]]; then elif [[ ${#PARTNERSHIP_AUTH_STACK[@]} -eq 0 ]]; then
warn "PARTNERSHIP_AUTH_STACK not set in ${MY_ID} conf — skipping auth stack deploy" warn "PARTNERSHIP_AUTH_STACK not set in ${MY_ID} conf — skipping auth stack deploy"
warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to master_host${MY_ID: -1}.conf" warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to host${MY_ID: -1}.conf"
STEP_AUTH_OK=false STEP_AUTH_OK=false
else else
deploy_xml_stack PARTNERSHIP_AUTH_STACK deploy_xml_stack PARTNERSHIP_AUTH_STACK
+5 -5
View File
@@ -7,7 +7,7 @@
# unRAID-Gmer4Lfe → gmer4lfe_rsync_automation # unRAID-Gmer4Lfe → gmer4lfe_rsync_automation
# unRAID-Jayred365 → jayred365_rsync_automation # unRAID-Jayred365 → jayred365_rsync_automation
# Idempotent — skips generation if key already exists (use --force to regenerate). # Idempotent — skips generation if key already exists (use --force to regenerate).
# Updates master_host*.conf with key path on success. # Updates host*.conf with key path on success.
# #
# ── MODES ───────────────────────────────────────────────────────────────────────────────────── # ── MODES ─────────────────────────────────────────────────────────────────────────────────────
# (default) — generate key if missing, copy to remote, update conf # (default) — generate key if missing, copy to remote, update conf
@@ -85,7 +85,7 @@ SSH_PUB_PATH="${SSH_KEY_PATH}.pub"
# ── Host conf path ──────────────────────────────────────────────────────────────────────────── # ── Host conf path ────────────────────────────────────────────────────────────────────────────
HOST_NUM="${MY_ID#HOST}" # "1" or "2" HOST_NUM="${MY_ID#HOST}" # "1" or "2"
HOST_CONF="$SCRIPTS_ROOT/master_host${HOST_NUM}.conf" HOST_CONF="$SCRIPTS_ROOT/host${HOST_NUM}.conf"
KEY_CONF_VAR="${MY_ID}_SSH_KEY" KEY_CONF_VAR="${MY_ID}_SSH_KEY"
# ── Strike state file ───────────────────────────────────────────────────────────────────────── # ── Strike state file ─────────────────────────────────────────────────────────────────────────
@@ -316,12 +316,12 @@ fi
# ── Update conf ─────────────────────────────────────────────────────────────────────────────── # ── Update conf ───────────────────────────────────────────────────────────────────────────────
echo "" echo ""
echo "━━━ $ICON_GEAR Update master_host${HOST_NUM}.conf ━━━" echo "━━━ $ICON_GEAR Update host${HOST_NUM}.conf ━━━"
if [[ "$DRY_RUN" == false ]]; then if [[ "$DRY_RUN" == false ]]; then
update_conf_key_path update_conf_key_path
else else
warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in master_host${HOST_NUM}.conf" warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in host${HOST_NUM}.conf"
fi fi
# ── Copy to remote ──────────────────────────────────────────────────────────────────────────── # ── Copy to remote ────────────────────────────────────────────────────────────────────────────
@@ -373,7 +373,7 @@ echo ""
echo "━━━━━ $ICON_SUMMARY SSH SETUP SUMMARY ━━━━━" echo "━━━━━ $ICON_SUMMARY SSH SETUP SUMMARY ━━━━━"
echo " Key: $SSH_KEY_PATH" echo " Key: $SSH_KEY_PATH"
echo " Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)" echo " Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)"
echo " Conf: ${KEY_CONF_VAR} in master_host${HOST_NUM}.conf" echo " Conf: ${KEY_CONF_VAR} in host${HOST_NUM}.conf"
echo "" echo ""
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \ [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
warn "$ICON_DONE DONE — SSH key ready for rsync automation ✅" warn "$ICON_DONE DONE — SSH key ready for rsync automation ✅"
+2 -2
View File
@@ -893,7 +893,7 @@ Background: NO
```bash ```bash
#!/bin/bash #!/bin/bash
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --status /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --status
``` ```
**What it shows:** **What it shows:**
@@ -1275,7 +1275,7 @@ Background: NO
```bash ```bash
#!/bin/bash #!/bin/bash
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --status /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --status
``` ```
**What it shows:** **What it shows:**
+4 -4
View File
@@ -462,8 +462,8 @@ Monitoring:
Unraid_Scripts/ Unraid_Scripts/
├── master.conf ← All shared configuration — the only file you edit regularly ├── master.conf ← All shared configuration — the only file you edit regularly
├── master_host1.conf ← HOST1-specific: share lists, container names, API keys ├── host1.conf ← HOST1-specific: share lists, container names, API keys
├── master_host2.conf ← HOST2-specific: same structure, different values ├── host2.conf ← HOST2-specific: same structure, different values
├── common.sh ← Shared library — all functions used by every script ├── common.sh ← Shared library — all functions used by every script
├── load_config.sh ← Sources all conf files and common.sh ├── load_config.sh ← Sources all conf files and common.sh
@@ -509,11 +509,11 @@ Unraid_Scripts/
# definitions, watchdog settings, arr cleanup config, DDNS timing, etc. # definitions, watchdog settings, arr cleanup config, DDNS timing, etc.
# Pushed to both servers via git. Never contains server-specific values. # Pushed to both servers via git. Never contains server-specific values.
# #
# master_host1.conf — sourced only on HOST1 # host1.conf — sourced only on HOST1
# HOST1_* prefixed variables: share lists, container names, API keys, # HOST1_* prefixed variables: share lists, container names, API keys,
# ramdisk size, specific paths, per-server toggle overrides. # ramdisk size, specific paths, per-server toggle overrides.
# #
# master_host2.conf — sourced only on HOST2 # host2.conf — sourced only on HOST2
# HOST2_* prefixed variables: same structure, different values. # HOST2_* prefixed variables: same structure, different values.
# #
# detect_hosts() in common.sh: # detect_hosts() in common.sh:
+10 -10
View File
@@ -130,7 +130,7 @@ access key (for script repository pulls).
### 3a — Rsync Automation Keys (ssh_setup.sh) ### 3a — Rsync Automation Keys (ssh_setup.sh)
Run on **each server**. `ssh_setup.sh` generates the key, copies it to the remote, Run on **each server**. `ssh_setup.sh` generates the key, copies it to the remote,
and updates `master_host*.conf` with the key path automatically. and updates `host*.conf` with the key path automatically.
```bash ```bash
# On HOST1 — after the repo is cloned: # On HOST1 — after the repo is cloned:
@@ -146,7 +146,7 @@ unRAID-Gmer4Lfe → /root/.ssh/gmer4lfe_rsync_automation
unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation
``` ```
`master_host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the `host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the
generated key. Run `--status` to verify: generated key. Run `--status` to verify:
```bash ```bash
@@ -188,8 +188,8 @@ find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \;
Expected structure after clone: Expected structure after clone:
``` ```
master.conf ← all shared configuration master.conf ← all shared configuration
master_host1.conf ← HOST1-specific configuration host1.conf ← HOST1-specific configuration
master_host2.conf ← HOST2-specific configuration host2.conf ← HOST2-specific configuration
common.sh ← shared library common.sh ← shared library
load_config.sh ← config loader load_config.sh ← config loader
Orchestrators/ Orchestrators/
@@ -232,7 +232,7 @@ SSH_PORT=221
### Daily Sync Shares ### Daily Sync Shares
```bash ```bash
# master_host1.conf # host1.conf
# Shares this server downloads to — pushed to the other server nightly. # Shares this server downloads to — pushed to the other server nightly.
# arr_sync.sh runs before rsync in the daily window, so the receiving # arr_sync.sh runs before rsync in the daily window, so the receiving
# server's arrs already track incoming files when they arrive. # server's arrs already track incoming files when they arrive.
@@ -247,7 +247,7 @@ HOST1_DAILY_SYNC_SHARES=(
"/mnt/user/stand-up_comedy" "/mnt/user/stand-up_comedy"
) )
# master_host2.conf # host2.conf
HOST2_DAILY_SYNC_SHARES=( HOST2_DAILY_SYNC_SHARES=(
"/mnt/user/Anime_Shows" "/mnt/user/Anime_Shows"
"/mnt/user/Anime_Movies" "/mnt/user/Anime_Movies"
@@ -274,8 +274,8 @@ to their unprefixed names. Scripts only ever reference the unprefixed name — t
work identically on both servers. work identically on both servers.
```bash ```bash
nano /mnt/user/appdata/unraid_scripts/master_host1.conf # on HOST1 nano /mnt/user/appdata/unraid_scripts/host1.conf # on HOST1
nano /mnt/user/appdata/unraid_scripts/master_host2.conf # on HOST2 nano /mnt/user/appdata/unraid_scripts/host2.conf # on HOST2
``` ```
Every variable is documented in the conf files. Key values to set: Every variable is documented in the conf files. Key values to set:
@@ -413,7 +413,7 @@ zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal zfs mount poolname/Gmer4Lfe-Personal
``` ```
### Add to master_host1.conf ### Add to host1.conf
```bash ```bash
HOST1_PERSONAL_SHARES=( HOST1_PERSONAL_SHARES=(
@@ -662,7 +662,7 @@ declare -A PROFILE_EXCLUDE_DIRS
declare -A PROFILE_REMOTE_RESTART_CONTAINERS declare -A PROFILE_REMOTE_RESTART_CONTAINERS
``` ```
### master_host*.conf ### host*.conf
```bash ```bash
# Daily sync shares — one list per server (mutually exclusive) # Daily sync shares — one list per server (mutually exclusive)
+2 -2
View File
@@ -432,7 +432,7 @@ library.db-wal — write-ahead log (uncommitted transactions)
the main library.db is also affected the main library.db is also affected
``` ```
### Configuration (master_host*.conf) ### Configuration (host*.conf)
```bash ```bash
HOST1_EMBY_CONTAINER="Emby" # aliased by detect_hosts() → EMBY_CONTAINER HOST1_EMBY_CONTAINER="Emby" # aliased by detect_hosts() → EMBY_CONTAINER
@@ -470,7 +470,7 @@ without triggering any error — until you try to read that specific file. By th
Run monthly. Also run after any disk replacement or power event. Run monthly. Also run after any disk replacement or power event.
Safe to run while the system is in use — scrub runs at low I/O priority. Safe to run while the system is in use — scrub runs at low I/O priority.
### Configuration (master_host*.conf) ### Configuration (host*.conf)
```bash ```bash
HOST1_ZFS_REPORT_IGNORE_POOLS=( HOST1_ZFS_REPORT_IGNORE_POOLS=(
+3 -3
View File
@@ -55,7 +55,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_EMBY_CONTAINER # HOST*_EMBY_CONTAINER
# Name of the Emby Docker container on this host. # Name of the Emby Docker container on this host.
@@ -122,7 +122,7 @@ fi
detect_hosts detect_hosts
if [[ -z "${EMBY_CONTAINER:-}" ]]; then if [[ -z "${EMBY_CONTAINER:-}" ]]; then
error "EMBY_CONTAINER not set for $MY_ID — check HOST*_EMBY_CONTAINER in master_host*.conf" error "EMBY_CONTAINER not set for $MY_ID — check HOST*_EMBY_CONTAINER in host*.conf"
exit 1 exit 1
fi fi
@@ -135,7 +135,7 @@ EMBY_CONFIG_HOST=$(timeout "$DOCKER_TIMEOUT" docker inspect "$EMBY_CONTAINER" 2>
if [[ -z "$EMBY_CONFIG_HOST" ]]; then if [[ -z "$EMBY_CONFIG_HOST" ]]; then
error "Could not detect Emby config path from Docker mounts" error "Could not detect Emby config path from Docker mounts"
error "Is $EMBY_CONTAINER the correct container name? Check HOST*_EMBY_CONTAINER in master_host*.conf" error "Is $EMBY_CONTAINER the correct container name? Check HOST*_EMBY_CONTAINER in host*.conf"
exit 1 exit 1
fi fi
+3 -3
View File
@@ -36,7 +36,7 @@
# modification. Safe to run multiple times — the second run finds nothing to add. # modification. Safe to run multiple times — the second run finds nothing to add.
# #
# ============================================================================================== # ==============================================================================================
# CONFIGURATION (master_host*.conf) # CONFIGURATION (host*.conf)
# ============================================================================================== # ==============================================================================================
# #
# HOST*_LIDARR_URL / HOST*_LIDARR_API_KEY — Lidarr connection (aliased by detect_hosts) # HOST*_LIDARR_URL / HOST*_LIDARR_API_KEY — Lidarr connection (aliased by detect_hosts)
@@ -73,12 +73,12 @@ if ! command -v jq >/dev/null 2>&1; then
fi fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf" error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then
error "LIDARR_URL / LIDARR_API_KEY not configured — check master_host*.conf" error "LIDARR_URL / LIDARR_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
+3 -3
View File
@@ -38,7 +38,7 @@
# without modification. Safe to run multiple times — the second run finds nothing to add. # without modification. Safe to run multiple times — the second run finds nothing to add.
# #
# ============================================================================================== # ==============================================================================================
# CONFIGURATION (master_host*.conf) # CONFIGURATION (host*.conf)
# ============================================================================================== # ==============================================================================================
# #
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY — Radarr connection (aliased by detect_hosts) # HOST*_RADARR_URL / HOST*_RADARR_API_KEY — Radarr connection (aliased by detect_hosts)
@@ -74,12 +74,12 @@ if ! command -v jq >/dev/null 2>&1; then
fi fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf" error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${RADARR_URL:-}" || -z "${RADARR_API_KEY:-}" ]]; then if [[ -z "${RADARR_URL:-}" || -z "${RADARR_API_KEY:-}" ]]; then
error "RADARR_URL / RADARR_API_KEY not configured — check master_host*.conf" error "RADARR_URL / RADARR_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
+3 -3
View File
@@ -38,7 +38,7 @@
# without modification. Safe to run multiple times — the second run finds nothing to add. # without modification. Safe to run multiple times — the second run finds nothing to add.
# #
# ============================================================================================== # ==============================================================================================
# CONFIGURATION (master.conf / master_host*.conf) # CONFIGURATION (master.conf / host*.conf)
# ============================================================================================== # ==============================================================================================
# #
# SONARR_EMBY_LIBRARIES — Emby library names to scan (master.conf) # SONARR_EMBY_LIBRARIES — Emby library names to scan (master.conf)
@@ -75,12 +75,12 @@ if ! command -v jq >/dev/null 2>&1; then
fi fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf" error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
if [[ -z "${SONARR_URL:-}" || -z "${SONARR_API_KEY:-}" ]]; then if [[ -z "${SONARR_URL:-}" || -z "${SONARR_API_KEY:-}" ]]; then
error "SONARR_URL / SONARR_API_KEY not configured — check master_host*.conf" error "SONARR_URL / SONARR_API_KEY not configured — check host*.conf"
exit 1 exit 1
fi fi
+1 -1
View File
@@ -50,7 +50,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_ZFS_REPORT_IGNORE_POOLS # HOST*_ZFS_REPORT_IGNORE_POOLS
# Pools to exclude from automatic scrub. Typically single-disk VM pools # Pools to exclude from automatic scrub. Typically single-disk VM pools
+4 -4
View File
@@ -216,7 +216,7 @@ TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
### Threshold Sizing ### Threshold Sizing
```bash ```bash
# master_host*.conf # host*.conf
HOST1_RAMDISK_SIZE="10G" HOST1_RAMDISK_SIZE="10G"
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
@@ -331,7 +331,7 @@ transcode_cleanup.sh --log # verbose per-file output
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback
# ── Per-Host (master_host*.conf) ─────────────────────────────────────────────── # ── Per-Host (host*.conf) ───────────────────────────────────────────────
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes" # ramdisk mount point HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes" # ramdisk mount point
HOST1_RAMDISK_SIZE="10G" # tmpfs ceiling (not a reservation) HOST1_RAMDISK_SIZE="10G" # tmpfs ceiling (not a reservation)
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
@@ -420,7 +420,7 @@ docker inspect Emby | grep -A3 "Mounts"
# Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB" # Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB"
# #
# If peak is close to RAMDISK_WARN_GB → increase ramdisk size: # If peak is close to RAMDISK_WARN_GB → increase ramdisk size:
# master_host1.conf # host1.conf
HOST1_RAMDISK_SIZE="12G" # increase by 2G HOST1_RAMDISK_SIZE="12G" # increase by 2G
HOST1_RAMDISK_WARN_GB=10.5 # adjust thresholds accordingly HOST1_RAMDISK_WARN_GB=10.5 # adjust thresholds accordingly
HOST1_RAMDISK_LOW_GB=8.5 HOST1_RAMDISK_LOW_GB=8.5
@@ -465,7 +465,7 @@ readlink /mnt/ram-transcode
### Increasing Ramdisk Size After Initial Setup ### Increasing Ramdisk Size After Initial Setup
```bash ```bash
# 1. Set new size and thresholds in master_host*.conf # 1. Set new size and thresholds in host*.conf
# 2. Unmount the existing ramdisk (no sessions should be active): # 2. Unmount the existing ramdisk (no sessions should be active):
umount /mnt/ramdisk_transcodes umount /mnt/ramdisk_transcodes
+1 -1
View File
@@ -53,7 +53,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_RAMDISK_SIZE # HOST*_RAMDISK_SIZE
# tmpfs ceiling (e.g. 10G). Must change together with WARN_GB and LOW_GB. # tmpfs ceiling (e.g. 10G). Must change together with WARN_GB and LOW_GB.
+1 -1
View File
@@ -65,7 +65,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD / HOST*_RAMDISK_LOW_GB # HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD / HOST*_RAMDISK_LOW_GB
# Aliased by detect_hosts() → RAMDISK_PATH / TRANSCODE_SSD / RAMDISK_LOW_GB. # Aliased by detect_hosts() → RAMDISK_PATH / TRANSCODE_SSD / RAMDISK_LOW_GB.
+1 -1
View File
@@ -65,7 +65,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD # HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD
# Ramdisk mount point and SSD fallback path. # Ramdisk mount point and SSD fallback path.
@@ -19,7 +19,7 @@
# ============================================================================================== # ==============================================================================================
# #
# Tier 1 — Strict Per-Container Monitoring # Tier 1 — Strict Per-Container Monitoring
# Applies only to containers explicitly configured in master_host*.conf. # Applies only to containers explicitly configured in host*.conf.
# #
# Memory hard limits — immediate restart if container exceeds MB ceiling # Memory hard limits — immediate restart if container exceeds MB ceiling
# Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of limit (no restart) # Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of limit (no restart)
@@ -145,7 +145,7 @@
# CONFIGURATION # CONFIGURATION
# ============================================================================================== # ==============================================================================================
# #
# master_host*.conf # host*.conf
# #
# HOST*_WATCHDOG_CONTAINERS # HOST*_WATCHDOG_CONTAINERS
# Memory hard limits per container. Format: "ContainerName:LimitInMB" # Memory hard limits per container. Format: "ContainerName:LimitInMB"
@@ -922,6 +922,7 @@ CYCLE_START=$(date +%s)
fi # WATCHDOG_SCAN_ALL fi # WATCHDOG_SCAN_ALL
# ── Send notifications ──────────────────────────────────────────────────────────────────── # ── Send notifications ────────────────────────────────────────────────────────────────────
flush_notify flush_notify
@@ -71,7 +71,7 @@
# RW_QBIT_ENABLED, RW_QBIT_DL_SOFT, RW_QBIT_DL_MEDIUM # RW_QBIT_ENABLED, RW_QBIT_DL_SOFT, RW_QBIT_DL_MEDIUM
# RW_CRITICAL_CONTAINERS — never paused or stopped regardless of pressure # RW_CRITICAL_CONTAINERS — never paused or stopped regardless of pressure
# #
# master_host*.conf (aliased by detect_hosts()) # host*.conf (aliased by detect_hosts())
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure # HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure # HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure
# HOST*_SABNZBD_URL, HOST*_SABNZBD_API_KEY # HOST*_SABNZBD_URL, HOST*_SABNZBD_API_KEY
+428
View File
@@ -0,0 +1,428 @@
#!/bin/bash
# ==============================================================================================
# ================================= Storage Watchdog ===========================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Pool and storage health monitoring — catches runaway data growth before it
# fills a pool. Runs as a single-pass script called by watchdog_orchestrator.sh
# every cycle. Sits between docker_watchdog.sh (container health) and
# system_watchdog.sh (last line of defense). Never reboots — detects, alerts,
# and optionally remediates.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Appdata Size Monitoring
# Two complementary checks run every cycle:
#
# Part 1 — Growth rate (zero-config catch-all):
# Reads per-container dir totals via du, compares to previous cycle baseline.
# Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused
# *.log scan inside that container's dir. No per-container config required —
# new containers are covered automatically. Baseline built on first cycle after
# boot; growth detection active from cycle 2.
#
# Part 2 — Absolute log size:
# Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB anywhere in
# WATCHDOG_APPDATA_PATHS. Catches logs already large but no longer actively
# growing. Independent strike counter per file.
#
# Strike System
# Reuses the same strike pattern as CPU/HTTP checks in docker_watchdog.sh:
#
# Strike 1 — warn + notify: condition first detected this run
# Strike 2 — warn + escalated notify: still present next cycle
# Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
# WATCHDOG_APPDATA_TRUNCATE_LOGS=true → truncate *.log in-place, clear strikes
# WATCHDOG_APPDATA_TRUNCATE_LOGS=false → critical notify, hold strikes until resolved
# Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
#
# Suppress Ceiling (WATCHDOG_APPDATA_SIZES)
# Containers in HOST*_WATCHDOG_APPDATA_SIZES suppress growth warnings while
# under their configured ceiling MB. Use ONLY when a container legitimately
# holds large stable data and would otherwise false-alarm (e.g. Tdarr cache).
# Zero-config growth detection covers everything else automatically.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Zero-config for new containers
# Growth rate detection requires no per-container configuration. Add a game
# server, spin up a new arr, install anything — it is monitored automatically
# from the second cycle after it appears. The suppress ceiling in conf is the
# exception, not the rule.
#
# Alert-only for data, truncate-only for logs
# Non-log growth (databases, game saves, caches) is detected and alerted but
# never touched. Only *.log / *.log.* files are candidates for truncation —
# and only when WATCHDOG_APPDATA_TRUNCATE_LOGS=true. Truncation zeroes the
# file in-place; the container keeps its open file handle, space is reclaimed
# immediately. Never deletes.
#
# Strike before acting
# One cycle of growth could be a legitimate library scan or game save burst.
# Three consecutive cycles of growth is a runaway. The strike system separates
# transient activity from sustained problems before any action fires.
#
# Silent when healthy
# Produces no output when all checks pass. Loud only when something needs
# attention.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master.conf
#
# WATCHDOG_CHECK_APPDATA
# Master toggle for all appdata checks (default: true)
#
# WATCHDOG_APPDATA_PATHS
# Array of paths to scan (e.g. "/mnt/docker-unraid/appdata")
#
# WATCHDOG_APPDATA_GROWTH_GB
# Per-cycle growth threshold in GB — flag containers growing more than this (default: 2)
#
# WATCHDOG_APPDATA_LOG_MAX_GB
# Absolute *.log file size alert threshold in GB (default: 2)
#
# WATCHDOG_APPDATA_TRUNCATE_LOGS
# Auto-truncate oversized *.log files on action cycle (default: false)
#
# WATCHDOG_APPDATA_STRIKE_LIMIT
# Consecutive cycles before action fires (default: 3)
#
# WATCHDOG_APPDATA_GROWTH_FILE
# Per-container size baseline — /tmp resets on reboot (correct: stale baseline
# after reboot would give false growth readings on first cycle)
#
# STORAGE_WATCHDOG_STATE_FILE
# Strike counts for this script — /tmp resets on reboot
#
# host*.conf (aliased by detect_hosts())
#
# HOST*_WATCHDOG_APPDATA_SIZES
# Per-container growth suppress ceilings in MB. Suppress growth alerts while
# a container's dir stays below this ceiling. Only needed when a container
# legitimately has large stable data. Growth detection covers everything else.
#
# ==============================================================================================
# STATE FILES
# ==============================================================================================
#
# STORAGE_WATCHDOG_STATE_FILE — strike counts (/tmp — resets on reboot ✅)
# WATCHDOG_APPDATA_GROWTH_FILE — per-container size baseline (/tmp — resets on reboot ✅)
#
# /tmp files reset on reboot — correct. Pre-reboot strikes and growth baselines are
# meaningless after a reboot. Both rebuild cleanly from cycle 1.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# storage_watchdog.sh
# Single-pass storage health check. Silent if all healthy.
#
# storage_watchdog.sh --dry-run
# Run all checks without truncating anything. Shows what would be actioned.
#
# storage_watchdog.sh --status
# Show configuration, active strikes, and growth baseline status. Then exit.
#
# storage_watchdog.sh --log
# Verbose output — every container checked, every size comparison, every decision.
#
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
acquire_lock
detect_hosts
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be truncated"
touch "$STORAGE_WATCHDOG_STATE_FILE" 2>/dev/null
touch "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null
# ==============================================================================================
# ━━━ Status ━━━
# ==============================================================================================
if [[ "$SHOW_STATUS" == true ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY STORAGE WATCHDOG STATUS ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_GEAR Appdata check: ${WATCHDOG_CHECK_APPDATA:-true}"
echo "$ICON_GEAR Paths: ${WATCHDOG_APPDATA_PATHS[*]:-none}"
echo "$ICON_GEAR Growth thresh: ${WATCHDOG_APPDATA_GROWTH_GB:-2}GB/cycle"
echo "$ICON_GEAR Log max: ${WATCHDOG_APPDATA_LOG_MAX_GB:-2}GB"
echo "$ICON_GEAR Truncate logs: ${WATCHDOG_APPDATA_TRUNCATE_LOGS:-false}"
echo "$ICON_GEAR Strike limit: ${WATCHDOG_APPDATA_STRIKE_LIMIT:-3}"
echo ""
echo "── Active Strikes ──"
if [[ -s "$STORAGE_WATCHDOG_STATE_FILE" ]]; then
while IFS=':' read -r _sk _sv; do
_sv_clean="${_sv//[^0-9]/}"
[[ "${_sv_clean:-0}" -gt 0 ]] && echo " $_sk$_sv_clean strikes"
done < "$STORAGE_WATCHDOG_STATE_FILE"
else
echo " none"
fi
echo ""
echo "── Growth Baseline ──"
if [[ -s "$WATCHDOG_APPDATA_GROWTH_FILE" ]]; then
_entries=$(wc -l < "$WATCHDOG_APPDATA_GROWTH_FILE")
_age=$(( $(date +%s) - $(stat -c %Y "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null || echo 0) ))
echo " Entries: $_entries containers Age: $(( _age / 60 ))m ago"
else
echo " No baseline yet (builds on first cycle after boot)"
fi
echo ""
echo "── Suppress Ceilings (this host) ──"
if [[ ${#WATCHDOG_APPDATA_SIZES[@]} -gt 0 ]]; then
for _c in "${!WATCHDOG_APPDATA_SIZES[@]}"; do
_ceil_gb=$(awk "BEGIN {printf \"%.0f\", ${WATCHDOG_APPDATA_SIZES[$_c]} / 1024}")
echo " $_c${_ceil_gb}GB"
done
else
echo " none configured"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
# ==============================================================================================
# ━━━ Appdata Size Monitoring ━━━
# ==============================================================================================
[[ "$WATCHDOG_CHECK_APPDATA" != "true" ]] && exit 0
log "$ICON_GEAR Storage watchdog — $MY_ID$(date '+%H:%M:%S')"
WARNINGS=0
_STRIKE_LIMIT=${WATCHDOG_APPDATA_STRIKE_LIMIT:-3}
_GROWTH_MB=$(( ${WATCHDOG_APPDATA_GROWTH_GB:-2} * 1024 ))
_LOG_KB=$(( ${WATCHDOG_APPDATA_LOG_MAX_GB:-2} * 1024 * 1024 ))
# Tracks files handled by Part 1 to prevent duplicate alerts in Part 2
declare -A _HANDLED=()
for _appdata_path in "${WATCHDOG_APPDATA_PATHS[@]:-}"; do
[[ -z "$_appdata_path" || ! -d "$_appdata_path" ]] && continue
# ── Part 1: Growth rate scan ──────────────────────────────────────────────────────────────
declare -A _PREV=()
if [[ -f "$WATCHDOG_APPDATA_GROWTH_FILE" ]]; then
while IFS='|' read -r _cn _cs _; do
[[ -n "$_cn" ]] && _PREV["$_cn"]="$_cs"
done < "$WATCHDOG_APPDATA_GROWTH_FILE"
fi
_growth_tmp=$(mktemp 2>/dev/null) || _growth_tmp=""
_now=$(date +%s)
while IFS= read -r _du_line; do
_curr_mb=$(echo "$_du_line" | awk '{print $1}')
_cdir=$(echo "$_du_line" | awk '{print $2}')
_cname=$(basename "$_cdir")
[[ -z "$_cname" || "$_cname" == "*" ]] && continue
[[ -n "$_growth_tmp" ]] && echo "${_cname}|${_curr_mb}|${_now}" >> "$_growth_tmp"
_prev_mb="${_PREV[$_cname]:-}"
[[ -z "$_prev_mb" ]] && continue # First run after boot — building baseline
_growth_mb=$(( _curr_mb - _prev_mb ))
_safe=$(echo "$_cname" | tr -cd '[:alnum:]_')
# Condition resolved — growth stopped, clear strikes
if [[ "$_growth_mb" -le 0 ]]; then
_existing=$(get_strikes "appdata_growth_${_safe}" "$STORAGE_WATCHDOG_STATE_FILE")
_existing="${_existing//[^0-9]/}"
[[ "${_existing:-0}" -gt 0 ]] && \
set_strikes "appdata_growth_${_safe}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
continue
fi
# Check suppress ceiling
_ceiling="${WATCHDOG_APPDATA_SIZES[$_cname]:-}"
if [[ -n "$_ceiling" && "$_curr_mb" -lt "$_ceiling" ]]; then
log "$_cname — growth suppressed (${_curr_mb}MB < ${_ceiling}MB ceiling)"
continue
fi
# Growth exceeds threshold — strike logic
if [[ "$_growth_mb" -ge "$_GROWTH_MB" ]]; then
_growth_gb=$(awk "BEGIN {printf \"%.1f\", $_growth_mb / 1024}")
_curr_gb=$(awk "BEGIN {printf \"%.1f\", $_curr_mb / 1024}")
_strikes=$(get_strikes "appdata_growth_${_safe}" "$STORAGE_WATCHDOG_STATE_FILE")
_strikes="${_strikes//[^0-9]/}"; _strikes="${_strikes:-0}"
[[ "$_strikes" -lt "$_STRIKE_LIMIT" ]] && {
_strikes=$(( _strikes + 1 ))
set_strikes "appdata_growth_${_safe}" "$_strikes" "$STORAGE_WATCHDOG_STATE_FILE"
}
warn "$_cname — grew ${_growth_gb}GB this cycle (total: ${_curr_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]"
(( WARNINGS++ ))
# Focused *.log scan inside the growing container's dir
_found_logs=()
while IFS= read -r _lf; do
[[ -n "$_lf" ]] && _found_logs+=("$_lf")
done < <(find "$_cdir" -maxdepth 3 -type f \
\( -name "*.log" -o -name "*.log.*" \) \
-size +${_LOG_KB}k -printf "%s %p\n" 2>/dev/null)
_log_summary=""
[[ ${#_found_logs[@]} -gt 0 ]] && _log_summary=$(printf '%s\n' "${_found_logs[@]}" | \
awk '{printf "%.1fGB %s | ", $1/1073741824, $2}' | head -c 200)
if [[ "$_strikes" -lt "$_STRIKE_LIMIT" ]]; then
if [[ -n "$_log_summary" ]]; then
notify "$_cname grew ${_growth_gb}GB on $(hostname) [strike ${_strikes}/${_STRIKE_LIMIT}] — logs: ${_log_summary}" \
"Storage Watchdog" "warning"
else
notify "$_cname grew ${_growth_gb}GB on $(hostname) [strike ${_strikes}/${_STRIKE_LIMIT}] — data growth, no log files" \
"Storage Watchdog" "warning"
fi
else
# Action cycle
error "$_cname — growth strike limit reached (${_STRIKE_LIMIT} consecutive cycles, ${_growth_gb}GB this cycle)"
if [[ "$WATCHDOG_APPDATA_TRUNCATE_LOGS" == "true" && ${#_found_logs[@]} -gt 0 ]]; then
for _lf_entry in "${_found_logs[@]}"; do
_lf_path=$(echo "$_lf_entry" | cut -d' ' -f2-)
_lf_gb=$(echo "$_lf_entry" | awk '{printf "%.1f", $1/1073741824}')
_HANDLED["$_lf_path"]=1
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would truncate $_lf_path (${_lf_gb}GB)"
else
if truncate -s 0 "$_lf_path" 2>/dev/null; then
success "Truncated runaway log: $_lf_path (was ${_lf_gb}GB)"
notify "Truncated runaway log on $(hostname): $_lf_path (was ${_lf_gb}GB)" \
"Storage Watchdog" "warning"
else
warn "Failed to truncate $_lf_path"
fi
fi
done
set_strikes "appdata_growth_${_safe}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
else
if [[ -n "$_log_summary" ]]; then
notify "$_cname appdata runaway on $(hostname)${_growth_gb}GB growth for ${_STRIKE_LIMIT} cycles — logs: ${_log_summary}" \
"Storage Watchdog" "critical"
else
notify "$_cname appdata runaway on $(hostname)${_growth_gb}GB growth for ${_STRIKE_LIMIT} cycles — data growth, manual investigation needed" \
"Storage Watchdog" "critical"
fi
fi
fi
# Mark found logs as handled to suppress Part 2 duplicates this cycle
for _lf_entry in "${_found_logs[@]}"; do
_HANDLED["$(echo "$_lf_entry" | cut -d' ' -f2-)"]=1
done
fi
done < <(du -sm "$_appdata_path"/*/ 2>/dev/null)
# Atomically update growth baseline
[[ -n "$_growth_tmp" ]] && mv "$_growth_tmp" "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null
# ── Part 2: Absolute log size scan ────────────────────────────────────────────────────────
# Catches *.log files already large but no longer actively growing this cycle.
# Same strike logic. Skips files already handled by Part 1 above.
declare -A _LOG_SEEN=()
while IFS= read -r _hit; do
[[ -z "$_hit" ]] && continue
_fpath=$(echo "$_hit" | cut -d' ' -f2-)
[[ -n "${_HANDLED[$_fpath]:-}" ]] && continue
_fsize_bytes=$(echo "$_hit" | awk '{print $1}')
_fsize_gb=$(awk "BEGIN {printf \"%.1f\", $_fsize_bytes / 1073741824}")
_safe_fkey=$(echo "$_fpath" | tr -cd '[:alnum:]_' | cut -c1-120)
_LOG_SEEN["appdata_log_${_safe_fkey}"]=1
_strikes=$(get_strikes "appdata_log_${_safe_fkey}" "$STORAGE_WATCHDOG_STATE_FILE")
_strikes="${_strikes//[^0-9]/}"; _strikes="${_strikes:-0}"
[[ "$_strikes" -lt "$_STRIKE_LIMIT" ]] && {
_strikes=$(( _strikes + 1 ))
set_strikes "appdata_log_${_safe_fkey}" "$_strikes" "$STORAGE_WATCHDOG_STATE_FILE"
}
warn "Oversized log: $_fpath (${_fsize_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]"
(( WARNINGS++ ))
if [[ "$_strikes" -lt "$_STRIKE_LIMIT" ]]; then
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]" \
"Storage Watchdog" "warning"
else
error "Oversized log persists for ${_STRIKE_LIMIT} cycles: $_fpath (${_fsize_gb}GB)"
if [[ "$WATCHDOG_APPDATA_TRUNCATE_LOGS" == "true" ]]; then
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would truncate $_fpath (${_fsize_gb}GB)"
else
if truncate -s 0 "$_fpath" 2>/dev/null; then
success "Truncated oversized log: $_fpath (was ${_fsize_gb}GB)"
notify "Truncated oversized log on $(hostname): $_fpath (was ${_fsize_gb}GB)" \
"Storage Watchdog" "warning"
set_strikes "appdata_log_${_safe_fkey}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
unset "_LOG_SEEN[appdata_log_${_safe_fkey}]"
else
warn "Failed to truncate $_fpath"
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) — ${_STRIKE_LIMIT} cycles, truncate failed" \
"Storage Watchdog" "critical"
fi
fi
else
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) — ${_STRIKE_LIMIT} cycles, manual intervention needed" \
"Storage Watchdog" "critical"
fi
fi
done < <(find "$_appdata_path" -maxdepth 4 -type f \
\( -name "*.log" -o -name "*.log.*" \) \
-size +${_LOG_KB}k -printf "%s %p\n" 2>/dev/null)
# Auto-clear strikes for log files no longer oversized this cycle
while IFS=':' read -r _sk _sv; do
[[ "$_sk" != appdata_log_* ]] && continue
_sv_clean="${_sv//[^0-9]/}"
[[ "${_sv_clean:-0}" -eq 0 ]] && continue
[[ -n "${_LOG_SEEN[$_sk]:-}" ]] && continue
set_strikes "$_sk" 0 "$STORAGE_WATCHDOG_STATE_FILE"
done < "$STORAGE_WATCHDOG_STATE_FILE" 2>/dev/null
unset _LOG_SEEN
done
unset _PREV _HANDLED
# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
if [[ "$WARNINGS" -gt 0 ]]; then
echo ""
echo "━━━ $ICON_GEAR Storage Watchdog — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo "$ICON_WATCHDOG Warnings: $WARNINGS"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
else
log "Storage healthy ✅ ($(date '+%H:%M:%S'))"
fi
+22 -30
View File
@@ -55,11 +55,11 @@
# check_local_disk_temps() — pre-rsync temp check with exit codes 0/1/2 # check_local_disk_temps() — pre-rsync temp check with exit codes 0/1/2
# stop/start local containers added alongside existing remote variants # stop/start local containers added alongside existing remote variants
# #
# v3.1 Three-file config split — master.conf + master_host1.conf + master_host2.conf # v3.1 Three-file config split — master.conf + host1.conf + host2.conf
# load_config.sh introduced — auto-discovers all master_host*.conf files # load_config.sh introduced — auto-discovers all host*.conf files
# detect_hosts() rewritten — sets MY_ID/REMOTE_ID and aliases all HOST* vars # detect_hosts() rewritten — sets MY_ID/REMOTE_ID and aliases all HOST* vars
# All scripts now source load_config.sh instead of conf files directly # All scripts now source load_config.sh instead of conf files directly
# Adding a new server = add master_host*.conf, zero script changes required # Adding a new server = add host*.conf, zero script changes required
# #
# v3.2 check_remote_disks() rewritten — three-tier detection: # v3.2 check_remote_disks() rewritten — three-tier detection:
# Tier 1: array disk paths (/mnt/disk*/sharename) # Tier 1: array disk paths (/mnt/disk*/sharename)
@@ -78,10 +78,6 @@
# Emby immediately removes ghost entries — no user-facing file-not-found errors # Emby immediately removes ghost entries — no user-facing file-not-found errors
# Called by lidarr/sonarr/radarr_cleanup.sh when files are deleted # Called by lidarr/sonarr/radarr_cleanup.sh when files are deleted
# #
# v3.4 Silent-by-default output model
# info() and success() now gated by SILENT_MODE — only warn/error always visible
# Exception: monitor scripts designed to produce output stay verbose
# Reduces notification spam — ecosystem only speaks when something is wrong
# #
# Three new safety functions added: # Three new safety functions added:
# check_unraid_version_parity() — refuses remote ops on version mismatch # check_unraid_version_parity() — refuses remote ops on version mismatch
@@ -190,27 +186,16 @@ ICON_SUCCESS="✅"
# ============================================================================================== # ==============================================================================================
# Standardised output functions used across all scripts. # Standardised output functions used across all scripts.
# #
# Silent-by-default model: # Two-tier model:
# SILENT_MODE=true (default) — only warn() and error() produce output # echo — always visible — summaries, status conclusions, section headers
# SILENT_MODE=false — all functions produce output # warn() — always visible — state transitions, warnings, important events
# --log flag — enables ENABLE_LOGGING (detailed [LOG] lines) # error() — always visible — something broke
# # log() — only with --log flag — per-item detail, internal checks
# Rules:
# error() — always visible — something broke
# warn() — always visible — something needs attention
# info() — silent by default — operational detail, visible when SILENT_MODE=false
# success() — silent by default — confirmation, visible when SILENT_MODE=false
# log() — debug detail — only when ENABLE_LOGGING=true
#
# Exception — monitor scripts are designed to produce output and set SILENT_MODE=false
# at the top of the script. All other scripts use the silent default.
# #
# All output goes to stdout — callers can redirect as needed. # All output goes to stdout — callers can redirect as needed.
info() { [[ "${SILENT_MODE:-true}" == false ]] && echo "$ICON_INFO [INFO] $*"; return 0; }
warn() { echo "$ICON_WARN [WARN] $*"; } warn() { echo "$ICON_WARN [WARN] $*"; }
error() { echo "$ICON_ERROR [ERROR] $*"; } error() { echo "$ICON_ERROR [ERROR] $*"; }
success() { [[ "${SILENT_MODE:-true}" == false ]] && echo "$ICON_SUCCESS [OK] $*"; return 0; }
log() { log() {
[[ "${ENABLE_LOGGING:-false}" == true ]] && echo "[LOG] $*" [[ "${ENABLE_LOGGING:-false}" == true ]] && echo "[LOG] $*"
@@ -223,7 +208,7 @@ log() {
# Sends a notification via unRAID native system and/or Discord webhook. # Sends a notification via unRAID native system and/or Discord webhook.
# Both channels are optional and independently controlled: # Both channels are optional and independently controlled:
# NOTIFY_UNRAID — shared toggle in master.conf # NOTIFY_UNRAID — shared toggle in master.conf
# MY_DISCORD_WEBHOOK — per-host in master_host*.conf, set by detect_hosts() # MY_DISCORD_WEBHOOK — per-host in host*.conf, set by detect_hosts()
# #
# Severity levels: normal, warning, alert # Severity levels: normal, warning, alert
# Usage: notify "message" "subject" "severity" # Usage: notify "message" "subject" "severity"
@@ -254,7 +239,7 @@ notify() {
-d "$payload" "$MY_DISCORD_WEBHOOK" >/dev/null 2>&1; then -d "$payload" "$MY_DISCORD_WEBHOOK" >/dev/null 2>&1; then
log "$ICON_NOTIFY Discord notification sent" log "$ICON_NOTIFY Discord notification sent"
else else
warn "Discord notification failed — check HOST*_DISCORD_WEBHOOK in master_host*.conf" warn "Discord notification failed — check HOST*_DISCORD_WEBHOOK in host*.conf"
fi fi
fi fi
} }
@@ -363,7 +348,7 @@ validate_int() {
# ── HOST DETECTION ──────────────────────────────────────────────────────────────────────────── # ── HOST DETECTION ────────────────────────────────────────────────────────────────────────────
# ============================================================================================== # ==============================================================================================
# Determines which server is local and which is remote by comparing hostname against # Determines which server is local and which is remote by comparing hostname against
# all HOST* values discovered from master_host*.conf files. # all HOST* values discovered from host*.conf files.
# #
# Sets: # Sets:
# MY_ID — "HOST1" or "HOST2" (the role key, not the hostname) # MY_ID — "HOST1" or "HOST2" (the role key, not the hostname)
@@ -384,6 +369,7 @@ validate_int() {
# WATCHDOG_REQUIRED_CONTAINERS ← HOST*_WATCHDOG_REQUIRED_CONTAINERS # WATCHDOG_REQUIRED_CONTAINERS ← HOST*_WATCHDOG_REQUIRED_CONTAINERS
# WATCHDOG_SCAN_IGNORE ← HOST*_WATCHDOG_SCAN_IGNORE # WATCHDOG_SCAN_IGNORE ← HOST*_WATCHDOG_SCAN_IGNORE
# WATCHDOG_DEPENDENCIES ← HOST*_WATCHDOG_DEPENDENCIES (associative) # WATCHDOG_DEPENDENCIES ← HOST*_WATCHDOG_DEPENDENCIES (associative)
# WATCHDOG_APPDATA_SIZES ← HOST*_WATCHDOG_APPDATA_SIZES (associative)
# NETWORK_CONNECT_CONTAINERS ← HOST*_NETWORK_CONNECT_CONTAINERS # NETWORK_CONNECT_CONTAINERS ← HOST*_NETWORK_CONNECT_CONTAINERS
# NETWORK_CONNECT_NETWORKS ← HOST*_NETWORK_CONNECT_NETWORKS # NETWORK_CONNECT_NETWORKS ← HOST*_NETWORK_CONNECT_NETWORKS
# MEDIA_PERMISSION_SHARES ← HOST*_MEDIA_PERMISSION_SHARES # MEDIA_PERMISSION_SHARES ← HOST*_MEDIA_PERMISSION_SHARES
@@ -423,7 +409,7 @@ detect_hosts() {
local host_var host_val local host_var host_val
# Scan all HOST* vars that hold a hostname value # Scan all HOST* vars that hold a hostname value
# HOST1, HOST2, HOST3 etc. — all defined in master_host*.conf files # HOST1, HOST2, HOST3 etc. — all defined in host*.conf files
for host_var in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do for host_var in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do
host_val="${!host_var:-}" host_val="${!host_var:-}"
[[ -z "$host_val" ]] && continue [[ -z "$host_val" ]] && continue
@@ -435,7 +421,7 @@ detect_hosts() {
if [[ -z "$MY_ID" ]]; then if [[ -z "$MY_ID" ]]; then
error "Unknown host: $local_hostname" error "Unknown host: $local_hostname"
error "Hostname must match a HOST* value in master_host*.conf" error "Hostname must match a HOST* value in host*.conf"
error "Available: $(for h in HOST1 HOST2 HOST3 HOST4; do error "Available: $(for h in HOST1 HOST2 HOST3 HOST4; do
[[ -n "${!h:-}" ]] && echo -n "${!h} "; done)" [[ -n "${!h:-}" ]] && echo -n "${!h} "; done)"
exit 1 exit 1
@@ -459,7 +445,7 @@ detect_hosts() {
local ssh_key_var="${MY_ID}_SSH_KEY" local ssh_key_var="${MY_ID}_SSH_KEY"
SSH_KEY="${!ssh_key_var:-}" SSH_KEY="${!ssh_key_var:-}"
if [[ -z "$SSH_KEY" ]]; then if [[ -z "$SSH_KEY" ]]; then
error "Missing SSH key: ${ssh_key_var} not set in master_host*.conf" error "Missing SSH key: ${ssh_key_var} not set in host*.conf"
exit 1 exit 1
fi fi
@@ -467,6 +453,11 @@ detect_hosts() {
MY_DISCORD_WEBHOOK_VAR="${MY_ID}_DISCORD_WEBHOOK" MY_DISCORD_WEBHOOK_VAR="${MY_ID}_DISCORD_WEBHOOK"
MY_DISCORD_WEBHOOK="${!MY_DISCORD_WEBHOOK_VAR:-}" MY_DISCORD_WEBHOOK="${!MY_DISCORD_WEBHOOK_VAR:-}"
OWNER_NAME_VAR="${MY_ID}_OWNER"
OWNER_NAME="${!OWNER_NAME_VAR:-}"
OWNER_EMAIL_VAR="${MY_ID}_OWNER_EMAIL"
OWNER_EMAIL="${!OWNER_EMAIL_VAR:-}"
EMBY_CONTAINER_VAR="${MY_ID}_EMBY_CONTAINER" EMBY_CONTAINER_VAR="${MY_ID}_EMBY_CONTAINER"
EMBY_CONTAINER="${!EMBY_CONTAINER_VAR:-}" EMBY_CONTAINER="${!EMBY_CONTAINER_VAR:-}"
EMBY_URL_VAR="${MY_ID}_EMBY_URL" EMBY_URL_VAR="${MY_ID}_EMBY_URL"
@@ -598,6 +589,7 @@ detect_hosts() {
_alias_assoc "WATCHDOG_CONTAINERS" _alias_assoc "WATCHDOG_CONTAINERS"
_alias_assoc "WATCHDOG_CONTAINER_URLS" _alias_assoc "WATCHDOG_CONTAINER_URLS"
_alias_assoc "WATCHDOG_DEPENDENCIES" _alias_assoc "WATCHDOG_DEPENDENCIES"
_alias_assoc "WATCHDOG_APPDATA_SIZES"
# ── Output ──────────────────────────────────────────────────────────────── # ── Output ────────────────────────────────────────────────────────────────
info "$ICON_HOST Host: $LOCAL_SERVER_NAME$REMOTE_SERVER_NAME" info "$ICON_HOST Host: $LOCAL_SERVER_NAME$REMOTE_SERVER_NAME"
@@ -1779,7 +1771,7 @@ show_status() {
echo "Remote ID: ${REMOTE_ID:-not set}" echo "Remote ID: ${REMOTE_ID:-not set}"
echo "unRAID ver: $local_ver" echo "unRAID ver: $local_ver"
echo "Profile: ${PROFILE_NAME:-n/a}" echo "Profile: ${PROFILE_NAME:-n/a}"
echo "Silent mode: ${SILENT_MODE:-true}"
echo "DryRun: ${DRY_RUN:-false}" echo "DryRun: ${DRY_RUN:-false}"
echo "Logging: ${ENABLE_LOGGING:-false}" echo "Logging: ${ENABLE_LOGGING:-false}"
echo "SSH Key: ${SSH_KEY:-not set}" echo "SSH Key: ${SSH_KEY:-not set}"
+13 -13
View File
@@ -8,19 +8,19 @@
# ── WHAT THIS SCRIPT DOES ───────────────────────────────────────────────────────────────────── # ── WHAT THIS SCRIPT DOES ─────────────────────────────────────────────────────────────────────
# 1. Detects which server it's running on via detect_hosts() (MY_ID) # 1. Detects which server it's running on via detect_hosts() (MY_ID)
# 2. Configures sparse checkout to exclude other servers' credential files # 2. Configures sparse checkout to exclude other servers' credential files
# Each server only pulls its own master_host*.conf — never sees peer credentials # Each server only pulls its own host*.conf — never sees peer credentials
# 3. Pulls or clones latest scripts from Gitea # 3. Pulls or clones latest scripts from Gitea
# 4. Sets executable permissions on all .sh files # 4. Sets executable permissions on all .sh files
# #
# ── SPARSE CHECKOUT ─────────────────────────────────────────────────────────────────────────── # ── SPARSE CHECKOUT ───────────────────────────────────────────────────────────────────────────
# Sparse checkout ensures each server only receives its own host conf: # Sparse checkout ensures each server only receives its own host conf:
# HOST1 pulls: master.conf + master_host1.conf + all scripts # HOST1 pulls: master.conf + host1.conf + all scripts
# HOST1 skips: master_host2.conf, master_host3.conf etc. # HOST1 skips: host2.conf, host3.conf etc.
# HOST2 pulls: master.conf + master_host2.conf + all scripts # HOST2 pulls: master.conf + host2.conf + all scripts
# HOST2 skips: master_host1.conf, master_host3.conf etc. # HOST2 skips: host1.conf, host3.conf etc.
# #
# Adding a new server: # Adding a new server:
# Create master_host3.conf in the repo # Create host3.conf in the repo
# All existing servers automatically exclude it on next pull # All existing servers automatically exclude it on next pull
# New server gets only its own conf ✅ # New server gets only its own conf ✅
# #
@@ -125,7 +125,7 @@ fi
# ============================================================================================== # ==============================================================================================
# ━━━ Sparse Checkout Configuration ━━━ # ━━━ Sparse Checkout Configuration ━━━
# ============================================================================================== # ==============================================================================================
# Build the list of master_host*.conf files that belong to OTHER servers. # Build the list of host*.conf files that belong to OTHER servers.
# This server pulls everything EXCEPT those files. # This server pulls everything EXCEPT those files.
# MY_ID is set by detect_hosts() — e.g. "HOST1" # MY_ID is set by detect_hosts() — e.g. "HOST1"
@@ -137,7 +137,7 @@ configure_sparse_checkout() {
# Enable sparse checkout # Enable sparse checkout
git -C "$repo_dir" config core.sparseCheckout true 2>/dev/null git -C "$repo_dir" config core.sparseCheckout true 2>/dev/null
# Build exclusion list — all master_host*.conf files except MY_ID's # Build exclusion list — all host*.conf files except MY_ID's
local sparse_file="$repo_dir/.git/info/sparse-checkout" local sparse_file="$repo_dir/.git/info/sparse-checkout"
mkdir -p "$(dirname "$sparse_file")" mkdir -p "$(dirname "$sparse_file")"
@@ -145,9 +145,9 @@ configure_sparse_checkout() {
echo "/*" > "$sparse_file" echo "/*" > "$sparse_file"
# Exclude each other server's conf file # Exclude each other server's conf file
# Find all master_host*.conf files present in the repo # Find all host*.conf files present in the repo
local excluded=0 local excluded=0
for conf_file in "$repo_dir"/master_host*.conf; do for conf_file in "$repo_dir"/host*.conf; do
[[ -f "$conf_file" ]] || continue [[ -f "$conf_file" ]] || continue
local conf_name local conf_name
conf_name=$(basename "$conf_file") conf_name=$(basename "$conf_file")
@@ -193,7 +193,7 @@ SYNC_SUCCESS=false
if [[ "$DRY_RUN" == true ]]; then if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would sync $REPO_SSH$TARGET_DIR" warn "DRY RUN — would sync $REPO_SSH$TARGET_DIR"
warn "DRY RUN — would configure sparse checkout for $MY_ID" warn "DRY RUN — would configure sparse checkout for $MY_ID"
warn "DRY RUN — would exclude peer master_host*.conf files" warn "DRY RUN — would exclude peer host*.conf files"
SYNC_SUCCESS=true SYNC_SUCCESS=true
else else
mkdir -p "$TARGET_DIR" mkdir -p "$TARGET_DIR"
@@ -233,7 +233,7 @@ else
echo " Clone successful" echo " Clone successful"
# Configure sparse checkout after clone # Configure sparse checkout after clone
# Now all master_host*.conf files are present — can detect exclusions # Now all host*.conf files are present — can detect exclusions
configure_sparse_checkout "$TARGET_DIR" configure_sparse_checkout "$TARGET_DIR"
# Apply sparse checkout — removes excluded files from working tree # Apply sparse checkout — removes excluded files from working tree
@@ -267,7 +267,7 @@ echo "━━━━━ $ICON_SUMMARY GIT SYNC SUMMARY ━━━━━"
echo "$ICON_NET Repo: $REPO_SSH" echo "$ICON_NET Repo: $REPO_SSH"
echo "$ICON_GEAR Target: $TARGET_DIR" echo "$ICON_GEAR Target: $TARGET_DIR"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)" echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_LOCK Excluded: peer master_host*.conf files" echo "$ICON_LOCK Excluded: peer host*.conf files"
echo "$ICON_TIME Duration: $(format_duration $((END - START)))" echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
if [[ "$DRY_RUN" == true ]]; then if [[ "$DRY_RUN" == true ]]; then
echo "$ICON_WARN Status: DRY RUN — no changes made" echo "$ICON_WARN Status: DRY RUN — no changes made"
+21 -2
View File
@@ -10,7 +10,7 @@
# HOST2 never sees HOST1 credentials — clean separation at the file level. # HOST2 never sees HOST1 credentials — clean separation at the file level.
# #
# DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf. # DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf.
# DO NOT put HOST2 variables here — they belong in master_host2.conf. # DO NOT put HOST2 variables here — they belong in host2.conf.
# #
# ── INDEX ───────────────────────────────────────────────────────────────────────────────────── # ── INDEX ─────────────────────────────────────────────────────────────────────────────────────
# #
@@ -59,6 +59,12 @@
# RADARR URL, API key, path map # RADARR URL, API key, path map
# ARR RECOVERY per-arr recovery toggles # ARR RECOVERY per-arr recovery toggles
# #
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
# SYSTEM WATCHDOG per-host check toggles and NIC configuration
#
# ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
# RESOURCE MANAGER containers paused/stopped under memory pressure
#
# ============================================================================================== # ==============================================================================================
# ============================================================================================== # ==============================================================================================
@@ -70,6 +76,8 @@
# SSH key used for all server-to-server operations — rsync, failover container commands. # SSH key used for all server-to-server operations — rsync, failover container commands.
# Must be in /root/.ssh/ and authorised in HOST2's /root/.ssh/authorized_keys. # Must be in /root/.ssh/ and authorised in HOST2's /root/.ssh/authorized_keys.
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation" HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
HOST1_OWNER="gmer4lfe"
HOST1_OWNER_EMAIL="gmer4lfe@gmail.com"
# ━━━ Emby ━━━ # ━━━ Emby ━━━
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh, # Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
@@ -319,6 +327,17 @@
["NextCloud"]="Postgres-NextCloud" ["NextCloud"]="Postgres-NextCloud"
) )
# Per-container appdata growth suppress ceilings in MB.
# ONLY needed in specific cases — growth rate detection covers all containers automatically.
# Use this when a container legitimately has large stable data and you want to guarantee
# it never triggers a false-positive growth alert. Growth warnings are suppressed while the
# container's dir stays below this ceiling; above it, warnings resume as normal.
# 50GB=51200 25GB=25600 20GB=20480 15GB=15360 10GB=10240 5GB=5120
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
["Tdarr"]="25600" # 25GB — transcode cache grows legitimately during active jobs
["7dtd"]="20480" # 20GB — game server world data, expected to be large
)
# ━━━ Docker Network Connect ━━━ # ━━━ Docker Network Connect ━━━
# Containers connected to custom networks at array start by docker_network_connect.sh. # Containers connected to custom networks at array start by docker_network_connect.sh.
# Networks created if they don't exist — idempotent, safe to re-run. # Networks created if they don't exist — idempotent, safe to re-run.
@@ -354,7 +373,7 @@
# ━━━ Fallback Tiers — HOST1 Runs for HOST2 ━━━ # ━━━ Fallback Tiers — HOST1 Runs for HOST2 ━━━
# Containers HOST1 starts when HOST2 goes down. # Containers HOST1 starts when HOST2 goes down.
# Tier 1 is always immediate — vital services cannot wait. # Tier 1 is always immediate — vital services cannot wait.
# Higher tiers activate after HOST2_TIER*_DELAY minutes (set in master_host2.conf). # Higher tiers activate after HOST2_TIER*_DELAY minutes (set in host2.conf).
FALLBACK_HOST1_COVERS_HOST2_TIER1=( FALLBACK_HOST1_COVERS_HOST2_TIER1=(
"Gmer4Lfe.us" "Gmer4Lfe.us"
"VaultWarden-Jayred365" "VaultWarden-Jayred365"
+36 -6
View File
@@ -10,7 +10,7 @@
# HOST1 never sees HOST2 credentials — clean separation at the file level. # HOST1 never sees HOST2 credentials — clean separation at the file level.
# #
# DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf. # DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf.
# DO NOT put HOST1 variables here — they belong in master_host1.conf. # DO NOT put HOST1 variables here — they belong in host1.conf.
# #
# ── STATUS ──────────────────────────────────────────────────────────────────────────────────── # ── STATUS ────────────────────────────────────────────────────────────────────────────────────
# HOST2 is currently being rebuilt — most sections scaffolded, fill in when back online. # HOST2 is currently being rebuilt — most sections scaffolded, fill in when back online.
@@ -61,6 +61,12 @@
# RADARR URL, API key, path map # RADARR URL, API key, path map
# ARR RECOVERY per-arr recovery toggles (no Lidarr on HOST2) # ARR RECOVERY per-arr recovery toggles (no Lidarr on HOST2)
# #
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
# SYSTEM WATCHDOG per-host check toggles and NIC configuration
#
# ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
# RESOURCE MANAGER containers paused/stopped under memory pressure
#
# ============================================================================================== # ==============================================================================================
# ============================================================================================== # ==============================================================================================
@@ -72,6 +78,8 @@
# SSH key used for all server-to-server operations — rsync, failover container commands. # SSH key used for all server-to-server operations — rsync, failover container commands.
# Must be in /root/.ssh/ and authorised in HOST1's /root/.ssh/authorized_keys. # Must be in /root/.ssh/ and authorised in HOST1's /root/.ssh/authorized_keys.
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key" HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"
HOST2_OWNER="jayred365"
HOST2_OWNER_EMAIL="" # fill in when HOST2 is back online
# ━━━ Emby ━━━ # ━━━ Emby ━━━
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh, # Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
@@ -284,6 +292,14 @@
# ["Authelia"]="Mariadb-Authelia Redis-Authelia" # ["Authelia"]="Mariadb-Authelia Redis-Authelia"
) )
# Per-container appdata growth suppress ceilings in MB.
# ONLY needed in specific cases — growth rate detection covers all containers automatically.
# Use when a container legitimately has large stable data and you want to suppress false-positive
# growth alerts. Add entries here only when a container triggers warnings it shouldn't.
declare -A HOST2_WATCHDOG_APPDATA_SIZES=(
# add HOST2 suppress entries here only as needed
)
# ━━━ Docker Network Connect ━━━ # ━━━ Docker Network Connect ━━━
# Containers connected to custom networks at array start by docker_network_connect.sh. # Containers connected to custom networks at array start by docker_network_connect.sh.
# Networks created if they don't exist — idempotent, safe to re-run. # Networks created if they don't exist — idempotent, safe to re-run.
@@ -318,7 +334,7 @@
# ━━━ Fallback Tiers — HOST2 Runs for HOST1 ━━━ # ━━━ Fallback Tiers — HOST2 Runs for HOST1 ━━━
# Containers HOST2 starts when HOST1 goes down. # Containers HOST2 starts when HOST1 goes down.
# Tier 1 is always immediate — vital services cannot wait. # Tier 1 is always immediate — vital services cannot wait.
# Higher tiers activate after HOST1_TIER*_DELAY minutes (set in master_host1.conf). # Higher tiers activate after HOST1_TIER*_DELAY minutes (set in host1.conf).
FALLBACK_HOST2_COVERS_HOST1_TIER1=( FALLBACK_HOST2_COVERS_HOST1_TIER1=(
"Gmer4Lfe.com" "Gmer4Lfe.com"
"Emby" "Emby"
@@ -501,10 +517,6 @@
HOST2_RADARR_RECOVERY=true HOST2_RADARR_RECOVERY=true
# HOST2_LIDARR_RECOVERY not set — Lidarr does not run on HOST2 # HOST2_LIDARR_RECOVERY not set — Lidarr does not run on HOST2
# ==============================================================================================
# ──────────────────────── End Of HOST2 Variables ──────────────────────────────────────────────
# ==============================================================================================
# ============================================================================================== # ==============================================================================================
# ── SYSTEM WATCHDOG ─────────────────────────────────────────────────────────────────────────── # ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
# ============================================================================================== # ==============================================================================================
@@ -597,6 +609,24 @@
# DISABLED — rebuild workloads may legitimately peg CPU. Enable after rebuild. # DISABLED — rebuild workloads may legitimately peg CPU. Enable after rebuild.
HOST2_SYS_WATCHDOG_CHECK_RUNAWAY=false HOST2_SYS_WATCHDOG_CHECK_RUNAWAY=false
# ==============================================================================================
# ── RESOURCE MANAGER ──────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Containers to manage under pressure — see master.conf RW_CRITICAL_CONTAINERS for exclusions.
# docker pause at medium pressure (RAM < RW_RAM_MEDIUM_GB or load > medium threshold)
# Suspended in-place — instant to pause/unpause, no state lost, no restart delay.
HOST2_RW_PAUSE_CONTAINERS=(
# fill in when HOST2 is back online
)
# docker stop at hard pressure (RAM < RW_RAM_HARD_GB)
# Full stop — these are optional/heavy services that free significant RAM when stopped.
# resource_watchdog.sh restarts them when pressure fully clears (RAM >= RW_RAM_RECOVER_GB).
HOST2_RW_STOP_CONTAINERS=(
# fill in when HOST2 is back online
)
# ============================================================================================== # ==============================================================================================
# ──────────────────────── End Of HOST2 Variables ────────────────────────────────────────────── # ──────────────────────── End Of HOST2 Variables ──────────────────────────────────────────────
# ============================================================================================== # ==============================================================================================
+11 -11
View File
@@ -7,23 +7,23 @@
# #
# ── HOW IT WORKS ────────────────────────────────────────────────────────────────────────────── # ── HOW IT WORKS ──────────────────────────────────────────────────────────────────────────────
# 1. Sources master.conf (shared config — hostnames, thresholds, toggles, profiles, job lists) # 1. Sources master.conf (shared config — hostnames, thresholds, toggles, profiles, job lists)
# 2. Auto-discovers and sources all master_host*.conf files in the same directory # 2. Auto-discovers and sources all host*.conf files in the same directory
# Each host conf extends the shared profile arrays and adds host-specific credentials # Each host conf extends the shared profile arrays and adds host-specific credentials
# 3. Sources common.sh (shared functions — detect_hosts, logging, notifications etc.) # 3. Sources common.sh (shared functions — detect_hosts, logging, notifications etc.)
# #
# ── WHY THIS EXISTS ─────────────────────────────────────────────────────────────────────────── # ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────
# Without this loader every script had to explicitly source each conf file: # Without this loader every script had to explicitly source each conf file:
# source master.conf # source master.conf
# source master_host1.conf # source host1.conf
# source master_host2.conf # source host2.conf
# source common.sh # source common.sh
# #
# Adding a new server meant updating every script. # Adding a new server meant updating every script.
# With this loader — add master_host3.conf to the git repo and every server # With this loader — add host3.conf to the git repo and every server
# auto-discovers it on next git pull. Zero script changes required. Ever. # auto-discovers it on next git pull. Zero script changes required. Ever.
# #
# ── ADDING A NEW SERVER ─────────────────────────────────────────────────────────────────────── # ── ADDING A NEW SERVER ───────────────────────────────────────────────────────────────────────
# 1. Create master_host3.conf following the same structure as HOST1/HOST2 # 1. Create host3.conf following the same structure as HOST1/HOST2
# 2. Commit and push to git repo # 2. Commit and push to git repo
# 3. All servers pull it automatically — no other changes needed # 3. All servers pull it automatically — no other changes needed
# #
@@ -40,9 +40,9 @@
# source "$SCRIPT_DIR/load_config.sh" # source "$SCRIPT_DIR/load_config.sh"
# #
# ── SPARSE CHECKOUT NOTE ────────────────────────────────────────────────────────────────────── # ── SPARSE CHECKOUT NOTE ──────────────────────────────────────────────────────────────────────
# Sparse checkout controls which master_host*.conf files each server receives. # Sparse checkout controls which host*.conf files each server receives.
# HOST1 only pulls master_host1.conf — never HOST2's credentials. # HOST1 only pulls host1.conf — never HOST2's credentials.
# HOST2 only pulls master_host2.conf — never HOST1's credentials. # HOST2 only pulls host2.conf — never HOST1's credentials.
# This loader sources whatever conf files ARE present — sparse checkout handles the rest. # This loader sources whatever conf files ARE present — sparse checkout handles the rest.
# Both servers pull all non-credential conf files (master.conf, common.sh, load_config.sh). # Both servers pull all non-credential conf files (master.conf, common.sh, load_config.sh).
# #
@@ -63,14 +63,14 @@
fi fi
source "$LOAD_CONFIG_DIR/master.conf" source "$LOAD_CONFIG_DIR/master.conf"
# ━━━ Auto-discover and source all master_host*.conf files ━━━ # ━━━ Auto-discover and source all host*.conf files ━━━
# Sorted for consistent load order — HOST1 before HOST2 before HOST3 etc. # Sorted for consistent load order — HOST1 before HOST2 before HOST3 etc.
# Each host conf extends the shared PROFILE_* arrays and adds host-specific vars. # Each host conf extends the shared PROFILE_* arrays and adds host-specific vars.
# Missing files are silently skipped — sparse checkout intentionally withholds some. # Missing files are silently skipped — sparse checkout intentionally withholds some.
# At least one host conf must be present or the ecosystem has no identity to work with. # At least one host conf must be present or the ecosystem has no identity to work with.
_host_confs_loaded=0 _host_confs_loaded=0
for _conf in $(ls "$LOAD_CONFIG_DIR"/master_host*.conf 2>/dev/null | sort); do for _conf in $(ls "$LOAD_CONFIG_DIR"/host*.conf 2>/dev/null | sort); do
if [[ -f "$_conf" ]]; then if [[ -f "$_conf" ]]; then
source "$_conf" source "$_conf"
(( _host_confs_loaded++ )) (( _host_confs_loaded++ ))
@@ -80,7 +80,7 @@
done done
if [[ "$_host_confs_loaded" -eq 0 ]]; then if [[ "$_host_confs_loaded" -eq 0 ]]; then
echo "[FATAL] No master_host*.conf files found in $LOAD_CONFIG_DIR" >&2 echo "[FATAL] No host*.conf files found in $LOAD_CONFIG_DIR" >&2
echo "[FATAL] At least one host conf required — check git pull and sparse checkout" >&2 echo "[FATAL] At least one host conf required — check git pull and sparse checkout" >&2
exit 1 exit 1
fi fi
+76 -51
View File
@@ -8,10 +8,10 @@
# ── HOW THE THREE-FILE SYSTEM WORKS ────────────────────────────────────────────────────────── # ── HOW THE THREE-FILE SYSTEM WORKS ──────────────────────────────────────────────────────────
# Scripts source all three files at startup: # Scripts source all three files at startup:
# source master.conf ← shared config (this file) # source master.conf ← shared config (this file)
# source master_host1.conf ← HOST1 credentials, shares, container lists # source host1.conf ← HOST1 credentials, shares, container lists
# source master_host2.conf ← HOST2 credentials, shares, container lists # source host2.conf ← HOST2 credentials, shares, container lists
# #
# Sparse checkout (git) ensures each server only pulls its own master_host*.conf. # Sparse checkout (git) ensures each server only pulls its own host*.conf.
# HOST2 never sees HOST1 credentials. HOST1 never sees HOST2 credentials. # HOST2 never sees HOST1 credentials. HOST1 never sees HOST2 credentials.
# #
# What belongs here: thresholds, toggles, intervals, profiles, job lists # What belongs here: thresholds, toggles, intervals, profiles, job lists
@@ -99,7 +99,7 @@
# Must match the exact unRAID hostname AND Tailscale device name (case sensitive). # Must match the exact unRAID hostname AND Tailscale device name (case sensitive).
# detect_hosts() in common.sh matches the local hostname against these to set MY_ID / REMOTE_ID. # detect_hosts() in common.sh matches the local hostname against these to set MY_ID / REMOTE_ID.
# Tailscale IP resolution uses these names — no hardcoded IPs needed. # Tailscale IP resolution uses these names — no hardcoded IPs needed.
# To add a new server: add HOST3="unRAID-NewServer" here + create master_host3.conf. # To add a new server: add HOST3="unRAID-NewServer" here + create host3.conf.
HOST1="unRAID-Gmer4Lfe" HOST1="unRAID-Gmer4Lfe"
HOST2="unRAID-Jayred365" HOST2="unRAID-Jayred365"
@@ -145,7 +145,7 @@
# Manages the relationship lifecycle between two unRAID servers. # Manages the relationship lifecycle between two unRAID servers.
# HOST1 is always the owner (source of truth) — HOST2 is always the mirror. # HOST1 is always the owner (source of truth) — HOST2 is always the mirror.
# PARTNERSHIP_OWNER_HOST flips to "HOST2" after a --transfer operation. # PARTNERSHIP_OWNER_HOST flips to "HOST2" after a --transfer operation.
# All identity vars (hostnames, SSH keys) live in master_host*.conf. # All identity vars (hostnames, SSH keys) live in host*.conf.
# Hostnames already match Tailscale device names — IP resolution is automatic. # Hostnames already match Tailscale device names — IP resolution is automatic.
# #
# State files on /boot/config — survives reboots, available before array starts: # State files on /boot/config — survives reboots, available before array starts:
@@ -164,13 +164,13 @@
PARTNERSHIP_ENABLED=false PARTNERSHIP_ENABLED=false
PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer
# Auth containers reconfigured on onboard/offboard — defined per host in master_host*.conf. # Auth containers reconfigured on onboard/offboard — defined per host in host*.conf.
# Format: "ContainerName|WebUIPort" # Format: "ContainerName|WebUIPort"
# HOST1_PARTNERSHIP_AUTH_WEBUIS / HOST2_PARTNERSHIP_AUTH_WEBUIS # HOST1_PARTNERSHIP_AUTH_WEBUIS / HOST2_PARTNERSHIP_AUTH_WEBUIS
# On onboard → WebUI pointed at owner's Tailscale IP # On onboard → WebUI pointed at owner's Tailscale IP
# On offboard → WebUI pointed back at localhost # On offboard → WebUI pointed back at localhost
# Paths to collect during the grace window after offboard — defined per host in master_host*.conf. # Paths to collect during the grace window after offboard — defined per host in host*.conf.
# HOST1_PARTNERSHIP_MIRROR_BACKUPS / HOST2_PARTNERSHIP_MIRROR_BACKUPS # HOST1_PARTNERSHIP_MIRROR_BACKUPS / HOST2_PARTNERSHIP_MIRROR_BACKUPS
# Notified on offboard — no auto-deletion, manual collection. # Notified on offboard — no auto-deletion, manual collection.
@@ -219,13 +219,6 @@
# ── LOGGING ─────────────────────────────────────────────────────────────────────────────────── # ── LOGGING ───────────────────────────────────────────────────────────────────────────────────
# ============================================================================================== # ==============================================================================================
# Silent-by-default output model — ecosystem only speaks when something is wrong.
# true = only warn() and error() produce output (default — reduces notification spam)
# false = all output visible — use for monitor scripts or debugging
# Override per-run: script --log sets ENABLE_LOGGING=true for [LOG] detail
# Monitor scripts (coffee_report, health_digest etc.) set SILENT_MODE=false themselves
SILENT_MODE=true
# Controls verbose [LOG] output across all scripts. # Controls verbose [LOG] output across all scripts.
# true = show detailed [LOG] lines — useful for debugging or first-time setup # true = show detailed [LOG] lines — useful for debugging or first-time setup
# false = show only user-facing output — cleaner for scheduled runs # false = show only user-facing output — cleaner for scheduled runs
@@ -239,7 +232,7 @@
# normal = job completed successfully / warning = something failed or needs attention # normal = job completed successfully / warning = something failed or needs attention
NOTIFY_UNRAID=true NOTIFY_UNRAID=true
# Discord webhook URL — defined per host in master_host*.conf. # Discord webhook URL — defined per host in host*.conf.
# HOST1_DISCORD_WEBHOOK / HOST2_DISCORD_WEBHOOK # HOST1_DISCORD_WEBHOOK / HOST2_DISCORD_WEBHOOK
# Allows different webhooks per server, or only one server notifying. # Allows different webhooks per server, or only one server notifying.
@@ -284,7 +277,6 @@
# Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via # Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via
# watchdog_orchestrator.sh — NOT launched here. # watchdog_orchestrator.sh — NOT launched here.
ARRAY_START_SCRIPTS=( ARRAY_START_SCRIPTS=(
"git_pull_execute.sh" # pull latest scripts before anything starts
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts "Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill "unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning "unRAID_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
@@ -298,11 +290,12 @@
# Schedule: * * * * * (every minute) # Schedule: * * * * * (every minute)
# NOT in ARRAY_START_SCRIPTS — has its own cron entry. # NOT in ARRAY_START_SCRIPTS — has its own cron entry.
# Order matters — resource first (frees pressure), docker second (heals with freed resources), # Order matters — resource first (frees pressure), docker second (heals with freed resources),
# system last (reboots only if prior layers failed). # storage third (pool/data health — never reboots), system last (last line of defense).
WATCHDOG_ORCHESTRATOR_SCRIPTS=( WATCHDOG_ORCHESTRATOR_SCRIPTS=(
"unRAID_Essentials/resource_watchdog.sh" # reduce system pressure before healing attempts "Watchdogs/resource_watchdog.sh" # reduce system pressure before healing attempts
"Docker_Essentials/docker_watchdog.sh" # heal containers with freed resources "Watchdogs/docker_watchdog.sh" # heal containers with freed resources
"unRAID_Essentials/system_watchdog.sh" # reboot if all else fails — last line of defense "Watchdogs/storage_watchdog.sh" # pool and appdata health — alert and remediate
"Watchdogs/system_watchdog.sh" # reboot if all else fails — last line of defense
) )
WATCHDOG_ORCHESTRATOR_HEARTBEAT=true WATCHDOG_ORCHESTRATOR_HEARTBEAT=true
@@ -317,7 +310,7 @@
"Docker_Essentials/downloaders_reset.sh" # clear stuck download states every 15min "Docker_Essentials/downloaders_reset.sh" # clear stuck download states every 15min
) )
# Shares synced every 15 minutes — defined per host in master_host*.conf. # Shares synced every 15 minutes — defined per host in host*.conf.
# HOST1_CRITICAL_SYNC_SHARES / HOST2_CRITICAL_SYNC_SHARES # HOST1_CRITICAL_SYNC_SHARES / HOST2_CRITICAL_SYNC_SHARES
# Format: "/path/to/share" or "/path/to/share|profile-name" # Format: "/path/to/share" or "/path/to/share|profile-name"
# Order matters — Critical-Data first (auth stack), then Emby dirty sync. # Order matters — Critical-Data first (auth stack), then Emby dirty sync.
@@ -326,7 +319,7 @@
# intermediate_sync_maintenance.sh runs every 4 hours — arr library sync, artwork fetch, # intermediate_sync_maintenance.sh runs every 4 hours — arr library sync, artwork fetch,
# and optional mid-day rsync for any shares that need sub-daily propagation. # and optional mid-day rsync for any shares that need sub-daily propagation.
# Schedule: 0 */4 * * * # Schedule: 0 */4 * * *
# INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in master_host*.conf. # INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in host*.conf.
INTERMEDIATE_RSYNC_ENABLED=true # set false to disable mid-day rsync without removing shares INTERMEDIATE_RSYNC_ENABLED=true # set false to disable mid-day rsync without removing shares
INTERMEDIATE_MAINTENANCE_SCRIPTS=( INTERMEDIATE_MAINTENANCE_SCRIPTS=(
@@ -346,7 +339,7 @@
"Media/media_cleaner.sh media" # remove junk from media shares "Media/media_cleaner.sh media" # remove junk from media shares
#"Media/lidarr_cleanup.sh" # remove orphaned music files — enable when ready #"Media/lidarr_cleanup.sh" # remove orphaned music files — enable when ready
#"Media/sonarr_cleanup.sh" # remove orphaned TV files — enable when ready #"Media/sonarr_cleanup.sh" # remove orphaned TV files — enable when ready
#"Media/radarr_cleanup.sh" # remove orphaned movie files — enable when ready "Media/radarr_cleanup.sh" # remove orphaned movie files
"Media/lidarr_missing_art.sh" # fetch missing album/artist artwork (HOST1 only — self-guards) "Media/lidarr_missing_art.sh" # fetch missing album/artist artwork (HOST1 only — self-guards)
"Media/radarr_tmdb_removed.sh" # remove movies dropped from TMDb "Media/radarr_tmdb_removed.sh" # remove movies dropped from TMDb
"Media/sonarr_tvdb_removed.sh" # remove series dropped from TVDB "Media/sonarr_tvdb_removed.sh" # remove series dropped from TVDB
@@ -360,7 +353,7 @@
DAILY_CONTAINER_UPDATES=true DAILY_CONTAINER_UPDATES=true
# Media shares synced daily by daily_sync_maintenance.sh. # Media shares synced daily by daily_sync_maintenance.sh.
# Defined per-host in master_host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES. # Defined per-host in host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
# Mesh model: every node pushes every media share. rsync has no --delete so pushes are additive. # Mesh model: every node pushes every media share. rsync has no --delete so pushes are additive.
# arr_sync (union) ensures all arr libraries converge first. arr_cleanup removes true orphans. # arr_sync (union) ensures all arr libraries converge first. arr_cleanup removes true orphans.
# Any node can download content to any share — it propagates to all nodes on the next cycle. # Any node can download content to any share — it propagates to all nodes on the next cycle.
@@ -368,7 +361,7 @@
# These shares use DEFAULT_RSYNC_OPTS — no profile entry needed. # These shares use DEFAULT_RSYNC_OPTS — no profile entry needed.
# For shares needing custom options or container stops — create a profile in RSYNC section. # For shares needing custom options or container stops — create a profile in RSYNC section.
# Personal encrypted shares defined per-host in master_host*.conf. # Personal encrypted shares defined per-host in host*.conf.
# ZFS encrypted at dataset level — remote receives encrypted blocks, cannot read content. # ZFS encrypted at dataset level — remote receives encrypted blocks, cannot read content.
# See README-Rsync_Setup.md for ZFS encryption setup before uncommenting. # See README-Rsync_Setup.md for ZFS encryption setup before uncommenting.
@@ -390,7 +383,7 @@
# Set false to skip — docker_weekly_restart.sh still runs regardless. # Set false to skip — docker_weekly_restart.sh still runs regardless.
WEEKLY_REMAINING_UPDATES=true WEEKLY_REMAINING_UPDATES=true
# Shares synced during the weekly maintenance window — defined per host in master_host*.conf. # Shares synced during the weekly maintenance window — defined per host in host*.conf.
# HOST1_WEEKLY_SYNC_SHARES / HOST2_WEEKLY_SYNC_SHARES # HOST1_WEEKLY_SYNC_SHARES / HOST2_WEEKLY_SYNC_SHARES
# Containers stopped both sides before sync — full clean state guaranteed. # Containers stopped both sides before sync — full clean state guaranteed.
# Profiles drive container stops, excludes, and options — configure in RSYNC section. # Profiles drive container stops, excludes, and options — configure in RSYNC section.
@@ -473,8 +466,8 @@
# called by weekly_sync_maintenance.sh — full clean sync weekly # called by weekly_sync_maintenance.sh — full clean sync weekly
# critical-fallback — dirty sync — auth stays running both sides, WAL excluded # critical-fallback — dirty sync — auth stays running both sides, WAL excluded
# called by critical_sync_maintenance.sh every 15min # called by critical_sync_maintenance.sh every 15min
# host1-appdata — HOST1 server-specific appdata — defined in master_host1.conf # host1-appdata — HOST1 server-specific appdata — defined in host1.conf
# host2-appdata — HOST2 server-specific appdata — defined in master_host2.conf # host2-appdata — HOST2 server-specific appdata — defined in host2.conf
# important-data — NextCloud + Postgres — NextCloud delayed start after Postgres # important-data — NextCloud + Postgres — NextCloud delayed start after Postgres
# emby — weekly clean sync — both Emby stopped, full mirror # emby — weekly clean sync — both Emby stopped, full mirror
# called by weekly_sync_maintenance.sh only — do NOT schedule separately # called by weekly_sync_maintenance.sh only — do NOT schedule separately
@@ -591,7 +584,7 @@
# Failover → start remote DDNS first (Tier 1) # Failover → start remote DDNS first (Tier 1)
# Handback → stop remote DDNS → rsync → start containers → start local DDNS last # Handback → stop remote DDNS → rsync → start containers → start local DDNS last
# #
# Per-host container lists and tier delays live in master_host*.conf. # Per-host container lists and tier delays live in host*.conf.
# Shared settings (intervals, state file, thresholds) live here. # Shared settings (intervals, state file, thresholds) live here.
EXTERNAL_IP="8.8.8.8" EXTERNAL_IP="8.8.8.8"
@@ -613,7 +606,7 @@
# ━━━ Downloaders Reset ━━━ # ━━━ Downloaders Reset ━━━
# Runs every 15 minutes via CRITICAL_MAINTENANCE_SCRIPTS. # Runs every 15 minutes via CRITICAL_MAINTENANCE_SCRIPTS.
# Clears stuck states, purges old history, prepares each download client for a clean cycle. # Clears stuck states, purges old history, prepares each download client for a clean cycle.
# Per-host URLs and API keys live in master_host*.conf. # Per-host URLs and API keys live in host*.conf.
DOWNLOADER_RETENTION_DAYS=7 # days — purge history older than this DOWNLOADER_RETENTION_DAYS=7 # days — purge history older than this
# qBittorrent failsafe — removes torrents older than threshold regardless of ratio # qBittorrent failsafe — removes torrents older than threshold regardless of ratio
@@ -623,7 +616,7 @@
# ━━━ Docker Daily Restart ━━━ # ━━━ Docker Daily Restart ━━━
# Containers restarted every day via DAILY_MAINTENANCE_SCRIPTS. # Containers restarted every day via DAILY_MAINTENANCE_SCRIPTS.
# Per-host lists live in master_host*.conf: # Per-host lists live in host*.conf:
# HOST1_DAILY_RESTART_CONTAINERS # HOST1_DAILY_RESTART_CONTAINERS
# HOST2_DAILY_RESTART_CONTAINERS # HOST2_DAILY_RESTART_CONTAINERS
# detect_hosts() sets DAILY_RESTART_CONTAINERS to the correct host array at runtime. # detect_hosts() sets DAILY_RESTART_CONTAINERS to the correct host array at runtime.
@@ -631,7 +624,7 @@
# ━━━ Docker Weekly Restart ━━━ # ━━━ Docker Weekly Restart ━━━
# Less critical services restarted weekly via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am). # Less critical services restarted weekly via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am).
# Containers already stopped for weekly sync — restart adds zero extra downtime. # Containers already stopped for weekly sync — restart adds zero extra downtime.
# Per-host lists live in master_host*.conf: # Per-host lists live in host*.conf:
# HOST1_WEEKLY_RESTART_CONTAINERS # HOST1_WEEKLY_RESTART_CONTAINERS
# HOST2_WEEKLY_RESTART_CONTAINERS # HOST2_WEEKLY_RESTART_CONTAINERS
@@ -653,7 +646,7 @@
# Dead containers — remove and restart # Dead containers — remove and restart
# Unexpected exits — non-zero exit code → restart # Unexpected exits — non-zero exit code → restart
# #
# All per-host container lists live in master_host*.conf: # All per-host container lists live in host*.conf:
# HOST*_WATCHDOG_CONTAINERS — memory hard limits per container # HOST*_WATCHDOG_CONTAINERS — memory hard limits per container
# HOST*_WATCHDOG_CONTAINER_URLS — HTTP health check URLs # HOST*_WATCHDOG_CONTAINER_URLS — HTTP health check URLs
# HOST*_WATCHDOG_REQUIRED_CONTAINERS — must always be running # HOST*_WATCHDOG_REQUIRED_CONTAINERS — must always be running
@@ -706,10 +699,42 @@
# Notification batching — one summary per cycle instead of one ping per event # Notification batching — one summary per cycle instead of one ping per event
WATCHDOG_BATCH_NOTIFY=true WATCHDOG_BATCH_NOTIFY=true
# Appdata size monitoring — two-part catch-all for runaway growth and oversized log files.
#
# Part 1 — Growth rate (zero-config):
# Reads per-container dir totals each cycle via du, compares to previous cycle.
# Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused *.log scan
# inside that container. No per-container config required — new containers are covered
# automatically. Baseline built on first run after boot; growth detection starts cycle 2.
#
# Part 2 — Absolute log size:
# Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB across all appdata paths.
# Catches logs that have already stabilised at a large size and are no longer actively growing.
#
# Strike system (reuses existing watchdog infrastructure):
# Strike 1 — warn + notify: condition first detected
# Strike 2 — warn + escalated notify: still present next cycle
# Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
# If WATCHDOG_APPDATA_TRUNCATE_LOGS=true: truncate *.log files in-place, clear strikes
# If false: critical notify only, strikes held until condition resolves
# Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
#
# HOST*_WATCHDOG_APPDATA_SIZES (in host*.conf) suppresses growth warnings for a
# container until its dir exceeds the configured ceiling. Only needed when a container
# legitimately has large stable data and you want to guarantee it never triggers a false alarm.
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2 # flag containers growing more than this per cycle
WATCHDOG_APPDATA_LOG_MAX_GB=2 # flag *.log files exceeding this size (absolute)
WATCHDOG_APPDATA_TRUNCATE_LOGS=false # set true to auto-truncate oversized *.log files on action cycle
WATCHDOG_APPDATA_STRIKE_LIMIT=3 # cycles before action fires (matches existing watchdog pattern)
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db" # /tmp — resets on reboot ✅
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db" # /tmp — resets on reboot ✅
# ━━━ Docker Network Connect ━━━ # ━━━ Docker Network Connect ━━━
# Ensures custom networks exist and connects containers at array start. # Ensures custom networks exist and connects containers at array start.
# Runs once via ARRAY_START_SCRIPTS — idempotent, safe to re-run. # Runs once via ARRAY_START_SCRIPTS — idempotent, safe to re-run.
# Container and network lists are host-specific — defined in master_host*.conf: # Container and network lists are host-specific — defined in host*.conf:
# HOST1_NETWORK_CONNECT_CONTAINERS / HOST2_NETWORK_CONNECT_CONTAINERS # HOST1_NETWORK_CONNECT_CONTAINERS / HOST2_NETWORK_CONNECT_CONTAINERS
# HOST1_NETWORK_CONNECT_NETWORKS / HOST2_NETWORK_CONNECT_NETWORKS # HOST1_NETWORK_CONNECT_NETWORKS / HOST2_NETWORK_CONNECT_NETWORKS
@@ -816,13 +841,13 @@
PERMISSIONS_FILE_MODE="664" # files — group read/write, no execute PERMISSIONS_FILE_MODE="664" # files — group read/write, no execute
PERMISSIONS_OWNER="nobody:users" PERMISSIONS_OWNER="nobody:users"
# Share list defined per host in master_host*.conf: # Share list defined per host in host*.conf:
# HOST1_MEDIA_PERMISSION_SHARES / HOST2_MEDIA_PERMISSION_SHARES # HOST1_MEDIA_PERMISSION_SHARES / HOST2_MEDIA_PERMISSION_SHARES
# ━━━ Media Cleaner ━━━ # ━━━ Media Cleaner ━━━
# Removes junk files from media shares — two profiles: anime and media. # Removes junk files from media shares — two profiles: anime and media.
# Called via DAILY_MAINTENANCE_SCRIPTS. Run manually: Media/media_cleaner.sh anime|media # Called via DAILY_MAINTENANCE_SCRIPTS. Run manually: Media/media_cleaner.sh anime|media
# Folder lists defined per host in master_host*.conf: # Folder lists defined per host in host*.conf:
# HOST1_ANIME_CLEAN_FOLDERS / HOST2_ANIME_CLEAN_FOLDERS # HOST1_ANIME_CLEAN_FOLDERS / HOST2_ANIME_CLEAN_FOLDERS
# HOST1_MEDIA_CLEAN_FOLDERS / HOST2_MEDIA_CLEAN_FOLDERS # HOST1_MEDIA_CLEAN_FOLDERS / HOST2_MEDIA_CLEAN_FOLDERS
@@ -904,7 +929,7 @@
# ━━━ Arr Cleanup ━━━ # ━━━ Arr Cleanup ━━━
# Orphan file cleanup via Lidarr, Sonarr, and Radarr APIs. # Orphan file cleanup via Lidarr, Sonarr, and Radarr APIs.
# Per-host URLs, API keys, and path maps live in master_host*.conf. # Per-host URLs, API keys, and path maps live in host*.conf.
# detect_hosts() selects correct host vars at runtime. # detect_hosts() selects correct host vars at runtime.
# #
# API versions — update MAJOR version here when script is updated for a new arr version: # API versions — update MAJOR version here when script is updated for a new arr version:
@@ -940,7 +965,7 @@
LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs
LIDARR_ART_RETRIES=2 # download retry attempts per image LIDARR_ART_RETRIES=2 # download retry attempts per image
LIDARR_ART_SLEEP_BETWEEN=0.2 # seconds between fanart.tv API calls LIDARR_ART_SLEEP_BETWEEN=0.2 # seconds between fanart.tv API calls
# HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in master_host*.conf # HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in host*.conf
# Lidarr discovery settings (playback_aware_lidarr_discovery.sh) # Lidarr discovery settings (playback_aware_lidarr_discovery.sh)
LIDARR_DISCOVERY_THRESHOLD=70 # score to accept candidate (0-100) LIDARR_DISCOVERY_THRESHOLD=70 # score to accept candidate (0-100)
@@ -1037,7 +1062,7 @@
# stalled — download stuck with no connections or progress # stalled — download stuck with no connections or progress
# #
# Items newer than ARR_IMPORT_RECOVERY_AGE are skipped — gives arr time to retry first. # Items newer than ARR_IMPORT_RECOVERY_AGE are skipped — gives arr time to retry first.
# Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in master_host*.conf. # Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in host*.conf.
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
# matches cron interval — items eligible after one missed cycle # matches cron interval — items eligible after one missed cycle
@@ -1057,7 +1082,7 @@
# Must exist before Emby starts so the symlink resolves correctly. # Must exist before Emby starts so the symlink resolves correctly.
RAMDISK_PATH="/mnt/ramdisk_transcodes" RAMDISK_PATH="/mnt/ramdisk_transcodes"
# Ramdisk size and flip thresholds — defined per host in master_host*.conf. # Ramdisk size and flip thresholds — defined per host in host*.conf.
# All three are coupled — if size changes, thresholds must change with it. # All three are coupled — if size changes, thresholds must change with it.
# HOST1_RAMDISK_SIZE / HOST2_RAMDISK_SIZE # HOST1_RAMDISK_SIZE / HOST2_RAMDISK_SIZE
# HOST1_RAMDISK_WARN_GB / HOST2_RAMDISK_WARN_GB ← flip to SSD at this usage # HOST1_RAMDISK_WARN_GB / HOST2_RAMDISK_WARN_GB ← flip to SSD at this usage
@@ -1068,7 +1093,7 @@
# Must match the container path configured in Emby's Extra Parameters. # Must match the container path configured in Emby's Extra Parameters.
TRANSCODE_LINK="/mnt/ram-transcode" TRANSCODE_LINK="/mnt/ram-transcode"
# SSD fallback location — defined per host in master_host*.conf (cache path differs per server): # SSD fallback location — defined per host in host*.conf (cache path differs per server):
# HOST1_TRANSCODE_SSD / HOST2_TRANSCODE_SSD # HOST1_TRANSCODE_SSD / HOST2_TRANSCODE_SSD
# Minimum free GB on SSD before allowing flip from ramdisk to SSD. # Minimum free GB on SSD before allowing flip from ramdisk to SSD.
@@ -1098,7 +1123,7 @@
# Format: "ContainerName|URL|APIKey|Type" — Type: emby | jellyfin | plex # Format: "ContainerName|URL|APIKey|Type" — Type: emby | jellyfin | plex
# Entries with placeholder API keys are skipped automatically. # Entries with placeholder API keys are skipped automatically.
# ⚠️ Tdarr does NOT belong here — keep Tdarr on SSD, not ramdisk. # ⚠️ Tdarr does NOT belong here — keep Tdarr on SSD, not ramdisk.
# Defined per host in master_host*.conf — Emby container names and keys differ per server: # Defined per host in host*.conf — Emby container names and keys differ per server:
# HOST1_TRANSCODE_SERVERS / HOST2_TRANSCODE_SERVERS # HOST1_TRANSCODE_SERVERS / HOST2_TRANSCODE_SERVERS
# ============================================================================================== # ==============================================================================================
@@ -1108,7 +1133,7 @@
# ━━━ Certificate Monitor ━━━ # ━━━ Certificate Monitor ━━━
# Checks SSL certificate expiry via direct openssl connection — no NPM dependency. # Checks SSL certificate expiry via direct openssl connection — no NPM dependency.
# Checks the actual certificate served by each domain, not what NPM thinks it has. # Checks the actual certificate served by each domain, not what NPM thinks it has.
# Domains defined per host in master_host*.conf — each server monitors its own domains: # Domains defined per host in host*.conf — each server monitors its own domains:
# HOST1_CERT_MONITOR_DOMAINS / HOST2_CERT_MONITOR_DOMAINS # HOST1_CERT_MONITOR_DOMAINS / HOST2_CERT_MONITOR_DOMAINS
CERT_WARN_DAYS=30 # warn when cert expires within this many days CERT_WARN_DAYS=30 # warn when cert expires within this many days
CERT_CRIT_DAYS=7 # critical alert within this many days CERT_CRIT_DAYS=7 # critical alert within this many days
@@ -1117,7 +1142,7 @@
# ━━━ Backup Verify ━━━ # ━━━ Backup Verify ━━━
# Verifies rsync mirror health by comparing random file checksums between servers. # Verifies rsync mirror health by comparing random file checksums between servers.
# Catches silent corruption or incomplete syncs that rsync itself wouldn't detect. # Catches silent corruption or incomplete syncs that rsync itself wouldn't detect.
# Defined per host in master_host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically: # Defined per host in host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically:
# HOST1_BACKUP_VERIFY_SHARES / HOST2_BACKUP_VERIFY_SHARES # HOST1_BACKUP_VERIFY_SHARES / HOST2_BACKUP_VERIFY_SHARES
BACKUP_VERIFY_SAMPLE=10 # random files to check per share BACKUP_VERIFY_SAMPLE=10 # random files to check per share
BACKUP_VERIFY_MIN_SIZE=1M # minimum file size to include in sample BACKUP_VERIFY_MIN_SIZE=1M # minimum file size to include in sample
@@ -1128,7 +1153,7 @@
# (hot/max/hotssd/maxssd) — these vars are fallback only if dynamix.cfg not found. # (hot/max/hotssd/maxssd) — these vars are fallback only if dynamix.cfg not found.
SMART_TEMP_WARN=45 # fallback — Celsius warn threshold SMART_TEMP_WARN=45 # fallback — Celsius warn threshold
SMART_TEMP_CRIT=55 # fallback — Celsius critical threshold SMART_TEMP_CRIT=55 # fallback — Celsius critical threshold
# Drives to ignore defined per host in master_host*.conf — hardware is server-specific: # Drives to ignore defined per host in host*.conf — hardware is server-specific:
# HOST1_SMART_IGNORE_DRIVES / HOST2_SMART_IGNORE_DRIVES # HOST1_SMART_IGNORE_DRIVES / HOST2_SMART_IGNORE_DRIVES
# ━━━ ZFS Memory Snapshot ━━━ # ━━━ ZFS Memory Snapshot ━━━
@@ -1138,7 +1163,7 @@
ZFS_REPORT_FREE_WARN_GB=10 # warn if less than this GB free RAM ZFS_REPORT_FREE_WARN_GB=10 # warn if less than this GB free RAM
ZFS_REPORT_AVAIL_WARN_GB=20 # warn if less than this GB available on ZFS pool ZFS_REPORT_AVAIL_WARN_GB=20 # warn if less than this GB available on ZFS pool
ZFS_REPORT_DOCKER_TOP=10 # how many top Docker containers to show by memory ZFS_REPORT_DOCKER_TOP=10 # how many top Docker containers to show by memory
# Pool ignore list defined per host in master_host*.conf — pool names are server-specific: # Pool ignore list defined per host in host*.conf — pool names are server-specific:
# HOST1_ZFS_REPORT_IGNORE_POOLS / HOST2_ZFS_REPORT_IGNORE_POOLS # HOST1_ZFS_REPORT_IGNORE_POOLS / HOST2_ZFS_REPORT_IGNORE_POOLS
# ━━━ Bandwidth Monitor ━━━ # ━━━ Bandwidth Monitor ━━━
@@ -1168,7 +1193,7 @@
# ━━━ Emby Session Report ━━━ # ━━━ Emby Session Report ━━━
# Weekly Emby usage statistics via API — no persistent writes, queries fresh each run. # Weekly Emby usage statistics via API — no persistent writes, queries fresh each run.
# URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in master_host*.conf. # URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in host*.conf.
EMBY_REPORT_DAYS=7 # days to include in the report period EMBY_REPORT_DAYS=7 # days to include in the report period
EMBY_REPORT_TOP_N=10 # number of top content items to show EMBY_REPORT_TOP_N=10 # number of top content items to show
@@ -1184,8 +1209,8 @@
# Level 3 (hard) — docker stop optional containers, signal docker_watchdog to defer # Level 3 (hard) — docker stop optional containers, signal docker_watchdog to defer
# #
# ── PER-HOST CONTAINER LISTS ────────────────────────────────────────────────────────────────── # ── PER-HOST CONTAINER LISTS ──────────────────────────────────────────────────────────────────
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (in master_host*.conf) # HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (in host*.conf)
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (in master_host*.conf) # HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (in host*.conf)
RW_ENABLED=true RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db" RW_STATE_FILE="/tmp/resource_watchdog_state.db"
@@ -1335,9 +1360,9 @@
SYS_WATCHDOG_MDSTAT_ERROR_LIMIT=5 # new errors in one cycle before acting SYS_WATCHDOG_MDSTAT_ERROR_LIMIT=5 # new errors in one cycle before acting
# ━━━ Check Toggles ━━━ # ━━━ Check Toggles ━━━
# Per-host — moved to master_host*.conf # Per-host — moved to host*.conf
# Different servers may have different hardware, NICs, and check requirements # Different servers may have different hardware, NICs, and check requirements
# See HOST*_SYS_WATCHDOG_CHECK_* in master_host*.conf # See HOST*_SYS_WATCHDOG_CHECK_* in host*.conf
# ━━━ Abort Toggles ━━━ # ━━━ Abort Toggles ━━━
# Conditions that prevent reboot even when a threshold is hit. # Conditions that prevent reboot even when a threshold is hit.
@@ -190,7 +190,7 @@ RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalat
### Per-Host Container Lists ### Per-Host Container Lists
```bash ```bash
# master_host1.conf # host1.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at medium pressure HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at medium pressure
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at hard pressure HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at hard pressure
``` ```
@@ -654,7 +654,7 @@ RW_QBIT_DL_MEDIUM=10240
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis") RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
# ── Per-Host (master_host*.conf) ─────────────────────────────────────────────── # ── Per-Host (host*.conf) ───────────────────────────────────────────────
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
HOST1_SABNZBD_URL="http://localhost:8080" HOST1_SABNZBD_URL="http://localhost:8080"
+24 -24
View File
@@ -44,7 +44,7 @@
# ── PATHS ───────────────────────────────────────────────────────────────────────────────────── # ── PATHS ─────────────────────────────────────────────────────────────────────────────────────
# Scripts: /mnt/user/appdata/unraid_scripts/ # Scripts: /mnt/user/appdata/unraid_scripts/
# Configuration: /mnt/user/appdata/unraid_scripts/master.conf # Configuration: /mnt/user/appdata/unraid_scripts/master.conf
# Per-host: /mnt/user/appdata/unraid_scripts/master_host1.conf (or master_host2.conf) # Per-host: /mnt/user/appdata/unraid_scripts/host1.conf (or host2.conf)
# Library: /mnt/user/appdata/unraid_scripts/common.sh # Library: /mnt/user/appdata/unraid_scripts/common.sh
# #
# ── PLUGIN SETTINGS (apply to every entry) ──────────────────────────────────────────────────── # ── PLUGIN SETTINGS (apply to every entry) ────────────────────────────────────────────────────
@@ -168,9 +168,9 @@
# Overlap protection: acquire_lock() exits immediately if a prior cycle is still running — # Overlap protection: acquire_lock() exits immediately if a prior cycle is still running —
# prevents pile-up when a restart attempt or daemon check takes longer than 60 seconds. # prevents pile-up when a restart attempt or daemon check takes longer than 60 seconds.
# #
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/watchdog_orchestrator.sh # bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_orchestrator.sh
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/watchdog_orchestrator.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_orchestrator.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/watchdog_orchestrator.sh --status # bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_orchestrator.sh --status
# ── TRANSCODE MANAGEMENT ────────────────────────────────────────────────────────────────────── # ── TRANSCODE MANAGEMENT ──────────────────────────────────────────────────────────────────────
@@ -570,7 +570,7 @@
# docker_watchdog.sh — two-tier container self-healing monitor (single-pass) # docker_watchdog.sh — two-tier container self-healing monitor (single-pass)
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass. # Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
# #
# Tier 1 — explicit per-container (configured in master_host*.conf): # Tier 1 — explicit per-container (configured in host*.conf):
# Memory hard limits: immediate restart when exceeded — no strikes, no waiting # Memory hard limits: immediate restart when exceeded — no strikes, no waiting
# CPU strike system: 2 consecutive cycles above HARD_CPU_THRESHOLD → restart # CPU strike system: 2 consecutive cycles above HARD_CPU_THRESHOLD → restart
# HTTP health checks: curl to configured URL — 2 consecutive failures → restart # HTTP health checks: curl to configured URL — 2 consecutive failures → restart
@@ -589,8 +589,8 @@
# RAM emergency: reads mem_shutdown_active from system_watchdog state file, defers all restarts # RAM emergency: reads mem_shutdown_active from system_watchdog state file, defers all restarts
# Silent on clean cycles — only outputs events and hourly heartbeat # Silent on clean cycles — only outputs events and hourly heartbeat
# #
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --status # bash /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --dry-run
# watchdog_skip_list_manager.sh — view and manage the container skip list # watchdog_skip_list_manager.sh — view and manage the container skip list
# When docker_watchdog restarts the same container 3 times in 1hr → skip-listed. # When docker_watchdog restarts the same container 3 times in 1hr → skip-listed.
@@ -605,11 +605,11 @@
# docker start ContainerName → confirm fix works before handing back to watchdog # docker start ContainerName → confirm fix works before handing back to watchdog
# watchdog resumes normal monitoring on next cycle automatically # watchdog resumes normal monitoring on next cycle automatically
# #
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --status # bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --status
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear ContainerName # bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --clear ContainerName
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear ContainerName --force # bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --clear ContainerName --force
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear-all # bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --clear-all
# bash /mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Watchdogs/watchdog_skip_list_manager.sh --dry-run
# docker_container_stop.sh — stop all running containers sequentially with verification # docker_container_stop.sh — stop all running containers sequentially with verification
# Called by array_stopping.sh as the final step in the planned shutdown sequence. # Called by array_stopping.sh as the final step in the planned shutdown sequence.
@@ -631,7 +631,7 @@
# Running → docker restart (graceful). Stopped → left stopped (state respected). Missing → skip. # Running → docker restart (graceful). Stopped → left stopped (state respected). Missing → skip.
# Dependency ordering via WATCHDOG_DEPENDENCIES — databases before applications. # Dependency ordering via WATCHDOG_DEPENDENCIES — databases before applications.
# Restart verification: checks container still up after settle period, notifies if not. # Restart verification: checks container still up after settle period, notifies if not.
# Configured via HOST*_DAILY_RESTART_CONTAINERS (master_host*.conf): # Configured via HOST*_DAILY_RESTART_CONTAINERS (host*.conf):
# NginxProxyManager, Authelia, Dispatcharr, Dispatcharr-Basic, ErsatzTV-Emby # NginxProxyManager, Authelia, Dispatcharr, Dispatcharr-Basic, ErsatzTV-Emby
# #
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh --dry-run
@@ -642,7 +642,7 @@
# Called by weekly_sync_maintenance.sh after sync completes and containers are back up. # Called by weekly_sync_maintenance.sh after sync completes and containers are back up.
# Targets services that benefit from weekly clean start but don't stop for the sync itself. # Targets services that benefit from weekly clean start but don't stop for the sync itself.
# Same rules as daily: running→restart, stopped→leave, missing→skip. # Same rules as daily: running→restart, stopped→leave, missing→skip.
# Configured via HOST*_WEEKLY_RESTART_CONTAINERS (master_host*.conf): # Configured via HOST*_WEEKLY_RESTART_CONTAINERS (host*.conf):
# NextCloud, AdGuard-Home, Immich # NextCloud, AdGuard-Home, Immich
# #
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh --dry-run
@@ -711,7 +711,7 @@
# system_watchdog.sh — three-tier server stability last-resort watchdog (single-pass) # system_watchdog.sh — three-tier server stability last-resort watchdog (single-pass)
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass. # Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
# All 18 checks independently toggleable per host in master_host*.conf. # All 18 checks independently toggleable per host in host*.conf.
# #
# Tier 1 CRITICAL — bypass ALL strikes, reboot immediately: # Tier 1 CRITICAL — bypass ALL strikes, reboot immediately:
# Docker daemon hung → attempt rc.docker restart → still hung → reboot # Docker daemon hung → attempt rc.docker restart → still hung → reboot
@@ -733,8 +733,8 @@
# Reboot loop protection: N reboots in X hours → shutdown instead. # Reboot loop protection: N reboots in X hours → shutdown instead.
# State file heartbeat: writes watchdog_cycle=N every cycle (docker_watchdog stale guard). # State file heartbeat: writes watchdog_cycle=N every cycle (docker_watchdog stale guard).
# #
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --status # bash /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --dry-run
# resource_watchdog.sh — three-level pressure reduction layer (single-pass, called by watchdog_orchestrator) # resource_watchdog.sh — three-level pressure reduction layer (single-pass, called by watchdog_orchestrator)
# Reduces system load intelligently BEFORE docker_watchdog attempts container restarts. # Reduces system load intelligently BEFORE docker_watchdog attempts container restarts.
@@ -748,11 +748,11 @@
# Recovery: pressure must stay below current threshold for RW_RECOVER_CYCLES consecutive runs # Recovery: pressure must stay below current threshold for RW_RECOVER_CYCLES consecutive runs
# before de-escalating. One level at a time — prevents flip-flopping. # before de-escalating. One level at a time — prevents flip-flopping.
# Coordination: at Level 3 writes mem_shutdown_active=true → docker_watchdog defers all restarts. # Coordination: at Level 3 writes mem_shutdown_active=true → docker_watchdog defers all restarts.
# HOST*_RW_PAUSE_CONTAINERS and HOST*_RW_STOP_CONTAINERS configured per host in master_host*.conf. # HOST*_RW_PAUSE_CONTAINERS and HOST*_RW_STOP_CONTAINERS configured per host in host*.conf.
# #
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh --status # bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh # bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh
# inotify_tuning.sh — raise Linux inotify kernel limits at array start # inotify_tuning.sh — raise Linux inotify kernel limits at array start
# Called by array_started.sh FIRST — must run before containers start (they inherit limits). # Called by array_started.sh FIRST — must run before containers start (they inherit limits).
@@ -858,7 +858,7 @@
# Run BEFORE arr cleanup — removes non-media files that would otherwise appear as orphans. # Run BEFORE arr cleanup — removes non-media files that would otherwise appear as orphans.
# Patterns: *.sfv *.md5 *.sha1 *.nfo *.url *.lnk *.rar *.zip *.info *.torrent # Patterns: *.sfv *.md5 *.sha1 *.nfo *.url *.lnk *.rar *.zip *.info *.torrent
# *.sample* *.proof* *sync-conflict* *.scr *.exe *.srr *.log *.json # *.sample* *.proof* *sync-conflict* *.scr *.exe *.srr *.log *.json
# Two profiles with separate folder lists (configured in master_host*.conf): # Two profiles with separate folder lists (configured in host*.conf):
# anime HOST*_ANIME_CLEAN_FOLDERS — anime share folders # anime HOST*_ANIME_CLEAN_FOLDERS — anime share folders
# media HOST*_MEDIA_CLEAN_FOLDERS — Movies, Tv_Shows, Music, Sports etc. # media HOST*_MEDIA_CLEAN_FOLDERS — Movies, Tv_Shows, Music, Sports etc.
# ALWAYS --dry-run when adding new patterns or folders — verify before committing. # ALWAYS --dry-run when adding new patterns or folders — verify before committing.
@@ -1080,7 +1080,7 @@
# Catches: renewed-but-not-reloaded (nginx not reloaded after certbot renewal), # Catches: renewed-but-not-reloaded (nginx not reloaded after certbot renewal),
# wrong cert served, chain issues visible externally but not internally. # wrong cert served, chain issues visible externally but not internally.
# If a user would see a certificate error in their browser, this catches it first. # If a user would see a certificate error in their browser, this catches it first.
# Configured via HOST*_CERT_MONITOR_DOMAINS in master_host*.conf. # Configured via HOST*_CERT_MONITOR_DOMAINS in host*.conf.
# Thresholds: > 30 days = silent, <= 30 = warning, <= CERT_CRIT_DAYS (7) = urgent. # Thresholds: > 30 days = silent, <= 30 = warning, <= CERT_CRIT_DAYS (7) = urgent.
# #
# bash /mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh --status # bash /mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh --status
@@ -1295,7 +1295,7 @@
# Orchestrators/array_stopping.sh (single entry — graceful shutdown) # Orchestrators/array_stopping.sh (single entry — graceful shutdown)
# #
# * * * * * every minute: # * * * * * every minute:
# Orchestrators/watchdog_orchestrator.sh # Watchdogs/watchdog_orchestrator.sh
# #
# */3 * * * * every 3 minutes: # */3 * * * * every 3 minutes:
# Orchestrators/transcode_management.sh # Orchestrators/transcode_management.sh