Compare commits

...
14 Commits
Author SHA1 Message Date
Gmer4Lfe 16e3b7651d Bump ramdisk to 10G; add ramdisk_stop.sh; fix info→log typo in common.sh
host1.conf: HOST1_RAMDISK_SIZE 8G→10G, WARN_GB 6.8→8.5, LOW_GB 5.5→7.
ramdisk_stop.sh: new Tools/ script — redirects symlink to SSD before unmount
  so Emby keeps writing during maintenance, auto-falls back to lazy umount
  when only directory handles are open (no active writes).
common.sh: fix `info` call at line 595 (undefined) → `log`.
Tools docs: added ramdisk_stop.sh to README table, categories, diagram, and Manual.
2026-05-22 17:35:16 -04:00
Gmer4Lfe 8a3e22c9b8 Watchdogs/ docs: new README + Manual, update affected folders
New docs:
  Watchdogs/README-Watchdogs.md  — design, relationships, script table, state file map
  Watchdogs/Manual-Watchdogs.md  — full config reference for all 4 watchdogs

Docker_Essentials/:
  README — remove docker_watchdog, update folder description and diagrams
  Manual  — strip watchdog config sections, add pointer to Watchdogs/Manual

unRAID_Essentials/:
  README — remove system/resource watchdog, update diagrams and script table
  Manual  — strip system/resource watchdog sections, update TOC + config reference

README.md:
  Add Watchdogs/ to folder structure
  Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start
  Fix daily cycle and monitoring diagrams
2026-05-22 17:26:21 -04:00
Gmer4Lfe ec79a926e8 Revert orchestrator + skip_list_manager to original folders
watchdog_orchestrator.sh stays in Orchestrators/ — it's an orchestrator, not a watchdog.
watchdog_skip_list_manager.sh stays in Tools/ — it's a management utility.

Only the 4 watchdog scripts belong in Watchdogs/:
  docker_watchdog.sh, resource_watchdog.sh, storage_watchdog.sh, system_watchdog.sh
2026-05-22 17:10:53 -04:00
Gmer4Lfe 95151c2278 Watchdogs/ folder + host conf rename
Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
2026-05-22 17:08:36 -04:00
Gmer4Lfe 9ee8af1a71 git_pull_execute: remove setup banner 2026-05-21 17:29:25 -04:00
Gmer4Lfe 84f133b99b Monitors echo/log audit pass
Status conclusions and clean-check summaries → echo: backup_verify (pre-flight
passed, all files match), bandwidth_monitor (logged transfer entry),
cert_monitor/smart_health (all healthy), system_tuning_monitor (snapshot written),
weekly_health_digest (smart no-findings exit, digest sent), zfs_memory_snapshot
(all pools ONLINE, all checks within thresholds). Manual-Monitors.md: added
Output Tiers section.
2026-05-21 17:28:01 -04:00
Gmer4Lfe ef8ef436cf Orchestrators echo/log audit pass
Status conclusions, clean-cycle summaries, skipped-feature lines, and phase
completion results → echo across all nine orchestrator scripts. Per-share,
per-job, and per-script detail lines inside loops remain log. README-Orchestrators:
added Output Tiers section.
2026-05-21 17:25:29 -04:00
Gmer4Lfe 3770bf80e7 unRAID_Essentials echo/log audit pass + exit traps
Exit traps: resource_watchdog registers/clears trap for stopped containers;
server_reboot registers trap to restart Docker service on abort; system_watchdog
records containers before bulk stop and registers trap to restart if reboot aborts.

echo/log audit: status conclusions, nothing-to-do, and clean-cycle confirmations
→ echo across clear_logs, docker_syslog_filter, mover_stop, php_fpm_max_children,
rsync_stop, user_scripts_stop. resource_watchdog: disabled flag, restore-level
messages, pressure status → echo. system_watchdog: clean-cycle summary → echo.
Manual-Unraid_Essentials.md: added Output Tiers section.
2026-05-21 17:21:20 -04:00
Gmer4Lfe 86b31256ed Partnership echo/log audit pass + exit trap
Exit trap (_pm_trap_restart_stopped) in partnership_manager.sh: restarts locally
stopped containers if script crashes mid-cleanup; partnership_offboard.sh registers
same trap. echo/log audit: step result lines (stack deployed counts, state writes,
onboard/offboard complete) → echo; check mode ACTIVE/INACTIVE status → echo;
ssh_setup.sh key-exists and install/verify results → echo. Manual-Partnership.md:
added Output Tiers section.
2026-05-21 17:11:05 -04:00
Gmer4Lfe 588cde1cc4 Transcodes echo/log audit pass
ramdisk_setup.sh: path/size detail lines → log (already in log block above);
status done → echo. transcode_cleanup.sh: flip-back decision line and status
done → echo. transcode_manager.sh: clean-run single-line summary → echo.
Manual-Transcoding.md: added Output Tiers section.
2026-05-21 17:07:23 -04:00
Gmer4Lfe 5ccfb895c1 Tools echo/log audit pass — 9 scripts + Manual
No setup banners to remove (Tools folder was already clean). Fix log→echo
on summary status/conclusion lines across container_data_export, emby_database_repair,
recreate_shares, zfs_pool_scrub, and fallback_state_reset (safety check + two summary
detail lines). Fix log→echo on status display lines in watchdog_skip_list_manager
(skip list empty, restart history lines). Move per-item "to add" preview echoes to
log in emby_to_lidarr/radarr/sonarr_sync; fix nothing-to-do log→echo in all three.
Add output tiers section to Manual-Tools.md.
2026-05-21 16:58:32 -04:00
Gmer4Lfe 789d376eb3 Rsync echo/log audit pass — rsync.sh + Manual
Fix log→echo on success status line in summary block. Add output tiers
section to Manual-Rsync.md. No setup banner (script was already clean);
all section headers and transfer identity block correctly remain echo.
2026-05-21 16:51:09 -04:00
Gmer4Lfe a4edeab5eb Media echo/log audit pass — 11 scripts + Manual
Remove setup banners from all 8 scripts that had them (discovery scripts
use comment-only setup sections). Move per-item/detail echoes to log in
arr_sync, lidarr_missing_art, lidarr/radarr/sonarr_cleanup (scan progress,
API query status, root/orphan config lines), and per-title lines in
radarr_tmdb_removed / sonarr_tvdb_removed. Fix log→echo on clean-status
summary lines in arrs_failed_stalled_recovery, media_cleaner, and both
removed-title scripts. Add output tiers section to Manual-Media.md.
2026-05-21 16:48:08 -04:00
Gmer4Lfe c1c12cbe73 Fallback echo/log audit pass — fallback.sh, fallback_test.sh + Manual
Remove setup banners from fallback.sh and fallback_test.sh. Add Host Detection
to fallback.sh OPERATIONAL SAFEGUARDS; add Lock Acquisition + Host Detection to
fallback_test.sh. Add output tiers section to Manual-Fallback.md documenting
daemon vs one-shot output behaviour.
2026-05-21 16:33:36 -04:00
92 changed files with 2568 additions and 1185 deletions
+29 -368
View File
@@ -6,235 +6,9 @@ Configuration reference, setup procedures, and operational workflows.
For folder overview and design philosophy see `README-Docker_Essentials.md`.
For per-script detail see the script headers directly.
---
## ━━━ WATCHDOG CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All watchdog configuration lives in `master_host*.conf` (per-container lists) and
`master.conf` (shared thresholds and toggles). `detect_hosts()` aliases all
`HOST1_` / `HOST2_` prefixed vars to their unprefixed names at runtime — scripts
always read the right values for the server they're running on.
---
### ── Memory Hard Limits ────────────────────────────────────────────────────────
```bash
# master_host1.conf
# Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
#
# Sizing guidance:
# Check normal peak: docker stats ContainerName
# Set limit at ~150-200% of normal peak
# Emby peaks ~12GB under heavy transcode load → 18GB gives headroom
# without triggering on legitimate load spikes
#
HOST1_WATCHDOG_CONTAINERS=(
"Emby:18432" # 18GB — peaks ~12GB under heavy transcode load
"LidaTube:6144" # 6GB — YouTube downloader, grows with large queues
"Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature
"Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous
)
```
A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
visibility into a container approaching its ceiling before a restart is triggered.
---
### ── CPU Thresholds ────────────────────────────────────────────────────────────
```bash
# master.conf
# CPU is normalised against total core count — meaningful regardless of hardware.
#
# Why normalised:
# 85% on one core of a 16-core machine = 5.3% normalised → ignore it
# 85% normalised on a 16-core machine = 13.6 cores worth → runaway process
#
# CPU uses a STRIKE SYSTEM — not immediate restart like memory.
# Brief spikes are normal (Tdarr encoding, Emby transcoding, SABnzbd unpacking).
# The strike system ignores spikes and acts on sustained high usage.
#
# Strike 1: above HARD_CPU_THRESHOLD this cycle → warn, increment strike
# Strike 2: above threshold next cycle → restart, reset counter
# Recovery: drops below threshold any cycle → reset counter to 0
#
SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only
HARD_CPU_THRESHOLD=85 # strike at 85% normalised
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
```
---
### ── HTTP Health Checks ────────────────────────────────────────────────────────
```bash
# master_host1.conf
# Format: "ContainerName:http://host:port/optional-path"
# Hits the actual service endpoint on every watchdog cycle.
# "Container running" and "service responding" are not the same thing.
#
# Uses a STRIKE SYSTEM — network hiccups and brief restarts happen.
# Two consecutive non-responses before acting prevents false positives.
#
# The path can be a lightweight health endpoint or the root URL.
# Docker's own HEALTHCHECK requires the image to define it — most don't.
# These checks work regardless of what the image defines.
#
HOST1_WATCHDOG_CONTAINER_URLS=(
"Emby:http://localhost:8096" # Emby WebUI root — fast to respond
"NginxProxyManager:http://localhost:81" # NPM admin interface
)
# master.conf
CURL_TIMEOUT=5 # seconds before non-response counts as a failure
RESP_FAIL_LIMIT=2 # consecutive failures before restart
```
---
### ── Required Containers ───────────────────────────────────────────────────────
```bash
# master_host1.conf
# Containers that must always be running.
# Found stopped → watchdog attempts restart every cycle until running or skip-listed.
# Uses the STRIKE SYSTEM — one miss might be mid-restart.
# Persistent failure → skip list → critical notification.
#
# These are the containers whose absence breaks everything else:
#
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
"NginxProxyManager" # reverse proxy — all external traffic routes through this
"Authelia" # SSO authentication — all protected services need it
"Mariadb-Authelia" # Authelia database — must be up before Authelia starts
"Redis-Authelia" # Authelia session cache — same startup dependency
)
```
---
### ── Dependency Ordering ───────────────────────────────────────────────────────
```bash
# master_host1.conf
# Format: "DependentContainer:dependency1 dependency2"
# Multiple dependencies space-separated. All must be running before dependent restarts.
#
# When a container and its dependency are both down:
# → restart the dependency first
# → skip the dependent this cycle
# → next cycle: dependency healthy → dependent restarts cleanly
#
# Without this: Authelia starts, can't connect to MariaDB (still starting),
# exits immediately, strike 1. Next cycle: same, strike 2. Skip list.
# MariaDB was fine the whole time.
#
HOST1_WATCHDOG_DEPENDENCIES=(
"Authelia:Mariadb-Authelia Redis-Authelia"
"Authelia-Secondary:Mariadb-Authelia Redis-Authelia-Secondary"
"NextCloud:Postgres-NextCloud"
)
```
The same dependency configuration is used by `docker_daily_restart.sh` and
`docker_weekly_restart.sh` — configure once, applies everywhere.
---
### ── Startup Grace Period ──────────────────────────────────────────────────────
```bash
# master.conf
# Suppress restart actions for N seconds after array start.
# Checks still run and log — only restart actions are suppressed.
# Clock starts from when the watchdog process itself starts.
#
# Without this: false-positive restarts fire in the first minutes after
# every array start while containers are still initialising.
#
WATCHDOG_STARTUP_GRACE=600 # 10 minutes
```
---
### ── Tier 2 Global Scan ────────────────────────────────────────────────────────
Tier 2 scans every running container when `WATCHDOG_SCAN_ALL=true`. No per-container
configuration required — it's the catch-all for everything not explicitly in Tier 1.
```bash
# master.conf
WATCHDOG_SCAN_ALL=true # enable global scan
WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures
WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills
WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts
WATCHDOG_RESTART_DEAD=true # recover containers in dead state
WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero
WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list
# Containers excluded from Tier 2 entirely.
# Use for containers you intentionally stop/start manually, or containers that
# have benign non-zero exits as part of their normal operation.
WATCHDOG_SCAN_IGNORE=(
"my-one-shot-container" # runs and exits normally — not a crash
)
```
Each toggle is independent — disable any check that produces false positives in your
environment without affecting the others.
---
### ── Restart Loop Protection ──────────────────────────────────────────────────
```bash
# master.conf
# N restarts within a rolling window → skip list + critical notification.
# The watchdog stops touching the container entirely.
# Skip list lives on /boot/config/ — survives reboots intentionally.
# A container bad enough to be skip-listed is still broken after a reboot.
#
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
```
**Auto-clear:** The watchdog checks the skip list every cycle and removes any container
it finds running. If the container recovers on its own, monitoring resumes automatically.
Manual clear is only needed when the container is stuck stopped.
---
### ── Notification Batching ────────────────────────────────────────────────────
```bash
# master.conf
# All events from one cycle collected → one notification at end of cycle.
#
# Why: a shared database going down can cascade 10+ containers failing
# simultaneously. Without batching: 10 individual pings. With batching:
# one summary listing all affected containers. Actionable vs overwhelming.
#
WATCHDOG_BATCH_NOTIFY=true
```
---
### ── State Files Reference ────────────────────────────────────────────────────
| File | Config Var | Location | Resets | Purpose |
|------|-----------|----------|--------|---------|
| Strike counts | `WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | Per-container CPU/HTTP strike counters |
| Skip list | `SYS_WATCHDOG_FAILED_FILE` | `/boot/config/` | Never (manual / auto-clear) | Containers that exhausted restart attempts |
| Restart history | `WATCHDOG_CONTAINER_RESTART_LOG` | `/boot/config/` | Auto-purge after window | Restart loop detection data |
| Shared state | `SYS_WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | RAM emergency flag + cycle heartbeat from `system_watchdog.sh` |
`/tmp/` resets on reboot — correct, strike counts before a reboot are meaningless after it.
`/boot/config/` survives reboots — correct, a skip-listed container is still broken after a reboot.
> **Watchdog configuration has moved.** `docker_watchdog.sh` is now in `Watchdogs/`.
> Memory limits, CPU thresholds, HTTP health checks, dependency ordering, skip list
> recovery, and all watchdog config vars are in `Watchdogs/Manual-Watchdogs.md`.
---
@@ -245,7 +19,7 @@ WATCHDOG_BATCH_NOTIFY=true
### ── Daily Restart List ───────────────────────────────────────────────────────
```bash
# master_host1.conf
# host1.conf
# Restarted every night at 1am via daily_sync_maintenance.sh.
#
# Good candidates:
@@ -271,7 +45,7 @@ their images updated daily before the restart. Add a container once, it gets bot
### ── Weekly Restart List ──────────────────────────────────────────────────────
```bash
# master_host1.conf
# host1.conf
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
#
@@ -291,7 +65,7 @@ HOST1_WEEKLY_RESTART_CONTAINERS=(
## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```bash
# master_host1.conf
# host1.conf
# Networks to ensure exist + containers to connect to each network.
# Many-to-many: every container connects to every network listed.
#
@@ -339,23 +113,11 @@ Everything gets updated at least once per week with no explicit configuration.
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
### ── master_host*.conf ────────────────────────────────────────────────────────
### ── host*.conf ────────────────────────────────────────────────────────
```bash
# Per-host — varies between HOST1 and HOST2
# Tier 1 — memory limits ("ContainerName:LimitInMB")
HOST1_WATCHDOG_CONTAINERS=()
# Tier 1 — HTTP health check endpoints ("ContainerName:http://host:port")
HOST1_WATCHDOG_CONTAINER_URLS=()
# Tier 1 — must always be running
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
# Tier 1 + 2 — dependency ordering ("Dependent:dep1 dep2")
HOST1_WATCHDOG_DEPENDENCIES=()
# Daily restart list (also drives docker_update.sh normal mode)
HOST1_DAILY_RESTART_CONTAINERS=()
@@ -367,6 +129,9 @@ HOST1_NETWORK_CONNECT_NETWORKS=()
# Containers to connect to every configured network
HOST1_NETWORK_CONNECT_CONTAINERS=()
# Watchdog config (memory limits, HTTP checks, required, dependencies):
# → see Watchdogs/Manual-Watchdogs.md
```
---
@@ -376,47 +141,16 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
```bash
# Shared — applies to both servers
# ── Watchdog timing ────────────────────────────────────────────────────────
DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes)
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
# ── Memory ─────────────────────────────────────────────────────────────────
SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart)
# ── CPU ────────────────────────────────────────────────────────────────────
SOFT_CPU_THRESHOLD=50 # warn threshold — normalised %
HARD_CPU_THRESHOLD=85 # strike threshold — normalised %
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
# ── HTTP health check ──────────────────────────────────────────────────────
CURL_TIMEOUT=5 # seconds before curl times out
RESP_FAIL_LIMIT=2 # consecutive failures before restart
# ── Restart loop protection ────────────────────────────────────────────────
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
# ── Tier 2 global scan ─────────────────────────────────────────────────────
WATCHDOG_SCAN_ALL=true
WATCHDOG_SCAN_IGNORE=()
WATCHDOG_RESTART_UNHEALTHY=true
WATCHDOG_NOTIFY_OOM=true
WATCHDOG_NOTIFY_CRASHLOOP=true
WATCHDOG_CRASH_LIMIT=5
WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
# ── Notifications ──────────────────────────────────────────────────────────
WATCHDOG_BATCH_NOTIFY=true
# ── Container updates ──────────────────────────────────────────────────────
# ── Container updates ──────────────────────────────────────────────────
DAILY_CONTAINER_UPDATES=true
WEEKLY_REMAINING_UPDATES=true
# ── Retry behaviour (shared by restart scripts) ────────────────────────────
# ── Retry behaviour (shared by restart scripts) ────────────────────────
RETRY_COUNT=3 # retry attempts before marking failed
SLEEP=5 # seconds between retry attempts
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
# → see Watchdogs/Manual-Watchdogs.md
```
---
@@ -425,47 +159,22 @@ SLEEP=5 # seconds between retry attempts
### ── Adding a Container to Monitoring ────────────────────────────────────────
Adding a container is purely additive — add the relevant lines to `master_host*.conf`.
Adding a container to watchdog monitoring is purely additive — add lines to `host*.conf`.
No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
See `Watchdogs/Manual-Watchdogs.md` for the full procedure.
To add a container to **daily restarts** (and daily image updates):
```bash
# master_host1.conf — example: adding "MyApp" to full Tier 1 + daily restarts
# 1. Memory hard limit — size at ~150-200% of normal peak (check: docker stats MyApp)
HOST1_WATCHDOG_CONTAINERS=(
...existing...
"MyApp:2048" # 2GB ceiling
)
# 2. HTTP health check
HOST1_WATCHDOG_CONTAINER_URLS=(
...existing...
"MyApp:http://localhost:8080/health" # or root URL if no /health endpoint
)
# 3. Required — add only if absence breaks other services
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
...existing...
"MyApp"
)
# 4. Dependency — add if MyApp needs another container up first
HOST1_WATCHDOG_DEPENDENCIES=(
...existing...
"MyApp:MyApp-Database"
)
# 5. Daily restart — add if MyApp degrades over time
# host1.conf
HOST1_DAILY_RESTART_CONTAINERS=(
...existing...
"MyApp" # also adds it to the daily image update
)
```
Tier 2 picks up MyApp automatically — no configuration needed. It will be included in
the global health scan from the next cycle onward.
To **exclude** a container from Tier 2 global scan (e.g. one-shot that exits normally):
To **exclude** MyApp from Tier 2 (e.g. it's a one-shot container that exits normally):
```bash
# master.conf
WATCHDOG_SCAN_IGNORE=(
@@ -477,45 +186,13 @@ WATCHDOG_SCAN_IGNORE=(
### ── Skip List Recovery ────────────────────────────────────────────────────────
Used when `docker_watchdog.sh` has skip-listed a container after exhausting restart
attempts. The watchdog stops touching it and sends a critical notification. Human
intervention required.
When `docker_watchdog.sh` skip-lists a container, use `Tools/watchdog_skip_list_manager.sh`.
Full procedure in `Watchdogs/Manual-Watchdogs.md`.
```bash
# Step 1 — understand the situation
# Shows skip list contents, container states, restart history
watchdog_skip_list_manager.sh --status
# Step 2 — fix the underlying problem first
# Check logs: docker logs ContainerName --tail 100
# Check disk: df -h /mnt/user
# Check database: docker exec ContainerName sqlite3 /path/to.db ".tables"
# Fix before clearing — clearing without fixing just resets the counter
# Step 3 — clear the container from the skip list + restart history
# Clearing history is important: the counter carries over otherwise and
# the container hits the limit again almost immediately on any startup trouble
watchdog_skip_list_manager.sh --clear ContainerName
# Step 4 — start the container manually
# Confirms your fix worked before handing back to the watchdog
docker start ContainerName
# Step 5 — monitoring resumes automatically
# Next watchdog cycle: container seen running → removed from skip list
# Restart history clean. Back to normal.
```
> ⚠️ **If `docker_watchdog.sh` is currently running when you clear the skip list**, it
> may re-add the container on its very next cycle if the container is still in a bad
> state. The script detects this and warns you. Fix the root cause **before** clearing.
Other skip list actions:
```bash
watchdog_skip_list_manager.sh # status (default)
watchdog_skip_list_manager.sh --clear-all # clear everything
watchdog_skip_list_manager.sh --clear-all --force # non-interactive
watchdog_skip_list_manager.sh --clear-all --dry-run # preview what would clear
Tools/watchdog_skip_list_manager.sh --status # see skip list + container states
Tools/watchdog_skip_list_manager.sh --clear MyApp # clear after fixing root cause
Tools/watchdog_skip_list_manager.sh --clear-all # clear everything
```
---
@@ -530,25 +207,9 @@ All scripts support these standard flags:
| `--status` | Show current config, container states, and relevant runtime info, then exit. |
| `--log` | Verbose mode — adds per-container banners, action lines, pull output, and list details. |
**Output tiers:** Scripts have two output levels. Without `--log`, each script processes
silently and always concludes with a summary block: identity, duration, counts, and
a status line. Per-container detail — individual container names, pull Status lines,
skip reasons — only appears with `--log`. Warnings and errors are always visible
regardless of `--log`.
`docker_watchdog.sh` is the exception — it runs 96 cycles/day as a background daemon
and is silent on clean cycles by design. Its summary block only fires when there are
restarts or warnings. Use `--log` to see per-cycle detail on clean cycles.
### `docker_watchdog.sh --status` shows:
Strike counts for all monitored containers, current skip list contents, whether grace
period is active and how long remains, whether RAM emergency deferral is active, last
cycle timing.
### `docker_watchdog.sh --dry-run` shows:
A full watchdog cycle without restarting anything. Shows what the watchdog would do
based on current container states. Useful for verifying configuration before enabling
automatic restarts.
**Output tiers:** Without `--log`, each script processes silently and concludes with a
summary block: identity, duration, counts, and a status line. Per-container detail only
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
### `docker_update.sh --remainder`
Switches to remainder mode — updates all running containers not in the managed daily/weekly
+22 -49
View File
@@ -154,24 +154,16 @@ its history after you've fixed the problem. No manual file editing required.
```
Docker_Essentials/ ← acts on containers (this folder)
unRAID_Essentials/ ← acts on the server itself
Watchdogs/ ← reactive monitoring + last-resort stability
Monitors/ ← observes, measures, reports
Rsync/ ← moves data between servers
```
Five distinct responsibilities, each handled by dedicated scripts:
> `docker_watchdog.sh` has moved to `Watchdogs/`. Container healing, memory limits,
> HTTP health checks, and skip list management are documented in
> `Watchdogs/README-Watchdogs.md` and `Watchdogs/Manual-Watchdogs.md`.
---
### 🔁 Reactive Healing — `docker_watchdog.sh`
Continuous two-tier monitoring that catches problems as they happen and acts on them
immediately. Runs as a background process started at array start. Every 15 minutes it
checks the full stack and fixes what it can — silently when everything is fine, visibly
when something needs attention.
Two tiers because different containers need different monitoring strategies:
- **Tier 1** — explicit per-container configuration with specific thresholds
- **Tier 2** — global scan of everything that's running with catch-all health checks
Four distinct responsibilities in this folder, each handled by dedicated scripts:
---
@@ -212,34 +204,22 @@ or in-progress downloads.
---
## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## ━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Two watchdogs run simultaneously. They are designed to work together, not compete:
`docker_watchdog.sh` has moved to `Watchdogs/` and is now one of four coordinated
single-pass scripts called every minute by `Orchestrators/watchdog_orchestrator.sh`.
```
system_watchdog.sh watches the server: RAM, CPU, disk, kernel, daemon health
docker_watchdog.sh ← watches the containers: memory, CPU, HTTP response, crashes
Watchdogs/resource_watchdog.sh ← reduces pressure before healing attempts
Watchdogs/docker_watchdog.sh heals containers (reads resource_watchdog state)
Watchdogs/storage_watchdog.sh ← pool growth + runaway log detection
Watchdogs/system_watchdog.sh ← last resort — reboots when healing has failed
```
**The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops
non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh`
would see stopped containers on its next cycle and restart them — directly undoing the
RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover.
The system would eventually hit the reboot threshold anyway, having accomplished nothing.
**The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When
`system_watchdog.sh` triggers a RAM emergency shutdown it writes
`mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every
cycle and defers all container restart logic until it clears. Health URL checks for
excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still
run. Everything else stands down.
**The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state
file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh`
checks how long ago the state file was modified. If it's more than 2 hours old while
`mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running.
`docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced
indefinitely by a stale flag from a process that's no longer running.
Scripts in this folder (daily restart, updates, network connect) are unaffected —
they run on their own schedules via the maintenance orchestrators and are not part
of the every-minute watchdog cycle. See `Watchdogs/README-Watchdogs.md` for the
full coordination model between all four watchdogs.
---
@@ -247,7 +227,6 @@ indefinitely by a stale flag from a process that's no longer running.
| Script | Role | When It Runs |
|--------|------|-------------|
| `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop — array start |
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window |
@@ -265,18 +244,12 @@ indefinitely by a stale flag from a process that's no longer running.
```
Array starts
── docker_network_connect.sh ────── run once at start
ensure networks + connections exist
silent if correct, notify if creating
└── docker_watchdog.sh ──────────── continuous background loop (every 15min)
Tier 1: memory, CPU, HTTP, required containers
Tier 2: global unhealthy / OOM / crash / dead scan
reads system_watchdog state (RAM emergency deferral)
│ (on skip list event → operator uses)
└── Tools/watchdog_skip_list_manager.sh
inspect state, clear after fixing root cause
── docker_network_connect.sh ────── run once at start
ensure networks + connections exist
silent if correct, notify if creating
Every minute (Orchestrators/watchdog_orchestrator.sh):
└── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md
Daily maintenance window (1am):
daily_sync_maintenance.sh
+2 -2
View File
@@ -60,7 +60,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_DAILY_RESTART_CONTAINERS
# Containers restarted nightly. Also used by docker_update.sh normal mode
@@ -126,7 +126,7 @@ detect_hosts
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
exit 0
fi
+3 -3
View File
@@ -78,7 +78,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_NETWORK_CONNECT_NETWORKS
# Networks to ensure exist at array start. Aliased by detect_hosts() →
@@ -144,13 +144,13 @@ fi
# Empty array guards
if [[ ${#NETWORK_CONNECT_NETWORKS[@]} -eq 0 ]]; then
warn "NETWORK_CONNECT_NETWORKS is empty for $MY_ID — nothing to do"
warn "Check HOST*_NETWORK_CONNECT_NETWORKS in master_host*.conf"
warn "Check HOST*_NETWORK_CONNECT_NETWORKS in host*.conf"
exit 0
fi
if [[ ${#NETWORK_CONNECT_CONTAINERS[@]} -eq 0 ]]; then
warn "NETWORK_CONNECT_CONTAINERS is empty for $MY_ID — no containers to connect"
warn "Check HOST*_NETWORK_CONNECT_CONTAINERS in master_host*.conf"
warn "Check HOST*_NETWORK_CONNECT_CONTAINERS in host*.conf"
exit 0
fi
+2 -2
View File
@@ -95,7 +95,7 @@
# Container names for emby and critical-data profiles — excluded from
# remainder mode (already updated by the weekly sync window)
#
# master_host*.conf
# host*.conf
#
# HOST*_DAILY_RESTART_CONTAINERS
# Containers updated in normal mode. Aliased by detect_hosts() →
@@ -205,7 +205,7 @@ else
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
exit 0
fi
+1 -1
View File
@@ -52,7 +52,7 @@
# Enable or disable this script. (default: true)
# To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
#
# master_host*.conf
# host*.conf
#
# HOST*_DAILY_RESTART_CONTAINERS
# Excluded from this script — already updated daily. Aliased by
+2 -2
View File
@@ -43,7 +43,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_WEEKLY_RESTART_CONTAINERS
# Containers restarted weekly. Aliased by detect_hosts() →
@@ -108,7 +108,7 @@ detect_hosts
if [[ ${#WEEKLY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
warn "WEEKLY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
warn "Check HOST*_WEEKLY_RESTART_CONTAINERS in master_host*.conf"
warn "Check HOST*_WEEKLY_RESTART_CONTAINERS in host*.conf"
exit 0
fi
+2 -2
View File
@@ -83,7 +83,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_SLSKD_URL / HOST*_SLSKD_API_KEY / HOST*_SLSKD_FAILED_IMPORTS_DIR
# slskd connection and failed imports path. Aliased by detect_hosts()
@@ -567,7 +567,7 @@ if [[ -n "$QBIT_URL" ]] && [[ -n "$QBIT_USERNAME" ]]; then
if [[ -z "$QBIT_COOKIE" ]]; then
error "Failed to authenticate with qBittorrent — check QBIT_USERNAME/PASSWORD"
notify "qBittorrent auth failed on $(hostname) — check credentials in master_host*.conf" "Downloaders Reset" "warning"
notify "qBittorrent auth failed on $(hostname) — check credentials in host*.conf" "Downloaders Reset" "warning"
((TOTAL_FAIL++))
else
TORRENTS=$(curl -sf --max-time 15 \
+22 -4
View File
@@ -80,7 +80,7 @@ container start time. At 3 strikes × 30s + ~2min rsync + ~1min container start:
---
## ━━━ CONFIGURATION — master_host*.conf ━━━
## ━━━ CONFIGURATION — host*.conf ━━━
```
HOST*_DDNS_CONTAINERS=("...")
@@ -181,7 +181,7 @@ FALLBACK_TEST_HANDBACK_WAIT=300
---
### master_host2.conf — HOST2 covering HOST1
### host2.conf — HOST2 covering HOST1
```bash
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
@@ -259,7 +259,7 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
---
### master_host1.conf — HOST1 covering HOST2
### host1.conf — HOST1 covering HOST2
```bash
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
@@ -486,7 +486,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
1. Create the container on the covering server (stopped), with volume mounts pointing at
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in master_host*.conf
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf
in the appropriate tier position (dependency ordering — databases before apps)
3. Verify: `fallback.sh --status` shows the container in the expected tier list
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
@@ -568,6 +568,24 @@ If it has happened:
---
## ━━━ OUTPUT TIERS ━━━
Both scripts have two output levels controlled by `--log`.
**fallback.sh** — daemon, runs continuously. Without `--log`, only state transitions,
warnings, errors, and the startup banner are visible. Per-cycle detail (ping results,
state evaluation) is suppressed — the daemon runs every 1530 seconds and clean cycles
produce no output by design. State transitions are always visible because they are the
events that matter. Use `--log` when debugging why the state machine is or is not
acting.
**fallback_test.sh** — one-shot test harness. Without `--log`, phase headers, pass/fail
results, and the final test report are always visible. Per-container checks within each
phase are suppressed. With `--log`, every check in every phase is shown. Warnings and
errors are always visible regardless of `--log`.
---
## ━━━ FLAG REFERENCE ━━━
### fallback.sh
+2 -2
View File
@@ -219,8 +219,8 @@ determines which server is local and which is remote at runtime, then selects th
container arrays and tier delays from config via MY_ID.
```
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in master_host2.conf)
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in master_host1.conf)
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf)
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf)
```
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
+5 -3
View File
@@ -111,6 +111,10 @@
# acquire_lock() prevents two instances running simultaneously. Both would
# modify containers and DDNS independently — conflict is certain.
#
# Host Detection
# detect_hosts() resolves MY_ID / REMOTE_ID from master.conf at startup.
# Exits if the host cannot be identified — prevents running on an unknown machine.
#
# Silent by Default
# State transitions: warn() — always visible. Healthy routine cycles: log()
# — suppressed unless --log. Produces no output on clean cycles.
@@ -128,7 +132,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_DDNS_CONTAINERS
# DDNS containers this host manages — stopped on internet loss, started
@@ -246,8 +250,6 @@ fi
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
+9 -3
View File
@@ -60,6 +60,14 @@
# Remote Docker Daemon Check
# Pre-flight confirms remote Docker daemon is responsive before Phase 2.
#
# Lock Acquisition
# acquire_lock() prevents concurrent test runs. Running two tests simultaneously
# would produce conflicting iptables rules and unreliable results.
#
# Host Detection
# detect_hosts() resolves MY_ID / REMOTE_ID from master.conf at startup.
# Exits if the host cannot be identified — prevents testing on an unknown machine.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
@@ -134,8 +142,6 @@ trap cleanup EXIT
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
@@ -283,7 +289,7 @@ fi
# Tier 1 containers configured
if [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then
error "No Tier 1 containers configured for $MY_ID$REMOTE_ID"
error "Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in master_host*.conf"
error "Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in host*.conf"
phase_fail "Pre-flight"
exit 1
fi
+24 -7
View File
@@ -228,7 +228,7 @@ LIDARR_DISCOVERY_REJECT_COOLDOWN=30 # days before re-evaluating a Stage 2 reje
LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db"
```
Requires `HOST*_LASTFM_API_KEY` in `master_host*.conf`.
Requires `HOST*_LASTFM_API_KEY` in `host*.conf`.
---
@@ -246,7 +246,7 @@ RADARR_DISCOVERY_SEED_LIBRARIES=("Movies") # Emby libraries to draw seeds from
RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db"
```
Requires `HOST*_TMDB_API_KEY` in `master_host*.conf`.
Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
---
@@ -266,7 +266,7 @@ SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db"
# SONARR_EMBY_LIBRARIES is shared with emby_to_sonarr_sync — see Orchestrator Job Order
```
Requires `HOST*_TMDB_API_KEY` in `master_host*.conf`.
Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
> **MONITOR_MODE note:** Use `"all"` (default) to have Sonarr search all existing seasons
> after adding a show. `"future"` only marks upcoming seasons as monitored — shows where
@@ -289,9 +289,9 @@ MEDIA_MAINTENANCE_JOBS=(
---
## ━━━ CONFIGURATION — master_host*.conf ━━━
## ━━━ CONFIGURATION — host*.conf ━━━
### master_host1.conf
### host1.conf
```bash
# Shares this server applies permissions to
@@ -429,7 +429,7 @@ cp radarr_cleanup.sh readarr_cleanup.sh
# 2. Replace RADARR_ prefix with READARR_ throughout
# Update API endpoint, tracked file API path, extension list, protected patterns
# 3. Add to master_host*.conf
# 3. Add to host*.conf
HOST1_READARR_URL="http://192.168.50.2:8787"
HOST1_READARR_API_KEY="your-api-key"
HOST1_READARR_BOOKS_ROOT="/mnt/user/Books"
@@ -534,7 +534,7 @@ If ghosts persist:
1. Is Emby's API responding?
curl -s "http://[emby-ip]:8096/System/Info/Public"
2. Is EMBY_URL / EMBY_API_KEY correct in master_host*.conf?
2. Is EMBY_URL / EMBY_API_KEY correct in host*.conf?
Run: sonarr_cleanup.sh --status (shows Emby config)
3. Trigger manually in Emby:
@@ -543,6 +543,23 @@ If ghosts persist:
---
## ━━━ OUTPUT TIERS ━━━
All scripts have two output levels controlled by `--log`.
Without `--log`, each script processes silently and always concludes with a summary
block: identity, duration, counts (files removed, items added, arrs cleaned), and a
status line. Warnings and errors are always visible.
With `--log`, per-item detail appears: individual titles being processed, API query
progress, per-node sync results, and per-file examination output. Use this when
debugging unexpected results or validating configuration before the first scheduled run.
Dry-run output follows the same tiers — `--dry-run` alone shows the summary of what
would happen; `--dry-run --log` shows the full per-item preview list.
---
## ━━━ FLAG REFERENCE ━━━
### media_shares_permissions.sh
+2 -5
View File
@@ -78,7 +78,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_LIDARR_URL / HOST*_LIDARR_API_KEY — local Lidarr (aliased by detect_hosts)
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY — local Sonarr
@@ -154,9 +154,6 @@ fi
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -822,7 +819,7 @@ _sync_arr() {
fi
fi
echo " $node_name: +${#to_add_local[@]} local | +${#to_add_remote[@]} remote | $total_skipped blocklisted"
log " $node_name: +${#to_add_local[@]} local | +${#to_add_remote[@]} remote | $total_skipped blocklisted"
unset remote_ids
declare -A remote_ids
+2 -5
View File
@@ -54,7 +54,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_RECOVERY
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_RECOVERY
@@ -92,9 +92,6 @@ parse_args "$@"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -451,7 +448,7 @@ elif [[ "$TOTAL_ACTIONED" -gt 0 ]]; then
notify "Arr recovery on $(hostname)$TOTAL_ACTIONED item(s) blocklisted and re-searched" \
"Arr Recovery" "warning"
else
log "$ICON_DONE Done — nothing to recover (all arrs clean)"
echo "$ICON_DONE Done — nothing to recover (all arrs clean)"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+6 -10
View File
@@ -58,7 +58,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY / HOST1_LIDARR_MUSIC_ROOT
# HOST1_LIDARR_PATH_MAP — container path → host path translation
@@ -136,9 +136,6 @@ fi
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -347,7 +344,7 @@ SCAN_CMD_ID=$(echo "$SCAN_RESPONSE" | jq -r '.id // empty' 2>/dev/null)
if [[ -z "$SCAN_CMD_ID" ]]; then
warn "Could not trigger import scan — proceeding without pre-flight"
else
echo " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
log " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
POLL_TIMEOUT=${LIDARR_IMPORT_SCAN_TIMEOUT:-600}
POLLED=0
while [[ "$POLLED" -lt "$POLL_TIMEOUT" ]]; do
@@ -382,7 +379,7 @@ fi
# Safety Layer 3 — API version check
check_arr_version "$LIDARR_URL" "$LIDARR_API_KEY" "v1" "$LIDARR_VERSION_MAJOR" "Lidarr" || exit 1
echo " Querying Lidarr API..."
log " Querying Lidarr API..."
# Fetch all artists
ARTIST_RESPONSE=$(lidarr_api "artist") || {
@@ -443,7 +440,7 @@ if [[ "$TRACKED_COUNT" -eq 0 ]]; then
exit 1
fi
echo " $ARTIST_COUNT artists | $TRACKED_COUNT tracked files"
log " $ARTIST_COUNT artists | $TRACKED_COUNT tracked files"
# Safety Layer 6 — percentage drop vs last known count
if [[ -f "$LIDARR_TRACKED_COUNT_FILE" ]]; then
@@ -471,8 +468,7 @@ echo "$TRACKED_COUNT" > "$LIDARR_TRACKED_COUNT_FILE"
# ==============================================================================================
echo ""
echo "━━━ $ICON_CLEAN Scanning Music Root ━━━"
echo " Root: $LIDARR_MUSIC_ROOT | Orphan age: ${LIDARR_ORPHAN_AGE} days"
echo ""
log " Root: $LIDARR_MUSIC_ROOT | Orphan age: ${LIDARR_ORPHAN_AGE} days"
START=$(date +%s)
ORPHAN_COUNT=0
@@ -605,7 +601,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no files deleted"
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
log "$ICON_DONE Clean — nothing to remove"
echo "$ICON_DONE Clean — nothing to remove"
else
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
notify "Lidarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" "Lidarr Cleanup" "warning"
+9 -12
View File
@@ -42,7 +42,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY
# Aliased by detect_hosts() — script uses LIDARR_URL / LIDARR_API_KEY
@@ -76,9 +76,6 @@ parse_args "$@"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -110,7 +107,7 @@ if [[ -z "$LIDARR_URL" ]]; then
exit 0
fi
echo " $MY_ID ($LOCAL_SERVER_NAME) — tools OK"
log " $MY_ID ($LOCAL_SERVER_NAME) — tools OK"
# ==============================================================================================
# ━━━ Status ━━━
@@ -218,7 +215,7 @@ if ! curl_json "$LIDARR_URL/api/v1/system/status?apikey=$LIDARR_API_KEY" | jq -e
notify "lidarr_missing_art failed — Lidarr API unreachable on $(hostname)" "Lidarr Missing Art" "warning"
exit 1
fi
echo " Reachable — $LIDARR_URL"
log " Reachable — $LIDARR_URL"
START=$(date +%s)
@@ -238,7 +235,7 @@ echo "━━━ $ICON_SYNC Building Album Directory Map ━━━"
declare -A ALBUM_DIR_MAP
_artist_list=$(curl_json "$LIDARR_URL/api/v1/artist?apikey=$LIDARR_API_KEY")
_map_artist_count=$(echo "$_artist_list" | jq '. | length')
echo " Fetching track files for $_map_artist_count artists..."
log " Fetching track files for $_map_artist_count artists..."
while IFS= read -r _artist_id; do
[[ -z "$_artist_id" ]] && continue
@@ -250,7 +247,7 @@ while IFS= read -r _artist_id; do
done < <(echo "$_artist_list" | jq -r '.[].id')
unset _artist_list _map_artist_count _artist_id _album_id _track_path
echo " Mapped ${#ALBUM_DIR_MAP[@]} albums with local tracks"
log " Mapped ${#ALBUM_DIR_MAP[@]} albums with local tracks"
# ==============================================================================================
# ━━━ Albums ━━━
@@ -267,7 +264,7 @@ if [[ -z "$albums" || "$albums" == "null" ]]; then
fi
total_albums=$(echo "$albums" | jq '. | length')
echo " Processing $total_albums albums..."
log " Processing $total_albums albums..."
while IFS=$'\t' read -r mbid artist_name album_name album_id; do
raw_dir="${ALBUM_DIR_MAP[$album_id]:-}"
@@ -335,7 +332,7 @@ wait
ALBUM_FETCHED=$(wc -l < "$LIDARR_TMP/album_fetches" 2>/dev/null || echo 0)
ALBUM_FAILED=$(wc -l < "$LIDARR_TMP/album_fails" 2>/dev/null || echo 0)
ALBUM_MISSING=$(( ALBUMS_CHECKED - ALBUMS_COMPLETE ))
echo " Checked: $ALBUMS_CHECKED | Complete: $ALBUMS_COMPLETE | Needed art: $ALBUM_MISSING | Fetched: $ALBUM_FETCHED | Failed: $ALBUM_FAILED"
log " Checked: $ALBUMS_CHECKED | Complete: $ALBUMS_COMPLETE | Needed art: $ALBUM_MISSING | Fetched: $ALBUM_FETCHED | Failed: $ALBUM_FAILED"
# ==============================================================================================
# ━━━ Artists ━━━
@@ -352,7 +349,7 @@ if [[ -z "$artists" || "$artists" == "null" ]]; then
fi
total_artists=$(echo "$artists" | jq '. | length')
echo " Processing $total_artists artists..."
log " Processing $total_artists artists..."
while IFS=$'\t' read -r local_path mbid name; do
local_path=$(translate_path "$local_path")
@@ -430,7 +427,7 @@ wait
ARTIST_FETCHED=$(wc -l < "$LIDARR_TMP/artist_fetches" 2>/dev/null || echo 0)
ARTIST_FAILED=$(wc -l < "$LIDARR_TMP/artist_fails" 2>/dev/null || echo 0)
ARTIST_MISSING=$(( ARTISTS_CHECKED - ARTISTS_COMPLETE ))
echo " Checked: $ARTISTS_CHECKED | Complete: $ARTISTS_COMPLETE | Needed art: $ARTIST_MISSING | Fetched: $ARTIST_FETCHED | Failed: $ARTIST_FAILED"
log " Checked: $ARTISTS_CHECKED | Complete: $ARTISTS_COMPLETE | Needed art: $ARTIST_MISSING | Fetched: $ARTIST_FETCHED | Failed: $ARTIST_FAILED"
END=$(date +%s)
+30 -20
View File
@@ -5,10 +5,10 @@
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Remove junk files from media shares using configurable file patterns. Two
# profiles — anime and media — each with their own folder list and patterns.
# Runs second in the daily maintenance window, after permissions and before
# arr cleanup, so orphan detection only encounters actual media files.
# Remove scene debris, tool artifacts, and unsafe files from media shares
# before the orphan scan — so arr cleanup only encounters actual media files.
# Runs second in the daily window, after permissions and before arr cleanup.
# Two profiles — anime and media — each with their own folder list and patterns.
#
# ==============================================================================================
# OPERATIONAL MODEL
@@ -18,15 +18,29 @@
# anime — cleans ANIME_CLEAN_FOLDERS using ANIME_FILE_PATTERNS
# media — cleans MEDIA_CLEAN_FOLDERS using MEDIA_FILE_PATTERNS (adds *.iso *.lrc)
#
# Removes junk left by download clients, scene releases, and tools:
# *.sfv *.md5 *.sha1 — checksum files — useless post-download
# *.nfo *.url *.lnk — scene info files — not needed in media library
# *.rar *.zip — archives — source files not needed after extraction
# *.sample* *.proof* — scene samples — never needed
# *sync-conflict* — Syncthing conflict files
# *.scr *.exe — executables — should never be in a media folder
# *.torrent torrent files left by download clients
# *.log *.json — tool output files
# Scene debris — left by scene releases and download clients:
# *.sfv *.md5 *.sha1 — checksums — useless post-download
# *.nzb *.torrent — download files left by clients
# *.url *.lnk *.info *.diz — scene metadata
# *.sample* *.proof* — scene samples — never needed in library
# *sync-conflict* — Syncthing conflict copies
# *.rar *.zip *.7z *.ace — archives — source not needed after extraction
# *.r00-*.r09 *.srrmulti-part rar segments and repair files
# *.001 *.002 *.003 — split archive parts
# *.gz *.tar *.bz2 — linux archives
#
# Tool artifacts — incomplete or stale files from download clients:
# *.!ut *.!qB — uTorrent / qBittorrent incomplete markers
# *.crdownload *.opdownload — Chrome / Opera incomplete downloads
# *.part — partial download files
#
# Unsafe files — executables that should never appear in a media folder:
# *.exe *.scr *.com — Windows executables
# *.bat *.cmd *.vbs *.ps1 — Windows scripts
# *.msi *.dll *.sys — Windows system files
# *.sh — shell scripts in media folders = suspicious
#
# Media profile also removes: *.iso *.lrc
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
@@ -37,13 +51,12 @@
# Empty array guards — warns and exits cleanly if no folders or patterns configured
# Folder existence — skips missing folders with warning, continues others
# validate_unraid_cmd — notify script validated before use
# Silent by default — only problems and removals produce output
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_ANIME_CLEAN_FOLDERS — folders cleaned by the anime profile on this host
# HOST*_MEDIA_CLEAN_FOLDERS — folders cleaned by the media profile on this host
@@ -88,9 +101,6 @@ parse_args "${RAW_ARGS[@]}"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -131,7 +141,7 @@ esac
# Empty array guards
if [[ ${#CLEAN_FOLDERS[@]} -eq 0 ]]; then
warn "No folders configured for profile '$PROFILE' on $MY_ID"
warn "Check HOST*_${PROFILE^^}_CLEAN_FOLDERS in master_host*.conf"
warn "Check HOST*_${PROFILE^^}_CLEAN_FOLDERS in host*.conf"
exit 0
fi
@@ -247,7 +257,7 @@ elif [[ ${#FAILED[@]} -gt 0 ]]; then
notify "Media cleaner ($PROFILE) failed on $(hostname)${FAILED[*]}" \
"Media Cleaner" "warning"
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
log "$ICON_DONE Status: clean — nothing to remove"
echo "$ICON_DONE Status: clean — nothing to remove"
else
warn "$ICON_TRASH Removed: $TOTAL_REMOVED file(s)"
notify "Media cleaner ($PROFILE) on $(hostname)$TOTAL_REMOVED file(s) removed" \
+2 -5
View File
@@ -34,7 +34,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_MEDIA_PERMISSION_SHARES — shares this host applies permissions to
# Aliased by detect_hosts() — script uses MEDIA_PERMISSION_SHARES
@@ -65,9 +65,6 @@ parse_args "$@"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -86,7 +83,7 @@ detect_hosts
# Empty array guard
if [[ ${#MEDIA_PERMISSION_SHARES[@]} -eq 0 ]]; then
warn "MEDIA_PERMISSION_SHARES is empty for $MY_ID — nothing to do"
warn "Check HOST*_MEDIA_PERMISSION_SHARES in master_host*.conf"
warn "Check HOST*_MEDIA_PERMISSION_SHARES in host*.conf"
exit 0
fi
+4 -4
View File
@@ -57,7 +57,7 @@
# ==============================================================================================
#
# Last.fm API key — required for both stages
# Configure HOST*_LASTFM_API_KEY in master_host*.conf
# Configure HOST*_LASTFM_API_KEY in host*.conf
#
# ==============================================================================================
# CONFIGURATION (master.conf)
@@ -135,18 +135,18 @@ if ! command -v jq >/dev/null 2>&1; then
fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then
error "LIDARR_URL / LIDARR_API_KEY not configured — check master_host*.conf"
error "LIDARR_URL / LIDARR_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${LASTFM_API_KEY:-}" ]]; then
error "LASTFM_API_KEY not configured — required for discovery"
error "Configure HOST*_LASTFM_API_KEY in master_host*.conf"
error "Configure HOST*_LASTFM_API_KEY in host*.conf"
exit 1
fi
+4 -4
View File
@@ -55,7 +55,7 @@
# ==============================================================================================
#
# TMDB API key — required for Stage 2 recommendations
# Configure HOST*_TMDB_API_KEY in master_host*.conf
# Configure HOST*_TMDB_API_KEY in host*.conf
# Free key at: https://www.themoviedb.org/settings/api
#
# ==============================================================================================
@@ -136,19 +136,19 @@ if ! command -v jq >/dev/null 2>&1; then
fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${RADARR_URL:-}" || -z "${RADARR_API_KEY:-}" ]]; then
error "RADARR_URL / RADARR_API_KEY not configured — check master_host*.conf"
error "RADARR_URL / RADARR_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${TMDB_API_KEY:-}" ]]; then
error "TMDB_API_KEY not configured — required for discovery"
error "Get a free key at https://www.themoviedb.org/settings/api"
error "Configure HOST*_TMDB_API_KEY in master_host*.conf"
error "Configure HOST*_TMDB_API_KEY in host*.conf"
exit 1
fi
+4 -4
View File
@@ -62,7 +62,7 @@
# ==============================================================================================
#
# TMDB API key — required for Stage 2 recommendations and external_ids lookup
# Configure HOST*_TMDB_API_KEY in master_host*.conf
# Configure HOST*_TMDB_API_KEY in host*.conf
# Free key at: https://www.themoviedb.org/settings/api
#
# ==============================================================================================
@@ -150,19 +150,19 @@ if ! command -v jq >/dev/null 2>&1; then
fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${SONARR_URL:-}" || -z "${SONARR_API_KEY:-}" ]]; then
error "SONARR_URL / SONARR_API_KEY not configured — check master_host*.conf"
error "SONARR_URL / SONARR_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${TMDB_API_KEY:-}" ]]; then
error "TMDB_API_KEY not configured — required for discovery"
error "Get a free key at https://www.themoviedb.org/settings/api"
error "Configure HOST*_TMDB_API_KEY in master_host*.conf"
error "Configure HOST*_TMDB_API_KEY in host*.conf"
exit 1
fi
+6 -10
View File
@@ -53,7 +53,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_MOVIES_ROOT
# HOST*_RADARR_PATH_MAP — container path → host path translation
@@ -125,9 +125,6 @@ fi
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -333,7 +330,7 @@ SCAN_CMD_ID=$(echo "$SCAN_RESPONSE" | jq -r '.id // empty' 2>/dev/null)
if [[ -z "$SCAN_CMD_ID" ]]; then
warn "Could not trigger import scan — proceeding without pre-flight"
else
echo " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
log " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
POLL_TIMEOUT=${RADARR_IMPORT_SCAN_TIMEOUT:-600}
POLLED=0
while [[ "$POLLED" -lt "$POLL_TIMEOUT" ]]; do
@@ -368,7 +365,7 @@ fi
# Safety Layer 3 — API version check
check_arr_version "$RADARR_URL" "$RADARR_API_KEY" "v3" "$RADARR_VERSION_MAJOR" "Radarr" || exit 1
echo " Querying Radarr API..."
log " Querying Radarr API..."
# Fetch all movies
MOVIES_RESPONSE=$(radarr_api "movie") || {
@@ -433,15 +430,14 @@ if [[ "$TRACKED_COUNT" -eq 0 ]]; then
exit 1
fi
echo " $MOVIE_COUNT movies | $TRACKED_COUNT tracked movie files"
log " $MOVIE_COUNT movies | $TRACKED_COUNT tracked movie files"
# ==============================================================================================
# ━━━ Scan Movies Root ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_CLEAN Scanning Movies Root ━━━"
echo " Root: $RADARR_MOVIES_ROOT | Orphan age: ${RADARR_ORPHAN_AGE} days"
echo ""
log " Root: $RADARR_MOVIES_ROOT | Orphan age: ${RADARR_ORPHAN_AGE} days"
START=$(date +%s)
ORPHAN_COUNT=0
@@ -571,7 +567,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no files deleted"
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
log "$ICON_DONE Clean — nothing to remove"
echo "$ICON_DONE Clean — nothing to remove"
else
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
notify "Radarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
+4 -7
View File
@@ -39,7 +39,7 @@
#
# RADARR_DROPPED_ADD_EXCLUSION — add removed movies to import exclusion (default: true)
#
# master_host*.conf
# host*.conf
#
# HOST1_RADARR_URL / HOST1_RADARR_API_KEY
# Aliased by detect_hosts() — script uses RADARR_URL / RADARR_API_KEY
@@ -76,9 +76,6 @@ parse_args "${FILTERED_ARGS[@]}"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -176,9 +173,9 @@ while IFS=$'\t' read -r id title year tmdb_id has_file file_size; do
SIZE_HUMAN=""
if [[ "$has_file" == "true" && "$file_size" -gt 0 ]]; then
SIZE_HUMAN=$(awk "BEGIN {printf \"%.1fGB\", $file_size / 1073741824}")
echo "$ICON_WARN $title ($year) [tmdbid $tmdb_id] — HAS FILE: $SIZE_HUMAN"
log "$ICON_WARN $title ($year) [tmdbid $tmdb_id] — HAS FILE: $SIZE_HUMAN"
else
echo "$ICON_TRASH $title ($year) [tmdbid $tmdb_id] — no file"
log "$ICON_TRASH $title ($year) [tmdbid $tmdb_id] — no file"
fi
if [[ "$DRY_RUN" == true ]]; then
@@ -242,7 +239,7 @@ echo "$ICON_TRASH Removed: ${#REMOVED[@]} of $DROPPED"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ ${#FAILED[@]} -eq 0 ]]; then
log "$ICON_DONE Status: done ✅"
echo "$ICON_DONE Status: done ✅"
else
warn "Status: ${#FAILED[@]} removal(s) failed"
fi
+6 -10
View File
@@ -53,7 +53,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_TV_ROOT
# HOST*_SONARR_PATH_MAP — container path → host path translation
@@ -125,9 +125,6 @@ fi
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -333,7 +330,7 @@ SCAN_CMD_ID=$(echo "$SCAN_RESPONSE" | jq -r '.id // empty' 2>/dev/null)
if [[ -z "$SCAN_CMD_ID" ]]; then
warn "Could not trigger import scan — proceeding without pre-flight"
else
echo " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
log " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
POLL_TIMEOUT=${SONARR_IMPORT_SCAN_TIMEOUT:-600}
POLLED=0
while [[ "$POLLED" -lt "$POLL_TIMEOUT" ]]; do
@@ -368,7 +365,7 @@ fi
# Safety Layer 3 — API version check
check_arr_version "$SONARR_URL" "$SONARR_API_KEY" "v3" "$SONARR_VERSION_MAJOR" "Sonarr" || exit 1
echo " Querying Sonarr API..."
log " Querying Sonarr API..."
# Fetch all series
SERIES_RESPONSE=$(sonarr_api "series") || {
@@ -432,15 +429,14 @@ if [[ "$TRACKED_COUNT" -eq 0 ]]; then
exit 1
fi
echo " $SERIES_COUNT series | $TRACKED_COUNT tracked episode files"
log " $SERIES_COUNT series | $TRACKED_COUNT tracked episode files"
# ==============================================================================================
# ━━━ Scan TV Root ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_CLEAN Scanning TV Root ━━━"
echo " Root: $SONARR_TV_ROOT | Orphan age: ${SONARR_ORPHAN_AGE} days"
echo ""
log " Root: $SONARR_TV_ROOT | Orphan age: ${SONARR_ORPHAN_AGE} days"
START=$(date +%s)
ORPHAN_COUNT=0
@@ -570,7 +566,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no files deleted"
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
log "$ICON_DONE Clean — nothing to remove"
echo "$ICON_DONE Clean — nothing to remove"
else
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
notify "Sonarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
+4 -7
View File
@@ -38,7 +38,7 @@
#
# SONARR_DROPPED_ADD_EXCLUSION — add removed series to import exclusion (default: true)
#
# master_host*.conf
# host*.conf
#
# HOST1_SONARR_URL / HOST1_SONARR_API_KEY
# Aliased by detect_hosts() — script uses SONARR_URL / SONARR_API_KEY
@@ -75,9 +75,6 @@ parse_args "${FILTERED_ARGS[@]}"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -178,9 +175,9 @@ while IFS=$'\t' read -r id title year tvdb_id episode_file_count size_on_disk; d
fi
if [[ "$episode_file_count" -gt 0 ]]; then
echo "$ICON_WARN $title ($year) [tvdbid $tvdb_id] — $episode_file_count episode files${SIZE_HUMAN:+, $SIZE_HUMAN}"
log "$ICON_WARN $title ($year) [tvdbid $tvdb_id] — $episode_file_count episode files${SIZE_HUMAN:+, $SIZE_HUMAN}"
else
echo "$ICON_TRASH $title ($year) [tvdbid $tvdb_id] — no files"
log "$ICON_TRASH $title ($year) [tvdbid $tvdb_id] — no files"
fi
if [[ "$DRY_RUN" == true ]]; then
@@ -244,7 +241,7 @@ echo "$ICON_TRASH Removed: ${#REMOVED[@]} of $DROPPED"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ ${#FAILED[@]} -eq 0 ]]; then
log "$ICON_DONE Status: done ✅"
echo "$ICON_DONE Status: done ✅"
else
warn "Status: ${#FAILED[@]} removal(s) failed"
fi
+29 -14
View File
@@ -13,7 +13,7 @@ Each domain and subdomain has an independent TLS certificate and must be listed
separately. `cert_monitor.sh` makes one openssl connection per entry.
```bash
# master_host1.conf
# host1.conf
HOST1_CERT_MONITOR_DOMAINS=(
"Gmer4Lfe.com"
"Gmer4Lfe.us"
@@ -21,7 +21,7 @@ HOST1_CERT_MONITOR_DOMAINS=(
# "cloud.Gmer4Lfe.com"
)
# master_host2.conf
# host2.conf
HOST2_CERT_MONITOR_DOMAINS=(
"Jayred365.com"
# "auth.Jayred365.com"
@@ -63,12 +63,12 @@ CERT_TIMEOUT=10 # seconds per domain before declaring FAILED
### Drive Ignore List
```bash
# master_host1.conf
# host1.conf
HOST1_SMART_IGNORE_DRIVES=(
"sda" # boot USB flash drive — no meaningful SMART data
)
# master_host2.conf
# host2.conf
HOST2_SMART_IGNORE_DRIVES=(
"sda" # boot USB flash drive
)
@@ -109,7 +109,7 @@ if `dynamix.cfg` is not found (e.g., running outside of unRAID).
### Share Configuration
```bash
# master_host1.conf
# host1.conf
HOST1_BACKUP_VERIFY_SHARES=(
# empty — uses HOST1_DAILY_SYNC_SHARES automatically
# "/mnt/user/Movies" # override to check specific shares only
@@ -230,9 +230,9 @@ data — it just doesn't trigger a notification for that condition.
| State File | Source | What It Shows |
|-----------|--------|---------------|
| `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
| `SYS_WATCHDOG_FAILED_FILE` | `Docker_Essentials/docker_watchdog.sh` | Container skip list — needs human attention |
| `WATCHDOG_STATE_FILE` | `Docker_Essentials/docker_watchdog.sh` | Active container strike counts |
| `SYS_WATCHDOG_STATE_FILE` | `unRAID_Essentials/system_watchdog.sh` | Active system watchdog strikes |
| `SYS_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
| `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts |
| `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/system_watchdog.sh` | Active system watchdog strikes |
| `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |
| `TRANSCODE_DAILY_LOG` | `Transcodes/` | Weekly transcode statistics |
| `CERT_MONITOR_DOMAINS` | live openssl check | Current cert status per domain |
@@ -244,11 +244,11 @@ data — it just doesn't trigger a notification for that condition.
### Emby Credentials
```bash
# master_host1.conf
# host1.conf
HOST1_EMBY_URL="http://192.168.50.2:8096"
HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829"
# master_host2.conf
# host2.conf
HOST2_EMBY_URL="http://192.168.50.3:8096"
HOST2_EMBY_API_KEY="<host2_api_key>"
```
@@ -306,7 +306,7 @@ in `unRAID_Essentials/`.
### Pool Ignore List
```bash
# master_host1.conf
# host1.conf
HOST1_ZFS_REPORT_IGNORE_POOLS=(
"disk10" # JBOD member — high usage expected, exclude from report noise
"disk9"
@@ -315,7 +315,7 @@ HOST1_ZFS_REPORT_IGNORE_POOLS=(
"disk5"
)
# master_host2.conf
# host2.conf
HOST2_ZFS_REPORT_IGNORE_POOLS=(
# list host2's JBOD members here
)
@@ -392,7 +392,7 @@ ZFS_REPORT_AVAIL_WARN_GB=20
ZFS_REPORT_DOCKER_TOP=10
```
### master_host*.conf
### host*.conf
```bash
# cert_monitor.sh
@@ -445,6 +445,21 @@ HOST2_ZFS_REPORT_IGNORE_POOLS=()
---
## ━━━ OUTPUT TIERS ━━━
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
Most monitors are one-shot scripts that run on a schedule. Without `--log`, phase
headers, step conclusions (pre-flight passed, snapshot written, digest sent), and
the final status line are visible. Per-drive, per-cert, per-file, and per-domain
detail lines inside loops are suppressed.
`weekly_health_digest.sh` with `profile=smart` exits silently (one `echo` line)
when there are no findings worth reporting — no noise on healthy weeks.
---
## ━━━ FLAG REFERENCE ━━━
All monitor scripts support the same four flags:
@@ -470,7 +485,7 @@ emby_session_report.sh --dry-run # test connectivity, generate report, no noti
### --status
Shows current configuration and exits without running checks. Use to verify
configuration is loaded correctly after editing `master.conf` or `master_host*.conf`.
configuration is loaded correctly after editing `master.conf` or `host*.conf`.
```bash
cert_monitor.sh --status # domain list, CERT_WARN_DAYS, CERT_CRIT_DAYS, timeout
+4 -4
View File
@@ -63,7 +63,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_BACKUP_VERIFY_SHARES
# Shares to verify. Leave empty to use HOST*_DAILY_SYNC_SHARES automatically.
@@ -140,7 +140,7 @@ fi
if [[ ${#VERIFY_SHARES[@]} -eq 0 ]]; then
warn "No shares configured for $MY_ID — nothing to verify"
warn "Check HOST*_BACKUP_VERIFY_SHARES or HOST*_DAILY_SYNC_SHARES in master_host*.conf"
warn "Check HOST*_BACKUP_VERIFY_SHARES or HOST*_DAILY_SYNC_SHARES in host*.conf"
exit 0
fi
@@ -193,7 +193,7 @@ if ! check_remote_array; then
exit 1
fi
log "Pre-flight passed ✅"
echo "Pre-flight passed ✅"
# ==============================================================================================
# ━━━ Backup Verification ━━━
@@ -319,7 +319,7 @@ elif [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 ]]; then
notify "Backup verify FAILED on $(hostname)$REMOTE_SERVER_NAME — mismatches: $TOTAL_MISMATCH missing: $TOTAL_MISSING — shares: ${SHARES_WITH_ISSUES[*]}" \
"Backup Verify" "warning"
else
log "$ICON_DONE Status: all $TOTAL_CHECKED files match across ${#VERIFY_SHARES[@]} shares ✅"
echo "$ICON_DONE Status: all $TOTAL_CHECKED files match across ${#VERIFY_SHARES[@]} shares ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+1 -4
View File
@@ -90,9 +90,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
# Monitor script — output is the point
SILENT_MODE=false
parse_args "$@"
# ── Parse mode from PARSED_ARGS ───────────────────────────────────────────────────────────────
@@ -188,7 +185,7 @@ if [[ "$LOG_TRANSFER_MODE" == true ]]; then
# Append entry — format: date|time|profile|duration|status|bytes|warn_flag
echo "${TODAY}|${NOW}|${TRANSFER_PROFILE}|${TRANSFER_DURATION}|${TRANSFER_STATUS}|${TRANSFER_BYTES}|${WARN_FLAG}" \
>> "$BANDWIDTH_LOG"
log "$ICON_BANDWIDTH Logged: $TRANSFER_PROFILE${DURATION_FMT}$TRANSFER_STATUS${WARN_FLAG:+ [$WARN_FLAG]}"
echo "$ICON_BANDWIDTH Logged: $TRANSFER_PROFILE${DURATION_FMT}$TRANSFER_STATUS${WARN_FLAG:+ [$WARN_FLAG]}"
# Trim entries older than retention — atomic write via temp file
CUTOFF=$(date -d "${BANDWIDTH_LOG_RETENTION} days ago" '+%Y-%m-%d')
+3 -6
View File
@@ -51,7 +51,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_CERT_MONITOR_DOMAINS
# Domains this host monitors. Each domain and subdomain is a separate entry —
@@ -91,9 +91,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
# Monitor script — output is the point
SILENT_MODE=false
parse_args "$@"
# ==============================================================================================
@@ -126,7 +123,7 @@ detect_hosts
# Empty array guard
if [[ ${#CERT_MONITOR_DOMAINS[@]} -eq 0 ]]; then
warn "CERT_MONITOR_DOMAINS is empty for $MY_ID"
warn "Check HOST*_CERT_MONITOR_DOMAINS in master_host*.conf"
warn "Check HOST*_CERT_MONITOR_DOMAINS in host*.conf"
exit 0
fi
@@ -277,7 +274,7 @@ elif [[ ${#CRITICAL[@]} -gt 0 || ${#FAILED[@]} -gt 0 ]]; then
elif [[ ${#WARNING[@]} -gt 0 ]]; then
warn "Status: WARNINGS — renewal recommended"
else
log "$ICON_DONE Status: all ${#HEALTHY[@]} certs healthy ✅"
echo "$ICON_DONE Status: all ${#HEALTHY[@]} certs healthy ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+1 -4
View File
@@ -42,7 +42,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_EMBY_URL
# Emby server URL for this host. Aliased by detect_hosts() → EMBY_URL.
@@ -87,9 +87,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
# Monitor/report script — output is the point
SILENT_MODE=false
parse_args "$@"
# ==============================================================================================
+2 -5
View File
@@ -43,7 +43,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_SMART_IGNORE_DRIVES
# Drives skipped in SMART monitoring. Aliased by detect_hosts() →
@@ -80,9 +80,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
# Monitor script — output is the point
SILENT_MODE=false
parse_args "$@"
# ==============================================================================================
@@ -367,7 +364,7 @@ elif [[ ${#DRIVES_WARN[@]} -gt 0 ]]; then
notify "SMART WARNING on $(hostname) — drives showing wear: ${DRIVES_WARN[*]}" \
"SMART Health" "warning"
else
log "$ICON_DONE Status: all ${#DRIVES_OK[@]} drives healthy ✅"
echo "$ICON_DONE Status: all ${#DRIVES_OK[@]} drives healthy ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+1 -1
View File
@@ -245,4 +245,4 @@ fi
echo "${DATE}|${TIME}|${INOTIFY_USED}|${INOTIFY_LIMIT}|${INOTIFY_PCT}|${INOTIFY_WARN}|${PHPFPM_ACTIVE}|${PHPFPM_MAX}|${PHPFPM_PCT}|${PHPFPM_WARN}" \
>> "$TUNING_MONITOR_LOG"
log "Snapshot written: inotify ${INOTIFY_PCT}% php-fpm ${PHPFPM_PCT}%"
echo "Snapshot written: inotify ${INOTIFY_PCT}% php-fpm ${PHPFPM_PCT}%"
+2 -5
View File
@@ -116,9 +116,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
# Report/monitor script — output is the point when sending
SILENT_MODE=false
parse_args "$@"
# ==============================================================================================
@@ -341,7 +338,7 @@ fi
# ── Smart profile — exit silently if nothing to report ────────────────────────────────────────
# ==============================================================================================
if [[ "$DIGEST_PROFILE" == "smart" && "$SHOULD_SEND" == false ]]; then
log "Profile: smart — no findings worth reporting — silent exit"
echo "Profile: smart — no findings worth reporting — silent exit"
exit 0
fi
@@ -378,5 +375,5 @@ if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — digest generated but not sent"
elif [[ "$SHOULD_SEND" == true ]]; then
notify "$NOTIFY_MSG" "Health Digest" "$NOTIFY_SEV"
log "Digest sent"
echo "Digest sent"
fi
+3 -6
View File
@@ -67,7 +67,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_ZFS_REPORT_IGNORE_POOLS
# Pools excluded from health reporting. Single-disk JBOD members generate
@@ -113,9 +113,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
# Monitor/report script — output is the point
SILENT_MODE=false
parse_args "$@"
DOCKER_TIMEOUT=15
@@ -227,7 +224,7 @@ else
error "One or more ZFS pools are NOT ONLINE: $UNHEALTHY"
WARNINGS+=("ZFS pool unhealthy: $UNHEALTHY")
else
log "All monitored ZFS pools are ONLINE ✅"
echo "All monitored ZFS pools are ONLINE ✅"
fi
if [[ ${#ZFS_REPORT_IGNORE_POOLS[@]} -gt 0 ]]; then
@@ -349,7 +346,7 @@ echo "$ICON_ZFS Log: $ZFS_REPORT_LOG"
echo ""
if [[ ${#WARNINGS[@]} -eq 0 ]]; then
log "$ICON_DONE All checks within thresholds ✅"
echo "$ICON_DONE All checks within thresholds ✅"
else
echo "$ICON_WARN Warnings: ${#WARNINGS[@]}"
for w in "${WARNINGS[@]}"; do
+1 -1
View File
@@ -17,7 +17,7 @@
#
# ADDING A NEW ARR
# ─────────────────────────────────────────────────────────────────────────────
# 1. Add HOST*_<ARR>_URL, API_KEY, MEDIA_ROOT, PATH_MAP to master_host*.conf
# 1. Add HOST*_<ARR>_URL, API_KEY, MEDIA_ROOT, PATH_MAP to host*.conf
# 2. Add <ARR>_ORPHAN_AGE, MAX_DELETE_GB, EXTENSIONS, etc. to master.conf
# 3. Add a profile entry to ARR_PROFILES in arr_cleanup.py (6 values)
# 4. Add an export block for the new arr below (copy Sonarr block, change prefix)
+25 -5
View File
@@ -142,6 +142,26 @@ Single notification → one bell per run, not one per job
---
## ━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
**One-shot orchestrators** (`array_started.sh`, `array_stopping.sh`, `sunday_morning_coffee_report.sh`):
without `--log`, section headers, per-phase results, and the final summary are
visible. Per-item detail suppressed.
**Periodic orchestrators** (`critical_sync_maintenance.sh`, `daily_sync_maintenance.sh`,
`intermediate_sync_maintenance.sh`, `weekly_sync_maintenance.sh`): without `--log`,
phase headers, per-phase completion status, and the final summary are visible. Per-share
and per-job detail suppressed.
**Daemon orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent
during clean cycles. Only state transitions, errors, and startup-grace expiry shown
without `--log`.
---
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
| Script | What It Orchestrates | Schedule |
@@ -194,10 +214,10 @@ ARRAY_START_SCRIPTS=(
# watchdogs check container states
# ── Continuous scripts — run until array stops ─────────────────────────────
"unRAID_Essentials/system_watchdog.sh" # system health BEFORE docker watchdog —
"Watchdogs/system_watchdog.sh" # system health BEFORE docker watchdog —
# system watchdog writes state file that
# docker watchdog reads every cycle
"Docker_Essentials/docker_watchdog.sh" # container health BEFORE failover —
"Watchdogs/docker_watchdog.sh" # container health BEFORE failover —
# containers must be healthy for failover
# to make reliable decisions
"Failover/failover.sh" # failover LAST — needs everything else stable
@@ -445,7 +465,7 @@ new search — hands-free recovery while you sleep.
### ── Host Awareness ───────────────────────────────────────────────────────────
```bash
# master.conf + master_host*.conf
# master.conf + host*.conf
# ─────────────────────────────────────────────────────────────────────────────
# Each arr is independently toggled per host.
# Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action.
@@ -560,7 +580,7 @@ DAILY_MAINTENANCE_SCRIPTS=(
"Docker_Essentials/docker_daily_restart.sh" # POST-SYNC — restarts after everything
)
# master_host1.conf
# host1.conf
HOST1_DAILY_SYNC_SHARES=(
"/mnt/user/Movies" # HOST1 source of truth — push to HOST2
"/mnt/user/Tv_Shows" # HOST1 source of truth
@@ -575,7 +595,7 @@ HOST1_PERSONAL_SHARES=(
"/mnt/user/Personal" # encrypted personal share — appended after standard
)
# master_host2.conf
# host2.conf
HOST2_DAILY_SYNC_SHARES=(
"/mnt/user/Anime_Shows" # HOST2 source of truth — push to HOST1
"/mnt/user/Anime_Movies" # HOST2 source of truth
+3 -3
View File
@@ -18,8 +18,8 @@
# Docker_Essentials/docker_network_connect.sh — ensure networks + container connections
#
# CONTINUOUS (run until array stops):
# unRAID_Essentials/system_watchdog.sh — system health monitor (last line of defense)
# Docker_Essentials/docker_watchdog.sh — container health monitor
# Watchdogs/system_watchdog.sh — system health monitor (last line of defense)
# Watchdogs/docker_watchdog.sh — container health monitor
# Fallback/fallback.sh — mutual failover monitor
#
# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
@@ -212,6 +212,6 @@ elif [[ "$FAILED" -gt 0 ]]; then
notify "Array start on $(hostname) ($MY_ID) — $FAILED script(s) failed: ${FAILED_SCRIPTS[*]}" \
"Array Start" "warning"
else
log "$ICON_DONE Status: all $LAUNCHED script(s) launched ✅"
echo "$ICON_DONE Status: all $LAUNCHED script(s) launched ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+1 -1
View File
@@ -169,7 +169,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ ${#FAILED[@]} -eq 0 ]]; then
log "$ICON_DONE Status: all $STEP step(s) complete ✅"
echo "$ICON_DONE Status: all $STEP step(s) complete ✅"
notify "Array stop complete on $(hostname) ($MY_ID) — $STEP step(s) done" \
"Array Stop" "normal"
else
+6 -6
View File
@@ -120,12 +120,12 @@ PASS=()
FAIL=()
if ! check_rsync_enabled "CRITICAL"; then
log "Critical rsync disabled — skipping sync, running partnership check only"
echo "Critical rsync disabled — skipping sync, running partnership check only"
elif [[ ${#CRITICAL_SYNC_SHARES[@]} -eq 0 ]]; then
warn "CRITICAL_RSYNC_ENABLED=true but CRITICAL_SYNC_SHARES is empty for $MY_ID"
warn "Check HOST*_CRITICAL_SYNC_SHARES in master_host*.conf"
warn "Check HOST*_CRITICAL_SYNC_SHARES in host*.conf"
else
log "Critical sync — $MY_ID$REMOTE_ID$(date '+%H:%M:%S')"
echo "Critical sync — $MY_ID$REMOTE_ID$(date '+%H:%M:%S')"
# Build dry-run flag to pass through
RSYNC_DRY=""
@@ -204,7 +204,7 @@ if [[ "${PARTNERSHIP_ENABLED:-false}" == true ]]; then
--check --remote-unseen $PARTNER_DRY
fi
else
log "Partnership disabled — skipping check"
echo "Partnership disabled — skipping check"
fi
# ==============================================================================================
@@ -219,14 +219,14 @@ if [[ ${#FAIL[@]} -gt 0 ]]; then
echo "━━━━━ $ICON_SUMMARY CRITICAL SYNC SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Duration: $DURATION"
[[ ${#PASS[@]} -gt 0 ]] && log "Synced: ${PASS[*]}"
[[ ${#PASS[@]} -gt 0 ]] && echo "Synced: ${PASS[*]}"
echo "$ICON_ERROR Failed: ${FAIL[*]}"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
notify "Critical sync failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
"Critical Sync" "warning"
exit 1
else
log "Critical sync complete — $MY_ID${DURATION}${#PASS[@]} share(s)"
echo "Critical sync complete — $MY_ID${DURATION}${#PASS[@]} share(s)"
fi
exit 0
+5 -5
View File
@@ -53,7 +53,7 @@
# Summary always shown — gives window timing and share/job counts.
# Notify only on failure — successful daily maintenance doesn't need notification.
#
# ── CONFIGURATION (master.conf + master_host*.conf) ───────────────────────────────────────────
# ── CONFIGURATION (master.conf + host*.conf) ───────────────────────────────────────────
# HOST*_DAILY_SYNC_SHARES — shares pushed to mirror each day
# HOST*_PERSONAL_SHARES — encrypted personal shares
# DAILY_MAINTENANCE_SCRIPTS — maintenance jobs (permissions, cleanup, restart)
@@ -213,7 +213,7 @@ echo "━━━ $ICON_SYNC Arr Sync ━━━"
ARR_SYNC_SCRIPT="$SCRIPTS_ROOT/Media/arr_sync.sh"
if [[ "${ARR_SYNC_ENABLED:-true}" != "true" ]]; then
log "ARR_SYNC_ENABLED=false — skipping"
echo "ARR_SYNC_ENABLED=false — skipping"
elif [[ ! -f "$ARR_SYNC_SCRIPT" ]]; then
warn "arr_sync.sh not found at $ARR_SYNC_SCRIPT — skipping"
JOB_FAIL+=("arr_sync.sh")
@@ -221,7 +221,7 @@ else
_arr_sync_args=()
[[ "$DRY_RUN" == true ]] && _arr_sync_args+=("--dry-run")
if bash "$ARR_SYNC_SCRIPT" "${_arr_sync_args[@]}"; then
log "Arr sync complete ✅"
echo "Arr sync complete ✅"
JOB_PASS+=("arr_sync.sh")
else
warn "Arr sync completed with errors — continuing to rsync"
@@ -244,7 +244,7 @@ if ! check_rsync_enabled "DAILY"; then
warn "Daily rsync disabled — skipping all $SHARE_COUNT share syncs"
warn "Proceeding to maintenance jobs..."
elif [[ "$SHARE_COUNT" -eq 0 ]]; then
warn "No shares configured for $MY_ID — check HOST*_DAILY_SYNC_SHARES in master_host*.conf"
warn "No shares configured for $MY_ID — check HOST*_DAILY_SYNC_SHARES in host*.conf"
else
# Pre-flight — connectivity then remote rootfs
check_connectivity
@@ -354,7 +354,7 @@ if [[ "$TOTAL_FAIL" -gt 0 ]]; then
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
exit 1
else
log "$ICON_DONE Status: all complete — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
echo "$ICON_DONE Status: all complete — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
@@ -24,7 +24,7 @@
#
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
# detect_hosts() sets MY_ID and aliases HOST*_INTERMEDIATE_SYNC_SHARES → INTERMEDIATE_SYNC_SHARES.
# Each server can have a different set of mid-day shares — configure in master_host*.conf.
# Each server can have a different set of mid-day shares — configure in host*.conf.
# Each script in INTERMEDIATE_MAINTENANCE_SCRIPTS handles its own host logic.
#
# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
@@ -41,7 +41,7 @@
# Silent on success — runs 4x/day, only failures warrant notification
#
# ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
# master_host*.conf: HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped)
# host*.conf: HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped)
# master.conf: INTERMEDIATE_RSYNC_ENABLED — enable/disable rsync section (default: true)
# master.conf: INTERMEDIATE_MAINTENANCE_SCRIPTS — jobs run after rsync
# master.conf: ARR_SYNC_ENABLED — toggle inside arr_sync.sh
@@ -162,7 +162,7 @@ echo "━━━ $ICON_SYNC Arr Sync ━━━"
ARR_SYNC_SCRIPT="$SCRIPTS_ROOT/Media/arr_sync.sh"
if [[ "${ARR_SYNC_ENABLED:-true}" != "true" ]]; then
log "ARR_SYNC_ENABLED=false — skipping"
echo "ARR_SYNC_ENABLED=false — skipping"
elif [[ ! -f "$ARR_SYNC_SCRIPT" ]]; then
warn "arr_sync.sh not found at $ARR_SYNC_SCRIPT — skipping"
JOB_FAIL+=("arr_sync.sh")
@@ -170,7 +170,7 @@ else
_arr_sync_args=()
[[ "$DRY_RUN" == true ]] && _arr_sync_args+=("--dry-run")
if bash "$ARR_SYNC_SCRIPT" "${_arr_sync_args[@]}"; then
log "Arr sync complete ✅"
echo "Arr sync complete ✅"
JOB_PASS+=("arr_sync.sh")
else
warn "Arr sync completed with errors — continuing"
@@ -192,8 +192,8 @@ SHARE_INDEX=0
ABORT_ALL_SYNCS=false
if [[ "$SHARE_COUNT" -eq 0 ]]; then
log "No INTERMEDIATE_SYNC_SHARES configured — skipping"
log "Add shares to INTERMEDIATE_SYNC_SHARES in master.conf to enable mid-day sync"
echo "No INTERMEDIATE_SYNC_SHARES configured — skipping"
echo "Add shares to INTERMEDIATE_SYNC_SHARES in master.conf to enable mid-day sync"
elif ! check_rsync_enabled "INTERMEDIATE"; then
warn "Intermediate rsync disabled — skipping all $SHARE_COUNT share sync(s)"
else
@@ -300,7 +300,7 @@ TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
log "$ICON_DONE Status: all complete ✅ — ${#JOB_PASS[@]} job(s) run, ${#PASS[@]}/$SHARE_COUNT share(s) synced"
echo "$ICON_DONE Status: all complete ✅ — ${#JOB_PASS[@]} job(s) run, ${#PASS[@]}/$SHARE_COUNT share(s) synced"
else
warn "Status: $TOTAL_FAIL failure(s)"
notify "Intermediate sync failed on $(hostname) ($MY_ID) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
+89 -3
View File
@@ -154,6 +154,8 @@ line() { REPORT+=(" $1"); }
issue() { ISSUES+=("$1"); REPORT+=(" ⚠️ $1"); }
finding() { FINDINGS+=("$1"); REPORT+=(" $1"); }
get_array() { eval "echo \"\${${1}[*]}\""; }
format_bytes() {
local bytes=$1
if (( bytes > 1073741824 )); then
@@ -656,6 +658,90 @@ if command -v tailscale >/dev/null 2>&1; then
fi
fi
# ==============================================================================================
# ━━━ 🔗 MESH ━━━
# ==============================================================================================
section "🔗 MESH"
_ALL_HOST_IDS=()
for _h in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do
[[ -n "${!_h:-}" ]] && _ALL_HOST_IDS+=("$_h")
done
# ── Members ──
if [[ ${#_ALL_HOST_IDS[@]} -eq 0 ]]; then
line "No hosts defined"
else
line "Members:"
for _h in "${_ALL_HOST_IDS[@]}"; do
_server="${!_h}"
_owner_var="${_h}_OWNER"; _owner="${!_owner_var:-unknown}"
_email_var="${_h}_OWNER_EMAIL"; _email="${!_email_var:-(not set)}"
line " $_h $_server / $_owner / $_email"
done
fi
# ── Protected Services ──
_coverage_found=false
for _covered in "${_ALL_HOST_IDS[@]}"; do
_covered_owner_var="${_covered}_OWNER"; _covered_owner="${!_covered_owner_var:-$_covered}"
_covered_email_var="${_covered}_OWNER_EMAIL"; _covered_email="${!_covered_email_var:-}"
declare -A _tier_containers=()
declare -A _tier_delays=()
_covered_by=""
_any_tiers=false
for _covering in "${_ALL_HOST_IDS[@]}"; do
[[ "$_covering" == "$_covered" ]] && continue
for _tier in 1 2 3 4; do
_containers=$(get_array "FALLBACK_${_covering}_COVERS_${_covered}_TIER${_tier}")
[[ -z "$_containers" ]] && continue
_any_tiers=true
_tier_containers[$_tier]="$_containers"
_delay_var="${_covered}_TIER${_tier}_DELAY"
_tier_delays[$_tier]="${!_delay_var:-0}"
done
if [[ "$_any_tiers" == true ]]; then
_cov_owner_var="${_covering}_OWNER"; _cov_owner="${!_cov_owner_var:-$_covering}"
_covered_by="$_covering ($_cov_owner)"
fi
done
[[ "$_any_tiers" == false ]] && { unset _tier_containers _tier_delays; declare -A _tier_containers=() _tier_delays=(); continue; }
_coverage_found=true
_hdr="$_covered_owner"
[[ -n "$_covered_email" ]] && _hdr+="$_covered_email"
line "Protected: $_hdr"
for _tier in 1 2 3 4; do
[[ -z "${_tier_containers[$_tier]:-}" ]] && continue
_d="${_tier_delays[$_tier]:-0}"
if (( _d == 0 )); then _dlabel="immediate"
elif (( _d >= 1440 )); then _dlabel="$(( _d / 1440 ))d"
elif (( _d >= 60 )); then _dlabel="$(( _d / 60 ))hr"
else _dlabel="${_d}min"
fi
line " Tier $_tier (${_dlabel}): ${_tier_containers[$_tier]// /, }"
done
[[ -n "$_covered_by" ]] && line " Covered by: $_covered_by"
unset _tier_containers _tier_delays
declare -A _tier_containers=() _tier_delays=()
done
[[ "$_coverage_found" == false ]] && line "No fallback coverage configured"
# ── Partnership ──
if [[ "${PARTNERSHIP_ENABLED:-false}" == true ]]; then
_po_host="${PARTNERSHIP_OWNER_HOST:-HOST1}"
_po_server="${!_po_host:-unknown}"
_po_name_var="${_po_host}_OWNER"; _po_name="${!_po_name_var:-unknown}"
line "Partnership: enabled — owner $_po_host ($_po_server / $_po_name), sync every ${PARTNERSHIP_SYNC_INTERVAL:-15}min"
else
line "Partnership: disabled"
fi
# ==============================================================================================
# ━━━ 🔐 SECURITY ━━━
# ==============================================================================================
@@ -860,7 +946,7 @@ else
NOTIFY_SCRIPT="/usr/local/emhttp/plugins/dynamix/scripts/notify"
if [[ -x "$NOTIFY_SCRIPT" ]]; then
"$NOTIFY_SCRIPT" -s "☕ Weekly Report — $MY_ID" -d "$BODY" -i "normal" 2>/dev/null
log "unRAID notification sent"
echo "unRAID notification sent"
fi
fi
if [[ -n "${DISCORD_WEBHOOK:-}" ]]; then
@@ -871,8 +957,8 @@ else
PAYLOAD="{\"content\": ${ESCAPED_BODY}}"
curl -sf -H "Content-Type: application/json" \
-d "$PAYLOAD" "$DISCORD_WEBHOOK" >/dev/null 2>&1 && \
log "Discord notification sent" || \
echo "Discord notification sent" || \
warn "Discord notification failed"
fi
log "Report generated — $MY_ID${#ISSUES[@]} issue(s) ${#FINDINGS[@]} finding(s)"
echo "Report generated — $MY_ID${#ISSUES[@]} issue(s) ${#FINDINGS[@]} finding(s)"
fi
+1 -1
View File
@@ -87,7 +87,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
if [[ "$UPTIME_S" -lt "$WATCHDOG_STARTUP_GRACE" ]]; then
warn "Within startup grace — $(format_duration $UPTIME_S) / $(format_duration $WATCHDOG_STARTUP_GRACE)"
else
log "Past startup grace — $(format_duration $UPTIME_S) uptime"
echo "Past startup grace — $(format_duration $UPTIME_S) uptime"
fi
echo ""
+4 -4
View File
@@ -231,7 +231,7 @@ if [[ "$WEEKLY_SYNC_UPDATES" == true ]]; then
done
fi
else
log "WEEKLY_SYNC_UPDATES=false — skipping local updates"
echo "WEEKLY_SYNC_UPDATES=false — skipping local updates"
fi
if [[ "$WEEKLY_SYNC_UPDATES_REMOTE" == true ]]; then
@@ -261,7 +261,7 @@ if [[ "$WEEKLY_SYNC_UPDATES_REMOTE" == true ]]; then
done
fi
else
log "WEEKLY_SYNC_UPDATES_REMOTE=false — skipping remote updates"
echo "WEEKLY_SYNC_UPDATES_REMOTE=false — skipping remote updates"
fi
# ==============================================================================================
@@ -352,7 +352,7 @@ else
_remainder_args=("--remainder")
[[ "$DRY_RUN" == true ]] && _remainder_args+=("--dry-run")
if bash "$DOCKER_UPDATE_SCRIPT" "${_remainder_args[@]}"; then
log "Remainder updates complete ✅"
echo "Remainder updates complete ✅"
JOB_PASS+=("docker_update.sh --remainder")
else
warn "Remainder updates completed with errors"
@@ -392,7 +392,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
log "$ICON_DONE Status: all complete ✅ — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
echo "$ICON_DONE Status: all complete ✅ — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
else
warn "Status: $TOTAL_FAIL failure(s)"
notify "Weekly maintenance failed on $(hostname) ($MY_ID) — sync: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
+28 -5
View File
@@ -130,7 +130,7 @@ PARTNERSHIP_ONBOARD_VERIFY=true # curl-verify each WebUI after onboard
PARTNERSHIP_ONBOARD_NOTIFY=true # notify both servers on successful onboard
```
### master_host1.conf (owner side)
### host1.conf (owner side)
```bash
# Containers whose WebUI URLs are redirected to owner's Tailscale IP on onboard.
@@ -179,7 +179,7 @@ HOST1_PARTNERSHIP_OWN_CONTAINERS=(
)
```
### master_host2.conf (mirror side)
### host2.conf (mirror side)
```bash
# Containers whose WebUI URLs are redirected on onboard.
@@ -342,9 +342,9 @@ Partnership/partnership_onboard.sh
### Adding a New Container to the Auth Stack
1. Create the XML template on HOST1 (`/boot/config/plugins/dockerMan/templates-user/my-NewContainer.xml`)
2. Add the XML filename to `HOST1_PARTNERSHIP_AUTH_STACK` in `master_host1.conf`
2. Add the XML filename to `HOST1_PARTNERSHIP_AUTH_STACK` in `host1.conf`
- If it has a database dependency, put the dep earlier in the array
3. Add the container name to `HOST2_PARTNERSHIP_REPLACE_CONTAINERS` in `master_host2.conf`
3. Add the container name to `HOST2_PARTNERSHIP_REPLACE_CONTAINERS` in `host2.conf`
4. Re-run the auth stack portion:
```bash
Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync
@@ -371,6 +371,29 @@ Counter resets after `SSH_STRIKE_RESET_HRS` of clean connectivity.
---
## ━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
**partnership_onboard.sh** — one-shot setup. Without `--log`, step headers, per-step
result lines (deployed/failed counts), and the final summary are visible. Per-container
deploy detail suppressed.
**partnership_offboard.sh** — one-shot teardown. Without `--log`, step headers, key
state transitions, and the final checklist summary are visible. Per-container cleanup
detail suppressed.
**partnership_manager.sh** — check/status/transfer/onboard modes. Without `--log`,
mode-specific result lines (`--check` prints ACTIVE/INACTIVE status always), state
transitions, warnings, and summary blocks are visible. Per-operation detail suppressed.
**ssh_setup.sh** — one-shot key setup. Without `--log`, section headers and per-step
results (key created, key installed, auth verified) are visible. Strike-counter
management (`--validate`) uses `log` for healthy cycles; `warn` for failures.
---
## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
### partnership_onboard.sh
@@ -491,7 +514,7 @@ automatically, but if HOST2 had no pre-existing auth stack of its own, it needs
docker ps
# Verify own parked containers came back up:
# (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in master_host2.conf)
# (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in host2.conf)
# If you need a fresh auth stack, deploy from HOST2's own XML templates:
docker create ... && docker start NginxProxyManager # etc.
+24 -5
View File
@@ -141,7 +141,7 @@
# TAILSCALE_API_KEY / TAILSCALE_TAILNET
# Required when PARTNERSHIP_REMOVE_TAILSCALE=true
#
# master_host*.conf
# host*.conf
#
# HOST*_PARTNERSHIP_AUTH_WEBUIS
# Containers reconfigured on onboard/offboard. Format: "ContainerName|WebUIPort"
@@ -253,7 +253,7 @@ OWNER="${!OWNER_ID}" # hostname string
MIRROR="${!MIRROR_ID}"
# SSH_KEY (set by detect_hosts) is this server's own private key.
# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
# With sparse checkout, each server only has its own master_host{N}.conf — the other server's
# With sparse checkout, each server only has its own host{N}.conf — the other server's
# key path is never available here. Use SSH_KEY for all outbound SSH regardless of mode.
MIRROR_SSH_KEY="$SSH_KEY"
OWNER_SSH_KEY="$SSH_KEY"
@@ -270,6 +270,21 @@ OWNER_STATE_FILE="/boot/config/partnership_${OWNER}.db"
MIRROR_STATE_FILE="/boot/config/partnership_${MIRROR}.db"
OFFLINE_COUNTER="/boot/config/partnership_offline_days.db"
# ── Exit Trap — restart locally stopped containers if script crashes mid-cleanup ──────────────
# Used by folderview3_remove_partner_folder() and cleanup_partner_containers() — also shared
# with partnership_offboard.sh which sources this file and registers the same trap.
declare -a _PM_TRAP_STOPPED=()
_pm_trap_restart_stopped() {
[[ ${#_PM_TRAP_STOPPED[@]} -eq 0 ]] && return
for c in "${_PM_TRAP_STOPPED[@]}"; do
[[ -z "$c" ]] && continue
if docker inspect "$c" >/dev/null 2>&1; then
warn "Exit trap: restarting $c (stopped but not removed)"
docker start "$c" >/dev/null 2>&1 || warn " Failed to restart $c"
fi
done
}
if [[ "${PARTNERSHIP_LIB_MODE:-}" != "1" ]]; then
if [[ -z "$MODE" ]]; then
error "No mode specified"
@@ -297,6 +312,8 @@ if [[ "${PARTNERSHIP_LIB_MODE:-}" != "1" ]]; then
# Lock for all modes except --check (frequent) and --offboard (offboard script holds its own)
[[ "$MODE" != "check" && "$MODE" != "offboard" ]] && acquire_lock "strict"
trap _pm_trap_restart_stopped EXIT
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
fi
@@ -679,6 +696,7 @@ folderview3_remove_partner_folder() {
if timeout "${DOCKER_TIMEOUT:-30}" docker inspect "$container" >/dev/null 2>&1; then
if timeout "${DOCKER_TIMEOUT:-30}" docker stop "$container" >/dev/null 2>&1; then
log "$container stopped ✅"
_PM_TRAP_STOPPED+=("$container")
(( stopped++ ))
else
warn "$container stop failed"
@@ -822,6 +840,7 @@ cleanup_partner_containers() {
fi
if timeout "${DOCKER_TIMEOUT:-30}" docker inspect "$container" >/dev/null 2>&1; then
timeout "${DOCKER_TIMEOUT:-30}" docker stop "$container" >/dev/null 2>&1 || true
_PM_TRAP_STOPPED+=("$container")
timeout "${DOCKER_TIMEOUT:-30}" docker rm "$container" >/dev/null 2>&1 && \
log "$container removed ✅" || warn "$container rm failed"
else
@@ -1337,7 +1356,7 @@ if [[ "$MODE" == "check" ]]; then
# Both agree and active — healthy, silent
if [[ "$LOCAL_STATE" == "$REMOTE_STATE" ]] && [[ "$LOCAL_STATE" == "ACTIVE" ]]; then
log "Partnership check — ACTIVE, both servers agree ✅"
echo "Partnership check — ACTIVE, both servers agree ✅"
exit 0
fi
@@ -1397,7 +1416,7 @@ if [[ "$MODE" == "check" ]]; then
# Both inactive — nothing to do
if [[ "$LOCAL_STATE" == "INACTIVE" ]] && [[ "$REMOTE_STATE" == "INACTIVE" ]]; then
log "Partnership check — INACTIVE on both servers"
echo "Partnership check — INACTIVE on both servers"
exit 0
fi
@@ -1489,7 +1508,7 @@ if [[ "$MODE" == "onboard" ]]; then
if [[ "$DRY_RUN" == false ]]; then
write_state_file "$LOCAL_STATE_FILE" "ACTIVE" "$NOW" "" "$LOCAL_SERVER_NAME" "onboard"
log "Local state: ACTIVE ✅"
echo "Local state: ACTIVE ✅"
remove_from_blocklist "$MIRROR"
push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"
echo "0" > "$OFFLINE_COUNTER"
+6 -3
View File
@@ -39,7 +39,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_PARTNERSHIP_AUTH_STACK
# Auth container XMLs to push during onboard — used on offboard to identify what
@@ -124,6 +124,8 @@ OFFLINE_COUNTER="/boot/config/partnership_offline_days.db"
acquire_lock "strict"
trap _pm_trap_restart_stopped EXIT
# Check already offboarded
if [[ -f "$LOCAL_STATE_FILE" ]]; then
CURRENT_STATE=$(read_state_file "$LOCAL_STATE_FILE" "state")
@@ -265,6 +267,7 @@ cleanup_deployed_stack_locally() {
--format '{{range .HostConfig.Binds}}{{println .}}{{end}}' \
"$cname" 2>/dev/null | awk -F: '{print $1}' | grep '^/mnt/.*/appdata')
timeout "${DOCKER_TIMEOUT:-30}" docker stop "$cname" >/dev/null 2>&1 || true
_PM_TRAP_STOPPED+=("$cname")
timeout "${DOCKER_TIMEOUT:-30}" docker rm "$cname" >/dev/null 2>&1 && \
log " $cname removed ✅" || warn " $cname rm failed"
else
@@ -428,7 +431,7 @@ if [[ "$AM_MIRROR" == true ]]; then
if [[ "$DRY_RUN" == false ]]; then
write_state_file "$LOCAL_STATE_FILE" \
"INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
log "Local state: INACTIVE ✅"
echo "Local state: INACTIVE ✅"
add_to_blocklist "$OWNER" "$REASON"
else
warn "DRY RUN — would write INACTIVE state and blocklist $OWNER"
@@ -626,7 +629,7 @@ fi
if [[ ${#PARTNERSHIP_MIRROR_BACKUPS[@]} -gt 0 ]]; then
echo ""
echo "━━━ $ICON_DISK Backup Handover ━━━"
log "Backups available for $MIRROR:"
echo "Backups available for $MIRROR:"
for path in "${PARTNERSHIP_MIRROR_BACKUPS[@]}"; do
[[ -z "$path" ]] && continue
echo " $path"
+12 -12
View File
@@ -45,7 +45,7 @@
#
# Dependency ordering in the auth stack is owner-enforced
# PARTNERSHIP_AUTH_STACK order matters: Mariadb and Redis must come before Authelia.
# The array is ordered correctly in master_host1.conf. After each Mariadb/Redis deploy,
# The array is ordered correctly in host1.conf. After each Mariadb/Redis deploy,
# the script waits for the container to be healthy before continuing. This is a remote
# health check — the container must be running (or report healthy) before the next
# dependent is deployed.
@@ -73,7 +73,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_PARTNERSHIP_AUTH_STACK
# XML filenames (from this server's templates-user/) to push and deploy on the
@@ -82,7 +82,7 @@
#
# HOST*_PARTNERSHIP_REPLACE_CONTAINERS
# Containers to stop on the mirror before deploying the auth stack.
# Defined in the MIRROR's own conf (master_host*.conf on HOST2) — never in HOST1's conf.
# Defined in the MIRROR's own conf (host*.conf on HOST2) — never in HOST1's conf.
# Read live from the mirror via SSH during Step 3 (sources mirror's load_config.sh at
# the same $SCRIPTS_ROOT path — convention: both servers use the same repo location).
# Leave empty on HOST2 if no conflicting containers exist (fresh mirror: nothing to stop).
@@ -171,7 +171,7 @@ OWNER="${!OWNER_ID}"
MIRROR="${!MIRROR_ID}"
# SSH_KEY (set by detect_hosts) is this server's own private key.
# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
# HOST{N}_SSH_KEY lives in master_host{N}.conf — with sparse checkout, the other server's
# HOST{N}_SSH_KEY lives in host{N}.conf — with sparse checkout, the other server's
# conf is never present here. Always use SSH_KEY (local private key) for outbound SSH.
MIRROR_SSH_KEY="$SSH_KEY"
@@ -359,7 +359,7 @@ wait_for_container_healthy() {
#
# SSHes to the mirror, sources its load_config.sh at the same $SCRIPTS_ROOT path (both servers
# use the same convention), and reads the named config array from the mirror's own conf.
# HOST2's container list stays in HOST2's master_host2.conf — not duplicated in HOST1's conf.
# HOST2's container list stays in HOST2's host2.conf — not duplicated in HOST1's conf.
# Fails gracefully if scripts aren't present yet or the array is empty (nothing to stop).
#
# deploy_container_from_xml() already stops/removes containers with the same name as what's
@@ -547,13 +547,13 @@ if [[ "$SKIP_AUTH_STACK" == true ]]; then
warn "Skipping (--skip-auth-stack)"
elif [[ ${#PARTNERSHIP_AUTH_STACK[@]} -eq 0 ]]; then
warn "PARTNERSHIP_AUTH_STACK not set in ${MY_ID} conf — skipping auth stack deploy"
warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to master_host${MY_ID: -1}.conf"
warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to host${MY_ID: -1}.conf"
STEP_AUTH_OK=false
else
deploy_xml_stack PARTNERSHIP_AUTH_STACK
AUTH_DEPLOYED=$_STACK_DEPLOYED
AUTH_FAILED=$_STACK_FAILED
log "Auth stack: $AUTH_DEPLOYED deployed, $AUTH_FAILED failed"
echo "Auth stack: $AUTH_DEPLOYED deployed, $AUTH_FAILED failed"
[[ "$AUTH_FAILED" -gt 0 ]] && STEP_AUTH_OK=false
fi
@@ -580,7 +580,7 @@ else
deploy_xml_stack PARTNERSHIP_ARR_STACK
ARR_DEPLOYED=$_STACK_DEPLOYED
ARR_FAILED=$_STACK_FAILED
log "Arr stack: $ARR_DEPLOYED deployed, $ARR_FAILED failed"
echo "Arr stack: $ARR_DEPLOYED deployed, $ARR_FAILED failed"
[[ "$ARR_FAILED" -gt 0 ]] && STEP_ARR_OK=false
fi
@@ -589,7 +589,7 @@ echo ""
echo "━━━ Step 7/8 — Partnership Onboard ━━━"
if bash "$SCRIPTS_ROOT/Partnership/partnership_manager.sh" --onboard "${EXTRA_FLAGS[@]}"; then
log "Partnership onboard complete ✅"
echo "Partnership onboard complete ✅"
ONBOARD_OK=true
else
error "Partnership onboard failed"
@@ -607,7 +607,7 @@ elif [[ "$SKIP_ARR_SYNC" == true ]]; then
elif [[ ! -f "$SCRIPTS_ROOT/Media/arr_sync.sh" ]]; then
warn "arr_sync.sh not found — run Media/arr_sync.sh manually once arrs are live"
elif bash "$SCRIPTS_ROOT/Media/arr_sync.sh" "${EXTRA_FLAGS[@]}"; then
log "Arr bootstrap complete ✅"
echo "Arr bootstrap complete ✅"
ARR_SYNC_OK=true
else
warn "Arr sync had errors — partnership still valid"
@@ -637,8 +637,8 @@ echo ""
if [[ "$ONBOARD_OK" == true ]]; then
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
log "$ICON_DONE DONE — partnership established ✅"
log "Verify with: Partnership/partnership_manager.sh --status"
echo "$ICON_DONE DONE — partnership established ✅"
echo "Verify with: Partnership/partnership_manager.sh --status"
else
error "Setup incomplete — resolve errors above and re-run"
fi
+9 -9
View File
@@ -7,7 +7,7 @@
# unRAID-Gmer4Lfe → gmer4lfe_rsync_automation
# unRAID-Jayred365 → jayred365_rsync_automation
# Idempotent — skips generation if key already exists (use --force to regenerate).
# Updates master_host*.conf with key path on success.
# Updates host*.conf with key path on success.
#
# ── MODES ─────────────────────────────────────────────────────────────────────────────────────
# (default) — generate key if missing, copy to remote, update conf
@@ -85,7 +85,7 @@ SSH_PUB_PATH="${SSH_KEY_PATH}.pub"
# ── Host conf path ────────────────────────────────────────────────────────────────────────────
HOST_NUM="${MY_ID#HOST}" # "1" or "2"
HOST_CONF="$SCRIPTS_ROOT/master_host${HOST_NUM}.conf"
HOST_CONF="$SCRIPTS_ROOT/host${HOST_NUM}.conf"
KEY_CONF_VAR="${MY_ID}_SSH_KEY"
# ── Strike state file ─────────────────────────────────────────────────────────────────────────
@@ -290,8 +290,8 @@ echo "━━━ $ICON_GEAR Key Generation ━━━"
if [[ -f "$SSH_KEY_PATH" ]] && [[ "$FORCE" == false ]]; then
local_fp=$(ssh-keygen -lf "$SSH_KEY_PATH" 2>/dev/null || echo "unreadable")
log "Key already exists — skipping generation (--force to regenerate)"
log " $local_fp"
echo "Key already exists — skipping generation (--force to regenerate)"
echo " $local_fp"
else
if [[ "$FORCE" == true ]] && [[ -f "$SSH_KEY_PATH" ]]; then
warn "Regenerating key (--force) — existing key will be replaced"
@@ -316,12 +316,12 @@ fi
# ── Update conf ───────────────────────────────────────────────────────────────────────────────
echo ""
echo "━━━ $ICON_GEAR Update master_host${HOST_NUM}.conf ━━━"
echo "━━━ $ICON_GEAR Update host${HOST_NUM}.conf ━━━"
if [[ "$DRY_RUN" == false ]]; then
update_conf_key_path
else
warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in master_host${HOST_NUM}.conf"
warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in host${HOST_NUM}.conf"
fi
# ── Copy to remote ────────────────────────────────────────────────────────────────────────────
@@ -339,7 +339,7 @@ else
if ssh-copy-id -i "$SSH_PUB_PATH" -o ConnectTimeout="${SSH_TIMEOUT:-15}" \
root@"$REMOTE_SERVER" 2>/dev/null; then
log "Public key installed on $REMOTE_SERVER_NAME"
echo "Public key installed on $REMOTE_SERVER_NAME"
else
error "ssh-copy-id failed — check that:"
error " 1. Remote server is reachable: tailscale status"
@@ -355,7 +355,7 @@ echo "━━━ $ICON_VERIFY Verify SSH Auth ━━━"
if [[ "$DRY_RUN" == false ]]; then
if test_ssh_auth "$REMOTE_SERVER"; then
log "SSH auth to $REMOTE_SERVER_NAME working ✅"
echo "SSH auth to $REMOTE_SERVER_NAME working ✅"
# Reset any existing strikes
if [[ -f "$SSH_STRIKE_FILE" ]]; then
write_strike_file 0 "" "$(date '+%Y-%m-%d %H:%M:%S')"
@@ -373,7 +373,7 @@ echo ""
echo "━━━━━ $ICON_SUMMARY SSH SETUP SUMMARY ━━━━━"
echo " Key: $SSH_KEY_PATH"
echo " Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)"
echo " Conf: ${KEY_CONF_VAR} in master_host${HOST_NUM}.conf"
echo " Conf: ${KEY_CONF_VAR} in host${HOST_NUM}.conf"
echo ""
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
warn "$ICON_DONE DONE — SSH key ready for rsync automation ✅"
+2 -2
View File
@@ -893,7 +893,7 @@ Background: NO
```bash
#!/bin/bash
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --status
/mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --status
```
**What it shows:**
@@ -1275,7 +1275,7 @@ Background: NO
```bash
#!/bin/bash
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --status
/mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --status
```
**What it shows:**
+24 -12
View File
@@ -448,8 +448,8 @@ Failover coverage:
serves Gmer4Lfe.us via DDNS
Monitoring:
Both servers system_watchdog.sh system_watchdog.sh
docker_watchdog.sh docker_watchdog.sh
Both servers resource/docker/storage/ resource/docker/storage/
system_watchdog.sh system_watchdog.sh
failover.sh failover.sh
Sunday morning coffee report Sunday morning coffee report
```
@@ -462,8 +462,8 @@ Monitoring:
Unraid_Scripts/
├── master.conf ← All shared configuration — the only file you edit regularly
├── master_host1.conf ← HOST1-specific: share lists, container names, API keys
├── master_host2.conf ← HOST2-specific: same structure, different values
├── host1.conf ← HOST1-specific: share lists, container names, API keys
├── host2.conf ← HOST2-specific: same structure, different values
├── common.sh ← Shared library — all functions used by every script
├── load_config.sh ← Sources all conf files and common.sh
@@ -476,10 +476,13 @@ Unraid_Scripts/
├── Failover/ ← Mutual automatic failover — continuous background process
│ README: README-Failover.md
├── Docker_Essentials/ ← Container lifecycle: watchdog, restarts, networks
├── Watchdogs/ ← All watchdog scripts: resource, docker, storage, system
│ README: README-Watchdogs.md
├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks
│ README: README-Docker_Essentials.md
├── unRAID_Essentials/ ← Server-level: system watchdog, WebGUI, log hygiene, tuning
├── unRAID_Essentials/ ← Server-level: WebGUI, log hygiene, kernel tuning
│ README: README-Unraid_Essentials.md
├── Media/ ← Library health + behavior-driven discovery: permissions, junk cleanup, orphan removal, weekly arr adds
@@ -509,11 +512,11 @@ Unraid_Scripts/
# definitions, watchdog settings, arr cleanup config, DDNS timing, etc.
# Pushed to both servers via git. Never contains server-specific values.
#
# master_host1.conf — sourced only on HOST1
# host1.conf — sourced only on HOST1
# HOST1_* prefixed variables: share lists, container names, API keys,
# ramdisk size, specific paths, per-server toggle overrides.
#
# master_host2.conf — sourced only on HOST2
# host2.conf — sourced only on HOST2
# HOST2_* prefixed variables: same structure, different values.
#
# detect_hosts() in common.sh:
@@ -539,10 +542,15 @@ At Startup of Array:
→ php_fpm_max_children.sh WebGUI tuning before first request
→ ramdisk_setup.sh create ramdisk before Emby starts
→ docker_network_connect.sh connect containers to extra networks
→ system_watchdog.sh continuous — last resort server stability
→ docker_watchdog.sh continuous — two-tier container healing
→ failover.sh continuous — mutual failover state machine
Every minute:
watchdog_orchestrator.sh fires each watchdog in sequence
→ resource_watchdog.sh reduce pressure before healing attempts
→ docker_watchdog.sh two-tier container healing
→ storage_watchdog.sh pool growth + runaway log detection
→ system_watchdog.sh last resort — reboots when all else fails
Every 3 minutes:
transcode_management.sh cleanup → manager (order non-negotiable)
@@ -582,9 +590,13 @@ Sunday morning block (611am):
What actually happens on a typical day, from the ecosystem's perspective:
```
Throughout the day:
Throughout the day (every minute via watchdog_orchestrator.sh):
resource_watchdog.sh managing: system pressure (throttle/pause/stop)
docker_watchdog.sh healing: memory leaks, HTTP failures, required containers
storage_watchdog.sh watching: appdata growth rate, runaway log files
system_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
docker_watchdog.sh watching: memory, CPU, HTTP health, required containers
Throughout the day:
failover.sh watching: remote server, internet connectivity
transcode_management.sh managing: ramdisk ↔ SSD, session cleanup (every 3min)
critical_sync_maintenance.sh keeping: auth stack + Emby current (every 15min)
+23 -10
View File
@@ -130,7 +130,7 @@ access key (for script repository pulls).
### 3a — Rsync Automation Keys (ssh_setup.sh)
Run on **each server**. `ssh_setup.sh` generates the key, copies it to the remote,
and updates `master_host*.conf` with the key path automatically.
and updates `host*.conf` with the key path automatically.
```bash
# On HOST1 — after the repo is cloned:
@@ -146,7 +146,7 @@ unRAID-Gmer4Lfe → /root/.ssh/gmer4lfe_rsync_automation
unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation
```
`master_host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the
`host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the
generated key. Run `--status` to verify:
```bash
@@ -188,8 +188,8 @@ find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \;
Expected structure after clone:
```
master.conf ← all shared configuration
master_host1.conf ← HOST1-specific configuration
master_host2.conf ← HOST2-specific configuration
host1.conf ← HOST1-specific configuration
host2.conf ← HOST2-specific configuration
common.sh ← shared library
load_config.sh ← config loader
Orchestrators/
@@ -232,7 +232,7 @@ SSH_PORT=221
### Daily Sync Shares
```bash
# master_host1.conf
# host1.conf
# Shares this server downloads to — pushed to the other server nightly.
# arr_sync.sh runs before rsync in the daily window, so the receiving
# server's arrs already track incoming files when they arrive.
@@ -247,7 +247,7 @@ HOST1_DAILY_SYNC_SHARES=(
"/mnt/user/stand-up_comedy"
)
# master_host2.conf
# host2.conf
HOST2_DAILY_SYNC_SHARES=(
"/mnt/user/Anime_Shows"
"/mnt/user/Anime_Movies"
@@ -274,8 +274,8 @@ to their unprefixed names. Scripts only ever reference the unprefixed name — t
work identically on both servers.
```bash
nano /mnt/user/appdata/unraid_scripts/master_host1.conf # on HOST1
nano /mnt/user/appdata/unraid_scripts/master_host2.conf # on HOST2
nano /mnt/user/appdata/unraid_scripts/host1.conf # on HOST1
nano /mnt/user/appdata/unraid_scripts/host2.conf # on HOST2
```
Every variable is documented in the conf files. Key values to set:
@@ -413,7 +413,7 @@ zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase
zfs mount poolname/Gmer4Lfe-Personal
```
### Add to master_host1.conf
### Add to host1.conf
```bash
HOST1_PERSONAL_SHARES=(
@@ -662,7 +662,7 @@ declare -A PROFILE_EXCLUDE_DIRS
declare -A PROFILE_REMOTE_RESTART_CONTAINERS
```
### master_host*.conf
### host*.conf
```bash
# Daily sync shares — one list per server (mutually exclusive)
@@ -680,6 +680,19 @@ HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"
---
## ━━━ OUTPUT TIERS ━━━
Without `--log`, each run shows: section headers (Pre-flight, Transfer, Stop/Start
Containers), the transfer identity block (source, remote, profile, identity), a
progress indicator while rsync is running, and a summary block with duration,
bytes transferred, and status. Warnings and errors are always visible.
With `--log`, every decision is shown — pre-flight check results, profile resolution,
per-container stop/start state, retry details, and bandwidth log confirmation.
Use for first-run validation or when investigating unexpected behavior.
---
## ━━━ FLAG REFERENCE ━━━
All flags work on `rsync.sh` and all orchestrators.
+1 -1
View File
@@ -435,7 +435,7 @@ echo "$ICON_TIME Duration: $(format_duration $DURATION)"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ "$RSYNC_SUCCESS" == true ]]; then
log "$ICON_DONE Status: $ICON_SUCCESS DONE"
echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
else
echo "$ICON_ERROR Status: FAILED after $RETRY_COUNT attempts"
notify "Rsync FAILED — $DIRECTORY ($PROFILE_NAME) after $RETRY_COUNT attempts on $(hostname)" \
+87 -2
View File
@@ -20,10 +20,28 @@ making any changes.
- [recreate_shares.sh](#recreate_sharessh)
- [continuous_scripts_status.sh](#continuous_scripts_statussh)
- [claude_startup.sh](#claude_startupsh)
- [ramdisk_stop.sh](#ramdisk_stopsh)
- [Adding a New Tool](#adding-a-new-tool)
---
## Output Tiers
All tools have two output levels controlled by `--log`.
Without `--log`, each script processes and always concludes with a summary block
showing identity, duration, counts, and a status line. Warnings and errors are
always visible. State-display scripts (watchdog_skip_list_manager, zfs_pool_scrub
`--status`, fallback_state_reset current-state section) always show their state
output — `--log` adds configuration detail and per-item resolution within each
section.
With `--log`, per-item detail appears: individual items added/skipped, per-database
check results, per-pool scan lines, per-container stop/start state, per-directory
creation results. Use when debugging unexpected results or confirming a first run.
---
## emby_to_lidarr_sync.sh
One-shot bootstrap tool. Scans Emby play history, finds artists you've actually
@@ -415,7 +433,7 @@ library.db-wal — write-ahead log (uncommitted transactions)
the main library.db is also affected
```
### Configuration (master_host*.conf)
### Configuration (host*.conf)
```bash
HOST1_EMBY_CONTAINER="Emby" # aliased by detect_hosts() → EMBY_CONTAINER
@@ -453,7 +471,7 @@ without triggering any error — until you try to read that specific file. By th
Run monthly. Also run after any disk replacement or power event.
Safe to run while the system is in use — scrub runs at low I/O priority.
### Configuration (master_host*.conf)
### Configuration (host*.conf)
```bash
HOST1_ZFS_REPORT_IGNORE_POOLS=(
@@ -614,6 +632,73 @@ claude_startup.sh --setup # set up symlinks only — no launch (for array_star
---
## ramdisk_stop.sh
Safely stops the transcode ramdisk: redirects the transcode symlink to the SSD
fallback first (so Emby continues writing without interruption), then unmounts the
tmpfs and updates the state file. Primary use case is stopping the current ramdisk
before re-running `ramdisk_setup.sh` with new size or threshold values.
### When to Use
```
Bumping RAMDISK_SIZE — setup script is idempotent, skips remount if already mounted
→ stop first, then re-run ramdisk_setup.sh with new HOST*_RAMDISK_SIZE value
Adjusting RAMDISK_WARN_GB / RAMDISK_LOW_GB thresholds
→ no need to stop for threshold changes (transcode_manager reads vars live)
→ only needed if you're also changing the size
Temporarily freeing ramdisk RAM — reclaim tmpfs back to general memory pool
→ stop, restart later with ramdisk_setup.sh
```
### Stop Sequence
```
1. Redirect symlink: TRANSCODE_LINK → TRANSCODE_SSD
Emby immediately writes to SSD — no broken-path window during unmount
2. Check for active transcode files on ramdisk (warn, don't block)
Files in progress on the ramdisk are lost on unmount — expected for maintenance
3. Unmount ramdisk
Regular umount first; if busy (directory handles only, no active writes)
falls back to lazy unmount automatically
4. Update /tmp/transcode_state.db → current_target=TRANSCODE_SSD
transcode_manager.sh reads this on its next cycle
```
### transcode_manager Warning
If `transcode_manager.sh` is running, it may flip the symlink back to the ramdisk
on its next cycle (once the ramdisk is unmounted, that flip will fail). Stop
`transcode_manager.sh` first if you need the SSD redirect to hold before remounting.
### After Stopping
```bash
# Update host*.conf with new size values:
# HOST1_RAMDISK_SIZE="10G"
# HOST1_RAMDISK_WARN_GB=8.5
# HOST1_RAMDISK_LOW_GB=7
# Remount at new size:
bash Transcodes/ramdisk_setup.sh
```
### Usage
```bash
ramdisk_stop.sh --status # show mount state, symlink, active files — always check first
ramdisk_stop.sh --dry-run # show what would happen without making changes
ramdisk_stop.sh # stop the ramdisk
ramdisk_stop.sh --log # verbose — show each step
```
---
## Adding a New Tool
Write the tool when you solve a problem manually with bash commands. You'll face it again.
+3 -1
View File
@@ -76,7 +76,7 @@ something new, write the tool. Store it here. Find it at 2am next time.
`bulk_permissions_repair.sh`, `zfs_pool_scrub.sh`
**Lifecycle Tools** — Backup, setup, and migration support
`container_data_export.sh`, `recreate_shares.sh`, `claude_startup.sh`
`container_data_export.sh`, `recreate_shares.sh`, `claude_startup.sh`, `ramdisk_stop.sh`
**Library Sync Bootstrap** — Close the gap between Emby and arr libraries
`emby_to_lidarr_sync.sh`, `emby_to_sonarr_sync.sh`, `emby_to_radarr_sync.sh`
@@ -125,6 +125,7 @@ The relationship is one-way: Tools act on state that other scripts have written.
| `recreate_shares.sh` | Share directories missing after fresh install or disk rebuild | After fresh unRAID install or disk replacement on HOST2 |
| `continuous_scripts_status.sh` | Need a live view of watchdog and fallback state | Any time — manual dashboard, no schedule |
| `claude_startup.sh` | Claude Code session setup after reboot — symlinks persistent storage | After each unRAID reboot, or called by array_started.sh |
| `ramdisk_stop.sh` | Safely stop the transcode ramdisk — redirect symlink to SSD, unmount, update state | Before re-running ramdisk_setup.sh with new size or thresholds |
| `emby_to_lidarr_sync.sh` | Add all Emby album artists not yet tracked in Lidarr | After Lidarr setup, database wipe, or when you suspect gaps |
| `emby_to_sonarr_sync.sh` | Add all Emby TV series not yet tracked in Sonarr | After Sonarr setup, database wipe, or when you suspect gaps |
| `emby_to_radarr_sync.sh` | Add all Emby movies not yet tracked in Radarr | After Radarr setup, database wipe, or when you suspect gaps |
@@ -151,6 +152,7 @@ Situation arises
│ recreate_shares ◄── fresh HOST2 setup or disk rebuild │
│ continuous_scripts_status ◄── manual status check at any time │
│ claude_startup ◄── after each unRAID reboot │
│ ramdisk_stop ◄── before ramdisk resize / remount │
│ │
│ emby_to_lidarr_sync ◄── Lidarr setup / database wipe / gap │
│ emby_to_sonarr_sync ◄── Sonarr setup / database wipe / gap │
+1 -1
View File
@@ -304,7 +304,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ "$ARCHIVE_VERIFIED" == true && "$RESTART_OK" == true ]]; then
log "$ICON_DONE Status: done — $ARCHIVE_NAME"
echo "$ICON_DONE Status: done — $ARCHIVE_NAME"
elif [[ "$ARCHIVE_VERIFIED" == false ]]; then
echo "$ICON_ERROR Status: archive verification FAILED — check backup before relying on it"
else
+5 -5
View File
@@ -55,7 +55,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_EMBY_CONTAINER
# Name of the Emby Docker container on this host.
@@ -122,7 +122,7 @@ fi
detect_hosts
if [[ -z "${EMBY_CONTAINER:-}" ]]; then
error "EMBY_CONTAINER not set for $MY_ID — check HOST*_EMBY_CONTAINER in master_host*.conf"
error "EMBY_CONTAINER not set for $MY_ID — check HOST*_EMBY_CONTAINER in host*.conf"
exit 1
fi
@@ -135,7 +135,7 @@ EMBY_CONFIG_HOST=$(timeout "$DOCKER_TIMEOUT" docker inspect "$EMBY_CONTAINER" 2>
if [[ -z "$EMBY_CONFIG_HOST" ]]; then
error "Could not detect Emby config path from Docker mounts"
error "Is $EMBY_CONTAINER the correct container name? Check HOST*_EMBY_CONTAINER in master_host*.conf"
error "Is $EMBY_CONTAINER the correct container name? Check HOST*_EMBY_CONTAINER in host*.conf"
exit 1
fi
@@ -352,7 +352,7 @@ echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
echo " $ICON_SUCCESS Passed: ${#PASS_DBS[@]}"
[[ ${#FAIL_DBS[@]} -gt 0 ]] && echo " $ICON_ERROR Failed: ${#FAIL_DBS[@]}"
[[ ${#MISSING_DBS[@]} -gt 0 ]] && log "Skipped: ${#MISSING_DBS[@]} (not found)"
[[ ${#MISSING_DBS[@]} -gt 0 ]] && echo " Skipped: ${#MISSING_DBS[@]} (not found)"
echo ""
[[ ${#PASS_DBS[@]} -gt 0 ]] && for db in "${PASS_DBS[@]}"; do log " $ICON_SUCCESS $db"; done
@@ -378,7 +378,7 @@ elif [[ ${#FAIL_DBS[@]} -gt 0 ]]; then
notify "Emby database CORRUPTION on $(hostname) — failed: ${FAIL_DBS[*]} — manual intervention needed" \
"Emby DB Repair" "warning"
else
log "$ICON_DONE Status: all ${#PASS_DBS[@]} databases healthy ✅"
echo "$ICON_DONE Status: all ${#PASS_DBS[@]} databases healthy ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+5 -5
View File
@@ -36,7 +36,7 @@
# modification. Safe to run multiple times — the second run finds nothing to add.
#
# ==============================================================================================
# CONFIGURATION (master_host*.conf)
# CONFIGURATION (host*.conf)
# ==============================================================================================
#
# HOST*_LIDARR_URL / HOST*_LIDARR_API_KEY — Lidarr connection (aliased by detect_hosts)
@@ -73,12 +73,12 @@ if ! command -v jq >/dev/null 2>&1; then
fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then
error "LIDARR_URL / LIDARR_API_KEY not configured — check master_host*.conf"
error "LIDARR_URL / LIDARR_API_KEY not configured — check host*.conf"
exit 1
fi
@@ -224,13 +224,13 @@ IFS=$'\n' MISSING=($(printf '%s\n' "${MISSING[@]}" | sort))
echo " Already tracked: $ALREADY | Missing from Lidarr: ${#MISSING[@]}"
if [[ "${#MISSING[@]}" -eq 0 ]]; then
log "Lidarr already tracks everything played on Emby"
echo "Lidarr already tracks everything played on Emby"
exit 0
fi
echo ""
echo "━━━ $ICON_SUMMARY Artists to add (${#MISSING[@]}) ━━━"
for a in "${MISSING[@]}"; do echo " $a"; done
for a in "${MISSING[@]}"; do log " $a"; done
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN complete — run without --dry-run to add these artists"
+5 -5
View File
@@ -38,7 +38,7 @@
# without modification. Safe to run multiple times — the second run finds nothing to add.
#
# ==============================================================================================
# CONFIGURATION (master_host*.conf)
# CONFIGURATION (host*.conf)
# ==============================================================================================
#
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY — Radarr connection (aliased by detect_hosts)
@@ -74,12 +74,12 @@ if ! command -v jq >/dev/null 2>&1; then
fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${RADARR_URL:-}" || -z "${RADARR_API_KEY:-}" ]]; then
error "RADARR_URL / RADARR_API_KEY not configured — check master_host*.conf"
error "RADARR_URL / RADARR_API_KEY not configured — check host*.conf"
exit 1
fi
@@ -229,7 +229,7 @@ IFS=$'\n' SORTED_MISSING=($(printf '%s\n' "${!MISSING[@]}" | sort))
echo " Already tracked: $ALREADY | Missing from Radarr: ${#MISSING[@]}"
if [[ "${#MISSING[@]}" -eq 0 ]]; then
log "Radarr already tracks everything in Emby"
echo "Radarr already tracks everything in Emby"
exit 0
fi
@@ -237,7 +237,7 @@ echo ""
echo "━━━ $ICON_SUMMARY Movies to add (${#MISSING[@]}) ━━━"
for name in "${SORTED_MISSING[@]}"; do
_tmdb="${MISSING[$name]}"
echo " $name${_tmdb:+ (TMDB: $_tmdb)}"
log " $name${_tmdb:+ (TMDB: $_tmdb)}"
done
if [[ "$DRY_RUN" == true ]]; then
+5 -5
View File
@@ -38,7 +38,7 @@
# without modification. Safe to run multiple times — the second run finds nothing to add.
#
# ==============================================================================================
# CONFIGURATION (master.conf / master_host*.conf)
# CONFIGURATION (master.conf / host*.conf)
# ==============================================================================================
#
# SONARR_EMBY_LIBRARIES — Emby library names to scan (master.conf)
@@ -75,12 +75,12 @@ if ! command -v jq >/dev/null 2>&1; then
fi
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
exit 1
fi
if [[ -z "${SONARR_URL:-}" || -z "${SONARR_API_KEY:-}" ]]; then
error "SONARR_URL / SONARR_API_KEY not configured — check master_host*.conf"
error "SONARR_URL / SONARR_API_KEY not configured — check host*.conf"
exit 1
fi
@@ -239,7 +239,7 @@ IFS=$'\n' SORTED_MISSING=($(printf '%s\n' "${!MISSING[@]}" | sort))
echo " Already tracked: $ALREADY | Missing from Sonarr: ${#MISSING[@]}"
if [[ "${#MISSING[@]}" -eq 0 ]]; then
log "Sonarr already tracks everything in Emby"
echo "Sonarr already tracks everything in Emby"
exit 0
fi
@@ -247,7 +247,7 @@ echo ""
echo "━━━ $ICON_SUMMARY Series to add (${#MISSING[@]}) ━━━"
for name in "${SORTED_MISSING[@]}"; do
_tvdb="${MISSING[$name]}"
echo " $name${_tvdb:+ (TVDB: $_tvdb)}"
log " $name${_tvdb:+ (TVDB: $_tvdb)}"
done
if [[ "$DRY_RUN" == true ]]; then
+3 -3
View File
@@ -151,7 +151,7 @@ if pgrep -f "fallback.sh" >/dev/null 2>&1; then
echo ""
warn "If you are sure you want to proceed anyway, confirm below"
else
log "fallback.sh is not running — safe to reset ✅"
echo "fallback.sh is not running — safe to reset ✅"
fi
# Check current state — if already NORMAL warn user
@@ -238,8 +238,8 @@ if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
else
warn "$ICON_DONE State reset to NORMAL"
log "fallback.sh will resume from NORMAL on next cycle"
log "No containers were started or stopped"
echo "fallback.sh will resume from NORMAL on next cycle"
echo "No containers were started or stopped"
echo ""
[[ "$FALLBACK_RUNNING" == true ]] && \
warn "⚠️ fallback.sh was running during reset — monitor next cycle carefully"
+339
View File
@@ -0,0 +1,339 @@
#!/bin/bash
# ==============================================================================================
# ================================= Ramdisk Stop ===============================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Safely stops the transcode ramdisk: redirects the transcode symlink to the
# SSD fallback before unmounting so Emby continues writing without interruption,
# then unmounts the tmpfs and updates the state file.
#
# Primary use case: stopping the current ramdisk before re-running
# ramdisk_setup.sh with new size or threshold values (setup is idempotent —
# if the ramdisk is mounted, it skips the mount and reports status, so you
# must stop it first to change the size).
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Executes in safe order:
# 1. Validate — ramdisk mounted, SSD fallback exists
# 2. Redirect symlink → SSD (Emby immediately writes to SSD instead)
# 3. Warn if active transcode files still on ramdisk (informational — not a blocker)
# 4. Unmount ramdisk tmpfs
# 5. Update /tmp/transcode_state.db → current_target=TRANSCODE_SSD
#
# The symlink redirect happens before unmount so there is no window where Emby
# has nowhere to write. Existing in-progress transcode files on the ramdisk are
# lost on unmount — warn the user but proceed (this is expected for maintenance).
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Root Required
# umount requires root.
#
# Single Instance Lock
# acquire_lock prevents concurrent stop attempts.
#
# Mounted Check
# Exits cleanly if ramdisk is not mounted — nothing to do.
#
# Symlink-First Order
# Symlink is redirected before unmount — Emby never sees a broken path.
#
# transcode_manager Warning
# Warns if transcode_manager.sh is running — it may flip the symlink back
# to ramdisk on its next cycle. Stop transcode_manager before running this
# if you need the SSD redirect to hold.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# host*.conf
# HOST*_TRANSCODE_SSD SSD fallback directory — redirect target during stop.
# Aliased by detect_hosts() → TRANSCODE_SSD.
#
# master.conf
# TRANSCODE_LINK Symlink Emby uses. Must match Emby's transcode path setting.
# RAMDISK_PATH tmpfs mount point.
#
# ==============================================================================================
# STATE FILES
# ==============================================================================================
#
# /tmp/transcode_state.db — updated to current_target=TRANSCODE_SSD after stop.
# transcode_manager.sh reads this on its next cycle.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# ramdisk_stop.sh
# Stop the ramdisk: redirect symlink → SSD, unmount, update state.
#
# ramdisk_stop.sh --dry-run
# Show what would happen without making any changes.
#
# ramdisk_stop.sh --status
# Show current mount state, symlink target, active files on ramdisk. Exit.
#
# ramdisk_stop.sh --log
# Verbose output for each step.
#
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root — umount requires root"
exit 1
fi
validate_unraid_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
acquire_lock
detect_hosts
STATE_FILE="/tmp/transcode_state.db"
log "Identity: $MY_ID ($LOCAL_SERVER_NAME)"
log "Ramdisk: $RAMDISK_PATH"
log "Fallback: $TRANSCODE_SSD"
log "Symlink: $TRANSCODE_LINK"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
# ==============================================================================================
# ━━━ Status ━━━
# ==============================================================================================
if [[ "$SHOW_STATUS" == true ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_RAM Ramdisk path: $RAMDISK_PATH"
echo "$ICON_DISK SSD fallback: $TRANSCODE_SSD"
echo "$ICON_LINK Symlink: $TRANSCODE_LINK"
echo ""
if mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
USAGE=$(df -BG "$RAMDISK_PATH" | awk 'NR==2 {print $3}')
AVAIL=$(df -BG "$RAMDISK_PATH" | awk 'NR==2 {print $4}')
echo " $ICON_RAM Ramdisk: mounted — $USAGE used / $AVAIL available ✅"
FILE_COUNT=$(find "$RAMDISK_PATH" -type f 2>/dev/null | wc -l)
echo " $ICON_RAM Active files on ramdisk: $FILE_COUNT"
else
echo " $ICON_RAM Ramdisk: NOT mounted"
fi
if [[ -L "$TRANSCODE_LINK" ]]; then
TARGET=$(readlink "$TRANSCODE_LINK")
echo " $ICON_LINK Symlink: $TRANSCODE_LINK$TARGET"
else
echo " $ICON_LINK Symlink: not set"
fi
if [[ -f "$STATE_FILE" ]]; then
echo ""
echo " State file ($STATE_FILE):"
while IFS='=' read -r key value; do
[[ -z "$key" ]] && continue
echo " $key = $value"
done < "$STATE_FILE"
else
echo " $ICON_INFO State file: not found (ramdisk never started this boot)"
fi
if pgrep -f "transcode_manager.sh" >/dev/null 2>&1; then
echo ""
warn "transcode_manager.sh is currently RUNNING"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
# ==============================================================================================
# ━━━ Preflight ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Preflight ━━━"
# Bail if not mounted — nothing to do
if ! mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
warn "Ramdisk is not mounted at $RAMDISK_PATH — nothing to stop"
exit 0
fi
log "Ramdisk is mounted ✅"
# Warn if transcode_manager is running — it may flip symlink back on next cycle
if pgrep -f "transcode_manager.sh" >/dev/null 2>&1; then
warn "transcode_manager.sh is currently RUNNING"
warn "It may flip the symlink back to ramdisk on its next cycle"
warn "Stop transcode_manager.sh first if you need the SSD redirect to hold"
echo ""
fi
# Confirm SSD fallback exists
if [[ ! -d "$TRANSCODE_SSD" ]]; then
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — SSD fallback does not exist: $TRANSCODE_SSD"
warn "DRY RUN — would create it before redirecting symlink"
else
warn "SSD fallback does not exist — creating: $TRANSCODE_SSD"
mkdir -p "$TRANSCODE_SSD" || {
error "Failed to create SSD fallback: $TRANSCODE_SSD"
error "Cannot safely redirect symlink — aborting"
exit 1
}
log "SSD fallback created ✅"
fi
else
log "SSD fallback exists: $TRANSCODE_SSD"
fi
START=$(date +%s)
STOP_SUCCESS=true
# ==============================================================================================
# ━━━ Redirect Symlink → SSD ━━━
# ==============================================================================================
# Redirect BEFORE unmount — Emby continues writing to SSD with no broken path window.
echo ""
echo "━━━ $ICON_LINK Redirect Symlink → SSD ━━━"
if [[ -L "$TRANSCODE_LINK" ]]; then
CURRENT_TARGET=$(readlink "$TRANSCODE_LINK")
if [[ "$CURRENT_TARGET" == "$TRANSCODE_SSD" ]]; then
log "Symlink already points to SSD — no change needed"
else
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would redirect: $TRANSCODE_LINK$TRANSCODE_SSD"
else
ln -sfn "$TRANSCODE_SSD" "$TRANSCODE_LINK" && \
warn "Symlink redirected: $TRANSCODE_LINK$TRANSCODE_SSD" || {
error "Failed to redirect symlink"
STOP_SUCCESS=false
}
fi
fi
elif [[ -e "$TRANSCODE_LINK" ]]; then
warn "$TRANSCODE_LINK exists but is not a symlink — leaving as-is"
else
warn "Symlink $TRANSCODE_LINK does not exist — nothing to redirect"
fi
# ==============================================================================================
# ━━━ Active Files Warning ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_RAM Active Files Check ━━━"
FILE_COUNT=$(find "$RAMDISK_PATH" -type f 2>/dev/null | wc -l)
if [[ "$FILE_COUNT" -gt 0 ]]; then
warn "⚠️ $FILE_COUNT file(s) still on ramdisk — will be lost on unmount"
warn "Active transcode sessions should be stopped before unmounting"
warn "Proceeding regardless (this is expected for maintenance)"
if [[ "$LOG" == true ]]; then
find "$RAMDISK_PATH" -type f 2>/dev/null | while read -r f; do
log " $f"
done
fi
else
log "No active files on ramdisk ✅"
fi
# ==============================================================================================
# ━━━ Unmount ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_RAM Unmount Ramdisk ━━━"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would unmount: $RAMDISK_PATH"
else
if umount "$RAMDISK_PATH" 2>/dev/null; then
warn "Ramdisk unmounted: $RAMDISK_PATH"
else
# Regular unmount failed — check if only directory handles are open (no active writes)
OPEN_FILES=$(lsof +D "$RAMDISK_PATH" 2>/dev/null | awk 'NR>1 && $5 != "DIR"' | wc -l)
if [[ "$OPEN_FILES" -eq 0 ]]; then
warn "Busy — only directory handles open, no active writes — trying lazy unmount"
if umount -l "$RAMDISK_PATH"; then
warn "Ramdisk lazy-unmounted: $RAMDISK_PATH"
warn "Handles will release when owning processes next check the directory"
else
error "Lazy unmount also failed — $RAMDISK_PATH"
STOP_SUCCESS=false
fi
else
error "Failed to unmount $RAMDISK_PATH$OPEN_FILES file(s) still open for writing"
error "Stop active transcode sessions and retry"
STOP_SUCCESS=false
fi
fi
fi
# ==============================================================================================
# ━━━ Update State File ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR State File ━━━"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would update $STATE_FILE: current_target=$TRANSCODE_SSD"
elif [[ "$STOP_SUCCESS" == true ]]; then
NOW=$(date +%s)
cat > "$STATE_FILE" <<EOF
current_target=$TRANSCODE_SSD
last_flip_time=$NOW
flip_count_hour=0
flip_hour_start=$NOW
EOF
log "State file updated: current_target=$TRANSCODE_SSD"
else
warn "Skipping state file update — stop had errors"
fi
END=$(date +%s)
# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
echo ""
echo "━━━━━ $ICON_SUMMARY RAMDISK STOP SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_RAM Ramdisk: $RAMDISK_PATH"
echo "$ICON_DISK Fallback: $TRANSCODE_SSD"
echo "$ICON_LINK Symlink: $TRANSCODE_LINK$(readlink "$TRANSCODE_LINK" 2>/dev/null || echo "not set")"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ "$STOP_SUCCESS" == true ]]; then
echo "$ICON_DONE Status: done ✅"
echo "Run ramdisk_setup.sh to remount with new configuration"
else
echo "$ICON_ERROR Status: STOP HAD ERRORS"
notify "Ramdisk stop errors on $(hostname) ($MY_ID) — check output" \
"Ramdisk Stop" "warning"
exit 1
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+1 -1
View File
@@ -293,6 +293,6 @@ elif [[ ${#FAILED[@]} -gt 0 ]]; then
"Recreate Shares" "warning"
exit 1
else
log "$ICON_DONE Status: done — ${#CREATED[@]} created, ${#SKIPPED[@]} skipped"
echo "$ICON_DONE Status: done — ${#CREATED[@]} created, ${#SKIPPED[@]} skipped"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+3 -3
View File
@@ -152,7 +152,7 @@ fi
echo ""
if [[ "$SKIP_COUNT" -eq 0 ]]; then
log "Skip list: empty — all containers monitored normally ✅"
echo "Skip list: empty — all containers monitored normally ✅"
else
warn "$SKIP_COUNT container(s) on skip list — manual intervention needed:"
echo ""
@@ -177,9 +177,9 @@ fi
echo ""
echo "━━━ $ICON_WATCHDOG Restart History ━━━"
if [[ "$RESTART_COUNT" -eq 0 ]]; then
log "No restart history"
echo "No restart history"
else
log "$RESTART_COUNT restart entries (window: ${WATCHDOG_CONTAINER_RESTART_WINDOW}h)"
echo "$RESTART_COUNT restart entries (window: ${WATCHDOG_CONTAINER_RESTART_WINDOW}h)"
echo ""
awk -F'|' '{counts[$1]++} END {
for (c in counts)
+3 -3
View File
@@ -50,7 +50,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_ZFS_REPORT_IGNORE_POOLS
# Pools to exclude from automatic scrub. Typically single-disk VM pools
@@ -281,7 +281,7 @@ for pool in "${STARTED[@]}"; do
error " $ERRORS"
POOLS_ERRORS+=("$pool")
else
log "$pool$SCAN_LINE"
echo "$pool$SCAN_LINE"
POOLS_OK+=("$pool")
fi
done
@@ -304,7 +304,7 @@ if [[ ${#POOLS_ERRORS[@]} -gt 0 ]]; then
notify "ZFS scrub errors on $(hostname) ($MY_ID) — pools with errors: ${POOLS_ERRORS[*]}" \
"ZFS Scrub" "warning"
elif [[ ${#POOLS_OK[@]} -gt 0 ]]; then
log "$ICON_DONE Status: all ${#POOLS_OK[@]} pools clean ✅"
echo "$ICON_DONE Status: all ${#POOLS_OK[@]} pools clean ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+22 -4
View File
@@ -18,6 +18,24 @@ for the ramdisk transcode system. Read the Docker mount section before anything
---
## Output Tiers
All three scripts follow a two-tier output model: `echo` lines are always visible;
`log` lines only appear when `--log` is passed.
**ramdisk_setup.sh** — one-shot at array start. Without `--log`, section headers
and the final summary are visible. Per-step creation detail suppressed.
**transcode_manager.sh** — runs every 3 minutes. Without `--log`, only state
transitions (flips, warnings, errors) and active session display are shown. When
nothing changes, a single one-line confirmation is printed. Per-check detail suppressed.
**transcode_cleanup.sh** — runs every 3 minutes. Without `--log`, the cleanup
summary (files removed, space freed) is always visible. Per-file deletion detail
suppressed.
---
## Docker Mount — Required Configuration
> **This is the most important configuration requirement in this folder.**
@@ -198,7 +216,7 @@ TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
### Threshold Sizing
```bash
# master_host*.conf
# host*.conf
HOST1_RAMDISK_SIZE="10G"
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
@@ -313,7 +331,7 @@ transcode_cleanup.sh --log # verbose per-file output
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback
# ── Per-Host (master_host*.conf) ───────────────────────────────────────────────
# ── Per-Host (host*.conf) ───────────────────────────────────────────────
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes" # ramdisk mount point
HOST1_RAMDISK_SIZE="10G" # tmpfs ceiling (not a reservation)
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
@@ -402,7 +420,7 @@ docker inspect Emby | grep -A3 "Mounts"
# Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB"
#
# If peak is close to RAMDISK_WARN_GB → increase ramdisk size:
# master_host1.conf
# host1.conf
HOST1_RAMDISK_SIZE="12G" # increase by 2G
HOST1_RAMDISK_WARN_GB=10.5 # adjust thresholds accordingly
HOST1_RAMDISK_LOW_GB=8.5
@@ -447,7 +465,7 @@ readlink /mnt/ram-transcode
### Increasing Ramdisk Size After Initial Setup
```bash
# 1. Set new size and thresholds in master_host*.conf
# 1. Set new size and thresholds in host*.conf
# 2. Unmount the existing ramdisk (no sessions should be active):
umount /mnt/ramdisk_transcodes
+4 -4
View File
@@ -53,7 +53,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_RAMDISK_SIZE
# tmpfs ceiling (e.g. 10G). Must change together with WARN_GB and LOW_GB.
@@ -175,8 +175,8 @@ fi
# ==============================================================================================
echo ""
echo "━━━ $ICON_RAM Ramdisk — $MY_ID ━━━"
echo "$ICON_RAM Path: $RAMDISK_PATH"
echo "$ICON_RAM Size: $RAMDISK_SIZE"
log "$ICON_RAM Path: $RAMDISK_PATH"
log "$ICON_RAM Size: $RAMDISK_SIZE"
echo ""
START=$(date +%s)
@@ -355,7 +355,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
elif [[ "$SETUP_SUCCESS" == true ]]; then
log "$ICON_DONE Status: done ✅"
echo "$ICON_DONE Status: done ✅"
else
echo "$ICON_ERROR Status: SETUP HAD ERRORS"
notify "Ramdisk setup errors on $(hostname) ($MY_ID) — check output" \
+3 -3
View File
@@ -65,7 +65,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD / HOST*_RAMDISK_LOW_GB
# Aliased by detect_hosts() → RAMDISK_PATH / TRANSCODE_SSD / RAMDISK_LOW_GB.
@@ -292,7 +292,7 @@ if [[ "$DRY_RUN" == false ]] && mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
CURRENT_TARGET=$(grep "^current_target=" "$STATE_FILE" 2>/dev/null | cut -d= -f2)
if [[ "$LOW_RECOVERED" == "1" && "$CURRENT_TARGET" == "$TRANSCODE_SSD" ]]; then
log "Ramdisk has space after cleanup (${RAMDISK_USED_GB}GB < ${RAMDISK_LOW_GB}GB) — triggering manager to flip back"
echo "Ramdisk has space after cleanup (${RAMDISK_USED_GB}GB < ${RAMDISK_LOW_GB}GB) — triggering manager to flip back"
bash "$SCRIPT_DIR/transcode_manager.sh" --no-log
fi
fi
@@ -314,6 +314,6 @@ echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no files deleted"
else
log "$ICON_DONE Status: done ✅"
echo "$ICON_DONE Status: done ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+2 -2
View File
@@ -65,7 +65,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD
# Ramdisk mount point and SSD fallback path.
@@ -650,5 +650,5 @@ if [[ "$SOMETHING_HAPPENED" == true ]]; then
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
else
log "Transcode manager — all healthy — $MY_ID ($(format_duration $(( END - START ))))"
echo "Transcode manager — all healthy — $MY_ID ($(format_duration $(( END - START ))))"
fi
+641
View File
@@ -0,0 +1,641 @@
# ━━━━━ WATCHDOGS — Manual ━━━━━
Configuration reference, operational procedures, and troubleshooting for all four
watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
---
## ━━━ CONTENTS ━━━
- [Output Tiers](#output-tiers)
- [resource_watchdog.sh](#resource_watchdogsh)
- [docker_watchdog.sh](#docker_watchdogsh)
- [storage_watchdog.sh](#storage_watchdogsh)
- [system_watchdog.sh](#system_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
---
## Output Tiers
All watchdog scripts use a two-tier output model: `echo` lines are always visible;
`log` lines only appear when `--log` is passed.
All four watchdogs are **single-pass scripts** called once per minute by the orchestrator.
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
visible. Per-check detail is suppressed on clean cycles.
`docker_watchdog.sh` is the exception — it is **silent on clean cycles by design**.
96 cycles/day means clean-cycle noise would bury real events. Its output only appears
when there are restarts, skip-list events, or RAM deferral. Use `--log` to see
per-cycle detail on clean cycles.
---
## resource_watchdog.sh
Runs first in the orchestrator sequence. Reduces system pressure before docker_watchdog
attempts any container restarts. Containers restarted into a RAM-pressured system just
fail again — this script ensures docker_watchdog has breathing room.
### Pressure Levels
Three escalating levels, each additive:
```bash
# master.conf
RW_RAM_SOFT_GB=20 # Level 1 trigger — throttle downloaders
RW_RAM_MEDIUM_GB=15 # Level 2 trigger — throttle + pause containers
RW_RAM_HARD_GB=10 # Level 3 trigger — stop containers + defer docker_watchdog
RW_RAM_RECOVER_GB=25 # de-escalate only after RAM reaches this
RW_LOAD_SOFT_MULTIPLIER=2.0 # load > 2× cpu count = level 1
RW_LOAD_MEDIUM_MULTIPLIER=3.0 # load > 3× cpu count = level 2
RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalating
```
**Level 1 (soft):** Throttle SABnzbd + qBittorrent download speeds.
**Level 2 (medium):** Further throttle + `docker pause` non-critical containers.
**Level 3 (hard):** `docker stop` optional services + write `mem_shutdown_active=true`
to `RW_STATE_FILE`. docker_watchdog.sh reads this flag and skips all restart logic
until pressure clears. Without this coordination, docker_watchdog would immediately
restart containers that resource_watchdog just stopped to free RAM.
Recovery de-escalates one level at a time — prevents flip-flopping between states.
### Per-Host Container Lists
```bash
# host1.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at level 2
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at level 3
```
Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
pressure level. Default: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
### Downloader Throttle Config
```bash
# master.conf
RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M" # throttled at level 1
RW_SABNZBD_SPEED_MEDIUM="10M" # throttled further at level 2
RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200 # KB/s — level 1
RW_QBIT_DL_MEDIUM=10240 # KB/s — level 2
# host1.conf (API access)
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"
```
### Usage
```bash
resource_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
resource_watchdog.sh --dry-run # show what would be throttled/paused/stopped
resource_watchdog.sh --status # current level, active actions, recovery cycle count
resource_watchdog.sh --log # verbose per-check output
```
---
## docker_watchdog.sh
Runs second in the orchestrator sequence. Two-tier container healing — explicit
per-container configuration (Tier 1) plus a global catch-all scan (Tier 2).
Reads `RW_STATE_FILE` at cycle start — if `mem_shutdown_active=true`, skips all
container restart logic (resource_watchdog is managing the situation). Health URL
checks for excluded containers still run.
### Memory Hard Limits
```bash
# host1.conf
# Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
#
# Sizing: check normal peak with "docker stats ContainerName"
# Set limit at ~150-200% of normal peak
#
HOST1_WATCHDOG_CONTAINERS=(
"Emby:18432" # 18GB — peaks ~12GB under heavy transcode load
"LidaTube:6144" # 6GB — YouTube downloader, grows with large queues
"Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature
"Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous
)
```
A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
visibility into a container approaching its ceiling before a restart is triggered.
### CPU Thresholds
```bash
# master.conf
# CPU is normalised against total core count.
# 85% normalised on a 16-core machine = 13.6 cores worth of a single process.
#
# CPU uses a STRIKE SYSTEM — brief spikes are normal (Tdarr, Emby transcoding, SABnzbd).
# Strike 1: above HARD_CPU_THRESHOLD → warn, increment strike
# Strike 2: above threshold → restart, reset counter
# Recovery: drops below threshold any cycle → reset to 0
#
SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only
HARD_CPU_THRESHOLD=85 # strike at 85% normalised
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
```
### HTTP Health Checks
```bash
# host1.conf
# Format: "ContainerName:http://host:port/optional-path"
# Two consecutive non-responses trigger a restart.
# "Container running" and "service responding" are not the same thing.
#
HOST1_WATCHDOG_CONTAINER_URLS=(
"Emby:http://localhost:8096" # Emby WebUI root
"NginxProxyManager:http://localhost:81" # NPM admin interface
)
# master.conf
CURL_TIMEOUT=5 # seconds before non-response counts as a failure
RESP_FAIL_LIMIT=2 # consecutive failures before restart
```
### Required Containers
```bash
# host1.conf
# Found stopped → restart attempted every cycle until running or skip-listed.
#
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
"NginxProxyManager"
"Authelia"
"Mariadb-Authelia"
"Redis-Authelia"
)
```
### Dependency Ordering
```bash
# host1.conf
# Format: "DependentContainer:dependency1 dependency2"
# All dependencies must be running before the dependent is restarted.
# Prevents Authelia crash-looping while MariaDB is still starting.
#
HOST1_WATCHDOG_DEPENDENCIES=(
"Authelia:Mariadb-Authelia Redis-Authelia"
"NextCloud:Postgres-NextCloud"
)
```
Same dependency config is used by `docker_daily_restart.sh` and
`docker_weekly_restart.sh` — configure once, applies everywhere.
### Startup Grace Period
```bash
# master.conf
# Suppress restart actions for N seconds after array start.
# Checks still run and log — only restart actions are suppressed.
#
WATCHDOG_STARTUP_GRACE=600 # 10 minutes
```
### Tier 2 Global Scan
```bash
# master.conf
WATCHDOG_SCAN_ALL=true # enable global scan
WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures
WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills
WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts
WATCHDOG_RESTART_DEAD=true # recover containers in dead state
WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero
WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list
# Exclude containers from Tier 2 (one-shots, manually managed, benign exits):
WATCHDOG_SCAN_IGNORE=(
"my-one-shot-container"
)
```
### Restart Loop Protection
```bash
# master.conf
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
```
After hitting the limit: skip list + critical notification. The watchdog stops
touching the container. Auto-clear: if the container recovers on its own and is
found running, it's removed from the skip list automatically. Manual clear is only
needed when the container is stuck stopped — use `Tools/watchdog_skip_list_manager.sh`.
### Notification Batching
```bash
# master.conf
WATCHDOG_BATCH_NOTIFY=true
# All events from one cycle → one notification at the end.
# A shared DB going down can cascade 10+ containers. Without batching: 10 pings.
# With batching: one summary listing all affected containers.
```
### Usage
```bash
docker_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
docker_watchdog.sh --dry-run # full cycle preview without restarting anything
docker_watchdog.sh --status # skip list, strike counts, grace period, RAM deferral state
docker_watchdog.sh --log # verbose per-cycle output
```
### Skip List Recovery
```bash
# Step 1 — understand the situation
Tools/watchdog_skip_list_manager.sh --status
# Step 2 — fix the underlying problem
# docker logs ContainerName --tail 100
# df -h /mnt/user
# Step 3 — clear the container
Tools/watchdog_skip_list_manager.sh --clear ContainerName
# Step 4 — start manually (confirms fix before handing back to watchdog)
docker start ContainerName
# Step 5 — monitoring resumes automatically on next cycle
```
---
## storage_watchdog.sh
Runs third in the orchestrator sequence. Two independent checks per cycle:
growth rate detection (automatic, zero config) and oversize log detection.
Uses its own strike state file — independent from docker_watchdog.
### Growth Rate Detection
```bash
# master.conf
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2 # growth per cycle that triggers a strike
WATCHDOG_APPDATA_STRIKE_LIMIT=3 # strikes before alert
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db" # size baseline
```
Runs `du -sm appdata/*/` each cycle — pure inode metadata, very lightweight on NVMe.
Compares each container's current size to the baseline from the previous cycle.
Growth > `WATCHDOG_APPDATA_GROWTH_GB` per cycle increments the container's strike count.
Strike 1: warn. Strike 2: escalate. Strike 3: critical alert.
Strikes auto-clear when growth drops to zero (condition resolved).
**Zero configuration required for new containers.** Growth rate detection covers all
containers automatically. The suppress array below is only for known-legitimate growth.
### Growth Suppress Ceilings
```bash
# host1.conf
# ONLY needed in specific cases — growth rate detection covers everything automatically.
# Use when a container's appdata legitimately grows fast during normal operation
# and you want to suppress false positives above a known-safe threshold.
#
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
["Tdarr"]="25600" # 25GB — transcode cache grows during active jobs
["7dtd"]="20480" # 20GB — game server world data, expected large
)
```
### Log File Detection
```bash
# master.conf
WATCHDOG_APPDATA_LOG_MAX_GB=2 # *.log / *.log.* files above this trigger a strike
WATCHDOG_APPDATA_TRUNCATE_LOGS=false # true: truncate at strike limit; false: alert only
```
Scans all `*.log` and `*.log.*` files across all appdata paths. Files over the threshold
increment per-file strike counts. At strike limit: truncates in-place with `truncate -s 0`
(container keeps its file handle — space reclaimed immediately without container restart)
or sends a critical alert if truncation is disabled.
Log strikes auto-clear when the file drops below threshold.
### Usage
```bash
storage_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings
storage_watchdog.sh --dry-run # show what would be alerted/truncated
storage_watchdog.sh --log # verbose per-container output
```
---
## system_watchdog.sh
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
to reboot. Watches the server itself — not containers, not storage. Reboots only when
healing at every other layer has failed or when the failure is non-recoverable.
### Three-Tier Response
**Tier 1 — CRITICAL (immediate reboot, no strikes)**
| Condition | Threshold | Why immediate |
|-----------|-----------|---------------|
| Docker daemon unresponsive | N/A | Every docker command hangs — nothing can be healed |
| rootfs usage | `SYS_WATCHDOG_ROOTFS_CRITICAL_PCT` (99%) | SSH stops; state files fail silently |
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
| File descriptor exhaustion | `SYS_WATCHDOG_FD_CRITICAL_PCT` (95%) | New connections silently failing |
| /boot read-only | write test fails | Config writes silently failing |
**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
RAM below `MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`.
Both conditions required — RAM alone uses the standard strike system.
OOM confirms the system is dying faster than watchdogs can heal.
**Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKES` consecutive failures → reboot)**
| Check | Threshold |
|-------|-----------|
| Free RAM | `MEM_WARN_GB``MEM_SHUTDOWN_GB``MEM_GB` |
| Load average | `SYS_WATCHDOG_LOAD_MULTIPLIER` × cpu_count |
| CPU temperature | `SYS_WATCHDOG_CPU_TEMP` |
| Zombie processes | `SYS_WATCHDOG_ZOMBIES` |
| /var/log usage | `SYS_WATCHDOG_VAR_LOG_PCT` |
| /tmp usage | `SYS_WATCHDOG_TMP_PCT` |
| Array disk errors | mdstat error delta > 0 |
| NIC state | interface operstate != "up" |
| Required containers | containers in `SYS_WATCHDOG_REQUIRED_CONTAINERS` |
### RAM Tiers
```
MEM_WARN_GB (10GB) → warn + notify, no action
MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery
MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM)
MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart
```
At `MEM_SHUTDOWN_GB`, all containers NOT listed in `SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED`
are stopped. Adjust in master.conf for your critical services.
### Abort Conditions
Prevent reboot — running them would risk data loss:
```bash
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
SYS_WATCHDOG_ABORT_ON_PARITY=true # Parity check/rebuild running
SYS_WATCHDOG_ABORT_ON_MOVER=true # Mover running
```
Tier 1 CRITICAL bypasses all abort conditions — an imminent crash outweighs data
safety concerns.
### Reboot Rate Limit
```bash
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # window in hours
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots within the window
```
If the server reboots `SYS_WATCHDOG_MAX_REBOOTS` times within the window, the watchdog
switches from rebooting to notifying only. Prevents a boot loop where the watchdog
reboots → something crashes again immediately → reboot again.
### Usage
```bash
system_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
system_watchdog.sh --dry-run # run detection logic without rebooting
system_watchdog.sh --status # thresholds, current state, strike counts
system_watchdog.sh --log # verbose per-check output
```
---
## Full Configuration Reference
```bash
# master.conf
# ── Resource Watchdog ──────────────────────────────────────────────────────────
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
RW_RAM_SOFT_GB=20
RW_RAM_MEDIUM_GB=15
RW_RAM_HARD_GB=10
RW_RAM_RECOVER_GB=25
RW_LOAD_SOFT_MULTIPLIER=2.0
RW_LOAD_MEDIUM_MULTIPLIER=3.0
RW_RECOVER_CYCLES=3
RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"
RW_SABNZBD_SPEED_MEDIUM="10M"
RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200 # KB/s
RW_QBIT_DL_MEDIUM=10240
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
# host*.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"
# ── Docker Watchdog ────────────────────────────────────────────────────────────
DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes)
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart)
SOFT_CPU_THRESHOLD=50
HARD_CPU_THRESHOLD=85
CPU_FAIL_LIMIT=2
CURL_TIMEOUT=5
RESP_FAIL_LIMIT=2
WATCHDOG_CONTAINER_RESTART_LIMIT=3
WATCHDOG_CONTAINER_RESTART_WINDOW=1
WATCHDOG_SCAN_ALL=true
WATCHDOG_SCAN_IGNORE=()
WATCHDOG_RESTART_UNHEALTHY=true
WATCHDOG_NOTIFY_OOM=true
WATCHDOG_NOTIFY_CRASHLOOP=true
WATCHDOG_CRASH_LIMIT=5
WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
WATCHDOG_BATCH_NOTIFY=true
# State files:
WATCHDOG_STATE_FILE="/tmp/watchdog_state.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
# host*.conf
HOST1_WATCHDOG_CONTAINERS=() # "ContainerName:LimitMB"
HOST1_WATCHDOG_CONTAINER_URLS=() # "ContainerName:http://host:port"
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
HOST1_WATCHDOG_DEPENDENCIES=() # "Dependent:dep1 dep2"
# ── Storage Watchdog ───────────────────────────────────────────────────────────
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2
WATCHDOG_APPDATA_LOG_MAX_GB=2
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
WATCHDOG_APPDATA_STRIKE_LIMIT=3
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
# host*.conf (optional — only for suppress ceilings)
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
# ["Tdarr"]="25600"
# )
# ── System Watchdog ────────────────────────────────────────────────────────────
SYS_WATCHDOG_STRIKE_LIMIT=2
SYSTEM_WATCHDOG_INTERVAL=300
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2
SYS_WATCHDOG_MAX_REBOOTS=3
SYS_WATCHDOG_OOM_LIMIT=3
MEM_WARN_GB=10
MEM_SHUTDOWN_GB=6
MEM_GB=4
MEM_RECOVER_GB=30
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99
SYS_WATCHDOG_FD_CRITICAL_PCT=95
SYS_WATCHDOG_LOAD_MULTIPLIER=4
SYS_WATCHDOG_CPU_TEMP=85
SYS_WATCHDOG_ZOMBIES=20
SYS_WATCHDOG_VAR_LOG_PCT=80
SYS_WATCHDOG_TMP_PCT=85
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=true
SYS_WATCHDOG_ABORT_ON_MOVER=true
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
"NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
)
SYS_WATCHDOG_REQUIRED_CONTAINERS=()
# State files:
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
```
---
## Troubleshooting
### resource_watchdog Paused Containers It Shouldn't
```bash
# Check current state:
resource_watchdog.sh --status
# Add to RW_CRITICAL_CONTAINERS in master.conf:
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
# Un-pause manually if needed:
docker unpause MyContainer
```
### docker_watchdog Keeps Restarting a Healthy Container
```bash
# Check what's triggering it — memory, CPU, HTTP, or required:
docker_watchdog.sh --status
# Check CPU normalised — brief spikes should not trigger (2-strike system):
# If triggering on CPU: check if HARD_CPU_THRESHOLD is set appropriately
# for containers with legitimate burst usage (Tdarr encoding, SABnzbd unpacking)
# Check HTTP — is the health endpoint returning 200?
curl -sf --max-time 5 http://localhost:PORT && echo "OK" || echo "FAIL"
```
### Container on the Skip List After Fixing the Problem
```bash
# See skip list and container state:
Tools/watchdog_skip_list_manager.sh --status
# Fix root cause first, then clear:
Tools/watchdog_skip_list_manager.sh --clear ContainerName
# Start manually to confirm fix before handing back to watchdog:
docker start ContainerName
```
### storage_watchdog Alerting on a Container That Grows Legitimately
```bash
# Check what triggered it:
storage_watchdog.sh --status
# Add a suppress ceiling to host*.conf:
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
# ["ContainerName"]="10240" # 10GB ceiling — legitimate growth, suppress below this
# )
```
### system_watchdog Rebooted Unexpectedly
```bash
# Check the reboot log (survives reboots):
cat /boot/config/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot
# Check syslog near the reboot time:
grep "system_watchdog" /var/log/syslog | tail -20
```
### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
```bash
# All four watchdogs run as single-pass scripts — there is no background process to check.
# If the orchestrator reports a timeout, one pass took longer than expected.
# Check the orchestrator itself:
Orchestrators/watchdog_orchestrator.sh --status
# Run the slow watchdog directly with --log to see where it's hanging:
Watchdogs/system_watchdog.sh --log --dry-run
```
+176
View File
@@ -0,0 +1,176 @@
# ━━━━━ WATCHDOGS ━━━━━
**Four single-pass scripts that run every minute through `watchdog_orchestrator.sh`,
each with a clear lane:** reduce system pressure → heal containers → protect storage →
reboot if nothing else worked. They never run standalone loops. The orchestrator calls
them in order, once per cron cycle.
---
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
**Container Memory Leaks Going Undetected for Days**
Emby's transcode session handling occasionally leaks memory. SABnzbd's Python process
expands slowly across downloads. Neither crashes dramatically — they just consume more
RAM until the system starts swapping. Docker reports both containers as `Up 14 days`.
Nothing alerts. By the time someone notices, the system has been degraded for hours.
Fix: `docker_watchdog.sh` — hard per-container memory ceilings. When a container
exceeds its limit the watchdog restarts it immediately. No strikes, no waiting.
A memory leak is not a transient spike.
**Containers That Look Running But Aren't Responding**
Docker reports a container as `Up` while its application layer has been frozen for
hours. The reverse proxy forwards traffic to a service that returns nothing. Users see
a broken page. Docker sees a healthy container.
Fix: `docker_watchdog.sh` — HTTP health checks on the actual service port every cycle.
Two consecutive non-responses trigger a restart. Process running and service responding
are not the same thing.
**System Pressure Causing Docker Watchdog to Undo Itself**
Resource pressure builds. RAM drops. docker_watchdog.sh tries to restart a container
into a system that's already swapping — the restarted container fails immediately
and goes on the skip list. The real problem (RAM pressure) is never addressed.
Fix: `resource_watchdog.sh` runs first in the orchestrator sequence. At Level 1 it
throttles downloaders. At Level 2 it pauses non-critical containers. At Level 3 it
stops heavy services and signals docker_watchdog to defer all restarts. By the time
docker_watchdog runs, the system has breathing room to actually heal.
**Runaway Log Files Filling a Pool Before Anyone Notices**
A game server container was offline for a year, restarted for a weekend, and wrote
130GB of logs to the appdata pool. The pool grew 13% in days. No alert fired —
nothing was watching for growth at the data level, only at the container level.
Fix: `storage_watchdog.sh` — growth-rate scan of every container's appdata directory
every cycle. No per-container configuration required. Runaway growth gets three
cycles to be confirmed, then alerts and (optionally) truncates logs automatically.
**Rootfs at 99% With SSH Failing Silently**
Rootfs fills. SSH stops accepting new connections. Docker can't write log files. State
files fail silently. The server is functionally dead but still technically running.
Nothing in the container layer can detect or recover from this — it requires a reboot.
Fix: `system_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
sequence so container healing and pressure reduction always get a chance first.
---
## ━━━ WHAT THIS FOLDER DOES ━━━
Four watchdogs. One purpose each. Fixed execution order via `watchdog_orchestrator.sh`.
```
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
Storage protection storage_watchdog.sh — pool growth rate + runaway log detection
Last resort system_watchdog.sh — reboot only when nothing else can recover
```
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
attempts restarts — containers restarted into a pressure-bound system just fail again.
Storage is checked after containers are healed — no false alerts from containers that
were already being restarted. System watchdog runs last — reboot is always the last
option, not the first.
**None of these scripts run standalone loops.** Each is a single-pass script called
once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles
startup grace, overlap protection, heartbeat, and sequencing.
---
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
```
Orchestrators/
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
──────────────────► docker_watchdog.sh (2nd)
──────────────────► storage_watchdog.sh (3rd)
──────────────────► system_watchdog.sh (4th — last resort)
Tools/
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
(operator utility — inspect + clear after fixing a crash-looping container)
Docker_Essentials/
All container lifecycle scripts (daily restart, updates, network) — unaffected.
docker_watchdog.sh coordinates with them via shared state, not direct calls.
unRAID_Essentials/
Server-level scripts (WebGUI restart, inotify tuning, log hygiene) — unaffected.
system_watchdog.sh runs in the same ecosystem but is independent of those scripts.
```
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
**watchdog_skip_list_manager.sh stays in Tools/** — it's an operator utility, not a watchdog.
---
## ━━━ SCRIPTS IN THIS FOLDER ━━━
| Script | Role | Called By |
|--------|------|-----------|
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute |
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute |
| `storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `watchdog_orchestrator.sh` — 3rd every minute |
| `system_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
---
## ━━━ HOW THE SCRIPTS RELATE ━━━
```
Every minute — watchdog_orchestrator.sh fires:
Step 1 — resource_watchdog.sh
│ RAM/load OK → pass through (no action)
│ Level 1 (soft): throttle SABnzbd + qBit download speeds
│ Level 2 (medium): further throttle + docker pause non-critical containers
│ Level 3 (hard): docker stop optional services
│ writes mem_shutdown_active=true → RW_STATE_FILE
│ ↓
Step 2 — docker_watchdog.sh
│ reads RW_STATE_FILE — if mem_shutdown_active=true: skip all restarts
│ Tier 1 — explicit per-container checks (configured in host*.conf):
│ memory hard limits → immediate restart (no strikes)
│ CPU high sustained → 2-strike restart
│ HTTP non-response → 2-strike restart
│ required stopped → restart (dependency ordering respected)
│ Tier 2 — global scan of all running containers:
│ unhealthy / OOM / crashloop / dead / non-zero exit → restart
│ N restarts in window → skip list + critical notify → human required
│ (skip list management) → Tools/watchdog_skip_list_manager.sh
Step 3 — storage_watchdog.sh
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
Step 4 — system_watchdog.sh
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
Tier 1 CRITICAL → immediate reboot (no strikes)
Tier 2 URGENT → reboot if OOM confirmed
Tier 3 STANDARD → N consecutive failures → reboot
Abort conditions → ZFS unhealthy / parity running / mover active
Rate limit → max N reboots per window before switching to notify
```
**State file coordination between scripts:**
| State File | Written By | Read By | Purpose |
|-----------|-----------|---------|---------|
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
| `SYS_WATCHDOG_STATE_FILE` | `system_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
| `STORAGE_WATCHDOG_STATE_FILE` | `storage_watchdog.sh` | itself | Growth + log strike counts |
| `WATCHDOG_APPDATA_GROWTH_FILE` | `storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
@@ -19,7 +19,7 @@
# ==============================================================================================
#
# Tier 1 — Strict Per-Container Monitoring
# Applies only to containers explicitly configured in master_host*.conf.
# Applies only to containers explicitly configured in host*.conf.
#
# Memory hard limits — immediate restart if container exceeds MB ceiling
# Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of limit (no restart)
@@ -145,7 +145,7 @@
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
# host*.conf
#
# HOST*_WATCHDOG_CONTAINERS
# Memory hard limits per container. Format: "ContainerName:LimitInMB"
@@ -922,6 +922,7 @@ CYCLE_START=$(date +%s)
fi # WATCHDOG_SCAN_ALL
# ── Send notifications ────────────────────────────────────────────────────────────────────
flush_notify
@@ -71,7 +71,7 @@
# RW_QBIT_ENABLED, RW_QBIT_DL_SOFT, RW_QBIT_DL_MEDIUM
# RW_CRITICAL_CONTAINERS — never paused or stopped regardless of pressure
#
# master_host*.conf (aliased by detect_hosts())
# host*.conf (aliased by detect_hosts())
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure
# HOST*_SABNZBD_URL, HOST*_SABNZBD_API_KEY
@@ -118,7 +118,7 @@ if [[ "$EUID" -ne 0 ]]; then
fi
if [[ "${RW_ENABLED:-true}" != "true" ]]; then
log "Resource Manager disabled (RW_ENABLED=false)"
echo "Resource Manager disabled (RW_ENABLED=false)"
exit 0
fi
@@ -138,6 +138,21 @@ touch "$RW_STATE_FILE" 2>/dev/null || {
exit 1
}
# ── Exit Trap — restart containers stopped this run if script crashes ──────────────────────────
declare -a _RW_TRAP_STOPPED=()
_rw_trap_restart_stopped() {
[[ ${#_RW_TRAP_STOPPED[@]} -eq 0 ]] && return
for c in "${_RW_TRAP_STOPPED[@]}"; do
[[ -z "$c" ]] && continue
if docker inspect "$c" >/dev/null 2>&1; then
warn "Exit trap: restarting $c (stopped but state not persisted)"
docker start "$c" >/dev/null 2>&1 || warn " Failed to restart $c"
fi
done
}
trap _rw_trap_restart_stopped EXIT
# ==============================================================================================
# ━━━ State Helpers ━━━
# ==============================================================================================
@@ -379,6 +394,7 @@ stop_containers() {
if timeout "$DOCKER_TIMEOUT" docker stop "$container" >/dev/null 2>&1; then
warn "Stopped $container (hard pressure)"
actually_stopped+=("$container")
_RW_TRAP_STOPPED+=("$container")
else
error "Failed to stop $container"
fi
@@ -468,7 +484,7 @@ apply_level_3() {
# ==============================================================================================
restore_level_3() {
log "Restoring from level 3 — starting stopped containers"
echo "Restoring from level 3 — starting stopped containers"
if [[ -n "$STOPPED_LIST" ]]; then
start_containers "$STOPPED_LIST"
STOPPED_LIST=""
@@ -478,7 +494,7 @@ restore_level_3() {
}
restore_level_2() {
log "Restoring from level 2 — unpausing containers"
echo "Restoring from level 2 — unpausing containers"
if [[ -n "$PAUSED_LIST" ]]; then
unpause_containers "$PAUSED_LIST"
PAUSED_LIST=""
@@ -486,7 +502,7 @@ restore_level_2() {
}
restore_level_1() {
log "Restoring from level 1 — removing downloader throttle"
echo "Restoring from level 1 — removing downloader throttle"
sabnzbd_set_speed "0"
qbit_set_dl_limit 0
}
@@ -552,9 +568,9 @@ elif [[ "$TARGET_LEVEL" -lt "$CURRENT_LEVEL" ]]; then
else
# ── Steady state ────────────────────────────────────────────────────────────────────────
if [[ "$CURRENT_LEVEL" -gt 0 ]]; then
log "Pressure holding at level $CURRENT_LEVEL — waiting for sustained recovery"
echo "Pressure holding at level $CURRENT_LEVEL — waiting for sustained recovery"
else
log "System at normal pressure ✅"
echo "System at normal pressure ✅"
fi
rm_state_set "rm_recover_cycles" 0
fi
@@ -564,5 +580,6 @@ fi
# ==============================================================================================
rm_state_set "rm_paused_containers" "$PAUSED_LIST"
rm_state_set "rm_stopped_containers" "$STOPPED_LIST"
trap - EXIT # state persisted — stopped containers recorded, trap no longer needed
# Touch state file each run so docker_watchdog stale guard sees fresh mtime
touch "$RW_STATE_FILE" 2>/dev/null
+428
View File
@@ -0,0 +1,428 @@
#!/bin/bash
# ==============================================================================================
# ================================= Storage Watchdog ===========================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Pool and storage health monitoring — catches runaway data growth before it
# fills a pool. Runs as a single-pass script called by watchdog_orchestrator.sh
# every cycle. Sits between docker_watchdog.sh (container health) and
# system_watchdog.sh (last line of defense). Never reboots — detects, alerts,
# and optionally remediates.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Appdata Size Monitoring
# Two complementary checks run every cycle:
#
# Part 1 — Growth rate (zero-config catch-all):
# Reads per-container dir totals via du, compares to previous cycle baseline.
# Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused
# *.log scan inside that container's dir. No per-container config required —
# new containers are covered automatically. Baseline built on first cycle after
# boot; growth detection active from cycle 2.
#
# Part 2 — Absolute log size:
# Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB anywhere in
# WATCHDOG_APPDATA_PATHS. Catches logs already large but no longer actively
# growing. Independent strike counter per file.
#
# Strike System
# Reuses the same strike pattern as CPU/HTTP checks in docker_watchdog.sh:
#
# Strike 1 — warn + notify: condition first detected this run
# Strike 2 — warn + escalated notify: still present next cycle
# Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
# WATCHDOG_APPDATA_TRUNCATE_LOGS=true → truncate *.log in-place, clear strikes
# WATCHDOG_APPDATA_TRUNCATE_LOGS=false → critical notify, hold strikes until resolved
# Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
#
# Suppress Ceiling (WATCHDOG_APPDATA_SIZES)
# Containers in HOST*_WATCHDOG_APPDATA_SIZES suppress growth warnings while
# under their configured ceiling MB. Use ONLY when a container legitimately
# holds large stable data and would otherwise false-alarm (e.g. Tdarr cache).
# Zero-config growth detection covers everything else automatically.
#
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# Zero-config for new containers
# Growth rate detection requires no per-container configuration. Add a game
# server, spin up a new arr, install anything — it is monitored automatically
# from the second cycle after it appears. The suppress ceiling in conf is the
# exception, not the rule.
#
# Alert-only for data, truncate-only for logs
# Non-log growth (databases, game saves, caches) is detected and alerted but
# never touched. Only *.log / *.log.* files are candidates for truncation —
# and only when WATCHDOG_APPDATA_TRUNCATE_LOGS=true. Truncation zeroes the
# file in-place; the container keeps its open file handle, space is reclaimed
# immediately. Never deletes.
#
# Strike before acting
# One cycle of growth could be a legitimate library scan or game save burst.
# Three consecutive cycles of growth is a runaway. The strike system separates
# transient activity from sustained problems before any action fires.
#
# Silent when healthy
# Produces no output when all checks pass. Loud only when something needs
# attention.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master.conf
#
# WATCHDOG_CHECK_APPDATA
# Master toggle for all appdata checks (default: true)
#
# WATCHDOG_APPDATA_PATHS
# Array of paths to scan (e.g. "/mnt/docker-unraid/appdata")
#
# WATCHDOG_APPDATA_GROWTH_GB
# Per-cycle growth threshold in GB — flag containers growing more than this (default: 2)
#
# WATCHDOG_APPDATA_LOG_MAX_GB
# Absolute *.log file size alert threshold in GB (default: 2)
#
# WATCHDOG_APPDATA_TRUNCATE_LOGS
# Auto-truncate oversized *.log files on action cycle (default: false)
#
# WATCHDOG_APPDATA_STRIKE_LIMIT
# Consecutive cycles before action fires (default: 3)
#
# WATCHDOG_APPDATA_GROWTH_FILE
# Per-container size baseline — /tmp resets on reboot (correct: stale baseline
# after reboot would give false growth readings on first cycle)
#
# STORAGE_WATCHDOG_STATE_FILE
# Strike counts for this script — /tmp resets on reboot
#
# host*.conf (aliased by detect_hosts())
#
# HOST*_WATCHDOG_APPDATA_SIZES
# Per-container growth suppress ceilings in MB. Suppress growth alerts while
# a container's dir stays below this ceiling. Only needed when a container
# legitimately has large stable data. Growth detection covers everything else.
#
# ==============================================================================================
# STATE FILES
# ==============================================================================================
#
# STORAGE_WATCHDOG_STATE_FILE — strike counts (/tmp — resets on reboot ✅)
# WATCHDOG_APPDATA_GROWTH_FILE — per-container size baseline (/tmp — resets on reboot ✅)
#
# /tmp files reset on reboot — correct. Pre-reboot strikes and growth baselines are
# meaningless after a reboot. Both rebuild cleanly from cycle 1.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# storage_watchdog.sh
# Single-pass storage health check. Silent if all healthy.
#
# storage_watchdog.sh --dry-run
# Run all checks without truncating anything. Shows what would be actioned.
#
# storage_watchdog.sh --status
# Show configuration, active strikes, and growth baseline status. Then exit.
#
# storage_watchdog.sh --log
# Verbose output — every container checked, every size comparison, every decision.
#
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
acquire_lock
detect_hosts
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be truncated"
touch "$STORAGE_WATCHDOG_STATE_FILE" 2>/dev/null
touch "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null
# ==============================================================================================
# ━━━ Status ━━━
# ==============================================================================================
if [[ "$SHOW_STATUS" == true ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY STORAGE WATCHDOG STATUS ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_GEAR Appdata check: ${WATCHDOG_CHECK_APPDATA:-true}"
echo "$ICON_GEAR Paths: ${WATCHDOG_APPDATA_PATHS[*]:-none}"
echo "$ICON_GEAR Growth thresh: ${WATCHDOG_APPDATA_GROWTH_GB:-2}GB/cycle"
echo "$ICON_GEAR Log max: ${WATCHDOG_APPDATA_LOG_MAX_GB:-2}GB"
echo "$ICON_GEAR Truncate logs: ${WATCHDOG_APPDATA_TRUNCATE_LOGS:-false}"
echo "$ICON_GEAR Strike limit: ${WATCHDOG_APPDATA_STRIKE_LIMIT:-3}"
echo ""
echo "── Active Strikes ──"
if [[ -s "$STORAGE_WATCHDOG_STATE_FILE" ]]; then
while IFS=':' read -r _sk _sv; do
_sv_clean="${_sv//[^0-9]/}"
[[ "${_sv_clean:-0}" -gt 0 ]] && echo " $_sk$_sv_clean strikes"
done < "$STORAGE_WATCHDOG_STATE_FILE"
else
echo " none"
fi
echo ""
echo "── Growth Baseline ──"
if [[ -s "$WATCHDOG_APPDATA_GROWTH_FILE" ]]; then
_entries=$(wc -l < "$WATCHDOG_APPDATA_GROWTH_FILE")
_age=$(( $(date +%s) - $(stat -c %Y "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null || echo 0) ))
echo " Entries: $_entries containers Age: $(( _age / 60 ))m ago"
else
echo " No baseline yet (builds on first cycle after boot)"
fi
echo ""
echo "── Suppress Ceilings (this host) ──"
if [[ ${#WATCHDOG_APPDATA_SIZES[@]} -gt 0 ]]; then
for _c in "${!WATCHDOG_APPDATA_SIZES[@]}"; do
_ceil_gb=$(awk "BEGIN {printf \"%.0f\", ${WATCHDOG_APPDATA_SIZES[$_c]} / 1024}")
echo " $_c${_ceil_gb}GB"
done
else
echo " none configured"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
# ==============================================================================================
# ━━━ Appdata Size Monitoring ━━━
# ==============================================================================================
[[ "$WATCHDOG_CHECK_APPDATA" != "true" ]] && exit 0
log "$ICON_GEAR Storage watchdog — $MY_ID$(date '+%H:%M:%S')"
WARNINGS=0
_STRIKE_LIMIT=${WATCHDOG_APPDATA_STRIKE_LIMIT:-3}
_GROWTH_MB=$(( ${WATCHDOG_APPDATA_GROWTH_GB:-2} * 1024 ))
_LOG_KB=$(( ${WATCHDOG_APPDATA_LOG_MAX_GB:-2} * 1024 * 1024 ))
# Tracks files handled by Part 1 to prevent duplicate alerts in Part 2
declare -A _HANDLED=()
for _appdata_path in "${WATCHDOG_APPDATA_PATHS[@]:-}"; do
[[ -z "$_appdata_path" || ! -d "$_appdata_path" ]] && continue
# ── Part 1: Growth rate scan ──────────────────────────────────────────────────────────────
declare -A _PREV=()
if [[ -f "$WATCHDOG_APPDATA_GROWTH_FILE" ]]; then
while IFS='|' read -r _cn _cs _; do
[[ -n "$_cn" ]] && _PREV["$_cn"]="$_cs"
done < "$WATCHDOG_APPDATA_GROWTH_FILE"
fi
_growth_tmp=$(mktemp 2>/dev/null) || _growth_tmp=""
_now=$(date +%s)
while IFS= read -r _du_line; do
_curr_mb=$(echo "$_du_line" | awk '{print $1}')
_cdir=$(echo "$_du_line" | awk '{print $2}')
_cname=$(basename "$_cdir")
[[ -z "$_cname" || "$_cname" == "*" ]] && continue
[[ -n "$_growth_tmp" ]] && echo "${_cname}|${_curr_mb}|${_now}" >> "$_growth_tmp"
_prev_mb="${_PREV[$_cname]:-}"
[[ -z "$_prev_mb" ]] && continue # First run after boot — building baseline
_growth_mb=$(( _curr_mb - _prev_mb ))
_safe=$(echo "$_cname" | tr -cd '[:alnum:]_')
# Condition resolved — growth stopped, clear strikes
if [[ "$_growth_mb" -le 0 ]]; then
_existing=$(get_strikes "appdata_growth_${_safe}" "$STORAGE_WATCHDOG_STATE_FILE")
_existing="${_existing//[^0-9]/}"
[[ "${_existing:-0}" -gt 0 ]] && \
set_strikes "appdata_growth_${_safe}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
continue
fi
# Check suppress ceiling
_ceiling="${WATCHDOG_APPDATA_SIZES[$_cname]:-}"
if [[ -n "$_ceiling" && "$_curr_mb" -lt "$_ceiling" ]]; then
log "$_cname — growth suppressed (${_curr_mb}MB < ${_ceiling}MB ceiling)"
continue
fi
# Growth exceeds threshold — strike logic
if [[ "$_growth_mb" -ge "$_GROWTH_MB" ]]; then
_growth_gb=$(awk "BEGIN {printf \"%.1f\", $_growth_mb / 1024}")
_curr_gb=$(awk "BEGIN {printf \"%.1f\", $_curr_mb / 1024}")
_strikes=$(get_strikes "appdata_growth_${_safe}" "$STORAGE_WATCHDOG_STATE_FILE")
_strikes="${_strikes//[^0-9]/}"; _strikes="${_strikes:-0}"
[[ "$_strikes" -lt "$_STRIKE_LIMIT" ]] && {
_strikes=$(( _strikes + 1 ))
set_strikes "appdata_growth_${_safe}" "$_strikes" "$STORAGE_WATCHDOG_STATE_FILE"
}
warn "$_cname — grew ${_growth_gb}GB this cycle (total: ${_curr_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]"
(( WARNINGS++ ))
# Focused *.log scan inside the growing container's dir
_found_logs=()
while IFS= read -r _lf; do
[[ -n "$_lf" ]] && _found_logs+=("$_lf")
done < <(find "$_cdir" -maxdepth 3 -type f \
\( -name "*.log" -o -name "*.log.*" \) \
-size +${_LOG_KB}k -printf "%s %p\n" 2>/dev/null)
_log_summary=""
[[ ${#_found_logs[@]} -gt 0 ]] && _log_summary=$(printf '%s\n' "${_found_logs[@]}" | \
awk '{printf "%.1fGB %s | ", $1/1073741824, $2}' | head -c 200)
if [[ "$_strikes" -lt "$_STRIKE_LIMIT" ]]; then
if [[ -n "$_log_summary" ]]; then
notify "$_cname grew ${_growth_gb}GB on $(hostname) [strike ${_strikes}/${_STRIKE_LIMIT}] — logs: ${_log_summary}" \
"Storage Watchdog" "warning"
else
notify "$_cname grew ${_growth_gb}GB on $(hostname) [strike ${_strikes}/${_STRIKE_LIMIT}] — data growth, no log files" \
"Storage Watchdog" "warning"
fi
else
# Action cycle
error "$_cname — growth strike limit reached (${_STRIKE_LIMIT} consecutive cycles, ${_growth_gb}GB this cycle)"
if [[ "$WATCHDOG_APPDATA_TRUNCATE_LOGS" == "true" && ${#_found_logs[@]} -gt 0 ]]; then
for _lf_entry in "${_found_logs[@]}"; do
_lf_path=$(echo "$_lf_entry" | cut -d' ' -f2-)
_lf_gb=$(echo "$_lf_entry" | awk '{printf "%.1f", $1/1073741824}')
_HANDLED["$_lf_path"]=1
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would truncate $_lf_path (${_lf_gb}GB)"
else
if truncate -s 0 "$_lf_path" 2>/dev/null; then
success "Truncated runaway log: $_lf_path (was ${_lf_gb}GB)"
notify "Truncated runaway log on $(hostname): $_lf_path (was ${_lf_gb}GB)" \
"Storage Watchdog" "warning"
else
warn "Failed to truncate $_lf_path"
fi
fi
done
set_strikes "appdata_growth_${_safe}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
else
if [[ -n "$_log_summary" ]]; then
notify "$_cname appdata runaway on $(hostname)${_growth_gb}GB growth for ${_STRIKE_LIMIT} cycles — logs: ${_log_summary}" \
"Storage Watchdog" "critical"
else
notify "$_cname appdata runaway on $(hostname)${_growth_gb}GB growth for ${_STRIKE_LIMIT} cycles — data growth, manual investigation needed" \
"Storage Watchdog" "critical"
fi
fi
fi
# Mark found logs as handled to suppress Part 2 duplicates this cycle
for _lf_entry in "${_found_logs[@]}"; do
_HANDLED["$(echo "$_lf_entry" | cut -d' ' -f2-)"]=1
done
fi
done < <(du -sm "$_appdata_path"/*/ 2>/dev/null)
# Atomically update growth baseline
[[ -n "$_growth_tmp" ]] && mv "$_growth_tmp" "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null
# ── Part 2: Absolute log size scan ────────────────────────────────────────────────────────
# Catches *.log files already large but no longer actively growing this cycle.
# Same strike logic. Skips files already handled by Part 1 above.
declare -A _LOG_SEEN=()
while IFS= read -r _hit; do
[[ -z "$_hit" ]] && continue
_fpath=$(echo "$_hit" | cut -d' ' -f2-)
[[ -n "${_HANDLED[$_fpath]:-}" ]] && continue
_fsize_bytes=$(echo "$_hit" | awk '{print $1}')
_fsize_gb=$(awk "BEGIN {printf \"%.1f\", $_fsize_bytes / 1073741824}")
_safe_fkey=$(echo "$_fpath" | tr -cd '[:alnum:]_' | cut -c1-120)
_LOG_SEEN["appdata_log_${_safe_fkey}"]=1
_strikes=$(get_strikes "appdata_log_${_safe_fkey}" "$STORAGE_WATCHDOG_STATE_FILE")
_strikes="${_strikes//[^0-9]/}"; _strikes="${_strikes:-0}"
[[ "$_strikes" -lt "$_STRIKE_LIMIT" ]] && {
_strikes=$(( _strikes + 1 ))
set_strikes "appdata_log_${_safe_fkey}" "$_strikes" "$STORAGE_WATCHDOG_STATE_FILE"
}
warn "Oversized log: $_fpath (${_fsize_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]"
(( WARNINGS++ ))
if [[ "$_strikes" -lt "$_STRIKE_LIMIT" ]]; then
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]" \
"Storage Watchdog" "warning"
else
error "Oversized log persists for ${_STRIKE_LIMIT} cycles: $_fpath (${_fsize_gb}GB)"
if [[ "$WATCHDOG_APPDATA_TRUNCATE_LOGS" == "true" ]]; then
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would truncate $_fpath (${_fsize_gb}GB)"
else
if truncate -s 0 "$_fpath" 2>/dev/null; then
success "Truncated oversized log: $_fpath (was ${_fsize_gb}GB)"
notify "Truncated oversized log on $(hostname): $_fpath (was ${_fsize_gb}GB)" \
"Storage Watchdog" "warning"
set_strikes "appdata_log_${_safe_fkey}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
unset "_LOG_SEEN[appdata_log_${_safe_fkey}]"
else
warn "Failed to truncate $_fpath"
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) — ${_STRIKE_LIMIT} cycles, truncate failed" \
"Storage Watchdog" "critical"
fi
fi
else
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) — ${_STRIKE_LIMIT} cycles, manual intervention needed" \
"Storage Watchdog" "critical"
fi
fi
done < <(find "$_appdata_path" -maxdepth 4 -type f \
\( -name "*.log" -o -name "*.log.*" \) \
-size +${_LOG_KB}k -printf "%s %p\n" 2>/dev/null)
# Auto-clear strikes for log files no longer oversized this cycle
while IFS=':' read -r _sk _sv; do
[[ "$_sk" != appdata_log_* ]] && continue
_sv_clean="${_sv//[^0-9]/}"
[[ "${_sv_clean:-0}" -eq 0 ]] && continue
[[ -n "${_LOG_SEEN[$_sk]:-}" ]] && continue
set_strikes "$_sk" 0 "$STORAGE_WATCHDOG_STATE_FILE"
done < "$STORAGE_WATCHDOG_STATE_FILE" 2>/dev/null
unset _LOG_SEEN
done
unset _PREV _HANDLED
# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
if [[ "$WARNINGS" -gt 0 ]]; then
echo ""
echo "━━━ $ICON_GEAR Storage Watchdog — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo "$ICON_WATCHDOG Warnings: $WARNINGS"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
else
log "Storage healthy ✅ ($(date '+%H:%M:%S'))"
fi
@@ -350,6 +350,17 @@ run_strike_check() {
return 1
}
# ── Exit Trap — restart containers stopped before an aborted reboot ───────────────────────────
_SYS_REBOOT_STOPPED=()
_trap_sys_reboot_restart() {
[[ ${#_SYS_REBOOT_STOPPED[@]} -eq 0 ]] && return
warn "Exit trap: restarting containers stopped before aborted reboot"
for c in "${_SYS_REBOOT_STOPPED[@]}"; do
[[ -z "$c" ]] && continue
docker inspect "$c" >/dev/null 2>&1 && docker start "$c" >/dev/null 2>&1 || true
done
}
# ==============================================================================================
# ── DO REBOOT ─────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
@@ -424,6 +435,8 @@ do_reboot() {
warn "Stopping Docker containers..."
if command -v docker >/dev/null 2>&1; then
mapfile -t _SYS_REBOOT_STOPPED < <(docker ps --format '{{.Names}}' 2>/dev/null)
trap _trap_sys_reboot_restart EXIT
timeout 60 docker ps -q 2>/dev/null | xargs -r docker stop >/dev/null 2>&1
fi
@@ -433,6 +446,7 @@ do_reboot() {
warn "Syncing disks..."
sync
trap - EXIT # committed to reboot — containers should stay down
sleep 5
/sbin/reboot
}
@@ -784,7 +798,7 @@ echo "━━━━━━━━━━━━━━━━━━━━━━━━
do_reboot "standard" "${TRIGGERS[@]}"
exit 0
else
log "System healthy ✅ ($(date '+%H:%M:%S'))"
echo "System healthy ✅ ($(date '+%H:%M:%S'))"
fi
# Keep state file mtime fresh — docker_watchdog stale guard checks this
+23 -31
View File
@@ -55,11 +55,11 @@
# check_local_disk_temps() — pre-rsync temp check with exit codes 0/1/2
# stop/start local containers added alongside existing remote variants
#
# v3.1 Three-file config split — master.conf + master_host1.conf + master_host2.conf
# load_config.sh introduced — auto-discovers all master_host*.conf files
# v3.1 Three-file config split — master.conf + host1.conf + host2.conf
# load_config.sh introduced — auto-discovers all host*.conf files
# detect_hosts() rewritten — sets MY_ID/REMOTE_ID and aliases all HOST* vars
# All scripts now source load_config.sh instead of conf files directly
# Adding a new server = add master_host*.conf, zero script changes required
# Adding a new server = add host*.conf, zero script changes required
#
# v3.2 check_remote_disks() rewritten — three-tier detection:
# Tier 1: array disk paths (/mnt/disk*/sharename)
@@ -78,10 +78,6 @@
# Emby immediately removes ghost entries — no user-facing file-not-found errors
# Called by lidarr/sonarr/radarr_cleanup.sh when files are deleted
#
# v3.4 Silent-by-default output model
# info() and success() now gated by SILENT_MODE — only warn/error always visible
# Exception: monitor scripts designed to produce output stay verbose
# Reduces notification spam — ecosystem only speaks when something is wrong
#
# Three new safety functions added:
# check_unraid_version_parity() — refuses remote ops on version mismatch
@@ -190,27 +186,16 @@ ICON_SUCCESS="✅"
# ==============================================================================================
# Standardised output functions used across all scripts.
#
# Silent-by-default model:
# SILENT_MODE=true (default) — only warn() and error() produce output
# SILENT_MODE=false — all functions produce output
# --log flag — enables ENABLE_LOGGING (detailed [LOG] lines)
#
# Rules:
# error() — always visible — something broke
# warn() — always visible — something needs attention
# info() — silent by default — operational detail, visible when SILENT_MODE=false
# success() — silent by default — confirmation, visible when SILENT_MODE=false
# log() — debug detail — only when ENABLE_LOGGING=true
#
# Exception — monitor scripts are designed to produce output and set SILENT_MODE=false
# at the top of the script. All other scripts use the silent default.
# Two-tier model:
# echo — always visible — summaries, status conclusions, section headers
# warn() — always visible — state transitions, warnings, important events
# error() — always visible — something broke
# log() — only with --log flag — per-item detail, internal checks
#
# All output goes to stdout — callers can redirect as needed.
info() { [[ "${SILENT_MODE:-true}" == false ]] && echo "$ICON_INFO [INFO] $*"; return 0; }
warn() { echo "$ICON_WARN [WARN] $*"; }
error() { echo "$ICON_ERROR [ERROR] $*"; }
success() { [[ "${SILENT_MODE:-true}" == false ]] && echo "$ICON_SUCCESS [OK] $*"; return 0; }
log() {
[[ "${ENABLE_LOGGING:-false}" == true ]] && echo "[LOG] $*"
@@ -223,7 +208,7 @@ log() {
# Sends a notification via unRAID native system and/or Discord webhook.
# Both channels are optional and independently controlled:
# NOTIFY_UNRAID — shared toggle in master.conf
# MY_DISCORD_WEBHOOK — per-host in master_host*.conf, set by detect_hosts()
# MY_DISCORD_WEBHOOK — per-host in host*.conf, set by detect_hosts()
#
# Severity levels: normal, warning, alert
# Usage: notify "message" "subject" "severity"
@@ -254,7 +239,7 @@ notify() {
-d "$payload" "$MY_DISCORD_WEBHOOK" >/dev/null 2>&1; then
log "$ICON_NOTIFY Discord notification sent"
else
warn "Discord notification failed — check HOST*_DISCORD_WEBHOOK in master_host*.conf"
warn "Discord notification failed — check HOST*_DISCORD_WEBHOOK in host*.conf"
fi
fi
}
@@ -363,7 +348,7 @@ validate_int() {
# ── HOST DETECTION ────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Determines which server is local and which is remote by comparing hostname against
# all HOST* values discovered from master_host*.conf files.
# all HOST* values discovered from host*.conf files.
#
# Sets:
# MY_ID — "HOST1" or "HOST2" (the role key, not the hostname)
@@ -384,6 +369,7 @@ validate_int() {
# WATCHDOG_REQUIRED_CONTAINERS ← HOST*_WATCHDOG_REQUIRED_CONTAINERS
# WATCHDOG_SCAN_IGNORE ← HOST*_WATCHDOG_SCAN_IGNORE
# WATCHDOG_DEPENDENCIES ← HOST*_WATCHDOG_DEPENDENCIES (associative)
# WATCHDOG_APPDATA_SIZES ← HOST*_WATCHDOG_APPDATA_SIZES (associative)
# NETWORK_CONNECT_CONTAINERS ← HOST*_NETWORK_CONNECT_CONTAINERS
# NETWORK_CONNECT_NETWORKS ← HOST*_NETWORK_CONNECT_NETWORKS
# MEDIA_PERMISSION_SHARES ← HOST*_MEDIA_PERMISSION_SHARES
@@ -423,7 +409,7 @@ detect_hosts() {
local host_var host_val
# Scan all HOST* vars that hold a hostname value
# HOST1, HOST2, HOST3 etc. — all defined in master_host*.conf files
# HOST1, HOST2, HOST3 etc. — all defined in host*.conf files
for host_var in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do
host_val="${!host_var:-}"
[[ -z "$host_val" ]] && continue
@@ -435,7 +421,7 @@ detect_hosts() {
if [[ -z "$MY_ID" ]]; then
error "Unknown host: $local_hostname"
error "Hostname must match a HOST* value in master_host*.conf"
error "Hostname must match a HOST* value in host*.conf"
error "Available: $(for h in HOST1 HOST2 HOST3 HOST4; do
[[ -n "${!h:-}" ]] && echo -n "${!h} "; done)"
exit 1
@@ -459,7 +445,7 @@ detect_hosts() {
local ssh_key_var="${MY_ID}_SSH_KEY"
SSH_KEY="${!ssh_key_var:-}"
if [[ -z "$SSH_KEY" ]]; then
error "Missing SSH key: ${ssh_key_var} not set in master_host*.conf"
error "Missing SSH key: ${ssh_key_var} not set in host*.conf"
exit 1
fi
@@ -467,6 +453,11 @@ detect_hosts() {
MY_DISCORD_WEBHOOK_VAR="${MY_ID}_DISCORD_WEBHOOK"
MY_DISCORD_WEBHOOK="${!MY_DISCORD_WEBHOOK_VAR:-}"
OWNER_NAME_VAR="${MY_ID}_OWNER"
OWNER_NAME="${!OWNER_NAME_VAR:-}"
OWNER_EMAIL_VAR="${MY_ID}_OWNER_EMAIL"
OWNER_EMAIL="${!OWNER_EMAIL_VAR:-}"
EMBY_CONTAINER_VAR="${MY_ID}_EMBY_CONTAINER"
EMBY_CONTAINER="${!EMBY_CONTAINER_VAR:-}"
EMBY_URL_VAR="${MY_ID}_EMBY_URL"
@@ -598,9 +589,10 @@ detect_hosts() {
_alias_assoc "WATCHDOG_CONTAINERS"
_alias_assoc "WATCHDOG_CONTAINER_URLS"
_alias_assoc "WATCHDOG_DEPENDENCIES"
_alias_assoc "WATCHDOG_APPDATA_SIZES"
# ── Output ────────────────────────────────────────────────────────────────
info "$ICON_HOST Host: $LOCAL_SERVER_NAME$REMOTE_SERVER_NAME"
log "$ICON_HOST Host: $LOCAL_SERVER_NAME$REMOTE_SERVER_NAME"
}
# ==============================================================================================
@@ -1779,7 +1771,7 @@ show_status() {
echo "Remote ID: ${REMOTE_ID:-not set}"
echo "unRAID ver: $local_ver"
echo "Profile: ${PROFILE_NAME:-n/a}"
echo "Silent mode: ${SILENT_MODE:-true}"
echo "DryRun: ${DRY_RUN:-false}"
echo "Logging: ${ENABLE_LOGGING:-false}"
echo "SSH Key: ${SSH_KEY:-not set}"
+13 -16
View File
@@ -8,19 +8,19 @@
# ── WHAT THIS SCRIPT DOES ─────────────────────────────────────────────────────────────────────
# 1. Detects which server it's running on via detect_hosts() (MY_ID)
# 2. Configures sparse checkout to exclude other servers' credential files
# Each server only pulls its own master_host*.conf — never sees peer credentials
# Each server only pulls its own host*.conf — never sees peer credentials
# 3. Pulls or clones latest scripts from Gitea
# 4. Sets executable permissions on all .sh files
#
# ── SPARSE CHECKOUT ───────────────────────────────────────────────────────────────────────────
# Sparse checkout ensures each server only receives its own host conf:
# HOST1 pulls: master.conf + master_host1.conf + all scripts
# HOST1 skips: master_host2.conf, master_host3.conf etc.
# HOST2 pulls: master.conf + master_host2.conf + all scripts
# HOST2 skips: master_host1.conf, master_host3.conf etc.
# HOST1 pulls: master.conf + host1.conf + all scripts
# HOST1 skips: host2.conf, host3.conf etc.
# HOST2 pulls: master.conf + host2.conf + all scripts
# HOST2 skips: host1.conf, host3.conf etc.
#
# Adding a new server:
# Create master_host3.conf in the repo
# Create host3.conf in the repo
# All existing servers automatically exclude it on next pull
# New server gets only its own conf ✅
#
@@ -55,9 +55,6 @@ parse_args "$@"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
@@ -128,7 +125,7 @@ fi
# ==============================================================================================
# ━━━ Sparse Checkout Configuration ━━━
# ==============================================================================================
# Build the list of master_host*.conf files that belong to OTHER servers.
# Build the list of host*.conf files that belong to OTHER servers.
# This server pulls everything EXCEPT those files.
# MY_ID is set by detect_hosts() — e.g. "HOST1"
@@ -140,7 +137,7 @@ configure_sparse_checkout() {
# Enable sparse checkout
git -C "$repo_dir" config core.sparseCheckout true 2>/dev/null
# Build exclusion list — all master_host*.conf files except MY_ID's
# Build exclusion list — all host*.conf files except MY_ID's
local sparse_file="$repo_dir/.git/info/sparse-checkout"
mkdir -p "$(dirname "$sparse_file")"
@@ -148,9 +145,9 @@ configure_sparse_checkout() {
echo "/*" > "$sparse_file"
# Exclude each other server's conf file
# Find all master_host*.conf files present in the repo
# Find all host*.conf files present in the repo
local excluded=0
for conf_file in "$repo_dir"/master_host*.conf; do
for conf_file in "$repo_dir"/host*.conf; do
[[ -f "$conf_file" ]] || continue
local conf_name
conf_name=$(basename "$conf_file")
@@ -196,7 +193,7 @@ SYNC_SUCCESS=false
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would sync $REPO_SSH$TARGET_DIR"
warn "DRY RUN — would configure sparse checkout for $MY_ID"
warn "DRY RUN — would exclude peer master_host*.conf files"
warn "DRY RUN — would exclude peer host*.conf files"
SYNC_SUCCESS=true
else
mkdir -p "$TARGET_DIR"
@@ -236,7 +233,7 @@ else
echo " Clone successful"
# Configure sparse checkout after clone
# Now all master_host*.conf files are present — can detect exclusions
# Now all host*.conf files are present — can detect exclusions
configure_sparse_checkout "$TARGET_DIR"
# Apply sparse checkout — removes excluded files from working tree
@@ -270,7 +267,7 @@ echo "━━━━━ $ICON_SUMMARY GIT SYNC SUMMARY ━━━━━"
echo "$ICON_NET Repo: $REPO_SSH"
echo "$ICON_GEAR Target: $TARGET_DIR"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_LOCK Excluded: peer master_host*.conf files"
echo "$ICON_LOCK Excluded: peer host*.conf files"
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
if [[ "$DRY_RUN" == true ]]; then
echo "$ICON_WARN Status: DRY RUN — no changes made"
+26 -7
View File
@@ -10,7 +10,7 @@
# HOST2 never sees HOST1 credentials — clean separation at the file level.
#
# DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf.
# DO NOT put HOST2 variables here — they belong in master_host2.conf.
# DO NOT put HOST2 variables here — they belong in host2.conf.
#
# ── INDEX ─────────────────────────────────────────────────────────────────────────────────────
#
@@ -59,6 +59,12 @@
# RADARR URL, API key, path map
# ARR RECOVERY per-arr recovery toggles
#
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
# SYSTEM WATCHDOG per-host check toggles and NIC configuration
#
# ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
# RESOURCE MANAGER containers paused/stopped under memory pressure
#
# ==============================================================================================
# ==============================================================================================
@@ -70,6 +76,8 @@
# SSH key used for all server-to-server operations — rsync, failover container commands.
# Must be in /root/.ssh/ and authorised in HOST2's /root/.ssh/authorized_keys.
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
HOST1_OWNER="gmer4lfe"
HOST1_OWNER_EMAIL="gmer4lfe@gmail.com"
# ━━━ Emby ━━━
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
@@ -319,6 +327,17 @@
["NextCloud"]="Postgres-NextCloud"
)
# Per-container appdata growth suppress ceilings in MB.
# ONLY needed in specific cases — growth rate detection covers all containers automatically.
# Use this when a container legitimately has large stable data and you want to guarantee
# it never triggers a false-positive growth alert. Growth warnings are suppressed while the
# container's dir stays below this ceiling; above it, warnings resume as normal.
# 50GB=51200 25GB=25600 20GB=20480 15GB=15360 10GB=10240 5GB=5120
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
["Tdarr"]="25600" # 25GB — transcode cache grows legitimately during active jobs
["7dtd"]="20480" # 20GB — game server world data, expected to be large
)
# ━━━ Docker Network Connect ━━━
# Containers connected to custom networks at array start by docker_network_connect.sh.
# Networks created if they don't exist — idempotent, safe to re-run.
@@ -354,7 +373,7 @@
# ━━━ Fallback Tiers — HOST1 Runs for HOST2 ━━━
# Containers HOST1 starts when HOST2 goes down.
# Tier 1 is always immediate — vital services cannot wait.
# Higher tiers activate after HOST2_TIER*_DELAY minutes (set in master_host2.conf).
# Higher tiers activate after HOST2_TIER*_DELAY minutes (set in host2.conf).
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
"Gmer4Lfe.us"
"VaultWarden-Jayred365"
@@ -494,13 +513,13 @@
# ==============================================================================================
# Ramdisk size ceiling — tmpfs only uses RAM actually needed, not the full size upfront.
# Real-world: 9 streams peaked at ~5.5GB — 8G gives comfortable headroom on 128GB RAM.
HOST1_RAMDISK_SIZE="8G"
# Real-world: 9 streams peaked at ~5.5GB — 10G gives generous headroom on 128GB RAM.
HOST1_RAMDISK_SIZE="10G"
# Usage thresholds — coupled to HOST1_RAMDISK_SIZE, adjust all three together if size changes.
# Hysteresis gap (6.8 - 5.5 = 1.3GB) prevents flip-flop between ramdisk and SSD.
HOST1_RAMDISK_WARN_GB=6.8 # flip to SSD when ramdisk usage reaches this
HOST1_RAMDISK_LOW_GB=5.5 # flip back to ramdisk when usage drops to this
# Hysteresis gap (8.5 - 7 = 1.5GB) prevents flip-flop between ramdisk and SSD.
HOST1_RAMDISK_WARN_GB=8.5 # flip to SSD when ramdisk usage reaches this
HOST1_RAMDISK_LOW_GB=7 # flip back to ramdisk when usage drops to this
# SSD fallback path — where transcodes land when ramdisk exceeds HOST1_RAMDISK_WARN_GB.
# Must be on cache pool — array disks too slow for active transcode writes.
+36 -6
View File
@@ -10,7 +10,7 @@
# HOST1 never sees HOST2 credentials — clean separation at the file level.
#
# DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf.
# DO NOT put HOST1 variables here — they belong in master_host1.conf.
# DO NOT put HOST1 variables here — they belong in host1.conf.
#
# ── STATUS ────────────────────────────────────────────────────────────────────────────────────
# HOST2 is currently being rebuilt — most sections scaffolded, fill in when back online.
@@ -61,6 +61,12 @@
# RADARR URL, API key, path map
# ARR RECOVERY per-arr recovery toggles (no Lidarr on HOST2)
#
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
# SYSTEM WATCHDOG per-host check toggles and NIC configuration
#
# ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
# RESOURCE MANAGER containers paused/stopped under memory pressure
#
# ==============================================================================================
# ==============================================================================================
@@ -72,6 +78,8 @@
# SSH key used for all server-to-server operations — rsync, failover container commands.
# Must be in /root/.ssh/ and authorised in HOST1's /root/.ssh/authorized_keys.
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"
HOST2_OWNER="jayred365"
HOST2_OWNER_EMAIL="" # fill in when HOST2 is back online
# ━━━ Emby ━━━
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
@@ -284,6 +292,14 @@
# ["Authelia"]="Mariadb-Authelia Redis-Authelia"
)
# Per-container appdata growth suppress ceilings in MB.
# ONLY needed in specific cases — growth rate detection covers all containers automatically.
# Use when a container legitimately has large stable data and you want to suppress false-positive
# growth alerts. Add entries here only when a container triggers warnings it shouldn't.
declare -A HOST2_WATCHDOG_APPDATA_SIZES=(
# add HOST2 suppress entries here only as needed
)
# ━━━ Docker Network Connect ━━━
# Containers connected to custom networks at array start by docker_network_connect.sh.
# Networks created if they don't exist — idempotent, safe to re-run.
@@ -318,7 +334,7 @@
# ━━━ Fallback Tiers — HOST2 Runs for HOST1 ━━━
# Containers HOST2 starts when HOST1 goes down.
# Tier 1 is always immediate — vital services cannot wait.
# Higher tiers activate after HOST1_TIER*_DELAY minutes (set in master_host1.conf).
# Higher tiers activate after HOST1_TIER*_DELAY minutes (set in host1.conf).
FALLBACK_HOST2_COVERS_HOST1_TIER1=(
"Gmer4Lfe.com"
"Emby"
@@ -501,10 +517,6 @@
HOST2_RADARR_RECOVERY=true
# HOST2_LIDARR_RECOVERY not set — Lidarr does not run on HOST2
# ==============================================================================================
# ──────────────────────── End Of HOST2 Variables ──────────────────────────────────────────────
# ==============================================================================================
# ==============================================================================================
# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
# ==============================================================================================
@@ -597,6 +609,24 @@
# DISABLED — rebuild workloads may legitimately peg CPU. Enable after rebuild.
HOST2_SYS_WATCHDOG_CHECK_RUNAWAY=false
# ==============================================================================================
# ── RESOURCE MANAGER ──────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Containers to manage under pressure — see master.conf RW_CRITICAL_CONTAINERS for exclusions.
# docker pause at medium pressure (RAM < RW_RAM_MEDIUM_GB or load > medium threshold)
# Suspended in-place — instant to pause/unpause, no state lost, no restart delay.
HOST2_RW_PAUSE_CONTAINERS=(
# fill in when HOST2 is back online
)
# docker stop at hard pressure (RAM < RW_RAM_HARD_GB)
# Full stop — these are optional/heavy services that free significant RAM when stopped.
# resource_watchdog.sh restarts them when pressure fully clears (RAM >= RW_RAM_RECOVER_GB).
HOST2_RW_STOP_CONTAINERS=(
# fill in when HOST2 is back online
)
# ==============================================================================================
# ──────────────────────── End Of HOST2 Variables ──────────────────────────────────────────────
# ==============================================================================================
+11 -11
View File
@@ -7,23 +7,23 @@
#
# ── HOW IT WORKS ──────────────────────────────────────────────────────────────────────────────
# 1. Sources master.conf (shared config — hostnames, thresholds, toggles, profiles, job lists)
# 2. Auto-discovers and sources all master_host*.conf files in the same directory
# 2. Auto-discovers and sources all host*.conf files in the same directory
# Each host conf extends the shared profile arrays and adds host-specific credentials
# 3. Sources common.sh (shared functions — detect_hosts, logging, notifications etc.)
#
# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────
# Without this loader every script had to explicitly source each conf file:
# source master.conf
# source master_host1.conf
# source master_host2.conf
# source host1.conf
# source host2.conf
# source common.sh
#
# Adding a new server meant updating every script.
# With this loader — add master_host3.conf to the git repo and every server
# With this loader — add host3.conf to the git repo and every server
# auto-discovers it on next git pull. Zero script changes required. Ever.
#
# ── ADDING A NEW SERVER ───────────────────────────────────────────────────────────────────────
# 1. Create master_host3.conf following the same structure as HOST1/HOST2
# 1. Create host3.conf following the same structure as HOST1/HOST2
# 2. Commit and push to git repo
# 3. All servers pull it automatically — no other changes needed
#
@@ -40,9 +40,9 @@
# source "$SCRIPT_DIR/load_config.sh"
#
# ── SPARSE CHECKOUT NOTE ──────────────────────────────────────────────────────────────────────
# Sparse checkout controls which master_host*.conf files each server receives.
# HOST1 only pulls master_host1.conf — never HOST2's credentials.
# HOST2 only pulls master_host2.conf — never HOST1's credentials.
# Sparse checkout controls which host*.conf files each server receives.
# HOST1 only pulls host1.conf — never HOST2's credentials.
# HOST2 only pulls host2.conf — never HOST1's credentials.
# This loader sources whatever conf files ARE present — sparse checkout handles the rest.
# Both servers pull all non-credential conf files (master.conf, common.sh, load_config.sh).
#
@@ -63,14 +63,14 @@
fi
source "$LOAD_CONFIG_DIR/master.conf"
# ━━━ Auto-discover and source all master_host*.conf files ━━━
# ━━━ Auto-discover and source all host*.conf files ━━━
# Sorted for consistent load order — HOST1 before HOST2 before HOST3 etc.
# Each host conf extends the shared PROFILE_* arrays and adds host-specific vars.
# Missing files are silently skipped — sparse checkout intentionally withholds some.
# At least one host conf must be present or the ecosystem has no identity to work with.
_host_confs_loaded=0
for _conf in $(ls "$LOAD_CONFIG_DIR"/master_host*.conf 2>/dev/null | sort); do
for _conf in $(ls "$LOAD_CONFIG_DIR"/host*.conf 2>/dev/null | sort); do
if [[ -f "$_conf" ]]; then
source "$_conf"
(( _host_confs_loaded++ ))
@@ -80,7 +80,7 @@
done
if [[ "$_host_confs_loaded" -eq 0 ]]; then
echo "[FATAL] No master_host*.conf files found in $LOAD_CONFIG_DIR" >&2
echo "[FATAL] No host*.conf files found in $LOAD_CONFIG_DIR" >&2
echo "[FATAL] At least one host conf required — check git pull and sparse checkout" >&2
exit 1
fi
+76 -51
View File
@@ -8,10 +8,10 @@
# ── HOW THE THREE-FILE SYSTEM WORKS ──────────────────────────────────────────────────────────
# Scripts source all three files at startup:
# source master.conf ← shared config (this file)
# source master_host1.conf ← HOST1 credentials, shares, container lists
# source master_host2.conf ← HOST2 credentials, shares, container lists
# source host1.conf ← HOST1 credentials, shares, container lists
# source host2.conf ← HOST2 credentials, shares, container lists
#
# Sparse checkout (git) ensures each server only pulls its own master_host*.conf.
# Sparse checkout (git) ensures each server only pulls its own host*.conf.
# HOST2 never sees HOST1 credentials. HOST1 never sees HOST2 credentials.
#
# What belongs here: thresholds, toggles, intervals, profiles, job lists
@@ -99,7 +99,7 @@
# Must match the exact unRAID hostname AND Tailscale device name (case sensitive).
# detect_hosts() in common.sh matches the local hostname against these to set MY_ID / REMOTE_ID.
# Tailscale IP resolution uses these names — no hardcoded IPs needed.
# To add a new server: add HOST3="unRAID-NewServer" here + create master_host3.conf.
# To add a new server: add HOST3="unRAID-NewServer" here + create host3.conf.
HOST1="unRAID-Gmer4Lfe"
HOST2="unRAID-Jayred365"
@@ -145,7 +145,7 @@
# Manages the relationship lifecycle between two unRAID servers.
# HOST1 is always the owner (source of truth) — HOST2 is always the mirror.
# PARTNERSHIP_OWNER_HOST flips to "HOST2" after a --transfer operation.
# All identity vars (hostnames, SSH keys) live in master_host*.conf.
# All identity vars (hostnames, SSH keys) live in host*.conf.
# Hostnames already match Tailscale device names — IP resolution is automatic.
#
# State files on /boot/config — survives reboots, available before array starts:
@@ -164,13 +164,13 @@
PARTNERSHIP_ENABLED=false
PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer
# Auth containers reconfigured on onboard/offboard — defined per host in master_host*.conf.
# Auth containers reconfigured on onboard/offboard — defined per host in host*.conf.
# Format: "ContainerName|WebUIPort"
# HOST1_PARTNERSHIP_AUTH_WEBUIS / HOST2_PARTNERSHIP_AUTH_WEBUIS
# On onboard → WebUI pointed at owner's Tailscale IP
# On offboard → WebUI pointed back at localhost
# Paths to collect during the grace window after offboard — defined per host in master_host*.conf.
# Paths to collect during the grace window after offboard — defined per host in host*.conf.
# HOST1_PARTNERSHIP_MIRROR_BACKUPS / HOST2_PARTNERSHIP_MIRROR_BACKUPS
# Notified on offboard — no auto-deletion, manual collection.
@@ -219,13 +219,6 @@
# ── LOGGING ───────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Silent-by-default output model — ecosystem only speaks when something is wrong.
# true = only warn() and error() produce output (default — reduces notification spam)
# false = all output visible — use for monitor scripts or debugging
# Override per-run: script --log sets ENABLE_LOGGING=true for [LOG] detail
# Monitor scripts (coffee_report, health_digest etc.) set SILENT_MODE=false themselves
SILENT_MODE=true
# Controls verbose [LOG] output across all scripts.
# true = show detailed [LOG] lines — useful for debugging or first-time setup
# false = show only user-facing output — cleaner for scheduled runs
@@ -239,7 +232,7 @@
# normal = job completed successfully / warning = something failed or needs attention
NOTIFY_UNRAID=true
# Discord webhook URL — defined per host in master_host*.conf.
# Discord webhook URL — defined per host in host*.conf.
# HOST1_DISCORD_WEBHOOK / HOST2_DISCORD_WEBHOOK
# Allows different webhooks per server, or only one server notifying.
@@ -284,7 +277,6 @@
# Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via
# watchdog_orchestrator.sh — NOT launched here.
ARRAY_START_SCRIPTS=(
"git_pull_execute.sh" # pull latest scripts before anything starts
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
@@ -298,11 +290,12 @@
# Schedule: * * * * * (every minute)
# NOT in ARRAY_START_SCRIPTS — has its own cron entry.
# Order matters — resource first (frees pressure), docker second (heals with freed resources),
# system last (reboots only if prior layers failed).
# storage third (pool/data health — never reboots), system last (last line of defense).
WATCHDOG_ORCHESTRATOR_SCRIPTS=(
"unRAID_Essentials/resource_watchdog.sh" # reduce system pressure before healing attempts
"Docker_Essentials/docker_watchdog.sh" # heal containers with freed resources
"unRAID_Essentials/system_watchdog.sh" # reboot if all else fails — last line of defense
"Watchdogs/resource_watchdog.sh" # reduce system pressure before healing attempts
"Watchdogs/docker_watchdog.sh" # heal containers with freed resources
"Watchdogs/storage_watchdog.sh" # pool and appdata health — alert and remediate
"Watchdogs/system_watchdog.sh" # reboot if all else fails — last line of defense
)
WATCHDOG_ORCHESTRATOR_HEARTBEAT=true
@@ -317,7 +310,7 @@
"Docker_Essentials/downloaders_reset.sh" # clear stuck download states every 15min
)
# Shares synced every 15 minutes — defined per host in master_host*.conf.
# Shares synced every 15 minutes — defined per host in host*.conf.
# HOST1_CRITICAL_SYNC_SHARES / HOST2_CRITICAL_SYNC_SHARES
# Format: "/path/to/share" or "/path/to/share|profile-name"
# Order matters — Critical-Data first (auth stack), then Emby dirty sync.
@@ -326,7 +319,7 @@
# intermediate_sync_maintenance.sh runs every 4 hours — arr library sync, artwork fetch,
# and optional mid-day rsync for any shares that need sub-daily propagation.
# Schedule: 0 */4 * * *
# INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in master_host*.conf.
# INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in host*.conf.
INTERMEDIATE_RSYNC_ENABLED=true # set false to disable mid-day rsync without removing shares
INTERMEDIATE_MAINTENANCE_SCRIPTS=(
@@ -346,7 +339,7 @@
"Media/media_cleaner.sh media" # remove junk from media shares
#"Media/lidarr_cleanup.sh" # remove orphaned music files — enable when ready
#"Media/sonarr_cleanup.sh" # remove orphaned TV files — enable when ready
#"Media/radarr_cleanup.sh" # remove orphaned movie files — enable when ready
"Media/radarr_cleanup.sh" # remove orphaned movie files
"Media/lidarr_missing_art.sh" # fetch missing album/artist artwork (HOST1 only — self-guards)
"Media/radarr_tmdb_removed.sh" # remove movies dropped from TMDb
"Media/sonarr_tvdb_removed.sh" # remove series dropped from TVDB
@@ -360,7 +353,7 @@
DAILY_CONTAINER_UPDATES=true
# Media shares synced daily by daily_sync_maintenance.sh.
# Defined per-host in master_host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
# Defined per-host in host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
# Mesh model: every node pushes every media share. rsync has no --delete so pushes are additive.
# arr_sync (union) ensures all arr libraries converge first. arr_cleanup removes true orphans.
# Any node can download content to any share — it propagates to all nodes on the next cycle.
@@ -368,7 +361,7 @@
# These shares use DEFAULT_RSYNC_OPTS — no profile entry needed.
# For shares needing custom options or container stops — create a profile in RSYNC section.
# Personal encrypted shares defined per-host in master_host*.conf.
# Personal encrypted shares defined per-host in host*.conf.
# ZFS encrypted at dataset level — remote receives encrypted blocks, cannot read content.
# See README-Rsync_Setup.md for ZFS encryption setup before uncommenting.
@@ -390,7 +383,7 @@
# Set false to skip — docker_weekly_restart.sh still runs regardless.
WEEKLY_REMAINING_UPDATES=true
# Shares synced during the weekly maintenance window — defined per host in master_host*.conf.
# Shares synced during the weekly maintenance window — defined per host in host*.conf.
# HOST1_WEEKLY_SYNC_SHARES / HOST2_WEEKLY_SYNC_SHARES
# Containers stopped both sides before sync — full clean state guaranteed.
# Profiles drive container stops, excludes, and options — configure in RSYNC section.
@@ -473,8 +466,8 @@
# called by weekly_sync_maintenance.sh — full clean sync weekly
# critical-fallback — dirty sync — auth stays running both sides, WAL excluded
# called by critical_sync_maintenance.sh every 15min
# host1-appdata — HOST1 server-specific appdata — defined in master_host1.conf
# host2-appdata — HOST2 server-specific appdata — defined in master_host2.conf
# host1-appdata — HOST1 server-specific appdata — defined in host1.conf
# host2-appdata — HOST2 server-specific appdata — defined in host2.conf
# important-data — NextCloud + Postgres — NextCloud delayed start after Postgres
# emby — weekly clean sync — both Emby stopped, full mirror
# called by weekly_sync_maintenance.sh only — do NOT schedule separately
@@ -591,7 +584,7 @@
# Failover → start remote DDNS first (Tier 1)
# Handback → stop remote DDNS → rsync → start containers → start local DDNS last
#
# Per-host container lists and tier delays live in master_host*.conf.
# Per-host container lists and tier delays live in host*.conf.
# Shared settings (intervals, state file, thresholds) live here.
EXTERNAL_IP="8.8.8.8"
@@ -613,7 +606,7 @@
# ━━━ Downloaders Reset ━━━
# Runs every 15 minutes via CRITICAL_MAINTENANCE_SCRIPTS.
# Clears stuck states, purges old history, prepares each download client for a clean cycle.
# Per-host URLs and API keys live in master_host*.conf.
# Per-host URLs and API keys live in host*.conf.
DOWNLOADER_RETENTION_DAYS=7 # days — purge history older than this
# qBittorrent failsafe — removes torrents older than threshold regardless of ratio
@@ -623,7 +616,7 @@
# ━━━ Docker Daily Restart ━━━
# Containers restarted every day via DAILY_MAINTENANCE_SCRIPTS.
# Per-host lists live in master_host*.conf:
# Per-host lists live in host*.conf:
# HOST1_DAILY_RESTART_CONTAINERS
# HOST2_DAILY_RESTART_CONTAINERS
# detect_hosts() sets DAILY_RESTART_CONTAINERS to the correct host array at runtime.
@@ -631,7 +624,7 @@
# ━━━ Docker Weekly Restart ━━━
# Less critical services restarted weekly via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am).
# Containers already stopped for weekly sync — restart adds zero extra downtime.
# Per-host lists live in master_host*.conf:
# Per-host lists live in host*.conf:
# HOST1_WEEKLY_RESTART_CONTAINERS
# HOST2_WEEKLY_RESTART_CONTAINERS
@@ -653,7 +646,7 @@
# Dead containers — remove and restart
# Unexpected exits — non-zero exit code → restart
#
# All per-host container lists live in master_host*.conf:
# All per-host container lists live in host*.conf:
# HOST*_WATCHDOG_CONTAINERS — memory hard limits per container
# HOST*_WATCHDOG_CONTAINER_URLS — HTTP health check URLs
# HOST*_WATCHDOG_REQUIRED_CONTAINERS — must always be running
@@ -706,10 +699,42 @@
# Notification batching — one summary per cycle instead of one ping per event
WATCHDOG_BATCH_NOTIFY=true
# Appdata size monitoring — two-part catch-all for runaway growth and oversized log files.
#
# Part 1 — Growth rate (zero-config):
# Reads per-container dir totals each cycle via du, compares to previous cycle.
# Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused *.log scan
# inside that container. No per-container config required — new containers are covered
# automatically. Baseline built on first run after boot; growth detection starts cycle 2.
#
# Part 2 — Absolute log size:
# Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB across all appdata paths.
# Catches logs that have already stabilised at a large size and are no longer actively growing.
#
# Strike system (reuses existing watchdog infrastructure):
# Strike 1 — warn + notify: condition first detected
# Strike 2 — warn + escalated notify: still present next cycle
# Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
# If WATCHDOG_APPDATA_TRUNCATE_LOGS=true: truncate *.log files in-place, clear strikes
# If false: critical notify only, strikes held until condition resolves
# Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
#
# HOST*_WATCHDOG_APPDATA_SIZES (in host*.conf) suppresses growth warnings for a
# container until its dir exceeds the configured ceiling. Only needed when a container
# legitimately has large stable data and you want to guarantee it never triggers a false alarm.
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2 # flag containers growing more than this per cycle
WATCHDOG_APPDATA_LOG_MAX_GB=2 # flag *.log files exceeding this size (absolute)
WATCHDOG_APPDATA_TRUNCATE_LOGS=false # set true to auto-truncate oversized *.log files on action cycle
WATCHDOG_APPDATA_STRIKE_LIMIT=3 # cycles before action fires (matches existing watchdog pattern)
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db" # /tmp — resets on reboot ✅
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db" # /tmp — resets on reboot ✅
# ━━━ Docker Network Connect ━━━
# Ensures custom networks exist and connects containers at array start.
# Runs once via ARRAY_START_SCRIPTS — idempotent, safe to re-run.
# Container and network lists are host-specific — defined in master_host*.conf:
# Container and network lists are host-specific — defined in host*.conf:
# HOST1_NETWORK_CONNECT_CONTAINERS / HOST2_NETWORK_CONNECT_CONTAINERS
# HOST1_NETWORK_CONNECT_NETWORKS / HOST2_NETWORK_CONNECT_NETWORKS
@@ -816,13 +841,13 @@
PERMISSIONS_FILE_MODE="664" # files — group read/write, no execute
PERMISSIONS_OWNER="nobody:users"
# Share list defined per host in master_host*.conf:
# Share list defined per host in host*.conf:
# HOST1_MEDIA_PERMISSION_SHARES / HOST2_MEDIA_PERMISSION_SHARES
# ━━━ Media Cleaner ━━━
# Removes junk files from media shares — two profiles: anime and media.
# Called via DAILY_MAINTENANCE_SCRIPTS. Run manually: Media/media_cleaner.sh anime|media
# Folder lists defined per host in master_host*.conf:
# Folder lists defined per host in host*.conf:
# HOST1_ANIME_CLEAN_FOLDERS / HOST2_ANIME_CLEAN_FOLDERS
# HOST1_MEDIA_CLEAN_FOLDERS / HOST2_MEDIA_CLEAN_FOLDERS
@@ -904,7 +929,7 @@
# ━━━ Arr Cleanup ━━━
# Orphan file cleanup via Lidarr, Sonarr, and Radarr APIs.
# Per-host URLs, API keys, and path maps live in master_host*.conf.
# Per-host URLs, API keys, and path maps live in host*.conf.
# detect_hosts() selects correct host vars at runtime.
#
# API versions — update MAJOR version here when script is updated for a new arr version:
@@ -940,7 +965,7 @@
LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs
LIDARR_ART_RETRIES=2 # download retry attempts per image
LIDARR_ART_SLEEP_BETWEEN=0.2 # seconds between fanart.tv API calls
# HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in master_host*.conf
# HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in host*.conf
# Lidarr discovery settings (playback_aware_lidarr_discovery.sh)
LIDARR_DISCOVERY_THRESHOLD=70 # score to accept candidate (0-100)
@@ -1037,7 +1062,7 @@
# stalled — download stuck with no connections or progress
#
# Items newer than ARR_IMPORT_RECOVERY_AGE are skipped — gives arr time to retry first.
# Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in master_host*.conf.
# Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in host*.conf.
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
# matches cron interval — items eligible after one missed cycle
@@ -1057,7 +1082,7 @@
# Must exist before Emby starts so the symlink resolves correctly.
RAMDISK_PATH="/mnt/ramdisk_transcodes"
# Ramdisk size and flip thresholds — defined per host in master_host*.conf.
# Ramdisk size and flip thresholds — defined per host in host*.conf.
# All three are coupled — if size changes, thresholds must change with it.
# HOST1_RAMDISK_SIZE / HOST2_RAMDISK_SIZE
# HOST1_RAMDISK_WARN_GB / HOST2_RAMDISK_WARN_GB ← flip to SSD at this usage
@@ -1068,7 +1093,7 @@
# Must match the container path configured in Emby's Extra Parameters.
TRANSCODE_LINK="/mnt/ram-transcode"
# SSD fallback location — defined per host in master_host*.conf (cache path differs per server):
# SSD fallback location — defined per host in host*.conf (cache path differs per server):
# HOST1_TRANSCODE_SSD / HOST2_TRANSCODE_SSD
# Minimum free GB on SSD before allowing flip from ramdisk to SSD.
@@ -1098,7 +1123,7 @@
# Format: "ContainerName|URL|APIKey|Type" — Type: emby | jellyfin | plex
# Entries with placeholder API keys are skipped automatically.
# ⚠️ Tdarr does NOT belong here — keep Tdarr on SSD, not ramdisk.
# Defined per host in master_host*.conf — Emby container names and keys differ per server:
# Defined per host in host*.conf — Emby container names and keys differ per server:
# HOST1_TRANSCODE_SERVERS / HOST2_TRANSCODE_SERVERS
# ==============================================================================================
@@ -1108,7 +1133,7 @@
# ━━━ Certificate Monitor ━━━
# Checks SSL certificate expiry via direct openssl connection — no NPM dependency.
# Checks the actual certificate served by each domain, not what NPM thinks it has.
# Domains defined per host in master_host*.conf — each server monitors its own domains:
# Domains defined per host in host*.conf — each server monitors its own domains:
# HOST1_CERT_MONITOR_DOMAINS / HOST2_CERT_MONITOR_DOMAINS
CERT_WARN_DAYS=30 # warn when cert expires within this many days
CERT_CRIT_DAYS=7 # critical alert within this many days
@@ -1117,7 +1142,7 @@
# ━━━ Backup Verify ━━━
# Verifies rsync mirror health by comparing random file checksums between servers.
# Catches silent corruption or incomplete syncs that rsync itself wouldn't detect.
# Defined per host in master_host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically:
# Defined per host in host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically:
# HOST1_BACKUP_VERIFY_SHARES / HOST2_BACKUP_VERIFY_SHARES
BACKUP_VERIFY_SAMPLE=10 # random files to check per share
BACKUP_VERIFY_MIN_SIZE=1M # minimum file size to include in sample
@@ -1128,7 +1153,7 @@
# (hot/max/hotssd/maxssd) — these vars are fallback only if dynamix.cfg not found.
SMART_TEMP_WARN=45 # fallback — Celsius warn threshold
SMART_TEMP_CRIT=55 # fallback — Celsius critical threshold
# Drives to ignore defined per host in master_host*.conf — hardware is server-specific:
# Drives to ignore defined per host in host*.conf — hardware is server-specific:
# HOST1_SMART_IGNORE_DRIVES / HOST2_SMART_IGNORE_DRIVES
# ━━━ ZFS Memory Snapshot ━━━
@@ -1138,7 +1163,7 @@
ZFS_REPORT_FREE_WARN_GB=10 # warn if less than this GB free RAM
ZFS_REPORT_AVAIL_WARN_GB=20 # warn if less than this GB available on ZFS pool
ZFS_REPORT_DOCKER_TOP=10 # how many top Docker containers to show by memory
# Pool ignore list defined per host in master_host*.conf — pool names are server-specific:
# Pool ignore list defined per host in host*.conf — pool names are server-specific:
# HOST1_ZFS_REPORT_IGNORE_POOLS / HOST2_ZFS_REPORT_IGNORE_POOLS
# ━━━ Bandwidth Monitor ━━━
@@ -1168,7 +1193,7 @@
# ━━━ Emby Session Report ━━━
# Weekly Emby usage statistics via API — no persistent writes, queries fresh each run.
# URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in master_host*.conf.
# URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in host*.conf.
EMBY_REPORT_DAYS=7 # days to include in the report period
EMBY_REPORT_TOP_N=10 # number of top content items to show
@@ -1184,8 +1209,8 @@
# Level 3 (hard) — docker stop optional containers, signal docker_watchdog to defer
#
# ── PER-HOST CONTAINER LISTS ──────────────────────────────────────────────────────────────────
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (in master_host*.conf)
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (in master_host*.conf)
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (in host*.conf)
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (in host*.conf)
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
@@ -1335,9 +1360,9 @@
SYS_WATCHDOG_MDSTAT_ERROR_LIMIT=5 # new errors in one cycle before acting
# ━━━ Check Toggles ━━━
# Per-host — moved to master_host*.conf
# Per-host — moved to host*.conf
# Different servers may have different hardware, NICs, and check requirements
# See HOST*_SYS_WATCHDOG_CHECK_* in master_host*.conf
# See HOST*_SYS_WATCHDOG_CHECK_* in host*.conf
# ━━━ Abort Toggles ━━━
# Conditions that prevent reboot even when a threshold is hit.
+27 -251
View File
@@ -4,13 +4,15 @@ Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start.
> **Watchdog scripts have moved.** `system_watchdog.sh` and `resource_watchdog.sh`
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
---
## ━━━ CONTENTS ━━━
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
- [system_watchdog.sh](#system_watchdogsh)
- [resource_watchdog.sh](#resource_watchdogsh)
- [webgui_restart.sh](#webgui_restartsh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
@@ -25,6 +27,22 @@ adding or reordering scripts at array start.
---
## Output Tiers
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
**Daemon scripts** (`webgui_restart.sh`): run on every cycle. Without `--log`, only
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
Per-check detail suppressed.
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
final summary are visible. Per-item detail suppressed.
---
## ARRAY_START_SCRIPTS Order
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
@@ -43,8 +61,9 @@ ARRAY_START_SCRIPTS=(
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
...
"system_watchdog.sh" # LAST or near-last — starts background loop
)
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every minute via
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
```
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
@@ -57,151 +76,6 @@ generates veth messages — these will appear in syslog if the filter isn't acti
---
## system_watchdog.sh
### Three-Tier Response
The watchdog categorizes every failure into one of three tiers:
**Tier 1 — CRITICAL (immediate reboot, no strikes)**
| Condition | Threshold | Why immediate |
|-----------|-----------|---------------|
| Docker daemon unresponsive | N/A | Nothing can be healed; every docker command hangs |
| rootfs usage | SYS_WATCHDOG_ROOTFS_CRITICAL_PCT (99%) | SSH stops; state files fail silently |
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
| File descriptor exhaustion | SYS_WATCHDOG_FD_CRITICAL_PCT (95%) | New connections silently failing |
| /boot read-only unexpectedly | write test fails | Config writes silently failing |
**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
RAM below MEM_GB AND OOM kills this cycle >= SYS_WATCHDOG_OOM_LIMIT.
Both conditions required — RAM alone without OOM uses the standard strike system.
OOM confirms the system is dying faster than watchdogs can heal.
**Tier 3 — STANDARD (SYS_WATCHDOG_STRIKES consecutive failures → reboot)**
| Check | Threshold |
|-------|-----------|
| Free RAM | MEM_WARN_GB → MEM_SHUTDOWN_GB → MEM_GB |
| Load average | SYS_WATCHDOG_LOAD_MULTIPLIER × cpu_count |
| CPU temperature | SYS_WATCHDOG_CPU_TEMP |
| Zombie processes | SYS_WATCHDOG_ZOMBIES |
| /var/log usage | SYS_WATCHDOG_VAR_LOG_PCT |
| /tmp usage | SYS_WATCHDOG_TMP_PCT |
| Array disk errors | mdstat error delta > 0 |
| NIC state | interface operstate != "up" |
| Required containers | containers in SYS_WATCHDOG_REQUIRED_CONTAINERS |
### RAM Tiers
```
MEM_WARN_GB (10GB) → warn + notify, no action
MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery
MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM)
MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart
```
At MEM_SHUTDOWN_GB, all containers NOT listed in
`SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED` are stopped. Excluded containers by default:
NginxProxyManager, Authelia, Mariadb, Redis, Emby, Dispatcharr. Adjust in
master.conf for your critical services.
### Abort Conditions
These conditions prevent a reboot — running them would cause data loss:
```bash
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
SYS_WATCHDOG_ABORT_ON_PARITY=true # Parity check/rebuild running
SYS_WATCHDOG_ABORT_ON_MOVER=true # Mover running
```
CRITICAL tier bypasses all abort conditions — an imminent crash outweighs
data safety concerns.
### Strike System
The strike count tracks consecutive failures. A single recovery resets strikes
to 0. The reboot only fires after SYS_WATCHDOG_STRIKES consecutive failures on
the same check — transient spikes (a brief load burst, a momentary RAM dip) don't
trigger reboots.
### Reboot Rate Limit
```bash
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # window in hours
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots within the window
```
If the server has rebooted SYS_WATCHDOG_MAX_REBOOTS times within the window,
system_watchdog stops rebooting and notifies instead. This prevents a boot loop
where the watchdog reboots → something crashes again immediately → reboot again.
### Usage
```bash
system_watchdog.sh # start continuous monitoring loop
system_watchdog.sh --dry-run # run detection logic without rebooting
system_watchdog.sh --status # show thresholds, current state, strike counts
system_watchdog.sh --log # verbose per-cycle output
```
### Verify Running
```bash
pgrep -a -f system_watchdog.sh
# Expected: shows PID and path
```
---
## resource_watchdog.sh
### Pressure Levels
```bash
# master.conf
RW_RAM_SOFT_GB=20 # Level 1 trigger — throttle downloaders
RW_RAM_MEDIUM_GB=15 # Level 2 trigger — throttle + pause containers
RW_RAM_HARD_GB=10 # Level 3 trigger — stop containers
RW_RAM_RECOVER_GB=25 # recover to this before un-stopping at level 3
RW_LOAD_SOFT_MULTIPLIER=2.0 # load > 2× cpu count = level 1
RW_LOAD_MEDIUM_MULTIPLIER=3.0 # load > 3× cpu count = level 2
RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalating
```
### Per-Host Container Lists
```bash
# master_host1.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at medium pressure
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at hard pressure
```
Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
pressure level. Default includes: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
### docker_watchdog Coordination
At level 3, resource_watchdog writes `mem_shutdown_active=true` to `RW_STATE_FILE`.
docker_watchdog.sh reads this flag each cycle and skips all container restart logic
while it is set. Without this coordination, docker_watchdog would immediately
restart containers that resource_watchdog just stopped to free RAM.
The flag is cleared when level 3 pressure resolves and containers are restarted.
### Usage
```bash
resource_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
resource_watchdog.sh --dry-run # show what would be throttled/paused/stopped
resource_watchdog.sh --status # current level, active actions, recovery cycle count
resource_watchdog.sh --log # verbose per-check output
```
---
## webgui_restart.sh
### Escalation Logic
@@ -578,75 +452,12 @@ server_reboot.sh --reason="disk work" # include reason in notification
## Full Configuration Reference
> Watchdog configuration (`system_watchdog.sh`, `resource_watchdog.sh`,
> `docker_watchdog.sh`, `storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
```bash
# master.conf
# ── System Watchdog ────────────────────────────────────────────────────────────
SYS_WATCHDOG_INTERVAL=300 # seconds between check cycles
SYS_WATCHDOG_STRIKES=3 # consecutive failures before reboot
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # rate limit window
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots in window
SYS_WATCHDOG_OOM_LIMIT=3 # OOM kills/cycle for URGENT bypass
MEM_WARN_GB=10 # warn + notify
MEM_SHUTDOWN_GB=6 # stop non-essential containers
MEM_GB=4 # strike → reboot
MEM_RECOVER_GB=30 # recovery threshold
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99 # Tier 1 trigger
SYS_WATCHDOG_FD_CRITICAL_PCT=95 # Tier 1 trigger
SYS_WATCHDOG_LOAD_MULTIPLIER=4 # Tier 3 — ×cpu_count
SYS_WATCHDOG_CPU_TEMP=85 # Tier 3 — Celsius
SYS_WATCHDOG_ZOMBIES=20 # Tier 3 — process count
SYS_WATCHDOG_VAR_LOG_PCT=80 # Tier 3 — percent full
SYS_WATCHDOG_TMP_PCT=85 # Tier 3 — percent full
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=true
SYS_WATCHDOG_ABORT_ON_MOVER=true
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
"NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
)
SYS_WATCHDOG_REQUIRED_CONTAINERS=() # containers that must be running
# State files:
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
SYS_WATCHDOG_REBOOT_LOG="/tmp/sys_watchdog_reboots.db"
SYS_WATCHDOG_FAILED_FILE="/tmp/sys_watchdog_failed.db"
SYS_WATCHDOG_OOM_FILE="/tmp/sys_watchdog_oom.db"
# ── Resource Watchdog ──────────────────────────────────────────────────────────
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
RW_RAM_SOFT_GB=20
RW_RAM_MEDIUM_GB=15
RW_RAM_HARD_GB=10
RW_RAM_RECOVER_GB=25
RW_LOAD_SOFT_MULTIPLIER=2.0
RW_LOAD_MEDIUM_MULTIPLIER=3.0
RW_RECOVER_CYCLES=3
RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"
RW_SABNZBD_SPEED_MEDIUM="10M"
RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200 # KB/s
RW_QBIT_DL_MEDIUM=10240
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
# ── Per-Host (master_host*.conf) ───────────────────────────────────────────────
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
@@ -683,43 +494,8 @@ REBOOT_VM_WAIT=30
## Troubleshooting
### system_watchdog.sh Not Starting
```bash
# Check if already running (acquire_lock prevents second instance):
pgrep -a -f system_watchdog.sh
# Check state file permissions:
ls -la /tmp/sys_watchdog_*.db
# Run with --log to see startup:
system_watchdog.sh --log
```
### system_watchdog Rebooted Unexpectedly
```bash
# Check reboot log:
cat /tmp/sys_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot
# Check what condition triggered it:
# Look in /var/log/syslog for "system_watchdog" near the reboot time
grep "system_watchdog" /var/log/syslog | tail -20
```
### resource_watchdog Paused Containers It Shouldn't Have
```bash
# Check current state:
resource_watchdog.sh --status
# Add the container to RW_CRITICAL_CONTAINERS in master.conf:
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
# Un-pause manually if needed:
docker unpause MyContainer
```
> Watchdog troubleshooting (system_watchdog, resource_watchdog, docker_watchdog,
> storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
### rsync_stop Killed the Wrong Thing
+14 -32
View File
@@ -71,8 +71,6 @@ SIGTERM (graceful — finishes current file), SIGKILL only if needed.
## ━━━ WHAT THIS FOLDER DOES ━━━
```
Last-resort stability system_watchdog.sh — reboots before crash
Pressure reduction resource_watchdog.sh — throttle/pause/stop under load
WebGUI availability webgui_restart.sh — nginx → php-fpm → emhttp escalation
Kernel tuning inotify_tuning.sh — file watch limits
php_fpm_max_children.sh — PHP worker count
@@ -84,6 +82,9 @@ Graceful operations mover_stop.sh — clean mover stop
server_reboot.sh — clean reboot with pre-flight warnings
```
> `system_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
---
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
@@ -93,33 +94,22 @@ Orchestrators/
array_start.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
─────────────────────────► docker_syslog_filter.sh (second)
─────────────────────────► php_fpm_max_children.sh
─────────────────────────► system_watchdog.sh (background loop)
watchdog_orchestrator.sh ───────────────► resource_watchdog.sh (every minute)
weekly_maintenance.sh ──────────────────► clear_logs.sh
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
Docker_Essentials/
docker_watchdog.sh ◄─── reads ──────────── resource_watchdog.sh state
(mem_shutdown_active flag)
Watchdogs/
system_watchdog.sh and resource_watchdog.sh now live here.
See Watchdogs/README-Watchdogs.md for how they relate to each other
and to docker_watchdog.sh and storage_watchdog.sh.
```
`system_watchdog.sh` and `docker_watchdog.sh` (in Docker_Essentials/) are
designed to work together — docker_watchdog heals containers first,
system_watchdog reboots only when healing has failed. `resource_watchdog.sh`
coordinates with docker_watchdog via the `mem_shutdown_active` state flag to
prevent docker_watchdog from restarting containers that resource_watchdog just
stopped to free RAM.
---
## ━━━ SCRIPTS IN THIS FOLDER ━━━
| Script | Role | When It Runs |
|--------|------|-------------|
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop via array_start.sh |
| `resource_watchdog.sh` | Pressure reduction — throttle/pause/stop under load | Every minute via watchdog_orchestrator.sh |
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
@@ -137,20 +127,14 @@ stopped to free RAM.
```
Array starts
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
─ php_fpm_max_children.sh ← before WebGUI is under load
└─ system_watchdog.sh ← starts background loop
Every minute (watchdog_orchestrator.sh):
└─ resource_watchdog.sh
Level 1 (soft): throttle SABnzbd + qBit download speeds
Level 2 (medium): further throttle + docker pause non-essential containers
Level 3 (hard): docker stop optional services + set mem_shutdown_active=true
docker_watchdog.sh reads mem_shutdown_active — defers restarts
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
─ php_fpm_max_children.sh ← before WebGUI is under load
Every minute (watchdog_orchestrator.sh in Orchestrators/):
→ Watchdogs/resource_watchdog.sh → Watchdogs/docker_watchdog.sh
→ Watchdogs/storage_watchdog.sh → Watchdogs/system_watchdog.sh
(see Watchdogs/README-Watchdogs.md for full flow)
Every 10 minutes (User Scripts):
└─ webgui_restart.sh
@@ -161,13 +145,11 @@ Every 10 minutes (User Scripts):
Step 3: restart emhttp → recheck
All failed → notify, exit 1
Weekly (weekly_maintenance.sh):
└─ clear_logs.sh
System logs: clear if > LOG_MIN_SIZE_MB
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
Manual operations:
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
+2 -2
View File
@@ -277,12 +277,12 @@ if [[ "$TOTAL_CLEARED" -gt 0 || ${#FAILED[@]} -gt 0 ]]; then
notify "Log clear failed on $(hostname) ($MY_ID) — ${FAILED[*]}" \
"Clear Logs" "warning"
else
log "$ICON_DONE Status: done — ${TOTAL_FREED_H} freed"
echo "$ICON_DONE Status: done — ${TOTAL_FREED_H} freed"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
else
# All logs under threshold — completely silent
log "All logs under threshold — nothing to clear"
echo "All logs under threshold — nothing to clear"
fi
[[ ${#FAILED[@]} -gt 0 ]] && exit 1
+3 -3
View File
@@ -189,12 +189,12 @@ fi
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would restart rsyslog"
else
log "Restarting rsyslog..."
echo "Restarting rsyslog..."
if /etc/rc.d/rc.rsyslogd restart >/dev/null 2>&1; then
sleep 2
# Verify rsyslog actually running after restart
if pgrep -x rsyslogd >/dev/null 2>&1; then
log "rsyslog restarted and running ✅"
echo "rsyslog restarted and running ✅"
else
error "rsyslog not running after restart"
notify "rsyslog failed to start after filter update on $(hostname) ($MY_ID)" \
@@ -224,7 +224,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
else
log "$ICON_DONE Status: done — Docker veth noise suppressed ✅"
echo "$ICON_DONE Status: done — Docker veth noise suppressed ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+3 -3
View File
@@ -127,7 +127,7 @@ fi
START=$(date +%s)
if ! pgrep -f "emhttp.*Mover" >/dev/null 2>&1; then
log "Mover is not running — nothing to do"
echo "Mover is not running — nothing to do"
exit 0
fi
@@ -137,7 +137,7 @@ warn "Mover is running (PID $MOVER_PID) — stopping in ${MOVER_STOP_TIMEOUT}s"
# ── Warn users via wall ───────────────────────────────────────────────────────────────────────
if [[ "$DRY_RUN" == false ]]; then
wall "$ICON_WARN $MY_ID ($LOCAL_SERVER_NAME) — unRAID Mover stopping in ${MOVER_STOP_TIMEOUT}s"
log "Wall message sent — waiting ${MOVER_STOP_TIMEOUT}s..."
echo "Wall message sent — waiting ${MOVER_STOP_TIMEOUT}s..."
sleep "$MOVER_STOP_TIMEOUT"
else
warn "DRY RUN — would send wall warning and wait ${MOVER_STOP_TIMEOUT}s"
@@ -187,6 +187,6 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
else
log "$ICON_DONE Status: done — mover stopped ✅"
echo "$ICON_DONE Status: done — mover stopped ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+1 -1
View File
@@ -238,7 +238,7 @@ echo "$ICON_GEAR Config file: $PHP_CONF"
echo "$ICON_PHP Applied: pm.max_children = $PHP_MAX_CHILDREN"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
log "$ICON_DONE Status: done ✅"
echo "$ICON_DONE Status: done ✅"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
+1 -1
View File
@@ -501,7 +501,7 @@ echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
else
log "$ICON_DONE Status: done ✅"
echo "$ICON_DONE Status: done ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+11
View File
@@ -121,6 +121,15 @@ acquire_lock
detect_hosts
# ── Exit Trap — restart Docker service if reboot sequence aborts after stopping it ────────────
_REBOOT_DOCKER_STOPPED=false
_trap_restart_docker_service() {
[[ "$_REBOOT_DOCKER_STOPPED" == true ]] || return
warn "Exit trap: restarting Docker service after aborted reboot sequence"
/etc/rc.d/rc.docker start >/dev/null 2>&1 || true
}
trap _trap_restart_docker_service EXIT
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made, no reboot will occur"
# ==============================================================================================
@@ -280,6 +289,7 @@ if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would stop Docker service"
else
if /etc/rc.d/rc.docker stop >/dev/null 2>&1; then
_REBOOT_DOCKER_STOPPED=true
warn "Docker stopped ✅"
else
warn "Docker stop returned non-zero — may already be stopped"
@@ -317,5 +327,6 @@ if [[ "$DRY_RUN" == true ]]; then
else
warn "$ICON_REBOOT Rebooting $MY_ID now..."
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
trap - EXIT # committed to reboot — Docker should stay down
/sbin/reboot
fi
+3 -3
View File
@@ -172,7 +172,7 @@ FAILED=()
SKIPPED=()
if [[ ${#PIDS[@]} -eq 0 ]]; then
log "No User Script processes running — nothing to do"
echo "No User Script processes running — nothing to do"
else
warn "${#PIDS[@]} User Script process(es) found"
echo ""
@@ -232,7 +232,7 @@ echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
if [[ ${#PIDS[@]} -eq 0 ]]; then
log "No processes were running"
echo "No processes were running"
elif [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would have stopped ${#SKIPPED[@]} process(es): ${SKIPPED[*]}"
else
@@ -248,7 +248,7 @@ elif [[ ${#FAILED[@]} -gt 0 ]]; then
notify "User Scripts stop failed on $(hostname) ($MY_ID) — unkillable: ${FAILED[*]}" \
"User Scripts Stop" "warning"
else
log "$ICON_DONE Status: done ✅"
echo "$ICON_DONE Status: done ✅"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+15 -15
View File
@@ -44,7 +44,7 @@
# ── PATHS ─────────────────────────────────────────────────────────────────────────────────────
# Scripts: /mnt/user/appdata/unraid_scripts/
# Configuration: /mnt/user/appdata/unraid_scripts/master.conf
# Per-host: /mnt/user/appdata/unraid_scripts/master_host1.conf (or master_host2.conf)
# Per-host: /mnt/user/appdata/unraid_scripts/host1.conf (or host2.conf)
# Library: /mnt/user/appdata/unraid_scripts/common.sh
#
# ── PLUGIN SETTINGS (apply to every entry) ────────────────────────────────────────────────────
@@ -570,7 +570,7 @@
# docker_watchdog.sh — two-tier container self-healing monitor (single-pass)
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
#
# Tier 1 — explicit per-container (configured in master_host*.conf):
# Tier 1 — explicit per-container (configured in host*.conf):
# Memory hard limits: immediate restart when exceeded — no strikes, no waiting
# CPU strike system: 2 consecutive cycles above HARD_CPU_THRESHOLD → restart
# HTTP health checks: curl to configured URL — 2 consecutive failures → restart
@@ -589,8 +589,8 @@
# RAM emergency: reads mem_shutdown_active from system_watchdog state file, defers all restarts
# Silent on clean cycles — only outputs events and hourly heartbeat
#
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --dry-run
# watchdog_skip_list_manager.sh — view and manage the container skip list
# When docker_watchdog restarts the same container 3 times in 1hr → skip-listed.
@@ -631,7 +631,7 @@
# Running → docker restart (graceful). Stopped → left stopped (state respected). Missing → skip.
# Dependency ordering via WATCHDOG_DEPENDENCIES — databases before applications.
# Restart verification: checks container still up after settle period, notifies if not.
# Configured via HOST*_DAILY_RESTART_CONTAINERS (master_host*.conf):
# Configured via HOST*_DAILY_RESTART_CONTAINERS (host*.conf):
# NginxProxyManager, Authelia, Dispatcharr, Dispatcharr-Basic, ErsatzTV-Emby
#
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh --dry-run
@@ -642,7 +642,7 @@
# Called by weekly_sync_maintenance.sh after sync completes and containers are back up.
# Targets services that benefit from weekly clean start but don't stop for the sync itself.
# Same rules as daily: running→restart, stopped→leave, missing→skip.
# Configured via HOST*_WEEKLY_RESTART_CONTAINERS (master_host*.conf):
# Configured via HOST*_WEEKLY_RESTART_CONTAINERS (host*.conf):
# NextCloud, AdGuard-Home, Immich
#
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh --dry-run
@@ -711,7 +711,7 @@
# system_watchdog.sh — three-tier server stability last-resort watchdog (single-pass)
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
# All 18 checks independently toggleable per host in master_host*.conf.
# All 18 checks independently toggleable per host in host*.conf.
#
# Tier 1 CRITICAL — bypass ALL strikes, reboot immediately:
# Docker daemon hung → attempt rc.docker restart → still hung → reboot
@@ -733,8 +733,8 @@
# Reboot loop protection: N reboots in X hours → shutdown instead.
# State file heartbeat: writes watchdog_cycle=N every cycle (docker_watchdog stale guard).
#
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --dry-run
# resource_watchdog.sh — three-level pressure reduction layer (single-pass, called by watchdog_orchestrator)
# Reduces system load intelligently BEFORE docker_watchdog attempts container restarts.
@@ -748,11 +748,11 @@
# Recovery: pressure must stay below current threshold for RW_RECOVER_CYCLES consecutive runs
# before de-escalating. One level at a time — prevents flip-flopping.
# Coordination: at Level 3 writes mem_shutdown_active=true → docker_watchdog defers all restarts.
# HOST*_RW_PAUSE_CONTAINERS and HOST*_RW_STOP_CONTAINERS configured per host in master_host*.conf.
# HOST*_RW_PAUSE_CONTAINERS and HOST*_RW_STOP_CONTAINERS configured per host in host*.conf.
#
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh --status
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh --dry-run
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh
# inotify_tuning.sh — raise Linux inotify kernel limits at array start
# Called by array_started.sh FIRST — must run before containers start (they inherit limits).
@@ -858,7 +858,7 @@
# Run BEFORE arr cleanup — removes non-media files that would otherwise appear as orphans.
# Patterns: *.sfv *.md5 *.sha1 *.nfo *.url *.lnk *.rar *.zip *.info *.torrent
# *.sample* *.proof* *sync-conflict* *.scr *.exe *.srr *.log *.json
# Two profiles with separate folder lists (configured in master_host*.conf):
# Two profiles with separate folder lists (configured in host*.conf):
# anime HOST*_ANIME_CLEAN_FOLDERS — anime share folders
# media HOST*_MEDIA_CLEAN_FOLDERS — Movies, Tv_Shows, Music, Sports etc.
# ALWAYS --dry-run when adding new patterns or folders — verify before committing.
@@ -1080,7 +1080,7 @@
# Catches: renewed-but-not-reloaded (nginx not reloaded after certbot renewal),
# wrong cert served, chain issues visible externally but not internally.
# If a user would see a certificate error in their browser, this catches it first.
# Configured via HOST*_CERT_MONITOR_DOMAINS in master_host*.conf.
# Configured via HOST*_CERT_MONITOR_DOMAINS in host*.conf.
# Thresholds: > 30 days = silent, <= 30 = warning, <= CERT_CRIT_DAYS (7) = urgent.
#
# bash /mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh --status