Compare commits
14
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
16e3b7651d | ||
|
|
8a3e22c9b8 | ||
|
|
ec79a926e8 | ||
|
|
95151c2278 | ||
|
|
9ee8af1a71 | ||
|
|
84f133b99b | ||
|
|
ef8ef436cf | ||
|
|
3770bf80e7 | ||
|
|
86b31256ed | ||
|
|
588cde1cc4 | ||
|
|
5ccfb895c1 | ||
|
|
789d376eb3 | ||
|
|
a4edeab5eb | ||
|
|
c1c12cbe73 |
@@ -6,235 +6,9 @@ Configuration reference, setup procedures, and operational workflows.
|
||||
For folder overview and design philosophy see `README-Docker_Essentials.md`.
|
||||
For per-script detail see the script headers directly.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WATCHDOG CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
All watchdog configuration lives in `master_host*.conf` (per-container lists) and
|
||||
`master.conf` (shared thresholds and toggles). `detect_hosts()` aliases all
|
||||
`HOST1_` / `HOST2_` prefixed vars to their unprefixed names at runtime — scripts
|
||||
always read the right values for the server they're running on.
|
||||
|
||||
---
|
||||
|
||||
### ── Memory Hard Limits ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# Format: "ContainerName:LimitInMB"
|
||||
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
|
||||
#
|
||||
# Sizing guidance:
|
||||
# Check normal peak: docker stats ContainerName
|
||||
# Set limit at ~150-200% of normal peak
|
||||
# Emby peaks ~12GB under heavy transcode load → 18GB gives headroom
|
||||
# without triggering on legitimate load spikes
|
||||
#
|
||||
HOST1_WATCHDOG_CONTAINERS=(
|
||||
"Emby:18432" # 18GB — peaks ~12GB under heavy transcode load
|
||||
"LidaTube:6144" # 6GB — YouTube downloader, grows with large queues
|
||||
"Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature
|
||||
"Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous
|
||||
)
|
||||
```
|
||||
|
||||
A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
|
||||
visibility into a container approaching its ceiling before a restart is triggered.
|
||||
|
||||
---
|
||||
|
||||
### ── CPU Thresholds ────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# CPU is normalised against total core count — meaningful regardless of hardware.
|
||||
#
|
||||
# Why normalised:
|
||||
# 85% on one core of a 16-core machine = 5.3% normalised → ignore it
|
||||
# 85% normalised on a 16-core machine = 13.6 cores worth → runaway process
|
||||
#
|
||||
# CPU uses a STRIKE SYSTEM — not immediate restart like memory.
|
||||
# Brief spikes are normal (Tdarr encoding, Emby transcoding, SABnzbd unpacking).
|
||||
# The strike system ignores spikes and acts on sustained high usage.
|
||||
#
|
||||
# Strike 1: above HARD_CPU_THRESHOLD this cycle → warn, increment strike
|
||||
# Strike 2: above threshold next cycle → restart, reset counter
|
||||
# Recovery: drops below threshold any cycle → reset counter to 0
|
||||
#
|
||||
SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only
|
||||
HARD_CPU_THRESHOLD=85 # strike at 85% normalised
|
||||
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── HTTP Health Checks ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# Format: "ContainerName:http://host:port/optional-path"
|
||||
# Hits the actual service endpoint on every watchdog cycle.
|
||||
# "Container running" and "service responding" are not the same thing.
|
||||
#
|
||||
# Uses a STRIKE SYSTEM — network hiccups and brief restarts happen.
|
||||
# Two consecutive non-responses before acting prevents false positives.
|
||||
#
|
||||
# The path can be a lightweight health endpoint or the root URL.
|
||||
# Docker's own HEALTHCHECK requires the image to define it — most don't.
|
||||
# These checks work regardless of what the image defines.
|
||||
#
|
||||
HOST1_WATCHDOG_CONTAINER_URLS=(
|
||||
"Emby:http://localhost:8096" # Emby WebUI root — fast to respond
|
||||
"NginxProxyManager:http://localhost:81" # NPM admin interface
|
||||
)
|
||||
|
||||
# master.conf
|
||||
CURL_TIMEOUT=5 # seconds before non-response counts as a failure
|
||||
RESP_FAIL_LIMIT=2 # consecutive failures before restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Required Containers ───────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# Containers that must always be running.
|
||||
# Found stopped → watchdog attempts restart every cycle until running or skip-listed.
|
||||
# Uses the STRIKE SYSTEM — one miss might be mid-restart.
|
||||
# Persistent failure → skip list → critical notification.
|
||||
#
|
||||
# These are the containers whose absence breaks everything else:
|
||||
#
|
||||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
||||
"NginxProxyManager" # reverse proxy — all external traffic routes through this
|
||||
"Authelia" # SSO authentication — all protected services need it
|
||||
"Mariadb-Authelia" # Authelia database — must be up before Authelia starts
|
||||
"Redis-Authelia" # Authelia session cache — same startup dependency
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Dependency Ordering ───────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# Format: "DependentContainer:dependency1 dependency2"
|
||||
# Multiple dependencies space-separated. All must be running before dependent restarts.
|
||||
#
|
||||
# When a container and its dependency are both down:
|
||||
# → restart the dependency first
|
||||
# → skip the dependent this cycle
|
||||
# → next cycle: dependency healthy → dependent restarts cleanly
|
||||
#
|
||||
# Without this: Authelia starts, can't connect to MariaDB (still starting),
|
||||
# exits immediately, strike 1. Next cycle: same, strike 2. Skip list.
|
||||
# MariaDB was fine the whole time.
|
||||
#
|
||||
HOST1_WATCHDOG_DEPENDENCIES=(
|
||||
"Authelia:Mariadb-Authelia Redis-Authelia"
|
||||
"Authelia-Secondary:Mariadb-Authelia Redis-Authelia-Secondary"
|
||||
"NextCloud:Postgres-NextCloud"
|
||||
)
|
||||
```
|
||||
|
||||
The same dependency configuration is used by `docker_daily_restart.sh` and
|
||||
`docker_weekly_restart.sh` — configure once, applies everywhere.
|
||||
|
||||
---
|
||||
|
||||
### ── Startup Grace Period ──────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# Suppress restart actions for N seconds after array start.
|
||||
# Checks still run and log — only restart actions are suppressed.
|
||||
# Clock starts from when the watchdog process itself starts.
|
||||
#
|
||||
# Without this: false-positive restarts fire in the first minutes after
|
||||
# every array start while containers are still initialising.
|
||||
#
|
||||
WATCHDOG_STARTUP_GRACE=600 # 10 minutes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Tier 2 Global Scan ────────────────────────────────────────────────────────
|
||||
|
||||
Tier 2 scans every running container when `WATCHDOG_SCAN_ALL=true`. No per-container
|
||||
configuration required — it's the catch-all for everything not explicitly in Tier 1.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WATCHDOG_SCAN_ALL=true # enable global scan
|
||||
WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures
|
||||
WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills
|
||||
WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts
|
||||
WATCHDOG_RESTART_DEAD=true # recover containers in dead state
|
||||
WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero
|
||||
WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list
|
||||
|
||||
# Containers excluded from Tier 2 entirely.
|
||||
# Use for containers you intentionally stop/start manually, or containers that
|
||||
# have benign non-zero exits as part of their normal operation.
|
||||
WATCHDOG_SCAN_IGNORE=(
|
||||
"my-one-shot-container" # runs and exits normally — not a crash
|
||||
)
|
||||
```
|
||||
|
||||
Each toggle is independent — disable any check that produces false positives in your
|
||||
environment without affecting the others.
|
||||
|
||||
---
|
||||
|
||||
### ── Restart Loop Protection ──────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# N restarts within a rolling window → skip list + critical notification.
|
||||
# The watchdog stops touching the container entirely.
|
||||
# Skip list lives on /boot/config/ — survives reboots intentionally.
|
||||
# A container bad enough to be skip-listed is still broken after a reboot.
|
||||
#
|
||||
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list
|
||||
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
|
||||
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
|
||||
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
||||
```
|
||||
|
||||
**Auto-clear:** The watchdog checks the skip list every cycle and removes any container
|
||||
it finds running. If the container recovers on its own, monitoring resumes automatically.
|
||||
Manual clear is only needed when the container is stuck stopped.
|
||||
|
||||
---
|
||||
|
||||
### ── Notification Batching ────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# All events from one cycle collected → one notification at end of cycle.
|
||||
#
|
||||
# Why: a shared database going down can cascade 10+ containers failing
|
||||
# simultaneously. Without batching: 10 individual pings. With batching:
|
||||
# one summary listing all affected containers. Actionable vs overwhelming.
|
||||
#
|
||||
WATCHDOG_BATCH_NOTIFY=true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── State Files Reference ────────────────────────────────────────────────────
|
||||
|
||||
| File | Config Var | Location | Resets | Purpose |
|
||||
|------|-----------|----------|--------|---------|
|
||||
| Strike counts | `WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | Per-container CPU/HTTP strike counters |
|
||||
| Skip list | `SYS_WATCHDOG_FAILED_FILE` | `/boot/config/` | Never (manual / auto-clear) | Containers that exhausted restart attempts |
|
||||
| Restart history | `WATCHDOG_CONTAINER_RESTART_LOG` | `/boot/config/` | Auto-purge after window | Restart loop detection data |
|
||||
| Shared state | `SYS_WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | RAM emergency flag + cycle heartbeat from `system_watchdog.sh` |
|
||||
|
||||
`/tmp/` resets on reboot — correct, strike counts before a reboot are meaningless after it.
|
||||
`/boot/config/` survives reboots — correct, a skip-listed container is still broken after a reboot.
|
||||
> **Watchdog configuration has moved.** `docker_watchdog.sh` is now in `Watchdogs/`.
|
||||
> Memory limits, CPU thresholds, HTTP health checks, dependency ordering, skip list
|
||||
> recovery, and all watchdog config vars are in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
---
|
||||
|
||||
@@ -245,7 +19,7 @@ WATCHDOG_BATCH_NOTIFY=true
|
||||
### ── Daily Restart List ───────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
# Restarted every night at 1am via daily_sync_maintenance.sh.
|
||||
#
|
||||
# Good candidates:
|
||||
@@ -271,7 +45,7 @@ their images updated daily before the restart. Add a container once, it gets bot
|
||||
### ── Weekly Restart List ──────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
# Restarted every Sunday at 2:30am via weekly_sync_maintenance.sh.
|
||||
# Runs AFTER the sync window's own restart of critical containers (Emby, auth stack).
|
||||
#
|
||||
@@ -291,7 +65,7 @@ HOST1_WEEKLY_RESTART_CONTAINERS=(
|
||||
## ━━━ NETWORK CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
# Networks to ensure exist + containers to connect to each network.
|
||||
# Many-to-many: every container connects to every network listed.
|
||||
#
|
||||
@@ -339,23 +113,11 @@ Everything gets updated at least once per week with no explicit configuration.
|
||||
|
||||
## ━━━ FULL CONFIGURATION REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
### ── master_host*.conf ────────────────────────────────────────────────────────
|
||||
### ── host*.conf ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# Per-host — varies between HOST1 and HOST2
|
||||
|
||||
# Tier 1 — memory limits ("ContainerName:LimitInMB")
|
||||
HOST1_WATCHDOG_CONTAINERS=()
|
||||
|
||||
# Tier 1 — HTTP health check endpoints ("ContainerName:http://host:port")
|
||||
HOST1_WATCHDOG_CONTAINER_URLS=()
|
||||
|
||||
# Tier 1 — must always be running
|
||||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
|
||||
|
||||
# Tier 1 + 2 — dependency ordering ("Dependent:dep1 dep2")
|
||||
HOST1_WATCHDOG_DEPENDENCIES=()
|
||||
|
||||
# Daily restart list (also drives docker_update.sh normal mode)
|
||||
HOST1_DAILY_RESTART_CONTAINERS=()
|
||||
|
||||
@@ -367,6 +129,9 @@ HOST1_NETWORK_CONNECT_NETWORKS=()
|
||||
|
||||
# Containers to connect to every configured network
|
||||
HOST1_NETWORK_CONNECT_CONTAINERS=()
|
||||
|
||||
# Watchdog config (memory limits, HTTP checks, required, dependencies):
|
||||
# → see Watchdogs/Manual-Watchdogs.md
|
||||
```
|
||||
|
||||
---
|
||||
@@ -376,47 +141,16 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
|
||||
```bash
|
||||
# Shared — applies to both servers
|
||||
|
||||
# ── Watchdog timing ────────────────────────────────────────────────────────
|
||||
DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes)
|
||||
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
|
||||
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
|
||||
|
||||
# ── Memory ─────────────────────────────────────────────────────────────────
|
||||
SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart)
|
||||
|
||||
# ── CPU ────────────────────────────────────────────────────────────────────
|
||||
SOFT_CPU_THRESHOLD=50 # warn threshold — normalised %
|
||||
HARD_CPU_THRESHOLD=85 # strike threshold — normalised %
|
||||
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
|
||||
|
||||
# ── HTTP health check ──────────────────────────────────────────────────────
|
||||
CURL_TIMEOUT=5 # seconds before curl times out
|
||||
RESP_FAIL_LIMIT=2 # consecutive failures before restart
|
||||
|
||||
# ── Restart loop protection ────────────────────────────────────────────────
|
||||
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts before skip list
|
||||
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
|
||||
|
||||
# ── Tier 2 global scan ─────────────────────────────────────────────────────
|
||||
WATCHDOG_SCAN_ALL=true
|
||||
WATCHDOG_SCAN_IGNORE=()
|
||||
WATCHDOG_RESTART_UNHEALTHY=true
|
||||
WATCHDOG_NOTIFY_OOM=true
|
||||
WATCHDOG_NOTIFY_CRASHLOOP=true
|
||||
WATCHDOG_CRASH_LIMIT=5
|
||||
WATCHDOG_RESTART_DEAD=true
|
||||
WATCHDOG_RESTART_CRASHED=true
|
||||
|
||||
# ── Notifications ──────────────────────────────────────────────────────────
|
||||
WATCHDOG_BATCH_NOTIFY=true
|
||||
|
||||
# ── Container updates ──────────────────────────────────────────────────────
|
||||
# ── Container updates ──────────────────────────────────────────────────
|
||||
DAILY_CONTAINER_UPDATES=true
|
||||
WEEKLY_REMAINING_UPDATES=true
|
||||
|
||||
# ── Retry behaviour (shared by restart scripts) ────────────────────────────
|
||||
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
||||
RETRY_COUNT=3 # retry attempts before marking failed
|
||||
SLEEP=5 # seconds between retry attempts
|
||||
|
||||
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
||||
# → see Watchdogs/Manual-Watchdogs.md
|
||||
```
|
||||
|
||||
---
|
||||
@@ -425,47 +159,22 @@ SLEEP=5 # seconds between retry attempts
|
||||
|
||||
### ── Adding a Container to Monitoring ────────────────────────────────────────
|
||||
|
||||
Adding a container is purely additive — add the relevant lines to `master_host*.conf`.
|
||||
Adding a container to watchdog monitoring is purely additive — add lines to `host*.conf`.
|
||||
No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
|
||||
See `Watchdogs/Manual-Watchdogs.md` for the full procedure.
|
||||
|
||||
To add a container to **daily restarts** (and daily image updates):
|
||||
|
||||
```bash
|
||||
# master_host1.conf — example: adding "MyApp" to full Tier 1 + daily restarts
|
||||
|
||||
# 1. Memory hard limit — size at ~150-200% of normal peak (check: docker stats MyApp)
|
||||
HOST1_WATCHDOG_CONTAINERS=(
|
||||
...existing...
|
||||
"MyApp:2048" # 2GB ceiling
|
||||
)
|
||||
|
||||
# 2. HTTP health check
|
||||
HOST1_WATCHDOG_CONTAINER_URLS=(
|
||||
...existing...
|
||||
"MyApp:http://localhost:8080/health" # or root URL if no /health endpoint
|
||||
)
|
||||
|
||||
# 3. Required — add only if absence breaks other services
|
||||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
||||
...existing...
|
||||
"MyApp"
|
||||
)
|
||||
|
||||
# 4. Dependency — add if MyApp needs another container up first
|
||||
HOST1_WATCHDOG_DEPENDENCIES=(
|
||||
...existing...
|
||||
"MyApp:MyApp-Database"
|
||||
)
|
||||
|
||||
# 5. Daily restart — add if MyApp degrades over time
|
||||
# host1.conf
|
||||
HOST1_DAILY_RESTART_CONTAINERS=(
|
||||
...existing...
|
||||
"MyApp" # also adds it to the daily image update
|
||||
)
|
||||
```
|
||||
|
||||
Tier 2 picks up MyApp automatically — no configuration needed. It will be included in
|
||||
the global health scan from the next cycle onward.
|
||||
To **exclude** a container from Tier 2 global scan (e.g. one-shot that exits normally):
|
||||
|
||||
To **exclude** MyApp from Tier 2 (e.g. it's a one-shot container that exits normally):
|
||||
```bash
|
||||
# master.conf
|
||||
WATCHDOG_SCAN_IGNORE=(
|
||||
@@ -477,45 +186,13 @@ WATCHDOG_SCAN_IGNORE=(
|
||||
|
||||
### ── Skip List Recovery ────────────────────────────────────────────────────────
|
||||
|
||||
Used when `docker_watchdog.sh` has skip-listed a container after exhausting restart
|
||||
attempts. The watchdog stops touching it and sends a critical notification. Human
|
||||
intervention required.
|
||||
When `docker_watchdog.sh` skip-lists a container, use `Tools/watchdog_skip_list_manager.sh`.
|
||||
Full procedure in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
```bash
|
||||
# Step 1 — understand the situation
|
||||
# Shows skip list contents, container states, restart history
|
||||
watchdog_skip_list_manager.sh --status
|
||||
|
||||
# Step 2 — fix the underlying problem first
|
||||
# Check logs: docker logs ContainerName --tail 100
|
||||
# Check disk: df -h /mnt/user
|
||||
# Check database: docker exec ContainerName sqlite3 /path/to.db ".tables"
|
||||
# Fix before clearing — clearing without fixing just resets the counter
|
||||
|
||||
# Step 3 — clear the container from the skip list + restart history
|
||||
# Clearing history is important: the counter carries over otherwise and
|
||||
# the container hits the limit again almost immediately on any startup trouble
|
||||
watchdog_skip_list_manager.sh --clear ContainerName
|
||||
|
||||
# Step 4 — start the container manually
|
||||
# Confirms your fix worked before handing back to the watchdog
|
||||
docker start ContainerName
|
||||
|
||||
# Step 5 — monitoring resumes automatically
|
||||
# Next watchdog cycle: container seen running → removed from skip list
|
||||
# Restart history clean. Back to normal.
|
||||
```
|
||||
|
||||
> ⚠️ **If `docker_watchdog.sh` is currently running when you clear the skip list**, it
|
||||
> may re-add the container on its very next cycle if the container is still in a bad
|
||||
> state. The script detects this and warns you. Fix the root cause **before** clearing.
|
||||
|
||||
Other skip list actions:
|
||||
```bash
|
||||
watchdog_skip_list_manager.sh # status (default)
|
||||
watchdog_skip_list_manager.sh --clear-all # clear everything
|
||||
watchdog_skip_list_manager.sh --clear-all --force # non-interactive
|
||||
watchdog_skip_list_manager.sh --clear-all --dry-run # preview what would clear
|
||||
Tools/watchdog_skip_list_manager.sh --status # see skip list + container states
|
||||
Tools/watchdog_skip_list_manager.sh --clear MyApp # clear after fixing root cause
|
||||
Tools/watchdog_skip_list_manager.sh --clear-all # clear everything
|
||||
```
|
||||
|
||||
---
|
||||
@@ -530,25 +207,9 @@ All scripts support these standard flags:
|
||||
| `--status` | Show current config, container states, and relevant runtime info, then exit. |
|
||||
| `--log` | Verbose mode — adds per-container banners, action lines, pull output, and list details. |
|
||||
|
||||
**Output tiers:** Scripts have two output levels. Without `--log`, each script processes
|
||||
silently and always concludes with a summary block: identity, duration, counts, and
|
||||
a status line. Per-container detail — individual container names, pull Status lines,
|
||||
skip reasons — only appears with `--log`. Warnings and errors are always visible
|
||||
regardless of `--log`.
|
||||
|
||||
`docker_watchdog.sh` is the exception — it runs 96 cycles/day as a background daemon
|
||||
and is silent on clean cycles by design. Its summary block only fires when there are
|
||||
restarts or warnings. Use `--log` to see per-cycle detail on clean cycles.
|
||||
|
||||
### `docker_watchdog.sh --status` shows:
|
||||
Strike counts for all monitored containers, current skip list contents, whether grace
|
||||
period is active and how long remains, whether RAM emergency deferral is active, last
|
||||
cycle timing.
|
||||
|
||||
### `docker_watchdog.sh --dry-run` shows:
|
||||
A full watchdog cycle without restarting anything. Shows what the watchdog would do
|
||||
based on current container states. Useful for verifying configuration before enabling
|
||||
automatic restarts.
|
||||
**Output tiers:** Without `--log`, each script processes silently and concludes with a
|
||||
summary block: identity, duration, counts, and a status line. Per-container detail only
|
||||
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
||||
|
||||
### `docker_update.sh --remainder`
|
||||
Switches to remainder mode — updates all running containers not in the managed daily/weekly
|
||||
|
||||
@@ -154,24 +154,16 @@ its history after you've fixed the problem. No manual file editing required.
|
||||
```
|
||||
Docker_Essentials/ ← acts on containers (this folder)
|
||||
unRAID_Essentials/ ← acts on the server itself
|
||||
Watchdogs/ ← reactive monitoring + last-resort stability
|
||||
Monitors/ ← observes, measures, reports
|
||||
Rsync/ ← moves data between servers
|
||||
```
|
||||
|
||||
Five distinct responsibilities, each handled by dedicated scripts:
|
||||
> `docker_watchdog.sh` has moved to `Watchdogs/`. Container healing, memory limits,
|
||||
> HTTP health checks, and skip list management are documented in
|
||||
> `Watchdogs/README-Watchdogs.md` and `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
---
|
||||
|
||||
### 🔁 Reactive Healing — `docker_watchdog.sh`
|
||||
|
||||
Continuous two-tier monitoring that catches problems as they happen and acts on them
|
||||
immediately. Runs as a background process started at array start. Every 15 minutes it
|
||||
checks the full stack and fixes what it can — silently when everything is fine, visibly
|
||||
when something needs attention.
|
||||
|
||||
Two tiers because different containers need different monitoring strategies:
|
||||
- **Tier 1** — explicit per-container configuration with specific thresholds
|
||||
- **Tier 2** — global scan of everything that's running with catch-all health checks
|
||||
Four distinct responsibilities in this folder, each handled by dedicated scripts:
|
||||
|
||||
---
|
||||
|
||||
@@ -212,34 +204,22 @@ or in-progress downloads.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Two watchdogs run simultaneously. They are designed to work together, not compete:
|
||||
`docker_watchdog.sh` has moved to `Watchdogs/` and is now one of four coordinated
|
||||
single-pass scripts called every minute by `Orchestrators/watchdog_orchestrator.sh`.
|
||||
|
||||
```
|
||||
system_watchdog.sh ← watches the server: RAM, CPU, disk, kernel, daemon health
|
||||
docker_watchdog.sh ← watches the containers: memory, CPU, HTTP response, crashes
|
||||
Watchdogs/resource_watchdog.sh ← reduces pressure before healing attempts
|
||||
Watchdogs/docker_watchdog.sh ← heals containers (reads resource_watchdog state)
|
||||
Watchdogs/storage_watchdog.sh ← pool growth + runaway log detection
|
||||
Watchdogs/system_watchdog.sh ← last resort — reboots when healing has failed
|
||||
```
|
||||
|
||||
**The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops
|
||||
non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh`
|
||||
would see stopped containers on its next cycle and restart them — directly undoing the
|
||||
RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover.
|
||||
The system would eventually hit the reboot threshold anyway, having accomplished nothing.
|
||||
|
||||
**The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When
|
||||
`system_watchdog.sh` triggers a RAM emergency shutdown it writes
|
||||
`mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every
|
||||
cycle and defers all container restart logic until it clears. Health URL checks for
|
||||
excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still
|
||||
run. Everything else stands down.
|
||||
|
||||
**The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state
|
||||
file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh`
|
||||
checks how long ago the state file was modified. If it's more than 2 hours old while
|
||||
`mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running.
|
||||
`docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced
|
||||
indefinitely by a stale flag from a process that's no longer running.
|
||||
Scripts in this folder (daily restart, updates, network connect) are unaffected —
|
||||
they run on their own schedules via the maintenance orchestrators and are not part
|
||||
of the every-minute watchdog cycle. See `Watchdogs/README-Watchdogs.md` for the
|
||||
full coordination model between all four watchdogs.
|
||||
|
||||
---
|
||||
|
||||
@@ -247,7 +227,6 @@ indefinitely by a stale flag from a process that's no longer running.
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop — array start |
|
||||
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
||||
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
||||
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window |
|
||||
@@ -265,18 +244,12 @@ indefinitely by a stale flag from a process that's no longer running.
|
||||
```
|
||||
Array starts
|
||||
│
|
||||
├── docker_network_connect.sh ────── run once at start
|
||||
│ ensure networks + connections exist
|
||||
│ silent if correct, notify if creating
|
||||
│
|
||||
└── docker_watchdog.sh ──────────── continuous background loop (every 15min)
|
||||
Tier 1: memory, CPU, HTTP, required containers
|
||||
Tier 2: global unhealthy / OOM / crash / dead scan
|
||||
reads system_watchdog state (RAM emergency deferral)
|
||||
│
|
||||
│ (on skip list event → operator uses)
|
||||
└── Tools/watchdog_skip_list_manager.sh
|
||||
inspect state, clear after fixing root cause
|
||||
└── docker_network_connect.sh ────── run once at start
|
||||
ensure networks + connections exist
|
||||
silent if correct, notify if creating
|
||||
|
||||
Every minute (Orchestrators/watchdog_orchestrator.sh):
|
||||
└── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md
|
||||
|
||||
Daily maintenance window (1am):
|
||||
daily_sync_maintenance.sh
|
||||
|
||||
@@ -60,7 +60,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Containers restarted nightly. Also used by docker_update.sh normal mode
|
||||
@@ -126,7 +126,7 @@ detect_hosts
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
|
||||
@@ -78,7 +78,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_NETWORK_CONNECT_NETWORKS
|
||||
# Networks to ensure exist at array start. Aliased by detect_hosts() →
|
||||
@@ -144,13 +144,13 @@ fi
|
||||
# Empty array guards
|
||||
if [[ ${#NETWORK_CONNECT_NETWORKS[@]} -eq 0 ]]; then
|
||||
warn "NETWORK_CONNECT_NETWORKS is empty for $MY_ID — nothing to do"
|
||||
warn "Check HOST*_NETWORK_CONNECT_NETWORKS in master_host*.conf"
|
||||
warn "Check HOST*_NETWORK_CONNECT_NETWORKS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#NETWORK_CONNECT_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "NETWORK_CONNECT_CONTAINERS is empty for $MY_ID — no containers to connect"
|
||||
warn "Check HOST*_NETWORK_CONNECT_CONTAINERS in master_host*.conf"
|
||||
warn "Check HOST*_NETWORK_CONNECT_CONTAINERS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
|
||||
@@ -95,7 +95,7 @@
|
||||
# Container names for emby and critical-data profiles — excluded from
|
||||
# remainder mode (already updated by the weekly sync window)
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Containers updated in normal mode. Aliased by detect_hosts() →
|
||||
@@ -205,7 +205,7 @@ else
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
|
||||
@@ -52,7 +52,7 @@
|
||||
# Enable or disable this script. (default: true)
|
||||
# To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Excluded from this script — already updated daily. Aliased by
|
||||
|
||||
@@ -43,7 +43,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_WEEKLY_RESTART_CONTAINERS
|
||||
# Containers restarted weekly. Aliased by detect_hosts() →
|
||||
@@ -108,7 +108,7 @@ detect_hosts
|
||||
|
||||
if [[ ${#WEEKLY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "WEEKLY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
|
||||
warn "Check HOST*_WEEKLY_RESTART_CONTAINERS in master_host*.conf"
|
||||
warn "Check HOST*_WEEKLY_RESTART_CONTAINERS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
|
||||
@@ -83,7 +83,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_SLSKD_URL / HOST*_SLSKD_API_KEY / HOST*_SLSKD_FAILED_IMPORTS_DIR
|
||||
# slskd connection and failed imports path. Aliased by detect_hosts()
|
||||
@@ -567,7 +567,7 @@ if [[ -n "$QBIT_URL" ]] && [[ -n "$QBIT_USERNAME" ]]; then
|
||||
|
||||
if [[ -z "$QBIT_COOKIE" ]]; then
|
||||
error "Failed to authenticate with qBittorrent — check QBIT_USERNAME/PASSWORD"
|
||||
notify "qBittorrent auth failed on $(hostname) — check credentials in master_host*.conf" "Downloaders Reset" "warning"
|
||||
notify "qBittorrent auth failed on $(hostname) — check credentials in host*.conf" "Downloaders Reset" "warning"
|
||||
((TOTAL_FAIL++))
|
||||
else
|
||||
TORRENTS=$(curl -sf --max-time 15 \
|
||||
|
||||
@@ -80,7 +80,7 @@ container start time. At 3 strikes × 30s + ~2min rsync + ~1min container start:
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONFIGURATION — master_host*.conf ━━━
|
||||
## ━━━ CONFIGURATION — host*.conf ━━━
|
||||
|
||||
```
|
||||
HOST*_DDNS_CONTAINERS=("...")
|
||||
@@ -181,7 +181,7 @@ FALLBACK_TEST_HANDBACK_WAIT=300
|
||||
|
||||
---
|
||||
|
||||
### master_host2.conf — HOST2 covering HOST1
|
||||
### host2.conf — HOST2 covering HOST1
|
||||
|
||||
```bash
|
||||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||||
@@ -259,7 +259,7 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
|
||||
|
||||
---
|
||||
|
||||
### master_host1.conf — HOST1 covering HOST2
|
||||
### host1.conf — HOST1 covering HOST2
|
||||
|
||||
```bash
|
||||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||||
@@ -486,7 +486,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
|
||||
|
||||
1. Create the container on the covering server (stopped), with volume mounts pointing at
|
||||
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
|
||||
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in master_host*.conf
|
||||
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf
|
||||
in the appropriate tier position (dependency ordering — databases before apps)
|
||||
3. Verify: `fallback.sh --status` shows the container in the expected tier list
|
||||
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
|
||||
@@ -568,6 +568,24 @@ If it has happened:
|
||||
|
||||
---
|
||||
|
||||
## ━━━ OUTPUT TIERS ━━━
|
||||
|
||||
Both scripts have two output levels controlled by `--log`.
|
||||
|
||||
**fallback.sh** — daemon, runs continuously. Without `--log`, only state transitions,
|
||||
warnings, errors, and the startup banner are visible. Per-cycle detail (ping results,
|
||||
state evaluation) is suppressed — the daemon runs every 15–30 seconds and clean cycles
|
||||
produce no output by design. State transitions are always visible because they are the
|
||||
events that matter. Use `--log` when debugging why the state machine is or is not
|
||||
acting.
|
||||
|
||||
**fallback_test.sh** — one-shot test harness. Without `--log`, phase headers, pass/fail
|
||||
results, and the final test report are always visible. Per-container checks within each
|
||||
phase are suppressed. With `--log`, every check in every phase is shown. Warnings and
|
||||
errors are always visible regardless of `--log`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FLAG REFERENCE ━━━
|
||||
|
||||
### fallback.sh
|
||||
|
||||
@@ -219,8 +219,8 @@ determines which server is local and which is remote at runtime, then selects th
|
||||
container arrays and tier delays from config via MY_ID.
|
||||
|
||||
```
|
||||
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in master_host2.conf)
|
||||
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in master_host1.conf)
|
||||
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf)
|
||||
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf)
|
||||
```
|
||||
|
||||
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
|
||||
|
||||
@@ -111,6 +111,10 @@
|
||||
# acquire_lock() prevents two instances running simultaneously. Both would
|
||||
# modify containers and DDNS independently — conflict is certain.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() resolves MY_ID / REMOTE_ID from master.conf at startup.
|
||||
# Exits if the host cannot be identified — prevents running on an unknown machine.
|
||||
#
|
||||
# Silent by Default
|
||||
# State transitions: warn() — always visible. Healthy routine cycles: log()
|
||||
# — suppressed unless --log. Produces no output on clean cycles.
|
||||
@@ -128,7 +132,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DDNS_CONTAINERS
|
||||
# DDNS containers this host manages — stopped on internet loss, started
|
||||
@@ -246,8 +250,6 @@ fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
|
||||
@@ -60,6 +60,14 @@
|
||||
# Remote Docker Daemon Check
|
||||
# Pre-flight confirms remote Docker daemon is responsive before Phase 2.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock() prevents concurrent test runs. Running two tests simultaneously
|
||||
# would produce conflicting iptables rules and unreliable results.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() resolves MY_ID / REMOTE_ID from master.conf at startup.
|
||||
# Exits if the host cannot be identified — prevents testing on an unknown machine.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
@@ -134,8 +142,6 @@ trap cleanup EXIT
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
@@ -283,7 +289,7 @@ fi
|
||||
# Tier 1 containers configured
|
||||
if [[ ${#TIER1_CONTAINERS[@]} -eq 0 ]]; then
|
||||
error "No Tier 1 containers configured for $MY_ID → $REMOTE_ID"
|
||||
error "Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in master_host*.conf"
|
||||
error "Check FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER1 in host*.conf"
|
||||
phase_fail "Pre-flight"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
+24
-7
@@ -228,7 +228,7 @@ LIDARR_DISCOVERY_REJECT_COOLDOWN=30 # days before re-evaluating a Stage 2 reje
|
||||
LIDARR_DISCOVERY_HISTORY="$DATA_DIR/lidarr_discovery_history.db"
|
||||
```
|
||||
|
||||
Requires `HOST*_LASTFM_API_KEY` in `master_host*.conf`.
|
||||
Requires `HOST*_LASTFM_API_KEY` in `host*.conf`.
|
||||
|
||||
---
|
||||
|
||||
@@ -246,7 +246,7 @@ RADARR_DISCOVERY_SEED_LIBRARIES=("Movies") # Emby libraries to draw seeds from
|
||||
RADARR_DISCOVERY_HISTORY="$DATA_DIR/radarr_discovery_history.db"
|
||||
```
|
||||
|
||||
Requires `HOST*_TMDB_API_KEY` in `master_host*.conf`.
|
||||
Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
|
||||
|
||||
---
|
||||
|
||||
@@ -266,7 +266,7 @@ SONARR_DISCOVERY_HISTORY="$DATA_DIR/sonarr_discovery_history.db"
|
||||
# SONARR_EMBY_LIBRARIES is shared with emby_to_sonarr_sync — see Orchestrator Job Order
|
||||
```
|
||||
|
||||
Requires `HOST*_TMDB_API_KEY` in `master_host*.conf`.
|
||||
Requires `HOST*_TMDB_API_KEY` in `host*.conf`.
|
||||
|
||||
> **MONITOR_MODE note:** Use `"all"` (default) to have Sonarr search all existing seasons
|
||||
> after adding a show. `"future"` only marks upcoming seasons as monitored — shows where
|
||||
@@ -289,9 +289,9 @@ MEDIA_MAINTENANCE_JOBS=(
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONFIGURATION — master_host*.conf ━━━
|
||||
## ━━━ CONFIGURATION — host*.conf ━━━
|
||||
|
||||
### master_host1.conf
|
||||
### host1.conf
|
||||
|
||||
```bash
|
||||
# Shares this server applies permissions to
|
||||
@@ -429,7 +429,7 @@ cp radarr_cleanup.sh readarr_cleanup.sh
|
||||
# 2. Replace RADARR_ prefix with READARR_ throughout
|
||||
# Update API endpoint, tracked file API path, extension list, protected patterns
|
||||
|
||||
# 3. Add to master_host*.conf
|
||||
# 3. Add to host*.conf
|
||||
HOST1_READARR_URL="http://192.168.50.2:8787"
|
||||
HOST1_READARR_API_KEY="your-api-key"
|
||||
HOST1_READARR_BOOKS_ROOT="/mnt/user/Books"
|
||||
@@ -534,7 +534,7 @@ If ghosts persist:
|
||||
1. Is Emby's API responding?
|
||||
curl -s "http://[emby-ip]:8096/System/Info/Public"
|
||||
|
||||
2. Is EMBY_URL / EMBY_API_KEY correct in master_host*.conf?
|
||||
2. Is EMBY_URL / EMBY_API_KEY correct in host*.conf?
|
||||
Run: sonarr_cleanup.sh --status (shows Emby config)
|
||||
|
||||
3. Trigger manually in Emby:
|
||||
@@ -543,6 +543,23 @@ If ghosts persist:
|
||||
|
||||
---
|
||||
|
||||
## ━━━ OUTPUT TIERS ━━━
|
||||
|
||||
All scripts have two output levels controlled by `--log`.
|
||||
|
||||
Without `--log`, each script processes silently and always concludes with a summary
|
||||
block: identity, duration, counts (files removed, items added, arrs cleaned), and a
|
||||
status line. Warnings and errors are always visible.
|
||||
|
||||
With `--log`, per-item detail appears: individual titles being processed, API query
|
||||
progress, per-node sync results, and per-file examination output. Use this when
|
||||
debugging unexpected results or validating configuration before the first scheduled run.
|
||||
|
||||
Dry-run output follows the same tiers — `--dry-run` alone shows the summary of what
|
||||
would happen; `--dry-run --log` shows the full per-item preview list.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FLAG REFERENCE ━━━
|
||||
|
||||
### media_shares_permissions.sh
|
||||
|
||||
+2
-5
@@ -78,7 +78,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_LIDARR_URL / HOST*_LIDARR_API_KEY — local Lidarr (aliased by detect_hosts)
|
||||
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY — local Sonarr
|
||||
@@ -154,9 +154,6 @@ fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -822,7 +819,7 @@ _sync_arr() {
|
||||
fi
|
||||
fi
|
||||
|
||||
echo " $node_name: +${#to_add_local[@]} local | +${#to_add_remote[@]} remote | $total_skipped blocklisted"
|
||||
log " $node_name: +${#to_add_local[@]} local | +${#to_add_remote[@]} remote | $total_skipped blocklisted"
|
||||
|
||||
unset remote_ids
|
||||
declare -A remote_ids
|
||||
|
||||
@@ -54,7 +54,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_RECOVERY
|
||||
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_RECOVERY
|
||||
@@ -92,9 +92,6 @@ parse_args "$@"
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -451,7 +448,7 @@ elif [[ "$TOTAL_ACTIONED" -gt 0 ]]; then
|
||||
notify "Arr recovery on $(hostname) — $TOTAL_ACTIONED item(s) blocklisted and re-searched" \
|
||||
"Arr Recovery" "warning"
|
||||
else
|
||||
log "$ICON_DONE Done — nothing to recover (all arrs clean)"
|
||||
echo "$ICON_DONE Done — nothing to recover (all arrs clean)"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
+6
-10
@@ -58,7 +58,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY / HOST1_LIDARR_MUSIC_ROOT
|
||||
# HOST1_LIDARR_PATH_MAP — container path → host path translation
|
||||
@@ -136,9 +136,6 @@ fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -347,7 +344,7 @@ SCAN_CMD_ID=$(echo "$SCAN_RESPONSE" | jq -r '.id // empty' 2>/dev/null)
|
||||
if [[ -z "$SCAN_CMD_ID" ]]; then
|
||||
warn "Could not trigger import scan — proceeding without pre-flight"
|
||||
else
|
||||
echo " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
|
||||
log " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
|
||||
POLL_TIMEOUT=${LIDARR_IMPORT_SCAN_TIMEOUT:-600}
|
||||
POLLED=0
|
||||
while [[ "$POLLED" -lt "$POLL_TIMEOUT" ]]; do
|
||||
@@ -382,7 +379,7 @@ fi
|
||||
# Safety Layer 3 — API version check
|
||||
check_arr_version "$LIDARR_URL" "$LIDARR_API_KEY" "v1" "$LIDARR_VERSION_MAJOR" "Lidarr" || exit 1
|
||||
|
||||
echo " Querying Lidarr API..."
|
||||
log " Querying Lidarr API..."
|
||||
|
||||
# Fetch all artists
|
||||
ARTIST_RESPONSE=$(lidarr_api "artist") || {
|
||||
@@ -443,7 +440,7 @@ if [[ "$TRACKED_COUNT" -eq 0 ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo " $ARTIST_COUNT artists | $TRACKED_COUNT tracked files"
|
||||
log " $ARTIST_COUNT artists | $TRACKED_COUNT tracked files"
|
||||
|
||||
# Safety Layer 6 — percentage drop vs last known count
|
||||
if [[ -f "$LIDARR_TRACKED_COUNT_FILE" ]]; then
|
||||
@@ -471,8 +468,7 @@ echo "$TRACKED_COUNT" > "$LIDARR_TRACKED_COUNT_FILE"
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_CLEAN Scanning Music Root ━━━"
|
||||
echo " Root: $LIDARR_MUSIC_ROOT | Orphan age: ${LIDARR_ORPHAN_AGE} days"
|
||||
echo ""
|
||||
log " Root: $LIDARR_MUSIC_ROOT | Orphan age: ${LIDARR_ORPHAN_AGE} days"
|
||||
|
||||
START=$(date +%s)
|
||||
ORPHAN_COUNT=0
|
||||
@@ -605,7 +601,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no files deleted"
|
||||
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
|
||||
log "$ICON_DONE Clean — nothing to remove"
|
||||
echo "$ICON_DONE Clean — nothing to remove"
|
||||
else
|
||||
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
|
||||
notify "Lidarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" "Lidarr Cleanup" "warning"
|
||||
|
||||
@@ -42,7 +42,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST1_LIDARR_URL / HOST1_LIDARR_API_KEY
|
||||
# Aliased by detect_hosts() — script uses LIDARR_URL / LIDARR_API_KEY
|
||||
@@ -76,9 +76,6 @@ parse_args "$@"
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -110,7 +107,7 @@ if [[ -z "$LIDARR_URL" ]]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo " $MY_ID ($LOCAL_SERVER_NAME) — tools OK"
|
||||
log " $MY_ID ($LOCAL_SERVER_NAME) — tools OK"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
@@ -218,7 +215,7 @@ if ! curl_json "$LIDARR_URL/api/v1/system/status?apikey=$LIDARR_API_KEY" | jq -e
|
||||
notify "lidarr_missing_art failed — Lidarr API unreachable on $(hostname)" "Lidarr Missing Art" "warning"
|
||||
exit 1
|
||||
fi
|
||||
echo " Reachable — $LIDARR_URL"
|
||||
log " Reachable — $LIDARR_URL"
|
||||
|
||||
START=$(date +%s)
|
||||
|
||||
@@ -238,7 +235,7 @@ echo "━━━ $ICON_SYNC Building Album Directory Map ━━━"
|
||||
declare -A ALBUM_DIR_MAP
|
||||
_artist_list=$(curl_json "$LIDARR_URL/api/v1/artist?apikey=$LIDARR_API_KEY")
|
||||
_map_artist_count=$(echo "$_artist_list" | jq '. | length')
|
||||
echo " Fetching track files for $_map_artist_count artists..."
|
||||
log " Fetching track files for $_map_artist_count artists..."
|
||||
|
||||
while IFS= read -r _artist_id; do
|
||||
[[ -z "$_artist_id" ]] && continue
|
||||
@@ -250,7 +247,7 @@ while IFS= read -r _artist_id; do
|
||||
done < <(echo "$_artist_list" | jq -r '.[].id')
|
||||
unset _artist_list _map_artist_count _artist_id _album_id _track_path
|
||||
|
||||
echo " Mapped ${#ALBUM_DIR_MAP[@]} albums with local tracks"
|
||||
log " Mapped ${#ALBUM_DIR_MAP[@]} albums with local tracks"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Albums ━━━
|
||||
@@ -267,7 +264,7 @@ if [[ -z "$albums" || "$albums" == "null" ]]; then
|
||||
fi
|
||||
|
||||
total_albums=$(echo "$albums" | jq '. | length')
|
||||
echo " Processing $total_albums albums..."
|
||||
log " Processing $total_albums albums..."
|
||||
|
||||
while IFS=$'\t' read -r mbid artist_name album_name album_id; do
|
||||
raw_dir="${ALBUM_DIR_MAP[$album_id]:-}"
|
||||
@@ -335,7 +332,7 @@ wait
|
||||
ALBUM_FETCHED=$(wc -l < "$LIDARR_TMP/album_fetches" 2>/dev/null || echo 0)
|
||||
ALBUM_FAILED=$(wc -l < "$LIDARR_TMP/album_fails" 2>/dev/null || echo 0)
|
||||
ALBUM_MISSING=$(( ALBUMS_CHECKED - ALBUMS_COMPLETE ))
|
||||
echo " Checked: $ALBUMS_CHECKED | Complete: $ALBUMS_COMPLETE | Needed art: $ALBUM_MISSING | Fetched: $ALBUM_FETCHED | Failed: $ALBUM_FAILED"
|
||||
log " Checked: $ALBUMS_CHECKED | Complete: $ALBUMS_COMPLETE | Needed art: $ALBUM_MISSING | Fetched: $ALBUM_FETCHED | Failed: $ALBUM_FAILED"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Artists ━━━
|
||||
@@ -352,7 +349,7 @@ if [[ -z "$artists" || "$artists" == "null" ]]; then
|
||||
fi
|
||||
|
||||
total_artists=$(echo "$artists" | jq '. | length')
|
||||
echo " Processing $total_artists artists..."
|
||||
log " Processing $total_artists artists..."
|
||||
|
||||
while IFS=$'\t' read -r local_path mbid name; do
|
||||
local_path=$(translate_path "$local_path")
|
||||
@@ -430,7 +427,7 @@ wait
|
||||
ARTIST_FETCHED=$(wc -l < "$LIDARR_TMP/artist_fetches" 2>/dev/null || echo 0)
|
||||
ARTIST_FAILED=$(wc -l < "$LIDARR_TMP/artist_fails" 2>/dev/null || echo 0)
|
||||
ARTIST_MISSING=$(( ARTISTS_CHECKED - ARTISTS_COMPLETE ))
|
||||
echo " Checked: $ARTISTS_CHECKED | Complete: $ARTISTS_COMPLETE | Needed art: $ARTIST_MISSING | Fetched: $ARTIST_FETCHED | Failed: $ARTIST_FAILED"
|
||||
log " Checked: $ARTISTS_CHECKED | Complete: $ARTISTS_COMPLETE | Needed art: $ARTIST_MISSING | Fetched: $ARTIST_FETCHED | Failed: $ARTIST_FAILED"
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
|
||||
+30
-20
@@ -5,10 +5,10 @@
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Remove junk files from media shares using configurable file patterns. Two
|
||||
# profiles — anime and media — each with their own folder list and patterns.
|
||||
# Runs second in the daily maintenance window, after permissions and before
|
||||
# arr cleanup, so orphan detection only encounters actual media files.
|
||||
# Remove scene debris, tool artifacts, and unsafe files from media shares
|
||||
# before the orphan scan — so arr cleanup only encounters actual media files.
|
||||
# Runs second in the daily window, after permissions and before arr cleanup.
|
||||
# Two profiles — anime and media — each with their own folder list and patterns.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
@@ -18,15 +18,29 @@
|
||||
# anime — cleans ANIME_CLEAN_FOLDERS using ANIME_FILE_PATTERNS
|
||||
# media — cleans MEDIA_CLEAN_FOLDERS using MEDIA_FILE_PATTERNS (adds *.iso *.lrc)
|
||||
#
|
||||
# Removes junk left by download clients, scene releases, and tools:
|
||||
# *.sfv *.md5 *.sha1 — checksum files — useless post-download
|
||||
# *.nfo *.url *.lnk — scene info files — not needed in media library
|
||||
# *.rar *.zip — archives — source files not needed after extraction
|
||||
# *.sample* *.proof* — scene samples — never needed
|
||||
# *sync-conflict* — Syncthing conflict files
|
||||
# *.scr *.exe — executables — should never be in a media folder
|
||||
# *.torrent — torrent files left by download clients
|
||||
# *.log *.json — tool output files
|
||||
# Scene debris — left by scene releases and download clients:
|
||||
# *.sfv *.md5 *.sha1 — checksums — useless post-download
|
||||
# *.nzb *.torrent — download files left by clients
|
||||
# *.url *.lnk *.info *.diz — scene metadata
|
||||
# *.sample* *.proof* — scene samples — never needed in library
|
||||
# *sync-conflict* — Syncthing conflict copies
|
||||
# *.rar *.zip *.7z *.ace — archives — source not needed after extraction
|
||||
# *.r00-*.r09 *.srr — multi-part rar segments and repair files
|
||||
# *.001 *.002 *.003 — split archive parts
|
||||
# *.gz *.tar *.bz2 — linux archives
|
||||
#
|
||||
# Tool artifacts — incomplete or stale files from download clients:
|
||||
# *.!ut *.!qB — uTorrent / qBittorrent incomplete markers
|
||||
# *.crdownload *.opdownload — Chrome / Opera incomplete downloads
|
||||
# *.part — partial download files
|
||||
#
|
||||
# Unsafe files — executables that should never appear in a media folder:
|
||||
# *.exe *.scr *.com — Windows executables
|
||||
# *.bat *.cmd *.vbs *.ps1 — Windows scripts
|
||||
# *.msi *.dll *.sys — Windows system files
|
||||
# *.sh — shell scripts in media folders = suspicious
|
||||
#
|
||||
# Media profile also removes: *.iso *.lrc
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
@@ -37,13 +51,12 @@
|
||||
# Empty array guards — warns and exits cleanly if no folders or patterns configured
|
||||
# Folder existence — skips missing folders with warning, continues others
|
||||
# validate_unraid_cmd — notify script validated before use
|
||||
# Silent by default — only problems and removals produce output
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_ANIME_CLEAN_FOLDERS — folders cleaned by the anime profile on this host
|
||||
# HOST*_MEDIA_CLEAN_FOLDERS — folders cleaned by the media profile on this host
|
||||
@@ -88,9 +101,6 @@ parse_args "${RAW_ARGS[@]}"
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -131,7 +141,7 @@ esac
|
||||
# Empty array guards
|
||||
if [[ ${#CLEAN_FOLDERS[@]} -eq 0 ]]; then
|
||||
warn "No folders configured for profile '$PROFILE' on $MY_ID"
|
||||
warn "Check HOST*_${PROFILE^^}_CLEAN_FOLDERS in master_host*.conf"
|
||||
warn "Check HOST*_${PROFILE^^}_CLEAN_FOLDERS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -247,7 +257,7 @@ elif [[ ${#FAILED[@]} -gt 0 ]]; then
|
||||
notify "Media cleaner ($PROFILE) failed on $(hostname) — ${FAILED[*]}" \
|
||||
"Media Cleaner" "warning"
|
||||
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
|
||||
log "$ICON_DONE Status: clean — nothing to remove"
|
||||
echo "$ICON_DONE Status: clean — nothing to remove"
|
||||
else
|
||||
warn "$ICON_TRASH Removed: $TOTAL_REMOVED file(s)"
|
||||
notify "Media cleaner ($PROFILE) on $(hostname) — $TOTAL_REMOVED file(s) removed" \
|
||||
|
||||
@@ -34,7 +34,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_MEDIA_PERMISSION_SHARES — shares this host applies permissions to
|
||||
# Aliased by detect_hosts() — script uses MEDIA_PERMISSION_SHARES
|
||||
@@ -65,9 +65,6 @@ parse_args "$@"
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -86,7 +83,7 @@ detect_hosts
|
||||
# Empty array guard
|
||||
if [[ ${#MEDIA_PERMISSION_SHARES[@]} -eq 0 ]]; then
|
||||
warn "MEDIA_PERMISSION_SHARES is empty for $MY_ID — nothing to do"
|
||||
warn "Check HOST*_MEDIA_PERMISSION_SHARES in master_host*.conf"
|
||||
warn "Check HOST*_MEDIA_PERMISSION_SHARES in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
|
||||
@@ -57,7 +57,7 @@
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Last.fm API key — required for both stages
|
||||
# Configure HOST*_LASTFM_API_KEY in master_host*.conf
|
||||
# Configure HOST*_LASTFM_API_KEY in host*.conf
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION (master.conf)
|
||||
@@ -135,18 +135,18 @@ if ! command -v jq >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then
|
||||
error "LIDARR_URL / LIDARR_API_KEY not configured — check master_host*.conf"
|
||||
error "LIDARR_URL / LIDARR_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${LASTFM_API_KEY:-}" ]]; then
|
||||
error "LASTFM_API_KEY not configured — required for discovery"
|
||||
error "Configure HOST*_LASTFM_API_KEY in master_host*.conf"
|
||||
error "Configure HOST*_LASTFM_API_KEY in host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@
|
||||
# ==============================================================================================
|
||||
#
|
||||
# TMDB API key — required for Stage 2 recommendations
|
||||
# Configure HOST*_TMDB_API_KEY in master_host*.conf
|
||||
# Configure HOST*_TMDB_API_KEY in host*.conf
|
||||
# Free key at: https://www.themoviedb.org/settings/api
|
||||
#
|
||||
# ==============================================================================================
|
||||
@@ -136,19 +136,19 @@ if ! command -v jq >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${RADARR_URL:-}" || -z "${RADARR_API_KEY:-}" ]]; then
|
||||
error "RADARR_URL / RADARR_API_KEY not configured — check master_host*.conf"
|
||||
error "RADARR_URL / RADARR_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${TMDB_API_KEY:-}" ]]; then
|
||||
error "TMDB_API_KEY not configured — required for discovery"
|
||||
error "Get a free key at https://www.themoviedb.org/settings/api"
|
||||
error "Configure HOST*_TMDB_API_KEY in master_host*.conf"
|
||||
error "Configure HOST*_TMDB_API_KEY in host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
@@ -62,7 +62,7 @@
|
||||
# ==============================================================================================
|
||||
#
|
||||
# TMDB API key — required for Stage 2 recommendations and external_ids lookup
|
||||
# Configure HOST*_TMDB_API_KEY in master_host*.conf
|
||||
# Configure HOST*_TMDB_API_KEY in host*.conf
|
||||
# Free key at: https://www.themoviedb.org/settings/api
|
||||
#
|
||||
# ==============================================================================================
|
||||
@@ -150,19 +150,19 @@ if ! command -v jq >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${SONARR_URL:-}" || -z "${SONARR_API_KEY:-}" ]]; then
|
||||
error "SONARR_URL / SONARR_API_KEY not configured — check master_host*.conf"
|
||||
error "SONARR_URL / SONARR_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${TMDB_API_KEY:-}" ]]; then
|
||||
error "TMDB_API_KEY not configured — required for discovery"
|
||||
error "Get a free key at https://www.themoviedb.org/settings/api"
|
||||
error "Configure HOST*_TMDB_API_KEY in master_host*.conf"
|
||||
error "Configure HOST*_TMDB_API_KEY in host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
+6
-10
@@ -53,7 +53,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY / HOST*_RADARR_MOVIES_ROOT
|
||||
# HOST*_RADARR_PATH_MAP — container path → host path translation
|
||||
@@ -125,9 +125,6 @@ fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -333,7 +330,7 @@ SCAN_CMD_ID=$(echo "$SCAN_RESPONSE" | jq -r '.id // empty' 2>/dev/null)
|
||||
if [[ -z "$SCAN_CMD_ID" ]]; then
|
||||
warn "Could not trigger import scan — proceeding without pre-flight"
|
||||
else
|
||||
echo " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
|
||||
log " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
|
||||
POLL_TIMEOUT=${RADARR_IMPORT_SCAN_TIMEOUT:-600}
|
||||
POLLED=0
|
||||
while [[ "$POLLED" -lt "$POLL_TIMEOUT" ]]; do
|
||||
@@ -368,7 +365,7 @@ fi
|
||||
# Safety Layer 3 — API version check
|
||||
check_arr_version "$RADARR_URL" "$RADARR_API_KEY" "v3" "$RADARR_VERSION_MAJOR" "Radarr" || exit 1
|
||||
|
||||
echo " Querying Radarr API..."
|
||||
log " Querying Radarr API..."
|
||||
|
||||
# Fetch all movies
|
||||
MOVIES_RESPONSE=$(radarr_api "movie") || {
|
||||
@@ -433,15 +430,14 @@ if [[ "$TRACKED_COUNT" -eq 0 ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo " $MOVIE_COUNT movies | $TRACKED_COUNT tracked movie files"
|
||||
log " $MOVIE_COUNT movies | $TRACKED_COUNT tracked movie files"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Scan Movies Root ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_CLEAN Scanning Movies Root ━━━"
|
||||
echo " Root: $RADARR_MOVIES_ROOT | Orphan age: ${RADARR_ORPHAN_AGE} days"
|
||||
echo ""
|
||||
log " Root: $RADARR_MOVIES_ROOT | Orphan age: ${RADARR_ORPHAN_AGE} days"
|
||||
|
||||
START=$(date +%s)
|
||||
ORPHAN_COUNT=0
|
||||
@@ -571,7 +567,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no files deleted"
|
||||
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
|
||||
log "$ICON_DONE Clean — nothing to remove"
|
||||
echo "$ICON_DONE Clean — nothing to remove"
|
||||
else
|
||||
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
|
||||
notify "Radarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
#
|
||||
# RADARR_DROPPED_ADD_EXCLUSION — add removed movies to import exclusion (default: true)
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST1_RADARR_URL / HOST1_RADARR_API_KEY
|
||||
# Aliased by detect_hosts() — script uses RADARR_URL / RADARR_API_KEY
|
||||
@@ -76,9 +76,6 @@ parse_args "${FILTERED_ARGS[@]}"
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -176,9 +173,9 @@ while IFS=$'\t' read -r id title year tmdb_id has_file file_size; do
|
||||
SIZE_HUMAN=""
|
||||
if [[ "$has_file" == "true" && "$file_size" -gt 0 ]]; then
|
||||
SIZE_HUMAN=$(awk "BEGIN {printf \"%.1fGB\", $file_size / 1073741824}")
|
||||
echo "$ICON_WARN $title ($year) [tmdbid $tmdb_id] — HAS FILE: $SIZE_HUMAN"
|
||||
log "$ICON_WARN $title ($year) [tmdbid $tmdb_id] — HAS FILE: $SIZE_HUMAN"
|
||||
else
|
||||
echo "$ICON_TRASH $title ($year) [tmdbid $tmdb_id] — no file"
|
||||
log "$ICON_TRASH $title ($year) [tmdbid $tmdb_id] — no file"
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
@@ -242,7 +239,7 @@ echo "$ICON_TRASH Removed: ${#REMOVED[@]} of $DROPPED"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ ${#FAILED[@]} -eq 0 ]]; then
|
||||
log "$ICON_DONE Status: done ✅"
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
else
|
||||
warn "Status: ${#FAILED[@]} removal(s) failed"
|
||||
fi
|
||||
|
||||
+6
-10
@@ -53,7 +53,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_SONARR_URL / HOST*_SONARR_API_KEY / HOST*_SONARR_TV_ROOT
|
||||
# HOST*_SONARR_PATH_MAP — container path → host path translation
|
||||
@@ -125,9 +125,6 @@ fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -333,7 +330,7 @@ SCAN_CMD_ID=$(echo "$SCAN_RESPONSE" | jq -r '.id // empty' 2>/dev/null)
|
||||
if [[ -z "$SCAN_CMD_ID" ]]; then
|
||||
warn "Could not trigger import scan — proceeding without pre-flight"
|
||||
else
|
||||
echo " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
|
||||
log " Import scan queued (command ID: $SCAN_CMD_ID) — waiting for completion..."
|
||||
POLL_TIMEOUT=${SONARR_IMPORT_SCAN_TIMEOUT:-600}
|
||||
POLLED=0
|
||||
while [[ "$POLLED" -lt "$POLL_TIMEOUT" ]]; do
|
||||
@@ -368,7 +365,7 @@ fi
|
||||
# Safety Layer 3 — API version check
|
||||
check_arr_version "$SONARR_URL" "$SONARR_API_KEY" "v3" "$SONARR_VERSION_MAJOR" "Sonarr" || exit 1
|
||||
|
||||
echo " Querying Sonarr API..."
|
||||
log " Querying Sonarr API..."
|
||||
|
||||
# Fetch all series
|
||||
SERIES_RESPONSE=$(sonarr_api "series") || {
|
||||
@@ -432,15 +429,14 @@ if [[ "$TRACKED_COUNT" -eq 0 ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo " $SERIES_COUNT series | $TRACKED_COUNT tracked episode files"
|
||||
log " $SERIES_COUNT series | $TRACKED_COUNT tracked episode files"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Scan TV Root ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_CLEAN Scanning TV Root ━━━"
|
||||
echo " Root: $SONARR_TV_ROOT | Orphan age: ${SONARR_ORPHAN_AGE} days"
|
||||
echo ""
|
||||
log " Root: $SONARR_TV_ROOT | Orphan age: ${SONARR_ORPHAN_AGE} days"
|
||||
|
||||
START=$(date +%s)
|
||||
ORPHAN_COUNT=0
|
||||
@@ -570,7 +566,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no files deleted"
|
||||
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
|
||||
log "$ICON_DONE Clean — nothing to remove"
|
||||
echo "$ICON_DONE Clean — nothing to remove"
|
||||
else
|
||||
warn "$ICON_DONE Removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)"
|
||||
notify "Sonarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" \
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
#
|
||||
# SONARR_DROPPED_ADD_EXCLUSION — add removed series to import exclusion (default: true)
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST1_SONARR_URL / HOST1_SONARR_API_KEY
|
||||
# Aliased by detect_hosts() — script uses SONARR_URL / SONARR_API_KEY
|
||||
@@ -75,9 +75,6 @@ parse_args "${FILTERED_ARGS[@]}"
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -178,9 +175,9 @@ while IFS=$'\t' read -r id title year tvdb_id episode_file_count size_on_disk; d
|
||||
fi
|
||||
|
||||
if [[ "$episode_file_count" -gt 0 ]]; then
|
||||
echo "$ICON_WARN $title ($year) [tvdbid $tvdb_id] — $episode_file_count episode files${SIZE_HUMAN:+, $SIZE_HUMAN}"
|
||||
log "$ICON_WARN $title ($year) [tvdbid $tvdb_id] — $episode_file_count episode files${SIZE_HUMAN:+, $SIZE_HUMAN}"
|
||||
else
|
||||
echo "$ICON_TRASH $title ($year) [tvdbid $tvdb_id] — no files"
|
||||
log "$ICON_TRASH $title ($year) [tvdbid $tvdb_id] — no files"
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
@@ -244,7 +241,7 @@ echo "$ICON_TRASH Removed: ${#REMOVED[@]} of $DROPPED"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ ${#FAILED[@]} -eq 0 ]]; then
|
||||
log "$ICON_DONE Status: done ✅"
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
else
|
||||
warn "Status: ${#FAILED[@]} removal(s) failed"
|
||||
fi
|
||||
|
||||
+29
-14
@@ -13,7 +13,7 @@ Each domain and subdomain has an independent TLS certificate and must be listed
|
||||
separately. `cert_monitor.sh` makes one openssl connection per entry.
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
HOST1_CERT_MONITOR_DOMAINS=(
|
||||
"Gmer4Lfe.com"
|
||||
"Gmer4Lfe.us"
|
||||
@@ -21,7 +21,7 @@ HOST1_CERT_MONITOR_DOMAINS=(
|
||||
# "cloud.Gmer4Lfe.com"
|
||||
)
|
||||
|
||||
# master_host2.conf
|
||||
# host2.conf
|
||||
HOST2_CERT_MONITOR_DOMAINS=(
|
||||
"Jayred365.com"
|
||||
# "auth.Jayred365.com"
|
||||
@@ -63,12 +63,12 @@ CERT_TIMEOUT=10 # seconds per domain before declaring FAILED
|
||||
### Drive Ignore List
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
HOST1_SMART_IGNORE_DRIVES=(
|
||||
"sda" # boot USB flash drive — no meaningful SMART data
|
||||
)
|
||||
|
||||
# master_host2.conf
|
||||
# host2.conf
|
||||
HOST2_SMART_IGNORE_DRIVES=(
|
||||
"sda" # boot USB flash drive
|
||||
)
|
||||
@@ -109,7 +109,7 @@ if `dynamix.cfg` is not found (e.g., running outside of unRAID).
|
||||
### Share Configuration
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
HOST1_BACKUP_VERIFY_SHARES=(
|
||||
# empty — uses HOST1_DAILY_SYNC_SHARES automatically
|
||||
# "/mnt/user/Movies" # override to check specific shares only
|
||||
@@ -230,9 +230,9 @@ data — it just doesn't trigger a notification for that condition.
|
||||
| State File | Source | What It Shows |
|
||||
|-----------|--------|---------------|
|
||||
| `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
|
||||
| `SYS_WATCHDOG_FAILED_FILE` | `Docker_Essentials/docker_watchdog.sh` | Container skip list — needs human attention |
|
||||
| `WATCHDOG_STATE_FILE` | `Docker_Essentials/docker_watchdog.sh` | Active container strike counts |
|
||||
| `SYS_WATCHDOG_STATE_FILE` | `unRAID_Essentials/system_watchdog.sh` | Active system watchdog strikes |
|
||||
| `SYS_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
|
||||
| `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts |
|
||||
| `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/system_watchdog.sh` | Active system watchdog strikes |
|
||||
| `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |
|
||||
| `TRANSCODE_DAILY_LOG` | `Transcodes/` | Weekly transcode statistics |
|
||||
| `CERT_MONITOR_DOMAINS` | live openssl check | Current cert status per domain |
|
||||
@@ -244,11 +244,11 @@ data — it just doesn't trigger a notification for that condition.
|
||||
### Emby Credentials
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
HOST1_EMBY_URL="http://192.168.50.2:8096"
|
||||
HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829"
|
||||
|
||||
# master_host2.conf
|
||||
# host2.conf
|
||||
HOST2_EMBY_URL="http://192.168.50.3:8096"
|
||||
HOST2_EMBY_API_KEY="<host2_api_key>"
|
||||
```
|
||||
@@ -306,7 +306,7 @@ in `unRAID_Essentials/`.
|
||||
### Pool Ignore List
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||||
"disk10" # JBOD member — high usage expected, exclude from report noise
|
||||
"disk9"
|
||||
@@ -315,7 +315,7 @@ HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||||
"disk5"
|
||||
)
|
||||
|
||||
# master_host2.conf
|
||||
# host2.conf
|
||||
HOST2_ZFS_REPORT_IGNORE_POOLS=(
|
||||
# list host2's JBOD members here
|
||||
)
|
||||
@@ -392,7 +392,7 @@ ZFS_REPORT_AVAIL_WARN_GB=20
|
||||
ZFS_REPORT_DOCKER_TOP=10
|
||||
```
|
||||
|
||||
### master_host*.conf
|
||||
### host*.conf
|
||||
|
||||
```bash
|
||||
# cert_monitor.sh
|
||||
@@ -445,6 +445,21 @@ HOST2_ZFS_REPORT_IGNORE_POOLS=()
|
||||
|
||||
---
|
||||
|
||||
## ━━━ OUTPUT TIERS ━━━
|
||||
|
||||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||||
lines only appear when `--log` is passed.
|
||||
|
||||
Most monitors are one-shot scripts that run on a schedule. Without `--log`, phase
|
||||
headers, step conclusions (pre-flight passed, snapshot written, digest sent), and
|
||||
the final status line are visible. Per-drive, per-cert, per-file, and per-domain
|
||||
detail lines inside loops are suppressed.
|
||||
|
||||
`weekly_health_digest.sh` with `profile=smart` exits silently (one `echo` line)
|
||||
when there are no findings worth reporting — no noise on healthy weeks.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FLAG REFERENCE ━━━
|
||||
|
||||
All monitor scripts support the same four flags:
|
||||
@@ -470,7 +485,7 @@ emby_session_report.sh --dry-run # test connectivity, generate report, no noti
|
||||
### --status
|
||||
|
||||
Shows current configuration and exits without running checks. Use to verify
|
||||
configuration is loaded correctly after editing `master.conf` or `master_host*.conf`.
|
||||
configuration is loaded correctly after editing `master.conf` or `host*.conf`.
|
||||
|
||||
```bash
|
||||
cert_monitor.sh --status # domain list, CERT_WARN_DAYS, CERT_CRIT_DAYS, timeout
|
||||
|
||||
@@ -63,7 +63,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_BACKUP_VERIFY_SHARES
|
||||
# Shares to verify. Leave empty to use HOST*_DAILY_SYNC_SHARES automatically.
|
||||
@@ -140,7 +140,7 @@ fi
|
||||
|
||||
if [[ ${#VERIFY_SHARES[@]} -eq 0 ]]; then
|
||||
warn "No shares configured for $MY_ID — nothing to verify"
|
||||
warn "Check HOST*_BACKUP_VERIFY_SHARES or HOST*_DAILY_SYNC_SHARES in master_host*.conf"
|
||||
warn "Check HOST*_BACKUP_VERIFY_SHARES or HOST*_DAILY_SYNC_SHARES in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -193,7 +193,7 @@ if ! check_remote_array; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log "Pre-flight passed ✅"
|
||||
echo "Pre-flight passed ✅"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Backup Verification ━━━
|
||||
@@ -319,7 +319,7 @@ elif [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 ]]; then
|
||||
notify "Backup verify FAILED on $(hostname) → $REMOTE_SERVER_NAME — mismatches: $TOTAL_MISMATCH missing: $TOTAL_MISSING — shares: ${SHARES_WITH_ISSUES[*]}" \
|
||||
"Backup Verify" "warning"
|
||||
else
|
||||
log "$ICON_DONE Status: all $TOTAL_CHECKED files match across ${#VERIFY_SHARES[@]} shares ✅"
|
||||
echo "$ICON_DONE Status: all $TOTAL_CHECKED files match across ${#VERIFY_SHARES[@]} shares ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
@@ -90,9 +90,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Monitor script — output is the point
|
||||
SILENT_MODE=false
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ── Parse mode from PARSED_ARGS ───────────────────────────────────────────────────────────────
|
||||
@@ -188,7 +185,7 @@ if [[ "$LOG_TRANSFER_MODE" == true ]]; then
|
||||
# Append entry — format: date|time|profile|duration|status|bytes|warn_flag
|
||||
echo "${TODAY}|${NOW}|${TRANSFER_PROFILE}|${TRANSFER_DURATION}|${TRANSFER_STATUS}|${TRANSFER_BYTES}|${WARN_FLAG}" \
|
||||
>> "$BANDWIDTH_LOG"
|
||||
log "$ICON_BANDWIDTH Logged: $TRANSFER_PROFILE — ${DURATION_FMT} — $TRANSFER_STATUS${WARN_FLAG:+ [$WARN_FLAG]}"
|
||||
echo "$ICON_BANDWIDTH Logged: $TRANSFER_PROFILE — ${DURATION_FMT} — $TRANSFER_STATUS${WARN_FLAG:+ [$WARN_FLAG]}"
|
||||
|
||||
# Trim entries older than retention — atomic write via temp file
|
||||
CUTOFF=$(date -d "${BANDWIDTH_LOG_RETENTION} days ago" '+%Y-%m-%d')
|
||||
|
||||
@@ -51,7 +51,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_CERT_MONITOR_DOMAINS
|
||||
# Domains this host monitors. Each domain and subdomain is a separate entry —
|
||||
@@ -91,9 +91,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Monitor script — output is the point
|
||||
SILENT_MODE=false
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -126,7 +123,7 @@ detect_hosts
|
||||
# Empty array guard
|
||||
if [[ ${#CERT_MONITOR_DOMAINS[@]} -eq 0 ]]; then
|
||||
warn "CERT_MONITOR_DOMAINS is empty for $MY_ID"
|
||||
warn "Check HOST*_CERT_MONITOR_DOMAINS in master_host*.conf"
|
||||
warn "Check HOST*_CERT_MONITOR_DOMAINS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -277,7 +274,7 @@ elif [[ ${#CRITICAL[@]} -gt 0 || ${#FAILED[@]} -gt 0 ]]; then
|
||||
elif [[ ${#WARNING[@]} -gt 0 ]]; then
|
||||
warn "Status: WARNINGS — renewal recommended"
|
||||
else
|
||||
log "$ICON_DONE Status: all ${#HEALTHY[@]} certs healthy ✅"
|
||||
echo "$ICON_DONE Status: all ${#HEALTHY[@]} certs healthy ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_EMBY_URL
|
||||
# Emby server URL for this host. Aliased by detect_hosts() → EMBY_URL.
|
||||
@@ -87,9 +87,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Monitor/report script — output is the point
|
||||
SILENT_MODE=false
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -43,7 +43,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_SMART_IGNORE_DRIVES
|
||||
# Drives skipped in SMART monitoring. Aliased by detect_hosts() →
|
||||
@@ -80,9 +80,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Monitor script — output is the point
|
||||
SILENT_MODE=false
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -367,7 +364,7 @@ elif [[ ${#DRIVES_WARN[@]} -gt 0 ]]; then
|
||||
notify "SMART WARNING on $(hostname) — drives showing wear: ${DRIVES_WARN[*]}" \
|
||||
"SMART Health" "warning"
|
||||
else
|
||||
log "$ICON_DONE Status: all ${#DRIVES_OK[@]} drives healthy ✅"
|
||||
echo "$ICON_DONE Status: all ${#DRIVES_OK[@]} drives healthy ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
@@ -245,4 +245,4 @@ fi
|
||||
echo "${DATE}|${TIME}|${INOTIFY_USED}|${INOTIFY_LIMIT}|${INOTIFY_PCT}|${INOTIFY_WARN}|${PHPFPM_ACTIVE}|${PHPFPM_MAX}|${PHPFPM_PCT}|${PHPFPM_WARN}" \
|
||||
>> "$TUNING_MONITOR_LOG"
|
||||
|
||||
log "Snapshot written: inotify ${INOTIFY_PCT}% php-fpm ${PHPFPM_PCT}%"
|
||||
echo "Snapshot written: inotify ${INOTIFY_PCT}% php-fpm ${PHPFPM_PCT}%"
|
||||
@@ -116,9 +116,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Report/monitor script — output is the point when sending
|
||||
SILENT_MODE=false
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -341,7 +338,7 @@ fi
|
||||
# ── Smart profile — exit silently if nothing to report ────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
if [[ "$DIGEST_PROFILE" == "smart" && "$SHOULD_SEND" == false ]]; then
|
||||
log "Profile: smart — no findings worth reporting — silent exit"
|
||||
echo "Profile: smart — no findings worth reporting — silent exit"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -378,5 +375,5 @@ if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — digest generated but not sent"
|
||||
elif [[ "$SHOULD_SEND" == true ]]; then
|
||||
notify "$NOTIFY_MSG" "Health Digest" "$NOTIFY_SEV"
|
||||
log "Digest sent"
|
||||
echo "Digest sent"
|
||||
fi
|
||||
@@ -67,7 +67,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_ZFS_REPORT_IGNORE_POOLS
|
||||
# Pools excluded from health reporting. Single-disk JBOD members generate
|
||||
@@ -113,9 +113,6 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Monitor/report script — output is the point
|
||||
SILENT_MODE=false
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
DOCKER_TIMEOUT=15
|
||||
@@ -227,7 +224,7 @@ else
|
||||
error "One or more ZFS pools are NOT ONLINE: $UNHEALTHY"
|
||||
WARNINGS+=("ZFS pool unhealthy: $UNHEALTHY")
|
||||
else
|
||||
log "All monitored ZFS pools are ONLINE ✅"
|
||||
echo "All monitored ZFS pools are ONLINE ✅"
|
||||
fi
|
||||
|
||||
if [[ ${#ZFS_REPORT_IGNORE_POOLS[@]} -gt 0 ]]; then
|
||||
@@ -349,7 +346,7 @@ echo "$ICON_ZFS Log: $ZFS_REPORT_LOG"
|
||||
echo ""
|
||||
|
||||
if [[ ${#WARNINGS[@]} -eq 0 ]]; then
|
||||
log "$ICON_DONE All checks within thresholds ✅"
|
||||
echo "$ICON_DONE All checks within thresholds ✅"
|
||||
else
|
||||
echo "$ICON_WARN Warnings: ${#WARNINGS[@]}"
|
||||
for w in "${WARNINGS[@]}"; do
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
#
|
||||
# ADDING A NEW ARR
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Add HOST*_<ARR>_URL, API_KEY, MEDIA_ROOT, PATH_MAP to master_host*.conf
|
||||
# 1. Add HOST*_<ARR>_URL, API_KEY, MEDIA_ROOT, PATH_MAP to host*.conf
|
||||
# 2. Add <ARR>_ORPHAN_AGE, MAX_DELETE_GB, EXTENSIONS, etc. to master.conf
|
||||
# 3. Add a profile entry to ARR_PROFILES in arr_cleanup.py (6 values)
|
||||
# 4. Add an export block for the new arr below (copy Sonarr block, change prefix)
|
||||
|
||||
@@ -142,6 +142,26 @@ Single notification → one bell per run, not one per job
|
||||
|
||||
---
|
||||
|
||||
## ━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||||
lines only appear when `--log` is passed.
|
||||
|
||||
**One-shot orchestrators** (`array_started.sh`, `array_stopping.sh`, `sunday_morning_coffee_report.sh`):
|
||||
without `--log`, section headers, per-phase results, and the final summary are
|
||||
visible. Per-item detail suppressed.
|
||||
|
||||
**Periodic orchestrators** (`critical_sync_maintenance.sh`, `daily_sync_maintenance.sh`,
|
||||
`intermediate_sync_maintenance.sh`, `weekly_sync_maintenance.sh`): without `--log`,
|
||||
phase headers, per-phase completion status, and the final summary are visible. Per-share
|
||||
and per-job detail suppressed.
|
||||
|
||||
**Daemon orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent
|
||||
during clean cycles. Only state transitions, errors, and startup-grace expiry shown
|
||||
without `--log`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
| Script | What It Orchestrates | Schedule |
|
||||
@@ -194,10 +214,10 @@ ARRAY_START_SCRIPTS=(
|
||||
# watchdogs check container states
|
||||
|
||||
# ── Continuous scripts — run until array stops ─────────────────────────────
|
||||
"unRAID_Essentials/system_watchdog.sh" # system health BEFORE docker watchdog —
|
||||
"Watchdogs/system_watchdog.sh" # system health BEFORE docker watchdog —
|
||||
# system watchdog writes state file that
|
||||
# docker watchdog reads every cycle
|
||||
"Docker_Essentials/docker_watchdog.sh" # container health BEFORE failover —
|
||||
"Watchdogs/docker_watchdog.sh" # container health BEFORE failover —
|
||||
# containers must be healthy for failover
|
||||
# to make reliable decisions
|
||||
"Failover/failover.sh" # failover LAST — needs everything else stable
|
||||
@@ -445,7 +465,7 @@ new search — hands-free recovery while you sleep.
|
||||
### ── Host Awareness ───────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf + master_host*.conf
|
||||
# master.conf + host*.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Each arr is independently toggled per host.
|
||||
# Lidarr only runs on HOST1 — exits cleanly on HOST2 with no action.
|
||||
@@ -560,7 +580,7 @@ DAILY_MAINTENANCE_SCRIPTS=(
|
||||
"Docker_Essentials/docker_daily_restart.sh" # POST-SYNC — restarts after everything
|
||||
)
|
||||
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
HOST1_DAILY_SYNC_SHARES=(
|
||||
"/mnt/user/Movies" # HOST1 source of truth — push to HOST2
|
||||
"/mnt/user/Tv_Shows" # HOST1 source of truth
|
||||
@@ -575,7 +595,7 @@ HOST1_PERSONAL_SHARES=(
|
||||
"/mnt/user/Personal" # encrypted personal share — appended after standard
|
||||
)
|
||||
|
||||
# master_host2.conf
|
||||
# host2.conf
|
||||
HOST2_DAILY_SYNC_SHARES=(
|
||||
"/mnt/user/Anime_Shows" # HOST2 source of truth — push to HOST1
|
||||
"/mnt/user/Anime_Movies" # HOST2 source of truth
|
||||
|
||||
@@ -18,8 +18,8 @@
|
||||
# Docker_Essentials/docker_network_connect.sh — ensure networks + container connections
|
||||
#
|
||||
# CONTINUOUS (run until array stops):
|
||||
# unRAID_Essentials/system_watchdog.sh — system health monitor (last line of defense)
|
||||
# Docker_Essentials/docker_watchdog.sh — container health monitor
|
||||
# Watchdogs/system_watchdog.sh — system health monitor (last line of defense)
|
||||
# Watchdogs/docker_watchdog.sh — container health monitor
|
||||
# Fallback/fallback.sh — mutual failover monitor
|
||||
#
|
||||
# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
|
||||
@@ -212,6 +212,6 @@ elif [[ "$FAILED" -gt 0 ]]; then
|
||||
notify "Array start on $(hostname) ($MY_ID) — $FAILED script(s) failed: ${FAILED_SCRIPTS[*]}" \
|
||||
"Array Start" "warning"
|
||||
else
|
||||
log "$ICON_DONE Status: all $LAUNCHED script(s) launched ✅"
|
||||
echo "$ICON_DONE Status: all $LAUNCHED script(s) launched ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -169,7 +169,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ ${#FAILED[@]} -eq 0 ]]; then
|
||||
log "$ICON_DONE Status: all $STEP step(s) complete ✅"
|
||||
echo "$ICON_DONE Status: all $STEP step(s) complete ✅"
|
||||
notify "Array stop complete on $(hostname) ($MY_ID) — $STEP step(s) done" \
|
||||
"Array Stop" "normal"
|
||||
else
|
||||
|
||||
@@ -120,12 +120,12 @@ PASS=()
|
||||
FAIL=()
|
||||
|
||||
if ! check_rsync_enabled "CRITICAL"; then
|
||||
log "Critical rsync disabled — skipping sync, running partnership check only"
|
||||
echo "Critical rsync disabled — skipping sync, running partnership check only"
|
||||
elif [[ ${#CRITICAL_SYNC_SHARES[@]} -eq 0 ]]; then
|
||||
warn "CRITICAL_RSYNC_ENABLED=true but CRITICAL_SYNC_SHARES is empty for $MY_ID"
|
||||
warn "Check HOST*_CRITICAL_SYNC_SHARES in master_host*.conf"
|
||||
warn "Check HOST*_CRITICAL_SYNC_SHARES in host*.conf"
|
||||
else
|
||||
log "Critical sync — $MY_ID → $REMOTE_ID — $(date '+%H:%M:%S')"
|
||||
echo "Critical sync — $MY_ID → $REMOTE_ID — $(date '+%H:%M:%S')"
|
||||
|
||||
# Build dry-run flag to pass through
|
||||
RSYNC_DRY=""
|
||||
@@ -204,7 +204,7 @@ if [[ "${PARTNERSHIP_ENABLED:-false}" == true ]]; then
|
||||
--check --remote-unseen $PARTNER_DRY
|
||||
fi
|
||||
else
|
||||
log "Partnership disabled — skipping check"
|
||||
echo "Partnership disabled — skipping check"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -219,14 +219,14 @@ if [[ ${#FAIL[@]} -gt 0 ]]; then
|
||||
echo "━━━━━ $ICON_SUMMARY CRITICAL SYNC SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $DURATION"
|
||||
[[ ${#PASS[@]} -gt 0 ]] && log "Synced: ${PASS[*]}"
|
||||
[[ ${#PASS[@]} -gt 0 ]] && echo "Synced: ${PASS[*]}"
|
||||
echo "$ICON_ERROR Failed: ${FAIL[*]}"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
notify "Critical sync failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
|
||||
"Critical Sync" "warning"
|
||||
exit 1
|
||||
else
|
||||
log "Critical sync complete — $MY_ID — ${DURATION} — ${#PASS[@]} share(s)"
|
||||
echo "Critical sync complete — $MY_ID — ${DURATION} — ${#PASS[@]} share(s)"
|
||||
fi
|
||||
|
||||
exit 0
|
||||
@@ -53,7 +53,7 @@
|
||||
# Summary always shown — gives window timing and share/job counts.
|
||||
# Notify only on failure — successful daily maintenance doesn't need notification.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf + master_host*.conf) ───────────────────────────────────────────
|
||||
# ── CONFIGURATION (master.conf + host*.conf) ───────────────────────────────────────────
|
||||
# HOST*_DAILY_SYNC_SHARES — shares pushed to mirror each day
|
||||
# HOST*_PERSONAL_SHARES — encrypted personal shares
|
||||
# DAILY_MAINTENANCE_SCRIPTS — maintenance jobs (permissions, cleanup, restart)
|
||||
@@ -213,7 +213,7 @@ echo "━━━ $ICON_SYNC Arr Sync ━━━"
|
||||
|
||||
ARR_SYNC_SCRIPT="$SCRIPTS_ROOT/Media/arr_sync.sh"
|
||||
if [[ "${ARR_SYNC_ENABLED:-true}" != "true" ]]; then
|
||||
log "ARR_SYNC_ENABLED=false — skipping"
|
||||
echo "ARR_SYNC_ENABLED=false — skipping"
|
||||
elif [[ ! -f "$ARR_SYNC_SCRIPT" ]]; then
|
||||
warn "arr_sync.sh not found at $ARR_SYNC_SCRIPT — skipping"
|
||||
JOB_FAIL+=("arr_sync.sh")
|
||||
@@ -221,7 +221,7 @@ else
|
||||
_arr_sync_args=()
|
||||
[[ "$DRY_RUN" == true ]] && _arr_sync_args+=("--dry-run")
|
||||
if bash "$ARR_SYNC_SCRIPT" "${_arr_sync_args[@]}"; then
|
||||
log "Arr sync complete ✅"
|
||||
echo "Arr sync complete ✅"
|
||||
JOB_PASS+=("arr_sync.sh")
|
||||
else
|
||||
warn "Arr sync completed with errors — continuing to rsync"
|
||||
@@ -244,7 +244,7 @@ if ! check_rsync_enabled "DAILY"; then
|
||||
warn "Daily rsync disabled — skipping all $SHARE_COUNT share syncs"
|
||||
warn "Proceeding to maintenance jobs..."
|
||||
elif [[ "$SHARE_COUNT" -eq 0 ]]; then
|
||||
warn "No shares configured for $MY_ID — check HOST*_DAILY_SYNC_SHARES in master_host*.conf"
|
||||
warn "No shares configured for $MY_ID — check HOST*_DAILY_SYNC_SHARES in host*.conf"
|
||||
else
|
||||
# Pre-flight — connectivity then remote rootfs
|
||||
check_connectivity
|
||||
@@ -354,7 +354,7 @@ if [[ "$TOTAL_FAIL" -gt 0 ]]; then
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 1
|
||||
else
|
||||
log "$ICON_DONE Status: all complete — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
|
||||
echo "$ICON_DONE Status: all complete — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
@@ -24,7 +24,7 @@
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_INTERMEDIATE_SYNC_SHARES → INTERMEDIATE_SYNC_SHARES.
|
||||
# Each server can have a different set of mid-day shares — configure in master_host*.conf.
|
||||
# Each server can have a different set of mid-day shares — configure in host*.conf.
|
||||
# Each script in INTERMEDIATE_MAINTENANCE_SCRIPTS handles its own host logic.
|
||||
#
|
||||
# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
|
||||
@@ -41,7 +41,7 @@
|
||||
# Silent on success — runs 4x/day, only failures warrant notification
|
||||
#
|
||||
# ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
|
||||
# master_host*.conf: HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped)
|
||||
# host*.conf: HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped)
|
||||
# master.conf: INTERMEDIATE_RSYNC_ENABLED — enable/disable rsync section (default: true)
|
||||
# master.conf: INTERMEDIATE_MAINTENANCE_SCRIPTS — jobs run after rsync
|
||||
# master.conf: ARR_SYNC_ENABLED — toggle inside arr_sync.sh
|
||||
@@ -162,7 +162,7 @@ echo "━━━ $ICON_SYNC Arr Sync ━━━"
|
||||
|
||||
ARR_SYNC_SCRIPT="$SCRIPTS_ROOT/Media/arr_sync.sh"
|
||||
if [[ "${ARR_SYNC_ENABLED:-true}" != "true" ]]; then
|
||||
log "ARR_SYNC_ENABLED=false — skipping"
|
||||
echo "ARR_SYNC_ENABLED=false — skipping"
|
||||
elif [[ ! -f "$ARR_SYNC_SCRIPT" ]]; then
|
||||
warn "arr_sync.sh not found at $ARR_SYNC_SCRIPT — skipping"
|
||||
JOB_FAIL+=("arr_sync.sh")
|
||||
@@ -170,7 +170,7 @@ else
|
||||
_arr_sync_args=()
|
||||
[[ "$DRY_RUN" == true ]] && _arr_sync_args+=("--dry-run")
|
||||
if bash "$ARR_SYNC_SCRIPT" "${_arr_sync_args[@]}"; then
|
||||
log "Arr sync complete ✅"
|
||||
echo "Arr sync complete ✅"
|
||||
JOB_PASS+=("arr_sync.sh")
|
||||
else
|
||||
warn "Arr sync completed with errors — continuing"
|
||||
@@ -192,8 +192,8 @@ SHARE_INDEX=0
|
||||
ABORT_ALL_SYNCS=false
|
||||
|
||||
if [[ "$SHARE_COUNT" -eq 0 ]]; then
|
||||
log "No INTERMEDIATE_SYNC_SHARES configured — skipping"
|
||||
log "Add shares to INTERMEDIATE_SYNC_SHARES in master.conf to enable mid-day sync"
|
||||
echo "No INTERMEDIATE_SYNC_SHARES configured — skipping"
|
||||
echo "Add shares to INTERMEDIATE_SYNC_SHARES in master.conf to enable mid-day sync"
|
||||
elif ! check_rsync_enabled "INTERMEDIATE"; then
|
||||
warn "Intermediate rsync disabled — skipping all $SHARE_COUNT share sync(s)"
|
||||
else
|
||||
@@ -300,7 +300,7 @@ TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
|
||||
log "$ICON_DONE Status: all complete ✅ — ${#JOB_PASS[@]} job(s) run, ${#PASS[@]}/$SHARE_COUNT share(s) synced"
|
||||
echo "$ICON_DONE Status: all complete ✅ — ${#JOB_PASS[@]} job(s) run, ${#PASS[@]}/$SHARE_COUNT share(s) synced"
|
||||
else
|
||||
warn "Status: $TOTAL_FAIL failure(s)"
|
||||
notify "Intermediate sync failed on $(hostname) ($MY_ID) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
|
||||
|
||||
@@ -154,6 +154,8 @@ line() { REPORT+=(" $1"); }
|
||||
issue() { ISSUES+=("$1"); REPORT+=(" ⚠️ $1"); }
|
||||
finding() { FINDINGS+=("$1"); REPORT+=(" ℹ️ $1"); }
|
||||
|
||||
get_array() { eval "echo \"\${${1}[*]}\""; }
|
||||
|
||||
format_bytes() {
|
||||
local bytes=$1
|
||||
if (( bytes > 1073741824 )); then
|
||||
@@ -656,6 +658,90 @@ if command -v tailscale >/dev/null 2>&1; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ 🔗 MESH ━━━
|
||||
# ==============================================================================================
|
||||
section "🔗 MESH"
|
||||
|
||||
_ALL_HOST_IDS=()
|
||||
for _h in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do
|
||||
[[ -n "${!_h:-}" ]] && _ALL_HOST_IDS+=("$_h")
|
||||
done
|
||||
|
||||
# ── Members ──
|
||||
if [[ ${#_ALL_HOST_IDS[@]} -eq 0 ]]; then
|
||||
line "No hosts defined"
|
||||
else
|
||||
line "Members:"
|
||||
for _h in "${_ALL_HOST_IDS[@]}"; do
|
||||
_server="${!_h}"
|
||||
_owner_var="${_h}_OWNER"; _owner="${!_owner_var:-unknown}"
|
||||
_email_var="${_h}_OWNER_EMAIL"; _email="${!_email_var:-(not set)}"
|
||||
line " $_h $_server / $_owner / $_email"
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Protected Services ──
|
||||
_coverage_found=false
|
||||
for _covered in "${_ALL_HOST_IDS[@]}"; do
|
||||
_covered_owner_var="${_covered}_OWNER"; _covered_owner="${!_covered_owner_var:-$_covered}"
|
||||
_covered_email_var="${_covered}_OWNER_EMAIL"; _covered_email="${!_covered_email_var:-}"
|
||||
|
||||
declare -A _tier_containers=()
|
||||
declare -A _tier_delays=()
|
||||
_covered_by=""
|
||||
_any_tiers=false
|
||||
|
||||
for _covering in "${_ALL_HOST_IDS[@]}"; do
|
||||
[[ "$_covering" == "$_covered" ]] && continue
|
||||
for _tier in 1 2 3 4; do
|
||||
_containers=$(get_array "FALLBACK_${_covering}_COVERS_${_covered}_TIER${_tier}")
|
||||
[[ -z "$_containers" ]] && continue
|
||||
_any_tiers=true
|
||||
_tier_containers[$_tier]="$_containers"
|
||||
_delay_var="${_covered}_TIER${_tier}_DELAY"
|
||||
_tier_delays[$_tier]="${!_delay_var:-0}"
|
||||
done
|
||||
if [[ "$_any_tiers" == true ]]; then
|
||||
_cov_owner_var="${_covering}_OWNER"; _cov_owner="${!_cov_owner_var:-$_covering}"
|
||||
_covered_by="$_covering ($_cov_owner)"
|
||||
fi
|
||||
done
|
||||
|
||||
[[ "$_any_tiers" == false ]] && { unset _tier_containers _tier_delays; declare -A _tier_containers=() _tier_delays=(); continue; }
|
||||
|
||||
_coverage_found=true
|
||||
_hdr="$_covered_owner"
|
||||
[[ -n "$_covered_email" ]] && _hdr+=" — $_covered_email"
|
||||
line "Protected: $_hdr"
|
||||
for _tier in 1 2 3 4; do
|
||||
[[ -z "${_tier_containers[$_tier]:-}" ]] && continue
|
||||
_d="${_tier_delays[$_tier]:-0}"
|
||||
if (( _d == 0 )); then _dlabel="immediate"
|
||||
elif (( _d >= 1440 )); then _dlabel="$(( _d / 1440 ))d"
|
||||
elif (( _d >= 60 )); then _dlabel="$(( _d / 60 ))hr"
|
||||
else _dlabel="${_d}min"
|
||||
fi
|
||||
line " Tier $_tier (${_dlabel}): ${_tier_containers[$_tier]// /, }"
|
||||
done
|
||||
[[ -n "$_covered_by" ]] && line " Covered by: $_covered_by"
|
||||
|
||||
unset _tier_containers _tier_delays
|
||||
declare -A _tier_containers=() _tier_delays=()
|
||||
done
|
||||
|
||||
[[ "$_coverage_found" == false ]] && line "No fallback coverage configured"
|
||||
|
||||
# ── Partnership ──
|
||||
if [[ "${PARTNERSHIP_ENABLED:-false}" == true ]]; then
|
||||
_po_host="${PARTNERSHIP_OWNER_HOST:-HOST1}"
|
||||
_po_server="${!_po_host:-unknown}"
|
||||
_po_name_var="${_po_host}_OWNER"; _po_name="${!_po_name_var:-unknown}"
|
||||
line "Partnership: enabled — owner $_po_host ($_po_server / $_po_name), sync every ${PARTNERSHIP_SYNC_INTERVAL:-15}min"
|
||||
else
|
||||
line "Partnership: disabled"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ 🔐 SECURITY ━━━
|
||||
# ==============================================================================================
|
||||
@@ -860,7 +946,7 @@ else
|
||||
NOTIFY_SCRIPT="/usr/local/emhttp/plugins/dynamix/scripts/notify"
|
||||
if [[ -x "$NOTIFY_SCRIPT" ]]; then
|
||||
"$NOTIFY_SCRIPT" -s "☕ Weekly Report — $MY_ID" -d "$BODY" -i "normal" 2>/dev/null
|
||||
log "unRAID notification sent"
|
||||
echo "unRAID notification sent"
|
||||
fi
|
||||
fi
|
||||
if [[ -n "${DISCORD_WEBHOOK:-}" ]]; then
|
||||
@@ -871,8 +957,8 @@ else
|
||||
PAYLOAD="{\"content\": ${ESCAPED_BODY}}"
|
||||
curl -sf -H "Content-Type: application/json" \
|
||||
-d "$PAYLOAD" "$DISCORD_WEBHOOK" >/dev/null 2>&1 && \
|
||||
log "Discord notification sent" || \
|
||||
echo "Discord notification sent" || \
|
||||
warn "Discord notification failed"
|
||||
fi
|
||||
log "Report generated — $MY_ID — ${#ISSUES[@]} issue(s) ${#FINDINGS[@]} finding(s)"
|
||||
echo "Report generated — $MY_ID — ${#ISSUES[@]} issue(s) ${#FINDINGS[@]} finding(s)"
|
||||
fi
|
||||
@@ -87,7 +87,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
|
||||
if [[ "$UPTIME_S" -lt "$WATCHDOG_STARTUP_GRACE" ]]; then
|
||||
warn "Within startup grace — $(format_duration $UPTIME_S) / $(format_duration $WATCHDOG_STARTUP_GRACE)"
|
||||
else
|
||||
log "Past startup grace — $(format_duration $UPTIME_S) uptime"
|
||||
echo "Past startup grace — $(format_duration $UPTIME_S) uptime"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
@@ -231,7 +231,7 @@ if [[ "$WEEKLY_SYNC_UPDATES" == true ]]; then
|
||||
done
|
||||
fi
|
||||
else
|
||||
log "WEEKLY_SYNC_UPDATES=false — skipping local updates"
|
||||
echo "WEEKLY_SYNC_UPDATES=false — skipping local updates"
|
||||
fi
|
||||
|
||||
if [[ "$WEEKLY_SYNC_UPDATES_REMOTE" == true ]]; then
|
||||
@@ -261,7 +261,7 @@ if [[ "$WEEKLY_SYNC_UPDATES_REMOTE" == true ]]; then
|
||||
done
|
||||
fi
|
||||
else
|
||||
log "WEEKLY_SYNC_UPDATES_REMOTE=false — skipping remote updates"
|
||||
echo "WEEKLY_SYNC_UPDATES_REMOTE=false — skipping remote updates"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -352,7 +352,7 @@ else
|
||||
_remainder_args=("--remainder")
|
||||
[[ "$DRY_RUN" == true ]] && _remainder_args+=("--dry-run")
|
||||
if bash "$DOCKER_UPDATE_SCRIPT" "${_remainder_args[@]}"; then
|
||||
log "Remainder updates complete ✅"
|
||||
echo "Remainder updates complete ✅"
|
||||
JOB_PASS+=("docker_update.sh --remainder")
|
||||
else
|
||||
warn "Remainder updates completed with errors"
|
||||
@@ -392,7 +392,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$TOTAL_FAIL" -eq 0 ]]; then
|
||||
log "$ICON_DONE Status: all complete ✅ — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
|
||||
echo "$ICON_DONE Status: all complete ✅ — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
|
||||
else
|
||||
warn "Status: $TOTAL_FAIL failure(s)"
|
||||
notify "Weekly maintenance failed on $(hostname) ($MY_ID) — sync: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
|
||||
|
||||
@@ -130,7 +130,7 @@ PARTNERSHIP_ONBOARD_VERIFY=true # curl-verify each WebUI after onboard
|
||||
PARTNERSHIP_ONBOARD_NOTIFY=true # notify both servers on successful onboard
|
||||
```
|
||||
|
||||
### master_host1.conf (owner side)
|
||||
### host1.conf (owner side)
|
||||
|
||||
```bash
|
||||
# Containers whose WebUI URLs are redirected to owner's Tailscale IP on onboard.
|
||||
@@ -179,7 +179,7 @@ HOST1_PARTNERSHIP_OWN_CONTAINERS=(
|
||||
)
|
||||
```
|
||||
|
||||
### master_host2.conf (mirror side)
|
||||
### host2.conf (mirror side)
|
||||
|
||||
```bash
|
||||
# Containers whose WebUI URLs are redirected on onboard.
|
||||
@@ -342,9 +342,9 @@ Partnership/partnership_onboard.sh
|
||||
### Adding a New Container to the Auth Stack
|
||||
|
||||
1. Create the XML template on HOST1 (`/boot/config/plugins/dockerMan/templates-user/my-NewContainer.xml`)
|
||||
2. Add the XML filename to `HOST1_PARTNERSHIP_AUTH_STACK` in `master_host1.conf`
|
||||
2. Add the XML filename to `HOST1_PARTNERSHIP_AUTH_STACK` in `host1.conf`
|
||||
- If it has a database dependency, put the dep earlier in the array
|
||||
3. Add the container name to `HOST2_PARTNERSHIP_REPLACE_CONTAINERS` in `master_host2.conf`
|
||||
3. Add the container name to `HOST2_PARTNERSHIP_REPLACE_CONTAINERS` in `host2.conf`
|
||||
4. Re-run the auth stack portion:
|
||||
```bash
|
||||
Partnership/partnership_onboard.sh --skip-ssh --skip-arr-stack --skip-arr-sync
|
||||
@@ -371,6 +371,29 @@ Counter resets after `SSH_STRIKE_RESET_HRS` of clean connectivity.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ OUTPUT TIERS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||||
lines only appear when `--log` is passed.
|
||||
|
||||
**partnership_onboard.sh** — one-shot setup. Without `--log`, step headers, per-step
|
||||
result lines (deployed/failed counts), and the final summary are visible. Per-container
|
||||
deploy detail suppressed.
|
||||
|
||||
**partnership_offboard.sh** — one-shot teardown. Without `--log`, step headers, key
|
||||
state transitions, and the final checklist summary are visible. Per-container cleanup
|
||||
detail suppressed.
|
||||
|
||||
**partnership_manager.sh** — check/status/transfer/onboard modes. Without `--log`,
|
||||
mode-specific result lines (`--check` prints ACTIVE/INACTIVE status always), state
|
||||
transitions, warnings, and summary blocks are visible. Per-operation detail suppressed.
|
||||
|
||||
**ssh_setup.sh** — one-shot key setup. Without `--log`, section headers and per-step
|
||||
results (key created, key installed, auth verified) are visible. Strike-counter
|
||||
management (`--validate`) uses `log` for healthy cycles; `warn` for failures.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FLAG REFERENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
### partnership_onboard.sh
|
||||
@@ -491,7 +514,7 @@ automatically, but if HOST2 had no pre-existing auth stack of its own, it needs
|
||||
docker ps
|
||||
|
||||
# Verify own parked containers came back up:
|
||||
# (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in master_host2.conf)
|
||||
# (listed in HOST2_PARTNERSHIP_OWN_CONTAINERS in host2.conf)
|
||||
|
||||
# If you need a fresh auth stack, deploy from HOST2's own XML templates:
|
||||
docker create ... && docker start NginxProxyManager # etc.
|
||||
|
||||
@@ -141,7 +141,7 @@
|
||||
# TAILSCALE_API_KEY / TAILSCALE_TAILNET
|
||||
# Required when PARTNERSHIP_REMOVE_TAILSCALE=true
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_PARTNERSHIP_AUTH_WEBUIS
|
||||
# Containers reconfigured on onboard/offboard. Format: "ContainerName|WebUIPort"
|
||||
@@ -253,7 +253,7 @@ OWNER="${!OWNER_ID}" # hostname string
|
||||
MIRROR="${!MIRROR_ID}"
|
||||
# SSH_KEY (set by detect_hosts) is this server's own private key.
|
||||
# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
|
||||
# With sparse checkout, each server only has its own master_host{N}.conf — the other server's
|
||||
# With sparse checkout, each server only has its own host{N}.conf — the other server's
|
||||
# key path is never available here. Use SSH_KEY for all outbound SSH regardless of mode.
|
||||
MIRROR_SSH_KEY="$SSH_KEY"
|
||||
OWNER_SSH_KEY="$SSH_KEY"
|
||||
@@ -270,6 +270,21 @@ OWNER_STATE_FILE="/boot/config/partnership_${OWNER}.db"
|
||||
MIRROR_STATE_FILE="/boot/config/partnership_${MIRROR}.db"
|
||||
OFFLINE_COUNTER="/boot/config/partnership_offline_days.db"
|
||||
|
||||
# ── Exit Trap — restart locally stopped containers if script crashes mid-cleanup ──────────────
|
||||
# Used by folderview3_remove_partner_folder() and cleanup_partner_containers() — also shared
|
||||
# with partnership_offboard.sh which sources this file and registers the same trap.
|
||||
declare -a _PM_TRAP_STOPPED=()
|
||||
_pm_trap_restart_stopped() {
|
||||
[[ ${#_PM_TRAP_STOPPED[@]} -eq 0 ]] && return
|
||||
for c in "${_PM_TRAP_STOPPED[@]}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
if docker inspect "$c" >/dev/null 2>&1; then
|
||||
warn "Exit trap: restarting $c (stopped but not removed)"
|
||||
docker start "$c" >/dev/null 2>&1 || warn " Failed to restart $c"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
if [[ "${PARTNERSHIP_LIB_MODE:-}" != "1" ]]; then
|
||||
if [[ -z "$MODE" ]]; then
|
||||
error "No mode specified"
|
||||
@@ -297,6 +312,8 @@ if [[ "${PARTNERSHIP_LIB_MODE:-}" != "1" ]]; then
|
||||
# Lock for all modes except --check (frequent) and --offboard (offboard script holds its own)
|
||||
[[ "$MODE" != "check" && "$MODE" != "offboard" ]] && acquire_lock "strict"
|
||||
|
||||
trap _pm_trap_restart_stopped EXIT
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
fi
|
||||
|
||||
@@ -679,6 +696,7 @@ folderview3_remove_partner_folder() {
|
||||
if timeout "${DOCKER_TIMEOUT:-30}" docker inspect "$container" >/dev/null 2>&1; then
|
||||
if timeout "${DOCKER_TIMEOUT:-30}" docker stop "$container" >/dev/null 2>&1; then
|
||||
log "$container stopped ✅"
|
||||
_PM_TRAP_STOPPED+=("$container")
|
||||
(( stopped++ ))
|
||||
else
|
||||
warn "$container stop failed"
|
||||
@@ -822,6 +840,7 @@ cleanup_partner_containers() {
|
||||
fi
|
||||
if timeout "${DOCKER_TIMEOUT:-30}" docker inspect "$container" >/dev/null 2>&1; then
|
||||
timeout "${DOCKER_TIMEOUT:-30}" docker stop "$container" >/dev/null 2>&1 || true
|
||||
_PM_TRAP_STOPPED+=("$container")
|
||||
timeout "${DOCKER_TIMEOUT:-30}" docker rm "$container" >/dev/null 2>&1 && \
|
||||
log "$container removed ✅" || warn "$container rm failed"
|
||||
else
|
||||
@@ -1337,7 +1356,7 @@ if [[ "$MODE" == "check" ]]; then
|
||||
|
||||
# Both agree and active — healthy, silent
|
||||
if [[ "$LOCAL_STATE" == "$REMOTE_STATE" ]] && [[ "$LOCAL_STATE" == "ACTIVE" ]]; then
|
||||
log "Partnership check — ACTIVE, both servers agree ✅"
|
||||
echo "Partnership check — ACTIVE, both servers agree ✅"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -1397,7 +1416,7 @@ if [[ "$MODE" == "check" ]]; then
|
||||
|
||||
# Both inactive — nothing to do
|
||||
if [[ "$LOCAL_STATE" == "INACTIVE" ]] && [[ "$REMOTE_STATE" == "INACTIVE" ]]; then
|
||||
log "Partnership check — INACTIVE on both servers"
|
||||
echo "Partnership check — INACTIVE on both servers"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -1489,7 +1508,7 @@ if [[ "$MODE" == "onboard" ]]; then
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
write_state_file "$LOCAL_STATE_FILE" "ACTIVE" "$NOW" "" "$LOCAL_SERVER_NAME" "onboard"
|
||||
log "Local state: ACTIVE ✅"
|
||||
echo "Local state: ACTIVE ✅"
|
||||
remove_from_blocklist "$MIRROR"
|
||||
push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"
|
||||
echo "0" > "$OFFLINE_COUNTER"
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_PARTNERSHIP_AUTH_STACK
|
||||
# Auth container XMLs to push during onboard — used on offboard to identify what
|
||||
@@ -124,6 +124,8 @@ OFFLINE_COUNTER="/boot/config/partnership_offline_days.db"
|
||||
|
||||
acquire_lock "strict"
|
||||
|
||||
trap _pm_trap_restart_stopped EXIT
|
||||
|
||||
# Check already offboarded
|
||||
if [[ -f "$LOCAL_STATE_FILE" ]]; then
|
||||
CURRENT_STATE=$(read_state_file "$LOCAL_STATE_FILE" "state")
|
||||
@@ -265,6 +267,7 @@ cleanup_deployed_stack_locally() {
|
||||
--format '{{range .HostConfig.Binds}}{{println .}}{{end}}' \
|
||||
"$cname" 2>/dev/null | awk -F: '{print $1}' | grep '^/mnt/.*/appdata')
|
||||
timeout "${DOCKER_TIMEOUT:-30}" docker stop "$cname" >/dev/null 2>&1 || true
|
||||
_PM_TRAP_STOPPED+=("$cname")
|
||||
timeout "${DOCKER_TIMEOUT:-30}" docker rm "$cname" >/dev/null 2>&1 && \
|
||||
log " $cname removed ✅" || warn " $cname rm failed"
|
||||
else
|
||||
@@ -428,7 +431,7 @@ if [[ "$AM_MIRROR" == true ]]; then
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
write_state_file "$LOCAL_STATE_FILE" \
|
||||
"INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
|
||||
log "Local state: INACTIVE ✅"
|
||||
echo "Local state: INACTIVE ✅"
|
||||
add_to_blocklist "$OWNER" "$REASON"
|
||||
else
|
||||
warn "DRY RUN — would write INACTIVE state and blocklist $OWNER"
|
||||
@@ -626,7 +629,7 @@ fi
|
||||
if [[ ${#PARTNERSHIP_MIRROR_BACKUPS[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_DISK Backup Handover ━━━"
|
||||
log "Backups available for $MIRROR:"
|
||||
echo "Backups available for $MIRROR:"
|
||||
for path in "${PARTNERSHIP_MIRROR_BACKUPS[@]}"; do
|
||||
[[ -z "$path" ]] && continue
|
||||
echo " $path"
|
||||
|
||||
@@ -45,7 +45,7 @@
|
||||
#
|
||||
# Dependency ordering in the auth stack is owner-enforced
|
||||
# PARTNERSHIP_AUTH_STACK order matters: Mariadb and Redis must come before Authelia.
|
||||
# The array is ordered correctly in master_host1.conf. After each Mariadb/Redis deploy,
|
||||
# The array is ordered correctly in host1.conf. After each Mariadb/Redis deploy,
|
||||
# the script waits for the container to be healthy before continuing. This is a remote
|
||||
# health check — the container must be running (or report healthy) before the next
|
||||
# dependent is deployed.
|
||||
@@ -73,7 +73,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_PARTNERSHIP_AUTH_STACK
|
||||
# XML filenames (from this server's templates-user/) to push and deploy on the
|
||||
@@ -82,7 +82,7 @@
|
||||
#
|
||||
# HOST*_PARTNERSHIP_REPLACE_CONTAINERS
|
||||
# Containers to stop on the mirror before deploying the auth stack.
|
||||
# Defined in the MIRROR's own conf (master_host*.conf on HOST2) — never in HOST1's conf.
|
||||
# Defined in the MIRROR's own conf (host*.conf on HOST2) — never in HOST1's conf.
|
||||
# Read live from the mirror via SSH during Step 3 (sources mirror's load_config.sh at
|
||||
# the same $SCRIPTS_ROOT path — convention: both servers use the same repo location).
|
||||
# Leave empty on HOST2 if no conflicting containers exist (fresh mirror: nothing to stop).
|
||||
@@ -171,7 +171,7 @@ OWNER="${!OWNER_ID}"
|
||||
MIRROR="${!MIRROR_ID}"
|
||||
# SSH_KEY (set by detect_hosts) is this server's own private key.
|
||||
# The remote accepts it because this server's PUBLIC key was installed there via ssh_setup.sh.
|
||||
# HOST{N}_SSH_KEY lives in master_host{N}.conf — with sparse checkout, the other server's
|
||||
# HOST{N}_SSH_KEY lives in host{N}.conf — with sparse checkout, the other server's
|
||||
# conf is never present here. Always use SSH_KEY (local private key) for outbound SSH.
|
||||
MIRROR_SSH_KEY="$SSH_KEY"
|
||||
|
||||
@@ -359,7 +359,7 @@ wait_for_container_healthy() {
|
||||
#
|
||||
# SSHes to the mirror, sources its load_config.sh at the same $SCRIPTS_ROOT path (both servers
|
||||
# use the same convention), and reads the named config array from the mirror's own conf.
|
||||
# HOST2's container list stays in HOST2's master_host2.conf — not duplicated in HOST1's conf.
|
||||
# HOST2's container list stays in HOST2's host2.conf — not duplicated in HOST1's conf.
|
||||
# Fails gracefully if scripts aren't present yet or the array is empty (nothing to stop).
|
||||
#
|
||||
# deploy_container_from_xml() already stops/removes containers with the same name as what's
|
||||
@@ -547,13 +547,13 @@ if [[ "$SKIP_AUTH_STACK" == true ]]; then
|
||||
warn "Skipping (--skip-auth-stack)"
|
||||
elif [[ ${#PARTNERSHIP_AUTH_STACK[@]} -eq 0 ]]; then
|
||||
warn "PARTNERSHIP_AUTH_STACK not set in ${MY_ID} conf — skipping auth stack deploy"
|
||||
warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to master_host${MY_ID: -1}.conf"
|
||||
warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to host${MY_ID: -1}.conf"
|
||||
STEP_AUTH_OK=false
|
||||
else
|
||||
deploy_xml_stack PARTNERSHIP_AUTH_STACK
|
||||
AUTH_DEPLOYED=$_STACK_DEPLOYED
|
||||
AUTH_FAILED=$_STACK_FAILED
|
||||
log "Auth stack: $AUTH_DEPLOYED deployed, $AUTH_FAILED failed"
|
||||
echo "Auth stack: $AUTH_DEPLOYED deployed, $AUTH_FAILED failed"
|
||||
[[ "$AUTH_FAILED" -gt 0 ]] && STEP_AUTH_OK=false
|
||||
fi
|
||||
|
||||
@@ -580,7 +580,7 @@ else
|
||||
deploy_xml_stack PARTNERSHIP_ARR_STACK
|
||||
ARR_DEPLOYED=$_STACK_DEPLOYED
|
||||
ARR_FAILED=$_STACK_FAILED
|
||||
log "Arr stack: $ARR_DEPLOYED deployed, $ARR_FAILED failed"
|
||||
echo "Arr stack: $ARR_DEPLOYED deployed, $ARR_FAILED failed"
|
||||
[[ "$ARR_FAILED" -gt 0 ]] && STEP_ARR_OK=false
|
||||
fi
|
||||
|
||||
@@ -589,7 +589,7 @@ echo ""
|
||||
echo "━━━ Step 7/8 — Partnership Onboard ━━━"
|
||||
|
||||
if bash "$SCRIPTS_ROOT/Partnership/partnership_manager.sh" --onboard "${EXTRA_FLAGS[@]}"; then
|
||||
log "Partnership onboard complete ✅"
|
||||
echo "Partnership onboard complete ✅"
|
||||
ONBOARD_OK=true
|
||||
else
|
||||
error "Partnership onboard failed"
|
||||
@@ -607,7 +607,7 @@ elif [[ "$SKIP_ARR_SYNC" == true ]]; then
|
||||
elif [[ ! -f "$SCRIPTS_ROOT/Media/arr_sync.sh" ]]; then
|
||||
warn "arr_sync.sh not found — run Media/arr_sync.sh manually once arrs are live"
|
||||
elif bash "$SCRIPTS_ROOT/Media/arr_sync.sh" "${EXTRA_FLAGS[@]}"; then
|
||||
log "Arr bootstrap complete ✅"
|
||||
echo "Arr bootstrap complete ✅"
|
||||
ARR_SYNC_OK=true
|
||||
else
|
||||
warn "Arr sync had errors — partnership still valid"
|
||||
@@ -637,8 +637,8 @@ echo ""
|
||||
|
||||
if [[ "$ONBOARD_OK" == true ]]; then
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
|
||||
log "$ICON_DONE DONE — partnership established ✅"
|
||||
log "Verify with: Partnership/partnership_manager.sh --status"
|
||||
echo "$ICON_DONE DONE — partnership established ✅"
|
||||
echo "Verify with: Partnership/partnership_manager.sh --status"
|
||||
else
|
||||
error "Setup incomplete — resolve errors above and re-run"
|
||||
fi
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
# unRAID-Gmer4Lfe → gmer4lfe_rsync_automation
|
||||
# unRAID-Jayred365 → jayred365_rsync_automation
|
||||
# Idempotent — skips generation if key already exists (use --force to regenerate).
|
||||
# Updates master_host*.conf with key path on success.
|
||||
# Updates host*.conf with key path on success.
|
||||
#
|
||||
# ── MODES ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# (default) — generate key if missing, copy to remote, update conf
|
||||
@@ -85,7 +85,7 @@ SSH_PUB_PATH="${SSH_KEY_PATH}.pub"
|
||||
|
||||
# ── Host conf path ────────────────────────────────────────────────────────────────────────────
|
||||
HOST_NUM="${MY_ID#HOST}" # "1" or "2"
|
||||
HOST_CONF="$SCRIPTS_ROOT/master_host${HOST_NUM}.conf"
|
||||
HOST_CONF="$SCRIPTS_ROOT/host${HOST_NUM}.conf"
|
||||
KEY_CONF_VAR="${MY_ID}_SSH_KEY"
|
||||
|
||||
# ── Strike state file ─────────────────────────────────────────────────────────────────────────
|
||||
@@ -290,8 +290,8 @@ echo "━━━ $ICON_GEAR Key Generation ━━━"
|
||||
|
||||
if [[ -f "$SSH_KEY_PATH" ]] && [[ "$FORCE" == false ]]; then
|
||||
local_fp=$(ssh-keygen -lf "$SSH_KEY_PATH" 2>/dev/null || echo "unreadable")
|
||||
log "Key already exists — skipping generation (--force to regenerate)"
|
||||
log " $local_fp"
|
||||
echo "Key already exists — skipping generation (--force to regenerate)"
|
||||
echo " $local_fp"
|
||||
else
|
||||
if [[ "$FORCE" == true ]] && [[ -f "$SSH_KEY_PATH" ]]; then
|
||||
warn "Regenerating key (--force) — existing key will be replaced"
|
||||
@@ -316,12 +316,12 @@ fi
|
||||
|
||||
# ── Update conf ───────────────────────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Update master_host${HOST_NUM}.conf ━━━"
|
||||
echo "━━━ $ICON_GEAR Update host${HOST_NUM}.conf ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
update_conf_key_path
|
||||
else
|
||||
warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in master_host${HOST_NUM}.conf"
|
||||
warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in host${HOST_NUM}.conf"
|
||||
fi
|
||||
|
||||
# ── Copy to remote ────────────────────────────────────────────────────────────────────────────
|
||||
@@ -339,7 +339,7 @@ else
|
||||
|
||||
if ssh-copy-id -i "$SSH_PUB_PATH" -o ConnectTimeout="${SSH_TIMEOUT:-15}" \
|
||||
root@"$REMOTE_SERVER" 2>/dev/null; then
|
||||
log "Public key installed on $REMOTE_SERVER_NAME ✅"
|
||||
echo "Public key installed on $REMOTE_SERVER_NAME ✅"
|
||||
else
|
||||
error "ssh-copy-id failed — check that:"
|
||||
error " 1. Remote server is reachable: tailscale status"
|
||||
@@ -355,7 +355,7 @@ echo "━━━ $ICON_VERIFY Verify SSH Auth ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
if test_ssh_auth "$REMOTE_SERVER"; then
|
||||
log "SSH auth to $REMOTE_SERVER_NAME working ✅"
|
||||
echo "SSH auth to $REMOTE_SERVER_NAME working ✅"
|
||||
# Reset any existing strikes
|
||||
if [[ -f "$SSH_STRIKE_FILE" ]]; then
|
||||
write_strike_file 0 "" "$(date '+%Y-%m-%d %H:%M:%S')"
|
||||
@@ -373,7 +373,7 @@ echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY SSH SETUP SUMMARY ━━━━━"
|
||||
echo " Key: $SSH_KEY_PATH"
|
||||
echo " Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)"
|
||||
echo " Conf: ${KEY_CONF_VAR} in master_host${HOST_NUM}.conf"
|
||||
echo " Conf: ${KEY_CONF_VAR} in host${HOST_NUM}.conf"
|
||||
echo ""
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
|
||||
warn "$ICON_DONE DONE — SSH key ready for rsync automation ✅"
|
||||
|
||||
@@ -893,7 +893,7 @@ Background: NO
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
/mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --status
|
||||
/mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --status
|
||||
```
|
||||
|
||||
**What it shows:**
|
||||
@@ -1275,7 +1275,7 @@ Background: NO
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --status
|
||||
/mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --status
|
||||
```
|
||||
|
||||
**What it shows:**
|
||||
|
||||
@@ -448,8 +448,8 @@ Failover coverage:
|
||||
serves Gmer4Lfe.us via DDNS
|
||||
|
||||
Monitoring:
|
||||
Both servers system_watchdog.sh system_watchdog.sh
|
||||
docker_watchdog.sh docker_watchdog.sh
|
||||
Both servers resource/docker/storage/ resource/docker/storage/
|
||||
system_watchdog.sh system_watchdog.sh
|
||||
failover.sh failover.sh
|
||||
Sunday morning coffee report Sunday morning coffee report
|
||||
```
|
||||
@@ -462,8 +462,8 @@ Monitoring:
|
||||
Unraid_Scripts/
|
||||
│
|
||||
├── master.conf ← All shared configuration — the only file you edit regularly
|
||||
├── master_host1.conf ← HOST1-specific: share lists, container names, API keys
|
||||
├── master_host2.conf ← HOST2-specific: same structure, different values
|
||||
├── host1.conf ← HOST1-specific: share lists, container names, API keys
|
||||
├── host2.conf ← HOST2-specific: same structure, different values
|
||||
├── common.sh ← Shared library — all functions used by every script
|
||||
├── load_config.sh ← Sources all conf files and common.sh
|
||||
│
|
||||
@@ -476,10 +476,13 @@ Unraid_Scripts/
|
||||
├── Failover/ ← Mutual automatic failover — continuous background process
|
||||
│ README: README-Failover.md
|
||||
│
|
||||
├── Docker_Essentials/ ← Container lifecycle: watchdog, restarts, networks
|
||||
├── Watchdogs/ ← All watchdog scripts: resource, docker, storage, system
|
||||
│ README: README-Watchdogs.md
|
||||
│
|
||||
├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks
|
||||
│ README: README-Docker_Essentials.md
|
||||
│
|
||||
├── unRAID_Essentials/ ← Server-level: system watchdog, WebGUI, log hygiene, tuning
|
||||
├── unRAID_Essentials/ ← Server-level: WebGUI, log hygiene, kernel tuning
|
||||
│ README: README-Unraid_Essentials.md
|
||||
│
|
||||
├── Media/ ← Library health + behavior-driven discovery: permissions, junk cleanup, orphan removal, weekly arr adds
|
||||
@@ -509,11 +512,11 @@ Unraid_Scripts/
|
||||
# definitions, watchdog settings, arr cleanup config, DDNS timing, etc.
|
||||
# Pushed to both servers via git. Never contains server-specific values.
|
||||
#
|
||||
# master_host1.conf — sourced only on HOST1
|
||||
# host1.conf — sourced only on HOST1
|
||||
# HOST1_* prefixed variables: share lists, container names, API keys,
|
||||
# ramdisk size, specific paths, per-server toggle overrides.
|
||||
#
|
||||
# master_host2.conf — sourced only on HOST2
|
||||
# host2.conf — sourced only on HOST2
|
||||
# HOST2_* prefixed variables: same structure, different values.
|
||||
#
|
||||
# detect_hosts() in common.sh:
|
||||
@@ -539,10 +542,15 @@ At Startup of Array:
|
||||
→ php_fpm_max_children.sh WebGUI tuning before first request
|
||||
→ ramdisk_setup.sh create ramdisk before Emby starts
|
||||
→ docker_network_connect.sh connect containers to extra networks
|
||||
→ system_watchdog.sh continuous — last resort server stability
|
||||
→ docker_watchdog.sh continuous — two-tier container healing
|
||||
→ failover.sh continuous — mutual failover state machine
|
||||
|
||||
Every minute:
|
||||
watchdog_orchestrator.sh fires each watchdog in sequence
|
||||
→ resource_watchdog.sh reduce pressure before healing attempts
|
||||
→ docker_watchdog.sh two-tier container healing
|
||||
→ storage_watchdog.sh pool growth + runaway log detection
|
||||
→ system_watchdog.sh last resort — reboots when all else fails
|
||||
|
||||
Every 3 minutes:
|
||||
transcode_management.sh cleanup → manager (order non-negotiable)
|
||||
|
||||
@@ -582,9 +590,13 @@ Sunday morning block (6–11am):
|
||||
What actually happens on a typical day, from the ecosystem's perspective:
|
||||
|
||||
```
|
||||
Throughout the day:
|
||||
Throughout the day (every minute via watchdog_orchestrator.sh):
|
||||
resource_watchdog.sh managing: system pressure (throttle/pause/stop)
|
||||
docker_watchdog.sh healing: memory leaks, HTTP failures, required containers
|
||||
storage_watchdog.sh watching: appdata growth rate, runaway log files
|
||||
system_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
|
||||
docker_watchdog.sh watching: memory, CPU, HTTP health, required containers
|
||||
|
||||
Throughout the day:
|
||||
failover.sh watching: remote server, internet connectivity
|
||||
transcode_management.sh managing: ramdisk ↔ SSD, session cleanup (every 3min)
|
||||
critical_sync_maintenance.sh keeping: auth stack + Emby current (every 15min)
|
||||
|
||||
+23
-10
@@ -130,7 +130,7 @@ access key (for script repository pulls).
|
||||
### 3a — Rsync Automation Keys (ssh_setup.sh)
|
||||
|
||||
Run on **each server**. `ssh_setup.sh` generates the key, copies it to the remote,
|
||||
and updates `master_host*.conf` with the key path automatically.
|
||||
and updates `host*.conf` with the key path automatically.
|
||||
|
||||
```bash
|
||||
# On HOST1 — after the repo is cloned:
|
||||
@@ -146,7 +146,7 @@ unRAID-Gmer4Lfe → /root/.ssh/gmer4lfe_rsync_automation
|
||||
unRAID-Jayred365 → /root/.ssh/jayred365_rsync_automation
|
||||
```
|
||||
|
||||
`master_host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the
|
||||
`host*.conf` is updated automatically with `HOST*_SSH_KEY` pointing to the
|
||||
generated key. Run `--status` to verify:
|
||||
|
||||
```bash
|
||||
@@ -188,8 +188,8 @@ find /mnt/user/appdata/unraid_scripts -name "*.sh" -exec chmod +x {} \;
|
||||
Expected structure after clone:
|
||||
```
|
||||
master.conf ← all shared configuration
|
||||
master_host1.conf ← HOST1-specific configuration
|
||||
master_host2.conf ← HOST2-specific configuration
|
||||
host1.conf ← HOST1-specific configuration
|
||||
host2.conf ← HOST2-specific configuration
|
||||
common.sh ← shared library
|
||||
load_config.sh ← config loader
|
||||
Orchestrators/
|
||||
@@ -232,7 +232,7 @@ SSH_PORT=221
|
||||
### Daily Sync Shares
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
# Shares this server downloads to — pushed to the other server nightly.
|
||||
# arr_sync.sh runs before rsync in the daily window, so the receiving
|
||||
# server's arrs already track incoming files when they arrive.
|
||||
@@ -247,7 +247,7 @@ HOST1_DAILY_SYNC_SHARES=(
|
||||
"/mnt/user/stand-up_comedy"
|
||||
)
|
||||
|
||||
# master_host2.conf
|
||||
# host2.conf
|
||||
HOST2_DAILY_SYNC_SHARES=(
|
||||
"/mnt/user/Anime_Shows"
|
||||
"/mnt/user/Anime_Movies"
|
||||
@@ -274,8 +274,8 @@ to their unprefixed names. Scripts only ever reference the unprefixed name — t
|
||||
work identically on both servers.
|
||||
|
||||
```bash
|
||||
nano /mnt/user/appdata/unraid_scripts/master_host1.conf # on HOST1
|
||||
nano /mnt/user/appdata/unraid_scripts/master_host2.conf # on HOST2
|
||||
nano /mnt/user/appdata/unraid_scripts/host1.conf # on HOST1
|
||||
nano /mnt/user/appdata/unraid_scripts/host2.conf # on HOST2
|
||||
```
|
||||
|
||||
Every variable is documented in the conf files. Key values to set:
|
||||
@@ -413,7 +413,7 @@ zfs load-key poolname/Gmer4Lfe-Personal # prompts for passphrase
|
||||
zfs mount poolname/Gmer4Lfe-Personal
|
||||
```
|
||||
|
||||
### Add to master_host1.conf
|
||||
### Add to host1.conf
|
||||
|
||||
```bash
|
||||
HOST1_PERSONAL_SHARES=(
|
||||
@@ -662,7 +662,7 @@ declare -A PROFILE_EXCLUDE_DIRS
|
||||
declare -A PROFILE_REMOTE_RESTART_CONTAINERS
|
||||
```
|
||||
|
||||
### master_host*.conf
|
||||
### host*.conf
|
||||
|
||||
```bash
|
||||
# Daily sync shares — one list per server (mutually exclusive)
|
||||
@@ -680,6 +680,19 @@ HOST2_SSH_KEY="/root/.ssh/jayred365_rsync_automation"
|
||||
|
||||
---
|
||||
|
||||
## ━━━ OUTPUT TIERS ━━━
|
||||
|
||||
Without `--log`, each run shows: section headers (Pre-flight, Transfer, Stop/Start
|
||||
Containers), the transfer identity block (source, remote, profile, identity), a
|
||||
progress indicator while rsync is running, and a summary block with duration,
|
||||
bytes transferred, and status. Warnings and errors are always visible.
|
||||
|
||||
With `--log`, every decision is shown — pre-flight check results, profile resolution,
|
||||
per-container stop/start state, retry details, and bandwidth log confirmation.
|
||||
Use for first-run validation or when investigating unexpected behavior.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FLAG REFERENCE ━━━
|
||||
|
||||
All flags work on `rsync.sh` and all orchestrators.
|
||||
|
||||
+1
-1
@@ -435,7 +435,7 @@ echo "$ICON_TIME Duration: $(format_duration $DURATION)"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$RSYNC_SUCCESS" == true ]]; then
|
||||
log "$ICON_DONE Status: $ICON_SUCCESS DONE"
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
|
||||
else
|
||||
echo "$ICON_ERROR Status: FAILED after $RETRY_COUNT attempts"
|
||||
notify "Rsync FAILED — $DIRECTORY ($PROFILE_NAME) after $RETRY_COUNT attempts on $(hostname)" \
|
||||
|
||||
+87
-2
@@ -20,10 +20,28 @@ making any changes.
|
||||
- [recreate_shares.sh](#recreate_sharessh)
|
||||
- [continuous_scripts_status.sh](#continuous_scripts_statussh)
|
||||
- [claude_startup.sh](#claude_startupsh)
|
||||
- [ramdisk_stop.sh](#ramdisk_stopsh)
|
||||
- [Adding a New Tool](#adding-a-new-tool)
|
||||
|
||||
---
|
||||
|
||||
## Output Tiers
|
||||
|
||||
All tools have two output levels controlled by `--log`.
|
||||
|
||||
Without `--log`, each script processes and always concludes with a summary block
|
||||
showing identity, duration, counts, and a status line. Warnings and errors are
|
||||
always visible. State-display scripts (watchdog_skip_list_manager, zfs_pool_scrub
|
||||
`--status`, fallback_state_reset current-state section) always show their state
|
||||
output — `--log` adds configuration detail and per-item resolution within each
|
||||
section.
|
||||
|
||||
With `--log`, per-item detail appears: individual items added/skipped, per-database
|
||||
check results, per-pool scan lines, per-container stop/start state, per-directory
|
||||
creation results. Use when debugging unexpected results or confirming a first run.
|
||||
|
||||
---
|
||||
|
||||
## emby_to_lidarr_sync.sh
|
||||
|
||||
One-shot bootstrap tool. Scans Emby play history, finds artists you've actually
|
||||
@@ -415,7 +433,7 @@ library.db-wal — write-ahead log (uncommitted transactions)
|
||||
the main library.db is also affected
|
||||
```
|
||||
|
||||
### Configuration (master_host*.conf)
|
||||
### Configuration (host*.conf)
|
||||
|
||||
```bash
|
||||
HOST1_EMBY_CONTAINER="Emby" # aliased by detect_hosts() → EMBY_CONTAINER
|
||||
@@ -453,7 +471,7 @@ without triggering any error — until you try to read that specific file. By th
|
||||
Run monthly. Also run after any disk replacement or power event.
|
||||
Safe to run while the system is in use — scrub runs at low I/O priority.
|
||||
|
||||
### Configuration (master_host*.conf)
|
||||
### Configuration (host*.conf)
|
||||
|
||||
```bash
|
||||
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||||
@@ -614,6 +632,73 @@ claude_startup.sh --setup # set up symlinks only — no launch (for array_star
|
||||
|
||||
---
|
||||
|
||||
## ramdisk_stop.sh
|
||||
|
||||
Safely stops the transcode ramdisk: redirects the transcode symlink to the SSD
|
||||
fallback first (so Emby continues writing without interruption), then unmounts the
|
||||
tmpfs and updates the state file. Primary use case is stopping the current ramdisk
|
||||
before re-running `ramdisk_setup.sh` with new size or threshold values.
|
||||
|
||||
### When to Use
|
||||
|
||||
```
|
||||
Bumping RAMDISK_SIZE — setup script is idempotent, skips remount if already mounted
|
||||
→ stop first, then re-run ramdisk_setup.sh with new HOST*_RAMDISK_SIZE value
|
||||
|
||||
Adjusting RAMDISK_WARN_GB / RAMDISK_LOW_GB thresholds
|
||||
→ no need to stop for threshold changes (transcode_manager reads vars live)
|
||||
→ only needed if you're also changing the size
|
||||
|
||||
Temporarily freeing ramdisk RAM — reclaim tmpfs back to general memory pool
|
||||
→ stop, restart later with ramdisk_setup.sh
|
||||
```
|
||||
|
||||
### Stop Sequence
|
||||
|
||||
```
|
||||
1. Redirect symlink: TRANSCODE_LINK → TRANSCODE_SSD
|
||||
Emby immediately writes to SSD — no broken-path window during unmount
|
||||
|
||||
2. Check for active transcode files on ramdisk (warn, don't block)
|
||||
Files in progress on the ramdisk are lost on unmount — expected for maintenance
|
||||
|
||||
3. Unmount ramdisk
|
||||
Regular umount first; if busy (directory handles only, no active writes)
|
||||
falls back to lazy unmount automatically
|
||||
|
||||
4. Update /tmp/transcode_state.db → current_target=TRANSCODE_SSD
|
||||
transcode_manager.sh reads this on its next cycle
|
||||
```
|
||||
|
||||
### transcode_manager Warning
|
||||
|
||||
If `transcode_manager.sh` is running, it may flip the symlink back to the ramdisk
|
||||
on its next cycle (once the ramdisk is unmounted, that flip will fail). Stop
|
||||
`transcode_manager.sh` first if you need the SSD redirect to hold before remounting.
|
||||
|
||||
### After Stopping
|
||||
|
||||
```bash
|
||||
# Update host*.conf with new size values:
|
||||
# HOST1_RAMDISK_SIZE="10G"
|
||||
# HOST1_RAMDISK_WARN_GB=8.5
|
||||
# HOST1_RAMDISK_LOW_GB=7
|
||||
|
||||
# Remount at new size:
|
||||
bash Transcodes/ramdisk_setup.sh
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
ramdisk_stop.sh --status # show mount state, symlink, active files — always check first
|
||||
ramdisk_stop.sh --dry-run # show what would happen without making changes
|
||||
ramdisk_stop.sh # stop the ramdisk
|
||||
ramdisk_stop.sh --log # verbose — show each step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Tool
|
||||
|
||||
Write the tool when you solve a problem manually with bash commands. You'll face it again.
|
||||
|
||||
@@ -76,7 +76,7 @@ something new, write the tool. Store it here. Find it at 2am next time.
|
||||
`bulk_permissions_repair.sh`, `zfs_pool_scrub.sh`
|
||||
|
||||
**Lifecycle Tools** — Backup, setup, and migration support
|
||||
`container_data_export.sh`, `recreate_shares.sh`, `claude_startup.sh`
|
||||
`container_data_export.sh`, `recreate_shares.sh`, `claude_startup.sh`, `ramdisk_stop.sh`
|
||||
|
||||
**Library Sync Bootstrap** — Close the gap between Emby and arr libraries
|
||||
`emby_to_lidarr_sync.sh`, `emby_to_sonarr_sync.sh`, `emby_to_radarr_sync.sh`
|
||||
@@ -125,6 +125,7 @@ The relationship is one-way: Tools act on state that other scripts have written.
|
||||
| `recreate_shares.sh` | Share directories missing after fresh install or disk rebuild | After fresh unRAID install or disk replacement on HOST2 |
|
||||
| `continuous_scripts_status.sh` | Need a live view of watchdog and fallback state | Any time — manual dashboard, no schedule |
|
||||
| `claude_startup.sh` | Claude Code session setup after reboot — symlinks persistent storage | After each unRAID reboot, or called by array_started.sh |
|
||||
| `ramdisk_stop.sh` | Safely stop the transcode ramdisk — redirect symlink to SSD, unmount, update state | Before re-running ramdisk_setup.sh with new size or thresholds |
|
||||
| `emby_to_lidarr_sync.sh` | Add all Emby album artists not yet tracked in Lidarr | After Lidarr setup, database wipe, or when you suspect gaps |
|
||||
| `emby_to_sonarr_sync.sh` | Add all Emby TV series not yet tracked in Sonarr | After Sonarr setup, database wipe, or when you suspect gaps |
|
||||
| `emby_to_radarr_sync.sh` | Add all Emby movies not yet tracked in Radarr | After Radarr setup, database wipe, or when you suspect gaps |
|
||||
@@ -151,6 +152,7 @@ Situation arises
|
||||
│ recreate_shares ◄── fresh HOST2 setup or disk rebuild │
|
||||
│ continuous_scripts_status ◄── manual status check at any time │
|
||||
│ claude_startup ◄── after each unRAID reboot │
|
||||
│ ramdisk_stop ◄── before ramdisk resize / remount │
|
||||
│ │
|
||||
│ emby_to_lidarr_sync ◄── Lidarr setup / database wipe / gap │
|
||||
│ emby_to_sonarr_sync ◄── Sonarr setup / database wipe / gap │
|
||||
|
||||
@@ -304,7 +304,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$ARCHIVE_VERIFIED" == true && "$RESTART_OK" == true ]]; then
|
||||
log "$ICON_DONE Status: done — $ARCHIVE_NAME"
|
||||
echo "$ICON_DONE Status: done — $ARCHIVE_NAME"
|
||||
elif [[ "$ARCHIVE_VERIFIED" == false ]]; then
|
||||
echo "$ICON_ERROR Status: archive verification FAILED — check backup before relying on it"
|
||||
else
|
||||
|
||||
@@ -55,7 +55,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_EMBY_CONTAINER
|
||||
# Name of the Emby Docker container on this host.
|
||||
@@ -122,7 +122,7 @@ fi
|
||||
detect_hosts
|
||||
|
||||
if [[ -z "${EMBY_CONTAINER:-}" ]]; then
|
||||
error "EMBY_CONTAINER not set for $MY_ID — check HOST*_EMBY_CONTAINER in master_host*.conf"
|
||||
error "EMBY_CONTAINER not set for $MY_ID — check HOST*_EMBY_CONTAINER in host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -135,7 +135,7 @@ EMBY_CONFIG_HOST=$(timeout "$DOCKER_TIMEOUT" docker inspect "$EMBY_CONTAINER" 2>
|
||||
|
||||
if [[ -z "$EMBY_CONFIG_HOST" ]]; then
|
||||
error "Could not detect Emby config path from Docker mounts"
|
||||
error "Is $EMBY_CONTAINER the correct container name? Check HOST*_EMBY_CONTAINER in master_host*.conf"
|
||||
error "Is $EMBY_CONTAINER the correct container name? Check HOST*_EMBY_CONTAINER in host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -352,7 +352,7 @@ echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
echo " $ICON_SUCCESS Passed: ${#PASS_DBS[@]}"
|
||||
[[ ${#FAIL_DBS[@]} -gt 0 ]] && echo " $ICON_ERROR Failed: ${#FAIL_DBS[@]}"
|
||||
[[ ${#MISSING_DBS[@]} -gt 0 ]] && log "Skipped: ${#MISSING_DBS[@]} (not found)"
|
||||
[[ ${#MISSING_DBS[@]} -gt 0 ]] && echo " Skipped: ${#MISSING_DBS[@]} (not found)"
|
||||
echo ""
|
||||
|
||||
[[ ${#PASS_DBS[@]} -gt 0 ]] && for db in "${PASS_DBS[@]}"; do log " $ICON_SUCCESS $db"; done
|
||||
@@ -378,7 +378,7 @@ elif [[ ${#FAIL_DBS[@]} -gt 0 ]]; then
|
||||
notify "Emby database CORRUPTION on $(hostname) — failed: ${FAIL_DBS[*]} — manual intervention needed" \
|
||||
"Emby DB Repair" "warning"
|
||||
else
|
||||
log "$ICON_DONE Status: all ${#PASS_DBS[@]} databases healthy ✅"
|
||||
echo "$ICON_DONE Status: all ${#PASS_DBS[@]} databases healthy ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
@@ -36,7 +36,7 @@
|
||||
# modification. Safe to run multiple times — the second run finds nothing to add.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION (master_host*.conf)
|
||||
# CONFIGURATION (host*.conf)
|
||||
# ==============================================================================================
|
||||
#
|
||||
# HOST*_LIDARR_URL / HOST*_LIDARR_API_KEY — Lidarr connection (aliased by detect_hosts)
|
||||
@@ -73,12 +73,12 @@ if ! command -v jq >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${LIDARR_URL:-}" || -z "${LIDARR_API_KEY:-}" ]]; then
|
||||
error "LIDARR_URL / LIDARR_API_KEY not configured — check master_host*.conf"
|
||||
error "LIDARR_URL / LIDARR_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -224,13 +224,13 @@ IFS=$'\n' MISSING=($(printf '%s\n' "${MISSING[@]}" | sort))
|
||||
echo " Already tracked: $ALREADY | Missing from Lidarr: ${#MISSING[@]}"
|
||||
|
||||
if [[ "${#MISSING[@]}" -eq 0 ]]; then
|
||||
log "Lidarr already tracks everything played on Emby"
|
||||
echo "Lidarr already tracks everything played on Emby"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_SUMMARY Artists to add (${#MISSING[@]}) ━━━"
|
||||
for a in "${MISSING[@]}"; do echo " $a"; done
|
||||
for a in "${MISSING[@]}"; do log " $a"; done
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN complete — run without --dry-run to add these artists"
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
# without modification. Safe to run multiple times — the second run finds nothing to add.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION (master_host*.conf)
|
||||
# CONFIGURATION (host*.conf)
|
||||
# ==============================================================================================
|
||||
#
|
||||
# HOST*_RADARR_URL / HOST*_RADARR_API_KEY — Radarr connection (aliased by detect_hosts)
|
||||
@@ -74,12 +74,12 @@ if ! command -v jq >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${RADARR_URL:-}" || -z "${RADARR_API_KEY:-}" ]]; then
|
||||
error "RADARR_URL / RADARR_API_KEY not configured — check master_host*.conf"
|
||||
error "RADARR_URL / RADARR_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -229,7 +229,7 @@ IFS=$'\n' SORTED_MISSING=($(printf '%s\n' "${!MISSING[@]}" | sort))
|
||||
echo " Already tracked: $ALREADY | Missing from Radarr: ${#MISSING[@]}"
|
||||
|
||||
if [[ "${#MISSING[@]}" -eq 0 ]]; then
|
||||
log "Radarr already tracks everything in Emby"
|
||||
echo "Radarr already tracks everything in Emby"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -237,7 +237,7 @@ echo ""
|
||||
echo "━━━ $ICON_SUMMARY Movies to add (${#MISSING[@]}) ━━━"
|
||||
for name in "${SORTED_MISSING[@]}"; do
|
||||
_tmdb="${MISSING[$name]}"
|
||||
echo " $name${_tmdb:+ (TMDB: $_tmdb)}"
|
||||
log " $name${_tmdb:+ (TMDB: $_tmdb)}"
|
||||
done
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
# without modification. Safe to run multiple times — the second run finds nothing to add.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION (master.conf / master_host*.conf)
|
||||
# CONFIGURATION (master.conf / host*.conf)
|
||||
# ==============================================================================================
|
||||
#
|
||||
# SONARR_EMBY_LIBRARIES — Emby library names to scan (master.conf)
|
||||
@@ -75,12 +75,12 @@ if ! command -v jq >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
if [[ -z "${EMBY_URL:-}" || -z "${EMBY_API_KEY:-}" ]]; then
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check master_host*.conf"
|
||||
error "EMBY_URL / EMBY_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${SONARR_URL:-}" || -z "${SONARR_API_KEY:-}" ]]; then
|
||||
error "SONARR_URL / SONARR_API_KEY not configured — check master_host*.conf"
|
||||
error "SONARR_URL / SONARR_API_KEY not configured — check host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -239,7 +239,7 @@ IFS=$'\n' SORTED_MISSING=($(printf '%s\n' "${!MISSING[@]}" | sort))
|
||||
echo " Already tracked: $ALREADY | Missing from Sonarr: ${#MISSING[@]}"
|
||||
|
||||
if [[ "${#MISSING[@]}" -eq 0 ]]; then
|
||||
log "Sonarr already tracks everything in Emby"
|
||||
echo "Sonarr already tracks everything in Emby"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -247,7 +247,7 @@ echo ""
|
||||
echo "━━━ $ICON_SUMMARY Series to add (${#MISSING[@]}) ━━━"
|
||||
for name in "${SORTED_MISSING[@]}"; do
|
||||
_tvdb="${MISSING[$name]}"
|
||||
echo " $name${_tvdb:+ (TVDB: $_tvdb)}"
|
||||
log " $name${_tvdb:+ (TVDB: $_tvdb)}"
|
||||
done
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
|
||||
@@ -151,7 +151,7 @@ if pgrep -f "fallback.sh" >/dev/null 2>&1; then
|
||||
echo ""
|
||||
warn "If you are sure you want to proceed anyway, confirm below"
|
||||
else
|
||||
log "fallback.sh is not running — safe to reset ✅"
|
||||
echo "fallback.sh is not running — safe to reset ✅"
|
||||
fi
|
||||
|
||||
# Check current state — if already NORMAL warn user
|
||||
@@ -238,8 +238,8 @@ if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
else
|
||||
warn "$ICON_DONE State reset to NORMAL"
|
||||
log "fallback.sh will resume from NORMAL on next cycle"
|
||||
log "No containers were started or stopped"
|
||||
echo "fallback.sh will resume from NORMAL on next cycle"
|
||||
echo "No containers were started or stopped"
|
||||
echo ""
|
||||
[[ "$FALLBACK_RUNNING" == true ]] && \
|
||||
warn "⚠️ fallback.sh was running during reset — monitor next cycle carefully"
|
||||
|
||||
Executable
+339
@@ -0,0 +1,339 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Ramdisk Stop ===============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Safely stops the transcode ramdisk: redirects the transcode symlink to the
|
||||
# SSD fallback before unmounting so Emby continues writing without interruption,
|
||||
# then unmounts the tmpfs and updates the state file.
|
||||
#
|
||||
# Primary use case: stopping the current ramdisk before re-running
|
||||
# ramdisk_setup.sh with new size or threshold values (setup is idempotent —
|
||||
# if the ramdisk is mounted, it skips the mount and reports status, so you
|
||||
# must stop it first to change the size).
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Executes in safe order:
|
||||
# 1. Validate — ramdisk mounted, SSD fallback exists
|
||||
# 2. Redirect symlink → SSD (Emby immediately writes to SSD instead)
|
||||
# 3. Warn if active transcode files still on ramdisk (informational — not a blocker)
|
||||
# 4. Unmount ramdisk tmpfs
|
||||
# 5. Update /tmp/transcode_state.db → current_target=TRANSCODE_SSD
|
||||
#
|
||||
# The symlink redirect happens before unmount so there is no window where Emby
|
||||
# has nowhere to write. Existing in-progress transcode files on the ramdisk are
|
||||
# lost on unmount — warn the user but proceed (this is expected for maintenance).
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# umount requires root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent stop attempts.
|
||||
#
|
||||
# Mounted Check
|
||||
# Exits cleanly if ramdisk is not mounted — nothing to do.
|
||||
#
|
||||
# Symlink-First Order
|
||||
# Symlink is redirected before unmount — Emby never sees a broken path.
|
||||
#
|
||||
# transcode_manager Warning
|
||||
# Warns if transcode_manager.sh is running — it may flip the symlink back
|
||||
# to ramdisk on its next cycle. Stop transcode_manager before running this
|
||||
# if you need the SSD redirect to hold.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# host*.conf
|
||||
# HOST*_TRANSCODE_SSD SSD fallback directory — redirect target during stop.
|
||||
# Aliased by detect_hosts() → TRANSCODE_SSD.
|
||||
#
|
||||
# master.conf
|
||||
# TRANSCODE_LINK Symlink Emby uses. Must match Emby's transcode path setting.
|
||||
# RAMDISK_PATH tmpfs mount point.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# /tmp/transcode_state.db — updated to current_target=TRANSCODE_SSD after stop.
|
||||
# transcode_manager.sh reads this on its next cycle.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ramdisk_stop.sh
|
||||
# Stop the ramdisk: redirect symlink → SSD, unmount, update state.
|
||||
#
|
||||
# ramdisk_stop.sh --dry-run
|
||||
# Show what would happen without making any changes.
|
||||
#
|
||||
# ramdisk_stop.sh --status
|
||||
# Show current mount state, symlink target, active files on ramdisk. Exit.
|
||||
#
|
||||
# ramdisk_stop.sh --log
|
||||
# Verbose output for each step.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — umount requires root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
validate_unraid_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
STATE_FILE="/tmp/transcode_state.db"
|
||||
|
||||
log "Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
log "Ramdisk: $RAMDISK_PATH"
|
||||
log "Fallback: $TRANSCODE_SSD"
|
||||
log "Symlink: $TRANSCODE_LINK"
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_RAM Ramdisk path: $RAMDISK_PATH"
|
||||
echo "$ICON_DISK SSD fallback: $TRANSCODE_SSD"
|
||||
echo "$ICON_LINK Symlink: $TRANSCODE_LINK"
|
||||
echo ""
|
||||
|
||||
if mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
|
||||
USAGE=$(df -BG "$RAMDISK_PATH" | awk 'NR==2 {print $3}')
|
||||
AVAIL=$(df -BG "$RAMDISK_PATH" | awk 'NR==2 {print $4}')
|
||||
echo " $ICON_RAM Ramdisk: mounted — $USAGE used / $AVAIL available ✅"
|
||||
FILE_COUNT=$(find "$RAMDISK_PATH" -type f 2>/dev/null | wc -l)
|
||||
echo " $ICON_RAM Active files on ramdisk: $FILE_COUNT"
|
||||
else
|
||||
echo " $ICON_RAM Ramdisk: NOT mounted"
|
||||
fi
|
||||
|
||||
if [[ -L "$TRANSCODE_LINK" ]]; then
|
||||
TARGET=$(readlink "$TRANSCODE_LINK")
|
||||
echo " $ICON_LINK Symlink: $TRANSCODE_LINK → $TARGET"
|
||||
else
|
||||
echo " $ICON_LINK Symlink: not set"
|
||||
fi
|
||||
|
||||
if [[ -f "$STATE_FILE" ]]; then
|
||||
echo ""
|
||||
echo " State file ($STATE_FILE):"
|
||||
while IFS='=' read -r key value; do
|
||||
[[ -z "$key" ]] && continue
|
||||
echo " $key = $value"
|
||||
done < "$STATE_FILE"
|
||||
else
|
||||
echo " $ICON_INFO State file: not found (ramdisk never started this boot)"
|
||||
fi
|
||||
|
||||
if pgrep -f "transcode_manager.sh" >/dev/null 2>&1; then
|
||||
echo ""
|
||||
warn "transcode_manager.sh is currently RUNNING"
|
||||
fi
|
||||
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Preflight ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Preflight ━━━"
|
||||
|
||||
# Bail if not mounted — nothing to do
|
||||
if ! mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
|
||||
warn "Ramdisk is not mounted at $RAMDISK_PATH — nothing to stop"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
log "Ramdisk is mounted ✅"
|
||||
|
||||
# Warn if transcode_manager is running — it may flip symlink back on next cycle
|
||||
if pgrep -f "transcode_manager.sh" >/dev/null 2>&1; then
|
||||
warn "transcode_manager.sh is currently RUNNING"
|
||||
warn "It may flip the symlink back to ramdisk on its next cycle"
|
||||
warn "Stop transcode_manager.sh first if you need the SSD redirect to hold"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Confirm SSD fallback exists
|
||||
if [[ ! -d "$TRANSCODE_SSD" ]]; then
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — SSD fallback does not exist: $TRANSCODE_SSD"
|
||||
warn "DRY RUN — would create it before redirecting symlink"
|
||||
else
|
||||
warn "SSD fallback does not exist — creating: $TRANSCODE_SSD"
|
||||
mkdir -p "$TRANSCODE_SSD" || {
|
||||
error "Failed to create SSD fallback: $TRANSCODE_SSD"
|
||||
error "Cannot safely redirect symlink — aborting"
|
||||
exit 1
|
||||
}
|
||||
log "SSD fallback created ✅"
|
||||
fi
|
||||
else
|
||||
log "SSD fallback exists: $TRANSCODE_SSD ✅"
|
||||
fi
|
||||
|
||||
START=$(date +%s)
|
||||
STOP_SUCCESS=true
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Redirect Symlink → SSD ━━━
|
||||
# ==============================================================================================
|
||||
# Redirect BEFORE unmount — Emby continues writing to SSD with no broken path window.
|
||||
echo ""
|
||||
echo "━━━ $ICON_LINK Redirect Symlink → SSD ━━━"
|
||||
|
||||
if [[ -L "$TRANSCODE_LINK" ]]; then
|
||||
CURRENT_TARGET=$(readlink "$TRANSCODE_LINK")
|
||||
if [[ "$CURRENT_TARGET" == "$TRANSCODE_SSD" ]]; then
|
||||
log "Symlink already points to SSD — no change needed"
|
||||
else
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would redirect: $TRANSCODE_LINK → $TRANSCODE_SSD"
|
||||
else
|
||||
ln -sfn "$TRANSCODE_SSD" "$TRANSCODE_LINK" && \
|
||||
warn "Symlink redirected: $TRANSCODE_LINK → $TRANSCODE_SSD ✅" || {
|
||||
error "Failed to redirect symlink"
|
||||
STOP_SUCCESS=false
|
||||
}
|
||||
fi
|
||||
fi
|
||||
elif [[ -e "$TRANSCODE_LINK" ]]; then
|
||||
warn "$TRANSCODE_LINK exists but is not a symlink — leaving as-is"
|
||||
else
|
||||
warn "Symlink $TRANSCODE_LINK does not exist — nothing to redirect"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Active Files Warning ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_RAM Active Files Check ━━━"
|
||||
|
||||
FILE_COUNT=$(find "$RAMDISK_PATH" -type f 2>/dev/null | wc -l)
|
||||
if [[ "$FILE_COUNT" -gt 0 ]]; then
|
||||
warn "⚠️ $FILE_COUNT file(s) still on ramdisk — will be lost on unmount"
|
||||
warn "Active transcode sessions should be stopped before unmounting"
|
||||
warn "Proceeding regardless (this is expected for maintenance)"
|
||||
if [[ "$LOG" == true ]]; then
|
||||
find "$RAMDISK_PATH" -type f 2>/dev/null | while read -r f; do
|
||||
log " $f"
|
||||
done
|
||||
fi
|
||||
else
|
||||
log "No active files on ramdisk ✅"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Unmount ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_RAM Unmount Ramdisk ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would unmount: $RAMDISK_PATH"
|
||||
else
|
||||
if umount "$RAMDISK_PATH" 2>/dev/null; then
|
||||
warn "Ramdisk unmounted: $RAMDISK_PATH ✅"
|
||||
else
|
||||
# Regular unmount failed — check if only directory handles are open (no active writes)
|
||||
OPEN_FILES=$(lsof +D "$RAMDISK_PATH" 2>/dev/null | awk 'NR>1 && $5 != "DIR"' | wc -l)
|
||||
if [[ "$OPEN_FILES" -eq 0 ]]; then
|
||||
warn "Busy — only directory handles open, no active writes — trying lazy unmount"
|
||||
if umount -l "$RAMDISK_PATH"; then
|
||||
warn "Ramdisk lazy-unmounted: $RAMDISK_PATH ✅"
|
||||
warn "Handles will release when owning processes next check the directory"
|
||||
else
|
||||
error "Lazy unmount also failed — $RAMDISK_PATH"
|
||||
STOP_SUCCESS=false
|
||||
fi
|
||||
else
|
||||
error "Failed to unmount $RAMDISK_PATH — $OPEN_FILES file(s) still open for writing"
|
||||
error "Stop active transcode sessions and retry"
|
||||
STOP_SUCCESS=false
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Update State File ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR State File ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would update $STATE_FILE: current_target=$TRANSCODE_SSD"
|
||||
elif [[ "$STOP_SUCCESS" == true ]]; then
|
||||
NOW=$(date +%s)
|
||||
cat > "$STATE_FILE" <<EOF
|
||||
current_target=$TRANSCODE_SSD
|
||||
last_flip_time=$NOW
|
||||
flip_count_hour=0
|
||||
flip_hour_start=$NOW
|
||||
EOF
|
||||
log "State file updated: current_target=$TRANSCODE_SSD"
|
||||
else
|
||||
warn "Skipping state file update — stop had errors"
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY RAMDISK STOP SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_RAM Ramdisk: $RAMDISK_PATH"
|
||||
echo "$ICON_DISK Fallback: $TRANSCODE_SSD"
|
||||
echo "$ICON_LINK Symlink: $TRANSCODE_LINK → $(readlink "$TRANSCODE_LINK" 2>/dev/null || echo "not set")"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$STOP_SUCCESS" == true ]]; then
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
echo "Run ramdisk_setup.sh to remount with new configuration"
|
||||
else
|
||||
echo "$ICON_ERROR Status: STOP HAD ERRORS"
|
||||
notify "Ramdisk stop errors on $(hostname) ($MY_ID) — check output" \
|
||||
"Ramdisk Stop" "warning"
|
||||
exit 1
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -293,6 +293,6 @@ elif [[ ${#FAILED[@]} -gt 0 ]]; then
|
||||
"Recreate Shares" "warning"
|
||||
exit 1
|
||||
else
|
||||
log "$ICON_DONE Status: done — ${#CREATED[@]} created, ${#SKIPPED[@]} skipped"
|
||||
echo "$ICON_DONE Status: done — ${#CREATED[@]} created, ${#SKIPPED[@]} skipped"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -152,7 +152,7 @@ fi
|
||||
|
||||
echo ""
|
||||
if [[ "$SKIP_COUNT" -eq 0 ]]; then
|
||||
log "Skip list: empty — all containers monitored normally ✅"
|
||||
echo "Skip list: empty — all containers monitored normally ✅"
|
||||
else
|
||||
warn "$SKIP_COUNT container(s) on skip list — manual intervention needed:"
|
||||
echo ""
|
||||
@@ -177,9 +177,9 @@ fi
|
||||
echo ""
|
||||
echo "━━━ $ICON_WATCHDOG Restart History ━━━"
|
||||
if [[ "$RESTART_COUNT" -eq 0 ]]; then
|
||||
log "No restart history"
|
||||
echo "No restart history"
|
||||
else
|
||||
log "$RESTART_COUNT restart entries (window: ${WATCHDOG_CONTAINER_RESTART_WINDOW}h)"
|
||||
echo "$RESTART_COUNT restart entries (window: ${WATCHDOG_CONTAINER_RESTART_WINDOW}h)"
|
||||
echo ""
|
||||
awk -F'|' '{counts[$1]++} END {
|
||||
for (c in counts)
|
||||
|
||||
@@ -50,7 +50,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_ZFS_REPORT_IGNORE_POOLS
|
||||
# Pools to exclude from automatic scrub. Typically single-disk VM pools
|
||||
@@ -281,7 +281,7 @@ for pool in "${STARTED[@]}"; do
|
||||
error " $ERRORS"
|
||||
POOLS_ERRORS+=("$pool")
|
||||
else
|
||||
log "$pool — $SCAN_LINE"
|
||||
echo "$pool — $SCAN_LINE"
|
||||
POOLS_OK+=("$pool")
|
||||
fi
|
||||
done
|
||||
@@ -304,7 +304,7 @@ if [[ ${#POOLS_ERRORS[@]} -gt 0 ]]; then
|
||||
notify "ZFS scrub errors on $(hostname) ($MY_ID) — pools with errors: ${POOLS_ERRORS[*]}" \
|
||||
"ZFS Scrub" "warning"
|
||||
elif [[ ${#POOLS_OK[@]} -gt 0 ]]; then
|
||||
log "$ICON_DONE Status: all ${#POOLS_OK[@]} pools clean ✅"
|
||||
echo "$ICON_DONE Status: all ${#POOLS_OK[@]} pools clean ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
@@ -18,6 +18,24 @@ for the ramdisk transcode system. Read the Docker mount section before anything
|
||||
|
||||
---
|
||||
|
||||
## Output Tiers
|
||||
|
||||
All three scripts follow a two-tier output model: `echo` lines are always visible;
|
||||
`log` lines only appear when `--log` is passed.
|
||||
|
||||
**ramdisk_setup.sh** — one-shot at array start. Without `--log`, section headers
|
||||
and the final summary are visible. Per-step creation detail suppressed.
|
||||
|
||||
**transcode_manager.sh** — runs every 3 minutes. Without `--log`, only state
|
||||
transitions (flips, warnings, errors) and active session display are shown. When
|
||||
nothing changes, a single one-line confirmation is printed. Per-check detail suppressed.
|
||||
|
||||
**transcode_cleanup.sh** — runs every 3 minutes. Without `--log`, the cleanup
|
||||
summary (files removed, space freed) is always visible. Per-file deletion detail
|
||||
suppressed.
|
||||
|
||||
---
|
||||
|
||||
## Docker Mount — Required Configuration
|
||||
|
||||
> **This is the most important configuration requirement in this folder.**
|
||||
@@ -198,7 +216,7 @@ TRANSCODE_MANAGER_MODE="smart" # smart | ramdisk | ssd
|
||||
### Threshold Sizing
|
||||
|
||||
```bash
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
HOST1_RAMDISK_SIZE="10G"
|
||||
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
|
||||
HOST1_RAMDISK_LOW_GB=6.5 # flip back below this
|
||||
@@ -313,7 +331,7 @@ transcode_cleanup.sh --log # verbose per-file output
|
||||
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at
|
||||
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback
|
||||
|
||||
# ── Per-Host (master_host*.conf) ───────────────────────────────────────────────
|
||||
# ── Per-Host (host*.conf) ───────────────────────────────────────────────
|
||||
HOST1_RAMDISK_PATH="/mnt/ramdisk_transcodes" # ramdisk mount point
|
||||
HOST1_RAMDISK_SIZE="10G" # tmpfs ceiling (not a reservation)
|
||||
HOST1_RAMDISK_WARN_GB=8.8 # flip to SSD above this
|
||||
@@ -402,7 +420,7 @@ docker inspect Emby | grep -A3 "Mounts"
|
||||
# Check peak usage from the weekly health digest: Transcodes → "Week peak: X.XGB"
|
||||
#
|
||||
# If peak is close to RAMDISK_WARN_GB → increase ramdisk size:
|
||||
# master_host1.conf
|
||||
# host1.conf
|
||||
HOST1_RAMDISK_SIZE="12G" # increase by 2G
|
||||
HOST1_RAMDISK_WARN_GB=10.5 # adjust thresholds accordingly
|
||||
HOST1_RAMDISK_LOW_GB=8.5
|
||||
@@ -447,7 +465,7 @@ readlink /mnt/ram-transcode
|
||||
### Increasing Ramdisk Size After Initial Setup
|
||||
|
||||
```bash
|
||||
# 1. Set new size and thresholds in master_host*.conf
|
||||
# 1. Set new size and thresholds in host*.conf
|
||||
# 2. Unmount the existing ramdisk (no sessions should be active):
|
||||
umount /mnt/ramdisk_transcodes
|
||||
|
||||
|
||||
@@ -53,7 +53,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_RAMDISK_SIZE
|
||||
# tmpfs ceiling (e.g. 10G). Must change together with WARN_GB and LOW_GB.
|
||||
@@ -175,8 +175,8 @@ fi
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_RAM Ramdisk — $MY_ID ━━━"
|
||||
echo "$ICON_RAM Path: $RAMDISK_PATH"
|
||||
echo "$ICON_RAM Size: $RAMDISK_SIZE"
|
||||
log "$ICON_RAM Path: $RAMDISK_PATH"
|
||||
log "$ICON_RAM Size: $RAMDISK_SIZE"
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
@@ -355,7 +355,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$SETUP_SUCCESS" == true ]]; then
|
||||
log "$ICON_DONE Status: done ✅"
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
else
|
||||
echo "$ICON_ERROR Status: SETUP HAD ERRORS"
|
||||
notify "Ramdisk setup errors on $(hostname) ($MY_ID) — check output" \
|
||||
|
||||
@@ -65,7 +65,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD / HOST*_RAMDISK_LOW_GB
|
||||
# Aliased by detect_hosts() → RAMDISK_PATH / TRANSCODE_SSD / RAMDISK_LOW_GB.
|
||||
@@ -292,7 +292,7 @@ if [[ "$DRY_RUN" == false ]] && mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
|
||||
CURRENT_TARGET=$(grep "^current_target=" "$STATE_FILE" 2>/dev/null | cut -d= -f2)
|
||||
|
||||
if [[ "$LOW_RECOVERED" == "1" && "$CURRENT_TARGET" == "$TRANSCODE_SSD" ]]; then
|
||||
log "Ramdisk has space after cleanup (${RAMDISK_USED_GB}GB < ${RAMDISK_LOW_GB}GB) — triggering manager to flip back"
|
||||
echo "Ramdisk has space after cleanup (${RAMDISK_USED_GB}GB < ${RAMDISK_LOW_GB}GB) — triggering manager to flip back"
|
||||
bash "$SCRIPT_DIR/transcode_manager.sh" --no-log
|
||||
fi
|
||||
fi
|
||||
@@ -314,6 +314,6 @@ echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no files deleted"
|
||||
else
|
||||
log "$ICON_DONE Status: done ✅"
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -65,7 +65,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_RAMDISK_PATH / HOST*_TRANSCODE_SSD
|
||||
# Ramdisk mount point and SSD fallback path.
|
||||
@@ -650,5 +650,5 @@ if [[ "$SOMETHING_HAPPENED" == true ]]; then
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
log "Transcode manager — all healthy — $MY_ID ($(format_duration $(( END - START ))))"
|
||||
echo "Transcode manager — all healthy — $MY_ID ($(format_duration $(( END - START ))))"
|
||||
fi
|
||||
@@ -0,0 +1,641 @@
|
||||
# ━━━━━ WATCHDOGS — Manual ━━━━━
|
||||
|
||||
Configuration reference, operational procedures, and troubleshooting for all four
|
||||
watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
|
||||
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [Output Tiers](#output-tiers)
|
||||
- [resource_watchdog.sh](#resource_watchdogsh)
|
||||
- [docker_watchdog.sh](#docker_watchdogsh)
|
||||
- [storage_watchdog.sh](#storage_watchdogsh)
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Output Tiers
|
||||
|
||||
All watchdog scripts use a two-tier output model: `echo` lines are always visible;
|
||||
`log` lines only appear when `--log` is passed.
|
||||
|
||||
All four watchdogs are **single-pass scripts** called once per minute by the orchestrator.
|
||||
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
|
||||
visible. Per-check detail is suppressed on clean cycles.
|
||||
|
||||
`docker_watchdog.sh` is the exception — it is **silent on clean cycles by design**.
|
||||
96 cycles/day means clean-cycle noise would bury real events. Its output only appears
|
||||
when there are restarts, skip-list events, or RAM deferral. Use `--log` to see
|
||||
per-cycle detail on clean cycles.
|
||||
|
||||
---
|
||||
|
||||
## resource_watchdog.sh
|
||||
|
||||
Runs first in the orchestrator sequence. Reduces system pressure before docker_watchdog
|
||||
attempts any container restarts. Containers restarted into a RAM-pressured system just
|
||||
fail again — this script ensures docker_watchdog has breathing room.
|
||||
|
||||
### Pressure Levels
|
||||
|
||||
Three escalating levels, each additive:
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
RW_RAM_SOFT_GB=20 # Level 1 trigger — throttle downloaders
|
||||
RW_RAM_MEDIUM_GB=15 # Level 2 trigger — throttle + pause containers
|
||||
RW_RAM_HARD_GB=10 # Level 3 trigger — stop containers + defer docker_watchdog
|
||||
RW_RAM_RECOVER_GB=25 # de-escalate only after RAM reaches this
|
||||
|
||||
RW_LOAD_SOFT_MULTIPLIER=2.0 # load > 2× cpu count = level 1
|
||||
RW_LOAD_MEDIUM_MULTIPLIER=3.0 # load > 3× cpu count = level 2
|
||||
|
||||
RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalating
|
||||
```
|
||||
|
||||
**Level 1 (soft):** Throttle SABnzbd + qBittorrent download speeds.
|
||||
**Level 2 (medium):** Further throttle + `docker pause` non-critical containers.
|
||||
**Level 3 (hard):** `docker stop` optional services + write `mem_shutdown_active=true`
|
||||
to `RW_STATE_FILE`. docker_watchdog.sh reads this flag and skips all restart logic
|
||||
until pressure clears. Without this coordination, docker_watchdog would immediately
|
||||
restart containers that resource_watchdog just stopped to free RAM.
|
||||
|
||||
Recovery de-escalates one level at a time — prevents flip-flopping between states.
|
||||
|
||||
### Per-Host Container Lists
|
||||
|
||||
```bash
|
||||
# host1.conf
|
||||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at level 2
|
||||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at level 3
|
||||
```
|
||||
|
||||
Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
|
||||
pressure level. Default: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
|
||||
|
||||
### Downloader Throttle Config
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
RW_SABNZBD_ENABLED=true
|
||||
RW_SABNZBD_SPEED_SOFT="50M" # throttled at level 1
|
||||
RW_SABNZBD_SPEED_MEDIUM="10M" # throttled further at level 2
|
||||
|
||||
RW_QBIT_ENABLED=true
|
||||
RW_QBIT_DL_SOFT=51200 # KB/s — level 1
|
||||
RW_QBIT_DL_MEDIUM=10240 # KB/s — level 2
|
||||
|
||||
# host1.conf (API access)
|
||||
HOST1_SABNZBD_URL="http://localhost:8080"
|
||||
HOST1_SABNZBD_API_KEY="your-api-key"
|
||||
HOST1_QBIT_URL="http://localhost:8090"
|
||||
HOST1_QBIT_USERNAME="admin"
|
||||
HOST1_QBIT_PASSWORD="your-password"
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
resource_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
resource_watchdog.sh --dry-run # show what would be throttled/paused/stopped
|
||||
resource_watchdog.sh --status # current level, active actions, recovery cycle count
|
||||
resource_watchdog.sh --log # verbose per-check output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## docker_watchdog.sh
|
||||
|
||||
Runs second in the orchestrator sequence. Two-tier container healing — explicit
|
||||
per-container configuration (Tier 1) plus a global catch-all scan (Tier 2).
|
||||
|
||||
Reads `RW_STATE_FILE` at cycle start — if `mem_shutdown_active=true`, skips all
|
||||
container restart logic (resource_watchdog is managing the situation). Health URL
|
||||
checks for excluded containers still run.
|
||||
|
||||
### Memory Hard Limits
|
||||
|
||||
```bash
|
||||
# host1.conf
|
||||
# Format: "ContainerName:LimitInMB"
|
||||
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
|
||||
#
|
||||
# Sizing: check normal peak with "docker stats ContainerName"
|
||||
# Set limit at ~150-200% of normal peak
|
||||
#
|
||||
HOST1_WATCHDOG_CONTAINERS=(
|
||||
"Emby:18432" # 18GB — peaks ~12GB under heavy transcode load
|
||||
"LidaTube:6144" # 6GB — YouTube downloader, grows with large queues
|
||||
"Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature
|
||||
"Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous
|
||||
)
|
||||
```
|
||||
|
||||
A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
|
||||
visibility into a container approaching its ceiling before a restart is triggered.
|
||||
|
||||
### CPU Thresholds
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# CPU is normalised against total core count.
|
||||
# 85% normalised on a 16-core machine = 13.6 cores worth of a single process.
|
||||
#
|
||||
# CPU uses a STRIKE SYSTEM — brief spikes are normal (Tdarr, Emby transcoding, SABnzbd).
|
||||
# Strike 1: above HARD_CPU_THRESHOLD → warn, increment strike
|
||||
# Strike 2: above threshold → restart, reset counter
|
||||
# Recovery: drops below threshold any cycle → reset to 0
|
||||
#
|
||||
SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only
|
||||
HARD_CPU_THRESHOLD=85 # strike at 85% normalised
|
||||
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
|
||||
```
|
||||
|
||||
### HTTP Health Checks
|
||||
|
||||
```bash
|
||||
# host1.conf
|
||||
# Format: "ContainerName:http://host:port/optional-path"
|
||||
# Two consecutive non-responses trigger a restart.
|
||||
# "Container running" and "service responding" are not the same thing.
|
||||
#
|
||||
HOST1_WATCHDOG_CONTAINER_URLS=(
|
||||
"Emby:http://localhost:8096" # Emby WebUI root
|
||||
"NginxProxyManager:http://localhost:81" # NPM admin interface
|
||||
)
|
||||
|
||||
# master.conf
|
||||
CURL_TIMEOUT=5 # seconds before non-response counts as a failure
|
||||
RESP_FAIL_LIMIT=2 # consecutive failures before restart
|
||||
```
|
||||
|
||||
### Required Containers
|
||||
|
||||
```bash
|
||||
# host1.conf
|
||||
# Found stopped → restart attempted every cycle until running or skip-listed.
|
||||
#
|
||||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
||||
"NginxProxyManager"
|
||||
"Authelia"
|
||||
"Mariadb-Authelia"
|
||||
"Redis-Authelia"
|
||||
)
|
||||
```
|
||||
|
||||
### Dependency Ordering
|
||||
|
||||
```bash
|
||||
# host1.conf
|
||||
# Format: "DependentContainer:dependency1 dependency2"
|
||||
# All dependencies must be running before the dependent is restarted.
|
||||
# Prevents Authelia crash-looping while MariaDB is still starting.
|
||||
#
|
||||
HOST1_WATCHDOG_DEPENDENCIES=(
|
||||
"Authelia:Mariadb-Authelia Redis-Authelia"
|
||||
"NextCloud:Postgres-NextCloud"
|
||||
)
|
||||
```
|
||||
|
||||
Same dependency config is used by `docker_daily_restart.sh` and
|
||||
`docker_weekly_restart.sh` — configure once, applies everywhere.
|
||||
|
||||
### Startup Grace Period
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# Suppress restart actions for N seconds after array start.
|
||||
# Checks still run and log — only restart actions are suppressed.
|
||||
#
|
||||
WATCHDOG_STARTUP_GRACE=600 # 10 minutes
|
||||
```
|
||||
|
||||
### Tier 2 Global Scan
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WATCHDOG_SCAN_ALL=true # enable global scan
|
||||
WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures
|
||||
WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills
|
||||
WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts
|
||||
WATCHDOG_RESTART_DEAD=true # recover containers in dead state
|
||||
WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero
|
||||
WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list
|
||||
|
||||
# Exclude containers from Tier 2 (one-shots, manually managed, benign exits):
|
||||
WATCHDOG_SCAN_IGNORE=(
|
||||
"my-one-shot-container"
|
||||
)
|
||||
```
|
||||
|
||||
### Restart Loop Protection
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list
|
||||
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
|
||||
```
|
||||
|
||||
After hitting the limit: skip list + critical notification. The watchdog stops
|
||||
touching the container. Auto-clear: if the container recovers on its own and is
|
||||
found running, it's removed from the skip list automatically. Manual clear is only
|
||||
needed when the container is stuck stopped — use `Tools/watchdog_skip_list_manager.sh`.
|
||||
|
||||
### Notification Batching
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WATCHDOG_BATCH_NOTIFY=true
|
||||
# All events from one cycle → one notification at the end.
|
||||
# A shared DB going down can cascade 10+ containers. Without batching: 10 pings.
|
||||
# With batching: one summary listing all affected containers.
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
docker_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
docker_watchdog.sh --dry-run # full cycle preview without restarting anything
|
||||
docker_watchdog.sh --status # skip list, strike counts, grace period, RAM deferral state
|
||||
docker_watchdog.sh --log # verbose per-cycle output
|
||||
```
|
||||
|
||||
### Skip List Recovery
|
||||
|
||||
```bash
|
||||
# Step 1 — understand the situation
|
||||
Tools/watchdog_skip_list_manager.sh --status
|
||||
|
||||
# Step 2 — fix the underlying problem
|
||||
# docker logs ContainerName --tail 100
|
||||
# df -h /mnt/user
|
||||
|
||||
# Step 3 — clear the container
|
||||
Tools/watchdog_skip_list_manager.sh --clear ContainerName
|
||||
|
||||
# Step 4 — start manually (confirms fix before handing back to watchdog)
|
||||
docker start ContainerName
|
||||
|
||||
# Step 5 — monitoring resumes automatically on next cycle
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## storage_watchdog.sh
|
||||
|
||||
Runs third in the orchestrator sequence. Two independent checks per cycle:
|
||||
growth rate detection (automatic, zero config) and oversize log detection.
|
||||
Uses its own strike state file — independent from docker_watchdog.
|
||||
|
||||
### Growth Rate Detection
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WATCHDOG_CHECK_APPDATA=true
|
||||
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
|
||||
WATCHDOG_APPDATA_GROWTH_GB=2 # growth per cycle that triggers a strike
|
||||
WATCHDOG_APPDATA_STRIKE_LIMIT=3 # strikes before alert
|
||||
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db" # size baseline
|
||||
```
|
||||
|
||||
Runs `du -sm appdata/*/` each cycle — pure inode metadata, very lightweight on NVMe.
|
||||
Compares each container's current size to the baseline from the previous cycle.
|
||||
Growth > `WATCHDOG_APPDATA_GROWTH_GB` per cycle increments the container's strike count.
|
||||
Strike 1: warn. Strike 2: escalate. Strike 3: critical alert.
|
||||
Strikes auto-clear when growth drops to zero (condition resolved).
|
||||
|
||||
**Zero configuration required for new containers.** Growth rate detection covers all
|
||||
containers automatically. The suppress array below is only for known-legitimate growth.
|
||||
|
||||
### Growth Suppress Ceilings
|
||||
|
||||
```bash
|
||||
# host1.conf
|
||||
# ONLY needed in specific cases — growth rate detection covers everything automatically.
|
||||
# Use when a container's appdata legitimately grows fast during normal operation
|
||||
# and you want to suppress false positives above a known-safe threshold.
|
||||
#
|
||||
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||||
["Tdarr"]="25600" # 25GB — transcode cache grows during active jobs
|
||||
["7dtd"]="20480" # 20GB — game server world data, expected large
|
||||
)
|
||||
```
|
||||
|
||||
### Log File Detection
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WATCHDOG_APPDATA_LOG_MAX_GB=2 # *.log / *.log.* files above this trigger a strike
|
||||
WATCHDOG_APPDATA_TRUNCATE_LOGS=false # true: truncate at strike limit; false: alert only
|
||||
```
|
||||
|
||||
Scans all `*.log` and `*.log.*` files across all appdata paths. Files over the threshold
|
||||
increment per-file strike counts. At strike limit: truncates in-place with `truncate -s 0`
|
||||
(container keeps its file handle — space reclaimed immediately without container restart)
|
||||
or sends a critical alert if truncation is disabled.
|
||||
|
||||
Log strikes auto-clear when the file drops below threshold.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
storage_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings
|
||||
storage_watchdog.sh --dry-run # show what would be alerted/truncated
|
||||
storage_watchdog.sh --log # verbose per-container output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## system_watchdog.sh
|
||||
|
||||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||||
to reboot. Watches the server itself — not containers, not storage. Reboots only when
|
||||
healing at every other layer has failed or when the failure is non-recoverable.
|
||||
|
||||
### Three-Tier Response
|
||||
|
||||
**Tier 1 — CRITICAL (immediate reboot, no strikes)**
|
||||
| Condition | Threshold | Why immediate |
|
||||
|-----------|-----------|---------------|
|
||||
| Docker daemon unresponsive | N/A | Every docker command hangs — nothing can be healed |
|
||||
| rootfs usage | `SYS_WATCHDOG_ROOTFS_CRITICAL_PCT` (99%) | SSH stops; state files fail silently |
|
||||
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
|
||||
| File descriptor exhaustion | `SYS_WATCHDOG_FD_CRITICAL_PCT` (95%) | New connections silently failing |
|
||||
| /boot read-only | write test fails | Config writes silently failing |
|
||||
|
||||
**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
|
||||
RAM below `MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`.
|
||||
Both conditions required — RAM alone uses the standard strike system.
|
||||
OOM confirms the system is dying faster than watchdogs can heal.
|
||||
|
||||
**Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKES` consecutive failures → reboot)**
|
||||
| Check | Threshold |
|
||||
|-------|-----------|
|
||||
| Free RAM | `MEM_WARN_GB` → `MEM_SHUTDOWN_GB` → `MEM_GB` |
|
||||
| Load average | `SYS_WATCHDOG_LOAD_MULTIPLIER` × cpu_count |
|
||||
| CPU temperature | `SYS_WATCHDOG_CPU_TEMP` |
|
||||
| Zombie processes | `SYS_WATCHDOG_ZOMBIES` |
|
||||
| /var/log usage | `SYS_WATCHDOG_VAR_LOG_PCT` |
|
||||
| /tmp usage | `SYS_WATCHDOG_TMP_PCT` |
|
||||
| Array disk errors | mdstat error delta > 0 |
|
||||
| NIC state | interface operstate != "up" |
|
||||
| Required containers | containers in `SYS_WATCHDOG_REQUIRED_CONTAINERS` |
|
||||
|
||||
### RAM Tiers
|
||||
|
||||
```
|
||||
MEM_WARN_GB (10GB) → warn + notify, no action
|
||||
MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery
|
||||
MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM)
|
||||
MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart
|
||||
```
|
||||
|
||||
At `MEM_SHUTDOWN_GB`, all containers NOT listed in `SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED`
|
||||
are stopped. Adjust in master.conf for your critical services.
|
||||
|
||||
### Abort Conditions
|
||||
|
||||
Prevent reboot — running them would risk data loss:
|
||||
|
||||
```bash
|
||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=true # Parity check/rebuild running
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=true # Mover running
|
||||
```
|
||||
|
||||
Tier 1 CRITICAL bypasses all abort conditions — an imminent crash outweighs data
|
||||
safety concerns.
|
||||
|
||||
### Reboot Rate Limit
|
||||
|
||||
```bash
|
||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # window in hours
|
||||
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots within the window
|
||||
```
|
||||
|
||||
If the server reboots `SYS_WATCHDOG_MAX_REBOOTS` times within the window, the watchdog
|
||||
switches from rebooting to notifying only. Prevents a boot loop where the watchdog
|
||||
reboots → something crashes again immediately → reboot again.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
system_watchdog.sh --dry-run # run detection logic without rebooting
|
||||
system_watchdog.sh --status # thresholds, current state, strike counts
|
||||
system_watchdog.sh --log # verbose per-check output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
|
||||
# ── Resource Watchdog ──────────────────────────────────────────────────────────
|
||||
RW_ENABLED=true
|
||||
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
|
||||
|
||||
RW_RAM_SOFT_GB=20
|
||||
RW_RAM_MEDIUM_GB=15
|
||||
RW_RAM_HARD_GB=10
|
||||
RW_RAM_RECOVER_GB=25
|
||||
|
||||
RW_LOAD_SOFT_MULTIPLIER=2.0
|
||||
RW_LOAD_MEDIUM_MULTIPLIER=3.0
|
||||
RW_RECOVER_CYCLES=3
|
||||
|
||||
RW_SABNZBD_ENABLED=true
|
||||
RW_SABNZBD_SPEED_SOFT="50M"
|
||||
RW_SABNZBD_SPEED_MEDIUM="10M"
|
||||
RW_QBIT_ENABLED=true
|
||||
RW_QBIT_DL_SOFT=51200 # KB/s
|
||||
RW_QBIT_DL_MEDIUM=10240
|
||||
|
||||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
|
||||
|
||||
# host*.conf
|
||||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
|
||||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
|
||||
HOST1_SABNZBD_URL="http://localhost:8080"
|
||||
HOST1_SABNZBD_API_KEY="your-api-key"
|
||||
HOST1_QBIT_URL="http://localhost:8090"
|
||||
HOST1_QBIT_USERNAME="admin"
|
||||
HOST1_QBIT_PASSWORD="your-password"
|
||||
|
||||
# ── Docker Watchdog ────────────────────────────────────────────────────────────
|
||||
DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes)
|
||||
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
|
||||
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
|
||||
|
||||
SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart)
|
||||
|
||||
SOFT_CPU_THRESHOLD=50
|
||||
HARD_CPU_THRESHOLD=85
|
||||
CPU_FAIL_LIMIT=2
|
||||
|
||||
CURL_TIMEOUT=5
|
||||
RESP_FAIL_LIMIT=2
|
||||
|
||||
WATCHDOG_CONTAINER_RESTART_LIMIT=3
|
||||
WATCHDOG_CONTAINER_RESTART_WINDOW=1
|
||||
|
||||
WATCHDOG_SCAN_ALL=true
|
||||
WATCHDOG_SCAN_IGNORE=()
|
||||
WATCHDOG_RESTART_UNHEALTHY=true
|
||||
WATCHDOG_NOTIFY_OOM=true
|
||||
WATCHDOG_NOTIFY_CRASHLOOP=true
|
||||
WATCHDOG_CRASH_LIMIT=5
|
||||
WATCHDOG_RESTART_DEAD=true
|
||||
WATCHDOG_RESTART_CRASHED=true
|
||||
WATCHDOG_BATCH_NOTIFY=true
|
||||
|
||||
# State files:
|
||||
WATCHDOG_STATE_FILE="/tmp/watchdog_state.db"
|
||||
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
||||
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
|
||||
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
|
||||
|
||||
# host*.conf
|
||||
HOST1_WATCHDOG_CONTAINERS=() # "ContainerName:LimitMB"
|
||||
HOST1_WATCHDOG_CONTAINER_URLS=() # "ContainerName:http://host:port"
|
||||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
|
||||
HOST1_WATCHDOG_DEPENDENCIES=() # "Dependent:dep1 dep2"
|
||||
|
||||
# ── Storage Watchdog ───────────────────────────────────────────────────────────
|
||||
WATCHDOG_CHECK_APPDATA=true
|
||||
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
|
||||
WATCHDOG_APPDATA_GROWTH_GB=2
|
||||
WATCHDOG_APPDATA_LOG_MAX_GB=2
|
||||
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
|
||||
WATCHDOG_APPDATA_STRIKE_LIMIT=3
|
||||
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
|
||||
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
|
||||
|
||||
# host*.conf (optional — only for suppress ceilings)
|
||||
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||||
# ["Tdarr"]="25600"
|
||||
# )
|
||||
|
||||
# ── System Watchdog ────────────────────────────────────────────────────────────
|
||||
SYS_WATCHDOG_STRIKE_LIMIT=2
|
||||
SYSTEM_WATCHDOG_INTERVAL=300
|
||||
|
||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2
|
||||
SYS_WATCHDOG_MAX_REBOOTS=3
|
||||
SYS_WATCHDOG_OOM_LIMIT=3
|
||||
|
||||
MEM_WARN_GB=10
|
||||
MEM_SHUTDOWN_GB=6
|
||||
MEM_GB=4
|
||||
MEM_RECOVER_GB=30
|
||||
|
||||
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99
|
||||
SYS_WATCHDOG_FD_CRITICAL_PCT=95
|
||||
SYS_WATCHDOG_LOAD_MULTIPLIER=4
|
||||
SYS_WATCHDOG_CPU_TEMP=85
|
||||
SYS_WATCHDOG_ZOMBIES=20
|
||||
SYS_WATCHDOG_VAR_LOG_PCT=80
|
||||
SYS_WATCHDOG_TMP_PCT=85
|
||||
|
||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=true
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=true
|
||||
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
|
||||
"NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
|
||||
)
|
||||
SYS_WATCHDOG_REQUIRED_CONTAINERS=()
|
||||
|
||||
# State files:
|
||||
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
|
||||
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### resource_watchdog Paused Containers It Shouldn't
|
||||
|
||||
```bash
|
||||
# Check current state:
|
||||
resource_watchdog.sh --status
|
||||
|
||||
# Add to RW_CRITICAL_CONTAINERS in master.conf:
|
||||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
|
||||
|
||||
# Un-pause manually if needed:
|
||||
docker unpause MyContainer
|
||||
```
|
||||
|
||||
### docker_watchdog Keeps Restarting a Healthy Container
|
||||
|
||||
```bash
|
||||
# Check what's triggering it — memory, CPU, HTTP, or required:
|
||||
docker_watchdog.sh --status
|
||||
|
||||
# Check CPU normalised — brief spikes should not trigger (2-strike system):
|
||||
# If triggering on CPU: check if HARD_CPU_THRESHOLD is set appropriately
|
||||
# for containers with legitimate burst usage (Tdarr encoding, SABnzbd unpacking)
|
||||
|
||||
# Check HTTP — is the health endpoint returning 200?
|
||||
curl -sf --max-time 5 http://localhost:PORT && echo "OK" || echo "FAIL"
|
||||
```
|
||||
|
||||
### Container on the Skip List After Fixing the Problem
|
||||
|
||||
```bash
|
||||
# See skip list and container state:
|
||||
Tools/watchdog_skip_list_manager.sh --status
|
||||
|
||||
# Fix root cause first, then clear:
|
||||
Tools/watchdog_skip_list_manager.sh --clear ContainerName
|
||||
|
||||
# Start manually to confirm fix before handing back to watchdog:
|
||||
docker start ContainerName
|
||||
```
|
||||
|
||||
### storage_watchdog Alerting on a Container That Grows Legitimately
|
||||
|
||||
```bash
|
||||
# Check what triggered it:
|
||||
storage_watchdog.sh --status
|
||||
|
||||
# Add a suppress ceiling to host*.conf:
|
||||
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||||
# ["ContainerName"]="10240" # 10GB ceiling — legitimate growth, suppress below this
|
||||
# )
|
||||
```
|
||||
|
||||
### system_watchdog Rebooted Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check the reboot log (survives reboots):
|
||||
cat /boot/config/system_watchdog_reboots.db
|
||||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||||
|
||||
# Check syslog near the reboot time:
|
||||
grep "system_watchdog" /var/log/syslog | tail -20
|
||||
```
|
||||
|
||||
### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
|
||||
|
||||
```bash
|
||||
# All four watchdogs run as single-pass scripts — there is no background process to check.
|
||||
# If the orchestrator reports a timeout, one pass took longer than expected.
|
||||
|
||||
# Check the orchestrator itself:
|
||||
Orchestrators/watchdog_orchestrator.sh --status
|
||||
|
||||
# Run the slow watchdog directly with --log to see where it's hanging:
|
||||
Watchdogs/system_watchdog.sh --log --dry-run
|
||||
```
|
||||
@@ -0,0 +1,176 @@
|
||||
# ━━━━━ WATCHDOGS ━━━━━
|
||||
|
||||
**Four single-pass scripts that run every minute through `watchdog_orchestrator.sh`,
|
||||
each with a clear lane:** reduce system pressure → heal containers → protect storage →
|
||||
reboot if nothing else worked. They never run standalone loops. The orchestrator calls
|
||||
them in order, once per cron cycle.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Container Memory Leaks Going Undetected for Days**
|
||||
Emby's transcode session handling occasionally leaks memory. SABnzbd's Python process
|
||||
expands slowly across downloads. Neither crashes dramatically — they just consume more
|
||||
RAM until the system starts swapping. Docker reports both containers as `Up 14 days`.
|
||||
Nothing alerts. By the time someone notices, the system has been degraded for hours.
|
||||
|
||||
Fix: `docker_watchdog.sh` — hard per-container memory ceilings. When a container
|
||||
exceeds its limit the watchdog restarts it immediately. No strikes, no waiting.
|
||||
A memory leak is not a transient spike.
|
||||
|
||||
**Containers That Look Running But Aren't Responding**
|
||||
Docker reports a container as `Up` while its application layer has been frozen for
|
||||
hours. The reverse proxy forwards traffic to a service that returns nothing. Users see
|
||||
a broken page. Docker sees a healthy container.
|
||||
|
||||
Fix: `docker_watchdog.sh` — HTTP health checks on the actual service port every cycle.
|
||||
Two consecutive non-responses trigger a restart. Process running and service responding
|
||||
are not the same thing.
|
||||
|
||||
**System Pressure Causing Docker Watchdog to Undo Itself**
|
||||
Resource pressure builds. RAM drops. docker_watchdog.sh tries to restart a container
|
||||
into a system that's already swapping — the restarted container fails immediately
|
||||
and goes on the skip list. The real problem (RAM pressure) is never addressed.
|
||||
|
||||
Fix: `resource_watchdog.sh` runs first in the orchestrator sequence. At Level 1 it
|
||||
throttles downloaders. At Level 2 it pauses non-critical containers. At Level 3 it
|
||||
stops heavy services and signals docker_watchdog to defer all restarts. By the time
|
||||
docker_watchdog runs, the system has breathing room to actually heal.
|
||||
|
||||
**Runaway Log Files Filling a Pool Before Anyone Notices**
|
||||
A game server container was offline for a year, restarted for a weekend, and wrote
|
||||
130GB of logs to the appdata pool. The pool grew 13% in days. No alert fired —
|
||||
nothing was watching for growth at the data level, only at the container level.
|
||||
|
||||
Fix: `storage_watchdog.sh` — growth-rate scan of every container's appdata directory
|
||||
every cycle. No per-container configuration required. Runaway growth gets three
|
||||
cycles to be confirmed, then alerts and (optionally) truncates logs automatically.
|
||||
|
||||
**Rootfs at 99% With SSH Failing Silently**
|
||||
Rootfs fills. SSH stops accepting new connections. Docker can't write log files. State
|
||||
files fail silently. The server is functionally dead but still technically running.
|
||||
Nothing in the container layer can detect or recover from this — it requires a reboot.
|
||||
|
||||
Fix: `system_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
|
||||
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
|
||||
sequence so container healing and pressure reduction always get a chance first.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
Four watchdogs. One purpose each. Fixed execution order via `watchdog_orchestrator.sh`.
|
||||
|
||||
```
|
||||
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
|
||||
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
|
||||
Storage protection storage_watchdog.sh — pool growth rate + runaway log detection
|
||||
Last resort system_watchdog.sh — reboot only when nothing else can recover
|
||||
```
|
||||
|
||||
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
|
||||
attempts restarts — containers restarted into a pressure-bound system just fail again.
|
||||
Storage is checked after containers are healed — no false alerts from containers that
|
||||
were already being restarted. System watchdog runs last — reboot is always the last
|
||||
option, not the first.
|
||||
|
||||
**None of these scripts run standalone loops.** Each is a single-pass script called
|
||||
once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles
|
||||
startup grace, overlap protection, heartbeat, and sequencing.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
```
|
||||
Orchestrators/
|
||||
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
|
||||
──────────────────► docker_watchdog.sh (2nd)
|
||||
──────────────────► storage_watchdog.sh (3rd)
|
||||
──────────────────► system_watchdog.sh (4th — last resort)
|
||||
|
||||
Tools/
|
||||
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
|
||||
(operator utility — inspect + clear after fixing a crash-looping container)
|
||||
|
||||
Docker_Essentials/
|
||||
All container lifecycle scripts (daily restart, updates, network) — unaffected.
|
||||
docker_watchdog.sh coordinates with them via shared state, not direct calls.
|
||||
|
||||
unRAID_Essentials/
|
||||
Server-level scripts (WebGUI restart, inotify tuning, log hygiene) — unaffected.
|
||||
system_watchdog.sh runs in the same ecosystem but is independent of those scripts.
|
||||
```
|
||||
|
||||
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
|
||||
**watchdog_skip_list_manager.sh stays in Tools/** — it's an operator utility, not a watchdog.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | Called By |
|
||||
|--------|------|-----------|
|
||||
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute |
|
||||
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute |
|
||||
| `storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `watchdog_orchestrator.sh` — 3rd every minute |
|
||||
| `system_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
|
||||
|
||||
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
|
||||
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
Every minute — watchdog_orchestrator.sh fires:
|
||||
|
||||
Step 1 — resource_watchdog.sh
|
||||
│ RAM/load OK → pass through (no action)
|
||||
│ Level 1 (soft): throttle SABnzbd + qBit download speeds
|
||||
│ Level 2 (medium): further throttle + docker pause non-critical containers
|
||||
│ Level 3 (hard): docker stop optional services
|
||||
│ writes mem_shutdown_active=true → RW_STATE_FILE
|
||||
│ ↓
|
||||
Step 2 — docker_watchdog.sh
|
||||
│ reads RW_STATE_FILE — if mem_shutdown_active=true: skip all restarts
|
||||
│
|
||||
│ Tier 1 — explicit per-container checks (configured in host*.conf):
|
||||
│ memory hard limits → immediate restart (no strikes)
|
||||
│ CPU high sustained → 2-strike restart
|
||||
│ HTTP non-response → 2-strike restart
|
||||
│ required stopped → restart (dependency ordering respected)
|
||||
│
|
||||
│ Tier 2 — global scan of all running containers:
|
||||
│ unhealthy / OOM / crashloop / dead / non-zero exit → restart
|
||||
│ N restarts in window → skip list + critical notify → human required
|
||||
│
|
||||
│ (skip list management) → Tools/watchdog_skip_list_manager.sh
|
||||
│
|
||||
Step 3 — storage_watchdog.sh
|
||||
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
|
||||
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
|
||||
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
|
||||
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
|
||||
│
|
||||
Step 4 — system_watchdog.sh
|
||||
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
|
||||
Tier 1 CRITICAL → immediate reboot (no strikes)
|
||||
Tier 2 URGENT → reboot if OOM confirmed
|
||||
Tier 3 STANDARD → N consecutive failures → reboot
|
||||
Abort conditions → ZFS unhealthy / parity running / mover active
|
||||
Rate limit → max N reboots per window before switching to notify
|
||||
```
|
||||
|
||||
**State file coordination between scripts:**
|
||||
|
||||
| State File | Written By | Read By | Purpose |
|
||||
|-----------|-----------|---------|---------|
|
||||
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
|
||||
| `SYS_WATCHDOG_STATE_FILE` | `system_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
|
||||
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
|
||||
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
|
||||
| `STORAGE_WATCHDOG_STATE_FILE` | `storage_watchdog.sh` | itself | Growth + log strike counts |
|
||||
| `WATCHDOG_APPDATA_GROWTH_FILE` | `storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
|
||||
@@ -19,7 +19,7 @@
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Tier 1 — Strict Per-Container Monitoring
|
||||
# Applies only to containers explicitly configured in master_host*.conf.
|
||||
# Applies only to containers explicitly configured in host*.conf.
|
||||
#
|
||||
# Memory hard limits — immediate restart if container exceeds MB ceiling
|
||||
# Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of limit (no restart)
|
||||
@@ -145,7 +145,7 @@
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_WATCHDOG_CONTAINERS
|
||||
# Memory hard limits per container. Format: "ContainerName:LimitInMB"
|
||||
@@ -922,6 +922,7 @@ CYCLE_START=$(date +%s)
|
||||
|
||||
fi # WATCHDOG_SCAN_ALL
|
||||
|
||||
|
||||
# ── Send notifications ────────────────────────────────────────────────────────────────────
|
||||
flush_notify
|
||||
|
||||
@@ -71,7 +71,7 @@
|
||||
# RW_QBIT_ENABLED, RW_QBIT_DL_SOFT, RW_QBIT_DL_MEDIUM
|
||||
# RW_CRITICAL_CONTAINERS — never paused or stopped regardless of pressure
|
||||
#
|
||||
# master_host*.conf (aliased by detect_hosts())
|
||||
# host*.conf (aliased by detect_hosts())
|
||||
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure
|
||||
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure
|
||||
# HOST*_SABNZBD_URL, HOST*_SABNZBD_API_KEY
|
||||
@@ -118,7 +118,7 @@ if [[ "$EUID" -ne 0 ]]; then
|
||||
fi
|
||||
|
||||
if [[ "${RW_ENABLED:-true}" != "true" ]]; then
|
||||
log "Resource Manager disabled (RW_ENABLED=false)"
|
||||
echo "Resource Manager disabled (RW_ENABLED=false)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -138,6 +138,21 @@ touch "$RW_STATE_FILE" 2>/dev/null || {
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ── Exit Trap — restart containers stopped this run if script crashes ──────────────────────────
|
||||
declare -a _RW_TRAP_STOPPED=()
|
||||
|
||||
_rw_trap_restart_stopped() {
|
||||
[[ ${#_RW_TRAP_STOPPED[@]} -eq 0 ]] && return
|
||||
for c in "${_RW_TRAP_STOPPED[@]}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
if docker inspect "$c" >/dev/null 2>&1; then
|
||||
warn "Exit trap: restarting $c (stopped but state not persisted)"
|
||||
docker start "$c" >/dev/null 2>&1 || warn " Failed to restart $c"
|
||||
fi
|
||||
done
|
||||
}
|
||||
trap _rw_trap_restart_stopped EXIT
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ State Helpers ━━━
|
||||
# ==============================================================================================
|
||||
@@ -379,6 +394,7 @@ stop_containers() {
|
||||
if timeout "$DOCKER_TIMEOUT" docker stop "$container" >/dev/null 2>&1; then
|
||||
warn "Stopped $container (hard pressure)"
|
||||
actually_stopped+=("$container")
|
||||
_RW_TRAP_STOPPED+=("$container")
|
||||
else
|
||||
error "Failed to stop $container"
|
||||
fi
|
||||
@@ -468,7 +484,7 @@ apply_level_3() {
|
||||
# ==============================================================================================
|
||||
|
||||
restore_level_3() {
|
||||
log "Restoring from level 3 — starting stopped containers"
|
||||
echo "Restoring from level 3 — starting stopped containers"
|
||||
if [[ -n "$STOPPED_LIST" ]]; then
|
||||
start_containers "$STOPPED_LIST"
|
||||
STOPPED_LIST=""
|
||||
@@ -478,7 +494,7 @@ restore_level_3() {
|
||||
}
|
||||
|
||||
restore_level_2() {
|
||||
log "Restoring from level 2 — unpausing containers"
|
||||
echo "Restoring from level 2 — unpausing containers"
|
||||
if [[ -n "$PAUSED_LIST" ]]; then
|
||||
unpause_containers "$PAUSED_LIST"
|
||||
PAUSED_LIST=""
|
||||
@@ -486,7 +502,7 @@ restore_level_2() {
|
||||
}
|
||||
|
||||
restore_level_1() {
|
||||
log "Restoring from level 1 — removing downloader throttle"
|
||||
echo "Restoring from level 1 — removing downloader throttle"
|
||||
sabnzbd_set_speed "0"
|
||||
qbit_set_dl_limit 0
|
||||
}
|
||||
@@ -552,9 +568,9 @@ elif [[ "$TARGET_LEVEL" -lt "$CURRENT_LEVEL" ]]; then
|
||||
else
|
||||
# ── Steady state ────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$CURRENT_LEVEL" -gt 0 ]]; then
|
||||
log "Pressure holding at level $CURRENT_LEVEL — waiting for sustained recovery"
|
||||
echo "Pressure holding at level $CURRENT_LEVEL — waiting for sustained recovery"
|
||||
else
|
||||
log "System at normal pressure ✅"
|
||||
echo "System at normal pressure ✅"
|
||||
fi
|
||||
rm_state_set "rm_recover_cycles" 0
|
||||
fi
|
||||
@@ -564,5 +580,6 @@ fi
|
||||
# ==============================================================================================
|
||||
rm_state_set "rm_paused_containers" "$PAUSED_LIST"
|
||||
rm_state_set "rm_stopped_containers" "$STOPPED_LIST"
|
||||
trap - EXIT # state persisted — stopped containers recorded, trap no longer needed
|
||||
# Touch state file each run so docker_watchdog stale guard sees fresh mtime
|
||||
touch "$RW_STATE_FILE" 2>/dev/null
|
||||
Executable
+428
@@ -0,0 +1,428 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Storage Watchdog ===========================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Pool and storage health monitoring — catches runaway data growth before it
|
||||
# fills a pool. Runs as a single-pass script called by watchdog_orchestrator.sh
|
||||
# every cycle. Sits between docker_watchdog.sh (container health) and
|
||||
# system_watchdog.sh (last line of defense). Never reboots — detects, alerts,
|
||||
# and optionally remediates.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Appdata Size Monitoring
|
||||
# Two complementary checks run every cycle:
|
||||
#
|
||||
# Part 1 — Growth rate (zero-config catch-all):
|
||||
# Reads per-container dir totals via du, compares to previous cycle baseline.
|
||||
# Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused
|
||||
# *.log scan inside that container's dir. No per-container config required —
|
||||
# new containers are covered automatically. Baseline built on first cycle after
|
||||
# boot; growth detection active from cycle 2.
|
||||
#
|
||||
# Part 2 — Absolute log size:
|
||||
# Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB anywhere in
|
||||
# WATCHDOG_APPDATA_PATHS. Catches logs already large but no longer actively
|
||||
# growing. Independent strike counter per file.
|
||||
#
|
||||
# Strike System
|
||||
# Reuses the same strike pattern as CPU/HTTP checks in docker_watchdog.sh:
|
||||
#
|
||||
# Strike 1 — warn + notify: condition first detected this run
|
||||
# Strike 2 — warn + escalated notify: still present next cycle
|
||||
# Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
|
||||
# WATCHDOG_APPDATA_TRUNCATE_LOGS=true → truncate *.log in-place, clear strikes
|
||||
# WATCHDOG_APPDATA_TRUNCATE_LOGS=false → critical notify, hold strikes until resolved
|
||||
# Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
|
||||
#
|
||||
# Suppress Ceiling (WATCHDOG_APPDATA_SIZES)
|
||||
# Containers in HOST*_WATCHDOG_APPDATA_SIZES suppress growth warnings while
|
||||
# under their configured ceiling MB. Use ONLY when a container legitimately
|
||||
# holds large stable data and would otherwise false-alarm (e.g. Tdarr cache).
|
||||
# Zero-config growth detection covers everything else automatically.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Zero-config for new containers
|
||||
# Growth rate detection requires no per-container configuration. Add a game
|
||||
# server, spin up a new arr, install anything — it is monitored automatically
|
||||
# from the second cycle after it appears. The suppress ceiling in conf is the
|
||||
# exception, not the rule.
|
||||
#
|
||||
# Alert-only for data, truncate-only for logs
|
||||
# Non-log growth (databases, game saves, caches) is detected and alerted but
|
||||
# never touched. Only *.log / *.log.* files are candidates for truncation —
|
||||
# and only when WATCHDOG_APPDATA_TRUNCATE_LOGS=true. Truncation zeroes the
|
||||
# file in-place; the container keeps its open file handle, space is reclaimed
|
||||
# immediately. Never deletes.
|
||||
#
|
||||
# Strike before acting
|
||||
# One cycle of growth could be a legitimate library scan or game save burst.
|
||||
# Three consecutive cycles of growth is a runaway. The strike system separates
|
||||
# transient activity from sustained problems before any action fires.
|
||||
#
|
||||
# Silent when healthy
|
||||
# Produces no output when all checks pass. Loud only when something needs
|
||||
# attention.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# WATCHDOG_CHECK_APPDATA
|
||||
# Master toggle for all appdata checks (default: true)
|
||||
#
|
||||
# WATCHDOG_APPDATA_PATHS
|
||||
# Array of paths to scan (e.g. "/mnt/docker-unraid/appdata")
|
||||
#
|
||||
# WATCHDOG_APPDATA_GROWTH_GB
|
||||
# Per-cycle growth threshold in GB — flag containers growing more than this (default: 2)
|
||||
#
|
||||
# WATCHDOG_APPDATA_LOG_MAX_GB
|
||||
# Absolute *.log file size alert threshold in GB (default: 2)
|
||||
#
|
||||
# WATCHDOG_APPDATA_TRUNCATE_LOGS
|
||||
# Auto-truncate oversized *.log files on action cycle (default: false)
|
||||
#
|
||||
# WATCHDOG_APPDATA_STRIKE_LIMIT
|
||||
# Consecutive cycles before action fires (default: 3)
|
||||
#
|
||||
# WATCHDOG_APPDATA_GROWTH_FILE
|
||||
# Per-container size baseline — /tmp resets on reboot (correct: stale baseline
|
||||
# after reboot would give false growth readings on first cycle)
|
||||
#
|
||||
# STORAGE_WATCHDOG_STATE_FILE
|
||||
# Strike counts for this script — /tmp resets on reboot
|
||||
#
|
||||
# host*.conf (aliased by detect_hosts())
|
||||
#
|
||||
# HOST*_WATCHDOG_APPDATA_SIZES
|
||||
# Per-container growth suppress ceilings in MB. Suppress growth alerts while
|
||||
# a container's dir stays below this ceiling. Only needed when a container
|
||||
# legitimately has large stable data. Growth detection covers everything else.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# STORAGE_WATCHDOG_STATE_FILE — strike counts (/tmp — resets on reboot ✅)
|
||||
# WATCHDOG_APPDATA_GROWTH_FILE — per-container size baseline (/tmp — resets on reboot ✅)
|
||||
#
|
||||
# /tmp files reset on reboot — correct. Pre-reboot strikes and growth baselines are
|
||||
# meaningless after a reboot. Both rebuild cleanly from cycle 1.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# storage_watchdog.sh
|
||||
# Single-pass storage health check. Silent if all healthy.
|
||||
#
|
||||
# storage_watchdog.sh --dry-run
|
||||
# Run all checks without truncating anything. Shows what would be actioned.
|
||||
#
|
||||
# storage_watchdog.sh --status
|
||||
# Show configuration, active strikes, and growth baseline status. Then exit.
|
||||
#
|
||||
# storage_watchdog.sh --log
|
||||
# Verbose output — every container checked, every size comparison, every decision.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be truncated"
|
||||
|
||||
touch "$STORAGE_WATCHDOG_STATE_FILE" 2>/dev/null
|
||||
touch "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STORAGE WATCHDOG STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Appdata check: ${WATCHDOG_CHECK_APPDATA:-true}"
|
||||
echo "$ICON_GEAR Paths: ${WATCHDOG_APPDATA_PATHS[*]:-none}"
|
||||
echo "$ICON_GEAR Growth thresh: ${WATCHDOG_APPDATA_GROWTH_GB:-2}GB/cycle"
|
||||
echo "$ICON_GEAR Log max: ${WATCHDOG_APPDATA_LOG_MAX_GB:-2}GB"
|
||||
echo "$ICON_GEAR Truncate logs: ${WATCHDOG_APPDATA_TRUNCATE_LOGS:-false}"
|
||||
echo "$ICON_GEAR Strike limit: ${WATCHDOG_APPDATA_STRIKE_LIMIT:-3}"
|
||||
echo ""
|
||||
echo "── Active Strikes ──"
|
||||
if [[ -s "$STORAGE_WATCHDOG_STATE_FILE" ]]; then
|
||||
while IFS=':' read -r _sk _sv; do
|
||||
_sv_clean="${_sv//[^0-9]/}"
|
||||
[[ "${_sv_clean:-0}" -gt 0 ]] && echo " $_sk → $_sv_clean strikes"
|
||||
done < "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
else
|
||||
echo " none"
|
||||
fi
|
||||
echo ""
|
||||
echo "── Growth Baseline ──"
|
||||
if [[ -s "$WATCHDOG_APPDATA_GROWTH_FILE" ]]; then
|
||||
_entries=$(wc -l < "$WATCHDOG_APPDATA_GROWTH_FILE")
|
||||
_age=$(( $(date +%s) - $(stat -c %Y "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null || echo 0) ))
|
||||
echo " Entries: $_entries containers Age: $(( _age / 60 ))m ago"
|
||||
else
|
||||
echo " No baseline yet (builds on first cycle after boot)"
|
||||
fi
|
||||
echo ""
|
||||
echo "── Suppress Ceilings (this host) ──"
|
||||
if [[ ${#WATCHDOG_APPDATA_SIZES[@]} -gt 0 ]]; then
|
||||
for _c in "${!WATCHDOG_APPDATA_SIZES[@]}"; do
|
||||
_ceil_gb=$(awk "BEGIN {printf \"%.0f\", ${WATCHDOG_APPDATA_SIZES[$_c]} / 1024}")
|
||||
echo " $_c → ${_ceil_gb}GB"
|
||||
done
|
||||
else
|
||||
echo " none configured"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Appdata Size Monitoring ━━━
|
||||
# ==============================================================================================
|
||||
[[ "$WATCHDOG_CHECK_APPDATA" != "true" ]] && exit 0
|
||||
|
||||
log "$ICON_GEAR Storage watchdog — $MY_ID — $(date '+%H:%M:%S')"
|
||||
|
||||
WARNINGS=0
|
||||
|
||||
_STRIKE_LIMIT=${WATCHDOG_APPDATA_STRIKE_LIMIT:-3}
|
||||
_GROWTH_MB=$(( ${WATCHDOG_APPDATA_GROWTH_GB:-2} * 1024 ))
|
||||
_LOG_KB=$(( ${WATCHDOG_APPDATA_LOG_MAX_GB:-2} * 1024 * 1024 ))
|
||||
|
||||
# Tracks files handled by Part 1 to prevent duplicate alerts in Part 2
|
||||
declare -A _HANDLED=()
|
||||
|
||||
for _appdata_path in "${WATCHDOG_APPDATA_PATHS[@]:-}"; do
|
||||
[[ -z "$_appdata_path" || ! -d "$_appdata_path" ]] && continue
|
||||
|
||||
# ── Part 1: Growth rate scan ──────────────────────────────────────────────────────────────
|
||||
declare -A _PREV=()
|
||||
if [[ -f "$WATCHDOG_APPDATA_GROWTH_FILE" ]]; then
|
||||
while IFS='|' read -r _cn _cs _; do
|
||||
[[ -n "$_cn" ]] && _PREV["$_cn"]="$_cs"
|
||||
done < "$WATCHDOG_APPDATA_GROWTH_FILE"
|
||||
fi
|
||||
|
||||
_growth_tmp=$(mktemp 2>/dev/null) || _growth_tmp=""
|
||||
_now=$(date +%s)
|
||||
|
||||
while IFS= read -r _du_line; do
|
||||
_curr_mb=$(echo "$_du_line" | awk '{print $1}')
|
||||
_cdir=$(echo "$_du_line" | awk '{print $2}')
|
||||
_cname=$(basename "$_cdir")
|
||||
[[ -z "$_cname" || "$_cname" == "*" ]] && continue
|
||||
|
||||
[[ -n "$_growth_tmp" ]] && echo "${_cname}|${_curr_mb}|${_now}" >> "$_growth_tmp"
|
||||
|
||||
_prev_mb="${_PREV[$_cname]:-}"
|
||||
[[ -z "$_prev_mb" ]] && continue # First run after boot — building baseline
|
||||
|
||||
_growth_mb=$(( _curr_mb - _prev_mb ))
|
||||
_safe=$(echo "$_cname" | tr -cd '[:alnum:]_')
|
||||
|
||||
# Condition resolved — growth stopped, clear strikes
|
||||
if [[ "$_growth_mb" -le 0 ]]; then
|
||||
_existing=$(get_strikes "appdata_growth_${_safe}" "$STORAGE_WATCHDOG_STATE_FILE")
|
||||
_existing="${_existing//[^0-9]/}"
|
||||
[[ "${_existing:-0}" -gt 0 ]] && \
|
||||
set_strikes "appdata_growth_${_safe}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Check suppress ceiling
|
||||
_ceiling="${WATCHDOG_APPDATA_SIZES[$_cname]:-}"
|
||||
if [[ -n "$_ceiling" && "$_curr_mb" -lt "$_ceiling" ]]; then
|
||||
log "$_cname — growth suppressed (${_curr_mb}MB < ${_ceiling}MB ceiling)"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Growth exceeds threshold — strike logic
|
||||
if [[ "$_growth_mb" -ge "$_GROWTH_MB" ]]; then
|
||||
_growth_gb=$(awk "BEGIN {printf \"%.1f\", $_growth_mb / 1024}")
|
||||
_curr_gb=$(awk "BEGIN {printf \"%.1f\", $_curr_mb / 1024}")
|
||||
|
||||
_strikes=$(get_strikes "appdata_growth_${_safe}" "$STORAGE_WATCHDOG_STATE_FILE")
|
||||
_strikes="${_strikes//[^0-9]/}"; _strikes="${_strikes:-0}"
|
||||
|
||||
[[ "$_strikes" -lt "$_STRIKE_LIMIT" ]] && {
|
||||
_strikes=$(( _strikes + 1 ))
|
||||
set_strikes "appdata_growth_${_safe}" "$_strikes" "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
}
|
||||
|
||||
warn "$_cname — grew ${_growth_gb}GB this cycle (total: ${_curr_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]"
|
||||
(( WARNINGS++ ))
|
||||
|
||||
# Focused *.log scan inside the growing container's dir
|
||||
_found_logs=()
|
||||
while IFS= read -r _lf; do
|
||||
[[ -n "$_lf" ]] && _found_logs+=("$_lf")
|
||||
done < <(find "$_cdir" -maxdepth 3 -type f \
|
||||
\( -name "*.log" -o -name "*.log.*" \) \
|
||||
-size +${_LOG_KB}k -printf "%s %p\n" 2>/dev/null)
|
||||
|
||||
_log_summary=""
|
||||
[[ ${#_found_logs[@]} -gt 0 ]] && _log_summary=$(printf '%s\n' "${_found_logs[@]}" | \
|
||||
awk '{printf "%.1fGB %s | ", $1/1073741824, $2}' | head -c 200)
|
||||
|
||||
if [[ "$_strikes" -lt "$_STRIKE_LIMIT" ]]; then
|
||||
if [[ -n "$_log_summary" ]]; then
|
||||
notify "$_cname grew ${_growth_gb}GB on $(hostname) [strike ${_strikes}/${_STRIKE_LIMIT}] — logs: ${_log_summary}" \
|
||||
"Storage Watchdog" "warning"
|
||||
else
|
||||
notify "$_cname grew ${_growth_gb}GB on $(hostname) [strike ${_strikes}/${_STRIKE_LIMIT}] — data growth, no log files" \
|
||||
"Storage Watchdog" "warning"
|
||||
fi
|
||||
else
|
||||
# Action cycle
|
||||
error "$_cname — growth strike limit reached (${_STRIKE_LIMIT} consecutive cycles, ${_growth_gb}GB this cycle)"
|
||||
if [[ "$WATCHDOG_APPDATA_TRUNCATE_LOGS" == "true" && ${#_found_logs[@]} -gt 0 ]]; then
|
||||
for _lf_entry in "${_found_logs[@]}"; do
|
||||
_lf_path=$(echo "$_lf_entry" | cut -d' ' -f2-)
|
||||
_lf_gb=$(echo "$_lf_entry" | awk '{printf "%.1f", $1/1073741824}')
|
||||
_HANDLED["$_lf_path"]=1
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would truncate $_lf_path (${_lf_gb}GB)"
|
||||
else
|
||||
if truncate -s 0 "$_lf_path" 2>/dev/null; then
|
||||
success "Truncated runaway log: $_lf_path (was ${_lf_gb}GB)"
|
||||
notify "Truncated runaway log on $(hostname): $_lf_path (was ${_lf_gb}GB)" \
|
||||
"Storage Watchdog" "warning"
|
||||
else
|
||||
warn "Failed to truncate $_lf_path"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
set_strikes "appdata_growth_${_safe}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
else
|
||||
if [[ -n "$_log_summary" ]]; then
|
||||
notify "$_cname appdata runaway on $(hostname) — ${_growth_gb}GB growth for ${_STRIKE_LIMIT} cycles — logs: ${_log_summary}" \
|
||||
"Storage Watchdog" "critical"
|
||||
else
|
||||
notify "$_cname appdata runaway on $(hostname) — ${_growth_gb}GB growth for ${_STRIKE_LIMIT} cycles — data growth, manual investigation needed" \
|
||||
"Storage Watchdog" "critical"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Mark found logs as handled to suppress Part 2 duplicates this cycle
|
||||
for _lf_entry in "${_found_logs[@]}"; do
|
||||
_HANDLED["$(echo "$_lf_entry" | cut -d' ' -f2-)"]=1
|
||||
done
|
||||
fi
|
||||
done < <(du -sm "$_appdata_path"/*/ 2>/dev/null)
|
||||
|
||||
# Atomically update growth baseline
|
||||
[[ -n "$_growth_tmp" ]] && mv "$_growth_tmp" "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null
|
||||
|
||||
# ── Part 2: Absolute log size scan ────────────────────────────────────────────────────────
|
||||
# Catches *.log files already large but no longer actively growing this cycle.
|
||||
# Same strike logic. Skips files already handled by Part 1 above.
|
||||
declare -A _LOG_SEEN=()
|
||||
|
||||
while IFS= read -r _hit; do
|
||||
[[ -z "$_hit" ]] && continue
|
||||
_fpath=$(echo "$_hit" | cut -d' ' -f2-)
|
||||
[[ -n "${_HANDLED[$_fpath]:-}" ]] && continue
|
||||
|
||||
_fsize_bytes=$(echo "$_hit" | awk '{print $1}')
|
||||
_fsize_gb=$(awk "BEGIN {printf \"%.1f\", $_fsize_bytes / 1073741824}")
|
||||
_safe_fkey=$(echo "$_fpath" | tr -cd '[:alnum:]_' | cut -c1-120)
|
||||
_LOG_SEEN["appdata_log_${_safe_fkey}"]=1
|
||||
|
||||
_strikes=$(get_strikes "appdata_log_${_safe_fkey}" "$STORAGE_WATCHDOG_STATE_FILE")
|
||||
_strikes="${_strikes//[^0-9]/}"; _strikes="${_strikes:-0}"
|
||||
|
||||
[[ "$_strikes" -lt "$_STRIKE_LIMIT" ]] && {
|
||||
_strikes=$(( _strikes + 1 ))
|
||||
set_strikes "appdata_log_${_safe_fkey}" "$_strikes" "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
}
|
||||
|
||||
warn "Oversized log: $_fpath (${_fsize_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]"
|
||||
(( WARNINGS++ ))
|
||||
|
||||
if [[ "$_strikes" -lt "$_STRIKE_LIMIT" ]]; then
|
||||
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]" \
|
||||
"Storage Watchdog" "warning"
|
||||
else
|
||||
error "Oversized log persists for ${_STRIKE_LIMIT} cycles: $_fpath (${_fsize_gb}GB)"
|
||||
if [[ "$WATCHDOG_APPDATA_TRUNCATE_LOGS" == "true" ]]; then
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would truncate $_fpath (${_fsize_gb}GB)"
|
||||
else
|
||||
if truncate -s 0 "$_fpath" 2>/dev/null; then
|
||||
success "Truncated oversized log: $_fpath (was ${_fsize_gb}GB)"
|
||||
notify "Truncated oversized log on $(hostname): $_fpath (was ${_fsize_gb}GB)" \
|
||||
"Storage Watchdog" "warning"
|
||||
set_strikes "appdata_log_${_safe_fkey}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
unset "_LOG_SEEN[appdata_log_${_safe_fkey}]"
|
||||
else
|
||||
warn "Failed to truncate $_fpath"
|
||||
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) — ${_STRIKE_LIMIT} cycles, truncate failed" \
|
||||
"Storage Watchdog" "critical"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) — ${_STRIKE_LIMIT} cycles, manual intervention needed" \
|
||||
"Storage Watchdog" "critical"
|
||||
fi
|
||||
fi
|
||||
done < <(find "$_appdata_path" -maxdepth 4 -type f \
|
||||
\( -name "*.log" -o -name "*.log.*" \) \
|
||||
-size +${_LOG_KB}k -printf "%s %p\n" 2>/dev/null)
|
||||
|
||||
# Auto-clear strikes for log files no longer oversized this cycle
|
||||
while IFS=':' read -r _sk _sv; do
|
||||
[[ "$_sk" != appdata_log_* ]] && continue
|
||||
_sv_clean="${_sv//[^0-9]/}"
|
||||
[[ "${_sv_clean:-0}" -eq 0 ]] && continue
|
||||
[[ -n "${_LOG_SEEN[$_sk]:-}" ]] && continue
|
||||
set_strikes "$_sk" 0 "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
done < "$STORAGE_WATCHDOG_STATE_FILE" 2>/dev/null
|
||||
|
||||
unset _LOG_SEEN
|
||||
|
||||
done
|
||||
|
||||
unset _PREV _HANDLED
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$WARNINGS" -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Storage Watchdog — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_WATCHDOG Warnings: $WARNINGS"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
log "Storage healthy ✅ ($(date '+%H:%M:%S'))"
|
||||
fi
|
||||
@@ -350,6 +350,17 @@ run_strike_check() {
|
||||
return 1
|
||||
}
|
||||
|
||||
# ── Exit Trap — restart containers stopped before an aborted reboot ───────────────────────────
|
||||
_SYS_REBOOT_STOPPED=()
|
||||
_trap_sys_reboot_restart() {
|
||||
[[ ${#_SYS_REBOOT_STOPPED[@]} -eq 0 ]] && return
|
||||
warn "Exit trap: restarting containers stopped before aborted reboot"
|
||||
for c in "${_SYS_REBOOT_STOPPED[@]}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
docker inspect "$c" >/dev/null 2>&1 && docker start "$c" >/dev/null 2>&1 || true
|
||||
done
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── DO REBOOT ─────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
@@ -424,6 +435,8 @@ do_reboot() {
|
||||
|
||||
warn "Stopping Docker containers..."
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
mapfile -t _SYS_REBOOT_STOPPED < <(docker ps --format '{{.Names}}' 2>/dev/null)
|
||||
trap _trap_sys_reboot_restart EXIT
|
||||
timeout 60 docker ps -q 2>/dev/null | xargs -r docker stop >/dev/null 2>&1
|
||||
fi
|
||||
|
||||
@@ -433,6 +446,7 @@ do_reboot() {
|
||||
warn "Syncing disks..."
|
||||
sync
|
||||
|
||||
trap - EXIT # committed to reboot — containers should stay down
|
||||
sleep 5
|
||||
/sbin/reboot
|
||||
}
|
||||
@@ -784,7 +798,7 @@ echo "━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
do_reboot "standard" "${TRIGGERS[@]}"
|
||||
exit 0
|
||||
else
|
||||
log "System healthy ✅ ($(date '+%H:%M:%S'))"
|
||||
echo "System healthy ✅ ($(date '+%H:%M:%S'))"
|
||||
fi
|
||||
|
||||
# Keep state file mtime fresh — docker_watchdog stale guard checks this
|
||||
@@ -55,11 +55,11 @@
|
||||
# check_local_disk_temps() — pre-rsync temp check with exit codes 0/1/2
|
||||
# stop/start local containers added alongside existing remote variants
|
||||
#
|
||||
# v3.1 Three-file config split — master.conf + master_host1.conf + master_host2.conf
|
||||
# load_config.sh introduced — auto-discovers all master_host*.conf files
|
||||
# v3.1 Three-file config split — master.conf + host1.conf + host2.conf
|
||||
# load_config.sh introduced — auto-discovers all host*.conf files
|
||||
# detect_hosts() rewritten — sets MY_ID/REMOTE_ID and aliases all HOST* vars
|
||||
# All scripts now source load_config.sh instead of conf files directly
|
||||
# Adding a new server = add master_host*.conf, zero script changes required
|
||||
# Adding a new server = add host*.conf, zero script changes required
|
||||
#
|
||||
# v3.2 check_remote_disks() rewritten — three-tier detection:
|
||||
# Tier 1: array disk paths (/mnt/disk*/sharename)
|
||||
@@ -78,10 +78,6 @@
|
||||
# Emby immediately removes ghost entries — no user-facing file-not-found errors
|
||||
# Called by lidarr/sonarr/radarr_cleanup.sh when files are deleted
|
||||
#
|
||||
# v3.4 Silent-by-default output model
|
||||
# info() and success() now gated by SILENT_MODE — only warn/error always visible
|
||||
# Exception: monitor scripts designed to produce output stay verbose
|
||||
# Reduces notification spam — ecosystem only speaks when something is wrong
|
||||
#
|
||||
# Three new safety functions added:
|
||||
# check_unraid_version_parity() — refuses remote ops on version mismatch
|
||||
@@ -190,27 +186,16 @@ ICON_SUCCESS="✅"
|
||||
# ==============================================================================================
|
||||
# Standardised output functions used across all scripts.
|
||||
#
|
||||
# Silent-by-default model:
|
||||
# SILENT_MODE=true (default) — only warn() and error() produce output
|
||||
# SILENT_MODE=false — all functions produce output
|
||||
# --log flag — enables ENABLE_LOGGING (detailed [LOG] lines)
|
||||
#
|
||||
# Rules:
|
||||
# Two-tier model:
|
||||
# echo — always visible — summaries, status conclusions, section headers
|
||||
# warn() — always visible — state transitions, warnings, important events
|
||||
# error() — always visible — something broke
|
||||
# warn() — always visible — something needs attention
|
||||
# info() — silent by default — operational detail, visible when SILENT_MODE=false
|
||||
# success() — silent by default — confirmation, visible when SILENT_MODE=false
|
||||
# log() — debug detail — only when ENABLE_LOGGING=true
|
||||
#
|
||||
# Exception — monitor scripts are designed to produce output and set SILENT_MODE=false
|
||||
# at the top of the script. All other scripts use the silent default.
|
||||
# log() — only with --log flag — per-item detail, internal checks
|
||||
#
|
||||
# All output goes to stdout — callers can redirect as needed.
|
||||
|
||||
info() { [[ "${SILENT_MODE:-true}" == false ]] && echo "$ICON_INFO [INFO] $*"; return 0; }
|
||||
warn() { echo "$ICON_WARN [WARN] $*"; }
|
||||
error() { echo "$ICON_ERROR [ERROR] $*"; }
|
||||
success() { [[ "${SILENT_MODE:-true}" == false ]] && echo "$ICON_SUCCESS [OK] $*"; return 0; }
|
||||
|
||||
log() {
|
||||
[[ "${ENABLE_LOGGING:-false}" == true ]] && echo "[LOG] $*"
|
||||
@@ -223,7 +208,7 @@ log() {
|
||||
# Sends a notification via unRAID native system and/or Discord webhook.
|
||||
# Both channels are optional and independently controlled:
|
||||
# NOTIFY_UNRAID — shared toggle in master.conf
|
||||
# MY_DISCORD_WEBHOOK — per-host in master_host*.conf, set by detect_hosts()
|
||||
# MY_DISCORD_WEBHOOK — per-host in host*.conf, set by detect_hosts()
|
||||
#
|
||||
# Severity levels: normal, warning, alert
|
||||
# Usage: notify "message" "subject" "severity"
|
||||
@@ -254,7 +239,7 @@ notify() {
|
||||
-d "$payload" "$MY_DISCORD_WEBHOOK" >/dev/null 2>&1; then
|
||||
log "$ICON_NOTIFY Discord notification sent"
|
||||
else
|
||||
warn "Discord notification failed — check HOST*_DISCORD_WEBHOOK in master_host*.conf"
|
||||
warn "Discord notification failed — check HOST*_DISCORD_WEBHOOK in host*.conf"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
@@ -363,7 +348,7 @@ validate_int() {
|
||||
# ── HOST DETECTION ────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Determines which server is local and which is remote by comparing hostname against
|
||||
# all HOST* values discovered from master_host*.conf files.
|
||||
# all HOST* values discovered from host*.conf files.
|
||||
#
|
||||
# Sets:
|
||||
# MY_ID — "HOST1" or "HOST2" (the role key, not the hostname)
|
||||
@@ -384,6 +369,7 @@ validate_int() {
|
||||
# WATCHDOG_REQUIRED_CONTAINERS ← HOST*_WATCHDOG_REQUIRED_CONTAINERS
|
||||
# WATCHDOG_SCAN_IGNORE ← HOST*_WATCHDOG_SCAN_IGNORE
|
||||
# WATCHDOG_DEPENDENCIES ← HOST*_WATCHDOG_DEPENDENCIES (associative)
|
||||
# WATCHDOG_APPDATA_SIZES ← HOST*_WATCHDOG_APPDATA_SIZES (associative)
|
||||
# NETWORK_CONNECT_CONTAINERS ← HOST*_NETWORK_CONNECT_CONTAINERS
|
||||
# NETWORK_CONNECT_NETWORKS ← HOST*_NETWORK_CONNECT_NETWORKS
|
||||
# MEDIA_PERMISSION_SHARES ← HOST*_MEDIA_PERMISSION_SHARES
|
||||
@@ -423,7 +409,7 @@ detect_hosts() {
|
||||
local host_var host_val
|
||||
|
||||
# Scan all HOST* vars that hold a hostname value
|
||||
# HOST1, HOST2, HOST3 etc. — all defined in master_host*.conf files
|
||||
# HOST1, HOST2, HOST3 etc. — all defined in host*.conf files
|
||||
for host_var in HOST1 HOST2 HOST3 HOST4 HOST5 HOST6 HOST7 HOST8; do
|
||||
host_val="${!host_var:-}"
|
||||
[[ -z "$host_val" ]] && continue
|
||||
@@ -435,7 +421,7 @@ detect_hosts() {
|
||||
|
||||
if [[ -z "$MY_ID" ]]; then
|
||||
error "Unknown host: $local_hostname"
|
||||
error "Hostname must match a HOST* value in master_host*.conf"
|
||||
error "Hostname must match a HOST* value in host*.conf"
|
||||
error "Available: $(for h in HOST1 HOST2 HOST3 HOST4; do
|
||||
[[ -n "${!h:-}" ]] && echo -n "${!h} "; done)"
|
||||
exit 1
|
||||
@@ -459,7 +445,7 @@ detect_hosts() {
|
||||
local ssh_key_var="${MY_ID}_SSH_KEY"
|
||||
SSH_KEY="${!ssh_key_var:-}"
|
||||
if [[ -z "$SSH_KEY" ]]; then
|
||||
error "Missing SSH key: ${ssh_key_var} not set in master_host*.conf"
|
||||
error "Missing SSH key: ${ssh_key_var} not set in host*.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -467,6 +453,11 @@ detect_hosts() {
|
||||
MY_DISCORD_WEBHOOK_VAR="${MY_ID}_DISCORD_WEBHOOK"
|
||||
MY_DISCORD_WEBHOOK="${!MY_DISCORD_WEBHOOK_VAR:-}"
|
||||
|
||||
OWNER_NAME_VAR="${MY_ID}_OWNER"
|
||||
OWNER_NAME="${!OWNER_NAME_VAR:-}"
|
||||
OWNER_EMAIL_VAR="${MY_ID}_OWNER_EMAIL"
|
||||
OWNER_EMAIL="${!OWNER_EMAIL_VAR:-}"
|
||||
|
||||
EMBY_CONTAINER_VAR="${MY_ID}_EMBY_CONTAINER"
|
||||
EMBY_CONTAINER="${!EMBY_CONTAINER_VAR:-}"
|
||||
EMBY_URL_VAR="${MY_ID}_EMBY_URL"
|
||||
@@ -598,9 +589,10 @@ detect_hosts() {
|
||||
_alias_assoc "WATCHDOG_CONTAINERS"
|
||||
_alias_assoc "WATCHDOG_CONTAINER_URLS"
|
||||
_alias_assoc "WATCHDOG_DEPENDENCIES"
|
||||
_alias_assoc "WATCHDOG_APPDATA_SIZES"
|
||||
|
||||
# ── Output ────────────────────────────────────────────────────────────────
|
||||
info "$ICON_HOST Host: $LOCAL_SERVER_NAME → $REMOTE_SERVER_NAME"
|
||||
log "$ICON_HOST Host: $LOCAL_SERVER_NAME → $REMOTE_SERVER_NAME"
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -1779,7 +1771,7 @@ show_status() {
|
||||
echo "Remote ID: ${REMOTE_ID:-not set}"
|
||||
echo "unRAID ver: $local_ver"
|
||||
echo "Profile: ${PROFILE_NAME:-n/a}"
|
||||
echo "Silent mode: ${SILENT_MODE:-true}"
|
||||
|
||||
echo "DryRun: ${DRY_RUN:-false}"
|
||||
echo "Logging: ${ENABLE_LOGGING:-false}"
|
||||
echo "SSH Key: ${SSH_KEY:-not set}"
|
||||
|
||||
+13
-16
@@ -8,19 +8,19 @@
|
||||
# ── WHAT THIS SCRIPT DOES ─────────────────────────────────────────────────────────────────────
|
||||
# 1. Detects which server it's running on via detect_hosts() (MY_ID)
|
||||
# 2. Configures sparse checkout to exclude other servers' credential files
|
||||
# Each server only pulls its own master_host*.conf — never sees peer credentials
|
||||
# Each server only pulls its own host*.conf — never sees peer credentials
|
||||
# 3. Pulls or clones latest scripts from Gitea
|
||||
# 4. Sets executable permissions on all .sh files
|
||||
#
|
||||
# ── SPARSE CHECKOUT ───────────────────────────────────────────────────────────────────────────
|
||||
# Sparse checkout ensures each server only receives its own host conf:
|
||||
# HOST1 pulls: master.conf + master_host1.conf + all scripts
|
||||
# HOST1 skips: master_host2.conf, master_host3.conf etc.
|
||||
# HOST2 pulls: master.conf + master_host2.conf + all scripts
|
||||
# HOST2 skips: master_host1.conf, master_host3.conf etc.
|
||||
# HOST1 pulls: master.conf + host1.conf + all scripts
|
||||
# HOST1 skips: host2.conf, host3.conf etc.
|
||||
# HOST2 pulls: master.conf + host2.conf + all scripts
|
||||
# HOST2 skips: host1.conf, host3.conf etc.
|
||||
#
|
||||
# Adding a new server:
|
||||
# Create master_host3.conf in the repo
|
||||
# Create host3.conf in the repo
|
||||
# All existing servers automatically exclude it on next pull
|
||||
# New server gets only its own conf ✅
|
||||
#
|
||||
@@ -55,9 +55,6 @@ parse_args "$@"
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
@@ -128,7 +125,7 @@ fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Sparse Checkout Configuration ━━━
|
||||
# ==============================================================================================
|
||||
# Build the list of master_host*.conf files that belong to OTHER servers.
|
||||
# Build the list of host*.conf files that belong to OTHER servers.
|
||||
# This server pulls everything EXCEPT those files.
|
||||
# MY_ID is set by detect_hosts() — e.g. "HOST1"
|
||||
|
||||
@@ -140,7 +137,7 @@ configure_sparse_checkout() {
|
||||
# Enable sparse checkout
|
||||
git -C "$repo_dir" config core.sparseCheckout true 2>/dev/null
|
||||
|
||||
# Build exclusion list — all master_host*.conf files except MY_ID's
|
||||
# Build exclusion list — all host*.conf files except MY_ID's
|
||||
local sparse_file="$repo_dir/.git/info/sparse-checkout"
|
||||
mkdir -p "$(dirname "$sparse_file")"
|
||||
|
||||
@@ -148,9 +145,9 @@ configure_sparse_checkout() {
|
||||
echo "/*" > "$sparse_file"
|
||||
|
||||
# Exclude each other server's conf file
|
||||
# Find all master_host*.conf files present in the repo
|
||||
# Find all host*.conf files present in the repo
|
||||
local excluded=0
|
||||
for conf_file in "$repo_dir"/master_host*.conf; do
|
||||
for conf_file in "$repo_dir"/host*.conf; do
|
||||
[[ -f "$conf_file" ]] || continue
|
||||
local conf_name
|
||||
conf_name=$(basename "$conf_file")
|
||||
@@ -196,7 +193,7 @@ SYNC_SUCCESS=false
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would sync $REPO_SSH → $TARGET_DIR"
|
||||
warn "DRY RUN — would configure sparse checkout for $MY_ID"
|
||||
warn "DRY RUN — would exclude peer master_host*.conf files"
|
||||
warn "DRY RUN — would exclude peer host*.conf files"
|
||||
SYNC_SUCCESS=true
|
||||
else
|
||||
mkdir -p "$TARGET_DIR"
|
||||
@@ -236,7 +233,7 @@ else
|
||||
echo " Clone successful"
|
||||
|
||||
# Configure sparse checkout after clone
|
||||
# Now all master_host*.conf files are present — can detect exclusions
|
||||
# Now all host*.conf files are present — can detect exclusions
|
||||
configure_sparse_checkout "$TARGET_DIR"
|
||||
|
||||
# Apply sparse checkout — removes excluded files from working tree
|
||||
@@ -270,7 +267,7 @@ echo "━━━━━ $ICON_SUMMARY GIT SYNC SUMMARY ━━━━━"
|
||||
echo "$ICON_NET Repo: $REPO_SSH"
|
||||
echo "$ICON_GEAR Target: $TARGET_DIR"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_LOCK Excluded: peer master_host*.conf files"
|
||||
echo "$ICON_LOCK Excluded: peer host*.conf files"
|
||||
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
echo "$ICON_WARN Status: DRY RUN — no changes made"
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
# HOST2 never sees HOST1 credentials — clean separation at the file level.
|
||||
#
|
||||
# DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf.
|
||||
# DO NOT put HOST2 variables here — they belong in master_host2.conf.
|
||||
# DO NOT put HOST2 variables here — they belong in host2.conf.
|
||||
#
|
||||
# ── INDEX ─────────────────────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
@@ -59,6 +59,12 @@
|
||||
# RADARR URL, API key, path map
|
||||
# ARR RECOVERY per-arr recovery toggles
|
||||
#
|
||||
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
|
||||
# SYSTEM WATCHDOG per-host check toggles and NIC configuration
|
||||
#
|
||||
# ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
|
||||
# RESOURCE MANAGER containers paused/stopped under memory pressure
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -70,6 +76,8 @@
|
||||
# SSH key used for all server-to-server operations — rsync, failover container commands.
|
||||
# Must be in /root/.ssh/ and authorised in HOST2's /root/.ssh/authorized_keys.
|
||||
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
|
||||
HOST1_OWNER="gmer4lfe"
|
||||
HOST1_OWNER_EMAIL="gmer4lfe@gmail.com"
|
||||
|
||||
# ━━━ Emby ━━━
|
||||
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
|
||||
@@ -319,6 +327,17 @@
|
||||
["NextCloud"]="Postgres-NextCloud"
|
||||
)
|
||||
|
||||
# Per-container appdata growth suppress ceilings in MB.
|
||||
# ONLY needed in specific cases — growth rate detection covers all containers automatically.
|
||||
# Use this when a container legitimately has large stable data and you want to guarantee
|
||||
# it never triggers a false-positive growth alert. Growth warnings are suppressed while the
|
||||
# container's dir stays below this ceiling; above it, warnings resume as normal.
|
||||
# 50GB=51200 25GB=25600 20GB=20480 15GB=15360 10GB=10240 5GB=5120
|
||||
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||||
["Tdarr"]="25600" # 25GB — transcode cache grows legitimately during active jobs
|
||||
["7dtd"]="20480" # 20GB — game server world data, expected to be large
|
||||
)
|
||||
|
||||
# ━━━ Docker Network Connect ━━━
|
||||
# Containers connected to custom networks at array start by docker_network_connect.sh.
|
||||
# Networks created if they don't exist — idempotent, safe to re-run.
|
||||
@@ -354,7 +373,7 @@
|
||||
# ━━━ Fallback Tiers — HOST1 Runs for HOST2 ━━━
|
||||
# Containers HOST1 starts when HOST2 goes down.
|
||||
# Tier 1 is always immediate — vital services cannot wait.
|
||||
# Higher tiers activate after HOST2_TIER*_DELAY minutes (set in master_host2.conf).
|
||||
# Higher tiers activate after HOST2_TIER*_DELAY minutes (set in host2.conf).
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
|
||||
"Gmer4Lfe.us"
|
||||
"VaultWarden-Jayred365"
|
||||
@@ -494,13 +513,13 @@
|
||||
# ==============================================================================================
|
||||
|
||||
# Ramdisk size ceiling — tmpfs only uses RAM actually needed, not the full size upfront.
|
||||
# Real-world: 9 streams peaked at ~5.5GB — 8G gives comfortable headroom on 128GB RAM.
|
||||
HOST1_RAMDISK_SIZE="8G"
|
||||
# Real-world: 9 streams peaked at ~5.5GB — 10G gives generous headroom on 128GB RAM.
|
||||
HOST1_RAMDISK_SIZE="10G"
|
||||
|
||||
# Usage thresholds — coupled to HOST1_RAMDISK_SIZE, adjust all three together if size changes.
|
||||
# Hysteresis gap (6.8 - 5.5 = 1.3GB) prevents flip-flop between ramdisk and SSD.
|
||||
HOST1_RAMDISK_WARN_GB=6.8 # flip to SSD when ramdisk usage reaches this
|
||||
HOST1_RAMDISK_LOW_GB=5.5 # flip back to ramdisk when usage drops to this
|
||||
# Hysteresis gap (8.5 - 7 = 1.5GB) prevents flip-flop between ramdisk and SSD.
|
||||
HOST1_RAMDISK_WARN_GB=8.5 # flip to SSD when ramdisk usage reaches this
|
||||
HOST1_RAMDISK_LOW_GB=7 # flip back to ramdisk when usage drops to this
|
||||
|
||||
# SSD fallback path — where transcodes land when ramdisk exceeds HOST1_RAMDISK_WARN_GB.
|
||||
# Must be on cache pool — array disks too slow for active transcode writes.
|
||||
@@ -10,7 +10,7 @@
|
||||
# HOST1 never sees HOST2 credentials — clean separation at the file level.
|
||||
#
|
||||
# DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf.
|
||||
# DO NOT put HOST1 variables here — they belong in master_host1.conf.
|
||||
# DO NOT put HOST1 variables here — they belong in host1.conf.
|
||||
#
|
||||
# ── STATUS ────────────────────────────────────────────────────────────────────────────────────
|
||||
# HOST2 is currently being rebuilt — most sections scaffolded, fill in when back online.
|
||||
@@ -61,6 +61,12 @@
|
||||
# RADARR URL, API key, path map
|
||||
# ARR RECOVERY per-arr recovery toggles (no Lidarr on HOST2)
|
||||
#
|
||||
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
|
||||
# SYSTEM WATCHDOG per-host check toggles and NIC configuration
|
||||
#
|
||||
# ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
|
||||
# RESOURCE MANAGER containers paused/stopped under memory pressure
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -72,6 +78,8 @@
|
||||
# SSH key used for all server-to-server operations — rsync, failover container commands.
|
||||
# Must be in /root/.ssh/ and authorised in HOST1's /root/.ssh/authorized_keys.
|
||||
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"
|
||||
HOST2_OWNER="jayred365"
|
||||
HOST2_OWNER_EMAIL="" # fill in when HOST2 is back online
|
||||
|
||||
# ━━━ Emby ━━━
|
||||
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
|
||||
@@ -284,6 +292,14 @@
|
||||
# ["Authelia"]="Mariadb-Authelia Redis-Authelia"
|
||||
)
|
||||
|
||||
# Per-container appdata growth suppress ceilings in MB.
|
||||
# ONLY needed in specific cases — growth rate detection covers all containers automatically.
|
||||
# Use when a container legitimately has large stable data and you want to suppress false-positive
|
||||
# growth alerts. Add entries here only when a container triggers warnings it shouldn't.
|
||||
declare -A HOST2_WATCHDOG_APPDATA_SIZES=(
|
||||
# add HOST2 suppress entries here only as needed
|
||||
)
|
||||
|
||||
# ━━━ Docker Network Connect ━━━
|
||||
# Containers connected to custom networks at array start by docker_network_connect.sh.
|
||||
# Networks created if they don't exist — idempotent, safe to re-run.
|
||||
@@ -318,7 +334,7 @@
|
||||
# ━━━ Fallback Tiers — HOST2 Runs for HOST1 ━━━
|
||||
# Containers HOST2 starts when HOST1 goes down.
|
||||
# Tier 1 is always immediate — vital services cannot wait.
|
||||
# Higher tiers activate after HOST1_TIER*_DELAY minutes (set in master_host1.conf).
|
||||
# Higher tiers activate after HOST1_TIER*_DELAY minutes (set in host1.conf).
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
||||
"Gmer4Lfe.com"
|
||||
"Emby"
|
||||
@@ -501,10 +517,6 @@
|
||||
HOST2_RADARR_RECOVERY=true
|
||||
# HOST2_LIDARR_RECOVERY not set — Lidarr does not run on HOST2
|
||||
|
||||
# ==============================================================================================
|
||||
# ──────────────────────── End Of HOST2 Variables ──────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ==============================================================================================
|
||||
# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
@@ -597,6 +609,24 @@
|
||||
# DISABLED — rebuild workloads may legitimately peg CPU. Enable after rebuild.
|
||||
HOST2_SYS_WATCHDOG_CHECK_RUNAWAY=false
|
||||
|
||||
# ==============================================================================================
|
||||
# ── RESOURCE MANAGER ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Containers to manage under pressure — see master.conf RW_CRITICAL_CONTAINERS for exclusions.
|
||||
|
||||
# docker pause at medium pressure (RAM < RW_RAM_MEDIUM_GB or load > medium threshold)
|
||||
# Suspended in-place — instant to pause/unpause, no state lost, no restart delay.
|
||||
HOST2_RW_PAUSE_CONTAINERS=(
|
||||
# fill in when HOST2 is back online
|
||||
)
|
||||
|
||||
# docker stop at hard pressure (RAM < RW_RAM_HARD_GB)
|
||||
# Full stop — these are optional/heavy services that free significant RAM when stopped.
|
||||
# resource_watchdog.sh restarts them when pressure fully clears (RAM >= RW_RAM_RECOVER_GB).
|
||||
HOST2_RW_STOP_CONTAINERS=(
|
||||
# fill in when HOST2 is back online
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ──────────────────────── End Of HOST2 Variables ──────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
+11
-11
@@ -7,23 +7,23 @@
|
||||
#
|
||||
# ── HOW IT WORKS ──────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Sources master.conf (shared config — hostnames, thresholds, toggles, profiles, job lists)
|
||||
# 2. Auto-discovers and sources all master_host*.conf files in the same directory
|
||||
# 2. Auto-discovers and sources all host*.conf files in the same directory
|
||||
# Each host conf extends the shared profile arrays and adds host-specific credentials
|
||||
# 3. Sources common.sh (shared functions — detect_hosts, logging, notifications etc.)
|
||||
#
|
||||
# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────
|
||||
# Without this loader every script had to explicitly source each conf file:
|
||||
# source master.conf
|
||||
# source master_host1.conf
|
||||
# source master_host2.conf
|
||||
# source host1.conf
|
||||
# source host2.conf
|
||||
# source common.sh
|
||||
#
|
||||
# Adding a new server meant updating every script.
|
||||
# With this loader — add master_host3.conf to the git repo and every server
|
||||
# With this loader — add host3.conf to the git repo and every server
|
||||
# auto-discovers it on next git pull. Zero script changes required. Ever.
|
||||
#
|
||||
# ── ADDING A NEW SERVER ───────────────────────────────────────────────────────────────────────
|
||||
# 1. Create master_host3.conf following the same structure as HOST1/HOST2
|
||||
# 1. Create host3.conf following the same structure as HOST1/HOST2
|
||||
# 2. Commit and push to git repo
|
||||
# 3. All servers pull it automatically — no other changes needed
|
||||
#
|
||||
@@ -40,9 +40,9 @@
|
||||
# source "$SCRIPT_DIR/load_config.sh"
|
||||
#
|
||||
# ── SPARSE CHECKOUT NOTE ──────────────────────────────────────────────────────────────────────
|
||||
# Sparse checkout controls which master_host*.conf files each server receives.
|
||||
# HOST1 only pulls master_host1.conf — never HOST2's credentials.
|
||||
# HOST2 only pulls master_host2.conf — never HOST1's credentials.
|
||||
# Sparse checkout controls which host*.conf files each server receives.
|
||||
# HOST1 only pulls host1.conf — never HOST2's credentials.
|
||||
# HOST2 only pulls host2.conf — never HOST1's credentials.
|
||||
# This loader sources whatever conf files ARE present — sparse checkout handles the rest.
|
||||
# Both servers pull all non-credential conf files (master.conf, common.sh, load_config.sh).
|
||||
#
|
||||
@@ -63,14 +63,14 @@
|
||||
fi
|
||||
source "$LOAD_CONFIG_DIR/master.conf"
|
||||
|
||||
# ━━━ Auto-discover and source all master_host*.conf files ━━━
|
||||
# ━━━ Auto-discover and source all host*.conf files ━━━
|
||||
# Sorted for consistent load order — HOST1 before HOST2 before HOST3 etc.
|
||||
# Each host conf extends the shared PROFILE_* arrays and adds host-specific vars.
|
||||
# Missing files are silently skipped — sparse checkout intentionally withholds some.
|
||||
# At least one host conf must be present or the ecosystem has no identity to work with.
|
||||
_host_confs_loaded=0
|
||||
|
||||
for _conf in $(ls "$LOAD_CONFIG_DIR"/master_host*.conf 2>/dev/null | sort); do
|
||||
for _conf in $(ls "$LOAD_CONFIG_DIR"/host*.conf 2>/dev/null | sort); do
|
||||
if [[ -f "$_conf" ]]; then
|
||||
source "$_conf"
|
||||
(( _host_confs_loaded++ ))
|
||||
@@ -80,7 +80,7 @@
|
||||
done
|
||||
|
||||
if [[ "$_host_confs_loaded" -eq 0 ]]; then
|
||||
echo "[FATAL] No master_host*.conf files found in $LOAD_CONFIG_DIR" >&2
|
||||
echo "[FATAL] No host*.conf files found in $LOAD_CONFIG_DIR" >&2
|
||||
echo "[FATAL] At least one host conf required — check git pull and sparse checkout" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
+76
-51
@@ -8,10 +8,10 @@
|
||||
# ── HOW THE THREE-FILE SYSTEM WORKS ──────────────────────────────────────────────────────────
|
||||
# Scripts source all three files at startup:
|
||||
# source master.conf ← shared config (this file)
|
||||
# source master_host1.conf ← HOST1 credentials, shares, container lists
|
||||
# source master_host2.conf ← HOST2 credentials, shares, container lists
|
||||
# source host1.conf ← HOST1 credentials, shares, container lists
|
||||
# source host2.conf ← HOST2 credentials, shares, container lists
|
||||
#
|
||||
# Sparse checkout (git) ensures each server only pulls its own master_host*.conf.
|
||||
# Sparse checkout (git) ensures each server only pulls its own host*.conf.
|
||||
# HOST2 never sees HOST1 credentials. HOST1 never sees HOST2 credentials.
|
||||
#
|
||||
# What belongs here: thresholds, toggles, intervals, profiles, job lists
|
||||
@@ -99,7 +99,7 @@
|
||||
# Must match the exact unRAID hostname AND Tailscale device name (case sensitive).
|
||||
# detect_hosts() in common.sh matches the local hostname against these to set MY_ID / REMOTE_ID.
|
||||
# Tailscale IP resolution uses these names — no hardcoded IPs needed.
|
||||
# To add a new server: add HOST3="unRAID-NewServer" here + create master_host3.conf.
|
||||
# To add a new server: add HOST3="unRAID-NewServer" here + create host3.conf.
|
||||
HOST1="unRAID-Gmer4Lfe"
|
||||
HOST2="unRAID-Jayred365"
|
||||
|
||||
@@ -145,7 +145,7 @@
|
||||
# Manages the relationship lifecycle between two unRAID servers.
|
||||
# HOST1 is always the owner (source of truth) — HOST2 is always the mirror.
|
||||
# PARTNERSHIP_OWNER_HOST flips to "HOST2" after a --transfer operation.
|
||||
# All identity vars (hostnames, SSH keys) live in master_host*.conf.
|
||||
# All identity vars (hostnames, SSH keys) live in host*.conf.
|
||||
# Hostnames already match Tailscale device names — IP resolution is automatic.
|
||||
#
|
||||
# State files on /boot/config — survives reboots, available before array starts:
|
||||
@@ -164,13 +164,13 @@
|
||||
PARTNERSHIP_ENABLED=false
|
||||
PARTNERSHIP_OWNER_HOST="HOST1" # "HOST1" or "HOST2" — flips on --transfer
|
||||
|
||||
# Auth containers reconfigured on onboard/offboard — defined per host in master_host*.conf.
|
||||
# Auth containers reconfigured on onboard/offboard — defined per host in host*.conf.
|
||||
# Format: "ContainerName|WebUIPort"
|
||||
# HOST1_PARTNERSHIP_AUTH_WEBUIS / HOST2_PARTNERSHIP_AUTH_WEBUIS
|
||||
# On onboard → WebUI pointed at owner's Tailscale IP
|
||||
# On offboard → WebUI pointed back at localhost
|
||||
|
||||
# Paths to collect during the grace window after offboard — defined per host in master_host*.conf.
|
||||
# Paths to collect during the grace window after offboard — defined per host in host*.conf.
|
||||
# HOST1_PARTNERSHIP_MIRROR_BACKUPS / HOST2_PARTNERSHIP_MIRROR_BACKUPS
|
||||
# Notified on offboard — no auto-deletion, manual collection.
|
||||
|
||||
@@ -219,13 +219,6 @@
|
||||
# ── LOGGING ───────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# Silent-by-default output model — ecosystem only speaks when something is wrong.
|
||||
# true = only warn() and error() produce output (default — reduces notification spam)
|
||||
# false = all output visible — use for monitor scripts or debugging
|
||||
# Override per-run: script --log sets ENABLE_LOGGING=true for [LOG] detail
|
||||
# Monitor scripts (coffee_report, health_digest etc.) set SILENT_MODE=false themselves
|
||||
SILENT_MODE=true
|
||||
|
||||
# Controls verbose [LOG] output across all scripts.
|
||||
# true = show detailed [LOG] lines — useful for debugging or first-time setup
|
||||
# false = show only user-facing output — cleaner for scheduled runs
|
||||
@@ -239,7 +232,7 @@
|
||||
# normal = job completed successfully / warning = something failed or needs attention
|
||||
NOTIFY_UNRAID=true
|
||||
|
||||
# Discord webhook URL — defined per host in master_host*.conf.
|
||||
# Discord webhook URL — defined per host in host*.conf.
|
||||
# HOST1_DISCORD_WEBHOOK / HOST2_DISCORD_WEBHOOK
|
||||
# Allows different webhooks per server, or only one server notifying.
|
||||
|
||||
@@ -284,7 +277,6 @@
|
||||
# Watchdogs (resource_watchdog, docker_watchdog, system_watchdog) are cronned via
|
||||
# watchdog_orchestrator.sh — NOT launched here.
|
||||
ARRAY_START_SCRIPTS=(
|
||||
"git_pull_execute.sh" # pull latest scripts before anything starts
|
||||
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
|
||||
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
|
||||
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
|
||||
@@ -298,11 +290,12 @@
|
||||
# Schedule: * * * * * (every minute)
|
||||
# NOT in ARRAY_START_SCRIPTS — has its own cron entry.
|
||||
# Order matters — resource first (frees pressure), docker second (heals with freed resources),
|
||||
# system last (reboots only if prior layers failed).
|
||||
# storage third (pool/data health — never reboots), system last (last line of defense).
|
||||
WATCHDOG_ORCHESTRATOR_SCRIPTS=(
|
||||
"unRAID_Essentials/resource_watchdog.sh" # reduce system pressure before healing attempts
|
||||
"Docker_Essentials/docker_watchdog.sh" # heal containers with freed resources
|
||||
"unRAID_Essentials/system_watchdog.sh" # reboot if all else fails — last line of defense
|
||||
"Watchdogs/resource_watchdog.sh" # reduce system pressure before healing attempts
|
||||
"Watchdogs/docker_watchdog.sh" # heal containers with freed resources
|
||||
"Watchdogs/storage_watchdog.sh" # pool and appdata health — alert and remediate
|
||||
"Watchdogs/system_watchdog.sh" # reboot if all else fails — last line of defense
|
||||
)
|
||||
|
||||
WATCHDOG_ORCHESTRATOR_HEARTBEAT=true
|
||||
@@ -317,7 +310,7 @@
|
||||
"Docker_Essentials/downloaders_reset.sh" # clear stuck download states every 15min
|
||||
)
|
||||
|
||||
# Shares synced every 15 minutes — defined per host in master_host*.conf.
|
||||
# Shares synced every 15 minutes — defined per host in host*.conf.
|
||||
# HOST1_CRITICAL_SYNC_SHARES / HOST2_CRITICAL_SYNC_SHARES
|
||||
# Format: "/path/to/share" or "/path/to/share|profile-name"
|
||||
# Order matters — Critical-Data first (auth stack), then Emby dirty sync.
|
||||
@@ -326,7 +319,7 @@
|
||||
# intermediate_sync_maintenance.sh runs every 4 hours — arr library sync, artwork fetch,
|
||||
# and optional mid-day rsync for any shares that need sub-daily propagation.
|
||||
# Schedule: 0 */4 * * *
|
||||
# INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in master_host*.conf.
|
||||
# INTERMEDIATE_SYNC_SHARES is host-specific — configure HOST*_INTERMEDIATE_SYNC_SHARES in host*.conf.
|
||||
INTERMEDIATE_RSYNC_ENABLED=true # set false to disable mid-day rsync without removing shares
|
||||
|
||||
INTERMEDIATE_MAINTENANCE_SCRIPTS=(
|
||||
@@ -346,7 +339,7 @@
|
||||
"Media/media_cleaner.sh media" # remove junk from media shares
|
||||
#"Media/lidarr_cleanup.sh" # remove orphaned music files — enable when ready
|
||||
#"Media/sonarr_cleanup.sh" # remove orphaned TV files — enable when ready
|
||||
#"Media/radarr_cleanup.sh" # remove orphaned movie files — enable when ready
|
||||
"Media/radarr_cleanup.sh" # remove orphaned movie files
|
||||
"Media/lidarr_missing_art.sh" # fetch missing album/artist artwork (HOST1 only — self-guards)
|
||||
"Media/radarr_tmdb_removed.sh" # remove movies dropped from TMDb
|
||||
"Media/sonarr_tvdb_removed.sh" # remove series dropped from TVDB
|
||||
@@ -360,7 +353,7 @@
|
||||
DAILY_CONTAINER_UPDATES=true
|
||||
|
||||
# Media shares synced daily by daily_sync_maintenance.sh.
|
||||
# Defined per-host in master_host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
|
||||
# Defined per-host in host*.conf — HOST1_DAILY_SYNC_SHARES and HOST2_DAILY_SYNC_SHARES.
|
||||
# Mesh model: every node pushes every media share. rsync has no --delete so pushes are additive.
|
||||
# arr_sync (union) ensures all arr libraries converge first. arr_cleanup removes true orphans.
|
||||
# Any node can download content to any share — it propagates to all nodes on the next cycle.
|
||||
@@ -368,7 +361,7 @@
|
||||
# These shares use DEFAULT_RSYNC_OPTS — no profile entry needed.
|
||||
# For shares needing custom options or container stops — create a profile in RSYNC section.
|
||||
|
||||
# Personal encrypted shares defined per-host in master_host*.conf.
|
||||
# Personal encrypted shares defined per-host in host*.conf.
|
||||
# ZFS encrypted at dataset level — remote receives encrypted blocks, cannot read content.
|
||||
# See README-Rsync_Setup.md for ZFS encryption setup before uncommenting.
|
||||
|
||||
@@ -390,7 +383,7 @@
|
||||
# Set false to skip — docker_weekly_restart.sh still runs regardless.
|
||||
WEEKLY_REMAINING_UPDATES=true
|
||||
|
||||
# Shares synced during the weekly maintenance window — defined per host in master_host*.conf.
|
||||
# Shares synced during the weekly maintenance window — defined per host in host*.conf.
|
||||
# HOST1_WEEKLY_SYNC_SHARES / HOST2_WEEKLY_SYNC_SHARES
|
||||
# Containers stopped both sides before sync — full clean state guaranteed.
|
||||
# Profiles drive container stops, excludes, and options — configure in RSYNC section.
|
||||
@@ -473,8 +466,8 @@
|
||||
# called by weekly_sync_maintenance.sh — full clean sync weekly
|
||||
# critical-fallback — dirty sync — auth stays running both sides, WAL excluded
|
||||
# called by critical_sync_maintenance.sh every 15min
|
||||
# host1-appdata — HOST1 server-specific appdata — defined in master_host1.conf
|
||||
# host2-appdata — HOST2 server-specific appdata — defined in master_host2.conf
|
||||
# host1-appdata — HOST1 server-specific appdata — defined in host1.conf
|
||||
# host2-appdata — HOST2 server-specific appdata — defined in host2.conf
|
||||
# important-data — NextCloud + Postgres — NextCloud delayed start after Postgres
|
||||
# emby — weekly clean sync — both Emby stopped, full mirror
|
||||
# called by weekly_sync_maintenance.sh only — do NOT schedule separately
|
||||
@@ -591,7 +584,7 @@
|
||||
# Failover → start remote DDNS first (Tier 1)
|
||||
# Handback → stop remote DDNS → rsync → start containers → start local DDNS last
|
||||
#
|
||||
# Per-host container lists and tier delays live in master_host*.conf.
|
||||
# Per-host container lists and tier delays live in host*.conf.
|
||||
# Shared settings (intervals, state file, thresholds) live here.
|
||||
|
||||
EXTERNAL_IP="8.8.8.8"
|
||||
@@ -613,7 +606,7 @@
|
||||
# ━━━ Downloaders Reset ━━━
|
||||
# Runs every 15 minutes via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Clears stuck states, purges old history, prepares each download client for a clean cycle.
|
||||
# Per-host URLs and API keys live in master_host*.conf.
|
||||
# Per-host URLs and API keys live in host*.conf.
|
||||
DOWNLOADER_RETENTION_DAYS=7 # days — purge history older than this
|
||||
|
||||
# qBittorrent failsafe — removes torrents older than threshold regardless of ratio
|
||||
@@ -623,7 +616,7 @@
|
||||
|
||||
# ━━━ Docker Daily Restart ━━━
|
||||
# Containers restarted every day via DAILY_MAINTENANCE_SCRIPTS.
|
||||
# Per-host lists live in master_host*.conf:
|
||||
# Per-host lists live in host*.conf:
|
||||
# HOST1_DAILY_RESTART_CONTAINERS
|
||||
# HOST2_DAILY_RESTART_CONTAINERS
|
||||
# detect_hosts() sets DAILY_RESTART_CONTAINERS to the correct host array at runtime.
|
||||
@@ -631,7 +624,7 @@
|
||||
# ━━━ Docker Weekly Restart ━━━
|
||||
# Less critical services restarted weekly via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am).
|
||||
# Containers already stopped for weekly sync — restart adds zero extra downtime.
|
||||
# Per-host lists live in master_host*.conf:
|
||||
# Per-host lists live in host*.conf:
|
||||
# HOST1_WEEKLY_RESTART_CONTAINERS
|
||||
# HOST2_WEEKLY_RESTART_CONTAINERS
|
||||
|
||||
@@ -653,7 +646,7 @@
|
||||
# Dead containers — remove and restart
|
||||
# Unexpected exits — non-zero exit code → restart
|
||||
#
|
||||
# All per-host container lists live in master_host*.conf:
|
||||
# All per-host container lists live in host*.conf:
|
||||
# HOST*_WATCHDOG_CONTAINERS — memory hard limits per container
|
||||
# HOST*_WATCHDOG_CONTAINER_URLS — HTTP health check URLs
|
||||
# HOST*_WATCHDOG_REQUIRED_CONTAINERS — must always be running
|
||||
@@ -706,10 +699,42 @@
|
||||
# Notification batching — one summary per cycle instead of one ping per event
|
||||
WATCHDOG_BATCH_NOTIFY=true
|
||||
|
||||
# Appdata size monitoring — two-part catch-all for runaway growth and oversized log files.
|
||||
#
|
||||
# Part 1 — Growth rate (zero-config):
|
||||
# Reads per-container dir totals each cycle via du, compares to previous cycle.
|
||||
# Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused *.log scan
|
||||
# inside that container. No per-container config required — new containers are covered
|
||||
# automatically. Baseline built on first run after boot; growth detection starts cycle 2.
|
||||
#
|
||||
# Part 2 — Absolute log size:
|
||||
# Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB across all appdata paths.
|
||||
# Catches logs that have already stabilised at a large size and are no longer actively growing.
|
||||
#
|
||||
# Strike system (reuses existing watchdog infrastructure):
|
||||
# Strike 1 — warn + notify: condition first detected
|
||||
# Strike 2 — warn + escalated notify: still present next cycle
|
||||
# Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
|
||||
# If WATCHDOG_APPDATA_TRUNCATE_LOGS=true: truncate *.log files in-place, clear strikes
|
||||
# If false: critical notify only, strikes held until condition resolves
|
||||
# Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
|
||||
#
|
||||
# HOST*_WATCHDOG_APPDATA_SIZES (in host*.conf) suppresses growth warnings for a
|
||||
# container until its dir exceeds the configured ceiling. Only needed when a container
|
||||
# legitimately has large stable data and you want to guarantee it never triggers a false alarm.
|
||||
WATCHDOG_CHECK_APPDATA=true
|
||||
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
|
||||
WATCHDOG_APPDATA_GROWTH_GB=2 # flag containers growing more than this per cycle
|
||||
WATCHDOG_APPDATA_LOG_MAX_GB=2 # flag *.log files exceeding this size (absolute)
|
||||
WATCHDOG_APPDATA_TRUNCATE_LOGS=false # set true to auto-truncate oversized *.log files on action cycle
|
||||
WATCHDOG_APPDATA_STRIKE_LIMIT=3 # cycles before action fires (matches existing watchdog pattern)
|
||||
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db" # /tmp — resets on reboot ✅
|
||||
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db" # /tmp — resets on reboot ✅
|
||||
|
||||
# ━━━ Docker Network Connect ━━━
|
||||
# Ensures custom networks exist and connects containers at array start.
|
||||
# Runs once via ARRAY_START_SCRIPTS — idempotent, safe to re-run.
|
||||
# Container and network lists are host-specific — defined in master_host*.conf:
|
||||
# Container and network lists are host-specific — defined in host*.conf:
|
||||
# HOST1_NETWORK_CONNECT_CONTAINERS / HOST2_NETWORK_CONNECT_CONTAINERS
|
||||
# HOST1_NETWORK_CONNECT_NETWORKS / HOST2_NETWORK_CONNECT_NETWORKS
|
||||
|
||||
@@ -816,13 +841,13 @@
|
||||
PERMISSIONS_FILE_MODE="664" # files — group read/write, no execute
|
||||
PERMISSIONS_OWNER="nobody:users"
|
||||
|
||||
# Share list defined per host in master_host*.conf:
|
||||
# Share list defined per host in host*.conf:
|
||||
# HOST1_MEDIA_PERMISSION_SHARES / HOST2_MEDIA_PERMISSION_SHARES
|
||||
|
||||
# ━━━ Media Cleaner ━━━
|
||||
# Removes junk files from media shares — two profiles: anime and media.
|
||||
# Called via DAILY_MAINTENANCE_SCRIPTS. Run manually: Media/media_cleaner.sh anime|media
|
||||
# Folder lists defined per host in master_host*.conf:
|
||||
# Folder lists defined per host in host*.conf:
|
||||
# HOST1_ANIME_CLEAN_FOLDERS / HOST2_ANIME_CLEAN_FOLDERS
|
||||
# HOST1_MEDIA_CLEAN_FOLDERS / HOST2_MEDIA_CLEAN_FOLDERS
|
||||
|
||||
@@ -904,7 +929,7 @@
|
||||
|
||||
# ━━━ Arr Cleanup ━━━
|
||||
# Orphan file cleanup via Lidarr, Sonarr, and Radarr APIs.
|
||||
# Per-host URLs, API keys, and path maps live in master_host*.conf.
|
||||
# Per-host URLs, API keys, and path maps live in host*.conf.
|
||||
# detect_hosts() selects correct host vars at runtime.
|
||||
#
|
||||
# API versions — update MAJOR version here when script is updated for a new arr version:
|
||||
@@ -940,7 +965,7 @@
|
||||
LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs
|
||||
LIDARR_ART_RETRIES=2 # download retry attempts per image
|
||||
LIDARR_ART_SLEEP_BETWEEN=0.2 # seconds between fanart.tv API calls
|
||||
# HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in master_host*.conf
|
||||
# HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in host*.conf
|
||||
|
||||
# Lidarr discovery settings (playback_aware_lidarr_discovery.sh)
|
||||
LIDARR_DISCOVERY_THRESHOLD=70 # score to accept candidate (0-100)
|
||||
@@ -1037,7 +1062,7 @@
|
||||
# stalled — download stuck with no connections or progress
|
||||
#
|
||||
# Items newer than ARR_IMPORT_RECOVERY_AGE are skipped — gives arr time to retry first.
|
||||
# Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in master_host*.conf.
|
||||
# Per-host recovery toggles (HOST1_SONARR_RECOVERY etc.) live in host*.conf.
|
||||
ARR_IMPORT_RECOVERY_AGE=6 # hours — skip items newer than this
|
||||
# matches cron interval — items eligible after one missed cycle
|
||||
|
||||
@@ -1057,7 +1082,7 @@
|
||||
# Must exist before Emby starts so the symlink resolves correctly.
|
||||
RAMDISK_PATH="/mnt/ramdisk_transcodes"
|
||||
|
||||
# Ramdisk size and flip thresholds — defined per host in master_host*.conf.
|
||||
# Ramdisk size and flip thresholds — defined per host in host*.conf.
|
||||
# All three are coupled — if size changes, thresholds must change with it.
|
||||
# HOST1_RAMDISK_SIZE / HOST2_RAMDISK_SIZE
|
||||
# HOST1_RAMDISK_WARN_GB / HOST2_RAMDISK_WARN_GB ← flip to SSD at this usage
|
||||
@@ -1068,7 +1093,7 @@
|
||||
# Must match the container path configured in Emby's Extra Parameters.
|
||||
TRANSCODE_LINK="/mnt/ram-transcode"
|
||||
|
||||
# SSD fallback location — defined per host in master_host*.conf (cache path differs per server):
|
||||
# SSD fallback location — defined per host in host*.conf (cache path differs per server):
|
||||
# HOST1_TRANSCODE_SSD / HOST2_TRANSCODE_SSD
|
||||
|
||||
# Minimum free GB on SSD before allowing flip from ramdisk to SSD.
|
||||
@@ -1098,7 +1123,7 @@
|
||||
# Format: "ContainerName|URL|APIKey|Type" — Type: emby | jellyfin | plex
|
||||
# Entries with placeholder API keys are skipped automatically.
|
||||
# ⚠️ Tdarr does NOT belong here — keep Tdarr on SSD, not ramdisk.
|
||||
# Defined per host in master_host*.conf — Emby container names and keys differ per server:
|
||||
# Defined per host in host*.conf — Emby container names and keys differ per server:
|
||||
# HOST1_TRANSCODE_SERVERS / HOST2_TRANSCODE_SERVERS
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -1108,7 +1133,7 @@
|
||||
# ━━━ Certificate Monitor ━━━
|
||||
# Checks SSL certificate expiry via direct openssl connection — no NPM dependency.
|
||||
# Checks the actual certificate served by each domain, not what NPM thinks it has.
|
||||
# Domains defined per host in master_host*.conf — each server monitors its own domains:
|
||||
# Domains defined per host in host*.conf — each server monitors its own domains:
|
||||
# HOST1_CERT_MONITOR_DOMAINS / HOST2_CERT_MONITOR_DOMAINS
|
||||
CERT_WARN_DAYS=30 # warn when cert expires within this many days
|
||||
CERT_CRIT_DAYS=7 # critical alert within this many days
|
||||
@@ -1117,7 +1142,7 @@
|
||||
# ━━━ Backup Verify ━━━
|
||||
# Verifies rsync mirror health by comparing random file checksums between servers.
|
||||
# Catches silent corruption or incomplete syncs that rsync itself wouldn't detect.
|
||||
# Defined per host in master_host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically:
|
||||
# Defined per host in host*.conf — leave empty to use HOST*_DAILY_SYNC_SHARES automatically:
|
||||
# HOST1_BACKUP_VERIFY_SHARES / HOST2_BACKUP_VERIFY_SHARES
|
||||
BACKUP_VERIFY_SAMPLE=10 # random files to check per share
|
||||
BACKUP_VERIFY_MIN_SIZE=1M # minimum file size to include in sample
|
||||
@@ -1128,7 +1153,7 @@
|
||||
# (hot/max/hotssd/maxssd) — these vars are fallback only if dynamix.cfg not found.
|
||||
SMART_TEMP_WARN=45 # fallback — Celsius warn threshold
|
||||
SMART_TEMP_CRIT=55 # fallback — Celsius critical threshold
|
||||
# Drives to ignore defined per host in master_host*.conf — hardware is server-specific:
|
||||
# Drives to ignore defined per host in host*.conf — hardware is server-specific:
|
||||
# HOST1_SMART_IGNORE_DRIVES / HOST2_SMART_IGNORE_DRIVES
|
||||
|
||||
# ━━━ ZFS Memory Snapshot ━━━
|
||||
@@ -1138,7 +1163,7 @@
|
||||
ZFS_REPORT_FREE_WARN_GB=10 # warn if less than this GB free RAM
|
||||
ZFS_REPORT_AVAIL_WARN_GB=20 # warn if less than this GB available on ZFS pool
|
||||
ZFS_REPORT_DOCKER_TOP=10 # how many top Docker containers to show by memory
|
||||
# Pool ignore list defined per host in master_host*.conf — pool names are server-specific:
|
||||
# Pool ignore list defined per host in host*.conf — pool names are server-specific:
|
||||
# HOST1_ZFS_REPORT_IGNORE_POOLS / HOST2_ZFS_REPORT_IGNORE_POOLS
|
||||
|
||||
# ━━━ Bandwidth Monitor ━━━
|
||||
@@ -1168,7 +1193,7 @@
|
||||
|
||||
# ━━━ Emby Session Report ━━━
|
||||
# Weekly Emby usage statistics via API — no persistent writes, queries fresh each run.
|
||||
# URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in master_host*.conf.
|
||||
# URL and API key pulled from HOST*_EMBY_URL and HOST*_EMBY_API_KEY in host*.conf.
|
||||
EMBY_REPORT_DAYS=7 # days to include in the report period
|
||||
EMBY_REPORT_TOP_N=10 # number of top content items to show
|
||||
|
||||
@@ -1184,8 +1209,8 @@
|
||||
# Level 3 (hard) — docker stop optional containers, signal docker_watchdog to defer
|
||||
#
|
||||
# ── PER-HOST CONTAINER LISTS ──────────────────────────────────────────────────────────────────
|
||||
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (in master_host*.conf)
|
||||
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (in master_host*.conf)
|
||||
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (in host*.conf)
|
||||
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (in host*.conf)
|
||||
|
||||
RW_ENABLED=true
|
||||
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
|
||||
@@ -1335,9 +1360,9 @@
|
||||
SYS_WATCHDOG_MDSTAT_ERROR_LIMIT=5 # new errors in one cycle before acting
|
||||
|
||||
# ━━━ Check Toggles ━━━
|
||||
# Per-host — moved to master_host*.conf
|
||||
# Per-host — moved to host*.conf
|
||||
# Different servers may have different hardware, NICs, and check requirements
|
||||
# See HOST*_SYS_WATCHDOG_CHECK_* in master_host*.conf
|
||||
# See HOST*_SYS_WATCHDOG_CHECK_* in host*.conf
|
||||
|
||||
# ━━━ Abort Toggles ━━━
|
||||
# Conditions that prevent reboot even when a threshold is hit.
|
||||
|
||||
@@ -4,13 +4,15 @@ Configuration reference, operational procedures, and troubleshooting for
|
||||
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
|
||||
adding or reordering scripts at array start.
|
||||
|
||||
> **Watchdog scripts have moved.** `system_watchdog.sh` and `resource_watchdog.sh`
|
||||
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
|
||||
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [resource_watchdog.sh](#resource_watchdogsh)
|
||||
- [webgui_restart.sh](#webgui_restartsh)
|
||||
- [inotify_tuning.sh](#inotify_tuningsh)
|
||||
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
|
||||
@@ -25,6 +27,22 @@ adding or reordering scripts at array start.
|
||||
|
||||
---
|
||||
|
||||
## Output Tiers
|
||||
|
||||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||||
lines only appear when `--log` is passed.
|
||||
|
||||
**Daemon scripts** (`webgui_restart.sh`): run on every cycle. Without `--log`, only
|
||||
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
|
||||
Per-check detail suppressed.
|
||||
|
||||
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
|
||||
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
|
||||
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
|
||||
final summary are visible. Per-item detail suppressed.
|
||||
|
||||
---
|
||||
|
||||
## ARRAY_START_SCRIPTS Order
|
||||
|
||||
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
|
||||
@@ -43,8 +61,9 @@ ARRAY_START_SCRIPTS=(
|
||||
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
|
||||
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
|
||||
...
|
||||
"system_watchdog.sh" # LAST or near-last — starts background loop
|
||||
)
|
||||
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every minute via
|
||||
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
|
||||
```
|
||||
|
||||
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
|
||||
@@ -57,151 +76,6 @@ generates veth messages — these will appear in syslog if the filter isn't acti
|
||||
|
||||
---
|
||||
|
||||
## system_watchdog.sh
|
||||
|
||||
### Three-Tier Response
|
||||
|
||||
The watchdog categorizes every failure into one of three tiers:
|
||||
|
||||
**Tier 1 — CRITICAL (immediate reboot, no strikes)**
|
||||
| Condition | Threshold | Why immediate |
|
||||
|-----------|-----------|---------------|
|
||||
| Docker daemon unresponsive | N/A | Nothing can be healed; every docker command hangs |
|
||||
| rootfs usage | SYS_WATCHDOG_ROOTFS_CRITICAL_PCT (99%) | SSH stops; state files fail silently |
|
||||
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
|
||||
| File descriptor exhaustion | SYS_WATCHDOG_FD_CRITICAL_PCT (95%) | New connections silently failing |
|
||||
| /boot read-only unexpectedly | write test fails | Config writes silently failing |
|
||||
|
||||
**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
|
||||
|
||||
RAM below MEM_GB AND OOM kills this cycle >= SYS_WATCHDOG_OOM_LIMIT.
|
||||
Both conditions required — RAM alone without OOM uses the standard strike system.
|
||||
OOM confirms the system is dying faster than watchdogs can heal.
|
||||
|
||||
**Tier 3 — STANDARD (SYS_WATCHDOG_STRIKES consecutive failures → reboot)**
|
||||
| Check | Threshold |
|
||||
|-------|-----------|
|
||||
| Free RAM | MEM_WARN_GB → MEM_SHUTDOWN_GB → MEM_GB |
|
||||
| Load average | SYS_WATCHDOG_LOAD_MULTIPLIER × cpu_count |
|
||||
| CPU temperature | SYS_WATCHDOG_CPU_TEMP |
|
||||
| Zombie processes | SYS_WATCHDOG_ZOMBIES |
|
||||
| /var/log usage | SYS_WATCHDOG_VAR_LOG_PCT |
|
||||
| /tmp usage | SYS_WATCHDOG_TMP_PCT |
|
||||
| Array disk errors | mdstat error delta > 0 |
|
||||
| NIC state | interface operstate != "up" |
|
||||
| Required containers | containers in SYS_WATCHDOG_REQUIRED_CONTAINERS |
|
||||
|
||||
### RAM Tiers
|
||||
|
||||
```
|
||||
MEM_WARN_GB (10GB) → warn + notify, no action
|
||||
MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery
|
||||
MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM)
|
||||
MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart
|
||||
```
|
||||
|
||||
At MEM_SHUTDOWN_GB, all containers NOT listed in
|
||||
`SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED` are stopped. Excluded containers by default:
|
||||
NginxProxyManager, Authelia, Mariadb, Redis, Emby, Dispatcharr. Adjust in
|
||||
master.conf for your critical services.
|
||||
|
||||
### Abort Conditions
|
||||
|
||||
These conditions prevent a reboot — running them would cause data loss:
|
||||
|
||||
```bash
|
||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=true # Parity check/rebuild running
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=true # Mover running
|
||||
```
|
||||
|
||||
CRITICAL tier bypasses all abort conditions — an imminent crash outweighs
|
||||
data safety concerns.
|
||||
|
||||
### Strike System
|
||||
|
||||
The strike count tracks consecutive failures. A single recovery resets strikes
|
||||
to 0. The reboot only fires after SYS_WATCHDOG_STRIKES consecutive failures on
|
||||
the same check — transient spikes (a brief load burst, a momentary RAM dip) don't
|
||||
trigger reboots.
|
||||
|
||||
### Reboot Rate Limit
|
||||
|
||||
```bash
|
||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # window in hours
|
||||
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots within the window
|
||||
```
|
||||
|
||||
If the server has rebooted SYS_WATCHDOG_MAX_REBOOTS times within the window,
|
||||
system_watchdog stops rebooting and notifies instead. This prevents a boot loop
|
||||
where the watchdog reboots → something crashes again immediately → reboot again.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # start continuous monitoring loop
|
||||
system_watchdog.sh --dry-run # run detection logic without rebooting
|
||||
system_watchdog.sh --status # show thresholds, current state, strike counts
|
||||
system_watchdog.sh --log # verbose per-cycle output
|
||||
```
|
||||
|
||||
### Verify Running
|
||||
|
||||
```bash
|
||||
pgrep -a -f system_watchdog.sh
|
||||
# Expected: shows PID and path
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## resource_watchdog.sh
|
||||
|
||||
### Pressure Levels
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
RW_RAM_SOFT_GB=20 # Level 1 trigger — throttle downloaders
|
||||
RW_RAM_MEDIUM_GB=15 # Level 2 trigger — throttle + pause containers
|
||||
RW_RAM_HARD_GB=10 # Level 3 trigger — stop containers
|
||||
RW_RAM_RECOVER_GB=25 # recover to this before un-stopping at level 3
|
||||
|
||||
RW_LOAD_SOFT_MULTIPLIER=2.0 # load > 2× cpu count = level 1
|
||||
RW_LOAD_MEDIUM_MULTIPLIER=3.0 # load > 3× cpu count = level 2
|
||||
|
||||
RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalating
|
||||
```
|
||||
|
||||
### Per-Host Container Lists
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at medium pressure
|
||||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at hard pressure
|
||||
```
|
||||
|
||||
Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
|
||||
pressure level. Default includes: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
|
||||
|
||||
### docker_watchdog Coordination
|
||||
|
||||
At level 3, resource_watchdog writes `mem_shutdown_active=true` to `RW_STATE_FILE`.
|
||||
docker_watchdog.sh reads this flag each cycle and skips all container restart logic
|
||||
while it is set. Without this coordination, docker_watchdog would immediately
|
||||
restart containers that resource_watchdog just stopped to free RAM.
|
||||
|
||||
The flag is cleared when level 3 pressure resolves and containers are restarted.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
resource_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
resource_watchdog.sh --dry-run # show what would be throttled/paused/stopped
|
||||
resource_watchdog.sh --status # current level, active actions, recovery cycle count
|
||||
resource_watchdog.sh --log # verbose per-check output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## webgui_restart.sh
|
||||
|
||||
### Escalation Logic
|
||||
@@ -578,75 +452,12 @@ server_reboot.sh --reason="disk work" # include reason in notification
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
> Watchdog configuration (`system_watchdog.sh`, `resource_watchdog.sh`,
|
||||
> `docker_watchdog.sh`, `storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
|
||||
# ── System Watchdog ────────────────────────────────────────────────────────────
|
||||
SYS_WATCHDOG_INTERVAL=300 # seconds between check cycles
|
||||
SYS_WATCHDOG_STRIKES=3 # consecutive failures before reboot
|
||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # rate limit window
|
||||
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots in window
|
||||
SYS_WATCHDOG_OOM_LIMIT=3 # OOM kills/cycle for URGENT bypass
|
||||
|
||||
MEM_WARN_GB=10 # warn + notify
|
||||
MEM_SHUTDOWN_GB=6 # stop non-essential containers
|
||||
MEM_GB=4 # strike → reboot
|
||||
MEM_RECOVER_GB=30 # recovery threshold
|
||||
|
||||
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99 # Tier 1 trigger
|
||||
SYS_WATCHDOG_FD_CRITICAL_PCT=95 # Tier 1 trigger
|
||||
SYS_WATCHDOG_LOAD_MULTIPLIER=4 # Tier 3 — ×cpu_count
|
||||
SYS_WATCHDOG_CPU_TEMP=85 # Tier 3 — Celsius
|
||||
SYS_WATCHDOG_ZOMBIES=20 # Tier 3 — process count
|
||||
SYS_WATCHDOG_VAR_LOG_PCT=80 # Tier 3 — percent full
|
||||
SYS_WATCHDOG_TMP_PCT=85 # Tier 3 — percent full
|
||||
|
||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=true
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=true
|
||||
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
|
||||
"NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
|
||||
)
|
||||
SYS_WATCHDOG_REQUIRED_CONTAINERS=() # containers that must be running
|
||||
|
||||
# State files:
|
||||
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="/tmp/sys_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_FAILED_FILE="/tmp/sys_watchdog_failed.db"
|
||||
SYS_WATCHDOG_OOM_FILE="/tmp/sys_watchdog_oom.db"
|
||||
|
||||
# ── Resource Watchdog ──────────────────────────────────────────────────────────
|
||||
RW_ENABLED=true
|
||||
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
|
||||
|
||||
RW_RAM_SOFT_GB=20
|
||||
RW_RAM_MEDIUM_GB=15
|
||||
RW_RAM_HARD_GB=10
|
||||
RW_RAM_RECOVER_GB=25
|
||||
|
||||
RW_LOAD_SOFT_MULTIPLIER=2.0
|
||||
RW_LOAD_MEDIUM_MULTIPLIER=3.0
|
||||
RW_RECOVER_CYCLES=3
|
||||
|
||||
RW_SABNZBD_ENABLED=true
|
||||
RW_SABNZBD_SPEED_SOFT="50M"
|
||||
RW_SABNZBD_SPEED_MEDIUM="10M"
|
||||
RW_QBIT_ENABLED=true
|
||||
RW_QBIT_DL_SOFT=51200 # KB/s
|
||||
RW_QBIT_DL_MEDIUM=10240
|
||||
|
||||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
|
||||
|
||||
# ── Per-Host (master_host*.conf) ───────────────────────────────────────────────
|
||||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
|
||||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
|
||||
HOST1_SABNZBD_URL="http://localhost:8080"
|
||||
HOST1_SABNZBD_API_KEY="your-api-key"
|
||||
HOST1_QBIT_URL="http://localhost:8090"
|
||||
HOST1_QBIT_USERNAME="admin"
|
||||
HOST1_QBIT_PASSWORD="your-password"
|
||||
|
||||
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
|
||||
WEBGUI_URL="http://localhost"
|
||||
WEBGUI_TIMEOUT=5
|
||||
@@ -683,43 +494,8 @@ REBOOT_VM_WAIT=30
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### system_watchdog.sh Not Starting
|
||||
|
||||
```bash
|
||||
# Check if already running (acquire_lock prevents second instance):
|
||||
pgrep -a -f system_watchdog.sh
|
||||
|
||||
# Check state file permissions:
|
||||
ls -la /tmp/sys_watchdog_*.db
|
||||
|
||||
# Run with --log to see startup:
|
||||
system_watchdog.sh --log
|
||||
```
|
||||
|
||||
### system_watchdog Rebooted Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check reboot log:
|
||||
cat /tmp/sys_watchdog_reboots.db
|
||||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||||
|
||||
# Check what condition triggered it:
|
||||
# Look in /var/log/syslog for "system_watchdog" near the reboot time
|
||||
grep "system_watchdog" /var/log/syslog | tail -20
|
||||
```
|
||||
|
||||
### resource_watchdog Paused Containers It Shouldn't Have
|
||||
|
||||
```bash
|
||||
# Check current state:
|
||||
resource_watchdog.sh --status
|
||||
|
||||
# Add the container to RW_CRITICAL_CONTAINERS in master.conf:
|
||||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
|
||||
|
||||
# Un-pause manually if needed:
|
||||
docker unpause MyContainer
|
||||
```
|
||||
> Watchdog troubleshooting (system_watchdog, resource_watchdog, docker_watchdog,
|
||||
> storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
### rsync_stop Killed the Wrong Thing
|
||||
|
||||
|
||||
@@ -71,8 +71,6 @@ SIGTERM (graceful — finishes current file), SIGKILL only if needed.
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
```
|
||||
Last-resort stability system_watchdog.sh — reboots before crash
|
||||
Pressure reduction resource_watchdog.sh — throttle/pause/stop under load
|
||||
WebGUI availability webgui_restart.sh — nginx → php-fpm → emhttp escalation
|
||||
Kernel tuning inotify_tuning.sh — file watch limits
|
||||
php_fpm_max_children.sh — PHP worker count
|
||||
@@ -84,6 +82,9 @@ Graceful operations mover_stop.sh — clean mover stop
|
||||
server_reboot.sh — clean reboot with pre-flight warnings
|
||||
```
|
||||
|
||||
> `system_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
|
||||
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
@@ -93,33 +94,22 @@ Orchestrators/
|
||||
array_start.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
|
||||
─────────────────────────► docker_syslog_filter.sh (second)
|
||||
─────────────────────────► php_fpm_max_children.sh
|
||||
─────────────────────────► system_watchdog.sh (background loop)
|
||||
|
||||
watchdog_orchestrator.sh ───────────────► resource_watchdog.sh (every minute)
|
||||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||||
|
||||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||||
|
||||
Docker_Essentials/
|
||||
docker_watchdog.sh ◄─── reads ──────────── resource_watchdog.sh state
|
||||
(mem_shutdown_active flag)
|
||||
Watchdogs/
|
||||
system_watchdog.sh and resource_watchdog.sh now live here.
|
||||
See Watchdogs/README-Watchdogs.md for how they relate to each other
|
||||
and to docker_watchdog.sh and storage_watchdog.sh.
|
||||
```
|
||||
|
||||
`system_watchdog.sh` and `docker_watchdog.sh` (in Docker_Essentials/) are
|
||||
designed to work together — docker_watchdog heals containers first,
|
||||
system_watchdog reboots only when healing has failed. `resource_watchdog.sh`
|
||||
coordinates with docker_watchdog via the `mem_shutdown_active` state flag to
|
||||
prevent docker_watchdog from restarting containers that resource_watchdog just
|
||||
stopped to free RAM.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop via array_start.sh |
|
||||
| `resource_watchdog.sh` | Pressure reduction — throttle/pause/stop under load | Every minute via watchdog_orchestrator.sh |
|
||||
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
|
||||
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
|
||||
@@ -139,18 +129,12 @@ Array starts
|
||||
│
|
||||
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
|
||||
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
|
||||
├─ php_fpm_max_children.sh ← before WebGUI is under load
|
||||
└─ system_watchdog.sh ← starts background loop
|
||||
|
||||
|
||||
Every minute (watchdog_orchestrator.sh):
|
||||
└─ resource_watchdog.sh
|
||||
Level 1 (soft): throttle SABnzbd + qBit download speeds
|
||||
Level 2 (medium): further throttle + docker pause non-essential containers
|
||||
Level 3 (hard): docker stop optional services + set mem_shutdown_active=true
|
||||
↓
|
||||
docker_watchdog.sh reads mem_shutdown_active — defers restarts
|
||||
└─ php_fpm_max_children.sh ← before WebGUI is under load
|
||||
|
||||
Every minute (watchdog_orchestrator.sh in Orchestrators/):
|
||||
→ Watchdogs/resource_watchdog.sh → Watchdogs/docker_watchdog.sh
|
||||
→ Watchdogs/storage_watchdog.sh → Watchdogs/system_watchdog.sh
|
||||
(see Watchdogs/README-Watchdogs.md for full flow)
|
||||
|
||||
Every 10 minutes (User Scripts):
|
||||
└─ webgui_restart.sh
|
||||
@@ -161,13 +145,11 @@ Every 10 minutes (User Scripts):
|
||||
Step 3: restart emhttp → recheck
|
||||
All failed → notify, exit 1
|
||||
|
||||
|
||||
Weekly (weekly_maintenance.sh):
|
||||
└─ clear_logs.sh
|
||||
System logs: clear if > LOG_MIN_SIZE_MB
|
||||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||||
|
||||
|
||||
Manual operations:
|
||||
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
|
||||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||||
|
||||
@@ -277,12 +277,12 @@ if [[ "$TOTAL_CLEARED" -gt 0 || ${#FAILED[@]} -gt 0 ]]; then
|
||||
notify "Log clear failed on $(hostname) ($MY_ID) — ${FAILED[*]}" \
|
||||
"Clear Logs" "warning"
|
||||
else
|
||||
log "$ICON_DONE Status: done — ${TOTAL_FREED_H} freed"
|
||||
echo "$ICON_DONE Status: done — ${TOTAL_FREED_H} freed"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
# All logs under threshold — completely silent
|
||||
log "All logs under threshold — nothing to clear"
|
||||
echo "All logs under threshold — nothing to clear"
|
||||
fi
|
||||
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && exit 1
|
||||
|
||||
@@ -189,12 +189,12 @@ fi
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart rsyslog"
|
||||
else
|
||||
log "Restarting rsyslog..."
|
||||
echo "Restarting rsyslog..."
|
||||
if /etc/rc.d/rc.rsyslogd restart >/dev/null 2>&1; then
|
||||
sleep 2
|
||||
# Verify rsyslog actually running after restart
|
||||
if pgrep -x rsyslogd >/dev/null 2>&1; then
|
||||
log "rsyslog restarted and running ✅"
|
||||
echo "rsyslog restarted and running ✅"
|
||||
else
|
||||
error "rsyslog not running after restart"
|
||||
notify "rsyslog failed to start after filter update on $(hostname) ($MY_ID)" \
|
||||
@@ -224,7 +224,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
else
|
||||
log "$ICON_DONE Status: done — Docker veth noise suppressed ✅"
|
||||
echo "$ICON_DONE Status: done — Docker veth noise suppressed ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
@@ -127,7 +127,7 @@ fi
|
||||
START=$(date +%s)
|
||||
|
||||
if ! pgrep -f "emhttp.*Mover" >/dev/null 2>&1; then
|
||||
log "Mover is not running — nothing to do"
|
||||
echo "Mover is not running — nothing to do"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -137,7 +137,7 @@ warn "Mover is running (PID $MOVER_PID) — stopping in ${MOVER_STOP_TIMEOUT}s"
|
||||
# ── Warn users via wall ───────────────────────────────────────────────────────────────────────
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
wall "$ICON_WARN $MY_ID ($LOCAL_SERVER_NAME) — unRAID Mover stopping in ${MOVER_STOP_TIMEOUT}s"
|
||||
log "Wall message sent — waiting ${MOVER_STOP_TIMEOUT}s..."
|
||||
echo "Wall message sent — waiting ${MOVER_STOP_TIMEOUT}s..."
|
||||
sleep "$MOVER_STOP_TIMEOUT"
|
||||
else
|
||||
warn "DRY RUN — would send wall warning and wait ${MOVER_STOP_TIMEOUT}s"
|
||||
@@ -187,6 +187,6 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
else
|
||||
log "$ICON_DONE Status: done — mover stopped ✅"
|
||||
echo "$ICON_DONE Status: done — mover stopped ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -238,7 +238,7 @@ echo "$ICON_GEAR Config file: $PHP_CONF"
|
||||
echo "$ICON_PHP Applied: pm.max_children = $PHP_MAX_CHILDREN"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
log "$ICON_DONE Status: done ✅"
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
exit 0
|
||||
@@ -501,7 +501,7 @@ echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
else
|
||||
log "$ICON_DONE Status: done ✅"
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
@@ -121,6 +121,15 @@ acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
# ── Exit Trap — restart Docker service if reboot sequence aborts after stopping it ────────────
|
||||
_REBOOT_DOCKER_STOPPED=false
|
||||
_trap_restart_docker_service() {
|
||||
[[ "$_REBOOT_DOCKER_STOPPED" == true ]] || return
|
||||
warn "Exit trap: restarting Docker service after aborted reboot sequence"
|
||||
/etc/rc.d/rc.docker start >/dev/null 2>&1 || true
|
||||
}
|
||||
trap _trap_restart_docker_service EXIT
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made, no reboot will occur"
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -280,6 +289,7 @@ if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would stop Docker service"
|
||||
else
|
||||
if /etc/rc.d/rc.docker stop >/dev/null 2>&1; then
|
||||
_REBOOT_DOCKER_STOPPED=true
|
||||
warn "Docker stopped ✅"
|
||||
else
|
||||
warn "Docker stop returned non-zero — may already be stopped"
|
||||
@@ -317,5 +327,6 @@ if [[ "$DRY_RUN" == true ]]; then
|
||||
else
|
||||
warn "$ICON_REBOOT Rebooting $MY_ID now..."
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
trap - EXIT # committed to reboot — Docker should stay down
|
||||
/sbin/reboot
|
||||
fi
|
||||
@@ -172,7 +172,7 @@ FAILED=()
|
||||
SKIPPED=()
|
||||
|
||||
if [[ ${#PIDS[@]} -eq 0 ]]; then
|
||||
log "No User Script processes running — nothing to do"
|
||||
echo "No User Script processes running — nothing to do"
|
||||
else
|
||||
warn "${#PIDS[@]} User Script process(es) found"
|
||||
echo ""
|
||||
@@ -232,7 +232,7 @@ echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
|
||||
if [[ ${#PIDS[@]} -eq 0 ]]; then
|
||||
log "No processes were running"
|
||||
echo "No processes were running"
|
||||
elif [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would have stopped ${#SKIPPED[@]} process(es): ${SKIPPED[*]}"
|
||||
else
|
||||
@@ -248,7 +248,7 @@ elif [[ ${#FAILED[@]} -gt 0 ]]; then
|
||||
notify "User Scripts stop failed on $(hostname) ($MY_ID) — unkillable: ${FAILED[*]}" \
|
||||
"User Scripts Stop" "warning"
|
||||
else
|
||||
log "$ICON_DONE Status: done ✅"
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
|
||||
+15
-15
@@ -44,7 +44,7 @@
|
||||
# ── PATHS ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# Scripts: /mnt/user/appdata/unraid_scripts/
|
||||
# Configuration: /mnt/user/appdata/unraid_scripts/master.conf
|
||||
# Per-host: /mnt/user/appdata/unraid_scripts/master_host1.conf (or master_host2.conf)
|
||||
# Per-host: /mnt/user/appdata/unraid_scripts/host1.conf (or host2.conf)
|
||||
# Library: /mnt/user/appdata/unraid_scripts/common.sh
|
||||
#
|
||||
# ── PLUGIN SETTINGS (apply to every entry) ────────────────────────────────────────────────────
|
||||
@@ -570,7 +570,7 @@
|
||||
# docker_watchdog.sh — two-tier container self-healing monitor (single-pass)
|
||||
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
|
||||
#
|
||||
# Tier 1 — explicit per-container (configured in master_host*.conf):
|
||||
# Tier 1 — explicit per-container (configured in host*.conf):
|
||||
# Memory hard limits: immediate restart when exceeded — no strikes, no waiting
|
||||
# CPU strike system: 2 consecutive cycles above HARD_CPU_THRESHOLD → restart
|
||||
# HTTP health checks: curl to configured URL — 2 consecutive failures → restart
|
||||
@@ -589,8 +589,8 @@
|
||||
# RAM emergency: reads mem_shutdown_active from system_watchdog state file, defers all restarts
|
||||
# Silent on clean cycles — only outputs events and hourly heartbeat
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_watchdog.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/docker_watchdog.sh --dry-run
|
||||
|
||||
# watchdog_skip_list_manager.sh — view and manage the container skip list
|
||||
# When docker_watchdog restarts the same container 3 times in 1hr → skip-listed.
|
||||
@@ -631,7 +631,7 @@
|
||||
# Running → docker restart (graceful). Stopped → left stopped (state respected). Missing → skip.
|
||||
# Dependency ordering via WATCHDOG_DEPENDENCIES — databases before applications.
|
||||
# Restart verification: checks container still up after settle period, notifies if not.
|
||||
# Configured via HOST*_DAILY_RESTART_CONTAINERS (master_host*.conf):
|
||||
# Configured via HOST*_DAILY_RESTART_CONTAINERS (host*.conf):
|
||||
# NginxProxyManager, Authelia, Dispatcharr, Dispatcharr-Basic, ErsatzTV-Emby
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_daily_restart.sh --dry-run
|
||||
@@ -642,7 +642,7 @@
|
||||
# Called by weekly_sync_maintenance.sh after sync completes and containers are back up.
|
||||
# Targets services that benefit from weekly clean start but don't stop for the sync itself.
|
||||
# Same rules as daily: running→restart, stopped→leave, missing→skip.
|
||||
# Configured via HOST*_WEEKLY_RESTART_CONTAINERS (master_host*.conf):
|
||||
# Configured via HOST*_WEEKLY_RESTART_CONTAINERS (host*.conf):
|
||||
# NextCloud, AdGuard-Home, Immich
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh --dry-run
|
||||
@@ -711,7 +711,7 @@
|
||||
|
||||
# system_watchdog.sh — three-tier server stability last-resort watchdog (single-pass)
|
||||
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
|
||||
# All 18 checks independently toggleable per host in master_host*.conf.
|
||||
# All 18 checks independently toggleable per host in host*.conf.
|
||||
#
|
||||
# Tier 1 CRITICAL — bypass ALL strikes, reboot immediately:
|
||||
# Docker daemon hung → attempt rc.docker restart → still hung → reboot
|
||||
@@ -733,8 +733,8 @@
|
||||
# Reboot loop protection: N reboots in X hours → shutdown instead.
|
||||
# State file heartbeat: writes watchdog_cycle=N every cycle (docker_watchdog stale guard).
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/system_watchdog.sh --dry-run
|
||||
|
||||
# resource_watchdog.sh — three-level pressure reduction layer (single-pass, called by watchdog_orchestrator)
|
||||
# Reduces system load intelligently BEFORE docker_watchdog attempts container restarts.
|
||||
@@ -748,11 +748,11 @@
|
||||
# Recovery: pressure must stay below current threshold for RW_RECOVER_CYCLES consecutive runs
|
||||
# before de-escalating. One level at a time — prevents flip-flopping.
|
||||
# Coordination: at Level 3 writes mem_shutdown_active=true → docker_watchdog defers all restarts.
|
||||
# HOST*_RW_PAUSE_CONTAINERS and HOST*_RW_STOP_CONTAINERS configured per host in master_host*.conf.
|
||||
# HOST*_RW_PAUSE_CONTAINERS and HOST*_RW_STOP_CONTAINERS configured per host in host*.conf.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/resource_watchdog.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/resource_watchdog.sh
|
||||
|
||||
# inotify_tuning.sh — raise Linux inotify kernel limits at array start
|
||||
# Called by array_started.sh FIRST — must run before containers start (they inherit limits).
|
||||
@@ -858,7 +858,7 @@
|
||||
# Run BEFORE arr cleanup — removes non-media files that would otherwise appear as orphans.
|
||||
# Patterns: *.sfv *.md5 *.sha1 *.nfo *.url *.lnk *.rar *.zip *.info *.torrent
|
||||
# *.sample* *.proof* *sync-conflict* *.scr *.exe *.srr *.log *.json
|
||||
# Two profiles with separate folder lists (configured in master_host*.conf):
|
||||
# Two profiles with separate folder lists (configured in host*.conf):
|
||||
# anime HOST*_ANIME_CLEAN_FOLDERS — anime share folders
|
||||
# media HOST*_MEDIA_CLEAN_FOLDERS — Movies, Tv_Shows, Music, Sports etc.
|
||||
# ALWAYS --dry-run when adding new patterns or folders — verify before committing.
|
||||
@@ -1080,7 +1080,7 @@
|
||||
# Catches: renewed-but-not-reloaded (nginx not reloaded after certbot renewal),
|
||||
# wrong cert served, chain issues visible externally but not internally.
|
||||
# If a user would see a certificate error in their browser, this catches it first.
|
||||
# Configured via HOST*_CERT_MONITOR_DOMAINS in master_host*.conf.
|
||||
# Configured via HOST*_CERT_MONITOR_DOMAINS in host*.conf.
|
||||
# Thresholds: > 30 days = silent, <= 30 = warning, <= CERT_CRIT_DAYS (7) = urgent.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh --status
|
||||
|
||||
Reference in New Issue
Block a user