844 lines
29 KiB
Markdown
844 lines
29 KiB
Markdown
# ━━━━━ WATCHDOGS — Manual ━━━━━
|
||
|
||
Configuration reference, operational procedures, and troubleshooting for all watchdog
|
||
scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
|
||
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
|
||
|
||
---
|
||
|
||
## ━━━ CONTENTS ━━━
|
||
|
||
- [Output Tiers](#output-tiers)
|
||
- [resource_watchdog.sh](#resource_watchdogsh)
|
||
- [docker_watchdog.sh](#docker_watchdogsh)
|
||
- [system_watchdog.sh](#system_watchdogsh)
|
||
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) ← Plugin/unraid/Watchdogs/System/
|
||
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
|
||
- [System/conf_cache_watchdog.sh](#systemconf_cache_watchdogsh)
|
||
- [stability_watchdog.sh](#stability_watchdogsh)
|
||
- [Full Configuration Reference](#full-configuration-reference)
|
||
- [Troubleshooting](#troubleshooting)
|
||
|
||
---
|
||
|
||
## Output Tiers
|
||
|
||
All watchdog scripts use a two-tier output model: `echo` lines are always visible;
|
||
`log` lines only appear when `--log` is passed.
|
||
|
||
All watchdog scripts are **single-pass scripts** called once every 15 minutes by the orchestrator.
|
||
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
|
||
visible. Per-check detail is suppressed on clean cycles.
|
||
|
||
`docker_watchdog.sh` is the exception — it is **silent on clean cycles by design**.
|
||
96 cycles/day means clean-cycle noise would bury real events. Its output only appears
|
||
when there are restarts, skip-list events, or RAM deferral. Use `--log` to see
|
||
per-cycle detail on clean cycles.
|
||
|
||
---
|
||
|
||
## resource_watchdog.sh
|
||
|
||
Runs first in the orchestrator sequence. Reduces system pressure before docker_watchdog
|
||
attempts any container restarts. Containers restarted into a RAM-pressured system just
|
||
fail again — this script ensures docker_watchdog has breathing room.
|
||
|
||
### Pressure Levels
|
||
|
||
Three escalating levels, each additive:
|
||
|
||
```bash
|
||
# master.conf
|
||
RW_RAM_SOFT_GB=20 # Level 1 trigger — throttle downloaders
|
||
RW_RAM_MEDIUM_GB=15 # Level 2 trigger — throttle + pause containers
|
||
RW_RAM_HARD_GB=10 # Level 3 trigger — stop containers + defer docker_watchdog
|
||
RW_RAM_RECOVER_GB=25 # de-escalate only after RAM reaches this
|
||
|
||
RW_LOAD_SOFT_MULTIPLIER=2.0 # load > 2× cpu count = level 1
|
||
RW_LOAD_MEDIUM_MULTIPLIER=3.0 # load > 3× cpu count = level 2
|
||
|
||
RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalating
|
||
```
|
||
|
||
**Level 1 (soft):** Throttle SABnzbd + qBittorrent download speeds.
|
||
**Level 2 (medium):** Further throttle + `docker pause` non-critical containers.
|
||
**Level 3 (hard):** `docker stop` optional services + write `mem_shutdown_active=true`
|
||
to `RW_STATE_FILE`. docker_watchdog.sh reads this flag and skips all restart logic
|
||
until pressure clears. Without this coordination, docker_watchdog would immediately
|
||
restart containers that resource_watchdog just stopped to free RAM.
|
||
|
||
Recovery de-escalates one level at a time — prevents flip-flopping between states.
|
||
|
||
### Per-Host Container Lists
|
||
|
||
```bash
|
||
# host1.conf
|
||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at level 2
|
||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at level 3
|
||
```
|
||
|
||
Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
|
||
pressure level. Default: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
|
||
|
||
### Downloader Throttle Config
|
||
|
||
```bash
|
||
# master.conf
|
||
RW_SABNZBD_ENABLED=true
|
||
RW_SABNZBD_SPEED_SOFT="50M" # throttled at level 1
|
||
RW_SABNZBD_SPEED_MEDIUM="10M" # throttled further at level 2
|
||
|
||
RW_QBIT_ENABLED=true
|
||
RW_QBIT_DL_SOFT=51200 # KB/s — level 1
|
||
RW_QBIT_DL_MEDIUM=10240 # KB/s — level 2
|
||
|
||
# host1.conf (API access)
|
||
HOST1_SABNZBD_URL="http://localhost:8080"
|
||
HOST1_SABNZBD_API_KEY="your-api-key"
|
||
HOST1_QBIT_URL="http://localhost:8090"
|
||
HOST1_QBIT_USERNAME="admin"
|
||
HOST1_QBIT_PASSWORD="your-password"
|
||
```
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
resource_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||
resource_watchdog.sh --dry-run # show what would be throttled/paused/stopped
|
||
resource_watchdog.sh --status # current level, active actions, recovery cycle count
|
||
resource_watchdog.sh --log # verbose per-check output
|
||
```
|
||
|
||
---
|
||
|
||
## docker_watchdog.sh
|
||
|
||
Runs second in the orchestrator sequence. Two-tier container healing — explicit
|
||
per-container configuration (Tier 1) plus a global catch-all scan (Tier 2).
|
||
|
||
Reads `RW_STATE_FILE` at cycle start — if `mem_shutdown_active=true`, skips all
|
||
container restart logic (resource_watchdog is managing the situation). Health URL
|
||
checks for excluded containers still run.
|
||
|
||
### Memory Hard Limits
|
||
|
||
```bash
|
||
# host1.conf
|
||
# Format: "ContainerName:LimitInMB"
|
||
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
|
||
#
|
||
# Sizing: check normal peak with "docker stats ContainerName"
|
||
# Set limit at ~150-200% of normal peak
|
||
#
|
||
HOST1_WATCHDOG_CONTAINERS=(
|
||
"Emby:18432" # 18GB — peaks ~12GB under heavy transcode load
|
||
"LidaTube:6144" # 6GB — YouTube downloader, grows with large queues
|
||
"Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature
|
||
"Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous
|
||
)
|
||
```
|
||
|
||
A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
|
||
visibility into a container approaching its ceiling before a restart is triggered.
|
||
|
||
### CPU Thresholds
|
||
|
||
```bash
|
||
# master.conf
|
||
# CPU is normalised against total core count.
|
||
# 85% normalised on a 16-core machine = 13.6 cores worth of a single process.
|
||
#
|
||
# CPU uses a STRIKE SYSTEM — brief spikes are normal (Tdarr, Emby transcoding, SABnzbd).
|
||
# Strike 1: above HARD_CPU_THRESHOLD → warn, increment strike
|
||
# Strike 2: above threshold → restart, reset counter
|
||
# Recovery: drops below threshold any cycle → reset to 0
|
||
#
|
||
SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only
|
||
HARD_CPU_THRESHOLD=85 # strike at 85% normalised
|
||
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
|
||
```
|
||
|
||
### HTTP Health Checks
|
||
|
||
```bash
|
||
# host1.conf
|
||
# Format: "ContainerName:http://host:port/optional-path"
|
||
# Two consecutive non-responses trigger a restart.
|
||
# "Container running" and "service responding" are not the same thing.
|
||
#
|
||
HOST1_WATCHDOG_CONTAINER_URLS=(
|
||
"Emby:http://localhost:8096" # Emby WebUI root
|
||
"NginxProxyManager:http://localhost:81" # NPM admin interface
|
||
)
|
||
|
||
# master.conf
|
||
CURL_TIMEOUT=5 # seconds before non-response counts as a failure
|
||
RESP_FAIL_LIMIT=2 # consecutive failures before restart
|
||
```
|
||
|
||
### Required Containers
|
||
|
||
```bash
|
||
# host1.conf
|
||
# Found stopped → restart attempted every cycle until running or skip-listed.
|
||
#
|
||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
||
"NginxProxyManager"
|
||
"Authelia"
|
||
"Mariadb-Authelia"
|
||
"Redis-Authelia"
|
||
)
|
||
```
|
||
|
||
### Dependency Ordering
|
||
|
||
```bash
|
||
# host1.conf
|
||
# Format: "DependentContainer:dependency1 dependency2"
|
||
# All dependencies must be running before the dependent is restarted.
|
||
# Prevents Authelia crash-looping while MariaDB is still starting.
|
||
#
|
||
HOST1_WATCHDOG_DEPENDENCIES=(
|
||
"Authelia:Mariadb-Authelia Redis-Authelia"
|
||
"NextCloud:Postgres-NextCloud"
|
||
)
|
||
```
|
||
|
||
Same dependency config is used by `docker_daily_restart.sh` and
|
||
`docker_weekly_restart.sh` — configure once, applies everywhere.
|
||
|
||
### Startup Grace Period
|
||
|
||
```bash
|
||
# master.conf
|
||
# Suppress restart actions for N seconds after array start.
|
||
# Checks still run and log — only restart actions are suppressed.
|
||
#
|
||
WATCHDOG_STARTUP_GRACE=600 # 10 minutes
|
||
```
|
||
|
||
### Tier 2 Global Scan
|
||
|
||
```bash
|
||
# master.conf
|
||
WATCHDOG_SCAN_ALL=true # enable global scan
|
||
WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures
|
||
WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills
|
||
WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts
|
||
WATCHDOG_RESTART_DEAD=true # recover containers in dead state
|
||
WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero
|
||
WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list
|
||
|
||
# Exclude containers from Tier 2 (one-shots, manually managed, benign exits):
|
||
WATCHDOG_SCAN_IGNORE=(
|
||
"my-one-shot-container"
|
||
)
|
||
```
|
||
|
||
### Restart Loop Protection
|
||
|
||
```bash
|
||
# master.conf
|
||
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list
|
||
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
|
||
```
|
||
|
||
After hitting the limit: skip list + critical notification. The watchdog stops
|
||
touching the container. Auto-clear: if the container recovers on its own and is
|
||
found running, it's removed from the skip list automatically. Manual clear is only
|
||
needed when the container is stuck stopped — use `Tools/watchdog_skip_list_manager.sh`.
|
||
|
||
### Notification Batching
|
||
|
||
```bash
|
||
# master.conf
|
||
WATCHDOG_BATCH_NOTIFY=true
|
||
# All events from one cycle → one notification at the end.
|
||
# A shared DB going down can cascade 10+ containers. Without batching: 10 pings.
|
||
# With batching: one summary listing all affected containers.
|
||
```
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
docker_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||
docker_watchdog.sh --dry-run # full cycle preview without restarting anything
|
||
docker_watchdog.sh --status # skip list, strike counts, grace period, RAM deferral state
|
||
docker_watchdog.sh --log # verbose per-cycle output
|
||
```
|
||
|
||
### Skip List Recovery
|
||
|
||
```bash
|
||
# Step 1 — understand the situation
|
||
Tools/watchdog_skip_list_manager.sh --status
|
||
|
||
# Step 2 — fix the underlying problem
|
||
# docker logs ContainerName --tail 100
|
||
# df -h /mnt/user
|
||
|
||
# Step 3 — clear the container
|
||
Tools/watchdog_skip_list_manager.sh --clear ContainerName
|
||
|
||
# Step 4 — start manually (confirms fix before handing back to watchdog)
|
||
docker start ContainerName
|
||
|
||
# Step 5 — monitoring resumes automatically on next cycle
|
||
```
|
||
|
||
---
|
||
|
||
## system_watchdog.sh
|
||
|
||
Thin orchestrator — runs `SYSTEM_WATCHDOG_SCRIPTS` from master.conf sequentially each cycle.
|
||
Called third by `watchdog_orchestrator.sh`. Covers all system component watchdogs.
|
||
Can also be run standalone to check all system components at once.
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
system_watchdog.sh # run all system component watchdogs
|
||
system_watchdog.sh --status # show configured scripts and their paths
|
||
system_watchdog.sh --dry-run # preview without executing anything
|
||
system_watchdog.sh --log # verbose output
|
||
```
|
||
|
||
---
|
||
|
||
## System/storage_watchdog.sh
|
||
|
||
Called by `system_watchdog.sh` each cycle. Two independent checks per cycle:
|
||
growth rate detection (automatic, zero config) and oversize log detection.
|
||
Uses its own strike state file — independent from docker_watchdog.
|
||
|
||
### Growth Rate Detection
|
||
|
||
```bash
|
||
# master.conf
|
||
WATCHDOG_CHECK_APPDATA=true
|
||
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
|
||
WATCHDOG_APPDATA_GROWTH_GB=2 # growth per cycle that triggers a strike
|
||
WATCHDOG_APPDATA_STRIKE_LIMIT=3 # strikes before alert
|
||
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db" # size baseline
|
||
```
|
||
|
||
Runs `du -sm appdata/*/` each cycle — pure inode metadata, very lightweight on NVMe.
|
||
Compares each container's current size to the baseline from the previous cycle.
|
||
Growth > `WATCHDOG_APPDATA_GROWTH_GB` per cycle increments the container's strike count.
|
||
Strike 1: warn. Strike 2: escalate. Strike 3: critical alert.
|
||
Strikes auto-clear when growth drops to zero (condition resolved).
|
||
|
||
**Zero configuration required for new containers.** Growth rate detection covers all
|
||
containers automatically. The suppress array below is only for known-legitimate growth.
|
||
|
||
### Growth Suppress Ceilings
|
||
|
||
```bash
|
||
# host1.conf
|
||
# ONLY needed in specific cases — growth rate detection covers everything automatically.
|
||
# Use when a container's appdata legitimately grows fast during normal operation
|
||
# and you want to suppress false positives above a known-safe threshold.
|
||
#
|
||
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||
["Tdarr"]="25600" # 25GB — transcode cache grows during active jobs
|
||
["7dtd"]="20480" # 20GB — game server world data, expected large
|
||
)
|
||
```
|
||
|
||
### Log File Detection
|
||
|
||
```bash
|
||
# master.conf
|
||
WATCHDOG_APPDATA_LOG_MAX_GB=2 # *.log / *.log.* files above this trigger a strike
|
||
WATCHDOG_APPDATA_TRUNCATE_LOGS=false # true: truncate at strike limit; false: alert only
|
||
```
|
||
|
||
Scans all `*.log` and `*.log.*` files across all appdata paths. Files over the threshold
|
||
increment per-file strike counts. At strike limit: truncates in-place with `truncate -s 0`
|
||
(container keeps its file handle — space reclaimed immediately without container restart)
|
||
or sends a critical alert if truncation is disabled.
|
||
|
||
Log strikes auto-clear when the file drops below threshold.
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
storage_watchdog.sh # single pass (called by system_watchdog.sh)
|
||
storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings
|
||
storage_watchdog.sh --dry-run # show what would be alerted/truncated
|
||
storage_watchdog.sh --log # verbose per-container output
|
||
```
|
||
|
||
---
|
||
|
||
## System/webgui_watchdog.sh
|
||
|
||
> **Lives in `Plugin/unraid/Watchdogs/System/webgui_watchdog.sh`** — calls Unraid-specific
|
||
> service commands (`rc.nginx`, `rc.php-fpm`, `emhttp`) via the platform adapter. Called by
|
||
> `system_watchdog.sh` via `SYSTEM_WATCHDOG_SCRIPTS` in master.conf.
|
||
|
||
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
|
||
through three restart steps if unresponsive. Silent when healthy.
|
||
|
||
### Escalation Path
|
||
|
||
```
|
||
WebGUI responding → exit 0 (silent)
|
||
|
||
Not responding:
|
||
Step 1 — nginx restart → wait WEBGUI_NGINX_WAIT → recheck
|
||
Step 2 — php-fpm restart → wait WEBGUI_PHP_WAIT → recheck
|
||
Step 3 — emhttp restart → wait WEBGUI_EMHTTP_WAIT → recheck
|
||
All three failed → critical notify, manual intervention needed
|
||
```
|
||
|
||
### Configuration
|
||
|
||
```bash
|
||
# master.conf
|
||
WEBGUI_URL="http://localhost"
|
||
WEBGUI_TIMEOUT=5
|
||
WEBGUI_NGINX_WAIT=15
|
||
WEBGUI_PHP_WAIT=10
|
||
WEBGUI_EMHTTP_WAIT=30
|
||
```
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
webgui_watchdog.sh # single pass (called by system_watchdog.sh)
|
||
webgui_watchdog.sh --status # current WebGUI state + nginx/php-fpm/emhttp status
|
||
webgui_watchdog.sh --dry-run # show which services would be restarted
|
||
webgui_watchdog.sh --log # verbose per-step output
|
||
```
|
||
|
||
---
|
||
|
||
## System/network_watchdog.sh
|
||
|
||
Called by `system_watchdog.sh` each cycle. Checks that the outside world can actually
|
||
reach what it needs to reach. Internet reachability gates all other checks — if upstream
|
||
is down, DDNS and NPM checks are skipped to prevent false positives.
|
||
|
||
### Check Sequence
|
||
|
||
```
|
||
1. Internet → curl NETWORK_WATCHDOG_INTERNET_URL
|
||
fail → alert + exit (skip all remaining checks)
|
||
|
||
2. DDNS → public IP (ifconfig.me) vs DNS record (dig @1.1.1.1)
|
||
match → pass (silent)
|
||
mismatch → restart HOST*_NETWORK_WATCHDOG_DDNS_CONTAINER + notify
|
||
|
||
3. Tailscale → tailscale status --json BackendState
|
||
Running → pass (silent)
|
||
not Running → notify (no auto-restart — warrants human review)
|
||
|
||
4. NPM proxy → curl HOST*_NETWORK_WATCHDOG_NPM_URL (external)
|
||
reachable → pass, clear strikes
|
||
not reachable → strike 1: warn + notify
|
||
strike 2: restart NginxProxyManager + notify + clear strikes
|
||
```
|
||
|
||
### Configuration
|
||
|
||
```bash
|
||
# master.conf
|
||
NETWORK_WATCHDOG_ENABLED=true
|
||
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
|
||
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
|
||
|
||
# host*.conf (host-specific)
|
||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||
```
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
network_watchdog.sh # run all connectivity checks (silent when healthy)
|
||
network_watchdog.sh --status # current IP, DNS record, Tailscale state, NPM strike count
|
||
network_watchdog.sh --dry-run # check without restarting any containers
|
||
network_watchdog.sh --log # verbose per-check output
|
||
```
|
||
|
||
### Troubleshooting
|
||
|
||
**DDNS keeps restarting the container but record stays stale**
|
||
```bash
|
||
# Check if the container is actually running after restart:
|
||
docker ps | grep "Gmer4Lfe.com"
|
||
|
||
# Check container logs for Cloudflare API errors:
|
||
docker logs "Gmer4Lfe.com" --tail 20
|
||
|
||
# Verify public IP detection:
|
||
curl -sf https://ifconfig.me
|
||
|
||
# Verify DNS resolution:
|
||
dig +short gmer4lfe.com @1.1.1.1
|
||
```
|
||
|
||
**NPM strikes accumulating but NPM is running**
|
||
```bash
|
||
# Check if the external URL is actually responding:
|
||
curl -sv https://gmer4lfe.com 2>&1 | head -20
|
||
|
||
# NPM may be running but a backend container is down — check the specific service
|
||
# the URL routes to, not just NginxProxyManager itself.
|
||
|
||
# Check NPM strike count:
|
||
network_watchdog.sh --status
|
||
```
|
||
|
||
**Tailscale showing not Running**
|
||
```bash
|
||
# Check tailscale state directly:
|
||
tailscale status
|
||
|
||
# Check the backend state specifically:
|
||
tailscale status --json | grep BackendState
|
||
|
||
# Reconnect manually if needed:
|
||
tailscale up
|
||
```
|
||
|
||
---
|
||
|
||
## System/conf_cache_watchdog.sh
|
||
|
||
Called by `system_watchdog.sh` each cycle. Maintains the persistent partner conf backup
|
||
at `$PERSISTENT_CONF_CACHE` while the partner is offline.
|
||
|
||
### Behaviour
|
||
|
||
**Remote online:** removes the persistent backup if one exists. It is not needed —
|
||
`conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
|
||
|
||
**Remote offline:** copies partner confs from the RAM cache (`$CONF_RAM_CACHE_DIR`, `/tmp/varaverk/conf/`) to
|
||
`$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
|
||
an extended outage. If this host reboots while the partner is still down,
|
||
`conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
|
||
valid partner vars.
|
||
|
||
Silent when remote is online and no backup exists (the normal steady state).
|
||
|
||
### Gates
|
||
|
||
- `FALLBACK_ENABLED=false` → no-op (no fallback means no need for partner vars)
|
||
- `CONF_SYNC_ENABLED=false` → no-op
|
||
- `PARTNERSHIP_ENABLED=false` → exits silently (require_partnership gate)
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
conf_cache_watchdog.sh # single pass (called by system_watchdog.sh)
|
||
conf_cache_watchdog.sh --dry-run # show what would be written or removed
|
||
conf_cache_watchdog.sh --log # verbose output
|
||
```
|
||
|
||
---
|
||
|
||
## stability_watchdog.sh
|
||
|
||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||
to reboot. Watches the server itself — not containers, not storage. Reboots only when
|
||
healing at every other layer has failed or when the failure is non-recoverable.
|
||
|
||
### Three-Tier Response
|
||
|
||
**Tier 1 — CRITICAL (immediate reboot, no strikes)**
|
||
| Condition | Threshold | Why immediate |
|
||
|-----------|-----------|---------------|
|
||
| Docker daemon unresponsive | N/A | Every docker command hangs — nothing can be healed |
|
||
| rootfs usage | `SYS_WATCHDOG_ROOTFS_CRITICAL_PCT` (99%) | SSH stops; state files fail silently |
|
||
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
|
||
| File descriptor exhaustion | `SYS_WATCHDOG_FD_CRITICAL_PCT` (95%) | New connections silently failing |
|
||
| /boot read-only | write test fails | Config writes silently failing |
|
||
|
||
**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
|
||
RAM below `SYS_WATCHDOG_MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`.
|
||
Both conditions required — RAM alone uses the standard strike system.
|
||
OOM confirms the system is dying faster than watchdogs can heal.
|
||
|
||
**Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKE_LIMIT` consecutive failures → reboot)**
|
||
| Check | Threshold |
|
||
|-------|-----------|
|
||
| Free RAM | `SYS_WATCHDOG_MEM_GB` (reboot trigger — earlier tiers handled by resource_watchdog) |
|
||
| Load average | `SYS_WATCHDOG_LOAD_MULTIPLIER` × cpu_count |
|
||
| CPU temperature | `SYS_WATCHDOG_CPU_TEMP_MAX` |
|
||
| Zombie processes | `SYS_WATCHDOG_ZOMBIE_LIMIT` |
|
||
| /var/log usage | `SYS_WATCHDOG_LOG_PCT` |
|
||
| /tmp usage | `SYS_WATCHDOG_TMP_PCT` |
|
||
| Array disk errors | mdstat error delta > 0 |
|
||
| NIC state | interface operstate != "up" |
|
||
|
||
### RAM Tiers
|
||
|
||
RAM pressure is a graduated response split across resource_watchdog and stability_watchdog:
|
||
|
||
```
|
||
RW_RAM_SOFT_GB (12GB) → throttle downloads, reduce background load (resource_watchdog)
|
||
RW_RAM_MEDIUM_GB (8GB) → pause background containers (resource_watchdog)
|
||
RW_RAM_HARD_GB (6GB) → stop optional containers, wait for recovery (resource_watchdog)
|
||
SYS_WATCHDOG_MEM_GB (4GB) → strike → reboot (last resort) (stability_watchdog)
|
||
RW_RAM_RECOVER_GB (20GB) → RAM must reach this before stopped containers restart
|
||
```
|
||
|
||
`SYS_WATCHDOG_MEM_GB` must always be below `RW_RAM_HARD_GB` — resource_watchdog acts first.
|
||
Containers stopped at the hard tier use `HOST*_RW_STOP_CONTAINERS` in host*.conf.
|
||
Containers paused at the medium tier use `HOST*_RW_PAUSE_CONTAINERS` in host*.conf.
|
||
|
||
### Abort Conditions
|
||
|
||
Prevent reboot — running them would risk data loss:
|
||
|
||
```bash
|
||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
|
||
SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity mid-check is worse than crashing
|
||
SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting move mid-run is worse than crashing
|
||
```
|
||
|
||
Tier 1 CRITICAL bypasses all abort conditions — an imminent crash outweighs data
|
||
safety concerns.
|
||
|
||
### Reboot Rate Limit
|
||
|
||
```bash
|
||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=12 # window in hours
|
||
SYS_WATCHDOG_REBOOT_LIMIT=3 # max reboots within the window
|
||
```
|
||
|
||
If the server reboots `SYS_WATCHDOG_REBOOT_LIMIT` times within the window, the watchdog
|
||
switches from rebooting to notifying only. Prevents a boot loop where the watchdog
|
||
reboots → something crashes again immediately → reboot again.
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
stability_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||
stability_watchdog.sh --dry-run # run detection logic without rebooting
|
||
stability_watchdog.sh --status # thresholds, current state, strike counts
|
||
stability_watchdog.sh --log # verbose per-check output
|
||
```
|
||
|
||
---
|
||
|
||
## Full Configuration Reference
|
||
|
||
```bash
|
||
# master.conf
|
||
|
||
# ── Resource Watchdog ──────────────────────────────────────────────────────────
|
||
RW_ENABLED=true
|
||
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
|
||
|
||
RW_RAM_SOFT_GB=20
|
||
RW_RAM_MEDIUM_GB=15
|
||
RW_RAM_HARD_GB=10
|
||
RW_RAM_RECOVER_GB=25
|
||
|
||
RW_LOAD_SOFT_MULTIPLIER=2.0
|
||
RW_LOAD_MEDIUM_MULTIPLIER=3.0
|
||
RW_RECOVER_CYCLES=3
|
||
|
||
RW_SABNZBD_ENABLED=true
|
||
RW_SABNZBD_SPEED_SOFT="50M"
|
||
RW_SABNZBD_SPEED_MEDIUM="10M"
|
||
RW_QBIT_ENABLED=true
|
||
RW_QBIT_DL_SOFT=51200 # KB/s
|
||
RW_QBIT_DL_MEDIUM=10240
|
||
|
||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
|
||
|
||
# host*.conf
|
||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
|
||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
|
||
HOST1_SABNZBD_URL="http://localhost:8080"
|
||
HOST1_SABNZBD_API_KEY="your-api-key"
|
||
HOST1_QBIT_URL="http://localhost:8090"
|
||
HOST1_QBIT_USERNAME="admin"
|
||
HOST1_QBIT_PASSWORD="your-password"
|
||
|
||
# ── Docker Watchdog ────────────────────────────────────────────────────────────
|
||
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
|
||
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
|
||
|
||
SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart)
|
||
|
||
SOFT_CPU_THRESHOLD=50
|
||
HARD_CPU_THRESHOLD=85
|
||
CPU_FAIL_LIMIT=2
|
||
|
||
CURL_TIMEOUT=5
|
||
RESP_FAIL_LIMIT=2
|
||
|
||
WATCHDOG_CONTAINER_RESTART_LIMIT=3
|
||
WATCHDOG_CONTAINER_RESTART_WINDOW=1
|
||
|
||
WATCHDOG_SCAN_ALL=true
|
||
WATCHDOG_SCAN_IGNORE=()
|
||
WATCHDOG_RESTART_UNHEALTHY=true
|
||
WATCHDOG_NOTIFY_OOM=true
|
||
WATCHDOG_NOTIFY_CRASHLOOP=true
|
||
WATCHDOG_CRASH_LIMIT=5
|
||
WATCHDOG_RESTART_DEAD=true
|
||
WATCHDOG_RESTART_CRASHED=true
|
||
WATCHDOG_BATCH_NOTIFY=true
|
||
|
||
# State files (all paths adapt to storage mode via STATE_DIR / DATA_DIR):
|
||
WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
|
||
WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"
|
||
|
||
# host*.conf
|
||
HOST1_WATCHDOG_CONTAINERS=() # "ContainerName:LimitMB"
|
||
HOST1_WATCHDOG_CONTAINER_URLS=() # "ContainerName:http://host:port"
|
||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
|
||
HOST1_WATCHDOG_DEPENDENCIES=() # "Dependent:dep1 dep2"
|
||
|
||
# ── Storage Watchdog ───────────────────────────────────────────────────────────
|
||
WATCHDOG_CHECK_APPDATA=true
|
||
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
|
||
WATCHDOG_APPDATA_GROWTH_GB=2
|
||
WATCHDOG_APPDATA_LOG_MAX_GB=2
|
||
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
|
||
WATCHDOG_APPDATA_STRIKE_LIMIT=3
|
||
WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
|
||
STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"
|
||
|
||
# host*.conf (optional — only for suppress ceilings)
|
||
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||
# ["Tdarr"]="25600"
|
||
# )
|
||
|
||
# ── System Watchdog ────────────────────────────────────────────────────────────
|
||
# Reboot trigger only — warn/shutdown/recover RAM tiers handled by resource_watchdog.sh
|
||
# RW_RAM_HARD_GB > SYS_WATCHDOG_MEM_GB always (resource_watchdog acts before watchdog reboots)
|
||
SYS_WATCHDOG_STRIKE_LIMIT=2
|
||
|
||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=12
|
||
SYS_WATCHDOG_REBOOT_LIMIT=3
|
||
SYS_WATCHDOG_OOM_LIMIT=3
|
||
|
||
SYS_WATCHDOG_MEM_GB=4 # strike system → reboot (last resort — below resource_watchdog hard stop)
|
||
|
||
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99
|
||
SYS_WATCHDOG_FD_CRITICAL_PCT=95
|
||
SYS_WATCHDOG_LOAD_MULTIPLIER=4
|
||
SYS_WATCHDOG_CPU_TEMP_MAX=95
|
||
SYS_WATCHDOG_ZOMBIE_LIMIT=50
|
||
SYS_WATCHDOG_LOG_PCT=95
|
||
SYS_WATCHDOG_TMP_PCT=90
|
||
|
||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
|
||
SYS_WATCHDOG_ABORT_ON_PARITY=false
|
||
SYS_WATCHDOG_ABORT_ON_MOVER=false
|
||
|
||
# State files — all in $STATE_DIR (survive reboots; adapt to storage mode):
|
||
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
|
||
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
|
||
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
|
||
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
|
||
RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"
|
||
|
||
# ── Network Watchdog ───────────────────────────────────────────────────────────
|
||
NETWORK_WATCHDOG_ENABLED=true
|
||
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
|
||
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"
|
||
|
||
# host*.conf (host-specific)
|
||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||
```
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### resource_watchdog Paused Containers It Shouldn't
|
||
|
||
```bash
|
||
# Check current state:
|
||
resource_watchdog.sh --status
|
||
|
||
# Add to RW_CRITICAL_CONTAINERS in master.conf:
|
||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
|
||
|
||
# Un-pause manually if needed:
|
||
docker unpause MyContainer
|
||
```
|
||
|
||
### docker_watchdog Keeps Restarting a Healthy Container
|
||
|
||
```bash
|
||
# Check what's triggering it — memory, CPU, HTTP, or required:
|
||
docker_watchdog.sh --status
|
||
|
||
# Check CPU normalised — brief spikes should not trigger (2-strike system):
|
||
# If triggering on CPU: check if HARD_CPU_THRESHOLD is set appropriately
|
||
# for containers with legitimate burst usage (Tdarr encoding, SABnzbd unpacking)
|
||
|
||
# Check HTTP — is the health endpoint returning 200?
|
||
curl -sf --max-time 5 http://localhost:PORT && echo "OK" || echo "FAIL"
|
||
```
|
||
|
||
### Container on the Skip List After Fixing the Problem
|
||
|
||
```bash
|
||
# See skip list and container state:
|
||
Tools/watchdog_skip_list_manager.sh --status
|
||
|
||
# Fix root cause first, then clear:
|
||
Tools/watchdog_skip_list_manager.sh --clear ContainerName
|
||
|
||
# Start manually to confirm fix before handing back to watchdog:
|
||
docker start ContainerName
|
||
```
|
||
|
||
### storage_watchdog Alerting on a Container That Grows Legitimately
|
||
|
||
```bash
|
||
# Check what triggered it:
|
||
storage_watchdog.sh --status
|
||
|
||
# Add a suppress ceiling to host*.conf:
|
||
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||
# ["ContainerName"]="10240" # 10GB ceiling — legitimate growth, suppress below this
|
||
# )
|
||
```
|
||
|
||
### stability_watchdog Rebooted Unexpectedly
|
||
|
||
```bash
|
||
# Check the reboot log (survives reboots — in data/state/):
|
||
cat /boot/config/plugins/varaverk/data/state/system_watchdog_reboots.db
|
||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||
|
||
# Check syslog near the reboot time:
|
||
grep "stability_watchdog" /var/log/syslog | tail -20
|
||
```
|
||
|
||
### stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
|
||
|
||
```bash
|
||
# All watchdogs run as single-pass scripts — there is no background process to check.
|
||
# If the orchestrator reports a timeout, one pass took longer than expected.
|
||
|
||
# Check the orchestrator itself:
|
||
Orchestrators/watchdog_orchestrator.sh --status
|
||
|
||
# Run the slow watchdog directly with --log to see where it's hanging:
|
||
Watchdogs/stability_watchdog.sh --log --dry-run
|
||
```
|