Varaverk/Watchdogs/Manual-Watchdogs.md

━━━━━ WATCHDOGS — Manual ━━━━━

Configuration reference, operational procedures, and troubleshooting for all four watchdog scripts. For design philosophy and script relationships see README-Watchdogs.md. For the orchestrator that calls these scripts see Orchestrators/watchdog_orchestrator.sh.

---

━━━ CONTENTS ━━━

• Output Tiers
• resource_watchdog.sh
• docker_watchdog.sh
• storage_watchdog.sh
• system_watchdog.sh
• Full Configuration Reference
• Troubleshooting

---

Output Tiers

All watchdog scripts use a two-tier output model: echo lines are always visible; log lines only appear when --log is passed.

All four watchdogs are single-pass scripts called once per minute by the orchestrator. Without --log, only state transitions, warnings, errors, and the conclusion line are visible. Per-check detail is suppressed on clean cycles.

docker_watchdog.sh is the exception — it is silent on clean cycles by design. 96 cycles/day means clean-cycle noise would bury real events. Its output only appears when there are restarts, skip-list events, or RAM deferral. Use --log to see per-cycle detail on clean cycles.

---

resource_watchdog.sh

Runs first in the orchestrator sequence. Reduces system pressure before docker_watchdog attempts any container restarts. Containers restarted into a RAM-pressured system just fail again — this script ensures docker_watchdog has breathing room.

Pressure Levels

Three escalating levels, each additive:

# master.conf
RW_RAM_SOFT_GB=20           # Level 1 trigger — throttle downloaders
RW_RAM_MEDIUM_GB=15         # Level 2 trigger — throttle + pause containers
RW_RAM_HARD_GB=10           # Level 3 trigger — stop containers + defer docker_watchdog
RW_RAM_RECOVER_GB=25        # de-escalate only after RAM reaches this

RW_LOAD_SOFT_MULTIPLIER=2.0    # load > 2× cpu count = level 1
RW_LOAD_MEDIUM_MULTIPLIER=3.0  # load > 3× cpu count = level 2

RW_RECOVER_CYCLES=3         # consecutive under-threshold runs before de-escalating

Level 1 (soft): Throttle SABnzbd + qBittorrent download speeds. Level 2 (medium): Further throttle + docker pause non-critical containers. Level 3 (hard): docker stop optional services + write mem_shutdown_active=true to RW_STATE_FILE. docker_watchdog.sh reads this flag and skips all restart logic until pressure clears. Without this coordination, docker_watchdog would immediately restart containers that resource_watchdog just stopped to free RAM.

Recovery de-escalates one level at a time — prevents flip-flopping between states.

Per-Host Container Lists

# host1.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")        # paused at level 2
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")    # stopped at level 3

Containers in RW_CRITICAL_CONTAINERS are never paused or stopped regardless of pressure level. Default: Emby, NginxProxyManager, Authelia, Mariadb, Redis.

Downloader Throttle Config

# master.conf
RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"        # throttled at level 1
RW_SABNZBD_SPEED_MEDIUM="10M"      # throttled further at level 2

RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200              # KB/s — level 1
RW_QBIT_DL_MEDIUM=10240            # KB/s — level 2

# host1.conf (API access)
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"

Usage

resource_watchdog.sh            # single pass (called by watchdog_orchestrator.sh)
resource_watchdog.sh --dry-run  # show what would be throttled/paused/stopped
resource_watchdog.sh --status   # current level, active actions, recovery cycle count
resource_watchdog.sh --log      # verbose per-check output

---

docker_watchdog.sh

Runs second in the orchestrator sequence. Two-tier container healing — explicit per-container configuration (Tier 1) plus a global catch-all scan (Tier 2).

Reads RW_STATE_FILE at cycle start — if mem_shutdown_active=true, skips all container restart logic (resource_watchdog is managing the situation). Health URL checks for excluded containers still run.

Memory Hard Limits

# host1.conf
# Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
#
# Sizing: check normal peak with "docker stats ContainerName"
# Set limit at ~150-200% of normal peak
#
HOST1_WATCHDOG_CONTAINERS=(
    "Emby:18432"        # 18GB — peaks ~12GB under heavy transcode load
    "LidaTube:6144"     # 6GB  — YouTube downloader, grows with large queues
    "Tdarr:6144"        # 6GB  — video transcoder, memory-intensive by nature
    "Code-Server:1024"  # 1GB  — IDE, should be light; 1GB is generous
)

A soft warning fires at SOFT_MEM_THRESHOLD=80 percent of the hard limit — early visibility into a container approaching its ceiling before a restart is triggered.

CPU Thresholds

# master.conf
# CPU is normalised against total core count.
# 85% normalised on a 16-core machine = 13.6 cores worth of a single process.
#
# CPU uses a STRIKE SYSTEM — brief spikes are normal (Tdarr, Emby transcoding, SABnzbd).
# Strike 1: above HARD_CPU_THRESHOLD → warn, increment strike
# Strike 2: above threshold → restart, reset counter
# Recovery: drops below threshold any cycle → reset to 0
#
SOFT_CPU_THRESHOLD=50    # warn at 50% normalised — informational only
HARD_CPU_THRESHOLD=85    # strike at 85% normalised
CPU_FAIL_LIMIT=2         # consecutive strikes before restart

HTTP Health Checks

# host1.conf
# Format: "ContainerName:http://host:port/optional-path"
# Two consecutive non-responses trigger a restart.
# "Container running" and "service responding" are not the same thing.
#
HOST1_WATCHDOG_CONTAINER_URLS=(
    "Emby:http://localhost:8096"            # Emby WebUI root
    "NginxProxyManager:http://localhost:81" # NPM admin interface
)

# master.conf
CURL_TIMEOUT=5      # seconds before non-response counts as a failure
RESP_FAIL_LIMIT=2   # consecutive failures before restart

Required Containers

# host1.conf
# Found stopped → restart attempted every cycle until running or skip-listed.
#
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
    "NginxProxyManager"
    "Authelia"
    "Mariadb-Authelia"
    "Redis-Authelia"
)

Dependency Ordering

# host1.conf
# Format: "DependentContainer:dependency1 dependency2"
# All dependencies must be running before the dependent is restarted.
# Prevents Authelia crash-looping while MariaDB is still starting.
#
HOST1_WATCHDOG_DEPENDENCIES=(
    "Authelia:Mariadb-Authelia Redis-Authelia"
    "NextCloud:Postgres-NextCloud"
)

Same dependency config is used by docker_daily_restart.sh and docker_weekly_restart.sh — configure once, applies everywhere.

Startup Grace Period

# master.conf
# Suppress restart actions for N seconds after array start.
# Checks still run and log — only restart actions are suppressed.
#
WATCHDOG_STARTUP_GRACE=600   # 10 minutes

Tier 2 Global Scan

# master.conf
WATCHDOG_SCAN_ALL=true               # enable global scan
WATCHDOG_RESTART_UNHEALTHY=true      # act on Docker HEALTHCHECK failures
WATCHDOG_NOTIFY_OOM=true             # detect and notify kernel OOM kills
WATCHDOG_NOTIFY_CRASHLOOP=true       # detect escalating restart counts
WATCHDOG_RESTART_DEAD=true           # recover containers in dead state
WATCHDOG_RESTART_CRASHED=true        # restart containers that exited non-zero
WATCHDOG_CRASH_LIMIT=5               # RestartCount above this → restart + skip list

# Exclude containers from Tier 2 (one-shots, manually managed, benign exits):
WATCHDOG_SCAN_IGNORE=(
    "my-one-shot-container"
)

Restart Loop Protection

# master.conf
WATCHDOG_CONTAINER_RESTART_LIMIT=3    # restarts in the window before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1   # rolling window in hours

After hitting the limit: skip list + critical notification. The watchdog stops touching the container. Auto-clear: if the container recovers on its own and is found running, it's removed from the skip list automatically. Manual clear is only needed when the container is stuck stopped — use Tools/watchdog_skip_list_manager.sh.

Notification Batching

# master.conf
WATCHDOG_BATCH_NOTIFY=true
# All events from one cycle → one notification at the end.
# A shared DB going down can cascade 10+ containers. Without batching: 10 pings.
# With batching: one summary listing all affected containers.

Usage

docker_watchdog.sh           # single pass (called by watchdog_orchestrator.sh)
docker_watchdog.sh --dry-run # full cycle preview without restarting anything
docker_watchdog.sh --status  # skip list, strike counts, grace period, RAM deferral state
docker_watchdog.sh --log     # verbose per-cycle output

Skip List Recovery

# Step 1 — understand the situation
Tools/watchdog_skip_list_manager.sh --status

# Step 2 — fix the underlying problem
# docker logs ContainerName --tail 100
# df -h /mnt/user

# Step 3 — clear the container
Tools/watchdog_skip_list_manager.sh --clear ContainerName

# Step 4 — start manually (confirms fix before handing back to watchdog)
docker start ContainerName

# Step 5 — monitoring resumes automatically on next cycle

---

storage_watchdog.sh

Runs third in the orchestrator sequence. Two independent checks per cycle: growth rate detection (automatic, zero config) and oversize log detection. Uses its own strike state file — independent from docker_watchdog.

Growth Rate Detection

# master.conf
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2          # growth per cycle that triggers a strike
WATCHDOG_APPDATA_STRIKE_LIMIT=3       # strikes before alert
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"   # size baseline

Runs du -sm appdata/*/ each cycle — pure inode metadata, very lightweight on NVMe. Compares each container's current size to the baseline from the previous cycle. Growth > WATCHDOG_APPDATA_GROWTH_GB per cycle increments the container's strike count. Strike 1: warn. Strike 2: escalate. Strike 3: critical alert. Strikes auto-clear when growth drops to zero (condition resolved).

Zero configuration required for new containers. Growth rate detection covers all containers automatically. The suppress array below is only for known-legitimate growth.

Growth Suppress Ceilings

# host1.conf
# ONLY needed in specific cases — growth rate detection covers everything automatically.
# Use when a container's appdata legitimately grows fast during normal operation
# and you want to suppress false positives above a known-safe threshold.
#
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
    ["Tdarr"]="25600"       # 25GB — transcode cache grows during active jobs
    ["7dtd"]="20480"        # 20GB — game server world data, expected large
)

Log File Detection

# master.conf
WATCHDOG_APPDATA_LOG_MAX_GB=2         # *.log / *.log.* files above this trigger a strike
WATCHDOG_APPDATA_TRUNCATE_LOGS=false  # true: truncate at strike limit; false: alert only

Scans all *.log and *.log.* files across all appdata paths. Files over the threshold increment per-file strike counts. At strike limit: truncates in-place with truncate -s 0 (container keeps its file handle — space reclaimed immediately without container restart) or sends a critical alert if truncation is disabled.

Log strikes auto-clear when the file drops below threshold.

Usage

storage_watchdog.sh            # single pass (called by watchdog_orchestrator.sh)
storage_watchdog.sh --status   # strikes, growth baseline age, suppress ceilings
storage_watchdog.sh --dry-run  # show what would be alerted/truncated
storage_watchdog.sh --log      # verbose per-container output

---

system_watchdog.sh

Runs last in the orchestrator sequence. The only script in the ecosystem authorized to reboot. Watches the server itself — not containers, not storage. Reboots only when healing at every other layer has failed or when the failure is non-recoverable.

Three-Tier Response

Tier 1 — CRITICAL (immediate reboot, no strikes)

Condition	Threshold	Why immediate
Docker daemon unresponsive	N/A	Every docker command hangs — nothing can be healed
rootfs usage	SYS_WATCHDOG_ROOTFS_CRITICAL_PCT (99%)	SSH stops; state files fail silently
Kernel oops/BUG in dmesg	delta > 0	Kernel running with corrupted state
File descriptor exhaustion	SYS_WATCHDOG_FD_CRITICAL_PCT (95%)	New connections silently failing
/boot read-only	write test fails	Config writes silently failing

Tier 2 — URGENT (bypass strikes with OOM confirmation) RAM below MEM_GB AND OOM kills this cycle ≥ SYS_WATCHDOG_OOM_LIMIT. Both conditions required — RAM alone uses the standard strike system. OOM confirms the system is dying faster than watchdogs can heal.

Tier 3 — STANDARD (SYS_WATCHDOG_STRIKES consecutive failures → reboot)

Check	Threshold
Free RAM	MEM_WARN_GB → MEM_SHUTDOWN_GB → MEM_GB
Load average	SYS_WATCHDOG_LOAD_MULTIPLIER × cpu_count
CPU temperature	SYS_WATCHDOG_CPU_TEMP
Zombie processes	SYS_WATCHDOG_ZOMBIES
/var/log usage	SYS_WATCHDOG_VAR_LOG_PCT
/tmp usage	SYS_WATCHDOG_TMP_PCT
Array disk errors	mdstat error delta > 0
NIC state	interface operstate != "up"
Required containers	containers in SYS_WATCHDOG_REQUIRED_CONTAINERS

RAM Tiers

MEM_WARN_GB (10GB)      → warn + notify, no action
MEM_SHUTDOWN_GB (6GB)   → stop non-essential containers, wait for recovery
MEM_GB (4GB)            → strike → reboot (URGENT bypass with OOM)
MEM_RECOVER_GB (30GB)   → RAM must reach this before stopped containers restart

At MEM_SHUTDOWN_GB, all containers NOT listed in SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED are stopped. Adjust in master.conf for your critical services.

Abort Conditions

Prevent reboot — running them would risk data loss:

SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true   # ZFS pool degraded/faulted
SYS_WATCHDOG_ABORT_ON_PARITY=true          # Parity check/rebuild running
SYS_WATCHDOG_ABORT_ON_MOVER=true           # Mover running

Tier 1 CRITICAL bypasses all abort conditions — an imminent crash outweighs data safety concerns.

Reboot Rate Limit

SYS_WATCHDOG_REBOOT_WINDOW_HRS=2   # window in hours
SYS_WATCHDOG_MAX_REBOOTS=3         # max reboots within the window

If the server reboots SYS_WATCHDOG_MAX_REBOOTS times within the window, the watchdog switches from rebooting to notifying only. Prevents a boot loop where the watchdog reboots → something crashes again immediately → reboot again.

Usage

system_watchdog.sh              # single pass (called by watchdog_orchestrator.sh)
system_watchdog.sh --dry-run    # run detection logic without rebooting
system_watchdog.sh --status     # thresholds, current state, strike counts
system_watchdog.sh --log        # verbose per-check output

---

Full Configuration Reference

# master.conf

# ── Resource Watchdog ──────────────────────────────────────────────────────────
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"

RW_RAM_SOFT_GB=20
RW_RAM_MEDIUM_GB=15
RW_RAM_HARD_GB=10
RW_RAM_RECOVER_GB=25

RW_LOAD_SOFT_MULTIPLIER=2.0
RW_LOAD_MEDIUM_MULTIPLIER=3.0
RW_RECOVER_CYCLES=3

RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"
RW_SABNZBD_SPEED_MEDIUM="10M"
RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200        # KB/s
RW_QBIT_DL_MEDIUM=10240

RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")

# host*.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"

# ── Docker Watchdog ────────────────────────────────────────────────────────────
DOCKER_WATCHDOG_INTERVAL=900         # seconds between cycles (15 minutes)
WATCHDOG_STARTUP_GRACE=600           # seconds before restarts begin after boot
CONTAINER_DELAY=15                   # seconds between dependency + dependent restart

SOFT_MEM_THRESHOLD=80                # warn at % of hard limit (no restart)

SOFT_CPU_THRESHOLD=50
HARD_CPU_THRESHOLD=85
CPU_FAIL_LIMIT=2

CURL_TIMEOUT=5
RESP_FAIL_LIMIT=2

WATCHDOG_CONTAINER_RESTART_LIMIT=3
WATCHDOG_CONTAINER_RESTART_WINDOW=1

WATCHDOG_SCAN_ALL=true
WATCHDOG_SCAN_IGNORE=()
WATCHDOG_RESTART_UNHEALTHY=true
WATCHDOG_NOTIFY_OOM=true
WATCHDOG_NOTIFY_CRASHLOOP=true
WATCHDOG_CRASH_LIMIT=5
WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
WATCHDOG_BATCH_NOTIFY=true

# State files:
WATCHDOG_STATE_FILE="/tmp/watchdog_state.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"

# host*.conf
HOST1_WATCHDOG_CONTAINERS=()          # "ContainerName:LimitMB"
HOST1_WATCHDOG_CONTAINER_URLS=()      # "ContainerName:http://host:port"
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
HOST1_WATCHDOG_DEPENDENCIES=()        # "Dependent:dep1 dep2"

# ── Storage Watchdog ───────────────────────────────────────────────────────────
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2
WATCHDOG_APPDATA_LOG_MAX_GB=2
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
WATCHDOG_APPDATA_STRIKE_LIMIT=3
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"

# host*.conf (optional — only for suppress ceilings)
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
#     ["Tdarr"]="25600"
# )

# ── System Watchdog ────────────────────────────────────────────────────────────
SYS_WATCHDOG_STRIKE_LIMIT=2
SYSTEM_WATCHDOG_INTERVAL=300

SYS_WATCHDOG_REBOOT_WINDOW_HRS=2
SYS_WATCHDOG_MAX_REBOOTS=3
SYS_WATCHDOG_OOM_LIMIT=3

MEM_WARN_GB=10
MEM_SHUTDOWN_GB=6
MEM_GB=4
MEM_RECOVER_GB=30

SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99
SYS_WATCHDOG_FD_CRITICAL_PCT=95
SYS_WATCHDOG_LOAD_MULTIPLIER=4
SYS_WATCHDOG_CPU_TEMP=85
SYS_WATCHDOG_ZOMBIES=20
SYS_WATCHDOG_VAR_LOG_PCT=80
SYS_WATCHDOG_TMP_PCT=85

SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=true
SYS_WATCHDOG_ABORT_ON_MOVER=true

SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
    "NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
)
SYS_WATCHDOG_REQUIRED_CONTAINERS=()

# State files:
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"

---

Troubleshooting

resource_watchdog Paused Containers It Shouldn't

# Check current state:
resource_watchdog.sh --status

# Add to RW_CRITICAL_CONTAINERS in master.conf:
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")

# Un-pause manually if needed:
docker unpause MyContainer

docker_watchdog Keeps Restarting a Healthy Container

# Check what's triggering it — memory, CPU, HTTP, or required:
docker_watchdog.sh --status

# Check CPU normalised — brief spikes should not trigger (2-strike system):
# If triggering on CPU: check if HARD_CPU_THRESHOLD is set appropriately
# for containers with legitimate burst usage (Tdarr encoding, SABnzbd unpacking)

# Check HTTP — is the health endpoint returning 200?
curl -sf --max-time 5 http://localhost:PORT && echo "OK" || echo "FAIL"

Container on the Skip List After Fixing the Problem

# See skip list and container state:
Tools/watchdog_skip_list_manager.sh --status

# Fix root cause first, then clear:
Tools/watchdog_skip_list_manager.sh --clear ContainerName

# Start manually to confirm fix before handing back to watchdog:
docker start ContainerName

storage_watchdog Alerting on a Container That Grows Legitimately

# Check what triggered it:
storage_watchdog.sh --status

# Add a suppress ceiling to host*.conf:
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
#     ["ContainerName"]="10240"    # 10GB ceiling — legitimate growth, suppress below this
# )

system_watchdog Rebooted Unexpectedly

# Check the reboot log (survives reboots):
cat /boot/config/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot

# Check syslog near the reboot time:
grep "system_watchdog" /var/log/syslog | tail -20

system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout

# All four watchdogs run as single-pass scripts — there is no background process to check.
# If the orchestrator reports a timeout, one pass took longer than expected.

# Check the orchestrator itself:
Orchestrators/watchdog_orchestrator.sh --status

# Run the slow watchdog directly with --log to see where it's hanging:
Watchdogs/system_watchdog.sh --log --dry-run