FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0621f668895cc3598082d2d7091b3c6c198b8f52
Varaverk/Watchdogs/Manual-Watchdogs.md
T

29 KiB
Raw Blame History

━━━━━ WATCHDOGS — Manual ━━━━━

Configuration reference, operational procedures, and troubleshooting for all watchdog scripts. For design philosophy and script relationships see README-Watchdogs.md. For the orchestrator that calls these scripts see Orchestrators/watchdog_orchestrator.sh.

━━━ CONTENTS ━━━

Output Tiers

All watchdog scripts use a two-tier output model: echo lines are always visible; log lines only appear when --log is passed.

All watchdog scripts are single-pass scripts called once every 15 minutes by the orchestrator. Without --log, only state transitions, warnings, errors, and the conclusion line are visible. Per-check detail is suppressed on clean cycles.

docker_watchdog.sh is the exception — it is silent on clean cycles by design. 96 cycles/day means clean-cycle noise would bury real events. Its output only appears when there are restarts, skip-list events, or RAM deferral. Use --log to see per-cycle detail on clean cycles.

resource_watchdog.sh

Runs first in the orchestrator sequence. Reduces system pressure before docker_watchdog attempts any container restarts. Containers restarted into a RAM-pressured system just fail again — this script ensures docker_watchdog has breathing room.

Pressure Levels

Three escalating levels, each additive:

# master.conf
RW_RAM_SOFT_GB=20           # Level 1 trigger — throttle downloaders
RW_RAM_MEDIUM_GB=15         # Level 2 trigger — throttle + pause containers
RW_RAM_HARD_GB=10           # Level 3 trigger — stop containers + defer docker_watchdog
RW_RAM_RECOVER_GB=25        # de-escalate only after RAM reaches this

RW_LOAD_SOFT_MULTIPLIER=2.0    # load > 2× cpu count = level 1
RW_LOAD_MEDIUM_MULTIPLIER=3.0  # load > 3× cpu count = level 2

RW_RECOVER_CYCLES=3         # consecutive under-threshold runs before de-escalating

Level 1 (soft): Throttle SABnzbd + qBittorrent download speeds. Level 2 (medium): Further throttle + docker pause non-critical containers. Level 3 (hard): docker stop optional services + write mem_shutdown_active=true to RW_STATE_FILE. docker_watchdog.sh reads this flag and skips all restart logic until pressure clears. Without this coordination, docker_watchdog would immediately restart containers that resource_watchdog just stopped to free RAM.

Recovery de-escalates one level at a time — prevents flip-flopping between states.

Per-Host Container Lists

# host1.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")        # paused at level 2
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")    # stopped at level 3

Containers in RW_CRITICAL_CONTAINERS are never paused or stopped regardless of pressure level. Default: Emby, NginxProxyManager, Authelia, Mariadb, Redis.

Downloader Throttle Config

# master.conf
RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"        # throttled at level 1
RW_SABNZBD_SPEED_MEDIUM="10M"      # throttled further at level 2

RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200              # KB/s — level 1
RW_QBIT_DL_MEDIUM=10240            # KB/s — level 2

# host1.conf (API access)
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"

Usage

resource_watchdog.sh            # single pass (called by watchdog_orchestrator.sh)
resource_watchdog.sh --dry-run  # show what would be throttled/paused/stopped
resource_watchdog.sh --status   # current level, active actions, recovery cycle count
resource_watchdog.sh --log      # verbose per-check output

docker_watchdog.sh

Runs second in the orchestrator sequence. Two-tier container healing — explicit per-container configuration (Tier 1) plus a global catch-all scan (Tier 2).

Reads RW_STATE_FILE at cycle start — if mem_shutdown_active=true, skips all container restart logic (resource_watchdog is managing the situation). Health URL checks for excluded containers still run.

Memory Hard Limits

# host1.conf
# Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
#
# Sizing: check normal peak with "docker stats ContainerName"
# Set limit at ~150-200% of normal peak
#
HOST1_WATCHDOG_CONTAINERS=(
    "Emby:18432"        # 18GB — peaks ~12GB under heavy transcode load
    "LidaTube:6144"     # 6GB  — YouTube downloader, grows with large queues
    "Tdarr:6144"        # 6GB  — video transcoder, memory-intensive by nature
    "Code-Server:1024"  # 1GB  — IDE, should be light; 1GB is generous
)

A soft warning fires at SOFT_MEM_THRESHOLD=80 percent of the hard limit — early visibility into a container approaching its ceiling before a restart is triggered.

CPU Thresholds

# master.conf
# CPU is normalised against total core count.
# 85% normalised on a 16-core machine = 13.6 cores worth of a single process.
#
# CPU uses a STRIKE SYSTEM — brief spikes are normal (Tdarr, Emby transcoding, SABnzbd).
# Strike 1: above HARD_CPU_THRESHOLD → warn, increment strike
# Strike 2: above threshold → restart, reset counter
# Recovery: drops below threshold any cycle → reset to 0
#
SOFT_CPU_THRESHOLD=50    # warn at 50% normalised — informational only
HARD_CPU_THRESHOLD=85    # strike at 85% normalised
CPU_FAIL_LIMIT=2         # consecutive strikes before restart

HTTP Health Checks

# host1.conf
# Format: "ContainerName:http://host:port/optional-path"
# Two consecutive non-responses trigger a restart.
# "Container running" and "service responding" are not the same thing.
#
HOST1_WATCHDOG_CONTAINER_URLS=(
    "Emby:http://localhost:8096"            # Emby WebUI root
    "NginxProxyManager:http://localhost:81" # NPM admin interface
)

# master.conf
CURL_TIMEOUT=5      # seconds before non-response counts as a failure
RESP_FAIL_LIMIT=2   # consecutive failures before restart

Required Containers

# host1.conf
# Found stopped → restart attempted every cycle until running or skip-listed.
#
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
    "NginxProxyManager"
    "Authelia"
    "Mariadb-Authelia"
    "Redis-Authelia"
)

Dependency Ordering

# host1.conf
# Format: "DependentContainer:dependency1 dependency2"
# All dependencies must be running before the dependent is restarted.
# Prevents Authelia crash-looping while MariaDB is still starting.
#
HOST1_WATCHDOG_DEPENDENCIES=(
    "Authelia:Mariadb-Authelia Redis-Authelia"
    "NextCloud:Postgres-NextCloud"
)

Same dependency config is used by docker_daily_restart.sh and docker_weekly_restart.sh — configure once, applies everywhere.

Startup Grace Period

# master.conf
# Suppress restart actions for N seconds after array start.
# Checks still run and log — only restart actions are suppressed.
#
WATCHDOG_STARTUP_GRACE=600   # 10 minutes

Tier 2 Global Scan

# master.conf
WATCHDOG_SCAN_ALL=true               # enable global scan
WATCHDOG_RESTART_UNHEALTHY=true      # act on Docker HEALTHCHECK failures
WATCHDOG_NOTIFY_OOM=true             # detect and notify kernel OOM kills
WATCHDOG_NOTIFY_CRASHLOOP=true       # detect escalating restart counts
WATCHDOG_RESTART_DEAD=true           # recover containers in dead state
WATCHDOG_RESTART_CRASHED=true        # restart containers that exited non-zero
WATCHDOG_CRASH_LIMIT=5               # RestartCount above this → restart + skip list

# Exclude containers from Tier 2 (one-shots, manually managed, benign exits):
WATCHDOG_SCAN_IGNORE=(
    "my-one-shot-container"
)

Restart Loop Protection

# master.conf
WATCHDOG_CONTAINER_RESTART_LIMIT=3    # restarts in the window before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1   # rolling window in hours

After hitting the limit: skip list + critical notification. The watchdog stops touching the container. Auto-clear: if the container recovers on its own and is found running, it's removed from the skip list automatically. Manual clear is only needed when the container is stuck stopped — use Tools/watchdog_skip_list_manager.sh.

Notification Batching

# master.conf
WATCHDOG_BATCH_NOTIFY=true
# All events from one cycle → one notification at the end.
# A shared DB going down can cascade 10+ containers. Without batching: 10 pings.
# With batching: one summary listing all affected containers.

Usage

docker_watchdog.sh           # single pass (called by watchdog_orchestrator.sh)
docker_watchdog.sh --dry-run # full cycle preview without restarting anything
docker_watchdog.sh --status  # skip list, strike counts, grace period, RAM deferral state
docker_watchdog.sh --log     # verbose per-cycle output

Skip List Recovery

# Step 1 — understand the situation
Tools/watchdog_skip_list_manager.sh --status

# Step 2 — fix the underlying problem
# docker logs ContainerName --tail 100
# df -h /mnt/user

# Step 3 — clear the container
Tools/watchdog_skip_list_manager.sh --clear ContainerName

# Step 4 — start manually (confirms fix before handing back to watchdog)
docker start ContainerName

# Step 5 — monitoring resumes automatically on next cycle

system_watchdog.sh

Thin orchestrator — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially each cycle. Called third by watchdog_orchestrator.sh. Covers all system component watchdogs. Can also be run standalone to check all system components at once.

Usage

system_watchdog.sh              # run all system component watchdogs
system_watchdog.sh --status     # show configured scripts and their paths
system_watchdog.sh --dry-run    # preview without executing anything
system_watchdog.sh --log        # verbose output

System/storage_watchdog.sh

Called by system_watchdog.sh each cycle. Two independent checks per cycle: growth rate detection (automatic, zero config) and oversize log detection. Uses its own strike state file — independent from docker_watchdog.

Growth Rate Detection

# master.conf
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2          # growth per cycle that triggers a strike
WATCHDOG_APPDATA_STRIKE_LIMIT=3       # strikes before alert
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"   # size baseline

Runs du -sm appdata/*/ each cycle — pure inode metadata, very lightweight on NVMe. Compares each container's current size to the baseline from the previous cycle. Growth > WATCHDOG_APPDATA_GROWTH_GB per cycle increments the container's strike count. Strike 1: warn. Strike 2: escalate. Strike 3: critical alert. Strikes auto-clear when growth drops to zero (condition resolved).

Zero configuration required for new containers. Growth rate detection covers all containers automatically. The suppress array below is only for known-legitimate growth.

Growth Suppress Ceilings

# host1.conf
# ONLY needed in specific cases — growth rate detection covers everything automatically.
# Use when a container's appdata legitimately grows fast during normal operation
# and you want to suppress false positives above a known-safe threshold.
#
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
    ["Tdarr"]="25600"       # 25GB — transcode cache grows during active jobs
    ["7dtd"]="20480"        # 20GB — game server world data, expected large
)

Log File Detection

# master.conf
WATCHDOG_APPDATA_LOG_MAX_GB=2         # *.log / *.log.* files above this trigger a strike
WATCHDOG_APPDATA_TRUNCATE_LOGS=false  # true: truncate at strike limit; false: alert only

Scans all *.log and *.log.* files across all appdata paths. Files over the threshold increment per-file strike counts. At strike limit: truncates in-place with truncate -s 0 (container keeps its file handle — space reclaimed immediately without container restart) or sends a critical alert if truncation is disabled.

Log strikes auto-clear when the file drops below threshold.

Usage

storage_watchdog.sh            # single pass (called by system_watchdog.sh)
storage_watchdog.sh --status   # strikes, growth baseline age, suppress ceilings
storage_watchdog.sh --dry-run  # show what would be alerted/truncated
storage_watchdog.sh --log      # verbose per-container output

System/webgui_watchdog.sh

Lives in Plugin/unraid/Watchdogs/System/webgui_watchdog.sh — calls Unraid-specific service commands (rc.nginx, rc.php-fpm, emhttp) via the platform adapter. Called by system_watchdog.sh via SYSTEM_WATCHDOG_SCRIPTS in master.conf.

Called by system_watchdog.sh each cycle. Monitors WebGUI availability and escalates through three restart steps if unresponsive. Silent when healthy.

Escalation Path

WebGUI responding → exit 0 (silent)

Not responding:
  Step 1 — nginx restart       → wait WEBGUI_NGINX_WAIT → recheck
  Step 2 — php-fpm restart     → wait WEBGUI_PHP_WAIT   → recheck
  Step 3 — emhttp restart      → wait WEBGUI_EMHTTP_WAIT → recheck
  All three failed → critical notify, manual intervention needed

Configuration

# master.conf
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30

Usage

webgui_watchdog.sh             # single pass (called by system_watchdog.sh)
webgui_watchdog.sh --status    # current WebGUI state + nginx/php-fpm/emhttp status
webgui_watchdog.sh --dry-run   # show which services would be restarted
webgui_watchdog.sh --log       # verbose per-step output

System/network_watchdog.sh

Called by system_watchdog.sh each cycle. Checks that the outside world can actually reach what it needs to reach. Internet reachability gates all other checks — if upstream is down, DDNS and NPM checks are skipped to prevent false positives.

Check Sequence

1. Internet → curl NETWORK_WATCHDOG_INTERNET_URL
   fail → alert + exit (skip all remaining checks)

2. DDNS → public IP (ifconfig.me) vs DNS record (dig @1.1.1.1)
   match  → pass (silent)
   mismatch → restart HOST*_NETWORK_WATCHDOG_DDNS_CONTAINER + notify

3. Tailscale → tailscale status --json BackendState
   Running → pass (silent)
   not Running → notify (no auto-restart — warrants human review)

4. NPM proxy → curl HOST*_NETWORK_WATCHDOG_NPM_URL (external)
   reachable → pass, clear strikes
   not reachable → strike 1: warn + notify
                   strike 2: restart NginxProxyManager + notify + clear strikes

Configuration

# master.conf
NETWORK_WATCHDOG_ENABLED=true
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"

# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"

Usage

network_watchdog.sh              # run all connectivity checks (silent when healthy)
network_watchdog.sh --status     # current IP, DNS record, Tailscale state, NPM strike count
network_watchdog.sh --dry-run    # check without restarting any containers
network_watchdog.sh --log        # verbose per-check output

Troubleshooting

DDNS keeps restarting the container but record stays stale

# Check if the container is actually running after restart:
docker ps | grep "Gmer4Lfe.com"

# Check container logs for Cloudflare API errors:
docker logs "Gmer4Lfe.com" --tail 20

# Verify public IP detection:
curl -sf https://ifconfig.me

# Verify DNS resolution:
dig +short gmer4lfe.com @1.1.1.1

NPM strikes accumulating but NPM is running

# Check if the external URL is actually responding:
curl -sv https://gmer4lfe.com 2>&1 | head -20

# NPM may be running but a backend container is down — check the specific service
# the URL routes to, not just NginxProxyManager itself.

# Check NPM strike count:
network_watchdog.sh --status

Tailscale showing not Running

# Check tailscale state directly:
tailscale status

# Check the backend state specifically:
tailscale status --json | grep BackendState

# Reconnect manually if needed:
tailscale up

System/conf_cache_watchdog.sh

Called by system_watchdog.sh each cycle. Maintains the persistent partner conf backup at $PERSISTENT_CONF_CACHE while the partner is offline.

Behaviour

Remote online: removes the persistent backup if one exists. It is not needed — conf_sync.sh will pull a fresh copy on the next boot. Silent when backup is already absent.

Remote offline: copies partner confs from the RAM cache ($CONF_RAM_CACHE_DIR, /tmp/varaverk/conf/) to $PERSISTENT_CONF_CACHE. Runs every 15 minutes, so the backup stays current throughout an extended outage. If this host reboots while the partner is still down, conf_cache_restore.sh will load the backup into RAM and fallback.sh will have valid partner vars.

Silent when remote is online and no backup exists (the normal steady state).

Gates

  • FALLBACK_ENABLED=false → no-op (no fallback means no need for partner vars)
  • CONF_SYNC_ENABLED=false → no-op
  • PARTNERSHIP_ENABLED=false → exits silently (require_partnership gate)

Usage

conf_cache_watchdog.sh              # single pass (called by system_watchdog.sh)
conf_cache_watchdog.sh --dry-run    # show what would be written or removed
conf_cache_watchdog.sh --log        # verbose output

stability_watchdog.sh

Runs last in the orchestrator sequence. The only script in the ecosystem authorized to reboot. Watches the server itself — not containers, not storage. Reboots only when healing at every other layer has failed or when the failure is non-recoverable.

Three-Tier Response

Tier 1 — CRITICAL (immediate reboot, no strikes)

Condition Threshold Why immediate
Docker daemon unresponsive N/A Every docker command hangs — nothing can be healed
rootfs usage SYS_WATCHDOG_ROOTFS_CRITICAL_PCT (99%) SSH stops; state files fail silently
Kernel oops/BUG in dmesg delta > 0 Kernel running with corrupted state
File descriptor exhaustion SYS_WATCHDOG_FD_CRITICAL_PCT (95%) New connections silently failing
/boot read-only write test fails Config writes silently failing

Tier 2 — URGENT (bypass strikes with OOM confirmation) RAM below SYS_WATCHDOG_MEM_GB AND OOM kills this cycle ≥ SYS_WATCHDOG_OOM_LIMIT. Both conditions required — RAM alone uses the standard strike system. OOM confirms the system is dying faster than watchdogs can heal.

Tier 3 — STANDARD (SYS_WATCHDOG_STRIKE_LIMIT consecutive failures → reboot)

Check Threshold
Free RAM SYS_WATCHDOG_MEM_GB (reboot trigger — earlier tiers handled by resource_watchdog)
Load average SYS_WATCHDOG_LOAD_MULTIPLIER × cpu_count
CPU temperature SYS_WATCHDOG_CPU_TEMP_MAX
Zombie processes SYS_WATCHDOG_ZOMBIE_LIMIT
/var/log usage SYS_WATCHDOG_LOG_PCT
/tmp usage SYS_WATCHDOG_TMP_PCT
Array disk errors mdstat error delta > 0
NIC state interface operstate != "up"

RAM Tiers

RAM pressure is a graduated response split across resource_watchdog and stability_watchdog:

RW_RAM_SOFT_GB (12GB)   → throttle downloads, reduce background load  (resource_watchdog)
RW_RAM_MEDIUM_GB (8GB)  → pause background containers                 (resource_watchdog)
RW_RAM_HARD_GB (6GB)    → stop optional containers, wait for recovery  (resource_watchdog)
SYS_WATCHDOG_MEM_GB (4GB) → strike → reboot (last resort)             (stability_watchdog)
RW_RAM_RECOVER_GB (20GB) → RAM must reach this before stopped containers restart

SYS_WATCHDOG_MEM_GB must always be below RW_RAM_HARD_GB — resource_watchdog acts first. Containers stopped at the hard tier use HOST*_RW_STOP_CONTAINERS in host*.conf. Containers paused at the medium tier use HOST*_RW_PAUSE_CONTAINERS in host*.conf.

Abort Conditions

Prevent reboot — running them would risk data loss:

SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true   # ZFS pool degraded/faulted
SYS_WATCHDOG_ABORT_ON_PARITY=false         # aborting parity mid-check is worse than crashing
SYS_WATCHDOG_ABORT_ON_MOVER=false          # aborting move mid-run is worse than crashing

Tier 1 CRITICAL bypasses all abort conditions — an imminent crash outweighs data safety concerns.

Reboot Rate Limit

SYS_WATCHDOG_REBOOT_WINDOW_HRS=12  # window in hours
SYS_WATCHDOG_REBOOT_LIMIT=3         # max reboots within the window

If the server reboots SYS_WATCHDOG_REBOOT_LIMIT times within the window, the watchdog switches from rebooting to notifying only. Prevents a boot loop where the watchdog reboots → something crashes again immediately → reboot again.

Usage

stability_watchdog.sh              # single pass (called by watchdog_orchestrator.sh)
stability_watchdog.sh --dry-run    # run detection logic without rebooting
stability_watchdog.sh --status     # thresholds, current state, strike counts
stability_watchdog.sh --log        # verbose per-check output

Full Configuration Reference

# master.conf

# ── Resource Watchdog ──────────────────────────────────────────────────────────
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"

RW_RAM_SOFT_GB=20
RW_RAM_MEDIUM_GB=15
RW_RAM_HARD_GB=10
RW_RAM_RECOVER_GB=25

RW_LOAD_SOFT_MULTIPLIER=2.0
RW_LOAD_MEDIUM_MULTIPLIER=3.0
RW_RECOVER_CYCLES=3

RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"
RW_SABNZBD_SPEED_MEDIUM="10M"
RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200        # KB/s
RW_QBIT_DL_MEDIUM=10240

RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")

# host*.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"

# ── Docker Watchdog ────────────────────────────────────────────────────────────
WATCHDOG_STARTUP_GRACE=600           # seconds before restarts begin after boot
CONTAINER_DELAY=15                   # seconds between dependency + dependent restart

SOFT_MEM_THRESHOLD=80                # warn at % of hard limit (no restart)

SOFT_CPU_THRESHOLD=50
HARD_CPU_THRESHOLD=85
CPU_FAIL_LIMIT=2

CURL_TIMEOUT=5
RESP_FAIL_LIMIT=2

WATCHDOG_CONTAINER_RESTART_LIMIT=3
WATCHDOG_CONTAINER_RESTART_WINDOW=1

WATCHDOG_SCAN_ALL=true
WATCHDOG_SCAN_IGNORE=()
WATCHDOG_RESTART_UNHEALTHY=true
WATCHDOG_NOTIFY_OOM=true
WATCHDOG_NOTIFY_CRASHLOOP=true
WATCHDOG_CRASH_LIMIT=5
WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
WATCHDOG_BATCH_NOTIFY=true

# State files (all paths adapt to storage mode via STATE_DIR / DATA_DIR):
WATCHDOG_STATE_FILE="$STATE_DIR/container_watchdog_state.db"
WATCHDOG_CONTAINER_RESTART_LOG="$DATA_DIR/container_restart_history.db"

# host*.conf
HOST1_WATCHDOG_CONTAINERS=()          # "ContainerName:LimitMB"
HOST1_WATCHDOG_CONTAINER_URLS=()      # "ContainerName:http://host:port"
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
HOST1_WATCHDOG_DEPENDENCIES=()        # "Dependent:dep1 dep2"

# ── Storage Watchdog ───────────────────────────────────────────────────────────
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2
WATCHDOG_APPDATA_LOG_MAX_GB=2
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
WATCHDOG_APPDATA_STRIKE_LIMIT=3
WATCHDOG_APPDATA_GROWTH_FILE="$STATE_DIR/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="$STATE_DIR/storage_watchdog_state.db"

# host*.conf (optional — only for suppress ceilings)
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
#     ["Tdarr"]="25600"
# )

# ── System Watchdog ────────────────────────────────────────────────────────────
# Reboot trigger only — warn/shutdown/recover RAM tiers handled by resource_watchdog.sh
# RW_RAM_HARD_GB > SYS_WATCHDOG_MEM_GB always (resource_watchdog acts before watchdog reboots)
SYS_WATCHDOG_STRIKE_LIMIT=2

SYS_WATCHDOG_REBOOT_WINDOW_HRS=12
SYS_WATCHDOG_REBOOT_LIMIT=3
SYS_WATCHDOG_OOM_LIMIT=3

SYS_WATCHDOG_MEM_GB=4              # strike system → reboot (last resort — below resource_watchdog hard stop)

SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99
SYS_WATCHDOG_FD_CRITICAL_PCT=95
SYS_WATCHDOG_LOAD_MULTIPLIER=4
SYS_WATCHDOG_CPU_TEMP_MAX=95
SYS_WATCHDOG_ZOMBIE_LIMIT=50
SYS_WATCHDOG_LOG_PCT=95
SYS_WATCHDOG_TMP_PCT=90

SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=false
SYS_WATCHDOG_ABORT_ON_MOVER=false

# State files — all in $STATE_DIR (survive reboots; adapt to storage mode):
SYS_WATCHDOG_STATE_FILE="$STATE_DIR/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="$STATE_DIR/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="$STATE_DIR/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="$STATE_DIR/system_watchdog_oom.db"
RW_STATE_FILE="$STATE_DIR/resource_watchdog_state.db"

# ── Network Watchdog ───────────────────────────────────────────────────────────
NETWORK_WATCHDOG_ENABLED=true
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="$STATE_DIR/network_watchdog_state.db"

# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"

Troubleshooting

resource_watchdog Paused Containers It Shouldn't

# Check current state:
resource_watchdog.sh --status

# Add to RW_CRITICAL_CONTAINERS in master.conf:
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")

# Un-pause manually if needed:
docker unpause MyContainer

docker_watchdog Keeps Restarting a Healthy Container

# Check what's triggering it — memory, CPU, HTTP, or required:
docker_watchdog.sh --status

# Check CPU normalised — brief spikes should not trigger (2-strike system):
# If triggering on CPU: check if HARD_CPU_THRESHOLD is set appropriately
# for containers with legitimate burst usage (Tdarr encoding, SABnzbd unpacking)

# Check HTTP — is the health endpoint returning 200?
curl -sf --max-time 5 http://localhost:PORT && echo "OK" || echo "FAIL"

Container on the Skip List After Fixing the Problem

# See skip list and container state:
Tools/watchdog_skip_list_manager.sh --status

# Fix root cause first, then clear:
Tools/watchdog_skip_list_manager.sh --clear ContainerName

# Start manually to confirm fix before handing back to watchdog:
docker start ContainerName

storage_watchdog Alerting on a Container That Grows Legitimately

# Check what triggered it:
storage_watchdog.sh --status

# Add a suppress ceiling to host*.conf:
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
#     ["ContainerName"]="10240"    # 10GB ceiling — legitimate growth, suppress below this
# )

stability_watchdog Rebooted Unexpectedly

# Check the reboot log (survives reboots — in data/state/):
cat /boot/config/plugins/varaverk/data/state/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot

# Check syslog near the reboot time:
grep "stability_watchdog" /var/log/syslog | tail -20

stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout

# All watchdogs run as single-pass scripts — there is no background process to check.
# If the orchestrator reports a timeout, one pass took longer than expected.

# Check the orchestrator itself:
Orchestrators/watchdog_orchestrator.sh --status

# Run the slow watchdog directly with --log to see where it's hanging:
Watchdogs/stability_watchdog.sh --log --dry-run
Reference in New Issue View Git Blame Copy Permalink