Watchdogs/ docs: new README + Manual, update affected folders

New docs: Watchdogs/README-Watchdogs.md — design, relationships, script table, state file map Watchdogs/Manual-Watchdogs.md — full config reference for all 4 watchdogs Docker_Essentials/: README — remove docker_watchdog, update folder description and diagrams Manual — strip watchdog config sections, add pointer to Watchdogs/Manual unRAID_Essentials/: README — remove system/resource watchdog, update diagrams and script table Manual — strip system/resource watchdog sections, update TOC + config reference README.md: Add Watchdogs/ to folder structure Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start Fix daily cycle and monitoring diagrams
2026-05-22 17:26:21 -04:00
parent ec79a926e8
commit 8a3e22c9b8
7 changed files with 912 additions and 707 deletions
@@ -0,0 +1,641 @@
+# ━━━━━ WATCHDOGS — Manual ━━━━━
+
+Configuration reference, operational procedures, and troubleshooting for all four
+watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
+For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
+
+---
+
+## ━━━ CONTENTS ━━━
+
+- [Output Tiers](#output-tiers)
+- [resource_watchdog.sh](#resource_watchdogsh)
+- [docker_watchdog.sh](#docker_watchdogsh)
+- [storage_watchdog.sh](#storage_watchdogsh)
+- [system_watchdog.sh](#system_watchdogsh)
+- [Full Configuration Reference](#full-configuration-reference)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Output Tiers
+
+All watchdog scripts use a two-tier output model: `echo` lines are always visible;
+`log` lines only appear when `--log` is passed.
+
+All four watchdogs are **single-pass scripts** called once per minute by the orchestrator.
+Without `--log`, only state transitions, warnings, errors, and the conclusion line are
+visible. Per-check detail is suppressed on clean cycles.
+
+`docker_watchdog.sh` is the exception — it is **silent on clean cycles by design**.
+96 cycles/day means clean-cycle noise would bury real events. Its output only appears
+when there are restarts, skip-list events, or RAM deferral. Use `--log` to see
+per-cycle detail on clean cycles.
+
+---
+
+## resource_watchdog.sh
+
+Runs first in the orchestrator sequence. Reduces system pressure before docker_watchdog
+attempts any container restarts. Containers restarted into a RAM-pressured system just
+fail again — this script ensures docker_watchdog has breathing room.
+
+### Pressure Levels
+
+Three escalating levels, each additive:
+
+```bash
+# master.conf
+RW_RAM_SOFT_GB=20           # Level 1 trigger — throttle downloaders
+RW_RAM_MEDIUM_GB=15         # Level 2 trigger — throttle + pause containers
+RW_RAM_HARD_GB=10           # Level 3 trigger — stop containers + defer docker_watchdog
+RW_RAM_RECOVER_GB=25        # de-escalate only after RAM reaches this
+
+RW_LOAD_SOFT_MULTIPLIER=2.0    # load > 2× cpu count = level 1
+RW_LOAD_MEDIUM_MULTIPLIER=3.0  # load > 3× cpu count = level 2
+
+RW_RECOVER_CYCLES=3         # consecutive under-threshold runs before de-escalating
+```
+
+**Level 1 (soft):** Throttle SABnzbd + qBittorrent download speeds.
+**Level 2 (medium):** Further throttle + `docker pause` non-critical containers.
+**Level 3 (hard):** `docker stop` optional services + write `mem_shutdown_active=true`
+to `RW_STATE_FILE`. docker_watchdog.sh reads this flag and skips all restart logic
+until pressure clears. Without this coordination, docker_watchdog would immediately
+restart containers that resource_watchdog just stopped to free RAM.
+
+Recovery de-escalates one level at a time — prevents flip-flopping between states.
+
+### Per-Host Container Lists
+
+```bash
+# host1.conf
+HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")        # paused at level 2
+HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")    # stopped at level 3
+```
+
+Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
+pressure level. Default: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
+
+### Downloader Throttle Config
+
+```bash
+# master.conf
+RW_SABNZBD_ENABLED=true
+RW_SABNZBD_SPEED_SOFT="50M"        # throttled at level 1
+RW_SABNZBD_SPEED_MEDIUM="10M"      # throttled further at level 2
+
+RW_QBIT_ENABLED=true
+RW_QBIT_DL_SOFT=51200              # KB/s — level 1
+RW_QBIT_DL_MEDIUM=10240            # KB/s — level 2
+
+# host1.conf (API access)
+HOST1_SABNZBD_URL="http://localhost:8080"
+HOST1_SABNZBD_API_KEY="your-api-key"
+HOST1_QBIT_URL="http://localhost:8090"
+HOST1_QBIT_USERNAME="admin"
+HOST1_QBIT_PASSWORD="your-password"
+```
+
+### Usage
+
+```bash
+resource_watchdog.sh            # single pass (called by watchdog_orchestrator.sh)
+resource_watchdog.sh --dry-run  # show what would be throttled/paused/stopped
+resource_watchdog.sh --status   # current level, active actions, recovery cycle count
+resource_watchdog.sh --log      # verbose per-check output
+```
+
+---
+
+## docker_watchdog.sh
+
+Runs second in the orchestrator sequence. Two-tier container healing — explicit
+per-container configuration (Tier 1) plus a global catch-all scan (Tier 2).
+
+Reads `RW_STATE_FILE` at cycle start — if `mem_shutdown_active=true`, skips all
+container restart logic (resource_watchdog is managing the situation). Health URL
+checks for excluded containers still run.
+
+### Memory Hard Limits
+
+```bash
+# host1.conf
+# Format: "ContainerName:LimitInMB"
+# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
+#
+# Sizing: check normal peak with "docker stats ContainerName"
+# Set limit at ~150-200% of normal peak
+#
+HOST1_WATCHDOG_CONTAINERS=(
+    "Emby:18432"        # 18GB — peaks ~12GB under heavy transcode load
+    "LidaTube:6144"     # 6GB  — YouTube downloader, grows with large queues
+    "Tdarr:6144"        # 6GB  — video transcoder, memory-intensive by nature
+    "Code-Server:1024"  # 1GB  — IDE, should be light; 1GB is generous
+)
+```
+
+A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
+visibility into a container approaching its ceiling before a restart is triggered.
+
+### CPU Thresholds
+
+```bash
+# master.conf
+# CPU is normalised against total core count.
+# 85% normalised on a 16-core machine = 13.6 cores worth of a single process.
+#
+# CPU uses a STRIKE SYSTEM — brief spikes are normal (Tdarr, Emby transcoding, SABnzbd).
+# Strike 1: above HARD_CPU_THRESHOLD → warn, increment strike
+# Strike 2: above threshold → restart, reset counter
+# Recovery: drops below threshold any cycle → reset to 0
+#
+SOFT_CPU_THRESHOLD=50    # warn at 50% normalised — informational only
+HARD_CPU_THRESHOLD=85    # strike at 85% normalised
+CPU_FAIL_LIMIT=2         # consecutive strikes before restart
+```
+
+### HTTP Health Checks
+
+```bash
+# host1.conf
+# Format: "ContainerName:http://host:port/optional-path"
+# Two consecutive non-responses trigger a restart.
+# "Container running" and "service responding" are not the same thing.
+#
+HOST1_WATCHDOG_CONTAINER_URLS=(
+    "Emby:http://localhost:8096"            # Emby WebUI root
+    "NginxProxyManager:http://localhost:81" # NPM admin interface
+)
+
+# master.conf
+CURL_TIMEOUT=5      # seconds before non-response counts as a failure
+RESP_FAIL_LIMIT=2   # consecutive failures before restart
+```
+
+### Required Containers
+
+```bash
+# host1.conf
+# Found stopped → restart attempted every cycle until running or skip-listed.
+#
+HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
+    "NginxProxyManager"
+    "Authelia"
+    "Mariadb-Authelia"
+    "Redis-Authelia"
+)
+```
+
+### Dependency Ordering
+
+```bash
+# host1.conf
+# Format: "DependentContainer:dependency1 dependency2"
+# All dependencies must be running before the dependent is restarted.
+# Prevents Authelia crash-looping while MariaDB is still starting.
+#
+HOST1_WATCHDOG_DEPENDENCIES=(
+    "Authelia:Mariadb-Authelia Redis-Authelia"
+    "NextCloud:Postgres-NextCloud"
+)
+```
+
+Same dependency config is used by `docker_daily_restart.sh` and
+`docker_weekly_restart.sh` — configure once, applies everywhere.
+
+### Startup Grace Period
+
+```bash
+# master.conf
+# Suppress restart actions for N seconds after array start.
+# Checks still run and log — only restart actions are suppressed.
+#
+WATCHDOG_STARTUP_GRACE=600   # 10 minutes
+```
+
+### Tier 2 Global Scan
+
+```bash
+# master.conf
+WATCHDOG_SCAN_ALL=true               # enable global scan
+WATCHDOG_RESTART_UNHEALTHY=true      # act on Docker HEALTHCHECK failures
+WATCHDOG_NOTIFY_OOM=true             # detect and notify kernel OOM kills
+WATCHDOG_NOTIFY_CRASHLOOP=true       # detect escalating restart counts
+WATCHDOG_RESTART_DEAD=true           # recover containers in dead state
+WATCHDOG_RESTART_CRASHED=true        # restart containers that exited non-zero
+WATCHDOG_CRASH_LIMIT=5               # RestartCount above this → restart + skip list
+
+# Exclude containers from Tier 2 (one-shots, manually managed, benign exits):
+WATCHDOG_SCAN_IGNORE=(
+    "my-one-shot-container"
+)
+```
+
+### Restart Loop Protection
+
+```bash
+# master.conf
+WATCHDOG_CONTAINER_RESTART_LIMIT=3    # restarts in the window before skip list
+WATCHDOG_CONTAINER_RESTART_WINDOW=1   # rolling window in hours
+```
+
+After hitting the limit: skip list + critical notification. The watchdog stops
+touching the container. Auto-clear: if the container recovers on its own and is
+found running, it's removed from the skip list automatically. Manual clear is only
+needed when the container is stuck stopped — use `Tools/watchdog_skip_list_manager.sh`.
+
+### Notification Batching
+
+```bash
+# master.conf
+WATCHDOG_BATCH_NOTIFY=true
+# All events from one cycle → one notification at the end.
+# A shared DB going down can cascade 10+ containers. Without batching: 10 pings.
+# With batching: one summary listing all affected containers.
+```
+
+### Usage
+
+```bash
+docker_watchdog.sh           # single pass (called by watchdog_orchestrator.sh)
+docker_watchdog.sh --dry-run # full cycle preview without restarting anything
+docker_watchdog.sh --status  # skip list, strike counts, grace period, RAM deferral state
+docker_watchdog.sh --log     # verbose per-cycle output
+```
+
+### Skip List Recovery
+
+```bash
+# Step 1 — understand the situation
+Tools/watchdog_skip_list_manager.sh --status
+
+# Step 2 — fix the underlying problem
+# docker logs ContainerName --tail 100
+# df -h /mnt/user
+
+# Step 3 — clear the container
+Tools/watchdog_skip_list_manager.sh --clear ContainerName
+
+# Step 4 — start manually (confirms fix before handing back to watchdog)
+docker start ContainerName
+
+# Step 5 — monitoring resumes automatically on next cycle
+```
+
+---
+
+## storage_watchdog.sh
+
+Runs third in the orchestrator sequence. Two independent checks per cycle:
+growth rate detection (automatic, zero config) and oversize log detection.
+Uses its own strike state file — independent from docker_watchdog.
+
+### Growth Rate Detection
+
+```bash
+# master.conf
+WATCHDOG_CHECK_APPDATA=true
+WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
+WATCHDOG_APPDATA_GROWTH_GB=2          # growth per cycle that triggers a strike
+WATCHDOG_APPDATA_STRIKE_LIMIT=3       # strikes before alert
+WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"   # size baseline
+```
+
+Runs `du -sm appdata/*/` each cycle — pure inode metadata, very lightweight on NVMe.
+Compares each container's current size to the baseline from the previous cycle.
+Growth > `WATCHDOG_APPDATA_GROWTH_GB` per cycle increments the container's strike count.
+Strike 1: warn. Strike 2: escalate. Strike 3: critical alert.
+Strikes auto-clear when growth drops to zero (condition resolved).
+
+**Zero configuration required for new containers.** Growth rate detection covers all
+containers automatically. The suppress array below is only for known-legitimate growth.
+
+### Growth Suppress Ceilings
+
+```bash
+# host1.conf
+# ONLY needed in specific cases — growth rate detection covers everything automatically.
+# Use when a container's appdata legitimately grows fast during normal operation
+# and you want to suppress false positives above a known-safe threshold.
+#
+declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
+    ["Tdarr"]="25600"       # 25GB — transcode cache grows during active jobs
+    ["7dtd"]="20480"        # 20GB — game server world data, expected large
+)
+```
+
+### Log File Detection
+
+```bash
+# master.conf
+WATCHDOG_APPDATA_LOG_MAX_GB=2         # *.log / *.log.* files above this trigger a strike
+WATCHDOG_APPDATA_TRUNCATE_LOGS=false  # true: truncate at strike limit; false: alert only
+```
+
+Scans all `*.log` and `*.log.*` files across all appdata paths. Files over the threshold
+increment per-file strike counts. At strike limit: truncates in-place with `truncate -s 0`
+(container keeps its file handle — space reclaimed immediately without container restart)
+or sends a critical alert if truncation is disabled.
+
+Log strikes auto-clear when the file drops below threshold.
+
+### Usage
+
+```bash
+storage_watchdog.sh            # single pass (called by watchdog_orchestrator.sh)
+storage_watchdog.sh --status   # strikes, growth baseline age, suppress ceilings
+storage_watchdog.sh --dry-run  # show what would be alerted/truncated
+storage_watchdog.sh --log      # verbose per-container output
+```
+
+---
+
+## system_watchdog.sh
+
+Runs last in the orchestrator sequence. The only script in the ecosystem authorized
+to reboot. Watches the server itself — not containers, not storage. Reboots only when
+healing at every other layer has failed or when the failure is non-recoverable.
+
+### Three-Tier Response
+
+**Tier 1 — CRITICAL (immediate reboot, no strikes)**
+| Condition | Threshold | Why immediate |
+|-----------|-----------|---------------|
+| Docker daemon unresponsive | N/A | Every docker command hangs — nothing can be healed |
+| rootfs usage | `SYS_WATCHDOG_ROOTFS_CRITICAL_PCT` (99%) | SSH stops; state files fail silently |
+| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
+| File descriptor exhaustion | `SYS_WATCHDOG_FD_CRITICAL_PCT` (95%) | New connections silently failing |
+| /boot read-only | write test fails | Config writes silently failing |
+
+**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
+RAM below `MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`.
+Both conditions required — RAM alone uses the standard strike system.
+OOM confirms the system is dying faster than watchdogs can heal.
+
+**Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKES` consecutive failures → reboot)**
+| Check | Threshold |
+|-------|-----------|
+| Free RAM | `MEM_WARN_GB` → `MEM_SHUTDOWN_GB` → `MEM_GB` |
+| Load average | `SYS_WATCHDOG_LOAD_MULTIPLIER` × cpu_count |
+| CPU temperature | `SYS_WATCHDOG_CPU_TEMP` |
+| Zombie processes | `SYS_WATCHDOG_ZOMBIES` |
+| /var/log usage | `SYS_WATCHDOG_VAR_LOG_PCT` |
+| /tmp usage | `SYS_WATCHDOG_TMP_PCT` |
+| Array disk errors | mdstat error delta > 0 |
+| NIC state | interface operstate != "up" |
+| Required containers | containers in `SYS_WATCHDOG_REQUIRED_CONTAINERS` |
+
+### RAM Tiers
+
+```
+MEM_WARN_GB (10GB)      → warn + notify, no action
+MEM_SHUTDOWN_GB (6GB)   → stop non-essential containers, wait for recovery
+MEM_GB (4GB)            → strike → reboot (URGENT bypass with OOM)
+MEM_RECOVER_GB (30GB)   → RAM must reach this before stopped containers restart
+```
+
+At `MEM_SHUTDOWN_GB`, all containers NOT listed in `SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED`
+are stopped. Adjust in master.conf for your critical services.
+
+### Abort Conditions
+
+Prevent reboot — running them would risk data loss:
+
+```bash
+SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true   # ZFS pool degraded/faulted
+SYS_WATCHDOG_ABORT_ON_PARITY=true          # Parity check/rebuild running
+SYS_WATCHDOG_ABORT_ON_MOVER=true           # Mover running
+```
+
+Tier 1 CRITICAL bypasses all abort conditions — an imminent crash outweighs data
+safety concerns.
+
+### Reboot Rate Limit
+
+```bash
+SYS_WATCHDOG_REBOOT_WINDOW_HRS=2   # window in hours
+SYS_WATCHDOG_MAX_REBOOTS=3         # max reboots within the window
+```
+
+If the server reboots `SYS_WATCHDOG_MAX_REBOOTS` times within the window, the watchdog
+switches from rebooting to notifying only. Prevents a boot loop where the watchdog
+reboots → something crashes again immediately → reboot again.
+
+### Usage
+
+```bash
+system_watchdog.sh              # single pass (called by watchdog_orchestrator.sh)
+system_watchdog.sh --dry-run    # run detection logic without rebooting
+system_watchdog.sh --status     # thresholds, current state, strike counts
+system_watchdog.sh --log        # verbose per-check output
+```
+
+---
+
+## Full Configuration Reference
+
+```bash
+# master.conf
+
+# ── Resource Watchdog ──────────────────────────────────────────────────────────
+RW_ENABLED=true
+RW_STATE_FILE="/tmp/resource_watchdog_state.db"
+
+RW_RAM_SOFT_GB=20
+RW_RAM_MEDIUM_GB=15
+RW_RAM_HARD_GB=10
+RW_RAM_RECOVER_GB=25
+
+RW_LOAD_SOFT_MULTIPLIER=2.0
+RW_LOAD_MEDIUM_MULTIPLIER=3.0
+RW_RECOVER_CYCLES=3
+
+RW_SABNZBD_ENABLED=true
+RW_SABNZBD_SPEED_SOFT="50M"
+RW_SABNZBD_SPEED_MEDIUM="10M"
+RW_QBIT_ENABLED=true
+RW_QBIT_DL_SOFT=51200        # KB/s
+RW_QBIT_DL_MEDIUM=10240
+
+RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
+
+# host*.conf
+HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
+HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
+HOST1_SABNZBD_URL="http://localhost:8080"
+HOST1_SABNZBD_API_KEY="your-api-key"
+HOST1_QBIT_URL="http://localhost:8090"
+HOST1_QBIT_USERNAME="admin"
+HOST1_QBIT_PASSWORD="your-password"
+
+# ── Docker Watchdog ────────────────────────────────────────────────────────────
+DOCKER_WATCHDOG_INTERVAL=900         # seconds between cycles (15 minutes)
+WATCHDOG_STARTUP_GRACE=600           # seconds before restarts begin after boot
+CONTAINER_DELAY=15                   # seconds between dependency + dependent restart
+
+SOFT_MEM_THRESHOLD=80                # warn at % of hard limit (no restart)
+
+SOFT_CPU_THRESHOLD=50
+HARD_CPU_THRESHOLD=85
+CPU_FAIL_LIMIT=2
+
+CURL_TIMEOUT=5
+RESP_FAIL_LIMIT=2
+
+WATCHDOG_CONTAINER_RESTART_LIMIT=3
+WATCHDOG_CONTAINER_RESTART_WINDOW=1
+
+WATCHDOG_SCAN_ALL=true
+WATCHDOG_SCAN_IGNORE=()
+WATCHDOG_RESTART_UNHEALTHY=true
+WATCHDOG_NOTIFY_OOM=true
+WATCHDOG_NOTIFY_CRASHLOOP=true
+WATCHDOG_CRASH_LIMIT=5
+WATCHDOG_RESTART_DEAD=true
+WATCHDOG_RESTART_CRASHED=true
+WATCHDOG_BATCH_NOTIFY=true
+
+# State files:
+WATCHDOG_STATE_FILE="/tmp/watchdog_state.db"
+SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
+WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
+SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
+
+# host*.conf
+HOST1_WATCHDOG_CONTAINERS=()          # "ContainerName:LimitMB"
+HOST1_WATCHDOG_CONTAINER_URLS=()      # "ContainerName:http://host:port"
+HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
+HOST1_WATCHDOG_DEPENDENCIES=()        # "Dependent:dep1 dep2"
+
+# ── Storage Watchdog ───────────────────────────────────────────────────────────
+WATCHDOG_CHECK_APPDATA=true
+WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
+WATCHDOG_APPDATA_GROWTH_GB=2
+WATCHDOG_APPDATA_LOG_MAX_GB=2
+WATCHDOG_APPDATA_TRUNCATE_LOGS=false
+WATCHDOG_APPDATA_STRIKE_LIMIT=3
+WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
+STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"
+
+# host*.conf (optional — only for suppress ceilings)
+# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
+#     ["Tdarr"]="25600"
+# )
+
+# ── System Watchdog ────────────────────────────────────────────────────────────
+SYS_WATCHDOG_STRIKE_LIMIT=2
+SYSTEM_WATCHDOG_INTERVAL=300
+
+SYS_WATCHDOG_REBOOT_WINDOW_HRS=2
+SYS_WATCHDOG_MAX_REBOOTS=3
+SYS_WATCHDOG_OOM_LIMIT=3
+
+MEM_WARN_GB=10
+MEM_SHUTDOWN_GB=6
+MEM_GB=4
+MEM_RECOVER_GB=30
+
+SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99
+SYS_WATCHDOG_FD_CRITICAL_PCT=95
+SYS_WATCHDOG_LOAD_MULTIPLIER=4
+SYS_WATCHDOG_CPU_TEMP=85
+SYS_WATCHDOG_ZOMBIES=20
+SYS_WATCHDOG_VAR_LOG_PCT=80
+SYS_WATCHDOG_TMP_PCT=85
+
+SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
+SYS_WATCHDOG_ABORT_ON_PARITY=true
+SYS_WATCHDOG_ABORT_ON_MOVER=true
+
+SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
+    "NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
+)
+SYS_WATCHDOG_REQUIRED_CONTAINERS=()
+
+# State files:
+SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
+SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
+SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
+SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
+```
+
+---
+
+## Troubleshooting
+
+### resource_watchdog Paused Containers It Shouldn't
+
+```bash
+# Check current state:
+resource_watchdog.sh --status
+
+# Add to RW_CRITICAL_CONTAINERS in master.conf:
+RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
+
+# Un-pause manually if needed:
+docker unpause MyContainer
+```
+
+### docker_watchdog Keeps Restarting a Healthy Container
+
+```bash
+# Check what's triggering it — memory, CPU, HTTP, or required:
+docker_watchdog.sh --status
+
+# Check CPU normalised — brief spikes should not trigger (2-strike system):
+# If triggering on CPU: check if HARD_CPU_THRESHOLD is set appropriately
+# for containers with legitimate burst usage (Tdarr encoding, SABnzbd unpacking)
+
+# Check HTTP — is the health endpoint returning 200?
+curl -sf --max-time 5 http://localhost:PORT && echo "OK" || echo "FAIL"
+```
+
+### Container on the Skip List After Fixing the Problem
+
+```bash
+# See skip list and container state:
+Tools/watchdog_skip_list_manager.sh --status
+
+# Fix root cause first, then clear:
+Tools/watchdog_skip_list_manager.sh --clear ContainerName
+
+# Start manually to confirm fix before handing back to watchdog:
+docker start ContainerName
+```
+
+### storage_watchdog Alerting on a Container That Grows Legitimately
+
+```bash
+# Check what triggered it:
+storage_watchdog.sh --status
+
+# Add a suppress ceiling to host*.conf:
+# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
+#     ["ContainerName"]="10240"    # 10GB ceiling — legitimate growth, suppress below this
+# )
+```
+
+### system_watchdog Rebooted Unexpectedly
+
+```bash
+# Check the reboot log (survives reboots):
+cat /boot/config/system_watchdog_reboots.db
+# Shows timestamp and reason for each watchdog-triggered reboot
+
+# Check syslog near the reboot time:
+grep "system_watchdog" /var/log/syslog | tail -20
+```
+
+### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
+
+```bash
+# All four watchdogs run as single-pass scripts — there is no background process to check.
+# If the orchestrator reports a timeout, one pass took longer than expected.
+
+# Check the orchestrator itself:
+Orchestrators/watchdog_orchestrator.sh --status
+
+# Run the slow watchdog directly with --log to see where it's hanging:
+Watchdogs/system_watchdog.sh --log --dry-run
+```