# ━━━━━ WATCHDOGS — Manual ━━━━━

Configuration reference, operational procedures, and troubleshooting for all watchdog
scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.

---

## ━━━ CONTENTS ━━━

- [Output Tiers](#output-tiers)
- [resource_watchdog.sh](#resource_watchdogsh)
- [docker_watchdog.sh](#docker_watchdogsh)
- [system_watchdog.sh](#system_watchdogsh)
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
- [stability_watchdog.sh](#stability_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)

---

## Output Tiers

All watchdog scripts use a two-tier output model: `echo` lines are always visible;
`log` lines only appear when `--log` is passed.

All watchdog scripts are **single-pass scripts** called once every 15 minutes by the orchestrator.
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
visible. Per-check detail is suppressed on clean cycles.

`docker_watchdog.sh` is the exception — it is **silent on clean cycles by design**.
96 cycles/day means clean-cycle noise would bury real events. Its output only appears
when there are restarts, skip-list events, or RAM deferral. Use `--log` to see
per-cycle detail on clean cycles.

---

## resource_watchdog.sh

Runs first in the orchestrator sequence. Reduces system pressure before docker_watchdog
attempts any container restarts. Containers restarted into a RAM-pressured system just
fail again — this script ensures docker_watchdog has breathing room.

### Pressure Levels

Three escalating levels, each additive:

```bash
# master.conf
RW_RAM_SOFT_GB=20           # Level 1 trigger — throttle downloaders
RW_RAM_MEDIUM_GB=15         # Level 2 trigger — throttle + pause containers
RW_RAM_HARD_GB=10           # Level 3 trigger — stop containers + defer docker_watchdog
RW_RAM_RECOVER_GB=25        # de-escalate only after RAM reaches this

RW_LOAD_SOFT_MULTIPLIER=2.0    # load > 2× cpu count = level 1
RW_LOAD_MEDIUM_MULTIPLIER=3.0  # load > 3× cpu count = level 2

RW_RECOVER_CYCLES=3         # consecutive under-threshold runs before de-escalating
```

**Level 1 (soft):** Throttle SABnzbd + qBittorrent download speeds.
**Level 2 (medium):** Further throttle + `docker pause` non-critical containers.
**Level 3 (hard):** `docker stop` optional services + write `mem_shutdown_active=true`
to `RW_STATE_FILE`. docker_watchdog.sh reads this flag and skips all restart logic
until pressure clears. Without this coordination, docker_watchdog would immediately
restart containers that resource_watchdog just stopped to free RAM.

Recovery de-escalates one level at a time — prevents flip-flopping between states.

### Per-Host Container Lists

```bash
# host1.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")        # paused at level 2
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")    # stopped at level 3
```

Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
pressure level. Default: Emby, NginxProxyManager, Authelia, Mariadb, Redis.

### Downloader Throttle Config

```bash
# master.conf
RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"        # throttled at level 1
RW_SABNZBD_SPEED_MEDIUM="10M"      # throttled further at level 2

RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200              # KB/s — level 1
RW_QBIT_DL_MEDIUM=10240            # KB/s — level 2

# host1.conf (API access)
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"
```

### Usage

```bash
resource_watchdog.sh            # single pass (called by watchdog_orchestrator.sh)
resource_watchdog.sh --dry-run  # show what would be throttled/paused/stopped
resource_watchdog.sh --status   # current level, active actions, recovery cycle count
resource_watchdog.sh --log      # verbose per-check output
```

---

## docker_watchdog.sh

Runs second in the orchestrator sequence. Two-tier container healing — explicit
per-container configuration (Tier 1) plus a global catch-all scan (Tier 2).

Reads `RW_STATE_FILE` at cycle start — if `mem_shutdown_active=true`, skips all
container restart logic (resource_watchdog is managing the situation). Health URL
checks for excluded containers still run.

### Memory Hard Limits

```bash
# host1.conf
# Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
#
# Sizing: check normal peak with "docker stats ContainerName"
# Set limit at ~150-200% of normal peak
#
HOST1_WATCHDOG_CONTAINERS=(
    "Emby:18432"        # 18GB — peaks ~12GB under heavy transcode load
    "LidaTube:6144"     # 6GB  — YouTube downloader, grows with large queues
    "Tdarr:6144"        # 6GB  — video transcoder, memory-intensive by nature
    "Code-Server:1024"  # 1GB  — IDE, should be light; 1GB is generous
)
```

A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
visibility into a container approaching its ceiling before a restart is triggered.

### CPU Thresholds

```bash
# master.conf
# CPU is normalised against total core count.
# 85% normalised on a 16-core machine = 13.6 cores worth of a single process.
#
# CPU uses a STRIKE SYSTEM — brief spikes are normal (Tdarr, Emby transcoding, SABnzbd).
# Strike 1: above HARD_CPU_THRESHOLD → warn, increment strike
# Strike 2: above threshold → restart, reset counter
# Recovery: drops below threshold any cycle → reset to 0
#
SOFT_CPU_THRESHOLD=50    # warn at 50% normalised — informational only
HARD_CPU_THRESHOLD=85    # strike at 85% normalised
CPU_FAIL_LIMIT=2         # consecutive strikes before restart
```

### HTTP Health Checks

```bash
# host1.conf
# Format: "ContainerName:http://host:port/optional-path"
# Two consecutive non-responses trigger a restart.
# "Container running" and "service responding" are not the same thing.
#
HOST1_WATCHDOG_CONTAINER_URLS=(
    "Emby:http://localhost:8096"            # Emby WebUI root
    "NginxProxyManager:http://localhost:81" # NPM admin interface
)

# master.conf
CURL_TIMEOUT=5      # seconds before non-response counts as a failure
RESP_FAIL_LIMIT=2   # consecutive failures before restart
```

### Required Containers

```bash
# host1.conf
# Found stopped → restart attempted every cycle until running or skip-listed.
#
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
    "NginxProxyManager"
    "Authelia"
    "Mariadb-Authelia"
    "Redis-Authelia"
)
```

### Dependency Ordering

```bash
# host1.conf
# Format: "DependentContainer:dependency1 dependency2"
# All dependencies must be running before the dependent is restarted.
# Prevents Authelia crash-looping while MariaDB is still starting.
#
HOST1_WATCHDOG_DEPENDENCIES=(
    "Authelia:Mariadb-Authelia Redis-Authelia"
    "NextCloud:Postgres-NextCloud"
)
```

Same dependency config is used by `docker_daily_restart.sh` and
`docker_weekly_restart.sh` — configure once, applies everywhere.

### Startup Grace Period

```bash
# master.conf
# Suppress restart actions for N seconds after array start.
# Checks still run and log — only restart actions are suppressed.
#
WATCHDOG_STARTUP_GRACE=600   # 10 minutes
```

### Tier 2 Global Scan

```bash
# master.conf
WATCHDOG_SCAN_ALL=true               # enable global scan
WATCHDOG_RESTART_UNHEALTHY=true      # act on Docker HEALTHCHECK failures
WATCHDOG_NOTIFY_OOM=true             # detect and notify kernel OOM kills
WATCHDOG_NOTIFY_CRASHLOOP=true       # detect escalating restart counts
WATCHDOG_RESTART_DEAD=true           # recover containers in dead state
WATCHDOG_RESTART_CRASHED=true        # restart containers that exited non-zero
WATCHDOG_CRASH_LIMIT=5               # RestartCount above this → restart + skip list

# Exclude containers from Tier 2 (one-shots, manually managed, benign exits):
WATCHDOG_SCAN_IGNORE=(
    "my-one-shot-container"
)
```

### Restart Loop Protection

```bash
# master.conf
WATCHDOG_CONTAINER_RESTART_LIMIT=3    # restarts in the window before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1   # rolling window in hours
```

After hitting the limit: skip list + critical notification. The watchdog stops
touching the container. Auto-clear: if the container recovers on its own and is
found running, it's removed from the skip list automatically. Manual clear is only
needed when the container is stuck stopped — use `Tools/watchdog_skip_list_manager.sh`.

### Notification Batching

```bash
# master.conf
WATCHDOG_BATCH_NOTIFY=true
# All events from one cycle → one notification at the end.
# A shared DB going down can cascade 10+ containers. Without batching: 10 pings.
# With batching: one summary listing all affected containers.
```

### Usage

```bash
docker_watchdog.sh           # single pass (called by watchdog_orchestrator.sh)
docker_watchdog.sh --dry-run # full cycle preview without restarting anything
docker_watchdog.sh --status  # skip list, strike counts, grace period, RAM deferral state
docker_watchdog.sh --log     # verbose per-cycle output
```

### Skip List Recovery

```bash
# Step 1 — understand the situation
Tools/watchdog_skip_list_manager.sh --status

# Step 2 — fix the underlying problem
# docker logs ContainerName --tail 100
# df -h /mnt/user

# Step 3 — clear the container
Tools/watchdog_skip_list_manager.sh --clear ContainerName

# Step 4 — start manually (confirms fix before handing back to watchdog)
docker start ContainerName

# Step 5 — monitoring resumes automatically on next cycle
```

---

## system_watchdog.sh

Thin orchestrator — runs `SYSTEM_WATCHDOG_SCRIPTS` from master.conf sequentially each cycle.
Called third by `watchdog_orchestrator.sh`. Covers all system component watchdogs.
Can also be run standalone to check all system components at once.

### Usage

```bash
system_watchdog.sh              # run all system component watchdogs
system_watchdog.sh --status     # show configured scripts and their paths
system_watchdog.sh --dry-run    # preview without executing anything
system_watchdog.sh --log        # verbose output
```

---

## System/storage_watchdog.sh

Called by `system_watchdog.sh` each cycle. Two independent checks per cycle:
growth rate detection (automatic, zero config) and oversize log detection.
Uses its own strike state file — independent from docker_watchdog.

### Growth Rate Detection

```bash
# master.conf
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2          # growth per cycle that triggers a strike
WATCHDOG_APPDATA_STRIKE_LIMIT=3       # strikes before alert
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"   # size baseline
```

Runs `du -sm appdata/*/` each cycle — pure inode metadata, very lightweight on NVMe.
Compares each container's current size to the baseline from the previous cycle.
Growth > `WATCHDOG_APPDATA_GROWTH_GB` per cycle increments the container's strike count.
Strike 1: warn. Strike 2: escalate. Strike 3: critical alert.
Strikes auto-clear when growth drops to zero (condition resolved).

**Zero configuration required for new containers.** Growth rate detection covers all
containers automatically. The suppress array below is only for known-legitimate growth.

### Growth Suppress Ceilings

```bash
# host1.conf
# ONLY needed in specific cases — growth rate detection covers everything automatically.
# Use when a container's appdata legitimately grows fast during normal operation
# and you want to suppress false positives above a known-safe threshold.
#
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
    ["Tdarr"]="25600"       # 25GB — transcode cache grows during active jobs
    ["7dtd"]="20480"        # 20GB — game server world data, expected large
)
```

### Log File Detection

```bash
# master.conf
WATCHDOG_APPDATA_LOG_MAX_GB=2         # *.log / *.log.* files above this trigger a strike
WATCHDOG_APPDATA_TRUNCATE_LOGS=false  # true: truncate at strike limit; false: alert only
```

Scans all `*.log` and `*.log.*` files across all appdata paths. Files over the threshold
increment per-file strike counts. At strike limit: truncates in-place with `truncate -s 0`
(container keeps its file handle — space reclaimed immediately without container restart)
or sends a critical alert if truncation is disabled.

Log strikes auto-clear when the file drops below threshold.

### Usage

```bash
storage_watchdog.sh            # single pass (called by system_watchdog.sh)
storage_watchdog.sh --status   # strikes, growth baseline age, suppress ceilings
storage_watchdog.sh --dry-run  # show what would be alerted/truncated
storage_watchdog.sh --log      # verbose per-container output
```

---

## System/webgui_watchdog.sh

Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
through three restart steps if unresponsive. Silent when healthy.

### Escalation Path

```
WebGUI responding → exit 0 (silent)

Not responding:
  Step 1 — nginx restart       → wait WEBGUI_NGINX_WAIT → recheck
  Step 2 — php-fpm restart     → wait WEBGUI_PHP_WAIT   → recheck
  Step 3 — emhttp restart      → wait WEBGUI_EMHTTP_WAIT → recheck
  All three failed → critical notify, manual intervention needed
```

### Configuration

```bash
# master.conf
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30
```

### Usage

```bash
webgui_watchdog.sh             # single pass (called by system_watchdog.sh)
webgui_watchdog.sh --status    # current WebGUI state + nginx/php-fpm/emhttp status
webgui_watchdog.sh --dry-run   # show which services would be restarted
webgui_watchdog.sh --log       # verbose per-step output
```

---

## System/network_watchdog.sh

Called by `system_watchdog.sh` each cycle. Checks that the outside world can actually
reach what it needs to reach. Internet reachability gates all other checks — if upstream
is down, DDNS and NPM checks are skipped to prevent false positives.

### Check Sequence

```
1. Internet → curl NETWORK_WATCHDOG_INTERNET_URL
   fail → alert + exit (skip all remaining checks)

2. DDNS → public IP (ifconfig.me) vs DNS record (dig @1.1.1.1)
   match  → pass (silent)
   mismatch → restart HOST*_NETWORK_WATCHDOG_DDNS_CONTAINER + notify

3. Tailscale → tailscale status --json BackendState
   Running → pass (silent)
   not Running → notify (no auto-restart — warrants human review)

4. NPM proxy → curl HOST*_NETWORK_WATCHDOG_NPM_URL (external)
   reachable → pass, clear strikes
   not reachable → strike 1: warn + notify
                   strike 2: restart NginxProxyManager + notify + clear strikes
```

### Configuration

```bash
# master.conf
NETWORK_WATCHDOG_ENABLED=true
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"

# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
```

### Usage

```bash
network_watchdog.sh              # run all connectivity checks (silent when healthy)
network_watchdog.sh --status     # current IP, DNS record, Tailscale state, NPM strike count
network_watchdog.sh --dry-run    # check without restarting any containers
network_watchdog.sh --log        # verbose per-check output
```

### Troubleshooting

**DDNS keeps restarting the container but record stays stale**
```bash
# Check if the container is actually running after restart:
docker ps | grep "Gmer4Lfe.com"

# Check container logs for Cloudflare API errors:
docker logs "Gmer4Lfe.com" --tail 20

# Verify public IP detection:
curl -sf https://ifconfig.me

# Verify DNS resolution:
dig +short gmer4lfe.com @1.1.1.1
```

**NPM strikes accumulating but NPM is running**
```bash
# Check if the external URL is actually responding:
curl -sv https://gmer4lfe.com 2>&1 | head -20

# NPM may be running but a backend container is down — check the specific service
# the URL routes to, not just NginxProxyManager itself.

# Check NPM strike count:
network_watchdog.sh --status
```

**Tailscale showing not Running**
```bash
# Check tailscale state directly:
tailscale status

# Check the backend state specifically:
tailscale status --json | grep BackendState

# Reconnect manually if needed:
tailscale up
```

---

## stability_watchdog.sh

Runs last in the orchestrator sequence. The only script in the ecosystem authorized
to reboot. Watches the server itself — not containers, not storage. Reboots only when
healing at every other layer has failed or when the failure is non-recoverable.

### Three-Tier Response

**Tier 1 — CRITICAL (immediate reboot, no strikes)**
| Condition | Threshold | Why immediate |
|-----------|-----------|---------------|
| Docker daemon unresponsive | N/A | Every docker command hangs — nothing can be healed |
| rootfs usage | `SYS_WATCHDOG_ROOTFS_CRITICAL_PCT` (99%) | SSH stops; state files fail silently |
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
| File descriptor exhaustion | `SYS_WATCHDOG_FD_CRITICAL_PCT` (95%) | New connections silently failing |
| /boot read-only | write test fails | Config writes silently failing |

**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
RAM below `SYS_WATCHDOG_MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`.
Both conditions required — RAM alone uses the standard strike system.
OOM confirms the system is dying faster than watchdogs can heal.

**Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKE_LIMIT` consecutive failures → reboot)**
| Check | Threshold |
|-------|-----------|
| Free RAM | `SYS_WATCHDOG_MEM_GB` (reboot trigger — earlier tiers handled by resource_watchdog) |
| Load average | `SYS_WATCHDOG_LOAD_MULTIPLIER` × cpu_count |
| CPU temperature | `SYS_WATCHDOG_CPU_TEMP_MAX` |
| Zombie processes | `SYS_WATCHDOG_ZOMBIE_LIMIT` |
| /var/log usage | `SYS_WATCHDOG_LOG_PCT` |
| /tmp usage | `SYS_WATCHDOG_TMP_PCT` |
| Array disk errors | mdstat error delta > 0 |
| NIC state | interface operstate != "up" |

### RAM Tiers

RAM pressure is a graduated response split across resource_watchdog and stability_watchdog:

```
RW_RAM_SOFT_GB (12GB)   → throttle downloads, reduce background load  (resource_watchdog)
RW_RAM_MEDIUM_GB (8GB)  → pause background containers                 (resource_watchdog)
RW_RAM_HARD_GB (6GB)    → stop optional containers, wait for recovery  (resource_watchdog)
SYS_WATCHDOG_MEM_GB (4GB) → strike → reboot (last resort)             (stability_watchdog)
RW_RAM_RECOVER_GB (20GB) → RAM must reach this before stopped containers restart
```

`SYS_WATCHDOG_MEM_GB` must always be below `RW_RAM_HARD_GB` — resource_watchdog acts first.
Containers stopped at the hard tier use `HOST*_RW_STOP_CONTAINERS` in host*.conf.
Containers paused at the medium tier use `HOST*_RW_PAUSE_CONTAINERS` in host*.conf.

### Abort Conditions

Prevent reboot — running them would risk data loss:

```bash
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true   # ZFS pool degraded/faulted
SYS_WATCHDOG_ABORT_ON_PARITY=false         # aborting parity mid-check is worse than crashing
SYS_WATCHDOG_ABORT_ON_MOVER=false          # aborting move mid-run is worse than crashing
```

Tier 1 CRITICAL bypasses all abort conditions — an imminent crash outweighs data
safety concerns.

### Reboot Rate Limit

```bash
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2   # window in hours
SYS_WATCHDOG_REBOOT_LIMIT=3         # max reboots within the window
```

If the server reboots `SYS_WATCHDOG_REBOOT_LIMIT` times within the window, the watchdog
switches from rebooting to notifying only. Prevents a boot loop where the watchdog
reboots → something crashes again immediately → reboot again.

### Usage

```bash
stability_watchdog.sh              # single pass (called by watchdog_orchestrator.sh)
stability_watchdog.sh --dry-run    # run detection logic without rebooting
stability_watchdog.sh --status     # thresholds, current state, strike counts
stability_watchdog.sh --log        # verbose per-check output
```

---

## Full Configuration Reference

```bash
# master.conf

# ── Resource Watchdog ──────────────────────────────────────────────────────────
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"

RW_RAM_SOFT_GB=20
RW_RAM_MEDIUM_GB=15
RW_RAM_HARD_GB=10
RW_RAM_RECOVER_GB=25

RW_LOAD_SOFT_MULTIPLIER=2.0
RW_LOAD_MEDIUM_MULTIPLIER=3.0
RW_RECOVER_CYCLES=3

RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"
RW_SABNZBD_SPEED_MEDIUM="10M"
RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200        # KB/s
RW_QBIT_DL_MEDIUM=10240

RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")

# host*.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"

# ── Docker Watchdog ────────────────────────────────────────────────────────────
WATCHDOG_STARTUP_GRACE=600           # seconds before restarts begin after boot
CONTAINER_DELAY=15                   # seconds between dependency + dependent restart

SOFT_MEM_THRESHOLD=80                # warn at % of hard limit (no restart)

SOFT_CPU_THRESHOLD=50
HARD_CPU_THRESHOLD=85
CPU_FAIL_LIMIT=2

CURL_TIMEOUT=5
RESP_FAIL_LIMIT=2

WATCHDOG_CONTAINER_RESTART_LIMIT=3
WATCHDOG_CONTAINER_RESTART_WINDOW=1

WATCHDOG_SCAN_ALL=true
WATCHDOG_SCAN_IGNORE=()
WATCHDOG_RESTART_UNHEALTHY=true
WATCHDOG_NOTIFY_OOM=true
WATCHDOG_NOTIFY_CRASHLOOP=true
WATCHDOG_CRASH_LIMIT=5
WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
WATCHDOG_BATCH_NOTIFY=true

# State files:
WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"

# host*.conf
HOST1_WATCHDOG_CONTAINERS=()          # "ContainerName:LimitMB"
HOST1_WATCHDOG_CONTAINER_URLS=()      # "ContainerName:http://host:port"
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
HOST1_WATCHDOG_DEPENDENCIES=()        # "Dependent:dep1 dep2"

# ── Storage Watchdog ───────────────────────────────────────────────────────────
WATCHDOG_CHECK_APPDATA=true
WATCHDOG_APPDATA_PATHS=("/mnt/docker-unraid/appdata")
WATCHDOG_APPDATA_GROWTH_GB=2
WATCHDOG_APPDATA_LOG_MAX_GB=2
WATCHDOG_APPDATA_TRUNCATE_LOGS=false
WATCHDOG_APPDATA_STRIKE_LIMIT=3
WATCHDOG_APPDATA_GROWTH_FILE="/tmp/watchdog_appdata_growth.db"
STORAGE_WATCHDOG_STATE_FILE="/tmp/storage_watchdog_state.db"

# host*.conf (optional — only for suppress ceilings)
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
#     ["Tdarr"]="25600"
# )

# ── System Watchdog ────────────────────────────────────────────────────────────
# Reboot trigger only — warn/shutdown/recover RAM tiers handled by resource_watchdog.sh
# RW_RAM_HARD_GB > SYS_WATCHDOG_MEM_GB always (resource_watchdog acts before watchdog reboots)
SYS_WATCHDOG_STRIKE_LIMIT=2

SYS_WATCHDOG_REBOOT_WINDOW_HRS=12
SYS_WATCHDOG_REBOOT_LIMIT=3
SYS_WATCHDOG_OOM_LIMIT=3

SYS_WATCHDOG_MEM_GB=4              # strike system → reboot (last resort — below resource_watchdog hard stop)

SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99
SYS_WATCHDOG_FD_CRITICAL_PCT=95
SYS_WATCHDOG_LOAD_MULTIPLIER=4
SYS_WATCHDOG_CPU_TEMP_MAX=95
SYS_WATCHDOG_ZOMBIE_LIMIT=50
SYS_WATCHDOG_LOG_PCT=95
SYS_WATCHDOG_TMP_PCT=90

SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=false
SYS_WATCHDOG_ABORT_ON_MOVER=false

# State files (all in State_Files/ — survive reboots):
SYS_WATCHDOG_STATE_FILE="State_Files/system_watchdog_state.db"
DOCKER_WATCHDOG_FAILED_FILE="State_Files/docker_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="State_Files/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="State_Files/system_watchdog_oom.db"

# ── Network Watchdog ───────────────────────────────────────────────────────────
NETWORK_WATCHDOG_ENABLED=true
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"

# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
```

---

## Troubleshooting

### resource_watchdog Paused Containers It Shouldn't

```bash
# Check current state:
resource_watchdog.sh --status

# Add to RW_CRITICAL_CONTAINERS in master.conf:
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")

# Un-pause manually if needed:
docker unpause MyContainer
```

### docker_watchdog Keeps Restarting a Healthy Container

```bash
# Check what's triggering it — memory, CPU, HTTP, or required:
docker_watchdog.sh --status

# Check CPU normalised — brief spikes should not trigger (2-strike system):
# If triggering on CPU: check if HARD_CPU_THRESHOLD is set appropriately
# for containers with legitimate burst usage (Tdarr encoding, SABnzbd unpacking)

# Check HTTP — is the health endpoint returning 200?
curl -sf --max-time 5 http://localhost:PORT && echo "OK" || echo "FAIL"
```

### Container on the Skip List After Fixing the Problem

```bash
# See skip list and container state:
Tools/watchdog_skip_list_manager.sh --status

# Fix root cause first, then clear:
Tools/watchdog_skip_list_manager.sh --clear ContainerName

# Start manually to confirm fix before handing back to watchdog:
docker start ContainerName
```

### storage_watchdog Alerting on a Container That Grows Legitimately

```bash
# Check what triggered it:
storage_watchdog.sh --status

# Add a suppress ceiling to host*.conf:
# declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
#     ["ContainerName"]="10240"    # 10GB ceiling — legitimate growth, suppress below this
# )
```

### stability_watchdog Rebooted Unexpectedly

```bash
# Check the reboot log (survives reboots):
cat /boot/config/system_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot

# Check syslog near the reboot time:
grep "stability_watchdog" /var/log/syslog | tail -20
```

### stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout

```bash
# All watchdogs run as single-pass scripts — there is no background process to check.
# If the orchestrator reports a timeout, one pass took longer than expected.

# Check the orchestrator itself:
Orchestrators/watchdog_orchestrator.sh --status

# Run the slow watchdog directly with --log to see where it's hanging:
Watchdogs/stability_watchdog.sh --log --dry-run
```