Docs: add network_watchdog to README-Watchdogs and Manual-Watchdogs
README-Watchdogs: tier diagram, folder relationship, script table, execution flow, state file table Manual-Watchdogs: TOC entry, full System/network_watchdog.sh section (check sequence, config, usage, troubleshooting), config reference block
This commit is contained in:
@@ -14,6 +14,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
|
||||
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
|
||||
- [stability_watchdog.sh](#stability_watchdogsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
@@ -409,6 +410,102 @@ webgui_watchdog.sh --log # verbose per-step output
|
||||
|
||||
---
|
||||
|
||||
## System/network_watchdog.sh
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Checks that the outside world can actually
|
||||
reach what it needs to reach. Internet reachability gates all other checks — if upstream
|
||||
is down, DDNS and NPM checks are skipped to prevent false positives.
|
||||
|
||||
### Check Sequence
|
||||
|
||||
```
|
||||
1. Internet → curl NETWORK_WATCHDOG_INTERNET_URL
|
||||
fail → alert + exit (skip all remaining checks)
|
||||
|
||||
2. DDNS → public IP (ifconfig.me) vs DNS record (dig @1.1.1.1)
|
||||
match → pass (silent)
|
||||
mismatch → restart HOST*_NETWORK_WATCHDOG_DDNS_CONTAINER + notify
|
||||
|
||||
3. Tailscale → tailscale status --json BackendState
|
||||
Running → pass (silent)
|
||||
not Running → notify (no auto-restart — warrants human review)
|
||||
|
||||
4. NPM proxy → curl HOST*_NETWORK_WATCHDOG_NPM_URL (external)
|
||||
reachable → pass, clear strikes
|
||||
not reachable → strike 1: warn + notify
|
||||
strike 2: restart NginxProxyManager + notify + clear strikes
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
NETWORK_WATCHDOG_ENABLED=true
|
||||
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
|
||||
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||
|
||||
# host*.conf (host-specific)
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
network_watchdog.sh # run all connectivity checks (silent when healthy)
|
||||
network_watchdog.sh --status # current IP, DNS record, Tailscale state, NPM strike count
|
||||
network_watchdog.sh --dry-run # check without restarting any containers
|
||||
network_watchdog.sh --log # verbose per-check output
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**DDNS keeps restarting the container but record stays stale**
|
||||
```bash
|
||||
# Check if the container is actually running after restart:
|
||||
docker ps | grep "Gmer4Lfe.com"
|
||||
|
||||
# Check container logs for Cloudflare API errors:
|
||||
docker logs "Gmer4Lfe.com" --tail 20
|
||||
|
||||
# Verify public IP detection:
|
||||
curl -sf https://ifconfig.me
|
||||
|
||||
# Verify DNS resolution:
|
||||
dig +short gmer4lfe.com @1.1.1.1
|
||||
```
|
||||
|
||||
**NPM strikes accumulating but NPM is running**
|
||||
```bash
|
||||
# Check if the external URL is actually responding:
|
||||
curl -sv https://gmer4lfe.com 2>&1 | head -20
|
||||
|
||||
# NPM may be running but a backend container is down — check the specific service
|
||||
# the URL routes to, not just NginxProxyManager itself.
|
||||
|
||||
# Check NPM strike count:
|
||||
network_watchdog.sh --status
|
||||
```
|
||||
|
||||
**Tailscale showing not Running**
|
||||
```bash
|
||||
# Check tailscale state directly:
|
||||
tailscale status
|
||||
|
||||
# Check the backend state specifically:
|
||||
tailscale status --json | grep BackendState
|
||||
|
||||
# Reconnect manually if needed:
|
||||
tailscale up
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## stability_watchdog.sh
|
||||
|
||||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||||
@@ -616,6 +713,20 @@ SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
|
||||
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
|
||||
|
||||
# ── Network Watchdog ───────────────────────────────────────────────────────────
|
||||
NETWORK_WATCHDOG_ENABLED=true
|
||||
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
|
||||
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||
|
||||
# host*.conf (host-specific)
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user