Docs: add network_watchdog to README-Watchdogs and Manual-Watchdogs

README-Watchdogs: tier diagram, folder relationship, script table,
  execution flow, state file table
Manual-Watchdogs: TOC entry, full System/network_watchdog.sh section
  (check sequence, config, usage, troubleshooting), config reference block
This commit is contained in:
Gmer4Lfe
2026-05-22 20:54:04 -04:00
parent 4535804c5d
commit d05df84d00
2 changed files with 121 additions and 0 deletions
+111
View File
@@ -14,6 +14,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
- [system_watchdog.sh](#system_watchdogsh)
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
- [stability_watchdog.sh](#stability_watchdogsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)
@@ -409,6 +410,102 @@ webgui_watchdog.sh --log # verbose per-step output
---
## System/network_watchdog.sh
Called by `system_watchdog.sh` each cycle. Checks that the outside world can actually
reach what it needs to reach. Internet reachability gates all other checks — if upstream
is down, DDNS and NPM checks are skipped to prevent false positives.
### Check Sequence
```
1. Internet → curl NETWORK_WATCHDOG_INTERNET_URL
fail → alert + exit (skip all remaining checks)
2. DDNS → public IP (ifconfig.me) vs DNS record (dig @1.1.1.1)
match → pass (silent)
mismatch → restart HOST*_NETWORK_WATCHDOG_DDNS_CONTAINER + notify
3. Tailscale → tailscale status --json BackendState
Running → pass (silent)
not Running → notify (no auto-restart — warrants human review)
4. NPM proxy → curl HOST*_NETWORK_WATCHDOG_NPM_URL (external)
reachable → pass, clear strikes
not reachable → strike 1: warn + notify
strike 2: restart NginxProxyManager + notify + clear strikes
```
### Configuration
```bash
# master.conf
NETWORK_WATCHDOG_ENABLED=true
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
```
### Usage
```bash
network_watchdog.sh # run all connectivity checks (silent when healthy)
network_watchdog.sh --status # current IP, DNS record, Tailscale state, NPM strike count
network_watchdog.sh --dry-run # check without restarting any containers
network_watchdog.sh --log # verbose per-check output
```
### Troubleshooting
**DDNS keeps restarting the container but record stays stale**
```bash
# Check if the container is actually running after restart:
docker ps | grep "Gmer4Lfe.com"
# Check container logs for Cloudflare API errors:
docker logs "Gmer4Lfe.com" --tail 20
# Verify public IP detection:
curl -sf https://ifconfig.me
# Verify DNS resolution:
dig +short gmer4lfe.com @1.1.1.1
```
**NPM strikes accumulating but NPM is running**
```bash
# Check if the external URL is actually responding:
curl -sv https://gmer4lfe.com 2>&1 | head -20
# NPM may be running but a backend container is down — check the specific service
# the URL routes to, not just NginxProxyManager itself.
# Check NPM strike count:
network_watchdog.sh --status
```
**Tailscale showing not Running**
```bash
# Check tailscale state directly:
tailscale status
# Check the backend state specifically:
tailscale status --json | grep BackendState
# Reconnect manually if needed:
tailscale up
```
---
## stability_watchdog.sh
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
@@ -616,6 +713,20 @@ SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
# ── Network Watchdog ───────────────────────────────────────────────────────────
NETWORK_WATCHDOG_ENABLED=true
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
NETWORK_WATCHDOG_NPM_TIMEOUT=10
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
# host*.conf (host-specific)
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
```
---