Docs: add network_watchdog to README-Watchdogs and Manual-Watchdogs
README-Watchdogs: tier diagram, folder relationship, script table, execution flow, state file table Manual-Watchdogs: TOC entry, full System/network_watchdog.sh section (check sequence, config, usage, troubleshooting), config reference block
This commit is contained in:
Gmer4Lfe
@@ -14,6 +14,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
|
||||
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
|
||||
- [stability_watchdog.sh](#stability_watchdogsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
@@ -409,6 +410,102 @@ webgui_watchdog.sh --log # verbose per-step output
|
||||
|
||||
---
|
||||
|
||||
## System/network_watchdog.sh
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Checks that the outside world can actually
|
||||
reach what it needs to reach. Internet reachability gates all other checks — if upstream
|
||||
is down, DDNS and NPM checks are skipped to prevent false positives.
|
||||
|
||||
### Check Sequence
|
||||
|
||||
```
|
||||
1. Internet → curl NETWORK_WATCHDOG_INTERNET_URL
|
||||
fail → alert + exit (skip all remaining checks)
|
||||
|
||||
2. DDNS → public IP (ifconfig.me) vs DNS record (dig @1.1.1.1)
|
||||
match → pass (silent)
|
||||
mismatch → restart HOST*_NETWORK_WATCHDOG_DDNS_CONTAINER + notify
|
||||
|
||||
3. Tailscale → tailscale status --json BackendState
|
||||
Running → pass (silent)
|
||||
not Running → notify (no auto-restart — warrants human review)
|
||||
|
||||
4. NPM proxy → curl HOST*_NETWORK_WATCHDOG_NPM_URL (external)
|
||||
reachable → pass, clear strikes
|
||||
not reachable → strike 1: warn + notify
|
||||
strike 2: restart NginxProxyManager + notify + clear strikes
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
NETWORK_WATCHDOG_ENABLED=true
|
||||
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
|
||||
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||
|
||||
# host*.conf (host-specific)
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
network_watchdog.sh # run all connectivity checks (silent when healthy)
|
||||
network_watchdog.sh --status # current IP, DNS record, Tailscale state, NPM strike count
|
||||
network_watchdog.sh --dry-run # check without restarting any containers
|
||||
network_watchdog.sh --log # verbose per-check output
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**DDNS keeps restarting the container but record stays stale**
|
||||
```bash
|
||||
# Check if the container is actually running after restart:
|
||||
docker ps | grep "Gmer4Lfe.com"
|
||||
|
||||
# Check container logs for Cloudflare API errors:
|
||||
docker logs "Gmer4Lfe.com" --tail 20
|
||||
|
||||
# Verify public IP detection:
|
||||
curl -sf https://ifconfig.me
|
||||
|
||||
# Verify DNS resolution:
|
||||
dig +short gmer4lfe.com @1.1.1.1
|
||||
```
|
||||
|
||||
**NPM strikes accumulating but NPM is running**
|
||||
```bash
|
||||
# Check if the external URL is actually responding:
|
||||
curl -sv https://gmer4lfe.com 2>&1 | head -20
|
||||
|
||||
# NPM may be running but a backend container is down — check the specific service
|
||||
# the URL routes to, not just NginxProxyManager itself.
|
||||
|
||||
# Check NPM strike count:
|
||||
network_watchdog.sh --status
|
||||
```
|
||||
|
||||
**Tailscale showing not Running**
|
||||
```bash
|
||||
# Check tailscale state directly:
|
||||
tailscale status
|
||||
|
||||
# Check the backend state specifically:
|
||||
tailscale status --json | grep BackendState
|
||||
|
||||
# Reconnect manually if needed:
|
||||
tailscale up
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## stability_watchdog.sh
|
||||
|
||||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||||
@@ -616,6 +713,20 @@ SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
|
||||
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
|
||||
|
||||
# ── Network Watchdog ───────────────────────────────────────────────────────────
|
||||
NETWORK_WATCHDOG_ENABLED=true
|
||||
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
|
||||
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||
|
||||
# host*.conf (host-specific)
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -69,6 +69,7 @@ Container healing docker_watchdog.sh — memory, CPU, HTTP, required con
|
||||
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
|
||||
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
|
||||
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
|
||||
network_watchdog.sh — internet, DDNS sync, Tailscale, NPM proxy
|
||||
Last resort stability_watchdog.sh — reboot only when nothing else can recover
|
||||
```
|
||||
|
||||
@@ -96,6 +97,7 @@ Orchestrators/
|
||||
Watchdogs/System/ ← called by system_watchdog.sh each cycle
|
||||
storage_watchdog.sh pool growth rate + runaway log detection
|
||||
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
|
||||
network_watchdog.sh internet reachability, DDNS sync, Tailscale, NPM proxy
|
||||
|
||||
Tools/
|
||||
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
|
||||
@@ -121,6 +123,7 @@ Docker_Essentials/
|
||||
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
|
||||
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute |
|
||||
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute |
|
||||
| `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every minute |
|
||||
|
||||
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
|
||||
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
|
||||
@@ -170,6 +173,12 @@ Every minute — watchdog_orchestrator.sh fires:
|
||||
│ still down → emhttp restart → wait → recheck
|
||||
│ all three failed → critical notify, manual intervention needed
|
||||
│
|
||||
│ └─ network_watchdog.sh
|
||||
│ internet check → fail = alert + skip remaining (prevents false positives)
|
||||
│ DDNS: public IP vs dig @1.1.1.1 → mismatch = restart Cloudflare DDNS container
|
||||
│ Tailscale: status Running → pass; not running = notify (no auto-restart)
|
||||
│ NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
|
||||
│
|
||||
Step 4 — stability_watchdog.sh
|
||||
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
|
||||
Tier 1 CRITICAL → immediate reboot (no strikes)
|
||||
@@ -189,3 +198,4 @@ Every minute — watchdog_orchestrator.sh fires:
|
||||
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
|
||||
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
|
||||
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
|
||||
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
||||
|
||||
Reference in New Issue
Block a user