Docs: add network_watchdog to README-Watchdogs and Manual-Watchdogs
README-Watchdogs: tier diagram, folder relationship, script table, execution flow, state file table Manual-Watchdogs: TOC entry, full System/network_watchdog.sh section (check sequence, config, usage, troubleshooting), config reference block
This commit is contained in:
@@ -14,6 +14,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
|||||||
- [system_watchdog.sh](#system_watchdogsh)
|
- [system_watchdog.sh](#system_watchdogsh)
|
||||||
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
||||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
|
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
|
||||||
|
- [System/network_watchdog.sh](#systemnetwork_watchdogsh)
|
||||||
- [stability_watchdog.sh](#stability_watchdogsh)
|
- [stability_watchdog.sh](#stability_watchdogsh)
|
||||||
- [Full Configuration Reference](#full-configuration-reference)
|
- [Full Configuration Reference](#full-configuration-reference)
|
||||||
- [Troubleshooting](#troubleshooting)
|
- [Troubleshooting](#troubleshooting)
|
||||||
@@ -409,6 +410,102 @@ webgui_watchdog.sh --log # verbose per-step output
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## System/network_watchdog.sh
|
||||||
|
|
||||||
|
Called by `system_watchdog.sh` each cycle. Checks that the outside world can actually
|
||||||
|
reach what it needs to reach. Internet reachability gates all other checks — if upstream
|
||||||
|
is down, DDNS and NPM checks are skipped to prevent false positives.
|
||||||
|
|
||||||
|
### Check Sequence
|
||||||
|
|
||||||
|
```
|
||||||
|
1. Internet → curl NETWORK_WATCHDOG_INTERNET_URL
|
||||||
|
fail → alert + exit (skip all remaining checks)
|
||||||
|
|
||||||
|
2. DDNS → public IP (ifconfig.me) vs DNS record (dig @1.1.1.1)
|
||||||
|
match → pass (silent)
|
||||||
|
mismatch → restart HOST*_NETWORK_WATCHDOG_DDNS_CONTAINER + notify
|
||||||
|
|
||||||
|
3. Tailscale → tailscale status --json BackendState
|
||||||
|
Running → pass (silent)
|
||||||
|
not Running → notify (no auto-restart — warrants human review)
|
||||||
|
|
||||||
|
4. NPM proxy → curl HOST*_NETWORK_WATCHDOG_NPM_URL (external)
|
||||||
|
reachable → pass, clear strikes
|
||||||
|
not reachable → strike 1: warn + notify
|
||||||
|
strike 2: restart NginxProxyManager + notify + clear strikes
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# master.conf
|
||||||
|
NETWORK_WATCHDOG_ENABLED=true
|
||||||
|
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
|
||||||
|
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||||
|
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||||
|
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||||
|
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||||
|
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||||
|
|
||||||
|
# host*.conf (host-specific)
|
||||||
|
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||||
|
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||||||
|
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
network_watchdog.sh # run all connectivity checks (silent when healthy)
|
||||||
|
network_watchdog.sh --status # current IP, DNS record, Tailscale state, NPM strike count
|
||||||
|
network_watchdog.sh --dry-run # check without restarting any containers
|
||||||
|
network_watchdog.sh --log # verbose per-check output
|
||||||
|
```
|
||||||
|
|
||||||
|
### Troubleshooting
|
||||||
|
|
||||||
|
**DDNS keeps restarting the container but record stays stale**
|
||||||
|
```bash
|
||||||
|
# Check if the container is actually running after restart:
|
||||||
|
docker ps | grep "Gmer4Lfe.com"
|
||||||
|
|
||||||
|
# Check container logs for Cloudflare API errors:
|
||||||
|
docker logs "Gmer4Lfe.com" --tail 20
|
||||||
|
|
||||||
|
# Verify public IP detection:
|
||||||
|
curl -sf https://ifconfig.me
|
||||||
|
|
||||||
|
# Verify DNS resolution:
|
||||||
|
dig +short gmer4lfe.com @1.1.1.1
|
||||||
|
```
|
||||||
|
|
||||||
|
**NPM strikes accumulating but NPM is running**
|
||||||
|
```bash
|
||||||
|
# Check if the external URL is actually responding:
|
||||||
|
curl -sv https://gmer4lfe.com 2>&1 | head -20
|
||||||
|
|
||||||
|
# NPM may be running but a backend container is down — check the specific service
|
||||||
|
# the URL routes to, not just NginxProxyManager itself.
|
||||||
|
|
||||||
|
# Check NPM strike count:
|
||||||
|
network_watchdog.sh --status
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tailscale showing not Running**
|
||||||
|
```bash
|
||||||
|
# Check tailscale state directly:
|
||||||
|
tailscale status
|
||||||
|
|
||||||
|
# Check the backend state specifically:
|
||||||
|
tailscale status --json | grep BackendState
|
||||||
|
|
||||||
|
# Reconnect manually if needed:
|
||||||
|
tailscale up
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## stability_watchdog.sh
|
## stability_watchdog.sh
|
||||||
|
|
||||||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||||||
@@ -616,6 +713,20 @@ SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
|
|||||||
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
||||||
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
|
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
|
||||||
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
|
SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db"
|
||||||
|
|
||||||
|
# ── Network Watchdog ───────────────────────────────────────────────────────────
|
||||||
|
NETWORK_WATCHDOG_ENABLED=true
|
||||||
|
NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1"
|
||||||
|
NETWORK_WATCHDOG_INTERNET_TIMEOUT=5
|
||||||
|
NETWORK_WATCHDOG_CHECK_TAILSCALE=true
|
||||||
|
NETWORK_WATCHDOG_NPM_TIMEOUT=10
|
||||||
|
NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2
|
||||||
|
NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db"
|
||||||
|
|
||||||
|
# host*.conf (host-specific)
|
||||||
|
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||||
|
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||||||
|
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -69,6 +69,7 @@ Container healing docker_watchdog.sh — memory, CPU, HTTP, required con
|
|||||||
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
|
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
|
||||||
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
|
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
|
||||||
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
|
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
|
||||||
|
network_watchdog.sh — internet, DDNS sync, Tailscale, NPM proxy
|
||||||
Last resort stability_watchdog.sh — reboot only when nothing else can recover
|
Last resort stability_watchdog.sh — reboot only when nothing else can recover
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -96,6 +97,7 @@ Orchestrators/
|
|||||||
Watchdogs/System/ ← called by system_watchdog.sh each cycle
|
Watchdogs/System/ ← called by system_watchdog.sh each cycle
|
||||||
storage_watchdog.sh pool growth rate + runaway log detection
|
storage_watchdog.sh pool growth rate + runaway log detection
|
||||||
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
|
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
|
||||||
|
network_watchdog.sh internet reachability, DDNS sync, Tailscale, NPM proxy
|
||||||
|
|
||||||
Tools/
|
Tools/
|
||||||
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
|
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
|
||||||
@@ -121,6 +123,7 @@ Docker_Essentials/
|
|||||||
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
|
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
|
||||||
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute |
|
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute |
|
||||||
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute |
|
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute |
|
||||||
|
| `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every minute |
|
||||||
|
|
||||||
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
|
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
|
||||||
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
|
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
|
||||||
@@ -170,6 +173,12 @@ Every minute — watchdog_orchestrator.sh fires:
|
|||||||
│ still down → emhttp restart → wait → recheck
|
│ still down → emhttp restart → wait → recheck
|
||||||
│ all three failed → critical notify, manual intervention needed
|
│ all three failed → critical notify, manual intervention needed
|
||||||
│
|
│
|
||||||
|
│ └─ network_watchdog.sh
|
||||||
|
│ internet check → fail = alert + skip remaining (prevents false positives)
|
||||||
|
│ DDNS: public IP vs dig @1.1.1.1 → mismatch = restart Cloudflare DDNS container
|
||||||
|
│ Tailscale: status Running → pass; not running = notify (no auto-restart)
|
||||||
|
│ NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager
|
||||||
|
│
|
||||||
Step 4 — stability_watchdog.sh
|
Step 4 — stability_watchdog.sh
|
||||||
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
|
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
|
||||||
Tier 1 CRITICAL → immediate reboot (no strikes)
|
Tier 1 CRITICAL → immediate reboot (no strikes)
|
||||||
@@ -189,3 +198,4 @@ Every minute — watchdog_orchestrator.sh fires:
|
|||||||
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
|
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
|
||||||
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
|
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
|
||||||
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
|
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
|
||||||
|
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
||||||
|
|||||||
Reference in New Issue
Block a user