diff --git a/Watchdogs/Manual-Watchdogs.md b/Watchdogs/Manual-Watchdogs.md index b6713f1..bdf1839 100644 --- a/Watchdogs/Manual-Watchdogs.md +++ b/Watchdogs/Manual-Watchdogs.md @@ -14,6 +14,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches - [system_watchdog.sh](#system_watchdogsh) - [System/storage_watchdog.sh](#systemstorage_watchdogsh) - [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) +- [System/network_watchdog.sh](#systemnetwork_watchdogsh) - [stability_watchdog.sh](#stability_watchdogsh) - [Full Configuration Reference](#full-configuration-reference) - [Troubleshooting](#troubleshooting) @@ -409,6 +410,102 @@ webgui_watchdog.sh --log # verbose per-step output --- +## System/network_watchdog.sh + +Called by `system_watchdog.sh` each cycle. Checks that the outside world can actually +reach what it needs to reach. Internet reachability gates all other checks — if upstream +is down, DDNS and NPM checks are skipped to prevent false positives. + +### Check Sequence + +``` +1. Internet → curl NETWORK_WATCHDOG_INTERNET_URL + fail → alert + exit (skip all remaining checks) + +2. DDNS → public IP (ifconfig.me) vs DNS record (dig @1.1.1.1) + match → pass (silent) + mismatch → restart HOST*_NETWORK_WATCHDOG_DDNS_CONTAINER + notify + +3. Tailscale → tailscale status --json BackendState + Running → pass (silent) + not Running → notify (no auto-restart — warrants human review) + +4. NPM proxy → curl HOST*_NETWORK_WATCHDOG_NPM_URL (external) + reachable → pass, clear strikes + not reachable → strike 1: warn + notify + strike 2: restart NginxProxyManager + notify + clear strikes +``` + +### Configuration + +```bash +# master.conf +NETWORK_WATCHDOG_ENABLED=true +NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1" +NETWORK_WATCHDOG_INTERNET_TIMEOUT=5 +NETWORK_WATCHDOG_CHECK_TAILSCALE=true +NETWORK_WATCHDOG_NPM_TIMEOUT=10 +NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2 +NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db" + +# host*.conf (host-specific) +HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com" +HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com" +HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com" +``` + +### Usage + +```bash +network_watchdog.sh # run all connectivity checks (silent when healthy) +network_watchdog.sh --status # current IP, DNS record, Tailscale state, NPM strike count +network_watchdog.sh --dry-run # check without restarting any containers +network_watchdog.sh --log # verbose per-check output +``` + +### Troubleshooting + +**DDNS keeps restarting the container but record stays stale** +```bash +# Check if the container is actually running after restart: +docker ps | grep "Gmer4Lfe.com" + +# Check container logs for Cloudflare API errors: +docker logs "Gmer4Lfe.com" --tail 20 + +# Verify public IP detection: +curl -sf https://ifconfig.me + +# Verify DNS resolution: +dig +short gmer4lfe.com @1.1.1.1 +``` + +**NPM strikes accumulating but NPM is running** +```bash +# Check if the external URL is actually responding: +curl -sv https://gmer4lfe.com 2>&1 | head -20 + +# NPM may be running but a backend container is down — check the specific service +# the URL routes to, not just NginxProxyManager itself. + +# Check NPM strike count: +network_watchdog.sh --status +``` + +**Tailscale showing not Running** +```bash +# Check tailscale state directly: +tailscale status + +# Check the backend state specifically: +tailscale status --json | grep BackendState + +# Reconnect manually if needed: +tailscale up +``` + +--- + ## stability_watchdog.sh Runs last in the orchestrator sequence. The only script in the ecosystem authorized @@ -616,6 +713,20 @@ SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db" SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db" SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db" SYS_WATCHDOG_OOM_FILE="/tmp/system_watchdog_oom.db" + +# ── Network Watchdog ─────────────────────────────────────────────────────────── +NETWORK_WATCHDOG_ENABLED=true +NETWORK_WATCHDOG_INTERNET_URL="https://1.1.1.1" +NETWORK_WATCHDOG_INTERNET_TIMEOUT=5 +NETWORK_WATCHDOG_CHECK_TAILSCALE=true +NETWORK_WATCHDOG_NPM_TIMEOUT=10 +NETWORK_WATCHDOG_NPM_STRIKE_LIMIT=2 +NETWORK_WATCHDOG_NPM_STATE_FILE="/tmp/network_watchdog_state.db" + +# host*.conf (host-specific) +HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com" +HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com" +HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com" ``` --- diff --git a/Watchdogs/README-Watchdogs.md b/Watchdogs/README-Watchdogs.md index 071d9a1..0cf77ed 100644 --- a/Watchdogs/README-Watchdogs.md +++ b/Watchdogs/README-Watchdogs.md @@ -69,6 +69,7 @@ Container healing docker_watchdog.sh — memory, CPU, HTTP, required con System components system_watchdog.sh — thin orchestrator: storage + WebGUI health └─ System/ storage_watchdog.sh — pool growth rate + runaway log detection webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp + network_watchdog.sh — internet, DDNS sync, Tailscale, NPM proxy Last resort stability_watchdog.sh — reboot only when nothing else can recover ``` @@ -96,6 +97,7 @@ Orchestrators/ Watchdogs/System/ ← called by system_watchdog.sh each cycle storage_watchdog.sh pool growth rate + runaway log detection webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation + network_watchdog.sh internet reachability, DDNS sync, Tailscale, NPM proxy Tools/ watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list @@ -121,6 +123,7 @@ Docker_Essentials/ | `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute | | `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute | | `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute | +| `System/network_watchdog.sh` | Internet reachability, DDNS sync, Tailscale, NPM proxy | `system_watchdog.sh` — every minute | > `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh` > is in `Tools/`. Neither is a watchdog — they sit at the edges of this system. @@ -170,6 +173,12 @@ Every minute — watchdog_orchestrator.sh fires: │ still down → emhttp restart → wait → recheck │ all three failed → critical notify, manual intervention needed │ + │ └─ network_watchdog.sh + │ internet check → fail = alert + skip remaining (prevents false positives) + │ DDNS: public IP vs dig @1.1.1.1 → mismatch = restart Cloudflare DDNS container + │ Tailscale: status Running → pass; not running = notify (no auto-restart) + │ NPM proxy: curl external URL → 2-strike system → restart NginxProxyManager + │ Step 4 — stability_watchdog.sh checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon Tier 1 CRITICAL → immediate reboot (no strikes) @@ -189,3 +198,4 @@ Every minute — watchdog_orchestrator.sh fires: | `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list | | `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts | | `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate | +| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |