Commit Graph
13 Commits
Author SHA1 Message Date
Gmer4Lfe 049f633667 Bound the watchdogs' docker calls — a hung daemon is what they exist to catch 2026-08-24 18:39:59 -04:00
Gmer4Lfe 77b24e9823 Resolve network watchdog host config after detect_hosts
MY_ID is not exported and is unset until detect_hosts runs, so the DDNS and NPM checks
were silently resolving empty config and skipping themselves every cycle.
2026-08-01 20:37:58 -04:00
Gmer4Lfe 6623d1e776 Fix dead/incorrect vars and consolidate duplicated logic into common.sh
Codebase-wide audit pass: fixed real bugs (SSH hangs missing BatchMode,
local-outside-function no-ops, variable name collisions, a truncated
ratio calc, wrong state-dir path, DARK vs NO_INTERNET drift, and more),
then pulled logic that was duplicated across multiple scripts — arr
cleanup safety gates, docker restart ordering, container maintenance
stop/restart, watchdog state-file helpers, partnership role resolution,
cert expiry checks, remote node discovery, and TMDB discovery scoring —
into common.sh so each now has a single implementation.
2026-07-03 23:52:33 -04:00
Gmer4Lfe 3964f6fb46 audit echo vs log across all scripts — outcomes always visible, verbose for per-item loops 2026-06-14 12:40:15 -04:00
Gmer4Lfe 27bfc21cb0 Platform-agnostic refactor: eliminate OS-specific hardcodes from core scripts
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.

New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.

Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
2026-06-14 00:59:19 -04:00
Gmer4Lfe 369a9e6c19 Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites
- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename)
- Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API
  for storage health, service management, mover, user scripts, notifications,
  disk temps, and platform command validation
- Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR,
  auto-source Plugin/$PLATFORM/adapter.sh after common.sh
- Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg,
  disks.ini, and validate_unraid_cmd calls with platform_*() functions across
  watchdogs, orchestrators, and System_Essentials scripts
- Update all documentation: rename refs, update webgui escalation logic,
  add platform adapter section to Plugin README, update main README with
  portability vision and corrected self-healing stack description
2026-06-04 18:14:34 -04:00
Gmer4Lfe 08fc551d9f Add verbose log() coverage across all watchdog scripts
docker_watchdog: config dump at startup (thresholds/limits), skip list shown when active, per-container healthy log for Tier1 required + mem/CPU monitored containers
stability_watchdog: config dump with all tier thresholds, log() on pass for rootfs/log/tmp/load/zombies/NIC checks (previously silent on clean)
resource_watchdog: config dump with all pressure thresholds and container lists, log normal pressure state with live RAM/load values
system_watchdog: per-script timing on each child script run
network_watchdog: config dump (internet URL, DDNS domain/container, NPM URL, strike limit)
storage_watchdog: config dump (growth threshold, log max, paths, suppress ceilings)
webgui_watchdog: log nginx worker and php-fpm worker counts on healthy check
2026-06-01 20:56:46 -04:00
Gmer4Lfe 5e89f334bb output: watchdog visibility — parity/grace/disabled exits now echo, docker_watchdog container count, api_key_renew status always visible 2026-06-01 20:22:52 -04:00
Gmer4Lfe ea9f39a803 watchdogs: make healthy-state outputs always visible
log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
2026-05-24 17:25:33 -04:00
Gmer4Lfe afd9129869 watchdogs: add echo banners to network/storage/stability
All watchdog scripts now open with a consistent echo banner:
  ━━━ <icon> <Name> Watchdog — <timestamp> ━━━

Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.

Also: network_watchdog now logs "Network healthy " on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
2026-05-24 17:20:55 -04:00
Gmer4Lfe b78fdddb04 fix(watchdogs): Tailscale false positive + system watchdog banner
network_watchdog: Tailscale JSON uses "BackendState": "Running" (with space)
but grep pattern matched "BackendState":"Running" (no space) — always failed.
Switched to grep -qE with \s* to handle both forms.

system_watchdog: add echo banner at run start so its output is visually
separated from docker_watchdog in orchestrator logs. Previously appeared
as a continuation of the docker section since docker_watchdog has a banner
but system_watchdog did not.
2026-05-24 17:11:54 -04:00
Gmer4Lfe f85493cf35 watchdogs: fix Tailscale error propagation and docker watchdog output order
- network_watchdog: remove ISSUES++ for Tailscale — it's notify-only by
  design, no restart possible; cascading to system watchdog exit 1 was noise
- docker_watchdog: move banner to top of cycle so body errors appear under
  the correct section header, not visually inside Resource Manager output
2026-05-24 16:56:32 -04:00
Gmer4Lfe 4535804c5d Add System/network_watchdog.sh — services-layer connectivity watchdog
Checks:
  1. Internet reachability (gates all remaining checks)
  2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
  3. Tailscale status — notify only, no auto-restart
  4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart

Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle

Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
2026-05-22 20:51:22 -04:00