Commit Graph
16 Commits
Author SHA1 Message Date
Gmer4Lfe 7a12c3eee6 Fix watchdog orchestrator schedule comment — every 15 min not every minute 2026-06-19 18:08:29 -04:00
Gmer4Lfe 82810ab168 Add conf_cache_watchdog.sh — watchdog-driven persistent conf backup
Writes partner confs from RAM cache to /boot/config/.cache/vv/d/ while
remote is offline, and removes the backup when remote comes back. Called
each minute via SYSTEM_WATCHDOG_SCRIPTS so crashes and power loss are
covered — not just graceful shutdowns.
2026-06-19 18:06:31 -04:00
Gmer4Lfe 3964f6fb46 audit echo vs log across all scripts — outcomes always visible, verbose for per-item loops 2026-06-14 12:40:15 -04:00
Gmer4Lfe 27bfc21cb0 Platform-agnostic refactor: eliminate OS-specific hardcodes from core scripts
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.

New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.

Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
2026-06-14 00:59:19 -04:00
Gmer4Lfe 9c3ace95a7 Auth stack certs tab, arrs db fallbacks, cert monitor cache, conf parser fix
- Auth stack: fold cert monitor into Auth Stack page as fourth tab (Certs);
  remove standalone cert page and top-level tab
- cert_monitor.sh: write JSON status cache to State_Files/cert_status.json
  after each run; expose per-domain days/expiry via _CERT_DAYS/_CERT_EXPIRY globals
- api/cert.php: new — serves cached cert status; falls back to configured
  domains as UNKN when no cache exists; POST action=run triggers live check
- arrs db fallbacks: vv_arr_cleanup_stats/discovery_stats/recovery_stats now
  read from data/*.db files when log JSON files don't yet exist
- config.php vv_conf_vars(): unescape bash \$ → $ so passwords with dollar
  signs read correctly from conf files
- host1.conf: fill in HOST1_NPM_USER/PASS and HOST1_LLDAP_USER/PASS
- Partnership adapter pattern: Unraid-specific container logic extracted to
  Plugin/unraid/Partnership/; platform-agnostic structure stays in Partnership/
- First-run wizard: uniform multi-step flow for all hosts; HOST2 pull moved
  to checklist; auto SSH keygen and API key creation on save
- api/checklist.php: live setup checklist with pull_master action
- Fullscreen toggle: hide Unraid header/menu; state persists via localStorage
2026-06-05 23:17:30 -04:00
Gmer4Lfe 369a9e6c19 Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites
- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename)
- Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API
  for storage health, service management, mover, user scripts, notifications,
  disk temps, and platform command validation
- Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR,
  auto-source Plugin/$PLATFORM/adapter.sh after common.sh
- Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg,
  disks.ini, and validate_unraid_cmd calls with platform_*() functions across
  watchdogs, orchestrators, and System_Essentials scripts
- Update all documentation: rename refs, update webgui escalation logic,
  add platform adapter section to Plugin README, update main README with
  portability vision and corrected self-healing stack description
2026-06-04 18:14:34 -04:00
Gmer4Lfe 08fc551d9f Add verbose log() coverage across all watchdog scripts
docker_watchdog: config dump at startup (thresholds/limits), skip list shown when active, per-container healthy log for Tier1 required + mem/CPU monitored containers
stability_watchdog: config dump with all tier thresholds, log() on pass for rootfs/log/tmp/load/zombies/NIC checks (previously silent on clean)
resource_watchdog: config dump with all pressure thresholds and container lists, log normal pressure state with live RAM/load values
system_watchdog: per-script timing on each child script run
network_watchdog: config dump (internet URL, DDNS domain/container, NPM URL, strike limit)
storage_watchdog: config dump (growth threshold, log max, paths, suppress ceilings)
webgui_watchdog: log nginx worker and php-fpm worker counts on healthy check
2026-06-01 20:56:46 -04:00
Gmer4Lfe 5e89f334bb output: watchdog visibility — parity/grace/disabled exits now echo, docker_watchdog container count, api_key_renew status always visible 2026-06-01 20:22:52 -04:00
Gmer4Lfe d3454a3dab docs: update watchdog headers — single-pass cron model, correct caller refs, stability vs system watchdog 2026-06-01 19:31:16 -04:00
Gmer4Lfe 6b30768853 webgui_watchdog: mark executable 2026-05-24 23:33:39 -04:00
Gmer4Lfe ea9f39a803 watchdogs: make healthy-state outputs always visible
log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
2026-05-24 17:25:33 -04:00
Gmer4Lfe afd9129869 watchdogs: add echo banners to network/storage/stability
All watchdog scripts now open with a consistent echo banner:
  ━━━ <icon> <Name> Watchdog — <timestamp> ━━━

Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.

Also: network_watchdog now logs "Network healthy " on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
2026-05-24 17:20:55 -04:00
Gmer4Lfe b78fdddb04 fix(watchdogs): Tailscale false positive + system watchdog banner
network_watchdog: Tailscale JSON uses "BackendState": "Running" (with space)
but grep pattern matched "BackendState":"Running" (no space) — always failed.
Switched to grep -qE with \s* to handle both forms.

system_watchdog: add echo banner at run start so its output is visually
separated from docker_watchdog in orchestrator logs. Previously appeared
as a continuation of the docker section since docker_watchdog has a banner
but system_watchdog did not.
2026-05-24 17:11:54 -04:00
Gmer4Lfe f85493cf35 watchdogs: fix Tailscale error propagation and docker watchdog output order
- network_watchdog: remove ISSUES++ for Tailscale — it's notify-only by
  design, no restart possible; cascading to system watchdog exit 1 was noise
- docker_watchdog: move banner to top of cycle so body errors appear under
  the correct section header, not visually inside Resource Manager output
2026-05-24 16:56:32 -04:00
Gmer4Lfe 4535804c5d Add System/network_watchdog.sh — services-layer connectivity watchdog
Checks:
  1. Internet reachability (gates all remaining checks)
  2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
  3. Tailscale status — notify only, no auto-restart
  4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart

Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle

Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
2026-05-22 20:51:22 -04:00
Gmer4Lfe cf21efaeea Structural reorganization: watchdog taxonomy + server_reboot integration
Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
2026-05-22 20:17:22 -04:00