Structural reorganization: watchdog taxonomy + server_reboot integration

Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
This commit is contained in:
Gmer4Lfe
2026-05-22 20:17:22 -04:00
parent 670f25fbbd
commit cf21efaeea
18 changed files with 962 additions and 824 deletions
@@ -13,7 +13,7 @@ adding or reordering scripts at array start.
## ━━━ CONTENTS ━━━
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
- [webgui_restart.sh](#webgui_restartsh)
- [webgui_watchdog.sh](#webgui_watchdogsh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
@@ -32,7 +32,7 @@ adding or reordering scripts at array start.
All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.
**Daemon scripts** (`webgui_restart.sh`): run on every cycle. Without `--log`, only
**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
Per-check detail suppressed.
@@ -76,7 +76,7 @@ generates veth messages — these will appear in syslog if the filter isn't acti
---
## webgui_restart.sh
## webgui_watchdog.sh
### Escalation Logic
@@ -113,7 +113,7 @@ WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
```bash
# Check which services are running:
webgui_restart.sh --status
webgui_watchdog.sh --status
# Try manual restart sequence (same as the script):
/etc/rc.d/rc.nginx restart
@@ -196,8 +196,8 @@ grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
```bash
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
# Look at the webgui_restart.sh escalation — step 2 (php-fpm restart) is specifically
# for worker exhaustion. If webgui_restart.sh is regularly hitting step 2, the
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
# pm.max_children value may still be too low, or there's a PHP worker leak.
# Check running worker count: