Structural reorganization: watchdog taxonomy + server_reboot integration
Watchdog renames and moves: system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots) storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention) New thin orchestrator: Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain System/ subfolder is the growth seam for future system component watchdogs master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui server_reboot.sh: Calls array_stopping.sh before VM shutdown for guaranteed safe array stop Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
This commit is contained in:
@@ -13,7 +13,7 @@ adding or reordering scripts at array start.
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
|
||||
- [webgui_restart.sh](#webgui_restartsh)
|
||||
- [webgui_watchdog.sh](#webgui_watchdogsh)
|
||||
- [inotify_tuning.sh](#inotify_tuningsh)
|
||||
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
|
||||
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
|
||||
@@ -32,7 +32,7 @@ adding or reordering scripts at array start.
|
||||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||||
lines only appear when `--log` is passed.
|
||||
|
||||
**Daemon scripts** (`webgui_restart.sh`): run on every cycle. Without `--log`, only
|
||||
**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
|
||||
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
|
||||
Per-check detail suppressed.
|
||||
|
||||
@@ -76,7 +76,7 @@ generates veth messages — these will appear in syslog if the filter isn't acti
|
||||
|
||||
---
|
||||
|
||||
## webgui_restart.sh
|
||||
## webgui_watchdog.sh
|
||||
|
||||
### Escalation Logic
|
||||
|
||||
@@ -113,7 +113,7 @@ WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
|
||||
|
||||
```bash
|
||||
# Check which services are running:
|
||||
webgui_restart.sh --status
|
||||
webgui_watchdog.sh --status
|
||||
|
||||
# Try manual restart sequence (same as the script):
|
||||
/etc/rc.d/rc.nginx restart
|
||||
@@ -196,8 +196,8 @@ grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
|
||||
|
||||
```bash
|
||||
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
|
||||
# Look at the webgui_restart.sh escalation — step 2 (php-fpm restart) is specifically
|
||||
# for worker exhaustion. If webgui_restart.sh is regularly hitting step 2, the
|
||||
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
|
||||
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
|
||||
# pm.max_children value may still be too low, or there's a PHP worker leak.
|
||||
|
||||
# Check running worker count:
|
||||
|
||||
Reference in New Issue
Block a user